Interaction Loop¶
AutoMobile runs an observe → execute → observe loop with UI-stability checks before and after the action. Observation captures the current UI state and hierarchy; the client sends an action, and AutoMobile returns the result with a fresh observation, so the next choice never relies on stale screen state.
sequenceDiagram
participant Agent as AI Agent
participant MCP as MCP Server
participant Device
Agent->>MCP: 🤖 Interaction Request
MCP->>Device: 👀 Observe
Device-->>MCP: 📱 UI State/Data (Cached)
MCP->>Device: ⚡ Execute Actions
Device-->>MCP: ✅ Result
MCP->>Device: 👀 Observe
Device-->>MCP: 📱 UI State/Data
MCP-->>Agent: 🔄 Interaction Response with UI State
Most action tools return the updated observation. Use
observe to inspect the current screen without changing it. When a screen is
loading, wait for the target state before acting and prefer stable text,
resource IDs, content descriptions, or app-defined test tags over coordinates.
The default observe skeleton marks a disabled Android or iOS control with
enabled: false. Enabled rows omit the field. Toggle rows also carry their
checked boolean. Adjustable range widgets carry adjust: an Android SeekBar,
AbsSeekBar, RatingBar or Material Slider class, or a finite range-info with a
scroll_forward, scroll_backward or set_progress action. A read-only
ProgressBar does not. Every row with valid numeric state shows it as
range: {current, min, max}. Unlabelled range controls share the toggle’s
container and preceding-sibling label rules in both skeleton and searchable
projections. Selected and focused state remain in full/raw output.
The swipeOn tool compares the resolved target’s range with the post-action
hierarchy by resource ID (falling back to element ID), using bounds to
distinguish duplicate IDs. Agreeing merged and window copies produce one
effect.value: {before, after} only when numeric state changes. Missing, invalid
or ambiguous state omits that field. Full/raw hierarchy and element output
preserve the explicit disabled enabled value. The marker describes state and
does not change tapOn selection or execution. An enabled-state transition
appears in hierarchy diffs and prevents the settle comparator from treating the
two captures as equal.
The skeleton stays actionable-only; non-actionable rows such as static status
text go to context. A positive waitFor whose only element predicate is
text or textAny first searches the hierarchy, then the rows of both arrays
(their label and sublabel). Any text an agent can read in the default
output is therefore awaitable. Compound, container- and layer-scoped
waits, selectionStrategy and absent stay on the hierarchy search, which
describes one source node. A text that any hierarchy node carries also stays
there, so a row label hoisted from an off-screen child cannot override that
search’s visibility verdict.
The row search skips a row whose label comes from several sources when any
hierarchy text source inside its bounds that the label or sublabel contains has
no visible part on screen. Rows do not record their sources, so the search infers
contribution from bounds and text. The condition form follows the same rules for
for: "appear" and for: "textEquals" with only text (textEquals keeps its
exact, case-sensitive comparison against the row). A clickable wait needs a
tappable source node, disappear must not pass on a hierarchy miss, and
countStable counts hierarchy nodes, so those stay hierarchy-only.
The hierarchy appear and disappear predicates ignore a match on a tappable
row with hoisted descendant text when the selector matches none of the row’s
text that only visible sources back. That text is the row’s own text, its
composed label when a visible descendant supplies the folded part, and each
visible descendant’s label. The comparison follows the resolver’s exact, contains
and regex rules, and an unscoped wait then takes any other match.
Selector resolution for tapOn and other actions skips this check: a row stays
tappable while part of it is on screen and uncovered, whichever sources
built its label. The tap point lies in that part, away from the keyboard,
application windows above the row’s window and AutoMobile’s prototype windows.
For a row inside a scroll container, it also avoids controls drawn over the row
in its own window. iOS counts a control outside that container, or after the row
inside it. Android counts a control outside the container that comes after it in
capture order, and only in captures with per-window trees. When covers leave no
exposed point, the tap fails with a no-visible-tap-area error that names the
cover; the same check makes observe report that row occluded, with no
affordances. A row with no on-screen part fails with an error that says to
scroll it into view.
iOS simulator lock readings are shared per device. Unlocked known readings use
knownTtlMs (30 seconds), locked ones lockedTtlMs (2 seconds), and unknown
ones failureTtlMs (5 seconds). The first known reading after
invalidation is capped by settleTtlMs (2 seconds); subsequent readings use the
normal TTL. Stale in-flight reads cannot consume this settling period. Button
presses, wake-and-unlock (before and after), and device release invalidate the
cache. Observe waits at most 750 ms for a reading. The
deviceLockUnknown.reason field is pending while the shared 4-second probe
continues, or timeout, error, or unparseable after a failed read. Unknown
readings omit deviceLock; they never imply an unlocked device. Probe reads
never reject.
The cache serves an expired known reading stale while one background refresh runs
(120 seconds past expiry for unlocked, 10 seconds for locked; staleUnlockedMs /
staleLockedMs). A burst of observes on a loaded host therefore does not report
deviceLockUnknown because the TTL lapsed (#15311). Invalidation drops the
stale reading; only a device with no recent reading can still report pending.
A successful power press always leaves the device locked, but it acknowledges
before SpringBoard publishes the new lockstate, so it invalidates with
expectLocked. For lockTransitionMs (5 seconds) the cache re-polls an unlocked
reading every lockTransitionPollMs (250 ms) instead of caching or serving it.
Observe then reports pending rather than a pre-transition locked: false
(#17770). The first locked reading ends the window; past it the host reading is
trusted. Any other invalidation cancels a pending transition. Caller abort
detaches cache/budget waits with an unknown pending reading and message
caller aborted; the shared read continues and can populate the cache. A
notifyutil executor abort becomes an unknown error reading.
Settled embedded observations¶
A navigation-class action (tapOn, tapAny, openLink, homeScreen,
recentApps, a navigation pressButton, and a submitting sendKeys) replaces
the screen. Android can return the destination’s hierarchy before it finishes
inflating, missing a child that has not attached yet. For Settings rows that is
the switchWidget node that alone carries toggle / checked, so the parent
row’s content-derived s2-… elementId would change on the next observe.
For those actions AutoMobile gates the embedded capture on hierarchy
stability, using the settle primitive of standalone
observe(waitFor: {for: "stable"}). It re-observes until two consecutive
hierarchies are structurally equal, bounded at 1000 ms and cancelled with the
request. In-place and scroll actions keep their single capture.
The settle treats a node as volatile for its remaining comparisons when only the node’s text or visibility changes in a comparison and in either of the two before it. Examples: a running clock or stopwatch, a paused stopwatch blinking its time text, a typing indicator and a live counter. Looking back two comparisons catches a blink whose polls land on the same phase twice in a row. The settle ignores the node’s changes in later comparisons, so such a screen settles once everything else holds still. The two comparisons that mark the node still count, so a real transition that rewrites one node’s text twice and then stops does not pass for a ticking node.
The first comparison may be the action’s own capture against the first poll when both show the same screen. It feeds only volatile-node detection, never the stable pair. The settle identifies the node by resource id, bounds and sibling position. Structural changes still count: a node added or removed without a counterpart, a volatile node changing anything else, and any change to another node. The settle forgets volatile nodes when consecutive reads are not the same screen.
The budget is a hard cap. Each poll read, the terminal back-stack read and the
attribution reconciliation capture race the remaining budget. The loop aborts
the read the budget cuts off at once and returns the newest trustworthy capture
it already has. A stable pair whose back-stack read or reconciliation capture the
budget cut off is unconfirmed and reports settled: false. No poll starts with
less than 100 ms left, since such a read can only return a stale cache.
While the only changes since the last tick are volatile nodes ticking or
blinking, the loop extends the budget by the smaller of 500 ms and half the
budget. A running stopwatch then has room for the three reads and the back-stack
read that prove it: at about 300 ms per read on an emulator, roughly 1.4 s. Any
other change withdraws the extension, so a still-moving screen stops at the plain
budget. A wall-clock fence at the budget plus a margin (the smaller of
1000 ms and the budget) still aborts the loop’s signal. It races the whole gate,
for terminal work outside those races and a read that never yields. The tapOn
tool takes a per-call settleTimeoutMs (0–3000 ms) that replaces the 1000 ms
budget; 0 skips the gate and returns the action capture with settled: false.
Every action observation carries a settled boolean saying whether that
capture passed the gate:
settled: true— two consecutive stable hierarchies: the settled post-action screen. Android synthetics2-…element ids are valid only for the observation that returned them. A sibling leaving the screen can change an id’s suffix; re-observe for the new id if an old id is stale.settled: false— the capture was not stability-checked. Either the action was not navigation-class, or the bound expired on a screen that never reaches structural stability (an animation that moves or adds nodes, a list still loading). Re-observe if screen completeness matters.
When the gate ran, the observation carries settleMs (milliseconds the gate
spent re-observing) and settlePolls (how many re-observations it took). A
settled: false result with settleMs near the budget means the screen
kept changing or device reads were slow; settlePolls tells the two apart. A
settle read that fails or is cancelled returns the action’s own capture with
settled: false and settleMs but no settlePolls. Both fields are absent when
the action was never gated.
In diff mode (--actions-diff-observe) settled, settleMs and settlePolls
ride on the diff alongside activeWindow and freshness, so one accessor works
in both modes. A diff with no node, display or field change, projected or not,
carries unchanged: true when its skeleton, context and keyboard equal the
ones the baseline’s response delivered. It then leaves those three out; every
other diff carries them. A baseline whose response carried no skeleton never yields
unchanged: true. The advertised diff arm
requires skeleton or unchanged.
A skeleton-projected diff leaves bounds-only changes (moves, resizes or both) out of
changed, because the skeleton already has the new bounds. The diff drops them for
nodes with no affordance. It counts them for actionable nodes in
relayout: {count, dx?, dy?}. dx and dy appear only when every counted change
shares the same non-zero left or top offset. A resize in place adds neither.
A requested post-action screenshot is published under the same observationId
as the hierarchy it should match. Only a settle-gated action can replace its
hierarchy with a later capture, so only it defers the screenshot to the
response boundary and captures the adopted frame. Every other action captures
the screenshot right after its final hierarchy read. The published
observation carries screenshotHierarchyLagMs: screenshotCapturedAt (host
file time) minus the hierarchy’s host receipt time (viewHierarchy.receivedAt).
Both operands are on the host clock, so device clock skew cannot shift it.
Apple platforms share the host clock and fall back to the device updatedAt
when no receipt time exists; an Android tree without one gets no lag. Above
500 ms (SCREENSHOT_HIERARCHY_LAG_THRESHOLD_MS, injectable through the settle
context) it also carries screenshotHierarchyMismatch: true and a
freshness.warning. A negative lag (screenshot captured before the tree
arrived) is never marked.
Android one-shot native captures without prototype hiding bound the CtrlProxy
request at ANDROID_CTRLPROXY_CAPTURE_TIMEOUT_MS (3000 ms) before ADB fallback.
The CtrlProxyCaptureCooldown shares per-device failures across
TakeScreenshot instances for 60,000 ms, matching the observation-stream reprobe
interval. Timeouts and capture failures arm it; caller cancellation and the
rate-limit result do not. Captures during the cooldown skip CtrlProxy and report
ctrlproxy_recent_failure. The first capture at expiry probes again, a failed
probe re-arms the cooldown, and success clears it. Explicit display captures
keep their CtrlProxy error instead of falling back, including during cooldown.
Prototype-hidden captures bypass this cooldown, keep the 10,000 ms timeout,
and never fall back to ADB.
Successful one-shot Android results carry screenshotCaptureSource,
screenshotFallback, optional screenshotFallbackReason and
screenshotStageDurationsMs (ctrlProxyMs?, adbMs?, totalMs). The injected
host timer measures them through encoding and persistence; an omitted stage was
not attempted. Observation-scoped state keeps those diagnostics only for the
matching screenshot path; observe keeps its existing freshness meaning of
screenshotCaptureSource. Stream screenshot updates forward the same optional
stage timings alongside their backend source and fallback fields. Each one-shot
capture logs its source, fallback reason and stage durations.
After an Android hierarchy sync exhausts its budget with at least 3000 ms of waiting and no reply, later reads on that client probe for at most 2000 ms. That lasts until a read succeeds or the 60,000 ms cooldown expires; shorter caller budgets stay (#18151).
The default compact action metadata section is on its own page: