Skip to content

๐Ÿงฐ Tools

Every tool works from the CLI (bunx @kaeawc/auto-mobile --cli <tool>), MCP, or the daemon HTTP endpoint โ€” same names, same arguments.

๐Ÿ“– Full arguments: the registered schema (schemas/tool-definitions.json) or automobile:tools. Availability varies by platform and feature gates: see the Platforms and Needs columns.

๐Ÿ” Platforms and Needs columns

Each tool’s _meta["automobile/requires"] list, shown as emoji in the tables below: Platforms lists the devices the tool drives, Needs lists what to turn on first. Hover an emoji for its name. On a phone, tap Show descriptions above a table to add the What it does column.

Emoji Column Requirement Meaning
๐Ÿค– Platforms platform:android the tool drives Android devices
๐ŸŽ Platforms platform:ios the tool drives iOS devices
โŒš Platforms platform:watchos the tool drives watchOS devices
๐Ÿ“บ Platforms platform:tvos the tool drives tvOS devices
๐Ÿฅฝ Platforms platform:visionos the tool drives visionOS devices
๐Ÿงฉ Needs sdkEmbed start the daemon with --embedded-sdk
๐Ÿž Needs debugBuild start the daemon with --debug (the debug feature flag)
๐Ÿšฉ Needs featureFlag:<key> some or all calls are refused while that feature flag (named beside the emoji) is off; enable it with auto-mobile --daemon set-feature-flag <key> true
  • A tool may still answer unsupported for an action its description limits to one platform.
  • A โ€” Platforms cell: a host tool that needs no device. A โ€” Needs cell: nothing to turn on.

๐Ÿท๏ธ Tool discovery metadata

  • ๐Ÿ“‹ tools/list entries carry a human-readable title and SDK ToolAnnotations.
  • ๐Ÿ›ก๏ธ Hints are untrusted client guidance, never authorization.
  • ๐Ÿ—‚๏ธ Classification lives in src/server/toolAnnotations.ts.
  • ๐Ÿ”€ Mixed-action tools take the conservative union of all supported actions.
  • ๐Ÿ“ฆ The wire omits default readOnlyHint: false, idempotentHint: false, and openWorldHint: true.
  • ๐Ÿ“– Read-only tools omit the meaningless destructive and idempotent hints.
  • ๐ŸŽ›๏ธ Controls always declare destructiveHint explicitly.
  • ๐Ÿ”‘ Metadata is independent of deviceReadOnly dispatch and ownership enforcement.
  • ๐Ÿงช readOnlyHint follows deviceReadOnly, except tools that also write host state (compareScreenshot record mode).

๐Ÿ” Shared options

  • ๐Ÿ”‘ sessionUuid โ€” run on a held device session (device tools)
  • ๐Ÿ’ก keepScreenAwake โ€” wake and keep a physical Android device awake (default on at session setup)
  • ๐Ÿชž raw / project (skeleton default, full) / includeExtras โ€” observation output on action tools
  • ๐Ÿ”— resourceLink โ€” getLogs, exportPlan, and videoRecording stop: return the complete result as an automobile:tool-output resource and a resource_link block, keeping only success, error, and the result’s required fields (a stop’s action, count, recordings, each bounded) inline (default off, inline)
  • ๐Ÿ“ฑ deviceId โ€” target a device explicitly on reads

Reading a device you do not own (deviceId)

A read never writes settings on a device another session holds. A sessionless read of an unheld running device may ready it (install CtrlProxy / runner) to answer. See which read tools ready an unheld device.

๐Ÿ‘€ Observe & navigate

Tool What it does Platforms Needs
๐Ÿ“ธ snapshotOf Save a PNG crop of one exposed element or a platform-native screen rectangle ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ†š compareScreenshot Visual regression: record a named screenshot baseline or compare the current screen, an element, or a rectangle against it, ignoring masked regions and system bars ๐Ÿค– ๐ŸŽ ๐Ÿ“บ โ€”
๐Ÿ‘๏ธ observe Get screen view hierarchy and screenshot, with optional PNG crop from a fresh settled capture of one display ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ ๐Ÿฅฝ โ€”
๐ŸŽฏ hitTest Preview hierarchy-bounds candidates at a platform-native screen point ๐Ÿค– ๐ŸŽ โ€”
๐Ÿงญ explore Automatically interact with the app to discover screens and record transitions in the navigation graph ๐Ÿค– ๐ŸŽ ๐Ÿ“บ ๐Ÿงฉ
๐Ÿ—บ๏ธ navigateTo Navigate to a learned screen by replaying recorded transitions from the navigation graph ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐Ÿ•ธ๏ธ getNavigationGraph Get the learned navigation graph for the foreground or given app ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐Ÿ”Ž identifyInteractions Suggest likely interactions ๐Ÿค– ๐ŸŽ ๐Ÿž
๐Ÿงช prototype Show agent-authored UI on the device ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ’ป exportPrototypeWeb Write the automobile:prototype/<id>/web page to an HTML file โ€” โ€”
๐Ÿ“ก blePeripheral Manage a session-owned virtual BLE peripheral on Android netsim โ€” โ€”
๐Ÿ”ฆ highlight Draw a red hand-drawn circle around a UI element ๐Ÿค– ๐ŸŽ โ€”

๐Ÿ‘† Interact

Tool What it does Platforms Needs
๐Ÿ“ tapAt Tap, long press, or double tap at screen coordinates in native, normalized, or percent units ๐Ÿค– ๐ŸŽ โŒš โ€”
๐Ÿ‘† tapOn Tap an element by text/content-desc, resource-id, or Android test tag ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ ๐Ÿฅฝ โ€”
๐Ÿค tapAny Tap any clickable element ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ‘‰ swipeOn Swipe/scroll on screen or elements ๐Ÿค– ๐ŸŽ โŒš โ€”
โ†”๏ธ swipeAt Swipe from one screen point to another, including diagonals, in native, normalized, or percent units ๐Ÿค– ๐ŸŽ โ€”
๐Ÿซณ dragAndDrop Drag and drop element ๐Ÿค– ๐ŸŽ โ€”
ใ€ฐ๏ธ dragPath Drag a finger along a polyline with segment timing, an initial hold and a release hold ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ–ฑ๏ธ hoverAt Hover a stylus or mouse over a path with no contact ๐Ÿค– ๐ŸŽ โ€”
๐ŸคŒ pinchOn Pinch to zoom ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ–๏ธ multiTouch Simultaneous 2-5 finger tap, long press, or swipe at a screen point ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”ค sendKeys Execute ordered typing, clear, raw-key, and semantic IME commands ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ ๐Ÿฅฝ โ€”
๐Ÿ•น๏ธ focusNavigate Move D-pad focus to an element on a focus-driven UI such as Android TV or Apple TV ๐Ÿค– ๐ŸŽ ๐Ÿ“บ โ€”
โ˜‘๏ธ setUIState Set multiple form fields by desired state ๐Ÿค– ๐ŸŽ ๐Ÿž
๐Ÿ“ selectAllText Select all text in focused input ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”˜ pressButton Press a device or navigation button ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ โ€”
โŒจ๏ธ keyboard Open, close, detect, list/select installed Android IMEs, or switch the AutoMobile typing profile ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ“‹ clipboard Clipboard operations ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ—ฃ๏ธ siri iOS: activate Siri with a spoken phrase, e.g. to invoke an App Shortcut / App Intent, then return the observation ๐Ÿค– ๐ŸŽ โ€”

๐Ÿ“ฆ Apps, files & data

Tool What it does Platforms Needs
๐Ÿ—‚๏ธ listApps List installed apps on a device, with optional display label and launchable fields when reported by the platform or transport ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ ๐Ÿฅฝ โ€”
๐Ÿš€ launchApp Launch app by package name ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ ๐Ÿฅฝ โ€”
๐Ÿ›‘ terminateApp Terminate app by package name ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ ๐Ÿฅฝ โ€”
๐Ÿ’ฅ crashApp Intentionally crash a running app through the platform crash path ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ โ€”
๐Ÿ”„ appLifecycle State-preserving background-process kill for saved-state restoration tests ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”จ buildApp Build an app from source for a simulator or emulator and return the installable artifact path โ€” โ€”
๐Ÿ“ฒ installApp Install app on device ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ ๐Ÿฅฝ โ€”
๐Ÿ—‘๏ธ uninstallApp Uninstall app by package name or bundle identifier ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ ๐Ÿฅฝ โ€”
๐Ÿ” inspectPackageSigning Fresh read of one app’s presence and signing ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”— getDeepLinks Query app deep links ๐Ÿค– ๐ŸŽ ๐Ÿ“บ โ€”
๐Ÿ“ฅ putAppFile Write files into a bounded logical storage target ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ›’ storeKitTest Test in-app purchases with StoreKit Testing on an iOS simulator ๐Ÿค– ๐ŸŽ โ€”
๐Ÿงน resetAppLogs Reset explicitly named app-container log files on the session’s device, reporting a per-path outcome ๐Ÿค– ๐ŸŽ โ€”
โš™๏ธ getPreference Read an Android system property, Android SharedPreferences key, or iOS UserDefaults key ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”ง setPreference Write an Android system property, Android SharedPreferences key, or iOS UserDefaults key and return read-back verification ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ—๏ธ deleteKeychainItem Delete one exact, host-allowlisted, app-owned iOS generic-password Keychain item ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐Ÿ”‘ setKeyValue Set app key-value storage entry ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
โŒ removeKeyValue Remove app key-value storage entry ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐Ÿงผ clearKeyValueFile Clear app key-value storage file ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐Ÿ—„๏ธ listDataStores List app Jetpack DataStore instances ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐Ÿ—ƒ๏ธ getDataStore Read entries from an app Jetpack DataStore instance ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐Ÿงฎ sqlQuery Execute SQL on an app SQLite database, or read a registered Core Data entity on iOS using store and entity ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐Ÿ” resetKeychain Reset secure test state ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ–‡๏ธ setAppLinks Android 12+: change an app’s App Links state so https links open the app directly ๐Ÿค– ๐ŸŽ โ€”
๐Ÿšฆ performanceAudit Pass/fail performance checks of an app on the current screen, against the configured thresholds ๐Ÿค– ๐ŸŽ โ€”
โฑ๏ธ measureStartup Android: measure cold/warm/hot app startup over N launches ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ“œ getLogs Read device logs with filters ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ ๐Ÿฅฝ โ€”
๐Ÿ“ป sendBroadcast Android: send an explicit broadcast to the app under test with a typed intent, e.g. to toggle a debug flag or seed data ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ› ๏ธ backgroundWork Android: read and trigger an app’s background work without waiting hours ๐Ÿค– ๐ŸŽ โ€”

๐Ÿ“ฑ Devices & system

Tool What it does Platforms Needs
๐Ÿ“ƒ listDevices List booted devices โ€” โ€”
๐Ÿ’ฟ listDeviceImages List device images โ€” โ€”
๐Ÿค– getAndroid Find or recover an Android AVD and prepare it for automation โ€” โ€”
๐ŸŽ getApple Find or recover an iOS Simulator and prepare it for automation โ€” โ€”
๐Ÿ—๏ธ provisionDevice Provision exact virtual device โ€” โ€”
๐Ÿงฐ setDeviceResources Set selected device resources and verify native state, including disabled iOS service impacts ๐Ÿค– ๐ŸŽ โ€”
โš–๏ธ reconcileDeviceResources Compare an iOS Simulator with a requested resource map and report typed drift ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ“Œ setActiveDevice Set active device using existing session binding or a sessionless per-MCP-connection device pin that lasts for that connection โ€” โ€”
๐Ÿ”Œ killDevice Kill device โ€” โ€”
โšฐ๏ธ deleteDevice Stop and permanently delete a device, with verified platform-inventory absence โ€” โ€”
๐Ÿ’พ deviceSnapshot Capture, restore or delete a device snapshot ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”ƒ rotate Rotate device orientation ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ” portReverse Android: manage adb reverse mappings so the app or browser on the device reaches a host server at localhost:<port> ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ“ setPosture Set Android device posture or iPhone Duo simulator posture, with an optional Resizable Android display preset ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”‹ statusBar Deterministic status bar for screenshots and video ๐Ÿค– ๐ŸŽ โ€”
๐ŸชŸ windowMode Android: put an app’s task into fullscreen, picture-in-picture, freeform or split screen with a second app, and read the mode and bounds back ๐Ÿค– ๐ŸŽ โ€”
๐ŸŽ›๏ธ rotaryScroll Wear OS: scroll with the rotary crown ๐Ÿค– ๐ŸŽ โ€”
๐Ÿฅฝ glassesInput Android XR glasses touchpad tap and forward/back swipe as focus input ๐Ÿค– ๐ŸŽ โ€”
๐ŸŒ™ ambient Wear OS watches only: enter sends KEYCODE_SLEEP, exit sends KEYCODE_WAKEUP, then reads wakefulness and the watch face’s inAmbient from dumpsys wear_service, reported as ambient ๐Ÿค– ๐ŸŽ โ€”
โŒš wearSurface Wear OS watches only: tiles, watch faces and complications ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ‘‘ rotateCrown watchOS only: rotate the Apple Watch Digital Crown by a signed, non-zero delta in full revolutions, with optional positive velocity โŒš โ€”
๐ŸŒ openLink Open URL in browser ๐Ÿค– ๐ŸŽ ๐Ÿ“บ โ€”
๐Ÿ  homeScreen Go to home screen ๐Ÿค– ๐ŸŽ ๐Ÿ“บ โ€”
๐Ÿ•˜ recentApps Open recent apps ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”” systemTray System tray actions for notifications ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”“ wakeAndUnlock Wake a device and unlock its keyguard ๐Ÿค– ๐ŸŽ โ€”
๐ŸŒ changeLocalization Change locale, time zone, text direction, time format, and calendar system ๐Ÿค– ๐ŸŽ โ€”
๐Ÿฉบ getDeviceState Read device-level state including clock, Do Not Disturb, the connectivity toggles, iOS Simulator biometric enrollment, network condition, and Android emulator virtual sensor values ๐Ÿค– ๐ŸŽ โ€”
๐ŸŽš๏ธ setDeviceState Set device state such as Do Not Disturb, Android connectivity toggles, static location or background route playback on an Android emulator, iOS Simulator, or physical iOS device, iOS Simulator biometric enrollment, and device-wide network condition ๐Ÿค– ๐ŸŽ ๐Ÿšฉ device-condition
๐Ÿ–ฅ๏ธ displayConfig Read or set the visual display configuration โ€” font/text scale, effective display density, light/dark theme, and on Android the runtime screen size, navigation mode and display-cutout emulation โ€” for adaptive-layout, edge-to-edge and large-font accessibility testing ๐Ÿค– ๐ŸŽ โ€”
๐ŸŽจ displayProfile Android: apply an OEM-style display profile in one call, composing displayConfig and setPosture ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”ฎ getSessionRequirementCapability Report the OS a session device requirement would resolve to on this broker host and whether it is satisfiable now, without acquiring a device โ€” โ€”
๐Ÿชช biometricAuth Simulate biometric auth: Android emulator/SDK hook, or iOS via the app’s AutoMobile SDK or Simulator ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ“ณ shake Shake device; iOS Simulator and Android emulator only ๐Ÿค– ๐ŸŽ โ€”
๐Ÿงฒ setSensor Set or reset one Android emulator virtual sensor through the emulator console ๐Ÿค– ๐ŸŽ โ€”
๐ŸŽ™๏ธ injectAudioInput Play a WAV file or a built-in tone into an Android emulator’s or iOS Simulator’s microphone, so an app under test records known audio ๐Ÿค– ๐ŸŽ โ€”
โน๏ธ stopAudioInput Stop an audio injection started by injectAudioInput, on the device it runs on ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”Š captureAudio Record up to 20 s of an Android emulator’s or iOS Simulator’s audio output and report its level ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ“ž phoneCall Simulate a phone call: Android emulator gsm commands, or CallKit via the app’s AutoMobile iOS SDK ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ’ฌ sendSms Send simulated incoming SMS: Android emulator, or a notification via the app’s AutoMobile iOS SDK ๐Ÿค– ๐ŸŽ โ€”
๐ŸŒฑ seedContent Seed Android contacts, calendar events, call history and SMS inbox, tagged automobile.<namespace> ๐Ÿค– ๐ŸŽ โ€”
๐ŸŽต mediaSession Android media sessions: read playback state, position, speed, metadata and actions ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ“ฃ postNotification Post notification via Android SDK hooks or iOS/watchOS/tvOS simulator simctl push ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ โ€”
๐Ÿ”• getNotificationPolicy Read app notification/DND policy state ๐Ÿค– ๐ŸŽ โ€”
๐Ÿคซ setNotificationPolicy Set app notification/DND policy state ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ›‚ getAppPermissions Read app permission state ๐Ÿค– ๐ŸŽ โ€”
๐ŸŽซ setAppPermissions Grant or revoke app permissions ๐Ÿค– ๐ŸŽ ๐Ÿ“บ โ€”
โ˜ ๏ธ killProfile Android: simulate an OEM background-kill policy on a stock device with standby buckets, appops, Doze, permission revocation, foreground-change kills, timed screen-off cleanup, boot-receiver observations and force-stop ๐Ÿค– ๐ŸŽ โ€”
๐Ÿญ manufacturerProfile Android: apply an OEM manufacturer profile on a stock device by composing the keyboard profile, killProfile and displayProfile executors ๐Ÿค– ๐ŸŽ โ€”
โš ๏ธ systemAlerts Set how OS permission and system alerts are handled before and after each UI interaction on this device, including executePlan steps ๐Ÿค– ๐ŸŽ โ€”

๐ŸŒ Network, plans & recording

Tool What it does Platforms Needs
๐Ÿ“ถ network Control network capture and error simulation ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐ŸŽญ mockNetwork Add mock network response rule ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐Ÿงฝ clearMockNetwork Clear mock network response rules ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
๐Ÿ“Š getNetworkGraph Get aggregate captured network graph ๐Ÿค– ๐ŸŽ ๐Ÿงฉ
โ–ถ๏ธ executePlan Execute YAML plan steps ๐Ÿค– ๐ŸŽ โŒš ๐Ÿ“บ ๐Ÿฅฝ โ€”
๐Ÿ”’ criticalSection Synchronize multiple devices at a barrier, then run steps serially ๐Ÿค– ๐ŸŽ โ€”
๐Ÿšง barrier Synchronize multiple devices at a barrier, then let all proceed concurrently ๐Ÿค– ๐ŸŽ โ€”
โบ๏ธ recordToolCalls Record MCP tool calls to a YAML plan โ€” ๐Ÿšฉ tool-call-recording
๐ŸŽฌ recordDeviceGestures Start recording on-device user input for exportPlan ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ“ค exportPlan Stop the active recordDeviceGestures or recordToolCalls recording and export a YAML plan โ€” โ€”
๐ŸŽฅ videoRecording Start or stop device video recording ๐Ÿค– ๐ŸŽ ๐Ÿ“บ โ€”
๐Ÿ“ˆ performanceTrace Record a native performance trace ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”ฌ captureTrace Android: record a Perfetto trace from typed categories ๐Ÿค– ๐ŸŽ โ€”
๐Ÿง  heapDump Dump a running app’s heap to a host file ๐Ÿค– ๐ŸŽ โ€”

โ™ฟ Accessibility & tooling

Tool What it does Platforms Needs
๐Ÿฆฎ accessibility Check or control accessibility services ๐Ÿค– ๐ŸŽ โ€”
๐Ÿ”ฒ accessibilityFocus Set accessibility focus on Android and iOS by resourceId, text, or contentDesc ๐Ÿค– ๐ŸŽ ๐Ÿž
๐Ÿซถ accessibilityAction Perform an Android accessibility action on an element, the way TalkBack or Switch Access does ๐Ÿค– ๐ŸŽ โ€”
โœ… accessibilityAudit Audit the accessibility of the current screen ๐Ÿค– ๐ŸŽ โ€”
๐Ÿงฉ setToolEnabled Control tools/list discovery for this MCP session โ€” โ€”

๐Ÿ“บ TV focus navigation

focusNavigate({ until, direction?, maxSteps?, settleMs?, select? }) (#14947) drives focus-based UIs on Android TV and Apple TV. It observes, and while the input-focused element (focusedElement) does not match until (text, contentDesc, resourceId, or a lone label/elementId resolved to the element tapOn would tap) it presses the D-pad arrow toward the nearest match, or direction when given, then observes again. With no focused element or no match on screen it presses down. When a press leaves focus unchanged, or keeps moving focus without getting closer, the other arrows are tried. It returns the path of focused elements and a status: focused, max_steps (default 30, at most 100) or stuck (every arrow from the focused element was tried; an explicit direction stops after two unchanged presses). With select: true it presses enter once focused and reports selected. On Apple TV the arrows are Siri Remote directional presses, select presses the remote’s select button, and focus is re-read after each press (see docs/how-it-works/plat/android/form-factors.md and docs/how-it-works/plat/tvos/xcuitest-spike.md).

๐Ÿ“ Notes

  • โŒจ๏ธ Compact observations fold soft-keyboard keys into one keyboard: { visible: true, package: "โ€ฆ" } summary plus at most one skeleton row (<ime> | Keyboard (โ€ฆ) | input). The row appears only with at least one bounded accessible descendant.
  • ๐Ÿ“ธ Screenshot paths are kept for at least 10 minutes after return unless evicted early under capacity pressure; each path has a screenshotExpiresAt/expiresAt (host epoch ms, not a promise). Retention starts at 128 MiB and scales by 64 MiB per active device up to 1 GiB, within a limit of 4096 files. Each deviceโ€™s newest 3 screenshots (counting its incoming frame) are evicted only as a last resort, and a capture is admitted even if nothing fits. The byte limits and per-device allowance are configurable. After a restart, existing files get a process-start grace of ten minutes.
  • ๐Ÿงน holdSessionOnFailure keeps plan sessions instead of auto-releasing them.

No audio input (audio_input_unavailable)

When a device cannot record audio, observe returns diagnostics: [{ code: "audio_input_unavailable", message, retryable: false }], and a setAppPermissions grant of microphone (iOS) or RECORD_AUDIO (Android) still succeeds but carries the same diagnostics entry. This applies to an iOS simulator on a host with no audio input device, and to an Android emulator launched with -no-audio (AUTOMOBILE_EMULATOR_AUDIO=false) or with hw.audioInput=no. Recordings on that device contain no audio. Treat this as an environment limitation: do not retry, and do not report it as an app failure. Fix the host (for example, make a virtual loopback device the default input) or the AVD audio settings. The message names the cause.

Stylus pointer on tapAt, swipeAt and dragPath

pointer: { toolType: "stylus" } injects a stylus instead of a finger (Android only; never falls back to a finger).

pointer shape Route Needs
{ toolType: "stylus" }, pressure omitted or 1 (tapAt, swipeAt) adb shell input stylus Android 12+ (API 31)
pressure other than 1, tilt (radians, 0 to PI/2), orientation (radians), or buttons (primary, secondary) CtrlProxy instrumentation stylus_input_v1 (Android 12+)
dragPath with any pointer CtrlProxy instrumentation stylus_input_v1 (Android 12+)

On an emulator or userdebug build whose adb shell already runs as root (after adb root) and has the uinput tool, a stylus stroke with pressure, tilt, orientation, buttons and a dragPath also works without stylus_input_v1, through a virtual pen; each stroke starts about 4 seconds later while the pen attaches.

pointer: { toolType: "eraser" } touches with the pen’s eraser end (apps see TOOL_TYPE_ERASER) on tapAt, swipeAt and dragPath, with optional pressure, tilt, orientation and buttons. It needs that same root adb shell with the uinput tool, and is refused before any input everywhere else.

dragPath points also take pressure and tilt (they need pointer), interpolated between points; holdMs and releaseHoldMs become holds at the first and last point. A device or runner that cannot honour the request is refused before anything is sent, naming the missing capability. tilt, orientation and buttons are Android only; doubleTap and a panel display are not supported with a stylus yet. Details and the unverified parts are in the stylus design note.

hoverAt

Hover enter, move and exit over points with no contact (Android only; the reasons iOS is refused are in the stylus design note). It takes the dragPath point shape (x, y, per-point durationMs) plus an optional per-point distance (stylus only), and the same coordinateSpace, display, snapshotId and layer.

toolType chooses who hovers, and apps see the difference:

toolType What the app receives
stylus (default) ACTION_HOVER_* with TOOL_TYPE_STYLUS, SOURCE_STYLUS and an AXIS_DISTANCE value
mouse ACTION_HOVER_* with TOOL_TYPE_MOUSE and SOURCE_MOUSE; no distance; desktop-style UI

It needs the hover_input_v1 runner capability, or, for a stylus hover, an adb shell that already runs as root with a writable /dev/uinput (after adb root on an emulator or userdebug build). A device or runner without either, and iOS, are refused before anything is sent with the missing capability named, and a hover is never turned into a touch. CtrlProxy advertises the capability and injects the hover through its UiAutomation instrumentation (see the stylus design note). The tool type, source and hover actions above are confirmed on API 31 to 37 emulators; the stylus AXIS_DISTANCE value and physical pen devices are unverified.

prototype

Show agent-authored UI on a device: show, patch, inspect, dismiss {id} / {all: true}, status. Android can hold up to four prototypes (replaceOthers: false); iOS shows one at a time. Features (data model, gestures, checks) require the matching device capability and are refused before upload otherwise. Spec: prototype UX. Camera and media support per runtime: camera and media guide.

Watch-sized mockups

Prototypes cannot run on watchOS, but a watch-sized spec gives a quick visual mockup of a watch screen on an iPhone simulator or Android emulator. Float the window in the centre and give the root box the watch screen’s size, a large corner radius (which also clips its content) and a black background:

Case width ร— height (dp/pt) cornerRadius
41 mm 176 ร— 215 36
45 mm 198 ร— 242 40
49 mm Ultra 205 ร— 251 44
{
  "id": "watch-41mm",
  "window": {
    "placement": { "type": "floating", "gravity": "center", "offset": { "x": 0, "y": 0 } }
  },
  "components": {
    "watchRow": {
      "root": {
        "type": "row",
        "style": {
          "width": "fill",
          "background": "#222222",
          "cornerRadius": 16,
          "padding": { "start": 10, "end": 10, "top": 8, "bottom": 8 }
        },
        "children": [{ "type": "text", "text": "{props.title}", "style": { "color": "#FFFFFF" } }]
      }
    }
  },
  "root": {
    "type": "box",
    "style": {
      "width": { "dp": 176 },
      "height": { "dp": 215 },
      "cornerRadius": 36,
      "background": "#000000"
    },
    "children": [
      {
        "type": "scroll",
        "child": {
          "type": "column",
          "style": { "spacing": 6, "padding": { "start": 8, "end": 8, "top": 12 } },
          "children": [
            { "type": "use", "component": "watchRow", "props": { "title": "Drink water" } },
            { "type": "use", "component": "watchRow", "props": { "title": "Stand up" } }
          ]
        }
      }
    ]
  }
}

Put each common watch pattern in components and reuse it with use: a vertically scrolling list of rows, a full-screen metric (one large number with a label and unit), a two-button confirmation (two buttons with weight: 1 in a row, swapped in with visibleWhen), and a now-playing card (title, artist, previous / play-pause / next iconButtons and a linear progress).

The mockup is a phone overlay sized like a watch. It does not simulate the Digital Crown, Always-On or wrist-down dimming, real watchOS or Wear OS fonts and controls, or the round Wear OS screen, and nothing runs on a watch. Check the final design on a real watch. To drive a watch app itself, see Apple Watch simulators.