Tools

Every tool below can be driven three ways: from the CLI (bunx @kaeawc/auto-mobile --cli <tool>), over MCP from an AI client, or directly against the daemon’s HTTP endpoint. The tool names and arguments are the same across all three.

This page reflects the current tool schema. Availability can still vary by platform, runner, and enabled feature gates; inspect the registered schema for the exact arguments supported by your connection.

Observe & navigate

Screen-coordinate contract

observe reports platform-native, current-orientation screen coordinates. The origin is the top-left of the complete current screen, including system UI:

• Android uses physical pixels; iOS uses XCTest logical points.
• screenSize, skeleton bounds, full-hierarchy bounds, and future absolute coordinate input use that same platform-native coordinate space.
• Valid coordinates are half-open: 0 <= x < width and 0 <= y < height.
• A point already in the platform-native space is not density-, inset-, Retina-scale-, canonical-pixel-, or rotation-transformed.

This is separate from the daemon observation-stream’s canonical-pixel mapping. That stream contract is intentional and does not transform MCP observe or native absolute-input coordinates.

observe, observe.screenSize, and tapAt use the device’s current-orientation native coordinate space described above. captureScreenshot returns a native-scale PNG whose orientation follows the device framebuffer: on the iOS Simulator, the framebuffer can remain portrait after rotate, even while the device orientation is landscape (this is simulator framebuffer behavior, not an AutoMobile bug); on Android, the raster rotates with the device. Therefore, after rotation, callers must apply a platform- and orientation-specific transform before correlating iOS observe or tapAt coordinates with captureScreenshot pixels. No such transform is needed on Android.

Tool	What it does
👀 observe	Gets the current screen view hierarchy.
🔍 explore	Explores an app to build a navigation graph.
🗺️ navigateTo	Navigates using the learned navigation graph.
📊 getNavigationGraph	Retrieves the navigation graph for debugging.
🔗 identifyInteractions	Suggests likely interactions.
🖍️ highlight	Draws a visual highlight around a UI element.
🔍 debugSearch	Shows selector matches, the chosen match, and near-misses.

Interact with the UI

Tool	What it does
👆 tapOn	Taps by text, content description, resource ID, or Android test tag; can idempotently ensure a toggle is checked or unchecked.
🎯 tapAny	Taps any clickable element, optionally scoped to a container.
👉 swipeOn	Swipes or scrolls the screen or an element.
↔️ dragAndDrop	Drags one element to another.
🤏 pinchOn	Pinches to zoom.
⌨️ sendKeys	Runs ordered text, clear, raw-key, and semantic-key commands.
⌨️ inputText	Legacy text input retained for compatibility; disabled by default.
🧩 setUIState	Sets multiple form fields to a desired state.
🗑️ clearText	Legacy focused-input clear; disabled by default.
✨ selectAllText	Selects all text in the focused input.
↩️ imeAction	Legacy standalone IME action; disabled by default.
🔘 pressButton	Presses a device or navigation button.
⌨️ keyboard	Opens, closes, or detects the on-screen keyboard.
📋 clipboard	Copies, pastes, clears, or reads the clipboard.

sendKeys accepts one optional field selector and an ordered sequence of up to 100 commands:

{
  "selector": { "text": "Email" },
  "commands": [
    { "action": "type", "text": "name@example.com" },
    { "action": "key", "key": "tab" },
    { "action": "type", "text": "replacement", "operation": "replace", "mode": "a11y" },
    { "action": "key", "key": "enter", "modifiers": ["shift"] }
  ]
}

Text defaults to operation: "insert" and mode: "auto". Modes are auto, a11y, eventLast, eventAll, and eventOnly. They select Android delivery strategies; iOS accepts the same values for cross-platform plans and reports the actual xcuiTypeText mechanism as resolvedMode. Raw keys are enter, tab, escape, backspace, delete, and the four arrow keys; they accept shift, ctrl, alt, and meta. Semantic keys next, previous, done, search, send, and go perform the corresponding IME action and ignore modifiers. A standalone { "action": "clear" } command clears the focused field. Execution stops on the first failure and returns compact command metadata plus the final observation without copying type-command text into the metadata.

inputText reserves an error response for a failed text write. When the text lands but the optional dismissKeyboard cleanup cannot be confirmed, the response stays a success carrying keyboardDismissed: false and a warnings: ["keyboard dismissal failed: ..."] entry — so a client never has to parse prose to decide whether retyping would double the text. Inside executePlan those warnings are promoted to the response’s own warnings list, each entry naming the stepIndex, tool, and (for multi-device plans) the device that reported it, so an ordinary plan does not have to opt into the captureObserveSteps debug trace to see them.

sendKeys is the default text-input path on any AutoMobile release whose CtrlProxy artifacts are 0.0.68 or newer, which is the case for current releases. On older pinned releases inputText and clearText are default-enabled instead and sendKeys is off; enable sendKeys there explicitly with setToolEnabled or --enable-tool sendKeys.

Pinch rotation semantics

pinchOn.rotationDegrees describes how far the two-finger axis rotates during the pinch. The fingers start horizontally and finish on the rotated axis, so a non-zero value combines pinch and rotation. The default 0 is a plain pinch. Android and iOS share this convention.

Apps, files & app data

Tool	What it does
📱 listApps	Lists installed apps with optional label/launchability when reported (device, type, search, profile; default type=launchable).
🚀 launchApp	Launches an app by package name; on Android an app already in the foreground is a success flagged alreadyForeground.
❌ terminateApp	Terminates an app by package name.
💥 crashApp	Intentionally crashes a running app through the platform crash path.
📦 installApp	Installs an APK, app bundle, or IPA.
🗑️ uninstallApp	Uninstalls an app by package name or bundle identifier.
🔗 getDeepLinks	Queries an app’s deep links.
📄 putAppFile	Writes local-file, UTF-8, or base64 content into an app container.
🧾 resetAppLogs	Resets explicitly named app-container log files and their rotated siblings on the session device, with per-path outcomes.
📥 stageSharedStorage	Stages host-file, UTF-8, or base64 fixtures into a bounded Android Downloads namespace for system pickers (Android only).
📁 stageSessionDownloads	Stages fixtures into one bounded child directory of the session device’s shared Downloads tree, with optional reset and per-file media indexing (Android only).
⚙️ getPreference / ⚙️ setPreference	Reads or writes Android system properties, SharedPreferences, or iOS UserDefaults.
🔑 setKeyValue / 🔑 removeKeyValue / 🔑 clearKeyValueFile	Manages an app key-value storage file.
🗃️ listDataStores / 🗃️ getDataStore	Lists or reads Android Jetpack DataStore entries with the SDK adapter.
🗄️ sqlQuery	Executes SQL against an app SQLite database.
🔐 resetKeychain	Resets all Keychain data on an iOS Simulator after explicit confirmation; unsupported on Android and physical iOS devices.

Intentional crash contract

crashApp accepts only an appId; it never accepts a PID, signal, or shell command. Android uses ActivityManager’s VM-crash path for the resolved user. iOS Simulator sends SIGABRT to the exact launchd application process. Physical iOS devices return supported: false and never fall back to normal termination.

Every result reports success, supported, platform, appId, mechanism, timestamp, and confirmed. It reports wasRunning whenever preflight established process state; confirmed crashes also report processId when available and include immediate OS diagnostic evidence. success: true and confirmed: true require fresh, target-specific crash evidence, not merely command dispatch or process disappearance.

Copy a fixture into an app container

{
  "tool": "putAppFile",
  "params": {
    "platform": "ios",
    "target": {
      "domain": "app_containers",
      "appId": "com.example.app",
      "container": "documents"
    },
    "files": [
      {
        "sourcePath": "/Users/me/fixtures/welcome.png",
        "destinationPath": "fixtures/welcome.png"
      }
    ]
  }
}

File containers

Android externalFiles maps to /sdcard/Android/data/{appId}/files. Private containers (documents, cache, and tmp) use run-as and require a debuggable app. iOS simulator containers include documents, library, cache, and tmp.

Devices & system state

Tool	What it does
📋 listDevices	Lists booted devices using the shared device description: identity, runtime, form factor, lifecycle, and session summary; a note points to MCP resources for image detail.
🖼️ listDeviceImages	Lists configured images using the same canonical identity, runtime, display, lifecycle, provenance, and capability inventory shape as the images resource.
🤖 getAndroid / 🍎 getApple	Finds or recovers an Android AVD or iOS Simulator for automation; Android identity includes API level and OS version when known.
🧱 provisionDevice	Provisions an exact virtual-device identity, with optional resource configuration before automation readiness.
⚙️ setDeviceResources	Configures selected device resources and returns verified, unsupported, or unknown state; omitted settings stay unchanged. Disabled by default: enable it with setToolEnabled (case-sensitive setDeviceResources) or --enable-tool setDeviceResources before use.
🔧 setActiveDevice	Sets the active device.
❌ killDevice / 🧹 deleteDevice	Stops a device, or stops and permanently deletes it. Both accept force: true, which drops every AVD-name comparison for a wedged Android emulator — the emulator-console confirmation and the platform kill’s own re-discovery check — and acts on whatever occupies the serial; it does not bypass serial selection, nor the refusals raised when the pooled entry was retired and replaced mid-action, or when no booted target can be identified at all.
📸 deviceSnapshot	Captures or restores a device snapshot.
🔄 rotate	Changes device orientation.
🌐 openLink	Opens web URLs or routes app and universal deep links.
🧰 homeScreen / recentApps / systemTray	Controls core system surfaces and notifications.
🔓 wakeAndUnlock	Wakes and unlocks the keyguard.
🌍 changeLocalization	Changes locale, time zone, text direction, time format, and calendar.
⚙️ getDeviceState / ⚙️ setDeviceState	Reads or changes Do Not Disturb, simulator biometric enrollment, and network condition. getDeviceState also reads back the Android connectivity toggles — airplaneMode, wifiEnabled, bluetoothEnabled, locationEnabled — in a single adb round-trip, so a toggle can be checked before it is flipped; a bare call returns doNotDisturb + connectivity, and include selects any subset. A connectivity field is true/false, or omitted when the device could not answer it (key absent on this API level, or an unparsable value) — omitted never means off. Connectivity is unsupported on iOS: Airplane mode, Wi-Fi, Bluetooth and Location have no simctl/devicectl read verb, and a simulator shares the host’s network stack. Degraded profiles — including offline — are best-effort cellular shaping on an Android emulator (adb emu network …/gsm data off plus a best-effort Wi-Fi disable), reported partial: they may not affect Wi-Fi or app traffic. Only reset to none is fully verified. A session restores the network to a clean none state on release. Unsupported on physical Android and all iOS.
🔠 displayConfig	Reads or sets font/text scale, effective display density, and light/dark theme for adaptive-layout and large-font accessibility testing. Android supports all three fields (density overrides are best-effort on physical devices); the iOS Simulator supports theme only, via simctl ui appearance; physical iOS is unsupported. Disabled by default — enable it with setToolEnabled (case-sensitive displayConfig) or --enable-tool displayConfig before use.
🧬 getIosSimulatorCapabilities	Discovers biometrics for a selected iOS Simulator device type and runtime.
🫆 biometricAuth	Simulates biometric authentication.
📳 shake	Shakes an Android emulator or iOS Simulator.
📞 phoneCall / 💬 sendSms	Simulates an Android emulator phone call or incoming SMS.
🔔 postNotification	Posts a notification through Android SDK hooks or iOS Simulator push.
🔔 getNotificationPolicy / 🔔 setNotificationPolicy	Reads or changes app notification and Do Not Disturb policy.
🛂 getAppPermissions / 🛂 setAppPermissions	Reads or changes app permissions.

During canonical-shape phase 1, every device surface returns the complete shared description: static facts are top-level, changing state is under runtime, and the previous nested and flat aliases remain present as compatibility fields.

Keeping an Android orientation locked

rotate preserves its existing behavior when lockOrientation is omitted: it temporarily disables auto-rotate when necessary, then restores the prior setting. To keep portrait or landscape orientation in effect for subsequent actions, pass lockOrientation: true:

{ "orientation": "landscape", "lockOrientation": true }

The result reports orientationLockState as locked, unlocked, or unknown. A persistent request succeeds only after live rotation and the lock are confirmed. If lock verification fails, currentOrientation reports the latest confirmed live orientation, or unknown when it cannot be read.

To restore automatic rotation, pass lockOrientation: false, for example { "orientation": "landscape", "lockOrientation": false }. These lock options are supported only on Android.

Acquiring a device: avdName, udid, and the deviceId alias

getAndroid and getApple each accept two ways to name a target; pass one.

• getAndroid — avdName names a configured Android Virtual Device (the name field of automobile:devices/images/android). It is the identity AutoMobile uses to boot and coordinate a named AVD: the avdName path passes matchExactName, androidAvdName, and a stableTarget for exact AVD-identity and lifecycle coordination. deviceId is the copy-paste-from-discovery convenience: it accepts either an already-booted serial such as emulator-5554 (the runtime.deviceId field of automobile:devices/booted/android) or an AVD image name — if it names a defined-but-unbooted AVD, getAndroid cold-boots that image by name. The difference is the coordination hints the avdName path passes up front — matchExactName, an androidAvdName startup-lease hint, and an eager stableTarget — so prefer avdName when you specifically want to boot or coordinate a named AVD; use deviceId to attach to a running device or to boot straight from a discovered identifier.
• getApple — udid is the iOS Simulator UDID. deviceId is an accepted alias for udid: a booted simulator’s runtime.deviceId (from automobile:devices/booted/ios) is its udid, so both fields resolve to the same value.

The deviceId fields exist so the value at runtime.deviceId in listDevices and the automobile:devices/booted/* resources can be copied straight into getAndroid/getApple — the discovery→acquire path (#5870). See the FAQ for the CLI equivalents.

Network, plans & recording

Tool	What it does
🌐 network	Controls network capture and error simulation.
🎭 mockNetwork / 🧹 clearMockNetwork	Adds or clears mock network response rules.
🕸️ getNetworkGraph	Returns the aggregate captured network graph.
🧪 executePlan	Executes YAML plan steps and stops at the first failure.
🔒 criticalSection	Synchronizes devices, then runs steps serially.
🚧 barrier	Synchronizes devices, then lets them proceed concurrently.
📝 recordSteps	Records MCP calls to YAML; begin and end require –mcp-recording.
⏺️ startTestRecording	Starts recording user interactions for exportPlan.
📤 exportPlan	Stops the active recording and exports a YAML plan.
🎥 videoRecording	Starts or stops device video recording.

Accessibility & session tools

Tool	What it does
♿ accessibility	Reads or controls Android TalkBack and iOS VoiceOver, returning fresh device state.
🎯 accessibilityFocus	Sets or clears Android TalkBack focus by resource ID, text, or content description.
🔀 setToolEnabled	Enables or disables AutoMobile tools for the current MCP session — one exact name via toolName, or a batch via toolNames; unknown or hidden names reject the batch, while always-on names are returned in skipped.

Before acquiring a device, read automobile:tools for every tool’s default enabled/gated state.

On Android and iOS, compact observations fold soft-keyboard keys into a single keyboard: { visible: true, package: "…" } summary plus at most one skeleton row:

<ime> | Keyboard (com.google.android.inputmethod.latin) | input

On iOS, the row is labelled Keyboard (com.apple.keyboard).

That row appears only when the IME exposed at least one bounded accessible descendant. A keyboard whose window carries no accessible keys — some IMEs expose none — is still announced by the keyboard summary, with no <ime> row, because a synthetic row must never claim a box it cannot measure. Treat the summary as the presence signal and the row as optional.

<ime> is a marker, not a selector — use sendKeys for text input and semantic keys, or keyboard to open or close it. Everything the IME itself owns folds into that row, including its toolbar, emoji and clipboard affordances — Gboard gives those the same key_pos_* ids it gives letter keys (key_pos_header_access_points_menu, key_pos_switch_to_symbol), and many keys carry no resource-id at all, so nothing distinguishes them from a keycap. What stays individually actionable is framework chrome sharing the window: a control whose resource-id belongs to a DIFFERENT package than the IME, such as android:id/input_method_nav_back. observe with project: "full" or raw: true retains the individual keys. Keys are identified from the IME window identity a re-cut control proxy supplies, and otherwise from the …:id/key_pos_* keycap resource-id family, so the fold also applies on older proxy builds and on the uiautomator dump path. An absent summary means no IME identity was captured, not a confirmed hidden keyboard.

A skeleton row omits label entirely when it has none — the key is never present with a placeholder value. A state-carrying container with no text of its own (the switchWidget of a Settings row, a scrollable fragment root) takes the label of its nearest labelled enclosing row, so a checked state is attributable to the setting it belongs to (com.android.settings:id/switchWidget | Airplane mode | toggle checked=false).

Action-observation diffs under the default skeleton projection use those same compact row fields for added and removed nodes; rows with no affordances are omitted. Their fields.layoutWarnings value is an { added, removed } pair rather than a whole { from, to } envelope, and excludes Android system-status bar notification chrome. An artifact-spilled diff keeps its skeleton, context, and capture metadata inline while the bulky node and field deltas are available from its artifact pointer.

For the observe → act → observe behavior behind interaction tools, see the interaction loop. For per-session public tool selection, see Dynamic Tools.