Skip to content

Overview

✅ Implemented 🧪 Tested

Current state: Fully implemented. Observation includes typed, source-attributed inset data when the platform can provide it. See the Status Glossary for chip definitions.

Each observation captures a snapshot of the current state of a device’s screen and UI. When executed, it collects multiple data points in parallel to minimize observation latency. These operations are incredibly platform specific and will likely require a different ordering of steps per platform. All of this is to drive the interaction loop.

All collected data is assembled into an object containing (fields may be omitted when unavailable):

  • updatedAt: device timestamp (or server timestamp fallback)
  • screenSize: current screen dimensions (rotation-aware)
  • insets: typed safe-area and system-inset snapshot, including availability, source, units, system bars, cutouts, and Android gesture regions when available
  • systemInsets: compatibility alias for the stable system-bar edges; prefer insets for new consumers
  • rotation: current device rotation value
  • activeWindow: current app/activity information when resolved
  • viewHierarchy: complete UI hierarchy (if available)
  • focusedElement: currently focused UI element (if any)
  • intentChooserDetected: whether a system intent chooser is visible
  • wakefulness and backStack: Android-specific state
  • displayedTimeMetrics (Android launchApp “Displayed” startup timings), performanceAudit, and accessibilityAudit: present when the relevant modes are enabled
  • perfTiming: collected internally for debug/perf capture diagnostics but stripped from the sanitized MCP tool output to reduce payload size
  • gfxMetrics: emitted in sanitized output for action UI-stability summaries; frame timing fields may be trimmed when performanceAudit.metrics already carries non-null computed replacements
  • error: error messages encountered during observation

Start AutoMobile with --safe-area-warnings or its equivalent alias --edge-to-edge-warnings to add report-only layoutWarnings. These flag text or interactive elements that may overlap safe areas, system bars, display cutouts, or Android gesture regions. Intentional edge-to-edge backgrounds and scrollable content remain advisory rather than failures.

The observation gracefully handles various error conditions:

  • Screen off or device locked states
  • Missing accessibility service
  • Network timeouts or ADB connection issues
  • Partial failures (returns available data even if some operations fail)

Each error is captured in the result object without causing the entire observation to fail, ensuring maximum data availability for automation workflows.

See Also