Skip to content

observe

Tool What it does Platforms Needs
👁️ observe Get screen view hierarchy and screenshot, with optional PNG crop from a fresh settled capture of one display 🤖 🍎 ⌚ 📺 🥽 —

Call observe to read the current screen before choosing a target. The default skeleton and context rows describe controls and visible text; project: "full" returns the hierarchy. Bounds tuples are [left, top, right, bottom].

Adjustable range controls, including sliders, carry the adjust affordance. Any row with numeric state, including a read-only progress indicator, carries range: {current, min, max}. To adjust one, use swipeOn with an adjust row’s elementId as the container.

Reading an observation’s screenshot later

Each observation returns an observationScreenshotResourceUri. Read that URI to get the screenshot of exactly that observation. It stays readable while it is one of the device’s last 10 observations and is under 5 minutes old, even after later actions or app switches. Read it before then if you record evidence after the agent’s next step.

When the screenshot is gone, the read returns JSON with code OBSERVATION_NOT_RETAINED. evictedBecause says why:

evictedBecause Meaning
newer-observation 10 newer observations pushed it out; supersededBy names the newest.
ttl It is older than 5 minutes.
cache-cleared The device’s cache was cleared, for example when its session ended.

With no evictedBecause, the id is unknown. In every case, call observe again.

Apps with several scenes on iOS

When an iOS app has more than one connected scene, observe and the element-based action tools such as tapOn fail with a multi_scene error that lists the scenes. Close all but one scene, then retry. The tools do not fall back to a different observation source.

Toggle labels

A toggle or range row may have no label of its own or from its enclosing row. It then takes the text shown immediately before it, such as the Dark mode text beside a Switch. That text still appears in context.

Some text follows a toggle that has no label and no text before it. That text may sit in the same coordinate space (window or ornament), directly or inside a plain container. It is then that toggle’s trailing label and never labels the next toggle. When a toggle has its own label or takes the text before it, the text after it labels the next toggle. Text selectors match the same label.

Android XR panels

For an Android XR app with an embedded SpatialPanel, xrPanelSpace identifies the panel’s windowId and window bounds. Elements and skeleton/context rows carry boundsSpace: "panel" when their bounds are relative to that panel window. They carry boundsSpace: "window" when they belong to another window, such as a caption or status bar.

These windows do not share a coordinate space: do not compare their bounds across windows or pass panel bounds to tapAt. Use tapOn or tapAny to click a panel control by its label or identifier, or use keyboard focus keys. Non-XR observations, and XR apps without an identified embedded SpatialPanel, omit these fields.