Icons and images¶
Part of the agent-authored prototype specification.
Built-in names are a curated set of 300 Material icons in snake_case (home, alarm_add); the
sorted source is schemas/prototype-icon-names.json, which generates the iconName definition in
schemas/prototype-spec-contract.json shared by all validators. CtrlProxy’s generated R8 rules
retain only those names in all five styles from material-icons-extended. An icon
node may set variant (filled default, outlined, rounded, sharp, twoTone); nav items are always
filled. Material Symbols names without a counterpart and symbol weight / fill axes (which need a bundled
variable font) are unavailable. Custom text fonts ship as host-uploaded assets (see “Custom fonts”).
Unknown names reject the spec at the offending field with close substring suggestions and an allowed-icon lookup URI. Images are ID-based (#9301): {type: "image", asset: "opaque-id"} and nav
image: "opaque-id". Existence is not validated at spec level; a missing asset renders a placeholder and is
reported in missingAssets. Bytes, MIME types, URLs, cache paths and screenshot handles are not spec
properties.
Asset transport (#9301)¶
put_prototype_asset {id, mimeType, dataBase64} uploads one asset and
remove_prototype_asset {id} deletes one; each gets one prototype_result carrying the
request ID. Bytes travel as base64 in the single JSON text frame, like screenshots.
Heap, not the 64 MiB frame limit, is the binding constraint, so the caps are conservative. The
host shares them through schemas/prototype-asset-contract.json:
- 4 MiB per asset, 32 assets, 16 MiB total, and ids of 1 to 256 characters;
image/png,image/jpegorimage/webp(exact lowercase) whose bytes must start with the matching signature;font/ttfandfont/otf(2 MiB each, see Custom fonts).
Putting an existing ID replaces it; a full store rejects the put with a clear error and never
evicts. Removing an unknown ID succeeds. Assets sit in the CtrlProxy cache directory. The runtime
clears them when the prototype session ends: on any dismissal, on service start, unbind or
teardown, on dismiss_prototype with all, and when the last client disconnects (even with no
prototype showing). A show replacement and a temporary lock-screen hide keep them.
Custom shader assets¶
application/vnd.automobile.agsl assets carry UTF-8 AGSL source, capped at MAX_PROTOTYPE_SHADER_ASSET_BYTES = 16384. Upload a local .agsl file through assets: [{id, path}], then reference its id with style.shader: {asset, uniforms?} (also accepted in styleWhen). The host refuses uploads and shader specs unless AUTOMOBILE_PROTOTYPE_CUSTOM_SHADERS=1; the Android runtime must advertise prototype_custom_shaders_v1. Shader assets share the existing count and total-byte limits. Available assets must have the AGSL MIME type and every supplied uniform must be declared as a scalar float in that source. iOS (including simulator saves) and ProtoViewer bundles refuse custom shaders. Native host export refuses shader specs on both platforms until the generated hosts support shader assets.
Source rules: valid UTF-8 with no leading BOM; the scanner strips comments (// and /* */) to
separators before token scanning; code tokens are ASCII. A source declares at most 16 scalar
uniform float name; uniforms plus one optional uniform shader content; (a shader may sample
only the node’s own rendered content). Validators refuse preprocessor directives, strings,
escapes, qualifiers, arrays, initializers, comma declarations, duplicate names and other uniform
types or samplers. Reserved uniform names are content, main, GLSL/AGSL reserved words and
names starting with sk_ or gl_. The style.shader.uniforms object supplies at most 16
declared float names matching [A-Za-z_][A-Za-z0-9_]*, with finite values in
[-1000000, 1000000]. AUTOMOBILE_PROTOTYPE_CUSTOM_SHADERS is off by default.
The Android runtime advertises prototype_custom_shaders_v1 only on API 33+. It compiles on put_prototype_asset before writing any source; unsupported APIs return a typed refusal naming API 33, and invalid source returns the compiler message capped at 2048 UTF-8 bytes without splitting a character. Failed replacements preserve the previous asset. Validation runs in three stages:
- Device lexical preflight restricts inputs to scalar floats and the node’s own
content. - Show/patch validation checks every style alternative and hidden descendant against device-compiled scalar declarations.
- Per-node compilation rechecks all supplied names before setting any values, clamps finite values to [-1000000, 1000000], and gives each node an independent shader instance.
Procedural sources without a content declaration receive an unused uniform shader content; so the same RenderEffect path can bind the node’s content.
Embedded applications supply AGSL bytes through PrototypeAssets.withShaders and its optional shader resolver; image and shader lookups use separate typed namespaces even when ids collide. These sources receive the same lexical, UTF-8 and compiler validation before rendering. Each composed node owns its compiled shader, reused across uniform changes and recompiled when its source asset or version changes. Removed float uniforms reset to zero. Compiler diagnostics refer to authored source lines, accounting for the injected content declaration. Each node applies its shader through a clipped Compose graphics layer using createRuntimeShaderEffect(shader, "content"). Content remains unshaded while preparation runs, while an asset is missing/unreadable, or on API levels below 33. Missing shader assets follow the existing missingAssets warning/re-upload path and retry on asset changes.
One session-level frame-clock loop runs only while at least one visible shaded node is composed (excluding settled-hidden retained subtrees) and the prototype lifecycle is RESUMED. The control-proxy pauses that lifecycle while hiding its window for capture. Only the newest composed window root owns sampling during a window replacement; retiring an older root cannot hide or pause the new one, and a failed replacement restores the older root. Gaps discard the first interval after resuming without clearing budget evidence.
The runtime permanently disables all shaders in the prototype after 8 consecutive intervals strictly above 24 ms, or more than 30 slow intervals in the last 60 sampled intervals. Intervals at or below 24 ms reset only the consecutive streak; the loop ignores duplicate/backwards timestamps. Status lists this once as frame budget exceeded while shaders were active: it measures whole frame intervals and cannot attribute the delay to a shader. The loop cannot measure a static expensive shader without fresh frame work, and cannot catch asynchronous RenderThread/GPU faults.
Synchronous compile/effect/draw exceptions disable only the affected composition instance. Duplicate repeat identities have separate guards; disposed registrations are evicted. The get_prototype_status result reports at most 64 node failures with unique instance ids and bounded reasons, plus a +N more summary, in optional shaderFailures. Host inspect adopts these into the next host-local status. An in-place show preserves the session budget and recorded failures; dismissal or an explicit reset starts a new lifetime. A remounted node gets a fresh instance guard, while a tripped session budget continues to disable all shaders.
The wire protocol has no host opt-in channel. The existing host gate AUTOMOBILE_PROTOTYPE_CUSTOM_SHADERS=1 therefore remains the authorization boundary; capability advertisement alone does not enable the host to send shaders. Direct device clients are responsible for the same opt-in policy.
Device media items (#14192)¶
Which camera, media-library, scanner and microphone routes work on each emulator, simulator and device, and how to seed them, is in the camera and media guide.
Camera, picker and recorder producers write a media item (or an array of them) to an into
pointer in /data, under the rules of applyPrototypeDataWrite and with no nulls. The device
mints its ids with the reserved media: prefix and keeps the bytes in its own store, so an image
can show something the device produced:
{
"asset": "media:5b1f…",
"kind": "image",
"mimeType": "image/jpeg",
"width": 3024,
"height": 4032,
"bytes": 2481152,
"source": "camera",
"createdAt": "2026-10-21T10:00:00Z",
"thumbnail": "media:5b1f…/thumb"
}
kind is image, video or audio (durationMs for the last two); source is camera,
library, microphone or injected; thumbnail is a JPEG of at most 512 px.
imageAssetgains{data: "/pointer"}, a pointer into/datathat resolves to a media id, on animageassetand on a tab or bottom-nav itemimage. It is exactly one key, a pointer string. A repeat placeholder also binds in a stringasset("asset": "{item.thumbnail}"), so a gallery is agridplusrepeatover/data/picked.media:is reserved. Hostput_prototype_asset,show.assets, bundles and saved layouts refuse an asset id that starts with it, and amedia:id is never listed inmissingAssets(an unknown or deleted one draws the broken-image placeholder).- No image budget. A
{data}slot names no uploaded asset and costs 0 againstMAX_PROTOTYPE_IMAGES(animagenode still counts 1). - Limits in the contract:
MAX_PROTOTYPE_MEDIA_ITEMS64,MAX_PROTOTYPE_MEDIA_TOTAL_BYTES512 MiB, and per item 32 MiB for an image, 256 MiB for a video and 32 MiB for audio. - Capability.
prototype_platform_media_items_v1gatesimageAsset.data. It is a spec-language gate, not a platform module, so it needs norequires.platformentry. The validators, gate and shared corpus (media-*fixtures) exist. Device stores, EXIF handling, the read-onlyprototype {action: "media"}host action and themediaItems.injectsimulation do not, so no renderer advertises it and the host refuses a show that uses it.
Rendering assets (#9301)¶
An image node draws its asset with contentScale: fit letterboxes, crop fills the box and clips, fill stretches. Bytes are decoded off the main thread with BitmapFactory and inSampleSize, downsampled to the node’s laid-out size (screen size when an axis wraps) and never above 4 Mi pixels. The decoded-bitmap cache keys on asset id and a power-of-two size bucket and evicts least recently used first. It holds at most 32 MiB of decoded pixels (a larger single bitmap is drawn but not retained). A replaced, removed or cleared asset drops its decoded copies immediately and showing nodes reload; uploading a previously missing id fills the placeholder.
A nav item draws its image when ready, then its built-in icon, then a gray square. While an image decodes the node shows a plain gray box. A gray box with a broken-image glyph marks an unknown id, undecodable bytes, or a file the OS evicted from the cache directory (the store lists it but read returns null).
show_prototype lists referenced ids the device has no copy of in prototype_result.missingAssets, in first-use order. It is a warning: success stays true and placeholders show, so the host can upload and the nodes fill in without another show. The field is omitted when nothing is missing and from every other result. An id the store lists but whose file was evicted is not reported at show; it renders the placeholder.
Host surface: the prototype tool’s show takes
assets: [{id, path}], an absolute daemon-readable file path per asset. The host
reads and validates every file first (signature-detected MIME type, the contract
limits, unique ids), then uploads sequentially before the prototype request. Any
failure fails the call before the prototype changes and names the assets already
stored.
An entry may instead be {id, observation}, an
automobile:observation/{deviceId}/{observationId}/screenshot URI. The host reads it
through the same handler as that resource (current-observation check, pending-capture
wait, retention lease). Any client can read that resource, so no access is widened.
Sometimes prototype_result.missingAssets lists an id the same call uploaded, because the device
cleared its store between the upload and the show. The host then re-uploads those assets
once from the bytes it already holds and re-sends the show once. It never
loops: if they are still missing, or the retry fails or is cancelled, the first
successful result is returned with missingAssets and a warning on the tool output.
Ids the call did not supply are only reported. See docs/tools.md for the result and
deadline model.
Custom fonts (#10443)¶
style.fontFamily also accepts {asset: "<id>"}, an opaque id of a font uploaded through
the same asset pipeline as images (put_prototype_asset, or assets: [{id, path}] on the
host). The spec carries only the id; font bytes, MIME types and caps stay out of it.
- Transport:
font/ttfandfont/otfjoin the contract’s MIME types. The bytes must start with an sfnt version tag,0x00010000,trueorOTTO, accepted for either MIME type because an OpenType file may carry TrueType outlines. TrueType collections (ttcf) and WOFF are rejected. Fonts are never decoded as images. The per-font cap is 2 MiB (MAX_PROTOTYPE_FONT_ASSET_BYTES); fonts count toward the 32-asset and 16 MiB totals like images. The host detects a font from the file signature, not the extension. - Missing assets:
prototype_result.missingAssetslists a referenced font the device has no copy of, with the image ids (references instyleandstyleWhencount, including hidden nodes). The text draws in the default family until the font is uploaded. - Rendering: the runtime loads the font from the stored asset file with
FontFamily(Font(file)), once per asset, and caches it. A file that cannot be parsed logs one warning and draws with the default family; the font never fails the prototype. Replacing an asset reloads it; removing it (or ending the session) while a spec still references it falls back to the default family, the same as a removed image becoming a placeholder. - Weights: one file is one face, so
fontWeightselects that face rather than a bolder one. Upload a font file per weight and usestyleWhento choose between them.