Prototype spec: media players, camera, casting and haptics¶
Part of the agent-authored prototype specification.
Media players (#14253, contract only). The root media block declares long-lived players
that outlive a screen (mini-player, background audio, picture-in-picture):
{
"requires": { "platform": ["media"] },
"media": {
"players": {
"main": { "kind": "video", "source": { "asset": "trailer" }, "loop": false },
"podcast": { "kind": "audio", "source": { "sample": "tone440" } }
}
}
}
- At most
MAX_PROTOTYPE_MEDIA_PLAYERS(4) players; each id is a state key.kindisvideooraudio;loopis optional. sourceis exactly one of{asset: id}(an uploaded media asset or a devicemedia:id),{url}(an absolutehttporhttpsURL, themediaUrlpattern),{data: "/pointer"}(resolved at runtime) or{sample}. For{sample},tone440andsilenceare synthesized audio,colorBarsandcountdownare generated video.- The block needs
mediainrequires.platform, soprototype_platform_media_v1gates it.
The media module (prototype_platform_media_v1) has the calls play, pause, toggle, stop,
seek (positionMs, 0-86400000) and seekBy (deltaMs, ±86400000). It also has setRate (rate, 0.25-4),
setVolume (volume, 0-1, the player’s own volume, never the device stream), setMuted and
setLoop. Every call takes a player argument; a literal {value} must name a declared player.
Its events are media.statusChanged, media.ended and media.error. State is per player at
/platform/media/players/<id>/<key>: playing, positionMs, durationMs (-1 while unknown),
buffering, status (idle, loading, ready, ended, error) and error
(source_not_found, decode, network, unsupported_format, blocked_egress). A read must name
a declared player and one of those keys. The asset transport and the renderers are separate
slices; no production runtime advertises the module yet.
The videoPlayer node renders exactly one of a declared player or an inline source (the same
one-key union), which is a node-scoped player released when the node leaves composition:
{ "type": "videoPlayer", "player": "main", "autoplay": "whenVisible", "muted": true }
- An inline player is addressed by the node
id, which must be a state key that no declared player uses;media.*calls and/platform/media/players/<id>reads accept it like a declared id. - Optional fields:
poster(an image asset id),controls(builtinornone),contentScale(fit,crop,fill),autoplay(true,falseor"whenVisible"),muted,loop,startMs(0-86400000) andpositionKey. ThepositionKeyfield is a declared number state key the device mirrors the position into. A slider bound to it holds device writes while it is dragged and seeks on release. - Every
videoPlayerneedsmediainrequires.platform. Authored controls are ordinary nodes callingmedia.*. - At most
MAX_PROTOTYPE_VIDEO_NODES(64)videoPlayernodes, counted in the authored tree and again after repeat expansion, so every repeat instance counts. Apostercosts one image againstMAX_PROTOTYPE_IMAGESin both counts.
Captions (#14259, contract only). A videoPlayer may list up to 8 captions tracks
{id, language, label, source}, where source is an uploaded WebVTT {asset} or {data}, a /data
pointer to an array of {startMs, endMs, text} cues that each end after they start. Track ids are
unique per player; hls: ids are reserved for in-band HLS tracks. media.setCaptions (player,
track, "" for off) selects a track and /platform/media/players/<id>/captionTrack reports it.
The cue parser and caption overlay are separate slices.
Feed autoplay. A videoPlayer with
autoplay: "whenVisible" may set visibleThreshold (0.1-1, default 0.6) and resumePosition;
either field without "whenVisible" is an error. Feed players share at most
MAX_PROTOTYPE_ACTIVE_DECODERS (3) decoders, allocated by a pure function that Kotlin and Swift
both replay against test/fixtures/prototype-media-pool:
- The anchor is the settled pager page, or else the most visible player, with ties going to the topmost.
- The window is the anchor, the next item and the previous item, in that order.
- The anchor plays only in the foreground, and only when its page has settled or, in a list, when
it is at least
visibleThresholdvisible. - Other window players preload; held players outside the window are released.
- When nothing is visible, every player is released.
The allocator only decides which players play, preload or release. Each renderer reports visible fractions, the settled page and foreground state, binds pooled players to surfaces and preloads.
Queues (#14249, contract only). A declared player takes exactly one of source or queue, a
/data pointer to an array of {id, title, artist?, album?, artwork?, source, durationMs?} items
with unique ids; appendData and removeData edit the queue at runtime. A queued player, and the
calls media.next, previous (restarts after 3 s), skipTo (itemId), setShuffle,
setRepeat (off, one, all) and setSleepTimer (timer {ms}, 0 cancels, or
{endOfItem: true}), need prototype_platform_media_queue_v1. Per-player state adds
currentItemId, currentIndex, rate, sleepRemainingMs, shuffle and repeat, and
media.itemChanged reports an item change. No runtime advertises the capability yet.
Queue semantics are a pure state machine (PrototypeQueue) that Kotlin and Swift both replay against
test/fixtures/prototype-media-queue; the host only validates. Renderers report ended, the
playhead, clock ticks and queue edits, and map the state onto their engine:
- The play order is the queue order, or a Fisher-Yates shuffle driven by an injected
Randomthat keeps the current item first. Turning shuffle off restores queue order. nextignores repeatone; an item ending replays under it. Past the last item, repeatallwraps andoffstops on that item.previousrestarts after 3 s, and at the first item wraps underallor otherwise restarts.- A
{ms}sleep timer counts down on the renderer’s clock and pauses at zero;{endOfItem}pauses when the current item ends instead of advancing. - When a data action edits the queue, play order keeps its relative order and appends new items.
Removing the current item moves to the next surviving one, or with none left wraps under
allor stops on the last item.skipToan unknown id fails withitem_not_found.
Fake media engine and simulate (#14257). PrototypeFakeMediaEngine (TypeScript, Kotlin in
android/protocol, Swift in the prototype agent core) is a deterministic player that decodes
nothing. All three replay the event scripts and expected /platform/media state in
test/fixtures/prototype-media-sim:
- Time is virtual and moves only when the runtime’s injected clock calls
advance. A playing, ready, non-buffering player advances, then ends atdurationMs, or wraps when it loops. AdurationMsof-1never ends. media.statusChangedandmedia.errorare reported only on a status orplayingchange, andmedia.endedandmedia.erroronce per transition.bufferingstalls the playhead without an event.- An errored player ignores
playandseekuntil an event sets itready. Seeking back fromendedreturns toready.
The host delivers an event with prototype {action: "simulate", id, event, data}. The event is
media.statusChanged (data {player, status, code?} with status buffering, ready, ended
or error), media.ended ({player}) or media.error ({player, code}). code is one of the
state’s error codes and is required exactly when the status is error. An optional advanceMs
(1-3600000) advances the fake engine’s clock first. The host validates data before it asks the
device (invalid_args). It refuses advanceMs on a native engine (unsupported) and reports the
engine as mediaEngine. No runtime exposes its media module to simulate yet, so every device
answers unavailable. Selecting the fake engine (platformMode) and the Media3 adapter belong to
the platform simulation runtime (#14300).
The fake engine proves spec logic, not playback. Real playback is confirmed separately. On Android
the mediaSession tool reads the session’s playback state and position (#14117), and captureAudio
checks that the output is not silent on hosts where emulator or Simulator audio capture works
(#13629). Generated test clips come from scripts/media/generate-prototype-test-media.sh.
Live camera (#14202, calls only). The camera module also drives an in-app viewfinder:
camera.takePhoto writes an image item to into, camera.startRecording (optional audio boolean and
maxDurationMs 1-600000) and camera.stopRecording (writes a video item to into). Each takes
preview, the id of the viewfinder it acts on, needs camera in requires.permissions and is
sensitive. Recording also needs prototype_platform_camera_video_v1. The camera.statusChanged event and the
/platform/camera/status state report starting, ready, capturing, recording, unavailable,
permission_denied or error; /platform/camera/flashAvailable, recordingMs and source
(camera or injected, the fake frame source on a Simulator) complete the state.
An iOS Simulator has no capture devices (AVFoundation discovery returns none) but does have an audio
input. It therefore advertises prototype_platform_audioRecorder_v1 and not prototype_platform_camera_v1,
and serves the viewfinder from injected frames (source is injected). A physical iPhone advertises a
module only when its hardware is present and its usage description (NSCameraUsageDescription,
NSMicrophoneUsageDescription) is in the Info.plist; a missing key reports unavailable with detail
missing_usage_description. The rule is PrototypeMediaCaptureAvailability.
The cameraPreview node is that viewfinder; a story-capture screen pairs it with ordinary buttons:
{
"type": "cameraPreview",
"lensKey": "lens",
"flashKey": "flash",
"zoomKey": "zoom",
"aspectRatio": "4:3",
"contentScale": "crop",
"tapToFocus": true,
"pinchToZoom": true,
"mode": "photo"
}
- A spec has at most one
cameraPreview. It needs anidthat is a state key, whichcamera.*calls name as their literalpreview(platform-camera-live.jsonis the full story-capture spec). - It needs
camerainrequires.platformand inrequires.permissions. lensKeyandflashKeyname declared string state keys (backorfront;off,on,autoortorch) andzoomKeya declared number key (0-1), sosetStateand a slider drive them.- Optional:
aspectRatio(4:3,16:9,1:1),contentScale(fit,crop,fill),tapToFocus,pinchToZoomandmode(photoorvideo). No renderer draws it yet.
Casting (#14239, module only). The cast module (prototype_platform_cast_v1) covers Google
Cast on Android and AirPlay on iOS with one vocabulary. Its calls are the remote-only controls:
cast.disconnect (end the session and resume locally), cast.stop (end it and stop the receiver
app) and cast.setVolume (volume 0-1, the remote device volume). Play, pause and seek stay on the
media.* calls. A platform that cannot do an op (AirPlay disconnect or setVolume) fails with
unsupported. Event data is a closed object (unknown keys are refused):
| event | data |
|---|---|
routesChanged |
state (as /platform/cast/state), routeCount (integer >= 0) |
sessionStarting |
player (1-64 chars), protocol, deviceName (0-256 chars) |
sessionStarted |
as sessionStarting, plus positionMs (integer >= 0, the remote position) |
sessionEnded |
as sessionStarted; positionMs is where local playback resumes |
sessionFailed |
player, reason (media_not_castable, play_services_unavailable, network, receiver), optional protocol |
remoteVolume |
volume (0-1) |
The host decodes them with prototypeCastEventOf (truncated data does not decode). Kotlin
PrototypeCastEvents and Swift PrototypeCastEvents are the twins, and all three replay
test/fixtures/prototype-spec/cast/events.json. The /platform/cast/state value is unavailable, noDevices, notConnected,
connecting or connected. It carries deviceName, protocol (googleCast or airplay), protocols
(the unique protocols the device supports, read whole as {platform:/cast/protocols}; an iOS agent
reports ["airplay"]), player (the cast media player id) and volume. CtrlProxy never advertises
the module.
The castButton node is the system route picker that starts a session; there is no start call. Its
required player names a declared media.players id or an inline videoPlayer id, and optional
protocols (1-2 unique of googleCast, airplay) limits the routes it offers. Every castButton
needs cast in requires.platform. No renderer draws it yet.
Haptics and sound (#14329). The haptics module has one call,
haptics.perform with effect one of click, tick, heavyClick, doubleClick, selection,
success, warning, error, impactLight, impactMedium, impactHeavy or longPress
(capability prototype_platform_haptics_v1). The sound module has sound.play with sound
one of click, send, receive or shutter and an optional volume, a whole number 0-100
(prototype_platform_sound_v1). Both lists are closed: a literal {asset: ...} sound is refused
(there is no audio asset type, #14254). Neither module has events or /platform state, needs a
runtime permission (VIBRATE is a normal permission, declared by CtrlProxy) or touches personal
data. CtrlProxy and the iOS prototype agent register both modules and advertise the capabilities;
the embedded Compose renderer leaves modules to the app.
- Android haptics use the
Vibrator. From API 29 they useVibrationEffect.createPredefined(click and impactMedium click; tick, selection and impactLight tick; heavyClick, impactHeavy, longPress and warning heavy click; doubleClick and success double click), anderroris a two-pulse waveform. Below API 29 they use a tuned one-shot or waveform. There is no tappedViewto callperformHapticFeedback(CONFIRM/REJECT)on, so success, warning and error use these fallbacks. The renderer reports touch feedback off in system settings, or a device without a vibrator (suppressed/unavailable). Theshuttercue isMediaActionSound,clickis the system click sound effect, andsend/receiveare short system tones. The renderer bundles no audio, and the ringer (silent or vibrate) and the touch-sounds setting suppress them. - iOS haptics use
UIImpactFeedbackGenerator,UISelectionFeedbackGeneratorandUINotificationFeedbackGenerator(doubleClick is two impacts 80 ms apart). The Simulator does not render haptics, but the call still succeeds and the agent logs it. “System Haptics” off and the silent switch are not detectable, so the agent reports nothing as suppressed. Sounds areAudioServicessystem sounds (click 1104, send 1001, receive 1007, shutter 1108); the agent ignoresvolumeand never touches the host’sAVAudioSession. feedbackLog. Each shown prototype keeps its last 64 calls, oldest first, andprototype inspect(andstatus) returns them asfeedbackLog: [{effect | sound, nodeId?, timeMs, suppressed?}]. ThenodeIdfield is the one node whose tap, long-press or double-tap action made the call, absent for a platform handler or when several nodes author the same call. ThetimeMsfield is the device clock. The log skips failed calls; a show or dismissal clears the log.
| Field | Rule |
|---|---|
call |
Required: "<module>.<op>" naming a call in platformModules, else Unknown platform call at .call. Its module must be listed in requires.platform, its permissions in requires.permissions and its background names in requires.background; each error is at .call. |
args |
Up to MAX_PROTOTYPE_PLATFORM_ARGS (16) entries, exactly the call’s declared arguments (Unknown argument, Required argument at .args.<name>). Each is a value binding: {value}, {state}, {data}, {platform}, {item} / {index: true} in a repeat, and {event} inside a handler. A literal {value} must satisfy the argument’s rule (error at .args.<name>.value). |
into |
Optional /data/<pointer>, the request rule: an object member, not an array element, so <into>__meta can be written beside it. /platform is read-only, so an into under it is refused. Written with applyPrototypeDataWrite and reported as a change event. |
statusKey |
A declared string state key, moved loading → success or error, as for request. |
timeoutMs |
Optional integer, 1 to MAX_PROTOTYPE_PLATFORM_TIMEOUT_MS (300000, long enough for a system picker); the default is DEFAULT_PROTOTYPE_PLATFORM_TIMEOUT_MS (10000). |
onSuccess, onError |
Optional action lists (1 to 32). They count toward MAX_PROTOTYPE_ACTION_DEPTH. |
A failed call writes {code, message} to <into>__meta.error, with code one of unsupported,
unavailable, permission_denied, invalid_args, cancelled, timeout, too_many_in_flight
(more than MAX_PROTOTYPE_PLATFORM_CALLS_IN_FLIGHT, 4) or platform_error.