Skip to content

WebRTC Screen Streaming (WHIP egress)

⚠️ Partial 🧪 Tested 🤖 Android + iOS Video

Current state: The AutoMobile publish path is implemented and tested — an Android or iOS device screen video is captured as H.264, packetized to RTP, and pushed to a coordination server over WHIP using werift (pure-TypeScript WebRTC). Control is via the daemon webrtc-stream.sock Unix socket. A reference coordination server + browser viewer ships under examples/webrtc-coordination-server/.

For the WebRTC protocol choices behind this implementation, see the WebRTC standards map.

Live Android capture depends on adb screenrecord or the persistent video-server jar on a real device/emulator; iOS capture depends on the CtrlProxy screen streaming helper plus local ffmpeg H.264 encoding. Everything up to and including the WebRTC media transport is covered by a real werift↔werift loopback and a full publisher→server→subscriber end-to-end test.

See the Status Glossary for chip definitions.

This is the browser/CI-facing streaming path. It is distinct from: - the videoRecording MCP tool (records a clip to a file), and - the desktop live-mirroring path over video-stream.sock + an in-process FFmpeg decoder (screen-streaming.md).

Motivation

A CI worker running the AutoMobile daemon should be able to push a live view of the device it is driving to a central web server, which fans the stream out to browsers (dashboards, debugging UIs, pair-debugging). WebRTC is the natural fit: H.264 is a first-class WebRTC codec, latency is sub-second, and the browser needs no plugin. WHIP (WebRTC-HTTP Ingestion Protocol) is the standard way to publish a WebRTC stream to a server with a single HTTP POST, and is supported by common media servers (MediaMTX, LiveKit, Janus, Cloudflare).

Architecture

┌──────────────┐   adb exec-out       ┌───────────────────────────────┐
│ Android      │   screenrecord       │ AutoMobile daemon (CI worker) │
│  screen      │──--output-format=────▶│                               │
│              │   h264 (stdout)      │  AndroidH264Source            │
└──────────────┘                      │    │ Annex-B H.264            │
                                      │    ▼                          │
                                      │  RtpH264TrackWriter (RFC 6184)│
                                      │    │ RTP packets              │
                                      │    ▼                          │
                                      │  WebRtcPublisher (werift)     │
                                      │    │ SDP offer / ICE / DTLS   │
                                      └────┼──────────────────────────┘
                                           │  WHIP: POST offer, get answer,
                                           │  Location: resource URL
                                           ▼
                                 ┌───────────────────────┐   WHEP    ┌─────────┐
                                 │ Coordination server   │◀─────────▶│ Browser │
                                 │ (WHIP ingest → forward │  offer/   │ <video> │
                                 │  RTP → WHEP egress)   │  answer   └─────────┘
                                 │  GET /api/streams  ◀──┼── reconnect API
                                 └───────────────────────┘

Components (in src/features/webrtc/)

File Responsibility
h264.ts Annex-B NAL splitter, access-unit assembler, RFC 6184 RTP packetizer (single-NAL + FU-A)
RtpH264TrackWriter.ts Turns the elementary stream into werift RtpPackets; wall-clock 90 kHz timestamps, marker bit on the last packet of a frame
RtpPcmuTrackWriter.ts Turns 8 kHz mono PCM16LE audio into PCMU/G.711 RTP packets
AndroidH264Source.ts Runs adb exec-out screenrecord --output-format=h264 -; rotates segments before the 180 s --time-limit cap so the stream stays continuous
PersistentEncoderH264Source.ts Runs the long-lived video-server (VirtualDisplay + MediaCodec, plus optional playback audio) via app_process; parsed by VideoServerStreamParser.ts
androidH264CaptureSourceFactory.ts Prefers the persistent encoder when automobile-video.jar is resolvable (videoServerJar.ts), falling back to screenrecord on unavailability or start failure
WhipClient.ts WHIP POST (offer→answer) and DELETE; resolves the Location resource URL used to reconnect/tear down
ReconnectController.ts Connect / reconnect with injectable backoff (default exponential 1 s→30 s)
WebRtcPublisher.ts werift RTCPeerConnection (H.264 sendonly) + WHIP + auto-reconnect; exposes a reconnect descriptor
webrtcStreamingConfig.ts Resolves config from AUTOMOBILE_WEBRTC_* env vars + per-request overrides

Control plane:

File Responsibility
src/server/webrtcStreamManager.ts Per-device stream lifecycle; wires source ⇄ publisher; restarts the source on reconnect so a fresh keyframe follows
src/daemon/webrtcStreamSocketServer.ts webrtc-stream.sock request/response control (start/stop/status/list)
src/daemon/webrtcStreamClient.ts Minimal client for scripts/tooling

Why the daemon socket (not an MCP tool)

Streaming is a long-lived side channel owned by the daemon, not a discrete agent action. It is controlled the same way as other daemon services (video-recording.sock, device-snapshot.sock) — a request/response Unix socket — so a CI worker can start/stop a stream without going through the MCP tool surface. See src/daemon/CLAUDE.md.

Control protocol (~/.auto-mobile/webrtc-stream.sock)

Newline-delimited JSON request/response.

Request

{
  "id": "1",                    // optional correlation id
  "action": "start",            // "start" | "stop" | "status" | "list"
  "deviceId": "emulator-5554",  // optional; defaults to the sole Android device
  "streamId": "ci-run-42",      // optional; generated if omitted
  "whipEndpoint": "https://coord:8080/whip",  // optional override of env
  "whipToken": "…",             // optional bearer token
  "iceServers": [{ "urls": "stun:stun.l.google.com:19302" }],
  "bitrateKbps": 4000,          // optional
  "size": { "width": 720, "height": 1280 },   // optional downscale
  "audio": true                               // optional Android audio
}

Response

{
  "id": "1",
  "success": true,
  "type": "webrtc_stream_response",
  "action": "start",
  "stream": {
    "streamId": "ci-run-42",
    "state": "connected",       // idle|connecting|connected|reconnecting|failed|stopped
    "whipEndpoint": "https://coord:8080/whip",
    "resourceUrl": "https://coord:8080/whip/ci-run-42",
    "iceServers": [ … ],
    "framesSent": 0,
    "packetsSent": 0,
    "audioPacketsSent": 0,
    "audioSamplesSent": 0
  }
}

Configuration

Environment variable Meaning
AUTOMOBILE_WEBRTC_WHIP_ENDPOINT WHIP ingest URL (required unless passed per request)
AUTOMOBILE_WEBRTC_WHIP_TOKEN Bearer token for the ingest endpoint
AUTOMOBILE_WEBRTC_ICE_SERVERS Comma-separated STUN/TURN URLs, or a JSON array of {urls,username,credential}
AUTOMOBILE_WEBRTC_BITRATE_KBPS Target encoder bitrate
AUTOMOBILE_WEBRTC_MAX_SIZE Capture downscale, WIDTHxHEIGHT
AUTOMOBILE_WEBRTC_AUDIO Enable optional audio (audio: true per request). Android requires the persistent video-server jar; iOS supports Simulator-window audio only.

Per-request fields override the environment defaults.

Reconnect model

Two independent reconnection layers:

  1. Publisher → server. WebRtcPublisher watches the peer connectionState; on failed/disconnected it tears down and re-publishes over WHIP with backoff, restarting the capture source so a new SPS/PPS + keyframe follows immediately. The WHIP Location resource URL is retained so the stale session can be DELETEd.
  2. Browser → server. The coordination server’s reconnect API (GET /api/streams, GET /api/streams/{id}) returns a StreamDescriptor containing the whepUrl and iceServers — everything a frontend needs to (re)connect. The reference viewer.html re-subscribes automatically when its peer connection drops.

Quality / bitrate

Capture prefers the persistent on-device encoder (video-server): a single long-lived VirtualDisplay + MediaCodec pipeline with no ~175 s rotation seam and one continuous timestamp base. It honors --bit-rate (from bitrateKbps) and --size (from size).

When the jar is not resolvable — or the persistent encoder fails to start — capture falls back to screenrecord, which encodes on-device H.264 with the same --bit-rate/--size controls. Its 180 s --time-limit cap is worked around by segment rotation (ANDROID_STREAM_SEGMENT_ROTATE_MS); each new segment emits a fresh keyframe.

Persistent-encoder delivery (automobile-video.jar)

The video-server jar ships as a GitHub release asset, mirroring the CtrlProxy APK/IPA delivery convention — so the persistent encoder is the production default rather than falling back to screenrecord because the jar wasn’t distributed.

Resolution precedence (resolved once at stream start, off the per-frame path, in webrtcStreamManager; the capture-source factory stays synchronous):

  1. AUTOMOBILE_VIDEO_SERVER_JAR — explicit local override (an already-built jar).
  2. A valid cached download at ~/.auto-mobile/video-server/automobile-video.jar (with a video-server-jar.json sidecar recording {version, sha256, size, downloadedAt}).
  3. A fresh download from the release, sha256-verified against the pinned release’s videoJarSha256 in the checksum registry.
  4. The local Gradle build output (android/video-server/build/libs/automobile-video.jar, from ./gradlew :video-server:d8Dex) — for development.
  5. else null → screenrecord.

AUTOMOBILE_VERSION (pin one coherent version) and AUTOMOBILE_ASSET_BASE_URL (offline mirror host) apply to the jar exactly as they do to the APK/IPA. The jar is warmed by a background prefetch at daemon startup, but only when WebRTC streaming is configured (AUTOMOBILE_WEBRTC_WHIP_ENDPOINT is set) — daemons that never stream download nothing.

Fail-modes. The jar is optional (the encoder already degrades to screenrecord), so unavailability degrades gracefully — but integrity is never compromised:

Situation Behavior
Checksum known + matches use the jar
Checksum known + mismatch fatal ActionableError from stream start, even in degrade mode — a corrupted/tampered download is never silently accepted
Checksum absent (a pin predating jar delivery / unknown) degrade to screenrecord
AUTOMOBILE_REQUIRE_VIDEO_SERVER=1 any degrade case becomes a hard error — for CI that must not silently fall back
AUTOMOBILE_SKIP_VIDEO_SERVER_DOWNLOAD=1 local-only (override / Gradle build); the network is never touched. A dedicated flag — it does not overload AUTOMOBILE_SKIP_CTRL_PROXY_DOWNLOAD, whose CtrlProxy APK is mandatory

See Environment variables for the full flag reference.

Optional audio

Audio is opt-in (AUTOMOBILE_WEBRTC_AUDIO=1 or "audio": true). The publisher adds a sendonly PCMU audio track. Android video-server captures 8 kHz mono PCM16 from REMOTE_SUBMIX, while iOS Simulator-window capture uses ScreenCaptureKit; the TypeScript publisher converts the PCM to PCMU RTP. Because screenrecord is video-only, Android audio-enabled streams require the persistent automobile-video.jar path. If its jar is unavailable or REMOTE_SUBMIX cannot initialize on the device build, stream start fails instead of silently publishing video-only audio. Physical iOS playback capture is unavailable through public APIs.

Android playback capture is policy-limited: some apps/usages cannot be captured, and REMOTE_SUBMIX is privileged. The reference coordination server forwards both the H.264 and PCMU tracks to WHEP subscribers and exposes audio plus audioPacketsForwarded in its reconnect descriptor.

Known limitations / future work

  • Platform capture differs. Android capture prefers the persistent video-server jar and falls back to screenrecord; iOS capture uses the CtrlProxy helper and local ffmpeg H.264 encoding. Keep both helper binaries available on workers that may stream either platform.
  • ~~Persistent-encoder distribution.~~ Resolved: the video-server jar now ships as a checksum-verified GitHub release asset and is downloaded/cached on demand, so production installs use the persistent encoder by default. See Persistent-encoder delivery above.
  • Audio is opt-in and platform-constrained. iOS audio requires Simulator-window capture; physical devices remain video-only. Android audio depends on REMOTE_SUBMIX availability for the shell-owned video-server process.
  • Reference server is single-process/in-memory. Use a hardened WHIP/WHEP SFU (MediaMTX, LiveKit, Janus, Cloudflare) in production; the publisher is unchanged.
  • Trickle ICE is opt-in. By default the publisher gathers candidates before POSTing the WHIP offer (non-trickle), which is simplest and widely compatible. Set AUTOMOBILE_WEBRTC_TRICKLE_ICE=1 (or trickleIce: true) to publish the offer immediately and PATCH candidates incrementally (application/trickle-ice-sdpfrag) so setup doesn’t stall on the gathering timeout — requires an ingest server that supports the WHIP trickle extension (the bundled reference server does; see trickleIce.ts / WhipClient.patchCandidate).

References