Manual-test harness: broker scenarios, clients and evidence¶
Part of the manual-test harness.
Broker scenarios and acceptance¶
broker-scenario.ts list reports the registry and class. The run NAME command
selects an isolated in-memory broker with a fake native port unless the entry is
device-only. Device entries always require explicit --device (for names in both
classes --device selects the device variant). Fake runs create no daemon or device and do
not prove native isolation. Device runs require an already-running private broker
batch, strict preflight, capability probing, owned persistent proxies, caps and boot
permits; the batch needs positive totalDeviceCap <= --max-devices, sessionCap >=
totalDeviceCap and zero pooling. Start with no live client drivers and verified
cleanup of earlier ledger grants; run scenarios separately from broker1 up traffic.
bun scripts/manual-test/broker-scenario.ts list
bun scripts/manual-test/broker-scenario.ts run capacity-refusal-and-replay \
--batch-dir "$B"
bun scripts/manual-test/broker-scenario.ts run parallel-acquire \
--batch-dir "$B" --device --max-devices 2 --dry-run
bun scripts/manual-test/broker-scenario.ts run daemon-restart-with-live-sessions \
--batch-dir "$B" --device --crash --max-devices 2 < /dev/null
--dry-run prints the plan without execution or evidence writes. --batch-dir (or
MT_BATCH) selects the evidence root; --max-devices defaults to 2, hard 4;
--crash is restricted to daemon-restart-with-live-sessions --device. Evidence is
evidence/<scenario>/summary.json: redacted requests, replies/errors, durations,
ledger deltas and explicit assertion limits.
| Scenario | Class | Expected evidence |
|---|---|---|
parallel-acquire |
device | At least two clients within cap; distinct session/native identities, matching profiles, observe per client, release and absence. |
capacity-refusal-and-replay |
fake + device | Device batch needs capacity + 1 clients, where capacity is device ops totalDeviceCap; cap winners plus one retryable refusal (retryRequires: "new_request_id") with no tools/grant; same refusal on replay after release; fresh attempt succeeds. |
release-destroy-verification |
fake + device | Released receipt, cleanup complete, native absence, old session refused, fresh session/native identity. Fake proves only fake-native absence. |
owner-death-and-abandon |
fake + device | Fake: abandon pending/unknown/ready identities and join cleanup. Device: disposable-owner exit closes the proxy (its closed.json reason is the transport-local proxy_exited_or_transport_closed) and the broker’s terminal receipt is session_owner_exited; ready abandonment yields session_request_abandoned and old proxy refusal. |
cross-client-isolation |
fake | Wrong-token status/release/rebind and generic lifecycle tools refused; legitimate client still observes. Use raw bound frames for live cross-client checks. |
daemon-restart-with-live-sessions |
device | Graceful restart preserves session/native identities and observe. --crash: interrupted requests become terminal session_owner_exited, replay is identical, destruction precedes fresh successors. |
#12451-restart-handoff |
device | Graceful restart handoff using the same restart assertions; no crash flag on this alias. |
#12452-retryable-prep-failure |
fake | Retryable catalog_unavailable replay is unchanged; unknown archetype fails validation; an absent exact Duo model is terminal unsupported_hardware. |
#12452-prep-timeout |
device | 1 ms preparation timeout is retryable (timeout, daemon_unavailable or provision_failed, whichever the deadline hits first) with settled cleanup; an absent exact Duo model is a nonretryable unsupported_hardware control. |
#12491-pid-reuse-fencing |
fake | Reused PID/start identity refuses before allocation; matching identity acquires and releases. |
#12491-pid-reuse-positive-control |
device | Recorded live disposable owner is accepted, then owner death closes proxy and destroys its device. |
The live adapter marks destruction/handoff resource samples not-verifiable
until the product exposes a read-only resource-state diagnostic. Native inventory
and receipts still verify their own assertions but do not prove internal resource
transitions or every instant of a restart handoff, so a passed summary is not full
resource-state qualification.
live-device-acceptance.ts --scenario broker uses the same adapter and runs
parallel-acquire, release-destroy-verification, capacity-refusal-and-replay
and daemon-restart-with-live-sessions against the existing private batch (it
creates no daemon). Select enough clients for two parallel winners and capacity + 1
for the refusal check; --clients a,b,c filters batch clients, --only takes a
comma-separated subset of the four names, --allow-crash adds the restart crash
variant. There is no --platform flag; platform comes from client requirements.
AUTOMOBILE_ACCEPTANCE_LIVE=1 bun scripts/live-device-acceptance.ts \
--scenario broker --broker-batch "$B" --max-devices 2 \
--confirm-live --test-owned-devices < /dev/null
--dry-run prints a plan without live confirmation. Live runs require all three
confirmation inputs, keep caps at default 2/hard 4, write signed
evidence/acceptance-broker-<id>.json and per-scenario summaries, and preserve
not-verifiable assertions. The full exact-device lane remains separate; see
Live device acceptance.
Slot-era to broker mapping¶
| Slot-era interface or assumption | Broker equivalent |
|---|---|
--managed-binding |
--broker-binding <client-or-request.json> on the same frame connection. |
proxy-session --config <managed-slot json> |
--client NAME, optionally --request <SessionConfig-json-or-path> for a fresh attempt. |
{{ownSessionId:<slotIndex>}} |
{{ownSessionId}} for the single grant; {{clientSessionId:<name>}} for another batch client. |
init.json.managedSlots |
init.json.session (flat outcome, UUID, profile, failure, cleanup). |
heldBy: managed_slot rows |
Broker-held: tolerate all hold strings and refuse shutdown locally; use release --client NAME. |
| Boot-gate “slot” | “permit”: permit-N.lock, JSON permit, waitingFor: permit. |
AUTOMOBILE_MANAGED_SLOT_* env |
Harness-owned AUTOMOBILE_SESSION_BROKER_CONFIG / AUTOMOBILE_SESSION_CONFIG; inherited values scrubbed. |
| “Managed release retains the slot’s device assignment” | Zero-pooling release destroys the device; verify receipt and native absence. |
amslot-* names |
Recognized through the product predicate for cleanup classification, never hardcoded in the harness; name alone never allows a device, the batch ledger does. |
Auxiliary sockets and observer sessions¶
aux derives recording, video and WebRTC paths from the private aux directory. The
product’s WebRTC resolver supports a socket-path override; initialized harness
environments clear inherited overrides to keep all sockets private. WebRTC requests
reuse sendWebRtcStreamRequest.
Recording start, stop and status require --session-uuid: a device session
(the owner for start/stop; any live device session may read status). A retained stop
result can still be fetched with its former owner’s UUID. Get sessionUuid from
init.json.session printed by proxy up/status or amN daemon active-sessions
--json. Recording refuses --observer at parse time. The socket is
device-gesture-recording.sock; the tools are recordToolCalls and
recordDeviceGestures, with daemon flag --tool-call-recording.
For broker video/WebRTC use --session-uuid for the granted session. In a separate
plain batch, --observer provides a sessionless viewer. It mints a UUID, registers
it on the control socket and heartbeats every 2000 ms by default (--heartbeat-ms
must be below the returned heartbeat timeout). It then injects sessionUuid, and
releases on exit.
--observer-name sets the client name; observer and explicit session UUID are
mutually exclusive. Observers grant viewing admission, not device ownership; see
Streams
and Recordings.
bash scripts/manual-test/in-daemon-env.sh --batch-dir "$B" --daemon 1 \
aux recording status --session-uuid your-session-uuid
bash scripts/manual-test/in-daemon-env.sh --batch-dir "$B" --daemon 1 \
aux recording start --device-id your-device-id --platform android \
--plan-name manual-plan --session-uuid your-session-uuid
bash scripts/manual-test/in-daemon-env.sh --batch-dir "$B" --daemon 1 \
aux recording stop --recording-id your-recording-id --session-uuid your-session-uuid
bash scripts/manual-test/in-daemon-env.sh --batch-dir "$B" --daemon 1 \
aux video '{"action":"subscribe","deviceId":"your-device-id","platform":"android"}' \
--observer --hold-ms 5000
bash scripts/manual-test/in-daemon-env.sh --batch-dir "$B" --daemon 1 \
aux webrtc list --observer --timeout-ms 45000
bash scripts/manual-test/in-daemon-env.sh --batch-dir "$B" --daemon 1 \
aux webrtc start '{"deviceId":"your-device-id","whipEndpoint":"https://example.com/whip"}' \
--observer
Recording prints one full response line. Video prints its acknowledgement, then
counts bytes after it for --hold-ms (default 5000) or until the server closes. It
then prints {ack, bytesAfterAck, closedBy} (binary newlines counted as bytes; the
product exports no packet decoder, so no packet counts are inferred). WebRTC prints
the full response; start requires whipEndpoint; other actions are stop,
status, list, await.
Exit codes. Frame and aux: 0 success, 1 typed refusal, 2 usage or
private-environment/device-start guard, 3 timeout or connection failure.
Private-daemon: 0 success/status; 1 typed non-success (daemon still alive, device not
stopped, inventory errors, preflight --strict with warnings); 2 usage/guard/I/O or
boot-gate wait deadline. Its start subcommand propagates the product launch
exit code, and boot-gate -- COMMAND exits with the command’s. Proxy: 0 success/status, 1
failed/refused response, 2 usage/guard/wait deadline, 3 serve lost its transport, 4
serve initialization failed. Inspect full evidence rather than treating exit 0 as
device-side proof. All three client CLIs support --help, -h, help and
<subcommand> --help on stdout (exit 0). The help lists flags, defaults, examples
and exit codes; parse errors go to stderr with a --help pointer and exit 2.
Python MCP SDK stdio clients¶
Use a separate non-broker private batch: a broker batch forbids startDevice and
inventory-resource calls. The scripts/manual-test/python/ client retains every request
and full response while reproducing #11951 (stdio close) and #11950 (multiline
typing). It uses the official Python mcp SDK, locked to 2.3.0: the TypeScript
proxy-session.ts and @modelcontextprotocol/client cannot reproduce Python/anyio
close-path errors, and the official SDK is the thing under test. Tests use
unittest like the repo’s other Python tests; the only third-party imports are
mcp and its dependency anyio; no claude-agent-sdk or LLM loop is needed.
Initialize and start a private daemon first. The Python client resolves its launch
through in-daemon-env.sh and harness-guard.ts (the TypeScript guard owns all
private namespace rules); resolve requires a live daemon pid file and a responding
private control socket. The proxy receives only the server entry point, with no
daemon startup flags. Real and fake modes both check every device-start call
(including afterPeerClosed) before spawning; selector-only starts need
--allow-selector. There is no arbitrary command, socket, port or real-server
environment override.
Set up Python 3.11+ inside an existing absolute batch directory. The setup script never creates the batch directory, defaults to no batch, never deletes a venv, and prints just the interpreter path on stdout. Its optional version override installs after the hashed lock and reports UNLOCKED on stderr (use it only to compare SDK lines; the 1.30.0 line reproduces the transport failure).
B=/absolute/private/batch
PY="$(bash scripts/manual-test/python/setup-venv.sh --batch-dir "$B")"
# Optional comparison (reuses this batch's venv):
PY="$(bash scripts/manual-test/python/setup-venv.sh --batch-dir "$B" --mcp-version 1.30.0)"
printf '%s\n' "$PY"
Single-client close modes all run the script first, except mid-call, which starts
the last call/read without awaiting it and leaves the contexts after the delay.
normalleaves the contexts.after-server-exitcloses the captured child’s stdin and waits for exit.sigterm-serverandkill-serversignal only that captured child, then wait.--server-exit-wait-ms 0races the signal with context close.
A failed process capture refuses modes needing a process handle before any step (no
pid is guessed). SDK contexts always close in their owning task; close deadlines kill
only the captured child and report CLOSE_HANG.
B=/absolute/private/batch
PY="$B/python-venv/bin/python"
SCRIPT=scripts/manual-test/python/scripts/observe-then-close.json
for MODE in normal after-server-exit mid-call sigterm-server kill-server; do
"$PY" scripts/manual-test/python/mcp_stdio_client.py \
--batch-dir "$B" --daemon 1 --script "$SCRIPT" --var platform=android --var deviceId=your-device-id \
--out-dir "$B/python-$MODE" --close-mode "$MODE" \
--server-exit-wait-ms 5000 --mid-call-delay-ms 200 \
--init-timeout-ms 60000 --step-timeout-ms 60000 --close-timeout-ms 30000
done
For two emulators sharing one private daemon, each stdio server belongs to its own asyncio task and both initialize before traffic.
a-first: A closes as soon as its steps finish while B continues and runsafterPeerClosedafter A’s close plus--stagger-ms.b-firstmirrors it.simultaneousholds both at a barrier and offsets B by the stagger.
Repeat five lifetimes to retain each close separately:
B=/absolute/private/batch
PY="$B/python-venv/bin/python"
"$PY" scripts/manual-test/python/two_clients.py \
--batch-dir "$B" --daemon 1 \
--script scripts/manual-test/python/scripts/sendkeys-multiline.json \
--var-a platform=android --var-a deviceId=emulator-a \
--var-b platform=android --var-b deviceId=emulator-b \
--out-dir "$B/python-pair" --lifetimes 5 --close-order a-first --stagger-ms 200 \
--close-mode-a normal --close-mode-b normal
observe-then-close.json: lists tools, observes the explicit device twice, readsautomobile:devices/booted, idles briefly; its peer continuation observes again.sendkeys-multiline.jsonrequires an editor already focused: default multiline typing, repeated type commands in one call, clear then type, explicitmode: "ime", three character-at-a-time calls with retries, a final observation. Retrying a typing error can duplicate text; inspect the retained attempts.- Both first call
startDevicewith requiredplatformand explicitdeviceIdvariables; the stdio proxy binds the returned session and injects it into later calls, so observe/sendKeys omitdevice/deviceId(deviceis a plan label). --script-a/--script-breplace--scriptfor asymmetric traffic. Script keys and operations are strict;${name}substitution uses--var(or--var-a/--var-b) in string values including nested arguments. Unknown keys, missing variables, non-object arguments and negative numbers refuse before launch.retriesmeans extra attempts, for exceptions orisError.
For a dry run, replace the real launch flags with --fake-server; --fake-env K=V
applies only to the bundled fake. The fake uses the installed SDK’s server API
(FastMCP in 1.x, MCPServer in 2.x), implements scripted tools and the
fake://status resource, and bounds every fake to 60 s. The FAKE_ON_EOF variable
accepts exit (default), write-then-exit:20 or ignore; FAKE_IGNORE_SIGTERM=1 adds the force-kill case.
B=/absolute/private/batch
PY="$B/python-venv/bin/python"
"$PY" scripts/manual-test/python/mcp_stdio_client.py --fake-server \
--fake-env FAKE_ON_EOF=write-then-exit:20 \
--script scripts/manual-test/python/scripts/sendkeys-multiline.json \
--var platform=android --var deviceId=fake-device --out-dir "$B/python-fake"
"$PY" scripts/manual-test/python/characterize_close.py
"$PY" scripts/manual-test/python/characterize_close.py --json --only normal
Evidence¶
The output directory is absolute, created mode 0700. Reusing one containing any named
evidence file (or lifetime-001 for two clients) is a usage error before guard
checks or spawn. Files:
A.jsonl: append-only, 0600, flush per record; fsync per record except notifications, which fsync every 100 and on close.A.identity.json: Python/platform/package versions, command/args, environment keys only, close mode, script path and SHA256.A.server.stderr.log.
JSONL records spawn-request and initialize-request before launch/initialization
(retained even when spawn fails: exception with phase: "spawn"). It then records
spawn pid, initialization, the request before every attempt, and the full
model_dump(mode="json", by_alias=True) response or exception. It also records
notifications when the SDK offers a handler, server-request method/params,
transport-exception details, server exit status/timing, and close.
Step timeouts bound only the wait: a timed-out attempt records timedOut: true and
stillPending: true, counts as failed, and may retry while its request continues.
The eventual result is a late-response/late-exception with the same
index/attempt and lateByMs. After context exit pending requests get a bounded 3 s
wait with afterClose: true; the client cancels the rest and records them abandoned.
Summaries count lateResponses and abandonedRequests; step stdout includes
timedOutAttempts. Initialization has its own --init-timeout-ms (default 60000).
Exceptions include flattened exception-group leaves and full tracebacks (close
tracebacks also go to stderr). Per-step stdout JSON is compact with error text
truncated at 300 characters. Application failures and close failures stay separate
in the final summary. Two-client evidence uses lifetime-001/{A,B}.* and a plain
summary table followed by JSON; monotonic absolute timestamps allow overlap and peer
ordering checks.
Exit codes: 0 every step passed and CLOSE_OK; 1 a step failed and close
succeeded; 3 close raised or hung; 2 usage or guard refusal. For multiple
lifetimes the precedence is 2, 3, 1, 0. A close result classifies transport
failures, cancel-scope misuse, server-gone, timeout or other exceptions, and an
earlier failure is never replaced by cleanup. Cancellation, KeyboardInterrupt and
SystemExit persist the attempt exception, stop remaining steps, record the
application failure once, then close and write a summary (a cancelled run with
CLOSE_OK exits 1). Deliberate mid-call cancellation remains a scenario.
characterize_close.py measures every ordering in a fresh child interpreter with a
hard timeout (default 30 s). A hang is CLOSE_HANG; it terminates that interpreter
by pid and kills only recorded spawn pids whose command line still contains the
absolute bundled fake-server path (it skips unverified pids). It prints identity
plus a Markdown table, or machine JSON with --json (--only selects smoke
scenarios), and exits 0 regardless of outcomes. It annotates expected outcomes from
the same SDK-major table the tests use, and never rewrites measured outcomes to fit.
Evidence is under scratch/pyharness/characterize-* unless --out-dir is given.
The harness’s own fake-server runs on Python 3.11.10, anyio 4.15.1 and pydantic
2.14.1 measured these outcomes with mcp 2.3.0 and 1.30.0. The table is regenerated
with characterize_close.py; CLOSE_RAISED describes context cleanup separately
from application errors. notification-spam-then-close on 1.x is timing
dependent: a slow notification interval sometimes closes cleanly.
| Ordering | mcp 2.3.0 outcome | mcp 1.30.0 outcome | leaf classes on 1.30.0 | Avoidable by |
|---|---|---|---|---|
| normal | CLOSE_OK | CLOSE_OK | — | none needed |
| server-exits-then-idle-close | CLOSE_OK | CLOSE_OK | — | none needed |
| server-exits-then-call | CLOSE_OK | CLOSE_OK | — | none needed |
| server-exits-during-call | CLOSE_OK | CLOSE_OK | — | none needed |
| mid-call-cancel | CLOSE_OK | CLOSE_OK | — | none needed |
| mid-call-leave-contexts | CLOSE_OK | CLOSE_OK | — | none needed |
| sigterm-then-close | CLOSE_OK | CLOSE_OK | — | none needed |
| sigkill-then-close | CLOSE_OK | CLOSE_OK | — | none needed |
| server-closes-stdout-then-close | CLOSE_OK | CLOSE_OK | — | none needed |
| server-ignores-stdin-eof | CLOSE_OK | CLOSE_OK | — | server |
| server-ignores-eof-and-sigterm | CLOSE_OK | CLOSE_OK | — | server |
| notification-spam-then-close | CLOSE_OK | CLOSE_RAISED | anyio.BrokenResourceError | SDK |
| server-writes-after-stdin-eof | CLOSE_OK | CLOSE_RAISED | anyio.BrokenResourceError | server |
| late-response-after-leave | CLOSE_OK | CLOSE_RAISED | anyio.BrokenResourceError | SDK |
| session-closed-transport-open-spam | CLOSE_OK | CLOSE_RAISED | asyncio.exceptions.CancelledError, anyio.BrokenResourceError | SDK |
| two-clients-own-tasks | CLOSE_OK | CLOSE_OK | — | none needed |
| two-clients-exit-stack-non-lifo | CLOSE_RAISED | CLOSE_RAISED | builtins.RuntimeError | client |
| close-in-other-task | CLOSE_HANG | CLOSE_RAISED | builtins.RuntimeError | client |
| earlier-application-exception | CLOSE_OK | CLOSE_OK | — | none needed |
| caller-cancellation | CLOSE_OK | CLOSE_OK | — | none needed |
In 1.x, after the session closes its read stream, any further stdout line makes the
transport reader send into a stream with no receiver. That raises
anyio.BrokenResourceError; 1.x catches only ClosedResourceError, so its task
group fails and transport exit raises an exception group. 2.x catches both. The
trigger is any stdout write between session closure and transport process reaping. A
correct client keeps contexts in one task and closes them LIFO. For the 1.x
transport bug, upgrade to 2.x or catch the exception group at stdio_client exit;
client ordering alone cannot fix a server’s post-EOF writes.
Where auto-mobile’s stdio server falls: OutputGatedTransport
(src/server/outputGatedTransport.ts, #11951, #12049) drops writes after the client
closes. After stdin end/close/error, the server runs requests it
already received to completion and writes their results or errors, in direct
(--no-proxy) and proxy mode. Everything unsolicited is dropped (list-changed,
resource-updated, logging, progress even for a received request, and server-to-client
requests). The gate closes fully after the last answer’s write settles, then normal
shutdown releases sessions and exits 0; idle EOF closes the gate synchronously.
Each call keeps its own deadline (120 s by default via
AUTOMOBILE_DAEMON_TIMEOUT_MS). The overall EOF answer wait is capped at
MAX_CALLER_MCP_REQUEST_TIMEOUT_MS (30 min), before the normal 9 s shutdown bound
and 100 ms finalization. At that bound the gate stays answers-only while shutdown
starts: the proxy aborts a remaining call and its typed “MCP proxy is closing” answer
is still written. A stdout error (EPIPE) closes the gate fully and starts shutdown
at once; SIGTERM/SIGINT also start shutdown immediately, including during the EOF
wait, with typed errors still written while the gate allows received answers. EOF
before broker session acquisition finishes still shuts down immediately and aborts
acquisition (#11173). test/server/stdioClientGoneSilence.test.ts pins these
orderings with fake streams; they have not yet been observed against a live daemon.
The remaining 1.x Python SDK race is a client closing while its own request is
still in flight. That response can land in its close window and raise
anyio.BrokenResourceError (the late-response-after-leave row; answering a
received request after EOF is intentional). A client closing idle gets CLOSE_OK;
unsolicited post-EOF output is dropped.
To confirm on devices, use the 1.30.0 venv (the line that raises) against a private
daemon built from the commit under test. Run two clients on two emulators, ten
lifetimes each, closing idle after their requests finish. Every close record must be
CLOSE_OK, each client’s session must be gone from the daemon within about 2 s of
its close, and no stdio server process may remain:
B=/absolute/private/batch
PY="$(bash scripts/manual-test/python/setup-venv.sh --batch-dir "$B" --mcp-version 1.30.0)"
for ORDER in a-first simultaneous; do
"$PY" scripts/manual-test/python/two_clients.py \
--batch-dir "$B" --daemon 1 \
--script scripts/manual-test/python/scripts/observe-then-close.json \
--var-a platform=android --var-a deviceId=emulator-a \
--var-b platform=android --var-b deviceId=emulator-b \
--out-dir "$B/python-pair-1x-$ORDER" --lifetimes 10 --close-order "$ORDER" \
--stagger-ms 200 --close-mode-a normal --close-mode-b normal
done
A piped one-shot run must print both answers and exit 0, in proxy mode and direct
mode (--no-proxy), using the server built from the commit under test:
printf '%s\n' \
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"one-shot","version":"1"}}}' \
'{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"listDevices","arguments":{}}}' \
| "$B/bin/am1" bun "$PWD/dist/src/index.js"
printf 'exit=%s\n' "$?"
The single-client --close-mode mid-call check on the 1.30.0 venv can still raise
anyio.BrokenResourceError when its own answer reaches the close window; it is not
an idle-close confirmation.
The Python check uses python3 >=3.11, falling back to a uv-managed interpreter, and
runs when scripts/manual-test/python/ or the validator changes (path-selected in
scripts/prepush-shell.sh) and locally via bash scripts/all_fast_validate_checks.sh --only manual-test-python.