Manual-test harness: daemons and boots¶
Part of the manual-test harness.
Initialize and run private daemons¶
Run from the repository root after building the entry script, with unused ports,
an absolute evidence directory and a short socket root. Redirect stdin with
< /dev/null when invoking from an agent tool call.
Use init --broker-batch /absolute/broker-batch.json --count 1 for the broker
harness: daemon 1 is the sole broker, and device scenarios and broker acceptance
require exactly one daemon total. The batch input has schemaVersion: 1, limits
(totalDeviceCap, sessionCap, idlePoolCap), reuseCleanDevices, optional
reuseAndroidCleanDevices, explicit per-platform defaults, and a clients
array. Each client has a unique name matching ^[a-z0-9-]{1,32}$, platform,
and optional hardware (archetype string or product hardware selector), os
(product requirement contract), idleTimeoutMs, preparationTimeoutMs. The
harness uses the product parsers; see Session requirements
and the session broker operator/consumer contract.
The batch also carries the product’s catalog classification keys. The host-config
parser validates them and the harness copies them into broker/host-config.json:
androidProfileArchetypes (AVD profile id to phone, tablet, or foldable, e.g.
{ "medium_phone": "phone" }) and iosClassifications (simulator device type
identifier to model type). Omit them and the first acquisition fails with
catalog_metadata_invalid (“lacks current archetype metadata”).
Android os values are the release major or version ({ "kind": "major", "major": 16 }
with preferredVersion: "16"), never the API level (36). Among installed images of
the same release the resolver prefers google_apis over google_apis_playstore,
which can stay adb unauthorized for many minutes after boot.
--max-devices defaults to 2, hard cap 4, and must cover totalDeviceCap.
Zero pooling is the default (idlePoolCap: 0, reuseCleanDevices: false,
reuseAndroidCleanDevices: false). init --allow-reuse only permits nonzero
pooling; there is no named clean-reuse runner scenario, and the scenario/acceptance
adapter requires zero pooling. Do not use reuse for the ordinary batch or claim
destruction evidence from it.
B="$PWD/scratch/manual-broker-batch"
bun scripts/manual-test/private-daemon.ts init --batch-dir "$B" \
--count 1 --base-port 3160 --aux-root /tmp/am-manual-broker \
--server "$PWD/dist/src/index.js" --broker-batch /absolute/broker-batch.json \
--max-devices 2 --protect protected-device-name \
--daemon-arg --plan-execution-lock-scope --daemon-arg session \
--daemon-arg --tool-call-recording < /dev/null
bun scripts/manual-test/private-daemon.ts start --batch-dir "$B" < /dev/null
"$B/bin/broker1" up --all --max-devices 2 < /dev/null
"$B/bin/broker1" status < /dev/null
"$B/bin/broker1" down < /dev/null
"$B/bin/broker1" report < /dev/null
bun scripts/manual-test/private-daemon.ts stop --batch-dir "$B" --daemon 1 < /dev/null
init only writes files; it starts no process. Topology:
<batch>/batch.json private manifest, broker daemon 1
<batch>/broker/host-config.json rendered operator defaults and limits (0600)
<batch>/broker/clients.json client profiles, initial request IDs, token paths (0600)
<batch>/daemon1/env.sh private paths; pid.txt mirrors the started daemon
<batch>/bin/am1 proxy1 frame1 aux1 broker1
<batch>/sessions/<client>/ request, owner token, attempts and connection evidence
<batch>/evidence/ledger.jsonl append-only grants/native bindings/cleanup evidence
amN cli … invokes the one-shot product CLI; proxyN, frameN and auxN select
daemon N; every brokerN wrapper selects daemon 1. Run broker clients through
proxy1/frame1. wrappers --batch-dir "$B" regenerates the wrappers alone; a
batch command uses MT_BATCH from its wrapper.
broker-batch.ts up (broker1) defaults to all clients, accepts --clients a,b or
--all, --stagger-ms N (default 0) and --max-devices N. It reruns
preflight --strict, requires a live session-broker/v1 socket, and holds a
boot-gate permit through every proxy initialization. Planned ready clients finish
before clients expected to get capacity_exhausted start (extra clients deliberately
test refusals). The capacity_exhausted refusal reports retryable: true with
retryRequires: "new_request_id": replaying the same request ID returns the
identical stored refusal, so a retry needs a fresh ID. The acquisition deadline is
26 minutes including the boot gate; a timeout signals no process. Inspect status,
then down before another batch. The down command closes clients in reverse start order,
waits for receipts and verifies native absence (240 s budget). The report command writes
evidence/broker-report.json (outcomes, timings, ledger data).
--env K=V is repeatable for all daemons; --daemon-env N:K=V overrides per
daemon. --daemon-arg TOKEN records each daemon argv token (N:TOKEN selects a
daemon; --daemon-arg=--debug also works). Restarts reuse recorded env and argv;
start -- … adds one-off argv. The harness owns --port and --strict-port.
init records AUTOMOBILE_PROJECT_ROOT as the harness checkout, so local runner
source builds find ios/control-proxy; --env/--daemon-env override it.
start refuses a recorded live PID, then mirrors the daemon’s JSON PID to
daemonN/pid.txt. The stop command confirms identity through ide/status on the
private control socket, sends SIGTERM to the recorded PID, and waits up to --wait-ms
(default 30000). It prints one JSON line with outcome (stopped, still-alive,
gone-before-signal) and via (signal, product-cli, none). When identity is
not confirmed it delegates to the product CLI (amN daemon stop remains the product
stop path). It never escalates to SIGKILL; exit 1 means still-alive and preserves
pid.txt, success removes it. A stop --wait-ms 0 signals and reports still-alive
unless the daemon is already gone.
For the explicit crash-recovery test only: private-daemon.ts stop --batch-dir "$B"
--daemon 1 --crash sends SIGKILL only to this batch’s recorded daemon PID. It first
confirms PID, socket and PID file through ide/status, matching
daemon-start.json, and re-checks the recorded process start identity right
before signalling. It never delegates, signals children or process groups, or kills
by name; missing or mismatched identity refuses. Success reports crashed.
status reports JSON/plain PID agreement (pidTxt: ok, missing, stale),
liveness, socket response, port and env file. The status --devices form appends host
inventories and the daemon’s own view (daemonDevicesMethod:
"daemon/availableDevices"). It is read-only apart from that method’s inventory
ownership/discovery refresh and boots or claims no device. Android inventory reads
host:devices-l over the local ADB server socket without launching adb (it skips this
when no server listens). Each diagnostic failure comes back as evidence.
Restart means stop then start; wrappers refuse daemon start* and daemon
restart* shorthands. env --batch-dir "$B" --daemon 1 prints an env file path.
in-daemon-env.sh --cd /absolute/existing/dir opts into changing cwd (otherwise it
preserves the caller’s); its cli shorthand matches amN cli.
"$B/bin/frame1" session-info your-session-uuid < /dev/null
bash scripts/manual-test/in-daemon-env.sh --batch-dir "$B" --daemon 1 \
--cd "$PWD" daemon status < /dev/null
bun scripts/manual-test/private-daemon.ts status --batch-dir "$B" --devices < /dev/null
bun scripts/manual-test/private-daemon.ts wrappers --batch-dir "$B" < /dev/null
Path policy¶
Every client requires all initialized state variables and refuses host defaults,
canonical aliases and resident socket/PID/lock basenames. The audited policy is
exhaustive for configurable product state destinations:
| Kind | Variables | Harness policy |
|---|---|---|
state |
AUTOMOBILE_DATA_DIR, AUTOMOBILE_LOG_DIR, AUTOMOBILE_DB_DIR, AUTOMOBILE_COORDINATION_DIR |
Required per-daemon defaults under the evidence root. |
state |
AUTOMOBILE_ADB_SERVER_COORDINATION_DIR |
Batch shared default, or per-daemon with --split-adb-coordination. |
state |
AUTOMOBILE_AUX_SOCKET_DIR, AUTOMOBILE_DAEMON_SOCKET_PATH |
Short auxiliary root and d.sock. |
state |
AUTOMOBILE_DAEMON_PID_FILE_PATH, AUTOMOBILE_DAEMON_LOCK_FILE_PATH, AUTOMOBILE_DAEMON_TOKEN_FILE_PATH, AUTOMOBILE_DAEMON_LAUNCH_LOG_PATH |
Per-daemon PID, lock and launch log. |
state |
AUTOMOBILE_CTRL_PROXY_IOS_CACHE_DIR, AUTOMOBILE_CTRL_PROXY_IOS_DERIVED_DATA |
Private path selection only; optional private overrides. |
opt-in |
AUTOMOBILE_IOS_CTRLPROXY_RECORD_DIR, AUTOMOBILE_CAPTURE_FIXTURES_DIR, AUTOMOBILE_PLAN_HEALTH_DIR, AUTOMOBILE_TOOL_OUTPUTS_DIR, AUTOMOBILE_STARTUP_BENCHMARK_OUTPUT |
Unset by default; strip inherited values in both spellings. Explicit --env/--daemon-env requires an absolute private path. These enable exchange capture, fixture capture, plan health files/results, tool-output artifacts, or benchmark output when benchmarking is enabled. |
unset |
AUTOMOBILE_DB_PATH, AUTOMOBILE_WEBRTC_STREAM_SOCKET_PATH |
Strip competing overrides in both spellings. |
input |
AUTOMOBILE_CTRL_PROXY_APK_PATH, AUTOMOBILE_CTRL_PROXY_IOS_BUNDLE_PATH, AUTOMOBILE_CTRL_PROXY_IOS_IPA_PATH, AUTOMOBILE_WATCH_RUNNER_XCTESTRUN, AUTOMOBILE_CWEBP_PATH, AUTOMOBILE_DWEBP_PATH |
Caller-selected read-only artifacts/executables; no harness defaults. |
input |
AUTOMOBILE_DAEMON_LAUNCH_CWD, AUTOMOBILE_IOS_CODE_SIGN_ENTITLEMENTS_PATH, AUTOMOBILE_MIGRATIONS_DIR, AUTOMOBILE_NETWORK_FILTER_APP_PATH, AUTOMOBILE_PROJECT_ROOT |
Caller-selected working directory or build/signing inputs. |
input |
AUTOMOBILE_UNIT_TEST_ISOLATED_DATA_DIR |
Test preload isolation marker, not a state destination. |
input |
AUTOMOBILE_SESSION_BROKER_CONFIG, AUTOMOBILE_SESSION_CONFIG |
Harness-owned: inherited values stripped; forbidden in --env/--daemon-env. Host config is set only on broker daemon 1 start; client requests use --client. |
input |
AUTOMOBILE_FFMPEG, AUTOMOBILE_VIDEO_SERVER_JAR, AUTOMOBILE_IOS_SCREEN_CAPTURE_HELPER, AUTOMOBILE_IOS_PROTOTYPE_AGENT |
Caller-selected read-only encoder/helper inputs. |
Legacy managed-slot configuration and AUTO_MOBILE_* values are stripped. env.sh
unsets opt-ins unless configured, so sourcing it cannot silently enable them.
Clients validate opt-in paths only when supplied and discard inherited opt-ins;
daemon manifest overrides are retained on launch.
The registry is scripts/manual-test/productPathEnv.ts, checked exhaustively
against product env reads with its sorted non-path list. Harness entry scripts
preserve explicit stderr sinks. File logging before product imports requires an
explicit absolute log directory outside both home defaults, checked lexically and
through existing ancestors’ realpaths. All other cases (invalid sink values,
data-dir-only environments) force the canonical stderr sink. Child environments
restore every harness sink override so private daemons log to their private files.
Control sockets live under the short aux root, normally
/tmp/am-mt-<hash>/<N>/d.sock, not under the batch directory. Each daemon
isolates data, logs, database, coordination, auxiliary sockets, control socket,
PID, lock, launch log and iOS cache/build paths.
Known uncovered path: the libwebp cache has no environment override
(src/utils/image/webp/WebpBinaryResolver.ts), so a private daemon that downloads
libwebp writes to ~/.auto-mobile/libwebp. Supply AUTOMOBILE_CWEBP_PATH and
AUTOMOBILE_DWEBP_PATH to avoid it.
Daemons still share the host’s adb server, AVD inventory and simulator set. By
default the batch uses one shared adb-server coordination directory; the broker
also has a private per-daemon coordination namespace for its host lock and registry.
The init --split-adb-coordination option is only for scenarios whose daemons must not see
each other’s coordination state (a split registry can report false capacity
findings, #11905); it creates no separate adb servers.
Host preflight and staggered boots¶
preflight --batch-dir "$B" [--strict] reports platform, power source and charge,
system-wide sleep assertions, lid state and clamshell sleep. It also reports low-power
mode (from lowpowermode or powermode; absence of both is not a warning), the three
load averages, CPU count and free/total memory bytes. On macOS it reads pmset -g batt,
pmset -g assertions, pmset -g and ioreg; other platforms report those sections
unsupported; failed readers carry unavailable reasons. It needs a private batch
path but no batch.json, prints one JSON document and exits 0 (--strict exits 1
when warnings exist).
init (after writing the batch) and start (before launch) run the same checks as
warnings, record shared/preflight-<ms>.json, and print PREFLIGHT WARNING: lines
plus the caffeinate command; failures never refuse initialization or change the
launch result.
bun scripts/manual-test/private-daemon.ts preflight --batch-dir "$B" --strict
nohup caffeinate -dims > /dev/null 2>&1 & echo $! > "$B"/shared/caffeinate.pid
Keep AC attached and the lid open or an external display connected: caffeinate cannot prevent a clamshell sleep on battery. A host sleep expires every idle session on wake, so timings across a sleep are invalid (#12212); discard timings measured during a host load spike too.
Every broker acquisition holds a boot-gate permit across the whole proxy up,
including failed preparation (broker1 up and device scenarios do this for you):
bun scripts/manual-test/private-daemon.ts boot-gate --batch-dir "$B" --role a-phone \
-- "$B/bin/proxy1" up --client a-phone --expect ready < /dev/null
Roles match ^[A-Za-z0-9._-]{1,64}$. Defaults: --max-concurrent 1 (1-8),
--max-load twice the CPU count, --min-interval-ms 30000, --timeout-ms 600000,
--poll-ms 2000; keep concurrent acquisitions within the batch device cap. Each role
takes a shared permit-N.lock, waits for 1-minute load strictly below the threshold
and the last-grant interval, then records shared/boot-gate/last-grant.json. Dead
holders’ permits are reclaimed by the canonical file lock; with multiple permits the
minimum interval is best-effort; signal termination leaves a permit lock until its
PID is gone.
A wrapped command holds the permit until it finishes and propagates its exit code
(command tokens keep the caller’s environment and are not daemon flags; an empty
-- refuses). A bare gate releases at exit and its JSON says so. Grant JSON has
role, outcome, permit, waitedMs, loadAverage1m, maxLoad; a timeout
records waitingFor (permit, load, interval), releases the permit, runs
nothing and exits 2. Gate-only success exits 0.