Unit lane scaling¶
How scripts/test-ts.sh unit spreads the Node unit lane over shards and processes (#10583). Under
--isolate most of the lane’s time is per-file realm setup and module re-import, not test bodies, so
files that cannot leak state share one process.
Shards¶
scripts/test-ts.sh assigns files round-robin: it walks the sorted find test -name '*.test.ts'
list (shared-allowlist files first) and gives file i to shard i mod N, so each shard gets an even
share of both groups. Each shard runs its shared files in one bun test process without --isolate
(scripts/lib/bun-unit-groups.sh), then its other files under --isolate. Both use the canonical
flags and preloads and run under the shard’s single watchdog. JUnit goes to shard-N-shared.xml and
shard-N-iso-K.xml; the timing gate globs *.xml.
Isolated chunks¶
The isolated group runs in sequential bun test --isolate processes of
AUTOMOBILE_UNIT_ISOLATED_CHUNK_SIZE files (default 30; 0 = one process), because a long isolated
process slows as it grows (#10213). Chunked shards (AUTOMOBILE_UNIT_TEST_CHUNK_FILES, nightly macOS)
chunk each group separately and never mix them.
Wall time, reaping and retries¶
Each shard subshell writes its own elapsed seconds to scratch/test-ts-<mode>-shards/shard-N.wall,
and test-ts: unit shard N/M wall= reports that, falling back to the reap-time measurement only when a
killed shard wrote none. The bats test
unit shard wall time is each shard's own duration, not its reap time (#10583) covers it.
The parent reaps attempts in completion order. It polls the live
shard pids (AUTOMOBILE_UNIT_SHARD_POLL_SECONDS, default 0.2 s, because macOS
Bash 3.2 has no wait -n). A shard that infra-exits early is retried at once,
and its retry window is judged against the lane cap at its own end time. A
wall-budget timeout whose own attempt calibration reaches
AUTOMOBILE_UNIT_CONTENDED_SLOWDOWN (default 2.5x) skips the same-runner retry
and reports RUNNER-CONTENDED.
A failing unit job (Node Unit Tests, Node TypeScript Build and Test, a TypeScript Code Coverage (N/2) part) qualifies
itself. Its last step (if: failure()) runs
scripts/ci/rerun-contended-runner.sh dispatch against the job’s own step summary.
On attempt 1 only, that step dispatches the workflow_dispatch Rerun contended runner
workflow (#12758). That workflow waits, bounded, for the run to finish,
re-verifies from the fetched logs that every failed job is a unit lane carrying
the marker, then reruns the failed jobs once. Fork PRs have a read-only token, so
the dispatch is skipped or logs a warning and never fails the job; a log-fetch
failure is a logged skip. The per-shard .wall record still excludes the end probe and the poll latency.
Which files share a process¶
| Tier | Rule | Runs |
|---|---|---|
| A: must isolate | mock.module, native/global patching, chdir. These change the module registry or process-wide natives, and afterEach cannot reliably undo them. |
Isolated |
| B: restorable state | Any other signal: spies not proven restored, env writes, singletons, DB, testOverrides, system time, moduleState. |
Isolated |
| C: no signal | Pure tests over injected fakes, and spy-only files whose every spy is proven restored. | Shared |
moduleState is free-function set…(/reset…( calls (setDeviceToolsDependencies,
setObserveCacheStore, resetAdbClientCaches) and X.resetForTests(), not methods on fakes or timer
globals. Files that write state through a production default collaborator a static check cannot see
are listed in SHARED_PROCESS_EXCLUSIONS with the reason.
Spy restore proof¶
Spies must be installed directly in a test or beforeEach callback; beforeAll, module/describe
bodies and helper-installed spies remain isolated, even with a restore hook.
The AST proof accepts an immediate .mockRestore(), a using declaration,
or a matching lexical variable/static field/array handle restored
unconditionally in the same test or a covering afterEach/afterAll.
Known mockImplementation/return-value chains preserve the handle. A covering
unconditional mock.restore(), restoreAllMocks() or jest.restoreAllMocks()
teardown also qualifies. Sibling/child describe hooks, conditional cleanup,
shadowed or overwritten handles, and unknown data/control flow do not prove a
restore. This is a bounded structural proof, not full exception-path analysis;
use try/finally for cleanup that must survive a failed assertion. There is no
blanket restore preload: it breaks suites that deliberately retain spies.
bun scripts/test/classify-shared-safe.ts --explain prints
file:line:reason for files blocked only by unproven spies. Supply a unit path
to inspect its unproven sites even when other signals also block it. --check
reports these sites if a listed file loses its restore proof.
Allow-list, runner and leak detection¶
- The list.
test/shared-process-allowlist.txtholds the shared files.bun scripts/test/classify-shared-safe.tsgenerates it from the tier signals above. Fast Validation runs the script with--check(shared-process-allowlist). The check fails when a listed file gained a tier A or B signal, no longer exists, is excluded, or when the list is unsorted or has duplicates. A new tier C file that is not listed only prints a hint: files that are not listed run isolated. - The runner. See Shards. The changed lane, coverage and explicit file
targets do not split groups.
AUTOMOBILE_UNIT_SHARED_PROCESS=0restores the all-isolated lane. Windows unit tests run in concurrent Bun--shardprocesses without isolation;AUTOMOBILE_UNIT_TEST_WORKERS=1restores the single command. Windows host integration runs all existing unsafe-tier files in one isolated process beside safe shards that also include unlisted files;AUTOMOBILE_INTEGRATION_TEST_WORKERS=1restores its single command. Unlikescripts/prepush-integration-safe.sh, the Windows lane does not give integration processes isolatedAUTOMOBILE_*state paths. A safe-tier file that falls back to a default path can therefore meet the unsafe process. Fix a Windows-only flake that appears only with concurrency in the test or reclassify the fileunsafe, withAUTOMOBILE_INTEGRATION_TEST_WORKERS=1as the kill switch. Any passthrough argument or non-emptyAUTOMOBILE_TEST_WALL_TIMEOUT_SECONDSkeeps both Windows lanes in a single process. - Timing probe. A shared process runs the preloads once. The probe
therefore records one entry per shared process, labelled through
AUTOMOBILE_TEST_TIMING_GROUP_LABEL, for exampleunit shard 0 shared process (360 files). The watchdog names that label, and the per-file headers in Bun’s log name the file. - Leak detection. The nightly
Node Randomized Unit Testsjob runs every unit file, including the shared group, in one non-isolated process with--randomize --seed=<run number>, so the shared group runs in a new random order every night.
A leak no seed has hit can surface on an unrelated PR, because adding or removing a file shifts which files share a process and their order. Fix a listed victim by excluding the victim or polluter (or fixing its reset), not by re-running.
Host coordination and cancellation¶
Unit and changed lanes acquire host slots before choosing their shard
count. AUTOMOBILE_HOST_TEST_SLOTS defaults to ceil(logical cores / 2), minimum
two; off disables it. The shared pool lives in the coordination directory’s
test-slots subdirectory. A sole holder of the entire pool can acquire extra
slots up to its existing worker demand, retaining idle-host and hosted-runner
parallelism. Other concurrent lanes use remaining base slots, or one unslotted
worker if none are free. Explicit AUTOMOBILE_UNIT_TEST_WORKERS remains an
upper bound. Targeted --parallel also respects the selected file count.
Signal/EXIT cleanup stops shard and watchdog process groups and releases
slots. Watchdogs notice a dead caller within a one-second poll, and chunk
loops check the original lane owner before each group. See
scripts/README.md for the environment controls and overloaded-host retry
suppression. Changed-file selection remains Bun’s responsibility.
Coverage lane¶
On Merge’s instrumented lane (scripts/test-ts.sh coverage) runs as two
TypeScript Code Coverage (N/2) part jobs of two shards each. A 4-core runner
is already CPU-bound at two instrumented Bun shards, so a third shard on the
same host does not help. The split halves wall time instead, keeping each shard
inside its 720 s budget on the slower runner class. Part P
(AUTOMOBILE_COVERAGE_PART/AUTOMOBILE_COVERAGE_PARTS) takes every second
discovered file from offset P-1 and numbers its shards after the earlier
parts’. The parts therefore interleave like one four-shard lane and their chunk
reports and calibration labels never collide. The TypeScript Code Coverage
job joins the parts (scripts/ci/merge-ts-coverage-parts.sh) for the badge
and runs the 100 ms gate over every chunk report. Coverage discovery skips
test/lint/** and test/scripts/**: they scan repository files rather than
exercise src/, and still run uninstrumented in the unit lanes.