Persona robustness harness¶
scripts/persona-robustness/run.ts is a research prototype (#14061). It replays one test plan
under a stock baseline and a matrix of device perturbations, and reports which perturbations
break the plan and why. It is a script over the public MCP surface, not a product tool.
Matrix¶
| Kind | Applied with | Default set |
|---|---|---|
baseline |
manufacturerProfile apply |
pixel-baseline (always runs first) |
persona |
manufacturerProfile apply |
every other catalog manufacturer profile |
display |
displayProfile apply |
font-scale-1.3, density-larger, force-dark, cutout-tall |
facet |
keyboard, killProfile or displayProfile |
one per facet of a failing persona, with --ablate |
Fold posture is left out of the default display set because it needs a foldable AVD.
Replay loop¶
- Acquire one device session (
getAndroid, or--session) and hold it for the whole run, soexecutePlannever releases it (#15333). The session enables the harness’s tools (manufacturerProfile,displayProfile,killProfile,keyboard,executePlan), several of which are off by default. manufacturerProfileresetonce, so a previous run’s changes do not leak in.- For each perturbation and repeat: apply,
executePlan, thenmanufacturerProfilereset. That reset also undoes directdisplayProfile,keyboardsetProfileandkillProfilechanges, so one call restores every kind. - A failed reset stops the run: the device state is unknown and later replays would be misattributed. The report keeps the results so far and names the reason.
Scoring¶
| Outcome | Rule |
|---|---|
pass |
executePlan succeeded with no skipped optional steps and no warnings |
recovered |
succeeded, but skipped optional steps or reported step warnings |
failed |
executePlan failed; the report keeps failedStep |
error |
apply failed or reported a status other than applied (an unsupported overlay, a partial persona), or no plan result came back; says nothing about the app |
Repeats aggregate into a verdict: a single outcome is the verdict, pass with recovered is
recovered, any error is error, and failed mixed with success is flaky.
recovered reflects only the executor’s own tolerance (optional steps, warnings). It does not
measure agent-driven recovery; that needs an agent in the loop, as in scripts/agent-benchmark/.
Background survival matrix¶
The kill-profile survival harness (scripts/kill-profile-survival/) applies each kill profile on a
rootable emulator, advances the device clock and Doze, and scores eight fixture events as ran,
delayed or never. The expected scores live in src/features/utility/KillProfileExpectedScores.ts;
the harness requires API 33+.
| Profile | workmanager-periodic | workmanager-oneshot | jobscheduler | alarm-exact | alarm-inexact | foreground-service | boot-receiver | push-broadcast |
|---|---|---|---|---|---|---|---|---|
| samsung-sleeping | ran | ran | delayed | ran | delayed | ran | ran | ran |
| samsung-deep-sleeping | never | never | never | never | never | never | never | never |
| samsung-auto-disable | never | never | never | never | never | never | never | never |
| xiaomi-autostart-off | never | never | never | never | never | never | never | never |
| oneplus-deep-optimization | never | never | never | never | never | never | never | never |
| recents-swipe-kill | ran | ran | delayed | ran | delayed | ran | ran | ran |
| recents-swipe-force-stop | never | never | never | never | never | never | never | never |
| oneplus-sleep-standby | ran | ran | delayed | ran | delayed | ran | ran | ran |
| motorola-adaptive | ran | ran | delayed | ran | delayed | ran | ran | ran |
| user-restricted | ran | ran | never | ran | never | delayed | ran | ran |
| exact-alarm-denied | ran | delayed | delayed | never | delayed | never | ran | ran |
| notifications-off | ran | ran | delayed | ran | delayed | ran | ran | ran |
| huawei-powergenie | ran | ran | never | ran | never | delayed | ran | ran |
| vivo-autostart | ran | ran | never | ran | never | delayed | ran | ran |
| realme-power-saver | ran | ran | never | ran | never | delayed | ran | ran |
| meizu-flyme-background | ran | ran | never | ran | never | delayed | ran | ran |
| asus-power-master | ran | ran | never | ran | never | delayed | ran | ran |
| sony-stamina | never | never | never | never | never | never | ran | ran |
| tecno-app-halt | ran | ran | never | ran | never | delayed | ran | ran |
Scores were measured on an Android 15 (API 35) google_apis arm64 emulator
(EXPECTED_SCORES_API_LEVEL); other API levels have no baseline. samsung-foreground-kill and
screen-off-stop have no measured scores. The harness scores a profile only when
every step reports applied. Scores reflect emulator behaviour, not confirmed OEM effects.
A real reboot is not exercised: BOOT_COMPLETED is delivered by adb broadcast as root with
exclude-stopped flags, because shell cannot send this protected broadcast. On stock API 35 default
delivery requires a receiver, RECEIVE_BOOT_COMPLETED and a non-stopped package; restricted buckets
and background appops do not themselves block it. The boot-receiver check observes these conditions
without broadcasting.
Attribution¶
A perturbation that did not succeed is attributed against the baseline:
- When the baseline did not succeed, nothing is attributable.
- A display stressor or facet ablation changes one thing, so it is its own cause.
- A persona changes several facets. With
--ablate, each facet that differs from the baseline replays alone; the facets that fail alone are the cause. A persona is reported as an interaction only when every facet replayed alone and passed; if none failed alone but a facet replay errored or never ran (an aborted run), the attribution is inconclusive. Without--ablate, the report lists the candidate facets.
Every persona facet’s evidence status is printed with the report, so a simulated effect is never read as behaviour measured on the OEM device.
Limits¶
- The kill facet only acts when the app goes to the background. A plan that never leaves the app cannot fail because of it.
- One plan and one device per run. Comparing skins across plans means combining reports.