The category ownership the reconciliation keeps (DeclaredCategories.kept)
was in UserDefaults.standard, and every XCTest process shares one
standard domain. In a full run, every other class's session that reached
data activation reconciled categories against the ownership this class
had just written there, found nothing registered, and wrote it back
empty. The class then took its own category for an embedder's and refused
the action ("notification action refused: … is not declared in
approve,note"), in whichever tests the other processes happened to cross.
It passed alone. The race has been there since ac741eca1.
DeclaredCategories names its defaults; the app's are the standard ones,
and the test names a suite per test and removes it after.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
28 KiB
Writing Exact's tests
How to write and change a test in this repository so that it stays trustworthy
and fast. Stage your files and run bun scripts/gate.mjs before landing.
It is the only blocking gate: at most 30 seconds warm after a one-crate edit
(rules/RULES.md). Every part runs after failures, and the command prints each
part's time and the total. Provisioning and the first compile are cold costs.
The gate checks formatting, caps, boot, default-member compilation, the unit tests of number text, colour, motion, plan, Contract syntax and types, and its own Bun control tests. Full build, tests, lint, workspace and wasm checking, semantic difftests, app drives, XCTests and platform builds run hourly. Run the gate, push, and fix what the hourly lane attributes to you. Use a targeted local test or drive to investigate; it does not add another blocking check.
Principles
- Tests MUST be reliable. Prioritize reliability over speed, although both are important. Optimizing a test to run faster must keep or strengthen the semantic correctness of the test.
- Conscientiously identify the unit and the integrations. Decide what a test covers: one function, one crate, a host over the runner, the whole app, etc. Test behaviors at the lowest level that exhibits them. Test an integration between subcomponents only where the test is about that integration.
- Combine semantically related checks. Combine semantically related tests to reduce the number of setups.
- Deduplicate. Delete tests that truly check the same thing.
- Delete vacuous tests, with care. A test that asserts nothing about this
repository's code (a check of Foundation's
Timer, say) or truly does not test anything meaningful is unnecessary. Read it first: an assertion can hide in a helper. - No external I/O unless it is the point. External I/O is relatively slow and unreliable. A subprocess, a network call, a real file or a nested build costs time and adds ways to fail. Tests that use compiled fixtures should use a binary the lane has built (below).
- Keep coverage. A randomized test, a fuzz loop or a stress test keeps its seeds and iteration counts. Change compiler optimization levels only for a whole test suite or when the intent is to test a specific optimization level.
Whose failure it is, and what has run
bun scripts/affected.mjslists the suites a branch's changes can affect (the working tree included) and how to run each.--runruns the selected Cargo and Bun suites, and--why <suite>shows the changed file that selected a suite and how the suite reaches it. A suite's inputs come from its sources: Cargo's dependencies, imports, and the paths, packages, apps and platforms its tests name. The hourly lane makes the same selection over all commits since its last run, keeps a failure until a run covers it again, and runs every step in full at least daily. It checks the newest tip once, then bisects each new failure through the commits that can affect it. Probes keep the failing affected suites and their prerequisites; a compile failure, unavailable prerequisite or timeout cannot count as a passing test. The lane confirms a reproduced failure and a pass at its actual parent before assigning a commit. An obstructed probe reports an unresolved range. Results and probe logs stay in the lane'starget/async/; issues and git notes carry the attribution. For a test to be selected when what it reads changes, name what it reads: a path (from the root or from the test's own directory),--app <name>orresolveApp('<name>'),-p <package>, or the platform in the call that drives it (spawnSync('bun', [agent, 'ios', …]),open({ host: 'web' })). Shared code (scripts/, a library) that names an app by name, or another platform's host, is taken to build or drive what its caller asks for, so a test that relies on it reading a particular app must name that app itself.--audit <suite>runs a suite with every unit outside its inputs unreadable (sandbox-exec), to find a read the selection misses. It sees what the suite reads as it runs, not what an earlier build read: a file a Rust crate includes (include_str!,#[path]) or a build script's input shows only when the audit builds that crate again.bun scripts/known.mjs -- <command>runscargo test …,bun test …orbun host/apple/build.mjs --test …and, when it fails, sorts each failing test against the async lane's record of origin/main (LLP 1146 D1): known (it fails there too, with how often it failed in the lane's last ten runs), new (the lane ran it there and it passed), or unknown (no lane run there covers it). The exit status stays the command's. The lane attaches what each commit's checks found to that commit as a git note (refs/notes/async, tier 2'sasync2), which it fetches from origin.bun scripts/evidence.mjs --app <name>lists the drives, test runs andverifyruns recorded against the app's builds (LLP 1146 D2,<app>/.exact/runs.jsonl): current while the build they ran on is still the host's and still fresh, else stale, saying what changed. It names the hosts the branch changed that no current run covers;--draftprints lines for a diary's Checkpoints or a PR's Test Plan.evidence.mjs note <host> "<what>" [--device "<model, OS>"]records something seen by hand. Web, macOS and iOS-simulator builds are identified; other hosts' runs are listed as unchecked.
Waiting
- Wait on events rather than sleeping for a duration. A fixed sleep is too
long on a fast machine and too short on a slow one. Some ways to wait for
events are:
- Rust: a channel, a condition variable, a count to observe.
- The TypeScript executor (
exact-js):Module::set_turn_steptimes each turn as one step of a clock that moves at no other time;exact_render::expire_rendersbrings render deadlines to now;Module::hold_compressionsholds compressions until released, and a storage wait with only held work in flight runs out at once. - Swift: an expectation fulfilled by the callback, or a run-loop pass that
returns once work has run:
RunLoop.main.run(mode: .default, before:).RunLoop.main.run(until:)waits out its whole slice; on a loaded Mac a 10 ms slice takes about 80 ms. The XCTests share three waits, inhost/apple/tests/ExactKitTests/EventWaits.swift:waitForwaits for a condition,waitForMainQueuefor the blocks already queued on the main queue, andwaitForActivationfor what follows a session's first pixel. - Browsers: an event or a promise, or the agent's
clock settleand virtual time. - A drive or an authored test:
clock settleorclock datafor what the app has in flight. A stream counts as landed when its first mapped value commits (LLP 1016.000 D5), so for state a protocol reaches later (a socket's READY after its HELLO) writeclock until state <path> == <literal> within <ms>, notclock +N real: the clock stays frozen while it pollsstateon real time, and a miss fails naming the last value. A task due a commit later (after(1, …)) needs aclock settlebefore it.
- Poll only when nothing notifies, and poll cheaply. Every 50–100 ms, with a
limit of tens of seconds that only stops hanging tests. A shorter interval can
be considered for state that is cheap to read in the test's own process, when
the poll significantly affects the test's speed. In a browser,
setTimeout(fn, 0)is already throttled and coalesced: HTML clamps it to at least 4 ms once nested five deep, and browsers slow and batch the timers of a hidden page. A check that something did not happen should wait until the work it watches has ended (a count of operations in flight, a join) and then check. - An iOS XCTest that hangs fails at its time allowance.
build.mjs --test --iossets the default per-test execution time allowance to 600 s: a test still running at its allowance fails with "Test exceeded execution time allowance", and the run goes on with the next test in a new process. - Keep waits simple. A wait reads as one condition and one bound. If
observing an event needs a new hook, the hook is small and read-only (for
example
Module::storage_in_flight).
Structure and isolation
- Spread independent cases across the CPU cores. When a test checks many
generated inputs (random bit patterns for fuzzing against a reference
implementation, the repository's documents), generate all the inputs first,
in order, then check them on several threads (
num'stests::par,std::thread::scope). Generating first keeps the inputs the same however many threads check them. Independent fixtures (runners, kernels) each run on a thread of their own. - Each test sets up what it uses. macOS XCTest classes run one per process
(
host/apple/xctest.mjs) and iOS classes one per simulator clone, so a class cannot rely on an earlier class. Install what it uses inclass func setUp(ExactSurfaces.install()for a canvas) and create what it reads (NSApplication.sharedbeforeNSApp).UserDefaults.standardis one domain for every macOS XCTest process (com.apple.dt.xctest.tool): a test that sets a key the host also writes gives the host a suite of its own (DeclaredCategories.defaults). - Run the classes a change touches.
bun host/apple/build.mjs --test --only FrameHealthMetricsTests,FrameHealthMacTestsbuilds as a full run does and runs only the classes named, on macOS;--test --ios --only FrameHealthMetricsTests,FrameHealthIOSTestsruns iOS classes on a simulator (--simpicks one). When a branch changes no input of the XCTests but Swift files intests/ExactKitTests,bun scripts/affected.mjsnames, for each platform, the classes those files declare or extend and the classes of any file that names one of them (a subclass, a caller of a helper), and after the push the async lane runs those iOS classes alone. A change to a file of shared helpers (one that declares a function, a type or an extension other files can use) or to anything else the XCTests read (the host, a fixture, a resource) runs every class. The lane runs no macOS class. - A Bun file that mocks a module runs in a global of its own. Bun's
mock.modulereplaces a module for the whole process and does not relink a module already loaded, so files that mock or import the same module (host/web-js/rt.js) break each other in one process: a mock that does not take can leave a test spinning on the real module, past any test timeout. Run several such files withbun test --isolate(a fresh global and module registry per file) or--parallel, which implies it and is how the async lane's glue step runs them. - Platform availability belongs in Swift. The iOS run includes every test
compiled for its destination, including shared classes without an
IOSTestssuffix. Guard platform-specific assertions with#if os(...). - Mac-wide state is serial. A class whose file uses the general pasteboard,
app activation, the mouse location or a screen capture shares one process with
the other such classes. A window's key status and first responder belong to
its own process and need nothing. A class that opens a menu that tracks (its
file observes
NSMenu.didBeginTrackingNotificationwithaddObserver(forName:)) runs alone after every other class: beside another test process, its menu can close before the test reads it. - Helpers are simple and correct under sharing. A helper shared by tests or
threads holds no mutable state, or holds it behind one obvious rule (a queue
index read and advanced with no
awaitbetween, a value drawn before the threads start). A browser shared by a file's tests gives each test a context of its own. - Build once, outside the test, where you can. A test compiles a Contract
fixture with the debug
contractbinary the lane's build step has built, run directly; notcargo run --release, and not acargo runper fixture. The compiler's output does not depend on its build profile. - An app's Rust tests are its data crate's. An app's logic and its Rust tests
live in
apps/<app>/data; itsweb,linux,appleandandroidcrates are host shells, tested mainly by driving the app (conformance, the smokes,scripts/agent.mjs). A workspace package without tests setstest = falseandbench = falseon its library and each binary target, socargo test --testsandcargo build --all-targetsbuild no empty test binary for it; remove both settings when adding the package's first test.bun scripts/caps.mjschecks this both ways from a scan of the package's sources, so a test that a macro or a generated file produces needs a reviewer's eye.
For an iOS simulator drive without a GUI, set EXACT_SHOW_SIMULATOR=0. The simulator
boots and runs the app, but the driver opens neither Simulator nor Device Hub.
Use a simulator you created when another person shares the Mac.
Apple agent drives keep iframe website data in memory. Unpartitioned frames share one store across all sessions in the process, separate from named partitions. A new drive process starts empty, even when it reuses an Exact scratch-store name. Ordinary app launches keep their saved website data.
UI tests
- Programmatic input over gestures. Typing text and programmatic presses are fast and exact. Use a gesture (a real touch, a drag, a swipe) when the gesture is what the test is about.
- Observe the UI instead of waiting for it. XCUITest's
waitForExistence(timeout:)and its non-existence counterpart poll with a delay of about a second. Query the app's state (tree,state, an accessibility read) for the condition instead. - Safe-area tests fill the screen. Size a UIKit test window to its screen's bounds and boot the session at that size. A fixed phone size may stop above another simulator's home indicator and correctly receive no bottom inset.
- A settled UI is a condition. After a transition or an animation, wait for
the condition the product itself uses (
Agent.nativeInFlight()on Apple, the agent'sclock settle), not a hardcoded sleep timeout.
For a contextPopover source, use tap <source> drag 0 0 press 1000 under
--touch platform, then clock settle. tap <action> selects UIKit's
row. An explicit tap <root> at <x> <y> on UIKit's backdrop dismisses it;
settle again before checking that state.navigation.popover is null.
tap <source> contextmenu delivers the recognized event and does not
qualify a real long press. scripts/smoke-touch.mjs covers ordinary and
projected grouped-list sources in both timing modes.
A real drag with during ops fails when the touch runner finishes before a
dispatch-log read sees the touch begin. The failure carries the aim, the injection
result and the log, and says whether the touch began and lifted. The log's t and
dispatched are seconds on the mach clock of UITouch.timestamp; the injection's
end is wall-clock milliseconds, so do not subtract one from the other. A dispatch-log
read, operation or clock step still pending a second after the runner finishes stops
the app's carrier, and the failure names it.
With an iOS session using touch: 'platform', await session.back({ native: true })
locates UIKit's Back button through XCTest inside the active navigation container
and taps it. Embedded sessions target their own container. The driver refuses missing
or ambiguous containers or buttons, so it cannot choose another session's Back.
Settle the session and assert its resulting route or screen.
The ordinary back() invokes the platform's handler;
it does not establish touch delivery. XCTest can wait a minute for the app and another
minute for SpringBoard to idle before tapping, so the native call has a 150-second bound.
ModalStackIOSTests also drives UIKit's animated popToViewController to
check multi-entry Back in primary and presented stacks. It asserts the retained
controllers and the router's chosen visit, including return to the sheet root
without changing the background stack. It also checks that header search commits
and blurs once before dispatch and keeps the typed query, newer app navigation
wins during a pop (including navigation from blur), and an unmapped destination
restores the declared stack.
A test exercises UIKit's registered search-cancellation action and the controller
dismissal used by Duo's inner-display Close, including dismissal in a presented stack.
It checks that cancellation clears the live and committed query, commits once and
keeps the route. It repeats cancellation after the Search key dismisses the keyboard
and checks that no second blur is sent.
The runner tests distinguish visits with the same URL and reject stale or
cross-tab jumps. These tests use UIKit's public stack API; they do not establish
activation of its Back-history menu.
ModalStackIOSTests.testBackHistoryMenuInThePrimaryStack and
testBackHistoryMenuInThePresentedStack use the XCTest touch runner to hold the
active stack's Back control for 0.8 seconds. They require an exposed menu before
selecting a destination, then check controller and visit identity with repeated
URLs, with and without authored navigation. A missing menu keeps its accessibility
snapshot and reports an unqualified interaction as a skip.
On iPhone 18 Pro with iOS 27.0, the long press performs an adjacent Back in both
fixtures and exposes no menu. Nonadjacent selection, repeated-URL identity and
authored navigation through the menu remain unqualified there.
Tests that declare missing("touch", …) run on the selected simulator without
clones, so their fixture and the touch runner share one screen.
With EXACT_SIM naming an iPhone Duo, rebuild Duo Lab and the host fixture, then run
bun scripts/smoke.mjs duo --only navigation,navigation-gestures,host-navigation.
These legs cover native Back taps, completed and cancelled edge and content swipes,
and Back in independent embedded sessions. The gesture checks observe UIKit's held
interactive transition and check content clearance within the moving route. After
release they assert the route, draft, focus and scroll position. UIKit owns the
content swipe recognizer.
--only secondary-navigation checks an unfolded Duo's centered UIKit sheet
with its own navigation stack over a background stack that also has Back.
It covers native Back taps, completed and cancelled edge and content swipes,
the secondary editor's focus and draft, destination scroll restoration, and
horizontal clearance during the gesture. The sheet's root refuses a native
Back tap rather than targeting the stack behind it.
host-navigation also opens the sample host fully open with EXACT_HOST_COLUMNS=3.
A UIStackView places a native label in a leading column before the two sessions.
The smoke checks that the three columns are equal across the width the stacked
panes fill, with the middle pane across the display's middle and the trailing pane
at its end. In each pane it then exercises native Back taps and completed and
cancelled swipes from the leading edge and the content. It checks each stack's
draft, focus, scroll restoration and horizontal clearance, that editing and
interactive Back keep the session's size and the editor's route in its column,
and that the other stack stays unchanged. The host uses UIKit's
custom containment
for this fixed arrangement; native Split View's adaptive column selection remains
a separate coverage limit.
The runtime refuses a host call made from a callback it waits on
(Owner.shared.callMain inside Owner.shared.sync). To test a report made there,
change the native state inside that pair, then wait for the runner to apply it
(testACoverFromAServedCallbackReachesTheRunner, testAReportWaitsForTheServedCallbackToReturn).
Browsers
- Launch Chromium through
chromium(), never by a path. It (scripts/agent-launch.mjs) answersCHROMEwhen it is set (a browser that is not there is reported as unavailable), else the Chromium that Playwright pins inpackage.json: by default its headless shell, which on an M5 Pro launched in 74 ms against the full build's 185 ms and used 263 MB of memory against 939 MB (the medians of 7 launches, every process of the browser counted).exact setupinstalls both builds (bunx playwright@<version> install chromium chromium-headless-shelldoes the same). - Run the device runner's own tests alone.
scripts/agent-test.test.mjsdrives a real app on the web, so it needs the pinned Chromium orCHROMEpointing at an installed Chrome, and one of its tests builds the app's web dist inside its 180 s limit: beside acargo buildorcargo testholding the Cargo lock, that build is killed at the limit and the test fails withReceived: null, which is contention, not a fault (seen twice, 2026-10-28). - Name a need when the shell lacks what the test uses.
chromium({ needs })answers the full build (Chrome for Testing, run with--headless=new) forgpu(WebGPU and WebGL on the GPU: the shell has only SwiftShader, whose fallback adapter a GPU app takes and then draws wrong),pdf(the PDF viewer),hevc,bfcache(the shell restores no page from the back/forward cache) andperf(timings that should match a person's browser). A web drive of an app with a GPU module asks forgpuitself (webNeeds). - A Playwright test takes
suiteBrowser(). It shares one browser in the process for each engine, build and set of launch options.newPageandnewContexteach make a browser context that belongs to the suite;closecloses those, not the browser. A test that speaks CDP itself takescdpBrowser()on the same terms:page(url)opens a target in the suite's context. Its browser quits after the test file, or before normal process exit outside Bun's test runner. A test about the browser process (it kills the browser, or needs a kept profile, a headed window or flags of its own) launches its own. - Wait for the event, not for its first sign. The shell is faster than the
full build, so a test that reads a state while the browser still changes it
fails there first.
document.fullscreenElementis set before the display-mode media changes are sent;fullscreenchangecomes after them.
Windows: the NucBox
Windows's oracle and its consumers' drives run on the NucBox, a Windows
machine on the team's tailnet (LLP 1115.004 D13 names it). Below, <host> is
its name there and <user> your account on it. As recorded on 2026-10-23:
- The machine. Windows 11 Pro 25H2, build 26200, x64; an AMD Ryzen AI
Max+ 395 with Radeon 8060S graphics. It has Rust's
x86_64-pc-windows-msvc, Bun 1.4.2, and the .NET SDKs 9.0.313 and 10.0.203. - The display and input. One display, a Samsung C49RG9x at 5120×1440, with no display scale or text scale set. It has no precision touchpad; D8 needs one added.
- Reaching it.
ssh -l "<user>" <host>; the shell is PowerShell.scprefuses a space inuser@host, so for an account name with one, name the user in a config file (User "<user>") and pass it with-F. An SSH session has no GitHub credentials, so send commits as a bundle:git bundle create, copy it, thengit fetch <bundle> main:refs/remotes/origin/mainandgit merge --ff-only. - SSH is not the desktop. An SSH session sees one 1024×768 stand-in
display (
WinDisc), so a window it opens is never shown. Headless drives work (bun scripts/agent.mjs windows --app shelf --storage <name> …); a real window must open in the signed-in console session, ashost/windows/reference/draw.mjsopens WinUI's, with a one-off scheduled task whose principal logs on interactively. WinUI launched from SSH exits with0xC000027Bbefore its first frame. - The WinUI reference pins Windows App SDK 1.8.260921001
(
host/windows/reference/Reference.csprojand itspackages.lock.json) and builds with the .NET 9 SDK:bun host/windows/reference/draw.mjsdraws each specimen ofspecimens.jsonin light and dark, at the display's 100%. - The drive's font. The agent pins DejaVu Sans on Windows as on Linux
(
scripts/agent-stdio.mjs), so a drive draws the fixture font, not Segoe UI Variable. SetEXACT_FONTto a family to draw another, or to a name that is not installed (EXACT_FONT=unpinned) to draw the system's own. In PowerShell,$env:EXACT_FONT = ""removes the variable, and the pin returns. - Confirm focus independently when reports disagree. Sky's focused-element
field can remain on Excord's composer after Tab moves to a button. Windows SDK
Inspect reports the button's UI Automation focus correctly, and Enter activates
it. Check the visible focus, action and an independent accessibility client
before changing the provider (
target/windows-chat-comparison/focus-inspect-qualification.json).
Pixels
- Downsample and crop. Compare at 1x or lower, cropped to the region the test is about. When higher resolution is needed, like for font rasterization and hairline borders, render them small or crop them.
- Capture once. Lay several cases out in one window, capture it once and crop each case so that we wait only once for the window server.
Measuring to optimize tests
Record uptime with every wall time on a shared machine. Warm the exact command
first, repeat the key samples, and distinguish compilation from test runtime.
Measure the gate after an actual one-line edit in a dependency as well as with
nothing changed. Adding a fast test whose dependencies take a minute to compile
still breaks the gate's budget.
- Measure with interactive QoS. A process started by a background service
(an agent's session, a launchd job without
ProcessType Interactive) can run at background QoS, where the kernel coalesces timers: a 10 ms sleep takes 60–90 ms and every wait inflates. Measure from a terminal or an interactive launchd job. - Measure with a pruned
target/. Cargo never deletes old object files fromtarget/debug/deps, and on macOS a test that touches the main bundle lists that directory, so hundreds of thousands of files there cost seconds of system time per test process. For URLSession startup stalls inCFBundleCopyBundleLocalizations, changing the working directory is insufficient: Foundation inspects the executable's directory. Copy the test executable into an empty temporary directory and run that copy with the same filters and deadlines to isolate directory accumulation. - Gatekeeper scans each newly built binary when it first runs. After a kernel edit, the scans added about 95–130% to the wall time of rebuilding and running ten packages' Rust tests. Adding the program that starts the test processes (a terminal, say) to System Settings → Privacy & Security → Developer Tools skips them.
- Find the bottleneck first. Time each binary and each test (
cargo testbinaries with-Zunstable-options --report-timeunderRUSTC_BOOTSTRAP=1; the XCTest per-case lines;bun test's per-test times). A binary's wall time is at least its slowest test, andcargo test's time is the sum over its binaries. - Sample a slow test before changing it.
sample <pid>shows whether the time is work, a lock, a timer or an idle wait. An idle wait is usually a missing notification, in the test or in the product. - Check the harness's own costs. Xcode's Thread Performance Checker writes a
symbolicated backtrace at every wait on a lower-priority thread, and its
diagnostics collection can wait out a 600 s timeout;
build.mjs --test --iosruns without both.