virtualjaguar-libretro/docs/agent/testing.md

Testing detail

Detail for AGENTS.md. Local-only RetroAchievements validation — no RA account/API/server. test/tools/test_rcheevos_e2e.sh downloads pinned RCHEEVOS_REF and verifies rc_libretro mapping (RC_CONSOLE_ATARI_JAGUAR) matches host RAM.

Shared test harness (test/harness/)

New tests should use test/harness/harness.h — shared lib eliminating dlopen/init/run boilerplate (see header's AGENT QUICK-START). Features:

• Common CLI: --json, --frames N, --bios, --option K=V, --quiet. --bios sets only the CART BIOS option; CD boot needs --option virtualjaguar_cd_boot_mode=bios (check the [BOOT] CD game, mode=BIOS log line). --input-bitmasks answers GET_INPUT_BITMASKS like RetroArch does (default harness takes the per-button input path).
• Scripted input: --press FRAME:BUTTON[:HOLD] (repeatable; buttons up down left right a b c pause option 0-6) — enough to navigate menus into gameplay headlessly. Programmatic: harness_press() / harness_input_cb.
• Log verbosity env: VJ_HARNESS_LOG_INFO=1 passes core INFO (CD trace dumps), VJ_HARNESS_LOG_DEBUG=1 adds DEBUG (per-call CD HLE trace).
• Automatic audio/video stats; harness_dlsym() probes internal state; --json machine output.
• Probe modules: dsp_probe.h (DSP registers, PC escape, LTXD ratio, RAM dumps), timing_probe.h (per-frame halflines, cycles, wall time, speed ratio).

Build: cc -O2 -Wall -std=c99 $(INCFLAGS) -o test_foo test_foo.c test/harness/harness.c [probe.c...] -ldl -lm. New probe: create test/harness/foo_probe.h + .c, resolve via harness_dlsym().

Key harnesses

• test/regression_test.sh — screenshot regression vs test/baselines/ via miniretro (built from source first run; MINIRETRO_BIN env to skip). Baselines not committed — no-baseline ROM reports NEW and prints the cp command; first run on fresh clone establishes them.
• test/tools/test_dsp_audio_diag.c — DSP audio diagnostic (make dsp-diag DSP_DIAG_ROM=path); detects PC escape, bank init failures, silent LTXD.
• test/tools/test_frame_timing.c — per-frame timing (make frame-timing FRAME_TIMING_ROM=path); halflines/cycles/VBlanks per frame, wall-clock speed ratio, anomaly detection. --csv, --json.
• test/test_audio_clipping.c — detects loud-broken audio (saturation density, run length, sustained loud RMS). Catches Skyhammer/IS2 "saturated square wave".
• test/test_audio_presence.c — counterpart: asserts audio present in a known-good envelope (RMS in [floor, ceiling], onset reached, no long zero runs). Required to catch the silencing-regression class where a "fix" drops RMS to zero (clipping passes, game silent). Iron Soldier 1 baseline: RMS ~1175 on develop.
• test/tools/hires_box_check.c — internal-res (hi-res Stage 1) inertness gate: every 2x frame must be exact 2×2 box replication of a 1x reference (frame_hash_ab CSV). Frames adjacent to a presented-dimension change excluded.
• test/tools/hires_state_digest.c — savestate FNV digests at frames 300/600/900; diff a 1x run vs virtualjaguar_internal_resolution=2x to prove the machine can't see the option.
• test/tools/test_memory_map.c — asserts SET_MEMORY_MAPS, SET_SUPPORT_ACHIEVEMENTS=true, descriptor layout.
• test/tools/blit_memo_verify + test/tools/blit_memo_sweep.sh — soundness gate for blit memoization (virtualjaguar_blit_memo, #411, docs/blit-memo.md). Verify mode never skips: runs every would-be skip live and compares write log + post-launch state. Run before adding virtualjaguar_blit_memo to any titledb row. Exit 3 = "thin" (too few checks) — a title that never repeats a blit stream is unverified, NOT clean; drive into gameplay with --press. No public ROM repeats a stream → no CI gate; private-corpus sweep is the gate.
• test/tools/test_blitter_compare — fast vs accurate blitter diff. Not in default make: cc -O2 -Wall -std=c99 -I./libretro-common/include -o test/tools/test_blitter_compare test/tools/test_blitter_compare.c -ldl. Usage: <core.so|.dylib> <rom> [frames] --load-state <file> [--frame-window F L] [--cmd-filter MASK VAL] [--verbose-dump] (note --load-state, not --savestate).
• test/test_dsp_mac40.c — DSP 40-bit MAC accumulator (dsp_acc40.h).
• test/tools/dram_scale_sweep.sh — gate for any GPU/68K timing-model change (#406). Sweeps VJ_DRAM_SCALE 1..16 with dram_timing + gpu_pipeline_timing on; asserts a per-window liveness floor on Doom's demo (default 30 flips per 300 fields; healthy 90-150), not merely "flips != 0". Exit 77 = private ROM absent (skip, never silent pass). Timing wedges relocate rather than disappear — a single-scale run proves nothing (#406 wedged at scales 4,8 before Verilator constants, at 3 after). Env: VJ_SCALES VJ_WINDOW VJ_MIN_FLIPS VJ_WARMUP_W.
• test/tools/test_doom_pace_gate + test/tools/test_cf_canary — timing-campaign S0 gates (docs/timing-campaign-plan.md, #408/#401), run by make TEST_EXPORTS=1 test, ROM-gated (exit 77 -> recorded SKIP). Tracking gates: expected values live in test/tools/timing_gate_baseline.txt = today's numbers beside the silicon target; a regression (moving AWAY from the target, liveness breach) fails, movement TOWARD it prints MOVED-TOWARD-TARGET (re-baseline in the same PR).
  • Doom: attract demo, fields 600..1500, prints DOOM_PACE_RESULT (mean fields/flip, gap histogram, per-300-field flips, GPU busy-time/flip). Silicon bar 4.0 fields/flip (#434, band [3.5,4.5]); develop = 1.993 (gaps 1:4 2:446 3:1). Pins the FAST blitter (census config); echoes options. Busy-time = core gpu_busy_* census (src/tom/gpu.c, TEST_EXPORTS only): cycles outside Doom's three spin loops (windows in the tool, none in src/), normalised by cycles/field (442,718 = GPU saturated). The only instrument that moves below the game's 2-field floor. Cross-checks: --option virtualjaguar_risc_idle_skip=disabled must give identical numbers; --no-arm leaves the census off (overhead timing). --measure-only --print-baseline re-measures.
  • Checkered Flag: loads states/Checkered Flag (1994).state.auto (rzip, extracted by the Makefile), no scripted input. Reads the frame-cap budget $43B0 and fields per game frame ($41AC is a 32-bit counter). The budget is a game-mode variable, not a load output (0 boot, 1 first menu, 2 race; reads 2 here), so the gate is fields/game-frame vs cap (budget+1): 3.000 today; FAIL > baseline+0.25 (all three timing options: 5.4; pipeline alone: 4.9) or if the budget rises above baseline; WARN on more overrun frames than baseline (plan 5.3: budget-1/overrun in heavy scenes needs reporter/BigPEmu corroboration).
• test/tools/bus_census_probe + test/tools/contention_model.py — timing-campaign S2 bus census (not in make test; build with make TEST_EXPORTS=1 BENCH_PROFILE=1 test/tools/bus_census_probe — BENCH_PROFILE only for blitter phrase counts, accurate blitter pinned by the tool). Core side: vjtrace_bus_* (src/core/vjtrace.c, VJ_TRACE only, release objects byte-identical): per-halfline 68K bus cycles/clocks, GPU external transactions, DSP 16-bit transactions, OP occupancy (model rule, UNGATED by dram_timing), RELEASE/BUSHI/DMAEN counts, split work/spin by --m68k-win / --dsp-win / gpu_busy windows. Writes per-halfline CSV (--out) + per-field flips (--frames-csv) + hot PCs with code bytes; --disasm-gpu/--disasm-dsp ADDR:N. The model script multiplies that traffic by the fixed latency function (plan S2, gpu-timing-spec.md §15). VJ_S2_INJECT=1|2|3 (low/central/high, VJ_TRACE builds) auto-arms the census and charges GPU external accesses that function so any tool (S0 gate, CF canary) sees the result — a one-shot cross-check, never a tuning knob. Doom 68K wait loops: 9C80:9C8A 4662:4670 (GPU done flag), E36A:E386 E30E:E336 (dspfinished), 12F74:12F80 4670:4688 ($279D4); DSP idle F1B092:F1B0A0.
• test/tools/corpus_ab_sweep.sh — A/B two cores across a ROM corpus, comparing the full framebuffer hash stream: corpus_ab_sweep.sh <base-core> <new-core> [rom-dir] (FRAMES, TMO, VJ_ROMS). Use it before merging anything that touches rendering. Two guards, both earned: every run is under a timeout (Chroma-Luma Color Pick hangs retro_run forever, #659 — the first sweep of this kind wedged on it for 2h23m emitting nothing, which is how #641's regression reached develop), and it compares the hash stream rather than the transition count (Super Burnout reads 143 transitions on both sides of #632 while rendering different pixels, so a count comparison calls a real change "no change"). Exit 1 on any SKIP: an unchecked ROM must not look like a clean sweep. AUDIO=1 also hashes each frame's audio (frame_hash_ab --audio-hash) and reports video/audio differing-frame counts -- use it for any DSP/audio change; ARGS="--bios --option ..." passes extra args to both arms. Only top-level ROMs in the dir are swept: for a parallel run, symlink chunks of the corpus into scratch dirs (never into the ROM tree).
• test/sram_test.sh — SRAM round-trip.
• test/tools/cd_boot_matrix.sh — per-title CD boot-stage matrix (HLE + BIOS) vs docs/cd-boot-matrix.md. Env: CD_MATRIX_FRAMES CD_MATRIX_TIMEOUT CD_MATRIX_MAX_RUNS CD_MATRIX_LOGDIR CD_MATRIX_OUT CD_MATRIX_ROMS_ROOT; chunked/resumable. Rows stamped with build id (<!-- build:<rev> -->); resume skips only same-build rows, re-runs others. (Resuming from an older build used to resurrect ancient rows as "fresh" — the phantom Battle Morph bios ? (pc_escape) final_pc=$8FBFB758 was such a stale row.)
• test/tools/cart_boot_matrix.sh (cart counterpart; classifier in test/tools/cart_classify.sh, probe cart_boot_probe.c): BIOS-mode rows are scored from the boot-ROM -> cart handoff (--post-handoff N, cap CART_MATRIX_BIOS_BOOT_FRAMES, default +700 frames), classes GAME_CODE/BIOS_REJECT (boot ROM halted on BRA.S *, never handed off: red "Jaguar" screen, an authentic reject)/? (bios_trap) (cart ran, PC fell back into $E00000)/? (...)/LOAD_FAIL. bios_ran=0 (headerless RAM-loaded .jag, e.g. Music Demo) has no boot phase: whole run scored. Crash-watchdog signatures are never handoff-relative (an early escape is a real bug, #853).
• test/tools/matrix_diff.py OLD.md NEW.md — never-backward gate (#749) between two copies of the same matrix (cart or CD): prints regressed / still-asymmetric / improved, exit 1 on any backward stage move or new crash-watchdog signature, 2 on parse error. BIOS_REJECT ranks equal to GAME_CODE (GAME_CODE -> BIOS_REJECT = lateral scoring correction; HLE GAME_CODE / BIOS BIOS_REJECT is not an asymmetry). Release-time, corpus machine only (docs/release-process.md). Self-test test/test_matrix_diff.sh runs in make test.
• CD trace ring: option virtualjaguar_cd_trace (or env VJ_CD_TRACE=1) records DSA_TX/DSA_RX/SEEK_START/SEEK_DONE/FIFO_FILL/FIFO_DRAIN/STOP/HLE_READ; dumped on cd_seek_wedge or on request.
• test/tools/cd_wedge_probe.c — catches intermittent CD lockups: detects a frozen framebuffer (--arm N --freeze-frames N, script repro with --press), dumps 68K regs, pcQueue traceback ring, CD counters + trace ring, RAM hexdump around stuck PC; exits 42 when caught. --ram-dump BASE writes full 2MB main-RAM + GPU-RAM at wedge; --snap FRAME (repeatable, --snap-prefix BASE) for clean-vs-wedged diffing. Found Hover Strike instant-CD_read code-stomp and B-skip stale-GPU-IRQ-latch corruption. Caveat: a static loading screen can false-positive short freeze windows — confirm with --freeze-frames 900.
• test/tools/cd_visual_verify.c — automated visual+audio verify for CD titles (make cd-visual CD_VISUAL_DISC=<image.cue>): per-second frame-motion timeline, non-black coverage, audio RMS, periodic PPM screenshots (sips -s format png to view). Replaces most "boot it and look" checks. Headless read-path caveat still applies for final sign-off.
• test/tools/netlink_discover_probe.c (--selftest) — validates LAN link-discovery beacon (src/jerry/jlink_discover.c). Lives inside the core-adjacent binary on purpose: macOS grants Local Network permission per binary, so a probe under python3's identity can pass while the shipped binary is denied (bit the #500 CI fix once). netlink_discover_pair.sh drives two instances, PID-spreading the discovery port (VJ_DISC_PORT, JLinkDiscPort()) so parallel CI runs don't collide on JLINK_DISC_PORT (42170). --selftest exits 77 when same-host broadcast isn't available in the sandbox (some macOS CI runners) — a skip, not a failure. netlink_rebuild_witness.c proves the SET_CORE_OPTIONS_V2 rebuild path (option-visibility re-push on link-mode change) executes.
• test/tools/netlink_sim.c — deterministic two-instance RetroArch-netpacket simulator (virtual clock, not part of make test). Measures the netlink reply wait without host-load noise: a core built from a THROWAWAY worktree patched by netlink_sim_clockpatch.py (clock/sleep + byte-event hooks, never applied to src/). Doom 2P deathmatch tics/s is the headline number: a tic is 12 one-way single-byte hops (6 per side) and must fit a whole number of video frames, so rates sit on 60/k plateaus (30/20/15/12 tics/s). Knobs: --delay-ms/--jitter-ms/--spike-pct/--spike-ms, --loss-pct with --rto-ms (TCP) or --udp-rto-ms (UDP+retransmit model), --emu-ms, --pump-us (cost of one poll_receive; results are sensitive to it, keep it identical across arms), --audio-block-ms (frontend cost AFTER the core's frame-end hook), --btrace (per-byte link events), --log (core log incl. the 5 s link stats line), --corrupt N:XX (flip one in-flight byte). Every arm of a comparison must use the same flags; --emu-ms 2 is optimistic for Doom at 2x, also run 6.

vjtrace flight recorder (dev-build-only)

Event ring + memory watch (src/core/vjtrace.h/.c): make TEST_EXPORTS=1 defines VJ_TRACE; plain make compiles every VJT_* to nothing and exports zero vjtrace symbols (costs nothing shipped). Every harness inherits tracing via trace_probe_attach() (test/harness/trace_probe.h; test/tools/vjtrace_smoke.c = generic driver). Flags: --trace-out FILE (binary ring at exit), --watch A[:LEN][:r|w|rw] (memory watch, max 16), --field-csv FILE (one row/frame: IRQ/GPU/OP/blitter counters, pad state, FB hash), --snap FRAME (VJSN snapshot), --mark FRAME:TAG (host marker event).

Analyzer CLIs (offline, no core/harness dep): trace_dump (print/filter a ring by --type/--who/--frame), trace_diff (structural diff of two rings, tolerant of frame offset), trace_memdiff (diff two --snap VJSN files by section), field_diff (row/col diff of two --field-csv). test/tools/vjtrace_selftest.sh is part of make test.

Ring sizing: default VJ_TRACE_RING = 1<<20 records; fills fast (OP_OBJECT/OP_LIST_START emit unconditionally, ~2,578 events/frame on test/roms/yarc.j64, ROM-dependent). A 1800-frame run needs ~4.6M records → set VJ_TRACE_RING=6000000. Both trace_dump/trace_diff print WARNING: ring wrapped to stderr when a dump lost events — don't treat a clean-looking dump as complete without checking.

Coverage caveats (authoritative in vjtrace.h WATCH COVERAGE block): GPU/DSP writes to their own local RAM ($F03000-$F03FFF / $F1B000-$F1CFFF) never route through watched dispatch — invisible to --watch; only accesses through JaguarRead/WriteByte/Word/Long and the 68K bus fast path are covered. A 32-bit 68K access = ONE record for main RAM/cart ROM but TWO (at addr and addr+2) for TOM/JERRY/CDROM/unknown → always round a watch's low bound down to a multiple of 4.

Audio / DSP work — required tests

Any change to src/jerry/dac.c, src/jerry/dsp.c, the HLE BIOS DSP/audio path in src/core/jaguar.c, or the DSP IRQ return-address logic MUST clear BOTH audio tests. A clipping check alone is insufficient: PR #170 (closed) took Iron Soldier 2 from 17% saturated to RMS=521 (silent) and the clipping test passed because silence has 0% saturation.

Required before declaring an audio change done:

1. make TEST_EXPORTS=1 test exits 0. Both test_audio_clipping and test_audio_presence are in the suite. Presence on IS1 uses develop's envelope (--rms-floor 200 --rms-ceiling 25000); if your change moves IS1's RMS outside that band you've changed audio behavior — verify intent.
2. Sanity-check that previously-clipping titles (Skyhammer, IS2) didn't go loud-broken → silent- broken. Both fixed as of 2026-08-05 by the MMULT secondary-bank fix; both assert clean. Skyhammer (World).j64: 0.000% saturated, window RMS 3079.8, first audio frame 171.
3. Verify in RetroArch on a real game. Headless can't tell music from structured noise at the right RMS, nor catch BIOS-mode crashes (PR #170's BIOS crash + HLE silence were invisible to the suite).

Do not relax thresholds in test_audio_clipping.c / test_audio_presence.c to pass. A real fix that legitimately quiets a known-broken title is a separate deliberate baseline update — call it out in the commit.

Runtime crash watchdog (src/core/crash_detect.c)

Runs once/frame from retro_run, logs to RetroArch log on these signatures. Option virtualjaguar_crash_detect = enabled (default) / disabled / verbose (adds a state heartbeat every 600 frames). Cost enabled: one indirect call + ~256-px hash/frame.

• gpu_pc_escape — GPU PC outside [$F03000,$F03FFF] ∪ [$0,$E3FFFF] (matches JaguarReadX decoding: main RAM mirrors bottom 8MB, cart ROM, boot ROM).
• dsp_pc_escape — DSP PC outside [$F1B000,$F1CFFF] ∪ [$0,$E3FFFF], tested on the raw PC (no 24-bit mask, unlike the GPU): DSPExec's own bail-out refuses a high-byte PC, so it executes nothing. The mask hid Music Demo's HLE runaway to $C1903000 (#853).
• inframe_hang — fired at blit dispatch (not per frame) when B_COUNT asks for >2^24 pixels, counted the way the accurate (default) engine runs it: outer 0 = 65536 lines (JTRM), inner 0 = one step (≤64 px — INNER.NET's zero/underflow detect, which BlitterMidsummer2 matches; Williams/Telegames carts write B_COUNT=0 at boot and run fine). Also fires with never_ends=1 for a blit whose inner counter never decrements: phrase mode below 8bpp, where INNER.NET's decrement is dstxp[0] and phrase-aligned X keeps it 0 — real hardware never finishes it either (#800). approx=1 marks a hang whose memory effect below is only a one-pass approximation (BlitterNeverEndsApprox). Policy: hang the emulated blitter, never the host -- the accurate engine runs the stuck step for one destination wrap period (exact when the step is idempotent on memory), then B_CMD reads busy and further starts are ignored until reset (test/tools/test_blitter_hung; saved in the v16 "BLH1" chunk). Known limit: on hardware a B_COUNT write while hung would un-stick it; not modelled. Absurd-size blits are still run synchronously and can keep retro_run from returning for hours; this line is logged before the blit runs. No real-ROM reproducer remains: Music Demo (ScatoLOGIC) --bios (#794, approx=1) was a misloaded raw binary (fixed by #818, now test_raw_binary_boot), and Native Demo (bin) HLE reached the blit only through a GPU runaway into data that the #611 NORMI fix reshuffled. The synthetic test covers both engines. Matrix runs killed at the wall-clock cap carry it in their evidence column.
  • Seen as garbage blits from a runaway 68K whose exception frames land on the blitter's $F0A238 mirror. Uploaded executables (raw BJL/.abs) that end in ILLEGAL used to do this in HLE; JaguarReset now parks vector 4 at $1000 (test/tools/test_upload_illegal_park, #800).
  • A raw binary parked at $1000 early is often a wrong inferred load base, not a bad title: file.c picks between overlapping candidate bases by entry-point consistency (JSR targets right after an RTS). test/tools/test_raw_binary_boot --expect-base pins it (#818).
• test/tools/cart_boot_matrix.sh runs the probe's default Fast blitter. Sweep the shipped default with CART_MATRIX_PROBE_ARGS="--option virtualjaguar_usefastblitter=disabled" plus an explicit CART_MATRIX_OUT (args are folded into the row cache id and default log dir).
• gpu_wedge / dsp_wedge — flagged running but zero opcodes for ≥180 / 600 frames (gpu_exec_opcode_count / dsp_exec_opcode_count). A stable sampled PC alone is NOT a wedge: deterministic slice budgets land the per-frame PC on the same instruction of a healthy wait/spin loop (Super Burnout spins ~446k GPU ops/frame at one sampled PC — the #378 false positive; regression test/tools/test_wedge_spin).
• video_stall — FB hash unchanged 300 frames while a processor runs.
• cd_seek_wedge — FIFO drain frozen 300 frames while a processor runs AND the drive still owes data: a seek outstanding, or a transfer still open (BUTCH low byte bits 0+1 = master + FIFO IRQ armed, AND I2CNTRL bit 2 = FIFO data enabled); dumps the CD trace ring. A game ends a transfer by clearing BUTCH bit 0 (Myst, Primal Rage, BrainDead 13 …) or I2CNTRL bit 2 (Philia); those idle drives used to fire it (#741). The line prints butch_int=/i2s_ctrl=. Pinned by test/test_crash_detect_cd_wedge (CI) + test/tools/cd_seek_wedge_regress.sh (private discs). test/test_cd_bios_boot FAILs a disc when the watchdog fired (CrashDetectCDSeekWedgeFires(), sticky, [CD-WEDGE] line) -- its PC checks alone pass on a dead transfer (#799).

Triaging "X crashes/hangs/black screen": the RetroArch log shows the signature — no save state or input recording needed. Add new signatures here when you find a recurring failure mode not already covered; don't sprinkle one-off LOG_ERR across subsystem files.

Acid suite CI gating

"Acid suite (linux x86_64)" can show conclusion: failure for two unrelated reasons — read the summary first:

• make -C test/acid test exits non-zero by design (returns FAIL count). Job uses set +e and gates on check-baseline.py. Real regression = Regressions: N (N>0) in acid-summary.txt.
• Conclusion failure with OK: no regressions in step output = the artifact-upload step timed out (Operation could not be completed within the specified time). Re-run the job; no code action.

Performance / profiling

make benchmark runs test/tools/test_benchmark headless vs a fixed ROM (default test/roms/yarc.j64, 600 frames), prints FPS / ms-per-frame. Use as a same-host commit-to-commit delta — don't compare across machines. Full guide: docs/profiling.md (Instruments / perf / flame graphs, SIMD A/B knob).

Measurement hazards: concurrent make swaps the dylib mid-sweep; sequential A/B measures host drift (an -O3 "+12.5%" flipped to -11.7% when interleaved) — always A/B/B/A, check uptime first.

Frame-time distribution (stutter, not average speed): test/tools/frame_cost_probe gives the per-frame ms CSV + p50/p95/p99/max and over-budget counts. An armed vjtrace gates the GPU/DSP idle skip OFF (idleSkipActive), so the default run measures skip-OFF whatever risc_idle_skip says; pass --no-arm to keep the skip live (event columns then read 0 — pair with an armed run of the same config and join on frame, the emulation is deterministic and the skip bit-exact). Feed the CSV to app/tests/pacing_trace.cpp to see what the app's pacer does with that cost trace (uneven image gaps, vsync release/restore cycles); virtualjaguar --perf-frames N with VJ_PERF_CSV=f.csv records the real thing (iter_ms,emu_frames,swap_sync,underruns,input_lat_ms,present_lat_ms; present = the swap that first shows a frame which polled the event; --perf-input injects a key every 30 iterations so the latency column has samples; --emu-thread on|off picks the loop, #967). --option k=v lists in an ssh command run under zsh must be built in bash (zsh does not word-split $OPTS; the option is then silently not applied and the harness prints no option line).

App emulation thread tests (#967)

The CTest label emu_thread (app/cmake/VJEmuThreadTests.cmake, plus the RA twins and ach_event_unit in VJAchTests.cmake) is what the ThreadSanitizer CI job runs. Locally the label runs on the normal build: ctest --test-dir app/build -C Release -L emu_thread. The CI job (tsan in .github/workflows/app.yml) configures app/build-tsan with -DVJ_APP_TSAN=ON -DVJ_APP_LTO=OFF (clang, RelWithDebInfo) and runs TSAN_OPTIONS="halt_on_error=1 second_deadlock_stack=1 suppressions=app/tests/tsan.supp" ctest --test-dir app/build-tsan -C RelWithDebInfo -L emu_thread --output-on-failure (Linux only; sudo sysctl vm.mmap_rnd_bits=28 first on a new runner kernel). Phase 2 added, on top of the phase 1 units and smoke_*_threaded:

• Units: core_events_unit (the core-event queue: order, bounded log lines, content switch, two threads) and ach_event_unit (AchEvent, an rc_client event copied across threads; in app/cmake/VJAchTests.cmake).
• RetroAchievements twins: every RA test is registered through vj_ra_test() in VJAchTests.cmake, which adds <name> (inline) and <name>_threaded (same run with --emu-thread on, label emu_thread; fixture names and the saved-cheat data dir get a _threaded suffix so the chains never share state). A test with PASS is a control: its twin fails only on owner violation or a TSan report. ra_reply_owner_threaded is threaded-only: login/game-load replies must run on the core owner, serialized with frames. It requires the mock login/load/session requests and rejects off-thread address reads. ra_ui_client_access and its threaded twin overlap owner-side hardcore/reset/evaluation commands with copied dashboard/profile/Discord snapshots, achievement rows, and image URLs (--ra-step FRAME:ui:stress). They also exercise unofficial/enabled toggles and password sign-in, and retain copied UI data across signout. Without the shared client lock, TSan reports a race between the hardcore getter and setter. UI code must never retain rcheevos pointers: use clientSnapshot() or the copied rows/URLs.
• Phase 3a (the Tools windows on the emulation thread): debug_views_unit (the request table, the buffers and the epoch; concurrent set/fill/read under TSan), the lane cases in emu_commands_unit / emu_thread_unit, debug_views_threaded (the REAL core on a threaded EmuThread, the session wired as App wires it: views equal a debugCall peek, a paused request fills with no frame run, the first read after a reload is the new ROM, inline == threaded for the Dram view, hardcore detaches), debug_on_threaded / gdb_enabled(_threaded) (nothing is switched off while threaded; the silent gdb host attaches at load) and gdb_hardcore_locked_* / gdb_hardcore_flip_* (hardcore locks the stub; a mid-game flip reloads). Each window's smoke test draws from view buffers only and pins zero core calls from the UI side (app/tests/debug_ui_fixture.h).
• Phase 3c (Neuro on the emulation thread): Neuro runs threaded, pinned by neuro_on_threaded (vj_neuro_test app ... on: a fake server, a 30-frame press, 30 frames of input played and emulation thread)), neuro_watch_threaded (a profile watch read through the DebugViews request, the cheat byte in the state line) and neuro_loopback_threaded (the loopback run with the input poll on its own thread; its frame-numbered press vector must equal the inline run's). neuro_core_e2e pins the replica against the real core inline, in lock-step and free-running.
• Phase 3b (the halt and the GDB stub on the emulation thread): debug_session_unit (now label emu_thread: the verdict serial, frameEnded/stepping_, detach and quit mid-halt, Detach and Attach ordered through the lane), emu_thread_unit testCallGivesUp / testAdvanceGivesUp / testServeDebugLane, emu_commands_unit testDebugLaneWait, debug_e2e_threaded (the REAL core on a threaded EmuThread, the rig wired as App: break / step / continue / detach / hardcore / quit mid-halt from a UI thread that never blocks over 50 ms; it opens a free TCP port for the stub), gdb_threaded_halt (Python gdb client: the threaded headless app halted at boot, S05, memory read, continue; SIGTERM mid-halt exits 0; needs Python3; gdb_inline_halt / gdb_inline_halt_hardcore are the inline runs), debug_on_threaded, gdb_enabled(_threaded), gdb_hardcore_locked_* and gdb_hardcore_flip_*. emu_thread_start_fallback no longer expects anything to be "back on" (nothing is switched off while threaded) and fails on "are back on" / "are off while".
• Phase 3d / Gate G1: no_threaded_limits (static, app/tests/no_threaded_limits.sh: the six checks -- removed gates, "off while threaded" texts, the old limit tests and their replacements, the GDB-stub lock keyed on hardcore only, the threaded_branch_allowlist.txt counts, tsan.supp size; raise a count only together with the new line's reason), emu_thread_default_unit, emu_thread_default_smoke_{unset,false,true,flag} (four because a headless run is inline, so the stored-choice line pins the rule; _flag is the threaded one; expectations derive from the configured channel), the Linux x86_64 (nightly channel) row in app.yml, and emu_thread_gate_check.sh (Studio only, VJ_WINDOWED_OK=1, --show-window tools, --perf-halt, ports from the OS). Runs: A every tool open + F9 halt (rows 130-410 parked, the window presents; Continue; a Neuro client answers presses), B a gdb halt at boot held 6 s (a parked run >= 200 rows, window presents, frames resume), C the plain path (>= 250 of 600 UI frames show an emulated frame). Each run wakes the display first; a still-asleep display skips the run ("display asleep, run skipped", exit 3), which is the environment, not a hang. studio.sh -D... args persist in app/build's CMakeCache: follow a nightly/release configure with -DVJ_BUILD_CHANNEL=dev.
• Audio scope: debug_blitter_unit also has the emu_thread label. Its concurrent producer/reader/clear test covers the sample ring under TSan; the UI open flag is atomic. Windowed TSan hardcore toggles are still required: they exercise the scope close callback and a freely running RA client while the UI pumps replies.
• Netplay: netplay_pair_threaded, netplay_pair_voicemodem_threaded, netplay_pair_mixed (one threaded copy linked to one single-thread copy) and netplay_no_partner_threaded (a copy quitting with no peer exits 1, not 3). They are RUN_SERIAL and real-time: a "no link within 30s" under heavy host load is load, so re-run them alone before reading anything into it.
• Busy spinner: smoke_busy_threaded runs with VJ_TEST_SLOW_CALL_MS=400 (the load call() sleeps that long on the emulation thread, max 5000) and expects [emu] busy: a call has taken over 250 ms.
• Failed start: emu_thread_start_fallback runs with VJ_TEST_EMU_START_FAIL=1, which takes the real failure branch of App::startEmuThread: the run falls back inline.

Headless framebuffer caveat

The miniretro harness in test/regression_test.sh doesn't expose the same composited framebuffer RetroArch reads. Symptom: jag_240p_test_suite main menu shows ~1k non-black pixels via miniretro vs tens of thousands via RetroArch. Treat as a headless read-path / presentation bug (OP+blitter output vs what the host reads), not a 240p timing or __muldi3 perf bug. Verify against RetroArch before treating a regression as real.