0 Fork 0

mirror of https://github.com/libretro/virtualjaguar-libretro.git synced 2026-11-07 12:47:58 -10:00

Files

T

Joe MattielloandClaude Opus 5.5 a314dd3aa4 ci: GL qualification: windowed boot + every shader preset on Linux, Windows x64/arm64 and macOS (#1155 ) (#1160 )

* feat(app): --shader PRESET|off for one run, and a [shader] result line per chain

`--shader PRESET.slangp` runs this one run through that preset and `--shader off`
through none; neither is written back to config.ini (like --theme).  A missing or
malformed preset is a usage error (exit 2) instead of a plain picture with a toast.

shaders.cpp now logs one stderr line when a chain is built or cannot be:
  [shader] <preset> built: GLSL 410 on OpenGL 4.1
  [shader] <preset> refused: ... (the GL_ARB_arrays_of_arrays limit, macOS GL 4.1)
  [shader] <preset> failed: ... (librashader's message)
so a CI run has a record of what the chain did (#1155 GL qualification).

Refs #1155

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci: GL qualification workflow: windowed boot + every shader preset on Linux, Windows, macOS

One reusable workflow, gl-qualification.yml, boots the desktop app windowed with a
real OpenGL context on hosted runners and renders every bundled .slangp through
librashader, keeping the screenshots.  Legs: Linux (ubuntu-24.04, Xvfb + llvmpipe),
Windows x64 (hosted, no vendor GL: the automatic bundled-Mesa relaunch of #1138/#1143
must fire), Windows arm64 (folds in windows-gl-test.yml, which is retired), macOS
arm64 (Apple OpenGL 4.1; the 15 arrays-of-arrays presets are allowlisted as refused).

Per leg: one build, cmake --install into stage/, vj_shader_tests gl run directly
(exit 77 = skipped = failure here), then app/test/gl-qualification/gl_qualify.py
launches --shader off and every preset with --screenshot and asserts exit 0, the
last [gl] renderer line, a non-black PNG, a [shader] built line and a picture that
differs from the plain one.  Artifacts upload pass or fail; the job summary lists
failures with the [gl]/[shader] log tail.  Goldens are designed
(app/test/golden/gl/<os>/<preset>.png, refresh_goldens input) but not gated.

Triggers: pull_request on the GL/shader/packaging paths, workflow_call (release.yml
requires it on tags and release/**), nightly schedule, workflow_dispatch.

Closes #1155

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci(gl-qualification): judge shaders_gl after the preset loop, fail fast, time the x64 fallback

First run (41565368188) findings: shaders_gl fails on llvmpipe and on Apple's software
renderer (crt-royale renders black in the 128x64 test picture) and the step stopped the
leg before any screenshot existed; the x64 automatic-fallback boot hit `timeout 120`
where arm64 took 3 s; the arm64 shaders_gl could not load the upstream prebuilt
librashader.dll (eager D3DX9_43.dll import, #1153/#1161).

- shaders_gl runs with continue-on-error and a "shaders_gl verdict" step fails the job
  after the preset loop, so the screenshots and logs upload either way.
- gl_qualify.py --fail-fast N (default 8): the loop stops after N consecutive failed
  presets and the summary says how many were not run.
- Windows fallback step: timeout 300 s, elapsed seconds logged (automatic-seconds.txt).

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci(gl-qualification): empty-array expansion safe for macOS bash 3.2 (set -u)

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci(gl-qualification): Windows leg records the boot, runs timing diagnostics, gates the loop

Second run (41567744247): the x64 automatic-fallback boot did not return within 300 s
although the screenshot was written (frame 20 reached); arm64 does the same in 2 s.

- the loader probe (context create/draw/destroy through the bundled Mesa) runs before
  the app boot, timed;
- the boot step records exit code, seconds and each side condition (automatic-seconds.txt)
  instead of aborting at the first failed check; a verdict step fails the leg later;
- when the boot did not come back, three bounded experiments (the Mesa-side child run
  directly, --emu-thread off, video.vsync=0 seeded) go to timing.txt;
- the preset loop runs only once the plain boot works (500 launches of a boot that does
  not return are hours of nothing); per-launch timeout 150 s.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci(gl-qualification): known-broken allowlist for presets librashader rejects everywhere

The second Linux run (41567744247) built and rendered 494 of the 500 bundled presets
in ~13 minutes of wall time; the six nedi bilateral-variant presets (all but variant2)
fail reflection with UnknownSemantics("play"): they chain vhs/shaders/vhs.slang, whose
`play` sampler is an LUT the presets never declare.  A preset bug, not a GL one, the
same on every leg, so it gets the same treatment as the macOS refused set: an allowlist
with the reason that may only shrink (a listed preset that builds is an error).

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci(gl-qualification): verdict steps run after an earlier failed step too (always())

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* test(app): shaders_gl renders the bundled presets at 640x480, not the 128x64 pixel-check picture

On llvmpipe (Linux) and Apple's software renderer (hosted macOS) crt-royale came out
black in shaders_gl: its mask passes scale by 0.0625 of the viewport and the preset says
"Safe for >= 341.333 horizontal triads at viewport size", so at 128x64 they are 8x4 px.
The app at 960x760 renders it (gl-qualification run 41567744247, 76% of pixels differ
from the plain picture).  The pixel checks keep their 128x64 (an exact 4x of the 32x16
source); the bundled loop and VJ_TEST_PRESETS render at a real-world 640x480.

Refs #1155

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci(gl-qualification): every Windows Mesa launch bounded; llvmpipe single-thread and softpipe probes

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci(gl-qualification): own librashader artifacts, macOS from source and sized down, release staging excludes qualification artifacts

Review (Codex P1/P2 on #1160) and run 41571727219:
- librashader.yml gains an `artifact_suffix` input; this workflow calls it with `-glq`, so a
  release run (app-package.yml + gl-qualification.yml) uploads no colliding artifact names.
- release.yml's two staging sweeps exclude artifacts/gl-qualification-* (screenshots, logs and
  the arm64 test zip are not release assets).
- macOS leg: the from-source aarch64 runtime the package ships, not upstream's prebuilt 0.12.0
  (crt-black_crush-koko.slangp fails to parse on the prebuilt, parses on the pinned source);
  3 launches at once in a 480x360 window (small-window.ini via gl_qualify.py --seed-config):
  Apple's software renderer managed 205 of 500 presets in 150 min at 960x760 and 2 jobs.
- refused-<os>.txt also covers presets the leg's GL cannot COMPILE (macOS: fsr and smaa+fsr need
  packHalf2x16, GLSL 4.20); a listed preset that builds is an error, the set only shrinks.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* test(app): gl_context_integration prints unbuffered stage markers (where does the x64 Mesa probe stop?)

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci(gl-qualification): tracked known failures; the x64 exit hang counted, not masked; loop before shaders_gl

app/test/gl-qualification/known-failures.txt lists one failure per line (<leg> <kind> <reference>);
the workflow reads it and turns exactly that failure into a recorded pass with a note. Nothing else
on the leg is forgiven, and removing the line makes the leg fail again. First entry: windows-x64
exit-hang (#1167), the app never exits on the bundled MSYS2 MINGW64 Mesa 26.2.3 llvmpipe
(SDL_DestroyWindow/SDL_Quit hang after "context destroyed"; LP_NUM_THREADS=0 and softpipe exit
at once, arm64 Mesa exits in 2 s).

gl_qualify.py: --known-exit-hang <ref> polls the launch, kills its process tree 5 s after the
screenshot is complete (IEND present) and counts the preset, noting the known failure in
summary.md and results.json; a launch without a complete screenshot still times out and fails.
The Windows boot step accepts exit 124 when the screenshot and the [gl] lines are there, the
loaders step accepts it on PASS, and shaders_gl passes without its final OK line only when
nothing failed or skipped and its last check (the broken shader) ran.

Every leg runs the preset loop before shaders_gl: on the macOS software renderer a SIGALRM-killed
GL process left the next window creation hanging. shaders_gl's bundled loop and VJ_TEST_PRESETS
render at 384x288 (crt-royale's mask passes need >= 341 px; 640x480 exceeded a 600 s alarm on
Apple's software renderer), the macOS alarm is 1200 s. gl_context_integration prints unbuffered
stage markers so a log shows where a probe stops. docs/agent/testing.md describes the lists.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci(gl-qualification): capture waits for core frames; unbuffered shaders_gl; macOS 640x480 + capture diagnostics

Run 41727418224 (3eef3305) read per leg:
- Windows x64: the preset loop passed under the tracked exit hang (494 built, 501 launches killed 5 s after
  their screenshot, #1167) but the shaders_gl allowance failed: the test's stdout verdict lines were lost when
  the hung process was killed (a pipe is fully buffered; only the stderr [shader] lines survived).
  vj_shader_tests writes stdout unbuffered from main() on.
- Windows arm64: 493 built, CuNNy-8x32-nvl-2x-rgb timed out at 150 s on the 128-bit llvmpipe. The Windows
  per-launch timeout is 240 s like the other legs.
- macOS: shaders_gl passes at 384x288 (10 min). In the seeded 480x360 window every capture, --shader off
  included, came back a uniform white picture at 1.9 s a launch; not the game (yarc is black at core frame
  1-2 and shows its picture from frame 10 on), not the theme. The leg seeds 640x480 (fits the hosted display,
  keeps crt-royale above 341 px) and a "Capture diagnostics" step launches `off` once per window
  configuration (default, 480x360, +frame 120, +emu frames, no viewports, no vsync, 480x362, 640x480,
  960x760) and prints each picture's statistics to the step summary, so the artifact names the variable.

App: --screenshot-emu-frames M (dev): the capture also waits until the core has shown M frames, with a line
and a capture anyway 600 UI frames later if it never does; gl_qualify.py --emu-frames passes it, every leg
uses 60. VideoOut::emuFrames() exposes the counter.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* ci(gl-qualification): even indentation in the x64 shaders_gl allowance (editorconfig-checker)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* ci(gl-qualification): macOS captures after 4 UI frames, 240 min budget; 480x360 white = ImGui viewports

Run 41760278887 (6afaae1a): Linux, Windows x64 (loop and shaders_gl under #1167) and Windows arm64
green. macOS: the capture diagnostics named the white picture's variable. With ImGui's extra OS
windows on (ui.viewports, the default) a 480x360 or 480x362 window captures uniform white; with them
off, or at 640x480 and up, it captures the game. The 640x480 leg then built every chain it reached but
ran out of time: 172 built, 57 timeouts at 240 s (49 of them had built the chain; Apple's software
renderer spends ~12 s a frame on the heavy ones), fail-fast after 250 presets, and the job hit its
150 min limit right after the upload.

- The macOS loop captures after 4 UI frames instead of 20; --emu-frames 60 already makes the capture
  the game's picture. The other legs keep 20.
- macOS timeout-minutes 240 (build ~4 min, shaders_gl 10-18 min), shaders_gl alarm 1800 s.
- The diagnostics step keeps four launches (default, 480x360 with and without viewports, 640x480).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* ci(gl-qualification): shard the macOS leg 3 ways, track presets over 120 s as macos-arm64 slow

Apple's software OpenGL 4.1 renderer on hosted Macs cannot render ~500 presets in one job
(runs 41760278887 and 41776934638 each ran out of time).  The macOS job becomes a 3-shard
matrix (gl_qualify.py --shard I/N: sorted preset index modulo N, every preset once, each
shard with its own `off` baseline, 2 launches at once), the GL probe, capture diagnostics and
shaders_gl in shard 0 only, and a macos-summary job joins the shards with --merge, failing
when a shard is missing or the shards did not run every preset exactly once.  Only the
joined table goes to the run summary.

known-failures.txt gains a `macos-arm64 slow <measured>@<run> <preset>` kind: 68 presets slow
in every run that measured them.  A listed preset is stopped at --slow-threshold (120 s) and
counted as a known failure, not a checked picture; an unlisted one over the limit fails and
the summary prints its line to paste; a listed one that renders in time is flagged for
removal.  A launch with no [gl] line is retried once and never counts as slow; POSIX kills
send SIGTERM first (--graceful-kill) and report a surviving process group.  Every run writes
times.tsv and a "Slowest launches" table.  macOS captures after SHOT_FRAME UI frames again.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* ci(gl-qualification): no exit allowance for app launches (#1172); test programs under #1179

#1172 ends the app's bundled-Mesa child without unloading Mesa, so the `windows-x64 exit-hang
#1167` entry and every app-launch allowance go: the automatic-fallback boot step and the preset
loop (--known-exit-hang) fail again if an app launch does not exit on its own.

The two test programs, vj_gl_context_integration and `vj_shader_tests gl`, load the bundled Mesa
in-process and still hang in its DllMain after SDL_Quit; their allowances stay, unchanged in what
they require (the probe's `PASS GL loader` line; shaders_gl with no FAIL/SKIP and its last check
run), now keyed to `windows-x64 test-exit-hang #1179`.

The macOS summary states plainly how many presets are listed `macos-arm64 slow` and therefore
not rendered, with their measured times (finished median and range, timeouts counted apart) and
how many this run stopped.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* ci(gl-qualification): macOS slow list from the first sharded run; ntsc-blastem refused on GL 4.1

Run 41891214865 (3 shards, all 500 presets, every one run once): Linux, Windows x64 and
Windows arm64 green; macOS 384 built, 57 listed presets stopped at 120 s, 36 failures.

- 31 presets took over 120 s with the chain built (12 timed out at 240 s): listed as
  `macos-arm64 slow` with their measured times.  7 listed presets that rendered in under 100 s
  are dropped; 4 that took 102-114 s stay listed (a listed preset that renders in time is still
  checked).  92 presets are listed.
- ntsc/ntsc-blastem fails to link on Apple's GL 4.1 ("Input of fragment shader
  'LIBRA_VARYING_2' differs in type/qualifiers"); it builds on GL 4.6: refused-macos.txt.
- 4 consecutive shard-0 launches (and their retries) never got a window for ~15 min, then the
  runner recovered, as in run 41760278887: on macOS a launch that times out with no [gl] line
  now leaves a `sample` stack of where the app was stuck.

The summary sentence now says a listed preset stopped at the limit is not rendered or checked,
and counts how many this run stopped and how many rendered in time.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* ci(gl-qualification): fail a slow macOS preset only if it never renders; reset Apple's GL compile service; no Windows exit allowances

Gate change (option b): an unlisted preset now FAILS only if it does not render within the
240 s timeout.  One that renders but takes over SLOW_LIMIT (120 s) is a ::notice:: with the
line to paste into known-failures.txt and a count in the summary ("Unlisted, rendered but over
120 s").  Hosted-Mac times swing ~2x between runs (sabr-hybrid-deposterize: 131 s, 91 s,
229 s), so the hard 120 s line failed a few new presets every run.  Listed presets keep the
120 s cap; the summary sentence states both rules.

Run 41957577821 (fd76515e): Linux, Windows x64, Windows arm64 green; macOS 18 failures:
- 10 rendered in 121-205 s: notices under the new gate, not listed.
- 2 built and timed out at 240 s: listed (vectorscale-exact-energy-single-aa, xbr-lv2-sharp).
  94 presets are listed; fd76515e's three lines (all rendered within 240 s) are not.
- 6 never got a window, again in the vectorscale/xbr stretch (third run).  The new stack
  samples all wait in CGLChoosePixelFormat -> cvms_connection_create on an XPC reply from
  CVMServer (com.apple.cvmsServ, Apple's GL compile service) that never came.  --reset-cvms
  (macOS leg) kills CVMCompiler/CVMServer with sudo -n before the no-[gl] retry; launchd starts
  a fresh one; the summary counts the resets and how many retries then got a window.

#1181 fixed #1179 (the GL test programs now exit without unloading the bundled Mesa): the
`windows-x64 test-exit-hang #1179` entry, the Known failures step and both test-program
allowances are gone, so any Windows process that does not exit on its own fails the leg.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* ci(gl-qualification): list the 2 macOS presets that timed out with a chain on run 42110580090

crt-yah/presets/pure-lcd (204.9 s on 41957577821, over 240 s now) and vectorscale (no window on
41957577821, the CVMServer stall; chain built but no picture in 240 s now) are added as
macos-arm64 slow, 96 listed.  The 15 unlisted presets that rendered in 121-205 s on that run are
notices under the current rule and are not listed.

Also corrects the SMOKE_PRESETS comment (Kimi review): PR runs render all presets; only
workflow_dispatch's preset_set=smoke selects the subset.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Fable 5.1 <[email protected]>

2026-11-05 20:04:02 -05:00

39 KiB

Raw Permalink Blame History

Testing detail

Detail for AGENTS.md. Local-only RetroAchievements validation — no RA account/API/server. test/tools/test_rcheevos_e2e.sh downloads pinned RCHEEVOS_REF and verifies rc_libretro mapping (RC_CONSOLE_ATARI_JAGUAR) matches host RAM.

Shared test harness (`test/harness/`)

New tests should use test/harness/harness.h — shared lib eliminating dlopen/init/run boilerplate (see header's AGENT QUICK-START). Features:

Common CLI: --json, --frames N, --bios, --option K=V, --quiet. --bios sets only the CART BIOS option; CD boot needs --option virtualjaguar_cd_boot_mode=bios (check the [BOOT] CD game, mode=BIOS log line). --input-bitmasks answers GET_INPUT_BITMASKS like RetroArch does (default harness takes the per-button input path).
Scripted input: --press FRAME:BUTTON[:HOLD] (repeatable; buttons up down left right a b c pause option 0-6) — enough to navigate menus into gameplay headlessly. Programmatic: harness_press() / harness_input_cb.
Log verbosity env: VJ_HARNESS_LOG_INFO=1 passes core INFO (CD trace dumps), VJ_HARNESS_LOG_DEBUG=1 adds DEBUG (per-call CD HLE trace).
Automatic audio/video stats; harness_dlsym() probes internal state; --json machine output.
Probe modules: dsp_probe.h (DSP registers, PC escape, LTXD ratio, RAM dumps), timing_probe.h (per-frame halflines, cycles, wall time, speed ratio).

Build: cc -O2 -Wall -std=c99 $(INCFLAGS) -o test_foo test_foo.c test/harness/harness.c [probe.c...] -ldl -lm. New probe: create test/harness/foo_probe.h + .c, resolve via harness_dlsym().

Key harnesses

test/regression_test.sh — screenshot regression vs test/baselines/ via miniretro (built from source first run; MINIRETRO_BIN env to skip). Baselines not committed — no-baseline ROM reports NEW and prints the cp command; first run on fresh clone establishes them.
test/tools/test_dsp_audio_diag.c — DSP audio diagnostic (make dsp-diag DSP_DIAG_ROM=path); detects PC escape, bank init failures, silent LTXD.
test/tools/test_frame_timing.c — per-frame timing (make frame-timing FRAME_TIMING_ROM=path); halflines/cycles/VBlanks per frame, wall-clock speed ratio, anomaly detection. --csv, --json.
test/test_audio_clipping.c — detects loud-broken audio (saturation density, run length, sustained loud RMS). Catches Skyhammer/IS2 "saturated square wave".
test/test_audio_presence.c — counterpart: asserts audio present in a known-good envelope (RMS in [floor, ceiling], onset reached, no long zero runs). Required to catch the silencing-regression class where a "fix" drops RMS to zero (clipping passes, game silent). Iron Soldier 1 baseline: RMS ~1175 on develop.
test/tools/hires_box_check.c — internal-res (hi-res Stage 1) inertness gate: every 2x frame must be exact 2×2 box replication of a 1x reference (frame_hash_ab CSV). Frames adjacent to a presented-dimension change excluded.
test/tools/hires_state_digest.c — savestate FNV digests at frames 300/600/900; diff a 1x run vs virtualjaguar_internal_resolution=2x to prove the machine can't see the option.
test/tools/test_memory_map.c — asserts SET_MEMORY_MAPS, SET_SUPPORT_ACHIEVEMENTS=true, descriptor layout.
test/tools/blit_memo_verify + test/tools/blit_memo_sweep.sh — soundness gate for blit memoization (virtualjaguar_blit_memo, #411, docs/blit-memo.md). Verify mode never skips: runs every would-be skip live and compares write log + post-launch state. Run before adding virtualjaguar_blit_memo to any titledb row. Exit 3 = "thin" (too few checks) — a title that never repeats a blit stream is unverified, NOT clean; drive into gameplay with --press. No public ROM repeats a stream → no CI gate; private-corpus sweep is the gate.
test/tools/test_blitter_compare — fast vs accurate blitter diff. Not in default make: cc -O2 -Wall -std=c99 -I./libretro-common/include -o test/tools/test_blitter_compare test/tools/test_blitter_compare.c -ldl. Usage: <core.so|.dylib> <rom> [frames] --load-state <file> [--frame-window F L] [--cmd-filter MASK VAL] [--verbose-dump] (note --load-state, not --savestate).
test/test_dsp_mac40.c — DSP 40-bit MAC accumulator (dsp_acc40.h).
test/tools/dram_scale_sweep.sh — gate for any GPU/68K timing-model change (#406). Sweeps VJ_DRAM_SCALE 1..16 with dram_timing + gpu_pipeline_timing on; asserts a per-window liveness floor on Doom's demo (default 30 flips per 300 fields; healthy 90-150), not merely "flips != 0". Exit 77 = private ROM absent (skip, never silent pass). Timing wedges relocate rather than disappear — a single-scale run proves nothing (#406 wedged at scales 4,8 before Verilator constants, at 3 after). Env: VJ_SCALES VJ_WINDOW VJ_MIN_FLIPS VJ_WARMUP_W.
test/tools/test_doom_pace_gate + test/tools/test_cf_canary — timing-campaign S0 gates (docs/timing-campaign-plan.md, #408/#401), run by make TEST_EXPORTS=1 test, ROM-gated (exit 77 -> recorded SKIP). Tracking gates: expected values live in test/tools/timing_gate_baseline.txt = today's numbers beside the silicon target; a regression (moving AWAY from the target, liveness breach) fails, movement TOWARD it prints MOVED-TOWARD-TARGET (re-baseline in the same PR).
- Doom: attract demo, fields 600..1500, prints DOOM_PACE_RESULT (mean fields/flip, gap histogram, per-300-field flips, GPU busy-time/flip). Silicon bar 4.0 fields/flip (#434, band [3.5,4.5]); develop = 1.993 (gaps 1:4 2:446 3:1). Pins the FAST blitter (census config); echoes options. Busy-time = core gpu_busy_* census (src/tom/gpu.c, TEST_EXPORTS only): cycles outside Doom's three spin loops (windows in the tool, none in src/), normalised by cycles/field (442,718 = GPU saturated). The only instrument that moves below the game's 2-field floor. Cross-checks: --option virtualjaguar_risc_idle_skip=disabled must give identical numbers; --no-arm leaves the census off (overhead timing). --measure-only --print-baseline re-measures.
- Checkered Flag: loads states/Checkered Flag (1994).state.auto (rzip, extracted by the Makefile), no scripted input. Reads the frame-cap budget $43B0 and fields per game frame ($41AC is a 32-bit counter). The budget is a game-mode variable, not a load output (0 boot, 1 first menu, 2 race; reads 2 here), so the gate is fields/game-frame vs cap (budget+1): 3.000 today; FAIL > baseline+0.25 (all three timing options: 5.4; pipeline alone: 4.9) or if the budget rises above baseline; WARN on more overrun frames than baseline (plan 5.3: budget-1/overrun in heavy scenes needs reporter/BigPEmu corroboration).
test/tools/bus_census_probe + test/tools/contention_model.py — timing-campaign S2 bus census (not in make test; build with make TEST_EXPORTS=1 BENCH_PROFILE=1 test/tools/bus_census_probe — BENCH_PROFILE only for blitter phrase counts, accurate blitter pinned by the tool). Core side: vjtrace_bus_* (src/core/vjtrace.c, VJ_TRACE only, release objects byte-identical): per-halfline 68K bus cycles/clocks, GPU external transactions, DSP 16-bit transactions, OP occupancy (model rule, UNGATED by dram_timing), RELEASE/BUSHI/DMAEN counts, split work/spin by --m68k-win / --dsp-win / gpu_busy windows. Writes per-halfline CSV (--out) + per-field flips (--frames-csv) + hot PCs with code bytes; --disasm-gpu/--disasm-dsp ADDR:N. The model script multiplies that traffic by the fixed latency function (plan S2, gpu-timing-spec.md §15). VJ_S2_INJECT=1|2|3 (low/central/high, VJ_TRACE builds) auto-arms the census and charges GPU external accesses that function so any tool (S0 gate, CF canary) sees the result — a one-shot cross-check, never a tuning knob. Doom 68K wait loops: 9C80:9C8A 4662:4670 (GPU done flag), E36A:E386 E30E:E336 (dspfinished), 12F74:12F80 4670:4688 ($279D4); DSP idle F1B092:F1B0A0.
test/tools/corpus_ab_sweep.sh — A/B two cores across a ROM corpus, comparing the full framebuffer hash stream: corpus_ab_sweep.sh <base-core> <new-core> [rom-dir] (FRAMES, TMO, VJ_ROMS). Use it before merging anything that touches rendering. Two guards, both earned: every run is under a timeout (Chroma-Luma Color Pick hangs retro_run forever, #659 — the first sweep of this kind wedged on it for 2h23m emitting nothing, which is how #641's regression reached develop), and it compares the hash stream rather than the transition count (Super Burnout reads 143 transitions on both sides of #632 while rendering different pixels, so a count comparison calls a real change "no change"). Exit 1 on any SKIP: an unchecked ROM must not look like a clean sweep. AUDIO=1 also hashes each frame's audio (frame_hash_ab --audio-hash) and reports video/audio differing-frame counts -- use it for any DSP/audio change; ARGS="--bios --option ..." passes extra args to both arms. Only top-level ROMs in the dir are swept: for a parallel run, symlink chunks of the corpus into scratch dirs (never into the ROM tree).
test/sram_test.sh — SRAM round-trip.
test/tools/cd_boot_matrix.sh — per-title CD boot-stage matrix (HLE + BIOS) vs docs/cd-boot-matrix.md. Env: CD_MATRIX_FRAMES CD_MATRIX_TIMEOUT CD_MATRIX_MAX_RUNS CD_MATRIX_LOGDIR CD_MATRIX_OUT CD_MATRIX_ROMS_ROOT; chunked/resumable. Rows stamped with build id (); resume skips only same-build rows, re-runs others. (Resuming from an older build used to resurrect ancient rows as "fresh" — the phantom Battle Morph bios ? (pc_escape) final_pc=$8FBFB758 was such a stale row.)
test/tools/cart_boot_matrix.sh (cart counterpart; classifier in test/tools/cart_classify.sh, probe cart_boot_probe.c): BIOS-mode rows are scored from the boot-ROM -> cart handoff (--post-handoff N, cap CART_MATRIX_BIOS_BOOT_FRAMES, default +700 frames), classes GAME_CODE/BIOS_REJECT (boot ROM halted on BRA.S *, never handed off: red "Jaguar" screen, an authentic reject)/? (bios_trap) (cart ran, PC fell back into $E00000)/? (...)/LOAD_FAIL. bios_ran=0 (headerless RAM-loaded .jag, e.g. Music Demo) has no boot phase: whole run scored. Every row ends in an  identity stamp (#976: the probe's jaguarMainROMCRC32, byte 0 of the image at $800000, the filedb row) that the site reads to say WHY the boot ROM rejected a file and to fold same-byte twins into one row; test/tools/cart_matrix_identify.py docs/cart-boot-matrix.md backfills or --checks the stamps from the files alone (no core), scripts/test_build_site_cart.py pins the labels. Crash-watchdog signatures are never handoff-relative (an early escape is a real bug, #853).
test/tools/matrix_diff.py OLD.md NEW.md — never-backward gate (#749) between two copies of the same matrix (cart or CD): prints regressed / still-asymmetric / improved, exit 1 on any backward stage move or new crash-watchdog signature, 2 on parse error. BIOS_REJECT ranks equal to GAME_CODE (GAME_CODE -> BIOS_REJECT = lateral scoring correction; HLE GAME_CODE / BIOS BIOS_REJECT is not an asymmetry). Release-time, corpus machine only (docs/release-process.md). Self-test test/test_matrix_diff.sh runs in make test.
CD trace ring: option virtualjaguar_cd_trace (or env VJ_CD_TRACE=1) records DSA_TX/DSA_RX/SEEK_START/SEEK_DONE/FIFO_FILL/FIFO_DRAIN/STOP/HLE_READ; dumped on cd_seek_wedge or on request.
test/tools/cd_wedge_probe.c — catches intermittent CD lockups: detects a frozen framebuffer (--arm N --freeze-frames N, script repro with --press), dumps 68K regs, pcQueue traceback ring, CD counters + trace ring, RAM hexdump around stuck PC; exits 42 when caught. --ram-dump BASE writes full 2MB main-RAM + GPU-RAM at wedge; --snap FRAME (repeatable, --snap-prefix BASE) for clean-vs-wedged diffing. Found Hover Strike instant-CD_read code-stomp and B-skip stale-GPU-IRQ-latch corruption. Caveat: a static loading screen can false-positive short freeze windows — confirm with --freeze-frames 900.
test/tools/cd_visual_verify.c — automated visual+audio verify for CD titles (make cd-visual CD_VISUAL_DISC=<image.cue>): per-second frame-motion timeline, non-black coverage, audio RMS, periodic PPM screenshots (sips -s format png to view). Replaces most "boot it and look" checks. Headless read-path caveat still applies for final sign-off.
test/tools/netlink_discover_probe.c (--selftest) — validates LAN link-discovery beacon (src/jerry/jlink_discover.c). Lives inside the core-adjacent binary on purpose: macOS grants Local Network permission per binary, so a probe under python3's identity can pass while the shipped binary is denied (bit the #500 CI fix once). netlink_discover_pair.sh drives two instances, PID-spreading the discovery port (VJ_DISC_PORT, JLinkDiscPort()) so parallel CI runs don't collide on JLINK_DISC_PORT (42170). --selftest exits 77 when same-host broadcast isn't available in the sandbox (some macOS CI runners) — a skip, not a failure. netlink_rebuild_witness.c proves the SET_CORE_OPTIONS_V2 rebuild path (option-visibility re-push on link-mode change) executes.
test/tools/netlink_sim.c — deterministic two-instance RetroArch-netpacket simulator (virtual clock, not part of make test). Measures the netlink reply wait without host-load noise: a core built from a THROWAWAY worktree patched by netlink_sim_clockpatch.py (clock/sleep + byte-event hooks, never applied to src/). Doom 2P deathmatch tics/s is the headline number: a tic is 12 one-way single-byte hops (6 per side) and must fit a whole number of video frames, so rates sit on 60/k plateaus (30/20/15/12 tics/s). Knobs: --delay-ms/--jitter-ms/--spike-pct/--spike-ms, --loss-pct with --rto-ms (TCP) or --udp-rto-ms (UDP+retransmit model), --emu-ms, --pump-us (cost of one poll_receive; results are sensitive to it, keep it identical across arms), --audio-block-ms (frontend cost AFTER the core's frame-end hook), --btrace (per-byte link events), --log (core log incl. the 5 s link stats line), --corrupt N:XX (flip one in-flight byte). Every arm of a comparison must use the same flags; --emu-ms 2 is optimistic for Doom at 2x, also run 6.

vjtrace flight recorder (dev-build-only)

Event ring + memory watch (src/core/vjtrace.h/.c): make TEST_EXPORTS=1 defines VJ_TRACE; plain make compiles every VJT_* to nothing and exports zero vjtrace symbols (costs nothing shipped). Every harness inherits tracing via trace_probe_attach() (test/harness/trace_probe.h; test/tools/vjtrace_smoke.c = generic driver). Flags: --trace-out FILE (binary ring at exit), --watch A[:LEN][:r|w|rw] (memory watch, max 16), --field-csv FILE (one row/frame: IRQ/GPU/OP/blitter counters, pad state, FB hash), --snap FRAME (VJSN snapshot), --mark FRAME:TAG (host marker event).

Analyzer CLIs (offline, no core/harness dep): trace_dump (print/filter a ring by --type/--who/--frame), trace_diff (structural diff of two rings, tolerant of frame offset), trace_memdiff (diff two --snap VJSN files by section), field_diff (row/col diff of two --field-csv). test/tools/vjtrace_selftest.sh is part of make test.

Ring sizing: default VJ_TRACE_RING = 1<<20 records; fills fast (OP_OBJECT/OP_LIST_START emit unconditionally, ~2,578 events/frame on test/roms/yarc.j64, ROM-dependent). A 1800-frame run needs ~4.6M records → set VJ_TRACE_RING=6000000. Both trace_dump/trace_diff print WARNING: ring wrapped to stderr when a dump lost events — don't treat a clean-looking dump as complete without checking.

Coverage caveats (authoritative in vjtrace.h WATCH COVERAGE block): GPU/DSP writes to their own local RAM ($F03000-$F03FFF / $F1B000-$F1CFFF) never route through watched dispatch — invisible to --watch; only accesses through JaguarRead/WriteByte/Word/Long and the 68K bus fast path are covered. A 32-bit 68K access = ONE record for main RAM/cart ROM but TWO (at addr and addr+2) for TOM/JERRY/CDROM/unknown → always round a watch's low bound down to a multiple of 4.

Audio / DSP work — required tests

Any change to src/jerry/dac.c, src/jerry/dsp.c, the HLE BIOS DSP/audio path in src/core/jaguar.c, or the DSP IRQ return-address logic MUST clear BOTH audio tests. A clipping check alone is insufficient: PR #170 (closed) took Iron Soldier 2 from 17% saturated to RMS=521 (silent) and the clipping test passed because silence has 0% saturation.

Required before declaring an audio change done:

make TEST_EXPORTS=1 test exits 0. Both test_audio_clipping and test_audio_presence are in the suite. Presence on IS1 uses develop's envelope (--rms-floor 200 --rms-ceiling 25000); if your change moves IS1's RMS outside that band you've changed audio behavior — verify intent.
Sanity-check that previously-clipping titles (Skyhammer, IS2) didn't go loud-broken → silent- broken. Both fixed as of 2026-08-05 by the MMULT secondary-bank fix; both assert clean. Skyhammer (World).j64: 0.000% saturated, window RMS 3079.8, first audio frame 171.
Verify in RetroArch on a real game. Headless can't tell music from structured noise at the right RMS, nor catch BIOS-mode crashes (PR #170's BIOS crash + HLE silence were invisible to the suite).

Do not relax thresholds in test_audio_clipping.c / test_audio_presence.c to pass. A real fix that legitimately quiets a known-broken title is a separate deliberate baseline update — call it out in the commit.

Runtime crash watchdog (`src/core/crash_detect.c`)

Runs once/frame from retro_run, logs to RetroArch log on these signatures. Option virtualjaguar_crash_detect = enabled (default) / disabled / verbose (adds a state heartbeat every 600 frames). Cost enabled: one indirect call + ~256-px hash/frame.

gpu_pc_escape — GPU PC outside [$F03000,$F03FFF] ∪ [$0,$E3FFFF] (matches JaguarReadX decoding: main RAM mirrors bottom 8MB, cart ROM, boot ROM).
dsp_pc_escape — DSP PC outside [$F1B000,$F1CFFF] ∪ [$0,$E3FFFF], tested on the raw PC (no 24-bit mask, unlike the GPU): DSPExec's own bail-out refuses a high-byte PC, so it executes nothing. The mask hid Music Demo's HLE runaway to $C1903000 (#853).
inframe_hang — fired at blit dispatch (not per frame) when B_COUNT asks for >2^24 pixels, counted the way the accurate (default) engine runs it: outer 0 = 65536 lines (JTRM), inner 0 = one step (≤64 px — INNER.NET's zero/underflow detect, which BlitterMidsummer2 matches; Williams/Telegames carts write B_COUNT=0 at boot and run fine). Also fires with never_ends=1 for a blit whose inner counter never decrements: phrase mode below 8bpp, where INNER.NET's decrement is dstxp[0] and phrase-aligned X keeps it 0 — real hardware never finishes it either (#800). approx=1 marks a hang whose memory effect below is only a one-pass approximation (BlitterNeverEndsApprox). Policy: hang the emulated blitter, never the host -- the accurate engine runs the stuck step for one destination wrap period (exact when the step is idempotent on memory), then B_CMD reads busy and further starts are ignored until reset (test/tools/test_blitter_hung; saved in the v16 "BLH1" chunk). Known limit: on hardware a B_COUNT write while hung would un-stick it; not modelled. Absurd-size blits are still run synchronously and can keep retro_run from returning for hours; this line is logged before the blit runs. No real-ROM reproducer remains: Music Demo (ScatoLOGIC) --bios (#794, approx=1) was a misloaded raw binary (fixed by #818, now test_raw_binary_boot), and Native Demo (bin) HLE reached the blit only through a GPU runaway into data that the #611 NORMI fix reshuffled. The synthetic test covers both engines. Matrix runs killed at the wall-clock cap carry it in their evidence column.
- Seen as garbage blits from a runaway 68K whose exception frames land on the blitter's $F0A238 mirror. Uploaded executables (raw BJL/.abs) that end in ILLEGAL used to do this in HLE; JaguarReset now parks vector 4 at $1000 (test/tools/test_upload_illegal_park, #800).
- A raw binary parked at $1000 early is often a wrong inferred load base, not a bad title: file.c picks between overlapping candidate bases by entry-point consistency (JSR targets right after an RTS). test/tools/test_raw_binary_boot --expect-base pins it (#818).
test/tools/cart_boot_matrix.sh runs the probe's default Fast blitter. Sweep the shipped default with CART_MATRIX_PROBE_ARGS="--option virtualjaguar_usefastblitter=disabled" plus an explicit CART_MATRIX_OUT (args are folded into the row cache id and default log dir).
gpu_wedge / dsp_wedge — flagged running but zero opcodes for ≥180 / 600 frames (gpu_exec_opcode_count / dsp_exec_opcode_count). A stable sampled PC alone is NOT a wedge: deterministic slice budgets land the per-frame PC on the same instruction of a healthy wait/spin loop (Super Burnout spins ~446k GPU ops/frame at one sampled PC — the #378 false positive; regression test/tools/test_wedge_spin).
video_stall — FB hash unchanged 300 frames while a processor runs.
cd_seek_wedge — FIFO drain frozen 300 frames while a processor runs AND the drive still owes data: a seek outstanding, or a transfer still open (BUTCH low byte bits 0+1 = master + FIFO IRQ armed, AND I2CNTRL bit 2 = FIFO data enabled); dumps the CD trace ring. A game ends a transfer by clearing BUTCH bit 0 (Myst, Primal Rage, BrainDead 13 …) or I2CNTRL bit 2 (Philia); those idle drives used to fire it (#741). The line prints butch_int=/i2s_ctrl=. Pinned by test/test_crash_detect_cd_wedge (CI) + test/tools/cd_seek_wedge_regress.sh (private discs). test/test_cd_bios_boot FAILs a disc when the watchdog fired (CrashDetectCDSeekWedgeFires(), sticky, [CD-WEDGE] line) -- its PC checks alone pass on a dead transfer (#799).

Triaging "X crashes/hangs/black screen": the RetroArch log shows the signature — no save state or input recording needed. Add new signatures here when you find a recurring failure mode not already covered; don't sprinkle one-off LOG_ERR across subsystem files.

Acid suite CI gating

"Acid suite (linux x86_64)" can show conclusion: failure for two unrelated reasons — read the summary first:

make -C test/acid test exits non-zero by design (returns FAIL count). Job uses set +e and gates on check-baseline.py. Real regression = Regressions: N (N>0) in acid-summary.txt.
Conclusion failure with OK: no regressions in step output = the artifact-upload step timed out (Operation could not be completed within the specified time). Re-run the job; no code action.

Performance / profiling

make benchmark runs test/tools/test_benchmark headless vs a fixed ROM (default test/roms/yarc.j64, 600 frames), prints FPS / ms-per-frame. Use as a same-host commit-to-commit delta — don't compare across machines. Full guide: docs/profiling.md (Instruments / perf / flame graphs, SIMD A/B knob).

Measurement hazards: concurrent make swaps the dylib mid-sweep; sequential A/B measures host drift (an -O3 "+12.5%" flipped to -11.7% when interleaved) — always A/B/B/A, check uptime first.

Frame-time distribution (stutter, not average speed): test/tools/frame_cost_probe gives the per-frame ms CSV + p50/p95/p99/max and over-budget counts. An armed vjtrace gates the GPU/DSP idle skip OFF (idleSkipActive), so the default run measures skip-OFF whatever risc_idle_skip says; pass --no-arm to keep the skip live (event columns then read 0 — pair with an armed run of the same config and join on frame, the emulation is deterministic and the skip bit-exact). Feed the CSV to app/tests/pacing_trace.cpp to see what the app's pacer does with that cost trace (uneven image gaps, vsync release/restore cycles); virtualjaguar --perf-frames N with VJ_PERF_CSV=f.csv records the real thing (iter_ms,emu_frames,swap_sync,underruns,input_lat_ms,present_lat_ms; present = the swap that first shows a frame which polled the event; --perf-input injects a key every 30 iterations so the latency column has samples; --emu-thread on|off picks the loop, #967). --option k=v lists in an ssh command run under zsh must be built in bash (zsh does not word-split $OPTS; the option is then silently not applied and the harness prints no option line).

App emulation thread tests (#967)

The CTest label emu_thread (app/cmake/VJEmuThreadTests.cmake, plus the RA twins and ach_event_unit in VJAchTests.cmake) is what the ThreadSanitizer CI job runs. Locally the label runs on the normal build: ctest --test-dir app/build -C Release -L emu_thread. The CI job (tsan in .github/workflows/app.yml) configures app/build-tsan with -DVJ_APP_TSAN=ON -DVJ_APP_LTO=OFF (clang, RelWithDebInfo) and runs TSAN_OPTIONS="halt_on_error=1 second_deadlock_stack=1 suppressions=app/tests/tsan.supp" ctest --test-dir app/build-tsan -C RelWithDebInfo -L emu_thread --output-on-failure (Linux only; sudo sysctl vm.mmap_rnd_bits=28 first on a new runner kernel). Phase 2 added, on top of the phase 1 units and smoke_*_threaded:

Units: core_events_unit (the core-event queue: order, bounded log lines, content switch, two threads) and ach_event_unit (AchEvent, an rc_client event copied across threads; in app/cmake/VJAchTests.cmake).
RetroAchievements twins: every RA test is registered through vj_ra_test() in VJAchTests.cmake, which adds <name> (inline) and <name>_threaded (same run with --emu-thread on, label emu_thread; fixture names and the saved-cheat data dir get a _threaded suffix so the chains never share state). A test with PASS is a control: its twin fails only on owner violation or a TSan report. ra_reply_owner_threaded is threaded-only: login/game-load replies must run on the core owner, serialized with frames. It requires the mock login/load/session requests and rejects off-thread address reads. ra_ui_client_access and its threaded twin overlap owner-side hardcore/reset/evaluation commands with copied dashboard/profile/Discord snapshots, achievement rows, and image URLs (--ra-step FRAME:ui:stress). They also exercise unofficial/enabled toggles and password sign-in, and retain copied UI data across signout. Without the shared client lock, TSan reports a race between the hardcore getter and setter. UI code must never retain rcheevos pointers: use clientSnapshot() or the copied rows/URLs.
Phase 3a (the Tools windows on the emulation thread): debug_views_unit (the request table, the buffers and the epoch; concurrent set/fill/read under TSan), the lane cases in emu_commands_unit / emu_thread_unit, debug_views_threaded (the REAL core on a threaded EmuThread, the session wired as App wires it: views equal a debugCall peek, a paused request fills with no frame run, the first read after a reload is the new ROM, inline == threaded for the Dram view, hardcore detaches), debug_on_threaded / gdb_enabled(_threaded) (nothing is switched off while threaded; the silent gdb host attaches at load) and gdb_hardcore_locked_* / gdb_hardcore_flip_* (hardcore locks the stub; a mid-game flip reloads). Each window's smoke test draws from view buffers only and pins zero core calls from the UI side (app/tests/debug_ui_fixture.h).
Phase 3c (Neuro on the emulation thread): Neuro runs threaded, pinned by neuro_on_threaded (vj_neuro_test app ... on: a fake server, a 30-frame press, 30 frames of input played and emulation thread)), neuro_watch_threaded (a profile watch read through the DebugViews request, the cheat byte in the state line) and neuro_loopback_threaded (the loopback run with the input poll on its own thread; its frame-numbered press vector must equal the inline run's). neuro_core_e2e pins the replica against the real core inline, in lock-step and free-running.
Phase 3b (the halt and the GDB stub on the emulation thread): debug_session_unit (now label emu_thread: the verdict serial, frameEnded/stepping_, detach and quit mid-halt, Detach and Attach ordered through the lane), emu_thread_unit testCallGivesUp / testAdvanceGivesUp / testServeDebugLane, emu_commands_unit testDebugLaneWait, debug_e2e_threaded (the REAL core on a threaded EmuThread, the rig wired as App: break / step / continue / detach / hardcore / quit mid-halt from a UI thread that never blocks over 50 ms; it opens a free TCP port for the stub), gdb_threaded_halt (Python gdb client: the threaded headless app halted at boot, S05, memory read, continue; SIGTERM mid-halt exits 0; needs Python3; gdb_inline_halt / gdb_inline_halt_hardcore are the inline runs), debug_on_threaded, gdb_enabled(_threaded), gdb_hardcore_locked_* and gdb_hardcore_flip_*. emu_thread_start_fallback no longer expects anything to be "back on" (nothing is switched off while threaded) and fails on "are back on" / "are off while".
Phase 3d / Gate G1: no_threaded_limits (static, app/tests/no_threaded_limits.sh: the six checks -- removed gates, "off while threaded" texts, the old limit tests and their replacements, the GDB-stub lock keyed on hardcore only, the threaded_branch_allowlist.txt counts, tsan.supp size; raise a count only together with the new line's reason), emu_thread_default_unit, emu_thread_default_smoke_{unset,false,true,flag} (four because a headless run is inline, so the stored-choice line pins the rule; _flag is the threaded one; expectations derive from the configured channel), the Linux x86_64 (nightly channel, Neuro off) row in app.yml, and emu_thread_gate_check.sh (Studio only, VJ_WINDOWED_OK=1, --show-window tools, --perf-halt, ports from the OS). Runs: A every tool open + F9 halt (rows 130-410 parked, the window presents; Continue; a Neuro client answers presses), B a gdb halt at boot held 6 s (a parked run >= 200 rows, window presents, frames resume), C the plain path (>= 250 of 600 UI frames show an emulated frame). Each run wakes the display first; a still-asleep display skips the run ("display asleep, run skipped", exit 3), which is the environment, not a hang. studio.sh -D... args persist in app/build's CMakeCache: follow a nightly/release configure with -DVJ_BUILD_CHANNEL=dev.
Audio scope: debug_blitter_unit also has the emu_thread label. Its concurrent producer/reader/clear test covers the sample ring under TSan; the UI open flag is atomic. Windowed TSan hardcore toggles are still required: they exercise the scope close callback and a freely running RA client while the UI pumps replies.
Netplay: netplay_pair_threaded, netplay_pair_voicemodem_threaded, netplay_pair_mixed (one threaded copy linked to one single-thread copy) and netplay_no_partner_threaded (a copy quitting with no peer exits 1, not 3). They are RUN_SERIAL and real-time: a "no link within 30s" under heavy host load is load, so re-run them alone before reading anything into it.
Busy spinner: smoke_busy_threaded runs with VJ_TEST_SLOW_CALL_MS=400 (the load call() sleeps that long on the emulation thread, max 5000) and expects [emu] busy: a call has taken over 250 ms.
Failed start: emu_thread_start_fallback runs with VJ_TEST_EMU_START_FAIL=1, which takes the real failure branch of App::startEmuThread: the run falls back inline.

GL qualification (desktop app, windowed, every shader preset)

.github/workflows/gl-qualification.yml (#1155) boots app/ windowed with a real OpenGL context on hosted runners, once per bundled .slangp, and keeps the screenshots. Legs: Linux (ubuntu-24.04, Xvfb + Mesa llvmpipe), Windows x64 (hosted, no vendor GL: the app must relaunch beside the bundled MSYS2 Mesa, #1138/#1143), Windows arm64 (same, plus the portable test build Store testers used), macOS arm64 (Apple OpenGL 4.1). Runs on PRs touching the GL/shader path, nightly, on dispatch, and release.yml requires it on tags and release/**.

Per leg: one build, cmake --install into stage/, the preset loop (below), then vj_shader_tests gl run directly after it (exit 77 = skipped = failure here; everywhere else it is -E shaders_gl; the step is continue-on-error and a "shaders_gl verdict" step fails the job, so the screenshots exist even when the test is red, and a test that has to be killed cannot wedge the window server before the loop). The loop: app/test/gl-qualification/gl_qualify.py launches stage/bin/virtualjaguar --shader <preset> --screenshot <png> --screenshot-frame 20 --screenshot-emu-frames 60 test/roms/yarc.j64 for --shader off and every preset (the capture waits for 20 UI frames and 60 core frames, so it is the game's picture however fast or crowded the host is). Per (leg, preset) it asserts: exit 0 before the timeout; the [gl] version=...; renderer= line matches the leg (llvmpipe / llvmpipe|D3D12 after automatically retrying with bundled Mesa OpenGL / Apple); the PNG decodes and is not black; a [shader] <preset> built line (shaders.cpp); the picture differs from the plain capture unless the preset is in identical-to-plain.txt. After 8 consecutive failed presets the loop stops (--fail-fast; the summary says how many were not run), so a leg with no working runtime fails in minutes. A [shader] ... refused line (macOS OpenGL 4.1 has no GL_ARB_arrays_of_arrays), or a ... failed line for a preset the leg's GL cannot compile (packHalf2x16 on macOS), passes only for presets in refused-<os>.txt, so that set can shrink but never grow. The macOS leg runs in a 640x480 window (small-window.ini, --seed-config): the hosted display is too small for 960x760 (the window came back 960x642, ~45 s a launch), and a seeded 480x360 window captures uniform white while ImGui's extra OS windows are on (the default; off, or at 640x480 and up, it captures the game), which its "Capture diagnostics" step keeps showing.

The macOS leg is sharded: Apple's software renderer (hosted runners have no GPU) takes minutes over a heavy chain's first frame, so one runner cannot render ~500 presets in a job. Three macos jobs each take the presets whose sorted index modulo 3 is their shard (--shard I/3, each with its own off baseline, 2 launches at once); the GL probe, the capture diagnostics and shaders_gl run in shard 0 only; macos-summary joins them (gl_qualify.py --merge ... --expect-shards 3), failing if a shard is missing or the shards did not run every preset exactly once, and uploads the one gl-qualification-macos-arm64 summary (screenshots and logs stay in the gl-qualification-macos-arm64-shard-<n> artifacts). A preset that alone takes over SLOW_LIMIT (120 s) there is a macos-arm64 slow <measured>@<run> <preset> line in known-failures.txt: it is stopped at the limit and listed as a known failure ("chain built, not rendered" / "no chain"), not a checked picture. An unlisted preset fails only if it does not render within the 240 s timeout; one that renders but takes over 120 s is a ::notice:: with its line to paste and a count in the summary (hosted-Mac times swing ~2x between runs, so a hard 120 s gate failed new presets every run); a listed one that renders in time is checked like any other (the summary counts both; drop entries that stay well under the limit). A launch with no [gl] line is retried once and never counts as slow. On macOS such a timeout leaves a stack sample (logs/<preset>.sample.txt, sample(1)), and the retry first restarts Apple's GL compile service (--reset-cvms: sudo -n killall -9 CVMCompiler CVMServer; launchd starts a fresh one). Three runs (41760278887, 41891214865, 41957577821) had every launch stall for ~15 minutes in the vectorscale/xbr stretch of the list; the samples all end in CGLChoosePixelFormat -> cvms_connection_create, an XPC call to CVMServer that never answered. Each folder has times.tsv (seconds per preset) and the summary a "Slowest launches" table. Goldens are a second pass: app/test/golden/gl/<os>/<preset>.png is compared and reported (not gated) when present; the refresh_goldens dispatch input collects candidates into the artifact.

Reading a failure. The job summary lists failed presets with the reason and the last [gl] / [shader] lines. Download gl-qualification-<leg> (always uploaded): shots/<preset>.png (/ in the preset path becomes __), logs/<preset>.log (the full stderr of that launch), results.json, summary.md, plus the leg's probes (glxinfo.txt, gl-context-integration.txt, shaders_gl.txt; Windows adds native-driver*.txt, automatic-stderr.txt, legacy-loader.txt).

no \[gl] version=...`line → the app never got a context: read the log head (SDL error) and the probe files; on Windows checknative-driver*.txt` and whether the retry line is there.
renderer ... does not match → the leg landed on the wrong driver (e.g. Windows picked WARP instead of llvmpipe/d3d12; Linux lost LIBGL_ALWAYS_SOFTWARE).
shader failed: ... → librashader's compile/link message; the preset or the runtime pin broke. One preset = that preset; all presets = the chain/loader (look at shaders_gl.txt too). Presets the bundled librashader rejects on every GL live in known-broken.txt with the reason (they must keep failing: one that starts building is reported so the list shrinks).
refused and not in the allowlist → a preset newly needs a GL feature the leg lacks: fix it or, if genuinely platform-limited, add it to refused-<os>.txt with the reason.
no \[shader] ... built` line→ the chain never ran: the game had not drawn by frame 20 or the preset was not enabled; compareshots/off.pngand the log's[emu]` lines.
identical to the --shader off capture → the preset changed nothing; either the chain fell back to the plain presenter (check for a [shader] line) or it is a true no-op at this scale (list it).
screenshot is black → the window drew nothing (lost context, hidden window, swap failure).
known-failures.txt holds tracked failures, one line each; nothing else on that leg is forgiven. macos-arm64 slow: presets Apple's software renderer cannot render within 120 s (above). The Windows exit hangs are fixed and no longer listed: the app's (#1167, by #1172) and the two GL test programs' (#1179, by #1181); a Windows process that does not exit on its own fails the leg.
Windows "Automatic Mesa fallback verdict" red → the stock-Windows path (#1138) regressed or the boot did not come back: automatic-seconds.txt has the exit code, the seconds and which of the checks failed (the retry line, the renderer= line, config.ini in the parent's data dir, no saves/instance-2); timing.txt then has three bounded experiments (the Mesa-side child run directly, the same with --emu-thread off, the same with video.vsync=0) that split "the parent never returns" from "the window never finishes a swap". The preset loop is skipped on that leg until the plain boot works. If native-driver-gl33.txt ends in exit 0 the runner image has grown a real GL and the fallback is not exercised there (skipped with a warning); arm64 is then the proof.

Headless framebuffer caveat

The miniretro harness in test/regression_test.sh doesn't expose the same composited framebuffer RetroArch reads. Symptom: jag_240p_test_suite main menu shows ~1k non-black pixels via miniretro vs tens of thousands via RetroArch. Treat as a headless read-path / presentation bug (OP+blitter output vs what the host reads), not a 240p timing or __muldi3 perf bug. Verify against RetroArch before treating a regression as real.

39 KiB Raw Permalink Blame History Unescape Escape