ci(gl-qualification): tracked known failures; the x64 exit hang counted, not masked; loop before shaders_gl

app/test/gl-qualification/known-failures.txt lists one failure per line (<leg> <kind> <reference>);
the workflow reads it and turns exactly that failure into a recorded pass with a note. Nothing else
on the leg is forgiven, and removing the line makes the leg fail again. First entry: windows-x64
exit-hang (#1167), the app never exits on the bundled MSYS2 MINGW64 Mesa 26.2.3 llvmpipe
(SDL_DestroyWindow/SDL_Quit hang after "context destroyed"; LP_NUM_THREADS=0 and softpipe exit
at once, arm64 Mesa exits in 2 s).

gl_qualify.py: --known-exit-hang <ref> polls the launch, kills its process tree 5 s after the
screenshot is complete (IEND present) and counts the preset, noting the known failure in
summary.md and results.json; a launch without a complete screenshot still times out and fails.
The Windows boot step accepts exit 124 when the screenshot and the [gl] lines are there, the
loaders step accepts it on PASS, and shaders_gl passes without its final OK line only when
nothing failed or skipped and its last check (the broken shader) ran.

Every leg runs the preset loop before shaders_gl: on the macOS software renderer a SIGALRM-killed
GL process left the next window creation hanging. shaders_gl's bundled loop and VJ_TEST_PRESETS
render at 384x288 (crt-royale's mask passes need >= 341 px; 640x480 exceeded a 600 s alarm on
Apple's software renderer), the macOS alarm is 1200 s. gl_context_integration prints unbuffered
stage markers so a log shows where a probe stops. docs/agent/testing.md describes the lists.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Joseph MattielloandClaude Fable 5.1 committed 2026-11-03 17:03:29 -05:00
1 parent 754171f71f
commit 3eef3305e2
6 files changed
+164 -58

No files matched your search

+68 -36
View File
@@ -145,17 +145,6 @@ jobs:
xvfb-run -a -s '-screen 0 1600x1200x24 +extension GLX' glxinfo -B | tee glq/glxinfo.txt
xvfb-run -a -s '-screen 0 1600x1200x24 +extension GLX' build/vj_gl_context_integration | tee glq/gl-context-integration.txt
# The headless shader test on a real context: a skip (77) is a failure on this leg. Recorded
# here and judged after the preset loop, so a red test still leaves the screenshots as evidence.
- name: shaders_gl (skip = failure)
id: shaders_gl
continue-on-error: true
run: |
set -eu
export LIBGL_ALWAYS_SOFTWARE=1 SDL_VIDEO_DRIVER=x11 VJ_LIBRASHADER="$PWD/stage/bin/librashader.so"
xvfb-run -a -s '-screen 0 1600x1200x24 +extension GLX' build/vj_shader_tests gl 2>&1 | tee glq/shaders_gl.txt
test "${PIPESTATUS[0]}" -eq 0
- name: Render every bundled preset
run: |
set -eu
@@ -168,6 +157,18 @@ jobs:
--refused-allowlist "$LISTS/refused-linux.txt" --identical-ok "$LISTS/identical-to-plain.txt" --known-broken "$LISTS/known-broken.txt" \
--goldens app/test/golden/gl/linux --preset-set "$PRESET_SET" ${refresh[@]+"${refresh[@]}"}
# The headless shader test on a real context: a skip (77) is a failure on this leg. Recorded
# here and judged after the preset loop, so a red test still leaves the screenshots as evidence.
- name: shaders_gl (skip = failure)
id: shaders_gl
if: always()
continue-on-error: true
run: |
set -eu
export LIBGL_ALWAYS_SOFTWARE=1 SDL_VIDEO_DRIVER=x11 VJ_LIBRASHADER="$PWD/stage/bin/librashader.so"
xvfb-run -a -s '-screen 0 1600x1200x24 +extension GLX' build/vj_shader_tests gl 2>&1 | tee glq/shaders_gl.txt
test "${PIPESTATUS[0]}" -eq 0
- name: shaders_gl verdict
if: always() && steps.shaders_gl.outcome != 'success'
run: |
@@ -284,6 +285,15 @@ jobs:
echo "fallback=yes" >>"$GITHUB_OUTPUT"
fi
# Tracked known failures of this leg (app/test/gl-qualification/known-failures.txt).
- name: Known failures of this leg
id: known
run: |
set -eu
ref="$(awk -v leg="windows-$ARCH" '$1 == leg && $2 == "exit-hang" {print $3; exit}' "$LISTS/known-failures.txt")"
echo "exit_hang=$ref" >>"$GITHUB_OUTPUT"
[ -z "$ref" ] || echo "::notice::windows-$ARCH: the app's exit hang is a tracked known failure ($ref); a launch is killed 5 s after its screenshot"
# The pre-#1138 loader against the bundled Mesa (before/after evidence), then a context
# created, drawn to, read back and destroyed through the same library the fallback child
# uses: the smallest program that must come back promptly on this runner.
@@ -308,6 +318,10 @@ jobs:
code=${PIPESTATUS[0]}
set -e
echo "context create/draw/destroy: exit $code after $SECONDS s" | tee -a glq/sdl-loader.txt
if [ "$code" -eq 124 ] && [ -n "${{ steps.known.outputs.exit_hang }}" ] && grep -q '^PASS GL loader' glq/sdl-loader.txt; then
echo "probe drew and read back, then did not exit: the tracked known failure (${{ steps.known.outputs.exit_hang }})" | tee -a glq/sdl-loader.txt
code=0
fi
# The same with llvmpipe's rasterizer threads off, then with softpipe (no LLVM at all):
# splits "llvmpipe's thread pool never winds down" from "WGL/the process itself".
for v in "LP_NUM_THREADS=0" "GALLIUM_DRIVER=softpipe"; do
@@ -342,6 +356,10 @@ jobs:
echo "automatic fallback launch: exit $code after $SECONDS s" | tee glq/automatic-seconds.txt
cat glq/automatic-stderr.txt
ok=yes
if [ "$code" -eq 124 ] && [ -n "${{ steps.known.outputs.exit_hang }}" ] && [ -s glq/automatic-fallback.png ]; then
echo "booted, screenshot written, did not exit: the tracked known failure (${{ steps.known.outputs.exit_hang }})" | tee -a glq/automatic-seconds.txt
code=0
fi
[ "$code" -eq 0 ] || ok=no
test -s glq/automatic-fallback.png || ok=no
grep -q 'automatically retrying with bundled Mesa OpenGL' glq/automatic-stderr.txt || ok=no
@@ -357,7 +375,7 @@ jobs:
# no-vsync the same with video.vsync=0 seeded in config.ini (a swap interval that never
# returns on a session without a compositor?)
- name: Fallback timing diagnostics
if: steps.native.outputs.fallback == 'yes' && steps.fallback.outputs.booted != 'yes'
if: steps.native.outputs.fallback == 'yes' && steps.fallback.outputs.booted != 'yes' && steps.known.outputs.exit_hang == ''
run: |
set -eu
export PATH="$PWD/stage/bin/software-opengl:$PATH"
@@ -383,20 +401,6 @@ jobs:
LP_NUM_THREADS=0 run_one lp-single
cat glq/timing.txt
# shaders_gl: without a GL 3.3 context it would skip (77); here it runs on the bundled Mesa,
# and a skip is a failure (judged after the preset loop).
- name: shaders_gl on the bundled Mesa (skip = failure)
id: shaders_gl
continue-on-error: true
run: |
set -eu
export PATH="$PWD/stage/bin/software-opengl:$PATH"
export SDL_OPENGL_LIBRARY="$(cygpath -w "$PWD/stage/bin/software-opengl/opengl32.dll")"
export GALLIUM_DRIVER=llvmpipe LIBGL_ALWAYS_SOFTWARE=true
export VJ_LIBRASHADER="$(cygpath -w "$PWD/stage/bin/librashader.dll")"
timeout 600 build/vj_shader_tests.exe gl 2>&1 | tee glq/shaders_gl.txt
test "${PIPESTATUS[0]}" -eq 0
# Every preset through the SHIPPED path: the stage/bin exe, which retries beside Mesa on its
# own (so each launch is the parent plus the child; the renderer line is the child's). Only
# once the plain boot came back: 500 launches of a boot that does not are hours of nothing.
@@ -409,12 +413,39 @@ jobs:
if [ "${{ steps.native.outputs.fallback }}" = yes ]; then
expect_log=(--expect-log 'automatically retrying with bundled Mesa OpenGL')
fi
hang=(); [ -z "${{ steps.known.outputs.exit_hang }}" ] || hang=(--known-exit-hang "${{ steps.known.outputs.exit_hang }}")
python -I "$QUALIFY" --os windows --exe stage/bin/virtualjaguar.exe --shaders stage/bin/shaders \
--out glq --content "$CONTENT_ROM" --frame "$SHOT_FRAME" --jobs 1 --timeout 150 \
--out glq --content "$CONTENT_ROM" --frame "$SHOT_FRAME" --jobs 1 --timeout 150 ${hang[@]+"${hang[@]}"} \
--expect-renderer 'llvmpipe|D3D12' ${expect_log[@]+"${expect_log[@]}"} \
--refused-allowlist "$LISTS/refused-windows.txt" --identical-ok "$LISTS/identical-to-plain.txt" --known-broken "$LISTS/known-broken.txt" \
--goldens app/test/golden/gl/windows --preset-set "$PRESET_SET" ${refresh[@]+"${refresh[@]}"}
# shaders_gl: without a GL 3.3 context it would skip (77); here it runs on the bundled Mesa,
# and a skip is a failure (judged after the preset loop).
- name: shaders_gl on the bundled Mesa (skip = failure)
id: shaders_gl
if: always()
continue-on-error: true
run: |
set -eu
export PATH="$PWD/stage/bin/software-opengl:$PATH"
export SDL_OPENGL_LIBRARY="$(cygpath -w "$PWD/stage/bin/software-opengl/opengl32.dll")"
export GALLIUM_DRIVER=llvmpipe LIBGL_ALWAYS_SOFTWARE=true
export VJ_LIBRASHADER="$(cygpath -w "$PWD/stage/bin/librashader.dll")"
set +e
timeout 600 build/vj_shader_tests.exe gl 2>&1 | tee glq/shaders_gl.txt
code=${PIPESTATUS[0]}
set -e
# The tracked exit hang: the test's verdict line comes after SDL_Quit, which never
# returns there, so the evidence is "nothing failed or skipped and the last check ran".
if [ "$code" -eq 124 ] && [ -n "${{ steps.known.outputs.exit_hang }}" ] \
&& ! grep -q 'shader_tests: FAIL\|shader_tests gl: SKIP' glq/shaders_gl.txt \
&& grep -q '^broken shader -> chain build failed' glq/shaders_gl.txt; then
echo "shader_tests gl: every check passed, then the process did not exit: the tracked known failure (${{ steps.known.outputs.exit_hang }})" | tee -a glq/shaders_gl.txt
code=0
fi
test "$code" -eq 0
- name: Automatic Mesa fallback verdict
if: always() && steps.native.outputs.fallback == 'yes' && (steps.fallback.outputs.booted != 'yes' || steps.loaders.outcome != 'success')
run: |
@@ -481,15 +512,6 @@ jobs:
perl -e 'alarm shift; exec @ARGV' 120 build/vj_gl_context_integration 2>&1 | tee glq/gl-context-integration.txt
test "${PIPESTATUS[0]}" -eq 0
- name: shaders_gl (skip = failure)
id: shaders_gl
continue-on-error: true
run: |
set -eu
export VJ_LIBRASHADER="$PWD/stage/bin/librashader.dylib"
perl -e 'alarm shift; exec @ARGV' 600 build/vj_shader_tests gl 2>&1 | tee glq/shaders_gl.txt
test "${PIPESTATUS[0]}" -eq 0
- name: Render every bundled preset
run: |
set -eu
@@ -503,6 +525,16 @@ jobs:
--refused-allowlist "$LISTS/refused-macos.txt" --identical-ok "$LISTS/identical-to-plain.txt" --known-broken "$LISTS/known-broken.txt" \
--goldens app/test/golden/gl/macos --preset-set "$PRESET_SET" ${refresh[@]+"${refresh[@]}"}
- name: shaders_gl (skip = failure)
id: shaders_gl
if: always()
continue-on-error: true
run: |
set -eu
export VJ_LIBRASHADER="$PWD/stage/bin/librashader.dylib"
perl -e 'alarm shift; exec @ARGV' 1200 build/vj_shader_tests gl 2>&1 | tee glq/shaders_gl.txt
test "${PIPESTATUS[0]}" -eq 0
- name: shaders_gl verdict
if: always() && steps.shaders_gl.outcome != 'success'
run: |
+64 -12
View File
@@ -32,6 +32,7 @@ import os
import pathlib
import re
import shutil
import signal
import subprocess
import sys
import time
@@ -177,6 +178,31 @@ def read_list(path):
return out
def png_complete(path):
"""The PNG is on disk and ends with its IEND chunk (SDL_SavePNG writes it last)."""
try:
with open(path, "rb") as f:
f.seek(-12, 2)
return f.read(12)[4:8] == b"IEND"
except (OSError, ValueError):
return False
def kill_tree(proc):
"""Kill the launch and whatever it spawned (on Windows the parent waits on the Mesa child)."""
if os.name == "nt":
subprocess.run(["taskkill", "/T", "/F", "/PID", str(proc.pid)], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
else:
try:
os.killpg(proc.pid, signal.SIGKILL) # its own session (start_new_session): the whole group
except (ProcessLookupError, PermissionError):
proc.kill()
try:
proc.wait(timeout=30)
except subprocess.TimeoutExpired:
pass
def launch(args, label, shader_arg, out_dir):
"""Run the app once. Returns a result dict (no picture analysis yet)."""
shots = out_dir / "shots"
@@ -198,16 +224,36 @@ def launch(args, label, shader_arg, out_dir):
cmd.append(str(args.content))
res = {"preset": label, "cmd": cmd, "png": str(png), "log": str(log), "errors": [], "notes": []}
t0 = time.monotonic()
try:
p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=args.timeout)
text = p.stdout.decode("utf-8", "replace")
res["exit"] = p.returncode
except subprocess.TimeoutExpired as e:
text = (e.stdout or b"").decode("utf-8", "replace") if isinstance(e.stdout, bytes) else ""
res["exit"] = None
res["errors"].append("timed out after %d s" % args.timeout)
# stdout+stderr straight to the log file (no pipe to drain), the process polled: a leg with a
# tracked exit hang (--known-exit-hang) is given a few seconds after its screenshot is complete,
# then its process tree is killed and the launch counts, with a note, instead of timing out.
with open(log, "wb") as lf:
proc = subprocess.Popen(cmd, stdout=lf, stderr=subprocess.STDOUT, start_new_session=(os.name != "nt"))
shot_done_at = None
while True:
code = proc.poll()
if code is not None:
res["exit"] = code
break
now = time.monotonic()
if now - t0 > args.timeout:
kill_tree(proc)
res["exit"] = None
res["errors"].append("timed out after %d s" % args.timeout)
break
if args.known_exit_hang:
if shot_done_at is None and png_complete(png):
shot_done_at = now
elif shot_done_at is not None and now - shot_done_at > args.hang_grace:
kill_tree(proc)
res["exit"] = None
res["notes"].append("did not exit %d s after its screenshot; killed (known failure: %s)"
% (args.hang_grace, args.known_exit_hang))
res["known_exit_hang"] = True
break
time.sleep(0.25)
res["seconds"] = round(time.monotonic() - t0, 1)
log.write_text(text, encoding="utf-8")
text = log.read_text(encoding="utf-8", errors="replace")
if res["exit"] not in (None, 0):
res["errors"].append("exit code %d" % res["exit"])
# The LAST renderer line: on stock Windows the first names the GDI Generic 1.1 context the
@@ -328,6 +374,8 @@ def main():
ap.add_argument("--frame", type=int, default=20, help="--screenshot-frame (UI frames before the capture)")
ap.add_argument("--timeout", type=int, default=180, help="seconds per launch")
ap.add_argument("--jobs", type=int, default=1, help="launches at once (each has its own --data-dir)")
ap.add_argument("--known-exit-hang", default="", help="this leg's app is known not to exit (cite the issue): a launch with a complete screenshot is killed after --hang-grace seconds and passes with a note")
ap.add_argument("--hang-grace", type=int, default=5, help="seconds a launch may keep running after its screenshot before --known-exit-hang kills it")
ap.add_argument("--fail-fast", type=int, default=8, help="stop after this many consecutive failed presets (0 = never): a leg whose GL or runtime is broken fails in minutes, not hours")
ap.add_argument("--low-dpi", action="store_true", help="pass --low-dpi (a 1x window on Retina)")
ap.add_argument("--extra-arg", action="append", default=[], help="another app argument (repeatable)")
@@ -420,10 +468,11 @@ def main():
built = [r for r in results if r["verdict"] == "built" and not r["errors"]]
refused = [r for r in results if r["verdict"] == "refused" and not r["errors"]]
broken = [r for r in results if r["verdict"] == "known-broken" and not r["errors"]]
hung = [r for r in results if r.get("known_exit_hang")]
(args.out / "results.json").write_text(json.dumps({
"os": args.os, "exe": str(args.exe), "frame": args.frame, "presets": len(presets),
"built": len(built), "refused": sorted(r["preset"] for r in refused),
"known_broken": sorted(r["preset"] for r in broken), "failed": len(failed),
"known_broken": sorted(r["preset"] for r in broken), "did_not_exit": len(hung), "failed": len(failed),
"results": results}, indent=1), encoding="utf-8")
md = ["## GL qualification: %s" % args.os, "",
@@ -434,6 +483,7 @@ def main():
"| Built | %d |" % len(built),
"| Refused (allowlisted) | %d |" % len(refused),
"| Known broken (allowlisted) | %d |" % len(broken),
"| Did not exit (known failure%s) | %d |" % ((": " + args.known_exit_hang) if args.known_exit_hang else "", len(hung)),
"| **Failed** | **%d** |" % len(failed), ""]
if refused:
md += ["<details><summary>Refused presets (allowlisted)</summary>", ""]
@@ -456,10 +506,12 @@ def main():
md += ["", "```", *gl[-12:], "```", ""]
except OSError:
md += [""]
notes = [r for r in results if [n for n in r["notes"] if n != "no golden"] and not r["errors"] and r["verdict"] == "built"]
def visible_notes(r):
return [n for n in r["notes"] if n != "no golden" and not n.startswith("did not exit")]
notes = [r for r in results if visible_notes(r) and not r["errors"] and r["verdict"] == "built"]
if notes:
md += ["<details><summary>Notes (%d)</summary>" % len(notes), ""]
md += ["- `%s`: %s" % (r["preset"], "; ".join(n for n in r["notes"] if n != "no golden")) for r in notes]
md += ["- `%s`: %s" % (r["preset"], "; ".join(visible_notes(r))) for r in notes]
md += ["", "</details>", ""]
summary = "\n".join(md) + "\n"
(args.out / "summary.md").write_text(summary, encoding="utf-8")
@@ -0,0 +1,15 @@
# Tracked known failures of gl-qualification.yml, one per line: <leg> <kind> <reference>.
# The workflow reads this file; an entry turns one specific failure into a recorded pass with
# a note, nothing else on that leg is forgiven. Remove the line when the fix lands: the leg
# then fails again if the failure comes back.
#
# windows-x64 exit-hang the app (and every process that made an llvmpipe context) never
# exits on the MSYS2 MINGW64 Mesa 26.2.3 the x64 package bundles:
# SDL_DestroyWindow/SDL_Quit hang after "context destroyed";
# LP_NUM_THREADS=0 and softpipe exit at once, arm64 Mesa exits in 2 s.
# gl_qualify.py kills a launch 5 s after its screenshot is complete and
# counts it; the boot step accepts the timeout when the screenshot and
# the [gl] lines are there; shaders_gl passes without its final OK line
# when nothing failed or skipped and its last check (the broken shader)
# ran. Product bug found by #1155 (PR #1160); tracked in #1167.
windows-x64 exit-hang #1167
+7 -6
View File
@@ -717,12 +717,13 @@ static int runGl() {
#ifdef VJ_SLANG_DIR
// ---- the bundled slang-shaders presets that must work on this GL ----
// At a real-world output size: the 128x64 picture the pixel checks above use is below what
// some presets are written for (crt-royale's mask passes scale by 0.0625 of the viewport and
// its preset says "Safe for >= 341.333 horizontal triads at viewport size"; at 128x64 those
// passes are 8x4 px and the chain comes out black on llvmpipe and Apple's software renderer,
// while the app at 960x760 renders it; gl-qualification.yml, #1155).
const int BW = 640, BH = 480;
// At a size the presets are written for: the 128x64 picture the pixel checks above use is
// below what some need (crt-royale's mask passes scale by 0.0625 of the viewport and its preset
// says "Safe for >= 341.333 horizontal triads at viewport size"; at 128x64 those passes are
// 8x4 px and the chain comes out black on llvmpipe and Apple's software renderer, while the
// app at 960x760 renders it; gl-qualification.yml, #1155). Not larger: on Apple's software
// renderer (hosted macOS) 640x480 did not finish the ~30 presets in 10 minutes.
const int BW = 384, BH = 288;
ShaderFrame fb = f;
fb.outW = BW;
fb.outH = BH;
+1
View File
@@ -58,6 +58,7 @@ int main(int argc, char** argv) {
SDL_GL_DestroyContext(context);
std::printf("context destroyed\n");
SDL_DestroyWindow(window);
std::printf("window destroyed\n");
SDL_Quit();
std::printf("SDL quit; exiting %d\n", passed ? 0 : 1);
return passed ? 0 : 1;
+9 -4
View File
@@ -381,10 +381,11 @@ beside the bundled MSYS2 Mesa, #1138/#1143), Windows arm64 (same, plus the porta
Store testers used), macOS arm64 (Apple OpenGL 4.1). Runs on PRs touching the GL/shader path,
nightly, on dispatch, and `release.yml` requires it on tags and `release/**`.
Per leg: one build, `cmake --install` into `stage/`, then `vj_shader_tests gl` run **directly**
(exit 77 = skipped = failure here; everywhere else it is `-E shaders_gl`; the step is
`continue-on-error` and a "shaders_gl verdict" step fails the job after the preset loop, so the
screenshots exist even when the test is red), then
Per leg: one build, `cmake --install` into `stage/`, the preset loop (below), then
`vj_shader_tests gl` run **directly** after it (exit 77 = skipped = failure here; everywhere else
it is `-E shaders_gl`; the step is `continue-on-error` and a "shaders_gl verdict" step fails the
job, so the screenshots exist even when the test is red, and a test that has to be killed cannot
wedge the window server before the loop). The loop:
`app/test/gl-qualification/gl_qualify.py` launches `stage/bin/virtualjaguar --shader <preset>
--screenshot <png> --screenshot-frame 20 test/roms/yarc.j64` for `--shader off` and every preset.
Per (leg, preset) it asserts: exit 0 before the timeout; the `[gl] version=...; renderer=` line
@@ -421,6 +422,10 @@ candidates into the artifact.
- `identical to the --shader off capture` → the preset changed nothing; either the chain fell back
to the plain presenter (check for a `[shader]` line) or it is a true no-op at this scale (list it).
- `screenshot is black` → the window drew nothing (lost context, hidden window, swap failure).
- `known-failures.txt` holds tracked failures with their issue (`windows-x64 exit-hang`, #1167: the app never
exits on the x64 bundled Mesa; a launch with a complete screenshot is killed after 5 s and counted,
the boot step and `shaders_gl` accept the timeout when their evidence is complete). Remove the line
when the fix lands; nothing else on that leg is forgiven.
- Windows "Automatic Mesa fallback verdict" red → the stock-Windows path (#1138) regressed or the
boot did not come back: `automatic-seconds.txt` has the exit code, the seconds and which of the
checks failed (the retry line, the `renderer=` line, `config.ini` in the parent's data dir, no