r2sleigh/doc/testing.md

Testing

Where a new test goes, the validation bar and --no-fail-fast are in AGENTS.md. This page covers the layers, harnesses and baselines.

Layers

• Unit tests, inline beside a lowering, predicate, certificate or fold.
• Crate integration tests in crates/<crate>/tests/, through the public surface only (crates/r2engine/tests/engine/native.rs is the model).
• Golden files in tests/db/, radare2's r2r format, run by crates/r2s/tests/golden.rs: a command's whole output on a fixed binary, so a pull request's diff shows what moved (tests/db/README.md).
• The claim oracle, crates/r2engine/tests/engine/oracle.rs, described below.
• End-to-end harnesses that run a built r2s over real binaries.

The claim oracle

The oracle checks every claim pd and pdf make against runs of the function's own lift, with r2il::eval as the machine: one concrete byte state per run, one step per operation as the P-code reference defines it, and nothing imported from an analysis. Constant folds in r2ssa answer through the same r2il::eval::apply, so the oracle checks the semantics the engine folds with.

• Entry states give every register read zero, one, all ones, each constant the body names and its neighbours, and seeded randoms; P-code booleans get only zero or one. The stack pointer and specification-tracked registers are pinned. A second run per state seeds every word read before written.
• A run stops at a call, an unmapped access, an unexecutable operation or its budget, and rules on nothing after that.
• A claim no run rules on fails, so an unchecked claim never passes; a second test injects the overclaims each check exists to reject. A new claim kind does not compile until verdict says how a run rules on it.

Harnesses

certify_render.py, diff_r2.py and run_equiv.py read target/debug/r2s unless given --r2s; the first two build it first unless given --no-build, and run_coverage.sh builds and uses the release binary.

Question	Harness
Does every named function render, reading nothing never written?	python3 scripts/certify_render.py --bins <radare2>/test/bins/elf --limit 24 --functions 8
Where do discovery, naming and decoding disagree with radare2?	python3 scripts/diff_r2.py --bins <radare2>/test/bins/elf --limit 30
How much of each corpus binary renders, against a baseline?	./tests/coverage/run_coverage.sh [--accept-baseline]
Does the rendered C compute what the binary computes (x86-64 Linux)?	tests/equiv/run_equiv.py --r2s target/debug/r2s --baseline tests/equiv/baseline.json
What does DecBench score, on its own protocol?	tests/decbench/run_decbench.sh (see its README)
Did any rendering move, base against head?	python3 scripts/census.py run --r2s <r2s> --out <dir> per side, then census.py diff <base> <head> --report <file> (CI: Census + Timing vs Base)
Did a large function get slower?	python3 scripts/census.py time --r2s <base> --r2s <head> --bins <radare2>/test/bins/elf (fails above 1.3x in CI)
How does pdd cost grow with function size (LX's exit)?	python3 scripts/census.py exponent --r2s <r2s> --max-exponent 1.05 <binary>... (Linux, perf)
What holds the heap at its peak, and which sites grow faster than the function?	cargo build --profile profiling -p r2s --features sleigh,dhat-heap, run R2S_HEAP_PROFILE=1 r2s -q -c 'pdd @ <f>' <binary> (writes dhat-heap.json), then python3 scripts/census.py heap [<instructions>=]<dhat-heap.json>...
How robust is one large binary?	tests/coverage/sweep_binary.sh <binary>

scripts/setup_corpus.py builds local corpora (coreutils, CGC, Juliet) under /tmp/r2sleigh-corpora. Keep binaries and reports out of the repository, and keep reports sorted so two revisions diff.

A disagreement with radare2 may be radare2's defect; judge it, never match it.

Baselines

tests/coverage/coverage-baseline.json records, per function, whether it rendered and the typed cause when it did not. A function that rendered and now refuses fails the gate; one that now renders needs --accept-baseline. tests/equiv/baseline.json says what each rendering computes; every record that is not equal carries its cause, and run_equiv.py exits 3 until a baseline is blessed with --write-baseline <path>.

Re-bless a baseline only after reading the new output and judging it correct, never because it differs. Agreement with a recorded output is evidence that nothing moved, not that the output is right; a score is never semantic proof.

Closed pipes

A test that closes the read end of a child's output and expects the child's next write to fail (exit 141 in crates/r2s/tests/commands.rs) races every other thread of its test binary that spawns. A spawn in flight holds a copy of each descriptor the process has open until its exec completes, which takes milliseconds under load, so the pipe stays open there and the child's write succeeds: the test saw exit 0 in 159 of 600 trials in a reproduction that closed the pipe of a sh after its prompt while eight threads spawned r2s, and in 0 of 600 when the closing thread held the spawn lock from the close to the exit. Every spawn in that binary goes through started, which takes SPAWNING, and the two pipe tests hold it from the close until the child has exited.

Diagnostics

R2DEC_TRACE_REFUSAL=1 prints every refusal_evidence! site with its predicate, file, line and operands. Use it to find where a refusal happens, then a debugger to see why. pdil (the lift) and pddo (the obligations) narrow a defect to the lift or the rendering.