radareorg/r2sleigh
1
0 Fork 0
mirror of https://github.com/radareorg/r2sleigh.git synced 2026-11-07 06:36:00 -10:00
Code Issues Packages Projects Releases Wiki Activity

DecBench 2026-10-24: r2s at master 7f696784, D65 renders 86 import-stub functions

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01456qmqCGpCWJkrg6JUZNNM
This commit is contained in:
Priyanshu KumarandClaude Sonnet 5.5 committed 2026-10-24 03:35:46 +05:30
1 parent 7f696784f7
commit 9bafe84a77
1 file changed
+201
Download Patch File Download Diff File Expand all files Collapse all files
tests/decbench/RESULT-2026-10-24.md
+201
View File
@@ -0,0 +1,201 @@
# DecBench, 2026-10-24: r2s at master 7f696784, against the corrected 2026-10-23 tables
r2s revision `7f696784` (`origin/master`, merge of #334), release build on contabo (rustc 1.99.0, cold target, 5 m 05 s),
`sha256:7df8377123cc5b47`. The 2026-10-23 run measured `8cff06fe` (`origin/structure-tails`,
`sha256:3764e777a7d6726f`). DecBench `ae070b8`, the same checkout as 2026-10-17 and 2026-10-23. Official protocol:
`scripts/run_benchmark.py` through `tests/decbench/run_decbench.sh`, stripped inputs, DWARF `low_pc` targets. The
script's sha256 gate passed (`results measured r2s sha256:7df8377123cc5b47`). Same 48 compiled binaries (8 projects, O2 and
O2-noinline). Only r2s was re-run (`DECBENCH_REDO_DECOMPILERS=r2sleigh_native` on a copy of the 2026-10-23 tree), so kuna,
ghidra and angr are the 2026-10-17 decompilations and their scores are the 2026-10-23 ones, bash included.
## Answer
1. **r2s gains 66 union rows, all from D65.** Over all rows: union-perfect 1,899 to 1,965 of 6,788 (28.0% to 28.9%),
byte_match 569 to 582, GED perfect 1,547 to 1,611, type_match 328 to 328. r2s stays third on union (kuna 32.0%, ghidra
29.3% with 26 rows more than r2s, angr 18.8%). It moves from third to second on GED perfect (1,611 against ghidra's 1,605).
2. **Non-bash, 2,736 rows: r2s is third on union, not fourth.** kuna 835 (30.5%), angr 834, r2s 726 (26.5%, was 695),
ghidra 715 (26.1%). r2s is first on byte_match (218) and third on GED perfect (623 against ghidra 602); it is
still fourth on type_match (91).
3. **The 66 come from 86 newly rendered functions.** The 2026-10-23 census holds 86 refusals `import stub at 0x...; this
symbol is the import X and has no body of its own`; this run holds none. Of the 86 rows, 66 are union-perfect
(GED 64, byte_match 13, type_match 0) and another tool is perfect on 78 (66 on GED). Per project: bash 43, zlib 20,
libexpat 9, bzip2 6, gzip 6, grep 2, diffutils 0, findutils 0.
4. **No-output goes 380 to 208, not to about 317.** Each newly rendered function removes two rows: its hex-named row
leaves the table (`0x41950`) and its DWARF-named row gets output. 380 - 2 x 86 = 208. The 317 estimate counted the 63
wrapper rows once. The other decompilers lose the same 86 hex-named rows (kuna 190 to 104, ghidra 323 to 237, angr 2,957
to 2,871), which is why their no-output counts fall too.
5. **The structurer stack changed the text of 50 of the 6,598 functions r2s already rendered and no perfect count.** The
2026-10-23 build was `origin/structure-tails`, the stack tip of that day. Of the 6,598 functions rendered by both builds, 6,548 have identical text and 50 differ: 4 in line order or the proof
comment, 10 with more residual markers, 36 in structure or naming (for example a `for (;;)` with two breaks rotated to
`do { } while (a && b)`). On the 6,598: GED better 2, worse 10; byte_match better 6, worse 8; type_match better 1; newly
perfect 0 and no longer perfect 0 on all three metrics; gotos 16,576 to 16,577; residual sites 160,780 to 160,829.
6. **No OOM, no lost evaluation, no lost function.** `journalctl -k` since the run start holds no OOM line. Both
`run_gate.py evaluated` checks passed (34 non-bash and 14 bash binaries). The coverage guard fired for 86 rows, all
hex-named, none added, no function lost.
7. **Mined set (bzip2, zlib, grep; O2 and O2-noinline) at master, scored with the owner mapping:** 1,595 functions scored,
261 GED-perfect, mean GED 43.99, total GED 70,163, 3,480 gotos in 507 functions. These replace 206 / 43.795 / 57,153
(1,305 functions then), which predate the owner-mapping fix.
## Method
* Build: `git archive` of `bench-master` (= `origin/master`) to a snapshot directory on contabo, then
`cargo build --release -p r2s --features sleigh` into a new target directory.
* Tree: a copy of the 2026-10-23 results tree. The refusal censuses of the earlier runs were deleted from the copy
(they are named by path, so two trees in one directory are counted twice by `census_decbench.py`).
* Non-bash: `run_decbench.sh --workers 1 --project bzip2 diffutils findutils grep gzip libexpat zlib`; then bash:
`run_decbench.sh --workers 1 --project bash`. `--workers 1` for both because `run_gate.py workers` reads the 13 MB
bash C that sits in the tree; four workers were not tried. Bash needs one worker for its evaluation anyway.
* `DECBENCH_ALLOW_DROPS=1`, so the driver's own finalize writes through the coverage guard. The row keys of the old and
new `function_results.json` were then diffed: 86 dropped, 0 added, every dropped key hex-named
(bash 43, zlib 20, libexpat 9, bzip2 6, gzip 6, grep 2), each a function the new run renders under its DWARF name.
* `report_decbench.py` exits 65 at the end of each invocation: the baseline is marked invalid, as on 2026-10-23.
* The tables below are `compare_runs.py` (scoreboard and restricted populations) and a script that counts per row set
(all, non-bash, bash, common, each project) from `function_results.json`. On the old tree it reproduces the
2026-10-23 tables exactly (1,899 / 2,169 / 1,991 / 1,278 union).
## 1. Tables
Full tree, 6,892 rows (6,788 with a metric measured for someone). Old r2s in brackets.
| Decompiler | Union-perfect | byte_match perfect | GED perfect (mean) | type_match perfect (mean) |
|---|---|---|---|---|
| kuna | 2169 of 6788 (32.0%), rank 1 | 465 (6.9%), rank 2 | 1830 of 6453 (28.4%), 17.7, rank 1 | 539 of 6365 (8.5%), 0.268, rank 1 |
| ghidra | 1991 (29.3%), rank 2 | 330 (4.9%), rank 3 | 1605 (24.9%), 19.2, rank 3 | 491 (7.7%), 0.295, rank 2 |
| r2sleigh_native | 1965 (28.9%), rank 3 [1899, 28.0%] | 582 (8.6%), rank 1 [569] | 1611 (25.0%), 29.6, rank 2 [1547, 29.9, rank 3] | 328 (5.2%), 0.191, rank 3 [328, 0.192] |
| angr | 1278 (18.8%), rank 4 | 234 (3.4%), rank 4 | 1073 (16.6%), 19.2, rank 4 | 250 (3.9%), 0.287, rank 4 |
Non-bash, 2,736 rows:
| Decompiler | Union-perfect | byte_match perfect | GED perfect (mean) | type_match perfect (mean) |
|---|---|---|---|---|
| kuna | 835 of 2736 (30.5%), rank 1 | 173 (6.3%), rank 2 | 724 of 2732 (26.5%), 18.7, rank 1 | 164 of 2664 (6.2%), 0.251, rank 1 |
| angr | 834 (30.5%), rank 2 | 155 (5.7%), rank 4 | 718 (26.3%), 19.9, rank 2 | 140 (5.3%), 0.287, rank 2 |
| r2sleigh_native | 726 (26.5%), rank 3 [695, 25.4%, rank 4] | 218 (8.0%), rank 1 [216] | 623 (22.8%), 38.5, rank 3 [594, 39.0, rank 4] | 91 (3.4%), 0.182, rank 4 [91, 0.185] |
| ghidra | 715 (26.1%), rank 4 | 172 (6.3%), rank 3 | 602 (22.0%), 20.9, rank 4 | 121 (4.5%), 0.279, rank 3 |
bash alone, 4,052 rows: kuna 1334 union (32.9%), ghidra 1276 (31.5%), r2s 1239 (30.6%, was 1204), angr 444 (11.0%).
GED perfect kuna 29.7%, ghidra 27.0%, r2s 26.6% (was 25.6%), angr 9.5%. type_match perfect kuna 375, ghidra 370, r2s 237,
angr 110. byte_match r2s 364 (9.0%, was 353), kuna 292, ghidra 158, angr 79.
Common rows, the rows all four decompilers scored (3,855 now; 3,814 on 2026-10-23, the 41 added are r2s rows that were
unscored). No-output is zero on these rows by definition.
| Decompiler | Union-perfect | byte_match perfect | GED perfect (mean) | type_match perfect (mean) |
|---|---|---|---|---|
| angr | 1219 (31.6%) | 227 | 1018 (18.4) | 240 (0.288) |
| kuna | 1170 (30.4%) | 245 | 1012 (17.8) | 234 (0.246) |
| ghidra | 1097 (28.5%) | 215 | 923 (19.2) | 221 (0.279) |
| r2sleigh_native | 1033 (26.8%) | 252 | 884 (34.7) | 146 (0.178) |
| r2sleigh_native, 2026-10-23, its 3,814 rows | 1004 (26.3%) | 252 | 855 (35.0) | 146 (0.180) |
On the 3,814 rows both runs scored, r2s's four counts are identical (1,004 / 252 / 855 / 146): the gain sits in rows the old
build left unscored.
r2s, old against new:
| Row set | Metric | 2026-10-23 | now | change |
|---|---|---|---|---|
| all rows | union-perfect | 1899 | 1965 | +66 |
| | byte_match perfect | 569 | 582 | +13 |
| | GED perfect, mean | 1547, 29.92 | 1611, 29.57 | +64, -0.35 |
| | type_match perfect, mean | 328, 0.192 | 328, 0.191 | 0, -0.002 |
| | rows with no output (all rows) | 380 of 6,978 | 208 of 6,892 | -172 |
| non-bash | union-perfect | 695 | 726 | +31 |
| | byte_match perfect | 216 | 218 | +2 |
| | GED perfect, mean | 594, 39.04 | 623, 38.49 | +29, -0.55 |
| | type_match perfect | 91 | 91 | 0 |
| | rows with no output (all rows) | 222 of 2,847 | 136 of 2,804 | -86 |
| bash | union-perfect | 1204 | 1239 | +35 |
| | byte_match perfect | 353 | 364 | +11 |
| | GED perfect, mean | 953, 23.37 | 988, 23.13 | +35, -0.23 |
| | type_match perfect | 237 | 237 | 0 |
| | rows with no output (all rows) | 158 of 4,131 | 72 of 4,088 | -86 |
Per project (union-perfect; rows are the measured rows):
| Project | Rows | r2s 2026-10-23 | r2s now | kuna | ghidra | angr | Newly rendered | no-output r2s, old to new (all rows) |
|---|---|---|---|---|---|---|---|---|
| bash | 4052 | 1204 | 1239 (+35) | 1334 | 1276 | 444 | 43 | 158 to 72 |
| zlib | 1321 | 233 | 243 (+10) | 300 | 224 | 280 | 20 | 100 to 60 |
| libexpat | 96 | 55 | 64 (+9) | 65 | 58 | 65 | 9 | 24 to 6 |
| gzip | 254 | 76 | 82 (+6) | 90 | 87 | 90 | 6 | 26 to 14 |
| bzip2 | 162 | 37 | 41 (+4) | 43 | 40 | 50 | 6 | 28 to 16 |
| grep | 156 | 40 | 42 (+2) | 50 | 39 | 43 | 2 | 8 to 4 |
| diffutils | 257 | 55 | 55 | 80 | 75 | 80 | 0 | 22 to 22 |
| findutils | 490 | 199 | 199 | 207 | 192 | 226 | 0 | 14 to 14 |
Per-project change in GED perfect: bash +35, zlib +10, libexpat +9, gzip +6, bzip2 +4, the rest 0. byte_match: bash +11,
grep +2, the rest 0. type_match: 0 everywhere. libexpat now ties kuna and angr within one row (64 against 65).
## 2. What moved
* Text of the 6,684 functions in r2s's 48 decompiled files: 6,548 identical, 50 changed, 86 new, none lost. Of the 50,
4 differ in line order (`pair_*` declarations) or the proof comment; 10 have more `r2sleigh_residual_*` markers (a read
before its definition has run is now a residual: `rbx_2 = r2sleigh_residual_u64(8)` where the old text had
`rbx_2 = rbx_5`); 36 differ in structure or naming (do-while rotation of a loop with two breaks, a merged condition, a
swapped `_next` suffix, renumbered residuals). GED is worse on 10 of the 6,269 scored functions and better on 2; none of
the 12 is perfect on either build.
* The 86 new functions are the `import stub` refusals of the 2026-10-23 census. `census_decbench.py` over the two sets of
censuses (6,794 functions each): refusal causes 190 to 104, import stub 86 to 0, native effect obligations 104 to 104,
coverage 0.972 to 0.985, proven 2,720 to 2,742.
* The mined-set view agrees (section 3): on the 1,569 functions both builds score, perfect 247 to 247, no function newly
perfect or lost, GED unchanged; 26 more functions are scored; 38 of 1,639 texts changed; refusals 68 to 40.
## 3. Mined set at master
`mined_set.py dump`, `score` and `pair master.json master.json` over `--tree` = this run's tree, `--set mining`, the same
r2s as the table above. The 2026-10-23 build (`3764e777`) was run through the same commands for the paired line.
| | scored | perfect | mean GED | total GED | gotos (all = scored) | functions with a goto | refused | functions |
|---|---|---|---|---|---|---|---|---|
| master, all binaries | 1595 | 261 | 43.99 | 70,163 | 3,480 | 507 | 40 | 1,639 |
| master, `-u` (one entry per function) | 585 | 108 | 54.58 | 31,931 | 1,800 | 199 | 14 | 603 |
| 2026-10-23 build, all binaries | 1569 | 247 | 44.63 | 70,027 | 3,480 | 507 | 68 | 1,639 |
| 2026-10-23 build, `-u` | 575 | 102 | 55.48 | 31,899 | 1,800 | 199 | 26 | 603 |
Paired, 2026-10-23 build to master, over the 1,569 functions both score: perfect 247 to 247, newly 0, lost 0, better 0,
worse 0, same 1,569; gotos 3,480 to 3,480. Master scores 26 functions the old build refused (and 10 in `-u`); 4 functions
have an owner and no source CFG (2 before). The earlier absolute figures (206 / 43.795 / 57,153) match the all-binaries
mode: 57,153 / 43.795 is 1,305 scored functions, 290 fewer than now.
## 4. Time and memory
| | 2026-10-23 | now |
|---|---|---|
| bash O2, r2s decompile phase | 1,253 s | 277 s |
| bash O2-noinline, r2s decompile phase | 1,723 s | 237 s |
| bash decompile phases, total | 2,976 s | 514 s |
| bash evaluation phases, one worker | 1,067 s | 820 s |
| non-bash decompile phases | 402 s | 465 s |
| all decompile phases | 3,378 s | 979 s |
| non-bash evaluation phases | 292 s (4 workers) | 242 s (1 worker) |
| wall time of the two invocations | not recorded | 2,173 s non-bash, 2,851 s bash (the rest is the driver's finalize) |
| peak RSS of r2s | 252 MB | 249 MB |
| peak RSS of the largest python process | 5.2 GB (an evaluation worker) | 4.58 GB |
The bash decompile phase falls to a fifth of its time, as the checkpoint-by-time change in `r2sleigh_raw.py` predicts
(README, "Checkpoints"); this run did not isolate it from the r2s change. The installed backend files differ from
2026-10-23 (`r2sleigh_raw.py` 2b92c863 to fd06f2bb, `r2s_batch.py` c21a0839 to 1c13d8e4). Host minimum available
memory was 1.28 GB; peak swap use was 43 MB of 10 GB. The sampler read `ps` RSS every 2 s.
## 5. What this leaves
* r2s trails ghidra by 26 union rows and kuna by 204 over all rows. The ranking of causes in the 2026-10-23 note stands
(library prototypes into values, a proven return or `void`, widths from reads); type_match did not move (328 to 328).
* The 208 remaining no-output rows are 104 functions counted twice, once as a hex-named row and once as the
DWARF-named row. The 104 are the native effect obligations refusals, unchanged from 2026-10-23. The import-stub class
is empty.
## Reproduce
```
tests/decbench/run_decbench.sh --decbench <decbench copy> --python <venv>/bin/python --out <copy of the 2026-10-23 tree> \
--r2s <r2s at 7f696784> --decompilers r2sleigh_native,kuna,ghidra,angr --opt-level O2 --opt-level O2-noinline \
--project <p>... --workers 1
# with DECBENCH_REDO_DECOMPILERS=r2sleigh_native DECBENCH_ALLOW_DROPS=1 PYTHONPATH=<decbench copy>
python tests/decbench/compare_runs.py --new <tree> --old <2026-10-23 tree> [--exclude-project bash | --only-project bash]
python tests/decbench/census_decbench.py <refusals dir>
tests/decbench/mined_set.py dump --tree <tree> --r2s <r2s> --out master --set mining
<venv>/bin/python tests/decbench/mined_set.py score --tree <tree> --dump master --out master.json --decbench <decbench copy>
tests/decbench/mined_set.py pair master.json master.json [-u]
```