mirror of
https://github.com/radareorg/r2sleigh.git
synced 2026-11-07 06:36:00 -10:00
DecBench: the mined-set comparison of two r2s builds lives in the tree
Two structuring items graded their changes with scratch scripts: dump every function of bzip2, zlib and grep at O2 and O2-noinline with each build, score with DecBench's GED, pair the two runs. mined_set.py is that, in three steps (dump, score, pair), with the pinned lists that split mining (bzip2, zlib, grep) from held-out (the other five projects, nightly and at an item's exit). pair reports the perfect count (newly, lost), mean and total GED, gotos, functions with a goto, functions whose text changed and newly refused ones, and refuses to pair a mining run with a held-out run. On the scratch scores of the s3 step it prints the numbers the scratch pairing script did (perfect 60 -> 55, mean GED 49.11 -> 48.33, total 21509 -> 21170, gotos 4143 -> 4080). dump finds the stripped binary by the driver's stem (libz.so.1.2.13 is libz.so.1.2); the scratch measure.py looked for a file named by the stem and would have skipped it. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01456qmqCGpCWJkrg6JUZNNM
This commit is contained in:
Priyanshu KumarandClaude Sonnet 5.5 committed
1 parent
b6fce39b59
commit
001ef1053d
3 files changed
+718
No files matched your search
@@ -153,10 +153,39 @@ into the tree (the scores it adds to a checkpoint).
|
||||
functions each cause alone keeps imperfect; `universe` says why a function has no row.
|
||||
* `ged_causes.py`, `counterfactual_ged.py`: block-graph deltas of the not-GED-perfect
|
||||
functions, and a re-score after a respelling, to bound what a renderer change gains.
|
||||
* `mined_set.py`: the mined-set comparison of two r2s builds (next section).
|
||||
* `run_gate.py`: the lost-evaluation and worker gates `run_decbench.sh` runs.
|
||||
* `evaluate_checkpointed.py`: evaluates decompilers already in a checkpoint (the driver
|
||||
only evaluates what it has just run); `r2s_timing.py`: r2s alone, without the backend's
|
||||
checkpoint, with peak RSS; `show_function.py`: source, rendering, scores and DWARF of one function.
|
||||
|
||||
The mined-set comparison
|
||||
------------------------
|
||||
|
||||
`mined_set.py` is how a structurer change is graded (ROADMAP D47): every function of
|
||||
the pinned mining set, bzip2, zlib and grep, at O2 and O2-noinline, rendered by two
|
||||
r2s builds and scored with DecBench's GED, then paired. It reads a results tree the
|
||||
driver wrote (`<tree>/<opt>/<project>/{stripped,compiled,decompiled}`: the targets
|
||||
are the addresses its `decompiled/*.toml` name) and runs where DecBench and Joern
|
||||
are installed.
|
||||
|
||||
```
|
||||
tests/decbench/mined_set.py dump --tree RUN --r2s base/r2s --out base --set mining
|
||||
tests/decbench/mined_set.py dump --tree RUN --r2s new/r2s --out new --set mining
|
||||
../decbench-venv/bin/python tests/decbench/mined_set.py score --tree RUN --dump base --out base.json --decbench ../decbench
|
||||
../decbench-venv/bin/python tests/decbench/mined_set.py score --tree RUN --dump new --out new.json --decbench ../decbench
|
||||
tests/decbench/mined_set.py pair base.json new.json # -u: one entry per function; -v: name them
|
||||
```
|
||||
|
||||
`pair` prints, over the functions both runs scored: perfect count with the ones newly
|
||||
perfect and lost, mean and total GED with better, worse and same, gotos (all, and over
|
||||
the scored functions) and the functions with a goto, the functions whose text changed,
|
||||
and refusals (newly refused). `--set heldout` selects the other five projects (bash,
|
||||
diffutils, findutils, gzip, libexpat): those are run nightly and at an item's exit,
|
||||
never to choose a rule, and `pair` refuses to compare a mining run with a held-out
|
||||
run. The source CFGs are cached beside the scores (`srccache/`); delete it when the
|
||||
sources are recompiled.
|
||||
|
||||
Local measurement
|
||||
-----------------
|
||||
|
||||
|
||||
@@ -0,0 +1,405 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The mined-set comparison: what a structurer change does to GED, on the functions it was mined from.
|
||||
|
||||
A change to the structurer (or to anything that moves a rendering) is graded on every function of one
|
||||
pinned set of DecBench projects, with the build before it and the build after it:
|
||||
|
||||
mined_set.py dump --tree <run> --r2s <base r2s> --out base/ --set mining
|
||||
mined_set.py dump --tree <run> --r2s <new r2s> --out new/ --set mining
|
||||
mined_set.py score --tree <run> --dump base/ --out base.json --decbench ../decbench
|
||||
mined_set.py score --tree <run> --dump new/ --out new.json --decbench ../decbench
|
||||
mined_set.py pair base.json new.json
|
||||
|
||||
* `dump` asks r2s for `pddj` of every function of every binary of the set, at O2 and O2-noinline, through
|
||||
the runner the backend uses (`tests/equiv/r2s_batch.py`, on the stripped binary). The functions are the
|
||||
addresses the run's own `decompiled/*.toml` name, the union over its decompilers, so the targets are the
|
||||
driver's: the DWARF functions of the project's own sources.
|
||||
* `score` computes DecBench's GED for each rendering against the source CFG of the `.i` that defines the
|
||||
function (`<tree>/<opt>/<project>/compiled/*.i`), by the flow of DecBench's `metrics/ged.py` (isomorphism,
|
||||
the size bound above `GED_MAX_NODES`, else VJ). It needs DecBench importable (`--decbench` or the venv) and
|
||||
Joern; it is not run by the unit tests.
|
||||
* `pair` reports two scored runs over the functions both scored: perfect count (newly, lost), mean and total
|
||||
GED (better, worse, same), gotos, functions with a goto, functions whose text changed and functions newly
|
||||
refused.
|
||||
|
||||
The pinned lists split what may be mined from what may only be reported: MINING (bzip2, zlib, grep) is run
|
||||
on every change; HELD_OUT is the rest of the sailr set, run nightly and at an item's exit only, and a
|
||||
regression there opens an investigation. A structurer rule is chosen by reading MINING; reading HELD_OUT to
|
||||
choose one defeats it. `pair` refuses to compare a mining run with a held-out run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import pickle
|
||||
import re
|
||||
import statistics
|
||||
import sys
|
||||
import tempfile
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
from run_gate import split_name # noqa: E402
|
||||
|
||||
MINING = ("bzip2", "zlib", "grep")
|
||||
HELD_OUT = ("bash", "diffutils", "findutils", "gzip", "libexpat")
|
||||
SETS = {"mining": MINING, "heldout": HELD_OUT}
|
||||
OPT_LEVELS = ("O2", "O2-noinline")
|
||||
DECOMPILERS = ("r2sleigh_native", "angr", "ghidra", "kuna")
|
||||
SCHEMA = 1
|
||||
GED_SECONDS = 120
|
||||
|
||||
|
||||
# --- targets and dump -------------------------------------------------------------------------
|
||||
|
||||
|
||||
def function_targets(project_dir: Path, decompilers=DECOMPILERS) -> dict[str, dict[int, str]]:
|
||||
"""`{binary stem: {address: DWARF name}}` from the tomls the run's decompilers wrote for a project."""
|
||||
found: dict[str, dict[int, str]] = {}
|
||||
for toml in sorted((project_dir / "decompiled").glob("*.toml")):
|
||||
named = split_name(toml.stem, list(decompilers))
|
||||
if named is None:
|
||||
continue
|
||||
data = tomllib.loads(toml.read_text())
|
||||
for key, value in data.items():
|
||||
if key.startswith("functions.") and isinstance(value, dict) and "address" in value:
|
||||
address = value["address"]
|
||||
address = int(address, 16) if isinstance(address, str) else int(address)
|
||||
found.setdefault(named[1], {}).setdefault(address, key[len("functions."):])
|
||||
return found
|
||||
|
||||
|
||||
def row_keys(opt: str, project: str, binary: str, targets: dict[int, str]) -> dict[int, str]:
|
||||
"""`opt|project|binary|name` per address; a name already taken by a lower address gets `@0x<address>`."""
|
||||
keys: dict[int, str] = {}
|
||||
taken: set[str] = set()
|
||||
for address in sorted(targets):
|
||||
key = f"{opt}|{project}|{binary}|{targets[address]}"
|
||||
if key in taken:
|
||||
key += f"@0x{address:x}"
|
||||
taken.add(key)
|
||||
keys[address] = key
|
||||
return keys
|
||||
|
||||
|
||||
def record_of(answer, name: str) -> dict:
|
||||
"""One dumped function: what r2s said, with the definition renamed to the DWARF name."""
|
||||
row = {"name": name, "addr": f"0x{answer.address:x}", "status": answer.kind,
|
||||
"refused": bool(answer.refusal)}
|
||||
if answer.ok and answer.record is not None:
|
||||
code = str(answer.record.get("code") or "")
|
||||
definition = str(answer.record.get("definition") or "")
|
||||
if definition and definition != name:
|
||||
code = re.sub(rf"\b{re.escape(definition)}\b", name, code)
|
||||
row["code"] = code
|
||||
else:
|
||||
row["cause"] = (answer.cause or "")[:300]
|
||||
return row
|
||||
|
||||
|
||||
def dump(tree: Path, r2s: Path, out: Path, projects, opt_levels=OPT_LEVELS, decompilers=DECOMPILERS,
|
||||
function_timeout: float = 300.0, startup_timeout: float = 600.0, set_name: str = "adhoc") -> int:
|
||||
sys.path.append(str(HERE.parent / "equiv"))
|
||||
import r2s_batch
|
||||
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
total = 0
|
||||
for opt in opt_levels:
|
||||
for project in projects:
|
||||
project_dir = tree / opt / project
|
||||
stripped = {p.stem: p for p in sorted((project_dir / "stripped").glob("*")) if p.is_file()}
|
||||
for binary, targets in sorted(function_targets(project_dir, decompilers).items()):
|
||||
if binary not in stripped:
|
||||
print(f"skip {opt}/{project}/{binary}: no stripped binary", file=sys.stderr)
|
||||
continue
|
||||
keys = row_keys(opt, project, binary, targets)
|
||||
report = r2s_batch.run_batch(r2s, stripped[binary], sorted(targets),
|
||||
function_timeout=function_timeout,
|
||||
startup_timeout=startup_timeout)
|
||||
rows = {keys[a.address]: record_of(a, targets[a.address]) for a in report.answers}
|
||||
(out / f"{opt}.{project}.{binary}.json").write_text(json.dumps(rows, sort_keys=True))
|
||||
total += len(rows)
|
||||
print(f"{opt} {project} {binary}: {len(rows)} functions", flush=True)
|
||||
manifest = {"schema": SCHEMA, "set": set_name, "projects": list(projects), "opt_levels": list(opt_levels),
|
||||
"r2s_sha256": hashlib.sha256(Path(r2s).read_bytes()).hexdigest(), "functions": total}
|
||||
(out / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")
|
||||
return total
|
||||
|
||||
|
||||
# --- scoring (DecBench) ------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Timeout(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def decbench_scorer(decbench: Path | None):
|
||||
"""DecBench's GED flow as `(source_cfgs, ged, dec_cfg_of)`; imports DecBench, so it needs the venv."""
|
||||
if decbench is not None:
|
||||
sys.path.insert(0, str(decbench))
|
||||
from decbench.metrics.ged import GED_MAX_NODES, _is_isomorphic
|
||||
from decbench.metrics.vj_ged import vj_ged
|
||||
from decbench.utils.cfg import (
|
||||
_parse_cfgs,
|
||||
canonicalize_infinite_loops,
|
||||
extract_cfgs_from_source,
|
||||
is_degenerate_source_cfg,
|
||||
preprocess_decompiled_c,
|
||||
sanitize_decompiled_c,
|
||||
)
|
||||
|
||||
def ged(source_cfg, decompiled_cfg):
|
||||
if is_degenerate_source_cfg(source_cfg):
|
||||
return None
|
||||
if _is_isomorphic(source_cfg, decompiled_cfg):
|
||||
return 0.0
|
||||
s, d = source_cfg.number_of_nodes(), decompiled_cfg.number_of_nodes()
|
||||
if s > GED_MAX_NODES or d > GED_MAX_NODES:
|
||||
return max(1.0, float(abs(s - d) + abs(source_cfg.number_of_edges() - decompiled_cfg.number_of_edges())))
|
||||
return max(1.0, float(vj_ged(source_cfg, decompiled_cfg)))
|
||||
|
||||
scratch = "/dev/shm" if os.path.isdir("/dev/shm") else None
|
||||
|
||||
def dec_cfg_of(text: str, name: str):
|
||||
marked = f"// Function: {name}\n{sanitize_decompiled_c(text)}"
|
||||
marked = canonicalize_infinite_loops(preprocess_decompiled_c(marked))
|
||||
with tempfile.TemporaryDirectory(dir=scratch) as directory:
|
||||
path = Path(directory) / "rendering.c"
|
||||
path.write_text(marked)
|
||||
return _parse_cfgs(path).get(name)
|
||||
|
||||
return (lambda path: extract_cfgs_from_source(Path(path))), ged, dec_cfg_of
|
||||
|
||||
|
||||
def source_cfgs_by_name(tree: Path, opt: str, project: str, cache: Path, extract) -> dict[str, list]:
|
||||
"""`{function: [(translation unit stem, cfg)]}` over the project's `.i` files; cached per (opt, project).
|
||||
|
||||
The cache is keyed by name alone: delete it when the sources are recompiled.
|
||||
"""
|
||||
pickled = cache / f"{opt}.{project}.pkl"
|
||||
if pickled.exists():
|
||||
return pickle.loads(pickled.read_bytes())
|
||||
by_name: dict[str, list] = {}
|
||||
for source in sorted((tree / opt / project / "compiled").glob("*.i")):
|
||||
try:
|
||||
cfgs = extract(source)
|
||||
except Exception as error: # noqa: BLE001
|
||||
print(f"no source CFGs from {source.name}: {error}", file=sys.stderr)
|
||||
continue
|
||||
for name, graph in cfgs.items():
|
||||
by_name.setdefault(name, []).append((source.stem, graph))
|
||||
cache.mkdir(parents=True, exist_ok=True)
|
||||
pickled.write_bytes(pickle.dumps(by_name))
|
||||
return by_name
|
||||
|
||||
|
||||
def pick_source(candidates: list, binary: str):
|
||||
"""The only definition, else the one in the translation unit named like the binary, else the first."""
|
||||
if len(candidates) == 1:
|
||||
return candidates[0][1]
|
||||
wanted = {binary, binary.split(".so")[0]}
|
||||
for stem, graph in candidates:
|
||||
if stem in wanted:
|
||||
return graph
|
||||
return candidates[0][1]
|
||||
|
||||
|
||||
def score(tree: Path, dump_dir: Path, out: Path, decbench: Path | None, cache: Path | None) -> int:
|
||||
import signal
|
||||
|
||||
extract, ged, dec_cfg_of = decbench_scorer(decbench)
|
||||
cache = cache or out.parent / "srccache"
|
||||
manifest = json.loads((dump_dir / "manifest.json").read_text())
|
||||
|
||||
def alarm(signum, frame):
|
||||
raise Timeout
|
||||
|
||||
signal.signal(signal.SIGALRM, alarm)
|
||||
rows: dict[str, dict] = {}
|
||||
for dumped in sorted(dump_dir.glob("*.json")):
|
||||
if dumped.name == "manifest.json":
|
||||
continue
|
||||
opt, project, binary = dumped.name[:-len(".json")].split(".", 2)
|
||||
cfgs = source_cfgs_by_name(tree, opt, project, cache, extract)
|
||||
for key, record in json.loads(dumped.read_text()).items():
|
||||
code = record.get("code") or ""
|
||||
row = {"gotos": len(re.findall(r"\bgoto\b", code)), "lines": code.count("\n"),
|
||||
"refused": bool(record.get("refused")), "status": record["status"],
|
||||
"loops": len(re.findall(r"for \(; ; \)", code)), "whiles": len(re.findall(r"\bwhile \(", code)),
|
||||
"text": hashlib.sha256(code.encode()).hexdigest()[:16], "ged": None}
|
||||
if record["status"] == "output" and not row["refused"] and record["name"] in cfgs:
|
||||
signal.alarm(GED_SECONDS)
|
||||
try:
|
||||
cfg = dec_cfg_of(code, record["name"])
|
||||
row["ged"] = ged(pick_source(cfgs[record["name"]], binary), cfg) if cfg is not None else None
|
||||
except Timeout:
|
||||
row["ged"] = "err:timeout"
|
||||
except Exception as error: # noqa: BLE001
|
||||
row["ged"] = "err:" + type(error).__name__
|
||||
finally:
|
||||
signal.alarm(0)
|
||||
rows[key] = row
|
||||
print(f"{opt} {project} {binary}: {len(rows)} scored so far", flush=True)
|
||||
meta = {k: manifest[k] for k in ("set", "r2s_sha256", "projects", "opt_levels") if k in manifest}
|
||||
out.write_text(json.dumps({"schema": SCHEMA, **meta, "rows": rows}))
|
||||
return len(rows)
|
||||
|
||||
|
||||
# --- pairing (pure) ------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def load_scores(path: Path) -> tuple[dict, dict]:
|
||||
"""`(rows, meta)` of a scored run; a bare `{key: row}` file (the scratch format) has no meta."""
|
||||
data = json.loads(Path(path).read_text())
|
||||
if isinstance(data.get("rows"), dict) and "schema" in data:
|
||||
return data["rows"], {k: v for k, v in data.items() if k != "rows"}
|
||||
return data, {}
|
||||
|
||||
|
||||
def has_ged(row: dict) -> bool:
|
||||
return isinstance(row.get("ged"), float)
|
||||
|
||||
|
||||
def one_per_function(keys) -> set[str]:
|
||||
"""One key per (opt, project, function): the first binary that holds it, libz before its static copies."""
|
||||
seen, kept = set(), set()
|
||||
for key in sorted(keys, key=lambda k: (0 if "libz" in k else 1, k)):
|
||||
opt, project, _binary, function = key.split("|", 3)
|
||||
if (opt, project, function) not in seen:
|
||||
seen.add((opt, project, function))
|
||||
kept.add(key)
|
||||
return kept
|
||||
|
||||
|
||||
def pair(base: dict, new: dict, unique: bool = False) -> dict:
|
||||
"""The comparison of two scored runs over the functions both scored."""
|
||||
if unique:
|
||||
kept = one_per_function(base)
|
||||
base = {k: v for k, v in base.items() if k in kept}
|
||||
new = {k: v for k, v in new.items() if k in kept}
|
||||
both = [k for k in sorted(base) if k in new and has_ged(base[k]) and has_ged(new[k])]
|
||||
only_base = [k for k in base if has_ged(base[k]) and not (k in new and has_ged(new[k]))]
|
||||
only_new = [k for k in new if has_ged(new[k]) and not (k in base and has_ged(base[k]))]
|
||||
old = {k: base[k]["ged"] for k in both}
|
||||
now = {k: new[k]["ged"] for k in both}
|
||||
shared = [k for k in sorted(base) if k in new]
|
||||
with_text = [k for k in shared if base[k].get("text") and new[k].get("text")]
|
||||
return {
|
||||
"scored": len(both),
|
||||
"lost_scoring": len(only_base),
|
||||
"gained_scoring": len(only_new),
|
||||
"perfect": [sum(1 for k in both if old[k] == 0), sum(1 for k in both if now[k] == 0)],
|
||||
"newly_perfect": sorted(k for k in both if old[k] > 0 and now[k] == 0),
|
||||
"lost_perfect": sorted(k for k in both if old[k] == 0 and now[k] > 0),
|
||||
"mean_ged": [statistics.mean(old.values()) if both else None, statistics.mean(now.values()) if both else None],
|
||||
"total_ged": [sum(old.values()), sum(now.values())],
|
||||
"better": sorted(k for k in both if now[k] < old[k]),
|
||||
"worse": sorted(k for k in both if now[k] > old[k]),
|
||||
"same": sum(1 for k in both if now[k] == old[k]),
|
||||
"gotos_all": [sum(r.get("gotos", 0) for r in base.values()), sum(r.get("gotos", 0) for r in new.values())],
|
||||
"gotos_scored": [sum(base[k].get("gotos", 0) for k in both), sum(new[k].get("gotos", 0) for k in both)],
|
||||
"functions_with_goto": [sum(1 for k in both if base[k].get("gotos", 0) > 0),
|
||||
sum(1 for k in both if new[k].get("gotos", 0) > 0)],
|
||||
"functions": len(shared),
|
||||
"text_changed": sum(1 for k in with_text if base[k]["text"] != new[k]["text"]) if with_text else None,
|
||||
"text_compared": len(with_text),
|
||||
"refused": [sum(1 for r in base.values() if r.get("refused")), sum(1 for r in new.values() if r.get("refused"))],
|
||||
"newly_refused": sorted(k for k in shared if new[k].get("refused") and not base[k].get("refused")),
|
||||
}
|
||||
|
||||
|
||||
def format_report(label: str, report: dict) -> str:
|
||||
mean = report["mean_ged"]
|
||||
lines = [
|
||||
f"{label}: scored {report['scored']} (lost scoring {report['lost_scoring']}, gained {report['gained_scoring']})",
|
||||
f" perfect {report['perfect'][0]} -> {report['perfect'][1]} newly {len(report['newly_perfect'])} "
|
||||
f"lost {len(report['lost_perfect'])}",
|
||||
f" mean GED {'n/a' if mean[0] is None else f'{mean[0]:.2f}'} -> {'n/a' if mean[1] is None else f'{mean[1]:.2f}'} "
|
||||
f"better {len(report['better'])} worse {len(report['worse'])} same {report['same']}",
|
||||
f" gotos all {report['gotos_all'][0]} -> {report['gotos_all'][1]}; scored {report['gotos_scored'][0]} -> "
|
||||
f"{report['gotos_scored'][1]}; scored fns with a goto {report['functions_with_goto'][0]} -> "
|
||||
f"{report['functions_with_goto'][1]}",
|
||||
f" total GED {report['total_ged'][0]:.0f} -> {report['total_ged'][1]:.0f}",
|
||||
]
|
||||
if report["text_changed"] is None:
|
||||
lines.append(f" text changed: n/a (these scores carry no text hash); refused {report['refused'][0]} -> "
|
||||
f"{report['refused'][1]}")
|
||||
else:
|
||||
lines.append(f" text changed {report['text_changed']} of {report['text_compared']} functions in both runs; "
|
||||
f"refused {report['refused'][0]} -> {report['refused'][1]}, newly refused "
|
||||
f"{len(report['newly_refused'])}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# --- command line ----------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def projects_of(args) -> tuple[str, tuple[str, ...]]:
|
||||
if args.project:
|
||||
return "adhoc", tuple(args.project)
|
||||
if args.set is None:
|
||||
raise SystemExit("name a pinned set (--set mining|heldout) or the projects (--project)")
|
||||
if args.set == "heldout":
|
||||
print("held-out set: run it nightly and at an item's exit, and do not choose rules by it", file=sys.stderr)
|
||||
return args.set, SETS[args.set]
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
d = sub.add_parser("dump", help="r2s pddj of every function of the set")
|
||||
d.add_argument("--tree", type=Path, required=True)
|
||||
d.add_argument("--r2s", type=Path, required=True)
|
||||
d.add_argument("--out", type=Path, required=True)
|
||||
d.add_argument("--set", choices=sorted(SETS))
|
||||
d.add_argument("--project", action="append")
|
||||
d.add_argument("--opt-level", action="append")
|
||||
d.add_argument("--decompilers", default=",".join(DECOMPILERS), help="whose tomls name the targets")
|
||||
s = sub.add_parser("score", help="DecBench GED of every dumped rendering")
|
||||
s.add_argument("--tree", type=Path, required=True)
|
||||
s.add_argument("--dump", type=Path, required=True)
|
||||
s.add_argument("--out", type=Path, required=True)
|
||||
s.add_argument("--decbench", type=Path)
|
||||
s.add_argument("--cache", type=Path)
|
||||
q = sub.add_parser("pair", help="compare two scored runs")
|
||||
q.add_argument("base", type=Path)
|
||||
q.add_argument("new", type=Path)
|
||||
q.add_argument("--label", default="")
|
||||
q.add_argument("-u", "--unique", action="store_true", help="one entry per function across binaries")
|
||||
q.add_argument("-v", "--verbose", action="store_true", help="name the worse, lost and newly refused")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
if args.command == "dump":
|
||||
name, projects = projects_of(args)
|
||||
total = dump(args.tree, args.r2s, args.out, projects, tuple(args.opt_level or OPT_LEVELS),
|
||||
tuple(d for d in args.decompilers.split(",") if d), set_name=name)
|
||||
print(f"dumped {total} functions of {name}: {', '.join(projects)}")
|
||||
return 0
|
||||
if args.command == "score":
|
||||
print(f"scored {score(args.tree, args.dump, args.out, args.decbench, args.cache)} functions")
|
||||
return 0
|
||||
|
||||
base, base_meta = load_scores(args.base)
|
||||
new, new_meta = load_scores(args.new)
|
||||
sets = {base_meta.get("set"), new_meta.get("set")} - {None}
|
||||
if len(sets) > 1:
|
||||
print(f"refusing to pair a {base_meta.get('set')} run with a {new_meta.get('set')} run", file=sys.stderr)
|
||||
return 2
|
||||
report = pair(base, new, args.unique)
|
||||
print(format_report(args.label or (next(iter(sets)) if sets else "pair"), report))
|
||||
if args.verbose:
|
||||
for heading, keys in (("worse", report["worse"]), ("lost perfect", report["lost_perfect"]),
|
||||
("newly perfect", report["newly_perfect"]), ("newly refused", report["newly_refused"])):
|
||||
for key in keys:
|
||||
print(f" {heading}: {key}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,284 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The mined-set comparison pairs two runs and reports them; the DecBench scoring is not run here."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
STUB = HERE.parents[1] / "tests" / "equiv" / "testdata" / "stub_r2s.py"
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
import mined_set # noqa: E402
|
||||
|
||||
|
||||
def row(ged, gotos=0, text="a", refused=False, status="output"):
|
||||
return {"ged": ged, "gotos": gotos, "text": text, "refused": refused, "status": status}
|
||||
|
||||
|
||||
BASE = {
|
||||
"O2|zlib|libz|keeps": row(0.0, text="1"),
|
||||
"O2|zlib|libz|gains": row(5.0, gotos=2, text="2"),
|
||||
"O2|zlib|libz|loses": row(0.0, text="3"),
|
||||
"O2|zlib|libz|better": row(4.0, gotos=1, text="4"),
|
||||
"O2|zlib|libz|same": row(2.0, text="5"),
|
||||
"O2|zlib|libz|unscored": row(1.0, text="6"),
|
||||
"O2|zlib|libz|gets_scored": row(None, text="7"),
|
||||
"O2|zlib|libz|dropped": row(9.0, text="8"),
|
||||
}
|
||||
NEW = {
|
||||
"O2|zlib|libz|keeps": row(0.0, text="1"),
|
||||
"O2|zlib|libz|gains": row(0.0, gotos=0, text="2b"),
|
||||
"O2|zlib|libz|loses": row(3.0, gotos=1, text="3b"),
|
||||
"O2|zlib|libz|better": row(2.0, gotos=1, text="4"),
|
||||
"O2|zlib|libz|same": row(2.0, text="5"),
|
||||
"O2|zlib|libz|unscored": row(None, text="6b", refused=True, status="decline"),
|
||||
"O2|zlib|libz|gets_scored": row(7.0, text="7"),
|
||||
}
|
||||
|
||||
|
||||
class Pairing(unittest.TestCase):
|
||||
def test_two_runs_are_paired_over_the_functions_both_scored(self):
|
||||
report = mined_set.pair(BASE, NEW)
|
||||
self.assertEqual(report["scored"], 5)
|
||||
self.assertEqual((report["lost_scoring"], report["gained_scoring"]), (2, 1))
|
||||
self.assertEqual(report["perfect"], [2, 2])
|
||||
self.assertEqual(report["newly_perfect"], ["O2|zlib|libz|gains"])
|
||||
self.assertEqual(report["lost_perfect"], ["O2|zlib|libz|loses"])
|
||||
self.assertEqual(report["mean_ged"], [2.2, 1.4])
|
||||
self.assertEqual(report["total_ged"], [11.0, 7.0])
|
||||
self.assertEqual(report["better"], ["O2|zlib|libz|better", "O2|zlib|libz|gains"])
|
||||
self.assertEqual(report["worse"], ["O2|zlib|libz|loses"])
|
||||
self.assertEqual(report["same"], 2)
|
||||
self.assertEqual(report["gotos_all"], [3, 2])
|
||||
self.assertEqual(report["gotos_scored"], [3, 2])
|
||||
self.assertEqual(report["functions_with_goto"], [2, 2])
|
||||
# Seven functions are in both runs; text differs in gains, loses and unscored.
|
||||
self.assertEqual((report["text_changed"], report["text_compared"]), (3, 7))
|
||||
self.assertEqual(report["refused"], [0, 1])
|
||||
self.assertEqual(report["newly_refused"], ["O2|zlib|libz|unscored"])
|
||||
|
||||
def test_the_report_names_each_figure_the_item_is_graded_by(self):
|
||||
text = mined_set.format_report("mining", mined_set.pair(BASE, NEW))
|
||||
self.assertEqual(text.splitlines(), [
|
||||
"mining: scored 5 (lost scoring 2, gained 1)",
|
||||
" perfect 2 -> 2 newly 1 lost 1",
|
||||
" mean GED 2.20 -> 1.40 better 2 worse 1 same 2",
|
||||
" gotos all 3 -> 2; scored 3 -> 2; scored fns with a goto 2 -> 2",
|
||||
" total GED 11 -> 7",
|
||||
" text changed 3 of 7 functions in both runs; refused 0 -> 1, newly refused 1",
|
||||
])
|
||||
|
||||
def test_scores_without_a_text_hash_say_so_and_an_empty_pairing_does_not_divide(self):
|
||||
legacy = {"O2|a|b|f": {"ged": 3.0, "gotos": 1, "refused": False}}
|
||||
report = mined_set.pair(legacy, legacy)
|
||||
self.assertIsNone(report["text_changed"])
|
||||
self.assertIn("text changed: n/a", mined_set.format_report("x", report))
|
||||
empty = mined_set.pair({}, {})
|
||||
self.assertEqual(empty["mean_ged"], [None, None])
|
||||
self.assertIn("mean GED n/a -> n/a", mined_set.format_report("x", empty))
|
||||
|
||||
def test_unique_keeps_one_entry_per_function_and_prefers_the_library(self):
|
||||
keys = ["O2|zlib|example|crc32", "O2|zlib|libz.so.1.2|crc32", "O2|zlib|example|main",
|
||||
"O2|zlib|minigzip|main", "O2-noinline|zlib|example|main"]
|
||||
self.assertEqual(sorted(mined_set.one_per_function(keys)), [
|
||||
"O2-noinline|zlib|example|main", "O2|zlib|example|main", "O2|zlib|libz.so.1.2|crc32"])
|
||||
rows = {k: row(1.0) for k in keys}
|
||||
self.assertEqual(mined_set.pair(rows, rows, unique=True)["scored"], 3)
|
||||
self.assertEqual(mined_set.pair(rows, rows)["scored"], 5)
|
||||
|
||||
def test_a_mining_run_is_not_paired_with_a_held_out_run(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
def write(name, set_name):
|
||||
path = Path(directory) / name
|
||||
path.write_text(json.dumps({"schema": 1, "set": set_name, "rows": BASE}))
|
||||
return str(path)
|
||||
|
||||
mining, held_out = write("a.json", "mining"), write("b.json", "heldout")
|
||||
err = io.StringIO()
|
||||
with contextlib.redirect_stderr(err), contextlib.redirect_stdout(io.StringIO()):
|
||||
self.assertEqual(mined_set.main(["pair", mining, held_out]), 2)
|
||||
self.assertIn("refusing to pair a mining run with a heldout run", err.getvalue())
|
||||
out = io.StringIO()
|
||||
with contextlib.redirect_stdout(out):
|
||||
self.assertEqual(mined_set.main(["pair", mining, mining]), 0)
|
||||
self.assertTrue(out.getvalue().startswith("mining: scored 7"))
|
||||
# The scratch format, a bare {key: row} file, still pairs.
|
||||
bare = Path(directory) / "bare.json"
|
||||
bare.write_text(json.dumps(BASE))
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.assertEqual(mined_set.main(["pair", str(bare), str(bare)]), 0)
|
||||
|
||||
|
||||
class Sets(unittest.TestCase):
|
||||
def test_mining_and_held_out_split_the_sailr_set(self):
|
||||
self.assertEqual(mined_set.MINING, ("bzip2", "zlib", "grep"))
|
||||
self.assertFalse(set(mined_set.MINING) & set(mined_set.HELD_OUT))
|
||||
self.assertEqual(sorted(mined_set.MINING + mined_set.HELD_OUT), sorted(
|
||||
["bash", "bzip2", "diffutils", "findutils", "grep", "gzip", "libexpat", "zlib"]))
|
||||
self.assertEqual(mined_set.OPT_LEVELS, ("O2", "O2-noinline"))
|
||||
|
||||
|
||||
class Dump(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.dir = tempfile.TemporaryDirectory()
|
||||
self.tree = Path(self.dir.name) / "run"
|
||||
project = self.tree / "O2" / "zlib"
|
||||
(project / "decompiled").mkdir(parents=True)
|
||||
(project / "stripped").mkdir()
|
||||
# The file is libz.so.1.2.13, the driver's name for it is the stem libz.so.1.2.
|
||||
(project / "stripped" / "libz.so.1.2.13").write_bytes(b"\x7fELF")
|
||||
(project / "decompiled" / "kuna_libz.so.1.2.toml").write_text(
|
||||
'binary = "libz.so.1.2"\n["functions.crc32"]\naddress = "0x401150"\n'
|
||||
'["functions.helper"]\naddress = "0x401160"\n')
|
||||
(project / "decompiled" / "angr_libz.so.1.2.toml").write_text(
|
||||
'binary = "libz.so.1.2"\n["functions.helper"]\naddress = "0x401160"\n'
|
||||
'["functions.helper_2"]\naddress = "0x4011f0"\n')
|
||||
self.out = Path(self.dir.name) / "dump"
|
||||
|
||||
def tearDown(self):
|
||||
self.dir.cleanup()
|
||||
|
||||
def dump(self, mode="minimal"):
|
||||
env = {"STUB_R2S_MODE": mode, "STUB_R2S_FAULTS": "", "STUB_R2S_STATE": self.dir.name}
|
||||
with mock.patch.dict(os.environ, env), contextlib.redirect_stdout(io.StringIO()):
|
||||
return mined_set.dump(self.tree, STUB, self.out, ("zlib",), ("O2",), set_name="mining",
|
||||
function_timeout=30.0)
|
||||
|
||||
def test_targets_are_the_union_of_the_decompilers_tomls(self):
|
||||
found = mined_set.function_targets(self.tree / "O2" / "zlib")
|
||||
self.assertEqual(found, {"libz.so.1.2": {0x401150: "crc32", 0x401160: "helper", 0x4011f0: "helper_2"}})
|
||||
|
||||
def test_a_name_two_addresses_share_is_told_apart_by_address(self):
|
||||
keys = mined_set.row_keys("O2", "grep", "grep", {0x2000: "usage", 0x1000: "usage", 0x3000: "main"})
|
||||
self.assertEqual(keys, {0x1000: "O2|grep|grep|usage", 0x2000: "O2|grep|grep|usage@0x2000",
|
||||
0x3000: "O2|grep|grep|main"})
|
||||
|
||||
def test_every_function_is_dumped_under_its_dwarf_name(self):
|
||||
self.assertEqual(self.dump(), 3)
|
||||
rows = json.loads((self.out / "O2.zlib.libz.so.1.2.json").read_text())
|
||||
self.assertEqual(sorted(rows), ["O2|zlib|libz.so.1.2|crc32", "O2|zlib|libz.so.1.2|helper",
|
||||
"O2|zlib|libz.so.1.2|helper_2"])
|
||||
crc = rows["O2|zlib|libz.so.1.2|crc32"]
|
||||
self.assertEqual((crc["status"], crc["refused"], crc["addr"]), ("output", False, "0x401150"))
|
||||
self.assertIn("void crc32(void)", crc["code"])
|
||||
self.assertNotIn("fcn_", crc["code"])
|
||||
manifest = json.loads((self.out / "manifest.json").read_text())
|
||||
self.assertEqual((manifest["set"], manifest["projects"], manifest["functions"]), ("mining", ["zlib"], 3))
|
||||
self.assertEqual(len(manifest["r2s_sha256"]), 64)
|
||||
|
||||
def test_a_refusal_is_dumped_as_a_refusal_with_no_code(self):
|
||||
self.dump(mode="refuse")
|
||||
rows = json.loads((self.out / "O2.zlib.libz.so.1.2.json").read_text())
|
||||
refused = rows["O2|zlib|libz.so.1.2|crc32"]
|
||||
self.assertEqual((refused["status"], refused["refused"]), ("decline", True))
|
||||
self.assertNotIn("code", refused)
|
||||
self.assertEqual(refused["cause"], "refused: stub refuses")
|
||||
|
||||
|
||||
class Graph:
|
||||
"""Stands in for a networkx CFG: a shape (`kind`) and a size."""
|
||||
|
||||
def __init__(self, kind: str, nodes: int):
|
||||
self.kind, self.nodes = kind, nodes
|
||||
|
||||
def number_of_nodes(self):
|
||||
return self.nodes
|
||||
|
||||
def number_of_edges(self):
|
||||
return self.nodes
|
||||
|
||||
|
||||
class Score(unittest.TestCase):
|
||||
"""`score` walks a dump, finds each source CFG, and writes one row per function.
|
||||
|
||||
DecBench is replaced by stand-ins that are isomorphic by `kind` and otherwise a distance of the node
|
||||
count, so this checks the walk, the pick of the source, the rows and their pairing; the GED itself is
|
||||
DecBench's.
|
||||
"""
|
||||
|
||||
def stand_ins(self):
|
||||
def module(name, **members):
|
||||
mod = types.ModuleType(name)
|
||||
mod.__dict__.update(members)
|
||||
return mod
|
||||
|
||||
parsed = lambda path: { # noqa: E731
|
||||
name: Graph("loop" if "while" in text else "line", text.count(";") + 1)
|
||||
for name, text in re.findall(r"// Function: (\w+)\n(.*?)(?=// Function:|\Z)", Path(path).read_text(), re.S)}
|
||||
cfg = module(
|
||||
"decbench.utils.cfg", _parse_cfgs=parsed, sanitize_decompiled_c=lambda t: t,
|
||||
preprocess_decompiled_c=lambda t: t, canonicalize_infinite_loops=lambda t: t,
|
||||
is_degenerate_source_cfg=lambda g: g.kind == "degenerate",
|
||||
extract_cfgs_from_source=lambda path: {"crc32": Graph("loop", 2), "tiny": Graph("degenerate", 1)}
|
||||
if path.stem == "crc32" else {"crc32": Graph("line", 99), "helper": Graph("line", 2)})
|
||||
return {
|
||||
"decbench": module("decbench"), "decbench.utils": module("decbench.utils"),
|
||||
"decbench.utils.cfg": cfg, "decbench.metrics": module("decbench.metrics"),
|
||||
"decbench.metrics.ged": module("decbench.metrics.ged", GED_MAX_NODES=200,
|
||||
_is_isomorphic=lambda a, b: a.kind == b.kind and a.nodes == b.nodes),
|
||||
"decbench.metrics.vj_ged": module("decbench.metrics.vj_ged",
|
||||
vj_ged=lambda a, b: abs(a.nodes - b.nodes) + (a.kind != b.kind)),
|
||||
}
|
||||
|
||||
def test_each_dumped_function_gets_a_row_and_two_runs_pair(self):
|
||||
with tempfile.TemporaryDirectory() as directory, mock.patch.dict(sys.modules, self.stand_ins()):
|
||||
root = Path(directory)
|
||||
compiled = root / "run" / "O2" / "zlib" / "compiled"
|
||||
compiled.mkdir(parents=True)
|
||||
for name in ("crc32", "other"):
|
||||
(compiled / f"{name}.i").write_text("")
|
||||
|
||||
def dump(name, helper_code, refused=False):
|
||||
folder = root / name
|
||||
folder.mkdir()
|
||||
rows = {
|
||||
"O2|zlib|libz.so.1.2|crc32": {"name": "crc32", "addr": "0x1", "status": "output",
|
||||
"refused": False, "code": "while (1) { goto out; }\n"},
|
||||
"O2|zlib|libz.so.1.2|helper": {"name": "helper", "addr": "0x2", "status": "output",
|
||||
"refused": False, "code": helper_code},
|
||||
"O2|zlib|libz.so.1.2|tiny": {"name": "tiny", "addr": "0x3", "status": "output",
|
||||
"refused": False, "code": "x;\n"},
|
||||
"O2|zlib|libz.so.1.2|nosource": {"name": "nosource", "addr": "0x4", "status": "output",
|
||||
"refused": False, "code": "y;\n"},
|
||||
}
|
||||
if refused:
|
||||
rows["O2|zlib|libz.so.1.2|helper"] = {"name": "helper", "addr": "0x2", "status": "decline",
|
||||
"refused": True, "cause": "refused: no"}
|
||||
(folder / "O2.zlib.libz.so.1.2.json").write_text(json.dumps(rows))
|
||||
(folder / "manifest.json").write_text(json.dumps({"set": "mining", "r2s_sha256": name}))
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
mined_set.score(root / "run", folder, root / f"{name}.json", None, root / "cache")
|
||||
return mined_set.load_scores(root / f"{name}.json")
|
||||
|
||||
base, meta = dump("base", "a; b;\n")
|
||||
self.assertEqual(meta["set"], "mining")
|
||||
crc = base["O2|zlib|libz.so.1.2|crc32"]
|
||||
# Two definitions: libz.so.1.2 has no translation unit of its name, so the first, crc32.i, is the one.
|
||||
self.assertEqual((crc["ged"], crc["gotos"], crc["loops"], crc["whiles"]), (0.0, 1, 0, 1))
|
||||
self.assertEqual(base["O2|zlib|libz.so.1.2|helper"]["ged"], 1.0) # 3 nodes against 2: distance 1
|
||||
self.assertIsNone(base["O2|zlib|libz.so.1.2|tiny"]["ged"]) # a degenerate source scores nothing
|
||||
self.assertIsNone(base["O2|zlib|libz.so.1.2|nosource"]["ged"])
|
||||
new, _ = dump("new", "a;\n")
|
||||
self.assertEqual(new["O2|zlib|libz.so.1.2|helper"]["ged"], 0.0)
|
||||
report = mined_set.pair(base, new)
|
||||
self.assertEqual((report["scored"], report["perfect"], report["text_changed"]), (2, [1, 2], 1))
|
||||
gone, _ = dump("gone", "", refused=True)
|
||||
report = mined_set.pair(base, gone)
|
||||
self.assertEqual((report["lost_scoring"], report["newly_refused"]),
|
||||
(1, ["O2|zlib|libz.so.1.2|helper"]))
|
||||
self.assertTrue((root / "cache" / "O2.zlib.pkl").exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user