mirror of
https://github.com/esengine/DeepSeek-Reasonix.git
synced 2026-11-07 10:45:59 -10:00
YHH
7703b22a12
* feat(bench): SWE-bench primitives verified against the real harness Image naming, conda activation, prompt construction, predictions encoding and grader-report parsing, each pinned by a test against output captured from swebench 4.1.0 on a real gold run. The orchestration that drives containers lands next; these are the pieces whose shape could only be learned by running the harness, not by reading it. * feat(bench): run SWE-bench inside the official instance images The agent now runs where the harnesses it is compared against run: inside the per-instance evaluation container, against /testbed, with the repo's own tests available to it. Anything less measures a different task. The container is the isolation boundary, so the in-container config turns the bash sandbox off — Reasonix otherwise refuses to run bash at all when it cannot find bubblewrap, which the official images do not ship, and an agent that cannot run tests is not doing SWE-bench. Nothing else deviates from the shipped defaults: the provider, model and context window are what a user gets on install, because a hand-tuned config would make the published number unreproducible. Credentials stream into the container over stdin rather than through argv or an image layer, and the issue text is fully shell-quoted before it becomes one argv element — a problem statement is untrusted input that routinely contains backticks and $(). Resolution is never decided here; predictions go to the official grader and its report is parsed back into the published failure taxonomy. * feat(bench): publish the SWE-bench Verified subset and how it was chosen The subset is 50 of the 500 Verified instances, and the script that produced it ships alongside it. Selection is deterministic — largest-remainder apportionment across repos, then evenly spaced picks within each repo ordered by difficulty — so re-running it against the same dataset revision reproduces the file byte for byte. A sample nobody can regenerate is a sample that can be quietly retuned after seeing results. Composition tracks the real benchmark: django 23 of 50 against 46.2% of the full set, sympy 8 against 15.0%, and difficulty lands at 18/28/4 against the full set's 38.8/52.2/8.4%. Nothing is excluded, including the psf/requests instance whose test suite makes live network calls; if it exhausts the grader timeout it is published as eval_timeout, because dropping the instances a harness handles badly is how a benchmark stops meaning anything. * feat(metrics): keep accounting when the agent is killed, and say where it came from A benchmark run that times out currently reports nothing: the metrics file is written once, after the controller returns, so a killed agent leaves no trace of the fifty tool calls it made. In the last SWE-bench run one solved instance landed in the totals as a free solve, quietly making every per-task figure cheaper than the truth. The sink now publishes a throttled snapshot to a `.partial` sidecar as the run proceeds, through the repository's existing atomic write so a reader only ever sees a whole file. The final record is marked complete and retires the sidecar, so the two can never both be counted. The harness prefers the final record and falls back to the snapshot, reporting recovered numbers as lower bounds rather than as fact. Usage is also split by origin. Steps counts every billed model call — executor, planner, sub-agents, compaction, classifier, router — which is why published rows read above a max_steps budget the main loop never exhausted. An origin the build does not recognise keeps its own key rather than being dropped, because a total that cannot reconcile is worse than one with an unfamiliar row. Snapshotting made an existing race consequential: Emit is serialized by the session's event.Sync wrapper, but the run command read the same struct directly while background jobs could still emit into it. The sink now owns a mutex and hands out deep copies. The race test needs `go test -race`, which cannot run on a host without cgo; CI's race job is what covers it.
89 lines
3.1 KiB
Python
89 lines
3.1 KiB
Python
#!/usr/bin/env python3
|
|
"""Select the published SWE-bench Verified subset.
|
|
|
|
Deterministic by construction: no random seed, no hand-picking. Re-running this
|
|
against the same dataset revision reproduces subset.json byte for byte, so the
|
|
sample cannot be quietly tuned after seeing results.
|
|
|
|
Rules:
|
|
1. Repos get slots in proportion to their share of the full 500, largest
|
|
remainder first, so the sample keeps the benchmark's real composition
|
|
(django is ~46% of SWE-bench Verified and stays ~46% here).
|
|
2. Within a repo, instances are ordered by (difficulty, instance_id) and
|
|
picked at evenly spaced indices, which spreads the sample across the
|
|
repo's own difficulty mix instead of clustering on easy ones.
|
|
|
|
Nothing is excluded. psf/requests instances exercise a test suite that makes
|
|
live network calls and can exhaust the grader timeout; such a run is reported
|
|
as eval_timeout rather than dropped, because silently removing the instances a
|
|
harness handles badly is how a benchmark stops meaning anything.
|
|
"""
|
|
|
|
import json
|
|
import sys
|
|
from datasets import load_dataset
|
|
|
|
TOTAL = 50
|
|
DIFFICULTY_ORDER = {"<15 min fix": 0, "15 min - 1 hour": 1, "1-4 hours": 2, ">4 hours": 3}
|
|
|
|
|
|
def allocate(counts, total):
|
|
"""Largest-remainder apportionment of `total` slots across repo counts."""
|
|
population = sum(counts.values())
|
|
exact = {repo: n * total / population for repo, n in counts.items()}
|
|
floors = {repo: int(v) for repo, v in exact.items()}
|
|
remaining = total - sum(floors.values())
|
|
order = sorted(exact, key=lambda r: (-(exact[r] - floors[r]), r))
|
|
for repo in order[:remaining]:
|
|
floors[repo] += 1
|
|
return floors
|
|
|
|
|
|
def evenly_spaced(items, k):
|
|
if k <= 0:
|
|
return []
|
|
if k >= len(items):
|
|
return items
|
|
return [items[(2 * i + 1) * len(items) // (2 * k)] for i in range(k)]
|
|
|
|
|
|
def main():
|
|
ds = load_dataset("princeton-nlp/SWE-bench_Verified", split="test")
|
|
by_repo = {}
|
|
for row in ds:
|
|
by_repo.setdefault(row["repo"], []).append(row)
|
|
|
|
slots = allocate({repo: len(rows) for repo, rows in by_repo.items()}, TOTAL)
|
|
|
|
selected = []
|
|
for repo in sorted(by_repo):
|
|
rows = sorted(
|
|
by_repo[repo],
|
|
key=lambda r: (DIFFICULTY_ORDER.get(r["difficulty"], 9), r["instance_id"]),
|
|
)
|
|
selected.extend(evenly_spaced(rows, slots[repo]))
|
|
|
|
selected.sort(key=lambda r: r["instance_id"])
|
|
out = [
|
|
{
|
|
"instance_id": r["instance_id"],
|
|
"repo": r["repo"],
|
|
"base_commit": r["base_commit"],
|
|
"problem_statement": r["problem_statement"],
|
|
"difficulty": r["difficulty"],
|
|
}
|
|
for r in selected
|
|
]
|
|
with open("benchmarks/swebench/subset.json", "w", encoding="utf-8") as f:
|
|
json.dump(out, f, indent=2, ensure_ascii=False)
|
|
f.write("\n")
|
|
|
|
print(f"selected {len(out)} of {len(ds)}", file=sys.stderr)
|
|
for repo in sorted(by_repo):
|
|
share = 100 * len(by_repo[repo]) / len(ds)
|
|
print(f" {slots[repo]:2d} {repo:32s} (full set {share:4.1f}%)", file=sys.stderr)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|