esengine 8a8f6e629e feat(bench): stratified e2e task corpus, first tranche ...

The committed suite measured toy tasks with increasing precision — the
planner A/B proved it: none of the five tasks ever routed past
ExecutorOnly. This starts the workload-stratified corpus the readouts
were built for: 16 new tasks across eight classes (atomic-bugfix,
repo-exploration, multi-file-bugfix, refactor, failing-test-diagnosis,
api-integration, ambiguous, long-horizon), two per class, with the
target stratification documented in the README. Multi-file, refactor
and long-horizon prompts are phrased to engage the planner gate
naturally; atomic fixes stay short and anchored so they keep routing
ExecutorOnly.

Every grader was validated in both directions before commit: verify.sh
fails on the pristine seed and passes on a reference solution.
failing-test-diagnosis graders checksum the test files so fixing the
test instead of the source cannot pass. Python graders pin a throwaway
PYTHONPYCACHEPREFIX — macOS system Python caches bytecode centrally by
absolute path, and a same-size same-second edit would otherwise run
stale bytecode while tracebacks display the new source.

Existing tasks join the taxonomy (fix-add-bug -> atomic-bugfix,
compaction -> repo-exploration); fizzbuzz/palindrome/subagent-delegation
stay as legacy smoke tasks.

2026-08-07 22:23:03 -07:00

..

quota.py

feat(bench): stratified e2e task corpus, first tranche

2026-08-07 22:23:03 -07:00

uploader.py

feat(bench): stratified e2e task corpus, first tranche

2026-08-07 22:23:03 -07:00