esengine 19107fcdeb feat(benchmarks): a generated corpus, held to both halves of its grader ...

Failing the pristine seed is only half of an authored task. A grader that can
never pass is indistinguishable from one with no solution: every attempt scores
zero, and a corpus of those teaches that nothing is ever accepted. Tasks that
ship a reference solution are now checked against it, and a generated task that
cannot satisfy its own grader is deleted rather than patched.

The generated tasks live apart from benchmarks/e2e on purpose. That suite is
the instrument every cross-version comparison is read against; mixing new
tasks into it would change what those numbers mean.

2026-08-25 10:05:52 -07:00

..

tasks

feat(benchmarks): a generated corpus, held to both halves of its grader

2026-08-25 10:05:52 -07:00