mirror of
https://github.com/esengine/DeepSeek-Reasonix.git
synced 2026-11-07 10:45:59 -10:00
esengine
e3031b5fd7
Both have failed every run of CI's windows-latest job since the corpus landed, and neither is a flake. gen-topo-cycle timed each call with signal.alarm. Windows Python has no SIGALRM, so the grader raised while installing the handler, before it graded anything. A daemon worker thread keeps the semantics exactly — _Timeout on a hang, the original exception re-raised, so the cyclic case still reads ValueError — and cannot hold the interpreter open at exit. gen-csv-quotes compared cli.py's stdout against an expected string. Windows Python writes CRLF in text mode, so that comparison could never hold; the carriage returns come off first, because the grader checks the fields and not the host's line endings. Its mktemp template also read TMPDIR, which under Git Bash is a Windows path — the rest of the corpus asks for a bare mktemp -d, and now so does this one.
Generated training corpus
Tasks here exist to produce training data, not to measure anything. The e2e suite is the instrument: mixing generated tasks into it would change what a cross-version comparison means, so the two stay apart.
Bundle contract
tasks/<id>/
task.toml prompt, class, timeout_sec
verify.sh the grader; run from the workspace root
workdir/ the seed the agent starts from
solution/ the reference fix, overlaid on workdir/
Both halves of the grader are enforced by cmd/e2ebench/corpus_test.go:
TestSolvableCorpusSeedsMustNotGradeClean— verify.sh must fail on the pristine seed, or the task scores the same whether it was solved or ignored.TestCorpusGradersPassTheReferenceSolution— verify.sh must pass with solution/ applied, or no attempt can ever satisfy it.solution/is optional in the hand-authored corpora and required here.
A generated task that fails either gate is deleted, never patched into the baseline.
Regenerating
./generate.sh <count>
Candidates are produced one per specification line in specs.txt, validated by
the gates above, and only survivors are kept.