YHHandreasonix f38ea2ca58 feat(ci): e2e benchmark bot triggered by a PR comment (#2574) ...

* feat(ci): e2e benchmark bot triggered by a PR comment

Comment "/e2e" on a pull request to run the committed task suite against
the real provider and get an accuracy / cache-hit / token / cost report
posted back. Gated to OWNER/MEMBER/COLLABORATOR because it checks out
PR-head code and runs it with the provider API key.

- reasonix run --metrics writes a JSON token/cache/cost summary of a run.
- cmd/e2ebench copies each task's seed workdir to a temp dir, runs the
  agent there via the built binary, then grades with verify.sh (dropped
  in only after the run so the answer key isn't readable), and aggregates
  the per-task metrics into a markdown report.
- benchmarks/e2e seeds three tasks (fizzbuzz, fix-add-bug, palindrome),
  each graded by a verify.sh that exits non-zero on failure.
- .github/workflows/e2e-bot.yml wires the comment trigger, writes the
  provider config, builds, runs the suite under a token budget, and posts
  the report.

Needs a DEEPSEEK_API_KEY repo secret; model/base URL are overridable via
repo variables.

* feat(ci): add diff-driven test-gen mode to the e2e bot

Comment "/e2e diff" on a PR to have the agent, on the PR branch, write
tests covering what the PR changed and run them — graded by the repo's
own tests, with the accuracy/cache/token/cost report posted back.

- e2ebench -mode diff: derives the changed Go source files and packages
  from base...HEAD, prompts the agent to add tests for them (not touch
  source), then grades: pass = the agent added >=1 new test function AND
  `go test` on the affected packages is green. A no-op can't pass since
  the suite was already green; non-test source edits by the agent are
  surfaced as a warning.
- run-metrics is reused for the cost/token/cache numbers.
- the workflow branches on "/e2e diff", computes the base via merge-base,
  and writes the provider config to the repo root too (project config
  wins over user config, and diff mode runs the agent in the repo root).

Plain "/e2e" still runs the fixed suite.

* feat(ci): differential check for diff test-gen; match default model

The diff grader now reverts the PR's source to its pre-change state and
re-runs the generated tests: they must FAIL on the old code, proving they
capture the change rather than asserting behavior that already held. Pass
now requires added tests + green on HEAD + failing on base.

Also point the workflow's default model/pricing at deepseek-v4-flash to
match the project's own provider config.

---------

Co-authored-by: reasonix <[email protected]>

2026-06-01 05:38:29 -07:00

..

calc.py

feat(ci): e2e benchmark bot triggered by a PR comment (#2574)

2026-06-01 05:38:29 -07:00