Files
zero/internal/perfbench/testdata/refactor/main.go
Vasanth T 727ad4d321 perf(turn-bench): Phase 0 — strengthen oracles so pass rate can't be misread as correctness (#712)
* perf(turn-bench): Phase 0 — strengthen oracles so pass rate can't be misread as correctness

The baseline manifest's pass/fail contract had three holes that let a weak
or no-op agent read as a pass:

  - edit grep oracles were substring checks: a reworded line (e.g.
    fmt.Println("debug: starting") -> fmt.Println("starting up")) passed
    `! grep 'debug: starting'`.
  - refactor lived in the build tier with a non-positive `go build ./...`,
    so a no-op refactor passed.
  - the 24 read-only tasks (nav/longproc/longctx/parallel) had no oracle;
    a zero-exit only proved the turn ran.

This makes the tiers honest:

  - edit/refactor now carry a stamped oracle_test.go compiled by
    `go test ./...` — the Go compiler is the structural verifier, so a
    no-op refactor, a missing field, or a reworded-but-not-removed line
    fails to compile/run. The test is stamped into the fixture copy AFTER
    the agent run, so it can't interfere with the agent's own build or be
    pre-seen.
  - nav gains an answer-oracle: the harness captures the agent's final
    answer (.zero-answer.txt) and greps it for determinable facts. Count
    tasks (nav-04/05/08) use a structured `count: N` token so the oracle is
    tight rather than a loose prose-contains on "0".
  - refactor graduates from build to correctness (its oracle is now
    positive); nav graduates from latency-only to correctness. The build
    tier is empty (buildOnlyClasses: []).
  - longproc/longctx/parallel stay permanently latency-only — there is no
    deterministic oracle for "report whether the build succeeds" or
    "summarize a large file"; documented in MANIFEST.md.

Tier counts: correctness 34 (edit 10 + fix 8 + nav 10 + refactor 6),
build 0, latency 14. Total 48 unchanged.

TurnSchemaVersion 2 -> 3: the result shape is unchanged (only class-list
membership shifts) but correctnessPassRate now means something strictly
stronger, so the bump prevents a v2->v2 cross-version comparison from
misreading the jump as a model improvement.

Prerequisite fix: the mutating fixtures had no go.mod, so `go test ./...`
in the temp copy failed with "cannot find main module". Added go.mod per
fixture (edit/fix/refactor/nav) and moved copyFixture's temp dir one level
below the system temp root — Go ignores go.mod in direct children of the
temp dir (a hijack guard), which would have broken every compiler-backed
oracle. This also fixes the pre-existing fix/refactor oracles, which were
silently broken because the test suite used fake runners that never ran
go test.

Tests: stub-binary gating tests prove each oracle fails the wrong thing
(no-op refactor, missing field, wrong nav answer, reworded debug print) and
passes the right thing (real refactor, added field, correct answer); an
edit-01 test proves the scoped negative grep doesn't false-fail a correct
rename that leaves the doc comment. Aggregation, schema, and unit tests
cover the v3 tier model and the answer-capture helper.

* Address CodeRabbit review on PR #712: tighten oracles + isolation

- nav count oracles (nav-01/04/05/08): anchored line-exact
  `^count: N$` with the fixture's exact expected count, so a substring
  like "the count: 0" or a wrong count like "count: 3" fails instead
  of rubber-stamping. nav-01 added to the structured-count contract.
- stampOracleAndAnswer now errors when a task has an oracle or
  verification but no workspace fixture, so verification can't run in
  the caller's cwd with the oracle never compiled (false-pass closed).
- copyFixture uses a unique 0700 os.MkdirTemp parent under the temp
  root and nests the copy beneath it (grandchild of temp root), so the
  Go go.mod hijack guard is satisfied and cleanup removes the whole
  parent. Avoids a predictable shared world-readable dir.
- Rejection tests require a non-empty VerifyErr via a shared
  assertVerifyFailed helper, proving the failure came from
  verification rather than a harness error or a silent non-pass.
- Unify streamJSONExitCode + streamJSONFinalText behind one
  parseStreamJSON parser (behavior-preserving; scanner limits and
  malformed-event handling now live in one place).

* Address gnanam1990 review on PR #712: close count=0 gameability + stronger oracles

- nav count tasks (nav-04/05/08) had a count=0 ground truth, so an agent
  that never opened the workspace and always emitted "count: 0" passed
  three correctness tasks -- the exact pass-rate-misread-as-correctness
  failure this PR closes. nav-04 now has a real TestGreet (count 1) and
  nav-05 a real TODO (count 1); nav-01's file count moves to 5. nav-08
  stays at 0 (a real third-party import would need a network fetch and
  break the offline suite) but its oracle now requires the answer to name
  both stdlib imports (fmt from main.go, testing from main_test.go), so a
  blind count:0 or a main.go-only read fails. nav-08's prompt now asks the
  agent to enumerate and classify every import, so naming the stdlib
  imports is part of a correct answer, not a false-fail.
- refactor-05: stamp a behavioral oracle (TestGreetWrapped) so deleting
  both Wrapper and greetWrapped, or stubbing the caller to return "",
  fails to compile/run. The negative grep alone passed that wrong edit.
- refactor-04: replace the brittle exact-text shell grep with a stamped
  file-reading oracle using a spacing-tolerant regex, so a bare
  map[string]int decl under any spacing/:= variant can't dodge it, and
  the Stats type must be referenced in main.go.
- edit-05: also assert the "debug: starting" message text is absent, so
  a respell onto fmt.Printf / backticks / log.Println / println that
  still prints the message fails (the fmt.Println(" check missed those).
- v3 doc comments on ClassSummary / TurnBenchResult /
  FormatTurnBenchSummary now describe the v3 tiers (correctness =
  edit/fix/nav/refactor, build empty, latency = longproc/longctx/
  parallel) instead of the stale v2 model.
- stampOracleAndAnswer failure now mutates the existing outcome instead
  of returning a fresh one, so the already-parsed trace and trace issue
  survive a stamp error (attribution is preserved, pass/fail unchanged).
- Schema comment provenance: "PR #701" -> neutral "Phase 0 oracle
  hardening" (this PR is #712).
- New gating + satisfiable stub tests for refactor-04/05 and edit-05, and
  count-oracle zero-guess / inspection-proof tests for nav-04/05/08.

* Fix macOS CI: copyFixture isolation test was slash-sensitive

TestCopyFixtureIsolatesSourceFromMutation compared filepath.Dir(parent)
against os.TempDir() verbatim. On macOS $TMPDIR ends in a trailing slash
("/var/folders/.../T/") while filepath.Dir strips it, so the equality
false-failed even though the parent is genuinely a direct child of the
temp root. Linux (/tmp, no slash) and Windows (no slash in this Go
build) matched by accident.

The production copyFixture nesting is correct on all platforms (the
grandchild copy sits one level below the temp root, satisfying Go's
go.mod hijack guard); only the assertion was slash-sensitive. Normalize
both sides with filepath.Clean.

* Close nav-04/05 blind-count-one gameability (anandh8x review)

nav-04 and nav-05 had a count-only oracle (anchored '^count:1$'). Once the
ground truth became 1, an agent that never inspects the fixture and always
emits 'count: 1' passed both — the blind-count-one surface the prior
non-zero-ground-truth fix opened. Pair the count with a workspace-derived
named fact that proves inspection, mirroring nav-08's fmt+testing:

- nav-04 now also requires 'TestGreet' (the one test function in
  main_test.go) — a specific, non-generic name a blind agent can't produce
  without reading main.go to learn a greet function exists. Fully closes
  the blind-count-one surface.
- nav-05 now also requires 'main.go' (the file holding the one TODO).
  main.go is the conventional entrypoint and the most guessable filename, so
  this reduces rather than fully closes the surface — a blind 'count: 1, in
  main.go' can still pass. Accepted for Phase 0: the blind floor drops below
  100% (a bare 'count: 1' fails), so a model still moves the pass rate by
  inspecting. MANIFEST.md documents this residual honestly.

Two gating tests per task: RejectsBlindCountOne (bare 'count: 1' now fails)
and, for nav-05, RejectsSubstring (main.go present but unanchored count
still fails — parity with nav-04). Existing accept tests still pass (their
answers already name TestGreet / main.go).

Doc nits also fixed: MANIFEST.md #701 -> #712; refactor-05 removed from the
'no oracleTest' inventory (it now has TestGreetWrapped); the nav-01 named-
fact prose corrected to the three files the oracle actually anchors on;
test comment #701 -> #712.
2026-07-17 17:32:45 +05:30

62 lines
1.6 KiB
Go

// Package refactor is the starting workspace for the multi-file refactor
// benchmark tasks. It is intentionally a little messy (duplicated logic, a
// bare map type, a thin wrapper) so the refactor prompts have real work to do.
// It builds cleanly at all times; the tasks keep it building.
package refactor
import "fmt"
// Config holds demo settings.
type Config struct {
Name string
Port int
}
// GreetFromConfig returns a greeting using the config's name.
func GreetFromConfig(c Config) string {
return fmt.Sprintf("hello, %s", c.Name)
}
// GreetByName returns a greeting using a bare name. BUG-note: this duplicates
// GreetFromConfig's formatting — refactor-01 extracts a shared helper.
func GreetByName(name string) string {
return fmt.Sprintf("hello, %s", name)
}
// Wrapper is a thin wrapper around GreetByName — refactor-05 inlines it.
func Wrapper(name string) string {
return GreetByName(name)
}
// greetWrapped is Wrapper's single caller — refactor-05 inlines Wrapper here
// and removes the Wrapper function.
func greetWrapped(name string) string {
return Wrapper(name)
}
// stats is a bare map used across the file — refactor-04 introduces a named type.
var stats = map[string]int{}
// Record bumps a stat by name.
func Record(name string) {
stats[name]++
}
// Lookup returns a stat by name.
func Lookup(name string) int {
return stats[name]
}
// load failed wrapped error — refactor-06 consolidates these.
func firstError() error {
return fmt.Errorf("load failed: missing input")
}
func secondError() error {
return fmt.Errorf("load failed: bad format")
}
func buildError(ctx string) error {
return fmt.Errorf("load failed: %s", ctx)
}