Address CodeRabbit review on 2bfb3406: omitempty + crash visibility

Three findings from CodeRabbit's review of the 3-tier pass-rate fix:

- BuildPassRate dropped its omitempty so an explicit 0.0 (all refactor
  tasks fail to compile) is no longer indistinguishable from 'no build
  tasks ran' in the JSON. Matches CorrectnessPassRate's always-serialize
  behavior, which is the whole point of the tier split: the report
  cannot be misread.
- A crashed sample (outcome.Err) now emits a Warning mirroring the
  TraceIssue path, so a run that died every iteration can't look like a
  normal benchmark with fewer measurements instead of a failure.
- cmd.Run()'s returned error is preserved on the no-run_end path: when
  stderr is empty and Run failed (binary missing, exec permission
  denied), the actual OS error is reported instead of collapsing to the
  generic 'missing terminal run_end event'.

go build/vet/gofmt clean; go test -race green for perfbench/agent/trace.
This commit is contained in:
Vasanthdev2004 committed 2026-07-16 14:46:42 +05:30
1 parent 2bfb340668
commit d061524989
1 file changed
+14 -2
+14 -2
View File
@@ -135,7 +135,7 @@ type TurnBenchResult struct {
BuildCheckedTasks int `json:"buildCheckedTasks"`
BuildPassedTasks int `json:"buildPassedTasks"`
CorrectnessPassRate float64 `json:"correctnessPassRate"`
BuildPassRate float64 `json:"buildPassRate,omitempty"`
BuildPassRate float64 `json:"buildPassRate"`
CorrectnessClasses []string `json:"correctnessClasses,omitempty"`
BuildOnlyClasses []string `json:"buildOnlyClasses,omitempty"`
LatencyOnlyClasses []string `json:"latencyOnlyClasses,omitempty"`
@@ -243,6 +243,13 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
})
}
if outcome.Err != nil {
// A crashed run must not look like a normal sample that's merely
// absent — surface it as a warning so a run that died every
// iteration can't pass as "fewer measurements."
result.Warnings = append(result.Warnings, Warning{
Metric: "run",
Message: fmt.Sprintf("task %s: %v", task.ID, outcome.Err),
})
passedForTask = false
continue
}
@@ -530,7 +537,6 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
start := time.Now()
runErr := cmd.Run()
wallMs := float64(time.Since(start).Microseconds()) / 1000
_ = runErr
exitCode, haveExit := streamJSONExitCode(outBuf.Bytes())
outcome := TurnTaskOutcome{WallMs: wallMs}
@@ -538,6 +544,12 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
outcome.VerifyErr = fmt.Sprintf("agent run_end exit code %d", exitCode)
} else if !haveExit {
detail := strings.TrimSpace(errBuf.String())
// If the process never produced a run_end event, the actual failure
// is in runErr (binary missing, exec permission denied, etc.) —
// prefer it over the generic fallback so the real reason survives.
if detail == "" && runErr != nil {
detail = runErr.Error()
}
if detail == "" {
detail = "missing terminal run_end event"
}