The workflow_dispatch input 'n_tasks' (default "8") was previously a
no-op — never read by the workflow. Now it flows as HERETEK_N_TASKS to
the shell script.
Logic in terminal_bench_ab.sh:
- HERETEK_N_TASKS="8" (default; matches push trigger with no input):
load the quick subset file (8 specific tasks via --include-task-name)
- HERETEK_N_TASKS=anything else: skip the subset filter, harbor runs all
dataset tasks (~89 for [email protected])
This is the manual-only path for tier=full:
- push trigger: unchanged, defaults to 8 tasks via workflow input default
- workflow_dispatch with "8" or unset: same quick behavior
- workflow_dispatch with "89" / "full" / etc: runs the full dataset
Push-to-main stays fast (~30 min, 8 tasks). Maintainer who wants
statistical power dispatches manually with n_tasks=89 (~4-6 hours).
Co-Authored-By: Claude <[email protected]>
The first real run (31615096459) failed 'Open comparison issue' because
the 'terminal-bench-ab' label was missing. The previous fix added a
precondition step that creates that label. But the issue step also uses
'--label automated', which is ALSO missing from the repo. The same
failure mode applies.
Extend the precondition step to create both labels (idempotent via
--force). Manual issue #238 was created by hand to close the loop on the
existing data; this fix prevents the same fail-silent on the next run.
Co-Authored-By: Claude <[email protected]>
The 'Open comparison issue' step failed silently on the first real run
(run 31615096459) because the 'terminal-bench-ab' label doesn't exist
in the repo. gh issue create exits 1 with 'could not add label:
terminal-bench-ab not found'. The continue-on-error: true mask on the
issue step swallowed the failure, so the workflow reported success
overall — but no issue was actually posted.
Add a precondition step that creates the label (idempotent via --force)
before the issue step. Belt-and-suspenders: also wraps the label step
in continue-on-error: true so a label-creation permission failure still
lets the issue step try (and fail gracefully).
Triggers-on-push verification: the next run will produce a real GitHub
issue with the aggregator's comparison body.
Co-Authored-By: Claude <[email protected]>
Real harbor runs reach the Actions log only at job completion (8 quick-
subset tasks take 10-30 minutes per agent). Without --debug, the GH log
shows zero progress during the run, leaving maintainers guessing if
harbor is actually working or stalled.
--debug surfaces trial resolution, dataset download progress, and
container lifecycle to stdout. Default verbosity already shows trial
progress bars; --debug adds the underlying detail.
Co-Authored-By: Claude <[email protected]>
harbor 0.21.0 declares requires-python >=3.12. The runner's setup-python@v5
was pinned to 3.10 (the version current when the workflow was first
written), which fails uv tool install with:
No solution found when resolving dependencies:
Because the current Python version (3.10.20) does not satisfy
Python>=3.12 and harbor==0.21.0 depends on Python>=3.12
The previous fix chain (c852341, 0c9236b, 78c59e7) addressed wheel-
availability flags but couldn't catch this because harbor's Python
requirement only surfaces when uv actually resolves the dependency.
Triggers-on-push verification: a real harbor run is now blocked only by
Docker availability + repo secrets, not by Python version.
Co-Authored-By: Claude <[email protected]>
The header docblock and the per-agent jobs dir comment still described
harbor's output layout as <jobs-dir>/<job-name>/trials/<trial-id>/...,
but the verified harbor 0.21.0 schema (HARBOR_NOTES.md and
docs/superpowers/specs/2026-08-12-aggregate-results-design.md §2) is
<jobs-dir>/<job-name>/<trial-name>/... (no trials/ subdir).
Comments only; no code or variable assignments changed.
Co-Authored-By: Claude <[email protected]>
The 'Output directory structure' section was written from an older
harbor version; corrected for 0.21.0 (subdirs for agent/verifier, no
trajectory.json, no agent.log file). The 'Follow-up summary.json
aggregation' section is replaced with a one-line pointer to the
shipped aggregator.
Co-Authored-By: Claude <[email protected]>
terminal_bench_ab.sh now emits <results-dir>/agent-{a,b}/summary.json
matching the Summary dataclass. comparison_report.py can now consume
real harbor output without falling over on FileNotFoundError.
Co-Authored-By: Claude <[email protected]>
Shared trial-dir factory in tests/_factories.py per the existing
convention. Round-trip test pins the aggregator output shape against
the prod load_summary code path — catches schema drift between the
aggregator and the Summary dataclass.
Co-Authored-By: Claude <[email protected]>
Reads harbor 0.21.0 per-trial result.json (verifier_result.rewards +
agent_result token fields + started_at/finished_at) and emits a dict
matching the Summary dataclass in scripts/comparison_report.py. Pure
stdlib, no harbor import. 11 hermetic tests with synthetic trial dirs.
Co-Authored-By: Claude <[email protected]>
5 tasks covering the aggregator implementation: (1) TDD aggregate_jobs_dir
with 11 hermetic tests, (2) factory helper + round-trip test, (3) wire
into terminal_bench_ab.sh, (4) HARBOR_NOTES.md corrections, (5) ci.sh
verification. Each task is independently testable and ends with a commit.
Co-Authored-By: Claude <[email protected]>
Per-trial result.json walker emits summary.json matching the existing
Summary dataclass. Closes the only remaining gap in the terminal-bench-ab
pipeline shipped in #235.
Verified harbor 0.21.0 trial schema by reading the installed harbor
source (the HARBOR_NOTES.md schema section is stale for this version —
spec includes the corrected layout). Per-trial result.json carries
verifier_result.rewards (verdict), agent_result.{n_input_tokens,
n_output_tokens} (tokens), and started_at/finished_at (wall_clock).
Co-Authored-By: Claude <[email protected]>
Previous fix used an inline # nosonar suppression based on the wrong
assumption that uv tool install has no --no-build equivalent. uv DOES
accept --no-build on (verified via 'uv tool install
--help' and a smoke-test install). This removes the nosonar hack and
uses the real flag, so the SonarCloud S2648 rule is satisfied at the
source rather than suppressed.
Co-Authored-By: Claude <[email protected]>
SonarCloud S2648 (--no-build) flagged the uv tool install harbor line.
uv tool install has no --no-build / --only-binary equivalent; harbor
0.21.0 ships as a pure-Python wheel on PyPI, no setup.py runs, and the
version is pinned. Added inline # nosonar + a 3-line rationale comment
so the next reader knows why we accept the (non-existent) risk.
Co-Authored-By: Claude <[email protected]>
SonarCloud flagged Duplicated Lines on New Code: 22 duplicated lines
in tests/test_comparison_diff.py (20.0%) and tests/test_comparison_report.py
(11.6%). The duplicate was the _summary/_make_summary helper (~20 lines
each) and the fixture-root path resolver.
Extracted into tests/_factories.py with two functions:
- make_summary(passed_ids, failed_ids, **kwargs) -> Summary
- fixture_root(name: str) -> Path
Tests now import from _factories. conftest.py adds tests/ to sys.path
(symmetric with the existing scripts/ sys.path injection). Both test
files lose ~22 duplicated lines each.
15/15 tests pass (was 15/15 before refactor; no behavior change).
Co-Authored-By: Claude <[email protected]>
SonarCloud flagged the unpinned 'pip install uv' and 'uv tool install
harbor' invocations. Pinned both to the versions we tested with
(uv==0.5.9, harbor==0.21.0). Added --only-binary :all: to the pip
install line to prevent running setup.py if a source distribution
is the only artifact available. Run values quoted to keep YAML
parser happy.
Co-Authored-By: Claude <[email protected]>
SonarCloud S5742 flagged test_missing_agent_b_raises_file_not_found for
having multiple invocations that could throw. Wrapping the
load_summary call in a lambda ensures pytest.raises only sees the
load_summary failure (and not, e.g., a path-building failure).
Co-Authored-By: Claude <[email protected]>
SonarCloud S3776 flagged render_markdown at complexity 36 (>15 allowed).
Extracted the function body into six small helpers:
- _render_header
- _render_headline_rows
- _signed + _render_delta_row
- _format_task_cell
- _note_for
- _render_per_task_table
- _render_list_section
render_markdown now just sequences lines.extend() calls. Golden test
still passes byte-for-byte (8/8 tests).
Co-Authored-By: Claude <[email protected]>
SonarCloud S-rules flagged single-bracket tests on lines 36/37/38
as less safe than [[ ]]. [[ ]] prevents word-splitting on unquoted
expansions and supports richer tests. Behavior unchanged; 9/9
shell tests still pass.
Co-Authored-By: Claude <[email protected]>
Verified end-to-end with a mocked harbor: terminal_bench_ab.sh invokes
harbor twice with the right flags, comparison_report.py renders the
Markdown body. Documents the open follow-up: harbor doesn't produce
summary.json directly — a future aggregator step must walk
<jobs-dir>/<job-name>/trials/*/result.json and emit summary.json
per agent before real harbor runs feed comparison_report.py.
Co-Authored-By: Claude <[email protected]>
Extends scripts/ci.sh with actionlint on the new GH workflow. actionlint
is optional (skip with a notice if not installed).
Co-Authored-By: Claude <[email protected]>
Push-to-main + workflow_dispatch trigger. concurrency group, required
permissions, harbor install, dual agent run, comparison render, issue
creation. Schema tests verify the rendered body has all required
sections and the workflow YAML has concurrency + permissions blocks.
Actions pinned to commit SHAs per project D20 (matches the convention
in the other workflows).
Co-Authored-By: Claude <[email protected]>
Detects sk-*, sk-cp-*, ghp_*, gho_*, xox*- tokens in rendered Markdown.
The render_with_secret_check wrapper aborts if any are found. Negative
fixture (case-task-id-with-secret) locks the behavior. Also wires the
guard into the __main__ block so 'python scripts/comparison_report.py'
refuses to write secrets to disk.
Co-Authored-By: Claude <[email protected]>
Claude