Summary:
- Add the repo-local update-harness skill and Claude skill symlink.
- Add a Bun-owned ci command and route GitHub CI through it.
- Tighten generated-doc freshness and workspace inventory generation.
Rationale:
- Keeps the harness upgrade workflow available in-repo for future agents.
- Gives local and hosted CI one shared command instead of duplicated YAML.
- Makes generated inventory depend on tracked files and catches real drift.
Tests:
- bun install --frozen-lockfile
- bun run ci
Co-authored-by: Codex <[email protected]>
## Intent
Remove server-layer imports of concrete SQLite persistence helpers by routing controllers and routes through typed persistence repository interfaces. This addresses SYM-35 and keeps backend-specific initialization inside persistence implementations.
## Behavior changes
No behavior changes. The server still uses SQLite for write-enabled start-server mode, while read/preset repository methods remain available through the typed SQLite and Postgres backends.
## Validation
- [x] `./scripts/fast-feedback.sh` passed
- [x] Behavior docs updated (if behavior changed): no behavior changes
Additional evidence:
- `rtk rg -n "providers/persistence/(sqlite|postgres)-|sqlite-run-history|sqlite-connection|postgres-backend|SqliteRunRecorder" src/runtime/server` returns no matches.
- `rtk bun run typecheck` passed.
- `rtk bun run test` passed: 139 tests before latest dev merge; fast feedback passed after the final `origin/dev` merge with 152 tests.
- GitHub CI passed on head `73a110c` after resolving the merge conflict against `dev`.
## Screenshots / video
N/A for CLI-only changes.
Summary:
- Add Happy DOM CompareView unit tests for the only-changes filter,
picker apply state, empty aligned rows, and null versus zero score
formatting.
- Extend comparison integration and controller tests for three-run
file/id alignment, empty comparison rows, malformed run IDs,
duplicate run IDs, and structured bad_request responses.
- Reject malformed or duplicate compare run IDs before repository
lookup, and document the run UUID validation contract.
Rationale:
- SYM-37 called out compare coverage gaps across both the dashboard UI
and /api/comparisons endpoint behavior.
- Duplicate run IDs previously deduped silently, which hid bad input and
made the endpoint validation contract weaker than the ticket requires.
Tests:
- bun test tests/unit/dashboard/compare-view.test.tsx
- bun test tests/integration/server/comparisons.test.ts
- bun test tests/unit/server/comparison.test.ts
- bun run docs:validate
- bun run lint
- bun run typecheck
- bun run fast-feedback
Co-authored-by: Symphony Agent <[email protected]>
Co-authored-by: Codex <[email protected]>
## Intent
Fix SYM-34 by making server-managed run executor failures observable outside
SSE subscribers. The controller now writes a structured `run_executor` stderr
line, persists the failure on the run record when a run ID exists, and still
publishes the terminal `run_error` stream event.
## Behavior changes
Failed `agentprobe start-server` runs that reach the executor catch path now
remain visible through `/api/runs/:runId` after the SSE session closes, with
`finalError` populated for the historical run detail. Operators also get a JSON
stderr log line with the run ID, error type, message, and stack.
## Validation
- [x] `./scripts/fast-feedback.sh` passed
- [x] Behavior docs updated (if behavior changed)
- [x] `bun test tests/integration/server/write-control.test.ts` passed
- [x] `bun run typecheck` passed
- [x] `bun run lint` passed
## Screenshots / video
N/A for CLI/server behavior changes.
## Summary
- `bun run test`, `bun run test:coverage`, and `bun run test:e2e` now pass `--timeout 30000` so the CLI-spawning e2e cases don't trip Bun's 5s default once `bunfig.toml` enables coverage instrumentation.
- Documented the choice and intent in `docs/HARNESS.md`.
## Test plan
- [x] `bun run test` (88 pass)
- [x] `bun run test:e2e` (19 pass)
- [x] `bun run docs:validate`
Linear: https://linear.app/autogpt/issue/SYM-40🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Intent
Fix SYM-27 by making database URL redaction handle passwords that contain reserved characters such as `@`, `:`, `/`, and `%`, then apply redaction consistently to operator-visible output.
## Behavior changes
Database URL passwords are now redacted using URL parsing for all schemes with userinfo credentials. Config errors, SQLite unsupported-URL errors, health/readiness output, migration output, and the start-server banner avoid exposing raw configured passwords.
The branch has also been synced with current `dev`; conflict resolution preserved the `dev` Postgres write-mode restriction, CORS/readiness updates, and Postgres batching tests while keeping the SYM-27 redaction behavior.
## Validation
- [ ] `./scripts/fast-feedback.sh` passed locally
- Latest landing run passed repo validation, lint, and typecheck, then hit known local 5s e2e timeout cases tracked separately as SYM-41: `agentprobe start-server > boots without OPEN_ROUTER_API_KEY and shuts down on SIGTERM` and `bun e2e baseline for the typescript cli > run records the suite in sqlite and report renders both explicit and discovered outputs`.
- [x] GitHub CI passed for commit `987af8fb5`.
- [x] Behavior docs updated (if behavior changed)
- [x] `bun test tests/unit/persistence/url.test.ts tests/unit/server/config.test.ts`
- [x] `bun run lint` via fast-feedback
- [x] `bun run docs:validate` via fast-feedback
- [x] `bunx tsc --noEmit && bun run --cwd dashboard typecheck` via fast-feedback
## Screenshots / video
N/A for CLI-only changes.
## Intent
Slim the Docker runtime image for SYM-30 by replacing the runtime-stage full-tree copy with a production dependency install and explicit runtime asset copies.
This removes tests, docs, scripts, agent metadata, and dev dependencies from the final image while preserving the Bun TypeScript server entrypoint, runtime data, and built dashboard bundle.
## Behavior changes
No CLI or API behavior changes. The published Docker image contents are narrower:
- Runtime stage now runs `bun install --production --frozen-lockfile`.
- Final image copies only `src`, `data`, and `dashboard/dist` from the build stage.
- `.dockerignore` excludes docs, scripts, tests, and agent metadata from the build context.
Image evidence:
- Before: `126398737` bytes; `/app` was `191M`.
- After: `71911094` bytes; `/app` is `15M`.
- Reduction: `54487643` bytes, about 43.1%.
- Final image inspection confirms `tests`, `dashboard/src`, `docs`, `scripts`, `.git`, `.agents`, `.claude`, `node_modules/@biomejs`, `node_modules/typescript`, `node_modules/bun-types`, and `node_modules/@types` are absent.
## Validation
- [ ] `./scripts/fast-feedback.sh` passed
- Not used as the final authority because its `bun run test` step hits existing 5s e2e timeout failures in this workspace; filed SYM-40.
- [x] Behavior docs updated (if behavior changed)
- No product behavior docs changed; Docker packaging only.
Validation run:
- `docker build -t agentprobe:sym-30-before .` on the original `origin/dev` Dockerfile for reproduction.
- `docker run --rm --entrypoint sh agentprobe:sym-30-before -c '...'` confirmed broad runtime tree and dev deps present.
- `docker build -t agentprobe:sym-30-after .`
- `docker run --rm --entrypoint sh agentprobe:sym-30-after -c '...'` confirmed excluded paths/dev deps absent and runtime paths present.
- `bun run docs:validate` passed.
- `bun run test` failed only on existing 5s e2e timeouts; see SYM-40.
- `bun run test:e2e` reproduced the same 5s e2e timeout failures; see SYM-40.
- `bun test --timeout 30000 tests/e2e` passed: 20 pass, 0 fail.
- `bun run test tests/integration/server` passed: 8 pass, 0 fail.
- `bun run lint` passed.
- `bun run typecheck` passed.
- `bun run dashboard:build` passed.
- `AGENTPROBE_SERVER_TOKEN=sym30-test-token OPEN_ROUTER_API_KEY=dummy-key docker compose -p sym30 up --build -d` passed.
- Compose smoke: `/healthz` returned 200, authenticated `/api/presets` returned 200, authenticated dry-run `/api/runs` completed with `passed: true` and `exitCode: 0`.
- `docker compose -p sym30 down -v` cleaned up the stack.
## Screenshots / video
N/A for CLI-only changes.
## Intent
Fix SYM-28 by removing the length-mismatch branch from bearer-token
comparison. `constantTimeEquals` now pads both UTF-8 byte arrays to a fixed
width, performs one timing-safe comparison, and only then gates the return value
on byte-length equality and compare-size bounds.
## Behavior changes
No user-visible behavior changes. Valid configured bearer tokens are still
accepted, invalid tokens are still rejected, and API auth coverage is unchanged.
The internal comparison path no longer has a distinct length-mismatch branch.
## Validation
- [x] `./scripts/fast-feedback.sh` passed
- [x] Behavior docs updated (if behavior changed)
- Not applicable; this is an internal security hardening with unchanged
public behavior.
- [x] `bun test tests/unit/server/auth.test.ts`
- [x] `git diff --check`
- [x] `bun test --timeout 20000 tests/e2e/start-server.e2e.test.ts`
- [x] `bun test --timeout 20000 tests/e2e/cli.e2e.test.ts --test-name-pattern "run records|openclaw commands"`
## Screenshots / video
N/A for CLI-only changes.
Add an `agentprobe` Docker Compose healthcheck so Compose can distinguish process start from server readiness. The probe calls `/readyz` from inside the container, and the server playbook documents the command plus failure debugging steps.
Validation:
- docker compose config
- docker compose up --build -d agentprobe; waited for healthy; curl /readyz; docker compose down -v
- bun run docs:validate
- bun test tests/integration/server/read-only.test.ts tests/unit/server/auth.test.ts tests/unit/server/config.test.ts
- git diff --check
- bun run fast-feedback
## Intent
Fix SYM-31 by adding explicit CORS handling for the AgentProbe control-plane API. The server now answers `/api/*` OPTIONS preflights centrally, attaches CORS response headers for allowed origins, and fails closed when operators expose the server externally without an explicit origin allow-list.
## Behavior changes
- `/api/*` OPTIONS preflights return `204` for allow-listed origins with allow-methods, allow-headers, allow-credentials, and max-age headers.
- `/api/*` OPTIONS preflights return `403` for unlisted origins.
- Non-preflight `/api/*` responses echo `Access-Control-Allow-Origin` for allowed origins.
- `AGENTPROBE_SERVER_CORS_ORIGINS` configures comma-separated exact `http://` or `https://` origins.
- `--unsafe-expose` / `AGENTPROBE_SERVER_UNSAFE_EXPOSE=true` now requires `AGENTPROBE_SERVER_CORS_ORIGINS` in addition to an auth token.
## Validation
- [x] `./scripts/fast-feedback.sh` passed on merge head `3774c52`
- [x] Behavior docs updated (if behavior changed)
- `bun test tests/unit/server/config.test.ts tests/integration/server/read-only.test.ts`
- `bun test tests/unit/server tests/integration/server`
- `bun run docs:validate`
- `bun test --timeout 15000` (pre-merge validation on original CORS head)
## Screenshots / video
N/A for CLI/server changes.
## Intent
Fix the Postgres \ 2N+1 query pattern by batching related scenario-selection and latest-run reads by \.
## Behavior changes
No user-visible behavior changes. Preset listing keeps the same ordering and returned fields while reducing Postgres round trips from 2N+1 to a constant query count.
## Validation
- [x] \bun test v1.3.12 (700fc117)
- [x] \bun test v1.3.12 (700fc117)
- [x] \error TS5025: Unknown compiler option '--noEmit\'. Did you mean 'noEmit'?
- [x] \
- [x] \
- [x] Behavior docs updated (if behavior changed): no behavior docs needed for this non-user-visible persistence optimization
## Screenshots / video
N/A for persistence-only changes.
## Intent
Fix the inline dashboard run-detail renderer so malformed scenario ordinals cannot break out of the scenario link href or body text. This is defense-in-depth for run data consumed from `/api/runs/:id`.
## Behavior changes
No intended user-visible behavior changes for valid run data. The inline dashboard now escapes malformed ordinal/count/score values before inserting them into generated HTML.
## Validation
- [x] `./scripts/fast-feedback.sh` passed
- [x] `bun test tests/unit/server/inline-dashboard.test.ts` passed
- [x] `bun test --timeout 15000` passed
- [x] Behavior docs updated (if behavior changed; N/A for this security hardening)
## Intent
Fix SYM-33 by making the Postgres run-recording limitation visible in the repository type system and by failing fast when the write-enabled server is configured with Postgres.
## Behavior changes
- `PostgresRepository` no longer exposes `createRecorder()`; only `RecordingRepository` implementations can create run recorders.
- `agentprobe start-server` rejects Postgres URLs while run write routes such as `POST /api/runs` are enabled, with a clear SQLite guidance error before schema probing or first write traffic.
- Server docs now state the current Postgres posture: migrations, preset CRUD, and historical reads are supported; run recording remains SQLite-only.
## Validation
- [x] `./scripts/fast-feedback.sh` passed
- [x] Behavior docs updated (if behavior changed)
- [x] `bun test --timeout 20000 tests/e2e/start-server.e2e.test.ts` passed
- [x] `bun test --timeout 20000 tests/e2e/cli.e2e.test.ts` passed
- [x] `git diff --check` passed
## Screenshots / video
N/A for CLI-only changes.
Phase 3 of the AgentProbe server: ship historical-run comparison as an API
and dashboard workspace, and introduce the persistence abstraction needed to
back the server with Postgres behind `AGENTPROBE_DB_URL`.
- Persistence contract: new PersistenceRepository interface, URL parser /
redactor, migration dispatcher (per-backend versioned) and a backend factory
that returns SqliteRepository or PostgresRepository. SQLite free-function
exports stay as compat wrappers; all new consumers go through the interface.
- Postgres backend: full DDL with jsonb columns and indexes for server
filters + comparison lookups, migration runner, boot-time schema-version
check that refuses to start when behind, preset/run/comparison reads, and
preset CRUD. Run recorder (writes) is deferred to SYM-25.
- CLI `agentprobe db:migrate`: accepts --db or AGENTPROBE_DB_URL, prints
backend / current / target / applied, fails clearly on unsupported schemes.
- Server config + /readyz + new /api/session expose the backend kind and a
redacted db_url. Postgres URLs are now accepted via --db and env.
- Comparison controller: loads 2–10 runs, chooses alignment (preset snapshot
→ preset id → scenario id → file::id), emits runs, scenarios with per-run
status/score/reason, delta_score, status_change, present_in, and summary
buckets. GET /api/comparisons rejects out-of-range counts with the common
error envelope and sets cache-control: no-store.
- Dashboard /compare workspace: ad-hoc multi-run picker, sticky summary bar,
per-run columns, missing-scenario rows, "only changes" toggle,
?run_ids=…&only=changes deep links, and a "Compare last two runs" CTA on
the preset detail view.
- Docker Compose example documents Postgres as an opt-in service; SQLite on
a named volume remains the default.
- Playbook expansion: Postgres setup/migration/rollback/backup, comparison
semantics, connection errors, and duplicate-scenario-id behaviour.
- Tests: unit coverage for url redaction + parse, migration runner (fresh,
idempotent, upgrade-in-place, unsupported schemes), factory dispatch, and
the comparison controller (alignment, delta/status_change, duplicate ids,
range enforcement, 404 routing). Integration coverage starts a real server
against seeded SQLite runs and exercises the /api/comparisons payload.
Follow-up SYM-25 tracks the buffered async Postgres recorder.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Summary:
- Add authenticated server write routes for starting and cancelling runs,
preset CRUD, preset launch history, and frozen preset snapshots.
- Extend SQLite run history to schema v4 with preset tables, server-run
metadata, cancellation timestamps, and WAL-enabled server connections.
- Teach runSuite to accept prepared file/id scenario selections and a
cooperative AbortSignal while preserving existing CLI and dashboard mode.
- Add dashboard start/preset/cancel views plus Docker and Compose packaging
with a token-protected SQLite volume deployment path.
Rationale:
- Phase 2 needs an operator workflow that can configure a run, save it as a
reusable preset, launch it from the server UI, and observe/cancel progress.
- Scenario references now use file plus id so duplicate ids across scenario
files remain deterministic, and write paths validate boundaries at data-root
and bearer-auth edges.
Tests:
- bun run docs:validate
- bun run test tests/unit/server
- bun run test tests/integration/server
- bun run test:e2e
- bun run dashboard:build
- bun run typecheck
- bun run fast-feedback
- COMPOSE_PROGRESS=plain AGENTPROBE_SERVER_TOKEN=... OPEN_ROUTER_API_KEY=...
docker compose -p agentprobe-sym22 up --build --detach
- curl -fsS http://127.0.0.1:7878/healthz
- curl -fsS -X POST http://127.0.0.1:7878/api/runs ... dry-run payload
- OPEN_ROUTER_API_KEY=... docker compose -p agentprobe-sym22-missing config
exited 1 for missing AGENTPROBE_SERVER_TOKEN
Co-authored-by: Codex <[email protected]>
Summary:
- Add start-server config, auth, routing, static dashboard, SQLite read
routes, suite discovery, report rendering, and SSE replay support.
- Add a dual-mode dashboard that preserves /api/state live polling and
adds read-only server views for overview, runs, scenarios, suites, and
settings.
- Add unit, integration, and e2e coverage for server config, auth,
streams, HTTP read paths, token protection, and CLI lifecycle.
Rationale:
- Phase 1 needs a stable read-only HTTP surface before later write paths
add run control, presets, cancellation, and other orchestration.
- Binding safety and token auth are enforced at boot and request edges so
non-loopback exposure cannot accidentally start unauthenticated.
Tests:
- bun run docs:validate
- bun run test tests/unit/server
- bun run test tests/integration/server
- bun run test:e2e
- bun run dashboard:build
- bun run typecheck
- bun run lint
- manual start-server smoke with /healthz, /readyz, /api/scenarios,
and /api/runs
Co-authored-by: Codex <[email protected]>
Turns the approved AgentProbe server design
(docs/design-docs/agent-probe-server.md) into a binding product contract
before any runtime code lands, per the Phase 0 exec plan
(docs/exec-plans/active/agent-probe-server-phase-0-contract-2026-04.md).
- platform.md: add "Server control plane" scenario group with 9 scenarios
covering default boot, exposure safety, read-only HTTP/UI history, live
SSE, run control, cancellation, presets, comparisons, and Docker SQLite
persistence.
- current-state.md: mirror all 9 new scenarios as unchecked (planned) and
refresh "Last validated against" to 2026-04-17.
- e2e-checklist.md: add a planned test-owner row per scenario covering
tests/e2e/server-e2e.test.ts, tests/integration/server/,
tests/unit/server/, and dashboard component tests.
scripts/check-behaviour-docs.ts reports zero drift across all 24 scenarios.
No runtime files modified; PR targets dev.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- Reframe cupel inspiration as design direction (palette, density,
list+detail split), not a visual copy.
- Add cross-file scenario selection, saved presets, and preset-aware
run history.
- Add /compare workspace and /api/comparisons for diffing multiple
runs of the same preset.
- Surface parallel-factor control on the /start run builder.
- Document Postgres backend behind AGENTPROBE_DB_URL for ephemeral
Docker deployments where SQLite-on-volume isn't viable; gated to
Phase 3.
- Fix Dockerfile CMD to match the server's non-loopback security
contract: bake --unsafe-expose into CMD and require
AGENTPROBE_SERVER_TOKEN at boot, failing fast when absent.
- Restructure rollout phases and acceptance criteria accordingly.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Captures the durable design for `agentprobe start-server`: long-running,
Dockerable control plane with a cupel-inspired UI, SSE-driven live run
view, browsable run history, and phased rollout that reuses the existing
`--dashboard` infrastructure.
Linear: SYM-18
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
When the SSE stream contains error events but no delta content,
extract errorText and show it as the assistant message so the
judge and transcript reflect what actually happened.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
When --scenarios points to a directory, use the directory itself as
the base for resolving attachment paths, not its parent.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
When loading scenarios from a directory, different files may define
different defaults (e.g. max_turns: 5 vs 10). Instead of throwing,
drop the conflicting key from the merged defaults. Each file's
defaults are already baked into its scenarios during parsing.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Fixture data files in data/fixtures/ contain non-scenario YAMLs
that fail strict parsing. Skip fixtures/ and hidden directories
during recursive YAML discovery.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Summary:
- Update scenario filters spec to describe --scenario flag, name matching,
comma-separated values, and improved error messages
- Add "List command shows available scenarios" spec section
- Add list command to current-state.md as implemented
- Add list command to e2e-checklist.md as planned
- Add e2e test for no-match scenario-id returning available IDs
Rationale:
- Product spec must reflect the --scenario and list command behavior
introduced in the previous commit before marking work complete.
Tests:
- bun run fast-feedback: all gates pass (repo validation, lint, typecheck, 85 tests)
Co-authored-by: Codex <[email protected]>
Summary:
- Add --scenario flag as alias for --scenario-id with name matching support
- Support comma-separated scenario IDs/names in --scenario flag
- Match scenarios by both id and name (e.g., --scenario "Refund smoke question")
- Improve error message to list available scenarios with IDs and names
- Add list command to display available scenarios for discovery
- Extend selectDashboardScenarios with same name-matching logic
- Fix db.test.ts expected error message to match new format
Rationale:
- Users want to run a single scenario without running the entire file.
- Matching by name is more ergonomic than requiring exact kebab-case IDs.
- The list command helps users discover scenario IDs/names before running.
- Comma-separated values allow running multiple specific scenarios.
Tests:
- bun run test: 82 pass, 0 fail
- bun run test:e2e: 15 pass, 0 fail
- Added unit test for scenarioId filtering by name in runSuite
- Added unit test for comma-separated scenario selection
- Added e2e test for --scenario flag alias
- Added e2e test for name-based scenario matching
Co-authored-by: Codex <[email protected]>
Summary:
- Add `FailureKind = "agent" | "harness"` type to contracts; extend
`RubricScore`, `ScenarioRunResult`, and `ScenarioRecord` with
`failureKind` field
- Update judge JSON schema to require `failure_kind` (enum: agent |
harness); instruct judge to set "harness" when the test infra caused
the failure (e.g. persona didn't pass required files), "agent"
otherwise
- Propagate `failureKind` from judge score through `run-suite.ts` into
`ScenarioRunResult`
- Bump DB schema to v3; add `failure_kind text` column to
`scenario_runs`, `scenario_harness_failed_count` column to `runs`,
and additive migration from v2 → v3 via `ensureColumn`
- Track `scenarioHarnessFailedCount` in `refreshRunCounts`; persist
`failure_kind` in `recordScenarioFinished`
- Extend `DashboardScenarioState.status` with `"harness_fail"`;
add `harness_failed` to `DashboardStateSnapshot`; surface
`failure_kind` in `DashboardScenarioDetail`
- Update `statusTone` / `statusLabel` in render-report to emit
`"warning"` / `"HARNESS FAIL"` for harness failures; add amber
badge and score-bar colour; add dedicated "Harness Failures" card
in the run header (separate from "Agent Failures")
Rationale:
- Binary pass/fail obscured whether a failing run reflected a real
agent defect or a broken test harness; this classification lets
teams filter harness noise from true regressions
- Schema migration is additive (ALTER TABLE ADD COLUMN with default)
so existing databases upgrade without data loss
- Keeping `failure_kind` in the judge output JSON means it is
archived alongside the rubric score for historical queries
Tests:
- bun run test: 78 pass, 0 fail
- bun run docs:validate: all checks passed
- Updated db.test.ts, report.test.ts, and support.ts assertions to
cover new schema version (3) and `scenarioHarnessFailedCount`
Co-authored-by: Codex <[email protected]>
* fix: join SSE content with spaces and collapse duplicates
SSE stream chunks joined with newlines produced unnatural text breaks.
Switch to space-join with duplicate space collapse for cleaner output.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: update SSE join expectation in adapter test
Align test with the newline-to-space join change in http-endpoint.ts.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Swifty