## Intent
Remove server-layer imports of concrete SQLite persistence helpers by routing controllers and routes through typed persistence repository interfaces. This addresses SYM-35 and keeps backend-specific initialization inside persistence implementations.
## Behavior changes
No behavior changes. The server still uses SQLite for write-enabled start-server mode, while read/preset repository methods remain available through the typed SQLite and Postgres backends.
## Validation
- [x] `./scripts/fast-feedback.sh` passed
- [x] Behavior docs updated (if behavior changed): no behavior changes
Additional evidence:
- `rtk rg -n "providers/persistence/(sqlite|postgres)-|sqlite-run-history|sqlite-connection|postgres-backend|SqliteRunRecorder" src/runtime/server` returns no matches.
- `rtk bun run typecheck` passed.
- `rtk bun run test` passed: 139 tests before latest dev merge; fast feedback passed after the final `origin/dev` merge with 152 tests.
- GitHub CI passed on head `73a110c` after resolving the merge conflict against `dev`.
## Screenshots / video
N/A for CLI-only changes.
Summary:
- Add Happy DOM CompareView unit tests for the only-changes filter,
picker apply state, empty aligned rows, and null versus zero score
formatting.
- Extend comparison integration and controller tests for three-run
file/id alignment, empty comparison rows, malformed run IDs,
duplicate run IDs, and structured bad_request responses.
- Reject malformed or duplicate compare run IDs before repository
lookup, and document the run UUID validation contract.
Rationale:
- SYM-37 called out compare coverage gaps across both the dashboard UI
and /api/comparisons endpoint behavior.
- Duplicate run IDs previously deduped silently, which hid bad input and
made the endpoint validation contract weaker than the ticket requires.
Tests:
- bun test tests/unit/dashboard/compare-view.test.tsx
- bun test tests/integration/server/comparisons.test.ts
- bun test tests/unit/server/comparison.test.ts
- bun run docs:validate
- bun run lint
- bun run typecheck
- bun run fast-feedback
Co-authored-by: Symphony Agent <[email protected]>
Co-authored-by: Codex <[email protected]>
## Intent
Fix SYM-34 by making server-managed run executor failures observable outside
SSE subscribers. The controller now writes a structured `run_executor` stderr
line, persists the failure on the run record when a run ID exists, and still
publishes the terminal `run_error` stream event.
## Behavior changes
Failed `agentprobe start-server` runs that reach the executor catch path now
remain visible through `/api/runs/:runId` after the SSE session closes, with
`finalError` populated for the historical run detail. Operators also get a JSON
stderr log line with the run ID, error type, message, and stack.
## Validation
- [x] `./scripts/fast-feedback.sh` passed
- [x] Behavior docs updated (if behavior changed)
- [x] `bun test tests/integration/server/write-control.test.ts` passed
- [x] `bun run typecheck` passed
- [x] `bun run lint` passed
## Screenshots / video
N/A for CLI/server behavior changes.
## Intent
Fix SYM-27 by making database URL redaction handle passwords that contain reserved characters such as `@`, `:`, `/`, and `%`, then apply redaction consistently to operator-visible output.
## Behavior changes
Database URL passwords are now redacted using URL parsing for all schemes with userinfo credentials. Config errors, SQLite unsupported-URL errors, health/readiness output, migration output, and the start-server banner avoid exposing raw configured passwords.
The branch has also been synced with current `dev`; conflict resolution preserved the `dev` Postgres write-mode restriction, CORS/readiness updates, and Postgres batching tests while keeping the SYM-27 redaction behavior.
## Validation
- [ ] `./scripts/fast-feedback.sh` passed locally
- Latest landing run passed repo validation, lint, and typecheck, then hit known local 5s e2e timeout cases tracked separately as SYM-41: `agentprobe start-server > boots without OPEN_ROUTER_API_KEY and shuts down on SIGTERM` and `bun e2e baseline for the typescript cli > run records the suite in sqlite and report renders both explicit and discovered outputs`.
- [x] GitHub CI passed for commit `987af8fb5`.
- [x] Behavior docs updated (if behavior changed)
- [x] `bun test tests/unit/persistence/url.test.ts tests/unit/server/config.test.ts`
- [x] `bun run lint` via fast-feedback
- [x] `bun run docs:validate` via fast-feedback
- [x] `bunx tsc --noEmit && bun run --cwd dashboard typecheck` via fast-feedback
## Screenshots / video
N/A for CLI-only changes.
## Intent
Fix SYM-28 by removing the length-mismatch branch from bearer-token
comparison. `constantTimeEquals` now pads both UTF-8 byte arrays to a fixed
width, performs one timing-safe comparison, and only then gates the return value
on byte-length equality and compare-size bounds.
## Behavior changes
No user-visible behavior changes. Valid configured bearer tokens are still
accepted, invalid tokens are still rejected, and API auth coverage is unchanged.
The internal comparison path no longer has a distinct length-mismatch branch.
## Validation
- [x] `./scripts/fast-feedback.sh` passed
- [x] Behavior docs updated (if behavior changed)
- Not applicable; this is an internal security hardening with unchanged
public behavior.
- [x] `bun test tests/unit/server/auth.test.ts`
- [x] `git diff --check`
- [x] `bun test --timeout 20000 tests/e2e/start-server.e2e.test.ts`
- [x] `bun test --timeout 20000 tests/e2e/cli.e2e.test.ts --test-name-pattern "run records|openclaw commands"`
## Screenshots / video
N/A for CLI-only changes.
## Intent
Fix SYM-31 by adding explicit CORS handling for the AgentProbe control-plane API. The server now answers `/api/*` OPTIONS preflights centrally, attaches CORS response headers for allowed origins, and fails closed when operators expose the server externally without an explicit origin allow-list.
## Behavior changes
- `/api/*` OPTIONS preflights return `204` for allow-listed origins with allow-methods, allow-headers, allow-credentials, and max-age headers.
- `/api/*` OPTIONS preflights return `403` for unlisted origins.
- Non-preflight `/api/*` responses echo `Access-Control-Allow-Origin` for allowed origins.
- `AGENTPROBE_SERVER_CORS_ORIGINS` configures comma-separated exact `http://` or `https://` origins.
- `--unsafe-expose` / `AGENTPROBE_SERVER_UNSAFE_EXPOSE=true` now requires `AGENTPROBE_SERVER_CORS_ORIGINS` in addition to an auth token.
## Validation
- [x] `./scripts/fast-feedback.sh` passed on merge head `3774c52`
- [x] Behavior docs updated (if behavior changed)
- `bun test tests/unit/server/config.test.ts tests/integration/server/read-only.test.ts`
- `bun test tests/unit/server tests/integration/server`
- `bun run docs:validate`
- `bun test --timeout 15000` (pre-merge validation on original CORS head)
## Screenshots / video
N/A for CLI/server changes.
## Intent
Fix the Postgres \ 2N+1 query pattern by batching related scenario-selection and latest-run reads by \.
## Behavior changes
No user-visible behavior changes. Preset listing keeps the same ordering and returned fields while reducing Postgres round trips from 2N+1 to a constant query count.
## Validation
- [x] \bun test v1.3.12 (700fc117)
- [x] \bun test v1.3.12 (700fc117)
- [x] \error TS5025: Unknown compiler option '--noEmit\'. Did you mean 'noEmit'?
- [x] \
- [x] \
- [x] Behavior docs updated (if behavior changed): no behavior docs needed for this non-user-visible persistence optimization
## Screenshots / video
N/A for persistence-only changes.
## Intent
Fix the inline dashboard run-detail renderer so malformed scenario ordinals cannot break out of the scenario link href or body text. This is defense-in-depth for run data consumed from `/api/runs/:id`.
## Behavior changes
No intended user-visible behavior changes for valid run data. The inline dashboard now escapes malformed ordinal/count/score values before inserting them into generated HTML.
## Validation
- [x] `./scripts/fast-feedback.sh` passed
- [x] `bun test tests/unit/server/inline-dashboard.test.ts` passed
- [x] `bun test --timeout 15000` passed
- [x] Behavior docs updated (if behavior changed; N/A for this security hardening)
## Intent
Fix SYM-33 by making the Postgres run-recording limitation visible in the repository type system and by failing fast when the write-enabled server is configured with Postgres.
## Behavior changes
- `PostgresRepository` no longer exposes `createRecorder()`; only `RecordingRepository` implementations can create run recorders.
- `agentprobe start-server` rejects Postgres URLs while run write routes such as `POST /api/runs` are enabled, with a clear SQLite guidance error before schema probing or first write traffic.
- Server docs now state the current Postgres posture: migrations, preset CRUD, and historical reads are supported; run recording remains SQLite-only.
## Validation
- [x] `./scripts/fast-feedback.sh` passed
- [x] Behavior docs updated (if behavior changed)
- [x] `bun test --timeout 20000 tests/e2e/start-server.e2e.test.ts` passed
- [x] `bun test --timeout 20000 tests/e2e/cli.e2e.test.ts` passed
- [x] `git diff --check` passed
## Screenshots / video
N/A for CLI-only changes.
Phase 3 of the AgentProbe server: ship historical-run comparison as an API
and dashboard workspace, and introduce the persistence abstraction needed to
back the server with Postgres behind `AGENTPROBE_DB_URL`.
- Persistence contract: new PersistenceRepository interface, URL parser /
redactor, migration dispatcher (per-backend versioned) and a backend factory
that returns SqliteRepository or PostgresRepository. SQLite free-function
exports stay as compat wrappers; all new consumers go through the interface.
- Postgres backend: full DDL with jsonb columns and indexes for server
filters + comparison lookups, migration runner, boot-time schema-version
check that refuses to start when behind, preset/run/comparison reads, and
preset CRUD. Run recorder (writes) is deferred to SYM-25.
- CLI `agentprobe db:migrate`: accepts --db or AGENTPROBE_DB_URL, prints
backend / current / target / applied, fails clearly on unsupported schemes.
- Server config + /readyz + new /api/session expose the backend kind and a
redacted db_url. Postgres URLs are now accepted via --db and env.
- Comparison controller: loads 2–10 runs, chooses alignment (preset snapshot
→ preset id → scenario id → file::id), emits runs, scenarios with per-run
status/score/reason, delta_score, status_change, present_in, and summary
buckets. GET /api/comparisons rejects out-of-range counts with the common
error envelope and sets cache-control: no-store.
- Dashboard /compare workspace: ad-hoc multi-run picker, sticky summary bar,
per-run columns, missing-scenario rows, "only changes" toggle,
?run_ids=…&only=changes deep links, and a "Compare last two runs" CTA on
the preset detail view.
- Docker Compose example documents Postgres as an opt-in service; SQLite on
a named volume remains the default.
- Playbook expansion: Postgres setup/migration/rollback/backup, comparison
semantics, connection errors, and duplicate-scenario-id behaviour.
- Tests: unit coverage for url redaction + parse, migration runner (fresh,
idempotent, upgrade-in-place, unsupported schemes), factory dispatch, and
the comparison controller (alignment, delta/status_change, duplicate ids,
range enforcement, 404 routing). Integration coverage starts a real server
against seeded SQLite runs and exercises the /api/comparisons payload.
Follow-up SYM-25 tracks the buffered async Postgres recorder.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Summary:
- Add authenticated server write routes for starting and cancelling runs,
preset CRUD, preset launch history, and frozen preset snapshots.
- Extend SQLite run history to schema v4 with preset tables, server-run
metadata, cancellation timestamps, and WAL-enabled server connections.
- Teach runSuite to accept prepared file/id scenario selections and a
cooperative AbortSignal while preserving existing CLI and dashboard mode.
- Add dashboard start/preset/cancel views plus Docker and Compose packaging
with a token-protected SQLite volume deployment path.
Rationale:
- Phase 2 needs an operator workflow that can configure a run, save it as a
reusable preset, launch it from the server UI, and observe/cancel progress.
- Scenario references now use file plus id so duplicate ids across scenario
files remain deterministic, and write paths validate boundaries at data-root
and bearer-auth edges.
Tests:
- bun run docs:validate
- bun run test tests/unit/server
- bun run test tests/integration/server
- bun run test:e2e
- bun run dashboard:build
- bun run typecheck
- bun run fast-feedback
- COMPOSE_PROGRESS=plain AGENTPROBE_SERVER_TOKEN=... OPEN_ROUTER_API_KEY=...
docker compose -p agentprobe-sym22 up --build --detach
- curl -fsS http://127.0.0.1:7878/healthz
- curl -fsS -X POST http://127.0.0.1:7878/api/runs ... dry-run payload
- OPEN_ROUTER_API_KEY=... docker compose -p agentprobe-sym22-missing config
exited 1 for missing AGENTPROBE_SERVER_TOKEN
Co-authored-by: Codex <[email protected]>
Summary:
- Add start-server config, auth, routing, static dashboard, SQLite read
routes, suite discovery, report rendering, and SSE replay support.
- Add a dual-mode dashboard that preserves /api/state live polling and
adds read-only server views for overview, runs, scenarios, suites, and
settings.
- Add unit, integration, and e2e coverage for server config, auth,
streams, HTTP read paths, token protection, and CLI lifecycle.
Rationale:
- Phase 1 needs a stable read-only HTTP surface before later write paths
add run control, presets, cancellation, and other orchestration.
- Binding safety and token auth are enforced at boot and request edges so
non-loopback exposure cannot accidentally start unauthenticated.
Tests:
- bun run docs:validate
- bun run test tests/unit/server
- bun run test tests/integration/server
- bun run test:e2e
- bun run dashboard:build
- bun run typecheck
- bun run lint
- manual start-server smoke with /healthz, /readyz, /api/scenarios,
and /api/runs
Co-authored-by: Codex <[email protected]>
Summary:
- Update scenario filters spec to describe --scenario flag, name matching,
comma-separated values, and improved error messages
- Add "List command shows available scenarios" spec section
- Add list command to current-state.md as implemented
- Add list command to e2e-checklist.md as planned
- Add e2e test for no-match scenario-id returning available IDs
Rationale:
- Product spec must reflect the --scenario and list command behavior
introduced in the previous commit before marking work complete.
Tests:
- bun run fast-feedback: all gates pass (repo validation, lint, typecheck, 85 tests)
Co-authored-by: Codex <[email protected]>
Summary:
- Add --scenario flag as alias for --scenario-id with name matching support
- Support comma-separated scenario IDs/names in --scenario flag
- Match scenarios by both id and name (e.g., --scenario "Refund smoke question")
- Improve error message to list available scenarios with IDs and names
- Add list command to display available scenarios for discovery
- Extend selectDashboardScenarios with same name-matching logic
- Fix db.test.ts expected error message to match new format
Rationale:
- Users want to run a single scenario without running the entire file.
- Matching by name is more ergonomic than requiring exact kebab-case IDs.
- The list command helps users discover scenario IDs/names before running.
- Comma-separated values allow running multiple specific scenarios.
Tests:
- bun run test: 82 pass, 0 fail
- bun run test:e2e: 15 pass, 0 fail
- Added unit test for scenarioId filtering by name in runSuite
- Added unit test for comma-separated scenario selection
- Added e2e test for --scenario flag alias
- Added e2e test for name-based scenario matching
Co-authored-by: Codex <[email protected]>
Summary:
- Add `FailureKind = "agent" | "harness"` type to contracts; extend
`RubricScore`, `ScenarioRunResult`, and `ScenarioRecord` with
`failureKind` field
- Update judge JSON schema to require `failure_kind` (enum: agent |
harness); instruct judge to set "harness" when the test infra caused
the failure (e.g. persona didn't pass required files), "agent"
otherwise
- Propagate `failureKind` from judge score through `run-suite.ts` into
`ScenarioRunResult`
- Bump DB schema to v3; add `failure_kind text` column to
`scenario_runs`, `scenario_harness_failed_count` column to `runs`,
and additive migration from v2 → v3 via `ensureColumn`
- Track `scenarioHarnessFailedCount` in `refreshRunCounts`; persist
`failure_kind` in `recordScenarioFinished`
- Extend `DashboardScenarioState.status` with `"harness_fail"`;
add `harness_failed` to `DashboardStateSnapshot`; surface
`failure_kind` in `DashboardScenarioDetail`
- Update `statusTone` / `statusLabel` in render-report to emit
`"warning"` / `"HARNESS FAIL"` for harness failures; add amber
badge and score-bar colour; add dedicated "Harness Failures" card
in the run header (separate from "Agent Failures")
Rationale:
- Binary pass/fail obscured whether a failing run reflected a real
agent defect or a broken test harness; this classification lets
teams filter harness noise from true regressions
- Schema migration is additive (ALTER TABLE ADD COLUMN with default)
so existing databases upgrade without data loss
- Keeping `failure_kind` in the judge output JSON means it is
archived alongside the rubric score for historical queries
Tests:
- bun run test: 78 pass, 0 fail
- bun run docs:validate: all checks passed
- Updated db.test.ts, report.test.ts, and support.ts assertions to
cover new schema version (3) and `scenarioHarnessFailedCount`
Co-authored-by: Codex <[email protected]>
* fix: join SSE content with spaces and collapse duplicates
SSE stream chunks joined with newlines produced unnatural text breaks.
Switch to space-join with duplicate space collapse for cleaner output.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: update SSE join expectation in adapter test
Align test with the newline-to-space join change in http-endpoint.ts.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Add `attachments` field to UserTurn schema, upload files via the
workspace API before sending chat messages, and pass file_ids in
the request body. Includes fixture CSVs and a supplier agreement
for scenarios 7, 21, and 27.
Also includes lint autofixes from biome across touched files.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Audit of changed lines vs origin/main found 159 uncovered executable lines
across runner.py, dashboard.py, cli.py, judge.py, report.py, scenarios.py,
and db.py. This commit adds targeted tests so every line we touched on this
branch runs under pytest, to guarantee the forthcoming TS port has 1-1
behavioral coverage to mirror.
New tests:
- tests/test_dashboard.py: DashboardState event handling, _serialize_state
with and without DB-backed aggregation (dimension scores, failure modes,
judge notes), _DashboardHandler GET routing, and end-to-end
DashboardServer start/shutdown.
- tests/test_scenarios_data.py: parse_time_offset branches and defaults
injection into missing scenario context.
- tests/test_runner.py: display_id iteration suffix, invalid base_date
warning, dry-run short-circuit, missing/unknown persona & rubric config
errors, _make_adapter_factory fallback through build_endpoint_adapter,
response_must_not_contain checkpoint, scoring_overrides below/above
auto-fail, parallel scenario_error progress event.
- tests/test_cli.py: --dashboard happy path, --dashboard missing-build
fallback, verbose log levels, --repeat forwarding.
- tests/test_judge.py: AuthenticationError wrap, non-retryable 4xx,
retry-on-5xx, retry-on-429, retry-on-invalid-JSON, exhausted retries.
- tests/test_report.py: session boundary parsing + role labeling.
- tests/test_db.py: latest_run_for_suite naive-datetime cutoff.
Also bundles the in-tree cli.py/dashboard.py refactor from file-write
dashboard to HTTP-server dashboard (DashboardServer); the new tests
exercise the server path. Tags the unreachable defensive return in
parse_time_offset with `pragma: no cover`.
Diff-vs-main coverage: 400/400 executable lines (100%).
Full suite: 143 passed.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
run_suite previously short-circuited to AUTOGPT_USER_ID (if set) when
pinning a user_id per scenario+iteration, which collapsed every repeat
iteration onto a single memory graph and invalidated averages across
runs. Always mint a fresh uuid per (iteration, scenario) instead; the
env-var escape hatch still exists in autogpt_auth.resolve_auth for
single-shot dev flows that bypass run_suite.
Add a regression test that sets AUTOGPT_USER_ID and asserts --repeat 3
produces three distinct user_ids.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Add Session model with id, time_offset, and reset policy (none/new/fresh_agent)
- Add FailureMode, ResetPolicy, and tester_note to scenario expectations
- Add persona/rubric to ScenarioDefaults for multi-session YAML format
- Implement multi-session execution loop in runner with session boundary tracking
- Support fresh_agent reset (new adapter instance) and new reset (clear context)
- Maintain separate session_transcript and full_transcript for proper judge context
- Add session boundary markers to full_transcript for HTML report visibility
- Add multi-session-memory rubric (retention, fabrication, context separation, temporal)
- Fix dry_run alignment in endpoint yaml and tests
- Parse failure_modes YAML shorthand (- name: description)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Swifty