anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-23 02:36:57 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

Epic: add-regression-detection #9110

New Issue
Open
opened 2026-02-16 18:11:40 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 18:11:40 -05:00
Owner
Copy Link
Copy Source

Originally created by @florianleibert on GitHub (Feb 11, 2026).

Originally assigned to: @thdxr on GitHub.

Epic: add-regression-detection

Overview

Add a built-in quality monitoring system to opencode. Every AI response is scored 0-100 using code correctness, user feedback, and static analysis signals. Scores are stored in a per-project SQLite database (via bun:sqlite). The system detects regressions by comparing against rolling averages, generates diagnostic reasoning for low scores, and exposes a /quality-stats command with dashboard summary and drill-down. A toggleable log file provides full diagnostic tracing.

Architecture Decisions

  • bun:sqlite for storage — Bun has built-in SQLite support, zero additional dependencies needed. Per-project DB stored at .opencode/quality.db (configurable).
  • Event bus integration — Subscribe to existing Session.Event.Updated, Session.Event.Diff, LSPClient.Event.Diagnostics, and Command.Event.Executed events. No modifications to existing event producers needed.
  • Async scoring pipeline — Scoring runs in a microtask after response delivery via queueMicrotask() or setTimeout(0) to ensure zero impact on response latency.
  • New src/quality/ module — Self-contained module following existing patterns (Instance.state() for per-project state, Zod schemas for type safety, Bus.subscribe() for event collection).
  • Config extension — Add quality key to Config.Info schema following existing patterns (e.g., similar to compaction or experimental sections).
  • /quality-stats as a slash command — Register via the existing command system (src/command/) as a built-in command, rendering formatted output to the TUI.

Technical Approach

New Module: src/quality/

packages/opencode/src/quality/
  index.ts        — Public API, Instance.state() init, event subscriptions
  schema.ts       — Zod schemas (ResponseScore, SessionScore, Regression, Config)
  store.ts        — SQLite wrapper (init, migrations, CRUD, queries, retention cleanup)
  scorer.ts       — Scoring engine (sub-score computation, composite weighting)
  collector.ts    — Event listeners, signal aggregation, user feedback inference
  detector.ts     — Regression detection (rolling averages, severity, diagnostic reasoning)
  logger.ts       — Diagnostic log file (rotation, levels, formatting)

Data Flow

Response delivered → Session.Event.Updated fires
    → collector.ts captures: modelID, providerID, tokens, cost, tool results, errors
    → scorer.ts computes sub-scores asynchronously:
        - correctness: LSP diagnostics delta, tool success/failure, error presence
        - userFeedback: deferred — updated when user accepts/reverts/redoes
        - staticAnalysis: LSP diagnostic count delta, severity-weighted
    → store.ts persists to SQLite
    → detector.ts compares against rolling average, flags regressions
    → logger.ts writes to log file
    → If regression severity >= alertLevel, emit Quality.Event.Regression for TUI warning

SQLite Schema

CREATE TABLE responses (
  id TEXT PRIMARY KEY,
  session_id TEXT NOT NULL,
  model_id TEXT NOT NULL,
  provider_id TEXT NOT NULL,
  timestamp INTEGER NOT NULL,
  prompt_hash TEXT,
  response_hash TEXT,
  composite_score REAL,
  correctness_score REAL,
  feedback_score REAL,
  analysis_score REAL,
  diagnostic_notes TEXT,  -- JSON
  is_collaborative INTEGER DEFAULT 0,
  tokens_input INTEGER,
  tokens_output INTEGER,
  cost REAL
);

CREATE TABLE regressions (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  response_id TEXT NOT NULL REFERENCES responses(id),
  detected_at INTEGER NOT NULL,
  score_delta REAL NOT NULL,
  severity TEXT NOT NULL,  -- minor|moderate|severe
  component TEXT,          -- correctness|feedback|analysis|composite
  description TEXT
);

CREATE INDEX idx_responses_session ON responses(session_id);
CREATE INDEX idx_responses_model ON responses(model_id);
CREATE INDEX idx_responses_timestamp ON responses(timestamp);
CREATE INDEX idx_regressions_severity ON regressions(severity);

Sessions and model aggregates computed via SQL queries (no denormalized tables needed — SQLite is fast enough for local data).

User Feedback Signal Collection

Infer implicit feedback from observable user actions:

  • Accept: User continues working after response (no revert, no redo) — positive signal
  • Revert: Session.Info.revert is populated — strong negative signal
  • Redo: User sends a correction prompt referencing the same topic — moderate negative signal (detected via prompt similarity heuristic)
  • Edit after apply: User modifies files that were just changed by the AI — mild negative signal

Feedback scores are updated retroactively in the database as signals arrive.

Regression Detection Algorithm

  1. Fetch rolling window (last N responses, default 20) from SQLite
  2. Compute rolling average for composite and each sub-score
  3. Calculate delta: current_score - rolling_average
  4. Classify severity:
    • minor: delta between -5 and -15
    • moderate: delta between -15 and -30
    • severe: delta below -30
  5. Generate diagnostic reasoning by identifying which sub-score(s) dropped and correlating with observable signals (e.g., "3 new type errors detected", "user reverted previous change")

/quality-stats Command

Registered as built-in command. Output modes:

Default summary — formatted terminal table:

Quality Dashboard (last 30 days)
━━━━━━━���━━━━━━━━━━━━━━━━━━━━━━━
Overall: 78/100 (▲ +3 from last week)

Model Comparison:
  claude-opus-4-6    82 avg  (142 responses)  ▲ +2
  claude-sonnet-4-5  74 avg  (89 responses)   ▼ -1

Recent Regressions:
  ⚠️  moderate  2h ago  correctness dropped (3 type errors)
  ●   minor    1d ago  user reverted change

This Session: 81/100 (4 responses)

--session <id> — per-response score timeline with diagnostic notes
--model <id> — score distribution and trend for specific model
--export json|csv — machine-readable export

Configuration

Extend Config.Info with:

quality: z.object({
  enabled: z.boolean().default(true),
  logging: z.boolean().default(true),
  logLevel: z.enum(["minimal", "normal", "verbose"]).default("normal"),
  logPath: z.string().default(".opencode/quality.log"),
  dbPath: z.string().default(".opencode/quality.db"),
  retentionDays: z.number().default(90),
  weights: z.object({
    correctness: z.number().default(40),
    userFeedback: z.number().default(35),
    staticAnalysis: z.number().default(25),
  }).default({}),
  regression: z.object({
    threshold: z.number().default(15),
    windowSize: z.number().default(20),
    alertLevel: z.enum(["minor", "moderate", "severe"]).default("moderate"),
  }).default({}),
}).optional()

Implementation Strategy

Phase 1 (Foundation): Storage + config + scoring engine — get data flowing into SQLite
Phase 2 (Intelligence): Regression detection + diagnostic reasoning + logging
Phase 3 (Presentation): /quality-stats command + TUI regression warnings

Each phase is independently testable and delivers incremental value.

Task Breakdown Preview

  • Task 1: SQLite storage layer — Schema, migrations, CRUD operations, retention cleanup, bun:sqlite integration
  • Task 2: Quality config — Extend Config.Info with quality section, defaults, validation
  • Task 3: Scoring engine — Sub-score computation (correctness, feedback, static analysis), composite weighting, async execution
  • Task 4: Event collector — Subscribe to session/LSP/command events, aggregate signals, infer user feedback
  • Task 5: Regression detection — Rolling average comparison, severity classification, diagnostic reasoning generation
  • Task 6: Diagnostic logging — Log file writer, rotation, level filtering, formatting
  • Task 7: /quality-stats command — Summary dashboard, session drill-down, model drill-down, JSON/CSV export
  • Task 8: Tests — Unit tests for scorer, detector, store; integration test for full pipeline

Dependencies

Internal

  • src/bus/ — Event subscriptions (no changes needed)
  • src/session/ — Message metadata access (no changes needed)
  • src/lsp/ — Diagnostic data (no changes needed)
  • src/provider/ — Model/provider identification (no changes needed)
  • src/config/config.ts — Add quality config key (schema extension)
  • src/command/index.ts — Register /quality-stats command

External

  • bun:sqlite — Built-in, no package installation needed

Success Criteria (Technical)

  • Scoring adds < 5ms to response processing (async, non-blocking)
  • SQLite queries for /quality-stats summary complete in < 200ms with 10K responses
  • Database stays under 50MB for 1 year of daily use
  • Zero test failures in existing test suite after integration
  • Scoring failures are caught and logged without affecting main workflow
  • All 6 PRD user stories have corresponding implementation coverage

Estimated Effort

  • 8 tasks total, each independently implementable
  • Tasks 1-4 (foundation) can be parallelized partially (1+2 in parallel, then 3+4)
  • Tasks 5-6 (intelligence) depend on 1-4
  • Task 7 (presentation) depends on 1-5
  • Task 8 (tests) runs alongside all phases
  • Critical path: Storage → Scoring → Detection → Stats command

Tasks Created

  • #13202 - SQLite storage layer (parallel: true)
  • #13207 - Quality configuration (parallel: true)
  • #13208 - Scoring engine (parallel: false, depends: #13202, #13207)
  • #13209 - Event collector and user feedback tracking (parallel: false, depends: #13202, #13207, #13208)
  • #13203 - Regression detection and diagnostic reasoning (parallel: false, depends: #13202, #13208)
  • #13204 - Diagnostic logging (parallel: true, depends: #13207, #13208)
  • #13205 - /quality-stats command (parallel: false, depends: #13202-#13208, #13203)
  • #13206 - Tests for quality monitoring system (parallel: false, depends: all)

Total tasks: 8
Parallel tasks: 3 (001, 002, 006)
Sequential tasks: 5 (003, 004, 005, 007, 008)
Estimated total effort: 51-63 hours

Originally created by @florianleibert on GitHub (Feb 11, 2026). Originally assigned to: @thdxr on GitHub. # Epic: add-regression-detection ## Overview Add a built-in quality monitoring system to opencode. Every AI response is scored 0-100 using code correctness, user feedback, and static analysis signals. Scores are stored in a per-project SQLite database (via `bun:sqlite`). The system detects regressions by comparing against rolling averages, generates diagnostic reasoning for low scores, and exposes a `/quality-stats` command with dashboard summary and drill-down. A toggleable log file provides full diagnostic tracing. ## Architecture Decisions - **`bun:sqlite`** for storage — Bun has built-in SQLite support, zero additional dependencies needed. Per-project DB stored at `.opencode/quality.db` (configurable). - **Event bus integration** — Subscribe to existing `Session.Event.Updated`, `Session.Event.Diff`, `LSPClient.Event.Diagnostics`, and `Command.Event.Executed` events. No modifications to existing event producers needed. - **Async scoring pipeline** — Scoring runs in a microtask after response delivery via `queueMicrotask()` or `setTimeout(0)` to ensure zero impact on response latency. - **New `src/quality/` module** — Self-contained module following existing patterns (`Instance.state()` for per-project state, Zod schemas for type safety, `Bus.subscribe()` for event collection). - **Config extension** — Add `quality` key to `Config.Info` schema following existing patterns (e.g., similar to `compaction` or `experimental` sections). - **`/quality-stats` as a slash command** — Register via the existing command system (`src/command/`) as a built-in command, rendering formatted output to the TUI. ## Technical Approach ### New Module: `src/quality/` ``` packages/opencode/src/quality/ index.ts — Public API, Instance.state() init, event subscriptions schema.ts — Zod schemas (ResponseScore, SessionScore, Regression, Config) store.ts — SQLite wrapper (init, migrations, CRUD, queries, retention cleanup) scorer.ts — Scoring engine (sub-score computation, composite weighting) collector.ts — Event listeners, signal aggregation, user feedback inference detector.ts — Regression detection (rolling averages, severity, diagnostic reasoning) logger.ts — Diagnostic log file (rotation, levels, formatting) ``` ### Data Flow ``` Response delivered → Session.Event.Updated fires → collector.ts captures: modelID, providerID, tokens, cost, tool results, errors → scorer.ts computes sub-scores asynchronously: - correctness: LSP diagnostics delta, tool success/failure, error presence - userFeedback: deferred — updated when user accepts/reverts/redoes - staticAnalysis: LSP diagnostic count delta, severity-weighted → store.ts persists to SQLite → detector.ts compares against rolling average, flags regressions → logger.ts writes to log file → If regression severity >= alertLevel, emit Quality.Event.Regression for TUI warning ``` ### SQLite Schema ```sql CREATE TABLE responses ( id TEXT PRIMARY KEY, session_id TEXT NOT NULL, model_id TEXT NOT NULL, provider_id TEXT NOT NULL, timestamp INTEGER NOT NULL, prompt_hash TEXT, response_hash TEXT, composite_score REAL, correctness_score REAL, feedback_score REAL, analysis_score REAL, diagnostic_notes TEXT, -- JSON is_collaborative INTEGER DEFAULT 0, tokens_input INTEGER, tokens_output INTEGER, cost REAL ); CREATE TABLE regressions ( id INTEGER PRIMARY KEY AUTOINCREMENT, response_id TEXT NOT NULL REFERENCES responses(id), detected_at INTEGER NOT NULL, score_delta REAL NOT NULL, severity TEXT NOT NULL, -- minor|moderate|severe component TEXT, -- correctness|feedback|analysis|composite description TEXT ); CREATE INDEX idx_responses_session ON responses(session_id); CREATE INDEX idx_responses_model ON responses(model_id); CREATE INDEX idx_responses_timestamp ON responses(timestamp); CREATE INDEX idx_regressions_severity ON regressions(severity); ``` Sessions and model aggregates computed via SQL queries (no denormalized tables needed — SQLite is fast enough for local data). ### User Feedback Signal Collection Infer implicit feedback from observable user actions: - **Accept**: User continues working after response (no revert, no redo) — positive signal - **Revert**: `Session.Info.revert` is populated — strong negative signal - **Redo**: User sends a correction prompt referencing the same topic — moderate negative signal (detected via prompt similarity heuristic) - **Edit after apply**: User modifies files that were just changed by the AI — mild negative signal Feedback scores are updated retroactively in the database as signals arrive. ### Regression Detection Algorithm 1. Fetch rolling window (last N responses, default 20) from SQLite 2. Compute rolling average for composite and each sub-score 3. Calculate delta: `current_score - rolling_average` 4. Classify severity: - **minor**: delta between -5 and -15 - **moderate**: delta between -15 and -30 - **severe**: delta below -30 5. Generate diagnostic reasoning by identifying which sub-score(s) dropped and correlating with observable signals (e.g., "3 new type errors detected", "user reverted previous change") ### `/quality-stats` Command Registered as built-in command. Output modes: **Default summary** — formatted terminal table: ``` Quality Dashboard (last 30 days) ━━━━━━━���━━━━━━━━━━━━━━━━━━━━━━━ Overall: 78/100 (▲ +3 from last week) Model Comparison: claude-opus-4-6 82 avg (142 responses) ▲ +2 claude-sonnet-4-5 74 avg (89 responses) ▼ -1 Recent Regressions: ⚠️ moderate 2h ago correctness dropped (3 type errors) ● minor 1d ago user reverted change This Session: 81/100 (4 responses) ``` **`--session <id>`** — per-response score timeline with diagnostic notes **`--model <id>`** — score distribution and trend for specific model **`--export json|csv`** — machine-readable export ### Configuration Extend `Config.Info` with: ```typescript quality: z.object({ enabled: z.boolean().default(true), logging: z.boolean().default(true), logLevel: z.enum(["minimal", "normal", "verbose"]).default("normal"), logPath: z.string().default(".opencode/quality.log"), dbPath: z.string().default(".opencode/quality.db"), retentionDays: z.number().default(90), weights: z.object({ correctness: z.number().default(40), userFeedback: z.number().default(35), staticAnalysis: z.number().default(25), }).default({}), regression: z.object({ threshold: z.number().default(15), windowSize: z.number().default(20), alertLevel: z.enum(["minor", "moderate", "severe"]).default("moderate"), }).default({}), }).optional() ``` ## Implementation Strategy **Phase 1 (Foundation)**: Storage + config + scoring engine — get data flowing into SQLite **Phase 2 (Intelligence)**: Regression detection + diagnostic reasoning + logging **Phase 3 (Presentation)**: `/quality-stats` command + TUI regression warnings Each phase is independently testable and delivers incremental value. ## Task Breakdown Preview - [ ] Task 1: SQLite storage layer — Schema, migrations, CRUD operations, retention cleanup, `bun:sqlite` integration - [ ] Task 2: Quality config — Extend `Config.Info` with `quality` section, defaults, validation - [ ] Task 3: Scoring engine — Sub-score computation (correctness, feedback, static analysis), composite weighting, async execution - [ ] Task 4: Event collector — Subscribe to session/LSP/command events, aggregate signals, infer user feedback - [ ] Task 5: Regression detection — Rolling average comparison, severity classification, diagnostic reasoning generation - [ ] Task 6: Diagnostic logging — Log file writer, rotation, level filtering, formatting - [ ] Task 7: `/quality-stats` command — Summary dashboard, session drill-down, model drill-down, JSON/CSV export - [ ] Task 8: Tests — Unit tests for scorer, detector, store; integration test for full pipeline ## Dependencies ### Internal - `src/bus/` — Event subscriptions (no changes needed) - `src/session/` — Message metadata access (no changes needed) - `src/lsp/` — Diagnostic data (no changes needed) - `src/provider/` — Model/provider identification (no changes needed) - `src/config/config.ts` — Add `quality` config key (schema extension) - `src/command/index.ts` — Register `/quality-stats` command ### External - `bun:sqlite` — Built-in, no package installation needed ## Success Criteria (Technical) - Scoring adds < 5ms to response processing (async, non-blocking) - SQLite queries for `/quality-stats` summary complete in < 200ms with 10K responses - Database stays under 50MB for 1 year of daily use - Zero test failures in existing test suite after integration - Scoring failures are caught and logged without affecting main workflow - All 6 PRD user stories have corresponding implementation coverage ## Estimated Effort - **8 tasks** total, each independently implementable - Tasks 1-4 (foundation) can be parallelized partially (1+2 in parallel, then 3+4) - Tasks 5-6 (intelligence) depend on 1-4 - Task 7 (presentation) depends on 1-5 - Task 8 (tests) runs alongside all phases - **Critical path**: Storage → Scoring → Detection → Stats command ## Tasks Created - [ ] #13202 - SQLite storage layer (parallel: true) - [ ] #13207 - Quality configuration (parallel: true) - [ ] #13208 - Scoring engine (parallel: false, depends: #13202, #13207) - [ ] #13209 - Event collector and user feedback tracking (parallel: false, depends: #13202, #13207, #13208) - [ ] #13203 - Regression detection and diagnostic reasoning (parallel: false, depends: #13202, #13208) - [ ] #13204 - Diagnostic logging (parallel: true, depends: #13207, #13208) - [ ] #13205 - /quality-stats command (parallel: false, depends: #13202-#13208, #13203) - [ ] #13206 - Tests for quality monitoring system (parallel: false, depends: all) Total tasks: 8 Parallel tasks: 3 (001, 002, 006) Sequential tasks: 5 (003, 004, 005, 007, 008) Estimated total effort: 51-63 hours
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#9110
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?