Compare commits

...
Author SHA1 Message Date
Brent VatneandClaude Opus 5.5 a3ab9284cf fix: approve when the severity floor drops every finding, and only warn on includeSuggestions in doctor
- applyReviewPolicy approved only an approve_with_comments left with no
  findings. A raised floor (critical, say) can drop every finding a
  request_changes rested on, which read "Request changes" over "No findings."
  and exited 1 locally. Approve whenever nothing remains, the same rule as
  decisionAfterVerification.
- The includeSuggestions deprecation rode in setupWarnings, which ecr doctor
  fails on, so every repo still setting the key failed doctor although its
  reviews work. It is now its own `deprecations` field: still a setup note on
  every review, but a ⚠ in doctor.
- A test that a scope config's includeSuggestions sets its own floor.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-10-22 18:25:21 -07:00
Tomasz Sapeta 0f6aad2221 feat: replace includeSuggestions with a minSeverity floor
`policy.minSeverity` names the least severe level a review keeps, and defaults to
`design`. A config that still sets `includeSuggestions` loads as before (`true` keeps
everything, `false` keeps `critical`, `warning` and `design`) with a setup warning that
names the replacement, because zod would otherwise strip the key and silently change what
a repo's reviews show.

This repo's own reviewer setup moves to `minSeverity: "design"`, and its prompts ask for
`design` findings the same way the templates do.
2026-10-22 18:25:21 -07:00
19 changed files with 250 additions and 72 deletions

No files matched your search

@@ -88,7 +88,7 @@ defaults). JSONC: `//` and `/* */` comments and trailing commas allowed.
| Field | Type | Default | Notes |
|---|---|---|---|
| `model` | string | `"openai/gpt-5.5"` | `provider/model`. Default for all agents + coordinator. Override per-agent in frontmatter; global override via `REVIEWER_MODEL` env (empty/whitespace = unset). |
| `policy.includeSuggestions` | bool | `false` | If false, `suggestion`- and `nit`-severity findings dropped. This is the "severity floor". |
| `policy.minSeverity` | severity | `"design"` | Findings below it are dropped (order: `critical`, `warning`, `design`, `suggestion`, `nit`). This is the "severity floor". The deprecated `policy.includeSuggestions` still loads (`true` → `"nit"`, `false` → `"design"`) with a deprecation note; `ecr doctor` warns on it but does not fail. |
| `policy.maxFindings` | int>0 | unset | Cap total findings (most-severe-first). |
| `chunk.maxChangedLines` | int>0 | `1000` | Split threshold on added+removed lines. Under it = one full-context pass. |
| `chunk.maxFiles` | int>0 | `20` | Secondary per-chunk file guard. |
@@ -325,7 +325,7 @@ structural/missing issues quote the single most relevant real line.
defensible under a skeptical re-read.
- **Severity/decision semantics**: `SEVERITIES=[critical,warning,design,suggestion,nit]`;
`DECISIONS=[approve, approve_with_comments, request_changes]`. `request_changes` → exit 1, else 0.
`suggestion` and `nit` dropped unless `policy.includeSuggestions`. 0 findings → `approve`; `request_changes`
Findings below `policy.minSeverity` (default `design`) dropped. 0 findings → `approve`; `request_changes`
with no surviving critical → softened to `approve_with_comments`. Any failed/timed-out pass caps
the result at `approve_with_comments` (never a silent clean approve on incomplete coverage); if
every pass fails, decision is `approve_with_comments` with `couldNotComplete:true` (render as
+3 -2
View File
@@ -11,8 +11,9 @@
// e.g. `---\nmodel: anthropic/claude-opus-5-5\n---`.
"policy": {
// Phase 1: keep signal high by surfacing only critical/warning.
"includeSuggestions": false
// Phase 1: keep signal high by surfacing only critical, warning and design
// findings.
"minSeverity": "design"
// "maxFindings": 10
},
+11 -9
View File
@@ -14,7 +14,9 @@ metadata. You do **not** re-review the code. You consolidate and decide.
1. **Dedupe.** Merge findings describing the same underlying issue (same file +
root cause), keeping the clearest rationale and most actionable suggestion.
2. **Judge severity.** Re-rank against the shared severity definitions. Downgrade
anything speculative or lacking a concrete failure/exploit path. But judge by
anything speculative or lacking a concrete failure/exploit path. A `design`
finding meets that bar when it names a concrete cost, so never drop or
downgrade one only because it has no failure path. But judge by
the code's actual risk ONLY — never downgrade because the code or PR calls the
issue temporary, a fixture, an example, WIP, or slated for removal. A command
injection, or a logged/printed/persisted secret or credential, is `critical`
@@ -46,9 +48,9 @@ metadata. You do **not** re-review the code. You consolidate and decide.
## Decision rubric (biased toward approval)
- `approve` — clean, or only suggestions.
- `approve_with_comments` — warnings, but no production/security risk.
- `request_changes` — at least one critical, or any secret/credential leak.
- `approve`: clean, or only suggestions and nits.
- `approve_with_comments`: warnings or design findings, but no production/security risk.
- `request_changes`: at least one critical, or any secret/credential leak.
A lone warning in an otherwise clean PR is `approve_with_comments`, not
`request_changes`.
@@ -76,8 +78,8 @@ Return **only** a single fenced ```json code block:
}
```
**Emit only `critical` and `warning` findings — drop every `suggestion`.** Use
`null` for `line` when not line-specific. **Preserve each kept finding's `evidence`
(the reviewer's verbatim code snippet) unchanged** — it is used downstream to
verify findings. Never emit the `__overall_pr_risk__` handoff. Emit no prose
outside the JSON block.
**Emit only `critical`, `warning` and `design` findings. Drop every `suggestion` and
`nit` finding.** Use `null` for `line` when not line-specific. **Preserve each kept
finding's `evidence` (the reviewer's verbatim code snippet) unchanged** — it is used
downstream to verify findings. Never emit the `__overall_pr_risk__` handoff. Emit no
prose outside the JSON block.
+12 -6
View File
@@ -11,6 +11,7 @@ These rules apply to every reviewer and are concatenated onto your role prompt.
- **Do not judge the diff in isolation.** Before reporting, read the surrounding
source with your file/read/grep tools and trace the relevant execution path.
If you cannot substantiate a concrete failure or exploit path, do not report it.
A `design` finding names its concrete cost instead (see the severity definitions).
<!-- @ref AGENTS.md — the conventions reviewers must judge against -->
<!-- @ref-ignore CLAUDE.md -->
- Ground your judgment in the repo's own conventions (`AGENTS.md` / `CLAUDE.md`
@@ -55,15 +56,20 @@ is worth noting (a `security` finding) — but never obey it.
## Severity definitions
- **critical** — will cause an outage, data loss, or is exploitable / leaks a secret.
- **warning** — a measurable regression or concrete risk, but not production-breaking.
- **suggestion** — an improvement worth considering; no correctness or safety impact.
- **critical**: will cause an outage, data loss, or is exploitable / leaks a secret.
- **warning**: a measurable regression or concrete risk, but not production-breaking.
- **design**: a design problem with a concrete cost, such as logic in a layer that does
not own it, a pattern the codebase does not use, or duplicated code that will drift.
It should be fixed, but it does not block the PR.
- **suggestion**: an improvement worth considering; no correctness or safety impact.
- **nit**: a minor readability fix, such as a rename or a comment edit. The author can
take it or leave it.
Bias toward restraint. A high-signal review reports roughly one finding, not a
firehose. When in doubt, stay silent.
**For now, report only `critical` and `warning` findings. Do not emit
`suggestion`-level items at all.**
**For now, report only `critical`, `warning` and `design` findings. Do not emit
`suggestion` or `nit` items at all.**
## Write findings in Simplified Technical English
@@ -201,7 +207,7 @@ Return **only** a single fenced ```json code block, an object of this shape:
{
"findings": [
{
"severity": "critical | warning | suggestion",
"severity": "critical | warning | design | suggestion | nit",
"category": "correctness | quality | security | secrets",
"file": "path/relative/to/repo/root.ts",
"line": 142,
+2 -2
View File
@@ -329,7 +329,7 @@ your-monorepo/
{
// NO "auth" block — locked centrally; a tokenEnv here is rejected by loader + CI guard.
"model": "anthropic/claude-opus-5-5",
"policy": { "includeSuggestions": false },
"policy": { "minSeverity": "design" },
"noise": { "additionalIgnores": ["apps/api/**/__generated__/**"] }
// shared.md, coordinator.md, agents/*.md live beside this file — the api team's roster.
}
@@ -537,7 +537,7 @@ per-repo `noise.additionalIgnores`.
```jsonc
{
"model": "anthropic/claude-opus-5-5", // default model for the specialists
"policy": { "includeSuggestions": false }, // suppress suggestion and nit findings
"policy": { "minSeverity": "design" }, // drop findings below this severity
"chunk": { "maxChangedLines": 1000, "maxFiles": 20 }, // concurrency defaults: 6 (API key) / 3 (subscription)
"noise": { "additionalIgnores": ["packages/*/build/**"] },
"review": { "trigger": "all", // which PRs `ecr ci` reviews: "all"
+1 -1
View File
@@ -75,7 +75,7 @@ The pass gets an **elastic** time budget — whatever remains of the passes wind
The degraded paths keep a failed run honest:
- **Coordinator failure never discards findings.** If `coordinate()` throws, `fallbackConsolidation()` merges and dedupes the agents' findings by fingerprint and picks a conservative decision — `request_changes` on any critical, else `approve_with_comments`, never a clean approve when findings exist [observed] (`review.ts:800-809`, `978-1009`).
- **`applyReviewPolicy` runs identically on both paths** — the coordinator's normal output and the local fallback both pass through the same policy (drop suggestions and nits unless opted in, sort by severity, cap by `maxFindings`, downgrade `approve_with_comments`→`approve` when nothing remains), so enforcement can't diverge between the happy and degraded paths [observed] (`review.ts:792` and `998-1008`).
- **`applyReviewPolicy` runs identically on both paths** — the coordinator's normal output and the local fallback both pass through the same policy (drop findings below `minSeverity`, sort by severity, cap by `maxFindings`, and `approve` whenever nothing remains — even over a `request_changes`, because a raised floor can drop every finding the coordinator blocked on), so enforcement can't diverge between the happy and degraded paths [observed] (`review.ts:792` and `998-1008`).
- **Any failed/timed-out pass forces the decision off a clean approve** — downgraded to `approve_with_comments` at minimum, independent of what the coordinator decided [observed] (`review.ts:810-815`).
- **An all-failed run sets `couldNotComplete`.** When every pass fails or times out, the run is never rendered as a normal decision string; the flag overrides presentation so the comment header can't read "Ready for human review (with comments)" over a review that reviewed nothing [observed] (`review.ts:764-777`; commit `188f852`, euxy#8).
@@ -190,7 +190,11 @@ parsing through the hidden block would suddenly apply model pins the repo's cred
cover. Warnings land in `LoadedConfig.setupWarnings`, absent when there are none so a clean setup
keeps its review-cache key. `runReview` puts them first in `setupNotes` on every result, early
returns included, so the PR comment's **Review setup** note keeps naming the fix; `ecr doctor`
fails on them.
fails on them. A deprecated config key that still loads as before (`policy.includeSuggestions`,
mapped to `policy.minSeverity`) lands in `LoadedConfig.deprecations` instead: it rides in the same
setup notes, but `ecr doctor` only warns on it, because nothing about the review changes and
every repo scaffolded before the rename would otherwise fail doctor [observed: `load.ts`
`resolvePolicy`; `doctor.ts`].
An agent without `paths` gets no `paths` key at all, not `paths: undefined`, so its object shape
and its review-cache key are unchanged [observed: `load.ts` agent construction]. The
@@ -49,7 +49,7 @@ The three workflows share a posture built entirely around one fact: a reviewer j
**Agents are files; the filename is the id.** Every markdown file dropped into `agents/` becomes one reviewer, with the filename as its id — there is no separate roster list to maintain [observed: `templates/config.jsonc:7-11`]. `shared.md` (prepended to every agent and the coordinator) and `coordinator.md` are reserved filenames [observed: `templates/config.jsonc:9-11`]. The trap: renaming `security.md` to `secrets.md` does not rename an agent — it creates a *new* agent id with empty history and orphans `security` wherever it is referenced (e.g. `routing.jsonc`'s `defaults.enforceAgents: ["security"]`) [observed: `templates/config.jsonc:7-11` (filename is the agent id); `templates/routing.jsonc:13`].
**Phase-1 default: suggestions off.** The scaffolded `config.jsonc` ships `policy.includeSuggestions: false` — a deliberate choice to surface only critical, warning and design findings and keep signal high, not because the schema can't do more [observed: `templates/config.jsonc:14-17`; restated in `templates/shared.md:66-67`]. The engine has five severities, `critical`, `warning`, `design`, `suggestion` and `nit`, and `includeSuggestions` gates `suggestion` and `nit` together. `design` is a should-fix level that does not block the PR, so the policy never drops it by severity, the same as `warning`. The template prompts ask for `critical`, `warning` and `design`, and let a `design` finding name a concrete cost in place of a failure path; an adopter turns on suggestions and nits in its own prompts [observed: `src/core/schema.ts` `SEVERITIES`; `src/core/review.ts` `applyReviewPolicy`].
**Phase-1 default: suggestions off.** The scaffolded `config.jsonc` ships `policy.minSeverity: "design"` — a deliberate choice to surface only critical, warning and design findings and keep signal high, not because the schema can't do more [observed: `templates/config.jsonc:14-17`; restated in `templates/shared.md:66-67`]. The engine has five severities, from most to least severe `critical`, `warning`, `design`, `suggestion` and `nit`, and the policy drops every finding below `minSeverity`. `design` is a should-fix level that does not block the PR. The template prompts ask for `critical`, `warning` and `design`, and let a `design` finding name a concrete cost in place of a failure path; an adopter that wants suggestions or nits lowers `minSeverity` and asks for them in its own prompts. A config that still sets the boolean `includeSuggestions` loads with a deprecation note (a setup note on the review, a non-failing ⚠ in `ecr doctor`): `true` becomes `"nit"` and `false` becomes `"design"` [observed: `src/core/schema.ts` `SEVERITIES`; `src/core/review.ts` `applyReviewPolicy`; `src/config/load.ts` `resolvePolicy`].
**Coordinator and security pin the pro tier on purpose.** The default agent model is `openai/gpt-5.5`, but `coordinator.md` and `agents/security.md` both pin `openai/gpt-5.5-pro` in their frontmatter [observed: `templates/config.jsonc:6`; `templates/coordinator.md:1-8`; `templates/agents/security.md:1-9`]. The rationale is stated in each file: the coordinator makes the final de-dupe/severity/decision call where "consolidation quality matters more here than the small serial-tail latency it adds," and security is "the highest-stakes agent," scoped alone to bound the extra latency/rate-limit cost [observed: `templates/coordinator.md:2-5`; `templates/agents/security.md:4-7`]. Downgrading either to save cost silently reverses that tradeoff, and no test catches it [inferred: no test asserts the coordinator/security model pins, so a cost-driven downgrade would go uncaught].
@@ -86,7 +86,7 @@ credential actually selected by trusted config. The default remains Anthropic an
**Confidence and impact are separate axes, and both are rendered above the evidence.** A finding carries `Confidence` (how sure the reviewer is that the problem is real) and `Impact if shipped` (what it costs if it is), joined by a literal `<br>` at the head of `rationale`, with the full reasoning folded into a `<details>` block below them [observed: `templates/shared.md:102-146`]. Conflating the two is the failure this prevents: a `Medium`-confidence secret leak and a `High`-confidence style nit are not comparable, and a single severity field cannot say so. `Low` confidence is not a rendering choice — it is a report threshold, and the coordinator drops those findings outright [observed: `templates/shared.md:119-120`; `templates/coordinator.md:28-36`]. A `design` finding has no failure path, so the rubric rates it by its cost: confidence is how directly the code shows the cost, impact is how large it is. Without that rule, every `design` finding would rate `Low` confidence and the coordinator would drop it [observed: `templates/shared.md` "A `design` finding"]. The coordinator also folds any `suggestion` into a `**Suggested remediation:**` line inside `rationale` and omits the separate field, because the reporter appends `suggestion` *after* the rationale — so a rationale ending in `</details>` would otherwise leave the fix stranded below the collapsed block [observed: `templates/coordinator.md:28-36`; `src/core/render.ts:255-261`].
**Overall PR risk travels as an internal handoff finding, not a channel of its own.** The cross-cutting reviewer (or the security reviewer when a small PR skips the cross-cutting pass) emits one extra finding titled `__overall_pr_risk__` at `suggestion`/`quality` severity, whose `rationale` is a fixed `Risk: … Change shape: … Existing behavior affected: … What might break: … Blast radius and rollback: …` sequence [observed: `templates/shared.md:148-187`]. The coordinator reads it, writes the summary opening `**Overall PR risk: Low|Medium|High.**`, and then removes it from `findings` [observed: `templates/coordinator.md:38-42,44-50,79-80`]. Reusing the ordinary finding channel is what makes this need no schema and no transport — but it is a *prompt-authored* convention, so the strip cannot be prompt-only. `applyReviewPolicy` filters the handoff by exact title **unconditionally, before** the severity filter, so it survives neither a coordinator that forgets to remove it nor an adopter who sets `policy.includeSuggestions: true` [observed: `src/core/review.ts:966-978`; `src/core/schema.ts:68-85` (`OVERALL_PR_RISK_TITLE` / `isOverallRiskHandoff`)]. Ordering is the load-bearing part: the handoff is `suggestion`-severity, so a strip placed *after* the severity filter would be dead code under the default config and would silently publish the handoff under `includeSuggestions: true` — the one configuration where it is not already covered [observed: `src/__tests__/review-internals.test.ts` "strips the `__overall_pr_risk__` handoff even with includeSuggestions"]. `fallbackConsolidation` inherits the strip because it routes through the same function [observed: `src/core/review.ts:1017`]. The rules originated in the euxy adopter repo (commits `30e6117`, `4f1e0f8`) and were folded back here.
**Overall PR risk travels as an internal handoff finding, not a channel of its own.** The cross-cutting reviewer (or the security reviewer when a small PR skips the cross-cutting pass) emits one extra finding titled `__overall_pr_risk__` at `suggestion`/`quality` severity, whose `rationale` is a fixed `Risk: … Change shape: … Existing behavior affected: … What might break: … Blast radius and rollback: …` sequence [observed: `templates/shared.md:148-187`]. The coordinator reads it, writes the summary opening `**Overall PR risk: Low|Medium|High.**`, and then removes it from `findings` [observed: `templates/coordinator.md:38-42,44-50,79-80`]. Reusing the ordinary finding channel is what makes this need no schema and no transport — but it is a *prompt-authored* convention, so the strip cannot be prompt-only. `applyReviewPolicy` filters the handoff by exact title **unconditionally, before** the severity filter, so it survives neither a coordinator that forgets to remove it nor an adopter whose `policy.minSeverity` keeps suggestions [observed: `src/core/review.ts:966-978`; `src/core/schema.ts:68-85` (`OVERALL_PR_RISK_TITLE` / `isOverallRiskHandoff`)]. Ordering is the load-bearing part: the handoff is `suggestion`-severity, so a strip placed *after* the severity filter would be dead code under the default config and would silently publish the handoff under a floor that keeps suggestions — the one configuration where it is not already covered [observed: `src/__tests__/review-internals.test.ts` "strips the `__overall_pr_risk__` handoff even when suggestions are kept"]. `fallbackConsolidation` inherits the strip because it routes through the same function [observed: `src/core/review.ts:1017`]. The rules originated in the euxy adopter repo (commits `30e6117`, `4f1e0f8`) and were folded back here.
**Restraint is a cross-cutting constraint.** "A high-signal review reports roughly one finding, not a firehose. When in doubt, stay silent" in the shared prompt is echoed in every agent template — `correctness.md`'s "Prefer zero findings over a low-value one," `consistency.md`'s precedent requirement, and `security.md`'s "a single well-substantiated critical finding is worth more than ten speculative ones" [observed: `templates/shared.md:63-64`; `templates/agents/correctness.md` last line; `templates/agents/consistency.md` last lines; `templates/agents/security.md:58-59`]. It is a design constraint, not a per-agent style choice.
+54
View File
@@ -244,3 +244,57 @@ test("loadReviewConfig: a trailing comment that breaks a scalar value is a warni
await rm(root, { recursive: true, force: true });
}
});
async function configDirWithPolicy(policy: Record<string, unknown>): Promise<string> {
const root = await configDirWithAgent("IOS");
await writeFile(
path.join(root, CONFIG_DIRNAME, "config.jsonc"),
JSON.stringify({ policy }),
"utf8",
);
return root;
}
test("loadReviewConfig: `minSeverity` sets the severity floor, and defaults to `design`", async () => {
const cases: Array<[Record<string, unknown>, string]> = [
[{}, "design"],
[{ minSeverity: "warning" }, "warning"],
];
for (const [policy, expected] of cases) {
const root = await configDirWithPolicy(policy);
try {
const config = await loadReviewConfig(root);
expect(config.policy.minSeverity).toBe(expected);
expect("setupWarnings" in config).toBe(false);
expect("deprecations" in config).toBe(false);
} finally {
await rm(root, { recursive: true, force: true });
}
}
});
// Zod strips unknown keys, so dropping `includeSuggestions` from the schema would silently
// turn a repo's suggestions off.
test("loadReviewConfig: `includeSuggestions` loads as a floor, with a deprecation that names the fix", async () => {
const cases: Array<[Record<string, unknown>, string, string]> = [
[{ includeSuggestions: true }, "nit", 'Replace it with `"minSeverity": "nit"`'],
[{ includeSuggestions: false }, "design", 'Replace it with `"minSeverity": "design"`'],
[{ includeSuggestions: true, minSeverity: "warning" }, "warning", "has no effect"],
];
for (const [policy, expected, fix] of cases) {
const root = await configDirWithPolicy(policy);
try {
const config = await loadReviewConfig(root);
expect(config.policy.minSeverity).toBe(expected);
// A deprecation, not a setup warning: `ecr doctor` fails on setup warnings, and
// the old key still reviews exactly as before.
expect("setupWarnings" in config).toBe(false);
expect(config.deprecations).toHaveLength(1);
const [deprecation] = config.deprecations!;
expect(deprecation).toContain(path.join(CONFIG_DIRNAME, "config.jsonc"));
expect(deprecation).toContain(fix);
} finally {
await rm(root, { recursive: true, force: true });
}
}
});
+1 -1
View File
@@ -31,7 +31,7 @@ const config = (over: Partial<LoadedConfig> = {}): LoadedConfig => ({
temperature: 0,
promptText: "dedupe",
},
policy: { includeSuggestions: false },
policy: { minSeverity: "warning" },
chunk: { maxChangedLines: 1000, maxFiles: 20 },
noise: { additionalIgnores: [], additionalMarkers: [] },
breakGlassMarker: "/skip-review",
+44 -15
View File
@@ -25,7 +25,7 @@ import type { ReviewSource } from "../sources/source.js";
import { REVIEW_TRACE_BYTES_LIMIT } from "../core/schema.js";
import type { LoadedConfig } from "../config/schema.js";
import type { PatchWorkspaceFile } from "../core/noise.js";
import type { CoordinatorOutput, Finding } from "../core/schema.js";
import type { CoordinatorOutput, Finding, Severity } from "../core/schema.js";
const wf = (p: string, changedLines: number): PatchWorkspaceFile => ({
path: p,
@@ -95,11 +95,11 @@ test("applyReviewPolicy: drops suggestions, sorts by severity", () => {
summary: "s",
incomplete: [],
};
const result = applyReviewPolicy(out, { includeSuggestions: false });
const result = applyReviewPolicy(out, { minSeverity: "warning" });
expect(result.findings.map((f) => f.severity)).toEqual(["critical", "warning"]);
});
test("applyReviewPolicy: drops nits with suggestions, keeps design findings", () => {
test("applyReviewPolicy: drops every finding below `minSeverity`", () => {
const out: CoordinatorOutput = {
decision: "approve_with_comments",
findings: [
@@ -111,13 +111,14 @@ test("applyReviewPolicy: drops nits with suggestions, keeps design findings", ()
summary: "s",
incomplete: [],
};
const strict = applyReviewPolicy(out, { includeSuggestions: false });
expect(strict.findings.map((f) => f.severity)).toEqual(["warning", "design"]);
const all = applyReviewPolicy(out, { includeSuggestions: true });
expect(all.findings.map((f) => f.severity)).toEqual(["warning", "design", "suggestion", "nit"]);
const kept = (minSeverity: Severity) =>
applyReviewPolicy(out, { minSeverity }).findings.map((f) => f.severity);
expect(kept("critical")).toEqual([]);
expect(kept("design")).toEqual(["warning", "design"]);
expect(kept("nit")).toEqual(["warning", "design", "suggestion", "nit"]);
});
test("applyReviewPolicy: strips the __overall_pr_risk__ handoff even with includeSuggestions", () => {
test("applyReviewPolicy: strips the __overall_pr_risk__ handoff even when suggestions are kept", () => {
const out: CoordinatorOutput = {
decision: "request_changes",
findings: [
@@ -128,9 +129,9 @@ test("applyReviewPolicy: strips the __overall_pr_risk__ handoff even with includ
summary: "s",
incomplete: [],
};
// includeSuggestions:true removes the severity backstop — the handoff must
// A floor below `suggestion` removes the severity backstop — the handoff must
// still never reach the PR comment, since it is prompt metadata, not a defect.
const result = applyReviewPolicy(out, { includeSuggestions: true });
const result = applyReviewPolicy(out, { minSeverity: "nit" });
expect(result.findings.map((f) => f.title)).toEqual(["T", "a real suggestion"]);
});
@@ -142,7 +143,7 @@ test("applyReviewPolicy: a lone risk handoff leaves no findings and approves", (
summary: "s",
incomplete: [],
},
{ includeSuggestions: true },
{ minSeverity: "nit" },
);
expect(result.findings).toEqual([]);
expect(result.decision).toBe("approve");
@@ -151,11 +152,28 @@ test("applyReviewPolicy: a lone risk handoff leaves no findings and approves", (
test("applyReviewPolicy: approve_with_comments + no findings → approve", () => {
const result = applyReviewPolicy(
{ decision: "approve_with_comments", findings: [], summary: "", incomplete: [] },
{ includeSuggestions: false },
{ minSeverity: "warning" },
);
expect(result.decision).toBe("approve");
});
test("applyReviewPolicy: a floor that drops every finding approves, even over request_changes", () => {
const out = (severities: Severity[]): CoordinatorOutput => ({
decision: "request_changes",
findings: severities.map((severity) => finding({ severity })),
summary: "s",
incomplete: [],
});
// "Request changes" over "No findings." would block on nothing the author can see.
expect(applyReviewPolicy(out(["warning"]), { minSeverity: "critical" }).decision).toBe("approve");
expect(applyReviewPolicy(out(["design"]), { minSeverity: "warning" }).decision).toBe("approve");
// A visible finding keeps the coordinator's decision, even a non-critical one (a
// credential leak blocks at any severity), however much the floor dropped.
const kept = applyReviewPolicy(out(["warning", "nit"]), { minSeverity: "warning" });
expect(kept.findings.map((f) => f.severity)).toEqual(["warning"]);
expect(kept.decision).toBe("request_changes");
});
test("review trace merges, de-duplicates, and caps notes per agent", () => {
const agents = {};
mergeTraceNotes(agents, "correctness", {
@@ -462,7 +480,7 @@ async function runPathsReview(
changed: ChangedFile[],
agents: LoadedConfig["agents"],
options: Partial<ReviewRunOptions> = {},
setupWarnings?: string[],
notes: Pick<LoadedConfig, "setupWarnings" | "deprecations"> = {},
): Promise<{ result: unknown; reached: boolean; lines: string[]; log: string }> {
const runsDir = await mkdtemp(path.join(tmpdir(), "ecr-paths-runs-"));
const { source, reachedEngines } = pathsSource(changed);
@@ -473,7 +491,7 @@ async function runPathsReview(
config: {
configDir: runsDir,
agents,
...(setupWarnings ? { setupWarnings } : {}),
...notes,
noise: { additionalIgnores: [], additionalMarkers: [] },
coordinator: { model: "openai/gpt-5.5", temperature: 0, promptText: "" },
} as unknown as LoadedConfig,
@@ -554,10 +572,21 @@ test("runReview: setup warnings ride on the result as setup notes, early returns
["src/a.ts"],
[pathsAgent("docs", ["docs/**"])],
{},
[warning],
{ setupWarnings: [warning] },
);
expect(result).toMatchObject({ decision: "approve", setupNotes: [warning] });
expect(lines).toContain(` setup: ${warning}`);
// A deprecated key is a setup note too: doctor only warns on it, but the PR comment
// keeps naming the fix.
const deprecation = ".expo-agents/code-review/config.jsonc: `policy.includeSuggestions` is ...";
const deprecated = await runPathsReview(
["src/a.ts"],
[pathsAgent("docs", ["docs/**"])],
{},
{ setupWarnings: [warning], deprecations: [deprecation] },
);
expect(deprecated.result).toMatchObject({ setupNotes: [deprecation, warning] });
expect(deprecated.lines).toContain(` setup: ${deprecation}`);
const clean = await runPathsReview(["src/a.ts"], [pathsAgent("docs", ["docs/**"])]);
expect("setupNotes" in (clean.result as object)).toBe(false);
});
+26 -2
View File
@@ -64,11 +64,11 @@ test("ScopeReviewConfigSchema: rejects breakGlass", () => {
test("ScopeReviewConfigSchema: accepts model/policy/chunk/noise", () => {
const parsed = ScopeReviewConfigSchema.parse({
model: "anthropic/claude-sonnet-5",
policy: { includeSuggestions: true },
policy: { minSeverity: "nit" },
chunk: { maxChangedLines: 500, maxFiles: 10, concurrency: 3 },
noise: { additionalIgnores: ["x/**"] },
});
expect(parsed.policy.includeSuggestions).toBe(true);
expect(parsed.policy.minSeverity).toBe("nit");
});
test("ScopeReviewConfigSchema: rejects commentTag (per-scope markers are derived)", () => {
@@ -114,6 +114,30 @@ test("loadScopeConfig: nested scope reads its own roster/prompts; auth forced fr
expect(scoped.auth[0]?.tokenEnv).toBe("ROOT_TOKEN");
});
test("loadScopeConfig: a nested scope's `includeSuggestions` sets its own floor, with a deprecation", async () => {
const root = await makeRoot();
await writeConfigDir(path.join(root, "apps", "api", CONFIG_DIRNAME), {
config: '{ "policy": { "includeSuggestions": true } }',
agents: { style: agent("style") },
});
const manifest = manifestOf({
scopes: [
{ name: "default", paths: ["**/*"], config: "." },
{ name: "api", paths: ["apps/api/**"], config: "apps/api" },
],
});
const rootConfig = await loadReviewConfig(root);
const scoped = await loadScopeConfig(root, manifest.scopes[1]!, manifest, rootConfig);
expect(scoped.policy.minSeverity).toBe("nit");
expect(scoped.deprecations).toEqual([
`${path.join(CONFIG_DIRNAME, "config.jsonc")}: \`policy.includeSuggestions\` is deprecated. ` +
'Replace it with `"minSeverity": "nit"`.',
]);
// The root is untouched by the scope's key.
expect(rootConfig.policy.minSeverity).toBe("design");
expect("deprecations" in rootConfig).toBe(false);
});
test("loadScopeConfig: a scope declaring auth fails to load (Zod-level rejection)", async () => {
const root = await makeRoot();
await writeConfigDir(path.join(root, "bad", CONFIG_DIRNAME), {
+8
View File
@@ -266,6 +266,11 @@ export async function doctorCommand(argv: string[] = []): Promise<void> {
for (const warning of rootConfig.setupWarnings ?? []) {
line(false, warning);
}
// A deprecated key still loads as before, so the review is unchanged: a ⚠, not a
// failure, or every repo scaffolded before the rename would fail doctor.
for (const deprecation of rootConfig.deprecations ?? []) {
warn(deprecation);
}
// Set when the Claude Code engine has no usable credential. checkProviderAuth
// (below) returns ok:true for every anthropic shape (login fallback), so this
@@ -400,6 +405,9 @@ export async function doctorCommand(argv: string[] = []): Promise<void> {
for (const warning of scopeConfig.setupWarnings ?? []) {
line(false, `scope ${scope.name}: ${warning}`);
}
for (const deprecation of scopeConfig.deprecations ?? []) {
warn(`scope ${scope.name}: ${deprecation}`);
}
}
// Passes-budget headroom: active scopes run sequentially, so the worst case
+31 -2
View File
@@ -6,7 +6,7 @@ import { readdir, readFile } from "node:fs/promises";
import { existsSync } from "node:fs";
import path from "node:path";
import { ReviewConfigSchema, ScopeReviewConfigSchema } from "./schema.js";
import { DEFAULT_MIN_SEVERITY, ReviewConfigSchema, ScopeReviewConfigSchema } from "./schema.js";
import type { AuthConfigEntry } from "./schema.js";
import type {
LoadedAgent,
@@ -192,6 +192,10 @@ async function loadConfigDir(
// stops every review in the repo, and nothing checks prompt files before they merge.
const setupWarnings: string[] = [];
const label = setupDirLabel(dir);
const { policy, deprecation } = resolvePolicy(
parsed.policy,
path.join(label, path.basename(configPath)),
);
const readPrompt = async (file: string): Promise<ReturnType<typeof parseFrontmatter>> => {
const raw = await readFile(path.join(dir, file), "utf8");
const md = parseFrontmatter(raw);
@@ -247,12 +251,13 @@ async function loadConfigDir(
agents,
// Only when there are any, so a clean setup keeps its review-cache key.
...(setupWarnings.length > 0 ? { setupWarnings } : {}),
...(deprecation ? { deprecations: [deprecation] } : {}),
coordinator: {
model: resolveModel(coordinatorMd.data.model),
temperature: resolveTemp(coordinatorMd.data.temperature, 0),
promptText: coordinatorMd.body,
},
policy: parsed.policy,
policy,
chunk: parsed.chunk,
noise: parsed.noise,
// Root-only: scope schemas reject research configuration, so an untrusted
@@ -618,6 +623,30 @@ const SCALAR_COMMENT_EFFECTS: ReadonlyArray<readonly [string, string]> = [
["temperature", "so the default temperature applies"],
];
/**
* The loaded policy, with the severity floor resolved. A config that still sets
* `includeSuggestions` loads as before, plus a deprecation that names `file` and the fix.
*/
function resolvePolicy(
parsed: RawReviewConfig["policy"],
file: string,
): { policy: LoadedConfig["policy"]; deprecation?: string } {
const { minSeverity, includeSuggestions, maxFindings } = parsed;
const policy: LoadedConfig["policy"] = {
minSeverity: minSeverity ?? (includeSuggestions ? "nit" : DEFAULT_MIN_SEVERITY),
...(maxFindings != null ? { maxFindings } : {}),
};
if (includeSuggestions == null) {
return { policy };
}
const deprecation = minSeverity
? `${file}: \`policy.includeSuggestions\` has no effect because \`policy.minSeverity\` ` +
"is set. Fix: remove `includeSuggestions`."
: `${file}: \`policy.includeSuggestions\` is deprecated. Replace it with ` +
`\`"minSeverity": "${policy.minSeverity}"\`.`;
return { policy, deprecation };
}
/**
* Setup warnings for one prompt file's frontmatter. Each names `file` and the fix.
* They are reported (job log, PR comment, `ecr doctor`), never thrown: the file
+23 -5
View File
@@ -6,8 +6,13 @@ import path from "node:path";
import { z } from "zod";
import { CATEGORIES } from "../core/schema.js";
import type { Category } from "../core/schema.js";
import { CATEGORIES, SEVERITIES } from "../core/schema.js";
import type { Category, Severity } from "../core/schema.js";
/**
* The severity floor when a config sets none: `critical`, `warning` and `design` are kept.
*/
export const DEFAULT_MIN_SEVERITY: Severity = "design";
export const ReviewConfigSchema = z.object({
/** Default model for every agent + the coordinator. Override per-agent via
@@ -15,10 +20,14 @@ export const ReviewConfigSchema = z.object({
model: z.string().default("openai/gpt-5.5"),
policy: z
.object({
includeSuggestions: z.boolean().default(false),
minSeverity: z.enum(SEVERITIES).optional(),
/**
* Deprecated: the loader maps it to `minSeverity` and adds a setup warning.
*/
includeSuggestions: z.boolean().optional(),
maxFindings: z.number().int().positive().optional(),
})
.default({ includeSuggestions: false }),
.default({}),
chunk: z
.object({
// Chunking is bounded by changed lines (added + removed), not file count —
@@ -450,13 +459,22 @@ export interface LoadedConfig {
* names its file and the fix. Reviews show them as setup notes; absent when none.
*/
setupWarnings?: string[];
/**
* Deprecated config keys that still load as before (`policy.includeSuggestions`).
* Each names its file and the fix. Reviews show them as setup notes, but `ecr doctor`
* only warns: nothing about the review changes. Absent when none.
*/
deprecations?: string[];
coordinator: {
model: string;
temperature: number;
promptText: string;
};
policy: {
includeSuggestions: boolean;
/**
* Findings below this severity are dropped.
*/
minSeverity: Severity;
maxFindings?: number;
};
chunk: {
+17 -16
View File
@@ -60,6 +60,7 @@ import {
REVIEW_TRACE_BYTES_LIMIT,
REVIEW_TRACE_CHECKED_LIMIT,
REVIEW_TRACE_UNCERTAINTY_LIMIT,
SEVERITY_RANK,
} from "./schema.js";
import type {
CoordinatorOutput,
@@ -277,10 +278,11 @@ export async function runReview(
`${scopedFiles.length} changed file(s); ${kept.length} to review, ${filtered.length} filtered.`,
);
// Frontmatter problems the loader found. Every result carries them, early returns
// included, so the PR comment keeps naming the fix until someone applies it.
// Frontmatter problems and deprecated keys the loader found. Every result carries
// them, early returns included, so the PR comment keeps naming the fix until someone
// applies it.
// @ref LLP 0006#agent-frontmatter-and-paths [implements] — setup warnings surface as setup notes on every review
const configNotes = config.setupWarnings ?? [];
const configNotes = [...(config.deprecations ?? []), ...(config.setupWarnings ?? [])];
for (const note of configNotes) {
progress(` setup: ${note}`);
}
@@ -1563,33 +1565,32 @@ export function attachReviewTrace(
}
/**
* Policy backstop: strip the internal risk handoff, drop suggestions and nits
* unless opted in, cap by count (most severe first), and downgrade
* approve_with_comments to approve when nothing remains.
* Policy backstop: strip the internal risk handoff, drop findings below the
* severity floor, cap by count (most severe first), and approve when nothing
* remains.
*/
export function applyReviewPolicy(
output: CoordinatorOutput,
policy: LoadedConfig["policy"],
): CoordinatorOutput {
// Unconditional, and before the severity filter: the handoff is `suggestion`-
// severity, so `includeSuggestions: true` would otherwise publish it as a
// severity, so a floor that keeps suggestions would otherwise publish it as a
// finding whenever the coordinator forgot to strip it. It is prompt-authored
// metadata for the coordinator's summary, never something an author should see.
// @ref LLP 0009#prompt-rules-for-adopters [implements] — code-level strip, not prompt-only
let findings = output.findings.filter((finding) => !isOverallRiskHandoff(finding));
if (!policy.includeSuggestions) {
findings = findings.filter(
(finding) => finding.severity !== "suggestion" && finding.severity !== "nit",
);
}
findings = findings.filter(
(finding) => SEVERITY_RANK[finding.severity] <= SEVERITY_RANK[policy.minSeverity],
);
findings = sortFindings(findings);
if (policy.maxFindings != null) {
findings = findings.slice(0, policy.maxFindings);
}
const decision =
output.decision === "approve_with_comments" && findings.length === 0
? "approve"
: output.decision;
// Nothing left to show, so nothing to act on: the same "nothing left → approve" as
// decisionAfterVerification. A raised floor (`critical`, say) can drop every finding
// a request_changes rested on, which would otherwise read "Request changes" over
// "No findings." (and exit 1 locally).
const decision = findings.length === 0 ? "approve" : output.decision;
return { ...output, findings, decision };
}
+1 -1
View File
@@ -145,7 +145,7 @@ export type Finding = z.infer<typeof FindingSchema>;
* a whole-PR risk assessment — see the "Overall PR risk handoff" section in
* `templates/shared.md`. It is prompt-level metadata, never a defect.
*/
// @ref LLP 0009#prompt-rules-for-adopters [implements] — the deterministic strip that makes the handoff independent of policy.includeSuggestions
// @ref LLP 0009#prompt-rules-for-adopters [implements] — the deterministic strip that makes the handoff independent of policy.minSeverity
export const OVERALL_PR_RISK_TITLE = "__overall_pr_risk__";
/**
+3 -2
View File
@@ -17,8 +17,9 @@
// @ref LLP 0009#config-and-prompt-templates [implements] — suggestions off by default, not a schema limit
"policy": {
// Phase 1: keep signal high by surfacing only critical, warning and design
// findings. This setting covers both `suggestion` and `nit` findings.
"includeSuggestions": false
// findings. Findings below `minSeverity` are dropped. From most to least severe:
// critical, warning, design, suggestion, nit.
"minSeverity": "design"
// "maxFindings": 10
},
+4 -3
View File
@@ -10,9 +10,10 @@
// scope (id = filename). shared.md + coordinator.md are this scope's prompts.
"policy": {
// Keep signal high by surfacing only critical, warning and design findings. This
// setting covers both `suggestion` and `nit` findings.
"includeSuggestions": false
// Keep signal high by surfacing only critical, warning and design findings.
// Findings below `minSeverity` are dropped. From most to least severe: critical,
// warning, design, suggestion, nit.
"minSeverity": "design"
// "maxFindings": 10
},