Files
code-review-cli/src/__tests__/render-aggregate.test.ts
124c2bf0f5 feat: let an agent mention configured people when a change needs a human review (#90)
* feat: show each reviewer's own summary in the review comment

Each reviewer pass may return a short `summary` of what it reviewed and concluded. The
engine keeps one row per roster agent (its summary, or why it did not run: `paths` or
the router) in an engine-owned `agentSummaries` field, and the comment lists them in a
collapsed section under the overall summary. Summaries are model text, so the reporter
flattens them to one line, escapes HTML, and breaks `@` mentions.

* feat: let an agent mention configured people when a change needs a human review

An agent's front matter can list GitHub logins or teams under `mention`. When the agent
sets `humanReview` with a reason, the review comment shows a visible line that mentions
those people. Only the trusted config names people; the model writes only the reason,
and the reporter breaks every `@` in it. An invalid `mention` is a setup warning.

* fix: close mention bypasses in agent summaries, and keep the review when the comment is too big

- plainModelLine let `_@login_` (rendered as `<em>@login</em>`) and
  character references (`&#64;login`, `&commat;login`, `@&#111;ctocat`)
  through, and GitHub decodes references before it looks for mentions.
  Escape every reference's `&`, and break each `@` that doesn't follow an
  ASCII letter or digit, which keeps email addresses whole.
- Only the aggregate comment checked MAX_COMMENT_CHARS. Escaping and the
  base64 state copy make summaries several times their byte budget, and a
  single comment over GitHub's limit failed to post, so the finished review
  was replaced by the failure notice. Over the limit, leave the summaries out
  with the same note the aggregate uses.
- Summary parts lived in a plain object, so an agent named `constructor` or
  `toString` crashed the whole run; they are now a Map. Parts are kept in
  diff order rather than pass-completion order, and a `cross-cutting` agent
  is listed once.
- Clipping no longer leaves half an emoji, and an unknown "not run" reason
  from an old or newer engine's state reads "not run".

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix: escape only the character references that spell `@`

Escaping every reference made model text like `&lt;T&gt;` or `&amp;` show
up literally. Only a reference that decodes to `@` itself can start a
mention: one after the `@` spells a login character, but that `@` is broken
like any other.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix: only configured mentions notify, once, and a request is never folded away

- Titles, rationales, suggestions, source titles, the summary and the
  requalification reason went through stripStateMarkers only, so model text
  could still notify anyone (a title citing `@param` or `@types/node` pings a
  user or team). They now go through the same mention breaking as the
  summaries, in the single, aggregate and inline bodies.
- In the aggregate comment, a scope with no findings kept its fold closed, so
  the person a request notified landed on a collapsed section. A mention line
  now opens it.
- humanReview is recomputed every run, and GitHub notifies when an edit adds
  a mention, so a request that dropped out and came back notified again. The
  comment state now remembers who was notified; a returning handle renders
  without notifying.
- MENTION_TARGET_RE now follows GitHub's rules: single inner hyphens, at most
  39 characters, an optional EMU `_shortcode` suffix, and team slugs. One
  invalid entry no longer drops the agent's whole `mention` list: it is left
  out with a warning naming it.
- Document that team mentions posted with GITHUB_TOKEN are untested.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix: break every `@` of a run like `@@login`

The capture in `(^|[^A-Za-z0-9])@` consumed the first `@` of `@@login`, so
the second never matched and `@​@login` still notified. A lookbehind
breaks each one. Found by the repo's own AI review.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Brent Vatne <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
2026-10-22 18:51:29 -07:00

380 lines
15 KiB
TypeScript

import { test, expect } from "bun:test";
import {
renderAggregateMarkdown,
renderMarkdown,
parseReviewState,
parseEmbeddedFingerprints,
commentMarker,
worstDecision,
} from "../core/render.js";
import type { ScopeReviewResult } from "../core/render.js";
import { fingerprintFinding, scopedFingerprint } from "../core/schema.js";
import type { CoordinatorOutput, Finding } from "../core/schema.js";
const finding = (over: Partial<Finding> = {}): Finding => ({
severity: "warning",
category: "quality",
file: "a.ts",
line: 1,
title: "T",
rationale: "r",
evidence: "const somethingLongEnough = 1;",
...over,
});
const review = (over: Partial<CoordinatorOutput> = {}): CoordinatorOutput => ({
decision: "approve",
findings: [],
summary: "s",
incomplete: [],
...over,
});
const scope = (
name: string,
isDefault: boolean,
r: Partial<CoordinatorOutput>,
): ScopeReviewResult => ({
scope: name,
isDefault,
review: review(r),
});
test("worstDecision: picks the most severe across scopes", () => {
expect(worstDecision(["approve", "approve_with_comments", "request_changes"])).toBe(
"request_changes",
);
expect(worstDecision(["approve", "approve_with_comments"])).toBe("approve_with_comments");
expect(worstDecision(["approve"])).toBe("approve");
});
test("aggregate: root marker first line, a table row + <details> per scope, worst decision at top", () => {
const results = [
scope("default", true, { decision: "approve", findings: [finding({ title: "D1" })] }),
scope("api", false, {
decision: "request_changes",
findings: [finding({ title: "W1", severity: "critical", category: "security" })],
}),
];
const body = renderAggregateMarkdown(results, "tag", []);
expect(body.split("\n")[0]).toBe(commentMarker("tag"));
expect(body).toContain("**Decision:** Request changes"); // worst across scopes
expect(body).toContain("| Scope | Decision | Findings |");
expect(body).toContain("| default | Ready for human review | 1 |");
expect(body).toContain("| api | Request changes | 1 |");
// one details block per scope, in manifest order
expect(body.indexOf("<summary>default")).toBeLessThan(body.indexOf("<summary>api"));
expect((body.match(/<summary>(default|api) —/g) ?? []).length).toBe(2);
});
test("aggregate: default scope uses plain fingerprints, non-default uses scoped ids; embedded block is the union", () => {
const dFinding = finding({ title: "D", evidence: "const defaultScopedThing = 1;" });
const wFinding = finding({ title: "W", evidence: "const apiScopedThing = 2;" });
const results = [
scope("default", true, { findings: [dFinding] }),
scope("api", false, { findings: [wFinding] }),
];
const body = renderAggregateMarkdown(results, "tag", []);
const plainId = fingerprintFinding(dFinding);
const scopedId = scopedFingerprint("api", wFinding);
expect(body).toContain(`id:${plainId}`);
expect(body).toContain(`id:${scopedId}`);
const embedded = parseEmbeddedFingerprints(body, "tag");
expect(embedded).toContain(plainId);
expect(embedded).toContain(scopedId);
});
test("aggregate: state round-trips through parseReviewState (scopes intact); a v1 body still parses", () => {
const results = [
{
...scope("default", true, { findings: [finding({ title: "D" })] }),
inputHash: "a".repeat(64),
},
{ ...scope("api", false, { findings: [finding({ title: "W" })] }), inputHash: "b".repeat(64) },
];
const body = renderAggregateMarkdown(results, "tag", []);
const state = parseReviewState(body, "tag");
expect(state).not.toBeNull();
expect(state!.scopes?.map((s) => s.scope)).toEqual(["default", "api"]);
expect(state!.scopes?.map((s) => s.inputHash)).toEqual(["a".repeat(64), "b".repeat(64)]);
// A pre-routing v1 comment still parses (no scopes field).
const v1 = renderMarkdown(review({ findings: [finding()] }), "tag");
const v1State = parseReviewState(v1, "tag");
expect(v1State).not.toBeNull();
expect(v1State!.scopes).toBeUndefined();
});
test("aggregate: each scope keeps its hidden review trace across re-rendering", () => {
const results = [
scope("api", false, {
reviewTrace: {
version: 1,
trust: "unverified-model-diagnostics",
agents: {
security: {
checked: ["Traced authentication state through the API scope."],
uncertainties: [],
},
},
},
}),
];
const body = renderAggregateMarkdown(results, "tag", []);
expect(body).not.toContain("Traced authentication state");
const state = parseReviewState(body, "tag")!;
expect(state.scopes![0]!.review.reviewTrace).toEqual(results[0]!.review.reviewTrace);
const rerendered = renderAggregateMarkdown(state.scopes!, "tag", []);
expect(parseReviewState(rerendered, "tag")!.scopes![0]!.review.reviewTrace).toEqual(
results[0]!.review.reviewTrace,
);
});
test("aggregate: each scope lists its own agent summaries inside its fold", () => {
const results = [
scope("api", false, {
summary: "API scope summary.",
agentSummaries: [{ agent: "security", summary: "Checked the token handling." }],
}),
];
const body = renderAggregateMarkdown(results, "tag", []);
const scopeSummary = body.indexOf("API scope summary.");
const section = body.indexOf("🔎 Reviewer summaries (1)");
expect(section).toBeGreaterThan(scopeSummary);
expect(body).toContain("- **security:** Checked the token handling.");
});
test("aggregate: a scope's mention renders under that scope's summary", () => {
const results = [
scope("docs", false, {
summary: "Docs scope summary.",
mentions: [{ agent: "docs", logins: ["amandeepmittal"], reason: "New guide." }],
}),
];
const body = renderAggregateMarkdown(results, "tag", []);
const line = "> 📣 @amandeepmittal: the `docs` reviewer asks for your review. New guide.";
expect(body.indexOf(line)).toBeGreaterThan(body.indexOf("Docs scope summary."));
});
test("aggregate: a scope with a human-review request and no findings opens its fold", () => {
const results = [
scope("docs", false, {
summary: "Docs summary for @summary-user.",
mentions: [{ agent: "docs", logins: ["amandeepmittal"], reason: "New guide." }],
}),
scope("api", false, {}),
];
const body = renderAggregateMarkdown(results, "tag", []);
expect(body).toContain("<details open>\n<summary>docs — ");
expect(body).toContain("<details>\n<summary>api — ");
expect(body).toContain("Docs summary for @\u200Bsummary-user.");
});
test("aggregate: the mention state rides the comment, so a returning request does not notify", () => {
const asking = [
scope("docs", false, {
mentions: [{ agent: "docs", logins: ["amandeepmittal"], reason: "New guide." }],
}),
];
const first = renderAggregateMarkdown(asking, "tag", []);
expect(first).toContain("> 📣 @amandeepmittal:");
const dropped = renderAggregateMarkdown([scope("docs", false, {})], "tag", [], undefined, {
mentioned: parseReviewState(first, "tag")!.mentioned!,
});
const back = renderAggregateMarkdown(asking, "tag", [], undefined, {
mentioned: parseReviewState(dropped, "tag")!.mentioned!,
});
expect(back).toContain("> 📣 @\u200Bamandeepmittal:");
});
test("aggregate: a default-scope finding dismissed pre-routing (plain fp) lands in Dismissed (risk 9)", () => {
const dFinding = finding({ title: "Carried", evidence: "const carriedOverDismissal = 1;" });
const plainFp = fingerprintFinding(dFinding);
const results = [scope("default", true, { findings: [dFinding] })];
const body = renderAggregateMarkdown(results, "tag", [
{ fp: plainFp, by: "me", reason: "known" },
]);
expect(body).toContain("Dismissed on this PR (1)");
expect(body).toContain(`id:${plainFp}`);
expect(body).toContain("| default | Ready for human review | 0 |"); // moved out of the active count
});
test("aggregate: dismissed findings survive in embedded state so /undismiss can restore them", () => {
const dFinding = finding({ title: "DismissedOne", evidence: "const dismissedThing = 1;" });
const kFinding = finding({ title: "KeptOne", evidence: "const keptThing = 2;" });
const results = [scope("api", false, { findings: [dFinding, kFinding] })];
const fp = scopedFingerprint("api", dFinding);
const body = renderAggregateMarkdown(results, "tag", [{ fp, by: "me", reason: "known" }]);
expect(body).toContain("Dismissed on this PR (1)");
// The dismissed finding stays in the embedded state (and fingerprint union),
// not just the kept one — a re-render from state must not lose it.
const state = parseReviewState(body, "tag")!;
expect(state.scopes![0]!.review.findings.map((f) => f.title)).toContain("DismissedOne");
expect(parseEmbeddedFingerprints(body, "tag")).toContain(fp);
// Round-trip: re-render from the embedded state with the dismissal removed
// (what /undismiss does) restores the finding to the active list.
const restored = renderAggregateMarkdown(state.scopes!, "tag", []);
expect(restored).toContain(`id:${fp}`);
expect(restored).toContain("| api | Ready for human review | 2 |");
expect(restored).not.toContain("Dismissed on this PR");
});
test("aggregate: a freshly-reviewed scope's per-scope feedback array is stripped from embedded state", () => {
// Finding stale-scope-feedback (render side): a freshly-reviewed scope's
// ReviewRunResult carries `review.feedback`. Embedding it verbatim would leave a
// stale per-scope copy that a later carried-over run reads back and re-applies. The
// top-level feedback array is the single source of truth, so the per-scope copy must
// not survive into the embedded state.
const f = finding({ title: "F", evidence: "const perScopeFeedbackStrip = 1;" });
const fp = scopedFingerprint("api", f);
const staleRecord = { fp, by: "octocat", commentId: 9, maintainer: true, applied: false };
const results: ScopeReviewResult[] = [
{
scope: "api",
isDefault: false,
review: {
decision: "approve",
findings: [f],
summary: "s",
incomplete: [],
feedback: [staleRecord],
} as CoordinatorOutput,
},
];
const body = renderAggregateMarkdown(results, "tag", [], undefined, undefined, [staleRecord]);
const state = parseReviewState(body, "tag")!;
const apiScope = state.scopes!.find((s) => s.scope === "api")!;
expect((apiScope.review as { feedback?: unknown }).feedback).toBeUndefined();
// The record still rides the top-level feedback array (the single source of truth).
expect(state.feedback).toHaveLength(1);
});
test("aggregate: a scope whose reply cleared every finding opens its fold (visible suppression)", () => {
// Finding aggregate-audit-fold: when replies suppress a scope's only finding, shown
// and requalifiedAll are both empty. The fold must still open so the audit note is
// visible, matching renderMarkdown and LLP 0010's visible-suppression rule.
const f = finding({ title: "Cleared by reply", evidence: "const replyClearedThing = 1;" });
const fp = scopedFingerprint("api", f);
const body = renderAggregateMarkdown(
[scope("api", false, { findings: [f] })],
"tag",
[],
undefined,
undefined,
[
{
fp,
by: "octocat",
commentId: 9,
url: "https://github.com/o/r/pull/1#issuecomment-9",
maintainer: true,
applied: true,
},
],
);
// The finding is suppressed (moved to Dismissed); the scope shows no active finding.
expect(body).toContain("<summary>api — Ready for human review (0)</summary>");
// ...but the fold OPENS so the audit note is above-the-fold, not collapsed away.
expect(body).toMatch(/<details open>\n<summary>api —/);
expect(body).toContain("have an author response");
});
test('aggregate: oversized findings truncate with a "+N more" note and stay under 65k', () => {
const many = Array.from({ length: 400 }, (_, i) =>
finding({
title: `Finding ${i}`,
severity: "critical",
evidence: `const uniqueEvidenceNumber${i} = ${i};`,
rationale: "x".repeat(300),
}),
);
const results = [scope("api", false, { decision: "request_changes", findings: many })];
const body = renderAggregateMarkdown(results, "tag", []);
expect(body.length).toBeLessThan(65_000);
expect(body).toContain("more finding(s) — see the workflow log.");
});
test("aggregate: oversized agent summaries go before any finding, and the comment says so", () => {
// Each scope stays inside its own summary budget; together they overflow the body.
const agentSummaries = Array.from({ length: 20 }, (_, i) => ({
agent: `agent-${i}`,
summary: `${"word ".repeat(55)}${i}`,
}));
const results = Array.from({ length: 10 }, (_, i) =>
scope(`scope-${i}`, false, {
decision: "approve_with_comments",
findings: [finding({ title: `Scope ${i} finding`, evidence: `const scopeFinding${i} = 1;` })],
agentSummaries,
}),
);
const body = renderAggregateMarkdown(results, "tag", []);
expect(body.length).toBeLessThan(65_000);
expect(body).not.toContain("<summary>🔎 Reviewer summaries (");
expect(body).toContain("🔎 Reviewer summaries were left out to fit the comment size limit.");
for (let i = 0; i < 10; i++) {
expect(body).toContain(`Scope ${i} finding`);
}
const state = parseReviewState(body, "tag")!;
expect(state.scopes!.every((entry) => entry.review.agentSummaries === undefined)).toBe(true);
});
test("aggregate: unmatched files render as a coverage note", () => {
const results = [scope("default", true, { findings: [] })];
const body = renderAggregateMarkdown(results, "tag", [], undefined, {
unmatchedFiles: ["weird/path.txt"],
});
expect(body).toContain("Coverage note");
expect(body).toContain("weird/path.txt");
});
// Finding 135b432cb46b: the `/undismiss` pin set is a maintainer decision about a
// finding. It rides the aggregate state whole — never indexed by the records or
// findings this render happens to show, and never trimmed by the size cap loop, which
// would silently release a restore no later run could recover.
test("aggregate: the /undismiss pin set survives a render with no records and a truncating one", () => {
const pinned = finding({ title: "Pinned", evidence: "const pinnedEvidence = 1;" });
const pinnedFp = scopedFingerprint("api", pinned);
const pins = [{ fp: pinnedFp, commentId: 42 }];
// No feedback record matched this run (the author edited the reply away).
const plain = renderAggregateMarkdown(
[scope("api", false, { findings: [pinned] })],
"tag",
[],
undefined,
undefined,
[],
pins,
);
expect(parseReviewState(plain, "tag")!.pins).toEqual(pins);
const many = Array.from({ length: 400 }, (_, i) =>
finding({
title: `Finding ${i}`,
severity: "critical",
evidence: `const uniqueEvidenceNumber${i} = ${i};`,
rationale: "x".repeat(300),
}),
);
const big = renderAggregateMarkdown(
[
{
...scope("api", false, { decision: "request_changes", findings: [pinned, ...many] }),
inputHash: "c".repeat(64),
},
],
"tag",
[],
undefined,
undefined,
[],
pins,
);
expect(big.length).toBeLessThan(65_000);
expect(parseReviewState(big, "tag")!.pins).toEqual(pins);
expect(parseReviewState(big, "tag")!.scopes![0]!.inputHash).toBeUndefined();
});