feat(sentry): add opt-in run failure reporting with saved run references (#16300)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs retain execution results and output in the instance. > - Operators can report run failures to Sentry for investigation. > - A backend Sentry DSN currently enables those reports with no separate choice. > - A repair bot also needs the exact instance, company, agent, and saved run. > - This pull request adds an explicit reporting mode and authenticated run references. > - The benefit is that operators can investigate every failure on selected deployments while keeping other deployments quiet. ## Linked Issues or Issue Description **What existing behavior does this improve?** Optional server reporting of terminal agent run failures to Sentry. **Current behavior** A backend DSN enables run failure reports. Known account, quota, and connection failures are excluded. Reports omit company and agent references. Runs without a task are skipped. Tool-name collision errors omit the conflicting bindings. **Proposed behavior** Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` to retain the existing filters. Set it to `all` to report every failed or timed-out run, including known setup and quota failures. The default is `off`. Expected stops stay excluded. Each report supplies identifiers and API paths that a bot can use with its own authorized Paperclip connection. **Reason and benefit** Operators can enable broad failure reporting on one staging deployment. A bot can fetch the saved run, events, and logs instead of relying on a short error label or copying complete transcripts into Sentry. **Breaking changes** A backend DSN alone no longer enables run failure reports. Existing operators must set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` to retain the previous policy. Ordinary server exception capture remains enabled by the DSN. Failure grouping gains the phase and original cause code when available. Refs #16281, #16297, #16298. These cover related tool discovery and failure diagnostics. This change retains their existing boundaries and does not depend on their open changes. ## What Changed - Add `off`, `unexpected`, and `all` modes. Missing and invalid values keep reporting off before diagnostic reads. - Keep the Sentry backend DSN and audited optional SDK requirements. - Include company, agent, run, and optional task identifiers. Include versioned paths for the saved run, events, log, and task. Add absolute run URLs only for a valid public HTTP(S) origin. - Report company runs without a task. Prefer the first-class issue binding when selecting the task reference. - Select saved native provider/model settings, timestamps, and original failure codes. Preserve gateway reason codes through redacted exception projection. - Group by error code, adapter, and available phase/cause. Keep run IDs and error text out of grouping. - Include the tool names and connection bindings in collision errors without copying tool schemas or provider metadata. - Document configuration, upgrade behavior, bot access, redaction, retention, and best-effort delivery. ## Verification - All 56 checks passed for `cccb47a721eaa1cc36e63278fc177b9101477541`. Two unrelated Storybook jobs were skipped by their path filters. No checks remain pending. - Complete [Paperclip CI run](https://github.com/paperclipai/paperclip/actions/runs/38591422284): passed. This covers general and serialized tests, Runner checks, browser E2E shards, typecheck, build, release-registry checks, canary dry run, and Docker checks. - Greptile reviewed that exact head: 5/5, with no unresolved threads or actionable findings. Superagent, Socket, and Snyk checks passed. - Local focused reporting, diagnostics, collision, and real SDK contract tests: 410 passed across five suites. This includes the external-wake filter in both reporting modes. - The audited `@sentry/[email protected].0` tests use an in-memory transport. No events were sent to Sentry. - `pnpm -r typecheck` and `pnpm build`: passed locally after the final rebase. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - The full local `pnpm test:run` command was stopped to avoid duplicating the complete CI shards. It did not finish locally. The full suite passed in CI. ## Risks - The new default disables existing run failure exports until the operator selects a mode. The upgrade setting is documented. - `all` can increase event volume and include known user-action failures. Secret redaction and diagnostic limits still apply. - Reporting remains asynchronous and best effort. This PR adds no durable delivery queue and does not install a bot or Sentry alert rule. - API references grant no authority. Existing company, private-task, and run-output checks still apply. Complete transcripts stay in Paperclip. - This changes optional Sentry error reporting. It changes neither first-party Telemetry nor OpenTelemetry span exports. It adds no database migration. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, shell, file editing, and test execution. The exact backend deployment ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>
2026-11-07 06:16:06 -10:00 · 2026-10-14 10:56:40 -05:00
1 parent 09a2f0b7b0
commit 21ee5c9c85
@@ -3,6 +3,7 @@ PORT=3100
 SERVE_UI=false
 BETTER_AUTH_SECRET=paperclip-dev-secret
 PAPERCLIP_TOOL_ACTION_SIGNING_SECRET=paperclip-dev-tool-action-signing-secret-change-me
+PAPERCLIP_SENTRY_RUN_FAILURES=off

 # Optional Paperclip ID Gmail OAuth broker. Enroll the instance first; keep both
 # private keys in the deployment secret manager. HTTP base URLs must be loopback.
@@ -473,15 +473,71 @@ below.

 ### Run failure context

-Terminal run failures also carry the run and task IDs, adapter, error code,
-run status, and redacted error message. Their fingerprint is the error code
-and adapter. These fields are passed directly to that event's capture call.
+Run failure reporting requires a separate operator choice in addition to the
+backend Sentry DSN and the audited SDK. Set `PAPERCLIP_SENTRY_RUN_FAILURES` to:
+
+| Value | Reports |
+| --- | --- |
+| `off` (default) | No run failure events. Ordinary server exception reporting still uses the backend DSN. |
+| `unexpected` | Failed and timed-out runs, plus unexpected started cancellations, with the existing configuration, quota, and external connection filters described below. |
+| `all` | Every failed or timed-out run, including the known configuration, quota, and external connection failures. Unexpected started cancellations also report. |
+
+Missing, blank, or unrecognized values keep run reporting off. Expected operator
+stops, shutdown interruptions, successful runs, and queued work do not report in
+either enabled mode. The mode does not change run status, retries, recovery,
+Telemetry, or OpenTelemetry export. It is process-wide, so an operator can enable
+it on one staging instance without enabling it on other deployments. Restart the
+server after changing deployment environment variables.
+
+**Upgrade:** A backend DSN alone previously enabled run failure reports. Set
+`PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` to preserve that reporting policy.
+Use `all` to investigate every failed run in a selected deployment.
+
+Terminal run failures carry the company, agent, run and optional task IDs,
+adapter, error code, run status, and redacted error message. Company-scoped runs
+without a task also report. The `event_kind=run_failure` tag identifies these
+events. Their fingerprint contains the error code and adapter, followed by the
+setup/execute phase and original failure code when available. IDs, release,
+model, and error text do not split groups. These fields are passed directly to
+that event's capture call.
 They do not change the ambient Sentry scope, whose isolation is unavailable
 without an OpenTelemetry context manager. Later, unrelated exceptions must
 not inherit a previous run's identity or fingerprint.

+The versioned `paperclip` context supplies company and agent IDs and relative
+API paths for the saved run, events, log, and optional task. A valid HTTP(S)
+origin in `PAPERCLIP_PUBLIC_URL` also supplies absolute run, events, and log URLs.
+Values with credentials, query parameters, fragments, or a path are omitted.
+The event never includes an API key or an access token. These references grant
+no access. A bot must authenticate to the correct instance and pass its existing
+company, run-output, and private-task access checks.
+
+The bot should use the exact run ID to read `GET /api/heartbeat-runs/{runId}`,
+`GET /api/heartbeat-runs/{runId}/events`, and
+`GET /api/heartbeat-runs/{runId}/log`. These retain the saved result, run context,
+and available output. Prefer the saved provider configuration over the agent's
+current settings. The Sentry execution context includes the saved native
+provider kind and model, start/finish timestamps, and original failure code when
+available. Tool-name collision errors identify the conflicting tool names and
+connection bindings without serializing tool schemas or provider metadata.
+Gateway reason codes survive exception projection. All diagnostic text still
+passes through the existing secret redaction and size limits.
+
+Use the Sentry group to combine repeated investigations, and the instance and
+run ID to identify each occurrence. Automated retries or proposed code fixes
+must use normal Paperclip authority and approval checks. Receiving an event
+does not authorize a bot to replay a run. This change provides the reporting
+contract; it does not install a bot or configure Sentry alerts.
+
+Reporting remains asynchronous and best effort. Shutdown drains pending report
+work before the SDK flushes. There is no durable delivery queue or receipt from
+Sentry, so a crash or network outage can lose events. `all` selects failures; it
+does not promise delivery or export complete transcripts. Detailed output stays
+in the instance behind the existing authenticated APIs, with its normal retention.
+
 Known missing-secret configuration blockers are kept in the task's run log,
 blocked state, and owner recovery action, without a Sentry run-failure event.
+The exclusions in this section apply to `unexpected` mode only.
 This requires a failed `configuration_incomplete` run in the preparing stage,
 a setup-phase report, explicit proof that provider work did not start, and a
 nonempty list of recognized missing or inactive secret bindings. Process exit
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
 import * as codexAdapter from "@paperclipai/adapter-codex-local/server";
 import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
 import { randomUUID } from "node:crypto";
@@ -567,3 +568,6 @@ describe("hired agents sharing a subscription", () => {
    }
  });
 });
+
+
+enableRunFailureReporting();
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
 import { randomUUID } from "node:crypto";
 import { mkdtemp, rm } from "node:fs/promises";
@@ -448,3 +449,6 @@ describe("durable, authorized connection routing", () => {
    } finally { await first.cleanup(); }
  });
 });
+
+
+enableRunFailureReporting();
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
 import { createHash, randomUUID } from "node:crypto";
 import { trace } from "@opentelemetry/api";
 import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
@@ -241,3 +242,6 @@ describePostgres("heartbeat execution configuration through adapter dispatch", (
    expect(input).toBeUndefined();
  });
 });
+
+
+enableRunFailureReporting();
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
 import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
 import * as conversationContinuation from "../services/conversation-continuation.js";
 import { environmentRuntimeService } from "../services/environment-runtime.js";
@@ -16789,3 +16790,6 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
  });

 });
+
+
+enableRunFailureReporting();
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
 import { randomUUID } from "node:crypto";
 import { eq } from "drizzle-orm";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
@@ -143,3 +144,6 @@ describeEmbeddedPostgres(
    });
  },
 );
+
+
+enableRunFailureReporting();
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
 import { randomUUID } from "node:crypto";
 import fs from "node:fs/promises";
 import os from "node:os";
@@ -1077,3 +1078,6 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => {
    expect(holderRun?.status).toBe("running");
  });
 });
+
+
+enableRunFailureReporting();
@@ -0,0 +1,13 @@
+import { afterEach, beforeEach } from "vitest";
+
+export function enableRunFailureReporting(): void {
+  let previousMode: string | undefined;
+  beforeEach(() => {
+    previousMode = process.env.PAPERCLIP_SENTRY_RUN_FAILURES;
+    process.env.PAPERCLIP_SENTRY_RUN_FAILURES = "unexpected";
+  });
+  afterEach(() => {
+    if (previousMode === undefined) delete process.env.PAPERCLIP_SENTRY_RUN_FAILURES;
+    else process.env.PAPERCLIP_SENTRY_RUN_FAILURES = previousMode;
+  });
+}
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
 import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
 import { spawn } from "node:child_process";
 import { randomUUID } from "node:crypto";
@@ -1457,3 +1458,6 @@ describe.each(["unchanged", "newer_active", "stale_idle"] as const)(
    }, 30_000);
  },
 );
+
+
+enableRunFailureReporting();
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
 import { captureEnvironmentAcquisitionDiagnostic } from "../services/environment-acquisition-diagnostics.js";
 import { createRequire } from "node:module";
 import fs from "node:fs/promises";
@@ -374,6 +375,8 @@ describe.skipIf(!sentryPackage)("run failure context with the real Sentry SDK",
    await sentryReady;

    const first = {
+      companyId: "55555555-5555-4555-8555-555555555555",
+      agentId: "66666666-6666-4666-8666-666666666666",
      taskId: "11111111-1111-4111-8111-111111111111",
      runId: "22222222-2222-4222-8222-222222222222",
      errorMessage: "first run failed",
@@ -432,7 +435,7 @@ describe.skipIf(!sentryPackage)("run failure context with the real Sentry SDK",
        contexts: { run_failure: { runId: run.runId, taskId: run.taskId,
          errorMessage: run.errorMessage, errorCode: run.errorCode, agentAdapter: run.agentAdapter,
          exitCode: run.exitCode, signal: run.signal } },
-        fingerprint: [run.errorCode, run.agentAdapter],
+        fingerprint: [run.errorCode, run.agentAdapter, ...(run === first ? ["execute", "ECONNRESET"] : [])],
      });
    }
    expect(captured("malformed process metadata")).toMatchObject({
@@ -441,6 +444,7 @@ describe.skipIf(!sentryPackage)("run failure context with the real Sentry SDK",
    });
    expect(JSON.stringify(events)).not.toContain("private-signal-payload");
    const firstCapture = captured(first.errorMessage)!;
+    expect(firstCapture).toHaveProperty("contexts.paperclip.runPath", `/api/heartbeat-runs/${first.runId}`);
    const exceptions = (firstCapture.exception as { values: Array<Record<string, unknown>> }).values;
    expect(exceptions).toHaveLength(2);
    expect(JSON.stringify(exceptions)).toContain("originalAdapter");
@@ -610,3 +614,6 @@ describe.skipIf(!sentryPackage)("run failure context with the real Sentry SDK",
  });

 });
+
+
+enableRunFailureReporting();
@@ -71,12 +71,14 @@ async function importFreshSentryWithGateOpen() {
 }

 beforeEach(() => {
+  vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "unexpected");
  delete process.env[DSN_ENV];
  delete process.env[FRONTEND_DSN_ENV];
  delete process.env[BACKEND_DSN_ENV];
 });

 afterEach(() => {
+  vi.unstubAllEnvs();
  if (originalDsn === undefined) delete process.env[DSN_ENV];
  else process.env[DSN_ENV] = originalDsn;
  if (originalFrontendDsn === undefined) delete process.env[FRONTEND_DSN_ENV];
@@ -89,6 +91,46 @@ afterEach(() => {
 });

 describe("captureRunFailure", () => {
+  it.each([undefined, "off", "true", "invalid"])("keeps ordinary server errors enabled while run reporting is disabled: %s", async mode => {
+    const { sentryModule, captureException } = await importFreshSentryWithGateOpen();
+    vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", mode);
+    if (mode === undefined) delete process.env.PAPERCLIP_SENTRY_RUN_FAILURES;
+    sentryModule.captureRunFailure(baseEvent());
+    const error = new Error("ordinary server failure");
+    sentryModule.captureException(error);
+    expect(captureException).toHaveBeenCalledExactlyOnceWith(error);
+  });
+
+  it("provides authenticated run references and groups repeated causes without leaking scope", async () => {
+    const { sentryModule, captureException } = await importFreshSentryWithGateOpen();
+    vi.stubEnv("PAPERCLIP_PUBLIC_URL", "https://paperclip.example.com");
+    const event = baseEvent({ companyId: "company", agentId: "agent", taskId: null,
+      diagnostics: { execution: { failurePhase: "setup", originalFailureCode: "tool_name_collision" }, adapter: {}, provider: {}, exceptions: [], truncatedFields: [] } });
+    sentryModule.captureRunFailure(event);
+    sentryModule.captureRunFailure({ ...event, runId: "another-run", errorMessage: "Another tool failed" });
+    sentryModule.captureException(new Error("unrelated"));
+    const first = captureException.mock.calls[0]![1]!;
+    expect(first.tags).toMatchObject({ event_kind: "run_failure", company_id: "company", agent_id: "agent", root_error_code: "tool_name_collision" });
+    expect(first.tags).not.toHaveProperty("task_id");
+    expect(first.contexts.paperclip).toEqual({ schemaVersion: 1, companyId: "company", agentId: "agent",
+      runPath: `/api/heartbeat-runs/${event.runId}`, eventsPath: `/api/heartbeat-runs/${event.runId}/events`, logPath: `/api/heartbeat-runs/${event.runId}/log`,
+      runUrl: `https://paperclip.example.com/api/heartbeat-runs/${event.runId}`, eventsUrl: `https://paperclip.example.com/api/heartbeat-runs/${event.runId}/events`,
+      logUrl: `https://paperclip.example.com/api/heartbeat-runs/${event.runId}/log` });
+    expect(first.fingerprint).toEqual(["process_lost", "claude-code", "setup", "tool_name_collision"]);
+    expect(captureException.mock.calls[1]![1]?.fingerprint).toEqual(first.fingerprint);
+    expect(captureException.mock.calls[2]![1]).toBeUndefined();
+  });
+
+  it.each([undefined, "invalid", "ftp://example.com", "https://user:[email protected]", "https://example.com?token=private", "https://example.com/#private", "https://example.com/private"])("omits unsafe or absent instance URLs: %s", async value => {
+    const { sentryModule, captureException } = await importFreshSentryWithGateOpen();
+    vi.stubEnv("PAPERCLIP_PUBLIC_URL", value);
+    sentryModule.captureRunFailure(baseEvent({ companyId: "company", agentId: "agent" }));
+    const context = captureException.mock.calls[0]![1]!.contexts.paperclip;
+    expect(context).toHaveProperty("runPath");
+    expect(context).not.toHaveProperty("runUrl");
+    expect(JSON.stringify(context)).not.toMatch(/password|private/);
+  });
+
  it("sets the fingerprint [errorCode, agentAdapter] in that order", async () => {
    const { sentryModule, captureException } = await importFreshSentryWithGateOpen();

@@ -195,6 +237,7 @@ describe("captureRunFailure", () => {
    sentryModule.captureRunFailure(event);

    expect(captureException.mock.calls[0]![1]?.tags).toEqual({
+      event_kind: "run_failure",
      run_id: event.runId,
      task_id: event.taskId,
      error_code: event.errorCode,
@@ -0,0 +1,6 @@
+export type RunFailureReportingMode = "off" | "unexpected" | "all";
+
+export function readRunFailureReportingMode(): RunFailureReportingMode {
+  const value = process.env.PAPERCLIP_SENTRY_RUN_FAILURES?.trim();
+  return value === "unexpected" || value === "all" ? value : "off";
+}
@@ -62,6 +62,7 @@ import type { RunFailureDiagnostics } from "./services/run-failure-diagnostics.j
 import { readBuildCommit } from "./build-commit.js";
 import { checkExactPeerVersions } from "./peer-version-check.js";
 import { resolveSentryDsns } from "./sentry-dsn.js";
+import { readRunFailureReportingMode } from "./sentry-run-failure-policy.js";

 const { backend: dsn, legacyFallbackUsed } = resolveSentryDsns();

@@ -142,8 +143,9 @@ export type RunFailureStatus = "failed" | "timed_out" | "cancelled";
 * event. `errorCode` is `null` when the run holds no error code.
 */
 export interface RunFailureEvent {
-  /** The task UUID the run belongs to. */
-  taskId: string;
+  taskId: string | null;
+  companyId?: string;
+  agentId?: string;
  /** The `heartbeat_runs` row id. */
  runId: string;
  /** The redacted error message. */
@@ -185,13 +187,9 @@ function normalizeRunSignal(value: unknown): string | null {
 * or when the gate never opens. Never throws — observability must not
 * change run control flow.
 *
- * Sets the fingerprint to `[errorCode, agentAdapter]`, in that order, so
- * Sentry groups events by error code and adapter. The error message stays
- * out of the fingerprint — it still travels as the exception message and as
- * a field of the `run_failure` context.
 */
 export function captureRunFailure(event: RunFailureEvent): void {
-  if (!sentryHandle) return;
+  if (!sentryHandle || readRunFailureReportingMode() === "off") return;
  const handle = sentryHandle;
  try {
    const errorCode = event.errorCode ?? "unknown";
@@ -199,7 +197,34 @@ export function captureRunFailure(event: RunFailureEvent): void {
    // A withScope mutation can then persist into unrelated later captures.
    // Pass these fields on this event only; do not mutate the ambient scope.
    const diagnostics = event.diagnostics;
+    const rootCodeCandidate = diagnostics?.execution.originalFailureCode ??
+      diagnostics?.exceptions.at(-1)?.code;
+    const rootErrorCode = typeof rootCodeCandidate === "string" && /^[a-zA-Z][a-zA-Z0-9_]{1,99}$/.test(rootCodeCandidate)
+      ? rootCodeCandidate : undefined;
+    const phase = diagnostics?.execution.failurePhase;
+    const failurePhase = phase === "setup" || phase === "execute" ? phase : undefined;
    const contexts: Record<string, Record<string, unknown>> = {};
+    if (event.companyId && event.agentId) {
+      const runPath = `/api/heartbeat-runs/${encodeURIComponent(event.runId)}`;
+      contexts.paperclip = {
+        schemaVersion: 1,
+        companyId: event.companyId,
+        agentId: event.agentId,
+        runPath,
+        eventsPath: `${runPath}/events`,
+        logPath: `${runPath}/log`,
+        ...(event.taskId ? { taskPath: `/api/issues/${encodeURIComponent(event.taskId)}` } : {}),
+      };
+      try {
+        const base = new URL(process.env.PAPERCLIP_PUBLIC_URL ?? "");
+        if ((base.protocol === "https:" || base.protocol === "http:") &&
+            !base.username && !base.password && !base.search && !base.hash && base.pathname === "/") {
+          contexts.paperclip.runUrl = new URL(runPath, base).href;
+          contexts.paperclip.eventsUrl = new URL(`${runPath}/events`, base).href;
+          contexts.paperclip.logUrl = new URL(`${runPath}/log`, base).href;
+        }
+      } catch {}
+    }
    if (diagnostics) {
      contexts.run_execution = { ...diagnostics.execution, truncatedFields: diagnostics.truncatedFields };
      if (Object.keys(diagnostics.adapter).length) contexts.adapter_failure = diagnostics.adapter;
@@ -226,8 +251,13 @@ export function captureRunFailure(event: RunFailureEvent): void {
    }
    handle.captureException(exception, {
      tags: {
+        event_kind: "run_failure",
+        ...(rootErrorCode ? { root_error_code: rootErrorCode } : {}),
+        ...(failurePhase ? { failure_phase: failurePhase } : {}),
        run_id: event.runId,
-        task_id: event.taskId,
+        ...(event.taskId ? { task_id: event.taskId } : {}),
+        ...(event.companyId ? { company_id: event.companyId } : {}),
+        ...(event.agentId ? { agent_id: event.agentId } : {}),
        error_code: errorCode,
        agent_adapter: event.agentAdapter,
        run_status: event.runStatus,
@@ -244,7 +274,7 @@ export function captureRunFailure(event: RunFailureEvent): void {
          signal: normalizeRunSignal(event.signal),
        },
      },
-      fingerprint: [errorCode, event.agentAdapter],
+      fingerprint: [errorCode, event.agentAdapter, ...(failurePhase ? [failurePhase] : []), ...(rootErrorCode ? [rootErrorCode] : [])],
    });
  } catch (err) {
    // eslint-disable-next-line no-console
@@ -8,6 +8,20 @@ const run = (overrides: Partial<Run> = {}) => ({ resultJson: null, ...overrides
 const collect = (error: unknown) => collectRunFailureDiagnostics(run(), { error });

 describe("run failure diagnostics", () => {
+  it("selects the saved provider configuration and gateway error code without reading credentials or tool schemas", () => {
+    const failure = Object.assign(new Error('Tool collision: "calendar.search"'), {
+      reasonCode: "tool_name_collision", details: { token: "private-token" },
+    });
+    const diagnostic = collectRunFailureDiagnostics(run({
+      startedAt: new Date("2026-10-14T00:00:00Z"), finishedAt: new Date("2026-10-14T00:00:01Z"),
+      runnerProfileJson: { nativeExecutionInput: { provider: { kind: "codex", model: "saved-model", env: { TOKEN: "private-token" } } } },
+      resultJson: { originalFailureCode: "tool_name_collision" },
+    }), { error: failure });
+    expect(diagnostic.execution).toMatchObject({ providerKind: "codex", model: "saved-model", durationMs: 1000,
+      originalFailureCode: "tool_name_collision", startedAt: "2026-10-14T00:00:00.000Z", finishedAt: "2026-10-14T00:00:01.000Z" });
+    expect(diagnostic.exceptions[0]).toMatchObject({ code: "tool_name_collision" });
+    expect(JSON.stringify(diagnostic)).not.toContain("private-token");
+  });
  it("reports only private acquisition observations for the exact failed setup run", () => {
    const scope = { companyId: "company-fixture", environmentId: "environment-fixture", runId: "run-fixture" };
    const diagnostic = { phase: "workspace", elapsedMs: 100, budgetMs: 2_000 } as const;
@@ -63,6 +63,7 @@ describeEmbeddedPostgres("reportRunFailure", () => {
      ["PATH", "HOME", "USER", "USERNAME", "LOGNAME", "USERPROFILE", "TMPDIR", "TEMP", "TMP"]
        .flatMap((key) => inheritedEnv[key] === undefined ? [] : [[key, inheritedEnv[key]]]),
    );
+    process.env.PAPERCLIP_SENTRY_RUN_FAILURES = "unexpected";
  });

  afterEach(async () => {
@@ -126,6 +127,55 @@ describeEmbeddedPostgres("reportRunFailure", () => {
    } as unknown as typeof heartbeatRuns.$inferSelect;
  }

+  it.each([undefined, "", "off", "true", "invalid"])("does no reporting work without an explicit valid mode: %s", async mode => {
+    vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", mode);
+    if (mode === undefined) delete process.env.PAPERCLIP_SENTRY_RUN_FAILURES;
+    const select = vi.fn();
+    await reportRunFailure({ select } as unknown as Db, buildRun());
+    expect(select).not.toHaveBeenCalled();
+    expect(mockCaptureRunFailure).not.toHaveBeenCalled();
+  });
+
+  it.each(["quota", "git", "secret", "selection", "workspace", "hermes"])("reports known %s failures in all mode", async kind => {
+    vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
+    await seedCompanyAndAgent(kind === "hermes" ? { adapterType: "hermes_gateway" } : {});
+    const fixtures = {
+      quota: providerQuotaRun,
+      git: gitConnectionRun,
+      secret: missingSecretRun,
+      selection: aiSelectionRun,
+      workspace: localPathConfigurationRun,
+      hermes: hermesConnectionRun,
+    };
+    const fixture = fixtures[kind as keyof typeof fixtures]();
+    const phase = ["git", "secret", "selection", "workspace"].includes(kind) ? "setup" : "execute";
+    const before = structuredClone(fixture);
+    await reportRunFailure(db, fixture, { phase });
+    expect(mockCaptureRunFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
+      companyId, agentId, runId: fixture.id, runStatus: "failed", errorCode: fixture.errorCode,
+    }));
+    expect(fixture).toEqual(before);
+  });
+
+  it("reports company-scoped runs that have no task", async () => {
+    await seedCompanyAndAgent();
+    const fixture = buildRun({ nativeIssueId: null, contextSnapshot: null });
+    await reportRunFailure(db, fixture);
+    expect(mockCaptureRunFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
+      companyId, agentId, taskId: null, runId: fixture.id,
+    }));
+  });
+
+  it.each(["succeeded", "queued", "running", "interrupted", "cancelled"])("all mode excludes %s runs and expected operator stops", async status => {
+    vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
+    const select = vi.fn();
+    await reportRunFailure({ select } as unknown as Db, buildRun({ status, startedAt: new Date(), resultJson: {
+      cancellation: { source: "operator", expected: true, initiator: { type: "user" }, reason: "Stopped", recordedAt: new Date().toISOString() },
+    } }));
+    expect(select).not.toHaveBeenCalled();
+    expect(mockCaptureRunFailure).not.toHaveBeenCalled();
+  });
+
  it.each(["fresh", "resume", "provider_started"] as const)("keeps workspace base-ref observations reportable: %s", async mode => {
    await seedCompanyAndAgent();
    const run = buildRun({ errorCode: "configuration_incomplete", executionStage: "preparing", exitCode: null, signal: null,
@@ -388,6 +438,15 @@ describeEmbeddedPostgres("reportRunFailure", () => {
    expect(mockCaptureRunFailure).not.toHaveBeenCalled();
  });

+  it("reports a rejected Grok wake in all mode", async () => {
+    vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
+    const run = await expiredGrokOffer({ wakes: [{ status: "rejected", httpStatus: 400 }] });
+    await reportRunFailure(db, run);
+    expect(mockCaptureRunFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
+      companyId, agentId, runId: run.id, errorCode: "native_provider_terminal_failed",
+    }));
+  });
+
  it.each([
    ["no wake was sent", { wakes: [] }],
    ["a wake was rejected locally without HTTP evidence", { wakes: [{ status: "rejected", httpStatus: 400 }, { status: "rejected", httpStatus: null }] }],
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "../__tests__/helpers/run-failure-reporting.js";
 import { randomUUID } from "node:crypto";
 import { eq } from "drizzle-orm";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
@@ -187,3 +188,6 @@ describeEmbeddedPostgres("native review execution reconciliation admission", ()
    expect(review?.status).toBe("pending");
  });
 });
+
+
+enableRunFailureReporting();
@@ -47,6 +47,14 @@ describe("assigned MCP runner tools", () => {
    expect(f.executeTool).not.toHaveBeenCalled();
  });

+  it("identifies the conflicting tool and both connection bindings without exporting descriptor metadata", async () => {
+    const tool = { ...descriptor("calendar.search"), connectionId: "first-connection", providerMetadata: { token: "private-token" } };
+    const f = fixture([tool, { ...tool, connectionId: "second-connection" }]);
+    await expect(createAssignedMcpTools(f)).rejects.toMatchObject({ reasonCode: "tool_name_collision",
+      message: 'Conflicting tool definitions share the name "calendar.search" (connections ["first-connection","second-connection"]).' });
+    expect(f.executeTool).not.toHaveBeenCalled();
+  });
+
  it("keeps same-action tools from distinct connections separately callable", async () => {
    const f = fixture([descriptor("mcp.memory-first:search"), descriptor("mcp.memory-second:search")]);
    const assigned = await createAssignedMcpTools(f);
@@ -94,7 +94,12 @@ export async function createAssignedMcpTools(input: {
  const tools = new Map<string, ToolGatewayDescriptor>();
  for (const descriptor of deduplicateToolDescriptors(listed)) {
    const name = assignedToolName(descriptor.name);
-    if (tools.has(name)) throw new Error("assigned_mcp_tool_name_collision");
+    const previous = tools.get(name);
+    if (previous) {
+      throw new ToolGatewayHttpError(409,
+        `Assigned MCP tool name collision for ${JSON.stringify([previous.name, descriptor.name])}.`,
+        "assigned_mcp_tool_name_collision");
+    }
    tools.set(name, descriptor);
  }
  const permits = (tool: ToolGatewayDescriptor, mode: WorkMode = input.workMode ?? "standard") => mode === "standard" || tool.risk === "read";
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "../../__tests__/helpers/run-failure-reporting.js";
 import { randomUUID } from "node:crypto";
 import { eq } from "drizzle-orm";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
@@ -917,3 +918,6 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a
    expect(agentTaskRunCalls(callsBefore)).toHaveLength(1);
  });
 });
+
+
+enableRunFailureReporting();
@@ -1,3 +1,4 @@
+import { enableRunFailureReporting } from "../../__tests__/helpers/run-failure-reporting.js";
 import { hasNativeLocalProcessStop } from "../native-local-process-stop.js";
 import { randomUUID } from "node:crypto";
 import { spawn } from "node:child_process";
@@ -1151,3 +1152,6 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
    ).toBe(true);
  }, 30_000);
 });
+
+
+enableRunFailureReporting();
@@ -181,7 +181,8 @@ export function collectRunFailureSecretValues(
      PUBLIC_ENV_KEYS.has(key.toUpperCase()) ||
      (PUBLIC_NUMBER_ENV_KEYS.has(key.toUpperCase()) && /^\d+$/.test(value)) ||
      (key === "BASE_URL" && value === "/") ||
-      (key === "MODE" && /^(?:test|development|production)$/.test(value))
+      (key === "MODE" && /^(?:test|development|production)$/.test(value)) ||
+      (key === "PAPERCLIP_SENTRY_RUN_FAILURES" && /^(?:off|unexpected|all)$/.test(value))
    )) ||
      (PUBLIC_BOOLEAN_ENV_KEYS.has(key.toUpperCase()) && /^(?:0|1|true|false)$/i.test(value));
    if (declared.has(key) || !publicSetting) values.push(value);
@@ -219,6 +220,19 @@ export function collectRunFailureDiagnostics(run: Run, options: RunFailureReport
    "runtimeMode", "executionStage", "nativePhase", "driverKind", "driverVersion",
  ]);
  if (options.phase) execution.failurePhase = options.phase;
+  const providerConfig = read(read(run.runnerProfileJson, "nativeExecutionInput"), "provider");
+  const providerKind = read(providerConfig, "kind");
+  const model = read(providerConfig, "model");
+  if (typeof providerKind === "string") execution.providerKind = providerKind;
+  if (typeof model === "string") execution.model = model;
+  for (const field of ["startedAt", "finishedAt"] as const) {
+    const value = run[field];
+    if (value instanceof Date && Number.isFinite(value.getTime())) execution[field] = value.toISOString();
+  }
+  const originalFailureCode = read(run.resultJson, "originalFailureCode");
+  if (typeof originalFailureCode === "string" && /^[a-z][a-z0-9_]{1,99}$/.test(originalFailureCode)) {
+    execution.originalFailureCode = originalFailureCode;
+  }
  if (run.startedAt && run.finishedAt) {
    const durationMs = run.finishedAt.getTime() - run.startedAt.getTime();
    if (Number.isFinite(durationMs) && durationMs >= 0) execution.durationMs = durationMs;
@@ -368,6 +382,10 @@ export function collectRunFailureDiagnostics(run: Run, options: RunFailureReport
      if (typeof value === "string") entry[field] = value;
      else if (field === "code" && typeof value === "number" && Number.isFinite(value)) entry.code = String(value);
    }
+    if (entry.code === undefined) {
+      const reasonCode = read(error, "reasonCode");
+      if (typeof reasonCode === "string") entry.code = reasonCode;
+    }
    const status = read(error, "status") ?? read(error, "statusCode");
    if (typeof status === "number" && Number.isInteger(status) && status >= 100 && status <= 599) entry.status = status;
    const requestId = read(error, "requestId") ?? read(error, "request_id");
@@ -13,6 +13,7 @@ import { logger } from "../middleware/logger.js";
 import { isUnexpectedRunCancellation } from "./run-cancellation.js";
 import { readConnectionFailure, type ConnectionFailure } from "@paperclipai/adapter-utils/connection-failure";
 import { isAiConnectionConfigurationReason } from "./ai-connection-configuration-failure.js";
+import { readRunFailureReportingMode, type RunFailureReportingMode } from "../sentry-run-failure-policy.js";

 type HeartbeatRun = typeof heartbeatRuns.$inferSelect;

@@ -37,6 +38,7 @@ function isRunFailureStatus(status: string): status is RunFailureStatus {
 }

 function readTaskId(run: HeartbeatRun): string | null {
+  if (run.issueId) return run.issueId;
  if (run.nativeIssueId) return run.nativeIssueId;
  const contextIssueId = run.contextSnapshot?.issueId;
  return typeof contextIssueId === "string" && contextIssueId.length > 0 ? contextIssueId : null;
@@ -207,15 +209,19 @@ async function isRejectedExternalBotWake(db: Db, run: HeartbeatRun): Promise<boo
 * shutdown find and wait for the report — see `waitForPendingRunFailureReports`.
 */
 export function reportRunFailure(db: Db, run: HeartbeatRun, options: RunFailureReportOptions = {}): Promise<void> {
+  const mode = readRunFailureReportingMode();
+  if (mode === "off") return Promise.resolve();
  if (!isRunFailureStatus(run.status)) return Promise.resolve();
  if (run.status === "cancelled" && !isUnexpectedRunCancellation(run)) return Promise.resolve();
-  if (isMissingSecretConfigurationBlocker(run, options)) return Promise.resolve();
-  if (isAiConnectionSelectionBlocker(run, options)) return Promise.resolve();
-  if (isLocalPathWorkspaceConfigurationBlocker(run, options)) return Promise.resolve();
-  if (isGitConnectionFailure(run, options)) return Promise.resolve();
-  if (isProviderQuotaFailure(run, options)) return Promise.resolve();
+  if (mode === "unexpected" && (
+    isMissingSecretConfigurationBlocker(run, options) ||
+    isAiConnectionSelectionBlocker(run, options) ||
+    isLocalPathWorkspaceConfigurationBlocker(run, options) ||
+    isGitConnectionFailure(run, options) ||
+    isProviderQuotaFailure(run, options)
+  )) return Promise.resolve();
  const runStatus = run.status;
-  const report = captureTerminalRunFailure(db, run, runStatus, options);
+  const report = captureTerminalRunFailure(db, run, runStatus, options, mode);
  pendingRunFailureReports.add(report);
  void report.finally(() => pendingRunFailureReports.delete(report));
  return report;
@@ -226,6 +232,7 @@ async function captureTerminalRunFailure(
  run: HeartbeatRun,
  runStatus: RunFailureStatus,
  options: RunFailureReportOptions,
+  mode: RunFailureReportingMode,
 ): Promise<void> {
  try {
    const snapshot = redactRunFailureSecretValues({
@@ -242,19 +249,16 @@ async function captureTerminalRunFailure(
      .where(and(eq(agents.id, run.agentId), eq(agents.companyId, run.companyId)))
      .then((rows) => rows[0] ?? null);

-    if (isHermesConnectionFailure(run, options, agent?.adapterType)) return;
-    // An unreadable receipt proves nothing; keep the failure reportable.
-    const rejectedExternally = await isRejectedExternalBotWake(db, run).catch((err) => {
-      logger.warn({ err, runId: run.id }, "failed to read bot wake receipts for run failure report");
-      return false;
-    });
-    if (rejectedExternally) return;
+    if (mode === "unexpected") {
+      if (isHermesConnectionFailure(run, options, agent?.adapterType)) return;
+      const rejectedExternally = await isRejectedExternalBotWake(db, run).catch((err) => {
+        logger.warn({ err, runId: run.id }, "failed to read bot wake receipts for run failure report");
+        return false;
+      });
+      if (rejectedExternally) return;
+    }

    const taskId = readTaskId(run);
-    if (!taskId) {
-      logger.warn({ runId: run.id }, "run failure report has no task id, skipping Sentry report");
-      return;
-    }

    // Resolve registered values before truncation. A failed resolution must
    // not send an incompletely redacted report.
@@ -264,6 +268,8 @@ async function captureTerminalRunFailure(
      redacted = await createRunSecretRedactionRegistry(db).redactForRun(run.companyId, run.id, snapshot);
    }
    captureRunFailure({
+      companyId: run.companyId,
+      agentId: run.agentId,
      taskId,
      runId: run.id,
      errorMessage: sanitizeRunFailureText(redacted.errorMessage, MAX_ERROR_MESSAGE_LENGTH),
@@ -362,7 +362,9 @@ export function deduplicateToolDescriptors(tools: readonly ToolGatewayDescriptor
    const previous = unique.get(tool.name);
    if (previous) {
      if (!isDeepStrictEqual(previous, tool)) {
-        throw new ToolGatewayHttpError(409, "Conflicting tool definitions share the same name.", "tool_name_collision");
+        throw new ToolGatewayHttpError(409,
+          `Conflicting tool definitions share the name ${JSON.stringify(tool.name)} (connections ${JSON.stringify([previous.connectionId ?? null, tool.connectionId ?? null])}).`,
+          "tool_name_collision");
      }
      continue;
    }