test(evals): pin reviewed outcome and native handoff repairs (#18976)

## Thinking Path

> - The catalog must measure the original human requests through
ordinary product controls.
> - The latest four-case run exposed two oracle false negatives and a
real suppressed native handoff.
> - Preserve those original failures and fix only their demonstrated
causes.
> - Keep the reviewed company-selector successor and add the accepted
timeline, approved-replay and personal-completion corrections.
> - Select native standing-chat terminal answers by authority while
preserving governed waits and ordinary-task behavior.
> - Bind the resulting immutable candidate and its three changed oracle
versions through the existing manifest and descriptor snapshot.
> - The next personal-secret attempt must prove the complete
secure-entry and worker journey before company selection or a final
ten-case regression.

## Linked Issues or Issue Description

Refs #18971, #18967, #18968, #18972, #18977. Follows #18974.

The latest four-case run retained original failures for timeline,
approved tool replay and personal-secret setup. Team installation
independently passed. The personal native run had actionable final
directions, but presentation selected an earlier ordinary comment and
suppressed that answer. This pin makes the reviewed corrections
available; it does not regrade those attempts or claim a new native
pass.

## What Changed

- Pin immutable candidate `5f5e2ca8f18063d051c3d948105f0049562fe4ce` on
`codex/chat-catalog-four-outcome-repairs-20261104`.
- Refresh the existing descriptor snapshot and matching manifest
metadata: timeline v5, app-review v5 and personal-secret v4. Company v3
is retained. Preserve the prior reviewed company recipient correction.
- Include the reviewed structural standing-chat response selection and
finished-turn recovery, preserving earlier comment history, governed
interaction ownership and no-final fallback. The superseded prose
classifier is absent.
- Preserve all 244 identities, original queries, provider/model, human
and native counts, clocks, launcher bytes, private evidence limits and
qualification holds. The existing SDK registry/compiler proof is
unchanged.
- The infrastructure change is three metadata files. No product fix is
merged by this PR; its owning core PR remains separate.

## Verification

- Exact final candidate: 510 owning tests passed across 19 suites,
including catalog contract/consistency and lazy declaration loading. The
candidate declares 242 cases plus two existing legacy tasks; no fixture
callbacks execute during admission.
- Candidate response summary suite: 96 tests passed. The owner
separately reports 103 summary/notice controls and 14 recovery
PostgreSQL controls; these are distinct from candidate results.
- Candidate PostgreSQL scope/recovery controls: four passed (three
native/ordinary/legacy caller scopes and the new server-finished yielded
unknown-channel recovery). The focused filter left 52 unrelated controls
unselected. Earlier setup attempts failed before assertions due missing
native library links and plugin-sdk output; package-cwd shipped
hydration and one owning SDK compiler build enabled this final actual
run. No source workaround, timeout change or acceptance weakening was
used.
- Two deliberate baseline joins preserve the frozen candidate: the
caller test adds only its required `issueComments` import; recovery adds
only current-conversation matching yielded result/terminal with no owned
interaction, retaining all existing
succeeded/completed/session/ownership/authorization guards. The test
adds only `statusDecisions` and needed fixture options. Fresh-master
ordinary `needs_review` behavior and unrelated imports were not copied.
- All 131 existing pin, TestOps contract/API, adapter, reconciliation
and streaming controls passed. The descriptor snapshot exactly matches
the candidate declaration bytes. Diff whitespace checks passed.
- SDK source guard: 506 inputs, `changed: false`, `runtimePinsChanged:
false`, registry
`efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. The
local compiler invocation supplied missing ignored test output; it is
not a renewed SDK qualification or runtime certificate.
- Existing reporting-lock guard returned `already-reviewed`. Fresh
protected master `86814c6550f52e99ede150a883bdc773cbb540e3` has admitted
lock `60f4c06bca47af3ab9d74281f2ef2e1ff559d566b862ba51c2fbe6c8e5bac35b`.
Repeat the existing serving-master guard after merge and before a new
Submit.
- The earlier pin run exposed a pre-existing remote-failure-retention
fixture failure. Normal protected merge #18984 repaired that baseline on
master `04a24f4e06da68043378bdf3646f50292efd5316`; its 47 real jobs and
aggregates succeeded despite delayed GitHub UI/API status projection.
This branch normally merges that actual master. Its three pin metadata
files are byte-identical to prior reviewed head
`dec78737cc520ce6d489fb8bfccafb460b79e870`, with the same immutable
candidate, descriptor bytes and model/count/clock/provenance fields. No
baseline fixture patch was duplicated in this PR.
- On updated head `bd5402d4688e690ffd9f1f80081577733ceeda25`, the
existing 131 controls pass, reporting lock is already reviewed, and
existing `selectedCases` admits exactly case 580 without execution.
Fresh actual master lock remains admitted.
- Exact-head normal CI run 41865757115 has completed its full actual job
set: 48 succeeded and one legitimate selector skip, including `ci / e2e`
and `ci / verify`. RunsOn completion records and exact GitHub aggregate
logs corroborate the results. Windows run 41865757064 completed SUCCESS;
its exact log reports all executed Rust test results as passing. GitHub
UI/API/check-rollup projections still contain stale queued/in-progress
rows, as independently observed during #18984; this is not a claim that
those displayed rows are green. A normal protected merge must be
accepted by GitHub without override.
- Fresh exact-head Greptile is 5/5, the current Paperclip assessment
explicitly says 5/5 and Ship, all review threads are resolved (zero
threads), and the security scan succeeded. Earlier company-only checks
are not used to qualify this head. Full repository build/typecheck/test
were not repeated locally for this metadata PR; owning source checks and
ordinary hosted CI provide the relevant gates.
- No model run, dispatch, retry, cancellation, secret mutation or merge
was performed by this update.

## Risks

The structural correction changes final-answer selection only for
admitted native standing conversations; it still needs a new real
secure-entry journey. Oracle corrections can admit previously rejected
wording, so retained positive and negative controls remain required.
Original failed and unknown attempts are immutable. Company selection
remains gated on actual personal-secret secure-entry/worker proof, and
all 244 cases remain unqualified as a whole.

## Model Used

OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis,
isolated integration and local checks. The host does not expose the
context-window limit. The evaluated model remains OpenCode/OpenRouter
`deepseek/deepseek-v4.1-flash-1017`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes —
manifest and expected-pin assertion agree
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green — actual job/aggregate results
passed; stale UI/API projection is disclosed above
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <[email protected]>

tests/runner-e2e/chat-full-catalog-testops-manifest.json

@@ -1,6 +1,6 @@
 {
   "schema": "paperclip.chat-full-catalog-testops-manifest/v1",
-  "targetSha": "e757e447371c4781a3eff3a0ea9dc9c4eceb060a",
+  "targetSha": "5f5e2ca8f18063d051c3d948105f0049562fe4ce",
   "metadataSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
   "profile": "runner-opencode",
   "model": "openrouter/deepseek/deepseek-v4.1-flash-1017",
@@ -172,7 +172,7 @@
       "baseTargetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16",
       "previousTargetSha": "ab2019e976076ec0eb60b026844f3d9211cb0208",
       "definitionSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
-      "candidateDescriptorSha256": "e7681aa6bb41a8b71f6e30ae3279e0579c72f55dabdaeedbf984e22268b070db",
+      "candidateDescriptorSha256": "640743a65be417330144b66858704e1e378fe2edf26a7977910d1756fde9b692",
       "changedDefinitionFiles": [
         {
           "path": "tests/runner-e2e/chat-scenarios/companies.yaml",
@@ -232,7 +232,7 @@
           "key": "activity.work-timeline",
           "field": "oracleVersion",
           "previous": "paperclip.chat-catalog.internal.v1",
-          "current": "paperclip.chat-catalog.work-timeline.v4"
+          "current": "paperclip.chat-catalog.work-timeline.v5"
         },
         {
           "key": "activity.work-timeline",
@@ -308,7 +308,7 @@
           "key": "app-permissions.ask-before-action",
           "field": "oracleVersion",
           "previous": "paperclip.chat-catalog.external.v1",
-          "current": "paperclip.chat-catalog.app-review.v4"
+          "current": "paperclip.chat-catalog.app-review.v5"
         },
         {
           "key": "app-permissions.test-as-agent",
@@ -694,7 +694,7 @@
           "key": "secrets.personal-secret",
           "field": "oracleVersion",
           "previous": "paperclip.chat-catalog.external.v1",
-          "current": "paperclip.chat-catalog.personal-secret.v3"
+          "current": "paperclip.chat-catalog.personal-secret.v4"
         },
         {
           "key": "skills.author-skill",
@@ -938,7 +938,7 @@
           "key": "activity.work-timeline",
           "field": "oracleVersion",
           "previous": "paperclip.chat-catalog.internal.v1",
-          "current": "paperclip.chat-catalog.work-timeline.v4"
+          "current": "paperclip.chat-catalog.work-timeline.v5"
         },
         {
           "key": "agents.choose-primary",
@@ -1008,7 +1008,7 @@
           "key": "app-permissions.ask-before-action",
           "field": "oracleVersion",
           "previous": "paperclip.chat-catalog.external.v1",
-          "current": "paperclip.chat-catalog.app-review.v4"
+          "current": "paperclip.chat-catalog.app-review.v5"
         },
         {
           "key": "app-permissions.test-as-agent",
@@ -1381,7 +1381,7 @@
           "key": "secrets.personal-secret",
           "field": "oracleVersion",
           "previous": "paperclip.chat-catalog.external.v1",
-          "current": "paperclip.chat-catalog.personal-secret.v3"
+          "current": "paperclip.chat-catalog.personal-secret.v4"
         },
         {
           "key": "skills.author-skill",
@@ -1745,7 +1745,7 @@
       },
       "candidateLauncherSnapshot": {
         "path": "fixtures/chat-repair-candidate-launcher.txt",
-        "targetSha": "e757e447371c4781a3eff3a0ea9dc9c4eceb060a",
+        "targetSha": "5f5e2ca8f18063d051c3d948105f0049562fe4ce",
         "sourcePath": "tests/runner-e2e/launch.ts",
         "gitBlobOid": "dba1467e211611f37937e9e1fc4c0c5be4329cbc"
       }
@@ -2288,7 +2288,7 @@
         "activity/timeline"
       ],
       "checks": null,
-      "oracleVersion": "paperclip.chat-catalog.work-timeline.v4",
+      "oracleVersion": "paperclip.chat-catalog.work-timeline.v5",
       "sourceFiles": [
         "chat-catalog-domains/internal-definition.ts",
         "chat-catalog-domains/internal-support.ts",
@@ -3366,7 +3366,7 @@
         "questions-and-approvals/tool-review"
       ],
       "checks": null,
-      "oracleVersion": "paperclip.chat-catalog.app-review.v4",
+      "oracleVersion": "paperclip.chat-catalog.app-review.v5",
       "sourceFiles": [
         "chat-catalog-domains/external-app-review.ts",
         "chat-catalog-domains/external-app-review-outcomes.ts",
@@ -11196,7 +11196,7 @@
         "secrets/user-secrets"
       ],
       "checks": null,
-      "oracleVersion": "paperclip.chat-catalog.personal-secret.v3",
+      "oracleVersion": "paperclip.chat-catalog.personal-secret.v4",
       "sourceFiles": [
         "chat-catalog-domains/external-personal-secret.ts",
         "chat-catalog-domains/external-secret-question.ts",

tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs

@@ -103,7 +103,7 @@ test('repair source metadata binds the corrected candidate and explicit changed
   const repair = manifest.source.repairMetadata;
   assert.equal(repair.baseTargetSha, '3b66c3c144360ddf32714a0742fd774b8ceafb16');
   assert.equal(repair.previousTargetSha, 'ab2019e976076ec0eb60b026844f3d9211cb0208');
-  assert.equal(manifest.targetSha, 'e757e447371c4781a3eff3a0ea9dc9c4eceb060a');
+  assert.equal(manifest.targetSha, '5f5e2ca8f18063d051c3d948105f0049562fe4ce');
   assert.equal(repair.definitionSourceSha, manifest.metadataSourceSha);
   assert.equal(repair.candidateDescriptorSnapshot, 'fixtures/chat-repair-candidate-descriptors.json');
   const descriptorBytes = readFileSync(new URL(repair.candidateDescriptorSnapshot, import.meta.url));

tests/runner-e2e/fixtures/chat-repair-candidate-descriptors.json

@@ -258,7 +258,7 @@
       "../../server/src/services/native-runtime/status-decision-committer.ts",
       "../../server/src/services/work-timeline.ts"
     ],
-    "oracleVersion": "paperclip.chat-catalog.work-timeline.v4",
+    "oracleVersion": "paperclip.chat-catalog.work-timeline.v5",
     "coverageGaps": [],
     "expectedRunCount": 3,
     "minimumRunCount": 3,
@@ -830,7 +830,7 @@
       "../../ui/src/components/task-chat/TaskChatComposer.tsx",
       "chat-scenarios/app-permissions.yaml"
     ],
-    "oracleVersion": "paperclip.chat-catalog.app-review.v4",
+    "oracleVersion": "paperclip.chat-catalog.app-review.v5",
     "expectedRunCount": 6,
     "expectedHumanMessageCount": 3,
     "minimumRunCount": 6,
@@ -4981,7 +4981,7 @@
       "../../ui/src/pages/secrets/SetMyUserSecretDialog.tsx",
       "chat-scenarios/secrets.yaml"
     ],
-    "oracleVersion": "paperclip.chat-catalog.personal-secret.v3",
+    "oracleVersion": "paperclip.chat-catalog.personal-secret.v4",
     "expectedRunCount": 3,
     "expectedHumanMessageCount": 2,
     "minimumRunCount": 3,