mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 09:09:58 -10:00
test(evals): renew the reporting lock after iMessage update (#19283)
## Thinking Path > - Paperclip verifies the selected source and reporting dependencies before it obtains model credentials. > - The reporting helper admits only an exact reviewed lock or its exact reviewed correction. > - Master updated the iMessage dependency and protobuf after the last reporting-policy review. > - The controller lock therefore stopped matching either admitted hash. > - A real five-case launch failed at this guard before any case execution. > - This PR renews the exact hashes and keeps the existing patch and rejection mechanism unchanged. > - A normal-CI repository check now exercises the real helper against the committed lock before paid execution. ## Linked Issues or Issue Description Refs: #19261, #18914. **What happened?** The five selected execution jobs in the retained launch 3002 stopped during “Verify source-bound build and authenticated selection before credentials.” The first observed error was `Reporting lock changed; review the dependency correction before install` in the trusted controller helper. Model-key and original case execution steps were skipped. The failed attempt remains infrastructure evidence; this PR makes no native product-grade claim. **Expected behavior** Admit the exact reviewed current controller dependency graph. Keep rejecting unknown locks, changed correction patches, and wrong output bytes before credentials or provider execution. **Steps to reproduce** Run the reporting helper against the committed current-master lock. The parent policy recognizes 754… and 60f…, while the actual controller lock is 6f252b…. The parent stops at the same pre-credential boundary as AWS. The existing correction patch reverses and reapplies cleanly to that lock. ## What Changed - Renew only the reporting policy input and output SHA256 values. The output is the exact committed current-master lock after `@photon-ai/advanced-imessage` 2.1.0→2.2.0 and `@bufbuild/protobuf` 2.15.0→2.16.0. - Derive input `ea91060774e3289d5d07bc58f72d8953f5a6dcb9db98aa9a1adf202476bf89ac` by reversing the existing correction patch from the current lock. Reapplying that byte-identical patch yields output `6f252b0c8099ed837a297f97f1fc5e14e96bc7530b0239065d0789b33b2ee477` exactly. - Keep patch SHA256 `2adced9a66052ba5752140ec6cc6670f679552a9a8559bd1faea10036afeeb97`, helper code, and guard behavior unchanged. Update the policy basis to describe the review truthfully. - Add one normal server-suite check. It copies the committed lock and reviewed policy into a temporary directory and runs the real helper. The prior focused suite already covered this contract, but its runner-e2e location is outside normal root Vitest projects. `server/vitest.config.ts` already includes `scripts/**/*.test.mjs`; no workflow change is needed. No `pnpm-lock.yaml`, workflow, source pin, descriptor, oracle, model, permission, credential, scanner, or case bound changes. Immutable source 6773 and the failed launch remain unchanged. ## Verification - Parent existing reporting suite: 4 fail and 3 pass because the committed lock is outside the admitted pair. Renewed policy: all 7 pass, including unknown-lock, changed-patch, wrong-output, and refresh-path rejection controls. - New normal-CI repository control fails with the exact paid-step reporting-lock error on the parent policy. The corrected combined run passes 8 controls across 2 suites. - Exact patch reverse/forward roundtrip preserves the full current lock bytes. The actual CLI returns `already-reviewed`; the working lock hash stays 6f252b… and is not modified. - Focused command: `node node_modules/vitest/vitest.mjs run --config ../report-lock-imessage-vitest.config.mts`. The external local config selects only the two affected suites with the existing 30-second test limit. - `git diff --check` passes. No SDK/native/source build, model call, evaluation retry, or private archive download was needed for local proof. Full workspace typecheck/build were not run for this policy-and-test change. Latest-head required CI and genuine review remain pending. ## Risks This policy deliberately rejects the next unreviewed lock change. The normal-CI check should expose that drift before paid fanout. The input is the current lock with the already approved correction reversed; it is not a claim that a new dependency graph was resolved or installed locally. Local guard controls do not prove a repaired AWS execution or any of the five product journeys. The next measured attempt still requires normal protected merge and fresh actual-controller admission. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository tools, and local code execution. The host did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <[email protected]>
This commit is contained in:
1 parent
29afc633f6
commit
3fca8a1500
2 files changed
+29
-3
No files matched your search
@@ -0,0 +1,26 @@
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { copyFile, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { expect, it } from 'vitest';
|
||||
import { prepareReportLock } from '../../tests/runner-e2e/prepare-report-lock.mjs';
|
||||
|
||||
it('admits the committed controller lock through the reviewed reporting policy before paid execution', async () => {
|
||||
const repository = fileURLToPath(new URL('../../', import.meta.url));
|
||||
const root = await mkdtemp(path.join(tmpdir(), 'controller-report-policy-'));
|
||||
const relative = 'tests/runner-e2e';
|
||||
try {
|
||||
await mkdir(path.join(root, relative), { recursive: true });
|
||||
for (const file of ['report-lock.json', 'report-lock.patch']) {
|
||||
await copyFile(path.join(repository, relative, file), path.join(root, relative, file));
|
||||
}
|
||||
await writeFile(path.join(root, 'pnpm-lock.yaml'), execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: repository }));
|
||||
await prepareReportLock(root);
|
||||
const policy = JSON.parse(await readFile(path.join(root, relative, 'report-lock.json'), 'utf8'));
|
||||
expect(createHash('sha256').update(await readFile(path.join(root, 'pnpm-lock.yaml'))).digest('hex')).toBe(policy.outputSha256);
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"inputSha256": "7540075e96f9a93924a1e54775c17346c5760dd5d02902d83fb5228f918996ba",
|
||||
"outputSha256": "60f4c06bca47af3ab9d74281f2ef2e1ff559d566b862ba51c2fbe6c8e5bac35b",
|
||||
"inputSha256": "ea91060774e3289d5d07bc58f72d8953f5a6dcb9db98aa9a1adf202476bf89ac",
|
||||
"outputSha256": "6f252b0c8099ed837a297f97f1fc5e14e96bc7530b0239065d0789b33b2ee477",
|
||||
"patchSha256": "2adced9a66052ba5752140ec6cc6670f679552a9a8559bd1faea10036afeeb97",
|
||||
"basis": "Generated with the trusted master lockfile refresh. Review this offline correction and its hashes with the dependency changes."
|
||||
"basis": "Reviewed current master after the advanced-imessage 2.2.0 and protobuf 2.16.0 lock update. The existing correction patch is unchanged; its reverse/forward roundtrip reproduces this exact output. Unknown locks, changed patches and wrong outputs still reject."
|
||||
}
|
||||
Reference in new issue
Block a user