mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 06:16:06 -10:00
fix(native): preserve web files without settling remote recovery (#19814)
## Thinking Path > - Paperclip manages work that AI agents perform for users. > - Native sessions retain files on the web server and state in a remote provider. > - Replacing the web server must preserve its files without changing remote recovery authority. > - The existing maintenance owner requires destroyed-provider receipts even when all web writers have settled. > - This can keep the old web volume required only because a remote recovery obligation remains unresolved. > - This pull request adds a closed web-byte evidence mode to the existing preservation owner. > - The benefit is durable web-file coverage with unchanged remote references, failures and continuation policy. ## Linked Issues or Issue Description Refs #19748, #19806. A search for native harness filesystem evidence and open web/harness changes found no duplicate implementation. **What happened?** A settled successful native source can retain a raw controller, a separate warm supervisor and provider-home files while its historical external lease has no termination receipt. The existing filesystem-independence owner refuses before it can preserve these web files. One retained skill file can also remain physically present after its version row was deleted. **Expected behavior** Preserve every nonexcluded web file as authenticated evidence. Keep missing files absent and deleted skill versions deleted. Leave VM references, recovery actions, failures and execution authority unchanged. Validate the published web-byte receipt on an empty replacement using database bindings only. **Steps to reproduce** Use the included successful-source fixture with a settled web owner, an unresolved external lease, real controller/supervisor files and a retained skill whose version row is deleted. The baseline rejects at its destroyed-provider guard. The new explicit maintenance mode preserves the files and validates on a replacement. A separate fixture uses a prior successful source, a failed retained last run, a genuinely absent controller, a warm supervisor and 267 durable checkpoint parts. **Paperclip version/commit** Baseline `384fa2b7be`; final branch rebased onto `585c885168`. Current head: `990210d7e9`. **Deployment mode** Original-volume maintenance followed by an ephemeral replacement. No hosted operation or provider request is part of this PR. ## What Changed - Add explicit `web_evidence` maintenance mode and a separate `paperclip.native-harness-web-evidence.v1` receipt. Keep the existing terminated-provider mode and old manifest shapes. - Archive the closed canonical session-root inventory, the separate warm supervisor and exact admitted skill manifests through the existing encrypted `original_volume_source_evidence` owner. Preserve bytes and modes. Keep canonical credential exclusions. - Preserve local skill edits and deleted-version files as forensic evidence. Create no skill version, executable checkpoint, provider backup or recovery authority. - Require original-volume capability, settled web writers, physical/finalizer locks, exact task/source/cohort/input bindings and final byte/mode/membership comparisons. Keep genuinely absent raw controllers absent. - Bind allocation and stamp identity while allowing later external status and cleanup observations. Validate replacements from durable database and archive bindings without the original anchor, filesystem or storage reads. ## Verification - Node 24.22.2, PostgreSQL 18.1, macOS arm64. The unchanged baseline fails at the real destroyed-provider guard. The final revision passes all 85 owner, original-volume capability and unchanged continuation-policy cases in 21.14 seconds. - Tests perform real encryption, persistence through the S3 storage interface backed by the owned local fixture, authenticated restore and exact byte/mode comparisons. The 281-byte file remains present while its version row remains absent. The evidence archive stays separate from the provider-session backup scope. - The supervisor-only case preserves 267 database checkpoint parts, keeps the raw controller absent and validates without original-volume identity. Existing strict-mode and old receipt controls pass. - Negative controls cover active web writers, expired but owned finalizers, physical owners, foreign controller/supervisor identities, missing manifests, symlinks, changed bytes/modes/membership/source during upload, changed durable bindings, incomplete receipt coverage and invalid late publication. Concurrent attempts retain one valid winner and one audit event. - Server TypeScript, module boundaries, diff checks and private-data scans pass. The verification planner's first invocation used a wrong script path; its canonical invocation refused a plan because fixture inputs changed during selection. Neither attempt counts as a pass. Required CI will provide broader application coverage. - Earlier fixture and treatment failures remain recorded. They exposed a missing helper, JSONB property-order instability, missing separate-supervisor mode coverage and fixture timing/source/count errors. The final same owner suite passes without weakened assertions. - A canonical cleanup audit found two additional observation fields: the cleanup controller and manual-attempt ID. Their real publication updates reproduced false receipt invalidation on the first PR head. The narrow correction keeps WEB coverage valid through those observations while changed allocation identity still fails. The exact same 85-case suite passes afterward. The final `990210d7e9` head has fresh Greptile 5/5 with no comments or unresolved threads. Required CI run [42370607882](https://github.com/paperclipai/paperclip/actions/runs/42370607882) passes: 55 successful checks and four intended skips, plus successful Snyk. The old head's CI was superseded by this correction and is not counted as a pass. From `server/`: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts src/services/native-runtime/native-harness-original-volume.test.ts src/services/native-runtime/native-harness-filesystem-independence-continuation.integration.test.ts --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks This changes a maintenance trust boundary. It proves web-byte preservation, not provider recovery or termination. External recovery obligations and existing resume failures remain. Unknown layouts, missing manifests, active local owners and changed authority fail closed. Invalid receipts are not automatically replaced. The inventory has independent limits of 128 entries, 16 MiB per file and 64 MiB total. Larger valid data stays on the original volume and blocks this mode. Canonical credential exclusions remain excluded. Evidence uses the existing encrypted ledger and retention roots; it does not become an executable backup. Old receipts remain byte-compatible. Startup adds bounded per-retained-scope database reads. Measure the affected cohort's actual serving startup cost before enrollment. Validation adds no filesystem or object read, worker or recurring poll. Live source qualification and publication remain separate after review, merge and normal delivery. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and PostgreSQL/filesystem test execution. The runtime does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>
This commit is contained in:
1 parent
585c885168
commit
ae1f781cac
5 files changed
+551
-63
No files matched your search
+28
-2
@@ -91,8 +91,34 @@ validate the published receipt using only database bindings, without the old
|
||||
volume, a fresh source witness, or storage reads. Changed or missing bindings
|
||||
invalidate it. Existing resume and replacement policies remain authoritative,
|
||||
including governed-wait failures and warm-session refusals. No provider history
|
||||
is reconstructed or declared recovered, and uncertain provider sources remain
|
||||
enrollment blockers.
|
||||
is reconstructed or declared recovered.
|
||||
|
||||
The same original-volume maintenance owner has a separate `web_evidence` mode.
|
||||
It stores a `paperclip.native-harness-web-evidence.v1` receipt. This mode proves
|
||||
that the web server's retained files have durable byte coverage. It does not
|
||||
require an external VM's termination receipt and does not settle that VM's
|
||||
recovery obligation. It requires the same settled web controller, finalizer,
|
||||
task and physical-writer fences. The encrypted `original_volume_source_evidence`
|
||||
archive includes a closed inventory of all nonexcluded files under the canonical
|
||||
session root, the separate warm supervisor, and each admitted skill manifest.
|
||||
Files retain their bytes and modes, including local edits and a skill whose
|
||||
version row was deleted. Such a skill remains forensic evidence: the owner does
|
||||
not recreate its version or make it executable. Canonical credential exclusions
|
||||
stay excluded. Unknown directories, symlinks, incomplete manifests, active web
|
||||
writers and changed source facts fail closed. The same 128-entry, 16-MiB-file and
|
||||
64-MiB-total limits apply independently to this inventory.
|
||||
|
||||
A genuinely absent raw controller stays absent. A supervisor-only archive still
|
||||
requires the exact durable controller checkpoint and source identity. Old
|
||||
terminated-provider receipts and history-free manifests keep their existing
|
||||
shape. External lease status and cleanup observations can change without
|
||||
creating a new web filesystem dependency. Allocation, stamp, selected source,
|
||||
checkpoint and admitted-input changes invalidate coverage. VM references,
|
||||
recovery actions, warnings and continuation policies stay unchanged. Replacement
|
||||
validation uses only the published archive and database bindings; it needs no
|
||||
original-volume anchor or filesystem read. These validations add bounded
|
||||
per-retained-scope database reads during startup. They do not add a worker or a
|
||||
poll. Missing or invalid byte evidence remains an enrollment blocker.
|
||||
|
||||
Receipt validation adds bounded per-scope database reads to startup eligibility.
|
||||
It does not add a recurring scan or network work. Qualification must measure the
|
||||
|
||||
@@ -13,7 +13,7 @@ export type SettledControllerArchiveOwner = {
|
||||
kind: "settled_controller_archive";
|
||||
companyId: string; issueId: string; agentId: string; runId: string; sessionId: string;
|
||||
selectedSessionId: string; selectedLastRunId: string;
|
||||
profileDigest: string; coordinatorDigest: string; controllerDigest: string; supervisorDigest: string | null;
|
||||
profileDigest: string; coordinatorDigest: string; controllerDigest: string | null; supervisorDigest: string | null;
|
||||
filesystemIndependence?: { source: OriginalHarnessVolumeCapability; scopeDigest: string; factsDigest: string };
|
||||
};
|
||||
const fileDigest = (value: string) => createHash("sha256").update(value).digest("hex");
|
||||
@@ -33,6 +33,7 @@ export function nativeControllerArchiveCoordinatorDigest(value: {
|
||||
|
||||
export async function assertSettledControllerArchiveSource(db: Db, owner: SettledControllerArchiveOwner) {
|
||||
function fail(): never { throw new Error("native_controller_archive_source_unsettled"); }
|
||||
if (owner.controllerDigest === null && !owner.filesystemIndependence) fail();
|
||||
if (owner.filesystemIndependence) {
|
||||
await assertOriginalHarnessVolumeSource(owner.filesystemIndependence.source);
|
||||
await (await import("./native-harness-filesystem-independence.js")).assertNativeHarnessFilesystemArchiveSource(db, owner);
|
||||
@@ -68,7 +69,7 @@ export async function assertSettledControllerArchiveSource(db: Db, owner: Settle
|
||||
normalizedSessionId: owner.sessionId, beforeCreatedAt: selectedRun.createdAt, includeProgressBarrier: true });
|
||||
if (!selected || selected.id !== owner.runId || nativeControllerArchiveProfileDigest(selected.runnerProfileJson) !== owner.profileDigest) fail();
|
||||
const material = await readLegacyNativeControllerArchiveDigests(selected);
|
||||
if (material.controllerDigest !== owner.controllerDigest
|
||||
if ((material.controllerDigest ?? null) !== owner.controllerDigest
|
||||
|| (material.supervisorDigest ?? null) !== owner.supervisorDigest) fail();
|
||||
}
|
||||
|
||||
|
||||
@@ -7,25 +7,36 @@ import type { StorageService } from "../../storage/types.js";
|
||||
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
|
||||
import { persistNativeHarnessBackup, verifyDurableNativeHarnessBackup } from "./native-harness-backup-storage.js";
|
||||
import type { SettledControllerArchiveOwner } from "./native-controller-archive-source.js";
|
||||
import { assertSkillSnapshotPath } from "../skill-snapshot.js";
|
||||
|
||||
export type NativeControllerHistoryEntry = { path: string; mode: number; type: "directory" }
|
||||
| { path: string; mode: number; type: "file"; content: string };
|
||||
export type NativeControllerHistoryManifestEntry = { path: string; mode: number; type: "directory" }
|
||||
| { path: string; mode: number; type: "file"; size: number; sha256: string };
|
||||
type Source = { companyId: string; issueId: string; runId: string; controller: string; supervisor?: string;
|
||||
export type NativeControllerWebEntry = { path: string; mode: number; type: "directory" }
|
||||
| { path: string; mode: number; type: "file"; content: Buffer };
|
||||
type WebEvidence<T> = { entries: readonly T[]; bindingDigest: string };
|
||||
type Source = { companyId: string; issueId: string; runId: string; controller?: string; supervisor?: string;
|
||||
history?: readonly NativeControllerHistoryEntry[];
|
||||
historyBindingDigest?: string;
|
||||
web?: WebEvidence<NativeControllerWebEntry>;
|
||||
purpose?: "filesystem_independence" };
|
||||
const sha256 = (value: string) => createHash("sha256").update(value).digest("hex");
|
||||
const sha256 = (value: string | Buffer) => createHash("sha256").update(value).digest("hex");
|
||||
|
||||
export function nativeControllerHistoryManifest(history: readonly NativeControllerHistoryEntry[]): NativeControllerHistoryManifestEntry[] {
|
||||
return history.map(entry => entry.type === "directory" ? { ...entry }
|
||||
: { path: entry.path, mode: entry.mode, type: entry.type, size: Buffer.byteLength(entry.content), sha256: sha256(entry.content) });
|
||||
}
|
||||
|
||||
export function nativeControllerWebManifest(entries: readonly NativeControllerWebEntry[]): NativeControllerHistoryManifestEntry[] {
|
||||
return entries.map(entry => entry.type === "directory" ? { ...entry }
|
||||
: { path: entry.path, mode: entry.mode, type: entry.type, size: entry.content.length, sha256: sha256(entry.content) });
|
||||
}
|
||||
|
||||
export function nativeControllerEvidenceDescriptor(source: { companyId: string; issueId: string; runId: string;
|
||||
controllerDigest: string; supervisorDigest: string | null; history?: readonly NativeControllerHistoryManifestEntry[];
|
||||
controllerDigest: string | null; supervisorDigest: string | null; history?: readonly NativeControllerHistoryManifestEntry[];
|
||||
historyBindingDigest?: string;
|
||||
web?: WebEvidence<NativeControllerHistoryManifestEntry>;
|
||||
purpose?: "filesystem_independence" }) {
|
||||
const history = source.history?.length ? source.history.map(entry => entry.type === "directory"
|
||||
? { path: entry.path, mode: entry.mode, type: entry.type }
|
||||
@@ -49,20 +60,45 @@ export function nativeControllerEvidenceDescriptor(source: { companyId: string;
|
||||
}
|
||||
}
|
||||
}
|
||||
const web = source.web ? { bindingDigest: source.web.bindingDigest,
|
||||
entries: source.web.entries.map(entry => entry.type === "directory" ? { path: entry.path, mode: entry.mode, type: entry.type }
|
||||
: { path: entry.path, mode: entry.mode, type: entry.type, size: entry.size, sha256: entry.sha256 }) } : undefined;
|
||||
if (source.controllerDigest === null && !web) throw new Error("native_controller_archive_authority_invalid");
|
||||
if (web) {
|
||||
const paths = new Set<string>(); let bytes = 0;
|
||||
if (source.purpose !== "filesystem_independence" || web.entries.length > 128 || !/^[a-f0-9]{64}$/.test(web.bindingDigest)) {
|
||||
throw new Error("native_controller_archive_authority_invalid");
|
||||
}
|
||||
for (const entry of web.entries) {
|
||||
assertSkillSnapshotPath(entry.path);
|
||||
if (!/^(?:(web-home|web-state|web-manifests)(\/|$)|web-supervisor\.json$)/.test(entry.path) || paths.has(entry.path)
|
||||
|| !["directory", "file"].includes(entry.type)
|
||||
|| !Number.isInteger(entry.mode) || entry.mode < 0 || (entry.mode & ~0o777) !== 0
|
||||
|| entry.path.includes("/") && !paths.has(entry.path.slice(0, entry.path.lastIndexOf("/")))
|
||||
|| entry.type === "file" && (!Number.isSafeInteger(entry.size) || entry.size < 0 || entry.size > 16 * 1024 * 1024
|
||||
|| !/^[a-f0-9]{64}$/.test(entry.sha256) || (bytes += entry.size) > 64 * 1024 * 1024)) {
|
||||
throw new Error("native_controller_archive_authority_invalid");
|
||||
}
|
||||
paths.add(entry.path);
|
||||
}
|
||||
}
|
||||
const manifest = JSON.stringify({ schema: "paperclip.legacy-controller-archive.v1", companyId: source.companyId,
|
||||
issueId: source.issueId, runId: source.runId, reason: source.purpose === "filesystem_independence"
|
||||
? "original_volume_source_evidence" : "original_runner_artifact_unproven",
|
||||
files: { "control-plane-state.json": source.controllerDigest,
|
||||
files: { ...(source.controllerDigest === null ? {} : { "control-plane-state.json": source.controllerDigest }),
|
||||
...(source.supervisorDigest === null ? {} : { "supervisor.json": source.supervisorDigest }),
|
||||
...(history ? Object.fromEntries(history.filter(entry => entry.type === "file").map(entry => [entry.path, entry.sha256])) : {}) },
|
||||
...(history ? { history, historyBindingDigest: source.historyBindingDigest } : {}) });
|
||||
...(history ? Object.fromEntries(history.filter(entry => entry.type === "file").map(entry => [entry.path, entry.sha256])) : {}),
|
||||
...(web ? Object.fromEntries(web.entries.filter(entry => entry.type === "file").map(entry => [entry.path, entry.sha256])) : {}) },
|
||||
...(history ? { history, historyBindingDigest: source.historyBindingDigest } : {}),
|
||||
...(web ? { web: { entries: web.entries, bindingDigest: web.bindingDigest, controllerAbsent: source.controllerDigest === null } } : {}) });
|
||||
return { manifest, scopeDigest: sha256(manifest), manifestSha256: `sha256:${sha256(manifest)}` };
|
||||
}
|
||||
|
||||
export function legacyNativeControllerArchive(source: Source) {
|
||||
return nativeControllerEvidenceDescriptor({ ...source, controllerDigest: sha256(source.controller),
|
||||
return nativeControllerEvidenceDescriptor({ ...source, controllerDigest: source.controller === undefined ? null : sha256(source.controller),
|
||||
supervisorDigest: source.supervisor === undefined ? null : sha256(source.supervisor),
|
||||
history: source.history ? nativeControllerHistoryManifest(source.history) : undefined });
|
||||
history: source.history ? nativeControllerHistoryManifest(source.history) : undefined,
|
||||
web: source.web ? { entries: nativeControllerWebManifest(source.web.entries), bindingDigest: source.web.bindingDigest } : undefined });
|
||||
}
|
||||
|
||||
export async function hasArchivedLegacyNativeController(db: Db, input: Source & { storage: StorageService; signal?: AbortSignal }) {
|
||||
@@ -73,7 +109,7 @@ export async function hasArchivedLegacyNativeController(db: Db, input: Source &
|
||||
export async function archiveFailedLegacyNativeController(db: Db, input: Source & {
|
||||
storage: StorageService; writersStopped?: true; onlineOwner?: SettledControllerArchiveOwner; signal: AbortSignal;
|
||||
}) {
|
||||
if (input.purpose || input.history?.length || input.onlineOwner?.filesystemIndependence) throw new Error("native_controller_archive_authority_invalid");
|
||||
if (input.controller === undefined || input.web || input.purpose || input.history?.length || input.onlineOwner?.filesystemIndependence) throw new Error("native_controller_archive_authority_invalid");
|
||||
return persistLegacyControllerEvidence(db, input);
|
||||
}
|
||||
|
||||
@@ -83,7 +119,8 @@ export async function archiveNativeControllerSourceEvidence(db: Db, input: Sourc
|
||||
if (input.purpose !== "filesystem_independence" || !input.onlineOwner.filesystemIndependence) {
|
||||
throw new Error("native_controller_archive_authority_invalid");
|
||||
}
|
||||
input = { ...input, history: input.history?.map(entry => ({ ...entry })) };
|
||||
input = { ...input, history: input.history?.map(entry => ({ ...entry })), web: input.web ? { ...input.web,
|
||||
entries: input.web.entries.map(entry => entry.type === "file" ? { ...entry, content: Buffer.from(entry.content) } : { ...entry }) } : undefined };
|
||||
(await import("./native-harness-filesystem-independence.js")).assertNativeHarnessFilesystemArchiveContents(
|
||||
input.onlineOwner, legacyNativeControllerArchive(input));
|
||||
await db.transaction(async tx => (await import("./native-controller-archive-source.js"))
|
||||
@@ -96,7 +133,7 @@ async function persistLegacyControllerEvidence(db: Db, input: Source & {
|
||||
}) {
|
||||
if (input.onlineOwner ? input.writersStopped || input.onlineOwner.kind !== "settled_controller_archive"
|
||||
|| input.onlineOwner.companyId !== input.companyId || input.onlineOwner.issueId !== input.issueId
|
||||
|| input.onlineOwner.runId !== input.runId || input.onlineOwner.controllerDigest !== sha256(input.controller)
|
||||
|| input.onlineOwner.runId !== input.runId || input.onlineOwner.controllerDigest !== (input.controller === undefined ? null : sha256(input.controller))
|
||||
|| input.onlineOwner.supervisorDigest !== (input.supervisor === undefined ? null : sha256(input.supervisor)) : !input.writersStopped) {
|
||||
throw new Error("native_controller_archive_authority_invalid");
|
||||
}
|
||||
@@ -105,13 +142,18 @@ async function persistLegacyControllerEvidence(db: Db, input: Source & {
|
||||
const root = await mkdtemp(join(tmpdir(), "pc-legacy-controller-archive-"));
|
||||
try {
|
||||
await writeFile(join(root, "manifest.json"), descriptor.manifest, { mode: 0o600 });
|
||||
await writeFile(join(root, "control-plane-state.json"), input.controller, { mode: 0o600 });
|
||||
if (input.controller !== undefined) await writeFile(join(root, "control-plane-state.json"), input.controller, { mode: 0o600 });
|
||||
if (input.supervisor !== undefined) await writeFile(join(root, "supervisor.json"), input.supervisor, { mode: 0o600 });
|
||||
for (const entry of input.history ?? []) {
|
||||
if (entry.type === "directory") await mkdir(join(root, entry.path), { mode: 0o700 });
|
||||
else await writeFile(join(root, entry.path), entry.content, { mode: entry.mode });
|
||||
}
|
||||
for (const entry of [...(input.history ?? [])].reverse()) await chmod(join(root, entry.path), entry.mode);
|
||||
for (const entry of input.web?.entries ?? []) {
|
||||
if (entry.type === "directory") await mkdir(join(root, entry.path), { mode: 0o700 });
|
||||
else await writeFile(join(root, entry.path), entry.content, { mode: entry.mode });
|
||||
}
|
||||
for (const entry of [...(input.web?.entries ?? [])].reverse()) await chmod(join(root, entry.path), entry.mode);
|
||||
await persistNativeHarnessBackup({ db, storage: input.storage, root, scopeDigest: descriptor.scopeDigest, signal: input.signal,
|
||||
owner: input.onlineOwner ?? { kind: "offline_migration", writersStopped: true, terminalFailureOnly: true,
|
||||
companyId: input.companyId, issueId: input.issueId, runId: input.runId } });
|
||||
|
||||
+199
-8
@@ -1,5 +1,5 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { chmod, cp, lstat, mkdir, mkdtemp, readFile, realpath, rename, rm, truncate, writeFile } from "node:fs/promises";
|
||||
import { chmod, cp, lstat, mkdir, mkdtemp, readFile, realpath, rename, rm, symlink, truncate, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { basename, join } from "node:path";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
@@ -43,7 +43,7 @@ describeEmbeddedPostgres("owner-verified native harness filesystem independence"
|
||||
afterEach(async () => { vi.restoreAllMocks(); vi.unstubAllEnvs(); await db.delete(agentTaskSessions); for (const root of roots.splice(0)) await removeRuntimeAssetTree(root); });
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); });
|
||||
|
||||
async function fixture(kind: "unversioned" | "repository" | "package" | "none" = "unversioned", managedWorkspace = false) {
|
||||
async function fixture(kind: "unversioned" | "repository" | "package" | "deleted" | "none" = "unversioned", managedWorkspace = false) {
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "61".repeat(32));
|
||||
const mount = await realpath(await mkdtemp(join(tmpdir(), "native-harness-original-"))); roots.push(mount);
|
||||
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), sessionId = randomUUID(), runnerId = randomUUID();
|
||||
@@ -60,9 +60,10 @@ describeEmbeddedPostgres("owner-verified native harness filesystem independence"
|
||||
mountIdentity: { dev: m.dev, ino: m.ino }, instanceIdentity: { dev: instance.dev, ino: instance.ino },
|
||||
createdFrom: { deploymentId: randomUUID(), commit: "c".repeat(40), providerWitnessSha256: "d".repeat(64) }, createdAt: new Date().toISOString() };
|
||||
const bytes = Buffer.from(JSON.stringify(anchor)); await writeFile(originalHarnessVolumeAnchorPath(), bytes, { mode: 0o600 });
|
||||
const observed = Date.now();
|
||||
const witness: ReviewedOriginalHarnessVolumeWitness = { projectId, serviceId, environmentId, volumeId, volumeInstanceId: anchor.volumeInstanceId,
|
||||
mountPath: mount, currentDeploymentId: deploymentId, currentCommit: "a".repeat(40), currentMountIdentity: anchor.mountIdentity,
|
||||
currentInstanceIdentity: anchor.instanceIdentity, observedAt: new Date().toISOString(), expiresAt: new Date(Date.now() + 120_000).toISOString(),
|
||||
currentInstanceIdentity: anchor.instanceIdentity, observedAt: new Date(observed).toISOString(), expiresAt: new Date(observed + 120_000).toISOString(),
|
||||
anchorSha256: createHash("sha256").update(bytes).digest("hex"), providerWitnessSha256: "e".repeat(64) };
|
||||
const source = await verifyOriginalHarnessVolumeSource(witness);
|
||||
const storage = createStorageService({ ...createLocalDiskStorageProvider(join(mount, "objects")), id: "s3" });
|
||||
@@ -72,14 +73,16 @@ describeEmbeddedPostgres("owner-verified native harness filesystem independence"
|
||||
const runtimeContext = nativeRuntimeContextFixture();
|
||||
if (kind !== "none") {
|
||||
let versionId: string | null = null;
|
||||
const files: CompanySkillVersionFileInventoryEntry[] = [{ path: "SKILL.md", kind: "skill", content: "# original immutable skill\n" },
|
||||
{ path: "tool.sh", kind: "other" as const, content: "#!/bin/sh\nexit 0\n", executable: true }];
|
||||
const files: CompanySkillVersionFileInventoryEntry[] = kind === "deleted"
|
||||
? [{ path: "SKILL.md", kind: "skill", content: "# Historical acceptance skill\n\n".padEnd(281, "x") }]
|
||||
: [{ path: "SKILL.md", kind: "skill", content: "# original immutable skill\n" },
|
||||
{ path: "tool.sh", kind: "other" as const, content: "#!/bin/sh\nexit 0\n", executable: true }];
|
||||
let projected = files;
|
||||
if (kind === "repository") {
|
||||
files[0]!.content = "---\nname: retained\ndescription: retained\n---\n# original immutable skill\n";
|
||||
projected = repositorySkillProjection(files, "SKILL.md");
|
||||
}
|
||||
if (kind === "repository" || kind === "package") {
|
||||
if (kind === "repository" || kind === "package" || kind === "deleted") {
|
||||
const packageId = await skillPackageStore(db, storage).prepare(companyId, files, true);
|
||||
const [skill] = await db.insert(companySkills).values({ companyId, key: "retained", slug: "retained", name: "Retained", markdown: projected[0]!.content }).returning();
|
||||
const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 1,
|
||||
@@ -125,8 +128,8 @@ describeEmbeddedPostgres("owner-verified native harness filesystem independence"
|
||||
return { input, runId, issueId, agentId, leaseId, execution, witness, storage, mount, sourceRoot: scopedRunnerdStateRoot(execution) };
|
||||
}
|
||||
|
||||
async function controllerFixture(managedWorkspace = false) {
|
||||
const f = await fixture("unversioned", managedWorkspace);
|
||||
async function controllerFixture(managedWorkspace = false, kind: Parameters<typeof fixture>[0] = "unversioned", checkpointParts?: number) {
|
||||
const f = await fixture(kind, managedWorkspace);
|
||||
const completion = await ensureNativeCompletionContract({ db, companyId: f.input.companyId,
|
||||
issue: { id: f.issueId, title: "Retain controller evidence", description: null }, actorId: f.agentId });
|
||||
const execution = parseNativeExecutionInput({ ...f.execution, session: { ...f.execution.session, lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 } },
|
||||
@@ -152,6 +155,13 @@ describeEmbeddedPostgres("owner-verified native harness filesystem independence"
|
||||
{ sourceSeq: 2, sourceEventId: "raw-terminal", eventType: "run.terminal", priority: 0,
|
||||
envelope: { ...identity }, deliveryCount: 1, logicalEffectCount: 1 });
|
||||
control.store.state.ackedSourceSeq = 2;
|
||||
if (checkpointParts) {
|
||||
const artifact = { version: "raw-evidence-fixture", digest: `sha256:${"a".repeat(64)}` };
|
||||
for (let count = 0; count < 512 && captureLegacyDurablePrpCheckpoint(JSON.stringify(control.store.state), identity, artifact).records.size < checkpointParts; count++) {
|
||||
Object.assign(control.queueCommand("runner.suspend"), { status: "completed", result: { status: "completed" } });
|
||||
}
|
||||
expect(captureLegacyDurablePrpCheckpoint(JSON.stringify(control.store.state), identity, artifact).records.size).toBe(checkpointParts);
|
||||
}
|
||||
const raw = JSON.stringify(control.store.state);
|
||||
control.setCheckpointPersistence(nativeRunnerCheckpointWriter(db, { companyId: f.input.companyId, issueId: f.issueId,
|
||||
runId: f.runId, authorityRunId: f.runId, bootId, generation: 1, leaseOwner }));
|
||||
@@ -174,6 +184,187 @@ describeEmbeddedPostgres("owner-verified native harness filesystem independence"
|
||||
return { ...f, execution, controllerPath, raw, supervisor, supervisorPath: descriptor.supervisorPath, identity };
|
||||
}
|
||||
|
||||
it("preserves WEB-only evidence with a succeeded source and unresolved external lease", async () => {
|
||||
const f = await controllerFixture(true, "deleted");
|
||||
if (!("runtimeContext" in f.execution)) throw new Error("fixture runtime context missing");
|
||||
const versionId = f.execution.runtimeContext.skills[0]!.versionId!;
|
||||
await db.delete(companySkillVersions).where(eq(companySkillVersions.id, versionId));
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
|
||||
const metadata = { ...lease!.metadata }; delete metadata.remoteExecutionTermination;
|
||||
await db.update(environmentLeases).set({ metadata }).where(eq(environmentLeases.id, f.leaseId));
|
||||
const beforeLeases = await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId));
|
||||
const beforeRuns = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId));
|
||||
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("unverified");
|
||||
const result = await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" });
|
||||
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
|
||||
expect(await hasLocalNativeHarnessObligations(db)).toBe(false);
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
const receipt = run!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { schema: string; controller: { archive: { id: string } } };
|
||||
expect(receipt.schema).toBe("paperclip.native-harness-web-evidence.v1");
|
||||
const [archive] = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, receipt.controller.archive.id));
|
||||
const target = join(f.mount, "isolated-web-evidence");
|
||||
expect(await restoreDurableNativeHarnessBackup({ db, storage: f.storage, companyId: f.input.companyId,
|
||||
scopeDigest: archive!.scopeDigest, backupRoot: target })).toBe(true);
|
||||
expect(await readFile(join(target, "current/control-plane-state.json"), "utf8")).toBe(f.raw);
|
||||
expect(await readFile(join(target, "current/supervisor.json"), "utf8")).toBe(f.supervisor);
|
||||
const original = join(f.sourceRoot, "codex-home/skills/retained/SKILL.md"), restored = join(target, "current/web-home/codex-home/skills/retained/SKILL.md");
|
||||
expect(await readFile(restored)).toEqual(await readFile(original));
|
||||
expect((await lstat(restored)).mode & 0o777).toBe(0o444);
|
||||
expect((await readFile(restored)).length).toBe(281);
|
||||
expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, versionId))).toEqual([]);
|
||||
expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.input.scopeDigest))).toEqual([]);
|
||||
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId))).toEqual(beforeLeases);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId))).toEqual(beforeRuns.map(r => ({ ...r,
|
||||
runnerProfileJson: { ...r.runnerProfileJson, nativeHarnessFilesystemIndependence: run!.runnerProfileJson!.nativeHarnessFilesystemIndependence } })));
|
||||
vi.stubEnv("PAPERCLIP_HOME", join(f.mount, "absent-web")); vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(f.mount, "absent-web/state"));
|
||||
vi.stubEnv("RAILWAY_VOLUME_ID", ""); vi.spyOn(f.storage, "getObject").mockRejectedValue(new Error("replacement must not read storage"));
|
||||
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
|
||||
expect(result.receiptDigest).toMatch(/^[a-f0-9]{64}$/);
|
||||
});
|
||||
|
||||
async function webFixture(checkpointParts?: number) {
|
||||
const f = await controllerFixture(true, "unversioned", checkpointParts);
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
|
||||
const metadata = { ...lease!.metadata }; delete metadata.remoteExecutionTermination;
|
||||
await db.update(environmentLeases).set({ status: "retained", cleanupStatus: "failed", releasedAt: null, metadata })
|
||||
.where(eq(environmentLeases.id, f.leaseId));
|
||||
return { ...f, input: { ...f.input, mode: "web_evidence" as const } };
|
||||
}
|
||||
|
||||
it("preserves a genuinely absent controller with its supervisor and 267 durable checkpoint parts", async () => {
|
||||
const f = await webFixture(267);
|
||||
await rm(f.controllerPath);
|
||||
const before = await db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId));
|
||||
expect(before).toHaveLength(267);
|
||||
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
const lastRunId = randomUUID(), execution = parseNativeExecutionInput({ ...f.execution, binding: { ...f.execution.binding, runId: lastRunId } });
|
||||
await db.insert(heartbeatRuns).values({ ...source!, id: lastRunId, status: "failed", processPid: null, processStartedAt: null,
|
||||
createdAt: new Date(source!.createdAt.getTime() + 1_000), runnerProfileJson: { nativeExecutionInput: execution, retainedWarning: "existing failed admission" } });
|
||||
await db.insert(nativeRunFinalizations).values({ companyId: f.input.companyId, issueId: f.issueId, runId: lastRunId,
|
||||
phase: "terminal_failure", recoveryState: "blocked" });
|
||||
await db.update(agentTaskSessions).set({ lastRunId }).where(eq(agentTaskSessions.id, f.input.selectedSessionId));
|
||||
await preserveNativeHarnessFilesystemIndependence(db, f.input);
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, lastRunId));
|
||||
const receipt = run!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { web: { controllerAbsent: boolean };
|
||||
controller: { archive: { id: string; controllerDigest: null } } };
|
||||
expect(receipt.web.controllerAbsent).toBe(true);
|
||||
expect(receipt.controller.archive.controllerDigest).toBeNull();
|
||||
const [archive] = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, receipt.controller.archive.id));
|
||||
const target = join(f.mount, "isolated-supervisor-only");
|
||||
expect(await restoreDurableNativeHarnessBackup({ db, storage: f.storage, companyId: f.input.companyId,
|
||||
scopeDigest: archive!.scopeDigest, backupRoot: target })).toBe(true);
|
||||
await expect(lstat(join(target, "current/control-plane-state.json"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await readFile(join(target, "current/supervisor.json"), "utf8")).toBe(f.supervisor);
|
||||
expect(await db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toEqual(before);
|
||||
vi.stubEnv("PAPERCLIP_HOME", join(f.mount, "absent-supervisor-host"));
|
||||
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(f.mount, "absent-supervisor-host/state")); vi.stubEnv("RAILWAY_VOLUME_ID", "");
|
||||
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps WEB coverage valid through external cleanup observations without changing recovery authority", async () => {
|
||||
const f = await webFixture();
|
||||
const result = await preserveNativeHarnessFilesystemIndependence(db, f.input);
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
|
||||
await db.update(environmentLeases).set({ status: "expired", cleanupStatus: "pending", expiresAt: new Date(),
|
||||
lastUsedAt: new Date(), releasedAt: new Date(), metadata: { ...lease!.metadata, sandboxState: "archived",
|
||||
pendingCleanupRetryAfterMs: Date.now(), pendingCleanupManualAttemptId: randomUUID(),
|
||||
pendingCleanupController: { attemptId: randomUUID(), countedAttempt: true }, remoteExecutionTermination: { observation: "not a stop receipt" } } })
|
||||
.where(eq(environmentLeases.id, f.leaseId));
|
||||
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
|
||||
expect((await preserveNativeHarnessFilesystemIndependence(db, f.input)).receiptDigest).toBe(result.receiptDigest);
|
||||
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
|
||||
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
|
||||
await db.update(environmentLeases).set({ providerLeaseId: "different-allocation" }).where(eq(environmentLeases.id, f.leaseId));
|
||||
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
|
||||
});
|
||||
|
||||
it.each(["active-web", "owned-finalizer", "physical-writer", "foreign-supervisor", "missing-manifest", "symlink", "foreign-controller"])(
|
||||
"refuses WEB evidence with %s and retains the original recovery state", async control => {
|
||||
const f = await webFixture(), where = eq(heartbeatRuns.id, f.runId);
|
||||
if (control === "active-web") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }).where(where);
|
||||
if (control === "owned-finalizer") await db.update(nativeRunFinalizations).set({ leaseOwner: "expired-but-still-owned", leaseExpiresAt: new Date(0) })
|
||||
.where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
if (control === "physical-writer") { const [run] = await db.select().from(heartbeatRuns).where(where);
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson, nativeWorkspaceFinalizationOwner: { token: "owned" } } }).where(where); }
|
||||
if (control === "foreign-supervisor") { const saved = JSON.parse(f.supervisor); saved.snapshot.identity.runId = randomUUID();
|
||||
await writeFile(f.supervisorPath, JSON.stringify(saved)); }
|
||||
if (control === "foreign-controller") { const saved = JSON.parse(f.raw); saved.identity.runId = randomUUID(); await writeFile(f.controllerPath, JSON.stringify(saved)); }
|
||||
if (control === "missing-manifest") { if (!("runtimeContext" in f.execution)) throw new Error("fixture runtime context missing");
|
||||
await rm(join(f.mount, "instances/default/runtime-context-assets/manifests", `${f.execution.runtimeContext.skills[0]!.bundle.digest}.json`)); }
|
||||
if (control === "symlink") { const file = join(f.sourceRoot, "codex-home/history.jsonl"); await symlink(f.controllerPath, file); }
|
||||
try {
|
||||
const before = await db.select().from(heartbeatRuns).where(where);
|
||||
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
|
||||
expect(await db.select().from(heartbeatRuns).where(where)).toEqual(before);
|
||||
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
|
||||
} finally { if (control === "symlink") await rm(join(f.sourceRoot, "codex-home/history.jsonl")); }
|
||||
});
|
||||
|
||||
it.each(["changed-byte", "changed-mode", "added-file", "controller-appeared", "changed-source"])(
|
||||
"does not publish WEB evidence after %s changes during encrypted upload", async control => {
|
||||
const f = await webFixture();
|
||||
const originalMode = (await lstat(f.supervisorPath)).mode & 0o777;
|
||||
if (control === "controller-appeared") await rm(f.controllerPath);
|
||||
const put = f.storage.putFile.bind(f.storage); let changed = false;
|
||||
vi.spyOn(f.storage, "putFile").mockImplementation(async (...args) => {
|
||||
const result = await put(...args);
|
||||
if (!changed) { changed = true;
|
||||
if (control === "changed-byte") await writeFile(f.supervisorPath, `${f.supervisor} `);
|
||||
if (control === "changed-mode") await chmod(f.supervisorPath, originalMode === 0o400 ? 0o600 : 0o400);
|
||||
if (control === "added-file") await writeFile(join(f.sourceRoot, "codex-home/history.jsonl"), "new uncovered bytes");
|
||||
if (control === "controller-appeared") await writeFile(f.controllerPath, f.raw);
|
||||
if (control === "changed-source") { const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson, retainedWarning: "new source binding" } }).where(eq(heartbeatRuns.id, f.runId)); }
|
||||
} return result;
|
||||
});
|
||||
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
|
||||
expect(changed).toBe(true);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson)
|
||||
.not.toHaveProperty("nativeHarnessFilesystemIndependence");
|
||||
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
|
||||
});
|
||||
|
||||
it.each([false, true])("retains the authenticated WEB publication winner after a late upload (invalid: %s)", async invalid => {
|
||||
const f = await webFixture();
|
||||
let release!: () => void, entered!: () => void, delay = true;
|
||||
const blocked = new Promise<void>(resolve => { release = resolve; }), ready = new Promise<void>(resolve => { entered = resolve; });
|
||||
const get = f.storage.getObject.bind(f.storage);
|
||||
vi.spyOn(f.storage, "getObject").mockImplementation(async (...args) => {
|
||||
if (delay) { delay = false; entered(); await blocked; } return get(...args);
|
||||
});
|
||||
const late = preserveNativeHarnessFilesystemIndependence(db, f.input);
|
||||
try {
|
||||
await ready;
|
||||
const winner = await preserveNativeHarnessFilesystemIndependence(db, f.input);
|
||||
let [published] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (invalid) {
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...published!.runnerProfileJson,
|
||||
nativeHarnessFilesystemIndependence: { ...(published!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>), assetsDigest: "0".repeat(64) } } })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
[published] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
}
|
||||
release();
|
||||
if (invalid) await expect(late).rejects.toThrow("unverified");
|
||||
else expect((await late).receiptDigest).toBe(winner.receiptDigest);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(published);
|
||||
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
|
||||
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
|
||||
} finally { release(); await late.catch(() => undefined); }
|
||||
});
|
||||
|
||||
it.each(["missing-entry", "changed-source"])("refuses existing WEB coverage after %s without replacing its receipt", async control => {
|
||||
const f = await webFixture(); await preserveNativeHarnessFilesystemIndependence(db, f.input);
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
const receipt = run!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown> & { web: { entries: Array<{ path: string }> } };
|
||||
const profile = control === "missing-entry" ? { ...run!.runnerProfileJson, nativeHarnessFilesystemIndependence: { ...receipt,
|
||||
web: { ...receipt.web, entries: receipt.web.entries.filter(entry => entry.path !== "web-supervisor.json") } } }
|
||||
: { ...run!.runnerProfileJson, sessionCheckpoint: { changed: true } };
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
|
||||
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
|
||||
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson).toEqual(profile);
|
||||
});
|
||||
|
||||
async function historyFixture(keepHistoricalHeaders = false, includeRunnerState = false) {
|
||||
const f = await controllerFixture(true);
|
||||
const get = f.storage.getObject.bind(f.storage);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { constants } from "node:fs";
|
||||
import type { Stats } from "node:fs";
|
||||
import { lstat, open, opendir, readdir, realpath } from "node:fs/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import { basename, join, posix } from "node:path";
|
||||
@@ -6,6 +7,7 @@ import { and, asc, eq, inArray, or, sql } from "drizzle-orm";
|
||||
import { agentTaskSessions, companySkillRepositorySnapshots, companySkillVersions, environmentLeases,
|
||||
heartbeatRuns, issues, nativeHarnessBackups, nativeRunnerCheckpoints, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import type { StorageService } from "../../storage/types.js";
|
||||
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
|
||||
import { adapterExecutionControls } from "../adapter-execution-control.js";
|
||||
import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
|
||||
import { hasRemoteTerminationReceipt, resolveRemoteTerminationLease } from "../remote-execution-termination.js";
|
||||
@@ -19,14 +21,15 @@ import { captureLegacyDurablePrpCheckpoint, inspectDurablePrpCheckpointHeader, p
|
||||
import { nativeSha256 } from "./canonical.js";
|
||||
import { readRuntimeSkillPreservation } from "./runtime-skill-preservation-reference.js";
|
||||
import { findNativeSessionResumeRun } from "./native-session-resume.js";
|
||||
import { resolveNativeHarnessPersistenceProfile, scopedRunnerdStateRoot } from "./native-session-executor.js";
|
||||
import { nativeControllerMigrationDescriptor, resolveNativeHarnessPersistenceProfile, scopedRunnerdStateRoot } from "./native-session-executor.js";
|
||||
import { readNativeHarnessBackupStamp } from "./native-harness-backup-stamp.js";
|
||||
import { tryLockNativeWorkspaceFinalization } from "./native-workspace-finalization-ownership.js";
|
||||
import { assertOriginalHarnessVolumeSource, type OriginalHarnessVolumeCapability } from "./native-harness-original-volume.js";
|
||||
import { isKnownNativeSessionRootEntry } from "./native-session-root-entries.js";
|
||||
import { readLegacyNativeControllerArchiveSource, readLegacyNativeControllerArchiveDigests } from "./native-controller-migration.js";
|
||||
import { archiveNativeControllerSourceEvidence, legacyNativeControllerArchive, nativeControllerEvidenceDescriptor,
|
||||
nativeControllerHistoryManifest, type NativeControllerHistoryEntry, type NativeControllerHistoryManifestEntry } from "./native-controller-archive.js";
|
||||
nativeControllerHistoryManifest, nativeControllerWebManifest, type NativeControllerWebEntry,
|
||||
type NativeControllerHistoryEntry, type NativeControllerHistoryManifestEntry } from "./native-controller-archive.js";
|
||||
import { nativeControllerArchiveCoordinatorDigest, nativeControllerArchiveProfileDigest, type SettledControllerArchiveOwner } from "./native-controller-archive-source.js";
|
||||
import { loadNativeRunnerCheckpoint, loadNativeRunnerCheckpointHeader } from "./native-runner-checkpoint.js";
|
||||
import { readNativeControllerRecovery } from "./native-controller-recovery.js";
|
||||
@@ -34,6 +37,7 @@ import { verifyDurableNativeHarnessBackup } from "./native-harness-backup-storag
|
||||
|
||||
const KEY = "nativeHarnessFilesystemIndependence";
|
||||
const SCHEMA = "paperclip.native-harness-filesystem-independence.v1";
|
||||
const WEB_SCHEMA = "paperclip.native-harness-web-evidence.v1";
|
||||
const terminal = ["succeeded", "failed", "cancelled", "interrupted", "timed_out"];
|
||||
const sha = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex");
|
||||
const digest = (value: unknown) => nativeSha256(JSON.parse(JSON.stringify(value)));
|
||||
@@ -44,17 +48,30 @@ type VersionBinding = { id: string; digest: string };
|
||||
type HistoryEpoch = { name: string; runId: string; identity: DurableRecoveryIdentity; identityDigest: string; headerDigest: string | null };
|
||||
type HistoryBinding = { entries: NativeControllerHistoryManifestEntry[]; epochs: HistoryEpoch[] };
|
||||
type CapturedHistory = { entries: NativeControllerHistoryEntry[]; binding: HistoryBinding; localDigest: string };
|
||||
type WebSkill = { runId: string; key: string; runtimeName: string; versionId: string | null; bundleDigest: string;
|
||||
manifestDigest: string; fileCount: number; totalBytes: number };
|
||||
type WebBinding = { entries: NativeControllerHistoryManifestEntry[]; skills: WebSkill[]; rootPresent: boolean;
|
||||
profileDigest: string; controllerAbsent: boolean; excluded: Array<{ path: string; type: "directory" | "file"; mode: number; size: number }> };
|
||||
type CapturedWeb = { entries: NativeControllerWebEntry[]; binding: WebBinding; localDigest: string };
|
||||
type ControllerBinding = { checkpointPartsDigest: string; history?: HistoryBinding; archive: null | { id: string; digest: string;
|
||||
controllerDigest: string; supervisorDigest: string | null } };
|
||||
type Receipt = { schema: typeof SCHEMA; selectedSessionId: string; lastRunId: string; sourceRunId: string; scopeDigest: string;
|
||||
controllerDigest: string | null; supervisorDigest: string | null } };
|
||||
type Receipt = { schema: typeof SCHEMA | typeof WEB_SCHEMA; selectedSessionId: string; lastRunId: string; sourceRunId: string; scopeDigest: string;
|
||||
factsDigest: string; sourceCapabilityFingerprint: string; localFactsDigest: string; assetsDigest: string;
|
||||
packages: PackageBinding[]; versions: VersionBinding[]; controller: ControllerBinding; observedAt: string };
|
||||
packages: PackageBinding[]; versions: VersionBinding[]; controller: ControllerBinding; observedAt: string; web?: WebBinding };
|
||||
type Scope = { companyId: string; selectedSessionId: string; scopeDigest: string };
|
||||
type Run = typeof heartbeatRuns.$inferSelect;
|
||||
const evidenceOwners = new WeakMap<SettledControllerArchiveOwner, { scope: Scope; factsDigest: string;
|
||||
history?: CapturedHistory; archiveManifestSha256: string; signal: AbortSignal }>();
|
||||
history?: CapturedHistory; web?: CapturedWeb; archiveManifestSha256: string; signal: AbortSignal }>();
|
||||
|
||||
async function databaseFacts(db: Db, input: Scope, lock: boolean) {
|
||||
function webLeaseBinding(lease: typeof environmentLeases.$inferSelect) {
|
||||
const fields = Object.fromEntries(Object.entries(lease).filter(([key]) => !["status", "releasedAt", "updatedAt", "lastUsedAt", "expiresAt", "failureReason", "cleanupStatus"].includes(key)));
|
||||
const metadata = Object.fromEntries(Object.entries(lease.metadata ?? {}).filter(([key]) => !["pendingCleanupAttemptId", "pendingCleanupInFlight",
|
||||
"pendingCleanupLeaseExpiresAtMs", "pendingCleanupRetryAfterMs", "pendingCleanupRetryAttempts", "pendingCleanupRetryCapWarned",
|
||||
"pendingCleanupController", "pendingCleanupManualAttemptId", "remoteExecutionTermination", "sandboxStopAndRetainReceipt", "sandboxState", "resumedFromState"].includes(key)));
|
||||
return { ...fields, metadata };
|
||||
}
|
||||
|
||||
async function databaseFacts(db: Db, input: Scope, lock: boolean, webOnly = false) {
|
||||
const [selection] = await db.select().from(agentTaskSessions).where(and(eq(agentTaskSessions.companyId, input.companyId),
|
||||
eq(agentTaskSessions.id, input.selectedSessionId), eq(agentTaskSessions.adapterType, "paperclip_runner")));
|
||||
if (!selection?.lastRunId || !/^[a-f0-9]{64}$/.test(input.scopeDigest)) fail();
|
||||
@@ -113,10 +130,12 @@ async function databaseFacts(db: Db, input: Scope, lock: boolean) {
|
||||
let matchingStamp = false;
|
||||
const terminations = [];
|
||||
for (const lease of leases) {
|
||||
const resolved = await resolveRemoteTerminationLease(db, lease);
|
||||
const receipt = resolved.metadata?.remoteExecutionTermination as { state?: unknown } | undefined;
|
||||
if (!hasRemoteTerminationReceipt(resolved) || receipt?.state !== "destroyed") fail();
|
||||
terminations.push({ leaseId: lease.id, receipt });
|
||||
if (!webOnly) {
|
||||
const resolved = await resolveRemoteTerminationLease(db, lease);
|
||||
const receipt = resolved.metadata?.remoteExecutionTermination as { state?: unknown } | undefined;
|
||||
if (!hasRemoteTerminationReceipt(resolved) || receipt?.state !== "destroyed") fail();
|
||||
terminations.push({ leaseId: lease.id, receipt });
|
||||
}
|
||||
const stamp = readNativeHarnessBackupStamp(lease.metadata?.nativeHarnessBackup, lease.providerLeaseId ?? "");
|
||||
if (stamp?.sessionScopeSha256 === input.scopeDigest && stamp.normalizedSessionId === anchor.nativeSessionId
|
||||
&& runs.some(run => run.runnerInstanceId === stamp.runnerInstanceId)
|
||||
@@ -124,7 +143,8 @@ async function databaseFacts(db: Db, input: Scope, lock: boolean) {
|
||||
}
|
||||
if (!matchingStamp) fail();
|
||||
const factsDigest = digest({ selection, runs: runs.map(run => ({ ...run, runnerProfileJson: profile(run.runnerProfileJson) })), owners,
|
||||
task: { id: task.id, executionRunId: task.executionRunId, checkoutRunId: task.checkoutRunId }, leases, terminations,
|
||||
task: { id: task.id, executionRunId: task.executionRunId, checkoutRunId: task.checkoutRunId },
|
||||
leases: webOnly ? leases.map(webLeaseBinding) : leases, terminations,
|
||||
sourceRunId: source.id, sourceProfile: source.runnerProfileJson });
|
||||
return { anchor, execution, source, factsDigest, runs };
|
||||
}
|
||||
@@ -169,35 +189,46 @@ async function assetBindings(db: Db, anchor: Run, sourceRunId: string) {
|
||||
}
|
||||
|
||||
function matchingReceipt(value: unknown, facts: Awaited<ReturnType<typeof databaseFacts>>, input: Scope,
|
||||
bindings: Awaited<ReturnType<typeof assetBindings>>["bindings"], controller: ControllerBinding): Receipt | null {
|
||||
bindings: Awaited<ReturnType<typeof assetBindings>>["bindings"], controller: ControllerBinding, web?: WebBinding): Receipt | null {
|
||||
const receipt = value as Receipt | undefined;
|
||||
if (receipt?.schema !== SCHEMA || receipt.selectedSessionId !== input.selectedSessionId || receipt.scopeDigest !== input.scopeDigest
|
||||
if (receipt?.schema !== (web ? WEB_SCHEMA : SCHEMA) || receipt.selectedSessionId !== input.selectedSessionId || receipt.scopeDigest !== input.scopeDigest
|
||||
|| receipt.lastRunId !== facts.anchor.id || receipt.sourceRunId !== facts.source.id || receipt.factsDigest !== facts.factsDigest
|
||||
|| !/^[a-f0-9]{64}$/.test(receipt.sourceCapabilityFingerprint) || !/^[a-f0-9]{64}$/.test(receipt.localFactsDigest)
|
||||
|| !Number.isFinite(Date.parse(receipt.observedAt)) || !Array.isArray(receipt.packages) || !Array.isArray(receipt.versions)) return null;
|
||||
return digest(bindings) === receipt.assetsDigest && digest(receipt.packages) === digest(bindings.packages)
|
||||
&& digest(receipt.versions) === digest(bindings.versions) && receipt.controller != null
|
||||
&& digest(receipt.controller) === digest(controller) ? receipt : null;
|
||||
&& digest(receipt.controller) === digest(controller) && digest(receipt.web ?? null) === digest(web ?? null) ? receipt : null;
|
||||
}
|
||||
|
||||
export async function hasValidNativeHarnessFilesystemIndependence(db: Db, input: Scope): Promise<boolean> {
|
||||
try {
|
||||
const facts = await databaseFacts(db, input, false);
|
||||
const assets = await assetBindings(db, facts.anchor, facts.source.id);
|
||||
const [anchor] = await db.select({ profile: heartbeatRuns.runnerProfileJson }).from(agentTaskSessions)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, agentTaskSessions.lastRunId), eq(heartbeatRuns.companyId, agentTaskSessions.companyId),
|
||||
eq(heartbeatRuns.agentId, agentTaskSessions.agentId)))
|
||||
.where(and(eq(agentTaskSessions.id, input.selectedSessionId), eq(agentTaskSessions.companyId, input.companyId)));
|
||||
const value = anchor?.profile?.[KEY] as Receipt | undefined;
|
||||
const webOnly = value?.schema === WEB_SCHEMA;
|
||||
const facts = await databaseFacts(db, input, false, webOnly);
|
||||
const existing = facts.anchor.runnerProfileJson?.[KEY] as Receipt | undefined;
|
||||
const controller = await controllerBinding(db, facts, existing?.controller?.archive ?? null, false, existing?.controller?.history);
|
||||
return matchingReceipt(existing, facts, input, assets.bindings, controller) !== null;
|
||||
if (webOnly && (!existing?.web || !validWebBinding(facts, existing.web))) return false;
|
||||
const bindings = webOnly ? { packages: [], versions: [] } : (await assetBindings(db, facts.anchor, facts.source.id)).bindings;
|
||||
const controller = await controllerBinding(db, facts, existing?.controller?.archive ?? null, false, existing?.controller?.history, existing?.web);
|
||||
return matchingReceipt(existing, facts, input, bindings, controller, existing?.web) !== null;
|
||||
} catch { return false; }
|
||||
}
|
||||
|
||||
export async function assertNativeHarnessFilesystemArchiveSource(db: Db, owner: SettledControllerArchiveOwner) {
|
||||
const registered = evidenceOwners.get(owner);
|
||||
if (!registered || !owner.filesystemIndependence) fail();
|
||||
const facts = await databaseFacts(db, registered.scope, true);
|
||||
const facts = await databaseFacts(db, registered.scope, true, registered.web !== undefined);
|
||||
if (facts.factsDigest !== registered.factsDigest || facts.factsDigest !== owner.filesystemIndependence.factsDigest
|
||||
|| facts.source.id !== owner.runId || facts.anchor.id !== owner.selectedLastRunId) fail();
|
||||
const history = await captureAuthorityHistory(db, facts, registered.signal);
|
||||
if (history?.localDigest !== registered.history?.localDigest || digest(history?.binding ?? null) !== digest(registered.history?.binding ?? null)) fail();
|
||||
if (registered.web) {
|
||||
const web = await captureWebEvidence(facts, registered.signal);
|
||||
if (web.localDigest !== registered.web.localDigest || digest(web.binding) !== digest(registered.web.binding)) fail();
|
||||
}
|
||||
}
|
||||
|
||||
export function assertNativeHarnessFilesystemArchiveContents(owner: SettledControllerArchiveOwner, archive: { manifestSha256: string }) {
|
||||
@@ -313,12 +344,15 @@ async function captureAuthorityHistory(db: Db, facts: Awaited<ReturnType<typeof
|
||||
}
|
||||
|
||||
async function controllerBinding(db: Db, facts: Awaited<ReturnType<typeof databaseFacts>>, archive: ControllerBinding["archive"], lock: boolean,
|
||||
history?: HistoryBinding): Promise<ControllerBinding> {
|
||||
history?: HistoryBinding, web?: WebBinding): Promise<ControllerBinding> {
|
||||
const partsQuery = db.select().from(nativeRunnerCheckpoints).where(and(eq(nativeRunnerCheckpoints.companyId, facts.anchor.companyId),
|
||||
eq(nativeRunnerCheckpoints.issueId, facts.anchor.nativeIssueId!), eq(nativeRunnerCheckpoints.runId, facts.source.id)))
|
||||
.orderBy(asc(nativeRunnerCheckpoints.part));
|
||||
const parts = await (lock ? partsQuery.for("share") : partsQuery);
|
||||
if (archive) {
|
||||
if (web && (!validWebBinding(facts, web) || web.controllerAbsent !== (archive.controllerDigest === null))) fail();
|
||||
if (web && (web.entries.find(entry => entry.path === "web-supervisor.json" && entry.type === "file") as
|
||||
Extract<NativeControllerHistoryManifestEntry, { type: "file" }> | undefined)?.sha256 !== (archive.supervisorDigest ?? undefined)) fail();
|
||||
if (!parts.some(row => row.part === "header") || !readNativeControllerRecovery(
|
||||
facts.source.runnerProfileJson.nativeControllerRecovery, facts.source.runnerProfileJson.sessionCheckpoint)) fail();
|
||||
const rowQuery = db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.companyId, facts.anchor.companyId),
|
||||
@@ -339,9 +373,10 @@ async function controllerBinding(db: Db, facts: Awaited<ReturnType<typeof databa
|
||||
}
|
||||
const descriptor = nativeControllerEvidenceDescriptor({ companyId: facts.anchor.companyId, issueId: facts.anchor.nativeIssueId!,
|
||||
runId: facts.source.id, controllerDigest: archive.controllerDigest, supervisorDigest: archive.supervisorDigest,
|
||||
purpose: "filesystem_independence", history: history?.entries, historyBindingDigest: history ? digest(history) : undefined });
|
||||
purpose: "filesystem_independence", history: history?.entries, historyBindingDigest: history ? digest(history) : undefined,
|
||||
...(web ? { web: { entries: web.entries, bindingDigest: digest(web) } } : {}) });
|
||||
if (row.scopeDigest !== descriptor.scopeDigest || row.manifestSha256 !== descriptor.manifestSha256) fail();
|
||||
} else if (history) {
|
||||
} else if (history || web) {
|
||||
fail();
|
||||
}
|
||||
return { checkpointPartsDigest: digest(parts), archive, ...(history ? { history } : {}) };
|
||||
@@ -349,25 +384,33 @@ async function controllerBinding(db: Db, facts: Awaited<ReturnType<typeof databa
|
||||
|
||||
async function preserveControllerEvidence(db: Db, facts: Awaited<ReturnType<typeof databaseFacts>>, input: Scope & {
|
||||
source: OriginalHarnessVolumeCapability; storage: StorageService; signal: AbortSignal;
|
||||
}, history?: CapturedHistory): Promise<ControllerBinding> {
|
||||
}, history?: CapturedHistory, web?: CapturedWeb): Promise<ControllerBinding> {
|
||||
const material = await readLegacyNativeControllerArchiveSource(facts.source);
|
||||
if (material.paths.root !== scopedRunnerdStateRoot(facts.execution)) fail();
|
||||
if (material.controller === undefined) {
|
||||
if (material.controller === undefined && !web) {
|
||||
if (material.supervisor !== undefined || history) fail();
|
||||
return controllerBinding(db, facts, null, false);
|
||||
}
|
||||
const identity = JSON.parse(material.controller).identity as Record<string, unknown> | undefined;
|
||||
if (identity?.runId !== facts.source.id || identity.runnerInstanceId !== facts.source.runnerInstanceId
|
||||
|| identity.normalizedSessionId !== facts.anchor.nativeSessionId) fail();
|
||||
const checkpoint = await loadNativeRunnerCheckpoint(db, { companyId: input.companyId, issueId: facts.anchor.nativeIssueId!, runId: facts.source.id });
|
||||
if (!checkpoint || !readNativeControllerRecovery(facts.source.runnerProfileJson.nativeControllerRecovery,
|
||||
facts.source.runnerProfileJson.sessionCheckpoint)) fail();
|
||||
const captured = captureLegacyDurablePrpCheckpoint(material.controller, { runId: facts.source.id,
|
||||
runnerInstanceId: facts.source.runnerInstanceId!, normalizedSessionId: facts.anchor.nativeSessionId! },
|
||||
{ version: checkpoint.runnerVersion, digest: checkpoint.runnerDigest });
|
||||
if (captured.records.size !== checkpoint.records.size || [...captured.records].some(([key, value]) => {
|
||||
const durable = checkpoint.records.get(key); return durable === undefined || digest(JSON.parse(value)) !== digest(JSON.parse(durable));
|
||||
})) fail();
|
||||
if (web && material.supervisor !== undefined) {
|
||||
const saved = JSON.parse(material.supervisor), identity = saved.snapshot?.identity;
|
||||
if (saved.schema !== "paperclip.native-session-supervisor.v1" || identity?.companyId !== facts.anchor.companyId
|
||||
|| identity.agentId !== facts.anchor.agentId || identity.issueId !== facts.anchor.nativeIssueId
|
||||
|| identity.sessionId !== facts.anchor.nativeSessionId || !facts.runs.some(run => run.id === identity.runId)) fail();
|
||||
}
|
||||
if (material.controller !== undefined) {
|
||||
const identity = JSON.parse(material.controller).identity as Record<string, unknown> | undefined;
|
||||
if (identity?.runId !== facts.source.id || identity.runnerInstanceId !== facts.source.runnerInstanceId
|
||||
|| identity.normalizedSessionId !== facts.anchor.nativeSessionId) fail();
|
||||
const captured = captureLegacyDurablePrpCheckpoint(material.controller, { runId: facts.source.id,
|
||||
runnerInstanceId: facts.source.runnerInstanceId!, normalizedSessionId: facts.anchor.nativeSessionId! },
|
||||
{ version: checkpoint.runnerVersion, digest: checkpoint.runnerDigest });
|
||||
if (captured.records.size !== checkpoint.records.size || [...captured.records].some(([key, value]) => {
|
||||
const durable = checkpoint.records.get(key); return durable === undefined || digest(JSON.parse(value)) !== digest(JSON.parse(durable));
|
||||
})) fail();
|
||||
}
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId),
|
||||
eq(nativeRunFinalizations.runId, facts.source.id)));
|
||||
if (!coordinator) fail();
|
||||
@@ -375,14 +418,15 @@ async function preserveControllerEvidence(db: Db, facts: Awaited<ReturnType<type
|
||||
issueId: facts.anchor.nativeIssueId!, agentId: facts.anchor.agentId, runId: facts.source.id, sessionId: facts.anchor.nativeSessionId!,
|
||||
selectedSessionId: input.selectedSessionId, selectedLastRunId: facts.anchor.id,
|
||||
profileDigest: nativeControllerArchiveProfileDigest(facts.source.runnerProfileJson), coordinatorDigest: nativeControllerArchiveCoordinatorDigest(coordinator),
|
||||
controllerDigest: sha(material.controller), supervisorDigest: material.supervisor === undefined ? null : sha(material.supervisor),
|
||||
controllerDigest: material.controller === undefined ? null : sha(material.controller), supervisorDigest: material.supervisor === undefined ? null : sha(material.supervisor),
|
||||
filesystemIndependence: Object.freeze({ source: input.source, scopeDigest: input.scopeDigest, factsDigest: facts.factsDigest }) });
|
||||
const source = { companyId: input.companyId, issueId: facts.anchor.nativeIssueId!, runId: facts.source.id,
|
||||
controller: material.controller, supervisor: material.supervisor, purpose: "filesystem_independence" as const,
|
||||
...(history ? { history: history.entries, historyBindingDigest: digest(history.binding) } : {}) };
|
||||
...(history ? { history: history.entries, historyBindingDigest: digest(history.binding) } : {}),
|
||||
...(web ? { web: { entries: web.entries, bindingDigest: digest(web.binding) } } : {}) };
|
||||
const descriptor = legacyNativeControllerArchive(source);
|
||||
evidenceOwners.set(owner, { scope: { companyId: input.companyId, selectedSessionId: input.selectedSessionId, scopeDigest: input.scopeDigest },
|
||||
factsDigest: facts.factsDigest, history, archiveManifestSha256: descriptor.manifestSha256, signal: input.signal });
|
||||
factsDigest: facts.factsDigest, history, web, archiveManifestSha256: descriptor.manifestSha256, signal: input.signal });
|
||||
await archiveNativeControllerSourceEvidence(db, { ...source, onlineOwner: owner, storage: input.storage, signal: input.signal });
|
||||
const [row] = await db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.companyId, input.companyId),
|
||||
eq(nativeHarnessBackups.runId, facts.source.id), eq(nativeHarnessBackups.scopeDigest, descriptor.scopeDigest),
|
||||
@@ -391,7 +435,7 @@ async function preserveControllerEvidence(db: Db, facts: Awaited<ReturnType<type
|
||||
if (!await verifyDurableNativeHarnessBackup({ db, storage: input.storage, companyId: input.companyId,
|
||||
scopeDigest: descriptor.scopeDigest, manifestSha256: descriptor.manifestSha256, backupId: row.id, signal: input.signal })) fail();
|
||||
return controllerBinding(db, facts, { id: row.id, digest: digest(row), controllerDigest: owner.controllerDigest,
|
||||
supervisorDigest: owner.supervisorDigest }, false, history?.binding);
|
||||
supervisorDigest: owner.supervisorDigest }, false, history?.binding, web?.binding);
|
||||
}
|
||||
|
||||
function runtimeFiles(skill: { bundle: { digest: string; manifestDigest: string; fileCount: number; totalBytes: number } },
|
||||
@@ -414,6 +458,146 @@ function runtimeFiles(skill: { bundle: { digest: string; manifestDigest: string;
|
||||
return files;
|
||||
}
|
||||
|
||||
function webSkills(facts: Awaited<ReturnType<typeof databaseFacts>>): WebSkill[] {
|
||||
const rows = facts.source.id === facts.anchor.id ? [facts.anchor] : [facts.anchor, facts.source];
|
||||
return rows.flatMap(row => {
|
||||
const input = parseNativeExecutionInput(row.runnerProfileJson?.nativeExecutionInput);
|
||||
if (!("runtimeContext" in input) || input.binding.companyId !== row.companyId || input.binding.agentId !== row.agentId
|
||||
|| input.binding.issueId !== row.nativeIssueId || input.binding.runId !== row.id
|
||||
|| input.session.normalizedSessionId !== row.nativeSessionId) fail();
|
||||
return input.runtimeContext.skills.map(skill => ({ runId: row.id, key: skill.key, runtimeName: skill.runtimeName,
|
||||
versionId: skill.versionId, bundleDigest: skill.bundle.digest, manifestDigest: skill.bundle.manifestDigest,
|
||||
fileCount: skill.bundle.fileCount, totalBytes: skill.bundle.totalBytes }));
|
||||
}).sort((a, b) => `${a.runId}:${a.key}:${a.runtimeName}`.localeCompare(`${b.runId}:${b.key}:${b.runtimeName}`));
|
||||
}
|
||||
|
||||
function webProfile(facts: Awaited<ReturnType<typeof databaseFacts>>) {
|
||||
return resolveNativeHarnessPersistenceProfile(facts.execution).directories
|
||||
.map(directory => ({ name: directory.name, location: directory.location, excludeEntries: directory.excludeEntries }));
|
||||
}
|
||||
|
||||
function validWebBinding(facts: Awaited<ReturnType<typeof databaseFacts>>, web: WebBinding): boolean {
|
||||
try {
|
||||
if (!web || typeof web.rootPresent !== "boolean" || typeof web.controllerAbsent !== "boolean" || !Array.isArray(web.entries)
|
||||
|| !Array.isArray(web.skills) || !Array.isArray(web.excluded) || web.entries.length > 128 || web.excluded.length > 128
|
||||
|| web.profileDigest !== digest(webProfile(facts)) || digest(web.skills) !== digest(webSkills(facts))) return false;
|
||||
const profiles = webProfile(facts), manifests = new Map(web.skills.map(skill => [skill.bundleDigest, skill.manifestDigest]));
|
||||
for (const [bundle, manifest] of manifests) if (!web.entries.some(entry => entry.path === `web-manifests/${bundle}.json`
|
||||
&& entry.type === "file" && entry.sha256 === manifest)) return false;
|
||||
if (web.entries.some(entry => {
|
||||
const [prefix, name, ...rest] = entry.path.split("/");
|
||||
if (prefix === "web-supervisor.json") return name !== undefined || entry.type !== "file";
|
||||
if (prefix === "web-manifests") return name === undefined ? entry.type !== "directory"
|
||||
: !/^[a-f0-9]{64}\.json$/.test(name) || rest.length > 0 || entry.type !== "file" || manifests.get(name.slice(0, -5)) !== entry.sha256;
|
||||
if (!["web-home", "web-state"].includes(prefix!)) return true;
|
||||
if (!web.rootPresent && name !== undefined) return true;
|
||||
if (name === undefined) return entry.type !== "directory";
|
||||
if (prefix === "web-state") return !isKnownNativeSessionRootEntry(name) || profiles.some(profile => profile.name === name);
|
||||
const profile = profiles.find(profile => profile.name === name), rel = rest.join("/");
|
||||
return !profile || profile.excludeEntries.some(excluded => rel === excluded || rel.startsWith(`${excluded}/`));
|
||||
})) return false;
|
||||
return web.excluded.every(entry => {
|
||||
const [name, ...rest] = entry.path.split("/"), rel = rest.join("/"), profile = profiles.find(profile => profile.name === name);
|
||||
assertSkillSnapshotPath(entry.path);
|
||||
return profile?.excludeEntries.some(excluded => rel === excluded || rel.startsWith(`${excluded}/`))
|
||||
&& ["directory", "file"].includes(entry.type) && Number.isInteger(entry.mode) && (entry.mode & ~0o777) === 0
|
||||
&& Number.isSafeInteger(entry.size) && entry.size >= 0;
|
||||
});
|
||||
} catch { return false; }
|
||||
}
|
||||
|
||||
async function captureWebEvidence(facts: Awaited<ReturnType<typeof databaseFacts>>, signal: AbortSignal): Promise<CapturedWeb> {
|
||||
const root = scopedRunnerdStateRoot(facts.execution), profiles = webProfile(facts), entries: NativeControllerWebEntry[] = [];
|
||||
const metadata: unknown[] = [], excluded: WebBinding["excluded"] = [], inodes = new Set<string>();
|
||||
let bytes = 0, count = 0;
|
||||
const directoryNames = async (path: string) => {
|
||||
const names: string[] = [];
|
||||
for await (const entry of await opendir(path)) {
|
||||
signal.throwIfAborted(); if (names.length >= 128) fail(); names.push(entry.name);
|
||||
}
|
||||
return names.sort();
|
||||
};
|
||||
const stat = (path: string) => lstat(path).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw error; });
|
||||
const rootStat = await stat(root), instance = await lstat(resolvePaperclipInstanceRoot());
|
||||
const read = async (file: string, s: Stats) => {
|
||||
const key = `${s.dev}:${s.ino}`;
|
||||
if (!s.isFile() || s.nlink !== 1 || inodes.has(key) || s.dev !== instance.dev || s.uid !== instance.uid
|
||||
|| (s.mode & 0o7000) !== 0 || await realpath(file) !== file
|
||||
|| s.size > 16 * 1024 * 1024 || (bytes += s.size) > 64 * 1024 * 1024) fail();
|
||||
inodes.add(key);
|
||||
const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
|
||||
try {
|
||||
const opened = await handle.stat();
|
||||
if (!opened.isFile() || opened.dev !== s.dev || opened.ino !== s.ino || opened.size !== s.size) fail();
|
||||
const buffer = Buffer.alloc(s.size + 1); let length = 0;
|
||||
while (length < buffer.length) { signal.throwIfAborted(); const result = await handle.read(buffer, length, buffer.length - length, length);
|
||||
if (!result.bytesRead) break; length += result.bytesRead; }
|
||||
const after = await handle.stat(), named = await lstat(file);
|
||||
if (length !== s.size || after.size !== s.size || after.mtimeMs !== s.mtimeMs || after.ctimeMs !== s.ctimeMs
|
||||
|| named.dev !== s.dev || named.ino !== s.ino) fail();
|
||||
metadata.push([file, s.dev, s.ino, s.size, s.mode, s.mtimeMs, s.ctimeMs]);
|
||||
return buffer.subarray(0, length);
|
||||
} finally { await handle.close(); }
|
||||
};
|
||||
const visit = async (file: string, path: string, relative: string): Promise<void> => {
|
||||
signal.throwIfAborted(); if (++count > 128) fail(); assertSkillSnapshotPath(path);
|
||||
const s = await lstat(file);
|
||||
if (s.isSymbolicLink() || !s.isDirectory() && !s.isFile() || s.dev !== instance.dev || s.uid !== instance.uid || (s.mode & 0o7000) !== 0) fail();
|
||||
metadata.push([path, s.dev, s.ino, s.size, s.mode, s.mtimeMs, s.ctimeMs]);
|
||||
const [name, ...rest] = relative.split("/"), rel = rest.join("/"), profile = profiles.find(profile => profile.name === name);
|
||||
if (profile?.excludeEntries.some(exclusion => rel === exclusion || rel.startsWith(`${exclusion}/`))) {
|
||||
excluded.push({ path: relative, type: s.isDirectory() ? "directory" : "file", mode: s.mode & 0o777, size: s.isFile() ? s.size : 0 }); return;
|
||||
}
|
||||
if (s.isFile()) { entries.push({ path, type: "file", mode: s.mode & 0o777, content: await read(file, s) }); return; }
|
||||
entries.push({ path, type: "directory", mode: s.mode & 0o777 });
|
||||
const names = await directoryNames(file);
|
||||
for (const name of names) await visit(join(file, name), `${path}/${name}`, `${relative}/${name}`);
|
||||
const after = await lstat(file);
|
||||
if (after.dev !== s.dev || after.ino !== s.ino || after.ctimeMs !== s.ctimeMs || digest(await directoryNames(file)) !== digest(names)) fail();
|
||||
};
|
||||
if (rootStat) {
|
||||
if (!rootStat.isDirectory() || rootStat.isSymbolicLink() || await realpath(root) !== root || rootStat.dev !== instance.dev || rootStat.uid !== instance.uid) fail();
|
||||
metadata.push([root, rootStat.dev, rootStat.ino, rootStat.mode, rootStat.mtimeMs, rootStat.ctimeMs]);
|
||||
const names = await directoryNames(root);
|
||||
if (names.some(name => !isKnownNativeSessionRootEntry(name))) fail();
|
||||
for (const name of ["continuity-breaks", "provider-artifacts", "failover-backups"]) {
|
||||
const s = await stat(join(root, name));
|
||||
if (s && (!s.isDirectory() || s.isSymbolicLink() || (await directoryNames(join(root, name))).length)) fail();
|
||||
}
|
||||
entries.push({ path: "web-home", type: "directory", mode: rootStat.mode & 0o777 },
|
||||
{ path: "web-state", type: "directory", mode: rootStat.mode & 0o777 });
|
||||
for (const name of names) {
|
||||
if (["runner", "codex-home", "opencode", "acpx"].includes(name) && !profiles.some(profile => profile.name === name)) fail();
|
||||
await visit(join(root, name), `${profiles.some(profile => profile.name === name) ? "web-home" : "web-state"}/${name}`, name);
|
||||
}
|
||||
if (digest(await directoryNames(root)) !== digest(names)) fail();
|
||||
}
|
||||
const sourceExecution = parseNativeExecutionInput(facts.source.runnerProfileJson.nativeExecutionInput);
|
||||
const supervisor = nativeControllerMigrationDescriptor(sourceExecution, facts.source.runnerProfileJson.nativeToolContractFingerprint).supervisorPath;
|
||||
const supervisorStat = sourceExecution.session.lifecyclePolicy.mode === "warm" ? await stat(supervisor) : null;
|
||||
if (supervisorStat) {
|
||||
const content = await read(supervisor, supervisorStat);
|
||||
entries.push({ path: "web-supervisor.json", type: "file", mode: supervisorStat.mode & 0o777, content });
|
||||
} else metadata.push(["web-supervisor.json", null]);
|
||||
const skills = webSkills(facts);
|
||||
if (skills.length) entries.push({ path: "web-manifests", type: "directory", mode: 0o700 });
|
||||
const seen = new Set<string>();
|
||||
for (const skill of skills) {
|
||||
if (seen.has(skill.bundleDigest)) continue; seen.add(skill.bundleDigest);
|
||||
if (!/^[a-f0-9]{64}$/.test(skill.bundleDigest)) fail();
|
||||
const path = join(resolvePaperclipInstanceRoot(), "runtime-context-assets/manifests", `${skill.bundleDigest}.json`), s = await lstat(path);
|
||||
const content = await read(path, s);
|
||||
runtimeFiles({ bundle: { digest: skill.bundleDigest, manifestDigest: skill.manifestDigest, fileCount: skill.fileCount, totalBytes: skill.totalBytes } }, content.toString("utf8"));
|
||||
entries.push({ path: `web-manifests/${skill.bundleDigest}.json`, type: "file", mode: s.mode & 0o777, content });
|
||||
}
|
||||
if (entries.length > 128) fail();
|
||||
const material = await readLegacyNativeControllerArchiveDigests(facts.source);
|
||||
const binding: WebBinding = { entries: nativeControllerWebManifest(entries), skills, rootPresent: rootStat !== null,
|
||||
profileDigest: digest(profiles), controllerAbsent: material.controllerDigest === undefined, excluded };
|
||||
if (!validWebBinding(facts, binding)) fail();
|
||||
return { entries, binding, localDigest: digest(metadata) };
|
||||
}
|
||||
|
||||
async function localFacts(facts: Awaited<ReturnType<typeof databaseFacts>>, assets: Awaited<ReturnType<typeof assetBindings>>,
|
||||
storage: StorageService, db: Db, verifyStorage: boolean, signal: AbortSignal, controller: ControllerBinding, plans = new Map<string, FileManifestEntry[]>()) {
|
||||
const root = scopedRunnerdStateRoot(facts.execution), metadata: unknown[] = [], store = skillPackageStore(db, storage);
|
||||
@@ -516,11 +700,55 @@ async function localFacts(facts: Awaited<ReturnType<typeof databaseFacts>>, asse
|
||||
return { digest: digest(metadata), plans };
|
||||
}
|
||||
|
||||
export async function preserveNativeHarnessFilesystemIndependence(db: Db, input: Scope & {
|
||||
async function preserveWebEvidence(db: Db, input: Scope & {
|
||||
source: OriginalHarnessVolumeCapability; storage: StorageService; signal: AbortSignal;
|
||||
}) {
|
||||
const facts = await databaseFacts(db, input, false, true), history = await captureAuthorityHistory(db, facts, input.signal);
|
||||
const web = await captureWebEvidence(facts, input.signal);
|
||||
const controller = await preserveControllerEvidence(db, facts, input, history, web), bindings = { packages: [], versions: [] };
|
||||
const previous = facts.anchor.runnerProfileJson?.[KEY];
|
||||
if (previous != null) {
|
||||
if (!await hasValidNativeHarnessFilesystemIndependence(db, input)) fail();
|
||||
return { scopeDigest: input.scopeDigest, sourceRunId: facts.source.id, receiptDigest: digest(previous) };
|
||||
}
|
||||
const receipt: Receipt = { schema: WEB_SCHEMA, selectedSessionId: input.selectedSessionId, scopeDigest: input.scopeDigest,
|
||||
lastRunId: facts.anchor.id, sourceRunId: facts.source.id, factsDigest: facts.factsDigest,
|
||||
sourceCapabilityFingerprint: input.source.fingerprint, localFactsDigest: web.localDigest,
|
||||
assetsDigest: digest(bindings), ...bindings, controller, web: web.binding, observedAt: new Date().toISOString() };
|
||||
let published = receipt;
|
||||
const publications: ActivityPublication[] = [];
|
||||
await db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
await tx.execute(sql`set local lock_timeout = '1s'`);
|
||||
const current = await databaseFacts(tx, input, true, true);
|
||||
if (current.factsDigest !== facts.factsDigest) fail();
|
||||
const currentController = await controllerBinding(tx, current, controller.archive, true, controller.history, web.binding);
|
||||
const currentWeb = await captureWebEvidence(current, input.signal);
|
||||
if (digest(currentController) !== digest(controller) || currentWeb.localDigest !== web.localDigest
|
||||
|| digest(currentWeb.binding) !== digest(web.binding)) fail();
|
||||
await assertOriginalHarnessVolumeSource(input.source); input.signal.throwIfAborted();
|
||||
const existing = current.anchor.runnerProfileJson?.[KEY];
|
||||
if (existing != null) {
|
||||
const winner = matchingReceipt(existing, current, input, bindings, currentController, web.binding);
|
||||
if (!winner) fail(); published = winner; return;
|
||||
}
|
||||
await tx.update(heartbeatRuns).set({ runnerProfileJson: { ...current.anchor.runnerProfileJson, [KEY]: receipt } })
|
||||
.where(and(eq(heartbeatRuns.id, current.anchor.id), eq(heartbeatRuns.companyId, input.companyId)));
|
||||
await logActivity(tx, { companyId: input.companyId, actorType: "system", actorId: "native-harness-preservation",
|
||||
action: "native_harness_filesystem_independence_verified", entityType: "heartbeat_run", entityId: current.anchor.id,
|
||||
agentId: current.anchor.agentId, runId: current.anchor.id, details: { scopeDigest: input.scopeDigest, sourceRunId: current.source.id,
|
||||
sourceCapabilityFingerprint: receipt.sourceCapabilityFingerprint, factsDigest: receipt.factsDigest, coverage: "web_evidence" } }, publications);
|
||||
});
|
||||
for (const publication of publications) publishActivity(publication);
|
||||
return { scopeDigest: input.scopeDigest, sourceRunId: facts.source.id, receiptDigest: digest(published) };
|
||||
}
|
||||
|
||||
export async function preserveNativeHarnessFilesystemIndependence(db: Db, input: Scope & {
|
||||
source: OriginalHarnessVolumeCapability; storage: StorageService; signal: AbortSignal; mode?: "web_evidence";
|
||||
}) {
|
||||
await assertOriginalHarnessVolumeSource(input.source); input.signal.throwIfAborted();
|
||||
if (input.storage.provider !== "s3") fail();
|
||||
if (input.mode === "web_evidence") return preserveWebEvidence(db, input);
|
||||
const facts = await databaseFacts(db, input, false), assets = await assetBindings(db, facts.anchor, facts.source.id);
|
||||
const history = await captureAuthorityHistory(db, facts, input.signal);
|
||||
const controller = await preserveControllerEvidence(db, facts, input, history);
|
||||
|
||||
Reference in new issue
Block a user