mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-07 10:36:05 -10:00
feat(extension-host): DSH sessions.flush through an acknowledged Engine checkpoint (CW-7054)
Lands the DSH caller-checkpoint candidate (previously only artifact patches qualified on a Boat snapshot) on current main. A plugin tool injecting `sessions` can call ctx.sessions.flush(exec.agent.session) during its own live invocation; the host sends session/checkpoint with the invocation's Engine ticket, the Engine captures the committed prefix, and the owning Runtime / terminal / exec store acknowledges persistence before the call resolves. Runtime checkpoint turns use schema 5 (older readers refuse); restore, compaction, fork and crash repair honour the private reference. Also: install of plugin bytes that changed after review returns a typed 409 (PluginContentMismatch) instead of 500. Compatibility table advertises `sessions` (flush only), which invalidates earlier trust by design. Applied, in order: checkpoint-bridge-7e7c2842, digest-hex-repair-7001d139, terminal-checkpoint-integration-v4 (a80a918c), exec-checkpoint-7c218953, runtime-prefix-checkpoint-4e8e93d4, runtime-api-checkpoint-fixtures-07a8413f, sessions-flush-adapter-v2 (2649f2a1), install-content-conflict-c1a9c47d, checkpoint-generated-protocol, checkpoint-cancel-wait-repair-8980519a (artifacts under CW/artifacts/dsh-*). Rebase repairs: two hunks hand-merged (core_call.rs imports, install/mod.rs PluginContentMismatch) and crate::utils::spawn_supervised -> codewhale_util::utils after S14. dist/ rebuilt from HEAD + these sources (builtin modules unchanged). Paths: CHANGELOG.md crates/tui/CHANGELOG.md crates/tui/extension-host/dist/agent-presets.mjs crates/tui/extension-host/dist/codewhale-extension-host.mjs crates/tui/extension-host/dist/dsh-compatibility.json crates/tui/extension-host/dist/dsh-composition-review.mjs crates/tui/extension-host/dist/protocol.mjs crates/tui/extension-host/dist/rpc.mjs crates/tui/extension-host/src/dsh/compatibility.json crates/tui/extension-host/src/protocol.generated.ts crates/tui/extension-host/src/root.ts crates/tui/extension-host/test/harness.mjs crates/tui/extension-host/test/host.test.mjs crates/tui/src/core/engine.rs crates/tui/src/core/engine/turn_loop.rs crates/tui/src/core/events.rs crates/tui/src/core/protocol_parity.rs crates/tui/src/exec_agent.rs crates/tui/src/extension_host/core_call.rs crates/tui/src/extension_host/core_call_tests.rs crates/tui/src/extension_host/mod.rs crates/tui/src/extension_host/protocol.rs crates/tui/src/extension_host/protocol/tests.rs crates/tui/src/extension_host/supervisor.rs crates/tui/src/extension_host/tests.rs crates/tui/src/extension_host/tool.rs crates/tui/src/lib.rs crates/tui/src/plugins/install/mod.rs crates/tui/src/plugins/install/tests.rs crates/tui/src/runtime_api/plugins.rs crates/tui/src/runtime_api/tests.rs crates/tui/src/runtime_threads.rs crates/tui/src/runtime_threads/tests.rs crates/tui/src/tools/codemode.rs crates/tui/src/tui/persistence_actor.rs crates/tui/src/tui/ui/event_loop.rs crates/tui/src/tui/ui/frame.rs crates/tui/src/tui/ui/tests.rs docs/EXTENSIONS.md docs/zh_hans/EXTENSIONS.md Verified (macOS, this tree incl. peers' uncommitted MCP work at test time): - extension-host typecheck; Node 420 tests: 413 pass, 7 skip, 0 fail; Bun 420: 414 pass, 6 skip, 0 fail. - CODEWHALE_EXT_HOST_TESTS=1 + compiled Bun host: scripts/dev-test.sh tui extension_host:: tools::codemode:: tui::persistence_actor:: runtime_threads:: runtime_api:: plugins::install:: exec_agent:: core::engine::tests:: core::protocol_parity tui::ui::tests:: -> 2493 run, 2493 passed (includes typescript_protocol_is_generated_from_the_rust_types and compiled Native containment). CLI build passed. - Live Engine HTTP (artifacts/dsh-e2e-20261027/logs/c1, loopback provider, no spend): DSH bundle tool calling ctx.sessions.flush returned true on turn one; saved turn is schema 5 with prefix_checkpoint; after Engine restart the checkpointed turn reloads with the plugin result and the killed turn is interrupted, not replayed. Not verified: TUI/App UI, Linux/Windows on this source, hosted CI. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
57adfdfea6
commit
17ebeb5f61
40 files changed
+2831
-215
No files matched your search
+1
-1
@@ -16,7 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- **Remote control works again:** `codewhale rc` (`/rc`) can attach to the hosted control plane for web Chat again. The control plane had only accepted the retired 0.9.13 runtime identity, so every 0.10.x Engine was refused with `runner_runtime_identity_unavailable`; it now accepts the exact released runtime identity (commit, image and digest) and fails closed when they are not configured.
|
||||
- **Modern MCP client:** MCP servers now connect through the official MCP SDK (2.4) behind Codewhale's Rust broker by default (`mcp_backend = "host"`; `"rust"` stays available for one release). Servers that speak the 2026-07-28 revision are discovered with `server/discover`; other stdio and HTTP servers fall back to the legacy `initialize` handshake, and configured SSE servers keep theirs. Rust keeps process, network, credential, catalog, approval and cancellation authority. (CW-6713)
|
||||
- **MCP compatibility fixes:** older servers that answer the discovery probe with an error or an unrelated result now connect through the legacy handshake instead of being refused after the probe. A server that rejects sign-in reports its real 401 and the `/mcp login` hint rather than the probe's "method not found". A stale OAuth token is refreshed once when the probe itself is rejected. A server whose command cannot start now says why (for example, command not found) instead of a generic broker refusal.
|
||||
- **DeepSeek Harness plugins:** reviewed DSH plugins (definition API pinned to DSH 0.2.1-rc.2 in a shared compatibility table) run in the Native extension host. Tool output is validated against its declared schema, unsupported options are refused instead of ignored, and plugin commands can submit a prompt with visible plugin attribution. Terminal command receipts are kept in thread snapshots without entering model history. Changing the compatibility table invalidates earlier trust. A newly enabled plugin's tools are offered on the first turn: the turn waits up to 10 seconds for the plugin host to finish loading instead of adding them only from the next turn. Plugin UI views stay disabled, and the import receipt says why: WebKit's network restrictions do not stop WebRTC, so plugin UI waits for an isolated renderer. (CW-7054, CW-7549)
|
||||
- **DeepSeek Harness plugins:** reviewed DSH plugins (definition API pinned to DSH 0.2.1-rc.2 in a shared compatibility table) run in the Native extension host. Tool output is validated against its declared schema, unsupported options are refused instead of ignored, and plugin commands can submit a prompt with visible plugin attribution. Terminal command receipts are kept in thread snapshots without entering model history. Changing the compatibility table invalidates earlier trust. A newly enabled plugin's tools are offered on the first turn: the turn waits up to 10 seconds for the plugin host to finish loading instead of adding them only from the next turn. Plugin UI views stay disabled, and the import receipt says why: WebKit's network restrictions do not stop WebRTC, so plugin UI waits for an isolated renderer. A plugin tool can call `ctx.sessions.flush(exec.agent.session)` to have the Engine save the conversation so far before it continues (Runtime, terminal and `exec` sessions); it reports failure instead of claiming a save. Installing a plugin whose bytes changed after review now returns a 409 conflict instead of a server error. (CW-7054, CW-7549)
|
||||
- **Computer Use is installed, not bundled:** the Computer Use plugin is no longer vendored into the binary. Install it from the `codewhale` marketplace, which points at its independent release; its consent tools load on first use, and decisions still require a human approval card. (CW-6732)
|
||||
- **Tool image validation:** an image returned by a tool must decode before it is sent, so truncated or corrupt files are omitted with a receipt even when they are already within the delivery budget.
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- **Remote control works again:** `codewhale rc` (`/rc`) can attach to the hosted control plane for web Chat again. The control plane had only accepted the retired 0.9.13 runtime identity, so every 0.10.x Engine was refused with `runner_runtime_identity_unavailable`; it now accepts the exact released runtime identity (commit, image and digest) and fails closed when they are not configured.
|
||||
- **Modern MCP client:** MCP servers now connect through the official MCP SDK (2.4) behind Codewhale's Rust broker by default (`mcp_backend = "host"`; `"rust"` stays available for one release). Servers that speak the 2026-07-28 revision are discovered with `server/discover`; other stdio and HTTP servers fall back to the legacy `initialize` handshake, and configured SSE servers keep theirs. Rust keeps process, network, credential, catalog, approval and cancellation authority. (CW-6713)
|
||||
- **MCP compatibility fixes:** older servers that answer the discovery probe with an error or an unrelated result now connect through the legacy handshake instead of being refused after the probe. A server that rejects sign-in reports its real 401 and the `/mcp login` hint rather than the probe's "method not found". A stale OAuth token is refreshed once when the probe itself is rejected. A server whose command cannot start now says why (for example, command not found) instead of a generic broker refusal.
|
||||
- **DeepSeek Harness plugins:** reviewed DSH plugins (definition API pinned to DSH 0.2.1-rc.2 in a shared compatibility table) run in the Native extension host. Tool output is validated against its declared schema, unsupported options are refused instead of ignored, and plugin commands can submit a prompt with visible plugin attribution. Terminal command receipts are kept in thread snapshots without entering model history. Changing the compatibility table invalidates earlier trust. A newly enabled plugin's tools are offered on the first turn: the turn waits up to 10 seconds for the plugin host to finish loading instead of adding them only from the next turn. Plugin UI views stay disabled, and the import receipt says why: WebKit's network restrictions do not stop WebRTC, so plugin UI waits for an isolated renderer. (CW-7054, CW-7549)
|
||||
- **DeepSeek Harness plugins:** reviewed DSH plugins (definition API pinned to DSH 0.2.1-rc.2 in a shared compatibility table) run in the Native extension host. Tool output is validated against its declared schema, unsupported options are refused instead of ignored, and plugin commands can submit a prompt with visible plugin attribution. Terminal command receipts are kept in thread snapshots without entering model history. Changing the compatibility table invalidates earlier trust. A newly enabled plugin's tools are offered on the first turn: the turn waits up to 10 seconds for the plugin host to finish loading instead of adding them only from the next turn. Plugin UI views stay disabled, and the import receipt says why: WebKit's network restrictions do not stop WebRTC, so plugin UI waits for an isolated renderer. A plugin tool can call `ctx.sessions.flush(exec.agent.session)` to have the Engine save the conversation so far before it continues (Runtime, terminal and `exec` sessions); it reports failure instead of claiming a save. Installing a plugin whose bytes changed after review now returns a 409 conflict instead of a server error. (CW-7054, CW-7549)
|
||||
- **Computer Use is installed, not bundled:** the Computer Use plugin is no longer vendored into the binary. Install it from the `codewhale` marketplace, which points at its independent release; its consent tools load on first use, and decisions still require a human approval card. (CW-6732)
|
||||
- **Tool image validation:** an image returned by a tool must decode before it is sent, so truncated or corrupt files are omitted with a receipt even when they are already within the delivery budget.
|
||||
|
||||
|
||||
+6
-2
@@ -12079,7 +12079,8 @@ var init_compatibility = __esm({
|
||||
"logger",
|
||||
"events",
|
||||
"reflect",
|
||||
"registry"
|
||||
"registry",
|
||||
"sessions"
|
||||
],
|
||||
refused_services: [
|
||||
"shellHooks",
|
||||
@@ -12322,7 +12323,10 @@ var init_compatibility = __esm({
|
||||
sha256: "66ee7f28877afa912496ba944a581ac74e7b87390e6a47c4c640d8f477bdc6e2"
|
||||
}
|
||||
},
|
||||
tool_caller_projection: "readonly-id-session-header-cwd"
|
||||
tool_caller_projection: "readonly-id-session-header-cwd",
|
||||
session_methods: {
|
||||
flush: "invocation-bound-acknowledged-checkpoint-with-host-redaction-policy"
|
||||
}
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
+56
-6
@@ -13801,7 +13801,8 @@ var init_compatibility = __esm({
|
||||
"logger",
|
||||
"events",
|
||||
"reflect",
|
||||
"registry"
|
||||
"registry",
|
||||
"sessions"
|
||||
],
|
||||
refused_services: [
|
||||
"shellHooks",
|
||||
@@ -14044,7 +14045,10 @@ var init_compatibility = __esm({
|
||||
sha256: "66ee7f28877afa912496ba944a581ac74e7b87390e6a47c4c640d8f477bdc6e2"
|
||||
}
|
||||
},
|
||||
tool_caller_projection: "readonly-id-session-header-cwd"
|
||||
tool_caller_projection: "readonly-id-session-header-cwd",
|
||||
session_methods: {
|
||||
flush: "invocation-bound-acknowledged-checkpoint-with-host-redaction-policy"
|
||||
}
|
||||
};
|
||||
}
|
||||
});
|
||||
@@ -14441,6 +14445,7 @@ var init_protocol_generated = __esm({
|
||||
{ name: "registry/register", direction: "host_to_core", request: true, params: "RegisterParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "registry/unregister", direction: "host_to_core", request: true, params: "UnregisterParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "core/call", direction: "host_to_core", request: true, params: "CoreCallParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "session/checkpoint", direction: "host_to_core", request: true, params: "SessionCheckpointParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "proc/launch", direction: "host_to_core", request: true, params: "ProcLaunchParams", tiers: ["builtin"] },
|
||||
{ name: "proc/read", direction: "host_to_core", request: true, params: "ProcSessionParams", tiers: ["builtin"] },
|
||||
{ name: "proc/write", direction: "host_to_core", request: true, params: "ProcWriteParams", tiers: ["builtin"] },
|
||||
@@ -14626,10 +14631,15 @@ var init_protocol_generated = __esm({
|
||||
required: { code: "integer", message: "string" },
|
||||
optional: { data: "json" }
|
||||
},
|
||||
SessionCheckpointParams: {
|
||||
strict: true,
|
||||
required: { owner: { ref: "OwnerRef" }, ticket: "string" },
|
||||
optional: {}
|
||||
},
|
||||
ToolCallParams: {
|
||||
strict: false,
|
||||
required: { handle: "uint", call_id: "string", input: "json", deadline_ms: "uint" },
|
||||
optional: { workspace: "string", ticket: "string", session_id: "string", agent_id: "string", origin_turn_id: "string" }
|
||||
optional: { workspace: "string", ticket: "string", checkpoint_ticket: "string", session_id: "string", agent_id: "string", origin_turn_id: "string" }
|
||||
},
|
||||
UnregisterParams: {
|
||||
strict: true,
|
||||
@@ -15827,11 +15837,35 @@ var init_root = __esm({
|
||||
return this.api().delete(key);
|
||||
}
|
||||
}
|
||||
class SessionsShim extends Service {
|
||||
constructor(ctx) {
|
||||
super(ctx, "sessions");
|
||||
}
|
||||
async flush(session) {
|
||||
const invocation = host2.toolInvocations.getStore();
|
||||
const owner = this.ctx[OWNER];
|
||||
const live = () => invocation && invocation.active && !invocation.signal.aborted && invocation.owner === owner && owner?.state === "active" && !owner.disposing && invocation.ticket !== void 0 && session === invocation.session && session !== void 0;
|
||||
if (!live()) throw new RpcError(ErrorCode.NotAvailable, "sessions.flush requires this live tool invocation’s session");
|
||||
const call = invocation;
|
||||
const run = async () => {
|
||||
if (!live()) throw new RpcError(ErrorCode.NotAvailable, "session checkpoint invocation ended");
|
||||
const result2 = await host2.rpc.request("session/checkpoint", { owner: call.owner.ref, ticket: call.ticket }, call.signal);
|
||||
if (!live()) throw new RpcError(ErrorCode.NotAvailable, "session checkpoint invocation ended");
|
||||
if (typeof result2 !== "boolean") throw new RpcError(ErrorCode.Internal, "invalid session checkpoint receipt");
|
||||
return result2;
|
||||
};
|
||||
const result = call.pending.then(run);
|
||||
call.pending = result.then(() => void 0, () => void 0);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
Object.freeze(SessionsShim.prototype);
|
||||
Object.freeze(StorageShim.prototype);
|
||||
shimClasses.set("tools", ToolsShim);
|
||||
shimClasses.set("commands", CommandsShim);
|
||||
shimClasses.set("prompt", PromptShim);
|
||||
shimClasses.set("storage", StorageShim);
|
||||
shimClasses.set("sessions", SessionsShim);
|
||||
shimClasses.set("skills", SkillsShim);
|
||||
shimClasses.set("avatars", AvatarsShim);
|
||||
shimClasses.set("mcp", McpShim);
|
||||
@@ -15840,6 +15874,7 @@ var init_root = __esm({
|
||||
root.plugin(CommandsShim);
|
||||
root.plugin(PromptShim);
|
||||
root.plugin(StorageShim);
|
||||
root.plugin(SessionsShim);
|
||||
root.plugin(SkillsShim);
|
||||
root.plugin(AvatarsShim);
|
||||
root.plugin(McpShim);
|
||||
@@ -15850,6 +15885,7 @@ var init_root = __esm({
|
||||
rpc;
|
||||
tier;
|
||||
root;
|
||||
toolInvocations = new AsyncLocalStorage();
|
||||
owners = /* @__PURE__ */ new Map();
|
||||
toolRegistrations;
|
||||
commandRegistrations;
|
||||
@@ -16105,11 +16141,25 @@ var init_root = __esm({
|
||||
session: Object.freeze({ header: Object.freeze({ cwd: workspace }) })
|
||||
}) : void 0;
|
||||
const exec = Object.freeze({ signal, callId, args: input, ...callContext(local.owner, workspace, identity), ...core2, ...agent === void 0 ? {} : { agent } });
|
||||
const run = ownerStorage.run(local.owner, async () => {
|
||||
const closed = new AbortController();
|
||||
const invocation = {
|
||||
owner: local.owner,
|
||||
session: agent?.session,
|
||||
ticket: identity?.checkpoint_ticket,
|
||||
signal: AbortSignal.any([signal, closed.signal]),
|
||||
active: true,
|
||||
pending: Promise.resolve()
|
||||
};
|
||||
const run = ownerStorage.run(local.owner, () => this.toolInvocations.run(invocation, async () => {
|
||||
const value = await definition.execute(input, exec);
|
||||
return renderResult(definition, input, value);
|
||||
});
|
||||
return Promise.race([run, abortedBy(signal)]);
|
||||
}));
|
||||
try {
|
||||
return await Promise.race([run, abortedBy(signal)]);
|
||||
} finally {
|
||||
invocation.active = false;
|
||||
closed.abort();
|
||||
}
|
||||
}
|
||||
/** `command/run`: only for a handle the core admitted, and only on a user's own invocation. */
|
||||
async callCommand(handle, rawInput, commandId, signal, workspace, identity) {
|
||||
|
||||
+6
-2
@@ -13,7 +13,8 @@
|
||||
"logger",
|
||||
"events",
|
||||
"reflect",
|
||||
"registry"
|
||||
"registry",
|
||||
"sessions"
|
||||
],
|
||||
"refused_services": [
|
||||
"shellHooks",
|
||||
@@ -256,5 +257,8 @@
|
||||
"sha256": "66ee7f28877afa912496ba944a581ac74e7b87390e6a47c4c640d8f477bdc6e2"
|
||||
}
|
||||
},
|
||||
"tool_caller_projection": "readonly-id-session-header-cwd"
|
||||
"tool_caller_projection": "readonly-id-session-header-cwd",
|
||||
"session_methods": {
|
||||
"flush": "invocation-bound-acknowledged-checkpoint-with-host-redaction-policy"
|
||||
}
|
||||
}
|
||||
@@ -12030,7 +12030,8 @@ var init_compatibility = __esm({
|
||||
"logger",
|
||||
"events",
|
||||
"reflect",
|
||||
"registry"
|
||||
"registry",
|
||||
"sessions"
|
||||
],
|
||||
refused_services: [
|
||||
"shellHooks",
|
||||
@@ -12273,7 +12274,10 @@ var init_compatibility = __esm({
|
||||
sha256: "66ee7f28877afa912496ba944a581ac74e7b87390e6a47c4c640d8f477bdc6e2"
|
||||
}
|
||||
},
|
||||
tool_caller_projection: "readonly-id-session-header-cwd"
|
||||
tool_caller_projection: "readonly-id-session-header-cwd",
|
||||
session_methods: {
|
||||
flush: "invocation-bound-acknowledged-checkpoint-with-host-redaction-policy"
|
||||
}
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
+7
-1
@@ -37,6 +37,7 @@ var METHODS = [
|
||||
{ name: "registry/register", direction: "host_to_core", request: true, params: "RegisterParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "registry/unregister", direction: "host_to_core", request: true, params: "UnregisterParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "core/call", direction: "host_to_core", request: true, params: "CoreCallParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "session/checkpoint", direction: "host_to_core", request: true, params: "SessionCheckpointParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "proc/launch", direction: "host_to_core", request: true, params: "ProcLaunchParams", tiers: ["builtin"] },
|
||||
{ name: "proc/read", direction: "host_to_core", request: true, params: "ProcSessionParams", tiers: ["builtin"] },
|
||||
{ name: "proc/write", direction: "host_to_core", request: true, params: "ProcWriteParams", tiers: ["builtin"] },
|
||||
@@ -222,10 +223,15 @@ var SHAPES = {
|
||||
required: { code: "integer", message: "string" },
|
||||
optional: { data: "json" }
|
||||
},
|
||||
SessionCheckpointParams: {
|
||||
strict: true,
|
||||
required: { owner: { ref: "OwnerRef" }, ticket: "string" },
|
||||
optional: {}
|
||||
},
|
||||
ToolCallParams: {
|
||||
strict: false,
|
||||
required: { handle: "uint", call_id: "string", input: "json", deadline_ms: "uint" },
|
||||
optional: { workspace: "string", ticket: "string", session_id: "string", agent_id: "string", origin_turn_id: "string" }
|
||||
optional: { workspace: "string", ticket: "string", checkpoint_ticket: "string", session_id: "string", agent_id: "string", origin_turn_id: "string" }
|
||||
},
|
||||
UnregisterParams: {
|
||||
strict: true,
|
||||
|
||||
+7
-1
@@ -34,6 +34,7 @@ var METHODS = [
|
||||
{ name: "registry/register", direction: "host_to_core", request: true, params: "RegisterParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "registry/unregister", direction: "host_to_core", request: true, params: "UnregisterParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "core/call", direction: "host_to_core", request: true, params: "CoreCallParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "session/checkpoint", direction: "host_to_core", request: true, params: "SessionCheckpointParams", tiers: ["plugin", "builtin"] },
|
||||
{ name: "proc/launch", direction: "host_to_core", request: true, params: "ProcLaunchParams", tiers: ["builtin"] },
|
||||
{ name: "proc/read", direction: "host_to_core", request: true, params: "ProcSessionParams", tiers: ["builtin"] },
|
||||
{ name: "proc/write", direction: "host_to_core", request: true, params: "ProcWriteParams", tiers: ["builtin"] },
|
||||
@@ -219,10 +220,15 @@ var SHAPES = {
|
||||
required: { code: "integer", message: "string" },
|
||||
optional: { data: "json" }
|
||||
},
|
||||
SessionCheckpointParams: {
|
||||
strict: true,
|
||||
required: { owner: { ref: "OwnerRef" }, ticket: "string" },
|
||||
optional: {}
|
||||
},
|
||||
ToolCallParams: {
|
||||
strict: false,
|
||||
required: { handle: "uint", call_id: "string", input: "json", deadline_ms: "uint" },
|
||||
optional: { workspace: "string", ticket: "string", session_id: "string", agent_id: "string", origin_turn_id: "string" }
|
||||
optional: { workspace: "string", ticket: "string", checkpoint_ticket: "string", session_id: "string", agent_id: "string", origin_turn_id: "string" }
|
||||
},
|
||||
UnregisterParams: {
|
||||
strict: true,
|
||||
|
||||
@@ -13,7 +13,8 @@
|
||||
"logger",
|
||||
"events",
|
||||
"reflect",
|
||||
"registry"
|
||||
"registry",
|
||||
"sessions"
|
||||
],
|
||||
"refused_services": [
|
||||
"shellHooks",
|
||||
@@ -256,5 +257,8 @@
|
||||
"sha256": "66ee7f28877afa912496ba944a581ac74e7b87390e6a47c4c640d8f477bdc6e2"
|
||||
}
|
||||
},
|
||||
"tool_caller_projection": "readonly-id-session-header-cwd"
|
||||
"tool_caller_projection": "readonly-id-session-header-cwd",
|
||||
"session_methods": {
|
||||
"flush": "invocation-bound-acknowledged-checkpoint-with-host-redaction-policy"
|
||||
}
|
||||
}
|
||||
@@ -45,6 +45,7 @@ export const METHODS = [
|
||||
{ name: 'registry/register', direction: 'host_to_core', request: true, params: 'RegisterParams', tiers: ['plugin', 'builtin'] },
|
||||
{ name: 'registry/unregister', direction: 'host_to_core', request: true, params: 'UnregisterParams', tiers: ['plugin', 'builtin'] },
|
||||
{ name: 'core/call', direction: 'host_to_core', request: true, params: 'CoreCallParams', tiers: ['plugin', 'builtin'] },
|
||||
{ name: 'session/checkpoint', direction: 'host_to_core', request: true, params: 'SessionCheckpointParams', tiers: ['plugin', 'builtin'] },
|
||||
{ name: 'proc/launch', direction: 'host_to_core', request: true, params: 'ProcLaunchParams', tiers: ['builtin'] },
|
||||
{ name: 'proc/read', direction: 'host_to_core', request: true, params: 'ProcSessionParams', tiers: ['builtin'] },
|
||||
{ name: 'proc/write', direction: 'host_to_core', request: true, params: 'ProcWriteParams', tiers: ['builtin'] },
|
||||
@@ -251,10 +252,15 @@ export const SHAPES: { readonly [name: string]: Shape } = {
|
||||
required: { code: 'integer', message: 'string' },
|
||||
optional: { data: 'json' },
|
||||
},
|
||||
SessionCheckpointParams: {
|
||||
strict: true,
|
||||
required: { owner: { ref: 'OwnerRef' }, ticket: 'string' },
|
||||
optional: {},
|
||||
},
|
||||
ToolCallParams: {
|
||||
strict: false,
|
||||
required: { handle: 'uint', call_id: 'string', input: 'json', deadline_ms: 'uint' },
|
||||
optional: { workspace: 'string', ticket: 'string', session_id: 'string', agent_id: 'string', origin_turn_id: 'string' },
|
||||
optional: { workspace: 'string', ticket: 'string', checkpoint_ticket: 'string', session_id: 'string', agent_id: 'string', origin_turn_id: 'string' },
|
||||
},
|
||||
UnregisterParams: {
|
||||
strict: true,
|
||||
@@ -517,6 +523,11 @@ export interface RpcErrorWire {
|
||||
data?: Json
|
||||
}
|
||||
|
||||
export interface SessionCheckpointParams {
|
||||
owner: OwnerRef
|
||||
ticket: string
|
||||
}
|
||||
|
||||
export interface ToolCallParams {
|
||||
handle: number
|
||||
call_id: string
|
||||
@@ -524,6 +535,7 @@ export interface ToolCallParams {
|
||||
deadline_ms: number
|
||||
workspace?: string
|
||||
ticket?: string
|
||||
checkpoint_ticket?: string
|
||||
session_id?: string
|
||||
agent_id?: string
|
||||
origin_turn_id?: string
|
||||
|
||||
@@ -132,8 +132,18 @@ function withDeadline<T>(promise: Promise<T>, ms: number, label: string): Promis
|
||||
return Promise.race([promise, deadline]).finally(() => clearTimeout(timer))
|
||||
}
|
||||
|
||||
interface ToolInvocation {
|
||||
owner: OwnerRecord
|
||||
session: object | undefined
|
||||
ticket: string | undefined
|
||||
signal: AbortSignal
|
||||
active: boolean
|
||||
pending: Promise<void>
|
||||
}
|
||||
|
||||
export class HostRoot {
|
||||
readonly root: any
|
||||
private readonly toolInvocations = new AsyncLocalStorage<ToolInvocation>()
|
||||
readonly owners = new Map<string, OwnerRecord>()
|
||||
private readonly toolRegistrations: OwnedRegistrations<OwnerRecord, LocalTool>
|
||||
private readonly commandRegistrations: OwnedRegistrations<OwnerRecord, LocalCommand<OwnerRecord>>
|
||||
@@ -274,11 +284,35 @@ export class HostRoot {
|
||||
set(key: string, value: Json) { return this.api().set(key, value) }
|
||||
delete(key: string) { return this.api().delete(key) }
|
||||
}
|
||||
class SessionsShim extends Service {
|
||||
constructor(ctx: any) { super(ctx, 'sessions') }
|
||||
async flush(session: unknown): Promise<boolean> {
|
||||
const invocation = host.toolInvocations.getStore()
|
||||
const owner: OwnerRecord | undefined = (this.ctx as any)[OWNER]
|
||||
const live = () => invocation && invocation.active && !invocation.signal.aborted
|
||||
&& invocation.owner === owner && owner?.state === 'active' && !owner.disposing
|
||||
&& invocation.ticket !== undefined && session === invocation.session && session !== undefined
|
||||
if (!live()) throw new RpcError(ErrorCode.NotAvailable, 'sessions.flush requires this live tool invocation’s session')
|
||||
const call = invocation!
|
||||
const run = async () => {
|
||||
if (!live()) throw new RpcError(ErrorCode.NotAvailable, 'session checkpoint invocation ended')
|
||||
const result = await host.rpc.request<unknown>('session/checkpoint', { owner: call.owner.ref, ticket: call.ticket }, call.signal)
|
||||
if (!live()) throw new RpcError(ErrorCode.NotAvailable, 'session checkpoint invocation ended')
|
||||
if (typeof result !== 'boolean') throw new RpcError(ErrorCode.Internal, 'invalid session checkpoint receipt')
|
||||
return result
|
||||
}
|
||||
const result = call.pending.then(run)
|
||||
call.pending = result.then(() => undefined, () => undefined)
|
||||
return result
|
||||
}
|
||||
}
|
||||
Object.freeze(SessionsShim.prototype)
|
||||
Object.freeze(StorageShim.prototype)
|
||||
shimClasses.set('tools', ToolsShim)
|
||||
shimClasses.set('commands', CommandsShim)
|
||||
shimClasses.set('prompt', PromptShim)
|
||||
shimClasses.set('storage', StorageShim)
|
||||
shimClasses.set('sessions', SessionsShim)
|
||||
shimClasses.set('skills', SkillsShim)
|
||||
shimClasses.set('avatars', AvatarsShim)
|
||||
shimClasses.set('mcp',McpShim)
|
||||
@@ -287,6 +321,7 @@ export class HostRoot {
|
||||
root.plugin(CommandsShim)
|
||||
root.plugin(PromptShim)
|
||||
root.plugin(StorageShim)
|
||||
root.plugin(SessionsShim)
|
||||
root.plugin(SkillsShim)
|
||||
root.plugin(AvatarsShim)
|
||||
root.plugin(McpShim)
|
||||
@@ -553,11 +588,25 @@ export class HostRoot {
|
||||
session: Object.freeze({ header: Object.freeze({ cwd: workspace }) }),
|
||||
}) : undefined
|
||||
const exec = Object.freeze({ signal, callId, args: input, ...callContext(local.owner, workspace, identity), ...core, ...(agent === undefined ? {} : { agent }) })
|
||||
const run = ownerStorage.run(local.owner, async () => {
|
||||
const closed = new AbortController()
|
||||
const invocation: ToolInvocation = {
|
||||
owner: local.owner,
|
||||
session: agent?.session,
|
||||
ticket: identity?.checkpoint_ticket,
|
||||
signal: AbortSignal.any([signal, closed.signal]),
|
||||
active: true,
|
||||
pending: Promise.resolve(),
|
||||
}
|
||||
const run = ownerStorage.run(local.owner, () => this.toolInvocations.run(invocation, async () => {
|
||||
const value = await definition.execute(input, exec)
|
||||
return renderResult(definition, input, value)
|
||||
})
|
||||
return Promise.race([run, abortedBy(signal)])
|
||||
}))
|
||||
try {
|
||||
return await Promise.race([run, abortedBy(signal)])
|
||||
} finally {
|
||||
invocation.active = false
|
||||
closed.abort()
|
||||
}
|
||||
}
|
||||
|
||||
/** `command/run`: only for a handle the core admitted, and only on a user's own invocation. */
|
||||
@@ -603,6 +652,7 @@ export class HostRoot {
|
||||
|
||||
/** The read-only context a call carries beyond its input; a field the core did not send is absent. */
|
||||
interface InvocationIdentityWire {
|
||||
checkpoint_ticket?: string
|
||||
session_id?: string
|
||||
agent_id?: string
|
||||
origin_turn_id?: string
|
||||
|
||||
@@ -51,7 +51,7 @@ export const LIMITS = { max_frame: 32 * 1024 * 1024, max_inflight: 256, dispose_
|
||||
* cancelled is not answered. Requests are recorded in `coreCalls`, the host's
|
||||
* `$/cancel` ids in `cancels`.
|
||||
*/
|
||||
export async function startHost({ admit, env, ownGroup = false, tier = 'plugin', coreCall, compiledHost, cwd } = {}) {
|
||||
export async function startHost({ admit, env, ownGroup = false, tier = 'plugin', coreCall, checkpoint, compiledHost, cwd } = {}) {
|
||||
const started = performance.now()
|
||||
// `ownGroup` spawns the host as a process-group leader and tells it so, as
|
||||
// the Rust core does on Unix.
|
||||
@@ -71,6 +71,7 @@ export async function startHost({ admit, env, ownGroup = false, tier = 'plugin',
|
||||
logs: [],
|
||||
faulted: [],
|
||||
coreCalls: [],
|
||||
checkpoints: [],
|
||||
cancels: [],
|
||||
registry: [],
|
||||
hello: null,
|
||||
@@ -142,9 +143,12 @@ export async function startHost({ admit, env, ownGroup = false, tier = 'plugin',
|
||||
case '$/cancel':
|
||||
host.cancels.push(message.params.id)
|
||||
break
|
||||
case 'session/checkpoint':
|
||||
case 'core/call': {
|
||||
host.coreCalls.push({ id: message.id, ...message.params })
|
||||
const answer = coreCall ? coreCall(message.params, message.id) : { error: { code: -32002, message: 'no core/call handler' } }
|
||||
const handler = message.method === 'session/checkpoint' ? checkpoint : coreCall
|
||||
const records = message.method === 'session/checkpoint' ? host.checkpoints : host.coreCalls
|
||||
records.push({ id: message.id, ...message.params })
|
||||
const answer = handler ? handler(message.params, message.id) : { error: { code: -32002, message: 'no handler for ' + message.method } }
|
||||
const id = message.id
|
||||
Promise.resolve(answer).then((value) => {
|
||||
if (host.cancels.includes(id) || child.exitCode !== null || stopping) return
|
||||
|
||||
@@ -174,11 +174,11 @@ test('injecting a service the host does not provide fails with its name', async
|
||||
const { result } = await activate(host, 'needs-secrets', plugin.entry)
|
||||
assert.equal(result.status, 'failed')
|
||||
assert.match(result.diagnostic, /requires `secrets`/)
|
||||
const provided = tempPlugin("export const inject = ['shellHooks']; export function apply(ctx) { if (!ctx.shellHooks) throw new Error('missing shell hooks') }\n")
|
||||
const provided = tempPlugin("export const inject = ['shellHooks', 'sessions']; export function apply(ctx) { if (!ctx.shellHooks || typeof ctx.sessions.flush !== 'function') throw new Error('missing provided service') }\n")
|
||||
t.after(() => provided.cleanup())
|
||||
const available = await activate(host, 'needs-shell-hooks', provided.entry)
|
||||
assert.equal(available.result.status, 'ok', available.result.diagnostic)
|
||||
for (const service of ['llm', 'sessions', 'systemPrompt', 'agents', 'subagents', 'userQuestions']) {
|
||||
for (const service of ['llm', 'systemPrompt', 'agents', 'subagents', 'userQuestions']) {
|
||||
const pure = tempPlugin(`import { createUserMessage } from '@deepseek-ai/dsh-llm'
|
||||
export const inject = [${JSON.stringify(service)}]
|
||||
export function apply() { createUserMessage({ content: [], source: { kind: 'fixture' } }) }
|
||||
@@ -1154,14 +1154,16 @@ test('several core calls can be in flight at once, each answered to its own requ
|
||||
})
|
||||
|
||||
test('storage survives owner reload, refuses disposed owner access, and call identity is per invocation', async (t) => {
|
||||
const host = await startHost()
|
||||
let checkpointResult = true
|
||||
const host = await startHost({ checkpoint: () => typeof checkpointResult === 'function' ? checkpointResult() : checkpointResult })
|
||||
t.after(() => host.stop())
|
||||
const plugin = tempPlugin(`
|
||||
export const inject = ['tools', 'commands', 'storage']
|
||||
export const inject = ['tools', 'commands', 'storage', 'sessions']
|
||||
let savedStorage
|
||||
let savedAgent
|
||||
let savedSessions
|
||||
export async function apply(ctx, config) {
|
||||
if (config.capture) savedStorage = ctx.storage
|
||||
if (config.capture) { savedStorage = ctx.storage; savedSessions = ctx.sessions }
|
||||
const identity = (exec) => ({
|
||||
frozen: Object.isFrozen(exec),
|
||||
sessionId: exec.sessionId ?? null,
|
||||
@@ -1177,6 +1179,13 @@ export async function apply(ctx, config) {
|
||||
})
|
||||
ctx.tools.register({ name: config.tool, description: 'Read owner-local state.', parameters: { type: 'object' },
|
||||
async execute(input, exec) {
|
||||
if (input.flush) {
|
||||
try {
|
||||
const target = input.flush === 'forged' ? { header: { cwd: exec.workspace } } : input.flush === 'retained' ? savedAgent?.session : exec.agent?.session
|
||||
const service = input.flush === 'foreign-owner' ? savedSessions : ctx.sessions
|
||||
return { flushed: input.many ? await Promise.all([service.flush(target), service.flush(target)]) : await service.flush(target) }
|
||||
} catch (error) { return { flushError: error.code } }
|
||||
}
|
||||
if (input.captureAgent) savedAgent = exec.agent
|
||||
if (input.readSavedAgent) return { previous: agentIdentity(savedAgent), current: agentIdentity(exec.agent) }
|
||||
if (input.saved) { try { await savedStorage.get('state'); return { accessed: true } } catch (error) { return { error: error.code } } }
|
||||
@@ -1212,12 +1221,52 @@ export async function apply(ctx, config) {
|
||||
assert.equal((await tool('first_state', {}, { workspace: '/workspace/only' })).structured.agent, null)
|
||||
assert.equal((await tool('first_state', {}, { ...identity, session_id: '' })).structured.agent, null)
|
||||
assert.equal((await tool('first_state', {}, { ...identity, workspace: '' })).structured.agent, null)
|
||||
const checkpointIdentity = { ...rootIdentity, checkpoint_ticket: 'exact-checkpoint-ticket' }
|
||||
assert.deepEqual((await tool('first_state', { flush: 'current' }, rootIdentity)).structured, { flushError: ErrorCode.NotAvailable })
|
||||
assert.deepEqual((await tool('first_state', { flush: 'forged' }, checkpointIdentity)).structured, { flushError: ErrorCode.NotAvailable })
|
||||
assert.deepEqual((await tool('first_state', { flush: 'retained' }, checkpointIdentity)).structured, { flushError: ErrorCode.NotAvailable })
|
||||
assert.equal(host.checkpoints.length, 0)
|
||||
assert.deepEqual((await tool('first_state', { flush: 'current' }, checkpointIdentity)).structured, { flushed: true })
|
||||
assert.deepEqual(host.checkpoints[0], { id: host.checkpoints[0].id, owner: first.ref, ticket: 'exact-checkpoint-ticket' })
|
||||
checkpointResult = false
|
||||
assert.deepEqual((await tool('first_state', { flush: 'current' }, checkpointIdentity)).structured, { flushed: false })
|
||||
checkpointResult = 'not-a-receipt'
|
||||
assert.deepEqual((await tool('first_state', { flush: 'current' }, checkpointIdentity)).structured, { flushError: ErrorCode.Internal })
|
||||
checkpointResult = { error: { code: ErrorCode.Refused, message: 'checkpoint refused' } }
|
||||
assert.deepEqual((await tool('first_state', { flush: 'current' }, checkpointIdentity)).structured, { flushError: ErrorCode.Refused })
|
||||
let inFlight = 0
|
||||
let peak = 0
|
||||
checkpointResult = async () => {
|
||||
inFlight++
|
||||
peak = Math.max(peak, inFlight)
|
||||
await new Promise(resolve => setTimeout(resolve, 10))
|
||||
inFlight--
|
||||
return true
|
||||
}
|
||||
assert.deepEqual((await tool('first_state', { flush: 'current', many: true }, checkpointIdentity)).structured, { flushed: [true, true] })
|
||||
assert.equal(peak, 1, 'one invocation serializes its checkpoint requests')
|
||||
let releaseCheckpoint
|
||||
checkpointResult = new Promise(resolve => { releaseCheckpoint = resolve })
|
||||
const pendingCheckpoint = host.request('tool/call', {
|
||||
handle: handle('first_state'), call_id: 'cancel-checkpoint', input: { flush: 'current' }, deadline_ms: 5000,
|
||||
...rootIdentity, checkpoint_ticket: 'cancel-checkpoint-ticket',
|
||||
})
|
||||
await host.waitFor(message => message.method === 'session/checkpoint' && message.params.ticket === 'cancel-checkpoint-ticket')
|
||||
const checkpointRequest = host.checkpoints.find(request => request.ticket === 'cancel-checkpoint-ticket')
|
||||
const cancelledCheckpoint = host.waitFor(message => message.method === '$/cancel' && message.params.id === checkpointRequest.id)
|
||||
host.cancel(pendingCheckpoint.id)
|
||||
await assert.rejects(pendingCheckpoint.promise, error => error.code === ErrorCode.Cancelled)
|
||||
await cancelledCheckpoint
|
||||
releaseCheckpoint(true)
|
||||
checkpointResult = true
|
||||
assert.deepEqual(await host.call('host/ping', {}), {})
|
||||
assert.equal(host.coreCalls.length, 0, 'read-only projections do not grant core calls')
|
||||
const command = await host.call('command/run', { handle: handle('first-state'), command_id: 'state-command', raw_input: '', deadline_ms: 5000, ...identity })
|
||||
assert.deepEqual(JSON.parse(command.text), { frozen: true, sessionId: 'session-one', agentId: 'agent-one', originTurnId: 'turn-one' })
|
||||
await host.call('ext/deactivate', { owner: first.ref })
|
||||
const other = await activate(host, 'other-state-owner', plugin.entry, { config: { tool: 'other_state', command: 'other-state' }, data_dir: otherDir })
|
||||
assert.equal(other.result.status, 'ok')
|
||||
assert.deepEqual((await tool('other_state', { flush: 'foreign-owner' }, checkpointIdentity)).structured, { flushError: ErrorCode.NotAvailable })
|
||||
assert.deepEqual((await tool('other_state', { saved: true })).structured, { error: 'not_available' })
|
||||
assert.equal((await tool('other_state')).structured.value, null, 'another owner directory has separate state')
|
||||
assert.deepEqual((await tool('other_state', { readSavedAgent: true }, rootIdentity)).structured,
|
||||
|
||||
@@ -573,6 +573,7 @@ pub struct EngineConfig {
|
||||
/// Interactive TUI sessions enable this; headless and machine-readable
|
||||
/// hosts disable it so stdout remains protocol-clean.
|
||||
pub terminal_chrome_enabled: bool,
|
||||
pub session_checkpoint_backend: Option<crate::core::events::SessionCheckpointProjection>,
|
||||
/// Resolved advisor watcher configuration (#3982). Off by default.
|
||||
/// Updated live by `Op::SetAdvisorEnabled`.
|
||||
pub advisor_config: crate::tools::subagent::AdvisorConfig,
|
||||
@@ -686,6 +687,7 @@ impl Default for EngineConfig {
|
||||
workspace_follow_symlinks: false,
|
||||
exec_policy_engine: codewhale_execpolicy::ExecPolicyEngine::new(Vec::new(), Vec::new()),
|
||||
terminal_chrome_enabled: true,
|
||||
session_checkpoint_backend: None,
|
||||
advisor_config: crate::tools::subagent::AdvisorConfig::disabled(),
|
||||
}
|
||||
}
|
||||
@@ -4746,6 +4748,7 @@ impl Engine {
|
||||
}
|
||||
let _ = self
|
||||
.send_event(Event::SessionUpdated {
|
||||
checkpoint: None,
|
||||
session_id: self.session.id.clone(),
|
||||
messages: self.session.messages.snapshot(),
|
||||
system_prompt: self.session.system_prompt.clone(),
|
||||
|
||||
@@ -3744,6 +3744,7 @@ impl Engine {
|
||||
(Some(caller), Some(registry)) => gate.for_extension(caller.clone(), registry.all()),
|
||||
_ => gate,
|
||||
};
|
||||
let checkpoint_turn_id = context.execution.origin_turn_id.clone();
|
||||
context.execution.nested_call_gate = Some(gate);
|
||||
let cancel = self.cancel_token.clone();
|
||||
let run = Self::execute_tool_with_lock(
|
||||
@@ -3770,6 +3771,60 @@ impl Engine {
|
||||
}
|
||||
result = &mut run => return (result, false),
|
||||
Some(request) = requests.recv() => {
|
||||
let request = match request {
|
||||
crate::tools::codemode::NestedCallRequest::Tool(request) => request,
|
||||
crate::tools::codemode::NestedCallRequest::Checkpoint { caller, withdraw, reply } => {
|
||||
if reply.is_closed() || withdraw.is_cancelled() {
|
||||
continue;
|
||||
}
|
||||
let Some(projection) = self.config.session_checkpoint_backend else {
|
||||
let _ = reply.send(Err(ToolError::not_available("owning host has no acknowledged session checkpoint backend")));
|
||||
continue;
|
||||
};
|
||||
if extension.is_none() || self.child_host.is_some() || caller.agent_id.is_some()
|
||||
|| caller.session_id != self.session.id || caller.workspace != self.session.workspace
|
||||
|| Some(caller.turn_id.as_str()) != checkpoint_turn_id.as_deref()
|
||||
{
|
||||
let _ = reply.send(Err(ToolError::not_available("session checkpoint does not belong to the serving root turn")));
|
||||
continue;
|
||||
}
|
||||
let snapshot = Arc::new(self.session_snapshot());
|
||||
let prefix_sha256 = match crate::core::events::SessionCheckpointRequest::prefix_hash(&snapshot.messages) {
|
||||
Ok(hash) => hash,
|
||||
Err(error) => {
|
||||
let _ = reply.send(Err(ToolError::not_available(error)));
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let request = crate::core::events::SessionCheckpointRequest::new(caller, snapshot, prefix_sha256, projection, withdraw, reply);
|
||||
let event = Event::SessionUpdated {
|
||||
checkpoint: Some(request.clone()),
|
||||
session_id: self.session.id.clone(),
|
||||
messages: self.session.messages.snapshot(),
|
||||
system_prompt: self.session.system_prompt.clone(),
|
||||
instruction_assembly: self.session.instruction_assembly.clone(),
|
||||
model: self.session.model.clone(),
|
||||
workspace: self.session.workspace.clone(),
|
||||
};
|
||||
let sent = tokio::select! {
|
||||
biased;
|
||||
() = cancel.cancelled() => {
|
||||
request.fail("session checkpoint turn was cancelled");
|
||||
return (Ok(RichToolResult::plain(interrupted_active_tool_result())), true);
|
||||
}
|
||||
() = request.withdraw.cancelled() => None,
|
||||
result = &mut run => {
|
||||
request.fail("extension invocation ended before checkpoint delivery");
|
||||
return (result, false);
|
||||
}
|
||||
sent = self.send_event(event) => Some(sent),
|
||||
};
|
||||
if !matches!(sent, Some(Ok(()))) {
|
||||
request.fail("owning host session checkpoint event channel is unavailable");
|
||||
}
|
||||
continue;
|
||||
}
|
||||
};
|
||||
// Nobody is waiting for this one any more (a withdrawn
|
||||
// extension call): no plan, no card.
|
||||
if request.is_stale() {
|
||||
|
||||
@@ -3,7 +3,13 @@
|
||||
//! These events flow from the engine to the TUI via a channel,
|
||||
//! enabling non-blocking, real-time updates.
|
||||
|
||||
use std::{path::PathBuf, sync::Arc};
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::{
|
||||
path::{Path, PathBuf},
|
||||
sync::{Arc, Mutex},
|
||||
};
|
||||
use tokio::sync::oneshot;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use serde_json::Value;
|
||||
@@ -201,6 +207,163 @@ impl AgentProgressEventMeta {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub(crate) struct SessionCheckpointCaller {
|
||||
pub(crate) session_id: String,
|
||||
pub(crate) agent_id: Option<String>,
|
||||
pub(crate) turn_id: String,
|
||||
pub(crate) call_id: String,
|
||||
pub(crate) workspace: PathBuf,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum SessionCheckpointProjection {
|
||||
Exact,
|
||||
RuntimeReceiptRedacted,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub(crate) struct SessionCheckpointReceipt {
|
||||
pub(crate) persisted_sha256: String,
|
||||
pub(crate) messages_len: usize,
|
||||
pub(crate) projection: SessionCheckpointProjection,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct SessionCheckpointRequest {
|
||||
pub(crate) checkpoint_id: String,
|
||||
pub(crate) caller: SessionCheckpointCaller,
|
||||
pub(crate) prefix_sha256: String,
|
||||
pub(crate) snapshot: Arc<crate::core::ops::SessionSnapshot>,
|
||||
pub(crate) withdraw: CancellationToken,
|
||||
projection: SessionCheckpointProjection,
|
||||
reply: Mutex<Option<oneshot::Sender<Result<bool, ToolError>>>>,
|
||||
}
|
||||
|
||||
impl SessionCheckpointRequest {
|
||||
pub(crate) fn new(
|
||||
caller: SessionCheckpointCaller,
|
||||
snapshot: Arc<crate::core::ops::SessionSnapshot>,
|
||||
prefix_sha256: String,
|
||||
projection: SessionCheckpointProjection,
|
||||
withdraw: CancellationToken,
|
||||
reply: oneshot::Sender<Result<bool, ToolError>>,
|
||||
) -> Arc<Self> {
|
||||
Arc::new(Self {
|
||||
checkpoint_id: uuid::Uuid::new_v4().to_string(),
|
||||
caller,
|
||||
prefix_sha256,
|
||||
snapshot,
|
||||
withdraw,
|
||||
projection,
|
||||
reply: Mutex::new(Some(reply)),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn prefix_hash(messages: &[Message]) -> Result<String, String> {
|
||||
serde_json::to_vec(messages)
|
||||
.map(|bytes| {
|
||||
Sha256::digest(bytes)
|
||||
.iter()
|
||||
.map(|byte| format!("{byte:02x}"))
|
||||
.collect()
|
||||
})
|
||||
.map_err(|error| format!("session checkpoint could not encode its prefix: {error}"))
|
||||
}
|
||||
|
||||
pub(crate) fn validate_prefix(
|
||||
&self,
|
||||
session_id: &str,
|
||||
workspace: &Path,
|
||||
messages: &[Message],
|
||||
) -> Result<(), String> {
|
||||
if self.is_stale() {
|
||||
return Err("session checkpoint was withdrawn".into());
|
||||
}
|
||||
if self.caller.agent_id.is_some()
|
||||
|| session_id != self.caller.session_id
|
||||
|| session_id != self.snapshot.session_id
|
||||
|| workspace != self.caller.workspace
|
||||
|| workspace != self.snapshot.workspace
|
||||
|| messages.len() != self.snapshot.messages.len()
|
||||
|| Self::prefix_hash(messages)? != self.prefix_sha256
|
||||
{
|
||||
return Err("session checkpoint does not match its captured owner and prefix".into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn is_stale(&self) -> bool {
|
||||
self.withdraw.is_cancelled()
|
||||
|| self
|
||||
.reply
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.as_ref()
|
||||
.is_none_or(oneshot::Sender::is_closed)
|
||||
}
|
||||
|
||||
fn finish(&self, result: Result<bool, ToolError>) {
|
||||
let result = if self.withdraw.is_cancelled() {
|
||||
Err(ToolError::cancelled("session checkpoint was withdrawn"))
|
||||
} else {
|
||||
result
|
||||
};
|
||||
if let Some(reply) = self
|
||||
.reply
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.take()
|
||||
{
|
||||
let _ = reply.send(result);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn succeed(&self, receipt: SessionCheckpointReceipt) {
|
||||
let valid_hash = receipt.persisted_sha256.len() == 64
|
||||
&& receipt
|
||||
.persisted_sha256
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte));
|
||||
let allowed = match receipt.projection {
|
||||
SessionCheckpointProjection::Exact => receipt.persisted_sha256 == self.prefix_sha256,
|
||||
SessionCheckpointProjection::RuntimeReceiptRedacted => {
|
||||
self.projection == SessionCheckpointProjection::RuntimeReceiptRedacted
|
||||
}
|
||||
};
|
||||
if !valid_hash || !allowed || receipt.messages_len != self.snapshot.messages.len() {
|
||||
self.fail(
|
||||
"session checkpoint persistence receipt does not match its captured projection",
|
||||
);
|
||||
return;
|
||||
}
|
||||
self.finish(Ok(true));
|
||||
}
|
||||
|
||||
pub(crate) fn fail(&self, reason: impl Into<String>) {
|
||||
self.finish(Err(ToolError::not_available(reason.into())));
|
||||
}
|
||||
|
||||
pub(crate) fn no_backend(&self) {
|
||||
self.finish(Ok(false));
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for SessionCheckpointRequest {
|
||||
fn drop(&mut self) {
|
||||
if let Some(reply) = self
|
||||
.reply
|
||||
.get_mut()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.take()
|
||||
{
|
||||
let _ = reply.send(Err(ToolError::not_available(
|
||||
"owning host did not acknowledge the session checkpoint",
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Events emitted by the engine to update the UI.
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum Event {
|
||||
@@ -717,6 +880,7 @@ pub enum Event {
|
||||
/// text block, and that assistant message still has to be persisted for
|
||||
/// later `reasoning_content` replay.
|
||||
SessionUpdated {
|
||||
checkpoint: Option<Arc<SessionCheckpointRequest>>,
|
||||
session_id: String,
|
||||
/// Shared history snapshot (#6214 T2): the engine hands out an `Arc`
|
||||
/// instead of deep-copying the transcript per event.
|
||||
|
||||
@@ -964,6 +964,7 @@ pub fn event_to_protocol(event: &Event, ids: &ProtocolIds) -> wire::EventMsg {
|
||||
request: user_input_to_wire(request),
|
||||
},
|
||||
Event::SessionUpdated {
|
||||
checkpoint: _,
|
||||
session_id: engine_session_id,
|
||||
messages,
|
||||
system_prompt,
|
||||
|
||||
+413
-40
@@ -287,6 +287,103 @@ pub(crate) fn exec_automation_services(
|
||||
/// tool-call markup that the engine stripped from the visible output.
|
||||
pub(crate) const ONE_SHOT_TOOL_CALL_NOTICE: &str = "codewhale exec: the model tried to call a tool, but this run offers none, so the tool call was removed from the answer. Re-run with --auto (or --allowed-tools) to let it use tools.";
|
||||
|
||||
async fn persist_exec_checkpoint(
|
||||
request: Arc<crate::core::events::SessionCheckpointRequest>,
|
||||
manager: Option<Arc<SessionManager>>,
|
||||
session_id: &str,
|
||||
workspace: &Path,
|
||||
turn_id: Option<&str>,
|
||||
engine: &crate::core::engine::EngineHandle,
|
||||
deadline: Instant,
|
||||
) {
|
||||
use crate::core::events::{
|
||||
SessionCheckpointProjection, SessionCheckpointReceipt, SessionCheckpointRequest,
|
||||
};
|
||||
|
||||
if turn_id != Some(request.caller.turn_id.as_str()) {
|
||||
request.fail("exec session checkpoint does not match its active turn");
|
||||
return;
|
||||
}
|
||||
if let Err(error) = request.validate_prefix(session_id, workspace, &request.snapshot.messages) {
|
||||
request.fail(error);
|
||||
return;
|
||||
}
|
||||
let turn_cancel = engine.captured_turn_cancel();
|
||||
if turn_cancel.is_cancelled() {
|
||||
request.fail("exec session checkpoint cancelled before persistence");
|
||||
return;
|
||||
}
|
||||
if Instant::now() >= deadline {
|
||||
request.fail("exec session checkpoint deadline expired before persistence");
|
||||
engine.cancel_with_reason(crate::core::engine::CancelReason::External);
|
||||
return;
|
||||
}
|
||||
let Some(manager) = manager else {
|
||||
request.no_backend();
|
||||
return;
|
||||
};
|
||||
let write_request = Arc::clone(&request);
|
||||
let mut writer = tokio::task::spawn_blocking(move || -> Result<SessionCheckpointReceipt> {
|
||||
let snapshot = &write_request.snapshot;
|
||||
let id = persist_exec_session(
|
||||
&snapshot.messages,
|
||||
&snapshot.model,
|
||||
PersistedProviderRoute {
|
||||
kind: &snapshot.model_provider,
|
||||
id: snapshot.model_provider_id.as_deref(),
|
||||
},
|
||||
&snapshot.workspace,
|
||||
&snapshot.system_prompt,
|
||||
Some(&snapshot.session_id),
|
||||
snapshot.total_tokens,
|
||||
None,
|
||||
manager.as_ref(),
|
||||
&[],
|
||||
Some(write_request.as_ref()),
|
||||
)?;
|
||||
let persisted = manager
|
||||
.load_session_snapshot(&id)
|
||||
.context("could not read back exec session checkpoint")?;
|
||||
write_request
|
||||
.validate_prefix(
|
||||
&persisted.metadata.id,
|
||||
&persisted.metadata.workspace,
|
||||
&persisted.messages,
|
||||
)
|
||||
.map_err(anyhow::Error::msg)?;
|
||||
Ok(SessionCheckpointReceipt {
|
||||
persisted_sha256: SessionCheckpointRequest::prefix_hash(&persisted.messages)
|
||||
.map_err(anyhow::Error::msg)?,
|
||||
messages_len: persisted.messages.len(),
|
||||
projection: SessionCheckpointProjection::Exact,
|
||||
})
|
||||
});
|
||||
let result = tokio::select! {
|
||||
biased;
|
||||
() = request.withdraw.cancelled() => {
|
||||
request.fail("exec session checkpoint was withdrawn; its write may have completed");
|
||||
writer.await
|
||||
}
|
||||
() = turn_cancel.cancelled() => {
|
||||
request.fail("exec session checkpoint was cancelled; its write may have completed");
|
||||
request.withdraw.cancel();
|
||||
writer.await
|
||||
}
|
||||
() = tokio::time::sleep_until(deadline.into()) => {
|
||||
request.fail("exec session checkpoint deadline expired; its write may have completed");
|
||||
request.withdraw.cancel();
|
||||
engine.cancel_with_reason(crate::core::engine::CancelReason::External);
|
||||
writer.await
|
||||
}
|
||||
result = &mut writer => result,
|
||||
};
|
||||
match result {
|
||||
Ok(Ok(receipt)) => request.succeed(receipt),
|
||||
Ok(Err(error)) => request.fail(format!("exec session checkpoint failed: {error:#}")),
|
||||
Err(error) => request.fail(format!("exec session checkpoint writer failed: {error}")),
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub(crate) async fn run_exec_agent(
|
||||
config: &Config,
|
||||
@@ -376,6 +473,40 @@ pub(crate) async fn run_exec_agent(
|
||||
_ => anyhow::bail!("incomplete Fleet session capture destination"),
|
||||
};
|
||||
|
||||
let exec_session_id = fleet_capture
|
||||
.as_ref()
|
||||
.map(|(id, _)| id.clone())
|
||||
.or_else(|| {
|
||||
resume_session
|
||||
.as_ref()
|
||||
.map(|saved| saved.metadata.id.clone())
|
||||
})
|
||||
.unwrap_or_else(|| uuid::Uuid::new_v4().to_string());
|
||||
let exec_session_workspace = resume_session
|
||||
.as_ref()
|
||||
.map(|saved| saved.metadata.workspace.clone())
|
||||
.unwrap_or_else(|| workspace.clone());
|
||||
let mut should_persist_session = !one_shot
|
||||
|| resume_session.is_some()
|
||||
|| fleet_capture.is_some()
|
||||
|| output_format == ExecOutputFormat::StreamJson;
|
||||
let mut exec_session_manager = fleet_capture.map(|(_, manager)| Arc::new(manager));
|
||||
let mut open_exec_session_manager = || -> Result<Arc<SessionManager>> {
|
||||
if exec_session_manager.is_none() {
|
||||
exec_session_manager = Some(Arc::new(
|
||||
SessionManager::default_location().context("could not open exec session store")?,
|
||||
));
|
||||
}
|
||||
Ok(Arc::clone(
|
||||
exec_session_manager
|
||||
.as_ref()
|
||||
.expect("opened exec session store"),
|
||||
))
|
||||
};
|
||||
if should_persist_session {
|
||||
open_exec_session_manager()?;
|
||||
}
|
||||
|
||||
let route = resolve_cli_exec_route(config, model, prompt, force_configured_route).await?;
|
||||
let execution_config = config_for_cli_route(config, &route)?;
|
||||
crate::provider_lake::refresh_account_catalog_before_first_request(&execution_config).await;
|
||||
@@ -566,14 +697,8 @@ pub(crate) async fn run_exec_agent(
|
||||
model: effective_model.clone(),
|
||||
active_route_limits,
|
||||
workspace: workspace.clone(),
|
||||
session_id: fleet_capture
|
||||
.as_ref()
|
||||
.map(|(id, _)| id.clone())
|
||||
.or_else(|| {
|
||||
resume_session
|
||||
.as_ref()
|
||||
.map(|saved| saved.metadata.id.clone())
|
||||
}),
|
||||
session_id: Some(exec_session_id.clone()),
|
||||
session_checkpoint_backend: Some(crate::core::events::SessionCheckpointProjection::Exact),
|
||||
subagent_state_root: None,
|
||||
plugin_registry: Some(std::sync::Arc::clone(&engine_plugin_registry)),
|
||||
allow_shell: exec_allow_shell,
|
||||
@@ -741,7 +866,6 @@ pub(crate) async fn run_exec_agent(
|
||||
eprintln!("warning: {sandbox_posture} — shell commands will run unrestricted");
|
||||
}
|
||||
|
||||
let resuming_session = resume_session.is_some();
|
||||
let mut loaded_session_id = None;
|
||||
if let Some(saved) = resume_session {
|
||||
let saved_id = saved.metadata.id.clone();
|
||||
@@ -792,8 +916,7 @@ pub(crate) async fn run_exec_agent(
|
||||
)
|
||||
})
|
||||
.unwrap_or_else(codewhale_hooks::LifecycleOutbox::disabled);
|
||||
// Wall clock for the outbox `turn_end` duration. `exec` never receives
|
||||
// a TurnStarted engine event, so the start is marked at the same
|
||||
// Wall clock for the outbox `turn_end` duration starts at the same
|
||||
// `Op::SendMessage` boundary where `turn_start` is emitted below.
|
||||
let exec_turn_started_at = Instant::now();
|
||||
|
||||
@@ -828,9 +951,8 @@ pub(crate) async fn run_exec_agent(
|
||||
}))
|
||||
.await?;
|
||||
|
||||
// Lifecycle outbox: the clean headless turn-start boundary. `exec` has
|
||||
// no TurnStarted engine event; the message submission above is exactly
|
||||
// where the engine begins the turn. No-op when the feature is disabled.
|
||||
// Lifecycle outbox: the headless submission boundary, before the Engine
|
||||
// reports TurnStarted. No-op when the feature is disabled.
|
||||
lifecycle_outbox.emit(codewhale_hooks::LifecycleEvent {
|
||||
event: "turn_start".to_string(),
|
||||
kind: "turn.started".to_string(),
|
||||
@@ -861,8 +983,7 @@ pub(crate) async fn run_exec_agent(
|
||||
let mut last_error_category = None;
|
||||
let mut reported_sandbox_contract = false;
|
||||
|
||||
let mut should_persist_session =
|
||||
!one_shot || resuming_session || output_format == ExecOutputFormat::StreamJson;
|
||||
let mut active_turn_id = None;
|
||||
let mut image_request_receipts = Vec::new();
|
||||
let mut latest_session_id = loaded_session_id;
|
||||
let mut latest_messages: Arc<Vec<Message>> = Arc::new(Vec::new());
|
||||
@@ -891,6 +1012,9 @@ pub(crate) async fn run_exec_agent(
|
||||
};
|
||||
|
||||
match event {
|
||||
Event::TurnStarted { turn_id, .. } => {
|
||||
active_turn_id = Some(turn_id);
|
||||
}
|
||||
Event::MessageDelta { content, .. } => {
|
||||
summary.output.push_str(&content);
|
||||
if output_format == ExecOutputFormat::StreamJson {
|
||||
@@ -1333,30 +1457,34 @@ pub(crate) async fn run_exec_agent(
|
||||
last_error_category.map(|category| category.to_string());
|
||||
}
|
||||
let saved_session_id = if should_persist_session && !latest_messages.is_empty() {
|
||||
match persist_exec_session(
|
||||
&latest_messages,
|
||||
&latest_model,
|
||||
PersistedProviderRoute {
|
||||
kind: effective_identity.persisted_kind(),
|
||||
id: effective_identity.persisted_id(),
|
||||
},
|
||||
&latest_workspace,
|
||||
&latest_system_prompt,
|
||||
latest_session_id.as_deref(),
|
||||
u64::from(usage.input_tokens) + u64::from(usage.output_tokens),
|
||||
(terminal_status == crate::core::events::TurnOutcomeStatus::Failed)
|
||||
.then_some(summary.error.as_deref().unwrap_or_else(|| {
|
||||
crate::core::termination::classify_turn_termination(
|
||||
terminal_status,
|
||||
last_error_category,
|
||||
tool_error_seen,
|
||||
approval_required,
|
||||
)
|
||||
.as_str()
|
||||
})),
|
||||
fleet_capture.as_ref().map(|(_, manager)| manager),
|
||||
&image_request_receipts,
|
||||
) {
|
||||
let saved = open_exec_session_manager().and_then(|manager| {
|
||||
persist_exec_session(
|
||||
&latest_messages,
|
||||
&latest_model,
|
||||
PersistedProviderRoute {
|
||||
kind: effective_identity.persisted_kind(),
|
||||
id: effective_identity.persisted_id(),
|
||||
},
|
||||
&latest_workspace,
|
||||
&latest_system_prompt,
|
||||
latest_session_id.as_deref(),
|
||||
u64::from(usage.input_tokens) + u64::from(usage.output_tokens),
|
||||
(terminal_status == crate::core::events::TurnOutcomeStatus::Failed)
|
||||
.then_some(summary.error.as_deref().unwrap_or_else(|| {
|
||||
crate::core::termination::classify_turn_termination(
|
||||
terminal_status,
|
||||
last_error_category,
|
||||
tool_error_seen,
|
||||
approval_required,
|
||||
)
|
||||
.as_str()
|
||||
})),
|
||||
manager.as_ref(),
|
||||
&image_request_receipts,
|
||||
None,
|
||||
)
|
||||
});
|
||||
match saved {
|
||||
Ok(id) => {
|
||||
if output_format == ExecOutputFormat::Text && !json_output {
|
||||
eprintln!("{}", exec_saved_session_line(&id));
|
||||
@@ -1538,7 +1666,47 @@ pub(crate) async fn run_exec_agent(
|
||||
instruction_assembly: _,
|
||||
model,
|
||||
workspace,
|
||||
checkpoint,
|
||||
} => {
|
||||
if let Some(checkpoint) = checkpoint {
|
||||
if session_id != exec_session_id
|
||||
|| workspace != exec_session_workspace
|
||||
|| active_turn_id.as_deref() != Some(checkpoint.caller.turn_id.as_str())
|
||||
|| model != checkpoint.snapshot.model
|
||||
|| system_prompt != checkpoint.snapshot.system_prompt
|
||||
{
|
||||
checkpoint
|
||||
.fail("exec checkpoint event does not match its captured root turn");
|
||||
continue;
|
||||
}
|
||||
if let Err(error) =
|
||||
checkpoint.validate_prefix(&session_id, &workspace, &messages)
|
||||
{
|
||||
checkpoint.fail(error);
|
||||
continue;
|
||||
}
|
||||
let manager = if should_persist_session {
|
||||
open_exec_session_manager().map(Some)
|
||||
} else {
|
||||
Ok(None)
|
||||
};
|
||||
match manager {
|
||||
Ok(manager) => {
|
||||
persist_exec_checkpoint(
|
||||
checkpoint,
|
||||
manager,
|
||||
&exec_session_id,
|
||||
&exec_session_workspace,
|
||||
active_turn_id.as_deref(),
|
||||
&engine_handle,
|
||||
exec_deadline,
|
||||
)
|
||||
.await
|
||||
}
|
||||
Err(error) => checkpoint
|
||||
.fail(format!("could not open exec checkpoint store: {error:#}")),
|
||||
}
|
||||
}
|
||||
latest_session_id = Some(session_id);
|
||||
latest_messages = messages;
|
||||
latest_system_prompt = system_prompt;
|
||||
@@ -1827,6 +1995,211 @@ mod tests {
|
||||
engine.rx_op.try_recv().is_err(),
|
||||
"the event reader does not send early Shutdown"
|
||||
);
|
||||
|
||||
use crate::core::events::{
|
||||
SessionCheckpointCaller, SessionCheckpointProjection, SessionCheckpointRequest,
|
||||
};
|
||||
use crate::core::ops::SessionSnapshot;
|
||||
use std::sync::Arc;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let manager = Arc::new(
|
||||
crate::session_manager::SessionManager::new(tmp.path().join("fleet-capture")).unwrap(),
|
||||
);
|
||||
let workspace = tmp.path().join("workspace");
|
||||
let messages: Vec<codewhale_models::Message> = serde_json::from_value(serde_json::json!([
|
||||
{"role":"user","content":[{"type":"text","text":"review the changes"}]},
|
||||
{"role":"assistant","content":[{"type":"tool_use","id":"provider-call","execution_id":"exec-call","name":"review","input":{}}]}
|
||||
])).unwrap();
|
||||
let snapshot = Arc::new(SessionSnapshot {
|
||||
session_id: "exec-checkpoint".into(),
|
||||
messages: messages.clone(),
|
||||
total_tokens: 23,
|
||||
model: "review-model".into(),
|
||||
model_provider: "custom".into(),
|
||||
model_provider_id: Some("review-route".into()),
|
||||
workspace: workspace.clone(),
|
||||
system_prompt: None,
|
||||
instruction_assembly: None,
|
||||
mode: "exec".into(),
|
||||
});
|
||||
let caller = SessionCheckpointCaller {
|
||||
session_id: snapshot.session_id.clone(),
|
||||
agent_id: None,
|
||||
turn_id: "exec-turn".into(),
|
||||
call_id: "exec-call".into(),
|
||||
workspace: workspace.clone(),
|
||||
};
|
||||
let checkpoint = |caller: SessionCheckpointCaller, snapshot: Arc<SessionSnapshot>| {
|
||||
let (reply, receive) = tokio::sync::oneshot::channel();
|
||||
let hash = SessionCheckpointRequest::prefix_hash(&snapshot.messages).unwrap();
|
||||
(
|
||||
SessionCheckpointRequest::new(
|
||||
caller,
|
||||
snapshot,
|
||||
hash,
|
||||
SessionCheckpointProjection::Exact,
|
||||
CancellationToken::new(),
|
||||
reply,
|
||||
),
|
||||
receive,
|
||||
)
|
||||
};
|
||||
let (request, receive) = checkpoint(caller.clone(), Arc::clone(&snapshot));
|
||||
super::persist_exec_checkpoint(
|
||||
Arc::clone(&request),
|
||||
Some(Arc::clone(&manager)),
|
||||
&snapshot.session_id,
|
||||
&workspace,
|
||||
Some("exec-turn"),
|
||||
&engine.handle,
|
||||
Instant::now() + Duration::from_secs(3),
|
||||
)
|
||||
.await;
|
||||
assert!(receive.await.unwrap().unwrap());
|
||||
let persisted = manager.load_session_snapshot(&snapshot.session_id).unwrap();
|
||||
assert_eq!(persisted.messages, messages);
|
||||
assert_eq!(persisted.metadata.total_tokens, 23);
|
||||
assert_eq!(
|
||||
persisted.metadata.model_provider_id.as_deref(),
|
||||
Some("review-route")
|
||||
);
|
||||
assert_eq!(persisted.metadata.workspace, workspace);
|
||||
assert_eq!(persisted.metadata.mode.as_deref(), Some("exec"));
|
||||
|
||||
let mut completed = messages.clone();
|
||||
completed.push(serde_json::from_value(serde_json::json!({
|
||||
"role":"user","content":[{"type":"tool_result","tool_use_id":"provider-call","execution_id":"exec-call","content":"review complete"}]
|
||||
})).unwrap());
|
||||
crate::persist_exec_session(
|
||||
&completed,
|
||||
&snapshot.model,
|
||||
crate::PersistedProviderRoute {
|
||||
kind: "custom",
|
||||
id: Some("review-route"),
|
||||
},
|
||||
&workspace,
|
||||
&None,
|
||||
Some(&snapshot.session_id),
|
||||
42,
|
||||
None,
|
||||
manager.as_ref(),
|
||||
&[],
|
||||
None,
|
||||
)
|
||||
.unwrap();
|
||||
super::persist_exec_checkpoint(
|
||||
request,
|
||||
Some(Arc::clone(&manager)),
|
||||
&snapshot.session_id,
|
||||
&workspace,
|
||||
Some("exec-turn"),
|
||||
&engine.handle,
|
||||
Instant::now() + Duration::from_secs(3),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(
|
||||
manager
|
||||
.load_session_snapshot(&snapshot.session_id)
|
||||
.unwrap()
|
||||
.messages,
|
||||
completed
|
||||
);
|
||||
|
||||
for mismatch in ["session", "workspace", "child", "turn"] {
|
||||
let mut foreign = caller.clone();
|
||||
match mismatch {
|
||||
"session" => foreign.session_id = "foreign-session".into(),
|
||||
"workspace" => foreign.workspace = tmp.path().join("foreign-workspace"),
|
||||
"child" => foreign.agent_id = Some("child-agent".into()),
|
||||
"turn" => foreign.turn_id = "foreign-turn".into(),
|
||||
_ => unreachable!(),
|
||||
}
|
||||
let (request, receive) = checkpoint(foreign, Arc::clone(&snapshot));
|
||||
super::persist_exec_checkpoint(
|
||||
request,
|
||||
Some(Arc::clone(&manager)),
|
||||
&snapshot.session_id,
|
||||
&workspace,
|
||||
Some("exec-turn"),
|
||||
&engine.handle,
|
||||
Instant::now() + Duration::from_secs(3),
|
||||
)
|
||||
.await;
|
||||
assert!(receive.await.unwrap().is_err(), "{mismatch}");
|
||||
}
|
||||
assert_eq!(
|
||||
manager
|
||||
.load_session_snapshot(&snapshot.session_id)
|
||||
.unwrap()
|
||||
.messages,
|
||||
completed
|
||||
);
|
||||
assert!(!manager.sessions_dir().join("foreign-session.json").exists());
|
||||
|
||||
let (request, receive) = checkpoint(caller.clone(), Arc::clone(&snapshot));
|
||||
request.withdraw.cancel();
|
||||
super::persist_exec_checkpoint(
|
||||
request,
|
||||
Some(Arc::clone(&manager)),
|
||||
&snapshot.session_id,
|
||||
&workspace,
|
||||
Some("exec-turn"),
|
||||
&engine.handle,
|
||||
Instant::now() + Duration::from_secs(3),
|
||||
)
|
||||
.await;
|
||||
assert!(receive.await.unwrap().is_err());
|
||||
let (request, receive) = checkpoint(caller.clone(), Arc::clone(&snapshot));
|
||||
super::persist_exec_checkpoint(
|
||||
request,
|
||||
None,
|
||||
&snapshot.session_id,
|
||||
&workspace,
|
||||
Some("exec-turn"),
|
||||
&engine.handle,
|
||||
Instant::now() + Duration::from_secs(3),
|
||||
)
|
||||
.await;
|
||||
assert!(!receive.await.unwrap().unwrap());
|
||||
|
||||
let failed_store = Arc::new(
|
||||
crate::session_manager::SessionManager::new(tmp.path().join("failed-store")).unwrap(),
|
||||
);
|
||||
std::fs::create_dir(failed_store.sessions_dir().join("exec-checkpoint.json")).unwrap();
|
||||
let (request, receive) = checkpoint(caller.clone(), Arc::clone(&snapshot));
|
||||
super::persist_exec_checkpoint(
|
||||
request,
|
||||
Some(failed_store),
|
||||
&snapshot.session_id,
|
||||
&workspace,
|
||||
Some("exec-turn"),
|
||||
&engine.handle,
|
||||
Instant::now() + Duration::from_secs(3),
|
||||
)
|
||||
.await;
|
||||
assert!(receive.await.unwrap().is_err());
|
||||
let (request, receive) = checkpoint(caller, Arc::clone(&snapshot));
|
||||
super::persist_exec_checkpoint(
|
||||
request,
|
||||
Some(Arc::clone(&manager)),
|
||||
&snapshot.session_id,
|
||||
&workspace,
|
||||
Some("exec-turn"),
|
||||
&engine.handle,
|
||||
Instant::now(),
|
||||
)
|
||||
.await;
|
||||
assert!(receive.await.unwrap().is_err());
|
||||
assert!(engine.handle.is_cancelled());
|
||||
assert_eq!(
|
||||
manager
|
||||
.load_session_snapshot(&snapshot.session_id)
|
||||
.unwrap()
|
||||
.messages,
|
||||
completed
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -75,11 +75,15 @@ use serde_json::{Value, json};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
use super::ManagerShared;
|
||||
use super::protocol::{ContentBlockWire, CoreCallParams, RpcErrorWire, ToolResultWire, error_code};
|
||||
use super::protocol::{
|
||||
ContentBlockWire, CoreCallParams, RpcErrorWire, SessionCheckpointParams, ToolResultWire,
|
||||
error_code,
|
||||
};
|
||||
use super::supervisor::HostRequestContext;
|
||||
use super::ticket::{Grant, Presented, Ticket, TicketKind};
|
||||
use super::tier::HostTier;
|
||||
use crate::core::authority::{ToolCategory, get_tool_category_for_call};
|
||||
use crate::core::events::SessionCheckpointCaller;
|
||||
use crate::tools::codemode::{
|
||||
CodemodeInvoker, ExtensionCaller, NestedCallGate, NestedDecision, NestedFailure, PauseClock,
|
||||
};
|
||||
@@ -88,6 +92,7 @@ use codewhale_config::OwnerRef;
|
||||
|
||||
/// The protocol method an invocation ticket is good for.
|
||||
pub(crate) const METHOD: &str = "core/call";
|
||||
const CHECKPOINT_METHOD: &str = "session/checkpoint";
|
||||
/// `core/call`s one invocation may make in all.
|
||||
pub(crate) const MAX_CALLS_PER_INVOCATION: u32 = 50;
|
||||
/// A backstop only: an invocation ends (and revokes its ticket) long before
|
||||
@@ -254,6 +259,9 @@ pub(crate) struct Invocation {
|
||||
plugins: Option<Arc<crate::plugins::PluginRegistry>>,
|
||||
content_hash: String,
|
||||
calls: CodemodeInvoker,
|
||||
gate: NestedCallGate,
|
||||
checkpoint_caller: Option<SessionCheckpointCaller>,
|
||||
checkpoint_lock: tokio::sync::Mutex<()>,
|
||||
/// Fires when the invocation ends or is revoked: withdraws every
|
||||
/// `core/call` still waiting or running for it.
|
||||
cancel: CancellationToken,
|
||||
@@ -328,6 +336,38 @@ impl CoreCalls {
|
||||
ttl: TICKET_TTL,
|
||||
uses: MAX_CALLS_PER_INVOCATION,
|
||||
});
|
||||
let checkpoint_caller = context
|
||||
.execution
|
||||
.session_objects
|
||||
.as_ref()
|
||||
.filter(|session| !session.session_id.is_empty())
|
||||
.zip(
|
||||
context
|
||||
.execution
|
||||
.origin_turn_id
|
||||
.as_ref()
|
||||
.filter(|id| !id.is_empty()),
|
||||
)
|
||||
.filter(|_| context.execution.owner_agent_id.is_none())
|
||||
.map(|(session, turn_id)| SessionCheckpointCaller {
|
||||
session_id: session.session_id.clone(),
|
||||
agent_id: None,
|
||||
turn_id: turn_id.clone(),
|
||||
call_id: call_id.to_string(),
|
||||
workspace: context.workspace.clone(),
|
||||
});
|
||||
let checkpoint_ticket = checkpoint_caller.as_ref().map(|_| {
|
||||
self.tickets.mint(Grant {
|
||||
kind: TicketKind::Invocation,
|
||||
tier,
|
||||
host_generation,
|
||||
owner: owner.clone(),
|
||||
method: CHECKPOINT_METHOD,
|
||||
target: json!({ "call_id": call_id }),
|
||||
ttl: TICKET_TTL,
|
||||
uses: MAX_CALLS_PER_INVOCATION,
|
||||
})
|
||||
});
|
||||
let invocation = Arc::new(Invocation {
|
||||
tier,
|
||||
host_generation,
|
||||
@@ -336,13 +376,23 @@ impl CoreCalls {
|
||||
plugins: context.plugin_registry.clone(),
|
||||
content_hash,
|
||||
calls,
|
||||
gate: gate.clone(),
|
||||
checkpoint_caller,
|
||||
checkpoint_lock: tokio::sync::Mutex::new(()),
|
||||
cancel: CancellationToken::new(),
|
||||
});
|
||||
self.invocations
|
||||
.lock()
|
||||
.expect("invocations lock")
|
||||
.insert(ticket.expose().to_string(), Arc::clone(&invocation));
|
||||
if let Some(checkpoint_ticket) = checkpoint_ticket.as_ref() {
|
||||
self.invocations.lock().expect("invocations lock").insert(
|
||||
checkpoint_ticket.expose().to_string(),
|
||||
Arc::clone(&invocation),
|
||||
);
|
||||
}
|
||||
Some(InvocationGuard {
|
||||
checkpoint_ticket,
|
||||
core_calls: Arc::clone(self),
|
||||
ticket,
|
||||
invocation,
|
||||
@@ -425,6 +475,156 @@ impl CoreCalls {
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn redeem_invocation(
|
||||
&self,
|
||||
shared: &ManagerShared,
|
||||
tier: HostTier,
|
||||
host_generation: u64,
|
||||
owner: &OwnerRef,
|
||||
ticket: &str,
|
||||
method: &str,
|
||||
cx: &HostRequestContext,
|
||||
) -> Result<Arc<Invocation>, RpcErrorWire> {
|
||||
let refuse = |code, message| RpcErrorWire {
|
||||
code,
|
||||
message,
|
||||
data: None,
|
||||
};
|
||||
let redeemed = self.tickets.redeem(&Presented {
|
||||
ticket,
|
||||
kind: TicketKind::Invocation,
|
||||
tier,
|
||||
host_generation,
|
||||
owner,
|
||||
method,
|
||||
target: None,
|
||||
});
|
||||
if let Err(refused) = redeemed {
|
||||
if refused.violation {
|
||||
cx.violation(format!(
|
||||
"too many invalid {method} tickets from one host process"
|
||||
));
|
||||
}
|
||||
return Err(refuse(
|
||||
error_code::REFUSED,
|
||||
refused.reason.describe().to_string(),
|
||||
));
|
||||
}
|
||||
// The redemption proved the ticket is live and ours; the owner must
|
||||
// still be the current one of this tier.
|
||||
let live = shared
|
||||
.registry
|
||||
.lock()
|
||||
.expect("registry lock")
|
||||
.tier_of(owner)
|
||||
== Some(tier);
|
||||
let Some(invocation) = self.invocation(ticket).filter(|_| live) else {
|
||||
return Err(refuse(
|
||||
error_code::REFUSED,
|
||||
format!("the invocation this {method} belongs to has ended"),
|
||||
));
|
||||
};
|
||||
shared
|
||||
.check_selection(
|
||||
invocation
|
||||
.plugins
|
||||
.as_ref()
|
||||
.and_then(|plugins| plugins.caller_selection()),
|
||||
invocation.plugins.as_deref(),
|
||||
&owner.plugin_id,
|
||||
&invocation.content_hash,
|
||||
invocation.scope.as_ref(),
|
||||
)
|
||||
.map_err(|message| refuse(error_code::REFUSED, message))?;
|
||||
if invocation.cancel.is_cancelled() {
|
||||
return Err(refuse(
|
||||
error_code::REFUSED,
|
||||
"the invocation has ended".to_string(),
|
||||
));
|
||||
}
|
||||
Ok(invocation)
|
||||
}
|
||||
|
||||
pub(crate) async fn serve_checkpoint(
|
||||
&self,
|
||||
shared: &ManagerShared,
|
||||
tier: HostTier,
|
||||
host_generation: u64,
|
||||
params: SessionCheckpointParams,
|
||||
cx: HostRequestContext,
|
||||
) -> Result<Value, RpcErrorWire> {
|
||||
let refuse = |code, message| RpcErrorWire {
|
||||
code,
|
||||
message,
|
||||
data: None,
|
||||
};
|
||||
let invocation = self.redeem_invocation(
|
||||
shared,
|
||||
tier,
|
||||
host_generation,
|
||||
¶ms.owner,
|
||||
¶ms.ticket,
|
||||
CHECKPOINT_METHOD,
|
||||
&cx,
|
||||
)?;
|
||||
let caller = invocation.checkpoint_caller.clone().ok_or_else(|| {
|
||||
refuse(
|
||||
error_code::NOT_AVAILABLE,
|
||||
"session checkpoint requires a live root invocation".into(),
|
||||
)
|
||||
})?;
|
||||
let withdraw = invocation.cancel.child_token();
|
||||
let _withdraw_guard = withdraw.clone().drop_guard();
|
||||
let outcome = tokio::select! {
|
||||
biased;
|
||||
() = cx.cancel.cancelled() => Err(crate::tools::spec::ToolError::cancelled("session checkpoint was cancelled by its host")),
|
||||
() = withdraw.cancelled() => Err(crate::tools::spec::ToolError::cancelled("session checkpoint invocation ended")),
|
||||
result = async {
|
||||
let _checkpoint = invocation.checkpoint_lock.lock().await;
|
||||
invocation.gate.checkpoint(caller, withdraw.clone()).await
|
||||
} => result,
|
||||
};
|
||||
if cx.cancel.is_cancelled() || invocation.cancel.is_cancelled() {
|
||||
return Err(refuse(
|
||||
error_code::CANCELLED,
|
||||
"session checkpoint invocation was withdrawn".into(),
|
||||
));
|
||||
}
|
||||
if shared
|
||||
.registry
|
||||
.lock()
|
||||
.expect("registry lock")
|
||||
.tier_of(¶ms.owner)
|
||||
!= Some(tier)
|
||||
{
|
||||
return Err(refuse(
|
||||
error_code::REFUSED,
|
||||
"session checkpoint owner is no longer current".into(),
|
||||
));
|
||||
}
|
||||
shared
|
||||
.check_selection(
|
||||
invocation
|
||||
.plugins
|
||||
.as_ref()
|
||||
.and_then(|plugins| plugins.caller_selection()),
|
||||
invocation.plugins.as_deref(),
|
||||
¶ms.owner.plugin_id,
|
||||
&invocation.content_hash,
|
||||
invocation.scope.as_ref(),
|
||||
)
|
||||
.map_err(|message| refuse(error_code::REFUSED, message))?;
|
||||
outcome.map(Value::Bool).map_err(|error| {
|
||||
let code = if matches!(&error, crate::tools::spec::ToolError::Cancelled { .. }) {
|
||||
error_code::CANCELLED
|
||||
} else {
|
||||
error_code::NOT_AVAILABLE
|
||||
};
|
||||
refuse(code, error.to_string())
|
||||
})
|
||||
}
|
||||
|
||||
/// Redeem `params` presented by `tier`'s host process `host_generation`
|
||||
/// and run the call. An invalid presentation is a refusal, and a burst of
|
||||
/// them a violation `cx` reports.
|
||||
@@ -441,52 +641,15 @@ impl CoreCalls {
|
||||
message,
|
||||
data: None,
|
||||
};
|
||||
let redeemed = self.tickets.redeem(&Presented {
|
||||
ticket: ¶ms.ticket,
|
||||
kind: TicketKind::Invocation,
|
||||
let invocation = self.redeem_invocation(
|
||||
shared,
|
||||
tier,
|
||||
host_generation,
|
||||
owner: ¶ms.owner,
|
||||
method: METHOD,
|
||||
target: None,
|
||||
});
|
||||
if let Err(refused) = redeemed {
|
||||
if refused.violation {
|
||||
cx.violation(
|
||||
"too many invalid core/call tickets from one host process".to_string(),
|
||||
);
|
||||
}
|
||||
return Err(refuse(
|
||||
error_code::REFUSED,
|
||||
refused.reason.describe().to_string(),
|
||||
));
|
||||
}
|
||||
// The redemption proved the ticket is live and ours; the owner must
|
||||
// still be the current one of this tier.
|
||||
let live = shared
|
||||
.registry
|
||||
.lock()
|
||||
.expect("registry lock")
|
||||
.tier_of(¶ms.owner)
|
||||
== Some(tier);
|
||||
let Some(invocation) = self.invocation(¶ms.ticket).filter(|_| live) else {
|
||||
return Err(refuse(
|
||||
error_code::REFUSED,
|
||||
"the invocation this core/call belongs to has ended".to_string(),
|
||||
));
|
||||
};
|
||||
shared
|
||||
.check_selection(
|
||||
invocation
|
||||
.plugins
|
||||
.as_ref()
|
||||
.and_then(|plugins| plugins.caller_selection()),
|
||||
invocation.plugins.as_deref(),
|
||||
¶ms.owner.plugin_id,
|
||||
&invocation.content_hash,
|
||||
invocation.scope.as_ref(),
|
||||
)
|
||||
.map_err(|message| refuse(error_code::REFUSED, message))?;
|
||||
¶ms.owner,
|
||||
¶ms.ticket,
|
||||
METHOD,
|
||||
&cx,
|
||||
)?;
|
||||
// Withdrawn when the host cancels this request, its owner is revoked,
|
||||
// the host exits (`cx.cancel`), or the invocation ends.
|
||||
let withdraw = invocation.cancel.child_token();
|
||||
@@ -587,6 +750,7 @@ pub(crate) fn wire_from_response(response: ToolCallResponse) -> ToolResultWire {
|
||||
pub(crate) struct InvocationGuard {
|
||||
core_calls: Arc<CoreCalls>,
|
||||
ticket: Ticket,
|
||||
checkpoint_ticket: Option<Ticket>,
|
||||
invocation: Arc<Invocation>,
|
||||
}
|
||||
|
||||
@@ -596,6 +760,10 @@ impl InvocationGuard {
|
||||
self.ticket.expose()
|
||||
}
|
||||
|
||||
pub(crate) fn checkpoint_ticket(&self) -> Option<&str> {
|
||||
self.checkpoint_ticket.as_ref().map(Ticket::expose)
|
||||
}
|
||||
|
||||
/// The clock the `tool/call` deadline runs on: paused while one of this
|
||||
/// invocation's core calls waits on the gate.
|
||||
pub(crate) fn clock(&self) -> Arc<Mutex<PauseClock>> {
|
||||
@@ -613,5 +781,8 @@ impl InvocationGuard {
|
||||
impl Drop for InvocationGuard {
|
||||
fn drop(&mut self) {
|
||||
self.core_calls.end(&self.ticket);
|
||||
if let Some(ticket) = self.checkpoint_ticket.as_ref() {
|
||||
self.core_calls.end(ticket);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -55,6 +55,15 @@ struct Server {
|
||||
fn serve(mut requests: mpsc::Receiver<NestedCallRequest>, server: Arc<Server>) {
|
||||
tokio::spawn(async move {
|
||||
while let Some(request) = requests.recv().await {
|
||||
let request = match request {
|
||||
NestedCallRequest::Tool(request) => request,
|
||||
NestedCallRequest::Checkpoint { reply, .. } => {
|
||||
let _ = reply.send(Err(ToolError::not_available(
|
||||
"fixture has no session checkpoint backend",
|
||||
)));
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if request.is_stale() {
|
||||
server.stale.fetch_add(1, Ordering::SeqCst);
|
||||
continue;
|
||||
@@ -786,18 +795,271 @@ async fn a_spoofed_ticket_is_refused_whoever_presents_it_and_a_burst_ends_the_ho
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_ticket_for_a_revoked_owner_is_refused_even_if_it_was_never_revoked() {
|
||||
let f = fixture(true);
|
||||
let mut f = fixture(true);
|
||||
let incomplete = f.begin(7, &f.owner);
|
||||
assert!(incomplete.checkpoint_ticket().is_none());
|
||||
drop(incomplete);
|
||||
f.context = f
|
||||
.context
|
||||
.clone()
|
||||
.with_session_objects(crate::rlm::session::SessionObjectSnapshot::new(
|
||||
"checkpoint-session".into(),
|
||||
"fixture-model".into(),
|
||||
f.context.workspace.clone(),
|
||||
None,
|
||||
Arc::new(Vec::new()),
|
||||
))
|
||||
.with_origin_turn_id("checkpoint-turn");
|
||||
let guard = f.begin(7, &f.owner);
|
||||
let checkpoint_ticket = guard.checkpoint_ticket().unwrap().to_string();
|
||||
let (cx, _violations, _cancel) = HostRequestContext::for_test(2);
|
||||
let unavailable = f
|
||||
.manager
|
||||
.shared
|
||||
.core_calls
|
||||
.serve_checkpoint(
|
||||
&f.manager.shared,
|
||||
HostTier::Plugin,
|
||||
7,
|
||||
super::protocol::SessionCheckpointParams {
|
||||
owner: f.owner.clone(),
|
||||
ticket: checkpoint_ticket.clone(),
|
||||
},
|
||||
cx,
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert_eq!(unavailable.code, error_code::NOT_AVAILABLE);
|
||||
assert!(
|
||||
unavailable
|
||||
.message
|
||||
.contains("fixture has no session checkpoint backend")
|
||||
);
|
||||
let (wrong_method, _) = f
|
||||
.serve(HostTier::Plugin, 7, &f.owner, &checkpoint_ticket)
|
||||
.await;
|
||||
assert!(refused(&wrong_method));
|
||||
for (tier, generation, owner, ticket) in [
|
||||
(
|
||||
HostTier::Plugin,
|
||||
7,
|
||||
f.owner.clone(),
|
||||
guard.ticket().to_string(),
|
||||
),
|
||||
(
|
||||
HostTier::Plugin,
|
||||
7,
|
||||
f.other.clone(),
|
||||
checkpoint_ticket.clone(),
|
||||
),
|
||||
(
|
||||
HostTier::Builtin,
|
||||
7,
|
||||
f.owner.clone(),
|
||||
checkpoint_ticket.clone(),
|
||||
),
|
||||
(
|
||||
HostTier::Plugin,
|
||||
8,
|
||||
f.owner.clone(),
|
||||
checkpoint_ticket.clone(),
|
||||
),
|
||||
] {
|
||||
let (cx, _violations, _cancel) = HostRequestContext::for_test(3);
|
||||
let result = f
|
||||
.manager
|
||||
.shared
|
||||
.core_calls
|
||||
.serve_checkpoint(
|
||||
&f.manager.shared,
|
||||
tier,
|
||||
generation,
|
||||
super::protocol::SessionCheckpointParams { owner, ticket },
|
||||
cx,
|
||||
)
|
||||
.await;
|
||||
assert!(refused(&result), "{result:?}");
|
||||
}
|
||||
let (caller, specs) = f.gate.extension().unwrap();
|
||||
let caller = caller.clone();
|
||||
let specs = specs.to_vec();
|
||||
let (tx, _events) = mpsc::channel(1);
|
||||
let (gate, mut requests) = NestedCallGate::new(None, tx, Duration::from_secs(60));
|
||||
f.gate = gate.for_extension(caller, specs);
|
||||
let pending = f.begin(7, &f.owner);
|
||||
let (cx, _violations, cancel) = HostRequestContext::for_test(4);
|
||||
let checkpoint = f.manager.shared.core_calls.serve_checkpoint(
|
||||
&f.manager.shared,
|
||||
HostTier::Plugin,
|
||||
7,
|
||||
super::protocol::SessionCheckpointParams {
|
||||
owner: f.owner.clone(),
|
||||
ticket: pending.checkpoint_ticket().unwrap().to_string(),
|
||||
},
|
||||
cx,
|
||||
);
|
||||
tokio::pin!(checkpoint);
|
||||
let request = tokio::select! {
|
||||
result = &mut checkpoint => panic!("checkpoint unexpectedly settled: {result:?}"),
|
||||
request = requests.recv() => request.unwrap(),
|
||||
};
|
||||
cancel.cancel();
|
||||
assert_eq!(checkpoint.await.unwrap_err().code, error_code::CANCELLED);
|
||||
let NestedCallRequest::Checkpoint {
|
||||
caller,
|
||||
withdraw,
|
||||
reply,
|
||||
} = request
|
||||
else {
|
||||
panic!("expected typed checkpoint")
|
||||
};
|
||||
assert_eq!(caller.session_id, "checkpoint-session");
|
||||
assert_eq!(caller.turn_id, "checkpoint-turn");
|
||||
assert!(withdraw.is_cancelled());
|
||||
assert!(reply.is_closed());
|
||||
drop(pending);
|
||||
let mut child_context = f.context.clone();
|
||||
child_context.execution.owner_agent_id = Some("child".into());
|
||||
let child = f
|
||||
.manager
|
||||
.shared
|
||||
.core_calls
|
||||
.begin(
|
||||
HostTier::Plugin,
|
||||
7,
|
||||
&f.owner,
|
||||
"child-call",
|
||||
&f.caller(),
|
||||
&child_context,
|
||||
&f.gate,
|
||||
)
|
||||
.unwrap();
|
||||
assert!(child.checkpoint_ticket().is_none());
|
||||
drop(child);
|
||||
// The owner is replaced by a new activation: its token changes, so the
|
||||
// old one the host still holds no longer names the current owner.
|
||||
let ticket = guard.ticket().to_string();
|
||||
f.manager.shared.registry.lock().unwrap().revoke_owner("a");
|
||||
let (result, _) = f.serve(HostTier::Plugin, 7, &f.owner, &ticket).await;
|
||||
assert!(refused(&result), "{result:?}");
|
||||
let (cx, _violations, _cancel) = HostRequestContext::for_test(5);
|
||||
assert!(refused(
|
||||
&f.manager
|
||||
.shared
|
||||
.core_calls
|
||||
.serve_checkpoint(
|
||||
&f.manager.shared,
|
||||
HostTier::Plugin,
|
||||
7,
|
||||
super::protocol::SessionCheckpointParams {
|
||||
owner: f.owner.clone(),
|
||||
ticket: checkpoint_ticket
|
||||
},
|
||||
cx,
|
||||
)
|
||||
.await
|
||||
));
|
||||
drop(guard);
|
||||
assert_eq!(f.manager.shared.core_calls.live_tickets(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_wire_result_is_the_tools_text_and_json_and_says_when_it_was_cut() {
|
||||
use crate::core::events::{
|
||||
SessionCheckpointCaller, SessionCheckpointProjection, SessionCheckpointReceipt,
|
||||
SessionCheckpointRequest,
|
||||
};
|
||||
let snapshot = Arc::new(crate::core::ops::SessionSnapshot {
|
||||
session_id: "session".into(),
|
||||
messages: Vec::new(),
|
||||
total_tokens: 0,
|
||||
model: "fixture".into(),
|
||||
model_provider: "fixture".into(),
|
||||
model_provider_id: None,
|
||||
workspace: "/workspace".into(),
|
||||
system_prompt: None,
|
||||
instruction_assembly: None,
|
||||
mode: "agent".into(),
|
||||
});
|
||||
let hash = SessionCheckpointRequest::prefix_hash(&snapshot.messages).unwrap();
|
||||
let make = |projection| {
|
||||
let (reply, receiver) = tokio::sync::oneshot::channel();
|
||||
(
|
||||
SessionCheckpointRequest::new(
|
||||
SessionCheckpointCaller {
|
||||
session_id: "session".into(),
|
||||
agent_id: None,
|
||||
turn_id: "turn".into(),
|
||||
call_id: "call".into(),
|
||||
workspace: "/workspace".into(),
|
||||
},
|
||||
snapshot.clone(),
|
||||
hash.clone(),
|
||||
projection,
|
||||
tokio_util::sync::CancellationToken::new(),
|
||||
reply,
|
||||
),
|
||||
receiver,
|
||||
)
|
||||
};
|
||||
let (request, mut answer) = make(SessionCheckpointProjection::Exact);
|
||||
request
|
||||
.validate_prefix("session", Path::new("/workspace"), &snapshot.messages)
|
||||
.unwrap();
|
||||
assert!(
|
||||
request
|
||||
.validate_prefix("other", Path::new("/workspace"), &snapshot.messages)
|
||||
.is_err()
|
||||
);
|
||||
request.succeed(SessionCheckpointReceipt {
|
||||
persisted_sha256: hash.clone(),
|
||||
messages_len: 0,
|
||||
projection: SessionCheckpointProjection::Exact,
|
||||
});
|
||||
assert!(answer.try_recv().unwrap().unwrap());
|
||||
assert!(request.is_stale());
|
||||
let (request, mut answer) = make(SessionCheckpointProjection::Exact);
|
||||
request.succeed(SessionCheckpointReceipt {
|
||||
persisted_sha256: "0".repeat(64),
|
||||
messages_len: 0,
|
||||
projection: SessionCheckpointProjection::Exact,
|
||||
});
|
||||
assert!(answer.try_recv().unwrap().is_err());
|
||||
let (request, mut answer) = make(SessionCheckpointProjection::Exact);
|
||||
request.succeed(SessionCheckpointReceipt {
|
||||
persisted_sha256: "0".repeat(64),
|
||||
messages_len: 0,
|
||||
projection: SessionCheckpointProjection::RuntimeReceiptRedacted,
|
||||
});
|
||||
assert!(answer.try_recv().unwrap().is_err());
|
||||
let (request, mut answer) = make(SessionCheckpointProjection::RuntimeReceiptRedacted);
|
||||
request.succeed(SessionCheckpointReceipt {
|
||||
persisted_sha256: "0".repeat(64),
|
||||
messages_len: 0,
|
||||
projection: SessionCheckpointProjection::RuntimeReceiptRedacted,
|
||||
});
|
||||
assert!(answer.try_recv().unwrap().unwrap());
|
||||
let (request, mut answer) = make(SessionCheckpointProjection::Exact);
|
||||
request.withdraw.cancel();
|
||||
request.succeed(SessionCheckpointReceipt {
|
||||
persisted_sha256: hash.clone(),
|
||||
messages_len: 0,
|
||||
projection: SessionCheckpointProjection::Exact,
|
||||
});
|
||||
assert!(answer.try_recv().unwrap().is_err());
|
||||
let (request, mut answer) = make(SessionCheckpointProjection::Exact);
|
||||
request.no_backend();
|
||||
assert!(!answer.try_recv().unwrap().unwrap());
|
||||
let (request, mut answer) = make(SessionCheckpointProjection::Exact);
|
||||
drop(request);
|
||||
assert!(
|
||||
answer
|
||||
.try_recv()
|
||||
.unwrap()
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("did not acknowledge")
|
||||
);
|
||||
let text = |wire: &super::protocol::ToolResultWire| match &wire.content[..] {
|
||||
[ContentBlockWire::Text { text }] => text.clone(),
|
||||
other => panic!("{other:?}"),
|
||||
|
||||
@@ -943,6 +943,24 @@ impl HostEvents for Events {
|
||||
.serve(&shared, self.generation, params, cx)
|
||||
.await
|
||||
}
|
||||
protocol::HostRequest::SessionCheckpoint(params) => {
|
||||
let shared = self
|
||||
.shared
|
||||
.upgrade()
|
||||
.ok_or_else(|| refuse("extension host manager is gone"))?;
|
||||
if shared
|
||||
.tier_runtime(self.tier)
|
||||
.host_generation
|
||||
.load(Ordering::SeqCst)
|
||||
!= self.generation
|
||||
{
|
||||
return Err(refuse("stale host generation"));
|
||||
}
|
||||
shared
|
||||
.core_calls
|
||||
.serve_checkpoint(&shared, self.tier, self.generation, params, cx)
|
||||
.await
|
||||
}
|
||||
protocol::HostRequest::CoreCall(params) => {
|
||||
let Some(shared) = self.shared.upgrade() else {
|
||||
return Err(refuse("extension host manager is gone"));
|
||||
|
||||
@@ -171,6 +171,7 @@ pub const METHODS: &[MethodSpec] = &[
|
||||
row(Direction::HostToCore, "registry/register", true),
|
||||
row(Direction::HostToCore, "registry/unregister", true),
|
||||
row(Direction::HostToCore, "core/call", true),
|
||||
row(Direction::HostToCore, "session/checkpoint", true),
|
||||
MethodSpec {
|
||||
tiers: &[HostTier::Builtin],
|
||||
..row(Direction::HostToCore, "proc/launch", true)
|
||||
@@ -525,6 +526,14 @@ pub struct CoreCallParams {
|
||||
pub input: Value,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[cfg_attr(test, derive(schemars::JsonSchema))]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct SessionCheckpointParams {
|
||||
pub owner: OwnerRef,
|
||||
pub ticket: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[cfg_attr(test, derive(schemars::JsonSchema))]
|
||||
#[serde(deny_unknown_fields)]
|
||||
@@ -711,6 +720,7 @@ pub enum HostRequest {
|
||||
Register(RegisterParams),
|
||||
Unregister(UnregisterParams),
|
||||
CoreCall(CoreCallParams),
|
||||
SessionCheckpoint(SessionCheckpointParams),
|
||||
ExecutionRedeem(ExecutionRedeemParams),
|
||||
ProcClose(ProcSessionParams),
|
||||
NetStart(ProcLaunchParams),
|
||||
@@ -731,6 +741,7 @@ impl HostRequest {
|
||||
Self::Register(params) => ¶ms.owner.plugin_id,
|
||||
Self::Unregister(params) => ¶ms.owner.plugin_id,
|
||||
Self::CoreCall(params) => ¶ms.owner.plugin_id,
|
||||
Self::SessionCheckpoint(params) => ¶ms.owner.plugin_id,
|
||||
Self::ExecutionRedeem(params) => ¶ms.owner.plugin_id,
|
||||
Self::ProcClose(params) => ¶ms.owner.plugin_id,
|
||||
Self::NetClose(params) => ¶ms.owner.plugin_id,
|
||||
@@ -848,6 +859,10 @@ pub fn parse_host_message(value: Value, tier: HostTier) -> Result<HostMessage, P
|
||||
id,
|
||||
request: HostRequest::Unregister(params(&method, p)?),
|
||||
},
|
||||
("session/checkpoint", Some(id)) => HostMessage::Request {
|
||||
id,
|
||||
request: HostRequest::SessionCheckpoint(params(&method, p)?),
|
||||
},
|
||||
("core/call", Some(id)) => HostMessage::Request {
|
||||
id,
|
||||
request: HostRequest::CoreCall(params(&method, p)?),
|
||||
@@ -950,6 +965,9 @@ impl HostMessage {
|
||||
}
|
||||
HostRequest::ExecutionRedeem(p) => request_value(*id, "exec/redeem", to_value(p)),
|
||||
HostRequest::CoreCall(p) => request_value(*id, "core/call", to_value(p)),
|
||||
HostRequest::SessionCheckpoint(p) => {
|
||||
request_value(*id, "session/checkpoint", to_value(p))
|
||||
}
|
||||
HostRequest::ProcClose(p) => request_value(*id, "proc/close", to_value(p)),
|
||||
HostRequest::NetStart(p) => request_value(*id, "net/start", to_value(p)),
|
||||
HostRequest::NetFetch(p) => request_value(*id, "net/fetch", to_value(p)),
|
||||
@@ -1048,6 +1066,8 @@ pub struct ToolCallParams {
|
||||
/// then has no way to ask the core for anything.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub ticket: Option<String>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub checkpoint_ticket: Option<String>,
|
||||
/// Per-invocation identity; never cached at activation or by the host root.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub session_id: Option<String>,
|
||||
|
||||
@@ -49,6 +49,7 @@ fn params_schema(method: &str, generator: &mut SchemaGenerator) -> Schema {
|
||||
"registry/register" => generator.subschema_for::<RegisterParams>(),
|
||||
"registry/unregister" => generator.subschema_for::<UnregisterParams>(),
|
||||
"core/call" => generator.subschema_for::<CoreCallParams>(),
|
||||
"session/checkpoint" => generator.subschema_for::<SessionCheckpointParams>(),
|
||||
"ext/faulted" => generator.subschema_for::<FaultedParams>(),
|
||||
"log" => generator.subschema_for::<LogParams>(),
|
||||
other => panic!("method `{other}` has no params type in the TypeScript generator"),
|
||||
@@ -477,8 +478,8 @@ fn typescript_protocol_is_generated_from_the_rust_types() {
|
||||
|
||||
/// Authority only the core may hold (CURRENT_DECISIONS §26): the event
|
||||
/// authority, the store, approval, secrets and credentials, the turn loop,
|
||||
/// sessions and the prompt. A method whose name mentions any of these is
|
||||
/// refused outright, whatever its reviewed reason.
|
||||
/// sessions and the prompt. Names remain refused except the exact reviewed
|
||||
/// invocation-bound checkpoint request, whose state and writer stay in Core.
|
||||
const CORE_ONLY: &[&str] = &[
|
||||
"event",
|
||||
"store",
|
||||
@@ -496,6 +497,11 @@ const CORE_ONLY: &[&str] = &[
|
||||
/// Every method, with why it gives the host no core authority. Adding a
|
||||
/// method means adding its row here, in review, with that reason.
|
||||
const REVIEWED: &[(&str, &str, &str)] = &[
|
||||
(
|
||||
"host_to_core",
|
||||
"session/checkpoint",
|
||||
"opaque live root-invocation ticket only; Core captures its own prefix and the existing host persistence owner acknowledges the exact request; no host-supplied session selector, state, writer, restore or turn authority",
|
||||
),
|
||||
(
|
||||
"core_to_host",
|
||||
"harness/run",
|
||||
@@ -663,7 +669,11 @@ fn host_protocol_never_gains_core_authority() {
|
||||
let name = spec.name.to_ascii_lowercase();
|
||||
for word in CORE_ONLY {
|
||||
assert!(
|
||||
!name.contains(word),
|
||||
!name.contains(word)
|
||||
|| (*word == "session"
|
||||
&& name == "session/checkpoint"
|
||||
&& spec.direction == Direction::HostToCore
|
||||
&& spec.request),
|
||||
"{} method `{}` reaches for core-only authority (`{word}`); \
|
||||
the extension host must never hold it (CURRENT_DECISIONS §26)",
|
||||
spec.direction.as_str(),
|
||||
|
||||
@@ -1369,6 +1369,11 @@ pub(crate) fn registry_host_request<E: HostEvents + ?Sized>(
|
||||
events.unregister(¶ms);
|
||||
Ok(json!({}))
|
||||
}
|
||||
HostRequest::SessionCheckpoint(_) => Err(RpcErrorWire {
|
||||
code: error_code::REFUSED,
|
||||
message: "session/checkpoint is not served here".to_string(),
|
||||
data: None,
|
||||
}),
|
||||
HostRequest::ExecutionRedeem(_)
|
||||
| HostRequest::CoreCall(_)
|
||||
| HostRequest::ProcLaunch(_)
|
||||
|
||||
@@ -2694,6 +2694,7 @@ async fn ordinary_exit_rejects_requests_from_a_drained_calls_waker() {
|
||||
deadline_ms: 60_000,
|
||||
workspace: None,
|
||||
ticket: None,
|
||||
checkpoint_ticket: None,
|
||||
session_id: None,
|
||||
agent_id: None,
|
||||
origin_turn_id: None,
|
||||
@@ -2754,6 +2755,7 @@ async fn idle_retirement_seals_admission_and_does_not_wait_for_heartbeat() {
|
||||
deadline_ms: 5000,
|
||||
workspace: None,
|
||||
ticket: None,
|
||||
checkpoint_ticket: None,
|
||||
session_id: None,
|
||||
agent_id: None,
|
||||
origin_turn_id: None,
|
||||
@@ -2850,6 +2852,7 @@ async fn two_dirty_teardowns_wait_for_a_live_call_then_replay_without_spending_c
|
||||
deadline_ms: 10000,
|
||||
workspace: None,
|
||||
ticket: None,
|
||||
checkpoint_ticket: None,
|
||||
session_id: None,
|
||||
agent_id: None,
|
||||
origin_turn_id: None,
|
||||
|
||||
@@ -373,6 +373,9 @@ impl ToolSpec for HostToolSpec {
|
||||
// learns where else this process has workspaces.
|
||||
workspace: context.workspace.to_str().map(str::to_owned),
|
||||
ticket: invocation.as_ref().map(|i| i.ticket().to_string()),
|
||||
checkpoint_ticket: invocation
|
||||
.as_ref()
|
||||
.and_then(|i| i.checkpoint_ticket().map(str::to_string)),
|
||||
session_id: context
|
||||
.execution
|
||||
.session_objects
|
||||
|
||||
+13
-10
@@ -15081,19 +15081,12 @@ fn persist_exec_session(
|
||||
session_id: Option<&str>,
|
||||
total_tokens: u64,
|
||||
turn_failure: Option<&str>,
|
||||
session_manager: Option<&SessionManager>,
|
||||
manager: &SessionManager,
|
||||
image_request_receipts: &[serde_json::Value],
|
||||
checkpoint: Option<&crate::core::events::SessionCheckpointRequest>,
|
||||
) -> Result<String> {
|
||||
let default_manager;
|
||||
let manager = if let Some(manager) = session_manager {
|
||||
manager
|
||||
} else {
|
||||
default_manager = SessionManager::default_location()
|
||||
.context("could not open session manager for save")?;
|
||||
&default_manager
|
||||
};
|
||||
let mut saved = if let Some(id) = session_id.filter(|id| !id.trim().is_empty()) {
|
||||
match manager.load_session(id) {
|
||||
match manager.load_session_snapshot(id) {
|
||||
Ok(existing) => session_manager::update_session(
|
||||
existing,
|
||||
messages,
|
||||
@@ -15148,6 +15141,16 @@ fn persist_exec_session(
|
||||
}
|
||||
}
|
||||
}
|
||||
saved.make_storage_compatible();
|
||||
if let Some(checkpoint) = checkpoint {
|
||||
checkpoint
|
||||
.validate_prefix(
|
||||
&saved.metadata.id,
|
||||
&saved.metadata.workspace,
|
||||
&saved.messages,
|
||||
)
|
||||
.map_err(anyhow::Error::msg)?;
|
||||
}
|
||||
let id = saved.metadata.id.clone();
|
||||
manager
|
||||
.save_session(&saved)
|
||||
|
||||
@@ -57,6 +57,15 @@ use thiserror::Error;
|
||||
#[error("{0}")]
|
||||
pub struct PluginNameConflict(pub String);
|
||||
|
||||
#[derive(Debug, Error)]
|
||||
#[error(
|
||||
"plugin source changed after review: expected content hash {expected}, copied bytes hash is {actual}; nothing was installed"
|
||||
)]
|
||||
pub struct PluginContentMismatch {
|
||||
pub expected: String,
|
||||
pub actual: String,
|
||||
}
|
||||
|
||||
use crate::plugins::manifest::PluginManifest;
|
||||
use crate::skills::install::{
|
||||
self as skill_install, FetchOutcome, InstallSource, InstalledFromMarker, fetch_tarball,
|
||||
@@ -546,9 +555,11 @@ fn verify_expected_content_hash(
|
||||
}
|
||||
let actual = staged.content_hash.clone();
|
||||
let _ = fs::remove_dir_all(&staged.staged_path);
|
||||
bail!(
|
||||
"plugin source changed after review: expected content hash {expected}, copied bytes hash is {actual}; nothing was installed"
|
||||
)
|
||||
Err(PluginContentMismatch {
|
||||
expected: expected.to_string(),
|
||||
actual,
|
||||
}
|
||||
.into())
|
||||
}
|
||||
|
||||
/// Stage and finalize an already-downloaded remote tarball. Kept separate
|
||||
|
||||
@@ -326,6 +326,10 @@ async fn exact_local_install_rejects_changed_bytes_before_placement() {
|
||||
.unwrap()
|
||||
.content_hash;
|
||||
fs::write(source.join("README.md"), "changed after review\n").unwrap();
|
||||
let actual =
|
||||
crate::plugins::manifest::PluginManifest::validate_from_path(&source.join("plugin.toml"))
|
||||
.unwrap()
|
||||
.content_hash;
|
||||
|
||||
let error = install_with_expected_content_hash(
|
||||
PluginInstallSource::parse(source.to_str().unwrap()).unwrap(),
|
||||
@@ -338,7 +342,15 @@ async fn exact_local_install_rejects_changed_bytes_before_placement() {
|
||||
.await
|
||||
.unwrap_err();
|
||||
|
||||
assert!(format!("{error:#}").contains("source changed after review"));
|
||||
let mismatch = error.downcast_ref::<PluginContentMismatch>().unwrap();
|
||||
assert_eq!(mismatch.expected, expected);
|
||||
assert_eq!(mismatch.actual, actual);
|
||||
assert_eq!(
|
||||
error.to_string(),
|
||||
format!(
|
||||
"plugin source changed after review: expected content hash {expected}, copied bytes hash is {actual}; nothing was installed"
|
||||
)
|
||||
);
|
||||
assert!(!plugins.join("exact-demo").exists());
|
||||
assert!(
|
||||
fs::read_dir(&plugins).unwrap().all(|entry| !entry
|
||||
|
||||
@@ -416,6 +416,9 @@ async fn run_plugin_mutation(
|
||||
if error
|
||||
.downcast_ref::<crate::plugins::install::PluginNameConflict>()
|
||||
.is_some()
|
||||
|| error
|
||||
.downcast_ref::<crate::plugins::install::PluginContentMismatch>()
|
||||
.is_some()
|
||||
{
|
||||
ApiError::conflict(format!("plugin mutation failed: {error:#}"))
|
||||
} else {
|
||||
|
||||
@@ -478,6 +478,7 @@ fn messages_from_thread_detail_batches_tool_results() {
|
||||
last_read: None,
|
||||
};
|
||||
let turn = TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: 2,
|
||||
@@ -9936,6 +9937,7 @@ async fn session_save_merges_thread_cost_split_and_records_coverage() -> Result<
|
||||
let store = runtime_threads.test_store();
|
||||
let now = Utc::now();
|
||||
let mut turn = TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: 2,
|
||||
@@ -10136,6 +10138,7 @@ async fn session_save_persists_parent_cny_unpriced_reasons_without_double_count(
|
||||
let store = runtime_threads.test_store();
|
||||
let now = Utc::now();
|
||||
let mut turn = TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: 2,
|
||||
@@ -11489,6 +11492,7 @@ fn seed_summary_search_transcript(
|
||||
item_ids.push(item_id);
|
||||
}
|
||||
store.save_turn(&TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: 2,
|
||||
@@ -18676,6 +18680,36 @@ async fn dsh_package_preview_then_exact_install_over_http() -> Result<()> {
|
||||
"preview installs nothing"
|
||||
);
|
||||
|
||||
let wrong_hash = "0".repeat(64);
|
||||
assert_ne!(preview["content_hash"].as_str(), Some(wrong_hash.as_str()));
|
||||
let mismatch = client
|
||||
.post(format!("http://{addr}/v1/apps/plugins/install"))
|
||||
.json(&serde_json::json!({
|
||||
"source": preview["install_source"],
|
||||
"expected_content_hash": wrong_hash.as_str(),
|
||||
}))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(mismatch.status(), StatusCode::CONFLICT);
|
||||
let mismatch: serde_json::Value = mismatch.json().await?;
|
||||
assert_eq!(mismatch["error"]["status"], 409);
|
||||
assert_eq!(
|
||||
mismatch["error"]["message"],
|
||||
format!(
|
||||
"plugin mutation failed: plugin source changed after review: expected content hash {wrong_hash}, copied bytes hash is {}; nothing was installed",
|
||||
preview["content_hash"].as_str().unwrap()
|
||||
)
|
||||
);
|
||||
let plugins_dir = tmp.path().join("runtime/plugins");
|
||||
assert!(!plugins_dir.join("docs-dsh").exists());
|
||||
assert!(fs::read_dir(&plugins_dir)?.all(|entry| {
|
||||
!entry
|
||||
.unwrap()
|
||||
.file_name()
|
||||
.to_string_lossy()
|
||||
.starts_with(".staging-")
|
||||
}));
|
||||
|
||||
let installed = client
|
||||
.post(format!("http://{addr}/v1/apps/plugins/install"))
|
||||
.json(&serde_json::json!({
|
||||
|
||||
@@ -519,6 +519,7 @@ const CURRENT_RUNTIME_SCHEMA_VERSION: u32 = 2;
|
||||
const IMAGE_RUNTIME_SCHEMA_VERSION: u32 = 3;
|
||||
// Explicit allowances need a newer reader so old binaries cannot retry uncapped.
|
||||
const OUTPUT_LIMIT_RUNTIME_SCHEMA_VERSION: u32 = 4;
|
||||
const PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION: u32 = 5;
|
||||
const MAX_SUPPORTED_RUNTIME_SCHEMA_VERSION: u32 = OUTPUT_LIMIT_RUNTIME_SCHEMA_VERSION;
|
||||
|
||||
fn is_zero_u64(value: &u64) -> bool {
|
||||
@@ -1159,20 +1160,19 @@ fn session_recovery_projection(messages: &[Message]) -> Vec<Value> {
|
||||
/// on, and the fork rebuilds the kept exchanges from the newest one it keeps
|
||||
/// plus the raw turns after it (`RuntimeThreadStore::history_from_records`).
|
||||
///
|
||||
/// The compaction names the turn it ran for, so the records answer the
|
||||
/// question directly. Any compaction in the thread is enough: the turns before
|
||||
/// it are the ones whose transcript was replaced, and a cut after them would
|
||||
/// still slice an abbreviation.
|
||||
fn saved_transcript_is_compacted(
|
||||
/// Compaction and active-prefix checkpoints name their owning turn. Either
|
||||
/// requires the canonical record history instead of a saved abbreviation.
|
||||
fn saved_transcript_requires_record_history(
|
||||
turns: &[TurnRecord],
|
||||
items_by_turn: &HashMap<String, Vec<TurnItemRecord>>,
|
||||
) -> bool {
|
||||
turns.iter().any(|turn| {
|
||||
items_by_turn.get(&turn.id).is_some_and(|items| {
|
||||
items
|
||||
.iter()
|
||||
.any(|item| item.kind == TurnItemKind::ContextCompaction)
|
||||
})
|
||||
turn.prefix_checkpoint.is_some()
|
||||
|| items_by_turn.get(&turn.id).is_some_and(|items| {
|
||||
items
|
||||
.iter()
|
||||
.any(|item| item.kind == TurnItemKind::ContextCompaction)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1468,8 +1468,85 @@ pub struct RuntimeOperationActivity {
|
||||
pub overflowed: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub(crate) struct RuntimePrefixCheckpoint {
|
||||
checkpoint_id: String,
|
||||
session_id: String,
|
||||
agent_id: Option<String>,
|
||||
runtime_turn_id: String,
|
||||
engine_turn_id: String,
|
||||
call_id: String,
|
||||
owner_id: String,
|
||||
source_sha256: String,
|
||||
persisted_sha256: String,
|
||||
messages_len: usize,
|
||||
projection: String,
|
||||
item_ids: Vec<String>,
|
||||
covered_items: HashMap<String, String>,
|
||||
pending_items: Vec<String>,
|
||||
tool_declarations: HashMap<String, String>,
|
||||
}
|
||||
|
||||
fn checkpoint_tool_declarations(messages: &[Message]) -> Result<HashMap<String, String>> {
|
||||
let mut declarations = HashMap::new();
|
||||
for block in messages.iter().flat_map(|message| &message.content) {
|
||||
if let ContentBlock::ToolUse {
|
||||
execution_id, id, ..
|
||||
} = block
|
||||
{
|
||||
declarations.insert(
|
||||
serde_json::to_string(&(execution_id, id))?,
|
||||
hex_digest(Sha256::digest(serde_json::to_vec(block)?)),
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok(declarations)
|
||||
}
|
||||
|
||||
fn checkpoint_item_digest(item: &TurnItemRecord) -> Result<String> {
|
||||
Ok(hex_digest(Sha256::digest(serde_json::to_vec(&json!({
|
||||
"kind": item.kind,
|
||||
"summary": item.summary,
|
||||
"detail": item.detail,
|
||||
"metadata": item.metadata,
|
||||
}))?)))
|
||||
}
|
||||
|
||||
fn ordered_turn_items(
|
||||
turn: &TurnRecord,
|
||||
items_by_turn: &HashMap<String, Vec<TurnItemRecord>>,
|
||||
) -> Result<Vec<TurnItemRecord>> {
|
||||
let stored = items_by_turn.get(&turn.id).cloned().unwrap_or_default();
|
||||
let mut by_id = HashMap::new();
|
||||
for item in &stored {
|
||||
anyhow::ensure!(
|
||||
item.turn_id == turn.id,
|
||||
"Stored item belongs to another turn"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
by_id.insert(item.id.clone(), item.clone()).is_none(),
|
||||
"Duplicate stored item"
|
||||
);
|
||||
}
|
||||
let mut ordered = Vec::with_capacity(stored.len());
|
||||
for id in &turn.item_ids {
|
||||
if let Some(item) = by_id.remove(id) {
|
||||
ordered.push(item);
|
||||
}
|
||||
}
|
||||
for item in stored {
|
||||
if by_id.remove(&item.id).is_some() {
|
||||
ordered.push(item);
|
||||
}
|
||||
}
|
||||
Ok(ordered)
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct TurnRecord {
|
||||
#[serde(default, skip_serializing)]
|
||||
pub(crate) prefix_checkpoint: Option<RuntimePrefixCheckpoint>,
|
||||
/// Admitted per-request allowance. Older turns have no explicit allowance.
|
||||
#[serde(
|
||||
default,
|
||||
@@ -1682,15 +1759,22 @@ impl TurnRecord {
|
||||
{
|
||||
bail!("Turn decision evidence exceeds its bound");
|
||||
}
|
||||
if self.schema_version > MAX_SUPPORTED_RUNTIME_SCHEMA_VERSION {
|
||||
anyhow::ensure!(
|
||||
self.prefix_checkpoint.is_none()
|
||||
|| self.schema_version >= PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION,
|
||||
"Runtime prefix checkpoint requires its recovery schema"
|
||||
);
|
||||
if self.schema_version > PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION {
|
||||
bail!(
|
||||
"Turn schema v{} is newer than supported v{}",
|
||||
self.schema_version,
|
||||
MAX_SUPPORTED_RUNTIME_SCHEMA_VERSION
|
||||
PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION
|
||||
);
|
||||
}
|
||||
if self.max_output_tokens.is_some()
|
||||
!= (self.schema_version == OUTPUT_LIMIT_RUNTIME_SCHEMA_VERSION)
|
||||
if (self.max_output_tokens.is_some()
|
||||
&& self.schema_version < OUTPUT_LIMIT_RUNTIME_SCHEMA_VERSION)
|
||||
|| (self.max_output_tokens.is_none()
|
||||
&& self.schema_version == OUTPUT_LIMIT_RUNTIME_SCHEMA_VERSION)
|
||||
{
|
||||
bail!("Turn output allowance does not match its schema");
|
||||
}
|
||||
@@ -1973,6 +2057,7 @@ fn settle_unaccepted_routed_usage(
|
||||
} else {
|
||||
let now = Utc::now();
|
||||
TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
@@ -3141,7 +3226,11 @@ impl RuntimeThreadStore {
|
||||
turn.validate_output_token_limit()?;
|
||||
validated_record_id(&turn.thread_id, "thread id")?;
|
||||
let path = self.turn_path(&turn.id)?;
|
||||
write_json_atomic(&path, turn).with_context(|| {
|
||||
let mut stored = serde_json::to_value(turn)?;
|
||||
if let Some(checkpoint) = &turn.prefix_checkpoint {
|
||||
stored["prefix_checkpoint"] = serde_json::to_value(checkpoint)?;
|
||||
}
|
||||
write_json_atomic(&path, &stored).with_context(|| {
|
||||
RuntimeStoreRecordFailure::new(
|
||||
RuntimeStoreOperation::Write,
|
||||
RuntimeStoreRecordKind::Turn,
|
||||
@@ -3287,21 +3376,349 @@ impl RuntimeThreadStore {
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
/// The model-visible history the records hold: the newest settled
|
||||
/// history, then every later turn rebuilt from its items.
|
||||
fn load_prefix_history(&self, checkpoint: &RuntimePrefixCheckpoint) -> Result<Vec<Message>> {
|
||||
anyhow::ensure!(
|
||||
checkpoint.persisted_sha256.len() == 64
|
||||
&& checkpoint
|
||||
.persisted_sha256
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_hexdigit())
|
||||
&& checkpoint.projection == "runtime_receipt_redacted",
|
||||
"Invalid Runtime prefix checkpoint reference"
|
||||
);
|
||||
let path = self.settled_history_path(&checkpoint.persisted_sha256)?;
|
||||
reject_symlinked_store_file(&path)?;
|
||||
let bytes = fs::read(&path).context("Runtime prefix checkpoint is unreadable")?;
|
||||
anyhow::ensure!(
|
||||
hex_digest(Sha256::digest(&bytes)) == checkpoint.persisted_sha256,
|
||||
"Runtime prefix checkpoint digest mismatch"
|
||||
);
|
||||
let messages: Vec<Message> = serde_json::from_slice(&bytes)?;
|
||||
anyhow::ensure!(
|
||||
messages.len() == checkpoint.messages_len,
|
||||
"Runtime prefix checkpoint length mismatch"
|
||||
);
|
||||
Ok(messages)
|
||||
}
|
||||
|
||||
fn checkpoint_tail(
|
||||
&self,
|
||||
turn: &TurnRecord,
|
||||
items_by_turn: &HashMap<String, Vec<TurnItemRecord>>,
|
||||
messages: &[Message],
|
||||
) -> Result<HashMap<String, Vec<TurnItemRecord>>> {
|
||||
let checkpoint = turn
|
||||
.prefix_checkpoint
|
||||
.as_ref()
|
||||
.context("Missing Runtime prefix checkpoint")?;
|
||||
anyhow::ensure!(
|
||||
checkpoint.session_id == turn.thread_id
|
||||
&& checkpoint.runtime_turn_id == turn.id
|
||||
&& checkpoint.agent_id.is_none()
|
||||
&& checkpoint.owner_id == self.owner_id
|
||||
&& !checkpoint.engine_turn_id.is_empty()
|
||||
&& !checkpoint.call_id.is_empty()
|
||||
&& !checkpoint.checkpoint_id.is_empty()
|
||||
&& checkpoint.source_sha256.len() == 64
|
||||
&& checkpoint
|
||||
.source_sha256
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_hexdigit()),
|
||||
"Runtime prefix checkpoint owner mismatch"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
turn.item_ids.starts_with(&checkpoint.item_ids)
|
||||
&& checkpoint.item_ids.iter().collect::<HashSet<_>>().len()
|
||||
== checkpoint.item_ids.len()
|
||||
&& checkpoint
|
||||
.covered_items
|
||||
.keys()
|
||||
.all(|id| checkpoint.item_ids.contains(id))
|
||||
&& checkpoint
|
||||
.pending_items
|
||||
.iter()
|
||||
.all(|id| checkpoint.item_ids.contains(id)
|
||||
&& !checkpoint.covered_items.contains_key(id)),
|
||||
"Runtime prefix checkpoint item cursor changed"
|
||||
);
|
||||
let mut calls = HashMap::<(Option<String>, String), ContentBlock>::new();
|
||||
let mut results = HashMap::<(Option<String>, String), ContentBlock>::new();
|
||||
let mut call_order = Vec::new();
|
||||
let mut execution_providers = HashMap::new();
|
||||
for block in messages.iter().flat_map(|message| &message.content) {
|
||||
match block {
|
||||
ContentBlock::ToolUse {
|
||||
execution_id, id, ..
|
||||
} => {
|
||||
check_execution_provider(
|
||||
&mut execution_providers,
|
||||
execution_id.as_deref(),
|
||||
id,
|
||||
)?;
|
||||
let key = (execution_id.clone(), id.clone());
|
||||
anyhow::ensure!(
|
||||
!calls.contains_key(&key) || results.contains_key(&key),
|
||||
"Checkpoint contains duplicate open tool identity"
|
||||
);
|
||||
results.remove(&key);
|
||||
call_order.retain(|previous| previous != &key);
|
||||
call_order.push(key.clone());
|
||||
calls.insert(key, block.clone());
|
||||
}
|
||||
ContentBlock::ToolResult {
|
||||
execution_id,
|
||||
tool_use_id,
|
||||
..
|
||||
} => {
|
||||
check_execution_provider(
|
||||
&mut execution_providers,
|
||||
execution_id.as_deref(),
|
||||
tool_use_id,
|
||||
)?;
|
||||
let key = (execution_id.clone(), tool_use_id.clone());
|
||||
anyhow::ensure!(
|
||||
calls.contains_key(&key) && results.insert(key, block.clone()).is_none(),
|
||||
"Checkpoint contains an unpaired tool result"
|
||||
);
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
anyhow::ensure!(
|
||||
calls.len() == checkpoint.tool_declarations.len()
|
||||
&& calls
|
||||
.keys()
|
||||
.all(
|
||||
|key| serde_json::to_string(key).ok().is_some_and(|key| checkpoint
|
||||
.tool_declarations
|
||||
.get(&key)
|
||||
.is_some_and(|digest| digest.len() == 64
|
||||
&& digest.bytes().all(|byte| byte.is_ascii_hexdigit())))
|
||||
),
|
||||
"Runtime prefix checkpoint tool declarations are incomplete"
|
||||
);
|
||||
let items = ordered_turn_items(turn, items_by_turn)?;
|
||||
let present: HashSet<_> = items.iter().map(|item| item.id.as_str()).collect();
|
||||
anyhow::ensure!(
|
||||
checkpoint
|
||||
.item_ids
|
||||
.iter()
|
||||
.all(|id| present.contains(id.as_str())),
|
||||
"Runtime prefix checkpoint item is missing"
|
||||
);
|
||||
let mut retained = Vec::new();
|
||||
let mut answered = HashSet::new();
|
||||
for mut item in items {
|
||||
if let Some(expected) = checkpoint.covered_items.get(&item.id) {
|
||||
anyhow::ensure!(
|
||||
checkpoint_item_digest(&item)? == *expected
|
||||
&& (projects_tool_history(&item)
|
||||
|| !matches!(
|
||||
item.status,
|
||||
TurnItemLifecycleStatus::Queued | TurnItemLifecycleStatus::Canceled
|
||||
)),
|
||||
"Runtime prefix checkpoint covered item changed"
|
||||
);
|
||||
if !projects_tool_history(&item) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
anyhow::ensure!(
|
||||
!checkpoint.item_ids.contains(&item.id)
|
||||
|| checkpoint.pending_items.contains(&item.id)
|
||||
|| !matches!(
|
||||
item.kind,
|
||||
TurnItemKind::UserMessage
|
||||
| TurnItemKind::AgentMessage
|
||||
| TurnItemKind::AgentReasoning
|
||||
)
|
||||
|| matches!(
|
||||
item.status,
|
||||
TurnItemLifecycleStatus::Queued | TurnItemLifecycleStatus::Canceled
|
||||
),
|
||||
"Runtime prefix checkpoint covered item reference is missing"
|
||||
);
|
||||
if projects_tool_history(&item) {
|
||||
let identity =
|
||||
RuntimeToolIdentity::read(item.metadata.as_ref(), "tool_use_id")?.or(
|
||||
RuntimeToolIdentity::read(item.metadata.as_ref(), "tool_result_for")?,
|
||||
);
|
||||
if let Some(identity) = identity {
|
||||
check_execution_provider(
|
||||
&mut execution_providers,
|
||||
identity.execution_id,
|
||||
identity.provider_id,
|
||||
)?;
|
||||
let key = (
|
||||
identity.execution_id.map(str::to_owned),
|
||||
identity.provider_id.to_owned(),
|
||||
);
|
||||
if calls.contains_key(&key) {
|
||||
let declaration = checkpoint
|
||||
.tool_declarations
|
||||
.get(&serde_json::to_string(&key)?)
|
||||
.context("Runtime prefix tool declaration reference is missing")?;
|
||||
let mut isolated = turn.clone();
|
||||
isolated.item_ids = vec![item.id.clone()];
|
||||
let projected = RuntimeThreadManager::reconstruct_messages_from_turns_with(
|
||||
&[isolated],
|
||||
&HashMap::from([(turn.id.clone(), vec![item.clone()])]),
|
||||
)?;
|
||||
let mut outcome = None;
|
||||
for block in projected.into_iter().flat_map(|message| message.content) {
|
||||
match &block {
|
||||
ContentBlock::ToolUse { .. } => anyhow::ensure!(
|
||||
hex_digest(Sha256::digest(serde_json::to_vec(&block)?))
|
||||
== *declaration,
|
||||
"Runtime prefix tool declaration changed"
|
||||
),
|
||||
ContentBlock::ToolResult { .. } => outcome = Some(block),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
if results.contains_key(&key) {
|
||||
anyhow::ensure!(
|
||||
checkpoint.covered_items.contains_key(&item.id),
|
||||
"Runtime prefix completed tool receipt is not bound to its cursor"
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if let Some(outcome) = outcome {
|
||||
anyhow::ensure!(
|
||||
answered.insert(key.clone()),
|
||||
"Duplicate Runtime prefix tool result"
|
||||
);
|
||||
let ContentBlock::ToolResult {
|
||||
content,
|
||||
is_error,
|
||||
content_blocks,
|
||||
..
|
||||
} = outcome
|
||||
else {
|
||||
unreachable!()
|
||||
};
|
||||
let metadata = item
|
||||
.metadata
|
||||
.as_mut()
|
||||
.and_then(Value::as_object_mut)
|
||||
.context("Invalid Runtime prefix tool metadata")?;
|
||||
metadata.remove("tool_use_id");
|
||||
metadata.insert(
|
||||
"tool_result_for".into(),
|
||||
json!(key.0.as_deref().unwrap_or(&key.1)),
|
||||
);
|
||||
metadata.insert("is_error".into(), json!(is_error.unwrap_or(false)));
|
||||
if let Some(content_blocks) = content_blocks {
|
||||
metadata.insert("content_blocks".into(), json!(content_blocks));
|
||||
}
|
||||
item.detail = Some(content);
|
||||
retained.push(item);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
anyhow::ensure!(
|
||||
!checkpoint.item_ids.contains(&item.id),
|
||||
"Runtime prefix does not contain a covered tool declaration"
|
||||
);
|
||||
}
|
||||
}
|
||||
retained.push(item);
|
||||
}
|
||||
if !turn.status.is_active_work() {
|
||||
let mut missing = Vec::new();
|
||||
for key in call_order
|
||||
.iter()
|
||||
.filter(|key| !results.contains_key(*key) && !answered.contains(*key))
|
||||
{
|
||||
let id = format!("checkpoint_result_{}", missing.len());
|
||||
missing.push(TurnItemRecord {
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id,
|
||||
turn_id: turn.id.clone(),
|
||||
kind: TurnItemKind::ToolCall,
|
||||
status: TurnItemLifecycleStatus::Interrupted,
|
||||
summary: String::new(),
|
||||
detail: Some(
|
||||
codewhale_runtime::tool_history_repair::CRASH_REPAIR_CONTENT.to_string(),
|
||||
),
|
||||
metadata: Some(match &key.0 {
|
||||
Some(execution) => {
|
||||
json!({"tool_result_for": execution, "execution_id": execution,
|
||||
"provider_tool_use_id": key.1, "is_error": true})
|
||||
}
|
||||
None => json!({"tool_result_for": key.1, "is_error": true}),
|
||||
}),
|
||||
artifact_refs: Vec::new(),
|
||||
artifacts: Vec::new(),
|
||||
started_at: None,
|
||||
ended_at: None,
|
||||
});
|
||||
}
|
||||
let boundary = retained
|
||||
.iter()
|
||||
.position(|item| {
|
||||
matches!(
|
||||
item.kind,
|
||||
TurnItemKind::UserMessage
|
||||
| TurnItemKind::AgentMessage
|
||||
| TurnItemKind::AgentReasoning
|
||||
)
|
||||
})
|
||||
.unwrap_or(retained.len());
|
||||
retained.splice(boundary..boundary, missing);
|
||||
}
|
||||
let mut tail = items_by_turn.clone();
|
||||
tail.insert(turn.id.clone(), retained);
|
||||
Ok(tail)
|
||||
}
|
||||
|
||||
fn history_with_prefix(
|
||||
&self,
|
||||
turns: &[TurnRecord],
|
||||
items_by_turn: &HashMap<String, Vec<TurnItemRecord>>,
|
||||
mut messages: Vec<Message>,
|
||||
mut covered: usize,
|
||||
) -> Result<Vec<Message>> {
|
||||
if let Some((settled, history)) = self.settled_history(&turns[covered..], items_by_turn)? {
|
||||
covered += settled;
|
||||
messages = history;
|
||||
}
|
||||
if let Some((index, turn)) = turns
|
||||
.iter()
|
||||
.enumerate()
|
||||
.skip(covered)
|
||||
.rev()
|
||||
.find(|(_, turn)| turn.prefix_checkpoint.is_some())
|
||||
{
|
||||
messages = self.load_prefix_history(turn.prefix_checkpoint.as_ref().unwrap())?;
|
||||
let tail = self.checkpoint_tail(turn, items_by_turn, &messages)?;
|
||||
let mut remaining = turns[index..].to_vec();
|
||||
remaining[0].item_ids = tail
|
||||
.get(&turn.id)
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.map(|item| item.id.clone())
|
||||
.collect();
|
||||
messages.extend(RuntimeThreadManager::reconstruct_messages_from_turns_with(
|
||||
&remaining, &tail,
|
||||
)?);
|
||||
} else {
|
||||
messages.extend(RuntimeThreadManager::reconstruct_messages_from_turns_with(
|
||||
&turns[covered..],
|
||||
items_by_turn,
|
||||
)?);
|
||||
}
|
||||
Ok(messages)
|
||||
}
|
||||
|
||||
/// The model-visible history the records hold: the newest canonical
|
||||
/// checkpoint, then later item outcomes and turns.
|
||||
pub(crate) fn history_from_records(
|
||||
&self,
|
||||
turns: &[TurnRecord],
|
||||
items_by_turn: &HashMap<String, Vec<TurnItemRecord>>,
|
||||
) -> Result<Vec<Message>> {
|
||||
let (covered, mut messages) = self
|
||||
.settled_history(turns, items_by_turn)?
|
||||
.unwrap_or_default();
|
||||
messages.extend(RuntimeThreadManager::reconstruct_messages_from_turns_with(
|
||||
&turns[covered..],
|
||||
items_by_turn,
|
||||
)?);
|
||||
Ok(messages)
|
||||
self.history_with_prefix(turns, items_by_turn, Vec::new(), 0)
|
||||
}
|
||||
|
||||
/// Journal for one history seed into an existing thread (#6555), kept
|
||||
@@ -13175,7 +13592,8 @@ impl RuntimeThreadManager {
|
||||
let (captured_source, saved_prefix) = self
|
||||
.capture_native_fork_source(&source, &source_turns, sessions_dir)
|
||||
.await?;
|
||||
let fork_prefix = if saved_transcript_is_compacted(&source_turns, &items_by_turn) {
|
||||
let fork_prefix = if saved_transcript_requires_record_history(&source_turns, &items_by_turn)
|
||||
{
|
||||
(
|
||||
self.store
|
||||
.history_from_records(&source_turns, &items_by_turn)?,
|
||||
@@ -13192,6 +13610,7 @@ impl RuntimeThreadManager {
|
||||
let mut cloned_turn = source_turn.clone();
|
||||
cloned_turn.id = runtime_record_id("turn");
|
||||
cloned_turn.thread_id = forked.id.clone();
|
||||
cloned_turn.prefix_checkpoint = None;
|
||||
if let Some(checkpoint) = forked.saved_session_checkpoint.as_mut()
|
||||
&& checkpoint.covered_turn_id.as_deref() == Some(source_turn.id.as_str())
|
||||
{
|
||||
@@ -13681,13 +14100,13 @@ impl RuntimeThreadManager {
|
||||
if let Some((messages, covered)) = saved_prefix {
|
||||
let kept_turns = covered.min(cutoff_turn_idx);
|
||||
// A compacted source has no transcript to cut — see
|
||||
// `saved_transcript_is_compacted`. Every boundary in its saved
|
||||
// `saved_transcript_requires_record_history`. Every boundary in its saved
|
||||
// messages names a turn while carrying only that turn's prompt, so
|
||||
// a slice of them would claim coverage of exchanges the fork's own
|
||||
// document does not hold. There is nothing for the boundary search
|
||||
// to find either, so it is skipped rather than risk a refusal for a
|
||||
// trim the fork is not going to take.
|
||||
let compacted = saved_transcript_is_compacted(&source_turns, &items_by_turn);
|
||||
let compacted = saved_transcript_requires_record_history(&source_turns, &items_by_turn);
|
||||
let retained_messages = if compacted {
|
||||
None
|
||||
} else if covered <= cutoff_turn_idx {
|
||||
@@ -13793,6 +14212,7 @@ impl RuntimeThreadManager {
|
||||
let mut cloned_turn = source_turn.clone();
|
||||
cloned_turn.id = runtime_record_id("turn");
|
||||
cloned_turn.thread_id = forked.id.clone();
|
||||
cloned_turn.prefix_checkpoint = None;
|
||||
if let Some(checkpoint) = forked.saved_session_checkpoint.as_mut()
|
||||
&& checkpoint.covered_turn_id.as_deref() == Some(source_turn.id.as_str())
|
||||
{
|
||||
@@ -14396,6 +14816,7 @@ impl RuntimeThreadManager {
|
||||
// Only create a turn if there's content.
|
||||
if !item_ids.is_empty() {
|
||||
seed_turns.push(TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: if turn_seed.image_content.is_empty() {
|
||||
@@ -16355,6 +16776,7 @@ impl RuntimeThreadManager {
|
||||
.clone()
|
||||
.unwrap_or_else(|| summarize_text(&prompt, SUMMARY_LIMIT));
|
||||
let mut turn = TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens,
|
||||
schema_version: if max_output_tokens.is_some() { OUTPUT_LIMIT_RUNTIME_SCHEMA_VERSION } else { CURRENT_RUNTIME_SCHEMA_VERSION },
|
||||
@@ -16889,6 +17311,7 @@ impl RuntimeThreadManager {
|
||||
thread.auto_approve,
|
||||
);
|
||||
let turn = TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
@@ -17253,6 +17676,9 @@ impl RuntimeThreadManager {
|
||||
features.disable(codewhale_runtime::features::Feature::Mcp);
|
||||
}
|
||||
let engine_cfg = EngineConfig {
|
||||
session_checkpoint_backend: Some(
|
||||
crate::core::events::SessionCheckpointProjection::RuntimeReceiptRedacted,
|
||||
),
|
||||
model: route_model.clone(),
|
||||
active_route_limits: route_limits,
|
||||
workspace: thread.workspace.clone(),
|
||||
@@ -17629,6 +18055,13 @@ impl RuntimeThreadManager {
|
||||
tokio::pin!(notified);
|
||||
notified.as_mut().enable();
|
||||
let Some(active) = active.upgrade() else {
|
||||
if let EngineEvent::SessionUpdated {
|
||||
checkpoint: Some(checkpoint),
|
||||
..
|
||||
} = &event
|
||||
{
|
||||
checkpoint.fail("Runtime checkpoint active owner was released");
|
||||
}
|
||||
break 'pump;
|
||||
};
|
||||
let owns_turn = {
|
||||
@@ -17642,6 +18075,12 @@ impl RuntimeThreadManager {
|
||||
};
|
||||
if !owns_turn {
|
||||
match &event {
|
||||
EngineEvent::SessionUpdated {
|
||||
checkpoint: Some(checkpoint),
|
||||
..
|
||||
} => {
|
||||
checkpoint.fail("Runtime checkpoint has no active event owner");
|
||||
}
|
||||
EngineEvent::ApprovalRequired { id, .. }
|
||||
| EngineEvent::ElevationRequired { tool_id: id, .. } => {
|
||||
let _ = handle
|
||||
@@ -17661,10 +18100,20 @@ impl RuntimeThreadManager {
|
||||
tokio::select! {
|
||||
permit = forward.reserve() => match permit {
|
||||
Ok(permit) => { permit.send(event); break; },
|
||||
Err(_) => break 'pump,
|
||||
Err(_) => {
|
||||
if let EngineEvent::SessionUpdated { checkpoint: Some(checkpoint), .. } = &event {
|
||||
checkpoint.fail("Runtime checkpoint event receiver closed");
|
||||
}
|
||||
break 'pump;
|
||||
},
|
||||
},
|
||||
_ = &mut notified => {},
|
||||
_ = stop.cancelled() => break 'pump,
|
||||
_ = stop.cancelled() => {
|
||||
if let EngineEvent::SessionUpdated { checkpoint: Some(checkpoint), .. } = &event {
|
||||
checkpoint.fail("Runtime checkpoint event owner stopped");
|
||||
}
|
||||
break 'pump;
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -17865,7 +18314,7 @@ impl RuntimeThreadManager {
|
||||
|
||||
fn restore_thread_messages(&self, thread: &ThreadRecord) -> Result<Vec<Message>> {
|
||||
let turns = self.store.list_turns_for_thread(&thread.id)?;
|
||||
let (mut messages, mut covered) = match self.saved_session_prefix(thread, &turns) {
|
||||
let (messages, covered) = match self.saved_session_prefix(thread, &turns) {
|
||||
Ok(prefix) => prefix.unwrap_or_default(),
|
||||
Err(error) => {
|
||||
let Some(stale) = error.downcast_ref::<StaleSessionBinding>() else {
|
||||
@@ -17879,18 +18328,8 @@ impl RuntimeThreadManager {
|
||||
}
|
||||
};
|
||||
let items_by_turn = self.prepared_items_for_turns(&turns[covered..])?;
|
||||
if let Some((settled, history)) = self
|
||||
.store
|
||||
.settled_history(&turns[covered..], &items_by_turn)?
|
||||
{
|
||||
messages = history;
|
||||
covered += settled;
|
||||
}
|
||||
messages.extend(Self::reconstruct_messages_from_turns_with(
|
||||
&turns[covered..],
|
||||
&items_by_turn,
|
||||
)?);
|
||||
Ok(messages)
|
||||
self.store
|
||||
.history_with_prefix(&turns, &items_by_turn, messages, covered)
|
||||
}
|
||||
|
||||
/// Drop `thread`'s saved-session binding when the stored record still
|
||||
@@ -18147,28 +18586,7 @@ impl RuntimeThreadManager {
|
||||
let mut messages = Vec::new();
|
||||
let mut execution_providers = HashMap::new();
|
||||
for turn in turns {
|
||||
let stored_items = items_by_turn.get(&turn.id).cloned().unwrap_or_default();
|
||||
let items = if turn.item_ids.is_empty() {
|
||||
stored_items
|
||||
} else {
|
||||
let mut by_id: HashMap<String, TurnItemRecord> = stored_items
|
||||
.iter()
|
||||
.cloned()
|
||||
.map(|item| (item.id.clone(), item))
|
||||
.collect();
|
||||
let mut ordered = Vec::new();
|
||||
for item_id in &turn.item_ids {
|
||||
if let Some(item) = by_id.remove(item_id) {
|
||||
ordered.push(item);
|
||||
}
|
||||
}
|
||||
for item in stored_items {
|
||||
if by_id.contains_key(&item.id) {
|
||||
ordered.push(item);
|
||||
}
|
||||
}
|
||||
ordered
|
||||
};
|
||||
let items = ordered_turn_items(turn, items_by_turn)?;
|
||||
|
||||
for item in items.iter().filter(|item| projects_tool_history(item)) {
|
||||
for side in ["tool_use_id", "tool_result_for"] {
|
||||
@@ -18462,6 +18880,218 @@ impl RuntimeThreadManager {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn persist_runtime_checkpoint(
|
||||
&self,
|
||||
thread_id: &str,
|
||||
turn_id: &str,
|
||||
engine: &EngineHandle,
|
||||
engine_turn_id: Option<&str>,
|
||||
request: Arc<crate::core::events::SessionCheckpointRequest>,
|
||||
receipt_secrets: &[String],
|
||||
) -> Result<crate::core::events::SessionCheckpointReceipt> {
|
||||
let thread = self.store.load_thread(thread_id)?;
|
||||
request
|
||||
.validate_prefix(thread_id, &thread.workspace, &request.snapshot.messages)
|
||||
.map_err(anyhow::Error::msg)?;
|
||||
anyhow::ensure!(
|
||||
request.caller.agent_id.is_none()
|
||||
&& engine_turn_id == Some(request.caller.turn_id.as_str())
|
||||
&& !request.is_stale(),
|
||||
"Runtime checkpoint caller is not the active root turn"
|
||||
);
|
||||
{
|
||||
let active = self.active.lock().await;
|
||||
anyhow::ensure!(
|
||||
active.engines.get(thread_id).is_some_and(|state| {
|
||||
Arc::ptr_eq(&state.engine.rx_event, &engine.rx_event)
|
||||
&& state.active_turn.as_ref().is_some_and(|turn| {
|
||||
turn.turn_id == turn_id && !turn.interrupt_requested
|
||||
})
|
||||
}),
|
||||
"Runtime checkpoint active owner changed"
|
||||
);
|
||||
}
|
||||
let tool_declarations = checkpoint_tool_declarations(&request.snapshot.messages)?;
|
||||
let private_results: HashSet<_> = request
|
||||
.snapshot
|
||||
.messages
|
||||
.iter()
|
||||
.flat_map(|message| &message.content)
|
||||
.filter_map(|block| match block {
|
||||
ContentBlock::ToolUse {
|
||||
execution_id,
|
||||
id,
|
||||
name,
|
||||
..
|
||||
} if name == REQUEST_USER_INPUT_TOOL_NAME => {
|
||||
Some((execution_id.clone(), id.clone()))
|
||||
}
|
||||
_ => None,
|
||||
})
|
||||
.collect();
|
||||
let settled_calls: HashSet<_> = request
|
||||
.snapshot
|
||||
.messages
|
||||
.iter()
|
||||
.flat_map(|message| &message.content)
|
||||
.filter_map(|block| match block {
|
||||
ContentBlock::ToolResult {
|
||||
execution_id,
|
||||
tool_use_id,
|
||||
..
|
||||
} => Some((execution_id.clone(), tool_use_id.clone())),
|
||||
_ => None,
|
||||
})
|
||||
.collect();
|
||||
let mut messages = request.snapshot.messages.clone();
|
||||
for message in &mut messages {
|
||||
crate::core::engine::redact_tool_blocks(message, |text| {
|
||||
crate::client::redact_model_bound_text(text, receipt_secrets)
|
||||
});
|
||||
for block in &mut message.content {
|
||||
if let ContentBlock::ToolResult {
|
||||
execution_id,
|
||||
tool_use_id,
|
||||
content,
|
||||
content_blocks,
|
||||
..
|
||||
} = block
|
||||
&& private_results.contains(&(execution_id.clone(), tool_use_id.clone()))
|
||||
{
|
||||
*content = REDACTED_USER_INPUT_RECEIPT.to_string();
|
||||
*content_blocks = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
let store = self.store.clone();
|
||||
let owner = self._process_owner_lock.clone();
|
||||
let owner_path = self
|
||||
.session_store_binding()
|
||||
.data_dir
|
||||
.join(RUNTIME_PROCESS_OWNER_LOCK_FILE);
|
||||
let turn_id_owned = turn_id.to_owned();
|
||||
let pending = request.clone();
|
||||
let receipt = tokio::task::spawn_blocking(
|
||||
move || -> Result<crate::core::events::SessionCheckpointReceipt> {
|
||||
anyhow::ensure!(
|
||||
!pending.is_stale() && owner_lock_is_at_path(&owner._file, &owner_path)?,
|
||||
"Runtime checkpoint owner lease expired"
|
||||
);
|
||||
let _mutation = store.turn_mutation.lock();
|
||||
let mut turn = store.load_turn(&turn_id_owned)?;
|
||||
anyhow::ensure!(
|
||||
turn.thread_id == pending.caller.session_id && turn.status.is_active_work(),
|
||||
"Runtime checkpoint turn is no longer active"
|
||||
);
|
||||
let items_by_turn =
|
||||
store.list_items_for_turns_map(std::slice::from_ref(&turn.id))?;
|
||||
let items = ordered_turn_items(&turn, &items_by_turn)?;
|
||||
anyhow::ensure!(
|
||||
items.iter().map(|item| &item.id).eq(turn.item_ids.iter()),
|
||||
"Runtime checkpoint item cursor is incomplete"
|
||||
);
|
||||
let mut covered_items = HashMap::new();
|
||||
let mut pending_items = Vec::new();
|
||||
for item in &items {
|
||||
if projects_tool_history(item)
|
||||
&& let Some(identity) =
|
||||
RuntimeToolIdentity::read(item.metadata.as_ref(), "tool_result_for")?
|
||||
&& settled_calls.contains(&(
|
||||
identity.execution_id.map(str::to_owned),
|
||||
identity.provider_id.to_owned(),
|
||||
))
|
||||
{
|
||||
covered_items.insert(item.id.clone(), checkpoint_item_digest(item)?);
|
||||
}
|
||||
if matches!(item.kind, TurnItemKind::UserMessage)
|
||||
&& matches!(
|
||||
item.status,
|
||||
TurnItemLifecycleStatus::Queued | TurnItemLifecycleStatus::Canceled
|
||||
)
|
||||
{
|
||||
pending_items.push(item.id.clone());
|
||||
}
|
||||
if !projects_tool_history(item)
|
||||
&& matches!(
|
||||
item.kind,
|
||||
TurnItemKind::UserMessage
|
||||
| TurnItemKind::AgentMessage
|
||||
| TurnItemKind::AgentReasoning
|
||||
)
|
||||
&& !matches!(
|
||||
item.status,
|
||||
TurnItemLifecycleStatus::Queued | TurnItemLifecycleStatus::Canceled
|
||||
)
|
||||
{
|
||||
covered_items.insert(item.id.clone(), checkpoint_item_digest(item)?);
|
||||
}
|
||||
}
|
||||
let digest = session_messages_sha256(&messages)?;
|
||||
turn.schema_version = turn
|
||||
.schema_version
|
||||
.max(PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION);
|
||||
turn.prefix_checkpoint = Some(RuntimePrefixCheckpoint {
|
||||
checkpoint_id: pending.checkpoint_id.clone(),
|
||||
session_id: pending.caller.session_id.clone(),
|
||||
agent_id: pending.caller.agent_id.clone(),
|
||||
runtime_turn_id: turn.id.clone(),
|
||||
engine_turn_id: pending.caller.turn_id.clone(),
|
||||
call_id: pending.caller.call_id.clone(),
|
||||
owner_id: store.owner_id.clone(),
|
||||
source_sha256: pending.prefix_sha256.clone(),
|
||||
persisted_sha256: digest.clone(),
|
||||
messages_len: messages.len(),
|
||||
projection: "runtime_receipt_redacted".to_string(),
|
||||
item_ids: turn.item_ids.clone(),
|
||||
covered_items,
|
||||
pending_items,
|
||||
tool_declarations,
|
||||
});
|
||||
store.checkpoint_tail(&turn, &items_by_turn, &messages)?;
|
||||
let reference = store.save_settled_history(&messages)?;
|
||||
anyhow::ensure!(
|
||||
reference["sha256"].as_str() == Some(digest.as_str()),
|
||||
"Runtime checkpoint stored projection changed"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
!pending.is_stale() && owner_lock_is_at_path(&owner._file, &owner_path)?,
|
||||
"Runtime checkpoint was withdrawn before publication"
|
||||
);
|
||||
store.save_turn(&turn)?;
|
||||
let published = store.load_turn(&turn.id)?;
|
||||
let checkpoint = published
|
||||
.prefix_checkpoint
|
||||
.as_ref()
|
||||
.context("Runtime checkpoint publication was lost")?;
|
||||
anyhow::ensure!(
|
||||
checkpoint.checkpoint_id == pending.checkpoint_id
|
||||
&& store.load_prefix_history(checkpoint)? == messages,
|
||||
"Runtime checkpoint publication could not be verified"
|
||||
);
|
||||
Ok(crate::core::events::SessionCheckpointReceipt {
|
||||
persisted_sha256: digest,
|
||||
messages_len: messages.len(),
|
||||
projection:
|
||||
crate::core::events::SessionCheckpointProjection::RuntimeReceiptRedacted,
|
||||
})
|
||||
},
|
||||
)
|
||||
.await
|
||||
.context("Runtime checkpoint writer stopped; persistence is uncertain")??;
|
||||
let active = self.active.lock().await;
|
||||
anyhow::ensure!(
|
||||
!request.is_stale()
|
||||
&& active.engines.get(thread_id).is_some_and(|state| {
|
||||
Arc::ptr_eq(&state.engine.rx_event, &engine.rx_event)
|
||||
&& state.active_turn.as_ref().is_some_and(|turn| {
|
||||
turn.turn_id == turn_id && !turn.interrupt_requested
|
||||
})
|
||||
}),
|
||||
"Runtime checkpoint owner changed after writing; persistence is uncertain"
|
||||
);
|
||||
Ok(receipt)
|
||||
}
|
||||
|
||||
async fn monitor_turn(
|
||||
&self,
|
||||
thread_id: String,
|
||||
@@ -18583,6 +19213,28 @@ impl RuntimeThreadManager {
|
||||
bail!("engine event channel closed before turn {turn_id} completed");
|
||||
};
|
||||
|
||||
if let EngineEvent::SessionUpdated {
|
||||
checkpoint: Some(checkpoint),
|
||||
..
|
||||
} = &event
|
||||
{
|
||||
match self
|
||||
.persist_runtime_checkpoint(
|
||||
&thread_id,
|
||||
&turn_id,
|
||||
&engine,
|
||||
engine_turn_id.as_deref(),
|
||||
checkpoint.clone(),
|
||||
&receipt_secrets,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(receipt) => checkpoint.succeed(receipt),
|
||||
Err(error) => checkpoint.fail(format!("Runtime checkpoint failed: {error:#}")),
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// SyncSession and configuration operations emit control status
|
||||
// receipts on the same channel before SendMessage is processed.
|
||||
// They belong to engine setup, not to the next claimed turn.
|
||||
@@ -19736,6 +20388,9 @@ impl RuntimeThreadManager {
|
||||
if !event_channel_closed =>
|
||||
{
|
||||
match event {
|
||||
Some(EngineEvent::SessionUpdated { checkpoint: Some(checkpoint), .. }) => {
|
||||
checkpoint.fail("Runtime checkpoint cannot wait behind an approval");
|
||||
}
|
||||
Some(EngineEvent::ApprovalWithdrawn { id: withdrawn }) if withdrawn == id => {
|
||||
self.cancel_pending_approval(&approval_id);
|
||||
break None;
|
||||
|
||||
@@ -2474,6 +2474,7 @@ fn sample_thread(thread_id: &str) -> ThreadRecord {
|
||||
fn sample_turn(thread_id: &str, turn_id: &str, status: RuntimeTurnStatus) -> TurnRecord {
|
||||
let now = Utc::now();
|
||||
TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
@@ -6912,7 +6913,7 @@ fn store_load_scenario() {
|
||||
let store = RuntimeThreadStore::open(dir.clone()).expect("open store");
|
||||
|
||||
let mut turn = sample_turn("thr_t", "trn_future", RuntimeTurnStatus::InProgress);
|
||||
turn.schema_version = MAX_SUPPORTED_RUNTIME_SCHEMA_VERSION + 1;
|
||||
turn.schema_version = PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION + 1;
|
||||
|
||||
let path = store.turns_dir.join(format!("{}.json", turn.id));
|
||||
std::fs::create_dir_all(path.parent().unwrap()).expect("mkdirs");
|
||||
@@ -18438,6 +18439,8 @@ async fn compaction_lifecycle_emits_item_events_with_compaction_counts() -> Resu
|
||||
let harness = install_mock_engine(&manager, &thread.id).await;
|
||||
let mut rx_op = harness.rx_op;
|
||||
let tx_event = harness.tx_event;
|
||||
let checkpoint_thread = thread.id.clone();
|
||||
let checkpoint_workspace = thread.workspace.clone();
|
||||
tokio::spawn(async move {
|
||||
let mut op_count = 0usize;
|
||||
let mut snapshots = 0usize;
|
||||
@@ -18464,6 +18467,61 @@ async fn compaction_lifecycle_emits_item_events_with_compaction_counts() -> Resu
|
||||
submission_id: None,
|
||||
})
|
||||
.await;
|
||||
for (agent_id, withdrawn, accepted) in [
|
||||
(Some("child".to_string()), false, false),
|
||||
(None, true, false),
|
||||
(None, false, true),
|
||||
(None, false, true),
|
||||
] {
|
||||
let messages: Vec<Message> = serde_json::from_value(json!([
|
||||
{"role":"user","content":[{"type":"text","text":"auto checkpoint prefix"}]}
|
||||
])).unwrap();
|
||||
let snapshot = Arc::new(crate::core::ops::SessionSnapshot {
|
||||
session_id: checkpoint_thread.clone(),
|
||||
messages: messages.clone(),
|
||||
total_tokens: 0,
|
||||
model: String::new(),
|
||||
model_provider: String::new(),
|
||||
model_provider_id: None,
|
||||
workspace: checkpoint_workspace.clone(),
|
||||
system_prompt: None,
|
||||
instruction_assembly: None,
|
||||
mode: "agent".into(),
|
||||
});
|
||||
let (reply, received) = oneshot::channel();
|
||||
let withdraw = CancellationToken::new();
|
||||
if withdrawn {
|
||||
withdraw.cancel();
|
||||
}
|
||||
let request = crate::core::events::SessionCheckpointRequest::new(
|
||||
crate::core::events::SessionCheckpointCaller {
|
||||
session_id: checkpoint_thread.clone(), agent_id, turn_id: "engine_turn_auto".into(),
|
||||
call_id: "mock_checkpoint_call".into(), workspace: checkpoint_workspace.clone(),
|
||||
}, snapshot, session_messages_sha256(&messages).unwrap(),
|
||||
crate::core::events::SessionCheckpointProjection::RuntimeReceiptRedacted, withdraw, reply,
|
||||
);
|
||||
tx_event
|
||||
.send(EngineEvent::SessionUpdated {
|
||||
checkpoint: Some(request),
|
||||
session_id: checkpoint_thread.clone(),
|
||||
messages: Arc::new(messages),
|
||||
system_prompt: None,
|
||||
instruction_assembly: None,
|
||||
model: String::new(),
|
||||
workspace: checkpoint_workspace.clone(),
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
let outcome = tokio::time::timeout(Duration::from_secs(5), received)
|
||||
.await
|
||||
.expect("checkpoint must not call back into the Engine mailbox")
|
||||
.expect("checkpoint writer must settle its sender");
|
||||
if accepted {
|
||||
assert!(matches!(outcome, Ok(true)));
|
||||
} else {
|
||||
assert!(outcome.is_err());
|
||||
}
|
||||
}
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::CompactionStarted {
|
||||
id: "auto_compact_1".to_string(),
|
||||
@@ -18627,6 +18685,13 @@ async fn compaction_lifecycle_emits_item_events_with_compaction_counts() -> Resu
|
||||
// Each compacting turn stores the history the engine settled on; a
|
||||
// reload restores the newest one instead of replaying the turns it
|
||||
// replaced, and the thread record carries no summary of its own.
|
||||
assert!(
|
||||
manager
|
||||
.store
|
||||
.list_turns_for_thread(&thread.id)?
|
||||
.iter()
|
||||
.any(|turn| turn.prefix_checkpoint.is_some())
|
||||
);
|
||||
let record = manager.get_thread(&thread.id).await?;
|
||||
assert_eq!(record.system_prompt, None);
|
||||
let restored = manager.restore_thread_messages(&record)?;
|
||||
@@ -18725,6 +18790,7 @@ fn opening_manager_recovers_stale_queued_and_in_progress_work() -> Result<()> {
|
||||
manager.store.save_item(&queued_item)?;
|
||||
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
@@ -18765,6 +18831,7 @@ fn opening_manager_recovers_stale_queued_and_in_progress_work() -> Result<()> {
|
||||
workspace_snapshots: Vec::new(),
|
||||
})?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
@@ -18963,6 +19030,7 @@ fn seed_turns_with_user_messages(
|
||||
ended_at: Some(created_at),
|
||||
})?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
@@ -19720,6 +19788,7 @@ fn restart_rebuild_restores_tool_call_identity_from_persisted_items() -> Result<
|
||||
manager.store.save_item(&user_item)?;
|
||||
manager.store.save_item(&call_item)?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
@@ -19792,6 +19861,186 @@ fn restart_rebuild_restores_tool_call_identity_from_persisted_items() -> Result<
|
||||
other => panic!("expected restored tool result, got {other:?}"),
|
||||
}
|
||||
|
||||
let mut checkpoint_turn = sample_turn(&thread.id, "turn_prefix", RuntimeTurnStatus::InProgress);
|
||||
checkpoint_turn.schema_version = PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION;
|
||||
checkpoint_turn.max_output_tokens = std::num::NonZeroU32::new(1500);
|
||||
let mut checkpoint_user = user_item.clone();
|
||||
checkpoint_user.id = "item_prefix_user".into();
|
||||
checkpoint_user.turn_id = checkpoint_turn.id.clone();
|
||||
checkpoint_user.detail = Some("checkpoint input".into());
|
||||
checkpoint_user.summary = "checkpoint input".into();
|
||||
let mut first = call_item.clone();
|
||||
first.id = "item_prefix_first".into();
|
||||
first.turn_id = checkpoint_turn.id.clone();
|
||||
first.status = TurnItemLifecycleStatus::InProgress;
|
||||
first.detail = Some(r#"{"path":"first","token":"fixture-secret"}"#.into());
|
||||
first.metadata = Some(
|
||||
json!({"tool_use_id":"execution_first", "execution_id":"execution_first",
|
||||
"provider_tool_use_id":"provider_first", "tool_name":"read", "tool_input":r#"{"path":"first","token":"fixture-secret"}"#}),
|
||||
);
|
||||
let mut closed = first.clone();
|
||||
closed.id = "item_prefix_closed".into();
|
||||
closed.status = TurnItemLifecycleStatus::Completed;
|
||||
closed.detail = Some("raw durable result".into());
|
||||
closed.metadata = Some(
|
||||
json!({"tool_use_id":"execution_closed", "execution_id":"execution_closed",
|
||||
"provider_tool_use_id":"provider_closed", "tool_name":"read", "tool_input":r#"{"path":"closed"}"#,
|
||||
"tool_result_for":"execution_closed"}),
|
||||
);
|
||||
checkpoint_turn.item_ids = vec![
|
||||
checkpoint_user.id.clone(),
|
||||
closed.id.clone(),
|
||||
first.id.clone(),
|
||||
];
|
||||
manager.store.save_item(&checkpoint_user)?;
|
||||
manager.store.save_item(&closed)?;
|
||||
manager.store.save_item(&first)?;
|
||||
let mut prefix = messages.clone();
|
||||
prefix.extend(serde_json::from_value::<Vec<Message>>(json!([
|
||||
{"role":"user", "content":[{"type":"text","text":"<turn_meta>canonical</turn_meta>checkpoint input"}]},
|
||||
{"role":"assistant", "content":[{"type":"tool_use","execution_id":"execution_closed","id":"provider_closed","name":"read","input":{"path":"closed"}}]},
|
||||
{"role":"user", "content":[{"type":"tool_result","execution_id":"execution_closed","tool_use_id":"provider_closed","content":"compacted result\n[hook context] retained canonical detail"}]},
|
||||
{"role":"assistant", "content":[
|
||||
{"type":"thinking","thinking":"canonical thought","signature":"retained-signature"},
|
||||
{"type":"tool_use","execution_id":"execution_first","id":"provider_first","name":"read","input":{"path":"first","token":"fixture-secret"}},
|
||||
{"type":"tool_use","execution_id":"execution_second","id":"provider_second","name":"read","input":{"path":"second"}}
|
||||
]}
|
||||
]))?);
|
||||
let source_sha256 = session_messages_sha256(&prefix)?;
|
||||
let tool_declarations = checkpoint_tool_declarations(&prefix)?;
|
||||
for message in &mut prefix {
|
||||
crate::core::engine::redact_tool_blocks(message, |text| {
|
||||
text.replace("fixture-secret", "[REDACTED]")
|
||||
});
|
||||
}
|
||||
let reference = manager.store.save_settled_history(&prefix)?;
|
||||
checkpoint_turn.prefix_checkpoint = Some(RuntimePrefixCheckpoint {
|
||||
checkpoint_id: "checkpoint_fixture".into(),
|
||||
session_id: thread.id.clone(),
|
||||
agent_id: None,
|
||||
runtime_turn_id: checkpoint_turn.id.clone(),
|
||||
engine_turn_id: "engine_prefix".into(),
|
||||
call_id: "invocation_prefix".into(),
|
||||
owner_id: manager.store.owner_id.clone(),
|
||||
source_sha256,
|
||||
persisted_sha256: reference["sha256"].as_str().unwrap().into(),
|
||||
messages_len: prefix.len(),
|
||||
projection: "runtime_receipt_redacted".into(),
|
||||
item_ids: checkpoint_turn.item_ids.clone(),
|
||||
tool_declarations,
|
||||
pending_items: Vec::new(),
|
||||
covered_items: HashMap::from([
|
||||
(
|
||||
checkpoint_user.id.clone(),
|
||||
checkpoint_item_digest(&checkpoint_user)?,
|
||||
),
|
||||
(closed.id.clone(), checkpoint_item_digest(&closed)?),
|
||||
]),
|
||||
});
|
||||
manager.store.save_turn(&checkpoint_turn)?;
|
||||
assert!(
|
||||
serde_json::to_value(&checkpoint_turn)?
|
||||
.get("prefix_checkpoint")
|
||||
.is_none()
|
||||
);
|
||||
assert!(
|
||||
manager
|
||||
.store
|
||||
.load_turn(&checkpoint_turn.id)?
|
||||
.prefix_checkpoint
|
||||
.is_some()
|
||||
);
|
||||
assert_eq!(manager.restore_thread_messages(&thread)?, prefix);
|
||||
checkpoint_turn.prefix_checkpoint.as_mut().unwrap().owner_id = "foreign_owner".into();
|
||||
manager.store.save_turn(&checkpoint_turn)?;
|
||||
assert!(manager.restore_thread_messages(&thread).is_err());
|
||||
checkpoint_turn.prefix_checkpoint.as_mut().unwrap().owner_id = manager.store.owner_id.clone();
|
||||
manager.store.save_turn(&checkpoint_turn)?;
|
||||
|
||||
first.status = TurnItemLifecycleStatus::Completed;
|
||||
first.detail = Some("first result".into());
|
||||
first.metadata.as_mut().unwrap()["tool_result_for"] = json!("execution_first");
|
||||
manager.store.save_item(&first)?;
|
||||
let mut second = first.clone();
|
||||
second.id = "item_prefix_second".into();
|
||||
second.status = TurnItemLifecycleStatus::Interrupted;
|
||||
second.detail = Some(r#"{"path":"second"}"#.into());
|
||||
second.metadata = Some(
|
||||
json!({"tool_use_id":"execution_second", "execution_id":"execution_second",
|
||||
"provider_tool_use_id":"provider_second", "tool_name":"read", "tool_input":r#"{"path":"second"}"#}),
|
||||
);
|
||||
let mut later = checkpoint_user.clone();
|
||||
later.id = "item_prefix_later".into();
|
||||
later.kind = TurnItemKind::AgentMessage;
|
||||
later.detail = Some("later assistant".into());
|
||||
later.summary = "later assistant".into();
|
||||
manager.store.save_item(&second)?;
|
||||
manager.store.save_item(&later)?;
|
||||
checkpoint_turn
|
||||
.item_ids
|
||||
.extend([second.id.clone(), later.id.clone()]);
|
||||
checkpoint_turn.status = RuntimeTurnStatus::Interrupted;
|
||||
manager.store.save_turn(&checkpoint_turn)?;
|
||||
let mut expected = prefix.clone();
|
||||
expected.extend(serde_json::from_value::<Vec<Message>>(json!([
|
||||
{"role":"user","content":[
|
||||
{"type":"tool_result","execution_id":"execution_first","tool_use_id":"provider_first","content":"first result"},
|
||||
{"type":"tool_result","execution_id":"execution_second","tool_use_id":"provider_second",
|
||||
"content":codewhale_runtime::tool_history_repair::CRASH_REPAIR_CONTENT,"is_error":true}
|
||||
]},
|
||||
{"role":"assistant","content":[{"type":"text","text":"later assistant"}]}
|
||||
]))?);
|
||||
assert_eq!(manager.restore_thread_messages(&thread)?, expected);
|
||||
let turn = manager.store.load_turn(&checkpoint_turn.id)?;
|
||||
let items = manager.prepared_items_for_turns(std::slice::from_ref(&turn))?;
|
||||
assert_eq!(
|
||||
manager.history_from_records(&[turn.clone()], &items)?,
|
||||
expected
|
||||
);
|
||||
first.metadata.as_mut().unwrap()["tool_input"] = json!(r#"{"path":"changed"}"#);
|
||||
manager.store.save_item(&first)?;
|
||||
assert!(manager.restore_thread_messages(&thread).is_err());
|
||||
first.metadata.as_mut().unwrap()["tool_input"] =
|
||||
json!(r#"{"path":"first","token":"fixture-secret"}"#);
|
||||
manager.store.save_item(&first)?;
|
||||
let history_path = manager
|
||||
.store
|
||||
.settled_history_path(reference["sha256"].as_str().unwrap())?;
|
||||
fs::write(&history_path, b"corrupt")?;
|
||||
assert!(manager.restore_thread_messages(&thread).is_err());
|
||||
manager.store.save_settled_history(&prefix)?;
|
||||
fs::remove_file(manager.store.item_path(&checkpoint_user.id)?)?;
|
||||
assert!(manager.restore_thread_messages(&thread).is_err());
|
||||
manager.store.save_item(&checkpoint_user)?;
|
||||
fs::remove_file(manager.store.item_path(&second.id)?)?;
|
||||
checkpoint_turn.item_ids.retain(|id| id != &second.id);
|
||||
manager.store.save_turn(&checkpoint_turn)?;
|
||||
assert_eq!(manager.restore_thread_messages(&thread)?, expected);
|
||||
drop(manager);
|
||||
let manager = test_manager(dir.clone())?;
|
||||
assert_eq!(manager.restore_thread_messages(&thread)?, expected);
|
||||
|
||||
let mut compacted = sample_turn(
|
||||
&thread.id,
|
||||
"turn_after_prefix",
|
||||
RuntimeTurnStatus::Completed,
|
||||
);
|
||||
compacted.created_at = checkpoint_turn.created_at + chrono::Duration::seconds(1);
|
||||
let compacted_history: Vec<Message> = serde_json::from_value(json!([
|
||||
{"role":"user","content":[{"type":"text","text":"newer canonical compaction"}]}
|
||||
]))?;
|
||||
let compacted_reference = manager.store.save_settled_history(&compacted_history)?;
|
||||
let mut compaction_item = later.clone();
|
||||
compaction_item.id = "item_after_prefix_compaction".into();
|
||||
compaction_item.turn_id = compacted.id.clone();
|
||||
compaction_item.kind = TurnItemKind::ContextCompaction;
|
||||
compaction_item.metadata = Some(json!({"settled_history":compacted_reference}));
|
||||
compacted.item_ids = vec![compaction_item.id.clone()];
|
||||
manager.store.save_item(&compaction_item)?;
|
||||
manager.store.save_turn(&compacted)?;
|
||||
fs::write(&history_path, b"superseded old prefix")?;
|
||||
assert_eq!(manager.restore_thread_messages(&thread)?, compacted_history);
|
||||
|
||||
let _ = std::fs::remove_dir_all(dir);
|
||||
Ok(())
|
||||
}
|
||||
@@ -19827,6 +20076,7 @@ fn restart_rebuild_keeps_in_flight_tool_call_identity() -> Result<()> {
|
||||
};
|
||||
manager.store.save_item(&call_item)?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
@@ -19929,6 +20179,7 @@ fn restart_rebuild_skips_steers_the_engine_never_delivered() -> Result<()> {
|
||||
manager.store.save_item(&dropped)?;
|
||||
manager.store.save_item(&pending)?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
@@ -20032,6 +20283,7 @@ fn restart_rebuild_skips_legacy_tool_items_without_identity() -> Result<()> {
|
||||
manager.store.save_item(&user_item)?;
|
||||
manager.store.save_item(&legacy_tool_item)?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
|
||||
@@ -214,7 +214,16 @@ pub(crate) enum NestedCallVerdict {
|
||||
}
|
||||
|
||||
/// One nested call waiting on the turn loop's gate.
|
||||
pub(crate) struct NestedCallRequest {
|
||||
pub(crate) enum NestedCallRequest {
|
||||
Tool(NestedToolCallRequest),
|
||||
Checkpoint {
|
||||
caller: crate::core::events::SessionCheckpointCaller,
|
||||
withdraw: CancellationToken,
|
||||
reply: oneshot::Sender<Result<bool, ToolError>>,
|
||||
},
|
||||
}
|
||||
|
||||
pub(crate) struct NestedToolCallRequest {
|
||||
pub(crate) name: String,
|
||||
pub(crate) input: Value,
|
||||
pub(crate) reply: oneshot::Sender<NestedCallVerdict>,
|
||||
@@ -226,7 +235,7 @@ pub(crate) struct NestedCallRequest {
|
||||
pub(crate) withdraw: Option<CancellationToken>,
|
||||
}
|
||||
|
||||
impl NestedCallRequest {
|
||||
impl NestedToolCallRequest {
|
||||
/// Nobody is waiting for the answer any more.
|
||||
pub(crate) fn is_stale(&self) -> bool {
|
||||
self.reply.is_closed()
|
||||
@@ -314,6 +323,32 @@ impl NestedCallGate {
|
||||
.map(|gate| (&gate.caller, gate.specs.as_slice()))
|
||||
}
|
||||
|
||||
pub(crate) async fn checkpoint(
|
||||
&self,
|
||||
caller: crate::core::events::SessionCheckpointCaller,
|
||||
withdraw: CancellationToken,
|
||||
) -> Result<bool, ToolError> {
|
||||
if self.extension.is_none() {
|
||||
return Err(ToolError::not_available(
|
||||
"session checkpoint requires a directly served extension invocation",
|
||||
));
|
||||
}
|
||||
let (reply, answer) = oneshot::channel();
|
||||
let request = NestedCallRequest::Checkpoint {
|
||||
caller,
|
||||
withdraw: withdraw.clone(),
|
||||
reply,
|
||||
};
|
||||
tokio::select! {
|
||||
biased;
|
||||
() = withdraw.cancelled() => Err(ToolError::cancelled("session checkpoint was withdrawn")),
|
||||
result = async {
|
||||
self.requests.send(request).await.map_err(|_| ToolError::not_available("the turn no longer serves session checkpoints"))?;
|
||||
answer.await.map_err(|_| ToolError::not_available("owning host did not acknowledge the session checkpoint"))?
|
||||
} => result,
|
||||
}
|
||||
}
|
||||
|
||||
/// Ask the serving turn loop to decide one nested call. A gate nobody
|
||||
/// serves any more refuses.
|
||||
pub(crate) async fn ask(&self, name: String, input: Value) -> NestedCallVerdict {
|
||||
@@ -340,12 +375,12 @@ impl NestedCallGate {
|
||||
decision: NestedDecision::Refused,
|
||||
};
|
||||
let (reply, answer) = oneshot::channel();
|
||||
let request = NestedCallRequest {
|
||||
let request = NestedCallRequest::Tool(NestedToolCallRequest {
|
||||
name,
|
||||
input,
|
||||
reply,
|
||||
withdraw: withdraw.clone(),
|
||||
};
|
||||
});
|
||||
let wait = async {
|
||||
if self.requests.send(request).await.is_err() {
|
||||
return unavailable();
|
||||
@@ -386,6 +421,15 @@ impl NestedCallGate {
|
||||
let (gate, mut requests) = Self::new(None, tx_event, Duration::from_secs(60));
|
||||
tokio::spawn(async move {
|
||||
while let Some(request) = requests.recv().await {
|
||||
let request = match request {
|
||||
NestedCallRequest::Tool(request) => request,
|
||||
NestedCallRequest::Checkpoint { reply, .. } => {
|
||||
let _ = reply.send(Err(ToolError::not_available(
|
||||
"fixture has no session checkpoint backend",
|
||||
)));
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let _ = request.reply.send(answer(&request.name, &request.input));
|
||||
}
|
||||
});
|
||||
@@ -1603,6 +1647,15 @@ mod tests {
|
||||
let counter = asked.clone();
|
||||
tokio::spawn(async move {
|
||||
while let Some(request) = requests.recv().await {
|
||||
let request = match request {
|
||||
NestedCallRequest::Tool(request) => request,
|
||||
NestedCallRequest::Checkpoint { reply, .. } => {
|
||||
let _ = reply.send(Err(ToolError::not_available(
|
||||
"fixture has no session checkpoint backend",
|
||||
)));
|
||||
continue;
|
||||
}
|
||||
};
|
||||
counter.fetch_add(1, Ordering::SeqCst);
|
||||
tokio::time::sleep(delay).await;
|
||||
let verdict = answer(&request.name, &request.input);
|
||||
|
||||
@@ -47,6 +47,10 @@ pub enum PersistRequest {
|
||||
/// Write a crash-recovery checkpoint (in-flight turn state) to the
|
||||
/// session's own file (`checkpoints/<session_id>.json`).
|
||||
SaveCheckpoint { session: SavedSession },
|
||||
AcknowledgedCheckpoint {
|
||||
session: SavedSession,
|
||||
reply: oneshot::Sender<Result<(), String>>,
|
||||
},
|
||||
/// Write a full session snapshot (completed turn, durable save).
|
||||
SessionSnapshot(SavedSession),
|
||||
/// Compound completion commit: write the completed session snapshot,
|
||||
@@ -145,6 +149,7 @@ const ACTOR_COMMAND_CAPACITY: usize = 8;
|
||||
/// flushes outside it, so disk I/O never blocks a sender.
|
||||
#[derive(Debug, Default)]
|
||||
struct SharedPending {
|
||||
closed: bool,
|
||||
pending: PendingState,
|
||||
/// Whether a `WorkReady` is already queued (or being queued) and no
|
||||
/// `take_pending` has observed it since. Reset only by `take_pending` —
|
||||
@@ -196,6 +201,22 @@ fn persistence_request_channel() -> (PersistRequestSender, PersistRequestReceive
|
||||
)
|
||||
}
|
||||
|
||||
impl Drop for PersistRequestReceiver {
|
||||
fn drop(&mut self) {
|
||||
let mut guard = self
|
||||
.shared
|
||||
.lock()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner());
|
||||
guard.closed = true;
|
||||
guard.notified = false;
|
||||
for (_, reply) in std::mem::take(&mut guard.pending.checkpoint_replies) {
|
||||
let _ = reply.send(Err(
|
||||
"persistence actor shut down before checkpoint write".into()
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl PersistRequestReceiver {
|
||||
/// Await the next actor command. `None` means every sender is gone.
|
||||
async fn recv(&mut self) -> Option<ActorCommand> {
|
||||
@@ -231,8 +252,12 @@ impl PersistActorHandle {
|
||||
/// session), so repeated snapshots of one session retain only the
|
||||
/// newest. Returns `false` when the actor is already shut down.
|
||||
pub fn try_send(&self, mut request: PersistRequest) -> bool {
|
||||
if self.tx.cmd_tx.is_closed() {
|
||||
return false;
|
||||
}
|
||||
match &mut request {
|
||||
PersistRequest::SaveCheckpoint { session }
|
||||
| PersistRequest::AcknowledgedCheckpoint { session, .. }
|
||||
| PersistRequest::SessionSnapshot(session)
|
||||
| PersistRequest::CompletedCommit { session } => {
|
||||
session.compact_for_persistence_queue();
|
||||
@@ -245,6 +270,9 @@ impl PersistActorHandle {
|
||||
.shared
|
||||
.lock()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner());
|
||||
if guard.closed {
|
||||
return false;
|
||||
}
|
||||
guard.pending.absorb(request)
|
||||
};
|
||||
match control {
|
||||
@@ -358,6 +386,7 @@ pub(crate) fn flush_before_transition() -> Result<(), String> {
|
||||
fn request_label(request: &PersistRequest) -> &'static str {
|
||||
match request {
|
||||
PersistRequest::SaveCheckpoint { .. } => "SaveCheckpoint",
|
||||
PersistRequest::AcknowledgedCheckpoint { .. } => "AcknowledgedCheckpoint",
|
||||
PersistRequest::SessionSnapshot(_) => "SessionSnapshot",
|
||||
PersistRequest::CompletedCommit { .. } => "CompletedCommit",
|
||||
PersistRequest::OfflineQueue { .. } => "OfflineQueue",
|
||||
@@ -453,6 +482,7 @@ struct PendingState {
|
||||
/// (mirroring `sessions` below) so concurrent sessions can interleave
|
||||
/// saves and clears without clobbering each other.
|
||||
checkpoints: BTreeMap<String, SavedSession>,
|
||||
checkpoint_replies: BTreeMap<String, oneshot::Sender<Result<(), String>>>,
|
||||
/// Session ids whose checkpoint file should be removed.
|
||||
checkpoint_clears: BTreeSet<String>,
|
||||
/// Latest-wins per session id. Coalescing into one global slot can
|
||||
@@ -493,7 +523,25 @@ impl PendingState {
|
||||
}
|
||||
|
||||
fn absorb(&mut self, req: PersistRequest) -> Control {
|
||||
let replaced = match &req {
|
||||
PersistRequest::SaveCheckpoint { session }
|
||||
| PersistRequest::AcknowledgedCheckpoint { session, .. }
|
||||
| PersistRequest::SessionSnapshot(session)
|
||||
| PersistRequest::CompletedCommit { session } => Some(session.metadata.id.as_str()),
|
||||
PersistRequest::ClearCheckpoint { session_id } => Some(session_id.as_str()),
|
||||
_ => None,
|
||||
};
|
||||
if let Some(id) = replaced
|
||||
&& let Some(reply) = self.checkpoint_replies.remove(id)
|
||||
{
|
||||
let _ = reply.send(Err("checkpoint superseded before persistence".into()));
|
||||
}
|
||||
match req {
|
||||
PersistRequest::AcknowledgedCheckpoint { session, reply } => {
|
||||
let id = session.metadata.id.clone();
|
||||
self.absorb(PersistRequest::SaveCheckpoint { session });
|
||||
self.checkpoint_replies.insert(id, reply);
|
||||
}
|
||||
PersistRequest::SaveCheckpoint { session } => {
|
||||
// Last-writer-wins per session: a fresh checkpoint supersedes
|
||||
// a pending clear for the same session so the two never both
|
||||
@@ -641,10 +689,15 @@ fn flush_inner(
|
||||
);
|
||||
}
|
||||
for (session_id, session) in std::mem::take(&mut pending.checkpoints) {
|
||||
record(
|
||||
format!("checkpoint:{session_id}"),
|
||||
manager.save_checkpoint_owned(session).map(|_| ()),
|
||||
);
|
||||
let reply = pending.checkpoint_replies.remove(&session_id);
|
||||
if reply.as_ref().is_some_and(oneshot::Sender::is_closed) {
|
||||
continue;
|
||||
}
|
||||
let result = manager.save_checkpoint_owned(session).map(|_| ());
|
||||
if let Some(reply) = reply {
|
||||
let _ = reply.send(result.as_ref().map(|_| ()).map_err(ToString::to_string));
|
||||
}
|
||||
record(format!("checkpoint:{session_id}"), result);
|
||||
}
|
||||
for (_, request) in std::mem::take(&mut pending.offline_queue) {
|
||||
match request {
|
||||
@@ -861,6 +914,27 @@ mod tests {
|
||||
None,
|
||||
Some("agent"),
|
||||
);
|
||||
let (tx, mut receiver) = persistence_request_channel();
|
||||
let retained = PersistActorHandle { tx };
|
||||
assert!(retained.try_send(PersistRequest::Shutdown));
|
||||
assert!(matches!(
|
||||
receiver.recv().await,
|
||||
Some(ActorCommand::Shutdown)
|
||||
));
|
||||
assert!(receiver.take_pending().is_empty());
|
||||
let (reply, late_checkpoint) = oneshot::channel();
|
||||
assert!(retained.try_send(PersistRequest::AcknowledgedCheckpoint {
|
||||
session: session.clone(),
|
||||
reply,
|
||||
}));
|
||||
drop(receiver);
|
||||
assert!(late_checkpoint.await.expect("shutdown rejection").is_err());
|
||||
let (reply, after_shutdown) = oneshot::channel();
|
||||
assert!(!retained.try_send(PersistRequest::AcknowledgedCheckpoint {
|
||||
session: session.clone(),
|
||||
reply,
|
||||
}));
|
||||
assert!(after_shutdown.await.is_err());
|
||||
let session_id = session.metadata.id.clone();
|
||||
let (handle, task) = spawn_persistence_actor(manager);
|
||||
|
||||
@@ -987,12 +1061,38 @@ mod tests {
|
||||
None,
|
||||
Some("agent"),
|
||||
);
|
||||
for replacement in [
|
||||
PersistRequest::SaveCheckpoint {
|
||||
session: session.clone(),
|
||||
},
|
||||
PersistRequest::CompletedCommit {
|
||||
session: session.clone(),
|
||||
},
|
||||
PersistRequest::ClearCheckpoint {
|
||||
session_id: session.metadata.id.clone(),
|
||||
},
|
||||
] {
|
||||
let mut pending = PendingState::default();
|
||||
let (reply, receiver) = oneshot::channel();
|
||||
pending.absorb(PersistRequest::AcknowledgedCheckpoint {
|
||||
session: session.clone(),
|
||||
reply,
|
||||
});
|
||||
pending.absorb(replacement);
|
||||
assert!(receiver.await.expect("supersession reply").is_err());
|
||||
assert!(pending.checkpoint_replies.is_empty());
|
||||
}
|
||||
let (handle, task) = spawn_persistence_actor(manager);
|
||||
|
||||
handle.try_send(PersistRequest::SaveCheckpoint { session });
|
||||
let (checkpoint_reply, checkpoint_result) = oneshot::channel();
|
||||
assert!(handle.try_send(PersistRequest::AcknowledgedCheckpoint {
|
||||
session,
|
||||
reply: checkpoint_reply
|
||||
}));
|
||||
let (reply_tx, reply_rx) = tokio::sync::oneshot::channel();
|
||||
handle.try_send(PersistRequest::FlushAndReport { reply: reply_tx });
|
||||
let report = reply_rx.await.expect("flush report reply");
|
||||
assert!(checkpoint_result.await.expect("checkpoint reply").is_ok());
|
||||
// Whether the checkpoint was written by an earlier background cycle
|
||||
// or by this flush, the accumulated report must count it and show no
|
||||
// failures — and the actor keeps running afterwards.
|
||||
@@ -1076,10 +1176,15 @@ mod tests {
|
||||
let session_id = session.metadata.id.clone();
|
||||
let (handle, task) = spawn_persistence_actor(manager);
|
||||
|
||||
handle.try_send(PersistRequest::SaveCheckpoint { session });
|
||||
let (checkpoint_reply, checkpoint_result) = oneshot::channel();
|
||||
assert!(handle.try_send(PersistRequest::AcknowledgedCheckpoint {
|
||||
session,
|
||||
reply: checkpoint_reply
|
||||
}));
|
||||
let (reply_tx, reply_rx) = tokio::sync::oneshot::channel();
|
||||
handle.try_send(PersistRequest::FlushAndReport { reply: reply_tx });
|
||||
let report = reply_rx.await.expect("flush report reply");
|
||||
assert!(checkpoint_result.await.expect("checkpoint reply").is_err());
|
||||
|
||||
assert!(
|
||||
report
|
||||
|
||||
@@ -111,10 +111,28 @@ pub(super) fn apply_engine_session_projection(
|
||||
instruction_assembly,
|
||||
model,
|
||||
workspace,
|
||||
checkpoint,
|
||||
} = event
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
if let Some(request) = checkpoint.as_ref() {
|
||||
if request.is_stale()
|
||||
|| request.caller.agent_id.is_some()
|
||||
|| app.current_session_id.as_deref() != Some(request.caller.session_id.as_str())
|
||||
|| app.runtime_turn_id.as_deref() != Some(request.caller.turn_id.as_str())
|
||||
|| app.workspace != request.caller.workspace
|
||||
|| !app.is_loading
|
||||
|| app.suppress_stream_events_until_turn_complete
|
||||
{
|
||||
request.fail("checkpoint does not belong to the active terminal turn");
|
||||
return false;
|
||||
}
|
||||
if let Err(error) = request.validate_prefix(&session_id, &workspace, &messages) {
|
||||
request.fail(error);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
// SetModel can emit the old session while a host-owned
|
||||
// SyncSession is still queued. Reject that entire stale
|
||||
// projection before changing transcript or persistence.
|
||||
@@ -152,6 +170,67 @@ pub(super) fn apply_engine_session_projection(
|
||||
if app.workspace != workspace {
|
||||
apply_workspace_runtime_state(app, config, workspace);
|
||||
}
|
||||
if let Some(request) = checkpoint {
|
||||
let prepared = SessionManager::default_location()
|
||||
.map_err(|error| error.to_string())
|
||||
.and_then(|manager| build_session_snapshot(app, &manager));
|
||||
let mut session = match prepared {
|
||||
Ok(session) => session,
|
||||
Err(error) => {
|
||||
request.fail(error);
|
||||
return false;
|
||||
}
|
||||
};
|
||||
session.make_storage_compatible();
|
||||
if let Err(error) = request.validate_prefix(
|
||||
&session.metadata.id,
|
||||
&session.metadata.workspace,
|
||||
&session.messages,
|
||||
) {
|
||||
request.fail(error);
|
||||
return false;
|
||||
}
|
||||
let (reply, receiver) = tokio::sync::oneshot::channel();
|
||||
if !crate::tui::persistence_actor::try_persist(PersistRequest::AcknowledgedCheckpoint {
|
||||
session,
|
||||
reply,
|
||||
}) {
|
||||
request.fail("terminal persistence actor is unavailable");
|
||||
return false;
|
||||
}
|
||||
if app
|
||||
.runtime_services
|
||||
.work
|
||||
.as_ref()
|
||||
.is_some_and(|work| work.has_pending_publish())
|
||||
&& let Err(error) = app.publish_pending_work_state()
|
||||
{
|
||||
request.fail(format!(
|
||||
"checkpoint queued but Work projection failed: {error}"
|
||||
));
|
||||
return false;
|
||||
}
|
||||
codewhale_util::utils::spawn_supervised(
|
||||
"terminal-session-checkpoint",
|
||||
std::panic::Location::caller(),
|
||||
async move {
|
||||
tokio::select! {
|
||||
biased;
|
||||
() = request.withdraw.cancelled() => request.fail("checkpoint withdrawn; a write already in progress may have completed"),
|
||||
result = receiver => match result {
|
||||
Ok(Ok(())) => request.succeed(crate::core::events::SessionCheckpointReceipt {
|
||||
persisted_sha256: request.prefix_sha256.clone(),
|
||||
messages_len: request.snapshot.messages.len(),
|
||||
projection: crate::core::events::SessionCheckpointProjection::Exact,
|
||||
}),
|
||||
Ok(Err(error)) => request.fail(error),
|
||||
Err(_) => request.fail("terminal persistence actor dropped checkpoint acknowledgement"),
|
||||
},
|
||||
}
|
||||
},
|
||||
);
|
||||
return true;
|
||||
}
|
||||
if (app.is_loading || app.is_compacting || app.is_purging)
|
||||
&& let Ok(manager) = SessionManager::default_location()
|
||||
{
|
||||
@@ -2505,6 +2584,16 @@ pub(crate) async fn run_event_loop(
|
||||
// cancel redraws owed to other events in the same batch.
|
||||
let redraw_requested_before_event = received_engine_event;
|
||||
received_engine_event = true;
|
||||
if matches!(
|
||||
&event,
|
||||
EngineEvent::SessionUpdated {
|
||||
checkpoint: Some(_),
|
||||
..
|
||||
}
|
||||
) {
|
||||
apply_engine_session_projection(app, config, event);
|
||||
continue;
|
||||
}
|
||||
capture_turn_started_metadata(app, &event);
|
||||
observe_human_request_settlement(app, &event);
|
||||
// Child approval bookkeeping runs before every filter: it is
|
||||
|
||||
@@ -952,6 +952,7 @@ pub(crate) fn build_engine_config(app: &App, config: &Config) -> EngineConfig {
|
||||
.max_subagents
|
||||
.clamp(1, codewhale_runtime::config::MAX_SUBAGENTS);
|
||||
EngineConfig {
|
||||
session_checkpoint_backend: Some(crate::core::events::SessionCheckpointProjection::Exact),
|
||||
model: app.model.clone(),
|
||||
active_route_limits: app.active_route_limits,
|
||||
workspace: app.workspace.clone(),
|
||||
|
||||
@@ -13973,6 +13973,7 @@ async fn a_steer_settles_even_though_the_engine_stored_it_trimmed() {
|
||||
&mut app,
|
||||
&Config::default(),
|
||||
EngineEvent::SessionUpdated {
|
||||
checkpoint: None,
|
||||
session_id: session,
|
||||
messages: std::sync::Arc::new(vec![
|
||||
text_message("user", "original request"),
|
||||
@@ -14051,6 +14052,7 @@ async fn steer_becomes_the_newest_transcript_entry_only_when_the_engine_records_
|
||||
&mut app,
|
||||
&Config::default(),
|
||||
EngineEvent::SessionUpdated {
|
||||
checkpoint: None,
|
||||
session_id: session,
|
||||
messages: std::sync::Arc::new(vec![
|
||||
text_message("user", "original request"),
|
||||
@@ -19104,6 +19106,7 @@ fn local_cancel_marks_late_stream_events_for_suppression() {
|
||||
));
|
||||
assert!(suppress_engine_event_after_local_cancel(
|
||||
&EngineEvent::SessionUpdated {
|
||||
checkpoint: None,
|
||||
session_id: "session".to_string(),
|
||||
messages: std::sync::Arc::new(Vec::new()),
|
||||
system_prompt: None,
|
||||
@@ -33493,6 +33496,7 @@ fn stale_session_projection_cannot_rewind_a_host_launch() {
|
||||
&mut app,
|
||||
&Config::default(),
|
||||
EngineEvent::SessionUpdated {
|
||||
checkpoint: None,
|
||||
session_id: previous,
|
||||
messages: std::sync::Arc::new(vec![text_message("user", "stale transcript")]),
|
||||
system_prompt: Some(SystemPrompt::Text("stale prompt".into())),
|
||||
@@ -33515,10 +33519,96 @@ fn stale_session_projection_cannot_rewind_a_host_launch() {
|
||||
assert_eq!(app.input, "typing remains responsive");
|
||||
assert_eq!(app.history.len(), history_length);
|
||||
assert_eq!(app.status_toasts.len(), notice_count);
|
||||
app.runtime_turn_id = Some("checkpoint-turn".into());
|
||||
app.is_loading = true;
|
||||
for invalid in ["session", "turn", "workspace", "child", "withdrawn"] {
|
||||
let snapshot = std::sync::Arc::new(crate::core::ops::SessionSnapshot {
|
||||
session_id: if invalid == "session" {
|
||||
"stale-session".into()
|
||||
} else {
|
||||
current.clone()
|
||||
},
|
||||
messages: vec![text_message("user", "must not replace the transcript")],
|
||||
total_tokens: 0,
|
||||
model: model.clone(),
|
||||
model_provider: "fixture".into(),
|
||||
model_provider_id: None,
|
||||
workspace: if invalid == "workspace" {
|
||||
PathBuf::from("/must-not-be-adopted")
|
||||
} else {
|
||||
workspace.clone()
|
||||
},
|
||||
system_prompt: None,
|
||||
instruction_assembly: None,
|
||||
mode: "agent".into(),
|
||||
});
|
||||
let (reply, mut answer) = tokio::sync::oneshot::channel();
|
||||
let request = crate::core::events::SessionCheckpointRequest::new(
|
||||
crate::core::events::SessionCheckpointCaller {
|
||||
session_id: snapshot.session_id.clone(),
|
||||
agent_id: (invalid == "child").then(|| "child".into()),
|
||||
turn_id: if invalid == "turn" {
|
||||
"wrong-turn".into()
|
||||
} else {
|
||||
"checkpoint-turn".into()
|
||||
},
|
||||
call_id: "checkpoint-call".into(),
|
||||
workspace: snapshot.workspace.clone(),
|
||||
},
|
||||
snapshot.clone(),
|
||||
crate::core::events::SessionCheckpointRequest::prefix_hash(&snapshot.messages).unwrap(),
|
||||
crate::core::events::SessionCheckpointProjection::Exact,
|
||||
tokio_util::sync::CancellationToken::new(),
|
||||
reply,
|
||||
);
|
||||
if invalid == "withdrawn" {
|
||||
request.withdraw.cancel();
|
||||
}
|
||||
assert!(
|
||||
!super::event_loop::apply_engine_session_projection(
|
||||
&mut app,
|
||||
&Config::default(),
|
||||
EngineEvent::SessionUpdated {
|
||||
checkpoint: Some(request),
|
||||
session_id: snapshot.session_id.clone(),
|
||||
messages: std::sync::Arc::new(snapshot.messages.clone()),
|
||||
system_prompt: None,
|
||||
instruction_assembly: None,
|
||||
model: snapshot.model.clone(),
|
||||
workspace: snapshot.workspace.clone(),
|
||||
},
|
||||
),
|
||||
"{invalid}"
|
||||
);
|
||||
assert!(
|
||||
answer
|
||||
.try_recv()
|
||||
.expect("explicit checkpoint refusal")
|
||||
.is_err(),
|
||||
"{invalid}"
|
||||
);
|
||||
assert_eq!(app.current_session_id.as_deref(), Some(current.as_str()));
|
||||
assert_eq!(
|
||||
*app.api_messages,
|
||||
vec![text_message("user", "current host transcript")]
|
||||
);
|
||||
assert_eq!(
|
||||
app.system_prompt,
|
||||
Some(SystemPrompt::Text("current prompt".into()))
|
||||
);
|
||||
assert_eq!(app.workspace, workspace);
|
||||
assert_eq!(app.model, model);
|
||||
assert_eq!(app.input, "typing remains responsive");
|
||||
assert_eq!(app.history.len(), history_length);
|
||||
assert_eq!(app.status_toasts.len(), notice_count);
|
||||
}
|
||||
app.runtime_turn_id = None;
|
||||
app.is_loading = false;
|
||||
assert!(super::event_loop::apply_engine_session_projection(
|
||||
&mut app,
|
||||
&Config::default(),
|
||||
EngineEvent::SessionUpdated {
|
||||
checkpoint: None,
|
||||
session_id: current.clone(),
|
||||
messages: std::sync::Arc::new(vec![
|
||||
text_message("user", "expected engine transcript",)
|
||||
|
||||
+19
-3
@@ -217,9 +217,25 @@ Tool and command invocations also expose frozen `sessionId`, `agentId` and
|
||||
`originTurnId` strings when Rust supplies them for that particular call. Tool invocations
|
||||
with an authoritative session and workspace also expose frozen `exec.agent.id`
|
||||
and `exec.agent.session.header.cwd`; root identity uses the session id and a
|
||||
child uses its supplied agent id. This is an inert read-only projection, not a
|
||||
general agent handle or session persistence service. Retaining it grants no
|
||||
operations after the invocation ends.
|
||||
child uses its supplied agent id. The projection has no methods and retaining it
|
||||
grants no operations after the invocation ends.
|
||||
|
||||
An entry injecting `sessions` can call `ctx.sessions.flush(exec.agent.session)`
|
||||
while its directly invoked root tool holds an Engine checkpoint ticket. The
|
||||
argument must be that invocation's exact session object. The Engine captures
|
||||
its committed prefix and waits for the owning Runtime, terminal, or exec store
|
||||
to acknowledge persistence. Runtime retains its existing receipt-secret
|
||||
redaction policy; success describes that persisted projection. It does not
|
||||
include the running tool's future result or the plugin's separate journal.
|
||||
Runtime recovery retains later tool results without replaying the tool.
|
||||
Checkpoint-bearing Runtime turns require schema version 5; older readers
|
||||
refuse them rather than silently reconstructing incomplete history.
|
||||
|
||||
`false` is reserved for a deliberately ephemeral host with no persistence
|
||||
backend. Missing authority, foreign or expired handles, child checkpoints,
|
||||
superseded writes, storage failure, cancellation and unavailable consumers
|
||||
reject. Cancellation can race a disk write, so a rejected call does not prove
|
||||
nothing was written. Other `sessions` methods remain unsupported.
|
||||
|
||||
## MCP definitions
|
||||
|
||||
|
||||
@@ -157,8 +157,19 @@ Rust 未提供的字段缺省;激活时不缓存会话身份。这些字符串
|
||||
会话或 agent 的权利。用户命令通常只有已知的 session 标签。
|
||||
当 Rust 提供会话和工作区时,工具调用还包含冻结的 `exec.agent.id` 和
|
||||
`exec.agent.session.header.cwd`;根身份使用会话 id,子 agent 使用 Core 提供的 id。
|
||||
这只是只读投影,不提供通用 agent 操作或会话持久化服务,保留该对象也不会获得
|
||||
调用结束后的操作权限。
|
||||
投影本身没有方法;保留该对象不会获得调用结束后的操作权限。
|
||||
|
||||
注入 `sessions` 的插件可在直接调用的根工具持有 Engine checkpoint ticket 时调用
|
||||
`ctx.sessions.flush(exec.agent.session)`,参数必须是本次调用的原始 session 对象。
|
||||
Engine 捕获已提交的前缀,由所属 Runtime、终端或 exec 存储实际写入并确认。
|
||||
Runtime 保留现有的回执秘密脱敏规则;成功确认的是持久化投影,不包含当前工具
|
||||
未来的结果或插件自己的日志。恢复时保留后续工具结果,不重新执行工具。
|
||||
含 checkpoint 的 Runtime turn 使用 schema 5,旧读取器会拒绝,避免静默丢失历史。
|
||||
|
||||
只有明确选择无持久化后端的临时宿主返回 `false`。缺少权限、外来或过期对象、
|
||||
子 agent checkpoint、被后续请求替代的写入、存储失败、取消及不可用的消费者均拒绝。
|
||||
取消可能与磁盘写入竞争,因此拒绝不能证明磁盘完全未写入。其他 `sessions` 方法
|
||||
仍未支持。
|
||||
|
||||
当 Rust 提供调用者会话时,`invocation.agent` 另外提供只读 id 和
|
||||
`followup(message)`。一次调用只接收一条 JSON user 消息,包含有界 id、仅有 kind
|
||||
|
||||
Reference in new issue
Block a user