mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-07 10:36:05 -10:00
Merge current Engine source and resolve React dependency updates
Preserve original React19.3.1/DOM and types19.3.0 intent, with current Stripe23.1.0 and Node types26.6.6. Structured three-way dependency delta, root/lock consistency, exact dependency pins and diff checks pass; hosted PR CI supplies runtime/build qualification.
This commit is contained in:
commit
290d3e4a4f
272 files changed
+21470
-10684
No files matched your search
@@ -169,11 +169,11 @@ casually it reads as taking the work even when credit is preserved.
|
||||
- The model-facing subagent tool is `agent`; `agent_open`/`agent_eval`/
|
||||
`agent_close`/`delegate_to_agent` are removed surfaces. If the shape must
|
||||
move, move the code and its existing guard test with it.
|
||||
- `BASE_PROMPT` in `crates/tui/src/prompts/text.rs` is the sole base prompt
|
||||
- `BASE_PROMPT` in `crates/engine/src/prompts/text.rs` is the sole base prompt
|
||||
by convention. Same rule: move the code, not the prose, if that changes.
|
||||
- There is exactly one turn loop: `Engine::run_turn` in
|
||||
`crates/tui/src/core/engine/turn_loop.rs`. Note that `crates/tui/src/core/`
|
||||
is a module inside the TUI crate — it is not `crates/core`, which owns
|
||||
`crates/engine/src/core/engine/turn_loop.rs`. Note that `crates/engine/src/core/`
|
||||
is a module inside the engine crate (`codewhale-engine`) — it is not `crates/core`, which owns
|
||||
request construction, bounded fragments, and thread/session types and
|
||||
runs no turns. A guard test (`crates/core/tests/single_turn_loop.rs`)
|
||||
fails on a second loop; changing the shape means changing the guard
|
||||
@@ -184,7 +184,7 @@ casually it reads as taking the work even when credit is preserved.
|
||||
volatile fact into the prefix; append it as a user-role message.
|
||||
- These active modules are repeatedly misidentified as dead; verify consumers
|
||||
before removal: `runtime/src/context_budget.rs`,
|
||||
`runtime/src/prompt_zones.rs`, `tui/src/tools/remember.rs`, and
|
||||
`runtime/src/prompt_zones.rs`, `engine/src/tools/remember.rs`, and
|
||||
`config/src/route/`. Native memory lives in `runtime/src/native_memory.rs`;
|
||||
`tools/remember.rs` is its capture path.
|
||||
- Environment-specific behavior belongs in `docs/ENVIRONMENTS.md`, not here.
|
||||
|
||||
+29
-1
@@ -75,6 +75,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Changed
|
||||
|
||||
- Internal coarse split: shared bounded owner work and frontend selection types live in Runtime, protected socket layout lives in Paths, and Engine/terminal avatars share one Protocol data contract. Existing App Server and terminal-kit aliases remain available. Engine and Sandbox no longer depend on App Server; the host-band move and Stage B remain open (#5586).
|
||||
- **Newer DeepSeek Harness plugin compatibility:** the Native host targets
|
||||
DSH 0.2.1-rc.5 with Cordis 4.0.5 and refreshed reviewed composition
|
||||
packages. Boolean `deferLoading` hints are accepted while the Engine keeps
|
||||
@@ -159,6 +160,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Fixed
|
||||
|
||||
- Session publication compares the document and crash checkpoint captured before work was queued. A delayed save refuses changed records without replacing newer history or clearing its checkpoint. Ordered completion retains real checkpoint acknowledgments, and native turn/control identity follows restores, undo and terminal saves. Legacy checkpoints require explicit recovery; canonical history recovery retains its original source and exact prepared generation.
|
||||
- Damaged usage sidecars keep valid recovery data readable with incomplete coverage and no publication receipt. Known retirement, changed raw records and unsafe writes still refuse.
|
||||
- Local control and ACP output serialize within the existing 8 MiB frame limit and stop a damaged or stalled output stream. Compatibility replies, events and relay stdout now share the 30-second write deadline; owner forwarding keeps its tighter 5 seconds. Standalone and attached ACP recover after complete invalid UTF-8 or oversized input frames, discard oversized tails through the real newline, and preserve cancellation, permissions and actual Core receipts.
|
||||
- MCP composition records actual schema or description changes before the next request, so a conservative batch withdrawal re-pins the declared tool surface instead of failing the C5 prefix guard. Semantically unchanged runtime refreshes preserve the original serialized catalog and schema key order. The `execute_tools` description also explains that visual input needs a direct capture call because nested results do not deliver inline images to the model.
|
||||
- Runtime SDK declarations for event, image and dynamic-tool messages are generated from the same Rust wire contracts. Output schemas retain required nullable fields while request defaults remain optional.
|
||||
- Inference requests stop after uncertain delivery or an interrupted stream instead of silently sending another request. Automatic retries require an ordinary rate limit or a transient connection failure before dispatch. Redirects and HTTP protocol retries cannot bypass this policy; retained output and missing usage coverage remain visible. Terminal `exec` failures return exit 1; error categories no longer imply that repeating a saved command is safe.
|
||||
|
||||
- MCP calls inside `execute_tools` expose a single bare JSON result directly through `content`. Rich or annotated results retain their envelopes, and oversized results keep the original raw size and spill receipt.
|
||||
|
||||
- Session metadata reads stop at a bounded scan and report unreadable records without hiding healthy picker rows. Queued saves preserve durable renames, archive state and Runtime ownership under the session write lock; uncertain metadata refuses writes and cleanup. Persistence batches run off the async executor, and explicit placeholder-title renames retain their intent.
|
||||
|
||||
- Worker coordination serializes durable admissions across processes, preserves foreign workers and winning completion receipts, and fences stale checkpoints. Fleet restart reservations retain uncertain prior processes until exact teardown is confirmed; final output, usage and the receipt publish together without duplicate accounting on retry. Retained transcripts follow committed ownership.
|
||||
|
||||
- JSONPath projections bound path length, traversal steps and intermediate matches before cloning results. Direct fetches share the same parser and enforce one aggregate serialized-field budget; oversized selections refuse with narrower-selection guidance.
|
||||
|
||||
- Long tool output and diffs use bounded display windows and a byte-limited row cache. Retained details load verified byte ranges and label what search and copy include; explicit source copy keeps stored text with control-sequence cleanup. Oversized Unicode is shown safely without changing question answers; long answers retain their exact bytes in the existing owned output artifact before bounded model-context projection. New pager notices are translated across all 15 complete language packs.
|
||||
- Managed Fleet validates overlapping write roots with a component-sorted scan and deduplicates roots once. Large valid request bodies no longer spend quadratic CPU before admission; existing write-scope refusals and execution limits remain in force (#6554).
|
||||
- MCP transports, connection pools and child Engines retain the owning host credential context for placeholders, headers, bearer tokens and OAuth setup. Missing injected values never fall back to process secrets. Reviewed plugins keep their captured environment authority, whose Debug output now masks values. Durable OAuth token custody and full subprocess isolation remain separate work (#5637).
|
||||
- Python REPL drains stderr while code runs, retains at most 64 KiB of diagnostics and reports omitted bytes, so noisy code cannot block request completion (#6554).
|
||||
- Saved tool results use bounded streaming inspection while preserving session ownership, digest and generation checks. Text refuses oversized lines or selections with byte-recovery guidance; verification also has explicit scan and cancellation limits (#6554).
|
||||
- Telemetry bounds its pending event queue and disk-tail reads, keeps final flush admission available, and skips invalid UTF-8 records without discarding intact neighboring records. Existing opt-out and privacy ordering remain in force (#6554).
|
||||
- Workflow history summaries retain large child totals without allocating one row per claimed child. Missing outcomes, usage and elapsed time remain unknown, including in the workbar (#6554). The independent terminal kit advances to unpublished 0.2.0 for its `u64` total API; registry-based TUI packaging waits for separately approved kit publication.
|
||||
- Web build dependencies cap glob preprocessing and stop cancelled directory traversals through compatible, verified upstream backports. Raw dependency findings remain visible. Thanks [@isaacs](https://github.com/isaacs) for the [minimatch](https://github.com/isaacs/minimatch/commit/77a0f5c4b80bdc641c3c930a1d9edf5ad2db57f8) and [glob](https://github.com/isaacs/node-glob/commit/4682e8c9babee1b385202ede36389634e12076d7) fixes (#6560).
|
||||
- Repeated requests no longer make `/undo` restore files from an older invocation. `/undo select` chooses a request range and shows checking, restoring and saving progress. Missing, ambiguous or stale restore points refuse before changing files. Esc–Esc continues to rewind only the conversation. Thanks [@luestr](https://github.com/luestr) for reporting [#6970](https://github.com/codewhale-hq/Codewhale/issues/6970).
|
||||
- `web.run` refuses more than 64 operations across its command arrays before starting network work or changing its session cache (#6554).
|
||||
- Skill discovery and audit marker reads stop at 512 KiB. Approved linked global skills still load; oversized, invalid and non-regular files produce visible refusals instead of unbounded reads (#6554).
|
||||
- OpenSandbox caps successful responses at 8 MiB before parsing and limits remote error details to 4 KiB with terminal controls removed. Oversized responses fail explicitly. Remote execution cancellation remains a separate requirement (#6554).
|
||||
@@ -318,13 +343,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Security
|
||||
|
||||
- Saved tool-result reads refuse links in the selected session, including a linked session directory, while preserving deliberately relocated application storage. Owned files and digest-checked legacy results keep their existing access paths.
|
||||
- **Pinned marketplace archives:** a Codewhale or Claude marketplace entry whose tarball source carries a `sha256` now installs only when the downloaded bytes match it; a mismatch, a malformed pin or a pin on a non-archive source refuses the install with nothing written. The pin also applies to direct installs as `https://…/x.tar.gz#sha256=<hex>`.
|
||||
- **Sandboxed commands cannot plant git hooks or edit git config:** in workspace-write mode, `.git/hooks` and `.git/config` (and a linked worktree's shared `hooks` and `config`) are read-only to sandboxed commands, so a command cannot leave a hook or `core.hooksPath` that later runs outside the sandbox. The workspace's `.codewhale/` is protected even before it exists (on Linux it is created first, because bubblewrap can only protect existing paths). `git commit` and ordinary file writes are unaffected; `git config` writes and hook edits need an approved unsandboxed command.
|
||||
- **Tool image validation:** an image returned by a tool must decode before it is sent, so truncated or corrupt files are omitted with a receipt even when they are already within the delivery budget.
|
||||
|
||||
### Maintenance
|
||||
|
||||
- **`codewhale-ratatui` is re-versioned to 0.1.3:** the vendored terminal kit gained the `Accent` role after 0.1.2 was published, and `codewhale-tui` uses it, so crates.io needs the new version before `codewhale-tui` can build from the registry. `crates/tui` and `pet/tui` pin 0.1.3; publishing it stays a release-time step.
|
||||
- MCP regression fixtures retain private extension-host roots and wait for their own workers to finish, preventing parallel checks from borrowing a deleted fixture home. Production connection and credential policy is unchanged.
|
||||
- **Independent terminal kit candidate 0.2.0:** its workflow totals use `u64`, and its existing avatar API reexports the shared Protocol contract. All local consumers pin 0.2.0. Protocol 0.10.3 must be published and verified before the kit, followed by the dependent Engine packages; each publication retains its separate approval.
|
||||
- **`codewhale-plugins` crate:** plugin discovery, review, install and marketplace parsing move out of `codewhale-tui` into a new crate; MCP server config types move to `codewhale-mcp`, and the skill parse core and recommendations to `codewhale-skill-install`. Review and trust behavior is unchanged. (S19, #5316)
|
||||
- **`codewhale-config` modules (internal):** the 8,100-line `lib.rs` is split into 23 modules (provider TOML, permissions, fleet, config resolution, env overrides, state dirs, config store and backup, and others); `codewhale_config::` paths are unchanged. No behavior change. (S57, #5316)
|
||||
- **`codewhale-app-server` modules (internal):** `lib.rs` is carved into `http`, `stdio`, `turns`, `bridge` and `config_rpc` modules; `BoundedLines` and `runtime_compatibility_router` now live at `codewhale_app_server::stdio` and `::http`. No behavior change. (S49, #5316)
|
||||
@@ -357,6 +384,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Contributors
|
||||
|
||||
- **[@isaacs](https://github.com/isaacs)** — upstream minimatch preprocessing and glob cancellation fixes reused in compatible web build dependency backports.
|
||||
- **[@Gabriel-Degret](https://github.com/Gabriel-Degret)** — reported that a hosted output limit truncated reasoning on a custom endpoint ([#7179](https://github.com/codewhale-hq/Codewhale/issues/7179)).
|
||||
- **[@Felixkw12](https://github.com/Felixkw12)** — added Opper as a bundled provider ([#7131](https://github.com/codewhale-hq/Codewhale/pull/7131)).
|
||||
- **[@7jrxt42BxFZo4iAnN4CX](https://github.com/7jrxt42BxFZo4iAnN4CX)** — per-scope instruction budgets with visible truncation ([#7086](https://github.com/codewhale-hq/Codewhale/pull/7086), [#6526](https://github.com/codewhale-hq/Codewhale/issues/6526)).
|
||||
|
||||
Generated
+20
-4
@@ -872,6 +872,7 @@ dependencies = [
|
||||
name = "codewhale-app-server"
|
||||
version = "0.10.2"
|
||||
dependencies = [
|
||||
"codewhale-app-server-protocol",
|
||||
"anyhow",
|
||||
"axum",
|
||||
"chrono",
|
||||
@@ -882,6 +883,7 @@ dependencies = [
|
||||
"codewhale-paths",
|
||||
"codewhale-protocol",
|
||||
"codewhale-release",
|
||||
"codewhale-runtime",
|
||||
"codewhale-state",
|
||||
"reqwest",
|
||||
"rusqlite",
|
||||
@@ -899,6 +901,15 @@ dependencies = [
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "codewhale-app-server-protocol"
|
||||
version = "0.10.2"
|
||||
dependencies = [
|
||||
"schemars",
|
||||
"serde",
|
||||
"serde_json",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "codewhale-build-support"
|
||||
version = "0.10.2"
|
||||
@@ -1051,6 +1062,7 @@ dependencies = [
|
||||
name = "codewhale-engine"
|
||||
version = "0.10.2"
|
||||
dependencies = [
|
||||
"codewhale-app-server-protocol",
|
||||
"anyhow",
|
||||
"arboard",
|
||||
"async-stream",
|
||||
@@ -1061,7 +1073,6 @@ dependencies = [
|
||||
"clap",
|
||||
"codewhale-agent",
|
||||
"codewhale-agent-contract",
|
||||
"codewhale-app-server",
|
||||
"codewhale-build-support",
|
||||
"codewhale-command-contract",
|
||||
"codewhale-commands",
|
||||
@@ -1081,7 +1092,6 @@ dependencies = [
|
||||
"codewhale-paths",
|
||||
"codewhale-plugins",
|
||||
"codewhale-protocol",
|
||||
"codewhale-ratatui",
|
||||
"codewhale-release",
|
||||
"codewhale-repl",
|
||||
"codewhale-runtime",
|
||||
@@ -1185,6 +1195,7 @@ dependencies = [
|
||||
name = "codewhale-hooks"
|
||||
version = "0.10.2"
|
||||
dependencies = [
|
||||
"codewhale-app-server-protocol",
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
"chrono",
|
||||
@@ -1336,7 +1347,10 @@ dependencies = [
|
||||
name = "codewhale-paths"
|
||||
version = "0.10.2"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"dirs 7.0.0",
|
||||
"sha2 0.11.1",
|
||||
"thiserror 2.0.21",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1410,6 +1424,7 @@ dependencies = [
|
||||
name = "codewhale-protocol"
|
||||
version = "0.10.2"
|
||||
dependencies = [
|
||||
"codewhale-app-server-protocol",
|
||||
"chrono",
|
||||
"serde",
|
||||
"serde_json",
|
||||
@@ -1418,8 +1433,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "codewhale-ratatui"
|
||||
version = "0.1.3"
|
||||
version = "0.2.0"
|
||||
dependencies = [
|
||||
"codewhale-protocol",
|
||||
"crossterm",
|
||||
"libc",
|
||||
"ratatui",
|
||||
@@ -1525,7 +1541,6 @@ dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
"chrono",
|
||||
"codewhale-app-server",
|
||||
"codewhale-execpolicy",
|
||||
"codewhale-paths",
|
||||
"codewhale-release",
|
||||
@@ -1688,6 +1703,7 @@ dependencies = [
|
||||
name = "codewhale-tui"
|
||||
version = "0.10.2"
|
||||
dependencies = [
|
||||
"codewhale-app-server-protocol",
|
||||
"ahash",
|
||||
"ansi-to-tui",
|
||||
"anyhow",
|
||||
|
||||
@@ -4,6 +4,7 @@ members = [
|
||||
"crates/agent",
|
||||
"crates/agent-contract",
|
||||
"crates/app-server",
|
||||
"crates/app-server-protocol",
|
||||
"crates/build-support",
|
||||
"crates/cli",
|
||||
"crates/cloud-facts",
|
||||
|
||||
+3
-3
@@ -1180,9 +1180,9 @@ base_url = "https://api.deepseek.com/beta"
|
||||
[stream]
|
||||
open_timeout_secs = 45 # response headers, including connect (0 = env/default, 5-300)
|
||||
chunk_timeout_secs = 900 # per-SSE-chunk idle bound (0 = default, 1-3600)
|
||||
max_resumes = 3 # whole-request reissues after failed streams (0 = off, 0-10)
|
||||
max_transparent_retries = 2 # retry before any content (0 = off, 0-10)
|
||||
max_stream_errors = 5 # recoverable errors per stream (0 = default, 1-50)
|
||||
max_resumes = 3 # proven pre-dispatch connection recovery (0 = off, 0-10)
|
||||
max_transparent_retries = 2 # compatibility value; interrupted streams never reissue
|
||||
max_stream_errors = 5 # compatibility value; uncertain stream errors end the stream
|
||||
max_duration_secs = 1800 # one stream, not the whole turn (0 = default, 10-86400)
|
||||
max_content_mb = 10 # one stream's content (0 = default, 1-512)
|
||||
connect_timeout_secs = 30 # TCP/TLS setup (0 = default, 1-300)
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
[package]
|
||||
name = "codewhale-app-server-protocol"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
rust-version.workspace = true
|
||||
license.workspace = true
|
||||
repository.workspace = true
|
||||
description = "Client wire contracts and schema exports for Codewhale App Server"
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
|
||||
[features]
|
||||
schema = ["dep:schemars"]
|
||||
|
||||
[dependencies]
|
||||
serde.workspace = true
|
||||
serde_json = { workspace = true, features = ["preserve_order"] }
|
||||
schemars = { version = "1.2.1", optional = true, features = ["derive", "preserve_order"] }
|
||||
|
||||
[[bin]]
|
||||
name = "codewhale-app-server-schema"
|
||||
path = "src/bin/export.rs"
|
||||
required-features = ["schema"]
|
||||
@@ -0,0 +1,33 @@
|
||||
# Codewhale App Server protocol
|
||||
|
||||
This crate owns the existing Runtime image, event-envelope, capability and
|
||||
dynamic-tool contracts. Engine, App Server, hooks and the SDK consume that one
|
||||
family; the former `codewhale_protocol::runtime` module is removed.
|
||||
|
||||
The default dependency graph is serde-only. The optional `schema` feature uses
|
||||
the existing locked schemars dependency and shared JSON Schema to TypeScript
|
||||
renderer. It selects input/output schema contracts from the actual owner. Runtime output
|
||||
fields reflect serialization while accepted input defaults stay optional.
|
||||
DynamicToolSpecOutput is the serialized view of the same DynamicToolSpec Rust
|
||||
type used in thread-created output. Runtime TypeScript uses JSON-serializable
|
||||
object members (which may omit undefined values); root/array undefined remains
|
||||
excluded. The existing Native host generator keeps its prior text and
|
||||
present-value normalization.
|
||||
|
||||
Generate outputs with:
|
||||
|
||||
```sh
|
||||
cargo run -p codewhale-app-server-protocol --features schema --bin codewhale-app-server-schema -- --write
|
||||
```
|
||||
|
||||
The command writes `schema/json/runtime.schema.json`,
|
||||
`schema/typescript/runtime.d.ts`, and the matching packaged SDK declaration.
|
||||
`--check` compares all three without writing. Normal runtime startup does not
|
||||
generate files or enable schema machinery.
|
||||
|
||||
The selected future Codex wire target is official `rust-v0.166.0`, commit
|
||||
`6ff1e1ca3b30fc8b914720681fcf5fe62e95f856`. Its envelope, initialize lifecycle,
|
||||
dispatcher, thread/turn contracts, subscriptions and interactions are separate
|
||||
remaining adoption slices. This crate does not advertise them as implemented.
|
||||
The existing Runtime journal and native Engine receipt/store authority remain
|
||||
where they are; no serialized observation creates publication authority.
|
||||
@@ -0,0 +1,500 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$defs": {
|
||||
"RuntimeImageInput": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"mime": {
|
||||
"type": "string"
|
||||
},
|
||||
"dataBase64": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"mime",
|
||||
"dataBase64"
|
||||
],
|
||||
"description": "Inline bytes only: neither host paths nor remote URLs confer attachment authority."
|
||||
},
|
||||
"DynamicToolSpec": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"namespace": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
],
|
||||
"description": "Optional namespace that groups related tools (e.g. `\"tau_bench\"`).\nWhen present, the runtime may expose the tool as\n`<namespace>::<name>` to the model."
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Short tool name. Combined with `namespace` it forms a unique tool id."
|
||||
},
|
||||
"description": {
|
||||
"type": "string",
|
||||
"description": "Human-readable description exposed to the model."
|
||||
},
|
||||
"input_schema": {
|
||||
"description": "JSON Schema describing the tool's input parameters."
|
||||
},
|
||||
"defer_loading": {
|
||||
"type": "boolean",
|
||||
"description": "If true, the runtime may defer schema validation / tool loading until\nthe model actually calls the tool.\n\nDefaults to `false` so that older clients omitting this field still\nbehave the same way.",
|
||||
"default": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name",
|
||||
"description",
|
||||
"input_schema"
|
||||
],
|
||||
"description": "Specification for a dynamic external tool registered by a runtime client.\n\nExample JSON from the spec:\n\n```json\n{\n \"namespace\": \"tau_bench\",\n \"name\": \"get_reservation\",\n \"description\": \"Look up an airline reservation.\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"reservation_id\": { \"type\": \"string\" }\n },\n \"required\": [\"reservation_id\"],\n \"additionalProperties\": false\n }\n}\n```"
|
||||
},
|
||||
"DynamicToolItemStatus": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"in_progress",
|
||||
"completed",
|
||||
"failed"
|
||||
],
|
||||
"description": "Lifecycle status of a dynamic tool item shown in thread detail and event\npayloads."
|
||||
},
|
||||
"DynamicToolCallResult": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"success": {
|
||||
"type": "boolean",
|
||||
"description": "Whether the client-side tool execution succeeded."
|
||||
},
|
||||
"content": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/DynamicToolCallContent"
|
||||
},
|
||||
"description": "Content fragments returned by the tool.\n\nDefaults to an empty vector when omitted so clients can send a minimal\n`{ \"success\": false }` payload.",
|
||||
"default": []
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"success"
|
||||
],
|
||||
"description": "Result submitted by a runtime client after executing a dynamic tool."
|
||||
},
|
||||
"DynamicToolCallContent": {
|
||||
"oneOf": [
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"text": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"type": "string",
|
||||
"const": "input_text"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"type",
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"image_url": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"type": "string",
|
||||
"const": "input_image"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"type",
|
||||
"image_url"
|
||||
]
|
||||
}
|
||||
],
|
||||
"description": "A single content fragment inside a [`DynamicToolCallResult`]."
|
||||
},
|
||||
"TurnEnvironmentParams": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"environment_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"cwd": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"environment_id",
|
||||
"cwd"
|
||||
],
|
||||
"description": "Environment target selected for a turn's shell/filesystem work.\n\nExample JSON:\n\n```json\n{\n \"environment_id\": \"local\",\n \"cwd\": \"/workspace\"\n}\n```"
|
||||
},
|
||||
"RuntimeEventEnvelope": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"schema_version": {
|
||||
"type": "integer",
|
||||
"format": "uint32",
|
||||
"minimum": 0,
|
||||
"default": 1
|
||||
},
|
||||
"seq": {
|
||||
"type": "integer",
|
||||
"format": "uint64",
|
||||
"minimum": 0
|
||||
},
|
||||
"event": {
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"type": "string"
|
||||
},
|
||||
"thread_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"turn_id": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"item_id": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"timestamp": {
|
||||
"type": "string"
|
||||
},
|
||||
"created_at": {
|
||||
"type": "string"
|
||||
},
|
||||
"payload": true
|
||||
},
|
||||
"required": [
|
||||
"schema_version",
|
||||
"seq",
|
||||
"event",
|
||||
"kind",
|
||||
"thread_id",
|
||||
"turn_id",
|
||||
"item_id",
|
||||
"timestamp",
|
||||
"payload"
|
||||
],
|
||||
"additionalProperties": true
|
||||
},
|
||||
"RuntimeCapabilities": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"client_token_intents": {
|
||||
"type": "boolean",
|
||||
"description": "Device client tokens have immutable watch/drive intent. Watch cannot\nmutate Runtime state, acquire control, or forward display input.",
|
||||
"default": false
|
||||
},
|
||||
"account_session": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"account_model_owner": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"session_provider_keys": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"threads": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"thread_create_idempotency": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"thread_idle_precondition": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"thread_shell_consent": {
|
||||
"type": "boolean",
|
||||
"description": "Explicit per-thread shell opt-in is checked against loaded policy and\ncannot broaden a conversation while it has an active turn.",
|
||||
"default": false
|
||||
},
|
||||
"turns": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"turn_operation_idempotency": {
|
||||
"type": "boolean",
|
||||
"description": "`POST /v1/threads/{id}/turns` accepts a durable, thread-scoped\n`operation_key` and returns the original turn for exact retries.",
|
||||
"default": false
|
||||
},
|
||||
"turn_operation_lookup": {
|
||||
"type": "boolean",
|
||||
"description": "Read-only exact accepted-turn lookup by thread and operation key.",
|
||||
"default": false
|
||||
},
|
||||
"structured_review_turns": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"structured_review_version": {
|
||||
"type": "integer",
|
||||
"format": "uint32",
|
||||
"minimum": 0,
|
||||
"default": 0
|
||||
},
|
||||
"automation_mutation_preconditions": {
|
||||
"type": "boolean",
|
||||
"description": "Automation patch/run/pause/resume honor `expected_revision` (409\n`revision_conflict` carrying `current_revision`); create/patch/run/pause/\nresume honor a bounded, persisted `Idempotency-Key`. Delete honors\nneither.",
|
||||
"default": false
|
||||
},
|
||||
"turn_image_inputs": {
|
||||
"type": "boolean",
|
||||
"description": "Bounded inline image inputs, persisted and replayed with their turn.",
|
||||
"default": false
|
||||
},
|
||||
"turn_output_token_limit": {
|
||||
"type": "boolean",
|
||||
"description": "Per-turn maxOutputTokens is validated and intersected with the route ceiling.",
|
||||
"default": false
|
||||
},
|
||||
"profile_constitution": {
|
||||
"type": "boolean",
|
||||
"description": "Account profile snapshots and Engine-owned constitution preview.",
|
||||
"default": false
|
||||
},
|
||||
"turn_steer": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"turn_interrupt": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"event_replay": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"external_tools": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"environments": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"worker_runtime": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"fleet_run_create": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"thread_fleet_members": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"fleet_run_start": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"fleet_event_replay": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"fleet_event_stream": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"fleet_local_target": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"thread_goals": {
|
||||
"type": "boolean",
|
||||
"description": "`GET/PUT/DELETE /v1/threads/{id}/goal` and the `complete`/`block`\nlifecycle actions are available.",
|
||||
"default": false
|
||||
},
|
||||
"memory": {
|
||||
"type": "boolean",
|
||||
"description": "`GET /v1/memory` and `GET /v1/memory/{id}` are available for\nbounded inspection of the native memory store. `POST /v1/memory`\nand `DELETE /v1/memory` are also available (auth-gated via the\nstandard route layer) for lifecycle controls.",
|
||||
"default": false
|
||||
},
|
||||
"mcp_server_management": {
|
||||
"type": "boolean",
|
||||
"description": "Whether the runtime supports create/update/enable/disable/reconnect/delete\noperations on MCP server configuration via the `POST|GET|PATCH|DELETE\n/v1/apps/mcp/servers` family of endpoints.",
|
||||
"default": false
|
||||
},
|
||||
"skill_lifecycle": {
|
||||
"type": "boolean",
|
||||
"description": "Skill lifecycle operations (install, update, uninstall, trust, audit)\nare available via the HTTP API.",
|
||||
"default": false
|
||||
},
|
||||
"skill_detail": {
|
||||
"type": "boolean",
|
||||
"description": "`GET /v1/skills/{name}` returns one skill's full body and routing\nmetadata, so a client can compose an activation instruction for its\nnext turn. `GET /v1/skills` rows also carry `invocation`, `aliases`,\nand `bundled_tier` routing fields when this flag is set.",
|
||||
"default": false
|
||||
},
|
||||
"plugin_management": {
|
||||
"type": "boolean",
|
||||
"description": "Plugin bundle and marketplace lifecycle operations (list/detail,\ninstall/update/uninstall, trust/enable/disable/revoke, marketplace\nadd/remove/install) are available via the `/v1/apps/plugins` and\n`/v1/apps/marketplaces` endpoint families.",
|
||||
"default": false
|
||||
},
|
||||
"agent_mail": {
|
||||
"type": "boolean",
|
||||
"description": "Durable, workspace-scoped cross-task Agent Mail endpoints and events.",
|
||||
"default": false
|
||||
},
|
||||
"terminal_stream": {
|
||||
"type": "boolean",
|
||||
"description": "`GET /v1/terminal/{name}/output` — the resumable byte stream over a\npersistent Engine-owned terminal session, with absolute cursors.",
|
||||
"default": false
|
||||
},
|
||||
"terminal_input": {
|
||||
"type": "boolean",
|
||||
"description": "`POST /v1/terminal/{name}/input` — bytes into the live session.",
|
||||
"default": false
|
||||
},
|
||||
"terminal_resize": {
|
||||
"type": "boolean",
|
||||
"description": "`POST /v1/terminal/{name}/resize` — the window the child draws for.",
|
||||
"default": false
|
||||
},
|
||||
"terminal_kill": {
|
||||
"type": "boolean",
|
||||
"description": "`POST /v1/terminal/{name}/kill` — end the live session.",
|
||||
"default": false
|
||||
},
|
||||
"event_stream_resume": {
|
||||
"type": "boolean",
|
||||
"description": "`GET /v1/threads/{id}/events` puts the durable `seq` on every journal\nframe as the SSE `id:` and resumes from a `Last-Event-ID` header, so a\nbrowser `EventSource` reconnects without a cursor in the query string.",
|
||||
"default": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"client_token_intents",
|
||||
"account_session",
|
||||
"account_model_owner",
|
||||
"session_provider_keys",
|
||||
"threads",
|
||||
"thread_create_idempotency",
|
||||
"thread_idle_precondition",
|
||||
"thread_shell_consent",
|
||||
"turns",
|
||||
"turn_operation_idempotency",
|
||||
"turn_operation_lookup",
|
||||
"structured_review_turns",
|
||||
"structured_review_version",
|
||||
"automation_mutation_preconditions",
|
||||
"turn_image_inputs",
|
||||
"turn_output_token_limit",
|
||||
"profile_constitution",
|
||||
"turn_steer",
|
||||
"turn_interrupt",
|
||||
"event_replay",
|
||||
"external_tools",
|
||||
"environments",
|
||||
"worker_runtime",
|
||||
"fleet_run_create",
|
||||
"thread_fleet_members",
|
||||
"fleet_run_start",
|
||||
"fleet_event_replay",
|
||||
"fleet_event_stream",
|
||||
"fleet_local_target",
|
||||
"thread_goals",
|
||||
"memory",
|
||||
"mcp_server_management",
|
||||
"skill_lifecycle",
|
||||
"skill_detail",
|
||||
"plugin_management",
|
||||
"agent_mail",
|
||||
"terminal_stream",
|
||||
"terminal_input",
|
||||
"terminal_resize",
|
||||
"terminal_kill",
|
||||
"event_stream_resume"
|
||||
],
|
||||
"description": "Fixed capability map advertised by `GET /v1/runtime/info`.\n\nAll fields are required on serialization so clients can rely on the shape."
|
||||
},
|
||||
"RuntimeExperimentalCapabilities": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"environments": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"environments"
|
||||
],
|
||||
"description": "Experimental opt-in flags advertised by `GET /v1/runtime/info`.\n\nFields are additive and default to `false` when omitted by older servers."
|
||||
},
|
||||
"DynamicToolCallParams": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"thread_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"turn_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"call_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"namespace": {
|
||||
"type": "string",
|
||||
"description": "Optional namespace that was registered with the tool."
|
||||
},
|
||||
"tool": {
|
||||
"type": "string",
|
||||
"description": "Tool name that the model invoked."
|
||||
},
|
||||
"arguments": {
|
||||
"description": "Arguments supplied by the model, validated against `input_schema`."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"thread_id",
|
||||
"turn_id",
|
||||
"call_id",
|
||||
"tool",
|
||||
"arguments"
|
||||
],
|
||||
"description": "Parameters identifying a dynamic tool call request emitted by the runtime.\n\nThis is the typed payload for `tool_call.requested` events and also the\nnatural identifier used when the runtime looks up a pending call."
|
||||
},
|
||||
"DynamicToolSpecOutput": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"namespace": {
|
||||
"type": "string",
|
||||
"description": "Optional namespace that groups related tools (e.g. `\"tau_bench\"`).\nWhen present, the runtime may expose the tool as\n`<namespace>::<name>` to the model."
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Short tool name. Combined with `namespace` it forms a unique tool id."
|
||||
},
|
||||
"description": {
|
||||
"type": "string",
|
||||
"description": "Human-readable description exposed to the model."
|
||||
},
|
||||
"input_schema": {
|
||||
"description": "JSON Schema describing the tool's input parameters."
|
||||
},
|
||||
"defer_loading": {
|
||||
"type": "boolean",
|
||||
"description": "If true, the runtime may defer schema validation / tool loading until\nthe model actually calls the tool.\n\nDefaults to `false` so that older clients omitting this field still\nbehave the same way.",
|
||||
"default": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name",
|
||||
"description",
|
||||
"input_schema",
|
||||
"defer_loading"
|
||||
],
|
||||
"description": "Specification for a dynamic external tool registered by a runtime client.\n\nExample JSON from the spec:\n\n```json\n{\n \"namespace\": \"tau_bench\",\n \"name\": \"get_reservation\",\n \"description\": \"Look up an airline reservation.\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"reservation_id\": { \"type\": \"string\" }\n },\n \"required\": [\"reservation_id\"],\n \"additionalProperties\": false\n }\n}\n```"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
|
||||
export type Json = null | boolean | number | string | Json[] | { [key: string]: Json | undefined }
|
||||
|
||||
export type DynamicToolCallContent = { type: 'input_text'; text: string } | { type: 'input_image'; image_url: string }
|
||||
|
||||
export type DynamicToolCallParams = {
|
||||
thread_id: string
|
||||
turn_id: string
|
||||
call_id: string
|
||||
namespace?: string
|
||||
tool: string
|
||||
arguments: Json
|
||||
[key: string]: Json | undefined
|
||||
}
|
||||
|
||||
export type DynamicToolCallResult = {
|
||||
success: boolean
|
||||
content?: DynamicToolCallContent[]
|
||||
[key: string]: Json | undefined
|
||||
}
|
||||
|
||||
export type DynamicToolItemStatus = 'in_progress' | 'completed' | 'failed'
|
||||
|
||||
export type DynamicToolSpec = {
|
||||
namespace?: (string | null)
|
||||
name: string
|
||||
description: string
|
||||
input_schema: Json
|
||||
defer_loading?: boolean
|
||||
[key: string]: Json | undefined
|
||||
}
|
||||
|
||||
export type DynamicToolSpecOutput = {
|
||||
namespace?: string
|
||||
name: string
|
||||
description: string
|
||||
input_schema: Json
|
||||
defer_loading: boolean
|
||||
[key: string]: Json | undefined
|
||||
}
|
||||
|
||||
export type RuntimeCapabilities = {
|
||||
client_token_intents: boolean
|
||||
account_session: boolean
|
||||
account_model_owner: boolean
|
||||
session_provider_keys: boolean
|
||||
threads: boolean
|
||||
thread_create_idempotency: boolean
|
||||
thread_idle_precondition: boolean
|
||||
thread_shell_consent: boolean
|
||||
turns: boolean
|
||||
turn_operation_idempotency: boolean
|
||||
turn_operation_lookup: boolean
|
||||
structured_review_turns: boolean
|
||||
structured_review_version: number
|
||||
automation_mutation_preconditions: boolean
|
||||
turn_image_inputs: boolean
|
||||
turn_output_token_limit: boolean
|
||||
profile_constitution: boolean
|
||||
turn_steer: boolean
|
||||
turn_interrupt: boolean
|
||||
event_replay: boolean
|
||||
external_tools: boolean
|
||||
environments: boolean
|
||||
worker_runtime: boolean
|
||||
fleet_run_create: boolean
|
||||
thread_fleet_members: boolean
|
||||
fleet_run_start: boolean
|
||||
fleet_event_replay: boolean
|
||||
fleet_event_stream: boolean
|
||||
fleet_local_target: boolean
|
||||
thread_goals: boolean
|
||||
memory: boolean
|
||||
mcp_server_management: boolean
|
||||
skill_lifecycle: boolean
|
||||
skill_detail: boolean
|
||||
plugin_management: boolean
|
||||
agent_mail: boolean
|
||||
terminal_stream: boolean
|
||||
terminal_input: boolean
|
||||
terminal_resize: boolean
|
||||
terminal_kill: boolean
|
||||
event_stream_resume: boolean
|
||||
[key: string]: Json | undefined
|
||||
}
|
||||
|
||||
export type RuntimeEventEnvelope = {
|
||||
schema_version: number
|
||||
seq: number
|
||||
event: string
|
||||
kind: string
|
||||
thread_id: string
|
||||
turn_id: (string | null)
|
||||
item_id: (string | null)
|
||||
timestamp: string
|
||||
created_at?: string
|
||||
payload: Json
|
||||
[key: string]: Json | undefined
|
||||
}
|
||||
|
||||
export type RuntimeExperimentalCapabilities = {
|
||||
environments: boolean
|
||||
[key: string]: Json | undefined
|
||||
}
|
||||
|
||||
export type RuntimeImageInput = {
|
||||
mime: string
|
||||
dataBase64: string
|
||||
}
|
||||
|
||||
export type TurnEnvironmentParams = {
|
||||
environment_id: string
|
||||
cwd: string
|
||||
[key: string]: Json | undefined
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
fn retain(path: &Path, bytes: &[u8], write: bool) -> std::io::Result<()> {
|
||||
if write {
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)?;
|
||||
}
|
||||
std::fs::write(path, bytes)
|
||||
} else if std::fs::read(path)? == bytes {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(std::io::Error::other(format!(
|
||||
"generated contract drift: {}",
|
||||
path.display()
|
||||
)))
|
||||
}
|
||||
}
|
||||
|
||||
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
let arguments: Vec<String> = std::env::args().skip(1).collect();
|
||||
if !matches!(arguments.as_slice(), [mode] if mode == "--write" || mode == "--check") {
|
||||
return Err("use --write or --check against the declared package and SDK outputs".into());
|
||||
}
|
||||
let write = arguments[0] == "--write";
|
||||
let package = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||
let repository = package
|
||||
.parent()
|
||||
.and_then(Path::parent)
|
||||
.ok_or("package repository root")?;
|
||||
let schema = codewhale_app_server_protocol::export::runtime_schema();
|
||||
let typescript = codewhale_app_server_protocol::export::runtime_typescript(&schema);
|
||||
let json = serde_json::to_string_pretty(&schema)? + "\n";
|
||||
retain(
|
||||
&package.join("schema/json/runtime.schema.json"),
|
||||
json.as_bytes(),
|
||||
write,
|
||||
)?;
|
||||
retain(
|
||||
&package.join("schema/typescript/runtime.d.ts"),
|
||||
typescript.as_bytes(),
|
||||
write,
|
||||
)?;
|
||||
retain(
|
||||
&repository.join("npm/runtime-sdk/generated/runtime.d.ts"),
|
||||
typescript.as_bytes(),
|
||||
write,
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
use crate::runtime::*;
|
||||
use schemars::generate::SchemaSettings;
|
||||
use serde_json::{Value, json};
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
fn output_references(value: &mut Value) {
|
||||
match value {
|
||||
Value::Object(object) => {
|
||||
if object.get("$ref").and_then(Value::as_str) == Some("#/$defs/DynamicToolSpec") {
|
||||
object.insert(
|
||||
"$ref".into(),
|
||||
Value::String("#/$defs/DynamicToolSpecOutput".into()),
|
||||
);
|
||||
}
|
||||
for child in object.values_mut() {
|
||||
output_references(child);
|
||||
}
|
||||
}
|
||||
Value::Array(values) => {
|
||||
for child in values {
|
||||
output_references(child);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn runtime_schema() -> Value {
|
||||
let mut input = SchemaSettings::draft2020_12()
|
||||
.for_deserialize()
|
||||
.into_generator();
|
||||
let _ = input.subschema_for::<RuntimeImageInput>();
|
||||
let _ = input.subschema_for::<DynamicToolSpec>();
|
||||
let _ = input.subschema_for::<DynamicToolItemStatus>();
|
||||
let _ = input.subschema_for::<DynamicToolCallResult>();
|
||||
let _ = input.subschema_for::<DynamicToolCallContent>();
|
||||
let _ = input.subschema_for::<TurnEnvironmentParams>();
|
||||
let mut output = SchemaSettings::draft2020_12()
|
||||
.for_serialize()
|
||||
.into_generator();
|
||||
let _ = output.subschema_for::<RuntimeEventEnvelope>();
|
||||
let _ = output.subschema_for::<RuntimeCapabilities>();
|
||||
let _ = output.subschema_for::<RuntimeExperimentalCapabilities>();
|
||||
let _ = output.subschema_for::<DynamicToolCallParams>();
|
||||
let _ = output.subschema_for::<DynamicToolSpec>();
|
||||
let _ = output.subschema_for::<TurnEnvironmentParams>();
|
||||
let _ = output.subschema_for::<DynamicToolItemStatus>();
|
||||
let mut definitions = input.take_definitions(false);
|
||||
for (name, mut schema) in output.take_definitions(false) {
|
||||
output_references(&mut schema);
|
||||
let name = if name == "DynamicToolSpec" {
|
||||
"DynamicToolSpecOutput".to_string()
|
||||
} else {
|
||||
name
|
||||
};
|
||||
if let Some(previous) = definitions.get(&name) {
|
||||
assert_eq!(previous, &schema, "input/output schema conflict for {name}");
|
||||
} else {
|
||||
definitions.insert(name, schema);
|
||||
}
|
||||
}
|
||||
json!({"$schema": "https://json-schema.org/draft/2020-12/schema", "$defs": definitions})
|
||||
}
|
||||
|
||||
pub fn runtime_typescript(schema: &Value) -> String {
|
||||
let definitions: BTreeMap<String, crate::schema::Def> = schema["$defs"]
|
||||
.as_object()
|
||||
.expect("Runtime schema definitions")
|
||||
.iter()
|
||||
.map(|(name, schema)| {
|
||||
(
|
||||
name.clone(),
|
||||
crate::schema::parse_def(name, schema, crate::schema::SchemaMode::Runtime),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
crate::schema::typescript_defs(&definitions, crate::schema::SchemaMode::Runtime)
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
pub mod runtime;
|
||||
|
||||
#[cfg(feature = "schema")]
|
||||
pub mod schema;
|
||||
|
||||
#[cfg(feature = "schema")]
|
||||
pub mod export;
|
||||
@@ -14,6 +14,7 @@ pub const MAX_RUNTIME_IMAGE_BYTES: usize = 4 * 1024 * 1024;
|
||||
pub const MAX_RUNTIME_IMAGE_TOTAL_BYTES: usize = 5 * 1024 * 1024;
|
||||
|
||||
/// Inline bytes only: neither host paths nor remote URLs confer attachment authority.
|
||||
#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "camelCase", deny_unknown_fields)]
|
||||
pub struct RuntimeImageInput {
|
||||
@@ -21,6 +22,7 @@ pub struct RuntimeImageInput {
|
||||
pub data_base64: String,
|
||||
}
|
||||
|
||||
#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct RuntimeEventEnvelope {
|
||||
#[serde(default = "default_runtime_event_envelope_schema_version")]
|
||||
@@ -32,6 +34,10 @@ pub struct RuntimeEventEnvelope {
|
||||
pub turn_id: Option<String>,
|
||||
pub item_id: Option<String>,
|
||||
pub timestamp: String,
|
||||
#[cfg_attr(
|
||||
feature = "schema",
|
||||
schemars(default, schema_with = "optional_string_schema")
|
||||
)]
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub created_at: Option<String>,
|
||||
pub payload: Value,
|
||||
@@ -44,6 +50,15 @@ fn default_runtime_event_envelope_schema_version() -> u32 {
|
||||
RUNTIME_EVENT_ENVELOPE_SCHEMA_VERSION
|
||||
}
|
||||
|
||||
#[cfg(feature = "schema")]
|
||||
fn optional_string_schema(generator: &mut schemars::SchemaGenerator) -> schemars::Schema {
|
||||
if generator.contract().is_serialize() {
|
||||
generator.subschema_for::<String>()
|
||||
} else {
|
||||
generator.subschema_for::<Option<String>>()
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Capability advertisement
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -51,6 +66,7 @@ fn default_runtime_event_envelope_schema_version() -> u32 {
|
||||
/// Fixed capability map advertised by `GET /v1/runtime/info`.
|
||||
///
|
||||
/// All fields are required on serialization so clients can rely on the shape.
|
||||
#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct RuntimeCapabilities {
|
||||
/// Device client tokens have immutable watch/drive intent. Watch cannot
|
||||
@@ -174,6 +190,7 @@ pub struct RuntimeCapabilities {
|
||||
/// Experimental opt-in flags advertised by `GET /v1/runtime/info`.
|
||||
///
|
||||
/// Fields are additive and default to `false` when omitted by older servers.
|
||||
#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]
|
||||
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||
pub struct RuntimeExperimentalCapabilities {
|
||||
#[serde(default)]
|
||||
@@ -203,11 +220,16 @@ pub struct RuntimeExperimentalCapabilities {
|
||||
/// }
|
||||
/// }
|
||||
/// ```
|
||||
#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
pub struct DynamicToolSpec {
|
||||
/// Optional namespace that groups related tools (e.g. `"tau_bench"`).
|
||||
/// When present, the runtime may expose the tool as
|
||||
/// `<namespace>::<name>` to the model.
|
||||
#[cfg_attr(
|
||||
feature = "schema",
|
||||
schemars(default, schema_with = "optional_string_schema")
|
||||
)]
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub namespace: Option<String>,
|
||||
|
||||
@@ -231,6 +253,7 @@ pub struct DynamicToolSpec {
|
||||
|
||||
/// Lifecycle status of a dynamic tool item shown in thread detail and event
|
||||
/// payloads.
|
||||
#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]
|
||||
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum DynamicToolItemStatus {
|
||||
@@ -243,6 +266,7 @@ pub enum DynamicToolItemStatus {
|
||||
///
|
||||
/// This is the typed payload for `tool_call.requested` events and also the
|
||||
/// natural identifier used when the runtime looks up a pending call.
|
||||
#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
pub struct DynamicToolCallParams {
|
||||
pub thread_id: String,
|
||||
@@ -250,6 +274,10 @@ pub struct DynamicToolCallParams {
|
||||
pub call_id: String,
|
||||
|
||||
/// Optional namespace that was registered with the tool.
|
||||
#[cfg_attr(
|
||||
feature = "schema",
|
||||
schemars(default, schema_with = "optional_string_schema")
|
||||
)]
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub namespace: Option<String>,
|
||||
|
||||
@@ -261,6 +289,7 @@ pub struct DynamicToolCallParams {
|
||||
}
|
||||
|
||||
/// Result submitted by a runtime client after executing a dynamic tool.
|
||||
#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
pub struct DynamicToolCallResult {
|
||||
/// Whether the client-side tool execution succeeded.
|
||||
@@ -275,6 +304,7 @@ pub struct DynamicToolCallResult {
|
||||
}
|
||||
|
||||
/// A single content fragment inside a [`DynamicToolCallResult`].
|
||||
#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(tag = "type", rename_all = "snake_case")]
|
||||
pub enum DynamicToolCallContent {
|
||||
@@ -296,6 +326,7 @@ pub enum DynamicToolCallContent {
|
||||
/// "cwd": "/workspace"
|
||||
/// }
|
||||
/// ```
|
||||
#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
pub struct TurnEnvironmentParams {
|
||||
pub environment_id: String,
|
||||
@@ -0,0 +1,302 @@
|
||||
use serde_json::Value;
|
||||
use std::collections::BTreeMap;
|
||||
use std::fmt::Write as _;
|
||||
|
||||
#[derive(Clone, Copy, PartialEq, Eq)]
|
||||
pub enum SchemaMode {
|
||||
Host,
|
||||
Runtime,
|
||||
}
|
||||
|
||||
/// A field's wire type, normalized.
|
||||
pub enum Ty {
|
||||
Nullable(Box<Ty>),
|
||||
String,
|
||||
Boolean,
|
||||
Uint,
|
||||
Integer,
|
||||
/// A JSON object with any keys (`Map<String, Value>`).
|
||||
Object,
|
||||
/// Any JSON value.
|
||||
Json,
|
||||
Ref(String),
|
||||
/// A string tag inside a union member.
|
||||
Const(String),
|
||||
Array(Box<Ty>),
|
||||
}
|
||||
|
||||
pub struct Field {
|
||||
pub name: String,
|
||||
pub ty: Ty,
|
||||
pub required: bool,
|
||||
}
|
||||
|
||||
pub enum Def {
|
||||
Object {
|
||||
strict: bool,
|
||||
extra: bool,
|
||||
fields: Vec<Field>,
|
||||
},
|
||||
Enum(Vec<String>),
|
||||
Union(Vec<Vec<Field>>),
|
||||
}
|
||||
|
||||
pub fn ref_name(schema: &Value) -> Option<String> {
|
||||
let reference = schema.get("$ref")?.as_str()?;
|
||||
Some(
|
||||
reference
|
||||
.strip_prefix("#/$defs/")
|
||||
.unwrap_or_else(|| panic!("unexpected $ref `{reference}`"))
|
||||
.to_string(),
|
||||
)
|
||||
}
|
||||
|
||||
fn parse_ty(at: &str, schema: &Value, mode: SchemaMode) -> Ty {
|
||||
let Some(object) = schema.as_object() else {
|
||||
assert_eq!(schema, &Value::Bool(true), "{at}: unsupported schema");
|
||||
return Ty::Json;
|
||||
};
|
||||
if let Some(name) = ref_name(schema) {
|
||||
return Ty::Ref(name);
|
||||
}
|
||||
if let Some(variants) = object.get("anyOf").and_then(Value::as_array)
|
||||
&& variants.len() == 2
|
||||
&& variants
|
||||
.iter()
|
||||
.filter(|variant| variant.get("type").and_then(Value::as_str) == Some("null"))
|
||||
.count()
|
||||
== 1
|
||||
{
|
||||
let present = parse_ty(
|
||||
at,
|
||||
variants
|
||||
.iter()
|
||||
.find(|variant| variant.get("type").and_then(Value::as_str) != Some("null"))
|
||||
.expect("one non-null variant"),
|
||||
mode,
|
||||
);
|
||||
return if mode == SchemaMode::Runtime {
|
||||
Ty::Nullable(Box::new(present))
|
||||
} else {
|
||||
present
|
||||
};
|
||||
}
|
||||
if let Some(value) = object.get("const").and_then(Value::as_str) {
|
||||
return Ty::Const(value.to_string());
|
||||
}
|
||||
let types: Vec<&str> = match object.get("type") {
|
||||
Some(Value::String(ty)) => vec![ty.as_str()],
|
||||
Some(Value::Array(types)) => types
|
||||
.iter()
|
||||
.filter_map(Value::as_str)
|
||||
.filter(|ty| *ty != "null")
|
||||
.collect(),
|
||||
// Only annotations (`default`, `description`): any value.
|
||||
None if !["enum", "anyOf", "oneOf", "allOf", "properties", "items"]
|
||||
.iter()
|
||||
.any(|key| object.contains_key(*key)) =>
|
||||
{
|
||||
return Ty::Json;
|
||||
}
|
||||
_ => panic!("{at}: unsupported wire schema {schema}"),
|
||||
};
|
||||
let value = match types.as_slice() {
|
||||
["string"] => Ty::String,
|
||||
["boolean"] => Ty::Boolean,
|
||||
["integer"] if object.get("minimum").and_then(Value::as_u64) == Some(0) => Ty::Uint,
|
||||
["integer"] => Ty::Integer,
|
||||
["object"]
|
||||
if object.get("additionalProperties") == Some(&Value::Bool(true))
|
||||
&& !object.contains_key("properties") =>
|
||||
{
|
||||
Ty::Object
|
||||
}
|
||||
["array"] => Ty::Array(Box::new(parse_ty(
|
||||
&format!("{at}[]"),
|
||||
object
|
||||
.get("items")
|
||||
.unwrap_or_else(|| panic!("{at}: array without items")),
|
||||
mode,
|
||||
))),
|
||||
_ => panic!("{at}: unsupported wire schema {schema}"),
|
||||
};
|
||||
if mode == SchemaMode::Runtime
|
||||
&& object
|
||||
.get("type")
|
||||
.and_then(Value::as_array)
|
||||
.is_some_and(|types| types.iter().any(|ty| ty == "null"))
|
||||
{
|
||||
Ty::Nullable(Box::new(value))
|
||||
} else {
|
||||
value
|
||||
}
|
||||
}
|
||||
|
||||
fn object_fields(at: &str, schema: &Value, mode: SchemaMode) -> Vec<Field> {
|
||||
assert_eq!(
|
||||
schema.get("type").and_then(Value::as_str),
|
||||
Some("object"),
|
||||
"{at}: expected an object schema, got {schema}"
|
||||
);
|
||||
let required: Vec<&str> = schema
|
||||
.get("required")
|
||||
.and_then(Value::as_array)
|
||||
.map(|names| names.iter().filter_map(Value::as_str).collect())
|
||||
.unwrap_or_default();
|
||||
schema
|
||||
.get("properties")
|
||||
.and_then(Value::as_object)
|
||||
.map(|properties| {
|
||||
properties
|
||||
.iter()
|
||||
.map(|(name, schema)| Field {
|
||||
name: name.clone(),
|
||||
ty: parse_ty(&format!("{at}.{name}"), schema, mode),
|
||||
required: required.contains(&name.as_str()),
|
||||
})
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
pub fn parse_def(name: &str, schema: &Value, mode: SchemaMode) -> Def {
|
||||
let members = schema
|
||||
.get("oneOf")
|
||||
.or_else(|| schema.get("anyOf"))
|
||||
.and_then(Value::as_array);
|
||||
if let Some(members) = members {
|
||||
// Unit variants, with or without doc comments: a string enum.
|
||||
let consts: Option<Vec<String>> = members
|
||||
.iter()
|
||||
.map(|member| {
|
||||
member
|
||||
.get("const")
|
||||
.and_then(Value::as_str)
|
||||
.map(str::to_string)
|
||||
})
|
||||
.collect();
|
||||
return match consts {
|
||||
Some(values) => Def::Enum(values),
|
||||
None if members.len() == 1 => Def::Object {
|
||||
strict: members[0].get("additionalProperties") == Some(&Value::Bool(false)),
|
||||
extra: mode == SchemaMode::Runtime
|
||||
&& members[0].get("additionalProperties") != Some(&Value::Bool(false)),
|
||||
fields: object_fields(name, &members[0], mode),
|
||||
},
|
||||
None => Def::Union(
|
||||
members
|
||||
.iter()
|
||||
.map(|member| object_fields(name, member, mode))
|
||||
.collect(),
|
||||
),
|
||||
};
|
||||
}
|
||||
if let Some(values) = schema.get("enum").and_then(Value::as_array) {
|
||||
return Def::Enum(
|
||||
values
|
||||
.iter()
|
||||
.map(|value| {
|
||||
value
|
||||
.as_str()
|
||||
.unwrap_or_else(|| panic!("{name}: non-string enum value"))
|
||||
.to_string()
|
||||
})
|
||||
.collect(),
|
||||
);
|
||||
}
|
||||
Def::Object {
|
||||
strict: schema.get("additionalProperties") == Some(&Value::Bool(false)),
|
||||
extra: mode == SchemaMode::Runtime
|
||||
&& schema.get("additionalProperties") != Some(&Value::Bool(false)),
|
||||
fields: object_fields(name, schema, mode),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn quote(value: &str) -> String {
|
||||
assert!(
|
||||
!value.contains(['\'', '\\']),
|
||||
"`{value}` needs escaping in TypeScript"
|
||||
);
|
||||
format!("'{value}'")
|
||||
}
|
||||
|
||||
fn ts(ty: &Ty) -> String {
|
||||
match ty {
|
||||
Ty::Nullable(inner) => format!("({} | null)", ts(inner)),
|
||||
Ty::String => "string".into(),
|
||||
Ty::Boolean => "boolean".into(),
|
||||
Ty::Uint | Ty::Integer => "number".into(),
|
||||
Ty::Object => "{ [key: string]: Json }".into(),
|
||||
Ty::Json => "Json".into(),
|
||||
Ty::Ref(name) => name.clone(),
|
||||
Ty::Const(value) => quote(value),
|
||||
Ty::Array(item) => format!("{}[]", ts(item)),
|
||||
}
|
||||
}
|
||||
|
||||
/// An array's item type, however deeply nested.
|
||||
pub fn innermost(ty: &Ty) -> &Ty {
|
||||
match ty {
|
||||
Ty::Array(item) => innermost(item),
|
||||
ty => ty,
|
||||
}
|
||||
}
|
||||
|
||||
fn member(field: &Field) -> String {
|
||||
let optional = if field.required { "" } else { "?" };
|
||||
format!("{}{optional}: {}", field.name, ts(&field.ty))
|
||||
}
|
||||
|
||||
pub fn typescript_defs(defs: &BTreeMap<String, Def>, mode: SchemaMode) -> String {
|
||||
let mut out = String::from(if mode == SchemaMode::Host {
|
||||
"\nexport type Json = null | boolean | number | string | Json[] | { [key: string]: Json }\n"
|
||||
} else {
|
||||
"\nexport type Json = null | boolean | number | string | Json[] | { [key: string]: Json | undefined }\n"
|
||||
});
|
||||
for (name, def) in defs {
|
||||
out.push('\n');
|
||||
match def {
|
||||
Def::Object { fields, extra, .. } if fields.is_empty() && !extra => {
|
||||
if mode == SchemaMode::Host {
|
||||
let _ = writeln!(out, "export interface {name} {{}}");
|
||||
} else {
|
||||
let _ = writeln!(out, "export type {name} = {{}}");
|
||||
}
|
||||
}
|
||||
Def::Object { fields, extra, .. } => {
|
||||
let declaration = if mode == SchemaMode::Host {
|
||||
format!("export interface {name}")
|
||||
} else {
|
||||
format!("export type {name} =")
|
||||
};
|
||||
let _ = writeln!(out, "{declaration} {{");
|
||||
for field in fields {
|
||||
let _ = writeln!(out, " {}", member(field));
|
||||
}
|
||||
if *extra {
|
||||
out.push_str(" [key: string]: Json | undefined\n");
|
||||
}
|
||||
out.push_str("}\n");
|
||||
}
|
||||
Def::Enum(values) => {
|
||||
let values: Vec<String> = values.iter().map(|v| quote(v)).collect();
|
||||
let _ = writeln!(out, "export type {name} = {}", values.join(" | "));
|
||||
}
|
||||
Def::Union(members) => {
|
||||
let members: Vec<String> = members
|
||||
.iter()
|
||||
.map(|fields| {
|
||||
// Tags first, then the variant's fields in order.
|
||||
let (tags, rest): (Vec<&Field>, Vec<&Field>) = fields
|
||||
.iter()
|
||||
.partition(|field| matches!(field.ty, Ty::Const(_)));
|
||||
let parts: Vec<String> = tags.into_iter().chain(rest).map(member).collect();
|
||||
format!("{{ {} }}", parts.join("; "))
|
||||
})
|
||||
.collect();
|
||||
let _ = writeln!(out, "export type {name} = {}", members.join(" | "));
|
||||
}
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
#![cfg(feature = "schema")]
|
||||
|
||||
use codewhale_app_server_protocol::runtime::{
|
||||
DynamicToolCallResult, DynamicToolSpec, RuntimeEventEnvelope, RuntimeImageInput,
|
||||
};
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
#[test]
|
||||
fn runtime_schema_matches_real_nullable_and_flattened_wire_fields() {
|
||||
let schema = codewhale_app_server_protocol::export::runtime_schema();
|
||||
let definitions = schema["$defs"].as_object().unwrap();
|
||||
let event = &definitions["RuntimeEventEnvelope"];
|
||||
let turn = &event["properties"]["turn_id"];
|
||||
let nullable = turn["type"]
|
||||
.as_array()
|
||||
.is_some_and(|types| types.iter().any(|ty| ty == "null"))
|
||||
|| turn["anyOf"]
|
||||
.as_array()
|
||||
.is_some_and(|types| types.iter().any(|ty| ty["type"] == "null"));
|
||||
assert!(nullable);
|
||||
let required = event["required"].as_array().unwrap();
|
||||
for field in ["schema_version", "turn_id", "item_id", "payload"] {
|
||||
assert!(required.iter().any(|name| name == field));
|
||||
}
|
||||
assert!(!required.iter().any(|name| name == "created_at"));
|
||||
assert_eq!(event["properties"]["created_at"]["type"], "string");
|
||||
for name in ["RuntimeCapabilities", "RuntimeExperimentalCapabilities"] {
|
||||
let record = &definitions[name];
|
||||
let required = record["required"].as_array().unwrap();
|
||||
for field in record["properties"].as_object().unwrap().keys() {
|
||||
assert!(
|
||||
required
|
||||
.iter()
|
||||
.any(|name| name.as_str() == Some(field.as_str()))
|
||||
);
|
||||
}
|
||||
}
|
||||
let spec = &definitions["DynamicToolSpec"];
|
||||
assert!(
|
||||
!spec["required"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.any(|name| name == "defer_loading")
|
||||
);
|
||||
let published_spec = &definitions["DynamicToolSpecOutput"];
|
||||
assert!(
|
||||
published_spec["required"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.any(|name| name == "defer_loading")
|
||||
);
|
||||
assert_eq!(published_spec["properties"]["namespace"]["type"], "string");
|
||||
assert!(
|
||||
!definitions["DynamicToolCallResult"]["required"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.any(|name| name == "content")
|
||||
);
|
||||
let input: DynamicToolSpec = serde_json::from_value(
|
||||
serde_json::json!({"name":"tool","description":"fixture","input_schema":{}}),
|
||||
)
|
||||
.unwrap();
|
||||
assert!(!input.defer_loading);
|
||||
let result: DynamicToolCallResult =
|
||||
serde_json::from_value(serde_json::json!({"success":false})).unwrap();
|
||||
assert!(result.content.is_empty());
|
||||
assert!(
|
||||
serde_json::from_value::<RuntimeImageInput>(
|
||||
serde_json::json!({"mime":"image/png","dataBase64":"","extra":true})
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
assert!(event["properties"].get("extra").is_none());
|
||||
assert_eq!(
|
||||
definitions["RuntimeImageInput"]["additionalProperties"],
|
||||
false
|
||||
);
|
||||
let value = serde_json::to_value(RuntimeEventEnvelope {
|
||||
schema_version: 1,
|
||||
seq: 4,
|
||||
event: "future.name".into(),
|
||||
kind: "future.name".into(),
|
||||
thread_id: "thread".into(),
|
||||
turn_id: None,
|
||||
item_id: None,
|
||||
timestamp: "now".into(),
|
||||
created_at: None,
|
||||
payload: serde_json::json!({"receipt": null}),
|
||||
extra: BTreeMap::new(),
|
||||
})
|
||||
.unwrap();
|
||||
assert!(value["turn_id"].is_null());
|
||||
assert!(value.get("created_at").is_none());
|
||||
let typescript = codewhale_app_server_protocol::export::runtime_typescript(&schema);
|
||||
assert!(typescript.contains("turn_id: (string | null)"));
|
||||
assert!(!typescript.contains("turn_id?:"));
|
||||
assert!(typescript.contains("created_at?: string"));
|
||||
assert!(typescript.contains("export type RuntimeImageInput ="));
|
||||
assert!(typescript.contains("Json[] | { [key: string]: Json | undefined }"));
|
||||
assert!(typescript.contains("type: 'input_text'"));
|
||||
assert!(typescript.contains("dataBase64: string"));
|
||||
assert!(typescript.contains("[key: string]: Json | undefined"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn committed_runtime_schema_and_sdk_exports_match_one_rust_contract() {
|
||||
let package = std::path::Path::new(env!("CARGO_MANIFEST_DIR"));
|
||||
let repository = package.parent().and_then(std::path::Path::parent).unwrap();
|
||||
let schema = codewhale_app_server_protocol::export::runtime_schema();
|
||||
let json = serde_json::to_string_pretty(&schema).unwrap() + "\n";
|
||||
let typescript = codewhale_app_server_protocol::export::runtime_typescript(&schema);
|
||||
assert_eq!(
|
||||
std::fs::read_to_string(package.join("schema/json/runtime.schema.json")).unwrap(),
|
||||
json
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read_to_string(package.join("schema/typescript/runtime.d.ts")).unwrap(),
|
||||
typescript
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read_to_string(repository.join("npm/runtime-sdk/generated/runtime.d.ts")).unwrap(),
|
||||
typescript
|
||||
);
|
||||
}
|
||||
@@ -13,6 +13,7 @@ autobins = false
|
||||
workspace = true
|
||||
|
||||
[dependencies]
|
||||
codewhale-app-server-protocol = { path = "../app-server-protocol", version = "0.10.2" }
|
||||
anyhow.workspace = true
|
||||
axum.workspace = true
|
||||
chrono.workspace = true
|
||||
@@ -23,6 +24,7 @@ codewhale-hooks = { path = "../hooks", version = "0.10.2" }
|
||||
codewhale-paths = { path = "../paths", version = "0.10.2" }
|
||||
codewhale-protocol = { path = "../protocol", version = "0.10.2" }
|
||||
codewhale-release = { path = "../release", version = "0.10.2" }
|
||||
codewhale-runtime = { path = "../runtime", version = "0.10.2" }
|
||||
codewhale-state = { path = "../state", version = "0.10.2" }
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
@@ -33,8 +35,6 @@ tower-http.workspace = true
|
||||
tracing.workspace = true
|
||||
reqwest = { workspace = true, features = ["json"] }
|
||||
uuid.workspace = true
|
||||
# The store-derived control endpoint hashes the runtime store root
|
||||
# (`daemon_socket::store_selected_socket_path`).
|
||||
sha2.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
|
||||
@@ -615,14 +615,14 @@ impl RuntimeBridge {
|
||||
}
|
||||
|
||||
if stream_result.is_ok() {
|
||||
let _ = emit_stdio_event(
|
||||
emit_stdio_event(
|
||||
writer,
|
||||
json!({
|
||||
"type": "response_end",
|
||||
"response_id": response_id,
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
.await?;
|
||||
if let Some(transcript) = transcript {
|
||||
transcript.events.push(EventFrame::ResponseEnd {
|
||||
response_id: response_id.clone(),
|
||||
|
||||
@@ -9,10 +9,17 @@ use std::path::PathBuf;
|
||||
mod platform {
|
||||
use super::*;
|
||||
use crate::daemon_socket::AuthorizedPeer;
|
||||
use crate::daemon_socket::{default_socket_path, owner_work};
|
||||
use crate::stdio::{BoundedLines, ParsedStdioLine, parse_stdio_line, write_stdio_line};
|
||||
#[cfg(windows)]
|
||||
use crate::daemon_socket::default_socket_path;
|
||||
use crate::daemon_socket::owner_work;
|
||||
use crate::stdio::{
|
||||
BoundedLines, ParsedStdioLine, is_frame_refusal, parse_stdio_line, serialize_control_frame,
|
||||
write_control_line, write_stdio_line,
|
||||
};
|
||||
use anyhow::{Context as _, bail};
|
||||
use codewhale_config::private_directory::PrivateDirectory;
|
||||
#[cfg(unix)]
|
||||
use codewhale_paths::daemon_socket::default_socket_path;
|
||||
use codewhale_protocol::RuntimeOwnerReceipt;
|
||||
use serde_json::{Value, json};
|
||||
use std::sync::Arc;
|
||||
@@ -32,6 +39,7 @@ mod platform {
|
||||
receipt: RuntimeOwnerReceipt,
|
||||
routing: Option<crate::RuntimeOwnerRouting>,
|
||||
frontend: crate::daemon_socket::AttachFrontend,
|
||||
output_failed: bool,
|
||||
}
|
||||
|
||||
/// Host startup is the only caller allowed to acquire the Runtime lease.
|
||||
@@ -598,6 +606,7 @@ mod platform {
|
||||
receipt,
|
||||
routing,
|
||||
frontend,
|
||||
output_failed: false,
|
||||
})))
|
||||
}
|
||||
|
||||
@@ -610,23 +619,29 @@ mod platform {
|
||||
&self.receipt
|
||||
}
|
||||
pub async fn send(&mut self, id: Value, method: &str, params: Value) -> Result<()> {
|
||||
let frame = json!({"jsonrpc":"2.0","id":id,"method":method,"params":params});
|
||||
let bytes = serde_json::to_vec(&frame)?;
|
||||
anyhow::ensure!(
|
||||
bytes.len() <= crate::MAX_RUNTIME_IMAGE_BODY_BYTES,
|
||||
"owner request exceeds transport bound"
|
||||
!self.output_failed,
|
||||
"owner transport write previously failed; outcomes uncertain, not replayed"
|
||||
);
|
||||
let frame = json!({"jsonrpc":"2.0","id":id,"method":method,"params":params});
|
||||
let bytes = serialize_control_frame(&frame)?;
|
||||
let check = self.peer.clone();
|
||||
owner_work(move || check.check()).await?;
|
||||
self.output_failed = true;
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(5),
|
||||
write_stdio_line(&mut self.write, &frame),
|
||||
write_control_line(&mut self.write, &bytes),
|
||||
)
|
||||
.await
|
||||
.context("owner write deadline expired; outcome uncertain, not replayed")??;
|
||||
self.output_failed = false;
|
||||
Ok(())
|
||||
}
|
||||
pub async fn recv(&mut self) -> Result<Option<Value>> {
|
||||
anyhow::ensure!(
|
||||
!self.output_failed,
|
||||
"owner transport write previously failed; outcomes uncertain, not replayed"
|
||||
);
|
||||
let Some(line) = self.read.next_line().await? else {
|
||||
return Ok(None);
|
||||
};
|
||||
@@ -639,25 +654,37 @@ mod platform {
|
||||
I: tokio::io::AsyncRead + Unpin,
|
||||
O: tokio::io::AsyncWrite + Unpin,
|
||||
{
|
||||
anyhow::ensure!(
|
||||
!self.output_failed,
|
||||
"owner transport write previously failed; outcomes uncertain, not replayed"
|
||||
);
|
||||
let mut input = BoundedLines::new(BufReader::new(input));
|
||||
let mut output = tokio::io::BufWriter::new(output);
|
||||
let mut input_open = true;
|
||||
loop {
|
||||
tokio::select! {
|
||||
line = input.next_line(), if input_open => match line? {
|
||||
line = input.next_line(), if input_open => match match line {
|
||||
Ok(line) => line,
|
||||
Err(error) if self.frontend == crate::daemon_socket::AttachFrontend::Acp && is_frame_refusal(&error) => {
|
||||
write_stdio_line(&mut output, &json!({"jsonrpc":"2.0","id":null,"error":{"code":-32700,"message":error.to_string()}})).await?;
|
||||
continue;
|
||||
}
|
||||
Err(error) => return Err(error.into()),
|
||||
} {
|
||||
None => {
|
||||
input_open=false;
|
||||
tokio::time::timeout(Duration::from_secs(5),write_stdio_line(&mut self.write,&json!({"jsonrpc":"2.0","method":"daemon/input_closed","params":{}}))).await.context("owner input-close write outcome uncertain, not replayed")??;
|
||||
self.write.shutdown().await?;
|
||||
tokio::time::timeout(Duration::from_secs(5), async {
|
||||
write_stdio_line(&mut self.write,&json!({"jsonrpc":"2.0","method":"daemon/input_closed","params":{}})).await?;
|
||||
self.write.shutdown().await?;
|
||||
Ok::<(), anyhow::Error>(())
|
||||
}).await.context("owner input-close write outcome uncertain, not replayed")??;
|
||||
}
|
||||
Some(line) if self.frontend==crate::daemon_socket::AttachFrontend::Acp => {
|
||||
let check=self.peer.clone();
|
||||
owner_work(move || check.check()).await?;
|
||||
let value:Value=serde_json::from_str(&line)?;
|
||||
anyhow::ensure!(value.is_object() && value["jsonrpc"]=="2.0","ACP frame must be a JSON-RPC object");
|
||||
// Preserve response IDs/permission replies verbatim.
|
||||
// Actual ACP authority/parser stays in AcpServer.
|
||||
tokio::time::timeout(Duration::from_secs(5),async {self.write.write_all(line.as_bytes()).await?;self.write.write_all(b"\n").await?;self.write.flush().await}).await.context("ACP write outcome uncertain; not replayed")??;
|
||||
tokio::time::timeout(Duration::from_secs(5),write_control_line(&mut self.write,line.as_bytes())).await.context("ACP write outcome uncertain; not replayed")??;
|
||||
}
|
||||
Some(line) => match parse_stdio_line(&line) {
|
||||
ParsedStdioLine::Blank => {},
|
||||
@@ -673,9 +700,7 @@ mod platform {
|
||||
None => bail!("selected owner connection closed; pending operation outcomes may be uncertain"),
|
||||
Some(line) => {
|
||||
let _: Value = serde_json::from_str(&line).context("invalid owner response")?;
|
||||
output.write_all(line.as_bytes()).await?;
|
||||
output.write_all(b"\n").await?;
|
||||
output.flush().await?;
|
||||
write_control_line(&mut output, line.as_bytes()).await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,18 +8,6 @@
|
||||
//! one addition in front of it: a `daemon/attach` handshake that establishes
|
||||
//! who this client is and whether it owns the daemon.
|
||||
//!
|
||||
//! # Endpoint resolution
|
||||
//!
|
||||
//! In precedence order (see [`resolve_socket_path`]):
|
||||
//!
|
||||
//! 1. an explicit path (`--socket-path`);
|
||||
//! 2. `$CODEWHALE_HOME/run/daemon.sock` when `CODEWHALE_HOME` is set — an
|
||||
//! explicit home is an isolation boundary, so its daemon must not collide
|
||||
//! with the default one;
|
||||
//! 3. `$XDG_RUNTIME_DIR/codewhale/daemon.sock`;
|
||||
//! 4. macOS: `~/Library/Application Support/codewhale/daemon.sock`;
|
||||
//! 5. `~/.codewhale/run/daemon.sock`.
|
||||
//!
|
||||
//! Windows derives a local named-pipe endpoint from the selected home and
|
||||
//! current principal. It shares the same dispatcher and ownership handshake;
|
||||
//! no supported platform falls back to TCP. Windows runtime proof is separate.
|
||||
@@ -38,99 +26,16 @@ use std::path::{Path, PathBuf};
|
||||
use codewhale_protocol::RuntimeOwnerReceipt;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Basename of the daemon socket inside the Codewhale runtime directory.
|
||||
pub const DAEMON_SOCKET_FILE_NAME: &str = "daemon.sock";
|
||||
|
||||
/// Historical reserved endpoint used in unsupported-platform diagnostics.
|
||||
/// Actual Windows owner endpoints are scoped to the selected home/principal.
|
||||
pub const WINDOWS_NAMED_PIPE: &str = r"\\.\pipe\codewhale-daemon";
|
||||
pub use codewhale_paths::daemon_socket::{
|
||||
DAEMON_SOCKET_FILE_NAME, DaemonSocketError, MAX_SOCKET_PATH_BYTES, SocketPathInputs,
|
||||
WINDOWS_NAMED_PIPE, resolve_socket_path,
|
||||
};
|
||||
#[cfg(unix)]
|
||||
pub use codewhale_paths::daemon_socket::{default_socket_path, store_selected_socket_path};
|
||||
|
||||
/// JSON-RPC method a client must send first on a daemon-socket connection.
|
||||
pub const ATTACH_METHOD: &str = "daemon/attach";
|
||||
|
||||
/// Longest socket path the kernel accepts (`sun_path` minus the NUL).
|
||||
pub const MAX_SOCKET_PATH_BYTES: usize = if cfg!(any(target_os = "macos", target_os = "ios")) {
|
||||
103
|
||||
} else {
|
||||
107
|
||||
};
|
||||
|
||||
/// Typed failures of the daemon socket transport.
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum DaemonSocketError {
|
||||
/// The platform has no daemon socket implementation. Never a silent
|
||||
/// fallback: the caller must pick another transport explicitly.
|
||||
#[error(
|
||||
"the daemon socket transport is not supported on {platform}; the reserved endpoint \
|
||||
there is the named pipe {planned_endpoint}, which is not implemented yet"
|
||||
)]
|
||||
UnsupportedPlatform {
|
||||
platform: &'static str,
|
||||
planned_endpoint: &'static str,
|
||||
},
|
||||
/// No home directory (or runtime directory) to derive a default path from.
|
||||
#[error(
|
||||
"cannot resolve the Codewhale runtime directory for the daemon socket: no home directory"
|
||||
)]
|
||||
RuntimeDirUnavailable,
|
||||
/// `CODEWHALE_HOME` is set but not a usable absolute path.
|
||||
#[error("invalid CODEWHALE_HOME override: {0}")]
|
||||
InvalidHomeOverride(String),
|
||||
/// Unix socket paths are limited to roughly one hundred bytes.
|
||||
#[error("daemon socket path {} is {len} bytes; this platform allows at most {max}", path.display())]
|
||||
PathTooLong {
|
||||
path: PathBuf,
|
||||
len: usize,
|
||||
max: usize,
|
||||
},
|
||||
/// Something other than a socket already sits at the path. Refused so a
|
||||
/// misconfigured path can never delete a user's file.
|
||||
#[error("{} exists and is not a unix socket; refusing to remove it", path.display())]
|
||||
NotASocket { path: PathBuf },
|
||||
/// A daemon answered on the socket: this one must not replace it.
|
||||
#[error(
|
||||
"a live listener already answers on {}; refusing to replace it (another codewhale daemon, or something else bound to this path)",
|
||||
path.display()
|
||||
)]
|
||||
AlreadyRunning { path: PathBuf },
|
||||
/// The liveness probe neither connected nor was refused within the
|
||||
/// budget. Refused rather than clobbered; remove the file by hand if the
|
||||
/// old daemon is truly gone.
|
||||
#[error("liveness probe of {} timed out; refusing to replace a socket that may be live", path.display())]
|
||||
ProbeTimedOut { path: PathBuf },
|
||||
/// Filesystem or socket I/O failed.
|
||||
#[error("{context} ({})", path.display())]
|
||||
Io {
|
||||
context: &'static str,
|
||||
path: PathBuf,
|
||||
#[source]
|
||||
source: std::io::Error,
|
||||
},
|
||||
/// The app-server state (config, state store, runtime) failed to build.
|
||||
///
|
||||
/// The source carries the real cause; a caller that prints only this
|
||||
/// message hides it. `serve --http` on a mount where the endpoint
|
||||
/// directory cannot be private used to surface here as a bare
|
||||
/// "failed to build daemon state", which named neither the directory nor
|
||||
/// the reason — see [`Self::EndpointDirectoryUnsupported`].
|
||||
#[error("failed to build daemon state")]
|
||||
State(#[source] anyhow::Error),
|
||||
/// The filesystem cannot hold the private directory an endpoint needs.
|
||||
///
|
||||
/// A private endpoint directory must be current-user-owned and closed to
|
||||
/// group and other. HarmonyOS hmdfs — the 2in1 user directory — reports the
|
||||
/// logged-in user's uid and a group-writable mode for every object created
|
||||
/// on it, so `chmod` answers `EPERM` and no directory there can satisfy that
|
||||
/// check.
|
||||
/// This is a missing capability, not a refusal: an owner with an HTTP
|
||||
/// frontend can run without its control socket, and does.
|
||||
#[error(
|
||||
"the filesystem cannot hold a private endpoint directory at {}: {reason}",
|
||||
path.display()
|
||||
)]
|
||||
EndpointDirectoryUnsupported { path: PathBuf, reason: String },
|
||||
}
|
||||
|
||||
/// How to start the daemon socket transport.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct DaemonSocketOptions {
|
||||
@@ -140,145 +45,6 @@ pub struct DaemonSocketOptions {
|
||||
pub config_path: Option<PathBuf>,
|
||||
}
|
||||
|
||||
/// Inputs to [`resolve_socket_path`], separated from the environment so the
|
||||
/// precedence rules are a pure, testable function.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct SocketPathInputs {
|
||||
/// `--socket-path`.
|
||||
pub explicit: Option<PathBuf>,
|
||||
/// A valid explicit `CODEWHALE_HOME`.
|
||||
pub codewhale_home_override: Option<PathBuf>,
|
||||
/// `$XDG_RUNTIME_DIR`, when set and non-empty.
|
||||
pub xdg_runtime_dir: Option<PathBuf>,
|
||||
/// The user's home directory.
|
||||
pub user_home: Option<PathBuf>,
|
||||
/// Whether the macOS Application Support layout applies.
|
||||
pub macos: bool,
|
||||
}
|
||||
|
||||
impl SocketPathInputs {
|
||||
/// Capture the live environment.
|
||||
pub fn from_environment(explicit: Option<PathBuf>) -> Result<Self, DaemonSocketError> {
|
||||
let codewhale_home_override = codewhale_paths::codewhale_home_override()
|
||||
.map_err(|err| DaemonSocketError::InvalidHomeOverride(err.to_string()))?;
|
||||
let xdg_runtime_dir = std::env::var_os("XDG_RUNTIME_DIR")
|
||||
.filter(|value| !value.is_empty())
|
||||
.map(PathBuf::from);
|
||||
Ok(Self {
|
||||
explicit,
|
||||
codewhale_home_override,
|
||||
xdg_runtime_dir,
|
||||
user_home: codewhale_paths::user_home(),
|
||||
macos: cfg!(target_os = "macos"),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Apply the precedence rules documented at the module level and enforce the
|
||||
/// kernel's path-length limit.
|
||||
pub fn resolve_socket_path(inputs: &SocketPathInputs) -> Result<PathBuf, DaemonSocketError> {
|
||||
let path = if let Some(explicit) = inputs.explicit.clone() {
|
||||
explicit
|
||||
} else if let Some(home) = inputs.codewhale_home_override.clone() {
|
||||
home.join("run").join(DAEMON_SOCKET_FILE_NAME)
|
||||
} else if let Some(runtime_dir) = inputs.xdg_runtime_dir.clone() {
|
||||
runtime_dir.join("codewhale").join(DAEMON_SOCKET_FILE_NAME)
|
||||
} else {
|
||||
let user_home = inputs
|
||||
.user_home
|
||||
.clone()
|
||||
.ok_or(DaemonSocketError::RuntimeDirUnavailable)?;
|
||||
if inputs.macos {
|
||||
user_home
|
||||
.join("Library")
|
||||
.join("Application Support")
|
||||
.join("codewhale")
|
||||
.join(DAEMON_SOCKET_FILE_NAME)
|
||||
} else {
|
||||
user_home
|
||||
.join(codewhale_paths::CODEWHALE_APP_DIR)
|
||||
.join("run")
|
||||
.join(DAEMON_SOCKET_FILE_NAME)
|
||||
}
|
||||
};
|
||||
let len = path.as_os_str().len();
|
||||
if len > MAX_SOCKET_PATH_BYTES {
|
||||
return Err(DaemonSocketError::PathTooLong {
|
||||
path,
|
||||
len,
|
||||
max: MAX_SOCKET_PATH_BYTES,
|
||||
});
|
||||
}
|
||||
Ok(path)
|
||||
}
|
||||
|
||||
/// The socket path this host would use with no explicit override.
|
||||
#[cfg(unix)]
|
||||
pub fn default_socket_path() -> Result<PathBuf, DaemonSocketError> {
|
||||
resolve_socket_path(&SocketPathInputs::from_environment(None)?)
|
||||
}
|
||||
|
||||
/// Digest bytes that name a derived socket. Six bytes (twelve hex characters,
|
||||
/// the same digest length the runtime-boundary id uses) make the basename a
|
||||
/// fixed 23 bytes with the `store-` prefix and the `.sock` suffix, which keeps
|
||||
/// the hard kernel path limit reachable only for an already-close default
|
||||
/// endpoint instead of consuming the remaining headroom.
|
||||
#[cfg(unix)]
|
||||
const DERIVED_STORE_HASH_BYTES: usize = 6;
|
||||
|
||||
/// The control endpoint for a selected runtime store: the well-known default
|
||||
/// socket when the store is the default selection, and a private sibling
|
||||
/// derived from the store root otherwise.
|
||||
///
|
||||
/// One control endpoint per runtime store, as a pure function of the store
|
||||
/// selection. A client that selects a non-default store (`CODEWHALE_RUNTIME_DIR`,
|
||||
/// e.g. the VS Code extension's per-workspace store) elects its own owner
|
||||
/// instead of colliding with the default store's owner on the shared socket;
|
||||
/// the default store keeps the historical path, so discovery, `doctor` and
|
||||
/// single-engine setups are unchanged.
|
||||
///
|
||||
/// `store_root` is the canonicalized runtime store root the caller selected;
|
||||
/// `None` means "the default selection" (no override in effect).
|
||||
///
|
||||
/// Unix only: a Windows owner endpoint is a named pipe, and its derived-name
|
||||
/// form is follow-up work. Windows keeps the well-known endpoint it has today.
|
||||
#[cfg(unix)]
|
||||
pub fn store_selected_socket_path(
|
||||
store_root: Option<&Path>,
|
||||
inputs: &SocketPathInputs,
|
||||
) -> Result<PathBuf, DaemonSocketError> {
|
||||
use anyhow::Context as _;
|
||||
use sha2::Digest as _;
|
||||
|
||||
let base = resolve_socket_path(inputs)?;
|
||||
let Some(store_root) = store_root else {
|
||||
return Ok(base);
|
||||
};
|
||||
let mut digest = sha2::Sha256::new();
|
||||
// The tag keeps this name from colliding with any other use of the store
|
||||
// root's digest in the same directory.
|
||||
digest.update(b"Codewhale/store-socket/v1\0");
|
||||
digest.update(store_root.as_os_str().as_encoded_bytes());
|
||||
let suffix: String = digest.finalize()[..DERIVED_STORE_HASH_BYTES]
|
||||
.iter()
|
||||
.map(|byte| format!("{byte:02x}"))
|
||||
.collect();
|
||||
let parent = base
|
||||
.parent()
|
||||
.context("default daemon socket has no parent directory")
|
||||
.map_err(DaemonSocketError::State)?;
|
||||
let derived = parent.join(format!("store-{suffix}.sock"));
|
||||
let len = derived.as_os_str().len();
|
||||
if len > MAX_SOCKET_PATH_BYTES {
|
||||
return Err(DaemonSocketError::PathTooLong {
|
||||
path: derived,
|
||||
len,
|
||||
max: MAX_SOCKET_PATH_BYTES,
|
||||
});
|
||||
}
|
||||
Ok(derived)
|
||||
}
|
||||
|
||||
/// The socket path this host would use with no explicit override.
|
||||
#[cfg(not(any(unix, windows)))]
|
||||
pub fn default_socket_path() -> Result<PathBuf, DaemonSocketError> {
|
||||
@@ -356,28 +122,8 @@ fn unsupported_platform() -> DaemonSocketError {
|
||||
}
|
||||
}
|
||||
|
||||
/// One bounded blocking boundary for held filesystem/process identity work.
|
||||
/// The worker retains its permit and captured handles if the waiter cancels.
|
||||
#[cfg(any(unix, windows))]
|
||||
pub async fn owner_work<T, F>(work: F) -> anyhow::Result<T>
|
||||
where
|
||||
T: Send + 'static,
|
||||
F: FnOnce() -> anyhow::Result<T> + Send + 'static,
|
||||
{
|
||||
static SLOTS: std::sync::OnceLock<std::sync::Arc<tokio::sync::Semaphore>> =
|
||||
std::sync::OnceLock::new();
|
||||
let permit = SLOTS
|
||||
.get_or_init(|| std::sync::Arc::new(tokio::sync::Semaphore::new(4)))
|
||||
.clone()
|
||||
.acquire_owned()
|
||||
.await?;
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let _permit = permit;
|
||||
work()
|
||||
})
|
||||
.await
|
||||
.map_err(anyhow::Error::from)?
|
||||
}
|
||||
pub use codewhale_runtime::runtime_control::owner_work;
|
||||
|
||||
// Blocking endpoint mutations retain the actual captured manager through
|
||||
// its existing frontend, even after cancellation of the async caller.
|
||||
@@ -882,6 +628,7 @@ mod platform {
|
||||
|
||||
use anyhow::Result;
|
||||
use codewhale_config::private_directory::{PrivateDirectory, PrivateSocketIdentity};
|
||||
use codewhale_paths::daemon_socket::{SocketPathInputs, resolve_socket_path};
|
||||
use tokio::net::{UnixListener, UnixStream};
|
||||
use tokio::sync::watch;
|
||||
|
||||
@@ -889,7 +636,7 @@ mod platform {
|
||||
use super::{ClientIdentity, ConnectionRegistry};
|
||||
use super::{
|
||||
ConnectionContext, ConnectionTasks, DaemonShutdownHandle, DaemonSocketError,
|
||||
DaemonSocketOptions, SocketPathInputs, resolve_socket_path, socket_owner_work,
|
||||
DaemonSocketOptions, socket_owner_work,
|
||||
};
|
||||
use crate::{AppState, AppTransport, build_state_off_runtime};
|
||||
|
||||
|
||||
@@ -5,12 +5,13 @@
|
||||
use crate::daemon_client::UnavailablePublication;
|
||||
use crate::daemon_socket::{
|
||||
AuthorizedPeer, ConnectionTasks, DaemonShutdownHandle, DaemonSocketError, DaemonSocketOptions,
|
||||
SocketPathInputs, connection_context, owner_work, run_authorized_connection, socket_owner_work,
|
||||
connection_context, owner_work, run_authorized_connection, socket_owner_work,
|
||||
};
|
||||
use crate::{AppState, AppTransport, build_state_off_runtime};
|
||||
use anyhow::{Context as _, Result};
|
||||
use codewhale_config::private_directory::PrivateDirectory;
|
||||
use codewhale_config::windows_identity::{CurrentWindowsUser, OwnerOnlyAcl, WindowsPeerProcess};
|
||||
use codewhale_paths::daemon_socket::SocketPathInputs;
|
||||
use codewhale_protocol::RuntimeOwnerReceipt;
|
||||
use std::fs::File;
|
||||
use std::os::windows::io::AsRawHandle as _;
|
||||
|
||||
@@ -12,7 +12,7 @@ use axum::middleware::Next;
|
||||
use axum::response::{IntoResponse as _, Response};
|
||||
use axum::routing::{get, post};
|
||||
use axum::{Json, Router, middleware};
|
||||
use codewhale_protocol::runtime::MAX_RUNTIME_IMAGE_BODY_BYTES;
|
||||
use codewhale_app_server_protocol::runtime::MAX_RUNTIME_IMAGE_BODY_BYTES;
|
||||
use codewhale_protocol::{AppRequest, AppResponse, PromptRequest, ThreadRequest};
|
||||
use serde_json::{Value, json};
|
||||
use std::path::PathBuf;
|
||||
|
||||
@@ -5,11 +5,13 @@ use anyhow::Context as _;
|
||||
use anyhow::{Result, anyhow};
|
||||
use axum::http::{Method, StatusCode};
|
||||
use codewhale_agent::ModelRegistry;
|
||||
#[cfg(test)]
|
||||
use codewhale_app_server_protocol::runtime::MAX_RUNTIME_IMAGE_BODY_BYTES;
|
||||
use codewhale_app_server_protocol::runtime::RuntimeImageInput;
|
||||
use codewhale_config::ConfigStore;
|
||||
use codewhale_core::Runtime;
|
||||
use codewhale_hooks::{HookDispatcher, JsonlHookSink, StdoutHookSink, UnixSocketHookSink};
|
||||
use codewhale_protocol::EventFrame;
|
||||
use codewhale_protocol::runtime::{MAX_RUNTIME_IMAGE_BODY_BYTES, RuntimeImageInput};
|
||||
use codewhale_state::StateStore;
|
||||
use serde::Deserialize;
|
||||
use serde::de::DeserializeOwned;
|
||||
@@ -100,14 +102,7 @@ impl std::fmt::Debug for AppServerOptions {
|
||||
}
|
||||
}
|
||||
|
||||
/// Selected frontend facts passed by the canonical CLI in memory.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum RuntimeControlFrontend {
|
||||
Stdio,
|
||||
Socket { path: Option<PathBuf> },
|
||||
LegacyHttp,
|
||||
Acp { model: String },
|
||||
}
|
||||
pub use codewhale_runtime::runtime_control::RuntimeControlFrontend;
|
||||
|
||||
/// A bounded listener request admitted only over the authenticated owner channel.
|
||||
/// The bearer is explicit operator input, never discovery or response data.
|
||||
@@ -339,6 +334,7 @@ struct JsonRpcError {
|
||||
code: i64,
|
||||
message: String,
|
||||
data: Option<Value>,
|
||||
transport_error: Option<anyhow::Error>,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
@@ -841,6 +837,7 @@ fn jsonrpc_error(id: Option<Value>, err: JsonRpcError) -> Value {
|
||||
impl JsonRpcError {
|
||||
fn parse_error(message: impl Into<String>) -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: -32700,
|
||||
message: message.into(),
|
||||
data: None,
|
||||
@@ -849,6 +846,7 @@ impl JsonRpcError {
|
||||
|
||||
fn invalid_request(message: impl Into<String>) -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: -32600,
|
||||
message: message.into(),
|
||||
data: None,
|
||||
@@ -857,6 +855,7 @@ impl JsonRpcError {
|
||||
|
||||
fn method_not_found(method: &str) -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: -32601,
|
||||
message: format!("unsupported method: {method}"),
|
||||
data: None,
|
||||
@@ -865,6 +864,7 @@ impl JsonRpcError {
|
||||
|
||||
fn invalid_params(message: impl Into<String>) -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: -32602,
|
||||
message: message.into(),
|
||||
data: None,
|
||||
@@ -873,6 +873,7 @@ impl JsonRpcError {
|
||||
|
||||
fn unsupported_interactive_execution() -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: -32602,
|
||||
message: "Compatibility transports cannot answer approvals or clarification requests. Use codewhale app-server --http and /v1/* for interactive turns and goal execution; tool-free turn requests require explicit allowed_tools: [].".to_string(),
|
||||
data: Some(json!({
|
||||
@@ -890,6 +891,7 @@ impl JsonRpcError {
|
||||
fn runtime_unavailable(message: impl Into<String>) -> Self {
|
||||
let message = message.into();
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: RUNTIME_UNAVAILABLE_CODE,
|
||||
message: message.clone(),
|
||||
data: Some(json!({
|
||||
@@ -902,6 +904,7 @@ impl JsonRpcError {
|
||||
/// Server error (-32000..-32099): the named thread does not exist.
|
||||
fn thread_not_found(thread_id: &str) -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: THREAD_NOT_FOUND_CODE,
|
||||
message: format!("thread not found: {thread_id}"),
|
||||
data: Some(json!({
|
||||
@@ -913,6 +916,7 @@ impl JsonRpcError {
|
||||
|
||||
fn internal(message: impl Into<String>) -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: -32603,
|
||||
message: message.into(),
|
||||
data: None,
|
||||
@@ -924,6 +928,7 @@ impl JsonRpcError {
|
||||
#[cfg(any(unix, windows))]
|
||||
fn attach_required(method: &str) -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: ATTACH_REQUIRED_CODE,
|
||||
message: format!("send daemon/attach before `{method}`"),
|
||||
data: Some(json!({
|
||||
@@ -938,6 +943,7 @@ impl JsonRpcError {
|
||||
#[cfg(any(unix, windows))]
|
||||
fn daemon_already_claimed(owner: &Value) -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: DAEMON_ALREADY_CLAIMED_CODE,
|
||||
message: "daemon already claimed by another client; attach with mode=attach"
|
||||
.to_string(),
|
||||
@@ -951,6 +957,7 @@ impl JsonRpcError {
|
||||
/// Server error (-32000..-32099): only the owner may stop the daemon.
|
||||
fn not_daemon_owner() -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: NOT_DAEMON_OWNER_CODE,
|
||||
message: "only the client that claimed this daemon may shut it down".to_string(),
|
||||
data: Some(json!({ "error": "not_daemon_owner" })),
|
||||
@@ -962,6 +969,7 @@ impl JsonRpcError {
|
||||
#[cfg(any(unix, windows))]
|
||||
fn daemon_version_skew(expected: &str, actual: &str) -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: DAEMON_VERSION_SKEW_CODE,
|
||||
message: format!(
|
||||
"daemon version {actual} does not match the client's expected {expected}"
|
||||
@@ -977,6 +985,7 @@ impl JsonRpcError {
|
||||
/// Server error (-32000..-32099): `daemon/attach` after attaching.
|
||||
fn already_attached() -> Self {
|
||||
Self {
|
||||
transport_error: None,
|
||||
code: ALREADY_ATTACHED_CODE,
|
||||
message: "this connection is already attached".to_string(),
|
||||
data: Some(json!({ "error": "already_attached" })),
|
||||
|
||||
+126
-21
@@ -8,7 +8,7 @@ use crate::{
|
||||
};
|
||||
use anyhow::Context as _;
|
||||
use anyhow::Result;
|
||||
use codewhale_protocol::runtime::MAX_RUNTIME_IMAGE_BODY_BYTES;
|
||||
use codewhale_app_server_protocol::runtime::MAX_RUNTIME_IMAGE_BODY_BYTES;
|
||||
use codewhale_protocol::{
|
||||
AppRequest, PromptRequest, ThreadGoalClearParams, ThreadGoalGetParams, ThreadGoalSetParams,
|
||||
ThreadRequest, ThreadResponse,
|
||||
@@ -20,11 +20,26 @@ use std::time::Duration;
|
||||
use tokio::io::{AsyncBufRead, AsyncWrite};
|
||||
use tokio::io::{AsyncBufReadExt as _, AsyncWriteExt as _};
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum FrameRefusal {
|
||||
#[error("control frame exceeds the 8 MiB transport limit")]
|
||||
TooLarge,
|
||||
#[error("control frame is not UTF-8")]
|
||||
InvalidUtf8,
|
||||
}
|
||||
|
||||
pub fn is_frame_refusal(error: &std::io::Error) -> bool {
|
||||
error
|
||||
.get_ref()
|
||||
.is_some_and(|source| source.is::<FrameRefusal>())
|
||||
}
|
||||
|
||||
/// Cancellation-safe, allocation-bounded newline framing shared by every
|
||||
/// local control transport. Consumed partial bytes remain owned by the reader.
|
||||
pub struct BoundedLines<R> {
|
||||
reader: R,
|
||||
pub(crate) pending: Vec<u8>,
|
||||
discarding: bool,
|
||||
}
|
||||
|
||||
impl<R: AsyncBufRead + Unpin> BoundedLines<R> {
|
||||
@@ -32,6 +47,7 @@ impl<R: AsyncBufRead + Unpin> BoundedLines<R> {
|
||||
Self {
|
||||
reader,
|
||||
pending: Vec::new(),
|
||||
discarding: false,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -42,29 +58,41 @@ impl<R: AsyncBufRead + Unpin> BoundedLines<R> {
|
||||
loop {
|
||||
let available = self.reader.fill_buf().await?;
|
||||
if available.is_empty() {
|
||||
self.discarding = false;
|
||||
if self.pending.is_empty() {
|
||||
return Ok(None);
|
||||
}
|
||||
if self.pending.len() > MAX_RUNTIME_IMAGE_BODY_BYTES {
|
||||
self.pending.clear();
|
||||
return Err(std::io::Error::new(
|
||||
std::io::ErrorKind::InvalidData,
|
||||
"control request exceeds the 8 MiB transport limit",
|
||||
FrameRefusal::TooLarge,
|
||||
));
|
||||
}
|
||||
let bytes = std::mem::take(&mut self.pending);
|
||||
return String::from_utf8(bytes).map(Some).map_err(|_| {
|
||||
std::io::Error::new(
|
||||
std::io::ErrorKind::InvalidData,
|
||||
"control frame is not UTF-8",
|
||||
)
|
||||
std::io::Error::new(std::io::ErrorKind::InvalidData, FrameRefusal::InvalidUtf8)
|
||||
});
|
||||
}
|
||||
let available = &available[..available.len().min(8192)];
|
||||
let newline = available.iter().position(|&byte| byte == b'\n');
|
||||
let count = newline.unwrap_or(available.len());
|
||||
if self.discarding {
|
||||
self.reader.consume(count + usize::from(newline.is_some()));
|
||||
if newline.is_some() {
|
||||
self.discarding = false;
|
||||
} else {
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if count > (MAX_RUNTIME_IMAGE_BODY_BYTES + 1).saturating_sub(self.pending.len()) {
|
||||
self.pending.clear();
|
||||
self.reader.consume(count + usize::from(newline.is_some()));
|
||||
self.discarding = newline.is_none();
|
||||
return Err(std::io::Error::new(
|
||||
std::io::ErrorKind::InvalidData,
|
||||
"control request exceeds the 8 MiB transport limit",
|
||||
FrameRefusal::TooLarge,
|
||||
));
|
||||
}
|
||||
// Do not let Vec's geometric growth double an almost-full frame.
|
||||
@@ -74,9 +102,11 @@ impl<R: AsyncBufRead + Unpin> BoundedLines<R> {
|
||||
if self.pending.len() > MAX_RUNTIME_IMAGE_BODY_BYTES
|
||||
&& self.pending.last() != Some(&b'\r')
|
||||
{
|
||||
self.pending.clear();
|
||||
self.discarding = newline.is_none();
|
||||
return Err(std::io::Error::new(
|
||||
std::io::ErrorKind::InvalidData,
|
||||
"control request exceeds the 8 MiB transport limit",
|
||||
FrameRefusal::TooLarge,
|
||||
));
|
||||
}
|
||||
if newline.is_some() {
|
||||
@@ -85,10 +115,7 @@ impl<R: AsyncBufRead + Unpin> BoundedLines<R> {
|
||||
}
|
||||
let bytes = std::mem::take(&mut self.pending);
|
||||
return String::from_utf8(bytes).map(Some).map_err(|_| {
|
||||
std::io::Error::new(
|
||||
std::io::ErrorKind::InvalidData,
|
||||
"control frame is not UTF-8",
|
||||
)
|
||||
std::io::Error::new(std::io::ErrorKind::InvalidData, FrameRefusal::InvalidUtf8)
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -214,7 +241,10 @@ where
|
||||
return Ok(StdioLoopExit::Shutdown);
|
||||
}
|
||||
}
|
||||
Err(err) => {
|
||||
Err(mut err) => {
|
||||
if let Some(error) = err.transport_error.take() {
|
||||
return Err(error);
|
||||
}
|
||||
write_stdio_line(&mut writer, &jsonrpc_error(id, err)).await?;
|
||||
}
|
||||
}
|
||||
@@ -379,14 +409,92 @@ pub(crate) fn is_control_detach(request: &JsonRpcRequest) -> bool {
|
||||
.is_some_and(serde_json::Map::is_empty)
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct ControlFrameBuffer {
|
||||
bytes: Vec<u8>,
|
||||
exceeded: bool,
|
||||
}
|
||||
|
||||
impl std::io::Write for ControlFrameBuffer {
|
||||
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
|
||||
if bytes.len() > MAX_RUNTIME_IMAGE_BODY_BYTES.saturating_sub(self.bytes.len()) {
|
||||
self.exceeded = true;
|
||||
return Err(std::io::Error::new(
|
||||
std::io::ErrorKind::InvalidData,
|
||||
FrameRefusal::TooLarge,
|
||||
));
|
||||
}
|
||||
let needed = self.bytes.len() + bytes.len();
|
||||
if needed > self.bytes.capacity() {
|
||||
let capacity = self
|
||||
.bytes
|
||||
.capacity()
|
||||
.saturating_mul(2)
|
||||
.max(needed)
|
||||
.min(MAX_RUNTIME_IMAGE_BODY_BYTES);
|
||||
self.bytes.reserve_exact(capacity - self.bytes.len());
|
||||
}
|
||||
self.bytes.extend_from_slice(bytes);
|
||||
Ok(bytes.len())
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> std::io::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
pub fn serialize_control_frame(value: &Value) -> std::io::Result<Vec<u8>> {
|
||||
let mut output = ControlFrameBuffer::default();
|
||||
if let Err(error) = serde_json::to_writer(&mut output, value) {
|
||||
return Err(if output.exceeded {
|
||||
std::io::Error::new(std::io::ErrorKind::InvalidData, FrameRefusal::TooLarge)
|
||||
} else {
|
||||
std::io::Error::new(std::io::ErrorKind::InvalidData, error)
|
||||
});
|
||||
}
|
||||
Ok(output.bytes)
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("control transport write failed; pending outcomes may be uncertain: {0}")]
|
||||
struct ControlWriteFailure(#[source] std::io::Error);
|
||||
|
||||
pub fn is_control_write_failure(error: &anyhow::Error) -> bool {
|
||||
error
|
||||
.chain()
|
||||
.any(|source| source.is::<ControlWriteFailure>())
|
||||
}
|
||||
|
||||
pub async fn write_control_line<W: AsyncWrite + Unpin + ?Sized>(
|
||||
writer: &mut W,
|
||||
body: &[u8],
|
||||
) -> Result<()> {
|
||||
if body.len() > MAX_RUNTIME_IMAGE_BODY_BYTES {
|
||||
return Err(
|
||||
std::io::Error::new(std::io::ErrorKind::InvalidData, FrameRefusal::TooLarge).into(),
|
||||
);
|
||||
}
|
||||
let result = tokio::time::timeout(Duration::from_secs(30), async {
|
||||
writer.write_all(body).await?;
|
||||
writer.write_all(b"\n").await?;
|
||||
writer.flush().await
|
||||
})
|
||||
.await
|
||||
.unwrap_or_else(|_| {
|
||||
Err(std::io::Error::new(
|
||||
std::io::ErrorKind::TimedOut,
|
||||
"control transport write did not settle within 30 seconds",
|
||||
))
|
||||
});
|
||||
result.map_err(|error| ControlWriteFailure(error).into())
|
||||
}
|
||||
|
||||
pub(crate) async fn write_stdio_line<W: AsyncWrite + Unpin>(
|
||||
writer: &mut W,
|
||||
response: &Value,
|
||||
) -> Result<()> {
|
||||
writer.write_all(&serde_json::to_vec(response)?).await?;
|
||||
writer.write_all(b"\n").await?;
|
||||
writer.flush().await?;
|
||||
Ok(())
|
||||
let frame = serialize_control_frame(response)?;
|
||||
write_control_line(writer, &frame).await
|
||||
}
|
||||
|
||||
pub(crate) async fn handle_stdio_thread_message<W: AsyncWrite + Unpin>(
|
||||
@@ -515,10 +623,7 @@ pub(crate) async fn emit_stdio_event<W: AsyncWrite + Unpin>(
|
||||
writer: &mut W,
|
||||
event: Value,
|
||||
) -> Result<()> {
|
||||
writer.write_all(&serde_json::to_vec(&event)?).await?;
|
||||
writer.write_all(b"\n").await?;
|
||||
writer.flush().await?;
|
||||
Ok(())
|
||||
write_stdio_line(writer, &event).await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -3,7 +3,7 @@ use crate::{
|
||||
AppState, JsonRpcError, RuntimeThreadHint, TurnTerminalStatus, TurnTranscript, thread_control,
|
||||
};
|
||||
use anyhow::{Result, anyhow};
|
||||
use codewhale_protocol::runtime::RuntimeImageInput;
|
||||
use codewhale_app_server_protocol::runtime::RuntimeImageInput;
|
||||
use codewhale_protocol::{PromptRequest, PromptResponse, ThreadRequest, ThreadResponse};
|
||||
use serde_json::{Value, json};
|
||||
use std::path::Path;
|
||||
@@ -160,7 +160,13 @@ pub(crate) async fn run_bridged_turn<W: AsyncWrite + Unpin>(
|
||||
let mut thread_map = state.runtime_thread_map.lock().await;
|
||||
bridge.forget_thread(&mut thread_map, turn.thread_key);
|
||||
}
|
||||
result.map_err(|err| JsonRpcError::internal(err.to_string()))
|
||||
result.map_err(|err| {
|
||||
let mut response = JsonRpcError::internal(err.to_string());
|
||||
if crate::stdio::is_control_write_failure(&err) {
|
||||
response.transport_error = Some(err);
|
||||
}
|
||||
response
|
||||
})
|
||||
}
|
||||
|
||||
/// Run a prompt as a genuine model turn and return what the model actually
|
||||
|
||||
@@ -15,7 +15,6 @@ use std::process::Command;
|
||||
use anyhow::{Context, Result, anyhow, bail};
|
||||
use clap::{Args, CommandFactory, FromArgMatches, Parser, Subcommand, ValueEnum};
|
||||
use clap_complete::{Shell, generate};
|
||||
use codewhale_app_server::RuntimeControlFrontend;
|
||||
use codewhale_config::credentials::{
|
||||
clear_provider_api_key_from_config, provider_slot, set_provider_api_key,
|
||||
};
|
||||
@@ -26,6 +25,7 @@ use codewhale_config::{
|
||||
classify_config_api_key_value, provider_base_url_is_official,
|
||||
};
|
||||
use codewhale_execpolicy::{AskForApproval, ExecPolicyContext, ExecPolicyEngine};
|
||||
use codewhale_runtime::runtime_control::RuntimeControlFrontend;
|
||||
use codewhale_secrets::Secrets;
|
||||
use codewhale_telemetry::{
|
||||
self as telemetry, Counters, DurationBucket, Errors, Event, ExitClass, SessionSource, Surface,
|
||||
|
||||
@@ -124,7 +124,7 @@ pub trait CommandDebugUndoContext {
|
||||
/// the conversation when it is still its last one. Preserves snapshot
|
||||
/// ownership, regular-file checks, backup and preflight, and asks for no
|
||||
/// trust: it only puts back, inside the workspace, what that request
|
||||
/// changed. `force` accepts a request with no recorded end. Never falls
|
||||
/// changed. `force` never waives owned identity or a complete write record. Never falls
|
||||
/// back to chat truncation: the portable handler decides that from the
|
||||
/// typed outcome.
|
||||
fn undo_files(&mut self, force: bool) -> DebugUndoOutcome;
|
||||
|
||||
@@ -70,6 +70,10 @@ pub enum DebugAction {
|
||||
diff: String,
|
||||
},
|
||||
OpenContextInspector,
|
||||
OpenUndoSelector,
|
||||
UndoLatestRequest {
|
||||
force: bool,
|
||||
},
|
||||
SendMessage(String),
|
||||
SyncSession(SessionSyncPayload),
|
||||
/// Install and persist a conversation rollback; `/retry` resends its
|
||||
|
||||
@@ -433,10 +433,10 @@ impl CommandDebugUndoContext for UndoFacet {
|
||||
}
|
||||
}
|
||||
|
||||
/// `force` is the only option `/undo` takes: it reaches the host as a typed
|
||||
/// flag, and anything else is reported before a file or message is touched.
|
||||
/// `/undo` defers mutation to the admitted host action; invalid options
|
||||
/// are reported before a file or message is touched.
|
||||
#[test]
|
||||
fn undo_passes_force_to_the_host_and_rejects_any_other_option() {
|
||||
fn undo_defers_latest_and_force_and_rejects_unknown_options() {
|
||||
let events = Rc::new(RefCell::new(vec![]));
|
||||
let mut history = HistoryFacet {
|
||||
events: events.clone(),
|
||||
@@ -447,13 +447,12 @@ fn undo_passes_force_to_the_host_and_rejects_any_other_option() {
|
||||
events: events.clone(),
|
||||
outcome: DebugUndoOutcome::OpenEnded,
|
||||
};
|
||||
for (arg, expected) in [
|
||||
(None, "undo_files"),
|
||||
(Some(" "), "undo_files"),
|
||||
(Some("force"), "undo_files_forced"),
|
||||
(Some(" FORCE "), "undo_files_forced"),
|
||||
for (arg, force) in [
|
||||
(None, false),
|
||||
(Some(" "), false),
|
||||
(Some("force"), true),
|
||||
(Some(" FORCE "), true),
|
||||
] {
|
||||
events.borrow_mut().clear();
|
||||
let result = undo::undo(
|
||||
CommandContexts::empty()
|
||||
.with_debug_undo(&mut facet)
|
||||
@@ -461,17 +460,16 @@ fn undo_passes_force_to_the_host_and_rejects_any_other_option() {
|
||||
.with_presentation(&mut UndoPresentation),
|
||||
arg,
|
||||
);
|
||||
assert_eq!(&*events.borrow(), &[expected], "{arg:?}");
|
||||
assert_eq!(
|
||||
result.message.as_deref(),
|
||||
Some(
|
||||
"Nothing was undone. There is no record of where your last request's changes end, so undoing it would also put back edits made since. To do that anyway, run `/undo force`."
|
||||
)
|
||||
assert!(
|
||||
events.borrow().is_empty(),
|
||||
"mutation precedes host admission: {arg:?}"
|
||||
);
|
||||
assert!(result.action.is_none());
|
||||
assert_eq!(
|
||||
result.action,
|
||||
Some(DebugAction::UndoLatestRequest { force })
|
||||
);
|
||||
assert!(result.message.is_none());
|
||||
}
|
||||
|
||||
events.borrow_mut().clear();
|
||||
let result = undo::undo(
|
||||
CommandContexts::empty()
|
||||
.with_debug_undo(&mut facet)
|
||||
@@ -605,43 +603,23 @@ fn undo_reports_the_request_each_file_and_the_conversation() {
|
||||
|
||||
#[test]
|
||||
fn undo_refusals_never_fall_back_to_history_or_parse_backend_text() {
|
||||
let events = Rc::new(RefCell::new(vec![]));
|
||||
let mut history = HistoryFacet {
|
||||
events: events.clone(),
|
||||
removed: 2,
|
||||
..Default::default()
|
||||
};
|
||||
for outcome in [
|
||||
DebugUndoOutcome::OpenEnded,
|
||||
DebugUndoOutcome::SnapshotPending,
|
||||
DebugUndoOutcome::ChangedSince {
|
||||
paths: vec!["a.txt".into()],
|
||||
},
|
||||
// Deliberately resembles the old English fallback prefixes. Its typed
|
||||
// failure must stay a refusal and cannot silently delete conversation.
|
||||
DebugUndoOutcome::RestoreBlocked("No snapshots found, but operation was refused".into()),
|
||||
DebugUndoOutcome::CompareFailed("read error".into()),
|
||||
DebugUndoOutcome::RestoreFailed("write error".into()),
|
||||
] {
|
||||
events.borrow_mut().clear();
|
||||
let mut facet = UndoFacet {
|
||||
events: events.clone(),
|
||||
outcome,
|
||||
};
|
||||
let result = undo::undo(
|
||||
CommandContexts::empty()
|
||||
.with_debug_undo(&mut facet)
|
||||
.with_debug_history(&mut history)
|
||||
.with_presentation(&mut UndoPresentation),
|
||||
None,
|
||||
);
|
||||
let result = undo::patch_result(outcome, &UndoPresentation);
|
||||
assert!(result.action.is_none());
|
||||
assert_eq!(&*events.borrow(), &["undo_files"]);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn undo_fallback_requires_both_facets_and_keeps_file_warning() {
|
||||
fn undo_requires_both_facets_and_defers_without_mutating_history() {
|
||||
let events = Rc::new(RefCell::new(vec![]));
|
||||
let mut facet = UndoFacet {
|
||||
events: events.clone(),
|
||||
@@ -661,38 +639,13 @@ fn undo_fallback_requires_both_facets_and_keeps_file_warning() {
|
||||
removed: 2,
|
||||
..Default::default()
|
||||
};
|
||||
let result = undo::undo(
|
||||
CommandContexts::empty()
|
||||
.with_debug_undo(&mut facet)
|
||||
.with_debug_history(&mut history)
|
||||
.with_presentation(&mut UndoPresentation),
|
||||
None,
|
||||
);
|
||||
assert_eq!(
|
||||
result.message.as_deref(),
|
||||
Some(
|
||||
"Removed 2 message(s)\nWorkspace files were NOT reverted — only the conversation was rolled back.\nSnapshot repo unavailable for /workspace: read denied"
|
||||
)
|
||||
);
|
||||
assert_eq!(&*events.borrow(), &["undo_files", "undo_chat"]);
|
||||
// A conversation-only undo always says what happened to the files.
|
||||
for (outcome, files_note) in [
|
||||
(
|
||||
DebugUndoOutcome::NoSnapshots,
|
||||
"Workspace files were NOT reverted — only the conversation was rolled back.",
|
||||
),
|
||||
(
|
||||
DebugUndoOutcome::NoSession,
|
||||
"Workspace files were NOT reverted — only the conversation was rolled back.",
|
||||
),
|
||||
(
|
||||
DebugUndoOutcome::NoOwnedSteps,
|
||||
"Workspace files were NOT reverted — only the conversation was rolled back.",
|
||||
),
|
||||
(DebugUndoOutcome::NoDifference, "No file changes to undo."),
|
||||
for outcome in [
|
||||
DebugUndoOutcome::NoSnapshots,
|
||||
DebugUndoOutcome::NoSession,
|
||||
DebugUndoOutcome::NoOwnedSteps,
|
||||
DebugUndoOutcome::NoDifference,
|
||||
] {
|
||||
facet.outcome = outcome;
|
||||
events.borrow_mut().clear();
|
||||
let result = undo::undo(
|
||||
CommandContexts::empty()
|
||||
.with_debug_undo(&mut facet)
|
||||
@@ -701,10 +654,14 @@ fn undo_fallback_requires_both_facets_and_keeps_file_warning() {
|
||||
None,
|
||||
);
|
||||
assert_eq!(
|
||||
result.message,
|
||||
Some(format!("Removed 2 message(s)\n{files_note}"))
|
||||
result.action,
|
||||
Some(DebugAction::UndoLatestRequest { force: false })
|
||||
);
|
||||
assert!(result.message.is_none());
|
||||
assert!(
|
||||
events.borrow().is_empty(),
|
||||
"an old facet cannot bypass asynchronous admission or discard history"
|
||||
);
|
||||
assert_eq!(&*events.borrow(), &["undo_files", "undo_chat"]);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@ registration!(
|
||||
registration!(
|
||||
UndoCmd,
|
||||
"undo",
|
||||
"/undo [force]",
|
||||
"/undo [force|select]",
|
||||
&[],
|
||||
"cmd_undo_description",
|
||||
Caps::DEBUG_UNDO
|
||||
@@ -242,96 +242,33 @@ fn conversation_report(
|
||||
|
||||
pub(super) fn undo(contexts: CommandContexts<'_>, arg: Option<&str>) -> CommandResult {
|
||||
let mut parts = contexts.into_parts();
|
||||
let Some(undo) = parts.debug_undo.as_deref_mut() else {
|
||||
if parts.debug_undo.is_none() {
|
||||
return CommandResult::error("Command capability unavailable: debug_undo");
|
||||
};
|
||||
let Some(history) = parts.debug_history.as_deref_mut() else {
|
||||
}
|
||||
if parts.debug_history.is_none() {
|
||||
return CommandResult::error("Command capability unavailable: debug_history");
|
||||
};
|
||||
}
|
||||
let Some(presentation) = parts.presentation.as_deref_mut() else {
|
||||
return CommandResult::error("Command capability unavailable: presentation");
|
||||
};
|
||||
match undo_report(undo, history, presentation, arg) {
|
||||
Ok(result) => result,
|
||||
Err(error) => CommandResult::error(error),
|
||||
}
|
||||
}
|
||||
|
||||
fn undo_report(
|
||||
undo: &mut dyn CommandDebugUndoContext,
|
||||
history: &mut dyn CommandDebugHistoryContext,
|
||||
presentation: &dyn CommandPresentationContext,
|
||||
arg: Option<&str>,
|
||||
) -> Result<CommandResult, String> {
|
||||
// `force` accepts a request whose end was never recorded. Anything else
|
||||
// is a mistake to report, not a request to undo.
|
||||
let force = match arg.map(str::trim).filter(|arg| !arg.is_empty()) {
|
||||
None => false,
|
||||
Some(arg) if arg.eq_ignore_ascii_case("force") => true,
|
||||
match arg.map(str::trim).filter(|arg| !arg.is_empty()) {
|
||||
None => CommandResult::action(DebugAction::UndoLatestRequest { force: false }),
|
||||
Some(arg) if arg.eq_ignore_ascii_case("force") => {
|
||||
CommandResult::action(DebugAction::UndoLatestRequest { force: true })
|
||||
}
|
||||
Some(arg) if arg.eq_ignore_ascii_case("select") => {
|
||||
CommandResult::action(DebugAction::OpenUndoSelector)
|
||||
}
|
||||
Some(other) => {
|
||||
return Ok(CommandResult::error(
|
||||
presentation.translate("cmd_undo_unknown_option", &[("option", other)])?,
|
||||
));
|
||||
match presentation.translate("cmd_undo_unknown_option", &[("option", other)]) {
|
||||
Ok(message) => CommandResult::error(message),
|
||||
Err(error) => CommandResult::error(error),
|
||||
}
|
||||
}
|
||||
};
|
||||
match undo.undo_files(force) {
|
||||
// No restore point of this conversation covers the request: say that
|
||||
// its files stayed as they are.
|
||||
DebugUndoOutcome::NoSnapshots
|
||||
| DebugUndoOutcome::NoSession
|
||||
| DebugUndoOutcome::NoOwnedSteps => conversation_only(
|
||||
history.undo_conversation(),
|
||||
presentation,
|
||||
"cmd_undo_files_not_reverted_note",
|
||||
),
|
||||
DebugUndoOutcome::NoDifference => conversation_only(
|
||||
history.undo_conversation(),
|
||||
presentation,
|
||||
"cmd_undo_no_file_changes_note",
|
||||
),
|
||||
DebugUndoOutcome::RepoUnavailable { workspace, error } => {
|
||||
let mut result = conversation_report(history.undo_conversation(), presentation)?;
|
||||
let note = format!(
|
||||
"{}\n{}",
|
||||
presentation.translate("cmd_undo_files_not_reverted_note", &[])?,
|
||||
presentation.translate(
|
||||
"cmd_undo_snapshot_repo_unavailable",
|
||||
&[
|
||||
("workspace", &workspace.display().to_string()),
|
||||
("error", &error),
|
||||
],
|
||||
)?
|
||||
);
|
||||
result.message = Some(match result.message.take() {
|
||||
Some(message) => format!("{message}\n{note}"),
|
||||
None => note,
|
||||
});
|
||||
Ok(result)
|
||||
}
|
||||
outcome => patch_report(outcome, presentation),
|
||||
}
|
||||
}
|
||||
|
||||
/// A conversation-only undo, with what happened to the files stated under it.
|
||||
fn conversation_only(
|
||||
undone: DebugConversationUndo,
|
||||
presentation: &dyn CommandPresentationContext,
|
||||
note_key: &str,
|
||||
) -> Result<CommandResult, String> {
|
||||
let removed = undone.removed;
|
||||
let mut result = conversation_report(undone, presentation)?;
|
||||
if removed > 0 {
|
||||
let note = presentation.translate(note_key, &[])?;
|
||||
result.message = result
|
||||
.message
|
||||
.take()
|
||||
.map(|message| format!("{message}\n{note}"));
|
||||
}
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
/// Format snapshot outcomes without invoking host operations. Host tests also
|
||||
/// use this for the file-only operation before the command's chat fallback.
|
||||
/// Format snapshot outcomes without invoking host operations.
|
||||
pub fn patch_result(
|
||||
outcome: DebugUndoOutcome,
|
||||
presentation: &dyn CommandPresentationContext,
|
||||
|
||||
@@ -73,14 +73,12 @@ path = "src/lib.rs"
|
||||
doctest = false
|
||||
|
||||
[dependencies]
|
||||
# Independent terminal kit, pinned and vendored for reproducible binary builds.
|
||||
codewhale-ratatui = { path = "../../vendor/codewhale-ratatui", version = "0.1.3" }
|
||||
codewhale-app-server-protocol = { path = "../app-server-protocol", version = "0.10.2" }
|
||||
codewhale-sanitize = { path = "../sanitize", version = "0.10.2" }
|
||||
codewhale-memory = { path = "../memory", version = "0.10.2" }
|
||||
anyhow.workspace = true
|
||||
codewhale-agent = { path = "../agent", version = "0.10.2" }
|
||||
codewhale-agent-contract = { path = "../agent-contract", version = "0.10.2" }
|
||||
codewhale-app-server = { path = "../app-server", version = "0.10.2" }
|
||||
codewhale-config = { path = "../config", version = "0.10.2" }
|
||||
codewhale-command-contract = { path = "../command-contract", version = "0.10.2" }
|
||||
codewhale-commands = { path = "../commands", version = "0.10.2" }
|
||||
@@ -190,6 +188,7 @@ wiremock = { version = "0.6", optional = true }
|
||||
codewhale-build-support = { path = "../build-support", version = "0.10.2" }
|
||||
|
||||
[dev-dependencies]
|
||||
codewhale-app-server-protocol = { path = "../app-server-protocol", version = "0.10.2", features = ["schema"] }
|
||||
codewhale-skills = { path = "../skills", version = "0.10.2", features = ["test-support"] }
|
||||
codewhale-dependencies = { path = "../dependencies", version = "0.10.2", features = ["test-support"] }
|
||||
codewhale-error-taxonomy = { path = "../error-taxonomy", version = "0.10.2", features = ["test-support"] }
|
||||
|
||||
+77
-61
@@ -822,30 +822,6 @@ fn request_query_secret_values(url: &reqwest::Url) -> Vec<String> {
|
||||
values
|
||||
}
|
||||
|
||||
/// Redirects are followed only within one host, over the same or a stronger
|
||||
/// scheme.
|
||||
///
|
||||
/// Provider keys ride in headers this stack may send under names reqwest does
|
||||
/// not strip on a cross-host hop (`x-api-key`, operator `http_headers`), so a
|
||||
/// compromised or open-redirecting endpoint could otherwise collect a key by
|
||||
/// answering `302`. A same-host hop to plaintext would expose the same key to
|
||||
/// anything on the path, so a downgrade stops too. Upgrades and same-scheme
|
||||
/// hops still work for the endpoints that use them.
|
||||
fn same_host_redirects() -> reqwest::redirect::Policy {
|
||||
reqwest::redirect::Policy::custom(|attempt| {
|
||||
let from = attempt.previous().last();
|
||||
let next = attempt.url();
|
||||
let same_host = from.is_some_and(|previous| previous.host_str() == next.host_str());
|
||||
let downgrade =
|
||||
from.is_some_and(|previous| previous.scheme() == "https" && next.scheme() == "http");
|
||||
if same_host && !downgrade && attempt.previous().len() < 10 {
|
||||
attempt.follow()
|
||||
} else {
|
||||
attempt.stop()
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
pub fn redact_model_bound_text(text: &str, exact_secret_values: &[String]) -> String {
|
||||
let mut redacted = text.to_string();
|
||||
for secret in exact_secret_values {
|
||||
@@ -1808,15 +1784,9 @@ impl CodewhaleClient {
|
||||
config,
|
||||
AuthHeaderSurface::Inference,
|
||||
)?;
|
||||
let http_client = if plugin_provider.is_some()
|
||||
|| api_key_source == "Claude sign-in"
|
||||
|| api_provider == ProviderKind::OpenaiCodex
|
||||
{
|
||||
http_client.redirect(reqwest::redirect::Policy::none())
|
||||
} else {
|
||||
http_client.redirect(same_host_redirects())
|
||||
}
|
||||
.build()?;
|
||||
let http_client = http_client
|
||||
.redirect(reqwest::redirect::Policy::none())
|
||||
.build()?;
|
||||
let models_http_client = Self::http_client_builder_with_auth_mode(
|
||||
&api_key,
|
||||
&http_headers,
|
||||
@@ -1830,9 +1800,8 @@ impl CodewhaleClient {
|
||||
)?
|
||||
.redirect(reqwest::redirect::Policy::none())
|
||||
.build()?;
|
||||
// Always keep an HTTP/1.1 twin for automatic stream-header fallback
|
||||
// when H2 stalls. When `force_http1` is pinned, both clients are
|
||||
// HTTP/1.1 and the fallback is a no-op retry path.
|
||||
// The HTTP/1.1 twin is eligible only after a proven pre-dispatch failure.
|
||||
// A stalled request may already have been accepted by the provider.
|
||||
let http1_client = Self::http_client_builder_with_auth_mode(
|
||||
&api_key,
|
||||
&http_headers,
|
||||
@@ -1844,12 +1813,9 @@ impl CodewhaleClient {
|
||||
config,
|
||||
AuthHeaderSurface::Inference,
|
||||
)?;
|
||||
let http1_client = if plugin_provider.is_some() || api_key_source == "Claude sign-in" {
|
||||
http1_client.redirect(reqwest::redirect::Policy::none())
|
||||
} else {
|
||||
http1_client
|
||||
}
|
||||
.build()?;
|
||||
let http1_client = http1_client
|
||||
.redirect(reqwest::redirect::Policy::none())
|
||||
.build()?;
|
||||
|
||||
let route_facts = crate::provider_lake::configured_catalog_offering_for_route(
|
||||
config,
|
||||
@@ -2420,7 +2386,8 @@ impl CodewhaleClient {
|
||||
.tcp_keepalive(config.tcp_keepalive())
|
||||
.http2_keep_alive_interval(config.http2_keep_alive_interval())
|
||||
.http2_keep_alive_timeout(config.http2_keep_alive_timeout())
|
||||
.min_tls_version(reqwest::tls::Version::TLS_1_2);
|
||||
.min_tls_version(reqwest::tls::Version::TLS_1_2)
|
||||
.retry(reqwest::retry::never());
|
||||
if api_provider == ProviderKind::OpenaiCodex {
|
||||
builder = builder.redirect(reqwest::redirect::Policy::none());
|
||||
}
|
||||
@@ -3891,6 +3858,7 @@ impl CodewhaleClient {
|
||||
NON_STREAMING_HTTP_TIMEOUT,
|
||||
build,
|
||||
&disclosure,
|
||||
crate::llm_client::RetryOperation::ReadOnly,
|
||||
)
|
||||
.await
|
||||
.map_err(ModelsFetchError::Interactive)?
|
||||
@@ -4489,8 +4457,13 @@ impl CodewhaleClient {
|
||||
where
|
||||
F: FnMut() -> reqwest::RequestBuilder,
|
||||
{
|
||||
self.send_with_retry_total_error_body(non_streaming_request_envelope(), build, disclosure)
|
||||
.await
|
||||
self.send_with_retry_total_error_body(
|
||||
non_streaming_request_envelope(),
|
||||
build,
|
||||
disclosure,
|
||||
crate::llm_client::RetryOperation::Inference,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// [`Self::send_with_retry_error_body`] with a caller-pinned per-attempt
|
||||
@@ -4503,6 +4476,7 @@ impl CodewhaleClient {
|
||||
total: Duration,
|
||||
mut build: F,
|
||||
disclosure: &ErrorBodyDisclosure,
|
||||
operation: crate::llm_client::RetryOperation,
|
||||
) -> Result<reqwest::Response>
|
||||
where
|
||||
F: FnMut() -> reqwest::RequestBuilder,
|
||||
@@ -4520,14 +4494,29 @@ impl CodewhaleClient {
|
||||
.authorize_plugin_request(request)
|
||||
.await
|
||||
.map_err(|error| LlmError::Other(error.to_string()))?;
|
||||
request
|
||||
.send()
|
||||
let (client, request) = request.build_split();
|
||||
let request = request
|
||||
.map_err(|error| LlmError::Other(error.without_url().to_string()))?;
|
||||
if operation == crate::llm_client::RetryOperation::ReadOnly
|
||||
&& request.method() != reqwest::Method::GET
|
||||
&& request.method() != reqwest::Method::HEAD
|
||||
{
|
||||
return Err(LlmError::Other(
|
||||
"Read-only retry requires GET or HEAD".to_string(),
|
||||
));
|
||||
}
|
||||
let pending = crate::llm_client::begin_outbound_transmission(request.method());
|
||||
let result = client
|
||||
.execute(request)
|
||||
.await
|
||||
.map_err(|error| LlmError::from_reqwest(&error.without_url()))
|
||||
.map_err(|error| LlmError::from_reqwest(&error.without_url()));
|
||||
crate::llm_client::settle_outbound_transmission(pending, &result);
|
||||
result
|
||||
}
|
||||
},
|
||||
Some(total),
|
||||
disclosure,
|
||||
operation,
|
||||
)
|
||||
.await
|
||||
}
|
||||
@@ -4546,9 +4535,18 @@ impl CodewhaleClient {
|
||||
let request = build(policy);
|
||||
async move {
|
||||
let request = self.authorize_plugin_request(request).await?;
|
||||
request.send().await.map_err(|error| {
|
||||
anyhow::Error::new(LlmError::from_reqwest(&error.without_url()))
|
||||
})
|
||||
let (client, request) = request.build_split();
|
||||
let request = request.map_err(|error| {
|
||||
anyhow::Error::new(LlmError::Other(error.without_url().to_string()))
|
||||
})?;
|
||||
let pending =
|
||||
crate::llm_client::begin_outbound_transmission(request.method());
|
||||
let result = client
|
||||
.execute(request)
|
||||
.await
|
||||
.map_err(|error| LlmError::from_reqwest(&error.without_url()));
|
||||
crate::llm_client::settle_outbound_transmission(pending, &result);
|
||||
result.map_err(anyhow::Error::new)
|
||||
}
|
||||
})
|
||||
.await
|
||||
@@ -4560,6 +4558,7 @@ impl CodewhaleClient {
|
||||
},
|
||||
None,
|
||||
&ErrorBodyDisclosure::Full,
|
||||
crate::llm_client::RetryOperation::Inference,
|
||||
)
|
||||
.await
|
||||
}
|
||||
@@ -4569,15 +4568,19 @@ impl CodewhaleClient {
|
||||
mut attempt: F,
|
||||
attempt_total: Option<Duration>,
|
||||
disclosure: &ErrorBodyDisclosure,
|
||||
operation: crate::llm_client::RetryOperation,
|
||||
) -> Result<reqwest::Response>
|
||||
where
|
||||
F: FnMut() -> Fut,
|
||||
Fut: std::future::Future<Output = std::result::Result<reqwest::Response, LlmError>>,
|
||||
{
|
||||
if self.isolated_request_state {
|
||||
return self.send_with_isolated_retry(attempt, disclosure).await;
|
||||
return self
|
||||
.send_with_isolated_retry(attempt, disclosure, operation)
|
||||
.await;
|
||||
}
|
||||
let retry_cfg: LlmRetryConfig = self.retry.clone().into();
|
||||
let mut retry_cfg: LlmRetryConfig = self.retry.clone().into();
|
||||
retry_cfg.operation = operation;
|
||||
let pause_scope = self.rate_limit_scope();
|
||||
let callback_scope = pause_scope.clone();
|
||||
// Non-streaming calls retain their whole-retry envelope. Streaming
|
||||
@@ -4609,12 +4612,14 @@ impl CodewhaleClient {
|
||||
let raw = bounded_error_text(response, ERROR_BODY_MAX_BYTES).await;
|
||||
let body_retry_after = retry_delay_from_error_body(&raw);
|
||||
let body = self.disclosed_http_error_body(disclosure, status.as_u16(), &raw);
|
||||
Err(self.http_error_with_route_context(
|
||||
let error = self.http_error_with_route_context(
|
||||
status.as_u16(),
|
||||
&body,
|
||||
header_retry_after,
|
||||
body_retry_after,
|
||||
))
|
||||
);
|
||||
crate::llm_client::settle_outbound_http_error(&error);
|
||||
Err(error)
|
||||
}
|
||||
},
|
||||
Some(Box::new(move |err, attempt, delay| {
|
||||
@@ -4703,12 +4708,14 @@ impl CodewhaleClient {
|
||||
&self,
|
||||
mut attempt: F,
|
||||
disclosure: &ErrorBodyDisclosure,
|
||||
operation: crate::llm_client::RetryOperation,
|
||||
) -> Result<reqwest::Response>
|
||||
where
|
||||
F: FnMut() -> Fut,
|
||||
Fut: std::future::Future<Output = std::result::Result<reqwest::Response, LlmError>>,
|
||||
{
|
||||
let retry_cfg: LlmRetryConfig = self.retry.clone().into();
|
||||
let mut retry_cfg: LlmRetryConfig = self.retry.clone().into();
|
||||
retry_cfg.operation = operation;
|
||||
let request_result = crate::llm_client::observe_request_retries(
|
||||
None,
|
||||
with_retry(
|
||||
@@ -4727,12 +4734,14 @@ impl CodewhaleClient {
|
||||
let body_retry_after = retry_delay_from_error_body(&raw);
|
||||
let body =
|
||||
self.disclosed_http_error_body(disclosure, status.as_u16(), &raw);
|
||||
Err(self.http_error_with_route_context(
|
||||
let error = self.http_error_with_route_context(
|
||||
status.as_u16(),
|
||||
&body,
|
||||
header_retry_after,
|
||||
body_retry_after,
|
||||
))
|
||||
);
|
||||
crate::llm_client::settle_outbound_http_error(&error);
|
||||
Err(error)
|
||||
}
|
||||
},
|
||||
Some(Box::new(|err, attempt, delay| {
|
||||
@@ -4810,7 +4819,14 @@ fn retry_reason_label_and_human(err: &LlmError) -> (&'static str, String) {
|
||||
("rate_limited", human)
|
||||
}
|
||||
LlmError::ServerError { status, .. } => ("server_error", format!("upstream {status}")),
|
||||
LlmError::NetworkError(_) => ("network_error", "network error".to_string()),
|
||||
LlmError::NetworkError(_) => (
|
||||
"network_error",
|
||||
"network error; request outcome unknown".to_string(),
|
||||
),
|
||||
LlmError::ConnectError { .. } => (
|
||||
"connect_error",
|
||||
"connection failed before dispatch".to_string(),
|
||||
),
|
||||
LlmError::Timeout(_) => ("timeout", "timeout".to_string()),
|
||||
_ => ("other", "other".to_string()),
|
||||
}
|
||||
@@ -6683,8 +6699,8 @@ pub mod system_one;
|
||||
|
||||
/// Longest a request may take to open its stream and deliver the first body
|
||||
/// byte before the client itself times out (#6184): the first-byte bound plus
|
||||
/// two header waits, because a failed or stalled open on the dual client
|
||||
/// retries once on the HTTP/1.1 twin under its own header wait. HTTP retries
|
||||
/// two header waits, because a proven pre-dispatch failure on the dual client
|
||||
/// may use the HTTP/1.1 twin under its own header wait. HTTP retries
|
||||
/// inside one open attempt run within that attempt's header wait. The engine
|
||||
/// heartbeat uses this as its awaiting-model bound, so it must cover the
|
||||
/// fallback or an in-progress recovery reads as a stall (#6711).
|
||||
|
||||
@@ -287,12 +287,17 @@ impl ProviderNativeSearchClient {
|
||||
pub(super) async fn get_json(&self, url: &str) -> Result<Value> {
|
||||
let response = self
|
||||
.inner
|
||||
.send_with_retry(|| {
|
||||
self.inner
|
||||
.http_client
|
||||
.get(url)
|
||||
.header("Accept", "application/json")
|
||||
})
|
||||
.send_with_retry_total_error_body(
|
||||
super::non_streaming_request_envelope(),
|
||||
|| {
|
||||
self.inner
|
||||
.http_client
|
||||
.get(url)
|
||||
.header("Accept", "application/json")
|
||||
},
|
||||
&super::ErrorBodyDisclosure::Full,
|
||||
crate::llm_client::RetryOperation::ReadOnly,
|
||||
)
|
||||
.await
|
||||
.context("provider-native web search request failed")?;
|
||||
response
|
||||
|
||||
@@ -181,77 +181,61 @@ pub fn should_retry_with_h1(policy: StreamHttpPolicy, err_text: &str) -> bool {
|
||||
|| lower.contains("frame size")
|
||||
}
|
||||
|
||||
/// Whether an error raised before response headers should retry through the
|
||||
/// HTTP/1.1 twin. Prefer typed transport errors, then retain the narrow text
|
||||
/// classifier for lower-level H2 errors that reqwest exposes only as prose.
|
||||
#[must_use]
|
||||
fn should_retry_error_with_h1(policy: StreamHttpPolicy, err: &anyhow::Error) -> bool {
|
||||
if policy != StreamHttpPolicy::DualWithH1Fallback {
|
||||
return false;
|
||||
}
|
||||
|
||||
typed_open_transport_failure(err)
|
||||
.unwrap_or_else(|| should_retry_with_h1(policy, &format!("{err:#}")))
|
||||
policy == StreamHttpPolicy::DualWithH1Fallback && is_stream_open_transport_failure(err)
|
||||
}
|
||||
|
||||
/// Classify `err` by its types alone, searching the whole context chain: an
|
||||
/// adapter's `.context("... request failed")` must not hide the transport
|
||||
/// cause (#6711). `Some(true)` for a typed `LlmError::NetworkError`/`Timeout`
|
||||
/// or a reqwest connect/timeout/request error that carries no HTTP status;
|
||||
/// `Some(false)` for any other typed `LlmError` or reqwest error; `None` when
|
||||
/// the chain holds neither type.
|
||||
fn typed_open_transport_failure(err: &anyhow::Error) -> Option<bool> {
|
||||
err.chain().find_map(|cause| {
|
||||
if let Some(llm_error) = cause.downcast_ref::<LlmError>() {
|
||||
return Some(matches!(
|
||||
llm_error,
|
||||
LlmError::NetworkError(_) | LlmError::Timeout(_)
|
||||
));
|
||||
}
|
||||
cause.downcast_ref::<reqwest::Error>().map(|reqwest_error| {
|
||||
reqwest_error.status().is_none()
|
||||
&& (reqwest_error.is_connect()
|
||||
|| reqwest_error.is_timeout()
|
||||
|| reqwest_error.is_request())
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
/// Whether a failed stream open is a real transport failure: the request
|
||||
/// never received response headers (#6699). Decided by type only: an untyped
|
||||
/// error, such as an adapter's `HTTP 500 ...` rejection whose provider body
|
||||
/// happens to mention a connection or timeout, is a provider answer and is
|
||||
/// never classified as a transport failure here.
|
||||
#[must_use]
|
||||
pub fn is_stream_open_transport_failure(err: &anyhow::Error) -> bool {
|
||||
typed_open_transport_failure(err).unwrap_or(false)
|
||||
err.chain()
|
||||
.find_map(|cause| {
|
||||
if let Some(error) = cause.downcast_ref::<LlmError>() {
|
||||
return Some(matches!(
|
||||
error,
|
||||
LlmError::ConnectError {
|
||||
transient: true,
|
||||
..
|
||||
}
|
||||
));
|
||||
}
|
||||
cause.downcast_ref::<reqwest::Error>().map(|error| {
|
||||
matches!(
|
||||
LlmError::from_reqwest(error),
|
||||
LlmError::ConnectError {
|
||||
transient: true,
|
||||
..
|
||||
}
|
||||
)
|
||||
})
|
||||
})
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
/// Preserve provider-semantic failures returned by the H1 attempt. Only a
|
||||
/// transport failure should be normalized into the shared retryable network
|
||||
/// error; otherwise an auth or invalid-request failure could be retried as if
|
||||
/// switching protocols had failed.
|
||||
fn h1_fallback_error(err: anyhow::Error) -> anyhow::Error {
|
||||
if let Some(llm_error) = err.downcast_ref::<LlmError>()
|
||||
&& !matches!(llm_error, LlmError::NetworkError(_) | LlmError::Timeout(_))
|
||||
{
|
||||
if err.downcast_ref::<LlmError>().is_some() {
|
||||
return err;
|
||||
}
|
||||
|
||||
let detail = format!("{err:#}");
|
||||
if let Some(unreachable) = connect_failure_summary(&err) {
|
||||
return anyhow::Error::new(LlmError::NetworkError(format!("{unreachable}: {detail}")));
|
||||
if let Some(error) = err
|
||||
.chain()
|
||||
.find_map(|cause| cause.downcast_ref::<reqwest::Error>())
|
||||
{
|
||||
let mut classified = LlmError::from_reqwest(error);
|
||||
if let Some(unreachable) = connect_failure_summary(&err)
|
||||
&& let LlmError::ConnectError { message, .. } = &mut classified
|
||||
{
|
||||
*message = format!("{unreachable}: {message}");
|
||||
}
|
||||
if has_tls_cause(&err)
|
||||
&& let LlmError::ConnectError { message, .. } = &mut classified
|
||||
{
|
||||
message.push_str(" On Windows or proxy networks, try CODEWHALE_FORCE_HTTP1=1 after checking the TLS configuration.");
|
||||
}
|
||||
return anyhow::Error::new(classified);
|
||||
}
|
||||
let mut message = format!("SSE stream request failed after HTTP/1.1 fallback: {detail}.");
|
||||
// The HTTP/1.1 hint only helps when the protocol or TLS layer failed; it
|
||||
// is noise for a refused connection or an unknown host.
|
||||
if has_tls_cause(&err) || is_protocol_or_tls_failure(&detail) {
|
||||
message.push_str(
|
||||
" `codewhale doctor` can still pass when non-streaming requests work; \
|
||||
on Windows or proxy networks, try `CODEWHALE_FORCE_HTTP1=1` and rerun `codewhale`.",
|
||||
);
|
||||
}
|
||||
anyhow::Error::new(LlmError::NetworkError(message))
|
||||
anyhow::Error::new(LlmError::NetworkError(format!(
|
||||
"SSE stream request failed: {err:#}"
|
||||
)))
|
||||
}
|
||||
|
||||
/// `Cannot reach host:port (<why>)` for a failure to open the connection.
|
||||
@@ -310,17 +294,7 @@ fn connect_failure_summary(err: &anyhow::Error) -> Option<String> {
|
||||
}
|
||||
|
||||
fn has_tls_cause(err: &anyhow::Error) -> bool {
|
||||
err.chain().any(|mut cause| {
|
||||
// hyper-rustls wraps tokio-rustls's IO error in another IO error;
|
||||
// Error::source skips those inners, so inspect get_ref explicitly.
|
||||
while let Some(inner) = cause
|
||||
.downcast_ref::<std::io::Error>()
|
||||
.and_then(std::io::Error::get_ref)
|
||||
{
|
||||
cause = inner;
|
||||
}
|
||||
cause.is::<rustls::Error>()
|
||||
})
|
||||
crate::llm_client::has_tls_cause(err.as_ref())
|
||||
}
|
||||
|
||||
fn is_protocol_or_tls_failure(detail: &str) -> bool {
|
||||
@@ -338,18 +312,6 @@ const COLD_MODEL_HINT: &str = "If this model has not been used recently, the pro
|
||||
still be loading it: wait a minute and send the message again, or allow a longer wait with \
|
||||
`codewhale config set stream.open_timeout_secs 180` (up to 300).";
|
||||
|
||||
/// Open an SSE response through the shared transport policy.
|
||||
///
|
||||
/// `attempt` builds and sends one wire-specific request on the client
|
||||
/// selected for the given policy (via [`client_for_policy`]); everything
|
||||
/// transport-shared lives here:
|
||||
///
|
||||
/// - the response-header wait is bounded by `open_req.open_timeout`;
|
||||
/// - a classified transport failure or header stall on the dual client
|
||||
/// retries exactly once on the HTTP/1.1 twin;
|
||||
/// - a failure on an already H1-pinned request never retries;
|
||||
/// - once response headers have been received the seam never retries —
|
||||
/// body/stream errors belong to the adapter's decode loop.
|
||||
pub async fn open_sse_response<F, Fut>(
|
||||
open_req: &StreamOpenRequest,
|
||||
attempt: F,
|
||||
@@ -358,55 +320,28 @@ where
|
||||
F: Fn(StreamHttpPolicy) -> Fut,
|
||||
Fut: Future<Output = Result<reqwest::Response>>,
|
||||
{
|
||||
let fallback_reason = match tokio::time::timeout(
|
||||
open_req.open_timeout,
|
||||
attempt(open_req.policy),
|
||||
)
|
||||
.await
|
||||
{
|
||||
match tokio::time::timeout(open_req.open_timeout, attempt(open_req.policy)).await {
|
||||
Ok(Ok(response)) => return Ok(response),
|
||||
Ok(Err(err)) => {
|
||||
if !should_retry_error_with_h1(open_req.policy, &err) {
|
||||
return Err(err);
|
||||
return Err(h1_fallback_error(err));
|
||||
}
|
||||
"transport error before response headers"
|
||||
}
|
||||
Err(_elapsed) => {
|
||||
if open_req.policy == StreamHttpPolicy::Http1Only {
|
||||
return Err(anyhow::Error::new(LlmError::NetworkError(format!(
|
||||
"SSE stream request did not receive response headers after {}s. \
|
||||
`codewhale doctor` can still pass when non-streaming requests work; \
|
||||
on Windows or proxy networks, try `CODEWHALE_FORCE_HTTP1=1` and rerun `codewhale`. \
|
||||
{COLD_MODEL_HINT}",
|
||||
open_req.open_timeout.as_secs()
|
||||
))));
|
||||
}
|
||||
"response-header timeout"
|
||||
Err(_) => {
|
||||
return Err(anyhow::Error::new(LlmError::NetworkError(format!(
|
||||
"SSE stream request did not receive response headers after {}s; request outcome unknown. {COLD_MODEL_HINT}",
|
||||
open_req.open_timeout.as_secs()
|
||||
))));
|
||||
}
|
||||
};
|
||||
|
||||
// No response body exists yet, so switching protocols and replaying the
|
||||
// request cannot corrupt stream state. It can still bill twice if the
|
||||
// provider accepted the first request (see the ambiguous-replay limit on
|
||||
// `llm_client::with_retry`). The policy guard above keeps this to exactly
|
||||
// one retry.
|
||||
}
|
||||
let h1_req = open_req.clone().with_h1_only();
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"SSE stream {fallback_reason}; retrying once with HTTP/1.1"
|
||||
));
|
||||
crate::llm_client::observe_transport_attempt();
|
||||
match tokio::time::timeout(h1_req.open_timeout, attempt(h1_req.policy)).await {
|
||||
Ok(Ok(response)) => Ok(response),
|
||||
Ok(Err(err)) => Err(h1_fallback_error(err)),
|
||||
// Typed, not a bare string: a header stall is a transport
|
||||
// failure, and `LlmError::NetworkError` is what the shared
|
||||
// retry layer recognizes as retryable. As an untyped anyhow
|
||||
// error this killed the whole turn outright.
|
||||
Err(_elapsed) => Err(anyhow::Error::new(LlmError::NetworkError(format!(
|
||||
"SSE stream request did not receive response headers after {}s \
|
||||
(HTTP/2 and HTTP/1.1). `codewhale doctor` can still pass when \
|
||||
non-streaming requests work; try `CODEWHALE_FORCE_HTTP1=1` and \
|
||||
rerun `codewhale`. {COLD_MODEL_HINT}",
|
||||
open_req.open_timeout.as_secs()
|
||||
Err(_) => Err(anyhow::Error::new(LlmError::NetworkError(format!(
|
||||
"SSE stream request did not receive response headers after {}s on HTTP/1.1; request outcome unknown. {COLD_MODEL_HINT}",
|
||||
h1_req.open_timeout.as_secs()
|
||||
)))),
|
||||
}
|
||||
}
|
||||
@@ -490,9 +425,11 @@ force_http1 = {pinned}
|
||||
if attempt == 0 {
|
||||
// Only a genuinely dual-protocol request may
|
||||
// retry this failure on its HTTP/1.1 twin.
|
||||
return Err(anyhow::Error::new(LlmError::NetworkError(
|
||||
"connection reset before response headers".into(),
|
||||
)));
|
||||
return Err(anyhow::Error::new(LlmError::ConnectError {
|
||||
message: "connection refused before dispatch".into(),
|
||||
transient: true,
|
||||
timed_out: false,
|
||||
}));
|
||||
}
|
||||
assert_eq!(policy, StreamHttpPolicy::Http1Only);
|
||||
Ok(reqwest::Response::from(
|
||||
@@ -538,7 +475,7 @@ force_http1 = {pinned}
|
||||
#[tokio::test]
|
||||
async fn header_stall_on_dual_policy_retries_exactly_once_on_h1() {
|
||||
let attempts = Arc::new(AtomicUsize::new(0));
|
||||
let response = open_sse_response(
|
||||
let error = open_sse_response(
|
||||
&open_req(
|
||||
StreamHttpPolicy::DualWithH1Fallback,
|
||||
Duration::from_millis(150),
|
||||
@@ -563,12 +500,12 @@ force_http1 = {pinned}
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("H1 fallback retry succeeds");
|
||||
assert_eq!(response.status(), 200);
|
||||
.expect_err("header stall must retain an unknown outcome without replay");
|
||||
assert!(error.to_string().contains("outcome unknown"));
|
||||
assert_eq!(
|
||||
attempts.load(Ordering::SeqCst),
|
||||
2,
|
||||
"exactly one fallback retry"
|
||||
1,
|
||||
"an unknown header stall must not fall back"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -587,9 +524,11 @@ force_http1 = {pinned}
|
||||
let attempt = attempts.fetch_add(1, Ordering::SeqCst);
|
||||
if attempt == 0 {
|
||||
assert_eq!(policy, StreamHttpPolicy::DualWithH1Fallback);
|
||||
return Err(anyhow::Error::new(LlmError::NetworkError(
|
||||
"connection reset before response headers".to_string(),
|
||||
))
|
||||
return Err(anyhow::Error::new(LlmError::ConnectError {
|
||||
message: "connection refused before dispatch".into(),
|
||||
transient: true,
|
||||
timed_out: false,
|
||||
})
|
||||
.context("Chat API request failed"));
|
||||
}
|
||||
assert_eq!(policy, StreamHttpPolicy::Http1Only);
|
||||
@@ -626,19 +565,14 @@ force_http1 = {pinned}
|
||||
assert_eq!(attempts.load(Ordering::SeqCst), 1, "no retry when pinned");
|
||||
let text = err.to_string();
|
||||
assert!(text.contains("did not receive response headers"), "{text}");
|
||||
// The whole point of the typed error: a header stall must reach the
|
||||
// shared retry layer as retryable. As an untyped anyhow error it
|
||||
// killed the turn outright, so a long root run lost all its work while
|
||||
// a sub-agent — which text-matches the same message in its own
|
||||
// classifier — would have retried and continued.
|
||||
let classified = err
|
||||
.downcast_ref::<crate::llm_client::LlmError>()
|
||||
.expect("header stall must be a typed LlmError");
|
||||
assert!(
|
||||
classified.is_retryable(),
|
||||
"header stall must be retryable: {classified:?}"
|
||||
!classified.is_retryable(),
|
||||
"header stall must not be replayable: {classified:?}"
|
||||
);
|
||||
assert!(text.contains("CODEWHALE_FORCE_HTTP1=1"), "{text}");
|
||||
assert!(text.contains("outcome unknown"), "{text}");
|
||||
// #6889: a header wait that ran out may be a model still loading.
|
||||
assert!(text.contains("may still be loading it"), "{text}");
|
||||
assert!(
|
||||
@@ -742,10 +676,10 @@ force_http1 = {pinned}
|
||||
anyhow::Error::new(LlmError::NetworkError("http2 protocol error".to_string()))
|
||||
.context("sending stream request"),
|
||||
);
|
||||
let text = protocol.to_string();
|
||||
let text = format!("{protocol:#}");
|
||||
assert!(text.contains("sending stream request: "), "{text}");
|
||||
assert!(text.contains("http2 protocol error"), "{text}");
|
||||
assert!(text.contains("CODEWHALE_FORCE_HTTP1=1"), "{text}");
|
||||
assert!(!protocol.downcast_ref::<LlmError>().unwrap().is_retryable());
|
||||
|
||||
let other = h1_fallback_error(anyhow::Error::new(LlmError::NetworkError(
|
||||
"unexpected end of body".to_string(),
|
||||
@@ -767,9 +701,11 @@ force_http1 = {pinned}
|
||||
let attempt = attempts.fetch_add(1, Ordering::SeqCst);
|
||||
if attempt == 0 {
|
||||
assert_eq!(policy, StreamHttpPolicy::DualWithH1Fallback);
|
||||
return Err(anyhow::Error::new(LlmError::NetworkError(
|
||||
"connection reset before response headers".to_string(),
|
||||
)));
|
||||
return Err(anyhow::Error::new(LlmError::ConnectError {
|
||||
message: "connection refused before dispatch".into(),
|
||||
transient: true,
|
||||
timed_out: false,
|
||||
}));
|
||||
}
|
||||
assert_eq!(policy, StreamHttpPolicy::Http1Only);
|
||||
Err(anyhow::Error::new(LlmError::InvalidRequest {
|
||||
@@ -833,9 +769,13 @@ force_http1 = {pinned}
|
||||
)
|
||||
.await
|
||||
.expect_err("double stall fails");
|
||||
assert_eq!(attempts.load(Ordering::SeqCst), 2, "one fallback, no more");
|
||||
assert_eq!(
|
||||
attempts.load(Ordering::SeqCst),
|
||||
1,
|
||||
"no replay after unknown stall"
|
||||
);
|
||||
let text = err.to_string();
|
||||
assert!(text.contains("HTTP/2 and HTTP/1.1"), "{text}");
|
||||
assert!(!text.contains("HTTP/2 and HTTP/1.1"), "{text}");
|
||||
// #6889: a header wait that ran out may be a model still loading.
|
||||
assert!(text.contains("may still be loading it"), "{text}");
|
||||
assert!(text.contains("stream.open_timeout_secs"), "{text}");
|
||||
|
||||
@@ -528,9 +528,10 @@ pub fn router_failure_from_error(error: &anyhow::Error) -> AutoRouterFailure {
|
||||
.map_or(AutoRouterFailure::Transport, |status| {
|
||||
AutoRouterFailure::Http { status }
|
||||
}),
|
||||
LlmError::NetworkError(_) | LlmError::Timeout(_) | LlmError::ParseError(_) => {
|
||||
AutoRouterFailure::Transport
|
||||
}
|
||||
LlmError::NetworkError(_)
|
||||
| LlmError::ConnectError { .. }
|
||||
| LlmError::Timeout(_)
|
||||
| LlmError::ParseError(_) => AutoRouterFailure::Transport,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+1577
-1581
File diff suppressed because it is too large.
Load diff
@@ -1286,17 +1286,7 @@ fn classify_compaction_failure(e: &anyhow::Error) -> CompactionFailureKind {
|
||||
};
|
||||
}
|
||||
|
||||
let text = e.to_string();
|
||||
if is_context_window_error_message(&text) {
|
||||
return CompactionFailureKind::ContextOverflow;
|
||||
}
|
||||
let category = codewhale_error_taxonomy::classify_error_message(&text);
|
||||
match category {
|
||||
codewhale_error_taxonomy::ErrorCategory::Network
|
||||
| codewhale_error_taxonomy::ErrorCategory::RateLimit
|
||||
| codewhale_error_taxonomy::ErrorCategory::Timeout => CompactionFailureKind::Transient,
|
||||
_ => CompactionFailureKind::Deterministic,
|
||||
}
|
||||
CompactionFailureKind::Deterministic
|
||||
}
|
||||
|
||||
fn llm_error_in_chain(error: &anyhow::Error) -> Option<&crate::llm_client::LlmError> {
|
||||
@@ -1363,29 +1353,16 @@ pub fn report_compaction_failure(
|
||||
format!("{prefix}: {detail}")
|
||||
}
|
||||
|
||||
/// Check if an error is transient and worth retrying. Categories that map to
|
||||
/// transient retry: Network, RateLimit, Timeout. Context overflow is *not*
|
||||
/// transient — it needs a larger route window, not the same payload.
|
||||
/// Only typed pre-dispatch connection failures and authoritative ordinary429
|
||||
/// refusals permit replay. Context overflow needs a different request.
|
||||
fn is_transient_error(e: &anyhow::Error) -> bool {
|
||||
classify_compaction_failure(e).is_transient()
|
||||
}
|
||||
|
||||
fn is_context_window_error_message(text: &str) -> bool {
|
||||
let lower = text.to_lowercase();
|
||||
lower.contains("too long for this model")
|
||||
|| lower.contains("prompt is too long")
|
||||
|| lower.contains("maximum prompt length")
|
||||
|| lower.contains("maximum context length")
|
||||
|| lower.contains("context_length_exceeded")
|
||||
|| lower.contains("context window")
|
||||
|| (lower.contains("context")
|
||||
&& (lower.contains("token") || lower.contains("too long") || lower.contains("maximum")))
|
||||
}
|
||||
|
||||
/// Compact messages with retry and backoff for transient errors.
|
||||
///
|
||||
/// This function wraps `compact_messages` with retry logic to handle
|
||||
/// transient network errors and rate limits. It uses exponential backoff
|
||||
/// proven pre-dispatch connection failures and ordinary429 refusals. It uses exponential backoff
|
||||
/// with delays of 1s, 2s, 4s between retries.
|
||||
///
|
||||
/// # Safety
|
||||
@@ -2238,29 +2215,11 @@ enum RequestSizeLadder {
|
||||
ImagesReplaced,
|
||||
}
|
||||
|
||||
/// Whether the provider refused the request body for size (HTTP 413 and its
|
||||
/// common wordings). A smaller payload can succeed where this one did not,
|
||||
/// which is exactly what the request-size ladder trades on.
|
||||
///
|
||||
/// Walks the whole error chain: the rejection may be stated by a fronting
|
||||
/// gateway (an HTML page from openresty reading "413 Request Entity Too
|
||||
/// Large") and then wrapped again by the client's own error text, so the
|
||||
/// top-level message alone is not reliable.
|
||||
fn is_request_too_large_error(e: &anyhow::Error) -> bool {
|
||||
matches!(
|
||||
e.downcast_ref::<crate::llm_client::LlmError>(),
|
||||
llm_error_in_chain(e),
|
||||
Some(crate::llm_client::LlmError::InvalidRequest { status: 413, .. })
|
||||
) || e.chain().any(|cause| {
|
||||
let lower = cause.to_string().to_lowercase();
|
||||
// The client always renders the status code into the message
|
||||
// ("HTTP 413"), so that token is the primary signal; the phrase list
|
||||
// below catches gateways that state the condition without it.
|
||||
lower.contains("http 413")
|
||||
|| lower.contains("payload too large")
|
||||
|| lower.contains("request entity too large")
|
||||
|| lower.contains("request body too large")
|
||||
|| lower.contains("length limit exceeded")
|
||||
})
|
||||
)
|
||||
}
|
||||
|
||||
/// Forward one user-visible progress sentence, when the host supplied a sink.
|
||||
@@ -2295,16 +2254,10 @@ fn replace_inline_images_for_retry(
|
||||
}
|
||||
|
||||
fn is_context_window_error(e: &anyhow::Error) -> bool {
|
||||
let text = e.to_string();
|
||||
if codewhale_error_taxonomy::classify_error_message(&text)
|
||||
!= codewhale_error_taxonomy::ErrorCategory::InvalidInput
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
let lower = text.to_lowercase();
|
||||
is_context_window_error_message(&text)
|
||||
|| (lower.contains("requested") && lower.contains("tokens") && lower.contains("maximum"))
|
||||
matches!(
|
||||
llm_error_in_chain(e),
|
||||
Some(crate::llm_client::LlmError::ContextLengthError(_))
|
||||
)
|
||||
}
|
||||
|
||||
/// Collect text from a user message without treating tool-result payloads
|
||||
@@ -2681,9 +2634,12 @@ mod tests {
|
||||
"request exceeds context window",
|
||||
] {
|
||||
assert!(
|
||||
is_context_window_error(&anyhow::anyhow!(msg)),
|
||||
!is_context_window_error(&anyhow::anyhow!(msg)),
|
||||
"expected context-window detection for `{msg}`",
|
||||
);
|
||||
assert!(is_context_window_error(&anyhow::Error::new(
|
||||
crate::llm_client::LlmError::ContextLengthError(msg.to_string())
|
||||
)));
|
||||
}
|
||||
|
||||
assert!(!is_context_window_error(&anyhow::anyhow!(
|
||||
@@ -2711,8 +2667,10 @@ mod tests {
|
||||
];
|
||||
let before = original.clone();
|
||||
let client = ScriptedSummaryClient::with_outcomes(vec![
|
||||
Err(anyhow::anyhow!(
|
||||
"prompt is too long for this model's context window"
|
||||
Err(anyhow::Error::new(
|
||||
crate::llm_client::LlmError::ContextLengthError(
|
||||
"prompt is too long for this model's context window".to_string(),
|
||||
),
|
||||
)),
|
||||
Ok(summary_content()),
|
||||
]);
|
||||
@@ -2943,10 +2901,11 @@ mod tests {
|
||||
/// the API's own body reader. Reported on 2026-09-26; the ladder must
|
||||
/// treat it as the same byte-side rejection.
|
||||
fn request_body_413_html_gateway() -> anyhow::Error {
|
||||
anyhow::anyhow!(
|
||||
"LLM error: HTTP 413: DeepSeek API returned an HTML error page (HTTP 413): \
|
||||
413 Request Entity Too Large 413 Request Entity Too Large openresty"
|
||||
crate::llm_client::LlmError::from_http_response(
|
||||
413,
|
||||
"413 Request Entity Too Large openresty",
|
||||
)
|
||||
.into()
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
@@ -3844,7 +3803,11 @@ mod tests {
|
||||
text: "...".to_string(),
|
||||
cache_control: None,
|
||||
}]),
|
||||
Err(anyhow::anyhow!("request timed out")),
|
||||
Err(crate::llm_client::LlmError::RateLimited {
|
||||
message: "rate limited".into(),
|
||||
retry_after: Some(Duration::ZERO),
|
||||
}
|
||||
.into()),
|
||||
Ok(vec![ContentBlock::Text {
|
||||
text: FIXED_SUMMARY.to_string(),
|
||||
cache_control: None,
|
||||
|
||||
@@ -164,7 +164,7 @@ fn untyped_transient_and_deterministic_classification_remains_compatible() {
|
||||
"503 Service Unavailable",
|
||||
"network error: connection refused",
|
||||
] {
|
||||
assert!(is_transient_error(&anyhow::anyhow!(message)), "{message}");
|
||||
assert!(!is_transient_error(&anyhow::anyhow!(message)), "{message}");
|
||||
}
|
||||
for message in [
|
||||
"401 Unauthorized: Invalid API key",
|
||||
@@ -177,7 +177,7 @@ fn untyped_transient_and_deterministic_classification_remains_compatible() {
|
||||
classify_compaction_failure(&anyhow::anyhow!(
|
||||
"prompt is too long for this model's context window"
|
||||
)),
|
||||
CompactionFailureKind::ContextOverflow
|
||||
CompactionFailureKind::Deterministic
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -15,9 +15,9 @@ use std::sync::{Arc, Mutex as StdMutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use anyhow::Result;
|
||||
use codewhale_app_server_protocol::runtime::DynamicToolSpec;
|
||||
use codewhale_config::route::CapabilityState;
|
||||
use codewhale_execpolicy::{AskForApproval, ExecPolicyContext};
|
||||
use codewhale_protocol::runtime::DynamicToolSpec;
|
||||
use futures_util::StreamExt;
|
||||
use futures_util::stream::FuturesUnordered;
|
||||
use serde_json::{Value, json};
|
||||
@@ -198,8 +198,7 @@ impl StructuredState {
|
||||
working_set.summary_block(&workspace, &crate::tools::file::working_set_read_allowed);
|
||||
|
||||
let subagent_snapshots = if let Some(handle) = subagents {
|
||||
let mut guard = handle.write().await;
|
||||
guard.cleanup_for_session(active_session_id, Duration::from_secs(60 * 60));
|
||||
let guard = handle.read().await;
|
||||
guard
|
||||
.list_for_session(active_session_id)
|
||||
.into_iter()
|
||||
@@ -316,6 +315,7 @@ pub struct EngineConfig {
|
||||
pub notes_path: PathBuf,
|
||||
/// Path to the MCP configuration file.
|
||||
pub mcp_config_path: PathBuf,
|
||||
pub mcp_auth_context: crate::mcp::McpAuthContext,
|
||||
/// OAuth callback overrides (`mcp_oauth_callback_port` / `_url`) so the
|
||||
/// self-serve MCP login tool honors a pre-registered redirect URI the
|
||||
/// same way `/mcp login` does.
|
||||
@@ -514,11 +514,9 @@ pub struct EngineConfig {
|
||||
/// from `[tui].stream_max_duration_secs`. Pre-R1 this was the hard-coded
|
||||
/// `DEFAULT_STREAM_MAX_DURATION_SECS`.
|
||||
pub stream_max_duration: Duration,
|
||||
/// Stream-level retry budgets (#6700): whole-request resumes (also spent
|
||||
/// by stream-open failures, #6699), in-stream transparent retries, and
|
||||
/// the per-stream error streak. Resolved from `[tui].stream_max_resumes`,
|
||||
/// `[tui].stream_max_transparent_retries` and `[tui].stream_max_errors`;
|
||||
/// the defaults are the historical compiled-in values.
|
||||
/// Proven pre-dispatch open retries from `[tui].stream_max_resumes`.
|
||||
/// The parsed max_transparent_retries/max_errors values are inactive
|
||||
/// compatibility fields and cannot authorize an inference POST replay.
|
||||
pub stream_retry_limits: turn_budget::StreamRetryLimits,
|
||||
/// Bounded wait for SSE response headers (#6700). Resolved from
|
||||
/// `[tui].stream_open_timeout_secs`, then
|
||||
@@ -598,6 +596,7 @@ impl Default for EngineConfig {
|
||||
trust_mode: false,
|
||||
notes_path: PathBuf::from("notes.txt"),
|
||||
mcp_config_path: PathBuf::from("mcp.json"),
|
||||
mcp_auth_context: Default::default(),
|
||||
mcp_oauth_callback_port: None,
|
||||
mcp_oauth_callback_url: None,
|
||||
skills_dir: codewhale_skills::default_skills_dir(),
|
||||
@@ -727,6 +726,7 @@ impl CancelReason {
|
||||
/// Handle to communicate with the engine
|
||||
#[derive(Clone)]
|
||||
pub struct EngineHandle {
|
||||
mcp_auth_context: Arc<dyn codewhale_runtime::credentials::AuthContext>,
|
||||
goal_state: SharedGoalState,
|
||||
pub sandbox_enforcement: codewhale_sandbox::policy::SandboxEnforcement,
|
||||
/// Send operations to the engine
|
||||
@@ -827,6 +827,10 @@ impl CompactionCancellationState {
|
||||
}
|
||||
|
||||
impl EngineHandle {
|
||||
pub fn mcp_auth_context(&self) -> Arc<dyn codewhale_runtime::credentials::AuthContext> {
|
||||
Arc::clone(&self.mcp_auth_context)
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn subagent_manager(&self) -> SharedSubAgentManager {
|
||||
self.subagent_manager.clone()
|
||||
@@ -2642,6 +2646,7 @@ impl Engine {
|
||||
child_host,
|
||||
};
|
||||
let handle = EngineHandle {
|
||||
mcp_auth_context: Arc::clone(&engine.config.mcp_auth_context.0),
|
||||
goal_state: engine.config.goal_state.clone(),
|
||||
sandbox_enforcement,
|
||||
tx_op,
|
||||
@@ -2740,7 +2745,13 @@ impl Engine {
|
||||
self.apply_runtime_mode_policy(&authority);
|
||||
self.discard_kernels_if_narrowed(&prior).await;
|
||||
|
||||
let previous_source = self.session.snapshot_source.clone();
|
||||
self.session.snapshot_source = Some(crate::core::ops::SessionSnapshotSource::turn(
|
||||
turn_id.clone(),
|
||||
None,
|
||||
));
|
||||
start_permit.send(Event::TurnStarted {
|
||||
previous_source,
|
||||
turn_id: turn_id.clone(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -3298,6 +3309,7 @@ impl Engine {
|
||||
system_prompt: Option<codewhale_models::SystemPrompt>,
|
||||
system_prompt_override: bool,
|
||||
) {
|
||||
self.session.snapshot_source = None;
|
||||
self.session.tool_activation_cache.clear();
|
||||
let compaction_checkpoint =
|
||||
compaction_checkpoint_for_restore(&messages, system_prompt.clone());
|
||||
@@ -3345,6 +3357,7 @@ impl Engine {
|
||||
let total_tokens =
|
||||
self.session.total_usage.input_tokens + self.session.total_usage.output_tokens;
|
||||
SessionSnapshot {
|
||||
source: self.session.snapshot_source.clone(),
|
||||
session_id: self.session.id.clone(),
|
||||
messages: self.session.messages.to_vec(),
|
||||
total_tokens,
|
||||
@@ -4043,12 +4056,28 @@ impl Engine {
|
||||
)
|
||||
};
|
||||
let event = if due {
|
||||
let mut manager = self.subagent_manager.write().await;
|
||||
manager.cleanup_for_session(
|
||||
&active_session_id,
|
||||
Duration::from_secs(60 * 60),
|
||||
);
|
||||
agent_list_event(&manager, &active_session_id)
|
||||
let manager = self.subagent_manager.clone();
|
||||
let session_id = active_session_id.clone();
|
||||
match codewhale_runtime::runtime_control::owner_work(move || {
|
||||
let mut manager = manager.blocking_write();
|
||||
manager.cleanup_for_session(
|
||||
&session_id,
|
||||
Duration::from_secs(60 * 60),
|
||||
)?;
|
||||
Ok(agent_list_event(&manager, &session_id))
|
||||
})
|
||||
.await
|
||||
{
|
||||
Ok(event) => event,
|
||||
Err(error) => {
|
||||
let _ = self
|
||||
.send_event(Event::error(ErrorEnvelope::transient(
|
||||
format!("Failed to clean up sub-agents: {error}"),
|
||||
)))
|
||||
.await;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
let manager = self.subagent_manager.read().await;
|
||||
agent_list_event(&manager, &active_session_id)
|
||||
@@ -4074,11 +4103,16 @@ impl Engine {
|
||||
}
|
||||
Op::CancelSubAgent { agent_id } => {
|
||||
let active_session_id = self.session.id.clone();
|
||||
let cancelled = self
|
||||
.subagent_manager
|
||||
.write()
|
||||
.await
|
||||
.cancel_agent_for_session(&active_session_id, &agent_id);
|
||||
let manager = self.subagent_manager.clone();
|
||||
let session_id = active_session_id.clone();
|
||||
let target = agent_id.clone();
|
||||
let cancelled =
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
manager
|
||||
.blocking_write()
|
||||
.cancel_agent_for_session(&session_id, &target)
|
||||
})
|
||||
.await;
|
||||
let result = match cancelled {
|
||||
Ok(snapshot) => {
|
||||
let snapshot = crate::tools::subagent::settle_requested_child(
|
||||
@@ -4151,18 +4185,27 @@ impl Engine {
|
||||
let active_session_id = self.session.id.clone();
|
||||
let runtime = self.off_turn_subagent_runtime();
|
||||
let manager_handle = Arc::clone(&self.subagent_manager);
|
||||
let (outcome, refresh) = {
|
||||
let mut manager = self.subagent_manager.write().await;
|
||||
let outcome = manager
|
||||
.continue_child_from_user_for_session(
|
||||
&active_session_id,
|
||||
manager_handle,
|
||||
runtime,
|
||||
&agent_id,
|
||||
&text,
|
||||
)
|
||||
.map_err(|err| err.to_string());
|
||||
(outcome, agent_list_event(&manager, &active_session_id))
|
||||
let manager = Arc::clone(&manager_handle);
|
||||
let session = active_session_id.clone();
|
||||
let target = agent_id.clone();
|
||||
let result =
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
let mut manager = manager.blocking_write();
|
||||
let outcome = manager
|
||||
.continue_child_from_user_for_session(
|
||||
&session,
|
||||
manager_handle,
|
||||
runtime,
|
||||
&target,
|
||||
&text,
|
||||
)
|
||||
.map_err(|err| err.to_string());
|
||||
Ok((outcome, agent_list_event(&manager, &session)))
|
||||
})
|
||||
.await;
|
||||
let (outcome, refresh) = match result {
|
||||
Ok((outcome, refresh)) => (outcome, Some(refresh)),
|
||||
Err(error) => (Err(error.to_string()), None),
|
||||
};
|
||||
let _ = self
|
||||
.send_event(Event::SubAgentFollowUp {
|
||||
@@ -4171,7 +4214,9 @@ impl Engine {
|
||||
outcome,
|
||||
})
|
||||
.await;
|
||||
if let Err(_e) = self.tx_event.try_send(refresh) {
|
||||
if let Some(refresh) = refresh
|
||||
&& self.tx_event.try_send(refresh).is_err()
|
||||
{
|
||||
tracing::debug!(
|
||||
"Event channel full; dropping FollowUpSubAgent refresh"
|
||||
);
|
||||
@@ -4306,6 +4351,7 @@ impl Engine {
|
||||
self.config.fleet_roster = roster;
|
||||
}
|
||||
Op::SyncSession {
|
||||
control,
|
||||
session_id,
|
||||
messages,
|
||||
system_prompt,
|
||||
@@ -4314,12 +4360,6 @@ impl Engine {
|
||||
workspace,
|
||||
mode,
|
||||
} => {
|
||||
// A fresh conversation arrives with its own
|
||||
// sources: re-arm the guard and rebuild the notice
|
||||
// from the assembly that installs its header.
|
||||
self.instruction_guard_key = None;
|
||||
self.pending_instruction_notices.clear();
|
||||
self.session.instruction_assembly = None;
|
||||
let plugin_workspace_changed =
|
||||
self.plugin_registry.workspace() != workspace.as_path();
|
||||
let previous_session_id = self.session.id.clone();
|
||||
@@ -4330,7 +4370,8 @@ impl Engine {
|
||||
} else {
|
||||
previous_session_id.clone()
|
||||
};
|
||||
let closed_session_id = self.install_synced_session_id(next_session_id);
|
||||
let closed_session_id = (next_session_id != previous_session_id)
|
||||
.then_some(previous_session_id);
|
||||
// SyncSession installs a conversation's identity; an id
|
||||
// change IS a conversation boundary in this runtime —
|
||||
// callers must pass their own conversation id for
|
||||
@@ -4341,11 +4382,28 @@ impl Engine {
|
||||
// the new conversation (#5372). Same-session reloads
|
||||
// keep their id and are deliberately left untouched.
|
||||
if let Some(closed_session_id) = closed_session_id {
|
||||
let finalized = self
|
||||
.subagent_manager
|
||||
.write()
|
||||
.await
|
||||
.finalize_session_close_for_session(&closed_session_id);
|
||||
let manager = self.subagent_manager.clone();
|
||||
let session_id = closed_session_id.clone();
|
||||
let finalized =
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
manager
|
||||
.blocking_write()
|
||||
.finalize_session_close_for_session(&session_id)
|
||||
})
|
||||
.await;
|
||||
let finalized = match finalized {
|
||||
Ok(finalized) => finalized,
|
||||
Err(error) => {
|
||||
let _ = self
|
||||
.send_event(Event::error(ErrorEnvelope::transient(
|
||||
format!(
|
||||
"Failed to close sub-agent session {closed_session_id}; session was not switched: {error}"
|
||||
),
|
||||
)))
|
||||
.await;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
crate::tools::subagent::settle_requested_children_for_session(
|
||||
&self.subagent_manager,
|
||||
&closed_session_id,
|
||||
@@ -4360,11 +4418,20 @@ impl Engine {
|
||||
);
|
||||
}
|
||||
}
|
||||
// A fresh conversation arrives with its own
|
||||
// sources: re-arm the guard and rebuild the notice
|
||||
// from the assembly that installs its header.
|
||||
self.instruction_guard_key = None;
|
||||
self.pending_instruction_notices.clear();
|
||||
self.session.instruction_assembly = None;
|
||||
let _ = self.install_synced_session_id(next_session_id);
|
||||
self.restore_session_history(
|
||||
messages,
|
||||
system_prompt,
|
||||
system_prompt_override,
|
||||
);
|
||||
self.session.snapshot_source =
|
||||
Some(crate::core::ops::SessionSnapshotSource::control(control));
|
||||
self.session.auto_model = model.trim().eq_ignore_ascii_case("auto");
|
||||
self.session.model = model;
|
||||
self.session.workspace = workspace.clone();
|
||||
@@ -4414,6 +4481,7 @@ impl Engine {
|
||||
self.emit_session_updated().await;
|
||||
}
|
||||
Op::RewindConversation {
|
||||
control,
|
||||
expected,
|
||||
messages,
|
||||
tx,
|
||||
@@ -4433,6 +4501,8 @@ impl Engine {
|
||||
self.session.system_prompt.clone(),
|
||||
self.session.system_prompt_override,
|
||||
);
|
||||
self.session.snapshot_source =
|
||||
Some(crate::core::ops::SessionSnapshotSource::control(control));
|
||||
self.session.rebuild_working_set();
|
||||
self.reconcile_restored_work_bindings().await;
|
||||
let _ = tx.send(Some(self.session_snapshot()));
|
||||
@@ -4697,19 +4767,41 @@ impl Engine {
|
||||
|
||||
// #freeze: flush any sub-agent checkpoint that the hot-path debounce
|
||||
// coalesced away, so a graceful shutdown keeps the latest progress.
|
||||
{
|
||||
let mut manager = self.subagent_manager.write().await;
|
||||
let children = manager.list_for_session(&self.session.id);
|
||||
let manager = self.subagent_manager.clone();
|
||||
let session_id = self.session.id.clone();
|
||||
let owner_agent_id = self
|
||||
.child_host
|
||||
.as_ref()
|
||||
.map(|owner| owner.authority.owner_agent_id.clone());
|
||||
let shutdown = codewhale_runtime::runtime_control::owner_work(move || {
|
||||
let mut manager = manager.blocking_write();
|
||||
let children = manager.list_for_session(&session_id);
|
||||
let mut failure = None;
|
||||
for child in children {
|
||||
let owned = self.child_host.as_ref().is_none_or(|owner| {
|
||||
child.parent_run_id.as_deref()
|
||||
== Some(owner.authority.owner_agent_id.as_str())
|
||||
});
|
||||
if owned && child.status == SubAgentStatus::Running {
|
||||
let _ = manager.cancel_agent_for_session(&self.session.id, &child.agent_id);
|
||||
let owned = owner_agent_id
|
||||
.as_ref()
|
||||
.is_none_or(|owner| child.parent_run_id.as_deref() == Some(owner.as_str()));
|
||||
if owned
|
||||
&& child.status == SubAgentStatus::Running
|
||||
&& let Err(error) =
|
||||
manager.cancel_agent_for_session(&session_id, &child.agent_id)
|
||||
{
|
||||
failure.get_or_insert(error);
|
||||
}
|
||||
}
|
||||
manager.flush_pending_persist();
|
||||
if let Err(error) = manager.flush_pending_persist() {
|
||||
failure.get_or_insert(error);
|
||||
}
|
||||
failure.map_or(Ok(()), Err)
|
||||
})
|
||||
.await;
|
||||
if let Err(error) = shutdown {
|
||||
let _ = self
|
||||
.send_event(Event::error(ErrorEnvelope::transient(format!(
|
||||
"Sub-agent shutdown is uncertain: {error}"
|
||||
))))
|
||||
.await;
|
||||
child_result = Some(Err(error));
|
||||
}
|
||||
crate::tools::subagent::settle_requested_children_for_session(
|
||||
&self.subagent_manager,
|
||||
@@ -4719,7 +4811,19 @@ impl Engine {
|
||||
.map(|child| child.authority.owner_agent_id.as_str()),
|
||||
)
|
||||
.await;
|
||||
self.subagent_manager.write().await.flush_pending_persist();
|
||||
let manager = self.subagent_manager.clone();
|
||||
if let Err(error) = codewhale_runtime::runtime_control::owner_work(move || {
|
||||
manager.blocking_write().flush_pending_persist()
|
||||
})
|
||||
.await
|
||||
{
|
||||
let _ = self
|
||||
.send_event(Event::error(ErrorEnvelope::transient(format!(
|
||||
"Sub-agent shutdown checkpoint is uncertain: {error}"
|
||||
))))
|
||||
.await;
|
||||
child_result = Some(Err(error));
|
||||
}
|
||||
|
||||
// #420: graceful MCP shutdown — send SIGTERM and give stdio servers
|
||||
// a brief window to exit before drop fires SIGKILL via kill_on_drop.
|
||||
@@ -4772,6 +4876,7 @@ impl Engine {
|
||||
}
|
||||
let _ = self
|
||||
.send_event(Event::SessionUpdated {
|
||||
source: self.session.snapshot_source.clone(),
|
||||
checkpoint: None,
|
||||
session_id: self.session.id.clone(),
|
||||
messages: self.session.messages.snapshot(),
|
||||
@@ -6116,6 +6221,7 @@ impl Engine {
|
||||
.with_max_spawn_depth(self.config.max_spawn_depth)
|
||||
.with_step_api_timeout(self.config.subagent_api_timeout)
|
||||
.with_speech_output_dir(self.config.speech_output_dir.clone())
|
||||
.with_mcp_auth_context(Arc::clone(&self.config.mcp_auth_context.0))
|
||||
.with_mcp_pool(mcp_pool.clone())
|
||||
.with_todos(self.config.todos.clone())
|
||||
.with_parent_completion_tx(self.tx_subagent_completion.clone())
|
||||
@@ -6891,7 +6997,13 @@ impl Engine {
|
||||
// Emit turn started event IMMEDIATELY so the UI knows the turn is
|
||||
// active. The snapshot below can take 30+ seconds on slow filesystems
|
||||
// (e.g. WSL2 /mnt/c) and must not delay the TurnStarted event.
|
||||
let previous_source = self.session.snapshot_source.clone();
|
||||
self.session.snapshot_source = Some(crate::core::ops::SessionSnapshotSource::turn(
|
||||
turn.id.clone(),
|
||||
submission_id.clone(),
|
||||
));
|
||||
start_permit.send(Event::TurnStarted {
|
||||
previous_source,
|
||||
turn_id: turn.id.clone(),
|
||||
created_at: turn_started_at,
|
||||
route: Some(turn_route),
|
||||
@@ -7799,6 +7911,7 @@ impl Engine {
|
||||
.with_max_spawn_depth(self.config.max_spawn_depth)
|
||||
.with_step_api_timeout(self.config.subagent_api_timeout)
|
||||
.with_speech_output_dir(self.config.speech_output_dir.clone())
|
||||
.with_mcp_auth_context(Arc::clone(&self.config.mcp_auth_context.0))
|
||||
.with_mcp_pool(self.mcp_pool.clone())
|
||||
.with_todos(self.config.todos.clone())
|
||||
.with_parent_completion_tx(self.tx_subagent_completion.clone())
|
||||
@@ -8200,10 +8313,15 @@ impl Engine {
|
||||
|
||||
async fn ensure_mcp_pool(&mut self) -> Result<Arc<AsyncMutex<McpPool>>, ToolError> {
|
||||
if let Some(pool) = self.mcp_pool.clone() {
|
||||
pool.lock()
|
||||
.await
|
||||
.bind_caller_plugins(Arc::clone(&self.plugin_registry))
|
||||
let mut held = pool.lock().await;
|
||||
if !Arc::ptr_eq(&held.auth_context(), &self.config.mcp_auth_context.0) {
|
||||
return Err(ToolError::not_available(
|
||||
"MCP pool belongs to another credential authority",
|
||||
));
|
||||
}
|
||||
held.bind_caller_plugins(Arc::clone(&self.plugin_registry))
|
||||
.map_err(|error| ToolError::not_available(error.to_string()))?;
|
||||
drop(held);
|
||||
self.ensure_mcp_supervisor();
|
||||
return Ok(pool);
|
||||
}
|
||||
@@ -8213,26 +8331,31 @@ impl Engine {
|
||||
// file and `/mcp reload` recover in-process — but the person is told
|
||||
// that the configured servers are gone, not left to find no tools.
|
||||
let mut load_failure = None;
|
||||
let mut pool = McpPool::from_config_path_with_workspace_and_plugins(
|
||||
let mut pool = McpPool::from_config_path_with_workspace_and_plugins_and_auth_context(
|
||||
&self.session.mcp_config_path,
|
||||
&self.session.workspace,
|
||||
Arc::clone(&self.plugin_registry),
|
||||
Arc::clone(&self.config.mcp_auth_context.0),
|
||||
)
|
||||
.unwrap_or_else(|e| {
|
||||
let reason = crate::mcp::format_mcp_error_for_display(&e);
|
||||
tracing::warn!("MCP config unavailable: {reason}");
|
||||
load_failure = Some(reason);
|
||||
McpPool::empty_with_workspace_config_sources(
|
||||
McpPool::empty_with_workspace_config_sources_and_auth_context(
|
||||
&self.session.mcp_config_path,
|
||||
&self.session.workspace,
|
||||
Arc::clone(&self.plugin_registry),
|
||||
Arc::clone(&self.config.mcp_auth_context.0),
|
||||
)
|
||||
.unwrap_or_else(|fallback_error| {
|
||||
tracing::warn!(
|
||||
"MCP reload source setup failed: {}",
|
||||
crate::mcp::format_mcp_error_for_display(&fallback_error)
|
||||
);
|
||||
McpPool::new(McpConfig::default())
|
||||
McpPool::new_with_auth_context(
|
||||
McpConfig::default(),
|
||||
Arc::clone(&self.config.mcp_auth_context.0),
|
||||
)
|
||||
})
|
||||
});
|
||||
if let Some(reason) = load_failure {
|
||||
@@ -8748,7 +8871,7 @@ impl Engine {
|
||||
let Some(pool) = self.mcp_pool.as_ref() else {
|
||||
return state;
|
||||
};
|
||||
let (pending, errors, timeouts, network_policy, generation, backend) = {
|
||||
let (pending, errors, timeouts, network_policy, generation, backend, auth_context) = {
|
||||
let mut pool = pool.lock().await;
|
||||
let (pending, errors) = pool.take_pending_connects_for(names);
|
||||
(
|
||||
@@ -8758,6 +8881,7 @@ impl Engine {
|
||||
pool.cloned_network_policy(),
|
||||
pool.current_catalog_generation(),
|
||||
pool.backend(),
|
||||
pool.auth_context(),
|
||||
)
|
||||
};
|
||||
for (name, error) in errors {
|
||||
@@ -8766,8 +8890,14 @@ impl Engine {
|
||||
}
|
||||
state.catalog_generation = generation;
|
||||
state.names = pending.iter().map(|(name, _)| name.clone()).collect();
|
||||
state.connects =
|
||||
McpPool::spawn_pending_connects(pending, timeouts, network_policy, generation, backend);
|
||||
state.connects = McpPool::spawn_pending_connects_with_auth_context(
|
||||
pending,
|
||||
timeouts,
|
||||
network_policy,
|
||||
generation,
|
||||
backend,
|
||||
auth_context,
|
||||
);
|
||||
state
|
||||
}
|
||||
|
||||
@@ -8858,7 +8988,15 @@ impl Engine {
|
||||
// those marked `lazy = true`, which connect on demand — a selected
|
||||
// turn, a `/mcp` connect, or a lazy tool-name resolution.
|
||||
let requested = self.explicit_mcp_tool_names(self.config.allowed_tools.as_deref());
|
||||
let (pending, auth_errors, timeouts, network_policy, catalog_generation, backend) = {
|
||||
let (
|
||||
pending,
|
||||
auth_errors,
|
||||
timeouts,
|
||||
network_policy,
|
||||
catalog_generation,
|
||||
backend,
|
||||
auth_context,
|
||||
) = {
|
||||
let mut pool = pool.lock().await;
|
||||
match refresh {
|
||||
McpConnectRefresh::IfChanged => {
|
||||
@@ -8891,6 +9029,7 @@ impl Engine {
|
||||
pool.cloned_network_policy(),
|
||||
pool.current_catalog_generation(),
|
||||
pool.backend(),
|
||||
pool.auth_context(),
|
||||
)
|
||||
};
|
||||
|
||||
@@ -8925,12 +9064,13 @@ impl Engine {
|
||||
async move {
|
||||
let mut remaining: HashSet<String> =
|
||||
pending.iter().map(|(name, _)| name.clone()).collect();
|
||||
let mut connects = McpPool::spawn_pending_connects(
|
||||
let mut connects = McpPool::spawn_pending_connects_with_auth_context(
|
||||
pending,
|
||||
timeouts,
|
||||
network_policy,
|
||||
catalog_generation,
|
||||
backend,
|
||||
auth_context,
|
||||
);
|
||||
let mut connection_errors = HashMap::new();
|
||||
while let Some(joined) = connects.join_next().await {
|
||||
@@ -9052,17 +9192,19 @@ impl Engine {
|
||||
pool.cloned_network_policy(),
|
||||
pool.current_catalog_generation(),
|
||||
pool.backend(),
|
||||
pool.auth_context(),
|
||||
)
|
||||
})
|
||||
};
|
||||
let attempt = match prepared {
|
||||
Ok((pending, timeouts, network_policy, generation, backend)) => {
|
||||
let mut connects = McpPool::spawn_pending_connects(
|
||||
Ok((pending, timeouts, network_policy, generation, backend, auth_context)) => {
|
||||
let mut connects = McpPool::spawn_pending_connects_with_auth_context(
|
||||
vec![pending],
|
||||
timeouts,
|
||||
network_policy,
|
||||
generation,
|
||||
backend,
|
||||
auth_context,
|
||||
);
|
||||
let connection = match connects.join_next().await {
|
||||
Some(Ok((_, connection))) => connection,
|
||||
@@ -9361,7 +9503,11 @@ impl Engine {
|
||||
}
|
||||
None => Default::default(),
|
||||
};
|
||||
tool_catalog::configure_composition_entry(catalog, batch.as_deref(), &members);
|
||||
if tool_catalog::configure_composition_entry(catalog, batch.as_deref(), &members) {
|
||||
self.session
|
||||
.pending_prefix_change_reason
|
||||
.get_or_insert_with(|| "tool_surface".to_string());
|
||||
}
|
||||
let previous =
|
||||
self.session.messages.iter().rev().find(|message| {
|
||||
crate::runtime_handoff::is_mcp_server_instructions_message(message)
|
||||
@@ -10354,27 +10500,32 @@ fn apply_patch_permission_paths(input: &Value) -> Vec<String> {
|
||||
}
|
||||
|
||||
/// Spawn the engine in a background task
|
||||
pub fn spawn_engine(
|
||||
pub async fn spawn_engine(
|
||||
config: EngineConfig,
|
||||
api_config: &Config,
|
||||
mcp_handoff: Option<crate::mcp::McpHandoff>,
|
||||
) -> EngineHandle {
|
||||
let (engine, handle) = Engine::new(config, api_config);
|
||||
engine
|
||||
.mcp_pool_slot
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.handoff = mcp_handoff;
|
||||
let api_config = api_config.clone();
|
||||
let (engine, handle) = codewhale_runtime::runtime_control::owner_work(move || {
|
||||
let (engine, handle) = Engine::new(config, &api_config);
|
||||
engine
|
||||
.mcp_pool_slot
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.handoff = mcp_handoff;
|
||||
|
||||
Ok((Box::new(engine), handle))
|
||||
})
|
||||
.await
|
||||
.expect("Engine initialization owner failed");
|
||||
// Box the run future before supervision. An extra async wrapper embeds
|
||||
// the large engine state again in both its own and the supervisor's poll
|
||||
// frames, which can overflow an ordinary worker-thread stack.
|
||||
spawn_supervised(
|
||||
"engine-event-loop",
|
||||
std::panic::Location::caller(),
|
||||
Box::pin(engine.run()),
|
||||
Box::pin((*engine).run()),
|
||||
);
|
||||
|
||||
handle
|
||||
}
|
||||
|
||||
@@ -10552,6 +10703,7 @@ pub fn mock_engine_handle() -> MockEngineHandle {
|
||||
)));
|
||||
let compaction_cancellation = Arc::new(StdMutex::new(CompactionCancellationState::default()));
|
||||
let handle = EngineHandle {
|
||||
mcp_auth_context: crate::mcp::process_mcp_auth_context(),
|
||||
goal_state: new_shared_goal_state(),
|
||||
sandbox_enforcement: codewhale_sandbox::policy::SandboxEnforcement::Unavailable,
|
||||
tx_op,
|
||||
@@ -11069,11 +11221,9 @@ use self::streaming::TOOL_CALL_START_MARKERS;
|
||||
#[cfg(test)]
|
||||
use self::streaming::filter_tool_call_delta;
|
||||
use self::streaming::{
|
||||
ContentBlockKind, StreamResume, StreamRetryBudget, ToolCallDeltaFilterState, ToolUseState,
|
||||
ContentBlockKind, StreamRetryBudget, ToolCallDeltaFilterState, ToolUseState,
|
||||
contains_fake_tool_wrapper, filter_tool_call_delta_with_state, flush_tool_call_delta_state,
|
||||
should_resume_after_network_drop, should_resume_after_sleep,
|
||||
should_resume_interactive_after_network_drop, should_transparently_retry_stream,
|
||||
sleep_gap_detected, stream_read_error_user_message,
|
||||
stream_read_error_user_message,
|
||||
};
|
||||
use self::tool_catalog::{
|
||||
CODE_EXECUTION_TOOL_NAME, EXECUTE_TOOLS_TOOL_NAME, JS_EXECUTION_TOOL_NAME, ToolSurfacePolicy,
|
||||
|
||||
@@ -64,6 +64,8 @@ impl Engine {
|
||||
config.session_id = Some(authority.runtime.context.state_namespace.clone());
|
||||
config.plugin_registry = authority.runtime.context.plugin_registry.clone();
|
||||
config.runtime_services = authority.runtime.context.runtime.clone();
|
||||
config.mcp_auth_context =
|
||||
crate::mcp::McpAuthContext(Arc::clone(&authority.runtime.mcp_auth_context));
|
||||
config.todos = authority.runtime.todos.clone();
|
||||
config.max_spawn_depth = authority.runtime.max_spawn_depth;
|
||||
config.allow_shell = authority.runtime.allow_shell;
|
||||
@@ -317,6 +319,17 @@ impl Engine {
|
||||
.clone()
|
||||
.unwrap_or_else(|| "Core child turn interrupted".into()),
|
||||
),
|
||||
(TurnOutcomeStatus::Failed, _, _)
|
||||
if self
|
||||
.child_host
|
||||
.as_ref()
|
||||
.and_then(|state| state.request_stop_reason)
|
||||
== Some("request_outcome_unknown") =>
|
||||
{
|
||||
SubAgentStatus::Interrupted(error.clone().unwrap_or_else(|| {
|
||||
"provider request outcome unknown; checkpoint preserved".into()
|
||||
}))
|
||||
}
|
||||
(TurnOutcomeStatus::Failed, Some(reason), _)
|
||||
if reason == super::dispatch::FLEET_NO_PROGRESS_STOP =>
|
||||
{
|
||||
|
||||
@@ -64,9 +64,13 @@ impl Engine {
|
||||
id: String,
|
||||
auto: bool,
|
||||
message: String,
|
||||
previous_source: Option<crate::core::ops::SessionSnapshotSource>,
|
||||
) {
|
||||
let _ = self
|
||||
.send_event(Event::CompactionStarted { id, auto, message })
|
||||
.send_event(Event::CompactionStarted {
|
||||
previous_source,
|
||||
id, auto, message,
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
@@ -315,8 +319,6 @@ impl Engine {
|
||||
// Manual compaction terminals identify the compaction operation, never
|
||||
// a surrounding host submission or a model turn.
|
||||
let compaction_id = Some(id.clone());
|
||||
self.emit_compaction_started(id.clone(), false, "Making room…".to_string())
|
||||
.await;
|
||||
let Some(cancel_token) = self.claim_compaction(&id) else {
|
||||
let message = "Making room stopped before it started".to_string();
|
||||
self.emit_compaction_cancelled(id, false, message).await;
|
||||
@@ -336,6 +338,14 @@ impl Engine {
|
||||
.await;
|
||||
return;
|
||||
};
|
||||
let previous_source = self.session.snapshot_source.clone();
|
||||
self.session.snapshot_source = Some(crate::core::ops::SessionSnapshotSource::compaction(
|
||||
id.clone(),
|
||||
));
|
||||
self.emit_compaction_started(id.clone(), false, "Making room…".to_string(),
|
||||
previous_source,
|
||||
)
|
||||
.await;
|
||||
if let Err(err) = self.install_resolved_runtime_route(route) {
|
||||
let message =
|
||||
format!("Cannot compact context because its provider route is not ready: {err}");
|
||||
@@ -708,7 +718,9 @@ impl Engine {
|
||||
let compaction_cancel = self
|
||||
.claim_compaction(&compaction_id)
|
||||
.expect("a fresh automatic compaction id cannot be pre-canceled");
|
||||
self.emit_compaction_started(compaction_id.clone(), true, "Making room…".to_string())
|
||||
self.emit_compaction_started(compaction_id.clone(), true, "Making room…".to_string(),
|
||||
self.session.snapshot_source.clone(),
|
||||
)
|
||||
.await;
|
||||
let auto_messages_before = self.session.messages.len();
|
||||
let auto_tokens_before = self.estimated_input_tokens();
|
||||
@@ -933,7 +945,9 @@ impl Engine {
|
||||
.emergency_compaction_attempts
|
||||
.saturating_add(1);
|
||||
let start_message = format!("Making room now ({reason})");
|
||||
self.emit_compaction_started(id.clone(), true, start_message)
|
||||
self.emit_compaction_started(id.clone(), true, start_message,
|
||||
self.session.snapshot_source.clone(),
|
||||
)
|
||||
.await;
|
||||
let Some(compaction_cancel) = self.claim_compaction(&id) else {
|
||||
self.emit_compaction_cancelled(
|
||||
|
||||
@@ -381,6 +381,10 @@ impl Engine {
|
||||
let observation = self.request_retry_observation();
|
||||
let transport_retries = observation.retries.clone();
|
||||
let mut pending = Some(super::turn_loop::PendingProviderRequestReceipt {
|
||||
request_id: observation.request_id.clone(),
|
||||
route: client.effective_route_envelope(&self.session.model, chrono::Utc::now()),
|
||||
primary: true,
|
||||
invoked: true,
|
||||
turn_id: turn.id.clone(),
|
||||
request_index: turn.stop_diagnostics.model_requests_started,
|
||||
prefix_check: serde_json::json!({
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
//! Streaming response state and guardrails.
|
||||
//!
|
||||
//! This module owns the local state used while decoding one model stream:
|
||||
//! content block kind tracking, streamed tool-use buffers, transparent retry
|
||||
//! policy, and scrubbers for text that looks like a forged tool-call wrapper.
|
||||
//! content block kind tracking, streamed tool-use buffers, retry observations,
|
||||
//! and scrubbers for text that looks like a forged tool-call wrapper.
|
||||
|
||||
use crate::core::events::TurnOutcomeStatus;
|
||||
use codewhale_models::ToolCaller;
|
||||
use std::time::Duration;
|
||||
|
||||
/// A send that did not enter the existing event queue. Cancellation is not
|
||||
/// evidence that the consumer closed, and neither is user-visible delivery.
|
||||
@@ -81,6 +80,7 @@ impl super::Engine {
|
||||
let cancel = self.cancel_token.clone();
|
||||
let quiet = self.api_config.notifications_config().quiet;
|
||||
crate::llm_client::RequestRetryObservation {
|
||||
request_id: format!("primary:{}", uuid::Uuid::new_v4().simple()),
|
||||
wire_observations: Default::default(),
|
||||
retries: std::sync::Arc::new(std::sync::atomic::AtomicU32::new(0)),
|
||||
emit: std::sync::Arc::new(move |message| {
|
||||
@@ -237,63 +237,7 @@ pub(super) fn tool_call_limit_error() -> codewhale_error_taxonomy::ErrorEnvelope
|
||||
)
|
||||
}
|
||||
|
||||
/// Decide whether a stream error is eligible for a transparent retry.
|
||||
///
|
||||
/// True only when ALL three conditions hold:
|
||||
/// 1. No content has been received on the current attempt. Reissuing after
|
||||
/// visible partial deltas needs a separate recovery policy. This content
|
||||
/// check is not evidence that the provider consumed or billed zero tokens.
|
||||
/// 2. We still have transparent-retry budget remaining (`max_attempts`,
|
||||
/// `[tui].stream_max_transparent_retries`, default
|
||||
/// [`codewhale_runtime::turn_budget::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES`]).
|
||||
/// 3. The turn has not been cancelled.
|
||||
///
|
||||
/// Extracted as a pure function so the four #103 retry cases can be exercised
|
||||
/// in unit tests without booting the full engine state machine.
|
||||
pub(super) fn should_transparently_retry_stream(
|
||||
any_content_received: bool,
|
||||
transparent_attempts: u32,
|
||||
max_attempts: u32,
|
||||
cancelled: bool,
|
||||
) -> bool {
|
||||
!any_content_received && transparent_attempts < max_attempts && !cancelled
|
||||
}
|
||||
|
||||
/// Typed, engine-internal state for one mid-stream drop recovery.
|
||||
///
|
||||
/// This enum **is** the retry mechanism. A resumed turn used to append a
|
||||
/// synthetic `[runtime]` *user* message to the persisted conversation, which
|
||||
/// polluted the transcript and — when only hidden reasoning had streamed —
|
||||
/// promised a preserved partial answer that never existed (0.9.10
|
||||
/// regression). The retry is now modeled as this value, carried out of the
|
||||
/// stream decoder in [`StreamOutcome`] and consumed exactly once per drop:
|
||||
///
|
||||
/// * it is never persisted to the user transcript, and
|
||||
/// * nothing it triggers is serialized into the provider request history as
|
||||
/// a user role — the retried request is simply the persisted conversation
|
||||
/// re-issued, ending (when a visible fragment was preserved) with that
|
||||
/// assistant fragment so the provider continues from it.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(super) enum StreamResume {
|
||||
/// The stream died before anything actionable was streamed (#103
|
||||
/// Phase 3): discard the fragment and re-issue the identical request.
|
||||
NoContentStreamDeath,
|
||||
/// The host slept mid-stream (#2990): the partial output predates the
|
||||
/// sleep and no operator watched it — discard and re-issue.
|
||||
AfterSleep,
|
||||
/// Mid-stream network drop on a headless host (v0.9.4 Terminal-Bench
|
||||
/// P0): the fragment was never committed and no tool from it ran, so
|
||||
/// discard and re-issue the identical request.
|
||||
HeadlessNetworkDrop,
|
||||
/// Mid-stream network drop in the interactive TUI. A fragment with
|
||||
/// sendable content is preserved as the trailing assistant message and
|
||||
/// the request is re-issued; a thinking-only fragment has nothing
|
||||
/// visible to preserve, so it is discarded exactly like the headless
|
||||
/// resume and the copy must never claim otherwise.
|
||||
InteractiveNetworkDrop,
|
||||
}
|
||||
|
||||
/// Bounded authorization for drop-resume retries.
|
||||
/// Bounded authorization for proven pre-dispatch connection retries.
|
||||
///
|
||||
/// Mechanism, not comment: [`StreamRetryBudget::authorize`] is the only way
|
||||
/// to spend a resume and it returns `None` once `limit` resumes (default
|
||||
@@ -323,7 +267,7 @@ impl StreamRetryBudget {
|
||||
self.limit
|
||||
}
|
||||
|
||||
/// Drop-resumes already issued without a healthy round in between.
|
||||
/// Safe stream-open retries already issued without a healthy round in between.
|
||||
pub(super) fn spent(&self) -> u32 {
|
||||
self.spent
|
||||
}
|
||||
@@ -440,98 +384,9 @@ mod repeated_upstream_failure_tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// Wall-clock vs monotonic divergence above which we conclude the host slept
|
||||
/// mid-stream (#2990). `Instant` pauses during system sleep (CLOCK_UPTIME_RAW
|
||||
/// on macOS, CLOCK_MONOTONIC on Linux) while `SystemTime` keeps advancing, so
|
||||
/// a large positive gap can only come from a suspend/resume cycle — ordinary
|
||||
/// network flakes never produce one. Windows `Instant` may keep ticking
|
||||
/// through sleep, in which case this simply never fires (no behavior change).
|
||||
pub(super) const SLEEP_GAP_THRESHOLD: Duration = Duration::from_secs(10);
|
||||
|
||||
/// True when the gap between wall-clock and monotonic elapsed time since the
|
||||
/// last stream progress says the host was suspended.
|
||||
pub(super) fn sleep_gap_detected(monotonic_elapsed: Duration, wallclock_elapsed: Duration) -> bool {
|
||||
wallclock_elapsed.saturating_sub(monotonic_elapsed) > SLEEP_GAP_THRESHOLD
|
||||
}
|
||||
|
||||
/// Decide whether a failed stream should be silently re-issued because the
|
||||
/// host slept mid-turn (#2990).
|
||||
///
|
||||
/// Unlike the transparent retry (#103), this fires even after content has
|
||||
/// streamed: the partial output predates the sleep, the user was not
|
||||
/// watching, and re-running the identical request is the correct
|
||||
/// user-visible behavior. The double-billing concern that blocks ordinary
|
||||
/// post-content retries is accepted here because the alternative is a dead
|
||||
/// turn the user must re-prompt (and pay for) anyway.
|
||||
pub(super) fn should_resume_after_sleep(
|
||||
sleep_detected: bool,
|
||||
retry_attempts: u32,
|
||||
retry_limit: u32,
|
||||
cancelled: bool,
|
||||
) -> bool {
|
||||
sleep_detected && retry_attempts < retry_limit && !cancelled
|
||||
}
|
||||
|
||||
/// Decide whether a failed stream should be re-issued after a mid-stream
|
||||
/// network drop in a headless host (`exec` / stream-json / app-server), even
|
||||
/// though content already streamed.
|
||||
///
|
||||
/// This extends the #2990 sleep-resume contract to ordinary transport drops
|
||||
/// for hosts with no operator watching: the partial assistant fragment has
|
||||
/// not been committed to the conversation and no tool call from the
|
||||
/// incomplete response has executed, so discarding the fragment and
|
||||
/// re-issuing the identical request cannot duplicate side effects. The
|
||||
/// double-billing risk that blocks post-content retries in the interactive
|
||||
/// TUI (#103) is accepted here because the alternative is a dead turn that
|
||||
/// forfeits the entire headless run — the exact tradeoff #2990 already makes
|
||||
/// for sleep-resume. Interactive sessions keep the #103 surface-the-warning
|
||||
/// behavior: the user saw the partial deltas, and replaying would render the
|
||||
/// same prefix twice.
|
||||
pub(super) fn should_resume_after_network_drop(
|
||||
headless_host: bool,
|
||||
network_class_error: bool,
|
||||
retry_attempts: u32,
|
||||
retry_limit: u32,
|
||||
cancelled: bool,
|
||||
) -> bool {
|
||||
headless_host && network_class_error && retry_attempts < retry_limit && !cancelled
|
||||
}
|
||||
|
||||
/// Decide whether an interactive TUI stream should be re-issued after a
|
||||
/// mid-stream network drop, preserving a visible partial reply.
|
||||
///
|
||||
/// Unlike the headless resume, this keeps a sendable fragment: the user has
|
||||
/// already seen the deltas, so the assistant message is committed and the
|
||||
/// re-issued request ends with that fragment, which is the provider-neutral
|
||||
/// continuation contract. No synthetic user turn is appended — the retry is
|
||||
/// typed state ([`StreamResume::InteractiveNetworkDrop`]), invisible to the
|
||||
/// transcript and to the provider request history as a user role. A
|
||||
/// thinking-only fragment preserves nothing and must not be described as a
|
||||
/// preserved reply. Tool calls are never resumed because an incomplete tool
|
||||
/// call could be re-issued and duplicate side effects. Bounded by
|
||||
/// `DEFAULT_STREAM_MAX_RESUMES` and gated on a network/timeout-class error so
|
||||
/// model/parse/auth failures still surface normally.
|
||||
pub(super) fn should_resume_interactive_after_network_drop(
|
||||
terminal_chrome_enabled: bool,
|
||||
network_class_error: bool,
|
||||
any_content_received: bool,
|
||||
tool_uses_empty: bool,
|
||||
retry_attempts: u32,
|
||||
retry_limit: u32,
|
||||
cancelled: bool,
|
||||
) -> bool {
|
||||
terminal_chrome_enabled
|
||||
&& network_class_error
|
||||
&& any_content_received
|
||||
&& tool_uses_empty
|
||||
&& retry_attempts < retry_limit
|
||||
&& !cancelled
|
||||
}
|
||||
|
||||
/// Convert low-level reqwest/hyper stream read errors into an operator-facing
|
||||
/// message. The raw provider error remains attached, but the lead sentence
|
||||
/// explains why Codewhale may retry before any output and why it must surface
|
||||
/// the warning once partial output has already streamed.
|
||||
/// retains the interrupted output and the unknown provider outcome.
|
||||
pub(super) fn stream_read_error_user_message(message: &str, any_content_received: bool) -> String {
|
||||
let lower = message.to_ascii_lowercase();
|
||||
let is_stream_read = lower.contains("stream read error")
|
||||
@@ -545,7 +400,7 @@ pub(super) fn stream_read_error_user_message(message: &str, any_content_received
|
||||
let retry_note = if any_content_received {
|
||||
"Some output had already streamed, so Codewhale is surfacing the warning instead of replaying the request and risking duplicated output."
|
||||
} else {
|
||||
"No output had streamed yet, so Codewhale will retry automatically while retry budget remains."
|
||||
"No output had streamed yet; the provider request outcome is unknown and Codewhale will not replay it automatically."
|
||||
};
|
||||
format!(
|
||||
"Provider stream connection dropped while reading the response body. {retry_note} Details: {message}"
|
||||
|
||||
@@ -198,7 +198,6 @@ async fn exhausted_stream_resume_never_executes_the_failed_batch() {
|
||||
client.clone(),
|
||||
);
|
||||
engine.session.auto_approve = true;
|
||||
let resumes = engine.config.stream_retry_limits.max_resumes as usize;
|
||||
let executions = Arc::new(AtomicUsize::new(0));
|
||||
let surface = write_surface(&engine, workspace.path(), executions.clone());
|
||||
let mut turn = crate::core::turn::TurnContext::new(4);
|
||||
@@ -210,7 +209,7 @@ async fn exhausted_stream_resume_never_executes_the_failed_batch() {
|
||||
0,
|
||||
"no attempt's batch may run"
|
||||
);
|
||||
assert_eq!(client.calls.load(Ordering::SeqCst), 1 + resumes);
|
||||
assert_eq!(client.calls.load(Ordering::SeqCst), 1);
|
||||
assert_eq!(status, TurnOutcomeStatus::Failed);
|
||||
assert!(
|
||||
error
|
||||
|
||||
@@ -791,6 +791,7 @@ struct ChildTimeoutClient {
|
||||
attempts: std::sync::atomic::AtomicUsize,
|
||||
held_attempts: usize,
|
||||
body: bool,
|
||||
tool_use: bool,
|
||||
}
|
||||
impl crate::llm_client::LlmClient for ChildTimeoutClient {
|
||||
fn provider_name(&self) -> &'static str {
|
||||
@@ -825,8 +826,24 @@ impl crate::llm_client::LlmClient for ChildTimeoutClient {
|
||||
..Default::default()
|
||||
};
|
||||
}
|
||||
let mut events = vec![
|
||||
start,
|
||||
crate::llm_client::mock::canned::text_block_start(0),
|
||||
crate::llm_client::mock::canned::text_delta(0, "retained incomplete child text"),
|
||||
crate::llm_client::mock::canned::block_stop(0),
|
||||
];
|
||||
if self.tool_use {
|
||||
events.extend([
|
||||
crate::llm_client::mock::canned::tool_use_block_start(1, "call_1", "read"),
|
||||
crate::llm_client::mock::canned::tool_input_delta(
|
||||
1,
|
||||
"{\"path\":\"must-not-run\"}",
|
||||
),
|
||||
crate::llm_client::mock::canned::block_stop(1),
|
||||
]);
|
||||
}
|
||||
return Ok(Box::pin(
|
||||
futures_util::stream::once(async { Ok(start) })
|
||||
futures_util::stream::iter(events.into_iter().map(Ok))
|
||||
.chain(futures_util::stream::pending()),
|
||||
));
|
||||
}
|
||||
@@ -835,9 +852,9 @@ impl crate::llm_client::LlmClient for ChildTimeoutClient {
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
async fn child_step_timeout_bounds_open_and_body_with_one_outer_retry_and_exact_usage_sources() {
|
||||
async fn child_step_timeout_interrupts_open_and_body_without_replay_and_keeps_exact_sources() {
|
||||
use crate::llm_client::mock::canned;
|
||||
for body in [false, true] {
|
||||
for (body, tool_use) in [(false, false), (true, false), (true, true)] {
|
||||
let dir = tempdir().unwrap();
|
||||
let _home = crate::test_support::SealedHome::at(dir.path());
|
||||
let (_old, _old_handle, captured, _) = fixture(dir.path(), Some(Vec::new()));
|
||||
@@ -865,6 +882,7 @@ async fn child_step_timeout_bounds_open_and_body_with_one_outer_retry_and_exact_
|
||||
attempts: std::sync::atomic::AtomicUsize::new(0),
|
||||
held_attempts: 2,
|
||||
body,
|
||||
tool_use,
|
||||
});
|
||||
core.model_client = Some(timeout_client.clone());
|
||||
let (_input_tx, input_rx) = tokio::sync::mpsc::unbounded_channel();
|
||||
@@ -875,22 +893,22 @@ async fn child_step_timeout_bounds_open_and_body_with_one_outer_retry_and_exact_
|
||||
.await
|
||||
.expect("per-step timeout includes both stream opening and reading")
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
assert!(matches!(
|
||||
result.status,
|
||||
crate::tools::subagent::SubAgentStatus::Completed
|
||||
);
|
||||
crate::tools::subagent::SubAgentStatus::Interrupted(_)
|
||||
));
|
||||
assert_eq!(
|
||||
timeout_client
|
||||
.attempts
|
||||
.load(std::sync::atomic::Ordering::Acquire),
|
||||
3
|
||||
1
|
||||
);
|
||||
assert_eq!(
|
||||
job.steps(),
|
||||
1,
|
||||
"retries do not consume a fresh logical work step"
|
||||
);
|
||||
assert_eq!(successful.call_count(), 1);
|
||||
assert_eq!(successful.call_count(), 0);
|
||||
let record = job
|
||||
.authority
|
||||
.runtime
|
||||
@@ -902,16 +920,16 @@ async fn child_step_timeout_bounds_open_and_body_with_one_outer_retry_and_exact_
|
||||
if body {
|
||||
assert_eq!(
|
||||
record.usage_source_fingerprints.len(),
|
||||
3,
|
||||
"each opened response settles once before its retry"
|
||||
1,
|
||||
"the sole opened response settles once"
|
||||
);
|
||||
assert_eq!(record.usage.input_tokens, Some(44));
|
||||
assert_eq!(record.usage.output_tokens, Some(12));
|
||||
assert_eq!(record.usage.input_tokens, Some(17));
|
||||
assert_eq!(record.usage.output_tokens, Some(3));
|
||||
} else {
|
||||
assert_eq!(record.usage_source_fingerprints.len(), 3);
|
||||
assert_eq!(record.missing_usage_sources.len(), 2);
|
||||
assert_eq!(record.usage.input_tokens, Some(10));
|
||||
assert_eq!(record.usage.output_tokens, Some(6));
|
||||
assert_eq!(record.usage_source_fingerprints.len(), 1);
|
||||
assert_eq!(record.missing_usage_sources.len(), 1);
|
||||
assert_eq!(record.usage.input_tokens, None);
|
||||
assert_eq!(record.usage.output_tokens, None);
|
||||
assert!(record.missing_usage_sources.values().all(|coverage| {
|
||||
coverage.reason
|
||||
== crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown
|
||||
@@ -922,7 +940,31 @@ async fn child_step_timeout_bounds_open_and_body_with_one_outer_retry_and_exact_
|
||||
!job.can_replace_first_request(),
|
||||
"opened/accepted response forbids route replacement"
|
||||
);
|
||||
assert!(result.checkpoint.is_some());
|
||||
let checkpoint = result.checkpoint.as_ref().unwrap();
|
||||
assert!(checkpoint.continuable);
|
||||
assert_eq!(checkpoint.reason, "request_outcome_unknown");
|
||||
assert!(result.needs_input.is_some());
|
||||
if body {
|
||||
assert_eq!(
|
||||
result.result.as_deref(),
|
||||
Some("retained incomplete child text")
|
||||
);
|
||||
assert!(checkpoint.messages.iter().any(|message| {
|
||||
message.role == Role::InterruptedAssistant
|
||||
&& message.content.iter().any(|block| {
|
||||
matches!(block, ContentBlock::Text { text, .. }
|
||||
if text == "retained incomplete child text")
|
||||
})
|
||||
}));
|
||||
}
|
||||
assert!(checkpoint.messages.iter().all(|message| {
|
||||
message.content.iter().all(|block| {
|
||||
!matches!(
|
||||
block,
|
||||
ContentBlock::ToolUse { .. } | ContentBlock::ToolResult { .. }
|
||||
)
|
||||
})
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -948,6 +990,7 @@ async fn child_timeout_exhaustion_interrupts_with_checkpoint_and_never_dispatche
|
||||
attempts: std::sync::atomic::AtomicUsize::new(0),
|
||||
held_attempts: usize::MAX,
|
||||
body: false,
|
||||
tool_use: false,
|
||||
});
|
||||
core.model_client = Some(timeout_client.clone());
|
||||
let (_input_tx, input_rx) = tokio::sync::mpsc::unbounded_channel();
|
||||
@@ -960,13 +1003,13 @@ async fn child_timeout_exhaustion_interrupts_with_checkpoint_and_never_dispatche
|
||||
.unwrap();
|
||||
assert!(
|
||||
matches!(result.status, crate::tools::subagent::SubAgentStatus::Interrupted(ref reason)
|
||||
if reason.contains("6 API attempt(s)"))
|
||||
if reason.contains("outcome unknown"))
|
||||
);
|
||||
assert_eq!(
|
||||
timeout_client
|
||||
.attempts
|
||||
.load(std::sync::atomic::Ordering::Acquire),
|
||||
6
|
||||
1
|
||||
);
|
||||
assert_eq!(successful.call_count(), 0);
|
||||
assert_eq!(job.steps(), 1);
|
||||
@@ -993,6 +1036,7 @@ async fn child_cancelled_dispatched_open_keeps_unknown_origin_while_queued_cance
|
||||
attempts: std::sync::atomic::AtomicUsize::new(0),
|
||||
held_attempts: usize::MAX,
|
||||
body: false,
|
||||
tool_use: false,
|
||||
});
|
||||
core.model_client = Some(held.clone());
|
||||
let (_tx, rx) = tokio::sync::mpsc::unbounded_channel();
|
||||
|
||||
@@ -94,6 +94,7 @@ async fn queued_manual_compaction_cancellation_is_idempotent_and_skips_route_act
|
||||
.expect("structurally resolve route without credential");
|
||||
let (mut engine, handle) = Engine::new(EngineConfig::default(), &route_config);
|
||||
let id = "compact-cancel-before-start";
|
||||
let before = engine.session_snapshot();
|
||||
|
||||
handle.cancel_compaction(id).expect("first cancel accepted");
|
||||
handle
|
||||
@@ -103,16 +104,17 @@ async fn queued_manual_compaction_cancellation_is_idempotent_and_skips_route_act
|
||||
.handle_manual_compaction_op(id.to_string(), route, CompactionConfig::default())
|
||||
.await;
|
||||
|
||||
assert_eq!(engine.session_snapshot(), before);
|
||||
|
||||
let mut events = handle.rx_event.write().await;
|
||||
let drained = std::iter::from_fn(|| events.try_recv().ok()).collect::<Vec<_>>();
|
||||
assert!(matches!(
|
||||
drained.as_slice(),
|
||||
[
|
||||
Event::CompactionStarted { id: started, auto: false, .. },
|
||||
Event::CompactionCancelled { id: cancelled, auto: false, .. },
|
||||
Event::TurnComplete { turn_id: None, submission_id: None,
|
||||
compaction_id: Some(terminal), status: TurnOutcomeStatus::Interrupted, .. }
|
||||
] if started == id && cancelled == id && terminal == id
|
||||
] if cancelled == id && terminal == id
|
||||
));
|
||||
assert!(
|
||||
!drained
|
||||
|
||||
@@ -631,6 +631,7 @@ async fn edit_last_turn_restores_the_exchange_when_the_replacement_never_starts(
|
||||
};
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("edit-restore".to_string()),
|
||||
messages: vec![
|
||||
text(Role::User, "original prompt"),
|
||||
|
||||
@@ -218,8 +218,17 @@ fn retry_status_count(events: &[Event]) -> usize {
|
||||
}
|
||||
|
||||
async fn verify_next_user_turn_after_loss(failure: Failure) {
|
||||
let _environment = lock_test_env();
|
||||
let workspace = tempdir().expect("owned stream recovery workspace");
|
||||
let fixture_state = workspace.path().join(".codewhale");
|
||||
fs::create_dir_all(&fixture_state).unwrap();
|
||||
fs::write(
|
||||
fixture_state.join("AGENTS.md"),
|
||||
"Preserve the exact operator input.\n",
|
||||
)
|
||||
.unwrap();
|
||||
let _state_home = EnvVarGuard::set("CODEWHALE_HOME", &fixture_state);
|
||||
let server = LoopbackSse::start(failure).await;
|
||||
let workspace = tempdir().unwrap();
|
||||
let config = {
|
||||
let __cw_config_init_0_0 = Some("custom".to_string());
|
||||
let __cw_config_init_0_1 =
|
||||
@@ -233,7 +242,7 @@ async fn verify_next_user_turn_after_loss(failure: Failure) {
|
||||
Some("synthetic-loopback-key".to_string()),
|
||||
Some(server.base_url.clone()),
|
||||
);
|
||||
let (engine, handle) = Engine::new(
|
||||
let (mut engine, handle) = Engine::new(
|
||||
EngineConfig {
|
||||
max_steps: 1,
|
||||
terminal_chrome_enabled: true,
|
||||
@@ -242,7 +251,14 @@ async fn verify_next_user_turn_after_loss(failure: Failure) {
|
||||
},
|
||||
&config,
|
||||
);
|
||||
let task = tokio::spawn(engine.run());
|
||||
engine.instruction_settings.home = Some(workspace.path().to_path_buf());
|
||||
engine.refresh_system_prompt();
|
||||
let env_scope = crate::test_support::env_scope_ticket();
|
||||
let mut run = Box::pin(engine.run());
|
||||
let task = tokio::spawn(std::future::poll_fn(move |context| {
|
||||
let _membership = crate::test_support::join_env_scope(env_scope);
|
||||
std::future::Future::poll(run.as_mut(), context)
|
||||
}));
|
||||
send_user(&handle, &config, FIRST_USER).await;
|
||||
let mut first = Vec::new();
|
||||
if matches!(failure, Failure::StalledBody) {
|
||||
@@ -278,12 +294,7 @@ async fn verify_next_user_turn_after_loss(failure: Failure) {
|
||||
&& envelope.message.contains("error decoding response body")))
|
||||
);
|
||||
}
|
||||
let partial_count = match failure {
|
||||
Failure::TruncatedBody => {
|
||||
usize::try_from(super::super::DEFAULT_STREAM_MAX_RESUMES).unwrap() + 1
|
||||
}
|
||||
Failure::StalledBody => 1,
|
||||
};
|
||||
let partial_count = 1;
|
||||
assert_eq!(
|
||||
server.requests.lock().unwrap().len(),
|
||||
partial_count,
|
||||
@@ -295,10 +306,7 @@ async fn verify_next_user_turn_after_loss(failure: Failure) {
|
||||
partial_count,
|
||||
"terminal parent-request count must match POSTs observed by the loopback"
|
||||
);
|
||||
let expected_resumes = match failure {
|
||||
Failure::TruncatedBody => super::super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
Failure::StalledBody => 0,
|
||||
};
|
||||
let expected_resumes = 0;
|
||||
assert_eq!(first_terminal.stream_resumes, expected_resumes);
|
||||
assert_eq!(first_terminal.transparent_stream_retries, 0);
|
||||
assert_eq!(
|
||||
@@ -353,7 +361,9 @@ async fn verify_next_user_turn_after_loss(failure: Failure) {
|
||||
),
|
||||
"this regression has no pending approval or user-input gate: {event:?}"
|
||||
);
|
||||
if let Event::SessionUpdated { session_id, .. } = event {
|
||||
if let Event::SessionUpdated {
|
||||
source: _,
|
||||
session_id, .. } = event {
|
||||
assert_eq!(session_id, SESSION_ID);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -208,7 +208,7 @@ async fn event_capacity_invalid_images_release_queued_control_and_keep_classifie
|
||||
unreachable!()
|
||||
};
|
||||
spec.images
|
||||
.push(codewhale_protocol::runtime::RuntimeImageInput {
|
||||
.push(codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
mime: "image/png".into(),
|
||||
data_base64: "invalid@base64".into(),
|
||||
});
|
||||
|
||||
@@ -64,7 +64,9 @@ async fn queued_goal_clear_refreshes_prompt_and_cancels_stale_continuation() {
|
||||
Event::TurnStarted { .. } => {
|
||||
panic!("queued clear must prevent a stale goal continuation")
|
||||
}
|
||||
Event::SessionUpdated { system_prompt, .. } => {
|
||||
Event::SessionUpdated {
|
||||
source: _,
|
||||
system_prompt, .. } => {
|
||||
let prompt = match system_prompt.expect("clear SessionUpdated prompt") {
|
||||
SystemPrompt::Text(text) => text,
|
||||
SystemPrompt::Blocks(blocks) => blocks
|
||||
|
||||
@@ -430,7 +430,9 @@ async fn truncated_response_continues_turn() {
|
||||
assert_eq!(reported.reasoning_tokens, Some(3));
|
||||
saw_turn_usage = true;
|
||||
}
|
||||
Event::SessionUpdated { messages, .. } => {
|
||||
Event::SessionUpdated {
|
||||
source: _,
|
||||
messages, .. } => {
|
||||
saw_truncation_observation |= messages.iter().any(|message| {
|
||||
message.content.iter().any(|block| {
|
||||
matches!(
|
||||
|
||||
@@ -83,8 +83,8 @@ fn agent_mode_elevates_writes_without_granting_network() {
|
||||
#[test]
|
||||
fn sandbox_policy_for_turn_returns_correct_default_policy_per_mode() {
|
||||
use crate::core::authority::{SandboxNetworkAccess, sandbox_policy_for_turn};
|
||||
use codewhale_sandbox::SandboxPolicy;
|
||||
use ApprovalMode;
|
||||
use codewhale_sandbox::SandboxPolicy;
|
||||
|
||||
let workspace = PathBuf::from("/tmp/example-workspace");
|
||||
|
||||
@@ -181,7 +181,9 @@ async fn session_update_preserves_reasoning_tool_only_turn() {
|
||||
let mut rx = handle.rx_event.write().await;
|
||||
rx.recv().await.expect("session update event")
|
||||
};
|
||||
let Event::SessionUpdated { messages, .. } = event else {
|
||||
let Event::SessionUpdated {
|
||||
source: _,
|
||||
messages, .. } = event else {
|
||||
panic!("expected session update event");
|
||||
};
|
||||
|
||||
@@ -342,8 +344,8 @@ async fn posture_change_status_uses_permissions_and_work() {
|
||||
|
||||
#[tokio::test]
|
||||
async fn live_runtime_authority_applies_latest_posture_and_sandbox_before_tools() {
|
||||
use codewhale_sandbox::SandboxPolicy;
|
||||
use ApprovalMode;
|
||||
use codewhale_sandbox::SandboxPolicy;
|
||||
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
let config = EngineConfig {
|
||||
@@ -1004,6 +1006,7 @@ async fn unchanged_compaction_config_is_acknowledged_silently() {
|
||||
let run = tokio::spawn(engine.run());
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("resumed-session".to_string()),
|
||||
messages: restored_messages.clone(),
|
||||
system_prompt: None,
|
||||
@@ -1110,7 +1113,9 @@ async fn change_mode_op_updates_current_mode_and_emits_status() {
|
||||
.await
|
||||
.expect("session update after mode switch")
|
||||
.expect("event");
|
||||
let Event::SessionUpdated { messages, .. } = session_updated else {
|
||||
let Event::SessionUpdated {
|
||||
source: _,
|
||||
messages, .. } = session_updated else {
|
||||
panic!("should emit SessionUpdated after mode change, got: {session_updated:?}");
|
||||
};
|
||||
assert!(
|
||||
@@ -1195,6 +1200,7 @@ async fn sync_session_restores_current_mode() {
|
||||
let run = tokio::spawn(engine.run());
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("plan-session".to_string()),
|
||||
messages: Vec::new(),
|
||||
system_prompt: None,
|
||||
@@ -1275,6 +1281,7 @@ async fn sync_session_without_prompt_repins_full_system_prompt_on_next_turn() {
|
||||
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("fresh-session".to_string()),
|
||||
messages: Vec::new(),
|
||||
system_prompt: None,
|
||||
@@ -1356,6 +1363,7 @@ async fn sync_session_same_id_does_not_finalize_live_worker() {
|
||||
// construction-time UUID.
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("session-keep".to_string()),
|
||||
messages: Vec::new(),
|
||||
system_prompt: None,
|
||||
@@ -1376,6 +1384,7 @@ async fn sync_session_same_id_does_not_finalize_live_worker() {
|
||||
// A same-id re-sync is a reload, not a conversation boundary.
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("session-keep".to_string()),
|
||||
messages: Vec::new(),
|
||||
system_prompt: None,
|
||||
@@ -1417,6 +1426,7 @@ async fn sync_session_different_id_finalizes_live_worker() {
|
||||
let run = tokio::spawn(engine.run());
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("session-a".to_string()),
|
||||
messages: Vec::new(),
|
||||
system_prompt: None,
|
||||
@@ -1440,6 +1450,7 @@ async fn sync_session_different_id_finalizes_live_worker() {
|
||||
// finalized with the session-closed reason.
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("session-b".to_string()),
|
||||
messages: Vec::new(),
|
||||
system_prompt: None,
|
||||
@@ -1496,6 +1507,7 @@ async fn sync_session_migrates_one_checkpoint_and_strips_its_system_carrier() {
|
||||
for round in 0..2 {
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("compacted-session".to_string()),
|
||||
messages,
|
||||
system_prompt: Some(carrier.clone()),
|
||||
|
||||
@@ -27,6 +27,7 @@ async fn sync_session_projects_persisted_subagent_handoff_for_headless_restore()
|
||||
let run = tokio::spawn(engine.run());
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("headless-resume".to_string()),
|
||||
messages,
|
||||
system_prompt: None,
|
||||
@@ -55,7 +56,8 @@ async fn sync_session_projects_persisted_subagent_handoff_for_headless_restore()
|
||||
|block| matches!(block, ContentBlock::Text { text, .. } if text == "Keep the original task")
|
||||
));
|
||||
let restored =
|
||||
codewhale_models::turn_classify::restored_subagent_checkpoint_display(&snapshot.messages[1])
|
||||
codewhale_models::turn_classify::restored_subagent_checkpoint_display(&snapshot.messages[1],
|
||||
)
|
||||
.expect("projected headless checkpoint");
|
||||
assert!(restored.contains("agent_headless"));
|
||||
assert!(restored.contains("Checkpoint: engine restore is covered."));
|
||||
@@ -164,6 +166,7 @@ async fn edit_last_turn_preserves_current_mode() {
|
||||
];
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("edit-mode-test".to_string()),
|
||||
messages: seeded_messages,
|
||||
system_prompt: None,
|
||||
@@ -320,6 +323,7 @@ async fn edit_last_turn_cuts_at_user_prompt_before_tool_results() {
|
||||
];
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("edit-cut-test".to_string()),
|
||||
messages: seeded_messages,
|
||||
system_prompt: None,
|
||||
@@ -424,6 +428,7 @@ async fn edit_last_turn_without_user_prompt_errors_and_sends_nothing() {
|
||||
// must surface an error instead of silently appending the message.
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("edit-no-user-test".to_string()),
|
||||
messages: vec![Message {
|
||||
role: Role::Assistant,
|
||||
@@ -546,6 +551,7 @@ async fn edit_last_turn_without_user_prompt_errors_and_sends_nothing() {
|
||||
];
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some("edit-unsupported-user-test".to_string()),
|
||||
messages: image_only_history.clone(),
|
||||
system_prompt: None,
|
||||
@@ -727,7 +733,8 @@ fn context_budget_scenario() {
|
||||
let _lock = lock_test_env();
|
||||
let budget = context_input_budget_for_provider(ProviderKind::OpenaiCodex, "gpt-5.5")
|
||||
.expect("OpenAI Codex should use a conservative fallback without route metadata");
|
||||
let expected = usize::try_from(codewhale_runtime::config::OPENAI_CODEX_EFFECTIVE_CONTEXT_WINDOW_TOKENS)
|
||||
let expected = usize::try_from(codewhale_runtime::config::OPENAI_CODEX_EFFECTIVE_CONTEXT_WINDOW_TOKENS,
|
||||
)
|
||||
.expect("context window fits usize")
|
||||
- crate::provider_lake::provider_capability(ProviderKind::OpenaiCodex, "gpt-5.5")
|
||||
.max_output
|
||||
|
||||
@@ -88,7 +88,9 @@ async fn trust_warning_is_internal_and_tracks_current_state() {
|
||||
engine.refresh_system_prompt_with_reason("model");
|
||||
engine.emit_session_updated().await;
|
||||
let event = handle.rx_event.write().await.recv().await.unwrap();
|
||||
let Event::SessionUpdated { messages, .. } = event else {
|
||||
let Event::SessionUpdated {
|
||||
source: _,
|
||||
messages, .. } = event else {
|
||||
panic!("expected session update");
|
||||
};
|
||||
assert_eq!(
|
||||
@@ -1563,83 +1565,10 @@ fn final_tool_scenario() {
|
||||
}
|
||||
}
|
||||
|
||||
// === #103 transparent stream-retry policy =====================================
|
||||
// === Proven pre-dispatch retry budget =========================================
|
||||
|
||||
#[test]
|
||||
fn stream_retry_scenario() {
|
||||
// Scenario consolidation of: stream_retry_zero_content_then_error_is_transparently_retried, stream_retry_after_content_received_surfaces_error, stream_retry_respects_cancellation, stream_retry_budget_caps_resumes_in_mechanism, stream_retry_threshold_relaxed_to_five
|
||||
// from stream_retry_zero_content_then_error_is_transparently_retried
|
||||
{
|
||||
// Case 2 from issue #103: stream yielded ZERO content then errored.
|
||||
// The decoder hit Err on the very first poll → engine should retry
|
||||
// because DeepSeek hasn't billed and the user has seen nothing.
|
||||
assert!(
|
||||
super::should_transparently_retry_stream(
|
||||
false,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
false
|
||||
),
|
||||
"first attempt with no content must be eligible for transparent retry"
|
||||
);
|
||||
assert!(
|
||||
super::should_transparently_retry_stream(
|
||||
false,
|
||||
1,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
false
|
||||
),
|
||||
"second attempt (one prior retry) with no content must still be eligible"
|
||||
);
|
||||
}
|
||||
// from stream_retry_after_content_received_surfaces_error
|
||||
{
|
||||
// Case 3 from issue #103: stream yielded content then errored. We must
|
||||
// NOT transparently retry — the model has emitted billed output tokens
|
||||
// and the UI has streamed deltas; resending would double-bill and the
|
||||
// user would see the same prefix twice.
|
||||
assert!(
|
||||
!super::should_transparently_retry_stream(
|
||||
true,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
false
|
||||
),
|
||||
"any content received → no transparent retry, even with full budget"
|
||||
);
|
||||
assert!(
|
||||
!super::should_transparently_retry_stream(
|
||||
true,
|
||||
1,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
false
|
||||
),
|
||||
"any content received → no transparent retry on subsequent attempts"
|
||||
);
|
||||
}
|
||||
// from stream_retry_respects_cancellation
|
||||
{
|
||||
// Cancellation overrides every other condition. If the user pressed
|
||||
// Esc / Ctrl-C, do not silently re-issue the request behind their back.
|
||||
assert!(
|
||||
!super::should_transparently_retry_stream(
|
||||
false,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
true
|
||||
),
|
||||
"cancelled turn must not be transparently retried"
|
||||
);
|
||||
assert!(
|
||||
!super::should_transparently_retry_stream(
|
||||
false,
|
||||
1,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
true
|
||||
),
|
||||
"cancelled turn must not be transparently retried even with budget"
|
||||
);
|
||||
}
|
||||
// from stream_retry_budget_caps_resumes_in_mechanism
|
||||
{
|
||||
// "At most one bounded retry per drop" is enforced by types, not by a
|
||||
@@ -1664,22 +1593,18 @@ fn stream_retry_scenario() {
|
||||
}
|
||||
// from stream_retry_threshold_relaxed_to_five
|
||||
{
|
||||
// Case 1+4 from issue #103: the consecutive-error threshold for marking
|
||||
// the turn failed was relaxed from 3 → 5 in v0.6.7 because the new
|
||||
// HTTP/2 keepalive defaults make spurious decode errors rarer.
|
||||
// This test pins the constant so a future regression to 3 fails loudly.
|
||||
// Historical values remain parsed for configuration compatibility.
|
||||
// They do not authorize replay or tolerance of an admitted stream error.
|
||||
assert_eq!(
|
||||
super::DEFAULT_STREAM_MAX_ERRORS,
|
||||
5,
|
||||
"the consecutive-stream-error threshold should be 5; \
|
||||
lowering it back to 3 will fail mid-turn under transient flakiness"
|
||||
"the historical error-count compatibility value remains5"
|
||||
);
|
||||
// And a regression guard on the transparent-retry cap.
|
||||
assert_eq!(
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
2,
|
||||
"transparent-retry cap should be 2; raising it risks hammering the \
|
||||
provider on real outages"
|
||||
"the historical transparent-retry compatibility value remains2"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -10,7 +10,7 @@ fn stream_read_scenario() {
|
||||
|
||||
assert!(message.contains("Provider stream connection dropped"));
|
||||
assert!(message.contains("No output had streamed yet"));
|
||||
assert!(message.contains("retry automatically"));
|
||||
assert!(message.contains("will not replay it automatically"));
|
||||
assert!(message.contains("Stream read error: error decoding response body"));
|
||||
}
|
||||
// from stream_read_error_message_explains_no_replay_after_output
|
||||
@@ -31,327 +31,6 @@ fn stream_read_scenario() {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stream_retry_budget_caps_transparent_retries_at_two() {
|
||||
// Case 4 from issue #103: after DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES attempts
|
||||
// we stop trying transparently and let the outer error path surface.
|
||||
// (The outer per-turn `stream_retry_attempts` retry is a separate layer
|
||||
// and is still in effect at the whole-turn level.)
|
||||
assert!(
|
||||
super::should_transparently_retry_stream(
|
||||
false,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES - 1,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
false,
|
||||
),
|
||||
"one short of the cap should still retry"
|
||||
);
|
||||
assert!(
|
||||
!super::should_transparently_retry_stream(
|
||||
false,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
false,
|
||||
),
|
||||
"at the cap, no further transparent retries"
|
||||
);
|
||||
assert!(
|
||||
!super::should_transparently_retry_stream(
|
||||
false,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES + 5,
|
||||
super::DEFAULT_STREAM_MAX_TRANSPARENT_RETRIES,
|
||||
false,
|
||||
),
|
||||
"well past the cap, definitely no transparent retries"
|
||||
);
|
||||
}
|
||||
|
||||
// === #2990 sleep-resume policy ================================================
|
||||
|
||||
#[test]
|
||||
fn sleep_gap_requires_wallclock_to_outrun_monotonic_clock() {
|
||||
use std::time::Duration;
|
||||
// No divergence: ordinary network failure, clocks agree.
|
||||
assert!(
|
||||
!super::sleep_gap_detected(Duration::from_secs(30), Duration::from_secs(30)),
|
||||
"equal elapsed times must not register as a sleep gap"
|
||||
);
|
||||
// Divergence below the threshold: NTP slew / scheduling jitter.
|
||||
assert!(
|
||||
!super::sleep_gap_detected(Duration::from_secs(5), Duration::from_secs(14)),
|
||||
"9s of divergence is below the 10s threshold"
|
||||
);
|
||||
// Divergence above the threshold: the host was suspended.
|
||||
assert!(
|
||||
super::sleep_gap_detected(Duration::from_secs(5), Duration::from_secs(16)),
|
||||
"11s of divergence must register as a sleep gap"
|
||||
);
|
||||
// Wall clock went backwards (NTP step): saturating_sub → zero gap.
|
||||
assert!(
|
||||
!super::sleep_gap_detected(Duration::from_secs(60), Duration::from_secs(5)),
|
||||
"wall clock behind monotonic must never register as a sleep gap"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sleep_resume_scenario() {
|
||||
// Scenario consolidation of: sleep_resume_retries_even_after_content_streamed, sleep_resume_requires_a_detected_gap, sleep_resume_respects_budget_and_cancellation
|
||||
// from sleep_resume_retries_even_after_content_streamed
|
||||
{
|
||||
// The whole point of #2990: unlike the #103 transparent retry, a
|
||||
// detected sleep gap retries regardless of streamed content — the
|
||||
// partial output predates the sleep and the user was not watching.
|
||||
assert!(
|
||||
super::should_resume_after_sleep(true, 0, super::DEFAULT_STREAM_MAX_RESUMES, false),
|
||||
"detected sleep with full budget must resume"
|
||||
);
|
||||
assert!(
|
||||
super::should_resume_after_sleep(
|
||||
true,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES - 1,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"detected sleep one short of the budget must still resume"
|
||||
);
|
||||
}
|
||||
// from sleep_resume_requires_a_detected_gap
|
||||
{
|
||||
// Without a sleep gap this layer stays out of the way entirely, so the
|
||||
// deliberate no-retry-after-content policy for ordinary flakes (#103)
|
||||
// is preserved.
|
||||
assert!(
|
||||
!super::should_resume_after_sleep(false, 0, super::DEFAULT_STREAM_MAX_RESUMES, false),
|
||||
"no sleep gap → never resume via this layer"
|
||||
);
|
||||
}
|
||||
// from sleep_resume_respects_budget_and_cancellation
|
||||
{
|
||||
assert!(
|
||||
!super::should_resume_after_sleep(
|
||||
true,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"budget exhausted → surface the failure instead of looping"
|
||||
);
|
||||
assert!(
|
||||
!super::should_resume_after_sleep(true, 0, super::DEFAULT_STREAM_MAX_RESUMES, true),
|
||||
"cancelled turn must not be resumed behind the user's back"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// === headless mid-stream network-drop resume (v0.9.4 Terminal-Bench P0) ======
|
||||
//
|
||||
// Terminal-Bench 2.1 on the 0.9.4 bundle forfeited tasks when the DeepSeek
|
||||
// stream dropped mid-response ("error decoding response body" after partial
|
||||
// content): the #103 policy surfaced the warning and failed the turn, and
|
||||
// `codewhale exec` exited 1. In a headless host no operator watches the
|
||||
// partial deltas and the fragment is never committed, so the turn loop now
|
||||
// re-issues the request instead (bounded by DEFAULT_STREAM_MAX_RESUMES), exactly
|
||||
// like the #2990 sleep-resume.
|
||||
|
||||
#[test]
|
||||
fn network_drop_scenario() {
|
||||
// Scenario consolidation of: network_drop_resume_only_fires_for_headless_hosts, network_drop_resume_requires_network_class_error, network_drop_resume_respects_budget_and_cancellation
|
||||
// from network_drop_resume_only_fires_for_headless_hosts
|
||||
{
|
||||
assert!(
|
||||
super::should_resume_after_network_drop(
|
||||
true,
|
||||
true,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"headless host + network-class drop with budget must resume"
|
||||
);
|
||||
assert!(
|
||||
!super::should_resume_after_network_drop(
|
||||
false,
|
||||
true,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"interactive sessions keep the #103 surface-the-warning policy: \
|
||||
the user saw the partial deltas and replay would render them twice"
|
||||
);
|
||||
}
|
||||
// from network_drop_resume_requires_network_class_error
|
||||
{
|
||||
assert!(
|
||||
!super::should_resume_after_network_drop(
|
||||
true,
|
||||
false,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"non-network failures (model/parse/auth) must never be replayed"
|
||||
);
|
||||
}
|
||||
// from network_drop_resume_respects_budget_and_cancellation
|
||||
{
|
||||
assert!(
|
||||
super::should_resume_after_network_drop(
|
||||
true,
|
||||
true,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES - 1,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false,
|
||||
),
|
||||
"one short of the budget should still resume"
|
||||
);
|
||||
assert!(
|
||||
!super::should_resume_after_network_drop(
|
||||
true,
|
||||
true,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"budget exhausted → surface the failure instead of looping"
|
||||
);
|
||||
assert!(
|
||||
!super::should_resume_after_network_drop(
|
||||
true,
|
||||
true,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
true
|
||||
),
|
||||
"cancelled turn must not be resumed behind the operator's back"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// === interactive mid-stream network-drop resume (0.9.4; reworked 0.9.10) =========
|
||||
//
|
||||
// The interactive TUI used to fail the turn when a provider stream dropped
|
||||
// after partial output because the #103 policy treated any post-content error
|
||||
// as terminal. The model now preserves a visible partial reply as a committed
|
||||
// assistant message and re-issues the request. Since 0.9.10 the recovery is
|
||||
// typed engine-internal state (`StreamResume`): no synthetic `[runtime]` user
|
||||
// continuation message is appended, and a thinking-only drop preserves
|
||||
// nothing and never claims it did.
|
||||
|
||||
#[test]
|
||||
fn interactive_network_scenario() {
|
||||
// Scenario consolidation of: interactive_network_drop_resume_only_fires_for_interactive_hosts, interactive_network_drop_resume_requires_partial_content_and_no_tools, interactive_network_drop_resume_requires_network_class_error
|
||||
// from interactive_network_drop_resume_only_fires_for_interactive_hosts
|
||||
{
|
||||
assert!(
|
||||
super::should_resume_interactive_after_network_drop(
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"interactive TUI + partial text + no tools + budget must resume"
|
||||
);
|
||||
assert!(
|
||||
!super::should_resume_interactive_after_network_drop(
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"headless hosts must use the headless resume path, not this one"
|
||||
);
|
||||
}
|
||||
// from interactive_network_drop_resume_requires_partial_content_and_no_tools
|
||||
{
|
||||
assert!(
|
||||
!super::should_resume_interactive_after_network_drop(
|
||||
true,
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"no streamed content → transparent retry or nothing-streamed path"
|
||||
);
|
||||
assert!(
|
||||
!super::should_resume_interactive_after_network_drop(
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
false,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"in-flight tool calls must never be resumed (side-effect duplication)"
|
||||
);
|
||||
}
|
||||
// from interactive_network_drop_resume_requires_network_class_error
|
||||
{
|
||||
assert!(
|
||||
!super::should_resume_interactive_after_network_drop(
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false
|
||||
),
|
||||
"non-network failures must surface normally"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn interactive_network_drop_resume_respects_budget_and_cancellation() {
|
||||
assert!(
|
||||
super::should_resume_interactive_after_network_drop(
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES - 1,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false,
|
||||
),
|
||||
"one short of the budget should still resume"
|
||||
);
|
||||
assert!(
|
||||
!super::should_resume_interactive_after_network_drop(
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
false,
|
||||
),
|
||||
"budget exhausted → surface the failure"
|
||||
);
|
||||
assert!(
|
||||
!super::should_resume_interactive_after_network_drop(
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
0,
|
||||
super::DEFAULT_STREAM_MAX_RESUMES,
|
||||
true
|
||||
),
|
||||
"cancelled turn must not resume"
|
||||
);
|
||||
}
|
||||
|
||||
/// Model client whose first `failures` streams emit partial content and then
|
||||
/// die with the network-class read error reqwest reports for a dropped
|
||||
/// chunked-transfer body; later streams complete a normal text turn.
|
||||
@@ -607,14 +286,14 @@ async fn headless_turn_retries_mid_stream_network_drop_and_recovers() {
|
||||
_ => None,
|
||||
})
|
||||
.expect("terminal diagnostics through existing event authority");
|
||||
assert_eq!(terminal.model_requests_started, 2);
|
||||
assert_eq!(terminal.stream_resumes, 1);
|
||||
assert_eq!(terminal.model_requests_started, 1);
|
||||
assert_eq!(terminal.stream_resumes, 0);
|
||||
assert_eq!(terminal.transparent_stream_retries, 0);
|
||||
|
||||
assert_eq!(
|
||||
model.calls.load(std::sync::atomic::Ordering::SeqCst),
|
||||
2,
|
||||
"the dropped stream must be re-issued exactly once"
|
||||
1,
|
||||
"an admitted dropped stream must not be replayed"
|
||||
);
|
||||
let (status, error) = events
|
||||
.iter()
|
||||
@@ -625,29 +304,30 @@ async fn headless_turn_retries_mid_stream_network_drop_and_recovers() {
|
||||
.expect("terminal TurnComplete");
|
||||
assert_eq!(
|
||||
*status,
|
||||
TurnOutcomeStatus::Completed,
|
||||
"a recovered retry must complete the turn: {error:?}"
|
||||
TurnOutcomeStatus::Failed,
|
||||
"an unknown dropped request must fail the turn: {error:?}"
|
||||
);
|
||||
assert!(error.is_none(), "recovered turn must not report an error");
|
||||
assert!(error.is_some(), "unknown outcome must remain visible");
|
||||
assert!(
|
||||
events.iter().any(|event| matches!(
|
||||
!events.iter().any(|event| matches!(
|
||||
event,
|
||||
Event::Status { message } if message.starts_with("Retry attempt: stream-resume 1/")
|
||||
)),
|
||||
"the first actual resume must remain visible: {events:?}"
|
||||
"an uncertain opened stream must not resume automatically: {events:?}"
|
||||
);
|
||||
assert!(
|
||||
!events
|
||||
events
|
||||
.iter()
|
||||
.any(|event| matches!(event, Event::Error { .. })),
|
||||
"a transient drop that the retry recovers must not surface an error event: {events:?}"
|
||||
"an unknown dropped response must surface its error: {events:?}"
|
||||
);
|
||||
// The discarded fragment from the dropped attempt must never reach the
|
||||
// transcript — only the retried turn's content is committed.
|
||||
// The actual dropped fragment survives without a second provider request.
|
||||
let transcript_text = events
|
||||
.iter()
|
||||
.filter_map(|event| match event {
|
||||
Event::SessionUpdated { messages, .. } => Some(messages),
|
||||
Event::SessionUpdated {
|
||||
source: _,
|
||||
messages, .. } => Some(messages),
|
||||
_ => None,
|
||||
})
|
||||
.flat_map(|messages| messages.iter())
|
||||
@@ -659,12 +339,12 @@ async fn headless_turn_retries_mid_stream_network_drop_and_recovers() {
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
assert!(
|
||||
transcript_text.contains("recovered after retry"),
|
||||
"retried content must be committed: {transcript_text}"
|
||||
!transcript_text.contains("recovered after retry"),
|
||||
"no retried content may be committed: {transcript_text}"
|
||||
);
|
||||
assert!(
|
||||
!transcript_text.contains("partial answer that must be discarded"),
|
||||
"the dropped attempt's fragment must be discarded, not committed: {transcript_text}"
|
||||
transcript_text.contains("partial answer that must be discarded"),
|
||||
"the dropped attempt's fragment must be retained: {transcript_text}"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -690,22 +370,22 @@ async fn terminal_diagnostics_count_transparent_stream_requests_without_extra_sn
|
||||
let surface = test_tool_surface(&engine, registry, None, AppMode::Agent);
|
||||
let mut turn = crate::core::turn::TurnContext::new(4);
|
||||
let (status, error) = engine.run_turn(&mut turn, surface, None, None).await;
|
||||
assert_eq!(status, TurnOutcomeStatus::Completed, "{error:?}");
|
||||
assert_eq!(model.calls.load(std::sync::atomic::Ordering::SeqCst), 2);
|
||||
assert_eq!(status, TurnOutcomeStatus::Failed, "{error:?}");
|
||||
assert_eq!(model.calls.load(std::sync::atomic::Ordering::SeqCst), 1);
|
||||
let snapshot = turn
|
||||
.terminal_request_snapshot(status)
|
||||
.expect("terminal snapshot");
|
||||
let terminal = snapshot.terminal.expect("terminal facts");
|
||||
assert_eq!(terminal.model_requests_started, 2);
|
||||
assert_eq!(terminal.transparent_stream_retries, 1);
|
||||
assert_eq!(terminal.model_requests_started, 1);
|
||||
assert_eq!(terminal.transparent_stream_retries, 0);
|
||||
assert_eq!(terminal.stream_resumes, 0);
|
||||
assert_eq!(terminal.model_step_index, 0);
|
||||
let mut receiver = handle.rx_event.write().await;
|
||||
let events = std::iter::from_fn(|| receiver.try_recv().ok()).collect::<Vec<_>>();
|
||||
assert!(events.iter().any(|event| matches!(event,
|
||||
assert!(!events.iter().any(|event| matches!(event,
|
||||
Event::Status { message } if message.starts_with("Retry attempt: transparent-stream 1/")
|
||||
)));
|
||||
assert!(events.iter().any(|event| matches!(event,
|
||||
assert!(!events.iter().any(|event| matches!(event,
|
||||
Event::Status { message } if message == "Retry recovery: transparent stream recovered after 1 retries"
|
||||
)));
|
||||
let snapshots = events
|
||||
@@ -767,16 +447,16 @@ async fn run_turn_with_stream_open_failures(
|
||||
async fn stream_open_failure_is_retried_through_the_resume_budget() {
|
||||
let (model, status, error, terminal, events) =
|
||||
run_turn_with_stream_open_failures(1, None, StreamOpenFailure::HeaderStall).await;
|
||||
assert_eq!(status, TurnOutcomeStatus::Completed, "{error:?}");
|
||||
assert_eq!(status, TurnOutcomeStatus::Failed, "{error:?}");
|
||||
assert_eq!(
|
||||
model.calls.load(std::sync::atomic::Ordering::SeqCst),
|
||||
2,
|
||||
1,
|
||||
"a request that never opened must be re-issued once"
|
||||
);
|
||||
assert_eq!(terminal.model_requests_started, 2);
|
||||
assert_eq!(terminal.stream_resumes, 1);
|
||||
assert_eq!(terminal.model_requests_started, 1);
|
||||
assert_eq!(terminal.stream_resumes, 0);
|
||||
assert!(
|
||||
!events
|
||||
events
|
||||
.iter()
|
||||
.any(|event| matches!(event, Event::Error { .. })),
|
||||
"a recovered open failure must not surface an error event: {events:?}"
|
||||
@@ -790,10 +470,10 @@ async fn stream_open_failure_fails_the_turn_once_the_budget_is_spent() {
|
||||
assert_eq!(status, TurnOutcomeStatus::Failed);
|
||||
assert_eq!(
|
||||
model.calls.load(std::sync::atomic::Ordering::SeqCst),
|
||||
1 + super::DEFAULT_STREAM_MAX_RESUMES as usize,
|
||||
1,
|
||||
"initial attempt plus the default resume budget, then the turn fails"
|
||||
);
|
||||
assert_eq!(terminal.stream_resumes, super::DEFAULT_STREAM_MAX_RESUMES);
|
||||
assert_eq!(terminal.stream_resumes, 0);
|
||||
assert!(
|
||||
error
|
||||
.as_deref()
|
||||
@@ -835,17 +515,22 @@ async fn stream_open_failure_behind_adapter_context_is_retried() {
|
||||
] {
|
||||
let (model, status, error, terminal, _events) =
|
||||
run_turn_with_stream_open_failures(1, None, open_failure).await;
|
||||
let safe = matches!(open_failure, StreamOpenFailure::WrappedConnectError);
|
||||
assert_eq!(
|
||||
status,
|
||||
TurnOutcomeStatus::Completed,
|
||||
if safe {
|
||||
TurnOutcomeStatus::Completed
|
||||
} else {
|
||||
TurnOutcomeStatus::Failed
|
||||
},
|
||||
"{open_failure:?}: {error:?}"
|
||||
);
|
||||
assert_eq!(
|
||||
model.calls.load(std::sync::atomic::Ordering::SeqCst),
|
||||
2,
|
||||
"{open_failure:?}: a wrapped transport failure must be re-issued"
|
||||
if safe { 2 } else { 1 },
|
||||
"only proven connect failure may be reissued"
|
||||
);
|
||||
assert_eq!(terminal.stream_resumes, 1, "{open_failure:?}");
|
||||
assert_eq!(terminal.stream_resumes, u32::from(safe), "{open_failure:?}");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1065,26 +750,28 @@ async fn transient_error_frame_with_no_content_is_retried_and_terminal_frame_is_
|
||||
canned::simple_text_turn("recovered answer"),
|
||||
])
|
||||
.await;
|
||||
assert_eq!(requests, 2, "the empty-upstream frame is re-issued once");
|
||||
assert!(events.iter().any(|event| matches!(event,
|
||||
assert_eq!(
|
||||
requests, 1,
|
||||
"an admitted error frame must not trigger another POST"
|
||||
);
|
||||
assert!(!events.iter().any(|event| matches!(event,
|
||||
Event::Status { message } if message.starts_with("Retry attempt: stream-resume 1/")
|
||||
)));
|
||||
assert!(events.iter().any(|event| matches!(event,
|
||||
assert!(!events.iter().any(|event| matches!(event,
|
||||
Event::Status { message } if message == "Retry recovery: stream recovered after 1 retries"
|
||||
)));
|
||||
assert!(
|
||||
events.iter().any(|event| matches!(
|
||||
event,
|
||||
Event::TurnComplete {
|
||||
status: TurnOutcomeStatus::Completed,
|
||||
error: None,
|
||||
status: TurnOutcomeStatus::Failed,
|
||||
..
|
||||
}
|
||||
)),
|
||||
"a successful retry completes the turn"
|
||||
);
|
||||
assert!(
|
||||
!events
|
||||
events
|
||||
.iter()
|
||||
.any(|event| matches!(event, Event::Error { .. })),
|
||||
"a retried frame leaves no error card behind"
|
||||
@@ -1121,10 +808,7 @@ async fn exhausted_transient_error_frame_posts_one_error_envelope() {
|
||||
}]
|
||||
};
|
||||
let (events, requests) = error_frame_turn_events((0..8).map(|_| frame()).collect()).await;
|
||||
assert!(
|
||||
(2..8).contains(&requests),
|
||||
"the frame is retried within the budget, then given up on: {requests}"
|
||||
);
|
||||
assert_eq!(requests, 1, "an unknown error frame must not be replayed");
|
||||
let cards: Vec<_> = events
|
||||
.iter()
|
||||
.filter_map(|event| match event {
|
||||
@@ -1442,6 +1126,16 @@ async fn cancellation_after_terminal_usage_still_charges_the_turn() {
|
||||
async fn run_interactive_turn_with_flaky_network(
|
||||
failures: usize,
|
||||
) -> (std::sync::Arc<FlakyNetworkDropModelClient>, Vec<Event>) {
|
||||
let _environment = lock_test_env();
|
||||
let workspace = tempdir().expect("owned stream recovery workspace");
|
||||
let fixture_state = workspace.path().join(".codewhale");
|
||||
fs::create_dir_all(&fixture_state).unwrap();
|
||||
fs::write(
|
||||
fixture_state.join("AGENTS.md"),
|
||||
"Preserve the exact operator input.\n",
|
||||
)
|
||||
.unwrap();
|
||||
let _state_home = EnvVarGuard::set("CODEWHALE_HOME", &fixture_state);
|
||||
let model = std::sync::Arc::new(FlakyNetworkDropModelClient {
|
||||
calls: std::sync::atomic::AtomicUsize::new(0),
|
||||
failures,
|
||||
@@ -1456,9 +1150,11 @@ async fn run_interactive_turn_with_flaky_network(
|
||||
snapshots_enabled: false,
|
||||
subagents_enabled: false,
|
||||
terminal_chrome_enabled: true,
|
||||
..EngineConfig::default()
|
||||
..deterministic_engine_config(workspace.path())
|
||||
};
|
||||
let (engine, handle) = Engine::new_with_model_client(engine_config, &config, client);
|
||||
let (mut engine, handle) = Engine::new_with_model_client(engine_config, &config, client);
|
||||
engine.instruction_settings.home = Some(workspace.path().to_path_buf());
|
||||
engine.refresh_system_prompt();
|
||||
let run_task = tokio::spawn(engine.run());
|
||||
|
||||
handle
|
||||
@@ -1520,8 +1216,8 @@ async fn interactive_turn_preserves_partial_reply_and_recovers_after_network_dro
|
||||
|
||||
assert_eq!(
|
||||
model.calls.load(std::sync::atomic::Ordering::SeqCst),
|
||||
2,
|
||||
"the dropped stream must be re-issued exactly once"
|
||||
1,
|
||||
"an admitted dropped stream must not be replayed"
|
||||
);
|
||||
let (status, error) = events
|
||||
.iter()
|
||||
@@ -1532,32 +1228,33 @@ async fn interactive_turn_preserves_partial_reply_and_recovers_after_network_dro
|
||||
.expect("terminal TurnComplete");
|
||||
assert_eq!(
|
||||
*status,
|
||||
TurnOutcomeStatus::Completed,
|
||||
"a recovered retry must complete the turn: {error:?}"
|
||||
TurnOutcomeStatus::Failed,
|
||||
"an unknown dropped request must fail the turn: {error:?}"
|
||||
);
|
||||
assert!(error.is_none(), "recovered turn must not report an error");
|
||||
assert!(error.is_some(), "unknown outcome must remain visible");
|
||||
assert!(
|
||||
events.iter().any(|event| matches!(
|
||||
!events.iter().any(|event| matches!(
|
||||
event,
|
||||
Event::Status { message } if message.starts_with("Retry attempt: stream-resume 1/")
|
||||
)),
|
||||
"the first actual resume must remain visible: {events:?}"
|
||||
"an uncertain opened stream must not resume automatically: {events:?}"
|
||||
);
|
||||
assert!(
|
||||
!events
|
||||
events
|
||||
.iter()
|
||||
.any(|event| matches!(event, Event::Error { .. })),
|
||||
"a transient drop that the retry recovers must not surface an error event: {events:?}"
|
||||
"an unknown dropped response must surface its error: {events:?}"
|
||||
);
|
||||
|
||||
// The visible fragment must survive as an assistant message, followed by
|
||||
// the retried assistant content — and nothing else. The recovery is
|
||||
// typed internal state, so no synthetic `[runtime]` user turn may appear.
|
||||
// The visible fragment survives as interrupted assistant content without
|
||||
// another provider request or a synthetic `[runtime]` user turn.
|
||||
let transcript = events
|
||||
.iter()
|
||||
.rev()
|
||||
.find_map(|event| match event {
|
||||
Event::SessionUpdated { messages, .. } => Some(messages.clone()),
|
||||
Event::SessionUpdated {
|
||||
source: _,
|
||||
messages, .. } => Some(messages.clone()),
|
||||
_ => None,
|
||||
})
|
||||
.expect("final SessionUpdated");
|
||||
@@ -1569,7 +1266,7 @@ async fn interactive_turn_preserves_partial_reply_and_recovers_after_network_dro
|
||||
ContentBlock::Text { text, .. } => !text.contains("[runtime]"),
|
||||
_ => true,
|
||||
}),
|
||||
"a retried turn must not insert a synthetic [runtime] user message: {transcript:?}"
|
||||
"a failed turn must not insert a synthetic [runtime] user message: {transcript:?}"
|
||||
);
|
||||
assert_eq!(
|
||||
transcript
|
||||
@@ -1579,13 +1276,13 @@ async fn interactive_turn_preserves_partial_reply_and_recovers_after_network_dro
|
||||
1,
|
||||
"the operator's own turn must be the only user message: {transcript:?}"
|
||||
);
|
||||
let assistant_cells = transcript
|
||||
let interrupted_assistant_cells = transcript
|
||||
.iter()
|
||||
.filter(|message| message.role == "assistant")
|
||||
.filter(|message| message.role == Role::InterruptedAssistant)
|
||||
.count();
|
||||
assert_eq!(
|
||||
assistant_cells, 2,
|
||||
"preserved fragment + one authoritative continuation: {transcript:?}"
|
||||
interrupted_assistant_cells, 1,
|
||||
"one preserved incomplete fragment: {transcript:?}"
|
||||
);
|
||||
let transcript_text = transcript
|
||||
.iter()
|
||||
@@ -1602,8 +1299,8 @@ async fn interactive_turn_preserves_partial_reply_and_recovers_after_network_dro
|
||||
);
|
||||
assert_eq!(
|
||||
transcript_text.matches("recovered after retry").count(),
|
||||
1,
|
||||
"exactly one authoritative final answer, not a duplicate: {transcript_text}"
|
||||
0,
|
||||
"no automatic replacement answer: {transcript_text}"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1711,8 +1408,8 @@ impl crate::core::model_client::ModelClient for TransportRetryModelClient {
|
||||
.attempts
|
||||
.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
|
||||
if attempt < self.failures {
|
||||
Err(crate::llm_client::LlmError::ServerError {
|
||||
status: 503,
|
||||
Err(crate::llm_client::LlmError::RateLimited {
|
||||
retry_after: None,
|
||||
message: "RAW-ENGINE-RETRY-FAILURE".into(),
|
||||
})
|
||||
} else {
|
||||
@@ -1844,8 +1541,8 @@ async fn full_retry_receipt_queue_cancellation_does_not_dispatch_an_extra_reques
|
||||
|| {
|
||||
calls += 1;
|
||||
async {
|
||||
Err::<(), _>(crate::llm_client::LlmError::ServerError {
|
||||
status: 503,
|
||||
Err::<(), _>(crate::llm_client::LlmError::RateLimited {
|
||||
retry_after: None,
|
||||
message: "private".into(),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -3,6 +3,16 @@
|
||||
/// authoritative assistant answer in the persisted conversation.
|
||||
#[tokio::test]
|
||||
async fn interactive_thinking_only_drop_preserves_nothing_and_never_claims_it_did() {
|
||||
let _environment = lock_test_env();
|
||||
let workspace = tempdir().expect("owned stream recovery workspace");
|
||||
let fixture_state = workspace.path().join(".codewhale");
|
||||
fs::create_dir_all(&fixture_state).unwrap();
|
||||
fs::write(
|
||||
fixture_state.join("AGENTS.md"),
|
||||
"Preserve the exact operator input.\n",
|
||||
)
|
||||
.unwrap();
|
||||
let _state_home = EnvVarGuard::set("CODEWHALE_HOME", &fixture_state);
|
||||
let model = std::sync::Arc::new(ThinkingOnlyDropModelClient {
|
||||
calls: std::sync::atomic::AtomicUsize::new(0),
|
||||
failures: 1,
|
||||
@@ -14,9 +24,11 @@ async fn interactive_thinking_only_drop_preserves_nothing_and_never_claims_it_di
|
||||
snapshots_enabled: false,
|
||||
subagents_enabled: false,
|
||||
terminal_chrome_enabled: true,
|
||||
..EngineConfig::default()
|
||||
..deterministic_engine_config(workspace.path())
|
||||
};
|
||||
let (engine, handle) = Engine::new_with_model_client(engine_config, &config, client);
|
||||
let (mut engine, handle) = Engine::new_with_model_client(engine_config, &config, client);
|
||||
engine.instruction_settings.home = Some(workspace.path().to_path_buf());
|
||||
engine.refresh_system_prompt();
|
||||
let run_task = tokio::spawn(engine.run());
|
||||
|
||||
handle
|
||||
@@ -72,8 +84,8 @@ async fn interactive_thinking_only_drop_preserves_nothing_and_never_claims_it_di
|
||||
|
||||
assert_eq!(
|
||||
model.calls.load(std::sync::atomic::Ordering::SeqCst),
|
||||
2,
|
||||
"the thinking-only drop must be re-issued exactly once"
|
||||
1,
|
||||
"thinking-only drop must not replay an admitted request"
|
||||
);
|
||||
let status = events
|
||||
.iter()
|
||||
@@ -82,10 +94,10 @@ async fn interactive_thinking_only_drop_preserves_nothing_and_never_claims_it_di
|
||||
_ => None,
|
||||
})
|
||||
.expect("terminal TurnComplete");
|
||||
assert_eq!(status, TurnOutcomeStatus::Completed);
|
||||
assert_eq!(status, TurnOutcomeStatus::Failed);
|
||||
|
||||
assert!(
|
||||
events.iter().any(|event| matches!(
|
||||
!events.iter().any(|event| matches!(
|
||||
event,
|
||||
Event::Status { message } if message.starts_with("Retry attempt: stream-resume 1/")
|
||||
)),
|
||||
@@ -99,7 +111,9 @@ async fn interactive_thinking_only_drop_preserves_nothing_and_never_claims_it_di
|
||||
.iter()
|
||||
.rev()
|
||||
.find_map(|event| match event {
|
||||
Event::SessionUpdated { messages, .. } => Some(messages.clone()),
|
||||
Event::SessionUpdated {
|
||||
source: _,
|
||||
messages, .. } => Some(messages.clone()),
|
||||
_ => None,
|
||||
})
|
||||
.expect("final SessionUpdated");
|
||||
@@ -116,7 +130,7 @@ async fn interactive_thinking_only_drop_preserves_nothing_and_never_claims_it_di
|
||||
.iter()
|
||||
.filter(|message| message.role == "assistant")
|
||||
.count(),
|
||||
1,
|
||||
0,
|
||||
"exactly one authoritative assistant answer after recovery: {transcript:?}"
|
||||
);
|
||||
let transcript_text = transcript
|
||||
@@ -140,7 +154,7 @@ async fn interactive_thinking_only_drop_preserves_nothing_and_never_claims_it_di
|
||||
transcript_text
|
||||
.matches("the one authoritative answer")
|
||||
.count(),
|
||||
1,
|
||||
0,
|
||||
"the recovered answer must be persisted exactly once: {transcript_text}"
|
||||
);
|
||||
}
|
||||
@@ -544,7 +558,7 @@ async fn headless_turn_fails_with_real_error_after_network_drop_budget_exhausted
|
||||
|
||||
assert_eq!(
|
||||
model.calls.load(std::sync::atomic::Ordering::SeqCst),
|
||||
1 + super::DEFAULT_STREAM_MAX_RESUMES as usize,
|
||||
1,
|
||||
"initial attempt plus the bounded resume budget, then the turn fails"
|
||||
);
|
||||
let (status, error) = events
|
||||
@@ -586,14 +600,9 @@ async fn headless_turn_fails_with_real_error_after_network_drop_budget_exhausted
|
||||
Event::Status { message } if message.starts_with("Retry attempt: stream-resume ")
|
||||
))
|
||||
.count(),
|
||||
super::DEFAULT_STREAM_MAX_RESUMES as usize,
|
||||
0,
|
||||
"every admitted resume must have its own numbered progress receipt"
|
||||
);
|
||||
for attempt in 1..=super::DEFAULT_STREAM_MAX_RESUMES {
|
||||
assert!(events.iter().any(|event| matches!(event,
|
||||
Event::Status { message } if message.starts_with(&format!("Retry attempt: stream-resume {attempt}/"))
|
||||
)));
|
||||
}
|
||||
}
|
||||
|
||||
// === Issue #66: error taxonomy wired through engine + audit + capacity ===
|
||||
@@ -871,6 +880,7 @@ fn engine_handle_try_send_does_not_block_when_op_channel_is_full() {
|
||||
// Construct a minimal EngineHandle with the tiny channel.
|
||||
let cancel_token = CancellationToken::new();
|
||||
let handle = EngineHandle {
|
||||
mcp_auth_context: crate::mcp::process_mcp_auth_context(),
|
||||
goal_state: new_shared_goal_state(),
|
||||
sandbox_enforcement: codewhale_sandbox::policy::SandboxEnforcement::Unavailable,
|
||||
tx_op,
|
||||
@@ -1078,6 +1088,14 @@ async fn first_turn_waits_for_explicit_mcp_schema_without_waiting_for_unrelated_
|
||||
return;
|
||||
};
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
let fixture_manager = Arc::new(crate::extension_host::ExtensionHostManager::new(
|
||||
crate::extension_host::ExtensionHostOptions {
|
||||
root: Some(tmp.path().join("host-home")),
|
||||
..Default::default()
|
||||
},
|
||||
));
|
||||
let _fixture_manager_guard =
|
||||
crate::extension_host::TestManagerGuard::install(Arc::clone(&fixture_manager));
|
||||
let server = tmp.path().join("server.mjs");
|
||||
let release = tmp.path().join("release-slow");
|
||||
let release_fast = tmp.path().join("release-fast");
|
||||
@@ -1300,6 +1318,7 @@ readline.createInterface({ input: process.stdin }).on('line', async line => {
|
||||
assert!(result.success);
|
||||
assert!(active.contains("mcp_slow_ready"));
|
||||
engine.wait_for_mcp_boot().await;
|
||||
fixture_manager.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -1317,6 +1336,14 @@ async fn assert_incremental_mcp_boot(invalidate_config: bool) {
|
||||
return;
|
||||
}
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
let fixture_manager = Arc::new(crate::extension_host::ExtensionHostManager::new(
|
||||
crate::extension_host::ExtensionHostOptions {
|
||||
root: Some(tmp.path().join("host-home")),
|
||||
..Default::default()
|
||||
},
|
||||
));
|
||||
let _fixture_manager_guard =
|
||||
crate::extension_host::TestManagerGuard::install(Arc::clone(&fixture_manager));
|
||||
let server = tmp.path().join("server.mjs");
|
||||
let release = tmp.path().join("release-slow");
|
||||
std::fs::write(
|
||||
@@ -1417,6 +1444,7 @@ lines.on('line', async (line) => {
|
||||
drop(events);
|
||||
handle.send(Op::Shutdown).await.expect("shutdown");
|
||||
task.await.expect("engine task");
|
||||
fixture_manager.shutdown().await;
|
||||
let progress = progress.expect("fast server must be visible before slow server is released");
|
||||
assert!(
|
||||
progress
|
||||
@@ -1622,6 +1650,8 @@ async fn mcp_boot_updates_preserve_authority_errors_and_replace_ordinary_errors(
|
||||
async fn mcp_search_fixture(
|
||||
allowed: Option<Vec<String>>,
|
||||
) -> (
|
||||
crate::extension_host::TestManagerGuard,
|
||||
Arc<crate::extension_host::ExtensionHostManager>,
|
||||
Engine,
|
||||
tool_catalog::ToolSurfacePolicy,
|
||||
tempfile::TempDir,
|
||||
@@ -1630,6 +1660,14 @@ async fn mcp_search_fixture(
|
||||
let node =
|
||||
codewhale_dependencies::resolve_node().expect("MCP discovery qualification requires Node");
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
let fixture_manager = Arc::new(crate::extension_host::ExtensionHostManager::new(
|
||||
crate::extension_host::ExtensionHostOptions {
|
||||
root: Some(tmp.path().join("host-home")),
|
||||
..Default::default()
|
||||
},
|
||||
));
|
||||
let fixture_manager_guard =
|
||||
crate::extension_host::TestManagerGuard::install(Arc::clone(&fixture_manager));
|
||||
let server = tmp.path().join("discovery.mjs");
|
||||
fs::write(&server, r#"import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
@@ -1711,12 +1749,20 @@ readline.createInterface({input:process.stdin}).on('line', line => {
|
||||
"discovery",
|
||||
)
|
||||
.await;
|
||||
(engine, build.surface, tmp, model)
|
||||
(
|
||||
fixture_manager_guard,
|
||||
fixture_manager,
|
||||
engine,
|
||||
build.surface,
|
||||
tmp,
|
||||
model,
|
||||
)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mcp_tool_search_discovers_optional_server_real_schema_without_eager_siblings() {
|
||||
let (mut engine, policy, tmp, model) = mcp_search_fixture(None).await;
|
||||
let (_fixture_manager_guard, fixture_manager, mut engine, policy, tmp, model) =
|
||||
mcp_search_fixture(None).await;
|
||||
let mut surface = crate::core::engine::ChildSurfaceProbe {
|
||||
policy,
|
||||
cache: crate::core::session::ToolActivationCache::default(),
|
||||
@@ -1763,11 +1809,12 @@ async fn mcp_tool_search_discovers_optional_server_real_schema_without_eager_sib
|
||||
0,
|
||||
"MCP discovery must not call a provider"
|
||||
);
|
||||
fixture_manager.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mcp_tool_search_respects_captured_turn_ceiling_and_disabled_servers() {
|
||||
let (mut engine, policy, tmp, model) =
|
||||
let (_fixture_manager_guard, fixture_manager, mut engine, policy, tmp, model) =
|
||||
mcp_search_fixture(Some(vec!["tool_search".into()])).await;
|
||||
let mut catalog = policy.catalog.clone();
|
||||
let mut active = policy.active_names.clone();
|
||||
@@ -1790,11 +1837,13 @@ async fn mcp_tool_search_respects_captured_turn_ceiling_and_disabled_servers() {
|
||||
0,
|
||||
"MCP discovery must not call a provider"
|
||||
);
|
||||
fixture_manager.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mcp_tool_search_failed_connection_is_bounded_without_guessed_tools() {
|
||||
let (mut engine, policy, _tmp, model) = mcp_search_fixture(None).await;
|
||||
let (_fixture_manager_guard, fixture_manager, mut engine, policy, _tmp, model) =
|
||||
mcp_search_fixture(None).await;
|
||||
let mut catalog = policy.catalog.clone();
|
||||
let mut active = policy.active_names.clone();
|
||||
tokio::time::timeout(
|
||||
@@ -1821,11 +1870,13 @@ async fn mcp_tool_search_failed_connection_is_bounded_without_guessed_tools() {
|
||||
0,
|
||||
"MCP discovery must not call a provider"
|
||||
);
|
||||
fixture_manager.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mcp_tool_search_cancel_aborts_handshake_and_clears_connecting() {
|
||||
let (mut engine, policy, tmp, model) = mcp_search_fixture(None).await;
|
||||
let (_fixture_manager_guard, fixture_manager, mut engine, policy, tmp, model) =
|
||||
mcp_search_fixture(None).await;
|
||||
let pool = engine.mcp_pool.clone().unwrap();
|
||||
let cancel = engine.cancel_token.clone();
|
||||
let mut catalog = policy.catalog.clone();
|
||||
@@ -1859,4 +1910,5 @@ async fn mcp_tool_search_cancel_aborts_handshake_and_clears_connecting() {
|
||||
0,
|
||||
"MCP discovery must not call a provider"
|
||||
);
|
||||
fixture_manager.shutdown().await;
|
||||
}
|
||||
@@ -432,9 +432,9 @@ pub(super) fn configure_composition_entry(
|
||||
catalog: &mut [Tool],
|
||||
name: Option<&str>,
|
||||
members: &[String],
|
||||
) {
|
||||
) -> bool {
|
||||
if !composition_eager(catalog) {
|
||||
return;
|
||||
return false;
|
||||
}
|
||||
let mut definition = crate::tools::codemode::execute_tools_tool_definition();
|
||||
if !members.is_empty()
|
||||
@@ -469,8 +469,18 @@ pub(super) fn configure_composition_entry(
|
||||
.iter_mut()
|
||||
.find(|t| t.name == EXECUTE_TOOLS_TOOL_NAME)
|
||||
{
|
||||
tool.description = definition.description;
|
||||
tool.input_schema = definition.input_schema;
|
||||
let mut changed = false;
|
||||
if tool.description != definition.description {
|
||||
tool.description = definition.description;
|
||||
changed = true;
|
||||
}
|
||||
if tool.input_schema != definition.input_schema {
|
||||
tool.input_schema = definition.input_schema;
|
||||
changed = true;
|
||||
}
|
||||
changed
|
||||
} else {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -71,6 +71,10 @@ struct PreparedModelStep {
|
||||
}
|
||||
|
||||
pub(super) struct PendingProviderRequestReceipt {
|
||||
pub(super) request_id: String,
|
||||
pub(super) route: crate::cost_status::EffectiveRouteEnvelope,
|
||||
pub(super) primary: bool,
|
||||
pub(super) invoked: bool,
|
||||
pub(super) turn_id: String,
|
||||
pub(super) request_index: u32,
|
||||
pub(super) prefix_check: serde_json::Value,
|
||||
@@ -159,24 +163,16 @@ struct StreamOutcome {
|
||||
pending_message_complete: bool,
|
||||
last_text_index: Option<usize>,
|
||||
stream_errors: u32,
|
||||
terminal_stream_error: bool,
|
||||
steer_preempted: bool,
|
||||
/// Unsettled steers queued mid-stream. Each is committed into the turn's
|
||||
/// record at a step boundary, or dropped — and dropping one reports
|
||||
/// `SteerOutcome::Dropped` to its sender, so an interrupted or failed
|
||||
/// turn cannot silently swallow user guidance (#6276).
|
||||
pending_steers: Vec<handle::PendingSteer>,
|
||||
/// Typed, engine-internal drop-recovery state. `Option` + consume-once
|
||||
/// means one drop schedules exactly one resume; see [`StreamResume`].
|
||||
pending_resume: Option<StreamResume>,
|
||||
stream_start: Instant,
|
||||
first_token_at: Option<Instant>,
|
||||
request_dispatched_at: Instant,
|
||||
stream_error: Option<String>,
|
||||
/// Envelope of a retryable provider error frame (#6795), held back so a
|
||||
/// retry that succeeds leaves no error card. Posted by the caller when the
|
||||
/// request is not re-issued.
|
||||
frame_error: Option<ErrorEnvelope>,
|
||||
}
|
||||
|
||||
pub(super) fn initial_stream_error_user_message(
|
||||
@@ -811,6 +807,8 @@ pub(super) fn replace_runtime_mcp_tools(
|
||||
let changed = *tool_catalog != catalog_before || *active_tool_names != active_before;
|
||||
if changed {
|
||||
super::tool_catalog::configure_composition_entry(tool_catalog, None, &[]);
|
||||
} else {
|
||||
*tool_catalog = catalog_before;
|
||||
}
|
||||
changed
|
||||
}
|
||||
@@ -986,32 +984,13 @@ impl Engine {
|
||||
})
|
||||
}
|
||||
|
||||
/// Post the retryable error frame `process_stream` held back (#6795), once,
|
||||
/// for a turn that ends without re-issuing the request. Every return that
|
||||
/// ends the turn between the stream and the retry decision calls this, so
|
||||
/// no exit leaves the TUI to synthesize its own amber warning instead.
|
||||
///
|
||||
/// Boxed so the several call sites embed a pointer, not this future, in
|
||||
/// the already very large model-step state machine (a debug-build test
|
||||
/// thread has a 2 MiB stack).
|
||||
pub(super) fn post_held_frame_error<'a>(
|
||||
&'a self,
|
||||
frame_error: &'a Option<ErrorEnvelope>,
|
||||
) -> std::pin::Pin<Box<dyn std::future::Future<Output = ()> + Send + 'a>> {
|
||||
Box::pin(async move {
|
||||
if let Some(envelope) = frame_error {
|
||||
let _ = self.send_stream_event(Event::error(envelope.clone())).await;
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// Tell the transcript why a turn stopped on one of Codewhale's own
|
||||
/// ceilings (step count, wall clock). Without an error event the TUI adds
|
||||
/// its own hard-coded amber warning for a failed turn (#6843); this card
|
||||
/// is a budget error that leaves the session online. A child run reports
|
||||
/// through its parent's receipt, so it posts only the status line.
|
||||
///
|
||||
/// Boxed for the same reason as [`Self::post_held_frame_error`].
|
||||
/// Boxed to keep the parent turn future bounded.
|
||||
pub(super) fn post_turn_budget_stop<'a>(
|
||||
&'a self,
|
||||
message: &'a str,
|
||||
@@ -1619,9 +1598,8 @@ impl Engine {
|
||||
// Turn-scoped budget for reasoning-only recovery. Some reasoning models
|
||||
// (and OpenAI-shim routes) close a turn after emitting only hidden
|
||||
// reasoning — a protocol-complete but answerless response that reaches
|
||||
// the failure tail with `stream_errors == 0`, so the transport resume
|
||||
// path above never sees it. A clean stop there is almost always
|
||||
// transient; re-request a bounded number of times before surfacing
|
||||
// the failure tail with `stream_errors == 0`. A clean stop there
|
||||
// permits a bounded answer-recovery request before surfacing
|
||||
// a hard failure. Each retry may incur provider usage and cost.
|
||||
let reasoning_only_reprompts: u32 = 0;
|
||||
// Turn-scoped budget for a clean terminal stop that carried nothing at
|
||||
@@ -1640,13 +1618,9 @@ impl Engine {
|
||||
// status receipt with its exact text (C02-04, "model-visible means
|
||||
// logged"): the runtime event log can reconstruct the request.
|
||||
let reasoning_only_nudge: Option<String> = None;
|
||||
// Outer stream-retry budget: when the chunked-transfer connection
|
||||
// dies mid-stream and either nothing useful was streamed (#103
|
||||
// Phase 3), the host slept mid-turn (#2990), or a host hit a
|
||||
// mid-stream network drop (v0.9.4 Terminal-Bench P0), we re-issue
|
||||
// the request up to `[tui].stream_max_resumes` times (default
|
||||
// DEFAULT_STREAM_MAX_RESUMES) before surfacing the failure to the user. A
|
||||
// stream that never opened (#6699) spends the same budget.
|
||||
// Outer stream-open retry budget: only a typed transient connection
|
||||
// failure proven to precede dispatch can spend it. An opened stream
|
||||
// never reissues its request automatically.
|
||||
// `StreamRetryBudget` enforces that bound in mechanism —
|
||||
// `authorize()` is the only way to spend a resume.
|
||||
let stream_retry_budget =
|
||||
@@ -3260,6 +3234,21 @@ impl Engine {
|
||||
Err(err) => Err(err),
|
||||
};
|
||||
|
||||
let result = preserve_tool_output_before_fanout(
|
||||
result.map(RichToolResult::plain),
|
||||
self.api_provider,
|
||||
&self.session.model,
|
||||
(
|
||||
self.active_route_limits,
|
||||
self.active_context_window.map(|window| window.tokens),
|
||||
),
|
||||
&self.session.id,
|
||||
(&tool_id, &tool_name),
|
||||
self.child_tool_result_token_cap(),
|
||||
)
|
||||
.await
|
||||
.map(RichToolResult::into_result);
|
||||
|
||||
let _ = self
|
||||
.send_event(Event::ToolCallComplete {
|
||||
model_call: plan.model_call.clone(),
|
||||
@@ -3867,6 +3856,7 @@ impl Engine {
|
||||
};
|
||||
let request = crate::core::events::SessionCheckpointRequest::new(caller, snapshot, prefix_sha256, projection, withdraw, reply);
|
||||
let event = Event::SessionUpdated {
|
||||
source: self.session.snapshot_source.clone(),
|
||||
checkpoint: Some(request.clone()),
|
||||
session_id: self.session.id.clone(),
|
||||
messages: self.session.messages.snapshot(),
|
||||
@@ -4683,11 +4673,52 @@ impl Engine {
|
||||
let Some(pending) = pending.take() else {
|
||||
return;
|
||||
};
|
||||
let usage = usage.filter(|usage| usage_has_reported_data(usage));
|
||||
let wire = pending.wire.drain();
|
||||
let source_id = wire
|
||||
.last_transmission
|
||||
.as_ref()
|
||||
.map_or(pending.request_id.as_str(), |attempt| {
|
||||
attempt.source_id.as_str()
|
||||
});
|
||||
let transmission = wire.last_transmission.as_ref().map_or(
|
||||
if pending.invoked {
|
||||
crate::llm_client::RequestTransmission::OutcomeUnknown
|
||||
} else {
|
||||
crate::llm_client::RequestTransmission::NotDispatched
|
||||
},
|
||||
|attempt| attempt.transmission,
|
||||
);
|
||||
let coverage = match transmission {
|
||||
crate::llm_client::RequestTransmission::NotDispatched => {
|
||||
crate::llm_client::ProviderUsageCoverage::NotDispatched
|
||||
}
|
||||
crate::llm_client::RequestTransmission::Rejected => {
|
||||
crate::llm_client::ProviderUsageCoverage::Rejected
|
||||
}
|
||||
crate::llm_client::RequestTransmission::OutcomeUnknown => {
|
||||
if usage.is_some() && matches!(outcome, "completed" | "review_response") {
|
||||
crate::llm_client::ProviderUsageCoverage::Reported
|
||||
} else if usage.is_some() {
|
||||
crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
} else {
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
}
|
||||
}
|
||||
};
|
||||
let _ = self.send_event(Event::ProviderRequestReceipt {
|
||||
receipt: json!({
|
||||
"session_id": self.session.id,
|
||||
"turn_id": pending.turn_id,
|
||||
"request_index": pending.request_index,
|
||||
"request_id": pending.request_id,
|
||||
"source_id": source_id,
|
||||
"route": pending.route.sanitized_for_persistence(),
|
||||
"primary": pending.primary,
|
||||
"transmission": transmission,
|
||||
"coverage": coverage,
|
||||
"physical_requests": wire.transmissions,
|
||||
"physical_requests_dropped": wire.transmissions_dropped,
|
||||
"prefix_check": pending.prefix_check,
|
||||
"reused_preparation": pending.reused_preparation,
|
||||
"wire_observations": wire.observations,
|
||||
@@ -4702,28 +4733,17 @@ impl Engine {
|
||||
}).await;
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn process_stream(
|
||||
&mut self,
|
||||
client: &dyn crate::core::model_client::ModelClient,
|
||||
stream: crate::llm_client::StreamEventBox,
|
||||
stream_request: &codewhale_models::MessageRequest,
|
||||
mut request_dispatched_at: Instant,
|
||||
drop_resumes_spent: u32,
|
||||
diagnostics: &mut crate::tool_inspection::TurnStopDiagnostics,
|
||||
request_dispatched_at: Instant,
|
||||
mut request_receipt: Option<PendingProviderRequestReceipt>,
|
||||
) -> StreamOutcome {
|
||||
// The stream value is itself `Pin<Box<dyn Stream + Send>>`, which
|
||||
// is `Unpin`, so we can rebind it on a transparent retry without
|
||||
// breaking the existing pin invariants.
|
||||
let mut stream = stream;
|
||||
let mut stream_error: Option<String> = None;
|
||||
let mut terminal_stream_error = false;
|
||||
let mut steer_preempted = false;
|
||||
let stream_cancel = self.cancel_token.clone();
|
||||
// #6795: the envelope of a retryable error frame, posted only if the
|
||||
// retry budget does not re-issue the request.
|
||||
let mut frame_error: Option<ErrorEnvelope> = None;
|
||||
|
||||
let mut current_text_raw = String::new();
|
||||
let mut current_text_visible = String::new();
|
||||
@@ -4739,7 +4759,6 @@ impl Engine {
|
||||
// later block must never overwrite an earlier one.
|
||||
let mut completed_thinking: Vec<ContentBlock> = Vec::new();
|
||||
let mut tool_uses: Vec<ToolUseState> = Vec::new();
|
||||
let mut settled_usage = Usage::default();
|
||||
let mut usage = Usage {
|
||||
input_tokens: 0,
|
||||
output_tokens: 0,
|
||||
@@ -4751,9 +4770,6 @@ impl Engine {
|
||||
// must not surface as fabricated zeros.
|
||||
let mut usage_reported = false;
|
||||
let mut attempt_usage_reported = false;
|
||||
let receipt_identity = request_receipt
|
||||
.as_ref()
|
||||
.map(|receipt| (receipt.turn_id.clone(), receipt.prefix_check.clone()));
|
||||
let mut stop_reason: Option<String> = None;
|
||||
let mut current_block_kind: Option<ContentBlockKind> = None;
|
||||
// Map block_index → tool_uses position. Required because the
|
||||
@@ -4772,30 +4788,10 @@ impl Engine {
|
||||
let mut pending_message_complete = false;
|
||||
let mut last_text_index: Option<usize> = None;
|
||||
let mut stream_errors = 0u32;
|
||||
// #103 transparent retry bookkeeping. `any_content_received` flips
|
||||
// on the first actionable content event so we know whether the user
|
||||
// has seen output. Absence of content does not establish zero usage.
|
||||
// This is distinct from the outer drop-resume budget (which
|
||||
// restarts the whole turn-step when a stream died with no
|
||||
// content-block delta delivered to the consumer).
|
||||
let mut any_content_received = false;
|
||||
let mut transparent_stream_retries = 0u32;
|
||||
let mut pending_steers: Vec<handle::PendingSteer> = Vec::new();
|
||||
// `stream_start` is reset on a transparent retry so the wall-clock
|
||||
// budget restarts with the fresh stream.
|
||||
let mut stream_start = Instant::now();
|
||||
// First content-bearing event of this model call, for TTFT.
|
||||
let stream_start = Instant::now();
|
||||
let mut first_token_at: Option<Instant> = None;
|
||||
// #2990 sleep-resume bookkeeping: monotonic and wall-clock stamps
|
||||
// of the last stream progress. `Instant` pauses across a host
|
||||
// suspend while `SystemTime` does not, so a large divergence on
|
||||
// the next error tells "machine slept" apart from "network died".
|
||||
let mut last_progress_mono = Instant::now();
|
||||
let mut last_progress_wall = std::time::SystemTime::now();
|
||||
// Typed drop-recovery state: at most one `StreamResume` is ever
|
||||
// scheduled per stream, and it is consumed exactly once by the
|
||||
// post-loop block. It never becomes a synthetic user message.
|
||||
let mut pending_resume: Option<StreamResume> = None;
|
||||
let mut stream_content_bytes: usize = 0;
|
||||
let (chunk_timeout_secs, chunk_timeout) = stream_chunk_timeout_budget(&self.config);
|
||||
// R1: the per-step stream caps are resolved from config rather than
|
||||
@@ -4805,15 +4801,9 @@ impl Engine {
|
||||
let max_duration = self.config.stream_max_duration;
|
||||
let max_duration_secs = max_duration.as_secs();
|
||||
let max_content_bytes = self.config.stream_max_content_bytes;
|
||||
let mut retry_limits = self.config.stream_retry_limits;
|
||||
let child_request_deadline = self
|
||||
.child_job()
|
||||
.map(|job| request_dispatched_at + job.authority.runtime.step_api_timeout);
|
||||
// Child retries must return through the one phase dispatch boundary,
|
||||
// so each attempt retains its own route/source and usage settlement.
|
||||
if self.child_host.is_some() {
|
||||
retry_limits.max_transparent_retries = 0;
|
||||
}
|
||||
|
||||
// Process stream events
|
||||
loop {
|
||||
@@ -4862,11 +4852,9 @@ impl Engine {
|
||||
bound: Some(chunk_timeout),
|
||||
},
|
||||
);
|
||||
// A stall is a stream error like any other:
|
||||
// count it so the nothing-streamed retry can
|
||||
// fire, and record it so an unrecovered stall
|
||||
// fails the turn with the real reason instead
|
||||
// of ending "Completed" over a frozen block.
|
||||
// A stalled admitted stream keeps its unknown outcome
|
||||
// and fails with the real reason rather than completing
|
||||
// over a frozen block.
|
||||
stream_errors = stream_errors.saturating_add(1);
|
||||
stream_error.get_or_insert(envelope.message.clone());
|
||||
let _ = self.send_stream_event(Event::error(envelope)).await;
|
||||
@@ -4903,8 +4891,6 @@ impl Engine {
|
||||
if let StreamEvent::MessageStart { message } = &e {
|
||||
self.turn_heartbeat.set_provider_request(message.id.clone());
|
||||
}
|
||||
last_progress_mono = Instant::now();
|
||||
last_progress_wall = std::time::SystemTime::now();
|
||||
// Only content-bearing events make a stream productive.
|
||||
// Ping, usage/terminal deltas, block stops, and MessageStop
|
||||
// are protocol bookkeeping; counting them as content hid
|
||||
@@ -4921,265 +4907,13 @@ impl Engine {
|
||||
let message = self.decorate_auth_error_message(e.to_string());
|
||||
let user_message =
|
||||
stream_read_error_user_message(&message, any_content_received);
|
||||
if self.rlm_host.is_none()
|
||||
&& let Some(rejection) = crate::llm_client::image_count_rejection(
|
||||
&format!("{e:#}").to_lowercase(),
|
||||
)
|
||||
{
|
||||
let sent = codewhale_models::image_window::count_request_images(
|
||||
&stream_request.messages,
|
||||
);
|
||||
let model = self.session.model.clone();
|
||||
if self.learn_image_cap(&model, sent, rejection).is_some() {
|
||||
terminal_stream_error = true;
|
||||
stream_error.get_or_insert(user_message.clone());
|
||||
let _ = self
|
||||
.send_stream_event(Event::error(
|
||||
codewhale_error_taxonomy::ErrorEnvelope::image_count_exceeded(
|
||||
user_message,
|
||||
),
|
||||
))
|
||||
.await;
|
||||
break;
|
||||
}
|
||||
}
|
||||
let envelope =
|
||||
crate::llm_error_envelope::envelope_for_llm_error(e, user_message.clone());
|
||||
if self.child_host.is_some() {
|
||||
terminal_stream_error = !envelope.recoverable;
|
||||
stream_error.get_or_insert(user_message);
|
||||
let _ = self.send_stream_event(Event::error(envelope)).await;
|
||||
break;
|
||||
}
|
||||
// Typed account, authorization, and protocol failures cannot
|
||||
// be repaired by sleep recovery or replaying the request.
|
||||
if !envelope.recoverable {
|
||||
terminal_stream_error = true;
|
||||
stream_error.get_or_insert(user_message);
|
||||
let _ = self.send_stream_event(Event::error(envelope)).await;
|
||||
break;
|
||||
}
|
||||
// #2990: wall-clock far ahead of the monotonic clock
|
||||
// since the last chunk means the host slept mid-stream.
|
||||
// The partial output predates the sleep and the user
|
||||
// was not watching — schedule a full request retry in
|
||||
// the post-loop block instead of failing the turn.
|
||||
let wall_elapsed = last_progress_wall
|
||||
.elapsed()
|
||||
.unwrap_or_else(|_| last_progress_mono.elapsed());
|
||||
if should_resume_after_sleep(
|
||||
sleep_gap_detected(last_progress_mono.elapsed(), wall_elapsed),
|
||||
drop_resumes_spent,
|
||||
retry_limits.max_resumes,
|
||||
self.cancel_token.is_cancelled(),
|
||||
) {
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"Stream error after suspected system sleep ({:?} monotonic vs {:?} wall since last chunk); scheduling request retry: {message}",
|
||||
last_progress_mono.elapsed(),
|
||||
wall_elapsed,
|
||||
));
|
||||
// Like the network-drop resumes below, keep the real
|
||||
// error as the prospective outcome: the retry clears
|
||||
// it, and an exhausted resume budget then fails the
|
||||
// turn with it instead of admitting the partial
|
||||
// response as if it had completed.
|
||||
stream_error.get_or_insert(stream_read_error_user_message(
|
||||
&message,
|
||||
any_content_received,
|
||||
));
|
||||
pending_resume = Some(StreamResume::AfterSleep);
|
||||
break;
|
||||
}
|
||||
// #103: when the stream errors before any content was
|
||||
// streamed AND we still have retry budget, transparently
|
||||
// resend the request. The user has seen nothing, but the
|
||||
// provider may already have consumed or billed tokens.
|
||||
if should_transparently_retry_stream(
|
||||
any_content_received,
|
||||
transparent_stream_retries,
|
||||
retry_limits.max_transparent_retries,
|
||||
self.cancel_token.is_cancelled(),
|
||||
) {
|
||||
transparent_stream_retries = transparent_stream_retries.saturating_add(1);
|
||||
codewhale_runtime::logging::info(format!(
|
||||
"Transparent stream retry {transparent_stream_retries}/{} (no content received yet): {message}",
|
||||
retry_limits.max_transparent_retries,
|
||||
));
|
||||
// Drop the failed stream before issuing the new
|
||||
// request to release the underlying connection.
|
||||
let _ = self.send_retry_status(format!(
|
||||
"Retry attempt: transparent-stream {transparent_stream_retries}/{}; stream failed before content",
|
||||
retry_limits.max_transparent_retries
|
||||
)).await;
|
||||
drop(stream);
|
||||
self.settle_provider_request_receipt(
|
||||
&mut request_receipt,
|
||||
attempt_usage_reported.then_some(&usage),
|
||||
"transparent_stream_retry",
|
||||
)
|
||||
.await;
|
||||
request_dispatched_at = Instant::now();
|
||||
let retry_observation = self.request_retry_observation();
|
||||
let transport_retries = retry_observation.retries.clone();
|
||||
request_receipt =
|
||||
receipt_identity.as_ref().map(|(turn_id, prefix_check)| {
|
||||
PendingProviderRequestReceipt {
|
||||
turn_id: turn_id.clone(),
|
||||
request_index: diagnostics
|
||||
.model_requests_started
|
||||
.saturating_add(1),
|
||||
prefix_check: prefix_check.clone(),
|
||||
reused_preparation: true,
|
||||
wire: retry_observation.wire_observations.clone(),
|
||||
transport_retries: transport_retries.clone(),
|
||||
started_at: request_dispatched_at,
|
||||
}
|
||||
});
|
||||
let retry_stream_result = tokio::select! {
|
||||
biased;
|
||||
() = stream_cancel.cancelled() => None,
|
||||
pending = self.wait_turn_steer() => {
|
||||
pending_steers.push(pending);
|
||||
steer_preempted = true;
|
||||
None
|
||||
},
|
||||
result = crate::llm_client::observe_request_retries(Some(retry_observation), async {
|
||||
diagnostics.transparent_stream_retries =
|
||||
diagnostics.transparent_stream_retries.saturating_add(1);
|
||||
diagnostics.model_requests_started =
|
||||
diagnostics.model_requests_started.saturating_add(1);
|
||||
client.create_message_stream(stream_request.clone()).await
|
||||
}) => Some(result),
|
||||
};
|
||||
diagnostics.transport_retries =
|
||||
diagnostics.transport_retries.saturating_add(
|
||||
transport_retries.load(std::sync::atomic::Ordering::Relaxed),
|
||||
);
|
||||
let Some(retry_stream_result) = retry_stream_result else {
|
||||
self.settle_provider_request_receipt(
|
||||
&mut request_receipt,
|
||||
None,
|
||||
if steer_preempted {
|
||||
"steered_before_stream"
|
||||
} else {
|
||||
"cancelled_before_stream"
|
||||
},
|
||||
)
|
||||
.await;
|
||||
break;
|
||||
};
|
||||
match retry_stream_result {
|
||||
Ok(fresh) => {
|
||||
if usage_has_reported_data(&usage) {
|
||||
crate::core::turn::add_usage_to(
|
||||
&mut settled_usage,
|
||||
&std::mem::take(&mut usage),
|
||||
);
|
||||
} else {
|
||||
usage = Usage::default();
|
||||
}
|
||||
attempt_usage_reported = false;
|
||||
stream = fresh;
|
||||
stream_start = Instant::now();
|
||||
// Roll back the error counter — this one
|
||||
// didn't surface to the user.
|
||||
stream_errors = stream_errors.saturating_sub(1);
|
||||
continue;
|
||||
}
|
||||
Err(retry_err) => {
|
||||
self.settle_provider_request_receipt(
|
||||
&mut request_receipt,
|
||||
None,
|
||||
"open_failed",
|
||||
)
|
||||
.await;
|
||||
let retry_msg = self.decorate_auth_error_message(format!(
|
||||
"Stream retry failed: {retry_err}"
|
||||
));
|
||||
stream_error.get_or_insert(retry_msg.clone());
|
||||
let envelope = crate::llm_error_envelope::envelope_for_llm_error(
|
||||
retry_err, retry_msg,
|
||||
);
|
||||
terminal_stream_error = !envelope.recoverable;
|
||||
let _ = self.send_stream_event(Event::error(envelope)).await;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
// Headless hosts (exec / stream-json): a mid-stream
|
||||
// network drop must not forfeit the whole session the
|
||||
// way it does interactively. No operator is watching
|
||||
// the partial deltas, the fragment was never committed
|
||||
// to the conversation, and no tool from the incomplete
|
||||
// response has executed, so break out and let the
|
||||
// post-loop block re-issue the request (bounded by
|
||||
// DEFAULT_STREAM_MAX_RESUMES), exactly like the #2990
|
||||
// sleep-resume. Do NOT emit an error event here: the
|
||||
// exec host forwards every error event onto the
|
||||
// stream-json error channel, and a successful retry
|
||||
// would leave that terminal-looking event on the
|
||||
// stream even though the turn recovered. When the
|
||||
// budget is already exhausted this check is false
|
||||
// and the normal surface-the-error path below runs,
|
||||
// so the final failure is still reported.
|
||||
let network_class_error = matches!(
|
||||
codewhale_error_taxonomy::classify_error_message(&message),
|
||||
ErrorCategory::Network | ErrorCategory::Timeout
|
||||
);
|
||||
if should_resume_after_network_drop(
|
||||
!self.config.terminal_chrome_enabled,
|
||||
network_class_error,
|
||||
drop_resumes_spent,
|
||||
retry_limits.max_resumes,
|
||||
self.cancel_token.is_cancelled(),
|
||||
) {
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"Headless stream resume: network drop after partial content; scheduling request retry: {message}"
|
||||
));
|
||||
// Keep the real error as the prospective turn
|
||||
// outcome; the post-loop retry clears it, and if
|
||||
// the turn still fails the last attempt surfaces
|
||||
// it through the normal path below.
|
||||
stream_error.get_or_insert(stream_read_error_user_message(
|
||||
&message,
|
||||
any_content_received,
|
||||
));
|
||||
pending_resume = Some(StreamResume::HeadlessNetworkDrop);
|
||||
break;
|
||||
}
|
||||
// Interactive TUI: a network/timeout-class stream drop
|
||||
// after partial text (but before any tool call) should
|
||||
// preserve the visible fragment and re-issue the
|
||||
// request, bounded by DEFAULT_STREAM_MAX_RESUMES. This keeps the
|
||||
// turn alive instead of failing with a terminal-looking
|
||||
// error. The resume is typed state — no synthetic user
|
||||
// continuation message is appended.
|
||||
if should_resume_interactive_after_network_drop(
|
||||
self.config.terminal_chrome_enabled,
|
||||
network_class_error,
|
||||
any_content_received,
|
||||
tool_uses.is_empty(),
|
||||
drop_resumes_spent,
|
||||
retry_limits.max_resumes,
|
||||
self.cancel_token.is_cancelled(),
|
||||
) {
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"Interactive stream resume: network drop after partial content; scheduling typed resume: {message}"
|
||||
));
|
||||
stream_error.get_or_insert(stream_read_error_user_message(
|
||||
&message,
|
||||
any_content_received,
|
||||
));
|
||||
pending_resume = Some(StreamResume::InteractiveNetworkDrop);
|
||||
break;
|
||||
}
|
||||
stream_error.get_or_insert(user_message.clone());
|
||||
// Recoverable failures retain their bounded retry tail.
|
||||
stream_error.get_or_insert(user_message);
|
||||
let mut envelope = envelope;
|
||||
envelope.recoverable = false;
|
||||
let _ = self.send_stream_event(Event::error(envelope)).await;
|
||||
if stream_errors >= retry_limits.max_errors {
|
||||
break;
|
||||
}
|
||||
continue;
|
||||
break;
|
||||
}
|
||||
};
|
||||
|
||||
@@ -5547,56 +5281,9 @@ impl Engine {
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"Provider stream error event: {message}"
|
||||
));
|
||||
if self.rlm_host.is_none()
|
||||
&& let Some(rejection) =
|
||||
crate::llm_client::image_count_rejection(&message.to_lowercase())
|
||||
{
|
||||
let sent = codewhale_models::image_window::count_request_images(
|
||||
&stream_request.messages,
|
||||
);
|
||||
let model = self.session.model.clone();
|
||||
if self.learn_image_cap(&model, sent, rejection).is_some() {
|
||||
terminal_stream_error = true;
|
||||
stream_error.get_or_insert(message.to_string());
|
||||
let _ = self
|
||||
.send_stream_event(Event::error(
|
||||
codewhale_error_taxonomy::ErrorEnvelope::image_count_exceeded(
|
||||
message.to_string(),
|
||||
),
|
||||
))
|
||||
.await;
|
||||
break;
|
||||
}
|
||||
}
|
||||
// #6795: a gateway can report a transient upstream failure
|
||||
// as an error frame inside a 200. With nothing actionable
|
||||
// streamed that is a no-content stream death like a
|
||||
// transport error or a stall: count it so the existing
|
||||
// retry budget re-issues the request, and keep it as the
|
||||
// prospective outcome so an exhausted budget fails the
|
||||
// turn with the provider's reason. No error event yet: a
|
||||
// retry that succeeds must not leave a terminal-looking
|
||||
// card behind. Auth, invalid-model and every other class
|
||||
// stays terminal on the first frame, as does any frame
|
||||
// after content (replaying would duplicate side effects).
|
||||
//
|
||||
// Either way the turn-ending envelope is non-recoverable,
|
||||
// so its severity is Error. A retryable frame holds it back
|
||||
// (`frame_error`): the post-loop retry either re-issues the
|
||||
// request and discards it, or the budget is spent and it is
|
||||
// posted then, once, so the card never promises a retry the
|
||||
// engine will not make.
|
||||
stream_errors = stream_errors.saturating_add(1);
|
||||
let envelope = ErrorEnvelope::classify(message.to_string(), false);
|
||||
let transient = matches!(
|
||||
envelope.category,
|
||||
ErrorCategory::Network | ErrorCategory::Timeout
|
||||
);
|
||||
if transient && !any_content_received {
|
||||
stream_errors = stream_errors.saturating_add(1);
|
||||
frame_error = Some(envelope);
|
||||
} else {
|
||||
let _ = self.send_stream_event(Event::error(envelope)).await;
|
||||
}
|
||||
let _ = self.send_stream_event(Event::error(envelope)).await;
|
||||
stream_error.get_or_insert(message.to_string());
|
||||
break;
|
||||
}
|
||||
@@ -5617,31 +5304,11 @@ impl Engine {
|
||||
self.finalize_streamed_tool_input(tool_state).await;
|
||||
}
|
||||
}
|
||||
if transparent_stream_retries > 0 && !steer_preempted {
|
||||
let message = if self.cancel_token.is_cancelled() {
|
||||
"Retry interrupted: transparent stream cancelled".to_string()
|
||||
} else if stream_errors == 0 && pending_message_complete {
|
||||
format!(
|
||||
"Retry recovery: transparent stream recovered after {transparent_stream_retries} retries"
|
||||
)
|
||||
} else if stream_errors > 0
|
||||
&& transparent_stream_retries >= retry_limits.max_transparent_retries
|
||||
{
|
||||
format!(
|
||||
"Retry exhaustion: transparent stream stopped after {transparent_stream_retries} retries; stream did not complete"
|
||||
)
|
||||
} else {
|
||||
format!(
|
||||
"Retry stopped: transparent stream ended after {transparent_stream_retries} retries; completion was not observed"
|
||||
)
|
||||
};
|
||||
let _ = self.send_retry_status(message).await;
|
||||
}
|
||||
let receipt_outcome = if self.cancel_token.is_cancelled() {
|
||||
"cancelled"
|
||||
} else if steer_preempted {
|
||||
"steered"
|
||||
} else if stream_error.is_some() || stream_errors > 0 || pending_resume.is_some() {
|
||||
} else if stream_error.is_some() || stream_errors > 0 {
|
||||
"incomplete"
|
||||
} else if pending_message_complete {
|
||||
"completed"
|
||||
@@ -5654,14 +5321,6 @@ impl Engine {
|
||||
receipt_outcome,
|
||||
)
|
||||
.await;
|
||||
let usage = if usage_has_reported_data(&settled_usage) {
|
||||
if usage_has_reported_data(&usage) {
|
||||
crate::core::turn::add_usage_to(&mut settled_usage, &usage);
|
||||
}
|
||||
settled_usage
|
||||
} else {
|
||||
usage
|
||||
};
|
||||
StreamOutcome {
|
||||
current_text_raw,
|
||||
current_text_visible,
|
||||
@@ -5676,15 +5335,12 @@ impl Engine {
|
||||
pending_message_complete,
|
||||
last_text_index,
|
||||
stream_errors,
|
||||
terminal_stream_error,
|
||||
steer_preempted,
|
||||
pending_steers,
|
||||
pending_resume,
|
||||
stream_start,
|
||||
first_token_at,
|
||||
request_dispatched_at,
|
||||
stream_error,
|
||||
frame_error,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6992,23 +6648,14 @@ mod tests {
|
||||
Ok(canned::message_stop()),
|
||||
]);
|
||||
let request = stream_backpressure_request();
|
||||
let mut diagnostics = crate::tool_inspection::TurnStopDiagnostics::default();
|
||||
let outcome = tokio::time::timeout(
|
||||
Duration::from_secs(1),
|
||||
engine.process_stream(
|
||||
model.as_ref(),
|
||||
Box::pin(stream),
|
||||
&request,
|
||||
Instant::now(),
|
||||
0,
|
||||
&mut diagnostics,
|
||||
None,
|
||||
),
|
||||
engine.process_stream(Box::pin(stream), &request, Instant::now(), None),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(outcome.terminal_stream_error);
|
||||
assert!(outcome.pending_resume.is_none());
|
||||
assert!(outcome.stream_error.is_some());
|
||||
assert_eq!(outcome.stream_errors, 1);
|
||||
assert!(!outcome.pending_message_complete);
|
||||
assert!(outcome.current_text_raw.is_empty());
|
||||
assert_eq!(
|
||||
@@ -7016,7 +6663,6 @@ mod tests {
|
||||
0,
|
||||
"terminal errors must not transparently retry"
|
||||
);
|
||||
assert_eq!(diagnostics.transparent_stream_retries, 0);
|
||||
assert!(
|
||||
matches!(rx.try_recv(), Ok(Event::Error { envelope, .. }) if envelope.code == "llm_quota_exhausted" && !envelope.recoverable)
|
||||
);
|
||||
@@ -7114,16 +6760,8 @@ mod tests {
|
||||
counted.fetch_add(1, Ordering::SeqCst);
|
||||
});
|
||||
let request = stream_backpressure_request();
|
||||
let mut diagnostics = crate::tool_inspection::TurnStopDiagnostics::default();
|
||||
let mut process = Box::pin(engine.process_stream(
|
||||
model.as_ref(),
|
||||
Box::pin(stream),
|
||||
&request,
|
||||
Instant::now(),
|
||||
0,
|
||||
&mut diagnostics,
|
||||
None,
|
||||
));
|
||||
let mut process =
|
||||
Box::pin(engine.process_stream(Box::pin(stream), &request, Instant::now(), None));
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_millis(25), &mut process)
|
||||
.await
|
||||
@@ -7155,10 +6793,6 @@ mod tests {
|
||||
outcome.usage.input_tokens, 17,
|
||||
"{label}: billed usage retained"
|
||||
);
|
||||
assert!(
|
||||
outcome.pending_resume.is_none(),
|
||||
"{label}: cancellation cannot retry"
|
||||
);
|
||||
assert_eq!(model.call_count(), 0, "{label}: no provider retry");
|
||||
assert_eq!(
|
||||
rx.len(),
|
||||
@@ -7181,7 +6815,7 @@ mod tests {
|
||||
use crate::llm_client::mock::canned;
|
||||
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
let (mut engine, model, mut rx) = stream_backpressure_fixture(tmp.path(), 1);
|
||||
let (mut engine, _model, mut rx) = stream_backpressure_fixture(tmp.path(), 1);
|
||||
engine
|
||||
.tx_event
|
||||
.send(Event::status("occupied"))
|
||||
@@ -7199,16 +6833,8 @@ mod tests {
|
||||
)),
|
||||
]);
|
||||
let request = stream_backpressure_request();
|
||||
let mut diagnostics = crate::tool_inspection::TurnStopDiagnostics::default();
|
||||
let mut process = Box::pin(engine.process_stream(
|
||||
model.as_ref(),
|
||||
Box::pin(stream),
|
||||
&request,
|
||||
Instant::now(),
|
||||
0,
|
||||
&mut diagnostics,
|
||||
None,
|
||||
));
|
||||
let mut process =
|
||||
Box::pin(engine.process_stream(Box::pin(stream), &request, Instant::now(), None));
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_millis(25), &mut process)
|
||||
.await
|
||||
@@ -7276,18 +6902,9 @@ mod tests {
|
||||
counted.fetch_add(1, Ordering::SeqCst);
|
||||
});
|
||||
let request = stream_backpressure_request();
|
||||
let mut diagnostics = crate::tool_inspection::TurnStopDiagnostics::default();
|
||||
let outcome = tokio::time::timeout(
|
||||
Duration::from_secs(1),
|
||||
engine.process_stream(
|
||||
model.as_ref(),
|
||||
Box::pin(stream),
|
||||
&request,
|
||||
Instant::now(),
|
||||
0,
|
||||
&mut diagnostics,
|
||||
None,
|
||||
),
|
||||
engine.process_stream(Box::pin(stream), &request, Instant::now(), None),
|
||||
)
|
||||
.await
|
||||
.expect("empty tool starts must be bounded");
|
||||
|
||||
@@ -107,6 +107,10 @@ impl Engine {
|
||||
let retry_observation = self.request_retry_observation();
|
||||
let transport_retries = retry_observation.retries.clone();
|
||||
let mut request_receipt = Some(PendingProviderRequestReceipt {
|
||||
request_id: retry_observation.request_id.clone(),
|
||||
route: client.effective_route_envelope(&stream_request.model, chrono::Utc::now()),
|
||||
primary: self.child_host.is_none() && self.rlm_host.is_none(),
|
||||
invoked: false,
|
||||
turn_id: turn.id.clone(),
|
||||
request_index: turn
|
||||
.stop_diagnostics
|
||||
@@ -136,6 +140,9 @@ impl Engine {
|
||||
let opening =
|
||||
crate::llm_client::observe_request_retries(Some(retry_observation), async {
|
||||
request_started = true;
|
||||
if let Some(receipt) = request_receipt.as_mut() {
|
||||
receipt.invoked = true;
|
||||
}
|
||||
turn.stop_diagnostics.model_requests_started = turn
|
||||
.stop_diagnostics
|
||||
.model_requests_started
|
||||
@@ -300,7 +307,10 @@ impl Engine {
|
||||
e.downcast_ref::<LlmError>(),
|
||||
Some(LlmError::InvalidRequest { status: 413, .. })
|
||||
);
|
||||
if !body_size_rejection
|
||||
let refusal_proven =
|
||||
crate::tools::subagent::engine::provider_request_refusal_proven(&e);
|
||||
if refusal_proven
|
||||
&& !body_size_rejection
|
||||
&& self.rlm_host.is_none()
|
||||
&& let Some(rejection) =
|
||||
crate::llm_client::image_count_rejection(&format!("{e:#}").to_lowercase())
|
||||
@@ -316,7 +326,8 @@ impl Engine {
|
||||
// Recovery/classification keeps its existing input. Expanding
|
||||
// diagnostics must not introduce another model request.
|
||||
let message = self.decorate_auth_error_message(e.to_string());
|
||||
if !body_size_rejection
|
||||
if refusal_proven
|
||||
&& !body_size_rejection
|
||||
&& self.rlm_host.is_none()
|
||||
&& is_context_length_error_message(&message)
|
||||
&& progress.context_recovery_attempts < MAX_CONTEXT_RECOVERY_ATTEMPTS
|
||||
@@ -339,7 +350,8 @@ impl Engine {
|
||||
if let Some(message) = self.report_context_recovery_failure(turn).await {
|
||||
return PhaseResult::Return((TurnOutcomeStatus::Failed, Some(message)));
|
||||
}
|
||||
if !body_size_rejection
|
||||
if refusal_proven
|
||||
&& !body_size_rejection
|
||||
&& self.rlm_host.is_none()
|
||||
&& is_image_input_rejection_message(&message)
|
||||
&& self.active_route_capabilities.image_input != CapabilityState::Unsupported
|
||||
@@ -370,16 +382,6 @@ impl Engine {
|
||||
});
|
||||
let mut envelope =
|
||||
crate::llm_error_envelope::envelope_for_llm_error(e, message.clone());
|
||||
// #6699: the request never became a stream (connect
|
||||
// failure, response-header stall). The transport layer
|
||||
// already spent its own retries; re-issue the identical
|
||||
// request from here through the same bounded resume
|
||||
// budget every other stream failure spends. Nothing
|
||||
// streamed, so there is no fragment to keep or discard,
|
||||
// and no error event is emitted for an attempt that is
|
||||
// retried — an exhausted budget falls through to the
|
||||
// normal failure below. Only a failure with no response
|
||||
// headers qualifies; a provider rejection never does.
|
||||
if self.rlm_host.is_none()
|
||||
&& open_transport_failure
|
||||
&& !self.cancel_token.is_cancelled()
|
||||
@@ -388,7 +390,7 @@ impl Engine {
|
||||
turn.stop_diagnostics.stream_resumes =
|
||||
turn.stop_diagnostics.stream_resumes.saturating_add(1);
|
||||
let _ = self.send_retry_status(format!(
|
||||
"Retry attempt: stream-open {attempt}/{}; connection failed before response headers",
|
||||
"Retry attempt: stream-open {attempt}/{}; connection failed before request dispatch",
|
||||
progress.stream_retry_budget.limit()
|
||||
)).await;
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
@@ -399,7 +401,7 @@ impl Engine {
|
||||
}
|
||||
if open_transport_failure && progress.stream_retry_budget.spent() > 0 {
|
||||
let _ = self.send_retry_status(format!(
|
||||
"Retry exhaustion: stream-open stopped after {} retries; connection failed before response headers",
|
||||
"Retry exhaustion: stream-open stopped after {} retries; connection failed before request dispatch",
|
||||
progress.stream_retry_budget.spent()
|
||||
)).await;
|
||||
}
|
||||
@@ -471,23 +473,17 @@ impl Engine {
|
||||
pending_message_complete,
|
||||
last_text_index,
|
||||
stream_errors,
|
||||
terminal_stream_error,
|
||||
steer_preempted,
|
||||
pending_steers,
|
||||
pending_resume,
|
||||
stream_start,
|
||||
first_token_at,
|
||||
request_dispatched_at,
|
||||
stream_error,
|
||||
frame_error,
|
||||
} = self
|
||||
.process_stream(
|
||||
client.as_ref(),
|
||||
stream,
|
||||
&stream_request,
|
||||
request_dispatched_at,
|
||||
progress.stream_retry_budget.spent(),
|
||||
&mut turn.stop_diagnostics,
|
||||
request_receipt,
|
||||
)
|
||||
.await;
|
||||
@@ -506,8 +502,7 @@ impl Engine {
|
||||
}
|
||||
// C02-05: a response whose stream failed or was steered — a provider error
|
||||
// frame, a transport error, a stall, or a cap — is not a complete
|
||||
// response. Unless the retry below re-issues the request, nothing
|
||||
// it collected may execute or continue the turn.
|
||||
// response. Nothing it collected may execute or continue the turn.
|
||||
let response_stream_failed = steer_preempted || stream_error.is_some();
|
||||
progress.turn_error = progress.turn_error.take().or(stream_error.clone());
|
||||
turn.stop_diagnostics
|
||||
@@ -615,6 +610,10 @@ impl Engine {
|
||||
.await;
|
||||
}
|
||||
|
||||
if response_stream_failed && let Some(state) = self.child_host.as_mut() {
|
||||
state.request_stop_reason = Some("request_outcome_unknown");
|
||||
}
|
||||
|
||||
if let Some(state) = self.rlm_host.as_mut() {
|
||||
state.usage.record_nested_event(serde_json::json!({"run_id": state.run_id, "depth_remaining": match state.mode { rlm_host::RlmMode::Completion => 0, rlm_host::RlmMode::Recursive { depth_remaining } => depth_remaining }, "kind":"code", "content":current_text_visible})).await;
|
||||
if is_incomplete_stop_reason(stop_reason.as_deref())
|
||||
@@ -635,45 +634,12 @@ impl Engine {
|
||||
};
|
||||
self.add_interrupted_assistant_text(¤t_text_visible)
|
||||
.await;
|
||||
self.post_held_frame_error(&frame_error).await;
|
||||
return PhaseResult::Return((TurnOutcomeStatus::Failed, Some(error)));
|
||||
}
|
||||
// Rejected fragments remain in the interrupted Session/code receipt,
|
||||
// but cannot replace the last admitted answer or execute FINAL/REPL.
|
||||
state.last_response = current_text_visible.clone();
|
||||
}
|
||||
if response_stream_failed && self.child_host.is_some() {
|
||||
let error = if progress
|
||||
.turn_error
|
||||
.as_deref()
|
||||
.is_some_and(|reason| reason.contains("child step API request timed out"))
|
||||
{
|
||||
anyhow::Error::new(LlmError::Timeout(
|
||||
child_job
|
||||
.as_ref()
|
||||
.expect("captured child")
|
||||
.authority
|
||||
.runtime
|
||||
.step_api_timeout,
|
||||
))
|
||||
} else {
|
||||
anyhow!(
|
||||
"{}",
|
||||
progress
|
||||
.turn_error
|
||||
.as_deref()
|
||||
.unwrap_or("child stream failed")
|
||||
)
|
||||
};
|
||||
if !terminal_stream_error
|
||||
&& stop_reason
|
||||
.as_deref()
|
||||
.is_none_or(|reason| reason.trim().is_empty())
|
||||
&& let Some(recovery) = self.recover_child_request(progress, &error).await
|
||||
{
|
||||
return recovery;
|
||||
}
|
||||
}
|
||||
if !response_stream_failed {
|
||||
progress.child_request_retries = Default::default();
|
||||
}
|
||||
@@ -760,215 +726,11 @@ impl Engine {
|
||||
)
|
||||
};
|
||||
codewhale_runtime::logging::warn(&error);
|
||||
self.post_held_frame_error(&frame_error).await;
|
||||
return PhaseResult::Return((TurnOutcomeStatus::Failed, Some(error)));
|
||||
}
|
||||
}
|
||||
|
||||
// #103 Phase 3 — transparent retry. The inner loop above bails
|
||||
// when reqwest yields chunk decode errors three times in a row;
|
||||
// most of the time those are recoverable proxy / HTTP/2 issues
|
||||
// and the request can simply be re-issued. Re-issue silently up
|
||||
// to DEFAULT_STREAM_MAX_RESUMES, but only when the stream produced
|
||||
// nothing actionable — if any tool call landed or text was
|
||||
// streamed, ship the partial state to the rest of the turn
|
||||
// pipeline so we don't double-bill the user by re-running it.
|
||||
// The post-content exceptions to that rule are the #2990
|
||||
// sleep-resume and the mid-stream network-drop resumes: those
|
||||
// discard the uncommitted fragment unless an operator watched
|
||||
// visible text land (see `StreamResume::InteractiveNetworkDrop`).
|
||||
//
|
||||
// The resume itself is typed state, consumed here by value, so
|
||||
// one drop schedules exactly one retry; and no resume path
|
||||
// appends a synthetic user message to the persisted
|
||||
// conversation — the retried request is the persisted
|
||||
// conversation re-issued, nothing else.
|
||||
// A flushed redacted block or non-blank thinking is received content,
|
||||
// the same rule `current_thinking` follows.
|
||||
let stream_died_with_nothing = stream_errors > 0
|
||||
&& !terminal_stream_error
|
||||
&& tool_uses.is_empty()
|
||||
&& current_text_visible.trim().is_empty()
|
||||
&& current_thinking.trim().is_empty()
|
||||
&& completed_thinking.iter().all(|block| {
|
||||
matches!(
|
||||
block,
|
||||
ContentBlock::Thinking { thinking, redacted_data: None, .. }
|
||||
if thinking.trim().is_empty()
|
||||
)
|
||||
})
|
||||
&& !pending_message_complete;
|
||||
let pending_resume = match pending_resume {
|
||||
Some(resume) => Some(resume),
|
||||
None if stream_died_with_nothing => Some(StreamResume::NoContentStreamDeath),
|
||||
None => None,
|
||||
};
|
||||
if self.rlm_host.is_none()
|
||||
&& self.child_host.is_none()
|
||||
&& !terminal_stream_error
|
||||
&& let Some(resume) = pending_resume
|
||||
&& let Some(attempt) = progress.stream_retry_budget.authorize()
|
||||
{
|
||||
let limit = progress.stream_retry_budget.limit();
|
||||
turn.stop_diagnostics.stream_resumes =
|
||||
turn.stop_diagnostics.stream_resumes.saturating_add(1);
|
||||
let reason = match resume {
|
||||
StreamResume::AfterSleep => "system sleep interrupted the stream",
|
||||
StreamResume::HeadlessNetworkDrop => "network drop; incomplete reply discarded",
|
||||
StreamResume::InteractiveNetworkDrop => {
|
||||
"network drop; continuing the admitted reply"
|
||||
}
|
||||
StreamResume::NoContentStreamDeath => {
|
||||
"stream ended before a response was completed"
|
||||
}
|
||||
};
|
||||
let _ = self
|
||||
.send_retry_status(format!(
|
||||
"Retry attempt: stream-resume {attempt}/{limit}; {reason}"
|
||||
))
|
||||
.await;
|
||||
match resume {
|
||||
StreamResume::AfterSleep => {
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"Resuming after system sleep (attempt {attempt}/{limit}); discarding partial output and retrying request"
|
||||
));
|
||||
// Finalize any partially-rendered assistant cell so
|
||||
// the retried stream renders fresh instead of
|
||||
// appending to the pre-sleep fragment.
|
||||
if pending_message_complete {
|
||||
let index = last_text_index.unwrap_or(0);
|
||||
let _ = self.send_event(Event::MessageComplete { index }).await;
|
||||
}
|
||||
}
|
||||
StreamResume::HeadlessNetworkDrop => {
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"Resuming headless turn after mid-stream network drop (attempt {attempt}/{limit}); discarding partial output and retrying request"
|
||||
));
|
||||
}
|
||||
StreamResume::InteractiveNetworkDrop => {
|
||||
// Commit the partial assistant message so the retried
|
||||
// request sees the prefix as already delivered. Build
|
||||
// the blocks inline; the outer `content_blocks`
|
||||
// variable is still empty at this point and will be
|
||||
// rebuilt on the next round.
|
||||
let mut resume_blocks: Vec<ContentBlock> = Vec::new();
|
||||
// Thinking blocks the stream already terminated stay in
|
||||
// the retry prefix, in wire order, ahead of the block
|
||||
// that was still open when the stream died.
|
||||
resume_blocks.append(&mut completed_thinking);
|
||||
// A wire-only placeholder must not ride into the
|
||||
// retry prefix as stored reasoning either.
|
||||
let thinking_is_placeholder_only =
|
||||
crate::client::is_reasoning_replay_placeholder(¤t_thinking);
|
||||
if (!current_thinking.is_empty() && !thinking_is_placeholder_only)
|
||||
|| current_thinking_state.is_some()
|
||||
{
|
||||
resume_blocks.push(ContentBlock::Thinking {
|
||||
thinking: current_thinking.clone(),
|
||||
signature: current_thinking_signature.clone(),
|
||||
state: current_thinking_state.clone(),
|
||||
redacted_data: None,
|
||||
});
|
||||
}
|
||||
if !current_text_visible.is_empty() {
|
||||
resume_blocks.push(ContentBlock::Text {
|
||||
text: current_text_visible.clone(),
|
||||
cache_control: None,
|
||||
});
|
||||
}
|
||||
for tool in &tool_uses {
|
||||
resume_blocks.push(ContentBlock::ToolUse {
|
||||
execution_id: Some(tool.execution_id.clone()),
|
||||
id: tool.id.clone(),
|
||||
name: tool.name.clone(),
|
||||
input: tool.input.clone(),
|
||||
caller: tool.caller.clone(),
|
||||
thought_signature: tool.thought_signature.clone(),
|
||||
});
|
||||
}
|
||||
let has_sendable_assistant_content = resume_blocks.iter().any(|block| {
|
||||
matches!(
|
||||
block,
|
||||
ContentBlock::Text { .. } | ContentBlock::ToolUse { .. }
|
||||
)
|
||||
});
|
||||
if !has_sendable_assistant_content {
|
||||
// Thinking-only drop: nothing visible streamed, so
|
||||
// nothing is preserved and nothing is committed.
|
||||
// The re-issued request is identical to the one
|
||||
// that died. Neither the log line nor the status
|
||||
// copy may claim a partial reply was preserved —
|
||||
// that claim is what minted the fake `[runtime]`
|
||||
// user turn in session 1589c05d.
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"Resuming interactive turn after mid-stream network drop (attempt {attempt}/{limit}); only hidden reasoning streamed — no partial reply to preserve, retrying request"
|
||||
));
|
||||
} else {
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"Resuming interactive turn after mid-stream network drop (attempt {attempt}/{limit}); preserving partial reply and retrying request"
|
||||
));
|
||||
// Finalize the partial text cell so the UI stops
|
||||
// streaming and the retried content lands in a
|
||||
// fresh cell instead of appending to an
|
||||
// unfinished one.
|
||||
if let Some(index) = last_text_index {
|
||||
let _ = self.send_event(Event::MessageComplete { index }).await;
|
||||
}
|
||||
// Persist the fragment the operator already saw —
|
||||
// exactly one assistant cell for it, and no
|
||||
// synthetic user turn after it. The retried
|
||||
// request therefore ends with this fragment, which
|
||||
// is the provider-neutral "continue from here"
|
||||
// contract; retry status receipts stay outside the
|
||||
// session messages and provider request history.
|
||||
// They never manufacture a user turn.
|
||||
self.add_session_message(Message {
|
||||
role: Role::Assistant,
|
||||
content: resume_blocks,
|
||||
})
|
||||
.await;
|
||||
}
|
||||
}
|
||||
StreamResume::NoContentStreamDeath => {
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"Stream died with no content (attempt {attempt}/{limit}); retrying request"
|
||||
));
|
||||
}
|
||||
}
|
||||
// Don't preserve the per-stream `turn_error` — we're
|
||||
// about to retry, and a successful retry should not
|
||||
// surface the transient error as the turn outcome.
|
||||
progress.turn_error = None;
|
||||
return PhaseResult::Retry;
|
||||
}
|
||||
// #6795: the request is not being re-issued, so a retryable error
|
||||
// frame that was held back is now the turn's outcome. Post it as the
|
||||
// non-recoverable envelope it was built as (severity Error), once.
|
||||
self.post_held_frame_error(&frame_error).await;
|
||||
if pending_resume.is_some() {
|
||||
if progress.stream_retry_budget.spent() > 0 {
|
||||
let _ = self
|
||||
.send_retry_status(format!(
|
||||
"Retry exhaustion: stream-resume stopped after {} retries; stream interrupted",
|
||||
progress.stream_retry_budget.spent()
|
||||
))
|
||||
.await;
|
||||
}
|
||||
codewhale_runtime::logging::warn(format!(
|
||||
"Stream retry budget exhausted ({} attempts); failing turn",
|
||||
progress.stream_retry_budget.spent()
|
||||
));
|
||||
} else if stream_errors == 0 {
|
||||
if progress.stream_retry_budget.spent() > 0 && pending_message_complete {
|
||||
let _ = self
|
||||
.send_retry_status(format!(
|
||||
"Retry recovery: stream recovered after {} retries",
|
||||
progress.stream_retry_budget.spent()
|
||||
))
|
||||
.await;
|
||||
}
|
||||
// Healthy round → reset retry budget so we don't carry over
|
||||
// state from a previous bad round.
|
||||
if stream_errors == 0 {
|
||||
progress.stream_retry_budget.reset();
|
||||
}
|
||||
|
||||
@@ -1013,8 +775,6 @@ impl Engine {
|
||||
// not-executed result and never reach planning, approval or a
|
||||
// handler; the visible text is kept as an interrupted fragment
|
||||
// and no tool_use enters history, so the transcript stays paired.
|
||||
// A retry never gets here: it `continue`d above with this batch
|
||||
// dropped, and the re-issued request streams its own calls.
|
||||
if response_stream_failed && !tool_uses.is_empty() {
|
||||
let error = progress
|
||||
.turn_error
|
||||
@@ -1141,7 +901,7 @@ impl Engine {
|
||||
});
|
||||
}
|
||||
|
||||
if pending_message_complete {
|
||||
if pending_message_complete && !response_stream_failed {
|
||||
let index = last_text_index.unwrap_or(0);
|
||||
let _ = self.send_event(Event::MessageComplete { index }).await;
|
||||
}
|
||||
@@ -1183,7 +943,11 @@ impl Engine {
|
||||
// Add assistant message to session
|
||||
if has_sendable_assistant_content {
|
||||
self.add_session_message(Message {
|
||||
role: Role::Assistant,
|
||||
role: if response_stream_failed {
|
||||
Role::InterruptedAssistant
|
||||
} else {
|
||||
Role::Assistant
|
||||
},
|
||||
content: content_blocks,
|
||||
})
|
||||
.await;
|
||||
|
||||
@@ -290,6 +290,12 @@ impl SessionCheckpointRequest {
|
||||
return Err("session checkpoint was withdrawn".into());
|
||||
}
|
||||
if self.caller.agent_id.is_some()
|
||||
|| self
|
||||
.snapshot
|
||||
.source
|
||||
.as_ref()
|
||||
.and_then(|source| source.turn_id.as_deref())
|
||||
!= Some(self.caller.turn_id.as_str())
|
||||
|| session_id != self.caller.session_id
|
||||
|| session_id != self.snapshot.session_id
|
||||
|| workspace != self.caller.workspace
|
||||
@@ -489,6 +495,7 @@ pub enum Event {
|
||||
// === Turn Lifecycle ===
|
||||
/// A new turn has started (user sent a message)
|
||||
TurnStarted {
|
||||
previous_source: Option<super::ops::SessionSnapshotSource>,
|
||||
turn_id: String,
|
||||
created_at: DateTime<Utc>,
|
||||
/// Legacy/non-model hosts may still attach a route at start. Model
|
||||
@@ -627,6 +634,7 @@ pub enum Event {
|
||||
|
||||
/// Context compaction started.
|
||||
CompactionStarted {
|
||||
previous_source: Option<super::ops::SessionSnapshotSource>,
|
||||
id: String,
|
||||
auto: bool,
|
||||
message: String,
|
||||
@@ -894,6 +902,7 @@ pub enum Event {
|
||||
/// text block, and that assistant message still has to be persisted for
|
||||
/// later `reasoning_content` replay.
|
||||
SessionUpdated {
|
||||
source: Option<crate::core::ops::SessionSnapshotSource>,
|
||||
checkpoint: Option<Arc<SessionCheckpointRequest>>,
|
||||
session_id: String,
|
||||
/// Shared history snapshot (#6214 T2): the engine hands out an `Arc`
|
||||
|
||||
@@ -6,11 +6,11 @@
|
||||
use crate::compaction::CompactionConfig;
|
||||
use crate::route_runtime::ResolvedRuntimeRoute;
|
||||
use crate::tools::goal::GoalStatus;
|
||||
use codewhale_app_server_protocol::runtime::DynamicToolSpec;
|
||||
use codewhale_config::AppMode;
|
||||
use codewhale_config::ProviderKind;
|
||||
use codewhale_execpolicy::ApprovalMode;
|
||||
use codewhale_models::{Message, SystemPrompt};
|
||||
use codewhale_protocol::runtime::DynamicToolSpec;
|
||||
use std::path::PathBuf;
|
||||
|
||||
/// Prefix used for tool-call ids created by local composer shell shortcuts.
|
||||
@@ -18,10 +18,85 @@ pub const USER_SHELL_TOOL_ID_PREFIX: &str = "user_shell_";
|
||||
|
||||
pub type FleetMemberStarted = Result<String, String>;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)]
|
||||
pub struct SessionControlId(String);
|
||||
|
||||
impl Default for SessionControlId {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
impl SessionControlId {
|
||||
pub fn new() -> Self {
|
||||
Self(uuid::Uuid::new_v4().to_string())
|
||||
}
|
||||
|
||||
pub(crate) fn for_admitted_history(id: String) -> Self {
|
||||
Self(id)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)]
|
||||
pub struct SessionSnapshotSource {
|
||||
pub turn_id: Option<String>,
|
||||
pub submission_id: Option<String>,
|
||||
pub compaction_id: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub control_id: Option<SessionControlId>,
|
||||
}
|
||||
|
||||
impl SessionSnapshotSource {
|
||||
pub fn turn(turn_id: String, submission_id: Option<String>) -> Self {
|
||||
Self {
|
||||
turn_id: Some(turn_id),
|
||||
submission_id,
|
||||
compaction_id: None,
|
||||
control_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn control(control_id: SessionControlId) -> Self {
|
||||
Self {
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
control_id: Some(control_id),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn compaction(compaction_id: String) -> Self {
|
||||
Self {
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: Some(compaction_id),
|
||||
control_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn terminal(
|
||||
turn_id: Option<String>,
|
||||
submission_id: Option<String>,
|
||||
compaction_id: Option<String>,
|
||||
) -> Option<Self> {
|
||||
if turn_id.is_some() == compaction_id.is_some() {
|
||||
return None;
|
||||
}
|
||||
Some(Self {
|
||||
turn_id,
|
||||
submission_id,
|
||||
compaction_id,
|
||||
control_id: None,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Snapshot of session state for saving to disk.
|
||||
/// Returned by `Op::GetSessionSnapshot` via a oneshot channel.
|
||||
#[derive(Debug, Clone, PartialEq, serde::Serialize)]
|
||||
pub struct SessionSnapshot {
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub source: Option<SessionSnapshotSource>,
|
||||
/// The live conversation id this engine session is running under.
|
||||
///
|
||||
/// Every workspace snapshot the conversation takes is tagged with it. A
|
||||
@@ -182,7 +257,7 @@ pub struct TurnSpec {
|
||||
pub max_output_tokens: Option<std::num::NonZeroU32>,
|
||||
pub content: String,
|
||||
/// Inline image bytes validated by Runtime admission; no file references.
|
||||
pub images: Vec<codewhale_protocol::runtime::RuntimeImageInput>,
|
||||
pub images: Vec<codewhale_app_server_protocol::runtime::RuntimeImageInput>,
|
||||
pub mode: AppMode,
|
||||
/// Exact, structurally resolved route authority for this turn. The
|
||||
/// engine activates its client before mutating turn state; injected
|
||||
@@ -385,6 +460,7 @@ pub enum Op {
|
||||
|
||||
/// Sync engine session state (used for resume/load)
|
||||
SyncSession {
|
||||
control: SessionControlId,
|
||||
session_id: Option<String>,
|
||||
messages: Vec<Message>,
|
||||
system_prompt: Option<SystemPrompt>,
|
||||
@@ -398,6 +474,7 @@ pub enum Op {
|
||||
/// compares the full expected state before changing history or caches.
|
||||
/// A rejected rewind returns None and performs no mutation or inference.
|
||||
RewindConversation {
|
||||
control: SessionControlId,
|
||||
expected: Box<SessionSnapshot>,
|
||||
messages: Vec<Message>,
|
||||
tx: tokio::sync::oneshot::Sender<Option<SessionSnapshot>>,
|
||||
|
||||
@@ -594,6 +594,7 @@ pub fn event_to_protocol(event: &Event, ids: &ProtocolIds) -> wire::EventMsg {
|
||||
created_at,
|
||||
route,
|
||||
submission_id,
|
||||
..
|
||||
} => wire::EventMsg::TurnStarted {
|
||||
thread_id,
|
||||
session_id,
|
||||
@@ -698,7 +699,8 @@ pub fn event_to_protocol(event: &Event, ids: &ProtocolIds) -> wire::EventMsg {
|
||||
interrupted: *interrupted,
|
||||
}
|
||||
}
|
||||
Event::CompactionStarted { id, auto, message } => wire::EventMsg::CompactionStarted {
|
||||
Event::CompactionStarted { id, auto, message, ..
|
||||
} => wire::EventMsg::CompactionStarted {
|
||||
thread_id,
|
||||
session_id,
|
||||
id: id.clone(),
|
||||
@@ -993,6 +995,7 @@ pub fn event_to_protocol(event: &Event, ids: &ProtocolIds) -> wire::EventMsg {
|
||||
},
|
||||
},
|
||||
Event::SessionUpdated {
|
||||
source: _,
|
||||
checkpoint: _,
|
||||
session_id: engine_session_id,
|
||||
messages,
|
||||
@@ -1254,6 +1257,7 @@ pub fn op_to_protocol(op: &Op) -> wire_op::Op {
|
||||
model,
|
||||
workspace,
|
||||
mode,
|
||||
..
|
||||
} => wire_op::Op::SyncSession {
|
||||
engine_session_id: session_id.clone(),
|
||||
messages: messages.iter().map(to_value).collect(),
|
||||
@@ -1267,6 +1271,7 @@ pub fn op_to_protocol(op: &Op) -> wire_op::Op {
|
||||
expected,
|
||||
messages,
|
||||
tx: _,
|
||||
..
|
||||
} => wire_op::Op::RewindConversation {
|
||||
expected: to_value(expected),
|
||||
messages: messages.iter().map(to_value).collect(),
|
||||
@@ -1576,6 +1581,7 @@ mod tests {
|
||||
result: Err(ToolError::Timeout { seconds: 9 }),
|
||||
},
|
||||
Event::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "turn-1".into(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -1711,6 +1717,8 @@ mod tests {
|
||||
#[test]
|
||||
fn conditional_rewind_projection_preserves_expected_state_and_reply_owner() {
|
||||
let expected = crate::core::ops::SessionSnapshot {
|
||||
source: None,
|
||||
|
||||
session_id: "observed-conversation".into(),
|
||||
messages: vec![],
|
||||
total_tokens: 9,
|
||||
@@ -1724,6 +1732,7 @@ mod tests {
|
||||
};
|
||||
let (tx, mut receive) = tokio::sync::oneshot::channel();
|
||||
let op = Op::RewindConversation {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
expected: Box::new(expected.clone()),
|
||||
messages: vec![],
|
||||
tx,
|
||||
|
||||
@@ -300,6 +300,7 @@ impl ToolActivationCache {
|
||||
/// Session state for the engine.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Session {
|
||||
pub snapshot_source: Option<crate::core::ops::SessionSnapshotSource>,
|
||||
/// Model being used
|
||||
pub model: String,
|
||||
|
||||
@@ -467,6 +468,7 @@ impl Session {
|
||||
approval_mode: ApprovalMode::Suggest,
|
||||
notes_path,
|
||||
mcp_config_path,
|
||||
snapshot_source: None,
|
||||
id: uuid::Uuid::new_v4().to_string(),
|
||||
project_context: None,
|
||||
last_system_prompt_hash: None,
|
||||
|
||||
@@ -1854,9 +1854,9 @@ pub fn restore_usage_source_ledger(
|
||||
});
|
||||
}
|
||||
|
||||
/// Mark a deleted origin's response handled in its original live generation.
|
||||
/// This suppresses legacy mailbox fallback without putting deleted usage or
|
||||
/// even its fingerprint into the pending pool or a durable session snapshot.
|
||||
/// Mark a captured response settled or retired in its original live generation.
|
||||
/// This suppresses duplicate mailbox pricing without adding money or a routed
|
||||
/// usage record. Durable source membership belongs to the owning projection.
|
||||
pub fn acknowledge_retired_usage_source(scope: CostScopeToken, source_id: &str) {
|
||||
with_pending_state_mut(|state| {
|
||||
if state.generation == scope.0 {
|
||||
|
||||
@@ -12,7 +12,7 @@ use codewhale_plugins::{
|
||||
activation::PluginActivationCapability,
|
||||
manifest::{PluginManifest, ValidatedManifest},
|
||||
};
|
||||
use codewhale_ratatui::avatar::{MAX_PACK_BYTES, MAX_PNG_BYTES, Pack, RegisteredPack};
|
||||
use codewhale_protocol::avatar::{MAX_PACK_BYTES, MAX_PNG_BYTES, Pack, RegisteredPack};
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::{
|
||||
io::Read,
|
||||
@@ -115,7 +115,7 @@ pub(super) async fn admit(
|
||||
if tier != HostTier::Plugin || !params.spec.description.is_empty() {
|
||||
return Err("avatar packs require a reviewed Native bundle and a manifest path".into());
|
||||
}
|
||||
if !codewhale_ratatui::avatar::relative_asset(¶ms.spec.name)
|
||||
if !codewhale_protocol::avatar::relative_asset(¶ms.spec.name)
|
||||
|| !params.spec.name.ends_with(".json")
|
||||
{
|
||||
return Err("avatar manifest must be a bounded bundle-relative JSON path".into());
|
||||
@@ -294,7 +294,7 @@ mod tests {
|
||||
"hash-b",
|
||||
)
|
||||
.unwrap();
|
||||
let pack = codewhale_ratatui::whale_girl::pack().clone();
|
||||
let pack = Pack::parse(br#"{"version":1,"id":"fixture","name":"Fixture","atlases":["atlas.png"],"columns":1,"rows":1,"tileWidth":8,"tileHeight":8,"actions":{"rest":{"frames":[0],"durationsMs":[100],"poster":0}},"states":{"rest":"rest"}}"#).unwrap();
|
||||
// Ownership does not depend on native artwork. Generate a valid page
|
||||
// so the test also works from a package without repository-only PNGs.
|
||||
let mut page = std::io::Cursor::new(Vec::new());
|
||||
|
||||
@@ -1277,6 +1277,11 @@ fn the_wire_result_is_the_tools_text_and_json_and_says_when_it_was_cut() {
|
||||
SessionCheckpointRequest,
|
||||
};
|
||||
let snapshot = Arc::new(crate::core::ops::SessionSnapshot {
|
||||
source: Some(crate::core::ops::SessionSnapshotSource::turn(
|
||||
"turn".into(),
|
||||
None,
|
||||
)),
|
||||
|
||||
session_id: "session".into(),
|
||||
messages: Vec::new(),
|
||||
total_tokens: 0,
|
||||
|
||||
@@ -60,6 +60,7 @@ struct Session {
|
||||
host_generation: u64,
|
||||
name: String,
|
||||
config: McpServerConfig,
|
||||
auth_context: Arc<dyn codewhale_runtime::credentials::AuthContext>,
|
||||
cancel: CancellationToken,
|
||||
broker: AsyncMutex<Option<BrokerSession>>,
|
||||
frames: AsyncMutex<mpsc::Receiver<Value>>,
|
||||
@@ -400,11 +401,12 @@ impl Broker {
|
||||
.command
|
||||
.as_deref()
|
||||
.context("MCP host stdio requires a command")?;
|
||||
let (mut broker, stdout) = BrokerSession::spawn(
|
||||
let (mut broker, stdout) = BrokerSession::spawn_with_auth_context(
|
||||
&session.name,
|
||||
command,
|
||||
&session.config,
|
||||
session.cancel.clone(),
|
||||
Arc::clone(&session.auth_context),
|
||||
)?;
|
||||
session.validate(shared, generation, ¶ms.owner)?;
|
||||
if session
|
||||
@@ -751,6 +753,25 @@ impl SdkTransport {
|
||||
cancel: CancellationToken,
|
||||
timeout: Duration,
|
||||
client: Option<crate::mcp::http_client::McpHttpClient>,
|
||||
) -> Result<Self> {
|
||||
Self::connect_with_http_and_auth_context(
|
||||
name,
|
||||
config,
|
||||
cancel,
|
||||
timeout,
|
||||
client,
|
||||
crate::mcp::process_mcp_auth_context(),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn connect_with_http_and_auth_context(
|
||||
name: &str,
|
||||
config: &McpServerConfig,
|
||||
cancel: CancellationToken,
|
||||
timeout: Duration,
|
||||
client: Option<crate::mcp::http_client::McpHttpClient>,
|
||||
auth_context: Arc<dyn codewhale_runtime::credentials::AuthContext>,
|
||||
) -> Result<Self> {
|
||||
if (config.url.is_some() != client.is_some())
|
||||
|| (config.url.is_none() && config.command.is_none())
|
||||
@@ -783,6 +804,7 @@ impl SdkTransport {
|
||||
host_generation,
|
||||
name: name.to_string(),
|
||||
config: config.clone(),
|
||||
auth_context,
|
||||
// The facade owns caller cancellation. Retiring this transport
|
||||
// must release its resources without cancelling that facade and
|
||||
// masking the concrete reply or failure it is about to receive.
|
||||
|
||||
@@ -14,6 +14,9 @@ use std::collections::{BTreeMap, BTreeSet};
|
||||
use std::fmt::Write as _;
|
||||
use std::path::Path;
|
||||
|
||||
use codewhale_app_server_protocol::schema::{
|
||||
Def, Field, SchemaMode, Ty, innermost, parse_def, quote, ref_name, typescript_defs,
|
||||
};
|
||||
use schemars::generate::SchemaSettings;
|
||||
use schemars::{Schema, SchemaGenerator};
|
||||
|
||||
@@ -59,199 +62,11 @@ fn params_schema(method: &str, generator: &mut SchemaGenerator) -> Schema {
|
||||
}
|
||||
}
|
||||
|
||||
/// A field's wire type, normalized.
|
||||
enum Ty {
|
||||
String,
|
||||
Boolean,
|
||||
Uint,
|
||||
Integer,
|
||||
/// A JSON object with any keys (`Map<String, Value>`).
|
||||
Object,
|
||||
/// Any JSON value.
|
||||
Json,
|
||||
Ref(String),
|
||||
/// A string tag inside a union member.
|
||||
Const(String),
|
||||
Array(Box<Ty>),
|
||||
}
|
||||
|
||||
struct Field {
|
||||
name: String,
|
||||
ty: Ty,
|
||||
required: bool,
|
||||
}
|
||||
|
||||
enum Def {
|
||||
Object { strict: bool, fields: Vec<Field> },
|
||||
Enum(Vec<String>),
|
||||
Union(Vec<Vec<Field>>),
|
||||
}
|
||||
|
||||
fn ref_name(schema: &Value) -> Option<String> {
|
||||
let reference = schema.get("$ref")?.as_str()?;
|
||||
Some(
|
||||
reference
|
||||
.strip_prefix("#/$defs/")
|
||||
.unwrap_or_else(|| panic!("unexpected $ref `{reference}`"))
|
||||
.to_string(),
|
||||
)
|
||||
}
|
||||
|
||||
fn parse_ty(at: &str, schema: &Value) -> Ty {
|
||||
let Some(object) = schema.as_object() else {
|
||||
assert_eq!(schema, &Value::Bool(true), "{at}: unsupported schema");
|
||||
return Ty::Json;
|
||||
};
|
||||
if let Some(name) = ref_name(schema) {
|
||||
return Ty::Ref(name);
|
||||
}
|
||||
// Optional object fields use anyOf(ref, null), while scalar options use a
|
||||
// nullable type array below. Normalize both to the present field's type:
|
||||
// Rust omits None and the host requires a valid value when a field exists.
|
||||
if let Some(variants) = object.get("anyOf").and_then(Value::as_array)
|
||||
&& variants.len() == 2
|
||||
&& variants
|
||||
.iter()
|
||||
.filter(|variant| variant.get("type").and_then(Value::as_str) == Some("null"))
|
||||
.count()
|
||||
== 1
|
||||
{
|
||||
return parse_ty(
|
||||
at,
|
||||
variants
|
||||
.iter()
|
||||
.find(|variant| variant.get("type").and_then(Value::as_str) != Some("null"))
|
||||
.expect("one non-null variant"),
|
||||
);
|
||||
}
|
||||
if let Some(value) = object.get("const").and_then(Value::as_str) {
|
||||
return Ty::Const(value.to_string());
|
||||
}
|
||||
let types: Vec<&str> = match object.get("type") {
|
||||
Some(Value::String(ty)) => vec![ty.as_str()],
|
||||
Some(Value::Array(types)) => types
|
||||
.iter()
|
||||
.filter_map(Value::as_str)
|
||||
.filter(|ty| *ty != "null")
|
||||
.collect(),
|
||||
// Only annotations (`default`, `description`): any value.
|
||||
None if !["enum", "anyOf", "oneOf", "allOf", "properties", "items"]
|
||||
.iter()
|
||||
.any(|key| object.contains_key(*key)) =>
|
||||
{
|
||||
return Ty::Json;
|
||||
}
|
||||
_ => panic!("{at}: unsupported wire schema {schema}"),
|
||||
};
|
||||
match types.as_slice() {
|
||||
["string"] => Ty::String,
|
||||
["boolean"] => Ty::Boolean,
|
||||
["integer"] if object.get("minimum").and_then(Value::as_u64) == Some(0) => Ty::Uint,
|
||||
["integer"] => Ty::Integer,
|
||||
["object"]
|
||||
if object.get("additionalProperties") == Some(&Value::Bool(true))
|
||||
&& !object.contains_key("properties") =>
|
||||
{
|
||||
Ty::Object
|
||||
}
|
||||
["array"] => Ty::Array(Box::new(parse_ty(
|
||||
&format!("{at}[]"),
|
||||
object
|
||||
.get("items")
|
||||
.unwrap_or_else(|| panic!("{at}: array without items")),
|
||||
))),
|
||||
_ => panic!("{at}: unsupported wire schema {schema}"),
|
||||
}
|
||||
}
|
||||
|
||||
fn object_fields(at: &str, schema: &Value) -> Vec<Field> {
|
||||
assert_eq!(
|
||||
schema.get("type").and_then(Value::as_str),
|
||||
Some("object"),
|
||||
"{at}: expected an object schema, got {schema}"
|
||||
);
|
||||
let required: Vec<&str> = schema
|
||||
.get("required")
|
||||
.and_then(Value::as_array)
|
||||
.map(|names| names.iter().filter_map(Value::as_str).collect())
|
||||
.unwrap_or_default();
|
||||
schema
|
||||
.get("properties")
|
||||
.and_then(Value::as_object)
|
||||
.map(|properties| {
|
||||
properties
|
||||
.iter()
|
||||
.map(|(name, schema)| Field {
|
||||
name: name.clone(),
|
||||
ty: parse_ty(&format!("{at}.{name}"), schema),
|
||||
required: required.contains(&name.as_str()),
|
||||
})
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
fn parse_def(name: &str, schema: &Value) -> Def {
|
||||
let members = schema
|
||||
.get("oneOf")
|
||||
.or_else(|| schema.get("anyOf"))
|
||||
.and_then(Value::as_array);
|
||||
if let Some(members) = members {
|
||||
// Unit variants, with or without doc comments: a string enum.
|
||||
let consts: Option<Vec<String>> = members
|
||||
.iter()
|
||||
.map(|member| {
|
||||
member
|
||||
.get("const")
|
||||
.and_then(Value::as_str)
|
||||
.map(str::to_string)
|
||||
})
|
||||
.collect();
|
||||
return match consts {
|
||||
Some(values) => Def::Enum(values),
|
||||
None if members.len() == 1 => Def::Object {
|
||||
strict: members[0].get("additionalProperties") == Some(&Value::Bool(false)),
|
||||
fields: object_fields(name, &members[0]),
|
||||
},
|
||||
None => Def::Union(
|
||||
members
|
||||
.iter()
|
||||
.map(|member| object_fields(name, member))
|
||||
.collect(),
|
||||
),
|
||||
};
|
||||
}
|
||||
if let Some(values) = schema.get("enum").and_then(Value::as_array) {
|
||||
return Def::Enum(
|
||||
values
|
||||
.iter()
|
||||
.map(|value| {
|
||||
value
|
||||
.as_str()
|
||||
.unwrap_or_else(|| panic!("{name}: non-string enum value"))
|
||||
.to_string()
|
||||
})
|
||||
.collect(),
|
||||
);
|
||||
}
|
||||
Def::Object {
|
||||
strict: schema.get("additionalProperties") == Some(&Value::Bool(false)),
|
||||
fields: object_fields(name, schema),
|
||||
}
|
||||
}
|
||||
|
||||
fn quote(value: &str) -> String {
|
||||
assert!(
|
||||
!value.contains(['\'', '\\']),
|
||||
"`{value}` needs escaping in TypeScript"
|
||||
);
|
||||
format!("'{value}'")
|
||||
}
|
||||
|
||||
/// The validator's kind for `ty`: a string enum is inlined, an object is a
|
||||
/// reference into `SHAPES`.
|
||||
fn kind(ty: &Ty, defs: &BTreeMap<String, Def>) -> String {
|
||||
match ty {
|
||||
Ty::Nullable(_) => panic!("host validator does not accept nullable fields"),
|
||||
Ty::String => "'string'".into(),
|
||||
Ty::Boolean => "'boolean'".into(),
|
||||
Ty::Uint => "'uint'".into(),
|
||||
@@ -289,32 +104,6 @@ fn kinds(fields: &[Field], required: bool, defs: &BTreeMap<String, Def>) -> Stri
|
||||
}
|
||||
}
|
||||
|
||||
fn ts(ty: &Ty) -> String {
|
||||
match ty {
|
||||
Ty::String => "string".into(),
|
||||
Ty::Boolean => "boolean".into(),
|
||||
Ty::Uint | Ty::Integer => "number".into(),
|
||||
Ty::Object => "{ [key: string]: Json }".into(),
|
||||
Ty::Json => "Json".into(),
|
||||
Ty::Ref(name) => name.clone(),
|
||||
Ty::Const(value) => quote(value),
|
||||
Ty::Array(item) => format!("{}[]", ts(item)),
|
||||
}
|
||||
}
|
||||
|
||||
/// An array's item type, however deeply nested.
|
||||
fn innermost(ty: &Ty) -> &Ty {
|
||||
match ty {
|
||||
Ty::Array(item) => innermost(item),
|
||||
ty => ty,
|
||||
}
|
||||
}
|
||||
|
||||
fn member(field: &Field) -> String {
|
||||
let optional = if field.required { "" } else { "?" };
|
||||
format!("{}{optional}: {}", field.name, ts(&field.ty))
|
||||
}
|
||||
|
||||
const HEADER: &str = "\
|
||||
// @generated from the Rust protocol types in crates/engine/src/extension_host/protocol.rs
|
||||
// by `extension_host::protocol::tests`. Do not edit: change the Rust side, re-record with
|
||||
@@ -369,7 +158,7 @@ fn render() -> String {
|
||||
let defs: BTreeMap<String, Def> = generator
|
||||
.definitions()
|
||||
.iter()
|
||||
.map(|(name, schema)| (name.clone(), parse_def(name, schema)))
|
||||
.map(|(name, schema)| (name.clone(), parse_def(name, schema, SchemaMode::Host)))
|
||||
.collect();
|
||||
|
||||
// What the validator checks: every params type and the error object,
|
||||
@@ -423,7 +212,7 @@ fn render() -> String {
|
||||
out.push_str(VALIDATOR_TYPES);
|
||||
out.push_str("\n/** Every method's params, and an error response's `error`. */\nexport const SHAPES: { readonly [name: string]: Shape } = {\n");
|
||||
for name in &validated {
|
||||
let Def::Object { strict, fields } = &defs[name] else {
|
||||
let Def::Object { strict, fields, .. } = &defs[name] else {
|
||||
unreachable!("only objects are validated");
|
||||
};
|
||||
let _ = writeln!(out, " {name}: {{");
|
||||
@@ -432,42 +221,8 @@ fn render() -> String {
|
||||
let _ = writeln!(out, " optional: {},", kinds(fields, false, &defs));
|
||||
out.push_str(" },\n");
|
||||
}
|
||||
out.push_str(
|
||||
"}\n\nexport type Json = null | boolean | number | string | Json[] | { [key: string]: Json }\n",
|
||||
);
|
||||
for (name, def) in &defs {
|
||||
out.push('\n');
|
||||
match def {
|
||||
Def::Object { fields, .. } if fields.is_empty() => {
|
||||
let _ = writeln!(out, "export interface {name} {{}}");
|
||||
}
|
||||
Def::Object { fields, .. } => {
|
||||
let _ = writeln!(out, "export interface {name} {{");
|
||||
for field in fields {
|
||||
let _ = writeln!(out, " {}", member(field));
|
||||
}
|
||||
out.push_str("}\n");
|
||||
}
|
||||
Def::Enum(values) => {
|
||||
let values: Vec<String> = values.iter().map(|v| quote(v)).collect();
|
||||
let _ = writeln!(out, "export type {name} = {}", values.join(" | "));
|
||||
}
|
||||
Def::Union(members) => {
|
||||
let members: Vec<String> = members
|
||||
.iter()
|
||||
.map(|fields| {
|
||||
// Tags first, then the variant's fields in order.
|
||||
let (tags, rest): (Vec<&Field>, Vec<&Field>) = fields
|
||||
.iter()
|
||||
.partition(|field| matches!(field.ty, Ty::Const(_)));
|
||||
let parts: Vec<String> = tags.into_iter().chain(rest).map(member).collect();
|
||||
format!("{{ {} }}", parts.join("; "))
|
||||
})
|
||||
.collect();
|
||||
let _ = writeln!(out, "export type {name} = {}", members.join(" | "));
|
||||
}
|
||||
}
|
||||
}
|
||||
out.push_str("}\n");
|
||||
out.push_str(&typescript_defs(&defs, SchemaMode::Host));
|
||||
out
|
||||
}
|
||||
|
||||
|
||||
@@ -723,7 +723,7 @@ impl OwnerRegistry {
|
||||
pub fn register_avatar(
|
||||
&mut self,
|
||||
params: &RegisterParams,
|
||||
pack: codewhale_ratatui::avatar::Pack,
|
||||
pack: codewhale_protocol::avatar::Pack,
|
||||
png: Vec<Vec<u8>>,
|
||||
host_generation: u64,
|
||||
) -> Result<u64, String> {
|
||||
|
||||
@@ -22,7 +22,9 @@
|
||||
|
||||
use anyhow::Result;
|
||||
use codewhale_config::FleetExecConfig;
|
||||
use codewhale_protocol::fleet::{FleetHostSpec, FleetTaskSpec, FleetWorkerEventPayload};
|
||||
use codewhale_protocol::fleet::{
|
||||
FleetHostSpec, FleetReceipt, FleetTaskSpec, FleetWorkerEventPayload,
|
||||
};
|
||||
|
||||
use super::host::{FleetHostAdapter, FleetWorkerCommand};
|
||||
use super::profile::AgentProfile;
|
||||
@@ -551,11 +553,11 @@ pub fn classify_worker_exit(exit_code: Option<i32>, stopped: bool) -> FleetWorke
|
||||
},
|
||||
Some(code) => FleetWorkerEventPayload::Failed {
|
||||
reason: format!("worker exited with code {code}"),
|
||||
recoverable: true,
|
||||
recoverable: false,
|
||||
},
|
||||
None => FleetWorkerEventPayload::Failed {
|
||||
reason: "worker exited without a status code".to_string(),
|
||||
recoverable: true,
|
||||
recoverable: false,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -595,6 +597,8 @@ struct WorkerStream {
|
||||
// corrupts valid UTF-8 when a multibyte code point crosses a read boundary.
|
||||
pending: Vec<u8>,
|
||||
terminal: bool,
|
||||
confirmed_stopped: bool,
|
||||
pending_terminal: Option<PendingFleetTerminal>,
|
||||
terminal_route: TerminalRouteEvidence,
|
||||
/// When this worker process was started, for per-task wall-clock limits (R5).
|
||||
started_at: std::time::Instant,
|
||||
@@ -720,20 +724,25 @@ pub struct FleetWorkerTerminalEvent {
|
||||
pub requires_reported_route: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub(crate) struct FleetPreparedTaskOutcome {
|
||||
pub payload: FleetWorkerEventPayload,
|
||||
pub tail_payloads: Vec<FleetWorkerEventPayload>,
|
||||
pub final_status: Option<super::ledger::FleetTaskLedgerStatus>,
|
||||
pub receipt: FleetReceipt,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
enum PendingFleetTerminal {
|
||||
Observed(FleetWorkerTerminalEvent),
|
||||
Prepared(FleetPreparedTaskOutcome),
|
||||
}
|
||||
|
||||
impl FleetExecutor {
|
||||
pub async fn refresh_tailnet_inventory(&mut self) {
|
||||
let snapshot = tokio::task::spawn_blocking(|| {
|
||||
super::tailnet::TailnetSnapshot::new(
|
||||
super::tailnet::inventory().map_err(|error| error.to_string()),
|
||||
)
|
||||
})
|
||||
.await
|
||||
.unwrap_or_else(|error| {
|
||||
super::tailnet::TailnetSnapshot::new(Err(format!(
|
||||
"Tailnet inventory task failed: {error}"
|
||||
)))
|
||||
});
|
||||
self.tailnet_snapshot = Some(std::sync::Arc::new(snapshot));
|
||||
pub fn refresh_tailnet_inventory(&mut self) {
|
||||
self.tailnet_snapshot = Some(std::sync::Arc::new(super::tailnet::TailnetSnapshot::new(
|
||||
super::tailnet::inventory().map_err(|error| error.to_string()),
|
||||
)));
|
||||
}
|
||||
|
||||
pub fn new(workspace: impl AsRef<std::path::Path>) -> Self {
|
||||
@@ -855,6 +864,8 @@ impl FleetExecutor {
|
||||
offset: 0,
|
||||
pending: Vec::new(),
|
||||
terminal: false,
|
||||
confirmed_stopped: false,
|
||||
pending_terminal: None,
|
||||
terminal_route: TerminalRouteEvidence::default(),
|
||||
started_at: std::time::Instant::now(),
|
||||
final_answer: None,
|
||||
@@ -879,6 +890,79 @@ impl FleetExecutor {
|
||||
.and_then(|stream| stream.attempt.clone())
|
||||
}
|
||||
|
||||
pub(crate) fn confirmed_stopped_attempts(
|
||||
&self,
|
||||
) -> std::collections::BTreeMap<String, FleetExecutorAttempt> {
|
||||
self.streams
|
||||
.iter()
|
||||
.filter_map(|(id, stream)| {
|
||||
stream
|
||||
.confirmed_stopped
|
||||
.then(|| stream.attempt.clone())
|
||||
.flatten()
|
||||
.map(|attempt| (id.clone(), attempt))
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub(crate) fn is_confirmed_stopped(&self, worker_id: &str) -> bool {
|
||||
self.streams
|
||||
.get(worker_id)
|
||||
.is_some_and(|stream| stream.confirmed_stopped)
|
||||
}
|
||||
|
||||
pub(crate) fn prepared_terminal_outcome(
|
||||
&self,
|
||||
worker_id: &str,
|
||||
) -> Option<FleetPreparedTaskOutcome> {
|
||||
match self.streams.get(worker_id)?.pending_terminal.as_ref()? {
|
||||
PendingFleetTerminal::Prepared(outcome) => Some(outcome.clone()),
|
||||
PendingFleetTerminal::Observed(_) => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn retain_prepared_terminal_outcome(
|
||||
&mut self,
|
||||
worker_id: &str,
|
||||
outcome: FleetPreparedTaskOutcome,
|
||||
) -> Result<()> {
|
||||
let stream = self
|
||||
.streams
|
||||
.get_mut(worker_id)
|
||||
.ok_or_else(|| anyhow::anyhow!("Fleet terminal stream is absent"))?;
|
||||
anyhow::ensure!(
|
||||
stream.confirmed_stopped,
|
||||
"Fleet terminal process is not confirmed stopped"
|
||||
);
|
||||
if let Some(attempt) = &stream.attempt {
|
||||
anyhow::ensure!(
|
||||
outcome.receipt.run_id == attempt.run_id
|
||||
&& outcome.receipt.task_id == attempt.task_id
|
||||
&& outcome.receipt.attempt == Some(attempt.attempt)
|
||||
&& outcome.receipt.worker_id == worker_id,
|
||||
"Fleet terminal candidate does not match its tracked attempt"
|
||||
);
|
||||
}
|
||||
stream.pending_terminal = Some(PendingFleetTerminal::Prepared(outcome));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn poll_terminal_for_commit(
|
||||
&mut self,
|
||||
worker_id: &str,
|
||||
) -> Option<FleetWorkerTerminalEvent> {
|
||||
if let Some(pending) = self.streams.get(worker_id)?.pending_terminal.as_ref() {
|
||||
return match pending {
|
||||
PendingFleetTerminal::Observed(event) => Some(event.clone()),
|
||||
PendingFleetTerminal::Prepared(_) => None,
|
||||
};
|
||||
}
|
||||
let event = self.poll_terminal_with_status(worker_id)?;
|
||||
self.streams.get_mut(worker_id)?.pending_terminal =
|
||||
Some(PendingFleetTerminal::Observed(event.clone()));
|
||||
Some(event)
|
||||
}
|
||||
|
||||
/// Wall-clock time this worker process has been running (R5). `None` when
|
||||
/// the worker is not tracked or its exit was already observed: a process
|
||||
/// that has exited is not running, so a deadline must not rewrite the
|
||||
@@ -912,6 +996,9 @@ impl FleetExecutor {
|
||||
} else {
|
||||
self.adapter.stop_worker(worker_id)?;
|
||||
}
|
||||
if let Some(stream) = self.streams.get_mut(worker_id) {
|
||||
stream.confirmed_stopped = true;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -1027,6 +1114,7 @@ impl FleetExecutor {
|
||||
tail_payloads.push(payload);
|
||||
}
|
||||
stream.terminal = true;
|
||||
stream.confirmed_stopped = true;
|
||||
let final_answer = stream.final_answer.take();
|
||||
// Surface the visible final answer on a successful completion so
|
||||
// report/summary tasks (no scorer, no file artifact) show their
|
||||
@@ -1198,6 +1286,8 @@ mod tests {
|
||||
offset: 0,
|
||||
pending: Vec::new(),
|
||||
terminal: false,
|
||||
confirmed_stopped: false,
|
||||
pending_terminal: None,
|
||||
terminal_route: TerminalRouteEvidence::default(),
|
||||
started_at: std::time::Instant::now(),
|
||||
final_answer: None,
|
||||
@@ -1866,13 +1956,15 @@ mod tests {
|
||||
classify_worker_exit(Some(0), false),
|
||||
FleetWorkerEventPayload::Completed { .. }
|
||||
));
|
||||
assert!(matches!(
|
||||
classify_worker_exit(Some(1), false),
|
||||
FleetWorkerEventPayload::Failed {
|
||||
recoverable: true,
|
||||
..
|
||||
}
|
||||
));
|
||||
for exit_code in [Some(1), Some(75), None] {
|
||||
assert!(matches!(
|
||||
classify_worker_exit(exit_code, false),
|
||||
FleetWorkerEventPayload::Failed {
|
||||
recoverable: false,
|
||||
..
|
||||
}
|
||||
));
|
||||
}
|
||||
assert!(matches!(
|
||||
classify_worker_exit(Some(0), true),
|
||||
FleetWorkerEventPayload::Cancelled { .. }
|
||||
|
||||
@@ -153,6 +153,8 @@ pub enum FleetLedgerRecord {
|
||||
/// publishing a terminal task without the receipt that explains it.
|
||||
TaskAttemptFinalized {
|
||||
event: FleetWorkerEvent,
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
tail_events: Vec<FleetWorkerEvent>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
final_status: Option<FleetTaskLedgerStatus>,
|
||||
receipt: Box<FleetReceipt>,
|
||||
@@ -340,6 +342,7 @@ pub struct FleetLedger {
|
||||
ledger_file: WorkspaceFile,
|
||||
lock_file: WorkspaceFile,
|
||||
original_lock: std::fs::File,
|
||||
durable_barrier: parking_lot::Mutex<Option<(std::fs::File, u64)>>,
|
||||
/// Stable advisory-lock inode shared by every manager for this workspace.
|
||||
///
|
||||
/// The ledger itself is replaced during compaction, so locking
|
||||
@@ -369,6 +372,7 @@ impl FleetLedger {
|
||||
ledger_file,
|
||||
lock_file,
|
||||
original_lock,
|
||||
durable_barrier: parking_lot::Mutex::new(None),
|
||||
ledger_path,
|
||||
lock_path,
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
@@ -444,6 +448,7 @@ impl FleetLedger {
|
||||
.ledger_file
|
||||
.open_update(true, true)
|
||||
.with_context(|| format!("opening fleet ledger {}", self.ledger_path.display()))?;
|
||||
self.durable_barrier.lock().take();
|
||||
// A process can die after writing only part of its final JSON record.
|
||||
// O_APPEND would otherwise concatenate the next valid record directly
|
||||
// onto that unterminated tail, causing replay to discard both. Preserve
|
||||
@@ -480,6 +485,8 @@ impl FleetLedger {
|
||||
.with_context(|| format!("flushing fleet ledger {}", self.ledger_path.display()))?;
|
||||
file.sync_data()
|
||||
.with_context(|| format!("syncing fleet ledger {}", self.ledger_path.display()))?;
|
||||
let length = file.metadata()?.len();
|
||||
*self.durable_barrier.lock() = Some((file, length));
|
||||
// Wake SSE subscribers after the bytes are durable (#6211 R7b). Every
|
||||
// record kind funnels through here, so a wake only promises "something
|
||||
// changed" — subscribers re-poll with their cursor.
|
||||
@@ -954,16 +961,40 @@ impl FleetLedger {
|
||||
timestamp: &str,
|
||||
payload: FleetWorkerEventPayload,
|
||||
final_status: Option<FleetTaskLedgerStatus>,
|
||||
receipt: FleetReceipt,
|
||||
) -> Result<Option<FleetWorkerEvent>> {
|
||||
self.finalize_task_attempt_with_tail_if_leased(
|
||||
run_id,
|
||||
worker_id,
|
||||
task_id,
|
||||
expected_attempts,
|
||||
timestamp,
|
||||
payload,
|
||||
Vec::new(),
|
||||
final_status,
|
||||
receipt,
|
||||
)
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub(crate) fn finalize_task_attempt_with_tail_if_leased(
|
||||
&self,
|
||||
run_id: &FleetRunId,
|
||||
worker_id: &str,
|
||||
task_id: &str,
|
||||
expected_attempts: u32,
|
||||
timestamp: &str,
|
||||
payload: FleetWorkerEventPayload,
|
||||
tail_payloads: Vec<FleetWorkerEventPayload>,
|
||||
final_status: Option<FleetTaskLedgerStatus>,
|
||||
mut receipt: FleetReceipt,
|
||||
) -> Result<Option<FleetWorkerEvent>> {
|
||||
if !matches!(
|
||||
&payload,
|
||||
FleetWorkerEventPayload::Completed { .. }
|
||||
| FleetWorkerEventPayload::Failed { .. }
|
||||
| FleetWorkerEventPayload::Cancelled { .. }
|
||||
) {
|
||||
bail!("attempt finalization requires a terminal worker event");
|
||||
}
|
||||
let payload_status = match &payload {
|
||||
FleetWorkerEventPayload::Completed { .. } => FleetTaskLedgerStatus::Completed,
|
||||
FleetWorkerEventPayload::Failed { .. } => FleetTaskLedgerStatus::Failed,
|
||||
FleetWorkerEventPayload::Cancelled { .. } => FleetTaskLedgerStatus::Cancelled,
|
||||
_ => bail!("attempt finalization requires a terminal worker event"),
|
||||
};
|
||||
if final_status.is_some_and(|status| {
|
||||
!matches!(
|
||||
status,
|
||||
@@ -984,28 +1015,126 @@ impl FleetLedger {
|
||||
{
|
||||
bail!("attempt receipt generation does not match its lease");
|
||||
}
|
||||
self.with_write_lock(|| {
|
||||
for tail in &tail_payloads {
|
||||
if matches!(
|
||||
tail,
|
||||
FleetWorkerEventPayload::Completed { .. }
|
||||
| FleetWorkerEventPayload::Failed { .. }
|
||||
| FleetWorkerEventPayload::Cancelled { .. }
|
||||
) {
|
||||
bail!("terminal tail contains another terminal outcome");
|
||||
}
|
||||
}
|
||||
let had_usage = tail_payloads
|
||||
.iter()
|
||||
.any(|payload| matches!(payload, FleetWorkerEventPayload::UsageReport { .. }));
|
||||
let tail_digest = (!tail_payloads.is_empty())
|
||||
.then(|| serde_json::to_vec(&tail_payloads))
|
||||
.transpose()?
|
||||
.map(|bytes| codewhale_runtime::hashing::sha256_hex(&bytes));
|
||||
let appended = self.with_write_lock(|| {
|
||||
let state = self.rebuild_state_unlocked()?;
|
||||
let key = task_key(&run_id.0, task_id);
|
||||
let Some(task) = state.tasks.get(&key) else {
|
||||
return Ok(None);
|
||||
};
|
||||
if task.status != FleetTaskLedgerStatus::Leased
|
||||
|| task.leased_to.as_deref() != Some(worker_id)
|
||||
if task.leased_to.as_deref() != Some(worker_id)
|
||||
|| task.entry.attempts != expected_attempts
|
||||
{
|
||||
return Ok(None);
|
||||
}
|
||||
let event = next_worker_event(&state, run_id, worker_id, task_id, timestamp, payload);
|
||||
let lifecycle_key = event_key(worker_id, &run_id.0, task_id);
|
||||
if task.status != FleetTaskLedgerStatus::Leased {
|
||||
let Some(existing_receipt) = state.receipts.get(&key) else {
|
||||
return Ok(None);
|
||||
};
|
||||
let Some(existing_event) = state.latest_events.get(&lifecycle_key) else {
|
||||
return Ok(None);
|
||||
};
|
||||
receipt.attempt = Some(expected_attempts);
|
||||
receipt.terminal_seq = Some(existing_event.seq);
|
||||
let mut expected_event = next_worker_event(
|
||||
&state,
|
||||
run_id,
|
||||
worker_id,
|
||||
task_id,
|
||||
timestamp,
|
||||
payload.clone(),
|
||||
);
|
||||
expected_event.seq = existing_event.seq;
|
||||
if let Some(digest) = &tail_digest {
|
||||
expected_event
|
||||
.extra
|
||||
.insert("terminal_tail_sha256".into(), Value::String(digest.clone()));
|
||||
}
|
||||
if task.status != final_status.unwrap_or(payload_status)
|
||||
|| serde_json::to_value(existing_receipt)? != serde_json::to_value(&receipt)?
|
||||
|| serde_json::to_value(existing_event)?
|
||||
!= serde_json::to_value(&expected_event)?
|
||||
{
|
||||
return Ok(None);
|
||||
}
|
||||
self.sync_durable_unlocked()
|
||||
.context("retrying exact Fleet terminal durability")?;
|
||||
return Ok(Some(existing_event.clone()));
|
||||
}
|
||||
let mut sequence = state.latest_seq.get(&lifecycle_key).copied().unwrap_or(0);
|
||||
let mut tail_events = Vec::with_capacity(tail_payloads.len());
|
||||
for tail in tail_payloads {
|
||||
let mut event =
|
||||
next_worker_event(&state, run_id, worker_id, task_id, timestamp, tail);
|
||||
sequence = sequence
|
||||
.checked_add(1)
|
||||
.ok_or_else(|| anyhow::anyhow!("Fleet event sequence exhausted"))?;
|
||||
event.seq = sequence;
|
||||
tail_events.push(event);
|
||||
}
|
||||
let mut event =
|
||||
next_worker_event(&state, run_id, worker_id, task_id, timestamp, payload);
|
||||
event.seq = sequence
|
||||
.checked_add(1)
|
||||
.ok_or_else(|| anyhow::anyhow!("Fleet event sequence exhausted"))?;
|
||||
if let Some(digest) = tail_digest {
|
||||
event
|
||||
.extra
|
||||
.insert("terminal_tail_sha256".into(), Value::String(digest));
|
||||
}
|
||||
receipt.attempt = Some(expected_attempts);
|
||||
receipt.terminal_seq = Some(event.seq);
|
||||
self.append_record_unlocked(&FleetLedgerRecord::TaskAttemptFinalized {
|
||||
event: event.clone(),
|
||||
tail_events,
|
||||
final_status,
|
||||
receipt: Box::new(receipt),
|
||||
})?;
|
||||
Ok(Some(event))
|
||||
})
|
||||
});
|
||||
if matches!(&appended, Ok(Some(_))) && had_usage {
|
||||
self.record_budget_exceeded_alert_once(run_id, timestamp)?;
|
||||
}
|
||||
appended
|
||||
}
|
||||
|
||||
pub(crate) fn ensure_durable(&self) -> Result<()> {
|
||||
self.with_write_lock(|| self.sync_durable_unlocked())
|
||||
}
|
||||
|
||||
fn sync_durable_unlocked(&self) -> Result<()> {
|
||||
let file = self.ledger_file.open_update(false, false)?;
|
||||
let length = file.metadata()?.len();
|
||||
let mut cached = self.durable_barrier.lock();
|
||||
if let Some((previous, durable_length)) = cached.take()
|
||||
&& durable_length == length
|
||||
&& same_file(&previous, &file)?
|
||||
{
|
||||
*cached = Some((previous, durable_length));
|
||||
return Ok(());
|
||||
}
|
||||
file.sync_data()
|
||||
.context("acknowledging Fleet ledger durability")?;
|
||||
let length = file.metadata()?.len();
|
||||
*cached = Some((file, length));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Append a scheduler-owned terminal event only if the stale lease version
|
||||
@@ -1770,6 +1899,7 @@ impl FleetLedger {
|
||||
if !contents.is_empty() {
|
||||
contents.push('\n');
|
||||
}
|
||||
self.durable_barrier.lock().take();
|
||||
self.ledger_file
|
||||
.replace(contents.as_bytes())
|
||||
.with_context(|| {
|
||||
@@ -1871,10 +2001,38 @@ fn runtime_events_from_record(
|
||||
worker_seq: None,
|
||||
payload: json!({ "status": status }),
|
||||
}],
|
||||
FleetLedgerRecord::TaskAttemptFinalized { event, receipt, .. } => vec![
|
||||
runtime_worker_event("worker", ledger_ordinal, replay_epoch, event),
|
||||
runtime_receipt_event("receipt", ledger_ordinal, replay_epoch, receipt),
|
||||
],
|
||||
FleetLedgerRecord::TaskAttemptFinalized {
|
||||
event,
|
||||
tail_events,
|
||||
receipt,
|
||||
..
|
||||
} => {
|
||||
let mut events = tail_events
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, event)| {
|
||||
runtime_worker_event(
|
||||
&format!("tail-{index}"),
|
||||
ledger_ordinal,
|
||||
replay_epoch,
|
||||
event,
|
||||
)
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
events.push(runtime_worker_event(
|
||||
"worker",
|
||||
ledger_ordinal,
|
||||
replay_epoch,
|
||||
event,
|
||||
));
|
||||
events.push(runtime_receipt_event(
|
||||
"receipt",
|
||||
ledger_ordinal,
|
||||
replay_epoch,
|
||||
receipt,
|
||||
));
|
||||
events
|
||||
}
|
||||
FleetLedgerRecord::EventAppended { event } => {
|
||||
vec![runtime_worker_event(
|
||||
"worker",
|
||||
@@ -2287,9 +2445,13 @@ fn apply_record(state: &mut FleetLedgerState, record: FleetLedgerRecord) {
|
||||
}
|
||||
FleetLedgerRecord::TaskAttemptFinalized {
|
||||
event,
|
||||
tail_events,
|
||||
final_status,
|
||||
receipt,
|
||||
} => {
|
||||
for tail in tail_events {
|
||||
apply_record(state, FleetLedgerRecord::EventAppended { event: tail });
|
||||
}
|
||||
let run_id = event.run_id.clone();
|
||||
let task_id = event.task_id.clone();
|
||||
let worker_id = event.worker_id.clone();
|
||||
|
||||
+541
-319
File diff suppressed because it is too large.
Load diff
@@ -548,15 +548,14 @@ pub fn fleet_write_roots(task_spec: &FleetTaskSpec) -> Result<Vec<String>> {
|
||||
(".", path) | (path, ".") => path.to_string(),
|
||||
(root, path) => format!("{root}/{path}"),
|
||||
};
|
||||
if !roots.contains(&claim_root) {
|
||||
roots.push(claim_root);
|
||||
}
|
||||
roots.push(claim_root);
|
||||
}
|
||||
Ok(roots)
|
||||
}
|
||||
|
||||
pub fn fleet_runtime_write_roots(task_spec: &FleetTaskSpec) -> Result<Vec<String>> {
|
||||
let mut roots = Vec::new();
|
||||
let mut seen = std::collections::BTreeSet::new();
|
||||
for path in task_spec
|
||||
.workspace
|
||||
.as_ref()
|
||||
@@ -565,7 +564,7 @@ pub fn fleet_runtime_write_roots(task_spec: &FleetTaskSpec) -> Result<Vec<String
|
||||
{
|
||||
let normalized =
|
||||
normalize_fleet_relative_path(path, &task_spec.id, "workspace.writable_paths")?;
|
||||
if !roots.contains(&normalized) {
|
||||
if seen.insert(normalized.clone()) {
|
||||
roots.push(normalized);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,7 +43,7 @@
|
||||
//! readable message beats letting one through to a provider-side 400.
|
||||
|
||||
use anyhow::{Result, bail};
|
||||
use codewhale_protocol::runtime::{
|
||||
use codewhale_app_server_protocol::runtime::{
|
||||
MAX_RUNTIME_IMAGE_BYTES, MAX_RUNTIME_IMAGE_TOTAL_BYTES, MAX_RUNTIME_IMAGES, RuntimeImageInput,
|
||||
};
|
||||
use image::codecs::jpeg::JpegEncoder;
|
||||
|
||||
@@ -10,7 +10,9 @@ pub const PNG_1X1: &[u8] = &[
|
||||
0x42, 0x60, 0x82,
|
||||
];
|
||||
|
||||
pub fn runtime_image_fixture(color: u8) -> codewhale_protocol::runtime::RuntimeImageInput {
|
||||
pub fn runtime_image_fixture(
|
||||
color: u8,
|
||||
) -> codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
let image = image::DynamicImage::ImageRgba8(image::RgbaImage::from_pixel(
|
||||
2,
|
||||
2,
|
||||
@@ -18,13 +20,15 @@ pub fn runtime_image_fixture(color: u8) -> codewhale_protocol::runtime::RuntimeI
|
||||
));
|
||||
let mut bytes = std::io::Cursor::new(Vec::new());
|
||||
image.write_to(&mut bytes, image::ImageFormat::Png).unwrap();
|
||||
codewhale_protocol::runtime::RuntimeImageInput {
|
||||
codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
mime: "image/png".into(),
|
||||
data_base64: STANDARD.encode(bytes.into_inner()),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn runtime_image_fixture_bytes(size: usize) -> codewhale_protocol::runtime::RuntimeImageInput {
|
||||
pub fn runtime_image_fixture_bytes(
|
||||
size: usize,
|
||||
) -> codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
let mut image = runtime_image_fixture(42);
|
||||
let mut bytes = STANDARD.decode(&image.data_base64).unwrap();
|
||||
bytes.resize(size, 0);
|
||||
|
||||
@@ -661,20 +661,20 @@ fn runtime_image_validation_preserves_exact_bytes_and_rejects_corruption() {
|
||||
std::slice::from_ref(&input)
|
||||
);
|
||||
for bad in [
|
||||
codewhale_protocol::runtime::RuntimeImageInput {
|
||||
codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
mime: "image/jpeg".into(),
|
||||
..input.clone()
|
||||
},
|
||||
codewhale_protocol::runtime::RuntimeImageInput {
|
||||
codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
data_base64: "not base64".into(),
|
||||
..input.clone()
|
||||
},
|
||||
codewhale_protocol::runtime::RuntimeImageInput {
|
||||
codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
data_base64: String::new(),
|
||||
..input.clone()
|
||||
},
|
||||
// Existing signature sniffing alone accepted this truncated PNG.
|
||||
codewhale_protocol::runtime::RuntimeImageInput {
|
||||
codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
data_base64: STANDARD.encode(b"\x89PNG\r\n\x1a\n"),
|
||||
..input.clone()
|
||||
},
|
||||
@@ -688,16 +688,18 @@ fn runtime_image_validation_bounds_count_encoded_size_and_decode_dimensions() {
|
||||
let input = runtime_image_fixture(7);
|
||||
assert!(prepare_runtime_images(&vec![input.clone(); 11]).is_err());
|
||||
assert!(
|
||||
prepare_runtime_images(&[codewhale_protocol::runtime::RuntimeImageInput {
|
||||
data_base64: "A".repeat(MAX_IMAGE_BYTES.div_ceil(3) * 4 + 1),
|
||||
..input
|
||||
}])
|
||||
prepare_runtime_images(
|
||||
&[codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
data_base64: "A".repeat(MAX_IMAGE_BYTES.div_ceil(3) * 4 + 1),
|
||||
..input
|
||||
}]
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
let wide = image::DynamicImage::ImageRgba8(image::RgbaImage::new(MAX_IMAGE_DIMENSION + 1, 1));
|
||||
let mut bytes = std::io::Cursor::new(Vec::new());
|
||||
wide.write_to(&mut bytes, image::ImageFormat::Png).unwrap();
|
||||
let input = codewhale_protocol::runtime::RuntimeImageInput {
|
||||
let input = codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
mime: "image/png".into(),
|
||||
data_base64: STANDARD.encode(bytes.into_inner()),
|
||||
};
|
||||
|
||||
@@ -363,11 +363,36 @@ impl QuotaExhaustionError {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum RequestTransmission {
|
||||
NotDispatched,
|
||||
Rejected,
|
||||
OutcomeUnknown,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ProviderUsageCoverage {
|
||||
Reported,
|
||||
Missing,
|
||||
Incomplete,
|
||||
NotDispatched,
|
||||
Rejected,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
pub enum RetryOperation {
|
||||
#[default]
|
||||
Inference,
|
||||
ReadOnly,
|
||||
}
|
||||
|
||||
/// Classified LLM errors with retryability information.
|
||||
///
|
||||
/// This enum categorizes API errors to enable smart retry decisions.
|
||||
/// Some errors (rate limits, transient server errors) are retryable,
|
||||
/// while others (auth failures, invalid requests) should fail immediately.
|
||||
/// Only authoritative rate limits and proven transient pre-dispatch failures
|
||||
/// permit an inference retry.
|
||||
#[derive(Debug)]
|
||||
pub enum LlmError {
|
||||
/// Rate limit exceeded (HTTP 429)
|
||||
@@ -390,6 +415,12 @@ pub enum LlmError {
|
||||
/// Network connectivity error
|
||||
NetworkError(String),
|
||||
|
||||
ConnectError {
|
||||
message: String,
|
||||
transient: bool,
|
||||
timed_out: bool,
|
||||
},
|
||||
|
||||
/// Request timed out
|
||||
Timeout(Duration),
|
||||
|
||||
@@ -429,6 +460,9 @@ impl std::fmt::Display for LlmError {
|
||||
write!(f, "Server error ({status}): {message}")
|
||||
}
|
||||
LlmError::NetworkError(msg) => write!(f, "Network error: {msg}"),
|
||||
LlmError::ConnectError { message, .. } => {
|
||||
write!(f, "Connection failed before request dispatch: {message}")
|
||||
}
|
||||
LlmError::Timeout(d) => write!(f, "Request timed out after {d:?}"),
|
||||
LlmError::AuthenticationError(auth) => {
|
||||
write!(f, "Authentication failed: {}", auth.to_user_message())
|
||||
@@ -449,28 +483,38 @@ impl std::fmt::Display for LlmError {
|
||||
impl std::error::Error for LlmError {}
|
||||
|
||||
impl LlmError {
|
||||
/// Determines if this error is potentially transient and worth retrying.
|
||||
///
|
||||
/// Retryable errors:
|
||||
/// - Rate limits (with backoff)
|
||||
/// - Server errors (5xx)
|
||||
/// - Network errors (connection issues)
|
||||
/// - Timeouts
|
||||
///
|
||||
/// Non-retryable errors:
|
||||
/// - Provider plan quota exhaustion
|
||||
/// - Authentication failures
|
||||
/// - Invalid requests
|
||||
/// - Content policy violations
|
||||
/// - Context length errors
|
||||
pub fn transmission(&self) -> RequestTransmission {
|
||||
match self {
|
||||
Self::ConnectError { .. } => RequestTransmission::NotDispatched,
|
||||
Self::RateLimited { .. }
|
||||
| Self::QuotaExhausted(_)
|
||||
| Self::AuthenticationError(_)
|
||||
| Self::AuthorizationError(_)
|
||||
| Self::InvalidRequest { .. }
|
||||
| Self::ModelError(_)
|
||||
| Self::ContentPolicyError(_)
|
||||
| Self::ContextLengthError(_) => RequestTransmission::Rejected,
|
||||
_ => RequestTransmission::OutcomeUnknown,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_retryable(&self) -> bool {
|
||||
self.is_retryable_for(RetryOperation::Inference)
|
||||
}
|
||||
|
||||
pub fn is_retryable_for(&self, operation: RetryOperation) -> bool {
|
||||
matches!(
|
||||
self,
|
||||
LlmError::RateLimited { .. }
|
||||
| LlmError::ServerError { .. }
|
||||
| LlmError::NetworkError(_)
|
||||
| LlmError::Timeout(_)
|
||||
)
|
||||
Self::RateLimited { .. }
|
||||
| Self::ConnectError {
|
||||
transient: true,
|
||||
..
|
||||
}
|
||||
) || (operation == RetryOperation::ReadOnly
|
||||
&& matches!(
|
||||
self,
|
||||
Self::ServerError { .. } | Self::NetworkError(_) | Self::Timeout(_)
|
||||
))
|
||||
}
|
||||
|
||||
/// Returns the server-suggested retry delay if available.
|
||||
@@ -504,6 +548,16 @@ impl LlmError {
|
||||
retry_after: Option<Duration>,
|
||||
body_retry_after: Option<Duration>,
|
||||
) -> Self {
|
||||
if (499..=599).contains(&status) {
|
||||
return Self::ServerError {
|
||||
status,
|
||||
message: body.to_string(),
|
||||
};
|
||||
}
|
||||
if !matches!(status, 400 | 401 | 402 | 403 | 404 | 413 | 429) {
|
||||
return Self::Other(format!("HTTP {status}: {body}"));
|
||||
}
|
||||
|
||||
if status == 413 {
|
||||
return LlmError::InvalidRequest {
|
||||
status,
|
||||
@@ -585,15 +639,6 @@ impl LlmError {
|
||||
}
|
||||
}
|
||||
}
|
||||
// Several OpenAI-compatible gateways use nginx's non-standard
|
||||
// 499 for an upstream request that was cancelled before response
|
||||
// streaming began. At this boundary no response body stream has
|
||||
// been exposed, so it is eligible for the same bounded retry
|
||||
// policy as a 5xx gateway failure.
|
||||
499..=599 => LlmError::ServerError {
|
||||
status,
|
||||
message: body.to_string(),
|
||||
},
|
||||
_ => LlmError::Other(format!("HTTP {status}: {body}")),
|
||||
}
|
||||
}
|
||||
@@ -679,11 +724,15 @@ impl LlmError {
|
||||
|
||||
/// Constructs an `LlmError` from a reqwest error.
|
||||
pub fn from_reqwest(err: &reqwest::Error) -> Self {
|
||||
if err.is_timeout() {
|
||||
if err.is_connect() {
|
||||
LlmError::ConnectError {
|
||||
message: err.to_string(),
|
||||
transient: !has_tls_cause(err),
|
||||
timed_out: err.is_timeout(),
|
||||
}
|
||||
} else if err.is_timeout() {
|
||||
LlmError::Timeout(Duration::from_secs(0))
|
||||
} else if err.is_connect() {
|
||||
LlmError::NetworkError(format!("Connection failed: {err}"))
|
||||
} else if err.is_request() {
|
||||
} else if err.is_request() || err.is_body() || err.is_decode() {
|
||||
LlmError::NetworkError(format!("Request failed: {err}"))
|
||||
} else {
|
||||
LlmError::Other(err.to_string())
|
||||
@@ -691,6 +740,24 @@ impl LlmError {
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn has_tls_cause(mut error: &(dyn std::error::Error + 'static)) -> bool {
|
||||
loop {
|
||||
if error.is::<rustls::Error>() {
|
||||
return true;
|
||||
}
|
||||
if let Some(inner) = error
|
||||
.downcast_ref::<std::io::Error>()
|
||||
.and_then(std::io::Error::get_ref)
|
||||
{
|
||||
error = inner;
|
||||
} else if let Some(source) = error.source() {
|
||||
error = source;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Format provider HTTP error bodies before they are surfaced in the TUI.
|
||||
///
|
||||
/// Providers sometimes return whole HTML error pages for gateway/WAF blocks.
|
||||
@@ -1168,6 +1235,7 @@ impl From<serde_json::Error> for LlmError {
|
||||
/// - `retryable_status_codes`: [429, 499, 500, 502, 503, 504]
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RetryConfig {
|
||||
pub operation: RetryOperation,
|
||||
/// Whether retry logic is enabled
|
||||
pub enabled: bool,
|
||||
|
||||
@@ -1205,6 +1273,7 @@ pub struct RetryConfig {
|
||||
impl Default for RetryConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
operation: RetryOperation::Inference,
|
||||
enabled: true,
|
||||
max_retries: 3,
|
||||
initial_delay: 1.0,
|
||||
@@ -1295,6 +1364,7 @@ impl RetryConfig {
|
||||
/// Checks if a given HTTP status code should trigger a retry
|
||||
pub fn is_retryable_status(&self, status: u16) -> bool {
|
||||
self.retryable_status_codes.contains(&status)
|
||||
&& (self.operation == RetryOperation::ReadOnly || status == 429)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1401,16 +1471,28 @@ pub struct OutboundWireObservation {
|
||||
pub image_projection: Option<ImageProjectionReceipt>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct OutboundTransmissionObservation {
|
||||
pub source_id: String,
|
||||
pub method: String,
|
||||
pub transmission: RequestTransmission,
|
||||
pub http_status: Option<u16>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default, serde::Serialize)]
|
||||
pub struct OutboundWireObservationBatch {
|
||||
pub observations: Vec<OutboundWireObservation>,
|
||||
pub dropped: u64,
|
||||
pub transmissions: Vec<OutboundTransmissionObservation>,
|
||||
pub transmissions_dropped: u64,
|
||||
pub last_transmission: Option<OutboundTransmissionObservation>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct OutboundWireObservationBuffer {
|
||||
batch: OutboundWireObservationBatch,
|
||||
retained_bytes: usize,
|
||||
next_transmission: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Default)]
|
||||
@@ -1428,6 +1510,59 @@ impl OutboundWireObservations {
|
||||
std::mem::take(&mut buffer.batch)
|
||||
}
|
||||
|
||||
fn begin_transmission(&self, request_id: &str, method: &reqwest::Method) -> String {
|
||||
let mut buffer = self
|
||||
.buffer
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
buffer.next_transmission = buffer.next_transmission.saturating_add(1);
|
||||
let source_id = format!("{request_id}:{}", buffer.next_transmission);
|
||||
let observation = OutboundTransmissionObservation {
|
||||
source_id: source_id.clone(),
|
||||
method: method.as_str().to_string(),
|
||||
transmission: RequestTransmission::OutcomeUnknown,
|
||||
http_status: None,
|
||||
};
|
||||
buffer.batch.last_transmission = Some(observation.clone());
|
||||
if buffer.batch.transmissions.len() < 32 {
|
||||
buffer.batch.transmissions.push(observation);
|
||||
} else {
|
||||
buffer.batch.transmissions_dropped =
|
||||
buffer.batch.transmissions_dropped.saturating_add(1);
|
||||
}
|
||||
source_id
|
||||
}
|
||||
|
||||
fn settle_transmission(
|
||||
&self,
|
||||
source_id: &str,
|
||||
transmission: RequestTransmission,
|
||||
http_status: Option<u16>,
|
||||
) {
|
||||
let mut buffer = self
|
||||
.buffer
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if let Some(attempt) = buffer
|
||||
.batch
|
||||
.transmissions
|
||||
.iter_mut()
|
||||
.find(|attempt| attempt.source_id == source_id)
|
||||
{
|
||||
attempt.transmission = transmission;
|
||||
attempt.http_status = http_status;
|
||||
}
|
||||
if let Some(attempt) = buffer
|
||||
.batch
|
||||
.last_transmission
|
||||
.as_mut()
|
||||
.filter(|attempt| attempt.source_id == source_id)
|
||||
{
|
||||
attempt.transmission = transmission;
|
||||
attempt.http_status = http_status;
|
||||
}
|
||||
}
|
||||
|
||||
fn record(&self, observation: OutboundWireObservation) {
|
||||
let retained_bytes = [
|
||||
observation
|
||||
@@ -1477,6 +1612,7 @@ impl OutboundWireObservations {
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct RequestRetryObservation {
|
||||
pub request_id: String,
|
||||
pub retries: std::sync::Arc<std::sync::atomic::AtomicU32>,
|
||||
pub emit: RetryStatusEmitter,
|
||||
pub wire_observations: OutboundWireObservations,
|
||||
@@ -1493,6 +1629,67 @@ pub async fn observe_request_retries<F: Future>(
|
||||
REQUEST_RETRY_OBSERVATION.scope(observation, future).await
|
||||
}
|
||||
|
||||
pub fn begin_outbound_transmission(
|
||||
method: &reqwest::Method,
|
||||
) -> Option<(OutboundWireObservations, String)> {
|
||||
REQUEST_RETRY_OBSERVATION
|
||||
.try_with(|observation| {
|
||||
observation.as_ref().map(|observation| {
|
||||
let wire = observation.wire_observations.clone();
|
||||
let source = wire.begin_transmission(&observation.request_id, method);
|
||||
(wire, source)
|
||||
})
|
||||
})
|
||||
.ok()
|
||||
.flatten()
|
||||
}
|
||||
|
||||
pub fn settle_outbound_transmission(
|
||||
pending: Option<(OutboundWireObservations, String)>,
|
||||
result: &Result<reqwest::Response, LlmError>,
|
||||
) {
|
||||
let Some((wire, source)) = pending else {
|
||||
return;
|
||||
};
|
||||
let (transmission, status) = match result {
|
||||
Ok(response) => {
|
||||
let status = response.status().as_u16();
|
||||
let transmission = if matches!(status, 401 | 403 | 413 | 429) {
|
||||
RequestTransmission::Rejected
|
||||
} else {
|
||||
RequestTransmission::OutcomeUnknown
|
||||
};
|
||||
(transmission, Some(status))
|
||||
}
|
||||
Err(error) => (error.transmission(), None),
|
||||
};
|
||||
wire.settle_transmission(&source, transmission, status);
|
||||
}
|
||||
|
||||
pub fn settle_outbound_http_error(error: &LlmError) {
|
||||
let _ = REQUEST_RETRY_OBSERVATION.try_with(|observation| {
|
||||
if let Some(observation) = observation {
|
||||
let mut buffer = observation
|
||||
.wire_observations
|
||||
.buffer
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if let Some(last) = buffer.batch.last_transmission.as_mut() {
|
||||
last.transmission = error.transmission();
|
||||
let source = last.source_id.clone();
|
||||
if let Some(attempt) = buffer
|
||||
.batch
|
||||
.transmissions
|
||||
.iter_mut()
|
||||
.find(|attempt| attempt.source_id == source)
|
||||
{
|
||||
attempt.transmission = error.transmission();
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
pub fn observe_outbound_wire(observation: impl FnOnce() -> OutboundWireObservation) {
|
||||
let observations = REQUEST_RETRY_OBSERVATION
|
||||
.try_with(|observation| {
|
||||
@@ -1544,7 +1741,8 @@ pub fn retry_reason_summary(error: &LlmError) -> String {
|
||||
match error {
|
||||
LlmError::RateLimited { .. } => "rate limited".into(),
|
||||
LlmError::ServerError { status, .. } => format!("upstream {status}"),
|
||||
LlmError::NetworkError(_) => "network error".into(),
|
||||
LlmError::NetworkError(_) => "network error; request outcome unknown".into(),
|
||||
LlmError::ConnectError { .. } => "connection failed before dispatch".into(),
|
||||
LlmError::Timeout(_) => "timeout".into(),
|
||||
_ => "non-retryable provider failure".into(),
|
||||
}
|
||||
@@ -1583,17 +1781,6 @@ async fn observe_transport_stopped(retries: u32, error: &LlmError, exhausted: bo
|
||||
/// * `Ok(T)` - The successful result from the operation
|
||||
/// * `Err(RetryError)` - All retries exhausted or non-retryable error encountered
|
||||
///
|
||||
/// # Known limitation: ambiguous failures are replayed
|
||||
///
|
||||
/// A timeout or connection loss after the request was written is retried
|
||||
/// like a connect failure, so a provider that already accepted the first
|
||||
/// attempt may bill a second completion. Every caller sends model inference
|
||||
/// (messages, FIM, translation, speech, provider web search): a replay costs
|
||||
/// compute but has no external side effect, and the provider APIs used here
|
||||
/// expose no idempotency key for these requests that could dedupe it. An
|
||||
/// operation with an external side effect must not be retried through this
|
||||
/// helper without an idempotency key the server honors.
|
||||
///
|
||||
/// # Example
|
||||
///
|
||||
/// ```ignore
|
||||
@@ -1666,7 +1853,7 @@ where
|
||||
}
|
||||
Err(err) => {
|
||||
// Non-retryable errors fail immediately
|
||||
if !err.is_retryable() {
|
||||
if !err.is_retryable_for(config.operation) {
|
||||
observe_transport_stopped(attempt, &err, false).await;
|
||||
return Err(RetryError {
|
||||
last_error: err,
|
||||
@@ -1856,6 +2043,9 @@ mod tests {
|
||||
#[test]
|
||||
fn test_retry_config_defaults() {
|
||||
let config = RetryConfig::default();
|
||||
assert_eq!(config.operation, RetryOperation::Inference);
|
||||
assert!(!config.is_retryable_status(500));
|
||||
assert!(config.is_retryable_status(429));
|
||||
assert!(config.enabled);
|
||||
assert_eq!(config.max_retries, 3);
|
||||
assert_f64_eq(config.initial_delay, 1.0);
|
||||
@@ -1934,7 +2124,10 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn test_is_retryable_status() {
|
||||
let config = RetryConfig::default();
|
||||
let config = RetryConfig {
|
||||
operation: RetryOperation::ReadOnly,
|
||||
..RetryConfig::default()
|
||||
};
|
||||
|
||||
assert!(config.is_retryable_status(429)); // Rate limit
|
||||
assert!(config.is_retryable_status(499)); // Upstream request cancelled
|
||||
@@ -2241,7 +2434,10 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_with_retry_success_first_attempt() {
|
||||
let config = RetryConfig::default();
|
||||
let config = RetryConfig {
|
||||
operation: RetryOperation::ReadOnly,
|
||||
..RetryConfig::default()
|
||||
};
|
||||
let mut call_count = 0;
|
||||
|
||||
let result = with_retry(
|
||||
@@ -2285,9 +2481,10 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_with_retry_eventual_success() {
|
||||
let config = RetryConfig::new()
|
||||
let mut config = RetryConfig::new()
|
||||
.with_max_retries(3)
|
||||
.with_initial_delay(0.01); // Fast for testing
|
||||
.with_initial_delay(0.01);
|
||||
config.operation = RetryOperation::ReadOnly; // Fast for testing
|
||||
|
||||
let call_count = std::sync::Arc::new(std::sync::atomic::AtomicU32::new(0));
|
||||
let cc = call_count.clone();
|
||||
@@ -2318,9 +2515,10 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_with_retry_exhausted() {
|
||||
let config = RetryConfig::new()
|
||||
let mut config = RetryConfig::new()
|
||||
.with_max_retries(2)
|
||||
.with_initial_delay(0.01);
|
||||
config.operation = RetryOperation::ReadOnly;
|
||||
|
||||
let call_count = std::sync::Arc::new(std::sync::atomic::AtomicU32::new(0));
|
||||
let cc = call_count.clone();
|
||||
@@ -2348,9 +2546,10 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_with_retry_callback() {
|
||||
let config = RetryConfig::new()
|
||||
let mut config = RetryConfig::new()
|
||||
.with_max_retries(2)
|
||||
.with_initial_delay(0.01);
|
||||
config.operation = RetryOperation::ReadOnly;
|
||||
|
||||
let callback_count = std::sync::Arc::new(std::sync::atomic::AtomicU32::new(0));
|
||||
let cc = callback_count.clone();
|
||||
|
||||
@@ -91,16 +91,21 @@ fn retryability_distinguishes_transient_failures_from_durable_failures() {
|
||||
message: "too many requests".into(),
|
||||
retry_after: None,
|
||||
},
|
||||
LlmError::ConnectError {
|
||||
message: "refused".into(),
|
||||
transient: true,
|
||||
timed_out: false,
|
||||
},
|
||||
] {
|
||||
assert!(error.is_retryable(), "expected transient error: {error}");
|
||||
}
|
||||
for error in [
|
||||
LlmError::ServerError {
|
||||
status: 500,
|
||||
message: "internal error".into(),
|
||||
},
|
||||
LlmError::NetworkError("connection refused".into()),
|
||||
LlmError::Timeout(Duration::from_secs(30)),
|
||||
] {
|
||||
assert!(error.is_retryable(), "expected transient error: {error}");
|
||||
}
|
||||
for error in [
|
||||
LlmError::authentication_error("invalid key"),
|
||||
LlmError::AuthorizationError("blocked".into()),
|
||||
LlmError::InvalidRequest {
|
||||
@@ -116,6 +121,18 @@ fn retryability_distinguishes_transient_failures_from_durable_failures() {
|
||||
|
||||
#[test]
|
||||
fn http_response_boundary_classifies_status_contract() {
|
||||
for status in [408, 409, 499, 500, 502, 503, 504] {
|
||||
let error = LlmError::from_http_response(
|
||||
status,
|
||||
r#"{"error":{"code":"usage_limit_reached","message":"context length,429 and HTTP413"}}"#,
|
||||
);
|
||||
assert_eq!(error.transmission(), RequestTransmission::OutcomeUnknown);
|
||||
assert!(!error.is_retryable());
|
||||
if status >= 499 {
|
||||
assert!(matches!(error, LlmError::ServerError { .. }));
|
||||
}
|
||||
}
|
||||
|
||||
let body = LlmError::from_http_response(413, "maximum images exceeded model token limit");
|
||||
assert!(matches!(body, LlmError::InvalidRequest { status: 413, .. }));
|
||||
assert!(!body.is_retryable());
|
||||
@@ -140,7 +157,8 @@ fn http_response_boundary_classifies_status_contract() {
|
||||
&cancelled,
|
||||
LlmError::ServerError { status: 499, .. }
|
||||
));
|
||||
assert!(cancelled.is_retryable());
|
||||
assert!(!cancelled.is_retryable());
|
||||
assert!(cancelled.is_retryable_for(RetryOperation::ReadOnly));
|
||||
assert!(matches!(
|
||||
LlmError::from_http_response(500, "internal server error"),
|
||||
LlmError::ServerError { status: 500, .. }
|
||||
@@ -346,23 +364,25 @@ fn missing_google_signature_400_explains_recovery_without_widening_gateway_prefl
|
||||
|
||||
#[tokio::test]
|
||||
async fn retry_loop_stops_after_one_typed_quota_failure() {
|
||||
let mut calls = 0;
|
||||
let result: RetryResult<i32> = with_retry(
|
||||
&RetryConfig::default(),
|
||||
|| {
|
||||
calls += 1;
|
||||
async {
|
||||
Err(LlmError::from_http_response(
|
||||
429,
|
||||
r#"{"error":{"code":"insufficient_quota"}}"#,
|
||||
))
|
||||
}
|
||||
},
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
assert_eq!(result.unwrap_err().attempts, 1);
|
||||
assert_eq!(calls, 1);
|
||||
for (status, body) in [
|
||||
(429, r#"{"error":{"code":"insufficient_quota"}}"#),
|
||||
(500, "upstream accepted then failed"),
|
||||
(408, "request timeout"),
|
||||
(409, "conflict"),
|
||||
] {
|
||||
let mut calls = 0;
|
||||
let result: RetryResult<i32> = with_retry(
|
||||
&RetryConfig::default(),
|
||||
|| {
|
||||
calls += 1;
|
||||
async move { Err(LlmError::from_http_response(status, body)) }
|
||||
},
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
assert_eq!(result.unwrap_err().attempts, 1);
|
||||
assert_eq!(calls, 1);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -389,6 +409,7 @@ fn retry_receipt_observation() -> (
|
||||
let captured = receipts.clone();
|
||||
(
|
||||
RequestRetryObservation {
|
||||
request_id: format!("fixture:{}", uuid::Uuid::new_v4().simple()),
|
||||
wire_observations: Default::default(),
|
||||
retries: std::sync::Arc::new(std::sync::atomic::AtomicU32::new(0)),
|
||||
emit: std::sync::Arc::new(move |message| {
|
||||
@@ -404,6 +425,7 @@ fn retry_receipt_observation() -> (
|
||||
|
||||
fn immediate_retry_policy(max_retries: u32) -> RetryConfig {
|
||||
RetryConfig {
|
||||
operation: RetryOperation::ReadOnly,
|
||||
max_retries,
|
||||
initial_delay: 0.0,
|
||||
jitter: false,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use crate::llm_client::LlmError;
|
||||
use codewhale_error_taxonomy::{
|
||||
ErrorCategory, ErrorEnvelope, ErrorSeverity, classify_error_message, is_spent_balance_message,
|
||||
ErrorCategory, ErrorEnvelope, ErrorSeverity, classify_error_message,
|
||||
};
|
||||
|
||||
/// Classify a boundary error from the typed [`LlmError`] when one is
|
||||
@@ -25,7 +25,7 @@ pub fn envelope_for_llm_error(error: anyhow::Error, display_message: String) ->
|
||||
}
|
||||
Err(error) => {
|
||||
drop(error);
|
||||
ErrorEnvelope::classify(display_message, true)
|
||||
ErrorEnvelope::classify(display_message, false)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -52,21 +52,30 @@ impl From<LlmError> for ErrorEnvelope {
|
||||
LlmError::ServerError { status, message } => Self::new(
|
||||
ErrorCategory::Internal,
|
||||
ErrorSeverity::Error,
|
||||
true,
|
||||
false,
|
||||
format!("llm_server_{status}"),
|
||||
message,
|
||||
),
|
||||
LlmError::NetworkError(message) => Self::new(
|
||||
ErrorCategory::Network,
|
||||
ErrorSeverity::Error,
|
||||
true,
|
||||
false,
|
||||
"llm_network_error",
|
||||
message,
|
||||
),
|
||||
LlmError::ConnectError {
|
||||
message, transient, ..
|
||||
} => Self::new(
|
||||
ErrorCategory::Network,
|
||||
ErrorSeverity::Error,
|
||||
transient,
|
||||
"llm_connect_before_dispatch",
|
||||
message,
|
||||
),
|
||||
LlmError::Timeout(duration) => Self::new(
|
||||
ErrorCategory::Timeout,
|
||||
ErrorSeverity::Warning,
|
||||
true,
|
||||
false,
|
||||
"llm_timeout",
|
||||
format!("Request timed out after {duration:?}"),
|
||||
),
|
||||
@@ -139,22 +148,12 @@ fn envelope_for_other_llm_error(message: String) -> ErrorEnvelope {
|
||||
return ErrorEnvelope::new(
|
||||
ErrorCategory::Internal,
|
||||
ErrorSeverity::Error,
|
||||
true,
|
||||
false,
|
||||
"llm_other",
|
||||
message,
|
||||
);
|
||||
}
|
||||
// Mirror the typed variants. Only a credential or authorization refusal, a
|
||||
// rejected input and a spent balance do not heal by resending; everything
|
||||
// else (a transient class, a spent budget, an unparseable chunk) keeps the
|
||||
// retry tail the legacy `Internal` label had.
|
||||
let recoverable = match category {
|
||||
ErrorCategory::Authentication
|
||||
| ErrorCategory::Authorization
|
||||
| ErrorCategory::InvalidInput => false,
|
||||
ErrorCategory::RateLimit => !is_spent_balance_message(&message),
|
||||
_ => true,
|
||||
};
|
||||
let recoverable = false;
|
||||
let mut envelope = ErrorEnvelope::classify(message, recoverable);
|
||||
// The typed `NetworkError` is an Error-severity card; an `Other` that reads
|
||||
// as one must not turn amber for a turn that later fails after its retries.
|
||||
@@ -227,7 +226,7 @@ mod tests {
|
||||
let envelope =
|
||||
envelope_for_llm_error(untyped, "stream read error: connection reset".into());
|
||||
assert_eq!(envelope.category, ErrorCategory::Network);
|
||||
assert!(envelope.recoverable);
|
||||
assert!(!envelope.recoverable);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -252,7 +251,7 @@ mod tests {
|
||||
let envelope = ErrorEnvelope::from(LlmError::Other("Unknown retry error".to_string()));
|
||||
assert_eq!(envelope.category, ErrorCategory::Internal);
|
||||
assert_eq!(envelope.code, "llm_other");
|
||||
assert!(envelope.recoverable);
|
||||
assert!(!envelope.recoverable);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -262,7 +261,7 @@ mod tests {
|
||||
// spent balance are terminal.
|
||||
for message in ["malformed chunk in stream", "error decoding response body"] {
|
||||
let envelope = ErrorEnvelope::from(LlmError::Other(message.to_string()));
|
||||
assert!(envelope.recoverable, "{message}");
|
||||
assert!(!envelope.recoverable, "{message}");
|
||||
}
|
||||
// A transient `Other` is an Error-severity card like the typed
|
||||
// `NetworkError`, not an amber one that outlives a failed retry.
|
||||
|
||||
+199
-25
@@ -17,6 +17,8 @@ use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use codewhale_config::McpBackend;
|
||||
use codewhale_runtime::credentials::AuthContext;
|
||||
use codewhale_runtime::credentials::context::ProcessAuthContext;
|
||||
use futures_util::FutureExt;
|
||||
use parking_lot::RwLock;
|
||||
use serde::{Deserialize, Serialize};
|
||||
@@ -119,16 +121,46 @@ fn reject_linked_workspace_state_path(path: &Path) -> Result<()> {
|
||||
.with_context(|| format!("MCP config {} is not safe to write", path.display()))
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct McpAuthContext(pub Arc<dyn AuthContext>);
|
||||
|
||||
impl std::fmt::Debug for McpAuthContext {
|
||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
formatter.write_str("McpAuthContext(<host authority>)")
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for McpAuthContext {
|
||||
fn default() -> Self {
|
||||
Self(process_mcp_auth_context())
|
||||
}
|
||||
}
|
||||
|
||||
pub fn process_mcp_auth_context() -> Arc<dyn AuthContext> {
|
||||
static CONTEXT: std::sync::OnceLock<Arc<dyn AuthContext>> = std::sync::OnceLock::new();
|
||||
Arc::clone(CONTEXT.get_or_init(|| Arc::new(ProcessAuthContext)))
|
||||
}
|
||||
|
||||
fn mcp_secret_var(
|
||||
reviewed_plugin: Option<&ReviewedPluginMcpSource>,
|
||||
name: &str,
|
||||
) -> Result<String, std::env::VarError> {
|
||||
auth_context: &dyn AuthContext,
|
||||
) -> Option<String> {
|
||||
match reviewed_plugin {
|
||||
Some(source) => source.host_environment.var(name),
|
||||
None => std::env::var(name),
|
||||
Some(source) => source.host_environment.var(name).ok(),
|
||||
None => auth_context.env(name),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn mcp_secret_available(
|
||||
server: &McpServerConfig,
|
||||
name: &str,
|
||||
auth_context: &dyn AuthContext,
|
||||
) -> bool {
|
||||
mcp_secret_var(server.reviewed_plugin_source(), name, auth_context)
|
||||
.is_some_and(|value| !value.trim().is_empty())
|
||||
}
|
||||
|
||||
/// Expand `${NAME}` placeholders in an MCP config value from its permitted
|
||||
/// environment source. This lets secrets (API keys, bearer tokens, …) be supplied
|
||||
/// through environment variables instead of being written in cleartext into
|
||||
@@ -140,6 +172,7 @@ fn mcp_secret_var(
|
||||
fn expand_env_placeholders_with(
|
||||
value: &str,
|
||||
reviewed_plugin: Option<&ReviewedPluginMcpSource>,
|
||||
auth_context: &dyn AuthContext,
|
||||
) -> Result<String> {
|
||||
let mut out = String::new();
|
||||
let mut rest = value;
|
||||
@@ -153,7 +186,7 @@ fn expand_env_placeholders_with(
|
||||
if name.is_empty() || !name.chars().all(|c| c.is_ascii_alphanumeric() || c == '_') {
|
||||
anyhow::bail!("invalid environment placeholder in MCP config value");
|
||||
}
|
||||
let env_value = mcp_secret_var(reviewed_plugin, name).with_context(|| {
|
||||
let env_value = mcp_secret_var(reviewed_plugin, name, auth_context).with_context(|| {
|
||||
format!("environment variable {name} required by MCP config is not set")
|
||||
})?;
|
||||
out.push_str(&env_value);
|
||||
@@ -165,7 +198,7 @@ fn expand_env_placeholders_with(
|
||||
|
||||
#[cfg(test)]
|
||||
fn expand_env_placeholders(value: &str) -> Result<String> {
|
||||
expand_env_placeholders_with(value, None)
|
||||
expand_env_placeholders_with(value, None, &ProcessAuthContext)
|
||||
}
|
||||
|
||||
/// Expand `${NAME}` placeholders across every value of an MCP config map
|
||||
@@ -175,12 +208,13 @@ fn expand_env_placeholders_map_with_environment(
|
||||
values: &HashMap<String, String>,
|
||||
context: &str,
|
||||
reviewed_plugin: Option<&ReviewedPluginMcpSource>,
|
||||
auth_context: &dyn AuthContext,
|
||||
) -> Result<HashMap<String, String>> {
|
||||
let mut expanded = HashMap::with_capacity(values.len());
|
||||
for (key, value) in values {
|
||||
expanded.insert(
|
||||
key.clone(),
|
||||
expand_env_placeholders_with(value, reviewed_plugin)
|
||||
expand_env_placeholders_with(value, reviewed_plugin, auth_context)
|
||||
.with_context(|| format!("failed to expand MCP {context} value for {key}"))?,
|
||||
);
|
||||
}
|
||||
@@ -192,21 +226,38 @@ fn expand_env_placeholders_map(
|
||||
values: &HashMap<String, String>,
|
||||
context: &str,
|
||||
) -> Result<HashMap<String, String>> {
|
||||
expand_env_placeholders_map_with_environment(values, context, None)
|
||||
expand_env_placeholders_map_with_environment(values, context, None, &ProcessAuthContext)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn expanded_mcp_stdio_env(config: &McpServerConfig) -> Result<HashMap<String, String>> {
|
||||
expanded_mcp_stdio_env_with_auth_context(config, &ProcessAuthContext)
|
||||
}
|
||||
|
||||
fn expanded_mcp_stdio_env_with_auth_context(
|
||||
config: &McpServerConfig,
|
||||
auth_context: &dyn AuthContext,
|
||||
) -> Result<HashMap<String, String>> {
|
||||
expand_env_placeholders_map_with_environment(
|
||||
&config.env,
|
||||
"env",
|
||||
config.reviewed_plugin_source(),
|
||||
auth_context,
|
||||
)
|
||||
}
|
||||
|
||||
/// Mirror the exact expanded and sanitized environment applied by the MCP
|
||||
/// stdio spawn path, without constructing or starting a process.
|
||||
#[cfg(test)]
|
||||
fn mcp_stdio_child_env(config: &McpServerConfig) -> Result<Vec<(OsString, OsString)>> {
|
||||
let expanded_env = expanded_mcp_stdio_env(config)?;
|
||||
mcp_stdio_child_env_with_auth_context(config, &ProcessAuthContext)
|
||||
}
|
||||
|
||||
fn mcp_stdio_child_env_with_auth_context(
|
||||
config: &McpServerConfig,
|
||||
auth_context: &dyn AuthContext,
|
||||
) -> Result<Vec<(OsString, OsString)>> {
|
||||
let expanded_env = expanded_mcp_stdio_env_with_auth_context(config, auth_context)?;
|
||||
let overrides = codewhale_runtime::child_env::string_map_env(&expanded_env);
|
||||
Ok(if let Some(source) = config.reviewed_plugin_source() {
|
||||
// Plugin reviews name every extra environment source explicitly. Do
|
||||
@@ -328,6 +379,13 @@ fn command_availability_on_path(
|
||||
/// Inspect an MCP stdio command using the same expanded, sanitized environment
|
||||
/// as the real spawn path, without starting the configured process.
|
||||
pub fn static_mcp_command_availability(server: &McpServerConfig) -> Result<McpCommandAvailability> {
|
||||
static_mcp_command_availability_with_auth_context(server, &ProcessAuthContext)
|
||||
}
|
||||
|
||||
pub fn static_mcp_command_availability_with_auth_context(
|
||||
server: &McpServerConfig,
|
||||
auth_context: &dyn AuthContext,
|
||||
) -> Result<McpCommandAvailability> {
|
||||
if server.url.is_some() {
|
||||
return Ok(McpCommandAvailability::NotApplicable);
|
||||
}
|
||||
@@ -340,7 +398,7 @@ pub fn static_mcp_command_availability(server: &McpServerConfig) -> Result<McpCo
|
||||
|
||||
// StdioTransport expands every configured env value before spawning, even
|
||||
// when the command itself is absolute. Mirror that failure boundary here.
|
||||
let child_env = mcp_stdio_child_env(server)?;
|
||||
let child_env = mcp_stdio_child_env_with_auth_context(server, auth_context)?;
|
||||
let path = Path::new(cmd);
|
||||
let is_absolute = path.is_absolute() || cmd.starts_with('/');
|
||||
if is_absolute {
|
||||
@@ -1745,6 +1803,7 @@ fn response_id_matches(id: Option<&serde_json::Value>, expected_id: &str) -> boo
|
||||
|
||||
/// Manages a single async connection to an MCP server
|
||||
pub struct McpConnection {
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
name: String,
|
||||
transport: Box<dyn McpTransport>,
|
||||
tools: Vec<McpTool>,
|
||||
@@ -1875,6 +1934,7 @@ async fn prepare_mcp_http_client(
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
cancel_token: &tokio_util::sync::CancellationToken,
|
||||
connect_timeout_secs: u64,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> Result<http_client::McpHttpClient> {
|
||||
let url = config.url.as_deref().context("MCP HTTP URL absent")?;
|
||||
// Per-domain network policy gate (#135). Only the HTTP/SSE transport
|
||||
@@ -1915,7 +1975,7 @@ async fn prepare_mcp_http_client(
|
||||
let oauth_runtime = if config.reviewed_plugin.is_some() {
|
||||
None
|
||||
} else {
|
||||
match oauth::build_default_headers(config) {
|
||||
match oauth::build_default_headers_with_auth_context(config, auth_context.as_ref()) {
|
||||
Ok(default_headers) => {
|
||||
let prepared = tokio::select! {
|
||||
biased;
|
||||
@@ -1971,7 +2031,12 @@ async fn prepare_mcp_http_client(
|
||||
}
|
||||
}
|
||||
};
|
||||
let client = client.with_mcp_auth(McpHttpAuth::from_config(name, config, oauth_runtime));
|
||||
let client = client.with_mcp_auth(McpHttpAuth::from_config_with_auth_context(
|
||||
name,
|
||||
config,
|
||||
oauth_runtime,
|
||||
auth_context,
|
||||
));
|
||||
Ok(client)
|
||||
}
|
||||
|
||||
@@ -2003,6 +2068,25 @@ impl McpConnection {
|
||||
global_timeouts: &McpTimeouts,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
backend: McpBackend,
|
||||
) -> Result<Self> {
|
||||
Self::connect_with_backend_and_auth_context(
|
||||
name,
|
||||
config,
|
||||
global_timeouts,
|
||||
network_policy,
|
||||
backend,
|
||||
process_mcp_auth_context(),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn connect_with_backend_and_auth_context(
|
||||
name: String,
|
||||
config: McpServerConfig,
|
||||
global_timeouts: &McpTimeouts,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
backend: McpBackend,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> Result<Self> {
|
||||
let connect_timeout_secs = config.effective_connect_timeout(global_timeouts);
|
||||
let read_timeout_secs = config.effective_read_timeout(global_timeouts);
|
||||
@@ -2033,6 +2117,7 @@ impl McpConnection {
|
||||
network_policy,
|
||||
&cancel_token,
|
||||
connect_timeout_secs,
|
||||
Arc::clone(&auth_context),
|
||||
)
|
||||
.await?,
|
||||
)
|
||||
@@ -2041,12 +2126,13 @@ impl McpConnection {
|
||||
};
|
||||
let transport: Box<dyn McpTransport> = if backend == McpBackend::Host {
|
||||
Box::new(
|
||||
crate::extension_host::mcp::SdkTransport::connect_with_http(
|
||||
crate::extension_host::mcp::SdkTransport::connect_with_http_and_auth_context(
|
||||
&name,
|
||||
&config,
|
||||
cancel_token.clone(),
|
||||
Duration::from_secs(connect_timeout_secs),
|
||||
http_client,
|
||||
Arc::clone(&auth_context),
|
||||
)
|
||||
.await?,
|
||||
)
|
||||
@@ -2085,11 +2171,12 @@ impl McpConnection {
|
||||
Box::new(http)
|
||||
}
|
||||
} else if let Some(command) = &config.command {
|
||||
Box::new(StdioTransport::spawn(
|
||||
Box::new(StdioTransport::spawn_with_auth_context(
|
||||
&name,
|
||||
command,
|
||||
&config,
|
||||
cancel_token.clone(),
|
||||
Arc::clone(&auth_context),
|
||||
)?)
|
||||
} else {
|
||||
anyhow::bail!("MCP server '{name}' config must have either 'command' or 'url'");
|
||||
@@ -2104,6 +2191,7 @@ impl McpConnection {
|
||||
let authority_watch = authority_watch.map(PendingAuthorityWatch::disarm);
|
||||
|
||||
let mut conn = Self {
|
||||
auth_context,
|
||||
name: name.clone(),
|
||||
transport,
|
||||
tools: Vec::new(),
|
||||
@@ -3238,6 +3326,7 @@ pub async fn authenticate_tool_via_pool(
|
||||
|
||||
/// Pool of MCP connections for reuse
|
||||
pub struct McpPool {
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
backend: McpBackend,
|
||||
/// Immutable operator ceiling; source reloads and shared child pools cannot relax it.
|
||||
disallowed_tools: Vec<String>,
|
||||
@@ -3312,6 +3401,7 @@ pub struct McpPool {
|
||||
}
|
||||
|
||||
pub struct McpHandoff {
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
workspace: Option<PathBuf>,
|
||||
backend: McpBackend,
|
||||
connections: Vec<(String, McpServerIdentity, McpConnection)>,
|
||||
@@ -3351,6 +3441,7 @@ pub struct SupervisionDue {
|
||||
/// One supervisor sweep's plan: candidates to redial plus the transitions
|
||||
/// the plan phase already knows (recoveries and newly parked servers).
|
||||
pub struct SupervisionPlan {
|
||||
pub auth_context: Arc<dyn AuthContext>,
|
||||
pub backend: McpBackend,
|
||||
pub due: Vec<SupervisionDue>,
|
||||
pub recovered: Vec<String>,
|
||||
@@ -3443,8 +3534,13 @@ pub fn tool_selection_covers_server(requested: &[String], server: &str) -> bool
|
||||
impl McpPool {
|
||||
/// Create a new pool with the given configuration
|
||||
pub fn new(config: McpConfig) -> Self {
|
||||
Self::new_with_auth_context(config, process_mcp_auth_context())
|
||||
}
|
||||
|
||||
pub fn new_with_auth_context(config: McpConfig, auth_context: Arc<dyn AuthContext>) -> Self {
|
||||
let config_hash = hash_mcp_config(&config);
|
||||
Self {
|
||||
auth_context,
|
||||
backend: McpBackend::default(),
|
||||
connections: HashMap::new(),
|
||||
disallowed_tools: Vec::new(),
|
||||
@@ -3474,6 +3570,10 @@ impl McpPool {
|
||||
self.backend = backend;
|
||||
self
|
||||
}
|
||||
pub fn auth_context(&self) -> Arc<dyn AuthContext> {
|
||||
Arc::clone(&self.auth_context)
|
||||
}
|
||||
|
||||
pub fn backend(&self) -> McpBackend {
|
||||
self.backend
|
||||
}
|
||||
@@ -3504,13 +3604,27 @@ impl McpPool {
|
||||
path: &std::path::Path,
|
||||
workspace: &Path,
|
||||
plugins: Arc<codewhale_plugins::PluginRegistry>,
|
||||
) -> Result<Self> {
|
||||
Self::from_config_path_with_workspace_and_plugins_and_auth_context(
|
||||
path,
|
||||
workspace,
|
||||
plugins,
|
||||
process_mcp_auth_context(),
|
||||
)
|
||||
}
|
||||
|
||||
pub fn from_config_path_with_workspace_and_plugins_and_auth_context(
|
||||
path: &std::path::Path,
|
||||
workspace: &Path,
|
||||
plugins: Arc<codewhale_plugins::PluginRegistry>,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> Result<Self> {
|
||||
if plugins.workspace() != workspace {
|
||||
anyhow::bail!("plugin registry workspace does not match MCP pool workspace");
|
||||
}
|
||||
let config = load_config_with_workspace_and_plugins(path, workspace, plugins.as_ref())?;
|
||||
let workspace = checked_workspace_path(workspace)?;
|
||||
let mut pool = Self::new(config);
|
||||
let mut pool = Self::new_with_auth_context(config, auth_context);
|
||||
pool.config_sources = vec![
|
||||
path.to_path_buf(),
|
||||
checked_workspace_mcp_config_path(&workspace)?,
|
||||
@@ -3535,13 +3649,27 @@ impl McpPool {
|
||||
path: &std::path::Path,
|
||||
workspace: &Path,
|
||||
plugins: Arc<codewhale_plugins::PluginRegistry>,
|
||||
) -> Result<Self> {
|
||||
Self::empty_with_workspace_config_sources_and_auth_context(
|
||||
path,
|
||||
workspace,
|
||||
plugins,
|
||||
process_mcp_auth_context(),
|
||||
)
|
||||
}
|
||||
|
||||
pub fn empty_with_workspace_config_sources_and_auth_context(
|
||||
path: &std::path::Path,
|
||||
workspace: &Path,
|
||||
plugins: Arc<codewhale_plugins::PluginRegistry>,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> Result<Self> {
|
||||
validate_mcp_config_path(path)?;
|
||||
if plugins.workspace() != workspace {
|
||||
anyhow::bail!("plugin registry workspace does not match MCP pool workspace");
|
||||
}
|
||||
let workspace = checked_workspace_path(workspace)?;
|
||||
let mut pool = Self::new(McpConfig::default());
|
||||
let mut pool = Self::new_with_auth_context(McpConfig::default(), auth_context);
|
||||
pool.config_sources = vec![
|
||||
path.to_path_buf(),
|
||||
checked_workspace_mcp_config_path(&workspace)?,
|
||||
@@ -3897,7 +4025,7 @@ impl McpPool {
|
||||
merge_plugin_mcp_servers(config, plugins.as_ref())?
|
||||
}
|
||||
};
|
||||
let mut pool = Self::new(config)
|
||||
let mut pool = Self::new_with_auth_context(config, Arc::clone(&self.auth_context))
|
||||
.with_backend(self.backend)
|
||||
.with_disallowed_tools(self.disallowed_tools.clone());
|
||||
pool.network_policy = self.network_policy.clone();
|
||||
@@ -4013,12 +4141,13 @@ impl McpPool {
|
||||
anyhow::bail!("Failed to connect MCP server '{server_name}': server is disabled");
|
||||
}
|
||||
|
||||
let mut connection = match McpConnection::connect_with_backend(
|
||||
let mut connection = match McpConnection::connect_with_backend_and_auth_context(
|
||||
server_name.to_string(),
|
||||
server_config,
|
||||
&self.config.timeouts,
|
||||
self.network_policy.as_ref(),
|
||||
self.backend,
|
||||
Arc::clone(&self.auth_context),
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -4053,12 +4182,13 @@ impl McpPool {
|
||||
/// the named transport.
|
||||
pub async fn retry_connection(&mut self, server_name: &str) -> Result<&mut McpConnection> {
|
||||
let server_config = self.retry_server_config(server_name)?;
|
||||
let mut connection = match McpConnection::connect_with_backend(
|
||||
let mut connection = match McpConnection::connect_with_backend_and_auth_context(
|
||||
server_name.to_string(),
|
||||
server_config,
|
||||
&self.config.timeouts,
|
||||
self.network_policy.as_ref(),
|
||||
self.backend,
|
||||
Arc::clone(&self.auth_context),
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -4131,6 +4261,10 @@ impl McpPool {
|
||||
connection: McpConnection,
|
||||
) -> Result<()> {
|
||||
self.require_server(&name)?;
|
||||
anyhow::ensure!(
|
||||
Arc::ptr_eq(&connection.auth_context, &self.auth_context),
|
||||
"MCP connection belongs to another credential authority"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
connection.catalog_generation == self.current_catalog_generation(),
|
||||
"MCP configuration changed while connecting {name}; retry against the current config"
|
||||
@@ -4160,11 +4294,17 @@ impl McpPool {
|
||||
let identity = mcp_server_identity(&self.config, &name)?;
|
||||
(identity.1.is_none()
|
||||
&& connection.config().url.is_none()
|
||||
&& !connection
|
||||
.config()
|
||||
.env
|
||||
.values()
|
||||
.any(|value| value.contains("${"))
|
||||
&& connection.is_transport_ready())
|
||||
.then_some((name, identity, connection))
|
||||
})
|
||||
.collect();
|
||||
McpHandoff {
|
||||
auth_context: Arc::clone(&self.auth_context),
|
||||
workspace: self.workspace.clone(),
|
||||
backend: self.backend,
|
||||
connections,
|
||||
@@ -4172,7 +4312,11 @@ impl McpPool {
|
||||
}
|
||||
|
||||
pub fn adopt_handoff(&mut self, handoff: McpHandoff) -> usize {
|
||||
if self.retired || handoff.workspace != self.workspace || handoff.backend != self.backend {
|
||||
if self.retired
|
||||
|| handoff.workspace != self.workspace
|
||||
|| handoff.backend != self.backend
|
||||
|| !Arc::ptr_eq(&handoff.auth_context, &self.auth_context)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
let generation = self.current_catalog_generation();
|
||||
@@ -4367,6 +4511,7 @@ impl McpPool {
|
||||
});
|
||||
}
|
||||
SupervisionPlan {
|
||||
auth_context: Arc::clone(&self.auth_context),
|
||||
backend: self.backend,
|
||||
due,
|
||||
recovered,
|
||||
@@ -4490,7 +4635,7 @@ impl McpPool {
|
||||
if plan.due.is_empty() && plan.recovered.is_empty() && plan.parked.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let mut connects = Self::spawn_pending_connects(
|
||||
let mut connects = Self::spawn_pending_connects_with_auth_context(
|
||||
plan.due
|
||||
.iter()
|
||||
.map(|due| (due.name.clone(), due.config.clone()))
|
||||
@@ -4499,6 +4644,7 @@ impl McpPool {
|
||||
plan.network_policy.clone(),
|
||||
plan.catalog_generation,
|
||||
plan.backend,
|
||||
Arc::clone(&plan.auth_context),
|
||||
);
|
||||
let mut update = McpSupervisorUpdate {
|
||||
recovered: plan.recovered,
|
||||
@@ -4856,6 +5002,24 @@ impl McpPool {
|
||||
network_policy: Option<NetworkPolicyDecider>,
|
||||
catalog_generation: u64,
|
||||
backend: McpBackend,
|
||||
) -> tokio::task::JoinSet<(String, Result<McpConnection, anyhow::Error>)> {
|
||||
Self::spawn_pending_connects_with_auth_context(
|
||||
pending,
|
||||
timeouts,
|
||||
network_policy,
|
||||
catalog_generation,
|
||||
backend,
|
||||
process_mcp_auth_context(),
|
||||
)
|
||||
}
|
||||
|
||||
pub fn spawn_pending_connects_with_auth_context(
|
||||
pending: Vec<McpPendingConnect>,
|
||||
timeouts: McpTimeouts,
|
||||
network_policy: Option<NetworkPolicyDecider>,
|
||||
catalog_generation: u64,
|
||||
backend: McpBackend,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> tokio::task::JoinSet<(String, Result<McpConnection, anyhow::Error>)> {
|
||||
let semaphore = std::sync::Arc::new(tokio::sync::Semaphore::new(Self::CONNECT_CONCURRENCY));
|
||||
let mut joins: tokio::task::JoinSet<(String, Result<McpConnection, anyhow::Error>)> =
|
||||
@@ -4863,15 +5027,17 @@ impl McpPool {
|
||||
for (name, config) in pending {
|
||||
let permit = semaphore.clone();
|
||||
let network_policy = network_policy.clone();
|
||||
let auth_context = Arc::clone(&auth_context);
|
||||
joins.spawn(async move {
|
||||
let connection = std::panic::AssertUnwindSafe(async {
|
||||
let _permit = permit.acquire_owned().await;
|
||||
McpConnection::connect_with_backend(
|
||||
McpConnection::connect_with_backend_and_auth_context(
|
||||
name.clone(),
|
||||
config,
|
||||
&timeouts,
|
||||
network_policy.as_ref(),
|
||||
backend,
|
||||
auth_context,
|
||||
)
|
||||
.await
|
||||
.map(|mut connection| {
|
||||
@@ -4944,12 +5110,13 @@ impl McpPool {
|
||||
break;
|
||||
}
|
||||
|
||||
let mut connects = Self::spawn_pending_connects(
|
||||
let mut connects = Self::spawn_pending_connects_with_auth_context(
|
||||
pending,
|
||||
self.config.timeouts,
|
||||
self.network_policy.clone(),
|
||||
self.catalog_generation.load(Ordering::SeqCst),
|
||||
self.backend,
|
||||
Arc::clone(&self.auth_context),
|
||||
);
|
||||
while let Some(joined) = connects.join_next().await {
|
||||
let (name, result) = joined
|
||||
@@ -5083,18 +5250,24 @@ impl McpPool {
|
||||
.get(server_name)
|
||||
.is_some_and(McpConnection::is_ready);
|
||||
if ready
|
||||
|| oauth::has_usable_stored_tokens(server_name, &server, self.network_policy.as_ref())
|
||||
.await?
|
||||
|| oauth::has_usable_stored_tokens_with_auth_context(
|
||||
server_name,
|
||||
&server,
|
||||
self.network_policy.as_ref(),
|
||||
Arc::clone(&self.auth_context),
|
||||
)
|
||||
.await?
|
||||
{
|
||||
return Ok(AuthenticateToolStart::AlreadyAuthorized);
|
||||
}
|
||||
let login = oauth::begin_oauth_login_for_server_tool(
|
||||
let login = oauth::begin_oauth_login_for_server_tool_with_auth_context(
|
||||
server_name,
|
||||
&server,
|
||||
None,
|
||||
self.oauth_callback_port,
|
||||
self.oauth_callback_url.as_deref(),
|
||||
self.network_policy.as_ref(),
|
||||
Arc::clone(&self.auth_context),
|
||||
)
|
||||
.await?;
|
||||
Ok(AuthenticateToolStart::Login(Box::new(login)))
|
||||
@@ -5399,6 +5572,7 @@ impl McpPool {
|
||||
}))
|
||||
.expect("minimal server config");
|
||||
let conn = McpConnection {
|
||||
auth_context: Arc::clone(&self.auth_context),
|
||||
name: server.to_string(),
|
||||
transport: Box::new(IdleTransport),
|
||||
tools: tools
|
||||
|
||||
@@ -22,8 +22,9 @@ use codewhale_mcp::config::McpServerConfig;
|
||||
use codewhale_net::guard::{guarded_reqwest_client_builder, is_restricted_ip};
|
||||
use codewhale_net::network_policy::{Decision, NetworkPolicyDecider};
|
||||
|
||||
#[derive(Clone, Default)]
|
||||
#[derive(Clone)]
|
||||
pub(super) struct McpHttpAuth {
|
||||
pub(super) auth_context: Arc<dyn codewhale_runtime::credentials::AuthContext>,
|
||||
pub(super) server_name: String,
|
||||
pub(super) headers: HashMap<String, String>,
|
||||
pub(super) env_headers: HashMap<String, String>,
|
||||
@@ -46,13 +47,45 @@ pub(super) struct McpHttpAuth {
|
||||
pub(super) reviewed_plugin: Option<ReviewedPluginMcpSource>,
|
||||
}
|
||||
|
||||
impl Default for McpHttpAuth {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
auth_context: super::process_mcp_auth_context(),
|
||||
server_name: String::new(),
|
||||
headers: HashMap::new(),
|
||||
env_headers: HashMap::new(),
|
||||
bearer_token_env_var: None,
|
||||
oauth: None,
|
||||
oauth_configured: false,
|
||||
suppress_server_error_details: false,
|
||||
reviewed_plugin: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl McpHttpAuth {
|
||||
#[cfg(test)]
|
||||
pub(super) fn from_config(
|
||||
server_name: &str,
|
||||
config: &McpServerConfig,
|
||||
oauth: Option<oauth::McpOAuthRuntime>,
|
||||
) -> Self {
|
||||
Self::from_config_with_auth_context(
|
||||
server_name,
|
||||
config,
|
||||
oauth,
|
||||
super::process_mcp_auth_context(),
|
||||
)
|
||||
}
|
||||
|
||||
pub(super) fn from_config_with_auth_context(
|
||||
server_name: &str,
|
||||
config: &McpServerConfig,
|
||||
oauth: Option<oauth::McpOAuthRuntime>,
|
||||
auth_context: Arc<dyn codewhale_runtime::credentials::AuthContext>,
|
||||
) -> Self {
|
||||
Self {
|
||||
auth_context,
|
||||
server_name: server_name.to_string(),
|
||||
headers: config.headers.clone(),
|
||||
env_headers: config.env_headers.clone(),
|
||||
@@ -78,8 +111,12 @@ impl McpHttpAuth {
|
||||
}
|
||||
let mut headers = self.headers.clone();
|
||||
for (name, env_var) in &self.env_headers {
|
||||
let value = super::mcp_secret_var(self.reviewed_plugin.as_ref(), env_var);
|
||||
if let Ok(value) = value
|
||||
let value = super::mcp_secret_var(
|
||||
self.reviewed_plugin.as_ref(),
|
||||
env_var,
|
||||
self.auth_context.as_ref(),
|
||||
);
|
||||
if let Some(value) = value
|
||||
&& !value.trim().is_empty()
|
||||
{
|
||||
headers.insert(name.clone(), value);
|
||||
@@ -87,7 +124,11 @@ impl McpHttpAuth {
|
||||
}
|
||||
if !mcp_headers_have_authorization(&headers)
|
||||
&& let Some(env_var) = self.bearer_token_env_var.as_deref()
|
||||
&& let Ok(token) = super::mcp_secret_var(self.reviewed_plugin.as_ref(), env_var)
|
||||
&& let Some(token) = super::mcp_secret_var(
|
||||
self.reviewed_plugin.as_ref(),
|
||||
env_var,
|
||||
self.auth_context.as_ref(),
|
||||
)
|
||||
{
|
||||
let token = token.trim();
|
||||
if !token.is_empty() {
|
||||
|
||||
+134
-19
@@ -1,6 +1,7 @@
|
||||
use super::http_client::McpHttpClient;
|
||||
use crate::mcp::McpServerConfigExt;
|
||||
use codewhale_net::network_policy::NetworkPolicyDecider;
|
||||
use codewhale_runtime::credentials::AuthContext;
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
@@ -1073,6 +1074,21 @@ pub async fn auth_status_for_server(
|
||||
name: &str,
|
||||
server: &McpServerConfig,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
) -> McpAuthStatus {
|
||||
auth_status_for_server_with_auth_context(
|
||||
name,
|
||||
server,
|
||||
network_policy,
|
||||
super::process_mcp_auth_context(),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn auth_status_for_server_with_auth_context(
|
||||
name: &str,
|
||||
server: &McpServerConfig,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> McpAuthStatus {
|
||||
if server.reviewed_plugin.is_some() || !server.is_enabled() || server.url.is_none() {
|
||||
return McpAuthStatus::Unsupported;
|
||||
@@ -1092,7 +1108,7 @@ pub async fn auth_status_for_server(
|
||||
}
|
||||
}
|
||||
|
||||
let headers = match build_default_headers(server) {
|
||||
let headers = match build_default_headers_with_auth_context(server, auth_context.as_ref()) {
|
||||
Ok(headers) => headers,
|
||||
Err(err) => {
|
||||
tracing::warn!(target: "mcp", server = %name, error = %err, "failed to build MCP OAuth discovery headers");
|
||||
@@ -1112,6 +1128,15 @@ pub async fn auth_status_for_server(
|
||||
pub async fn oauth_login_support(
|
||||
server: &McpServerConfig,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
) -> Result<Option<McpOAuthDiscovery>> {
|
||||
oauth_login_support_with_auth_context(server, network_policy, super::process_mcp_auth_context())
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn oauth_login_support_with_auth_context(
|
||||
server: &McpServerConfig,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> Result<Option<McpOAuthDiscovery>> {
|
||||
if server.reviewed_plugin.is_some() {
|
||||
return Ok(None);
|
||||
@@ -1122,7 +1147,7 @@ pub async fn oauth_login_support(
|
||||
if server_has_manual_authorization(server) {
|
||||
return Ok(None);
|
||||
}
|
||||
let headers = build_default_headers(server)?;
|
||||
let headers = build_default_headers_with_auth_context(server, auth_context.as_ref())?;
|
||||
discover_streamable_http_oauth_for_server(server, url, headers, network_policy).await
|
||||
}
|
||||
|
||||
@@ -1148,8 +1173,9 @@ fn oauth_login_client(
|
||||
server: &McpServerConfig,
|
||||
url: &str,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
auth_context: &dyn AuthContext,
|
||||
) -> Result<McpHttpClient> {
|
||||
let headers = build_default_headers(server)?;
|
||||
let headers = build_default_headers_with_auth_context(server, auth_context)?;
|
||||
Ok(oauth_http_client(server, url, network_policy)?.with_default_headers(headers))
|
||||
}
|
||||
|
||||
@@ -1218,14 +1244,40 @@ pub async fn perform_oauth_login_for_server(
|
||||
callback_url: Option<&str>,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
) -> Result<()> {
|
||||
perform_oauth_login_for_server_with_cancel(
|
||||
perform_oauth_login_for_server_with_auth_context(
|
||||
name,
|
||||
server,
|
||||
explicit_scopes,
|
||||
callback_port,
|
||||
callback_url,
|
||||
CancellationToken::new(),
|
||||
network_policy,
|
||||
super::process_mcp_auth_context(),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn perform_oauth_login_for_server_with_auth_context(
|
||||
name: &str,
|
||||
server: &McpServerConfig,
|
||||
explicit_scopes: Option<Vec<String>>,
|
||||
callback_port: Option<u16>,
|
||||
callback_url: Option<&str>,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> Result<()> {
|
||||
if server.reviewed_plugin.is_some() {
|
||||
bail!(
|
||||
"OAuth is disabled for plugin-contributed MCP servers; use a reviewed environment-backed header or bearer token"
|
||||
);
|
||||
}
|
||||
perform_oauth_login_for_server_inner(
|
||||
name,
|
||||
server,
|
||||
explicit_scopes,
|
||||
callback_port,
|
||||
callback_url,
|
||||
network_policy,
|
||||
auth_context,
|
||||
)
|
||||
.await
|
||||
}
|
||||
@@ -1251,13 +1303,14 @@ pub async fn perform_oauth_login_for_server_with_cancel(
|
||||
}
|
||||
run_cancellable_oauth(
|
||||
&cancellation_token,
|
||||
perform_oauth_login_for_server_inner(
|
||||
perform_oauth_login_for_server_with_auth_context(
|
||||
name,
|
||||
server,
|
||||
explicit_scopes,
|
||||
callback_port,
|
||||
callback_url,
|
||||
network_policy,
|
||||
super::process_mcp_auth_context(),
|
||||
),
|
||||
)
|
||||
.await
|
||||
@@ -1282,6 +1335,7 @@ async fn resolve_oauth_login(
|
||||
server: &McpServerConfig,
|
||||
explicit_scopes: Option<Vec<String>>,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> Result<(String, ResolvedMcpOAuthScopes)> {
|
||||
let Some(url) = server.url.as_deref() else {
|
||||
bail!("OAuth login is only supported for URL-based MCP servers");
|
||||
@@ -1291,7 +1345,8 @@ async fn resolve_oauth_login(
|
||||
}
|
||||
|
||||
let discovery = if explicit_scopes.is_none() && server.scopes.is_empty() {
|
||||
oauth_login_support(server, network_policy).await?
|
||||
oauth_login_support_with_auth_context(server, network_policy, Arc::clone(&auth_context))
|
||||
.await?
|
||||
} else {
|
||||
None
|
||||
};
|
||||
@@ -1310,14 +1365,21 @@ async fn perform_oauth_login_for_server_inner(
|
||||
callback_port: Option<u16>,
|
||||
callback_url: Option<&str>,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> Result<()> {
|
||||
let (url, resolved_scopes) =
|
||||
resolve_oauth_login(name, server, explicit_scopes, network_policy).await?;
|
||||
let (url, resolved_scopes) = resolve_oauth_login(
|
||||
name,
|
||||
server,
|
||||
explicit_scopes,
|
||||
network_policy,
|
||||
Arc::clone(&auth_context),
|
||||
)
|
||||
.await?;
|
||||
|
||||
match perform_oauth_login(
|
||||
name,
|
||||
&url,
|
||||
oauth_login_client(server, &url, network_policy)?,
|
||||
oauth_login_client(server, &url, network_policy, auth_context.as_ref())?,
|
||||
&resolved_scopes.scopes,
|
||||
server.oauth.as_ref(),
|
||||
server.oauth_resource.as_deref(),
|
||||
@@ -1337,7 +1399,7 @@ async fn perform_oauth_login_for_server_inner(
|
||||
perform_oauth_login(
|
||||
name,
|
||||
&url,
|
||||
oauth_login_client(server, &url, network_policy)?,
|
||||
oauth_login_client(server, &url, network_policy, auth_context.as_ref())?,
|
||||
&[],
|
||||
server.oauth.as_ref(),
|
||||
server.oauth_resource.as_deref(),
|
||||
@@ -1397,6 +1459,7 @@ pub struct McpOAuthToolLogin {
|
||||
server_name: String,
|
||||
server: McpServerConfig,
|
||||
scopes_source: McpOAuthScopesSource,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
network_policy: Option<NetworkPolicyDecider>,
|
||||
flow: OauthLoginFlow,
|
||||
open_browser: bool,
|
||||
@@ -1435,7 +1498,12 @@ impl McpOAuthToolLogin {
|
||||
OauthLoginFlow::new(
|
||||
&self.server_name,
|
||||
url,
|
||||
oauth_login_client(server, url, self.network_policy.as_ref())?,
|
||||
oauth_login_client(
|
||||
server,
|
||||
url,
|
||||
self.network_policy.as_ref(),
|
||||
self.auth_context.as_ref(),
|
||||
)?,
|
||||
&[],
|
||||
server.oauth.as_ref(),
|
||||
server.oauth_resource.as_deref(),
|
||||
@@ -1463,18 +1531,45 @@ pub async fn begin_oauth_login_for_server_tool(
|
||||
callback_port: Option<u16>,
|
||||
callback_url: Option<&str>,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
) -> Result<McpOAuthToolLogin> {
|
||||
begin_oauth_login_for_server_tool_with_auth_context(
|
||||
name,
|
||||
server,
|
||||
explicit_scopes,
|
||||
callback_port,
|
||||
callback_url,
|
||||
network_policy,
|
||||
super::process_mcp_auth_context(),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn begin_oauth_login_for_server_tool_with_auth_context(
|
||||
name: &str,
|
||||
server: &McpServerConfig,
|
||||
explicit_scopes: Option<Vec<String>>,
|
||||
callback_port: Option<u16>,
|
||||
callback_url: Option<&str>,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> Result<McpOAuthToolLogin> {
|
||||
if server.reviewed_plugin.is_some() {
|
||||
bail!(
|
||||
"OAuth is disabled for plugin-contributed MCP servers; use a reviewed environment-backed header or bearer token"
|
||||
);
|
||||
}
|
||||
let (url, resolved_scopes) =
|
||||
resolve_oauth_login(name, server, explicit_scopes, network_policy).await?;
|
||||
let (url, resolved_scopes) = resolve_oauth_login(
|
||||
name,
|
||||
server,
|
||||
explicit_scopes,
|
||||
network_policy,
|
||||
Arc::clone(&auth_context),
|
||||
)
|
||||
.await?;
|
||||
let flow = OauthLoginFlow::new(
|
||||
name,
|
||||
&url,
|
||||
oauth_login_client(server, &url, network_policy)?,
|
||||
oauth_login_client(server, &url, network_policy, auth_context.as_ref())?,
|
||||
&resolved_scopes.scopes,
|
||||
server.oauth.as_ref(),
|
||||
server.oauth_resource.as_deref(),
|
||||
@@ -1486,6 +1581,7 @@ pub async fn begin_oauth_login_for_server_tool(
|
||||
server_name: name.to_string(),
|
||||
server: server.clone(),
|
||||
scopes_source: resolved_scopes.source,
|
||||
auth_context,
|
||||
network_policy: network_policy.cloned(),
|
||||
flow,
|
||||
// The test build drives the loopback callback itself; a real browser
|
||||
@@ -1512,6 +1608,21 @@ pub async fn has_usable_stored_tokens(
|
||||
name: &str,
|
||||
server: &McpServerConfig,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
) -> Result<bool> {
|
||||
has_usable_stored_tokens_with_auth_context(
|
||||
name,
|
||||
server,
|
||||
network_policy,
|
||||
super::process_mcp_auth_context(),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn has_usable_stored_tokens_with_auth_context(
|
||||
name: &str,
|
||||
server: &McpServerConfig,
|
||||
network_policy: Option<&NetworkPolicyDecider>,
|
||||
auth_context: Arc<dyn AuthContext>,
|
||||
) -> Result<bool> {
|
||||
let Some(url) = server.url.as_deref() else {
|
||||
return Ok(false);
|
||||
@@ -1524,8 +1635,8 @@ pub async fn has_usable_stored_tokens(
|
||||
match McpOAuthRuntime::from_server_config_with_client(
|
||||
name,
|
||||
server,
|
||||
build_default_headers(server)?,
|
||||
oauth_login_client(server, url, network_policy)?,
|
||||
build_default_headers_with_auth_context(server, auth_context.as_ref())?,
|
||||
oauth_login_client(server, url, network_policy, auth_context.as_ref())?,
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -1578,13 +1689,17 @@ pub fn server_has_manual_authorization(server: &McpServerConfig) -> bool {
|
||||
|| contains_authorization_header(&server.env_headers)
|
||||
}
|
||||
|
||||
pub(super) fn build_default_headers(server: &McpServerConfig) -> Result<HeaderMap> {
|
||||
pub(super) fn build_default_headers_with_auth_context(
|
||||
server: &McpServerConfig,
|
||||
auth_context: &dyn AuthContext,
|
||||
) -> Result<HeaderMap> {
|
||||
let mut headers = HeaderMap::new();
|
||||
for (name, value) in &server.headers {
|
||||
insert_header(&mut headers, name, value)?;
|
||||
}
|
||||
for (name, env_var) in &server.env_headers {
|
||||
if let Ok(value) = super::mcp_secret_var(server.reviewed_plugin_source(), env_var)
|
||||
if let Some(value) =
|
||||
super::mcp_secret_var(server.reviewed_plugin_source(), env_var, auth_context)
|
||||
&& !value.trim().is_empty()
|
||||
{
|
||||
insert_header(&mut headers, name, &value)?;
|
||||
|
||||
@@ -93,6 +93,22 @@ impl BrokerSession {
|
||||
command: &str,
|
||||
config: &McpServerConfig,
|
||||
cancel_token: tokio_util::sync::CancellationToken,
|
||||
) -> Result<(Self, ChildStdout)> {
|
||||
Self::spawn_with_auth_context(
|
||||
server_name,
|
||||
command,
|
||||
config,
|
||||
cancel_token,
|
||||
super::process_mcp_auth_context(),
|
||||
)
|
||||
}
|
||||
|
||||
pub fn spawn_with_auth_context(
|
||||
server_name: &str,
|
||||
command: &str,
|
||||
config: &McpServerConfig,
|
||||
cancel_token: tokio_util::sync::CancellationToken,
|
||||
auth_context: Arc<dyn codewhale_runtime::credentials::AuthContext>,
|
||||
) -> Result<(Self, ChildStdout)> {
|
||||
let reviewed_launch = if let Some(reviewed_plugin) = config.reviewed_plugin_source() {
|
||||
// This is deliberately the last trust check before constructing
|
||||
@@ -165,11 +181,12 @@ impl BrokerSession {
|
||||
}
|
||||
|
||||
// Expand `${NAME}` placeholders so secret env values can be sourced
|
||||
// from the process environment instead of being stored in cleartext
|
||||
// from the owning credential context instead of being stored in cleartext
|
||||
// in the MCP config. The child env is allowlist-sanitized below, so
|
||||
// these vars would not otherwise be inherited by the child.
|
||||
let expanded_env = super::expanded_mcp_stdio_env(config)
|
||||
.with_context(|| format!("MCP server '{server_name}' env expansion failed"))?;
|
||||
let expanded_env =
|
||||
super::expanded_mcp_stdio_env_with_auth_context(config, auth_context.as_ref())
|
||||
.with_context(|| format!("MCP server '{server_name}' env expansion failed"))?;
|
||||
|
||||
// User-configured MCP keeps the compatibility-oriented Node/Python
|
||||
// bootstrap allowlist (#1244). Reviewed plugins receive only the base
|
||||
|
||||
@@ -15,13 +15,36 @@ pub(super) struct StdioTransport {
|
||||
}
|
||||
|
||||
impl StdioTransport {
|
||||
#[cfg(test)]
|
||||
pub(super) fn spawn(
|
||||
server_name: &str,
|
||||
command: &str,
|
||||
config: &McpServerConfig,
|
||||
cancel_token: tokio_util::sync::CancellationToken,
|
||||
) -> Result<Self> {
|
||||
let (session, stdout) = BrokerSession::spawn(server_name, command, config, cancel_token)?;
|
||||
Self::spawn_with_auth_context(
|
||||
server_name,
|
||||
command,
|
||||
config,
|
||||
cancel_token,
|
||||
super::process_mcp_auth_context(),
|
||||
)
|
||||
}
|
||||
|
||||
pub(super) fn spawn_with_auth_context(
|
||||
server_name: &str,
|
||||
command: &str,
|
||||
config: &McpServerConfig,
|
||||
cancel_token: tokio_util::sync::CancellationToken,
|
||||
auth_context: std::sync::Arc<dyn codewhale_runtime::credentials::AuthContext>,
|
||||
) -> Result<Self> {
|
||||
let (session, stdout) = BrokerSession::spawn_with_auth_context(
|
||||
server_name,
|
||||
command,
|
||||
config,
|
||||
cancel_token,
|
||||
auth_context,
|
||||
)?;
|
||||
Ok(Self {
|
||||
session,
|
||||
reader: tokio::io::BufReader::new(stdout),
|
||||
|
||||
@@ -3047,6 +3047,7 @@ fn test_server_config() -> McpServerConfig {
|
||||
|
||||
fn test_connection(transport: Box<dyn McpTransport>) -> McpConnection {
|
||||
McpConnection {
|
||||
auth_context: process_mcp_auth_context(),
|
||||
name: "mock".to_string(),
|
||||
transport,
|
||||
tools: Vec::new(),
|
||||
|
||||
@@ -59,7 +59,7 @@ pub struct RuntimeOptions {
|
||||
#[arg(long, hide = true)]
|
||||
pub remote_control: bool,
|
||||
#[arg(skip)]
|
||||
pub control_frontend: Option<codewhale_app_server::RuntimeControlFrontend>,
|
||||
pub control_frontend: Option<codewhale_runtime::runtime_control::RuntimeControlFrontend>,
|
||||
}
|
||||
|
||||
impl RuntimeOptions {
|
||||
|
||||
@@ -49,6 +49,10 @@ use crate::tools::subagent::{
|
||||
SharedSubAgentManager, SubAgentStatus, new_shared_runtime_subagent_manager,
|
||||
};
|
||||
use crate::tools::todo::new_shared_todo_list;
|
||||
use codewhale_app_server_protocol::runtime::{
|
||||
DynamicToolCallContent, DynamicToolCallParams, DynamicToolCallResult, DynamicToolSpec,
|
||||
TurnEnvironmentParams,
|
||||
};
|
||||
#[cfg(test)]
|
||||
use codewhale_config::AppMode;
|
||||
use codewhale_config::ProviderKind;
|
||||
@@ -65,10 +69,6 @@ use codewhale_protocol::agent_mail::{
|
||||
AgentMailSendRequest, AgentMailSendResponse, AgentMailStatus, MAX_AGENT_MAIL_DELIVERY_ATTEMPTS,
|
||||
MAX_AGENT_MAIL_SUMMARY_BYTES,
|
||||
};
|
||||
use codewhale_protocol::runtime::{
|
||||
DynamicToolCallContent, DynamicToolCallParams, DynamicToolCallResult, DynamicToolSpec,
|
||||
TurnEnvironmentParams,
|
||||
};
|
||||
use codewhale_runtime::config::{Config, MAX_SUBAGENTS, ProviderIdentity};
|
||||
use codewhale_runtime::runtime_policy::RuntimePolicyProjection;
|
||||
pub const RUNTIME_EVENT_REPLAY_BATCH_SIZE: usize = 256;
|
||||
@@ -411,6 +411,8 @@ pub struct ThreadReadMarker {
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct SavedSessionCheckpoint {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub publication_observation: Option<crate::session_manager::SessionObservation>,
|
||||
pub covered_turn_id: Option<String>,
|
||||
/// Fingerprint of the document's first `messages_len` messages.
|
||||
pub messages_sha256: String,
|
||||
@@ -683,6 +685,14 @@ fn unanswered_call_result(
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct RuntimeTurnRequestDiagnostics {
|
||||
#[serde(default)]
|
||||
pub primary_coverage_recorded: bool,
|
||||
#[serde(default)]
|
||||
pub primary_usage_reported: u32,
|
||||
#[serde(default)]
|
||||
pub primary_usage_unknown: u32,
|
||||
#[serde(default)]
|
||||
pub primary_usage_missing: u32,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub stop_reason: Option<RuntimeTurnStopReason>,
|
||||
/// Parent streaming model-client calls, including stream retries.
|
||||
@@ -716,6 +726,10 @@ pub enum RuntimeTurnStopReason {
|
||||
impl From<&crate::tool_inspection::TurnStopDiagnostics> for RuntimeTurnRequestDiagnostics {
|
||||
fn from(diagnostics: &crate::tool_inspection::TurnStopDiagnostics) -> Self {
|
||||
Self {
|
||||
primary_coverage_recorded: false,
|
||||
primary_usage_reported: 0,
|
||||
primary_usage_unknown: 0,
|
||||
primary_usage_missing: 0,
|
||||
stop_reason: diagnostics.reason.map(|reason| match reason {
|
||||
crate::tool_inspection::TurnStopReason::ProviderNoToolCall => {
|
||||
RuntimeTurnStopReason::ProviderNoToolCall
|
||||
@@ -966,8 +980,8 @@ pub struct TurnRecord {
|
||||
serialize_with = "serialize_routed_usage_drop_records"
|
||||
)]
|
||||
pub routed_usage_drop_records: Vec<RuntimeUsageDropRecord>,
|
||||
/// Fingerprints of provider-call identities already appended to this turn.
|
||||
/// This durable ledger makes mailbox delivery, direct sinks, fallback
|
||||
/// Fingerprints of provider-call identities already settled by this turn.
|
||||
/// Primary membership adds no routed amount. This ledger makes mailbox, sink, fallback
|
||||
/// recovery, and process restart idempotent without persisting raw ids.
|
||||
#[serde(
|
||||
default,
|
||||
@@ -4709,6 +4723,10 @@ impl HeldRuntimeStore {
|
||||
let mut thread = store.load_thread(thread_id)?;
|
||||
thread.session_id = Some(session.metadata.id.clone());
|
||||
thread.saved_session_checkpoint = Some(SavedSessionCheckpoint {
|
||||
publication_observation: session
|
||||
.write_receipt
|
||||
.as_ref()
|
||||
.and_then(|receipt| receipt.observation().ok()),
|
||||
covered_turn_id: thread.latest_turn_id.clone(),
|
||||
messages_sha256: session_messages_sha256(&session.messages)?,
|
||||
messages_len: Some(session.messages.len()),
|
||||
@@ -5057,6 +5075,8 @@ pub struct RuntimeHistoryOperation {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub target_document_digest: Option<String>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
publication_base: Option<crate::session_manager::SessionObservation>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
goal_source_digest: Option<String>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
goal_target_digest: Option<String>,
|
||||
@@ -5079,6 +5099,18 @@ pub struct RuntimeHistoryWitness {
|
||||
pub workspace: PathBuf,
|
||||
}
|
||||
impl RuntimeHistoryOperation {
|
||||
pub fn session_publication_id(&self) -> Result<String> {
|
||||
self.validate()?;
|
||||
Ok(codewhale_runtime::hashing::sha256_hex(
|
||||
codewhale_runtime::hashing::canonical_json(&serde_json::json!({
|
||||
"session_publication": 1,
|
||||
"key_fingerprint": self.key_fingerprint,
|
||||
"receipt": self.receipt,
|
||||
}))
|
||||
.as_bytes(),
|
||||
))
|
||||
}
|
||||
|
||||
fn validate(&self) -> Result<()> {
|
||||
anyhow::ensure!(
|
||||
self.version == 1 && self.receipt.version == 1,
|
||||
|
||||
@@ -1145,6 +1145,7 @@ pub struct RuntimeThreadManagerConfig {
|
||||
/// Selected document store, captured once before owner recovery or hydration.
|
||||
pub sessions_dir: Option<PathBuf>,
|
||||
pub max_active_threads: usize,
|
||||
pub mcp_auth_context: crate::mcp::McpAuthContext,
|
||||
}
|
||||
|
||||
impl RuntimeThreadManagerConfig {
|
||||
@@ -1170,6 +1171,7 @@ impl RuntimeThreadManagerConfig {
|
||||
task_data_dir,
|
||||
sessions_dir: None,
|
||||
max_active_threads: MAX_ACTIVE_THREADS_DEFAULT,
|
||||
mcp_auth_context: Default::default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1368,7 +1370,7 @@ pub struct StartTurnRequest {
|
||||
pub max_output_tokens: Option<std::num::NonZeroU32>,
|
||||
pub prompt: String,
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub images: Vec<codewhale_protocol::runtime::RuntimeImageInput>,
|
||||
pub images: Vec<codewhale_app_server_protocol::runtime::RuntimeImageInput>,
|
||||
/// Optional caller-supplied idempotency key, scoped to this Runtime store
|
||||
/// and thread. The raw key is validated but never persisted.
|
||||
#[serde(default, alias = "operationKey")]
|
||||
@@ -1499,7 +1501,7 @@ pub(super) fn runtime_turn_request_fingerprint(
|
||||
trust_mode: bool,
|
||||
dynamic_tools: &[DynamicToolSpec],
|
||||
environment_id: Option<&str>,
|
||||
images: &[codewhale_protocol::runtime::RuntimeImageInput],
|
||||
images: &[codewhale_app_server_protocol::runtime::RuntimeImageInput],
|
||||
max_output_tokens: Option<std::num::NonZeroU32>,
|
||||
) -> Result<String> {
|
||||
let mut payload = json!({
|
||||
@@ -2149,6 +2151,59 @@ pub(super) fn billed_prompt_tokens(usage: &Usage) -> u32 {
|
||||
.map_or(usage.input_tokens, |last| last.input_tokens)
|
||||
}
|
||||
|
||||
fn accumulate_primary_runtime_usage_record(
|
||||
totals: &mut UsageTotals,
|
||||
buckets: &mut std::collections::BTreeMap<String, UsageBucket>,
|
||||
group_by: UsageGroupBy,
|
||||
route: Option<&EffectiveRouteEnvelope>,
|
||||
usage: &Usage,
|
||||
turn: &TurnRecord,
|
||||
thread: &ThreadRecord,
|
||||
) {
|
||||
if let Some(facts) = turn
|
||||
.model_request_diagnostics
|
||||
.filter(|facts| facts.primary_coverage_recorded)
|
||||
{
|
||||
for (count, reason) in [
|
||||
(
|
||||
facts.primary_usage_unknown,
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown,
|
||||
),
|
||||
(
|
||||
facts.primary_usage_missing,
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage,
|
||||
),
|
||||
] {
|
||||
if let Some(route) = route {
|
||||
if count > 0 {
|
||||
accumulate_exact_runtime_usage_drop(
|
||||
totals,
|
||||
buckets,
|
||||
group_by,
|
||||
route,
|
||||
reason,
|
||||
turn,
|
||||
thread,
|
||||
u64::from(count),
|
||||
);
|
||||
}
|
||||
} else if count > 0 {
|
||||
totals.unpriced_turns = totals.unpriced_turns.saturating_add(u64::from(count));
|
||||
totals.cny_unpriced_turns =
|
||||
totals.cny_unpriced_turns.saturating_add(u64::from(count));
|
||||
totals.unpriced_reasons.insert(reason.label().to_string());
|
||||
totals
|
||||
.cny_unpriced_reasons
|
||||
.insert(reason.label().to_string());
|
||||
}
|
||||
}
|
||||
if facts.primary_usage_reported == 0 && usage == &Usage::default() {
|
||||
return;
|
||||
}
|
||||
}
|
||||
accumulate_runtime_usage_record(totals, buckets, group_by, route, usage, turn, thread);
|
||||
}
|
||||
|
||||
pub(super) fn accumulate_runtime_usage_record(
|
||||
totals: &mut UsageTotals,
|
||||
buckets: &mut std::collections::BTreeMap<String, UsageBucket>,
|
||||
@@ -2324,6 +2379,7 @@ pub(super) fn accumulate_runtime_child_usage_record(
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage,
|
||||
turn,
|
||||
thread,
|
||||
1,
|
||||
);
|
||||
} else {
|
||||
accumulate_runtime_usage_record(
|
||||
@@ -2338,6 +2394,7 @@ pub(super) fn accumulate_runtime_child_usage_record(
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub(super) fn accumulate_exact_runtime_usage_drop(
|
||||
totals: &mut UsageTotals,
|
||||
buckets: &mut std::collections::BTreeMap<String, UsageBucket>,
|
||||
@@ -2346,18 +2403,19 @@ pub(super) fn accumulate_exact_runtime_usage_drop(
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason,
|
||||
turn: &TurnRecord,
|
||||
thread: &ThreadRecord,
|
||||
count: u64,
|
||||
) {
|
||||
let nonmetered = matches!(
|
||||
route.billing_mode,
|
||||
RouteBillingMode::Subscription | RouteBillingMode::Local
|
||||
);
|
||||
totals.dropped_usage_records = totals.dropped_usage_records.saturating_add(1);
|
||||
totals.turns = totals.turns.saturating_add(1);
|
||||
totals.dropped_usage_records = totals.dropped_usage_records.saturating_add(count);
|
||||
totals.turns = totals.turns.saturating_add(count);
|
||||
if nonmetered {
|
||||
totals.nonmetered_turns = totals.nonmetered_turns.saturating_add(1);
|
||||
totals.nonmetered_turns = totals.nonmetered_turns.saturating_add(count);
|
||||
} else {
|
||||
totals.unpriced_turns = totals.unpriced_turns.saturating_add(1);
|
||||
totals.cny_unpriced_turns = totals.cny_unpriced_turns.saturating_add(1);
|
||||
totals.unpriced_turns = totals.unpriced_turns.saturating_add(count);
|
||||
totals.cny_unpriced_turns = totals.cny_unpriced_turns.saturating_add(count);
|
||||
totals.unpriced_reasons.insert(reason.label().to_string());
|
||||
totals
|
||||
.cny_unpriced_reasons
|
||||
@@ -2375,13 +2433,13 @@ pub(super) fn accumulate_exact_runtime_usage_drop(
|
||||
key,
|
||||
..UsageBucket::default()
|
||||
});
|
||||
bucket.dropped_usage_records = bucket.dropped_usage_records.saturating_add(1);
|
||||
bucket.turns = bucket.turns.saturating_add(1);
|
||||
bucket.dropped_usage_records = bucket.dropped_usage_records.saturating_add(count);
|
||||
bucket.turns = bucket.turns.saturating_add(count);
|
||||
if nonmetered {
|
||||
bucket.nonmetered_turns = bucket.nonmetered_turns.saturating_add(1);
|
||||
bucket.nonmetered_turns = bucket.nonmetered_turns.saturating_add(count);
|
||||
} else {
|
||||
bucket.unpriced_turns = bucket.unpriced_turns.saturating_add(1);
|
||||
bucket.cny_unpriced_turns = bucket.cny_unpriced_turns.saturating_add(1);
|
||||
bucket.unpriced_turns = bucket.unpriced_turns.saturating_add(count);
|
||||
bucket.cny_unpriced_turns = bucket.cny_unpriced_turns.saturating_add(count);
|
||||
bucket.unpriced_reasons.insert(reason.label().to_string());
|
||||
bucket
|
||||
.cny_unpriced_reasons
|
||||
@@ -2553,7 +2611,7 @@ pub struct PreparedThreadFork {
|
||||
pub(super) thread: ThreadRecord,
|
||||
pub(super) records: Vec<(TurnRecord, Vec<TurnItemRecord>)>,
|
||||
pub(super) original_user_text: Option<String>,
|
||||
pub(super) original_images: Vec<codewhale_protocol::runtime::RuntimeImageInput>,
|
||||
pub(super) original_images: Vec<codewhale_app_server_protocol::runtime::RuntimeImageInput>,
|
||||
pub(super) max_output_tokens: Option<std::num::NonZeroU32>,
|
||||
/// The checked full source/goal/lease, retained prefix and covered cloned
|
||||
/// turn travel together. The private target document is written only at
|
||||
@@ -3956,6 +4014,10 @@ impl RuntimeThreadManager {
|
||||
Ok(manager)
|
||||
}
|
||||
|
||||
pub fn mcp_auth_context(&self) -> Arc<dyn codewhale_runtime::credentials::AuthContext> {
|
||||
Arc::clone(&self.manager_cfg.mcp_auth_context.0)
|
||||
}
|
||||
|
||||
pub fn sessions_dir(&self) -> &Path {
|
||||
self.manager_cfg
|
||||
.sessions_dir
|
||||
@@ -7094,10 +7156,20 @@ impl RuntimeThreadManager {
|
||||
let parent_dispatched_at = parent_route
|
||||
.as_ref()
|
||||
.map_or(turn.created_at, |route| route.dispatched_at);
|
||||
if let Some(usage) = turn.effective_route_usage.as_ref().or(turn.usage.as_ref())
|
||||
&& usage_timestamp_in_range(parent_dispatched_at, since, until)
|
||||
if usage_timestamp_in_range(parent_dispatched_at, since, until)
|
||||
&& (turn.effective_route_usage.is_some()
|
||||
|| turn.usage.is_some()
|
||||
|| turn
|
||||
.model_request_diagnostics
|
||||
.is_some_and(|facts| facts.primary_coverage_recorded))
|
||||
{
|
||||
accumulate_runtime_usage_record(
|
||||
let absent = Usage::default();
|
||||
let usage = turn
|
||||
.effective_route_usage
|
||||
.as_ref()
|
||||
.or(turn.usage.as_ref())
|
||||
.unwrap_or(&absent);
|
||||
accumulate_primary_runtime_usage_record(
|
||||
&mut totals,
|
||||
&mut buckets,
|
||||
group_by,
|
||||
@@ -7146,6 +7218,7 @@ impl RuntimeThreadManager {
|
||||
drop_record.reason,
|
||||
turn,
|
||||
thread,
|
||||
1,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -7219,8 +7292,19 @@ impl RuntimeThreadManager {
|
||||
std::collections::BTreeMap::new();
|
||||
for turn in &turns {
|
||||
let parent_route = turn.effective_route_envelope();
|
||||
if let Some(usage) = turn.effective_route_usage.as_ref().or(turn.usage.as_ref()) {
|
||||
accumulate_runtime_usage_record(
|
||||
if turn.effective_route_usage.is_some()
|
||||
|| turn.usage.is_some()
|
||||
|| turn
|
||||
.model_request_diagnostics
|
||||
.is_some_and(|facts| facts.primary_coverage_recorded)
|
||||
{
|
||||
let absent = Usage::default();
|
||||
let usage = turn
|
||||
.effective_route_usage
|
||||
.as_ref()
|
||||
.or(turn.usage.as_ref())
|
||||
.unwrap_or(&absent);
|
||||
accumulate_primary_runtime_usage_record(
|
||||
&mut split.parent,
|
||||
&mut buckets,
|
||||
UsageGroupBy::Thread,
|
||||
@@ -7249,6 +7333,7 @@ impl RuntimeThreadManager {
|
||||
drop_record.reason,
|
||||
turn,
|
||||
&thread,
|
||||
1,
|
||||
);
|
||||
}
|
||||
// Dropped fallback receipts are routed-child records, so the
|
||||
@@ -7920,7 +8005,7 @@ impl RuntimeThreadManager {
|
||||
"invalid existing history target identity"
|
||||
);
|
||||
}
|
||||
let operation = RuntimeHistoryOperation {
|
||||
let mut operation = RuntimeHistoryOperation {
|
||||
version: 1,
|
||||
key_fingerprint: fingerprint.clone(),
|
||||
committed: false,
|
||||
@@ -7928,6 +8013,7 @@ impl RuntimeThreadManager {
|
||||
journal_witness: None,
|
||||
association: None,
|
||||
target_document_digest: None,
|
||||
publication_base: None,
|
||||
goal_source_digest: None,
|
||||
goal_target_digest: None,
|
||||
session_goal_source_digest: None,
|
||||
@@ -7947,6 +8033,15 @@ impl RuntimeThreadManager {
|
||||
.unwrap_or_else(|| uuid::Uuid::new_v4().to_string()),
|
||||
},
|
||||
};
|
||||
let sessions =
|
||||
crate::session_manager::SessionManager::new(self.sessions_dir().to_path_buf())?;
|
||||
let source = if target.is_some() {
|
||||
sessions.capture_session_write_source(&operation.receipt.session_id)?
|
||||
} else {
|
||||
sessions.expect_absent_session(&operation.receipt.session_id)?
|
||||
};
|
||||
operation.publication_base = Some(source.observation()?);
|
||||
operation.validate()?;
|
||||
self.store.save_history_operation(&operation)?;
|
||||
Ok(operation)
|
||||
})
|
||||
@@ -7991,14 +8086,14 @@ impl RuntimeThreadManager {
|
||||
|
||||
pub fn bind_history_operation_witness(
|
||||
&self,
|
||||
mut operation: RuntimeHistoryOperation,
|
||||
operation: RuntimeHistoryOperation,
|
||||
witness: RuntimeHistoryWitness,
|
||||
association: codewhale_protocol::CanonicalThreadOperationAssociation,
|
||||
) -> Result<RuntimeHistoryOperation> {
|
||||
let fingerprint = operation.key_fingerprint.clone();
|
||||
self.store
|
||||
.with_turn_operation_claim(Some(&fingerprint), || {
|
||||
let current = self
|
||||
let mut current = self
|
||||
.store
|
||||
.load_history_operation(&fingerprint)?
|
||||
.ok_or_else(|| anyhow!("history reservation is absent"))?;
|
||||
@@ -8017,11 +8112,11 @@ impl RuntimeThreadManager {
|
||||
!current.committed,
|
||||
"committed operation has no journal witness"
|
||||
);
|
||||
operation.journal_witness = Some(witness);
|
||||
operation.association = Some(association);
|
||||
operation.validate()?;
|
||||
self.store.save_history_operation(&operation)?;
|
||||
Ok(operation)
|
||||
current.journal_witness = Some(witness);
|
||||
current.association = Some(association);
|
||||
current.validate()?;
|
||||
self.store.save_history_operation(¤t)?;
|
||||
Ok(current)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -8117,6 +8212,7 @@ impl RuntimeThreadManager {
|
||||
};
|
||||
engine
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: Some(thread.id.clone()),
|
||||
messages: session.messages.clone(),
|
||||
system_prompt_override: system_prompt.is_some(),
|
||||
@@ -8371,6 +8467,124 @@ impl RuntimeThreadManager {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn saved_session_publication_source(
|
||||
&self,
|
||||
thread_id: &str,
|
||||
) -> Result<Option<crate::session_manager::SessionWriteReceipt>> {
|
||||
let _thread_mutation = self.store.thread_mutation.lock();
|
||||
let thread = self.store.load_thread(thread_id)?;
|
||||
let Some(id) = thread.session_id.as_deref() else {
|
||||
return Ok(None);
|
||||
};
|
||||
let Some(observation) = thread
|
||||
.saved_session_checkpoint
|
||||
.as_ref()
|
||||
.and_then(|checkpoint| checkpoint.publication_observation.clone())
|
||||
else {
|
||||
return Ok(None);
|
||||
};
|
||||
let source = crate::session_manager::SessionWriteReceipt::observed(observation);
|
||||
anyhow::ensure!(
|
||||
source.targets(self.sessions_dir(), id)?,
|
||||
"saved publication source belongs to another store or session"
|
||||
);
|
||||
Ok(Some(source))
|
||||
}
|
||||
|
||||
pub fn capture_history_publication_source(
|
||||
&self,
|
||||
operation: &RuntimeHistoryOperation,
|
||||
source: crate::session_manager::SessionWriteReceipt,
|
||||
) -> Result<crate::session_manager::SessionWriteReceipt> {
|
||||
validate_runtime_turn_operation_key(&operation.receipt.operation_key)?;
|
||||
let fingerprint = codewhale_runtime::hashing::sha256_hex(
|
||||
format!(
|
||||
"codewhale:history-operation:v1\0{}",
|
||||
operation.receipt.operation_key
|
||||
)
|
||||
.as_bytes(),
|
||||
);
|
||||
self.store
|
||||
.with_turn_operation_claim(Some(&fingerprint), || {
|
||||
let current = self
|
||||
.lookup_history_operation(
|
||||
&operation.receipt.operation_key,
|
||||
&operation.receipt.request_digest,
|
||||
)?
|
||||
.ok_or_else(|| anyhow!("history reservation is absent"))?;
|
||||
anyhow::ensure!(
|
||||
current.receipt == operation.receipt
|
||||
&& !current.committed
|
||||
&& current.journal_witness == operation.journal_witness
|
||||
&& current.association == operation.association,
|
||||
"history operation changed before native publication admission"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
source.targets(self.sessions_dir(), ¤t.receipt.session_id)?,
|
||||
"history publication source belongs to another store or target"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
current.publication_base.as_ref() == Some(&source.observation()?),
|
||||
"history original publication source is absent or changed; recovery required"
|
||||
);
|
||||
Ok(source.capture_history_control(¤t)?)
|
||||
})
|
||||
}
|
||||
|
||||
pub fn verify_prepared_history_publication(
|
||||
&self,
|
||||
operation: &RuntimeHistoryOperation,
|
||||
session: &crate::session_manager::SavedSession,
|
||||
) -> Result<()> {
|
||||
validate_runtime_turn_operation_key(&operation.receipt.operation_key)?;
|
||||
let fingerprint = codewhale_runtime::hashing::sha256_hex(
|
||||
format!(
|
||||
"codewhale:history-operation:v1\0{}",
|
||||
operation.receipt.operation_key
|
||||
)
|
||||
.as_bytes(),
|
||||
);
|
||||
self.store
|
||||
.with_turn_operation_claim(Some(&fingerprint), || {
|
||||
let current = self
|
||||
.lookup_history_operation(
|
||||
&operation.receipt.operation_key,
|
||||
&operation.receipt.request_digest,
|
||||
)?
|
||||
.ok_or_else(|| anyhow!("history reservation is absent"))?;
|
||||
anyhow::ensure!(
|
||||
current.receipt == operation.receipt && !current.committed,
|
||||
"prepared history operation changed before recovery"
|
||||
);
|
||||
let source = session.write_receipt.as_ref().ok_or_else(|| {
|
||||
anyhow!("prepared target lacks an admitted native read source")
|
||||
})?;
|
||||
anyhow::ensure!(
|
||||
source.targets(self.sessions_dir(), ¤t.receipt.session_id)?,
|
||||
"prepared target belongs to another store or session"
|
||||
);
|
||||
let observed = source.observation()?;
|
||||
anyhow::ensure!(
|
||||
session.metadata.publication_id.as_deref()
|
||||
== Some(current.session_publication_id()?.as_str()),
|
||||
"uncommitted target belongs to another publication generation; recovery required"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
current
|
||||
.publication_base
|
||||
.as_ref()
|
||||
.is_some_and(|base| base.same_checkpoint(&observed)),
|
||||
"prepared target checkpoint changed or its original source is unknown"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
current.target_document_digest.as_ref()
|
||||
== Some(&history::saved_document_digest(session)?),
|
||||
"uncommitted target is not its exact prepared publication; recovery required"
|
||||
);
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
/// Bind full-fidelity saved history to the exact prefix it covers. Callers
|
||||
/// hold session_checkpoint_guard across the snapshot, save and this write.
|
||||
pub async fn set_thread_session_checkpoint(
|
||||
@@ -8385,6 +8599,10 @@ impl RuntimeThreadManager {
|
||||
let mut thread = self.store.load_thread(thread_id)?;
|
||||
thread.session_id = Some(session_id.clone());
|
||||
thread.saved_session_checkpoint = Some(SavedSessionCheckpoint {
|
||||
publication_observation: session
|
||||
.write_receipt
|
||||
.as_ref()
|
||||
.and_then(|receipt| receipt.observation().ok()),
|
||||
covered_turn_id: thread.latest_turn_id.clone(),
|
||||
messages_sha256,
|
||||
messages_len: Some(session.messages.len()),
|
||||
@@ -8746,6 +8964,10 @@ impl RuntimeThreadManager {
|
||||
"native fork source document or local goal changed; source retained"
|
||||
);
|
||||
}
|
||||
let target_id = Uuid::new_v4().to_string();
|
||||
let source = manager
|
||||
.expect_absent_session(&target_id)?
|
||||
.capture(crate::session_manager::SessionWriteOwner::control())?;
|
||||
let mut session = match captured.session {
|
||||
Some(source) => {
|
||||
let parent = source.metadata.clone();
|
||||
@@ -8782,7 +9004,7 @@ impl RuntimeThreadManager {
|
||||
session.leaf_id = journal.leaf_id.clone();
|
||||
session.metadata.spawn_depth = journal.spawn_depth;
|
||||
session.messages = journal.to_messages();
|
||||
session.metadata.id = Uuid::new_v4().to_string();
|
||||
session.metadata.id = target_id;
|
||||
session.metadata.created_at = forked.created_at;
|
||||
session.metadata.updated_at = forked.updated_at;
|
||||
session.metadata.message_count = session.messages.len();
|
||||
@@ -8830,7 +9052,9 @@ impl RuntimeThreadManager {
|
||||
!manager.session_document_exists(&session_id),
|
||||
"native fork target already exists"
|
||||
);
|
||||
manager.save_session(&session)?;
|
||||
session.write_receipt = Some(source);
|
||||
let publication = manager.publish_session(&session)?;
|
||||
session.adopt_publication(&publication);
|
||||
let mut goal = captured.goal;
|
||||
if let Some(goal) = goal.as_mut()
|
||||
&& goal.status == crate::session_manager::SessionGoalStatus::Active
|
||||
@@ -8841,6 +9065,7 @@ impl RuntimeThreadManager {
|
||||
manager.save_session_goal(&session_id, goal.as_ref())?;
|
||||
forked.session_id = Some(session_id);
|
||||
forked.saved_session_checkpoint = Some(SavedSessionCheckpoint {
|
||||
publication_observation: Some(publication.observation().clone()),
|
||||
covered_turn_id,
|
||||
messages_sha256: session_messages_sha256(&session.messages)?,
|
||||
messages_len: Some(session.messages.len()),
|
||||
@@ -8995,7 +9220,7 @@ impl RuntimeThreadManager {
|
||||
) -> Result<(
|
||||
ThreadRecord,
|
||||
Option<String>,
|
||||
Vec<codewhale_protocol::runtime::RuntimeImageInput>,
|
||||
Vec<codewhale_app_server_protocol::runtime::RuntimeImageInput>,
|
||||
Option<std::num::NonZeroU32>,
|
||||
)> {
|
||||
self.fork_at_user_message_in_sessions_dir(id, depth, self.sessions_dir())
|
||||
@@ -9010,7 +9235,7 @@ impl RuntimeThreadManager {
|
||||
) -> Result<(
|
||||
ThreadRecord,
|
||||
Option<String>,
|
||||
Vec<codewhale_protocol::runtime::RuntimeImageInput>,
|
||||
Vec<codewhale_app_server_protocol::runtime::RuntimeImageInput>,
|
||||
Option<std::num::NonZeroU32>,
|
||||
)> {
|
||||
let (admission, _) = self.thread_restore_guard(id).await?;
|
||||
@@ -9059,7 +9284,7 @@ impl RuntimeThreadManager {
|
||||
) -> Result<(
|
||||
ThreadRecord,
|
||||
Option<String>,
|
||||
Vec<codewhale_protocol::runtime::RuntimeImageInput>,
|
||||
Vec<codewhale_app_server_protocol::runtime::RuntimeImageInput>,
|
||||
Option<std::num::NonZeroU32>,
|
||||
)> {
|
||||
self.fork_at_user_turn_in_sessions_dir(id, turn_id, self.sessions_dir())
|
||||
@@ -9074,7 +9299,7 @@ impl RuntimeThreadManager {
|
||||
) -> Result<(
|
||||
ThreadRecord,
|
||||
Option<String>,
|
||||
Vec<codewhale_protocol::runtime::RuntimeImageInput>,
|
||||
Vec<codewhale_app_server_protocol::runtime::RuntimeImageInput>,
|
||||
Option<std::num::NonZeroU32>,
|
||||
)> {
|
||||
let (admission, _) = self.thread_restore_guard(id).await?;
|
||||
@@ -9103,7 +9328,7 @@ impl RuntimeThreadManager {
|
||||
) -> Result<(
|
||||
ThreadRecord,
|
||||
Option<String>,
|
||||
Vec<codewhale_protocol::runtime::RuntimeImageInput>,
|
||||
Vec<codewhale_app_server_protocol::runtime::RuntimeImageInput>,
|
||||
Option<std::num::NonZeroU32>,
|
||||
)> {
|
||||
let (admission, _) = self.thread_restore_guard(id).await?;
|
||||
@@ -9480,6 +9705,8 @@ impl RuntimeThreadManager {
|
||||
),
|
||||
};
|
||||
forked.saved_session_checkpoint = Some(SavedSessionCheckpoint {
|
||||
publication_observation: None,
|
||||
|
||||
covered_turn_id: covered_turns
|
||||
.checked_sub(1)
|
||||
.map(|index| source_turns[index].id.clone()),
|
||||
@@ -9573,7 +9800,7 @@ impl RuntimeThreadManager {
|
||||
) -> Result<(
|
||||
ThreadRecord,
|
||||
Option<String>,
|
||||
Vec<codewhale_protocol::runtime::RuntimeImageInput>,
|
||||
Vec<codewhale_app_server_protocol::runtime::RuntimeImageInput>,
|
||||
Option<std::num::NonZeroU32>,
|
||||
)> {
|
||||
let (captured_source, prefix, covered_turn_id) = prepared.own_session;
|
||||
@@ -10183,6 +10410,8 @@ impl RuntimeThreadManager {
|
||||
);
|
||||
thread.session_id = Some(operation.receipt.session_id.clone());
|
||||
thread.saved_session_checkpoint = Some(SavedSessionCheckpoint {
|
||||
publication_observation: None,
|
||||
|
||||
covered_turn_id: thread.latest_turn_id.clone(),
|
||||
messages_sha256: session_messages_sha256(checkpoint_messages)?,
|
||||
messages_len: Some(checkpoint_messages.len()),
|
||||
@@ -13015,6 +13244,7 @@ impl RuntimeThreadManager {
|
||||
trust_mode: thread.trust_mode,
|
||||
notes_path: cfg.notes_path(),
|
||||
mcp_config_path: cfg.mcp_config_path(),
|
||||
mcp_auth_context: self.manager_cfg.mcp_auth_context.clone(),
|
||||
mcp_oauth_callback_port: cfg.mcp_oauth_callback_port,
|
||||
mcp_oauth_callback_url: cfg.mcp_oauth_callback_url.clone(),
|
||||
skills_dir: cfg.skills_dir(),
|
||||
@@ -13203,6 +13433,7 @@ impl RuntimeThreadManager {
|
||||
if !session_messages.is_empty() || sys_prompt.is_some() {
|
||||
engine
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
// Same identity the engine was built with: a
|
||||
// re-sync of the same conversation, never a
|
||||
// conversation boundary.
|
||||
@@ -13425,7 +13656,8 @@ impl RuntimeThreadManager {
|
||||
permit = forward.reserve() => match permit {
|
||||
Ok(permit) => { permit.send(event); break; },
|
||||
Err(_) => {
|
||||
if let EngineEvent::SessionUpdated { checkpoint: Some(checkpoint), .. } = &event {
|
||||
if let EngineEvent::SessionUpdated {
|
||||
source: _, checkpoint: Some(checkpoint), .. } = &event {
|
||||
checkpoint.fail("Runtime checkpoint event receiver closed");
|
||||
}
|
||||
break 'pump;
|
||||
@@ -13433,7 +13665,8 @@ impl RuntimeThreadManager {
|
||||
},
|
||||
_ = &mut notified => {},
|
||||
_ = stop.cancelled() => {
|
||||
if let EngineEvent::SessionUpdated { checkpoint: Some(checkpoint), .. } = &event {
|
||||
if let EngineEvent::SessionUpdated {
|
||||
source: _, checkpoint: Some(checkpoint), .. } = &event {
|
||||
checkpoint.fail("Runtime checkpoint event owner stopped");
|
||||
}
|
||||
break 'pump;
|
||||
@@ -13727,6 +13960,8 @@ impl RuntimeThreadManager {
|
||||
return;
|
||||
}
|
||||
stored.saved_session_checkpoint = Some(SavedSessionCheckpoint {
|
||||
publication_observation: None,
|
||||
|
||||
covered_turn_id: covered
|
||||
.checked_sub(1)
|
||||
.and_then(|index| turns.get(index))
|
||||
@@ -14244,6 +14479,10 @@ impl RuntimeThreadManager {
|
||||
// latter; a prepared request is not evidence of delivery or a model
|
||||
// call, and the terminal counters intentionally exclude HTTP retries.
|
||||
let mut turn_model_request_diagnostics: Option<RuntimeTurnRequestDiagnostics> = None;
|
||||
let mut primary_coverage_recorded = false;
|
||||
let mut primary_usage_reported = 0_u32;
|
||||
let mut primary_usage_unknown = 0_u32;
|
||||
let mut primary_usage_missing = 0_u32;
|
||||
let mut turn_status: Option<RuntimeTurnStatus> = None;
|
||||
let mut turn_error: Option<String> = None;
|
||||
let mut saw_engine_activity = false;
|
||||
@@ -14429,6 +14668,7 @@ impl RuntimeThreadManager {
|
||||
created_at,
|
||||
route,
|
||||
submission_id: _,
|
||||
..
|
||||
} => {
|
||||
saw_turn_started = true;
|
||||
engine_turn_id = Some(started_turn_id);
|
||||
@@ -15084,7 +15324,9 @@ impl RuntimeThreadManager {
|
||||
)?;
|
||||
}
|
||||
}
|
||||
EngineEvent::CompactionStarted { id, auto, message } => {
|
||||
EngineEvent::CompactionStarted {
|
||||
id, auto, message, ..
|
||||
} => {
|
||||
let item_id = runtime_record_id("item");
|
||||
compaction_items.insert(id.clone(), item_id.clone());
|
||||
let item = TurnItemRecord {
|
||||
@@ -15570,7 +15812,8 @@ impl RuntimeThreadManager {
|
||||
Some(EngineEvent::UserInputClosed { id, outcome }) => {
|
||||
self.user_input_settlement(&thread_id).close(&id, Some(&turn_id), outcome).await?;
|
||||
}
|
||||
Some(EngineEvent::SessionUpdated { checkpoint: Some(checkpoint), .. }) => {
|
||||
Some(EngineEvent::SessionUpdated {
|
||||
source: None, checkpoint: Some(checkpoint), .. }) => {
|
||||
checkpoint.fail("Runtime checkpoint cannot wait behind an approval");
|
||||
}
|
||||
Some(EngineEvent::ApprovalWithdrawn { id: withdrawn }) if withdrawn == id => {
|
||||
@@ -15945,6 +16188,56 @@ impl RuntimeThreadManager {
|
||||
.await?;
|
||||
}
|
||||
EngineEvent::ProviderRequestReceipt { receipt } => {
|
||||
if receipt["primary"].as_bool() == Some(true) {
|
||||
primary_coverage_recorded = true;
|
||||
let coverage = receipt.get("coverage").cloned().and_then(|value| {
|
||||
serde_json::from_value::<crate::llm_client::ProviderUsageCoverage>(
|
||||
value,
|
||||
)
|
||||
.ok()
|
||||
});
|
||||
let reported = receipt["usage_reported"].as_bool() == Some(true);
|
||||
if reported {
|
||||
primary_usage_reported = primary_usage_reported.saturating_add(1);
|
||||
}
|
||||
if matches!(
|
||||
coverage,
|
||||
Some(
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
| crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
)
|
||||
) {
|
||||
if receipt["outcome"] == "completed"
|
||||
|| receipt["outcome"] == "review_response"
|
||||
{
|
||||
primary_usage_missing = primary_usage_missing.saturating_add(1);
|
||||
} else {
|
||||
primary_usage_unknown = primary_usage_unknown.saturating_add(1);
|
||||
}
|
||||
}
|
||||
let _turn_mutation = self.store.turn_mutation.lock();
|
||||
let mut turn = self.store.load_turn(&turn_id)?;
|
||||
let mut facts = turn.model_request_diagnostics.unwrap_or_else(|| {
|
||||
RuntimeTurnRequestDiagnostics::from(
|
||||
&crate::tool_inspection::TurnStopDiagnostics::default(),
|
||||
)
|
||||
});
|
||||
facts.primary_coverage_recorded = primary_coverage_recorded;
|
||||
facts.primary_usage_reported = primary_usage_reported;
|
||||
facts.primary_usage_unknown = primary_usage_unknown;
|
||||
facts.primary_usage_missing = primary_usage_missing;
|
||||
turn.model_request_diagnostics = Some(facts);
|
||||
if reported && let Some(source) = receipt["source_id"].as_str() {
|
||||
let fingerprint = routed_usage_source_fingerprint(source);
|
||||
if !turn.routed_usage_source_ids.contains(&fingerprint)
|
||||
&& turn.routed_usage_source_ids.len()
|
||||
< MAX_ROUTED_USAGE_RECORDS_PER_TURN
|
||||
{
|
||||
turn.routed_usage_source_ids.push(fingerprint);
|
||||
}
|
||||
}
|
||||
self.store.save_turn(&turn)?;
|
||||
}
|
||||
if receipt["outcome"] == "steered"
|
||||
|| receipt["outcome"] == "steered_before_stream"
|
||||
{
|
||||
@@ -16012,6 +16305,10 @@ impl RuntimeThreadManager {
|
||||
if !admitted_acp {
|
||||
facts.stop_reason = None;
|
||||
}
|
||||
facts.primary_coverage_recorded = primary_coverage_recorded;
|
||||
facts.primary_usage_reported = primary_usage_reported;
|
||||
facts.primary_usage_unknown = primary_usage_unknown;
|
||||
facts.primary_usage_missing = primary_usage_missing;
|
||||
turn_model_request_diagnostics = Some(facts);
|
||||
}
|
||||
// Keep the existing bounded request projection in the
|
||||
@@ -16283,7 +16580,8 @@ impl RuntimeThreadManager {
|
||||
.routed_usage_dropped_records
|
||||
.saturating_add(background_residual)
|
||||
.saturating_add(turn_routed_usage_dropped_records);
|
||||
turn.model_request_diagnostics = turn_model_request_diagnostics;
|
||||
turn.model_request_diagnostics =
|
||||
turn_model_request_diagnostics.or(turn.model_request_diagnostics);
|
||||
turn.error = turn_error;
|
||||
// Item refs are final now. The workspace delta derives from the
|
||||
// pre/post-turn restore points recorded on this turn (every
|
||||
|
||||
@@ -8,7 +8,7 @@ pub async fn history_owner_work<T: Send + 'static>(
|
||||
) -> Result<T> {
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
let ticket = crate::test_support::env_scope_ticket();
|
||||
codewhale_app_server::daemon_socket::owner_work(move || {
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
let _membership = crate::test_support::join_env_scope(ticket);
|
||||
work()
|
||||
|
||||
@@ -6,6 +6,9 @@ use crate::core::events::{Event as EngineEvent, TurnOutcomeStatus};
|
||||
use crate::core::ops::{Op, TurnSpec};
|
||||
use crate::route_budget::known_route_limits;
|
||||
use crate::tools::subagent::SubAgentStatus;
|
||||
use codewhale_app_server_protocol::runtime::{
|
||||
DynamicToolCallContent, DynamicToolCallParams, DynamicToolCallResult,
|
||||
};
|
||||
use codewhale_execpolicy::ApprovalMode;
|
||||
use codewhale_models::{SystemPrompt, ToolCaller};
|
||||
use codewhale_protocol::agent_mail::{
|
||||
@@ -13,9 +16,6 @@ use codewhale_protocol::agent_mail::{
|
||||
AgentMailSendRequest,
|
||||
};
|
||||
use codewhale_protocol::agent_mail::{AgentMailDeliveryMode, AgentMailSender, AgentMailStatus};
|
||||
use codewhale_protocol::runtime::{
|
||||
DynamicToolCallContent, DynamicToolCallParams, DynamicToolCallResult,
|
||||
};
|
||||
use codewhale_runtime::config::Config;
|
||||
use codewhale_runtime::runtime_policy::RuntimePolicyProjection;
|
||||
use std::time::{Duration, Instant};
|
||||
@@ -82,6 +82,7 @@ mod recovery {
|
||||
fs::create_dir(&workspace)?;
|
||||
let runtime_dir = dir.path().join("runtime");
|
||||
let manager_cfg = RuntimeThreadManagerConfig {
|
||||
mcp_auth_context: Default::default(),
|
||||
max_active_threads: 1,
|
||||
..test_manager_config(runtime_dir)
|
||||
};
|
||||
@@ -196,6 +197,7 @@ mod recovery {
|
||||
let workspace = dir.path().join("workspace");
|
||||
fs::create_dir(&workspace)?;
|
||||
let manager_cfg = RuntimeThreadManagerConfig {
|
||||
mcp_auth_context: Default::default(),
|
||||
max_active_threads: 1,
|
||||
..test_manager_config(dir.path().join("runtime"))
|
||||
};
|
||||
@@ -533,6 +535,8 @@ mod recovery {
|
||||
];
|
||||
for len in 0..=messages.len() {
|
||||
let checkpoint = SavedSessionCheckpoint {
|
||||
publication_observation: None,
|
||||
|
||||
covered_turn_id: None,
|
||||
messages_sha256: session_messages_sha256(&messages[..len])?,
|
||||
messages_len: None,
|
||||
@@ -546,6 +550,8 @@ mod recovery {
|
||||
assert_eq!(checkpoint_prefix_len(&exact, &messages)?, Some(len));
|
||||
}
|
||||
let diverged = SavedSessionCheckpoint {
|
||||
publication_observation: None,
|
||||
|
||||
covered_turn_id: None,
|
||||
messages_sha256: session_messages_sha256(&[msg(Role::User, "other")])?,
|
||||
messages_len: Some(1),
|
||||
@@ -2585,6 +2591,7 @@ impl Drop for RuntimeEventChildGuard {
|
||||
|
||||
fn test_manager_config(data_dir: PathBuf) -> RuntimeThreadManagerConfig {
|
||||
RuntimeThreadManagerConfig {
|
||||
mcp_auth_context: Default::default(),
|
||||
task_data_dir: data_dir.clone(),
|
||||
sessions_dir: None,
|
||||
data_dir,
|
||||
@@ -4549,6 +4556,7 @@ async fn compact_lifecycle_outlives_caller_and_preserves_concurrent_thread_updat
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::CompactionStarted {
|
||||
previous_source: None,
|
||||
id: "manual_owned".to_string(),
|
||||
auto: false,
|
||||
message: "compaction started".to_string(),
|
||||
@@ -8663,6 +8671,7 @@ async fn thread_lifecycle_persists_across_restart() -> Result<()> {
|
||||
if matches!(rx_op.recv().await, Some(Op::SendMessage(TurnSpec { .. }))) {
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_turn_1".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -8834,6 +8843,7 @@ async fn initial_classifier_usage_is_persisted_before_terminal_and_merged_exactl
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_classifier_receipt".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -9422,6 +9432,7 @@ async fn monitor_deduplicates_sink_and_metadata_and_persists_metadata_only_missi
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_metadata_receipt".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -9509,6 +9520,7 @@ async fn monitor_separates_lifecycle_start_from_billing_dispatch_and_child_usage
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_route_receipt".to_string(),
|
||||
created_at: started_at,
|
||||
route: None,
|
||||
@@ -9687,6 +9699,7 @@ async fn monitor_separates_lifecycle_start_from_billing_dispatch_and_child_usage
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: second_engine_turn.to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -9770,6 +9783,7 @@ async fn monitor_persists_request_snapshots_and_matching_terminal_diagnostics()
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: engine_turn_id.to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -10006,6 +10020,10 @@ async fn monitor_persists_request_snapshots_and_matching_terminal_diagnostics()
|
||||
assert_eq!(
|
||||
completed.model_request_diagnostics,
|
||||
Some(RuntimeTurnRequestDiagnostics {
|
||||
primary_coverage_recorded: false,
|
||||
primary_usage_reported: 0,
|
||||
primary_usage_unknown: 0,
|
||||
primary_usage_missing: 0,
|
||||
stop_reason: None,
|
||||
model_requests_started: 2,
|
||||
transport_retries: 3,
|
||||
@@ -10096,6 +10114,7 @@ async fn monitor_persists_request_snapshots_and_matching_terminal_diagnostics()
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: second_engine_turn_id.to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -10191,6 +10210,7 @@ async fn completed_turn_without_engine_output_fails() -> Result<()> {
|
||||
if matches!(rx_op.recv().await, Some(Op::SendMessage(TurnSpec { .. }))) {
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_empty_turn".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -10319,6 +10339,7 @@ async fn worker_lifecycle_receipts_preserve_owner_outcome_and_durable_replay() -
|
||||
}
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_worker_lifecycle".into(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -10376,6 +10397,7 @@ async fn worker_lifecycle_receipts_preserve_owner_outcome_and_durable_replay() -
|
||||
}
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_next_parent".into(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -10667,6 +10689,7 @@ async fn preturn_control_status_does_not_make_empty_turn_succeed() -> Result<()>
|
||||
.await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_empty_after_control_status".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -10728,6 +10751,7 @@ async fn engine_error_remains_failed_after_nominal_turn_complete() -> Result<()>
|
||||
if matches!(rx_op.recv().await, Some(Op::SendMessage(TurnSpec { .. }))) {
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_error_then_complete".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -11311,6 +11335,7 @@ async fn compact_interrupt_persists_canceled_item_for_the_exact_request() -> Res
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::CompactionStarted {
|
||||
previous_source: None,
|
||||
id: compaction_id.clone(),
|
||||
auto: false,
|
||||
message: "compaction started".to_string(),
|
||||
@@ -11556,6 +11581,7 @@ async fn multi_turn_continuity_same_thread() -> Result<()> {
|
||||
turn_index = turn_index.saturating_add(1);
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: format!("engine_turn_{turn_index}"),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -11803,6 +11829,7 @@ async fn host_goal_loop_kickoff_arms_one_continuation_and_parks_at_engine_cap()
|
||||
let _ = tx_event.send(EngineEvent::GoalUpdated { snapshot }).await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: format!("engine_goal_{pass}"),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -11925,6 +11952,7 @@ async fn host_goal_loop_skips_rearm_without_update_goal_and_after_failed_pass()
|
||||
let _ = tx_event.send(EngineEvent::GoalUpdated { snapshot }).await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_goal_no_update".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -12019,6 +12047,7 @@ async fn host_goal_loop_skips_rearm_without_update_goal_and_after_failed_pass()
|
||||
.await;
|
||||
let _ = failed_tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_goal_failed".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -12120,6 +12149,7 @@ async fn host_goal_loop_mirrors_terminal_snapshot_and_does_not_rearm() -> Result
|
||||
let _ = tx_event.send(EngineEvent::GoalUpdated { snapshot }).await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: format!("engine_{status}"),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -12231,6 +12261,7 @@ async fn model_created_goal_persists_through_adopted_revision() -> Result<()> {
|
||||
let _ = tx_event.send(EngineEvent::GoalUpdated { snapshot }).await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_model_created".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -12318,6 +12349,7 @@ async fn model_created_goal_never_overwrites_concurrent_explicit_goal() -> Resul
|
||||
let _ = tx_event.send(EngineEvent::GoalUpdated { snapshot }).await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_concurrent".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -13014,6 +13046,7 @@ async fn interrupt_turn_marks_interrupted_after_cleanup() -> Result<()> {
|
||||
if matches!(rx_op.recv().await, Some(Op::SendMessage(TurnSpec { .. }))) {
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_turn_interrupt".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -14453,6 +14486,7 @@ async fn thread_detail_cursor_precedes_projection_reads_at_terminal_boundary() -
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "snapshot_terminal".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -14638,6 +14672,7 @@ async fn thread_detail_materializes_stream_prefixes_before_their_delta_cursor()
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "delta_snapshot".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -14776,6 +14811,7 @@ async fn superseded_reasoning_segment_closes_when_the_retried_stream_starts() ->
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "retried_stream".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -14925,6 +14961,7 @@ async fn turn_end_closes_a_tool_call_that_never_completed() -> Result<()> {
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "unanswered_tool_call".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -15086,6 +15123,7 @@ async fn thread_detail_delta_boundary_is_replay_idempotent() -> Result<()> {
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "delta_boundary".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -15445,6 +15483,7 @@ async fn dynamic_tool_result_settles_snapshot_and_emits_one_safe_resolution() ->
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "dynamic_result".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -15610,6 +15649,7 @@ async fn dynamic_tool_result_receipt_outlives_canceled_delivery_future() -> Resu
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "dynamic_detached_settlement".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -16556,6 +16596,7 @@ async fn dynamic_tool_timeout_clears_snapshot_and_emits_once() -> Result<()> {
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "dynamic_timeout".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -16647,6 +16688,7 @@ async fn terminal_turn_cancels_pending_dynamic_tool_exactly_once() -> Result<()>
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "dynamic_cancel".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -16758,6 +16800,7 @@ async fn engine_withdrawal_retires_approval_and_unblocks_queued_events() -> Resu
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_withdrawal".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -16913,6 +16956,7 @@ async fn approval_wait_heartbeat_is_never_sequenced_after_the_decision() -> Resu
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_turn_wait".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -18475,6 +18519,7 @@ async fn steer_turn_on_active_turn_records_item_and_event() -> Result<()> {
|
||||
if matches!(rx_op.recv().await, Some(Op::SendMessage(TurnSpec { .. }))) {
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_turn_steer".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -19144,6 +19189,8 @@ fn answer_session_snapshot(
|
||||
.expect("snapshot message");
|
||||
if let Some(tx) = tx.lock().expect("snapshot sender").take() {
|
||||
let _ = tx.send(crate::core::ops::SessionSnapshot {
|
||||
source: None,
|
||||
|
||||
session_id: String::new(),
|
||||
messages,
|
||||
total_tokens: 0,
|
||||
@@ -19201,6 +19248,7 @@ async fn compaction_lifecycle_emits_item_events_with_compaction_counts() -> Resu
|
||||
op_count = op_count.saturating_add(1);
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_turn_auto".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
route: None,
|
||||
@@ -19217,6 +19265,11 @@ async fn compaction_lifecycle_emits_item_events_with_compaction_counts() -> Resu
|
||||
{"role":"user","content":[{"type":"text","text":"auto checkpoint prefix"}]}
|
||||
])).unwrap();
|
||||
let snapshot = Arc::new(crate::core::ops::SessionSnapshot {
|
||||
source: Some(crate::core::ops::SessionSnapshotSource::turn(
|
||||
"engine_turn_auto".into(),
|
||||
None,
|
||||
)),
|
||||
|
||||
session_id: checkpoint_thread.clone(),
|
||||
messages: messages.clone(),
|
||||
total_tokens: 0,
|
||||
@@ -19242,6 +19295,7 @@ async fn compaction_lifecycle_emits_item_events_with_compaction_counts() -> Resu
|
||||
);
|
||||
tx_event
|
||||
.send(EngineEvent::SessionUpdated {
|
||||
source: request.snapshot.source.clone(),
|
||||
checkpoint: Some(request),
|
||||
session_id: checkpoint_thread.clone(),
|
||||
messages: Arc::new(messages),
|
||||
@@ -19264,6 +19318,7 @@ async fn compaction_lifecycle_emits_item_events_with_compaction_counts() -> Resu
|
||||
}
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::CompactionStarted {
|
||||
previous_source: None,
|
||||
id: "auto_compact_1".to_string(),
|
||||
auto: true,
|
||||
message: "auto compact begin".to_string(),
|
||||
@@ -19307,6 +19362,7 @@ async fn compaction_lifecycle_emits_item_events_with_compaction_counts() -> Resu
|
||||
op_count = op_count.saturating_add(1);
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::CompactionStarted {
|
||||
previous_source: None,
|
||||
id: "manual_compact_1".to_string(),
|
||||
auto: false,
|
||||
message: "manual compact begin".to_string(),
|
||||
@@ -21289,7 +21345,7 @@ mod runtime_image_inputs {
|
||||
(
|
||||
"look",
|
||||
"deepseek-v4-flash-vision-exp",
|
||||
codewhale_protocol::runtime::RuntimeImageInput {
|
||||
codewhale_app_server_protocol::runtime::RuntimeImageInput {
|
||||
mime: "image/jpeg".into(),
|
||||
..good
|
||||
},
|
||||
@@ -21502,6 +21558,7 @@ mod runtime_image_inputs {
|
||||
let run = tokio::spawn(engine.run());
|
||||
handle
|
||||
.send(Op::SyncSession {
|
||||
control: crate::core::ops::SessionControlId::new(),
|
||||
session_id: stored_thread.session_id.clone(),
|
||||
messages: reopened.restore_thread_messages(&stored_thread)?,
|
||||
system_prompt: None,
|
||||
@@ -21761,6 +21818,7 @@ mod runtime_image_inputs {
|
||||
// and observe its durable settlement before restoring that case.
|
||||
for event in [
|
||||
EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: turn.id.clone(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -21800,7 +21858,7 @@ mod runtime_image_inputs {
|
||||
fn runtime_image_native_fingerprint_keeps_legacy_text_and_binds_bytes() -> Result<()> {
|
||||
let thread = sample_thread("thr_image_legacy");
|
||||
let policy = RuntimePolicyProjection::from_persisted("agent", Some("ask"), false);
|
||||
let fingerprint = |images: &[codewhale_protocol::runtime::RuntimeImageInput]| {
|
||||
let fingerprint = |images: &[codewhale_app_server_protocol::runtime::RuntimeImageInput]| {
|
||||
runtime_turn_request_fingerprint(
|
||||
&thread,
|
||||
"hello",
|
||||
@@ -21949,6 +22007,7 @@ async fn notices_raise_from_engine_events_and_clear_on_settle_or_ack() -> Result
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: turn.id.clone(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -23012,6 +23071,7 @@ async fn engine_plumbing_items_are_tagged_internal_and_retry_hints_are_dropped()
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine_plumbing_visibility".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -23783,6 +23843,7 @@ async fn runtime_receipts_mask_configured_secrets() -> Result<()> {
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: turn.id.clone(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -24042,6 +24103,7 @@ async fn tool_completion_items_carry_typed_artifact_refs() -> Result<()> {
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: turn.id.clone(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -24285,6 +24347,7 @@ async fn run_workspace_turn(
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: "engine-turn".to_string(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -24896,6 +24959,7 @@ mod execution_identity {
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: turn.id.clone(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -25136,6 +25200,7 @@ mod execution_identity {
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnStarted {
|
||||
previous_source: None,
|
||||
turn_id: turn.id.clone(),
|
||||
created_at: Utc::now(),
|
||||
route: None,
|
||||
@@ -25328,6 +25393,10 @@ fn control_owner_capture_uses_held_lease_and_new_generation_after_restart() -> R
|
||||
#[test]
|
||||
fn ordinary_terminal_request_facts_omit_private_acp_stop_projection() {
|
||||
let facts = RuntimeTurnRequestDiagnostics {
|
||||
primary_coverage_recorded: false,
|
||||
primary_usage_reported: 0,
|
||||
primary_usage_unknown: 0,
|
||||
primary_usage_missing: 0,
|
||||
stop_reason: None,
|
||||
model_requests_started: 2,
|
||||
transport_retries: 0,
|
||||
|
||||
+1596
-412
File diff suppressed because it is too large.
Load diff
@@ -314,6 +314,8 @@
|
||||
assert_eq!(restored.metadata.cost.cny_unpriced_turns, 1);
|
||||
assert_eq!(restored.metadata.cost.usage_source_fingerprints.len(), 1);
|
||||
if malformed == "tombstone" {
|
||||
assert!(restored.write_receipt.is_none());
|
||||
assert!(restored.capture_publication(SessionWriteOwner::control()).is_err());
|
||||
assert!(
|
||||
manager.save_session(&restored).is_err(),
|
||||
"an invalid deletion marker must fail closed for writes"
|
||||
|
||||
@@ -7,11 +7,17 @@
|
||||
updated_at: DateTime<Utc>,
|
||||
) {
|
||||
let session = SavedSession {
|
||||
schema_version: CURRENT_SESSION_SCHEMA_VERSION,
|
||||
write_receipt: None,
|
||||
|
||||
schema_version: CURRENT_SESSION_SCHEMA_VERSION,
|
||||
messages: vec![make_test_message("user", "hi")],
|
||||
metadata: SessionMetadata {
|
||||
id: id.to_string(),
|
||||
publication_id: None,
|
||||
native_mutation_observation: None,
|
||||
|
||||
id: id.to_string(),
|
||||
title: format!("session-{id}"),
|
||||
title_is_custom: false,
|
||||
created_at: updated_at,
|
||||
updated_at,
|
||||
message_count: 1,
|
||||
@@ -51,11 +57,17 @@
|
||||
updated_at: DateTime<Utc>,
|
||||
) {
|
||||
let session = SavedSession {
|
||||
schema_version: CURRENT_SESSION_SCHEMA_VERSION,
|
||||
write_receipt: None,
|
||||
|
||||
schema_version: CURRENT_SESSION_SCHEMA_VERSION,
|
||||
messages: Vec::new(),
|
||||
metadata: SessionMetadata {
|
||||
id: id.to_string(),
|
||||
publication_id: None,
|
||||
native_mutation_observation: None,
|
||||
|
||||
id: id.to_string(),
|
||||
title: DEFAULT_SESSION_TITLE.to_string(),
|
||||
title_is_custom: false,
|
||||
created_at: updated_at,
|
||||
updated_at,
|
||||
message_count: 0,
|
||||
|
||||
@@ -579,6 +579,7 @@
|
||||
.unwrap()
|
||||
.is_some()
|
||||
);
|
||||
fs::remove_file(checkpoints.join("latest.json")).expect("remove malformed listing fixture");
|
||||
let mut collision = session.clone();
|
||||
collision.metadata.id = format!("{}-other", session.metadata.id);
|
||||
manager.save_checkpoint(&collision).expect("save colliding checkpoint");
|
||||
|
||||
@@ -346,8 +346,12 @@ mod tests {
|
||||
fn metadata(id: &str, title: &str, workspace: &str, minutes_ago: i64) -> SessionMetadata {
|
||||
let ts = Utc::now() - Duration::minutes(minutes_ago);
|
||||
SessionMetadata {
|
||||
publication_id: None,
|
||||
native_mutation_observation: None,
|
||||
|
||||
id: id.to_string(),
|
||||
title: title.to_string(),
|
||||
title_is_custom: false,
|
||||
created_at: ts,
|
||||
updated_at: ts,
|
||||
message_count: title.len(),
|
||||
|
||||
@@ -46,7 +46,10 @@ use serde_json::{Value, json};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use crate::runtime_threads::{RuntimeStoreBinding, ThreadRecord};
|
||||
use crate::session_manager::{SavedSession, SessionManager, is_runtime_store_dir_name};
|
||||
use crate::session_manager::{
|
||||
SavedSession, SessionManager, is_offline_queue_file, is_runtime_store_dir_name,
|
||||
read_session_reference_bytes,
|
||||
};
|
||||
|
||||
pub const SET_ASIDE_DIR: &str = ".set-aside";
|
||||
const RECONCILE_DIR: &str = ".reconcile";
|
||||
@@ -124,11 +127,14 @@ impl ReconcileSummary {
|
||||
!self.dry_run && self.actions() > 0
|
||||
}
|
||||
|
||||
/// One line for the first frame, only when something changed.
|
||||
/// One line for the first frame when something changed or inspection failed.
|
||||
#[must_use]
|
||||
pub fn notice(&self) -> Option<String> {
|
||||
if !self.changed() {
|
||||
return None;
|
||||
return self
|
||||
.errors
|
||||
.first()
|
||||
.map(|error| format!("Session inspection incomplete: {error}"));
|
||||
}
|
||||
let mut parts = Vec::new();
|
||||
if self.sessions_recovered > 0 {
|
||||
@@ -322,6 +328,10 @@ pub fn retire_unbound_store(
|
||||
return None;
|
||||
}
|
||||
let references = collect_document_references(sessions.sessions_dir(), false);
|
||||
if !references.incomplete.is_empty() {
|
||||
tracing::warn!(store = %store.display(), error = %references.incomplete[0], "released store kept because reference inspection is incomplete");
|
||||
return None;
|
||||
}
|
||||
if references.binds(store) {
|
||||
return None;
|
||||
}
|
||||
@@ -475,9 +485,14 @@ struct References {
|
||||
document_tokens: HashSet<String>,
|
||||
/// Document texts, kept only to answer non-uuid ids (rare).
|
||||
document_texts: Vec<String>,
|
||||
incomplete: Vec<String>,
|
||||
}
|
||||
|
||||
impl References {
|
||||
fn failed(&mut self, path: &Path, error: impl std::fmt::Display) {
|
||||
self.incomplete.push(format!("{}: {error}", path.display()));
|
||||
}
|
||||
|
||||
fn binds(&self, store: &Path) -> bool {
|
||||
self.bound_stores.contains(&canonical(store)) || self.bound_stores.contains(store)
|
||||
}
|
||||
@@ -525,28 +540,59 @@ fn document_id(path: &Path) -> Option<String> {
|
||||
/// `keep_text`, also collect what R6 needs to know which ids documents mention.
|
||||
fn collect_document_references(sessions_dir: &Path, keep_text: bool) -> References {
|
||||
let mut references = References::default();
|
||||
let checkpoints_dir = sessions_dir.join(CHECKPOINTS_DIR);
|
||||
let scan = |dir: &Path, references: &mut References, keep_text: bool| {
|
||||
let Ok(entries) = fs::read_dir(dir) else {
|
||||
return;
|
||||
let entries = match fs::read_dir(dir) {
|
||||
Ok(entries) => entries,
|
||||
Err(error) if error.kind() == io::ErrorKind::NotFound => return,
|
||||
Err(error) => {
|
||||
references.failed(dir, error);
|
||||
return;
|
||||
}
|
||||
};
|
||||
for entry in entries.flatten() {
|
||||
for entry in entries {
|
||||
let entry = match entry {
|
||||
Ok(entry) => entry,
|
||||
Err(error) => {
|
||||
references.failed(dir, error);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let path = entry.path();
|
||||
if dir == checkpoints_dir
|
||||
&& path
|
||||
.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.is_some_and(is_offline_queue_file)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let Some(id) = document_id(&path) else {
|
||||
continue;
|
||||
};
|
||||
let Ok(metadata) = SessionManager::load_session_metadata(&path) else {
|
||||
continue;
|
||||
};
|
||||
references.documents.insert(id);
|
||||
let metadata = match SessionManager::load_session_metadata(&path) {
|
||||
Ok(metadata) => metadata,
|
||||
Err(error) => {
|
||||
references.failed(&path, error);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
references.documents.insert(metadata.id);
|
||||
if let Some(binding) = metadata.runtime_store {
|
||||
references.bound_stores.insert(canonical(&binding.data_dir));
|
||||
references.bound_stores.insert(binding.data_dir);
|
||||
}
|
||||
if keep_text && let Ok(bytes) = fs::read(&path) {
|
||||
references.document_tokens.extend(uuid_tokens(&bytes));
|
||||
references
|
||||
.document_texts
|
||||
.push(String::from_utf8_lossy(&bytes).into_owned());
|
||||
if keep_text {
|
||||
match read_session_reference_bytes(&path) {
|
||||
Ok(bytes) => {
|
||||
references.document_tokens.extend(uuid_tokens(&bytes));
|
||||
references
|
||||
.document_texts
|
||||
.push(String::from_utf8_lossy(&bytes).into_owned());
|
||||
}
|
||||
Err(error) => references.failed(&path, error),
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
@@ -596,8 +642,11 @@ impl Run<'_> {
|
||||
fn execute(&mut self) {
|
||||
self.repair_documents();
|
||||
let mut references = collect_document_references(&self.sessions_dir, true);
|
||||
let stores = self.store_dirs();
|
||||
let stores = self.store_dirs(&mut references);
|
||||
self.collect_thread_references(&stores, &mut references);
|
||||
self.summary
|
||||
.errors
|
||||
.extend(references.incomplete.iter().cloned());
|
||||
for store in &stores {
|
||||
if !self.budget_left() {
|
||||
return;
|
||||
@@ -612,43 +661,91 @@ impl Run<'_> {
|
||||
|
||||
/// R1: set aside documents that cannot be read at all.
|
||||
fn repair_documents(&mut self) {
|
||||
let Ok(entries) = fs::read_dir(&self.sessions_dir) else {
|
||||
return;
|
||||
let entries = match fs::read_dir(&self.sessions_dir) {
|
||||
Ok(entries) => entries,
|
||||
Err(error) => {
|
||||
self.error("session documents", error);
|
||||
return;
|
||||
}
|
||||
};
|
||||
let mut paths: Vec<PathBuf> = entries.flatten().map(|entry| entry.path()).collect();
|
||||
let mut paths = Vec::new();
|
||||
for entry in entries {
|
||||
match entry {
|
||||
Ok(entry) => paths.push(entry.path()),
|
||||
Err(error) => self.error("session documents", error),
|
||||
}
|
||||
}
|
||||
paths.sort();
|
||||
for path in paths {
|
||||
let Some(id) = document_id(&path) else {
|
||||
continue;
|
||||
};
|
||||
if self.options.skip_session.as_deref() == Some(id.as_str())
|
||||
|| SessionManager::load_session_metadata(&path).is_ok()
|
||||
{
|
||||
if self.options.skip_session.as_deref() == Some(id.as_str()) {
|
||||
continue;
|
||||
}
|
||||
match classify_unlisted_document(&path) {
|
||||
DocumentState::Readable => continue,
|
||||
DocumentState::NewerSchema => {
|
||||
self.summary.documents_newer_schema += 1;
|
||||
match SessionManager::load_session_metadata(&path) {
|
||||
Ok(_) => continue,
|
||||
Err(error) if error.kind() == io::ErrorKind::InvalidData => {}
|
||||
Err(error) => {
|
||||
self.error(&path.display().to_string(), error);
|
||||
continue;
|
||||
}
|
||||
DocumentState::Unreadable(reason) => {
|
||||
if !self.budget_left() {
|
||||
return;
|
||||
}
|
||||
self.set_aside_path(&path, &format!("unreadable document: {reason}"), true);
|
||||
self.summary.documents_set_aside += 1;
|
||||
}
|
||||
let sessions = self.sessions;
|
||||
let result = sessions.with_session_file_lock(&id, || {
|
||||
match SessionManager::load_session_metadata(&path) {
|
||||
Ok(_) => return Ok(()),
|
||||
Err(error) if error.kind() == io::ErrorKind::InvalidData => {}
|
||||
Err(error) => return Err(error),
|
||||
}
|
||||
match classify_unlisted_document(&path) {
|
||||
DocumentState::Readable => {}
|
||||
DocumentState::NewerSchema => self.summary.documents_newer_schema += 1,
|
||||
DocumentState::Unavailable(reason) => {
|
||||
self.error(&path.display().to_string(), reason)
|
||||
}
|
||||
DocumentState::Unreadable(reason) => {
|
||||
if self.budget_left() {
|
||||
let errors = self.summary.errors.len();
|
||||
self.set_aside_path(
|
||||
&path,
|
||||
&format!("unreadable document: {reason}"),
|
||||
true,
|
||||
);
|
||||
if self.summary.errors.len() == errors {
|
||||
self.summary.documents_set_aside += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
});
|
||||
if let Err(error) = result {
|
||||
self.error(&path.display().to_string(), error);
|
||||
}
|
||||
if self.summary.limit_reached {
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn store_dirs(&self) -> Vec<StoreDir> {
|
||||
fn store_dirs(&self, references: &mut References) -> Vec<StoreDir> {
|
||||
let mut out = Vec::new();
|
||||
let Ok(entries) = fs::read_dir(&self.sessions_dir) else {
|
||||
return out;
|
||||
let entries = match fs::read_dir(&self.sessions_dir) {
|
||||
Ok(entries) => entries,
|
||||
Err(error) => {
|
||||
references.failed(&self.sessions_dir, error);
|
||||
return out;
|
||||
}
|
||||
};
|
||||
for entry in entries.flatten() {
|
||||
for entry in entries {
|
||||
let entry = match entry {
|
||||
Ok(entry) => entry,
|
||||
Err(error) => {
|
||||
references.failed(&self.sessions_dir, error);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let owner = entry.file_name();
|
||||
let Some(owner_id) = owner.to_str() else {
|
||||
continue;
|
||||
@@ -658,10 +755,21 @@ impl Run<'_> {
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let Ok(children) = fs::read_dir(entry.path()) else {
|
||||
continue;
|
||||
let children = match fs::read_dir(entry.path()) {
|
||||
Ok(children) => children,
|
||||
Err(error) => {
|
||||
references.failed(&entry.path(), error);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
for child in children.flatten() {
|
||||
for child in children {
|
||||
let child = match child {
|
||||
Ok(child) => child,
|
||||
Err(error) => {
|
||||
references.failed(&entry.path(), error);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if is_runtime_store_dir_name(&child.file_name()) && child.path().is_dir() {
|
||||
out.push(StoreDir {
|
||||
owner_id: owner_id.to_string(),
|
||||
@@ -681,14 +789,36 @@ impl Run<'_> {
|
||||
let mut roots: Vec<PathBuf> = stores.iter().map(|store| store.path.clone()).collect();
|
||||
let tasks = crate::task_manager::default_tasks_dir().join("runtime");
|
||||
roots.push(tasks.clone());
|
||||
if let Ok(entries) = fs::read_dir(&tasks) {
|
||||
roots.extend(entries.flatten().map(|entry| entry.path()));
|
||||
match fs::read_dir(&tasks) {
|
||||
Ok(entries) => {
|
||||
for entry in entries {
|
||||
match entry {
|
||||
Ok(entry) => roots.push(entry.path()),
|
||||
Err(error) => references.failed(&tasks, error),
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(error) if error.kind() == io::ErrorKind::NotFound => {}
|
||||
Err(error) => references.failed(&tasks, error),
|
||||
}
|
||||
for root in roots {
|
||||
let Ok(entries) = fs::read_dir(root.join("threads")) else {
|
||||
continue;
|
||||
let threads = root.join("threads");
|
||||
let entries = match fs::read_dir(&threads) {
|
||||
Ok(entries) => entries,
|
||||
Err(error) if error.kind() == io::ErrorKind::NotFound => continue,
|
||||
Err(error) => {
|
||||
references.failed(&threads, error);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
for entry in entries.flatten() {
|
||||
for entry in entries {
|
||||
let entry = match entry {
|
||||
Ok(entry) => entry,
|
||||
Err(error) => {
|
||||
references.failed(&threads, error);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let path = entry.path();
|
||||
if path.extension().is_none_or(|ext| ext != "json") {
|
||||
continue;
|
||||
@@ -698,17 +828,25 @@ impl Run<'_> {
|
||||
.thread_named
|
||||
.insert(crate::runtime_threads::thread_session_id(thread_id));
|
||||
}
|
||||
if let Ok(raw) = fs::read(&path)
|
||||
&& let Ok(value) = serde_json::from_slice::<Value>(&raw)
|
||||
&& let Some(session_id) = value.get("session_id").and_then(Value::as_str)
|
||||
{
|
||||
references.thread_named.insert(session_id.to_string());
|
||||
match read_session_reference_bytes(&path).and_then(|raw| {
|
||||
serde_json::from_slice::<Value>(&raw)
|
||||
.map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))
|
||||
}) {
|
||||
Ok(value) => {
|
||||
if let Some(session_id) = value.get("session_id").and_then(Value::as_str) {
|
||||
references.thread_named.insert(session_id.to_string());
|
||||
}
|
||||
}
|
||||
Err(error) => references.failed(&path, error),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn repair_store(&mut self, store: &StoreDir, references: &References) {
|
||||
if !references.incomplete.is_empty() {
|
||||
return;
|
||||
}
|
||||
let binding = match RuntimeStoreBinding::for_store_dir(&store.path) {
|
||||
Ok(binding) => binding,
|
||||
Err(error) => return self.error(&format!("store {}", store.path.display()), error),
|
||||
@@ -857,8 +995,12 @@ impl Run<'_> {
|
||||
// A run that stopped between saving the document and binding the
|
||||
// thread left the document behind at this same id; reuse it.
|
||||
if exists(&id) || self.sessions.session_document_exists(&id) {
|
||||
return Ok(self.sessions.load_session(&id)?);
|
||||
return Ok(self.sessions.resume_session(&id)?.session);
|
||||
}
|
||||
let source = self
|
||||
.sessions
|
||||
.expect_absent_session(&id)?
|
||||
.capture(crate::session_manager::SessionWriteOwner::control())?;
|
||||
let mut session = crate::session_manager::create_saved_session_with_id_and_mode(
|
||||
id,
|
||||
&recovered.messages,
|
||||
@@ -884,13 +1026,18 @@ impl Run<'_> {
|
||||
.unwrap_or_else(|| session.metadata.title.clone());
|
||||
session.metadata.title = recovered_title(&label);
|
||||
session.metadata.runtime_store = Some(held.binding().clone());
|
||||
self.sessions.save_session(&session)?;
|
||||
session.write_receipt = Some(source);
|
||||
let publication = self.sessions.publish_session(&session)?;
|
||||
session.adopt_publication(&publication);
|
||||
Ok(session)
|
||||
}
|
||||
|
||||
/// R6: directories with no document and no store that hold only
|
||||
/// artifacts or approval receipts nothing names.
|
||||
fn repair_artifact_dirs(&mut self, references: &References) {
|
||||
if !references.incomplete.is_empty() {
|
||||
return;
|
||||
}
|
||||
let Ok(entries) = fs::read_dir(&self.sessions_dir) else {
|
||||
return;
|
||||
};
|
||||
@@ -937,10 +1084,10 @@ impl Run<'_> {
|
||||
return;
|
||||
}
|
||||
let detail = if hash {
|
||||
fs::read(path).map_or(
|
||||
Value::Null,
|
||||
|bytes| json!({"sha256": file_sha256(&bytes), "bytes": bytes.len()}),
|
||||
)
|
||||
match read_session_reference_bytes(path) {
|
||||
Ok(bytes) => json!({"sha256": file_sha256(&bytes), "bytes": bytes.len()}),
|
||||
Err(error) => return self.error(&format!("set aside {}", path.display()), error),
|
||||
}
|
||||
} else {
|
||||
json!({"kind": "directory"})
|
||||
};
|
||||
@@ -970,14 +1117,15 @@ enum DocumentState {
|
||||
Readable,
|
||||
NewerSchema,
|
||||
Unreadable(String),
|
||||
Unavailable(String),
|
||||
}
|
||||
|
||||
/// A document `list_sessions` could not list: newer, readable after all, or
|
||||
/// genuinely unreadable.
|
||||
fn classify_unlisted_document(path: &Path) -> DocumentState {
|
||||
let bytes = match fs::read(path) {
|
||||
let bytes = match read_session_reference_bytes(path) {
|
||||
Ok(bytes) => bytes,
|
||||
Err(error) => return DocumentState::Unreadable(error.to_string()),
|
||||
Err(error) => return DocumentState::Unavailable(error.to_string()),
|
||||
};
|
||||
if let Ok(value) = serde_json::from_slice::<Value>(&bytes)
|
||||
&& value
|
||||
|
||||
@@ -92,6 +92,7 @@ impl Fixture {
|
||||
fixture_config(),
|
||||
self.home.path().to_path_buf(),
|
||||
RuntimeThreadManagerConfig {
|
||||
mcp_auth_context: Default::default(),
|
||||
data_dir: path.clone(),
|
||||
task_data_dir: self.home.path().join("tasks"),
|
||||
sessions_dir: None,
|
||||
|
||||
@@ -139,8 +139,12 @@ pub fn execute_tools_tool_definition() -> Tool {
|
||||
the user decides, and a denied or refused call throws inside the program (catch \
|
||||
it to continue). Inside a program, tools.call('tool_search', {query}) returns \
|
||||
matching tool names with their input schemas without loading them into the \
|
||||
conversation. Each result is {content, metadata, truncated}. When a result is \
|
||||
cut, truncated names its full size and saved copy and content is the leading \
|
||||
conversation. Each result is {content, metadata, truncated}. A bare single MCP \
|
||||
JSON text block is parsed directly into content; mixed/rich blocks retain \
|
||||
their MCP envelope. Nested results do not deliver inline images to the model; \
|
||||
for visual input, discover the capture tool with a direct tool_search call \
|
||||
and invoke it directly. When a result is cut, truncated names its full size \
|
||||
and saved copy and content is the leading \
|
||||
text of the raw output instead of parsed JSON, so check truncated before \
|
||||
reading fields. Not \
|
||||
available inside programs: agent, workflow, request_user_input, nested \
|
||||
@@ -1092,9 +1096,28 @@ impl CodemodeInvoker {
|
||||
let metadata = result.metadata.clone().unwrap_or(Value::Null);
|
||||
let content = serde_json::from_str(&result.content)
|
||||
.unwrap_or_else(|_| Value::String(result.content.clone()));
|
||||
let payload = json!({ "content": content, "metadata": metadata, "truncated": Value::Null });
|
||||
let mut payload =
|
||||
json!({ "content": content, "metadata": metadata, "truncated": Value::Null });
|
||||
let raw = payload.to_string();
|
||||
if raw.len() <= PER_CALL_RESULT_CAP_BYTES {
|
||||
if McpPool::is_mcp_tool(name)
|
||||
&& let Some(envelope) = payload["content"].as_object()
|
||||
&& envelope
|
||||
.keys()
|
||||
.all(|key| key == "content" || key == "isError")
|
||||
&& envelope
|
||||
.get("isError")
|
||||
.is_none_or(|value| value == &Value::Bool(false))
|
||||
&& let Some(blocks) = envelope.get("content").and_then(Value::as_array)
|
||||
&& let [block] = blocks.as_slice()
|
||||
&& let Some(fields) = block.as_object()
|
||||
&& fields.keys().all(|key| key == "type" || key == "text")
|
||||
&& block.get("type").and_then(Value::as_str) == Some("text")
|
||||
&& let Some(text) = block.get("text").and_then(Value::as_str)
|
||||
&& let Ok(content) = serde_json::from_str::<Value>(text)
|
||||
{
|
||||
payload["content"] = content;
|
||||
}
|
||||
return (payload, raw.len(), None);
|
||||
}
|
||||
let spill_path = spill(&self.spill_id(seq, name), &raw);
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
use async_trait::async_trait;
|
||||
use codewhale_protocol::runtime::DynamicToolSpec;
|
||||
use codewhale_app_server_protocol::runtime::DynamicToolSpec;
|
||||
use serde_json::Value;
|
||||
|
||||
use crate::tools::spec::{
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
//! (`format = "markdown"`); pass `format = "raw"` to keep the bytes intact
|
||||
//! when the model wants to do its own parsing.
|
||||
|
||||
use super::handle::query_jsonpath;
|
||||
use super::handle::{JSONPATH_MAX_BYTES, JSONPATH_MAX_PATH_BYTES, query_jsonpath_bounded};
|
||||
use super::pdf::PdfTextCommand;
|
||||
use super::spec::{
|
||||
ApprovalRequirement, ToolCapability, ToolContext, ToolError, ToolResult, ToolSpec, optional_u64,
|
||||
@@ -27,6 +27,7 @@ use std::collections::BTreeMap;
|
||||
use std::time::Duration;
|
||||
|
||||
const FETCH_ACCEPT: &str = "text/html,text/markdown,text/plain,application/json,application/pdf,image/*,audio/*,video/*,*/*;q=0.5";
|
||||
const MAX_JSON_FIELDS: usize = 32;
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum Format {
|
||||
@@ -132,8 +133,9 @@ impl ToolSpec for FetchUrlTool {
|
||||
},
|
||||
"fields": {
|
||||
"type": "array",
|
||||
"items": { "type": "string" },
|
||||
"description": "Optional JSONPath projections for JSON responses. Supports $, .field, [index], [*], and ['field']; returns matches under `fields`."
|
||||
"maxItems": MAX_JSON_FIELDS,
|
||||
"items": { "type": "string", "maxLength": JSONPATH_MAX_PATH_BYTES },
|
||||
"description": "Optional JSONPath projections for JSON responses. Supports $, .field, [index], [*], and ['field']; returns matches under `fields`. At most 32 paths of 4096 bytes each, 128 steps and 1024 intermediate matches per path, and 1 MiB of aggregate projected fields; narrow the selection on refusal."
|
||||
}
|
||||
},
|
||||
"required": ["url"]
|
||||
@@ -515,6 +517,11 @@ fn parse_fields(input: &Value) -> Result<Vec<String>, ToolError> {
|
||||
let Some(values) = values.as_array() else {
|
||||
return Err(ToolError::invalid_input("`fields` must be an array"));
|
||||
};
|
||||
if values.len() > MAX_JSON_FIELDS {
|
||||
return Err(ToolError::invalid_input(
|
||||
"`fields` accepts at most 32 paths",
|
||||
));
|
||||
}
|
||||
let mut fields = Vec::new();
|
||||
for value in values {
|
||||
let Some(field) = value.as_str() else {
|
||||
@@ -523,7 +530,12 @@ fn parse_fields(input: &Value) -> Result<Vec<String>, ToolError> {
|
||||
));
|
||||
};
|
||||
let field = field.trim();
|
||||
if !field.is_empty() {
|
||||
if field.len() > JSONPATH_MAX_PATH_BYTES {
|
||||
return Err(ToolError::invalid_input(
|
||||
"`fields` JSONPath exceeds 4096 bytes",
|
||||
));
|
||||
}
|
||||
if !field.is_empty() && !fields.iter().any(|existing| existing == field) {
|
||||
fields.push(field.to_string());
|
||||
}
|
||||
}
|
||||
@@ -547,10 +559,30 @@ fn project_json_fields(
|
||||
ToolError::execution_failed(format!("response body is not valid JSON for `fields`: {e}"))
|
||||
})?;
|
||||
let mut out = BTreeMap::new();
|
||||
let mut projected_bytes = 2usize;
|
||||
for field in fields {
|
||||
let matches = query_jsonpath(&body_json, field).map_err(|e| {
|
||||
ToolError::invalid_input(format!("invalid JSONPath `{field}` in `fields`: {e}"))
|
||||
if out.contains_key(field) {
|
||||
continue;
|
||||
}
|
||||
let mut counter = codewhale_util::utils::CountingWriter::new();
|
||||
serde_json::to_writer(&mut counter, field).map_err(|error| {
|
||||
ToolError::execution_failed(format!("JSON projection measurement failed: {error}"))
|
||||
})?;
|
||||
projected_bytes = projected_bytes
|
||||
.saturating_add(counter.count())
|
||||
.saturating_add(1 + usize::from(!out.is_empty()));
|
||||
let remaining = JSONPATH_MAX_BYTES
|
||||
.checked_sub(projected_bytes)
|
||||
.ok_or_else(|| {
|
||||
ToolError::invalid_input(
|
||||
"`fields` projection exceeds 1 MiB; request fewer or narrower fields",
|
||||
)
|
||||
})?;
|
||||
let (matches, bytes) =
|
||||
query_jsonpath_bounded(&body_json, field, remaining).map_err(|e| {
|
||||
ToolError::invalid_input(format!("invalid JSONPath `{field}` in `fields`: {e}"))
|
||||
})?;
|
||||
projected_bytes = projected_bytes.saturating_add(bytes);
|
||||
out.insert(field.clone(), matches);
|
||||
}
|
||||
Ok(Some(out))
|
||||
|
||||
@@ -31,6 +31,10 @@ const HANDLE_STORE_MAX_BYTES: usize = 64 * 1024 * 1024;
|
||||
const DEFAULT_MAX_CHARS: usize = 12_000;
|
||||
const HARD_MAX_CHARS: usize = 50_000;
|
||||
const REPR_PREVIEW_CHARS: usize = 160;
|
||||
pub(super) const JSONPATH_MAX_PATH_BYTES: usize = 4096;
|
||||
pub(super) const JSONPATH_MAX_BYTES: usize = 1024 * 1024;
|
||||
const JSONPATH_MAX_MATCHES: usize = 1024;
|
||||
const JSONPATH_MAX_STEPS: usize = 128;
|
||||
|
||||
pub type SharedHandleStore = Arc<Mutex<HandleStore>>;
|
||||
|
||||
@@ -311,7 +315,8 @@ impl ToolSpec for HandleReadTool {
|
||||
},
|
||||
"jsonpath": {
|
||||
"type": "string",
|
||||
"description": "Small JSONPath subset: $, .field, [index], [*], and ['field']."
|
||||
"maxLength": JSONPATH_MAX_PATH_BYTES,
|
||||
"description": "Small JSONPath subset: $, .field, [index], [*], and ['field']. Refuses paths over 4096 bytes, over 128 steps, over 1024 intermediate matches, or results over 1 MiB; narrow the path on refusal."
|
||||
},
|
||||
"introspect": {
|
||||
"type": "boolean",
|
||||
@@ -490,6 +495,11 @@ fn parse_projection(input: &Value) -> Result<Projection, ToolError> {
|
||||
"handle_read: jsonpath must not be empty",
|
||||
));
|
||||
}
|
||||
if path.len() > JSONPATH_MAX_PATH_BYTES {
|
||||
return Err(ToolError::invalid_input(
|
||||
"handle_read: JSONPath exceeds 4096 bytes",
|
||||
));
|
||||
}
|
||||
return Ok(Projection::JsonPath(path.to_string()));
|
||||
}
|
||||
if let Some(slice) = input.get("slice") {
|
||||
@@ -735,13 +745,29 @@ fn record_text(record: &HandleRecord) -> std::borrow::Cow<'_, str> {
|
||||
}
|
||||
|
||||
pub fn query_jsonpath(root: &Value, path: &str) -> Result<Vec<Value>, String> {
|
||||
query_jsonpath_bounded(root, path, JSONPATH_MAX_BYTES).map(|(values, _)| values)
|
||||
}
|
||||
|
||||
pub(super) fn query_jsonpath_bounded(
|
||||
root: &Value,
|
||||
path: &str,
|
||||
max_bytes: usize,
|
||||
) -> Result<(Vec<Value>, usize), String> {
|
||||
if path.len() > JSONPATH_MAX_PATH_BYTES {
|
||||
return Err("jsonpath exceeds 4096 bytes".to_string());
|
||||
}
|
||||
if !path.starts_with('$') {
|
||||
return Err("jsonpath must start with `$`".to_string());
|
||||
}
|
||||
let mut idx = 1usize;
|
||||
let bytes = path.as_bytes();
|
||||
let mut current = vec![root];
|
||||
let mut steps = 0usize;
|
||||
while idx < bytes.len() {
|
||||
steps += 1;
|
||||
if steps > JSONPATH_MAX_STEPS {
|
||||
return Err("jsonpath exceeds 128 steps; use a narrower projection".to_string());
|
||||
}
|
||||
match bytes[idx] {
|
||||
b'.' => {
|
||||
idx += 1;
|
||||
@@ -780,16 +806,35 @@ pub fn query_jsonpath(root: &Value, path: &str) -> Result<Vec<Value>, String> {
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(current.into_iter().cloned().collect())
|
||||
let mut counter = codewhale_util::utils::CountingWriter::new();
|
||||
serde_json::to_writer(&mut counter, ¤t)
|
||||
.map_err(|error| format!("jsonpath result measurement failed: {error}"))?;
|
||||
let max_bytes = max_bytes.min(JSONPATH_MAX_BYTES);
|
||||
if counter.count() > max_bytes {
|
||||
return Err(format!(
|
||||
"jsonpath result exceeds the {max_bytes}-byte projection budget; use a narrower projection"
|
||||
));
|
||||
}
|
||||
Ok((current.into_iter().cloned().collect(), counter.count()))
|
||||
}
|
||||
|
||||
fn apply_bracket_token<'a>(values: Vec<&'a Value>, token: &str) -> Result<Vec<&'a Value>, String> {
|
||||
if token == "*" {
|
||||
let mut out = Vec::new();
|
||||
let mut push = |item: &'a Value| -> Result<(), String> {
|
||||
if out.len() == JSONPATH_MAX_MATCHES {
|
||||
return Err(
|
||||
"jsonpath exceeds 1024 intermediate matches; use a narrower projection"
|
||||
.to_string(),
|
||||
);
|
||||
}
|
||||
out.push(item);
|
||||
Ok(())
|
||||
};
|
||||
for value in values {
|
||||
match value {
|
||||
Value::Array(items) => out.extend(items),
|
||||
Value::Object(map) => out.extend(map.values()),
|
||||
Value::Array(items) => items.iter().try_for_each(&mut push)?,
|
||||
Value::Object(map) => map.values().try_for_each(&mut push)?,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
Loaded 100 of 272 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user