test(managed-agent): gate the B3 round-and-cost-budget crash window (#15981)

* test(managed-agent): gate the B3 round-and-cost-budget crash window

Matrix row B3 ("after earlier tools") had focused/integration evidence
only: nothing crashed a replacement after several tool rounds and proved
the numeric round counter and the turn cost ledger survive it. A recovery
that restarts the round at 0 or zeroes the token ledger would pass CI.

Package the W2 window in hosted-failover-e2e: a multi-round fake fixture
runs two earlier tools (each its own approval, execution and file effect),
the replacement is crashed while the later post-tool request is held on
the fake-model side (no new serve hook), and the next generation must,
from MySQL alone after scrashing the crashed disk, resume the saved
logical round, restore the TurnBudget snapshot and per-model token ledger
(asserted as per-model output-token totals with non-vacuity guards, since
an unrestored ledger is a truthy {}), reissue the saved request deep-equal,
retract the published prefix, and project one terminal. "Cost budget" here
is the snapshot plus the ledger, not a cumulative spend gate.

Real-stack (.75) evidence: head green twice (round 2->2, budget 500000
deep-equal, ledger 600->700 = restored 600 + final round 100, parity equal,
both earlier Actions decided under their original IDs); three named
mutation kills (round reset, ledger wipe, snapshot re-baselining). The
execution-count assertion is defense-in-depth in this window: three
mutation cells survived because exactly-once rides three independent
guards (replay routing, continuation settled-batch adoption, Runtime
runtimeCallId idempotency), documented in the design.

No production behavior change; failover ceiling 26 -> 27 arms (285 min).

* test(managed-agent): harden the B3 arm per review (R1-1..R1-4)

R1-1: the W2 rationale claimed the saved request carries the budget; it
carries only the round (budget lives on the attempt's route/usage
resources). Corrected in both design languages.

R1-2: the behavioural fixture could not distinguish a newest-first from
an oldest-first budget/models scan, and the ledger assertion had only
text pins. Add a discriminating fixture (older attempt with a different
budget and smaller ledger) and lift the cost block into a top-level
assertB3CostContinuity helper with injected-state behavioural cases:
healthy growth passes; null-budget snapshot, missing snapshot,
re-baselined snapshot, empty before-ledger, zeroed after-ledger and
shrunk per-model totals all throw.

R1-3: the budget non-vacuity guard tested the snapshot object, but a
non-null snapshot with budget: null is a legal production state (lost
+Nk directive), so null == null passed vacuously. Guard the numeric
field instead.

R1-4: site-anchor the B3 post-tool request count (? 3) like the
neighbouring R1-7/F1 pins so a conflict resolution back to the pre-PR
ternary reds at unit speed, not only in the real-stack lane.

Each fix has a demonstrated witness: oldest-first scan, deleted ? 3
clause, object-truthy guard and candidates->prompt repath each red the
new cases/pins; all restored. Contract suites 132/132, typecheck green,
and the .75 head arm re-run with the hardened runner is green (round
2->2, budget 500000 deep-equal, ledger 600->700).
This commit is contained in:
Shaojin Wen authored and GitHub committed 2026-10-31 05:36:12 +00:00
1 parent f87fd6a037
commit bc8ad53f95
10 files changed
+1120 -28

No files matched your search

+8 -3
View File
@@ -1388,9 +1388,9 @@ jobs:
# that a failing integration test there no longer skips them: they need
# only the bundled CLI, the server jar and a runner-private mysqld, not
# its MySQL service or cgroup root. Each arm also runs when an earlier
# one failed. Ceilings: 26 arms × 10 = 260 plus ~13 minutes for the
# uncapped setup steps (Checkout.. Install MySQL binaries): 273 + 2.
timeout-minutes: 275
# one failed. Ceilings: 27 arms × 10 = 270 plus ~13 minutes for the
# uncapped setup steps (Checkout.. Install MySQL binaries): 283 + 2.
timeout-minutes: 285
env:
QWEN_E2E_REGISTER_STORAGE: '1'
steps:
@@ -1539,6 +1539,11 @@ jobs:
timeout-minutes: 10
run: >-
npm run test:e2e:managed-harness-restart-approval-sibling-approval
- name: 'Run after-earlier-tools crash E2E'
if: "${{ !cancelled() && steps.failover_prerequisites.outcome == 'success' }}"
timeout-minutes: 10
run: >-
npm run test:e2e:managed-harness-restart-approval-after-earlier-tools
- name: 'Run approval decision reply-loss E2E'
if: "${{ !cancelled() && steps.failover_prerequisites.outcome == 'success' }}"
timeout-minutes: 10
@@ -510,7 +510,7 @@ Integration regressions proved that model request preparation failures preserve
The machine-readable reports, manifests and retained failed diagnostics are under `.qwen/investigations/g3-step3-implementation/`; the current consolidated report is `packaged-verification.md`. These five packaged gates do not prove the whole matrix above. Since [replaceable-Harness D11](2026-10-02-hosted-replaceable-harness.md#d11--residual-gates-step-3-in-ci-the-model-round-arm-and-q2s-binding-half), they are CI steps of the `hosted-failover-e2e` job, which runs apart from the Hosted MySQL verify (see D11), next to a model-round arm that packages the main observations of B1 for a no-tool Session and of B2, though not their attempt and stream identities, the live SSE view or the admission count.
#14481's packaged gates add thirteen fault arms to the same job. Seven landed with
#14481's packaged gates add fourteen fault arms to the same job. Seven landed with
the second step. Five layer one fault on the Allow path: three crash the
replacement Harness after a
durable A5 barrier — once the Allow decision is recorded (A5's decision row),
@@ -553,11 +553,18 @@ generation must re-hang the original pending Action with its original deadline,
require that sibling's own Allow rather than inherit the earlier one, leave the
decided sibling un-re-asked, and settle both calls once each in their original
order ([A3 mixed-sibling approval](2026-10-31-hosted-a3-mixed-sibling-approval.md)).
A crash after several earlier tool rounds (B3) is packaged too: a multi-round
fixture runs two earlier tools, the replacement is crashed while a later post-tool
request is held, and the next generation must resume the saved logical round and
restore the turn cost snapshot and per-model token ledger rather than restart them,
with each earlier execution occurring once and the later request reissued deep-equal
([B3 round and cost budget](2026-10-31-hosted-b3-round-cost-budget.md)). "Cost budget"
here is the TurnBudget snapshot plus the token ledger, not a cumulative spend gate.
Malformed or foreign refs, snapshot overflow, the rewritten-input and
Hook-no-rerun half of A3 (a commandStep follow-up, since a native continuation
refuses Hook/MCP sessions), Hook/MCP/child
uncertainty (A6), the attempt and stream identities plus the live SSE view
(B2), the round and cost budgets (B3), and the mixed-version reader/bundle
(B2), and the mixed-version reader/bundle
closure (R2) stay focused or integration evidence. Full-owner restart, real-provider
acceptance and a real mixed-version cluster remain separate. Q2's binding
half has the D11 frozen control-plane arm for a woken former Spring's stale
@@ -416,7 +416,7 @@ Allow 保留原 Action、prepare key、request digest 和 Runtime binding,产
机器可读报告、manifest 和保留的失败诊断位于 `.qwen/investigations/g3-step3-implementation/`,当前汇总报告为 `packaged-verification.md`。这五种打包门禁不证明上述整个矩阵。自[可替换 Harness 设计 D11](2026-10-02-hosted-replaceable-harness.zh-CN.md#d11--残留门禁step-3-进-ci模型轮支路q2-的-binding-半边) 起,它们是 `hosted-failover-e2e` 任务的 CI 步骤,该任务独立于 Hosted MySQL 的 verify 运行(见 D11);同一任务里还有一条模型轮支路,把无工具 Session 的 B1 与 B2 的主要观察项做成打包门禁,但不覆盖它们的 attempt 与 stream 身份、实时 SSE 视图以及准入次数。
#14481 的打包门禁在同一任务里新增十三条故障支路,其中七条随第 2 步落地。五条在 Allow 路径上叠加一层故障:三条在持久 A5 屏障之后崩溃替代 Harness——一条在 Allow 决策落库后(A5 的决策行),一条在批准执行到达 `PREPARED`、Runtime Broker 的 `:start` 被扣住时(A5a 的部分 prepare),还有一条在批准结果已提交、post-tool 模型轮被扣住时(A5 的 result commit 屏障)——另两条在提交后丢弃控制面的回复:决策回复丢失后,同一 `Idempotency-Key` 重试必须落到同一条记录,只产生一个 `ACTION_RESPONSE` operation(A4 的决策半边);Turn 准入回复丢失后,重试只产生一行 Turn(C4)。五条都保留原 Action、prepare key、digest 与 Runtime 身份,启动第三代 Harness,并要求一次已结算执行、一次文件效果和一个终态;result-commit 支路还要求重发复用同一个已保存请求,并撤回已发布的片段。另外五条覆盖接下来的屏障:`turn.settled` 已提交但回复丢失时,替换后必须投影一次终态(不要求第二次 store 提交),且不产生推理或工具调用,公开文本仍必须是恢复出的答案(C1);终态写入被拒时,可付 checkpoint 必须由替换方恰好补偿一次,且恢复出的答案仍可见(C2)。清理释放被拒、随后再次崩溃(C3)也已打包:Turn 进入终态且其 Harness 代际已死之后,控制面通过 cleanup-only load 重试欠付清理,直到替换方确认([清理欠债重试](2026-10-28-hosted-cleanup-debt-retry.zh-CN.md))。接管 load 回复丢失(A4 的 load 半边)也已打包:替换方第一次 drive load 的回复被丢失一次(测试门控钩子),控制面把同一 load 重发进已启动的 active driver,原 Turn 仍须恰好投影一个终态并保留恢复出的答案——重发原样重述 active attachment,而不是把 Turn 卡死。早先观察到的卡 RUNNING 变体在当前代码树上复现不出,因此该臂作为回归门禁守住这条交接,而非标记一个未决缺口。主流 Finished 持久标记与 assistant 提交之间的崩溃(B3a)也已打包:测试门控钩子在替换方的 post-tool 轮记录 Finished 之后、assistant 提交之前把它扣在该窗口内,runner 在那里崩溃它,下一代必须 deep-equal 重发已保存请求并投影一个终态、保留恢复出的答案,而不是补偿一个已提交的输出。混合兄弟审批裂缝中的崩溃(A3 的 native 半边)也已打包:一个两调用批次按兄弟逐个审批,测试门控钩子在较早兄弟已决之后、替换方第一次询问审批时把它扣住,runner 在那里崩溃它,下一代必须用原 deadline 重挂原 pending Action、要求该兄弟自己的 Allow 而非继承较早的 Allow、不重问已决兄弟,并按原顺序各结算两个调用一次([A3 混合兄弟审批](2026-10-31-hosted-a3-mixed-sibling-approval.zh-CN.md))。损坏/外部 ref、快照超限、A3 的改写输入与 Hook 不重跑半边(commandStep 后续,因为 native continuation 拒绝 Hook/MCP 会话)、Hook/MCP/child 不确定性(A6)、attempt 与 stream 身份及实时 SSE 视图(B2)、轮次与成本预算(B3),以及混版本 reader/bundle 闭包(R2)仍由定向或集成证据覆盖。两侧 owner 重启、真实 provider 和真实混版本集群仍独立处理。Q2 的 binding 半边有 D11 的冻结控制面支路覆盖醒来的旧 Spring 的过期 claim,并有 store 测试钉住 bind 守卫;两个存活控制面仍属后续阶段。
#14481 的打包门禁在同一任务里新增十四条故障支路,其中七条随第 2 步落地。五条在 Allow 路径上叠加一层故障:三条在持久 A5 屏障之后崩溃替代 Harness——一条在 Allow 决策落库后(A5 的决策行),一条在批准执行到达 `PREPARED`、Runtime Broker 的 `:start` 被扣住时(A5a 的部分 prepare),还有一条在批准结果已提交、post-tool 模型轮被扣住时(A5 的 result commit 屏障)——另两条在提交后丢弃控制面的回复:决策回复丢失后,同一 `Idempotency-Key` 重试必须落到同一条记录,只产生一个 `ACTION_RESPONSE` operation(A4 的决策半边);Turn 准入回复丢失后,重试只产生一行 Turn(C4)。五条都保留原 Action、prepare key、digest 与 Runtime 身份,启动第三代 Harness,并要求一次已结算执行、一次文件效果和一个终态;result-commit 支路还要求重发复用同一个已保存请求,并撤回已发布的片段。另外五条覆盖接下来的屏障:`turn.settled` 已提交但回复丢失时,替换后必须投影一次终态(不要求第二次 store 提交),且不产生推理或工具调用,公开文本仍必须是恢复出的答案(C1);终态写入被拒时,可付 checkpoint 必须由替换方恰好补偿一次,且恢复出的答案仍可见(C2)。清理释放被拒、随后再次崩溃(C3)也已打包:Turn 进入终态且其 Harness 代际已死之后,控制面通过 cleanup-only load 重试欠付清理,直到替换方确认([清理欠债重试](2026-10-28-hosted-cleanup-debt-retry.zh-CN.md))。接管 load 回复丢失(A4 的 load 半边)也已打包:替换方第一次 drive load 的回复被丢失一次(测试门控钩子),控制面把同一 load 重发进已启动的 active driver,原 Turn 仍须恰好投影一个终态并保留恢复出的答案——重发原样重述 active attachment,而不是把 Turn 卡死。早先观察到的卡 RUNNING 变体在当前代码树上复现不出,因此该臂作为回归门禁守住这条交接,而非标记一个未决缺口。主流 Finished 持久标记与 assistant 提交之间的崩溃(B3a)也已打包:测试门控钩子在替换方的 post-tool 轮记录 Finished 之后、assistant 提交之前把它扣在该窗口内,runner 在那里崩溃它,下一代必须 deep-equal 重发已保存请求并投影一个终态、保留恢复出的答案,而不是补偿一个已提交的输出。混合兄弟审批裂缝中的崩溃(A3 的 native 半边)也已打包:一个两调用批次按兄弟逐个审批,测试门控钩子在较早兄弟已决之后、替换方第一次询问审批时把它扣住,runner 在那里崩溃它,下一代必须用原 deadline 重挂原 pending Action、要求该兄弟自己的 Allow 而非继承较早的 Allow、不重问已决兄弟,并按原顺序各结算两个调用一次([A3 混合兄弟审批](2026-10-31-hosted-a3-mixed-sibling-approval.zh-CN.md))。若干先前工具轮之后的崩溃(B3)也已打包:一个多轮夹具跑两个先前工具,替换方在较后的 post-tool 请求被扣住时崩溃,下一代必须续跑已保存的逻辑轮次、恢复回合成本快照与每模型 token 账本而非重启它们,每个先前执行恰好一次、较后请求 deep-equal 重发([B3 轮次与成本预算](2026-10-31-hosted-b3-round-cost-budget.zh-CN.md))。此处「成本预算」指 TurnBudget 快照加 token 账本,而非累计花费闸门。损坏/外部 ref、快照超限、A3 的改写输入与 Hook 不重跑半边(commandStep 后续,因为 native continuation 拒绝 Hook/MCP 会话)、Hook/MCP/child 不确定性(A6)、attempt 与 stream 身份及实时 SSE 视图(B2),以及混版本 reader/bundle 闭包(R2)仍由定向或集成证据覆盖。两侧 owner 重启、真实 provider 和真实混版本集群仍独立处理。Q2 的 binding 半边有 D11 的冻结控制面支路覆盖醒来的旧 Spring 的过期 claim,并有 store 测试钉住 bind 守卫;两个存活控制面仍属后续阶段。
## 10. 实现范围
@@ -0,0 +1,227 @@
# Hosted B3 — round and cost budget across a replacement
[English](2026-10-31-hosted-b3-round-cost-budget.md) | [简体中文](2026-10-31-hosted-b3-round-cost-budget.zh-CN.md)
Status: proposed, 2026-10-31. Part of [#14481][issue] (G3 Step 3 recovery-matrix
acceptance, tracks #12380). Extends the packaged fault arms of the
`hosted-failover-e2e` CI job described in
[G3 Step 3 recovery](2026-10-09-hosted-g3-step3-recovery.md). This slice packages
matrix row **B3 ("after earlier tools")**: a crash after several tool rounds must
recover the logical round and the turn cost ledger without resetting them.
## Problem
Matrix row B3 reads: _"Request uses saved results/history; each previous execution
occurs once; round and cost budget do not reset."_ Today B3 is focused/integration
evidence only. The existing `after-result` arm crashes after a **single** tool round
and proves the saved request is reissued deep-equal, but nothing crashes after
**several** tool rounds and proves the numeric round counter and the turn cost
ledger survive the replacement. A recovery that restarts the round at 0 (letting a
Turn run past its configured round limit) or zeroes the token ledger would pass CI.
**What "cost budget" actually is here (verified).** There is no free-standing
cumulative cost counter. The recoverable cost state is the `TurnBudget` snapshot
(the output-token budget parsed from a user `+Nk` directive, plus
`outputTokensAtTurnStart`) and the per-model session token metrics (uiTelemetry),
both persisted on the `model.attempt` route/usage resources and restored on a cold
load. The `toolOutputBatchBudget` is stateless per-generation config (harmless to
recompute) and the checkpoint `budgetConsumed` field is a constant-0 placeholder;
neither is a B3 assertion target. So "cost budget do not reset" is proven as
**snapshot + ledger restoration fidelity**, not as a hard spend gate in the main
model loop (the loop does not stop on token spend; only workflow/agent dispatch
consumes the budget).
## Current state (verified on `main` at `b3e9ce0cd1`)
- **Logical round.** The model loop starts at
`round = input.resumeModel?.round ?? input.initialRound ?? 0`
(`hosted-harness-model.ts:979`) and enforces
`maxRounds > 0 && round >= maxRounds → HostedModelRoundLimitError`
(`hosted-harness-model.ts:991`). `maxRounds = config.getMaxSessionTurns()` defaults
to **-1 (unlimited)**. Because the round index is absolute, the round limit alone
does not cleanly detect a reset, so this arm detects a round reset through the
exactly-once, deep-equal and round-continuity assertions rather than the limit.
The round is persisted two ways:
the `hosted-model-request` resource carries `round`
(`hosted-model-recovery.ts:33`), and the `hosted-approval-continuation` plan
carries `round` (`hosted-approval-continuation.ts:88`, set from `logicalRound` at
`hosted-workspace-tool-turn.ts:3738`). On recovery the loop resumes from
`resumeModel.round` (saved-request path) or `initialRound = plan.round + 1`
(`hosted-harness-session.ts:3981`).
- **Cost ledger.** `bindBudget` (`managed-hook-activation.ts:83-135`) on a cold load
reverse-scans `model.attempt` events, reads `budget` and `models` from the
`usageRef`/`routeRef` resources, restores the `TurnBudget` snapshot and calls
`restoreSessionModelMetrics`. It begins a new snapshot only when
`shared.snapshot?.promptId !== this.turnId`, so a same-turn recovery does not
restart the ledger. Each attempt writes the route resource (`budget`) and usage
resource (`budget` + `models`) (`managed-hook-activation.ts:176-183, 230-241`).
- **Exactly-once / saved results.** The saved-request replay is pinned by
`requestDigest` and refused if any assistant commit or `tool.intent` follows the
snapshot's `throughSequence` (`hosted-model-recovery.ts:213-222`); the tool-turn
fold is keyed on the functionResponse id (`hosted-workspace-tool-turn.ts:1666`),
and `resumeSavedNativeBatch` re-validates the original definition/argument digests
and reuses the original `runtimeCallId`/`prepareKey`.
- **Runner observability.** The round is NOT in the provider request body or the
public feed; it lives in the `hosted-model-request` / `hosted-approval-continuation`
MySQL resources. The budget/models live in the `managed-hosted-model-route` /
`managed-hosted-model-usage` resources. The runner already has both readers (journal
HEX parse and `ma_resource` inline read). There is currently **no** runner assertion
on the numeric round or the budget content.
## Goals and scope
**In scope.** A packaged `hosted-failover-e2e` arm that runs a **multi-round** tool
fixture (two earlier tool rounds), crashes the replacement while a later post-tool
model request is held, and proves, from MySQL alone after scrashing the crashed
generation's disk:
1. the recovered model request resumes at the **same logical round** as the saved
request (not restarted at 0/1);
2. the turn **cost snapshot and per-model token ledger** are restored, not zeroed or
re-baselined;
3. each earlier tool execution occurs **exactly once** with its original execution
identity, and the reissued request is deep-equal to the saved one;
4. the Turn projects exactly one terminal with the recovered answer.
**Out of scope.** A hard token-spend gate in the main model loop (none exists); the
`toolOutputBatchBudget` (stateless config); the checkpoint `budgetConsumed`
placeholder; the skill-only / manual-continue recovery paths that carry no
`initialRound` (their reachability in a packaged native-tool arm is unverified and
left as a follow-up); real-provider acceptance; full-owner restart.
**No production behavior change.** The crash is held on the fake-model side (the
existing `holdUntil` mechanism), so this arm needs **no new serve hook** — unlike
A3/B3a. The only production-source edit, if any, is test-gated and inert.
## Proposed solution
### Window choice (W2)
Crash in the **after-result window of a later round**: the Nth (N≥2) post-tool model
request has been published (snapshot + `model.attempt: started` durable) and is held
by the fake model before it finishes. This is the lowest-risk window — it reuses the
`after-result` hold/barrier machinery, the hold point is on the fake-server side (no
new serve hook), and the saved request carries the round while the crash-boundary
`managed-hosted-model-route` / `managed-hosted-model-usage` resources carry the
budget, so recovery must restore both and the runner reads each from its own
resource. (The alternative W1 — between a tool-result commit and
the next snapshot — may not be stoppable and could route to a round-resetting path;
it is deferred pending a dry-run. W3 overlaps B3a's boundary and adds only the
multi-round/budget dimension, folded into W2's assertions.)
### Multi-round fixture
The fake model decides by the count of `"role":"tool"` messages: the first two
requests return a `write_file` tool call each (two distinct paths/effects), and the
third returns the held final answer. The prompt carries a `+Nk` directive so the
`TurnBudget` snapshot is non-null and its restoration is numerically assertable.
Each response also charges non-zero usage (300/300/100 completion tokens): the fake
server's default usage is all zeros, which would leave the token ledger numerically
empty, and the earlier rounds outweigh the final round so a zeroed restore totals
less than the crash-boundary ledger instead of re-counting the same sum.
### Recovery assertions (new, on top of the after-result template)
- **Round continuity**: read the saved `hosted-model-request.round` before the crash
and the recovered request's round after; require equality (and ≥2, so a reset to 0
reds). A reset that re-runs the earlier rounds is additionally caught by the
exactly-once execution count and the deep-equal request parity.
- **Cost ledger continuity**: read the route/usage resource `budget` and `models`
before and after; require the crash-boundary snapshot to be non-null (a null one
would let a dropped restore pass vacuously as null == null) and deep-equal after
recovery, and every crash-boundary model's output-token total to survive (after ≥
before). The crash-boundary ledger must be non-empty, so a fixture that never
charged the ledger reds instead of passing vacuously.
- **Exactly-once + deep-equal**: two earlier executions, each `SETTLED` /
`dispatch_generation=1` under the original Runtime identity (binding / generation /
runtimeSessionId); the first approval plan carries only tool A, so the two physical
file effects give the per-tool evidence; the reissued post-tool request deep-equal
to the saved one; one terminal; recovered answer.
### Runner, CI and contract pins
A new `approvalFaults` entry (allow-only overlay), a barrier branch in the exhaustive
crash-arm guard, an npm script and a `sdk-java.yml` step, and a bump of the
`hosted-failover-e2e` arm ceiling and timeout. Contract pins pin the multi-round
fixture, the round/budget readers and their assertions; any new top-level runner
helper gets a behavioural test on injected inputs.
### Documentation
Update [G3 Step 3 recovery](2026-10-09-hosted-g3-step3-recovery.md) §9.1 in both
languages: thirteen → fourteen packaged arms, move B3 from focused to packaged, and
record the "cost budget = TurnBudget snapshot + token ledger" clarification.
## Design decisions and rationale
- **W2 over W1/W3** — W2 reuses the proven after-result hold with no new serve hook;
W1's recovery routing is unverified and may be unstoppable; W3 duplicates B3a's
boundary.
- **Cost = snapshot + ledger, not a spend gate** — matches the actual mechanism; the
arm asserts restoration fidelity, which is what "do not reset" means here.
- **Explicit `+Nk` directive** — the default null budget makes a ledger reset
unobservable; the fixture opts in so the snapshot restoration is numerically
assertable. No `maxSessionTurns` is set: the absolute round index means the limit
does not cleanly detect a reset, so reset detection rests on the exactly-once,
deep-equal and round-continuity assertions.
- **Multi-round gated on the fault name** — the shared fake model and assertions are
used by every approval arm; gating keeps the single-round arms unchanged.
## Constraints and risks
- **Shared fixture regression.** The fake model and execution-identity assertions are
shared; the multi-round path must be fully gated and the single-round arms re-run
green. Every gate the arm shares with `after-result` — arming AND releasing the
hold, capturing the published prefix, and the retraction proof — must enumerate
both fault names: a release gated to `after-result` alone left the recovery
generation's reissued request held forever (caught by the first .75 head run).
- **Round/budget readers must mirror recovery.** The round/budget assertions read the
same durable resources recovery reads; a behavioural test on injected rows proves
the readers, and a mutation must red them. Resources are read as `HEX(inline_bytes)`
and decoded in the runner: `mysql --batch` escapes backslashes, which corrupts the
saved request's `\"` sequences (the request carries the full system prompt).
- **Ledger shape (confirmed).** An unrestored ledger is `{}`, not null —
`getMetricsForSession` falls back to `createInitialMetrics()` — and `{}` is truthy,
so a null check alone would let a zeroed restore pass. The assertion therefore
compares per-model output-token totals (after ≥ before, the ledger grows as the
held round completes) and requires a non-empty crash-boundary ledger.
- **CI budget.** One more arm raises the `hosted-failover-e2e` ceiling and timeout.
## Validation plan
- **.75 real stack**: head arm green — round continuity, cost ledger continuity, two
earlier executions once each, deep-equal reissue, one terminal, recovery from MySQL
alone after scrash.
- **Demonstrated mutation reds**: reset the recovered round to 0 → round-continuity
red; zero the restored models ledger → ledger red; drop the `+Nk` snapshot restore
→ budget red (each killed at its named assertion on the .75 real stack). The
execution-count assertion is defense-in-depth in W2: three mutation cells
(empty fold predicate; broken replay routing; both together) all SURVIVED, because
exactly-once here rides three independent layers — the saved-request replay
routing, the continuation route's settled-batch adoption, and the Runtime's
runtimeCallId idempotency (Java-side, unreachable from CLI mutations). Reddening
the count would require a cross-language compound; that is recorded as a
follow-up. The count/identity assertions are shared with the packaged,
CI-proven sibling-approval and after-result arms.
- Contract suite + behavioural tests green; `npm run build && npm run typecheck &&
npm run bundle`.
## Acceptance criteria
- A packaged arm in `hosted-failover-e2e` whose runner asserts round continuity, cost
ledger continuity, exactly-once earlier executions and one terminal from MySQL alone
after scrashing the crashed generation.
- Each new round/budget/ledger signal has a demonstrated named mutation red; the
shared exactly-once count/identity assertions are defense-in-depth in W2, with
survived-mutation evidence documenting the independent guard layers.
- Bilingual design docs aligned; CI green; no production behavior change.
## Open questions
- Resolved: the fault name is `after-earlier-tools` and the directive is `+500k`
(the .75 crash-boundary evidence pins the snapshot at `budget: 500000`,
`outputTokensAtTurnStart: 0`).
- Whether the skill-only / manual-continue round-reset paths are reachable in a
packaged arm (deferred; would be a separate gap if confirmed).
[issue]: https://github.com/QwenLM/qwen-code/issues/14481
@@ -0,0 +1,180 @@
# Hosted B3 —— 跨替换的轮次与成本预算
[English](2026-10-31-hosted-b3-round-cost-budget.md) | [简体中文](2026-10-31-hosted-b3-round-cost-budget.zh-CN.md)
状态:提议中,2026-10-31。属于 [#14481][issue](G3 Step 3 恢复矩阵验收,跟踪
#12380)。扩展 [G3 Step 3 恢复](2026-10-09-hosted-g3-step3-recovery.zh-CN.md) 所述
`hosted-failover-e2e` CI job 的打包故障支路。本切片打包矩阵行 **B3(「after earlier
tools」)**:在若干工具轮之后崩溃,恢复必须不重置逻辑轮次与回合成本账本。
## 问题
矩阵行 B3 写道:_「请求使用已保存的结果/历史;每次先前执行恰好发生一次;轮次与成本
预算不重置。」_ 目前 B3 仅有 focused/integration 证据。现有 `after-result` 支路在
**单个**工具轮之后崩溃、证明已保存请求被 deep-equal 重发,但没有任何支路在**若干**
工具轮之后崩溃并证明数值轮次计数器与回合成本账本跨替换存活。一个把轮次重启为 0
(让 Turn 跑过其配置轮次上限)或把 token 账本归零的恢复,本来都能通过 CI。
**这里「成本预算」的真实含义(已实证)。** 本仓没有独立的累计成本计数器。可恢复的
成本状态是 `TurnBudget` 快照(从用户 `+Nk` 指令解析出的输出 token 预算,加
`outputTokensAtTurnStart`)与每模型会话 token 指标(uiTelemetry),二者持久化在
`model.attempt` 的 route/usage 资源上并在冷加载时恢复。`toolOutputBatchBudget` 是
每代重新计算的无状态配置(重算无害),checkpoint 的 `budgetConsumed` 字段是恒 0
占位符;二者都不是 B3 的断言对象。因此「成本预算不重置」被证明为**快照 + 账本的恢复
保真**,而非主模型循环里的硬性花费闸门(主循环不按 token 花费停轮;只有
workflow/agent 派发消费该预算)。
## 现状(在 `main` 的 `b3e9ce0cd1` 上实证)
- **逻辑轮次。** 模型循环从
`round = input.resumeModel?.round ?? input.initialRound ?? 0`
(`hosted-harness-model.ts:979`)起跳,并强制
`maxRounds > 0 && round >= maxRounds → HostedModelRoundLimitError`
(`hosted-harness-model.ts:991`)。`maxRounds = config.getMaxSessionTurns()` 默认
**-1(无限制)**。由于轮次索引是绝对的,轮次上限本身无法干净地检测重置,因此本臂
通过恰好一次、deep-equal 与轮次连续性断言来检测轮次重置,而非依赖上限。轮次经两条通道持久化:`hosted-model-request` 资源带 `round`
(`hosted-model-recovery.ts:33`),`hosted-approval-continuation` plan 带 `round`
(`hosted-approval-continuation.ts:88`,在 `hosted-workspace-tool-turn.ts:3738`
由 `logicalRound` 赋值)。恢复时循环从 `resumeModel.round`(saved-request 路径)或
`initialRound = plan.round + 1`(`hosted-harness-session.ts:3981`)续跑。
- **成本账本。** `bindBudget`(`managed-hook-activation.ts:83-135`)冷加载时逆序扫
`model.attempt` 事件,从 `usageRef`/`routeRef` 资源读回 `budget` 与 `models`,恢复
`TurnBudget` 快照并调用 `restoreSessionModelMetrics`。仅当
`shared.snapshot?.promptId !== this.turnId` 才起新快照,因此同轮恢复不会重启账本。
每次 attempt 写 route 资源(`budget`)与 usage 资源(`budget` + `models`)
(`managed-hook-activation.ts:176-183, 230-241`)。
- **恰好一次 / 已存结果。** saved-request 重放由 `requestDigest` 钉死,且若快照
`throughSequence` 之后出现 assistant commit 或 `tool.intent` 即拒绝
(`hosted-model-recovery.ts:213-222`);tool-turn fold 以 functionResponse id 为键
(`hosted-workspace-tool-turn.ts:1666`),`resumeSavedNativeBatch` 重验原始定义/参数
摘要并复用原 `runtimeCallId`/`prepareKey`。
- **runner 可观测性。** 轮次不在 provider 请求体、也不在公共 feed;它在
`hosted-model-request` / `hosted-approval-continuation` MySQL 资源里。budget/models
在 `managed-hosted-model-route` / `managed-hosted-model-usage` 资源里。runner 已有
这两种读法(journal HEX 解析与 `ma_resource` inline 读)。目前 runner **没有**对
数值轮次或 budget 内容的断言。
## 目标与范围
**范围内。** 一个打包的 `hosted-failover-e2e` 支路:跑一个**多轮**工具夹具(两个先前
工具轮),在较后的 post-tool 模型请求被扣住时崩溃替换进程,并在 scrash 崩溃代磁盘后
仅凭 MySQL 证明:
1. 恢复出的模型请求以与已保存请求**相同的逻辑轮次**续跑(不重启为 0/1);
2. 回合**成本快照与每模型 token 账本**被恢复,而非归零或重新起算;
3. 每个先前工具执行以原执行身份**恰好发生一次**,且重发请求与已保存请求 deep-equal;
4. Turn 恰好投影一个 terminal 并带恢复出的答案。
**范围外。** 主模型循环里的硬性 token 花费闸门(不存在);`toolOutputBatchBudget`
(无状态配置);checkpoint `budgetConsumed` 占位符;不带 `initialRound` 的 skill-only
/ 手动 continue 恢复路径(其在打包 native-tool 臂中的可达性未验证,留作后续);
real-provider 验收;两侧 owner 重启。
**无生产行为改动。** 崩溃由 fake 模型侧扣住(既有 `holdUntil` 机制),因此本支路
**无需新 serve 钩子**——不同于 A3/B3a。唯一的生产源码改动(若有)是测试 gated 且惰性。
## 方案
### 窗口选择(W2)
崩在**较后一轮的 after-result 窗口**:第 N(N≥2)轮 post-tool 模型请求已发布(快照 +
`model.attempt: started` 已 durable),被 fake 模型在完成前扣住。这是风险最低的窗口
——复用 `after-result` 的 hold/barrier 机器,hold 点在 fake 服务器侧(无需新 serve
钩子),且已保存请求携带轮次、崩溃边界的 `managed-hosted-model-route` /
`managed-hosted-model-usage` 资源携带预算,恢复必须把两者都恢复,runner 也从各自
的资源分别读取。(备选 W1——工具结果 commit
与下一快照之间——可能停不住、且可能落到重置轮次的路径,干跑确认前延后。W3 与 B3a
边界重合、仅增的多轮/budget 维度并入 W2 断言。)
### 多轮夹具
fake 模型按 `"role":"tool"` 消息计数决定:前两个请求各回一个 `write_file` 工具调用
(两个不同路径/效果),第三个回被扣住的最终答案。提示词带 `+Nk` 指令使 `TurnBudget`
快照非空、其恢复可数值断言。每个响应还计入非零 usage(completion tokens
300/300/100):fake 服务器默认 usage 全 0,会让 token 账本在数值上为空;且先前轮次
重于最终轮,使归零恢复的总量**低于**崩溃边界账本,而非把同一总和重算一遍。
### 恢复断言(在 after-result 模板之上新增)
- **轮次连续**:崩溃前读已保存 `hosted-model-request.round`,恢复后读恢复请求的轮次,
要求相等(且 ≥2,使重置为 0 变红)。重跑先前轮次的重置还会被恰好一次执行计数与
deep-equal 请求 parity 捕获。
- **成本账本连续**:崩溃前后读 route/usage 资源的 `budget` 与 `models`,要求崩溃边界
快照非空(null 会让丢弃恢复以 null == null 空过)且恢复后 deep-equal,且崩溃边界
每个模型的输出 token 总量存活(after ≥ before)。崩溃边界账本必须非空,使从未计入
账本的夹具变红而非空过。
- **恰好一次 + deep-equal**:两个先前执行各 `SETTLED`/`dispatch_generation=1`、带原
Runtime 身份(binding/generation/runtimeSessionId);首个审批 plan 只含工具 A,故
两个物理文件效果提供逐工具证据;重发 post-tool 请求与已保存请求 deep-equal;一个
terminal;恢复出的答案。
### runner、CI 与契约钉
新增一个 `approvalFaults` 条目(allow-only 叠加)、穷举崩溃臂守卫里的 barrier 分支、
一个 npm script 与一个 `sdk-java.yml` step,并抬 `hosted-failover-e2e` 的臂数上限与
timeout。契约钉钉住多轮夹具、轮次/budget 读法及其断言;任何新 runner 顶层 helper 配
注入输入的行为化测试。
### 文档
双语更新 [G3 Step 3 恢复](2026-10-09-hosted-g3-step3-recovery.zh-CN.md) §9.1:十三→
十四条打包支路,把 B3 从 focused 移到 packaged,并记录「成本预算 = TurnBudget 快照 +
token 账本」的澄清。
## 设计决定与理由
- **W2 优于 W1/W3** —— W2 复用已验证的 after-result hold、无需新 serve 钩子;W1 的
恢复路由未验证、可能停不住;W3 与 B3a 边界重复。
- **成本 = 快照 + 账本,非花费闸门** —— 符合真实机制;支路断言恢复保真,这正是此处
「不重置」的含义。
- **显式 `+Nk` 指令** —— 默认 null 预算使账本重置不可观测;夹具显式开启使快照恢复可
数值断言。不设 `maxSessionTurns`:绝对轮次索引意味着上限无法干净检测重置,故重置
检测依赖恰好一次、deep-equal 与轮次连续性断言。
- **多轮 gated 在故障名上** —— 共享 fake 模型与断言被每个审批支路使用;gating 使单轮
支路不变。
## 约束与风险
- **共享夹具回归。** fake 模型与执行身份断言共享;多轮路径必须完全 gated,单轮支路
重跑绿。本臂与 `after-result` 共享的每个 gate——hold 的 arm **与释放**、已发布前缀
的采集、前缀撤回证明——都必须枚举两个故障名:只 gate 在 `after-result` 的释放会让
恢复代的重发请求被永远扣住(第一次 .75 head 运行即抓住)。
- **轮次/budget 读法须镜像恢复。** 轮次/budget 断言读取与恢复相同的 durable 资源;
注入行的行为化测试证明读法,变异必须使其变红。资源以 `HEX(inline_bytes)` 读取并在
runner 内解码:`mysql --batch` 会转义反斜杠,损坏已保存请求 JSON 的 `\"` 序列
(请求携带完整 system prompt)。
- **账本形态(已确认)。** 未恢复的账本是 `{}` 而非 null——`getMetricsForSession`
回退到 `createInitialMetrics()`——而 `{}` 为 truthy,仅查 null 会让归零恢复通过。
因此断言比较逐模型输出 token 总量(after ≥ before,账本随被扣住轮次完成而增长),
并要求崩溃边界账本非空。
- **CI 预算。** 多一条臂抬 `hosted-failover-e2e` 上限与 timeout。
## 验证计划
- **.75 真机**:head 臂绿——轮次连续、成本账本连续、两个先前执行各一次、deep-equal
重发、一个 terminal、scrash 后仅凭 MySQL 恢复。
- **实测变异红**:把恢复轮次重置为 0→轮次连续红;把恢复的 models 账本归零→账本红;
丢弃 `+Nk` 快照恢复→budget 红(三者均在 .75 真机于命名断言处被杀)。执行计数断言
在 W2 是纵深防御:三个变异格(空 fold 谓词;破坏 replay 路由;两者同时)全部
**存活**,因为此处恰好一次依赖三道独立守卫——saved-request replay 路由、
continuation 路由的 settled-batch 采纳、Runtime 的 runtimeCallId 幂等(Java 侧,
CLI 变异不可达)。使计数变红需要跨语言复合变异;已记录为后续。计数/身份断言与
已打包且 CI 证明的 sibling-approval、after-result 支路共享。
- 契约套件 + 行为化测试绿;`npm run build && npm run typecheck && npm run bundle`。
## 验收标准
- `hosted-failover-e2e` 中一个打包支路,其 runner 在 scrash 崩溃代后仅凭 MySQL 断言
轮次连续、成本账本连续、先前执行恰好一次与一个 terminal。
- 每个新增轮次/budget/账本信号都有一次实测命名变异红;共享的恰好一次计数/身份断言
在 W2 为纵深防御,以存活变异证据记录各独立守卫层。
- 双语设计文档对齐;CI 绿;无生产行为改动。
## 开放问题
- 已解决:故障名为 `after-earlier-tools`,指令为 `+500k`(.75 崩溃边界证据钉住快照
`budget: 500000`、`outputTokensAtTurnStart: 0`)。
- skill-only / 手动 continue 的轮次重置路径在打包臂中是否可达(延后;若确认将是独立
缺口)。
[issue]: https://github.com/QwenLM/qwen-code/issues/14481
+1
View File
@@ -82,6 +82,7 @@
"test:e2e:managed-harness-restart-approval-after-result": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault after-result",
"test:e2e:managed-harness-restart-approval-finished-before-commit": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault finished-before-commit",
"test:e2e:managed-harness-restart-approval-sibling-approval": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault sibling-approval",
"test:e2e:managed-harness-restart-approval-after-earlier-tools": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault after-earlier-tools",
"test:e2e:managed-harness-restart-approval-cleanup-failed": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault cleanup-failed",
"test:e2e:managed-harness-restart-approval-decision-reply-lost": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault decision-reply-lost",
"test:e2e:managed-harness-restart-approval-admission-reply-lost": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault admission-reply-lost",
+14 -1
View File
@@ -1824,12 +1824,25 @@ un-re-asked, and settle both calls once each in their original order:
npm run test:e2e:managed-harness-restart-approval-sibling-approval
```
A crash after several earlier tool rounds (B3) runs a multi-round fixture with two
earlier tools and holds the replacement while a later post-tool request is open
(the hold is on the fake-model side, so no serve hook is needed). The runner kills
it there; the next generation must resume the saved logical round and restore the
turn cost snapshot and per-model token ledger rather than restart them, with each
earlier execution occurring once and the later request reissued deep-equal. "Cost
budget" here is the TurnBudget snapshot plus the token ledger, not a cumulative
spend gate:
```bash
npm run test:e2e:managed-harness-restart-approval-after-earlier-tools
```
They write their evidence under `.qwen/investigations/g3-step3-implementation/`;
`G3_VERIFY_REPORT_DIR` overrides the directory. The remaining rows stay
focused or integration tested for now: the rewritten-input and Hook-no-rerun half
of mixed calls (a commandStep follow-up, since a native continuation refuses
Hook/MCP sessions), Hook/MCP/child uncertainty, the attempt and stream
identities plus the live SSE view, the round and cost budgets, and the
identities plus the live SSE view, and the
mixed-version reader/bundle closure (#14481).
Two arms keep a former owner alive while its replacement finishes the Turn
+412 -17
View File
@@ -59,6 +59,9 @@ let originalApprovalPlan: SavedApprovalPlan | undefined;
// re-decided by the recovery generation.
let siblingApprovalId: string | undefined;
let siblingApprovalOptions: SavedApprovalOptions | undefined;
// B3: tool B's new Action, detected after Allow A and decided by the runner's
// second Allow; both earlier Actions must end decided under their original IDs.
let secondApprovalId: string | undefined;
let verifiedOriginalTerminal: string | undefined;
function saveEvidence(name: string, body: unknown): void {
if (!verificationMode) return;
@@ -91,6 +94,7 @@ const approvalFaults = [
'after-result',
'finished-before-commit',
'sibling-approval',
'after-earlier-tools',
'cleanup-failed',
'decision-reply-lost',
'admission-reply-lost',
@@ -137,7 +141,7 @@ for (let index = 0; index < argumentsList.length; index += 1) {
index += 1;
} else {
throw new Error(
'Usage: run-managed-agent-server-e2e.ts [--model ID] [--runtime-delay-ms N] [--settings PATH] [--session-failover|--inflight-failover|--continuation-failover|--model-round-failover|--big-output] [--harness-only] [--freeze] [--frozen-control-plane] [--g3-verify allow|deny|cancel|expiry|continuation] [--g3-fault after-decision|after-prepare|after-result|finished-before-commit|sibling-approval|cleanup-failed|decision-reply-lost|admission-reply-lost|load-reply-lost|terminal-reply-lost|terminal-write-failed]',
'Usage: run-managed-agent-server-e2e.ts [--model ID] [--runtime-delay-ms N] [--settings PATH] [--session-failover|--inflight-failover|--continuation-failover|--model-round-failover|--big-output] [--harness-only] [--freeze] [--frozen-control-plane] [--g3-verify allow|deny|cancel|expiry|continuation] [--g3-fault after-decision|after-prepare|after-result|finished-before-commit|sibling-approval|after-earlier-tools|cleanup-failed|decision-reply-lost|admission-reply-lost|load-reply-lost|terminal-reply-lost|terminal-write-failed]',
);
}
}
@@ -346,6 +350,15 @@ const inflightSideEffectContent = 'MANAGED_INFLIGHT_TOOL_EXECUTED\n';
const siblingSideEffectName = 'managed-sibling-side-effect.txt';
const siblingSideEffect = path.join(workspaceMount, siblingSideEffectName);
const siblingSideEffectContent = 'MANAGED_SIBLING_TOOL_EXECUTED\n';
// B3 after-earlier-tools arm: a second, later tool round writes its own file so
// each earlier round's single execution has its own physical effect. The +Nk
// budget directive makes the TurnBudget snapshot non-null so its restoration is
// numerically assertable (the default null budget hides a ledger reset). A round
// reset is caught by the exactly-once / deep-equal / round-continuity assertions.
const b3SecondSideEffectName = 'managed-b3-second-side-effect.txt';
const b3SecondSideEffect = path.join(workspaceMount, b3SecondSideEffectName);
const b3SecondSideEffectContent = 'MANAGED_B3_SECOND_TOOL_EXECUTED\n';
const b3BudgetDirective = '+500k';
const workspaceId = createHash('sha256')
.update(workspace)
.digest('hex')
@@ -1370,6 +1383,136 @@ function siblingFinalPlan(
return plan;
}
// B3: read the durable round and cost ledger the way recovery restores them.
// The round comes from the latest hosted-model-request (model.attempt.recoveryRef);
// the budget snapshot and per-model token metrics are reverse-scanned from the
// route/usage resources exactly as bindBudget does on a cold load.
interface B3RoundBudget {
round: number;
budget: unknown;
models: unknown;
}
function b3RoundAndBudget(port: number, sessionFilter: string): B3RoundBudget {
const rows = runMysql(
port,
`SELECT HEX(record_bytes) FROM qwen_managed_agent.ma_session_journal_tx WHERE ${sessionFilter} ORDER BY first_sequence`,
).split('\n');
const attempts: Array<{
sequence: number;
payload: Record<string, unknown>;
}> = [];
for (const row of rows) {
if (!row || row === 'NULL') continue;
for (const line of Buffer.from(row, 'hex').toString('utf8').split('\n')) {
if (!line) continue;
const record = JSON.parse(line) as {
subtype?: unknown;
managedSession?: {
kind?: unknown;
sequence?: unknown;
payload?: Record<string, unknown>;
};
};
if (record.subtype !== 'managed_session_event_v1') continue;
const event = record.managedSession;
if (
!event ||
typeof event.sequence !== 'number' ||
event.kind !== 'model.attempt'
)
continue;
attempts.push({ sequence: event.sequence, payload: event.payload ?? {} });
}
}
attempts.sort((a, b) => a.sequence - b.sequence);
if (attempts.length === 0) throw new Error('B3 window has no model attempt');
const readResource = (ref: unknown): Record<string, unknown> | undefined => {
const resourceId = (ref as { resourceId?: string } | null)?.resourceId;
if (!resourceId) return undefined;
// HEX + decode avoids the mysql --batch backslash escaping that would
// corrupt the JSON's \" sequences — the saved request carries the full
// system prompt, unlike the small approval-continuation plan.
const hex = runMysql(
port,
`SELECT HEX(inline_bytes) FROM qwen_managed_agent.ma_resource WHERE ${sessionFilter} AND resource_id=${sqlString(resourceId)}`,
);
if (!hex || hex === 'NULL') return undefined;
return JSON.parse(Buffer.from(hex, 'hex').toString('utf8')) as Record<
string,
unknown
>;
};
let round = -1;
for (let i = attempts.length - 1; i >= 0; i -= 1) {
const req = readResource(attempts[i].payload['recoveryRef']);
if (req && typeof req['round'] === 'number') {
round = req['round'];
break;
}
}
let budget: unknown;
let models: unknown;
for (let i = attempts.length - 1; i >= 0; i -= 1) {
for (const field of ['usageRef', 'routeRef']) {
const saved = readResource(attempts[i].payload[field]);
if (!saved) continue;
if (budget === undefined && 'budget' in saved) budget = saved['budget'];
if (models === undefined && saved['models']) models = saved['models'];
}
if (budget !== undefined && models !== undefined) break;
}
if (round < 0) throw new Error('B3 window has no saved model-request round');
return { round, budget: budget ?? null, models: models ?? null };
}
// B3: the turn cost snapshot and per-model token ledger must survive the
// replacement. The snapshot's NUMERIC budget must be present: a non-null
// snapshot with budget: null is a legal production state (no +Nk directive
// parsed), and an object-truthy guard would let null == null pass vacuously.
// The ledger may grow as the held round completes, but every crash-boundary
// model's output-token total must survive; an unrestored ledger is a truthy
// {}, so a null check alone would let a zeroed restore pass too.
function assertB3CostContinuity(
before: B3RoundBudget,
after: B3RoundBudget,
): void {
const b3Budget = (before.budget as { budget?: number | null } | null)?.budget;
if (b3Budget === null || b3Budget === undefined)
throw new Error(
`B3 crash-boundary turn budget was not exercised: ${JSON.stringify(before.budget)}`,
);
if (!isDeepStrictEqual(after.budget, before.budget))
throw new Error(
`B3 recovery reset the turn budget snapshot: before=${JSON.stringify(before.budget)} after=${JSON.stringify(after.budget)}`,
);
const b3ModelsBefore = (before.models ?? {}) as Record<
string,
{ tokens?: { candidates?: number } }
>;
const b3ModelsAfter = (after.models ?? {}) as Record<
string,
{ tokens?: { candidates?: number } }
>;
const b3LedgerEntries = Object.entries(b3ModelsBefore);
if (
b3LedgerEntries.length === 0 ||
b3LedgerEntries.every(
([, metric]) => (metric.tokens?.candidates ?? 0) === 0,
)
)
throw new Error(
'B3 crash-boundary per-model token ledger is empty: the fixture did not exercise the ledger',
);
for (const [model, metric] of b3LedgerEntries) {
const beforeTokens = metric.tokens?.candidates ?? 0;
const afterTokens = b3ModelsAfter[model]?.tokens?.candidates ?? 0;
if (afterTokens < beforeTokens)
throw new Error(
`B3 recovery wiped the per-model token ledger: model=${model} before=${beforeTokens} after=${afterTokens}`,
);
}
}
function assertPublicAnswer(events: PublicEvent[], expected: string): void {
const terminal = events.filter((event) => event.terminal);
const text = events
@@ -1641,6 +1784,77 @@ try {
}
if (continuationFailover && serialized.includes(continuationMarker)) {
// B3: two earlier tool rounds, then a held post-tool round. The fake
// model decides by the number of tool results already in the history:
// 0 -> tool A (round 0), 1 -> tool B (round 1), >=2 -> the held final
// answer (round 2). The crash lands on the round-2 request so recovery
// must resume the saved round and cost ledger, not restart them.
if (approvalFault === 'after-earlier-tools') {
const toolResultCount = (serialized.match(/"role":"tool"/g) ?? [])
.length;
// Non-zero usage makes the token ledger numerically observable (the
// fake's default usage is all zeros). The earlier rounds outweigh the
// final round so a zeroed restore totals less than the crash-boundary
// ledger, not just the same sum re-counted.
if (toolResultCount === 0) {
return {
toolCalls: [
fakeToolCall(
'write_file',
{
file_path: inflightSideEffectName,
content: inflightSideEffectContent,
},
'call_managed_b3_first',
),
],
usage: {
prompt_tokens: 0,
completion_tokens: 300,
total_tokens: 300,
},
};
}
if (toolResultCount === 1) {
return {
toolCalls: [
fakeToolCall(
'write_file',
{
file_path: b3SecondSideEffectName,
content: b3SecondSideEffectContent,
},
'call_managed_b3_second',
),
],
usage: {
prompt_tokens: 0,
completion_tokens: 300,
total_tokens: 300,
},
};
}
if (approvalResultHold) {
return {
contentChunks: [continuationPartial],
holdAfterChunks: 1,
holdUntil: approvalResultHold,
usage: {
prompt_tokens: 0,
completion_tokens: 100,
total_tokens: 100,
},
};
}
return {
content: continuationResponse,
usage: {
prompt_tokens: 0,
completion_tokens: 100,
total_tokens: 100,
},
};
}
if (!serialized.includes('"role":"tool"')) {
// A3: a two-call batch so the replacement can be crashed in the
// approval gap between a decided sibling and a pending one.
@@ -2041,7 +2255,9 @@ try {
{
type: 'text',
text: continuationFailover
? `${continuationMarker}. Execute the requested tool once and reply exactly ${continuationResponse}.`
? approvalFault === 'after-earlier-tools'
? `${continuationMarker}. ${b3BudgetDirective} Execute the requested tools and reply exactly ${continuationResponse}.`
: `${continuationMarker}. Execute the requested tool once and reply exactly ${continuationResponse}.`
: inflightFailover
? `${inflightMarker}. Execute the requested tool once and reply exactly ${inflightResponse}.`
: modelRoundFailover
@@ -2822,7 +3038,10 @@ try {
policy_revision: originalApprovalOptions!.policyRevision,
option_id: verificationMode,
});
if (approvalFault === 'after-result') {
if (
approvalFault === 'after-result' ||
approvalFault === 'after-earlier-tools'
) {
approvalResultHold = new Promise<void>((resolve) => {
releaseApprovalResultHold = resolve;
});
@@ -2862,6 +3081,60 @@ try {
`Lost decision reply produced ${operations} ACTION_RESPONSE operations, not one`,
);
}
if (approvalFault === 'after-earlier-tools') {
// The first Allow drives tool A (round 0); the model then asks approval
// for tool B (round 1) under a NEW Action. Detect that requested Action
// and allow it so both earlier rounds execute before the held round 2.
await waitUntil(
'Second earlier-tool approval requested',
() => {
const rows = runMysql(
mysqlPort,
`SELECT action_id, state FROM qwen_managed_agent.ma_session_record WHERE ${sessionFilter} AND record_kind='ACTION'`,
);
const requested = (rows ? rows.split('\n') : [])
.map((row) => row.split('\t'))
.filter(
([id, state]) =>
state === 'requested' && id !== originalApprovalId,
);
if (requested.length === 1) {
secondApprovalId = requested[0][0];
return true;
}
return false;
},
120_000,
replacementHarness,
);
const secondOptions = JSON.parse(
runMysql(
mysqlPort,
`SELECT options_json FROM qwen_managed_agent.ma_session_record WHERE ${sessionFilter} AND record_kind='ACTION' AND action_id=${sqlString(secondApprovalId!)}`,
),
);
const secondResponse = await fetch(
`${replacementSpringUrl}/v1/agents/sessions/${session.id}/actions/${secondApprovalId}/responses`,
{
method: 'POST',
headers: {
'content-type': 'application/json',
'idempotency-key': `g3-answer-${secondApprovalId}`,
...tenantHeaders(tenant),
},
body: JSON.stringify({
kind: 'permission',
input_revision: secondOptions.inputRevision,
policy_revision: secondOptions.policyRevision,
option_id: 'allow',
}),
},
);
if (secondResponse.status !== 202)
throw new Error(
`Second earlier-tool response returned ${secondResponse.status}: ${await secondResponse.text()}`,
);
}
} else if (verificationMode === 'cancel') {
const response = await fetch(
`${replacementSpringUrl}/v1/agents/sessions/${session.id}/events`,
@@ -2885,12 +3158,14 @@ try {
}
let publishedPrefixSequences: number[] = [];
let terminalAcceptsAtBarrier = 0;
let b3Before: B3RoundBudget | undefined;
if (
approvalFault === 'after-decision' ||
approvalFault === 'after-prepare' ||
approvalFault === 'after-result' ||
approvalFault === 'finished-before-commit' ||
approvalFault === 'sibling-approval' ||
approvalFault === 'after-earlier-tools' ||
approvalFault === 'cleanup-failed' ||
approvalFault === 'terminal-reply-lost' ||
approvalFault === 'terminal-write-failed'
@@ -2952,6 +3227,45 @@ try {
120_000,
replacementHarness,
);
} else if (approvalFault === 'after-earlier-tools') {
// Both earlier tool rounds must have committed their results and the
// round-2 post-tool request must be held with its prefix published, so
// the crash lands after the earlier tools but before the final output
// commits. Recovery must then resume the saved round and cost ledger.
await waitUntil(
'Earlier tools settled and the later round held',
async () => {
const rows = runMysql(
mysqlPort,
'SELECT execution_state, IF(result_json IS NULL, 0, 1) FROM qwen_managed_agent.ma_tool_execution ORDER BY execution_call_id',
);
const execs = rows ? rows.split('\n') : [];
if (execs.length !== 2 || execs.some((r) => r !== 'SETTLED\t1'))
return false;
const heldLaterRequest = (fake?.requests ?? []).some(
({ body }) => {
const messages = JSON.stringify(body['messages']);
return (
messages.includes(continuationMarker) &&
(messages.match(/"role":"tool"/g) ?? []).length >= 2
);
},
);
if (!heldLaterRequest) return false;
const view = await publicTranscript(
replacementSpringUrl,
tenant,
session.id,
);
return view.text.includes(continuationPartial);
},
120_000,
replacementHarness,
);
// Capture the durable round and cost ledger at the crash boundary so
// recovery can be proven to resume them rather than restart them.
b3Before = b3RoundAndBudget(mysqlPort, sessionFilter);
saveEvidence('b3-round-budget-before.json', b3Before);
} else if (
approvalFault === 'terminal-reply-lost' ||
approvalFault === 'terminal-write-failed'
@@ -3128,7 +3442,8 @@ try {
)
: undefined;
const publishedPrefix =
approvalFault === 'after-result'
approvalFault === 'after-result' ||
approvalFault === 'after-earlier-tools'
? await publicTranscript(replacementSpringUrl, tenant, session.id)
: undefined;
publishedPrefixSequences = (publishedPrefix?.events ?? [])
@@ -3160,7 +3475,8 @@ try {
}
: undefined,
publishedPrefix:
approvalFault === 'after-result'
approvalFault === 'after-result' ||
approvalFault === 'after-earlier-tools'
? {
sequences: publishedPrefixSequences,
text: publishedPrefix?.text,
@@ -3171,7 +3487,10 @@ try {
replacementHarness.child,
'Replacement Hosted Harness (fault barrier)',
);
if (approvalFault === 'after-result') {
if (
approvalFault === 'after-result' ||
approvalFault === 'after-earlier-tools'
) {
releaseApprovalResultHold();
approvalResultHold = undefined;
}
@@ -3328,7 +3647,11 @@ try {
// a compensation that only projects a terminal would pass. The
// result-commit arm already owns the same check with its retraction
// diagnostic, so it keeps that message.
if (verificationMode === 'allow' && approvalFault !== 'after-result') {
if (
verificationMode === 'allow' &&
approvalFault !== 'after-result' &&
approvalFault !== 'after-earlier-tools'
) {
const transcript = await publicTranscript(
replacementSpringUrl,
tenant,
@@ -3339,7 +3662,10 @@ try {
`Approval recovery answer differs: ${JSON.stringify(transcript.text)}`,
);
}
if (approvalFault === 'after-result') {
if (
approvalFault === 'after-result' ||
approvalFault === 'after-earlier-tools'
) {
// The reissue must retract the held prefix: only the recovered
// answer may remain in the public transcript.
const transcript = await publicTranscript(
@@ -3374,7 +3700,8 @@ try {
: [];
const expectedExecutions =
verificationMode === 'allow'
? approvalFault === 'sibling-approval'
? approvalFault === 'sibling-approval' ||
approvalFault === 'after-earlier-tools'
? 2
: 1
: 0;
@@ -3385,8 +3712,26 @@ try {
// Every approved call keeps its own original prepare key/digest and the
// original Runtime identity, settled once. Match by prepare key so the
// model's call order is checked, not the SQL row order; the A3 arm has
// two siblings, every other allow arm has one.
if (verificationMode === 'allow') {
// two siblings and B3 two earlier rounds, every other allow arm has one.
if (approvalFault === 'after-earlier-tools') {
// B3's two earlier tools live in separate rounds, so the first approval
// plan carries only tool A. Assert both executions' Runtime identity and
// settled-once directly; the two physical file effects give per-tool
// evidence, and a re-run of either earlier tool reds the count above.
for (const [ordinal, execution] of executions.entries()) {
if (
execution[2] !== originalApprovalPlan!.runtime.bindingId ||
execution[3] !== originalApprovalPlan!.runtime.generation ||
execution[4] !== originalApprovalPlan!.runtime.runtimeSessionId ||
execution[5] !== 'SETTLED' ||
execution[6] !== '1'
) {
throw new Error(
`B3 earlier execution ${ordinal} changed its identity or re-ran: ${execution.join(',')}`,
);
}
}
} else if (verificationMode === 'allow') {
const plan = originalApprovalPlan!;
const settledAtByOrdinal: number[] = [];
for (const [ordinal, call] of plan.calls.entries()) {
@@ -3448,6 +3793,26 @@ try {
`Sibling physical file assertion failed: ${JSON.stringify(siblingBytes)}`,
);
}
if (approvalFault === 'after-earlier-tools') {
const b3Bytes = existsSync(b3SecondSideEffect)
? readFileSync(b3SecondSideEffect, 'utf8')
: '';
if (b3Bytes !== b3SecondSideEffectContent)
throw new Error(
`B3 second physical file assertion failed: ${JSON.stringify(b3Bytes)}`,
);
// Round + cost ledger continuity: recovery must resume the saved round
// and restore the budget snapshot / token ledger, not restart them.
if (!b3Before)
throw new Error('B3 crash-boundary round/budget was not captured');
const b3After = b3RoundAndBudget(mysqlPort, sessionFilter);
saveEvidence('b3-round-budget-after.json', b3After);
if (b3After.round !== b3Before.round || b3After.round < 2)
throw new Error(
`B3 recovery reset the logical round: before=${b3Before.round} after=${b3After.round}`,
);
assertB3CostContinuity(b3Before, b3After);
}
const actionRows = runMysql(
mysqlPort,
`SELECT action_id, state FROM qwen_managed_agent.ma_session_record WHERE ${sessionFilter} AND record_kind='ACTION' ORDER BY action_id`,
@@ -3469,6 +3834,18 @@ try {
throw new Error(
`Sibling approval identities/states differ: ${actionRows}`,
);
} else if (approvalFault === 'after-earlier-tools') {
// Both earlier tools end decided under their ORIGINAL Action IDs: A by
// the first Allow, B by the runner's second Allow. Recovery must resume
// the saved model round, not re-ask either approval.
const expectedB3Actions = [
`${originalApprovalId}\t${expectedActionState}`,
`${secondApprovalId}\t${expectedActionState}`,
].sort();
if (actionRows !== expectedB3Actions.join('\n'))
throw new Error(
`B3 approval identities/states differ: ${actionRows}`,
);
} else if (
actionRows !== `${originalApprovalId}\t${expectedActionState}`
) {
@@ -3587,12 +3964,14 @@ try {
JSON.stringify(request.body.messages).includes('"role":"tool"'),
);
const expectedModelsAfterResults =
approvalFault === 'after-result' ||
approvalFault === 'finished-before-commit'
? 2
: ['allow', 'deny'].includes(verificationMode)
? 1
: 0;
approvalFault === 'after-earlier-tools'
? 3
: approvalFault === 'after-result' ||
approvalFault === 'finished-before-commit'
? 2
: ['allow', 'deny'].includes(verificationMode)
? 1
: 0;
if (
initialRequests.length !== 1 ||
modelAfterResults.length !== expectedModelsAfterResults
@@ -3617,6 +3996,22 @@ try {
'Result-commit recovery changed the saved model request',
);
}
if (approvalFault === 'after-earlier-tools') {
// The held round-2 request (index 1) must be reissued deep-equal by the
// recovery generation (index 2); index 0 is the earlier round-1 request.
const parity = isDeepStrictEqual(
modelAfterResults[1]?.body,
modelAfterResults[2]?.body,
);
saveEvidence('model-request-parity.json', {
requests: modelAfterResults.map((request) => request.body),
equal: parity,
});
if (!parity)
throw new Error(
'B3 recovery changed the saved later-round model request',
);
}
if (responseOperation) {
await waitUntil(
'Approval response command completion',
+14 -2
View File
@@ -144,6 +144,18 @@ const failoverArms = [
'sibling-approval',
],
],
[
'Run after-earlier-tools crash E2E',
'test:e2e:managed-harness-restart-approval-after-earlier-tools',
[
'--continuation-failover',
'--harness-only',
'--g3-verify',
'allow',
'--g3-fault',
'after-earlier-tools',
],
],
[
'Run approval decision reply-loss E2E',
'test:e2e:managed-harness-restart-approval-decision-reply-lost',
@@ -652,8 +664,8 @@ describe('Hosted real-process gates', () => {
// Step ceilings today (201 + 20 + 50 + 20): Verify, the Broker fault
// gates, admission and O4. The failover arms moved to their own job.
['hosted-harness-mysql', 291, 304],
// Twenty-six failover arms at 10 each.
['hosted-failover-e2e', 260, 275],
// Twenty-seven failover arms at 10 each.
['hosted-failover-e2e', 270, 285],
])(
'keeps the %s job ceiling above its summed step ceilings',
(name, steps, ceiling) => {
+254 -2
View File
@@ -15,7 +15,7 @@ import {
} from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { inspect } from 'node:util';
import { inspect, isDeepStrictEqual } from 'node:util';
import {
createSourceFile,
isArrayLiteralExpression,
@@ -594,7 +594,7 @@ describe('managed-agent-server e2e runner', () => {
// pinned separately and narrowing either one (2 -> 1) reds here at unit
// speed, not only at the real-stack barrier.
expect(source).toMatch(
/const expectedExecutions =[\s\S]*?approvalFault === 'sibling-approval'\s*\?\s*2\s*:\s*1/,
/const expectedExecutions =[\s\S]*?approvalFault === 'sibling-approval' \|\|\s*approvalFault === 'after-earlier-tools'\s*\?\s*2\s*:\s*1/,
);
// `\s*`, not `[\s\S]*?`: the gap here is whitespace-only, so a tight anchor
// cannot span past this site to the expectedExecutions ternary and mask a
@@ -648,6 +648,73 @@ describe('managed-agent-server e2e runner', () => {
expect(a3PlanFnBody).toContain("event.kind !== 'hosted.batch.planned'");
expect(a3PlanFnBody).toContain("payload['batchId'] !== batchId");
expect(a3PlanFnBody).toContain("plan.stage !== 'final'");
// B3 after-earlier-tools: the multi-round fixture, the second-Allow
// choreography, the round/budget continuity readers + assertions, and the
// later-round deep-equal parity are all pinned so a regression reds here.
expect(source).toContain("'after-earlier-tools'");
expect(source).toContain('call_managed_b3_second');
expect(source).toContain('b3BudgetDirective');
expect(source).toContain('Second earlier-tool approval requested');
expect(source).toContain('Earlier tools settled and the later round held');
expect(source).toContain('b3RoundAndBudget(');
expect(source).toContain('B3 recovery reset the logical round');
expect(source).toContain('B3 recovery reset the turn budget snapshot');
expect(source).toContain('B3 recovery wiped the per-model token ledger');
expect(source).toContain('B3 second physical file assertion failed');
expect(source).toContain(
'B3 recovery changed the saved later-round model request',
);
expect(source).toContain('B3 earlier execution');
expect(source).toContain('B3 approval identities/states differ');
// The hold armed for B3 must also be RELEASED after the crash: a release
// gated to after-result alone leaves the recovery generation's reissued
// request held forever (the .75 head run crash-looped on exactly that).
const releaseAt = source.indexOf('releaseApprovalResultHold();');
expect(releaseAt).toBeGreaterThan(-1);
expect(source.slice(releaseAt - 200, releaseAt)).toContain(
"approvalFault === 'after-earlier-tools'",
);
// B3 shares after-result's published-prefix capture and retraction proof.
expect(source).toMatch(
/approvalFault === 'after-result' \|\|\s*approvalFault === 'after-earlier-tools'\s*\) \{\s*\/\/ The reissue must retract the held prefix/,
);
// The fixture must charge non-zero usage (the fake default is all zeros,
// which makes the ledger continuity assertion vacuous) and the ledger
// check must compare per-model totals — an empty {} ledger is truthy.
expect(source).toContain('completion_tokens: 300');
expect(source).toContain(
'B3 crash-boundary per-model token ledger is empty',
);
expect(source).toContain('B3 crash-boundary turn budget was not exercised');
expect(source).toContain('afterTokens < beforeTokens');
expect(source).toContain('assertB3CostContinuity(');
// R1-3: the budget guard must test the NUMERIC field — a non-null snapshot
// with budget: null is a legal production state, so an object-truthy guard
// would let a lost +Nk directive pass as null == null.
const b3CostFn = source.indexOf('function assertB3CostContinuity(');
expect(b3CostFn).toBeGreaterThan(-1);
const b3CostFnBody = source.slice(
b3CostFn,
source.indexOf('\n}', b3CostFn),
);
expect(b3CostFnBody).toContain(
'b3Budget === null || b3Budget === undefined',
);
// R1-4: site-anchor the B3 post-tool request count like the neighbouring
// R1-7 / F1 pins — the identifier alone survives a conflict resolution
// back to the pre-PR ternary, which would only red in the real-stack lane.
// Deleting the after-earlier-tools ? 3 clause must red here.
expect(source).toMatch(
/const expectedModelsAfterResults =[\s\S]*?approvalFault === 'after-earlier-tools'\s*\?\s*3\s*:/,
);
const b3Fn = source.indexOf('function b3RoundAndBudget(');
expect(b3Fn).toBeGreaterThan(-1);
const b3FnBody = source.slice(b3Fn, source.indexOf('\n}', b3Fn));
expect(b3FnBody).toContain('qwen_managed_agent.ma_session_journal_tx');
expect(b3FnBody).toContain("event.kind !== 'model.attempt'");
expect(b3FnBody).toContain('SELECT HEX(inline_bytes)');
expect(b3FnBody).toContain("attempts[i].payload['recoveryRef']");
expect(b3FnBody).toContain("['usageRef', 'routeRef']");
});
// R4-2 (behavioural): the source pins above assert the window helper's
@@ -838,6 +905,191 @@ describe('managed-agent-server e2e runner', () => {
).toEqual(finalPlan);
});
// B3 (behavioural): exercise b3RoundAndBudget on injected journal rows and
// request/route/usage resources, so the latest-attempt round selection, the
// reverse-scan for budget/models, and both throw branches are proven on data.
// runMysql is injected through load() and dispatches on the query (journal
// HEX rows vs HEX resource bytes keyed by resource_id); sqlString is
// extracted too.
it('behaves b3RoundAndBudget on injected journal rows', () => {
const attempt = (sequence, refs) => ({
kind: 'model.attempt',
sequence,
payload: refs,
});
const journalRows = (events) =>
Buffer.from(
events
.map((event) =>
JSON.stringify({
subtype: 'managed_session_event_v1',
managedSession: event,
}),
)
.join('\n'),
'utf8',
).toString('hex');
const budgetSnapshot = { budget: 500000, outputTokensAtTurnStart: 100 };
const resources = {
// The saved request carries the full system prompt, so its JSON has \"
// and \\ sequences — the shape mysql --batch escaping corrupted when the
// helper read CONVERT(inline_bytes) instead of HEX.
req_round2: { round: 2, prompt: 'reply "exactly" then \\ stop' },
req_round0: { round: 0 },
usage1: { budget: budgetSnapshot, models: { m: { output: 150 } } },
route2: { budget: budgetSnapshot },
// R1-2: a discriminating pair — the older attempt carries a DIFFERENT
// budget and a smaller ledger, the newest its own. An oldest-first scan
// returns 900000/300 where the newest-first contract returns 500000/600,
// so the direction is proven on data, not only on the round selection.
usageOld: {
budget: { budget: 900000, outputTokensAtTurnStart: 0 },
models: { m: { tokens: { candidates: 300 } } },
},
usageNew: {
budget: budgetSnapshot,
models: { m: { tokens: { candidates: 600 } } },
},
};
const run = (events) =>
load(
['b3RoundAndBudget', 'sqlString'],
'b3RoundAndBudget',
'runMysql',
)((_port, query) => {
if (query.includes('ma_session_journal_tx')) return journalRows(events);
if (query.includes('ma_resource')) {
const resourceId = query.match(/resource_id='([^']+)'/)?.[1];
const res = resources[resourceId];
if (!res) throw new Error(`no resource ${String(resourceId)}`);
return Buffer.from(JSON.stringify(res), 'utf8').toString('hex');
}
throw new Error(`unexpected query: ${query}`);
}).b3RoundAndBudget(0, 'filter');
// The latest attempt's request gives the round; budget/models reverse-scan
// across the route/usage refs (budget from the newest, models from usage1).
expect(
run([
attempt(10, {
recoveryRef: { resourceId: 'req_round0' },
usageRef: { resourceId: 'usage1' },
}),
attempt(20, {
recoveryRef: { resourceId: 'req_round2' },
routeRef: { resourceId: 'route2' },
}),
]),
).toEqual({
round: 2,
budget: budgetSnapshot,
models: { m: { output: 150 } },
});
// Models still resolve from an earlier attempt when the newest has no usage.
expect(
run([
attempt(10, {
recoveryRef: { resourceId: 'req_round0' },
usageRef: { resourceId: 'usage1' },
}),
attempt(20, { recoveryRef: { resourceId: 'req_round2' } }),
]),
).toEqual({
round: 2,
budget: budgetSnapshot,
models: { m: { output: 150 } },
});
// R1-2: the newest attempt's own usage wins over an older, different one —
// an oldest-first budget/models scan reds here (900000/300 vs 500000/600).
expect(
run([
attempt(10, {
recoveryRef: { resourceId: 'req_round0' },
usageRef: { resourceId: 'usageOld' },
}),
attempt(20, {
recoveryRef: { resourceId: 'req_round2' },
usageRef: { resourceId: 'usageNew' },
}),
]),
).toEqual({
round: 2,
budget: budgetSnapshot,
models: { m: { tokens: { candidates: 600 } } },
});
// No attempt at all.
expect(() => run([])).toThrow(/no model attempt/);
// An attempt with no saved request round.
expect(() => run([attempt(10, {})])).toThrow(
/no saved model-request round/,
);
});
// R1-2 / R1-3 (behavioural): the cost-continuity assertion on injected
// before/after states. isDeepStrictEqual is injected through load() because
// the extracted body's free identifiers resolve against the harness scope.
it('behaves assertB3CostContinuity on injected before/after states', () => {
const assertCost = load(
['assertB3CostContinuity'],
'assertB3CostContinuity',
'isDeepStrictEqual',
)(isDeepStrictEqual).assertB3CostContinuity;
const snapshot = {
sessionId: 's',
promptId: 'p',
budget: 500000,
directiveText: '+500k',
outputTokensAtTurnStart: 0,
};
const ledger = (candidates) => ({
'fake-model': { tokens: { candidates } },
});
const before = { round: 2, budget: snapshot, models: ledger(600) };
// The healthy recovery: snapshot deep-equal, ledger grown by the final round.
expect(() =>
assertCost(before, { round: 2, budget: snapshot, models: ledger(700) }),
).not.toThrow();
// R1-3: a non-null snapshot whose numeric budget is null (the +Nk directive
// stopped being parsed) must red — an object-truthy guard passes it and the
// deep-equal then compares null with null.
expect(() =>
assertCost(
{ ...before, budget: { ...snapshot, budget: null } },
{
round: 2,
budget: { ...snapshot, budget: null },
models: ledger(700),
},
),
).toThrow(/turn budget was not exercised/);
// A missing snapshot reds the same guard.
expect(() =>
assertCost({ ...before, budget: null }, { ...before, budget: null }),
).toThrow(/turn budget was not exercised/);
// A re-baselined snapshot (different turn-start baseline) reds deep-equal.
expect(() =>
assertCost(before, {
round: 2,
budget: { ...snapshot, outputTokensAtTurnStart: 600 },
models: ledger(700),
}),
).toThrow(/reset the turn budget snapshot/);
// An empty crash-boundary ledger is vacuous and must red.
expect(() =>
assertCost({ ...before, models: {} }, { ...before, models: {} }),
).toThrow(/token ledger is empty/);
// A zeroed ({}) after-ledger must red — {} is truthy, so a null check passes it.
expect(() =>
assertCost(before, { round: 2, budget: snapshot, models: {} }),
).toThrow(/wiped the per-model token ledger/);
// A shrunk per-model total must red (the dropped-restore shape).
expect(() =>
assertCost(before, { round: 2, budget: snapshot, models: ledger(100) }),
).toThrow(
/wiped the per-model token ledger: model=fake-model before=600 after=100/,
);
});
// #12941: the Stage A acceptance criterion names a 15-second Runtime delay,
// but the ordering assertion was gated at 20 s, so a --runtime-delay-ms 15000
// run silently skipped it. Pin the threshold to the criterion's delay and