mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
test(managed-agent): gate the B3 round-and-cost-budget crash window (#15981)
* test(managed-agent): gate the B3 round-and-cost-budget crash window
Matrix row B3 ("after earlier tools") had focused/integration evidence
only: nothing crashed a replacement after several tool rounds and proved
the numeric round counter and the turn cost ledger survive it. A recovery
that restarts the round at 0 or zeroes the token ledger would pass CI.
Package the W2 window in hosted-failover-e2e: a multi-round fake fixture
runs two earlier tools (each its own approval, execution and file effect),
the replacement is crashed while the later post-tool request is held on
the fake-model side (no new serve hook), and the next generation must,
from MySQL alone after scrashing the crashed disk, resume the saved
logical round, restore the TurnBudget snapshot and per-model token ledger
(asserted as per-model output-token totals with non-vacuity guards, since
an unrestored ledger is a truthy {}), reissue the saved request deep-equal,
retract the published prefix, and project one terminal. "Cost budget" here
is the snapshot plus the ledger, not a cumulative spend gate.
Real-stack (.75) evidence: head green twice (round 2->2, budget 500000
deep-equal, ledger 600->700 = restored 600 + final round 100, parity equal,
both earlier Actions decided under their original IDs); three named
mutation kills (round reset, ledger wipe, snapshot re-baselining). The
execution-count assertion is defense-in-depth in this window: three
mutation cells survived because exactly-once rides three independent
guards (replay routing, continuation settled-batch adoption, Runtime
runtimeCallId idempotency), documented in the design.
No production behavior change; failover ceiling 26 -> 27 arms (285 min).
* test(managed-agent): harden the B3 arm per review (R1-1..R1-4)
R1-1: the W2 rationale claimed the saved request carries the budget; it
carries only the round (budget lives on the attempt's route/usage
resources). Corrected in both design languages.
R1-2: the behavioural fixture could not distinguish a newest-first from
an oldest-first budget/models scan, and the ledger assertion had only
text pins. Add a discriminating fixture (older attempt with a different
budget and smaller ledger) and lift the cost block into a top-level
assertB3CostContinuity helper with injected-state behavioural cases:
healthy growth passes; null-budget snapshot, missing snapshot,
re-baselined snapshot, empty before-ledger, zeroed after-ledger and
shrunk per-model totals all throw.
R1-3: the budget non-vacuity guard tested the snapshot object, but a
non-null snapshot with budget: null is a legal production state (lost
+Nk directive), so null == null passed vacuously. Guard the numeric
field instead.
R1-4: site-anchor the B3 post-tool request count (? 3) like the
neighbouring R1-7/F1 pins so a conflict resolution back to the pre-PR
ternary reds at unit speed, not only in the real-stack lane.
Each fix has a demonstrated witness: oldest-first scan, deleted ? 3
clause, object-truthy guard and candidates->prompt repath each red the
new cases/pins; all restored. Contract suites 132/132, typecheck green,
and the .75 head arm re-run with the hardened runner is green (round
2->2, budget 500000 deep-equal, ledger 600->700).
This commit is contained in:
1 parent
f87fd6a037
commit
bc8ad53f95
10 files changed
+1120
-28
No files matched your search
@@ -1388,9 +1388,9 @@ jobs:
|
||||
# that a failing integration test there no longer skips them: they need
|
||||
# only the bundled CLI, the server jar and a runner-private mysqld, not
|
||||
# its MySQL service or cgroup root. Each arm also runs when an earlier
|
||||
# one failed. Ceilings: 26 arms × 10 = 260 plus ~13 minutes for the
|
||||
# uncapped setup steps (Checkout.. Install MySQL binaries): 273 + 2.
|
||||
timeout-minutes: 275
|
||||
# one failed. Ceilings: 27 arms × 10 = 270 plus ~13 minutes for the
|
||||
# uncapped setup steps (Checkout.. Install MySQL binaries): 283 + 2.
|
||||
timeout-minutes: 285
|
||||
env:
|
||||
QWEN_E2E_REGISTER_STORAGE: '1'
|
||||
steps:
|
||||
@@ -1539,6 +1539,11 @@ jobs:
|
||||
timeout-minutes: 10
|
||||
run: >-
|
||||
npm run test:e2e:managed-harness-restart-approval-sibling-approval
|
||||
- name: 'Run after-earlier-tools crash E2E'
|
||||
if: "${{ !cancelled() && steps.failover_prerequisites.outcome == 'success' }}"
|
||||
timeout-minutes: 10
|
||||
run: >-
|
||||
npm run test:e2e:managed-harness-restart-approval-after-earlier-tools
|
||||
- name: 'Run approval decision reply-loss E2E'
|
||||
if: "${{ !cancelled() && steps.failover_prerequisites.outcome == 'success' }}"
|
||||
timeout-minutes: 10
|
||||
|
||||
@@ -510,7 +510,7 @@ Integration regressions proved that model request preparation failures preserve
|
||||
|
||||
The machine-readable reports, manifests and retained failed diagnostics are under `.qwen/investigations/g3-step3-implementation/`; the current consolidated report is `packaged-verification.md`. These five packaged gates do not prove the whole matrix above. Since [replaceable-Harness D11](2026-10-02-hosted-replaceable-harness.md#d11--residual-gates-step-3-in-ci-the-model-round-arm-and-q2s-binding-half), they are CI steps of the `hosted-failover-e2e` job, which runs apart from the Hosted MySQL verify (see D11), next to a model-round arm that packages the main observations of B1 for a no-tool Session and of B2, though not their attempt and stream identities, the live SSE view or the admission count.
|
||||
|
||||
#14481's packaged gates add thirteen fault arms to the same job. Seven landed with
|
||||
#14481's packaged gates add fourteen fault arms to the same job. Seven landed with
|
||||
the second step. Five layer one fault on the Allow path: three crash the
|
||||
replacement Harness after a
|
||||
durable A5 barrier — once the Allow decision is recorded (A5's decision row),
|
||||
@@ -553,11 +553,18 @@ generation must re-hang the original pending Action with its original deadline,
|
||||
require that sibling's own Allow rather than inherit the earlier one, leave the
|
||||
decided sibling un-re-asked, and settle both calls once each in their original
|
||||
order ([A3 mixed-sibling approval](2026-10-31-hosted-a3-mixed-sibling-approval.md)).
|
||||
A crash after several earlier tool rounds (B3) is packaged too: a multi-round
|
||||
fixture runs two earlier tools, the replacement is crashed while a later post-tool
|
||||
request is held, and the next generation must resume the saved logical round and
|
||||
restore the turn cost snapshot and per-model token ledger rather than restart them,
|
||||
with each earlier execution occurring once and the later request reissued deep-equal
|
||||
([B3 round and cost budget](2026-10-31-hosted-b3-round-cost-budget.md)). "Cost budget"
|
||||
here is the TurnBudget snapshot plus the token ledger, not a cumulative spend gate.
|
||||
Malformed or foreign refs, snapshot overflow, the rewritten-input and
|
||||
Hook-no-rerun half of A3 (a commandStep follow-up, since a native continuation
|
||||
refuses Hook/MCP sessions), Hook/MCP/child
|
||||
uncertainty (A6), the attempt and stream identities plus the live SSE view
|
||||
(B2), the round and cost budgets (B3), and the mixed-version reader/bundle
|
||||
(B2), and the mixed-version reader/bundle
|
||||
closure (R2) stay focused or integration evidence. Full-owner restart, real-provider
|
||||
acceptance and a real mixed-version cluster remain separate. Q2's binding
|
||||
half has the D11 frozen control-plane arm for a woken former Spring's stale
|
||||
|
||||
@@ -416,7 +416,7 @@ Allow 保留原 Action、prepare key、request digest 和 Runtime binding,产
|
||||
|
||||
机器可读报告、manifest 和保留的失败诊断位于 `.qwen/investigations/g3-step3-implementation/`,当前汇总报告为 `packaged-verification.md`。这五种打包门禁不证明上述整个矩阵。自[可替换 Harness 设计 D11](2026-10-02-hosted-replaceable-harness.zh-CN.md#d11--残留门禁step-3-进-ci模型轮支路q2-的-binding-半边) 起,它们是 `hosted-failover-e2e` 任务的 CI 步骤,该任务独立于 Hosted MySQL 的 verify 运行(见 D11);同一任务里还有一条模型轮支路,把无工具 Session 的 B1 与 B2 的主要观察项做成打包门禁,但不覆盖它们的 attempt 与 stream 身份、实时 SSE 视图以及准入次数。
|
||||
|
||||
#14481 的打包门禁在同一任务里新增十三条故障支路,其中七条随第 2 步落地。五条在 Allow 路径上叠加一层故障:三条在持久 A5 屏障之后崩溃替代 Harness——一条在 Allow 决策落库后(A5 的决策行),一条在批准执行到达 `PREPARED`、Runtime Broker 的 `:start` 被扣住时(A5a 的部分 prepare),还有一条在批准结果已提交、post-tool 模型轮被扣住时(A5 的 result commit 屏障)——另两条在提交后丢弃控制面的回复:决策回复丢失后,同一 `Idempotency-Key` 重试必须落到同一条记录,只产生一个 `ACTION_RESPONSE` operation(A4 的决策半边);Turn 准入回复丢失后,重试只产生一行 Turn(C4)。五条都保留原 Action、prepare key、digest 与 Runtime 身份,启动第三代 Harness,并要求一次已结算执行、一次文件效果和一个终态;result-commit 支路还要求重发复用同一个已保存请求,并撤回已发布的片段。另外五条覆盖接下来的屏障:`turn.settled` 已提交但回复丢失时,替换后必须投影一次终态(不要求第二次 store 提交),且不产生推理或工具调用,公开文本仍必须是恢复出的答案(C1);终态写入被拒时,可付 checkpoint 必须由替换方恰好补偿一次,且恢复出的答案仍可见(C2)。清理释放被拒、随后再次崩溃(C3)也已打包:Turn 进入终态且其 Harness 代际已死之后,控制面通过 cleanup-only load 重试欠付清理,直到替换方确认([清理欠债重试](2026-10-28-hosted-cleanup-debt-retry.zh-CN.md))。接管 load 回复丢失(A4 的 load 半边)也已打包:替换方第一次 drive load 的回复被丢失一次(测试门控钩子),控制面把同一 load 重发进已启动的 active driver,原 Turn 仍须恰好投影一个终态并保留恢复出的答案——重发原样重述 active attachment,而不是把 Turn 卡死。早先观察到的卡 RUNNING 变体在当前代码树上复现不出,因此该臂作为回归门禁守住这条交接,而非标记一个未决缺口。主流 Finished 持久标记与 assistant 提交之间的崩溃(B3a)也已打包:测试门控钩子在替换方的 post-tool 轮记录 Finished 之后、assistant 提交之前把它扣在该窗口内,runner 在那里崩溃它,下一代必须 deep-equal 重发已保存请求并投影一个终态、保留恢复出的答案,而不是补偿一个已提交的输出。混合兄弟审批裂缝中的崩溃(A3 的 native 半边)也已打包:一个两调用批次按兄弟逐个审批,测试门控钩子在较早兄弟已决之后、替换方第一次询问审批时把它扣住,runner 在那里崩溃它,下一代必须用原 deadline 重挂原 pending Action、要求该兄弟自己的 Allow 而非继承较早的 Allow、不重问已决兄弟,并按原顺序各结算两个调用一次([A3 混合兄弟审批](2026-10-31-hosted-a3-mixed-sibling-approval.zh-CN.md))。损坏/外部 ref、快照超限、A3 的改写输入与 Hook 不重跑半边(commandStep 后续,因为 native continuation 拒绝 Hook/MCP 会话)、Hook/MCP/child 不确定性(A6)、attempt 与 stream 身份及实时 SSE 视图(B2)、轮次与成本预算(B3),以及混版本 reader/bundle 闭包(R2)仍由定向或集成证据覆盖。两侧 owner 重启、真实 provider 和真实混版本集群仍独立处理。Q2 的 binding 半边有 D11 的冻结控制面支路覆盖醒来的旧 Spring 的过期 claim,并有 store 测试钉住 bind 守卫;两个存活控制面仍属后续阶段。
|
||||
#14481 的打包门禁在同一任务里新增十四条故障支路,其中七条随第 2 步落地。五条在 Allow 路径上叠加一层故障:三条在持久 A5 屏障之后崩溃替代 Harness——一条在 Allow 决策落库后(A5 的决策行),一条在批准执行到达 `PREPARED`、Runtime Broker 的 `:start` 被扣住时(A5a 的部分 prepare),还有一条在批准结果已提交、post-tool 模型轮被扣住时(A5 的 result commit 屏障)——另两条在提交后丢弃控制面的回复:决策回复丢失后,同一 `Idempotency-Key` 重试必须落到同一条记录,只产生一个 `ACTION_RESPONSE` operation(A4 的决策半边);Turn 准入回复丢失后,重试只产生一行 Turn(C4)。五条都保留原 Action、prepare key、digest 与 Runtime 身份,启动第三代 Harness,并要求一次已结算执行、一次文件效果和一个终态;result-commit 支路还要求重发复用同一个已保存请求,并撤回已发布的片段。另外五条覆盖接下来的屏障:`turn.settled` 已提交但回复丢失时,替换后必须投影一次终态(不要求第二次 store 提交),且不产生推理或工具调用,公开文本仍必须是恢复出的答案(C1);终态写入被拒时,可付 checkpoint 必须由替换方恰好补偿一次,且恢复出的答案仍可见(C2)。清理释放被拒、随后再次崩溃(C3)也已打包:Turn 进入终态且其 Harness 代际已死之后,控制面通过 cleanup-only load 重试欠付清理,直到替换方确认([清理欠债重试](2026-10-28-hosted-cleanup-debt-retry.zh-CN.md))。接管 load 回复丢失(A4 的 load 半边)也已打包:替换方第一次 drive load 的回复被丢失一次(测试门控钩子),控制面把同一 load 重发进已启动的 active driver,原 Turn 仍须恰好投影一个终态并保留恢复出的答案——重发原样重述 active attachment,而不是把 Turn 卡死。早先观察到的卡 RUNNING 变体在当前代码树上复现不出,因此该臂作为回归门禁守住这条交接,而非标记一个未决缺口。主流 Finished 持久标记与 assistant 提交之间的崩溃(B3a)也已打包:测试门控钩子在替换方的 post-tool 轮记录 Finished 之后、assistant 提交之前把它扣在该窗口内,runner 在那里崩溃它,下一代必须 deep-equal 重发已保存请求并投影一个终态、保留恢复出的答案,而不是补偿一个已提交的输出。混合兄弟审批裂缝中的崩溃(A3 的 native 半边)也已打包:一个两调用批次按兄弟逐个审批,测试门控钩子在较早兄弟已决之后、替换方第一次询问审批时把它扣住,runner 在那里崩溃它,下一代必须用原 deadline 重挂原 pending Action、要求该兄弟自己的 Allow 而非继承较早的 Allow、不重问已决兄弟,并按原顺序各结算两个调用一次([A3 混合兄弟审批](2026-10-31-hosted-a3-mixed-sibling-approval.zh-CN.md))。若干先前工具轮之后的崩溃(B3)也已打包:一个多轮夹具跑两个先前工具,替换方在较后的 post-tool 请求被扣住时崩溃,下一代必须续跑已保存的逻辑轮次、恢复回合成本快照与每模型 token 账本而非重启它们,每个先前执行恰好一次、较后请求 deep-equal 重发([B3 轮次与成本预算](2026-10-31-hosted-b3-round-cost-budget.zh-CN.md))。此处「成本预算」指 TurnBudget 快照加 token 账本,而非累计花费闸门。损坏/外部 ref、快照超限、A3 的改写输入与 Hook 不重跑半边(commandStep 后续,因为 native continuation 拒绝 Hook/MCP 会话)、Hook/MCP/child 不确定性(A6)、attempt 与 stream 身份及实时 SSE 视图(B2),以及混版本 reader/bundle 闭包(R2)仍由定向或集成证据覆盖。两侧 owner 重启、真实 provider 和真实混版本集群仍独立处理。Q2 的 binding 半边有 D11 的冻结控制面支路覆盖醒来的旧 Spring 的过期 claim,并有 store 测试钉住 bind 守卫;两个存活控制面仍属后续阶段。
|
||||
|
||||
## 10. 实现范围
|
||||
|
||||
|
||||
@@ -0,0 +1,227 @@
|
||||
# Hosted B3 — round and cost budget across a replacement
|
||||
|
||||
[English](2026-10-31-hosted-b3-round-cost-budget.md) | [简体中文](2026-10-31-hosted-b3-round-cost-budget.zh-CN.md)
|
||||
|
||||
Status: proposed, 2026-10-31. Part of [#14481][issue] (G3 Step 3 recovery-matrix
|
||||
acceptance, tracks #12380). Extends the packaged fault arms of the
|
||||
`hosted-failover-e2e` CI job described in
|
||||
[G3 Step 3 recovery](2026-10-09-hosted-g3-step3-recovery.md). This slice packages
|
||||
matrix row **B3 ("after earlier tools")**: a crash after several tool rounds must
|
||||
recover the logical round and the turn cost ledger without resetting them.
|
||||
|
||||
## Problem
|
||||
|
||||
Matrix row B3 reads: _"Request uses saved results/history; each previous execution
|
||||
occurs once; round and cost budget do not reset."_ Today B3 is focused/integration
|
||||
evidence only. The existing `after-result` arm crashes after a **single** tool round
|
||||
and proves the saved request is reissued deep-equal, but nothing crashes after
|
||||
**several** tool rounds and proves the numeric round counter and the turn cost
|
||||
ledger survive the replacement. A recovery that restarts the round at 0 (letting a
|
||||
Turn run past its configured round limit) or zeroes the token ledger would pass CI.
|
||||
|
||||
**What "cost budget" actually is here (verified).** There is no free-standing
|
||||
cumulative cost counter. The recoverable cost state is the `TurnBudget` snapshot
|
||||
(the output-token budget parsed from a user `+Nk` directive, plus
|
||||
`outputTokensAtTurnStart`) and the per-model session token metrics (uiTelemetry),
|
||||
both persisted on the `model.attempt` route/usage resources and restored on a cold
|
||||
load. The `toolOutputBatchBudget` is stateless per-generation config (harmless to
|
||||
recompute) and the checkpoint `budgetConsumed` field is a constant-0 placeholder;
|
||||
neither is a B3 assertion target. So "cost budget do not reset" is proven as
|
||||
**snapshot + ledger restoration fidelity**, not as a hard spend gate in the main
|
||||
model loop (the loop does not stop on token spend; only workflow/agent dispatch
|
||||
consumes the budget).
|
||||
|
||||
## Current state (verified on `main` at `b3e9ce0cd1`)
|
||||
|
||||
- **Logical round.** The model loop starts at
|
||||
`round = input.resumeModel?.round ?? input.initialRound ?? 0`
|
||||
(`hosted-harness-model.ts:979`) and enforces
|
||||
`maxRounds > 0 && round >= maxRounds → HostedModelRoundLimitError`
|
||||
(`hosted-harness-model.ts:991`). `maxRounds = config.getMaxSessionTurns()` defaults
|
||||
to **-1 (unlimited)**. Because the round index is absolute, the round limit alone
|
||||
does not cleanly detect a reset, so this arm detects a round reset through the
|
||||
exactly-once, deep-equal and round-continuity assertions rather than the limit.
|
||||
The round is persisted two ways:
|
||||
the `hosted-model-request` resource carries `round`
|
||||
(`hosted-model-recovery.ts:33`), and the `hosted-approval-continuation` plan
|
||||
carries `round` (`hosted-approval-continuation.ts:88`, set from `logicalRound` at
|
||||
`hosted-workspace-tool-turn.ts:3738`). On recovery the loop resumes from
|
||||
`resumeModel.round` (saved-request path) or `initialRound = plan.round + 1`
|
||||
(`hosted-harness-session.ts:3981`).
|
||||
- **Cost ledger.** `bindBudget` (`managed-hook-activation.ts:83-135`) on a cold load
|
||||
reverse-scans `model.attempt` events, reads `budget` and `models` from the
|
||||
`usageRef`/`routeRef` resources, restores the `TurnBudget` snapshot and calls
|
||||
`restoreSessionModelMetrics`. It begins a new snapshot only when
|
||||
`shared.snapshot?.promptId !== this.turnId`, so a same-turn recovery does not
|
||||
restart the ledger. Each attempt writes the route resource (`budget`) and usage
|
||||
resource (`budget` + `models`) (`managed-hook-activation.ts:176-183, 230-241`).
|
||||
- **Exactly-once / saved results.** The saved-request replay is pinned by
|
||||
`requestDigest` and refused if any assistant commit or `tool.intent` follows the
|
||||
snapshot's `throughSequence` (`hosted-model-recovery.ts:213-222`); the tool-turn
|
||||
fold is keyed on the functionResponse id (`hosted-workspace-tool-turn.ts:1666`),
|
||||
and `resumeSavedNativeBatch` re-validates the original definition/argument digests
|
||||
and reuses the original `runtimeCallId`/`prepareKey`.
|
||||
- **Runner observability.** The round is NOT in the provider request body or the
|
||||
public feed; it lives in the `hosted-model-request` / `hosted-approval-continuation`
|
||||
MySQL resources. The budget/models live in the `managed-hosted-model-route` /
|
||||
`managed-hosted-model-usage` resources. The runner already has both readers (journal
|
||||
HEX parse and `ma_resource` inline read). There is currently **no** runner assertion
|
||||
on the numeric round or the budget content.
|
||||
|
||||
## Goals and scope
|
||||
|
||||
**In scope.** A packaged `hosted-failover-e2e` arm that runs a **multi-round** tool
|
||||
fixture (two earlier tool rounds), crashes the replacement while a later post-tool
|
||||
model request is held, and proves, from MySQL alone after scrashing the crashed
|
||||
generation's disk:
|
||||
|
||||
1. the recovered model request resumes at the **same logical round** as the saved
|
||||
request (not restarted at 0/1);
|
||||
2. the turn **cost snapshot and per-model token ledger** are restored, not zeroed or
|
||||
re-baselined;
|
||||
3. each earlier tool execution occurs **exactly once** with its original execution
|
||||
identity, and the reissued request is deep-equal to the saved one;
|
||||
4. the Turn projects exactly one terminal with the recovered answer.
|
||||
|
||||
**Out of scope.** A hard token-spend gate in the main model loop (none exists); the
|
||||
`toolOutputBatchBudget` (stateless config); the checkpoint `budgetConsumed`
|
||||
placeholder; the skill-only / manual-continue recovery paths that carry no
|
||||
`initialRound` (their reachability in a packaged native-tool arm is unverified and
|
||||
left as a follow-up); real-provider acceptance; full-owner restart.
|
||||
|
||||
**No production behavior change.** The crash is held on the fake-model side (the
|
||||
existing `holdUntil` mechanism), so this arm needs **no new serve hook** — unlike
|
||||
A3/B3a. The only production-source edit, if any, is test-gated and inert.
|
||||
|
||||
## Proposed solution
|
||||
|
||||
### Window choice (W2)
|
||||
|
||||
Crash in the **after-result window of a later round**: the Nth (N≥2) post-tool model
|
||||
request has been published (snapshot + `model.attempt: started` durable) and is held
|
||||
by the fake model before it finishes. This is the lowest-risk window — it reuses the
|
||||
`after-result` hold/barrier machinery, the hold point is on the fake-server side (no
|
||||
new serve hook), and the saved request carries the round while the crash-boundary
|
||||
`managed-hosted-model-route` / `managed-hosted-model-usage` resources carry the
|
||||
budget, so recovery must restore both and the runner reads each from its own
|
||||
resource. (The alternative W1 — between a tool-result commit and
|
||||
the next snapshot — may not be stoppable and could route to a round-resetting path;
|
||||
it is deferred pending a dry-run. W3 overlaps B3a's boundary and adds only the
|
||||
multi-round/budget dimension, folded into W2's assertions.)
|
||||
|
||||
### Multi-round fixture
|
||||
|
||||
The fake model decides by the count of `"role":"tool"` messages: the first two
|
||||
requests return a `write_file` tool call each (two distinct paths/effects), and the
|
||||
third returns the held final answer. The prompt carries a `+Nk` directive so the
|
||||
`TurnBudget` snapshot is non-null and its restoration is numerically assertable.
|
||||
Each response also charges non-zero usage (300/300/100 completion tokens): the fake
|
||||
server's default usage is all zeros, which would leave the token ledger numerically
|
||||
empty, and the earlier rounds outweigh the final round so a zeroed restore totals
|
||||
less than the crash-boundary ledger instead of re-counting the same sum.
|
||||
|
||||
### Recovery assertions (new, on top of the after-result template)
|
||||
|
||||
- **Round continuity**: read the saved `hosted-model-request.round` before the crash
|
||||
and the recovered request's round after; require equality (and ≥2, so a reset to 0
|
||||
reds). A reset that re-runs the earlier rounds is additionally caught by the
|
||||
exactly-once execution count and the deep-equal request parity.
|
||||
- **Cost ledger continuity**: read the route/usage resource `budget` and `models`
|
||||
before and after; require the crash-boundary snapshot to be non-null (a null one
|
||||
would let a dropped restore pass vacuously as null == null) and deep-equal after
|
||||
recovery, and every crash-boundary model's output-token total to survive (after ≥
|
||||
before). The crash-boundary ledger must be non-empty, so a fixture that never
|
||||
charged the ledger reds instead of passing vacuously.
|
||||
- **Exactly-once + deep-equal**: two earlier executions, each `SETTLED` /
|
||||
`dispatch_generation=1` under the original Runtime identity (binding / generation /
|
||||
runtimeSessionId); the first approval plan carries only tool A, so the two physical
|
||||
file effects give the per-tool evidence; the reissued post-tool request deep-equal
|
||||
to the saved one; one terminal; recovered answer.
|
||||
|
||||
### Runner, CI and contract pins
|
||||
|
||||
A new `approvalFaults` entry (allow-only overlay), a barrier branch in the exhaustive
|
||||
crash-arm guard, an npm script and a `sdk-java.yml` step, and a bump of the
|
||||
`hosted-failover-e2e` arm ceiling and timeout. Contract pins pin the multi-round
|
||||
fixture, the round/budget readers and their assertions; any new top-level runner
|
||||
helper gets a behavioural test on injected inputs.
|
||||
|
||||
### Documentation
|
||||
|
||||
Update [G3 Step 3 recovery](2026-10-09-hosted-g3-step3-recovery.md) §9.1 in both
|
||||
languages: thirteen → fourteen packaged arms, move B3 from focused to packaged, and
|
||||
record the "cost budget = TurnBudget snapshot + token ledger" clarification.
|
||||
|
||||
## Design decisions and rationale
|
||||
|
||||
- **W2 over W1/W3** — W2 reuses the proven after-result hold with no new serve hook;
|
||||
W1's recovery routing is unverified and may be unstoppable; W3 duplicates B3a's
|
||||
boundary.
|
||||
- **Cost = snapshot + ledger, not a spend gate** — matches the actual mechanism; the
|
||||
arm asserts restoration fidelity, which is what "do not reset" means here.
|
||||
- **Explicit `+Nk` directive** — the default null budget makes a ledger reset
|
||||
unobservable; the fixture opts in so the snapshot restoration is numerically
|
||||
assertable. No `maxSessionTurns` is set: the absolute round index means the limit
|
||||
does not cleanly detect a reset, so reset detection rests on the exactly-once,
|
||||
deep-equal and round-continuity assertions.
|
||||
- **Multi-round gated on the fault name** — the shared fake model and assertions are
|
||||
used by every approval arm; gating keeps the single-round arms unchanged.
|
||||
|
||||
## Constraints and risks
|
||||
|
||||
- **Shared fixture regression.** The fake model and execution-identity assertions are
|
||||
shared; the multi-round path must be fully gated and the single-round arms re-run
|
||||
green. Every gate the arm shares with `after-result` — arming AND releasing the
|
||||
hold, capturing the published prefix, and the retraction proof — must enumerate
|
||||
both fault names: a release gated to `after-result` alone left the recovery
|
||||
generation's reissued request held forever (caught by the first .75 head run).
|
||||
- **Round/budget readers must mirror recovery.** The round/budget assertions read the
|
||||
same durable resources recovery reads; a behavioural test on injected rows proves
|
||||
the readers, and a mutation must red them. Resources are read as `HEX(inline_bytes)`
|
||||
and decoded in the runner: `mysql --batch` escapes backslashes, which corrupts the
|
||||
saved request's `\"` sequences (the request carries the full system prompt).
|
||||
- **Ledger shape (confirmed).** An unrestored ledger is `{}`, not null —
|
||||
`getMetricsForSession` falls back to `createInitialMetrics()` — and `{}` is truthy,
|
||||
so a null check alone would let a zeroed restore pass. The assertion therefore
|
||||
compares per-model output-token totals (after ≥ before, the ledger grows as the
|
||||
held round completes) and requires a non-empty crash-boundary ledger.
|
||||
- **CI budget.** One more arm raises the `hosted-failover-e2e` ceiling and timeout.
|
||||
|
||||
## Validation plan
|
||||
|
||||
- **.75 real stack**: head arm green — round continuity, cost ledger continuity, two
|
||||
earlier executions once each, deep-equal reissue, one terminal, recovery from MySQL
|
||||
alone after scrash.
|
||||
- **Demonstrated mutation reds**: reset the recovered round to 0 → round-continuity
|
||||
red; zero the restored models ledger → ledger red; drop the `+Nk` snapshot restore
|
||||
→ budget red (each killed at its named assertion on the .75 real stack). The
|
||||
execution-count assertion is defense-in-depth in W2: three mutation cells
|
||||
(empty fold predicate; broken replay routing; both together) all SURVIVED, because
|
||||
exactly-once here rides three independent layers — the saved-request replay
|
||||
routing, the continuation route's settled-batch adoption, and the Runtime's
|
||||
runtimeCallId idempotency (Java-side, unreachable from CLI mutations). Reddening
|
||||
the count would require a cross-language compound; that is recorded as a
|
||||
follow-up. The count/identity assertions are shared with the packaged,
|
||||
CI-proven sibling-approval and after-result arms.
|
||||
- Contract suite + behavioural tests green; `npm run build && npm run typecheck &&
|
||||
npm run bundle`.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A packaged arm in `hosted-failover-e2e` whose runner asserts round continuity, cost
|
||||
ledger continuity, exactly-once earlier executions and one terminal from MySQL alone
|
||||
after scrashing the crashed generation.
|
||||
- Each new round/budget/ledger signal has a demonstrated named mutation red; the
|
||||
shared exactly-once count/identity assertions are defense-in-depth in W2, with
|
||||
survived-mutation evidence documenting the independent guard layers.
|
||||
- Bilingual design docs aligned; CI green; no production behavior change.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Resolved: the fault name is `after-earlier-tools` and the directive is `+500k`
|
||||
(the .75 crash-boundary evidence pins the snapshot at `budget: 500000`,
|
||||
`outputTokensAtTurnStart: 0`).
|
||||
- Whether the skill-only / manual-continue round-reset paths are reachable in a
|
||||
packaged arm (deferred; would be a separate gap if confirmed).
|
||||
|
||||
[issue]: https://github.com/QwenLM/qwen-code/issues/14481
|
||||
@@ -0,0 +1,180 @@
|
||||
# Hosted B3 —— 跨替换的轮次与成本预算
|
||||
|
||||
[English](2026-10-31-hosted-b3-round-cost-budget.md) | [简体中文](2026-10-31-hosted-b3-round-cost-budget.zh-CN.md)
|
||||
|
||||
状态:提议中,2026-10-31。属于 [#14481][issue](G3 Step 3 恢复矩阵验收,跟踪
|
||||
#12380)。扩展 [G3 Step 3 恢复](2026-10-09-hosted-g3-step3-recovery.zh-CN.md) 所述
|
||||
`hosted-failover-e2e` CI job 的打包故障支路。本切片打包矩阵行 **B3(「after earlier
|
||||
tools」)**:在若干工具轮之后崩溃,恢复必须不重置逻辑轮次与回合成本账本。
|
||||
|
||||
## 问题
|
||||
|
||||
矩阵行 B3 写道:_「请求使用已保存的结果/历史;每次先前执行恰好发生一次;轮次与成本
|
||||
预算不重置。」_ 目前 B3 仅有 focused/integration 证据。现有 `after-result` 支路在
|
||||
**单个**工具轮之后崩溃、证明已保存请求被 deep-equal 重发,但没有任何支路在**若干**
|
||||
工具轮之后崩溃并证明数值轮次计数器与回合成本账本跨替换存活。一个把轮次重启为 0
|
||||
(让 Turn 跑过其配置轮次上限)或把 token 账本归零的恢复,本来都能通过 CI。
|
||||
|
||||
**这里「成本预算」的真实含义(已实证)。** 本仓没有独立的累计成本计数器。可恢复的
|
||||
成本状态是 `TurnBudget` 快照(从用户 `+Nk` 指令解析出的输出 token 预算,加
|
||||
`outputTokensAtTurnStart`)与每模型会话 token 指标(uiTelemetry),二者持久化在
|
||||
`model.attempt` 的 route/usage 资源上并在冷加载时恢复。`toolOutputBatchBudget` 是
|
||||
每代重新计算的无状态配置(重算无害),checkpoint 的 `budgetConsumed` 字段是恒 0
|
||||
占位符;二者都不是 B3 的断言对象。因此「成本预算不重置」被证明为**快照 + 账本的恢复
|
||||
保真**,而非主模型循环里的硬性花费闸门(主循环不按 token 花费停轮;只有
|
||||
workflow/agent 派发消费该预算)。
|
||||
|
||||
## 现状(在 `main` 的 `b3e9ce0cd1` 上实证)
|
||||
|
||||
- **逻辑轮次。** 模型循环从
|
||||
`round = input.resumeModel?.round ?? input.initialRound ?? 0`
|
||||
(`hosted-harness-model.ts:979`)起跳,并强制
|
||||
`maxRounds > 0 && round >= maxRounds → HostedModelRoundLimitError`
|
||||
(`hosted-harness-model.ts:991`)。`maxRounds = config.getMaxSessionTurns()` 默认
|
||||
**-1(无限制)**。由于轮次索引是绝对的,轮次上限本身无法干净地检测重置,因此本臂
|
||||
通过恰好一次、deep-equal 与轮次连续性断言来检测轮次重置,而非依赖上限。轮次经两条通道持久化:`hosted-model-request` 资源带 `round`
|
||||
(`hosted-model-recovery.ts:33`),`hosted-approval-continuation` plan 带 `round`
|
||||
(`hosted-approval-continuation.ts:88`,在 `hosted-workspace-tool-turn.ts:3738`
|
||||
由 `logicalRound` 赋值)。恢复时循环从 `resumeModel.round`(saved-request 路径)或
|
||||
`initialRound = plan.round + 1`(`hosted-harness-session.ts:3981`)续跑。
|
||||
- **成本账本。** `bindBudget`(`managed-hook-activation.ts:83-135`)冷加载时逆序扫
|
||||
`model.attempt` 事件,从 `usageRef`/`routeRef` 资源读回 `budget` 与 `models`,恢复
|
||||
`TurnBudget` 快照并调用 `restoreSessionModelMetrics`。仅当
|
||||
`shared.snapshot?.promptId !== this.turnId` 才起新快照,因此同轮恢复不会重启账本。
|
||||
每次 attempt 写 route 资源(`budget`)与 usage 资源(`budget` + `models`)
|
||||
(`managed-hook-activation.ts:176-183, 230-241`)。
|
||||
- **恰好一次 / 已存结果。** saved-request 重放由 `requestDigest` 钉死,且若快照
|
||||
`throughSequence` 之后出现 assistant commit 或 `tool.intent` 即拒绝
|
||||
(`hosted-model-recovery.ts:213-222`);tool-turn fold 以 functionResponse id 为键
|
||||
(`hosted-workspace-tool-turn.ts:1666`),`resumeSavedNativeBatch` 重验原始定义/参数
|
||||
摘要并复用原 `runtimeCallId`/`prepareKey`。
|
||||
- **runner 可观测性。** 轮次不在 provider 请求体、也不在公共 feed;它在
|
||||
`hosted-model-request` / `hosted-approval-continuation` MySQL 资源里。budget/models
|
||||
在 `managed-hosted-model-route` / `managed-hosted-model-usage` 资源里。runner 已有
|
||||
这两种读法(journal HEX 解析与 `ma_resource` inline 读)。目前 runner **没有**对
|
||||
数值轮次或 budget 内容的断言。
|
||||
|
||||
## 目标与范围
|
||||
|
||||
**范围内。** 一个打包的 `hosted-failover-e2e` 支路:跑一个**多轮**工具夹具(两个先前
|
||||
工具轮),在较后的 post-tool 模型请求被扣住时崩溃替换进程,并在 scrash 崩溃代磁盘后
|
||||
仅凭 MySQL 证明:
|
||||
|
||||
1. 恢复出的模型请求以与已保存请求**相同的逻辑轮次**续跑(不重启为 0/1);
|
||||
2. 回合**成本快照与每模型 token 账本**被恢复,而非归零或重新起算;
|
||||
3. 每个先前工具执行以原执行身份**恰好发生一次**,且重发请求与已保存请求 deep-equal;
|
||||
4. Turn 恰好投影一个 terminal 并带恢复出的答案。
|
||||
|
||||
**范围外。** 主模型循环里的硬性 token 花费闸门(不存在);`toolOutputBatchBudget`
|
||||
(无状态配置);checkpoint `budgetConsumed` 占位符;不带 `initialRound` 的 skill-only
|
||||
/ 手动 continue 恢复路径(其在打包 native-tool 臂中的可达性未验证,留作后续);
|
||||
real-provider 验收;两侧 owner 重启。
|
||||
|
||||
**无生产行为改动。** 崩溃由 fake 模型侧扣住(既有 `holdUntil` 机制),因此本支路
|
||||
**无需新 serve 钩子**——不同于 A3/B3a。唯一的生产源码改动(若有)是测试 gated 且惰性。
|
||||
|
||||
## 方案
|
||||
|
||||
### 窗口选择(W2)
|
||||
|
||||
崩在**较后一轮的 after-result 窗口**:第 N(N≥2)轮 post-tool 模型请求已发布(快照 +
|
||||
`model.attempt: started` 已 durable),被 fake 模型在完成前扣住。这是风险最低的窗口
|
||||
——复用 `after-result` 的 hold/barrier 机器,hold 点在 fake 服务器侧(无需新 serve
|
||||
钩子),且已保存请求携带轮次、崩溃边界的 `managed-hosted-model-route` /
|
||||
`managed-hosted-model-usage` 资源携带预算,恢复必须把两者都恢复,runner 也从各自
|
||||
的资源分别读取。(备选 W1——工具结果 commit
|
||||
与下一快照之间——可能停不住、且可能落到重置轮次的路径,干跑确认前延后。W3 与 B3a
|
||||
边界重合、仅增的多轮/budget 维度并入 W2 断言。)
|
||||
|
||||
### 多轮夹具
|
||||
|
||||
fake 模型按 `"role":"tool"` 消息计数决定:前两个请求各回一个 `write_file` 工具调用
|
||||
(两个不同路径/效果),第三个回被扣住的最终答案。提示词带 `+Nk` 指令使 `TurnBudget`
|
||||
快照非空、其恢复可数值断言。每个响应还计入非零 usage(completion tokens
|
||||
300/300/100):fake 服务器默认 usage 全 0,会让 token 账本在数值上为空;且先前轮次
|
||||
重于最终轮,使归零恢复的总量**低于**崩溃边界账本,而非把同一总和重算一遍。
|
||||
|
||||
### 恢复断言(在 after-result 模板之上新增)
|
||||
|
||||
- **轮次连续**:崩溃前读已保存 `hosted-model-request.round`,恢复后读恢复请求的轮次,
|
||||
要求相等(且 ≥2,使重置为 0 变红)。重跑先前轮次的重置还会被恰好一次执行计数与
|
||||
deep-equal 请求 parity 捕获。
|
||||
- **成本账本连续**:崩溃前后读 route/usage 资源的 `budget` 与 `models`,要求崩溃边界
|
||||
快照非空(null 会让丢弃恢复以 null == null 空过)且恢复后 deep-equal,且崩溃边界
|
||||
每个模型的输出 token 总量存活(after ≥ before)。崩溃边界账本必须非空,使从未计入
|
||||
账本的夹具变红而非空过。
|
||||
- **恰好一次 + deep-equal**:两个先前执行各 `SETTLED`/`dispatch_generation=1`、带原
|
||||
Runtime 身份(binding/generation/runtimeSessionId);首个审批 plan 只含工具 A,故
|
||||
两个物理文件效果提供逐工具证据;重发 post-tool 请求与已保存请求 deep-equal;一个
|
||||
terminal;恢复出的答案。
|
||||
|
||||
### runner、CI 与契约钉
|
||||
|
||||
新增一个 `approvalFaults` 条目(allow-only 叠加)、穷举崩溃臂守卫里的 barrier 分支、
|
||||
一个 npm script 与一个 `sdk-java.yml` step,并抬 `hosted-failover-e2e` 的臂数上限与
|
||||
timeout。契约钉钉住多轮夹具、轮次/budget 读法及其断言;任何新 runner 顶层 helper 配
|
||||
注入输入的行为化测试。
|
||||
|
||||
### 文档
|
||||
|
||||
双语更新 [G3 Step 3 恢复](2026-10-09-hosted-g3-step3-recovery.zh-CN.md) §9.1:十三→
|
||||
十四条打包支路,把 B3 从 focused 移到 packaged,并记录「成本预算 = TurnBudget 快照 +
|
||||
token 账本」的澄清。
|
||||
|
||||
## 设计决定与理由
|
||||
|
||||
- **W2 优于 W1/W3** —— W2 复用已验证的 after-result hold、无需新 serve 钩子;W1 的
|
||||
恢复路由未验证、可能停不住;W3 与 B3a 边界重复。
|
||||
- **成本 = 快照 + 账本,非花费闸门** —— 符合真实机制;支路断言恢复保真,这正是此处
|
||||
「不重置」的含义。
|
||||
- **显式 `+Nk` 指令** —— 默认 null 预算使账本重置不可观测;夹具显式开启使快照恢复可
|
||||
数值断言。不设 `maxSessionTurns`:绝对轮次索引意味着上限无法干净检测重置,故重置
|
||||
检测依赖恰好一次、deep-equal 与轮次连续性断言。
|
||||
- **多轮 gated 在故障名上** —— 共享 fake 模型与断言被每个审批支路使用;gating 使单轮
|
||||
支路不变。
|
||||
|
||||
## 约束与风险
|
||||
|
||||
- **共享夹具回归。** fake 模型与执行身份断言共享;多轮路径必须完全 gated,单轮支路
|
||||
重跑绿。本臂与 `after-result` 共享的每个 gate——hold 的 arm **与释放**、已发布前缀
|
||||
的采集、前缀撤回证明——都必须枚举两个故障名:只 gate 在 `after-result` 的释放会让
|
||||
恢复代的重发请求被永远扣住(第一次 .75 head 运行即抓住)。
|
||||
- **轮次/budget 读法须镜像恢复。** 轮次/budget 断言读取与恢复相同的 durable 资源;
|
||||
注入行的行为化测试证明读法,变异必须使其变红。资源以 `HEX(inline_bytes)` 读取并在
|
||||
runner 内解码:`mysql --batch` 会转义反斜杠,损坏已保存请求 JSON 的 `\"` 序列
|
||||
(请求携带完整 system prompt)。
|
||||
- **账本形态(已确认)。** 未恢复的账本是 `{}` 而非 null——`getMetricsForSession`
|
||||
回退到 `createInitialMetrics()`——而 `{}` 为 truthy,仅查 null 会让归零恢复通过。
|
||||
因此断言比较逐模型输出 token 总量(after ≥ before,账本随被扣住轮次完成而增长),
|
||||
并要求崩溃边界账本非空。
|
||||
- **CI 预算。** 多一条臂抬 `hosted-failover-e2e` 上限与 timeout。
|
||||
|
||||
## 验证计划
|
||||
|
||||
- **.75 真机**:head 臂绿——轮次连续、成本账本连续、两个先前执行各一次、deep-equal
|
||||
重发、一个 terminal、scrash 后仅凭 MySQL 恢复。
|
||||
- **实测变异红**:把恢复轮次重置为 0→轮次连续红;把恢复的 models 账本归零→账本红;
|
||||
丢弃 `+Nk` 快照恢复→budget 红(三者均在 .75 真机于命名断言处被杀)。执行计数断言
|
||||
在 W2 是纵深防御:三个变异格(空 fold 谓词;破坏 replay 路由;两者同时)全部
|
||||
**存活**,因为此处恰好一次依赖三道独立守卫——saved-request replay 路由、
|
||||
continuation 路由的 settled-batch 采纳、Runtime 的 runtimeCallId 幂等(Java 侧,
|
||||
CLI 变异不可达)。使计数变红需要跨语言复合变异;已记录为后续。计数/身份断言与
|
||||
已打包且 CI 证明的 sibling-approval、after-result 支路共享。
|
||||
- 契约套件 + 行为化测试绿;`npm run build && npm run typecheck && npm run bundle`。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- `hosted-failover-e2e` 中一个打包支路,其 runner 在 scrash 崩溃代后仅凭 MySQL 断言
|
||||
轮次连续、成本账本连续、先前执行恰好一次与一个 terminal。
|
||||
- 每个新增轮次/budget/账本信号都有一次实测命名变异红;共享的恰好一次计数/身份断言
|
||||
在 W2 为纵深防御,以存活变异证据记录各独立守卫层。
|
||||
- 双语设计文档对齐;CI 绿;无生产行为改动。
|
||||
|
||||
## 开放问题
|
||||
|
||||
- 已解决:故障名为 `after-earlier-tools`,指令为 `+500k`(.75 崩溃边界证据钉住快照
|
||||
`budget: 500000`、`outputTokensAtTurnStart: 0`)。
|
||||
- skill-only / 手动 continue 的轮次重置路径在打包臂中是否可达(延后;若确认将是独立
|
||||
缺口)。
|
||||
|
||||
[issue]: https://github.com/QwenLM/qwen-code/issues/14481
|
||||
@@ -82,6 +82,7 @@
|
||||
"test:e2e:managed-harness-restart-approval-after-result": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault after-result",
|
||||
"test:e2e:managed-harness-restart-approval-finished-before-commit": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault finished-before-commit",
|
||||
"test:e2e:managed-harness-restart-approval-sibling-approval": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault sibling-approval",
|
||||
"test:e2e:managed-harness-restart-approval-after-earlier-tools": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault after-earlier-tools",
|
||||
"test:e2e:managed-harness-restart-approval-cleanup-failed": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault cleanup-failed",
|
||||
"test:e2e:managed-harness-restart-approval-decision-reply-lost": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault decision-reply-lost",
|
||||
"test:e2e:managed-harness-restart-approval-admission-reply-lost": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault admission-reply-lost",
|
||||
|
||||
@@ -1824,12 +1824,25 @@ un-re-asked, and settle both calls once each in their original order:
|
||||
npm run test:e2e:managed-harness-restart-approval-sibling-approval
|
||||
```
|
||||
|
||||
A crash after several earlier tool rounds (B3) runs a multi-round fixture with two
|
||||
earlier tools and holds the replacement while a later post-tool request is open
|
||||
(the hold is on the fake-model side, so no serve hook is needed). The runner kills
|
||||
it there; the next generation must resume the saved logical round and restore the
|
||||
turn cost snapshot and per-model token ledger rather than restart them, with each
|
||||
earlier execution occurring once and the later request reissued deep-equal. "Cost
|
||||
budget" here is the TurnBudget snapshot plus the token ledger, not a cumulative
|
||||
spend gate:
|
||||
|
||||
```bash
|
||||
npm run test:e2e:managed-harness-restart-approval-after-earlier-tools
|
||||
```
|
||||
|
||||
They write their evidence under `.qwen/investigations/g3-step3-implementation/`;
|
||||
`G3_VERIFY_REPORT_DIR` overrides the directory. The remaining rows stay
|
||||
focused or integration tested for now: the rewritten-input and Hook-no-rerun half
|
||||
of mixed calls (a commandStep follow-up, since a native continuation refuses
|
||||
Hook/MCP sessions), Hook/MCP/child uncertainty, the attempt and stream
|
||||
identities plus the live SSE view, the round and cost budgets, and the
|
||||
identities plus the live SSE view, and the
|
||||
mixed-version reader/bundle closure (#14481).
|
||||
|
||||
Two arms keep a former owner alive while its replacement finishes the Turn
|
||||
|
||||
@@ -59,6 +59,9 @@ let originalApprovalPlan: SavedApprovalPlan | undefined;
|
||||
// re-decided by the recovery generation.
|
||||
let siblingApprovalId: string | undefined;
|
||||
let siblingApprovalOptions: SavedApprovalOptions | undefined;
|
||||
// B3: tool B's new Action, detected after Allow A and decided by the runner's
|
||||
// second Allow; both earlier Actions must end decided under their original IDs.
|
||||
let secondApprovalId: string | undefined;
|
||||
let verifiedOriginalTerminal: string | undefined;
|
||||
function saveEvidence(name: string, body: unknown): void {
|
||||
if (!verificationMode) return;
|
||||
@@ -91,6 +94,7 @@ const approvalFaults = [
|
||||
'after-result',
|
||||
'finished-before-commit',
|
||||
'sibling-approval',
|
||||
'after-earlier-tools',
|
||||
'cleanup-failed',
|
||||
'decision-reply-lost',
|
||||
'admission-reply-lost',
|
||||
@@ -137,7 +141,7 @@ for (let index = 0; index < argumentsList.length; index += 1) {
|
||||
index += 1;
|
||||
} else {
|
||||
throw new Error(
|
||||
'Usage: run-managed-agent-server-e2e.ts [--model ID] [--runtime-delay-ms N] [--settings PATH] [--session-failover|--inflight-failover|--continuation-failover|--model-round-failover|--big-output] [--harness-only] [--freeze] [--frozen-control-plane] [--g3-verify allow|deny|cancel|expiry|continuation] [--g3-fault after-decision|after-prepare|after-result|finished-before-commit|sibling-approval|cleanup-failed|decision-reply-lost|admission-reply-lost|load-reply-lost|terminal-reply-lost|terminal-write-failed]',
|
||||
'Usage: run-managed-agent-server-e2e.ts [--model ID] [--runtime-delay-ms N] [--settings PATH] [--session-failover|--inflight-failover|--continuation-failover|--model-round-failover|--big-output] [--harness-only] [--freeze] [--frozen-control-plane] [--g3-verify allow|deny|cancel|expiry|continuation] [--g3-fault after-decision|after-prepare|after-result|finished-before-commit|sibling-approval|after-earlier-tools|cleanup-failed|decision-reply-lost|admission-reply-lost|load-reply-lost|terminal-reply-lost|terminal-write-failed]',
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -346,6 +350,15 @@ const inflightSideEffectContent = 'MANAGED_INFLIGHT_TOOL_EXECUTED\n';
|
||||
const siblingSideEffectName = 'managed-sibling-side-effect.txt';
|
||||
const siblingSideEffect = path.join(workspaceMount, siblingSideEffectName);
|
||||
const siblingSideEffectContent = 'MANAGED_SIBLING_TOOL_EXECUTED\n';
|
||||
// B3 after-earlier-tools arm: a second, later tool round writes its own file so
|
||||
// each earlier round's single execution has its own physical effect. The +Nk
|
||||
// budget directive makes the TurnBudget snapshot non-null so its restoration is
|
||||
// numerically assertable (the default null budget hides a ledger reset). A round
|
||||
// reset is caught by the exactly-once / deep-equal / round-continuity assertions.
|
||||
const b3SecondSideEffectName = 'managed-b3-second-side-effect.txt';
|
||||
const b3SecondSideEffect = path.join(workspaceMount, b3SecondSideEffectName);
|
||||
const b3SecondSideEffectContent = 'MANAGED_B3_SECOND_TOOL_EXECUTED\n';
|
||||
const b3BudgetDirective = '+500k';
|
||||
const workspaceId = createHash('sha256')
|
||||
.update(workspace)
|
||||
.digest('hex')
|
||||
@@ -1370,6 +1383,136 @@ function siblingFinalPlan(
|
||||
return plan;
|
||||
}
|
||||
|
||||
// B3: read the durable round and cost ledger the way recovery restores them.
|
||||
// The round comes from the latest hosted-model-request (model.attempt.recoveryRef);
|
||||
// the budget snapshot and per-model token metrics are reverse-scanned from the
|
||||
// route/usage resources exactly as bindBudget does on a cold load.
|
||||
interface B3RoundBudget {
|
||||
round: number;
|
||||
budget: unknown;
|
||||
models: unknown;
|
||||
}
|
||||
function b3RoundAndBudget(port: number, sessionFilter: string): B3RoundBudget {
|
||||
const rows = runMysql(
|
||||
port,
|
||||
`SELECT HEX(record_bytes) FROM qwen_managed_agent.ma_session_journal_tx WHERE ${sessionFilter} ORDER BY first_sequence`,
|
||||
).split('\n');
|
||||
const attempts: Array<{
|
||||
sequence: number;
|
||||
payload: Record<string, unknown>;
|
||||
}> = [];
|
||||
for (const row of rows) {
|
||||
if (!row || row === 'NULL') continue;
|
||||
for (const line of Buffer.from(row, 'hex').toString('utf8').split('\n')) {
|
||||
if (!line) continue;
|
||||
const record = JSON.parse(line) as {
|
||||
subtype?: unknown;
|
||||
managedSession?: {
|
||||
kind?: unknown;
|
||||
sequence?: unknown;
|
||||
payload?: Record<string, unknown>;
|
||||
};
|
||||
};
|
||||
if (record.subtype !== 'managed_session_event_v1') continue;
|
||||
const event = record.managedSession;
|
||||
if (
|
||||
!event ||
|
||||
typeof event.sequence !== 'number' ||
|
||||
event.kind !== 'model.attempt'
|
||||
)
|
||||
continue;
|
||||
attempts.push({ sequence: event.sequence, payload: event.payload ?? {} });
|
||||
}
|
||||
}
|
||||
attempts.sort((a, b) => a.sequence - b.sequence);
|
||||
if (attempts.length === 0) throw new Error('B3 window has no model attempt');
|
||||
const readResource = (ref: unknown): Record<string, unknown> | undefined => {
|
||||
const resourceId = (ref as { resourceId?: string } | null)?.resourceId;
|
||||
if (!resourceId) return undefined;
|
||||
// HEX + decode avoids the mysql --batch backslash escaping that would
|
||||
// corrupt the JSON's \" sequences — the saved request carries the full
|
||||
// system prompt, unlike the small approval-continuation plan.
|
||||
const hex = runMysql(
|
||||
port,
|
||||
`SELECT HEX(inline_bytes) FROM qwen_managed_agent.ma_resource WHERE ${sessionFilter} AND resource_id=${sqlString(resourceId)}`,
|
||||
);
|
||||
if (!hex || hex === 'NULL') return undefined;
|
||||
return JSON.parse(Buffer.from(hex, 'hex').toString('utf8')) as Record<
|
||||
string,
|
||||
unknown
|
||||
>;
|
||||
};
|
||||
let round = -1;
|
||||
for (let i = attempts.length - 1; i >= 0; i -= 1) {
|
||||
const req = readResource(attempts[i].payload['recoveryRef']);
|
||||
if (req && typeof req['round'] === 'number') {
|
||||
round = req['round'];
|
||||
break;
|
||||
}
|
||||
}
|
||||
let budget: unknown;
|
||||
let models: unknown;
|
||||
for (let i = attempts.length - 1; i >= 0; i -= 1) {
|
||||
for (const field of ['usageRef', 'routeRef']) {
|
||||
const saved = readResource(attempts[i].payload[field]);
|
||||
if (!saved) continue;
|
||||
if (budget === undefined && 'budget' in saved) budget = saved['budget'];
|
||||
if (models === undefined && saved['models']) models = saved['models'];
|
||||
}
|
||||
if (budget !== undefined && models !== undefined) break;
|
||||
}
|
||||
if (round < 0) throw new Error('B3 window has no saved model-request round');
|
||||
return { round, budget: budget ?? null, models: models ?? null };
|
||||
}
|
||||
|
||||
// B3: the turn cost snapshot and per-model token ledger must survive the
|
||||
// replacement. The snapshot's NUMERIC budget must be present: a non-null
|
||||
// snapshot with budget: null is a legal production state (no +Nk directive
|
||||
// parsed), and an object-truthy guard would let null == null pass vacuously.
|
||||
// The ledger may grow as the held round completes, but every crash-boundary
|
||||
// model's output-token total must survive; an unrestored ledger is a truthy
|
||||
// {}, so a null check alone would let a zeroed restore pass too.
|
||||
function assertB3CostContinuity(
|
||||
before: B3RoundBudget,
|
||||
after: B3RoundBudget,
|
||||
): void {
|
||||
const b3Budget = (before.budget as { budget?: number | null } | null)?.budget;
|
||||
if (b3Budget === null || b3Budget === undefined)
|
||||
throw new Error(
|
||||
`B3 crash-boundary turn budget was not exercised: ${JSON.stringify(before.budget)}`,
|
||||
);
|
||||
if (!isDeepStrictEqual(after.budget, before.budget))
|
||||
throw new Error(
|
||||
`B3 recovery reset the turn budget snapshot: before=${JSON.stringify(before.budget)} after=${JSON.stringify(after.budget)}`,
|
||||
);
|
||||
const b3ModelsBefore = (before.models ?? {}) as Record<
|
||||
string,
|
||||
{ tokens?: { candidates?: number } }
|
||||
>;
|
||||
const b3ModelsAfter = (after.models ?? {}) as Record<
|
||||
string,
|
||||
{ tokens?: { candidates?: number } }
|
||||
>;
|
||||
const b3LedgerEntries = Object.entries(b3ModelsBefore);
|
||||
if (
|
||||
b3LedgerEntries.length === 0 ||
|
||||
b3LedgerEntries.every(
|
||||
([, metric]) => (metric.tokens?.candidates ?? 0) === 0,
|
||||
)
|
||||
)
|
||||
throw new Error(
|
||||
'B3 crash-boundary per-model token ledger is empty: the fixture did not exercise the ledger',
|
||||
);
|
||||
for (const [model, metric] of b3LedgerEntries) {
|
||||
const beforeTokens = metric.tokens?.candidates ?? 0;
|
||||
const afterTokens = b3ModelsAfter[model]?.tokens?.candidates ?? 0;
|
||||
if (afterTokens < beforeTokens)
|
||||
throw new Error(
|
||||
`B3 recovery wiped the per-model token ledger: model=${model} before=${beforeTokens} after=${afterTokens}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function assertPublicAnswer(events: PublicEvent[], expected: string): void {
|
||||
const terminal = events.filter((event) => event.terminal);
|
||||
const text = events
|
||||
@@ -1641,6 +1784,77 @@ try {
|
||||
}
|
||||
|
||||
if (continuationFailover && serialized.includes(continuationMarker)) {
|
||||
// B3: two earlier tool rounds, then a held post-tool round. The fake
|
||||
// model decides by the number of tool results already in the history:
|
||||
// 0 -> tool A (round 0), 1 -> tool B (round 1), >=2 -> the held final
|
||||
// answer (round 2). The crash lands on the round-2 request so recovery
|
||||
// must resume the saved round and cost ledger, not restart them.
|
||||
if (approvalFault === 'after-earlier-tools') {
|
||||
const toolResultCount = (serialized.match(/"role":"tool"/g) ?? [])
|
||||
.length;
|
||||
// Non-zero usage makes the token ledger numerically observable (the
|
||||
// fake's default usage is all zeros). The earlier rounds outweigh the
|
||||
// final round so a zeroed restore totals less than the crash-boundary
|
||||
// ledger, not just the same sum re-counted.
|
||||
if (toolResultCount === 0) {
|
||||
return {
|
||||
toolCalls: [
|
||||
fakeToolCall(
|
||||
'write_file',
|
||||
{
|
||||
file_path: inflightSideEffectName,
|
||||
content: inflightSideEffectContent,
|
||||
},
|
||||
'call_managed_b3_first',
|
||||
),
|
||||
],
|
||||
usage: {
|
||||
prompt_tokens: 0,
|
||||
completion_tokens: 300,
|
||||
total_tokens: 300,
|
||||
},
|
||||
};
|
||||
}
|
||||
if (toolResultCount === 1) {
|
||||
return {
|
||||
toolCalls: [
|
||||
fakeToolCall(
|
||||
'write_file',
|
||||
{
|
||||
file_path: b3SecondSideEffectName,
|
||||
content: b3SecondSideEffectContent,
|
||||
},
|
||||
'call_managed_b3_second',
|
||||
),
|
||||
],
|
||||
usage: {
|
||||
prompt_tokens: 0,
|
||||
completion_tokens: 300,
|
||||
total_tokens: 300,
|
||||
},
|
||||
};
|
||||
}
|
||||
if (approvalResultHold) {
|
||||
return {
|
||||
contentChunks: [continuationPartial],
|
||||
holdAfterChunks: 1,
|
||||
holdUntil: approvalResultHold,
|
||||
usage: {
|
||||
prompt_tokens: 0,
|
||||
completion_tokens: 100,
|
||||
total_tokens: 100,
|
||||
},
|
||||
};
|
||||
}
|
||||
return {
|
||||
content: continuationResponse,
|
||||
usage: {
|
||||
prompt_tokens: 0,
|
||||
completion_tokens: 100,
|
||||
total_tokens: 100,
|
||||
},
|
||||
};
|
||||
}
|
||||
if (!serialized.includes('"role":"tool"')) {
|
||||
// A3: a two-call batch so the replacement can be crashed in the
|
||||
// approval gap between a decided sibling and a pending one.
|
||||
@@ -2041,7 +2255,9 @@ try {
|
||||
{
|
||||
type: 'text',
|
||||
text: continuationFailover
|
||||
? `${continuationMarker}. Execute the requested tool once and reply exactly ${continuationResponse}.`
|
||||
? approvalFault === 'after-earlier-tools'
|
||||
? `${continuationMarker}. ${b3BudgetDirective} Execute the requested tools and reply exactly ${continuationResponse}.`
|
||||
: `${continuationMarker}. Execute the requested tool once and reply exactly ${continuationResponse}.`
|
||||
: inflightFailover
|
||||
? `${inflightMarker}. Execute the requested tool once and reply exactly ${inflightResponse}.`
|
||||
: modelRoundFailover
|
||||
@@ -2822,7 +3038,10 @@ try {
|
||||
policy_revision: originalApprovalOptions!.policyRevision,
|
||||
option_id: verificationMode,
|
||||
});
|
||||
if (approvalFault === 'after-result') {
|
||||
if (
|
||||
approvalFault === 'after-result' ||
|
||||
approvalFault === 'after-earlier-tools'
|
||||
) {
|
||||
approvalResultHold = new Promise<void>((resolve) => {
|
||||
releaseApprovalResultHold = resolve;
|
||||
});
|
||||
@@ -2862,6 +3081,60 @@ try {
|
||||
`Lost decision reply produced ${operations} ACTION_RESPONSE operations, not one`,
|
||||
);
|
||||
}
|
||||
if (approvalFault === 'after-earlier-tools') {
|
||||
// The first Allow drives tool A (round 0); the model then asks approval
|
||||
// for tool B (round 1) under a NEW Action. Detect that requested Action
|
||||
// and allow it so both earlier rounds execute before the held round 2.
|
||||
await waitUntil(
|
||||
'Second earlier-tool approval requested',
|
||||
() => {
|
||||
const rows = runMysql(
|
||||
mysqlPort,
|
||||
`SELECT action_id, state FROM qwen_managed_agent.ma_session_record WHERE ${sessionFilter} AND record_kind='ACTION'`,
|
||||
);
|
||||
const requested = (rows ? rows.split('\n') : [])
|
||||
.map((row) => row.split('\t'))
|
||||
.filter(
|
||||
([id, state]) =>
|
||||
state === 'requested' && id !== originalApprovalId,
|
||||
);
|
||||
if (requested.length === 1) {
|
||||
secondApprovalId = requested[0][0];
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
},
|
||||
120_000,
|
||||
replacementHarness,
|
||||
);
|
||||
const secondOptions = JSON.parse(
|
||||
runMysql(
|
||||
mysqlPort,
|
||||
`SELECT options_json FROM qwen_managed_agent.ma_session_record WHERE ${sessionFilter} AND record_kind='ACTION' AND action_id=${sqlString(secondApprovalId!)}`,
|
||||
),
|
||||
);
|
||||
const secondResponse = await fetch(
|
||||
`${replacementSpringUrl}/v1/agents/sessions/${session.id}/actions/${secondApprovalId}/responses`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'content-type': 'application/json',
|
||||
'idempotency-key': `g3-answer-${secondApprovalId}`,
|
||||
...tenantHeaders(tenant),
|
||||
},
|
||||
body: JSON.stringify({
|
||||
kind: 'permission',
|
||||
input_revision: secondOptions.inputRevision,
|
||||
policy_revision: secondOptions.policyRevision,
|
||||
option_id: 'allow',
|
||||
}),
|
||||
},
|
||||
);
|
||||
if (secondResponse.status !== 202)
|
||||
throw new Error(
|
||||
`Second earlier-tool response returned ${secondResponse.status}: ${await secondResponse.text()}`,
|
||||
);
|
||||
}
|
||||
} else if (verificationMode === 'cancel') {
|
||||
const response = await fetch(
|
||||
`${replacementSpringUrl}/v1/agents/sessions/${session.id}/events`,
|
||||
@@ -2885,12 +3158,14 @@ try {
|
||||
}
|
||||
let publishedPrefixSequences: number[] = [];
|
||||
let terminalAcceptsAtBarrier = 0;
|
||||
let b3Before: B3RoundBudget | undefined;
|
||||
if (
|
||||
approvalFault === 'after-decision' ||
|
||||
approvalFault === 'after-prepare' ||
|
||||
approvalFault === 'after-result' ||
|
||||
approvalFault === 'finished-before-commit' ||
|
||||
approvalFault === 'sibling-approval' ||
|
||||
approvalFault === 'after-earlier-tools' ||
|
||||
approvalFault === 'cleanup-failed' ||
|
||||
approvalFault === 'terminal-reply-lost' ||
|
||||
approvalFault === 'terminal-write-failed'
|
||||
@@ -2952,6 +3227,45 @@ try {
|
||||
120_000,
|
||||
replacementHarness,
|
||||
);
|
||||
} else if (approvalFault === 'after-earlier-tools') {
|
||||
// Both earlier tool rounds must have committed their results and the
|
||||
// round-2 post-tool request must be held with its prefix published, so
|
||||
// the crash lands after the earlier tools but before the final output
|
||||
// commits. Recovery must then resume the saved round and cost ledger.
|
||||
await waitUntil(
|
||||
'Earlier tools settled and the later round held',
|
||||
async () => {
|
||||
const rows = runMysql(
|
||||
mysqlPort,
|
||||
'SELECT execution_state, IF(result_json IS NULL, 0, 1) FROM qwen_managed_agent.ma_tool_execution ORDER BY execution_call_id',
|
||||
);
|
||||
const execs = rows ? rows.split('\n') : [];
|
||||
if (execs.length !== 2 || execs.some((r) => r !== 'SETTLED\t1'))
|
||||
return false;
|
||||
const heldLaterRequest = (fake?.requests ?? []).some(
|
||||
({ body }) => {
|
||||
const messages = JSON.stringify(body['messages']);
|
||||
return (
|
||||
messages.includes(continuationMarker) &&
|
||||
(messages.match(/"role":"tool"/g) ?? []).length >= 2
|
||||
);
|
||||
},
|
||||
);
|
||||
if (!heldLaterRequest) return false;
|
||||
const view = await publicTranscript(
|
||||
replacementSpringUrl,
|
||||
tenant,
|
||||
session.id,
|
||||
);
|
||||
return view.text.includes(continuationPartial);
|
||||
},
|
||||
120_000,
|
||||
replacementHarness,
|
||||
);
|
||||
// Capture the durable round and cost ledger at the crash boundary so
|
||||
// recovery can be proven to resume them rather than restart them.
|
||||
b3Before = b3RoundAndBudget(mysqlPort, sessionFilter);
|
||||
saveEvidence('b3-round-budget-before.json', b3Before);
|
||||
} else if (
|
||||
approvalFault === 'terminal-reply-lost' ||
|
||||
approvalFault === 'terminal-write-failed'
|
||||
@@ -3128,7 +3442,8 @@ try {
|
||||
)
|
||||
: undefined;
|
||||
const publishedPrefix =
|
||||
approvalFault === 'after-result'
|
||||
approvalFault === 'after-result' ||
|
||||
approvalFault === 'after-earlier-tools'
|
||||
? await publicTranscript(replacementSpringUrl, tenant, session.id)
|
||||
: undefined;
|
||||
publishedPrefixSequences = (publishedPrefix?.events ?? [])
|
||||
@@ -3160,7 +3475,8 @@ try {
|
||||
}
|
||||
: undefined,
|
||||
publishedPrefix:
|
||||
approvalFault === 'after-result'
|
||||
approvalFault === 'after-result' ||
|
||||
approvalFault === 'after-earlier-tools'
|
||||
? {
|
||||
sequences: publishedPrefixSequences,
|
||||
text: publishedPrefix?.text,
|
||||
@@ -3171,7 +3487,10 @@ try {
|
||||
replacementHarness.child,
|
||||
'Replacement Hosted Harness (fault barrier)',
|
||||
);
|
||||
if (approvalFault === 'after-result') {
|
||||
if (
|
||||
approvalFault === 'after-result' ||
|
||||
approvalFault === 'after-earlier-tools'
|
||||
) {
|
||||
releaseApprovalResultHold();
|
||||
approvalResultHold = undefined;
|
||||
}
|
||||
@@ -3328,7 +3647,11 @@ try {
|
||||
// a compensation that only projects a terminal would pass. The
|
||||
// result-commit arm already owns the same check with its retraction
|
||||
// diagnostic, so it keeps that message.
|
||||
if (verificationMode === 'allow' && approvalFault !== 'after-result') {
|
||||
if (
|
||||
verificationMode === 'allow' &&
|
||||
approvalFault !== 'after-result' &&
|
||||
approvalFault !== 'after-earlier-tools'
|
||||
) {
|
||||
const transcript = await publicTranscript(
|
||||
replacementSpringUrl,
|
||||
tenant,
|
||||
@@ -3339,7 +3662,10 @@ try {
|
||||
`Approval recovery answer differs: ${JSON.stringify(transcript.text)}`,
|
||||
);
|
||||
}
|
||||
if (approvalFault === 'after-result') {
|
||||
if (
|
||||
approvalFault === 'after-result' ||
|
||||
approvalFault === 'after-earlier-tools'
|
||||
) {
|
||||
// The reissue must retract the held prefix: only the recovered
|
||||
// answer may remain in the public transcript.
|
||||
const transcript = await publicTranscript(
|
||||
@@ -3374,7 +3700,8 @@ try {
|
||||
: [];
|
||||
const expectedExecutions =
|
||||
verificationMode === 'allow'
|
||||
? approvalFault === 'sibling-approval'
|
||||
? approvalFault === 'sibling-approval' ||
|
||||
approvalFault === 'after-earlier-tools'
|
||||
? 2
|
||||
: 1
|
||||
: 0;
|
||||
@@ -3385,8 +3712,26 @@ try {
|
||||
// Every approved call keeps its own original prepare key/digest and the
|
||||
// original Runtime identity, settled once. Match by prepare key so the
|
||||
// model's call order is checked, not the SQL row order; the A3 arm has
|
||||
// two siblings, every other allow arm has one.
|
||||
if (verificationMode === 'allow') {
|
||||
// two siblings and B3 two earlier rounds, every other allow arm has one.
|
||||
if (approvalFault === 'after-earlier-tools') {
|
||||
// B3's two earlier tools live in separate rounds, so the first approval
|
||||
// plan carries only tool A. Assert both executions' Runtime identity and
|
||||
// settled-once directly; the two physical file effects give per-tool
|
||||
// evidence, and a re-run of either earlier tool reds the count above.
|
||||
for (const [ordinal, execution] of executions.entries()) {
|
||||
if (
|
||||
execution[2] !== originalApprovalPlan!.runtime.bindingId ||
|
||||
execution[3] !== originalApprovalPlan!.runtime.generation ||
|
||||
execution[4] !== originalApprovalPlan!.runtime.runtimeSessionId ||
|
||||
execution[5] !== 'SETTLED' ||
|
||||
execution[6] !== '1'
|
||||
) {
|
||||
throw new Error(
|
||||
`B3 earlier execution ${ordinal} changed its identity or re-ran: ${execution.join(',')}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
} else if (verificationMode === 'allow') {
|
||||
const plan = originalApprovalPlan!;
|
||||
const settledAtByOrdinal: number[] = [];
|
||||
for (const [ordinal, call] of plan.calls.entries()) {
|
||||
@@ -3448,6 +3793,26 @@ try {
|
||||
`Sibling physical file assertion failed: ${JSON.stringify(siblingBytes)}`,
|
||||
);
|
||||
}
|
||||
if (approvalFault === 'after-earlier-tools') {
|
||||
const b3Bytes = existsSync(b3SecondSideEffect)
|
||||
? readFileSync(b3SecondSideEffect, 'utf8')
|
||||
: '';
|
||||
if (b3Bytes !== b3SecondSideEffectContent)
|
||||
throw new Error(
|
||||
`B3 second physical file assertion failed: ${JSON.stringify(b3Bytes)}`,
|
||||
);
|
||||
// Round + cost ledger continuity: recovery must resume the saved round
|
||||
// and restore the budget snapshot / token ledger, not restart them.
|
||||
if (!b3Before)
|
||||
throw new Error('B3 crash-boundary round/budget was not captured');
|
||||
const b3After = b3RoundAndBudget(mysqlPort, sessionFilter);
|
||||
saveEvidence('b3-round-budget-after.json', b3After);
|
||||
if (b3After.round !== b3Before.round || b3After.round < 2)
|
||||
throw new Error(
|
||||
`B3 recovery reset the logical round: before=${b3Before.round} after=${b3After.round}`,
|
||||
);
|
||||
assertB3CostContinuity(b3Before, b3After);
|
||||
}
|
||||
const actionRows = runMysql(
|
||||
mysqlPort,
|
||||
`SELECT action_id, state FROM qwen_managed_agent.ma_session_record WHERE ${sessionFilter} AND record_kind='ACTION' ORDER BY action_id`,
|
||||
@@ -3469,6 +3834,18 @@ try {
|
||||
throw new Error(
|
||||
`Sibling approval identities/states differ: ${actionRows}`,
|
||||
);
|
||||
} else if (approvalFault === 'after-earlier-tools') {
|
||||
// Both earlier tools end decided under their ORIGINAL Action IDs: A by
|
||||
// the first Allow, B by the runner's second Allow. Recovery must resume
|
||||
// the saved model round, not re-ask either approval.
|
||||
const expectedB3Actions = [
|
||||
`${originalApprovalId}\t${expectedActionState}`,
|
||||
`${secondApprovalId}\t${expectedActionState}`,
|
||||
].sort();
|
||||
if (actionRows !== expectedB3Actions.join('\n'))
|
||||
throw new Error(
|
||||
`B3 approval identities/states differ: ${actionRows}`,
|
||||
);
|
||||
} else if (
|
||||
actionRows !== `${originalApprovalId}\t${expectedActionState}`
|
||||
) {
|
||||
@@ -3587,12 +3964,14 @@ try {
|
||||
JSON.stringify(request.body.messages).includes('"role":"tool"'),
|
||||
);
|
||||
const expectedModelsAfterResults =
|
||||
approvalFault === 'after-result' ||
|
||||
approvalFault === 'finished-before-commit'
|
||||
? 2
|
||||
: ['allow', 'deny'].includes(verificationMode)
|
||||
? 1
|
||||
: 0;
|
||||
approvalFault === 'after-earlier-tools'
|
||||
? 3
|
||||
: approvalFault === 'after-result' ||
|
||||
approvalFault === 'finished-before-commit'
|
||||
? 2
|
||||
: ['allow', 'deny'].includes(verificationMode)
|
||||
? 1
|
||||
: 0;
|
||||
if (
|
||||
initialRequests.length !== 1 ||
|
||||
modelAfterResults.length !== expectedModelsAfterResults
|
||||
@@ -3617,6 +3996,22 @@ try {
|
||||
'Result-commit recovery changed the saved model request',
|
||||
);
|
||||
}
|
||||
if (approvalFault === 'after-earlier-tools') {
|
||||
// The held round-2 request (index 1) must be reissued deep-equal by the
|
||||
// recovery generation (index 2); index 0 is the earlier round-1 request.
|
||||
const parity = isDeepStrictEqual(
|
||||
modelAfterResults[1]?.body,
|
||||
modelAfterResults[2]?.body,
|
||||
);
|
||||
saveEvidence('model-request-parity.json', {
|
||||
requests: modelAfterResults.map((request) => request.body),
|
||||
equal: parity,
|
||||
});
|
||||
if (!parity)
|
||||
throw new Error(
|
||||
'B3 recovery changed the saved later-round model request',
|
||||
);
|
||||
}
|
||||
if (responseOperation) {
|
||||
await waitUntil(
|
||||
'Approval response command completion',
|
||||
|
||||
@@ -144,6 +144,18 @@ const failoverArms = [
|
||||
'sibling-approval',
|
||||
],
|
||||
],
|
||||
[
|
||||
'Run after-earlier-tools crash E2E',
|
||||
'test:e2e:managed-harness-restart-approval-after-earlier-tools',
|
||||
[
|
||||
'--continuation-failover',
|
||||
'--harness-only',
|
||||
'--g3-verify',
|
||||
'allow',
|
||||
'--g3-fault',
|
||||
'after-earlier-tools',
|
||||
],
|
||||
],
|
||||
[
|
||||
'Run approval decision reply-loss E2E',
|
||||
'test:e2e:managed-harness-restart-approval-decision-reply-lost',
|
||||
@@ -652,8 +664,8 @@ describe('Hosted real-process gates', () => {
|
||||
// Step ceilings today (201 + 20 + 50 + 20): Verify, the Broker fault
|
||||
// gates, admission and O4. The failover arms moved to their own job.
|
||||
['hosted-harness-mysql', 291, 304],
|
||||
// Twenty-six failover arms at 10 each.
|
||||
['hosted-failover-e2e', 260, 275],
|
||||
// Twenty-seven failover arms at 10 each.
|
||||
['hosted-failover-e2e', 270, 285],
|
||||
])(
|
||||
'keeps the %s job ceiling above its summed step ceilings',
|
||||
(name, steps, ceiling) => {
|
||||
|
||||
@@ -15,7 +15,7 @@ import {
|
||||
} from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { inspect } from 'node:util';
|
||||
import { inspect, isDeepStrictEqual } from 'node:util';
|
||||
import {
|
||||
createSourceFile,
|
||||
isArrayLiteralExpression,
|
||||
@@ -594,7 +594,7 @@ describe('managed-agent-server e2e runner', () => {
|
||||
// pinned separately and narrowing either one (2 -> 1) reds here at unit
|
||||
// speed, not only at the real-stack barrier.
|
||||
expect(source).toMatch(
|
||||
/const expectedExecutions =[\s\S]*?approvalFault === 'sibling-approval'\s*\?\s*2\s*:\s*1/,
|
||||
/const expectedExecutions =[\s\S]*?approvalFault === 'sibling-approval' \|\|\s*approvalFault === 'after-earlier-tools'\s*\?\s*2\s*:\s*1/,
|
||||
);
|
||||
// `\s*`, not `[\s\S]*?`: the gap here is whitespace-only, so a tight anchor
|
||||
// cannot span past this site to the expectedExecutions ternary and mask a
|
||||
@@ -648,6 +648,73 @@ describe('managed-agent-server e2e runner', () => {
|
||||
expect(a3PlanFnBody).toContain("event.kind !== 'hosted.batch.planned'");
|
||||
expect(a3PlanFnBody).toContain("payload['batchId'] !== batchId");
|
||||
expect(a3PlanFnBody).toContain("plan.stage !== 'final'");
|
||||
// B3 after-earlier-tools: the multi-round fixture, the second-Allow
|
||||
// choreography, the round/budget continuity readers + assertions, and the
|
||||
// later-round deep-equal parity are all pinned so a regression reds here.
|
||||
expect(source).toContain("'after-earlier-tools'");
|
||||
expect(source).toContain('call_managed_b3_second');
|
||||
expect(source).toContain('b3BudgetDirective');
|
||||
expect(source).toContain('Second earlier-tool approval requested');
|
||||
expect(source).toContain('Earlier tools settled and the later round held');
|
||||
expect(source).toContain('b3RoundAndBudget(');
|
||||
expect(source).toContain('B3 recovery reset the logical round');
|
||||
expect(source).toContain('B3 recovery reset the turn budget snapshot');
|
||||
expect(source).toContain('B3 recovery wiped the per-model token ledger');
|
||||
expect(source).toContain('B3 second physical file assertion failed');
|
||||
expect(source).toContain(
|
||||
'B3 recovery changed the saved later-round model request',
|
||||
);
|
||||
expect(source).toContain('B3 earlier execution');
|
||||
expect(source).toContain('B3 approval identities/states differ');
|
||||
// The hold armed for B3 must also be RELEASED after the crash: a release
|
||||
// gated to after-result alone leaves the recovery generation's reissued
|
||||
// request held forever (the .75 head run crash-looped on exactly that).
|
||||
const releaseAt = source.indexOf('releaseApprovalResultHold();');
|
||||
expect(releaseAt).toBeGreaterThan(-1);
|
||||
expect(source.slice(releaseAt - 200, releaseAt)).toContain(
|
||||
"approvalFault === 'after-earlier-tools'",
|
||||
);
|
||||
// B3 shares after-result's published-prefix capture and retraction proof.
|
||||
expect(source).toMatch(
|
||||
/approvalFault === 'after-result' \|\|\s*approvalFault === 'after-earlier-tools'\s*\) \{\s*\/\/ The reissue must retract the held prefix/,
|
||||
);
|
||||
// The fixture must charge non-zero usage (the fake default is all zeros,
|
||||
// which makes the ledger continuity assertion vacuous) and the ledger
|
||||
// check must compare per-model totals — an empty {} ledger is truthy.
|
||||
expect(source).toContain('completion_tokens: 300');
|
||||
expect(source).toContain(
|
||||
'B3 crash-boundary per-model token ledger is empty',
|
||||
);
|
||||
expect(source).toContain('B3 crash-boundary turn budget was not exercised');
|
||||
expect(source).toContain('afterTokens < beforeTokens');
|
||||
expect(source).toContain('assertB3CostContinuity(');
|
||||
// R1-3: the budget guard must test the NUMERIC field — a non-null snapshot
|
||||
// with budget: null is a legal production state, so an object-truthy guard
|
||||
// would let a lost +Nk directive pass as null == null.
|
||||
const b3CostFn = source.indexOf('function assertB3CostContinuity(');
|
||||
expect(b3CostFn).toBeGreaterThan(-1);
|
||||
const b3CostFnBody = source.slice(
|
||||
b3CostFn,
|
||||
source.indexOf('\n}', b3CostFn),
|
||||
);
|
||||
expect(b3CostFnBody).toContain(
|
||||
'b3Budget === null || b3Budget === undefined',
|
||||
);
|
||||
// R1-4: site-anchor the B3 post-tool request count like the neighbouring
|
||||
// R1-7 / F1 pins — the identifier alone survives a conflict resolution
|
||||
// back to the pre-PR ternary, which would only red in the real-stack lane.
|
||||
// Deleting the after-earlier-tools ? 3 clause must red here.
|
||||
expect(source).toMatch(
|
||||
/const expectedModelsAfterResults =[\s\S]*?approvalFault === 'after-earlier-tools'\s*\?\s*3\s*:/,
|
||||
);
|
||||
const b3Fn = source.indexOf('function b3RoundAndBudget(');
|
||||
expect(b3Fn).toBeGreaterThan(-1);
|
||||
const b3FnBody = source.slice(b3Fn, source.indexOf('\n}', b3Fn));
|
||||
expect(b3FnBody).toContain('qwen_managed_agent.ma_session_journal_tx');
|
||||
expect(b3FnBody).toContain("event.kind !== 'model.attempt'");
|
||||
expect(b3FnBody).toContain('SELECT HEX(inline_bytes)');
|
||||
expect(b3FnBody).toContain("attempts[i].payload['recoveryRef']");
|
||||
expect(b3FnBody).toContain("['usageRef', 'routeRef']");
|
||||
});
|
||||
|
||||
// R4-2 (behavioural): the source pins above assert the window helper's
|
||||
@@ -838,6 +905,191 @@ describe('managed-agent-server e2e runner', () => {
|
||||
).toEqual(finalPlan);
|
||||
});
|
||||
|
||||
// B3 (behavioural): exercise b3RoundAndBudget on injected journal rows and
|
||||
// request/route/usage resources, so the latest-attempt round selection, the
|
||||
// reverse-scan for budget/models, and both throw branches are proven on data.
|
||||
// runMysql is injected through load() and dispatches on the query (journal
|
||||
// HEX rows vs HEX resource bytes keyed by resource_id); sqlString is
|
||||
// extracted too.
|
||||
it('behaves b3RoundAndBudget on injected journal rows', () => {
|
||||
const attempt = (sequence, refs) => ({
|
||||
kind: 'model.attempt',
|
||||
sequence,
|
||||
payload: refs,
|
||||
});
|
||||
const journalRows = (events) =>
|
||||
Buffer.from(
|
||||
events
|
||||
.map((event) =>
|
||||
JSON.stringify({
|
||||
subtype: 'managed_session_event_v1',
|
||||
managedSession: event,
|
||||
}),
|
||||
)
|
||||
.join('\n'),
|
||||
'utf8',
|
||||
).toString('hex');
|
||||
const budgetSnapshot = { budget: 500000, outputTokensAtTurnStart: 100 };
|
||||
const resources = {
|
||||
// The saved request carries the full system prompt, so its JSON has \"
|
||||
// and \\ sequences — the shape mysql --batch escaping corrupted when the
|
||||
// helper read CONVERT(inline_bytes) instead of HEX.
|
||||
req_round2: { round: 2, prompt: 'reply "exactly" then \\ stop' },
|
||||
req_round0: { round: 0 },
|
||||
usage1: { budget: budgetSnapshot, models: { m: { output: 150 } } },
|
||||
route2: { budget: budgetSnapshot },
|
||||
// R1-2: a discriminating pair — the older attempt carries a DIFFERENT
|
||||
// budget and a smaller ledger, the newest its own. An oldest-first scan
|
||||
// returns 900000/300 where the newest-first contract returns 500000/600,
|
||||
// so the direction is proven on data, not only on the round selection.
|
||||
usageOld: {
|
||||
budget: { budget: 900000, outputTokensAtTurnStart: 0 },
|
||||
models: { m: { tokens: { candidates: 300 } } },
|
||||
},
|
||||
usageNew: {
|
||||
budget: budgetSnapshot,
|
||||
models: { m: { tokens: { candidates: 600 } } },
|
||||
},
|
||||
};
|
||||
const run = (events) =>
|
||||
load(
|
||||
['b3RoundAndBudget', 'sqlString'],
|
||||
'b3RoundAndBudget',
|
||||
'runMysql',
|
||||
)((_port, query) => {
|
||||
if (query.includes('ma_session_journal_tx')) return journalRows(events);
|
||||
if (query.includes('ma_resource')) {
|
||||
const resourceId = query.match(/resource_id='([^']+)'/)?.[1];
|
||||
const res = resources[resourceId];
|
||||
if (!res) throw new Error(`no resource ${String(resourceId)}`);
|
||||
return Buffer.from(JSON.stringify(res), 'utf8').toString('hex');
|
||||
}
|
||||
throw new Error(`unexpected query: ${query}`);
|
||||
}).b3RoundAndBudget(0, 'filter');
|
||||
|
||||
// The latest attempt's request gives the round; budget/models reverse-scan
|
||||
// across the route/usage refs (budget from the newest, models from usage1).
|
||||
expect(
|
||||
run([
|
||||
attempt(10, {
|
||||
recoveryRef: { resourceId: 'req_round0' },
|
||||
usageRef: { resourceId: 'usage1' },
|
||||
}),
|
||||
attempt(20, {
|
||||
recoveryRef: { resourceId: 'req_round2' },
|
||||
routeRef: { resourceId: 'route2' },
|
||||
}),
|
||||
]),
|
||||
).toEqual({
|
||||
round: 2,
|
||||
budget: budgetSnapshot,
|
||||
models: { m: { output: 150 } },
|
||||
});
|
||||
// Models still resolve from an earlier attempt when the newest has no usage.
|
||||
expect(
|
||||
run([
|
||||
attempt(10, {
|
||||
recoveryRef: { resourceId: 'req_round0' },
|
||||
usageRef: { resourceId: 'usage1' },
|
||||
}),
|
||||
attempt(20, { recoveryRef: { resourceId: 'req_round2' } }),
|
||||
]),
|
||||
).toEqual({
|
||||
round: 2,
|
||||
budget: budgetSnapshot,
|
||||
models: { m: { output: 150 } },
|
||||
});
|
||||
// R1-2: the newest attempt's own usage wins over an older, different one —
|
||||
// an oldest-first budget/models scan reds here (900000/300 vs 500000/600).
|
||||
expect(
|
||||
run([
|
||||
attempt(10, {
|
||||
recoveryRef: { resourceId: 'req_round0' },
|
||||
usageRef: { resourceId: 'usageOld' },
|
||||
}),
|
||||
attempt(20, {
|
||||
recoveryRef: { resourceId: 'req_round2' },
|
||||
usageRef: { resourceId: 'usageNew' },
|
||||
}),
|
||||
]),
|
||||
).toEqual({
|
||||
round: 2,
|
||||
budget: budgetSnapshot,
|
||||
models: { m: { tokens: { candidates: 600 } } },
|
||||
});
|
||||
// No attempt at all.
|
||||
expect(() => run([])).toThrow(/no model attempt/);
|
||||
// An attempt with no saved request round.
|
||||
expect(() => run([attempt(10, {})])).toThrow(
|
||||
/no saved model-request round/,
|
||||
);
|
||||
});
|
||||
|
||||
// R1-2 / R1-3 (behavioural): the cost-continuity assertion on injected
|
||||
// before/after states. isDeepStrictEqual is injected through load() because
|
||||
// the extracted body's free identifiers resolve against the harness scope.
|
||||
it('behaves assertB3CostContinuity on injected before/after states', () => {
|
||||
const assertCost = load(
|
||||
['assertB3CostContinuity'],
|
||||
'assertB3CostContinuity',
|
||||
'isDeepStrictEqual',
|
||||
)(isDeepStrictEqual).assertB3CostContinuity;
|
||||
const snapshot = {
|
||||
sessionId: 's',
|
||||
promptId: 'p',
|
||||
budget: 500000,
|
||||
directiveText: '+500k',
|
||||
outputTokensAtTurnStart: 0,
|
||||
};
|
||||
const ledger = (candidates) => ({
|
||||
'fake-model': { tokens: { candidates } },
|
||||
});
|
||||
const before = { round: 2, budget: snapshot, models: ledger(600) };
|
||||
// The healthy recovery: snapshot deep-equal, ledger grown by the final round.
|
||||
expect(() =>
|
||||
assertCost(before, { round: 2, budget: snapshot, models: ledger(700) }),
|
||||
).not.toThrow();
|
||||
// R1-3: a non-null snapshot whose numeric budget is null (the +Nk directive
|
||||
// stopped being parsed) must red — an object-truthy guard passes it and the
|
||||
// deep-equal then compares null with null.
|
||||
expect(() =>
|
||||
assertCost(
|
||||
{ ...before, budget: { ...snapshot, budget: null } },
|
||||
{
|
||||
round: 2,
|
||||
budget: { ...snapshot, budget: null },
|
||||
models: ledger(700),
|
||||
},
|
||||
),
|
||||
).toThrow(/turn budget was not exercised/);
|
||||
// A missing snapshot reds the same guard.
|
||||
expect(() =>
|
||||
assertCost({ ...before, budget: null }, { ...before, budget: null }),
|
||||
).toThrow(/turn budget was not exercised/);
|
||||
// A re-baselined snapshot (different turn-start baseline) reds deep-equal.
|
||||
expect(() =>
|
||||
assertCost(before, {
|
||||
round: 2,
|
||||
budget: { ...snapshot, outputTokensAtTurnStart: 600 },
|
||||
models: ledger(700),
|
||||
}),
|
||||
).toThrow(/reset the turn budget snapshot/);
|
||||
// An empty crash-boundary ledger is vacuous and must red.
|
||||
expect(() =>
|
||||
assertCost({ ...before, models: {} }, { ...before, models: {} }),
|
||||
).toThrow(/token ledger is empty/);
|
||||
// A zeroed ({}) after-ledger must red — {} is truthy, so a null check passes it.
|
||||
expect(() =>
|
||||
assertCost(before, { round: 2, budget: snapshot, models: {} }),
|
||||
).toThrow(/wiped the per-model token ledger/);
|
||||
// A shrunk per-model total must red (the dropped-restore shape).
|
||||
expect(() =>
|
||||
assertCost(before, { round: 2, budget: snapshot, models: ledger(100) }),
|
||||
).toThrow(
|
||||
/wiped the per-model token ledger: model=fake-model before=600 after=100/,
|
||||
);
|
||||
});
|
||||
|
||||
// #12941: the Stage A acceptance criterion names a 15-second Runtime delay,
|
||||
// but the ordering assertion was gated at 20 s, so a --runtime-delay-ms 15000
|
||||
// run silently skipped it. Pin the threshold to the criterion's delay and
|
||||
|
||||
Reference in new issue
Block a user