test(managed-agent): gate the A4 takeover-load reply-loss handoff (#15857)

* test(managed-agent): gate the A4 takeover-load reply-loss handoff

The A4 load half was recorded as a recovery gap (a lost takeover-load reply
leaving the original Turn stuck RUNNING). On the current tree that variant does
not reproduce: when the replacement loses the reply to its first drive load and
the control plane redrives the same load into the now-active driver, the redrive
restates the active attachment and the original Turn projects exactly one
terminal with the recovered answer. Two clean real-stack runs (Linux aarch64,
MySQL 8.0.46, JDK 21) recover correctly with the fault injected.

Rather than leave the row focused, package it as a regression gate so the
handoff cannot silently regress:

- A test-gated Harness hook (QWEN_E2E_DROP_LOAD_REPLY=<sessionId>, set only on
  the replacement by the arm, never in production) destroys the first load
  attach reply once, after the driver started.
- The arm asserts the reply was actually lost (via the replacement's captured
  log) before trusting the recovery, then requires one terminal, the recovered
  answer, one settled execution and one file effect.
- Refusing the redrive's active-branch restatement makes the arm fail at the
  adoption barrier, proving the gate is load-bearing.

The failover job grows to 24 arms (ceiling 255). The bilingual Step 3 design and
the server README flip the A4 load row from a focused gap to packaged and note
that the earlier stuck-RUNNING variant did not reproduce on the current tree.

* test(cli): document the A4 load-reply fault flag in the serve env guard

The packaged load-reply-lost arm reads QWEN_E2E_DROP_LOAD_REPLY once at session-bridge construction in hosted-harness-session.ts. The serve process.env guard requires every access to be a documented process-scoped exception, so register the test-gated flag with its reason. Fixes the Qwen Code CI env-guard red on the merged head.

.github/workflows/sdk-java.yml

@@ -1373,9 +1373,9 @@ jobs:
     # that a failing integration test there no longer skips them: they need
     # only the bundled CLI, the server jar and a runner-private mysqld, not
     # its MySQL service or cgroup root. Each arm also runs when an earlier
-    # one failed. Ceilings: 23 arms × 10 = 230 plus ~13 minutes for the
-    # uncapped setup steps (Checkout.. Install MySQL binaries): 243 + 2.
-    timeout-minutes: 245
+    # one failed. Ceilings: 24 arms × 10 = 240 plus ~13 minutes for the
+    # uncapped setup steps (Checkout.. Install MySQL binaries): 253 + 2.
+    timeout-minutes: 255
     env:
       QWEN_E2E_REGISTER_STORAGE: '1'
     steps:
@@ -1524,6 +1524,11 @@ jobs:
         timeout-minutes: 10
         run: >-
           npm run test:e2e:managed-harness-restart-approval-admission-reply-lost
+      - name: 'Run takeover load reply-loss E2E'
+        if: "${{ !cancelled() && steps.failover_prerequisites.outcome == 'success' }}"
+        timeout-minutes: 10
+        run: >-
+          npm run test:e2e:managed-harness-restart-approval-load-reply-lost
       - name: 'Run committed terminal reply-loss E2E'
         if: "${{ !cancelled() && steps.failover_prerequisites.outcome == 'success' }}"
         timeout-minutes: 10

docs/design/2026-10-09-hosted-g3-step3-recovery.md

@@ -510,7 +510,7 @@ Integration regressions proved that model request preparation failures preserve
 
 The machine-readable reports, manifests and retained failed diagnostics are under `.qwen/investigations/g3-step3-implementation/`; the current consolidated report is `packaged-verification.md`. These five packaged gates do not prove the whole matrix above. Since [replaceable-Harness D11](2026-10-02-hosted-replaceable-harness.md#d11--residual-gates-step-3-in-ci-the-model-round-arm-and-q2s-binding-half), they are CI steps of the `hosted-failover-e2e` job, which runs apart from the Hosted MySQL verify (see D11), next to a model-round arm that packages the main observations of B1 for a no-tool Session and of B2, though not their attempt and stream identities, the live SSE view or the admission count.
 
-#14481's packaged gates add ten fault arms to the same job. Seven landed with
+#14481's packaged gates add eleven fault arms to the same job. Seven landed with
 the second step. Five layer one fault on the Allow path: three crash the
 replacement Harness after a
 durable A5 barrier — once the Allow decision is recorded (A5's decision row),
@@ -523,7 +523,7 @@ admission must retry to one Turn row (C4). All five leave the original Action,
 prepare key, digest and Runtime identity in place, start a third generation,
 and require exactly one settled execution, one file effect and one terminal;
 the result-commit arm additionally reuses the saved request and retracts the
-published prefix. Three more cover the next barriers: a committed
+published prefix. Four more cover the next barriers: a committed
 `turn.settled` whose reply is lost must be projected once after replacement
 with zero inference or tool calls (no second store commit is required), and
 the surviving public text must still be the recovered answer (C1); and a
@@ -533,10 +533,13 @@ exactly once by the replacement, with the recovered answer still visible
 too: once the Turn is terminal and its generation is dead, the control plane
 re-drives the owed cleanup through a cleanup-only load until the replacement
 confirms it ([cleanup debt retry](2026-10-28-hosted-cleanup-debt-retry.md)).
-A lost takeover-load reply (A4's load half) stays focused: the packaged arm
-showed the redrive's active handoff leaves the original Turn RUNNING instead
-of projecting its terminal, so fixing that recovery path is separate from
-this acceptance slice. Malformed or foreign refs, snapshot overflow, mixed
+A lost takeover-load reply (A4's load half) is packaged as well: the replacement
+loses the reply to its first drive load once (a test-gated hook), the control
+plane redrives the same load into the now-active driver, and the original Turn
+must still project exactly one terminal with the recovered answer — the redrive
+restates the active attachment rather than wedging the Turn. An earlier observed
+stuck-RUNNING variant did not reproduce on the current tree, so the arm gates the
+handoff as a regression rather than marking an open gap. Malformed or foreign refs, snapshot overflow, mixed
 calls and rewritten inputs across a replacement (A3), Hook/MCP/child
 uncertainty (A6), the attempt and stream identities plus the live SSE view
 (B2), the round and cost budgets (B3), the crash between stream Finished and

docs/design/2026-10-09-hosted-g3-step3-recovery.zh-CN.md

@@ -416,7 +416,7 @@ Allow 保留原 Action、prepare key、request digest 和 Runtime binding，产
 
 机器可读报告、manifest 和保留的失败诊断位于 `.qwen/investigations/g3-step3-implementation/`，当前汇总报告为 `packaged-verification.md`。这五种打包门禁不证明上述整个矩阵。自[可替换 Harness 设计 D11](2026-10-02-hosted-replaceable-harness.zh-CN.md#d11--残留门禁step-3-进-ci模型轮支路q2-的-binding-半边) 起，它们是 `hosted-failover-e2e` 任务的 CI 步骤，该任务独立于 Hosted MySQL 的 verify 运行（见 D11）；同一任务里还有一条模型轮支路，把无工具 Session 的 B1 与 B2 的主要观察项做成打包门禁，但不覆盖它们的 attempt 与 stream 身份、实时 SSE 视图以及准入次数。
 
-#14481 的打包门禁在同一任务里新增十条故障支路，其中七条随第 2 步落地。五条在 Allow 路径上叠加一层故障：三条在持久 A5 屏障之后崩溃替代 Harness——一条在 Allow 决策落库后（A5 的决策行），一条在批准执行到达 `PREPARED`、Runtime Broker 的 `:start` 被扣住时（A5a 的部分 prepare），还有一条在批准结果已提交、post-tool 模型轮被扣住时（A5 的 result commit 屏障）——另两条在提交后丢弃控制面的回复：决策回复丢失后，同一 `Idempotency-Key` 重试必须落到同一条记录，只产生一个 `ACTION_RESPONSE` operation（A4 的决策半边）；Turn 准入回复丢失后，重试只产生一行 Turn（C4）。五条都保留原 Action、prepare key、digest 与 Runtime 身份，启动第三代 Harness，并要求一次已结算执行、一次文件效果和一个终态；result-commit 支路还要求重发复用同一个已保存请求，并撤回已发布的片段。另外三条覆盖接下来的屏障：`turn.settled` 已提交但回复丢失时，替换后必须投影一次终态（不要求第二次 store 提交），且不产生推理或工具调用，公开文本仍必须是恢复出的答案（C1）；终态写入被拒时，可付 checkpoint 必须由替换方恰好补偿一次，且恢复出的答案仍可见（C2）。清理释放被拒、随后再次崩溃（C3）也已打包：Turn 进入终态且其 Harness 代际已死之后，控制面通过 cleanup-only load 重试欠付清理，直到替换方确认（[清理欠债重试](2026-10-28-hosted-cleanup-debt-retry.zh-CN.md)）。接管 load 回复丢失（A4 的 load 半边）仍保持定向覆盖：打包支路显示重发的 active handoff 没有投影终态，原 Turn 停在 RUNNING，修复该恢复路径与本验收切片分开。损坏/外部 ref、快照超限、替换后的混合调用与重写输入（A3）、Hook/MCP/child 不确定性（A6）、attempt 与 stream 身份及实时 SSE 视图（B2）、轮次与成本预算（B3）、stream Finished 与 assistant 提交之间的崩溃（B3a），以及混版本 reader/bundle 闭包（R2）仍由定向或集成证据覆盖。两侧 owner 重启、真实 provider 和真实混版本集群仍独立处理。Q2 的 binding 半边有 D11 的冻结控制面支路覆盖醒来的旧 Spring 的过期 claim，并有 store 测试钉住 bind 守卫；两个存活控制面仍属后续阶段。
+#14481 的打包门禁在同一任务里新增十一条故障支路，其中七条随第 2 步落地。五条在 Allow 路径上叠加一层故障：三条在持久 A5 屏障之后崩溃替代 Harness——一条在 Allow 决策落库后（A5 的决策行），一条在批准执行到达 `PREPARED`、Runtime Broker 的 `:start` 被扣住时（A5a 的部分 prepare），还有一条在批准结果已提交、post-tool 模型轮被扣住时（A5 的 result commit 屏障）——另两条在提交后丢弃控制面的回复：决策回复丢失后，同一 `Idempotency-Key` 重试必须落到同一条记录，只产生一个 `ACTION_RESPONSE` operation（A4 的决策半边）；Turn 准入回复丢失后，重试只产生一行 Turn（C4）。五条都保留原 Action、prepare key、digest 与 Runtime 身份，启动第三代 Harness，并要求一次已结算执行、一次文件效果和一个终态；result-commit 支路还要求重发复用同一个已保存请求，并撤回已发布的片段。另外四条覆盖接下来的屏障：`turn.settled` 已提交但回复丢失时，替换后必须投影一次终态（不要求第二次 store 提交），且不产生推理或工具调用，公开文本仍必须是恢复出的答案（C1）；终态写入被拒时，可付 checkpoint 必须由替换方恰好补偿一次，且恢复出的答案仍可见（C2）。清理释放被拒、随后再次崩溃（C3）也已打包：Turn 进入终态且其 Harness 代际已死之后，控制面通过 cleanup-only load 重试欠付清理，直到替换方确认（[清理欠债重试](2026-10-28-hosted-cleanup-debt-retry.zh-CN.md)）。接管 load 回复丢失（A4 的 load 半边）也已打包：替换方第一次 drive load 的回复被丢失一次（测试门控钩子），控制面把同一 load 重发进已启动的 active driver，原 Turn 仍须恰好投影一个终态并保留恢复出的答案——重发原样重述 active attachment，而不是把 Turn 卡死。早先观察到的卡 RUNNING 变体在当前代码树上复现不出，因此该臂作为回归门禁守住这条交接，而非标记一个未决缺口。损坏/外部 ref、快照超限、替换后的混合调用与重写输入（A3）、Hook/MCP/child 不确定性（A6）、attempt 与 stream 身份及实时 SSE 视图（B2）、轮次与成本预算（B3）、stream Finished 与 assistant 提交之间的崩溃（B3a），以及混版本 reader/bundle 闭包（R2）仍由定向或集成证据覆盖。两侧 owner 重启、真实 provider 和真实混版本集群仍独立处理。Q2 的 binding 半边有 D11 的冻结控制面支路覆盖醒来的旧 Spring 的过期 claim，并有 store 测试钉住 bind 守卫；两个存活控制面仍属后续阶段。
 
 ## 10. 实现范围
 

package.json

@@ -83,6 +83,7 @@
     "test:e2e:managed-harness-restart-approval-cleanup-failed": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault cleanup-failed",
     "test:e2e:managed-harness-restart-approval-decision-reply-lost": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault decision-reply-lost",
     "test:e2e:managed-harness-restart-approval-admission-reply-lost": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault admission-reply-lost",
+    "test:e2e:managed-harness-restart-approval-load-reply-lost": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault load-reply-lost",
     "test:e2e:managed-harness-restart-approval-terminal-reply-lost": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault terminal-reply-lost",
     "test:e2e:managed-harness-restart-approval-terminal-write-failed": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify allow --g3-fault terminal-write-failed",
     "test:e2e:managed-harness-restart-continuation-closeout": "tsx scripts/run-managed-agent-server-e2e.ts --continuation-failover --harness-only --g3-verify continuation",

packages/cli/src/serve/hosted-harness-session.ts

@@ -4173,6 +4173,13 @@ export function registerHostedHarnessSessionRoutes(
   const unresolvedOpenings = new Set<string>();
   let initialReconciliations = 0;
   const epoch = contract.bootId.replaceAll('-', '_');
+  // E2E fault injection only (QWEN_E2E_DROP_LOAD_REPLY=<sessionId>, set by the
+  // packaged load-reply-lost arm on the replacement Harness): lose exactly the
+  // first load attach reply for that Session, after its driver has started, so
+  // the control plane must redrive the same load into the now-active driver.
+  // The flag is never set in production.
+  const dropLoadReplySession = process.env['QWEN_E2E_DROP_LOAD_REPLY'];
+  let loadReplyDropped = false;
   const released = new Map<
     string,
     { clientId: string; receipt: HostedHarnessReleaseReceipt }
@@ -4397,6 +4404,14 @@ export function registerHostedHarnessSessionRoutes(
     session: HostedSession,
     recovery?: HostedRuntimeRecoveryReport,
   ): void => {
+    if (dropLoadReplySession === sessionId && !loadReplyDropped) {
+      loadReplyDropped = true;
+      writeStderrLineSafe(
+        `qwen serve: E2E dropping first load reply for Session ${sessionId}`,
+      );
+      res.destroy();
+      return;
+    }
     res.status(200).json({
       sessionId,
       clientId: session.clientId,

packages/cli/src/serve/process-env-guard.test.ts

@@ -67,6 +67,14 @@ const allowedProcessEnvAccesses = normalizeAllowances([
       accesses: { 'key:QWEN_HOSTED_HARNESS_INSTANCE_ID': 1 },
     },
   ],
+  [
+    'packages/cli/src/serve/hosted-harness-session.ts',
+    {
+      reason:
+        'The packaged load-reply-lost E2E arm sets QWEN_E2E_DROP_LOAD_REPLY=<sessionId> on the replacement Harness only; the session bridge reads it once at construction to lose exactly the first load attach reply for that Session as fault injection. It is a process-scoped test switch, never set in production.',
+      accesses: { 'key:QWEN_E2E_DROP_LOAD_REPLY': 1 },
+    },
+  ],
   [
     'packages/cli/src/serve/workspace-recovery-worker.ts',
     {

packages/managed-agent-server/README.md

@@ -1800,11 +1800,20 @@ profile or unreadable evidence keeps its debt and waits for the operator
 recheck route. See
 [the cleanup debt design](../../docs/design/2026-10-28-hosted-cleanup-debt-retry.md).
 
+A lost takeover-load reply (A4's load half) drops the reply to the replacement's
+first drive load once, via a test-gated Harness hook (`QWEN_E2E_DROP_LOAD_REPLY`,
+never set in production). The control plane redrives the same load into the
+now-active driver; the arm asserts the reply was actually lost, then requires the
+original Turn to project exactly one terminal with the recovered answer — the
+redrive restates the active attachment instead of wedging the Turn:
+
+```bash
+npm run test:e2e:managed-harness-restart-approval-load-reply-lost
+```
+
 They write their evidence under `.qwen/investigations/g3-step3-implementation/`;
 `G3_VERIFY_REPORT_DIR` overrides the directory. The remaining rows stay
-focused or integration tested for now: the lost takeover-load reply (the
-packaged arm showed the redrive's active handoff leaves the original Turn
-RUNNING instead of projecting its terminal), mixed calls and rewritten inputs
+focused or integration tested for now: mixed calls and rewritten inputs
 across a replacement, Hook/MCP/child uncertainty, the attempt and stream
 identities plus the live SSE view, the round and cost budgets, and the
 mixed-version reader/bundle closure (#14481).

scripts/run-managed-agent-server-e2e.ts

@@ -81,6 +81,7 @@ const approvalFaults = [
   'cleanup-failed',
   'decision-reply-lost',
   'admission-reply-lost',
+  'load-reply-lost',
   'terminal-reply-lost',
   'terminal-write-failed',
 ] as const;
@@ -123,7 +124,7 @@ for (let index = 0; index < argumentsList.length; index += 1) {
     index += 1;
   } else {
     throw new Error(
-      'Usage: run-managed-agent-server-e2e.ts [--model ID] [--runtime-delay-ms N] [--settings PATH] [--session-failover|--inflight-failover|--continuation-failover|--model-round-failover|--big-output] [--harness-only] [--freeze] [--frozen-control-plane] [--g3-verify allow|deny|cancel|expiry|continuation] [--g3-fault after-decision|after-prepare|after-result|decision-reply-lost|admission-reply-lost|terminal-reply-lost|terminal-write-failed]',
+      'Usage: run-managed-agent-server-e2e.ts [--model ID] [--runtime-delay-ms N] [--settings PATH] [--session-failover|--inflight-failover|--continuation-failover|--model-round-failover|--big-output] [--harness-only] [--freeze] [--frozen-control-plane] [--g3-verify allow|deny|cancel|expiry|continuation] [--g3-fault after-decision|after-prepare|after-result|cleanup-failed|decision-reply-lost|admission-reply-lost|load-reply-lost|terminal-reply-lost|terminal-write-failed]',
     );
   }
 }
@@ -2505,6 +2506,11 @@ try {
             QWEN_MODEL: 'fake-model',
             QWEN_RUNTIME_BROKER_TOKEN: brokerToken,
             QWEN_RUNTIME_BROKER_URL: replacementBrokerUrl,
+            // A4: the replacement loses its first load attach reply once, so
+            // the control plane must redrive into the now-active driver.
+            ...(approvalFault === 'load-reply-lost'
+              ? { QWEN_E2E_DROP_LOAD_REPLY: session.id }
+              : {}),
           },
         },
         name,
@@ -2540,6 +2546,18 @@ try {
         120_000,
         replacementHarness,
       );
+      if (approvalFault === 'load-reply-lost') {
+        // The fault must actually have landed: the replacement lost its first
+        // load attach reply (after its driver started). A redrive that never
+        // lost a reply would prove nothing about the active-driver handoff.
+        await waitUntil(
+          'Takeover load reply lost',
+          () =>
+            replacementHarness.log().includes('E2E dropping first load reply'),
+          120_000,
+          replacementHarness,
+        );
+      }
       const actionBeforeResponse = await fetchJson<Record<string, unknown>>(
         `${replacementSpringUrl}/v1/agents/sessions/${session.id}/actions/${originalApprovalId}`,
         { headers: tenantHeaders(tenant) },

scripts/tests/hosted-process-ci.test.js

@@ -144,6 +144,18 @@ const failoverArms = [
       'admission-reply-lost',
     ],
   ],
+  [
+    'Run takeover load reply-loss E2E',
+    'test:e2e:managed-harness-restart-approval-load-reply-lost',
+    [
+      '--continuation-failover',
+      '--harness-only',
+      '--g3-verify',
+      'allow',
+      '--g3-fault',
+      'load-reply-lost',
+    ],
+  ],
   [
     'Run committed terminal reply-loss E2E',
     'test:e2e:managed-harness-restart-approval-terminal-reply-lost',
@@ -613,8 +625,8 @@ describe('Hosted real-process gates', () => {
     // Step ceilings today (201 + 20 + 50 + 20): Verify, the Broker fault
     // gates, admission and O4. The failover arms moved to their own job.
     ['hosted-harness-mysql', 291, 304],
-    // Twenty-three failover arms at 10 each.
-    ['hosted-failover-e2e', 230, 245],
+    // Twenty-four failover arms at 10 each.
+    ['hosted-failover-e2e', 240, 255],
   ])(
     'keeps the %s job ceiling above its summed step ceilings',
     (name, steps, ceiling) => {

scripts/tests/managed-agent-server-e2e.test.js

@@ -320,6 +320,7 @@ describe('managed-agent-server e2e runner', () => {
       'test:e2e:managed-harness-restart-approval-after-result',
       'test:e2e:managed-harness-restart-approval-decision-reply-lost',
       'test:e2e:managed-harness-restart-approval-admission-reply-lost',
+      'test:e2e:managed-harness-restart-approval-load-reply-lost',
       'test:e2e:managed-harness-restart-approval-terminal-reply-lost',
       'test:e2e:managed-harness-restart-approval-terminal-write-failed',
       'test:e2e:managed-harness-restart-approval-cleanup-failed',
@@ -438,6 +439,17 @@ describe('managed-agent-server e2e runner', () => {
       'Cleanup fault was not retried after the replacement',
     );
     expect(source).toContain("owner_kind='HOSTED_CLEANUP'");
+    // A4 loses the first takeover load reply once via a test-gated hook on the
+    // replacement, and asserts the fault landed before trusting the recovery
+    // (a redrive that never lost a reply would prove nothing).
+    expect(source).toContain('QWEN_E2E_DROP_LOAD_REPLY: session.id');
+    expect(source).toContain("'Takeover load reply lost'");
+    expect(source).toContain('E2E dropping first load reply');
+    const harnessSource = read(
+      'packages/cli/src/serve/hosted-harness-session.ts',
+    );
+    expect(harnessSource).toContain("process.env['QWEN_E2E_DROP_LOAD_REPLY']");
+    expect(harnessSource).toContain('E2E dropping first load reply');
   });
 
   // #12941: the Stage A acceptance criterion names a 15-second Runtime delay,