diff --git a/.github/workflows/sdk-java.yml b/.github/workflows/sdk-java.yml index fb03499744e..f90ab6d45f2 100644 --- a/.github/workflows/sdk-java.yml +++ b/.github/workflows/sdk-java.yml @@ -1097,7 +1097,7 @@ jobs: name: 'Hosted process fault gates / MySQL 8.4 / Java 21' if: "${{ inputs.tier != 'fast' }}" runs-on: 'ubuntu-latest' - # The step ceilings today (203 + 20 + 50 + 20 = 293 — the FG8 letters + # The step ceilings today (204 + 20 + 50 + 20 = 294 — the FG8 letters # grew the shared failsafe fork to 3600 s, FG9 slice 3 to 3990 s, the # FG10b–f workflow relay letters to 4484 s, the FG12a MCP reply-loss # letter to 4499 s, the FG12c restart matrix to 4548 s, the FG13c Hook @@ -1174,18 +1174,21 @@ jobs: # margin (12104 s of fork plus wrap, 16 s left); the FG12d two-server # release cell adds 32 s (16.2 s measured), so the fork moves to 11536 s # and Verify to 203 minutes, wrapping fork + 600 s plus margin (12136 s - # of fork plus wrap, 44 s left); + # of fork plus wrap, 44 s left); the FG12d detach-unknown cell adds 55 s + # (27.4 s measured), so the fork moves to 11591 s and Verify to 204 + # minutes, wrapping fork + 600 s plus margin (12191 s of fork plus wrap, + # 49 s left); # the Broker fault gates use 20, # and admission plus O4 keep per-fork ceilings) plus a 33-minute # allowance for the uncapped setup steps (Checkout.. cgroup delegation, # the report checks), which took under 10 minutes on every nightly from - # 10-14 to 10-30: 293 + 33. A Verify that timed out used to skip + # 10-14 to 10-30: 294 + 33. A Verify that timed out used to skip # admission, leaving that path 50 minutes spare; admission now runs # after it, so that path too can fill all four capped ceilings. The # allowance sits above the ~13-minute floor so that a slow setup (the # CLI install makes up to three attempts) still leaves the O4 gates # time to run. The failover arms run in hosted-failover-e2e. - timeout-minutes: 326 + timeout-minutes: 327 services: mysql: image: 'mysql:8.4.6' @@ -1285,10 +1288,10 @@ jobs: # fork leaves no per-test failure lines, so the main-CI failure # analyzer can only file an undiagnosable per-commit issue (#13503). # hosted-process-ci.test.js pins this ordering; the fork budget is - # 11536 s, so the step wraps to fork + 600 s plus margin (203 min, - # 44 s left before the step must move), including the provisional + # 11591 s, so the step wraps to fork + 600 s plus margin (204 min, + # 49 s left before the step must move), including the provisional # 180 s pool-fixture reservation pending CI measurement. - timeout-minutes: 203 + timeout-minutes: 204 env: MAVEN_ARGS: '--settings ${{ runner.temp }}/setup-java-m2/settings.xml --toolchains ${{ runner.temp }}/setup-java-m2/toolchains.xml' MYSQL_PORT: "${{ job.services.mysql.ports['3306'] }}" diff --git a/docs/design/2026-10-18-mcp-release-fault-gates-d.md b/docs/design/2026-10-18-mcp-release-fault-gates-d.md index a5119158e79..54757e3b2ac 100644 --- a/docs/design/2026-10-18-mcp-release-fault-gates-d.md +++ b/docs/design/2026-10-18-mcp-release-fault-gates-d.md @@ -132,3 +132,21 @@ Decisions: Mutation witness: `drainConfigurations` iterates `configurations.slice(0, 1)` in a copy of the packaged CLI. The two-server case is red at `settle must complete` because the second configuration stays active; the one-server case stays green, so this case is the one that pins the second configuration. The restored bundle is green. Budget: the two-server case measures 16.16–16.24 s standalone (two runs) on JDK 21, MySQL 8.4.6 and Linux x86_64; the ten-case class is 81.6 s. It adds 32 s (2×) to the shared healthy sum (11504 → 11536 s), so the `hosted-harness-mysql` fork moves to 11536 s and the Verify step to 203 minutes (12136 s of fork plus wrap); the job ceiling moves to 326 (293 + 33). The `hosted-mcp-gates` profile and the artifact root are unchanged. + +## Follow-up cell: detach refused while a tool result is unknown + +#15750's FG12d row also asks for a detach that refuses while a tool result is unknown. The class's existing unknown-result cases (`stdioKillDuringRequestSettlesOutcomeUnknownOnMySql` and `workerKillDuringOperationSettlesOutcomeUnknownOnMySql`) stop at the unknown; this follow-up attempts the Harness detach route while the record is `recovery_blocked` / `outcome_unknown`. + +| Case | Injection | Expect | +| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `detach-refused-unknown-result` | Start an `ext://hold` raw operation through the private route; kill the stdio fixture; wait for `outcome_unknown` / `recovery_blocked`; POST the Harness detach route | Detach answers 409 `hosted_turn_active` or 503 `managed_session_close_failed` (the close catches the MCP recovery error); no `mcp-release` control; the operation stays unknown; the Session remains attached and reopenable. | + +Decisions: + +1. **The Harness detach route is the gate.** The control-plane lifecycle close is exercised by the R1-1 and two-server cells; this cell calls `POST /session/{id}/detach` directly, which is the production detach that reaches `HostedMcpSession.close()`. +2. **The unknown is permanent, not a timed window.** The stdio fixture is killed with an `ext://hold` read in flight, so the operation can only settle through a conclusive answer from its original Runtime generation. The detach has no window to race. +3. **The refusal is one of the documented close answers.** The test accepts 409 `hosted_turn_active` (MCP busy/recovering) and 503 `managed_session_close_failed` / `hosted_mcp_recovery_required`; it rejects 200/204 and asserts no release control and an unchanged unknown record. + +Mutation witness: wrap `session.mcp?.close()` in the close path of a packaged CLI copy in `try { … } catch {}`; detach answers 204 and the case is red at the status assertion. The restored bundle is green. + +Budget: the case measures 27.4 s worst of two runs (27.4 s and 17.4 s) on JDK 21, MySQL 8.4.6 and Linux x86_64; booked at 55 s (2×). The `hosted-harness-mysql` fork moves to 11591 s, the Verify step to 204 minutes and the job ceiling to 327 (294 + 33). diff --git a/docs/design/2026-10-18-mcp-release-fault-gates-d.zh-CN.md b/docs/design/2026-10-18-mcp-release-fault-gates-d.zh-CN.md index a5d59f79d72..202cf323816 100644 --- a/docs/design/2026-10-18-mcp-release-fault-gates-d.zh-CN.md +++ b/docs/design/2026-10-18-mcp-release-fault-gates-d.zh-CN.md @@ -132,3 +132,21 @@ v2 关闭被准入之后，store 会 fence 普通写者续租，而 Harness 要 变异见证：在打包 CLI 的副本中让 `drainConfigurations` 只遍历 `configurations.slice(0, 1)`。两个 server 的用例在 `settle must complete` 处变红，因为第二个配置保持 active；一个 server 的用例仍绿，因此是这个用例钉住了第二个配置。还原后的 bundle 复绿。 预算：两个 server 的用例在 JDK 21、MySQL 8.4.6、Linux x86_64 上单独实测 16.16–16.24 秒（两次），十用例整类 81.6 秒。按 2× 记 32 秒加入共享健康总和（11504 → 11536 秒），`hosted-harness-mysql` fork 移到 11536 秒，Verify 步骤移到 203 分钟（含 wrap 共 12136 秒），job 上限移到 326（293 + 33）。`hosted-mcp-gates` profile 与产物根不变。 + +## 后续格：工具结果未知时 detach 被拒 + +#15750 的 FG12d 行还要求：工具结果未知时 detach 必须被拒。本类已有的 unknown 结果用例（`stdioKillDuringRequestSettlesOutcomeUnknownOnMySql` 与 `workerKillDuringOperationSettlesOutcomeUnknownOnMySql`）只停在 unknown；本后续格在记录处于 `recovery_blocked` / `outcome_unknown` 时调用 Harness 的 detach 路由。 + +| 用例 | 注入 | 期望 | +| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `detach-refused-unknown-result` | 通过私有路由发起 `ext://hold` 原始操作；杀掉 stdio fixture；等 `outcome_unknown` / `recovery_blocked`；POST Harness detach 路由 | detach 返回 409 `hosted_turn_active` 或 503 `managed_session_close_failed`（close 捕获 MCP recovery 错误）；没有 `mcp-release` 控制；操作保持 unknown；Session 保持 attached、可重开 | + +决策： + +1. **以 Harness detach 路由为门禁。** 控制面 lifecycle close 已由 R1-1 与两 server 用例覆盖；本格直接调 `POST /session/{id}/detach`，即生产 detach 到达 `HostedMcpSession.close()` 的路径。 +2. **unknown 是永久的，不是定时窗口。** 带 `ext://hold` 读在飞时杀掉 stdio fixture，操作只能由原 Runtime 代的确定性应答结算；detach 没有可赛跑的窗口。 +3. **拒绝码取自已记录的关闭应答。** 测试接受 409 `hosted_turn_active`（MCP busy/recovering）与 503 `managed_session_close_failed` / `hosted_mcp_recovery_required`；拒绝 200/204，并断言没有 release 控制、记录仍是 unknown。 + +变异见证：在打包 CLI 副本的 close 路径中把 `session.mcp?.close()` 包进 `try { … } catch {}`；detach 返回 204，用例在状态断言处变红。还原后的 bundle 复绿。 + +预算：该用例在 JDK 21、MySQL 8.4.6、Linux x86_64 上两次实测取最慢 27.4 秒（27.4/17.4），按 2× 记 55 秒。`hosted-harness-mysql` fork 移到 11591 秒，Verify 移到 204 分钟，job 上限移到 327（294 + 33）。 diff --git a/packages/managed-agent-server/verification/pom.xml b/packages/managed-agent-server/verification/pom.xml index 7451cf6944a..36910b92333 100644 --- a/packages/managed-agent-server/verification/pom.xml +++ b/packages/managed-agent-server/verification/pom.xml @@ -284,7 +284,7 @@ qwen.hosted.mysql.isolation = true - 11536 + 11591 diff --git a/packages/managed-agent-server/verification/src/test/java/com/alibaba/qwen/code/managedagent/HostedMcpRestartFaultGatesIT.java b/packages/managed-agent-server/verification/src/test/java/com/alibaba/qwen/code/managedagent/HostedMcpRestartFaultGatesIT.java index 6fbcb0da067..75e9636e470 100644 --- a/packages/managed-agent-server/verification/src/test/java/com/alibaba/qwen/code/managedagent/HostedMcpRestartFaultGatesIT.java +++ b/packages/managed-agent-server/verification/src/test/java/com/alibaba/qwen/code/managedagent/HostedMcpRestartFaultGatesIT.java @@ -74,23 +74,28 @@ class HostedMcpRestartFaultGatesIT { // Per-case evidence letter: FG12c keeps its directory, FG12d's release // cell lands under its own so the two letters' artefacts never mix. private String caseLetter; - private static final Map CASE_LETTERS = Map.of( - "harnessKillDuringConfigureResolvesThroughStatusOnMySql", - "fg12c", - "harnessKillDuringOperationLooksUpTheOriginalInvokeOnMySql", - "fg12c", - "workerKillWhileIdleNeverMintsAFreshOwnerOnMySql", "fg12c", - "workerKillDuringOperationSettlesOutcomeUnknownOnMySql", "fg12c", - "stdioKillDuringRequestSettlesOutcomeUnknownOnMySql", "fg12c", - "harnessKillDuringReleaseResumesTheOriginalReleaseOnMySql", - "fg12d", - "firstReleaseRefusedBeforeForwardingResumesUnderTheSameId", - "fg12d", - "firstReleaseRefusedWithTwoServersReleasesBothOnMySql", - "fg12d", - "cancelBeforeDispatchIsNotStartedProvenOnMySql", "fg12d", - "ownerReleaseRefusedThenSpringRestartReleasesTheHolderOnMySql", - "fg12d"); + private static final Map CASE_LETTERS = Map.ofEntries( + Map.entry("harnessKillDuringConfigureResolvesThroughStatusOnMySql", + "fg12c"), + Map.entry("harnessKillDuringOperationLooksUpTheOriginalInvokeOnMySql", + "fg12c"), + Map.entry("workerKillWhileIdleNeverMintsAFreshOwnerOnMySql", + "fg12c"), + Map.entry("workerKillDuringOperationSettlesOutcomeUnknownOnMySql", + "fg12c"), + Map.entry("stdioKillDuringRequestSettlesOutcomeUnknownOnMySql", + "fg12c"), + Map.entry("detachRefusedWhileToolResultUnknownOnMySql", "fg12d"), + Map.entry("harnessKillDuringReleaseResumesTheOriginalReleaseOnMySql", + "fg12d"), + Map.entry("firstReleaseRefusedBeforeForwardingResumesUnderTheSameId", + "fg12d"), + Map.entry("firstReleaseRefusedWithTwoServersReleasesBothOnMySql", + "fg12d"), + Map.entry("cancelBeforeDispatchIsNotStartedProvenOnMySql", + "fg12d"), + Map.entry("ownerReleaseRefusedThenSpringRestartReleasesTheHolderOnMySql", + "fg12d")); private static final String TOKEN = "extension-process-fixture-token"; private static final String CAPABILITY = "sha256:" + "a".repeat(64); private final ObjectMapper json = new ObjectMapper(); @@ -665,6 +670,45 @@ class HostedMcpRestartFaultGatesIT { pending.cancel(true); } + // FG12d (issue #14447): a tool result is unknown, so detach must refuse + // before it releases anything. The Session stays attached and the + // operation stays recovery_blocked; a later generation owns the only + // conclusive answer for its original id. + @Test + @Timeout(600) + void detachRefusedWhileToolResultUnknownOnMySql() throws Exception { + SessionRef session = createSession(); + baseline(session); + String operation = uuid(); + Future pending = caseExecutor.submit( + () -> privateRequest("POST", "/session/" + session.id() + + "/mcp/operations", Map.of("operationId", operation, + "serverId", "fixture", "request", Map.of("kind", + "resource_read", "uri", "ext://hold")), + session, 202)); + waitFor(() -> assertThat(fixtureEvents("mcp.hold")).hasSize(1)); + kill(fixturePid()); + waitFor(() -> assertThat(operations(session.id()).getLast() + .path("run").path("execution").asText()) + .isEqualTo("outcome_unknown")); + assertThat(operations(session.id()).getLast().path("run").path("state") + .asText()).isEqualTo("recovery_blocked"); + HttpResponse detach = rawRequest("POST", + "/session/" + session.id() + "/detach", "{}", + session.clientId()); + assertThat(detach.statusCode()).as("detach: %s", detach.body()) + .isIn(409, 503); + JsonNode refusal = json.readTree(detach.body()); + assertThat(refusal.path("error").asText( + refusal.path("code").asText())) + .isIn("hosted_turn_active", "managed_session_close_failed", + "hosted_mcp_recovery_required"); + assertThat(controls("mcp-release")).isEmpty(); + assertThat(operations(session.id()).getLast().path("run").path("execution") + .asText()).isEqualTo("outcome_unknown"); + pending.cancel(true); + } + private long fixturePid() { return fixtureEvents("mcp.start").getFirst().path("pid").asLong(); } diff --git a/scripts/tests/hosted-process-ci.test.js b/scripts/tests/hosted-process-ci.test.js index 49fe38a9267..a6d63522a51 100644 --- a/scripts/tests/hosted-process-ci.test.js +++ b/scripts/tests/hosted-process-ci.test.js @@ -661,13 +661,13 @@ describe('Hosted real-process gates', () => { }); it.each([ - // Step ceilings today (203 + 20 + 50 + 20): Verify, the Broker fault + // Step ceilings today (204 + 20 + 50 + 20): Verify, the Broker fault // gates, admission and O4. The failover arms moved to their own job. // The MySQL job's ceiling carries 33 minutes of setup allowance, above // the 13-minute floor: a Verify that timed out used to skip admission, // and now that path too can fill all four ceilings, so a slow setup // must still leave the O4 gates time to run. - ['hosted-harness-mysql', 293, 326, 33], + ['hosted-harness-mysql', 294, 327, 33], // Twenty-seven failover arms at 10 each; the allowance is ~13 minutes of // setup plus 2. ['hosted-failover-e2e', 270, 285, 15], @@ -930,11 +930,13 @@ describe('Hosted real-process gates', () => { // (21–23 s measured), moving the fork to 11504 and the step to 202 // minutes; the FG12d two-server release cell adds 32 s (2x its // measured 16.2 s), moving the fork to 11536 and the step to 203 - // minutes until CI measures it: the relational check above tolerates - // anything up to step*60, so the value is pinned literally here — - // an unnoticed raise has no other red to land on. + // minutes; the FG12d detach-unknown cell adds 55 s (2x its measured + // 27.4 s), moving the fork to 11591 and the step to 204 minutes until + // CI measures it: the relational check above tolerates anything up to + // step*60, so the value is pinned literally here — an unnoticed raise + // has no other red to land on. if (profile === 'hosted-harness-mysql') { - expect(forkSeconds).toBe(11536); + expect(forkSeconds).toBe(11591); } }, );