mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 17:46:16 -10:00
fix(managed-agent): bound the answered action response by its Action's expiry
Round 7 made "the Harness already answered" a row-level watermark: once budget_exempt_attempt is set, `answered` is true on every later attempt, so the terminal gate `!answered && ...` became unreachable and the only reset was a client replay. Java keeps no expiry scanner for managed_agent_action — only the journal projection writes it — so an answer whose decision never projected retried forever. That is not merely a spin: hasOpenOperation counts ACTION_RESPONSE rows and requireNoOpenOperation gates close, archive, delete, rename and unarchive, so one such row refuses every later lifecycle operation on its Session with 409 session_operation_active while reading as a healthy RUNNING/PENDING with a NULL error_code — the wedge this PR exists to remove, reintroduced. Bounds the exemption by the Action's own life, which is the condition the exemption exists for: waiting only helps while the Action can still be decided. Admission validates expiresAt as a required number greater than createdAt, so a real Action always carries that deadline; once it has passed while the Action is still `requested`, the operation completes FAILED with `action_response_decision_expired`. The code is its own rather than action_response_delivery_failed because the Harness did answer, and the record stays java_durable because with no projection Java cannot certify what was committed. An absent expiry is not read as a deadline, so a row this code should not judge keeps waiting instead of terminating on an invented one. Contract and docs follow: the failure_code description on both OpenAPI surfaces and the regenerated client, and the actions design doc in both languages — which also still described re-admission as needing only `requested` plus a Session that accepts responses, omitting the expiry condition the code has checked since round 4. Pinned by anAnsweredResponseTerminatesOnceItsActionExpired. Round 7's aLaterFailureAfterAnAnsweredAttemptStaysBudgetExempt stays green unchanged: its Action carries no expiry, so the exemption it pins still applies. Mutant-verified: removing the bound leaves the new test red with "wanted but not invoked", i.e. the row retries forever.
This commit is contained in:
1 parent
9989e1cea6
commit
d80cc925bd
6 files changed
+110
-7
No files matched your search
@@ -266,7 +266,8 @@ the Hosted prompt ID when a matching Java Turn exists.
|
||||
completes as FAILED with `action_response_delivery_failed` and stays
|
||||
`java_durable` — the admission stage never claims a Harness confirmation
|
||||
that did not happen. That FAILED is not the end of the answer: while the
|
||||
Action is still `requested` and the Session still accepts responses, a
|
||||
Action is still `requested`, has not expired, and the Session still accepts
|
||||
responses, a
|
||||
retried request under the same key and digest re-admits the delivery on
|
||||
the same operation row (back to `pending`, budget reset, receipt cleared)
|
||||
instead of returning the stale failure forever — one row per key keeps a
|
||||
@@ -275,7 +276,14 @@ the Hosted prompt ID when a matching Java Turn exists.
|
||||
place — `completed`, harness-confirmed, with the decision receipt —
|
||||
because the delivery did land and only Java's record was wrong. While the
|
||||
Action stays `requested`, for example on a recovery-blocked Session, the
|
||||
operation stays `running` within that budget. The WebShell request gains
|
||||
operation stays `running` within that budget. An answer that was delivered
|
||||
but never projected is bounded the same way: it waits budget-exempt while
|
||||
the Action can still be decided, and once the Action has expired while
|
||||
still `requested` the operation completes as FAILED with
|
||||
`action_response_decision_expired`, staying `java_durable` because no
|
||||
projection lets Java certify what the Harness committed. Without that
|
||||
bound the row would outlive its Action and, through the open-operation
|
||||
barrier, block every later lifecycle operation on the Session. The WebShell request gains
|
||||
`requestId`.
|
||||
|
||||
### 6.3 Checks
|
||||
|
||||
@@ -127,7 +127,7 @@ Session Store 已经会读取每一行已提交的 journal 来投影 Stage H 记
|
||||
|
||||
- **列表:** `GET …/actions` 与 WebShell `actions/query` 按从新到旧分页列出 Session 中 `requested` 的 Action,使用 Turn 列表的游标与 limit 规则。
|
||||
- **读取:** `GET …/actions/{actionId}` 与 WebShell `actions/get` 返回任何状态的 Action;已决定的 Action 带有 `decision_receipt_id`。
|
||||
- **回答:** `POST …/actions/{actionId}/responses` 与 WebShell `actions/respond` 以 `202` 返回 `action_response` command operation。受理前按 Action 的类型、版本、选项、状态与过期时间检查请求。该 operation 在 D4 的幂等域(租户、Session、类型、actor 与键)内幂等。worker 把它转发到 Harness 路由,失败的尝试按 dispatch 退避回到 pending;与 D4 一样,重试受 `dispatch.max-operation-retries`(默认 10 次)约束。worker 以 Java 从 journal 投影出的 Action 为准判断结果,从不根据时钟或仅凭失败的调用推断:Harness 已记录的决定可能已经让调用运行,只是它的回答丢失了,或者 Harness 已经重启。Harness 在返回 `200` 之前已经提交了决定,因此投影中已经能看到它。投影出的 Action 进入终态后,该 operation 才完成:若为 `decided` 且决定与本次回答相同(Java 按第 5.4 节的确切决定编码比较摘要),以 `action_resolution`(`decided`,带决定回执)完成;否则以其结束状态(`action_expired`、`action_cancelled` 或 `action_already_resolved`)完成,并通过 `failure_code`(WebShell 为 `failureCode`)公开。Harness 返回 `400` 时以该错误完成。重试预算耗尽而 Harness 从未应答时,operation 以 `action_response_delivery_failed` 失败完成,并保持 `java_durable`——准入阶段不会谎称一次从未发生的 Harness 确认。该 FAILED 并非应答的终点:只要 Action 仍为 `requested` 且 Session 仍接受应答,按同一幂等键与摘要重发的请求会在同一 operation 行上重新受理投递(回到 `pending`、预算清零、回执清空),而不是永远返回陈旧的失败——每个键只有一行,第二票仍不可能。一旦投影显示该 Action 已以本次应答自身的决定进入 `decided`,重放则改为就地结算该行——以 `completed` 完成、标记为 Harness 已确认并带决定回执——因为投递确实已送达,只有 Java 的记录是错的。Action 仍为 `requested` 时(例如在恢复阻塞的 Session 上),operation 在该预算内保持 `running`。WebShell 请求增加 `requestId`。
|
||||
- **回答:** `POST …/actions/{actionId}/responses` 与 WebShell `actions/respond` 以 `202` 返回 `action_response` command operation。受理前按 Action 的类型、版本、选项、状态与过期时间检查请求。该 operation 在 D4 的幂等域(租户、Session、类型、actor 与键)内幂等。worker 把它转发到 Harness 路由,失败的尝试按 dispatch 退避回到 pending;与 D4 一样,重试受 `dispatch.max-operation-retries`(默认 10 次)约束。worker 以 Java 从 journal 投影出的 Action 为准判断结果,从不根据时钟或仅凭失败的调用推断:Harness 已记录的决定可能已经让调用运行,只是它的回答丢失了,或者 Harness 已经重启。Harness 在返回 `200` 之前已经提交了决定,因此投影中已经能看到它。投影出的 Action 进入终态后,该 operation 才完成:若为 `decided` 且决定与本次回答相同(Java 按第 5.4 节的确切决定编码比较摘要),以 `action_resolution`(`decided`,带决定回执)完成;否则以其结束状态(`action_expired`、`action_cancelled` 或 `action_already_resolved`)完成,并通过 `failure_code`(WebShell 为 `failureCode`)公开。Harness 返回 `400` 时以该错误完成。重试预算耗尽而 Harness 从未应答时,operation 以 `action_response_delivery_failed` 失败完成,并保持 `java_durable`——准入阶段不会谎称一次从未发生的 Harness 确认。该 FAILED 并非应答的终点:只要 Action 仍为 `requested`、尚未过期且 Session 仍接受应答,按同一幂等键与摘要重发的请求会在同一 operation 行上重新受理投递(回到 `pending`、预算清零、回执清空),而不是永远返回陈旧的失败——每个键只有一行,第二票仍不可能。一旦投影显示该 Action 已以本次应答自身的决定进入 `decided`,重放则改为就地结算该行——以 `completed` 完成、标记为 Harness 已确认并带决定回执——因为投递确实已送达,只有 Java 的记录是错的。Action 仍为 `requested` 时(例如在恢复阻塞的 Session 上),operation 在该预算内保持 `running`。已送达但始终未投影的应答同样受约束:在 Action 仍可能被决定期间,它不计入预算地等待;而一旦 Action 在仍为 `requested` 时过期,operation 就以 `action_response_decision_expired` 失败完成,并保持 `java_durable`——没有投影,Java 无从证明 Harness 提交了什么。缺少这一上界时,该行会比它的 Action 活得更久,并经由未终结 operation 的准入屏障挡住该 Session 之后所有生命周期操作。WebShell 请求增加 `requestId`。
|
||||
|
||||
### 6.3 检查
|
||||
|
||||
|
||||
+43
-1
@@ -110,7 +110,8 @@ public class ActionResponseCoordinator {
|
||||
} catch (RuntimeException error) {
|
||||
// A lost answer may follow a committed decision. Inspect the projection
|
||||
// again before returning this command to the outbox.
|
||||
if (settled(op, actions.response(tenant, session, operation))) {
|
||||
Response current = actions.response(tenant, session, operation);
|
||||
if (settled(op, current)) {
|
||||
return;
|
||||
}
|
||||
// The answer is durable, not per-attempt: an answered attempt
|
||||
@@ -120,6 +121,34 @@ public class ActionResponseCoordinator {
|
||||
// answered attempt never records action_response_delivery_failed
|
||||
// for a decision the Harness already committed.
|
||||
boolean answered = harnessAnswered || op.budgetExemptAttempt() > 0;
|
||||
// That exemption is bounded by the Action's own life. Once it
|
||||
// expired while still `requested`, no projection can make this
|
||||
// delivery observable any more, so the wait can no longer
|
||||
// succeed — and because Java keeps no expiry scanner of its own,
|
||||
// nothing else would ever end it: the row would outlive the
|
||||
// Action and, through the open-operation barrier, every later
|
||||
// lifecycle operation on the Session. The delivery code would be
|
||||
// false here (the Harness did answer), so an expired decision
|
||||
// records its own. It stays java_durable: with no projection
|
||||
// there is nothing the Harness confirmed that Java could certify.
|
||||
if (answered && decisionExpired(tenant, session, current.actionId())) {
|
||||
LOG.error(
|
||||
"Action response outlived its Action tenant={} session={}"
|
||||
+ " operation={} attempts={}",
|
||||
tenant,
|
||||
session,
|
||||
operation,
|
||||
op.attemptCount(),
|
||||
error);
|
||||
actions.complete(
|
||||
op,
|
||||
owner,
|
||||
"action_response_decision_expired",
|
||||
null,
|
||||
false,
|
||||
clock.millis());
|
||||
return;
|
||||
}
|
||||
if (!answered
|
||||
&& op.attemptCount() - op.budgetExemptAttempt()
|
||||
>= dispatch.getMaxOperationRetries()) {
|
||||
@@ -171,6 +200,19 @@ public class ActionResponseCoordinator {
|
||||
}
|
||||
}
|
||||
|
||||
// Admission validates expiresAt as a required number greater than
|
||||
// createdAt, so a real Action always carries a deadline. One that is
|
||||
// absent is therefore not a deadline but a row this code should not
|
||||
// judge: absent a positive expiry the wait continues rather than
|
||||
// recording a terminal invented from a missing field.
|
||||
private boolean decisionExpired(String tenant, String session, String actionId) {
|
||||
long expiresAt =
|
||||
actions.find(tenant, session, actionId)
|
||||
.map(action -> action.options().path("expiresAt").asLong())
|
||||
.orElse(0L);
|
||||
return expiresAt > 0 && clock.millis() >= expiresAt;
|
||||
}
|
||||
|
||||
private boolean settled(OperationRecord op, Response response) {
|
||||
Action action =
|
||||
actions.find(op.tenantId(), op.sessionId(), response.actionId()).orElseThrow();
|
||||
|
||||
+2
-2
@@ -6469,7 +6469,7 @@
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 128,
|
||||
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
|
||||
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, a delivery that exhausted its retry budget (action_response_delivery_failed), or an answered delivery whose Action expired before its decision projected (action_response_decision_expired). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
|
||||
},
|
||||
"replayed": {
|
||||
"type": "boolean"
|
||||
@@ -6855,7 +6855,7 @@
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 128,
|
||||
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
|
||||
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, a delivery that exhausted its retry budget (action_response_delivery_failed), or an answered delivery whose Action expired before its decision projected (action_response_decision_expired). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
|
||||
},
|
||||
"replayed": {
|
||||
"type": "boolean"
|
||||
|
||||
+53
@@ -1009,6 +1009,59 @@ class OperationRetryTerminalStateTest {
|
||||
anyLong());
|
||||
}
|
||||
|
||||
// The exemption is bounded by the Action's own life. Once it expired
|
||||
// while still `requested`, no projection can make this delivery
|
||||
// observable any more, so waiting can no longer succeed — and since Java
|
||||
// keeps no expiry scanner, nothing else would ever end the row, which
|
||||
// would keep blocking every later lifecycle operation on the Session
|
||||
// through the open-operation barrier. It terminates with its own code
|
||||
// rather than action_response_delivery_failed, because the Harness did
|
||||
// answer, and as java_durable because no projection lets Java certify
|
||||
// what it committed.
|
||||
@Test
|
||||
void anAnsweredResponseTerminatesOnceItsActionExpired() throws Exception {
|
||||
AgentStateStore sessions = mock(AgentStateStore.class);
|
||||
ManagedActionStore actions = mock(ManagedActionStore.class);
|
||||
HarnessConnector harness = mock(HarnessConnector.class);
|
||||
// attemptCount == budgetExemptAttempt == 10: the Harness answered an
|
||||
// earlier attempt, so the row is on the exempt path.
|
||||
OperationRecord claimed = new OperationRecord("tenant", "session",
|
||||
"op-action", OperationKind.ACTION_RESPONSE, "digest",
|
||||
"RUNNING", "JAVA_DURABLE", "LEASED", "ACTIVE", null, "owner",
|
||||
3, 10, null, null, null, null, 10);
|
||||
JsonNode body = actionBody();
|
||||
when(sessions.claimOperation(eq("tenant"), eq("session"),
|
||||
eq("op-action"), anyString(), any(Duration.class)))
|
||||
.thenReturn(Optional.of(claimed));
|
||||
when(actions.response("tenant", "session", "op-action")).thenReturn(
|
||||
new ManagedActionStore.Response("action-1", body, null,
|
||||
null));
|
||||
when(actions.find("tenant", "session", "action-1")).thenReturn(
|
||||
Optional.of(new ManagedActionStore.Action("action-1",
|
||||
"requested", new ObjectMapper().readTree(
|
||||
"{\"expiresAt\":1}"),
|
||||
null, null)));
|
||||
DaemonHttpException unavailable = mock(DaemonHttpException.class);
|
||||
when(unavailable.getStatusCode()).thenReturn(503);
|
||||
doThrow(unavailable).when(harness).resolveAction("tenant", "session",
|
||||
"action-1", body);
|
||||
|
||||
ActionResponseCoordinator coordinator = new ActionResponseCoordinator(
|
||||
sessions, actions, harness,
|
||||
CoordinatorTestSupport.directExecutor(),
|
||||
Clock.systemUTC(), new ManagedAgentProperties());
|
||||
coordinator.dispatch("tenant", "session", "op-action");
|
||||
|
||||
verify(actions).complete(eq(claimed), anyString(),
|
||||
eq("action_response_decision_expired"), isNull(), eq(false),
|
||||
anyLong());
|
||||
verify(actions, never()).complete(any(), anyString(),
|
||||
eq("action_response_delivery_failed"), any(), anyBoolean(),
|
||||
anyLong());
|
||||
verify(sessions, never()).retryOperation(anyString(), anyString(),
|
||||
anyString(), anyString(), anyLong(), anyLong(), anyBoolean());
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "attemptCount = {0}")
|
||||
@ValueSource(ints = {1, 9})
|
||||
void actionResponseStillRetriesWhileTheBudgetLasts(int attemptCount)
|
||||
|
||||
@@ -767,7 +767,7 @@ export interface components {
|
||||
deliveryState: "pending" | "leased" | "confirmed" | "blocked";
|
||||
receiptId?: string;
|
||||
actionResolution?: components["schemas"]["WebShellActionResolution"];
|
||||
/** @description Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal). */
|
||||
/** @description Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, a delivery that exhausted its retry budget (action_response_delivery_failed), or an answered delivery whose Action expired before its decision projected (action_response_decision_expired). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal). */
|
||||
failureCode?: string;
|
||||
replayed: boolean;
|
||||
} & (unknown & unknown & unknown & unknown);
|
||||
|
||||
Reference in new issue
Block a user