fix(managed-agent): bound the answered action response by its Action's expiry

Round 7 made "the Harness already answered" a row-level watermark: once
budget_exempt_attempt is set, `answered` is true on every later attempt, so
the terminal gate `!answered && ...` became unreachable and the only reset was
a client replay. Java keeps no expiry scanner for managed_agent_action — only
the journal projection writes it — so an answer whose decision never projected
retried forever. That is not merely a spin: hasOpenOperation counts
ACTION_RESPONSE rows and requireNoOpenOperation gates close, archive, delete,
rename and unarchive, so one such row refuses every later lifecycle operation
on its Session with 409 session_operation_active while reading as a healthy
RUNNING/PENDING with a NULL error_code — the wedge this PR exists to remove,
reintroduced.

Bounds the exemption by the Action's own life, which is the condition the
exemption exists for: waiting only helps while the Action can still be
decided. Admission validates expiresAt as a required number greater than
createdAt, so a real Action always carries that deadline; once it has passed
while the Action is still `requested`, the operation completes FAILED with
`action_response_decision_expired`. The code is its own rather than
action_response_delivery_failed because the Harness did answer, and the record
stays java_durable because with no projection Java cannot certify what was
committed. An absent expiry is not read as a deadline, so a row this code
should not judge keeps waiting instead of terminating on an invented one.

Contract and docs follow: the failure_code description on both OpenAPI
surfaces and the regenerated client, and the actions design doc in both
languages — which also still described re-admission as needing only
`requested` plus a Session that accepts responses, omitting the expiry
condition the code has checked since round 4.

Pinned by anAnsweredResponseTerminatesOnceItsActionExpired. Round 7's
aLaterFailureAfterAnAnsweredAttemptStaysBudgetExempt stays green unchanged:
its Action carries no expiry, so the exemption it pins still applies.
Mutant-verified: removing the bound leaves the new test red with "wanted but
not invoked", i.e. the row retries forever.
This commit is contained in:
wenshao committed 2026-10-07 12:24:14 +08:00
1 parent 9989e1cea6
commit d80cc925bd
6 files changed
+110 -7

No files matched your search

@@ -266,7 +266,8 @@ the Hosted prompt ID when a matching Java Turn exists.
completes as FAILED with `action_response_delivery_failed` and stays
`java_durable` — the admission stage never claims a Harness confirmation
that did not happen. That FAILED is not the end of the answer: while the
Action is still `requested` and the Session still accepts responses, a
Action is still `requested`, has not expired, and the Session still accepts
responses, a
retried request under the same key and digest re-admits the delivery on
the same operation row (back to `pending`, budget reset, receipt cleared)
instead of returning the stale failure forever — one row per key keeps a
@@ -275,7 +276,14 @@ the Hosted prompt ID when a matching Java Turn exists.
place — `completed`, harness-confirmed, with the decision receipt —
because the delivery did land and only Java's record was wrong. While the
Action stays `requested`, for example on a recovery-blocked Session, the
operation stays `running` within that budget. The WebShell request gains
operation stays `running` within that budget. An answer that was delivered
but never projected is bounded the same way: it waits budget-exempt while
the Action can still be decided, and once the Action has expired while
still `requested` the operation completes as FAILED with
`action_response_decision_expired`, staying `java_durable` because no
projection lets Java certify what the Harness committed. Without that
bound the row would outlive its Action and, through the open-operation
barrier, block every later lifecycle operation on the Session. The WebShell request gains
`requestId`.
### 6.3 Checks
@@ -127,7 +127,7 @@ Session Store 已经会读取每一行已提交的 journal 来投影 Stage H 记
- **列表:** `GET …/actions` 与 WebShell `actions/query` 按从新到旧分页列出 Session 中 `requested` 的 Action,使用 Turn 列表的游标与 limit 规则。
- **读取:** `GET …/actions/{actionId}` 与 WebShell `actions/get` 返回任何状态的 Action;已决定的 Action 带有 `decision_receipt_id`。
- **回答:** `POST …/actions/{actionId}/responses` 与 WebShell `actions/respond` 以 `202` 返回 `action_response` command operation。受理前按 Action 的类型、版本、选项、状态与过期时间检查请求。该 operation 在 D4 的幂等域(租户、Session、类型、actor 与键)内幂等。worker 把它转发到 Harness 路由,失败的尝试按 dispatch 退避回到 pending;与 D4 一样,重试受 `dispatch.max-operation-retries`(默认 10 次)约束。worker 以 Java 从 journal 投影出的 Action 为准判断结果,从不根据时钟或仅凭失败的调用推断:Harness 已记录的决定可能已经让调用运行,只是它的回答丢失了,或者 Harness 已经重启。Harness 在返回 `200` 之前已经提交了决定,因此投影中已经能看到它。投影出的 Action 进入终态后,该 operation 才完成:若为 `decided` 且决定与本次回答相同(Java 按第 5.4 节的确切决定编码比较摘要),以 `action_resolution`(`decided`,带决定回执)完成;否则以其结束状态(`action_expired`、`action_cancelled` 或 `action_already_resolved`)完成,并通过 `failure_code`(WebShell 为 `failureCode`)公开。Harness 返回 `400` 时以该错误完成。重试预算耗尽而 Harness 从未应答时,operation 以 `action_response_delivery_failed` 失败完成,并保持 `java_durable`——准入阶段不会谎称一次从未发生的 Harness 确认。该 FAILED 并非应答的终点:只要 Action 仍为 `requested` 且 Session 仍接受应答,按同一幂等键与摘要重发的请求会在同一 operation 行上重新受理投递(回到 `pending`、预算清零、回执清空),而不是永远返回陈旧的失败——每个键只有一行,第二票仍不可能。一旦投影显示该 Action 已以本次应答自身的决定进入 `decided`,重放则改为就地结算该行——以 `completed` 完成、标记为 Harness 已确认并带决定回执——因为投递确实已送达,只有 Java 的记录是错的。Action 仍为 `requested` 时(例如在恢复阻塞的 Session 上),operation 在该预算内保持 `running`。WebShell 请求增加 `requestId`。
- **回答:** `POST …/actions/{actionId}/responses` 与 WebShell `actions/respond` 以 `202` 返回 `action_response` command operation。受理前按 Action 的类型、版本、选项、状态与过期时间检查请求。该 operation 在 D4 的幂等域(租户、Session、类型、actor 与键)内幂等。worker 把它转发到 Harness 路由,失败的尝试按 dispatch 退避回到 pending;与 D4 一样,重试受 `dispatch.max-operation-retries`(默认 10 次)约束。worker 以 Java 从 journal 投影出的 Action 为准判断结果,从不根据时钟或仅凭失败的调用推断:Harness 已记录的决定可能已经让调用运行,只是它的回答丢失了,或者 Harness 已经重启。Harness 在返回 `200` 之前已经提交了决定,因此投影中已经能看到它。投影出的 Action 进入终态后,该 operation 才完成:若为 `decided` 且决定与本次回答相同(Java 按第 5.4 节的确切决定编码比较摘要),以 `action_resolution`(`decided`,带决定回执)完成;否则以其结束状态(`action_expired`、`action_cancelled` 或 `action_already_resolved`)完成,并通过 `failure_code`(WebShell 为 `failureCode`)公开。Harness 返回 `400` 时以该错误完成。重试预算耗尽而 Harness 从未应答时,operation 以 `action_response_delivery_failed` 失败完成,并保持 `java_durable`——准入阶段不会谎称一次从未发生的 Harness 确认。该 FAILED 并非应答的终点:只要 Action 仍为 `requested`、尚未过期且 Session 仍接受应答,按同一幂等键与摘要重发的请求会在同一 operation 行上重新受理投递(回到 `pending`、预算清零、回执清空),而不是永远返回陈旧的失败——每个键只有一行,第二票仍不可能。一旦投影显示该 Action 已以本次应答自身的决定进入 `decided`,重放则改为就地结算该行——以 `completed` 完成、标记为 Harness 已确认并带决定回执——因为投递确实已送达,只有 Java 的记录是错的。Action 仍为 `requested` 时(例如在恢复阻塞的 Session 上),operation 在该预算内保持 `running`。已送达但始终未投影的应答同样受约束:在 Action 仍可能被决定期间,它不计入预算地等待;而一旦 Action 在仍为 `requested` 时过期,operation 就以 `action_response_decision_expired` 失败完成,并保持 `java_durable`——没有投影,Java 无从证明 Harness 提交了什么。缺少这一上界时,该行会比它的 Action 活得更久,并经由未终结 operation 的准入屏障挡住该 Session 之后所有生命周期操作。WebShell 请求增加 `requestId`。
### 6.3 检查
@@ -110,7 +110,8 @@ public class ActionResponseCoordinator {
} catch (RuntimeException error) {
// A lost answer may follow a committed decision. Inspect the projection
// again before returning this command to the outbox.
if (settled(op, actions.response(tenant, session, operation))) {
Response current = actions.response(tenant, session, operation);
if (settled(op, current)) {
return;
}
// The answer is durable, not per-attempt: an answered attempt
@@ -120,6 +121,34 @@ public class ActionResponseCoordinator {
// answered attempt never records action_response_delivery_failed
// for a decision the Harness already committed.
boolean answered = harnessAnswered || op.budgetExemptAttempt() > 0;
// That exemption is bounded by the Action's own life. Once it
// expired while still `requested`, no projection can make this
// delivery observable any more, so the wait can no longer
// succeed — and because Java keeps no expiry scanner of its own,
// nothing else would ever end it: the row would outlive the
// Action and, through the open-operation barrier, every later
// lifecycle operation on the Session. The delivery code would be
// false here (the Harness did answer), so an expired decision
// records its own. It stays java_durable: with no projection
// there is nothing the Harness confirmed that Java could certify.
if (answered && decisionExpired(tenant, session, current.actionId())) {
LOG.error(
"Action response outlived its Action tenant={} session={}"
+ " operation={} attempts={}",
tenant,
session,
operation,
op.attemptCount(),
error);
actions.complete(
op,
owner,
"action_response_decision_expired",
null,
false,
clock.millis());
return;
}
if (!answered
&& op.attemptCount() - op.budgetExemptAttempt()
>= dispatch.getMaxOperationRetries()) {
@@ -171,6 +200,19 @@ public class ActionResponseCoordinator {
}
}
// Admission validates expiresAt as a required number greater than
// createdAt, so a real Action always carries a deadline. One that is
// absent is therefore not a deadline but a row this code should not
// judge: absent a positive expiry the wait continues rather than
// recording a terminal invented from a missing field.
private boolean decisionExpired(String tenant, String session, String actionId) {
long expiresAt =
actions.find(tenant, session, actionId)
.map(action -> action.options().path("expiresAt").asLong())
.orElse(0L);
return expiresAt > 0 && clock.millis() >= expiresAt;
}
private boolean settled(OperationRecord op, Response response) {
Action action =
actions.find(op.tenantId(), op.sessionId(), response.actionId()).orElseThrow();
@@ -6469,7 +6469,7 @@
"type": "string",
"minLength": 1,
"maxLength": 128,
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, a delivery that exhausted its retry budget (action_response_delivery_failed), or an answered delivery whose Action expired before its decision projected (action_response_decision_expired). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
},
"replayed": {
"type": "boolean"
@@ -6855,7 +6855,7 @@
"type": "string",
"minLength": 1,
"maxLength": 128,
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, a delivery that exhausted its retry budget (action_response_delivery_failed), or an answered delivery whose Action expired before its decision projected (action_response_decision_expired). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
},
"replayed": {
"type": "boolean"
@@ -1009,6 +1009,59 @@ class OperationRetryTerminalStateTest {
anyLong());
}
// The exemption is bounded by the Action's own life. Once it expired
// while still `requested`, no projection can make this delivery
// observable any more, so waiting can no longer succeed — and since Java
// keeps no expiry scanner, nothing else would ever end the row, which
// would keep blocking every later lifecycle operation on the Session
// through the open-operation barrier. It terminates with its own code
// rather than action_response_delivery_failed, because the Harness did
// answer, and as java_durable because no projection lets Java certify
// what it committed.
@Test
void anAnsweredResponseTerminatesOnceItsActionExpired() throws Exception {
AgentStateStore sessions = mock(AgentStateStore.class);
ManagedActionStore actions = mock(ManagedActionStore.class);
HarnessConnector harness = mock(HarnessConnector.class);
// attemptCount == budgetExemptAttempt == 10: the Harness answered an
// earlier attempt, so the row is on the exempt path.
OperationRecord claimed = new OperationRecord("tenant", "session",
"op-action", OperationKind.ACTION_RESPONSE, "digest",
"RUNNING", "JAVA_DURABLE", "LEASED", "ACTIVE", null, "owner",
3, 10, null, null, null, null, 10);
JsonNode body = actionBody();
when(sessions.claimOperation(eq("tenant"), eq("session"),
eq("op-action"), anyString(), any(Duration.class)))
.thenReturn(Optional.of(claimed));
when(actions.response("tenant", "session", "op-action")).thenReturn(
new ManagedActionStore.Response("action-1", body, null,
null));
when(actions.find("tenant", "session", "action-1")).thenReturn(
Optional.of(new ManagedActionStore.Action("action-1",
"requested", new ObjectMapper().readTree(
"{\"expiresAt\":1}"),
null, null)));
DaemonHttpException unavailable = mock(DaemonHttpException.class);
when(unavailable.getStatusCode()).thenReturn(503);
doThrow(unavailable).when(harness).resolveAction("tenant", "session",
"action-1", body);
ActionResponseCoordinator coordinator = new ActionResponseCoordinator(
sessions, actions, harness,
CoordinatorTestSupport.directExecutor(),
Clock.systemUTC(), new ManagedAgentProperties());
coordinator.dispatch("tenant", "session", "op-action");
verify(actions).complete(eq(claimed), anyString(),
eq("action_response_decision_expired"), isNull(), eq(false),
anyLong());
verify(actions, never()).complete(any(), anyString(),
eq("action_response_delivery_failed"), any(), anyBoolean(),
anyLong());
verify(sessions, never()).retryOperation(anyString(), anyString(),
anyString(), anyString(), anyLong(), anyLong(), anyBoolean());
}
@ParameterizedTest(name = "attemptCount = {0}")
@ValueSource(ints = {1, 9})
void actionResponseStillRetriesWhileTheBudgetLasts(int attemptCount)
@@ -767,7 +767,7 @@ export interface components {
deliveryState: "pending" | "leased" | "confirmed" | "blocked";
receiptId?: string;
actionResolution?: components["schemas"]["WebShellActionResolution"];
/** @description Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal). */
/** @description Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, a delivery that exhausted its retry budget (action_response_delivery_failed), or an answered delivery whose Action expired before its decision projected (action_response_decision_expired). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal). */
failureCode?: string;
replayed: boolean;
} & (unknown & unknown & unknown & unknown);