fix(managed-agent): answer self-healing Harness conflicts as retryable 503 (#14645)

* fix(managed-agent): answer self-healing Harness conflicts as retryable 503

The public automation mutations relayed a dead writer's lease as
`409 managed_session_writer_conflict`, which a client following HTTP
semantics reads as final even though the conflict lapses on its own.

Cross the Harness 409s that heal by themselves as the retryable 503,
sharing the channel routes' conflict set, and keep every other 409.
The fault gate now accepts only 202 or 503, pinning the answer.

* test(managed-agent): pin every self-healing Harness conflict code

Parameterize the retryable-conflict test with a fixed list of the four
codes instead of checking only the writer conflict. Dropping a member
from the shared set now fails the test rather than passing silently.

* fix(managed-agent): keep deferred manual runs inside the scanner budget

A manual-run retry under the same key re-fired a still-firing occurrence
and deferred it again, so a client retrying at roughly 1 Hz could reach
MAX_FIRE_ATTEMPTS and settle the run unknown. Answer the retryable 503
while the recorded defer is still in the future; the scanner owns the
next fire.

Record the hosted_session_blocked exclusion and pin the gate's 503 code.

* fix(managed-agent): let the first lost-answer retry re-drive the run

The defer-window guard blocked the first same-key retry after a lost
answer, so the FG11a manual-run-reply case answered 503 where it expects
202. Apply the guard only once the first re-drive has been spent
(attempts > 1); the first retry still re-drives, and a later retry
answers the code the defer recorded.

Record the defer-window behaviour in the b2 design twins and reconcile
the hosted_session_blocked exclusion with the scanner.

* fix(managed-agent): keep an in-window retry in the recorded refusal class

The defer-window guard answered every in-window retry as a retryable 503,
including final refusals such as hosted_session_blocked, so the same key
and row switched from 409 to 503. Re-answer the class the recorded code
implies: 503 for a transient or lost answer, the same 409 for a final
refusal. Add unit cases for both branches and record the rule in the b2
design twins.

* test(managed-agent): accept the lost-answer code in the FG11b retry gate

The tightened retryUntilAccepted rejected a legitimate 503
automation_operation_unknown: on a cold restart the first retry can lose
its answer before the dead writer's lease is observable. Accept that code
alongside managed_session_writer_conflict, while still rejecting 4xx and
any other coded 503.
This commit is contained in:
Shaojin Wen authored and GitHub committed 2026-10-20 15:06:00 +08:00
1 parent e0d97c5569
commit f2b6ffb69e
8 files changed
+280 -42

No files matched your search

@@ -75,26 +75,35 @@ attempt was the same commit and that at least three were made.
re-drive. The case checks that a new writer generation holds the Session.
- **Retry until accepted.** After the restart, the case retries the same
request under the same key until it answers `202`. Before the lease
lapses, the cold Harness's load is refused, and the public route passes
that through as `409 managed_session_writer_conflict` (see Findings). Any
other refusal fails the case.
lapses, the cold Harness's load is refused, and the public route answers
that as the retryable `503 managed_session_writer_conflict` (see
Findings). Any other refusal, including a bare 409, fails the case.
- **Teardown.** It drops each trigger on its own, so one failing drop
leaves none of the others on the shared fork's schema. Then it drops the
refusal table.
## Findings
- **A transient writer conflict reaches the public caller as a 409.**
Right after a Harness restart, the public manual-run route answered
- **A self-healing Harness conflict crosses as a retryable 503.** Right
after a Harness restart, the public manual-run route used to answer
`409 managed_session_writer_conflict` until the dead writer's lease
lapsed. The update route relays through the same translation
(`ManagedAutomationService.translate`, a Harness 409 stays a 409). The
scanner re-drives this, because the code is not one of its definitive
refusals, so a manual run is not lost. But the public answer is a 4xx
conflict that a client is unlikely to retry, and nothing else re-drives a
definition update. A retry under the same key does recover, because the
receipt stays claimed and unanswered. This slice records the behaviour,
and the gate retries through it rather than pinning it.
(`ManagedAutomationService.translate`). The scanner re-drives the code
because it is not one of its definitive refusals, so a manual run was
never lost, but a client reading the 409 as final stopped, and nothing
else re-drives a definition update. The public automation routes now
cross the Harness conflicts that heal on their own — a dead
predecessor's writer lease, a busy or briefly unavailable Workspace, a
closing Session — as the retryable 503 that keeps the code, the way the
channel routes answer them. Every other 409 keeps its class. The gate
pins the answer: it accepts only `202` or `503`.
A same-key manual-run retry inside the occurrence's defer window answers
the class the recorded failure implies — the retryable 503 for a
transient or lost answer, the same 409 for a final refusal — without
spending an attempt once the first re-drive has been spent; the scanner
owns the next fire. The first retry after a lost answer still re-drives,
so the gate's second send reaches `202`.
## Mutation witnesses
@@ -130,9 +139,6 @@ so no step or job ceiling moves.
## Open questions
- Whether the public automation routes should answer a writer conflict as
a retryable `503`, the way they answer every other unobtained outcome.
See Findings.
- A store transport failure before the store sees the commit, on every
attempt, behaves like `revision-1`: nothing commits, and the Session
latches. This slice gates only the answer-lost variant, which is the one
@@ -58,19 +58,22 @@ Issue #14446 是 #12380 F 阶段的 FG11 轮。FG11b 覆盖存储故障。第 1
15 s,所以死掉的 writer 的 lease 很快过期。下一个到达这个 Session 的请求会重新加载它,可能是
用例自己的重试,也可能是扫描器的重驱。用例会检查已有新的 writer generation 持有这个 Session。
- **重试直到被接受。** 重启后,用例用同一个 key 重试同一个请求,直到它回答 `202`。在 lease 过期
之前,冷 Harness 的加载会被拒绝,公开路由把它原样传成 `409 managed_session_writer_conflict`
(见"发现")。其他任何拒绝都会让用例失败。
之前,冷 Harness 的加载会被拒绝,公开路由把它回答为可重试的
`503 managed_session_writer_conflict`(见"发现")。其他任何拒绝,包括裸的 409,都会让用例失败。
- **teardown。** 逐个删除 trigger,一个删除失败不会让其他 trigger 留在共享 fork 的 schema 上。
然后删除拒绝计数表。
## 发现
- **暂时性的 writer 冲突以 409 到达公开调用方。** Harness 重启后,在死掉的 writer 的 lease 过期
之前,公开的手动 run 路由回答了 `409 managed_session_writer_conflict`。更新路由经过同一个转换
(`ManagedAutomationService.translate`,Harness 的 409 仍是 409)。扫描器会重驱这种情况,因为这个
错误码不在它的终局拒绝之列,所以手动 run 不会丢。但公开应答是一个客户端不太会重试的 4xx 冲突,
而定义更新没有别的东西会重驱。用同一个 key 重试确实能恢复,因为回执保持已占用、未作答。本切片
记录这一行为,门禁会重试越过它,而不把它钉住。
- **会自愈的 Harness 冲突以可重试的 503 越过。** Harness 重启后,公开的手动 run 路由曾经在死掉的
writer 的 lease 过期之前回答 `409 managed_session_writer_conflict`。更新路由经过同一个转换
(`ManagedAutomationService.translate`)。扫描器会重驱这种情况,因为这个错误码不在它的终局拒绝
之列,所以手动 run 不会丢;但把 409 当作终局的客户端会就此停手,而定义更新没有别的东西会重驱。
现在,公开的 automation 路由把会自愈的 Harness 冲突——死掉的前任 writer 的 lease、繁忙或短暂
不可用的 Workspace、正在关闭的 Session——回答为保留原错误码的可重试 503,与 channel 路由的回答
一致。其他所有 409 保持原有的分类。门禁把这一应答钉住:只接受 `202` 或 `503`。
失败后的第一次同 key 重试被消耗后,defer 窗口内再次重试会回答上次失败所对应的类别——暂时性或丢失应答回答可重试的 `503`,终局拒答回答同样的 `409`——且不消耗尝试次数;下一次 fire 由扫描器拥有。丢失应答后的第一次重试仍会立即重驱,因此门禁的第二次发送会得到 `202`。
## 变异见证
@@ -100,7 +103,5 @@ Issue #14446 是 #12380 F 阶段的 FG11 轮。FG11b 覆盖存储故障。第 1
## 未决问题
- 公开的 automation 路由是否应当把 writer 冲突回答为可重试的 `503`,就像它回答其他所有结果不明
的情况那样。见"发现"。
- store 在看到提交之前、每次尝试都发生的传输故障,表现与 `revision-1` 相同:什么都不提交,Session
被锁存。本切片只覆盖应答丢失这一变体,也就是 store client 自己的重试能吸收的那种。
@@ -22,6 +22,7 @@ import com.alibaba.qwen.code.managedagent.persistence.AutomationLedgerStore.Sche
import com.alibaba.qwen.code.managedagent.persistence.AutomationLedgerStore.ScheduleRow;
import com.alibaba.qwen.code.managedagent.persistence.ManagedAgentStore;
import com.alibaba.qwen.code.managedagent.persistence.ManagedWorkspaceRegistry;
import com.alibaba.qwen.code.managedagent.session.HarnessConflictCodes;
import com.alibaba.qwen.code.managedagent.session.ManagedAgentService;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
@@ -349,12 +350,22 @@ public class ManagedAutomationService {
"The automation is being scanned; retry."));
if (AutomationLedgerStore.OUTCOME_FIRING.equals(
occurrence.outcome())) {
if (occurrence.nextRetryAt() > now
&& occurrence.attempts() > 1) {
// The first same-key retry after a failure re-drives,
// so a lost answer still recovers. Once that attempt
// is spent, a retry inside the window must not spend
// another one or move the scanner's schedule; the
// scanner owns the next fire when the delay lapses.
throw deferred(occurrence);
}
try {
scanner.fire(held, occurrence, owner, fence, now);
} catch (RuntimeException error) {
// The claim is durable and re-driven by the scanner
// with backoff; the same key answers it once known.
scanner.defer(held, occurrence, owner, fence, error, now);
scanner.defer(held, occurrence, owner, fence, error,
clock.get());
throw unobtained(error);
}
// A definitive route refusal settles the decision as
@@ -808,11 +819,44 @@ public class ManagedAutomationService {
.toString();
}
private static String deferredCode(OccurrenceRow occurrence) {
String error = occurrence.lastError();
if (error != null) {
int at = error.indexOf("\"code\":\"");
if (at >= 0) {
int end = error.indexOf('"', at + 8);
if (end > at) {
return error.substring(at + 8, end);
}
}
}
return "automation_operation_unknown";
}
private static ManagedAgentException deferred(OccurrenceRow occurrence) {
String code = deferredCode(occurrence);
boolean retryable = "automation_operation_unknown".equals(code)
|| HarnessConflictCodes.isTransient(code);
return new ManagedAgentException(
retryable ? HttpStatus.SERVICE_UNAVAILABLE : HttpStatus.CONFLICT,
code,
retryable
? "The Hosted Harness answer is pending; retry"
+ " the same key."
: "The Hosted Harness refused the automation operation.");
}
static ManagedAgentException translate(DaemonHttpException error) {
String code = AutomationScanner.errorCode(error);
// A self-healing Harness conflict (a dead predecessor's writer
// lease, a busy or briefly unavailable Workspace, a closing
// Session) answers the retryable 503 every other unobtained
// outcome takes: the same request under the same key succeeds
// once it lapses, so a 409 would tell the caller to stop.
HttpStatus status = switch (error.getStatusCode()) {
case 404 -> HttpStatus.NOT_FOUND;
case 409 -> HttpStatus.CONFLICT;
case 409 -> HarnessConflictCodes.isTransient(code)
? HttpStatus.SERVICE_UNAVAILABLE : HttpStatus.CONFLICT;
case 400 -> HttpStatus.BAD_REQUEST;
default -> HttpStatus.SERVICE_UNAVAILABLE;
};
@@ -36,6 +36,7 @@ import com.alibaba.qwen.code.managedagent.persistence.ChannelRouteRepository.Cha
import com.alibaba.qwen.code.managedagent.persistence.JdbcChannelDeliveryRepository;
import com.alibaba.qwen.code.managedagent.persistence.JdbcChannelRouteRepository;
import com.alibaba.qwen.code.managedagent.persistence.ManagedWorkspaceRegistry;
import com.alibaba.qwen.code.managedagent.session.HarnessConflictCodes;
import com.alibaba.qwen.code.managedagent.session.ManagedAgentService;
import com.alibaba.qwen.code.managedagent.workspace.WorkspaceSelection;
import com.fasterxml.jackson.databind.JsonNode;
@@ -82,11 +83,6 @@ public class ManagedChannelService {
private static final int MAX_CLAIM_SESSIONS = 64;
private static final int RECONCILE_LIMIT = 50;
private static final String SEPARATOR = "\u0000";
/** Harness 409 codes that heal on their own; see {@link #translate}. */
private static final java.util.Set<String> TRANSIENT_HARNESS_CONFLICTS =
java.util.Set.of("managed_session_writer_conflict",
"workspace_busy", "workspace_unavailable",
"hosted_session_closing");
private final ChannelInstanceStore instances;
private final ChannelRouteRepository routes;
@@ -780,7 +776,7 @@ public class ManagedChannelService {
// Session) are the opposite: the adapter drops every 409, so they
// cross as the retryable 503.
HttpStatus status = http.getStatusCode() == 409
? (TRANSIENT_HARNESS_CONFLICTS.contains(code)
? (HarnessConflictCodes.isTransient(code)
? HttpStatus.SERVICE_UNAVAILABLE : HttpStatus.CONFLICT)
: http.getStatusCode() == 400 ? HttpStatus.BAD_REQUEST
: HttpStatus.SERVICE_UNAVAILABLE;
@@ -0,0 +1,28 @@
package com.alibaba.qwen.code.managedagent.session;
import java.util.Set;
/**
* The Harness 409 codes that heal on their own: a fenced predecessor's
* writer lease lapses, a busy or briefly unavailable Workspace frees up,
* a closing Session finishes. A public route crosses them as the
* retryable 503, never as a conflict a client reads as final.
*
* <p>{@code hosted_session_blocked} is deliberately excluded: a recovery
* block can lift, but the cleanup-blocked paths may never do so, so a
* retry cannot be promised. The scanner still re-drives this code after a
* refused fire; the public route does not promise the same. Every other
* 409 stays a conflict for the same reason.
*/
public final class HarnessConflictCodes {
private static final Set<String> TRANSIENT = Set.of(
"managed_session_writer_conflict", "workspace_busy",
"workspace_unavailable", "hosted_session_closing");
public static boolean isTransient(String code) {
return code != null && TRANSIENT.contains(code);
}
private HarnessConflictCodes() {
}
}
@@ -632,17 +632,21 @@ class HostedAutomationFaultGatesIT {
// Retries the same request, under the same key, until the reloaded Session answers it: the
// dead writer's lease lapses first, and every attempt before that is the same operation.
// Until it lapses the cold Harness's load is refused, and the public route passes that
// through as 409 managed_session_writer_conflict (see the design's Findings); any other
// refusal fails the case.
// Until it lapses the cold Harness's load is refused, and the public route answers that as
// the retryable 503 managed_session_writer_conflict (see the design's Findings). A 409
// would tell the caller to stop, so only 202 or 503 is accepted; a 503 that names a code
// must name the writer conflict or the lost-answer code, which the cold restart can answer
// before the dead writer's lease is observable. Any other coded 503 fails at once.
private JsonNode retryUntilAccepted(String method, String path, Object body, String key) throws Exception {
AtomicReference<HttpResponse<String>> accepted = new AtomicReference<>();
waiting().atMost(Duration.ofSeconds(90)).pollInterval(Duration.ofSeconds(1)).until(() -> {
HttpResponse<String> response = send(method, path, body, key);
if (response.statusCode() == 409) {
assertThat(response.body()).contains("managed_session_writer_conflict");
} else {
assertThat(response.statusCode()).as(response.body()).isIn(202, 503);
assertThat(response.statusCode()).as(response.body()).isIn(202, 503);
if (response.statusCode() == 503
&& response.body().contains("\"code\":\"")) {
assertThat(response.body()).containsAnyOf(
"managed_session_writer_conflict",
"automation_operation_unknown");
}
accepted.set(response);
return response.statusCode() == 202;
@@ -1071,8 +1071,8 @@ class AutomationScannerTest {
assertThat(claim.attempts()).isEqualTo(1);
assertThat(claim.lastError()).isEqualTo("connection lost");
assertThat(claim.trigger()).isEqualTo(AutomationScanner.TRIGGER_MANUAL);
// The same key asked again re-drives the claim at once: the caller
// is answered what the Harness now says, under the same run id.
// The first same-key retry after a lost answer re-drives the claim:
// the caller is answered what the Harness now says.
var asked = service.run(tenant, ACTOR, id, "lost-1");
assertThat(asked.replayed()).isTrue();
assertThat(asked.body().outcome()).isEqualTo("fired");
@@ -1105,6 +1105,95 @@ class AutomationScannerTest {
.isNull();
}
@Test
void aRetryInsideTheDeferWindowDoesNotSpendTheScannerBudget() {
PublicAutomation automation = define("0 2 * * *", "allow", "none",
null, true);
String id = automation.id();
ScheduleRow held = ledger.findSchedule(tenant, id).orElseThrow();
long seedFence = ledger.claim(tenant, id, "seed",
clock.get() + 60_000, clock.get());
assertThat(seedFence).isPositive();
assertThat(ledger.recordOccurrence(AutomationLedgerStore.OccurrenceRow
.decision(held, "manual:seeded", null,
AutomationScanner.TRIGGER_MANUAL,
AutomationLedgerStore.OUTCOME_FIRING, null, seedFence,
clock.get()), "seed")).isPresent();
assertThat(ledger.deferOccurrence(tenant, id, "manual:seeded", 63,
clock.get() + 30_000, "seeded", "seed", seedFence,
clock.get())).isTrue();
ledger.release(tenant, id, "seed", seedFence, clock.get());
// One more fire would take attempts to MAX_FIRE_ATTEMPTS and settle
// the claim unknown; the guard answers 503 without reaching it.
fake.failNextFire = new DaemonException("must not fire");
assertThatThrownBy(() -> service.run(tenant, ACTOR, id, "seeded"))
.isInstanceOfSatisfying(ManagedAgentException.class, error -> {
assertThat(error.getStatus())
.isEqualTo(HttpStatus.SERVICE_UNAVAILABLE);
assertThat(error.getCode())
.isEqualTo("automation_operation_unknown");
});
assertThat(fake.fires()).isZero();
AutomationLedgerStore.OccurrenceRow after = ledger.findOccurrence(
tenant, id, "manual:seeded").orElseThrow();
assertThat(after.outcome())
.isEqualTo(AutomationLedgerStore.OUTCOME_FIRING);
assertThat(after.attempts()).isEqualTo(63);
}
@Test
void aDeferredRetryAnswersTheRecordedWorkspaceCode() {
PublicAutomation automation = define("0 2 * * *", "allow", "none",
null, true);
String id = automation.id();
seedDeferred(id, "manual:workspace", 2,
"fire failed with HTTP 409: {\"error\":\"workspace_busy\","
+ "\"code\":\"workspace_busy\"}");
fake.failNextFire = new DaemonException("must not fire");
assertThatThrownBy(() -> service.run(tenant, ACTOR, id, "workspace"))
.isInstanceOfSatisfying(ManagedAgentException.class, error -> {
assertThat(error.getStatus())
.isEqualTo(HttpStatus.SERVICE_UNAVAILABLE);
assertThat(error.getCode()).isEqualTo("workspace_busy");
});
assertThat(fake.fires()).isZero();
}
@Test
void aDeferredRetryKeepsAFinalRefusalFinal() {
PublicAutomation automation = define("0 2 * * *", "allow", "none",
null, true);
String id = automation.id();
seedDeferred(id, "manual:blocked", 2,
"fire failed with HTTP 409: {\"error\":\"hosted_session_blocked\","
+ "\"code\":\"hosted_session_blocked\"}");
fake.failNextFire = new DaemonException("must not fire");
assertThatThrownBy(() -> service.run(tenant, ACTOR, id, "blocked"))
.isInstanceOfSatisfying(ManagedAgentException.class, error -> {
assertThat(error.getStatus()).isEqualTo(HttpStatus.CONFLICT);
assertThat(error.getCode())
.isEqualTo("hosted_session_blocked");
});
assertThat(fake.fires()).isZero();
}
private void seedDeferred(String id, String occurrenceKey, int attempts,
String lastError) {
ScheduleRow held = ledger.findSchedule(tenant, id).orElseThrow();
long fence = ledger.claim(tenant, id, "seed",
clock.get() + 60_000, clock.get());
assertThat(fence).isPositive();
assertThat(ledger.recordOccurrence(AutomationLedgerStore.OccurrenceRow
.decision(held, occurrenceKey, null,
AutomationScanner.TRIGGER_MANUAL,
AutomationLedgerStore.OUTCOME_FIRING, null, fence,
clock.get()), "seed")).isPresent();
assertThat(ledger.deferOccurrence(tenant, id, occurrenceKey, attempts,
clock.get() + 30_000, lastError, "seed", fence,
clock.get())).isTrue();
ledger.release(tenant, id, "seed", fence, clock.get());
}
@Test
void theScannerRunsOnADedicatedScheduler() throws Exception {
assertThat(applicationContext.containsBean("managedAutomationScheduler"))
@@ -0,0 +1,70 @@
package com.alibaba.qwen.code.managedagent.automation;
import com.alibaba.qwen.code.daemon.DaemonHttpException;
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import org.springframework.http.HttpStatus;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
class ManagedAutomationServiceTest {
// A dead writer's lease lapses on its own. A 409 would tell the caller
// the conflict is final, so the public route crosses it as the
// retryable 503 the channel routes already answer. The inputs are a
// fixed list rather than the production set, so dropping a member from
// that set fails this test.
@ParameterizedTest
@ValueSource(strings = {"managed_session_writer_conflict",
"workspace_busy", "workspace_unavailable",
"hosted_session_closing"})
void crossesASelfHealingHarnessConflictAsRetryable(String code) {
ManagedAgentException conflict = ManagedAutomationService
.translate(refusal(409, code));
assertThat(conflict.getStatus())
.isEqualTo(HttpStatus.SERVICE_UNAVAILABLE);
assertThat(conflict.getCode()).isEqualTo(code);
}
// A refusal a retry cannot change keeps its class.
@Test
void keepsADeterministicHarnessConflictAsConflict() {
ManagedAgentException conflict = ManagedAutomationService.translate(
refusal(409, "hosted_automations_unavailable"));
assertThat(conflict.getStatus()).isEqualTo(HttpStatus.CONFLICT);
assertThat(conflict.getCode())
.isEqualTo("hosted_automations_unavailable");
}
// A recovery block can lift, but a cleanup block may never: the route
// does not promise a retry for this code.
@Test
void keepsHostedSessionBlockedAsConflict() {
ManagedAgentException conflict = ManagedAutomationService.translate(
refusal(409, "hosted_session_blocked"));
assertThat(conflict.getStatus()).isEqualTo(HttpStatus.CONFLICT);
assertThat(conflict.getCode()).isEqualTo("hosted_session_blocked");
}
// A 409 whose body names no code is not a self-healing conflict.
@Test
void keepsACodeLessHarnessConflictAsConflict() {
DaemonHttpException error = mock(DaemonHttpException.class);
when(error.getStatusCode()).thenReturn(409);
when(error.getResponseBody()).thenReturn(null);
ManagedAgentException conflict = ManagedAutomationService.translate(error);
assertThat(conflict.getStatus()).isEqualTo(HttpStatus.CONFLICT);
assertThat(conflict.getCode()).isEqualTo("automation_operation_failed");
}
private static DaemonHttpException refusal(int status, String code) {
DaemonHttpException error = mock(DaemonHttpException.class);
when(error.getStatusCode()).thenReturn(status);
when(error.getResponseBody()).thenReturn(
"{\"error\":\"" + code + "\",\"code\":\"" + code + "\"}");
return error;
}
}