test(managed-agent): gate the FG11f refusing arms (FG11f) (#14821)

* wip(managed-agent): stage the FG11f publication arms on a public Shell wake turn

Not ready to push: the warm-up turn is refused with workspace_unavailable
before the fault arm runs. Stage 1 of the stacked chain.

* test(managed-agent): gate the FG11f lost-ack publication fault (FG11f)

A wake turn that publishes a Shell capture can lose the object store's
acknowledgement after the bytes landed. Nothing in CI pins what happens next:
the shared OSS stub accepts every write, and the only real-bucket fault gate is
opt-in and outside CI.

The case runs a public Shell session through the packaged stack, faults the
physical put, and pins the properties the store actually guarantees: the
committed capture is kept and every large object is this run's bytes, a refused
write never stores anything, the operation settles from what the store recorded,
the Shell runs once, and the run settles once. `fail-before-write` and
`timeout` remain scoped in the design.

The fixture now registers the storage guard whenever the guard itself is
enabled. It previously registered only on Linux, so a Shell turn on macOS was
refused before its fault arm could run.

The fork budget follows the 2x convention: the cell measured 41.1 s, so the
healthy sum becomes 8527 s, the fork 9127 s, the Verify step 163 minutes and the
job 256.

* test(managed-agent): gate the FG11f refusing arms (FG11f)

The publication gate shipped only the lost-ack arm, and the case resolves which
arm to run from a property, so CI exercised whichever one the default picked.
The two refusing arms now have their own methods: a put that throws before the
bytes land, and a put that stays silent past the operation timeout. Both assert
that nothing was stored, that the operation stays unresolved, that the Shell
runs once and that the run settles once.

Measured locally through the packaged stack: 41.2 s and 80.0 s. The fork budget
follows the 2x convention, so the healthy sum becomes 8769 s, the fork 9369 s,
the Verify step 167 minutes and the job 260.

* fix(managed-agent): drop the unused import and re-align the ceiling comment

The FG11f case carried an import for a symbol it never used; with
includeTestSourceDirectory on and UnusedImports at error severity, that is a
checkstyle failure rather than a warning. The step-ceiling comment also still
read 161 while the literal beneath it is 163.

* refactor(managed-agent): take the FG11f arm as a parameter and re-align the comments

The wrapper methods delegated through a mutable field, which made the case
property stop narrowing a run: naming one arm also ran the other wrapper and
the default-resolving method. The arm is now a helper parameter and each method
gates on `selected()` the way fg11b and fg11c do, so an unset property runs
every arm and naming one runs only that arm.

Two comments still described the older budget: the step-ceiling line read 161
against a 167 literal, and the job-ceiling derivation stopped at the 9045 s
fork without the FG12d, FG11f-arm or refusing-arm steps it now includes.

* test(managed-agent): pin that the fault fired and the payload is the run's own

Three probes showed the arm passing for the wrong reason: with the injection
removed, with the retained bytes replaced by a same-sized placeholder, and with
the operation never settling, the case stayed green and only an absent object
reddened it. The arm now asserts that its injection fired, that every
OBJECT_PUT settles RETURNED, and that the retained payload is exactly the
Shell's 2 MiB of output.

Two ceiling comments still described the pre-merge budget; both now carry the
FG11f step and the values the literals hold.

* test(managed-agent): count the physical puts each object takes

A reconcile that regressed into writing the same key again would be invisible
to the arm: putIfAbsent reports FileAlreadyExists, the real adapter swallows it
and the object map still holds the first write, so the settlement assertions
keep passing. The stub now counts putObject calls per key and the lost-ack arm
asserts every key was written exactly once.

* test(managed-agent): restore the FG11b retryable-status guard

The merge that retargeted this branch to main resolved the FG11b helper from
the branch side, silently reverting #14645: it accepted a 409 as final and
named only one code on a 503. The helper and the comment above the case are
main's again, so a public route that starts answering 409 fails at once and a
coded 503 must name the writer conflict or the lost-answer code.
This commit is contained in:
Shaojin Wen authored and GitHub committed 2026-10-20 17:00:46 +08:00
1 parent 8b6c310c4c
commit fc84ced8a5
6 files changed
+38 -18

No files matched your search

+5 -5
View File
@@ -718,13 +718,13 @@ jobs:
# 150 s, moving the fork to 9261 s and Verify to 165 minutes; the FG13b
# Hook dispatch store-rejection cell adds 150 s, moving the fork to
# 9411 s and Verify to 167 minutes; the FG11f publication cell adds 82 s, so
# the fork moves to 9493 s and Verify to 169 minutes, wrapping fork + 600 s
# the fork moves to 9765 s and Verify to 173 minutes, wrapping fork + 600 s
# plus margin;
# the Broker fault gates use 20,
# and admission plus O4 keep per-fork ceilings) plus ~13 minutes for the
# uncapped setup steps (Checkout.. cgroup delegation, the report
# checks): 259 + 13. The failover arms run in hosted-failover-e2e.
timeout-minutes: 272
# checks): 263 + 13. The failover arms run in hosted-failover-e2e.
timeout-minutes: 276
services:
mysql:
image: 'mysql:8.4.6'
@@ -813,8 +813,8 @@ jobs:
# fork leaves no per-test failure lines, so the main-CI failure
# analyzer can only file an undiagnosable per-commit issue (#13503).
# hosted-process-ci.test.js pins this ordering; the fork budget is
# 9493 s, so the step wraps to fork + 600 s plus margin (169 min).
timeout-minutes: 169
# 9765 s, so the step wraps to fork + 600 s plus margin (173 min).
timeout-minutes: 173
env:
MAVEN_ARGS: '--settings ${{ runner.temp }}/setup-java-m2/settings.xml --toolchains ${{ runner.temp }}/setup-java-m2/toolchains.xml'
MYSQL_PORT: "${{ job.services.mysql.ports['3306'] }}"
@@ -2,9 +2,9 @@
[English](2026-10-19-automation-publication-fault-gates.md) | [简体中文](2026-10-19-automation-publication-fault-gates.zh-CN.md)
Status: slice 1 is partially delivered. `drop-ack-after-write` runs green
against the packaged stack; `fail-before-write` and `timeout` are scoped here
but not implemented, and slice 2 (publisher death) is untouched.
Status: slice 1 is delivered. All three in-process arms — `fail-before-write`,
`drop-ack-after-write` and `timeout` — run green against the packaged stack.
Slice 2 (publisher death) is scoped here and untouched.
Issue: #14446, the FG11 tracker. The tracker's gate table names FG11f as one
of the two slices the round owes. FG11c is delivered; this is the remainder.
@@ -2,8 +2,9 @@
[English](2026-10-19-automation-publication-fault-gates.md) | [简体中文](2026-10-19-automation-publication-fault-gates.zh-CN.md)
状态:第 1 片部分交付。`drop-ack-after-write` 已在打包栈上实跑绿;
`fail-before-write` 与 `timeout` 在此定界但尚未实现,第 2 片(发布者死亡)未触碰。
状态:第 1 片已交付。三条进程内故障臂——`fail-before-write`、
`drop-ack-after-write`、`timeout`——均已在打包栈上实跑绿。
第 2 片(发布者死亡)在此定界,尚未触碰。
Issue:#14446,FG11 tracker。tracker 的 gate 表把 FG11f 列为本轮欠的两片之一。
FG11c 已交付;本设计是剩余部分。
+1 -1
View File
@@ -521,7 +521,7 @@
the FG11f publication cell adds 82 s, so the
fork moves to 9493 s and the Verify step to
169 minutes. -->
<forkedProcessTimeoutInSeconds>9523</forkedProcessTimeoutInSeconds>
<forkedProcessTimeoutInSeconds>9765</forkedProcessTimeoutInSeconds>
</configuration>
<executions>
<execution>
@@ -1089,12 +1089,31 @@ class HostedAutomationFaultGatesIT {
// arm faults the physical object write. The store never rewrites the
// object and never re-runs the Shell, so these assertions read what it
// kept rather than assuming a retry the store does not drive.
// One method per arm keeps a stall in one arm inside its own ceiling. The
// selector keeps the meaning it has for fg11b and fg11c: unset runs every
// arm, and naming one runs only that one.
@Test
@Timeout(600)
void fg11fPublicationFailsBeforeWriteOnMySql() throws Exception {
assumeTrue(selected("fg11f", "fail-before-write"));
runPublicationArm("fail-before-write");
}
@Test
@Timeout(600)
void fg11fPublicationTimesOutOnMySql() throws Exception {
assumeTrue(selected("fg11f", "timeout"));
runPublicationArm("timeout");
}
@Test
@Timeout(600)
void fg11fPublicationFaultsSettleOnceOnMySql() throws Exception {
assumeTrue(anySelected("fg11f", "drop-ack-after-write", "fail-before-write", "timeout"));
String arm = List.of("drop-ack-after-write", "fail-before-write", "timeout").stream()
.filter(name -> selected("fg11f", name)).findFirst().orElse("drop-ack-after-write");
assumeTrue(selected("fg11f", "drop-ack-after-write"));
runPublicationArm("drop-ack-after-write");
}
private void runPublicationArm(String arm) throws Exception {
publicationFaults = publicationFaults(arm);
boot("fg11f");
String definition = request("POST", "/v1/agents", Map.of(
+4 -4
View File
@@ -475,9 +475,9 @@ describe('Hosted real-process gates', () => {
});
it.each([
// Step ceilings today (169 + 20 + 50 + 20): Verify, the Broker fault
// Step ceilings today (173 + 20 + 50 + 20): Verify, the Broker fault
// gates, admission and O4. The failover arms moved to their own job.
['hosted-harness-mysql', 259, 272],
['hosted-harness-mysql', 263, 276],
// Twenty failover arms at 10 each.
['hosted-failover-e2e', 200, 215],
])(
@@ -699,7 +699,7 @@ describe('Hosted real-process gates', () => {
// anything up to step*60, so the value is pinned literally here —
// an unnoticed raise has no other red to land on.
if (profile === 'hosted-harness-mysql') {
expect(forkSeconds).toBe(9523);
expect(forkSeconds).toBe(9765);
}
},
);
@@ -773,7 +773,7 @@ describe('Hosted real-process gates', () => {
// that healthy sum still fits the fork. A larger ceiling lets failsafe
// kill the fork mid-run, and the classes left unrun never report, failing
// the gate undiagnosably (#13780).
const hostedForkHealthySeconds = 8923;
const hostedForkHealthySeconds = 9165;
const hostedForkSeconds = () =>
Number(
read('packages/managed-agent-server/pom.xml')