mirror of
https://github.com/mvschwarz/openrig.git
synced 2026-11-08 17:31:56 -10:00
fix(slice-22): teardown compose before rig deletion on service_boot_failed (OPR.0.3.2.22 Bug 2 follow-up)
Closes a downstream-orphan finding from the per-commit review ona29c7883(guard verdict qitem-20260601051133-c9e23b07). The original Bug 2 commit rolled back the rig record on prelaunch-hook failure to free the spec name for clean retry. But ServiceOrchestrator.boot can already have started compose containers before failing during status/wait — and PodRigInstantiator's deleteRig immediately cascade-removes the rig_services row, taking the normal env/teardown handle with it. Result: orphaned compose resources with no rig-scoped path to teardown them. The fix (bootstrap-orchestrator.ts): - In buildServicePrelaunchHook's failure path, call serviceOrch.teardown(rigId) best-effort before returning service_boot_failed. Teardown errors are swallowed so they cannot mask the load-bearing boot-failure return; manual `docker compose down` remains available with the compose file path. - PodRigInstantiator's deleteRig at the same code path (commita29c7883) is unchanged — the rig-record rollback contract holds. Test addition (services-up.test.ts): - "service boot failure tears down compose before rig record is deleted (no compose orphan)" — vi.spyOn the teardown method on ServiceOrchestrator, force boot to return ok:false, run the bootstrap, assert teardown was called exactly once AND the rig record is still rolled back (findRigsByName returns empty). Discriminator-flip pattern: with the new teardown call removed, the spy assertion fails ("expected to have been called 1 times, but was called 0 times") while the rig-rollback assertion still passes — proves the test catches the specific compose-orphan regression class without false positives. Verification: - services-up suite: 5/5 PASS (4 existing + 1 new discriminator). - Full packages/daemon test/ suite: 271 files, ZERO failures (one transient cold-start timeout on a single file in the first pass cleared on retry, pre-existing baseline flakiness — not introduced by this change). - tsc --noEmit on @openrig/daemon EXIT=0. - git diff --cached --stat: 2 files only (bootstrap-orchestrator.ts +15, services-up.test.ts +68). Bug 3 WIP files (routes/info.ts, server.ts mount line) are unstaged and not in this commit per banked staged-diff scope check. Branch: slice-22-rig-up-paper-cut-fix-round off LOCAL main0c9e63f1. Chain:50488427→a29c7883→ this commit. LOCAL only, NO push.
This commit is contained in:
1 parent
a29c78834f
commit
ad6a522b69
2 files changed
+83
No files matched your search
@@ -777,6 +777,21 @@ export class BootstrapOrchestrator {
|
||||
status: "failed",
|
||||
detail: { code: bootResult.code, error: bootResult.error, receipt: bootResult.receipt },
|
||||
});
|
||||
// OPR.0.3.2.22 Bug 2 follow-up — serviceOrch.boot can already have
|
||||
// started compose resources before failing during status/wait. The
|
||||
// PodRigInstantiator will delete the rig record next, which cascades
|
||||
// away rig_services and the normal teardown handle — so any
|
||||
// already-started compose containers would orphan. Tear them down
|
||||
// here best-effort while the rig handle still exists. Teardown
|
||||
// errors are swallowed so they cannot mask the boot failure that
|
||||
// is the load-bearing return.
|
||||
try {
|
||||
await serviceOrch.teardown(rigId);
|
||||
} catch {
|
||||
// Best-effort. If teardown also fails, the boot-failure error
|
||||
// is what the operator needs; manual `docker compose down`
|
||||
// remains available with the compose file path from the spec.
|
||||
}
|
||||
return { ok: false, code: "service_boot_failed", message: `Service boot failed: ${bootResult.error}` };
|
||||
}
|
||||
|
||||
|
||||
@@ -196,6 +196,74 @@ pods:
|
||||
expect(serviceStage!.status).toBe("failed");
|
||||
});
|
||||
|
||||
// OPR.0.3.2.22 Bug 2 follow-up (guard BLOCKING on a29c7883) — when
|
||||
// boot returns ok:false after compose may have started (e.g.
|
||||
// wait_timeout), the prelaunch-hook MUST call teardown best-effort
|
||||
// before the rig record is deleted. Otherwise compose resources
|
||||
// orphan: rig_services cascade-removes with the rig, taking the
|
||||
// normal teardown handle with it. Discriminator: assert teardown
|
||||
// was called on boot failure AND the rig record is still removed.
|
||||
it("service boot failure tears down compose before rig record is deleted (no compose orphan)", async () => {
|
||||
const composePath = path.join(tmpDir, "docker-compose.yml");
|
||||
fs.writeFileSync(composePath, "version: '3'\nservices:\n vault:\n image: vault:1.15\n");
|
||||
|
||||
const specPath = writeSpec(`
|
||||
version: "0.2"
|
||||
name: teardown-on-boot-failure-rig
|
||||
services:
|
||||
kind: compose
|
||||
compose_file: docker-compose.yml
|
||||
wait_for:
|
||||
- url: http://localhost:8200/v1/sys/health
|
||||
pods:
|
||||
- id: infra
|
||||
label: Infra
|
||||
members:
|
||||
- id: server
|
||||
runtime: terminal
|
||||
agent_ref: "builtin:terminal"
|
||||
profile: none
|
||||
cwd: /tmp
|
||||
edges: []
|
||||
`);
|
||||
|
||||
const composeAdapter = new ComposeServicesAdapter(mockExec());
|
||||
const rigRepo = new RigRepository(db);
|
||||
const serviceOrch = new ServiceOrchestrator({ rigRepo, composeAdapter });
|
||||
|
||||
vi.spyOn(serviceOrch, "boot").mockResolvedValue({
|
||||
ok: false,
|
||||
code: "wait_timeout",
|
||||
error: "Service wait targets not healthy after 30s: HTTP probe failed: http://localhost:8200/v1/sys/health",
|
||||
receipt: {
|
||||
kind: "compose",
|
||||
composeFile: "docker-compose.yml",
|
||||
projectName: "teardown-on-boot-failure-rig",
|
||||
services: [{ name: "vault", status: "running", health: "starting" }],
|
||||
waitFor: [{ target: { url: "http://localhost:8200/v1/sys/health" }, status: "unhealthy", detail: "HTTP probe failed" }],
|
||||
capturedAt: new Date().toISOString(),
|
||||
},
|
||||
});
|
||||
const teardownSpy = vi.spyOn(serviceOrch, "teardown").mockResolvedValue({ ok: true });
|
||||
|
||||
const setup = createTestApp(db);
|
||||
(setup.bootstrapOrchestrator as any).deps.serviceOrchestrator = serviceOrch;
|
||||
(setup.bootstrapOrchestrator as any).deps.rigRepo = rigRepo;
|
||||
|
||||
const result = await setup.bootstrapOrchestrator.bootstrap({
|
||||
sourceRef: specPath,
|
||||
mode: "apply",
|
||||
});
|
||||
|
||||
expect(result.status).toBe("failed");
|
||||
// Discriminator: teardown MUST be called when boot fails — that is
|
||||
// the bug guard caught (compose orphan downstream of rig deletion).
|
||||
expect(teardownSpy, "expected serviceOrch.teardown to be called best-effort on boot failure").toHaveBeenCalledTimes(1);
|
||||
// And the rig record is still rolled back (Bug 2 original contract).
|
||||
const orphans = rigRepo.findRigsByName("teardown-on-boot-failure-rig");
|
||||
expect(orphans).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("plan mode does not boot services", async () => {
|
||||
const composePath = path.join(tmpDir, "docker-compose.yml");
|
||||
fs.writeFileSync(composePath, "version: '3'\nservices:\n vault:\n image: vault:1.15\n");
|
||||
|
||||
Reference in new issue
Block a user