fix(slice-22): teardown compose before rig deletion on service_boot_failed (OPR.0.3.2.22 Bug 2 follow-up)

Closes a downstream-orphan finding from the per-commit review on
a29c7883 (guard verdict qitem-20260601051133-c9e23b07). The original
Bug 2 commit rolled back the rig record on prelaunch-hook failure to
free the spec name for clean retry. But ServiceOrchestrator.boot can
already have started compose containers before failing during
status/wait — and PodRigInstantiator's deleteRig immediately
cascade-removes the rig_services row, taking the normal env/teardown
handle with it. Result: orphaned compose resources with no
rig-scoped path to teardown them.

The fix (bootstrap-orchestrator.ts):

- In buildServicePrelaunchHook's failure path, call
  serviceOrch.teardown(rigId) best-effort before returning
  service_boot_failed. Teardown errors are swallowed so they cannot
  mask the load-bearing boot-failure return; manual `docker compose
  down` remains available with the compose file path.
- PodRigInstantiator's deleteRig at the same code path (commit
  a29c7883) is unchanged — the rig-record rollback contract holds.

Test addition (services-up.test.ts):

- "service boot failure tears down compose before rig record is
  deleted (no compose orphan)" — vi.spyOn the teardown method on
  ServiceOrchestrator, force boot to return ok:false, run the
  bootstrap, assert teardown was called exactly once AND the rig
  record is still rolled back (findRigsByName returns empty).
  Discriminator-flip pattern: with the new teardown call removed,
  the spy assertion fails ("expected to have been called 1 times,
  but was called 0 times") while the rig-rollback assertion still
  passes — proves the test catches the specific compose-orphan
  regression class without false positives.

Verification:
- services-up suite: 5/5 PASS (4 existing + 1 new discriminator).
- Full packages/daemon test/ suite: 271 files, ZERO failures
  (one transient cold-start timeout on a single file in the first
  pass cleared on retry, pre-existing baseline flakiness — not
  introduced by this change).
- tsc --noEmit on @openrig/daemon EXIT=0.
- git diff --cached --stat: 2 files only (bootstrap-orchestrator.ts
  +15, services-up.test.ts +68). Bug 3 WIP files
  (routes/info.ts, server.ts mount line) are unstaged and not in
  this commit per banked staged-diff scope check.

Branch: slice-22-rig-up-paper-cut-fix-round off LOCAL main 0c9e63f1.
Chain: 50488427 → a29c7883 → this commit. LOCAL only, NO push.
This commit is contained in:
mvschwarz committed 2026-05-31 22:14:28 -07:00
1 parent a29c78834f
commit ad6a522b69
2 files changed
+83

No files matched your search

@@ -777,6 +777,21 @@ export class BootstrapOrchestrator {
status: "failed",
detail: { code: bootResult.code, error: bootResult.error, receipt: bootResult.receipt },
});
// OPR.0.3.2.22 Bug 2 follow-up — serviceOrch.boot can already have
// started compose resources before failing during status/wait. The
// PodRigInstantiator will delete the rig record next, which cascades
// away rig_services and the normal teardown handle — so any
// already-started compose containers would orphan. Tear them down
// here best-effort while the rig handle still exists. Teardown
// errors are swallowed so they cannot mask the boot failure that
// is the load-bearing return.
try {
await serviceOrch.teardown(rigId);
} catch {
// Best-effort. If teardown also fails, the boot-failure error
// is what the operator needs; manual `docker compose down`
// remains available with the compose file path from the spec.
}
return { ok: false, code: "service_boot_failed", message: `Service boot failed: ${bootResult.error}` };
}
+68
View File
@@ -196,6 +196,74 @@ pods:
expect(serviceStage!.status).toBe("failed");
});
// OPR.0.3.2.22 Bug 2 follow-up (guard BLOCKING on a29c7883) — when
// boot returns ok:false after compose may have started (e.g.
// wait_timeout), the prelaunch-hook MUST call teardown best-effort
// before the rig record is deleted. Otherwise compose resources
// orphan: rig_services cascade-removes with the rig, taking the
// normal teardown handle with it. Discriminator: assert teardown
// was called on boot failure AND the rig record is still removed.
it("service boot failure tears down compose before rig record is deleted (no compose orphan)", async () => {
const composePath = path.join(tmpDir, "docker-compose.yml");
fs.writeFileSync(composePath, "version: '3'\nservices:\n vault:\n image: vault:1.15\n");
const specPath = writeSpec(`
version: "0.2"
name: teardown-on-boot-failure-rig
services:
kind: compose
compose_file: docker-compose.yml
wait_for:
- url: http://localhost:8200/v1/sys/health
pods:
- id: infra
label: Infra
members:
- id: server
runtime: terminal
agent_ref: "builtin:terminal"
profile: none
cwd: /tmp
edges: []
`);
const composeAdapter = new ComposeServicesAdapter(mockExec());
const rigRepo = new RigRepository(db);
const serviceOrch = new ServiceOrchestrator({ rigRepo, composeAdapter });
vi.spyOn(serviceOrch, "boot").mockResolvedValue({
ok: false,
code: "wait_timeout",
error: "Service wait targets not healthy after 30s: HTTP probe failed: http://localhost:8200/v1/sys/health",
receipt: {
kind: "compose",
composeFile: "docker-compose.yml",
projectName: "teardown-on-boot-failure-rig",
services: [{ name: "vault", status: "running", health: "starting" }],
waitFor: [{ target: { url: "http://localhost:8200/v1/sys/health" }, status: "unhealthy", detail: "HTTP probe failed" }],
capturedAt: new Date().toISOString(),
},
});
const teardownSpy = vi.spyOn(serviceOrch, "teardown").mockResolvedValue({ ok: true });
const setup = createTestApp(db);
(setup.bootstrapOrchestrator as any).deps.serviceOrchestrator = serviceOrch;
(setup.bootstrapOrchestrator as any).deps.rigRepo = rigRepo;
const result = await setup.bootstrapOrchestrator.bootstrap({
sourceRef: specPath,
mode: "apply",
});
expect(result.status).toBe("failed");
// Discriminator: teardown MUST be called when boot fails — that is
// the bug guard caught (compose orphan downstream of rig deletion).
expect(teardownSpy, "expected serviceOrch.teardown to be called best-effort on boot failure").toHaveBeenCalledTimes(1);
// And the rig record is still rolled back (Bug 2 original contract).
const orphans = rigRepo.findRigsByName("teardown-on-boot-failure-rig");
expect(orphans).toHaveLength(0);
});
it("plan mode does not boot services", async () => {
const composePath = path.join(tmpDir, "docker-compose.yml");
fs.writeFileSync(composePath, "version: '3'\nservices:\n vault:\n image: vault:1.15\n");