# Service Lifecycle: Election, Restart, and Reconnect Status: in progress Incident: [#36688](https://github.com/anomalyco/opencode/issues/36688) ## Summary The managed V2 service keeps its current update policy: the background updater may install a new package, but only a freshly launched TUI activates that update after finding an older running service. Existing TUIs never replace a service; they only reconnect. The restart path changes in three places: 1. A process-held OS lock, not the HTTP port or registration file, elects exactly one server owner for its lifetime. 2. The elected process binds and registers a minimal lifecycle surface before it initializes the application, so clients can distinguish a slow winner from an absent server. 3. TUIs rediscover and reconnect indefinitely. Transport loss is never a terminal error by itself. Several clients may spawn small contenders during a restart. This is safe and intentional: one contender acquires the lock and initializes, while every loser exits before expensive server boot. The design does not require clients to agree on a single initiator. This proposal does not introduce a supervisor process, warm candidate server, protocol negotiation, idle background restart, or general execution-recovery framework. ## Architecture at a Glance ```text ╭───────────────────╮ │ CLI ServiceConfig │ ╰─────────┬─────────╯ │ ▼ ╭──────────────────────╮ │ CLI ServerConnection │ ╰───────────┬──────────╯ ╭──────────────────╰───────────────────╮ ▼ ▼ ╭──────────────────────────╮ ╭─────────────────────────╮ │ Client Service lifecycle │ │ CLI runPromiseWith seam │ ╰─────────────┬────────────╯ ╰─────────────┬───────────╯ ╰─────╮ │ ▼ ▼ ╭────────────────────────────╮ ╭─────────────╮ │ Background service process │ │ TUI / Solid │ ╰──────────────┬─────────────╯ ╰──────┬──────╯ │ │ ╰────────────◀────────────────────╯ ╭───────────────────────╮ │ Server HTTP transport │ ╰───────────┬───────────╯ │ ▼ ╭──────────────────╮ │ Core application │ ╰──────────────────╯ ``` | Owner | Responsibility | | ------------------------------------------------ | --------------------------------------------------------------------------------------------------- | | `packages/client/src/effect/service.ts` | Effect-native discovery, start, and stop lifecycle operations | | `packages/cli/src/services/service-config.ts` | CLI registration path, installed version, and daemon command | | `packages/cli/src/services/server-connection.ts` | Resolve an endpoint and, only for the shared service, grouped reconnect and restart Effects | | `packages/cli/src/server-process.ts` | Daemon election, registration, and server process boot | | `packages/server/src/process.ts` | HTTP lifecycle shell and application transport | | `packages/core` | Application behavior behind the transport | | CLI default handler | Convert lifecycle Effects with the outer `FileSystem` context and pass grouped Promise capabilities | | `packages/tui` Solid client context | Own event-stream reconnect, endpoint replacement, status, and user-triggered restart UI | ## Implementation Status | Area | State | | ------------------------- | --------------------------------------------------------------------- | | Lifetime ownership | Implemented on this branch with a scoped OS lock | | Contender behavior | Implemented; losers exit before the server module is imported | | Registration repair | Implemented; the owner reasserts deleted or corrupt discovery | | Channel isolation | Implemented with no-clobber migration for legacy preview discovery | | Client startup waiting | Implemented; slow winners are not killed and waiting is indefinite | | Lifecycle shell | Implemented; the owner binds and registers before application boot | | Failed-state latching | Implemented; deterministic boot failure stays bound and actionable | | Recovery diagnostics | Implemented; the TUI shows status instead of transport internals | | Cross-platform validation | macOS runtime verified; Linux and Windows run in the unit-test matrix | ## Context The V2 CLI runs a shared managed service that owns Sessions, location graphs, plugins, permissions, and tool execution. The service updater can replace the installed package while the current process continues running the old image. A later TUI launch then detects the version mismatch and replaces the service. Incident #36688 showed four failures in that replacement path: - Multiple TUIs spawned heavyweight server contenders. - A winner remained unobservable while it cold-booted, so another wave treated it as absent and displaced it. - A fresh TUI exhausted its reconnect budget and crashed with an unhandled transport defect. - A losing contender remained alive and consumed about 1 GB of RSS. The `origin/v2` baseline serializes service startup with `EffectFlock`. A contender acquires a three-second heartbeat lease, checks whether another service became discoverable, and only the winner crosses the application-boot boundary. This already prevents simultaneous heavy boots and makes startup losers exit. The lease is released immediately after registration, however, so it is not lifetime ownership. Registration then reverts to last-writer-wins authority: a deleted or corrupt registration can admit a second boot, a displaced server terminates itself through its 10-second registration self-check, and a stalled lease holder can be displaced after the three-second service staleness timeout. `Flock` and `EffectFlock` live in `packages/core/src/util` and are also used for config writes, MCP auth, npm installs, and repository caching. Despite the name, the primitive is an atomic-mkdir lease with heartbeat and staleness takeover, not an OS-held lock. It remains appropriate for bounded critical sections, including today's startup fence, but is not lifetime service ownership. The current implementation also mixes three different concepts: - **Ownership:** which process is allowed to be the managed server. - **Discovery:** where clients can reach that process. - **Lifecycle:** whether that process is starting, ready, stopping, or failed. This design gives each concept one authority. ```definitions [ { "term": "Owner", "definition": "The one process holding the process-held OS service lock." }, { "term": "Contender", "definition": "A small serve process attempting to acquire the service lock. It must not initialize the application before winning." }, { "term": "Registration", "definition": "An atomic discovery record containing the elected owner's identity and endpoint. Registration never grants ownership." }, { "term": "Lifecycle shell", "definition": "The minimal HTTP surface bound by the elected process before application initialization. It serves health and retryable startup responses." }, { "term": "Application", "definition": "The full server routes and global or location-scoped modules used for normal OpenCode work." } ] ``` ## Goals - At most one process initializes and serves the managed application. - Losing contenders exit before database, route, plugin, MCP, or location boot. - A slow winner becomes observable before expensive initialization. - Existing and freshly launched TUIs survive retryable service unavailability. - Reconnect follows service state instead of displaying retry counts or raw transport failures. - Version-mismatch replacement remains triggered by a fresh TUI launch. - A stale or malformed registration cannot create a second owner. - An unresponsive owner is never killed automatically by an arbitrary TUI. - Every spawned contender has a bounded path to ownership or exit. ## Non-goals - Restarting automatically when a background update finds an idle window. - Running old and candidate application servers concurrently. - Adding a permanent steward, proxy, or supervisor process. - Zero-downtime worker handoff or automatic rollback. - Application protocol negotiation or automatic TUI self-restart. - General hard-crash recovery for active Sessions. - Defining recovery semantics for provider attempts, tools, shells, sub-agents, permissions, questions, or background jobs. - Automatically killing a frozen owner. - Bounding concurrent location cold boots after clients reconnect. - Multi-machine or clustered service placement. ## Invariants 1. **The service lock is ownership.** Exactly one process may hold the OS lock for one installation channel and service profile. 2. **Ownership precedes boot.** A contender performs no expensive application initialization before it acquires the lock. 3. **Ownership lasts for the process lifetime.** The owner holds an open lock handle until the managed server exits. The OS releases it on process death without a cleanup callback. 4. **The port is transport, not election.** The owner may select a dynamic port after acquiring the lock. 5. **Registration is discovery, not election.** Deleting, corrupting, or replacing registration does not invalidate a live owner's lock. 6. **Only a fresh launch enforces package version.** Existing TUIs reconnect to the current owner without initiating version replacement. 7. **Transport loss is retryable.** It never terminates a TUI without a separate diagnosed, non-retryable cause. 8. **Clients do not kill an unresponsive owner automatically.** Destructive recovery requires the explicit `service restart` command. 9. **Lifecycle does not promise execution semantics.** Graceful replacement invokes Session suspension and resumption hooks, but tool-level continuity belongs to a separate design. ## System Model ```text ╭───────────────────────╮ ╭──────────────────────────────╮ │ Fresh or existing TUI │ │ Process-held OS service lock │ ╰───────────┬───────────╯ ╰───────────────┬──────────────╯ ╰─────┬ normal requests observe ───────────────────────╮ │ │ discover │ ├──╯ authorizes one owner ▼ │ ▼ ╭───────────────────╮ │ ╭─────────────────╮ │ Registration file │ │ │ Lifecycle shell │ ╰───────────────────╯ │ ╰────────┬────────╯ │ │ ├────────────────────────╯ ▼ ╭──────────────────────╮ │ OpenCode application │ ╰──────────────────────╯ ``` The lifecycle shell and application run in the same process. The distinction is initialization order and responsibility, not process topology. ## Service Status The server reports one small status value: ```typescript type ServiceStatus = | { type: "starting" } | { type: "ready" } | { type: "stopping" targetVersion?: string } | { type: "failed" message: string action: string } ``` The client adds only the discovery states needed by callers: ```typescript type Status = { type: "missing" } | { type: "unreachable" } | { type: "unresponsive" } | ServiceStatus ``` The health response retains the existing fields for old clients and adds the status discriminant: ```typescript type ServiceHealth = { healthy: true version: string pid: number instanceID: string status: ServiceStatus } ``` `healthy: true` means the registered lifecycle shell is responding and its identity matches registration. New clients use `status.type === "ready"` as the application-readiness signal. During `starting` or `stopping`, application requests are not held in memory. They receive an immediate retryable response: ```http HTTP/1.1 503 Service Unavailable Retry-After: 1 Content-Type: application/json {"code":"service_starting"} ``` `stopping` uses `service_stopping`. A failed application boot uses `service_failed` and includes a safe diagnostic message. A failed owner remains bound and keeps holding the service lock. Exiting on failure would let every waiting client's `ensureRunning` loop elect a new contender that repeats the same heavy failing boot, so staying bound turns a deterministic boot failure into one observable `failed` state instead of a client-driven respawn loop. Recovery still works: a fresh launch observes the failed instance through the stop path, and explicit `service restart` replaces it. ## Registration Contract Registration contains only discovery identity: ```typescript type ServiceRegistration = { schema: 1 instanceID: string version: string url: string pid: number } ``` Authentication continues to use the existing private service credential storage. The registration schema does not change that policy. The owner writes registration only after the lifecycle shell has bound: 1. Bind the lifecycle shell. 2. Write a temporary registration file with mode `0600`. 3. Atomically rename it over the old registration. 4. Serve lifecycle health as `starting`. On shutdown, the owner removes registration only if the current file still has its `instanceID`. An old finalizer can never remove a successor's registration. While running, the owner periodically asserts its registration. Because the lock guarantees exactly one live owner, any registration that does not name the owner is stale or corrupt, and the owner rewrites it. A deleted or clobbered registration therefore heals within one assertion interval instead of leaving clients waiting on absent discovery. This inverts today's self-check loop, which terminates the displaced process instead of repairing discovery. Legacy registration shapes are decoded by a compatibility adapter. The new domain type does not make fields optional to represent old formats. ## Election This design promotes today's startup fence into lifetime ownership. Last-writer-wins registration is replaced by a process-held OS lock that is acquired before any expensive boot work and held for the entire service lifetime. A heartbeat-and-staleness lease, including the existing `Flock` utility, is not sufficient for service ownership: the service configures a three-second stale timeout, after which its lock can be broken and recreated. An event-loop stall, a suspended machine, or a debugger pause can therefore make a live owner appear stale and allow a contender to displace it. Service ownership requires a process-held OS lock: `flock` on Unix and an exclusively bound named pipe on Windows. It cannot be broken because a heartbeat exceeded a timeout. Process death releases the lock through the OS. Neither Bun nor Node exposes `flock` directly, the existing `Flock` utility is an mkdir-plus-heartbeat lease rather than an OS-held lock, and the common lockfile packages are staleness-based leases as well. The platform layer uses `bun:ffi` to call `flock` on POSIX and Node's named-pipe server support on Windows, where Bun FFI is not available on every shipped architecture. It lives alongside the existing utility in `packages/core/src/util`. This primitive is the foundation of the design, so the delivery sequence spikes it first. ```text Contender Lock Lifecycle Application │ │ │ │ ├─ try acquire ───▶ │ │ │ │ │ │ ╭─ alt: lock held ────────────────────────────────────────────────╮ │ │ │ │ │ │ │ ◀─ busy ──────────┤ │ │ │ │ │ │ │ │ │ │ ├─────────╮ │ │ │ │ │ │ exit │ │ │ │ │ │ ◀─────────╯ │ │ │ │ │ │ │ │ │ │ ├─ else: lock acquired ───────────────────────────────────────────┤ │ │ │ │ │ │ │ ◀─ owner ─────────┤ │ │ │ │ │ │ │ │ │ │ ├─ bind, register, starting ────────▶ │ │ │ │ │ │ │ │ │ ├─ initialize ──────────────────────────────────────────────▶ │ │ │ │ │ │ │ │╭─ alt: boot succeeds ──────────────────────────────────────────╮│ ││ │ │ │ │ ││ ││ │ │ ◀─ ready ───────────────┤ ││ ││ │ │ │ │ ││ │├─ else: boot fails ────────────────────────────────────────────┤│ ││ │ │ │ │ ││ ││ │ │ ◀─ failed, stay bound ──┤ ││ ││ │ │ │ │ ││ │╰───────────────────────────────────────────────────────────────╯│ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────╯ │ │ │ │ ``` Lock acquisition by a contender is nonblocking or tightly bounded. A loser must exit before constructing application routes or importing startup-heavy modules. Several clients may spawn contenders concurrently. The design guarantees one heavy winner, not one process spawn. If the winner crashes during startup, the OS releases the lock and a later client retry starts another election. The lock is scoped by installation channel and service profile. Local, preview, and stable installations cannot displace one another. ## Update Activation Background update behavior remains unchanged: 1. The running service checks for an update. 2. The updater installs the package in the background. 3. The running process continues using its existing process image. 4. No idle check or automatic restart occurs. A fresh TUI launch activates the installed update: 1. Read registration and authenticate the responding service. 2. If its package version matches the fresh client, attach normally. 3. If the version differs, request graceful stop of that exact registered instance using the existing authenticated stop path. 4. Re-check instance identity before every signal or escalation in that path. 5. Wait for the old process to exit and release the service lock. 6. Call `ensureRunning` until a compatible service becomes ready. Concurrent fresh launchers may all observe the same old instance. Stopping that exact instance must be idempotent. Once registration names a different instance, a stale launcher stops signaling and returns to discovery. No durable restart-transition record is introduced. The initiating fresh TUI already knows the source and target versions and can display its update preflight. Existing TUIs may display `Updating...` if they observed `stopping`; otherwise `Waiting for background service...` is the honest fallback. ## Fresh Launch Versus Reconnect Fresh launch and reconnect deliberately have different version policies: ```typescript type ManagedConnection = | { type: "launch" requiredVersion: string } | { type: "reconnect" } ``` - `launch` requires the installed package version and may activate replacement. - `reconnect` accepts the current owner and never activates replacement. This preserves today's permissive reconnect behavior. Explicit application protocol negotiation and automatic TUI re-exec remain follow-ups. ## Client Reconnect Fresh and existing TUIs use the same status loop after startup: 1. Read registration on every attempt. Do not retry a stale URL indefinitely. 2. If registration is absent, call `ensureRunning` and continue waiting. 3. If registration is unreachable, call `ensureRunning`. A live owner prevents contenders from acquiring the lock; a dead owner does not. 4. If status is `starting` or `stopping`, wait. 5. If status is `failed`, show its actionable message. 6. If status is `ready`, rebuild HTTP and event-stream clients for the new endpoint and perform authoritative state reconciliation. Retry cadence is internal policy. Retry counts are telemetry, not user-facing state. The TUI waits until the service is ready or the user exits. Transport failures are handled at the TUI run boundary. A raw client transport error or Effect defect must not escape to the terminal. Hard exit is reserved for diagnosed causes such as invalid local configuration, failed authentication, or a foreign process occupying an explicitly configured port. The UI derives text from status: | Status | User-facing state | | ------------------------ | ----------------------------------- | | No registration | `Starting background service...` | | Registration unreachable | `Waiting for background service...` | | `starting` | `Starting OpenCode vX...` | | `stopping` | `Updating to vX...` | | `failed` | Actionable failure message | | `ready` | Normal TUI | ## Graceful Session Continuity Version-mismatch replacement uses the existing graceful Session suspension and resumption hooks: 1. The old server snapshots active Session IDs during graceful teardown. 2. The successor schedules those Sessions for continuation. 3. The runner reloads durable Session history before continuing. This lifecycle design does not define what an interrupted physical provider attempt or tool invocation means. It does not promise that external side effects did not occur, replay the exact interrupted tool, preserve an in-memory form, or recover process-local background work. Those concerns require a separate execution-continuity design covering tools, shells, sub-agents, permissions, questions, provider attempts, and hard-crash recovery. ## Unresponsive Owner An unreachable registration does not prove that the owner is dead. A contender attempts the service lock: - If the lock is free, the contender starts a replacement. - If the lock is held, the contender exits and the client keeps waiting. After a bounded diagnostic threshold, the client may show: ```text The background service owns the service lock but is not responding. Run `opencode service restart` to recover it. ``` Only explicit `service restart` may perform destructive recovery. It verifies the complete registration and process instance before signaling, waits for graceful exit, re-checks identity before escalation, and refuses to kill a process it cannot positively identify. Automatic frozen-owner recovery is deferred. ## Failure Walkthroughs ### Update with open TUIs 1. The old service installs vNext but keeps running. 2. A fresh vNext TUI finds the healthy vOld service and requests graceful stop. 3. The old service reports `stopping`, suspends active Sessions, and exits. 4. Open TUIs enter their indefinite status loops. 5. One or more clients spawn contenders. 6. One contender acquires the service lock. Losers exit before heavy boot. 7. The winner binds and registers the lifecycle shell as `starting`. 8. Clients stop spawning and wait on the observable winner. 9. The winner initializes the application and reports `ready`. 10. TUIs rebuild clients, reconcile state, and resume. ### Server crashes while ready 1. The endpoint becomes unreachable and registration may remain stale. 2. Clients call `ensureRunning`. 3. Process death has released the service lock. 4. One contender wins, replaces registration, and starts normally. 5. Detailed active-execution recovery is outside this design. ### Winner crashes during startup 1. Clients observed `starting` and remain alive. 2. Process death releases the service lock. 3. A later reconnect attempt starts another election. 4. One new contender wins; all other contenders exit. ### Registration is deleted while the owner is healthy 1. Clients may call `ensureRunning` because discovery is absent. 2. Every contender fails to acquire the owner's lock and exits. 3. No second application initializes. 4. The owner's next registration assertion republishes discovery. ### Owner is alive but unresponsive 1. Health fails, but the process still holds the service lock. 2. Contenders fail lock acquisition and exit. 3. Clients wait and eventually show explicit recovery guidance. 4. No TUI kills the owner automatically. ## TDD Verification Implementation should proceed test-first with real subprocesses and real locks. Mocks cannot establish process death, lock release, loser cleanup, or port behavior. ### Election tests | Scenario | Required result | | ----------------------------------------------------- | ------------------------------------------------------- | | Ten contenders start simultaneously | Exactly one crosses the application-boot boundary | | Winner pauses after lock acquisition | No loser initializes or remains alive | | Winner event loop pauses beyond the old stale timeout | Ownership is not displaced | | Winner crashes before bind | Lock releases; a later attempt wins | | Winner crashes after bind but before registration | Lock releases; a later attempt replaces stale discovery | | Registration is deleted while owner runs | No second owner initializes | | Registration is malformed | Lock still prevents a second owner | | Registration names a dead PID | New contender can acquire the released lock | | Two installation channels start | Each elects an independent owner | | Explicit configured port is foreign-owned | Fail diagnostically; do not kill the foreign process | The fixture records a marker immediately before application initialization. The tests assert that only one process writes that marker and that every loser exits within a bounded interval. The harness should also assert that a loser's peak RSS stays an order of magnitude below an application boot, since import weight was the observed incident cost. ### Lifecycle tests | Scenario | Required result | | ----------------------------------------------- | ---------------------------------------------------------------- | | Winner owns lock but application boot is paused | Health reports `starting` | | Application request arrives during startup | Immediate retryable `503` | | Application becomes ready | Status changes once from `starting` to `ready` | | Graceful replacement begins | Status reports `stopping` before disconnect | | Application initialization fails | Actionable `failed` status; owner stays bound and holds the lock | | Registration is deleted while owner runs | Owner republishes it within one assertion interval | | Owner exits | Registration is removed only if it still names that owner | ### Update tests | Scenario | Required result | | -------------------------------------- | -------------------------------------------------------- | | Background update installs vNext | Running vOld service does not restart | | Fresh vNext launch finds vOld | Exact old instance stops; vNext eventually becomes ready | | Two fresh vNext launches race | One heavy successor; both clients attach | | Existing vOld TUI reconnects to vNext | It never requests replacement | | Stale launcher observes a new instance | It does not signal the new instance | ### Reconnect tests | Scenario | Required result | | --------------------------------------------------- | -------------------------------------------------- | | Endpoint disappears and changes port | TUI rediscovers and rebuilds clients | | Service remains unavailable beyond old retry budget | TUI remains alive | | Event stream reconnects | Client performs authoritative state reconciliation | | Transport returns an unexpected defect | TUI formats it; no raw stack escapes | | Owner remains unresponsive | TUI waits and shows explicit restart guidance | ## Delivery Sequence 1. **Spike the lock primitive.** Prove a nonblocking, process-held OS lock under Bun on macOS, Linux, and Windows (`bun:ffi` to `flock` on POSIX and a named pipe on Windows), including release on hard kill and behavior across containers and network filesystems used in CI. 2. **Expand the subprocess test harness.** Begin from the baseline two-contender test and cover ten contenders, lock release on crash, a paused winner, deleted or corrupt registration, and bounded loser exit before changing ownership. 3. **Contain client failure.** Make transport loss nonterminal, rediscover on every cycle, and format unexpected failures at the TUI boundary. 4. **Promote the startup fence to process-held ownership.** Preserve the existing pre-boot acquisition seam, replace its lease with the OS lock, hold it until process exit, and invert the registration self-check from self-termination to reassertion. 5. **Bind the lifecycle shell first.** Publish registration and `starting`, return retryable `503` for application requests, then initialize the app. The health contract change is public API: regenerate clients from `packages/client` with `bun run generate`. 6. **Codify launch versus reconnect.** Fresh launch enforces installed version; reconnect never activates replacement. 7. **Integrate graceful replacement.** Preserve current background-install and fresh-launch activation behavior while invoking Session continuity hooks. 8. **Harden explicit recovery.** Verify exact process identity during explicit `service restart`; never automatically kill an unresponsive owner. 9. **Run the full multi-process suite.** Include repeated restart cycles and assert that no contender or child process remains afterward. ## Acceptance Criteria - Ten concurrent restart observers produce one application initialization. - No losing contender survives or builds a location graph. - A 30-second application boot remains continuously observable as `starting`. - A TUI remains alive through a service outage longer than the previous retry budget. - A service endpoint change does not require restarting an existing TUI. - Background installation alone does not restart the service. - A fresh mismatched TUI eventually attaches to the installed service version. - Existing reconnecting TUIs never replace the current owner. - Registration corruption cannot produce two owners. - A deleted registration heals without restarting the owner or any client. - An unresponsive owner is not killed without an explicit recovery command. - Raw transport defects never escape to the terminal. ## Follow-ups - Idle background update activation with an admission fence. - Application protocol compatibility and automatic local TUI re-exec. - Durable execution recovery for provider attempts and tools. - Shell, sub-agent, permission, question, and background-job continuity. - Automatic recovery for a positively identified frozen owner. - Cold-boot concurrency limits and interaction-prioritized location loading. - A steward or socket-handoff architecture if zero-downtime replacement becomes a real requirement.