feat(mcp): keep configured stdio MCP servers running across /new (CW-7430 G1)

Founder decision (2026-10-15): /new keeps configured MCP servers running. Until now every conversation boundary dropped the engine's MCP pool, so each /new killed and respawned every configured stdio server. A conversation boundary now hands live stdio processes to the next conversation's pool: - The old pool is taken with try_lock only, never awaited. When it is free, McpPool::retire_for_handoff marks it retired and moves out the ready connections of statically configured stdio servers. - Runtime-added and plugin servers are conversation capabilities and are closed. - HTTP/SSE connections are closed too. They reconnect cheaply and so pick up current OAuth credentials and [network] policy. A handed-off HTTP client would keep the policy it was built with and a token that a deferred /mcp logout had already removed. - A retired pool refuses every server: require_server fails and server_allowed is false. Any tool spec, child runtime, boot task or supervisor still holding the old pool therefore cannot reconnect and spawn a second copy of a handed-off server. - The new pool's session boot adopts the handoff under the lock it already takes, after its config reload and before it collects pending connects. A connection is adopted only if all of these are unchanged: - the workspace; - the MCP backend; - the server's own config identity. This is the per-server hash plus plugin authority that per-server reload already used, now a named function. - Adopted connections are restamped to the new catalog generation and stored through store_ready_connection, so the disallowed-tools ceiling and its bookkeeping apply. Anything not adopted is dropped, which stops its process. - The boot then dials only what is missing. Its existing McpSessionBoot events report adopted servers as connected. - Child engines share their parent's pool and never retire it. A busy pool (an MCP call still running) cannot hand off, and its servers restart as before. A forced reload never adopts. A handoff the boot did not take is released at the end of the sync. Docs: the connection lifecycle section of MCP.md and zh_hans/MCP.md now covers this. Only /new and same-provider, same-workspace picker switches hand off; /resume <id>, /load and provider switches start a new engine. The section also says that server-held state such as a browser session carries over, and that /mcp reload gives fresh servers. The derived web/data/agent-docs.generated.json is regenerated with the existing prebuild. An adversarial review ran four lenses over this diff and the companion #6978 commit, with a skeptic per finding. Twelve findings were refuted. Five findings on this diff were confirmed and are fixed here: - the OAuth credential carry-over, now avoided by the stdio-only scope; - the docs scope; - the stateful-server note; - two flags on an over-long comment edit, now cut to a minimal one. Two confirmed findings on the companion commit are fixed there. Not in this commit: the engine-respawn handoff, and the respawned engine's MCP event generation restarting at 0. Verification on one combined tree: b457a8affd plus this commit and the CW-7427 queue commit, which touch disjoint files. - cargo fmt --all -- --check: clean. - clippy -p codewhale-tui -p codewhale-localization --all-targets --locked with CI's flags: clean. - A scratch probe used real Node stdio servers and was removed before this commit: - Pool level: after retire and adopt, the same server process (same pid) answered tools/call, and the retired pool refused. - Engine A -> B (SyncSession with a new id, same config): the server was not restarted (spawn count 1, pid alive) and boot reported it connected. - B -> C with that server's args changed: it restarted (spawn count 2), the old pid was stopped, and the new one was alive. - nextest, focused on MCP, session boundary, sync session, supervisor, offline queue, persistence, queue, localization and toasts, plus the probe: 978 run, 977 passed, 1 failed. The failure is runtime_api::tests::mcp_management_selected_connection_and_retry_preserve_siblings. It comes from the runtime API management pool's config-hash check, which this diff does not touch, and it passed 5 of 5 isolated reruns. - npm test: 547 Node tests (540 passed, 7 skipped, 0 failed) and 772 Vitest passed. npm run check:web: pass after the regeneration. - Not run: the full suite (CI owns it), or an interactive TUI /new in a real terminal. Refs CW-7430 Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01NBKbhx19BRbZtvWK8rVKjz
2026-11-07 17:09:13 -10:00 · 2026-10-15 19:58:15 -07:00
1 parent b6ce70b12e
commit 9eff98f826
@@ -1004,6 +1004,7 @@ pub struct Engine {
    /// Abort the supervisor's reconnect handshakes when its pool is retired.
    mcp_supervisor_task: Option<tokio::task::AbortHandle>,
    mcp_boot_done: Option<tokio::sync::watch::Receiver<bool>>,
+    mcp_handoff: Option<crate::mcp::McpHandoff>,
    /// Generation owned by the currently installed boot receiver. Terminal
    /// cleanup is conditional on this exact value so an older pass can never
    /// clear a newer receiver.
@@ -2337,6 +2338,7 @@ impl Engine {
            mcp_supervisor_rx: None,
            mcp_supervisor_task: None,
            mcp_boot_done: None,
+            mcp_handoff: None,
            mcp_boot_generation: None,
            mcp_boot_task: None,
            mcp_event_generation: 0,
@@ -3109,8 +3111,17 @@ impl Engine {
        // authoritative schedule makes that token fail closed when drained.
        self.scheduled_goal_continuation = None;
        // Runtime-added MCP servers are conversation capabilities even when
-        // both conversations use the same workspace. The sync handler restarts
-        // the session boot once the new session is installed.
+        // both conversations use the same workspace, so the old pool retires.
+        // The sync handler starts the session boot once the new session is
+        // installed.
+        self.mcp_handoff = if self.child_host.is_none() {
+            self.mcp_pool
+                .as_ref()
+                .and_then(|pool| pool.try_lock().ok())
+                .map(|mut pool| pool.retire_for_handoff())
+        } else {
+            None
+        };
        self.drop_mcp_pool();
        // C02-17: cumulative usage belongs to the conversation that spent
        // it. The sync carries no prior total for the installed session, so
@@ -4037,6 +4048,7 @@ impl Engine {
                                crate::mcp::format_mcp_error_for_display(&error)
                            );
                        }
+                        self.mcp_handoff = None;
                        // SessionUpdated acknowledges the sync. A generic status
                        // would immediately cover the host's confirmed resume receipt.
                        self.emit_session_updated().await;
@@ -8213,6 +8225,10 @@ impl Engine {
    /// pool, skipped a session boot). Progress and the finished receipt flow
    /// as `Event::McpSessionBoot` updates under that generation.
    async fn start_mcp_session_boot(&mut self, refresh: McpConnectRefresh) -> anyhow::Result<u64> {
+        let handoff = self
+            .mcp_handoff
+            .take()
+            .filter(|_| matches!(refresh, McpConnectRefresh::IfChanged));
        if matches!(refresh, McpConnectRefresh::IfChanged)
            && (self.child_host.is_some() || !self.config.features.enabled(Feature::Mcp))
        {
@@ -8250,6 +8266,9 @@ impl Engine {
                // so a failed explicit reload leaves the live pool intact.
                McpConnectRefresh::Force => pool.force_reload_config_sources()?,
            }
+            if let Some(handoff) = handoff {
+                pool.adopt_handoff(handoff);
+            }
            let eager = pool.eager_boot_server_names(&requested);
            let (pending, auth_errors) = pool.collect_pending_connects(Some(&eager));
            (
@@ -3211,6 +3211,30 @@ pub struct McpPool {
    /// unconnected on-demand server (`lazy = true` or plugin-contributed) is
    /// one nobody has asked for, not one mid-handshake.
    connecting: HashSet<String>,
+    retired: bool,
+}
+
+pub(crate) struct McpHandoff {
+    workspace: Option<PathBuf>,
+    backend: McpBackend,
+    connections: Vec<(String, McpServerIdentity, McpConnection)>,
+}
+
+type McpServerIdentity = (u64, Option<crate::plugins::types::PluginAuthority>);
+
+fn mcp_server_identity(config: &McpConfig, name: &str) -> Option<McpServerIdentity> {
+    config.servers.get(name).map(|server| {
+        (
+            hash_mcp_config(&McpConfig {
+                timeouts: config.timeouts,
+                servers: HashMap::from([(name.to_string(), server.clone())]),
+            }),
+            server
+                .reviewed_plugin
+                .as_ref()
+                .map(|source| source.authority.clone()),
+        )
+    })
 }

 /// One server's cooldown: when to try again, and what to say until then.
@@ -3341,6 +3365,7 @@ impl McpPool {
            dynamic_servers: Arc::new(RwLock::new(HashMap::new())),
            needs_auth_servers: BTreeSet::new(),
            needs_auth_generation: 0,
+            retired: false,
        }
    }

@@ -3450,7 +3475,7 @@ impl McpPool {
    }

    fn server_allowed(&self, server: &str) -> bool {
-        !Self::server_denied_by(&self.disallowed_tools, server)
+        !self.retired && !Self::server_denied_by(&self.disallowed_tools, server)
    }

    pub(crate) fn tool_allowed(&self, name: &str) -> bool {
@@ -3458,6 +3483,10 @@ impl McpPool {
    }

    fn require_server(&self, server: &str) -> Result<()> {
+        anyhow::ensure!(
+            !self.retired,
+            "MCP server {server} belongs to a newer conversation; this conversation's MCP connections are closed"
+        );
        anyhow::ensure!(
            self.server_allowed(server),
            "Failed to find MCP server: {server}"
@@ -3618,28 +3647,14 @@ impl McpPool {
        if force {
            self.drop_all_connections("explicit config reload");
        } else {
-            let server_identity = |config: &McpConfig, name: &str| {
-                config.servers.get(name).map(|server| {
-                    (
-                        hash_mcp_config(&McpConfig {
-                            timeouts: config.timeouts,
-                            servers: HashMap::from([(name.to_string(), server.clone())]),
-                        }),
-                        server
-                            .reviewed_plugin
-                            .as_ref()
-                            .map(|source| source.authority.clone()),
-                    )
-                })
-            };
            let changed: HashSet<String> = self
                .config
                .servers
                .keys()
                .chain(new_config.servers.keys())
                .filter(|name| {
-                    server_identity(&self.config, name.as_str())
-                        != server_identity(&new_config, name.as_str())
+                    mcp_server_identity(&self.config, name.as_str())
+                        != mcp_server_identity(&new_config, name.as_str())
                })
                .cloned()
                .collect();
@@ -4041,6 +4056,45 @@ impl McpPool {
        Ok(())
    }

+    pub(crate) fn retire_for_handoff(&mut self) -> McpHandoff {
+        self.retired = true;
+        let connections = std::mem::take(&mut self.connections)
+            .into_iter()
+            .filter_map(|(name, connection)| {
+                let identity = mcp_server_identity(&self.config, &name)?;
+                (identity.1.is_none()
+                    && connection.config().url.is_none()
+                    && connection.is_transport_ready())
+                .then_some((name, identity, connection))
+            })
+            .collect();
+        McpHandoff {
+            workspace: self.workspace.clone(),
+            backend: self.backend,
+            connections,
+        }
+    }
+
+    pub(crate) fn adopt_handoff(&mut self, handoff: McpHandoff) -> usize {
+        if self.retired || handoff.workspace != self.workspace || handoff.backend != self.backend {
+            return 0;
+        }
+        let generation = self.current_catalog_generation();
+        let mut adopted = 0;
+        for (name, identity, mut connection) in handoff.connections {
+            if mcp_server_identity(&self.config, &name).as_ref() != Some(&identity)
+                || !connection.is_transport_ready()
+            {
+                continue;
+            }
+            connection.catalog_generation = generation;
+            if self.store_ready_connection(name, connection).is_ok() {
+                adopted += 1;
+            }
+        }
+        adopted
+    }
+
    /// Record a connect failure's auth classification. When the failure looks
    /// like a missing/expired OAuth login, the next model catalog offers the
    /// synthetic `mcp_<server>_authenticate` tool so the model can self-serve
@@ -557,6 +557,20 @@ and `codewhale mcp connect` run in their own process, so a server they just
 reached from the shell still reads `not connected` inside a running session
 until that session connects it.

+`/new`, and switching sessions from the resume picker without changing
+provider or workspace, keep user-configured stdio servers running: each live
+process moves to the new conversation without a restart, provided its config,
+the workspace and the MCP backend are unchanged. Anything the server holds in
+memory, such as a browser session and its logins, comes along; run
+`/mcp reload` for fresh servers. HTTP/SSE servers reconnect, so they pick up
+current credentials and `[network]` policy. Servers added at runtime and
+plugin-contributed servers are conversation capabilities and are closed.
+`/resume <id>`, `/load` and a provider or workspace switch start a new engine,
+and every server restarts. If an MCP call from the previous conversation is
+still running at the boundary, nothing is handed over and the servers restart
+as before; otherwise later calls from that conversation's background work fail
+instead of reconnecting.
+
 An MCP-focused `tool_search` is also explicit discovery intent: search a
 configured server name (for example `engram`), an exact `mcp_<server>_...`
 name, or use `{"query":"mcp_.*","match":"regex"}`. The current turn's
@@ -369,6 +369,8 @@ codewhale mcp tools codewhale

 会话启动时会在后台连接：每个已启用且被允许的用户配置服务器都会并发启动，不会阻塞第一个回合。把较重的服务器标记为 `lazy = true`，则在某个回合的 `allowed_tools`/`tools.always_load` 选择覆盖了它的 `mcp_<server>_*` 名称、某次模型调用解析到它的某个工具，或显式 `/mcp retry <name>` 请求它之前不会启动。标记为 `required` 的服务器即使是 `lazy` 也会在启动时连接，以便其失败在第一个回合之前暴露出来。尚未连接的服务器显示为 `not connected`，绝不会显示 `connecting`；`connecting` 标签只描述实际正在进行中的握手。OAuth 登录完成后，只有该服务器会自动重新连接。插件提供的服务器按需连接，除非标记为 `required` 或被会话的工具选择点名。

+执行 `/new`，或在不更换提供商和工作区的情况下从恢复选择器切换会话时，用户配置的 stdio 服务器会继续运行：只要其配置、工作区和 MCP 后端未变，正在运行的进程会直接转交给新对话，无需重启。服务器保存在内存中的状态（例如浏览器会话及其登录）也会一并保留；需要全新的服务器时请运行 `/mcp reload`。HTTP/SSE 服务器会重新连接，从而使用当前的凭据和 `[network]` 策略。运行时添加的服务器和插件提供的服务器属于对话能力，会被关闭。`/resume <id>`、`/load` 以及切换提供商或工作区会启动新的引擎，所有服务器都会重启。如果交接时上一个对话仍有 MCP 调用在执行，则不转交任何连接，服务器照旧重启；否则，上一个对话的后台工作此后发起的调用会直接失败，而不会重新连接。
+
 面向 MCP 的 `tool_search` 也是明确的发现请求：可搜索已配置的服务器名称
 （例如 `engram`）、完整的 `mcp_<server>_...` 名称，或使用
 `{"query":"mcp_.*","match":"regex"}`。当前回合的允许与拒绝范围先过滤服务器，
@@ -208,8 +208,8 @@
    "sources": [
      {
        "path": "docs/MCP.md",
-        "sha256": "6190d0cca1798694216d51530942dbbf4f149b789f1323a1fcfac2cbe4856173",
-        "content": "# MCP (External Tool Servers)\n\nIn the terminal, `/mcp` (also `/mcps`) opens **Extensions → MCP**. Enter opens the selected server’s recovery action or read-only details. An empty inventory offers server suggestions; browsing them installs nothing. Explicit subcommands such as `/mcp status`, `/mcp doctor`, `/mcp login`, and `/mcp add` retain their existing behavior.\n\n\n> 阅读简体中文版：[zh_hans/MCP.md](zh_hans/MCP.md)\n\ncodewhale can load additional tools via MCP (Model Context Protocol). MCP servers can be local stdio processes that the TUI starts, or remote URL-based servers that speak Streamable HTTP with legacy SSE fallback.\n\nBrowsing note:\n- `Web` is the canonical, deferred built-in browsing tool; it provides\n  `search`, `fetch`, and `wait` actions when network policy permits.\n- `web_search`, `fetch_url`, and `wait_for_dev_server` are hidden replay-only\n  aliases. New prompts and integrations should use `Web`.\n\nServer mode note:\n- `codewhale serve --mcp` runs the MCP stdio server.\n- `codewhale serve --http` runs the runtime HTTP/SSE API (separate mode).\n- `codewhale mcp-server` is an equivalent stdio entrypoint on the same\n  consolidated runtime.\n\nIn 0.10.1, the old child-server aggregation proxy is removed. `mcp-server`\nnow uses the same native tool server as `serve --mcp`; it does not launch the\nlegacy `mcp.server_definitions` list. Saved definitions are left intact. Configure\nexternal servers in `mcp.json` for the existing MCP client, or connect to those\nservers directly from your external client. Native server write tools remain\nwithheld unless the operator explicitly permits them in its server configuration.\n\n## Setup wizard vs manual MCP setup (#3407)\n\nThe `/setup` hub includes an optional **Tools and MCP**\nstep. That step is discovery/readiness only:\n\n| Wizard can do | Still requires manual / explicit action |\n| --- | --- |\n| Show configured servers as `healthy` / `needs_config` / `off` | Start or connect MCP servers |\n| Report config path presence (global + project) | Write or edit `mcp.json` contents |\n| Safe static health probe (missing command/url, broken absolute path, missing bearer env) | `codewhale mcp validate`, live connect, OAuth login |\n| Point at safe on-ramps (`/mcp`, `codewhale mcp init`, `codewhale doctor`) | Install community skills, trust skills, enable plugins |\n| Share Hotbar source counts from the same skill/MCP adapters (#3399) | Bind Hotbar slots (Hotbar step / `H`) |\n| Record optional/`needs_action` setup_state without blocking first-run | Anything that spawns processes or installs packages |\n\nEmpty inventory is **not** an error: first-run users see “nothing configured\nyet, that’s fine.” Failing or incomplete configured servers surface as\n`needs_config` with an actionable hint and never block setup completion.\nEnumeration never executes MCP/plugin commands beyond the static probe.\nSummaries redact commands, args, env, headers, and tokens.\n\n`codewhale doctor` reports MCP/skills/tools/plugins health with the same\noptional-surface intent (paths, counts, static checks) so wizard and doctor\nstay consistent.\n\n## Plugin-contributed MCP\n\nA reviewed local plugin bundle may contribute MCP servers without creating a\nsecond transport or approval system. The servers use the same MCP manager,\ntool approval, resource, prompt, timeout, and network-policy paths documented\nhere, and appear under namespaced `<plugin>-<server>` identities.\n\nThe bundle boundary is intentionally stricter than user-authored `mcp.json`:\nunknown fields and ambiguous transports fail closed; stdio environment values\nmust be exact environment-source references; remote literal headers and\nsecret-bearing URLs are rejected; declared network hosts must exactly match\nthe normalized endpoint host set; and redirects remain on the reviewed origin.\nReviewed plugin remotes also bypass ambient HTTP proxy configuration entirely;\nproxy credentials and proxy-observed traffic are not part of the v1 review.\nThe plugin review discloses local host-user authority, structural argv,\nenvironment provenance, endpoint, auth source names, scopes, and tool filters\nwithout reading or printing secret values.\n\nTrust stages reviewed content but does not enable it. Enablement attaches that\nstaged snapshot to the current workspace's MCP pool. Disable, revoke, and other\ncross-process generation changes remove catalog entries, cancel in-flight\noperations, and terminate plugin stdio children. Source or staged-tree drift is\nfully revalidated before each dispatch/catalogue boundary and fails the next\nboundary closed; v0.9.1 does not continuously hash mutable trees during an\nalready-running call and therefore does not promise drift-triggered mid-call\ncancellation. MCP subscriptions are not exposed through plugin bundles. See\n[Plugin bundles](PLUGIN_BUNDLES.md) for the complete lifecycle contract.\n\n## Bootstrap MCP Config\n\nCreate a startLine truncated
+        "sha256": "1246a1f9aa78314e0c7e304bb7219d5e15ba6d6b6bd060bbf08e77b5ea233285",
+        "content": "# MCP (External Tool Servers)\n\nIn the terminal, `/mcp` (also `/mcps`) opens **Extensions → MCP**. Enter opens the selected server’s recovery action or read-only details. An empty inventory offers server suggestions; browsing them installs nothing. Explicit subcommands such as `/mcp status`, `/mcp doctor`, `/mcp login`, and `/mcp add` retain their existing behavior.\n\n\n> 阅读简体中文版：[zh_hans/MCP.md](zh_hans/MCP.md)\n\ncodewhale can load additional tools via MCP (Model Context Protocol). MCP servers can be local stdio processes that the TUI starts, or remote URL-based servers that speak Streamable HTTP with legacy SSE fallback.\n\nBrowsing note:\n- `Web` is the canonical, deferred built-in browsing tool; it provides\n  `search`, `fetch`, and `wait` actions when network policy permits.\n- `web_search`, `fetch_url`, and `wait_for_dev_server` are hidden replay-only\n  aliases. New prompts and integrations should use `Web`.\n\nServer mode note:\n- `codewhale serve --mcp` runs the MCP stdio server.\n- `codewhale serve --http` runs the runtime HTTP/SSE API (separate mode).\n- `codewhale mcp-server` is an equivalent stdio entrypoint on the same\n  consolidated runtime.\n\nIn 0.10.1, the old child-server aggregation proxy is removed. `mcp-server`\nnow uses the same native tool server as `serve --mcp`; it does not launch the\nlegacy `mcp.server_definitions` list. Saved definitions are left intact. Configure\nexternal servers in `mcp.json` for the existing MCP client, or connect to those\nservers directly from your external client. Native server write tools remain\nwithheld unless the operator explicitly permits them in its server configuration.\n\n## Setup wizard vs manual MCP setup (#3407)\n\nThe `/setup` hub includes an optional **Tools and MCP**\nstep. That step is discovery/readiness only:\n\n| Wizard can do | Still requires manual / explicit action |\n| --- | --- |\n| Show configured servers as `healthy` / `needs_config` / `off` | Start or connect MCP servers |\n| Report config path presence (global + project) | Write or edit `mcp.json` contents |\n| Safe static health probe (missing command/url, broken absolute path, missing bearer env) | `codewhale mcp validate`, live connect, OAuth login |\n| Point at safe on-ramps (`/mcp`, `codewhale mcp init`, `codewhale doctor`) | Install community skills, trust skills, enable plugins |\n| Share Hotbar source counts from the same skill/MCP adapters (#3399) | Bind Hotbar slots (Hotbar step / `H`) |\n| Record optional/`needs_action` setup_state without blocking first-run | Anything that spawns processes or installs packages |\n\nEmpty inventory is **not** an error: first-run users see “nothing configured\nyet, that’s fine.” Failing or incomplete configured servers surface as\n`needs_config` with an actionable hint and never block setup completion.\nEnumeration never executes MCP/plugin commands beyond the static probe.\nSummaries redact commands, args, env, headers, and tokens.\n\n`codewhale doctor` reports MCP/skills/tools/plugins health with the same\noptional-surface intent (paths, counts, static checks) so wizard and doctor\nstay consistent.\n\n## Plugin-contributed MCP\n\nA reviewed local plugin bundle may contribute MCP servers without creating a\nsecond transport or approval system. The servers use the same MCP manager,\ntool approval, resource, prompt, timeout, and network-policy paths documented\nhere, and appear under namespaced `<plugin>-<server>` identities.\n\nThe bundle boundary is intentionally stricter than user-authored `mcp.json`:\nunknown fields and ambiguous transports fail closed; stdio environment values\nmust be exact environment-source references; remote literal headers and\nsecret-bearing URLs are rejected; declared network hosts must exactly match\nthe normalized endpoint host set; and redirects remain on the reviewed origin.\nReviewed plugin remotes also bypass ambient HTTP proxy configuration entirely;\nproxy credentials and proxy-observed traffic are not part of the v1 review.\nThe plugin review discloses local host-user authority, structural argv,\nenvironment provenance, endpoint, auth source names, scopes, and tool filters\nwithout reading or printing secret values.\n\nTrust stages reviewed content but does not enable it. Enablement attaches that\nstaged snapshot to the current workspace's MCP pool. Disable, revoke, and other\ncross-process generation changes remove catalog entries, cancel in-flight\noperations, and terminate plugin stdio children. Source or staged-tree drift is\nfully revalidated before each dispatch/catalogue boundary and fails the next\nboundary closed; v0.9.1 does not continuously hash mutable trees during an\nalready-running call and therefore does not promise drift-triggered mid-call\ncancellation. MCP subscriptions are not exposed through plugin bundles. See\n[Plugin bundles](PLUGIN_BUNDLES.md) for the complete lifecycle contract.\n\n## Bootstrap MCP Config\n\nCreate a startLine truncated
      }
    ]
  },