fix(runtime): retain worker ownership and late completion receipts (CW-7414)

A child can finish after its Runtime parent is idle or after a newer turn
starts. Share the actual worker manager and hold one permanent lifecycle
receiver, capture the dispatch owner and original Runtime turn, and publish
one durable terminal item/event to that origin. Retain owner admission policy,
shared write claims, process leases and pending terminal deliveries through
shutdown; protect live children from eviction and preserve journal failures.

Expose captured pending-approval input on authenticated canonical snapshots
and approval.required events, preserving missing versus explicit null values.
Runtime-minted single-use decision IDs, namespaced execution IDs, force/deny
routing and redacted global/legacy history remain unchanged. Existing fixtures
use actual origin-aware registration; no new test functions or code comments.

Local frozen source at 68f4dd3406, reconciled onto 601009c8b9:
- Upstream's 11 contributor-history commits have an identical committed tree;
  all 4092 tracked source inputs and eleven owned paths remained byte-identical.
- cargo fmt --all -- --check: exit 0.
- Existing focused Rust filters: 25 commands, 29 passed; 0 failed; every
  selection positive. Literal lines include:
  test result: ok. 1 passed; 0 failed; 0 ignored
  test result: ok. 3 passed; 0 failed; 0 ignored
- Scoped locked all-target Clippy: exit 0, using exactly canonical CI's three
  existing style allowances (uninlined_format_args, too_many_arguments,
  unnecessary_map_or) after fixing the new registry type complexity. The prior
  unconfigured strict failure is retained; no new lint allowance or test skip.
- npm test: 547 tests; 540 passed; 0 failed; 7 existing skipped.
- npm run check:web: exit 0 (facts/docs/tokens, lint, types, production web build).
- Actual CLI build: exit 0; SHA256
  458aaa00373b748df978a6281ae1ee75e32e1342dfdfd66c6163e14f75d01889.

Separate actual-process lanes, each with independent native identity/port
cleanup and unchanged source/binary receipts:
- Engine + PostgreSQL Work bridge: 8 named checks passed; 0 failed. Two owners,
  parent-first completion, original-turn exactly-once Runtime/PG/live/replay
  SSE delivery, producer reconnect, newer active turn and durable histories.
- Engine owner admission/lease/restart: 9 named checks passed; 0 failed. Real
  route refusal, independent-owner admission, both exclusive lease files,
  same-profile history recovery, current token 200/old token 401, no model replay.
- Engine scoped writers: 11 named checks passed; 0 failed. Normal three parent
  allow-once approvals, live overlapping-claim refusal, disjoint actual writes,
  exact nonce/readback hashes, paired durable ToolUse/ToolResult identities,
  captured Custom delegation and terminal-only release of claims.

All process model transport was scripted loopback. Earlier failed harness
attempts remain failed, including identity-domain/cursor/public-route/approval
note assumptions; they were corrected without relaxing product assertions.
Muse Wonder Full/max informed the earlier diagnosis; later timed-out requests
supplied no additional opinion. Dev linker oversized __eh_frame warning is
retained. This is source/local/process readiness, not hosted CI, a released
Engine, VM/provider isolation, paid billing, signing, deployment or production.
This commit is contained in:
CodeWhale Bot committed 2026-10-15 09:55:28 -07:00
1 parent 601009c8b9
commit 23e9119caf
11 files changed
+1729 -511

No files matched your search

+119 -30
View File
@@ -53,9 +53,9 @@ use crate::tools::spec::{
};
use crate::tools::subagent::{
FleetRole, ForegroundChildRegistry, Mailbox, MailboxMessage, SharedSubAgentManager,
SubAgentCompletion, SubAgentForkContext, SubAgentManager, SubAgentResult, SubAgentRuntime,
SubAgentStatus, agent_worker_owner_snapshot,
new_shared_subagent_manager_with_state_root_and_timeout,
SubAgentCompletion, SubAgentExecutionLimits, SubAgentForkContext, SubAgentManager,
SubAgentResult, SubAgentRuntime, SubAgentStatus, TerminalEventDelivery,
agent_worker_owner_snapshot, new_shared_subagent_manager_with_state_root_and_timeout,
};
use crate::tools::todo::{SharedTodoList, new_shared_todo_list};
use crate::tools::user_input::{UserInputRequest, UserInputResponse};
@@ -762,6 +762,7 @@ pub struct EngineHandle {
/// straight to it: the engine may be streaming or running tools and not
/// reading approvals for a long time.
subagent_manager: SharedSubAgentManager,
terminal_worker_events: Option<Arc<TerminalEventDelivery>>,
}
const MAX_PENDING_COMPACTION_CANCELLATIONS: usize = 64;
@@ -814,6 +815,35 @@ impl CompactionCancellationState {
}
impl EngineHandle {
#[must_use]
pub(crate) fn subagent_manager(&self) -> SharedSubAgentManager {
self.subagent_manager.clone()
}
pub(crate) fn mark_worker_event_delivery_failed(&self) {
if let Some(events) = self.terminal_worker_events.as_ref() {
events.mark_failed();
}
}
pub(crate) async fn wait_terminal_worker_events(&self) -> Result<()> {
if let Some(events) = self.terminal_worker_events.as_ref() {
tokio::time::timeout(FOREGROUND_CHILD_SETTLE_GRACE, events.wait())
.await
.map_err(|_| {
anyhow::anyhow!(
"terminal worker event delivery timed out during Runtime shutdown"
)
})??;
}
Ok(())
}
#[cfg(test)]
pub(crate) fn set_subagent_manager(&mut self, manager: SharedSubAgentManager) {
self.subagent_manager = manager;
}
/// Publish typed compaction cancellation immediately, then enqueue the
/// matching operation when capacity permits. The shared authority is what
/// stops a running provider future; the operation keeps the mailbox
@@ -943,6 +973,7 @@ pub struct Engine {
/// state stays alive across model turns.
repl_kernel: Option<crate::repl::PythonRuntime>,
subagent_manager: SharedSubAgentManager,
subagent_execution_limits: Option<SubAgentExecutionLimits>,
/// The deterministic Auto-Review policy shared with every child runtime
/// so children are gated by the same rules as the parent turn.
shared_auto_review_policy: Arc<crate::core::authority::auto_review::AutoReviewPolicy>,
@@ -1874,9 +1905,18 @@ impl Engine {
}
fn new_admitted(
config: EngineConfig,
api_config: &Config,
host: Option<EngineHostSetup>,
) -> (Self, EngineHandle) {
Self::new_admitted_with_subagent_manager(config, api_config, host, None)
}
fn new_admitted_with_subagent_manager(
mut config: EngineConfig,
api_config: &Config,
mut host: Option<EngineHostSetup>,
supplied_subagent_manager: Option<SharedSubAgentManager>,
) -> (Self, EngineHandle) {
crate::tls::ensure_rustls_crypto_provider();
@@ -2125,28 +2165,50 @@ impl Engine {
codewhale_core::prefix_cache::PrefixStabilityManager::new_unpinned()
});
let subagent_manager = host.as_ref().map_or_else(
|| {
let subagent_state_root = config
.subagent_state_root
.clone()
.unwrap_or_else(|| config.workspace.clone());
new_shared_subagent_manager_with_state_root_and_timeout(
config.workspace.clone(),
subagent_state_root,
let subagent_execution_limits = supplied_subagent_manager
.as_ref()
.map(|_| {
SubAgentExecutionLimits::new(
config.max_subagents,
config.max_admitted_subagents,
config.subagent_heartbeat_timeout,
config.launch_concurrency,
// #5324: defaults are captured operator config, not call fields.
api_config.subagent_default_max_steps(),
api_config
.subagent_default_wall_time_secs()
.map(std::time::Duration::from_secs),
.map(Duration::from_secs),
)
},
|host| Arc::clone(host.subagent_manager()),
);
})
.or_else(|| match host.as_ref() {
Some(EngineHostSetup::Child(child)) => {
child.authority.runtime.execution_limits.clone()
}
_ => None,
});
let subagent_manager = supplied_subagent_manager.unwrap_or_else(|| {
host.as_ref().map_or_else(
|| {
let subagent_state_root = config
.subagent_state_root
.clone()
.unwrap_or_else(|| config.workspace.clone());
new_shared_subagent_manager_with_state_root_and_timeout(
config.workspace.clone(),
subagent_state_root,
config.max_subagents,
config.max_admitted_subagents,
config.subagent_heartbeat_timeout,
config.launch_concurrency,
// #5324: defaults are captured operator config, not call fields.
api_config.subagent_default_max_steps(),
api_config
.subagent_default_wall_time_secs()
.map(std::time::Duration::from_secs),
)
},
|host| Arc::clone(host.subagent_manager()),
)
});
// The OS wrappers below only cover child processes. Codewhale's own
// `read_file`/`read`/`read_media` tools read in-process, so the same
// deny-list is installed process-wide for them to consult (S1).
@@ -2263,6 +2325,7 @@ impl Engine {
session,
repl_kernel: None,
subagent_manager,
subagent_execution_limits,
shared_auto_review_policy,
shell_manager,
file_read_tracker,
@@ -2342,6 +2405,10 @@ impl Engine {
compaction_cancellation,
turn_heartbeat: Arc::clone(&engine.turn_heartbeat),
subagent_manager: Arc::clone(&engine.subagent_manager),
terminal_worker_events: engine
.subagent_execution_limits
.as_ref()
.map(|limits| limits.terminal_events.clone()),
};
(engine, handle)
@@ -2351,6 +2418,7 @@ impl Engine {
/// client. The event loop, prompt assembly, tool registry/execution,
/// cancellation, and session projection are unchanged; only the model I/O
/// boundary is replaced.
#[cfg(test)]
pub fn new_with_model_client(
config: EngineConfig,
api_config: &Config,
@@ -3837,15 +3905,24 @@ impl Engine {
self.config.subagent_api_timeout = Duration::from_secs(api_timeout_secs);
self.config.subagent_heartbeat_timeout =
Duration::from_secs(heartbeat_timeout_secs);
let launch_gate_applied = {
let mut manager = self.subagent_manager.write().await;
manager.update_runtime_limits(
self.config.max_subagents,
self.config.max_admitted_subagents,
self.config.subagent_heartbeat_timeout,
self.config.launch_concurrency,
)
};
let launch_gate_applied =
if let Some(limits) = self.subagent_execution_limits.as_mut() {
limits.update(
self.config.max_subagents,
self.config.max_admitted_subagents,
self.config.subagent_heartbeat_timeout,
self.config.launch_concurrency,
);
true
} else {
let mut manager = self.subagent_manager.write().await;
manager.update_runtime_limits(
self.config.max_subagents,
self.config.max_admitted_subagents,
self.config.subagent_heartbeat_timeout,
self.config.launch_concurrency,
)
};
let launch_note = if launch_gate_applied {
""
} else {
@@ -5659,6 +5736,7 @@ impl Engine {
.with_locale_tag(route.locale_tag.clone())
.with_role_models(route.role_models.clone())
.with_api_config((*route.api_config).clone())
.with_execution_limits(self.subagent_execution_limits.clone())
.with_auto_model(route.auto_model)
.with_reasoning_effort(route.reasoning_effort.clone(), route.reasoning_effort_auto)
.with_agent_tool_surface_options(
@@ -7127,6 +7205,7 @@ impl Engine {
.with_locale_tag(self.config.locale_tag.clone())
.with_role_models(self.subagent_role_models())
.with_api_config(self.api_config.clone())
.with_execution_limits(self.subagent_execution_limits.clone())
.with_auto_model(self.session.auto_model)
.with_reasoning_effort(
self.session.reasoning_effort.clone(),
@@ -9343,6 +9422,8 @@ pub(crate) fn spawn_engine_with_authoritative_route_config(
host_profile: EngineHostProfile,
authoritative_route_config: Arc<parking_lot::RwLock<Config>>,
model_client: Option<SharedModelClient>,
subagent_manager: Option<SharedSubAgentManager>,
execution_admission: CancellationToken,
) -> (EngineHandle, tokio::task::JoinHandle<()>) {
// `model_client` replaces only the model I/O boundary (see
// `Engine::new_with_model_client`); hosts pass `None` for the provider
@@ -9352,10 +9433,17 @@ pub(crate) fn spawn_engine_with_authoritative_route_config(
config.goal_max_steps = None;
config.subagents_enabled = false;
}
let (mut engine, handle) = match model_client {
Some(client) => Engine::new_with_model_client(config, api_config, client),
None => Engine::new(config, api_config),
};
let (mut engine, mut handle) =
Engine::new_admitted_with_subagent_manager(config, api_config, None, subagent_manager);
if let Some(limits) = engine.subagent_execution_limits.as_mut() {
limits.admission = execution_admission;
}
if let Some(client) = model_client {
engine.model_client = Some(client);
engine.model_client_injected = true;
engine.codewhale_client_error = None;
handle.client_preflight_required = false;
}
engine.host_profile = host_profile;
engine.authoritative_route_config = Some(authoritative_route_config);
@@ -9520,6 +9608,7 @@ pub(crate) fn mock_engine_handle() -> MockEngineHandle {
std::env::temp_dir(),
1,
),
terminal_worker_events: Some(Arc::new(TerminalEventDelivery::default())),
};
MockEngineHandle {
@@ -892,6 +892,7 @@ fn engine_handle_try_send_does_not_block_when_op_channel_is_full() {
std::env::temp_dir(),
1,
),
terminal_worker_events: None,
};
// Fill the op channel with one message (capacity = 1).
+18 -15
View File
@@ -88,7 +88,7 @@ use crate::task_manager::{
};
use crate::tools::subagent::{
AgentWorkerRecord, AgentWorkerStatus, SharedSubAgentManager, SubAgentStatus,
new_shared_subagent_manager_with_timeout,
new_shared_runtime_subagent_manager,
};
#[cfg(test)]
pub(super) use codewhale_models::{ContentBlock, Message};
@@ -1369,15 +1369,11 @@ fn default_runtime_capabilities() -> RuntimeCapabilities {
}
}
fn runtime_api_sub_agent_manager(workspace: &FsPath, workers: usize) -> SharedSubAgentManager {
let max_agents = workers.max(1);
new_shared_subagent_manager_with_timeout(
workspace.to_path_buf(),
max_agents,
max_agents,
Duration::from_secs(crate::config::DEFAULT_SUBAGENT_HEARTBEAT_TIMEOUT_SECS),
max_agents,
)
fn runtime_api_sub_agent_manager(
workspace: &FsPath,
workers: usize,
) -> Result<SharedSubAgentManager> {
new_shared_runtime_subagent_manager(workspace.to_path_buf(), workers)
}
#[derive(Debug, Serialize)]
@@ -1971,6 +1967,13 @@ pub async fn run_http_server(
existing_binding.as_ref(),
)?;
let sessions_dir = runtime_threads.sessions_dir().to_path_buf();
let sub_agent_manager = runtime_api_sub_agent_manager(&workspace, options.workers)?;
runtime_threads
.attach_subagent_manager(sub_agent_manager.clone())
.await?;
let workspace_scopes =
RuntimeWorkspaceScopes::new(runtime_threads.clone(), sub_agent_manager.clone());
let workspace_scope = workspace_scopes.admit(workspace.clone()).await?;
if let Some(tap) = options.runtime_event_tap {
tap(runtime_threads.subscribe_events());
}
@@ -2020,7 +2023,6 @@ pub async fn run_http_server(
};
let skill_state = SkillStateStore::load_default()
.context("load persistent Skill activation state for Runtime API")?;
let sub_agent_manager = runtime_api_sub_agent_manager(&workspace, options.workers);
let shutdown = RuntimeServerShutdown::default();
// Opening a thread is every client's first read, and the store can only
// answer it after one pass over the whole items directory (an item's
@@ -2033,9 +2035,7 @@ pub async fn run_http_server(
tracing::warn!(%error, "thread item index warm-up failed");
}
});
let workspace_scopes =
RuntimeWorkspaceScopes::new(runtime_threads.clone(), sub_agent_manager.clone());
let workspace_scope = workspace_scopes.admit(workspace.clone()).await?;
let state = RuntimeApiState {
config: Arc::new(parking_lot::RwLock::new(config.clone())),
workspace,
@@ -13184,7 +13184,10 @@ base_url = "http://127.0.0.1:9/v1"
.await?;
let listener = TcpListener::bind("127.0.0.1:0").await?;
let addr = listener.local_addr()?;
let sub_agent_manager = runtime_api_sub_agent_manager(&workspace, 2);
let sub_agent_manager = runtime_api_sub_agent_manager(&workspace, 2)?;
runtime_threads
.attach_subagent_manager(sub_agent_manager.clone())
.await?;
let workspace_scopes =
RuntimeWorkspaceScopes::new(runtime_threads.clone(), sub_agent_manager.clone());
let workspace_scope = workspace_scopes.admit(workspace.clone()).await?;
+71 -13
View File
@@ -1288,9 +1288,13 @@ async fn build_test_server(
}
let auth_required = runtime_token.is_some();
let sub_agent_manager = overrides
.sub_agent_manager
.unwrap_or_else(|| runtime_api_sub_agent_manager(&workspace, 2));
let sub_agent_manager = match overrides.sub_agent_manager {
Some(manager) => manager,
None => runtime_api_sub_agent_manager(&workspace, 2)?,
};
runtime_threads
.attach_subagent_manager(sub_agent_manager.clone())
.await?;
// A std listener: the server thread registers it with its own runtime
// after setup is reported (see `spawn_product_stack_server`).
let listener = match std::net::TcpListener::bind("127.0.0.1:0") {
@@ -2657,7 +2661,7 @@ async fn fleet_status_runtime_api_exposes_state_and_actions() -> Result<()> {
let root = std::env::temp_dir().join(format!("codewhale-fleet-api-{}", Uuid::new_v4()));
let workspace = root.join("workspace");
fs::create_dir_all(&workspace)?;
let sub_agent_manager = runtime_api_sub_agent_manager(&workspace, 2);
let sub_agent_manager = runtime_api_sub_agent_manager(&workspace, 2)?;
let manager = FleetManager::open(&workspace)?
.with_sub_agent_manager(sub_agent_manager.clone())
.with_session_model(DEFAULT_TEXT_MODEL)
@@ -2972,6 +2976,7 @@ async fn agent_runs_runtime_api_exposes_persisted_worker_receipts() -> Result<()
launch_manifest: None,
},
owner_session_id: "session-receipt".to_string(),
runtime_origin_turn_id: None,
actor_kind: "subagent".to_string(),
parent_run_id: Some("parent_run".to_string()),
follow_up: AgentRunFollowUpTarget {
@@ -3051,15 +3056,14 @@ async fn agent_runs_runtime_api_exposes_persisted_worker_receipts() -> Result<()
)?;
let sessions_dir = root.join("sessions");
let Some((addr, _runtime_threads, handle)) =
spawn_test_server_with_root_token_mobile_workspace(
root.clone(),
sessions_dir,
None,
false,
workspace,
)
.await?
let Some((addr, runtime_threads, handle)) = spawn_test_server_with_root_token_mobile_workspace(
root.clone(),
sessions_dir,
None,
false,
workspace.clone(),
)
.await?
else {
return Ok(());
};
@@ -3105,6 +3109,60 @@ async fn agent_runs_runtime_api_exposes_persisted_worker_receipts() -> Result<()
.status();
assert_eq!(missing, StatusCode::NOT_FOUND);
assert!(run.get("runtime_origin_turn_id").is_none());
assert!(
serde_json::from_value::<AgentWorkerRecord>(run.clone())?
.runtime_origin_turn_id
.is_none()
);
let thread = runtime_threads
.create_thread(crate::runtime_threads::CreateThreadRequest::default())
.await?;
let engine = runtime_threads.get_engine(&thread.id).await?;
let actual = engine.subagent_manager();
let mut spec = serde_json::from_value::<AgentWorkerSpec>(run["spec"].clone())?;
spec.worker_id = "ordinary_receipt".into();
spec.run_id = "ordinary_run".into();
spec.parent_run_id = None;
actual
.write()
.await
.register_worker_for_session_with_origin(
spec,
&thread.id,
None,
Some("turn_original_receipt".into()),
);
let listed: serde_json::Value = client
.get(format!("http://{addr}/v1/agent-runs"))
.send()
.await?
.error_for_status()?
.json()
.await?;
let records = listed["runs"].as_array().unwrap();
assert!(
records
.iter()
.any(|record| record["spec"]["worker_id"] == "agent_receipt")
);
assert_eq!(
records
.iter()
.filter(|record| record["spec"]["worker_id"] == "ordinary_receipt")
.count(),
1
);
let ordinary: serde_json::Value = client
.get(format!("http://{addr}/v1/agent-runs/ordinary_run"))
.send()
.await?
.error_for_status()?
.json()
.await?;
assert_eq!(ordinary["owner_session_id"], thread.id);
assert_eq!(ordinary["runtime_origin_turn_id"], "turn_original_receipt");
runtime_threads.shutdown_and_wait().await?;
handle.abort();
Ok(())
}
@@ -777,7 +777,7 @@ async fn real_owner_two_workspace_services_share_scope_caches_and_settle_global_
#[cfg(windows)]
let socket = PathBuf::from(format!(r"\\.\pipe\scope-owner-{}", Uuid::new_v4()));
let token = "cwrt_scope_operator_only";
let workers = runtime_api_sub_agent_manager(&original_workspace, 2);
let workers = runtime_api_sub_agent_manager(&original_workspace, 2)?;
let (scope_tx, scope_rx) = oneshot::channel();
let (original, manager, server) =
spawn_test_server_with_root_token_mobile_workspace_and_overrides(
+594 -185
View File
@@ -23,7 +23,7 @@ use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use sha2::{Digest, Sha256};
use tokio::sync::{Mutex, RwLock as AsyncRwLock, broadcast, mpsc, oneshot, watch};
use tokio::sync::{Mutex, Notify, RwLock as AsyncRwLock, broadcast, mpsc, oneshot, watch};
use tokio_util::sync::CancellationToken;
use uuid::Uuid;
@@ -50,7 +50,9 @@ use crate::route_runtime::{ResolvedRuntimeRoute, resolve_runtime_route_for_ident
use crate::runtime_policy::RuntimePolicyProjection;
use crate::tools::plan::new_shared_plan_state;
use crate::tools::shell::{SharedShellManager, new_shared_shell_manager};
use crate::tools::subagent::SubAgentStatus;
use crate::tools::subagent::{
SharedSubAgentManager, SubAgentStatus, new_shared_runtime_subagent_manager,
};
use crate::tools::todo::new_shared_todo_list;
#[cfg(test)]
use codewhale_config::AppMode;
@@ -2370,6 +2372,11 @@ enum RuntimeEventMatch {
message_id: String,
attempt_count: u8,
},
WorkerLifecycle {
turn_id: String,
agent_id: String,
event_name: String,
},
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -4372,6 +4379,16 @@ impl RuntimeThreadStore {
event.event == "turn.completed"
&& event.turn_id.as_deref() == Some(turn_id.as_str())
}
RuntimeEventMatch::WorkerLifecycle {
turn_id,
agent_id,
event_name,
} => {
event.event == *event_name
&& event.turn_id.as_deref() == Some(turn_id.as_str())
&& event.payload.get("agent_id").and_then(Value::as_str)
== Some(agent_id.as_str())
}
RuntimeEventMatch::DynamicTerminal { turn_id, call_id } => {
matches!(
event.event.as_str(),
@@ -5324,6 +5341,12 @@ pub struct ThreadDetail {
pub approval_grants: Vec<RuntimeApprovalGrant>,
}
fn deserialize_present_approval_input<'de, D: serde::Deserializer<'de>>(
deserializer: D,
) -> Result<Option<Value>, D::Error> {
Value::deserialize(deserializer).map(Some)
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PendingApprovalRequest {
/// Runtime-owned, single-use approval ID. Clients echo this value; it is
@@ -5331,12 +5354,18 @@ pub struct PendingApprovalRequest {
pub id: String,
pub turn_id: String,
pub tool_name: String,
#[serde(
default,
skip_serializing_if = "Option::is_none",
deserialize_with = "deserialize_present_approval_input"
)]
pub input: Option<Value>,
pub description: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub intent_summary: Option<String>,
/// The provider tool-call ID this approval gates, mirroring the
/// The Engine tool-execution ID this approval gates, mirroring the
/// `tool_call_id` on `approval.required`. A client resuming from a
/// snapshot needs it to attach the prompt to the tool row it belongs to.
/// snapshot needs it to attach the prompt to the execution it belongs to.
/// It is a correlator, never a capability: `deliver_external_approval`
/// matches `id` only, so this value settles nothing.
#[serde(default, skip_serializing_if = "Option::is_none")]
@@ -6222,6 +6251,323 @@ async fn await_retained_completion(completion: &RuntimeCompletion) -> Result<()>
pub type SharedRuntimeThreadManager = Arc<RuntimeThreadManager>;
fn push_runtime_notice(
notices: &parking_lot::Mutex<HashMap<String, Vec<ActiveNotice>>>,
thread_id: &str,
kind: &str,
turn_id: &str,
subject: &str,
detail: String,
) -> String {
debug_assert!(
codewhale_config::notifications::NotificationEvent::parse(kind).is_some(),
"notice kind must be a NotificationEvent name: {kind}"
);
let id = runtime_record_id("notice");
let mut notices = notices.lock();
let list = notices.entry(thread_id.to_string()).or_default();
list.push(ActiveNotice {
id: id.clone(),
kind: kind.to_string(),
turn_id: turn_id.to_string(),
subject: subject.to_string(),
detail,
raised_at: Utc::now(),
});
if list.len() > MAX_NOTICES_PER_THREAD {
list.drain(..list.len() - MAX_NOTICES_PER_THREAD);
}
id
}
struct RuntimeWorkerEventSink {
thread_id: String,
workspace: PathBuf,
manager: SharedSubAgentManager,
store: RuntimeThreadStore,
projection: Arc<Mutex<()>>,
event_emit: Arc<Mutex<()>>,
event_tx: broadcast::Sender<RuntimeEventRecord>,
notices: Arc<parking_lot::Mutex<HashMap<String, Vec<ActiveNotice>>>>,
owner: std::sync::Weak<RuntimeProcessOwnerLock>,
}
impl RuntimeWorkerEventSink {
async fn publish(&self, event: EngineEvent) -> Result<Option<String>> {
let (owner, agent_id, event_name) = match &event {
EngineEvent::AgentSpawned {
owner_session_id,
id,
..
} => (owner_session_id, id.clone(), "agent.spawned"),
EngineEvent::AgentProgress {
owner_session_id,
id,
..
} => (owner_session_id, id.clone(), "agent.progress"),
EngineEvent::AgentComplete {
owner_session_id,
id,
..
} => (owner_session_id, id.clone(), "agent.completed"),
_ => return Ok(None),
};
if owner != &self.thread_id {
return Ok(None);
}
let _owner = self
.owner
.upgrade()
.context("Runtime event owner has closed")?;
let record = self
.manager
.read()
.await
.worker_records_for_workspace(&self.workspace)
.map_err(anyhow::Error::msg)?
.into_iter()
.map(|(record, _)| record)
.find(|record| {
record.owner_session_id == self.thread_id && record.spec.worker_id == agent_id
});
let Some(record) = record else {
return Ok(None);
};
let Some(turn_id) = record.runtime_origin_turn_id else {
return Ok(None);
};
let thread_id = self.thread_id.clone();
let _projection = self.projection.lock().await;
let store = self.store.clone();
let origin = turn_id.clone();
let selected_thread = thread_id.clone();
let selected_workspace = self.workspace.clone();
let selected_agent = agent_id.clone();
let duplicate = tokio::task::spawn_blocking(move || -> Result<bool> {
let thread = store.load_thread(&selected_thread)?;
let turn = store.load_turn(&origin)?;
anyhow::ensure!(
turn.thread_id == selected_thread && thread.workspace == selected_workspace,
"child Runtime turn owner has changed"
);
if event_name == "agent.progress" {
return Ok(false);
}
store.contains_event(
&selected_thread,
&RuntimeEventMatch::WorkerLifecycle {
turn_id: origin,
agent_id: selected_agent,
event_name: event_name.to_string(),
},
)
})
.await
.context("child receipt origin check failed")??;
if duplicate {
return Ok(Some(turn_id));
}
let item_id = if event_name == "agent.progress" {
runtime_record_id("item")
} else {
format!(
"item_{}",
crate::hashing::sha256_hex(format!("{turn_id}\0{agent_id}\0{event_name}"))
)
};
match event {
EngineEvent::AgentSpawned {
owner_session_id,
id,
prompt,
worker_status,
parent_run_id,
spawn_depth,
display_name,
..
} if owner_session_id == thread_id => {
// Hosts name the agent the way the TUI does (#6565).
let name = display_name.as_deref().unwrap_or(&id);
let message = format!(
"Sub-agent {name} spawned: {}",
summarize_text(&prompt, SUMMARY_LIMIT)
);
let item = TurnItemRecord {
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: item_id,
turn_id: turn_id.clone(),
kind: TurnItemKind::Status,
status: TurnItemLifecycleStatus::Completed,
summary: summarize_text(&message, SUMMARY_LIMIT),
detail: Some(message),
metadata: None,
artifact_refs: Vec::new(),
artifacts: Vec::new(),
started_at: Some(Utc::now()),
ended_at: Some(Utc::now()),
};
self.save_item(&item).await?;
self.emit_event(
&thread_id,
Some(&turn_id),
Some(&item.id),
"agent.spawned",
json!({ "item": item, "agent_id": id, "agent_name": display_name,
"worker_status": worker_status, "parent_run_id": parent_run_id,
"spawn_depth": spawn_depth }),
)
.await?;
}
EngineEvent::AgentProgress {
owner_session_id,
id,
status,
activity,
parent_run_id,
spawn_depth,
} if owner_session_id == thread_id => {
let message = format!("Sub-agent {id}: {status}");
let item = TurnItemRecord {
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: item_id,
turn_id: turn_id.clone(),
kind: TurnItemKind::Status,
status: TurnItemLifecycleStatus::Completed,
summary: summarize_text(&message, SUMMARY_LIMIT),
detail: Some(message),
metadata: None,
artifact_refs: Vec::new(),
artifacts: Vec::new(),
started_at: Some(Utc::now()),
ended_at: Some(Utc::now()),
};
self.save_item(&item).await?;
self.emit_event(
&thread_id,
Some(&turn_id),
Some(&item.id),
"agent.progress",
json!({ "item": item, "agent_id": id,
"worker_status": activity.worker_status, "step": activity.step,
"parent_run_id": parent_run_id, "spawn_depth": spawn_depth }),
)
.await?;
}
EngineEvent::AgentComplete {
owner_session_id,
id,
result,
outcome,
parent_run_id,
spawn_depth,
continuable,
usage,
display_name,
} if owner_session_id == thread_id => {
let worker_status = outcome
.as_ref()
.map(crate::tools::subagent::subagent_status_name);
let name = display_name.as_deref().unwrap_or(&id);
let message = format!(
"Sub-agent {name} {}: {}",
worker_status.unwrap_or("settled (outcome unconfirmed)"),
summarize_text(&result, SUMMARY_LIMIT)
);
let item = TurnItemRecord {
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: item_id,
turn_id: turn_id.clone(),
kind: TurnItemKind::Status,
status: TurnItemLifecycleStatus::Completed,
summary: summarize_text(&message, SUMMARY_LIMIT),
detail: Some(message),
metadata: None,
artifact_refs: Vec::new(),
artifacts: Vec::new(),
started_at: Some(Utc::now()),
ended_at: Some(Utc::now()),
};
self.save_item(&item).await?;
self.emit_event(
&thread_id,
Some(&turn_id),
Some(&item.id),
"agent.completed",
json!({ "item": item, "agent_id": id, "agent_name": display_name,
"worker_status": worker_status, "parent_run_id": parent_run_id,
"spawn_depth": spawn_depth, "continuable": continuable,
"usage": usage }),
)
.await?;
self.raise_notice(
&thread_id,
codewhale_config::notifications::NotificationEvent::SubagentTerminal.as_str(),
&turn_id,
&id,
format!(
"{name} {}",
match outcome.as_ref() {
Some(SubAgentStatus::Completed) => "finished",
Some(SubAgentStatus::Failed(_) | SubAgentStatus::BudgetExhausted) =>
"failed",
Some(SubAgentStatus::Cancelled | SubAgentStatus::Interrupted(_)) =>
"stopped",
Some(SubAgentStatus::Running) | None => "settled (outcome unconfirmed)",
}
),
);
}
_ => {}
}
Ok(Some(turn_id))
}
async fn save_item(&self, item: &TurnItemRecord) -> Result<()> {
let store = self.store.clone();
let item = item.clone();
tokio::task::spawn_blocking(move || -> Result<()> {
store.save_item(&item)?;
let _turn_mutation = store.turn_mutation.lock();
let mut turn = store.load_turn(&item.turn_id)?;
if !turn.item_ids.contains(&item.id) {
turn.item_ids.push(item.id);
store.save_turn(&turn)?;
}
Ok(())
})
.await
.context("child receipt persistence failed")?
}
async fn emit_event(
&self,
thread_id: &str,
turn_id: Option<&str>,
item_id: Option<&str>,
event: impl Into<String>,
payload: Value,
) -> Result<RuntimeEventRecord> {
let _emit = self.event_emit.lock().await;
let record = self
.store
.append_event(thread_id, turn_id, item_id, event, payload)
.await?;
let _ = self.event_tx.send(record.clone());
Ok(record)
}
fn raise_notice(
&self,
thread_id: &str,
kind: &str,
turn_id: &str,
subject: &str,
detail: String,
) -> String {
push_runtime_notice(&self.notices, thread_id, kind, turn_id, subject, detail)
}
}
#[derive(Clone)]
struct RecoveredTurnReceipt {
turn: TurnRecord,
@@ -6275,8 +6621,13 @@ pub struct RuntimeThreadManager {
config_admission: Arc<AsyncRwLock<()>>,
engine_load: Arc<Mutex<()>>,
shutdown_drain: Arc<Mutex<()>>,
engine_workers: Arc<parking_lot::Mutex<Vec<(EngineHandle, RuntimeCompletion)>>>,
engine_workers: Arc<parking_lot::Mutex<Vec<(String, EngineHandle, RuntimeCompletion)>>>,
subagent_manager: Arc<Mutex<Option<SharedSubAgentManager>>>,
turn_monitors: Arc<parking_lot::Mutex<Vec<RuntimeCompletion>>>,
event_monitors: Arc<parking_lot::Mutex<Vec<RuntimeCompletion>>>,
event_reader_wake: Arc<Notify>,
event_pump_stop: CancellationToken,
_event_pump_owner: Arc<tokio_util::sync::DropGuard>,
active: Arc<Mutex<ActiveThreads>>,
event_emit: Arc<Mutex<()>>,
projection_locks: Arc<parking_lot::Mutex<HashMap<String, Arc<Mutex<()>>>>>,
@@ -7718,6 +8069,7 @@ impl RuntimeThreadManager {
if let Some(reason) = grants.degraded_reason() {
tracing::warn!(%reason, "approval grants are unavailable; remembered approvals will not apply");
}
let event_pump_stop = CancellationToken::new();
let manager = Self {
host_profile,
config: Arc::new(parking_lot::RwLock::new(config)),
@@ -7729,7 +8081,12 @@ impl RuntimeThreadManager {
engine_load: Arc::new(Mutex::new(())),
shutdown_drain: Arc::new(Mutex::new(())),
engine_workers: Arc::new(parking_lot::Mutex::new(Vec::new())),
subagent_manager: Arc::new(Mutex::new(None)),
turn_monitors: Arc::new(parking_lot::Mutex::new(Vec::new())),
event_monitors: Arc::new(parking_lot::Mutex::new(Vec::new())),
event_reader_wake: Arc::new(Notify::new()),
event_pump_stop: event_pump_stop.clone(),
_event_pump_owner: Arc::new(event_pump_stop.drop_guard()),
active: Arc::new(Mutex::new(ActiveThreads::default())),
event_emit: Arc::new(Mutex::new(())),
projection_locks: Arc::new(parking_lot::Mutex::new(HashMap::new())),
@@ -7856,18 +8213,18 @@ impl RuntimeThreadManager {
let workers = self.engine_workers.lock().clone();
for engine in engines
.iter()
.chain(workers.iter().map(|(engine, _)| engine))
.chain(workers.iter().map(|(_, engine, _)| engine))
{
engine.cancel_with_reason(crate::core::engine::CancelReason::Internal);
}
for engine in engines
.iter()
.chain(workers.iter().map(|(engine, _)| engine))
.chain(workers.iter().map(|(_, engine, _)| engine))
{
let _ = engine.send(Op::Shutdown).await;
}
for (_, worker) in workers {
if let Err(error) = await_retained_completion(&worker).await {
for (_, _, worker) in &workers {
if let Err(error) = await_retained_completion(worker).await {
failure = Some(anyhow!("Runtime engine shutdown failed: {error}"));
}
}
@@ -7890,9 +8247,35 @@ impl RuntimeThreadManager {
break;
}
}
for (owner, engine, _) in &workers {
let authority = engine.subagent_manager();
let live = authority.read().await.live_count_for_session(owner);
if live > 0 {
failure = Some(anyhow!(
"Runtime shutdown left {live} unsettled workers for {owner}"
));
}
}
for engine in engines
.iter()
.chain(workers.iter().map(|(_, engine, _)| engine))
{
if let Err(error) = engine.wait_terminal_worker_events().await {
failure = Some(anyhow!("Runtime terminal worker delivery failed: {error}"));
}
}
self.event_pump_stop.cancel();
self.event_reader_wake.notify_waiters();
let pumps = self.event_monitors.lock().clone();
for pump in pumps {
if let Err(error) = await_retained_completion(&pump).await {
failure = Some(anyhow!("Runtime Engine event receiver failed: {error}"));
}
}
self.event_monitors.lock().clear();
self.engine_workers
.lock()
.retain(|(_, worker)| !completion_finished(worker));
.retain(|(_, _, worker)| !completion_finished(worker));
let mut active = self.active.lock().await;
active.engines.clear();
active.lru.clear();
@@ -8628,6 +9011,7 @@ impl RuntimeThreadManager {
id: String::new(),
turn_id: "test-turn".to_string(),
tool_name: "test-tool".to_string(),
input: None,
description: format!("test approval {label}"),
intent_summary: None,
// Stands in for the provider's raw call ID so tests can prove
@@ -10520,25 +10904,7 @@ impl RuntimeThreadManager {
subject: &str,
detail: String,
) -> String {
debug_assert!(
codewhale_config::notifications::NotificationEvent::parse(kind).is_some(),
"notice kind must be a NotificationEvent name: {kind}"
);
let id = runtime_record_id("notice");
let mut notices = self.notices.lock();
let list = notices.entry(thread_id.to_string()).or_default();
list.push(ActiveNotice {
id: id.clone(),
kind: kind.to_string(),
turn_id: turn_id.to_string(),
subject: subject.to_string(),
detail,
raised_at: Utc::now(),
});
if list.len() > MAX_NOTICES_PER_THREAD {
list.drain(..list.len() - MAX_NOTICES_PER_THREAD);
}
id
push_runtime_notice(&self.notices, thread_id, kind, turn_id, subject, detail)
}
/// Notices currently raised on a thread, oldest first. Unknown threads
@@ -14073,6 +14439,7 @@ impl RuntimeThreadManager {
"Runtime shutdown requested before turn monitoring started",
)
.await;
self.event_reader_wake.notify_waiters();
return;
}
@@ -14089,7 +14456,10 @@ impl RuntimeThreadManager {
.catch_unwind()
.await;
let failure = match result {
Ok(Ok(())) => return,
Ok(Ok(())) => {
self.event_reader_wake.notify_waiters();
return;
}
Ok(Err(error)) => {
let failure = format!("Failed to monitor {}: {error}", kind.label());
// An unreadable item or turn under the monitor is the
@@ -14111,6 +14481,7 @@ impl RuntimeThreadManager {
engine.cancel_with_reason(crate::core::engine::CancelReason::Internal);
self.settle_claimed_turn_failure(&thread_id, &turn_id, &failure)
.await;
self.event_reader_wake.notify_waiters();
}
fn spawn_claimed_turn_monitor(
@@ -14143,6 +14514,7 @@ impl RuntimeThreadManager {
manager
.settle_claimed_turn_failure(&turn.thread_id, &turn.id, &failure)
.await;
manager.event_reader_wake.notify_waiters();
return;
}
@@ -15808,17 +16180,33 @@ impl RuntimeThreadManager {
let model_client = self.test_model_client.lock().clone();
#[cfg(not(test))]
let model_client = None;
let subagent_manager = self.subagent_manager().await?;
let previous = self.engine_workers.lock().clone();
for (owner, old_engine, completion) in previous {
if owner == thread.id {
old_engine.cancel_with_reason(crate::core::engine::CancelReason::Internal);
let _ = old_engine.send(Op::Shutdown).await;
await_retained_completion(&completion).await?;
}
}
let (engine, worker) = spawn_engine_with_authoritative_route_config(
engine_cfg,
&cfg,
self.host_profile,
Arc::clone(&self.config),
model_client,
Some(subagent_manager),
self.cancel_token.clone(),
);
let engine = self.route_engine_events(&thread, engine).await;
{
let mut workers = self.engine_workers.lock();
workers.retain(|(_, worker)| !completion_finished(worker));
workers.push((engine.clone(), retained_completion(worker)));
workers.retain(|(_, _, worker)| !completion_finished(worker));
workers.push((
thread.id.clone(),
engine.clone(),
retained_completion(worker),
));
}
let sys_prompt = thread
@@ -15897,6 +16285,159 @@ impl RuntimeThreadManager {
}
}
async fn route_engine_events(
&self,
thread: &ThreadRecord,
mut engine: EngineHandle,
) -> EngineHandle {
let original = engine.rx_event.clone();
let capacity = original.read().await.max_capacity();
let (forward, receiver) = mpsc::channel(capacity);
engine.rx_event = Arc::new(AsyncRwLock::new(receiver));
let identity = Arc::downgrade(&engine.rx_event);
let active = Arc::downgrade(&self.active);
let wake = self.event_reader_wake.clone();
let stop = self.event_pump_stop.clone();
let sink = RuntimeWorkerEventSink {
thread_id: thread.id.clone(),
workspace: thread.workspace.clone(),
manager: engine.subagent_manager(),
store: self.store.clone(),
projection: self.projection_lock(&thread.id),
event_emit: self.event_emit.clone(),
event_tx: self.event_tx.clone(),
notices: self.notices.clone(),
owner: Arc::downgrade(&self._process_owner_lock),
};
let handle = engine.clone();
let thread_id = thread.id.clone();
let pump = tokio::spawn(async move {
'pump: loop {
let event = tokio::select! {
biased;
event = async { original.write().await.recv().await } => match event { Some(event) => event, None => break },
_ = stop.cancelled() => break,
};
if matches!(
&event,
EngineEvent::AgentSpawned { .. }
| EngineEvent::AgentProgress { .. }
| EngineEvent::AgentComplete { .. }
) {
let origin = match sink.publish(event.clone()).await {
Ok(origin) => origin,
Err(error) => {
handle.mark_worker_event_delivery_failed();
tracing::error!(thread_id = %thread_id, "Runtime child event publication failed: {error:#}");
if let Some(_owner) = sink.owner.upgrade() {
if let Some(failure) = RuntimeStoreRecordFailure::from_error(&error)
{
let _ = sink
.emit_event(
&thread_id,
None,
None,
RUNTIME_STORE_FAILURE_EVENT,
json!(failure.notice(&error, false)),
)
.await;
} else {
let _ = sink.emit_event(&thread_id, None, None, "error", json!({"message": "A background agent receipt could not be recorded."})).await;
}
}
break;
}
};
let owns_origin =
if let (Some(origin), Some(active)) = (origin, active.upgrade()) {
active
.lock()
.await
.engines
.get(&thread_id)
.and_then(|state| state.active_turn.as_ref())
.is_some_and(|turn| turn.turn_id == origin)
} else {
false
};
if !owns_origin {
continue;
}
}
loop {
let notified = wake.notified();
tokio::pin!(notified);
notified.as_mut().enable();
let Some(active) = active.upgrade() else {
break 'pump;
};
let owns_turn = {
let active = active.lock().await;
active.engines.get(&thread_id).is_some_and(|state| {
state.active_turn.is_some()
&& identity.upgrade().is_some_and(|receiver| {
Arc::ptr_eq(&receiver, &state.engine.rx_event)
})
})
};
if !owns_turn {
break;
}
tokio::select! {
permit = forward.reserve() => match permit {
Ok(permit) => { permit.send(event); break; },
Err(_) => break 'pump,
},
_ = &mut notified => {},
_ = stop.cancelled() => break 'pump,
}
}
}
handle.cancel_with_reason(crate::core::engine::CancelReason::Internal);
let _ = handle.try_send(Op::Shutdown);
});
let mut monitors = self.event_monitors.lock();
monitors.retain(|monitor| !completion_finished(monitor));
monitors.push(retained_completion(pump));
engine
}
pub(crate) async fn attach_subagent_manager(
&self,
manager: SharedSubAgentManager,
) -> Result<()> {
let _loading = self.engine_load.lock().await;
let mut selected = self.subagent_manager.lock().await;
if let Some(current) = selected.as_ref() {
anyhow::ensure!(
Arc::ptr_eq(current, &manager),
"Runtime already has another worker authority"
);
} else {
anyhow::ensure!(
self.active.lock().await.engines.is_empty(),
"Runtime engines started before worker authority admission"
);
*selected = Some(manager);
}
Ok(())
}
async fn subagent_manager(&self) -> Result<SharedSubAgentManager> {
let mut selected = self.subagent_manager.lock().await;
if let Some(manager) = selected.as_ref() {
return Ok(manager.clone());
}
let workspace = self.workspace.clone();
let manager = tokio::task::spawn_blocking(move || {
new_shared_runtime_subagent_manager(workspace, MAX_SUBAGENTS)
})
.await
.context("Runtime worker authority admission failed")??;
*selected = Some(manager.clone());
Ok(manager)
}
/// The thread's engine only when it is already live in this process.
/// Control routes that act on in-flight work (stopping an agent run) use
/// this: loading a cold engine cannot reach work that is not running here.
@@ -17505,154 +18046,6 @@ impl RuntimeThreadManager {
.await?;
}
}
EngineEvent::AgentSpawned {
owner_session_id,
id,
prompt,
worker_status,
parent_run_id,
spawn_depth,
display_name,
..
} if owner_session_id == thread_id => {
// Hosts name the agent the way the TUI does (#6565).
let name = display_name.as_deref().unwrap_or(&id);
let message = format!(
"Sub-agent {name} spawned: {}",
summarize_text(&prompt, SUMMARY_LIMIT)
);
let item = TurnItemRecord {
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: runtime_record_id("item"),
turn_id: turn_id.clone(),
kind: TurnItemKind::Status,
status: TurnItemLifecycleStatus::Completed,
summary: summarize_text(&message, SUMMARY_LIMIT),
detail: Some(message),
metadata: None,
artifact_refs: Vec::new(),
artifacts: Vec::new(),
started_at: Some(Utc::now()),
ended_at: Some(Utc::now()),
};
self.store.save_item(&item)?;
self.attach_item_to_turn(&turn_id, &item.id)?;
self.emit_event(
&thread_id,
Some(&turn_id),
Some(&item.id),
"agent.spawned",
json!({ "item": item, "agent_id": id, "agent_name": display_name,
"worker_status": worker_status, "parent_run_id": parent_run_id,
"spawn_depth": spawn_depth }),
)
.await?;
}
EngineEvent::AgentProgress {
owner_session_id,
id,
status,
activity,
parent_run_id,
spawn_depth,
} if owner_session_id == thread_id => {
let message = format!("Sub-agent {id}: {status}");
let item = TurnItemRecord {
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: runtime_record_id("item"),
turn_id: turn_id.clone(),
kind: TurnItemKind::Status,
status: TurnItemLifecycleStatus::Completed,
summary: summarize_text(&message, SUMMARY_LIMIT),
detail: Some(message),
metadata: None,
artifact_refs: Vec::new(),
artifacts: Vec::new(),
started_at: Some(Utc::now()),
ended_at: Some(Utc::now()),
};
self.store.save_item(&item)?;
self.attach_item_to_turn(&turn_id, &item.id)?;
self.emit_event(
&thread_id,
Some(&turn_id),
Some(&item.id),
"agent.progress",
json!({ "item": item, "agent_id": id,
"worker_status": activity.worker_status, "step": activity.step,
"parent_run_id": parent_run_id, "spawn_depth": spawn_depth }),
)
.await?;
}
EngineEvent::AgentComplete {
owner_session_id,
id,
result,
outcome,
parent_run_id,
spawn_depth,
continuable,
usage,
display_name,
} if owner_session_id == thread_id => {
let worker_status = outcome
.as_ref()
.map(crate::tools::subagent::subagent_status_name);
let name = display_name.as_deref().unwrap_or(&id);
let message = format!(
"Sub-agent {name} {}: {}",
worker_status.unwrap_or("settled (outcome unconfirmed)"),
summarize_text(&result, SUMMARY_LIMIT)
);
let item = TurnItemRecord {
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: runtime_record_id("item"),
turn_id: turn_id.clone(),
kind: TurnItemKind::Status,
status: TurnItemLifecycleStatus::Completed,
summary: summarize_text(&message, SUMMARY_LIMIT),
detail: Some(message),
metadata: None,
artifact_refs: Vec::new(),
artifacts: Vec::new(),
started_at: Some(Utc::now()),
ended_at: Some(Utc::now()),
};
self.store.save_item(&item)?;
self.attach_item_to_turn(&turn_id, &item.id)?;
self.emit_event(
&thread_id,
Some(&turn_id),
Some(&item.id),
"agent.completed",
json!({ "item": item, "agent_id": id, "agent_name": display_name,
"worker_status": worker_status, "parent_run_id": parent_run_id,
"spawn_depth": spawn_depth, "continuable": continuable,
"usage": usage }),
)
.await?;
self.raise_notice(
&thread_id,
codewhale_config::notifications::NotificationEvent::SubagentTerminal
.as_str(),
&turn_id,
&id,
format!(
"{name} {}",
match outcome.as_ref() {
Some(SubAgentStatus::Completed) => "finished",
Some(
SubAgentStatus::Failed(_) | SubAgentStatus::BudgetExhausted,
) => "failed",
Some(
SubAgentStatus::Cancelled | SubAgentStatus::Interrupted(_),
) => "stopped",
Some(SubAgentStatus::Running) | None =>
"settled (outcome unconfirmed)",
}
),
);
}
EngineEvent::AgentList {
owner_session_id,
agents,
@@ -17734,13 +18127,10 @@ impl RuntimeThreadManager {
);
let pending_request = PendingApprovalRequest {
// Replaced by the minted ID at registration. The raw
// provider call ID travels in `tool_call_id`, where it
// is a correlator a snapshot client can match against
// the tool row and never a value the API will accept.
id: String::new(),
turn_id: turn_id.clone(),
tool_name: tool_name.clone(),
input: Some(input.clone()),
description: description.clone(),
intent_summary: intent_summary.clone(),
tool_call_id: Some(id.clone()),
@@ -17830,6 +18220,7 @@ impl RuntimeThreadManager {
"approval_id": approval_id,
"tool_call_id": id,
"tool_name": tool_name,
"input": input,
"summary": summary,
"description": description,
"intent_summary": intent_summary,
@@ -17933,6 +18324,7 @@ impl RuntimeThreadManager {
"approval_id": approval_id,
"tool_call_id": id,
"tool_name": tool_name,
"input": input,
"summary": summary,
"description": description,
"intent_summary": intent_summary,
@@ -18000,6 +18392,7 @@ impl RuntimeThreadManager {
"approval_id": approval_id,
"tool_call_id": id,
"tool_name": tool_name,
"input": input,
"summary": summary,
"description": description,
"intent_summary": intent_summary,
@@ -19259,6 +19652,18 @@ impl RuntimeThreadManager {
let thread = self.get_thread(thread_id).await?;
let config = self.read_config().clone();
let route = self.resolved_route_for_thread(&config, &thread)?;
{
let mut selected = self.subagent_manager.lock().await;
if let Some(manager) = selected.as_ref() {
anyhow::ensure!(
Arc::ptr_eq(manager, &engine.subagent_manager()),
"fixture worker authority differs from Runtime owner"
);
} else {
*selected = Some(engine.subagent_manager());
}
}
let engine = self.route_engine_events(&thread, engine).await;
let mut active = self.active.lock().await;
active.engines.insert(
thread_id.to_string(),
@@ -19499,7 +19904,11 @@ fn enforce_lru_capacity(
.engines
.iter()
.filter_map(|(thread_id, state)| {
if state.active_turn.is_some() {
let manager = state.engine.subagent_manager();
let live_children = manager.try_read().map_or(true, |manager| {
manager.live_count_for_session(thread_id) > 0
});
if state.active_turn.is_some() || live_children {
Some(thread_id.clone())
} else {
None
+451 -210
View File
@@ -6708,7 +6708,10 @@ async fn install_mock_engine(
manager: &RuntimeThreadManager,
thread_id: &str,
) -> crate::core::engine::MockEngineHandle {
let harness = mock_engine_handle();
let mut harness = mock_engine_handle();
if let Some(authority) = manager.subagent_manager.lock().await.as_ref() {
harness.handle.set_subagent_manager(authority.clone());
}
manager
.install_test_engine(thread_id, harness.handle.clone())
.await
@@ -8104,6 +8107,23 @@ fn enforce_lru_capacity_does_not_loop_when_all_threads_are_active() {
assert!(evicted.is_empty(), "no idle threads should be evicted");
assert_eq!(active.engines.len(), 2);
assert_eq!(active.lru.len(), 2);
active.engines.get_mut("thr_a").unwrap().active_turn = None;
let authority = active.engines["thr_a"].engine.subagent_manager();
let child = {
let mut workers = authority.try_write().unwrap();
let child = workers.insert_test_running_agent("lru_child", &PathBuf::from("."));
workers.assign_test_session_owner(&child, "thr_a");
child
};
assert!(enforce_lru_capacity(&mut active, 2).is_empty());
let _ = authority
.try_write()
.unwrap()
.cancel_agent_for_session("thr_a", &child);
let evicted = enforce_lru_capacity(&mut active, 2);
assert_eq!(evicted.len(), 1);
assert!(!active.engines.contains_key("thr_a"));
assert!(active.engines.contains_key("thr_b"));
}
#[test]
@@ -9944,7 +9964,10 @@ async fn completed_turn_without_engine_output_fails() -> Result<()> {
#[tokio::test]
async fn worker_lifecycle_receipts_preserve_owner_outcome_and_durable_replay() -> Result<()> {
use crate::core::events::AgentProgressEventMeta;
use crate::tools::subagent::AgentWorkerStatus;
use crate::tools::subagent::{
AgentWorkerSpec, AgentWorkerStatus, AgentWorkerToolProfile, FleetRole, SubAgentManager,
};
use crate::worker_profile::WorkerRuntimeProfile;
let runtime_dir = test_runtime_dir();
let manager = test_manager(runtime_dir.clone())?;
@@ -9954,89 +9977,159 @@ async fn worker_lifecycle_receipts_preserve_owner_outcome_and_durable_replay() -
let foreign = manager
.create_thread(CreateThreadRequest::default())
.await?;
let harness = install_mock_engine(&manager, &thread.id).await;
let mut harness = mock_engine_handle();
harness
.handle
.set_subagent_manager(Arc::new(tokio::sync::RwLock::new(SubAgentManager::new(
thread.workspace.clone(),
1,
))));
manager
.install_test_engine(&thread.id, harness.handle.clone())
.await?;
let worker_manager = harness.handle.subagent_manager();
let mut rx_op = harness.rx_op;
let tx_event = harness.tx_event;
let thread_id = thread.id.clone();
let foreign_id = foreign.id.clone();
let (registered_tx, registered_rx) = oneshot::channel::<()>();
let (late_tx, late_rx) = oneshot::channel::<()>();
let mut events_rx = manager.subscribe_events();
let completed = EngineEvent::AgentComplete {
display_name: Some("audit docs".into()),
owner_session_id: thread.id.clone(),
id: "worker_completed".into(),
result: "Completed successfully".into(),
outcome: Some(SubAgentStatus::Completed),
parent_run_id: Some("parent".into()),
spawn_depth: Some(2),
continuable: Some(false),
usage: None,
};
tokio::spawn(async move {
if matches!(rx_op.recv().await, Some(Op::SendMessage(TurnSpec { .. }))) {
if !matches!(rx_op.recv().await, Some(Op::SendMessage(TurnSpec { .. }))) {
return;
}
if registered_rx.await.is_err() {
return;
}
let _ = tx_event
.send(EngineEvent::TurnStarted {
turn_id: "engine_worker_lifecycle".into(),
created_at: Utc::now(),
route: None,
submission_id: None,
})
.await;
for owner in [&thread_id, &foreign_id] {
let _ = tx_event
.send(EngineEvent::TurnStarted {
turn_id: "engine_worker_lifecycle".into(),
created_at: Utc::now(),
route: None,
submission_id: None,
.send(EngineEvent::AgentSpawned {
display_name: None,
owner_session_id: owner.to_string(),
id: "worker_spawn".into(),
prompt: "private prompt".into(),
worker_status: Some(AgentWorkerStatus::Queued),
parent_run_id: Some("parent".into()),
spawn_depth: 2,
model: "fixture".into(),
route_source: None,
})
.await;
for owner in [thread_id.clone(), foreign_id] {
let _ = tx_event
.send(EngineEvent::AgentSpawned {
display_name: None,
owner_session_id: owner.clone(),
id: "worker_spawn".into(),
prompt: "private prompt".into(),
worker_status: Some(AgentWorkerStatus::Queued),
parent_run_id: Some("parent".into()),
spawn_depth: 2,
model: "fixture".into(),
route_source: None,
})
.await;
let _ = tx_event
.send(EngineEvent::AgentProgress {
owner_session_id: owner.clone(),
id: "worker_spawn".into(),
status: "private progress".into(),
activity: AgentProgressEventMeta::new(AgentWorkerStatus::RunningTool)
.with_step(3)
.with_tool("private_tool"),
parent_run_id: Some("parent".into()),
spawn_depth: 2,
})
.await;
for (id, outcome) in [
("worker_completed", Some(SubAgentStatus::Completed)),
(
"worker_failed",
Some(SubAgentStatus::Failed("private failure".into())),
),
(
"worker_interrupted",
Some(SubAgentStatus::Interrupted("private reason".into())),
),
("worker_cancelled", Some(SubAgentStatus::Cancelled)),
("worker_budget", Some(SubAgentStatus::BudgetExhausted)),
("worker_legacy", None),
] {
let _ = tx_event
.send(EngineEvent::AgentComplete {
// #6565: hosts name the agent the way the TUI does.
display_name: (id == "worker_completed").then(|| "audit docs".into()),
owner_session_id: owner.clone(),
id: id.into(),
result: "Completed successfully".into(),
outcome,
parent_run_id: Some("parent".into()),
spawn_depth: Some(2),
continuable: Some(id == "worker_interrupted"),
usage: None,
})
.await;
}
}
let _ = tx_event
.send(EngineEvent::TurnComplete {
usage: Usage::default(),
parent_route_usage: Usage::default(),
routed_usage_dropped_records: 0,
status: TurnOutcomeStatus::Completed,
error: None,
tool_catalog: None,
base_url: None,
.send(EngineEvent::AgentProgress {
owner_session_id: owner.to_string(),
id: "worker_spawn".into(),
status: "private progress".into(),
activity: AgentProgressEventMeta::new(AgentWorkerStatus::RunningTool)
.with_step(3)
.with_tool("private_tool"),
parent_run_id: Some("parent".into()),
spawn_depth: 2,
})
.await;
}
let _ = tx_event
.send(EngineEvent::TurnComplete {
usage: Usage::default(),
parent_route_usage: Usage::default(),
routed_usage_dropped_records: 0,
status: TurnOutcomeStatus::Completed,
error: None,
tool_catalog: None,
base_url: None,
})
.await;
if late_rx.await.is_err() {
return;
}
let _ = tx_event.send(completed.clone()).await;
let _ = tx_event.send(completed.clone()).await;
if !matches!(rx_op.recv().await, Some(Op::SendMessage(TurnSpec { .. }))) {
return;
}
let _ = tx_event
.send(EngineEvent::TurnStarted {
turn_id: "engine_next_parent".into(),
created_at: Utc::now(),
route: None,
submission_id: None,
})
.await;
let _ = tx_event
.send(EngineEvent::MessageStarted { index: 0 })
.await;
let _ = tx_event
.send(EngineEvent::MessageDelta {
index: 0,
content: "Second parent response".into(),
})
.await;
let _ = tx_event
.send(EngineEvent::MessageComplete { index: 0 })
.await;
for owner in [&thread_id, &foreign_id] {
for (id, outcome) in [
(
"worker_failed",
Some(SubAgentStatus::Failed("private failure".into())),
),
(
"worker_interrupted",
Some(SubAgentStatus::Interrupted("private reason".into())),
),
("worker_cancelled", Some(SubAgentStatus::Cancelled)),
("worker_budget", Some(SubAgentStatus::BudgetExhausted)),
("worker_legacy", None),
("worker_unknown_origin", Some(SubAgentStatus::Completed)),
] {
let _ = tx_event
.send(EngineEvent::AgentComplete {
display_name: None,
owner_session_id: owner.to_string(),
id: id.into(),
result: "Completed successfully".into(),
continuable: Some(id == "worker_interrupted"),
outcome,
parent_run_id: Some("parent".into()),
spawn_depth: Some(2),
usage: None,
})
.await;
}
}
let _ = tx_event.send(completed).await;
let _ = tx_event
.send(EngineEvent::TurnComplete {
usage: Usage::default(),
parent_route_usage: Usage::default(),
routed_usage_dropped_records: 0,
status: TurnOutcomeStatus::Completed,
error: None,
tool_catalog: None,
base_url: None,
})
.await;
});
let turn = manager
.start_turn(
@@ -10047,7 +10140,87 @@ async fn worker_lifecycle_receipts_preserve_owner_outcome_and_durable_replay() -
},
)
.await?;
let _ = wait_for_terminal_turn(&manager, &turn.id).await?;
{
let mut workers = worker_manager.write().await;
for id in [
"worker_spawn",
"worker_completed",
"worker_failed",
"worker_interrupted",
"worker_cancelled",
"worker_budget",
"worker_legacy",
"worker_unknown_origin",
] {
workers.register_worker_for_session_with_origin(
AgentWorkerSpec {
worker_id: id.into(),
run_id: id.into(),
parent_run_id: Some("parent".into()),
workflow_run_id: None,
session_name: None,
objective: "fixture".into(),
role: None,
agent_type: FleetRole::Verifier,
model: "fixture".into(),
workspace: if id == "worker_completed" {
thread.workspace.join("isolated-child")
} else {
thread.workspace.clone()
},
git_branch: None,
context_mode: "fresh".into(),
fork_context: false,
tool_profile: AgentWorkerToolProfile::Inherited,
runtime_profile: WorkerRuntimeProfile::default(),
max_steps: 2,
spawn_depth: 2,
max_spawn_depth: 3,
child_route: None,
launch_manifest: None,
},
&thread.id,
None,
(id != "worker_unknown_origin").then(|| turn.id.clone()),
);
}
}
registered_tx
.send(())
.map_err(|_| anyhow!("fixture registration receiver closed"))?;
let original = wait_for_terminal_turn(&manager, &turn.id).await?;
assert_eq!(original.status, RuntimeTurnStatus::Completed);
assert!(
manager.active.lock().await.engines[&thread.id]
.active_turn
.is_none()
);
late_tx
.send(())
.map_err(|_| anyhow!("fixture completion receiver closed"))?;
tokio::time::timeout(TURN_SETTLEMENT_DEADLOCK_TIMEOUT, async {
loop {
let event = events_rx.recv().await?;
if event.event == "agent.completed" && event.payload["agent_id"] == "worker_completed" {
anyhow::ensure!(
event.turn_id.as_deref() == Some(turn.id.as_str()),
"idle completion changed parent"
);
return Ok::<_, anyhow::Error>(());
}
}
})
.await??;
let next = manager
.start_turn(
&thread.id,
StartTurnRequest {
prompt: "another parent".into(),
..Default::default()
},
)
.await?;
let _ = wait_for_terminal_turn(&manager, &next.id).await?;
let events = manager.events_since(&thread.id, None)?;
let workers: Vec<_> = events
.iter()
@@ -10061,7 +10234,12 @@ async fn worker_lifecycle_receipts_preserve_owner_outcome_and_durable_replay() -
assert_eq!(
workers.len(),
8,
"foreign-owner envelopes must never enter this journal"
"foreign, unknown-origin and duplicate envelopes must not enter this journal"
);
assert!(
workers
.iter()
.all(|event| event.turn_id.as_deref() == Some(turn.id.as_str()))
);
assert_eq!(workers[0].payload["worker_status"], "queued");
assert_eq!(workers[1].payload["worker_status"], "running_tool");
@@ -10096,6 +10274,27 @@ async fn worker_lifecycle_receipts_preserve_owner_outcome_and_durable_replay() -
.unwrap()
.starts_with("Sub-agent audit docs completed")
);
let original = manager.store.load_turn(&turn.id)?;
assert_eq!(original.status, RuntimeTurnStatus::Completed);
assert!(
workers
.iter()
.all(|event| original.item_ids.contains(event.item_id.as_ref().unwrap()))
);
let current = manager.store.load_turn(&next.id)?;
assert!(
workers
.iter()
.all(|event| !current.item_ids.contains(event.item_id.as_ref().unwrap()))
);
assert_eq!(
manager
.store
.load_thread(&thread.id)?
.latest_turn_id
.as_deref(),
Some(next.id.as_str())
);
let notices = manager.list_notices(&thread.id);
assert!(
notices
@@ -10115,9 +10314,6 @@ async fn worker_lifecycle_receipts_preserve_owner_outcome_and_durable_replay() -
.iter()
.all(|event| !event.event.starts_with("agent."))
);
// Reopen only the existing journal authority. This proves persisted event
// replay, not a fresh worker-state snapshot or process resurrection.
let reopened = RuntimeThreadStore::open(runtime_dir)?;
let replay = reopened.events_since(&thread.id, None)?;
assert_eq!(
@@ -10131,7 +10327,6 @@ async fn worker_lifecycle_receipts_preserve_owner_outcome_and_durable_replay() -
);
Ok(())
}
#[tokio::test]
async fn preturn_control_status_does_not_make_empty_turn_succeed() -> Result<()> {
let manager = test_manager(test_runtime_dir())?;
@@ -12683,144 +12878,169 @@ async fn await_approval_identity(
#[tokio::test]
async fn approval_required_awaits_external_decision_allow() -> Result<()> {
let manager = test_manager(test_runtime_dir())?;
let thread = manager
.create_thread(CreateThreadRequest {
model: None,
workspace: None,
mode: None,
allow_shell: None,
trust_mode: None,
auto_approve: None,
archived: false,
system_prompt: None,
task_id: None,
..Default::default()
})
.await?;
let mut harness = install_mock_engine(&manager, &thread.id).await;
let _turn = manager
.start_turn(
&thread.id,
StartTurnRequest {
prompt: "needs approval".to_string(),
input_summary: None,
for (call_id, tool_name, input) in [
(
"tool_external_allow",
"exec_command",
json!({"command":"echo fixture","metadata":{"values":[1,true,null]}}),
),
(
"agent:worker_fixture:approval:00000000-0000-4000-8000-000000000001",
"write",
json!({"path":"proof/shared.txt","content":"fixture\n"}),
),
("tool_external_null", "fixture_null", Value::Null),
] {
let manager = test_manager(test_runtime_dir())?;
let thread = manager
.create_thread(CreateThreadRequest {
model: None,
workspace: None,
mode: None,
allow_shell: None,
trust_mode: None,
auto_approve: None,
archived: false,
system_prompt: None,
task_id: None,
..Default::default()
},
)
.await?;
assert!(matches!(
harness.rx_op.recv().await,
Some(Op::SendMessage(TurnSpec { .. }))
));
})
.await?;
harness
.tx_event
.send(EngineEvent::ApprovalRequired {
approval_key: "key1".to_string(),
approval_grouping_key: "key1".to_string(),
id: "tool_external_allow".to_string(),
tool_name: "exec_command".to_string(),
description: "external allow".to_string(),
input: serde_json::json!({}),
intent_summary: Some("I will update the config file.".to_string()),
approval_force_prompt: false,
})
.await?;
let mut harness = install_mock_engine(&manager, &thread.id).await;
let _turn = manager
.start_turn(
&thread.id,
StartTurnRequest {
prompt: "needs approval".to_string(),
input_summary: None,
model: None,
mode: None,
allow_shell: None,
trust_mode: None,
auto_approve: None,
..Default::default()
},
)
.await?;
assert!(matches!(
harness.rx_op.recv().await,
Some(Op::SendMessage(TurnSpec { .. }))
));
let deadline = Instant::now() + APPROVAL_READINESS_TIMEOUT;
while Instant::now() < deadline && manager.pending_approvals_count() == 0 {
sleep(Duration::from_millis(20)).await;
harness
.tx_event
.send(EngineEvent::ApprovalRequired {
approval_key: "key1".to_string(),
approval_grouping_key: "key1".to_string(),
id: call_id.to_string(),
tool_name: tool_name.to_string(),
description: "external allow".to_string(),
input: input.clone(),
intent_summary: Some("I will update the config file.".to_string()),
approval_force_prompt: false,
})
.await?;
let deadline = Instant::now() + APPROVAL_READINESS_TIMEOUT;
while Instant::now() < deadline && manager.pending_approvals_count() == 0 {
sleep(Duration::from_millis(20)).await;
}
assert_eq!(manager.pending_approvals_count(), 1);
let approval_id = await_approval_identity(&manager, &thread.id, call_id).await?;
// The snapshot a reconnecting client resumes from carries the same opaque
// capability as the event, so both routes to the ID agree.
let detail = manager.get_thread_detail(&thread.id).await?;
assert_eq!(detail.pending_approvals.len(), 1);
assert_eq!(detail.pending_approvals[0].id, approval_id);
assert_eq!(detail.pending_approvals[0].turn_id, _turn.id);
assert_eq!(detail.pending_approvals[0].tool_name, tool_name);
// The snapshot keeps the raw correlator so a client that reloaded past the
// event can still attach this prompt to the tool row it gates.
assert_eq!(
detail.pending_approvals[0].tool_call_id.as_deref(),
Some(call_id)
);
assert_eq!(detail.pending_approvals[0].input.as_ref(), Some(&input));
let serialized = serde_json::to_value(&detail.pending_approvals[0])?;
assert_eq!(serialized.get("input"), Some(&input));
let reloaded: PendingApprovalRequest = serde_json::from_value(serialized.clone())?;
assert_eq!(reloaded.input, Some(input.clone()));
let mut legacy = serialized;
legacy
.as_object_mut()
.expect("approval object")
.remove("input");
let legacy: PendingApprovalRequest = serde_json::from_value(legacy)?;
assert_eq!(legacy.input, None);
assert!(serde_json::to_value(legacy)?.get("input").is_none());
assert_eq!(detail.pending_user_inputs.len(), 0);
let events = manager.events_since(&thread.id, None)?;
let approval_event = events
.iter()
.rev()
.find(|event| event.event == "approval.required")
.context("missing approval.required event")?;
assert_eq!(approval_event.payload.get("input"), Some(&input));
assert_eq!(
approval_event
.payload
.get("intent_summary")
.and_then(Value::as_str),
Some("I will update the config file.")
);
assert!(!manager.deliver_external_approval(
call_id,
ExternalApprovalDecision::Allow { remember: false },
));
assert_eq!(manager.pending_approvals_count(), 1);
assert!(manager.deliver_external_approval(
&approval_id,
ExternalApprovalDecision::Allow { remember: false },
));
assert_eq!(
harness.recv_approval_event().await,
Some(MockApprovalEvent::Approved {
id: call_id.to_string(),
})
);
assert_eq!(manager.pending_approvals_count(), 0);
assert!(
manager
.get_thread_detail(&thread.id)
.await?
.pending_approvals
.is_empty()
);
assert!(
manager.events_since(&thread.id, None)?.iter().any(|event| {
event.event == "approval.decided"
&& event.payload.get("approval_id").and_then(Value::as_str)
== Some(approval_id.as_str())
&& event.payload.get("tool_call_id").and_then(Value::as_str) == Some(call_id)
&& event.payload.get("decision").and_then(Value::as_str) == Some("allow")
}),
"approval.decided must correlate the opaque id with the tool call it settled"
);
harness
.tx_event
.send(EngineEvent::TurnComplete {
usage: Usage::default(),
parent_route_usage: Usage::default(),
routed_usage_dropped_records: 0,
status: TurnOutcomeStatus::Completed,
error: None,
tool_catalog: None,
base_url: None,
})
.await?;
}
assert_eq!(manager.pending_approvals_count(), 1);
let approval_id = await_approval_identity(&manager, &thread.id, "tool_external_allow").await?;
// The snapshot a reconnecting client resumes from carries the same opaque
// capability as the event, so both routes to the ID agree.
let detail = manager.get_thread_detail(&thread.id).await?;
assert_eq!(detail.pending_approvals.len(), 1);
assert_eq!(detail.pending_approvals[0].id, approval_id);
assert_eq!(detail.pending_approvals[0].turn_id, _turn.id);
assert_eq!(detail.pending_approvals[0].tool_name, "exec_command");
// The snapshot keeps the raw correlator so a client that reloaded past the
// event can still attach this prompt to the tool row it gates.
assert_eq!(
detail.pending_approvals[0].tool_call_id.as_deref(),
Some("tool_external_allow")
);
assert_eq!(detail.pending_user_inputs.len(), 0);
let events = manager.events_since(&thread.id, None)?;
let approval_event = events
.iter()
.rev()
.find(|event| event.event == "approval.required")
.context("missing approval.required event")?;
assert_eq!(
approval_event
.payload
.get("intent_summary")
.and_then(Value::as_str),
Some("I will update the config file.")
);
// The raw provider ID is not a credential: only the minted one settles.
assert!(!manager.deliver_external_approval(
"tool_external_allow",
ExternalApprovalDecision::Allow { remember: false },
));
assert_eq!(manager.pending_approvals_count(), 1);
assert!(manager.deliver_external_approval(
&approval_id,
ExternalApprovalDecision::Allow { remember: false },
));
// The engine is still addressed by the provider's own call ID.
assert_eq!(
harness.recv_approval_event().await,
Some(MockApprovalEvent::Approved {
id: "tool_external_allow".to_string(),
})
);
assert_eq!(manager.pending_approvals_count(), 0);
assert!(
manager
.get_thread_detail(&thread.id)
.await?
.pending_approvals
.is_empty()
);
assert!(
manager.events_since(&thread.id, None)?.iter().any(|event| {
event.event == "approval.decided"
&& event.payload.get("approval_id").and_then(Value::as_str)
== Some(approval_id.as_str())
&& event.payload.get("tool_call_id").and_then(Value::as_str)
== Some("tool_external_allow")
&& event.payload.get("decision").and_then(Value::as_str) == Some("allow")
}),
"approval.decided must correlate the opaque id with the tool call it settled"
);
harness
.tx_event
.send(EngineEvent::TurnComplete {
usage: Usage::default(),
parent_route_usage: Usage::default(),
routed_usage_dropped_records: 0,
status: TurnOutcomeStatus::Completed,
error: None,
tool_catalog: None,
base_url: None,
})
.await?;
Ok(())
}
@@ -15923,6 +16143,7 @@ async fn identical_raw_tool_call_ids_on_two_threads_stay_independently_gated() -
let manager = test_manager(test_runtime_dir())?;
let mut gated = Vec::new();
let mut inputs = std::collections::HashMap::new();
for prompt in ["first gated turn", "second gated turn"] {
let thread = manager
.create_thread(CreateThreadRequest::default())
@@ -15941,6 +16162,8 @@ async fn identical_raw_tool_call_ids_on_two_threads_stay_independently_gated() -
harness.rx_op.recv().await,
Some(Op::SendMessage(TurnSpec { .. }))
));
let input =
json!({"command":"echo fixture","owner_prompt":prompt,"nested":{"values":[null,true]}});
harness
.tx_event
.send(EngineEvent::ApprovalRequired {
@@ -15949,12 +16172,13 @@ async fn identical_raw_tool_call_ids_on_two_threads_stay_independently_gated() -
id: RAW_CALL_ID.to_string(),
tool_name: "exec_command".to_string(),
description: format!("gated {prompt}"),
input: serde_json::json!({}),
input: input.clone(),
intent_summary: None,
approval_force_prompt: false,
})
.await?;
let approval_id = await_approval_identity(&manager, &thread.id, RAW_CALL_ID).await?;
inputs.insert(thread.id.clone(), input);
gated.push((thread, harness, approval_id));
}
@@ -15976,6 +16200,10 @@ async fn identical_raw_tool_call_ids_on_two_threads_stay_independently_gated() -
// provider behavior this fix survives — while the capability that
// settles each one is distinct.
assert_eq!(pending[0].tool_call_id.as_deref(), Some(RAW_CALL_ID));
assert_eq!(pending[0].input.as_ref(), inputs.get(&thread.id));
let reloaded: PendingApprovalRequest =
serde_json::from_value(serde_json::to_value(&pending[0])?)?;
assert_eq!(reloaded.input.as_ref(), inputs.get(&thread.id));
}
// The shared correlator is visible to both clients and settles neither.
@@ -16016,6 +16244,7 @@ async fn identical_raw_tool_call_ids_on_two_threads_stay_independently_gated() -
.pending_approvals;
assert_eq!(still_pending.len(), 1);
assert_eq!(still_pending[0].id, approval_b);
assert_eq!(still_pending[0].input.as_ref(), inputs.get(&thread_b.id));
assert!(
manager
.get_thread_detail(&thread_a.id)
@@ -16124,6 +16353,7 @@ async fn runtime_approval_disposition_auto_paths_obey_force_and_posture() -> Res
} else {
None
};
let input = json!({"approval_key":"extcall:ext:claimed","command":"echo fixture","nested":{"values":[null,true,3]}});
harness
.tx_event
.send(EngineEvent::ApprovalRequired {
@@ -16133,7 +16363,7 @@ async fn runtime_approval_disposition_auto_paths_obey_force_and_posture() -> Res
tool_name: "bash".to_string(),
// Text and input cannot impersonate a Rust-minted origin key.
description: "Requested by extension:claimed".to_string(),
input: json!({"approval_key": "extcall:ext:claimed", "command": "echo fixture"}),
input: input.clone(),
intent_summary: None,
approval_force_prompt: forced,
})
@@ -16165,6 +16395,17 @@ async fn runtime_approval_disposition_auto_paths_obey_force_and_posture() -> Res
let detail = manager.get_thread_detail(&thread.id).await?;
assert!(detail.pending_approvals.is_empty());
assert_eq!(detail.approval_grants.len(), usize::from(remembered));
let required = manager
.events_since(&thread.id, None)?
.into_iter()
.find(|event| event.event == "approval.required");
if approved {
let required = required.context("automatic approval request receipt")?;
assert_eq!(required.payload.get("input"), Some(&input));
assert_eq!(required.payload["tool_call_id"], "policy-call");
} else {
assert!(required.is_none());
}
let decided = manager
.events_since(&thread.id, None)?
.into_iter()
@@ -91,6 +91,7 @@ async fn shutdown_waits_actual_engine_exit_and_terminal_monitor_before_releasing
let (engine, handle) = Engine::new_with_model_client(
EngineConfig {
workspace: root.path().into(),
session_id: Some(thread.id.clone()),
model: thread.model.clone(),
subagents_enabled: false,
snapshots_enabled: false,
@@ -115,10 +116,11 @@ async fn shutdown_waits_actual_engine_exit_and_terminal_monitor_before_releasing
blocked.await.unwrap();
engine.run().await;
});
runtime
.engine_workers
.lock()
.push((handle.clone(), retained_completion(worker)));
runtime.engine_workers.lock().push((
thread.id.clone(),
handle.clone(),
retained_completion(worker),
));
let turn = runtime
.start_turn(
&thread.id,
@@ -152,7 +154,7 @@ async fn shutdown_waits_actual_engine_exit_and_terminal_monitor_before_releasing
.await
.is_err()
);
let completion = runtime.engine_workers.lock()[0].1.clone();
let completion = runtime.engine_workers.lock()[0].2.clone();
tokio::time::timeout(Duration::from_secs(5), async {
while completion.try_lock().is_ok() {
sleep(Duration::from_millis(5)).await;
@@ -87,6 +87,7 @@ async fn completion_usage_live_terminal_counts_grandchildren_and_continuations_o
parent_completion_tx: Some(tx),
mailbox: None,
event_tx: Some(event_tx),
event_delivery: None,
session_id: "workspace".into(),
});
let mut result = terminal_result(&manager, &root);
+358 -41
View File
@@ -1029,6 +1029,8 @@ pub struct AgentWorkerRecord {
/// session-addressable without being guessed into a later conversation.
#[serde(default, skip_serializing_if = "String::is_empty")]
pub owner_session_id: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub runtime_origin_turn_id: Option<String>,
#[serde(default = "default_subagent_actor_kind")]
pub actor_kind: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
@@ -1152,6 +1154,7 @@ impl AgentWorkerRecord {
parent_run_id: spec.parent_run_id.clone(),
spec,
owner_session_id,
runtime_origin_turn_id: None,
actor_kind: default_subagent_actor_kind(),
follow_up,
takeover,
@@ -2407,6 +2410,55 @@ impl SubAgentCompletion {
})
}
}
#[derive(Default)]
pub(crate) struct TerminalEventDelivery {
pending: std::sync::atomic::AtomicUsize,
failed: std::sync::atomic::AtomicBool,
settled: tokio::sync::Notify,
}
impl TerminalEventDelivery {
pub(crate) fn mark_failed(&self) {
self.failed
.store(true, std::sync::atomic::Ordering::Release);
}
pub(crate) async fn wait(&self) -> Result<()> {
loop {
let settled = self.settled.notified();
tokio::pin!(settled);
settled.as_mut().enable();
if self.pending.load(std::sync::atomic::Ordering::Acquire) == 0 {
break;
}
settled.await;
}
anyhow::ensure!(
!self.failed.load(std::sync::atomic::Ordering::Acquire),
"a background worker lifecycle event could not reach its receiver or durable journal"
);
Ok(())
}
}
struct PendingTerminalEvent {
owner: Arc<TerminalEventDelivery>,
delivered: bool,
}
impl Drop for PendingTerminalEvent {
fn drop(&mut self) {
if !self.delivered {
self.owner
.failed
.store(true, std::sync::atomic::Ordering::Release);
}
self.owner
.pending
.fetch_sub(1, std::sync::atomic::Ordering::AcqRel);
self.owner.settled.notify_waiters();
}
}
/// Live-only sinks needed to publish one terminal child outcome.
///
@@ -2422,6 +2474,7 @@ struct SubAgentTerminalDeliveryContext {
parent_completion_tx: Option<mpsc::Sender<SubAgentCompletion>>,
mailbox: Option<Mailbox>,
event_tx: Option<mpsc::Sender<Event>>,
event_delivery: Option<Arc<TerminalEventDelivery>>,
/// Shared session namespace (root session id), cloned down the spawn
/// tree; over-budget final reports are spilled under it so the truncation
/// footer can name a retrievable artifact.
@@ -2435,6 +2488,10 @@ impl SubAgentTerminalDeliveryContext {
parent_completion_tx: runtime.parent_completion_tx.clone(),
mailbox: runtime.mailbox.clone(),
event_tx: runtime.event_tx.clone(),
event_delivery: runtime
.execution_limits
.as_ref()
.map(|limits| limits.terminal_events.clone()),
session_id: runtime.context.state_namespace.clone(),
}
}
@@ -2465,7 +2522,7 @@ impl SubAgentTerminalDeliveryContext {
}
if let Some(event_tx) = self.event_tx.as_ref() {
send_terminal_event(
send_terminal_event_with_delivery(
event_tx,
Event::AgentComplete {
owner_session_id: self.session_id.clone(),
@@ -2478,6 +2535,7 @@ impl SubAgentTerminalDeliveryContext {
usage: result.usage.clone(),
display_name: Some(subagent_result_display_name(result)),
},
self.event_delivery.as_ref(),
);
}
}
@@ -2494,20 +2552,53 @@ impl SubAgentTerminalDeliveryContext {
/// lossy: delivered late it could land after the completion and resurrect a
/// Running row, while a lost one is recovered by the completion itself.
pub(crate) fn send_terminal_event<T: Send + 'static>(event_tx: &mpsc::Sender<T>, event: T) {
send_terminal_event_with_delivery(event_tx, event, None);
}
fn send_terminal_event_with_delivery<T: Send + 'static>(
event_tx: &mpsc::Sender<T>,
event: T,
delivery: Option<&Arc<TerminalEventDelivery>>,
) {
let event = match event_tx.try_send(event) {
Ok(()) | Err(mpsc::error::TrySendError::Closed(_)) => return,
Ok(()) => return,
Err(mpsc::error::TrySendError::Closed(_)) => {
if let Some(delivery) = delivery {
delivery
.failed
.store(true, std::sync::atomic::Ordering::Release);
}
return;
}
Err(mpsc::error::TrySendError::Full(event)) => event,
};
let pending = delivery.map(|owner| {
owner
.pending
.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
PendingTerminalEvent {
owner: owner.clone(),
delivered: false,
}
});
let tx = event_tx.clone();
match tokio::runtime::Handle::try_current() {
Ok(handle) => {
handle.spawn(async move {
let _ = tx.send(event).await;
let mut pending = pending;
let delivered = tx.send(event).await.is_ok();
if let Some(pending) = pending.as_mut() {
pending.delivered = delivered;
}
});
}
Err(_) => {
std::thread::spawn(move || {
let _ = tx.blocking_send(event);
let mut pending = pending;
let delivered = tx.blocking_send(event).is_ok();
if let Some(pending) = pending.as_mut() {
pending.delivered = delivered;
}
});
}
}
@@ -2799,6 +2890,63 @@ impl Drop for ForegroundChildRegistration {
}
}
#[derive(Clone)]
pub(crate) struct SubAgentExecutionLimits {
pub max_admitted: usize,
pub heartbeat_timeout: Duration,
pub default_max_steps: Option<u32>,
pub default_wall_time: Option<Duration>,
pub launch_gate: Arc<governor::DynamicGate>,
pub terminal_events: Arc<TerminalEventDelivery>,
pub admission: CancellationToken,
}
impl SubAgentExecutionLimits {
pub(crate) fn new(
max_agents: usize,
max_admitted: usize,
heartbeat_timeout: Duration,
launch_concurrency: usize,
default_max_steps: Option<u32>,
default_wall_time: Option<Duration>,
) -> Self {
let max_agents = max_agents.clamp(1, MAX_SUBAGENTS);
Self {
max_admitted: max_admitted.clamp(max_agents, crate::config::MAX_SUBAGENT_ADMISSION),
heartbeat_timeout: if heartbeat_timeout.is_zero() {
Duration::from_secs(crate::config::DEFAULT_SUBAGENT_HEARTBEAT_TIMEOUT_SECS)
} else {
heartbeat_timeout
},
default_max_steps,
default_wall_time,
launch_gate: Arc::new(governor::DynamicGate::new(
launch_concurrency.clamp(1, max_agents),
)),
terminal_events: Arc::new(TerminalEventDelivery::default()),
admission: CancellationToken::new(),
}
}
pub(crate) fn update(
&mut self,
max_agents: usize,
max_admitted: usize,
heartbeat_timeout: Duration,
launch_concurrency: usize,
) {
let max_agents = max_agents.clamp(1, MAX_SUBAGENTS);
self.max_admitted = max_admitted.clamp(max_agents, crate::config::MAX_SUBAGENT_ADMISSION);
self.heartbeat_timeout = if heartbeat_timeout.is_zero() {
Duration::from_secs(crate::config::DEFAULT_SUBAGENT_HEARTBEAT_TIMEOUT_SECS)
} else {
heartbeat_timeout
};
self.launch_gate
.set_capacity(launch_concurrency.clamp(1, max_agents));
}
}
#[derive(Clone)]
pub struct SubAgentRuntime {
pub client: CodewhaleClient,
@@ -2853,6 +3001,7 @@ pub struct SubAgentRuntime {
/// Manager handle so children can recurse via `agent`. All agents
/// at every depth share the same manager.
pub manager: SharedSubAgentManager,
pub(crate) execution_limits: Option<SubAgentExecutionLimits>,
/// Depth in the spawn tree. 0 = top-level user turn; 1 = direct child;
/// etc. Children clone the parent runtime and increment this on spawn.
pub spawn_depth: u32,
@@ -2999,6 +3148,7 @@ impl SubAgentRuntime {
worker_profile: WorkerRuntimeProfile::for_role(FleetRole::Worker),
event_tx,
manager,
execution_limits: None,
spawn_depth: 0,
parent_agent_id: None,
max_spawn_depth: DEFAULT_MAX_SPAWN_DEPTH,
@@ -3236,6 +3386,12 @@ impl SubAgentRuntime {
self
}
#[must_use]
pub(crate) fn with_execution_limits(mut self, limits: Option<SubAgentExecutionLimits>) -> Self {
self.execution_limits = limits;
self
}
/// Preserve whether the parent session is using per-turn model routing.
#[must_use]
pub fn with_auto_model(mut self, auto_model: bool) -> Self {
@@ -3325,6 +3481,7 @@ impl SubAgentRuntime {
worker_profile: self.worker_profile.clone(),
event_tx: self.event_tx.clone(),
manager: self.manager.clone(),
execution_limits: self.execution_limits.clone(),
spawn_depth: self.spawn_depth.saturating_add(1),
parent_agent_id: self.parent_agent_id.clone(),
max_spawn_depth: self
@@ -3514,6 +3671,7 @@ pub struct SubAgent {
pub needs_input: Option<SubAgentNeedsInput>,
pub started_at: Instant,
pub last_activity_at: Instant,
heartbeat_timeout: Option<Duration>,
/// `None` = full registry inheritance, with approval-gated tools still
/// blocked unless the parent runtime is auto-approved.
/// `Some(list)` = explicit narrow allowlist (Custom agents, legacy).
@@ -3544,7 +3702,7 @@ pub struct SubAgent {
requested_stop: Option<(SubAgentStatus, Option<String>)>,
// An unresolved forced-stop retains the exact existing capacity/barrier
// guards until owner teardown; it cannot be replayed as completed work.
unsettled_launch: Option<governor::DynamicGatePermit>,
unsettled_launch: Option<Vec<governor::DynamicGatePermit>>,
unsettled_foreground: Option<ForegroundChildRegistration>,
}
@@ -3583,6 +3741,7 @@ impl SubAgent {
needs_input: None,
started_at,
last_activity_at: started_at,
heartbeat_timeout: None,
allowed_tools,
session_boot_id,
owner_session_id: String::new(),
@@ -3661,7 +3820,7 @@ pub const COORDINATION_LOCK_TIMEOUT_MARKER: &str =
"timed out acquiring delegated coordination lock";
impl CoordinationProcessLock {
fn acquire(state_root: &Path) -> Result<Self> {
fn acquire(state_root: &Path, exclusive: bool) -> Result<Self> {
let requested_root = normalize_subagent_workspace(state_root);
let lock_dir = requested_root.join(".codewhale").join("state");
// Refuse a linked `.codewhale` or `state` before creating anything:
@@ -3701,22 +3860,22 @@ impl CoordinationProcessLock {
let (ready_tx, ready_rx) = std::sync::mpsc::sync_channel(1);
let (release_tx, release_rx) = std::sync::mpsc::sync_channel(0);
let thread = std::thread::spawn(move || {
let lock = fd_lock::RwLock::new(lock_file);
// Changed from try_write (exclusive) to try_read (shared) so two
// sessions in the same state root can coexist without a sticky
// error — job rows still settle via atomic rename + sequence guard.
// Matches user intent: "no reason to have a lock like that".
match lock.try_read() {
Ok(guard) => {
// Shared lock: multiple sessions coexist, no pid stamping needed.
// The exclusive pid dance was for telling same-process handover
// from foreign owner — with shared locks every holder is equal
// and job rows settle via atomic rename, so we just hold the
// shared flock for the session lifetime.
let _guard = guard;
let mut lock = fd_lock::RwLock::new(lock_file);
let acquired = if exclusive {
lock.try_write().map(|guard| {
let _ = ready_tx.send(Ok::<(), String>(()));
let _guard = guard;
let _ = release_rx.recv();
}
})
} else {
lock.try_read().map(|guard| {
let _ = ready_tx.send(Ok::<(), String>(()));
let _guard = guard;
let _ = release_rx.recv();
})
};
match acquired {
Ok(()) => {}
Err(error) => {
let _ = ready_tx.send(Err(error.to_string()));
}
@@ -3810,8 +3969,10 @@ pub struct SubAgentManager {
/// isolate it without changing child execution cwd or file authority.
state_root: PathBuf,
state_path: Option<PathBuf>,
coordination_process_lock: std::sync::Mutex<Option<CoordinationProcessLock>>,
coordination_process_lock: std::sync::Mutex<Option<Arc<CoordinationProcessLock>>>,
coordination_process_lock_required: bool,
coordination_exclusive: bool,
workspace_writer_locks: HashMap<PathBuf, Arc<CoordinationProcessLock>>,
/// Configured default per-child model-turn budget (`[subagents]
/// default_max_steps`, #5324). `None` keeps the unbounded Fleet default;
/// an explicit spawn `max_steps` still wins. Zero means unbounded.
@@ -4323,6 +4484,8 @@ impl SubAgentManager {
state_path: None,
coordination_process_lock: std::sync::Mutex::new(None),
coordination_process_lock_required: false,
coordination_exclusive: false,
workspace_writer_locks: HashMap::new(),
max_steps: None,
wall_time: None,
max_agents,
@@ -4387,6 +4550,16 @@ impl SubAgentManager {
return Err("coordination workspace identity is unsupported on this platform".into());
#[cfg(any(unix, windows))]
{
if self.coordination_exclusive
&& normalize_subagent_workspace(&lexical)
!= normalize_subagent_workspace(&self.state_root)
&& !self.workspace_writer_locks.contains_key(&canonical_root)
{
let lock = CoordinationProcessLock::acquire(&lexical, true)
.map_err(|error| error.to_string())?;
self.workspace_writer_locks
.insert(canonical_root.clone(), Arc::new(lock));
}
let receipt = CoordinationClaimScope {
canonical_root,
platform: platform.into(),
@@ -5480,7 +5653,9 @@ impl SubAgentManager {
.coordination_process_lock
.get_mut()
.expect("coordination lock slot poisoned") =
CoordinationProcessLock::acquire(&self.state_root).ok();
CoordinationProcessLock::acquire(&self.state_root, self.coordination_exclusive)
.ok()
.map(Arc::new);
self
}
@@ -5499,9 +5674,9 @@ impl SubAgentManager {
if slot.is_some() {
return Ok(());
}
match CoordinationProcessLock::acquire(&self.state_root) {
match CoordinationProcessLock::acquire(&self.state_root, self.coordination_exclusive) {
Ok(lock) => {
*slot = Some(lock);
*slot = Some(Arc::new(lock));
Ok(())
}
Err(error) => Err(error.to_string()),
@@ -5648,9 +5823,18 @@ impl SubAgentManager {
return Ok(std::thread::spawn(|| {}));
};
let state_root = self.state_root.clone();
let writer_locks = self
.coordination_process_lock
.lock()
.expect("coordination lock slot poisoned")
.iter()
.cloned()
.chain(self.workspace_writer_locks.values().cloned())
.collect::<Vec<_>>();
// Spawn disk I/O off the write-lock hot path. `payload` is fully
// owned (cloned from `self.agents`) so it is `Send` and safe to move.
let handle = std::thread::spawn(move || {
let _writer_locks = writer_locks;
if let Err(err) = write_json_atomic(&state_root, &path, &payload) {
tracing::warn!(target: "subagent", ?err, "failed to persist sub-agent state");
}
@@ -5845,6 +6029,7 @@ impl SubAgentManager {
needs_input: persisted.needs_input,
started_at,
last_activity_at: started_at,
heartbeat_timeout: None,
allowed_tools,
// Empty string when loading pre-#405 records; the
// manager treats that the same as a non-matching id —
@@ -6064,6 +6249,21 @@ impl SubAgentManager {
spec: AgentWorkerSpec,
owner_session_id: &str,
precomputed_evidence: Option<DeliveryEvidence>,
) {
self.register_worker_for_session_with_origin(
spec,
owner_session_id,
precomputed_evidence,
None,
);
}
pub(crate) fn register_worker_for_session_with_origin(
&mut self,
spec: AgentWorkerSpec,
owner_session_id: &str,
precomputed_evidence: Option<DeliveryEvidence>,
runtime_origin_turn_id: Option<String>,
) {
let worker_id = spec.worker_id.clone();
let now_ms = epoch_millis_now();
@@ -6073,6 +6273,7 @@ impl SubAgentManager {
owner_session_id.to_string(),
precomputed_evidence,
);
record.runtime_origin_turn_id = runtime_origin_turn_id;
self.push_worker_event(
&mut record,
AgentWorkerStatus::Starting,
@@ -8393,7 +8594,10 @@ impl SubAgentManager {
agent.status == SubAgentStatus::Running
&& agent.task_handle.is_some()
&& agent.requested_stop.is_none()
&& agent.last_activity_at.elapsed() >= self.running_heartbeat_timeout
&& agent.last_activity_at.elapsed()
>= agent
.heartbeat_timeout
.unwrap_or(self.running_heartbeat_timeout)
}
pub fn touch(&mut self, agent_id: &str) -> bool {
@@ -8428,6 +8632,13 @@ impl SubAgentManager {
options: SubAgentSpawnOptions,
precomputed_delivery_evidence: Option<DeliveryEvidence>,
) -> Result<SubAgentResult> {
if runtime
.execution_limits
.as_ref()
.is_some_and(|limits| limits.admission.is_cancelled())
{
return Err(anyhow!("Runtime worker admission is closed"));
}
let mut origin = match options.origin.as_ref() {
Some(origin) => origin.clone(),
None => self
@@ -8469,6 +8680,16 @@ impl SubAgentManager {
self.cleanup(COMPLETED_AGENT_RETENTION);
self.check_admission_capacity()?;
if let Some(limits) = runtime.execution_limits.as_ref() {
let admitted = self.live_count_for_session(&runtime.context.state_namespace);
if admitted >= limits.max_admitted {
return Err(anyhow!(
"Sub-agent route admission limit reached (max_admitted {}, admitted {}). Wait for queued/running agents to finish or cancel unneeded agents.",
limits.max_admitted,
admitted
));
}
}
runtime.max_spawn_depth = runtime
.max_spawn_depth
.min(runtime.worker_profile.max_spawn_depth);
@@ -8501,7 +8722,14 @@ impl SubAgentManager {
narrow_optional_limit(runtime.max_output_tokens, options.max_output_tokens);
let mut max_steps = narrow_model_steps(
runtime.worker_profile.max_steps,
resolve_max_steps(agent_type.clone(), options.max_steps, self.max_steps),
resolve_max_steps(
agent_type.clone(),
options.max_steps,
runtime
.execution_limits
.as_ref()
.map_or(self.max_steps, |limits| limits.default_max_steps),
),
);
if let Some(source) = options
.resume_from_agent_id
@@ -8537,10 +8765,14 @@ impl SubAgentManager {
// `wall_time_secs` may raise it up to MAX_CHILD_WALL_TIME. An operator
// `[subagents] default_wall_time_secs` stays a ceiling, and inherited
// profiles and deadlines still only narrow.
let default_wall_time = runtime
.execution_limits
.as_ref()
.map_or(self.wall_time, |limits| limits.default_wall_time);
let wall_time = options
.wall_time
.unwrap_or_else(|| self.wall_time.unwrap_or(DEFAULT_CHILD_WALL_TIME))
.min(self.wall_time.unwrap_or(MAX_CHILD_WALL_TIME))
.unwrap_or_else(|| default_wall_time.unwrap_or(DEFAULT_CHILD_WALL_TIME))
.min(default_wall_time.unwrap_or(MAX_CHILD_WALL_TIME))
.min(
runtime
.worker_profile
@@ -8639,6 +8871,7 @@ impl SubAgentManager {
// live holder, so a reused name never resolves ambiguously.
if let Some(existing) = self.agents.values().find(|existing| {
existing.session_name == name
&& self.agent_is_owned_by_session(existing, &runtime.context.state_namespace)
&& !self.is_from_prior_session(existing)
&& !subagent_status_is_settled(&existing.status)
}) {
@@ -8733,7 +8966,18 @@ impl SubAgentManager {
));
}
}
let runtime_origin_turn_id = if runtime.context.runtime.active_thread_id.as_deref()
== Some(runtime.context.state_namespace.as_str())
{
runtime
.runtime_usage_lease
.as_ref()
.map(|lease| lease.owner().to_string())
} else {
None
};
let durable_registration = write_capable
|| runtime_origin_turn_id.is_some()
|| continuation_from.is_some()
|| origin
.selected_workspace
@@ -8918,12 +9162,17 @@ impl SubAgentManager {
}
};
agent.owner_session_id = runtime.context.state_namespace.clone();
agent.heartbeat_timeout = runtime
.execution_limits
.as_ref()
.map(|limits| limits.heartbeat_timeout);
agent.stop_token = Some(runtime.cancel_token.clone());
agent.terminal_delivery = Some(SubAgentTerminalDeliveryContext::from_runtime(&runtime));
self.register_worker_for_session(
self.register_worker_for_session_with_origin(
worker_spec,
&runtime.context.state_namespace,
precomputed_delivery_evidence,
runtime_origin_turn_id,
);
self.install_worker_origin(&agent_id, origin);
@@ -9315,7 +9564,11 @@ impl SubAgentManager {
let millis =
|duration: Duration| u64::try_from(duration.as_millis()).unwrap_or(u64::MAX);
snap.idle_ms = Some(millis(agent.last_activity_at.elapsed()));
snap.heartbeat_timeout_ms = Some(millis(self.running_heartbeat_timeout));
snap.heartbeat_timeout_ms = Some(millis(
agent
.heartbeat_timeout
.unwrap_or(self.running_heartbeat_timeout),
));
}
if let Some(record) = self.worker_records.get(&agent.id) {
snap.usage = Some(record.usage.clone());
@@ -9502,11 +9755,17 @@ impl SubAgentManager {
agent.status == SubAgentStatus::Running
&& !agent.completion_claimed
&& agent.requested_stop.is_none()
&& agent.last_activity_at.elapsed() >= timeout
&& agent.last_activity_at.elapsed()
>= agent.heartbeat_timeout.unwrap_or(timeout)
})
.map(|agent| agent.id.clone())
.collect::<Vec<_>>();
for agent_id in stale_agent_ids {
let timeout = self
.agents
.get(&agent_id)
.and_then(|agent| agent.heartbeat_timeout)
.unwrap_or(timeout);
let orphan = self
.agents
.get(&agent_id)
@@ -10972,6 +11231,44 @@ pub fn new_shared_subagent_manager_with_timeout(
)
}
pub(crate) fn new_shared_runtime_subagent_manager(
workspace: PathBuf,
workers: usize,
) -> Result<SharedSubAgentManager> {
type RuntimeManagerRegistry =
parking_lot::Mutex<HashMap<PathBuf, (usize, std::sync::Weak<RwLock<SubAgentManager>>)>>;
static MANAGERS: std::sync::OnceLock<RuntimeManagerRegistry> = std::sync::OnceLock::new();
let workers = workers.clamp(1, MAX_SUBAGENTS);
let state_path = default_state_path(&workspace)?;
let key = checked_subagent_state_path(&workspace, &state_path)?;
let mut managers = MANAGERS
.get_or_init(|| parking_lot::Mutex::new(HashMap::new()))
.lock();
managers.retain(|_, (_, manager)| manager.strong_count() > 0);
if let Some((capacity, manager)) = managers.get(&key)
&& let Some(manager) = manager.upgrade()
{
anyhow::ensure!(
*capacity == workers,
"Runtime worker authority already has aggregate capacity {capacity}; requested {workers}. Attach to its owner or stop it before changing workers."
);
return Ok(manager);
}
let mut manager = SubAgentManager::new(workspace.clone(), workers)
.with_admission_limit(workers)
.with_launch_concurrency(workers)
.with_state_path(state_path);
manager.coordination_exclusive = true;
manager = manager.require_coordination_process_lock();
manager
.ensure_coordination_process_lock()
.map_err(anyhow::Error::msg)?;
manager.load_state()?;
let manager = Arc::new(RwLock::new(manager));
managers.insert(key, (workers, Arc::downgrade(&manager)));
Ok(manager)
}
/// Create a shared sub-agent manager with an explicit control-plane state root.
///
/// `workspace` remains the child execution and file-authority root. The worker
@@ -11023,8 +11320,8 @@ pub fn new_shared_subagent_manager_with_state_root_and_timeout(
// empty ledger that would be written straight over the real one the
// moment the first session exited and the flock became acquirable
// (owner report, 2026-08-04). Read unconditionally; the write gate on
// `persist_state`/`persist_state_synchronously` is what keeps the file
// single-writer.
// `persist_state`/`persist_state_synchronously` still checks this
// manager's coordinator admission before publishing.
if let Err(err) = manager.load_state() {
// Routed through tracing instead of stderr — see comment in
// `persist_state_best_effort` above.
@@ -13727,12 +14024,20 @@ async fn run_subagent_task_inner(mut task: SubAgentTask) {
// slot after waiting starts its work clock then (#6015), still bounded by
// any inherited deadline. Cancellation while queued is handled by
// the admitted child actor before it emits Started/Starting.
let mut _launch_permit = None;
let mut _launch_permits = Vec::new();
let mut launch_wait_timed_out = false;
let mut launched_from_queue = false;
if let Some(gate) = task.launch_gate.as_ref() {
let owner_gate = (task.runtime.spawn_depth == 1)
.then(|| {
task.runtime
.execution_limits
.as_ref()
.map(|limits| &limits.launch_gate)
})
.flatten();
for gate in owner_gate.into_iter().chain(task.launch_gate.as_ref()) {
match Arc::clone(gate).try_acquire() {
Some(permit) => _launch_permit = Some(permit),
Some(permit) => _launch_permits.push(permit),
None => {
match tokio::time::timeout_at(
deadline.into(),
@@ -13741,10 +14046,17 @@ async fn run_subagent_task_inner(mut task: SubAgentTask) {
.await
{
Ok(permit) => {
launched_from_queue = permit.is_some();
_launch_permit = permit;
if let Some(permit) = permit {
launched_from_queue = true;
_launch_permits.push(permit);
} else {
break;
}
}
Err(_) => {
launch_wait_timed_out = true;
break;
}
Err(_) => launch_wait_timed_out = true,
}
}
}
@@ -13883,7 +14195,7 @@ async fn run_subagent_task_inner(mut task: SubAgentTask) {
Some(note.clone()),
)
});
agent.unsettled_launch = _launch_permit.take();
agent.unsettled_launch = Some(std::mem::take(&mut _launch_permits));
agent.unsettled_foreground = task._foreground_child_registration.take();
}
if let Some(record) = manager.worker_records.get_mut(&agent_id) {
@@ -19421,9 +19733,14 @@ mod owner_scoped_coordination_tests {
let selected = root.join("selected");
fs::create_dir_all(original.join("src")).unwrap();
fs::create_dir_all(selected.join("src")).unwrap();
let mut manager = SubAgentManager::new(original.clone(), 2);
let authority = new_shared_runtime_subagent_manager(original.clone(), 2).unwrap();
let mut manager = authority.try_write().unwrap();
admit(&mut manager, &original);
admit(&mut manager, &selected);
let independent =
SubAgentManager::new(selected.clone(), 2).require_coordination_process_lock();
assert!(independent.ensure_coordination_process_lock().is_err());
assert!(independent.persist_state_synchronously().is_err());
manager
.preflight_worker_coordination(&writer("original", &original, "src"))
.unwrap();
+108 -11
View File
@@ -1197,7 +1197,12 @@ fn coordination_hot_paths_count_only_active_authoritative_owners() {
#[test]
fn headless_worker_registration_enforces_live_claims_and_projects_context() {
let tmp = tempdir().expect("tempdir");
let mut manager = SubAgentManager::new(tmp.path().to_path_buf(), 8);
let authority_a = new_shared_runtime_subagent_manager(tmp.path().to_path_buf(), 8)
.expect("admit Runtime writer");
let authority_b = new_shared_runtime_subagent_manager(tmp.path().join("."), 8)
.expect("reuse canonical Runtime writer");
assert!(Arc::ptr_eq(&authority_a, &authority_b));
let mut manager = authority_a.try_write().expect("first authority lock");
manager
.record_coordination_decision(DecisionRecord {
decision_id: "decision-src-a".into(),
@@ -1218,7 +1223,7 @@ fn headless_worker_registration_enforces_live_claims_and_projects_context() {
spec.role = Some("worker".into());
spec.runtime_profile = WorkerRuntimeProfile::for_role(FleetRole::Builder);
spec.launch_manifest = Some(ChildLaunchManifest {
owner_session: "fleet-run".into(),
owner_session: format!("owner-{id}"),
child_id: id.into(),
profile: spec.runtime_profile.clone(),
prompt: spec.objective.clone(),
@@ -1237,11 +1242,14 @@ fn headless_worker_registration_enforces_live_claims_and_projects_context() {
};
let worker_a = worker("worker-a", "src/a");
let origin_a = manager
.capture_direct_worker_origin(tmp.path(), "owner-worker-a")
.expect("capture first owner");
manager
.preflight_worker_coordination(&worker_a)
.preflight_worker_coordination_in_origin(&worker_a, &origin_a)
.expect("first worker preflight");
manager
.register_worker_with_coordination(worker_a)
.register_worker_with_coordination_in_origin(worker_a, origin_a)
.expect("first worker registration");
let record = manager
.get_worker_record("worker-a")
@@ -1257,21 +1265,82 @@ fn headless_worker_registration_enforces_live_claims_and_projects_context() {
.contains("src-a-contract")
);
let origin_b = manager
.capture_direct_worker_origin(tmp.path(), "owner-worker-b")
.expect("capture overlapping owner");
let overlap = manager
.preflight_worker_coordination(&worker("worker-b", "src/a/nested"))
.preflight_worker_coordination_in_origin(&worker("worker-b", "src/a/nested"), &origin_b)
.expect_err("overlapping live Fleet writer must remain queued");
assert!(overlap.contains("worker-a"), "{overlap}");
assert_eq!(manager.coordination.contentions.len(), 1);
assert!(manager.get_worker_record("worker-b").is_none());
let worker_c = worker("worker-c", "src/b");
let origin_c = manager
.capture_direct_worker_origin(tmp.path(), "owner-worker-c")
.expect("capture non-overlapping owner");
manager
.preflight_worker_coordination(&worker_c)
.preflight_worker_coordination_in_origin(&worker_c, &origin_c)
.expect("non-overlapping worker preflight");
manager
.register_worker_with_coordination(worker_c)
.register_worker_with_coordination_in_origin(worker_c, origin_c)
.expect("non-overlapping workers may proceed concurrently");
assert_eq!(manager.coordination.write_claims.len(), 2);
manager
.worker_records
.get_mut("worker-a")
.unwrap()
.runtime_origin_turn_id = Some("turn-owner-a".into());
manager
.worker_records
.get_mut("worker-c")
.unwrap()
.runtime_origin_turn_id = Some("turn-owner-c".into());
manager
.persist_state_synchronously()
.expect("persist both owners");
drop(manager);
let persisted = load_persisted_agent_worker_records_with_state_root(tmp.path(), tmp.path())
.expect("read shared durable history");
assert_eq!(persisted.len(), 2);
assert!(
persisted
.iter()
.any(|record| record.owner_session_id == "owner-worker-a")
);
assert!(
persisted
.iter()
.any(|record| record.owner_session_id == "owner-worker-c")
);
let independent =
SubAgentManager::new(tmp.path().to_path_buf(), 8).require_coordination_process_lock();
assert!(independent.ensure_coordination_process_lock().is_err());
assert!(independent.persist_state_synchronously().is_err());
drop(independent);
drop(authority_a);
drop(authority_b);
let reopened = new_shared_runtime_subagent_manager(tmp.path().to_path_buf(), 8)
.expect("reopen after actual writer release");
let reopened = reopened.try_read().expect("reopened owner lock");
assert!(reopened.get_worker_record("worker-a").is_some());
assert!(reopened.get_worker_record("worker-c").is_some());
assert_eq!(
reopened
.get_worker_record("worker-a")
.unwrap()
.runtime_origin_turn_id
.as_deref(),
Some("turn-owner-a")
);
assert_eq!(
reopened
.get_worker_record("worker-c")
.unwrap()
.runtime_origin_turn_id
.as_deref(),
Some("turn-owner-c")
);
}
#[test]
@@ -6952,6 +7021,9 @@ async fn full_event_channel_still_delivers_agent_complete() {
.expect("fill the only slot");
let mut runtime = runtime_with_depth(1, None);
runtime.event_tx = Some(event_tx);
let limits = SubAgentExecutionLimits::new(2, 2, Duration::from_secs(60), 2, None, None);
let delivery = limits.terminal_events.clone();
runtime.execution_limits = Some(limits);
agent.terminal_delivery = Some(SubAgentTerminalDeliveryContext::from_runtime(&runtime));
manager.agents.insert(agent_id.clone(), agent);
manager.register_worker(make_worker_spec(&agent_id, tmp.path().to_path_buf()));
@@ -6964,6 +7036,9 @@ async fn full_event_channel_still_delivers_agent_complete() {
"the roster leaves Running"
);
let waiting = delivery.wait();
tokio::pin!(waiting);
assert!(futures_util::poll!(&mut waiting).is_pending());
let filler = event_rx.recv().await.expect("filler event");
assert!(matches!(filler, Event::Status { .. }));
let delivered = tokio::time::timeout(Duration::from_secs(5), event_rx.recv())
@@ -6974,6 +7049,18 @@ async fn full_event_channel_still_delivers_agent_complete() {
&delivered,
Event::AgentComplete { id, outcome: Some(SubAgentStatus::Cancelled), .. } if id == &agent_id
));
waiting.await.expect("actual terminal send completed");
let failed = Arc::new(TerminalEventDelivery::default());
let (tx, rx) = mpsc::channel(1);
tx.try_send(Event::status("full")).unwrap();
send_terminal_event_with_delivery(&tx, Event::status("terminal"), Some(&failed));
drop(rx);
assert!(
tokio::time::timeout(Duration::from_secs(5), failed.wait())
.await
.expect("closed sender settles without hanging")
.is_err()
);
}
/// #6560 D02-06: a nested parent's completion inbox is bounded and has no
@@ -11853,7 +11940,7 @@ fn coordination_lock_refuses_linked_codewhale_without_writing_through_it() {
std::fs::create_dir_all(&outside).expect("mkdir outside");
std::os::unix::fs::symlink(&outside, workspace.join(".codewhale")).expect("link .codewhale");
let Err(err) = CoordinationProcessLock::acquire(&workspace) else {
let Err(err) = CoordinationProcessLock::acquire(&workspace, false) else {
panic!("a linked .codewhale must be refused");
};
let message = format!("{err:#}");
@@ -15518,6 +15605,7 @@ pub(crate) fn stub_runtime() -> SubAgentRuntime {
worker_profile: WorkerRuntimeProfile::for_role(FleetRole::Worker),
event_tx: None,
manager: new_shared_subagent_manager(workspace, 5),
execution_limits: None,
spawn_depth: 0,
max_spawn_depth: DEFAULT_MAX_SPAWN_DEPTH,
cancel_token: CancellationToken::new(),
@@ -18207,10 +18295,14 @@ async fn launch_gate_queues_extra_direct_children() {
runtime.context = ToolContext::new(tmp.path());
runtime.mailbox = Some(mailbox);
let gate = Arc::new(governor::DynamicGate::new(1));
let held_launch_permit = Arc::clone(&gate)
runtime.spawn_depth = 1;
let limits = SubAgentExecutionLimits::new(1, 4, Duration::from_secs(60), 1, None, None);
let held_launch_permit = limits
.launch_gate
.try_acquire()
.expect("test holds the single launch permit");
.expect("hold owner route slot");
runtime.execution_limits = Some(limits);
let gate = manager.read().await.launch_gate.clone();
let spawn = |agent_id: &str, gate: Option<Arc<governor::DynamicGate>>| {
let (input_tx, input_rx) = mpsc::unbounded_channel();
let agent = SubAgent::new(
@@ -18286,6 +18378,11 @@ async fn launch_gate_queues_extra_direct_children() {
queued.is_ok(),
"second child must publish a visible queued reason: {messages:?}"
);
assert_eq!(
gate.available_permits(),
4,
"a route-blocked owner must not hold an aggregate slot"
);
drop(held_launch_permit);
let collected = tokio::time::timeout(Duration::from_secs(5), async {