qwen-code/docs/design/2026-10-07-managed-automation-runtime.md

Managed automation runtime (H6b, H6c persistent arm)

English | 简体中文

Status: implemented in this change for the persistent target mode; the per_run target (H4 child Sessions) and the delivery policy (H5 channel deliveries) stay refused until their runtime slices land, and production enablement stays behind a deployment opt-in. This is the runtime part of slice H6 of #12827, stage H of the Managed Agent proposal #12380: slice H6b (definitions, manual runs, the single-claim scanner, the run ledger, overlap and catch-up) and the persistent arm of slice H6c of the H6 design. It follows H6a (the managed-schedule and managed-automation_run record contract, PR #13536) and extends the obligations of H0b (record contract), H0c (authority) and H3 (background Shell and Monitor, whose embedded wake scheduler this slice reuses). Below, "the reference design" is sections 1, 7, 11, 12 and 14 of the extension runtime design, and "the automation design" is section 4 of the automation, Channels and child delivery design, at the commit that #12827 pins. Wherever this document and the H6 design disagree, this document records the supersession in "Decisions".

Problem and scope

H6a pinned what may be committed: a definition chain keyed by scheduleId with append-only definition revisions, and a run chain keyed by automationRunId with a closed occurrence identity. Nothing produces either: no Session may submit the domains, no public route creates a definition, no scanner derives slots or claims occurrences, and the two planned routes answer nothing. Issue #12380's H6 line is the exit this slice works toward: "Durable schedule/run claims, overlap/catch-up, persistent/per-run execution and delivery without model rerun."

This slice delivers the persistent vertical end to end:

• Definitions (H6b). The public routes create, revise, retire and list definitions. Every mutation is relayed by the control plane to the target Session's Hosted Harness, whose authority commits the definition revision — the prompt published as a Session resource first — and replays by Idempotency-Key.
• Occurrences (H6b). A control-plane scanner derives the due slots of every armed definition on a pinned tz database, claims each definition under a lease with a fencing token, records every occurrence decision in its own ledger (fired, skipped, missed), enforces the overlap and catch-up policies, and fires an occurrence as one idempotent Harness operation. A manual run is the same operation keyed by its Idempotency-Key.
• Execution (H6c, persistent). Firing commits the run's claim revision and then, in one journal transaction, its dispatch revision together with the run's notification input and the authority-generated wake; the Session's embedded wake scheduler runs the input as an ordinary text turn; the settled turn settles the run, on the settle path and again on the next open.
• Public reads. GET /v1/agent-automations and GET /v1/agent-automations/{automationId}/runs move from planned to partial, and gain the mutation routes and the definition detail read; definitions and occurrences page separately.
• Enablement. Both domains become submittable, gated per target mode: only persistent definitions are admitted.

Still deferred (not this slice): the per_run target through H4's child Session pipeline (#13550); the delivery policy and its H5 channel_delivery projection (#13572); webhook triggers; Goal/Live/channel-loop migration; new budget kinds; WebShell mirror routes; production AgentBundle enablement.

Current state

The facts below are from this slice's base, main at e92b60a282.

• Records. managed-automation-record.ts and the Java ManagedExtensionRecords mirror define both closed bodies and replay the shared managed-automation-record-v1 fixtures. Neither domain is in MANAGED_SESSION_ENABLED_DOMAINS; commitExtensionRecord refuses both. The authority's verifyExtensionResources closes no schedule reference, so a promptRef outside the Session commits on the local store and is refused after publication by the HTTP store (H6a verification, finding F1). No cross-record check ties a run to its definition, and a second run naming an already committed occurrenceKey under a new automationRunId commits.
• Public contract. The two automation routes are planned at 1.33.0 with no response shape; H0a assigned definition CRUD, manual run and run query to H6.
• Harness side. hosted-harness-session.ts hosts per-capability funnels and one control-plane operation route per capability; the H3 embedded wake scheduler runs pending notification inputs of source monitor as text turns, deriving the pending set from the journal (pendingSessionInputs), and the close path settles pending monitor inputs model-free.
• Control plane. HarnessConnector/HostedHarnessClient carry one verb per capability; @Scheduled workers reconcile from committed rows; public mutations validate an Idempotency-Key, answer 202 and mark replays with X-Qwen-Idempotent-Replay (the AgentDefinition routes, D8a). Flyway stands at V47; open branches claim V48–V52.
• Legacy scheduler. packages/core/src/utils/cronParser.ts evaluates five-field cron in the daemon host's local time with Vixie day semantics; its state is daemon-local and nothing survives a node replacement as a reconcilable fact.

Decisions

1. A definition's home is its target Session. A persistent definition's schedule chain and every automation_run it fires live in the target Session's journal, and both the authority and the Java store refuse a persistent body whose targetSessionId is not the committing Session. The prompt is published into that Session (managed-automation-prompt) before the definition revision commits, and verifyExtensionResources and the Java store close promptRef over the Session's own store — F1 of the H6a verification is decided for Session scope. The automation design's rule follows: the scanner only discovers candidates; the target Session's authority admits the run and its wake, with or without a live Harness attachment (the control plane loads one).
2. Identities are derived once and agree in both languages. automationRunId = arun_<sha256(scheduleId NUL occurrenceKey)>. The authority and the Java store refuse a first automation_run revision whose id is not the derivation of its own scheduleId and occurrenceKey, so two claims of one occurrence collide on the record key and the second claimant reads the committed run — the structural dedup the H6a verification asked H6b for. The run's input is <automationRunId>:input (its turn id is the same), run command ids are <automationRunId>:<revision>, and the control plane derives scheduleId as asch_<sha256(tenantId NUL Idempotency-Key)[0:32]>, so a create retried after a crash between the Harness answer and the command row meets the same definition instead of minting a second one. Every definition mutation commits under the relayed operation id the control plane derives from the same Idempotency-Key (a name-based UUID), so a re-driven relay re-sends it: the authority replays the operation's originally committed revision from the journal — however the chain moved since — instead of committing the retried content over a newer revision, and the control plane records the replayed answer as the operation's result while its mirror never regresses past the revision it already tracks. A mutation honored without a revision (its content identical to the committed definition) commits an operation.replayed marker naming the honored revision, so its lost-answer retry answers the honored result instead of re-applying old content over the moved chain. Every replay verifies the operation's target and, for defines, its requested content; a retry under one identity naming another target or another content is refused 409 automation_operation_conflict rather than answered with a different request's result.
3. A run binds to the live definition at the current revision. A first automation_run revision must name a committed, non-terminal schedule of the same Session at exactly its current definitionRevision, with the same sessionMode and targetSessionId. A scanner that derived a slot from an older revision is refused and re-reads; the per-definition watermark (decision 7) keeps the slot covered or not regardless of which revision reads it.
4. The run's revision chain, persistent arm. Revision 1 (claim): run admitted, execution intent, dispatchId = automationRunId, no runtime, no delivery. Revision 2 (dispatch): run running, execution dispatch_started, committed in one transaction with input.accepted (source automation, the envelope resource holding the turn text) and the generated wake.requested. Revision 3 (settle): run settled (turn completed), failed (turn error) or cancelled (turn cancelled), execution settled; an input the close path settled model-free ends cancelled with execution not_started_proven. Each step is one allowed move per state line (H0b), so a crash between any two leaves a chain the next step completes: a claim without its dispatch is re-driven by the scanner's firing row, and a settled turn without its settle revision is reconciled when the Session opens. A crash inside the turn has no lawful proof either way: the run ends once, failed with execution outcome_unknown (the H0b pairing admits exactly that pair, widened here from recovery-blocked-only) — visible beside the definition instead of running forever, and the slot frees.
5. One Harness operation fires; it is complete before any effect and replay-safe. POST /session/:id/automations/operations with kind: fire_run commits revisions 1 and 2 in the funnel's serial chain and answers the committed run; a replay answers the same run and completes a missing revision 2. The scanner inserts the occurrence row as firing before the call and marks it fired only after the answer, so a crash anywhere before that re-drives the same operation and never a second run. Every occurrence write names the lease its writer holds (owner and fence), so a scanner whose lease expired records and settles nothing; a manual run takes the same lease (409 automation_busy when a tick keeps it). A lost answer is re-driven with backoff (2s doubling, the shift clamped at 256 seconds — the 64 attempts total close to a four-hour window, not a 5-minute cap per attempt) and after 64 attempts the row is recorded unknown: visible, never re-driven, not counted as active. A replay of an occurrence whose run is already dispatched answers that run even after the definition moved or retired; a claim whose definition ended before its dispatch settles cancelled with execution not_started_proven. A revision_stale answer refreshes the mirror from the committed record and re-pins the claim to the live revision once before the row is recorded skipped(revision_stale). Any other definitive 4xx answer settles the occurrence skipped with the route's typed code recorded as its reason — a verdict the request alone decides ends the slot — while store and transport faults answer a retryable 503 the backoff re-drives; the reason recorded is the answer's own code, hosted_session_blocked included. A run settles from the journal's own turn.settled event of its input. define_schedule and retire_schedule commit definition revisions the same way, replaying by the relayed operation id (decision 2). The verb route's connector re-runs the Workspace admission submit performs before every relay, attachment cached or not, so a grant revoked after the Session attached stops fires and definition mutations alike. When the connector's cache misses but the Session was attached before (its record carries harnessBootId), the relay re-attaches through the same takeover load a Turn uses, so a control-plane redeploy never strands every automated Session on hosted_session_already_attached.
6. Exactly one scanner claims, and the fence guards the ledger. Each definition has one ledger row (qwen_managed_automation_schedule) with a lease (lease_owner, lease_until) and a monotonic fence; a scanner claims a row only when the lease is free or expired, bumps the fence, and writes every later ledger step with WHERE lease_owner = ? AND fence = ?. A scanner that lost its lease therefore cannot move the watermark or record occurrences, and could at most re-drive a Harness operation that is idempotent by the derived run id — the effect is guarded by the journal, the ledger by the fence. The scanner runs on a dedicated managedAutomationScheduler pool, never the one-thread default scheduler that also carries the message materializer, the turn-retry dispatch and the recovery ticks, because its blocking Harness calls would otherwise delay unrelated work by a request timeout at a time.
7. Slots, watermark, late slots and catch-up. A tick derives the slots of a definition in the window (from, now], where from is the greater of the covered watermark and the arming instant, and never older than the lookback horizon (qwen.managed-agent.automation.lookback, default 24 h). A slot is timely when now − slot ≤ late-tolerance (default 5 min) and late otherwise. Timely slots fire. Late slots are missed under none; under latest the newest late slot fires as a catch-up run and the rest are recorded missed; under bounded: N the newest N fire oldest first. Every slot in the window gets a ledger row, and the watermark advances to the newest, so a later revision — or a later tick — never re-arms a covered slot (H6 decision 2). The result set is bounded (max-slots-per-tick, default 1,000) — the minute walk itself follows the lookback horizon, so raising it widens per-tick work, never trims it; a window beyond the result bound is cut at the oldest end and the cut is logged — unbounded catch-up has no path here.
8. Overlap is the Session's input-queue admission. A persistent target runs one turn at a time, so an overlapping occurrence is one more pending input in the journal. active counts the definition's occurrences whose run is not terminal (firing rows included). skip fires only at active = 0, queue_one only at active ≤ 1 (one running, one waiting in the journal), allow up to the concurrency quota (default 4); a dropped occurrence is recorded skipped with reason overlap or count_limit, never as a run. A target Session that is not ACTIVE records skipped with reason session_not_active. An open public Turn on the target suppresses every fire under any policy as skipped with reason overlap: the wake pump starts a queued automation input the moment the Session idles, while the dispatcher retries the Turn with backoff, so a recurring definition could otherwise starve the Turn indefinitely; the Turn goes first and the slot fires when it ends. The count joins the hosted Session store's record projection, so mutations answer 409 automation_unavailable unless both qwen.managed-agent.automation.enabled and qwen.managed-agent.session-store.enabled are on.
9. Cron evaluation is pinned by a shared fixture. Both languages enumerate minute instants in UTC, read each one's wall clock in the definition's zone, and match it with the H6a grammar under Vixie day semantics (either day field matches when neither is *), 7 as Sunday, and N/step as N..max/step. A wall-clock minute inside a DST gap fires once at the first instant after the gap; a minute repeated by a DST fold fires once, at its first instant. managed-automation-slots-v1.fixtures.json pins the tables — including gap and fold witnesses in the seven H6a anchor zones — and TypeScript and Java both replay them. The Java scanner is the production evaluator; the TypeScript twin exists so the semantics are a contract, not an implementation detail.
10. Timezones resolve at admission on a pinned database. The H6a contract checks the name's shape only; the control plane refuses a definition whose zone the host tz database does not resolve (400 automation_timezone_unknown), and the scanner parks a definition whose zone stops resolving (blocked_reason) rather than firing it on a guess.
11. Definitions are Session-creator mutations, answered synchronously with 202. POST /v1/agent-automations creates a definition for a Workspace-bound Session the caller created (the same authorization as later Turns and cwd changes); POST /v1/agent-automations/{automationId} revises it; DELETE retires it (the chain ends cancelled and freezes); POST /v1/agent-automations/{automationId}/runs fires a manual run. Each takes an Idempotency-Key, is relayed to the Harness synchronously, answers 202 with the resource, and replays by key through the control plane's command ledger with X-Qwen-Idempotent-Replay: true; a different body under the same key answers 409 idempotency_conflict. The reference design's 202 + operationId is honoured as D8a honours it: the answer is the committed resource, and the control plane never holds a half-committed operation, because the Harness answers only after the journal did.
12. Three ledgers, one authority. The journal's schedule and automation_run chains are the only business facts. The control plane keeps three rebuildable indexes: the definition row (the scanner's view of the latest revision, refreshed from the record when its mirrored revision lags, plus the lease and watermark), the occurrence row (one per decision, keyed by occurrenceKey, carrying the derived run id, the trigger, the outcome and its reason), and the command row (public idempotency). None holds a second copy of a journal fact that the journal does not settle.
13. Enablement is mode-scoped and deployment-gated. schedule and automation_run join MANAGED_SESSION_ENABLED_DOMAINS; a new gate beside it, MANAGED_SESSION_ENABLED_SCHEDULE_SESSION_MODES = ['persistent'], is checked by the authority on every schedule revision (the mode is not a fixed key of the chain, so checking only the first would let a later revision name a mode no first revision may open), and a run inherits it through decision 3. The control plane's scanner and the mutation routes stay inert unless qwen.managed-agent.automation.enabled is true (default false); the read routes always answer. The server-first order holds: the Java store has validated both bodies since H6a and gains the cross-record checks here.
14. No WebShell surface, no Legacy change. The daemon's /scheduled-tasks routes, cronScheduler.ts and cronParser.ts keep their behavior; a Managed definition never fires through them.
15. A crashed wake turn has one aftermath, retried until provably complete. A wake turn its Harness died inside (classified recovery, never re-driven) ends through one idempotent sequence keyed on the journal and the checkpoint, not on which settle ran before it: its run fails with execution outcome_unknown; when its checkpoint holds in-progress Runtime executions they are stopped to terminal, settled cancelled — each cancelled tool_result committed once, so the checkpoint leaves await_runtime and the history stays shaped — and the wake session hands its Broker lease back, with the release attempted on every pass since the journal cannot say whether an earlier pass landed it; only then does the crashed input's own turn_result consume it, with a transient journal conflict retried, because an unproven consume keeps the input pending and every later load and every blocked pump pass re-runs the same settle — it is never dropped, or the re-block loop returns. The blocked pump's cycle re-arms itself across a dead pass, so a transient fault cannot take the reminders down with it. A crash whose run keeps the overlap count closed may have nothing left to load the Session (no slot fires, no Turn comes): the scanner's decision against a count-based refusal — never the open-Turn one — first asks the Harness a fourth verb, reconcile_run, for the blocking occurrences, committing no new fact; the verb's own load classifies and settles the crashed wake turn and answers its run, and the decision prices the admission off those answers rather than the lagging mirror. A prompt arriving while such a park is still unsettled is refused before admission with 409 hosted_turn_recovery_in_progress — the coordinator retries it past the pre-admission budget until the pump settles the park, since this is the only refusal that clears under the daemon's own recovery channel, while the durable hosted_turn_recovery_required still meets the budget; post-admission is exactly where no retry can reach. And a turn that ended without an answer (error or cancelled) keeps its prompt out of every later turn's model history, filtered against the journal at both history builders, so a crashed instruction never merges into an unrelated turn — the no-toolProfile path's older rule, now structural for every profile. Resolved runs price the retry at the mirror's own truth, never beside it: a reconcile's settle commits its mirror row in the same transaction that answers, so the recount subtracts only what still reads blocked rather than the answered runs once more; a reconcile counts an occurrence resolved only when its own run answers terminal — never on the route's bare repaired flag, whose session-scoped wake recovery may have settled an older neighbour while the asked run still lives; a reconcile that meets the definition's or the Session's death refresh-retires the mirror and records the slot skipped(definition_retired) instead of standing forever; and each decision's reconcile fan-out scales to the configured allow bound, at least two.

The pipeline, revision by revision

Step	Where	Commit
1	control plane create	command row; Harness define_schedule → prompt resource + schedule rev 1 (admitted, definitionRevision 1); definition row mirrored
2	control plane update	Harness define_schedule → schedule rev n (definitionRevision n, run unchanged); definition row refreshed
3	scanner tick	claim the row (lease + fence); derive slots; one occurrence row per slot of the window (firing, skipped or missed; a re-driven claim may end unknown)
4	Harness fire_run	automation_run rev 1 (admitted/intent), then rev 2 (running/dispatch_started) + input.accepted + wake.requested in one transaction
5	scanner	occurrence row fired; watermark advanced under the fence; lease released
6	Harness (wake)	the automation turn runs; turn.settled
7	Harness (settle / open)	automation_run rev 3 (settled, failed or cancelled, execution settled)
8	Harness (close)	pending automation inputs settle model-free; their runs end cancelled with execution not_started_proven
9	control plane retire	Harness retire_schedule → schedule terminal revision (cancelled); definition row retired

Each commit moves every state line at most one allowed step; every control-plane verb re-reads the committed record before acting and treats "already there" as its own replay.

Quotas and configuration

Bound	Value	Refusal / effect
Prompt	≤ 64 KiB UTF-8, and the wrapped execution input — the turn text plus the envelope, under the largest identity fields the route admits and real JSON escaping — ≤ the 65,536-byte inline resource bound, computed exactly at admission so a definition never commits whose fire could not publish the input	400 invalid_automation pre-relay; 400 invalid_automation_operation at the funnel
Goal	≤ 4,096 UTF-8 bytes, no control characters (H6a contract)	400 invalid_automation
Concurrent runs under allow	4 (qwen.managed-agent.automation.concurrency)	occurrence skipped, reason count_limit
Late tolerance	5 min (late-tolerance)	older slots follow the catch-up policy
Lookback horizon	24 h (lookback)	older slots are neither fired nor recorded
Slots per tick	1,000 (max-slots-per-tick)	the window is cut at its oldest end, logged
Scan period / lease	10 s (scan-delay) / 60 s (lease)	a stalled scanner yields its rows
Definitions per Session	32	409 automation_count_limit

Files affected

• packages/core/src/managed-runtime/: managed-session-records.ts (enablement and the mode gate), managed-automation-operations.ts (new: identities, digest, envelope, turn text, body builders, the final-input budget), managed-automation-slots.ts (new: the cron slot evaluator), managed-session-authority.ts (the mode gate, the run→definition cross-record check, the derived-id check, promptRef closure, the committed-operation read), the shared slots fixture, collocated tests.
• packages/cli/src/serve/: hosted-automation-session.ts (new funnel), hosted-harness-session.ts (funnel wiring, the operations route with reconcile_run, wake admission of automation inputs, the crashed wake turn's aftermath on the wake path and on open, close-time settlement, the prompt route's wake-park refusal, the journal-keyed history exclusion, the restore verifier's domain list), hosted-monitor-wake.ts (the turn's source on close, the blocked cycle's re-arm and crash-survival), hosted-harness-model.ts (history assembly consumes the builders' filtered records).
• packages/sdk-java/managed-agent-server: V57__managed_automation_ledger.sql, AutomationLedgerStore (plus the blocking-runs read), CronSlots, AutomationScanner (the reconcile fan-out before count-based refusals), ManagedAutomationService, ManagedAutomationController, HarnessConnector.runAutomationOperation and its connector implementation, the cross-record checks in ManagedExtensionRecordStore, the H0b pairing rule in ManagedExtensionRecords (the failed + outcome_unknown pair), PublicSurface, properties and application.yml, the OpenAPI routes and schemas, ApiModels records, and tests (slots, ledger, scanner, service, store, contract).
• packages/sdk-java/qwencode: HostedHarnessClient.runAutomationOperation.
• This design in both languages; the H6 design's status line and slice table in both languages; the H0b record contract's shared fixtures (managed-extension-record-v1.fixtures.json: the failed-with-unknown-outcome case flips as the F3 pairing lands); the actor-roles design pair's /v1/agent-automations surface-matrix rows in both languages.

Validation plan

• TypeScript: builder suites (every body the H6a parser accepts, every successor the rule admits, the derived identities, the digest, the turn text); authority suites for explicit enablement, the mode gate, the run→definition binding (missing, retired, stale revision, other target, other mode), the derived-id refusal, the structural occurrence dedup, promptRef closure, and reopen rebuild; funnel suites for define/revise/retire and their replays, a mutation retried after its answer was lost answering its committed revision while the moved chain stays put (also across a reopen), a definition whose wrapped input cannot be published refused at admission, fire_run committing revision 2 with its wake in one transaction, replay after a revision-1-only crash, settle from the turn result, reconciliation on open and after a close-time settlement; the slots fixture replay. The crashed wake turn's aftermath: the block survives an unsettled park and the pump's cycle finishes it (cancelled tool_result, lease released, consume retried), reconcile_run repairs a blocked Session and reports a consumed one, the consume's transient-conflict retry, the prompt route's pre-admission refusal while the park stands, and the unanswered turn's prompt absent from the next turn's model history.
• Java: the slots fixture replay; ledger suites for claim contention (two owners, one claimant), lease expiry handover, fenced writes refused after loss, occurrence uniqueness; scanner suites for the policy matrix (every overlap × catch-up pair against scripted timely, late and overlapping windows, asserting the exact occurrence population and outcomes), firing re-drive after a simulated crash, the non-active target, a wedged run reconciled into the very slot that asked, an unanswered reconcile keeping the refusal for the next slot, and an open Turn never triggering a reconcile; service suites for create/update/retire/manual with replay and conflict, a mutation retried after its answer was lost answering its committed result while the mirror keeps the newer revision; the connector suite pinning the Workspace admission gate over a cached attachment; store suites for the cross-record checks; the contract test exercising every new route and response.
• Fault injection (store/authority level): crash before the occurrence row; between the row and the Harness answer; between revision 1 and revision 2; between the turn settle and the settle revision; between the Harness answer and the watermark — each ends in at most one run per occurrence and one model turn per run.
• Mutation checks: the derivation, the binding check, the mode gate, each policy branch, the fence guard and the DST rules, disabled in turn, fail a test in both languages.

Acceptance criteria

• Reference design section 14, item 6: two scanners claim one run per occurrence; a Channel send failure after model completion never re-runs the model (no delivery exists in this slice, so nothing can).
• Section 14, item 10: every occurrence resolves to fired, skipped or missed, and every fired run to settled, failed or cancelled with a proven execution; a claim whose answer was lost is re-driven, never guessed.
• Section 11: definitions and occurrences page separately on the public routes; mutations are Idempotency-Key based and answer 202.
• Both domains are submittable only for the persistent mode; the contract test proves the gate is explicit and mode-scoped.
• All existing suites stay green — every H1–H5a/H6a contract replay, authority, store, tool-turn and lifecycle test, npm run typecheck && npm run lint && npm run build, and the managed-agent-server surefire suite.
• Still separate: real multi-instance control planes, the §13 product-stack acceptance recorded in #12380, and the per_run and delivery arms.

Open questions

1. Catch-up age bound (H6 open question 2): the lookback horizon bounds bounded: N by age as well as by count; whether a definition should carry its own horizon is left to usage.
2. Lease and scan sizing (H6 open question 3): the defaults above are deployment configuration; measured values precede enablement.
3. Manual-run scope (H6 open question 4): decided as the Session creator for all mutations; a reader role waits for the actor-role work.
4. Equal-ended cron ranges: the H6a contract refuses 5-5; the evaluator inherits that and the fixture pins it.

Follow-up work

Slice	Scope
H6c-run	The per_run target through H4b's child Session pipeline, once #13550 lands.
H6c-dlv	The delivery policy and its projection onto H5 channel_delivery entries, once #13572 lands.
Webhook	webhook:<eventId> triggers with a verified ingress surface.
API	WebShell mirror routes; a public operation query for automation mutations if clients need one.