feat(managed-agent): Support per-run automation cancellation (#14526)

This commit is contained in:
jinye authored and GitHub committed 2026-10-18 14:07:54 +00:00
1 parent 97d48447a2
commit 68aa5a8879
19 files changed
+981 -46

No files matched your search

@@ -0,0 +1,43 @@
# Cancel one Managed Automation run
[English](2026-10-18-automation-run-cancel.md) | [简体中文](2026-10-18-automation-run-cancel.zh-CN.md)
## Status and problem
Proposed implementation for [#14460](https://github.com/QwenLM/qwen-code/issues/14460), based on main `b3c5d3ed1e67`. Automation runs appear as tasks, but cannot advertise or accept cancellation. Existing run settlement already maps the committed Turn outcome to `cancelled`.
## Goals and boundaries
Cancel one run through the existing public and WebShell task routes. Preserve the schedule, future occurrences, sibling runs and unrelated foreground Turns. Keep `automation.enabled` defaulting to `false`. Do not add a public route, operation kind, record field, schema migration or UI control. General `CANCEL` recovery, Automation observability, `per_run` Sessions and the complete FG11 fault round remain separate work.
## Design
### Control plane
Add `automation_run` to the cancellable task kinds. Durable admission, authorization, idempotency and operation leasing continue to use `TASK_CANCEL`. The coordinator sends a private `cancel_run` Automation operation naming the original run, schedule and occurrence; the Hosted route verifies that the immutable pair derives that exact run before addressing its Turn. A mismatch is refused without cancelling any run. The Hosted connector uses stop authorization and passive attachment, rather than new-work authorization or session-wide cancellation takeover. A reload may start a queued run before Hosted accepts the stop; Java admission alone does not promise physical cancellation.
### Hosted authority
Serialize cancellation with the Automation funnel's fire and settlement writes. A claim still at `intent` can settle `cancelled/not_started_proven` immediately: no input was admitted. A dispatched run commits a turn-scoped `cancel.requested` event, coalesced by the original run identity. This is durable authority acceptance, not physical settlement. After the append, abort only a currently active controller whose prompt identity matches that run's Turn.
The wake runner rechecks the committed cancellation after its awaited attempt read, while holding its Turn slot and before calling the model. An unattempted cancelled input writes its own cancelled Turn result without calling the model, and follows normal run settlement. If this write fails, the slot is released and the scheduler retries the same input without blocking the Session or inventing an error result. A queued input waits behind any other active Turn; it does not interrupt it. Reopen derives cancellation from the journal again. An input already attempted keeps the existing interrupted-Turn recovery path: absence of a live controller never proves cancellation.
### Proof and races
Java completes this kind's cancel operation only when the committed `automation_run` is `cancelled`. A wire acknowledgment or a missing controller cannot complete it. A natural completed/failed outcome wins and remains unchanged; the operation reports `task_already_settled`. A lost reply retries the same target, and a lost run-settlement write reconciles from the original Turn result. Unproven outcomes retain the existing bounded attempts and `recovery_blocked` record-only reconciliation; a parked operation is never re-delivered.
The wake runner claims its slot and checks cancellation synchronously, with no intervening await. Cancellation arriving after that claim aborts the matching controller after committing its request. Cancellation arriving earlier is read by the runner before model start. Any attempted run whose crash leaves no terminal proof remains subject to the existing honest `failed/outcome_unknown` recovery semantics.
## Affected components
The control-plane task projection, cancel coordinator and Hosted connector route the stop. The Hosted Automation funnel commits it; the shared wake runner consumes queued cancellations. The OpenAPI task capability documentation and generated WebShell types describe the capability. Existing task UI and SDK entry points need no additional API. Focused tests cover each layer and the actual Hosted route.
## Validation and acceptance
First attempt the E2E plan with the global `qwen` baseline; then verify the local bundle and Java service. Cover claim-only and queued cancellation, active matching cancellation, sibling and foreground isolation, natural-completion races, repeat/reopen, unknown recovery, authorization, public/WebShell idempotency and parked reconciliation. Run affected TypeScript tests, root build/typecheck/bundle, Java focused tests and `mvn verify`; use disposable MySQL 8.4 and matching Hosted profiles for integration evidence. Record unavailable platform or deployed-environment acceptance explicitly.
For [#14446](https://github.com/QwenLM/qwen-code/issues/14446), FG11d must now expect per-run task cancellation and an unchanged schedule, rather than `task_action_unavailable`. That tracker update accompanies the PR; it does not claim the complete fault round has run.
## Risks and open questions
The main risk is cancelling or settling a different Turn across asynchronous boundaries; immutable target identity and the wake slot protect it. A queued cancellation may remain pending while unrelated work owns the Session. A cancellation acknowledgment can precede the physical result, while this kind's public receipt deliberately waits for positive cancelled-run evidence. No open contract decisions remain.
@@ -0,0 +1,43 @@
# 取消一次 Managed Automation 运行
[English](2026-10-18-automation-run-cancel.md) | [简体中文](2026-10-18-automation-run-cancel.zh-CN.md)
## 状态与问题
这是 [#14460](https://github.com/QwenLM/qwen-code/issues/14460) 的拟实现设计,基于 main `b3c5d3ed1e67`。Automation 运行已投影为任务,但不能声明或接受取消。现有运行结算已经能将已提交的 Turn 结果映射为 `cancelled`。
## 目标与边界
通过现有公开和 WebShell 任务入口取消一次运行,保留定时定义、后续触发、同级运行以及无关前台 Turn。`automation.enabled` 默认仍为 `false`。不增加公开路由、操作种类、记录字段、数据库迁移或 UI 控件。通用 `CANCEL` 恢复、Automation 可观测性、`per_run` Session 和完整 FG11 故障轮次仍单独处理。
## 设计
### 控制面
将 `automation_run` 纳入可取消的任务种类。持久受理、鉴权、幂等和操作租约继续使用 `TASK_CANCEL`。协调器通过私有 Automation 操作 `cancel_run` 指定原 run、schedule 和 occurrence;Hosted 路由先校验这个不可变组合派生的就是指定 run,再寻址它的 Turn。身份不匹配时拒绝请求,不取消任何运行。Hosted 连接器使用停止鉴权和被动挂载,不走新工作鉴权或整个 Session 的取消接管。重新加载可能在 Hosted 接受停止前启动排队运行;Java 受理本身不承诺已物理取消。
### Hosted 权威
取消与 Automation 漏斗的触发、结算写入串行化。仍处于 `intent` 的认领可以直接结算为 `cancelled/not_started_proven`,因为没有受理输入。已派发运行提交针对其 Turn 的 `cancel.requested` 事件,按原运行身份合并。这证明权威接受,不代表物理结算。追加完成后,只中止当前 prompt 身份与目标 Turn 匹配的控制器。
唤醒执行器在异步读取尝试记录后,持有 Turn 槽位并在调用模型前重新读取已提交的取消。尚未尝试且已取消的输入不调用模型,写入自己的取消 Turn 结果,再沿用正常运行结算。如果此写入失败,释放槽位,由调度器重试同一输入,不阻塞 Session,也不伪造错误结果。排队输入等待其他活动 Turn 结束,不中断它。重新打开时再次从日志派生取消。已经尝试的输入保留现有中断 Turn 恢复路径;没有活动控制器不能证明取消。
### 证明与竞争
Java 仅在已提交的 `automation_run` 为 `cancelled` 时完成此种取消操作。网络确认或控制器缺失都不能完成操作。自然完成或失败的结果优先且保持不变,操作报告 `task_already_settled`。响应丢失后仍重试同一目标,运行结算写入丢失则从原 Turn 结果对账。结果无法证明时保留现有有界重试和 `recovery_blocked` 仅凭记录对账;停放操作绝不再次投递。
唤醒执行器同步认领槽位并检查取消,中间没有 await。在认领后到达的取消先提交请求,再中止匹配控制器。更早到达的取消在启动模型前被执行器读到。已尝试运行若在崩溃后没有终态证据,继续遵守现有真实的 `failed/outcome_unknown` 恢复语义。
## 受影响组件
控制面的任务投影、取消协调器和 Hosted 连接器路由停止。Hosted Automation 漏斗提交停止,共用唤醒执行器消费排队取消。OpenAPI 的任务能力说明和生成的 WebShell 类型描述新能力。现有任务 UI 和 SDK 入口无需增加 API。针对性测试覆盖各层及实际 Hosted 路由。
## 验证与验收
先以全局 `qwen` 尝试 E2E 基线,再验证本地 bundle 和 Java 服务。覆盖仅认领和排队取消、匹配活动运行的取消、同级和前台隔离、自然完成竞争、重复请求及重开、未知结果恢复、鉴权、公开与 WebShell 幂等和停放对账。执行受影响 TypeScript 测试、根目录 build/typecheck/bundle、Java 针对性测试和 `mvn verify`;使用一次性 MySQL 8.4 和匹配的 Hosted profile 获取集成证据。明确记录无法执行的平台或已部署环境验收。
[#14446](https://github.com/QwenLM/qwen-code/issues/14446) 的 FG11d 应改为验收单次任务取消和定时定义不变,不再期待 `task_action_unavailable`。提交 PR 时同步该 tracker,但不声称已经完成整轮故障验收。
## 风险与待定问题
主要风险是跨异步边界取消或结算了另一 Turn,由不可变目标身份和唤醒槽位防护。无关工作占有 Session 时,排队取消可能保持待处理。取消确认可早于物理结果,此种任务的公开凭证则有意等待已取消运行的正向证据。没有待定契约决策。
@@ -206,6 +206,118 @@ function turnResult(
}
describe('hosted automation definitions', () => {
it('persists a targeted cancel across reopen without retiring the schedule', async () => {
const harness = await createHarness();
const runId = automationRunId(SCHEDULE_ID, SLOT_KEY);
const siblingId = automationRunId(SCHEDULE_ID, 'manual:sibling');
await withSession(harness, async (automations, authority) => {
await automations.define({
scheduleId: SCHEDULE_ID,
operationId: randomUUID(),
definition,
});
await automations.fire(fireParams());
await automations.fire(fireParams(1, 'manual:sibling'));
await automations.requestCancel(runId);
const before = authority.committedSequence;
await automations.requestCancel(runId);
expect(authority.committedSequence).toBe(before);
expect(automations.cancelRequested(automationInputId(runId))).toBe(true);
expect(automations.cancelRequested(automationInputId(siblingId))).toBe(
false,
);
expect(automations.run(runId)?.run.state).toBe('running');
expect(automations.schedule(SCHEDULE_ID)?.enabled).toBe(true);
});
await withSession(
harness,
async (automations) => {
expect(automations.cancelRequested(automationInputId(runId))).toBe(
true,
);
expect(automations.cancelRequested(automationInputId(siblingId))).toBe(
false,
);
expect(automations.schedule(SCHEDULE_ID)?.enabled).toBe(true);
},
{ create: false },
);
});
it('keeps a naturally completed result when cancellation arrives later', async () => {
const harness = await createHarness();
await withSession(harness, async (automations, authority) => {
await automations.define({
scheduleId: SCHEDULE_ID,
operationId: randomUUID(),
definition,
});
const fired = await automations.fire(fireParams());
harness.records.push(
turnRecord(harness, fired.inputId, 'user'),
turnResult(fired.inputId, 'completed'),
);
expect(
(await automations.requestCancel(fired.run.automationRunId)).run.state,
).toBe('settled');
expect(automations.cancelRequested(fired.inputId)).toBe(false);
const before = authority.committedSequence;
await automations.requestCancel(fired.run.automationRunId);
expect(authority.committedSequence).toBe(before);
});
});
it('cancels a claim before dispatch without admitting an input or retiring its definition', async () => {
const harness = await createHarness();
await withSession(harness, async (automations, authority) => {
const opened = await automations.define({
scheduleId: SCHEDULE_ID,
operationId: randomUUID(),
definition,
});
const claim = automationRunClaimBody({
schedule: opened.schedule,
occurrenceKey: SLOT_KEY,
});
await authority.commitExtensionRecord(
{
operation: 'fireRun',
commandId: `${claim.automationRunId}:1`,
sessionKey,
contentDigest: 'd'.repeat(64),
},
{ domain: 'automation_run', record: claim },
{ class: 'trusted_entry' },
);
const cancelled = await automations.requestCancel(claim.automationRunId);
expect(cancelled.run).toMatchObject({
state: 'cancelled',
execution: 'not_started_proven',
});
expect(
pendingSessionInputs(
authority.eventsInSequenceRange(1, authority.committedSequence),
),
).toEqual([]);
expect(automations.schedule(SCHEDULE_ID)?.enabled).toBe(true);
expect((await automations.fire(fireParams())).run).toEqual(cancelled);
expect(
(await automations.fire(fireParams(1, 'manual:next'))).run.run.state,
).toBe('running');
});
});
it('refuses a missing run without writing a cancellation', async () => {
const harness = await createHarness();
await withSession(harness, async (automations, authority) => {
const before = authority.committedSequence;
await expect(
automations.requestCancel(automationRunId(SCHEDULE_ID, SLOT_KEY)),
).rejects.toThrow(AutomationNotFoundError);
expect(authority.committedSequence).toBe(before);
});
});
it('opens, revises append-only, replays unchanged content and retires', async () => {
const harness = await createHarness();
await withSession(harness, async (automations, authority) => {
@@ -40,6 +40,7 @@ import {
import { isTerminalRunState } from '@qwen-code/qwen-code-core/managed-runtime/managed-extension-record.js';
import type {
CommittedExtensionOperation,
LocalManagedSessionAuthority,
ManagedSessionActor,
ManagedSessionCommand,
ManagedSessionInputRequest,
@@ -66,6 +67,7 @@ import { wakeHasPriorAttempt } from './hosted-monitor-wake.js';
export interface HostedAutomationStore {
readonly authority: {
readonly committedSequence: number;
appendExecutionEvent: LocalManagedSessionAuthority['appendExecutionEvent'];
eventsInSequenceRange(
from: number,
to: number,
@@ -243,6 +245,63 @@ export class HostedAutomationSession {
return existing ? parseAutomationRunRecord(existing.record) : undefined;
}
cancelRequested(turnId: string): boolean {
return this.store.authority
.eventsInSequenceRange(1, this.store.authority.committedSequence)
.some(
(event) =>
event.kind === 'cancel.requested' &&
(event.payload['target'] as { turnId?: unknown } | undefined)
?.turnId === turnId,
);
}
requestCancel(runId: string): Promise<AutomationRun> {
return this.serial(async () => {
const turnId = automationInputId(runId);
await this.settleRunUnserialized(turnId);
const run = this.run(runId);
if (run === undefined) throw new AutomationNotFoundError(runId);
if (isTerminalRunState(run.run.state)) return run;
if (run.run.execution === 'intent') {
const cancelled = automationRunSettleBody(run, 'cancelled', false);
await this.commitRun(
cancelled,
this.revisionOf(runId) + 1,
'settleRun',
);
return cancelled;
}
if (!this.cancelRequested(turnId)) {
const commandId = `automation-cancel:${runId}`;
await this.store.authority.appendExecutionEvent(
{
operation: 'requestCancel',
commandId,
contentDigest: digest(turnId),
sessionKey: this.key,
},
(sequence) => ({
v: 1,
sequence,
eventId: commandId,
sessionKey: this.key,
kind: 'cancel.requested',
occurredAt: Date.now(),
payload: {
requestId: commandId,
target: { turnId },
reason: 'user',
requestedBy: 'hosted',
},
}),
TRUSTED,
);
}
return run;
});
}
runs(): AutomationRun[] {
return this.store.authority
.extensionRecordsInDomain('automation_run')
@@ -2972,9 +2972,222 @@ describe('Hosted Harness no-tool session', () => {
expect(settled[0]!.payload['outcome']).toBe('completed');
});
it('cancels only the active automation occurrence and preserves later runs', async () => {
const server = await app();
const created = await headers(supertest(server).post('/session')).send({
sessionId: SESSION_ID,
sessionScope: 'thread',
managedSessionStore: store(),
});
expect(created.status).toBe(200);
const operations = (body: Record<string, unknown>) =>
headers(
supertest(server).post(`/session/${SESSION_ID}/automations/operations`),
)
.set('X-Qwen-Client-Id', created.body.clientId as string)
.send({ operationId: randomUUID(), ...body });
const cancel = (occurrenceKey = AUTOMATION_SLOT) =>
operations({
kind: 'cancel_run',
scheduleId: AUTOMATION_ID,
occurrenceKey,
automationRunId: automationRunId(AUTOMATION_ID, occurrenceKey),
});
expect(
(
await operations({
kind: 'cancel_run',
scheduleId: AUTOMATION_ID,
occurrenceKey: '',
automationRunId: automationRunId(AUTOMATION_ID, AUTOMATION_SLOT),
})
).status,
).toBe(400);
expect((await cancel('manual:missing')).status).toBe(404);
let signal!: AbortSignal;
let finish!: () => void;
state.model.mockImplementationOnce(
(input) =>
new Promise((_resolve, reject) => {
signal = input.signal;
finish = () => reject(new DOMException('Cancelled', 'AbortError'));
signal.addEventListener('abort', finish, { once: true });
}),
);
try {
expect(
(
await operations({
kind: 'define_schedule',
scheduleId: AUTOMATION_ID,
definition: automationDefinition,
})
).status,
).toBe(202);
expect((await operations(fireBody())).status).toBe(202);
await vi.waitFor(() => expect(state.model).toHaveBeenCalledTimes(1), {
timeout: 10_000,
});
expect(signal.aborted).toBe(false);
expect(
(
await operations({
kind: 'cancel_run',
scheduleId: AUTOMATION_ID,
occurrenceKey: AUTOMATION_SLOT,
automationRunId: automationRunId(AUTOMATION_ID, 'manual:other'),
})
).status,
).toBe(400);
expect(signal.aborted).toBe(false);
expect((await cancel()).status).toBe(202);
expect(signal.aborted).toBe(true);
await vi.waitFor(
async () =>
expect((await operations(fireBody())).body.run.state).toBe(
'cancelled',
),
{ timeout: 10_000 },
);
expect((await cancel()).body.run.state).toBe('cancelled');
expect(
(
await operations({
kind: 'define_schedule',
scheduleId: AUTOMATION_ID,
definition: automationDefinition,
})
).body.schedule,
).toMatchObject({ enabled: true, revision: 1 });
expect(
(await operations(fireBody(1, 'manual:later', 'manual'))).status,
).toBe(202);
await vi.waitFor(
async () =>
expect(
(await operations(fireBody(1, 'manual:later', 'manual'))).body.run
.state,
).toBe('settled'),
{ timeout: 10_000 },
);
expect((await cancel('manual:later')).body.run.state).toBe('settled');
expect(state.model).toHaveBeenCalledTimes(2);
} finally {
finish?.();
}
});
it('cancels one queued automation without aborting foreground or sibling work', async () => {
const server = await app();
const created = await headers(supertest(server).post('/session')).send({
sessionId: SESSION_ID,
sessionScope: 'thread',
managedSessionStore: store(),
});
expect(created.status).toBe(200);
const authorize = (request: supertest.Test) =>
headers(request).set('X-Qwen-Client-Id', created.body.clientId as string);
const operations = (body: Record<string, unknown>) =>
authorize(
supertest(server).post(`/session/${SESSION_ID}/automations/operations`),
).send({ operationId: randomUUID(), ...body });
let signal!: AbortSignal;
let finish!: () => void;
state.model.mockImplementationOnce(
(input) =>
new Promise((resolve) => {
signal = input.signal;
finish = () =>
resolve({ text: 'foreground done', model: 'test-model' });
}),
);
const prompt = [{ type: 'text', text: 'foreground held' }];
const payloadDigest = `sha256:${createHash('sha256').update(JSON.stringify(prompt)).digest('hex')}`;
try {
await authorize(supertest(server).post(`/session/${SESSION_ID}/prompt`))
.send({ prompt, promptId: PROMPT_ID, payloadDigest })
.expect(202);
await vi.waitFor(() => expect(state.model).toHaveBeenCalledTimes(1), {
timeout: 10_000,
});
expect(
(
await operations({
kind: 'define_schedule',
scheduleId: AUTOMATION_ID,
definition: automationDefinition,
})
).status,
).toBe(202);
expect((await operations(fireBody())).status).toBe(202);
expect(
(await operations(fireBody(1, 'manual:sibling', 'manual'))).status,
).toBe(202);
expect(
(
await operations({
kind: 'cancel_run',
scheduleId: AUTOMATION_ID,
occurrenceKey: AUTOMATION_SLOT,
automationRunId: automationRunId(AUTOMATION_ID, AUTOMATION_SLOT),
})
).status,
).toBe(202);
expect(signal.aborted).toBe(false);
expect(state.model).toHaveBeenCalledTimes(1);
finish();
await vi.waitFor(
async () => {
expect((await operations(fireBody())).body.run).toMatchObject({
state: 'cancelled',
execution: 'not_started_proven',
});
expect(
(await operations(fireBody(1, 'manual:sibling', 'manual'))).body.run
.state,
).toBe('settled');
},
{ timeout: 10_000 },
);
expect(state.model).toHaveBeenCalledTimes(2);
expect(signal.aborted).toBe(false);
} finally {
finish?.();
}
});
it('consumes a durable queued automation cancel on load without a model call', async () => {
await prewriteAutomationSession(undefined, undefined, true);
const server = await app();
const loaded = await headers(
supertest(server).post(`/session/${SESSION_ID}/load`),
).send({
managedSessionStore: store(),
});
expect(loaded.status).toBe(200);
await vi.waitFor(
async () => {
const replay = await headers(
supertest(server).post(
`/session/${SESSION_ID}/automations/operations`,
),
)
.set('X-Qwen-Client-Id', loaded.body.clientId as string)
.send({ operationId: randomUUID(), ...fireBody() });
expect(replay.body.run).toMatchObject({
state: 'cancelled',
execution: 'not_started_proven',
});
},
{ timeout: 10_000 },
);
expect(state.model).not.toHaveBeenCalled();
});
async function prewriteAutomationSession(
toolProfile?: string,
captureBytes?: number,
cancel = false,
): Promise<string> {
const key = {
tenantId: 'tenant',
@@ -3041,6 +3254,7 @@ describe('Hosted Harness no-tool session', () => {
trigger: 'scheduled',
firedAt: Date.parse('2026-03-08T07:00:05Z'),
});
if (cancel) await automations.requestCancel(fired.run.automationRunId);
return fired.inputId;
} finally {
await managed.close().catch(() => undefined);
@@ -3051,7 +3265,9 @@ describe('Hosted Harness no-tool session', () => {
// the journal carries the committed input, the transcript carries the
// turn attempt, and the hook pin keeps a bare load from refusing the
// crash the way a parked prompt would refuse it without one.
async function prewriteAutomationCrashedTurnSession(): Promise<string> {
async function prewriteAutomationCrashedTurnSession(
cancel = false,
): Promise<string> {
const key = {
tenantId: 'tenant',
workspaceId: 'workspace',
@@ -3129,6 +3345,7 @@ describe('Hosted Harness no-tool session', () => {
daemonPromptId: fired.inputId,
message: { role: 'user', parts: [{ text: 'queued' }] },
} as never);
if (cancel) await automations.requestCancel(fired.run.automationRunId);
return fired.inputId;
} finally {
await managed.close().catch(() => undefined);
@@ -6150,31 +6367,55 @@ describe('Hosted Harness no-tool session', () => {
await headers(supertest(server).delete(`/session/${SESSION_ID}`));
}, 45_000);
it('settles an automation run as failed/unknown when its turn went recovery-blocked', async () => {
await prewriteAutomationCrashedTurnSession();
mockBrokerBroker();
const server = await app(true);
const loaded = await headers(
supertest(server).post(`/session/${SESSION_ID}/load`),
).send({ managedSessionStore: store() });
expect(loaded.status).toBe(200);
const authorize = (request: supertest.Test) =>
headers(request).set('X-Qwen-Client-Id', loaded.body.clientId as string);
await vi.waitFor(
async () => {
const replayed = await authorize(
supertest(server).post(
`/session/${SESSION_ID}/automations/operations`,
),
).send({ operationId: randomUUID(), ...fireBody() });
expect(replayed.status).toBe(202);
expect(replayed.body.run.state).toBe('failed');
expect(replayed.body.run.execution).toBe('outcome_unknown');
},
{ timeout: 10_000, interval: 50 },
);
await headers(supertest(server).delete(`/session/${SESSION_ID}`));
});
it.each([false, true])(
'settles a crashed automation as failed/unknown with durable cancel=%s',
async (cancel) => {
const inputId = await prewriteAutomationCrashedTurnSession(cancel);
mockBrokerBroker();
const server = await app(true);
const loaded = await headers(
supertest(server).post(`/session/${SESSION_ID}/load`),
).send({ managedSessionStore: store() });
expect(loaded.status).toBe(200);
const authorize = (request: supertest.Test) =>
headers(request).set(
'X-Qwen-Client-Id',
loaded.body.clientId as string,
);
await vi.waitFor(
async () => {
const replayed = await authorize(
supertest(server).post(
`/session/${SESSION_ID}/automations/operations`,
),
).send({ operationId: randomUUID(), ...fireBody() });
expect(replayed.status).toBe(202);
expect(replayed.body.run.state).toBe('failed');
expect(replayed.body.run.execution).toBe('outcome_unknown');
},
{ timeout: 10_000, interval: 50 },
);
await vi.waitFor(async () => {
const journal = await LocalJsonlManagedSessionJournalStore.read(
path.join(state.root, `${SESSION_ID}.jsonl`),
{
tenantId: 'tenant',
workspaceId: 'workspace',
sessionId: SESSION_ID,
},
);
const settled = journal.events.filter(
(event) =>
event.kind === 'turn.settled' &&
event.payload['turnId'] === inputId,
);
expect(settled).toHaveLength(1);
expect(settled[0]!.payload['outcome']).toBe('error');
});
expect(state.model).not.toHaveBeenCalled();
await headers(supertest(server).delete(`/session/${SESSION_ID}`));
},
);
it('settles, consumes and unblocks on a crash, and later fires run once', async () => {
await prewriteAutomationCrashedTurnSession();
@@ -199,6 +199,7 @@ import {
} from './hosted-automation-session.js';
import {
AUTOMATION_INPUT_SOURCE,
automationInputId,
MANAGED_AUTOMATION_LIMITS,
automationRunId,
} from '@qwen-code/qwen-code-core/managed-runtime/managed-automation-operations.js';
@@ -4918,6 +4919,9 @@ export function registerHostedHarnessSessionRoutes(
),
busy: wakeBusy,
held: (turn) => stoppedMessage(turn.source),
cancelled: (turn) =>
turn.source === AUTOMATION_INPUT_SOURCE &&
session.automations?.cancelRequested(turn.turnId) === true,
needsRecovery: monitorWakeNeedsRecovery,
// F5/direction (a): channel turns interrupted mid-flight get
// terminal settlement from their own funnel; monitor turns
@@ -10395,6 +10399,29 @@ export function registerHostedHarnessSessionRoutes(
};
break;
}
case 'cancel_run': {
const occurrenceKey = body?.['occurrenceKey'];
if (
typeof occurrenceKey !== 'string' ||
occurrenceKey.length < 1 ||
occurrenceKey.length > 512 ||
body?.['automationRunId'] !==
automationRunId(scheduleId, occurrenceKey)
) {
return error(res, 400, 'invalid_automation_operation');
}
const runId = automationRunId(scheduleId, occurrenceKey);
const run = await automations.requestCancel(runId);
const turnId = automationInputId(runId);
if (
automations.cancelRequested(turnId) &&
session.active?.promptId === turnId
) {
session.active.abort.abort();
}
result = { run: runSummary(run) };
break;
}
case 'reconcile_run': {
const occurrenceKey = body?.['occurrenceKey'];
if (
@@ -11,7 +11,10 @@ import {
withChildAgentConsumption,
} from './hosted-monitor-wake-turn.js';
import { HostedToolRecoveryRequiredError } from './hosted-workspace-tool-turn.js';
import type { HostedMonitorWakeTurn } from './hosted-monitor-wake.js';
import {
HostedMonitorWakeScheduler,
type HostedMonitorWakeTurn,
} from './hosted-monitor-wake.js';
import type { HostedChildAgentSession } from './hosted-child-agent-session.js';
function world() {
@@ -53,6 +56,121 @@ function harness(
}
describe('withChildAgentConsumption', () => {
it('settles a queued cancellation without entering the model', async () => {
const { session } = world();
const execute = vi.fn();
const runTurn = createMonitorWakeRunTurn({
session,
sessionId: 'session-1',
cwd: '/work',
executeHostedTurn: execute,
busy: () => false,
needsRecovery: () => false,
writeStderr: () => {},
cancelled: () => true,
});
await expect(runTurn(TURN)).resolves.toBe('settled_incomplete');
expect(execute).not.toHaveBeenCalled();
expect(session.managed.sink.write).toHaveBeenCalledOnce();
expect(session.managed.sink.write).toHaveBeenCalledWith(
expect.objectContaining({
subtype: 'turn_result',
systemPayload: expect.objectContaining({
promptId: TURN.turnId,
state: 'cancelled',
}),
}),
);
});
it('rechecks cancellation after awaiting the attempt read', async () => {
const { session } = world();
let cancelled = false;
session.managed.sink.project.mockImplementationOnce(async () => {
cancelled = true;
return [];
});
const execute = vi.fn();
const runTurn = createMonitorWakeRunTurn({
session,
sessionId: 'session-1',
cwd: '/work',
executeHostedTurn: execute,
busy: () => false,
needsRecovery: () => false,
writeStderr: () => {},
cancelled: () => cancelled,
});
await expect(runTurn(TURN)).resolves.toBe('settled_incomplete');
expect(execute).not.toHaveBeenCalled();
});
it('retries a failed queued-cancel commit through the scheduler and continues later work', async () => {
const { session } = world();
const queue = [TURN, { ...TURN, turnId: 'later-turn' }];
session.managed.sink.write
.mockRejectedValueOnce(new Error('store unavailable'))
.mockImplementationOnce(async () => {
queue.shift();
});
const execute = vi.fn(async () => {
queue.shift();
return { systemPayload: { state: 'completed' } };
});
const writeStderr = vi.fn();
const runTurn = createMonitorWakeRunTurn({
session,
sessionId: 'session-1',
cwd: '/work',
executeHostedTurn: execute,
busy: () => false,
needsRecovery: () => false,
writeStderr,
cancelled: (turn) => turn.turnId === TURN.turnId,
});
const failed = vi.fn(() => {
session.blocked = true;
});
const scheduler = new HostedMonitorWakeScheduler(
{
next: async () => queue[0],
state: () => (session.blocked ? 'blocked' : 'idle'),
runTurn,
failed,
},
10,
);
vi.useFakeTimers();
try {
scheduler.kick();
await vi.advanceTimersByTimeAsync(50);
expect(session.blocked).toBe(false);
expect(failed).not.toHaveBeenCalled();
expect(queue).toEqual([]);
expect(session.managed.sink.write).toHaveBeenCalledTimes(2);
expect(session.managed.sink.write).toHaveBeenCalledWith(
expect.objectContaining({
systemPayload: expect.objectContaining({
promptId: TURN.turnId,
state: 'cancelled',
}),
}),
);
expect(execute).toHaveBeenCalledOnce();
expect(execute).toHaveBeenCalledWith(
'later-turn',
TURN.text,
expect.any(AbortController),
);
expect(writeStderr).toHaveBeenCalledWith(
expect.stringContaining('store unavailable'),
);
} finally {
scheduler.close();
vi.useRealTimers();
}
});
it('marks the result consumed after the wake turn settles durably', async () => {
const { session, agents } = world();
const turn = withChildAgentConsumption(
@@ -71,6 +71,7 @@ export function createMonitorWakeRunTurn(params: {
/** A turn its source no longer runs (H4d-b: a stopped run's message),
* re-read with the busy verdict right before the turn starts. */
readonly held?: (turn: HostedMonitorWakeTurn) => boolean;
readonly cancelled?: (turn: HostedMonitorWakeTurn) => boolean;
/**
* The recovery-required classification. It is injected because these
* error classes must be matched by type — their
@@ -160,6 +161,37 @@ export function createMonitorWakeRunTurn(params: {
if (params.busy() || session.blocked || params.held?.(turn)) return 'busy';
const abort = new AbortController();
session.active = { promptId: turn.turnId, digest: '', abort };
if (params.cancelled?.(turn)) {
try {
await session.managed.sink.write({
uuid: randomUUID(),
parentUuid: null,
sessionId: params.sessionId,
timestamp: new Date().toISOString(),
type: 'system',
cwd: params.cwd,
version: 'hosted-harness/1',
subtype: 'turn_result',
systemPayload: {
promptId: turn.turnId,
state: 'cancelled',
stopReason: 'cancelled',
endedAt: Date.now(),
},
} as ChatRecord);
return 'settled_incomplete';
} catch (cause) {
params.writeStderr(
'qwen serve: Queued wake cancellation ' +
turn.turnId +
' could not settle this pass; the next wake attempt drives it again: ' +
String(cause),
);
return 'busy';
} finally {
session.active = undefined;
}
}
let turnResult: { systemPayload?: Record<string, unknown> } | null;
try {
turnResult = (await params.executeHostedTurn(
@@ -117,8 +117,9 @@ public interface HarnessConnector extends AutoCloseable {
/**
* H6b/H6c: one automation operation onto the Session's journal, from the
* control plane's automation service and scanner (define_schedule,
* retire_schedule, fire_run). The Hosted side settles it before
* answering, and the answer carries the operation's result.
* retire_schedule, fire_run, reconcile_run, cancel_run). The answer
* carries the committed operation result; a cancel_run may still be
* awaiting its Turn's physical settlement.
*/
default Map<String, Object> runAutomationOperation(String tenantId,
String sessionId, Map<String, Object> body) {
@@ -690,6 +690,11 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
public Map<String, Object> runAutomationOperation(String tenantId,
String sessionId, Map<String, Object> body) {
try {
if ("cancel_run".equals(body.get("kind"))) {
HarnessSessionRef ref = committedStopAttachment(tenantId,
sessionId, false);
return client().runAutomationOperation(ref, body);
}
// Automation operations drive new work (a fire admits an input
// and wakes a turn), so a cached attachment is no admission:
// the Workspace checks submit and continueManagedRuntime run
@@ -776,6 +781,11 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
*/
private HarnessSessionRef messageStopAttachment(String tenantId,
String sessionId) {
return committedStopAttachment(tenantId, sessionId, true);
}
private HarnessSessionRef committedStopAttachment(String tenantId,
String sessionId, boolean stopMessages) {
SessionRecord session = sessions.requireSession(tenantId, sessionId);
requireShellApproval(session);
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
@@ -793,8 +803,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
}
workspaceExecution.authorizeCommittedStop(session);
}
HarnessSessionRef attached = client().loadSession(
new LoadHarnessSession(session.sessionId(),
LoadHarnessSession load = new LoadHarnessSession(session.sessionId(),
managedSessionStore(session),
session.harnessBootId() != null, toolProfile(session),
false, false,
@@ -803,8 +812,9 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
.withMcpServers(mcpServers(session))
.withHookCatalog(hookCatalog(session))
.withForegroundOnly(foregroundOnly(session))
.withChildWorkspaces(childWorkspaces && !foregroundOnly(session), childSnapshots && !foregroundOnly(session))
.withStoppedMessages());
.withChildWorkspaces(childWorkspaces && !foregroundOnly(session), childSnapshots && !foregroundOnly(session));
HarnessSessionRef attached = client().loadSession(
stopMessages ? load.withStoppedMessages() : load);
if (session.workspace() != null && !actions.approvalMode(tenantId,
sessionId).equals(attached.getApprovalMode())) {
throw new IllegalStateException(
@@ -65,7 +65,10 @@ import java.util.concurrent.atomic.AtomicBoolean;
* <li>a planned channel delivery takes its cancel through the Hosted
* Harness's channel operation route, whose {@code cancelRequested} flag is
* the request: a planned one settles cancelled with it, and one an adapter
* claimed meanwhile keeps sending with the flag recorded.</li>
* claimed meanwhile keeps sending with the flag recorded;</li>
* <li>an Automation run takes a turn-scoped request through the Hosted
* Harness; its cancelled terminal record proves the stop, so its receipt
* waits for settlement rather than trusting the request acknowledgment.</li>
* </ul>
* A delivery whose outcome stays unproven retries with the dispatch
* backoff, bounded at {@link #ATTEMPT_BUDGET} claims; past it the
@@ -239,6 +242,10 @@ public class TaskCancelCoordinator implements AcceptanceEvidence {
return new Verdict(TaskCancelOutcome.failed(
"task_action_unavailable"), target);
}
if ("automation_run".equals(target.kind())
&& "cancelled".equals(target.state())) {
return new Verdict(TaskCancelOutcome.completed(), target);
}
// The stopReason clause only adds a Monitor's settlement: for a
// child_run (child_agent, workflow, background_shell) the record
// invariant "childRun stop_requested needs its stop request"
@@ -406,9 +413,20 @@ public class TaskCancelCoordinator implements AcceptanceEvidence {
/**
* A child Session run's stop request onto the parent's journal, a
* background process's stop through the Broker, or a delivery's cancel
* onto its route Session's journal; all replay-safe.
* onto its route Session's journal, or a stop of one Automation wake
* Turn; all replay-safe.
*/
private void send(OperationRecord operation, TaskTarget target) {
if ("automation_run".equals(target.kind())) {
harness.runAutomationOperation(operation.tenantId(),
operation.sessionId(), Map.of(
"operationId", attemptId(operation),
"kind", "cancel_run",
"automationRunId", target.recordId(),
"scheduleId", target.body().path("scheduleId").asText(),
"occurrenceKey", target.body().path("occurrenceKey").asText()));
return;
}
if ("channel_delivery".equals(target.kind())) {
harness.runChannelOperation(operation.tenantId(),
operation.sessionId(), Map.of(
@@ -127,12 +127,13 @@ public final class ManagedExtensionProjection {
* the process through the Broker, and the worker's stop-marked end
* settles the record {@code stop_requested} (A8). A channel delivery
* (#13746 F3) cancels only while no adapter has claimed it: its
* cancelRequested flag is its stop request. An automation run has no
* stop path yet.
* cancelRequested flag is its stop request. An automation run stops
* its own wake Turn and proves cancellation through its cancelled
* terminal record.
*/
public static final Set<String> CANCELLABLE_TASK_KINDS = Set.of(
"child_agent", "workflow", "background_shell", "monitor",
"channel_delivery");
"channel_delivery", "automation_run");
/** Task states a cancel may still act on: the run has not ended and
* is not waiting on recovery reconciliation. */
private static final Set<String> CANCELLABLE_TASK_STATES = Set.of(
@@ -2,8 +2,8 @@
"openapi": "3.1.0",
"info": {
"title": "Qwen Managed Agent Public and WebShell API",
"version": "1.64.0",
"description": "Canonical contract for the public Managed Agent resources and the WebShell adapter. partial means that the route exists but generated types and full contract tests are pending; planned means that the route or field is not implemented. Workspace context and cwd operations are v1.9 target contracts, not current server capabilities. v1.10 adds planned command admission, Actions, lifecycle and actor authorization. v1.12 refines planned W0 creation discovery, actor-specific create hints and explicit default selection. Filter planned fields as well as routes from production SDKs. v1.13 moves this contract into the Qwen Code repository as its single source and records the shipped rename, archive, unarchive and delete routes as partial. v1.14 integrates W0d workspace discovery, the WebShell workspace lookup, and empty-session binding as partial; workspace_binding/workspaceBinding advertises this narrow flow without enabling workspace execution or context switching. v1.15 adds the archived, archiving and deleting Session statuses and the cancelling Turn status, requires request_id in error envelopes, declares the error responses that the server returns, and marks Session create, list and get on both surfaces implemented. v1.16 adds the planned Stage H task contract (SessionTaskView as PublicTask, task list, detail, events and cancel) and names the MCP catalog, hook catalog, automation and channel resources as planned; their shapes arrive with Stages H1 to H6. v1.17 implements event replay: events carry their schema and projection versions and a top-level Item and Part identity, JSON event pages return has_more and next_cursor and accept limits up to 1000, and a persisted replay floor answers expired cursors with 409 cursor_expired in JSON and one agent.session.resync_required frame over SSE; it marks the public event query and stream, the WebShell event stream and the WebShell transcript implemented. v1.18 implements the durable Session lifecycle (Stage D4): close, archive and delete answer 202 with a command operation on both surfaces, an archive requires a closed Session, a deleted Session leaves a tombstone whose operations stay readable, and the operation query serves them; unarchive restores a closed Session, and Session capabilities advertise session_lifecycle. v1.19 serves the task list and detail on both surfaces as partial (Stage H0c): the Session store projects them from the Stage H records the Session authority commits, a task.updated Session event announces each change of a task's view, and Session capabilities advertise tasks. Task events and cancel stay planned until a slice's tasks produce output and accept a cancel. v1.20 implements the Turn read model (Stage D5): the public Turn list and detail read a Session's Turns without their input, newest first by creation time in milliseconds and then Turn ID, with an opaque cursor; a deleted Session's Turns are not readable, and the durable admission fields of a Turn stay planned until D7. v1.21 declares on the task list, detail and event routes of both surfaces the 403 actor_scope_mismatch that the tenant filter answers on every /v1/agents/ and WebShell route, as the Session and Turn reads already do, and the shared Forbidden response names it; a caller that cannot read a task still gets 404. v1.22 adds deployment-opted-in creation with an initial Workspace file-tool Turn (G0) on the public and WebShell surfaces; the fixed server-owned profile does not enable later Workspace operations. v1.23 settles the planned task event and cancel semantics: a durable retention floor, committed-prefix publication, stable cursors, Artifact visibility before expiry, bounded backlog under archival failure, ordered idempotent cancellation and command outcomes. Clients tolerate unknown optional event fields across minor versions. Task events and cancel remain planned. v1.24 serves the read-only Session MCP catalog as partial (Stage H1), projecting display metadata and schemas from committed records without Runtime identities, credentials or production profile enablement. v1.25 implements permission Actions (Stage D6b) on both surfaces: list, get and respond, projected from the Session's action.changed records; a WebShell permission Action carries inputRevision, policyRevision, functionCallId, toolName and expiresAt, its option ids allow and deny are stable, a Turn has at most one requested approval at a time, and a response is a durable command operation carrying requestId. v1.26 declares the tenant filter's 403 actor_scope_mismatch on every covered public and WebShell route, including planned routes, and pins the refusal code in contract traffic. v1.27 adds O3 durable tool-result projection, artifact metadata, immutable range/streaming content and WebShell result reads. Artifact capability requires a configured O3 reader and a Workspace-bound Session; execution admission remains independent. v1.28 admits later Turns of a Workspace-bound Session for the SLine truncated
"version": "1.65.0",
"description": "Canonical contract for the public Managed Agent resources and the WebShell adapter. partial means that the route exists but generated types and full contract tests are pending; planned means that the route or field is not implemented. Workspace context and cwd operations are v1.9 target contracts, not current server capabilities. v1.10 adds planned command admission, Actions, lifecycle and actor authorization. v1.12 refines planned W0 creation discovery, actor-specific create hints and explicit default selection. Filter planned fields as well as routes from production SDKs. v1.13 moves this contract into the Qwen Code repository as its single source and records the shipped rename, archive, unarchive and delete routes as partial. v1.14 integrates W0d workspace discovery, the WebShell workspace lookup, and empty-session binding as partial; workspace_binding/workspaceBinding advertises this narrow flow without enabling workspace execution or context switching. v1.15 adds the archived, archiving and deleting Session statuses and the cancelling Turn status, requires request_id in error envelopes, declares the error responses that the server returns, and marks Session create, list and get on both surfaces implemented. v1.16 adds the planned Stage H task contract (SessionTaskView as PublicTask, task list, detail, events and cancel) and names the MCP catalog, hook catalog, automation and channel resources as planned; their shapes arrive with Stages H1 to H6. v1.17 implements event replay: events carry their schema and projection versions and a top-level Item and Part identity, JSON event pages return has_more and next_cursor and accept limits up to 1000, and a persisted replay floor answers expired cursors with 409 cursor_expired in JSON and one agent.session.resync_required frame over SSE; it marks the public event query and stream, the WebShell event stream and the WebShell transcript implemented. v1.18 implements the durable Session lifecycle (Stage D4): close, archive and delete answer 202 with a command operation on both surfaces, an archive requires a closed Session, a deleted Session leaves a tombstone whose operations stay readable, and the operation query serves them; unarchive restores a closed Session, and Session capabilities advertise session_lifecycle. v1.19 serves the task list and detail on both surfaces as partial (Stage H0c): the Session store projects them from the Stage H records the Session authority commits, a task.updated Session event announces each change of a task's view, and Session capabilities advertise tasks. Task events and cancel stay planned until a slice's tasks produce output and accept a cancel. v1.20 implements the Turn read model (Stage D5): the public Turn list and detail read a Session's Turns without their input, newest first by creation time in milliseconds and then Turn ID, with an opaque cursor; a deleted Session's Turns are not readable, and the durable admission fields of a Turn stay planned until D7. v1.21 declares on the task list, detail and event routes of both surfaces the 403 actor_scope_mismatch that the tenant filter answers on every /v1/agents/ and WebShell route, as the Session and Turn reads already do, and the shared Forbidden response names it; a caller that cannot read a task still gets 404. v1.22 adds deployment-opted-in creation with an initial Workspace file-tool Turn (G0) on the public and WebShell surfaces; the fixed server-owned profile does not enable later Workspace operations. v1.23 settles the planned task event and cancel semantics: a durable retention floor, committed-prefix publication, stable cursors, Artifact visibility before expiry, bounded backlog under archival failure, ordered idempotent cancellation and command outcomes. Clients tolerate unknown optional event fields across minor versions. Task events and cancel remain planned. v1.24 serves the read-only Session MCP catalog as partial (Stage H1), projecting display metadata and schemas from committed records without Runtime identities, credentials or production profile enablement. v1.25 implements permission Actions (Stage D6b) on both surfaces: list, get and respond, projected from the Session's action.changed records; a WebShell permission Action carries inputRevision, policyRevision, functionCallId, toolName and expiresAt, its option ids allow and deny are stable, a Turn has at most one requested approval at a time, and a response is a durable command operation carrying requestId. v1.26 declares the tenant filter's 403 actor_scope_mismatch on every covered public and WebShell route, including planned routes, and pins the refusal code in contract traffic. v1.27 adds O3 durable tool-result projection, artifact metadata, immutable range/streaming content and WebShell result reads. Artifact capability requires a configured O3 reader and a Workspace-bound Session; execution admission remains independent. v1.28 admits later Turns of a Workspace-bound Session for the SLine truncated
},
"servers": [
{
@@ -8497,7 +8497,7 @@
"type": "string",
"enum": ["cancel", "send_input", "read_output"],
"x-qwen-implementation-status": "partial",
"description": "Actions the task supports now, the same for every caller. cancel: the cancel route accepts a new command for this task (since 1.40 for child_agent, 1.46 for workflow and 1.54 for background_shell and monitor, tasks of those kinds in pending, running, waiting or degraded, and since 1.57 for channel_delivery, a delivery still pending because no adapter has claimed it, each in an active Session); whether a caller may use it is an authorization check (403 on the cancel route). send_input: reserved for a later capability route. read_output: the task events route returns output events for this task to any caller that can read it. read_output does not change during the task's life, and a task without it produces no output events: its output goes only to Artifacts, so the events route never filters out events that exist."
"description": "Actions the task supports now, the same for every caller. cancel: the cancel route accepts a new command for this task (since 1.40 for child_agent, 1.46 for workflow and 1.54 for background_shell and monitor, tasks of those kinds in pending, running, waiting or degraded, and since 1.57 for channel_delivery, a delivery still pending because no adapter has claimed it, each in an active Session, and since 1.65 for automation_run in pending, running, waiting or degraded in an active Session; only that occurrence is cancelled and its schedule is preserved); whether a caller may use it is an authorization check (403 on the cancel route). send_input: reserved for a later capability route. read_output: the task events route returns output events for this task to any caller that can read it. read_output does not change during the task's life, and a task without it produces no output events: its output goes only to Artifacts, so the events route never filters out events that exist."
},
"PublicTask": {
"type": "object",
@@ -11,6 +11,7 @@ import com.alibaba.qwen.code.daemon.ManagedSessionStoreConnection;
import com.alibaba.qwen.code.daemon.PromptReceipt;
import com.alibaba.qwen.code.daemon.StreamHarnessEvents;
import com.alibaba.qwen.code.daemon.SubmitHarnessTurn;
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
@@ -243,6 +244,7 @@ class HostedHarnessMySqlIT {
"--qwen.managed-agent.harness.base-url=http://127.0.0.1:" + harnessPort,
"--qwen.managed-agent.harness.token=" + TOKEN,
"--qwen.managed-agent.harness.capability-digest=" + DIGEST,
"--qwen.managed-agent.trusted-actor-header=X-Qwen-Actor-Id",
"--qwen.managed-agent.runtime-broker.enabled=false",
"--qwen.managed-agent.dispatch.scan-delay=100ms");
try {
@@ -272,6 +274,7 @@ class HostedHarnessMySqlIT {
api("GET", "/v1/agents/sessions/" + id, null, null, 404);
assertThat(writerState(id)).isEqualTo("DELETED");
assertThat(modelRequests).hasSize(1);
cancelAutomationOccurrence();
assertThat(modelFailure.get()).isNull();
} catch (Exception | AssertionError failure) {
System.err.println("HOSTED_MYSQL_HARNESS_LOG\n" + logs());
@@ -279,6 +282,57 @@ class HostedHarnessMySqlIT {
}
}
private void cancelAutomationOccurrence() throws Exception {
String id = api("POST", "/v1/agents/sessions", "automation-session", """
{"agent_id":"qwen-code",
"input":[{"type":"input_text","text":"AUTOMATION_INIT"}]}
""", 202).path("id").asText();
String sessionPath = "/v1/agents/sessions/" + id;
await(() -> api("GET", sessionPath + "/events", null, null, 200)
.path("data").toString().contains("\"turn.completed\""),
SETTLEMENT_POLL_SECONDS, "Automation Session ready");
// Public Automation creation requires a bound Workspace; prepare
// this text-only fixture through the real Harness instead.
HarnessConnector harness = spring.getBean(HarnessConnector.class);
String schedule = "asch_" + UUID.randomUUID().toString().replace("-", "");
Map<String, Object> definition = Map.of(
"operationId", UUID.randomUUID().toString(),
"kind", "define_schedule", "scheduleId", schedule,
"definition", Map.of("goal", "Cancel one occurrence", "cron", "0 0 1 1 *",
"timezone", "UTC", "prompt", "MYSQL_CANCEL"));
harness.runAutomationOperation(tenant, id, definition);
harness.runAutomationOperation(tenant, id, Map.of(
"operationId", UUID.randomUUID().toString(),
"kind", "fire_run", "scheduleId", schedule,
"definitionRevision", 1, "occurrenceKey", "manual:cancel",
"trigger", "manual", "firedAt", System.currentTimeMillis()));
assertThat(cancellationRequested.await(15, TimeUnit.SECONDS))
.as("The automation wake reaches the actual model request").isTrue();
JsonNode tasks = api("GET", sessionPath + "/tasks", null, null, 200).path("data");
JsonNode task = java.util.stream.StreamSupport.stream(tasks.spliterator(), false)
.filter(value -> "automation_run".equals(value.path("kind").asText()))
.findFirst().orElseThrow();
assertThat(task.path("action_capabilities").toString()).contains("cancel");
String taskPath = sessionPath + "/tasks/" + task.path("id").asText();
JsonNode admitted = api("POST", taskPath + "/cancel", "automation-cancel", null, 202);
assertThat(admitted.path("type").asText()).isEqualTo("task_cancel");
JsonNode completed = awaitOperation(id, admitted.path("id").asText());
assertThat(completed.path("admission_stage").asText()).isEqualTo("harness_confirmed");
assertThat(api("GET", taskPath, null, null, 200).path("state").asText())
.isEqualTo("cancelled");
assertThat(api("POST", taskPath + "/cancel", "automation-cancel", null, 202)
.path("id").asText()).isEqualTo(admitted.path("id").asText());
JsonNode preserved = json.valueToTree(harness.runAutomationOperation(tenant, id,
Map.of("operationId", UUID.randomUUID().toString(),
"kind", "define_schedule", "scheduleId", schedule,
"definition", Map.of())).get("schedule"));
assertThat(preserved.path("enabled").asBoolean()).isTrue();
assertThat(preserved.path("revision").asLong()).isEqualTo(1);
releaseCancellation.countDown();
JsonNode closing = api("POST", sessionPath + "/close", "automation-close", null, 202);
awaitOperation(id, closing.path("id").asText());
}
private JsonNode awaitOperation(String session, String operationId)
throws InterruptedException {
AtomicReference<JsonNode> operation = new AtomicReference<>();
@@ -297,6 +351,7 @@ class HostedHarnessMySqlIT {
URI.create("http://127.0.0.1:" + springPort + path))
.timeout(Duration.ofSeconds(10))
.header("X-Qwen-Tenant-Id", tenant)
.header("X-Qwen-Actor-Id", "actor")
.header("Accept", "application/json")
.method(method, body == null ? HttpRequest.BodyPublishers.noBody()
: HttpRequest.BodyPublishers.ofString(body));
@@ -2054,6 +2054,37 @@ class ManagedAgentApiContractTest {
assertThat(workflowWebShell.get("operationId").asText())
.isEqualTo(workflowAdmitted.get("id").asText());
assertThat(workflowWebShell.get("replayed").asBoolean()).isTrue();
String automationKey = "1".repeat(64);
jdbc.update("UPDATE ma_operation SET state = 'COMPLETED'"
+ " WHERE tenant_id = ? AND session_id = ?"
+ " AND operation_kind = 'TASK_CANCEL'"
+ " AND state IN ('PENDING', 'RUNNING')", tenant, sessionId);
jdbc.update("INSERT INTO ma_session_record (session_scope_key, record_key, tenant_id, workspace_id, session_id, domain, record_id, operation_hash, revision, record_resource_id, task_kind, task_state, runtime_state, delivery_target, delivery_state, gmt_create, started_at) VALUES (?, ?, ?, 'workspace-contract', ?, 'automation_run', 'arun-contract', ?, 2, 'resource-automation-contract', 'automation_run', 'running', 'ready', 'session', 'planned', 1, 2)",
com.alibaba.qwen.code.managedagent.store.ManagedSessionStore
.sessionScopeKey(tenant, sessionId),
automationKey, tenant, sessionId, "2".repeat(64));
childRunResource(tenant, sessionId, "resource-automation-contract",
"{\"scheduleId\":\"asch_" + "1".repeat(32)
+ "\",\"occurrenceKey\":\"manual:contract\"}");
String automationTask = "task_" + automationKey;
assertThat(json(exchange(drift, "getSessionTask", 200,
get("/v1/agents/sessions/{id}/tasks/{task}", sessionId,
automationTask).header(TENANT, tenant), null))
.get("action_capabilities").toString()).isEqualTo("[\"cancel\"]");
JsonNode automationAdmitted = json(exchange(drift, "cancelSessionTask", 202,
post(cancelPath, sessionId, automationTask).header(TENANT, tenant)
.header(IDEMPOTENCY_KEY, "contract-automation-cancel"), null));
assertThat(automationAdmitted.get("type").asText()).isEqualTo("task_cancel");
assertThat(automationAdmitted.get("task_id").asText()).isEqualTo(automationTask);
JsonNode automationWebShell = json(exchange(drift, "cancelWebShellTask", 202,
post(WEB_SHELL + "/tasks/cancel").header(TENANT, tenant),
("{\"sessionId\":\"%s\",\"taskId\":\"%s\","
+ "\"idempotencyKey\":\"contract-automation-cancel\"}")
.formatted(sessionId, automationTask)));
assertThat(automationWebShell.get("operationId").asText())
.isEqualTo(automationAdmitted.get("id").asText());
assertThat(automationWebShell.get("replayed").asBoolean()).isTrue();
}
/**
@@ -147,8 +147,6 @@ class ManagedTaskCancelOperationTest {
@Test
void aNewRequestNeedsTheTasksCancelAction() {
refused(() -> begin("key-1", "digest-1", AUTOMATION),
HttpStatus.CONFLICT, "task_action_unavailable");
refused(() -> begin("key-2", "digest-2", SETTLED_CHILD),
HttpStatus.CONFLICT, "task_action_unavailable");
refused(() -> begin("key-3", "digest-3", "task_" + "d".repeat(64)),
@@ -168,6 +166,23 @@ class ManagedTaskCancelOperationTest {
HttpStatus.CONFLICT, "task_action_unavailable");
}
@Test
void anAutomationRunAdmitsCancelAndReplaysAfterSettlement() {
OperationRecord admitted = begin("auto-key", "auto-digest", AUTOMATION)
.operation();
assertThat(admitted.kind()).isEqualTo(OperationKind.TASK_CANCEL);
assertThat(admitted.taskId()).isEqualTo(AUTOMATION);
refused(() -> begin("auto-key", "other-digest", AUTOMATION),
HttpStatus.CONFLICT, "idempotency_conflict");
settle(admitted, TaskCancelOutcome.completed());
jdbc.update("UPDATE ma_session_record SET task_state = 'cancelled'"
+ " WHERE record_key = ?", "c".repeat(64));
assertThat(begin("auto-key", "auto-digest", AUTOMATION).operation()
.operationId()).isEqualTo(admitted.operationId());
refused(() -> begin("auto-next", "auto-digest", AUTOMATION),
HttpStatus.CONFLICT, "task_action_unavailable");
}
// H4f-b: a workflow child_run takes the same admission rule as a child
// agent's — the cancellable states admit, terminal and
// recovery_blocked states refuse, and the retained key replays.
@@ -1407,6 +1407,51 @@ class QwenHostedHarnessConnectorTest {
verify(client, times(2)).runMessageOperation(attached, stop);
}
@Test
void automationCancelLoadsPassivelyUnderCommittedStopAuthority() {
HostedHarnessClient client = mock(HostedHarnessClient.class);
HostedHarnessCapabilities capabilities = mock(HostedHarnessCapabilities.class);
HarnessSessionRef attached = mock(HarnessSessionRef.class);
SessionRecord session = mock(SessionRecord.class);
AgentStateStore sessions = mock(AgentStateStore.class);
WorkspaceExecutionStore execution = mock(WorkspaceExecutionStore.class);
when(client.capabilities()).thenReturn(capabilities);
when(capabilities.getBootId()).thenReturn(BOOT_ID);
when(sessions.requireSession("tenant-a", SESSION_ID)).thenReturn(session);
when(session.tenantId()).thenReturn("tenant-a");
when(session.sessionId()).thenReturn(SESSION_ID);
when(session.harnessBootId()).thenReturn(BOOT_ID);
when(session.workspace()).thenReturn(new ContextBinding("tenant-a", "workspace", 1,
"storage", ".", "config", 1));
when(session.toolProfile()).thenReturn("hosted-workspace-shell/1");
when(client.loadSession(any(LoadHarnessSession.class))).thenReturn(attached);
when(attached.getHarnessBootId()).thenReturn(BOOT_ID);
when(attached.getApprovalMode()).thenReturn("default");
ManagedAgentProperties properties = properties();
properties.getHarness().setWorkspaceFilesEnabled(true);
ManagedActionStore actions = mock(ManagedActionStore.class);
when(actions.approvalMode("tenant-a", SESSION_ID)).thenReturn("default");
doThrow(WorkspaceExecutionStore.unavailable()).when(execution).authorize(session);
doThrow(WorkspaceExecutionStore.unavailable()).when(execution).authorizeCancellation(session);
QwenHostedHarnessConnector connector = new QwenHostedHarnessConnector(properties, sessions, execution, actions);
ReflectionTestUtils.setField(connector, "client", client);
Map<String, Object> cancel = Map.of("kind", "cancel_run",
"scheduleId", "asch_" + "1".repeat(32), "occurrenceKey", "manual:one",
"operationId", "66666666-6666-4666-8666-666666666666");
connector.runAutomationOperation("tenant-a", SESSION_ID, cancel);
ArgumentCaptor<LoadHarnessSession> loads = ArgumentCaptor.forClass(LoadHarnessSession.class);
verify(client).loadSession(loads.capture());
assertThat(ReflectionTestUtils.<Map<String, Object>>invokeMethod(loads.getValue(), "toJson"))
.containsEntry("passiveManagedRuntimeRecovery", true)
.doesNotContainKeys("stopMessages", "driveRuntimeRecovery", "cancellationTakeover");
verify(execution).authorizeCommittedStop(session);
verify(execution, never()).authorize(session);
verify(execution, never()).authorizeCancellation(session);
connector.runAutomationOperation("tenant-a", SESSION_ID, cancel);
verify(client, times(1)).loadSession(any(LoadHarnessSession.class));
verify(client, times(2)).runAutomationOperation(attached, cancel);
}
// D8b: a definition-pinned child Session's message stop loads through
// the same entry, so the load must carry the pin or the Harness would
// refuse the Session with hosted_agent_definition_conflict.
@@ -467,8 +467,8 @@ class TaskCancelCoordinatorTest {
@Test
void aTaskWithoutACancelPathFailsWithoutDelivery() {
target.set(new TaskTarget("automation_run", "automation-1",
"automation_run", "running", 1, JSON.createObjectNode()));
target.set(new TaskTarget("unknown", "unknown-1",
"unknown", "running", 1, JSON.createObjectNode()));
coordinator.dispatch(TENANT, SESSION, OPERATION);
assertThat(sent).isEmpty();
assertThat(mockingDetails(broker).getInvocations()).isEmpty();
@@ -478,6 +478,90 @@ class TaskCancelCoordinatorTest {
eq(0L));
}
private TaskTarget automationRun(String state) {
return new TaskTarget("automation_run", "arun_" + "1".repeat(64),
"automation_run", state, 2, JSON.createObjectNode()
.put("scheduleId", "asch_" + "1".repeat(32))
.put("occurrenceKey", "manual:original"));
}
private void automationSend() {
when(harness.runAutomationOperation(eq(TENANT), eq(SESSION), any()))
.thenAnswer(args -> {
sent.add(Map.copyOf(args.getArgument(2)));
if (afterSend.get() != null) {
target.set(afterSend.get());
}
if (harnessFailure.get() != null) {
throw harnessFailure.get();
}
return Map.of("state", "settled");
});
}
@Test
void anAutomationCancelTargetsOnlyItsOriginalOccurrence() {
automationSend();
target.set(automationRun("running"));
afterSend.set(automationRun("cancelled"));
coordinator.dispatch(TENANT, SESSION, OPERATION);
assertThat(sent).hasSize(1);
assertThat(sent.getFirst()).containsEntry("kind", "cancel_run")
.containsEntry("automationRunId", "arun_" + "1".repeat(64))
.containsEntry("scheduleId", "asch_" + "1".repeat(32))
.containsEntry("occurrenceKey", "manual:original");
verify(harness, never()).runChildOperation(any(), any(), any());
verify(store).settleTaskCancel(eq(TENANT), eq(SESSION), eq(OPERATION),
anyString(), eq(1L), eq(TaskCancelOutcome.completed()), eq(0L));
}
@Test
void anAutomationCancelReadsCommittedProofEvenWhenTheReplyWasLost() {
automationSend();
target.set(automationRun("running"));
afterSend.set(automationRun("cancelled"));
harnessFailure.set(new IllegalStateException("reply lost"));
coordinator.dispatch(TENANT, SESSION, OPERATION);
verify(store).settleTaskCancel(eq(TENANT), eq(SESSION), eq(OPERATION),
anyString(), eq(1L), eq(TaskCancelOutcome.completed()), eq(0L));
}
@Test
void anAutomationAcknowledgmentAloneIsNotCancellationProof() {
automationSend();
target.set(automationRun("running"));
coordinator.dispatch(TENANT, SESSION, OPERATION);
verify(store, never()).settleTaskCancel(any(), any(), any(), any(),
anyLong(), eq(TaskCancelOutcome.completed()), anyLong());
}
@Test
void aNaturallySettledAutomationKeepsItsOutcome() {
automationSend();
target.set(automationRun("running"));
afterSend.set(automationRun("completed"));
coordinator.dispatch(TENANT, SESSION, OPERATION);
verify(store).settleTaskCancel(eq(TENANT), eq(SESSION), eq(OPERATION),
anyString(), eq(1L),
eq(TaskCancelOutcome.failed("task_already_settled")), eq(0L));
}
@Test
void aParkedAutomationCancelReconcilesWithoutDelivery() {
ParkedOperation parked = new ParkedOperation(TENANT, SESSION,
OPERATION, "TASK_CANCEL");
when(store.findOperation(TENANT, SESSION, OPERATION))
.thenReturn(Optional.of(operation("RECOVERY_BLOCKED", 16)));
target.set(automationRun("running"));
assertThat(coordinator.read(parked).orElseThrow())
.isEqualTo(AcceptanceVerdict.unknown());
target.set(automationRun("cancelled"));
assertThat(coordinator.read(parked).orElseThrow())
.isEqualTo(AcceptanceVerdict.committed());
assertThat(sent).isEmpty();
verify(harness, never()).runAutomationOperation(any(), any(), any());
}
// H4f-b: a workflow run rides the same record-first delivery — the
// kind gate admits it and every outcome is read from its record.
@Test
@@ -1170,7 +1170,7 @@ export interface components {
*/
TaskRuntimeState: "unbound" | "provisioning" | "ready" | "draining" | "lost";
/**
* @description Actions the task supports now, the same for every caller. cancel: the cancel route accepts a new command for this task (since 1.40 for child_agent, 1.46 for workflow and 1.54 for background_shell and monitor, tasks of those kinds in pending, running, waiting or degraded, and since 1.57 for channel_delivery, a delivery still pending because no adapter has claimed it, each in an active Session); whether a caller may use it is an authorization check (403 on the cancel route). send_input: reserved for a later capability route. read_output: the task events route returns output events for this task to any caller that can read it. read_output does not change during the task's life, and a task without it produces no output events: its output goes only to Artifacts, so the events route never filters out events that exist.
* @description Actions the task supports now, the same for every caller. cancel: the cancel route accepts a new command for this task (since 1.40 for child_agent, 1.46 for workflow and 1.54 for background_shell and monitor, tasks of those kinds in pending, running, waiting or degraded, and since 1.57 for channel_delivery, a delivery still pending because no adapter has claimed it, each in an active Session, and since 1.65 for automation_run in pending, running, waiting or degraded in an active Session; only that occurrence is cancelled and its schedule is preserved); whether a caller may use it is an authorization check (403 on the cancel route). send_input: reserved for a later capability route. read_output: the task events route returns output events for this task to any caller that can read it. read_output does not change during the task's life, and a task without it produces no output events: its output goes only to Artifacts, so the events route never filters out events that exist.
* @enum {string}
*/
TaskActionCapability: "cancel" | "send_input" | "read_output";