mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
feat(managed-agent): add reliable ACTIVE Workspace deletion (L3) (#13354)
* feat(managed-agent): add reliable ACTIVE Workspace deletion * fix(managed-agent): align lifecycle CI fixtures * fix(hosted): authorize detach after durable lifecycle recovery (#13354) * fix(hosted): scope lifecycle fence locks and authorization errors * fix(managed-agent): resolve L3 migration version collision (#13354) * fix(managed-agent): address PR review feedback (#13354) * fix(managed-agent): address PR CI fixture failures (#13354) * codex: address PR review feedback (#13354) * codex: address PR review feedback (#13354) * fix(managed-agent): preserve main credential and migration contracts (#13354) * codex: address PR review feedback (#13354) * fix(managed): restore original lifecycle Hook owner before dispatch * codex: address PR review feedback (#13354) * test(managed-agent): repair lifecycle receipt journal fixture * fix(managed-agent): fail closed on stale lifecycle capability client * fix(managed-agent): align native lifecycle test contracts * codex: address PR review feedback (#13354) --------- Co-authored-by: Shaojin Wen <[email protected]>
This commit is contained in:
jinyeandShaojin Wen authored and GitHub committed
1 parent
e79a4a7966
commit
fe4d4e345c
86 files changed
+5695
-198
No files matched your search
@@ -17,7 +17,7 @@ Paths are under `packages/sdk-java/managed-agent-server/src/main/java/com/alibab
|
||||
- **Binding.** `V7__managed_workspace_binding.sql` adds the eight-column binding set (`workspace_id`, `workspace_generation`, `workspace_storage_id`, `cwd_relative`, `context_config_ref`, `context_revision`, `workspace_config_ref`, `workspace_policy_ref`) as one all-null-or-all-not-null unit behind the seven-field `ContextBinding` model, including `cwd_relative` and `context_revision` (hard-coded `1` at `store/ManagedWorkspaceRegistry.java:212`). No `context_state` column exists, and none is needed here: an open cwd operation is the gate.
|
||||
- **Operation ledger.** `V17__managed_session_operation.sql`'s `managed_agent_operation` is the durable 202 + Idempotency-Key substrate: per-(tenant, session, kind, actor, key) uniqueness, PENDING/RUNNING/COMPLETED states, lease/claim/retry columns, and a coordinator (`service/SessionLifecycleCoordinator.java`) with `@Scheduled recoverOperations()`. Kinds are bespoke (`CLOSE/ARCHIVE/DELETE/ACTION_RESPONSE` at `store/StoreModels.java:84`); V24 added per-kind columns, which is the template for the cwd columns. `findDeliverableOperations`, `claimOperation` and `retryOperation` are kind-generic; `beginOperation`/`completeOperation` are not (they assert the lifecycle status machine and reject bound Sessions).
|
||||
- **Admission barriers.** `insertTurnCommand` rejects `turn_active` when any Turn is ACCEPTED/RUNNING/CANCELLING and, since this slice's merge, refuses `session_context_busy` for a bound Session with an open context-changing operation (its own narrowed predicate — pending display mutations and `ACTION_RESPONSE` operations stay off the Turn barrier; see [Concurrency and races](#concurrency-and-races)). `requireNoOpenOperation` (`:2035`) rejects `session_operation_active` when any operation is open or any mutation command is PENDING; `beginOperation` and `beginSessionMutation` call it, and the cwd admission keeps the wider barrier beside it. All run under the Session row lock (`requireSessionForUpdate`, `SELECT ... FOR UPDATE`), which serializes competing admissions.
|
||||
- **Execution pivot.** `WorkspaceRuntimeResolver.resolve()` re-reads the persisted binding and rebuilds the scope on every call (`service/WorkspaceRuntimeResolver.java`'s `resolve()`, lines 62–75 at the merge commit); `WorkspaceRuntimeTransport.acquire` installs and activates **that** binding on a **fresh Runtime Session** for each tool turn (`service/WorkspaceRuntimeTransport.java`'s `acquire`, lines 53–78 at the merge commit). The worker records installations per Runtime Session ID (`packages/cli/src/serve/managed-context-envelope.ts:347-453`), re-resolves the effective directory from the installed binding on every tool call, and never uses `process.chdir`. **Committing new `cwd_relative`/`context_revision` to the Session row is therefore the only pivot; the next turn's production acquire path installs the new context with the full worker receipt. No TypeScript or worker-protocol change is required.**
|
||||
- **Execution pivot.** `WorkspaceRuntimeResolver.resolve()` re-reads the persisted binding and rebuilds the scope on every call (`service/WorkspaceRuntimeResolver.java`'s `resolve()`, lines 62–75 at the merge commit); `WorkspaceRuntimeTransport.acquire` installs and activates **that** binding on a **fresh Runtime Session** for each tool turn without a retained Hook owner (`service/WorkspaceRuntimeTransport.java`'s `acquire`, lines 53–78 at the merge commit). The worker records installations per Runtime Session ID (`packages/cli/src/serve/managed-context-envelope.ts:347-453`), re-resolves the effective directory from the installed binding on every tool call, and never uses `process.chdir`. **Committing new `cwd_relative`/`context_revision` to the Session row is therefore the only pivot; the next turn's production acquire path installs the new context with the full worker receipt. No TypeScript or worker-protocol change is required.**
|
||||
- **Authorization.** `WorkspaceExecutionStore.authorizePassiveAttachment` (`store/WorkspaceExecutionStore.java:63-116`) joins the Session, Registry, creation command and access rows and requires exact workspace/generation/storage, `ACTIVE` registry state, creator read/create grants and the frozen profile refs. `WorkspaceRuntimeResolver` additionally re-verifies the administrator mount (canonical path + `fileKey`).
|
||||
- **Contract.** `managed-agent-public-api.openapi.json` v1.27.0 pins at lines 1341/1479 the `planned` routes `POST /v1/agents/sessions/{sessionId}/cwd` (`changeSessionCwd`, Idempotency-Key header) and `POST /api/agent/web-shell/v1/sessions/cwd/change` (`webShellChangeCwd`), the `planned` schemas `ChangeCwdRequest`, `PublicCwdOperation`, `WebShellChangeCwdRequest`, `WebShellCwdOperation` and `CwdOperationStatus` (`pending/installing/completed/failed/recovery_blocked`), and per the contract text: "W2: same-workspace only. Replay the original operation before revision/busy checks. Reject active or queued input and context holds. Poll the operation or await session.context.changed; do not treat 202 as activation." `ManagedAgentApiContractTest` treats a mapped route whose spec status is `planned` as drift, so the routes and the status flip must land together.
|
||||
- **Contract gaps to close.** `session_context_busy` appears nowhere in the contract or code; `session.context.changed` is not in the event vocabulary; the planned `failure_code` field has no pinned value set.
|
||||
@@ -61,14 +61,14 @@ Non-goals for this slice (each recorded in [Boundaries](#boundaries-and-follow-u
|
||||
| 11 | Session status is `ACTIVE` | 409 `session_state_conflict` |
|
||||
| 12 | Registry row still matches the binding's workspace, generation and storage and is `ACTIVE` | 409 `workspace_unavailable` |
|
||||
| 13 | `expected_context_revision` equals the persisted `context_revision` | 409 `context_revision_conflict` |
|
||||
| 14 | No active Turn, no open operation or pending mutation command (the shared `hasOpenOperation` predicate; lifecycle admissions throw the same barrier as `session_operation_active`), and no answerable permission Action (`hasDecidableAction` — the lifecycle sibling refuses in exactly this state with 409 turn_active) | 409 `session_context_busy` |
|
||||
| 14 | No retained Runtime Session for the same tenant/Harness Session (only RELEASED permits a change), no active Turn, no open operation or pending mutation command (the shared `hasOpenOperation` predicate; lifecycle admissions throw the same barrier as `session_operation_active`), and no answerable permission Action (`hasDecidableAction` — the lifecycle sibling refuses in exactly this state with 409 turn_active) | 409 `session_context_busy` |
|
||||
| 15 | Insert the PENDING operation and dispatch it asynchronously; return 202. | — |
|
||||
|
||||
Notes on the order and semantics:
|
||||
|
||||
- Rows 2–4 run in the admission service before the store transaction opens (the G0 parse-then-digest precedent): a lexical check consults no Session fact, so its error cannot leak one, and the digest it feeds already pins the replay semantics of row 8.
|
||||
- The request digest covers `sessionId`, the fixed operation name, the **normalized** target directory and the expected revision (`RequestDigests` canonical SHA-256). A retry after the commit therefore replays the completed operation even though row 13 would now fail.
|
||||
- Rows 5 onward run under the Session row lock; replay stays ahead of the revision and busy checks with the V17 binary-safe key comparison, exactly as the contract text demands; rows 6–9 mirror `beginLifecycle`'s invisible-caller and replay-first discipline.
|
||||
- Rows 5 onward run under the Session row lock, after taking the tenant placement lock before any consistent read to serialize Runtime admission; replay stays ahead of the revision and busy checks with the V17 binary-safe key comparison, exactly as the contract text demands; rows 6–9 mirror `beginLifecycle`'s invisible-caller and replay-first discipline.
|
||||
- Creator-only matches the policy #13112 announced for later turns and the execution layer's own creator-grant recheck (`authorizePassiveAttachment` joins on the creation actor). Widening to non-creator Workspace writers is a separate admission decision. The refusal vocabulary for readable non-creators is `403 session_operation_forbidden` — the exact pairing the merged bound lifecycle (#13135's close, #13194's archive/delete) already ships; the first version of this design answered 404 everywhere and aligned with the sibling semantics at integration time. The settlement re-verifies the full grant set (`can_read` + `can_create` of the creation actor) at commit via `hasCwdChangeRegistryFacts`, so a grant revoked after admission still blocks the change.
|
||||
- A target equal to the current directory is admitted and completes, raising the revision. It is a legal re-validation transition; rejecting it would add a branch that protects nothing.
|
||||
- No event is appended at admission. The contract's only cwd event is completion's `session.context.changed`; the 202 body and the operation query carry the pending state.
|
||||
@@ -85,7 +85,7 @@ The `admission_stage` is `JAVA_DURABLE` and `delivery_state` starts `PENDING` wi
|
||||
- re-checks the mount's canonical path and `fileKey`, and runs the storage guard's mount verification of the **candidate** binding (target directory) when verified recovery is enabled, exactly as `resolve()` would for a real acquisition of that target;
|
||||
- runs the direct-question shape of the directory rule `acquire()` enforces: the acquire path keeps its long-standing boolean-predicate `requireDirectory` (every I/O anomaly terminal — an ENOTDIR-shaped cwd fails the Turn immediately with the accurate code, never a retryable mis-attribution), while the probe twin `requireDirectoryForProbe` probes with throwing calls — `readAttributes` with `NOFOLLOW_LINKS`, `toRealPath` identity, `checkAccess(READ, EXECUTE)` in place of the false-on-error `Files.is*` predicates — so the catch arms classify by the verdict: vanished target, permission denial and ENOTDIR/ELOOP structural shapes terminal (the last two reach some JDKs as bare `FileSystemException`, so the residual arm walks the ancestor chain — a plain-file or dangling/looping symlink ancestor is structural), only opaque blips (`ESTALE`/`EIO`, `ENAMETOOLONG`) retrying through the budget.
|
||||
- A structural failure is terminal for the operation: `failCwdChangeOperation` records `failure_code=workspace_unavailable` and the Session stays at its prior revision. Run against the **same host and mount snapshot the worker uses**, this probe has the same verdict the worker's installation verification would return — its guard runs against the _candidate_ binding through a probe-only guard entry (the shared acquire path keeps the terminal classification it always had), so a change away from a destroyed current directory stays reachable (the escape this feature exists for), and the shared directory rule carries the worker's own `access(R_OK|X_OK)` checks, so nothing passes here that the worker would later refuse as an untyped wedge. Structural probe refusals fold into the typed terminal verdict (re-issuable by the caller), and the refusal log carries the broker code, class, message and the preserved cause; a momentary I/O failure keeps its cause and retries through the delivery machine instead of certifying a verification that never happened — **bounded at 8 attempts** under the capped dispatch backoff, after which the change settles with the typed terminal failure: an accepted operation always terminates typed, the row leaves the deliverable set, and both admission barriers re-open with the Session unchanged and executable.
|
||||
3. **Commit** in one transaction: lock the Session row; re-verify `ACTIVE` status, expected revision, no active Turn, no other open operation, and the binding-matching Registry facts plus the creation actor's surviving grants (`hasCwdChangeRegistryFacts` — the same passive-attachment subset; the full execution-time set including frozen profile refs remains the next turn's acquire-time gate); then update `cwd_relative` and `context_revision = expected + 1` with the usual `version` bump, mark the operation COMPLETED with `result_context_revision = expected + 1` and `receipt_id`, and append `session.context.changed` (`data: {sessionId, operationId, workspaceId, cwdRelative, contextRevision}`, camelCase per the store's event-data convention, source `operation:<id>:completed`). The committed-event publisher pushes it after commit through the existing SSE hub.
|
||||
3. **Commit** in one transaction: lock placement before the Session row; re-verify `ACTIVE` status, expected revision, no active Turn, no other open operation, no retained same-tenant/Harness Runtime Session, and the binding-matching Registry facts plus the creation actor's surviving grants (`hasCwdChangeRegistryFacts` — the same passive-attachment subset; the full execution-time set including frozen profile refs remains the next turn's acquire-time gate); then update `cwd_relative` and `context_revision = expected + 1` with the usual `version` bump, mark the operation COMPLETED with `result_context_revision = expected + 1` and `receipt_id`, and append `session.context.changed` (`data: {sessionId, operationId, workspaceId, cwdRelative, contextRevision}`, camelCase per the store's event-data convention, source `operation:<id>:completed`). The committed-event publisher pushes it after commit through the existing SSE hub.
|
||||
- If the re-verified facts no longer hold — `context_revision_conflict` when the revision moved, `session_context_busy` when a Turn or another operation appeared, `workspace_unavailable` when the grant/registry facts changed — the operation ends as `failed` with that `failure_code`; the Session row provably keeps its prior context because nothing ran before this transaction.
|
||||
- Any other `RuntimeException` (transport, SQL, clock) retries through the existing `retryOperation` backoff, unbounded, matching the lifecycle contract that an accepted operation never silently dies.
|
||||
4. **Reclaim** of a LEASED operation whose owner died re-enters the same branch: the probe is read-only and its verdict deterministic, so rerunning it is idempotent. Double-settlement is fenced not by that CAS but by the delivery machine itself: a settled row is `CONFIRMED` and unclaimable, and a stale owner's completion or failure write is refused by the lease/owner/generation/live-lease predicate — a contested completion returns `null`, a contested failure write is skipped (`failCwdChangeOperation` reports `false`). The probe leaves no ownership or worker footprint (it claims nothing and talks to no worker), so a crash can strand nothing — unlike the turn path, which by design may retain the storage lease pending W0e recovery.
|
||||
@@ -96,7 +96,7 @@ The `admission_stage` is `JAVA_DURABLE` and `delivery_state` starts `PENDING` wi
|
||||
|
||||
The reference design asks for Runtime and Harness installation receipts before commit. This slice deliberately scopes that down, and the deviation is safe to take because of verified properties of the merged architecture:
|
||||
|
||||
- Every tool turn acquires a **fresh** Runtime Session and installs the then-current binding before any tool runs (`installations` are keyed by Runtime Session ID; a same-Session re-install with a different digest is refused). The production receipt for the new context therefore exists before the first post-change tool executes, and its absence blocks that turn with `managed_context_unavailable` — typed, never silently in the wrong directory.
|
||||
- A tool turn without a retained Hook owner acquires a **fresh** Runtime Session and installs the then-current binding before any tool runs (`installations` are keyed by Runtime Session ID; a same-Session re-install with a different digest is refused). The production receipt for the new context therefore exists before the first post-change tool executes, and its absence blocks that turn with `managed_context_unavailable` — typed, never silently in the wrong directory.
|
||||
- The worker is same-host, same-UID, same mount snapshot as the control plane (enforced by `validateWorkspaceFiles` and the resolver's construction-time mount checks), and the shared `requireDirectory` carries the worker's own `access(R_OK|X_OK)` checks, so the Java probe verdict and the worker's install verdict coincide on the same snapshot; a later divergence can only come from the filesystem changing after commit, where it still surfaces typed at the first turn, never misdirected.
|
||||
- Committing only after a Java-side deterministic check keeps the settled operation honest without inventing ownership-free access to the worker's installation route (which today exists only inside `acquire()`'s claim→install→activate chain, whose ownership claim may by design strand storage on an ambiguous crash — an unacceptable risk for a probe).
|
||||
|
||||
@@ -115,7 +115,7 @@ WebShell requests carry `sessionId`, `idempotencyKey`, `cwdRelative`, `expectedC
|
||||
- **Admission vs cancel**: a cancel is real work only while a Turn is active (`commandEffect` false otherwise), and an open operation gates any Turn, so a cancel admitted while a cwd operation is open is a completed no-op insertion — it cannot disturb the commit, which re-runs its fact checks afterwards anyway. The earlier version of this bullet promised a typed refusal such a cancel would produce; the shipped and pinned mechanism is the benign one.
|
||||
- **Probe vs commit**: the probe is read-only and needs no storage lease; the commit transaction re-checks every fact under the row lock, so a grant revoked or a Turn admitted between probe and commit fails the operation cleanly instead of committing.
|
||||
- **Crash windows**: admission-insert is one transaction; probe+commit has no footprint to strand; the commit is atomic with its event; reclaim re-runs the branch idempotently.
|
||||
- **Post-commit execution**: the next turn's `acquire` re-resolves the binding, claims storage, installs and activates the new context on a new Runtime Session, asserting ownership afterwards. A leaked pre-commit Runtime Session cannot execute — execution requires the ACQUIRING/READY record state that release closes, and an unsettled one keeps the Turn that owns it `RUNNING`, which blocks admission at row 14.
|
||||
- **Post-commit execution**: the next turn's `acquire` re-resolves the binding, claims storage, installs and activates the new context on a new Runtime Session, asserting ownership afterwards. Hook owners can remain READY after their public Turn completes. Admission and commit refuse `session_context_busy` while any same-tenant/Harness Runtime Session is not RELEASED, including uncertain ACQUIRING/RELEASING and unproven FAILED states. This preserves the original immutable worker context and its release identity; it neither reinstalls nor replaces that Runtime. Other Harness Sessions and tenants are outside this predicate. Both transactions take placement first, matching Broker admission; the exact-scope existence read adds scan cost without a fixed latency promise.
|
||||
|
||||
## Files affected
|
||||
|
||||
@@ -133,8 +133,8 @@ No change to `packages/cli/src/serve/**`, `packages/core/**`, the worker, the en
|
||||
## Validation plan
|
||||
|
||||
- **Java**: `mvn test` and `mvn checkstyle:check` in `packages/sdk-java/managed-agent-server` on JDK 21; focused store/coordinator/controller/contract tests.
|
||||
- **Store admission matrix**: every row of the admission table, including replay-before-CAS order, digest conflict, cross-tenant invisibility, unreadable-actor 404 `session_not_found` vs readable non-creator 403 `session_operation_forbidden`, revoked grant, draining/removed registry, generation drift, expected-revision mismatch, busy on the G0 initial Turn, and the no-op same-directory change.
|
||||
- **Settlement**: probe failure leaves the Session untouched with `failure_code=workspace_unavailable`; commit after external revision bump fails with `context_revision_conflict`; commit with a fresh active Turn fails with `session_context_busy`; retry after a transient failure succeeds once; a probe that never stops failing transiently settles terminally when the 8-attempt budget runs out — deliverable rows drain and both admission barriers re-open; a vanished mount root or target classifies terminal (pinned against real deleted directories) while a momentary I/O shape classifies retryable (pinned with a deterministic ENAMETOOLONG probe); reclaim of a dead owner's LEASED row completes exactly once; legacy lifecycle operations and `ACTION_RESPONSE` are untouched (their tests stay green).
|
||||
- **Store admission matrix**: every row of the admission table, including replay-before-CAS order, digest conflict, cross-tenant invisibility, unreadable-actor 404 `session_not_found` vs readable non-creator 403 `session_operation_forbidden`, revoked grant, draining/removed registry, generation drift, expected-revision mismatch, busy on the G0 initial Turn or retained Runtime context, and the no-op same-directory change.
|
||||
- **Settlement**: probe failure leaves the Session untouched with `failure_code=workspace_unavailable`; commit after external revision bump fails with `context_revision_conflict`; commit with a fresh active Turn or retained Runtime context fails with `session_context_busy`; retry after a transient failure succeeds once; a probe that never stops failing transiently settles terminally when the 8-attempt budget runs out — deliverable rows drain and both admission barriers re-open; a vanished mount root or target classifies terminal (pinned against real deleted directories) while a momentary I/O shape classifies retryable (pinned with a deterministic ENAMETOOLONG probe); reclaim of a dead owner's LEASED row completes exactly once; legacy lifecycle operations and `ACTION_RESPONSE` are untouched (their tests stay green).
|
||||
- **Contract**: mapped vs planned drift test, schema pins for the six touched schemas, WebShell twins, event documentation consistency.
|
||||
- **Hosted integration (H2, mirroring the G0 suite; added to the MySQL CI lane)**: create a bound Session with its initial file Turn in directory A (a real pre-change installation for the later Turn to escape), wait for that Turn, then change cwd across both surfaces with normalized replays, poll each operation to `completed` and assert the Session row revision/cwd plus the `session.context.changed` event on the public stream and through `/operations/query` — then submit a bound later Turn (merged #13112) and assert its file writes land in the committed directory while the initial Turn's own files stay intact. Negative group: flag off, legacy Session, busy, CAS conflict, idempotency replay and conflict, the WebShell failed projection carrying `failureCode`; the store pins the later-turn busy handshake both ways (open op refuses a Turn; completed op frees it) and the terminal-failure release of both surfaces.
|
||||
- **E2E plan**: `.qwen/e2e-tests/managed-workspace-w2-cwd-change.md`, dry-run against the global `qwen` CLI baseline first (route unmapped today).
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
- **绑定。** `V7__managed_workspace_binding.sql` 新增八列绑定集合(`workspace_id`、`workspace_generation`、`workspace_storage_id`、`cwd_relative`、`context_config_ref`、`context_revision`、`workspace_config_ref`、`workspace_policy_ref`),以全空或全非空为整体,承载七字段的 `ContextBinding` 模型,含 `cwd_relative` 与 `context_revision`(创建时硬编码为 `1`,见 `store/ManagedWorkspaceRegistry.java:212`)。不存在 `context_state` 列,本切片也不需要:一个未关闭的 cwd operation 本身就是闸门。
|
||||
- **operation 账本。** `V17__managed_session_operation.sql` 的 `managed_agent_operation` 是持久的 202 + Idempotency-Key 底座:按 (tenant, session, kind, actor, key) 唯一;PENDING/RUNNING/COMPLETED 状态;lease/claim/retry 列;以及带 `@Scheduled recoverOperations()` 的协调器(`service/SessionLifecycleCoordinator.java`)。kind 各自专属(`store/StoreModels.java:84` 的 `CLOSE/ARCHIVE/DELETE/ACTION_RESPONSE`);V24 增加了按 kind 的专有列,正是 cwd 列的模板。`findDeliverableOperations`、`claimOperation` 与 `retryOperation` 与 kind 无关可复用;`beginOperation`/`completeOperation` 不可复用(它们断言生命周期状态机,且拒绝绑定会话)。
|
||||
- **准入屏障。** `insertTurnCommand` 在存在 ACCEPTED/RUNNING/CANCELLING Turn 时拒绝 `turn_active`,并自本切片合并接入起,对存在未关闭"上下文变更型"operation 的绑定会话拒绝 `session_context_busy`(其专属收窄谓词——PENDING 展示变更与 `ACTION_RESPONSE` operation 不上 Turn 屏障;见[并发与竞态](#并发与竞态))。`requireNoOpenOperation`(`:2035`)在存在未关闭 operation 或 PENDING 变更命令时拒绝 `session_operation_active`,`beginOperation` 与 `beginSessionMutation` 都会调用它,cwd 准入并排保留更宽的屏障。以上均在会话行锁内运行(`requireSessionForUpdate`,`SELECT ... FOR UPDATE`),竞争的准入由此串行化。
|
||||
- **执行枢轴。** `WorkspaceRuntimeResolver.resolve()` 每次调用都重读持久化绑定并重建 scope(`service/WorkspaceRuntimeResolver.java` 的 `resolve()`,合并提交时在第 62–75 行);每个工具轮次由 `WorkspaceRuntimeTransport.acquire` 在**新的 Runtime Session** 上安装并激活**该**绑定(`service/WorkspaceRuntimeTransport.java` 的 `acquire`,合并提交时在第 53–78 行)。worker 按 Runtime Session ID 记录安装(`packages/cli/src/serve/managed-context-envelope.ts:347-453`),每次工具调用都从已安装绑定重新解析有效目录,且从不使用 `process.chdir`。**因此提交新的 `cwd_relative`/`context_revision` 到会话行是唯一的枢轴;下一轮的生产 acquire 路径会用完整的 worker 回执安装新上下文,不需要任何 TypeScript 或 worker 协议改动。**
|
||||
- **执行枢轴。** `WorkspaceRuntimeResolver.resolve()` 每次调用都重读持久化绑定并重建 scope(`service/WorkspaceRuntimeResolver.java` 的 `resolve()`,合并提交时在第 62–75 行);没有保留 Hook owner 的工具轮次由 `WorkspaceRuntimeTransport.acquire` 在**新的 Runtime Session** 上安装并激活**该**绑定(`service/WorkspaceRuntimeTransport.java` 的 `acquire`,合并提交时在第 53–78 行)。worker 按 Runtime Session ID 记录安装(`packages/cli/src/serve/managed-context-envelope.ts:347-453`),每次工具调用都从已安装绑定重新解析有效目录,且从不使用 `process.chdir`。**因此提交新的 `cwd_relative`/`context_revision` 到会话行是唯一的枢轴;下一轮的生产 acquire 路径会用完整的 worker 回执安装新上下文,不需要任何 TypeScript 或 worker 协议改动。**
|
||||
- **授权。** `WorkspaceExecutionStore.authorizePassiveAttachment`(`store/WorkspaceExecutionStore.java:63-116`)联接会话、Registry、创建命令与访问授权行,要求 workspace/generation/storage 精确一致、Registry 为 `ACTIVE`、创建者持有 read/create 授权以及冻结 profile 引用。`WorkspaceRuntimeResolver` 还复核管理员挂载(规范路径 + `fileKey`)。
|
||||
- **契约。** `managed-agent-public-api.openapi.json` v1.27.0 在第 1341/1479 行固定了 `planned` 路由 `POST /v1/agents/sessions/{sessionId}/cwd`(`changeSessionCwd`,Idempotency-Key 头)与 `POST /api/agent/web-shell/v1/sessions/cwd/change`(`webShellChangeCwd`),以及 `planned` schema `ChangeCwdRequest`、`PublicCwdOperation`、`WebShellChangeCwdRequest`、`WebShellCwdOperation` 和 `CwdOperationStatus`(`pending/installing/completed/failed/recovery_blocked`),契约文字写明:"W2: same-workspace only. Replay the original operation before revision/busy checks. Reject active or queued input and context holds. Poll the operation or await session.context.changed; do not treat 202 as activation." `ManagedAgentApiContractTest` 把"路由已映射但 spec 状态仍为 `planned`"视为漂移,因此路由与状态翻转必须同一 PR 落地。
|
||||
- **需要补齐的契约空隙。** `session_context_busy` 在契约与代码中都不存在;`session.context.changed` 不在事件词汇中;`failure_code` 字段没有固定取值集合。
|
||||
@@ -61,14 +61,14 @@
|
||||
| 11 | 会话状态为 `ACTIVE` | 409 `session_state_conflict` |
|
||||
| 12 | Registry 行仍与绑定的 workspace、generation、storage 一致且为 `ACTIVE` | 409 `workspace_unavailable` |
|
||||
| 13 | `expected_context_revision` 等于持久化的 `context_revision` | 409 `context_revision_conflict` |
|
||||
| 14 | 无活动 Turn,无未关闭 operation 或 PENDING 变更命令(共享的 `hasOpenOperation` 谓词;生命周期准入以 `session_operation_active` 抛出同一条屏障),且无待决的权限 Action(`hasDecidableAction`——生命周期兄弟在完全相同状态下答 409 turn_active) | 409 `session_context_busy` |
|
||||
| 14 | 同租户/Harness Session 无保留 Runtime Session(仅 RELEASED 允许变更)、无活动 Turn,无未关闭 operation 或 PENDING 变更命令(共享的 `hasOpenOperation` 谓词;生命周期准入以 `session_operation_active` 抛出同一条屏障),且无待决的权限 Action(`hasDecidableAction`——生命周期兄弟在完全相同状态下答 409 turn_active) | 409 `session_context_busy` |
|
||||
| 15 | 插入 PENDING operation 并异步派发;返回 202。 | — |
|
||||
|
||||
关于顺序与语义的说明:
|
||||
|
||||
- 第 2–4 行在准入服务中先于存储事务执行(与 G0 的"先解析后摘要"先例一致)——词法检查不读取任何会话事实,其错误不会泄露会话存在性,而它所喂养的摘要已固定第 8 行的重放语义。
|
||||
- 请求摘要覆盖 `sessionId`、固定操作名、**归一化**目标目录与期望 revision(`RequestDigests` 规范化 SHA-256)。提交完成后的重试会重放出已完成的 operation,即使此刻第 13 行必然失败。
|
||||
- 第 5 行起在会话行锁内运行;重放保持先于 revision 与繁忙检查,沿用 V17 的二进制安全键比较(契约文字的要求);第 6–9 行与 `beginLifecycle` 的调用者不可见与重放优先纪律一致。
|
||||
- 第 5 行起在会话行锁内运行,且在任何一致性读取前先取得租户 placement 锁,以串行化 Runtime 准入;重放保持先于 revision 与繁忙检查,沿用 V17 的二进制安全键比较(契约文字的要求);第 6–9 行与 `beginLifecycle` 的调用者不可见与重放优先纪律一致。
|
||||
- 仅创建者,与 #13112 为后续 Turn 宣布的策略及执行层自己的创建者授权复核一致(`authorizePassiveAttachment` 按创建 actor 联接)。开放给非创建者的 Workspace 写作者是另一个独立的准入决策。有读授权非创建者的拒绝词汇为 `403 session_operation_forbidden`——与已合入的绑定生命周期(#13135 的 close、#13194 的 archive/delete)发布的形态完全一致;本设计第一版对非创建者一律答 404,在集成时对齐了兄弟语义。结算在提交时仍经 `hasCwdChangeRegistryFacts` 复核完整授权集(创建 actor 的 `can_read` + `can_create`),准入之后授权被撤仍能阻止变更。
|
||||
- 目标与当前目录相同的请求也会被接纳并正常完成,revision 递增。这是合法的再校验转移;拒绝它只会增加一个不保护任何东西的分支。
|
||||
- 准入时不追加事件。契约唯一的 cwd 事件是完成时的 `session.context.changed`;pending 状态由 202 响应体与 operation 查询承载。
|
||||
@@ -85,7 +85,7 @@
|
||||
- 复核挂载的规范路径与 `fileKey`,并在已启用验证恢复时对**候选**绑定(目标目录)执行存储守卫的挂载校验——与 `resolve()` 对该目标的真实获取所做的完全一致;
|
||||
- 以探测专用的孪生形态执行目录规则:acquire 路径保持长期以来的谓词式 `requireDirectory`(任何 I/O 异常皆为终态——ENOTDIR 形态的 cwd 立即以准确错误码使 Turn 失败,绝不误报为可重试错误),探测孪生 `requireDirectoryForProbe` 一律用会抛异常的调用——`NOFOLLOW_LINKS` 的 `readAttributes`、`toRealPath` 恒等,以及取代 `Files.is*` 谓词的 `checkAccess(READ, EXECUTE)`——由 catch 分支按判定分类:目标消失、权限拒绝与 ENOTDIR/ELOOP 结构性形态均为终态(后两者在某些 JDK 上只现身为裸 `FileSystemException`,因此残余分支改为上溯祖先链——链上出现普通文件或悬空/成环符号链接即为结构性),只有不透明抖动(`ESTALE`/`EIO`、`ENAMETOOLONG`)经预算重试。
|
||||
- 结构性失败对该 operation 是终态:`failCwdChangeOperation` 记录 `failure_code=workspace_unavailable`,会话保持原 revision。探针运行在**与 worker 相同的主机与挂载快照**之上:其守卫经**探测专用入口**校验**候选**绑定(共享的 acquire 路径保持原有的终态语义;离开一个已被摧毁的当前目录仍可达——这正是本特性存在的理由),共享目录规则还带上 worker 自己的 `access(R_OK|X_OK)` 检查,probe 不会放过任何 worker 随后才以无类型化楔死呈现的目标。结构性探针拒绝折入类型化终态判定(调用方可重发),拒绝日志带 broker 错误码、异常类、消息与保留的异常因;一时的 I/O 失败保留异常因,经交付机械退避重试,而不是把一次从未发生的校验判定成终态——**以 8 次尝试为界**的封顶退避,用尽后以类型化终态结算:已接纳的 operation 必然以类型化状态终止,行离开可投递集合,两道准入屏障重新放行,会话不变且可执行。
|
||||
3. **提交**,单事务:锁会话行;复核 `ACTIVE`、期望 revision、无活动 Turn、无其他未关闭 operation,以及与绑定匹配的 Registry 事实加上创建者仍存的授权(`hasCwdChangeRegistryFacts`——即 passive-attachment 的子集;含冻结 profile 引用的完整执行期集合仍由下一轮的 acquire 准入把关);随后更新 `cwd_relative` 与 `context_revision = expected + 1` 并常规递增 `version`,把 operation 标记为 COMPLETED 并写入 `result_context_revision = expected + 1` 与 `receipt_id`,追加 `session.context.changed`(`data: {sessionId, operationId, workspaceId, cwdRelative, contextRevision}`,按 store 事件 data 的 camelCase 惯例,source 为 `operation:<id>:completed`)。committed-event publisher 在提交后经现有 SSE hub 推送。
|
||||
3. **提交**,单事务:先锁 placement 再锁会话行;复核 `ACTIVE`、期望 revision、无活动 Turn、无其他未关闭 operation、无同租户/Harness 保留 Runtime Session,以及与绑定匹配的 Registry 事实加上创建者仍存的授权(`hasCwdChangeRegistryFacts`——即 passive-attachment 的子集;含冻结 profile 引用的完整执行期集合仍由下一轮的 acquire 准入把关);随后更新 `cwd_relative` 与 `context_revision = expected + 1` 并常规递增 `version`,把 operation 标记为 COMPLETED 并写入 `result_context_revision = expected + 1` 与 `receipt_id`,追加 `session.context.changed`(`data: {sessionId, operationId, workspaceId, cwdRelative, contextRevision}`,按 store 事件 data 的 camelCase 惯例,source 为 `operation:<id>:completed`)。committed-event publisher 在提交后经现有 SSE hub 推送。
|
||||
- 若复核事实不再成立——revision 已变动为 `context_revision_conflict`、出现 Turn 或其他 operation 为 `session_context_busy`、授权/Registry 事实变化为 `workspace_unavailable`——operation 以对应 `failure_code` 进入 `failed`;由于该事务之前没有任何写入,会话行可证明地保持原上下文。
|
||||
- 其他 `RuntimeException`(传输、SQL、时钟)走现有 `retryOperation` 退避重试,不设上限,与生命周期契约一致:已接纳的 operation 不会悄悄死掉。
|
||||
4. **租约回收**:属主挂掉的 LEASED operation 重新进入同一分支——探针只读且判定确定,重跑幂等。防二次结算的不是那个 CAS,而是交付状态机本身:已结算的行是 `CONFIRMED`、不可再认领,陈旧属主的完成或失败写入都被租约/属主/代际/活租约谓词拦下——竞争失败时 complete 返回 `null`、失败写入被跳过(`failCwdChangeOperation` 报告 `false`)。探针不留任何 ownership 或 worker 痕迹(不 claim、不接触 worker),崩溃不会遗留任何东西——这与 Turn 路径不同,后者按设计可能在崩溃歧义时保留存储租约,留待 W0e 恢复。
|
||||
@@ -96,7 +96,7 @@
|
||||
|
||||
参考设计要求在提交前取得 Runtime 与 Harness 的安装回执。本切片有意收窄这一点,而能安全收窄是因为已验证的合入架构性质:
|
||||
|
||||
- 每个工具轮次都在任何工具运行前获取**新的** Runtime Session 并安装当时的绑定(安装按 Runtime Session ID 记账;同一会话以不同 digest 重装会被拒绝)。新上下文的生产回执因此必然先于首次变更后的工具执行存在;缺失时该轮次以 `managed_context_unavailable` 受阻——类型化受阻,永远不会静默落在错误目录。
|
||||
- 没有保留 Hook owner 的工具轮次在任何工具运行前获取**新的** Runtime Session 并安装当时的绑定(安装按 Runtime Session ID 记账;同一会话以不同 digest 重装会被拒绝)。新上下文的生产回执因此必然先于首次变更后的工具执行存在;缺失时该轮次以 `managed_context_unavailable` 受阻——类型化受阻,永远不会静默落在错误目录。
|
||||
- worker 与控制面同主机、同 UID、同一挂载快照(由 `validateWorkspaceFiles` 与 resolver 构造期挂载检查保证),且共享的 `requireDirectory` 带上了 worker 自己的 `access(R_OK|X_OK)` 检查,因此 Java 探针判定与 worker 安装判定在同一快照上完全一致;其后的分歧只可能来自提交之后文件系统的变化,届时仍在首个 Turn 以类型化暴露,不会错向。
|
||||
- 只在 Java 侧确定性检查之后提交,使结算真实可信,同时无需发明一条绕过所有权访问 worker 安装路由的通道(该路由今天只存在于 `acquire()` 的 claim→install→activate 链内,其 ownership claim 按设计可能在崩溃歧义时留置存储——探针绝不该引入这种风险)。
|
||||
|
||||
@@ -115,7 +115,7 @@ WebShell 请求体携带 `sessionId`、`idempotencyKey`、`cwdRelative`、`expec
|
||||
- **准入 vs cancel**:cancel 只在存在活动 Turn 时才有真实效果(否则 `commandEffect` 为 false),而未关闭 operation 会挡住一切 Turn,因此在 cwd operation 未关闭期间放行的 cancel 只是一行已完成的空插入——它扰动不了提交,因为提交事务随后会重核全部事实。本要点较早的版本曾许诺此种 cancel 会引出类型化拒绝;实际交付并被测试钉住的机制是良性的一种。
|
||||
- **探针 vs 提交**:探针只读,不需要存储租约;提交事务在行锁内复核全部事实,因此探针与提交之间撤销授权或准入 Turn 只会让 operation 干净失败,而不会错误提交。
|
||||
- **崩溃窗口**:准入插入是单事务;探针+提交没有可遗留的痕迹;提交与其事件同事务;回收会幂等地重跑分支。
|
||||
- **提交后执行**:下一轮的 `acquire` 重新解析绑定、claim 存储,在新的 Runtime Session 上安装并激活新上下文并随后断言 ownership。泄漏的提交前 Runtime Session 无法执行——执行要求 ACQUIRING/READY 的记录状态,release 会关闭它;未结算的执行让其所属 Turn 保持 `RUNNING`,进而在第 14 步挡住准入。
|
||||
- **提交后执行**:下一轮的 `acquire` 重新解析绑定、claim 存储,在新的 Runtime Session 上安装并激活新上下文并随后断言 ownership。Hook owner 可在公开 Turn 完成后仍保持 READY。准入与提交在同租户/Harness 存在任何非 RELEASED Runtime Session 时拒绝 `session_context_busy`,包括不确定的 ACQUIRING/RELEASING 与尚无释放证明的 FAILED。原 worker 上下文及释放身份保持不变,不重装也不替换 Runtime。其他 Harness Session 与租户不在此谓词范围。两个事务均先取得 placement 锁,与 Broker 准入同序;精确 scope 的存在性读取增加扫描成本,不承诺固定延迟。
|
||||
|
||||
## 涉及文件
|
||||
|
||||
@@ -133,8 +133,8 @@ WebShell 请求体携带 `sessionId`、`idempotencyKey`、`cwdRelative`、`expec
|
||||
## 验证计划
|
||||
|
||||
- **Java**:JDK 21 下 `packages/sdk-java/managed-agent-server` 的 `mvn test` 与 `mvn checkstyle:check`;聚焦的 store/coordinator/controller/contract 测试。
|
||||
- **store 准入矩阵**:准入表每一行,含 CAS 前的重放顺序、摘要冲突、跨租户不可见、无读授权 404 `session_not_found` 与有读授权非创建者 403 `session_operation_forbidden`、授权被撤、Registry draining/removed、generation 漂移、期望 revision 不符、G0 初始 Turn 期间的 busy,以及同目录空变更。
|
||||
- **结算**:探针失败时会话原样保留且 `failure_code=workspace_unavailable`;外部 revision 变动后提交以 `context_revision_conflict` 失败;提交时出现新活动 Turn 以 `session_context_busy` 失败;瞬时错误重试后成功一次;永不停歇的瞬时失败在 8 次尝试预算用尽后以类型化终态结算——可投递集合清空、两道准入屏障重新放行;消失的挂载根或目标判为终态(对照真实删除的目录钉住),一时的 I/O 形态判为可重试(以确定性的 ENAMETOOLONG 探测钉住);属主死亡的 LEASED 行回收后恰好完成一次;legacy 生命周期 operation 与 `ACTION_RESPONSE` 不受影响(其测试保持绿色)。
|
||||
- **store 准入矩阵**:准入表每一行,含 CAS 前的重放顺序、摘要冲突、跨租户不可见、无读授权 404 `session_not_found` 与有读授权非创建者 403 `session_operation_forbidden`、授权被撤、Registry draining/removed、generation 漂移、期望 revision 不符、G0 初始 Turn 或保留 Runtime 上下文繁忙,以及同目录无变化请求。
|
||||
- **结算**:探针失败时会话原样保留且 `failure_code=workspace_unavailable`;外部 revision 变动后提交以 `context_revision_conflict` 失败;出现新的活动 Turn 或保留 Runtime 上下文时以 `session_context_busy` 失败;瞬时错误重试后成功一次;永不停歇的瞬时失败在 8 次尝试预算用尽后以类型化终态结算——可投递集合清空、两道准入屏障重新放行;消失的挂载根或目标判为终态(对照真实删除的目录钉住),一时的 I/O 形态判为可重试(以确定性的 ENAMETOOLONG 探测钉住);属主死亡的 LEASED 行回收后恰好完成一次;legacy 生命周期 operation 与 `ACTION_RESPONSE` 不受影响(其测试保持绿色)。
|
||||
- **契约**:映射 vs planned 漂移测试、六个触及 schema 的钉、WebShell 孪生、事件文档一致性。
|
||||
- **Hosted 集成(H2,镜像 G0 套件;加入 MySQL CI 车道)**:在目录 A 以初始文件 Turn 创建绑定会话(为后续 Turn 准备一次真实的变更前安装),等待该 Turn 完成,再经两侧 API 变更目录并做归一化重放,轮询 operation 到 `completed` 并断言会话行 revision/cwd 与公开流上的 `session.context.changed` 事件及 `/operations/query` 读取——随后经已合入的 #13112 提交一个绑定后续 Turn,断言其文件写入落在已提交的目录,而初始 Turn 自己的文件保持完整。负向组:开关关闭、legacy 会话、busy、CAS 冲突、幂等重放与冲突,以及 WebShell 失败投影携带 `failureCode`;store 层双向钉住后续 Turn 繁忙屏障(op 未关闭挡 Turn;完成后放行)与两面在终态失败后的释放。
|
||||
- **E2E 计划**:`.qwen/e2e-tests/managed-workspace-w2-cwd-change.md`,先用全局 `qwen` CLI 基线做干跑(该路由今天未映射)。
|
||||
|
||||
@@ -31,7 +31,7 @@ Two gaps, different in kind:
|
||||
slice, per merged capability. For the 56 public and WebShell routes the API
|
||||
contract test already fails a mapped route that the OpenAPI contract lacks,
|
||||
and fires a cross-tenant probe at every contract operation. Nothing gates
|
||||
the 22 internal routes, nothing probes a caller below read or with read but
|
||||
the original slice-A baseline's 22 internal routes, nothing probes a caller below read or with read but
|
||||
without the family's power, and nothing ties a route to the admission rule
|
||||
it should follow — so a route can land with the wrong check and every test
|
||||
stays green, which is the failure mode that matters most while this surface
|
||||
@@ -72,7 +72,8 @@ tenant may mutate them today; internal store/publication routes admit a writer
|
||||
HMAC credential, not an actor. The public surface is `/v1/agents/**` plus
|
||||
`/api/agent/web-shell/v1/**` (`PublicSurface`), realised by ten Spring
|
||||
controllers — the section-10 matrix enumerates the current 32 public + 24
|
||||
WebShell + 22 internal routes (78 in total, counted by the slice-A gate).
|
||||
WebShell + 24 internal routes (80 in total, including the two L3 authorization
|
||||
routes, counted by the gate).
|
||||
|
||||
There is no production provisioning of workspace registry/access rows —
|
||||
today only tests and fixture entry points write them, and a deployment writes
|
||||
@@ -220,7 +221,7 @@ the published shape of the 56 public and WebShell routes and which the API
|
||||
contract test keeps in bijection with the mounted handlers. The rule class is
|
||||
deliberately not an `x-qwen-*` extension on the spec: the spec is the
|
||||
published, machine-consumed contract (the WebShell client types are generated
|
||||
from it), it does not describe the 22 internal routes, and an admission rule
|
||||
from it), it does not describe the 24 internal routes, and an admission rule
|
||||
class is a server-internal classification. A new public or WebShell route is
|
||||
therefore named three times — controller, spec, registry — and each pairing
|
||||
is gated: the contract test fails a route the spec lacks, and the
|
||||
@@ -365,8 +366,9 @@ Same as the issue's, plus the explicit deferrals named there:
|
||||
|
||||
## 10. Surface route matrix (the slice-A registry, bilingual summary)
|
||||
|
||||
`api/SurfaceRegistry.java` at slice-A head carries 78 route constants: 32
|
||||
public + 24 WebShell + 22 internal handler methods of the ten controllers.
|
||||
`api/SurfaceRegistry.java` integrated with L3 carries 80 route constants: 32
|
||||
public + 24 WebShell + 24 internal handler methods of the ten controllers,
|
||||
including the two L3 authorization routes.
|
||||
The gate derives everything from scanning, so the count is information, not
|
||||
an asserted constant.
|
||||
|
||||
@@ -374,7 +376,7 @@ Rule classes name today's admission: `WORKSPACE_CREATE` (2), `READER` (24),
|
||||
`READER_ACTOR` (6), `READER_ACTOR_POLICY` (1), `OPERATOR` as today's
|
||||
submitter family (4), `OWNER` as today's creator families — lifecycle and
|
||||
cwd plus Action respond — (12), `WORKSPACE_DISCOVERY` (4), `TENANT_SCOPED`
|
||||
(3), `INTERNAL_WRITER` (22). The design's `legacy_create` and
|
||||
(3), `INTERNAL_WRITER` (24). The design's `legacy_create` and
|
||||
`legacy_tenant` names are kept in the class documentation as the names of
|
||||
the legacy arms: a route carries exactly one rule class and, per the
|
||||
separation rule, it is the bound-Session one. Slice C flips cwd and Action
|
||||
@@ -448,6 +450,8 @@ store route and a publication route.
|
||||
| `POST /api/agent/web-shell/v1/artifacts/query` | WEBSHELL | ARTIFACT_LIST | READER_ACTOR |
|
||||
| `POST /api/agent/web-shell/v1/workspaces/query` | WEBSHELL | WORKSPACE_LIST | WORKSPACE_DISCOVERY |
|
||||
| `POST /api/agent/web-shell/v1/workspaces/get` | WEBSHELL | WORKSPACE_GET | WORKSPACE_DISCOVERY |
|
||||
| `POST /internal/managed-session-store/v1/sessions/{sessionId}/execution:authorize` | INTERNAL | STORE_EXECUTION_AUTHORIZE | INTERNAL_WRITER |
|
||||
| `POST /internal/managed-session-store/v1/sessions/{sessionId}/lifecycle:authorize` | INTERNAL | STORE_LIFECYCLE_AUTHORIZE | INTERNAL_WRITER |
|
||||
| `POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:acquire` | INTERNAL | STORE_WRITER_ACQUIRE | INTERNAL_WRITER |
|
||||
| `POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:renew` | INTERNAL | STORE_WRITER_RENEW | INTERNAL_WRITER |
|
||||
| `POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:seal` | INTERNAL | STORE_WRITER_SEAL | INTERNAL_WRITER |
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
两个性质不同的缺口：
|
||||
|
||||
- **R1 —— 没有角色词表。** 到目前为止合入的每个绑定 Session 操作都只准入其创建者，对其他任何人返回 `404` 或 `403`。`AuthenticatedTenantActor` 只带 `tenantId()` 与 `actorId()`，此外什么都没有。两个产品行为今天被阻塞：同一 Workspace 上的第二个操作者无法回答一个正阻塞 Turn 的审批；绑定 Workspace 的 Session 无法移交，因为没有任何东西能表达「与创建者不同的所有者」。
|
||||
- **R2 —— 没有系统性验收。** 准入覆盖是逐切片随各合入能力累积的。对 56 条公开与 WebShell 路由，API 契约测试已经会让契约里缺失的已挂载路由失败，并对每个契约 operation 发跨租户探针。但 22 条内部路由没有任何门禁，没有探针覆盖低于读权限或有读但无该族权限的调用方，也没有任何东西把一条路由与它应遵循的准入规则绑在一起 —— 于是一条路由可能带着错误的校验落地而所有测试保持全绿，而这个面还在快速增长，这恰恰是最要紧的失效模式。
|
||||
- **R2 —— 没有系统性验收。** 准入覆盖是逐切片随各合入能力累积的。对 56 条公开与 WebShell 路由，API 契约测试已经会让契约里缺失的已挂载路由失败，并对每个契约 operation 发跨租户探针。但原切片 A 基线的 22 条内部路由没有任何门禁，没有探针覆盖低于读权限或有读但无该族权限的调用方，也没有任何东西把一条路由与它应遵循的准入规则绑在一起 —— 于是一条路由可能带着错误的校验落地而所有测试保持全绿，而这个面还在快速增长，这恰恰是最要紧的失效模式。
|
||||
|
||||
## 2. 现状
|
||||
|
||||
@@ -32,7 +32,7 @@
|
||||
| 生命周期（close、archive、unarchive、delete）+ cwd 变更 | `POST …/close` / `…/archive` / `…/unarchive`、`DELETE`、`POST …/cwd`，及 WebShell 孪生 | `requireWorkspaceCreator`（先 can_read 再创建命令行） | 403 `session_operation_forbidden` |
|
||||
| Action（审批）回答 | `POST …/actions/{id}/responses`、WebShell `actions/respond` | `requireOwner`（creator_actor_key，回退创建命令） | 403 `action_forbidden` |
|
||||
|
||||
其余已实现的规则形态：绑定读取与全部 list/stream/catalog 路由要求 `can_read`（否则 404）；绑定创建要求 actor + `can_create` + `ACTIVE` Workspace（按失败点返回 401/404/403/409）；artifact 字节读取叠加部署策略门（`403 artifact_content_forbidden`）；Workspace 发现只列出 `can_read` 行（无 actor 返回 401）；legacy（未绑定）Session 与 agent 定义是**租户级**的 —— 租户内任何 actor 今天都可以改它们；内部 store/publication 路由准入 writer HMAC 凭据而非 actor。公开面是 `/v1/agents/**` 加 `/api/agent/web-shell/v1/**`（`PublicSurface`），由十个 Spring controller 实现 —— 第 10 节的矩阵枚举当前的 32 条公开 + 24 条 WebShell + 22 条内部路由（切片 A 门禁实计共 78 条）。
|
||||
其余已实现的规则形态：绑定读取与全部 list/stream/catalog 路由要求 `can_read`（否则 404）；绑定创建要求 actor + `can_create` + `ACTIVE` Workspace（按失败点返回 401/404/403/409）；artifact 字节读取叠加部署策略门（`403 artifact_content_forbidden`）；Workspace 发现只列出 `can_read` 行（无 actor 返回 401）；legacy（未绑定）Session 与 agent 定义是**租户级**的 —— 租户内任何 actor 今天都可以改它们；内部 store/publication 路由准入 writer HMAC 凭据而非 actor。公开面是 `/v1/agents/**` 加 `/api/agent/web-shell/v1/**`（`PublicSurface`），由十个 Spring controller 实现 —— 第 10 节的矩阵枚举当前的 32 条公开 + 24 条 WebShell + 24 条内部路由（含 L3 两条授权入口，门禁实计共 80 条）。
|
||||
|
||||
workspace registry/access 行没有任何生产置备路径 —— 今天只有测试与 fixture 入口写它们，部署环境靠带外方式写入；全仓没有 role 列、owner 列，也没有按租户存放 actor 的表。
|
||||
|
||||
@@ -112,7 +112,7 @@ owner 的更新路径（移交命令）是建在此列之上的后续切片；
|
||||
|
||||
注册表放在 `src/test/java`，因为生产代码里没有任何地方读它：下文的门禁与验收探针是它仅有的消费者，本切片如此，切片 C 也如此 —— C 的强制执行读的是存储的角色。只有出现运行时消费者时它才移到 `src/main`。
|
||||
|
||||
它不取代 OpenAPI 契约（`managed-agent-public-api.openapi.json`）：契约仍是 56 条公开与 WebShell 路由对外发布形态的唯一来源，API 契约测试让它与已挂载的 handler 保持双射。规则类刻意不作为 `x-qwen-*` 扩展写进契约：契约是对外发布、被机器消费的（WebShell 客户端类型由它生成），它不描述 22 条内部路由，而准入规则类是服务端内部的分类。因此一条新的公开或 WebShell 路由要被点名三次 —— controller、契约、注册表 —— 且每一对都有门禁：契约测试让契约缺失的路由失败，对账门禁让注册表缺失的路由失败。
|
||||
它不取代 OpenAPI 契约（`managed-agent-public-api.openapi.json`）：契约仍是 56 条公开与 WebShell 路由对外发布形态的唯一来源，API 契约测试让它与已挂载的 handler 保持双射。规则类刻意不作为 `x-qwen-*` 扩展写进契约：契约是对外发布、被机器消费的（WebShell 客户端类型由它生成），它不描述 24 条内部路由，而准入规则类是服务端内部的分类。因此一条新的公开或 WebShell 路由要被点名三次 —— controller、契约、注册表 —— 且每一对都有门禁：契约测试让契约缺失的路由失败，对账门禁让注册表缺失的路由失败。
|
||||
|
||||
### D6 —— 构建门禁
|
||||
|
||||
@@ -179,14 +179,14 @@ A ∥ B 是安全的：文件不相交（A 纯新增；B 改 store 侧）。C
|
||||
|
||||
## 10. Surface 路由矩阵（切片 A 注册表，双语摘要）
|
||||
|
||||
切片 A 头的 `api/SurfaceRegistry.java` 携带 78 条路由常量：十个
|
||||
controller 的 32 公开 + 24 WebShell + 22 internal handler 方法。门禁从扫描推导一切，计数只是信息，不是被断言的常量。
|
||||
与 L3 整合后的 `api/SurfaceRegistry.java` 携带 80 条路由常量：十个
|
||||
controller 的 32 公开 + 24 WebShell + 24 internal handler 方法（含 L3 两条授权入口）。门禁从扫描推导一切，计数只是信息，不是被断言的常量。
|
||||
|
||||
规则类按今天的准入命名：`WORKSPACE_CREATE`（2）、`READER`（24）、
|
||||
`READER_ACTOR`（6）、`READER_ACTOR_POLICY`（1）、`OPERATOR` 作为今天的
|
||||
submitter 族（4）、`OWNER` 作为今天的 creator 各族 —— lifecycle、cwd
|
||||
与 Action respond（12）、`WORKSPACE_DISCOVERY`（4）、`TENANT_SCOPED`
|
||||
（3）、`INTERNAL_WRITER`（22）。设计列出的 `legacy_create` 与
|
||||
（3）、`INTERNAL_WRITER`（24）。设计列出的 `legacy_create` 与
|
||||
`legacy_tenant` 两个名字保留在类的文档里，作为 legacy 分支的名字：
|
||||
每条路由恰有一个规则类，按分离规则取的是绑定 Session 的类。切片 C
|
||||
会把 cwd 与 Action respond 从 `OWNER` 翻到 `OPERATOR`，归一
|
||||
@@ -253,6 +253,8 @@ submitter 族的拒绝码，并且只有在探针需要不同期望时才拆开
|
||||
| `POST /api/agent/web-shell/v1/artifacts/query` | WEBSHELL | ARTIFACT_LIST | READER_ACTOR |
|
||||
| `POST /api/agent/web-shell/v1/workspaces/query` | WEBSHELL | WORKSPACE_LIST | WORKSPACE_DISCOVERY |
|
||||
| `POST /api/agent/web-shell/v1/workspaces/get` | WEBSHELL | WORKSPACE_GET | WORKSPACE_DISCOVERY |
|
||||
| `POST /internal/managed-session-store/v1/sessions/{sessionId}/execution:authorize` | INTERNAL | STORE_EXECUTION_AUTHORIZE | INTERNAL_WRITER |
|
||||
| `POST /internal/managed-session-store/v1/sessions/{sessionId}/lifecycle:authorize` | INTERNAL | STORE_LIFECYCLE_AUTHORIZE | INTERNAL_WRITER |
|
||||
| `POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:acquire` | INTERNAL | STORE_WRITER_ACQUIRE | INTERNAL_WRITER |
|
||||
| `POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:renew` | INTERNAL | STORE_WRITER_RENEW | INTERNAL_WRITER |
|
||||
| `POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:seal` | INTERNAL | STORE_WRITER_SEAL | INTERNAL_WRITER |
|
||||
|
||||
@@ -0,0 +1,289 @@
|
||||
# Reliable deletion of ACTIVE Workspace Sessions (L3)
|
||||
|
||||
[English](workspace-session-active-delete-l3.md) | [简体中文](workspace-session-active-delete-l3.zh-CN.md)
|
||||
|
||||
## 1. Status and scope
|
||||
|
||||
Implemented locally on `codex/workspace-session-l3`, 2026-10-03. Linux physical
|
||||
acceptance remains pending. This implements L3 of
|
||||
[#13164](https://github.com/QwenLM/qwen-code/issues/13164), on main after
|
||||
reliable close #13135, L1/L2 #13194, O4 retirement #13084 and H2 Hooks #13129.
|
||||
It admits deletion of idle `hosted-workspace-files/1` Sessions through the
|
||||
existing public and WebShell routes. Shell/MCP profiles, buttons, physical
|
||||
erasure, new roles and abandoning unknown effects remain out of scope.
|
||||
|
||||
| Operation | Lifecycle Hooks | Result |
|
||||
| ---------------------- | ------------------------------ | -------------------------------------------- |
|
||||
| ACTIVE close | SessionEnd | CLOSED, retained data |
|
||||
| ACTIVE delete | SessionEnd, then SessionDelete | Reliable stop, atomic retirement and DELETED |
|
||||
| CLOSED/ARCHIVED delete | None, including deferred Hooks | Existing L2 metadata deletion |
|
||||
| Detach | None | Attachment cleanup |
|
||||
|
||||
The current-readable creator remains the mutation authority. An accepted,
|
||||
running, cancelling or approval-waiting Turn rejects admission with
|
||||
`409 turn_active`. Idempotency, actor isolation and tombstone visibility retain
|
||||
their L2 semantics. Once means reusing committed outcomes and never redispatching
|
||||
an attempt that might have begun, not guaranteed eventual completion.
|
||||
|
||||
### Why close followed by L2 delete is insufficient
|
||||
|
||||
Reliable close settles SessionEnd and permanently stops the original Runtime.
|
||||
L2 deletion of CLOSED/ARCHIVED Sessions intentionally runs no Hooks. Composing
|
||||
those operations therefore cannot settle SessionDelete on the original Runtime
|
||||
after End and before shutdown. Restoring a replacement worker to run Delete
|
||||
would violate the no-replay boundary. L3 reuses reliable close's stop machinery
|
||||
and L2's atomic retirement transaction, but settles both required events before
|
||||
permanent draining. One delete operation remains DELETING throughout recovery;
|
||||
it does not publish a separate successful CLOSE and later attempt an unrelated
|
||||
L2 operation. The effects receipt lets successors skip committed Hook outcomes
|
||||
without treating a missing Harness or an HTTP response as completion evidence.
|
||||
|
||||
## 2. Protocol and evidence
|
||||
|
||||
Add private `POST /session/:id/lifecycle`, carrying Session scope, operationId,
|
||||
kind (`close` or `delete`) and claimGeneration. It settles only the required
|
||||
Hooks and returns references to committed H2 records, retaining the attachment,
|
||||
writer and owner until Java accepts those effects. Java validates authoritative
|
||||
Session-store records; HTTP success alone is not proof.
|
||||
|
||||
Operations save a lifecycle protocol version and an intermediate effects receipt.
|
||||
The receipt binds Session and operation identity to required event occurrences,
|
||||
plans and committed result references, or verified no-Hook evidence. The public
|
||||
receipt_id continues to be issued only by final completion.
|
||||
|
||||
The receipt is not a substitute for worker-stop evidence. Final completion checks
|
||||
the permanent fence, writer exclusion, all original Runtime resources and
|
||||
matching binding/generation/handle stop proofs. RELEASED alone is insufficient;
|
||||
historical bindings without verifiable stop or never-started evidence block.
|
||||
|
||||
The public routes and 202 operation responses remain unchanged. Advertise
|
||||
session_delete/sessionDelete for supported ACTIVE files Sessions independently
|
||||
of archive/unarchive. Capability denotes support, not authorization or idle
|
||||
state. Keep the aggregate session_lifecycle unchanged.
|
||||
|
||||
## 3. Execution and recovery
|
||||
|
||||
Admission saves CLOSING/DELETING and a durable LIFECYCLE_ONLY fence together.
|
||||
Ordinary warm, acquire, tool execution, input and other control mutations are
|
||||
excluded. Only the current operation and live claim may obtain lifecycle writer
|
||||
and execution authority in the original Workspace scope. This authority still
|
||||
requires private authentication and current execution authorization. Use the
|
||||
existing lock hierarchy to prevent authorization/admission races.
|
||||
|
||||
The placement guard precedes retention, Session and journal locks. Checking only
|
||||
the journal head or a missing fence row cannot exclude concurrent fence
|
||||
insertion; writer mutations and ordinary execution authorization must share
|
||||
admission's lock order. A tool-result receipt takes the placement guard before
|
||||
its original-result, publication tenant and journal locks, including receipt
|
||||
replay; its nested journal commit must not acquire placement in reverse order.
|
||||
Offline storage migration, including an ABORTED operation replay, follows the
|
||||
same placement-before-retention order as cleanup authorization for another
|
||||
Session in the tenant. This does not relax the stopped-writer/no-new-dispatch
|
||||
migration prerequisite.
|
||||
An ordinary journal transaction definitively refused with
|
||||
`workspace_lifecycle_admission_closed` before any uncertain commit response
|
||||
does not stop the cached Session's writes; current lifecycle authority is still
|
||||
required for subsequent settlement. An uncertain earlier response retains the
|
||||
write-failure fence.
|
||||
The pre-L3 Store already obtains the publication tenant
|
||||
lock through its Session lock helper. L3 adds the placement guard and ordinary
|
||||
authorization request, so its incremental contention and request cost require
|
||||
measurement. Drain lookups and mutations use the existing hashed primary key,
|
||||
with original identity checks, so their locking reads do not scan other tenants'
|
||||
fences. The key encoding is shared with the Broker; no new index is required.
|
||||
This protects persisted Session admission on every hosted
|
||||
attachment, including one without local lifecycle state; it does not grant
|
||||
every Session L3 delete support. A definite Store lifecycle-fence rejection
|
||||
returns 409. Unexpected Store, transport or writer-authority failures return
|
||||
503 and admit no execution. Authenticated cancellation of an admitted Turn is a
|
||||
local abort and does not require ordinary Store authorization; the local
|
||||
lifecycle fence still excludes cancellation. Protocol-zero close retains its
|
||||
scoped exception. Ordinary attachment mutations validate client identity before
|
||||
Store authorization or writer renewal; the legacy close control route retains
|
||||
its explicit missing-client exception. Lifecycle claims use database millisecond
|
||||
time and keep their non-retryable 409 through the Broker boundary. A DELETE
|
||||
classified before a concurrent CLOSE completes is reclassified under the locked
|
||||
Session state, preserving CLOSED/ARCHIVED deletion as L2.
|
||||
|
||||
Settle earlier operations, excluding this operation's lifecycle occurrences from
|
||||
generic cancellation. Stable occurrence IDs derive from Session, operation and
|
||||
event. Reuse H2 catalog, plan, dispatch intent and results. SessionDelete starts
|
||||
only after SessionEnd's children have committed their outcomes, including async
|
||||
Hooks; a settled plan marker alone is insufficient. Cold load restores saved state without a user
|
||||
Turn or startup Hook. Reuse a verifiable original Runtime; a replacement
|
||||
generation never replays effects. First initialization is permitted only when
|
||||
no Runtime has ever been created and authorization is valid.
|
||||
Plan identities use fixed compact JSON bytes, independent of application JSON
|
||||
formatting. Recovery rejects pending, failed or cancelled local Session routes
|
||||
without waiting for ordinary acquisition or propagating its outcome.
|
||||
|
||||
A successor lifecycle load may adopt an existing idle files attachment only
|
||||
after its normalized Store descriptor matches the original grant and the Store
|
||||
accepts the current claim on the original writer. It returns the original client
|
||||
identity without reopening the Session or replaying Hooks. Busy, foreign and
|
||||
refused claims retain the attachment and its previous authority. Client identity
|
||||
is checked before disclosing the local lifecycle fence, preserving the explicit
|
||||
legacy close exception. Cancelling a never-dispatched original execution may
|
||||
settle its durable cancellation through a drain fence without invoking the worker;
|
||||
new dispatch remains excluded. Typed transactional claim refusals retain their
|
||||
409 code and roll back all journal mutations. Confirmed lifecycle detach removes
|
||||
only the captured attachment and prompt state; a late response cannot clear a
|
||||
replacement's heartbeat or prompt watermark.
|
||||
|
||||
Before a new lifecycle Hook leaves intent, restore its saved original Broker
|
||||
owner by binding and generation. A surviving Harness attachment cannot infer
|
||||
that a restarted Broker still has that owner. Failed attestation leaves the
|
||||
child intent retryable without dispatch; current authority is checked again
|
||||
before dispatch. Already unknown outcomes remain blocked and are not replayed.
|
||||
|
||||
Before each new side-effect dispatch, check current ACL, mount and identity.
|
||||
After revocation, lookup and settlement of already dispatched work continue;
|
||||
undispatched Hooks stay recovery_blocked until authority returns. Unknown effects
|
||||
retain ownership and do not become cancellation or completion proofs. This
|
||||
inherits H2's potentially indefinite blocking, tracked separately in #13133.
|
||||
|
||||
Precheck authority before both plan and child dispatch, and recheck inside the
|
||||
journal commit transaction. A definite transactional authorization refusal leaves
|
||||
the writer authority retryable without consuming journal sequences. A missing
|
||||
response, or a refusal after an uncertain request, retains the write failure fence.
|
||||
The guard checks every Hook revision, including multiple revisions in one commit:
|
||||
leaving intent for a possibly started outcome requires authorization, while
|
||||
not_started_proven and original dispatched-result settlement remain available.
|
||||
Lifecycle settlement cannot change an already recorded Runtime identity.
|
||||
|
||||
After validating and saving the effects receipt, upgrade the fence monotonically
|
||||
to DRAINING. Detach without Hooks, release owners, seal the writer, then use the
|
||||
reliable-close drain/stop protocol. A successor skips Hooks when the effects
|
||||
receipt is saved, otherwise reconstructs progress from the same H2 occurrences.
|
||||
Harness 404, lease expiry and worker disappearance cannot establish completion.
|
||||
A confirmed same-boot detach 404 stops the SDK attachment heartbeat; refusals and
|
||||
ambiguous responses retain it. Receipt recovery selects the exact occurrence and
|
||||
resource through their existing primary keys, preserving raw identity and byte
|
||||
validation without scanning other tenants or looping over historical candidates.
|
||||
|
||||
Receipt recovery may skip the Harness lifecycle request entirely. Detach must
|
||||
therefore validate its operation authority against Session Store even when the
|
||||
live attachment has no local lifecycle state. Store authorization checks the
|
||||
current claim, original writer, saved effects and DRAINING fence; a rejected
|
||||
claim leaves the attachment and its prior authority intact. A successor without
|
||||
a cached attachment addresses the original Session ID with its current claim;
|
||||
it does not load a Runtime or dispatch Hooks to recover a client ID. Missing
|
||||
authority still requires the original client ID, and a supplied wrong client ID
|
||||
is rejected. Cleanup authorization accepts only the same writer identity, token
|
||||
and generation, including an expired or already sealed writer; it does not renew
|
||||
that writer. Hook dispatch still requires an active, unexpired writer. Ordinary
|
||||
detach still requires ordinary execution authorization.
|
||||
|
||||
After that cleanup authorization, stop activation renewal and seal the original
|
||||
writer without appending an activation release record. Expired or sealed writers
|
||||
cannot append, and an already-started renewal remains constrained by the Store
|
||||
fence and writer seal. The historical activation may still read as active; it is
|
||||
not evidence of a normal release. Completion relies on the permanent fence,
|
||||
effects receipt, writer seal and original Runtime stop proofs. Ordinary and
|
||||
legacy close continue to record activation release before sealing.
|
||||
|
||||
Final completion requires the live delivery claim, effects receipt, permanent
|
||||
fence, no live writer or unsettled execution, and verifiable original stop
|
||||
proofs. CLOSE commits CLOSED. DELETE commits O4 retirement, DELETED, operation
|
||||
completion and the terminal event atomically. Shared files and other Sessions'
|
||||
holders remain intact.
|
||||
|
||||
For a never-initialized Session, acquire the tenant-retention, public Session and
|
||||
journal-head exclusion in writer order, and prove no completed bootstrap, live
|
||||
writer, journal or Hook dispatch records. Save never-initialized no-Hook evidence.
|
||||
With a header, inspect the original definition/catalog instead. Either case
|
||||
still checks the complete Runtime binding set after the permanent fence.
|
||||
|
||||
Admission scans and JSON-parses the complete journal to prove there are no
|
||||
accepted, unsettled Turns. Its cost grows with retained history while the
|
||||
transaction holds its locks; bounded admission latency is not established.
|
||||
The current product initializes the compaction watermark to zero and has no
|
||||
production path that advances it. Existing cold recovery also rejects nonzero
|
||||
watermarks; L3 rejects them with `workspace_lifecycle_journal_unverified`.
|
||||
Future compaction requires durable idle-state evidence before L3 can
|
||||
accept those Sessions; this PR does not implement compaction recovery.
|
||||
|
||||
## 4. Compatibility and rollout
|
||||
|
||||
Add lifecycle migration V51 after the existing V35 tool-profile, V36–V39
|
||||
journal/query, V40 creator, V41–V44 CSI/dispatch, V45 H3 task-journal, V46
|
||||
W2 directory-change, V47 H5 channel route/delivery and V48–V50 W1c storage
|
||||
migration/index/identity migrations; preserve those migrations and V32. The
|
||||
unmerged lifecycle SQL is unchanged when its version moves from V48 to V51.
|
||||
Main's `/2` search profiles coexist with L3, but lifecycle
|
||||
admission still requires `hosted-workspace-files/1`. H5 channel contracts and
|
||||
persistence do not enable channel execution or lifecycle effects. Keep W2's
|
||||
directory/revision operation fields alongside the lifecycle protocol field.
|
||||
Its read-only probe and settlement remain separate from lifecycle Hook effects,
|
||||
drain and detach; the Session-row admission barrier serializes the operations.
|
||||
Runtime resolution retains the lifecycle claim on its actual resolved scope.
|
||||
W2 admission and commit take placement first and refuse `session_context_busy`
|
||||
while any same-tenant/Harness Runtime Session lacks confirmed RELEASED state.
|
||||
Hook owners can outlive a completed Turn; their immutable original context must
|
||||
remain available for L3 effects and release. Neighbor holders are unaffected,
|
||||
and no replacement Runtime or context reinstall is introduced. The existence
|
||||
read adds scan cost without a fixed latency guarantee.
|
||||
This integration preserves
|
||||
H3 background Shell/Monitor behavior and does not add lifecycle support for
|
||||
CSI, Shell or MCP profiles. The shared Harness closes its Monitor wake
|
||||
scheduler only after detach authorization succeeds, so a refused detach leaves
|
||||
the live Session and its scheduler intact. The synchronous closing/busy fence
|
||||
still prevents a wake from starting while authorization is pending.
|
||||
G3 generation adoption also covers lifecycle settlement and detach. A call
|
||||
that discovers a new Harness generation invalidates the stale client and
|
||||
attachment and still returns the generation error. The existing delivery retry
|
||||
renegotiates with the same operation and current claim; it does not redispatch
|
||||
within the discovering call, enable ordinary recovery or cancellation takeover,
|
||||
replace the original Runtime, or clear an unknown outcome. The new Harness boot
|
||||
is not stop evidence for the original Runtime; effects receipts, writer seals
|
||||
and original-handle stop proofs remain required.
|
||||
Historical admitted operations retain their original protocol and evidence,
|
||||
without new Hook identities.
|
||||
Operation reads before the lifecycle migration treat an absent protocol column
|
||||
as legacy protocol-zero; a present protocol-one value remains unchanged.
|
||||
This projection compatibility does not grant execution or bypass schema upgrades.
|
||||
Live protocol-zero close attachments retain their legacy DELETE and original
|
||||
Hook control path only while their persisted close claim is valid. Ordinary
|
||||
execution remains fenced; the exception cannot authorize L3 or MCP execution.
|
||||
Legacy claim validity compares database epoch milliseconds on both sides,
|
||||
independent of JVM, JDBC and database session time zones.
|
||||
Upgrade every Spring coordinator/Store first, then the Hosted Harnesses. There
|
||||
is no separate L3 enable switch. A new Harness against an old Store lacking
|
||||
ordinary execution authorization refuses all hosted Turns, including private
|
||||
Sessions; it must not treat a missing authorization route as permission.
|
||||
During the Store-first mixed-version interval, ordinary Turns remain available,
|
||||
but ACTIVE Workspace close and delete are unavailable until the Harnesses
|
||||
advertise the new lifecycle capability. Schedule this temporary loss of close
|
||||
availability as part of the rollout. A missing protocol capability rejects
|
||||
admission rather than falling back to legacy DELETE. Do not
|
||||
roll back to old coordinators while L3 operations remain unfinished. L2
|
||||
CLOSED/ARCHIVED deletion remains independent of Harness availability.
|
||||
|
||||
The configured Store writer-credential policy applies before lifecycle or
|
||||
ordinary writer state is read. Lifecycle loads carry the provisioned credential
|
||||
and transport policy unchanged. Cleanup still requires the original writer and
|
||||
current claim; an expired or SEALED cleanup grant does not bypass credential
|
||||
validation. A credential-policy change may leave recovery blocked and cannot
|
||||
justify a replacement writer or Runtime.
|
||||
|
||||
Implementation order: protocol and fences; scoped authority and receipts;
|
||||
close semantics; ACTIVE delete admission and capability; recovery validation.
|
||||
Synchronize canonical OpenAPI and related bilingual designs.
|
||||
|
||||
## 5. Validation and acceptance
|
||||
|
||||
Cover both HTTP surfaces, replay/actor isolation, every active Turn state, empty
|
||||
Sessions, absent catalogs and empty Hook plans. Assert close End=1/Delete=0,
|
||||
ACTIVE delete End=1/Delete=1, and detach/L2 Hook=0. Inject lost replies and crashes
|
||||
around dispatch, result commit, receipt, detach, stop and tombstone. A second
|
||||
server must resume without duplicate effects; stale claims cannot advance.
|
||||
|
||||
Exercise ACL/mount revocation between Hooks, restoration, unknown effects,
|
||||
unverifiable identity, historical missing stop proof, and ordinary admission
|
||||
races. Real MySQL checks transaction rollback and concurrency. Real Linux checks
|
||||
Harness/workers, host/boot/PID identity, retained shared files and neighbor
|
||||
holders. Simulation is reported separately from physical validation.
|
||||
|
||||
Run build, typecheck, bundle, focused TS/Java tests and the E2E plan, then two
|
||||
clean self-audit passes and independent review. Results and environment limits
|
||||
are recorded in `.qwen/e2e-tests/workspace-session-active-delete-l3.md`.
|
||||
@@ -0,0 +1,212 @@
|
||||
# ACTIVE Workspace 会话的可靠删除（L3）
|
||||
|
||||
[English](workspace-session-active-delete-l3.md) | [简体中文](workspace-session-active-delete-l3.zh-CN.md)
|
||||
|
||||
## 1. 状态与范围
|
||||
|
||||
已在 `codex/workspace-session-l3` 本地实现，2026-10-03，Linux 实体验收仍待完成。本文实现
|
||||
[#13164](https://github.com/QwenLM/qwen-code/issues/13164) 的 L3，基于 main 已合入的
|
||||
可靠 close #13135、L1/L2 #13194、O4 退役 #13084 和 H2 Hooks #13129。
|
||||
通过既有公开和 WebShell 路由接纳空闲的 `hosted-workspace-files/1` 会话删除。
|
||||
Shell/MCP profile、按钮、物理擦除、新角色和放弃 unknown 副作用不在范围内。
|
||||
|
||||
| 操作 | 生命周期 Hook | 结果 |
|
||||
| ---------------------- | ---------------------------- | ---------------------------- |
|
||||
| ACTIVE close | SessionEnd | CLOSED，保留数据 |
|
||||
| ACTIVE delete | SessionEnd，再 SessionDelete | 可靠停机，原子退役与 DELETED |
|
||||
| CLOSED/ARCHIVED delete | 不运行，也不补跑 Hook | 既有 L2 元数据删除 |
|
||||
| Detach | 不运行 | 清理 attachment |
|
||||
|
||||
mutation 仍要求当前可读的创建者。已接纳、运行中、取消中或等待审批的 Turn
|
||||
返回 `409 turn_active`。幂等、actor 隔离和墓碑可见性沿用 L2。
|
||||
运行一次指复用已提交结果，不重新派发可能已开始的尝试，不承诺最终一定完成。
|
||||
|
||||
### 为什么不能先 close 再执行 L2 delete
|
||||
|
||||
可靠 close 结算 SessionEnd 后永久停止原 Runtime；CLOSED/ARCHIVED 的 L2 删除
|
||||
明确不运行 Hook。组合两步无法在 End 之后、停机之前，用原 Runtime 结算
|
||||
SessionDelete。启动替代 worker 补跑 Delete 又会违反禁止重放的边界。L3 复用
|
||||
可靠 close 的停机机制和 L2 的原子退役事务，在永久 draining 前结算两个所需
|
||||
事件。单个 delete operation 在恢复期间持续保持 DELETING，不先公开完成一个
|
||||
CLOSE，再尝试无关的 L2 operation。effects receipt 让接管者跳过已提交 Hook
|
||||
结果，不把 Harness 消失或 HTTP 应答当成完成证明。
|
||||
|
||||
## 2. 协议与证据
|
||||
|
||||
新增私有 `POST /session/:id/lifecycle`，携带 Session scope、operationId、
|
||||
kind（`close` 或 `delete`）和 claimGeneration。它只结算所需 Hook，返回已提交
|
||||
H2 记录的引用；Java 接受这些副作用前保留 attachment、writer 和 owner。
|
||||
Java 核对权威 Session Store 记录，HTTP 成功本身不是证明。
|
||||
|
||||
operation 保存生命周期协议版本和中间 effects receipt。回执将 Session 和操作
|
||||
身份绑定到所需事件的 occurrence、plan、已提交结果引用，或经过核验的无 Hook
|
||||
证据。公开 receipt_id 仍仅在最终完成时生成。
|
||||
|
||||
该回执不能替代 worker 停机证明。最终完成检查永久围栏、writer 排他、全部原
|
||||
Runtime 资源及匹配 binding/generation/handle 的停机证明。RELEASED 本身不足；
|
||||
历史 binding 缺少可核验的 stop 或 never-started 证据时阻塞。
|
||||
|
||||
公开路由和 202 operation 响应不变。对支持的 ACTIVE files 会话独立宣告
|
||||
session_delete/sessionDelete，不联动 archive/unarchive。capability 表示支持，
|
||||
不表示授权或空闲状态。聚合 session_lifecycle 保持原值。
|
||||
|
||||
## 3. 执行与恢复
|
||||
|
||||
准入一起保存 CLOSING/DELETING 和持久 LIFECYCLE_ONLY 围栏，禁止普通 warm、
|
||||
acquire、工具执行、输入和其他控制 mutation。仅当前 operation 与有效 claim
|
||||
可以在原 Workspace scope 获取生命周期 writer 和执行权限。该权限仍要求
|
||||
私有认证和当前执行授权。使用既有锁层级消除授权与执行准入之间的竞态。
|
||||
|
||||
placement guard 先于 retention、Session 和 journal 锁。仅检查 journal head
|
||||
或尚不存在的围栏行不能排除并发插入围栏；writer 变更和普通执行授权必须共用
|
||||
准入的锁顺序。工具结果 receipt 在原结果、publication 租户和 journal 锁之前
|
||||
获取 placement guard，receipt 重放也遵循此顺序，避免嵌套 journal commit
|
||||
反向获取 placement 锁。离线 storage 迁移（包括 ABORTED operation 重放）与同租户
|
||||
其他 Session 的清理授权使用相同的 placement 先于 retention 顺序；不放宽迁移
|
||||
要求 writer 已停止且不新增派发的前提。普通 journal 事务在尚无不确定提交响应前收到明确的
|
||||
`workspace_lifecycle_admission_closed` 回滚拒绝时，不会使缓存 Session 永久停写；
|
||||
后续结算仍须有效的当前 lifecycle authority。若此前已有不确定响应，仍保留
|
||||
写失败围栏。L3 之前的 Store 已经通过 Session 锁 helper 获取 publication
|
||||
租户锁；新增的是 placement guard 和普通授权请求，其增量竞争和请求开销需要
|
||||
测量。围栏查询和变更使用既有哈希主键，同时保留原身份检查，避免锁定读取扫描
|
||||
其他租户的围栏；键编码与 Broker 共用，不新增索引。
|
||||
这保护所有 hosted attachment 的持久会话准入，包括没有本地 lifecycle
|
||||
状态的 attachment，不代表所有会话都支持 L3 删除。明确的 Store 生命周期围栏
|
||||
拒绝返回 409；意外 Store、传输或 writer 权限故障返回 503，均不放行执行。
|
||||
已接纳 Turn 的认证取消是本地 abort，不要求普通 Store 授权；本地生命周期围栏
|
||||
仍禁止取消生命周期操作。protocol-zero close 保留限定的例外。
|
||||
普通 attachment mutation 在 Store 授权或 writer 续租前核验 client 身份；
|
||||
legacy close 控制路由保留明确的缺失 client 例外。生命周期 claim 使用数据库
|
||||
毫秒时钟，其不可重试 409 穿过 Broker 边界后保持不变。DELETE 若在并发 CLOSE
|
||||
完成前被分类，仍按加锁后的 Session 状态重新分类，保留 CLOSED/ARCHIVED 的 L2 删除。
|
||||
|
||||
先结算此前操作，通用取消不得包含本 operation 的生命周期 occurrence。
|
||||
稳定 occurrence ID 由 Session、operation、事件派生，复用 H2 的 catalog、plan、
|
||||
dispatch intent 和结果。SessionEnd 的全部子执行提交结果后才开始 SessionDelete，
|
||||
包括异步 Hook；只有 plan marker 结算不足。冷加载只恢复
|
||||
保存状态，不创建用户 Turn 或 startup Hook。复用可核验的原 Runtime，不用替代
|
||||
代际重放副作用。仅从未创建 Runtime 且授权有效时允许首次初始化。
|
||||
Plan 身份使用固定的紧凑 JSON 字节，不受应用 JSON 格式化配置影响。
|
||||
恢复遇到在途、失败或已取消的本地 Session 路由时直接拒绝，
|
||||
不等待普通 acquire，也不传播其结果。
|
||||
|
||||
接管者的 lifecycle load 只有在规范化 Store descriptor 与原 grant 完全一致、
|
||||
且 Store 接受原 writer 上的当前 claim 后，才可复用已存在且空闲的 files attachment。
|
||||
应答返回原 client 身份，不重新打开 Session 或重放 Hook。busy、外来身份或 claim
|
||||
拒绝均保留原 attachment 与此前权限。返回本地生命周期围栏前核验 client 身份，
|
||||
仍保留明确的 legacy close 例外。原从未派发执行的取消可穿过 drain 围栏结算持久
|
||||
取消结果，不调用 worker；新派发仍被拒绝。事务中的 typed claim 拒绝保留原 409
|
||||
code，并回滚全部 journal mutation。确认 lifecycle detach 后仅删除请求前捕获的
|
||||
attachment 与 prompt 状态；晚到应答不得清除替代 attachment 的心跳或 prompt 水位。
|
||||
|
||||
新生命周期 Hook 离开 intent 前，凭已保存的原 binding 与 generation 恢复 Broker
|
||||
owner。存活的 Harness attachment 不能据此推断重启的 Broker 仍持有原 owner。
|
||||
原身份核验失败时保留可重试的 child intent，不进行派发；派发前再次核验当前权限。
|
||||
已有 unknown 结果继续阻塞，不重放副作用。
|
||||
|
||||
每次新副作用派发前核对当前 ACL、挂载和身份。撤销后仍可查询、结算已派发工作；
|
||||
未派发 Hook 保持 recovery_blocked，直至恢复权限。unknown 保留所有权，不能
|
||||
变成取消或完成证明。沿用 H2 可能无限期阻塞的限制，独立由 #13133 跟踪。
|
||||
|
||||
plan 和子执行派发前均预检权限，并在 journal 提交事务中重新检查。明确的事务
|
||||
授权拒绝不消耗 journal sequence，writer authority 可继续重试。应答丢失，或
|
||||
不确定请求后再收到拒绝，仍保留写入失败围栏。
|
||||
围栏核验每次 Hook revision，包括同一提交中的连续 revision：intent 转为可能
|
||||
已启动的结果须重新授权；not_started_proven 与原已派发工作结果的结算仍可提交。
|
||||
生命周期结算不得改变已记录的 Runtime 身份。
|
||||
|
||||
核对并保存 effects receipt 后，围栏单向升级到 DRAINING。不运行 Hook 地 detach，
|
||||
释放 owner、seal writer，再使用可靠 close 的 drain/stop 协议。接管者在 effects
|
||||
receipt 已保存时跳过 Hook，否则从相同 H2 occurrence 恢复进度。Harness 404、
|
||||
租约过期或 worker 消失均不能证明完成。
|
||||
明确且同 boot 的 detach 404 停止 SDK attachment 心跳；拒绝和不确定应答仍保留。
|
||||
receipt 恢复通过既有主键查找精确 occurrence 与 resource，保留原身份和字节校验，
|
||||
不扫描其他租户，也不遍历历史候选资源。
|
||||
|
||||
receipt 恢复可能完全跳过 Harness lifecycle 请求。因此，即使存活 attachment
|
||||
没有本地 lifecycle 状态，detach 也必须向 Session Store 核验 operation 权限。
|
||||
Store 授权检查当前 claim、原 writer、已保存的 effects 与 DRAINING 围栏；
|
||||
claim 被拒绝时保留 attachment 及其原权限。没有缓存 attachment 的接管者凭
|
||||
当前 claim 直接访问原 Session ID，不为恢复 client ID 加载 Runtime 或派发 Hook。
|
||||
缺少 authority 时仍要求原 client ID，提供错误 client ID 也会被拒绝。普通 detach
|
||||
仍须通过普通执行授权。清理授权仅接受原 writer 身份、token 与 generation，
|
||||
包括已过期或已封存的 writer，且不续租；Hook 派发仍要求 ACTIVE 且未过期的 writer。
|
||||
|
||||
通过该清理授权后，停止 activation 续租并封存原 writer，不追加 activation release
|
||||
记录。过期或已封存的 writer 无权追加；已开始的续租仍受 Store 围栏和 writer seal
|
||||
约束。历史 activation 可能仍显示 active，不能作为正常释放的证据。完成依据仍为
|
||||
永久围栏、effects receipt、writer 封存与原 Runtime 停机证明。普通与 legacy close
|
||||
仍在封存前记录 activation release。
|
||||
|
||||
最终完成要求有效 delivery claim、effects receipt、永久围栏、无有效 writer 或
|
||||
未结算执行，以及可核验的原停机证明。CLOSE 提交 CLOSED；DELETE 原子提交 O4
|
||||
退役、DELETED、operation 完成与终止事件。共享文件及其他会话的 holder 保留。
|
||||
|
||||
从未初始化的会话按 writer 顺序取得 tenant-retention、公开 Session 和 journal
|
||||
head 排他，证明无已完成 bootstrap、有效 writer、journal 或 Hook 派发记录，
|
||||
再保存 never-initialized 无 Hook 证据。有 header 时核对原 definition/catalog。
|
||||
两种情况都必须在永久围栏之后检查完整 Runtime binding 集合。
|
||||
|
||||
准入扫描并解析完整 journal，证明不存在已接纳但未结算的 Turn。成本随保留历史
|
||||
增长，事务期间持锁；尚未证明准入延迟存在固定上界。当前产品仅将 compaction
|
||||
watermark 初始化为零，没有推进它的生产路径。既有冷恢复也不支持非零 watermark；
|
||||
L3 返回 `workspace_lifecycle_journal_unverified`。未来 compaction 必须
|
||||
提供持久 idle-state 证据后，L3 才能接纳这些会话；本 PR 不实现压缩后的恢复。
|
||||
|
||||
## 4. 兼容与启用
|
||||
|
||||
在已有 V35 工具配置、V36–V39 journal/查询、V40 创建者、V41–V44 CSI/派发、V45 H3 任务 journal、V46 W2 目录切换及 V47 H5 channel route/delivery 以及 V48–V50 W1c storage 迁移/索引/身份迁移之后新增 V51 生命周期迁移，
|
||||
保留这些迁移和 V32。未合入的生命周期迁移从 V48 顺延为 V51，SQL 内容不变。
|
||||
主线 `/2` 搜索画像与 L3 共存，但生命周期准入仍仅接受 `hosted-workspace-files/1`。
|
||||
H5 channel 契约和持久化不启用 channel 执行或生命周期 effects。
|
||||
W2 目录/revision 操作字段与生命周期协议字段同时保留。
|
||||
W2 只读探测及结算独立于生命周期 Hook effects、drain 和 detach；Session 行上的准入围栏串行接纳两类操作。
|
||||
Runtime 解析在实际返回的 scope 中保留生命周期 claim。
|
||||
W2 准入与提交均先取得 placement 锁,在同租户/Harness 任何 Runtime Session 尚非确认 RELEASED 时拒绝 `session_context_busy`。
|
||||
Hook owner 可在 Turn 完成后继续保留;其不可变原上下文必须继续供 L3 effects 与释放使用。
|
||||
邻居 holder 不受影响,不替换 Runtime、不重装上下文;存在性读取增加扫描成本,不保证固定延迟。
|
||||
本次整合保留 H3 后台 Shell/Monitor 行为，不增加 CSI、Shell 或 MCP profile 生命周期支持。
|
||||
共享 Harness 仅在 detach 授权成功后关闭 Monitor 唤醒调度器；拒绝 detach 时，存活 Session 及其调度器保持可用。
|
||||
同步 closing/busy 围栏仍阻止授权等待期间的新唤醒。
|
||||
G3 代际恢复同样覆盖生命周期结算与 detach。发现新 Harness 代际的调用会清除旧客户端及 attachment，
|
||||
仍返回代际错误；既有交付重试以相同 operation 与当前 claim 重新协商。
|
||||
发现错误的调用内不再次派发，不启用普通恢复或 cancellation takeover，不替换原 Runtime，也不清除 unknown 结果。
|
||||
新 Harness boot 不是原 Runtime 的停机证据；effects receipt、writer seal 与原 handle 的停机证明仍为必需。
|
||||
升级前已接纳的操作沿用原协议与证据，不产生新 Hook 身份。
|
||||
生命周期迁移前读取 operation 时，缺少协议列表示历史 protocol-zero；已有 protocol-one 值原样保留。
|
||||
此投影兼容不授予执行权限，也不绕过 schema 升级。
|
||||
仍存活的 protocol-zero close attachment 仅在持久 close claim 有效时保留原 DELETE
|
||||
和原 Hook control 路径。普通执行继续被围栏阻止；此例外不能授权 L3 或 MCP 执行。
|
||||
旧 claim 的有效性统一比较数据库 epoch 毫秒，不受 JVM、JDBC 与数据库会话时区影响。
|
||||
先升级全部 Spring coordinator/Store，再升级 Hosted Harness；没有独立的 L3 启用开关。
|
||||
新 Harness 对接缺少普通执行授权路由的旧 Store 时，会拒绝全部托管 Turn，
|
||||
包括私有 Session；缺少授权路由不能当作执行许可。
|
||||
先升级 Store 的混合版本阶段，普通 Turn 仍可用，但 ACTIVE Workspace close 和
|
||||
delete 暂不可用，直到 Harness 宣告新生命周期能力。部署计划须包含这一暂时的
|
||||
close 不可用窗口。缺少新协议能力时拒绝准入，不回退到旧 DELETE。
|
||||
存在未完成 L3 操作时不回滚到旧 coordinator。
|
||||
L2 CLOSED/ARCHIVED 删除继续独立于 Harness 可用性。
|
||||
|
||||
读取生命周期或普通 writer 状态前，先执行已配置的 Store writer 凭证校验。
|
||||
生命周期加载原样传递签发的凭证及传输策略。cleanup 仍须使用原 writer 和当前
|
||||
claim；过期或 SEALED 的 cleanup grant 不绕过凭证校验。凭证策略变更可能使恢复
|
||||
持续阻塞，不能因此创建替代 writer 或 Runtime。
|
||||
|
||||
实施顺序：协议与围栏；限定权限和回执；close 语义；ACTIVE delete 准入与
|
||||
capability；恢复验证。同步 canonical OpenAPI 和相关中英文设计。
|
||||
|
||||
## 5. 验证与验收
|
||||
|
||||
覆盖两个 HTTP 入口、重放与 actor 隔离、全部活跃 Turn 状态、空会话、无 catalog
|
||||
与空 Hook plan。断言 close End=1/Delete=0、ACTIVE delete End=1/Delete=1、
|
||||
detach/L2 Hook=0。在 dispatch、结果提交、receipt、detach、stop 和墓碑边界
|
||||
注入应答丢失与崩溃。第二服务接管不能重复副作用，旧 claim 不能推进。
|
||||
|
||||
覆盖两个 Hook 之间撤销 ACL/挂载及恢复、unknown、身份无法核验、历史缺失停机
|
||||
证明和普通执行准入竞态。真实 MySQL 验证事务回滚与并发。真实 Linux 验证
|
||||
Harness/worker、host/boot/PID 身份、共享文件和邻居 holder 保留。
|
||||
模拟与物理验证分别报告。
|
||||
|
||||
执行 build、typecheck、bundle、定向 TS/Java 测试及 E2E 计划，再完成两轮干净
|
||||
自审和独立评审。结果和环境限制记录在
|
||||
`.qwen/e2e-tests/workspace-session-active-delete-l3.md`。
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
[English](workspace-session-archive-delete-l1-l2.md) | [简体中文](workspace-session-archive-delete-l1-l2.zh-CN.md)
|
||||
|
||||
This document preserves the original L1/L2 and reliable-close protocol. Newly admitted ACTIVE files close/delete use [L3](workspace-session-active-delete-l3.md): close runs SessionEnd only, delete runs SessionEnd followed by SessionDelete, and durable Hook evidence precedes permanent draining. L2 CLOSED/ARCHIVED deletion remains independent of Harness; operations admitted before upgrade resume under their saved protocol.
|
||||
|
||||
## 1. Status and baseline
|
||||
|
||||
Implemented and synchronized with main, 2026-10-02, on `codex/workspace-session-l1-l2`. This document
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
[English](workspace-session-archive-delete-l1-l2.md) | [简体中文](workspace-session-archive-delete-l1-l2.zh-CN.md)
|
||||
|
||||
本设计保留 L1/L2 与可靠 close 的原有协议。新接纳的 ACTIVE files 会话 close/delete 采用 [L3 设计](workspace-session-active-delete-l3.zh-CN.md)：close 仅运行 SessionEnd，delete 顺序运行 SessionEnd 与 SessionDelete，先提交 Hook 证据再永久停机。CLOSED/ARCHIVED 的 L2 删除仍不依赖 Harness；升级前接纳的 operation 按原版本恢复。
|
||||
|
||||
## 1. 状态与基线
|
||||
|
||||
实现完成并已同步 main，2026-10-02，分支 `codex/workspace-session-l1-l2`。本文覆盖
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
[English](workspace-session-reliable-close.md) | [简体中文](workspace-session-reliable-close.zh-CN.md)
|
||||
|
||||
This document preserves the original L1/L2 and reliable-close protocol. Newly admitted ACTIVE files close/delete use [L3](workspace-session-active-delete-l3.md): close runs SessionEnd only, delete runs SessionEnd followed by SessionDelete, and durable Hook evidence precedes permanent draining. L2 CLOSED/ARCHIVED deletion remains independent of Harness; operations admitted before upgrade resume under their saved protocol.
|
||||
|
||||
## Status and scope
|
||||
|
||||
Implemented and locally verified. This slice enables idle public `hosted-workspace-files/1` Sessions to close through existing public and WebShell lifecycle operations. Running, cancelling, accepted, or approval-waiting Turns remain a `409 turn_active`. Shell, MCP, archive/delete, and a new UI button are excluded. Close preserves transcripts, Artifacts, and shared Workspace files.
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
[English](workspace-session-reliable-close.md) | [简体中文](workspace-session-reliable-close.zh-CN.md)
|
||||
|
||||
本设计保留 L1/L2 与可靠 close 的原有协议。新接纳的 ACTIVE files 会话 close/delete 采用 [L3 设计](workspace-session-active-delete-l3.zh-CN.md)：close 仅运行 SessionEnd，delete 顺序运行 SessionEnd 与 SessionDelete，先提交 Hook 证据再永久停机。CLOSED/ARCHIVED 的 L2 删除仍不依赖 Harness；升级前接纳的 operation 按原版本恢复。
|
||||
|
||||
## 状态与范围
|
||||
|
||||
已实现并完成本地验证。本切片通过现有 public 和 WebShell lifecycle operation，为空闲的公开 `hosted-workspace-files/1` Session 开放 close。运行中、取消中、已接受或等待审批的 Turn 仍返回 `409 turn_active`。Shell、MCP、archive/delete 和新增 UI 按钮不在范围内。close 保留历史、Artifacts 和共享 Workspace 文件。
|
||||
|
||||
@@ -40,6 +40,7 @@ export async function startHostedSessionStore(sessionId: string) {
|
||||
const pending = new Set<Promise<void>>();
|
||||
let handle: ManagedSessionJournalHandle | undefined;
|
||||
let writerToken: string | undefined;
|
||||
let writerId: string | undefined;
|
||||
let writerGeneration = 0;
|
||||
let seals = 0;
|
||||
const scan = () =>
|
||||
@@ -77,6 +78,7 @@ export async function startHostedSessionStore(sessionId: string) {
|
||||
return;
|
||||
}
|
||||
writerToken = String(req.headers['x-qwen-managed-writer-token']);
|
||||
writerId = String(body.writerId);
|
||||
handle = await journal.open({ sessionKey });
|
||||
writerGeneration++;
|
||||
} else {
|
||||
@@ -98,6 +100,12 @@ export async function startHostedSessionStore(sessionId: string) {
|
||||
};
|
||||
if (route === '/writers:acquire' || route === '/writers:renew') {
|
||||
json({ ...head, leaseUntil: Date.now() + Number(body.leaseMillis) });
|
||||
} else if (route === '/execution:authorize') {
|
||||
assert.equal(req.method, 'POST');
|
||||
assert(handle);
|
||||
assert.equal(body.writerId, writerId);
|
||||
assert.equal(body.writerGeneration, writerGeneration);
|
||||
json({ authorized: true });
|
||||
} else if (route === '/restore') {
|
||||
json({
|
||||
...head,
|
||||
|
||||
@@ -43,6 +43,7 @@ describe('Hosted Harness private contract', () => {
|
||||
protocolVersions: { current: 1, supported: [1] },
|
||||
bootId: BOOT_ID,
|
||||
capabilityDigest: DIGEST,
|
||||
lifecycleProtocolVersion: 1,
|
||||
});
|
||||
expect(Object.isFrozen(contract)).toBe(true);
|
||||
expect(Object.isFrozen(contract.protocolVersions)).toBe(true);
|
||||
|
||||
@@ -44,6 +44,7 @@ export function createHostedHarnessContract(
|
||||
}),
|
||||
bootId: bootId.toLowerCase(),
|
||||
capabilityDigest,
|
||||
lifecycleProtocolVersion: 1,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -94,6 +94,7 @@ vi.mock(
|
||||
resourceStore,
|
||||
toolResultResources: resourceStore,
|
||||
assertWritable: state.assertWritable,
|
||||
authorizeOrdinary: async () => undefined,
|
||||
publication: {
|
||||
owner: async () => ({ writerId: BOOT_ID, writerGeneration: 1 }),
|
||||
request: (route: string, body: unknown, token?: string) =>
|
||||
|
||||
@@ -18,7 +18,15 @@ import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
import express from 'express';
|
||||
import supertest from 'supertest';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import {
|
||||
afterEach,
|
||||
beforeEach,
|
||||
describe,
|
||||
expect,
|
||||
it,
|
||||
vi,
|
||||
type MockInstance,
|
||||
} from 'vitest';
|
||||
import * as hostedHistory from './hosted-file-history.js';
|
||||
import type { HostedFileHistoryState } from './hosted-file-history-protocol.js';
|
||||
import {
|
||||
@@ -27,6 +35,7 @@ import {
|
||||
} from '@qwen-code/qwen-code-core/managed-runtime/local-jsonl-managed-session-journal-store.js';
|
||||
import { resetManagedRuntimeDispatchGatesForTest } from '@qwen-code/qwen-code-core/managed-runtime/managed-runtime-dispatch-gate.js';
|
||||
import { LocalManagedSessionAuthority } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-authority.js';
|
||||
import { ManagedSessionStoreHttpError } from '@qwen-code/qwen-code-core/managed-runtime/http-managed-session-store.js';
|
||||
import { openManagedSession } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js';
|
||||
import { LocalShellResultCapture } from '@qwen-code/qwen-code-core/managed-runtime/local-shell-result-capture.js';
|
||||
import { parseToolResultManifestBytes } from '@qwen-code/qwen-code-core/managed-runtime/managed-tool-result.js';
|
||||
@@ -82,6 +91,7 @@ import type {
|
||||
} from '@qwen-code/qwen-code-core/managed-runtime/managed-hook-protocol.js';
|
||||
import { HostedMonitorSession } from './hosted-monitor-session.js';
|
||||
import { HostedMonitorLoop } from './hosted-monitor-loop.js';
|
||||
import { HostedMonitorWakeScheduler } from './hosted-monitor-wake.js';
|
||||
import { HostedChildRunSession } from './hosted-child-run-session.js';
|
||||
import { LocalShellStreamCapture } from '@qwen-code/qwen-code-core/managed-runtime/local-shell-stream-capture.js';
|
||||
import { monitorWakeNeedsRecovery } from './hosted-monitor-wake-turn.js';
|
||||
@@ -133,6 +143,9 @@ const state = vi.hoisted(() => ({
|
||||
toolResults: null as DurableToolResultResourceStore | null,
|
||||
publicationRequest: vi.fn(),
|
||||
storeOptions: [] as Array<Record<string, unknown>>,
|
||||
setLifecycleAuthority: vi.fn(),
|
||||
authorizeLifecycle: vi.fn(async () => undefined),
|
||||
authorizeOrdinary: vi.fn(async (_kind?: 'legacy-close') => undefined),
|
||||
model: vi.fn(
|
||||
async (_input: {
|
||||
signal: AbortSignal;
|
||||
@@ -152,7 +165,10 @@ const state = vi.hoisted(() => ({
|
||||
|
||||
vi.mock(
|
||||
'@qwen-code/qwen-code-core/managed-runtime/http-managed-session-store.js',
|
||||
() => ({
|
||||
async (importOriginal) => ({
|
||||
...(await importOriginal<
|
||||
typeof import('@qwen-code/qwen-code-core/managed-runtime/http-managed-session-store.js')
|
||||
>()),
|
||||
HTTP_MANAGED_SESSION_STORE_CONTRACT: { maxInlineResourceBytes: 64 * 1024 },
|
||||
// The load route's catch classifies with instanceof against this class;
|
||||
// the wholesale module mock must still export it or the handler dies.
|
||||
@@ -192,6 +208,9 @@ vi.mock(
|
||||
resourceStore,
|
||||
toolResultResources: state.toolResults ?? resourceStore,
|
||||
assertWritable: state.assertWritable,
|
||||
setLifecycleAuthority: state.setLifecycleAuthority,
|
||||
authorizeOrdinary: state.authorizeOrdinary,
|
||||
authorizeLifecycle: state.authorizeLifecycle,
|
||||
publication: {
|
||||
owner: async () => ({ writerId: BOOT_ID, writerGeneration: 1 }),
|
||||
request: (route: string, body: unknown, token?: string) =>
|
||||
@@ -420,7 +439,7 @@ const hookPin = {
|
||||
catalogRevision: 1,
|
||||
definitionDigest: 'b'.repeat(64),
|
||||
};
|
||||
async function hookApp() {
|
||||
async function hookApp(asyncEnd = false) {
|
||||
const requests: ManagedHookControl[] = [];
|
||||
const catalog: ManagedHookCatalog = {
|
||||
...hookPin,
|
||||
@@ -432,7 +451,7 @@ async function hookApp() {
|
||||
hookId: eventName,
|
||||
eventName,
|
||||
sequential: false,
|
||||
async: false,
|
||||
async: asyncEnd && eventName === HookEventName.SessionEnd,
|
||||
failClosed: true,
|
||||
onceKey: null,
|
||||
config: { type: 'command' as const },
|
||||
@@ -448,6 +467,10 @@ async function hookApp() {
|
||||
};
|
||||
},
|
||||
);
|
||||
vi.spyOn(
|
||||
HostedWorkspaceBroker.prototype,
|
||||
'authorizeLifecycle',
|
||||
).mockResolvedValue();
|
||||
const release = vi
|
||||
.spyOn(HostedWorkspaceBroker.prototype, 'release')
|
||||
.mockResolvedValue();
|
||||
@@ -487,7 +510,15 @@ async function hookApp() {
|
||||
expect(created.status).toBe(200);
|
||||
const authorize = (request: supertest.Test) =>
|
||||
headers(request).set('X-Qwen-Client-Id', created.body.clientId);
|
||||
return { server, authorize, definition, catalog, requests, release };
|
||||
return {
|
||||
server,
|
||||
authorize,
|
||||
definition,
|
||||
catalog,
|
||||
requests,
|
||||
release,
|
||||
clientId: created.body.clientId,
|
||||
};
|
||||
}
|
||||
|
||||
describe('attributeShellReceipts', () => {
|
||||
@@ -615,6 +646,11 @@ describe('Hosted Harness no-tool session', () => {
|
||||
state.toolResults = null;
|
||||
state.assertWritable.mockReset();
|
||||
state.assertWritable.mockResolvedValue(undefined);
|
||||
state.authorizeLifecycle.mockReset();
|
||||
state.authorizeLifecycle.mockResolvedValue(undefined);
|
||||
state.setLifecycleAuthority.mockReset();
|
||||
state.authorizeOrdinary.mockReset();
|
||||
state.authorizeOrdinary.mockResolvedValue(undefined);
|
||||
state.model.mockReset();
|
||||
state.publicationRequest.mockReset();
|
||||
state.model.mockImplementation(async () => ({
|
||||
@@ -627,6 +663,716 @@ describe('Hosted Harness no-tool session', () => {
|
||||
await rm(state.root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it.each(['close', 'delete'] as const)(
|
||||
'settles %s lifecycle Hooks once and retains the attachment until detach',
|
||||
async (kind) => {
|
||||
const { server, authorize, requests, release } = await hookApp();
|
||||
const operation = {
|
||||
kind,
|
||||
sessionKey: {
|
||||
tenantId: 'tenant',
|
||||
workspaceId: 'workspace',
|
||||
sessionId: SESSION_ID,
|
||||
},
|
||||
authority: { operationId: 'op-l3', claimGeneration: 1 },
|
||||
};
|
||||
const first = await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/lifecycle`),
|
||||
).send(operation);
|
||||
expect(first.status).toBe(200);
|
||||
expect(
|
||||
first.body.effects.map((effect: { event: string }) => effect.event),
|
||||
).toEqual(
|
||||
kind === 'close' ? ['SessionEnd'] : ['SessionEnd', 'SessionDelete'],
|
||||
);
|
||||
expect(release).not.toHaveBeenCalled();
|
||||
const retry = await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/lifecycle`),
|
||||
).send({
|
||||
...operation,
|
||||
authority: { ...operation.authority, claimGeneration: 2 },
|
||||
});
|
||||
expect(retry.status).toBe(200);
|
||||
expect(retry.body).toEqual(first.body);
|
||||
expect(
|
||||
requests
|
||||
.filter((request) => request.kind === 'hook-execute')
|
||||
.map((request) => request.input.hook_event_name),
|
||||
).toEqual(
|
||||
kind === 'close' ? ['SessionEnd'] : ['SessionEnd', 'SessionDelete'],
|
||||
);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/prompt`))
|
||||
.send({})
|
||||
.expect(409);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/cancel`))
|
||||
.send({})
|
||||
.expect(409);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({ authority: { operationId: 'op-l3', claimGeneration: 2 } })
|
||||
.expect(204);
|
||||
expect(release).toHaveBeenCalled();
|
||||
expect(state.model).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it.each([true, false])(
|
||||
'authorizes detach after the coordinator recovers effects without a Harness lifecycle request (clientId=%s)',
|
||||
async (withClientId) => {
|
||||
const { server, authorize, requests } = await hookApp();
|
||||
const transcriptPath = path.join(state.root, `${SESSION_ID}.jsonl`);
|
||||
const transcript = await readFile(transcriptPath, 'utf8');
|
||||
const releaseActivation = vi
|
||||
.spyOn(LocalManagedSessionAuthority.prototype, 'releaseActivation')
|
||||
.mockRejectedValue(new Error('journal appends are fenced'));
|
||||
state.authorizeOrdinary.mockRejectedValue(
|
||||
new ManagedSessionStoreHttpError(
|
||||
409,
|
||||
'managed_session_lifecycle_active',
|
||||
'DRAINING',
|
||||
),
|
||||
);
|
||||
const authority = {
|
||||
operationId: 'recovered-effects',
|
||||
claimGeneration: 2,
|
||||
};
|
||||
await headers(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({})
|
||||
.expect(404);
|
||||
await headers(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.set('X-Qwen-Client-Id', 'wrong-client')
|
||||
.send({ authority })
|
||||
.expect(404);
|
||||
expect(state.authorizeLifecycle).not.toHaveBeenCalled();
|
||||
await (withClientId ? authorize : headers)(
|
||||
supertest(server).post(`/session/${SESSION_ID}/detach`),
|
||||
)
|
||||
.send({ authority })
|
||||
.expect(204);
|
||||
expect(state.setLifecycleAuthority).toHaveBeenLastCalledWith(authority);
|
||||
expect(state.authorizeLifecycle).toHaveBeenCalledExactlyOnceWith();
|
||||
expect(state.authorizeOrdinary).not.toHaveBeenCalled();
|
||||
expect(releaseActivation).not.toHaveBeenCalled();
|
||||
expect(await readFile(transcriptPath, 'utf8')).toBe(transcript);
|
||||
expect(requests).toEqual([]);
|
||||
expect(state.model).not.toHaveBeenCalled();
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/prompt`))
|
||||
.send({})
|
||||
.expect(404);
|
||||
},
|
||||
);
|
||||
|
||||
it.each(['close', 'delete'] as const)(
|
||||
'adopts the original idle attachment for %s without reopening or replaying Hooks',
|
||||
async (kind) => {
|
||||
const { server, authorize, definition, requests, clientId } =
|
||||
await hookApp();
|
||||
const descriptorCount = state.storeOptions.length;
|
||||
const transcriptPath = path.join(state.root, `${SESSION_ID}.jsonl`);
|
||||
const transcript = await readFile(transcriptPath, 'utf8');
|
||||
state.assertWritable.mockClear();
|
||||
const authority = { operationId: 'adopt-original', claimGeneration: 1 };
|
||||
const load = () =>
|
||||
headers(supertest(server).post(`/session/${SESSION_ID}/load`)).send({
|
||||
...definition,
|
||||
lifecycleAuthority: authority,
|
||||
});
|
||||
const adopted = await load();
|
||||
expect(adopted.status).toBe(200);
|
||||
expect(adopted.body.clientId).toBe(clientId);
|
||||
expect(state.storeOptions).toHaveLength(descriptorCount);
|
||||
expect(state.assertWritable).toHaveBeenCalledTimes(1);
|
||||
expect(await readFile(transcriptPath, 'utf8')).toBe(transcript);
|
||||
expect(requests).toEqual([]);
|
||||
const operation = {
|
||||
kind,
|
||||
sessionKey: {
|
||||
tenantId: 'tenant',
|
||||
workspaceId: 'workspace',
|
||||
sessionId: SESSION_ID,
|
||||
},
|
||||
authority,
|
||||
};
|
||||
await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/lifecycle`),
|
||||
)
|
||||
.send(operation)
|
||||
.expect(200);
|
||||
expect((await load()).body.clientId).toBe(adopted.body.clientId);
|
||||
await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/lifecycle`),
|
||||
)
|
||||
.send(operation)
|
||||
.expect(200);
|
||||
expect(
|
||||
requests
|
||||
.filter((request) => request.kind === 'hook-execute')
|
||||
.map((request) => request.input.hook_event_name),
|
||||
).toEqual(
|
||||
kind === 'close' ? ['SessionEnd'] : ['SessionEnd', 'SessionDelete'],
|
||||
);
|
||||
for (const route of ['prompt', 'cancel']) {
|
||||
await headers(
|
||||
supertest(server).post(`/session/${SESSION_ID}/${route}`),
|
||||
).expect(404);
|
||||
await headers(supertest(server).post(`/session/${SESSION_ID}/${route}`))
|
||||
.set('X-Qwen-Client-Id', randomUUID())
|
||||
.expect(404);
|
||||
await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/${route}`),
|
||||
).expect(409);
|
||||
}
|
||||
await headers(supertest(server).delete(`/session/${SESSION_ID}`))
|
||||
.set('X-Qwen-Client-Id', randomUUID())
|
||||
.expect(404);
|
||||
await headers(supertest(server).delete(`/session/${SESSION_ID}`)).expect(
|
||||
409,
|
||||
);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({ authority })
|
||||
.expect(204);
|
||||
expect(state.model).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
{ tenantId: 'foreign' },
|
||||
{ workspaceId: 'foreign' },
|
||||
{ writerToken: 'f'.repeat(40) },
|
||||
{ baseUrl: 'http://foreign-store.test' },
|
||||
{ writerId: randomUUID() },
|
||||
])(
|
||||
'refuses lifecycle adoption with a different original grant: %s',
|
||||
async (changed) => {
|
||||
const { server, authorize, definition, requests } = await hookApp();
|
||||
state.assertWritable.mockClear();
|
||||
await headers(supertest(server).post(`/session/${SESSION_ID}/load`))
|
||||
.send({
|
||||
...definition,
|
||||
managedSessionStore: { ...store(), ...changed },
|
||||
lifecycleAuthority: { operationId: 'adopt', claimGeneration: 1 },
|
||||
})
|
||||
.expect(409);
|
||||
expect(state.assertWritable).not.toHaveBeenCalled();
|
||||
expect(state.setLifecycleAuthority).not.toHaveBeenCalled();
|
||||
expect(requests).toEqual([]);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({})
|
||||
.expect(204);
|
||||
},
|
||||
);
|
||||
|
||||
it('restores original authority after refused adoption and guards concurrent claims', async () => {
|
||||
const { server, authorize, definition, requests } = await hookApp();
|
||||
const load = (operationId: string) =>
|
||||
headers(supertest(server).post(`/session/${SESSION_ID}/load`)).send({
|
||||
...definition,
|
||||
lifecycleAuthority: { operationId, claimGeneration: 1 },
|
||||
});
|
||||
state.assertWritable.mockRejectedValueOnce(new Error('claim refused'));
|
||||
expect((await load('stale')).status).toBe(503);
|
||||
expect(state.setLifecycleAuthority).toHaveBeenLastCalledWith(undefined);
|
||||
let release!: () => void;
|
||||
state.assertWritable.mockImplementationOnce(
|
||||
() =>
|
||||
new Promise<undefined>((resolve) => {
|
||||
release = () => resolve(undefined);
|
||||
}),
|
||||
);
|
||||
const pending = load('current').then((response) => response);
|
||||
await vi.waitFor(() => expect(release).toBeDefined());
|
||||
try {
|
||||
expect((await load('competing')).status).toBe(409);
|
||||
expect(requests).toEqual([]);
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
expect((await pending).status).toBe(200);
|
||||
expect((await load('other-operation')).status).toBe(409);
|
||||
await headers(supertest(server).post(`/session/${SESSION_ID}/load`))
|
||||
.send(definition)
|
||||
.expect(409);
|
||||
await headers(supertest(server).post('/session'))
|
||||
.send({
|
||||
...definition,
|
||||
lifecycleAuthority: { operationId: 'create', claimGeneration: 1 },
|
||||
})
|
||||
.expect(400);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({ authority: { operationId: 'current', claimGeneration: 1 } })
|
||||
.expect(204);
|
||||
});
|
||||
|
||||
it.each(['prompt', 'title', 'hooks/operations'])(
|
||||
'refuses a late ordinary %s preflight after lifecycle adoption',
|
||||
async (route) => {
|
||||
const { server, authorize, definition, requests } = await hookApp();
|
||||
const transcriptPath = path.join(state.root, `${SESSION_ID}.jsonl`);
|
||||
const before = await readFile(transcriptPath, 'utf8');
|
||||
let release!: () => void;
|
||||
state.authorizeOrdinary.mockImplementationOnce(
|
||||
() =>
|
||||
new Promise<undefined>((resolve) => {
|
||||
release = () => resolve(undefined);
|
||||
}),
|
||||
);
|
||||
const prompt = [{ type: 'text', text: 'late request' }];
|
||||
const pending = authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/${route}`),
|
||||
)
|
||||
.send(
|
||||
route === 'title'
|
||||
? { title: 'late title' }
|
||||
: route === 'hooks/operations'
|
||||
? {
|
||||
operationId: 'ea17a7bc-b8d6-4f42-9e82-3859309fe92a',
|
||||
event: 'Notification',
|
||||
input: { message: 'late', notification_type: 'test' },
|
||||
}
|
||||
: {
|
||||
prompt,
|
||||
promptId: PROMPT_ID,
|
||||
payloadDigest: `sha256:${createHash('sha256').update(JSON.stringify(prompt)).digest('hex')}`,
|
||||
},
|
||||
)
|
||||
.then((response) => response);
|
||||
await vi.waitFor(() => expect(release).toBeDefined());
|
||||
try {
|
||||
await headers(supertest(server).post(`/session/${SESSION_ID}/load`))
|
||||
.send({
|
||||
...definition,
|
||||
lifecycleAuthority: { operationId: 'adopt', claimGeneration: 1 },
|
||||
})
|
||||
.expect(200);
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
const refused = await pending;
|
||||
expect(refused.status).toBe(409);
|
||||
expect(refused.body.code).toBe('hosted_lifecycle_operation_active');
|
||||
expect(requests).toEqual([]);
|
||||
expect(state.model).not.toHaveBeenCalled();
|
||||
expect(await readFile(transcriptPath, 'utf8')).toBe(before);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({ authority: { operationId: 'adopt', claimGeneration: 1 } })
|
||||
.expect(204);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([true, false])(
|
||||
'retains the attachment and restores authority when a recovered detach claim is rejected (clientId=%s)',
|
||||
async (withClientId) => {
|
||||
const { server, authorize, requests } = await hookApp();
|
||||
state.authorizeOrdinary.mockRejectedValue(
|
||||
new ManagedSessionStoreHttpError(
|
||||
409,
|
||||
'managed_session_lifecycle_active',
|
||||
'DRAINING',
|
||||
),
|
||||
);
|
||||
state.authorizeLifecycle.mockRejectedValueOnce(new Error('stale claim'));
|
||||
await (withClientId ? authorize : headers)(
|
||||
supertest(server).post(`/session/${SESSION_ID}/detach`),
|
||||
)
|
||||
.send({ authority: { operationId: 'stale', claimGeneration: 1 } })
|
||||
.expect(503);
|
||||
expect(state.setLifecycleAuthority).toHaveBeenLastCalledWith(undefined);
|
||||
expect(requests).toEqual([]);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({})
|
||||
.expect(409);
|
||||
await (withClientId ? authorize : headers)(
|
||||
supertest(server).post(`/session/${SESSION_ID}/detach`),
|
||||
)
|
||||
.send({ authority: { operationId: 'valid', claimGeneration: 2 } })
|
||||
.expect(204);
|
||||
expect(state.authorizeLifecycle).toHaveBeenCalledTimes(2);
|
||||
expect(requests).toEqual([]);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([true, false])(
|
||||
'rejects malformed detach authority without bypassing a persistent fence (clientId=%s)',
|
||||
async (withClientId) => {
|
||||
const { server, authorize, requests } = await hookApp();
|
||||
state.authorizeOrdinary.mockRejectedValue(
|
||||
new ManagedSessionStoreHttpError(
|
||||
409,
|
||||
'managed_session_lifecycle_active',
|
||||
'DRAINING',
|
||||
),
|
||||
);
|
||||
await (withClientId ? authorize : headers)(
|
||||
supertest(server).post(`/session/${SESSION_ID}/detach`),
|
||||
)
|
||||
.send({ authority: { operationId: 'malformed', claimGeneration: 0 } })
|
||||
.expect(400);
|
||||
expect(state.authorizeLifecycle).not.toHaveBeenCalled();
|
||||
expect(state.setLifecycleAuthority).not.toHaveBeenCalled();
|
||||
await (withClientId ? authorize : headers)(
|
||||
supertest(server).post(`/session/${SESSION_ID}/detach`),
|
||||
)
|
||||
.send({ authority: { operationId: 'valid', claimGeneration: 2 } })
|
||||
.expect(204);
|
||||
expect(requests).toEqual([]);
|
||||
},
|
||||
);
|
||||
|
||||
it.each(['title', 'prompt'] as const)(
|
||||
'checks attachment identity before ordinary Store authorization for %s',
|
||||
async (route) => {
|
||||
const { server, authorize } = await hookApp();
|
||||
state.authorizeOrdinary.mockClear();
|
||||
for (const clientId of [undefined, 'wrong-client']) {
|
||||
const request = headers(
|
||||
supertest(server).post(`/session/${SESSION_ID}/${route}`),
|
||||
);
|
||||
if (clientId) request.set('X-Qwen-Client-Id', clientId);
|
||||
const rejected = await request.send({ title: 'renamed' });
|
||||
expect(rejected.status).toBe(404);
|
||||
expect(rejected.body.code).toBe('hosted_session_not_found');
|
||||
expect(state.authorizeOrdinary).not.toHaveBeenCalled();
|
||||
}
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/title`))
|
||||
.send({ title: 'renamed' })
|
||||
.expect(200);
|
||||
expect(state.authorizeOrdinary).toHaveBeenCalledOnce();
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({})
|
||||
.expect(204);
|
||||
},
|
||||
);
|
||||
|
||||
it('uses the legacy close admission for an attached protocol zero Session with Hooks', async () => {
|
||||
const { server, authorize, requests } = await hookApp();
|
||||
state.authorizeOrdinary.mockImplementation(async (kind) => {
|
||||
if (kind !== 'legacy-close')
|
||||
throw new ManagedSessionStoreHttpError(
|
||||
409,
|
||||
'managed_session_lifecycle_active',
|
||||
'Ordinary admission closed',
|
||||
);
|
||||
});
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/prompt`))
|
||||
.send({})
|
||||
.expect(409);
|
||||
await authorize(supertest(server).delete(`/session/${SESSION_ID}`)).expect(
|
||||
204,
|
||||
);
|
||||
expect(state.authorizeOrdinary).toHaveBeenLastCalledWith('legacy-close');
|
||||
expect(
|
||||
requests
|
||||
.filter((request) => request.kind === 'hook-execute')
|
||||
.map((request) => request.input.hook_event_name),
|
||||
).toEqual(['SessionEnd', 'SessionDelete']);
|
||||
});
|
||||
|
||||
it.each([
|
||||
['prompt', new TypeError('network timeout')],
|
||||
['detach', new TypeError('network timeout')],
|
||||
[
|
||||
'prompt',
|
||||
new ManagedSessionStoreHttpError(
|
||||
503,
|
||||
'internal_error',
|
||||
'Store unavailable',
|
||||
),
|
||||
],
|
||||
[
|
||||
'detach',
|
||||
new ManagedSessionStoreHttpError(
|
||||
503,
|
||||
'internal_error',
|
||||
'Store unavailable',
|
||||
),
|
||||
],
|
||||
[
|
||||
'prompt',
|
||||
new ManagedSessionStoreHttpError(
|
||||
409,
|
||||
'managed_session_writer_conflict',
|
||||
'Writer fenced',
|
||||
),
|
||||
],
|
||||
[
|
||||
'detach',
|
||||
new ManagedSessionStoreHttpError(
|
||||
409,
|
||||
'managed_session_writer_conflict',
|
||||
'Writer fenced',
|
||||
),
|
||||
],
|
||||
])(
|
||||
'reports an ordinary %s authorization failure as unavailable: %s',
|
||||
async (route, cause) => {
|
||||
const { server, authorize, requests } = await hookApp();
|
||||
state.authorizeOrdinary.mockRejectedValueOnce(cause);
|
||||
const rejected = await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/${route}`),
|
||||
).send({});
|
||||
expect(rejected.status).toBe(503);
|
||||
expect(rejected.body.code).toBe(
|
||||
'hosted_execution_authorization_unavailable',
|
||||
);
|
||||
expect(state.model).not.toHaveBeenCalled();
|
||||
expect(requests).toEqual([]);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({})
|
||||
.expect(204);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
new TypeError('network timeout'),
|
||||
new ManagedSessionStoreHttpError(
|
||||
503,
|
||||
'internal_error',
|
||||
'Store unavailable',
|
||||
),
|
||||
new ManagedSessionStoreHttpError(
|
||||
409,
|
||||
'managed_session_writer_conflict',
|
||||
'Writer fenced',
|
||||
),
|
||||
])(
|
||||
'cancels an active Turn despite ordinary authorization failure: %s',
|
||||
async (cause) => {
|
||||
const { server, authorize } = await hookApp();
|
||||
let signal!: AbortSignal;
|
||||
let finish!: () => void;
|
||||
state.model.mockImplementationOnce(
|
||||
(input) =>
|
||||
new Promise((resolve) => {
|
||||
signal = input.signal;
|
||||
finish = () => resolve({ text: '', model: 'test-model' });
|
||||
signal.addEventListener('abort', finish, { once: true });
|
||||
}),
|
||||
);
|
||||
const prompt = [{ type: 'text', text: 'wait for cancellation' }];
|
||||
const payloadDigest = `sha256:${createHash('sha256').update(JSON.stringify(prompt)).digest('hex')}`;
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/prompt`))
|
||||
.send({ prompt, promptId: PROMPT_ID, payloadDigest })
|
||||
.expect(202);
|
||||
await vi.waitFor(() => expect(state.model).toHaveBeenCalledTimes(1));
|
||||
state.authorizeOrdinary.mockClear();
|
||||
state.authorizeOrdinary.mockRejectedValue(cause);
|
||||
try {
|
||||
await headers(
|
||||
supertest(server).post(`/session/${SESSION_ID}/cancel`),
|
||||
).expect(404);
|
||||
expect(signal.aborted).toBe(false);
|
||||
await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/cancel`),
|
||||
).expect(204);
|
||||
expect(signal.aborted).toBe(true);
|
||||
expect(state.authorizeOrdinary).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
finish();
|
||||
}
|
||||
await vi.waitFor(async () => {
|
||||
const status = await authorize(
|
||||
supertest(server).get(`/session/${SESSION_ID}/status`),
|
||||
);
|
||||
expect(status.body.hasActivePrompt).toBe(false);
|
||||
});
|
||||
state.authorizeOrdinary.mockResolvedValue(undefined);
|
||||
await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/detach`),
|
||||
).expect(204);
|
||||
},
|
||||
);
|
||||
|
||||
it('resumes the undispatched Delete Hook after revocation without cancelling or repeating End', async () => {
|
||||
const { server, authorize, requests } = await hookApp();
|
||||
let permitted = true;
|
||||
const control = vi.mocked(HostedWorkspaceBroker.prototype.hookControl);
|
||||
const original = control.getMockImplementation()!;
|
||||
control.mockImplementation(async (request) => {
|
||||
const result = await original(request);
|
||||
if (request.kind === 'hook-execute') permitted = false;
|
||||
return result;
|
||||
});
|
||||
vi.mocked(
|
||||
HostedWorkspaceBroker.prototype.authorizeLifecycle,
|
||||
).mockImplementation(async () => {
|
||||
if (!permitted) throw new Error('ACL revoked');
|
||||
});
|
||||
const operation = {
|
||||
kind: 'delete',
|
||||
sessionKey: {
|
||||
tenantId: 'tenant',
|
||||
workspaceId: 'workspace',
|
||||
sessionId: SESSION_ID,
|
||||
},
|
||||
authority: { operationId: 'op-revoked', claimGeneration: 1 },
|
||||
};
|
||||
const first = await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/lifecycle`),
|
||||
).send(operation);
|
||||
expect(first.status).toBe(503);
|
||||
expect(
|
||||
requests
|
||||
.filter((request) => request.kind === 'hook-execute')
|
||||
.map((request) => request.input.hook_event_name),
|
||||
).toEqual(['SessionEnd']);
|
||||
vi.mocked(
|
||||
HostedWorkspaceBroker.prototype.authorizeLifecycle,
|
||||
).mockResolvedValue();
|
||||
const retry = await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/lifecycle`),
|
||||
).send({
|
||||
...operation,
|
||||
authority: { ...operation.authority, claimGeneration: 2 },
|
||||
});
|
||||
expect(retry.status).toBe(200);
|
||||
expect(
|
||||
requests
|
||||
.filter((request) => request.kind === 'hook-execute')
|
||||
.map((request) => request.input.hook_event_name),
|
||||
).toEqual(['SessionEnd', 'SessionDelete']);
|
||||
expect(
|
||||
requests.filter((request) => request.kind === 'hook-cancel'),
|
||||
).toHaveLength(0);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({ authority: { operationId: 'op-revoked', claimGeneration: 2 } })
|
||||
.expect(204);
|
||||
});
|
||||
|
||||
it('waits for an async End child to settle before dispatching Delete', async () => {
|
||||
const { server, authorize, requests } = await hookApp(true);
|
||||
const control = vi.mocked(HostedWorkspaceBroker.prototype.hookControl);
|
||||
const original = control.getMockImplementation()!;
|
||||
let endSettled = false;
|
||||
control.mockImplementation(async (request) => {
|
||||
if (
|
||||
!endSettled &&
|
||||
(request.kind === 'hook-status' ||
|
||||
(request.kind === 'hook-execute' &&
|
||||
request.input.hook_event_name === HookEventName.SessionEnd))
|
||||
) {
|
||||
requests.push(request);
|
||||
return { operationId: request.operationId, state: 'running' };
|
||||
}
|
||||
return original(request);
|
||||
});
|
||||
const operation = {
|
||||
kind: 'delete',
|
||||
sessionKey: {
|
||||
tenantId: 'tenant',
|
||||
workspaceId: 'workspace',
|
||||
sessionId: SESSION_ID,
|
||||
},
|
||||
authority: { operationId: 'async-end', claimGeneration: 1 },
|
||||
};
|
||||
try {
|
||||
await authorize(
|
||||
supertest(server).post(`/session/${SESSION_ID}/lifecycle`),
|
||||
)
|
||||
.send(operation)
|
||||
.expect(503);
|
||||
expect(
|
||||
requests
|
||||
.filter((request) => request.kind === 'hook-execute')
|
||||
.map((request) => request.input.hook_event_name),
|
||||
).toEqual(['SessionEnd']);
|
||||
} finally {
|
||||
endSettled = true;
|
||||
}
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/lifecycle`))
|
||||
.send({
|
||||
...operation,
|
||||
authority: { operationId: 'async-end', claimGeneration: 2 },
|
||||
})
|
||||
.expect(200);
|
||||
expect(
|
||||
requests
|
||||
.filter((request) => request.kind === 'hook-execute')
|
||||
.map((request) => request.input.hook_event_name),
|
||||
).toEqual(['SessionEnd', 'SessionDelete']);
|
||||
expect(
|
||||
requests.filter((request) => request.kind === 'hook-cancel'),
|
||||
).toEqual([]);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({ authority: { operationId: 'async-end', claimGeneration: 2 } })
|
||||
.expect(204);
|
||||
});
|
||||
|
||||
it('allows the valid operation after an invalid lifecycle claim is rejected', async () => {
|
||||
const { server, authorize, requests } = await hookApp();
|
||||
state.authorizeLifecycle.mockRejectedValueOnce(new Error('stale claim'));
|
||||
const operation = {
|
||||
kind: 'close',
|
||||
sessionKey: {
|
||||
tenantId: 'tenant',
|
||||
workspaceId: 'workspace',
|
||||
sessionId: SESSION_ID,
|
||||
},
|
||||
authority: { operationId: 'invalid', claimGeneration: 1 },
|
||||
};
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/lifecycle`))
|
||||
.send(operation)
|
||||
.expect(503);
|
||||
expect(requests).toHaveLength(0);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/lifecycle`))
|
||||
.send({
|
||||
...operation,
|
||||
kind: 'delete',
|
||||
authority: { operationId: 'valid', claimGeneration: 2 },
|
||||
})
|
||||
.expect(200);
|
||||
expect(
|
||||
requests
|
||||
.filter((request) => request.kind === 'hook-execute')
|
||||
.map((request) => request.input.hook_event_name),
|
||||
).toEqual(['SessionEnd', 'SessionDelete']);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({ authority: { operationId: 'valid', claimGeneration: 2 } })
|
||||
.expect(204);
|
||||
});
|
||||
|
||||
it('keeps an unknown End blocked without cancelling or redispatching it on takeover', async () => {
|
||||
const { server, authorize, requests } = await hookApp();
|
||||
const control = vi.mocked(HostedWorkspaceBroker.prototype.hookControl);
|
||||
const original = control.getMockImplementation()!;
|
||||
control.mockImplementation(async (request) => {
|
||||
if (request.kind === 'hook-catalog') return original(request);
|
||||
requests.push(request);
|
||||
return { operationId: request.operationId, state: 'outcome_unknown' };
|
||||
});
|
||||
const operation = {
|
||||
kind: 'delete',
|
||||
sessionKey: {
|
||||
tenantId: 'tenant',
|
||||
workspaceId: 'workspace',
|
||||
sessionId: SESSION_ID,
|
||||
},
|
||||
authority: { operationId: 'unknown-delete', claimGeneration: 1 },
|
||||
};
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/lifecycle`))
|
||||
.send(operation)
|
||||
.expect(503);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/lifecycle`))
|
||||
.send({
|
||||
...operation,
|
||||
authority: { ...operation.authority, claimGeneration: 2 },
|
||||
})
|
||||
.expect(503);
|
||||
expect(
|
||||
requests
|
||||
.filter((request) => request.kind === 'hook-execute')
|
||||
.map((request) => request.input.hook_event_name),
|
||||
).toEqual(['SessionEnd']);
|
||||
expect(
|
||||
requests.filter((request) => request.kind === 'hook-cancel'),
|
||||
).toHaveLength(0);
|
||||
expect(state.model).not.toHaveBeenCalled();
|
||||
control.mockImplementation(original);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({
|
||||
authority: { operationId: 'unknown-delete', claimGeneration: 2 },
|
||||
})
|
||||
.expect(204);
|
||||
});
|
||||
|
||||
it('refuses a new prompt while the journal close is still sealing', async () => {
|
||||
const server = await app();
|
||||
const created = await headers(supertest(server).post('/session')).send({
|
||||
@@ -846,6 +1592,52 @@ describe('Hosted Harness no-tool session', () => {
|
||||
);
|
||||
}
|
||||
|
||||
it.each([
|
||||
new TypeError('network timeout'),
|
||||
new ManagedSessionStoreHttpError(
|
||||
409,
|
||||
'managed_session_writer_conflict',
|
||||
'Writer fenced',
|
||||
),
|
||||
])(
|
||||
'keeps the Monitor wake scheduler after refused detach: %s',
|
||||
async (cause) => {
|
||||
mockBrokerBroker();
|
||||
const closeWake = vi.spyOn(HostedMonitorWakeScheduler.prototype, 'close');
|
||||
const server = await app(true);
|
||||
const created = await headers(supertest(server).post('/session'))
|
||||
.send({
|
||||
sessionId: SESSION_ID,
|
||||
sessionScope: 'thread',
|
||||
managedSessionStore: store(),
|
||||
toolProfile: 'hosted-workspace-shell/1',
|
||||
captureBytes: 1024 * 1024,
|
||||
})
|
||||
.expect(200);
|
||||
const authorize = (request: supertest.Test) =>
|
||||
headers(request).set(
|
||||
'X-Qwen-Client-Id',
|
||||
created.body.clientId as string,
|
||||
);
|
||||
state.authorizeOrdinary.mockRejectedValueOnce(cause);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({})
|
||||
.expect(503);
|
||||
expect(closeWake).not.toHaveBeenCalled();
|
||||
expect(HostedWorkspaceBroker.prototype.release).not.toHaveBeenCalled();
|
||||
await authorize(
|
||||
supertest(server).get(`/session/${SESSION_ID}/status`),
|
||||
).expect(200);
|
||||
await authorize(supertest(server).post(`/session/${SESSION_ID}/detach`))
|
||||
.send({})
|
||||
.expect(204);
|
||||
expect(closeWake).toHaveBeenCalledOnce();
|
||||
await authorize(
|
||||
supertest(server).get(`/session/${SESSION_ID}/status`),
|
||||
).expect(404);
|
||||
},
|
||||
);
|
||||
|
||||
it('settles a pending Monitor notification as session_closing when the Session closes', async () => {
|
||||
domainEnablement.monitorRun = true;
|
||||
const notificationTurnId = await prewriteMonitorSession();
|
||||
@@ -8729,7 +9521,7 @@ describe('Hosted Harness Runtime turn takeover', () => {
|
||||
|
||||
/** Captured per parked turn so tests can assert which loads acquire the
|
||||
* Runtime lease: a cold load never does, a passive takeover adopts it. */
|
||||
let acquireSpy: ReturnType<typeof vi.spyOn>;
|
||||
let acquireSpy: MockInstance<HostedWorkspaceBroker['acquire']>;
|
||||
|
||||
beforeEach(async () => {
|
||||
vi.spyOn(
|
||||
|
||||
@@ -33,6 +33,8 @@ import {
|
||||
HTTP_MANAGED_SESSION_STORE_CONTRACT,
|
||||
ManagedSessionStoreHttpError,
|
||||
type HttpToolPublicationOwner,
|
||||
type HttpManagedSessionStores,
|
||||
type ManagedSessionLifecycleAuthority,
|
||||
} from '@qwen-code/qwen-code-core/managed-runtime/http-managed-session-store.js';
|
||||
import {
|
||||
openManagedSession,
|
||||
@@ -105,6 +107,7 @@ import {
|
||||
type HostedRuntimeRecoveryReport,
|
||||
} from './hosted-runtime-recovery.js';
|
||||
import {
|
||||
HOSTED_WORKSPACE_FILE_PROFILE,
|
||||
HostedToolRecoveryRequiredError,
|
||||
HostedWorkspaceToolTurn,
|
||||
isHostedWorkspaceProfile,
|
||||
@@ -203,6 +206,10 @@ interface HostedSession {
|
||||
mcpRecovering?: boolean;
|
||||
approval?: HostedApprovalSettings;
|
||||
waiters: HostedApprovalWaiters;
|
||||
stores?: HttpManagedSessionStores;
|
||||
storeDescriptor: ReturnType<typeof parseBridgeManagedSessionStore>;
|
||||
lifecycle?: ManagedSessionLifecycleAuthority;
|
||||
lifecycleKind?: 'close' | 'delete';
|
||||
/** Fetched Workspace instructions; undefined until the first fetch. */
|
||||
workspaceContext?: string;
|
||||
monitorWake?: HostedMonitorWakeScheduler;
|
||||
@@ -258,6 +265,26 @@ function object(value: unknown): Record<string, unknown> | null {
|
||||
: null;
|
||||
}
|
||||
|
||||
function lifecycleAuthority(
|
||||
value: unknown,
|
||||
): ManagedSessionLifecycleAuthority | undefined {
|
||||
if (value === undefined) return undefined;
|
||||
const fields = object(value);
|
||||
if (
|
||||
!fields ||
|
||||
Object.keys(fields).sort().join(',') !== 'claimGeneration,operationId' ||
|
||||
typeof fields['operationId'] !== 'string' ||
|
||||
!/^[A-Za-z0-9._:-]{1,128}$/u.test(fields['operationId']) ||
|
||||
!Number.isSafeInteger(fields['claimGeneration']) ||
|
||||
(fields['claimGeneration'] as number) < 1
|
||||
)
|
||||
throw new Error('Invalid lifecycle authority.');
|
||||
return {
|
||||
operationId: fields['operationId'],
|
||||
claimGeneration: fields['claimGeneration'] as number,
|
||||
};
|
||||
}
|
||||
|
||||
function error(
|
||||
res: Response,
|
||||
status: number,
|
||||
@@ -273,6 +300,16 @@ function error(
|
||||
);
|
||||
}
|
||||
|
||||
function ordinaryAuthorizationError(res: Response, cause: unknown): void {
|
||||
if (
|
||||
cause instanceof ManagedSessionStoreHttpError &&
|
||||
cause.status === 409 &&
|
||||
cause.remoteCode === 'managed_session_lifecycle_active'
|
||||
)
|
||||
return error(res, 409, 'hosted_lifecycle_operation_active');
|
||||
error(res, 503, 'hosted_execution_authorization_unavailable');
|
||||
}
|
||||
|
||||
/** A takeover refusal that cannot change under retry: distinct from
|
||||
* `hosted_turn_recovery_required`, which stays retriable. */
|
||||
function recoveryDeclined(
|
||||
@@ -1863,6 +1900,15 @@ export function registerHostedHarnessSessionRoutes(
|
||||
error(res, 409, 'hosted_harness_generation_mismatch');
|
||||
return;
|
||||
}
|
||||
let lifecycle: ManagedSessionLifecycleAuthority | undefined;
|
||||
try {
|
||||
lifecycle = lifecycleAuthority(body?.['lifecycleAuthority']);
|
||||
if (lifecycle && create)
|
||||
throw new Error('Lifecycle load cannot create a Session.');
|
||||
} catch {
|
||||
error(res, 400, 'invalid_hosted_lifecycle_authority');
|
||||
return;
|
||||
}
|
||||
// Cold loads stay inert: only an explicit takeover request may touch
|
||||
// the Broker or settle anything.
|
||||
const takeoverFlags =
|
||||
@@ -1889,6 +1935,42 @@ export function registerHostedHarnessSessionRoutes(
|
||||
: {}),
|
||||
});
|
||||
const attached = sessions.get(sessionId);
|
||||
if (attached !== undefined && lifecycle) {
|
||||
if (
|
||||
!isDeepStrictEqual(store, attached.storeDescriptor) ||
|
||||
toolProfile !== HOSTED_WORKSPACE_FILE_PROFILE ||
|
||||
attached.toolProfile !== HOSTED_WORKSPACE_FILE_PROFILE ||
|
||||
(attached.lifecycle &&
|
||||
attached.lifecycle.operationId !== lifecycle.operationId)
|
||||
) {
|
||||
error(res, 409, 'hosted_session_already_attached');
|
||||
return;
|
||||
}
|
||||
if (
|
||||
attached.active ||
|
||||
attached.mcpBusy ||
|
||||
attached.mcpRecovering ||
|
||||
attached.hooksBusy
|
||||
) {
|
||||
error(res, 409, 'hosted_turn_active');
|
||||
return;
|
||||
}
|
||||
const previous = attached.lifecycle;
|
||||
attached.hooksBusy = true;
|
||||
attached.stores!.setLifecycleAuthority(lifecycle);
|
||||
try {
|
||||
await attached.stores!.assertWritable();
|
||||
attached.lifecycle = lifecycle;
|
||||
res.status(200).json(attachmentReply(attached));
|
||||
} catch (cause) {
|
||||
attached.stores!.setLifecycleAuthority(previous);
|
||||
debugLogger.warn('Hosted lifecycle attachment claim rejected:', cause);
|
||||
error(res, 503, 'managed_session_open_failed');
|
||||
} finally {
|
||||
attached.hooksBusy = false;
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (attached !== undefined && !create) {
|
||||
const passive = body?.['passiveManagedRuntimeRecovery'] === true;
|
||||
if (!attached.hooks && takeoverFlags) {
|
||||
@@ -2079,6 +2161,7 @@ export function registerHostedHarnessSessionRoutes(
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (lifecycle) stores.setLifecycleAuthority(lifecycle);
|
||||
opening.add(sessionId);
|
||||
let managed: ManagedSession | undefined;
|
||||
try {
|
||||
@@ -2188,6 +2271,9 @@ export function registerHostedHarnessSessionRoutes(
|
||||
admissions: new Map(),
|
||||
blocked: false,
|
||||
waiters: new HostedApprovalWaiters(),
|
||||
stores,
|
||||
storeDescriptor: store,
|
||||
lifecycle,
|
||||
...(toolProfile ? { toolProfile } : {}),
|
||||
...(isHostedWorkspaceShellProfile(toolProfile) &&
|
||||
captureBytes !== undefined
|
||||
@@ -2232,7 +2318,7 @@ export function registerHostedHarnessSessionRoutes(
|
||||
session.mcp = new HostedMcpSession(brokerOptions, managed, mcpServers);
|
||||
if (hookCatalog && brokerOptions)
|
||||
session.hooks = new HostedHookSession(
|
||||
brokerOptions,
|
||||
{ ...brokerOptions, lifecycleAuthority: () => session.lifecycle },
|
||||
managed,
|
||||
hookCatalog,
|
||||
session.mcp?.broker,
|
||||
@@ -2348,7 +2434,7 @@ export function registerHostedHarnessSessionRoutes(
|
||||
// Cold loads stay inert: only an explicit takeover request may touch
|
||||
// the Broker or settle anything. A bare load of a parked Session keeps
|
||||
// refusing with 409 so it never drives a Runtime by accident.
|
||||
const takeover = !session.hooks && takeoverFlags;
|
||||
const takeover = !lifecycle && !session.hooks && takeoverFlags;
|
||||
const fileHistory = await readHostedFileHistory(managed);
|
||||
if (
|
||||
fileHistory?.pendingUndo ||
|
||||
@@ -2595,6 +2681,7 @@ export function registerHostedHarnessSessionRoutes(
|
||||
}
|
||||
}
|
||||
} else if (
|
||||
!lifecycle &&
|
||||
restore.recoveryStatus === 'ok' &&
|
||||
(session.publication || session.hooks || fileHistory) &&
|
||||
brokerOptions
|
||||
@@ -2739,7 +2826,7 @@ export function registerHostedHarnessSessionRoutes(
|
||||
!inapplicableAnswer
|
||||
)
|
||||
session.blocked = true;
|
||||
await settleCancelledHookTurn(session);
|
||||
if (!session.lifecycle) await settleCancelledHookTurn(session);
|
||||
if (resume) {
|
||||
const abort = new AbortController();
|
||||
session.active = {
|
||||
@@ -2826,6 +2913,134 @@ export function registerHostedHarnessSessionRoutes(
|
||||
void open(req, res, false);
|
||||
});
|
||||
|
||||
app.use('/session/:id', async (req, res, next) => {
|
||||
const session = sessions.get(req.params['id']);
|
||||
if (
|
||||
session?.lifecycle &&
|
||||
req.method !== 'GET' &&
|
||||
!['/lifecycle', '/detach', '/heartbeat'].includes(req.path)
|
||||
) {
|
||||
const legacyClose = req.method === 'DELETE' && req.path === '/';
|
||||
if (!identity(req, sessions, legacyClose))
|
||||
return error(res, 404, 'hosted_session_not_found');
|
||||
return error(res, 409, 'hosted_lifecycle_operation_active');
|
||||
}
|
||||
if (
|
||||
session &&
|
||||
req.method !== 'GET' &&
|
||||
!['/lifecycle', '/detach', '/heartbeat', '/cancel'].includes(req.path) &&
|
||||
!session.lifecycle
|
||||
) {
|
||||
const legacyClose = req.method === 'DELETE' && req.path === '/';
|
||||
if (!identity(req, sessions, legacyClose))
|
||||
return error(res, 404, 'hosted_session_not_found');
|
||||
try {
|
||||
await session.stores!.authorizeOrdinary(
|
||||
legacyClose ? 'legacy-close' : undefined,
|
||||
);
|
||||
if (session.lifecycle)
|
||||
return error(res, 409, 'hosted_lifecycle_operation_active');
|
||||
} catch (cause) {
|
||||
return ordinaryAuthorizationError(res, cause);
|
||||
}
|
||||
}
|
||||
next();
|
||||
});
|
||||
|
||||
app.post('/session/:id/lifecycle', async (req, res) => {
|
||||
const session = identity(req, sessions);
|
||||
if (!session) return error(res, 404, 'hosted_session_not_found');
|
||||
const body = object(req.body);
|
||||
let authority: ManagedSessionLifecycleAuthority | undefined;
|
||||
try {
|
||||
authority = lifecycleAuthority(body?.['authority']);
|
||||
} catch {
|
||||
return error(res, 400, 'invalid_hosted_lifecycle_authority');
|
||||
}
|
||||
const kind = body?.['kind'];
|
||||
if (
|
||||
!authority ||
|
||||
(kind !== 'close' && kind !== 'delete') ||
|
||||
!isDeepStrictEqual(
|
||||
body?.['sessionKey'],
|
||||
session.managed.authority.sessionHeader.sessionKey,
|
||||
) ||
|
||||
session.toolProfile !== HOSTED_WORKSPACE_FILE_PROFILE
|
||||
)
|
||||
return error(res, 400, 'invalid_hosted_lifecycle_request');
|
||||
if (
|
||||
session.lifecycle &&
|
||||
(session.lifecycle.operationId !== authority.operationId ||
|
||||
(session.lifecycleKind && session.lifecycleKind !== kind))
|
||||
)
|
||||
return error(res, 409, 'hosted_lifecycle_operation_conflict');
|
||||
if (
|
||||
session.active ||
|
||||
session.mcpBusy ||
|
||||
session.mcpRecovering ||
|
||||
session.hooksBusy
|
||||
)
|
||||
return error(res, 409, 'hosted_turn_active');
|
||||
const previousAuthority = session.lifecycle;
|
||||
session.stores!.setLifecycleAuthority(authority);
|
||||
session.hooksBusy = true;
|
||||
let authorized = false;
|
||||
try {
|
||||
await session.stores!.authorizeLifecycle(kind);
|
||||
authorized = true;
|
||||
session.lifecycle = authority;
|
||||
session.lifecycleKind = kind;
|
||||
session.mcpClosing = true;
|
||||
const events =
|
||||
kind === 'close'
|
||||
? [HookEventName.SessionEnd]
|
||||
: [HookEventName.SessionEnd, HookEventName.SessionDelete];
|
||||
const occurrences = events.map((event) =>
|
||||
hostedHookOccurrenceId(event, authority.operationId),
|
||||
);
|
||||
await session.hooks?.drain(new Set(occurrences));
|
||||
const effects = [];
|
||||
if (session.hooks) {
|
||||
for (const event of events) {
|
||||
await runHostedLifecycleHook(
|
||||
session,
|
||||
event,
|
||||
authority.operationId,
|
||||
event === HookEventName.SessionEnd
|
||||
? { reason: 'other' }
|
||||
: { deleted_session_id: req.params['id'] },
|
||||
);
|
||||
await session.hooks.settleOccurrence(
|
||||
hostedHookOccurrenceId(event, authority.operationId),
|
||||
);
|
||||
const entry = session.managed.authority.extensionRecord(
|
||||
'hook_execution',
|
||||
hostedHookOccurrenceId(event, authority.operationId),
|
||||
);
|
||||
if (!entry || !parseHookExecution(entry.record).resultRef)
|
||||
throw new HostedHookRecoveryRequiredError();
|
||||
effects.push({ event, recordRef: entry.recordRef });
|
||||
}
|
||||
}
|
||||
res.json({
|
||||
protocolVersion: 1,
|
||||
sessionKey: session.managed.authority.sessionHeader.sessionKey,
|
||||
operationId: authority.operationId,
|
||||
kind,
|
||||
definitionRef: session.managed.authority.sessionHeader.definitionRef,
|
||||
effects,
|
||||
});
|
||||
} catch (cause) {
|
||||
writeStderrLineSafe(
|
||||
`qwen serve: Hosted lifecycle requires recovery: ${String(cause)}`,
|
||||
);
|
||||
error(res, 503, 'hosted_lifecycle_recovery_required');
|
||||
} finally {
|
||||
if (!authorized) session.stores!.setLifecycleAuthority(previousAuthority);
|
||||
session.hooksBusy = false;
|
||||
}
|
||||
});
|
||||
|
||||
app.post('/session/:id/prompt', (req, res) => {
|
||||
const session = identity(req, sessions);
|
||||
if (!session) return error(res, 404, 'hosted_session_not_found');
|
||||
@@ -3193,7 +3408,7 @@ export function registerHostedHarnessSessionRoutes(
|
||||
.then(async (execution) => {
|
||||
if (execution.hookId !== '__plan__')
|
||||
await session.hooks!.status(execution.occurrenceId);
|
||||
await settleCancelledHookTurn(session);
|
||||
if (!session.lifecycle) await settleCancelledHookTurn(session);
|
||||
res.json({
|
||||
operationId: execution.hookExecutionId,
|
||||
state: execution.run.state,
|
||||
@@ -4348,7 +4563,17 @@ export function registerHostedHarnessSessionRoutes(
|
||||
res: Response,
|
||||
allowMissingClientId = false,
|
||||
): Promise<void> => {
|
||||
const session = identity(req, sessions, allowMissingClientId);
|
||||
let authority: ManagedSessionLifecycleAuthority | undefined;
|
||||
try {
|
||||
authority = lifecycleAuthority(object(req.body)?.['authority']);
|
||||
} catch {
|
||||
return error(res, 400, 'invalid_hosted_lifecycle_authority');
|
||||
}
|
||||
const session = identity(
|
||||
req,
|
||||
sessions,
|
||||
allowMissingClientId || authority !== undefined,
|
||||
);
|
||||
if (!session) return error(res, 404, 'hosted_session_not_found');
|
||||
if (
|
||||
session.active ||
|
||||
@@ -4359,10 +4584,35 @@ export function registerHostedHarnessSessionRoutes(
|
||||
return error(res, 409, 'hosted_turn_active');
|
||||
session.mcpBusy = true;
|
||||
session.mcpClosing = true;
|
||||
// No wake turn may start once the Session is draining; pending
|
||||
// notifications still settle below before the log closes.
|
||||
session.monitorWake?.close();
|
||||
try {
|
||||
if (session.lifecycle || authority) {
|
||||
if (req.method === 'DELETE')
|
||||
return error(res, 409, 'hosted_lifecycle_operation_active');
|
||||
if (
|
||||
!authority ||
|
||||
(session.lifecycle &&
|
||||
authority.operationId !== session.lifecycle.operationId)
|
||||
)
|
||||
return error(res, 409, 'hosted_lifecycle_operation_conflict');
|
||||
const previousAuthority = session.lifecycle;
|
||||
session.stores!.setLifecycleAuthority(authority);
|
||||
try {
|
||||
await session.stores!.authorizeLifecycle();
|
||||
} catch (cause) {
|
||||
session.stores!.setLifecycleAuthority(previousAuthority);
|
||||
throw cause;
|
||||
}
|
||||
session.lifecycle = authority;
|
||||
} else if (req.method === 'POST') {
|
||||
try {
|
||||
await session.stores!.authorizeOrdinary();
|
||||
} catch (cause) {
|
||||
return ordinaryAuthorizationError(res, cause);
|
||||
}
|
||||
}
|
||||
// No wake turn may start once the authorized Session is draining;
|
||||
// a rejected close leaves its scheduler available for later wakes.
|
||||
session.monitorWake?.close();
|
||||
if (req.method === 'DELETE' && session.hooks) {
|
||||
session.hooksBusy = true;
|
||||
try {
|
||||
@@ -4414,7 +4664,9 @@ export function registerHostedHarnessSessionRoutes(
|
||||
sessionId: req.params['id'],
|
||||
cwd: session.cwd,
|
||||
});
|
||||
await session.managed.close();
|
||||
await session.managed.close(
|
||||
session.lifecycle ? { releaseActivation: false } : undefined,
|
||||
);
|
||||
for (const stop of session.streams) stop();
|
||||
sessions.delete(req.params['id']);
|
||||
res.sendStatus(204);
|
||||
@@ -4424,7 +4676,7 @@ export function registerHostedHarnessSessionRoutes(
|
||||
);
|
||||
error(res, 503, 'managed_session_close_failed');
|
||||
} finally {
|
||||
session.mcpClosing = false;
|
||||
session.mcpClosing = !!session.lifecycle;
|
||||
session.mcpBusy = false;
|
||||
}
|
||||
};
|
||||
|
||||
@@ -181,6 +181,198 @@ afterEach(async () => {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it.each([HookEventName.SessionEnd, HookEventName.SessionDelete])(
|
||||
'restores the original lifecycle Hook owner before dispatching %s after Broker restart',
|
||||
async (eventName) => {
|
||||
let lifecycle = false;
|
||||
let ownerPresent = true;
|
||||
hooks = new HostedHookSession(
|
||||
{
|
||||
...options,
|
||||
lifecycleAuthority: () =>
|
||||
lifecycle
|
||||
? { operationId: 'close-original', claimGeneration: 2 }
|
||||
: undefined,
|
||||
},
|
||||
session,
|
||||
pin,
|
||||
);
|
||||
catalog = { ...catalog, hooks: [{ ...catalog.hooks[0], eventName }] };
|
||||
const acquire = vi.mocked(HostedWorkspaceBroker.prototype.acquire);
|
||||
const originalAcquire = acquire.getMockImplementation()!;
|
||||
acquire.mockImplementation(async function (
|
||||
this: HostedWorkspaceBroker,
|
||||
expected,
|
||||
) {
|
||||
if (lifecycle) {
|
||||
expect(expected).toEqual({
|
||||
runtimeBindingId: 'binding',
|
||||
generation: '1',
|
||||
});
|
||||
expect(hooks.hasPendingOperations).toBe(true);
|
||||
}
|
||||
await originalAcquire.call(this, expected);
|
||||
ownerPresent = true;
|
||||
});
|
||||
vi.spyOn(
|
||||
HostedWorkspaceBroker.prototype,
|
||||
'authorizeLifecycle',
|
||||
).mockResolvedValue();
|
||||
const control = vi.mocked(HostedWorkspaceBroker.prototype.hookControl);
|
||||
const originalControl = control.getMockImplementation()!;
|
||||
control.mockImplementation(async function (
|
||||
this: HostedWorkspaceBroker,
|
||||
operation,
|
||||
) {
|
||||
if (operation.kind === 'hook-execute' && !ownerPresent)
|
||||
throw new HostedWorkspaceBrokerRejection(
|
||||
404,
|
||||
'runtime_session_not_found',
|
||||
);
|
||||
return originalControl.call(this, operation);
|
||||
});
|
||||
await hooks.ensureReady();
|
||||
const originalRuntimeSessionId = hooks.broker.runtimeSessionId;
|
||||
ownerPresent = false;
|
||||
lifecycle = true;
|
||||
acquire.mockClear();
|
||||
const output = await hooks.fire(
|
||||
eventName,
|
||||
'lifecycle-after-restart',
|
||||
{},
|
||||
signal(),
|
||||
);
|
||||
expect(output?.hookSpecificOutput?.['additionalContext']).toBe('checked');
|
||||
expect(acquire).toHaveBeenCalledExactlyOnceWith({
|
||||
runtimeBindingId: 'binding',
|
||||
generation: '1',
|
||||
});
|
||||
expect(hooks.broker.runtimeSessionId).toBe(originalRuntimeSessionId);
|
||||
expect(hooks.broker.runtime).toEqual({
|
||||
bindingId: 'binding',
|
||||
generation: '1',
|
||||
workspaceGeneration: '1',
|
||||
});
|
||||
expect(
|
||||
vi.mocked(HostedWorkspaceBroker.prototype.warm),
|
||||
).toHaveBeenCalledOnce();
|
||||
expect(
|
||||
await hooks.fire(eventName, 'lifecycle-after-restart', {}, signal()),
|
||||
).toEqual(output);
|
||||
expect(
|
||||
requests.filter((request) => request.kind === 'hook-execute'),
|
||||
).toHaveLength(1);
|
||||
expect(hooks.hasPendingOperations).toBe(false);
|
||||
},
|
||||
);
|
||||
|
||||
it('keeps a lifecycle Hook intent retryable when its original owner cannot be restored', async () => {
|
||||
let lifecycle = false;
|
||||
hooks = new HostedHookSession(
|
||||
{
|
||||
...options,
|
||||
lifecycleAuthority: () =>
|
||||
lifecycle
|
||||
? { operationId: 'close-original', claimGeneration: 2 }
|
||||
: undefined,
|
||||
},
|
||||
session,
|
||||
pin,
|
||||
);
|
||||
catalog = {
|
||||
...catalog,
|
||||
hooks: [{ ...catalog.hooks[0], eventName: HookEventName.SessionEnd }],
|
||||
};
|
||||
vi.spyOn(
|
||||
HostedWorkspaceBroker.prototype,
|
||||
'authorizeLifecycle',
|
||||
).mockResolvedValue();
|
||||
await hooks.ensureReady();
|
||||
lifecycle = true;
|
||||
const refusal = new HostedWorkspaceBrokerRejection(
|
||||
409,
|
||||
'workspace_close_identity_unverified',
|
||||
);
|
||||
const acquire = vi.mocked(HostedWorkspaceBroker.prototype.acquire);
|
||||
const originalAcquire = acquire.getMockImplementation()!;
|
||||
acquire.mockRejectedValue(refusal);
|
||||
await expect(
|
||||
hooks.fire(HookEventName.SessionEnd, 'owner-refused', {}, signal()),
|
||||
).rejects.toBe(refusal);
|
||||
const children = session.authority
|
||||
.extensionRecordsInDomain('hook_execution')
|
||||
.map((entry) => parseHookExecution(entry.record))
|
||||
.filter((record) => record.hookId !== '__plan__');
|
||||
expect(children).toHaveLength(1);
|
||||
expect(children[0].run.execution).toBe('intent');
|
||||
expect(children[0].run.reason).toBeNull();
|
||||
expect(
|
||||
requests.filter((request) => request.kind === 'hook-execute'),
|
||||
).toHaveLength(0);
|
||||
acquire.mockImplementation(originalAcquire);
|
||||
expect(
|
||||
(await hooks.fire(HookEventName.SessionEnd, 'owner-refused', {}, signal()))
|
||||
?.hookSpecificOutput?.['additionalContext'],
|
||||
).toBe('checked');
|
||||
expect(
|
||||
requests.filter((request) => request.kind === 'hook-execute'),
|
||||
).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('preserves an uncertain lifecycle Hook outcome after original-owner restoration', async () => {
|
||||
let lifecycle = false;
|
||||
hooks = new HostedHookSession(
|
||||
{
|
||||
...options,
|
||||
lifecycleAuthority: () =>
|
||||
lifecycle
|
||||
? { operationId: 'close-original', claimGeneration: 2 }
|
||||
: undefined,
|
||||
},
|
||||
session,
|
||||
pin,
|
||||
);
|
||||
catalog = {
|
||||
...catalog,
|
||||
hooks: [{ ...catalog.hooks[0], eventName: HookEventName.SessionEnd }],
|
||||
};
|
||||
vi.spyOn(
|
||||
HostedWorkspaceBroker.prototype,
|
||||
'authorizeLifecycle',
|
||||
).mockResolvedValue();
|
||||
await hooks.ensureReady();
|
||||
lifecycle = true;
|
||||
const control = vi.mocked(HostedWorkspaceBroker.prototype.hookControl);
|
||||
const originalControl = control.getMockImplementation()!;
|
||||
let dispatched = 0;
|
||||
control.mockImplementation(async function (
|
||||
this: HostedWorkspaceBroker,
|
||||
operation,
|
||||
) {
|
||||
if (operation.kind === 'hook-execute') {
|
||||
dispatched++;
|
||||
throw new TypeError('reply lost after possible worker effect');
|
||||
}
|
||||
return originalControl.call(this, operation);
|
||||
});
|
||||
await expect(
|
||||
hooks.fire(HookEventName.SessionEnd, 'unknown-original', {}, signal()),
|
||||
).rejects.toBeInstanceOf(HostedHookRecoveryRequiredError);
|
||||
await expect(
|
||||
hooks.fire(HookEventName.SessionEnd, 'unknown-original', {}, signal()),
|
||||
).rejects.toBeInstanceOf(HostedHookRecoveryRequiredError);
|
||||
const child = session.authority
|
||||
.extensionRecordsInDomain('hook_execution')
|
||||
.map((entry) => parseHookExecution(entry.record))
|
||||
.find((record) => record.hookId !== '__plan__')!;
|
||||
expect(child.run).toMatchObject({
|
||||
state: 'recovery_blocked',
|
||||
execution: 'outcome_unknown',
|
||||
reason: 'outcome_unknown',
|
||||
});
|
||||
expect(dispatched).toBe(1);
|
||||
});
|
||||
|
||||
it('commits a pinned plan before dispatch and replays the original occurrence after reconstruction', async () => {
|
||||
const output = await hooks.fire(
|
||||
HookEventName.PreToolUse,
|
||||
|
||||
@@ -761,6 +761,7 @@ export class HostedHookSession {
|
||||
await this.commit('hook_execution', occurrenceId, marker);
|
||||
}
|
||||
if (marker.run.execution === 'intent') {
|
||||
await this.broker.authorizeLifecycle();
|
||||
marker = {
|
||||
...marker,
|
||||
run: { ...marker.run, state: 'running', execution: 'dispatch_started' },
|
||||
@@ -926,6 +927,7 @@ export class HostedHookSession {
|
||||
return cancelBeforeDispatch();
|
||||
if (hook.config.type === HookType.Prompt && !prompt)
|
||||
throw new Error('Hook model activation is unavailable.');
|
||||
await this.broker.authorizeLifecycle();
|
||||
record = {
|
||||
...record,
|
||||
run: {
|
||||
@@ -1114,8 +1116,17 @@ export class HostedHookSession {
|
||||
);
|
||||
this.recoveredBrokers.set(record.runtimeSessionId, broker);
|
||||
}
|
||||
if (current && !reacquire) await this.acquire();
|
||||
else await broker.acquire();
|
||||
if (current && !reacquire) {
|
||||
await this.acquire();
|
||||
if (this.options.lifecycleAuthority?.() && broker.runtime)
|
||||
await broker.acquire({
|
||||
runtimeBindingId: broker.runtime.bindingId,
|
||||
generation: broker.runtime.generation,
|
||||
});
|
||||
} else
|
||||
await broker.acquire(
|
||||
reacquire && record.run.runtime ? record.run.runtime : undefined,
|
||||
);
|
||||
if (
|
||||
!broker.runtime ||
|
||||
this.executions().some(
|
||||
@@ -1273,6 +1284,19 @@ export class HostedHookSession {
|
||||
return this.children.byOccurrence.get(occurrenceId) ?? [];
|
||||
}
|
||||
|
||||
async settleOccurrence(occurrenceId: string): Promise<void> {
|
||||
const marker = await this.status(occurrenceId);
|
||||
if (
|
||||
!marker.resultRef ||
|
||||
this.childrenOf(occurrenceId).some(
|
||||
(child) =>
|
||||
child.run.state === 'recovery_blocked' ||
|
||||
(!child.resultRef && child.run.execution !== 'not_started_proven'),
|
||||
)
|
||||
)
|
||||
throw new HostedHookRecoveryRequiredError();
|
||||
}
|
||||
|
||||
async status(id: string, cancel = false): Promise<HookExecution> {
|
||||
const direct = this.session.authority.extensionRecord('hook_execution', id);
|
||||
const matches = direct
|
||||
@@ -1498,8 +1522,16 @@ export class HostedHookSession {
|
||||
: output;
|
||||
}
|
||||
|
||||
async drain(): Promise<void> {
|
||||
for (const record of this.executions()) {
|
||||
async drain(
|
||||
excludedOccurrences: ReadonlySet<string> = new Set(),
|
||||
): Promise<void> {
|
||||
const prior = () =>
|
||||
this.executions().filter(
|
||||
(record) =>
|
||||
!excludedOccurrences.has(record.hookExecutionId) &&
|
||||
!excludedOccurrences.has(record.occurrenceId),
|
||||
);
|
||||
for (const record of prior()) {
|
||||
if (!record.resultRef) await this.status(record.hookExecutionId, true);
|
||||
}
|
||||
await Promise.allSettled(
|
||||
@@ -1508,7 +1540,7 @@ export class HostedHookSession {
|
||||
await Promise.allSettled(
|
||||
[...this.occurrences.values()].map((occurrence) => occurrence.result),
|
||||
);
|
||||
for (const record of this.executions()) {
|
||||
for (const record of prior()) {
|
||||
const latest = await this.status(record.hookExecutionId, true);
|
||||
if (!latest.resultRef && latest.run.execution !== 'not_started_proven')
|
||||
throw new HostedHookRecoveryRequiredError();
|
||||
|
||||
@@ -32,10 +32,12 @@ import type {
|
||||
ManagedHookControl,
|
||||
ManagedHookOperationView,
|
||||
} from '@qwen-code/qwen-code-core/managed-runtime/managed-hook-protocol.js';
|
||||
import type { ManagedSessionLifecycleAuthority } from '@qwen-code/qwen-code-core/managed-runtime/http-managed-session-store.js';
|
||||
|
||||
export interface HostedWorkspaceBrokerOptions {
|
||||
baseUrl: string;
|
||||
token: string;
|
||||
lifecycleAuthority?: () => ManagedSessionLifecycleAuthority | undefined;
|
||||
}
|
||||
|
||||
function object(value: unknown): Record<string, unknown> {
|
||||
@@ -140,9 +142,23 @@ export class HostedWorkspaceBroker {
|
||||
await this.request('/runtimes:warm', {});
|
||||
}
|
||||
|
||||
async acquire(): Promise<void> {
|
||||
async authorizeLifecycle(): Promise<void> {
|
||||
if (this.options.lifecycleAuthority?.())
|
||||
await this.request('/runtimes:authorize-lifecycle', {});
|
||||
}
|
||||
|
||||
async acquire(expected?: {
|
||||
runtimeBindingId: string;
|
||||
generation: string;
|
||||
}): Promise<void> {
|
||||
const response = await this.request('/tool-sessions:acquire', {
|
||||
turnKind: 'bootstrap',
|
||||
...(expected
|
||||
? {
|
||||
recoveryBindingId: expected.runtimeBindingId,
|
||||
recoveryGeneration: expected.generation,
|
||||
}
|
||||
: {}),
|
||||
});
|
||||
const scope = object(response['scope']);
|
||||
if (
|
||||
@@ -611,11 +627,20 @@ export class HostedWorkspaceBroker {
|
||||
if (!body)
|
||||
for (const [key, value] of Object.entries(fields))
|
||||
url.searchParams.set(key, String(value));
|
||||
const authority = this.options.lifecycleAuthority?.();
|
||||
const response = await fetch(url, {
|
||||
method: body ? 'POST' : 'GET',
|
||||
headers: {
|
||||
Authorization: `Bearer ${this.options.token}`,
|
||||
'Content-Type': 'application/json',
|
||||
...(authority
|
||||
? {
|
||||
'X-Qwen-Lifecycle-Operation-Id': authority.operationId,
|
||||
'X-Qwen-Lifecycle-Claim-Generation': String(
|
||||
authority.claimGeneration,
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
},
|
||||
...(body ? { body: JSON.stringify(fields) } : {}),
|
||||
redirect: 'error',
|
||||
|
||||
@@ -61,6 +61,7 @@ import * as stdio from '../utils/stdioHelpers.js';
|
||||
|
||||
const broker = vi.hoisted(() => ({
|
||||
fileHistory: vi.fn(),
|
||||
authorizeLifecycle: vi.fn(),
|
||||
workspaceContext: vi.fn(),
|
||||
warm: vi.fn(),
|
||||
acquire: vi.fn(),
|
||||
@@ -85,6 +86,7 @@ vi.mock('./hosted-workspace-broker.js', async (importOriginal) => ({
|
||||
HostedWorkspaceBroker: class {
|
||||
readonly runtimeSessionId = 'prompt';
|
||||
fileHistory = broker.fileHistory;
|
||||
authorizeLifecycle = broker.authorizeLifecycle;
|
||||
workspaceContext = broker.workspaceContext;
|
||||
warm = broker.warm;
|
||||
acquire = broker.acquire;
|
||||
@@ -172,6 +174,7 @@ beforeEach(async () => {
|
||||
vi.resetAllMocks();
|
||||
expectWritesStopped = false;
|
||||
for (const method of [
|
||||
broker.authorizeLifecycle,
|
||||
broker.warm,
|
||||
broker.acquire,
|
||||
broker.cancel,
|
||||
|
||||
@@ -452,6 +452,7 @@ export interface ServeOptions {
|
||||
* `v` is the wire schema version; bumped only on breaking frame changes.
|
||||
*/
|
||||
export interface HostedHarnessCapabilities {
|
||||
readonly lifecycleProtocolVersion?: 1;
|
||||
readonly protocolVersions: {
|
||||
readonly current: 1;
|
||||
readonly supported: readonly [1];
|
||||
|
||||
@@ -296,6 +296,198 @@ describe('HTTP Managed Session store', () => {
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
it('scopes lifecycle authorization to the current claim and preserves writer identity', async () => {
|
||||
const server = new FakeManagedSessionStore();
|
||||
const observed: Array<{
|
||||
path: string;
|
||||
claim: string | null;
|
||||
operation: string | null;
|
||||
}> = [];
|
||||
const stores = createHttpManagedSessionStores({
|
||||
baseUrl: 'https://session-store.test',
|
||||
sessionKey: SESSION_KEY,
|
||||
writerId: 'harness-a',
|
||||
writerToken: TOKEN_A,
|
||||
fetchFn: async (input, init) => {
|
||||
const url = new URL(requestUrl(input));
|
||||
if (!url.pathname.endsWith(':authorize'))
|
||||
return server.fetch(input, init);
|
||||
const headers = new Headers(init?.headers);
|
||||
expect(headers.get('X-Qwen-Managed-Writer-Token')).toBe(TOKEN_A);
|
||||
expect(JSON.parse(String(init?.body))).toMatchObject({
|
||||
workspaceId: SESSION_KEY.workspaceId,
|
||||
writerId: 'harness-a',
|
||||
writerGeneration: 1,
|
||||
});
|
||||
observed.push({
|
||||
path: url.pathname.split('/').at(-1)!,
|
||||
claim: headers.get('X-Qwen-Lifecycle-Claim-Generation'),
|
||||
operation: headers.get('X-Qwen-Lifecycle-Operation-Id'),
|
||||
});
|
||||
return jsonResponse({});
|
||||
},
|
||||
});
|
||||
try {
|
||||
await stores.journalStore.open({ sessionKey: SESSION_KEY });
|
||||
stores.setLifecycleAuthority({
|
||||
operationId: 'delete-original',
|
||||
claimGeneration: 1,
|
||||
});
|
||||
await stores.authorizeLifecycle('delete');
|
||||
stores.setLifecycleAuthority({
|
||||
operationId: 'delete-original',
|
||||
claimGeneration: 2,
|
||||
});
|
||||
await stores.authorizeLifecycle();
|
||||
stores.setLifecycleAuthority();
|
||||
await stores.authorizeOrdinary();
|
||||
expect(observed).toEqual([
|
||||
{
|
||||
path: 'lifecycle:authorize',
|
||||
operation: 'delete-original',
|
||||
claim: '1',
|
||||
},
|
||||
{
|
||||
path: 'lifecycle:authorize',
|
||||
operation: 'delete-original',
|
||||
claim: '2',
|
||||
},
|
||||
{ path: 'execution:authorize', operation: null, claim: null },
|
||||
]);
|
||||
} finally {
|
||||
await stores.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('authorizes cleanup with the original expired or sealed writer without renewing it', async () => {
|
||||
vi.useFakeTimers({ now: 1_790_000_000_000, toFake: ['Date'] });
|
||||
const server = new FakeManagedSessionStore();
|
||||
const authorize = vi.fn();
|
||||
const stores = createHttpManagedSessionStores({
|
||||
baseUrl: 'https://session-store.test',
|
||||
sessionKey: SESSION_KEY,
|
||||
writerId: 'harness-a',
|
||||
writerToken: TOKEN_A,
|
||||
fetchFn: async (input, init) => {
|
||||
const path = new URL(requestUrl(input)).pathname;
|
||||
if (path.endsWith('/writers:renew'))
|
||||
throw new Error('Original writer lease expired');
|
||||
if (path.endsWith('/lifecycle:authorize')) {
|
||||
authorize();
|
||||
expect(JSON.parse(String(init?.body))).toEqual({
|
||||
workspaceId: SESSION_KEY.workspaceId,
|
||||
writerId: 'harness-a',
|
||||
writerGeneration: 1,
|
||||
});
|
||||
const headers = new Headers(init?.headers);
|
||||
expect(headers.get('X-Qwen-Managed-Writer-Token')).toBe(TOKEN_A);
|
||||
expect(headers.get('X-Qwen-Lifecycle-Operation-Id')).toBe(
|
||||
'delete-original',
|
||||
);
|
||||
expect(headers.get('X-Qwen-Lifecycle-Claim-Generation')).toBe('2');
|
||||
return jsonResponse({});
|
||||
}
|
||||
return server.fetch(input, init);
|
||||
},
|
||||
});
|
||||
try {
|
||||
await stores.journalStore.open({ sessionKey: SESSION_KEY });
|
||||
vi.setSystemTime(1_790_000_400_000);
|
||||
stores.setLifecycleAuthority({
|
||||
operationId: 'delete-original',
|
||||
claimGeneration: 2,
|
||||
});
|
||||
await stores.authorizeLifecycle();
|
||||
expect(authorize).toHaveBeenCalledTimes(1);
|
||||
await expect(stores.authorizeLifecycle('delete')).rejects.toThrow(
|
||||
'Original writer lease expired',
|
||||
);
|
||||
expect(authorize).toHaveBeenCalledTimes(1);
|
||||
await stores.close();
|
||||
await stores.authorizeLifecycle();
|
||||
expect(authorize).toHaveBeenCalledTimes(2);
|
||||
await expect(stores.authorizeLifecycle('delete')).rejects.toThrow(
|
||||
'writer is not active',
|
||||
);
|
||||
expect(authorize).toHaveBeenCalledTimes(2);
|
||||
} finally {
|
||||
await stores.close();
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
['ordinary', false],
|
||||
['previous', false],
|
||||
['same', false],
|
||||
['ordinary', true],
|
||||
] as const)(
|
||||
'rechecks a changed claim after an in-flight %s renewal (refused=%s)',
|
||||
async (previous, refused) => {
|
||||
const server = new FakeManagedSessionStore();
|
||||
const claims: Array<string | null> = [];
|
||||
let release!: () => void;
|
||||
const barrier = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const stores = createHttpManagedSessionStores({
|
||||
baseUrl: 'https://session-store.test',
|
||||
sessionKey: SESSION_KEY,
|
||||
writerId: 'harness-a',
|
||||
writerToken: TOKEN_A,
|
||||
fetchFn: async (input, init) => {
|
||||
if (requestUrl(input).endsWith('/writers:renew')) {
|
||||
const headers = new Headers(init?.headers);
|
||||
claims.push(headers.get('X-Qwen-Lifecycle-Claim-Generation'));
|
||||
if (claims.length === 1) await barrier;
|
||||
else if (refused)
|
||||
return jsonResponse(
|
||||
{
|
||||
error: {
|
||||
code: 'workspace_lifecycle_claim_fenced',
|
||||
message: 'The lifecycle claim is fenced.',
|
||||
},
|
||||
},
|
||||
409,
|
||||
);
|
||||
}
|
||||
return server.fetch(input, init);
|
||||
},
|
||||
});
|
||||
try {
|
||||
await stores.journalStore.open({ sessionKey: SESSION_KEY });
|
||||
if (previous !== 'ordinary')
|
||||
stores.setLifecycleAuthority({
|
||||
operationId: 'delete',
|
||||
claimGeneration: previous === 'same' ? 2 : 1,
|
||||
});
|
||||
const old = stores.assertWritable();
|
||||
await vi.waitFor(() => expect(claims).toHaveLength(1));
|
||||
stores.setLifecycleAuthority({
|
||||
operationId: 'delete',
|
||||
claimGeneration: 2,
|
||||
});
|
||||
const current = stores.assertWritable();
|
||||
release();
|
||||
await old;
|
||||
if (refused)
|
||||
await expect(current).rejects.toMatchObject({
|
||||
status: 409,
|
||||
remoteCode: 'workspace_lifecycle_claim_fenced',
|
||||
});
|
||||
else await current;
|
||||
expect(claims).toEqual(
|
||||
previous === 'same'
|
||||
? ['2']
|
||||
: [previous === 'ordinary' ? null : '1', '2'],
|
||||
);
|
||||
} finally {
|
||||
release();
|
||||
await stores.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it('publishes bounded tool output immediately under the original writer grant', async () => {
|
||||
const server = new FakeManagedSessionStore();
|
||||
let publication: Record<string, unknown> | undefined;
|
||||
@@ -503,6 +695,11 @@ describe('HTTP Managed Session store', () => {
|
||||
'exhausted-503',
|
||||
'changed-receipt',
|
||||
'invalid-json',
|
||||
'lifecycle-403',
|
||||
'lifecycle-409',
|
||||
'uncertain-lifecycle-403',
|
||||
'ordinary-lifecycle-409',
|
||||
'uncertain-ordinary-lifecycle-409',
|
||||
])(
|
||||
'preserves activation transaction identity and failure fencing after %s',
|
||||
async (failure) => {
|
||||
@@ -516,6 +713,14 @@ describe('HTTP Managed Session store', () => {
|
||||
failure === 'uncommitted-503' ||
|
||||
failure === 'lost-commit-response' ||
|
||||
failure === 'lost-response-body';
|
||||
const refused =
|
||||
failure === 'lifecycle-403' ||
|
||||
failure === 'lifecycle-409' ||
|
||||
failure === 'ordinary-lifecycle-409';
|
||||
const ordinaryLifecycle = failure.includes('ordinary-lifecycle-409');
|
||||
const uncertainLifecycle =
|
||||
failure === 'uncertain-lifecycle-403' ||
|
||||
failure === 'uncertain-ordinary-lifecycle-409';
|
||||
let armed = false;
|
||||
let replay: unknown;
|
||||
const stores = createHttpManagedSessionStores({
|
||||
@@ -531,6 +736,30 @@ describe('HTTP Managed Session store', () => {
|
||||
>;
|
||||
if (body['operation'] === 'installActivation') {
|
||||
requests.push(String(init?.body));
|
||||
if (ordinaryLifecycle)
|
||||
expect(
|
||||
new Headers(init?.headers).has(
|
||||
'X-Qwen-Lifecycle-Operation-Id',
|
||||
),
|
||||
).toBe(false);
|
||||
if (refused || uncertainLifecycle)
|
||||
return jsonResponse(
|
||||
{
|
||||
error: {
|
||||
message: 'Lifecycle authorization revoked',
|
||||
code: ordinaryLifecycle
|
||||
? 'workspace_lifecycle_admission_closed'
|
||||
: failure === 'lifecycle-409'
|
||||
? 'workspace_lifecycle_authorization_revoked'
|
||||
: 'workspace_access_denied',
|
||||
},
|
||||
},
|
||||
uncertainLifecycle && requests.length === 1
|
||||
? 503
|
||||
: ordinaryLifecycle || failure === 'lifecycle-409'
|
||||
? 409
|
||||
: 403,
|
||||
);
|
||||
if (failure === 'permanent-409' || failure === 'exhausted-503')
|
||||
return jsonResponse(
|
||||
{ error: { code: 'commit_rejected' } },
|
||||
@@ -595,6 +824,11 @@ describe('HTTP Managed Session store', () => {
|
||||
create: { definitionRef, rootSnapshotRef, createdBy: 'test' },
|
||||
});
|
||||
try {
|
||||
if ((refused || uncertainLifecycle) && !ordinaryLifecycle)
|
||||
stores.setLifecycleAuthority({
|
||||
operationId: 'lifecycle',
|
||||
claimGeneration: 1,
|
||||
});
|
||||
armed = true;
|
||||
const controller = new ManagedHookActivationController(session);
|
||||
const run = vi.fn(async () => 'completed');
|
||||
@@ -632,16 +866,45 @@ describe('HTTP Managed Session store', () => {
|
||||
await expect(
|
||||
controller.runTurn('next-turn', async () => 'accepted'),
|
||||
).resolves.toBe('accepted');
|
||||
} else if (refused) {
|
||||
await expect(operation).rejects.toThrow();
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
expect(requests).toHaveLength(2);
|
||||
expect(JSON.parse(requests[1])).toMatchObject({
|
||||
firstSequence: JSON.parse(requests[0]).firstSequence,
|
||||
previousCommitDigest: JSON.parse(requests[0]).previousCommitDigest,
|
||||
});
|
||||
expect(session.authority.writesStopped).toBe(false);
|
||||
armed = false;
|
||||
if (ordinaryLifecycle)
|
||||
stores.setLifecycleAuthority({
|
||||
operationId: 'lifecycle',
|
||||
claimGeneration: 1,
|
||||
});
|
||||
await expect(
|
||||
controller.runHookOperation(
|
||||
{
|
||||
operationId: 'notification-retry',
|
||||
occurrenceId: 'notification-retry',
|
||||
originTurnId: null,
|
||||
},
|
||||
run,
|
||||
),
|
||||
).resolves.toBe('completed');
|
||||
expect(run).toHaveBeenCalledOnce();
|
||||
} else {
|
||||
await expect(operation).rejects.toThrow('writes stopped');
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
expect(requests).toHaveLength(failure === 'exhausted-503' ? 3 : 1);
|
||||
expect(requests).toHaveLength(
|
||||
failure === 'exhausted-503' ? 3 : uncertainLifecycle ? 2 : 1,
|
||||
);
|
||||
expect(session.authority.writesStopped).toBe(true);
|
||||
await expect(controller.runTurn('next-turn', run)).rejects.toThrow(
|
||||
'current Session activation',
|
||||
);
|
||||
}
|
||||
expect(requests.every((body) => body === requests[0])).toBe(true);
|
||||
if (!refused)
|
||||
expect(requests.every((body) => body === requests[0])).toBe(true);
|
||||
} finally {
|
||||
await session.close();
|
||||
}
|
||||
|
||||
@@ -37,6 +37,7 @@ import {
|
||||
type DurableToolResultResourceStore,
|
||||
} from './resource-tool-result-store.js';
|
||||
import {
|
||||
ManagedSessionCommitRejectedError,
|
||||
scanManagedSessionJournal,
|
||||
type ManagedSessionJournalHandle,
|
||||
type ManagedSessionJournalScan,
|
||||
@@ -91,6 +92,11 @@ export interface HttpManagedSessionStoreOptions {
|
||||
readonly allowInsecureHttp?: boolean;
|
||||
}
|
||||
|
||||
export interface ManagedSessionLifecycleAuthority {
|
||||
readonly operationId: string;
|
||||
readonly claimGeneration: number;
|
||||
}
|
||||
|
||||
export interface HttpManagedSessionStores {
|
||||
readonly journalStore: ManagedSessionJournalStore;
|
||||
readonly resourceStore: ManagedSessionResourceStore;
|
||||
@@ -98,6 +104,9 @@ export interface HttpManagedSessionStores {
|
||||
readonly toolResultResources: DurableToolResultResourceStore;
|
||||
assertWritable(): Promise<void>;
|
||||
close(): Promise<void>;
|
||||
setLifecycleAuthority(authority?: ManagedSessionLifecycleAuthority): void;
|
||||
authorizeLifecycle(kind?: 'close' | 'delete'): Promise<void>;
|
||||
authorizeOrdinary(kind?: 'legacy-close'): Promise<void>;
|
||||
}
|
||||
|
||||
export interface HttpToolPublicationOwner {
|
||||
@@ -146,6 +155,10 @@ export function createHttpManagedSessionStores(
|
||||
},
|
||||
assertWritable: () => client.assertWritable(),
|
||||
close: () => client.seal(),
|
||||
setLifecycleAuthority: (authority) =>
|
||||
client.setLifecycleAuthority(authority),
|
||||
authorizeLifecycle: (kind) => client.authorizeLifecycle(kind),
|
||||
authorizeOrdinary: (kind) => client.authorizeOrdinary(kind),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -572,8 +585,39 @@ class ManagedSessionStoreHttpClient {
|
||||
private readonly fetchFn: typeof fetch;
|
||||
private grant: WriterGrant | undefined;
|
||||
private renewPromise: Promise<void> | undefined;
|
||||
private renewingAuthority?: ManagedSessionLifecycleAuthority;
|
||||
private renewTimer: NodeJS.Timeout | undefined;
|
||||
private sealed = false;
|
||||
private lifecycleAuthority?: ManagedSessionLifecycleAuthority;
|
||||
|
||||
setLifecycleAuthority(authority?: ManagedSessionLifecycleAuthority): void {
|
||||
this.lifecycleAuthority = authority;
|
||||
}
|
||||
|
||||
async authorizeOrdinary(kind?: 'legacy-close'): Promise<void> {
|
||||
await this.assertWritable();
|
||||
await this.json('/execution:authorize', 'POST', {
|
||||
...(kind ? { kind } : {}),
|
||||
workspaceId: this.sessionKey.workspaceId,
|
||||
writerId: this.writerId,
|
||||
writerGeneration: this.grant!.writerGeneration,
|
||||
});
|
||||
}
|
||||
|
||||
async authorizeLifecycle(kind?: 'close' | 'delete'): Promise<void> {
|
||||
if (kind) await this.assertWritable();
|
||||
const grant = this.grant;
|
||||
if (!grant)
|
||||
throw new ManagedSessionRecordError(
|
||||
'the HTTP Managed Session writer is not active.',
|
||||
);
|
||||
await this.json('/lifecycle:authorize', 'POST', {
|
||||
...(kind ? { kind } : {}),
|
||||
workspaceId: this.sessionKey.workspaceId,
|
||||
writerId: this.writerId,
|
||||
writerGeneration: grant.writerGeneration,
|
||||
});
|
||||
}
|
||||
private readonly publicationAdmissions = new Map<string, string>();
|
||||
|
||||
constructor(
|
||||
@@ -794,6 +838,18 @@ class ManagedSessionStoreHttpClient {
|
||||
);
|
||||
break;
|
||||
} catch (error) {
|
||||
if (
|
||||
attempt === 0 &&
|
||||
error instanceof ManagedSessionStoreHttpError &&
|
||||
((error.status === 409 &&
|
||||
error.remoteCode === 'workspace_lifecycle_admission_closed') ||
|
||||
(this.lifecycleAuthority &&
|
||||
(error.status === 403 ||
|
||||
(error.status === 409 &&
|
||||
(error.remoteCode.startsWith('workspace_lifecycle_') ||
|
||||
error.remoteCode === 'workspace_unavailable')))))
|
||||
)
|
||||
throw new ManagedSessionCommitRejectedError(error);
|
||||
const uncertain =
|
||||
(error instanceof ManagedSessionStoreHttpError &&
|
||||
(error.status === 429 || error.status >= 500)) ||
|
||||
@@ -1093,8 +1149,20 @@ class ManagedSessionStoreHttpClient {
|
||||
}
|
||||
|
||||
private renewWriter(): Promise<void> {
|
||||
if (this.renewPromise !== undefined) return this.renewPromise;
|
||||
if (this.renewPromise !== undefined) {
|
||||
if (
|
||||
this.lifecycleAuthority?.operationId ===
|
||||
this.renewingAuthority?.operationId &&
|
||||
this.lifecycleAuthority?.claimGeneration ===
|
||||
this.renewingAuthority?.claimGeneration
|
||||
)
|
||||
return this.renewPromise;
|
||||
return this.renewPromise.then(() => this.renewWriter());
|
||||
}
|
||||
const grant = this.requireGrant();
|
||||
this.renewingAuthority = this.lifecycleAuthority
|
||||
? { ...this.lifecycleAuthority }
|
||||
: undefined;
|
||||
const renewal = (async () => {
|
||||
const renewed = parseWriterGrant(
|
||||
await this.json('/writers:renew', 'POST', {
|
||||
@@ -1268,6 +1336,15 @@ class ManagedSessionStoreHttpClient {
|
||||
this.sessionKey.tenantId,
|
||||
[HTTP_MANAGED_SESSION_STORE_CONTRACT.writerTokenHeader]:
|
||||
this.writerToken,
|
||||
...(this.lifecycleAuthority
|
||||
? {
|
||||
'X-Qwen-Lifecycle-Operation-Id':
|
||||
this.lifecycleAuthority.operationId,
|
||||
'X-Qwen-Lifecycle-Claim-Generation': String(
|
||||
this.lifecycleAuthority.claimGeneration,
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
...(body === undefined ? {} : { 'Content-Type': 'application/json' }),
|
||||
},
|
||||
...(body === undefined ? {} : { body: JSON.stringify(body) }),
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
import * as fs from 'node:fs/promises';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import { Storage } from '../config/storage.js';
|
||||
import type { ChatRecord } from '../services/chatRecordingService.js';
|
||||
import { SessionWriterLease } from '../services/session-writer-lease.js';
|
||||
@@ -35,6 +35,8 @@ const sessionKey = { tenantId: 't1', workspaceId: 'w1', sessionId };
|
||||
const temporaryDirectories = new Set<string>();
|
||||
|
||||
afterEach(async () => {
|
||||
vi.useRealTimers();
|
||||
vi.restoreAllMocks();
|
||||
for (const directory of temporaryDirectories) {
|
||||
await fs.rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
@@ -227,6 +229,7 @@ describe('managed session assembly', () => {
|
||||
const session = await open(workspace);
|
||||
await session.sink.write(record({ uuid: 'rec-user-1' }));
|
||||
await session.close();
|
||||
expect(session.authority.currentActivation?.phase).toBe('released');
|
||||
|
||||
await expect(
|
||||
SessionWriterLease.acquire({
|
||||
@@ -237,6 +240,54 @@ describe('managed session assembly', () => {
|
||||
).rejects.toThrow();
|
||||
});
|
||||
|
||||
it('stops renewal and seals an externally fenced writer without appending activation release', async () => {
|
||||
vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] });
|
||||
const workspace = await createWorkspace();
|
||||
const session = await open(workspace);
|
||||
const committed = session.authority.commitProof;
|
||||
const transcript = await fs.readFile(workspace.transcriptPath, 'utf8');
|
||||
let finishRenewal!: () => void;
|
||||
const pendingRenewal = new Promise<void>((resolve) => {
|
||||
finishRenewal = resolve;
|
||||
});
|
||||
const renew = vi
|
||||
.spyOn(session.authority, 'renewActivation')
|
||||
.mockImplementation(async () => {
|
||||
await pendingRenewal;
|
||||
return undefined;
|
||||
});
|
||||
const release = vi
|
||||
.spyOn(session.authority, 'releaseActivation')
|
||||
.mockRejectedValue(new Error('journal appends are fenced'));
|
||||
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(20_000);
|
||||
expect(renew).toHaveBeenCalledTimes(1);
|
||||
await session.close({ releaseActivation: false });
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(renew).toHaveBeenCalledTimes(1);
|
||||
expect(release).not.toHaveBeenCalled();
|
||||
expect(await fs.readFile(workspace.transcriptPath, 'utf8')).toBe(
|
||||
transcript,
|
||||
);
|
||||
expect(session.authority.commitProof).toEqual(committed);
|
||||
expect(session.authority.currentActivation?.phase).toBe('active');
|
||||
const lockPath = path.join(
|
||||
workspace.runtimeBaseDir,
|
||||
'tmp',
|
||||
'session-writer-locks',
|
||||
`${encodeURIComponent(sessionId)}.lock`,
|
||||
);
|
||||
expect(JSON.parse(await fs.readFile(lockPath, 'utf8'))).toMatchObject({
|
||||
state: 'sealed',
|
||||
last_commit_sequence: committed.lastCommitSequence,
|
||||
committed_prefix_hash: committed.committedPrefixHash,
|
||||
});
|
||||
} finally {
|
||||
finishRenewal();
|
||||
}
|
||||
});
|
||||
|
||||
it('seals with the commit proof and reopens through a certified takeover', async () => {
|
||||
const workspace = await createWorkspace();
|
||||
const session = await open(workspace);
|
||||
|
||||
@@ -136,8 +136,11 @@ export interface ManagedSession {
|
||||
readonly activationId: string;
|
||||
readonly epoch: number;
|
||||
}>;
|
||||
/** Seals the writer, leaving the at-rest barrier in place. */
|
||||
close(): Promise<void>;
|
||||
/**
|
||||
* Seals the writer, leaving the at-rest barrier in place. Omit activation
|
||||
* release only when an external permanent lifecycle fence forbids appends.
|
||||
*/
|
||||
close(options?: { readonly releaseActivation?: boolean }): Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -253,13 +256,14 @@ export async function openManagedSession(
|
||||
return activation;
|
||||
},
|
||||
// Sealing is the at-rest barrier, but only the lease's owner may end it.
|
||||
// A call that owns the whole lifecycle also records the boundary, or the
|
||||
// activation would read as abandoned. An adopted lease still stops the
|
||||
// renewal it started; only the lease itself stays with its owner.
|
||||
close: async () => {
|
||||
// Normally the lifecycle owner also records the boundary, or the activation
|
||||
// reads as abandoned. An external permanent fence can forbid that append.
|
||||
// An adopted lease stops renewal but leaves the lease with its owner.
|
||||
close: async (options) => {
|
||||
renewal.stop();
|
||||
if (adopted) return;
|
||||
await authority.releaseActivation();
|
||||
if (options?.releaseActivation !== false)
|
||||
await authority.releaseActivation();
|
||||
await authority.close();
|
||||
},
|
||||
};
|
||||
|
||||
@@ -80,6 +80,7 @@ import {
|
||||
type ChildAcceptance,
|
||||
} from './managed-child-acceptance-record.js';
|
||||
import {
|
||||
ManagedSessionCommitRejectedError,
|
||||
managedSessionActivationStateFrom,
|
||||
managedSessionCommandKey,
|
||||
type ManagedSessionActivationState,
|
||||
@@ -2357,6 +2358,7 @@ export class LocalManagedSessionAuthority {
|
||||
await this.journal.appendTransaction(records);
|
||||
this.lastRecordUuid = final.uuid;
|
||||
} catch (cause) {
|
||||
if (cause instanceof ManagedSessionCommitRejectedError) throw cause;
|
||||
// Records may already be on disk, so the sequences this transaction
|
||||
// claimed are spent whether or not the marker landed.
|
||||
this.writeFailure =
|
||||
|
||||
@@ -23,6 +23,15 @@ import {
|
||||
type ManagedSessionKey,
|
||||
} from './managed-session-records.js';
|
||||
|
||||
/** A durable store refused the transaction without appending any records. */
|
||||
export class ManagedSessionCommitRejectedError extends ManagedSessionRecordError {
|
||||
constructor(cause: Error) {
|
||||
super(cause.message);
|
||||
this.cause = cause;
|
||||
this.name = 'ManagedSessionCommitRejectedError';
|
||||
}
|
||||
}
|
||||
|
||||
export interface ManagedSessionCommitReceipt {
|
||||
readonly transactionId: string;
|
||||
readonly commandId: string;
|
||||
|
||||
+42
-8
@@ -41,14 +41,45 @@ public class ManagedSessionStoreController {
|
||||
this.store = store;
|
||||
}
|
||||
|
||||
private static com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority lifecycleAuthority(HttpHeaders headers) {
|
||||
String operation = headers.getFirst(com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority.OPERATION_HEADER);
|
||||
String generation = headers.getFirst(com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority.GENERATION_HEADER);
|
||||
if (operation == null && generation == null) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
return new com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority(operation, Long.parseLong(generation));
|
||||
} catch (RuntimeException error) {
|
||||
throw new ApiException(org.springframework.http.HttpStatus.BAD_REQUEST, "invalid_lifecycle_authority", "Invalid lifecycle authority.");
|
||||
}
|
||||
}
|
||||
|
||||
@PostMapping("/execution:authorize")
|
||||
public java.util.Map<String, Object> authorizeOrdinary(TenantContext tenant, @PathVariable String sessionId,
|
||||
@RequestHeader(ManagedSessionStoreModels.WRITER_TOKEN_HEADER) String writerToken,
|
||||
@Valid @RequestBody ManagedSessionStoreModels.AuthorizeLifecycleRequest request) {
|
||||
store.authorizeOrdinary(tenant.tenantId(), sessionId, writerToken, request);
|
||||
return java.util.Map.of("authorized", true);
|
||||
}
|
||||
|
||||
@PostMapping("/lifecycle:authorize")
|
||||
public java.util.Map<String, Object> authorizeLifecycle(TenantContext tenant, @PathVariable String sessionId,
|
||||
@RequestHeader(ManagedSessionStoreModels.WRITER_TOKEN_HEADER) String writerToken,
|
||||
@Valid @RequestBody ManagedSessionStoreModels.AuthorizeLifecycleRequest request,
|
||||
@RequestHeader HttpHeaders headers) {
|
||||
store.authorizeLifecycle(tenant.tenantId(), sessionId, writerToken, request, lifecycleAuthority(headers));
|
||||
return java.util.Map.of("authorized", true);
|
||||
}
|
||||
|
||||
@PostMapping("/writers:acquire")
|
||||
public WriterGrant acquire(TenantContext tenant,
|
||||
@PathVariable String sessionId,
|
||||
@RequestHeader(ManagedSessionStoreModels.WRITER_TOKEN_HEADER)
|
||||
String writerToken,
|
||||
@Valid @RequestBody AcquireWriterRequest request) {
|
||||
@Valid @RequestBody AcquireWriterRequest request,
|
||||
@RequestHeader HttpHeaders headers) {
|
||||
return store.acquireWriter(tenant.tenantId(), sessionId,
|
||||
writerToken, request);
|
||||
writerToken, request, lifecycleAuthority(headers));
|
||||
}
|
||||
|
||||
@PostMapping("/writers:renew")
|
||||
@@ -56,9 +87,10 @@ public class ManagedSessionStoreController {
|
||||
@PathVariable String sessionId,
|
||||
@RequestHeader(ManagedSessionStoreModels.WRITER_TOKEN_HEADER)
|
||||
String writerToken,
|
||||
@Valid @RequestBody RenewWriterRequest request) {
|
||||
@Valid @RequestBody RenewWriterRequest request,
|
||||
@RequestHeader HttpHeaders headers) {
|
||||
return store.renewWriter(tenant.tenantId(), sessionId,
|
||||
writerToken, request);
|
||||
writerToken, request, lifecycleAuthority(headers));
|
||||
}
|
||||
|
||||
@PostMapping("/writers:seal")
|
||||
@@ -66,9 +98,10 @@ public class ManagedSessionStoreController {
|
||||
@PathVariable String sessionId,
|
||||
@RequestHeader(ManagedSessionStoreModels.WRITER_TOKEN_HEADER)
|
||||
String writerToken,
|
||||
@Valid @RequestBody SealWriterRequest request) {
|
||||
@Valid @RequestBody SealWriterRequest request,
|
||||
@RequestHeader HttpHeaders headers) {
|
||||
return store.sealWriter(tenant.tenantId(), sessionId,
|
||||
writerToken, request);
|
||||
writerToken, request, lifecycleAuthority(headers));
|
||||
}
|
||||
|
||||
@PostMapping("/recovery:block")
|
||||
@@ -86,9 +119,10 @@ public class ManagedSessionStoreController {
|
||||
@PathVariable String sessionId,
|
||||
@RequestHeader(ManagedSessionStoreModels.WRITER_TOKEN_HEADER)
|
||||
String writerToken,
|
||||
@Valid @RequestBody CommitTransactionRequest request) {
|
||||
@Valid @RequestBody CommitTransactionRequest request,
|
||||
@RequestHeader HttpHeaders headers) {
|
||||
return store.commit(tenant.tenantId(), sessionId,
|
||||
writerToken, request);
|
||||
writerToken, request, lifecycleAuthority(headers));
|
||||
}
|
||||
|
||||
@GetMapping("/restore")
|
||||
|
||||
+10
@@ -12,6 +12,16 @@ public interface HarnessConnector extends AutoCloseable {
|
||||
return false;
|
||||
}
|
||||
|
||||
default boolean supportsLifecycle() { return false; }
|
||||
|
||||
default JsonNode settleLifecycle(com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord operation) {
|
||||
throw new UnsupportedOperationException("Hosted lifecycle is unavailable");
|
||||
}
|
||||
|
||||
default void detachLifecycle(com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord operation) {
|
||||
throw new UnsupportedOperationException("Hosted lifecycle is unavailable");
|
||||
}
|
||||
|
||||
Attachment createOrLoad(String tenantId, String sessionId,
|
||||
boolean loadExisting);
|
||||
|
||||
|
||||
+70
@@ -123,6 +123,76 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean supportsLifecycle() {
|
||||
if (!isWorkspaceFilesAvailable()) {
|
||||
return false;
|
||||
}
|
||||
try {
|
||||
return client().capabilities().getLifecycleProtocolVersion() == 1;
|
||||
} catch (com.alibaba.qwen.code.daemon.DaemonException
|
||||
| IllegalStateException unavailable) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public com.fasterxml.jackson.databind.JsonNode settleLifecycle(
|
||||
com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord operation) {
|
||||
try {
|
||||
return doSettleLifecycle(operation);
|
||||
} catch (HostedHarnessGenerationException error) {
|
||||
adoptGeneration(error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
private com.fasterxml.jackson.databind.JsonNode doSettleLifecycle(
|
||||
com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord operation) {
|
||||
SessionRecord session = sessions.requireSession(operation.tenantId(), operation.sessionId());
|
||||
AttachmentKey key = new AttachmentKey(operation.tenantId(), operation.sessionId());
|
||||
HarnessSessionRef ref = attachments.get(key);
|
||||
if (ref == null) {
|
||||
ref = client().loadSession(new LoadHarnessSession(session.sessionId(), managedSessionStore(session),
|
||||
false, toolProfile(session), false).forLifecycle(operation.operationId(), operation.claimGeneration()));
|
||||
attachments.put(key, ref);
|
||||
}
|
||||
var authority = Map.<String, Object>of("operationId", operation.operationId(), "claimGeneration", operation.claimGeneration());
|
||||
var request = Map.<String, Object>of("sessionKey", Map.of("tenantId", session.tenantId(),
|
||||
"workspaceId", session.workspace().getWorkspaceId(), "sessionId", session.sessionId()),
|
||||
"kind", operation.kind().name().toLowerCase(java.util.Locale.ROOT), "authority", authority);
|
||||
return new com.fasterxml.jackson.databind.ObjectMapper().valueToTree(client().settleLifecycle(ref, request));
|
||||
}
|
||||
|
||||
@Override
|
||||
public void detachLifecycle(com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord operation) {
|
||||
try {
|
||||
doDetachLifecycle(operation);
|
||||
} catch (HostedHarnessGenerationException error) {
|
||||
adoptGeneration(error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
private void doDetachLifecycle(com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord operation) {
|
||||
AttachmentKey key = new AttachmentKey(operation.tenantId(), operation.sessionId());
|
||||
HarnessSessionRef ref = attachments.get(key);
|
||||
try {
|
||||
var authority = Map.<String, Object>of("operationId", operation.operationId(), "claimGeneration", operation.claimGeneration());
|
||||
if (ref == null) {
|
||||
client().detachLifecycle(operation.sessionId(), authority);
|
||||
} else {
|
||||
client().detachLifecycle(ref, authority);
|
||||
}
|
||||
} catch (DaemonHttpException error) {
|
||||
if (error.getStatusCode() != 404) {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
attachments.remove(key);
|
||||
pendingRecovery.remove(key);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isWorkspaceFilesAvailable() {
|
||||
return properties.isWorkspaceFilesEnabled();
|
||||
|
||||
+8
-2
@@ -118,7 +118,13 @@ public class EmbeddedRuntimeBroker implements RuntimeWarmer, AutoCloseable {
|
||||
}
|
||||
|
||||
@Override
|
||||
public java.util.concurrent.CompletionStage<RuntimeScope> resolve(String sessionId) {
|
||||
public CompletionStage<RuntimeScope> resolve(String sessionId) {
|
||||
return resolve(sessionId, null);
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletionStage<RuntimeScope> resolve(String sessionId,
|
||||
com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority authority) {
|
||||
SessionRecord session = store.findSessionById(sessionId)
|
||||
.orElse(null);
|
||||
if (session == null) {
|
||||
@@ -130,7 +136,7 @@ public class EmbeddedRuntimeBroker implements RuntimeWarmer, AutoCloseable {
|
||||
}
|
||||
if (session.workspace() != null) {
|
||||
if (workspaces != null) {
|
||||
return CompletableFuture.completedFuture(workspaces.resolve(sessionId).scope());
|
||||
return CompletableFuture.completedFuture(workspaces.resolve(sessionId, authority).scope());
|
||||
}
|
||||
return CompletableFuture.failedFuture(
|
||||
new RuntimeBrokerException(409,
|
||||
|
||||
+9
-3
@@ -96,7 +96,8 @@ public class ManagedAgentService {
|
||||
|
||||
private boolean supportsClose(SessionRecord session) {
|
||||
return session.workspace() == null || store.workspaceFilesEnabled()
|
||||
&& runtimeWarmer != null && runtimeWarmer.supportsWorkspaceClose();
|
||||
&& runtimeWarmer != null && runtimeWarmer.supportsWorkspaceClose()
|
||||
&& harness.supportsLifecycle();
|
||||
}
|
||||
|
||||
private boolean hasActions(SessionRecord session) {
|
||||
@@ -604,7 +605,7 @@ public class ManagedAgentService {
|
||||
true,
|
||||
session.workspace() == null,
|
||||
true,
|
||||
hasActions(session), supportsClose(session), retention, retention, retention),
|
||||
hasActions(session), supportsClose(session), retention, retention, supportsDelete(session, retention)),
|
||||
publicWorkspace(session));
|
||||
}
|
||||
|
||||
@@ -673,7 +674,12 @@ public class ManagedAgentService {
|
||||
// Stage H records its Session store holds (H0c).
|
||||
new WebShellSessionCapabilities(true, hasArtifacts(session),
|
||||
hasActions(session), maySubmit, supportsClose(session),
|
||||
retention, retention, retention));
|
||||
retention, retention, supportsDelete(session, retention)));
|
||||
}
|
||||
|
||||
private boolean supportsDelete(SessionRecord session, boolean retention) {
|
||||
return retention || session.workspace() != null && store.workspaceFilesEnabled()
|
||||
&& runtimeWarmer != null && runtimeWarmer.supportsWorkspaceClose() && harness.supportsLifecycle();
|
||||
}
|
||||
|
||||
private static WebShellWorkspace webShellWorkspace(SessionRecord session) {
|
||||
|
||||
+41
@@ -50,6 +50,17 @@ public class SessionLifecycleCoordinator {
|
||||
// is the documented transient case.
|
||||
private static final int CWD_CHANGE_ATTEMPT_BUDGET = 8;
|
||||
private final AgentStateStore store;
|
||||
private com.alibaba.qwen.code.managedagent.store.WorkspaceLifecycleStore lifecycle;
|
||||
|
||||
@org.springframework.beans.factory.annotation.Autowired(required = false)
|
||||
public void setWorkspaceLifecycleStore(com.alibaba.qwen.code.managedagent.store.WorkspaceLifecycleStore lifecycle) {
|
||||
this.lifecycle = lifecycle;
|
||||
}
|
||||
|
||||
public boolean supportsWorkspaceLifecycle() {
|
||||
return runtimeWarmer.supportsWorkspaceClose() && harness.supportsLifecycle();
|
||||
}
|
||||
|
||||
private final ManagedSessionStore sessionStore;
|
||||
private final HarnessConnector harness;
|
||||
private final RuntimeWarmer runtimeWarmer;
|
||||
@@ -169,6 +180,15 @@ public class SessionLifecycleCoordinator {
|
||||
blocked = "workspace_close_identity_unverified";
|
||||
}
|
||||
}
|
||||
if (cause instanceof com.alibaba.qwen.code.managedagent.api.ApiException apiError
|
||||
&& (apiError.getCode().startsWith("workspace_lifecycle") || apiError.getCode().startsWith("workspace_close"))) {
|
||||
blocked = apiError.getCode();
|
||||
}
|
||||
if ((cause instanceof com.alibaba.qwen.code.daemon.DaemonHttpException
|
||||
|| cause instanceof com.alibaba.qwen.code.daemon.MutationOutcomeUnknownException)
|
||||
&& operationIsLifecycle(claimed)) {
|
||||
blocked = "workspace_lifecycle_hooks_unsettled";
|
||||
}
|
||||
if (valid.get() && blocked != null) {
|
||||
store.blockLifecycleOperation(tenantId, sessionId, operationId, owner,
|
||||
claimed.claimGeneration(), blocked, Math.addExact(clock.millis(), delay));
|
||||
@@ -188,6 +208,10 @@ public class SessionLifecycleCoordinator {
|
||||
}
|
||||
}
|
||||
|
||||
private static boolean operationIsLifecycle(OperationRecord operation) {
|
||||
return operation.lifecycleProtocolVersion() == 1;
|
||||
}
|
||||
|
||||
// A cwd change settles without Harness or worker involvement: the probe
|
||||
// is read-only, and the commit transaction re-checks every fact it
|
||||
// depends on, so a reclaim can rerun this branch idempotently.
|
||||
@@ -267,6 +291,23 @@ public class SessionLifecycleCoordinator {
|
||||
&& ("CLOSED".equals(operation.sessionStatusBefore()) || "ARCHIVED".equals(operation.sessionStatusBefore()))) {
|
||||
return false;
|
||||
}
|
||||
if (bound && operation.lifecycleProtocolVersion() == 1) {
|
||||
if (lifecycle == null || !runtimeWarmer.supportsWorkspaceClose()) {
|
||||
throw com.alibaba.qwen.code.managedagent.store.WorkspaceLifecycleStore.blocked("workspace_close_identity_unverified");
|
||||
}
|
||||
if (lifecycle.recoverEffects(operation) == null) {
|
||||
if (!harness.supportsLifecycle()) {
|
||||
throw com.alibaba.qwen.code.managedagent.store.WorkspaceLifecycleStore.blocked("workspace_lifecycle_protocol_unavailable");
|
||||
}
|
||||
lifecycle.saveEffects(operation, harness.settleLifecycle(operation));
|
||||
}
|
||||
harness.detachLifecycle(operation);
|
||||
if (sessionStore.hasLiveWriter(operation.tenantId(), operation.sessionId())) {
|
||||
throw com.alibaba.qwen.code.managedagent.store.WorkspaceLifecycleStore.blocked("workspace_lifecycle_writer_active");
|
||||
}
|
||||
runtimeWarmer.closeWorkspace(operation.tenantId(), operation.sessionId()).toCompletableFuture().join();
|
||||
return true;
|
||||
}
|
||||
if (bound) {
|
||||
if (!runtimeWarmer.supportsWorkspaceClose()) {
|
||||
throw new com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException(409,
|
||||
|
||||
+4
-1
@@ -198,8 +198,11 @@ public class SessionLifecycleService {
|
||||
String digest = sessions.lifecycleDigest(sessionId, DIGEST_NAMES.get(kind));
|
||||
OperationAdmission admission;
|
||||
if (session.workspace() != null) {
|
||||
boolean activeLifecycle = "ACTIVE".equals(session.status()) && (kind == OperationKind.CLOSE || kind == OperationKind.DELETE);
|
||||
admission = store.beginWorkspaceLifecycle(tenantId, sessionId, kind, actorId, actorDigest(actorId),
|
||||
idempotencyKey, digest, kind == OperationKind.CLOSE && runtimeWarmer != null && runtimeWarmer.supportsWorkspaceClose());
|
||||
idempotencyKey, digest, activeLifecycle ? coordinator.supportsWorkspaceLifecycle()
|
||||
: kind == OperationKind.CLOSE && runtimeWarmer != null && runtimeWarmer.supportsWorkspaceClose(),
|
||||
activeLifecycle ? 1 : 0);
|
||||
} else {
|
||||
sessions.requireLegacyWorkspace(tenantId, actorId, sessionId);
|
||||
admission = store.beginOperation(tenantId, sessionId, kind, actorDigest(actorId), idempotencyKey, digest);
|
||||
|
||||
+2
-2
@@ -30,7 +30,7 @@ final class WorkspaceRuntimeProvisioner implements RuntimeProvisioner {
|
||||
if (!WorkspaceExecutionProfile.CAPABILITY_DIGEST.equals(scope.getCapabilityDigest())) {
|
||||
return delegate.createRequest(scope, isolationKey);
|
||||
}
|
||||
var resolved = resolver.resolve(isolationKey);
|
||||
var resolved = resolver.resolve(isolationKey, scope.getLifecycleAuthority());
|
||||
if (!resolved.scope().equals(scope)) {
|
||||
throw WorkspaceExecutionStore.unavailable();
|
||||
}
|
||||
@@ -70,7 +70,7 @@ final class WorkspaceRuntimeProvisioner implements RuntimeProvisioner {
|
||||
if (!request.isManagedContext() || !executionStore.verifiedRecoveryEnabled()) {
|
||||
return;
|
||||
}
|
||||
var resolved = resolver.resolve(request.getIsolationKey());
|
||||
var resolved = resolver.resolve(request.getIsolationKey(), request.getScope().getLifecycleAuthority());
|
||||
if (!resolved.scope().equals(request.getScope())
|
||||
|| !resolved.binding().getStorageId().equals(request.getStorageId())) {
|
||||
throw WorkspaceExecutionStore.unavailable();
|
||||
|
||||
+20
-2
@@ -61,9 +61,27 @@ final class WorkspaceRuntimeResolver {
|
||||
}
|
||||
|
||||
Resolved resolve(String sessionId) {
|
||||
return resolve(sessionId, null);
|
||||
}
|
||||
|
||||
Resolved resolve(String sessionId, com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority lifecycle) {
|
||||
return resolve(sessionId, lifecycle, false);
|
||||
}
|
||||
|
||||
Resolved resolveHook(String sessionId, com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority lifecycle) {
|
||||
return resolve(sessionId, lifecycle, true);
|
||||
}
|
||||
|
||||
private Resolved resolve(String sessionId, com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority lifecycle, boolean hook) {
|
||||
SessionRecord session = sessions.findSessionById(sessionId)
|
||||
.orElseThrow(WorkspaceExecutionStore::unavailable);
|
||||
authority.authorize(session);
|
||||
if (lifecycle == null && hook && "CLOSING".equals(session.status())) {
|
||||
authority.authorizeLegacyClose(session);
|
||||
} else if (lifecycle == null) {
|
||||
authority.authorize(session);
|
||||
} else {
|
||||
authority.authorizeLifecycle(session, lifecycle);
|
||||
}
|
||||
ContextBinding binding = session.workspace();
|
||||
Mount mount = mounts.get(new Storage(binding.getTenantId(), binding.getStorageId()));
|
||||
if (mount == null) {
|
||||
@@ -72,7 +90,7 @@ final class WorkspaceRuntimeResolver {
|
||||
verifyMountIntact(mount);
|
||||
return new Resolved(binding, new RuntimeScope(session.tenantId(), binding.getWorkspaceId(),
|
||||
Long.toString(binding.getWorkspaceGeneration()), mount.root().toString(),
|
||||
WorkspaceExecutionProfile.CAPABILITY_DIGEST, "session"));
|
||||
WorkspaceExecutionProfile.CAPABILITY_DIGEST, "session").withLifecycleAuthority(lifecycle));
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+10
-4
@@ -197,7 +197,7 @@ final class WorkspaceRuntimeTransport implements RuntimeTransport {
|
||||
if (!managed(session)) {
|
||||
throw WorkspaceExecutionStore.unavailable();
|
||||
}
|
||||
Context context = context(lease, session, false, true);
|
||||
Context context = context(lease, session, false, false, true);
|
||||
var runtime = context.runtime();
|
||||
if (!"kubernetes-workspace".equals(runtime.getRequest().getProvisionerKind())
|
||||
|| runtime.getState() != RuntimeBindingRecord.State.DRAINING || !runtime.isDrainRequested()
|
||||
@@ -332,7 +332,7 @@ final class WorkspaceRuntimeTransport implements RuntimeTransport {
|
||||
}
|
||||
boolean recovery = ManagedMcpProtocol.isRecovery(operation) || ManagedHookProtocol.isRecovery(operation)
|
||||
|| ManagedShellProtocol.isRecovery(operation);
|
||||
Context context = context(lease, session, !recovery);
|
||||
Context context = context(lease, session, !recovery, ManagedHookProtocol.isOperation(operation));
|
||||
if (context.runtime().getState() != RuntimeBindingRecord.State.READY
|
||||
&& !(recovery && context.runtime().getState() == RuntimeBindingRecord.State.DRAINING)) {
|
||||
throw WorkspaceExecutionStore.unavailable();
|
||||
@@ -380,10 +380,16 @@ final class WorkspaceRuntimeTransport implements RuntimeTransport {
|
||||
return context(lease, session, authorize, false);
|
||||
}
|
||||
|
||||
private Context context(RuntimeLease lease, RuntimeSession session, boolean authorize, boolean originalCsi) {
|
||||
private Context context(RuntimeLease lease, RuntimeSession session, boolean authorize, boolean hook) {
|
||||
return context(lease, session, authorize, hook, false);
|
||||
}
|
||||
|
||||
private Context context(RuntimeLease lease, RuntimeSession session, boolean authorize,
|
||||
boolean hook, boolean originalCsi) {
|
||||
ContextBinding binding;
|
||||
if (authorize) {
|
||||
var resolved = resolver.resolve(session.getHarnessSessionId());
|
||||
var resolved = hook ? resolver.resolveHook(session.getHarnessSessionId(), session.getScope().getLifecycleAuthority())
|
||||
: resolver.resolve(session.getHarnessSessionId(), session.getScope().getLifecycleAuthority());
|
||||
if (!resolved.scope().equals(session.getScope())) {
|
||||
throw WorkspaceExecutionStore.unavailable();
|
||||
}
|
||||
|
||||
+8
@@ -104,6 +104,14 @@ public interface AgentStateStore {
|
||||
OperationKind kind, String actorId, String actorDigest, String key,
|
||||
String digest, boolean closeSupported);
|
||||
|
||||
default OperationAdmission beginWorkspaceLifecycle(String tenantId, String sessionId, OperationKind kind,
|
||||
String actorId, String actorDigest, String key, String digest, boolean supported, int protocolVersion) {
|
||||
if (protocolVersion != 0) {
|
||||
throw new UnsupportedOperationException("Workspace lifecycle protocol is unavailable");
|
||||
}
|
||||
return beginWorkspaceLifecycle(tenantId, sessionId, kind, actorId, actorDigest, key, digest, supported);
|
||||
}
|
||||
|
||||
boolean hasCompletedWorkspaceClose(String tenantId, String sessionId);
|
||||
|
||||
/** The given Sessions with a completed workspace close, in one read. */
|
||||
|
||||
+107
-14
@@ -30,6 +30,7 @@ import com.alibaba.qwen.code.managedagent.store.StoreModels.TurnPage;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.TurnRecord;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.TurnSummary;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedWorkspaceRegistry.ResolvedBinding;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedworkspace.ContextBinding;
|
||||
import com.alibaba.qwen.code.runtimebroker.WorkspaceExecutionProfile;
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
@@ -90,6 +91,13 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
"turn_record." + String.join(", turn_record.",
|
||||
TURN_SUMMARY_COLUMNS.split(", "));
|
||||
private final JdbcTemplate jdbc;
|
||||
private WorkspaceLifecycleStore lifecycle;
|
||||
|
||||
@org.springframework.beans.factory.annotation.Autowired(required = false)
|
||||
public void setWorkspaceLifecycleStore(WorkspaceLifecycleStore lifecycle) {
|
||||
this.lifecycle = lifecycle;
|
||||
}
|
||||
|
||||
private final ObjectMapper objectMapper;
|
||||
private final Clock clock;
|
||||
private final String approvalMode;
|
||||
@@ -200,7 +208,9 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
additiveString(result, "target_cwd_relative"),
|
||||
additiveLong(result, "expected_context_revision"),
|
||||
additiveLong(result, "result_context_revision"),
|
||||
result.getString("error_code"));
|
||||
result.getString("error_code"),
|
||||
hasColumn(result, "lifecycle_protocol_version")
|
||||
? result.getInt("lifecycle_protocol_version") : 0);
|
||||
private final RowMapper<OperationTarget> operationTargetMapper =
|
||||
(result, row) -> new OperationTarget(
|
||||
result.getString("tenant_id"),
|
||||
@@ -700,6 +710,13 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
return beginLifecycle(tenantId, sessionId, kind, actorDigest, key, digest, actorId, closeSupported);
|
||||
}
|
||||
|
||||
@Override
|
||||
@Transactional
|
||||
public OperationAdmission beginWorkspaceLifecycle(String tenantId, String sessionId,
|
||||
OperationKind kind, String actorId, String actorDigest, String key, String digest, boolean supported, int protocolVersion) {
|
||||
return beginLifecycle(tenantId, sessionId, kind, actorDigest, key, digest, actorId, supported, protocolVersion);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean hasCompletedWorkspaceClose(String tenantId, String sessionId) {
|
||||
return completedWorkspaceCloses(tenantId, List.of(sessionId))
|
||||
@@ -773,7 +790,15 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
|
||||
private OperationAdmission beginLifecycle(String tenantId, String sessionId, OperationKind kind,
|
||||
String actorDigest, String idempotencyKey, String requestDigest, String actorId, boolean supported) {
|
||||
return beginLifecycle(tenantId, sessionId, kind, actorDigest, idempotencyKey, requestDigest, actorId, supported, 0);
|
||||
}
|
||||
|
||||
private OperationAdmission beginLifecycle(String tenantId, String sessionId, OperationKind kind,
|
||||
String actorDigest, String idempotencyKey, String requestDigest, String actorId, boolean supported, int protocolVersion) {
|
||||
WorkspaceMigrationAdmission.lockTenant(jdbc, tenantId);
|
||||
if (protocolVersion == 1) {
|
||||
ToolPublicationRetentionStore.lockTenant(jdbc, tenantId);
|
||||
}
|
||||
SessionRecord session = requireSessionForUpdate(tenantId, sessionId);
|
||||
if (session.workspace() != null) {
|
||||
requireWorkspaceCreator(session, actorId);
|
||||
@@ -802,8 +827,11 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
throw new ApiException(HttpStatus.NOT_FOUND, "session_not_found",
|
||||
"The Session was not found.");
|
||||
}
|
||||
if (kind == OperationKind.DELETE && List.of("CLOSED", "ARCHIVED").contains(session.status())) {
|
||||
protocolVersion = 0;
|
||||
}
|
||||
if (session.workspace() != null) {
|
||||
if (kind == OperationKind.CLOSE) {
|
||||
if (kind == OperationKind.CLOSE || kind == OperationKind.DELETE && "ACTIVE".equals(session.status()) && protocolVersion == 1) {
|
||||
if (!supported || !workspaceFilesEnabled) {
|
||||
throw workspaceExecutionUnavailable();
|
||||
}
|
||||
@@ -824,6 +852,9 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
}
|
||||
requireNoOpenOperation(tenantId, sessionId);
|
||||
validateOperationStart(session, kind);
|
||||
if (protocolVersion == 1) {
|
||||
WorkspaceLifecycleStore.requireIdleJournal(jdbc, objectMapper, tenantId, sessionId);
|
||||
}
|
||||
long now = lifecycleDatabaseTime();
|
||||
String operationId = publicId("op");
|
||||
Map<String, Object> data = Map.of("sessionId", sessionId,
|
||||
@@ -835,15 +866,21 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
+ " actor_digest, idempotency_key, request_digest,"
|
||||
+ " state, admission_stage, delivery_state,"
|
||||
+ " session_status_before, receipt_id, available_at,"
|
||||
+ " created_at, updated_at, completed_at) VALUES"
|
||||
+ " created_at, updated_at, completed_at, lifecycle_protocol_version) VALUES"
|
||||
+ " (?, ?, ?, ?, ?, ?, ?, ?, 'JAVA_DURABLE', ?, ?, ?,"
|
||||
+ " ?, ?, ?, ?)",
|
||||
+ " ?, ?, ?, ?, ?)",
|
||||
tenantId, sessionId, operationId, kind.name(), actorDigest,
|
||||
idempotencyKey, requestDigest,
|
||||
archive ? "COMPLETED" : "PENDING",
|
||||
archive ? "CONFIRMED" : "PENDING", session.status(),
|
||||
archive ? publicId("rcpt") : null, now, now, now,
|
||||
archive ? now : null);
|
||||
archive ? now : null, protocolVersion);
|
||||
if (protocolVersion == 1) {
|
||||
jdbc.execute((org.springframework.jdbc.core.ConnectionCallback<Void>) connection -> {
|
||||
com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository.beginHarnessLifecycle(connection, tenantId, sessionId, operationId);
|
||||
return null;
|
||||
});
|
||||
}
|
||||
jdbc.update("UPDATE managed_agent_session SET status = ?,"
|
||||
+ " updated_at = ?, version = version + 1 WHERE"
|
||||
+ " tenant_id = ? AND session_id = ?",
|
||||
@@ -919,10 +956,11 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
// turn_active), so the directory change must hold as well.
|
||||
if (hasOpenOperation(tenantId, sessionId)
|
||||
|| hasActiveTurn(tenantId, sessionId)
|
||||
|| hasDecidableAction(session)) {
|
||||
|| hasDecidableAction(session)
|
||||
|| hasRetainedRuntimeSession(tenantId, sessionId)) {
|
||||
throw new ApiException(HttpStatus.CONFLICT,
|
||||
"session_context_busy",
|
||||
"The Session has an active Turn or operation.");
|
||||
"The Session has an active Turn, operation or retained Runtime context.");
|
||||
}
|
||||
long now = lifecycleDatabaseTime();
|
||||
String operationId = publicId("op");
|
||||
@@ -983,7 +1021,8 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
failure = "context_revision_conflict";
|
||||
} else if (hasOpenOperation(tenantId, sessionId, operationId)
|
||||
|| hasActiveTurn(tenantId, sessionId)
|
||||
|| hasDecidableAction(session)) {
|
||||
|| hasDecidableAction(session)
|
||||
|| hasRetainedRuntimeSession(tenantId, sessionId)) {
|
||||
failure = "session_context_busy";
|
||||
} else if (!hasCwdChangeRegistryFacts(session)) {
|
||||
failure = "workspace_unavailable";
|
||||
@@ -1087,6 +1126,14 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
return hasOpenOperation(tenantId, sessionId, null);
|
||||
}
|
||||
|
||||
// Hook owners can outlive their Turn with an immutable worker context.
|
||||
// Only confirmed release permits changing that context's cwd/revision.
|
||||
private boolean hasRetainedRuntimeSession(String tenantId, String sessionId) {
|
||||
return !jdbc.queryForList("SELECT runtime_session_id FROM qwen_runtime_session"
|
||||
+ " WHERE tenant_id = ? AND harness_session_id = ?"
|
||||
+ " AND session_state <> 'RELEASED' LIMIT 1", tenantId, sessionId).isEmpty();
|
||||
}
|
||||
|
||||
// The bound later-Turn barrier: context-changing operations only —
|
||||
// pending command rows and permission-action operations are excluded,
|
||||
// matching the recovery scan's own population.
|
||||
@@ -1136,7 +1183,7 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
"SELECT tenant_id, session_id, operation_id FROM"
|
||||
+ " managed_agent_operation WHERE operation_kind <>"
|
||||
+ " 'ACTION_RESPONSE' AND (delivery_state = 'PENDING' OR (delivery_state = 'BLOCKED' AND (operation_kind = 'CLOSE'"
|
||||
+ " OR (operation_kind = 'DELETE' AND session_status_before IN ('CLOSED', 'ARCHIVED'))))) AND"
|
||||
+ " OR (operation_kind = 'DELETE' AND (session_status_before IN ('CLOSED', 'ARCHIVED') OR lifecycle_protocol_version = 1))))) AND"
|
||||
+ " available_at <= ? ORDER BY available_at LIMIT ?",
|
||||
operationTargetMapper,
|
||||
now,
|
||||
@@ -1160,6 +1207,7 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
public Optional<OperationRecord> claimOperation(String tenantId,
|
||||
String sessionId, String operationId, String owner,
|
||||
Duration leaseDuration) {
|
||||
WorkspaceLifecycleStore.lockPlacement(jdbc, tenantId);
|
||||
long now = lifecycleDatabaseTime();
|
||||
int updated = jdbc.update("UPDATE managed_agent_operation SET state ="
|
||||
+ " 'RUNNING', delivery_state = 'LEASED',"
|
||||
@@ -1168,10 +1216,13 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
+ " updated_at = ? WHERE tenant_id = ? AND"
|
||||
+ " session_id = ? AND operation_id = ? AND"
|
||||
+ " (((delivery_state = 'PENDING' OR (delivery_state = 'BLOCKED' AND (operation_kind = 'CLOSE'"
|
||||
+ " OR (operation_kind = 'DELETE' AND session_status_before IN ('CLOSED', 'ARCHIVED'))))) AND available_at <= ?)"
|
||||
+ " OR (operation_kind = 'DELETE' AND (session_status_before IN ('CLOSED', 'ARCHIVED') OR lifecycle_protocol_version = 1))))) AND available_at <= ?)"
|
||||
+ " OR (delivery_state = 'LEASED' AND lease_until < ?))",
|
||||
owner, Math.addExact(now, leaseDuration.toMillis()), now,
|
||||
tenantId, sessionId, operationId, now, now);
|
||||
if (updated == 1) {
|
||||
syncLifecycleClaim(tenantId, sessionId, operationId);
|
||||
}
|
||||
return updated == 1 ? findOperation(tenantId, sessionId, operationId)
|
||||
: Optional.empty();
|
||||
}
|
||||
@@ -1182,10 +1233,17 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
String operationId, String owner, long claimGeneration,
|
||||
boolean harnessConfirmed) {
|
||||
var target = findOperation(tenantId, sessionId, operationId).orElseThrow();
|
||||
if (target.kind() == OperationKind.DELETE) {
|
||||
if (target.lifecycleProtocolVersion() == 1) {
|
||||
WorkspaceLifecycleStore.lockPlacement(jdbc, tenantId);
|
||||
ToolPublicationRetentionStore.lockTenant(jdbc, tenantId);
|
||||
}
|
||||
if (target.kind() == OperationKind.DELETE && target.lifecycleProtocolVersion() != 1) {
|
||||
ToolPublicationRetentionStore.lockDeletion(jdbc, tenantId, sessionId);
|
||||
}
|
||||
SessionRecord session = requireSessionForUpdate(tenantId, sessionId);
|
||||
if (target.lifecycleProtocolVersion() == 1 && target.kind() == OperationKind.DELETE) {
|
||||
ToolPublicationRetentionStore.lockDeletion(jdbc, tenantId, sessionId);
|
||||
}
|
||||
OperationRecord operation = jdbc.query("SELECT * FROM"
|
||||
+ " managed_agent_operation WHERE tenant_id = ? AND"
|
||||
+ " session_id = ? AND operation_id = ? FOR UPDATE",
|
||||
@@ -1206,9 +1264,16 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
+ session.status() + " during its "
|
||||
+ operation.kind() + " operation");
|
||||
}
|
||||
if (operation.lifecycleProtocolVersion() == 1) {
|
||||
if (lifecycle == null) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_close_identity_unverified");
|
||||
}
|
||||
lifecycle.verifyCompletion(operation);
|
||||
}
|
||||
long now = lifecycleDatabaseTime();
|
||||
if (operation.kind() == OperationKind.DELETE) {
|
||||
if (session.workspace() != null && !List.of("CLOSED", "ARCHIVED").contains(operation.sessionStatusBefore())) {
|
||||
if (session.workspace() != null && !List.of("CLOSED", "ARCHIVED").contains(operation.sessionStatusBefore())
|
||||
&& operation.lifecycleProtocolVersion() != 1) {
|
||||
throw new IllegalStateException("Workspace deletion requires a closed Session");
|
||||
}
|
||||
ToolPublicationRetentionStore.retire(jdbc, tenantId, sessionId, operationId);
|
||||
@@ -1255,17 +1320,36 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
@Transactional
|
||||
public boolean renewLifecycleOperation(String tenantId, String sessionId, String operationId,
|
||||
String owner, long generation, Duration duration) {
|
||||
WorkspaceLifecycleStore.lockPlacement(jdbc, tenantId);
|
||||
long now = lifecycleDatabaseTime();
|
||||
return jdbc.update("UPDATE managed_agent_operation SET lease_until = ?, updated_at = ?"
|
||||
int updated = jdbc.update("UPDATE managed_agent_operation SET lease_until = ?, updated_at = ?"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND operation_id = ? AND delivery_state = 'LEASED'"
|
||||
+ " AND lease_owner = ? AND claim_generation = ? AND lease_until > ?",
|
||||
Math.addExact(now, duration.toMillis()), now, tenantId, sessionId, operationId, owner, generation, now) == 1;
|
||||
Math.addExact(now, duration.toMillis()), now, tenantId, sessionId, operationId, owner, generation, now);
|
||||
if (updated == 1) {
|
||||
syncLifecycleClaim(tenantId, sessionId, operationId);
|
||||
}
|
||||
return updated == 1;
|
||||
}
|
||||
|
||||
private void syncLifecycleClaim(String tenantId, String sessionId, String operationId) {
|
||||
var rows = jdbc.queryForList("SELECT lifecycle_protocol_version, claim_generation, lease_until FROM managed_agent_operation"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND operation_id = ?", tenantId, sessionId, operationId);
|
||||
if (!rows.isEmpty() && ((Number) rows.getFirst().get("lifecycle_protocol_version")).intValue() == 1) {
|
||||
var row = rows.getFirst();
|
||||
jdbc.update("UPDATE qwen_runtime_harness_drain SET claim_generation = ?, claim_lease_until = ?"
|
||||
+ " WHERE tenant_key = ? AND harness_key = ? AND tenant_id = ? AND harness_session_id = ?"
|
||||
+ " AND operation_id = ? AND phase = 'LIFECYCLE_ONLY'",
|
||||
row.get("claim_generation"), row.get("lease_until"), JdbcRuntimeBindingRepository.harnessDrainKey(tenantId),
|
||||
JdbcRuntimeBindingRepository.harnessDrainKey(sessionId), tenantId, sessionId, operationId);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
@Transactional
|
||||
public void blockLifecycleOperation(String tenantId, String sessionId, String operationId,
|
||||
String owner, long generation, String failureCode, long availableAt) {
|
||||
WorkspaceLifecycleStore.lockPlacement(jdbc, tenantId);
|
||||
long now = lifecycleDatabaseTime();
|
||||
jdbc.update("UPDATE managed_agent_operation SET state = 'RECOVERY_BLOCKED', delivery_state = 'BLOCKED',"
|
||||
+ " error_code = ?, available_at = ?, lease_owner = NULL, lease_until = NULL, updated_at = ?,"
|
||||
@@ -1274,6 +1358,10 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
+ " AND lease_owner = ? AND claim_generation = ? AND lease_until > ?",
|
||||
failureCode, Math.addExact(now, Math.max(0, availableAt - clock.millis())),
|
||||
now, tenantId, sessionId, operationId, owner, generation, now);
|
||||
jdbc.update("UPDATE qwen_runtime_harness_drain SET claim_lease_until = NULL WHERE tenant_key = ? AND harness_key = ? AND tenant_id = ?"
|
||||
+ " AND harness_session_id = ? AND operation_id = ? AND claim_generation = ?",
|
||||
JdbcRuntimeBindingRepository.harnessDrainKey(tenantId), JdbcRuntimeBindingRepository.harnessDrainKey(sessionId),
|
||||
tenantId, sessionId, operationId, generation);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -1281,6 +1369,7 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
public void retryOperation(String tenantId, String sessionId,
|
||||
String operationId, String owner, long claimGeneration,
|
||||
long availableAt) {
|
||||
WorkspaceLifecycleStore.lockPlacement(jdbc, tenantId);
|
||||
long delay = Math.max(0, availableAt - clock.millis());
|
||||
long now = lifecycleDatabaseTime();
|
||||
availableAt = Math.addExact(now, delay);
|
||||
@@ -1293,6 +1382,10 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
+ " AND lease_owner = ? AND claim_generation = ? AND lease_until > ?",
|
||||
availableAt, now, tenantId, sessionId,
|
||||
operationId, owner, claimGeneration, now);
|
||||
jdbc.update("UPDATE qwen_runtime_harness_drain SET claim_lease_until = NULL WHERE tenant_key = ? AND harness_key = ? AND tenant_id = ?"
|
||||
+ " AND harness_session_id = ? AND operation_id = ? AND claim_generation = ?",
|
||||
JdbcRuntimeBindingRepository.harnessDrainKey(tenantId), JdbcRuntimeBindingRepository.harnessDrainKey(sessionId),
|
||||
tenantId, sessionId, operationId, claimGeneration);
|
||||
}
|
||||
|
||||
public Admission replayCommand(String tenantId, String operation,
|
||||
|
||||
+101
@@ -307,6 +307,107 @@ public class ManagedExtensionRecordStore {
|
||||
return new ApplyResult(receipts, lastActivation);
|
||||
}
|
||||
|
||||
boolean hasNewLifecycleDispatch(String tenantId, String sessionId, byte[] bytes,
|
||||
Function<String, StoredResource> resources) {
|
||||
boolean dispatch = false;
|
||||
var revisions = new java.util.HashMap<String, JsonNode>();
|
||||
for (String line : new String(bytes, StandardCharsets.UTF_8).split("\n")) {
|
||||
JsonNode parsed = parse(line);
|
||||
if (parsed == null) {
|
||||
throw new ApiException(HttpStatus.BAD_REQUEST,
|
||||
ManagedSessionStoreModels.ERROR_INVALID_REQUEST,
|
||||
"Record line is not a JSON object the Session authority can read.");
|
||||
}
|
||||
JsonNode event = parsed.path("managedSession");
|
||||
JsonNode payload = event.path("payload");
|
||||
if ("input.accepted".equals(event.path("kind").asText())
|
||||
|| "tool.intent".equals(event.path("kind").asText())) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_admission_closed");
|
||||
}
|
||||
if ("model.attempt".equals(event.path("kind").asText())
|
||||
&& "started".equals(payload.path("state").asText())) {
|
||||
dispatch = true;
|
||||
}
|
||||
if (!"domain.committed".equals(event.path("kind").asText())) {
|
||||
continue;
|
||||
}
|
||||
String domain = payload.path("domain").asText();
|
||||
if (!List.of("hook_execution", "hook_registration").contains(domain)) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_admission_closed");
|
||||
}
|
||||
JsonNode next = readBody(resources.apply(payload.path("recordRef").path("resourceId").asText()));
|
||||
JsonNode previous = previousLifecycleRecord(tenantId, sessionId, domain, next, revisions);
|
||||
if (requiresLifecycleDispatch(previous, next)) {
|
||||
dispatch = true;
|
||||
}
|
||||
}
|
||||
return dispatch;
|
||||
}
|
||||
|
||||
void requireLifecycleSettlement(String tenantId, String sessionId, byte[] bytes,
|
||||
Function<String, StoredResource> resources) {
|
||||
var revisions = new java.util.HashMap<String, JsonNode>();
|
||||
for (String line : new String(bytes, StandardCharsets.UTF_8).split("\n")) {
|
||||
JsonNode parsed = parse(line);
|
||||
if (parsed == null) {
|
||||
throw new ApiException(HttpStatus.BAD_REQUEST,
|
||||
ManagedSessionStoreModels.ERROR_INVALID_REQUEST,
|
||||
"Record line is not a JSON object the Session authority can read.");
|
||||
}
|
||||
JsonNode event = parsed.path("managedSession");
|
||||
String kind = event.path("kind").asText();
|
||||
JsonNode payload = event.path("payload");
|
||||
if ("input.accepted".equals(kind) || "tool.intent".equals(kind)
|
||||
|| "model.attempt".equals(kind) && "started".equals(payload.path("state").asText())) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_admission_closed");
|
||||
}
|
||||
if (!"domain.committed".equals(kind)) {
|
||||
continue;
|
||||
}
|
||||
String domain = payload.path("domain").asText();
|
||||
if (!List.of("hook_execution", "hook_registration").contains(domain)) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_admission_closed");
|
||||
}
|
||||
JsonNode next = readBody(resources.apply(payload.path("recordRef").path("resourceId").asText()));
|
||||
JsonNode previous = previousLifecycleRecord(tenantId, sessionId, domain, next, revisions);
|
||||
if (previous == null || requiresLifecycleDispatch(previous, next)) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_admission_closed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private JsonNode previousLifecycleRecord(String tenantId, String sessionId, String domain, JsonNode next,
|
||||
java.util.Map<String, JsonNode> revisions) {
|
||||
String id = next.path("hook_execution".equals(domain) ? "hookExecutionId" : "registrationId").asText();
|
||||
String key = domain + ":" + id;
|
||||
JsonNode previous = revisions.get(key);
|
||||
if (previous == null) {
|
||||
previous = jdbc.queryForList("SELECT record_resource_id FROM qwen_managed_session_extension_record"
|
||||
+ " WHERE session_scope_key = ? AND record_key = ? AND tenant_id = ? AND session_id = ? AND domain = ? AND record_id = ?",
|
||||
String.class, ManagedSessionStore.sessionScopeKey(tenantId, sessionId),
|
||||
ManagedExtensionProjection.recordKey(sessionId, domain, id), tenantId, sessionId, domain, id).stream().map(resource -> {
|
||||
JsonNode record = readBody(readResource(tenantId, sessionId, resource));
|
||||
ManagedExtensionProjection.RECORD_BODIES.get(domain).require().accept(record);
|
||||
return record;
|
||||
}).findFirst().orElse(null);
|
||||
}
|
||||
revisions.put(key, next);
|
||||
return previous;
|
||||
}
|
||||
|
||||
private boolean requiresLifecycleDispatch(JsonNode previous, JsonNode next) {
|
||||
if (previous == null) {
|
||||
return "dispatch_started".equals(next.path("run").path("execution").asText());
|
||||
}
|
||||
JsonNode before = previous.path("run");
|
||||
JsonNode after = next.path("run");
|
||||
String execution = before.path("execution").asText();
|
||||
if (!List.of("", "intent").contains(execution) && !before.path("runtime").equals(after.path("runtime"))) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_admission_closed");
|
||||
}
|
||||
return "intent".equals(execution) && !List.of("intent", "not_started_proven").contains(after.path("execution").asText());
|
||||
}
|
||||
|
||||
public TaskPage listTasks(String tenantId, String sessionId,
|
||||
Long beforeCreatedAt, String beforeTaskId, int limit) {
|
||||
List<Object> arguments = new ArrayList<>();
|
||||
|
||||
+145
-1
@@ -17,6 +17,7 @@ import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.Stored
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.StoredTransaction;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.TransactionPage;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.WriterGrant;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.io.IOException;
|
||||
@@ -130,6 +131,95 @@ public class ManagedSessionStore {
|
||||
new org.springframework.jdbc.datasource.DataSourceTransactionManager(jdbc.getDataSource()));
|
||||
}
|
||||
|
||||
private WorkspaceExecutionStore lifecycleExecution;
|
||||
private AgentStateStore lifecycleSessions;
|
||||
|
||||
@Autowired(required = false)
|
||||
public void setLifecycleExecution(WorkspaceExecutionStore execution, AgentStateStore sessions) {
|
||||
this.lifecycleExecution = execution;
|
||||
this.lifecycleSessions = sessions;
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public void authorizeLifecycle(String tenantId, String sessionId, String writerToken,
|
||||
ManagedSessionStoreModels.AuthorizeLifecycleRequest request,
|
||||
com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority authority) {
|
||||
validateScope(tenantId, request.workspaceId(), sessionId);
|
||||
requireCredential(tenantId, request.workspaceId(), sessionId,
|
||||
writerToken);
|
||||
WorkspaceLifecycleStore.lockPlacement(jdbc, tenantId);
|
||||
ToolPublicationRetentionStore.lockTenant(jdbc, tenantId);
|
||||
jdbc.queryForList("SELECT session_id FROM managed_agent_session WHERE tenant_id = ? AND session_id = ? FOR UPDATE", tenantId, sessionId);
|
||||
WorkspaceLifecycleStore.requireClaim(jdbc, tenantId, sessionId, authority, true);
|
||||
if (request.kind() != null) {
|
||||
String kind = jdbc.queryForObject("SELECT operation_kind FROM managed_agent_operation WHERE tenant_id = ?"
|
||||
+ " AND session_id = ? AND operation_id = ?", String.class, tenantId, sessionId, authority.operationId());
|
||||
if (!kind.toLowerCase(java.util.Locale.ROOT).equals(request.kind())) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_operation_conflict");
|
||||
}
|
||||
} else if (!"DRAINING".equals(jdbc.queryForObject("SELECT phase FROM qwen_runtime_harness_drain"
|
||||
+ " WHERE tenant_key = ? AND harness_key = ? AND tenant_id = ? AND harness_session_id = ?", String.class,
|
||||
JdbcRuntimeBindingRepository.harnessDrainKey(tenantId), JdbcRuntimeBindingRepository.harnessDrainKey(sessionId),
|
||||
tenantId, sessionId))) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_hooks_unsettled");
|
||||
}
|
||||
HeadRow head = requireHeadForUpdate(tenantId, sessionId);
|
||||
requireHeadScope(head, tenantId, request.workspaceId(), sessionId);
|
||||
if (request.kind() != null) {
|
||||
requireWriter(head, request.writerId(), request.writerGeneration(), writerToken, databaseNow(), true);
|
||||
} else if (!List.of("ACTIVE", "SEALED").contains(head.state())
|
||||
|| request.writerGeneration() != head.writerGeneration()
|
||||
|| !request.writerId().equals(head.writerId())
|
||||
|| !secureEquals(tokenHash(writerToken), head.leaseTokenHash())) {
|
||||
throw writerConflict();
|
||||
}
|
||||
}
|
||||
|
||||
private void checkLifecycleWriter(String tenantId, String sessionId,
|
||||
com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority authority) {
|
||||
WorkspaceLifecycleStore.lockPlacement(jdbc, tenantId);
|
||||
ToolPublicationRetentionStore.lockTenant(jdbc, tenantId);
|
||||
if (authority != null) {
|
||||
WorkspaceLifecycleStore.requireClaim(jdbc, tenantId, sessionId, authority, true);
|
||||
}
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public void authorizeOrdinary(String tenantId, String sessionId, String writerToken,
|
||||
ManagedSessionStoreModels.AuthorizeLifecycleRequest request) {
|
||||
validateScope(tenantId, request.workspaceId(), sessionId);
|
||||
requireCredential(tenantId, request.workspaceId(), sessionId,
|
||||
writerToken);
|
||||
WorkspaceLifecycleStore.lockPlacement(jdbc, tenantId);
|
||||
ToolPublicationRetentionStore.lockTenant(jdbc, tenantId);
|
||||
if (!"legacy-close".equals(request.kind()) || !WorkspaceLifecycleStore.legacyClose(jdbc, tenantId, sessionId)) {
|
||||
requireOrdinaryAdmission(tenantId, sessionId);
|
||||
}
|
||||
HeadRow head = requireHeadForUpdate(tenantId, sessionId);
|
||||
requireHeadScope(head, tenantId, request.workspaceId(), sessionId);
|
||||
requireWriter(head, request.writerId(), request.writerGeneration(), writerToken, databaseNow(), true);
|
||||
}
|
||||
|
||||
private void requireOrdinaryAdmission(String tenantId, String sessionId) {
|
||||
if (lifecycleFenced(tenantId, sessionId)) {
|
||||
throw conflict("managed_session_lifecycle_active", "Ordinary execution admission is closed.");
|
||||
}
|
||||
}
|
||||
|
||||
private boolean lifecycleFenced(String tenantId, String sessionId) {
|
||||
return !jdbc.queryForList("SELECT phase FROM qwen_runtime_harness_drain WHERE tenant_key = ? AND harness_key = ?"
|
||||
+ " AND tenant_id = ? AND harness_session_id = ? FOR UPDATE",
|
||||
JdbcRuntimeBindingRepository.harnessDrainKey(tenantId), JdbcRuntimeBindingRepository.harnessDrainKey(sessionId),
|
||||
tenantId, sessionId).isEmpty();
|
||||
}
|
||||
|
||||
private boolean lifecycleProtocolFenced(String tenantId, String sessionId) {
|
||||
return !jdbc.queryForList("SELECT operation_id FROM qwen_runtime_harness_drain WHERE tenant_key = ? AND harness_key = ?"
|
||||
+ " AND tenant_id = ? AND harness_session_id = ? AND operation_id IS NOT NULL FOR UPDATE",
|
||||
JdbcRuntimeBindingRepository.harnessDrainKey(tenantId), JdbcRuntimeBindingRepository.harnessDrainKey(sessionId),
|
||||
tenantId, sessionId).isEmpty();
|
||||
}
|
||||
|
||||
boolean usesDataSource(javax.sql.DataSource source) {
|
||||
return source != null && jdbc.getDataSource() == source;
|
||||
}
|
||||
@@ -182,9 +272,21 @@ public class ManagedSessionStore {
|
||||
@Transactional
|
||||
public WriterGrant acquireWriter(String tenantId, String sessionId,
|
||||
String writerToken, AcquireWriterRequest request) {
|
||||
return acquireWriter(tenantId, sessionId, writerToken, request, null);
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public WriterGrant acquireWriter(String tenantId, String sessionId, String writerToken,
|
||||
AcquireWriterRequest request, com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority authority) {
|
||||
validateScope(tenantId, request.workspaceId(), sessionId);
|
||||
requireCredential(tenantId, request.workspaceId(), sessionId,
|
||||
writerToken);
|
||||
checkLifecycleWriter(tenantId, sessionId, authority);
|
||||
if (authority != null) {
|
||||
WorkspaceLifecycleStore.requireClaim(jdbc, tenantId, sessionId, authority, false);
|
||||
} else {
|
||||
requireOrdinaryAdmission(tenantId, sessionId);
|
||||
}
|
||||
validateStableId(request.writerId(), "writerId");
|
||||
validateLeaseMillis(request.leaseMillis());
|
||||
ToolPublicationRetentionStore.lockTenant(jdbc, tenantId);
|
||||
@@ -193,7 +295,7 @@ public class ManagedSessionStore {
|
||||
List<String> closed = jdbc.query("SELECT status FROM managed_agent_session"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND workspace_id IS NOT NULL FOR UPDATE",
|
||||
(row, index) -> row.getString("status"), tenantId, sessionId);
|
||||
if (!closed.isEmpty() && !"ACTIVE".equals(closed.getFirst())) {
|
||||
if (authority == null && !closed.isEmpty() && !"ACTIVE".equals(closed.getFirst())) {
|
||||
throw conflict("managed_session_not_writable", "The Session is closing or closed.");
|
||||
}
|
||||
String tokenHash = tokenHash(writerToken);
|
||||
@@ -277,9 +379,16 @@ public class ManagedSessionStore {
|
||||
@Transactional
|
||||
public WriterGrant renewWriter(String tenantId, String sessionId,
|
||||
String writerToken, RenewWriterRequest request) {
|
||||
return renewWriter(tenantId, sessionId, writerToken, request, null);
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public WriterGrant renewWriter(String tenantId, String sessionId, String writerToken,
|
||||
RenewWriterRequest request, com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority authority) {
|
||||
validateScope(tenantId, request.workspaceId(), sessionId);
|
||||
requireCredential(tenantId, request.workspaceId(), sessionId,
|
||||
writerToken);
|
||||
checkLifecycleWriter(tenantId, sessionId, authority);
|
||||
validateStableId(request.writerId(), "writerId");
|
||||
validateCounter(request.writerGeneration(), "writerGeneration", 1);
|
||||
validateLeaseMillis(request.leaseMillis());
|
||||
@@ -303,9 +412,16 @@ public class ManagedSessionStore {
|
||||
@Transactional
|
||||
public SealReceipt sealWriter(String tenantId, String sessionId,
|
||||
String writerToken, SealWriterRequest request) {
|
||||
return sealWriter(tenantId, sessionId, writerToken, request, null);
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public SealReceipt sealWriter(String tenantId, String sessionId, String writerToken,
|
||||
SealWriterRequest request, com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority authority) {
|
||||
validateScope(tenantId, request.workspaceId(), sessionId);
|
||||
requireCredential(tenantId, request.workspaceId(), sessionId,
|
||||
writerToken);
|
||||
checkLifecycleWriter(tenantId, sessionId, authority);
|
||||
validateStableId(request.writerId(), "writerId");
|
||||
validateCounter(request.writerGeneration(), "writerGeneration", 1);
|
||||
HeadRow head = requireHeadForUpdate(tenantId, sessionId);
|
||||
@@ -373,9 +489,16 @@ public class ManagedSessionStore {
|
||||
@Transactional
|
||||
public CommitReceipt commit(String tenantId, String sessionId,
|
||||
String writerToken, CommitTransactionRequest request) {
|
||||
return commit(tenantId, sessionId, writerToken, request, null);
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public CommitReceipt commit(String tenantId, String sessionId, String writerToken,
|
||||
CommitTransactionRequest request, com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority authority) {
|
||||
validateScope(tenantId, request.workspaceId(), sessionId);
|
||||
requireCredential(tenantId, request.workspaceId(), sessionId,
|
||||
writerToken);
|
||||
checkLifecycleWriter(tenantId, sessionId, authority);
|
||||
validateStableId(request.writerId(), "writerId");
|
||||
ValidatedCommit validated = validateCommit(request);
|
||||
HeadRow head = requireHeadForUpdate(tenantId, sessionId);
|
||||
@@ -397,6 +520,27 @@ public class ManagedSessionStore {
|
||||
String scopeKey = sessionScopeKey(tenantId, sessionId);
|
||||
commitResources(scopeKey, tenantId, sessionId, request, revision,
|
||||
now, validated.resources());
|
||||
if (authority != null && extensionRecords.hasNewLifecycleDispatch(tenantId, sessionId, validated.recordBytes(),
|
||||
resourceId -> storedResource(scopeKey, tenantId, request.workspaceId(), sessionId, resourceId))) {
|
||||
if (lifecycleExecution == null || lifecycleSessions == null) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_protocol_unavailable");
|
||||
}
|
||||
try {
|
||||
lifecycleExecution.authorizeLifecycle(lifecycleSessions.requireSession(tenantId, sessionId), authority);
|
||||
} catch (com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException refusal) {
|
||||
if (refusal.getStatusCode() == 409 && "workspace_unavailable".equals(refusal.getCode())) {
|
||||
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_authorization_revoked");
|
||||
}
|
||||
if (refusal.getStatusCode() == 409) {
|
||||
throw new ApiException(HttpStatus.CONFLICT, refusal.getCode(), refusal.getMessage());
|
||||
}
|
||||
throw refusal;
|
||||
}
|
||||
}
|
||||
if (authority == null && lifecycleProtocolFenced(tenantId, sessionId)) {
|
||||
extensionRecords.requireLifecycleSettlement(tenantId, sessionId, validated.recordBytes(),
|
||||
resourceId -> storedResource(scopeKey, tenantId, request.workspaceId(), sessionId, resourceId));
|
||||
}
|
||||
var applied = extensionRecords.apply(tenantId, request.workspaceId(), sessionId,
|
||||
request.firstSequence(), request.eventCount(),
|
||||
validated.recordBytes(), resourceId -> storedResource(
|
||||
|
||||
+10
@@ -60,6 +60,16 @@ public final class ManagedSessionStoreModels {
|
||||
private ManagedSessionStoreModels() {
|
||||
}
|
||||
|
||||
public record AuthorizeLifecycleRequest(
|
||||
@NotBlank @Size(max = 512) String workspaceId,
|
||||
@NotBlank @Size(max = 512) String writerId,
|
||||
@Min(1) @Max(MAX_SAFE_COUNTER) long writerGeneration,
|
||||
String kind) {
|
||||
public AuthorizeLifecycleRequest(String workspaceId, String writerId, long writerGeneration) {
|
||||
this(workspaceId, writerId, writerGeneration, null);
|
||||
}
|
||||
}
|
||||
|
||||
public record AcquireWriterRequest(
|
||||
@NotBlank @Size(max = 512) String workspaceId,
|
||||
@NotBlank @Size(max = 512) String writerId,
|
||||
|
||||
+27
-2
@@ -105,12 +105,37 @@ public final class StoreModels {
|
||||
String sessionStatusBefore, String receiptId, String leaseOwner,
|
||||
long claimGeneration, int attemptCount, String targetCwdRelative,
|
||||
Long expectedContextRevision, Long resultContextRevision,
|
||||
String failureCode) {
|
||||
String failureCode, int lifecycleProtocolVersion) {
|
||||
public OperationRecord(String tenantId, String sessionId, String operationId, OperationKind kind,
|
||||
String requestDigest, String state, String admissionStage, String deliveryState, String sessionStatusBefore,
|
||||
String receiptId, String leaseOwner, long claimGeneration, int attemptCount, String targetCwdRelative,
|
||||
Long expectedContextRevision, Long resultContextRevision, String failureCode) {
|
||||
this(tenantId, sessionId, operationId, kind, requestDigest, state, admissionStage, deliveryState,
|
||||
sessionStatusBefore, receiptId, leaseOwner, claimGeneration, attemptCount, targetCwdRelative,
|
||||
expectedContextRevision, resultContextRevision, failureCode, 0);
|
||||
}
|
||||
|
||||
public OperationRecord(String tenantId, String sessionId, String operationId, OperationKind kind,
|
||||
String requestDigest, String state, String admissionStage, String deliveryState, String sessionStatusBefore,
|
||||
String receiptId, String leaseOwner, long claimGeneration, int attemptCount, String failureCode,
|
||||
int lifecycleProtocolVersion) {
|
||||
this(tenantId, sessionId, operationId, kind, requestDigest, state, admissionStage, deliveryState,
|
||||
sessionStatusBefore, receiptId, leaseOwner, claimGeneration, attemptCount, null, null, null,
|
||||
failureCode, lifecycleProtocolVersion);
|
||||
}
|
||||
|
||||
public OperationRecord(String tenantId, String sessionId, String operationId, OperationKind kind,
|
||||
String requestDigest, String state, String admissionStage, String deliveryState, String sessionStatusBefore,
|
||||
String receiptId, String leaseOwner, long claimGeneration, int attemptCount, String failureCode) {
|
||||
this(tenantId, sessionId, operationId, kind, requestDigest, state, admissionStage, deliveryState,
|
||||
sessionStatusBefore, receiptId, leaseOwner, claimGeneration, attemptCount, failureCode, 0);
|
||||
}
|
||||
|
||||
public OperationRecord(String tenantId, String sessionId, String operationId, OperationKind kind,
|
||||
String requestDigest, String state, String admissionStage, String deliveryState,
|
||||
String sessionStatusBefore, String receiptId, String leaseOwner, long claimGeneration, int attemptCount) {
|
||||
this(tenantId, sessionId, operationId, kind, requestDigest, state, admissionStage, deliveryState,
|
||||
sessionStatusBefore, receiptId, leaseOwner, claimGeneration, attemptCount, null, null, null, null);
|
||||
sessionStatusBefore, receiptId, leaseOwner, claimGeneration, attemptCount, null, 0);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+1
@@ -351,6 +351,7 @@ public final class ToolPublicationAdmissionStore {
|
||||
"Original manifest resource conflicts");
|
||||
}
|
||||
return transactions.execute(status -> {
|
||||
WorkspaceLifecycleStore.lockPlacement(jdbc, text(key, "tenantId"));
|
||||
data.lockOriginalSettledResult(key, publicationId, finished);
|
||||
lockTenant(key);
|
||||
sessions.lockPublicationWriter(text(key, "tenantId"), text(key, "workspaceId"),
|
||||
|
||||
+36
-2
@@ -1,5 +1,6 @@
|
||||
package com.alibaba.qwen.code.managedagent.store;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.api.ApiException;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.SessionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
@@ -50,6 +51,28 @@ public class WorkspaceExecutionStore {
|
||||
}
|
||||
}
|
||||
|
||||
public void authorizeLifecycle(SessionRecord session, com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority authority) {
|
||||
try {
|
||||
WorkspaceLifecycleStore.requireClaim(jdbc, session.tenantId(), session.sessionId(), authority, false);
|
||||
} catch (ApiException error) {
|
||||
throw new RuntimeBrokerException(error.getStatus().value(), error.getCode(), error.getMessage(), false, error);
|
||||
}
|
||||
authorizePassiveAttachment(session, authority);
|
||||
if (storageGuard != null) {
|
||||
storageGuard.verify(session.workspace());
|
||||
}
|
||||
}
|
||||
|
||||
public void authorizeLegacyClose(SessionRecord session) {
|
||||
if (!WorkspaceLifecycleStore.legacyClose(jdbc, session.tenantId(), session.sessionId())) {
|
||||
throw unavailable();
|
||||
}
|
||||
authorizePassiveAttachment(session, null, true);
|
||||
if (storageGuard != null) {
|
||||
storageGuard.verify(session.workspace());
|
||||
}
|
||||
}
|
||||
|
||||
// The mount guard of an execution authority, without the Session-level
|
||||
// checks; the W2 settlement probe must not fail a Session it only
|
||||
// reads. It uses the guard's probe-only entry: momentary I/O failures
|
||||
@@ -62,8 +85,19 @@ public class WorkspaceExecutionStore {
|
||||
}
|
||||
|
||||
public void authorizePassiveAttachment(SessionRecord session) {
|
||||
authorizePassiveAttachment(session, null);
|
||||
}
|
||||
|
||||
private void authorizePassiveAttachment(SessionRecord session, com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority authority) {
|
||||
authorizePassiveAttachment(session, authority, false);
|
||||
}
|
||||
|
||||
private void authorizePassiveAttachment(SessionRecord session, com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority authority,
|
||||
boolean legacyClose) {
|
||||
ContextBinding binding = session.workspace();
|
||||
if (binding == null || !"ACTIVE".equals(session.status())
|
||||
String expectedStatus = authority == null && !legacyClose ? "ACTIVE" : session.status();
|
||||
if (binding == null || !(legacyClose ? "CLOSING".equals(session.status()) : authority == null ? "ACTIVE".equals(session.status())
|
||||
: java.util.List.of("CLOSING", "DELETING").contains(session.status()))
|
||||
|| session.deletedAt() != null || !"qwen-code".equals(session.agentId())
|
||||
|| !session.tenantId().equals(binding.getTenantId())
|
||||
|| !WorkspaceExecutionProfile.CONTEXT_CONFIG_REF.equals(
|
||||
@@ -96,7 +130,7 @@ public class WorkspaceExecutionStore {
|
||||
(row, index) -> session.tenantId().equals(row.getString("tenant_id"))
|
||||
&& session.sessionId().equals(row.getString("session_id"))
|
||||
&& "qwen-code".equals(row.getString("session_agent"))
|
||||
&& "ACTIVE".equals(row.getString("session_status"))
|
||||
&& expectedStatus.equals(row.getString("session_status"))
|
||||
&& row.getObject("session_deleted_at") == null
|
||||
&& binding.getWorkspaceId().equals(row.getString("session_workspace"))
|
||||
&& binding.getWorkspaceGeneration() == row.getLong("session_generation")
|
||||
|
||||
+363
@@ -0,0 +1,363 @@
|
||||
package com.alibaba.qwen.code.managedagent.store;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.api.ApiException;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
import org.springframework.beans.factory.ObjectProvider;
|
||||
import org.springframework.jdbc.core.ConnectionCallback;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.stereotype.Repository;
|
||||
import org.springframework.transaction.annotation.Transactional;
|
||||
|
||||
@Repository
|
||||
public class WorkspaceLifecycleStore {
|
||||
private static final ObjectMapper OCCURRENCE_JSON = new ObjectMapper();
|
||||
|
||||
public static boolean legacyClose(JdbcTemplate jdbc, String tenant, String session) {
|
||||
long now = jdbc.queryForObject("SELECT UNIX_TIMESTAMP(), EXTRACT(MICROSECOND FROM CURRENT_TIMESTAMP(6))",
|
||||
(row, index) -> row.getLong(1) * 1000 + row.getLong(2) / 1000);
|
||||
return !jdbc.queryForList("SELECT o.operation_id FROM managed_agent_operation o"
|
||||
+ " JOIN managed_agent_session s ON s.tenant_id = o.tenant_id AND s.session_id = o.session_id"
|
||||
+ " JOIN qwen_runtime_harness_drain f ON f.tenant_id = o.tenant_id AND f.harness_session_id = o.session_id"
|
||||
+ " WHERE o.tenant_id = ? AND o.session_id = ? AND o.operation_kind = 'CLOSE'"
|
||||
+ " AND o.lifecycle_protocol_version = 0 AND o.delivery_state = 'LEASED'"
|
||||
+ " AND o.lease_until > ? AND s.status = 'CLOSING'"
|
||||
+ " AND f.tenant_key = ? AND f.harness_key = ?"
|
||||
+ " AND f.phase = 'DRAINING' AND f.operation_id IS NULL FOR UPDATE", tenant, session, now,
|
||||
JdbcRuntimeBindingRepository.harnessDrainKey(tenant), JdbcRuntimeBindingRepository.harnessDrainKey(session)).isEmpty();
|
||||
}
|
||||
private final JdbcTemplate jdbc;
|
||||
private final ObjectMapper json;
|
||||
private final ManagedExtensionRecordStore records;
|
||||
private final ObjectProvider<RuntimeBindingRepository> bindings;
|
||||
|
||||
public WorkspaceLifecycleStore(JdbcTemplate jdbc, ObjectMapper json, ObjectProvider<RuntimeBindingRepository> bindings) {
|
||||
this.jdbc = jdbc;
|
||||
this.json = json;
|
||||
this.records = new ManagedExtensionRecordStore(jdbc);
|
||||
this.bindings = bindings;
|
||||
}
|
||||
|
||||
public static void lockPlacement(JdbcTemplate jdbc, String tenant) {
|
||||
jdbc.execute((ConnectionCallback<Void>) connection -> {
|
||||
JdbcRuntimeBindingRepository.lockPlacementDomain(connection, tenant);
|
||||
return null;
|
||||
});
|
||||
}
|
||||
|
||||
public static void requireClaim(JdbcTemplate jdbc, String tenant, String session,
|
||||
RuntimeLifecycleAuthority authority, boolean drainingAllowed) {
|
||||
if (authority == null) {
|
||||
throw blocked("workspace_lifecycle_claim_fenced");
|
||||
}
|
||||
var rows = jdbc.queryForList("SELECT o.claim_generation, o.lease_until, o.delivery_state, o.lifecycle_protocol_version,"
|
||||
+ " s.status, f.phase, f.operation_id FROM managed_agent_operation o"
|
||||
+ " JOIN managed_agent_session s ON s.tenant_id = o.tenant_id AND s.session_id = o.session_id"
|
||||
+ " JOIN qwen_runtime_harness_drain f ON f.tenant_id = s.tenant_id AND f.harness_session_id = s.session_id"
|
||||
+ " WHERE o.tenant_id = ? AND o.session_id = ? AND o.operation_id = ?"
|
||||
+ " AND f.tenant_key = ? AND f.harness_key = ? FOR UPDATE", tenant, session, authority.operationId(),
|
||||
JdbcRuntimeBindingRepository.harnessDrainKey(tenant), JdbcRuntimeBindingRepository.harnessDrainKey(session));
|
||||
long now = jdbc.queryForObject("SELECT UNIX_TIMESTAMP(), EXTRACT(MICROSECOND FROM CURRENT_TIMESTAMP(6))",
|
||||
(row, index) -> row.getLong(1) * 1000 + row.getLong(2) / 1000);
|
||||
if (rows.size() != 1) {
|
||||
throw blocked("workspace_lifecycle_claim_fenced");
|
||||
}
|
||||
var row = rows.getFirst();
|
||||
if (((Number) row.get("claim_generation")).longValue() != authority.claimGeneration()
|
||||
|| !(row.get("lease_until") instanceof Number until) || until.longValue() <= now
|
||||
|| !"LEASED".equals(row.get("delivery_state"))
|
||||
|| ((Number) row.get("lifecycle_protocol_version")).intValue() != 1
|
||||
|| !List.of("CLOSING", "DELETING").contains(row.get("status"))
|
||||
|| !authority.operationId().equals(row.get("operation_id"))
|
||||
|| !"LIFECYCLE_ONLY".equals(row.get("phase")) && !(drainingAllowed && "DRAINING".equals(row.get("phase")))) {
|
||||
throw blocked("workspace_lifecycle_claim_fenced");
|
||||
}
|
||||
}
|
||||
|
||||
public static void requireIdleJournal(JdbcTemplate jdbc, ObjectMapper json, String tenant, String session) {
|
||||
var heads = jdbc.queryForList("SELECT compacted_through_revision FROM qwen_managed_session_journal_head"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? FOR UPDATE", tenant, session);
|
||||
if (heads.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
if (((Number) heads.getFirst().get("compacted_through_revision")).longValue() != 0) {
|
||||
throw blocked("workspace_lifecycle_journal_unverified");
|
||||
}
|
||||
var pending = new java.util.HashSet<String>();
|
||||
jdbc.query("SELECT record_bytes FROM qwen_managed_session_journal_tx WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " ORDER BY journal_revision FOR UPDATE", row -> {
|
||||
for (String line : new String(row.getBytes(1), StandardCharsets.UTF_8).split("\n")) {
|
||||
try {
|
||||
JsonNode event = json.readTree(line).path("managedSession");
|
||||
String turn = event.path("payload").path("turnId").asText();
|
||||
if ("input.accepted".equals(event.path("kind").asText())) {
|
||||
pending.add(turn);
|
||||
} else if ("turn.settled".equals(event.path("kind").asText())) {
|
||||
pending.remove(turn);
|
||||
}
|
||||
} catch (java.io.IOException error) {
|
||||
throw blocked("workspace_lifecycle_journal_unverified");
|
||||
}
|
||||
}
|
||||
}, tenant, session);
|
||||
if (!pending.isEmpty()) {
|
||||
throw new ApiException(HttpStatus.CONFLICT, "turn_active", "The Session has an active Turn.");
|
||||
}
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public JsonNode recoverEffects(OperationRecord operation) {
|
||||
lock(operation);
|
||||
requireClaim(jdbc, operation.tenantId(), operation.sessionId(), authority(operation), true);
|
||||
String saved = jdbc.queryForObject("SELECT lifecycle_effects_receipt_json FROM managed_agent_operation"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND operation_id = ? FOR UPDATE", String.class,
|
||||
operation.tenantId(), operation.sessionId(), operation.operationId());
|
||||
if (saved != null) {
|
||||
return parse(saved);
|
||||
}
|
||||
JsonNode header = header(operation.tenantId(), operation.sessionId());
|
||||
var receipt = envelope(operation);
|
||||
var effects = receipt.putArray("effects");
|
||||
if (header == null) {
|
||||
if (!neverInitialized(operation)) {
|
||||
return null;
|
||||
}
|
||||
receipt.put("neverInitialized", true);
|
||||
} else {
|
||||
JsonNode definitionRef = header.path("definitionRef");
|
||||
JsonNode definition = records.readRecordResource(operation.tenantId(), operation.sessionId(), definitionRef);
|
||||
receipt.set("definitionRef", definitionRef);
|
||||
if (definition.has("hookCatalog")) {
|
||||
for (String event : events(operation)) {
|
||||
String resource = latestHookExecutionResource(operation, occurrence(event, operation.operationId()));
|
||||
if (resource == null) {
|
||||
return null;
|
||||
}
|
||||
JsonNode ref = resourceRef(operation, resource);
|
||||
JsonNode marker = records.readRecordResource(operation.tenantId(), operation.sessionId(), ref);
|
||||
if (!marker.hasNonNull("resultRef")) {
|
||||
return null;
|
||||
}
|
||||
effects.addObject().put("event", event).set("recordRef", ref);
|
||||
}
|
||||
}
|
||||
}
|
||||
saveLocked(operation, receipt);
|
||||
return receipt;
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public void saveEffects(OperationRecord operation, JsonNode receipt) {
|
||||
lock(operation);
|
||||
requireClaim(jdbc, operation.tenantId(), operation.sessionId(), authority(operation), false);
|
||||
saveLocked(operation, receipt);
|
||||
}
|
||||
|
||||
private void saveLocked(OperationRecord operation, JsonNode receipt) {
|
||||
verifyEffects(operation, receipt);
|
||||
jdbc.update("UPDATE managed_agent_operation SET lifecycle_effects_receipt_json = ?"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND operation_id = ? AND claim_generation = ?",
|
||||
receipt.toString(), operation.tenantId(), operation.sessionId(), operation.operationId(), operation.claimGeneration());
|
||||
jdbc.update("UPDATE qwen_runtime_harness_drain SET phase = 'DRAINING', claim_lease_until = NULL"
|
||||
+ " WHERE tenant_key = ? AND harness_key = ? AND tenant_id = ? AND harness_session_id = ? AND operation_id = ?",
|
||||
JdbcRuntimeBindingRepository.harnessDrainKey(operation.tenantId()), JdbcRuntimeBindingRepository.harnessDrainKey(operation.sessionId()),
|
||||
operation.tenantId(), operation.sessionId(), operation.operationId());
|
||||
}
|
||||
|
||||
public void verifyCompletion(OperationRecord operation) {
|
||||
String saved = jdbc.queryForObject("SELECT lifecycle_effects_receipt_json FROM managed_agent_operation"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND operation_id = ? FOR UPDATE", String.class,
|
||||
operation.tenantId(), operation.sessionId(), operation.operationId());
|
||||
if (saved == null) {
|
||||
throw blocked("workspace_lifecycle_hooks_unsettled");
|
||||
}
|
||||
verifyEffects(operation, parse(saved));
|
||||
var writers = jdbc.queryForList("SELECT state, CASE WHEN writer_lease_until > CURRENT_TIMESTAMP(6) THEN 1 ELSE 0 END AS live_writer"
|
||||
+ " FROM qwen_managed_session_journal_head WHERE tenant_id = ? AND session_id = ? FOR UPDATE",
|
||||
operation.tenantId(), operation.sessionId());
|
||||
if (!writers.isEmpty() && "ACTIVE".equals(writers.getFirst().get("state"))
|
||||
&& ((Number) writers.getFirst().get("live_writer")).intValue() != 0) {
|
||||
throw blocked("workspace_lifecycle_writer_active");
|
||||
}
|
||||
RuntimeBindingRepository repository = bindings.getIfAvailable();
|
||||
if (repository == null) {
|
||||
throw blocked("workspace_close_identity_unverified");
|
||||
}
|
||||
jdbc.execute((ConnectionCallback<Void>) connection -> {
|
||||
repository.verifyHarnessStopped(connection, operation.tenantId(), operation.sessionId());
|
||||
return null;
|
||||
});
|
||||
}
|
||||
|
||||
private void lock(OperationRecord operation) {
|
||||
lockPlacement(jdbc, operation.tenantId());
|
||||
ToolPublicationRetentionStore.lockTenant(jdbc, operation.tenantId());
|
||||
jdbc.queryForList("SELECT session_id FROM managed_agent_session WHERE tenant_id = ? AND session_id = ? FOR UPDATE",
|
||||
operation.tenantId(), operation.sessionId());
|
||||
}
|
||||
|
||||
private void verifyEffects(OperationRecord operation, JsonNode receipt) {
|
||||
if (receipt.path("protocolVersion").asInt() != 1 || !receipt.path("sessionKey").equals(sessionKey(operation))
|
||||
|| !operation.operationId().equals(receipt.path("operationId").asText())
|
||||
|| !operation.kind().name().toLowerCase(java.util.Locale.ROOT).equals(receipt.path("kind").asText())
|
||||
|| !receipt.path("effects").isArray()) {
|
||||
throw blocked("workspace_lifecycle_receipt_invalid");
|
||||
}
|
||||
JsonNode header = header(operation.tenantId(), operation.sessionId());
|
||||
if (receipt.path("neverInitialized").asBoolean()) {
|
||||
if (header != null || !receipt.path("effects").isEmpty() || !neverInitialized(operation)) {
|
||||
throw blocked("workspace_lifecycle_receipt_invalid");
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (header == null || !header.path("definitionRef").equals(receipt.path("definitionRef"))) {
|
||||
throw blocked("workspace_lifecycle_receipt_invalid");
|
||||
}
|
||||
JsonNode definition = records.readRecordResource(operation.tenantId(), operation.sessionId(), receipt.path("definitionRef"));
|
||||
if (!"hosted-workspace-files/1".equals(definition.path("toolProfile").asText())) {
|
||||
throw blocked("workspace_lifecycle_profile_unavailable");
|
||||
}
|
||||
for (JsonNode execution : records.listRecords(operation.tenantId(), operation.sessionId(), "hook_execution")) {
|
||||
if (execution.path("resultRef").isNull()
|
||||
&& !"not_started_proven".equals(execution.path("run").path("execution").asText())
|
||||
|| "recovery_blocked".equals(execution.path("run").path("state").asText())) {
|
||||
throw blocked("workspace_lifecycle_hooks_unsettled");
|
||||
}
|
||||
}
|
||||
if (!definition.has("hookCatalog")) {
|
||||
if (!receipt.path("effects").isEmpty()) {
|
||||
throw blocked("workspace_lifecycle_receipt_invalid");
|
||||
}
|
||||
return;
|
||||
}
|
||||
List<String> events = events(operation);
|
||||
if (receipt.path("effects").size() != events.size()) {
|
||||
throw blocked("workspace_lifecycle_hooks_unsettled");
|
||||
}
|
||||
for (int index = 0; index < events.size(); index++) {
|
||||
String event = events.get(index);
|
||||
JsonNode effect = receipt.path("effects").get(index);
|
||||
String latest = latestHookExecutionResource(operation, occurrence(event, operation.operationId()));
|
||||
if (latest == null || !latest.equals(effect.path("recordRef").path("resourceId").asText())) {
|
||||
throw blocked("workspace_lifecycle_receipt_invalid");
|
||||
}
|
||||
JsonNode record = records.readRecordResource(operation.tenantId(), operation.sessionId(), effect.path("recordRef"));
|
||||
ManagedHookRecords.requireExecution(record);
|
||||
if (!event.equals(effect.path("event").asText()) || !event.equals(record.path("eventName").asText())
|
||||
|| !occurrence(event, operation.operationId()).equals(record.path("hookExecutionId").asText())
|
||||
|| !"__plan__".equals(record.path("hookId").asText()) || record.path("resultRef").isNull()
|
||||
|| "recovery_blocked".equals(record.path("run").path("state").asText())) {
|
||||
throw blocked("workspace_lifecycle_hooks_unsettled");
|
||||
}
|
||||
records.readRecordResource(operation.tenantId(), operation.sessionId(), record.path("resultRef"));
|
||||
JsonNode plan = records.readRecordResource(operation.tenantId(), operation.sessionId(), record.path("planRef"));
|
||||
if (!event.equals(plan.path("input").path("hook_event_name").asText())
|
||||
|| !operation.sessionId().equals(plan.path("input").path("session_id").asText())) {
|
||||
throw blocked("workspace_lifecycle_receipt_invalid");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private boolean neverInitialized(OperationRecord operation) {
|
||||
var heads = jdbc.queryForList("SELECT journal_revision, CASE WHEN state = 'ACTIVE' AND"
|
||||
+ " writer_lease_until > CURRENT_TIMESTAMP(6) THEN 1 ELSE 0 END AS live_writer"
|
||||
+ " FROM qwen_managed_session_journal_head WHERE tenant_id = ? AND session_id = ? FOR UPDATE",
|
||||
operation.tenantId(), operation.sessionId());
|
||||
if (!heads.isEmpty() && (((Number) heads.getFirst().get("journal_revision")).longValue() != 0
|
||||
|| ((Number) heads.getFirst().get("live_writer")).intValue() != 0)) {
|
||||
return false;
|
||||
}
|
||||
return jdbc.queryForObject("SELECT harness_boot_id FROM managed_agent_session WHERE tenant_id = ? AND session_id = ?",
|
||||
String.class, operation.tenantId(), operation.sessionId()) == null
|
||||
&& jdbc.queryForObject("SELECT COUNT(*) FROM qwen_managed_session_journal_tx WHERE tenant_id = ? AND session_id = ?",
|
||||
Integer.class, operation.tenantId(), operation.sessionId()) == 0
|
||||
&& records.listRecords(operation.tenantId(), operation.sessionId(), "hook_execution").isEmpty()
|
||||
&& records.listRecords(operation.tenantId(), operation.sessionId(), "hook_registration").isEmpty();
|
||||
}
|
||||
|
||||
private String latestHookExecutionResource(OperationRecord operation, String occurrence) {
|
||||
return jdbc.queryForList("SELECT record_resource_id FROM qwen_managed_session_extension_record"
|
||||
+ " WHERE session_scope_key = ? AND record_key = ? AND tenant_id = ? AND session_id = ?"
|
||||
+ " AND domain = 'hook_execution' AND record_id = ? FOR UPDATE", String.class,
|
||||
ManagedSessionStore.sessionScopeKey(operation.tenantId(), operation.sessionId()),
|
||||
ManagedExtensionProjection.recordKey(operation.sessionId(), "hook_execution", occurrence),
|
||||
operation.tenantId(), operation.sessionId(), occurrence).stream().findFirst().orElse(null);
|
||||
}
|
||||
|
||||
private JsonNode resourceRef(OperationRecord operation, String resource) {
|
||||
return jdbc.query("SELECT kind, schema_version, byte_length, sha256 FROM qwen_managed_session_resource"
|
||||
+ " WHERE session_scope_key = ? AND tenant_id = ? AND session_id = ? AND resource_id = ?",
|
||||
(row, index) -> json.createObjectNode().put("resourceId", resource).put("kind", row.getString("kind"))
|
||||
.put("schemaVersion", row.getInt("schema_version")).put("byteLength", row.getLong("byte_length"))
|
||||
.put("digest", row.getString("sha256")), ManagedSessionStore.sessionScopeKey(operation.tenantId(), operation.sessionId()),
|
||||
operation.tenantId(), operation.sessionId(), resource).stream()
|
||||
.findFirst().orElseThrow(() -> blocked("workspace_lifecycle_receipt_invalid"));
|
||||
}
|
||||
|
||||
private JsonNode header(String tenant, String session) {
|
||||
var rows = jdbc.queryForList("SELECT record_bytes FROM qwen_managed_session_journal_tx WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " ORDER BY journal_revision LIMIT 1", byte[].class, tenant, session);
|
||||
if (rows.isEmpty()) {
|
||||
return null;
|
||||
}
|
||||
for (String line : new String(rows.getFirst(), StandardCharsets.UTF_8).split("\\n")) {
|
||||
JsonNode record = parse(line);
|
||||
if ("managed_session_header_v1".equals(record.path("subtype").asText())) {
|
||||
return record.path("managedSession");
|
||||
}
|
||||
}
|
||||
throw blocked("workspace_lifecycle_receipt_invalid");
|
||||
}
|
||||
|
||||
private com.fasterxml.jackson.databind.node.ObjectNode envelope(OperationRecord operation) {
|
||||
var node = json.createObjectNode().put("protocolVersion", 1).put("operationId", operation.operationId())
|
||||
.put("kind", operation.kind().name().toLowerCase(java.util.Locale.ROOT));
|
||||
node.set("sessionKey", sessionKey(operation));
|
||||
return node;
|
||||
}
|
||||
|
||||
private JsonNode sessionKey(OperationRecord operation) {
|
||||
String workspace = jdbc.queryForObject("SELECT workspace_id FROM managed_agent_session WHERE tenant_id = ? AND session_id = ?",
|
||||
String.class, operation.tenantId(), operation.sessionId());
|
||||
return json.createObjectNode().put("tenantId", operation.tenantId()).put("workspaceId", workspace).put("sessionId", operation.sessionId());
|
||||
}
|
||||
|
||||
private List<String> events(OperationRecord operation) {
|
||||
return operation.kind() == StoreModels.OperationKind.CLOSE ? List.of("SessionEnd") : List.of("SessionEnd", "SessionDelete");
|
||||
}
|
||||
|
||||
private String occurrence(String event, String operationId) {
|
||||
try {
|
||||
return "hook-plan-" + HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(
|
||||
OCCURRENCE_JSON.writeValueAsBytes(List.of(event, operationId))));
|
||||
} catch (java.io.IOException | NoSuchAlgorithmException error) {
|
||||
throw new IllegalStateException(error);
|
||||
}
|
||||
}
|
||||
|
||||
private JsonNode parse(String value) {
|
||||
try {
|
||||
return json.readTree(value);
|
||||
} catch (java.io.IOException error) {
|
||||
throw blocked("workspace_lifecycle_receipt_invalid");
|
||||
}
|
||||
}
|
||||
|
||||
public static RuntimeLifecycleAuthority authority(OperationRecord operation) {
|
||||
return new RuntimeLifecycleAuthority(operation.operationId(), operation.claimGeneration());
|
||||
}
|
||||
|
||||
public static ApiException blocked(String code) {
|
||||
return new ApiException(HttpStatus.CONFLICT, code, "Workspace lifecycle recovery is blocked.");
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -387,8 +387,8 @@ public final class WorkspaceMigrationStore {
|
||||
}
|
||||
|
||||
private void lockAuthority() {
|
||||
ToolPublicationRetentionStore.lockTenant(jdbc, tenant);
|
||||
WorkspaceMigrationAdmission.lockTenant(jdbc, tenant);
|
||||
ToolPublicationRetentionStore.lockTenant(jdbc, tenant);
|
||||
}
|
||||
|
||||
private void requireOwner() {
|
||||
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
ALTER TABLE managed_agent_operation ADD COLUMN lifecycle_protocol_version INT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE managed_agent_operation ADD COLUMN lifecycle_effects_receipt_json LONGTEXT;
|
||||
ALTER TABLE qwen_runtime_harness_drain ADD COLUMN phase VARCHAR(32) NOT NULL DEFAULT 'DRAINING';
|
||||
ALTER TABLE qwen_runtime_harness_drain ADD COLUMN operation_id VARCHAR(128);
|
||||
ALTER TABLE qwen_runtime_harness_drain ADD COLUMN claim_generation BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE qwen_runtime_harness_drain ADD COLUMN claim_lease_until BIGINT;
|
||||
+9
-9
@@ -204,7 +204,7 @@
|
||||
"tags": ["Public Sessions"],
|
||||
"operationId": "createSession",
|
||||
"x-qwen-implementation-status": "implemented",
|
||||
"description": "Workspace selection is persisted with the original actor-scoped creation receipt. With the deployment's G0 file admission opt-in, qwen-code Sessions in an authorized, preregistered Workspace may include initial input under the fixed preapproved file-tool profile. Public callers cannot select the profile. Otherwise Workspace creation must have empty input. The Session creator may submit later Turns and cancel its running Turns under the same opt-in, and may rename the Session, while the creator currently holds Workspace read and create grants on a registry row whose state is ACTIVE and the Session is an active, undeleted qwen-code Session on the frozen execution profile; Workspace close follows its separate close capability and lifecycle admission; archive, delete and unarchive follow their separate retention capabilities after reliable Workspace close; controlled same-Workspace cwd changes ship through the durable cwd_change operations (v1.32). This surface has no per-caller capability flag for later Turns, so callers must handle 409 workspace_unavailable. Check replay before resolving defaults; a durable binding is not proof of Runtime readiness.",
|
||||
"description": "Workspace selection is persisted with the original actor-scoped creation receipt. With the deployment's G0 file admission opt-in, qwen-code Sessions in an authorized, preregistered Workspace may include initial input under the fixed preapproved file-tool profile. Public callers cannot select the profile. Otherwise Workspace creation must have empty input. The Session creator may submit later Turns and cancel its running Turns under the same opt-in, and may rename the Session, while the creator currently holds Workspace read and create grants on a registry row whose state is ACTIVE and the Session is an active, undeleted qwen-code Session on the frozen execution profile; Workspace close follows its separate close capability and lifecycle admission; archive and unarchive follow retention capabilities after reliable Workspace close; delete supports either L3 idle ACTIVE files Sessions or L2 CLOSED/ARCHIVED Sessions; controlled same-Workspace cwd changes ship through the durable cwd_change operations (v1.32). This surface has no per-caller capability flag for later Turns, so callers must handle 409 workspace_unavailable. Check replay before resolving defaults; a durable binding is not proof of Runtime readiness.",
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/IdempotencyKey"
|
||||
@@ -398,7 +398,7 @@
|
||||
"tags": ["Public Sessions"],
|
||||
"operationId": "deleteSession",
|
||||
"x-qwen-implementation-status": "implemented",
|
||||
"description": "Unbound Sessions accept active, closed or archived state. Returns 409 turn_active while a Turn is active and session_operation_active while another lifecycle change is pending. The same key from the same actor returns the original operation, also after the Session was deleted. Workspace-bound delete accepts only closed or archived Sessions with completed reliable-close evidence and the creator with current read access. Active deletion returns 409 session_state_conflict. Completion atomically retires private recovery/publication access with the tombstone, operation receipt and terminal event; it calls neither Harness nor Runtime and erases no shared Workspace files or backup bytes. Operations remain readable under current read grants; the same authorized actor/key can replay after deletion. A readable non-creator returns 403; missing close evidence returns 409 workspace_unavailable.",
|
||||
"description": "Workspace-bound ACTIVE deletion supports idle hosted-workspace-files/1 Sessions when L3 lifecycle protocol and reliable Runtime stop are available. Only the creator with current read access may admit it; accepted, running, cancelling or approval-waiting Turns return 409 turn_active. Admission persists a lifecycle-only execution fence and a versioned actor-scoped operation. SessionEnd settles before SessionDelete; committed Hook results are verified in Session Store and saved before permanent draining, attachment cleanup and original worker stop. Unknown outcomes or missing original stop evidence remain recovery_blocked and are never replayed on a replacement worker. CLOSED/ARCHIVED deletion retains L2: completed reliable-close evidence is required and no Harness or Hook is invoked. Completion atomically commits retirement, tombstone, receipt and terminal event, retaining shared Workspace files and neighboring holders. Authorized same-actor/key replays return the original operation, including after deletion; readable non-creators return 403. Missing lifecycle capability or close evidence returns 409 workspace_unavailable.",
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/IdempotencyKey"
|
||||
@@ -406,7 +406,7 @@
|
||||
],
|
||||
"responses": {
|
||||
"202": {
|
||||
"description": "Deletion accepted. The Session stops accepting input and reads deleting. If it was active, the operation first closes it as a close does; it then drains the Session's Runtime binding and leaves a tombstone: Session reads return 404, while its operations stay readable. The shared Workspace is never deleted, and no Session content is erased yet. 202 does not mean that tools have stopped.",
|
||||
"description": "Deletion accepted. The Session stops accepting input and reads deleting. For a supported ACTIVE Workspace files Session, SessionEnd and then SessionDelete settle under L3 before permanent draining and verified original Runtime stop; CLOSED/ARCHIVED Workspace Sessions use independent L2 retirement. Unbound Sessions retain their existing lifecycle flow. Completion leaves a tombstone: Session reads return 404, while its operations stay readable. The shared Workspace is never deleted, and no Session content is erased yet. 202 does not mean that tools have stopped.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
@@ -1209,7 +1209,7 @@
|
||||
"tags": ["WebShell"],
|
||||
"operationId": "webShellCreateSession",
|
||||
"x-qwen-implementation-status": "implemented",
|
||||
"description": "Maps workspaceId/cwdRelative to public Workspace selection without using environmentId or absolute cwd. Shares G0's opt-in initial file-tool Turn admission and fixed server-owned profile with public Session creation. The Session creator may submit later Turns and cancel its running Turns under the same opt-in, and may rename the Session, while the creator currently holds Workspace read and create grants on a registry row whose state is ACTIVE and the Session is an active, undeleted qwen-code Session on the frozen execution profile; Workspace close follows its separate close capability and lifecycle admission; archive, delete and unarchive follow their separate retention capabilities after reliable Workspace close; controlled same-Workspace cwd changes ship through the durable cwd_change operations (v1.32). The per-caller workspaceTurns capability on this surface advertises the same rule. Freeze selection with the original idempotency key; admission does not prove physical directory readiness.",
|
||||
"description": "Maps workspaceId/cwdRelative to public Workspace selection without using environmentId or absolute cwd. Shares G0's opt-in initial file-tool Turn admission and fixed server-owned profile with public Session creation. The Session creator may submit later Turns and cancel its running Turns under the same opt-in, and may rename the Session, while the creator currently holds Workspace read and create grants on a registry row whose state is ACTIVE and the Session is an active, undeleted qwen-code Session on the frozen execution profile; Workspace close follows its separate close capability and lifecycle admission; archive and unarchive follow retention capabilities after reliable Workspace close; delete supports either L3 idle ACTIVE files Sessions or L2 CLOSED/ARCHIVED Sessions; controlled same-Workspace cwd changes ship through the durable cwd_change operations (v1.32). The per-caller workspaceTurns capability on this surface advertises the same rule. Freeze selection with the original idempotency key; admission does not prove physical directory readiness.",
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
@@ -1876,7 +1876,7 @@
|
||||
"tags": ["Public Sessions"],
|
||||
"operationId": "closeSession",
|
||||
"x-qwen-implementation-status": "implemented",
|
||||
"description": "Accepts an active Session; 409 session_state_conflict otherwise. Supported hosted-workspace-files/1 Sessions may close when session_close is true. Only the creator with current read access may close a bound Session: unreadable returns 404, readable non-creator returns 403. Active, cancelling or approval-waiting Turns return 409 turn_active; another pending lifecycle operation returns session_operation_active. Unsupported Workspace close returns workspace_unavailable. The same actor and key replay the original operation.",
|
||||
"description": "Accepts an active Session; 409 session_state_conflict otherwise. Supported hosted-workspace-files/1 Sessions may close when session_close is true. Only the creator with current read access may close a bound Session: unreadable returns 404, readable non-creator returns 403. Active, cancelling or approval-waiting Turns return 409 turn_active; another pending lifecycle operation returns session_operation_active. Unsupported Workspace close returns workspace_unavailable. The same actor and key replay the original operation. New L3 Workspace close operations run SessionEnd only, retain data, and finish after durable effects verification and original Runtime stop. SessionDelete is reserved for ACTIVE delete. Missing protocol support rejects new admission; previously admitted operations retain their saved protocol. Unknown outcomes remain recovery_blocked.",
|
||||
"responses": {
|
||||
"202": {
|
||||
"description": "Close durably accepted. The Session stops accepting input and reads closing. A bound Session completes only after its permanent Runtime admission fence is stored, no valid journal writer or unsettled execution remains, all original Runtime Sessions and holders are released, and original worker stop proof is persisted; the Session then reads closed. Unknown original identities or executions report recovery_blocked with a stable failure_code and keep the Session closing. History, Artifacts and Workspace files are retained. 202 does not mean that tools have stopped.",
|
||||
@@ -2200,7 +2200,7 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"description": "Uses the same close admission, creator authorization and durable cleanup semantics as the public close route. sessionClose describes deployment support; it does not grant actor authorization."
|
||||
"description": "Uses the same close admission, creator authorization and durable cleanup semantics as the public close route. sessionClose describes deployment support; it does not grant actor authorization. New L3 Workspace close operations run SessionEnd only, retain data, and finish after durable effects verification and original Runtime stop. SessionDelete is reserved for ACTIVE delete. Missing protocol support rejects new admission; previously admitted operations retain their saved protocol. Unknown outcomes remain recovery_blocked."
|
||||
}
|
||||
},
|
||||
"/api/agent/web-shell/v1/sessions/archive": {
|
||||
@@ -2296,7 +2296,7 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"description": "Workspace-bound delete accepts only closed or archived Sessions with completed reliable-close evidence and the creator with current read access. Active deletion returns 409 session_state_conflict. Completion atomically retires private recovery/publication access with the tombstone, operation receipt and terminal event; it calls neither Harness nor Runtime and erases no shared Workspace files or backup bytes. Operations remain readable under current read grants; the same authorized actor/key can replay after deletion. A readable non-creator returns 403; missing close evidence returns 409 workspace_unavailable."
|
||||
"description": "Workspace-bound ACTIVE deletion supports idle hosted-workspace-files/1 Sessions when L3 lifecycle protocol and reliable Runtime stop are available. Only the creator with current read access may admit it; accepted, running, cancelling or approval-waiting Turns return 409 turn_active. Admission persists a lifecycle-only execution fence and a versioned actor-scoped operation. SessionEnd settles before SessionDelete; committed Hook results are verified in Session Store and saved before permanent draining, attachment cleanup and original worker stop. Unknown outcomes or missing original stop evidence remain recovery_blocked and are never replayed on a replacement worker. CLOSED/ARCHIVED deletion retains L2: completed reliable-close evidence is required and no Harness or Hook is invoked. Completion atomically commits retirement, tombstone, receipt and terminal event, retaining shared Workspace files and neighboring holders. Authorized same-actor/key replays return the original operation, including after deletion; readable non-creators return 403. Missing lifecycle capability or close evidence returns 409 workspace_unavailable."
|
||||
}
|
||||
},
|
||||
"/api/agent/web-shell/v1/workspaces/get": {
|
||||
@@ -4165,7 +4165,7 @@
|
||||
"type": "boolean",
|
||||
"default": false,
|
||||
"x-qwen-implementation-status": "implemented",
|
||||
"description": "Optional support for this retention operation. For a Workspace-bound Session requires completed reliable-close evidence; authorization and source-state validation still apply."
|
||||
"description": "Optional support capability. ACTIVE Workspace files Sessions require L3 lifecycle protocol and reliable original Runtime stop; CLOSED/ARCHIVED Sessions use independent L2 close evidence. Current creator/read authorization and idle/source-state admission checks still apply. The aggregate session lifecycle capability is unchanged."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -5122,7 +5122,7 @@
|
||||
"type": "boolean",
|
||||
"default": false,
|
||||
"x-qwen-implementation-status": "implemented",
|
||||
"description": "Optional support for this retention operation. For a Workspace-bound Session requires completed reliable-close evidence; authorization and source-state validation still apply."
|
||||
"description": "Optional support capability. ACTIVE Workspace files Sessions require L3 lifecycle protocol and reliable original Runtime stop; CLOSED/ARCHIVED Sessions use independent L2 close evidence. Current creator/read authorization and idle/source-state admission checks still apply. The aggregate session lifecycle capability is unchanged."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+6
-6
@@ -24,7 +24,7 @@ import java.util.function.UnaryOperator;
|
||||
* Commits journal transactions to the Session store in the record format the
|
||||
* Session authority writes, each carrying one Stage H record revision.
|
||||
*/
|
||||
final class ExtensionRecordJournal {
|
||||
public final class ExtensionRecordJournal {
|
||||
private static final ObjectMapper JSON = new ObjectMapper();
|
||||
private static final String WRITER = "writer-extension";
|
||||
private static final String TOKEN = "extension-writer-token-0123456789";
|
||||
@@ -39,7 +39,7 @@ final class ExtensionRecordJournal {
|
||||
private String lastCommitDigest;
|
||||
private int domainEvents;
|
||||
|
||||
ExtensionRecordJournal(ManagedSessionStore store, String tenantId,
|
||||
public ExtensionRecordJournal(ManagedSessionStore store, String tenantId,
|
||||
String workspaceId, String sessionId) {
|
||||
this.store = store;
|
||||
this.tenantId = tenantId;
|
||||
@@ -48,7 +48,7 @@ final class ExtensionRecordJournal {
|
||||
}
|
||||
|
||||
/** Acquires the writer and commits the Session's genesis. */
|
||||
ExtensionRecordJournal open() {
|
||||
public ExtensionRecordJournal open() {
|
||||
acquire().commit(genesis(
|
||||
"{\"subtype\":\"session_execution_engine\"}\n"
|
||||
+ "{\"subtype\":\"managed_session_header_v1\"}\n",
|
||||
@@ -84,7 +84,7 @@ final class ExtensionRecordJournal {
|
||||
return receipt;
|
||||
}
|
||||
|
||||
CommitReceipt commit(CommitTransactionRequest request) {
|
||||
public CommitReceipt commit(CommitTransactionRequest request) {
|
||||
return store.commit(tenantId, sessionId, TOKEN, request);
|
||||
}
|
||||
|
||||
@@ -122,7 +122,7 @@ final class ExtensionRecordJournal {
|
||||
editRecords, extraEvents, "monitor_run", List.of());
|
||||
}
|
||||
|
||||
CommitTransactionRequest requestDomain(String commandId, String domain,
|
||||
public CommitTransactionRequest requestDomain(String commandId, String domain,
|
||||
JsonNode body, List<CommitResource> resources, long occurredAt) {
|
||||
return request("commitMcpRecord", commandId, bytes(body), occurredAt,
|
||||
event -> { }, records -> records, 0, domain, resources);
|
||||
@@ -222,7 +222,7 @@ final class ExtensionRecordJournal {
|
||||
}
|
||||
|
||||
/** Advances past a request the store committed. */
|
||||
void committed(CommitTransactionRequest request) {
|
||||
public void committed(CommitTransactionRequest request) {
|
||||
journalRevision++;
|
||||
sequence = request.lastSequence();
|
||||
lastCommitDigest = request.commitDigest();
|
||||
|
||||
+65
-10
@@ -797,8 +797,9 @@ class ManagedAgentMySqlIT {
|
||||
// Begin the delete mid-life: it stays pending while this journal's
|
||||
// writer holds the Session, so the Session is DELETING when the next
|
||||
// revision lands.
|
||||
agents.beginOperation(tenant, session, OperationKind.DELETE,
|
||||
"sha256:" + "d".repeat(64), "delete", "digest-delete");
|
||||
OperationRecord deletion = agents.beginOperation(tenant, session,
|
||||
OperationKind.DELETE, "sha256:" + "d".repeat(64), "delete",
|
||||
"digest-delete").operation();
|
||||
assertThat(jdbc.queryForObject("SELECT status FROM"
|
||||
+ " managed_agent_session WHERE tenant_id = ?"
|
||||
+ " AND session_id = ?", String.class, tenant,
|
||||
@@ -816,11 +817,48 @@ class ManagedAgentMySqlIT {
|
||||
+ " qwen_managed_session_extension_record WHERE"
|
||||
+ " tenant_id = ? AND session_id = ?", Long.class,
|
||||
tenant, session)).isEqualTo(2L);
|
||||
OperationRecord operation = inTransaction(transactions,
|
||||
() -> agents.claimOperation(tenant, session,
|
||||
deletion.operationId(), "worker",
|
||||
Duration.ofMinutes(1))).orElseThrow();
|
||||
assertThatThrownBy(() -> inTransaction(transactions,
|
||||
() -> agents.completeOperation(tenant, session,
|
||||
operation.operationId(), "worker",
|
||||
operation.claimGeneration(), true)))
|
||||
.isInstanceOfSatisfying(ApiException.class, error ->
|
||||
assertThat(error.getCode()).isEqualTo(
|
||||
"managed_session_writer_active"));
|
||||
assertThat(count(jdbc, "qwen_output_session_retirement",
|
||||
tenant, session)).isZero();
|
||||
long generation = jdbc.queryForObject("SELECT writer_generation FROM"
|
||||
+ " qwen_managed_session_journal_head WHERE tenant_id = ? AND"
|
||||
+ " session_id = ?", Long.class, tenant, session);
|
||||
inTransaction(transactions, () -> new ManagedSessionStore(jdbc)
|
||||
.sealWriter(tenant, session,
|
||||
"extension-writer-token-0123456789",
|
||||
new SealWriterRequest("mysql-journal-workspace",
|
||||
"writer-extension", generation)));
|
||||
assertThat(inTransaction(transactions,
|
||||
() -> agents.completeOperation(tenant, session,
|
||||
operation.operationId(), "worker",
|
||||
operation.claimGeneration(), true))).isTrue();
|
||||
assertThat(jdbc.queryForObject("SELECT state FROM"
|
||||
+ " qwen_managed_session_journal_head WHERE tenant_id = ? AND"
|
||||
+ " session_id = ?", String.class, tenant, session))
|
||||
.isEqualTo("DELETED");
|
||||
assertThat(count(jdbc, "qwen_managed_session_extension_record",
|
||||
tenant, session)).isEqualTo(1);
|
||||
List<String> events = jdbc.queryForList("SELECT event_type FROM"
|
||||
+ " managed_agent_event WHERE tenant_id = ? AND"
|
||||
+ " session_id = ? ORDER BY sequence_id",
|
||||
String.class, tenant, session);
|
||||
assertThat(events).endsWith("session.deleted")
|
||||
.doesNotContain("task.updated");
|
||||
}
|
||||
|
||||
@Test
|
||||
@Order(9)
|
||||
void journalsNothingWhenTheDeletionCommitsMidCommit() throws Exception {
|
||||
void serializesDeletionBehindTheRecordCommit() throws Exception {
|
||||
DriverManagerDataSource dataSource = dataSource();
|
||||
Flyway.configure().dataSource(dataSource)
|
||||
.locations("classpath:db/migration").load().migrate();
|
||||
@@ -848,8 +886,8 @@ class ManagedAgentMySqlIT {
|
||||
assertThat(count(jdbc, "qwen_managed_session_task_journal", tenant,
|
||||
session)).isEqualTo(1);
|
||||
// A second connection holds the record row, so the next commit
|
||||
// blocks at its UPDATE: after its first consistent read took the
|
||||
// snapshot, and before the deletion guard runs.
|
||||
// blocks at its UPDATE while retaining the placement guard that
|
||||
// deletion admission also needs.
|
||||
try (java.sql.Connection holder = dataSource.getConnection()) {
|
||||
holder.setAutoCommit(false);
|
||||
try (var lock = holder.prepareStatement("SELECT record_key FROM"
|
||||
@@ -883,19 +921,36 @@ class ManagedAgentMySqlIT {
|
||||
assertThat(System.nanoTime()).isLessThan(deadline);
|
||||
Thread.sleep(50);
|
||||
}
|
||||
// The deletion begins and commits while the record commit waits
|
||||
// behind its snapshot.
|
||||
agents.beginOperation(tenant, session, OperationKind.DELETE,
|
||||
"sha256:" + "e".repeat(64), "delete-mid", "digest-mid");
|
||||
CompletableFuture<Object> deletion = CompletableFuture.supplyAsync(
|
||||
() -> inTransaction(transactions,
|
||||
() -> agents.beginOperation(tenant, session,
|
||||
OperationKind.DELETE,
|
||||
"sha256:" + "e".repeat(64), "delete-mid",
|
||||
"digest-mid")));
|
||||
String admissionBlocked = "SELECT COUNT(*) FROM"
|
||||
+ " information_schema.PROCESSLIST WHERE DB = DATABASE()"
|
||||
+ " AND COMMAND = 'Query' AND TIME >= 1 AND INFO LIKE"
|
||||
+ " 'INSERT INTO qwen_runtime_placement_guard%'";
|
||||
while (jdbc.queryForObject(admissionBlocked, Integer.class) == 0) {
|
||||
assertThat(deletion).as("deletion admission must wait for"
|
||||
+ " the record commit's placement guard").isNotDone();
|
||||
assertThat(commit).isNotDone();
|
||||
assertThat(System.nanoTime()).isLessThan(deadline);
|
||||
Thread.sleep(50);
|
||||
}
|
||||
assertThat(deletion).isNotDone();
|
||||
holder.commit();
|
||||
commit.get(30, TimeUnit.SECONDS);
|
||||
deletion.get(30, TimeUnit.SECONDS);
|
||||
}
|
||||
assertThat(jdbc.queryForObject("SELECT revision FROM"
|
||||
+ " qwen_managed_session_extension_record WHERE"
|
||||
+ " tenant_id = ? AND session_id = ?", Long.class,
|
||||
tenant, session)).isEqualTo(2L);
|
||||
assertThat(count(jdbc, "qwen_managed_session_task_journal", tenant,
|
||||
session)).isEqualTo(1);
|
||||
session)).isEqualTo(2);
|
||||
assertThat(agents.requireSession(tenant, session).status())
|
||||
.isEqualTo("DELETING");
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
+94
-1
@@ -17,8 +17,12 @@ import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationAdmission;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationKind;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.SessionMutationKind;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeSessionRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeScope;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSession;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSessionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedworkspace.ContextBinding;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
@@ -41,6 +45,8 @@ import java.util.concurrent.atomic.AtomicLong;
|
||||
import org.flywaydb.core.Flyway;
|
||||
import org.h2.jdbcx.JdbcDataSource;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.EnumSource;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
@@ -104,6 +110,8 @@ class ManagedCwdChangeOperationTest {
|
||||
assertThat(settle(fixture,
|
||||
sessionId, claimed.operationId(), "owner",
|
||||
claimed.claimGeneration()).completed()).isTrue();
|
||||
fixture.insertRuntimeOwner(TENANT, sessionId,
|
||||
RuntimeSessionRecord.State.READY);
|
||||
OperationAdmission replay = begin(fixture, sessionId, "key-1",
|
||||
"digest-1", "services/b", 1);
|
||||
assertThat(replay.replayed()).isTrue();
|
||||
@@ -111,6 +119,75 @@ class ManagedCwdChangeOperationTest {
|
||||
assertThat(replay.operation().resultContextRevision()).isEqualTo(2);
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@EnumSource(value = RuntimeSessionRecord.State.class,
|
||||
names = {"ACQUIRING", "READY", "RELEASING", "FAILED"})
|
||||
void retainedRuntimeContextRefusesAdmissionWithoutAnActiveTurn(
|
||||
RuntimeSessionRecord.State state) {
|
||||
Fixture fixture = fixture(true);
|
||||
String sessionId = fixture.createBoundSession(TENANT, WS);
|
||||
fixture.insertRuntimeOwner(TENANT, sessionId, state);
|
||||
assertThatThrownBy(() -> fixture.tx(() -> begin(fixture, sessionId,
|
||||
"key", "digest", "services/b", 1)))
|
||||
.isInstanceOfSatisfying(ApiException.class,
|
||||
error -> assertRefusal(error, HttpStatus.CONFLICT,
|
||||
"session_context_busy"));
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM"
|
||||
+ " managed_agent_operation WHERE tenant_id = ? AND"
|
||||
+ " session_id = ?", Integer.class, TENANT, sessionId)).isZero();
|
||||
assertThat(fixture.store.requireSession(TENANT, sessionId)
|
||||
.workspace().getContextRevision()).isEqualTo(1);
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@EnumSource(value = RuntimeSessionRecord.State.class,
|
||||
names = {"ACQUIRING", "READY", "RELEASING", "FAILED"})
|
||||
void runtimeContextAcquiredAfterAdmissionRefusesCommit(
|
||||
RuntimeSessionRecord.State state) {
|
||||
Fixture fixture = fixture(true);
|
||||
String sessionId = fixture.createBoundSession(TENANT, WS);
|
||||
OperationRecord operation = fixture.tx(() -> begin(fixture,
|
||||
sessionId, "key", "digest", "services/b", 1)).operation();
|
||||
OperationRecord claimed = claim(fixture, sessionId,
|
||||
operation.operationId(), "owner");
|
||||
fixture.insertRuntimeOwner(TENANT, sessionId, state);
|
||||
CwdChangeOutcome outcome = settle(fixture, sessionId,
|
||||
operation.operationId(), "owner", claimed.claimGeneration());
|
||||
assertThat(outcome.completed()).isFalse();
|
||||
assertThat(outcome.failureCode()).isEqualTo("session_context_busy");
|
||||
assertFailed(fixture, sessionId, operation.operationId(),
|
||||
"session_context_busy");
|
||||
ContextBinding binding = fixture.store.requireSession(TENANT,
|
||||
sessionId).workspace();
|
||||
assertThat(binding.getCwdRelative()).isEqualTo("services/api");
|
||||
assertThat(binding.getContextRevision()).isEqualTo(1);
|
||||
assertThat(fixture.events(sessionId).stream().map(fixture::event)
|
||||
.filter(event -> "session.context.changed"
|
||||
.equals(event.path("type").asText())).count()).isZero();
|
||||
}
|
||||
|
||||
@Test
|
||||
void confirmedReleaseAndOtherHarnessOwnersPermitTheDirectoryChange() {
|
||||
Fixture fixture = fixture(true);
|
||||
String sessionId = fixture.createBoundSession(TENANT, WS);
|
||||
fixture.insertRuntimeOwner(TENANT, sessionId,
|
||||
RuntimeSessionRecord.State.RELEASED);
|
||||
fixture.insertRuntimeOwner("another-tenant", sessionId,
|
||||
RuntimeSessionRecord.State.READY);
|
||||
fixture.insertRuntimeOwner(TENANT, "another-harness",
|
||||
RuntimeSessionRecord.State.READY);
|
||||
OperationRecord operation = fixture.tx(() -> begin(fixture,
|
||||
sessionId, "key", "digest", "services/b", 1)).operation();
|
||||
OperationRecord claimed = claim(fixture, sessionId,
|
||||
operation.operationId(), "owner");
|
||||
assertThat(settle(fixture, sessionId, operation.operationId(),
|
||||
"owner", claimed.claimGeneration()).completed()).isTrue();
|
||||
ContextBinding binding = fixture.store.requireSession(TENANT,
|
||||
sessionId).workspace();
|
||||
assertThat(binding.getCwdRelative()).isEqualTo("services/b");
|
||||
assertThat(binding.getContextRevision()).isEqualTo(2);
|
||||
}
|
||||
|
||||
@Test
|
||||
void migrationFenceRefusesFreshCwdChangesButPreservesReceiptsAndOtherStorage() {
|
||||
Fixture fixture = fixture(true);
|
||||
@@ -1421,6 +1498,22 @@ class ManagedCwdChangeOperationTest {
|
||||
turnId, UUID.randomUUID().toString(), status);
|
||||
}
|
||||
|
||||
void insertRuntimeOwner(String tenant, String sessionId,
|
||||
RuntimeSessionRecord.State state) {
|
||||
RuntimeScope scope = new RuntimeScope(tenant, WS, "1",
|
||||
"/workspace", "workspace-files", "session");
|
||||
RuntimeSession session = new RuntimeSession(sessionId,
|
||||
UUID.randomUUID().toString(), "bootstrap", scope);
|
||||
var repository = new JdbcRuntimeSessionRepository(jdbc.getDataSource());
|
||||
RuntimeSessionRecord created = repository.findOrCreate(
|
||||
new RuntimeSessionRecord(session, "binding", 1,
|
||||
RuntimeSessionRecord.State.ACQUIRING, 0, clock.instant()));
|
||||
if (state != RuntimeSessionRecord.State.ACQUIRING) {
|
||||
assertThat(repository.compareAndSet(created,
|
||||
created.withState(state, clock.instant()))).isNotNull();
|
||||
}
|
||||
}
|
||||
|
||||
List<Map<String, Object>> events(String sessionId) {
|
||||
return jdbc.queryForList("SELECT data_json, event_type,"
|
||||
+ " source_key FROM managed_agent_event WHERE"
|
||||
|
||||
+2
-2
@@ -14,7 +14,7 @@ import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
class ManagedHookRecordContractTest {
|
||||
public class ManagedHookRecordContractTest {
|
||||
private static final ObjectMapper JSON = new ObjectMapper();
|
||||
|
||||
@Test
|
||||
@@ -50,7 +50,7 @@ class ManagedHookRecordContractTest {
|
||||
}
|
||||
}
|
||||
|
||||
static JsonNode fixtures() throws IOException {
|
||||
public static JsonNode fixtures() throws IOException {
|
||||
Path directory = Path.of("").toAbsolutePath();
|
||||
while (directory != null) {
|
||||
Path path = directory.resolve("packages/core/src/managed-runtime/contracts/managed-hook-record-v1.fixtures.json");
|
||||
|
||||
+10
-1
@@ -55,13 +55,22 @@ class ManagedOperationSchemaUpgradeTest {
|
||||
assertThat(operation.targetCwdRelative()).isNull();
|
||||
assertThat(operation.expectedContextRevision()).isNull();
|
||||
assertThat(operation.resultContextRevision()).isNull();
|
||||
assertThat(operation.lifecycleProtocolVersion()).isZero();
|
||||
});
|
||||
|
||||
// After the upgrade the same reads work unchanged.
|
||||
Flyway.configure().dataSource(dataSource)
|
||||
.locations("classpath:db/migration").load().migrate();
|
||||
assertThat(store.findOperation("t", "s", "op-1")).isPresent()
|
||||
.hasValueSatisfying(operation -> {
|
||||
assertThat(operation.expectedContextRevision()).isNull();
|
||||
assertThat(operation.lifecycleProtocolVersion()).isZero();
|
||||
});
|
||||
jdbc.update("UPDATE managed_agent_operation"
|
||||
+ " SET lifecycle_protocol_version = 1"
|
||||
+ " WHERE tenant_id = 't' AND operation_id = 'op-1'");
|
||||
assertThat(store.findOperation("t", "s", "op-1")).isPresent()
|
||||
.hasValueSatisfying(operation ->
|
||||
assertThat(operation.expectedContextRevision()).isNull());
|
||||
assertThat(operation.lifecycleProtocolVersion()).isEqualTo(1));
|
||||
}
|
||||
}
|
||||
+8
-9
@@ -672,15 +672,14 @@ class ManagedSessionLifecycleTest {
|
||||
.principal(actor(tenant, "actor-a")), tenant, "bound-close")
|
||||
.andExpect(status().isConflict())
|
||||
.andExpect(jsonPath("$.error.code").value("workspace_unavailable"));
|
||||
for (MockHttpServletRequestBuilder request : List.of(
|
||||
post("/v1/agents/sessions/{id}/archive", sessionId),
|
||||
delete("/v1/agents/sessions/{id}", sessionId))) {
|
||||
lifecycle(request.principal(actor(tenant, "actor-a")), tenant,
|
||||
"bound-" + UUID.randomUUID())
|
||||
.andExpect(status().isConflict())
|
||||
.andExpect(jsonPath("$.error.code")
|
||||
.value("session_state_conflict"));
|
||||
}
|
||||
lifecycle(post("/v1/agents/sessions/{id}/archive", sessionId)
|
||||
.principal(actor(tenant, "actor-a")), tenant, "bound-archive")
|
||||
.andExpect(status().isConflict())
|
||||
.andExpect(jsonPath("$.error.code").value("session_state_conflict"));
|
||||
lifecycle(delete("/v1/agents/sessions/{id}", sessionId)
|
||||
.principal(actor(tenant, "actor-a")), tenant, "bound-delete")
|
||||
.andExpect(status().isConflict())
|
||||
.andExpect(jsonPath("$.error.code").value("workspace_unavailable"));
|
||||
mvc.perform(post(WEB_SHELL + "/sessions/delete").header(TENANT, tenant)
|
||||
.principal(actor(tenant, "actor-b"))
|
||||
.contentType(MediaType.APPLICATION_JSON)
|
||||
|
||||
+10
-2
@@ -696,7 +696,8 @@ class SurfaceAdmissionAcceptanceTest {
|
||||
*/
|
||||
private void expectInternal(SurfaceRegistry entry) throws Exception {
|
||||
int status = switch (entry.capabilities().iterator().next()) {
|
||||
case STORE_WRITER_ACQUIRE, STORE_WRITER_RENEW, STORE_WRITER_SEAL,
|
||||
case STORE_EXECUTION_AUTHORIZE, STORE_LIFECYCLE_AUTHORIZE,
|
||||
STORE_WRITER_ACQUIRE, STORE_WRITER_RENEW, STORE_WRITER_SEAL,
|
||||
STORE_RECOVERY_BLOCK, STORE_TRANSACTION_COMMIT,
|
||||
STORE_RESTORE, STORE_TOOL_RESULT_PUBLISH,
|
||||
STORE_TRANSACTION_LIST, STORE_RESOURCE_GET, PUB_FINISHED,
|
||||
@@ -711,6 +712,12 @@ class SurfaceAdmissionAcceptanceTest {
|
||||
.as("%s answered %d for a wrong credential", entry.routeKey(),
|
||||
result.getResponse().getStatus())
|
||||
.isEqualTo(status);
|
||||
if (entry == SurfaceRegistry.INTERNAL_EXECUTION_AUTHORIZE
|
||||
|| entry == SurfaceRegistry.INTERNAL_LIFECYCLE_AUTHORIZE) {
|
||||
assertThat(JSON.readTree(result.getResponse().getContentAsString())
|
||||
.path("error").path("code").asText())
|
||||
.isEqualTo("writer_credential_invalid");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -950,7 +957,8 @@ class SurfaceAdmissionAcceptanceTest {
|
||||
case STORE_WRITER_RENEW -> "{\"workspaceId\":\"ws\","
|
||||
+ "\"writerId\":\"writer-probe\",\"writerGeneration\":1,"
|
||||
+ "\"leaseMillis\":60000}";
|
||||
case STORE_WRITER_SEAL -> "{\"workspaceId\":\"ws\","
|
||||
case STORE_EXECUTION_AUTHORIZE, STORE_LIFECYCLE_AUTHORIZE,
|
||||
STORE_WRITER_SEAL -> "{\"workspaceId\":\"ws\","
|
||||
+ "\"writerId\":\"writer-probe\",\"writerGeneration\":1}";
|
||||
case STORE_RECOVERY_BLOCK -> "{\"workspaceId\":\"ws\","
|
||||
+ "\"writerId\":\"writer-probe\",\"writerGeneration\":1,"
|
||||
|
||||
+231
@@ -0,0 +1,231 @@
|
||||
package com.alibaba.qwen.code.managedagent;
|
||||
|
||||
import static com.alibaba.qwen.code.managedagent.PublicationJournalFixture.COMMIT_MARKER;
|
||||
import static com.alibaba.qwen.code.managedagent.PublicationJournalFixture.PUBLICATION_TOKEN;
|
||||
import static com.alibaba.qwen.code.managedagent.PublicationJournalFixture.WRITER_TOKEN;
|
||||
import static com.alibaba.qwen.code.managedagent.PublicationJournalFixture.digest;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.api.WorkspaceSelection;
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedAgentStore;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.CommitResource;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.CommitTransactionRequest;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedWorkspaceRegistry;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationKind;
|
||||
import com.alibaba.qwen.code.managedagent.store.ToolPublicationAdmissionStore;
|
||||
import com.alibaba.qwen.code.managedagent.store.ToolPublicationDataStore;
|
||||
import com.alibaba.qwen.code.managedagent.store.ToolPublicationObjectStore;
|
||||
import com.alibaba.qwen.code.managedagent.store.WorkspaceLifecycleStore;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.WorkspaceExecutionProfile;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import java.io.InputStream;
|
||||
import java.lang.reflect.InvocationTargetException;
|
||||
import java.lang.reflect.Proxy;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.sql.Connection;
|
||||
import java.sql.PreparedStatement;
|
||||
import java.sql.SQLException;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.Base64;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import javax.sql.DataSource;
|
||||
import org.flywaydb.core.Flyway;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import org.springframework.beans.factory.support.DefaultListableBeanFactory;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.jdbc.datasource.DelegatingDataSource;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
import org.springframework.transaction.support.TransactionTemplate;
|
||||
|
||||
class ToolPublicationLifecycleMySqlIT {
|
||||
private static final ObjectMapper JSON = new ObjectMapper();
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(booleans = {false, true})
|
||||
void receiptAndReplayDoNotDeadlockWithSameTenantLifecycleCompletion(boolean replay) throws Exception {
|
||||
String url = System.getProperty("mysql.url");
|
||||
if (url == null || !url.matches("jdbc:mysql://[^/]+/[^?]+(?:\\?.*)?")) {
|
||||
throw new IllegalArgumentException("A MySQL test database URL is required");
|
||||
}
|
||||
String user = System.getProperty("mysql.user");
|
||||
String password = System.getProperty("mysql.password", "");
|
||||
var admin = new JdbcTemplate(new DriverManagerDataSource(url, user, password));
|
||||
String schema = "publication_lifecycle_" + UUID.randomUUID().toString().replace("-", "");
|
||||
admin.execute("CREATE DATABASE " + schema);
|
||||
try {
|
||||
var source = new ScheduledDataSource(new DriverManagerDataSource(
|
||||
url.replaceFirst("/[^/?]+(?=\\?|$)", "/" + schema), user, password));
|
||||
Flyway.configure().dataSource(source).load().migrate();
|
||||
var journal = PublicationJournalFixture.create(source, false);
|
||||
journal.reserve();
|
||||
var jdbc = journal.jdbc;
|
||||
var transaction = new TransactionTemplate(new DataSourceTransactionManager(source));
|
||||
ToolPublicationObjectStore objects = new ToolPublicationObjectStore() {
|
||||
@Override
|
||||
public void putIfAbsent(String key, byte[] bytes) {
|
||||
throw new AssertionError("Inline fixture expected");
|
||||
}
|
||||
@Override
|
||||
public InputStream open(String key) {
|
||||
throw new AssertionError("Inline fixture expected");
|
||||
}
|
||||
@Override
|
||||
public void requireUnversioned() {}
|
||||
};
|
||||
var data = new ToolPublicationDataStore(jdbc, journal.manager, journal.store, journal.sessions,
|
||||
objects, Duration.ofMinutes(2), Duration.ofSeconds(30),
|
||||
new ToolPublicationDataStore.VerificationBudget(16 * 1024 * 1024, Duration.ofMinutes(25)));
|
||||
var key = journal.binding.get("sessionKey");
|
||||
var envelope = JSON.createObjectNode().put("executionStatus", "error");
|
||||
envelope.putArray("responseParts");
|
||||
envelope.set("capture", JSON.createObjectNode().put("captureStatus", "unavailable")
|
||||
.put("captureReason", "storage_failed").put("previewTruncated", false)
|
||||
.put("deliveryStatus", "pending").putNull("manifest"));
|
||||
data.finish(key, "pub-1", PUBLICATION_TOKEN, "finish", envelope.toString().getBytes(StandardCharsets.UTF_8));
|
||||
var outcome = JSON.createObjectNode().put("schemaVersion", 1).put("decision", "blocked").putNull("manifestRef");
|
||||
outcome.set("envelope", envelope);
|
||||
var history = JSON.createObjectNode().put("messageId", "11111111-1111-4111-8111-111111111111")
|
||||
.put("timestamp", "2026-10-06T00:00:00Z").put("model", "test");
|
||||
history.putArray("parts").addObject().put("text", "Shell capture unavailable");
|
||||
outcome.set("history", history);
|
||||
var root = data.prepareAdmission(key, "pub-1", "writer-1", 1, WRITER_TOKEN, outcome);
|
||||
long sequence = journal.sequence + 1;
|
||||
var payload = JSON.createObjectNode().put("executionCallId", "execution-1")
|
||||
.put("historyRevision", sequence).putNull("resultRef");
|
||||
payload.set("toolOutcomeRef", root);
|
||||
payload.putArray("resources");
|
||||
String records = journal.event(sequence, "tool.receipt", payload) + COMMIT_MARKER;
|
||||
var request = new CommitTransactionRequest("workspace-1", "writer-1", 1, journal.revision,
|
||||
journal.sequence, "receipt", "recordToolResult", "execution-1", root.path("digest").asText(),
|
||||
sequence, sequence, 1, digest(records), journal.commitDigest, digest(records), 1, null, 2,
|
||||
Base64.getEncoder().encodeToString(records.getBytes(StandardCharsets.UTF_8)), digest(records),
|
||||
List.of(new CommitResource(root.path("resourceId").asText(), "managed-tool-outcome", 1,
|
||||
root.path("byteLength").asLong(), root.path("digest").asText(), null)));
|
||||
var admission = new ToolPublicationAdmissionStore(jdbc, journal.manager, journal.sessions, data);
|
||||
if (replay) {
|
||||
admission.commitReceipt(key, "pub-1", WRITER_TOKEN, request);
|
||||
}
|
||||
|
||||
var properties = new ManagedAgentProperties();
|
||||
properties.getHarness().setWorkspaceFilesEnabled(true);
|
||||
var agents = new ManagedAgentStore(jdbc, JSON, Clock.systemUTC(), ignored -> {},
|
||||
new ManagedWorkspaceRegistry(jdbc), properties);
|
||||
var beans = new DefaultListableBeanFactory();
|
||||
beans.registerSingleton("bindings", journal.bindings);
|
||||
var lifecycle = new WorkspaceLifecycleStore(jdbc, JSON, beans.getBeanProvider(RuntimeBindingRepository.class));
|
||||
agents.setWorkspaceLifecycleStore(lifecycle);
|
||||
jdbc.update("INSERT INTO managed_workspace_registry (tenant_id, workspace_id, workspace_generation,"
|
||||
+ " storage_id, display_name, config_ref, policy_ref, state)"
|
||||
+ " VALUES ('tenant-1', 'neighbor', 1, 'storage', 'Neighbor', ?, ?, 'ACTIVE')",
|
||||
WorkspaceExecutionProfile.CONFIG_REF, WorkspaceExecutionProfile.POLICY_REF);
|
||||
jdbc.update("INSERT INTO managed_workspace_access (tenant_id, workspace_id, actor_id, can_read, can_create)"
|
||||
+ " VALUES ('tenant-1', 'neighbor', ?, TRUE, TRUE)", "owner".getBytes(StandardCharsets.UTF_8));
|
||||
String session = transaction.execute(ignored -> agents.insertWorkspaceSessionCommand("tenant-1", "owner",
|
||||
"create", "digest", "qwen-code", null, null, List.of(), null,
|
||||
new WorkspaceSelection("neighbor", ".")).sessionId());
|
||||
String operationId = transaction.execute(ignored -> agents.beginWorkspaceLifecycle("tenant-1", session,
|
||||
OperationKind.CLOSE, "owner", "a".repeat(64), "close", "digest", true, 1).operation().operationId());
|
||||
var operation = transaction.execute(ignored -> agents.claimOperation("tenant-1", session, operationId,
|
||||
"worker", Duration.ofMinutes(5)).orElseThrow());
|
||||
assertThat(transaction.execute(ignored -> lifecycle.recoverEffects(operation)).path("neverInitialized").asBoolean()).isTrue();
|
||||
source.enabled = true;
|
||||
try (var workers = Executors.newFixedThreadPool(2)) {
|
||||
var completion = workers.submit(() -> {
|
||||
Thread.currentThread().setName("lifecycle");
|
||||
return transaction.execute(ignored -> agents.completeOperation("tenant-1", session, operationId,
|
||||
"worker", operation.claimGeneration(), true));
|
||||
});
|
||||
try {
|
||||
assertThat(source.lifecyclePlacement.await(10, TimeUnit.SECONDS)).isTrue();
|
||||
var receipt = workers.submit(() -> {
|
||||
Thread.currentThread().setName("receipt");
|
||||
return admission.commitReceipt(key, "pub-1", WRITER_TOKEN, request);
|
||||
});
|
||||
assertThat(source.receiptPlacement.await(10, TimeUnit.SECONDS)).isTrue();
|
||||
source.releaseLifecycle.countDown();
|
||||
assertThat(completion.get(10, TimeUnit.SECONDS)).isTrue();
|
||||
assertThat(receipt.get(10, TimeUnit.SECONDS).path("historyRevision").asLong()).isEqualTo(sequence);
|
||||
} finally {
|
||||
source.releaseLifecycle.countDown();
|
||||
}
|
||||
}
|
||||
source.enabled = false;
|
||||
assertThat(agents.requireSession("tenant-1", session).status()).isEqualTo("CLOSED");
|
||||
assertThat(agents.findOperation("tenant-1", session, operationId).orElseThrow().deliveryState()).isEqualTo("CONFIRMED");
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM qwen_managed_session_journal_tx"
|
||||
+ " WHERE tenant_id = 'tenant-1' AND session_id = 'session-1' AND operation = 'recordToolResult'", Integer.class)).isOne();
|
||||
assertThat(admission.commitReceipt(key, "pub-1", WRITER_TOKEN, request).path("historyRevision").asLong()).isEqualTo(sequence);
|
||||
} finally {
|
||||
admin.execute("DROP DATABASE " + schema);
|
||||
}
|
||||
}
|
||||
|
||||
private static final class ScheduledDataSource extends DelegatingDataSource {
|
||||
private final CountDownLatch lifecyclePlacement = new CountDownLatch(1);
|
||||
private final CountDownLatch receiptPlacement = new CountDownLatch(1);
|
||||
private final CountDownLatch releaseLifecycle = new CountDownLatch(1);
|
||||
private final AtomicBoolean paused = new AtomicBoolean();
|
||||
private volatile boolean enabled;
|
||||
|
||||
private ScheduledDataSource(DataSource source) {
|
||||
super(source);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Connection getConnection() throws SQLException {
|
||||
return wrap(super.getConnection());
|
||||
}
|
||||
|
||||
@Override
|
||||
public Connection getConnection(String user, String password) throws SQLException {
|
||||
return wrap(super.getConnection(user, password));
|
||||
}
|
||||
|
||||
private Connection wrap(Connection connection) {
|
||||
return (Connection) Proxy.newProxyInstance(getClass().getClassLoader(), new Class<?>[] {Connection.class},
|
||||
(proxy, method, args) -> {
|
||||
try {
|
||||
Object result = method.invoke(connection, args);
|
||||
if (!method.getName().equals("prepareStatement") || !(result instanceof PreparedStatement statement)) {
|
||||
return result;
|
||||
}
|
||||
String sql = (String) args[0];
|
||||
return Proxy.newProxyInstance(getClass().getClassLoader(), new Class<?>[] {PreparedStatement.class},
|
||||
(prepared, call, values) -> {
|
||||
boolean execute = enabled && call.getName().startsWith("execute");
|
||||
String role = Thread.currentThread().getName();
|
||||
if (execute && "receipt".equals(role) && sql.startsWith("INSERT INTO qwen_runtime_placement_guard")) {
|
||||
receiptPlacement.countDown();
|
||||
}
|
||||
try {
|
||||
Object value = call.invoke(statement, values);
|
||||
if (execute && "lifecycle".equals(role)
|
||||
&& sql.startsWith("SELECT tenant_id FROM qwen_runtime_placement_guard")
|
||||
&& paused.compareAndSet(false, true)) {
|
||||
lifecyclePlacement.countDown();
|
||||
if (!releaseLifecycle.await(10, TimeUnit.SECONDS)) {
|
||||
throw new AssertionError("Lifecycle hold timed out");
|
||||
}
|
||||
}
|
||||
return value;
|
||||
} catch (InvocationTargetException error) {
|
||||
throw error.getCause();
|
||||
}
|
||||
});
|
||||
} catch (InvocationTargetException error) {
|
||||
throw error.getCause();
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
+18
-2
@@ -5,13 +5,19 @@ import static org.awaitility.Awaitility.await;
|
||||
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
|
||||
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.api.ApiException;
|
||||
import com.alibaba.qwen.code.managedagent.api.AuthenticatedTenantActor;
|
||||
import com.alibaba.qwen.code.managedagent.api.TenantContextFilter;
|
||||
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector;
|
||||
import com.alibaba.qwen.code.managedagent.harness.UnavailableHarnessConnector;
|
||||
import com.alibaba.qwen.code.managedagent.service.RuntimeWarmer;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedAgentStore;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStore;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.AcquireWriterRequest;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.SealWriterRequest;
|
||||
import com.alibaba.qwen.code.runtimebroker.AesGcmSecretProtector;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.WorkspaceExecutionProfile;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
@@ -63,7 +69,7 @@ class WorkspaceSessionCloseTest {
|
||||
await().untilAsserted(() -> assertThat(runtime.fenced).containsKey(session));
|
||||
assertThat(store.requireSession(tenant, session).status()).isEqualTo("CLOSING");
|
||||
assertThatThrownBy(() -> journal.acquireWriter(tenant, session, "a".repeat(43),
|
||||
new AcquireWriterRequest("ws", "writer", 10_000L))).hasMessageContaining("closing");
|
||||
new AcquireWriterRequest("ws", "writer", 10_000L))).isInstanceOfSatisfying(ApiException.class, error -> assertThat(error.getCode()).isEqualTo("managed_session_lifecycle_active"));
|
||||
mvc.perform(post("/api/agent/web-shell/v1/sessions/close").header(TenantContextFilter.HEADER, tenant)
|
||||
.principal(actor(tenant, "owner")).contentType(MediaType.APPLICATION_JSON)
|
||||
.content("{\"sessionId\":\"" + session + "\",\"idempotencyKey\":\"close\"}"))
|
||||
@@ -197,6 +203,15 @@ class WorkspaceSessionCloseTest {
|
||||
|
||||
@TestConfiguration
|
||||
static class Configuration {
|
||||
@Bean RuntimeBindingRepository lifecycleBindings(javax.sql.DataSource source) {
|
||||
return new JdbcRuntimeBindingRepository(source, new AesGcmSecretProtector("test", new byte[32]));
|
||||
}
|
||||
@Bean @Primary HarnessConnector uninitializedHarness() {
|
||||
return new UnavailableHarnessConnector() {
|
||||
public boolean supportsLifecycle() { return true; }
|
||||
public void detachLifecycle(com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord operation) {}
|
||||
};
|
||||
}
|
||||
@Bean @Primary CloseRuntime closeRuntime() { return new CloseRuntime(); }
|
||||
@Bean @Primary ManagedAgentStore closeStore(JdbcTemplate jdbc, ObjectMapper mapper,
|
||||
com.alibaba.qwen.code.managedagent.store.ManagedWorkspaceRegistry registry) {
|
||||
@@ -213,8 +228,9 @@ class WorkspaceSessionCloseTest {
|
||||
public boolean supportsWorkspaceClose() { return true; }
|
||||
public CompletionStage<Void> warm(String id) { return CompletableFuture.completedFuture(null); }
|
||||
public CompletionStage<Void> drain(String id) { return CompletableFuture.completedFuture(null); }
|
||||
public void requestWorkspaceClose(String tenant, String id) { fenced.put(id, tenant); }
|
||||
public void requestWorkspaceClose(String tenant, String id) { throw new AssertionError("Unexpected legacy close"); }
|
||||
public CompletionStage<Void> closeWorkspace(String tenant, String id) {
|
||||
fenced.put(id, tenant);
|
||||
return results.computeIfAbsent(id, ignored -> CompletableFuture.completedFuture(null));
|
||||
}
|
||||
}
|
||||
|
||||
+5
-2
@@ -351,14 +351,17 @@ class WorkspaceSessionRetentionMySqlIT {
|
||||
assertThat(jdbc.queryForMap("SELECT * FROM managed_agent_session")).containsAllEntriesOf(originalSession);
|
||||
assertThat(jdbc.queryForMap("SELECT * FROM managed_agent_operation"))
|
||||
.containsAllEntriesOf(originalClose)
|
||||
.hasSize(originalClose.size() + 3)
|
||||
.hasSize(originalClose.size() + 5)
|
||||
.containsEntry("target_cwd_relative", null)
|
||||
.containsEntry("expected_context_revision", null)
|
||||
.containsEntry("result_context_revision", null);
|
||||
.containsEntry("result_context_revision", null)
|
||||
.containsEntry("lifecycle_protocol_version", 0)
|
||||
.containsEntry("lifecycle_effects_receipt_json", null);
|
||||
var properties = new ManagedAgentProperties();
|
||||
properties.getHarness().setWorkspaceFilesEnabled(true);
|
||||
var store = new ManagedAgentStore(jdbc, mapper, Clock.systemUTC(), ignored -> {}, new ManagedWorkspaceRegistry(jdbc), properties);
|
||||
assertThat(store.hasCompletedWorkspaceClose(tenant, session)).isTrue();
|
||||
assertThat(store.findOperation(tenant, session, "legacy-close").orElseThrow().lifecycleProtocolVersion()).isZero();
|
||||
assertThat(store.requireSession(tenant, session).status()).isEqualTo("CLOSED");
|
||||
assertThat(store.requireSession(tenant, session).toolProfile()).isEqualTo("hosted-workspace-files/1");
|
||||
assertThat(store.findOperation(tenant, session, "legacy-close").orElseThrow().receiptId()).isEqualTo("legacy-close-receipt");
|
||||
|
||||
+24
-9
@@ -6,9 +6,12 @@ import static org.awaitility.Awaitility.await;
|
||||
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
|
||||
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.api.ApiException;
|
||||
import com.alibaba.qwen.code.managedagent.api.AuthenticatedTenantActor;
|
||||
import com.alibaba.qwen.code.managedagent.api.TenantContextFilter;
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
|
||||
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector;
|
||||
import com.alibaba.qwen.code.managedagent.harness.UnavailableHarnessConnector;
|
||||
import com.alibaba.qwen.code.managedagent.service.RuntimeWarmer;
|
||||
import com.alibaba.qwen.code.managedagent.service.SessionLifecycleCoordinator;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedAgentStore;
|
||||
@@ -17,6 +20,9 @@ import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.Acquir
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.SealWriterRequest;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedWorkspaceRegistry;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationKind;
|
||||
import com.alibaba.qwen.code.runtimebroker.AesGcmSecretProtector;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.WorkspaceExecutionProfile;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
@@ -88,10 +94,10 @@ class WorkspaceSessionRetentionTest {
|
||||
request(post(PUBLIC + session + "/unarchive"), tenant, "owner", "unarchive")
|
||||
.andExpect(status().isOk()).andExpect(jsonPath("$.status").value("archived"));
|
||||
assertThatThrownBy(() -> journal.acquireWriter(tenant, session, "b".repeat(43),
|
||||
new AcquireWriterRequest("ws", "late", 60_000L))).hasMessageContaining("closing or closed");
|
||||
new AcquireWriterRequest("ws", "late", 60_000L))).isInstanceOfSatisfying(ApiException.class, error -> assertThat(error.getCode()).isEqualTo("managed_session_lifecycle_active"));
|
||||
assertThat(count(tenant, session, "session.archived")).isEqualTo(2);
|
||||
assertThat(count(tenant, session, "session.unarchived")).isEqualTo(1);
|
||||
assertThat(runtime.calls.get(session)).isEqualTo(2);
|
||||
assertThat(runtime.calls.get(session)).isEqualTo(1);
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@@ -119,7 +125,7 @@ class WorkspaceSessionRetentionTest {
|
||||
assertThat(jdbc.queryForObject("SELECT operation_id FROM qwen_output_session_retirement WHERE tenant_id = ?"
|
||||
+ " AND session_id = ?", String.class, tenant, session)).isEqualTo(id);
|
||||
assertThat(count(tenant, session, "session.deleted")).isEqualTo(1);
|
||||
assertThat(runtime.calls.get(session)).isEqualTo(2);
|
||||
assertThat(runtime.calls.get(session)).isEqualTo(1);
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@@ -155,7 +161,7 @@ class WorkspaceSessionRetentionTest {
|
||||
assertThat(jdbc.queryForObject("SELECT operation_id FROM qwen_output_session_retirement WHERE tenant_id = ?"
|
||||
+ " AND session_id = ?", String.class, tenant, session)).isEqualTo(id);
|
||||
assertThat(count(tenant, session, "session.deleted")).isEqualTo(1);
|
||||
assertThat(runtime.calls.get(session)).isEqualTo(2);
|
||||
assertThat(runtime.calls.get(session)).isEqualTo(1);
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -204,10 +210,10 @@ class WorkspaceSessionRetentionTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void sourceStatesAndMissingCloseProofAreRefused() throws Exception {
|
||||
void archiveAndUnarchiveSourceStatesAndMissingCloseProofAreRefused() throws Exception {
|
||||
String tenant = tenant();
|
||||
String session = create(tenant);
|
||||
for (String operation : new String[] {"archive", "unarchive", "delete"}) {
|
||||
for (String operation : new String[] {"archive", "unarchive"}) {
|
||||
web(operation, tenant, session, "owner", "key").andExpect(status().isConflict())
|
||||
.andExpect(jsonPath("$.error.code").value("session_state_conflict"));
|
||||
}
|
||||
@@ -248,7 +254,7 @@ class WorkspaceSessionRetentionTest {
|
||||
String id = json(web("delete", tenant, session, "owner", "delete").andExpect(status().isAccepted()))
|
||||
.path("operationId").asText();
|
||||
await().untilAsserted(() -> assertThat(store.findOperation(tenant, session, id).orElseThrow().state()).isEqualTo("COMPLETED"));
|
||||
assertThat(runtime.calls.get(session)).isEqualTo(2);
|
||||
assertThat(runtime.calls.get(session)).isEqualTo(1);
|
||||
} finally {
|
||||
runtime.supportRemoved = false;
|
||||
}
|
||||
@@ -351,6 +357,15 @@ class WorkspaceSessionRetentionTest {
|
||||
|
||||
@TestConfiguration
|
||||
static class Configuration {
|
||||
@Bean RuntimeBindingRepository lifecycleBindings(javax.sql.DataSource source) {
|
||||
return new JdbcRuntimeBindingRepository(source, new AesGcmSecretProtector("test", new byte[32]));
|
||||
}
|
||||
@Bean @Primary HarnessConnector uninitializedHarness() {
|
||||
return new UnavailableHarnessConnector() {
|
||||
public boolean supportsLifecycle() { return true; }
|
||||
public void detachLifecycle(com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord operation) {}
|
||||
};
|
||||
}
|
||||
@Bean @Primary OnceCloseRuntime retentionRuntime() { return new OnceCloseRuntime(); }
|
||||
@Bean @Primary ManagedAgentStore retentionStore(JdbcTemplate jdbc, ObjectMapper mapper, ManagedWorkspaceRegistry registry) {
|
||||
var properties = new ManagedAgentProperties();
|
||||
@@ -368,10 +383,10 @@ class WorkspaceSessionRetentionTest {
|
||||
}
|
||||
public CompletionStage<Void> warm(String id) { throw new AssertionError("Unexpected warm"); }
|
||||
public CompletionStage<Void> drain(String id) { throw new AssertionError("Unexpected drain"); }
|
||||
public void requestWorkspaceClose(String tenant, String id) { record(id); }
|
||||
public void requestWorkspaceClose(String tenant, String id) { throw new AssertionError("Unexpected legacy close"); }
|
||||
public CompletionStage<Void> closeWorkspace(String tenant, String id) { record(id); return CompletableFuture.completedFuture(null); }
|
||||
private void record(String id) {
|
||||
if (calls.merge(id, 1, Integer::sum) > 2) { throw new AssertionError("Cleanup repeated after close"); }
|
||||
if (calls.merge(id, 1, Integer::sum) > 1) { throw new AssertionError("Cleanup repeated after close"); }
|
||||
}
|
||||
}
|
||||
}
|
||||
+12
@@ -228,6 +228,16 @@ public enum SurfaceRegistry {
|
||||
Surface.WEBSHELL, RuleClass.WORKSPACE_DISCOVERY,
|
||||
EnumSet.of(Capability.WORKSPACE_GET)),
|
||||
// ManagedSessionStoreController: internal writer routes.
|
||||
INTERNAL_EXECUTION_AUTHORIZE(Method.POST,
|
||||
"/internal/managed-session-store/v1/sessions/{sessionId}"
|
||||
+ "/execution:authorize",
|
||||
Surface.INTERNAL, RuleClass.INTERNAL_WRITER,
|
||||
EnumSet.of(Capability.STORE_EXECUTION_AUTHORIZE)),
|
||||
INTERNAL_LIFECYCLE_AUTHORIZE(Method.POST,
|
||||
"/internal/managed-session-store/v1/sessions/{sessionId}"
|
||||
+ "/lifecycle:authorize",
|
||||
Surface.INTERNAL, RuleClass.INTERNAL_WRITER,
|
||||
EnumSet.of(Capability.STORE_LIFECYCLE_AUTHORIZE)),
|
||||
INTERNAL_WRITER_ACQUIRE(Method.POST,
|
||||
"/internal/managed-session-store/v1/sessions/{sessionId}"
|
||||
+ "/writers:acquire",
|
||||
@@ -451,6 +461,8 @@ public enum SurfaceRegistry {
|
||||
AGENT_DEFINITION_CREATE,
|
||||
AGENT_DEFINITION_GET,
|
||||
AGENT_DEFINITION_UPDATE,
|
||||
STORE_EXECUTION_AUTHORIZE,
|
||||
STORE_LIFECYCLE_AUTHORIZE,
|
||||
STORE_WRITER_ACQUIRE,
|
||||
STORE_WRITER_RENEW,
|
||||
STORE_WRITER_SEAL,
|
||||
|
||||
+150
@@ -28,6 +28,8 @@ import com.alibaba.qwen.code.daemon.SubmitHarnessTurn;
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
|
||||
import com.alibaba.qwen.code.managedagent.store.AgentStateStore;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedActionStore;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationKind;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.SessionRecord;
|
||||
import com.alibaba.qwen.code.managedagent.store.WorkspaceExecutionStore;
|
||||
import com.alibaba.qwen.code.managedagent.store.WriterCredentialPolicy;
|
||||
@@ -67,6 +69,68 @@ class QwenHostedHarnessConnectorTest {
|
||||
private static final String SUBMIT_DIGEST =
|
||||
SubmitHarnessTurn.computePayloadDigest(SUBMIT_CONTENT);
|
||||
|
||||
@Test
|
||||
void lifecycleCapabilityFailsClosedWhenTheCachedClientWasClosed() {
|
||||
HostedHarnessClient client = mock(HostedHarnessClient.class);
|
||||
HostedHarnessCapabilities unsupported = mock(HostedHarnessCapabilities.class);
|
||||
HostedHarnessCapabilities supported = mock(HostedHarnessCapabilities.class);
|
||||
when(supported.getLifecycleProtocolVersion()).thenReturn(1);
|
||||
when(client.capabilities()).thenReturn(unsupported, supported)
|
||||
.thenThrow(new IllegalStateException("HostedHarnessClient is closed"));
|
||||
ManagedAgentProperties properties = properties();
|
||||
properties.getHarness().setWorkspaceFilesEnabled(true);
|
||||
AgentStateStore sessions = mock(AgentStateStore.class);
|
||||
WorkspaceExecutionStore execution = mock(WorkspaceExecutionStore.class);
|
||||
QwenHostedHarnessConnector connector =
|
||||
new QwenHostedHarnessConnector(properties, sessions, execution);
|
||||
ReflectionTestUtils.setField(connector, "client", client);
|
||||
|
||||
assertThat(connector.supportsLifecycle()).isFalse();
|
||||
assertThat(connector.supportsLifecycle()).isTrue();
|
||||
assertThat(connector.supportsLifecycle()).isFalse();
|
||||
verify(client, times(3)).capabilities();
|
||||
verifyNoInteractions(sessions, execution);
|
||||
}
|
||||
|
||||
@Test
|
||||
void successorDetachesTheOriginalSessionWithoutLoadingAnAttachment() {
|
||||
HostedHarnessClient client = mock(HostedHarnessClient.class);
|
||||
QwenHostedHarnessConnector successor = connector(client);
|
||||
OperationRecord operation = mock(OperationRecord.class);
|
||||
when(operation.tenantId()).thenReturn("tenant-a");
|
||||
when(operation.sessionId()).thenReturn(SESSION_ID);
|
||||
when(operation.operationId()).thenReturn("delete-1");
|
||||
when(operation.claimGeneration()).thenReturn(2L);
|
||||
|
||||
successor.detachLifecycle(operation);
|
||||
|
||||
verify(client).detachLifecycle(SESSION_ID, Map.of("operationId", "delete-1", "claimGeneration", 2L));
|
||||
verify(client, never()).createSession(any());
|
||||
verify(client, never()).loadSession(any());
|
||||
verify(client, never()).settleLifecycle(any(), any());
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(ints = {404, 409, 503})
|
||||
void successorOnlyIgnoresAnAbsentAttachmentDuringLifecycleDetach(int status) {
|
||||
HostedHarnessClient client = mock(HostedHarnessClient.class);
|
||||
OperationRecord operation = mock(OperationRecord.class);
|
||||
when(operation.tenantId()).thenReturn("tenant-a");
|
||||
when(operation.sessionId()).thenReturn(SESSION_ID);
|
||||
when(operation.operationId()).thenReturn("delete-1");
|
||||
when(operation.claimGeneration()).thenReturn(2L);
|
||||
DaemonHttpException failure = mock(DaemonHttpException.class);
|
||||
when(failure.getStatusCode()).thenReturn(status);
|
||||
doThrow(failure).when(client).detachLifecycle(SESSION_ID,
|
||||
Map.of("operationId", "delete-1", "claimGeneration", 2L));
|
||||
QwenHostedHarnessConnector successor = connector(client);
|
||||
if (status == 404) {
|
||||
assertThatCode(() -> successor.detachLifecycle(operation)).doesNotThrowAnyException();
|
||||
} else {
|
||||
assertThatThrownBy(() -> successor.detachLifecycle(operation)).isSameAs(failure);
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"hosted-workspace-files/1", "hosted-workspace-files/2"})
|
||||
void boundCreateConflictLoadsOriginalWorkspaceAndProfileAndRechecksCachedGrant(String profile) {
|
||||
@@ -561,6 +625,92 @@ class QwenHostedHarnessConnectorTest {
|
||||
verify(newClient).submitTurn(any());
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(booleans = {false, true})
|
||||
void lifecycleGenerationMismatchAdoptsWithoutRedispatchingTheDiscoveringCall(boolean settle) {
|
||||
HostedHarnessClient oldClient = mock(HostedHarnessClient.class);
|
||||
HostedHarnessCapabilities oldCapabilities = mock(HostedHarnessCapabilities.class);
|
||||
HarnessSessionRef staleRef = mock(HarnessSessionRef.class);
|
||||
when(oldCapabilities.getBootId()).thenReturn(BOOT_ID);
|
||||
when(oldClient.capabilities()).thenReturn(oldCapabilities);
|
||||
when(oldClient.loadSession(any(LoadHarnessSession.class))).thenReturn(staleRef);
|
||||
when(staleRef.getHarnessBootId()).thenReturn(BOOT_ID);
|
||||
when(staleRef.getApprovalMode()).thenReturn("default");
|
||||
AgentStateStore sessions = mock(AgentStateStore.class);
|
||||
SessionRecord session = new SessionRecord("tenant-a", SESSION_ID, "qwen-code", null,
|
||||
null, "ACTIVE", null, null, 0, 0, 0, 1, 1, null, 1,
|
||||
new ContextBinding("tenant-a", "selected-workspace", 1, "storage", "child",
|
||||
WorkspaceExecutionProfile.CONTEXT_CONFIG_REF, 1), "yolo", "hosted-workspace-files/1");
|
||||
when(sessions.requireSession("tenant-a", SESSION_ID)).thenReturn(session);
|
||||
ManagedAgentProperties properties = properties();
|
||||
properties.getHarness().setWorkspaceFilesEnabled(true);
|
||||
ManagedActionStore actions = mock(ManagedActionStore.class);
|
||||
when(actions.approvalMode("tenant-a", SESSION_ID)).thenReturn("default");
|
||||
QwenHostedHarnessConnector connector = new QwenHostedHarnessConnector(
|
||||
properties, sessions, mock(WorkspaceExecutionStore.class), actions);
|
||||
ReflectionTestUtils.setField(connector, "client", oldClient);
|
||||
connector.createOrLoad("tenant-a", SESSION_ID, true);
|
||||
OperationRecord operation = mock(OperationRecord.class);
|
||||
when(operation.tenantId()).thenReturn("tenant-a");
|
||||
when(operation.sessionId()).thenReturn(SESSION_ID);
|
||||
when(operation.operationId()).thenReturn("close-original");
|
||||
when(operation.claimGeneration()).thenReturn(7L);
|
||||
when(operation.kind()).thenReturn(OperationKind.CLOSE);
|
||||
Map<String, Object> authority = Map.of("operationId", "close-original", "claimGeneration", 7L);
|
||||
HostedHarnessGenerationException mismatch = mock(HostedHarnessGenerationException.class);
|
||||
when(mismatch.getActualBootId()).thenReturn(NEW_BOOT_ID);
|
||||
if (settle) {
|
||||
when(oldClient.settleLifecycle(any(), any())).thenThrow(mismatch);
|
||||
} else {
|
||||
doThrow(mismatch).when(oldClient).detachLifecycle(staleRef, authority);
|
||||
}
|
||||
|
||||
assertThatThrownBy(() -> {
|
||||
if (settle) {
|
||||
connector.settleLifecycle(operation);
|
||||
} else {
|
||||
connector.detachLifecycle(operation);
|
||||
}
|
||||
}).isSameAs(mismatch);
|
||||
verify(oldClient).close();
|
||||
assertThat(ReflectionTestUtils.getField(connector, "client")).isNull();
|
||||
if (settle) {
|
||||
verify(oldClient).settleLifecycle(any(), any());
|
||||
} else {
|
||||
verify(oldClient).detachLifecycle(staleRef, authority);
|
||||
}
|
||||
|
||||
HostedHarnessClient newClient = mock(HostedHarnessClient.class);
|
||||
HostedHarnessCapabilities newCapabilities = mock(HostedHarnessCapabilities.class);
|
||||
when(newCapabilities.getBootId()).thenReturn(NEW_BOOT_ID);
|
||||
when(newClient.capabilities()).thenReturn(newCapabilities);
|
||||
ReflectionTestUtils.setField(connector, "client", newClient);
|
||||
if (settle) {
|
||||
HarnessSessionRef freshRef = mock(HarnessSessionRef.class);
|
||||
when(freshRef.getHarnessBootId()).thenReturn(NEW_BOOT_ID);
|
||||
when(newClient.loadSession(any(LoadHarnessSession.class))).thenReturn(freshRef);
|
||||
Map<String, Object> request = Map.of("sessionKey", Map.of("tenantId", "tenant-a",
|
||||
"workspaceId", "selected-workspace", "sessionId", SESSION_ID),
|
||||
"kind", "close", "authority", authority);
|
||||
when(newClient.settleLifecycle(freshRef, request)).thenReturn(Map.of("marker", "ok"));
|
||||
|
||||
assertThat(connector.settleLifecycle(operation).path("marker").asText()).isEqualTo("ok");
|
||||
ArgumentCaptor<LoadHarnessSession> load = ArgumentCaptor.forClass(LoadHarnessSession.class);
|
||||
verify(newClient).loadSession(load.capture());
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> loadJson = ReflectionTestUtils.invokeMethod(load.getValue(), "toJson");
|
||||
assertThat(loadJson).containsEntry("lifecycleAuthority", authority)
|
||||
.doesNotContainKeys("passiveManagedRuntimeRecovery", "driveRuntimeRecovery", "cancellationTakeover");
|
||||
verify(newClient).settleLifecycle(freshRef, request);
|
||||
} else {
|
||||
connector.detachLifecycle(operation);
|
||||
verify(newClient).detachLifecycle(SESSION_ID, authority);
|
||||
verify(newClient, never()).loadSession(any());
|
||||
verify(newClient, never()).settleLifecycle(any(), any());
|
||||
}
|
||||
verify(newClient, never()).createSession(any());
|
||||
}
|
||||
|
||||
// R4-13: each call site must fetch the client AFTER resolving the
|
||||
// attachment — the resolution runs a create/load round trip during
|
||||
// which an adoption can close and rebuild the client; a receiver
|
||||
|
||||
+28
@@ -678,6 +678,34 @@ class WorkspaceRuntimeTest {
|
||||
authority.assertHeld(session.workspace(), fixture.record());
|
||||
}
|
||||
|
||||
@Test
|
||||
void legacyCloseHooksUseTheOriginalRuntimeAndStillCheckCurrentAuthorization() throws Exception {
|
||||
SessionRecord session = createSession("storage", ".");
|
||||
var fixture = transport(session);
|
||||
var runtimeSession = fixture.record().getSession();
|
||||
authority.claim(session.workspace(), fixture.record());
|
||||
long now = System.currentTimeMillis();
|
||||
jdbc.update("UPDATE managed_agent_session SET status = 'CLOSING' WHERE tenant_id = ? AND session_id = ?",
|
||||
session.tenantId(), session.sessionId());
|
||||
jdbc.update("INSERT INTO managed_agent_operation (tenant_id, session_id, operation_id, operation_kind, actor_digest,"
|
||||
+ " idempotency_key, request_digest, state, admission_stage, delivery_state, session_status_before, lease_until,"
|
||||
+ " available_at, created_at, updated_at) VALUES (?, ?, ?, 'CLOSE', 'actor', 'close', 'digest', 'PENDING',"
|
||||
+ " 'ACCEPTED', 'LEASED', 'ACTIVE', ?, ?, ?, ?)", session.tenantId(), session.sessionId(), UUID.randomUUID().toString(),
|
||||
now + 60_000, now, now, now);
|
||||
fixture.bindings().requestHarnessDrain(session.tenantId(), session.sessionId());
|
||||
when(fixture.http().control(any(), any(), any())).thenReturn(CompletableFuture.completedFuture(Map.of("state", "settled")));
|
||||
for (String kind : List.of("hook-execute", "hook-catalog")) {
|
||||
Map<String, Object> hook = hookControl(session, kind);
|
||||
fixture.transport().control(fixture.lease(), runtimeSession, hook).toCompletableFuture().join();
|
||||
verify(fixture.http()).control(fixture.lease(), runtimeSession, hook);
|
||||
}
|
||||
assertUnavailable(() -> fixture.transport().control(fixture.lease(), runtimeSession, mcpControl(session, "mcp-configure")));
|
||||
jdbc.update("UPDATE managed_workspace_access SET can_read = FALSE WHERE tenant_id = ?", session.tenantId());
|
||||
assertUnavailable(() -> fixture.transport().control(fixture.lease(), runtimeSession, hookControl(session, "hook-execute")));
|
||||
fixture.transport().control(fixture.lease(), runtimeSession, hookControl(session, "hook-status")).toCompletableFuture().join();
|
||||
authority.assertHeld(session.workspace(), fixture.record());
|
||||
}
|
||||
|
||||
private static Map<String, Object> hookControl(SessionRecord session, String kind) {
|
||||
return Map.of("kind", kind, "operationId", kind,
|
||||
"sessionKey", Map.of("tenantId", session.tenantId(),
|
||||
|
||||
+3
-3
@@ -45,7 +45,7 @@ class WorkspaceRuntimeTransportTest {
|
||||
.thenReturn(CompletableFuture.completedFuture(Map.of("state", "ACKNOWLEDGED")));
|
||||
assertThat(acknowledge(fixture)).containsEntry("state", "ACKNOWLEDGED");
|
||||
verify(fixture.resolver()).savedBinding(fixture.session().getHarnessSessionId());
|
||||
verify(fixture.resolver(), never()).resolve(any());
|
||||
verify(fixture.resolver(), never()).resolve(any(), any());
|
||||
verifyNoInteractions(fixture.ownership());
|
||||
verify(fixture.http()).acknowledgeCsi(fixture.lease(), fixture.session(), fixture.boot(), fixture.pod(),
|
||||
fixture.request(), fixture.capture());
|
||||
@@ -56,7 +56,7 @@ class WorkspaceRuntimeTransportTest {
|
||||
@Test
|
||||
void dedicatedWorkspaceExemptionDoesNotOpenTheExistingGenericRoutes() throws Exception {
|
||||
var fixture = fixture();
|
||||
when(fixture.resolver().resolve(any())).thenThrow(WorkspaceExecutionStore.unavailable());
|
||||
when(fixture.resolver().resolve(any(), any())).thenThrow(WorkspaceExecutionStore.unavailable());
|
||||
Map<String, Object> reference = Map.of("runtimeProtocol", 3, "callId", "original");
|
||||
unavailable(() -> fixture.transport().statusV3(fixture.lease(), fixture.session(), reference, 0));
|
||||
unavailable(() -> fixture.transport().cancelV3(fixture.lease(), fixture.session(), reference));
|
||||
@@ -100,7 +100,7 @@ class WorkspaceRuntimeTransportTest {
|
||||
3, true, null, null, 0, 1, null, Instant.now(), Instant.now()));
|
||||
unavailable(() -> acknowledge(fixture));
|
||||
verifyNoInteractions(fixture.http(), fixture.ownership());
|
||||
verify(fixture.resolver(), never()).resolve(any());
|
||||
verify(fixture.resolver(), never()).resolve(any(), any());
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
+52
@@ -2,12 +2,18 @@ package com.alibaba.qwen.code.managedagent.store;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.reset;
|
||||
import static org.mockito.Mockito.verifyNoInteractions;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.api.ApiException;
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
|
||||
import org.flywaydb.core.Flyway;
|
||||
import org.h2.jdbcx.JdbcDataSource;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.CsvSource;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
|
||||
/**
|
||||
@@ -18,6 +24,52 @@ import org.springframework.jdbc.core.JdbcTemplate;
|
||||
class ManagedSessionStoreCredentialTest {
|
||||
private static final String KEY = "0123456789abcdef0123456789abcdef";
|
||||
|
||||
@ParameterizedTest
|
||||
@CsvSource({"ordinary", "lifecycle", "acquire", "renew", "seal", "commit"})
|
||||
void lifecycleWriterPathsRefuseForeignCredentialsBeforeStateLookup(String route) {
|
||||
JdbcTemplate jdbc = mock(JdbcTemplate.class);
|
||||
when(jdbc.getDataSource()).thenReturn(new JdbcDataSource());
|
||||
ManagedSessionStore store = new ManagedSessionStore(jdbc);
|
||||
ManagedAgentProperties properties = new ManagedAgentProperties();
|
||||
properties.getSessionStore().setBindingKey(KEY);
|
||||
WriterCredentialPolicy policy = new WriterCredentialPolicy(properties);
|
||||
store.setCredentials(policy);
|
||||
reset(jdbc);
|
||||
|
||||
for (String token : new String[]{null, "self-minted-token-self-minted-token-0",
|
||||
policy.issue("foreign-tenant", "workspace", "session"),
|
||||
policy.issue("tenant", "foreign-workspace", "session"),
|
||||
policy.issue("tenant", "workspace", "foreign-session")}) {
|
||||
assertThatThrownBy(() -> invokeWriterPath(store, route, token))
|
||||
.isInstanceOfSatisfying(ApiException.class, error -> {
|
||||
assertThat(error.getStatus().value()).isEqualTo(403);
|
||||
assertThat(error.getCode()).isEqualTo("writer_credential_invalid");
|
||||
});
|
||||
verifyNoInteractions(jdbc);
|
||||
}
|
||||
}
|
||||
|
||||
private static void invokeWriterPath(ManagedSessionStore store, String route, String token) {
|
||||
var authority = new com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority("operation", 1);
|
||||
switch (route) {
|
||||
case "ordinary" -> store.authorizeOrdinary("tenant", "session", token,
|
||||
new ManagedSessionStoreModels.AuthorizeLifecycleRequest("workspace", "writer", 1));
|
||||
case "lifecycle" -> store.authorizeLifecycle("tenant", "session", token,
|
||||
new ManagedSessionStoreModels.AuthorizeLifecycleRequest("workspace", "writer", 1), authority);
|
||||
case "acquire" -> store.acquireWriter("tenant", "session", token,
|
||||
new ManagedSessionStoreModels.AcquireWriterRequest("workspace", "writer", 60_000L), authority);
|
||||
case "renew" -> store.renewWriter("tenant", "session", token,
|
||||
new ManagedSessionStoreModels.RenewWriterRequest("workspace", "writer", 1, 60_000L), authority);
|
||||
case "seal" -> store.sealWriter("tenant", "session", token,
|
||||
new ManagedSessionStoreModels.SealWriterRequest("workspace", "writer", 1), authority);
|
||||
case "commit" -> store.commit("tenant", "session", token,
|
||||
new ManagedSessionStoreModels.CommitTransactionRequest("workspace", "writer", 1, 0, 0,
|
||||
"transaction", "operation", "command", "0".repeat(64), 0, 0, 0, null, null, null,
|
||||
0, null, 1, "", "0".repeat(64), java.util.List.of()), authority);
|
||||
default -> throw new IllegalArgumentException(route);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void publicationWriterLockRequiresTheBoundCredential() {
|
||||
JdbcDataSource dataSource = new JdbcDataSource();
|
||||
|
||||
+252
@@ -0,0 +1,252 @@
|
||||
package com.alibaba.qwen.code.managedagent.store;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.api.ApiException;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.AcquireWriterRequest;
|
||||
import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.AuthorizeLifecycleRequest;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationKind;
|
||||
import java.net.URLEncoder;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.TimeZone;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.TimeoutException;
|
||||
import java.util.stream.Stream;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.parallel.ResourceLock;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.Arguments;
|
||||
import org.junit.jupiter.params.provider.MethodSource;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
|
||||
class WorkspaceLifecycleMySqlIT extends WorkspaceLifecycleStoreTest {
|
||||
private DriverManagerDataSource source;
|
||||
private JdbcTemplate admin;
|
||||
private String schema;
|
||||
|
||||
@BeforeEach
|
||||
void openSchema() {
|
||||
String url = System.getProperty("mysql.url");
|
||||
if (url == null || !url.matches("jdbc:mysql://[^/]+/[^?]+(?:\\?.*)?")) {
|
||||
throw new IllegalArgumentException("A MySQL test database URL is required");
|
||||
}
|
||||
String user = System.getProperty("mysql.user", "root");
|
||||
String password = System.getProperty("mysql.password", "");
|
||||
admin = new JdbcTemplate(new DriverManagerDataSource(url, user, password));
|
||||
schema = "l3_" + UUID.randomUUID().toString().replace("-", "");
|
||||
admin.execute("CREATE DATABASE " + schema);
|
||||
source = new DriverManagerDataSource(url.replaceFirst("/[^/?]+(?=\\?|$)", "/" + schema), user, password);
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
void removeSchema() {
|
||||
if (admin != null && schema != null) {
|
||||
admin.execute("DROP DATABASE IF EXISTS " + schema);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
Fixture fixture() {
|
||||
return new Fixture(source);
|
||||
}
|
||||
|
||||
@Test
|
||||
void lifecycleEffectsKeepOtherTenantExtensionRecordsUnlocked() throws Exception {
|
||||
var fixture = fixture();
|
||||
String scope = ManagedSessionStore.sessionScopeKey("neighbor", "other-session");
|
||||
String record = ManagedExtensionProjection.recordKey("other-session", "hook_execution", "other-hook");
|
||||
fixture.jdbc.update("INSERT INTO qwen_managed_session_extension_record (session_scope_key, record_key, tenant_id, workspace_id,"
|
||||
+ " session_id, domain, record_id, operation_hash, revision, record_resource_id, created_at, first_sequence)"
|
||||
+ " VALUES (?, ?, 'neighbor', 'other-workspace', 'other-session', 'hook_execution', 'other-hook', ?, 1, 'other-resource', 1, 1)",
|
||||
scope, record, "a".repeat(64));
|
||||
var entered = new CountDownLatch(1);
|
||||
var release = new CountDownLatch(1);
|
||||
try (var workers = Executors.newFixedThreadPool(2)) {
|
||||
var held = workers.submit(() -> fixture.transactions.execute(ignored -> {
|
||||
try {
|
||||
new WorkspaceLifecycleStoreTest() {
|
||||
@Override
|
||||
Fixture fixture() {
|
||||
return fixture;
|
||||
}
|
||||
}.hookReceiptsUseCompactProtocolIdentitiesWithAnIndentedApplicationMapper(OperationKind.DELETE, true);
|
||||
entered.countDown();
|
||||
if (!release.await(10, TimeUnit.SECONDS)) throw new AssertionError("Lifecycle hold was not released");
|
||||
return null;
|
||||
} catch (Exception error) {
|
||||
throw new IllegalStateException(error);
|
||||
}
|
||||
}));
|
||||
try {
|
||||
assertThat(entered.await(10, TimeUnit.SECONDS)).isTrue();
|
||||
var neighbor = workers.submit(() -> new JdbcTemplate(source).update(
|
||||
"UPDATE qwen_managed_session_extension_record SET revision = revision + 1 WHERE session_scope_key = ? AND record_key = ?",
|
||||
scope, record));
|
||||
assertThat(neighbor.get(2, TimeUnit.SECONDS)).isEqualTo(1);
|
||||
} finally {
|
||||
release.countDown();
|
||||
}
|
||||
held.get(10, TimeUnit.SECONDS);
|
||||
}
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT revision FROM qwen_managed_session_extension_record"
|
||||
+ " WHERE session_scope_key = ? AND record_key = ?", Long.class, scope, record)).isEqualTo(2);
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@MethodSource("legacyCloseClockZones")
|
||||
@ResourceLock("java.util.TimeZone")
|
||||
void legacyCloseUsesTheDatabaseEpochAcrossTimeZones(String jvmZone, String databaseZone, String connectionZone) {
|
||||
TimeZone previous = TimeZone.getDefault();
|
||||
try {
|
||||
TimeZone.setDefault(TimeZone.getTimeZone(jvmZone));
|
||||
String url = source.getUrl();
|
||||
source.setUrl(url + (url.contains("?") ? "&" : "?")
|
||||
+ "connectionTimeZone=" + URLEncoder.encode(connectionZone, StandardCharsets.UTF_8)
|
||||
+ "&sessionVariables=" + URLEncoder.encode("time_zone='" + databaseZone + "'", StandardCharsets.UTF_8));
|
||||
var fixture = fixture();
|
||||
var journal = new ManagedSessionStore(fixture.jdbc);
|
||||
var writer = fixture.transactions.execute(ignored -> journal.acquireWriter("tenant", fixture.session,
|
||||
"w".repeat(32), new AcquireWriterRequest("workspace", "original", 60_000L)));
|
||||
String id = fixture.transactions.execute(ignored -> fixture.store.beginWorkspaceLifecycle("tenant", fixture.session,
|
||||
OperationKind.CLOSE, "owner", "a".repeat(64), "legacy", "digest", true).operation().operationId());
|
||||
var operation = fixture.transactions.execute(ignored -> fixture.store.claimOperation("tenant", fixture.session,
|
||||
id, "worker", Duration.ofMinutes(1)).orElseThrow());
|
||||
fixture.bindings.requestHarnessDrain("tenant", fixture.session);
|
||||
var legacy = new AuthorizeLifecycleRequest("workspace", "original", writer.writerGeneration(), "legacy-close");
|
||||
var ordinary = new AuthorizeLifecycleRequest("workspace", "original", writer.writerGeneration());
|
||||
var execution = new WorkspaceExecutionStore(fixture.jdbc, new DataSourceTransactionManager(source));
|
||||
assertThat(operation.lifecycleProtocolVersion()).isZero();
|
||||
assertThat(fixture.transactions.<Boolean>execute(ignored -> WorkspaceLifecycleStore.legacyClose(fixture.jdbc, "tenant", fixture.session)))
|
||||
.isTrue();
|
||||
fixture.transactions.executeWithoutResult(ignored -> journal.authorizeOrdinary("tenant", fixture.session, "w".repeat(32), legacy));
|
||||
execution.authorizeLegacyClose(fixture.store.requireSession("tenant", fixture.session));
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeOrdinary(
|
||||
"tenant", fixture.session, "w".repeat(32), ordinary))).isInstanceOfSatisfying(ApiException.class,
|
||||
error -> assertThat(error.getCode()).isEqualTo("managed_session_lifecycle_active"));
|
||||
fixture.jdbc.update("UPDATE managed_agent_operation SET lease_until = UNIX_TIMESTAMP() * 1000 - 1000 WHERE operation_id = ?", id);
|
||||
assertThat(fixture.transactions.<Boolean>execute(ignored -> WorkspaceLifecycleStore.legacyClose(fixture.jdbc, "tenant", fixture.session)))
|
||||
.isFalse();
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeOrdinary(
|
||||
"tenant", fixture.session, "w".repeat(32), legacy))).isInstanceOfSatisfying(ApiException.class,
|
||||
error -> assertThat(error.getCode()).isEqualTo("managed_session_lifecycle_active"));
|
||||
assertThatThrownBy(() -> execution.authorizeLegacyClose(fixture.store.requireSession("tenant", fixture.session)))
|
||||
.hasMessageContaining("unavailable");
|
||||
} finally {
|
||||
TimeZone.setDefault(previous);
|
||||
}
|
||||
}
|
||||
|
||||
static Stream<Arguments> legacyCloseClockZones() {
|
||||
return Stream.of(new String[]{"Asia/Shanghai", "+00:00"}, new String[]{"UTC", "+08:00"})
|
||||
.flatMap(zones -> List.of("LOCAL", "UTC", "+08:00", "Asia/Shanghai").stream()
|
||||
.map(connection -> Arguments.of(zones[0], zones[1], connection)));
|
||||
}
|
||||
|
||||
@Test
|
||||
void admissionSerializesAnOrdinaryWriterOnThePlacementFence() throws Exception {
|
||||
var fixture = fixture();
|
||||
var journal = new ManagedSessionStore(fixture.jdbc);
|
||||
try (var pool = Executors.newSingleThreadExecutor()) {
|
||||
var entered = new CountDownLatch(1);
|
||||
var queued = new java.util.concurrent.atomic.AtomicReference<java.util.concurrent.Future<Throwable>>();
|
||||
fixture.transactions.execute(ignored -> {
|
||||
WorkspaceLifecycleStore.lockPlacement(fixture.jdbc, "tenant");
|
||||
queued.set(pool.submit(() -> {
|
||||
entered.countDown();
|
||||
try {
|
||||
fixture.transactions.execute(transaction -> journal.acquireWriter("tenant", fixture.session,
|
||||
"w".repeat(32), new AcquireWriterRequest("workspace", "late-writer", 60_000L)));
|
||||
return null;
|
||||
} catch (RuntimeException error) {
|
||||
return error;
|
||||
}
|
||||
}));
|
||||
try {
|
||||
assertThat(entered.await(5, TimeUnit.SECONDS)).isTrue();
|
||||
assertThrows(TimeoutException.class, () -> queued.get().get(100, TimeUnit.MILLISECONDS));
|
||||
} catch (InterruptedException error) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException(error);
|
||||
}
|
||||
return fixture.admit(OperationKind.DELETE);
|
||||
});
|
||||
assertThat(queued.get().get(5, TimeUnit.SECONDS)).isInstanceOf(ApiException.class);
|
||||
}
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM qwen_managed_session_journal_head", Integer.class)).isZero();
|
||||
}
|
||||
|
||||
@Test
|
||||
void ordinaryAuthorizationKeepsFenceLocksInsideTheTenant() throws Exception {
|
||||
var fixture = fixture();
|
||||
var journal = new ManagedSessionStore(fixture.jdbc);
|
||||
String token = "w".repeat(32);
|
||||
String otherTenant = "other-tenant";
|
||||
String otherSession = "other-session";
|
||||
var original = fixture.transactions.execute(ignored -> journal.acquireWriter("tenant", fixture.session,
|
||||
token, new AcquireWriterRequest("workspace", "writer", 60_000L)));
|
||||
var neighbor = fixture.transactions.execute(ignored -> journal.acquireWriter("tenant", "neighbor-session",
|
||||
token, new AcquireWriterRequest("workspace", "writer", 60_000L)));
|
||||
var other = fixture.transactions.execute(ignored -> journal.acquireWriter(otherTenant, otherSession,
|
||||
token, new AcquireWriterRequest("workspace", "writer", 60_000L)));
|
||||
fixture.bindings.requestHarnessDrain("unrelated-tenant", "unrelated-session");
|
||||
var originalRequest = new AuthorizeLifecycleRequest("workspace", "writer", original.writerGeneration());
|
||||
var neighborRequest = new AuthorizeLifecycleRequest("workspace", "writer", neighbor.writerGeneration());
|
||||
var otherRequest = new AuthorizeLifecycleRequest("workspace", "writer", other.writerGeneration());
|
||||
try (var pool = Executors.newFixedThreadPool(2)) {
|
||||
var entered = new CountDownLatch(2);
|
||||
var sameTenant = new java.util.concurrent.atomic.AtomicReference<java.util.concurrent.Future<?>>();
|
||||
fixture.transactions.executeWithoutResult(ignored -> {
|
||||
journal.authorizeOrdinary("tenant", fixture.session, token, originalRequest);
|
||||
sameTenant.set(pool.submit(() -> {
|
||||
entered.countDown();
|
||||
fixture.transactions.executeWithoutResult(transaction ->
|
||||
journal.authorizeOrdinary("tenant", "neighbor-session", token, neighborRequest));
|
||||
}));
|
||||
var differentTenant = pool.submit(() -> {
|
||||
entered.countDown();
|
||||
fixture.transactions.executeWithoutResult(transaction ->
|
||||
journal.authorizeOrdinary(otherTenant, otherSession, token, otherRequest));
|
||||
});
|
||||
try {
|
||||
assertThat(entered.await(5, TimeUnit.SECONDS)).isTrue();
|
||||
assertThrows(TimeoutException.class, () -> sameTenant.get().get(100, TimeUnit.MILLISECONDS));
|
||||
differentTenant.get(5, TimeUnit.SECONDS);
|
||||
} catch (Exception error) {
|
||||
throw new IllegalStateException(error);
|
||||
}
|
||||
});
|
||||
sameTenant.get().get(5, TimeUnit.SECONDS);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void aFailureAfterRetirementRollsBackTheEntireDeleteAndRetryReusesEffects() {
|
||||
var fixture = fixture();
|
||||
var operation = fixture.admit(OperationKind.DELETE);
|
||||
var effects = fixture.transactions.execute(ignored -> fixture.lifecycle.recoverEffects(operation));
|
||||
fixture.jdbc.execute("CREATE TRIGGER l3_delete_failure BEFORE UPDATE ON managed_agent_session FOR EACH ROW"
|
||||
+ " BEGIN IF NEW.status = 'DELETED' THEN SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'l3 injected crash'; END IF; END");
|
||||
assertThatThrownBy(() -> fixture.transactions.execute(ignored -> fixture.store.completeOperation("tenant", fixture.session,
|
||||
operation.operationId(), "worker", operation.claimGeneration(), true))).hasMessageContaining("l3 injected crash");
|
||||
assertThat(fixture.store.requireSession("tenant", fixture.session).status()).isEqualTo("DELETING");
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM qwen_output_session_retirement", Integer.class)).isZero();
|
||||
assertThat(fixture.store.findOperation("tenant", fixture.session, operation.operationId()).orElseThrow().receiptId()).isNull();
|
||||
fixture.jdbc.execute("DROP TRIGGER l3_delete_failure");
|
||||
assertThat(fixture.transactions.<com.fasterxml.jackson.databind.JsonNode>execute(ignored -> fixture.lifecycle.recoverEffects(operation)))
|
||||
.isEqualTo(effects);
|
||||
assertThat(fixture.transactions.<Boolean>execute(ignored -> fixture.store.completeOperation("tenant", fixture.session,
|
||||
operation.operationId(), "worker", operation.claimGeneration(), true))).isTrue();
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM qwen_output_session_retirement", Integer.class)).isOne();
|
||||
}
|
||||
}
|
||||
+583
@@ -0,0 +1,583 @@
|
||||
package com.alibaba.qwen.code.managedagent.store;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.ExtensionRecordJournal;
|
||||
import com.alibaba.qwen.code.managedagent.ManagedHookRecordContractTest;
|
||||
import com.alibaba.qwen.code.managedagent.api.ApiException;
|
||||
import com.alibaba.qwen.code.managedagent.api.WorkspaceSelection;
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationKind;
|
||||
import com.alibaba.qwen.code.managedagent.store.StoreModels.OperationRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.AesGcmSecretProtector;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeProvisionRequest;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeScope;
|
||||
import com.alibaba.qwen.code.runtimebroker.WorkspaceExecutionProfile;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.databind.SerializationFeature;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.Base64;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import java.util.stream.Stream;
|
||||
import org.flywaydb.core.Flyway;
|
||||
import org.h2.jdbcx.JdbcDataSource;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.Arguments;
|
||||
import org.junit.jupiter.params.provider.MethodSource;
|
||||
import org.springframework.beans.factory.support.DefaultListableBeanFactory;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.transaction.support.TransactionTemplate;
|
||||
|
||||
class WorkspaceLifecycleStoreTest {
|
||||
@ParameterizedTest
|
||||
@MethodSource("fencedHookTransitions")
|
||||
void lifecycleHookTransitionsRecheckNewEffectsAndKeepOriginalSettlement(boolean authority, boolean granted,
|
||||
String before, String after, boolean replacement, boolean accepted, boolean batch, boolean draining) throws Exception {
|
||||
var fixture = fixture();
|
||||
var store = new ManagedSessionStore(fixture.jdbc);
|
||||
store.setLifecycleExecution(new WorkspaceExecutionStore(fixture.jdbc,
|
||||
new DataSourceTransactionManager(fixture.jdbc.getDataSource())), fixture.store);
|
||||
var journal = fixture.transactions.execute(ignored -> new ExtensionRecordJournal(store, "tenant", "workspace", fixture.session).open());
|
||||
var templates = ManagedHookRecordContractTest.fixtures().get("templates");
|
||||
var data = new ManagedSessionStoreModels.CommitResource("hook-data", "hook-data", 1, 2,
|
||||
HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest("{}".getBytes(StandardCharsets.UTF_8))), "e30=");
|
||||
ObjectNode registration = templates.get("hook_registration").deepCopy();
|
||||
registration.withObject("/catalogRef").put("digest", data.digest());
|
||||
for (String state : List.of("admitted", "running", "settled")) {
|
||||
registration.withObject("/run").put("state", state);
|
||||
var request = journal.requestDomain("register-" + state, "hook_registration", registration, List.of(data), 1000);
|
||||
fixture.transactions.executeWithoutResult(ignored -> journal.commit(request));
|
||||
journal.committed(request);
|
||||
}
|
||||
ObjectNode execution = templates.get("hook_execution").deepCopy();
|
||||
execution.set("planRef", registration.get("catalogRef"));
|
||||
execution.set("inputRef", registration.get("catalogRef"));
|
||||
execution.withObject("/run").put("execution", "intent").putObject("runtime")
|
||||
.put("runtimeBindingId", "original-binding").put("generation", "1");
|
||||
for (String state : "intent".equals(before) ? List.of("intent")
|
||||
: "dispatch_started".equals(before) ? List.of("intent", "dispatch_started")
|
||||
: List.of("intent", "dispatch_started", "outcome_unknown")) {
|
||||
hookState(execution, state, registration.get("catalogRef"));
|
||||
var request = journal.requestDomain("seed-" + state, "hook_execution", execution, List.of(), 1000);
|
||||
fixture.transactions.executeWithoutResult(ignored -> journal.commit(request));
|
||||
journal.committed(request);
|
||||
}
|
||||
var operation = fixture.admit(OperationKind.DELETE);
|
||||
if (draining) fixture.jdbc.update("UPDATE qwen_runtime_harness_drain SET phase = 'DRAINING'");
|
||||
fixture.jdbc.update("UPDATE managed_workspace_access SET can_create = ?", granted);
|
||||
ObjectNode next = execution.deepCopy();
|
||||
hookState(next, after, registration.get("catalogRef"));
|
||||
if (replacement && !batch) next.withObject("/run/runtime").put("generation", "2");
|
||||
assertThat(ManagedHookRecords.isExecutionSuccessor(execution, next)).isTrue();
|
||||
var first = journal.requestDomain("fenced", "hook_execution", next, List.of(), 1000);
|
||||
var request = first;
|
||||
if (batch) {
|
||||
ObjectNode resumed = next.deepCopy();
|
||||
hookState(resumed, "running_attached", registration.get("catalogRef"));
|
||||
if (replacement) resumed.withObject("/run/runtime").put("generation", "2");
|
||||
assertThat(ManagedHookRecords.isExecutionSuccessor(next, resumed)).isTrue();
|
||||
var second = journal.requestDomain("fenced", "hook_execution", resumed, List.of(), 1000);
|
||||
String[] firstLines = new String(Base64.getDecoder().decode(first.recordBytesBase64()), StandardCharsets.UTF_8).split("\n");
|
||||
String[] secondLines = new String(Base64.getDecoder().decode(second.recordBytesBase64()), StandardCharsets.UTF_8).split("\n");
|
||||
var event = (ObjectNode) fixture.json.readTree(secondLines[0]);
|
||||
event.withObject("/managedSession").put("sequence", first.lastSequence() + 1).put("eventId", "hook_execution:" + (first.lastSequence() + 1));
|
||||
byte[] bytes = (firstLines[0] + "\n" + event + "\n" + firstLines[1] + "\n").getBytes(StandardCharsets.UTF_8);
|
||||
var closure = new java.util.ArrayList<>(first.resources());
|
||||
closure.addAll(second.resources());
|
||||
request = new ManagedSessionStoreModels.CommitTransactionRequest(first.workspaceId(), first.writerId(), first.writerGeneration(),
|
||||
first.expectedJournalRevision(), first.expectedCommittedSequence(), first.transactionId(), first.operation(), first.commandId(),
|
||||
first.contentDigest(), first.firstSequence(), first.lastSequence() + 1, 2, first.eventsDigest(), first.previousCommitDigest(),
|
||||
first.commitDigest(), first.activationEpoch(), null, 3, Base64.getEncoder().encodeToString(bytes),
|
||||
HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)), closure);
|
||||
}
|
||||
var commitRequest = request;
|
||||
long revision = fixture.jdbc.queryForObject("SELECT journal_revision FROM qwen_managed_session_journal_head", Long.class);
|
||||
int resources = fixture.jdbc.queryForObject("SELECT COUNT(*) FROM qwen_managed_session_resource", Integer.class);
|
||||
Runnable commit = () -> fixture.transactions.executeWithoutResult(ignored -> store.commit("tenant", fixture.session,
|
||||
"extension-writer-token-0123456789", commitRequest, authority ? WorkspaceLifecycleStore.authority(operation) : null));
|
||||
if (accepted && !batch) {
|
||||
commit.run();
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT journal_revision FROM qwen_managed_session_journal_head", Long.class))
|
||||
.isEqualTo(revision + 1);
|
||||
} else {
|
||||
assertThatThrownBy(commit::run).isInstanceOfSatisfying(ApiException.class, error -> {
|
||||
assertThat(error.getStatus()).isEqualTo(HttpStatus.CONFLICT);
|
||||
assertThat(error.getCode()).isEqualTo(accepted && batch ? "managed_session_extension_record_rejected"
|
||||
: draining ? "workspace_lifecycle_claim_fenced" : authority && !replacement
|
||||
? "workspace_lifecycle_authorization_revoked" : "workspace_lifecycle_admission_closed");
|
||||
if (accepted && batch) assertThat(error.getMessage()).contains("at most one Stage H record");
|
||||
});
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT journal_revision FROM qwen_managed_session_journal_head", Long.class)).isEqualTo(revision);
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM qwen_managed_session_resource", Integer.class)).isEqualTo(resources);
|
||||
}
|
||||
}
|
||||
|
||||
static Stream<Arguments> fencedHookTransitions() {
|
||||
Stream<Arguments> single = Stream.concat(Stream.of(true, false).flatMap(authority -> Stream.of(
|
||||
Arguments.of(authority, false, "intent", "dispatch_started", false, false),
|
||||
Arguments.of(authority, false, "intent", "outcome_unknown", false, false),
|
||||
Arguments.of(authority, false, "intent", "corrupt", false, false),
|
||||
Arguments.of(authority, false, "intent", "not_started_proven", false, true),
|
||||
Arguments.of(authority, false, "dispatch_started", "outcome_unknown", false, true),
|
||||
Arguments.of(authority, false, "dispatch_started", "settled", false, true),
|
||||
Arguments.of(authority, false, "outcome_unknown", "running_attached", false, true),
|
||||
Arguments.of(authority, true, "outcome_unknown", "running_attached", true, false))),
|
||||
Stream.of(Arguments.of(true, true, "intent", "dispatch_started", false, true),
|
||||
Arguments.of(true, true, "intent", "outcome_unknown", false, true))).map(arguments -> {
|
||||
var values = java.util.Arrays.copyOf(arguments.get(), 7);
|
||||
values[6] = false;
|
||||
return Arguments.of(values);
|
||||
});
|
||||
Stream<Arguments> existing = Stream.concat(single, Stream.of(Arguments.of(true, true, "intent", "outcome_unknown", true, false, true),
|
||||
Arguments.of(true, true, "intent", "outcome_unknown", false, true, true))).map(arguments -> {
|
||||
var values = java.util.Arrays.copyOf(arguments.get(), 8);
|
||||
values[7] = false;
|
||||
return Arguments.of(values);
|
||||
});
|
||||
return Stream.concat(existing, Stream.of(
|
||||
Arguments.of(true, true, "intent", "dispatch_started", false, false, false, true),
|
||||
Arguments.of(true, true, "intent", "outcome_unknown", false, false, false, true),
|
||||
Arguments.of(true, true, "intent", "not_started_proven", false, true, false, true)));
|
||||
}
|
||||
|
||||
private static void hookState(ObjectNode record, String execution, JsonNode result) {
|
||||
var run = record.withObject("/run").put("execution", execution).putNull("reason");
|
||||
if (List.of("outcome_unknown", "corrupt").contains(execution)) {
|
||||
run.put("state", "recovery_blocked").put("reason", "corrupt".equals(execution) ? "execution_corrupt" : execution);
|
||||
} else if ("settled".equals(execution)) {
|
||||
run.put("state", "settled");
|
||||
record.set("resultRef", result);
|
||||
} else if ("not_started_proven".equals(execution)) {
|
||||
run.put("state", "cancelled");
|
||||
} else {
|
||||
run.put("state", "intent".equals(execution) ? "admitted" : "running");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void lifecycleClaimRefusalRemainsANonRetryableBrokerConflict() {
|
||||
var fixture = fixture();
|
||||
var operation = fixture.admit(OperationKind.DELETE);
|
||||
var execution = new WorkspaceExecutionStore(fixture.jdbc, new DataSourceTransactionManager(fixture.jdbc.getDataSource()));
|
||||
var session = fixture.store.requireSession("tenant", fixture.session);
|
||||
execution.authorizeLifecycle(session, WorkspaceLifecycleStore.authority(operation));
|
||||
assertThatThrownBy(() -> execution.authorizeLifecycle(session,
|
||||
new RuntimeLifecycleAuthority(operation.operationId(), operation.claimGeneration() + 1)))
|
||||
.isInstanceOfSatisfying(RuntimeBrokerException.class, error -> {
|
||||
assertThat(error.getStatusCode()).isEqualTo(409);
|
||||
assertThat(error.getCode()).isEqualTo("workspace_lifecycle_claim_fenced");
|
||||
assertThat(error.isRetryable()).isFalse();
|
||||
});
|
||||
fixture.jdbc.update("UPDATE managed_agent_operation SET lease_until = 0");
|
||||
assertThatThrownBy(() -> execution.authorizeLifecycle(session, WorkspaceLifecycleStore.authority(operation)))
|
||||
.isInstanceOfSatisfying(RuntimeBrokerException.class, error -> {
|
||||
assertThat(error.getStatusCode()).isEqualTo(409);
|
||||
assertThat(error.getCode()).isEqualTo("workspace_lifecycle_claim_fenced");
|
||||
assertThat(error.isRetryable()).isFalse();
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void brokerAdmissionRejectsAnExpiredMillisecondLifecycleClaim() {
|
||||
var fixture = fixture();
|
||||
var operation = fixture.admit(OperationKind.DELETE);
|
||||
long now = fixture.jdbc.queryForObject("SELECT UNIX_TIMESTAMP(), EXTRACT(MICROSECOND FROM CURRENT_TIMESTAMP(6))",
|
||||
(row, index) -> row.getLong(1) * 1000 + row.getLong(2) / 1000);
|
||||
fixture.jdbc.update("UPDATE qwen_runtime_harness_drain SET claim_lease_until = ?", now - 1);
|
||||
assertThatThrownBy(() -> fixture.bindings.requireHarnessAdmission(fixture.scope(), fixture.session,
|
||||
WorkspaceLifecycleStore.authority(operation))).isInstanceOfSatisfying(RuntimeBrokerException.class, error -> {
|
||||
assertThat(error.getStatusCode()).isEqualTo(409);
|
||||
assertThat(error.getCode()).isEqualTo("runtime_admission_closed");
|
||||
assertThat(error.isRetryable()).isFalse();
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void deleteReclassifiesACompletedCloseUnderTheSessionLock() {
|
||||
var fixture = fixture();
|
||||
var close = fixture.admit(OperationKind.CLOSE);
|
||||
assertThat(fixture.transactions.<JsonNode>execute(ignored -> fixture.lifecycle.recoverEffects(close))).isNotNull();
|
||||
fixture.transactions.execute(ignored -> fixture.store.completeOperation("tenant", fixture.session, close.operationId(),
|
||||
close.leaseOwner(), close.claimGeneration(), true));
|
||||
assertThat(fixture.store.requireSession("tenant", fixture.session).status()).isEqualTo("CLOSED");
|
||||
var admission = fixture.transactions.execute(ignored -> fixture.store.beginWorkspaceLifecycle("tenant", fixture.session,
|
||||
OperationKind.DELETE, "owner", "a".repeat(64), "stale-active-delete", "digest", true, 1));
|
||||
assertThat(admission.operation().lifecycleProtocolVersion()).isZero();
|
||||
assertThat(admission.operation().sessionStatusBefore()).isEqualTo("CLOSED");
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM qwen_runtime_harness_drain", Integer.class)).isEqualTo(1);
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT phase FROM qwen_runtime_harness_drain", String.class)).isEqualTo("DRAINING");
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@MethodSource("invalidJournalRecords")
|
||||
void lifecycleJournalValidationRejectsInvalidRecordsAndRollsBack(String line, String mode) throws Exception {
|
||||
var fixture = fixture();
|
||||
var journal = new ManagedSessionStore(fixture.jdbc);
|
||||
var writer = fixture.transactions.execute(ignored -> journal.acquireWriter("tenant", fixture.session,
|
||||
"w".repeat(32), new ManagedSessionStoreModels.AcquireWriterRequest("workspace", "original", 60_000L)));
|
||||
var operation = "ordinary".equals(mode) ? null : fixture.admit(OperationKind.DELETE);
|
||||
byte[] body = "{}".getBytes(StandardCharsets.UTF_8);
|
||||
var resource = new ManagedSessionStoreModels.CommitResource("candidate", "managed-message", 1, body.length,
|
||||
HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(body)), Base64.getEncoder().encodeToString(body));
|
||||
byte[] bytes = (line + "\n{}\n").getBytes(StandardCharsets.UTF_8);
|
||||
var request = new ManagedSessionStoreModels.CommitTransactionRequest("workspace", "original", writer.writerGeneration(),
|
||||
0, 0, "transaction", "session.create", "command", "a".repeat(64), 0, 0, 0, null, null, null, 0, null, 2,
|
||||
Base64.getEncoder().encodeToString(bytes), HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)),
|
||||
List.of(resource));
|
||||
assertThatThrownBy(() -> fixture.transactions.execute(ignored -> journal.commit("tenant", fixture.session, "w".repeat(32),
|
||||
request, "authority".equals(mode) ? WorkspaceLifecycleStore.authority(operation) : null)))
|
||||
.isInstanceOfSatisfying(ApiException.class, error -> {
|
||||
assertThat(error.getStatus()).isEqualTo(HttpStatus.BAD_REQUEST);
|
||||
assertThat(error.getCode()).isEqualTo(ManagedSessionStoreModels.ERROR_INVALID_REQUEST);
|
||||
});
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT journal_revision FROM qwen_managed_session_journal_head", Long.class)).isZero();
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM qwen_managed_session_journal_tx", Integer.class)).isZero();
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM qwen_managed_session_resource", Integer.class)).isZero();
|
||||
}
|
||||
|
||||
static Stream<Arguments> invalidJournalRecords() {
|
||||
return Stream.of("{", "[]", "null", "{\"n\":1e999}", "{\"a\":1,\"a\":2}")
|
||||
.flatMap(line -> Stream.of("ordinary", "authority", "settlement").map(mode -> Arguments.of(line, mode)));
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@MethodSource("hookReceiptPaths")
|
||||
void hookReceiptsUseCompactProtocolIdentitiesWithAnIndentedApplicationMapper(OperationKind kind, boolean recover) throws Exception {
|
||||
var fixture = fixture();
|
||||
fixture.json.enable(SerializationFeature.INDENT_OUTPUT);
|
||||
var journal = new ManagedSessionStore(fixture.jdbc);
|
||||
var writer = fixture.transactions.execute(ignored -> journal.acquireWriter("tenant", fixture.session,
|
||||
"w".repeat(32), new ManagedSessionStoreModels.AcquireWriterRequest("workspace", "original", 60_000L)));
|
||||
byte[] definition = "{\"toolProfile\":\"hosted-workspace-files/1\",\"hookCatalog\":{}}".getBytes(StandardCharsets.UTF_8);
|
||||
String digest = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(definition));
|
||||
var header = fixture.json.createObjectNode().put("subtype", "managed_session_header_v1");
|
||||
header.putObject("managedSession").putObject("definitionRef").put("resourceId", "definition")
|
||||
.put("kind", "managed-session-definition").put("schemaVersion", 1).put("byteLength", definition.length).put("digest", digest);
|
||||
byte[] bytes = ("{\"subtype\":\"session_execution_engine\"}\n" + header + "\n").getBytes(StandardCharsets.UTF_8);
|
||||
var commit = new ManagedSessionStoreModels.CommitTransactionRequest("workspace", "original", writer.writerGeneration(),
|
||||
0, 0, "transaction", "session.create", "command", "a".repeat(64), 0, 0, 0, null, null, null, 0, null, 2,
|
||||
Base64.getEncoder().encodeToString(bytes), HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)),
|
||||
List.of(new ManagedSessionStoreModels.CommitResource("definition", "managed-session-definition", 1,
|
||||
definition.length, digest, Base64.getEncoder().encodeToString(definition))));
|
||||
fixture.transactions.execute(ignored -> journal.commit("tenant", fixture.session, "w".repeat(32), commit));
|
||||
var operation = fixture.admit(kind);
|
||||
var receipt = fixture.json.createObjectNode().put("protocolVersion", 1).put("operationId", operation.operationId())
|
||||
.put("kind", kind.name().toLowerCase(java.util.Locale.ROOT));
|
||||
receipt.putObject("sessionKey").put("tenantId", "tenant").put("workspaceId", "workspace").put("sessionId", fixture.session);
|
||||
receipt.set("definitionRef", header.path("managedSession").path("definitionRef"));
|
||||
var effects = receipt.putArray("effects");
|
||||
for (String event : kind == OperationKind.CLOSE ? List.of("SessionEnd") : List.of("SessionEnd", "SessionDelete")) {
|
||||
// These bytes are JSON.stringify([event, operationId]) on the Harness wire.
|
||||
String id = "hook-plan-" + HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(
|
||||
("[\"" + event + "\",\"" + operation.operationId() + "\"]").getBytes(StandardCharsets.UTF_8)));
|
||||
var plan = fixture.json.createObjectNode();
|
||||
plan.putObject("input").put("hook_event_name", event).put("session_id", fixture.session);
|
||||
var planRef = fixture.resource(event + "-plan", "hook-data", plan);
|
||||
var resultRef = fixture.resource(event + "-result", "hook-data", fixture.json.createObjectNode());
|
||||
var record = fixture.json.createObjectNode().put("hookExecutionId", id).put("occurrenceId", operation.operationId())
|
||||
.put("runtimeSessionId", "original").put("registrationId", "registration").put("eventName", event)
|
||||
.put("ordinal", 0).put("hookId", "__plan__").put("cancelRequested", false);
|
||||
record.putNull("onceKey");
|
||||
record.set("planRef", planRef);
|
||||
record.set("inputRef", planRef);
|
||||
record.set("resultRef", resultRef);
|
||||
var run = record.putObject("run").put("state", "settled").put("effectId", id).put("execution", "settled");
|
||||
run.putObject("definition").put("definitionId", "catalog").put("definitionRevision", 1).put("definitionDigest", "b".repeat(64));
|
||||
for (String key : List.of("reason", "executionCallId", "dispatchId", "deliveryId", "runtime", "delivery")) {
|
||||
run.putNull(key);
|
||||
}
|
||||
ManagedHookRecords.requireExecution(record);
|
||||
var ref = fixture.resource(event + "-record", "hook-execution", record);
|
||||
fixture.jdbc.update("INSERT INTO qwen_managed_session_extension_record (session_scope_key, record_key, tenant_id, workspace_id,"
|
||||
+ " session_id, domain, record_id, operation_hash, revision, record_resource_id, task_kind, task_state, created_at, first_sequence)"
|
||||
+ " VALUES (?, ?, 'tenant', 'workspace', ?, 'hook_execution', ?, ?, 1, ?, 'hook', 'settled', 1, 1)",
|
||||
ManagedSessionStore.sessionScopeKey("tenant", fixture.session),
|
||||
ManagedExtensionProjection.recordKey(fixture.session, "hook_execution", id),
|
||||
fixture.session, id, "a".repeat(64), ref.path("resourceId").asText());
|
||||
effects.addObject().put("event", event).set("recordRef", ref);
|
||||
}
|
||||
if (recover) {
|
||||
JsonNode recovered = fixture.transactions.execute(ignored -> fixture.lifecycle.recoverEffects(operation));
|
||||
assertThat(recovered).isNotNull();
|
||||
assertThat(fixture.json.readTree(recovered.toString())).isEqualTo(receipt);
|
||||
} else {
|
||||
fixture.transactions.executeWithoutResult(ignored -> fixture.lifecycle.saveEffects(operation, receipt));
|
||||
}
|
||||
assertThat(fixture.json.readTree(fixture.jdbc.queryForObject("SELECT lifecycle_effects_receipt_json FROM managed_agent_operation",
|
||||
String.class))).isEqualTo(receipt);
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT phase FROM qwen_runtime_harness_drain", String.class)).isEqualTo("DRAINING");
|
||||
fixture.transactions.executeWithoutResult(ignored -> journal.sealWriter("tenant", fixture.session, "w".repeat(32),
|
||||
new ManagedSessionStoreModels.SealWriterRequest("workspace", "original", writer.writerGeneration()),
|
||||
WorkspaceLifecycleStore.authority(operation)));
|
||||
fixture.transactions.executeWithoutResult(ignored -> fixture.lifecycle.verifyCompletion(operation));
|
||||
}
|
||||
|
||||
static Stream<Arguments> hookReceiptPaths() {
|
||||
return Stream.of(OperationKind.CLOSE, OperationKind.DELETE).flatMap(kind -> Stream.of(false, true).map(recover -> Arguments.of(kind, recover)));
|
||||
}
|
||||
|
||||
@Test
|
||||
void detachOnlyAuthorizationAcceptsOriginalExpiredOrSealedWriterAfterEffects() throws Exception {
|
||||
var fixture = fixture();
|
||||
var journal = new ManagedSessionStore(fixture.jdbc);
|
||||
var writer = fixture.transactions.execute(ignored -> journal.acquireWriter("tenant", fixture.session,
|
||||
"w".repeat(32), new ManagedSessionStoreModels.AcquireWriterRequest("workspace", "original", 60_000L)));
|
||||
byte[] definition = "{\"toolProfile\":\"hosted-workspace-files/1\"}".getBytes(StandardCharsets.UTF_8);
|
||||
String digest = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(definition));
|
||||
var header = new ObjectMapper().createObjectNode().put("subtype", "managed_session_header_v1");
|
||||
header.putObject("managedSession").putObject("definitionRef").put("resourceId", "definition")
|
||||
.put("kind", "managed-session-definition").put("schemaVersion", 1).put("byteLength", definition.length).put("digest", digest);
|
||||
byte[] bytes = ("{\"subtype\":\"session_execution_engine\"}\n" + header + "\n").getBytes(StandardCharsets.UTF_8);
|
||||
var commit = new ManagedSessionStoreModels.CommitTransactionRequest("workspace", "original", writer.writerGeneration(),
|
||||
0, 0, "transaction", "session.create", "command", "a".repeat(64), 0, 0, 0, null, null, null, 0, null, 2,
|
||||
Base64.getEncoder().encodeToString(bytes), HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)),
|
||||
List.of(new ManagedSessionStoreModels.CommitResource("definition", "managed-session-definition", 1,
|
||||
definition.length, digest, Base64.getEncoder().encodeToString(definition))));
|
||||
fixture.transactions.execute(ignored -> journal.commit("tenant", fixture.session, "w".repeat(32), commit));
|
||||
var operation = fixture.admit(OperationKind.DELETE);
|
||||
var authority = WorkspaceLifecycleStore.authority(operation);
|
||||
var cleanup = new ManagedSessionStoreModels.AuthorizeLifecycleRequest("workspace", "original", writer.writerGeneration());
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeLifecycle(
|
||||
"tenant", fixture.session, "w".repeat(32), cleanup, authority))).hasMessageContaining("blocked");
|
||||
assertThat(fixture.transactions.<com.fasterxml.jackson.databind.JsonNode>execute(ignored -> fixture.lifecycle.recoverEffects(operation)))
|
||||
.isNotNull();
|
||||
fixture.jdbc.update("UPDATE qwen_managed_session_journal_head SET writer_lease_until = ?", new java.sql.Timestamp(0));
|
||||
assertThat(journal.hasLiveWriter("tenant", fixture.session)).isFalse();
|
||||
fixture.transactions.executeWithoutResult(ignored -> journal.authorizeLifecycle("tenant", fixture.session, "w".repeat(32), cleanup, authority));
|
||||
var dispatch = new ManagedSessionStoreModels.AuthorizeLifecycleRequest("workspace", "original", writer.writerGeneration(), "delete");
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeLifecycle(
|
||||
"tenant", fixture.session, "w".repeat(32), dispatch, authority))).hasMessageContaining("writer");
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.renewWriter("tenant", fixture.session, "w".repeat(32),
|
||||
new ManagedSessionStoreModels.RenewWriterRequest("workspace", "original", writer.writerGeneration(), 60_000L), authority)))
|
||||
.hasMessageContaining("writer");
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeLifecycle(
|
||||
"tenant", fixture.session, "x".repeat(32), cleanup, authority))).hasMessageContaining("writer");
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeLifecycle("tenant", fixture.session,
|
||||
"w".repeat(32), new ManagedSessionStoreModels.AuthorizeLifecycleRequest("workspace", "original", writer.writerGeneration() + 1),
|
||||
authority))).hasMessageContaining("writer");
|
||||
fixture.transactions.executeWithoutResult(ignored -> journal.sealWriter("tenant", fixture.session, "w".repeat(32),
|
||||
new ManagedSessionStoreModels.SealWriterRequest("workspace", "original", writer.writerGeneration()), authority));
|
||||
fixture.transactions.executeWithoutResult(ignored -> journal.authorizeLifecycle("tenant", fixture.session, "w".repeat(32), cleanup, authority));
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeLifecycle(
|
||||
"tenant", fixture.session, "w".repeat(32), dispatch, authority))).hasMessageContaining("writer");
|
||||
fixture.jdbc.update("UPDATE managed_agent_operation SET lease_until = 0");
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeLifecycle(
|
||||
"tenant", fixture.session, "w".repeat(32), cleanup, authority))).hasMessageContaining("blocked");
|
||||
}
|
||||
|
||||
@Test
|
||||
void legacyCloseKeepsItsOriginalHookPathButCannotAdmitOrdinaryWork() {
|
||||
var fixture = fixture();
|
||||
var journal = new ManagedSessionStore(fixture.jdbc);
|
||||
var writer = fixture.transactions.execute(ignored -> journal.acquireWriter("tenant", fixture.session,
|
||||
"w".repeat(32), new ManagedSessionStoreModels.AcquireWriterRequest("workspace", "original", 60_000L)));
|
||||
String id = fixture.transactions.execute(ignored -> fixture.store.beginWorkspaceLifecycle("tenant", fixture.session,
|
||||
OperationKind.CLOSE, "owner", "a".repeat(64), "legacy", "digest", true).operation().operationId());
|
||||
var operation = fixture.transactions.execute(ignored -> fixture.store.claimOperation("tenant", fixture.session, id,
|
||||
"worker", Duration.ofSeconds(30)).orElseThrow());
|
||||
fixture.bindings.requestHarnessDrain("tenant", fixture.session);
|
||||
assertThat(operation.lifecycleProtocolVersion()).isZero();
|
||||
var ordinary = new ManagedSessionStoreModels.AuthorizeLifecycleRequest("workspace", "original", writer.writerGeneration());
|
||||
var legacy = new ManagedSessionStoreModels.AuthorizeLifecycleRequest("workspace", "original", writer.writerGeneration(), "legacy-close");
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeOrdinary("tenant", fixture.session,
|
||||
"w".repeat(32), ordinary))).hasMessageContaining("admission is closed");
|
||||
fixture.transactions.executeWithoutResult(ignored -> journal.authorizeOrdinary("tenant", fixture.session, "w".repeat(32), legacy));
|
||||
fixture.bindings.requireHookAdmission(fixture.scope(), fixture.session, null);
|
||||
assertThatThrownBy(() -> fixture.bindings.requireHarnessAdmission(fixture.scope(), fixture.session, null))
|
||||
.hasMessageContaining("closed");
|
||||
var execution = new WorkspaceExecutionStore(fixture.jdbc, new DataSourceTransactionManager(fixture.jdbc.getDataSource()));
|
||||
execution.authorizeLegacyClose(fixture.store.requireSession("tenant", fixture.session));
|
||||
fixture.jdbc.update("UPDATE managed_workspace_access SET can_create = FALSE");
|
||||
assertThatThrownBy(() -> execution.authorizeLegacyClose(fixture.store.requireSession("tenant", fixture.session)))
|
||||
.hasMessageContaining("unavailable");
|
||||
fixture.jdbc.update("UPDATE managed_workspace_access SET can_create = TRUE");
|
||||
execution.authorizeLegacyClose(fixture.store.requireSession("tenant", fixture.session));
|
||||
fixture.jdbc.update("UPDATE managed_agent_operation SET lease_until = 0");
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeOrdinary("tenant", fixture.session,
|
||||
"w".repeat(32), legacy))).hasMessageContaining("admission is closed");
|
||||
}
|
||||
|
||||
@Test
|
||||
void l3FenceCannotUseTheLegacyCloseException() {
|
||||
var fixture = fixture();
|
||||
var journal = new ManagedSessionStore(fixture.jdbc);
|
||||
var writer = fixture.transactions.execute(ignored -> journal.acquireWriter("tenant", fixture.session,
|
||||
"w".repeat(32), new ManagedSessionStoreModels.AcquireWriterRequest("workspace", "original", 60_000L)));
|
||||
fixture.admit(OperationKind.CLOSE);
|
||||
var legacy = new ManagedSessionStoreModels.AuthorizeLifecycleRequest("workspace", "original", writer.writerGeneration(), "legacy-close");
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeOrdinary("tenant", fixture.session,
|
||||
"w".repeat(32), legacy))).hasMessageContaining("admission is closed");
|
||||
assertThatThrownBy(() -> fixture.bindings.requireHookAdmission(fixture.scope(), fixture.session, null)).hasMessageContaining("closed");
|
||||
fixture.bindings.requestHarnessDrain("tenant", fixture.session);
|
||||
assertThatThrownBy(() -> fixture.bindings.requireHookAdmission(fixture.scope(), fixture.session, null)).hasMessageContaining("closed");
|
||||
}
|
||||
@Test
|
||||
void lifecycleAttachmentCannotDetachBeforeEffectsAreSaved() {
|
||||
var fixture = fixture();
|
||||
var journal = new ManagedSessionStore(fixture.jdbc);
|
||||
var writer = fixture.transactions.execute(ignored -> journal.acquireWriter("tenant", fixture.session,
|
||||
"w".repeat(32), new ManagedSessionStoreModels.AcquireWriterRequest("workspace", "original", 60_000L)));
|
||||
var operation = fixture.admit(OperationKind.DELETE);
|
||||
var unverified = new ObjectMapper().createObjectNode().put("protocolVersion", 1)
|
||||
.put("operationId", operation.operationId()).put("kind", "delete").put("neverInitialized", true);
|
||||
unverified.putObject("sessionKey").put("tenantId", "tenant").put("workspaceId", "workspace").put("sessionId", fixture.session);
|
||||
unverified.putArray("effects");
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> fixture.lifecycle.saveEffects(operation, unverified)))
|
||||
.hasMessageContaining("blocked");
|
||||
var request = new ManagedSessionStoreModels.AuthorizeLifecycleRequest("workspace", "original", writer.writerGeneration());
|
||||
fixture.transactions.executeWithoutResult(ignored -> journal.authorizeLifecycle("tenant", fixture.session,
|
||||
"w".repeat(32), new ManagedSessionStoreModels.AuthorizeLifecycleRequest("workspace", "original", writer.writerGeneration(), "delete"),
|
||||
WorkspaceLifecycleStore.authority(operation)));
|
||||
assertThatThrownBy(() -> fixture.transactions.executeWithoutResult(ignored -> journal.authorizeLifecycle("tenant", fixture.session,
|
||||
"w".repeat(32), request, WorkspaceLifecycleStore.authority(operation)))).hasMessageContaining("blocked");
|
||||
assertThat(journal.hasLiveWriter("tenant", fixture.session)).isTrue();
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT phase FROM qwen_runtime_harness_drain", String.class)).isEqualTo("LIFECYCLE_ONLY");
|
||||
}
|
||||
|
||||
@Test
|
||||
void completedBootstrapWithoutTheOriginalHeaderIsNotNeverInitializedEvidence() {
|
||||
var fixture = fixture();
|
||||
var operation = fixture.admit(OperationKind.DELETE);
|
||||
fixture.jdbc.update("UPDATE managed_agent_session SET harness_boot_id = 'original-boot'");
|
||||
assertThat(fixture.transactions.<com.fasterxml.jackson.databind.JsonNode>execute(ignored -> fixture.lifecycle.recoverEffects(operation))).isNull();
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT lifecycle_effects_receipt_json FROM managed_agent_operation", String.class)).isNull();
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT phase FROM qwen_runtime_harness_drain", String.class)).isEqualTo("LIFECYCLE_ONLY");
|
||||
}
|
||||
|
||||
@Test
|
||||
void emptySessionCompletesAtomicallyOnlyAfterEffectsAndPermanentFence() {
|
||||
var fixture = fixture();
|
||||
OperationRecord operation = fixture.admit(OperationKind.DELETE);
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT phase FROM qwen_runtime_harness_drain", String.class))
|
||||
.isEqualTo("LIFECYCLE_ONLY");
|
||||
assertThatThrownBy(() -> fixture.transactions.execute(ignored -> fixture.store.completeOperation(
|
||||
"tenant", fixture.session, operation.operationId(), "worker", operation.claimGeneration(), true)))
|
||||
.hasMessageContaining("blocked");
|
||||
var effects = fixture.transactions.execute(ignored -> fixture.lifecycle.recoverEffects(operation));
|
||||
assertThat(effects.path("neverInitialized").asBoolean()).isTrue();
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT receipt_id FROM managed_agent_operation", String.class)).isNull();
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT phase FROM qwen_runtime_harness_drain", String.class))
|
||||
.isEqualTo("DRAINING");
|
||||
assertThat(fixture.transactions.<Boolean>execute(ignored -> fixture.store.completeOperation(
|
||||
"tenant", fixture.session, operation.operationId(), "worker", operation.claimGeneration(), true))).isTrue();
|
||||
assertThat(fixture.store.requireSession("tenant", fixture.session).status()).isEqualTo("DELETED");
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM qwen_output_session_retirement", Integer.class)).isOne();
|
||||
assertThat(fixture.store.findOperation("tenant", fixture.session, operation.operationId()).orElseThrow().receiptId()).isNotNull();
|
||||
}
|
||||
|
||||
@Test
|
||||
void expiredClaimCannotSaveEffectsOrAdmitRuntimeAndTakeoverKeepsOperationIdentity() {
|
||||
var fixture = fixture();
|
||||
OperationRecord first = fixture.admit(OperationKind.DELETE);
|
||||
RuntimeScope scope = fixture.scope();
|
||||
fixture.bindings.requireHarnessAdmission(scope, fixture.session, WorkspaceLifecycleStore.authority(first));
|
||||
assertThatThrownBy(() -> fixture.bindings.requireHarnessAdmission(scope, fixture.session, null))
|
||||
.hasMessageContaining("closed");
|
||||
fixture.jdbc.update("UPDATE managed_agent_operation SET lease_until = 0");
|
||||
OperationRecord second = fixture.transactions.execute(ignored -> fixture.store.claimOperation(
|
||||
"tenant", fixture.session, first.operationId(), "second", Duration.ofSeconds(30)).orElseThrow());
|
||||
assertThat(second.claimGeneration()).isGreaterThan(first.claimGeneration());
|
||||
assertThatThrownBy(() -> fixture.bindings.requireHarnessAdmission(scope, fixture.session, WorkspaceLifecycleStore.authority(first)))
|
||||
.hasMessageContaining("closed");
|
||||
assertThatThrownBy(() -> fixture.transactions.execute(ignored -> fixture.lifecycle.recoverEffects(first)))
|
||||
.hasMessageContaining("blocked");
|
||||
fixture.bindings.requireHarnessAdmission(scope, fixture.session, WorkspaceLifecycleStore.authority(second));
|
||||
assertThat(fixture.transactions.<com.fasterxml.jackson.databind.JsonNode>execute(ignored -> fixture.lifecycle.recoverEffects(second))).isNotNull();
|
||||
assertThat(fixture.transactions.<Boolean>execute(ignored -> fixture.store.completeOperation(
|
||||
"tenant", fixture.session, first.operationId(), "worker", first.claimGeneration(), true))).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
void releasedBindingWithoutOriginalStopProofCannotComplete() {
|
||||
var fixture = fixture();
|
||||
var binding = fixture.bindings.findOrCreate(new RuntimeProvisionRequest(fixture.scope(), fixture.session));
|
||||
fixture.jdbc.update("UPDATE qwen_runtime_binding SET binding_state = 'RELEASED' WHERE binding_id = ?", binding.getBindingId());
|
||||
OperationRecord operation = fixture.admit(OperationKind.DELETE);
|
||||
fixture.transactions.execute(ignored -> fixture.lifecycle.recoverEffects(operation));
|
||||
assertThatThrownBy(() -> fixture.transactions.execute(ignored -> fixture.store.completeOperation(
|
||||
"tenant", fixture.session, operation.operationId(), "worker", operation.claimGeneration(), true)))
|
||||
.hasMessageContaining("proof");
|
||||
assertThat(fixture.store.requireSession("tenant", fixture.session).status()).isEqualTo("DELETING");
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM qwen_output_session_retirement", Integer.class)).isZero();
|
||||
}
|
||||
|
||||
Fixture fixture() {
|
||||
return new Fixture();
|
||||
}
|
||||
|
||||
static final class Fixture {
|
||||
final JdbcTemplate jdbc;
|
||||
final ObjectMapper json = new ObjectMapper();
|
||||
final TransactionTemplate transactions;
|
||||
final ManagedAgentStore store;
|
||||
final WorkspaceLifecycleStore lifecycle;
|
||||
final JdbcRuntimeBindingRepository bindings;
|
||||
final String session;
|
||||
|
||||
Fixture() {
|
||||
this(h2());
|
||||
}
|
||||
|
||||
private static javax.sql.DataSource h2() {
|
||||
var source = new JdbcDataSource();
|
||||
source.setURL("jdbc:h2:mem:l3-" + UUID.randomUUID() + ";MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE");
|
||||
return source;
|
||||
}
|
||||
|
||||
Fixture(javax.sql.DataSource source) {
|
||||
Flyway.configure().dataSource(source).locations("classpath:db/migration").load().migrate();
|
||||
jdbc = new JdbcTemplate(source);
|
||||
transactions = new TransactionTemplate(new DataSourceTransactionManager(source));
|
||||
var properties = new ManagedAgentProperties();
|
||||
properties.getHarness().setWorkspaceFilesEnabled(true);
|
||||
store = new ManagedAgentStore(jdbc, json, Clock.systemUTC(), ignored -> {}, new ManagedWorkspaceRegistry(jdbc), properties);
|
||||
bindings = new JdbcRuntimeBindingRepository(source, new AesGcmSecretProtector("test", new byte[32]));
|
||||
var beans = new DefaultListableBeanFactory();
|
||||
beans.registerSingleton("bindings", bindings);
|
||||
lifecycle = new WorkspaceLifecycleStore(jdbc, json, beans.getBeanProvider(RuntimeBindingRepository.class));
|
||||
store.setWorkspaceLifecycleStore(lifecycle);
|
||||
jdbc.update("INSERT INTO managed_workspace_registry (tenant_id, workspace_id, workspace_generation, storage_id,"
|
||||
+ " display_name, config_ref, policy_ref, state) VALUES ('tenant', 'workspace', 1, 'storage', 'Workspace', ?, ?, 'ACTIVE')",
|
||||
WorkspaceExecutionProfile.CONFIG_REF, WorkspaceExecutionProfile.POLICY_REF);
|
||||
jdbc.update("INSERT INTO managed_workspace_access (tenant_id, workspace_id, actor_id, can_read, can_create)"
|
||||
+ " VALUES ('tenant', 'workspace', ?, TRUE, TRUE)", "owner".getBytes(StandardCharsets.UTF_8));
|
||||
session = transactions.execute(ignored -> store.insertWorkspaceSessionCommand("tenant", "owner", "create", "digest", "qwen-code",
|
||||
null, null, List.of(), null, new WorkspaceSelection("workspace", ".")).sessionId());
|
||||
}
|
||||
|
||||
ObjectNode resource(String id, String kind, JsonNode body) throws Exception {
|
||||
byte[] bytes = body.toString().getBytes(StandardCharsets.UTF_8);
|
||||
String digest = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes));
|
||||
jdbc.update("INSERT INTO qwen_managed_session_resource (session_scope_key, tenant_id, workspace_id, session_id, resource_id, kind,"
|
||||
+ " schema_version, byte_length, sha256, storage_kind, inline_bytes, publish_command_id, state, created_at)"
|
||||
+ " VALUES (?, 'tenant', 'workspace', ?, ?, ?, 1, ?, ?, 'MYSQL_INLINE', ?, 'committed-hooks', 'REFERENCED', CURRENT_TIMESTAMP(6))",
|
||||
ManagedSessionStore.sessionScopeKey("tenant", session), session, id, kind, bytes.length, digest, bytes);
|
||||
return json.createObjectNode().put("resourceId", id).put("kind", kind).put("schemaVersion", 1).put("byteLength", bytes.length).put("digest", digest);
|
||||
}
|
||||
|
||||
RuntimeScope scope() {
|
||||
return new RuntimeScope("tenant", "workspace", "1", "/workspace", "digest", "session");
|
||||
}
|
||||
|
||||
OperationRecord admit(OperationKind kind) {
|
||||
String id = transactions.execute(ignored -> store.beginWorkspaceLifecycle("tenant", session, kind, "owner", "a".repeat(64),
|
||||
"operation", "digest", true, 1).operation().operationId());
|
||||
return transactions.execute(ignored -> store.claimOperation("tenant", session, id, "worker", Duration.ofSeconds(30)).orElseThrow());
|
||||
}
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -62,7 +62,7 @@ class WorkspaceMigrationMySqlIT {
|
||||
+ " WHERE installed_rank <= ? ORDER BY installed_rank", lastRank)).isEqualTo(applied);
|
||||
assertThat(jdbc.queryForList("SELECT version FROM flyway_schema_history"
|
||||
+ " WHERE installed_rank > ? AND success = TRUE ORDER BY installed_rank",
|
||||
String.class, lastRank)).containsExactly("48", "49", "50");
|
||||
String.class, lastRank)).containsExactly("48", "49", "50", "51");
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM managed_workspace_migration", Integer.class)).isZero();
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM qwen_runtime_storage_fence", Integer.class)).isZero();
|
||||
}
|
||||
|
||||
+9
@@ -10,10 +10,19 @@ public final class HostedHarnessCapabilities {
|
||||
private final List<Integer> supportedProtocolVersions;
|
||||
private final String bootId;
|
||||
private final String capabilityDigest;
|
||||
private final int lifecycleProtocolVersion;
|
||||
|
||||
public int getLifecycleProtocolVersion() { return lifecycleProtocolVersion; }
|
||||
|
||||
HostedHarnessCapabilities(int currentProtocolVersion,
|
||||
List<Integer> supportedProtocolVersions, String bootId,
|
||||
String capabilityDigest) {
|
||||
this(currentProtocolVersion, supportedProtocolVersions, bootId, capabilityDigest, 0);
|
||||
}
|
||||
|
||||
HostedHarnessCapabilities(int currentProtocolVersion, List<Integer> supportedProtocolVersions,
|
||||
String bootId, String capabilityDigest, int lifecycleProtocolVersion) {
|
||||
this.lifecycleProtocolVersion = lifecycleProtocolVersion;
|
||||
this.currentProtocolVersion = currentProtocolVersion;
|
||||
this.supportedProtocolVersions = Collections.unmodifiableList(
|
||||
new ArrayList<>(supportedProtocolVersions));
|
||||
|
||||
+54
-1
@@ -529,6 +529,58 @@ public final class HostedHarnessClient implements AutoCloseable {
|
||||
json);
|
||||
}
|
||||
|
||||
public Map<String, Object> settleLifecycle(HarnessSessionRef session, Map<String, Object> request) {
|
||||
HarnessSessionRef ref = requireSessionRef(session);
|
||||
if (capabilities.getLifecycleProtocolVersion() != 1) {
|
||||
throw new DaemonProtocolException("Hosted lifecycle protocol 1 is required");
|
||||
}
|
||||
String operation = "POST /session/:id/lifecycle";
|
||||
HttpSupport.Response response = sendMutation(sessionPath(ref.getHarnessSessionId()) + "/lifecycle",
|
||||
request, ref.getHarnessClientId(), operation);
|
||||
requireMutationStatus(response, 200, operation);
|
||||
Map<String, Object> receipt = JsonSupport.parseObject(response.getBody(), operation);
|
||||
if (!java.util.Objects.equals(request.get("sessionKey"), receipt.get("sessionKey"))
|
||||
|| !java.util.Objects.equals(request.get("kind"), receipt.get("kind"))) {
|
||||
throw new DaemonProtocolException("Hosted lifecycle receipt identity differs");
|
||||
}
|
||||
return receipt;
|
||||
}
|
||||
|
||||
public void detachLifecycle(HarnessSessionRef session, Map<String, Object> authority) {
|
||||
HarnessSessionRef ref = requireSessionRef(session);
|
||||
AttachmentState state = attachments.get(ref.getHarnessSessionId());
|
||||
ActivePrompt prompt = activePrompts.get(ref.getHarnessSessionId());
|
||||
HttpSupport.Response response = sendMutation(sessionPath(ref.getHarnessSessionId()) + "/detach",
|
||||
Map.of("authority", authority), ref.getHarnessClientId(), "POST /session/:id/detach");
|
||||
if (response.getStatusCode() != 404) {
|
||||
requireMutationStatus(response, 204, "POST /session/:id/detach");
|
||||
}
|
||||
if (state != null && state.matches(ref) && attachments.remove(ref.getHarnessSessionId(), state)) {
|
||||
state.cancel();
|
||||
if (prompt != null) {
|
||||
activePrompts.remove(ref.getHarnessSessionId(), prompt);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public void detachLifecycle(String harnessSessionId, Map<String, Object> authority) {
|
||||
ensureOpen();
|
||||
String sessionId = requireUuid(harnessSessionId, "harnessSessionId");
|
||||
AttachmentState state = attachments.get(sessionId);
|
||||
ActivePrompt prompt = activePrompts.get(sessionId);
|
||||
HttpSupport.Response response = sendMutation(sessionPath(sessionId) + "/detach",
|
||||
Map.of("authority", authority), null, "POST /session/:id/detach");
|
||||
if (response.getStatusCode() != 404) {
|
||||
requireMutationStatus(response, 204, "POST /session/:id/detach");
|
||||
}
|
||||
if (state != null && attachments.remove(sessionId, state)) {
|
||||
state.cancel();
|
||||
if (prompt != null) {
|
||||
activePrompts.remove(sessionId, prompt);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public void detachSession(HarnessSessionRef session) {
|
||||
HarnessSessionRef ref = requireSessionRef(session);
|
||||
HttpSupport.Response response = sendMutation(
|
||||
@@ -766,7 +818,8 @@ public final class HostedHarnessClient implements AutoCloseable {
|
||||
expectedDigest, digest);
|
||||
}
|
||||
return new HostedHarnessCapabilities(current, supported, bootId,
|
||||
digest);
|
||||
digest, hosted.containsKey("lifecycleProtocolVersion")
|
||||
? JsonSupport.requiredInt(hosted, "lifecycleProtocolVersion", "capabilities.hostedHarness") : 0);
|
||||
}
|
||||
|
||||
private HarnessSessionRef parseSession(String body,
|
||||
|
||||
+14
@@ -11,6 +11,17 @@ public final class LoadHarnessSession {
|
||||
private final String toolProfile;
|
||||
private final boolean driveRuntimeRecovery;
|
||||
private final boolean cancellationTakeover;
|
||||
private Map<String, Object> lifecycleAuthority;
|
||||
|
||||
public LoadHarnessSession forLifecycle(String operationId, long claimGeneration) {
|
||||
if (operationId == null || !operationId.matches("[A-Za-z0-9._:-]{1,128}") || claimGeneration < 1) {
|
||||
throw new IllegalArgumentException("Invalid lifecycle authority");
|
||||
}
|
||||
LoadHarnessSession copy = new LoadHarnessSession(harnessSessionId, managedSessionStore,
|
||||
passiveManagedRuntimeRecovery, toolProfile, driveRuntimeRecovery, cancellationTakeover);
|
||||
copy.lifecycleAuthority = Map.of("operationId", operationId, "claimGeneration", claimGeneration);
|
||||
return copy;
|
||||
}
|
||||
|
||||
public LoadHarnessSession(String harnessSessionId) {
|
||||
this(harnessSessionId, null, false);
|
||||
@@ -83,6 +94,9 @@ public final class LoadHarnessSession {
|
||||
if (driveRuntimeRecovery) {
|
||||
result.put("driveRuntimeRecovery", true);
|
||||
}
|
||||
if (lifecycleAuthority != null) {
|
||||
result.put("lifecycleAuthority", lifecycleAuthority);
|
||||
}
|
||||
if (cancellationTakeover) {
|
||||
result.put("cancellationTakeover", true);
|
||||
}
|
||||
|
||||
+209
@@ -20,11 +20,16 @@ import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.Arguments;
|
||||
import org.junit.jupiter.params.provider.MethodSource;
|
||||
import java.util.stream.Stream;
|
||||
|
||||
class HostedHarnessClientTest {
|
||||
private static final String BOOT_ID =
|
||||
@@ -72,6 +77,210 @@ class HostedHarnessClientTest {
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@MethodSource("lifecycleDetachResponses")
|
||||
void lifecycleDetachStopsHeartbeatOnlyAfterConfirmedAbsence(boolean byId, int status, boolean sameBoot) throws Exception {
|
||||
createSessionRoute();
|
||||
AtomicBoolean detached = new AtomicBoolean();
|
||||
CountDownLatch firstHeartbeat = new CountDownLatch(1);
|
||||
CountDownLatch laterHeartbeats = new CountDownLatch(2);
|
||||
server.createContext("/session/" + SESSION_ID + "/heartbeat", exchange -> {
|
||||
firstHeartbeat.countDown();
|
||||
if (detached.get()) laterHeartbeats.countDown();
|
||||
sendSessionJson(exchange, 200, "{\"sessionId\":\"" + SESSION_ID
|
||||
+ "\",\"clientId\":\"" + CLIENT_ID + "\",\"lastSeenAt\":123}");
|
||||
});
|
||||
server.createContext("/session/" + SESSION_ID + "/detach", exchange -> {
|
||||
assertPrivateHeaders(exchange, !byId);
|
||||
if (status == 204) {
|
||||
exchange.getResponseHeaders().set(HostedHarnessClient.BOOT_ID_HEADER, BOOT_ID);
|
||||
exchange.getResponseHeaders().set("Connection", "close");
|
||||
exchange.sendResponseHeaders(204, -1);
|
||||
exchange.close();
|
||||
} else {
|
||||
exchange.getResponseHeaders().set(HostedHarnessClient.BOOT_ID_HEADER, sameBoot ? BOOT_ID : OTHER_BOOT_ID);
|
||||
sendJson(exchange, status, "{\"code\":\"session_not_found\"}", false);
|
||||
}
|
||||
});
|
||||
try (HostedHarnessClient client = HostedHarnessClient.builder().baseUri(baseUri).bearerToken("harness-token")
|
||||
.capabilityDigest(DIGEST).heartbeatInterval(Duration.ofMillis(20)).build()) {
|
||||
HarnessSessionRef session = createSession(client);
|
||||
assertTrue(firstHeartbeat.await(2, TimeUnit.SECONDS));
|
||||
var authority = Map.<String, Object>of("operationId", "delete-1", "claimGeneration", 2);
|
||||
Runnable detach = () -> {
|
||||
if (byId) client.detachLifecycle(SESSION_ID, authority);
|
||||
else client.detachLifecycle(session, authority);
|
||||
};
|
||||
boolean confirmed = sameBoot && (status == 204 || status == 404);
|
||||
if (confirmed) detach.run();
|
||||
else {
|
||||
Class<? extends DaemonException> expected = sameBoot ? status == 500 ? MutationOutcomeUnknownException.class
|
||||
: DaemonHttpException.class : HostedHarnessGenerationException.class;
|
||||
assertThrows(expected, detach::run);
|
||||
}
|
||||
detached.set(true);
|
||||
assertEquals(!confirmed, laterHeartbeats.await(200, TimeUnit.MILLISECONDS));
|
||||
}
|
||||
}
|
||||
|
||||
static Stream<Arguments> lifecycleDetachResponses() {
|
||||
return Stream.of(true, false).flatMap(byId -> Stream.of(Arguments.of(byId, 204, true),
|
||||
Arguments.of(byId, 404, true), Arguments.of(byId, 403, true), Arguments.of(byId, 409, true),
|
||||
Arguments.of(byId, 500, true), Arguments.of(byId, 404, false)));
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@MethodSource("confirmedLifecycleDetachResponses")
|
||||
void lifecycleDetachReleasesOnlyTheOriginalPrompt(boolean byId, int status, boolean replacement) throws Exception {
|
||||
createSessionRoute();
|
||||
CountDownLatch detaching = new CountDownLatch(1);
|
||||
CountDownLatch reply = new CountDownLatch(1);
|
||||
server.createContext("/session/" + SESSION_ID + "/detach", exchange -> {
|
||||
assertPrivateHeaders(exchange, !byId);
|
||||
detaching.countDown();
|
||||
if (replacement) {
|
||||
try { assertTrue(reply.await(2, TimeUnit.SECONDS)); }
|
||||
catch (InterruptedException error) { Thread.currentThread().interrupt(); throw new IOException(error); }
|
||||
}
|
||||
exchange.getResponseHeaders().set(HostedHarnessClient.BOOT_ID_HEADER, BOOT_ID);
|
||||
if (status == 204) {
|
||||
exchange.getResponseHeaders().set("Connection", "close");
|
||||
exchange.sendResponseHeaders(204, -1);
|
||||
exchange.close();
|
||||
} else sendJson(exchange, status, "{\"code\":\"session_not_found\"}", false);
|
||||
});
|
||||
server.createContext("/session/" + SESSION_ID + "/load", exchange ->
|
||||
sendSessionJson(exchange, 200, sessionJson().replace(CLIENT_ID, "replacement-client")));
|
||||
AtomicInteger calls = new AtomicInteger();
|
||||
server.createContext("/session/" + SESSION_ID + "/prompt", exchange -> {
|
||||
int call = calls.incrementAndGet();
|
||||
assertEquals(call == 1 ? CLIENT_ID : "replacement-client",
|
||||
exchange.getRequestHeaders().getFirst(HostedHarnessClient.CLIENT_ID_HEADER));
|
||||
var request = JsonSupport.parseObject(readBody(exchange), "prompt");
|
||||
exchange.getResponseHeaders().set(HostedHarnessClient.BOOT_ID_HEADER, BOOT_ID);
|
||||
sendJson(exchange, 202, "{\"promptId\":\"" + request.get("promptId")
|
||||
+ "\",\"lastEventId\":0,\"eventEpoch\":\"" + EVENT_EPOCH + "\"}", false);
|
||||
});
|
||||
ExecutorService executor = Executors.newSingleThreadExecutor();
|
||||
try (HostedHarnessClient client = newClient()) {
|
||||
HarnessSessionRef original = createSession(client);
|
||||
Map<String, Object> block = Map.of("type", "text", "text", "first");
|
||||
client.submitTurn(requestForSession(block, original));
|
||||
var detach = executor.submit(() -> {
|
||||
var authority = Map.<String, Object>of("operationId", "delete", "claimGeneration", 1);
|
||||
if (byId) client.detachLifecycle(SESSION_ID, authority);
|
||||
else client.detachLifecycle(original, authority);
|
||||
});
|
||||
assertTrue(detaching.await(2, TimeUnit.SECONDS));
|
||||
HarnessSessionRef next;
|
||||
if (!replacement) detach.get(2, TimeUnit.SECONDS);
|
||||
next = client.loadSession(new LoadHarnessSession(SESSION_ID,
|
||||
ManagedSessionStoreConnection.builder().baseUri(URI.create("https://store.example/"))
|
||||
.tenantId("tenant").workspaceId("workspace").writerId(BOOT_ID)
|
||||
.leaseDuration(Duration.ofSeconds(45)).build()));
|
||||
if (replacement) {
|
||||
// A completed old prompt permits the replacement's new prompt before the old detach returns.
|
||||
server.createContext("/session/" + SESSION_ID + "/status", exchange -> {
|
||||
exchange.getResponseHeaders().set(HostedHarnessClient.BOOT_ID_HEADER, BOOT_ID);
|
||||
sendJson(exchange, 200, "{\"sessionId\":\"" + SESSION_ID + "\",\"hasActivePrompt\":false}", false);
|
||||
});
|
||||
client.getStatus(next);
|
||||
}
|
||||
var second = SubmitHarnessTurn.builder().session(next).promptId(SECOND_PROMPT_ID).addContent(block)
|
||||
.payloadDigest(SubmitHarnessTurn.computePayloadDigest(List.of(block))).build();
|
||||
client.submitTurn(second);
|
||||
reply.countDown();
|
||||
detach.get(2, TimeUnit.SECONDS);
|
||||
assertEquals(2, calls.get());
|
||||
var third = SubmitHarnessTurn.builder().session(next).promptId(OTHER_BOOT_ID).addContent(block)
|
||||
.payloadDigest(SubmitHarnessTurn.computePayloadDigest(List.of(block))).build();
|
||||
assertThrows(DaemonException.class, () -> client.submitTurn(third));
|
||||
assertEquals(2, calls.get());
|
||||
} finally {
|
||||
reply.countDown();
|
||||
executor.shutdownNow();
|
||||
}
|
||||
}
|
||||
|
||||
static Stream<Arguments> confirmedLifecycleDetachResponses() {
|
||||
return Stream.of(true, false).flatMap(byId -> Stream.of(204, 404)
|
||||
.flatMap(status -> Stream.of(true, false).map(replacement -> Arguments.of(byId, status, replacement))));
|
||||
}
|
||||
|
||||
@Test
|
||||
void lifecycleSettlementRetainsAttachmentAndDetachCarriesTheNewClaim() {
|
||||
server.removeContext("/capabilities");
|
||||
server.createContext("/capabilities", exchange -> sendJson(exchange, 200,
|
||||
capabilitiesJson(DIGEST, BOOT_ID).replace("\"hostedHarness\":{", "\"hostedHarness\":{\"lifecycleProtocolVersion\":1,"), false));
|
||||
createSessionRoute();
|
||||
AtomicReference<Map<String, Object>> settlement = new AtomicReference<>();
|
||||
AtomicReference<Map<String, Object>> detach = new AtomicReference<>();
|
||||
server.createContext("/session/" + SESSION_ID + "/lifecycle", exchange -> {
|
||||
assertEquals("POST", exchange.getRequestMethod());
|
||||
assertEquals(CLIENT_ID, exchange.getRequestHeaders().getFirst(HostedHarnessClient.CLIENT_ID_HEADER));
|
||||
settlement.set(JsonSupport.parseObject(readBody(exchange), "lifecycle"));
|
||||
sendSessionJson(exchange, 200, "{\"protocolVersion\":1,\"operationId\":\"delete-1\",\"kind\":\"delete\","
|
||||
+ "\"sessionKey\":{\"tenantId\":\"tenant\",\"workspaceId\":\"workspace\",\"sessionId\":\"" + SESSION_ID + "\"},\"effects\":[]}");
|
||||
});
|
||||
server.createContext("/session/" + SESSION_ID + "/detach", exchange -> {
|
||||
detach.set(JsonSupport.parseObject(readBody(exchange), "detach"));
|
||||
sendSessionNoContent(exchange);
|
||||
});
|
||||
try (HostedHarnessClient client = newClient()) {
|
||||
var session = createSession(client);
|
||||
var authority = Map.<String, Object>of("operationId", "delete-1", "claimGeneration", 2);
|
||||
var request = Map.<String, Object>of("kind", "delete", "sessionKey", Map.of("tenantId", "tenant", "workspaceId", "workspace",
|
||||
"sessionId", SESSION_ID), "authority", authority);
|
||||
assertEquals(1, client.capabilities().getLifecycleProtocolVersion());
|
||||
assertEquals("delete-1", client.settleLifecycle(session, request).get("operationId"));
|
||||
assertEquals(request, settlement.get());
|
||||
assertNull(detach.get());
|
||||
client.detachLifecycle(session, authority);
|
||||
assertEquals(Map.of("authority", authority), detach.get());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void successorLifecycleDetachNeedsNoNewAttachmentOrLifecycleDispatch() {
|
||||
AtomicReference<Map<String, Object>> detach = new AtomicReference<>();
|
||||
server.createContext("/session/" + SESSION_ID + "/detach", exchange -> {
|
||||
assertEquals("POST", exchange.getRequestMethod());
|
||||
detach.set(JsonSupport.parseObject(readBody(exchange), "detach"));
|
||||
assertPrivateHeaders(exchange, false);
|
||||
exchange.getResponseHeaders().set(HostedHarnessClient.BOOT_ID_HEADER, BOOT_ID);
|
||||
exchange.getResponseHeaders().set("Connection", "close");
|
||||
exchange.sendResponseHeaders(204, -1);
|
||||
exchange.close();
|
||||
});
|
||||
AtomicInteger otherMutations = new AtomicInteger();
|
||||
server.createContext("/session", exchange -> {
|
||||
otherMutations.incrementAndGet();
|
||||
sendSessionJson(exchange, 500, "{}");
|
||||
});
|
||||
try (HostedHarnessClient client = newClient()) {
|
||||
var authority = Map.<String, Object>of("operationId", "delete-1", "claimGeneration", 2);
|
||||
client.detachLifecycle(SESSION_ID, authority);
|
||||
assertEquals(Map.of("authority", authority), detach.get());
|
||||
assertEquals(0, otherMutations.get());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void legacyHarnessCannotFallBackToDeleteForLifecycleSettlement() {
|
||||
AtomicInteger requests = new AtomicInteger();
|
||||
createSessionRoute();
|
||||
server.createContext("/session/" + SESSION_ID, exchange -> {
|
||||
requests.incrementAndGet();
|
||||
sendSessionNoContent(exchange);
|
||||
});
|
||||
try (HostedHarnessClient client = newClient()) {
|
||||
var session = createSession(client);
|
||||
assertEquals(0, client.capabilities().getLifecycleProtocolVersion());
|
||||
assertThrows(DaemonProtocolException.class, () -> client.settleLifecycle(session, Map.of()));
|
||||
assertEquals(0, requests.get());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void negotiatesAndFencesSessionCreation() {
|
||||
AtomicReference<String> authorization = new AtomicReference<>();
|
||||
|
||||
+8
@@ -12,6 +12,14 @@ import java.util.concurrent.CompletionStage;
|
||||
public interface HarnessSessionResolver {
|
||||
CompletionStage<RuntimeScope> resolve(String harnessSessionId);
|
||||
|
||||
default CompletionStage<RuntimeScope> resolve(String harnessSessionId, RuntimeLifecycleAuthority authority) {
|
||||
if (authority != null) {
|
||||
return java.util.concurrent.CompletableFuture.failedFuture(new RuntimeBrokerException(
|
||||
409, "runtime_lifecycle_unavailable", "Lifecycle authority is unavailable", false));
|
||||
}
|
||||
return resolve(harnessSessionId);
|
||||
}
|
||||
|
||||
default CompletionStage<String> resolveTenant(String harnessSessionId) {
|
||||
return resolve(harnessSessionId).thenApply(RuntimeScope::getTenantId);
|
||||
}
|
||||
|
||||
+7
@@ -13,6 +13,13 @@ import java.util.function.Supplier;
|
||||
/** Process-local Runtime binding repository for tests and single-node use. */
|
||||
public final class InMemoryRuntimeBindingRepository
|
||||
implements RuntimeBindingRepository {
|
||||
@Override
|
||||
public void requireHookAdmission(RuntimeScope scope, String harnessSessionId,
|
||||
RuntimeLifecycleAuthority authority) {
|
||||
if (authority != null) {
|
||||
requireHarnessAdmission(scope, harnessSessionId, authority);
|
||||
}
|
||||
}
|
||||
private final java.util.Set<java.util.List<String>> draining = new java.util.HashSet<>();
|
||||
private final Map<List<String>, String> storageFences = new HashMap<>();
|
||||
private final Clock clock;
|
||||
|
||||
+147
-5
@@ -206,7 +206,12 @@ public final class JdbcRuntimeBindingRepository
|
||||
RuntimeBindingRecord binding = selectById(connection,
|
||||
candidate.getBindingId(), true);
|
||||
if (binding != null) {
|
||||
requireHarnessAdmission(connection, binding.getRequest());
|
||||
if (binding.getRequest().getStorageId() != null && storageFence(connection,
|
||||
binding.getRequest().getScope().getTenantId(), binding.getRequest().getStorageId()) != null) {
|
||||
throw new RuntimeBrokerException(409, "workspace_migrating", "Storage is under maintenance.", false);
|
||||
}
|
||||
requireHarnessAdmission(connection, candidate.getSession().getScope(), candidate.getSession().getHarnessSessionId(),
|
||||
candidate.getSession().getScope().getLifecycleAuthority());
|
||||
}
|
||||
RuntimeAdmission.requireReady(binding, candidate.getRuntimeGeneration());
|
||||
RuntimeScope scope = candidate.getSession().getScope();
|
||||
@@ -285,7 +290,7 @@ public final class JdbcRuntimeBindingRepository
|
||||
lockPlacementDomain(connection, tenantId);
|
||||
try (PreparedStatement statement = connection.prepareStatement(
|
||||
"INSERT INTO qwen_runtime_harness_drain (tenant_key, harness_key, tenant_id, harness_session_id) VALUES (?, ?, ?, ?)"
|
||||
+ " ON DUPLICATE KEY UPDATE harness_session_id = harness_session_id")) {
|
||||
+ " ON DUPLICATE KEY UPDATE phase = 'DRAINING', claim_lease_until = NULL")) {
|
||||
statement.setString(1, JdbcRepositorySupport.valueKey(tenantId));
|
||||
statement.setString(2, JdbcRepositorySupport.valueKey(harnessSessionId));
|
||||
statement.setString(3, tenantId);
|
||||
@@ -296,8 +301,26 @@ public final class JdbcRuntimeBindingRepository
|
||||
});
|
||||
}
|
||||
|
||||
public static String harnessDrainKey(String value) {
|
||||
return JdbcRepositorySupport.valueKey(value);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isHarnessDraining(String tenantId, String harnessSessionId) {
|
||||
return JdbcRepositorySupport.read(dataSource, connection -> {
|
||||
try (PreparedStatement statement = connection.prepareStatement("SELECT phase FROM qwen_runtime_harness_drain"
|
||||
+ " WHERE tenant_key = ? AND harness_key = ?")) {
|
||||
statement.setString(1, JdbcRepositorySupport.valueKey(tenantId));
|
||||
statement.setString(2, JdbcRepositorySupport.valueKey(harnessSessionId));
|
||||
try (ResultSet row = statement.executeQuery()) {
|
||||
return row.next() && "DRAINING".equals(row.getString(1));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isHarnessAdmissionClosed(String tenantId, String harnessSessionId) {
|
||||
return JdbcRepositorySupport.read(dataSource, connection -> hasHarnessDrain(connection, tenantId, harnessSessionId));
|
||||
}
|
||||
|
||||
@@ -326,10 +349,87 @@ public final class JdbcRuntimeBindingRepository
|
||||
request.getScope().getTenantId(), request.getStorageId()) != null) {
|
||||
throw new RuntimeBrokerException(409, "workspace_migrating", "Storage is under maintenance.", false);
|
||||
}
|
||||
if ("session".equals(request.getScope().getIsolationClass())
|
||||
&& hasHarnessDrain(connection, request.getScope().getTenantId(), request.getIsolationKey())) {
|
||||
throw new RuntimeBrokerException(409, "runtime_admission_closed", "Session is draining.", false);
|
||||
requireHarnessAdmission(connection, request.getScope(), request.getIsolationKey(),
|
||||
request.getScope().getLifecycleAuthority());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void requireHarnessAdmission(RuntimeScope scope, String harnessSessionId, RuntimeLifecycleAuthority authority) {
|
||||
JdbcRepositorySupport.transaction(dataSource, connection -> {
|
||||
lockPlacementDomain(connection, scope.getTenantId());
|
||||
requireHarnessAdmission(connection, scope, harnessSessionId, authority);
|
||||
return null;
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public void requireHookAdmission(RuntimeScope scope, String harnessSessionId, RuntimeLifecycleAuthority authority) {
|
||||
JdbcRepositorySupport.transaction(dataSource, connection -> {
|
||||
lockPlacementDomain(connection, scope.getTenantId());
|
||||
requireHarnessAdmission(connection, scope, harnessSessionId, authority, true);
|
||||
return null;
|
||||
});
|
||||
}
|
||||
|
||||
public static void requireHarnessAdmission(Connection connection, RuntimeScope scope,
|
||||
String harnessSessionId, RuntimeLifecycleAuthority authority) throws SQLException {
|
||||
requireHarnessAdmission(connection, scope, harnessSessionId, authority, false);
|
||||
}
|
||||
|
||||
private static void requireHarnessAdmission(Connection connection, RuntimeScope scope,
|
||||
String harnessSessionId, RuntimeLifecycleAuthority authority, boolean legacyHook) throws SQLException {
|
||||
if (!"session".equals(scope.getIsolationClass())) {
|
||||
if (authority != null) {
|
||||
throw admissionClosed();
|
||||
}
|
||||
return;
|
||||
}
|
||||
try (PreparedStatement statement = connection.prepareStatement("SELECT tenant_id, harness_session_id, phase,"
|
||||
+ " operation_id, claim_generation, claim_lease_until FROM qwen_runtime_harness_drain"
|
||||
+ " WHERE tenant_key = ? AND harness_key = ?")) {
|
||||
statement.setString(1, JdbcRepositorySupport.valueKey(scope.getTenantId()));
|
||||
statement.setString(2, JdbcRepositorySupport.valueKey(harnessSessionId));
|
||||
try (ResultSet row = statement.executeQuery()) {
|
||||
if (!row.next()) {
|
||||
if (authority != null) {
|
||||
throw admissionClosed();
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (!scope.getTenantId().equals(row.getString("tenant_id"))
|
||||
|| !harnessSessionId.equals(row.getString("harness_session_id"))) {
|
||||
throw admissionClosed();
|
||||
}
|
||||
if (legacyHook && authority == null && "DRAINING".equals(row.getString("phase"))
|
||||
&& row.getString("operation_id") == null) {
|
||||
return;
|
||||
}
|
||||
if (authority == null || !"LIFECYCLE_ONLY".equals(row.getString("phase"))
|
||||
|| !authority.operationId().equals(row.getString("operation_id"))
|
||||
|| authority.claimGeneration() != row.getLong("claim_generation")
|
||||
|| row.getLong("claim_lease_until") <= JdbcRepositorySupport.databaseNowPrecise(connection).toEpochMilli()) {
|
||||
throw admissionClosed();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public static void beginHarnessLifecycle(Connection connection, String tenantId, String sessionId, String operationId)
|
||||
throws SQLException {
|
||||
try (PreparedStatement statement = connection.prepareStatement("INSERT INTO qwen_runtime_harness_drain"
|
||||
+ " (tenant_key, harness_key, tenant_id, harness_session_id, phase, operation_id)"
|
||||
+ " VALUES (?, ?, ?, ?, 'LIFECYCLE_ONLY', ?)")) {
|
||||
statement.setString(1, JdbcRepositorySupport.valueKey(tenantId));
|
||||
statement.setString(2, JdbcRepositorySupport.valueKey(sessionId));
|
||||
statement.setString(3, tenantId);
|
||||
statement.setString(4, sessionId);
|
||||
statement.setString(5, operationId);
|
||||
statement.executeUpdate();
|
||||
}
|
||||
}
|
||||
|
||||
private static RuntimeBrokerException admissionClosed() {
|
||||
return new RuntimeBrokerException(409, "runtime_admission_closed", "Session admission is closed", false);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -442,6 +542,45 @@ public final class JdbcRuntimeBindingRepository
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public void verifyHarnessStopped(Connection connection, String tenantId, String harnessSessionId) throws SQLException {
|
||||
lockPlacementDomain(connection, tenantId);
|
||||
try (PreparedStatement fence = connection.prepareStatement("SELECT tenant_id, harness_session_id, phase"
|
||||
+ " FROM qwen_runtime_harness_drain WHERE tenant_key = ? AND harness_key = ? FOR UPDATE")) {
|
||||
fence.setString(1, JdbcRepositorySupport.valueKey(tenantId));
|
||||
fence.setString(2, JdbcRepositorySupport.valueKey(harnessSessionId));
|
||||
try (ResultSet row = fence.executeQuery()) {
|
||||
if (!row.next() || !tenantId.equals(row.getString(1)) || !harnessSessionId.equals(row.getString(2))
|
||||
|| !"DRAINING".equals(row.getString(3))) {
|
||||
throw admissionClosed();
|
||||
}
|
||||
}
|
||||
}
|
||||
try (PreparedStatement statement = connection.prepareStatement("SELECT " + BINDING_COLUMNS
|
||||
+ " FROM qwen_runtime_binding WHERE tenant_id = ? AND isolation_class = 'session'"
|
||||
+ " AND isolation_key = ? ORDER BY binding_id FOR UPDATE")) {
|
||||
statement.setString(1, tenantId);
|
||||
statement.setString(2, harnessSessionId);
|
||||
try (ResultSet rows = statement.executeQuery()) {
|
||||
while (rows.next()) {
|
||||
RuntimeBindingRecord record = mapBinding(rows);
|
||||
if (!tenantId.equals(record.getRequest().getScope().getTenantId())
|
||||
|| !harnessSessionId.equals(record.getRequest().getIsolationKey())) {
|
||||
continue;
|
||||
}
|
||||
if (record.getState() != RuntimeBindingRecord.State.RELEASED
|
||||
|| record.getDrainReceipt() == null && !record.hasStoppedWriters()) {
|
||||
throw new RuntimeBrokerException(409, "workspace_close_identity_unverified", "Original stop proof is missing", false);
|
||||
}
|
||||
if (JdbcRuntimeSessionRepository.hasActiveByBinding(connection, record)
|
||||
|| JdbcToolExecutionRepository.hasActiveByBinding(connection, record.getBindingId(), record.getGeneration())) {
|
||||
throw new RuntimeBrokerException(409, "workspace_close_execution_unsettled", "Original resources remain active", false);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public ToolExecutionRecord authorizeDispatch(RuntimeSessionRepository sessions,
|
||||
ToolExecutionRepository executions, ToolExecutionRecord expected,
|
||||
@@ -488,6 +627,9 @@ public final class JdbcRuntimeBindingRepository
|
||||
return active;
|
||||
}
|
||||
|
||||
if (request.getScope().getLifecycleAuthority() != null && slot.lastGeneration != 0) {
|
||||
throw new RuntimeBrokerException(409, "workspace_close_identity_unverified", "Lifecycle cannot replace an original Runtime", false);
|
||||
}
|
||||
requireRecoverablePlacement(connection, request);
|
||||
long generation = slot.lastGeneration + 1;
|
||||
String bindingId = BrokerValues.requireId(idSupplier.get(),
|
||||
|
||||
+4
@@ -33,6 +33,10 @@ public final class JdbcRuntimeBrokerSchema {
|
||||
addColumn(statement, "qwen_runtime_binding", "loss_evidence_json", "LONGTEXT");
|
||||
addColumn(statement, "qwen_runtime_binding", "stop_evidence_json", "LONGTEXT");
|
||||
addColumn(statement, "qwen_runtime_binding", "drain_receipt_json", "LONGTEXT");
|
||||
addColumn(statement, "qwen_runtime_harness_drain", "phase", "VARCHAR(32) NOT NULL DEFAULT 'DRAINING'");
|
||||
addColumn(statement, "qwen_runtime_harness_drain", "operation_id", "VARCHAR(128)");
|
||||
addColumn(statement, "qwen_runtime_harness_drain", "claim_generation", "BIGINT NOT NULL DEFAULT 0");
|
||||
addColumn(statement, "qwen_runtime_harness_drain", "claim_lease_until", "BIGINT");
|
||||
addColumn(statement, "qwen_tool_execution", "abandoned_at", "DATETIME(6)");
|
||||
addColumn(statement, "qwen_tool_execution", "loss_evidence_id", "VARCHAR(512)");
|
||||
addColumn(statement, "qwen_tool_execution", "authorized_dispatch_generation", "BIGINT");
|
||||
|
||||
+9
@@ -238,6 +238,15 @@ public final class RuntimeBindingRecord {
|
||||
return request;
|
||||
}
|
||||
|
||||
RuntimeBindingRecord withLifecycleAuthority(RuntimeLifecycleAuthority authority) {
|
||||
RuntimeProvisionRequest scoped = new RuntimeProvisionRequest(request.getScope().withLifecycleAuthority(authority),
|
||||
request.getIsolationKey(), request.getProvisionerKind(), request.getStorageId());
|
||||
return new RuntimeBindingRecord(bindingId, scoped, provisionSeed, generation, state, lease,
|
||||
resourceHandle, attestationGeneration, drainRequested, operationOwner, operationLeaseUntil,
|
||||
operationGeneration, version, lastHealthAt, lastReconciledAt, lastActiveAt,
|
||||
lossEvidence, stopEvidence, drainReceipt);
|
||||
}
|
||||
|
||||
public RuntimeProvisionSeed getProvisionSeed() {
|
||||
return provisionSeed;
|
||||
}
|
||||
|
||||
+21
@@ -57,6 +57,27 @@ public interface RuntimeBindingRepository {
|
||||
|
||||
boolean isHarnessDraining(String tenantId, String harnessSessionId);
|
||||
|
||||
default boolean isHarnessAdmissionClosed(String tenantId, String harnessSessionId) {
|
||||
return isHarnessDraining(tenantId, harnessSessionId);
|
||||
}
|
||||
|
||||
default void requireHarnessAdmission(RuntimeScope scope, String harnessSessionId,
|
||||
RuntimeLifecycleAuthority authority) {
|
||||
if (authority != null || isHarnessDraining(scope.getTenantId(), harnessSessionId)) {
|
||||
throw new RuntimeBrokerException(409, "runtime_admission_closed", "Session admission is closed", false);
|
||||
}
|
||||
}
|
||||
|
||||
default void requireHookAdmission(RuntimeScope scope, String harnessSessionId,
|
||||
RuntimeLifecycleAuthority authority) {
|
||||
requireHarnessAdmission(scope, harnessSessionId, authority);
|
||||
}
|
||||
|
||||
default void verifyHarnessStopped(java.sql.Connection connection, String tenantId, String harnessSessionId)
|
||||
throws java.sql.SQLException {
|
||||
throw new RuntimeBrokerException(409, "workspace_close_identity_unverified", "Stop verification is unavailable", false);
|
||||
}
|
||||
|
||||
List<RuntimeBindingRecord> findByHarnessSession(String tenantId, String harnessSessionId,
|
||||
String afterBindingId, int limit);
|
||||
|
||||
|
||||
+43
-14
@@ -110,10 +110,19 @@ public final class RuntimeBrokerHttpServer implements AutoCloseable {
|
||||
requireProtocol(body);
|
||||
JsonCodec.requiredString(body, "requestId", "warm request");
|
||||
String sessionId = JsonCodec.requiredString(body, "harnessSessionId", "warm request");
|
||||
complete(exchange, service.warm(sessionId), ignored -> Map.of(
|
||||
complete(exchange, service.warm(sessionId, lifecycleAuthority(exchange)), ignored -> Map.of(
|
||||
"protocolVersion", 1, "harnessSessionId", sessionId, "ready", true));
|
||||
return;
|
||||
}
|
||||
if ("POST".equals(exchange.getRequestMethod()) && "/runtimes:authorize-lifecycle".equals(relative)) {
|
||||
Map<String, Object> body = requestBody(exchange, "lifecycle authorization");
|
||||
requireProtocol(body);
|
||||
String sessionId = JsonCodec.requiredString(body, "harnessSessionId", "lifecycle authorization");
|
||||
String runtimeSessionId = JsonCodec.requiredString(body, "runtimeSessionId", "lifecycle authorization");
|
||||
complete(exchange, service.authorizeLifecycle(sessionId, lifecycleAuthority(exchange)),
|
||||
ignored -> envelope(sessionId, runtimeSessionId, "authorized", true));
|
||||
return;
|
||||
}
|
||||
if ("POST".equals(exchange.getRequestMethod())
|
||||
&& "/tool-sessions:acquire".equals(relative)) {
|
||||
acquire(exchange);
|
||||
@@ -145,6 +154,19 @@ public final class RuntimeBrokerHttpServer implements AutoCloseable {
|
||||
}
|
||||
}
|
||||
|
||||
private static RuntimeLifecycleAuthority lifecycleAuthority(HttpExchange exchange) {
|
||||
String operation = exchange.getRequestHeaders().getFirst(RuntimeLifecycleAuthority.OPERATION_HEADER);
|
||||
String generation = exchange.getRequestHeaders().getFirst(RuntimeLifecycleAuthority.GENERATION_HEADER);
|
||||
if (operation == null && generation == null) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
return new RuntimeLifecycleAuthority(operation, Long.parseLong(generation));
|
||||
} catch (RuntimeException error) {
|
||||
throw new RuntimeBrokerException(400, "runtime_lifecycle_authority_invalid", "Invalid lifecycle authority", false);
|
||||
}
|
||||
}
|
||||
|
||||
private void acquire(HttpExchange exchange) throws IOException {
|
||||
Map<String, Object> body = requestBody(exchange, "acquire request");
|
||||
requireProtocol(body);
|
||||
@@ -155,18 +177,25 @@ public final class RuntimeBrokerHttpServer implements AutoCloseable {
|
||||
"runtimeSessionId", "acquire request");
|
||||
String turnKind = JsonCodec.requiredString(body, "turnKind",
|
||||
"acquire request");
|
||||
complete(exchange, service.acquire(harnessSessionId, runtimeSessionId,
|
||||
turnKind), record -> {
|
||||
Map<String, Object> response = new LinkedHashMap<>(envelope(harnessSessionId,
|
||||
runtimeSessionId, "acquired", true));
|
||||
RuntimeScope scope = record.getSession().getScope();
|
||||
response.put("scope", Map.of("tenantId", scope.getTenantId(),
|
||||
"workspaceId", scope.getWorkspaceId(), "workspaceGeneration", scope.getWorkspaceGeneration(),
|
||||
"capabilityDigest", scope.getCapabilityDigest()));
|
||||
response.put("runtime", Map.of("bindingId", record.getBindingId(),
|
||||
"generation", Long.toString(record.getRuntimeGeneration())));
|
||||
return response;
|
||||
});
|
||||
CompletionStage<RuntimeSessionRecord> acquisition;
|
||||
if (body.containsKey("recoveryBindingId")) {
|
||||
acquisition = service.acquireRecovery(harnessSessionId, runtimeSessionId,
|
||||
JsonCodec.requiredString(body, "recoveryBindingId", "acquire request"),
|
||||
Long.parseLong(JsonCodec.requiredString(body, "recoveryGeneration", "acquire request")));
|
||||
} else {
|
||||
acquisition = service.acquire(harnessSessionId, runtimeSessionId, turnKind, lifecycleAuthority(exchange));
|
||||
}
|
||||
complete(exchange, acquisition, record -> {
|
||||
Map<String, Object> response = new LinkedHashMap<>(envelope(harnessSessionId,
|
||||
runtimeSessionId, "acquired", true));
|
||||
RuntimeScope scope = record.getSession().getScope();
|
||||
response.put("scope", Map.of("tenantId", scope.getTenantId(),
|
||||
"workspaceId", scope.getWorkspaceId(), "workspaceGeneration", scope.getWorkspaceGeneration(),
|
||||
"capabilityDigest", scope.getCapabilityDigest()));
|
||||
response.put("runtime", Map.of("bindingId", record.getBindingId(),
|
||||
"generation", Long.toString(record.getRuntimeGeneration())));
|
||||
return response;
|
||||
});
|
||||
}
|
||||
|
||||
private void toolSession(HttpExchange exchange, String suffix)
|
||||
@@ -202,7 +231,7 @@ public final class RuntimeBrokerHttpServer implements AutoCloseable {
|
||||
Map<String, Object> operation = requiredObject(body, "operation",
|
||||
"control request");
|
||||
complete(exchange, service.control(harnessSessionId,
|
||||
runtimeSessionId, operation), result -> envelope(
|
||||
runtimeSessionId, operation, lifecycleAuthority(exchange)), result -> envelope(
|
||||
harnessSessionId, runtimeSessionId, "result",
|
||||
result));
|
||||
return;
|
||||
|
||||
+119
-14
@@ -264,7 +264,9 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
|
||||
private CompletionStage<Void> drainBinding(RuntimeBindingRecord saved) {
|
||||
if (saved.getState() == RuntimeBindingRecord.State.RELEASED) {
|
||||
return CompletableFuture.completedFuture(null);
|
||||
return saved.getDrainReceipt() != null || saved.hasStoppedWriters()
|
||||
? CompletableFuture.completedFuture(null)
|
||||
: failed(conflict("workspace_close_identity_unverified", "Released binding has no original stop proof"));
|
||||
}
|
||||
if (!saved.getRequest().isManagedContext() || !provisioner.supportsDrainedStop()
|
||||
|| saved.getState() == RuntimeBindingRecord.State.OPERATOR_RECOVERY
|
||||
@@ -385,11 +387,15 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
|
||||
public CompletionStage<RuntimeBindingRecord> warm(
|
||||
String harnessSessionId) {
|
||||
return warm(harnessSessionId, null);
|
||||
}
|
||||
|
||||
public CompletionStage<RuntimeBindingRecord> warm(String harnessSessionId, RuntimeLifecycleAuthority authority) {
|
||||
requireOpen();
|
||||
String harnessId = BrokerValues.requirePathSafe(
|
||||
BrokerValues.requireId(harnessSessionId, "harnessSessionId"),
|
||||
"harnessSessionId");
|
||||
return resolveScope(harnessId)
|
||||
return resolveScope(harnessId, authority)
|
||||
.thenCompose(scope -> ensureBinding(
|
||||
provisionRequest(scope, harnessId)))
|
||||
.thenApply(BindingContext::record);
|
||||
@@ -398,6 +404,11 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
public CompletionStage<RuntimeSessionRecord> acquire(
|
||||
String harnessSessionId, String runtimeSessionId,
|
||||
String turnKind) {
|
||||
return acquire(harnessSessionId, runtimeSessionId, turnKind, null);
|
||||
}
|
||||
|
||||
public CompletionStage<RuntimeSessionRecord> acquire(String harnessSessionId, String runtimeSessionId,
|
||||
String turnKind, RuntimeLifecycleAuthority authority) {
|
||||
requireOpen();
|
||||
String harnessId = BrokerValues.requirePathSafe(
|
||||
BrokerValues.requireId(harnessSessionId, "harnessSessionId"),
|
||||
@@ -410,6 +421,7 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
return current.thenApply(context -> {
|
||||
context.lock();
|
||||
try {
|
||||
bindingRepository.requireHarnessAdmission(context.session().getScope(), harnessId, authority);
|
||||
requireSameSession(context.session(), new RuntimeSession(harnessId,
|
||||
runtimeId, turnKind, context.session().getScope()));
|
||||
return requireReadySessionRecord(context);
|
||||
@@ -418,15 +430,69 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
}
|
||||
});
|
||||
}
|
||||
return resolveScope(harnessId).thenCompose(scope -> {
|
||||
return resolveScope(harnessId, authority).thenCompose(scope -> {
|
||||
RuntimeSession session = new RuntimeSession(harnessId,
|
||||
runtimeId, turnKind, scope);
|
||||
return acquireSession(session);
|
||||
});
|
||||
}
|
||||
|
||||
public CompletionStage<RuntimeSessionRecord> acquireRecovery(String harnessSessionId, String runtimeSessionId,
|
||||
String bindingId, long generation) {
|
||||
requireOpen();
|
||||
RuntimeBindingRecord original = bindingRepository.findById(bindingId);
|
||||
if (original == null || original.getGeneration() != generation
|
||||
|| !harnessSessionId.equals(original.getRequest().getIsolationKey())
|
||||
|| !"session".equals(original.getRequest().getScope().getIsolationClass())
|
||||
|| original.getState() != RuntimeBindingRecord.State.READY) {
|
||||
return failed(conflict("workspace_close_identity_unverified", "Original Hook Runtime is unavailable"));
|
||||
}
|
||||
RuntimeSessionRecord saved = sessionRepository.findById(original.getRequest().getScope(), runtimeSessionId);
|
||||
if (saved == null || !harnessSessionId.equals(saved.getSession().getHarnessSessionId())
|
||||
|| !bindingId.equals(saved.getBindingId()) || saved.getRuntimeGeneration() != generation
|
||||
|| saved.getState() != RuntimeSessionRecord.State.READY) {
|
||||
return failed(conflict("workspace_close_identity_unverified", "Original Hook Session is unavailable"));
|
||||
}
|
||||
CompletionStage<RuntimeBindingRecord> recovered = matchingLiveBinding(original) == null
|
||||
? recoverBinding(bindingId, generation, true) : CompletableFuture.completedFuture(original);
|
||||
return recovered.thenApply(record -> {
|
||||
if (record.getState() != RuntimeBindingRecord.State.READY || !record.sameIdentity(original)) {
|
||||
throw conflict("workspace_close_identity_unverified", "Original Hook Runtime cannot be recovered");
|
||||
}
|
||||
BindingContext live = requireLiveBinding(record);
|
||||
var context = CompletableFuture.completedFuture(new SessionContext(saved.getSession(), record, live.lease()));
|
||||
var existing = sessions.putIfAbsent(runtimeSessionId, context);
|
||||
if (existing != null) {
|
||||
if (!existing.isDone() || existing.isCompletedExceptionally()) {
|
||||
throw conflict("workspace_close_identity_unverified", "Original Hook Session is not attested in this Broker");
|
||||
}
|
||||
SessionContext current = existing.join();
|
||||
requireSameSession(current.session(), saved.getSession());
|
||||
if (!current.binding().sameIdentity(original)) {
|
||||
throw conflict("workspace_close_identity_unverified", "Hook Runtime identity changed");
|
||||
}
|
||||
}
|
||||
return saved;
|
||||
});
|
||||
}
|
||||
|
||||
public CompletionStage<RuntimeScope> authorizeLifecycle(String harnessSessionId, RuntimeLifecycleAuthority authority) {
|
||||
if (authority == null) {
|
||||
return failed(conflict("runtime_lifecycle_authority_required", "Lifecycle authority is required"));
|
||||
}
|
||||
return resolveScope(harnessSessionId, authority).thenApply(scope -> {
|
||||
bindingRepository.requireHarnessAdmission(scope, harnessSessionId, authority);
|
||||
return scope;
|
||||
});
|
||||
}
|
||||
|
||||
public CompletionStage<Object> control(String harnessSessionId,
|
||||
String runtimeSessionId, Map<String, Object> operation) {
|
||||
return control(harnessSessionId, runtimeSessionId, operation, null);
|
||||
}
|
||||
|
||||
public CompletionStage<Object> control(String harnessSessionId,
|
||||
String runtimeSessionId, Map<String, Object> operation, RuntimeLifecycleAuthority authority) {
|
||||
requireOpen();
|
||||
Map<String, Object> immutable = immutableMap(operation,
|
||||
"operation");
|
||||
@@ -441,6 +507,17 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
if (ManagedHookProtocol.isOperation(immutable)) {
|
||||
ManagedHookProtocol.validateSession(context.session(), immutable);
|
||||
}
|
||||
boolean recovery = ManagedMcpProtocol.isRecovery(immutable) || ManagedHookProtocol.isRecovery(immutable);
|
||||
if (!recovery) {
|
||||
if (authority != null && !ManagedHookProtocol.isOperation(immutable)) {
|
||||
throw conflict("runtime_lifecycle_operation_invalid", "Only Hook control may use lifecycle authority");
|
||||
}
|
||||
if (ManagedHookProtocol.isOperation(immutable)) {
|
||||
bindingRepository.requireHookAdmission(context.session().getScope(), harnessSessionId, authority);
|
||||
} else {
|
||||
bindingRepository.requireHarnessAdmission(context.session().getScope(), harnessSessionId, authority);
|
||||
}
|
||||
}
|
||||
context.lock();
|
||||
try {
|
||||
requireReadySessionRecord(context);
|
||||
@@ -453,7 +530,9 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
requireUsableLease(context);
|
||||
result = mapFailure(safeStage(() ->
|
||||
transport.control(context.lease(),
|
||||
context.session(), immutable)),
|
||||
authority == null && context.session().getScope().getLifecycleAuthority() == null ? context.session()
|
||||
: new RuntimeSession(context.session().getHarnessSessionId(), context.session().getRuntimeSessionId(),
|
||||
context.session().getTurnKind(), context.session().getScope().withLifecycleAuthority(authority)), immutable)),
|
||||
"runtime_control_failed",
|
||||
"Runtime control operation failed");
|
||||
} catch (RuntimeException | Error failure) {
|
||||
@@ -2016,6 +2095,8 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
return selected.thenApply(context -> {
|
||||
context.lock();
|
||||
try {
|
||||
bindingRepository.requireHarnessAdmission(session.getScope(), session.getHarnessSessionId(),
|
||||
session.getScope().getLifecycleAuthority());
|
||||
requireSameSession(context.session(), session);
|
||||
return requireReadySessionRecord(context);
|
||||
} finally {
|
||||
@@ -2094,6 +2175,12 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
private CompletionStage<BindingContext> ensureBinding(
|
||||
RuntimeProvisionRequest request) {
|
||||
RuntimeBindingRecord record = bindingRepository.findOrCreate(request);
|
||||
if (request.getScope().getLifecycleAuthority() != null
|
||||
&& (record.getState() == RuntimeBindingRecord.State.LOST
|
||||
|| record.getState() == RuntimeBindingRecord.State.DRAINING
|
||||
|| record.getState() == RuntimeBindingRecord.State.RELEASED)) {
|
||||
return failed(conflict("workspace_close_identity_unverified", "Original lifecycle Runtime is unavailable"));
|
||||
}
|
||||
if (record.getState() == RuntimeBindingRecord.State.READY) {
|
||||
CompletableFuture<BindingContext> finishing =
|
||||
bindingOperations.get(record.getBindingId());
|
||||
@@ -2169,7 +2256,7 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
if (existing != null) {
|
||||
return existing;
|
||||
}
|
||||
safeStage(() -> provisionBinding(record)).whenComplete(
|
||||
safeStage(() -> provisionBinding(record, request.getScope().getLifecycleAuthority())).whenComplete(
|
||||
(context, error) -> {
|
||||
bindingOperations.remove(record.getBindingId(), created);
|
||||
if (error == null) {
|
||||
@@ -2182,14 +2269,15 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
}
|
||||
|
||||
private CompletionStage<BindingContext> provisionBinding(
|
||||
RuntimeBindingRecord record) {
|
||||
RuntimeBindingRecord claimed = bindingRepository.claimOperation(
|
||||
RuntimeBindingRecord record, RuntimeLifecycleAuthority authority) {
|
||||
RuntimeBindingRecord savedClaim = bindingRepository.claimOperation(
|
||||
record.getBindingId(), brokerOwnerId,
|
||||
operationLeaseDuration);
|
||||
if (claimed == null) {
|
||||
if (savedClaim == null) {
|
||||
return failed(unavailable("runtime_provisioning_in_progress",
|
||||
"another Broker owns Runtime provisioning"));
|
||||
}
|
||||
RuntimeBindingRecord claimed = savedClaim.withLifecycleAuthority(authority);
|
||||
if (claimed.getState()
|
||||
!= RuntimeBindingRecord.State.PROVISIONING) {
|
||||
if (claimed.getState() == RuntimeBindingRecord.State.READY
|
||||
@@ -3035,6 +3123,10 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
|
||||
/** Trusted maintenance of the saved generation; never resolves current authorization or provisions a replacement. */
|
||||
public CompletionStage<RuntimeBindingRecord> recoverBinding(String bindingId, long expectedGeneration) {
|
||||
return recoverBinding(bindingId, expectedGeneration, false);
|
||||
}
|
||||
|
||||
private CompletionStage<RuntimeBindingRecord> recoverBinding(String bindingId, long expectedGeneration, boolean lookupOnly) {
|
||||
requireOpen();
|
||||
RuntimeBindingRecord record = bindingRepository.findById(bindingId);
|
||||
if (record == null || record.getGeneration() != expectedGeneration
|
||||
@@ -3043,9 +3135,9 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
}
|
||||
// Close owns retirement after journal settlement. Observing its retired
|
||||
// registration as journal loss would erase the clean-stop boundary.
|
||||
if (record.getState() != RuntimeBindingRecord.State.LOST
|
||||
if (!lookupOnly && record.getState() != RuntimeBindingRecord.State.LOST
|
||||
&& "session".equals(record.getRequest().getScope().getIsolationClass())
|
||||
&& (bindingRepository.isHarnessDraining(record.getRequest().getScope().getTenantId(),
|
||||
&& (bindingRepository.isHarnessAdmissionClosed(record.getRequest().getScope().getTenantId(),
|
||||
record.getRequest().getIsolationKey()) || record.getRequest().getStorageId() != null
|
||||
&& bindingRepository.isStorageFenced(record.getRequest().getScope().getTenantId(),
|
||||
record.getRequest().getStorageId(), null))) {
|
||||
@@ -3074,15 +3166,18 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
if (claimed == null || claimed.getGeneration() != expectedGeneration) {
|
||||
return failed(unavailable("runtime_reconcile_in_progress", "Another Broker owns Runtime recovery"));
|
||||
}
|
||||
if (claimed.getState() != RuntimeBindingRecord.State.LOST
|
||||
if (!lookupOnly && claimed.getState() != RuntimeBindingRecord.State.LOST
|
||||
&& "session".equals(claimed.getRequest().getScope().getIsolationClass())
|
||||
&& (bindingRepository.isHarnessDraining(claimed.getRequest().getScope().getTenantId(),
|
||||
&& (bindingRepository.isHarnessAdmissionClosed(claimed.getRequest().getScope().getTenantId(),
|
||||
claimed.getRequest().getIsolationKey()) || claimed.getRequest().getStorageId() != null
|
||||
&& bindingRepository.isStorageFenced(claimed.getRequest().getScope().getTenantId(),
|
||||
claimed.getRequest().getStorageId(), null))) {
|
||||
return CompletableFuture.completedFuture(claimed);
|
||||
}
|
||||
if (claimed.getState() == RuntimeBindingRecord.State.LOST) {
|
||||
if (lookupOnly) {
|
||||
return failed(conflict("workspace_close_identity_unverified", "Original Hook Runtime is lost"));
|
||||
}
|
||||
return cleanupLost(claimed);
|
||||
}
|
||||
if (!canReconcile(claimed) && claimed.getState() != RuntimeBindingRecord.State.DRAINING) {
|
||||
@@ -3095,6 +3190,9 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
if (observed == null) {
|
||||
return CompletableFuture.completedFuture(current);
|
||||
}
|
||||
if (lookupOnly && observed.getOutcome() != RuntimeObservation.Outcome.READY) {
|
||||
return failed(conflict("workspace_close_identity_unverified", "Original Hook Runtime is unverified"));
|
||||
}
|
||||
if (observed.getOutcome() == RuntimeObservation.Outcome.NOT_FOUND
|
||||
&& observed.getLossEvidence() != null) {
|
||||
RuntimeBindingRecord lost = bindingRepository.compareAndSet(current,
|
||||
@@ -3328,6 +3426,9 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
private void beginDispatch(SessionContext context,
|
||||
ToolExecutionRecord prepared, Map<String, Object> payload,
|
||||
RuntimePublicationGrant grant) {
|
||||
if (!prepared.isCancelRequested()) {
|
||||
bindingRepository.requireHarnessAdmission(context.session().getScope(), context.session().getHarnessSessionId(), null);
|
||||
}
|
||||
if (payload == null && ("deferred".equals(prepared.getReference().get("dispatchMode"))
|
||||
|| "deferred_v3".equals(prepared.getReference().get("dispatchMode")))
|
||||
&& !prepared.isCancelRequested()) {
|
||||
@@ -4101,10 +4202,14 @@ public final class RuntimeBrokerService implements AutoCloseable {
|
||||
provisioner.close();
|
||||
}
|
||||
|
||||
private CompletionStage<RuntimeScope> resolveScope(String harnessSessionId) {
|
||||
return resolveScope(harnessSessionId, null);
|
||||
}
|
||||
|
||||
private CompletionStage<RuntimeScope> resolveScope(
|
||||
String harnessSessionId) {
|
||||
String harnessSessionId, RuntimeLifecycleAuthority authority) {
|
||||
return mapFailure(safeStage(
|
||||
() -> sessionResolver.resolve(harnessSessionId)),
|
||||
() -> sessionResolver.resolve(harnessSessionId, authority)),
|
||||
"runtime_scope_resolution_failed",
|
||||
"Runtime scope resolution failed").thenApply(scope -> {
|
||||
if (scope == null) {
|
||||
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
package com.alibaba.qwen.code.runtimebroker;
|
||||
|
||||
/** Request-scoped authority; never part of persisted Runtime identity. */
|
||||
public record RuntimeLifecycleAuthority(String operationId, long claimGeneration) {
|
||||
public static final String OPERATION_HEADER = "X-Qwen-Lifecycle-Operation-Id";
|
||||
public static final String GENERATION_HEADER = "X-Qwen-Lifecycle-Claim-Generation";
|
||||
|
||||
public RuntimeLifecycleAuthority {
|
||||
BrokerValues.requirePathSafe(BrokerValues.requireId(operationId, "operationId"), "operationId");
|
||||
if (claimGeneration < 1) {
|
||||
throw new IllegalArgumentException("Lifecycle claim generation must be positive");
|
||||
}
|
||||
}
|
||||
}
|
||||
+16
@@ -10,10 +10,26 @@ public final class RuntimeScope {
|
||||
private final String canonicalCwd;
|
||||
private final String capabilityDigest;
|
||||
private final String isolationClass;
|
||||
private final RuntimeLifecycleAuthority lifecycleAuthority;
|
||||
|
||||
public RuntimeLifecycleAuthority getLifecycleAuthority() {
|
||||
return lifecycleAuthority;
|
||||
}
|
||||
|
||||
public RuntimeScope withLifecycleAuthority(RuntimeLifecycleAuthority authority) {
|
||||
return new RuntimeScope(tenantId, workspaceId, workspaceGeneration,
|
||||
canonicalCwd, capabilityDigest, isolationClass, authority);
|
||||
}
|
||||
|
||||
public RuntimeScope(String tenantId, String workspaceId,
|
||||
String workspaceGeneration, String canonicalCwd,
|
||||
String capabilityDigest, String isolationClass) {
|
||||
this(tenantId, workspaceId, workspaceGeneration, canonicalCwd, capabilityDigest, isolationClass, null);
|
||||
}
|
||||
|
||||
private RuntimeScope(String tenantId, String workspaceId, String workspaceGeneration,
|
||||
String canonicalCwd, String capabilityDigest, String isolationClass, RuntimeLifecycleAuthority authority) {
|
||||
this.lifecycleAuthority = authority;
|
||||
this.tenantId = BrokerValues.requireId(tenantId, "tenantId");
|
||||
this.workspaceId = BrokerValues.requireId(workspaceId, "workspaceId");
|
||||
this.workspaceGeneration = BrokerValues.requireId(
|
||||
|
||||
+4
@@ -148,6 +148,10 @@ CREATE TABLE IF NOT EXISTS qwen_runtime_harness_drain (
|
||||
harness_key VARCHAR(64) NOT NULL,
|
||||
tenant_id VARCHAR(512) NOT NULL,
|
||||
harness_session_id VARCHAR(512) NOT NULL,
|
||||
phase VARCHAR(32) NOT NULL DEFAULT 'DRAINING',
|
||||
operation_id VARCHAR(128),
|
||||
claim_generation BIGINT NOT NULL DEFAULT 0,
|
||||
claim_lease_until BIGINT,
|
||||
PRIMARY KEY (tenant_key, harness_key)
|
||||
);
|
||||
|
||||
|
||||
+113
@@ -107,6 +107,80 @@ class RuntimeBrokerServiceTest {
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void hookRecoveryUsesTheOriginalBindingAndNeverAcquiresAReplacement() {
|
||||
try (Fixture fixture = new Fixture(SESSION_SCOPE)) {
|
||||
RuntimeSessionRecord original = join(fixture.service.acquire("harness-a", "runtime-a", "bootstrap"));
|
||||
int acquires = fixture.transport.acquireCalls.get();
|
||||
int provisions = fixture.provisioner.calls.get();
|
||||
assertSame(original, join(fixture.service.acquireRecovery("harness-a", "runtime-a",
|
||||
original.getBindingId(), original.getRuntimeGeneration())));
|
||||
for (CompletionStage<RuntimeSessionRecord> request : List.of(
|
||||
fixture.service.acquireRecovery("harness-b", "runtime-a", original.getBindingId(), original.getRuntimeGeneration()),
|
||||
fixture.service.acquireRecovery("harness-a", "runtime-a", original.getBindingId(), original.getRuntimeGeneration() + 1),
|
||||
fixture.service.acquireRecovery("harness-a", "missing", original.getBindingId(), original.getRuntimeGeneration()))) {
|
||||
assertEquals("workspace_close_identity_unverified", failure(request).getCode());
|
||||
}
|
||||
join(fixture.service.release("harness-a", "runtime-a"));
|
||||
assertEquals("workspace_close_identity_unverified", failure(fixture.service.acquireRecovery("harness-a", "runtime-a",
|
||||
original.getBindingId(), original.getRuntimeGeneration())).getCode());
|
||||
assertEquals(acquires, fixture.transport.acquireCalls.get());
|
||||
assertEquals(provisions, fixture.provisioner.calls.get());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void hookRecoveryDoesNotWaitForAnOrdinaryAcquireOfTheSameRuntimeId() throws Exception {
|
||||
try (Fixture fixture = new Fixture(SESSION_SCOPE)) {
|
||||
RuntimeSessionRecord original = join(fixture.service.acquire("harness-a", "runtime-a", "bootstrap"));
|
||||
var field = RuntimeBrokerService.class.getDeclaredField("sessions");
|
||||
field.setAccessible(true);
|
||||
((Map<?, ?>) field.get(fixture.service)).remove("runtime-a");
|
||||
fixture.resolver.result = CompletableFuture.completedFuture(new RuntimeScope(
|
||||
"tenant", "other-workspace", "generation", "/other", "other-capability", "session"));
|
||||
var pending = new CompletableFuture<Void>();
|
||||
fixture.transport.acquireResult = pending;
|
||||
var ordinary = fixture.service.acquire("harness-b", "runtime-a", "bootstrap");
|
||||
int provisions = fixture.provisioner.calls.get();
|
||||
int acquires = fixture.transport.acquireCalls.get();
|
||||
try {
|
||||
assertTimeoutPreemptively(Duration.ofSeconds(1), () -> assertEquals("workspace_close_identity_unverified",
|
||||
failure(fixture.service.acquireRecovery("harness-a", "runtime-a",
|
||||
original.getBindingId(), original.getRuntimeGeneration())).getCode()));
|
||||
assertFalse(ordinary.toCompletableFuture().isDone());
|
||||
assertSame(original, fixture.sessionRepository.findById(SESSION_SCOPE, "runtime-a"));
|
||||
assertEquals(provisions, fixture.provisioner.calls.get());
|
||||
assertEquals(acquires, fixture.transport.acquireCalls.get());
|
||||
} finally {
|
||||
pending.completeExceptionally(new RuntimeBrokerException(503, "ordinary_failure", "failed", true));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@SuppressWarnings("unchecked")
|
||||
void hookRecoveryRejectsFailedAndCancelledLocalRoutes() throws Exception {
|
||||
try (Fixture fixture = new Fixture(SESSION_SCOPE)) {
|
||||
RuntimeSessionRecord original = join(fixture.service.acquire("harness-a", "runtime-a", "bootstrap"));
|
||||
var field = RuntimeBrokerService.class.getDeclaredField("sessions");
|
||||
field.setAccessible(true);
|
||||
var routes = (Map<String, CompletableFuture<?>>) field.get(fixture.service);
|
||||
var cancelled = new CompletableFuture<>();
|
||||
cancelled.cancel(false);
|
||||
int provisions = fixture.provisioner.calls.get();
|
||||
int acquires = fixture.transport.acquireCalls.get();
|
||||
for (var route : List.of(CompletableFuture.failedFuture(
|
||||
new RuntimeBrokerException(503, "ordinary_failure", "failed", true)), cancelled)) {
|
||||
routes.put("runtime-a", route);
|
||||
assertEquals("workspace_close_identity_unverified", failure(fixture.service.acquireRecovery(
|
||||
"harness-a", "runtime-a", original.getBindingId(), original.getRuntimeGeneration())).getCode());
|
||||
}
|
||||
assertSame(original, fixture.sessionRepository.findById(SESSION_SCOPE, "runtime-a"));
|
||||
assertEquals(provisions, fixture.provisioner.calls.get());
|
||||
assertEquals(acquires, fixture.transport.acquireCalls.get());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void concurrentAcquireOfOneSessionCallsRuntimeOnce() {
|
||||
try (Fixture fixture = new Fixture(WORKSPACE_SCOPE)) {
|
||||
@@ -2047,6 +2121,24 @@ class RuntimeBrokerServiceTest {
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void legacyDrainAllowsOnlyOriginalHookControlAndNoNewOrdinaryAdmission() {
|
||||
try (Fixture fixture = new Fixture(SESSION_SCOPE)) {
|
||||
RuntimeSessionRecord session = join(fixture.service.acquire("harness", "runtime", "bootstrap"));
|
||||
fixture.bindingRepository.requestHarnessDrain("tenant", "harness");
|
||||
Map<String, Object> hook = Map.of("kind", "hook-execute", "operationId", "legacy-end",
|
||||
"sessionKey", Map.of("tenantId", "tenant", "workspaceId", "workspace", "sessionId", "harness"));
|
||||
assertEquals("ok", join(fixture.service.control("harness", "runtime", hook)));
|
||||
assertEquals(session.getSession(), fixture.transport.lastSession);
|
||||
Map<String, Object> ordinary = Map.of("kind", "mcp-configure", "operationId", "ordinary",
|
||||
"sessionKey", Map.of("tenantId", "tenant", "workspaceId", "workspace", "sessionId", "harness"));
|
||||
assertEquals("runtime_admission_closed", failure(fixture.service.control("harness", "runtime", ordinary)).getCode());
|
||||
assertEquals("runtime_admission_closed", failure(fixture.service.acquire("harness", "new-runtime", "bootstrap")).getCode());
|
||||
assertEquals(1, fixture.provisioner.calls.get());
|
||||
assertEquals(1, fixture.transport.acquireCalls.get());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void observesTheOriginalOwnerAfterNewAdmissionIsRevoked() {
|
||||
try (Fixture fixture = new Fixture(SESSION_SCOPE)) {
|
||||
@@ -2069,6 +2161,27 @@ class RuntimeBrokerServiceTest {
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void legacyDrainSettlesCancelledDispatchWithoutInvokingTheWorker() {
|
||||
try (Fixture fixture = new Fixture(SESSION_SCOPE)) {
|
||||
RuntimeSessionRecord session = join(fixture.service.acquire("harness", "runtime", "bootstrap"));
|
||||
fixture.executionRepository.findOrCreate(ToolExecutionRecord.prepared("parked", "parked-key",
|
||||
session.getBindingId(), session.getRuntimeGeneration(), "harness", "runtime", "prompt", "call", "digest",
|
||||
reference("runtime", "digest")));
|
||||
fixture.executionRepository.claimDispatch("parked", "broker", Duration.ofMinutes(1));
|
||||
fixture.bindingRepository.requestHarnessDrain("tenant", "harness");
|
||||
ToolExecutionRecord settled = join(fixture.service.cancelExecution("harness", "runtime", "parked"));
|
||||
assertEquals(ToolExecutionRecord.State.SETTLED, settled.getState());
|
||||
assertEquals("cancelled", settled.getExecutionStatus());
|
||||
assertTrue(settled.isCancelRequested());
|
||||
assertSame(settled, join(fixture.service.cancelExecution("harness", "runtime", "parked")));
|
||||
assertEquals("runtime_admission_closed", failure(fixture.service.createExecution("harness", "runtime",
|
||||
"new-key", reference("runtime", "other"))).getCode());
|
||||
assertEquals(0, fixture.transport.executeCalls.get());
|
||||
assertEquals(0, fixture.transport.cancelCalls.get());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void settledCancellationWinsOverLateExecutionCompletion() {
|
||||
try (Fixture fixture = new Fixture(WORKSPACE_SCOPE)) {
|
||||
|
||||
+26
@@ -34,6 +34,32 @@ class RuntimeHarnessDrainTest {
|
||||
private final Provider provisioner = new Provider();
|
||||
private final Transport transport = new Transport();
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(booleans = {false, true})
|
||||
void storageFenceBlocksLateSessionAdmissionAfterBindingReady(boolean jdbc) {
|
||||
RuntimeBindingRepository registry = jdbc ? bindings : new InMemoryRuntimeBindingRepository();
|
||||
RuntimeSessionRepository sessionRepository = jdbc ? sessions : new InMemoryRuntimeSessionRepository();
|
||||
var binding = ready(registry);
|
||||
var original = candidate(binding, "original");
|
||||
registry.admitSession(sessionRepository, original);
|
||||
registry.requestStorageFence("tenant", "storage", "migration");
|
||||
|
||||
var failure = assertThrows(RuntimeBrokerException.class,
|
||||
() -> registry.admitSession(sessionRepository, candidate(binding, "late")));
|
||||
assertEquals(409, failure.getStatusCode());
|
||||
assertEquals("workspace_migrating", failure.getCode());
|
||||
assertFalse(failure.isRetryable());
|
||||
assertNull(sessionRepository.findById(scope, "late"));
|
||||
var retained = sessionRepository.findById(scope, "original");
|
||||
assertTrue(original.sameIdentity(retained));
|
||||
assertEquals(RuntimeSessionRecord.State.ACQUIRING, retained.getState());
|
||||
assertEquals(1, sessionRepository.countActiveByBinding(binding.getBindingId(), binding.getGeneration()));
|
||||
assertEquals(RuntimeBindingRecord.State.READY, registry.findById(binding.getBindingId()).getState());
|
||||
assertEquals(binding.getGeneration(), registry.findById(binding.getBindingId()).getGeneration());
|
||||
assertTrue(registry.isStorageFenced("tenant", "storage", "migration"));
|
||||
assertFalse(registry.isHarnessDraining("tenant", "harness"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void fenceSurvivesRestartAndBlocksLateWarmAndAdmissions() throws Exception {
|
||||
var binding = ready();
|
||||
|
||||
@@ -76,7 +76,7 @@ export interface paths {
|
||||
};
|
||||
get?: never;
|
||||
put?: never;
|
||||
/** @description Maps workspaceId/cwdRelative to public Workspace selection without using environmentId or absolute cwd. Shares G0's opt-in initial file-tool Turn admission and fixed server-owned profile with public Session creation. The Session creator may submit later Turns and cancel its running Turns under the same opt-in, and may rename the Session, while the creator currently holds Workspace read and create grants on a registry row whose state is ACTIVE and the Session is an active, undeleted qwen-code Session on the frozen execution profile; Workspace close follows its separate close capability and lifecycle admission; archive, delete and unarchive follow their separate retention capabilities after reliable Workspace close; controlled same-Workspace cwd changes ship through the durable cwd_change operations (v1.32). The per-caller workspaceTurns capability on this surface advertises the same rule. Freeze selection with the original idempotency key; admission does not prove physical directory readiness. */
|
||||
/** @description Maps workspaceId/cwdRelative to public Workspace selection without using environmentId or absolute cwd. Shares G0's opt-in initial file-tool Turn admission and fixed server-owned profile with public Session creation. The Session creator may submit later Turns and cancel its running Turns under the same opt-in, and may rename the Session, while the creator currently holds Workspace read and create grants on a registry row whose state is ACTIVE and the Session is an active, undeleted qwen-code Session on the frozen execution profile; Workspace close follows its separate close capability and lifecycle admission; archive and unarchive follow retention capabilities after reliable Workspace close; delete supports either L3 idle ACTIVE files Sessions or L2 CLOSED/ARCHIVED Sessions; controlled same-Workspace cwd changes ship through the durable cwd_change operations (v1.32). The per-caller workspaceTurns capability on this surface advertises the same rule. Freeze selection with the original idempotency key; admission does not prove physical directory readiness. */
|
||||
post: operations["webShellCreateSession"];
|
||||
delete?: never;
|
||||
options?: never;
|
||||
@@ -226,7 +226,7 @@ export interface paths {
|
||||
};
|
||||
get?: never;
|
||||
put?: never;
|
||||
/** @description Uses the same close admission, creator authorization and durable cleanup semantics as the public close route. sessionClose describes deployment support; it does not grant actor authorization. */
|
||||
/** @description Uses the same close admission, creator authorization and durable cleanup semantics as the public close route. sessionClose describes deployment support; it does not grant actor authorization. New L3 Workspace close operations run SessionEnd only, retain data, and finish after durable effects verification and original Runtime stop. SessionDelete is reserved for ACTIVE delete. Missing protocol support rejects new admission; previously admitted operations retain their saved protocol. Unknown outcomes remain recovery_blocked. */
|
||||
post: operations["closeWebShellSession"];
|
||||
delete?: never;
|
||||
options?: never;
|
||||
@@ -260,7 +260,7 @@ export interface paths {
|
||||
};
|
||||
get?: never;
|
||||
put?: never;
|
||||
/** @description Workspace-bound delete accepts only closed or archived Sessions with completed reliable-close evidence and the creator with current read access. Active deletion returns 409 session_state_conflict. Completion atomically retires private recovery/publication access with the tombstone, operation receipt and terminal event; it calls neither Harness nor Runtime and erases no shared Workspace files or backup bytes. Operations remain readable under current read grants; the same authorized actor/key can replay after deletion. A readable non-creator returns 403; missing close evidence returns 409 workspace_unavailable. */
|
||||
/** @description Workspace-bound ACTIVE deletion supports idle hosted-workspace-files/1 Sessions when L3 lifecycle protocol and reliable Runtime stop are available. Only the creator with current read access may admit it; accepted, running, cancelling or approval-waiting Turns return 409 turn_active. Admission persists a lifecycle-only execution fence and a versioned actor-scoped operation. SessionEnd settles before SessionDelete; committed Hook results are verified in Session Store and saved before permanent draining, attachment cleanup and original worker stop. Unknown outcomes or missing original stop evidence remain recovery_blocked and are never replayed on a replacement worker. CLOSED/ARCHIVED deletion retains L2: completed reliable-close evidence is required and no Harness or Hook is invoked. Completion atomically commits retirement, tombstone, receipt and terminal event, retaining shared Workspace files and neighboring holders. Authorized same-actor/key replays return the original operation, including after deletion; readable non-creators return 403. Missing lifecycle capability or close evidence returns 409 workspace_unavailable. */
|
||||
post: operations["deleteWebShellSession"];
|
||||
delete?: never;
|
||||
options?: never;
|
||||
@@ -552,7 +552,7 @@ export interface components {
|
||||
*/
|
||||
sessionUnarchive?: boolean;
|
||||
/**
|
||||
* @description Optional support for this retention operation. For a Workspace-bound Session requires completed reliable-close evidence; authorization and source-state validation still apply.
|
||||
* @description Optional support capability. ACTIVE Workspace files Sessions require L3 lifecycle protocol and reliable original Runtime stop; CLOSED/ARCHIVED Sessions use independent L2 close evidence. Current creator/read authorization and idle/source-state admission checks still apply. The aggregate session lifecycle capability is unchanged.
|
||||
* @default false
|
||||
*/
|
||||
sessionDelete?: boolean;
|
||||
|
||||
Reference in new issue
Block a user