mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
refactor(managed-agent): Separate runtime broker execution workflows (#16029)
* refactor(managed-agent): Separate runtime broker execution workflows Give process-local Binding and Session state a registry and separate execution reconciliation, takeover scans and cancellation while preserving the existing fenced state transitions and dispatch boundary. Add a deterministic cooldown completion-order regression and synchronized bilingual designs. Co-authored-by: Qwen-Coder <[email protected]> * style(managed-agent): Remove extra blank lines from runtime workflows Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]>
This commit is contained in:
jinyeandQwen-Coder authored and GitHub committed
1 parent
91a6274075
commit
456a49c14f
19 files changed
+2955
-1886
No files matched your search
@@ -256,29 +256,29 @@ cross-cutting "how the system works today" tour.
|
||||
with a dozen injected callbacks rather than a clean seam; it waits for the
|
||||
reconciliation path to become its own collaborator. Cancellation follows
|
||||
the same rule: extract only where the seam is clean. Known coverage gap
|
||||
carried into the reconciliation slice:
|
||||
the stamp-before-in-flight-slot-drop ordering in `lookupOnce` has no test
|
||||
pinning it (a pre-existing gap, identical at the base); pin it when the
|
||||
reconciliation path becomes its own collaborator and the ordering is a
|
||||
seam-level contract rather than a private lambda's internal sequence.
|
||||
carried into the reconciliation slice: the stamp-before-in-flight-slot-drop
|
||||
ordering in `lookupOnce` had no test pinning it at the base. The
|
||||
[workflow extraction](2026-11-01-runtime-execution-workflows.md) now adds a
|
||||
deterministic blocked-stamp regression at the reconciler boundary; its
|
||||
verification evidence is recorded with that increment.
|
||||
- Reconciliation, recovery scanning and cancellation are **not** clean
|
||||
standalone seams (investigated 2026-10-31). Unlike the cooldown and the
|
||||
hold inventory — which own their state — these paths share the service's
|
||||
core mutable state and lifecycle machinery: the `sessions`, `liveBindings`
|
||||
and `reconciliations` concurrent maps, the binding-lifecycle helpers
|
||||
(`requireAnswerableBinding`, `requireLiveBinding`, `invalidateBinding`,
|
||||
`provisioner.isUsable`), and the CAS execution settle
|
||||
(`absorbRuntimeStatus` → `settleUnstartedBackgroundSiblings`). Extracting
|
||||
any of them now would inject a dozen maps/helpers/callbacks into a
|
||||
"collaborator" that is the god class under another name. The prerequisite
|
||||
is a **shared-state registry** slice: move the three maps and the
|
||||
binding-lifecycle helpers behind a `RuntimeBindingRegistry` (or similar)
|
||||
that owns them, leaving the service to delegate. Once that registry
|
||||
exists, reconciliation/recovery/cancellation depend on a narrow owner
|
||||
instead of the service's internals and become extractable, and the M6
|
||||
ordering becomes a seam-level contract a test can pin. This registry
|
||||
slice is larger and riskier than the two clean seams and should be
|
||||
designed and reviewed on its own before implementation.
|
||||
standalone seams (investigated 2026-10-31). Their prerequisite is the
|
||||
[Runtime Binding Registry](2026-10-31-runtime-binding-registry.md) first
|
||||
slice: it owns `liveBindings`, `bindingOperations`, `sessions`, local
|
||||
Session contexts and Binding validation/invalidation helpers. The source
|
||||
audit corrects the original three-map sketch: `bindingOperations` must
|
||||
move with Binding lifecycle operations, whereas execution-keyed
|
||||
`reconciliations` stays with `ExecutionObservationCooldown` and moves in
|
||||
the later reconciliation slice. In the Registry slice the service retained
|
||||
provisioning, drain, recovery, execution settlement, renewal and scheduler
|
||||
lifecycle. This
|
||||
establishes a concrete owner without injecting service callbacks. The M6
|
||||
stamp-before-slot-removal coverage gap is addressed by the blocked-stamp
|
||||
regression in the [execution workflow extraction](2026-11-01-runtime-execution-workflows.md),
|
||||
which implements reconciliation, takeover scans and cancellation after
|
||||
the Registry. The Registry slice itself does not claim that coverage.
|
||||
See the linked bilingual designs for concurrency, cleanup and acceptance
|
||||
requirements; local and CI qualification are recorded separately.
|
||||
- Apply the same treatment to `ManagedSessionStore` and
|
||||
`QwenHostedHarnessConnector` afterwards, each as its own slice.
|
||||
|
||||
|
||||
@@ -217,22 +217,22 @@ sweep 管理这个陷阱。各字段的单位本身是一份被钉住的绑定
|
||||
`scanExecutions` 驱动 `reconcileInContext`，后台进程 settle 族拥有
|
||||
`SessionContext` 内存索引与 CAS settle 循环——现在提取会造出注入十来个
|
||||
回调的协作者而非干净的缝；它等待对账路径先成为独立协作者。取消遵循
|
||||
同一原则：只在缝干净处提取。带入对账切片的已知覆盖缺口：`lookupOnce`
|
||||
中"先 stamp 后释放
|
||||
在途槽位"的顺序没有测试钉住（base 上即存在的缺口）；待对账路径提取
|
||||
为独立协作者、该顺序成为缝级契约而非私有 lambda 内部时序时再钉。
|
||||
- 对账、恢复扫描与取消**不是**干净的独立缝（2026-10-31 调研）。与冷却、
|
||||
hold 清单（各自拥有自己的状态）不同，这些路径共享服务的核心可变状态
|
||||
与生命周期机制：`sessions`、`liveBindings`、`reconciliations` 三个并发
|
||||
map，绑定生命周期助手（`requireAnswerableBinding`、`requireLiveBinding`、
|
||||
`invalidateBinding`、`provisioner.isUsable`），以及 CAS 执行 settle
|
||||
（`absorbRuntimeStatus` → `settleUnstartedBackgroundSiblings`）。现在
|
||||
提取其中任何一个，都会把十来个 map/助手/回调注入一个换个名字的上帝类
|
||||
"协作者"。前置条件是一个**共享状态注册表**切片：把三个 map 与绑定生命
|
||||
周期助手移到一个拥有它们的 `RuntimeBindingRegistry`（或类似）之后，
|
||||
服务转为委托。注册表就位后，对账/恢复/取消依赖的是一个窄所有者而非
|
||||
服务内部，从而可提取，M6 顺序也成为测试可钉的缝级契约。该注册表切片
|
||||
比两条干净缝更大、风险更高，应在实现前单独设计与评审。
|
||||
同一原则：只在缝干净处提取。带入对账切片的覆盖缺口：base 中没有测试
|
||||
固定 `lookupOnce` 先 stamp 后释放在途槽位的顺序。
|
||||
[流程抽取](2026-11-01-runtime-execution-workflows.zh-CN.md) 现已在对账器边界
|
||||
增加确定性阻塞 stamp 回归；其验证证据随该增量记录。
|
||||
- 对账、恢复扫描与取消**不是**干净的独立边界（2026-10-31 调研）。前置条件是
|
||||
[Runtime Binding Registry](2026-10-31-runtime-binding-registry.zh-CN.md)
|
||||
首片：它持有 `liveBindings`、`bindingOperations`、`sessions`、本地
|
||||
Session context 及 Binding 校验/失效 helper。源码审计修正了最初三张 map
|
||||
的草案：`bindingOperations` 必须随 Binding 生命周期操作迁移；按 execution
|
||||
ID 索引的 `reconciliations` 则继续与 `ExecutionObservationCooldown` 放在
|
||||
一起，在后续对账切片迁移。Registry 首片中，服务仍保留供应、排空、恢复、
|
||||
execution 结算、续租和 scheduler 生命周期。这样无需注入 service 回调即可建立具体持有者。
|
||||
M6 先 stamp 后移除槽位的覆盖缺口由[执行流程抽取](2026-11-01-runtime-execution-workflows.zh-CN.md)
|
||||
中阻塞 stamp 的回归处理；该切片在 Registry 后实现对账、接管扫描和取消。
|
||||
Registry 首片本身不宣称具备该覆盖。并发、清理与验收要求见所链接的双语
|
||||
设计；本地与 CI 验证分别记录。
|
||||
- 之后对 `ManagedSessionStore` 与 `QwenHostedHarnessConnector` 做同样处理，
|
||||
各自独立成切片。
|
||||
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
# Runtime Binding Registry
|
||||
|
||||
[English](2026-10-31-runtime-binding-registry.md) | [简体中文](2026-10-31-runtime-binding-registry.zh-CN.md)
|
||||
|
||||
- Status: First implementation slice; verification results are recorded separately. Subsequent workflow extraction is described in the [execution workflow design](2026-11-01-runtime-execution-workflows.md).
|
||||
- Date: 2026-10-31
|
||||
- Source baseline: `cf7e77810efa776f29b52a3dc4bf24f17d5e80f8`.
|
||||
- Scope: module-internal extraction in `packages/managed-agent-server/runtime-broker`.
|
||||
|
||||
## Problem and current state
|
||||
|
||||
`RuntimeBrokerService` combines placement, acquire, execution, reconciliation, cancellation, recovery and shutdown in approximately 6,300 lines. Extracting reconciliation alone would require passing its mutable maps and numerous lifecycle callbacks into another class. The [structure review](2026-10-31-managed-agent-structure-review.md) therefore requires a state owner first.
|
||||
|
||||
The source audit refines that proposal: `bindingOperations` belongs with Binding state because provisioning, recovery, reclamation and drain all reserve the same Binding ID. `reconciliations` belongs with execution reconciliation because its execution-ID reservation must be released only after stamping `ExecutionObservationCooldown`. Moving these two maps in the opposite direction would retain the coupling we want to remove.
|
||||
|
||||
## Goals and boundaries
|
||||
|
||||
Establish one module-internal owner for local Binding and Runtime Session state, migrate all production consumers, and preserve observable behavior. This slice changes no public interface, HTTP route, schema, dependency, authentication rule or protocol byte. It does not change when `warm` runs; the separate on-demand Runtime proposal is outside this change.
|
||||
|
||||
Provisioning, attestation, drain, Session acquisition/release, execution settlement, recovery scans and renewal scheduling remain orchestrated by the service. They may use Registry operations, but the Registry never calls back into the service. Do not introduce an interface, generic operation runner, event bus or configurable lifecycle framework for this one owner.
|
||||
|
||||
## Ownership
|
||||
|
||||
| State or responsibility | Owner after this slice | Authority and consumers |
|
||||
| ---------------------------------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `liveBindings` | `RuntimeBindingRegistry` | Local attested generation/lease only; provisioning/adoption registers it; warm, recovery, reconciliation and extension controls consult it; drain, loss and invalidation remove it. |
|
||||
| `bindingOperations` | Registry | One local reservation per Binding ID, shared by provisioning, reconciliation of a Binding, lost-generation reclamation, maintenance recovery and drain. It never replaces a repository operation claim. |
|
||||
| `sessions` and `SessionContext` | Registry | Local Runtime Session acquisition/adoption futures and per-context lock, mount claim, active controls, background-process IDs and release futures. Acquire, execution/control, reconciliation, cancellation, release and drain use these contexts. |
|
||||
| `reconciliations` and `ExecutionObservationCooldown` | Service, then a future reconciliation collaborator | Execution-ID deduplication and the existing stamp-before-reservation-removal ordering move together in a later slice. |
|
||||
| `dispatches`, `invocations`, execution CAS and background-process settlement | Service | Dispatch and physical invocation tracking are not Binding ownership. |
|
||||
| Operation/dispatch renewal, schedulers, `closed`, provisioner lifecycle | Service | The same close fence and renewal tasks remain in force. |
|
||||
| Persisted Binding, Session, execution, admission and writer fencing | Repositories | Durable authority across Broker processes and restarts; Registry membership is never authorization or proof of stopped writers. |
|
||||
|
||||
`BindingContext` is an immutable record and lease pair used between orchestration steps. `SessionContext` retains its `ReentrantLock`: callers' compound decisions and the context's accessors use the same guard, including when repository calls block on JDK 21 virtual threads. The first slice preserves the background-process set's existing guarded use rather than changing settlement ownership at the same time.
|
||||
|
||||
## Registry operations
|
||||
|
||||
The Registry is package-private in the existing service package. It receives the Binding repository, provisioner, Broker owner ID, operation lease duration and clock directly. These are its concrete dependencies for the existing validation/invalidation operations; it has no scheduler and no service reference.
|
||||
|
||||
- Look up, reserve and conditionally remove Binding-operation futures. A caller keeps the future it installed; cleanup removes that exact reservation, so an old completion cannot drop a replacement.
|
||||
- Look up, register and remove Session futures. Failed acquire/adoption removes its exact future. Successful release retains the existing removal points after durable release or an already-`RELEASED` read.
|
||||
- Register or forget a locally attested Binding, match its generation and complete lease identity, and require that local route before use.
|
||||
- Validate that an execution's original persisted generation remains `READY` or `DRAINING`. This read remains separate from local lease validation and from tenant, Harness and admission checks at their existing call sites.
|
||||
- Invalidate an unusable Binding and reject unusable Session leases with the existing error codes.
|
||||
- Cancel pending Binding and Session futures during service shutdown. The service continues to cancel dispatch and execution-reconciliation futures, stop both schedulers and close the provisioner.
|
||||
|
||||
Raw maps never escape. Futures and contexts are operation handles, not new durable state. The Registry does not start work, refresh authorization, acquire a replacement generation or manufacture a Session on a read path.
|
||||
|
||||
## Concurrency, leases and cleanup
|
||||
|
||||
Preserve these source-order contracts:
|
||||
|
||||
1. Provisioning/adoption persists readiness under the repository claim before registering the attested lease locally. Cached readiness must match generation, Runtime instance ID, endpoint, token, lease ID and epoch; a stale endpoint never becomes a fallback route.
|
||||
2. Reserve a Binding ID with `putIfAbsent` before starting the corresponding local operation. Existing callers either join or receive the same operation-specific refusal as before. Conditional reservation removal precedes completion of its owned future, preserving synchronous continuation behavior.
|
||||
3. Reserve a Session future before acquire. On failure, remove that future before completing it exceptionally. Existing Session identity/admission checks, mount-claim replacement, control counters and release guards retain their locations and lock boundaries.
|
||||
4. Invalidation first revokes the local route, then attempts a repository operation claim and the existing `READY` to `LOST` CAS. A failed claim or write must not restore the route. Release the claim in `finally`, using the exact Broker owner and operation generation; a failed best-effort release expires naturally. Invalidation alone proves neither loss evidence nor stopped writers and does not release Sessions or physical resources.
|
||||
5. Physical Session release still follows durable `RELEASING` admission, worker acknowledgement and durable release completion. Failed/negative release resets the existing latch for retry. No local cache move bypasses active-execution or background-hold checks.
|
||||
6. Service close sets the existing atomic fence before canceling Registry futures, then cancels the other operation futures, shuts down schedulers and closes the provisioner. Registry cancellation introduces no separate closed flag, teardown order or physical cleanup policy.
|
||||
|
||||
Binding IDs name durable physical generations; repository operation generations independently fence successive owners of a claim. The extraction preserves both identities and all repository checks. Process-local futures only coalesce work inside one Broker and cannot establish cross-process exclusivity.
|
||||
|
||||
## Implementation and follow-up sequence
|
||||
|
||||
The first slice adds `RuntimeBindingRegistry`, moves its maps, local context types and Binding validation/invalidation helpers, and migrates every service call site. Test-only context installation follows the new owner through `BrokerServiceTestAccess`; production gains no test hook. Update both versions of the structure review to reference the corrected ownership boundary.
|
||||
|
||||
The [subsequent workflow design and implementation](2026-11-01-runtime-execution-workflows.md) follows this sequence:
|
||||
|
||||
1. Execution reconciliation, taking its execution-ID reservations and cooldown ordering together. Pin stamp-before-reservation-removal at that seam; this Registry slice does not claim to close the existing coverage gap; the subsequent extraction adds the blocked-stamp regression.
|
||||
2. Recovery scanning and its background-process settlement dependencies, after the reconciliation seam exists.
|
||||
3. Cancellation and its original-generation routing and durable intent handling.
|
||||
|
||||
Each follow-up has its own diff and acceptance evidence. Moving all workflows into this Registry would merely recreate the large service under a new name.
|
||||
|
||||
## Validation and acceptance
|
||||
|
||||
The change is an internal refactor, so use the existing Java service and HTTP regressions instead of inventing a global CLI baseline or a new behavior E2E plan. Run focused Registry tests for full lease matching, persistent-generation eligibility, invalidation order and claim cleanup, reservation replacement and shutdown. Existing service tests cover concurrent warm/acquire/release, changed leases, stale provisioning completion, claim renewal, reconciliation routing and close fencing. Include the affected HTTP tests whose context fixtures move, durable recovery tests, the virtual-thread guard regression and module architecture checks within the module's shared local verification budget.
|
||||
|
||||
Preflight JDK 21, Maven >= 3.8.9 and a source-matching SDK in the task's isolated Maven repository. Use the module's focused Maven test/Checkstyle command and record the exact selected classes, source state, results and omissions. Full Java/SpotBugs, MySQL and Hosted/fault gates remain CI work. Selected in-memory or H2 cases do not establish MySQL, real-process recovery or production acceptance. If the relevant real-process path cannot run locally within budget, record it as pending rather than replacing it with a fixture claim.
|
||||
|
||||
Acceptance requires no raw Registry maps or context definitions in the service, no service dependency or injected callbacks in the Registry, unchanged external contracts and repository algorithms, meaningful focused regression evidence, two clean self-audit passes, synchronized bilingual documents and the required CI gates on the accepted revision. Local completion of the extraction alone does not mean CI acceptance.
|
||||
|
||||
## Risks and resolved questions
|
||||
|
||||
The main risks are changed completion order, accidental weakening of local lease checks, altered exception classification and loss of the shared context guard. Mitigate them with mechanical moves, caller-by-caller comparison and the focused tests above. Keep existing unconditional Session removal and Binding retirement points; changing their policy would be a separate behavior change.
|
||||
|
||||
Resolved: Registry owns Binding-operation reservations, not execution reconciliation reservations. The service owns renewal and shutdown. Durable admission remains at the original repository call sites. No open product decision is required for this slice; the subsequent collaborator APIs and validation are specified in the linked workflow design.
|
||||
@@ -0,0 +1,86 @@
|
||||
# Runtime Binding Registry
|
||||
|
||||
[English](2026-10-31-runtime-binding-registry.md) | [简体中文](2026-10-31-runtime-binding-registry.zh-CN.md)
|
||||
|
||||
- 状态：首个实现切片；验证结果单独记录。后续流程抽取见[执行流程设计](2026-11-01-runtime-execution-workflows.zh-CN.md)。
|
||||
- 日期：2026-10-31
|
||||
- 源码基线：`cf7e77810efa776f29b52a3dc4bf24f17d5e80f8`。
|
||||
- 范围：`packages/managed-agent-server/runtime-broker` 模块内部抽取。
|
||||
|
||||
## 问题与当前状态
|
||||
|
||||
`RuntimeBrokerService` 在约 6,300 行中混合了放置、acquire、执行、对账、取消、恢复和关闭。单独抽取对账需要向新类传递可变 map 和大量生命周期回调。因此，[结构评审](2026-10-31-managed-agent-structure-review.zh-CN.md) 要求先明确状态持有者。
|
||||
|
||||
源码审计修正了原提案：`bindingOperations` 应与 Binding 状态归于同一持有者，因为供应、恢复、回收和排空都预留同一个 Binding ID。`reconciliations` 则属于执行对账，因为必须先写入 `ExecutionObservationCooldown`，再释放按 execution ID 预留的槽位。反向迁移这两张 map 会保留本次要解除的耦合。
|
||||
|
||||
## 目标与边界
|
||||
|
||||
为本地 Binding 和 Runtime Session 状态建立唯一的模块内部持有者，迁移全部生产调用方，并保持可观察行为。首片不改变公共接口、HTTP 路由、schema、依赖、认证规则或协议字节。不改变 `warm` 的调用时机；按需启动 Runtime 的独立提案不在本次范围内。
|
||||
|
||||
供应、身份核验、排空、Session 获取/释放、执行结算、恢复扫描和续租调度仍由服务编排。它们可以调用 Registry，但 Registry 不得回调服务。不为这一个持有者引入接口、通用操作运行器、事件总线或可配置生命周期框架。
|
||||
|
||||
## 状态归属
|
||||
|
||||
| 状态或职责 | 首片后的持有者 | 权威来源与调用方 |
|
||||
| --------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `liveBindings` | `RuntimeBindingRegistry` | 仅表示本地已核验的代际/租约；供应和接管登记；warm、恢复、对账和扩展控制读取；排空、丢失和失效处理移除。 |
|
||||
| `bindingOperations` | Registry | 按 Binding ID 预留本地操作，由供应、Binding 对账、丢失代际回收、维护恢复和排空共享；不替代 repository 操作租约。 |
|
||||
| `sessions` 与 `SessionContext` | Registry | 本地 Runtime Session 获取/接管 future，以及每个 context 的锁、mount claim、活跃 control、后台进程 ID 和 release future。acquire、执行/control、对账、取消、释放和排空使用这些 context。 |
|
||||
| `reconciliations` 与 `ExecutionObservationCooldown` | 当前服务，后续对账协作者 | 按 execution ID 去重，以及先 stamp 后移除预留槽位的顺序，在后续切片中一起迁移。 |
|
||||
| `dispatches`、`invocations`、execution CAS 和后台进程结算 | 服务 | 分发与物理调用跟踪不属于 Binding 状态归属。 |
|
||||
| 操作/分发续租、scheduler、`closed`、provisioner 生命周期 | 服务 | 保留同一关闭屏障与续租任务。 |
|
||||
| 持久化 Binding、Session、execution、准入和 writer fencing | Repository | 跨 Broker 进程与重启的持久权威；Registry 成员资格绝不是授权或 writer 已停止的证明。 |
|
||||
|
||||
`BindingContext` 是编排步骤间传递的不可变 record 与 lease 对。`SessionContext` 保留 `ReentrantLock`：调用方的复合决策与 context 访问器使用同一把锁，包括 repository 调用在 JDK 21 虚拟线程上阻塞的场景。首片保持后台进程集合现有的加锁使用方式，不同时修改结算归属。
|
||||
|
||||
## Registry 操作
|
||||
|
||||
Registry 是现有 service package 中的 package-private 类。直接接收 Binding repository、provisioner、Broker owner ID、operation lease duration 和 clock。这些是现有校验/失效操作的具体依赖；它没有 scheduler 或 service 引用。
|
||||
|
||||
- 查找、预留并有条件地移除 Binding 操作 future。调用方保留自己安装的 future；清理只移除该预留项，旧操作完成不能删除替代项。
|
||||
- 查找、登记和移除 Session future。acquire/接管失败时只移除自己的 future。成功释放保留持久化释放完成或读到已 `RELEASED` 后的现有移除位置。
|
||||
- 登记或移除本地已核验的 Binding，匹配代际及完整租约身份，并在使用前要求这条本地路由存在。
|
||||
- 验证 execution 的原持久化代际仍为 `READY` 或 `DRAINING`。该读取继续独立于本地租约校验，以及原位置的 tenant、Harness 与准入检查。
|
||||
- 将不可用 Binding 标为失效，并按现有错误码拒绝不可用的 Session lease。
|
||||
- 服务关闭时取消未完成的 Binding 与 Session future。服务继续取消分发与执行对账 future、停止两个 scheduler 并关闭 provisioner。
|
||||
|
||||
不暴露原始 map。future 和 context 是操作句柄，不是新的持久状态。Registry 不启动工作、不刷新授权、不获取替代代际，也不在读取路径上制造 Session。
|
||||
|
||||
## 并发、租约与清理
|
||||
|
||||
保留下列源码顺序契约：
|
||||
|
||||
1. 供应/接管在 repository claim 下持久化就绪后，才在本地登记核验后的 lease。缓存就绪必须匹配代际、Runtime instance ID、endpoint、token、lease ID 和 epoch；过期 endpoint 不能成为回退路由。
|
||||
2. 用 `putIfAbsent` 预留 Binding ID 后才启动相应本地操作。现有调用方继续 join 或收到原有的操作特定拒绝。有条件移除预留项先于完成其持有的 future，保持同步 continuation 行为。
|
||||
3. acquire 前先预留 Session future。失败时先移除该 future，再异常完成。现有 Session 身份/准入检查、mount claim 替换、control 计数和 release guard 保留原位置与锁边界。
|
||||
4. 失效处理先撤销本地路由，再尝试 repository 操作 claim 与现有 `READY` 到 `LOST` 的 CAS。claim 或写入失败不得恢复路由。在 `finally` 中以准确的 Broker owner 和 operation generation 释放 claim；尽力释放失败时由租约自然过期。失效本身不构成丢失证据或 writer 停止证明，也不释放 Session 或物理资源。
|
||||
5. 物理 Session 释放仍按持久化 `RELEASING` 准入、worker 确认、持久化释放完成的顺序执行。失败/否定回复重置现有 latch 以便重试。本地缓存迁移不得绕过活跃 execution 或后台 hold 检查。
|
||||
6. 服务 close 先设置现有原子屏障，再取消 Registry future，随后取消其他操作 future、关闭 scheduler 和 provisioner。Registry 取消操作不引入独立 closed 标志、拆除顺序或物理清理策略。
|
||||
|
||||
Binding ID 标识持久的物理代际；repository operation generation 独立隔离同一 claim 的历次 owner。抽取保留两种身份和全部 repository 检查。进程内 future 仅合并同一 Broker 中的工作，不能建立跨进程互斥。
|
||||
|
||||
## 实现与后续顺序
|
||||
|
||||
首片新增 `RuntimeBindingRegistry`，迁移其 map、本地 context 类型及 Binding 校验/失效 helper，并迁移全部 service 调用点。测试通过 `BrokerServiceTestAccess` 按新持有者安装 context；生产代码不新增测试入口。同时更新两个语言版本的结构评审，引用修正后的归属边界。
|
||||
|
||||
[后续流程设计与实现](2026-11-01-runtime-execution-workflows.zh-CN.md) 遵循以下顺序：
|
||||
|
||||
1. 执行对账，同时迁移按 execution ID 预留的槽位和 cooldown 顺序。在该边界固定先 stamp 后移除预留项的测试；Registry 首片不宣称补齐现有覆盖缺口；后续抽取增加阻塞 stamp 的回归。
|
||||
2. 对账边界具备之后，再拆恢复扫描及其后台进程结算依赖。
|
||||
3. 取消执行及其原代际路由和持久取消意图处理。
|
||||
|
||||
每项后续工作独立提供 diff 和验收证据。把所有流程搬进 Registry，只会换一个名字重新形成大服务。
|
||||
|
||||
## 验证与验收
|
||||
|
||||
本次是内部重构，复用现有 Java service 与 HTTP 回归，不构造全局 CLI 基线或新的行为 E2E 计划。定向 Registry 测试覆盖完整 lease 匹配、持久代际资格、失效顺序与 claim 清理、预留项替换和关闭。现有 service 测试覆盖并发 warm/acquire/release、租约变化、过期供应完成、claim 续租、对账路由和关闭屏障。在模块共享本地验证预算内，包括 context 夹具迁移所影响的 HTTP 测试、持久恢复测试、虚拟线程 guard 回归和模块架构检查。
|
||||
|
||||
预检 JDK 21、Maven >= 3.8.9，以及任务隔离 Maven 仓库中匹配源码的 SDK。按模块要求运行定向 Maven test/Checkstyle，记录准确的测试类、源码状态、结果和遗漏。完整 Java/SpotBugs、MySQL 和 Hosted/fault 门禁交由 CI。选定的内存或 H2 用例不构成 MySQL、真实进程恢复或生产验收。相关真实进程路径若无法在本地预算内运行，记录为待验证，不以 fixture 结果替代。
|
||||
|
||||
验收要求：service 不再持有 Registry 的原始 map 或 context 定义；Registry 不依赖 service 或注入回调；外部契约和 repository 算法保持不变；具备有意义的定向回归证据；完成两轮干净自审；双语文档同步；被验收版本通过必要 CI 门禁。本地抽取完成不表示 CI 已验收。
|
||||
|
||||
## 风险与已决问题
|
||||
|
||||
主要风险是完成顺序变化、本地 lease 校验被意外削弱、异常分类改变，以及丢失共享 context guard。通过机械迁移、逐调用方对比和上述定向测试控制风险。保留现有无条件移除 Session 和退役 Binding 的位置；更改该策略属于独立行为变更。
|
||||
|
||||
已决：Registry 持有 Binding 操作预留项，不持有执行对账预留项。续租与关闭归服务。持久准入保留在原 repository 调用位置。首片没有待定产品决策；后续协作者 API 和验证已在所链接的流程设计中明确。
|
||||
@@ -0,0 +1,77 @@
|
||||
# Runtime Execution Workflow Decomposition
|
||||
|
||||
[English](2026-11-01-runtime-execution-workflows.md) | [简体中文](2026-11-01-runtime-execution-workflows.zh-CN.md)
|
||||
|
||||
- Status: Implemented locally with focused validation passing; full CI qualification remains pending.
|
||||
- Date: 2026-11-01
|
||||
- Prerequisite: the local [Runtime Binding Registry](2026-10-31-runtime-binding-registry.md) slice on baseline `cf7e77810efa776f29b52a3dc4bf24f17d5e80f8`.
|
||||
|
||||
## Problem, goals and scope
|
||||
|
||||
The Registry gives Binding and Session state an owner, but execution reconciliation, takeover scans and cancellation still live in `RuntimeBrokerService`. Their shared execution reads, fenced settlement and background-hold settlement make a three-way copy of private helpers unsafe. Extract the three workflows with direct module-internal collaborators and one shared execution-state boundary. Preserve current observable behavior, including failure timing, async continuations, writer fencing and cleanup order.
|
||||
|
||||
No public interface, HTTP route, schema, protocol, dependency or Runtime launch policy changes. Provisioning, Binding recovery/drain, dispatch, renewal scheduling and physical Session release stay in the service. This is structural work; full integration acceptance remains separate from implementation completion.
|
||||
|
||||
## Ownership and dependencies
|
||||
|
||||
| Component | Owns | Consumers |
|
||||
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
|
||||
| `RuntimeBindingRegistry` | Existing local Binding/Session maps and contexts; READY/historical Session validation | Service and all three execution workflows |
|
||||
| `RuntimeExecutionState` | Existing execution ownership reads, fenced dispatch settlement/UNKNOWN transitions, lapsed-claim fencing and background-row settlement | Dispatch, reconciliation, recovery, cancellation and existing acknowledgement/control paths |
|
||||
| `RuntimeExecutionReconciler` | Execution-ID lookup reservations, `ExecutionObservationCooldown`, evidence validation and reconciliation CAS | Service observation endpoints and takeover scanner |
|
||||
| `RuntimeExecutionRecovery` | Bounded takeover scan and background-hold observation/sweep workflows | Acquire/adoption, routine/drain release, background observation and task stop |
|
||||
| `RuntimeExecutionCancellation` | Durable cancellation intent and physical cancellation/confirmation workflow | Service cancellation endpoint |
|
||||
| `RuntimeBrokerSupport` | Unchanged stateless future/error mapping, payload/reference validation and status projection | The extracted collaborators and service |
|
||||
| Service | Public facade, `closed`, dispatch reservations/invocations, schedulers, provisioner lifetime, Binding/Session lifecycle orchestration | Existing adapters and assembly |
|
||||
|
||||
All classes remain package-private in the existing runtime-broker service package; no new interface or Spring wiring is needed. Repositories remain the sole durable authority. Context locks still belong to the Registry, and every compound read/admit/release decision retains its current lock region. The support class shares existing rules across several real callers; it is not a configurable workflow framework.
|
||||
|
||||
## Reconciliation design
|
||||
|
||||
Move `reconcileExecution`, automatic `observeExecution`, the context-based lookup, its execution-ID future map, the cooldown and result absorption into `RuntimeExecutionReconciler`. The service keeps the public signatures and delegates. The reconciler receives the Registry, execution-state collaborator, execution repository, provisioner, transport, publication verifier, clock, lookup duration, shared close fence and existing cooldown instance. It owns no executor and never calls the service.
|
||||
|
||||
Keep this sequence: validate caller IDs; read owned execution; return non-UNKNOWN/terminal answers before requiring a live Session; consult a finished v3 publication if available; otherwise require the original persisted generation and locally attested route; perform a bounded status lookup. Only validated settled evidence resolves UNKNOWN, and a racing cancellation is reread before CAS. UNKNOWN or unreachable is never proof of `not_started`, never dispatches work and never provisions a Runtime.
|
||||
|
||||
Explicit reconciliation remains fresh. Automatic observation consults cooldown first. Concurrent requests join the existing execution-ID reservation. On completion, stamp cooldown before removing that exact reservation, then complete its future; JVM Errors do not become cached UNRESOLVED answers. This order prevents a sequential caller between removal and stamping from issuing another worker request. Service close cancels the reconciler's pending reservations at the same point in the original shutdown order.
|
||||
|
||||
Takeover lookup retains its extra close checks, original-generation routing, unresolved handling of ordinary failures/invalid status and refusal to swallow an Error. Background `not_started` settlement goes through the shared execution-state collaborator, avoiding a dependency cycle with the scanner.
|
||||
|
||||
## Recovery scan and hold design
|
||||
|
||||
Move `scanExecutions` into `RuntimeExecutionRecovery`, together with durable background-row pagination, fire-and-forget scan observations, routine/drain sweeps, physical hold control and the existing background observation/task-stop entry points. The dependency set is Registry, execution state, execution/session repositories, transport, reconciler and the shared close fence. It has no service callback or scheduler.
|
||||
|
||||
The scan still reads 100-row pages scoped to the saved Session and generation, rereads each candidate before observation and yields through `thenComposeAsync` between candidates/pages. It does not claim dispatch, resend tools or isolate a record merely because it is old. Acquire waits for the execution scan but never for background maintenance observations. Listing/control failures in the background scan remain isolated from acquire.
|
||||
|
||||
Both scan and release use the same durable background-row reader. A scan observes without `activeControls`; ordinary controls/sweeps keep their original guard accounting. The routine release sweep only observes, whereas drain alone follows status → stop → status. Only proven exit or no-start evidence settles the hold. Unknown evidence and per-row errors preserve the hold. Backfill/snapshot/removal of the context's background index retain the same lock regions. Physical Session release and its durable transitions remain in the service and call the recovery collaborator before their existing busy check.
|
||||
|
||||
Task stop keeps tenant/Harness ownership validation and the original Session route. It may stop only the invocation's own `:process` row. No recovery path creates a replacement Binding or Session.
|
||||
|
||||
## Cancellation design
|
||||
|
||||
Move cancellation admission, prepared-provider confirmation, physical cancel and cancellation-result validation into `RuntimeExecutionCancellation`. It uses Registry, execution state, execution repository, transport, the existing invocation set, coordination scheduler, cancellation duration and close fence. The service facade supplies its existing two-argument dispatch action for the one DISPATCHING branch; this per-call `BiConsumer` is the only service action passed to an extracted workflow. Do not store a service reference or pass maps/private lifecycle helpers as callbacks.
|
||||
|
||||
Preserve terminal-receipt ownership reads before requiring a live Session. A prepared provider's cancelled receipt still requires original-worker confirmation when that worker may retain the preparation. Nonterminal cancellation persists intent before sending a worker request. Hold rows and authorized private CSI calls retain their existing refusals. An invocation still running here may receive cancellation after lease lapse; an invocation absent here first follows the repository-clock dispatch fence. UNKNOWN remains UNKNOWN until reconciliation evidence resolves it.
|
||||
|
||||
An unsent DISPATCHING record still goes through the existing dispatcher, which reads its durable cancellation intent before deciding whether to execute; replacing that branch with a new direct settlement algorithm would change behavior. Physical cancellation status is validated and settled with the original dispatch owner/generation. A late result after claim lapse is fenced, and live-claim conflicts remain errors. Prepared-provider cancellation retains its bounded result future, 25 ms observation schedule, control accounting and refusal to treat forgotten state as confirmation.
|
||||
|
||||
## Shared state, files and migration order
|
||||
|
||||
`RuntimeExecutionState` moves the existing owned-execution checks, `settleExecution`, `markUnknown`, `fenceLapsedClaim`, UNKNOWN-on-closed-admission mapping and background settlement CAS loops without changing their algorithms. It receives the three repositories, Broker owner, dispatch lease and clock. `RuntimeBindingRegistry` additionally receives the existing Session repository and resolver for the READY/historical checks used across workflows; non-managed historical reads still revalidate the resolved scope, while managed history uses its saved parent identity. No resolver/admission check is dropped or moved to a cache-only path.
|
||||
|
||||
Implement in order: shared unchanged helpers/state and Registry queries; reconciler; recovery and hold maintenance; cancellation; facade wiring and shutdown. The five new implementation files above, the existing service/Registry/cooldown documentation, test constructors and focused reconciliation regression are the affected code surfaces. Update both Registry and structure-review design pairs to link this design and distinguish the earlier slice from the new implementation. The existing contract/schema/adapters remain unchanged.
|
||||
|
||||
## Validation and acceptance
|
||||
|
||||
Add a deterministic reconciler-level regression for stamp-before-reservation-removal: block cooldown stamping, submit another observation during that interval and assert that it joins without a second worker lookup. Cover both an ordinary status and an ordinary lookup failure, then release the stamp and verify completion/cooldown. The witness must fail if reservation removal is moved before stamping.
|
||||
|
||||
Reuse the existing service regressions for explicit versus automatic lookup, timeouts, malformed status, JVM Errors, publication results, concurrent cancellation and shutdown. Recovery regressions must cover multi-page takeover, generation/Session isolation, asynchronous background observations, durable-only holds and routine versus drain release. Cancellation regressions cover terminal ownership, prepared confirmation, unsent dispatch, lapsed claims, sticky intent, cross-Harness refusal and hold/CSI refusals. Keep their assertions and original ordering expectations when moving code.
|
||||
|
||||
Within the module's shared 10-minute local verification budget, reuse the source-matching SDK and JDK 21/Maven environment, run affected Java suites with Checkstyle and the minimum subprocess path, and inspect the actual selected reports. Full MySQL 8.4, bundled Hosted/fault recovery and SpotBugs remain required CI qualification; fixture/H2 or test-worker subprocess passes do not establish those results. No JavaScript build is required because this change has no TypeScript/shared-contract change.
|
||||
|
||||
Acceptance: all three facade entry families delegate to their named collaborators; lookup reservations and cooldown share one owner; no collaborator reaches back into service internals; the sole cancellation dispatch action is explicit; no SQL/schema/public-contract diff; moved algorithms retain their semantics; the ordering witness and focused regressions pass; bilingual documents and two clean self-audit passes are complete. CI acceptance is pending until its required gates run on the accepted revision.
|
||||
|
||||
## Risks and open questions
|
||||
|
||||
The main risks are altered synchronous exception mapping, completion ordering, guard accounting and cross-generation routing. Mechanical moves and unchanged call-site envelopes reduce these risks, while the existing integration tests exercise them through the public service facade. Shared immutable dependencies are passed directly rather than hidden in a generic context object.
|
||||
|
||||
The callback exception is deliberately limited to cancellation's unsent-dispatch branch; extracting the entire dispatcher would widen this task. No product decisions remain open. A later dispatcher extraction may replace that action with a concrete collaborator without changing cancellation semantics.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Runtime 执行流程拆分
|
||||
|
||||
[English](2026-11-01-runtime-execution-workflows.md) | [简体中文](2026-11-01-runtime-execution-workflows.zh-CN.md)
|
||||
|
||||
- 状态：本地已实现，定向验证通过；完整 CI 验收仍待完成。
|
||||
- 日期：2026-11-01
|
||||
- 前置条件：基于 `cf7e77810efa776f29b52a3dc4bf24f17d5e80f8` 的本地 [Runtime Binding Registry](2026-10-31-runtime-binding-registry.zh-CN.md) 切片。
|
||||
|
||||
## 问题、目标与范围
|
||||
|
||||
Registry 明确了 Binding 与 Session 状态持有者，但执行对账、接管扫描与取消仍在 `RuntimeBrokerService` 中。它们共享执行读取、带 fencing 的结算和后台 hold 结算，不能安全地把私有 helper 复制三遍。用直接依赖的模块内部协作者拆出三个流程，并建立一个共享执行状态边界。保留当前可观察行为，包括失败时机、异步 continuation、writer fencing 和清理顺序。
|
||||
|
||||
不修改公共接口、HTTP 路由、schema、协议、依赖或 Runtime 启动策略。供应、Binding 恢复/排空、分发、续租调度和物理 Session 释放仍在服务中。这是结构调整；完整集成验收与实现完成分别记录。
|
||||
|
||||
## 归属与依赖
|
||||
|
||||
| 组件 | 持有职责 | 调用方 |
|
||||
| ------------------------------ | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------- |
|
||||
| `RuntimeBindingRegistry` | 已有本地 Binding/Session map 与 context；READY/历史 Session 校验 | 服务与全部三个执行流程 |
|
||||
| `RuntimeExecutionState` | 已有 execution 归属读取、带 fencing 的分发结算/UNKNOWN 转换、过期 claim 隔离和后台行结算 | 分发、对账、恢复、取消及现有确认/control 路径 |
|
||||
| `RuntimeExecutionReconciler` | 按 execution ID 预留的查询槽位、`ExecutionObservationCooldown`、证据校验与对账 CAS | 服务观察入口与接管扫描 |
|
||||
| `RuntimeExecutionRecovery` | 有界接管扫描与后台 hold 观察/清理流程 | acquire/接管、普通/排空 release、后台观察与任务停止 |
|
||||
| `RuntimeExecutionCancellation` | 持久取消意图与物理取消/确认流程 | 服务取消入口 |
|
||||
| `RuntimeBrokerSupport` | 不变的无状态 future/错误映射、payload/reference 校验和 status 投影 | 抽出的协作者与服务 |
|
||||
| 服务 | 公共门面、`closed`、分发预留项/在途调用、scheduler、provisioner 生命周期、Binding/Session 生命周期编排 | 现有 adapter 与装配 |
|
||||
|
||||
全部类保留在现有 runtime-broker service package 内，采用 package-private，不新增接口或 Spring 装配。Repository 仍是唯一持久权威。context 的锁继续归 Registry，每个复合读取/准入/释放决策保留当前锁区域。support 类只共享多个真实调用方已有的规则，不引入可配置流程框架。
|
||||
|
||||
## 对账设计
|
||||
|
||||
将 `reconcileExecution`、自动 `observeExecution`、基于 context 的查询、按 execution ID 索引的 future map、cooldown 和结果吸收迁入 `RuntimeExecutionReconciler`。服务保留公共签名并委托。对账器接收 Registry、执行状态协作者、execution repository、provisioner、transport、publication verifier、clock、查询时限、共享关闭屏障和现有 cooldown 实例。不持有 executor，不调用服务。
|
||||
|
||||
保留以下顺序：校验调用方 ID；读取属于调用方的 execution；在要求活跃 Session 之前返回非 UNKNOWN/终态结果；若存在已完成的 v3 publication，则读取它；否则要求原持久代际与本地已核验路由，再执行有时限的 status 查询。只有合法的 settled 证据才能结算 UNKNOWN，并在 CAS 前重新读取并发取消。UNKNOWN 或不可达绝不构成 `not_started` 证明，不触发分发或 Runtime 供应。
|
||||
|
||||
显式对账继续读取新状态，自动观察先读 cooldown。并发请求加入已有 execution ID 预留项。完成时先 stamp cooldown，再移除准确的预留项，最后完成 future；JVM Error 不得缓存为 UNRESOLVED。这样可防止顺序调用方在移除与 stamp 之间发起第二次 worker 查询。服务关闭在原有关闭顺序的同一位置取消对账器待完成的预留项。
|
||||
|
||||
接管查询保留额外的关闭检查、原代际路由、普通失败/非法 status 的 unresolved 处理，以及不吞掉 Error 的规则。后台 `not_started` 结算经共享执行状态协作者执行，避免与扫描器形成依赖环。
|
||||
|
||||
## 恢复扫描与 hold 设计
|
||||
|
||||
把 `scanExecutions` 连同持久后台行分页、后台扫描观察、普通/排空清理、物理 hold control 和现有后台观察/任务停止入口迁入 `RuntimeExecutionRecovery`。依赖为 Registry、执行状态、execution/session repository、transport、对账器和共享关闭屏障。不回调服务，也不持有 scheduler。
|
||||
|
||||
扫描继续按已存 Session 与代际读取每页 100 行，观察前重读候选行，并通过 `thenComposeAsync` 在候选项/页面之间让出执行。它不 claim dispatch、不重发工具，也不因记录久远就隔离它。acquire 等待 execution 扫描，但不等待后台维护观察。后台扫描的列举/control 失败仍与 acquire 隔离。
|
||||
|
||||
扫描与 release 共用持久后台行读取。扫描观察不持有 `activeControls`，普通 control/清理保持原 guard 计数。普通 release 清理仅观察，只有 drain 执行 status → stop → status。仅已证明退出或未启动的证据能结算 hold；未知证据与逐行错误保留 hold。context 后台索引的回填、快照和移除保留原锁区域。物理 Session 释放及其持久转换仍归服务，在已有 busy 检查前调用恢复协作者。
|
||||
|
||||
任务停止保留 tenant/Harness 归属校验和原 Session 路由，只能停止该 invocation 自己的 `:process` 行。任何恢复路径都不创建替代 Binding 或 Session。
|
||||
|
||||
## 取消设计
|
||||
|
||||
把取消准入、prepared provider 确认、物理取消与结果校验迁入 `RuntimeExecutionCancellation`。它使用 Registry、执行状态、execution repository、transport、现有 invocation 集合、协调 scheduler、取消时限和关闭屏障。服务门面为唯一的 DISPATCHING 分支传入现有双参数分发动作；这个逐调用传入的 `BiConsumer` 是抽出流程接收的唯一 service 动作。不保存 service 引用，不把 map/私有生命周期 helper 作为回调传递。
|
||||
|
||||
保留在要求活跃 Session 前读取终态回执归属的顺序。若原 worker 可能仍保留 preparation，prepared provider 的已取消回执仍需原 worker 确认。非终态取消先持久化意图，再发送 worker 请求。hold 行和已授权的私有 CSI 调用保留原拒绝规则。租约过期后，本进程仍运行的 invocation 可继续接收取消；本进程未运行的 invocation 先按 repository 时钟执行 dispatch fence。UNKNOWN 只在对账证据足够时才能结算。
|
||||
|
||||
未发送的 DISPATCHING 记录继续走已有 dispatcher，由它读取持久取消意图后决定是否执行；用新的直接结算算法替代该分支会改变行为。物理取消 status 按原 dispatch owner/generation 校验并结算。claim 过期后的迟到结果被隔离，活跃 claim 冲突继续报错。prepared provider 取消保留有时限的结果 future、25 ms 观察调度、control 计数，以及不把已遗忘状态当作确认的规则。
|
||||
|
||||
## 共享状态、文件与迁移顺序
|
||||
|
||||
`RuntimeExecutionState` 原样迁移 execution 归属检查、`settleExecution`、`markUnknown`、`fenceLapsedClaim`、准入关闭时保留 UNKNOWN 的映射和后台结算 CAS 循环，不改变算法。接收三个 repository、Broker owner、dispatch lease 和 clock。`RuntimeBindingRegistry` 额外接收现有 Session repository 与 resolver，以提供跨流程使用的 READY/历史检查；非 managed 历史读取继续重验解析的 scope，managed 历史使用已存父身份。不删除 resolver/准入检查，也不把它们改成仅依赖缓存的路径。
|
||||
|
||||
实现顺序：共享的不变 helper/状态与 Registry 查询；对账器；恢复与 hold 维护；取消；门面装配和关闭。受影响代码为上面的五个新实现文件、已有 service/Registry/cooldown 说明、测试构造器和定向对账回归。同时更新 Registry 与结构评审的两组双语设计，链接本设计并区分早期切片与本次实现。现有 contract/schema/adapter 不变。
|
||||
|
||||
## 验证与验收
|
||||
|
||||
增加确定性的对账器级先 stamp 后移除预留项回归：阻塞 cooldown stamp，在该窗口内再次观察，断言加入已有操作且不产生第二次 worker 查询。覆盖普通 status 与普通查询失败，再放开 stamp 并验证完成/cooldown。把预留项移除移到 stamp 前时，该测试必须失败。
|
||||
|
||||
复用 service 回归覆盖显式/自动查询、超时、非法 status、JVM Error、publication 结果、并发取消与关闭。恢复回归须覆盖多页接管、代际/Session 隔离、异步后台观察、仅持久存在的 hold 和普通/排空 release 差异。取消回归覆盖终态归属、prepared 确认、未发送分发、过期 claim、粘性取消意图、跨 Harness 拒绝和 hold/CSI 拒绝。迁移代码时保留这些断言及原顺序预期。
|
||||
|
||||
在模块共享的 10 分钟本地验证预算内，复用匹配源码的 SDK 和 JDK 21/Maven 环境，运行受影响 Java 测试、Checkstyle 和最小子进程路径，并检查真实选中的报告。完整 MySQL 8.4、打包 Hosted/fault 恢复和 SpotBugs 仍需 CI 验收；fixture/H2 或测试 worker 子进程通过不构成这些结果。本次无 TypeScript/共享契约变化，不需要 JavaScript 构建。
|
||||
|
||||
验收要求：三个公共入口族委托给对应协作者；查询预留项与 cooldown 归同一持有者；协作者不访问 service 内部；唯一的取消分发动作显式传入；SQL/schema/公共契约无 diff；迁移算法语义不变；顺序回归与定向测试通过；双语文档及两轮干净自审完成。在被验收版本运行所需门禁前，CI 验收仍待完成。
|
||||
|
||||
## 风险与待定问题
|
||||
|
||||
主要风险是同步异常映射、完成顺序、guard 计数和跨代际路由改变。通过机械迁移和保留调用点包络降低风险，现有集成测试继续经公共服务门面验证它们。共享的不可变依赖直接传递，不隐藏在通用 context 对象中。
|
||||
|
||||
回调例外刻意限于取消的未发送分发分支；抽取整个 dispatcher 会扩大本次范围。没有待定产品决策。后续 dispatcher 抽取可在不改变取消语义的前提下，把该动作替换为具体协作者。
|
||||
+2
-1
@@ -18,7 +18,8 @@ import java.util.concurrent.TimeUnit;
|
||||
* worker (issue #13183 saw hundreds of status calls per execution).
|
||||
*
|
||||
* <p>Extracted from {@link RuntimeBrokerService} as the first P1 internal
|
||||
* seam. The service owns the scheduler this class schedules eviction on,
|
||||
* seam, now owned by RuntimeExecutionReconciler. The service owns the
|
||||
* scheduler this class schedules eviction on,
|
||||
* so the cooldown has no lifecycle of its own.
|
||||
*/
|
||||
final class ExecutionObservationCooldown {
|
||||
|
||||
+427
@@ -0,0 +1,427 @@
|
||||
package com.alibaba.qwen.code.runtimebroker.service;
|
||||
|
||||
import com.alibaba.qwen.code.runtimebroker.BrokerValues;
|
||||
import com.alibaba.qwen.code.runtimebroker.HarnessSessionResolver;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeLease;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeProvisioner;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSession;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSessionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSessionRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRecord;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.ConcurrentMap;
|
||||
import java.util.concurrent.locks.ReentrantLock;
|
||||
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.conflict;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.mapFailure;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.notFound;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.safeStage;
|
||||
|
||||
/** Process-local Binding and Session ownership; repositories remain authoritative. */
|
||||
final class RuntimeBindingRegistry {
|
||||
private final ConcurrentMap<String, LiveBinding> liveBindings = new ConcurrentHashMap<>();
|
||||
private final ConcurrentMap<String, CompletableFuture<BindingContext>> bindingOperations = new ConcurrentHashMap<>();
|
||||
private final ConcurrentMap<String, CompletableFuture<SessionContext>> sessions = new ConcurrentHashMap<>();
|
||||
private final RuntimeBindingRepository bindingRepository;
|
||||
private final RuntimeProvisioner provisioner;
|
||||
private final RuntimeSessionRepository sessionRepository;
|
||||
private final HarnessSessionResolver sessionResolver;
|
||||
private final String brokerOwnerId;
|
||||
private final Duration operationLeaseDuration;
|
||||
private final Clock clock;
|
||||
|
||||
RuntimeBindingRegistry(RuntimeBindingRepository bindingRepository,
|
||||
RuntimeProvisioner provisioner, RuntimeSessionRepository sessionRepository,
|
||||
HarnessSessionResolver sessionResolver, String brokerOwnerId,
|
||||
Duration operationLeaseDuration, Clock clock) {
|
||||
this.bindingRepository = bindingRepository;
|
||||
this.provisioner = provisioner;
|
||||
this.sessionRepository = sessionRepository;
|
||||
this.sessionResolver = sessionResolver;
|
||||
this.brokerOwnerId = brokerOwnerId;
|
||||
this.operationLeaseDuration = operationLeaseDuration;
|
||||
this.clock = clock;
|
||||
}
|
||||
|
||||
CompletableFuture<BindingContext> bindingOperation(String bindingId) {
|
||||
return bindingOperations.get(bindingId);
|
||||
}
|
||||
|
||||
CompletableFuture<BindingContext> reserveBinding(String bindingId,
|
||||
CompletableFuture<BindingContext> reservation) {
|
||||
return bindingOperations.putIfAbsent(bindingId, reservation);
|
||||
}
|
||||
|
||||
void finishBindingOperation(String bindingId, CompletableFuture<BindingContext> reservation) {
|
||||
bindingOperations.remove(bindingId, reservation);
|
||||
}
|
||||
|
||||
CompletableFuture<SessionContext> session(String runtimeSessionId) {
|
||||
return sessions.get(runtimeSessionId);
|
||||
}
|
||||
|
||||
CompletableFuture<SessionContext> registerSession(String runtimeSessionId,
|
||||
CompletableFuture<SessionContext> context) {
|
||||
return sessions.putIfAbsent(runtimeSessionId, context);
|
||||
}
|
||||
|
||||
void removeSession(String runtimeSessionId, CompletableFuture<SessionContext> context) {
|
||||
sessions.remove(runtimeSessionId, context);
|
||||
}
|
||||
|
||||
void removeSession(String runtimeSessionId) {
|
||||
sessions.remove(runtimeSessionId);
|
||||
}
|
||||
|
||||
void registerBinding(RuntimeBindingRecord record, RuntimeLease lease) {
|
||||
liveBindings.put(record.getBindingId(), new LiveBinding(record.getGeneration(), lease));
|
||||
}
|
||||
|
||||
boolean hasBinding(String bindingId) {
|
||||
return liveBindings.containsKey(bindingId);
|
||||
}
|
||||
|
||||
void forgetBinding(String bindingId) {
|
||||
liveBindings.remove(bindingId);
|
||||
}
|
||||
|
||||
void cancelPendingOperations() {
|
||||
bindingOperations.values().forEach(future -> future.cancel(false));
|
||||
sessions.values().forEach(future -> future.cancel(false));
|
||||
}
|
||||
|
||||
BindingContext matchingLiveBinding(RuntimeBindingRecord record) {
|
||||
LiveBinding live = liveBindings.get(record.getBindingId());
|
||||
return live != null && live.generation() == record.getGeneration()
|
||||
&& record.getLease() != null
|
||||
&& sameLease(live.lease(), record.getLease())
|
||||
? new BindingContext(record, live.lease()) : null;
|
||||
}
|
||||
|
||||
boolean isLiveBinding(RuntimeBindingRecord record) {
|
||||
return matchingLiveBinding(record) != null;
|
||||
}
|
||||
|
||||
BindingContext requireLiveBinding(RuntimeBindingRecord record) {
|
||||
BindingContext live = matchingLiveBinding(record);
|
||||
if (live == null) {
|
||||
throw unavailable("runtime_reconciliation_required",
|
||||
"persisted Runtime readiness requires adoption or "
|
||||
+ "reconciliation in this Broker process");
|
||||
}
|
||||
return live;
|
||||
}
|
||||
|
||||
CompletionStage<SessionContext> requireSession(
|
||||
String harnessSessionId, String runtimeSessionId) {
|
||||
String harnessId = BrokerValues.requireId(harnessSessionId,
|
||||
"harnessSessionId");
|
||||
String runtimeId = BrokerValues.requireId(runtimeSessionId,
|
||||
"runtimeSessionId");
|
||||
CompletableFuture<SessionContext> context = sessions.get(runtimeId);
|
||||
if (context == null) {
|
||||
return CompletableFuture.failedFuture(new RuntimeBrokerException(404,
|
||||
"runtime_session_not_found",
|
||||
"Runtime Session is not active in this Broker process", false));
|
||||
}
|
||||
return context.thenApply(value -> {
|
||||
if (!value.session().getHarnessSessionId().equals(harnessId)) {
|
||||
throw new RuntimeBrokerException(409, "runtime_session_conflict",
|
||||
"Runtime Session belongs to another Harness Session", false);
|
||||
}
|
||||
return value;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* An execution permanently points at the binding generation it was
|
||||
* dispatched to. Once that generation is retired, replaced, or gone, no
|
||||
* Runtime can answer for it, so polling must stop rather than retry.
|
||||
*/
|
||||
void requireAnswerableBinding(ToolExecutionRecord record) {
|
||||
RuntimeBindingRecord binding = bindingRepository.findById(
|
||||
record.getBindingId());
|
||||
if (binding == null
|
||||
|| binding.getGeneration() != record.getRuntimeGeneration()
|
||||
|| (binding.getState() != RuntimeBindingRecord.State.READY
|
||||
&& binding.getState()
|
||||
!= RuntimeBindingRecord.State.DRAINING)) {
|
||||
throw evidenceUnavailable();
|
||||
}
|
||||
}
|
||||
|
||||
void invalidateBinding(RuntimeBindingRecord record) {
|
||||
liveBindings.remove(record.getBindingId());
|
||||
RuntimeBindingRecord claimed = bindingRepository.claimOperation(
|
||||
record.getBindingId(), brokerOwnerId, operationLeaseDuration);
|
||||
if (claimed != null) {
|
||||
try {
|
||||
if (claimed.getState() == RuntimeBindingRecord.State.READY) {
|
||||
bindingRepository.compareAndSet(claimed, claimed.withState(
|
||||
RuntimeBindingRecord.State.LOST, claimed.getLease(), clock.instant()));
|
||||
}
|
||||
} finally {
|
||||
releaseOperationQuietly(claimed.getBindingId(), claimed.getOperationGeneration());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void requireUsableLease(SessionContext context) {
|
||||
if (!provisioner.isUsable(context.lease())) {
|
||||
invalidateBinding(context.binding());
|
||||
throw unavailable("runtime_provision_failed",
|
||||
"Managed Runtime process is not alive.");
|
||||
}
|
||||
}
|
||||
|
||||
void releaseOperationQuietly(String bindingId,
|
||||
long operationGeneration) {
|
||||
try {
|
||||
bindingRepository.releaseOperation(bindingId, brokerOwnerId,
|
||||
operationGeneration);
|
||||
} catch (RuntimeException ignored) {
|
||||
// The claim lapses on its own when the release cannot be written.
|
||||
}
|
||||
}
|
||||
|
||||
static boolean sameLease(RuntimeLease left,
|
||||
RuntimeLease right) {
|
||||
return left.getRuntimeInstanceId().equals(
|
||||
right.getRuntimeInstanceId())
|
||||
&& left.getEndpoint().equals(right.getEndpoint())
|
||||
&& left.getToken().equals(right.getToken())
|
||||
&& left.getLeaseId().equals(right.getLeaseId())
|
||||
&& left.getEpoch() == right.getEpoch();
|
||||
}
|
||||
|
||||
static RuntimeBrokerException evidenceUnavailable() {
|
||||
return new RuntimeBrokerException(409,
|
||||
"runtime_execution_evidence_unavailable",
|
||||
"The original Runtime generation cannot answer for this "
|
||||
+ "execution", false);
|
||||
}
|
||||
|
||||
private static RuntimeBrokerException unavailable(String code, String message) {
|
||||
return new RuntimeBrokerException(503, code, message, true);
|
||||
}
|
||||
|
||||
private record LiveBinding(long generation, RuntimeLease lease) {
|
||||
}
|
||||
|
||||
record BindingContext(RuntimeBindingRecord record,
|
||||
RuntimeLease lease) {
|
||||
}
|
||||
|
||||
static final class SessionContext {
|
||||
// One guard for the state below and every context.lock() region at
|
||||
// the callers. Like them it must not pin a virtual-thread carrier
|
||||
// when a repository call inside the guard blocks (JDK 21).
|
||||
private final ReentrantLock guard = new ReentrantLock();
|
||||
private final RuntimeSession session;
|
||||
private final RuntimeBindingRecord binding;
|
||||
private final RuntimeLease lease;
|
||||
// The background process rows this Broker admitted for the Session,
|
||||
// until each settles — the release sweep asks exactly these owners.
|
||||
private final Set<String> backgroundProcesses = new LinkedHashSet<>();
|
||||
private int activeControls;
|
||||
private CompletableFuture<Boolean> release;
|
||||
// One release attempt at a time per context: the sweep's controls
|
||||
// belong to whoever holds this, so a concurrent caller joins it
|
||||
// instead of sweeping against it and reading its own attempts busy.
|
||||
private CompletableFuture<Boolean> releaseAttempt;
|
||||
// The latest acquire's mount claim: only that acquirer may hand the
|
||||
// mount back (#13800).
|
||||
private String mountClaim;
|
||||
|
||||
SessionContext(RuntimeSession session, RuntimeBindingRecord binding,
|
||||
RuntimeLease lease) {
|
||||
this.session = session;
|
||||
this.binding = binding;
|
||||
this.lease = lease;
|
||||
}
|
||||
|
||||
void lock() {
|
||||
guard.lock();
|
||||
}
|
||||
|
||||
void unlock() {
|
||||
guard.unlock();
|
||||
}
|
||||
|
||||
RuntimeSession session() {
|
||||
return session;
|
||||
}
|
||||
|
||||
RuntimeBindingRecord binding() {
|
||||
return binding;
|
||||
}
|
||||
|
||||
RuntimeLease lease() {
|
||||
return lease;
|
||||
}
|
||||
|
||||
Set<String> backgroundProcesses() {
|
||||
lock();
|
||||
try {
|
||||
return backgroundProcesses;
|
||||
} finally {
|
||||
unlock();
|
||||
}
|
||||
}
|
||||
|
||||
void beginControl() {
|
||||
lock();
|
||||
try {
|
||||
activeControls++;
|
||||
} finally {
|
||||
unlock();
|
||||
}
|
||||
}
|
||||
|
||||
void endControl() {
|
||||
lock();
|
||||
try {
|
||||
activeControls--;
|
||||
} finally {
|
||||
unlock();
|
||||
}
|
||||
}
|
||||
|
||||
String mountClaim() {
|
||||
lock();
|
||||
try {
|
||||
return mountClaim;
|
||||
} finally {
|
||||
unlock();
|
||||
}
|
||||
}
|
||||
|
||||
void mountClaim(String next) {
|
||||
lock();
|
||||
try {
|
||||
mountClaim = next;
|
||||
} finally {
|
||||
unlock();
|
||||
}
|
||||
}
|
||||
|
||||
boolean hasActiveControl() {
|
||||
lock();
|
||||
try {
|
||||
return activeControls > 0;
|
||||
} finally {
|
||||
unlock();
|
||||
}
|
||||
}
|
||||
|
||||
CompletableFuture<Boolean> release() {
|
||||
lock();
|
||||
try {
|
||||
return release;
|
||||
} finally {
|
||||
unlock();
|
||||
}
|
||||
}
|
||||
|
||||
void release(CompletableFuture<Boolean> next) {
|
||||
lock();
|
||||
try {
|
||||
release = next;
|
||||
} finally {
|
||||
unlock();
|
||||
}
|
||||
}
|
||||
|
||||
CompletableFuture<Boolean> releaseAttempt() {
|
||||
lock();
|
||||
try {
|
||||
return releaseAttempt;
|
||||
} finally {
|
||||
unlock();
|
||||
}
|
||||
}
|
||||
|
||||
void releaseAttempt(CompletableFuture<Boolean> next) {
|
||||
lock();
|
||||
try {
|
||||
releaseAttempt = next;
|
||||
} finally {
|
||||
unlock();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
CompletionStage<SessionContext> requireReadySession(
|
||||
String harnessSessionId, String runtimeSessionId) {
|
||||
return requireSession(harnessSessionId, runtimeSessionId)
|
||||
.thenApply(value -> {
|
||||
requireReadySessionRecord(value);
|
||||
return value;
|
||||
});
|
||||
}
|
||||
|
||||
RuntimeSessionRecord requireReadySessionRecord(
|
||||
SessionContext context) {
|
||||
RuntimeSessionRecord record = sessionRepository.findById(
|
||||
context.session().getScope(),
|
||||
context.session().getRuntimeSessionId());
|
||||
if (record == null) {
|
||||
throw notFound("runtime_session_not_found",
|
||||
"Runtime Session was not found");
|
||||
}
|
||||
RuntimeBindingRecord parent = bindingRepository.findById(record.getBindingId());
|
||||
if (parent == null || parent.getGeneration() != record.getRuntimeGeneration()
|
||||
|| parent.getState() != RuntimeBindingRecord.State.READY
|
||||
&& parent.getState() != RuntimeBindingRecord.State.DRAINING) {
|
||||
throw conflict("runtime_admission_closed", "Runtime generation is no longer live");
|
||||
}
|
||||
if (!record.getBindingId().equals(context.binding().getBindingId())
|
||||
|| record.getRuntimeGeneration() != context.binding().getGeneration()
|
||||
|| record.getState() != RuntimeSessionRecord.State.READY) {
|
||||
throw conflict("runtime_session_not_ready",
|
||||
"Runtime Session is not ready");
|
||||
}
|
||||
return record;
|
||||
}
|
||||
|
||||
CompletionStage<RuntimeSessionRecord> persistedSession(
|
||||
String harnessSessionId, String runtimeSessionId) {
|
||||
return mapFailure(safeStage(() -> sessionResolver.resolveTenant(harnessSessionId)),
|
||||
"runtime_scope_resolution_failed", "Runtime scope resolution failed").thenCompose(tenant -> {
|
||||
RuntimeSessionRecord record = sessionRepository.findHistorical(tenant, harnessSessionId, runtimeSessionId);
|
||||
if (record == null) {
|
||||
throw notFound("runtime_session_not_found",
|
||||
"Runtime Session was not found");
|
||||
}
|
||||
if (!record.getSession().getHarnessSessionId().equals(
|
||||
harnessSessionId)) {
|
||||
throw conflict("runtime_session_conflict",
|
||||
"Runtime Session belongs to another Harness Session");
|
||||
}
|
||||
var parent = bindingRepository.findById(record.getBindingId());
|
||||
if (parent == null || parent.getGeneration() != record.getRuntimeGeneration()
|
||||
|| !parent.getRequest().getScope().equals(record.getSession().getScope())) {
|
||||
throw conflict("runtime_session_conflict", "Historical Runtime parent differs");
|
||||
}
|
||||
if (!parent.getRequest().isManagedContext()) {
|
||||
return RuntimeBrokerSupport.resolveScope(sessionResolver, harnessSessionId, false, null).thenApply(scope -> {
|
||||
if (!scope.equals(record.getSession().getScope())) {
|
||||
throw notFound("runtime_session_not_found", "Runtime Session was not found");
|
||||
}
|
||||
return record;
|
||||
});
|
||||
}
|
||||
return CompletableFuture.completedFuture(record);
|
||||
});
|
||||
}
|
||||
}
|
||||
+192
-1812
File diff suppressed because it is too large.
Load diff
+206
@@ -0,0 +1,206 @@
|
||||
package com.alibaba.qwen.code.runtimebroker.service;
|
||||
|
||||
import com.alibaba.qwen.code.runtimebroker.BrokerValues;
|
||||
import com.alibaba.qwen.code.runtimebroker.HarnessSessionResolver;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeScope;
|
||||
import com.alibaba.qwen.code.runtimebroker.StoreFailureException;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionException;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
final class RuntimeBrokerSupport {
|
||||
private RuntimeBrokerSupport() { }
|
||||
|
||||
static final Set<String> RUNTIME_STATUS_STATES = Set.of(
|
||||
"prepared", "executing", "cancel_requested", "settled",
|
||||
"unknown");
|
||||
static final int MAX_CAS_ATTEMPTS = 16;
|
||||
|
||||
static <T> CompletionStage<T> safeStage(
|
||||
Supplier<CompletionStage<T>> supplier) {
|
||||
try {
|
||||
CompletionStage<T> stage = supplier.get();
|
||||
if (stage == null) {
|
||||
return failed(new IllegalStateException(
|
||||
"operation returned no CompletionStage"));
|
||||
}
|
||||
return stage;
|
||||
} catch (RuntimeException | Error exception) {
|
||||
return failed(exception);
|
||||
}
|
||||
}
|
||||
|
||||
static <T> CompletionStage<T> mapFailure(
|
||||
CompletionStage<T> stage, String code, String message) {
|
||||
return stage.handle((value, error) -> {
|
||||
if (error == null) {
|
||||
return value;
|
||||
}
|
||||
Throwable cause = unwrap(error);
|
||||
if (cause instanceof RuntimeBrokerException) {
|
||||
throw new CompletionException(cause);
|
||||
}
|
||||
throw new CompletionException(
|
||||
unavailable(code, message, cause));
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Only a genuine store outage maps to a retryable 503 here: client-input
|
||||
* errors, permanent invariant violations, and JVM errors keep the
|
||||
* non-retryable mapping the HTTP layer gives them.
|
||||
*/
|
||||
static <T> CompletionStage<T> mapStoreFailure(
|
||||
CompletionStage<T> stage) {
|
||||
return stage.handle((value, error) -> {
|
||||
if (error == null) {
|
||||
return value;
|
||||
}
|
||||
Throwable cause = unwrap(error);
|
||||
if (cause instanceof RuntimeBrokerException) {
|
||||
throw new CompletionException(cause);
|
||||
}
|
||||
if (cause instanceof StoreFailureException) {
|
||||
throw new CompletionException(unavailable(
|
||||
"runtime_broker_store_unavailable",
|
||||
"Managed Runtime execution store is unavailable.",
|
||||
cause));
|
||||
}
|
||||
throw new CompletionException(cause);
|
||||
});
|
||||
}
|
||||
|
||||
static Throwable unwrap(Throwable error) {
|
||||
Throwable current = error;
|
||||
while ((current instanceof CompletionException)
|
||||
&& current.getCause() != null) {
|
||||
current = current.getCause();
|
||||
}
|
||||
return current;
|
||||
}
|
||||
|
||||
static <T> CompletableFuture<T> failed(Throwable error) {
|
||||
return CompletableFuture.failedFuture(error);
|
||||
}
|
||||
|
||||
static RuntimeBrokerException invalid(String code,
|
||||
String message) {
|
||||
return new RuntimeBrokerException(400, code, message, false);
|
||||
}
|
||||
|
||||
static RuntimeBrokerException notFound(String code,
|
||||
String message) {
|
||||
return new RuntimeBrokerException(404, code, message, false);
|
||||
}
|
||||
|
||||
static RuntimeBrokerException conflict(String code,
|
||||
String message) {
|
||||
return new RuntimeBrokerException(409, code, message, false);
|
||||
}
|
||||
|
||||
static RuntimeBrokerException conflict(String code,
|
||||
String message, Throwable cause) {
|
||||
return new RuntimeBrokerException(409, code, message, false, cause);
|
||||
}
|
||||
|
||||
static RuntimeBrokerException unavailable(String code,
|
||||
String message) {
|
||||
return new RuntimeBrokerException(503, code, message, true);
|
||||
}
|
||||
|
||||
static RuntimeBrokerException unavailable(String code,
|
||||
String message, Throwable cause) {
|
||||
return new RuntimeBrokerException(503, code, message, true, cause);
|
||||
}
|
||||
|
||||
static String referenceString(Map<String, Object> reference,
|
||||
String field) {
|
||||
Object value = reference == null ? null : reference.get(field);
|
||||
if (!(value instanceof String)) {
|
||||
throw invalid("runtime_reference_invalid",
|
||||
"reference " + field + " is required");
|
||||
}
|
||||
try {
|
||||
return BrokerValues.requireWellFormed(BrokerValues.requireId(
|
||||
(String) value, "reference." + field),
|
||||
"reference." + field);
|
||||
} catch (IllegalArgumentException exception) {
|
||||
throw invalid("runtime_reference_invalid",
|
||||
"reference " + field + " is invalid");
|
||||
}
|
||||
}
|
||||
|
||||
static Map<String, Object> immutableMap(Object value,
|
||||
String name) {
|
||||
if (!(value instanceof Map<?, ?>)) {
|
||||
throw invalid("runtime_payload_invalid",
|
||||
name + " must be an object");
|
||||
}
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, ?> source = (Map<String, ?>) value;
|
||||
try {
|
||||
return BrokerValues.immutableMap(source);
|
||||
} catch (IllegalArgumentException exception) {
|
||||
throw invalid("runtime_payload_invalid",
|
||||
name + " is invalid");
|
||||
}
|
||||
}
|
||||
|
||||
static Map<String, Object> runtimeMap(Object value,
|
||||
String name) {
|
||||
if (!(value instanceof Map<?, ?>)) {
|
||||
throw unavailable("runtime_execution_cancel_failed",
|
||||
name + " must be an object");
|
||||
}
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, ?> source = (Map<String, ?>) value;
|
||||
try {
|
||||
return BrokerValues.immutableMap(source);
|
||||
} catch (IllegalArgumentException exception) {
|
||||
throw unavailable("runtime_execution_cancel_failed",
|
||||
name + " is invalid", exception);
|
||||
}
|
||||
}
|
||||
|
||||
static Map<String, Object> projectV3Status(Map<String, Object> response) {
|
||||
Object state = response.get("state");
|
||||
if (!(state instanceof String name) || !RUNTIME_STATUS_STATES.contains(name)
|
||||
|| "settled".equals(name) != (response.get("result") instanceof Map)) {
|
||||
throw conflict("runtime_execution_conflict", "Tool v3 returned an invalid status");
|
||||
}
|
||||
return "settled".equals(name)
|
||||
? Map.of("state", name, "result", response.get("result"))
|
||||
: Map.of("state", name);
|
||||
}
|
||||
|
||||
static CompletionStage<RuntimeScope> resolveScope(HarnessSessionResolver sessionResolver,
|
||||
String harnessSessionId, boolean admission,
|
||||
RuntimeLifecycleAuthority authority) {
|
||||
return mapFailure(safeStage(
|
||||
() -> admission
|
||||
? sessionResolver.resolveAdmission(harnessSessionId,
|
||||
authority)
|
||||
: sessionResolver.resolve(harnessSessionId, authority)),
|
||||
"runtime_scope_resolution_failed",
|
||||
"Runtime scope resolution failed").thenApply(scope -> {
|
||||
if (scope == null) {
|
||||
throw unavailable("runtime_scope_resolution_failed",
|
||||
"Runtime scope resolver returned no scope");
|
||||
}
|
||||
return scope;
|
||||
});
|
||||
}
|
||||
|
||||
static void requireOpen(AtomicBoolean closed) {
|
||||
if (closed.get()) {
|
||||
throw new IllegalStateException("Runtime Broker is closed");
|
||||
}
|
||||
}
|
||||
}
|
||||
+343
@@ -0,0 +1,343 @@
|
||||
package com.alibaba.qwen.code.runtimebroker.service;
|
||||
|
||||
import com.alibaba.qwen.code.runtimebroker.BrokerValues;
|
||||
import com.alibaba.qwen.code.runtimebroker.CsiFilesRetirementProfile;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSessionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeTransport;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.protocol.ProviderRuntimeProtocol;
|
||||
import com.alibaba.qwen.code.runtimebroker.service.RuntimeBindingRegistry.SessionContext;
|
||||
import com.alibaba.qwen.code.runtimebroker.service.RuntimeExecutionState.OwnedExecution;
|
||||
import com.alibaba.qwen.code.runtimebroker.transport.ManagedHoldFamily;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.ScheduledExecutorService;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.function.BiConsumer;
|
||||
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.MAX_CAS_ATTEMPTS;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.conflict;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.mapFailure;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.mapStoreFailure;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.notFound;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.requireOpen;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.runtimeMap;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.safeStage;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.unavailable;
|
||||
|
||||
final class RuntimeExecutionCancellation {
|
||||
private final RuntimeBindingRegistry bindingRegistry;
|
||||
private final RuntimeExecutionState executionState;
|
||||
private final ToolExecutionRepository executionRepository;
|
||||
private final RuntimeTransport transport;
|
||||
private final Set<String> invocations;
|
||||
private final ScheduledExecutorService scheduler;
|
||||
private final Duration operationLeaseDuration;
|
||||
private final AtomicBoolean closed;
|
||||
|
||||
RuntimeExecutionCancellation(RuntimeBindingRegistry bindingRegistry,
|
||||
RuntimeExecutionState executionState,
|
||||
ToolExecutionRepository executionRepository,
|
||||
RuntimeTransport transport,
|
||||
Set<String> invocations,
|
||||
ScheduledExecutorService scheduler,
|
||||
Duration operationLeaseDuration,
|
||||
AtomicBoolean closed) {
|
||||
this.bindingRegistry = bindingRegistry;
|
||||
this.executionState = executionState;
|
||||
this.executionRepository = executionRepository;
|
||||
this.transport = transport;
|
||||
this.invocations = invocations;
|
||||
this.scheduler = scheduler;
|
||||
this.operationLeaseDuration = operationLeaseDuration;
|
||||
this.closed = closed;
|
||||
}
|
||||
|
||||
static final Set<String> RUNTIME_EXECUTION_STATES = Set.of(
|
||||
"prepared", "executing", "cancel_requested", "settled",
|
||||
"unknown");
|
||||
|
||||
CompletionStage<ToolExecutionRecord> cancelExecution(
|
||||
String harnessSessionId, String runtimeSessionId,
|
||||
String executionCallId, BiConsumer<SessionContext, ToolExecutionRecord> dispatch) {
|
||||
requireOpen(closed);
|
||||
String executionId = BrokerValues.requireId(executionCallId,
|
||||
"executionCallId");
|
||||
return mapStoreFailure(safeStage(() -> {
|
||||
OwnedExecution owned = executionState.requireOwnership(harnessSessionId,
|
||||
runtimeSessionId, executionId);
|
||||
ToolExecutionRecord stored = owned.record();
|
||||
if (stored.isTerminal()) {
|
||||
if (!isPreparedProviderCancellation(stored)) {
|
||||
return CompletableFuture.completedFuture(stored);
|
||||
}
|
||||
RuntimeBindingRecord binding = owned.binding();
|
||||
// Only a READY Session whose binding can still answer may hold
|
||||
// the worker's preparation; anywhere else the terminal receipt
|
||||
// stands alone, the same rule that lets Broker HTTP read
|
||||
// terminal receipts without a READY Session.
|
||||
if (owned.session().getState() != RuntimeSessionRecord.State.READY
|
||||
|| binding.getState() != RuntimeBindingRecord.State.READY
|
||||
&& binding.getState() != RuntimeBindingRecord.State.DRAINING) {
|
||||
return CompletableFuture.completedFuture(stored);
|
||||
}
|
||||
// That worker may still hold the preparation, so the
|
||||
// cancellation is not acknowledged without its evidence. A
|
||||
// process without a live Session (a replaced Broker, or an
|
||||
// acquire still adopting) asks for adoption, as reconciliation
|
||||
// does; another Harness's Session under the same id stays a
|
||||
// conflict below.
|
||||
CompletableFuture<SessionContext> local = bindingRegistry.session(runtimeSessionId);
|
||||
if (local == null || !local.isDone() || local.isCompletedExceptionally()) {
|
||||
throw unavailable("runtime_reconciliation_required",
|
||||
"Runtime Session is not active in this Broker process; "
|
||||
+ "acquire the Session again to confirm the cancellation");
|
||||
}
|
||||
}
|
||||
return executionState.unknownWhenAdmissionClosed(
|
||||
bindingRegistry.requireReadySession(harnessSessionId, runtimeSessionId)
|
||||
.thenCompose(context -> {
|
||||
ToolExecutionRecord requested;
|
||||
boolean publicationV3;
|
||||
context.lock();
|
||||
try {
|
||||
bindingRegistry.requireReadySessionRecord(context);
|
||||
ToolExecutionRecord current = executionState.requireExecution(
|
||||
context, executionId);
|
||||
// A hold row never dispatches, so a cancel would
|
||||
// settle it immediately — while its process
|
||||
// provably runs. It settles only on the owner's
|
||||
// stop evidence; stop goes through the
|
||||
// maintenance route.
|
||||
if (ManagedHoldFamily.of(current) != null) {
|
||||
throw conflict("runtime_execution_conflict",
|
||||
"A background process settles only on stop evidence");
|
||||
}
|
||||
if (CsiFilesRetirementProfile.isProfile(context.binding().getRequest().getScope())
|
||||
&& current.getAuthorizedDispatchGeneration() != null) {
|
||||
throw conflict("csi_execution_cancel_unavailable",
|
||||
"An authorized private CSI execution does not support cancellation");
|
||||
}
|
||||
requested = requestCancel(current);
|
||||
if (isPreparedProviderCancellation(requested)) {
|
||||
context.beginControl();
|
||||
return cancelPreparedProvider(context, requested);
|
||||
}
|
||||
publicationV3 = "deferred_v3".equals(
|
||||
requested.getReference().get("dispatchMode"));
|
||||
// An UNKNOWN record may still have an invocation
|
||||
// running in this process, or be one its original
|
||||
// Runtime can still answer for; it gets the
|
||||
// physical cancel below but is never settled from
|
||||
// here.
|
||||
if (requested.isTerminal()
|
||||
|| (requested.getState()
|
||||
== ToolExecutionRecord.State.UNKNOWN
|
||||
&& !invocations.contains(
|
||||
executionId)
|
||||
&& !publicationV3
|
||||
&& !requested.observableAfterLoss())) {
|
||||
return CompletableFuture.completedFuture(
|
||||
requested);
|
||||
}
|
||||
} finally {
|
||||
context.unlock();
|
||||
}
|
||||
if (requested.getState()
|
||||
== ToolExecutionRecord.State.DISPATCHING) {
|
||||
dispatch.accept(context, requested);
|
||||
ToolExecutionRecord latest = executionRepository
|
||||
.findByExecutionCallId(executionId);
|
||||
return CompletableFuture.completedFuture(
|
||||
latest == null ? requested : latest);
|
||||
}
|
||||
if (requested.getState()
|
||||
== ToolExecutionRecord.State
|
||||
.CANCEL_REQUESTED
|
||||
&& !invocations.contains(executionId)) {
|
||||
// Fence a lapsed claim nothing here is running. A
|
||||
// claim the repository still holds live falls
|
||||
// through to the physical cancel, as does an
|
||||
// invocation still running in this process.
|
||||
ToolExecutionRecord fenced;
|
||||
try {
|
||||
fenced = executionState.fenceLapsedClaim(requested);
|
||||
} catch (RuntimeException exception) {
|
||||
throw unavailable(
|
||||
"runtime_execution_cancel_failed",
|
||||
"Runtime execution cancellation failed",
|
||||
exception);
|
||||
}
|
||||
if (fenced == null || fenced.getState()
|
||||
!= ToolExecutionRecord.State
|
||||
.CANCEL_REQUESTED) {
|
||||
return CompletableFuture.completedFuture(
|
||||
fenced == null ? requested : fenced);
|
||||
}
|
||||
}
|
||||
if (requested.getState()
|
||||
!= ToolExecutionRecord.State
|
||||
.CANCEL_REQUESTED
|
||||
&& requested.getState()
|
||||
!= ToolExecutionRecord.State.UNKNOWN) {
|
||||
return CompletableFuture.completedFuture(requested);
|
||||
}
|
||||
bindingRegistry.requireUsableLease(context);
|
||||
return mapFailure(safeStage(() ->
|
||||
publicationV3
|
||||
? transport.cancelV3(context.lease(), context.session(),
|
||||
requested.getReference()).thenApply(
|
||||
RuntimeBrokerSupport::projectV3Status)
|
||||
: transport.cancel(context.lease(), context.session(),
|
||||
requested.getReference())),
|
||||
"runtime_execution_cancel_failed",
|
||||
"Runtime execution cancellation failed")
|
||||
.thenApply(status -> {
|
||||
absorbCancellationStatus(requested, status);
|
||||
ToolExecutionRecord latest =
|
||||
executionRepository
|
||||
.findByExecutionCallId(
|
||||
executionId);
|
||||
return latest == null ? requested : latest;
|
||||
});
|
||||
}),
|
||||
harnessSessionId, runtimeSessionId, executionId);
|
||||
}));
|
||||
}
|
||||
|
||||
private static boolean isPreparedProviderCancellation(ToolExecutionRecord record) {
|
||||
return record.isSettled() && record.isCancelRequested()
|
||||
&& record.getDispatchGeneration() == 0
|
||||
&& ProviderRuntimeProtocol.isReference(record.getReference());
|
||||
}
|
||||
|
||||
private CompletionStage<ToolExecutionRecord> cancelPreparedProvider(
|
||||
SessionContext context, ToolExecutionRecord requested) {
|
||||
CompletableFuture<ToolExecutionRecord> result = new CompletableFuture<>();
|
||||
result.orTimeout(Math.max(1, operationLeaseDuration.toMillis()), TimeUnit.MILLISECONDS);
|
||||
observePreparedCancellation(context, requested, safeStage(() -> {
|
||||
bindingRegistry.requireUsableLease(context);
|
||||
return transport.cancel(context.lease(), context.session(), requested.getReference());
|
||||
}), result);
|
||||
return mapFailure(result, "runtime_execution_cancel_failed",
|
||||
"Runtime execution cancellation failed")
|
||||
.whenComplete((ignored, error) -> context.endControl());
|
||||
}
|
||||
|
||||
private void observePreparedCancellation(SessionContext context, ToolExecutionRecord requested,
|
||||
CompletionStage<Map<String, Object>> observation, CompletableFuture<ToolExecutionRecord> completion) {
|
||||
observation.whenComplete((status, error) -> {
|
||||
if (completion.isDone()) {
|
||||
return;
|
||||
}
|
||||
if (error != null) {
|
||||
completion.completeExceptionally(error);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
if (status != null && "settled".equals(status.get("state"))
|
||||
&& status.get("result") instanceof Map<?, ?> result
|
||||
&& ("cancelled".equals(result.get("executionStatus"))
|
||||
|| "not_started".equals(result.get("executionStatus")))) {
|
||||
completion.complete(requested);
|
||||
} else if (status != null && ("prepared".equals(status.get("state"))
|
||||
|| "executing".equals(status.get("state"))
|
||||
|| "cancel_requested".equals(status.get("state")))) {
|
||||
scheduler.schedule(() -> {
|
||||
if (!completion.isDone()) {
|
||||
observePreparedCancellation(context, requested, safeStage(() -> {
|
||||
bindingRegistry.requireUsableLease(context);
|
||||
return transport.status(context.lease(), context.session(),
|
||||
requested.getReference(), requested.getLastSequence());
|
||||
}), completion);
|
||||
}
|
||||
}, 25, TimeUnit.MILLISECONDS);
|
||||
} else {
|
||||
// unknown or an unexpected terminal answer can never
|
||||
// become the confirmation, so a retry cannot converge;
|
||||
// fail non-retryably rather than loop the same round
|
||||
// trip. unknown never becomes not_started evidence.
|
||||
Object state = status == null ? null : status.get("state");
|
||||
throw new RuntimeBrokerException(409,
|
||||
"runtime_execution_cancel_unconfirmed",
|
||||
"unknown".equals(state)
|
||||
? "Runtime no longer retains the prepared invocation"
|
||||
: "Runtime did not confirm prepared invocation cancellation",
|
||||
false);
|
||||
}
|
||||
} catch (RuntimeException failure) {
|
||||
completion.completeExceptionally(failure);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private ToolExecutionRecord requestCancel(
|
||||
ToolExecutionRecord initial) {
|
||||
ToolExecutionRecord current = initial;
|
||||
for (int attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt++) {
|
||||
if (current.isTerminal()) {
|
||||
return current;
|
||||
}
|
||||
ToolExecutionRecord updated = executionRepository.requestCancel(
|
||||
current.getExecutionCallId(), current.getVersion());
|
||||
if (updated != null) {
|
||||
return updated;
|
||||
}
|
||||
current = executionRepository.findByExecutionCallId(
|
||||
current.getExecutionCallId());
|
||||
if (current == null) {
|
||||
throw notFound("runtime_execution_not_found",
|
||||
"Runtime execution was not found");
|
||||
}
|
||||
}
|
||||
throw conflict("runtime_execution_state_conflict",
|
||||
"Runtime execution changed while requesting cancellation");
|
||||
}
|
||||
|
||||
private void absorbCancellationStatus(ToolExecutionRecord requested,
|
||||
Map<String, Object> status) {
|
||||
Object state = status == null ? null : status.get("state");
|
||||
if (!(state instanceof String)
|
||||
|| !RUNTIME_EXECUTION_STATES.contains(state)) {
|
||||
throw unavailable("runtime_execution_cancel_failed",
|
||||
"Runtime cancellation returned an invalid status");
|
||||
}
|
||||
if (!"settled".equals(state)) {
|
||||
return;
|
||||
}
|
||||
Map<String, Object> result = runtimeMap(status.get("result"),
|
||||
"cancellation result");
|
||||
try {
|
||||
executionState.settleExecution(requested, result);
|
||||
} catch (IllegalArgumentException exception) {
|
||||
throw unavailable("runtime_execution_cancel_failed",
|
||||
"Runtime cancellation returned an invalid result",
|
||||
exception);
|
||||
} catch (RuntimeBrokerException exception) {
|
||||
if (!"runtime_execution_state_conflict".equals(
|
||||
exception.getCode())) {
|
||||
throw exception;
|
||||
}
|
||||
// The Runtime settled the call after this claim lapsed; fence it
|
||||
// for reconciliation instead of reporting a state conflict. The
|
||||
// conflict stands only while the repository still holds the
|
||||
// claim live; a record another writer settled is returned.
|
||||
ToolExecutionRecord latest = executionState.fenceLapsedClaim(
|
||||
executionRepository.findByExecutionCallId(
|
||||
requested.getExecutionCallId()));
|
||||
if (latest == null || (!latest.isTerminal()
|
||||
&& latest.getState()
|
||||
!= ToolExecutionRecord.State.UNKNOWN)) {
|
||||
throw exception;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+402
@@ -0,0 +1,402 @@
|
||||
package com.alibaba.qwen.code.runtimebroker.service;
|
||||
|
||||
import com.alibaba.qwen.code.runtimebroker.BrokerValues;
|
||||
import com.alibaba.qwen.code.runtimebroker.ExecutionReconciliation;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeProvisioner;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimePublicationVerifier;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeTransport;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.service.RuntimeBindingRegistry.SessionContext;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionException;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.ConcurrentMap;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBindingRegistry.evidenceUnavailable;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.MAX_CAS_ATTEMPTS;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.RUNTIME_STATUS_STATES;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.conflict;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.immutableMap;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.mapFailure;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.notFound;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.requireOpen;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.safeStage;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.unavailable;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.unwrap;
|
||||
|
||||
final class RuntimeExecutionReconciler {
|
||||
private final RuntimeBindingRegistry bindingRegistry;
|
||||
private final RuntimeExecutionState executionState;
|
||||
private final ToolExecutionRepository executionRepository;
|
||||
private final RuntimeProvisioner provisioner;
|
||||
private final RuntimeTransport transport;
|
||||
private final RuntimePublicationVerifier publicationVerifier;
|
||||
private final Duration operationLeaseDuration;
|
||||
private final Clock clock;
|
||||
private final AtomicBoolean closed;
|
||||
private final ExecutionObservationCooldown observationCooldown;
|
||||
|
||||
RuntimeExecutionReconciler(RuntimeBindingRegistry bindingRegistry,
|
||||
RuntimeExecutionState executionState,
|
||||
ToolExecutionRepository executionRepository,
|
||||
RuntimeProvisioner provisioner,
|
||||
RuntimeTransport transport,
|
||||
RuntimePublicationVerifier publicationVerifier,
|
||||
Duration operationLeaseDuration,
|
||||
Clock clock,
|
||||
AtomicBoolean closed,
|
||||
ExecutionObservationCooldown observationCooldown) {
|
||||
this.bindingRegistry = bindingRegistry;
|
||||
this.executionState = executionState;
|
||||
this.executionRepository = executionRepository;
|
||||
this.provisioner = provisioner;
|
||||
this.transport = transport;
|
||||
this.publicationVerifier = publicationVerifier;
|
||||
this.operationLeaseDuration = operationLeaseDuration;
|
||||
this.clock = clock;
|
||||
this.closed = closed;
|
||||
this.observationCooldown = observationCooldown;
|
||||
}
|
||||
|
||||
private final ConcurrentMap<String, CompletableFuture<ExecutionReconciliation>> reconciliations = new ConcurrentHashMap<>();
|
||||
|
||||
void cancelPendingOperations() {
|
||||
reconciliations.values().forEach(future -> future.cancel(false));
|
||||
}
|
||||
|
||||
static final Set<String> RUNTIME_STATUS_FIELDS = Set.of(
|
||||
"state", "result");
|
||||
|
||||
/**
|
||||
* Asks the original Runtime once whether an {@code UNKNOWN} execution
|
||||
* settled, and settles the record only on that evidence. Any other
|
||||
* answer, and any failure, leaves it {@code UNKNOWN}. This never executes
|
||||
* or re-claims the dispatch, and it never retries: polling belongs to
|
||||
* the caller. A record that is not {@code UNKNOWN} is answered from the
|
||||
* repository before any Session or liveness check.
|
||||
*/
|
||||
CompletionStage<ExecutionReconciliation> reconcileExecution(
|
||||
String harnessSessionId, String runtimeSessionId,
|
||||
String executionCallId) {
|
||||
requireOpen(closed);
|
||||
String harnessId = BrokerValues.requireId(harnessSessionId,
|
||||
"harnessSessionId");
|
||||
String runtimeId = BrokerValues.requireId(runtimeSessionId,
|
||||
"runtimeSessionId");
|
||||
String executionId = BrokerValues.requireId(executionCallId,
|
||||
"executionCallId");
|
||||
return mapFailure(safeStage(() -> reconcile(harnessId, runtimeId,
|
||||
executionId)), "runtime_execution_reconcile_failed",
|
||||
"Runtime execution reconciliation failed");
|
||||
}
|
||||
|
||||
/**
|
||||
* The HTTP face's automatic observation of an UNKNOWN execution: the
|
||||
* same evidence-only lookup as {@link #reconcileExecution}, except that
|
||||
* a lookup that just ran cools the execution for a short window, so
|
||||
* rapid polling shares the one answer instead of fanning every request
|
||||
* through to the worker (issue #13183 saw hundreds of status calls per
|
||||
* execution this way).
|
||||
*/
|
||||
CompletionStage<ExecutionReconciliation> observeExecution(
|
||||
String harnessSessionId, String runtimeSessionId,
|
||||
String executionCallId) {
|
||||
requireOpen(closed);
|
||||
String harnessId = BrokerValues.requireId(harnessSessionId,
|
||||
"harnessSessionId");
|
||||
String runtimeId = BrokerValues.requireId(runtimeSessionId,
|
||||
"runtimeSessionId");
|
||||
String executionId = BrokerValues.requireId(executionCallId,
|
||||
"executionCallId");
|
||||
return mapFailure(safeStage(() -> reconcile(harnessId, runtimeId,
|
||||
executionId, true)), "runtime_execution_reconcile_failed",
|
||||
"Runtime execution reconciliation failed");
|
||||
}
|
||||
|
||||
private CompletionStage<ExecutionReconciliation> reconcile(
|
||||
String harnessSessionId, String runtimeSessionId,
|
||||
String executionCallId) {
|
||||
return reconcile(harnessSessionId, runtimeSessionId, executionCallId,
|
||||
false);
|
||||
}
|
||||
|
||||
private CompletionStage<ExecutionReconciliation> reconcile(
|
||||
String harnessSessionId, String runtimeSessionId,
|
||||
String executionCallId, boolean cooled) {
|
||||
ToolExecutionRecord unknown = executionState.requireOwnedExecution(harnessSessionId,
|
||||
runtimeSessionId, executionCallId);
|
||||
if (unknown.getState() != ToolExecutionRecord.State.UNKNOWN) {
|
||||
return CompletableFuture.completedFuture(notUnknown(unknown,
|
||||
null));
|
||||
}
|
||||
if ("deferred_v3".equals(unknown.getReference().get("dispatchMode"))
|
||||
&& publicationVerifier != null) {
|
||||
Map<String, Object> durable = publicationVerifier.finished(unknown);
|
||||
if (durable != null) {
|
||||
return CompletableFuture.completedFuture(absorbRuntimeStatus(unknown,
|
||||
Map.of("state", "settled", "result", durable), false));
|
||||
}
|
||||
}
|
||||
CompletableFuture<SessionContext> local = bindingRegistry.session(
|
||||
runtimeSessionId);
|
||||
// Process-local Sessions are keyed by Runtime Session id alone, so a
|
||||
// Session another Harness holds under the same id is not a route.
|
||||
SessionContext context = local == null || !local.isDone()
|
||||
|| local.isCompletedExceptionally() ? null : local.join();
|
||||
if (context == null || !context.session().getHarnessSessionId()
|
||||
.equals(harnessSessionId)) {
|
||||
// No attested route in this process yet: ask for adoption only
|
||||
// while the original generation could still answer.
|
||||
return bindingRegistry.persistedSession(harnessSessionId, runtimeSessionId)
|
||||
.thenApply(ignored -> {
|
||||
bindingRegistry.requireAnswerableBinding(unknown);
|
||||
throw unavailable("runtime_reconciliation_required",
|
||||
"Runtime Session is not active in this "
|
||||
+ "Broker process; adopt its binding "
|
||||
+ "and acquire the Session again");
|
||||
});
|
||||
}
|
||||
return reconcileInContext(context, unknown, false, cooled);
|
||||
}
|
||||
|
||||
CompletionStage<ExecutionReconciliation> reconcileInContext(
|
||||
SessionContext context, ToolExecutionRecord unknown,
|
||||
boolean takeover) {
|
||||
return reconcileInContext(context, unknown, takeover, false);
|
||||
}
|
||||
|
||||
CompletionStage<ExecutionReconciliation> reconcileInContext(
|
||||
SessionContext context, ToolExecutionRecord unknown,
|
||||
boolean takeover, boolean cooled) {
|
||||
requireOpen(closed);
|
||||
bindingRegistry.requireAnswerableBinding(unknown);
|
||||
context.lock();
|
||||
try {
|
||||
bindingRegistry.requireReadySessionRecord(context);
|
||||
} finally {
|
||||
context.unlock();
|
||||
}
|
||||
// Only the binding generation the execution was dispatched to,
|
||||
// reached through this process's attested lease, may answer.
|
||||
bindingRegistry.requireAnswerableBinding(unknown);
|
||||
// A Session keeps its binding generation for life, so a mismatch
|
||||
// means inconsistent records rather than something a retry fixes.
|
||||
if (!unknown.getBindingId().equals(context.binding().getBindingId())
|
||||
|| unknown.getRuntimeGeneration()
|
||||
!= context.binding().getGeneration()) {
|
||||
throw conflict("runtime_execution_conflict",
|
||||
"Runtime execution belongs to another Runtime "
|
||||
+ "generation");
|
||||
}
|
||||
if (!bindingRegistry.hasBinding(context.binding().getBindingId())) {
|
||||
// Local invalidation revokes this route even if its persistent
|
||||
// fence failed. A new observation must establish recovery evidence.
|
||||
throw evidenceUnavailable();
|
||||
}
|
||||
if (!provisioner.isUsable(context.lease())) {
|
||||
// Same retirement as every other operation on a dead lease; the
|
||||
// original Runtime is gone, so no answer can come.
|
||||
bindingRegistry.invalidateBinding(context.binding());
|
||||
throw evidenceUnavailable();
|
||||
}
|
||||
// Adoption can re-register the binding at a different endpoint than
|
||||
// the one this Session was acquired with; that stale route must not be
|
||||
// used, so the lookup fails as runtime_reconciliation_required.
|
||||
bindingRegistry.requireLiveBinding(context.binding());
|
||||
return lookupOnce(context, unknown, takeover, cooled);
|
||||
}
|
||||
|
||||
private CompletionStage<ExecutionReconciliation> lookupOnce(
|
||||
SessionContext context, ToolExecutionRecord unknown,
|
||||
boolean takeover, boolean cooled) {
|
||||
String executionId = unknown.getExecutionCallId();
|
||||
if (cooled) {
|
||||
ExecutionReconciliation cached =
|
||||
observationCooldown.cooled(executionId, unknown);
|
||||
if (cached != null) {
|
||||
return CompletableFuture.completedFuture(cached);
|
||||
}
|
||||
}
|
||||
CompletableFuture<ExecutionReconciliation> created =
|
||||
new CompletableFuture<>();
|
||||
CompletableFuture<ExecutionReconciliation> existing =
|
||||
reconciliations.putIfAbsent(executionId, created);
|
||||
if (existing != null) {
|
||||
return existing;
|
||||
}
|
||||
// Bridge into a future this service owns, so nothing the transport
|
||||
// returns or throws can leave the in-flight slot claimed, and bound
|
||||
// it so a hung transport call cannot hold the slot and every later
|
||||
// poll. Repository work afterwards relies on its own timeouts.
|
||||
CompletableFuture<Map<String, Object>> lookup =
|
||||
new CompletableFuture<>();
|
||||
try {
|
||||
CompletionStage<Map<String, Object>> query = safeStage(() ->
|
||||
"deferred_v3".equals(unknown.getReference().get("dispatchMode"))
|
||||
? transport.statusV3(context.lease(), context.session(),
|
||||
unknown.getReference(), unknown.getLastSequence())
|
||||
.thenApply(RuntimeBrokerSupport::projectV3Status)
|
||||
: transport.status(context.lease(), context.session(),
|
||||
unknown.getReference(), unknown.getLastSequence()));
|
||||
query
|
||||
.whenComplete((status, error) -> {
|
||||
if (error == null) {
|
||||
lookup.complete(status);
|
||||
} else {
|
||||
lookup.completeExceptionally(error);
|
||||
}
|
||||
});
|
||||
} catch (RuntimeException | Error exception) {
|
||||
lookup.completeExceptionally(exception);
|
||||
}
|
||||
lookup.orTimeout(operationLeaseDuration.toMillis(),
|
||||
TimeUnit.MILLISECONDS);
|
||||
// Every caller, joined or not, maps failures in reconcileExecution.
|
||||
lookup.handle((status, error) -> {
|
||||
if (takeover) {
|
||||
requireOpen(closed);
|
||||
}
|
||||
if (error != null) {
|
||||
if (takeover && !(unwrap(error) instanceof Error)) {
|
||||
return new ExecutionReconciliation(unknown,
|
||||
ExecutionReconciliation.Outcome.UNRESOLVED, null);
|
||||
}
|
||||
throw new CompletionException(unwrap(error));
|
||||
}
|
||||
try {
|
||||
return absorbRuntimeStatus(unknown, status, takeover);
|
||||
} catch (RuntimeBrokerException failure) {
|
||||
if (takeover && "runtime_execution_status_invalid".equals(failure.getCode())) {
|
||||
return new ExecutionReconciliation(unknown,
|
||||
ExecutionReconciliation.Outcome.UNRESOLVED, null);
|
||||
}
|
||||
throw failure;
|
||||
}
|
||||
})
|
||||
.whenComplete((reconciled, error) -> {
|
||||
// An Error is not a lookup answer: both paths above
|
||||
// refuse to downgrade one, so the cache must not either.
|
||||
if (cooled && (error == null
|
||||
|| !(unwrap(error) instanceof Error))) {
|
||||
// Stamp before dropping the in-flight slot, so no
|
||||
// sequential observer can slip between them and fan
|
||||
// through to the worker. A failed lookup cools as
|
||||
// UNRESOLVED: the Runtime was just unreachable, and
|
||||
// hammering it changes nothing.
|
||||
observationCooldown.stamp(executionId,
|
||||
error == null ? reconciled
|
||||
: new ExecutionReconciliation(unknown,
|
||||
ExecutionReconciliation.Outcome
|
||||
.UNRESOLVED, null));
|
||||
}
|
||||
reconciliations.remove(executionId, created);
|
||||
if (error == null) {
|
||||
created.complete(reconciled);
|
||||
} else {
|
||||
created.completeExceptionally(unwrap(error));
|
||||
}
|
||||
});
|
||||
return created;
|
||||
}
|
||||
|
||||
private static ExecutionReconciliation notUnknown(
|
||||
ToolExecutionRecord record, String runtimeState) {
|
||||
return new ExecutionReconciliation(record,
|
||||
record.getState() == ToolExecutionRecord.State.ABANDONED
|
||||
? ExecutionReconciliation.Outcome.ABANDONED
|
||||
: record.isSettled() ? ExecutionReconciliation.Outcome.ALREADY_SETTLED
|
||||
: ExecutionReconciliation.Outcome.IN_FLIGHT, runtimeState);
|
||||
}
|
||||
|
||||
ExecutionReconciliation absorbRuntimeStatus(
|
||||
ToolExecutionRecord unknown, Map<String, Object> status, boolean takeover) {
|
||||
if (status == null) {
|
||||
throw invalidStatus(null);
|
||||
}
|
||||
for (Object field : status.keySet()) {
|
||||
if (!(field instanceof String)
|
||||
|| !RUNTIME_STATUS_FIELDS.contains(field)) {
|
||||
throw invalidStatus(null);
|
||||
}
|
||||
}
|
||||
Object state = status.get("state");
|
||||
if (!(state instanceof String)
|
||||
|| !RUNTIME_STATUS_STATES.contains(state)
|
||||
|| "settled".equals(state)
|
||||
!= status.containsKey("result")) {
|
||||
throw invalidStatus(null);
|
||||
}
|
||||
String runtimeState = (String) state;
|
||||
if (!"settled".equals(runtimeState)) {
|
||||
ToolExecutionRecord latest = executionRepository
|
||||
.findByExecutionCallId(unknown.getExecutionCallId());
|
||||
ToolExecutionRecord current = latest == null ? unknown : latest;
|
||||
return current.getState() == ToolExecutionRecord.State.UNKNOWN
|
||||
? new ExecutionReconciliation(current,
|
||||
ExecutionReconciliation.Outcome.UNRESOLVED,
|
||||
runtimeState)
|
||||
: notUnknown(current, runtimeState);
|
||||
}
|
||||
Map<String, Object> result;
|
||||
try {
|
||||
result = immutableMap(status.get("result"), "status result");
|
||||
// Validate before writing, so an invalid result is reported as
|
||||
// the Runtime's fault rather than as a repository failure. A
|
||||
// not_started status is accepted only as the Runtime's own
|
||||
// terminal answer; the Broker never derives it.
|
||||
unknown.resolveUnsettled(result, clock.instant());
|
||||
} catch (IllegalArgumentException | RuntimeBrokerException exception) {
|
||||
throw invalidStatus(exception);
|
||||
}
|
||||
ToolExecutionRecord current = unknown;
|
||||
for (int attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt++) {
|
||||
if (current == null) {
|
||||
throw notFound("runtime_execution_not_found",
|
||||
"Runtime execution was not found");
|
||||
}
|
||||
if (!current.needsReconciliation()
|
||||
|| !takeover && current.getState() != ToolExecutionRecord.State.UNKNOWN) {
|
||||
return notUnknown(current, runtimeState);
|
||||
}
|
||||
if (takeover) {
|
||||
requireOpen(closed);
|
||||
}
|
||||
if (!current.sameIdentity(unknown)) {
|
||||
throw conflict("runtime_execution_conflict", "Execution identity changed while reconciling");
|
||||
}
|
||||
// A racing cancel advances the version; re-read and retry.
|
||||
ToolExecutionRecord resolved = takeover
|
||||
? executionRepository.resolveUnsettled(current, result, clock.instant())
|
||||
: executionRepository.resolveUnknown(current, result, clock.instant());
|
||||
if (resolved != null) {
|
||||
executionState.settleUnstartedBackgroundSiblings(
|
||||
unknown.getExecutionCallId(), result);
|
||||
return new ExecutionReconciliation(resolved,
|
||||
ExecutionReconciliation.Outcome.RESOLVED,
|
||||
runtimeState);
|
||||
}
|
||||
current = executionRepository.findByExecutionCallId(
|
||||
unknown.getExecutionCallId());
|
||||
}
|
||||
// The record is still UNKNOWN, so the next poll can try again.
|
||||
throw unavailable("runtime_execution_reconcile_failed",
|
||||
"Runtime execution changed while reconciling");
|
||||
}
|
||||
|
||||
private static RuntimeBrokerException invalidStatus(Throwable cause) {
|
||||
return new RuntimeBrokerException(502,
|
||||
"runtime_execution_status_invalid",
|
||||
"Runtime execution lookup returned an invalid status", false,
|
||||
cause);
|
||||
}
|
||||
}
|
||||
+392
@@ -0,0 +1,392 @@
|
||||
package com.alibaba.qwen.code.runtimebroker.service;
|
||||
|
||||
import com.alibaba.qwen.code.runtimebroker.BrokerValues;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSessionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSessionRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeTransport;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.service.RuntimeBindingRegistry.SessionContext;
|
||||
import com.alibaba.qwen.code.runtimebroker.transport.ManagedHoldFamily;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.failed;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.invalid;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.mapFailure;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.notFound;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.referenceString;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.requireOpen;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.safeStage;
|
||||
|
||||
final class RuntimeExecutionRecovery {
|
||||
private final RuntimeBindingRegistry bindingRegistry;
|
||||
private final RuntimeExecutionState executionState;
|
||||
private final ToolExecutionRepository executionRepository;
|
||||
private final RuntimeSessionRepository sessionRepository;
|
||||
private final RuntimeTransport transport;
|
||||
private final RuntimeExecutionReconciler executionReconciler;
|
||||
private final AtomicBoolean closed;
|
||||
|
||||
RuntimeExecutionRecovery(RuntimeBindingRegistry bindingRegistry,
|
||||
RuntimeExecutionState executionState,
|
||||
ToolExecutionRepository executionRepository,
|
||||
RuntimeSessionRepository sessionRepository,
|
||||
RuntimeTransport transport,
|
||||
RuntimeExecutionReconciler executionReconciler,
|
||||
AtomicBoolean closed) {
|
||||
this.bindingRegistry = bindingRegistry;
|
||||
this.executionState = executionState;
|
||||
this.executionRepository = executionRepository;
|
||||
this.sessionRepository = sessionRepository;
|
||||
this.transport = transport;
|
||||
this.executionReconciler = executionReconciler;
|
||||
this.closed = closed;
|
||||
}
|
||||
|
||||
/**
|
||||
* Asks the physical owner of a background process for status. An
|
||||
* exited answer settles the process row with that evidence; anything
|
||||
* else keeps the row active and its hold, exactly the wedge semantics
|
||||
* — a status that cannot be proven never becomes a claimed end.
|
||||
*/
|
||||
CompletionStage<ToolExecutionRecord> observeBackgroundProcess(
|
||||
String harnessSessionId, String runtimeSessionId,
|
||||
String executionCallId) {
|
||||
requireOpen(closed);
|
||||
String invocationId = BrokerValues.requireId(executionCallId,
|
||||
"executionCallId");
|
||||
return bindingRegistry.requireReadySession(harnessSessionId, runtimeSessionId)
|
||||
.thenCompose(context -> {
|
||||
ToolExecutionRecord invocation = executionState.requireExecution(context, invocationId);
|
||||
ToolExecutionRecord process = executionState.requireExecution(context, invocationId + ":process");
|
||||
return observeProcessRow(context, invocation, process);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* H4f: the public task cancel's stop of one background Shell or
|
||||
* Monitor watch, named by the call that started it inside the Harness
|
||||
* Session that owns the task. It runs the drain sweep's status → stop
|
||||
* → status for that one hold row, so only the worker's exit evidence
|
||||
* settles the row, and the worker answers a stop only after the
|
||||
* stop-marked finalize that settles the task's record has returned.
|
||||
* It runs unfenced, like a client Monitor control: the hold row itself
|
||||
* keeps the Runtime Session from releasing while the process may run,
|
||||
* so a racing close's drain is never refused busy because of it.
|
||||
*/
|
||||
CompletionStage<ToolExecutionRecord> stopBackgroundProcess(
|
||||
String tenantId, String harnessSessionId,
|
||||
String executionCallId) {
|
||||
requireOpen(closed);
|
||||
String invocationId = BrokerValues.requireId(executionCallId,
|
||||
"executionCallId");
|
||||
ToolExecutionRecord row = executionRepository.findByExecutionCallId(
|
||||
invocationId + ":process");
|
||||
if (row == null || !row.getHarnessSessionId().equals(harnessSessionId)) {
|
||||
return failed(notFound("runtime_execution_not_found",
|
||||
"The background process was not found"));
|
||||
}
|
||||
return bindingRegistry.requireReadySession(harnessSessionId, row.getRuntimeSessionId())
|
||||
.thenCompose(context -> {
|
||||
if (!context.session().getScope().getTenantId()
|
||||
.equals(tenantId)) {
|
||||
throw notFound("runtime_execution_not_found",
|
||||
"The background process was not found");
|
||||
}
|
||||
ToolExecutionRecord invocation = executionState.requireExecution(context,
|
||||
invocationId);
|
||||
ToolExecutionRecord process = executionState.requireExecution(context,
|
||||
row.getExecutionCallId());
|
||||
return observeScannedProcessRow(context, invocation,
|
||||
process)
|
||||
.thenCompose(current -> current.isTerminal()
|
||||
? CompletableFuture.completedFuture(
|
||||
current)
|
||||
: controlProcessRow(context, invocation,
|
||||
current, stopKind(current),
|
||||
false))
|
||||
.thenCompose(current -> current.isTerminal()
|
||||
? CompletableFuture.completedFuture(
|
||||
current)
|
||||
: observeScannedProcessRow(context,
|
||||
invocation, current));
|
||||
});
|
||||
}
|
||||
|
||||
CompletionStage<SessionContext> scanExecutions(SessionContext context,
|
||||
RuntimeSessionRecord session, String afterExecutionCallId) {
|
||||
requireOpen(closed);
|
||||
List<ToolExecutionRecord> batch = executionRepository.findUnsettled(
|
||||
session, afterExecutionCallId, 100);
|
||||
CompletionStage<Void> scanned = CompletableFuture.completedFuture(null);
|
||||
for (ToolExecutionRecord candidate : batch) {
|
||||
// A timed-out lookup completes on the JVM's timeout thread;
|
||||
// the next repository read must not block that thread.
|
||||
scanned = scanned.thenComposeAsync(ignored -> {
|
||||
requireOpen(closed);
|
||||
ToolExecutionRecord current = executionState.requireExecution(context, candidate.getExecutionCallId());
|
||||
return current.needsReconciliation()
|
||||
? executionReconciler.reconcileInContext(context, current, true).thenApply(result -> null)
|
||||
: CompletableFuture.completedFuture(null);
|
||||
});
|
||||
}
|
||||
// Yield between pages even when every status completes synchronously.
|
||||
return scanned.thenComposeAsync(ignored -> {
|
||||
if (batch.size() < 100) {
|
||||
observeScannedBackgroundProcesses(context);
|
||||
return CompletableFuture.completedFuture(context);
|
||||
}
|
||||
return scanExecutions(context, session,
|
||||
batch.get(batch.size() - 1).getExecutionCallId());
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* A background `:process` row stays PREPARED for the process's life, so
|
||||
* the reconcile filter above never reaches it — after the last page,
|
||||
* observe each live one from durable evidence: a provable exit settles
|
||||
* the row here, and anything else keeps it holding. Every observation
|
||||
* is fire-and-forget (an acquire never waits on it) and per-row
|
||||
* isolated (a store fault or a broken control must never fail an
|
||||
* acquire); the release sweep or a later scan asks again. The release
|
||||
* sweep's per-row prelude in {@code settleUnprovenBackgroundRows} is
|
||||
* the same shape, kept separate because it owns the in-memory index
|
||||
* and the status→terminate→status follow-up.
|
||||
*/
|
||||
private void observeScannedBackgroundProcesses(SessionContext context) {
|
||||
List<ToolExecutionRecord> rows;
|
||||
try {
|
||||
rows = durableBackgroundProcessRows(context);
|
||||
} catch (RuntimeException listing) {
|
||||
return;
|
||||
}
|
||||
for (ToolExecutionRecord row : rows) {
|
||||
if (row.isTerminal()) {
|
||||
continue;
|
||||
}
|
||||
ToolExecutionRecord invocation;
|
||||
try {
|
||||
invocation = executionState.requireExecution(context, referenceString(
|
||||
row.getReference(), "processOf"));
|
||||
} catch (RuntimeException missing) {
|
||||
continue;
|
||||
}
|
||||
ToolExecutionRecord parent = invocation;
|
||||
CompletableFuture.completedFuture(null)
|
||||
.thenComposeAsync(ignored -> observeScannedProcessRow(
|
||||
context, parent, row))
|
||||
.exceptionally(failure -> null);
|
||||
}
|
||||
}
|
||||
|
||||
private CompletionStage<ToolExecutionRecord> observeProcessRow(
|
||||
SessionContext context, ToolExecutionRecord invocation,
|
||||
ToolExecutionRecord process) {
|
||||
return controlProcessRow(context, invocation, process,
|
||||
statusKind(process), true);
|
||||
}
|
||||
|
||||
/**
|
||||
* The scan's read-only maintenance ask runs unfenced: it must not hold
|
||||
* `activeControls` past the sweep's own proof, or a release that settled
|
||||
* every row would still refuse `runtime_session_busy` solely for the
|
||||
* outstanding status answer (#13830 review).
|
||||
*/
|
||||
private CompletionStage<ToolExecutionRecord> observeScannedProcessRow(
|
||||
SessionContext context, ToolExecutionRecord invocation,
|
||||
ToolExecutionRecord process) {
|
||||
return controlProcessRow(context, invocation, process,
|
||||
statusKind(process), false);
|
||||
}
|
||||
|
||||
/**
|
||||
* A hold row's family names its maintenance kinds: the background Shell
|
||||
* rides {@code shell-status}/{@code shell-terminate}, the Monitor watch
|
||||
* {@code monitor-status}/{@code monitor-stop}. The sweep and the scan
|
||||
* only ever touch hold rows, so the family is always present.
|
||||
*/
|
||||
private static String statusKind(ToolExecutionRecord process) {
|
||||
ManagedHoldFamily family = ManagedHoldFamily.of(process);
|
||||
if (family == null) {
|
||||
throw invalid("runtime_control_operation_invalid",
|
||||
"A hold row's dispatch mode is required");
|
||||
}
|
||||
return family.statusKind();
|
||||
}
|
||||
|
||||
private static String stopKind(ToolExecutionRecord process) {
|
||||
ManagedHoldFamily family = ManagedHoldFamily.of(process);
|
||||
if (family == null) {
|
||||
throw invalid("runtime_control_operation_invalid",
|
||||
"A hold row's dispatch mode is required");
|
||||
}
|
||||
return family.stopKind();
|
||||
}
|
||||
|
||||
/**
|
||||
* One maintenance operation against a background process's physical
|
||||
* owner. An `exited` answer settles the row with that evidence —
|
||||
* terminal for both the status read and the stop the release asks
|
||||
* for, whose answer carries the same receipt view; anything else
|
||||
* keeps the row active and holding, exactly the wedge semantics.
|
||||
*/
|
||||
private CompletionStage<ToolExecutionRecord> controlProcessRow(
|
||||
SessionContext context, ToolExecutionRecord invocation,
|
||||
ToolExecutionRecord process, String kind) {
|
||||
return controlProcessRow(context, invocation, process, kind, true);
|
||||
}
|
||||
|
||||
private CompletionStage<ToolExecutionRecord> controlProcessRow(
|
||||
SessionContext context, ToolExecutionRecord invocation,
|
||||
ToolExecutionRecord process, String kind, boolean fenced) {
|
||||
if (process.isTerminal()) {
|
||||
return CompletableFuture.completedFuture(process);
|
||||
}
|
||||
Map<String, Object> operation = new LinkedHashMap<>();
|
||||
operation.put("kind", kind);
|
||||
operation.put("sessionKey", Map.of(
|
||||
"tenantId", context.session().getScope().getTenantId(),
|
||||
"workspaceId", context.session().getScope().getWorkspaceId(),
|
||||
"sessionId", context.session().getHarnessSessionId()));
|
||||
operation.put("operationId", process.getExecutionCallId());
|
||||
operation.put("targetOperationId",
|
||||
referenceString(invocation.getReference(), "callId"));
|
||||
if (fenced) {
|
||||
context.beginControl();
|
||||
}
|
||||
return mapFailure(safeStage(() -> {
|
||||
bindingRegistry.requireUsableLease(context);
|
||||
return transport.control(context.lease(), context.session(),
|
||||
operation);
|
||||
}), "runtime_shell_status_failed", "Shell status lookup failed")
|
||||
.thenApply(view -> {
|
||||
if (!(view instanceof Map<?, ?> answer)
|
||||
|| !"exited".equals(answer.get("state"))) {
|
||||
return process;
|
||||
}
|
||||
executionState.settleBackgroundProcess(process, (Map<String, Object>) view);
|
||||
ToolExecutionRecord current = executionRepository
|
||||
.findByExecutionCallId(process.getExecutionCallId());
|
||||
return current == null ? process : current;
|
||||
})
|
||||
.whenComplete((ignored, error) -> {
|
||||
if (fenced) {
|
||||
context.endControl();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Before a release computes its busy check, every hold row this Broker
|
||||
* admitted for the Session asks its physical owner once: a proven exit
|
||||
* settles the row here, and anything else simply keeps it — the busy or
|
||||
* held answer behind it then stands on what is actually still running.
|
||||
* A failed lookup moves on, because busy remains the accurate answer
|
||||
* for what could not be proven. Only a drain issues each live row's
|
||||
* stop; a routine release observes and holds.
|
||||
*/
|
||||
CompletionStage<Void> settleUnprovenBackgroundRows(
|
||||
SessionContext context, boolean drain) {
|
||||
CompletionStage<Void> chain = CompletableFuture.completedFuture(null);
|
||||
List<String> processIds;
|
||||
List<ToolExecutionRecord> durable = durableBackgroundProcessRows(
|
||||
context);
|
||||
context.lock();
|
||||
try {
|
||||
for (ToolExecutionRecord row : durable) {
|
||||
context.backgroundProcesses().add(row.getExecutionCallId());
|
||||
}
|
||||
processIds = new ArrayList<>(context.backgroundProcesses());
|
||||
} finally {
|
||||
context.unlock();
|
||||
}
|
||||
for (String processId : processIds) {
|
||||
ToolExecutionRecord row = executionRepository
|
||||
.findByExecutionCallId(processId);
|
||||
if (row == null || row.isTerminal()) {
|
||||
context.lock();
|
||||
try {
|
||||
context.backgroundProcesses().remove(processId);
|
||||
} finally {
|
||||
context.unlock();
|
||||
}
|
||||
continue;
|
||||
}
|
||||
ToolExecutionRecord invocation;
|
||||
try {
|
||||
invocation = executionState.requireExecution(context,
|
||||
referenceString(row.getReference(), "processOf"));
|
||||
} catch (RuntimeException notOurs) {
|
||||
continue;
|
||||
}
|
||||
// Status first. A drain gives a row the owner still counts
|
||||
// running one stop it must prove — an unproven stop keeps the
|
||||
// hold, and the busy answer behind it stays accurate. A routine
|
||||
// release stops nothing: the run holds its Runtime for as long
|
||||
// as it needs it, and the release answers held instead.
|
||||
chain = chain.thenCompose(ignored -> observeProcessRow(
|
||||
context, invocation, row)
|
||||
.thenCompose(current -> !drain || current.isTerminal()
|
||||
? CompletableFuture.completedFuture(current)
|
||||
: controlProcessRow(context, invocation, current,
|
||||
stopKind(current)))
|
||||
.thenCompose(current -> !drain || current.isTerminal()
|
||||
? CompletableFuture.completedFuture(current)
|
||||
: observeProcessRow(context, invocation, current))
|
||||
.thenApply(current -> {
|
||||
if (current.isTerminal()) {
|
||||
context.lock();
|
||||
try {
|
||||
context.backgroundProcesses().remove(processId);
|
||||
} finally {
|
||||
context.unlock();
|
||||
}
|
||||
}
|
||||
return (Void) null;
|
||||
})
|
||||
.exceptionally(failure -> null));
|
||||
}
|
||||
return chain;
|
||||
}
|
||||
|
||||
/**
|
||||
* The durable ledger knows every background process this Session was
|
||||
* ever admitted for; the in-memory index only knows the ones this
|
||||
* Broker process admitted itself. Without the backfill a fresh context
|
||||
* — every release path after a Broker restart — sweeps nothing, and
|
||||
* the non-terminal `:process` row answers busy forever. The scan stays
|
||||
* inside the repository's Session-and-generation fence, so a stale
|
||||
* cross-generation row never enters the sweep or the busy-check
|
||||
* exclusion set.
|
||||
*/
|
||||
private List<ToolExecutionRecord> durableBackgroundProcessRows(
|
||||
SessionContext context) {
|
||||
RuntimeSessionRecord session = sessionRepository.findById(
|
||||
context.session().getScope(),
|
||||
context.session().getRuntimeSessionId());
|
||||
if (session == null) {
|
||||
return List.of();
|
||||
}
|
||||
List<ToolExecutionRecord> rows = new ArrayList<>();
|
||||
String after = null;
|
||||
for (;;) {
|
||||
List<ToolExecutionRecord> batch = executionRepository
|
||||
.findBackgroundProcesses(session, after, 100);
|
||||
if (batch.isEmpty()) {
|
||||
return rows;
|
||||
}
|
||||
rows.addAll(batch);
|
||||
if (batch.size() < 100) {
|
||||
return rows;
|
||||
}
|
||||
after = batch.get(batch.size() - 1).getExecutionCallId();
|
||||
}
|
||||
}
|
||||
}
|
||||
+295
@@ -0,0 +1,295 @@
|
||||
package com.alibaba.qwen.code.runtimebroker.service;
|
||||
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSessionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSessionRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.service.RuntimeBindingRegistry.SessionContext;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.CompletionException;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.MAX_CAS_ATTEMPTS;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.conflict;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.notFound;
|
||||
import static com.alibaba.qwen.code.runtimebroker.service.RuntimeBrokerSupport.unwrap;
|
||||
|
||||
final class RuntimeExecutionState {
|
||||
private final RuntimeBindingRepository bindingRepository;
|
||||
private final RuntimeSessionRepository sessionRepository;
|
||||
private final ToolExecutionRepository executionRepository;
|
||||
private final String brokerOwnerId;
|
||||
private final Duration dispatchLeaseDuration;
|
||||
private final Clock clock;
|
||||
|
||||
RuntimeExecutionState(RuntimeBindingRepository bindingRepository,
|
||||
RuntimeSessionRepository sessionRepository,
|
||||
ToolExecutionRepository executionRepository,
|
||||
String brokerOwnerId,
|
||||
Duration dispatchLeaseDuration,
|
||||
Clock clock) {
|
||||
this.bindingRepository = bindingRepository;
|
||||
this.sessionRepository = sessionRepository;
|
||||
this.executionRepository = executionRepository;
|
||||
this.brokerOwnerId = brokerOwnerId;
|
||||
this.dispatchLeaseDuration = dispatchLeaseDuration;
|
||||
this.clock = clock;
|
||||
}
|
||||
|
||||
void settleExecution(ToolExecutionRecord original, Map<String, Object> result) {
|
||||
String executionCallId = original.getExecutionCallId();
|
||||
long dispatchGeneration = original.getDispatchGeneration();
|
||||
ToolExecutionRecord current = executionRepository
|
||||
.findByExecutionCallId(executionCallId);
|
||||
for (int attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt++) {
|
||||
if (current == null || current.isTerminal()
|
||||
|| current.getState()
|
||||
== ToolExecutionRecord.State.UNKNOWN
|
||||
|| !ownsDispatch(current, dispatchGeneration)
|
||||
|| !current.sameIdentity(original) || !current.sameAuthorization(original)) {
|
||||
return;
|
||||
}
|
||||
ToolExecutionRecord replacement = current.withResult(result,
|
||||
current.getLastSequence(), clock.instant());
|
||||
ToolExecutionRecord updated = executionRepository.compareAndSet(
|
||||
current, replacement, brokerOwnerId,
|
||||
dispatchGeneration);
|
||||
if (updated != null) {
|
||||
return;
|
||||
}
|
||||
current = executionRepository.findByExecutionCallId(
|
||||
executionCallId);
|
||||
}
|
||||
throw conflict("runtime_execution_state_conflict",
|
||||
"Runtime execution changed while settling");
|
||||
}
|
||||
|
||||
void markUnknown(ToolExecutionRecord original) {
|
||||
String executionCallId = original.getExecutionCallId();
|
||||
long dispatchGeneration = original.getDispatchGeneration();
|
||||
ToolExecutionRecord current = executionRepository
|
||||
.findByExecutionCallId(executionCallId);
|
||||
for (int attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt++) {
|
||||
if (current == null || current.isTerminal()
|
||||
|| current.getState()
|
||||
== ToolExecutionRecord.State.UNKNOWN
|
||||
|| !ownsDispatch(current, dispatchGeneration)
|
||||
|| !current.sameIdentity(original) || !current.sameAuthorization(original)) {
|
||||
return;
|
||||
}
|
||||
ToolExecutionRecord updated = executionRepository.compareAndSet(
|
||||
current, current.withUnknown(), brokerOwnerId,
|
||||
dispatchGeneration);
|
||||
if (updated != null) {
|
||||
return;
|
||||
}
|
||||
current = executionRepository.findByExecutionCallId(
|
||||
executionCallId);
|
||||
}
|
||||
fenceLapsedClaim(current);
|
||||
}
|
||||
|
||||
/**
|
||||
* Once a claim has lapsed, no compare-and-set can settle the record or
|
||||
* mark it UNKNOWN; the takeover fence in {@code claimDispatch} can still
|
||||
* mark it UNKNOWN. {@code claimDispatch} judges the lease by the
|
||||
* repository's clock and never writes over a live claim, whoever holds
|
||||
* it. A DISPATCHING record is left alone, since claiming it would hold a
|
||||
* dispatch nothing here is running.
|
||||
*/
|
||||
ToolExecutionRecord fenceLapsedClaim(ToolExecutionRecord current) {
|
||||
if (current == null
|
||||
|| (current.getState() != ToolExecutionRecord.State.EXECUTING
|
||||
&& current.getState()
|
||||
!= ToolExecutionRecord.State
|
||||
.CANCEL_REQUESTED)) {
|
||||
return current;
|
||||
}
|
||||
// Only this broker's own live claim comes back; a settled or fenced
|
||||
// record, or another broker's live claim, yields null and is read
|
||||
// again.
|
||||
ToolExecutionRecord claimed = executionRepository.claimDispatch(
|
||||
current.getExecutionCallId(), brokerOwnerId,
|
||||
dispatchLeaseDuration);
|
||||
return claimed != null ? claimed : executionRepository
|
||||
.findByExecutionCallId(current.getExecutionCallId());
|
||||
}
|
||||
|
||||
ToolExecutionRecord requireOwnedExecution(
|
||||
String harnessSessionId, String runtimeSessionId,
|
||||
String executionCallId) {
|
||||
return requireOwnership(harnessSessionId, runtimeSessionId,
|
||||
executionCallId).record();
|
||||
}
|
||||
|
||||
/**
|
||||
* A terminal record comes back with the saved binding and Session rows
|
||||
* that prove its ownership; any other record comes back without them.
|
||||
*/
|
||||
OwnedExecution requireOwnership(
|
||||
String harnessSessionId, String runtimeSessionId,
|
||||
String executionCallId) {
|
||||
ToolExecutionRecord record = executionRepository
|
||||
.findByExecutionCallId(executionCallId);
|
||||
if (record == null) {
|
||||
throw notFound("runtime_execution_not_found",
|
||||
"Runtime execution was not found");
|
||||
}
|
||||
if (!record.getHarnessSessionId().equals(harnessSessionId)
|
||||
|| !record.getRuntimeSessionId().equals(runtimeSessionId)) {
|
||||
throw conflict("runtime_execution_conflict",
|
||||
"Runtime execution belongs to another Session");
|
||||
}
|
||||
RuntimeBindingRecord binding = null;
|
||||
RuntimeSessionRecord session = null;
|
||||
if (record.isTerminal()) {
|
||||
binding = bindingRepository.findById(record.getBindingId());
|
||||
session = binding == null ? null : sessionRepository.findById(
|
||||
binding.getRequest().getScope(), runtimeSessionId);
|
||||
if (binding == null || binding.getGeneration() != record.getRuntimeGeneration()
|
||||
|| session == null || !session.getBindingId().equals(record.getBindingId())
|
||||
|| session.getRuntimeGeneration() != record.getRuntimeGeneration()
|
||||
|| !session.getSession().getHarnessSessionId().equals(harnessSessionId)) {
|
||||
throw conflict("runtime_execution_conflict", "Saved execution ownership differs");
|
||||
}
|
||||
}
|
||||
return new OwnedExecution(record, binding, session);
|
||||
}
|
||||
|
||||
/**
|
||||
* An execution whose outcome is already UNKNOWN keeps its own answer
|
||||
* when the generation that could tell more is no longer live: nothing
|
||||
* can change the record anymore, so a closed admission reads as the
|
||||
* record's UNKNOWN instead of replacing it. Any other failure, and any
|
||||
* record the caller does not own, keeps the original error.
|
||||
*/
|
||||
CompletionStage<ToolExecutionRecord> unknownWhenAdmissionClosed(
|
||||
CompletionStage<ToolExecutionRecord> stage,
|
||||
String harnessSessionId, String runtimeSessionId,
|
||||
String executionCallId) {
|
||||
return stage.handle((record, error) -> {
|
||||
if (error == null) {
|
||||
return record;
|
||||
}
|
||||
Throwable cause = unwrap(error);
|
||||
if (cause instanceof RuntimeBrokerException exception
|
||||
&& "runtime_admission_closed".equals(exception.getCode())) {
|
||||
ToolExecutionRecord stored = executionRepository
|
||||
.findByExecutionCallId(executionCallId);
|
||||
if (stored != null
|
||||
&& stored.getState() == ToolExecutionRecord.State.UNKNOWN
|
||||
&& stored.getHarnessSessionId().equals(harnessSessionId)
|
||||
&& stored.getRuntimeSessionId().equals(runtimeSessionId)) {
|
||||
return stored;
|
||||
}
|
||||
}
|
||||
throw new CompletionException(cause);
|
||||
});
|
||||
}
|
||||
|
||||
ToolExecutionRecord requireExecution(SessionContext context,
|
||||
String executionCallId) {
|
||||
ToolExecutionRecord record = executionRepository
|
||||
.findByExecutionCallId(executionCallId);
|
||||
if (record == null) {
|
||||
throw notFound("runtime_execution_not_found",
|
||||
"Runtime execution was not found");
|
||||
}
|
||||
if (!record.getHarnessSessionId().equals(
|
||||
context.session().getHarnessSessionId())
|
||||
|| !record.getRuntimeSessionId().equals(
|
||||
context.session().getRuntimeSessionId())
|
||||
|| !record.getBindingId().equals(
|
||||
context.binding().getBindingId())
|
||||
|| record.getRuntimeGeneration()
|
||||
!= context.binding().getGeneration()) {
|
||||
throw conflict("runtime_execution_conflict",
|
||||
"Runtime execution belongs to another Session or "
|
||||
+ "Runtime generation");
|
||||
}
|
||||
return record;
|
||||
}
|
||||
|
||||
/**
|
||||
* The Runtime proved no process ever started: without this sibling
|
||||
* settle the `:process` row would count as alive forever, and no
|
||||
* shell release could ever pass busy again (round 6). Every arm that
|
||||
* settles the invocation with that proof — the dispatch poll, and the
|
||||
* reconcile resume — calls it, so no path back to SETTLED keeps the
|
||||
* sibling PREPARED.
|
||||
*/
|
||||
void settleUnstartedBackgroundSiblings(String executionCallId,
|
||||
Map<String, Object> result) {
|
||||
if (!"not_started".equals(result.get("executionStatus"))) {
|
||||
return;
|
||||
}
|
||||
ToolExecutionRecord process = executionRepository
|
||||
.findByExecutionCallId(executionCallId + ":process");
|
||||
if (process == null || process.isTerminal()) {
|
||||
return;
|
||||
}
|
||||
settleUnstartedBackgroundProcess(process, result);
|
||||
}
|
||||
|
||||
void settleUnstartedBackgroundProcess(ToolExecutionRecord process,
|
||||
Map<String, Object> result) {
|
||||
Map<String, Object> settled = new LinkedHashMap<>();
|
||||
settled.put("state", "not_started");
|
||||
settled.put("executionStatus", "not_started");
|
||||
settled.put("evidence", result.get("error"));
|
||||
for (int attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt++) {
|
||||
ToolExecutionRecord current = executionRepository
|
||||
.findByExecutionCallId(process.getExecutionCallId());
|
||||
if (current == null || current.isTerminal()) {
|
||||
return;
|
||||
}
|
||||
if (executionRepository.settlePrepared(current, settled,
|
||||
clock.instant()) != null) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
throw conflict("runtime_execution_state_conflict",
|
||||
"Background process could not settle with its own generation");
|
||||
}
|
||||
|
||||
void settleBackgroundProcess(ToolExecutionRecord process,
|
||||
Map<String, Object> view) {
|
||||
Map<String, Object> result = new LinkedHashMap<>();
|
||||
result.put("state", "exited");
|
||||
result.put("evidence", view.get("evidence"));
|
||||
Object exitCode = view.get("evidence") instanceof Map<?, ?> evidence
|
||||
? evidence.get("exitCode") : null;
|
||||
result.put("executionStatus", Integer.valueOf(0).equals(exitCode)
|
||||
? "success" : "error");
|
||||
for (int attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt++) {
|
||||
ToolExecutionRecord current = executionRepository
|
||||
.findByExecutionCallId(process.getExecutionCallId());
|
||||
if (current == null || current.isTerminal()) {
|
||||
return;
|
||||
}
|
||||
if (executionRepository.settlePrepared(current, result,
|
||||
clock.instant()) != null) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
throw conflict("runtime_execution_state_conflict",
|
||||
"Background process could not settle with its own generation");
|
||||
}
|
||||
|
||||
boolean ownsDispatch(ToolExecutionRecord current,
|
||||
long dispatchGeneration) {
|
||||
return dispatchGeneration == current.getDispatchGeneration()
|
||||
&& brokerOwnerId.equals(current.getDispatchOwner());
|
||||
}
|
||||
|
||||
record OwnedExecution(ToolExecutionRecord record,
|
||||
RuntimeBindingRecord binding, RuntimeSessionRecord session) {
|
||||
}
|
||||
}
|
||||
+4
-18
@@ -1383,16 +1383,8 @@ public class RuntimeBrokerHttpServerTest {
|
||||
csiSessions, csiExecutions, "broker", Duration.ofMinutes(1),
|
||||
Duration.ofMinutes(1));
|
||||
var csiClient = HttpClient.newHttpClient()) {
|
||||
var contextClass = Class.forName(
|
||||
RuntimeBrokerService.class.getName() + "$SessionContext");
|
||||
var contextConstructor = contextClass.getDeclaredConstructor(
|
||||
RuntimeSession.class, RuntimeBindingRecord.class, RuntimeLease.class);
|
||||
contextConstructor.setAccessible(true);
|
||||
var installed = RuntimeBrokerService.class.getDeclaredField("sessions");
|
||||
installed.setAccessible(true);
|
||||
((Map<String, CompletableFuture<?>>) installed.get(csiService)).put(csiSession,
|
||||
CompletableFuture.completedFuture(contextConstructor.newInstance(
|
||||
csiRuntime, csiBinding, csiLease)));
|
||||
com.alibaba.qwen.code.runtimebroker.service.BrokerServiceTestAccess.installSession(
|
||||
csiService, csiRuntime, csiBinding, csiLease);
|
||||
var csiServer = new RuntimeBrokerHttpServer(
|
||||
new InetSocketAddress("127.0.0.1", 0), "secret", csiService);
|
||||
csiServer.start();
|
||||
@@ -1495,14 +1487,8 @@ public class RuntimeBrokerHttpServerTest {
|
||||
sessions, new InMemoryToolExecutionRepository(Clock.systemUTC()),
|
||||
"broker", Duration.ofMinutes(1), Duration.ofMinutes(1));
|
||||
// Install the ready Runtime Session the way a completed acquire leaves it.
|
||||
var contextClass = Class.forName(RuntimeBrokerService.class.getName() + "$SessionContext");
|
||||
var constructor = contextClass.getDeclaredConstructor(RuntimeSession.class, RuntimeBindingRecord.class,
|
||||
RuntimeLease.class);
|
||||
constructor.setAccessible(true);
|
||||
var field = RuntimeBrokerService.class.getDeclaredField("sessions");
|
||||
field.setAccessible(true);
|
||||
((Map<String, CompletableFuture<?>>) field.get(service)).put(session,
|
||||
CompletableFuture.completedFuture(constructor.newInstance(runtime, binding, lease)));
|
||||
com.alibaba.qwen.code.runtimebroker.service.BrokerServiceTestAccess.installSession(
|
||||
service, runtime, binding, lease);
|
||||
try (RuntimeBrokerHttpServer server = new RuntimeBrokerHttpServer(
|
||||
new InetSocketAddress("127.0.0.1", 0), "secret", service);
|
||||
HttpClient client = HttpClient.newHttpClient()) {
|
||||
|
||||
+17
@@ -1,15 +1,19 @@
|
||||
package com.alibaba.qwen.code.runtimebroker.service;
|
||||
|
||||
import com.alibaba.qwen.code.runtimebroker.HarnessSessionResolver;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeLease;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeProvisioner;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimePublicationVerifier;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSession;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSessionRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeTransport;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRepository;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
public final class BrokerServiceTestAccess {
|
||||
@@ -41,6 +45,19 @@ public final class BrokerServiceTestAccess {
|
||||
sessionRepository, executionRepository, brokerOwnerId, operationLeaseDuration,
|
||||
dispatchLeaseDuration, clock, executionIdSupplier);
|
||||
}
|
||||
|
||||
static RuntimeBindingRegistry registry(RuntimeBrokerService service) throws ReflectiveOperationException {
|
||||
var field = RuntimeBrokerService.class.getDeclaredField("bindingRegistry");
|
||||
field.setAccessible(true);
|
||||
return (RuntimeBindingRegistry) field.get(service);
|
||||
}
|
||||
|
||||
public static void installSession(RuntimeBrokerService service, RuntimeSession session,
|
||||
RuntimeBindingRecord binding, RuntimeLease lease) throws ReflectiveOperationException {
|
||||
registry(service).registerSession(session.getRuntimeSessionId(), CompletableFuture.completedFuture(
|
||||
new RuntimeBindingRegistry.SessionContext(session, binding, lease)));
|
||||
}
|
||||
|
||||
public static long pollDelay(int attempt) { return RuntimeBrokerService.v3PollDelayMillis(attempt); }
|
||||
public static final java.time.Duration UNKNOWN_LOOKUP_COOLDOWN = ExecutionObservationCooldown.WINDOW;
|
||||
}
|
||||
+173
@@ -0,0 +1,173 @@
|
||||
package com.alibaba.qwen.code.runtimebroker.service;
|
||||
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeLease;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeProvisionRequest;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeProvisioner;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeScope;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.service.RuntimeBindingRegistry.BindingContext;
|
||||
import com.alibaba.qwen.code.runtimebroker.service.RuntimeBindingRegistry.SessionContext;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.net.URI;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertSame;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.inOrder;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.verifyNoInteractions;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class RuntimeBindingRegistryTest {
|
||||
private static final Instant NOW = Instant.parse("2026-10-31T00:00:00Z");
|
||||
private static final Duration LEASE_DURATION = Duration.ofSeconds(30);
|
||||
private static final RuntimeProvisionRequest REQUEST = new RuntimeProvisionRequest(
|
||||
new RuntimeScope("tenant", "workspace", "generation", "/cwd", "digest", "session"), "harness");
|
||||
private static final RuntimeLease LEASE = new RuntimeLease("runtime", URI.create("http://127.0.0.1:1"),
|
||||
"token", "lease", 1);
|
||||
|
||||
private final RuntimeBindingRepository bindings = mock(RuntimeBindingRepository.class);
|
||||
private final RuntimeProvisioner provisioner = mock(RuntimeProvisioner.class);
|
||||
private final RuntimeBindingRegistry registry = new RuntimeBindingRegistry(bindings, provisioner,
|
||||
mock(com.alibaba.qwen.code.runtimebroker.RuntimeSessionRepository.class),
|
||||
mock(com.alibaba.qwen.code.runtimebroker.HarnessSessionResolver.class), "broker", LEASE_DURATION, Clock.fixed(NOW, ZoneOffset.UTC));
|
||||
|
||||
@Test
|
||||
void routingRequiresTheExactLocallyAttestedGenerationAndLease() {
|
||||
RuntimeBindingRecord original = binding(7, RuntimeBindingRecord.State.READY, LEASE);
|
||||
assertNull(registry.matchingLiveBinding(original));
|
||||
registry.registerBinding(original, LEASE);
|
||||
assertSame(LEASE, registry.requireLiveBinding(original).lease());
|
||||
for (RuntimeLease changed : List.of(
|
||||
new RuntimeLease("other", LEASE.getEndpoint(), "token", "lease", 1),
|
||||
new RuntimeLease("runtime", URI.create("http://127.0.0.1:2"), "token", "lease", 1),
|
||||
new RuntimeLease("runtime", LEASE.getEndpoint(), "other", "lease", 1),
|
||||
new RuntimeLease("runtime", LEASE.getEndpoint(), "token", "other", 1),
|
||||
new RuntimeLease("runtime", LEASE.getEndpoint(), "token", "lease", 2))) {
|
||||
RuntimeBrokerException failure = assertThrows(RuntimeBrokerException.class,
|
||||
() -> registry.requireLiveBinding(binding(7, RuntimeBindingRecord.State.READY, changed)));
|
||||
assertEquals("runtime_reconciliation_required", failure.getCode());
|
||||
assertTrue(failure.isRetryable());
|
||||
}
|
||||
assertFalse(registry.isLiveBinding(binding(8, RuntimeBindingRecord.State.READY, LEASE)));
|
||||
assertFalse(registry.isLiveBinding(binding(7, RuntimeBindingRecord.State.READY, null)));
|
||||
registry.registerBinding(binding(8, RuntimeBindingRecord.State.READY, LEASE), LEASE);
|
||||
assertFalse(registry.isLiveBinding(original));
|
||||
verifyNoInteractions(bindings, provisioner);
|
||||
}
|
||||
|
||||
@Test
|
||||
void executionEligibilityUsesTheOriginalPersistedGeneration() {
|
||||
ToolExecutionRecord execution = mock(ToolExecutionRecord.class);
|
||||
when(execution.getBindingId()).thenReturn("binding");
|
||||
when(execution.getRuntimeGeneration()).thenReturn(7L);
|
||||
for (RuntimeBindingRecord.State state : RuntimeBindingRecord.State.values()) {
|
||||
when(bindings.findById("binding")).thenReturn(binding(7, state, LEASE));
|
||||
if (state == RuntimeBindingRecord.State.READY || state == RuntimeBindingRecord.State.DRAINING) {
|
||||
assertDoesNotThrow(() -> registry.requireAnswerableBinding(execution));
|
||||
} else {
|
||||
assertEvidenceUnavailable(execution);
|
||||
}
|
||||
}
|
||||
when(bindings.findById("binding")).thenReturn(binding(8, RuntimeBindingRecord.State.READY, LEASE));
|
||||
assertEvidenceUnavailable(execution);
|
||||
when(bindings.findById("binding")).thenReturn(null);
|
||||
assertEvidenceUnavailable(execution);
|
||||
verifyNoInteractions(provisioner);
|
||||
}
|
||||
|
||||
@Test
|
||||
void invalidationRevokesTheRouteBeforeClaimingAndReleasesItsExactClaimAfterAWriteFailure() {
|
||||
RuntimeBindingRecord original = binding(7, RuntimeBindingRecord.State.READY, LEASE);
|
||||
registry.registerBinding(original, LEASE);
|
||||
when(bindings.claimOperation("binding", "broker", LEASE_DURATION)).thenAnswer(call -> {
|
||||
assertFalse(registry.hasBinding("binding"));
|
||||
return original;
|
||||
});
|
||||
IllegalStateException outage = new IllegalStateException("repository unavailable");
|
||||
when(bindings.compareAndSet(eq(original), any())).thenAnswer(call -> {
|
||||
RuntimeBindingRecord lost = call.getArgument(1);
|
||||
assertEquals(RuntimeBindingRecord.State.LOST, lost.getState());
|
||||
assertEquals(original.getGeneration(), lost.getGeneration());
|
||||
assertSame(LEASE, lost.getLease());
|
||||
throw outage;
|
||||
});
|
||||
assertSame(outage, assertThrows(IllegalStateException.class, () -> registry.invalidateBinding(original)));
|
||||
assertFalse(registry.hasBinding("binding"));
|
||||
var order = inOrder(bindings);
|
||||
order.verify(bindings).claimOperation("binding", "broker", LEASE_DURATION);
|
||||
order.verify(bindings).compareAndSet(eq(original), any());
|
||||
order.verify(bindings).releaseOperation("binding", "broker", original.getOperationGeneration());
|
||||
order.verifyNoMoreInteractions();
|
||||
verifyNoInteractions(provisioner);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aLostClaimStillRevokesTheLocalRouteWithoutReleasingOtherResources() {
|
||||
RuntimeBindingRecord original = binding(7, RuntimeBindingRecord.State.READY, LEASE);
|
||||
registry.registerBinding(original, LEASE);
|
||||
registry.invalidateBinding(original);
|
||||
assertFalse(registry.hasBinding("binding"));
|
||||
var order = inOrder(bindings);
|
||||
order.verify(bindings).claimOperation("binding", "broker", LEASE_DURATION);
|
||||
order.verifyNoMoreInteractions();
|
||||
verifyNoInteractions(provisioner);
|
||||
}
|
||||
|
||||
@Test
|
||||
void lateOperationCleanupCannotRemoveReplacementReservations() {
|
||||
var firstBinding = new CompletableFuture<BindingContext>();
|
||||
var nextBinding = new CompletableFuture<BindingContext>();
|
||||
assertNull(registry.reserveBinding("binding", firstBinding));
|
||||
assertSame(firstBinding, registry.reserveBinding("binding", nextBinding));
|
||||
registry.finishBindingOperation("binding", firstBinding);
|
||||
assertNull(registry.reserveBinding("binding", nextBinding));
|
||||
registry.finishBindingOperation("binding", firstBinding);
|
||||
assertSame(nextBinding, registry.bindingOperation("binding"));
|
||||
|
||||
var firstSession = new CompletableFuture<SessionContext>();
|
||||
var nextSession = new CompletableFuture<SessionContext>();
|
||||
assertNull(registry.registerSession("session", firstSession));
|
||||
assertSame(firstSession, registry.registerSession("session", nextSession));
|
||||
registry.removeSession("session", firstSession);
|
||||
assertNull(registry.registerSession("session", nextSession));
|
||||
registry.removeSession("session", firstSession);
|
||||
assertSame(nextSession, registry.session("session"));
|
||||
|
||||
registry.cancelPendingOperations();
|
||||
assertTrue(nextBinding.isCancelled());
|
||||
assertTrue(nextSession.isCancelled());
|
||||
assertFalse(firstBinding.isCancelled());
|
||||
assertFalse(firstSession.isCancelled());
|
||||
verifyNoInteractions(bindings, provisioner);
|
||||
}
|
||||
|
||||
private void assertEvidenceUnavailable(ToolExecutionRecord execution) {
|
||||
RuntimeBrokerException failure = assertThrows(RuntimeBrokerException.class,
|
||||
() -> registry.requireAnswerableBinding(execution));
|
||||
assertEquals("runtime_execution_evidence_unavailable", failure.getCode());
|
||||
assertEquals(409, failure.getStatusCode());
|
||||
assertFalse(failure.isRetryable());
|
||||
}
|
||||
|
||||
private static RuntimeBindingRecord binding(long generation, RuntimeBindingRecord.State state, RuntimeLease lease) {
|
||||
return new RuntimeBindingRecord("binding", REQUEST, generation, state, lease,
|
||||
false, "broker", NOW.plus(LEASE_DURATION), 9, 1, NOW, NOW);
|
||||
}
|
||||
}
|
||||
+7
-17
@@ -104,13 +104,7 @@ class RuntimeBrokerServiceTest {
|
||||
var saved = fixture.sessionRepository.findOrCreate(new RuntimeSessionRecord(runtime, binding.getBindingId(),
|
||||
binding.getGeneration(), RuntimeSessionRecord.State.ACQUIRING, 0, START));
|
||||
fixture.sessionRepository.compareAndSet(saved, saved.withState(RuntimeSessionRecord.State.READY, START));
|
||||
var contextClass = Class.forName(RuntimeBrokerService.class.getName() + "$SessionContext");
|
||||
var constructor = contextClass.getDeclaredConstructor(RuntimeSession.class, RuntimeBindingRecord.class, RuntimeLease.class);
|
||||
constructor.setAccessible(true);
|
||||
var field = RuntimeBrokerService.class.getDeclaredField("sessions");
|
||||
field.setAccessible(true);
|
||||
((Map<String, CompletableFuture<?>>) field.get(fixture.service)).put(PROVIDER_SESSION,
|
||||
CompletableFuture.completedFuture(constructor.newInstance(runtime, binding, lease)));
|
||||
BrokerServiceTestAccess.installSession(fixture.service, runtime, binding, lease);
|
||||
var candidate = ToolExecutionRecord.prepared("execution", "idempotency", binding.getBindingId(),
|
||||
binding.getGeneration(), PROVIDER_SESSION, PROVIDER_SESSION, "prompt", "call", "sha256:" + "b".repeat(64),
|
||||
Map.of("dispatchMode", "deferred", "sessionId", PROVIDER_SESSION, "promptId", "prompt",
|
||||
@@ -292,9 +286,7 @@ class RuntimeBrokerServiceTest {
|
||||
void hookRecoveryDoesNotWaitForAnOrdinaryAcquireOfTheSameRuntimeId() throws Exception {
|
||||
try (Fixture fixture = new Fixture(SESSION_SCOPE)) {
|
||||
RuntimeSessionRecord original = join(fixture.service.acquire("harness-a", "runtime-a", "bootstrap"));
|
||||
var field = RuntimeBrokerService.class.getDeclaredField("sessions");
|
||||
field.setAccessible(true);
|
||||
((Map<?, ?>) field.get(fixture.service)).remove("runtime-a");
|
||||
BrokerServiceTestAccess.registry(fixture.service).removeSession("runtime-a");
|
||||
fixture.resolver.result = CompletableFuture.completedFuture(new RuntimeScope(
|
||||
"tenant", "other-workspace", "generation", "/other", "other-capability", "session"));
|
||||
var pending = new CompletableFuture<Void>();
|
||||
@@ -317,20 +309,18 @@ class RuntimeBrokerServiceTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@SuppressWarnings("unchecked")
|
||||
void hookRecoveryRejectsFailedAndCancelledLocalRoutes() throws Exception {
|
||||
try (Fixture fixture = new Fixture(SESSION_SCOPE)) {
|
||||
RuntimeSessionRecord original = join(fixture.service.acquire("harness-a", "runtime-a", "bootstrap"));
|
||||
var field = RuntimeBrokerService.class.getDeclaredField("sessions");
|
||||
field.setAccessible(true);
|
||||
var routes = (Map<String, CompletableFuture<?>>) field.get(fixture.service);
|
||||
var cancelled = new CompletableFuture<>();
|
||||
var registry = BrokerServiceTestAccess.registry(fixture.service);
|
||||
var cancelled = new CompletableFuture<RuntimeBindingRegistry.SessionContext>();
|
||||
cancelled.cancel(false);
|
||||
int provisions = fixture.provisioner.calls.get();
|
||||
int acquires = fixture.transport.acquireCalls.get();
|
||||
for (var route : List.of(CompletableFuture.failedFuture(
|
||||
for (var route : List.of(CompletableFuture.<RuntimeBindingRegistry.SessionContext>failedFuture(
|
||||
new RuntimeBrokerException(503, "ordinary_failure", "failed", true)), cancelled)) {
|
||||
routes.put("runtime-a", route);
|
||||
registry.removeSession("runtime-a");
|
||||
registry.registerSession("runtime-a", route);
|
||||
assertEquals("workspace_close_identity_unverified", failure(fixture.service.acquireRecovery(
|
||||
"harness-a", "runtime-a", original.getBindingId(), original.getRuntimeGeneration())).getCode());
|
||||
}
|
||||
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
package com.alibaba.qwen.code.runtimebroker.service;
|
||||
|
||||
import com.alibaba.qwen.code.runtimebroker.ExecutionReconciliation;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeLease;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeProvisioner;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeSession;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeTransport;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.ToolExecutionRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.service.RuntimeBindingRegistry.SessionContext;
|
||||
import org.junit.jupiter.api.Timeout;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionException;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.doAnswer;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.spy;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class RuntimeExecutionReconcilerTest {
|
||||
@ParameterizedTest
|
||||
@ValueSource(booleans = {false, true})
|
||||
@Timeout(15)
|
||||
void anObserverDuringCooldownStampJoinsTheLookupInsteadOfCallingTheWorkerAgain(boolean failed) throws Exception {
|
||||
var scheduler = Executors.newSingleThreadScheduledExecutor();
|
||||
var entered = new CountDownLatch(1);
|
||||
var release = new CountDownLatch(1);
|
||||
var closed = new AtomicBoolean();
|
||||
Clock clock = Clock.fixed(Instant.parse("2026-11-01T00:00:00Z"), ZoneOffset.UTC);
|
||||
var cooldown = spy(new ExecutionObservationCooldown(clock, scheduler));
|
||||
doAnswer(call -> {
|
||||
entered.countDown();
|
||||
assertTrue(release.await(5, TimeUnit.SECONDS), "cooldown stamp was not released");
|
||||
return call.callRealMethod();
|
||||
}).when(cooldown).stamp(eq("execution"), any());
|
||||
|
||||
var registry = mock(RuntimeBindingRegistry.class);
|
||||
var state = mock(RuntimeExecutionState.class);
|
||||
var executions = mock(ToolExecutionRepository.class);
|
||||
var provisioner = mock(RuntimeProvisioner.class);
|
||||
var transport = mock(RuntimeTransport.class);
|
||||
var unknown = mock(ToolExecutionRecord.class);
|
||||
var context = mock(SessionContext.class);
|
||||
var binding = mock(RuntimeBindingRecord.class);
|
||||
var session = mock(RuntimeSession.class);
|
||||
var lease = mock(RuntimeLease.class);
|
||||
when(unknown.getExecutionCallId()).thenReturn("execution");
|
||||
when(unknown.getState()).thenReturn(ToolExecutionRecord.State.UNKNOWN);
|
||||
when(unknown.getBindingId()).thenReturn("binding");
|
||||
when(unknown.getRuntimeGeneration()).thenReturn(7L);
|
||||
when(unknown.getReference()).thenReturn(Map.of("callId", "call"));
|
||||
when(state.requireOwnedExecution("harness", "runtime", "execution")).thenReturn(unknown);
|
||||
when(executions.findByExecutionCallId("execution")).thenReturn(unknown);
|
||||
when(registry.session("runtime")).thenReturn(CompletableFuture.completedFuture(context));
|
||||
when(registry.hasBinding("binding")).thenReturn(true);
|
||||
when(context.session()).thenReturn(session);
|
||||
when(context.binding()).thenReturn(binding);
|
||||
when(context.lease()).thenReturn(lease);
|
||||
when(session.getHarnessSessionId()).thenReturn("harness");
|
||||
when(binding.getBindingId()).thenReturn("binding");
|
||||
when(binding.getGeneration()).thenReturn(7L);
|
||||
when(provisioner.isUsable(lease)).thenReturn(true);
|
||||
var answer = new CompletableFuture<Map<String, Object>>();
|
||||
var lookups = new AtomicInteger();
|
||||
when(transport.status(eq(lease), eq(session), any(), eq(0L))).thenAnswer(call -> {
|
||||
lookups.incrementAndGet();
|
||||
return answer;
|
||||
});
|
||||
var reconciler = new RuntimeExecutionReconciler(registry, state, executions, provisioner, transport,
|
||||
null, Duration.ofSeconds(10), clock, closed, cooldown);
|
||||
Thread completer = null;
|
||||
try {
|
||||
var first = reconciler.observeExecution("harness", "runtime", "execution").toCompletableFuture();
|
||||
completer = Thread.ofPlatform().start(() -> {
|
||||
if (failed) {
|
||||
answer.completeExceptionally(new IllegalStateException("worker unavailable"));
|
||||
} else {
|
||||
answer.complete(Map.of("state", "executing"));
|
||||
}
|
||||
});
|
||||
assertTrue(entered.await(5, TimeUnit.SECONDS), "lookup never reached cooldown stamp");
|
||||
var submitted = CompletableFuture.supplyAsync(() -> reconciler.observeExecution(
|
||||
"harness", "runtime", "execution").toCompletableFuture());
|
||||
var joined = submitted.get(1, TimeUnit.SECONDS);
|
||||
assertEquals(1, lookups.get(), "a caller escaped between reservation removal and cooldown stamp");
|
||||
assertFalse(joined.isDone());
|
||||
release.countDown();
|
||||
completer.join(2000);
|
||||
assertFalse(completer.isAlive());
|
||||
if (failed) {
|
||||
assertThrows(CompletionException.class, first::join);
|
||||
assertThrows(CompletionException.class, joined::join);
|
||||
} else {
|
||||
assertEquals("executing", first.get(2, TimeUnit.SECONDS).getRuntimeState());
|
||||
assertEquals("executing", joined.get(2, TimeUnit.SECONDS).getRuntimeState());
|
||||
}
|
||||
assertEquals(ExecutionReconciliation.Outcome.UNRESOLVED,
|
||||
reconciler.observeExecution("harness", "runtime", "execution").toCompletableFuture()
|
||||
.get(2, TimeUnit.SECONDS).getOutcome());
|
||||
assertEquals(1, lookups.get(), "a sequential observer missed the completed cooldown");
|
||||
} finally {
|
||||
release.countDown();
|
||||
closed.set(true);
|
||||
reconciler.cancelPendingOperations();
|
||||
scheduler.shutdownNow();
|
||||
if (completer != null) {
|
||||
completer.join(2000);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user