mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
refactor(managed): Remove persisted public message snapshots (#15870)
Use bounded live message pages for public history and exports, remove the public snapshot schema and background lifecycle, and retain native recovery history. Co-authored-by: Qwen-Coder <[email protected]>
This commit is contained in:
1 parent
70be140e14
commit
619a601b39
61 files changed
+1320
-2825
No files matched your search
@@ -2,6 +2,8 @@
|
||||
|
||||
[English](2026-10-29-managed-live-message-pages.md) | [简体中文](2026-10-29-managed-live-message-pages.zh-CN.md)
|
||||
|
||||
The follow-up [snapshot removal](2026-10-29-managed-message-snapshot-removal.md) supersedes this change's optional read selector and retained snapshot defaults. Direct message reads become the default and the public snapshot table is removed there; the implementation and validation status of that follow-up are recorded separately.
|
||||
|
||||
## Problem and scope
|
||||
|
||||
The merged bounded query API reads published snapshots. It can lag message projection and the Java WebShell Provider still bootstraps through the full-snapshot transcript path. The requested behavior is direct SQL reads for the latest 100 Items, older pages on scroll, and live updates through the existing event stream. Export reads live message pages and does not promise frozen bodies. Only public Managed H2/MySQL WebShell reads change; daemon Legacy, execution recovery, private CSI, canonical journals and schema remain unchanged. Existing snapshot APIs and their authenticated cursors retain their default semantics; removing their other dependencies is separate work.
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
[English](2026-10-29-managed-live-message-pages.md) | [简体中文](2026-10-29-managed-live-message-pages.zh-CN.md)
|
||||
|
||||
后续[快照删除](2026-10-29-managed-message-snapshot-removal.zh-CN.md)替代本变更的可选读取选择项及保留快照默认路径。在该后续变更中,直接消息读取成为默认,并移除公开快照表;其实现及验证状态单独记录。
|
||||
|
||||
## 问题与范围
|
||||
|
||||
已合并的有界查询API读取已发布快照,可能落后于消息投影,Java WebShell Provider启动时仍使用完整快照transcript路径。目标是直接SQL读取最近100个Item、滚动加载更早页,并通过既有事件流实时更新。导出读取实时消息页,不承诺冻结正文。只调整公开Managed H2/MySQL WebShell读取;daemon Legacy、执行恢复、私有CSI、canonical journal和schema保持不变。既有快照API及认证游标保留默认语义,清理它们的其他依赖另行处理。
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
# Remove persisted Managed conversation snapshots
|
||||
|
||||
[English](2026-10-29-managed-message-snapshot-removal.md) | [简体中文](2026-10-29-managed-message-snapshot-removal.zh-CN.md)
|
||||
|
||||
## Status and scope
|
||||
|
||||
Initially implemented on top of the direct-message paging change in PR #15865 at `cf7a0edf37d390a58db00901a397286023ce62f5`, following merged bounded-history queries (#15839). See [Managed live message pages](2026-10-29-managed-live-message-pages.md) for the preceding change. The current diff removes persisted public conversation snapshots; acceptance remains pending the checks listed below.
|
||||
|
||||
After #15865 merged as `89d3aaea62af96849c4b9b7a2bc68e26347a7ac8` with final head `ca8970b84b96c6d3cc5f77fc43b577150a0944c8`, the removal increment was rebased onto `main`. The dependency's coverage validation and smoke-fixture repairs remain intact. The checks below predate this baseline synchronization and do not establish final-tree acceptance; no local Java allocation was reopened.
|
||||
|
||||
The user selected direct reads of mutable messages, recent-message paging, and live export without a fixed content version. Remove `ma_snapshot` and its persisted public message-page copies. Preserve canonical journals, private native journal snapshots, execution checkpoints, child launch snapshots, Workspace authorization, and Legacy daemon behavior.
|
||||
|
||||
## Current behavior and problem
|
||||
|
||||
Public message queries and the default Managed WebShell transcript/export paths use `ManagedSnapshotStore`. The materializer periodically freezes the complete message projection into manifest and page resources, publishes a `ma_snapshot` row, and advances `ma_event_stream.current_snapshot_id`. Signed cursors identify that persisted version. The event replay floor, Session summaries, resource collection, and CSI application inventory also depend on those rows.
|
||||
|
||||
Direct-message paging makes those copies unnecessary for the selected product behavior. Removing only the table would leave active SQL reads, background writes, retention jobs, and integrity checks referring to an absent object.
|
||||
|
||||
## Proposed behavior
|
||||
|
||||
### Message reads and live export
|
||||
|
||||
Use the direct `ma_message` pager for every public Managed message read: public Items, recent Items, default transcript, and export. The first request fixes a membership upper bound; each page reads current content in its own consistent SQL view. Keep the complete `(first_sequence, item_id)` order, direction and scope validation, projection-version checks, authorization on every request, and existing item and byte limits. Adapt numeric `after=N` explicitly: exclude every Item whose first-sequence is N, then use complete tuple boundaries for continuation so equal-sequence Items on later pages are retained. Preserve explicit `cursor` precedence and the existing nonnumeric `after` cursor alias. Old snapshot cursors are rejected; they are not reinterpreted as message cursors.
|
||||
|
||||
The default transcript changes from a 1–1000 event limit to a 1–100 message limit; omission still defaults to 100. Existing requests with limits 101–1000 now receive a parameter error and should page through older Items instead. The removed `view` selector is no longer interpreted; retain the existing JSON unknown-property handling rather than adding a global strict-deserialization policy. Clients should omit the selector under the new contract.
|
||||
|
||||
The default `/transcript/query` cursor changes from a positive decimal event sequence to an opaque message cursor. Previously issued decimal cursors are rejected with `400 invalid_message_cursor`, replacing the old `invalid_cursor` code; discard stored decimal cursors and reload the first message page after upgrading. Public Items numeric `after` and SSE `afterSequence` retain their existing numeric semantics.
|
||||
|
||||
The transcript response's resume sequence is the message projection's actual `covered_sequence`, not the public event tail. A client receives subsequent changes through SSE. Export reads forward within its initial membership bound and does not claim immutable content. If the initial projection is behind committed events, use the direct pager's explicit retryable pending response rather than silently exporting an empty or incomplete history.
|
||||
|
||||
Existing response field names that refer to snapshot coverage may remain as transitional wire names only where needed by current consumers; they do not imply a retained version. Distinguish their meanings by response: transcript coverage is the current page's actual `covered_sequence` for SSE handoff; export and public Item pagination coverage is the first request's fixed `throughSequence` membership bound and must not advance on later pages. Do not introduce a replacement snapshot table, persisted page store, or process-local history cache.
|
||||
|
||||
### Materialization and event replay
|
||||
|
||||
Continue updating `ma_message`, public resource references, `covered_sequence`, and `projection_version`. Remove public snapshot building, refresh thresholds, stale markers, and idle snapshot refresh scans. Projection work is pending exactly when `last_sequence > covered_sequence`.
|
||||
|
||||
Read replay coverage directly from the Session event-stream row. Preserve the monotonic replay floor, active-Turn protection, private CSI exclusions, and the current deployment enablement policy. A floor advance must never exceed readable message-projection coverage. Projection rebuild does not lower the floor: preserve the pager's retryable pending guard when `covered_sequence < replay_floor_sequence` and cursor invalidation when `projection_version` changes. If the floor advances between a message read and SSE subscription, expired replay reloads the default live-message path rather than a removed snapshot path. This change does not add public event deletion or compact canonical history.
|
||||
|
||||
### Resource lifetime and CSI inventory
|
||||
|
||||
Remove public `SNAPSHOT` owners, `managed-snapshot-page` and `managed-snapshot-manifest` resources, snapshot expiry scans, and public snapshot retirement batches. Retain message-body collection, read leases, recovery protection, and all native `NATIVE_SNAPSHOT` ownership and collection.
|
||||
|
||||
Update the CSI application inventory's declared schema, queries, allowed owners and resource kinds, and structural/content checks together. Continue rejecting unknown objects, references, or scope violations. Removing an obsolete snapshot branch must not weaken the checks for message bodies, journal resources, native history, output artifacts, or physical ownership.
|
||||
|
||||
### Shared schema and deployment
|
||||
|
||||
Edit the single canonical V1 baseline in place under the first-release policy. Remove `ma_snapshot`, `ma_event_stream.current_snapshot_id`, `snapshot_stale_since`, public snapshot page metadata in `ma_resource`, and their indexes, foreign keys, and checks. Remove the public `SNAPSHOT` resource-owner enum value while preserving `NATIVE_SNAPSHOT`. Update explicit projections, schema validation, CSI declarations, legacy responsibility mapping, and applicable tests. The inspected 39-table baseline becomes 38 business tables.
|
||||
|
||||
Development catalogs require recreation; this change supplies no V2 migration, `DROP TABLE` against a running catalog, or repair/history bypass. Preserve the other chat's running preview and its original database. Matching Server/Broker/Harness/WebShell builds are required for qualification.
|
||||
|
||||
## Affected components
|
||||
|
||||
| Component | Required work |
|
||||
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| `ManagedAgentStore`, `AgentStateStore`, `ManagedAgentService` | Replace remaining public snapshot reads and coverage joins; remove building and refresh orchestration. |
|
||||
| `ManagedSnapshotStore`, public snapshot model records | Remove obsolete persisted snapshot implementation and interfaces. |
|
||||
| Replay floor and SSE services | Use projection coverage and document message resynchronization semantics. |
|
||||
| Projection collector and Session retirement | Remove public snapshot collection; retain message and native-history protection. |
|
||||
| Canonical schema and `managedschema` contracts | Remove obsolete table, fields, owners and indexes consistently. |
|
||||
| `WorkspaceCsiApplicationInventory` | Remove public snapshot inventory and validation without broadening accepted unknown state. |
|
||||
| OpenAPI and WebShell | Align default live-read semantics, cursor rejection, export metadata and generated types. |
|
||||
| Tests and current operational documentation | Replace fixed-copy assertions with live paging, watermark, authorization, retirement and recovery invariants. |
|
||||
|
||||
## Validation and acceptance
|
||||
|
||||
Message-page errors retain their exact recovery meanings: `400 invalid_message_cursor` rejects invalid cursor bounds/scope/direction, `409 message_projection_changed` requires discarding the cursor after a projection rebuild, `503 message_projection_pending` is retryable projection lag, and `413 message_page_too_large` is a response budget failure. A single oversized Item remains unavailable through message paging even at limit 1. Signed POST request-body failures retain the separate `payload_too_large` code.
|
||||
|
||||
Review follow-up strengthens the body-read, write-free read, live-reference, coverage/control-event, transaction-annotation, unknown-selector and MySQL literal-drift witnesses; the CSI body-window negative keeps a valid internal range, and the fixture exporter preserves WebShell milliseconds directly from original events. These Java test edits have source review only until CI executes the new revision. The local Java allocation stays closed; earlier successful CI remains evidence only for its recorded inputs.
|
||||
|
||||
Use the direct-message regression cases for no-snapshot sessions, projection lag, duplicate first-sequences, bounded forward/backward paging, evolving content, scope and version errors, and finite export during continued writes. Exercise reconnect before and after replay-floor advancement and projection rebuild. Cover Session summaries, collector runs, retirement, strict schema admission and CSI inventories on the schema without `ma_snapshot`. Native journal snapshot publication, readback, restore and collection must retain their original behavior.
|
||||
|
||||
Run affected Java unit/H2 and disposable MySQL cases, API/surface contracts, and affected WebShell checks under applicable repository budgets; full required profiles remain CI gates. Reuse upstream evidence only for identical tested inputs. The upstream chat's closed Java and preview-startup allocations cannot be reused or reset by this follow-up. Record unavailable execution explicitly rather than treating source inspection as runtime acceptance.
|
||||
|
||||
Local build, typecheck, bundle and all 318 tests in the four affected WebShell suites passed. The latest collected unit/H2 reports contain 835 cases: 834 passed and one API contract fixture failed because an empty Session used a helper requiring two messages. The fixture now waits for the message query to become ready; that correction was not rerun locally.
|
||||
|
||||
Disposable MySQL 8.4.11 completed 141 cases: 140 passed, including 27 Managed API, 22 native journal snapshot, 24 retained artifact and 66 CSI inventory cases. One schema negative-test fixture still named a removed public snapshot CHECK; it now mutates the surviving resource length CHECK and remains unrerun. The Session-schema group was interrupted and the dedicated message-paging group did not start before the local budget ended. The temporary database server and files were cleaned up. The 24 retained-artifact cases passed before three negative fixtures were strengthened to preserve digest-derived resource identity and the original large-body length. Those strengthened fixtures were not compiled or rerun locally; the earlier report is not evidence for their new assertions.
|
||||
|
||||
This explicit table-removal request used its separate default 600-second local Java allocation; it is closed below that limit, including setup and cleanup. Earlier chat allocations remain closed and were not reused. Final API/schema-fixture reruns, the strengthened CSI negatives, complete MySQL paging/concurrency, Checkstyle, SpotBugs and relevant full CI profiles remain required. Independent source review also found an obsolete generated-expression normalization branch that assumed an OR in projection_pending. The branch was removed and missing/changed-expression diagnostic tests were added; this final validator change and its tests were not compiled or run locally under the closed budget. The recorded unit and MySQL runs are partial evidence, not a passing full gate or production acceptance.
|
||||
|
||||
Subsequent external CI run `40915271041` compiled the `6851d1a8` source tree (synthetic merge `abe373a3`, with the same tree digest). The schema diagnostic, retained-artifact, removal, direct-paging and API contract suites passed there, but the complete run failed: Checkstyle found two obsolete imports and the 5971-case unit phase reported ten failures from three stale physical-table-count assertions. The review follow-up removes those imports and changes those counts from 40 to 39 (38 business tables plus Flyway); verification of the follow-up revision remains pending. No local Java allocation was reopened.
|
||||
|
||||
Acceptance requires no production SQL references to the removed table or public snapshot columns, no generation of public snapshot copies, bounded default message reads and live export, a safe SSE resume watermark, consistent strict schema/CSI validation, and intact private recovery history. Outstanding integration or CI evidence must remain explicit.
|
||||
|
||||
## Decisions and boundaries
|
||||
|
||||
No fixed-version conversation export is required. The compatibility change is intentional and must be stated in API and deployment documentation. No original history is deleted and no existing user database is altered by development work. This is a follow-up to the other chat's reading-path change; do not edit its active checkout or claim its unfinished validation as evidence for this removal.
|
||||
@@ -0,0 +1,86 @@
|
||||
# 移除 Managed 持久化对话快照
|
||||
|
||||
[English](2026-10-29-managed-message-snapshot-removal.md) | [简体中文](2026-10-29-managed-message-snapshot-removal.zh-CN.md)
|
||||
|
||||
## 状态与范围
|
||||
|
||||
最初实现基于 PR #15865 的直接消息分页提交 `cf7a0edf37d390a58db00901a397286023ce62f5`,承接已合并的有界历史查询(#15839)。前一阶段见[Managed 实时消息分页](2026-10-29-managed-live-message-pages.zh-CN.md)。当前差异移除持久化公开对话快照;验收仍等待下列检查完成。
|
||||
|
||||
在 #15865 以 `89d3aaea62af96849c4b9b7a2bc68e26347a7ac8` 合并、最终 head 为 `ca8970b84b96c6d3cc5f77fc43b577150a0944c8` 后,删表增量已重放到 `main`。依赖中的水位校验和 smoke 夹具修正保持完整。下文检查早于此次基线同步,不能证明最终源码树验收通过;没有重新开启本地 Java 额度。
|
||||
|
||||
用户已选择直接读取可变消息、最近消息分页,以及不要求固定正文版本的实时导出。删除 `ma_snapshot` 及其持久化公开消息页副本。保留 canonical journal、私有原生 journal 快照、执行 checkpoint、child launch snapshot、Workspace 授权及 Legacy daemon 行为。
|
||||
|
||||
## 现有行为与问题
|
||||
|
||||
公开消息查询及 Managed WebShell 默认 transcript/export 路径使用 `ManagedSnapshotStore`。物化器定期将完整消息投影冻结为 manifest 和页面资源,发布 `ma_snapshot` 行,并推进 `ma_event_stream.current_snapshot_id`。签名游标标识该持久化版本。事件重放下界、Session 摘要、资源回收及 CSI 应用清单也依赖这些行。
|
||||
|
||||
直接消息分页使选定产品行为不再需要这些副本。仅删除表会留下查询缺失对象的 SQL、后台写入、保留任务及完整性检查。
|
||||
|
||||
## 目标行为
|
||||
|
||||
### 消息读取与实时导出
|
||||
|
||||
所有公开 Managed 消息读取统一使用直接 `ma_message` 分页:公开 Items、最近 Items、默认 transcript 及导出。首请求固定成员范围上界;每页在独立一致 SQL 视图读取当前正文。保留完整 `(first_sequence, item_id)` 排序、方向及 scope 校验、投影版本检查、每次请求的授权,以及现有条目和字节上限。明确适配数字 `after=N`:排除所有 first-sequence 等于 N 的 Item;后续使用完整双键边界,保留跨页的同序号 Item。保留显式 `cursor` 优先及非数字 `after` 作为游标的现有兼容解析。拒绝旧快照游标,不将其重新解释为消息游标。
|
||||
|
||||
默认 transcript 从接受 1–1000 的事件 limit 切换为每页 1–100 条消息;省略时仍默认为 100。原有 101–1000 的 limit 请求现在会收到参数错误,应改为向更早 Item 分页。已移除的 `view` 选择参数不再解释;保留现有 JSON 未知属性处理,不新增全局严格反序列化策略。客户端应按新契约省略该参数。
|
||||
|
||||
默认 `/transcript/query` 游标从正十进制事件序号改为不透明消息游标。此前发出的十进制游标返回 `400 invalid_message_cursor`,替代旧的 `invalid_cursor` 错误码;升级后应丢弃保存的十进制游标,从首个消息页重新读取。公开 Items 的数字 `after` 及 SSE 的 `afterSequence` 保留既有数字语义。
|
||||
|
||||
transcript 响应的续接序号为消息投影实际完成的 `covered_sequence`,不能使用公开事件 tail。客户端通过 SSE 接收后续变化。导出在初始成员上界内正向读取,不承诺正文不可变。初始投影落后于已提交事件时,沿用直接分页器明确且可重试的 pending 响应,不静默导出为空或缺失历史。
|
||||
|
||||
现有消费者确需时,可以暂时保留名称含 snapshot coverage 的响应字段,但不暗示存在保留版本。必须按响应区分其含义:transcript 使用本页实际 `covered_sequence` 续接 SSE;导出及公开 Item 分页使用首请求固定的 `throughSequence` 成员上界,后续页不得推进。不新增替代表、持久化分页存储或进程内历史缓存。
|
||||
|
||||
### 消息物化与事件重放
|
||||
|
||||
继续更新 `ma_message`、公开资源引用、`covered_sequence` 和 `projection_version`。删除公开快照构建、刷新阈值、stale 标记及空闲快照刷新扫描。只有 `last_sequence > covered_sequence` 时才存在待完成投影。
|
||||
|
||||
直接从 Session 事件流行读取重放覆盖水位。保留单调递增的 replay floor、活动 Turn 保护、私有 CSI 排除,以及现有部署启用策略。floor 不得越过可读取的消息投影水位。重建投影不降低 floor:保留分页器在 `covered_sequence < replay_floor_sequence` 时可重试的 pending 检查,以及 `projection_version` 变化后的游标失效。消息读取后、SSE 订阅前 floor 已推进时,过期重放重新加载默认实时消息路径,不再请求已删除的快照。本次不增加公开事件删除,不压缩 canonical 历史。
|
||||
|
||||
### 资源生命周期与 CSI 清单
|
||||
|
||||
删除公开 `SNAPSHOT` owner、`managed-snapshot-page` 和 `managed-snapshot-manifest` 资源、快照过期扫描及公开快照退役批次。保留消息正文回收、读租约、恢复保护,以及全部原生 `NATIVE_SNAPSHOT` 所有权与回收。
|
||||
|
||||
同步更新 CSI 应用清单声明的 schema、查询、允许的 owner 和资源种类,以及结构和内容校验。继续拒绝未知对象、引用或 scope 越界。删除过时快照分支不能削弱消息正文、journal 资源、原生历史、输出产物或物理所有权检查。
|
||||
|
||||
### 共享 schema 与部署
|
||||
|
||||
按首发前规则原地修改唯一 canonical V1 baseline。删除 `ma_snapshot`、`ma_event_stream.current_snapshot_id`、`snapshot_stale_since`、`ma_resource` 中的公开快照页元数据,以及对应索引、外键和 CHECK。删除公开 `SNAPSHOT` 资源 owner 枚举值,保留 `NATIVE_SNAPSHOT`。同步显式投影、schema 校验、CSI 声明、旧职责映射及适用测试。已核对的 39 表 baseline 将变为 38 张业务表。
|
||||
|
||||
开发库需要重建;本次不提供 V2 迁移、运行库 `DROP TABLE` 或 repair/history 绕过。保留另一会话正在运行的预览及其原数据库。验收要求 Server/Broker/Harness/WebShell 使用匹配版本。
|
||||
|
||||
## 受影响组件
|
||||
|
||||
| 组件 | 必要改动 |
|
||||
| ------------------------------------------------------------- | ------------------------------------------------------------ |
|
||||
| `ManagedAgentStore`、`AgentStateStore`、`ManagedAgentService` | 替换剩余公开快照读取和覆盖水位联接,移除构建与刷新编排。 |
|
||||
| `ManagedSnapshotStore`、公开快照模型记录 | 删除过时的持久化快照实现与接口。 |
|
||||
| Replay floor 与 SSE 服务 | 使用投影覆盖水位,并明确消息重新同步语义。 |
|
||||
| 投影回收器与 Session 退役 | 移除公开快照回收,保留消息及原生历史保护。 |
|
||||
| Canonical schema 与 `managedschema` 契约 | 一致删除过时表、字段、owner 和索引。 |
|
||||
| `WorkspaceCsiApplicationInventory` | 删除公开快照清单与校验,不扩大对未知状态的接受范围。 |
|
||||
| OpenAPI 与 WebShell | 对齐默认实时读取语义、游标拒绝、导出元数据及生成类型。 |
|
||||
| 测试及现役运维文档 | 将固定副本断言替换为实时分页、水位、授权、退役及恢复不变量。 |
|
||||
|
||||
## 验证与验收
|
||||
|
||||
消息分页错误保留各自恢复语义:`400 invalid_message_cursor` 拒绝非法游标边界、scope 或方向;`409 message_projection_changed` 要求在投影重建后丢弃游标;`503 message_projection_pending` 表示可重试的投影落后;`413 message_page_too_large` 表示响应字节预算不足。单个超大 Item 即使 limit 为 1 也不能通过消息分页取得。签名 POST 请求体过大仍使用独立的 `payload_too_large` 错误码。
|
||||
|
||||
评审后续加强正文读取、无写入读取、存活引用、水位/控制事件、事务注解、旧选择参数兼容及 MySQL 字面量漂移见证;CSI 正文窗口负例保持内部区间合法,夹具导出直接从原始事件保留 WebShell 毫秒值。这些 Java 测试改动在新修订 CI 执行前仅有源码审阅。本地 Java 额度保持关闭;此前成功 CI 只证明其已记录的输入。
|
||||
|
||||
复用直接消息回归场景:无快照会话、投影滞后、相同 first-sequence、有界双向分页、正文变化、scope 和版本错误,以及持续写入下有限导出。覆盖 replay floor 推进及投影重建前后的重连。在不含 `ma_snapshot` 的 schema 上覆盖 Session 摘要、回收器运行、退役、严格 schema 准入及 CSI 清单。原生 journal 快照发布、读回、恢复和回收保持原有行为。
|
||||
|
||||
按适用仓库预算运行受影响 Java 单元/H2、独立 MySQL、API/surface 契约及 WebShell 检查;完整必需 profile 仍是 CI 门禁。仅在测试输入相同时复用上游证据。不得复用另一会话已关闭的 Java 及预览启动额度,也不得通过本次后续工作重置。无法执行的验收明确记录,不能将源码检查当作运行验收。
|
||||
|
||||
本地 build、typecheck、bundle 及四个受影响 WebShell 测试文件的 318 项测试通过。最新已收集的单元/H2 报告包含 835 项:834 项通过,一项 API 契约夹具因空 Session 误用要求两条消息的等待 helper 失败。该夹具已改为等待消息查询就绪,但尚未在本地重跑。
|
||||
|
||||
独立 MySQL 8.4.11 完成 141 项:140 项通过,包括 Managed API 27 项、原生 journal 快照 22 项、保留产物 24 项及 CSI 清单 66 项。一项 schema 负例夹具仍引用已移除的公开快照 CHECK,已改为修改保留的资源长度 CHECK,尚未重跑。Session schema 组在本地预算截止时中断,专门的消息分页组未开始。临时数据库服务和文件已清理。保留产物的 24 项通过报告早于三项负例夹具加强:新版夹具保持摘要派生资源身份及原始大正文长度一致。这些加强后的夹具未在本地编译和重跑,旧报告不能证明新断言通过。
|
||||
|
||||
本次明确的删表请求使用独立的默认 600 秒本地 Java 额度,包含准备及清理,并已在上限内关闭。此前会话额度仍保持关闭,没有复用。最终 API/schema 夹具复验、加强后的 CSI 负例、完整 MySQL 分页及并发、Checkstyle、SpotBugs 和相关完整 CI profile 仍须完成。独立源码审阅还发现生成列表达式的旧规范化分支假定 projection_pending 含 OR;已删除该分支,并新增缺失或漂移表达式的诊断测试。预算关闭后,这项最终校验器改动及其测试未在本地编译和运行。已记录的单元及 MySQL 运行属于部分证据,不能视为完整门禁通过或生产验收。
|
||||
|
||||
后续外部 CI run `40915271041` 已编译 `6851d1a8` 的源码树(合成 merge 为 `abe373a3`,tree 摘要相同)。schema 诊断、保留产物、删除回归、直接分页及 API 契约组在该树上通过,但完整运行失败:Checkstyle 发现两条过时 import,5971 项单元阶段因三处旧物理表数断言出现十项失败。本轮审阅修正删除这两条 import,并将这三处计数从 40 改为 39(38 张业务表加 Flyway);修正后修订的验证仍待完成,没有重新开启本地 Java 额度。
|
||||
|
||||
验收要求:生产 SQL 不再引用已删除的表及公开快照字段,不再生成公开快照副本;默认消息读取及实时导出有界;SSE 续接水位安全;严格 schema/CSI 校验一致;私有恢复历史完整。未完成的集成或 CI 证据必须明确保留。
|
||||
|
||||
## 决策与边界
|
||||
|
||||
不要求固定版本的对话导出。这是有意的兼容语义变化,必须反映在 API 及部署文档中。不删除原始历史,开发工作不修改用户现有数据库。本次是另一会话读取路径变更的后续,不编辑其活动工作树,不把其未完成验证当成本次删除的证据。
|
||||
@@ -2,17 +2,19 @@
|
||||
|
||||
[English](2026-10-29-managed-sql-history-queries.md) | [简体中文](2026-10-29-managed-sql-history-queries.zh-CN.md)
|
||||
|
||||
The follow-up [public snapshot removal](2026-10-29-managed-message-snapshot-removal.md) supersedes this design's published-snapshot message reads and signed cursors. The recent-message contract below reflects bounded live message reads; journal queries retain their existing behavior. Follow-up implementation and validation evidence are recorded separately.
|
||||
|
||||
## Scope
|
||||
|
||||
Provide small, bounded reads for Managed local H2 and Hosted MySQL. Reuse existing SQL tables, published snapshots and authorization. This replaces the broader implementation in PR #15839; the original work remains preserved in Draft PR #15696 and the earlier commit history. No authority window, history cache, derived history tables, backfill, streaming full recovery, checkpoint/CSI redesign or Legacy behavior change is included.
|
||||
Provide small, bounded reads for Managed local H2 and Hosted MySQL. Reuse existing SQL tables, live message pages and authorization. This replaces the broader implementation in PR #15839; the original work remains preserved in Draft PR #15696 and the earlier commit history. No authority window, history cache, derived history tables, backfill, streaming full recovery, checkpoint/CSI redesign or Legacy behavior change is included.
|
||||
|
||||
## Recent messages
|
||||
|
||||
`POST /api/agent/web-shell/v1/transcript/items/query` accepts the existing `WebShellExportRequest`: `sessionId`, optional `cursor`, and `limit` from 1 to 100. The first request selects the newest published snapshot and reads at most the requested number of Items from its end. Each response is chronological; `nextCursor` moves toward older Items. The existing snapshot page resources bound physical reads without loading the complete snapshot.
|
||||
`POST /api/agent/web-shell/v1/transcript/items/query` accepts the existing `WebShellExportRequest`: `sessionId`, optional `cursor`, and `limit` from 1 to 100. The first request fixes a sequence membership upper bound and reads the newest bounded current message rows. Each response is chronological; `nextCursor` moves toward older Items in that initial range. SQL metadata lookahead and the row/byte budgets bound physical reads without creating or loading snapshot resources.
|
||||
|
||||
Reuse `WebShellExportPage`: `items`, `nextCursor`, `hasMore`, and `snapshotThroughSequence`. Coverage belongs to the captured published snapshot and can lag the current Session head. Empty Items with zero coverage do not certify that the Session is empty: a published snapshot may not be available. This API neither includes the live event tail nor establishes the latest execution state. Existing transcript, stream, task and approval APIs retain those responsibilities. The WebShell client exposes one-page access; the existing UI and transcript reconstruction behavior are unchanged.
|
||||
Reuse `WebShellExportPage`: `items`, `nextCursor`, `hasMore`, and `snapshotThroughSequence`. The last field retains its wire name and denotes the fixed membership upper bound; bodies and statuses remain live. Projection coverage below the replay floor returns retryable `503 message_projection_pending`, rather than certifying an empty Session with a zero-coverage page. This API neither includes the live event tail nor establishes the latest execution state. The transcript reports actual projection coverage for SSE continuation; stream, task and approval APIs retain their existing responsibilities. The WebShell client exposes one-page access.
|
||||
|
||||
The backward cursor uses version 2, authenticated with the existing snapshot secret. Scope, snapshot identity/version, projection and expiry remain pinned. Forward version-1 export cursors and backward cursors are not interchangeable. Every page checks current Session read authorization and existing resource read leases. Publication of a newer snapshot does not change an existing cursor.
|
||||
The backward message cursor uses version 1 with scope, direction, projection generation, membership upper bound and the complete `(first_sequence, item_id)` position. It is unsigned and has no snapshot expiry or identity. Forward and backward cursors are not interchangeable. Every page checks current Session read authorization and uses one consistent SQL read view. A projection rebuild returns `409 message_projection_changed`; discard the cursor and restart the read. Later message updates do not freeze or replace the cursor's initial membership range.
|
||||
|
||||
## Journal pages and point reads
|
||||
|
||||
@@ -24,10 +26,10 @@ Both internal routes preserve tenant/Workspace/Session credentials and database-
|
||||
|
||||
## Compatibility and deferred work
|
||||
|
||||
Canonical history, schema baseline, replay floor, journal format, transaction admission, cleanup and all execution consumers remain unchanged. Existing `/transcript/query`, forward export and `/transactions` semantics remain unchanged. Hosted scratch and checkpoint/CSI safety checks remain in the baseline. Full-snapshot UI pagination and replacement of execution-state reconstruction are separate work; this change does not claim bounded UI history, bounded recovery time or bounded total process RSS.
|
||||
The journal queries preserve canonical history, replay floor, journal format, transaction admission, cleanup and execution consumers. The follow-up removes the public snapshot schema and changes transcript/export to the bounded live message semantics above; `/transactions` remains unchanged. Hosted scratch and native checkpoint/CSI safety checks remain in place. Whole-history UI draining and replacement of execution-state reconstruction are separate work; bounded recent reads do not claim bounded recovery time or bounded total process RSS.
|
||||
|
||||
## Verification
|
||||
|
||||
Focused tests cover backward ordering across physical pages and sequence ties, cursor direction/tampering, fixed snapshots, byte and revision bounds, exact command reads, authorization and unchanged ownership. MySQL tests exercise the same journal queries and recent snapshot reads. OpenAPI, the surface registry, parity capture and generated WebShell types remain synchronized. The thin client requests one page and propagates opaque cursors and failures without automatically draining history.
|
||||
Focused tests cover backward ordering across pages and sequence ties, cursor direction/scope, fixed membership with live content, byte and revision bounds, exact command reads, authorization and unchanged ownership. MySQL tests target the same journal queries and recent message reads; qualification requires execution on the recorded final source. OpenAPI, the surface registry, parity capture and generated WebShell types remain synchronized. The thin client requests one page and propagates opaque cursors and failures without automatically draining history.
|
||||
|
||||
The original local Java budget remains CLOSED at 564.787/600 seconds; the unused 35.213 seconds are forbidden, including preparation, compilation, retries and agents. New Java/H2/MySQL/HTTP execution must be verified by CI on the final head. Local TypeScript build/typecheck and affected client tests are separate evidence. Earlier broad recovery/scale results do not qualify this implementation. No runtime publication, deployment or model acceptance is claimed.
|
||||
@@ -2,17 +2,19 @@
|
||||
|
||||
[English](2026-10-29-managed-sql-history-queries.md) | [简体中文](2026-10-29-managed-sql-history-queries.zh-CN.md)
|
||||
|
||||
后续[公开快照删除](2026-10-29-managed-message-snapshot-removal.zh-CN.md)取代本文的已发布快照消息读取和签名游标。下文最近消息契约已按有界实时消息读取更新;journal 查询保持既有行为。后续实现及验证证据单独记录。
|
||||
|
||||
## 范围
|
||||
|
||||
为 Managed 本地 H2 和 Hosted MySQL 提供小范围、有界读取,复用现有 SQL 表、已发布快照与授权。本方案替换 PR #15839 中较宽的实现;原工作保留在 Draft PR #15696 及此前提交历史中。不包含 authority 窗口、历史缓存、派生历史表、补建、全历史流式恢复、checkpoint/CSI 重构或 Legacy 行为变更。
|
||||
为 Managed 本地 H2 和 Hosted MySQL 提供小范围、有界读取,复用现有 SQL 表、实时消息分页与授权。本方案替换 PR #15839 中较宽的实现;原工作保留在 Draft PR #15696 及此前提交历史中。不包含 authority 窗口、历史缓存、派生历史表、补建、全历史流式恢复、checkpoint/CSI 重构或 Legacy 行为变更。
|
||||
|
||||
## 最近消息
|
||||
|
||||
`POST /api/agent/web-shell/v1/transcript/items/query` 使用既有 `WebShellExportRequest`:`sessionId`、可选 `cursor` 及 1 到 100 的 `limit`。首次请求选择最新已发布快照,从末尾读取不超过指定数量的 Item。每页按时间顺序返回,`nextCursor` 指向更早的 Item。复用现有快照页资源约束物理读取,不加载完整快照。
|
||||
`POST /api/agent/web-shell/v1/transcript/items/query` 使用既有 `WebShellExportRequest`:`sessionId`、可选 `cursor` 及 1 到 100 的 `limit`。首次请求固定序号成员范围上界,读取最新的一组有界当前消息行。每页按时间顺序返回,`nextCursor` 指向首请求范围内更早的 Item。SQL 元数据前瞻以及条目/字节预算约束物理读取,不创建或加载快照资源。
|
||||
|
||||
复用 `WebShellExportPage`:`items`、`nextCursor`、`hasMore` 与 `snapshotThroughSequence`。覆盖位置属于捕获的已发布快照,可能落后于当前 Session head。Item 为空且覆盖位置为零不能证明 Session 没有历史:可能尚无可用的已发布快照。此接口不包含实时事件尾部,也不证明最新执行状态;既有 transcript、事件流、任务和审批接口继续承担这些职责。WebShell 客户端提供单页访问,现有 UI 与 transcript 重建行为保持不变。
|
||||
复用 `WebShellExportPage`:`items`、`nextCursor`、`hasMore` 与 `snapshotThroughSequence`。最后一个字段保留 wire 名称,表示固定成员上界;正文和状态保持实时。投影覆盖落后于 replay floor 时返回可重试的 `503 message_projection_pending`,而非用零覆盖空页证明 Session 为空。此接口不包含实时事件尾部,也不证明最新执行状态。transcript 提供用于 SSE 续接的实际投影覆盖位置;事件流、任务和审批接口继续承担既有职责。WebShell 客户端提供单页访问。
|
||||
|
||||
反向游标使用版本 2,并由既有快照密钥认证。scope、快照身份/版本、projection 和到期时间保持固定。正向导出的版本 1 游标与反向游标不能互换。每页均检查当前 Session 读取权限及既有资源读租约。新快照发布不会改变已有游标的视图。
|
||||
反向消息游标使用版本 1,绑定 scope、方向、投影代次、成员上界及完整 `(first_sequence, item_id)` 位置。游标未签名,没有快照到期时间或身份。正反向游标不能互换。每页均检查当前 Session 读取权限,并使用一致的 SQL 读取视图。投影重建返回 `409 message_projection_changed`,应丢弃游标并重新开始读取。后续消息更新不会冻结内容,也不会替换游标的初始成员范围。
|
||||
|
||||
## Journal 分页与点查
|
||||
|
||||
@@ -24,10 +26,10 @@
|
||||
|
||||
## 兼容与延期工作
|
||||
|
||||
完整 canonical 历史、schema baseline、replay floor、journal 格式、事务准入、清理及所有执行消费者保持不变。既有 `/transcript/query`、正向导出和 `/transactions` 的语义保持。Hosted scratch 及 checkpoint/CSI 安全检查保留在基线中。完整快照的 UI 分页与替换执行状态重建另行处理;本次不宣称 UI 全历史、恢复耗时或进程总 RSS 已有界。
|
||||
Journal 查询保留 canonical 历史、replay floor、journal 格式、事务准入、清理及执行消费者。后续改动删除公开快照 schema,并将 transcript/导出改为上方有界实时消息语义;`/transactions` 保持不变。Hosted scratch 及原生 checkpoint/CSI 安全检查继续保留。UI 全历史自动遍历与替换执行状态重建另行处理;最近消息有界读取不表示恢复耗时或进程总 RSS 已有界。
|
||||
|
||||
## 验证
|
||||
|
||||
聚焦测试覆盖跨物理页及相同序列值的反向顺序、游标方向/篡改、固定快照、字节与版本边界、精确命令读取、授权和所有权不变。MySQL 测试执行同一 journal 查询及最近快照读取。同步 OpenAPI、surface registry、parity capture 与生成的 WebShell 类型。薄客户端只请求一页,原样传递不透明游标与失败,不自动遍历全部历史。
|
||||
聚焦测试覆盖跨页及相同序列值的反向顺序、游标方向/scope、固定成员范围与实时内容、字节与版本边界、精确命令读取、授权和所有权不变。MySQL 测试面向相同 journal 查询及最近消息读取;资格需要基于记录的最终源码执行验证。同步 OpenAPI、surface registry、parity capture 与生成的 WebShell 类型。薄客户端只请求一页,原样传递不透明游标与失败,不自动遍历全部历史。
|
||||
|
||||
原本地 Java 预算仍在 564.787/600 秒主动 CLOSED;剩余 35.213 秒禁用,包含准备、编译、重试与子代理。新增 Java/H2/MySQL/HTTP 执行交由最终 head 的 CI 验证。本地 TypeScript 构建/类型检查和受影响客户端测试是独立证据。旧的完整恢复/规模结果不能验收本实现。不宣称运行时发布、部署或模型验收完成。
|
||||
@@ -145,9 +145,20 @@ Items and resumes after their `snapshot_through_sequence`. A
|
||||
cursor below a Session's replay floor gets `409 cursor_expired` from the JSON
|
||||
event query and one `agent.session.resync_required` frame from either stream
|
||||
whenever the cursor is below the durable floor. Rebuilding projections or pruning
|
||||
all retained events never resets that floor. Snapshot-backed pages keep their immutable
|
||||
version during concurrent message changes; their scoped opaque cursor expires after
|
||||
30 minutes and retains the required snapshot and subsequent replay interval.
|
||||
all retained events never resets that floor. Message queries page directly over
|
||||
`ma_message`; a scoped opaque cursor fixes the membership range and complete
|
||||
ordering position while message bodies and statuses remain live. Projection
|
||||
rebuild invalidates the cursor. The default transcript returns recent Items and
|
||||
the actual materialized coverage for SSE handoff, without reading the whole
|
||||
history or unmaterialized event tail. Export reads forward within its initial
|
||||
membership range and does not promise a fixed content version. Its retained
|
||||
`snapshotThroughSequence` field names that fixed range; transcript coverage
|
||||
names the current read watermark. A projection behind the replay floor or an
|
||||
initial export behind committed events returns retryable
|
||||
`message_projection_pending`. The historical `reload_snapshot` resync action
|
||||
now means reload the live message view. See the
|
||||
[removal design](../../docs/design/2026-10-29-managed-message-snapshot-removal.md)
|
||||
([简体中文](../../docs/design/2026-10-29-managed-message-snapshot-removal.zh-CN.md)).
|
||||
`GET /v1/agents/sessions/{id}/turns` lists a Session's Turns newest first with
|
||||
an opaque cursor, and `GET /v1/agents/sessions/{id}/turns/{turnId}` reads one.
|
||||
Design: [English](../../docs/design/2026-09-27-managed-agent-api-contract.md) |
|
||||
@@ -485,7 +496,7 @@ and requires the retention deployment gates. With both
|
||||
`qwen.managed-agent.tool-publication.enabled` and
|
||||
`qwen.managed-agent.tool-publication.gc-enabled` true, a typed
|
||||
`SESSION_RESOURCE_GC` operation in `ma_operation` releases retired journal,
|
||||
message, record, snapshot and artifact owners and their `ma_resource_ref`
|
||||
message, record, native journal snapshot and artifact owners and their `ma_resource_ref`
|
||||
references in bounded transactions after the deletion grace. Unreferenced
|
||||
`PUBLISHED` or `REFERENCED` immutable `MYSQL_INLINE` bodies in `ma_resource`
|
||||
are then byte-dropped, keeping their identities with state `COLLECTED`.
|
||||
@@ -494,7 +505,8 @@ intent before physical deletion and retains its claim generation and cursor
|
||||
until the outcome is settled. Live readers, recovery protection and unresolved
|
||||
object writes block collection. Session/head tombstones and event-stream tail
|
||||
and retention-floor counters remain. Unreferenced bodies of active Sessions
|
||||
remain retained until Session retirement. Reads of the retired Session stay fenced with
|
||||
remain retained until Session retirement, except released public message bodies
|
||||
that the reader-aware projection collector can reclaim. Reads of the retired Session stay fenced with
|
||||
`tool_output_session_retired` exactly as before; a workspace-recovery read
|
||||
that physically reaches a collected byte copy fails the named check
|
||||
`resource_collected` instead of the generic `resource_layout_unsupported` or
|
||||
@@ -1255,7 +1267,7 @@ standalone reference keeps the one configured directory for legacy unbound
|
||||
Sessions. Persisted bound Sessions use the private Workspace execution path
|
||||
below.
|
||||
|
||||
Managed Server and Runtime Broker share the single 32-table baseline in
|
||||
Managed Server and Runtime Broker share the single 38-table baseline in
|
||||
[`src/main/resources/db/managed-schema/V1__managed_schema.sql`](src/main/resources/db/managed-schema/V1__managed_schema.sql).
|
||||
Until the first formal Managed Agent release, schema changes update this V1 baseline
|
||||
in place. Do not add V2+ migrations or increment the database schema version before
|
||||
@@ -1313,7 +1325,7 @@ Publication and CSI ACK references retain executions and their parent sessions a
|
||||
bindings regardless of age, including COLLECTED publications.
|
||||
After cleanup, existing missing-record responses apply. Callers must not reuse
|
||||
expired Runtime Session IDs or idempotency keys. Initialize a fresh database with
|
||||
the single canonical 32-table V1 baseline and keep cleanup disabled. Legacy
|
||||
the single canonical 38-table V1 baseline and keep cleanup disabled. Legacy
|
||||
catalogs are rejected; no in-place upgrade or old-data migration is provided.
|
||||
Verify repeated initialization preserves terminal receipts, retain the old
|
||||
environment for historical reads and settle its in-flight work before cutover.
|
||||
@@ -1403,7 +1415,7 @@ for record capacity and rewind semantics.
|
||||
|
||||
### Verified original-mount recovery (W1a)
|
||||
|
||||
The current fresh 32-table baseline requires this physical mount guard for LOCAL tool and child Git execution in a trusted, single-host OpenJDK 21 `local-process` deployment on Linux or macOS with a persistent, unambiguous root birth time. Enable `QWEN_MANAGED_AGENT_RUNTIME_VERIFIED_WORKSPACE_RECOVERY_ENABLED=true` and complete offline registration before admitting that work. Text-only Turns can run without registration; LOCAL tool acquisition fails with `409 workspace_unavailable` when the physical mapping is absent. Taking the next tool Turn after a Broker restart also requires `durable-local-process=true`. macOS requires `trusted-local-reboot-recovery=false` and supports same-boot adoption only. Linux whole-host restart succeeds only while the registered physical identity still matches. The shared V1 baseline stores registration, independent `mount_birth_time` and mount fence in `ma_storage` and `ma_storage_owner`. An unregistered mount is never registered from the directory found at startup.
|
||||
The current fresh 38-table baseline requires this physical mount guard for LOCAL tool and child Git execution in a trusted, single-host OpenJDK 21 `local-process` deployment on Linux or macOS with a persistent, unambiguous root birth time. Enable `QWEN_MANAGED_AGENT_RUNTIME_VERIFIED_WORKSPACE_RECOVERY_ENABLED=true` and complete offline registration before admitting that work. Text-only Turns can run without registration; LOCAL tool acquisition fails with `409 workspace_unavailable` when the physical mapping is absent. Taking the next tool Turn after a Broker restart also requires `durable-local-process=true`. macOS requires `trusted-local-reboot-recovery=false` and supports same-boot adoption only. Linux whole-host restart succeeds only while the registered physical identity still matches. The shared V1 baseline stores registration, independent `mount_birth_time` and mount fence in `ma_storage` and `ma_storage_owner`. An unregistered mount is never registered from the directory found at startup.
|
||||
|
||||
Marker v2 stores birth time as a canonical `Instant` string. Linux reads creation time, mtime, device and inode in one JDK `unix` attribute snapshot and rejects creation time at/before epoch or equal to mtime, which OpenJDK 21 may return when birth time is unsupported. A real Linux birth time equal to mtime is also conservatively refused; prepare the project layout or update the root's mtime offline before retrying. macOS reads native nanosecond birth time with `stat` and cross-checks its device/inode against the JDK canonical-directory snapshot, because JDK 21 truncates macOS birth time to seconds. A positive native macOS birth time is independent even when it equals mtime; fresh empty roots keep their original birth time. Mtime is not an identity field, and ordinary mtime changes do not invalidate an unchanged birth time. The marker contains an application-specific HMAC-SHA256 host ID with `Qwen-Code/verified-workspace/v2` as input. Its key is the trimmed machine-id UTF-8 bytes on Linux or the original `IOPlatformUUID` UTF-8 bytes on macOS; neither raw identity is stored in the marker.
|
||||
|
||||
@@ -1539,7 +1551,7 @@ Checkstyle and the Spring Boot repackage, and the full suite includes
|
||||
environment-sensitive timing tests that can fail on a local machine, so CI is
|
||||
the arbiter; for SpotBugs alone, run `mvn compile spotbugs:check`. The
|
||||
optional real-MySQL profile also verifies schema upgrade,
|
||||
exact bytes, public Item/Snapshot projection, and the independent-JVM Managed
|
||||
exact bytes, public Item projection, and the independent-JVM Managed
|
||||
Session Store crash/takeover path:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -51,7 +51,7 @@ the arbiter.
|
||||
|
||||
## JDBC persistence
|
||||
|
||||
`JdbcRuntimeBrokerSchema.initialize(DataSource)` initializes or validates the shared 32-table fresh-database baseline used by Broker and Managed Server. It rejects legacy, foreign, partial and system catalogs before mutation; it does not upgrade or copy existing data. The JDBC implementations use `javax.sql.DataSource` for
|
||||
`JdbcRuntimeBrokerSchema.initialize(DataSource)` initializes or validates the shared 38-table fresh-database baseline used by Broker and Managed Server. It rejects legacy, foreign, partial and system catalogs before mutation; it does not upgrade or copy existing data. The JDBC implementations use `javax.sql.DataSource` for
|
||||
database access and fastjson2 (2.0.65) as the `reference_json`/`result_json`
|
||||
codec; the embedding service owns the connection pool and schema lifecycle.
|
||||
`JdbcRuntimeBindingRepository` additionally requires a `SecretProtector`
|
||||
@@ -102,7 +102,7 @@ publications. After records are removed, callers must not reuse expired
|
||||
Runtime Session identifiers or idempotency keys. A missing execution is not
|
||||
permission to replay it. No permanent execution-key tombstone is retained.
|
||||
|
||||
The Broker and Managed Server share the canonical 32-table V1 baseline. Initialize
|
||||
The Broker and Managed Server share the canonical 38-table V1 baseline. Initialize
|
||||
a fresh database; old catalogs are refused rather than upgraded. Keep the old
|
||||
environment for historical reads and settle its work before cutover. Repeated
|
||||
initialization validates structure without repairing it or changing receipts.
|
||||
|
||||
-12
@@ -134,10 +134,6 @@ public class WebShellAgentController {
|
||||
@PostMapping("/sessions/export/query")
|
||||
public ApiModels.WebShellExportPage export(TenantContext tenant,
|
||||
@Valid @RequestBody ApiModels.WebShellExportRequest request) {
|
||||
if ("messages".equals(request.view())) {
|
||||
return service.messageExportPage(tenant.tenantId(), tenant.actorId(), request.sessionId(),
|
||||
request.cursor(), request.limit() == null ? 100 : request.limit(), false);
|
||||
}
|
||||
return service.exportPage(tenant.tenantId(), tenant.actorId(), request.sessionId(),
|
||||
request.cursor(), request.limit() == null ? 100 : request.limit());
|
||||
}
|
||||
@@ -179,10 +175,6 @@ public class WebShellAgentController {
|
||||
@PostMapping("/transcript/query")
|
||||
public WebShellTranscript transcript(TenantContext tenant,
|
||||
@Valid @RequestBody WebShellTranscriptRequest request) {
|
||||
if ("messages".equals(request.view())) {
|
||||
return service.messageTranscript(tenant.tenantId(), tenant.actorId(), request.sessionId(),
|
||||
request.cursor(), request.limit() == null ? 100 : request.limit());
|
||||
}
|
||||
return service.transcript(tenant.tenantId(), tenant.actorId(),
|
||||
request.sessionId(),
|
||||
request.cursor(),
|
||||
@@ -192,10 +184,6 @@ public class WebShellAgentController {
|
||||
@PostMapping("/transcript/items/query")
|
||||
public WebShellExportPage recentTranscriptItems(TenantContext tenant,
|
||||
@Valid @RequestBody WebShellExportRequest request) {
|
||||
if ("messages".equals(request.view())) {
|
||||
return service.messageExportPage(tenant.tenantId(), tenant.actorId(), request.sessionId(),
|
||||
request.cursor(), request.limit() == null ? 100 : request.limit(), true);
|
||||
}
|
||||
return service.recentTranscriptItems(tenant.tenantId(), tenant.actorId(), request.sessionId(),
|
||||
request.cursor(), request.limit() == null ? 100 : request.limit());
|
||||
}
|
||||
|
||||
+2
-10
@@ -439,11 +439,7 @@ public final class ApiModels {
|
||||
public record WebShellSessionSkills(List<WebShellSkill> skills) { }
|
||||
|
||||
public record WebShellTranscriptRequest(@NotBlank String sessionId,
|
||||
String cursor, Integer limit,
|
||||
@Pattern(regexp = "messages") String view) {
|
||||
public WebShellTranscriptRequest(String sessionId, String cursor, Integer limit) {
|
||||
this(sessionId, cursor, limit, null);
|
||||
}
|
||||
@Size(max = 4096) String cursor, @Min(1) @Max(100) Integer limit) {
|
||||
}
|
||||
|
||||
public record WebShellStreamRequest(@NotBlank String sessionId,
|
||||
@@ -1065,11 +1061,7 @@ public final class ApiModels {
|
||||
}
|
||||
|
||||
public record WebShellExportRequest(@NotBlank String sessionId,
|
||||
@Size(max = 4096) String cursor, @Min(1) @Max(100) Integer limit,
|
||||
@Pattern(regexp = "messages") String view) {
|
||||
public WebShellExportRequest(String sessionId, String cursor, Integer limit) {
|
||||
this(sessionId, cursor, limit, null);
|
||||
}
|
||||
@Size(max = 4096) String cursor, @Min(1) @Max(100) Integer limit) {
|
||||
}
|
||||
|
||||
public record WebShellExportPage(List<WebShellItem> items, String nextCursor,
|
||||
|
||||
+3
-12
@@ -99,11 +99,9 @@ public final class StoreModels {
|
||||
}
|
||||
|
||||
/**
|
||||
* Events at or below {@code floorSequence} may be pruned. A client that
|
||||
* falls below it reloads the Snapshot, which covers events through
|
||||
* {@code snapshotThroughSequence}. A stream reconciliation discards the
|
||||
* Snapshot without lowering the floor, so the floor can exceed the
|
||||
* coverage while the Items rebuild; the cursor is still served then.
|
||||
* Events at or below the floor may be pruned. Resynchronization reads the
|
||||
* message projection through snapshotThroughSequence, a retained wire name.
|
||||
* Reads wait when projection rebuild temporarily falls behind the floor.
|
||||
*/
|
||||
public record ReplayWindow(long floorSequence,
|
||||
long snapshotThroughSequence) {
|
||||
@@ -320,16 +318,9 @@ public final class StoreModels {
|
||||
long updatedAt, long revision, List<ItemPartRecord> content) {
|
||||
}
|
||||
|
||||
public record SnapshotPage(List<ItemRecord> items, boolean hasMore, String nextCursor, long coveredSequence) {}
|
||||
|
||||
public record MessagePage(List<ItemRecord> items, List<EventRecord> events,
|
||||
boolean hasMore, String nextCursor, long coveredSequence, long throughSequence) {}
|
||||
|
||||
public record SnapshotRecord(String tenantId, String sessionId,
|
||||
long version, long coveredSequence, List<ItemRecord> items,
|
||||
long createdAt, long updatedAt) {
|
||||
}
|
||||
|
||||
public record MaterializationTarget(String tenantId, String sessionId) {
|
||||
}
|
||||
|
||||
|
||||
+45
-145
@@ -41,7 +41,6 @@ import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionMutationComma
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionMutationKind;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionPage;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SnapshotRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.TurnPage;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.TurnRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.TurnSummary;
|
||||
@@ -107,14 +106,7 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
private static final TypeReference<Map<String, Object>> MAP_TYPE =
|
||||
new TypeReference<>() {
|
||||
};
|
||||
private static final TypeReference<List<ItemRecord>> ITEMS_TYPE =
|
||||
new TypeReference<>() {
|
||||
};
|
||||
private static final TypeReference<List<ItemPartRecord>> PARTS_TYPE = new TypeReference<>() {};
|
||||
// Between full rewrites the snapshot may lag the projection by this many
|
||||
// events or this long; the batch that ends a Turn rewrites it anyway.
|
||||
public static final int SNAPSHOT_REFRESH_EVENTS = 1000;
|
||||
private static final long SNAPSHOT_REFRESH_MILLIS = 5000;
|
||||
private static final long MESSAGE_PAGE_BYTES = 8L * 1024 * 1024;
|
||||
private static final String INSERT_EVENT = "INSERT INTO ma_public_event"
|
||||
+ " (tenant_id, session_id, sequence_id, event_id, turn_id,"
|
||||
@@ -149,7 +141,6 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
|
||||
private final ObjectMapper objectMapper;
|
||||
private final Clock clock;
|
||||
private final ManagedSnapshotStore snapshots;
|
||||
private final String approvalMode;
|
||||
private final CommittedEventPublisher eventPublisher;
|
||||
private final ManagedWorkspaceRegistry workspaces;
|
||||
@@ -353,7 +344,6 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
this.creationSkills = new ManagedSessionSkills(jdbc, objectMapper,
|
||||
new RequestDigests(), null);
|
||||
this.clock = clock;
|
||||
this.snapshots = new ManagedSnapshotStore(jdbc, objectMapper, clock, this::snapshotItems);
|
||||
this.approvalMode =
|
||||
properties.getHarness().getApprovalMode().toLowerCase(Locale.ROOT);
|
||||
this.eventPublisher = eventPublisher;
|
||||
@@ -5320,33 +5310,19 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
return new EventPage(List.copyOf(rows), hasMore);
|
||||
}
|
||||
|
||||
public Optional<SnapshotRecord> findSnapshot(String tenantId,
|
||||
String sessionId) {
|
||||
requireSession(tenantId, sessionId);
|
||||
return snapshots.latest(tenantId, sessionId);
|
||||
}
|
||||
|
||||
@Override
|
||||
public StoreModels.SnapshotPage findSnapshotPage(String tenantId, String sessionId,
|
||||
String cursor, long afterSequence, int limit) {
|
||||
requireSession(tenantId, sessionId);
|
||||
ManagedSnapshotStore.Page page = snapshots.page(tenantId, sessionId, cursor, afterSequence, limit);
|
||||
return new StoreModels.SnapshotPage(page.items(), page.hasMore(), page.nextCursor(), page.coveredSequence());
|
||||
}
|
||||
|
||||
@Override
|
||||
public StoreModels.SnapshotPage findRecentSnapshotPage(String tenantId, String sessionId,
|
||||
String cursor, int limit) {
|
||||
requireSession(tenantId, sessionId);
|
||||
ManagedSnapshotStore.Page page = snapshots.recentPage(tenantId, sessionId, cursor, limit);
|
||||
return new StoreModels.SnapshotPage(page.items(), page.hasMore(), page.nextCursor(), page.coveredSequence());
|
||||
@Transactional(transactionManager = "managedAgentTransactionManager", readOnly = true,
|
||||
isolation = org.springframework.transaction.annotation.Isolation.REPEATABLE_READ)
|
||||
public StoreModels.MessagePage findMessagePage(String tenantId, String sessionId,
|
||||
String cursor, int limit, boolean backward) {
|
||||
return findMessagePage(tenantId, sessionId, cursor, limit, backward, 0);
|
||||
}
|
||||
|
||||
@Override
|
||||
@Transactional(transactionManager = "managedAgentTransactionManager", readOnly = true,
|
||||
isolation = org.springframework.transaction.annotation.Isolation.REPEATABLE_READ)
|
||||
public StoreModels.MessagePage findMessagePage(String tenantId, String sessionId,
|
||||
String cursor, int limit, boolean backward) {
|
||||
String cursor, int limit, boolean backward, long afterSequence) {
|
||||
requireSession(tenantId, sessionId);
|
||||
ManagedLegacySessionGuard.requireLegacyMutation(Boolean.TRUE.equals(jdbc.queryForObject(
|
||||
"SELECT csi_guard FROM ma_session WHERE tenant_id = ? AND session_id = ?",
|
||||
@@ -5355,6 +5331,10 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
throw new ManagedAgentException(HttpStatus.BAD_REQUEST, "invalid_limit",
|
||||
"Limit must be between 1 and 100.");
|
||||
}
|
||||
if (afterSequence < 0) {
|
||||
throw new ManagedAgentException(HttpStatus.BAD_REQUEST, "invalid_event_cursor",
|
||||
"Item sequence must be non-negative.");
|
||||
}
|
||||
var progress = jdbc.queryForMap("SELECT covered_sequence, projection_version, last_sequence, replay_floor_sequence FROM ma_event_stream"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND stream_kind = 'SESSION' AND stream_id = X''",
|
||||
tenantId, sessionId);
|
||||
@@ -5374,18 +5354,25 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
}
|
||||
String comparison = backward ? "<" : ">";
|
||||
String order = backward ? " DESC" : " ASC";
|
||||
boolean numericBoundary = afterSequence > 0 && org.apache.commons.lang3.StringUtils.isBlank(cursor);
|
||||
List<Object> pageArguments = new ArrayList<>(List.of(scope, tenantId, sessionId, position.through()));
|
||||
if (numericBoundary) {
|
||||
pageArguments.add(afterSequence);
|
||||
}
|
||||
pageArguments.addAll(List.of(position.sequence(), position.sequence(),
|
||||
com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaKeys.identity(position.item()), limit + 1));
|
||||
List<MessageSize> candidates = jdbc.query(
|
||||
"SELECT m.item_id, m.first_sequence, OCTET_LENGTH(m.attributes_json) +"
|
||||
+ " OCTET_LENGTH(m.parts_json) + COALESCE(OCTET_LENGTH(r.inline_bytes), 0) AS byte_length"
|
||||
+ " FROM ma_message m LEFT JOIN ma_resource r ON r.resource_key = m.body_resource_key"
|
||||
+ " AND r.session_scope_key = ? AND r.tenant_id = m.tenant_id AND r.session_id = m.session_id"
|
||||
+ " WHERE m.tenant_id = ? AND m.session_id = ? AND m.first_sequence <= ?"
|
||||
+ (numericBoundary ? " AND m.first_sequence > ?" : "")
|
||||
+ " AND (m.first_sequence " + comparison + " ? OR (m.first_sequence = ? AND m.item_id "
|
||||
+ comparison + " ?)) ORDER BY m.first_sequence" + order + ", m.item_id" + order + " LIMIT ?",
|
||||
(row, index) -> new MessageSize(row.getString("item_id"), row.getLong("first_sequence"),
|
||||
row.getLong("byte_length")),
|
||||
scope, tenantId, sessionId, position.through(), position.sequence(), position.sequence(),
|
||||
com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaKeys.identity(position.item()), limit + 1);
|
||||
pageArguments.toArray());
|
||||
long bytes = 0;
|
||||
List<MessageSize> selected = new ArrayList<>();
|
||||
for (MessageSize candidate : candidates) {
|
||||
@@ -5504,8 +5491,8 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
|
||||
private record EventSize(long sequence, long bytes) {}
|
||||
|
||||
/** The snapshot's covered sequence of each given Session, in one read. */
|
||||
public Map<String, Long> findSnapshotCoveredSequences(String tenantId,
|
||||
/** The message projection coverage of each Session, in one read. */
|
||||
public Map<String, Long> findMessageCoveredSequences(String tenantId,
|
||||
List<String> sessionIds) {
|
||||
if (sessionIds.isEmpty()) {
|
||||
return Map.of();
|
||||
@@ -5517,9 +5504,8 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
RowCallbackHandler reader = row -> result.put(
|
||||
row.getString("session_id"), row.getLong("covered_sequence"));
|
||||
jdbc.query(
|
||||
"SELECT s.session_id, p.covered_sequence FROM ma_event_stream s JOIN ma_snapshot p"
|
||||
+ " ON p.tenant_id = s.tenant_id AND p.session_id = s.session_id AND"
|
||||
+ " p.snapshot_id = s.current_snapshot_id WHERE s.stream_kind = 'SESSION' AND"
|
||||
"SELECT s.session_id, s.covered_sequence FROM ma_event_stream s"
|
||||
+ " WHERE s.stream_kind = 'SESSION' AND"
|
||||
+ " s.stream_id = X'' AND s.tenant_id = ? AND s.session_id IN ("
|
||||
+ placeholders(sessionIds.size())
|
||||
+ ")",
|
||||
@@ -5531,10 +5517,8 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
public ReplayWindow findReplayWindow(String tenantId, String sessionId) {
|
||||
List<ReplayWindow> rows =
|
||||
jdbc.query(
|
||||
"SELECT s.replay_floor_sequence, p.covered_sequence FROM ma_event_stream s"
|
||||
+ " LEFT JOIN ma_snapshot p ON p.tenant_id = s.tenant_id AND"
|
||||
+ " p.session_id = s.session_id AND p.snapshot_id ="
|
||||
+ " s.current_snapshot_id WHERE s.tenant_id = ? AND s.session_id = ?"
|
||||
"SELECT s.replay_floor_sequence, s.covered_sequence FROM ma_event_stream s"
|
||||
+ " WHERE s.tenant_id = ? AND s.session_id = ?"
|
||||
+ " AND s.stream_kind = 'SESSION' AND s.stream_id = X''",
|
||||
(result, row) ->
|
||||
new ReplayWindow(
|
||||
@@ -5551,36 +5535,30 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
|
||||
public List<ReplayFloorTarget> findReplayFloorTargets(int limit) {
|
||||
return jdbc.query(
|
||||
"SELECT s.tenant_id, s.session_id FROM ma_event_stream s JOIN ma_snapshot p ON p.tenant_id = s.tenant_id AND p.session_id = s.session_id AND p.snapshot_id = s.current_snapshot_id WHERE s.stream_kind = 'SESSION' AND s.stream_id = X'' AND EXISTS (SELECT 1 FROM ma_session private_scope WHERE private_scope.tenant_id = s.tenant_id AND private_scope.session_id = s.session_id AND private_scope.csi_guard = FALSE) AND p.covered_sequence > s.replay_floor_sequence AND NOT EXISTS (SELECT 1 FROM ma_snapshot older WHERE older.tenant_id = s.tenant_id AND older.session_id = s.session_id AND older.state = 'PUBLISHED' AND older.retain_until > ? AND older.covered_sequence <= s.replay_floor_sequence) AND NOT EXISTS (SELECT 1 FROM ma_turn t JOIN ma_public_event e ON e.tenant_id = t.tenant_id AND e.session_id = t.session_id AND e.turn_id = t.turn_id WHERE t.tenant_id = s.tenant_id AND t.session_id = s.session_id AND t.status IN ('ACCEPTED', 'RUNNING', 'CANCELLING') AND e.stream_kind = 'SESSION' AND e.stream_id = X'' AND e.event_type = 'turn.accepted' AND e.sequence_id <= s.replay_floor_sequence + 1) ORDER BY s.gmt_modified ASC LIMIT ?",
|
||||
(row, index) ->
|
||||
new ReplayFloorTarget(
|
||||
row.getString("tenant_id"), row.getString("session_id")),
|
||||
clock.millis(),
|
||||
"SELECT s.tenant_id, s.session_id FROM ma_event_stream s"
|
||||
+ " WHERE s.stream_kind = 'SESSION' AND s.stream_id = X''"
|
||||
+ " AND s.covered_sequence > s.replay_floor_sequence"
|
||||
+ " AND EXISTS (SELECT 1 FROM ma_session private_scope WHERE"
|
||||
+ " private_scope.tenant_id = s.tenant_id AND private_scope.session_id = s.session_id"
|
||||
+ " AND private_scope.csi_guard = FALSE)"
|
||||
+ " AND NOT EXISTS (SELECT 1 FROM ma_turn t JOIN ma_public_event e ON"
|
||||
+ " e.tenant_id = t.tenant_id AND e.session_id = t.session_id AND e.turn_id = t.turn_id"
|
||||
+ " WHERE t.tenant_id = s.tenant_id AND t.session_id = s.session_id"
|
||||
+ " AND t.status IN ('ACCEPTED', 'RUNNING', 'CANCELLING')"
|
||||
+ " AND e.stream_kind = 'SESSION' AND e.stream_id = X''"
|
||||
+ " AND e.event_type = 'turn.accepted' AND e.sequence_id <= s.replay_floor_sequence + 1)"
|
||||
+ " ORDER BY s.gmt_modified ASC LIMIT ?",
|
||||
(row, index) -> new ReplayFloorTarget(row.getString("tenant_id"), row.getString("session_id")),
|
||||
limit);
|
||||
}
|
||||
|
||||
/**
|
||||
* Raises the replay floor, never above the Snapshot's covered sequence, so that a client told
|
||||
* to resync can resume from the Snapshot. Nothing prunes events yet; the retention work will
|
||||
* call this before pruning.
|
||||
*/
|
||||
/** Advances only through readable message coverage, retaining active-Turn evidence. */
|
||||
@Transactional
|
||||
public ReplayWindow advanceReplayFloor(String tenantId, String sessionId,
|
||||
long floorSequence) {
|
||||
requireLegacySessionForUpdate(tenantId, sessionId);
|
||||
ReplayWindow window = findReplayWindow(tenantId, sessionId);
|
||||
Long protectedSequence =
|
||||
jdbc.queryForObject(
|
||||
"SELECT MIN(covered_sequence) FROM ma_snapshot WHERE tenant_id = ? AND"
|
||||
+ " session_id = ? AND state = 'PUBLISHED' AND retain_until > ?",
|
||||
Long.class,
|
||||
tenantId,
|
||||
sessionId,
|
||||
clock.millis());
|
||||
long floor = Math.min(floorSequence, window.snapshotThroughSequence());
|
||||
if (protectedSequence != null) {
|
||||
floor = Math.min(floor, protectedSequence);
|
||||
}
|
||||
Long activeStart =
|
||||
jdbc.queryForObject(
|
||||
"SELECT MIN(e.sequence_id) FROM ma_public_event e JOIN ma_turn t ON"
|
||||
@@ -5608,16 +5586,11 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
}
|
||||
|
||||
public List<MaterializationTarget> findMaterializationTargets(int limit) {
|
||||
// The second disjunct re-selects a session whose drained batch
|
||||
// deferred the snapshot rewrite, once the deferral marker is old
|
||||
// enough; without it a caught-up idle session would stay stale
|
||||
// forever.
|
||||
return jdbc.query(
|
||||
"SELECT s.tenant_id, s.session_id FROM ma_session s JOIN ma_event_stream p ON p.tenant_id = s.tenant_id AND p.session_id = s.session_id AND p.stream_kind = 'SESSION' AND p.stream_id = X'' WHERE s.session_kind = 'PUBLIC' AND s.csi_guard = FALSE AND s.retired_at IS NULL AND s.status <> 'DELETED' AND p.projection_pending = TRUE AND (p.last_sequence > p.covered_sequence OR (p.snapshot_stale_since IS NOT NULL AND p.snapshot_stale_since <= ?)) ORDER BY p.gmt_modified ASC LIMIT ?",
|
||||
"SELECT s.tenant_id, s.session_id FROM ma_session s JOIN ma_event_stream p ON p.tenant_id = s.tenant_id AND p.session_id = s.session_id AND p.stream_kind = 'SESSION' AND p.stream_id = X'' WHERE s.session_kind = 'PUBLIC' AND s.csi_guard = FALSE AND s.retired_at IS NULL AND s.status <> 'DELETED' AND p.projection_pending = TRUE ORDER BY p.gmt_modified ASC LIMIT ?",
|
||||
(result, row) ->
|
||||
new MaterializationTarget(
|
||||
result.getString("tenant_id"), result.getString("session_id")),
|
||||
clock.millis() - SNAPSHOT_REFRESH_MILLIS,
|
||||
limit);
|
||||
}
|
||||
|
||||
@@ -5645,9 +5618,6 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
covered,
|
||||
limit);
|
||||
if (events.isEmpty()) {
|
||||
if (snapshotRefreshRequired(tenantId, sessionId, covered, clock.millis(), false)) {
|
||||
snapshots.afterCommit(tenantId, sessionId);
|
||||
}
|
||||
return new MaterializationResult(false, covered);
|
||||
}
|
||||
long expected = covered + 1;
|
||||
@@ -5717,20 +5687,13 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
flushParts(tenantId, sessionId, projectedParts);
|
||||
long nextCovered = events.get(events.size() - 1).sequence();
|
||||
long now = clock.millis();
|
||||
boolean refresh = snapshotRefreshRequired(tenantId, sessionId, nextCovered, now,
|
||||
events.stream().anyMatch(EventRecord::terminal));
|
||||
jdbc.update(
|
||||
"UPDATE ma_event_stream SET covered_sequence = ?, gmt_modified = ?,"
|
||||
+ " snapshot_stale_since = COALESCE(snapshot_stale_since, ?) WHERE tenant_id ="
|
||||
+ " ? AND session_id = ? AND stream_kind = 'SESSION' AND stream_id = X''",
|
||||
"UPDATE ma_event_stream SET covered_sequence = ?, gmt_modified = ?"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND stream_kind = 'SESSION' AND stream_id = X''",
|
||||
nextCovered,
|
||||
now,
|
||||
now,
|
||||
tenantId,
|
||||
sessionId);
|
||||
if (refresh) {
|
||||
snapshots.afterCommit(tenantId, sessionId);
|
||||
}
|
||||
return new MaterializationResult(true, nextCovered);
|
||||
}
|
||||
|
||||
@@ -6400,15 +6363,11 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
flushParts(tenantId, sessionId, parts);
|
||||
jdbc.update(
|
||||
"UPDATE ma_event_stream SET projection_version = projection_version + 1,"
|
||||
+ " snapshot_stale_since = ?, gmt_modified = ? WHERE tenant_id = ? AND session_id"
|
||||
+ " = ? AND stream_kind = 'SESSION' AND stream_id = X''",
|
||||
now,
|
||||
+ " gmt_modified = ? WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " AND stream_kind = 'SESSION' AND stream_id = X''",
|
||||
now,
|
||||
tenantId,
|
||||
sessionId);
|
||||
if (covered > 0) {
|
||||
snapshots.afterCommit(tenantId, sessionId);
|
||||
}
|
||||
}
|
||||
|
||||
// A Harness source key ends in the journal sequence of its journal event;
|
||||
@@ -6773,37 +6732,6 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
"The Session was not found."));
|
||||
}
|
||||
|
||||
private boolean snapshotRefreshRequired(String tenantId, String sessionId, long covered, long now, boolean terminal) {
|
||||
var published =
|
||||
jdbc.query(
|
||||
"SELECT p.published_at, p.covered_sequence, p.projection_version,"
|
||||
+ " s.projection_version AS current_projection FROM ma_event_stream s"
|
||||
+ " LEFT JOIN ma_snapshot p ON p.tenant_id = s.tenant_id AND"
|
||||
+ " p.session_id = s.session_id AND p.snapshot_id ="
|
||||
+ " s.current_snapshot_id WHERE s.tenant_id = ? AND s.session_id = ?"
|
||||
+ " AND s.stream_kind = 'SESSION' AND s.stream_id = X''",
|
||||
(row, index) ->
|
||||
new Long[] {
|
||||
nullableLong(row, "published_at"),
|
||||
nullableLong(row, "covered_sequence"),
|
||||
nullableLong(row, "projection_version"),
|
||||
nullableLong(row, "current_projection")
|
||||
},
|
||||
tenantId,
|
||||
sessionId);
|
||||
if (published.isEmpty() || published.getFirst()[0] == null) {
|
||||
return true;
|
||||
}
|
||||
Long[] current = published.getFirst();
|
||||
if (covered <= current[1] && java.util.Objects.equals(current[2], current[3])) {
|
||||
return false;
|
||||
}
|
||||
return terminal
|
||||
|| !java.util.Objects.equals(current[2], current[3])
|
||||
|| now - current[0] >= SNAPSHOT_REFRESH_MILLIS
|
||||
|| covered - current[1] >= SNAPSHOT_REFRESH_EVENTS;
|
||||
}
|
||||
|
||||
private void materializeEvent(EventRecord event, Map<String, List<ItemPartRecord>> projectedParts) {
|
||||
switch (event.type()) {
|
||||
case "turn.accepted", "item.input.added" -> materializeInput(event, projectedParts);
|
||||
@@ -7163,21 +7091,6 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
}
|
||||
}
|
||||
|
||||
private List<ItemRecord> snapshotItems(String tenantId, String sessionId, Runnable checkLease) {
|
||||
return withParts(
|
||||
jdbc.query(
|
||||
"SELECT " + ManagedSchemaProjection.MESSAGE.columns() + " FROM ma_message WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " ORDER BY first_sequence ASC, item_id ASC",
|
||||
(row, index) -> {
|
||||
if (index % 100 == 0 || row.getString("body_resource_key") != null) {
|
||||
checkLease.run();
|
||||
}
|
||||
return itemMapper.mapRow(row, index);
|
||||
},
|
||||
tenantId,
|
||||
sessionId));
|
||||
}
|
||||
|
||||
private List<ItemRecord> allItems(String tenantId, String sessionId) {
|
||||
return withParts(jdbc.query("SELECT " + ManagedSchemaProjection.MESSAGE.columns() + " FROM ma_message WHERE"
|
||||
+ " tenant_id = ? AND session_id = ? ORDER BY"
|
||||
@@ -7735,15 +7648,6 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
}
|
||||
}
|
||||
|
||||
private List<ItemRecord> readItems(String value) {
|
||||
try {
|
||||
return objectMapper.readValue(value, ITEMS_TYPE);
|
||||
} catch (JsonProcessingException error) {
|
||||
throw new IllegalStateException("Stored snapshot is invalid",
|
||||
error);
|
||||
}
|
||||
}
|
||||
|
||||
public static ContextBinding readBinding(java.sql.ResultSet result)
|
||||
throws java.sql.SQLException {
|
||||
String workspaceId = result.getString("workspace_id");
|
||||
@@ -7813,10 +7717,6 @@ public class ManagedAgentStore implements AgentStateStore {
|
||||
return String.join(", ", Collections.nCopies(count, "?"));
|
||||
}
|
||||
|
||||
private record SnapshotState(long version, long coveredSequence,
|
||||
long updatedAt) {
|
||||
}
|
||||
|
||||
private record ItemRow(String tenantId, String sessionId, String itemId,
|
||||
String turnId, String type, String role, String status,
|
||||
Map<String, Object> attributes, long firstSequence,
|
||||
|
||||
+3
-104
@@ -1,6 +1,5 @@
|
||||
package com.alibaba.qwen.code.managedagent.persistence;
|
||||
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaKeys;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaTime;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.transaction.PlatformTransactionManager;
|
||||
@@ -10,10 +9,7 @@ import org.springframework.transaction.support.TransactionTemplate;
|
||||
/** Reclaims released public projection bodies, including those of still-live Sessions. */
|
||||
final class ManagedProjectionResourceCollector {
|
||||
private static final int BATCH = 32;
|
||||
private static final String KINDS =
|
||||
"X'6d616e616765642d6d6573736167652d7061727473',"
|
||||
+ "X'6d616e616765642d736e617073686f742d70616765',"
|
||||
+ "X'6d616e616765642d736e617073686f742d6d616e6966657374'";
|
||||
private static final String KINDS = "X'6d616e616765642d6d6573736167652d7061727473'";
|
||||
private final JdbcTemplate jdbc;
|
||||
private final TransactionTemplate transactions;
|
||||
|
||||
@@ -24,8 +20,7 @@ final class ManagedProjectionResourceCollector {
|
||||
}
|
||||
|
||||
static void release(JdbcTemplate jdbc, String scope, String kind, String owner) {
|
||||
// A builder can still hold this body's old RR view before publishing its snapshot ref.
|
||||
// Mark a candidate here; the collector must wait for that builder's read lease.
|
||||
// Readers can still hold this body's old read view; collection waits for their leases.
|
||||
jdbc.update(
|
||||
"UPDATE ma_resource SET retention_until = " + ManagedSchemaTime.SQL_NOW + " WHERE session_scope_key = ? AND kind IN ("
|
||||
+ KINDS
|
||||
@@ -38,23 +33,8 @@ final class ManagedProjectionResourceCollector {
|
||||
owner);
|
||||
}
|
||||
|
||||
static void removeSnapshot(JdbcTemplate jdbc, String tenant, String session, String id,
|
||||
long version) {
|
||||
String scope = ManagedSessionStore.sessionScopeKey(tenant, session);
|
||||
String owner = ManagedSchemaKeys.key(scope, id);
|
||||
release(jdbc, scope, "SNAPSHOT", owner);
|
||||
jdbc.update(
|
||||
"DELETE FROM ma_resource_ref WHERE session_scope_key = ? AND owner_kind ="
|
||||
+ " 'SNAPSHOT' AND owner_key = ? AND owner_revision = ?",
|
||||
scope, owner, version);
|
||||
jdbc.update(
|
||||
"DELETE FROM ma_snapshot WHERE tenant_id = ? AND session_id = ? AND snapshot_id ="
|
||||
+ " ?",
|
||||
tenant, session, id);
|
||||
}
|
||||
|
||||
boolean runOnce() {
|
||||
boolean changed = expireSnapshots();
|
||||
boolean changed = false;
|
||||
var candidates = jdbc.queryForList(
|
||||
"SELECT resource_key, tenant_id, session_id FROM ma_resource WHERE retention_until <= " + ManagedSchemaTime.SQL_NOW + " AND NOT EXISTS (SELECT 1 FROM ma_session s WHERE s.tenant_id = ma_resource.tenant_id AND s.session_id = ma_resource.session_id AND s.csi_guard = TRUE) ORDER BY retention_until, resource_key LIMIT "
|
||||
+ BATCH);
|
||||
@@ -67,87 +47,6 @@ final class ManagedProjectionResourceCollector {
|
||||
return changed;
|
||||
}
|
||||
|
||||
private boolean expireSnapshots() {
|
||||
long now = ToolPublicationRetentionStore.now(jdbc);
|
||||
var candidates = jdbc.queryForList(
|
||||
"SELECT tenant_id, session_id, snapshot_id FROM ma_snapshot WHERE state ="
|
||||
+ " 'PUBLISHED' AND retain_until <= ? AND NOT EXISTS (SELECT 1 FROM"
|
||||
+ " ma_session s WHERE s.tenant_id = ma_snapshot.tenant_id AND"
|
||||
+ " s.session_id = ma_snapshot.session_id AND s.csi_guard = TRUE) ORDER"
|
||||
+ " BY retain_until LIMIT "
|
||||
+ BATCH, now);
|
||||
boolean changed = false;
|
||||
for (var candidate : candidates) {
|
||||
changed |=
|
||||
Boolean.TRUE.equals(
|
||||
ManagedSessionStore.skipContended(
|
||||
transactions,
|
||||
status -> {
|
||||
String tenant = (String) candidate.get("tenant_id");
|
||||
String session = (String) candidate.get("session_id");
|
||||
String id = (String) candidate.get("snapshot_id");
|
||||
ManagedSessionStore.lockBounded(() -> ToolPublicationRetentionStore.lockTenant(
|
||||
jdbc, tenant, ManagedSessionStore.GUARD_ACQUIRE_TIMEOUT_SECONDS));
|
||||
if (ManagedLegacySessionGuard.isPrivate(
|
||||
jdbc, tenant, session)) {
|
||||
return false;
|
||||
}
|
||||
ToolPublicationRetentionStore.lockSession(
|
||||
jdbc, tenant, session);
|
||||
var streams =
|
||||
jdbc.queryForList(
|
||||
"SELECT current_snapshot_id FROM"
|
||||
+ " ma_event_stream WHERE tenant_id = ?"
|
||||
+ " AND session_id = ? AND stream_kind"
|
||||
+ " = 'SESSION' AND stream_id = X'' FOR"
|
||||
+ " UPDATE",
|
||||
tenant,
|
||||
session);
|
||||
var roots =
|
||||
jdbc.queryForList(
|
||||
"SELECT snapshot_version, retain_until FROM"
|
||||
+ " ma_snapshot WHERE tenant_id = ? AND"
|
||||
+ " session_id = ? AND snapshot_id = ?"
|
||||
+ " AND state = 'PUBLISHED' FOR UPDATE",
|
||||
tenant,
|
||||
session,
|
||||
id);
|
||||
long time = ToolPublicationRetentionStore.now(jdbc);
|
||||
if (roots.isEmpty()
|
||||
|| ToolPublicationRetentionStore.number(
|
||||
roots.getFirst(), "retain_until")
|
||||
> time) {
|
||||
return false;
|
||||
}
|
||||
if (!streams.isEmpty()
|
||||
&& id.equals(
|
||||
streams.getFirst()
|
||||
.get("current_snapshot_id"))) {
|
||||
jdbc.update(
|
||||
"UPDATE ma_snapshot SET retain_until = ? WHERE"
|
||||
+ " tenant_id = ? AND session_id = ? AND"
|
||||
+ " snapshot_id = ?",
|
||||
time
|
||||
+ ManagedSnapshotStore
|
||||
.CURSOR_LIFETIME_MILLIS,
|
||||
tenant,
|
||||
session,
|
||||
id);
|
||||
return false;
|
||||
}
|
||||
removeSnapshot(
|
||||
jdbc,
|
||||
tenant,
|
||||
session,
|
||||
id,
|
||||
ToolPublicationRetentionStore.number(
|
||||
roots.getFirst(), "snapshot_version"));
|
||||
return true;
|
||||
}));
|
||||
}
|
||||
return changed;
|
||||
}
|
||||
|
||||
private boolean collect(String tenant, String session, String key) {
|
||||
ManagedSessionStore.lockBounded(() -> ToolPublicationRetentionStore.lockTenant(
|
||||
jdbc, tenant, ManagedSessionStore.GUARD_ACQUIRE_TIMEOUT_SECONDS));
|
||||
|
||||
+2
-34
@@ -39,38 +39,6 @@ final class ManagedSessionRetirementStore {
|
||||
if (!nativeSnapshots.isEmpty()) {
|
||||
return true;
|
||||
}
|
||||
var snapshots =
|
||||
jdbc.queryForList(
|
||||
"SELECT snapshot_id, snapshot_version FROM ma_snapshot WHERE tenant_id = ?"
|
||||
+ " AND session_id = ? ORDER BY snapshot_version LIMIT ? FOR UPDATE",
|
||||
tenant,
|
||||
session,
|
||||
limit);
|
||||
for (var row : snapshots) {
|
||||
String id = (String) row.get("snapshot_id");
|
||||
jdbc.update(
|
||||
"UPDATE ma_event_stream SET current_snapshot_id = NULL WHERE tenant_id = ? AND"
|
||||
+ " session_id = ? AND stream_kind = 'SESSION' AND stream_id = X'' AND"
|
||||
+ " current_snapshot_id = ?",
|
||||
tenant,
|
||||
session,
|
||||
id);
|
||||
jdbc.update(
|
||||
"DELETE FROM ma_resource_ref WHERE session_scope_key = ? AND owner_kind ="
|
||||
+ " 'SNAPSHOT' AND owner_key = ? AND owner_revision = ?",
|
||||
scope,
|
||||
ManagedSchemaKeys.key(scope, id),
|
||||
row.get("snapshot_version"));
|
||||
jdbc.update(
|
||||
"DELETE FROM ma_snapshot WHERE tenant_id = ? AND session_id = ? AND snapshot_id"
|
||||
+ " = ?",
|
||||
tenant,
|
||||
session,
|
||||
id);
|
||||
}
|
||||
if (!snapshots.isEmpty()) {
|
||||
return true;
|
||||
}
|
||||
var messages =
|
||||
jdbc.queryForList(
|
||||
"SELECT item_id FROM ma_message WHERE tenant_id = ? AND session_id = ?"
|
||||
@@ -227,8 +195,8 @@ final class ManagedSessionRetirementStore {
|
||||
return true;
|
||||
}
|
||||
jdbc.update(
|
||||
"UPDATE ma_event_stream SET covered_sequence = last_sequence, snapshot_stale_since"
|
||||
+ " = NULL WHERE tenant_id = ? AND session_id = ? AND stream_kind = 'SESSION'"
|
||||
"UPDATE ma_event_stream SET covered_sequence = last_sequence"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND stream_kind = 'SESSION'"
|
||||
+ " AND stream_id = X''",
|
||||
tenant,
|
||||
session);
|
||||
|
||||
-921
@@ -1,921 +0,0 @@
|
||||
package com.alibaba.qwen.code.managedagent.persistence;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.ItemRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SnapshotRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaKeys;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaProjection;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaTime;
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.apache.commons.codec.digest.HmacAlgorithms;
|
||||
import org.apache.commons.codec.digest.HmacUtils;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.transaction.TransactionDefinition;
|
||||
import org.springframework.transaction.support.TransactionSynchronization;
|
||||
import org.springframework.transaction.support.TransactionSynchronizationManager;
|
||||
import org.springframework.transaction.support.TransactionTemplate;
|
||||
|
||||
import java.security.GeneralSecurityException;
|
||||
import java.time.Clock;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Base64;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
|
||||
/** Immutable pages are built in one read view, then published in a short transaction. */
|
||||
final class ManagedSnapshotStore {
|
||||
private static final java.security.SecureRandom CURSOR_RANDOM = new java.security.SecureRandom();
|
||||
static final long CURSOR_LIFETIME_MILLIS = 30 * 60 * 1000L;
|
||||
private static final int PAGE_ITEMS = 100;
|
||||
private static final int PAGE_BYTES = 128 * 1024;
|
||||
private static final TypeReference<List<SnapshotItem>> ITEMS = new TypeReference<>() {};
|
||||
private final JdbcTemplate jdbc;
|
||||
private final ObjectMapper mapper;
|
||||
private final Clock clock;
|
||||
private final TransactionTemplate view;
|
||||
private final TransactionTemplate publish;
|
||||
private final ToolPublicationRetentionStore retention;
|
||||
private final ItemReader items;
|
||||
|
||||
ManagedSnapshotStore(JdbcTemplate jdbc, ObjectMapper mapper, Clock clock, ItemReader items) {
|
||||
this.jdbc = jdbc;
|
||||
this.mapper = mapper;
|
||||
this.clock = clock;
|
||||
this.items = items;
|
||||
var manager =
|
||||
new DataSourceTransactionManager(
|
||||
java.util.Objects.requireNonNull(jdbc.getDataSource()));
|
||||
view = new TransactionTemplate(manager);
|
||||
view.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
|
||||
view.setIsolationLevel(TransactionDefinition.ISOLATION_REPEATABLE_READ);
|
||||
view.setReadOnly(true);
|
||||
retention = new ToolPublicationRetentionStore(jdbc, manager);
|
||||
publish = new TransactionTemplate(manager);
|
||||
publish.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
|
||||
}
|
||||
|
||||
void afterCommit(String tenant, String session) {
|
||||
if (TransactionSynchronizationManager.isSynchronizationActive()) {
|
||||
TransactionSynchronizationManager.registerSynchronization(
|
||||
new TransactionSynchronization() {
|
||||
@Override
|
||||
public void afterCommit() {
|
||||
build(tenant, session);
|
||||
}
|
||||
});
|
||||
} else {
|
||||
build(tenant, session);
|
||||
}
|
||||
}
|
||||
|
||||
void build(String tenant, String session) {
|
||||
try (var lease = new SnapshotReadLease(tenant, session)) {
|
||||
Draft draft =
|
||||
view.execute(
|
||||
status -> {
|
||||
List<Map<String, Object>> progress =
|
||||
jdbc.queryForList(
|
||||
"SELECT covered_sequence, projection_version FROM"
|
||||
+ " ma_event_stream WHERE tenant_id = ? AND"
|
||||
+ " session_id = ? AND stream_kind = 'SESSION'"
|
||||
+ " AND stream_id = X''",
|
||||
tenant,
|
||||
session);
|
||||
if (progress.isEmpty()) {
|
||||
return null;
|
||||
}
|
||||
long sequence = number(progress.getFirst(), "covered_sequence");
|
||||
int projection =
|
||||
(int) number(progress.getFirst(), "projection_version");
|
||||
List<ItemRecord> rows = items.read(tenant, session, lease::check);
|
||||
lease.check();
|
||||
Map<String, String> bodyKeys = new java.util.HashMap<>();
|
||||
jdbc.query(
|
||||
"SELECT item_id, body_resource_key FROM ma_message WHERE"
|
||||
+ " tenant_id = ? AND session_id = ? AND"
|
||||
+ " body_resource_key IS NOT NULL",
|
||||
(org.springframework.jdbc.core.RowCallbackHandler)
|
||||
row ->
|
||||
bodyKeys.put(
|
||||
row.getString("item_id"),
|
||||
row.getString("body_resource_key")),
|
||||
tenant,
|
||||
session);
|
||||
List<DraftPage> pages = new ArrayList<>();
|
||||
List<SnapshotItem> page = new ArrayList<>();
|
||||
int size = 2;
|
||||
for (ItemRecord item : rows) {
|
||||
if (page.isEmpty()) {
|
||||
lease.check();
|
||||
}
|
||||
String body = bodyKeys.get(item.itemId());
|
||||
ItemRecord descriptor =
|
||||
body == null
|
||||
? item
|
||||
: new ItemRecord(
|
||||
item.tenantId(),
|
||||
item.sessionId(),
|
||||
item.itemId(),
|
||||
item.turnId(),
|
||||
item.type(),
|
||||
item.role(),
|
||||
item.status(),
|
||||
item.attributes(),
|
||||
item.firstSequence(),
|
||||
item.lastSequence(),
|
||||
item.createdAt(),
|
||||
item.updatedAt(),
|
||||
item.revision(),
|
||||
List.of());
|
||||
SnapshotItem frozen = new SnapshotItem(descriptor, body);
|
||||
int itemSize = json(frozen).length;
|
||||
if (!page.isEmpty()
|
||||
&& (page.size() >= PAGE_ITEMS
|
||||
|| size + itemSize > PAGE_BYTES)) {
|
||||
pages.add(draftPage(page));
|
||||
page = new ArrayList<>();
|
||||
size = 2;
|
||||
}
|
||||
page.add(frozen);
|
||||
size += itemSize + 1;
|
||||
}
|
||||
if (!page.isEmpty()) {
|
||||
pages.add(draftPage(page));
|
||||
}
|
||||
return new Draft(
|
||||
sequence,
|
||||
projection,
|
||||
rows.size(),
|
||||
List.copyOf(pages),
|
||||
List.copyOf(new java.util.HashSet<>(bodyKeys.values())));
|
||||
});
|
||||
if (draft == null) {
|
||||
return;
|
||||
}
|
||||
lease.check();
|
||||
publish.executeWithoutResult(
|
||||
status -> {
|
||||
var sessions =
|
||||
jdbc.queryForList(
|
||||
"SELECT retired_at, status FROM ma_session WHERE tenant_id"
|
||||
+ " = ? AND session_id = ? AND session_kind = 'PUBLIC'"
|
||||
+ " FOR UPDATE",
|
||||
tenant,
|
||||
session);
|
||||
if (sessions.size() != 1
|
||||
|| sessions.getFirst().get("retired_at") != null
|
||||
|| "DELETED".equals(sessions.getFirst().get("status"))) {
|
||||
return;
|
||||
}
|
||||
var streams =
|
||||
jdbc.queryForList(
|
||||
"SELECT current_snapshot_id, covered_sequence,"
|
||||
+ " projection_version FROM ma_event_stream WHERE"
|
||||
+ " tenant_id = ? AND session_id = ? AND stream_kind ="
|
||||
+ " 'SESSION' AND stream_id = X'' FOR UPDATE",
|
||||
tenant,
|
||||
session);
|
||||
if (streams.isEmpty()
|
||||
|| number(streams.getFirst(), "projection_version")
|
||||
!= draft.projection()) {
|
||||
return;
|
||||
}
|
||||
Map<String, Object> stream = streams.getFirst();
|
||||
List<Root> roots = current(tenant, session);
|
||||
if (!roots.isEmpty()
|
||||
&& roots.getFirst().projection() == draft.projection()
|
||||
&& roots.getFirst().sequence() >= draft.sequence()) {
|
||||
return;
|
||||
}
|
||||
long version = roots.isEmpty() ? 1 : roots.getFirst().version() + 1;
|
||||
String id = UUID.randomUUID().toString();
|
||||
String scope = ManagedSessionStore.sessionScopeKey(tenant, session);
|
||||
String owner = ManagedSchemaKeys.key(scope, id);
|
||||
long now = clock.millis();
|
||||
for (String body : draft.bodies()) {
|
||||
var available =
|
||||
jdbc.queryForList(
|
||||
"SELECT resource_id FROM ma_resource WHERE resource_key"
|
||||
+ " = ? AND session_scope_key = ? AND state ="
|
||||
+ " 'REFERENCED' AND deletion_operation_key IS NULL"
|
||||
+ " FOR UPDATE",
|
||||
body,
|
||||
scope);
|
||||
if (available.size() != 1) {
|
||||
throw new IllegalStateException(
|
||||
"Snapshot body resource is no longer available");
|
||||
}
|
||||
jdbc.update(
|
||||
"INSERT INTO ma_resource_ref (session_scope_key, owner_kind, owner_key, owner_revision, resource_key, tenant_id, session_id, resource_id, gmt_create) VALUES (?, 'SNAPSHOT', ?, ?, ?, ?, ?, ?, " + ManagedSchemaTime.SQL_NOW + ")",
|
||||
scope,
|
||||
owner,
|
||||
version,
|
||||
body,
|
||||
tenant,
|
||||
session,
|
||||
available.getFirst().get("resource_id"));
|
||||
}
|
||||
for (int page = 0; page < draft.pages().size(); page++) {
|
||||
writeResource(
|
||||
tenant,
|
||||
session,
|
||||
scope,
|
||||
owner,
|
||||
version,
|
||||
pageId(id, page),
|
||||
"managed-snapshot-page",
|
||||
draft.pages().get(page).bytes(),
|
||||
new PageLocation(
|
||||
id,
|
||||
page,
|
||||
draft.pages().get(page).firstSequence(),
|
||||
draft.pages().get(page).lastSequence()));
|
||||
}
|
||||
byte[] manifest =
|
||||
json(
|
||||
new Manifest(
|
||||
1,
|
||||
id,
|
||||
draft.sequence(),
|
||||
draft.projection(),
|
||||
draft.items(),
|
||||
draft.pages().size(),
|
||||
newCursorSecret()));
|
||||
String manifestKey =
|
||||
writeResource(
|
||||
tenant,
|
||||
session,
|
||||
scope,
|
||||
owner,
|
||||
version,
|
||||
"snapshot:" + id + ":manifest",
|
||||
"managed-snapshot-manifest",
|
||||
manifest);
|
||||
jdbc.update(
|
||||
"INSERT INTO ma_snapshot (tenant_id, session_id, snapshot_id,"
|
||||
+ " snapshot_version, covered_sequence, projection_version,"
|
||||
+ " manifest_resource_key, manifest_sha256, item_count, state,"
|
||||
+ " gmt_create, published_at, retain_until) VALUES (?, ?, ?, ?,"
|
||||
+ " ?, ?, ?, ?, ?, 'PUBLISHED', ?, ?, ?)",
|
||||
tenant,
|
||||
session,
|
||||
id,
|
||||
version,
|
||||
draft.sequence(),
|
||||
draft.projection(),
|
||||
manifestKey,
|
||||
WorkspaceEvidenceResources.digest(manifest),
|
||||
draft.items(),
|
||||
now,
|
||||
now,
|
||||
now + CURSOR_LIFETIME_MILLIS);
|
||||
jdbc.update(
|
||||
"UPDATE ma_event_stream SET current_snapshot_id = ?,"
|
||||
+ " snapshot_stale_since = CASE WHEN covered_sequence > ? THEN"
|
||||
+ " ? ELSE NULL END WHERE tenant_id = ? AND session_id = ? AND"
|
||||
+ " stream_kind = 'SESSION' AND stream_id = X''",
|
||||
id,
|
||||
draft.sequence(),
|
||||
now,
|
||||
tenant,
|
||||
session);
|
||||
expire(tenant, session, id, now);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
Optional<SnapshotRecord> latest(String tenant, String session) {
|
||||
try (var lease = new SnapshotReadLease(tenant, session)) {
|
||||
Root root =
|
||||
publish.execute(
|
||||
status -> {
|
||||
lockStream(tenant, session);
|
||||
List<Root> roots = current(tenant, session);
|
||||
if (roots.isEmpty()) {
|
||||
return null;
|
||||
}
|
||||
protect(roots.getFirst(), tenant, session);
|
||||
return roots.getFirst();
|
||||
});
|
||||
if (root == null) {
|
||||
return Optional.empty();
|
||||
}
|
||||
Optional<SnapshotRecord> result = view.execute(
|
||||
status -> {
|
||||
Manifest manifest = manifest(root, tenant, session);
|
||||
List<ItemRecord> rows = new ArrayList<>();
|
||||
for (int page = 0; page < manifest.pages(); page++) {
|
||||
lease.check();
|
||||
rows.addAll(readPage(tenant, session, root.id(), page));
|
||||
}
|
||||
lease.check();
|
||||
if (rows.size() != manifest.items()) {
|
||||
throw new IllegalStateException(
|
||||
"Snapshot item count differs from manifest");
|
||||
}
|
||||
return Optional.of(
|
||||
new SnapshotRecord(
|
||||
tenant,
|
||||
session,
|
||||
root.version(),
|
||||
root.sequence(),
|
||||
List.copyOf(rows),
|
||||
root.created(),
|
||||
root.created()));
|
||||
});
|
||||
lease.check();
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
Page page(String tenant, String session, String cursor, long afterSequence, int limit) {
|
||||
return page(tenant, session, cursor, afterSequence, limit, false);
|
||||
}
|
||||
|
||||
Page recentPage(String tenant, String session, String cursor, int limit) {
|
||||
return page(tenant, session, cursor, 0, limit, true);
|
||||
}
|
||||
|
||||
private Page page(String tenant, String session, String cursor, long afterSequence, int limit,
|
||||
boolean backward) {
|
||||
if (backward && (limit < 1 || limit > 100)) {
|
||||
throw new ManagedAgentException(HttpStatus.BAD_REQUEST, "invalid_limit", "Limit must be between 1 and 100.");
|
||||
}
|
||||
try (var lease = new SnapshotReadLease(tenant, session)) {
|
||||
Start start =
|
||||
publish.execute(
|
||||
status -> {
|
||||
Root root;
|
||||
Cursor position;
|
||||
if (StringUtils.isBlank(cursor)) {
|
||||
lockStream(tenant, session);
|
||||
List<Root> roots = current(tenant, session);
|
||||
if (roots.isEmpty()) {
|
||||
return null;
|
||||
}
|
||||
root = roots.getFirst();
|
||||
long expires = protect(root, tenant, session);
|
||||
position =
|
||||
new Cursor(
|
||||
backward ? 2 : 1,
|
||||
ManagedSessionStore.sessionScopeKey(
|
||||
tenant, session),
|
||||
root.id(),
|
||||
root.version(),
|
||||
root.projection(),
|
||||
0,
|
||||
0,
|
||||
expires,
|
||||
afterSequence);
|
||||
} else {
|
||||
position = decode(cursor);
|
||||
if (position.version() != (backward ? 2 : 1)
|
||||
|| position.scope() == null
|
||||
|| !position.scope()
|
||||
.equals(
|
||||
ManagedSessionStore.sessionScopeKey(
|
||||
tenant, session))) {
|
||||
throw invalidCursor();
|
||||
}
|
||||
if (position.expires() <= clock.millis()) {
|
||||
throw expiredCursor();
|
||||
}
|
||||
var roots =
|
||||
jdbc.query(
|
||||
"SELECT " + ManagedSchemaProjection.SNAPSHOT.columns() + " FROM ma_snapshot WHERE tenant_id = ?"
|
||||
+ " AND session_id = ? AND snapshot_id = ?"
|
||||
+ " AND state = 'PUBLISHED' AND"
|
||||
+ " retain_until >= ?",
|
||||
(row, i) -> root(row),
|
||||
tenant,
|
||||
session,
|
||||
position.snapshot(),
|
||||
position.expires());
|
||||
if (roots.size() != 1) {
|
||||
throw expiredCursor();
|
||||
}
|
||||
root = roots.getFirst();
|
||||
if (root.version() != position.snapshotVersion()
|
||||
|| root.projection() != position.projection()) {
|
||||
throw invalidCursor();
|
||||
}
|
||||
}
|
||||
return new Start(root, position);
|
||||
});
|
||||
if (start == null) {
|
||||
return new Page(List.of(), false, null, 0);
|
||||
}
|
||||
Page result = view.execute(
|
||||
status -> {
|
||||
Root root = start.root();
|
||||
Cursor position = start.cursor();
|
||||
Manifest manifest = manifest(root, tenant, session);
|
||||
if (StringUtils.isNotBlank(cursor)) {
|
||||
verifyCursor(cursor, manifest);
|
||||
}
|
||||
if (backward) {
|
||||
return recentItems(tenant, session, start, manifest, cursor, limit, lease);
|
||||
}
|
||||
if (position.page() < 0
|
||||
|| position.page() > manifest.pages()
|
||||
|| position.offset() < 0) {
|
||||
throw invalidCursor();
|
||||
}
|
||||
int page = position.page();
|
||||
int offset = position.offset();
|
||||
if (StringUtils.isBlank(cursor)) {
|
||||
List<Integer> starts =
|
||||
jdbc.query(
|
||||
"SELECT snapshot_page_ordinal FROM ma_resource WHERE"
|
||||
+ " session_scope_key = ? AND snapshot_id = ? AND"
|
||||
+ " snapshot_page_last_sequence > ? AND tenant_id ="
|
||||
+ " ? AND session_id = ? AND kind = ? AND"
|
||||
+ " schema_version = 1 AND storage_kind ="
|
||||
+ " 'MYSQL_INLINE' AND state = 'REFERENCED' AND"
|
||||
+ " deletion_operation_key IS NULL ORDER BY"
|
||||
+ " snapshot_page_last_sequence,"
|
||||
+ " snapshot_page_ordinal LIMIT 1",
|
||||
(row, index) -> row.getInt("snapshot_page_ordinal"),
|
||||
ManagedSessionStore.sessionScopeKey(tenant, session),
|
||||
root.id(),
|
||||
position.afterSequence(),
|
||||
tenant,
|
||||
session,
|
||||
"managed-snapshot-page");
|
||||
if (starts.isEmpty()) {
|
||||
lease.check();
|
||||
return new Page(List.of(), false, null, root.sequence());
|
||||
}
|
||||
page = starts.getFirst();
|
||||
if (page < 0 || page >= manifest.pages()) {
|
||||
throw new IllegalStateException(
|
||||
"Snapshot page directory is inconsistent");
|
||||
}
|
||||
}
|
||||
List<ItemRecord> selected = new ArrayList<>();
|
||||
while (page < manifest.pages() && selected.size() < limit + 1) {
|
||||
lease.check();
|
||||
List<ItemRecord> rows = readPage(tenant, session, root.id(), page);
|
||||
if (offset > rows.size()) {
|
||||
throw invalidCursor();
|
||||
}
|
||||
while (offset < rows.size() && selected.size() < limit + 1) {
|
||||
ItemRecord item = rows.get(offset++);
|
||||
if (item.firstSequence() > position.afterSequence()) {
|
||||
selected.add(item);
|
||||
}
|
||||
}
|
||||
if (offset == rows.size()) {
|
||||
page++;
|
||||
offset = 0;
|
||||
}
|
||||
}
|
||||
boolean more = selected.size() > limit;
|
||||
String next = null;
|
||||
if (more) {
|
||||
// The look-ahead item belongs to the next page request.
|
||||
if (offset == 0) {
|
||||
page--;
|
||||
offset = readPage(tenant, session, root.id(), page).size();
|
||||
}
|
||||
offset--;
|
||||
next =
|
||||
encode(
|
||||
new Cursor(
|
||||
1,
|
||||
position.scope(),
|
||||
root.id(),
|
||||
root.version(),
|
||||
root.projection(),
|
||||
page,
|
||||
offset,
|
||||
position.expires(),
|
||||
position.afterSequence()),
|
||||
manifest);
|
||||
selected.removeLast();
|
||||
}
|
||||
lease.check();
|
||||
return new Page(List.copyOf(selected), more, next, root.sequence());
|
||||
});
|
||||
lease.check();
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
private Page recentItems(String tenant, String session, Start start, Manifest manifest,
|
||||
String cursor, int limit, SnapshotReadLease lease) {
|
||||
Root root = start.root();
|
||||
Cursor position = start.cursor();
|
||||
int page = StringUtils.isBlank(cursor) ? manifest.pages() - 1 : position.page();
|
||||
int offset = StringUtils.isBlank(cursor) ? -1 : position.offset();
|
||||
if (page < -1 || page >= manifest.pages() || offset < -1
|
||||
|| StringUtils.isNotBlank(cursor) && (page < 0 || offset < 0)) {
|
||||
throw invalidCursor();
|
||||
}
|
||||
List<ItemRecord> selected = new ArrayList<>();
|
||||
String next = null;
|
||||
while (page >= 0) {
|
||||
lease.check();
|
||||
List<ItemRecord> rows = readPage(tenant, session, root.id(), page);
|
||||
if (offset == -1) {
|
||||
offset = rows.size();
|
||||
}
|
||||
if (offset > rows.size()) {
|
||||
throw invalidCursor();
|
||||
}
|
||||
while (offset > 0) {
|
||||
ItemRecord item = rows.get(--offset);
|
||||
if (selected.size() == limit) {
|
||||
next = encode(new Cursor(2, position.scope(), root.id(), root.version(), root.projection(),
|
||||
page, offset + 1, position.expires(), 0), manifest);
|
||||
break;
|
||||
}
|
||||
selected.add(item);
|
||||
}
|
||||
if (next != null) {
|
||||
break;
|
||||
}
|
||||
page--;
|
||||
offset = -1;
|
||||
}
|
||||
java.util.Collections.reverse(selected);
|
||||
lease.check();
|
||||
return new Page(List.copyOf(selected), next != null, next, root.sequence());
|
||||
}
|
||||
|
||||
@FunctionalInterface
|
||||
interface ItemReader {
|
||||
List<ItemRecord> read(String tenant, String session, Runnable checkLease);
|
||||
}
|
||||
|
||||
private record Start(Root root, Cursor cursor) {}
|
||||
|
||||
private final class SnapshotReadLease implements AutoCloseable {
|
||||
private final ToolPublicationRetentionStore.ReadLease lease;
|
||||
|
||||
SnapshotReadLease(String tenant, String session) {
|
||||
// afterCommit callbacks still have the completed transaction bound to the thread.
|
||||
lease = publish.execute(status -> retention.read(tenant, session));
|
||||
}
|
||||
|
||||
void check() {
|
||||
if (TransactionSynchronizationManager.isCurrentTransactionReadOnly()) {
|
||||
// A read view cannot commit a renewal; it must finish before the committed lease expires.
|
||||
lease.checkReadView();
|
||||
} else {
|
||||
publish.executeWithoutResult(status -> lease.check());
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
publish.executeWithoutResult(status -> lease.close());
|
||||
}
|
||||
}
|
||||
|
||||
private void lockStream(String tenant, String session) {
|
||||
jdbc.queryForList(
|
||||
"SELECT current_snapshot_id FROM ma_event_stream WHERE tenant_id = ? AND session_id"
|
||||
+ " = ? AND stream_kind = 'SESSION' AND stream_id = X'' FOR UPDATE",
|
||||
tenant,
|
||||
session);
|
||||
}
|
||||
|
||||
private void expire(String tenant, String session, String current, long now) {
|
||||
var rows =
|
||||
jdbc.queryForList(
|
||||
"SELECT snapshot_id, snapshot_version FROM ma_snapshot WHERE tenant_id = ?"
|
||||
+ " AND session_id = ? AND retain_until <= ? AND snapshot_id <> ? ORDER"
|
||||
+ " BY retain_until LIMIT 32 FOR UPDATE",
|
||||
tenant,
|
||||
session,
|
||||
now,
|
||||
current);
|
||||
for (var row : rows) {
|
||||
String id = (String) row.get("snapshot_id");
|
||||
ManagedProjectionResourceCollector.removeSnapshot(
|
||||
jdbc, tenant, session, id, number(row, "snapshot_version"));
|
||||
}
|
||||
}
|
||||
|
||||
private long protect(Root root, String tenant, String session) {
|
||||
long expires = clock.millis() + CURSOR_LIFETIME_MILLIS;
|
||||
jdbc.update(
|
||||
"UPDATE ma_snapshot SET retain_until = GREATEST(retain_until, ?) WHERE tenant_id ="
|
||||
+ " ? AND session_id = ? AND snapshot_id = ? AND state = 'PUBLISHED'",
|
||||
expires,
|
||||
tenant,
|
||||
session,
|
||||
root.id());
|
||||
return expires;
|
||||
}
|
||||
|
||||
private List<Root> current(String tenant, String session) {
|
||||
return jdbc.query(
|
||||
"SELECT " + ManagedSchemaProjection.SNAPSHOT.columns("p") + " FROM ma_event_stream s JOIN ma_snapshot p ON p.tenant_id = s.tenant_id"
|
||||
+ " AND p.session_id = s.session_id AND p.snapshot_id = s.current_snapshot_id"
|
||||
+ " WHERE s.tenant_id = ? AND s.session_id = ? AND s.stream_kind = 'SESSION'"
|
||||
+ " AND s.stream_id = X'' AND p.state = 'PUBLISHED'",
|
||||
(row, i) -> root(row),
|
||||
tenant,
|
||||
session);
|
||||
}
|
||||
|
||||
private Root root(java.sql.ResultSet row) throws java.sql.SQLException {
|
||||
return new Root(
|
||||
row.getString("snapshot_id"),
|
||||
row.getLong("snapshot_version"),
|
||||
row.getLong("covered_sequence"),
|
||||
row.getInt("projection_version"),
|
||||
row.getString("manifest_resource_key"),
|
||||
row.getString("manifest_sha256"),
|
||||
row.getLong("gmt_create"));
|
||||
}
|
||||
|
||||
private Manifest manifest(Root root, String tenant, String session) {
|
||||
byte[] bytes = resource(root.manifest(), tenant, session, "managed-snapshot-manifest");
|
||||
if (!WorkspaceEvidenceResources.digest(bytes).equals(root.digest())) {
|
||||
throw new IllegalStateException("Snapshot manifest digest differs");
|
||||
}
|
||||
try {
|
||||
Manifest result = decodeManifest(mapper, bytes);
|
||||
if (result.version() != 1
|
||||
|| !result.snapshot().equals(root.id())
|
||||
|| result.sequence() != root.sequence()
|
||||
|| result.projection() != root.projection()
|
||||
|| result.pages() < 0
|
||||
|| result.items() < 0) {
|
||||
throw new IllegalStateException("Snapshot manifest scope differs");
|
||||
}
|
||||
return result;
|
||||
} catch (java.io.IOException error) {
|
||||
throw new IllegalStateException("Snapshot manifest is invalid", error);
|
||||
}
|
||||
}
|
||||
|
||||
private List<ItemRecord> readPage(String tenant, String session, String snapshot, int page) {
|
||||
String scope = ManagedSessionStore.sessionScopeKey(tenant, session);
|
||||
try {
|
||||
List<SnapshotItem> frozen =
|
||||
decodePage(mapper,
|
||||
resource(
|
||||
ManagedSessionStore.resourceKey(scope, pageId(snapshot, page)),
|
||||
tenant,
|
||||
session,
|
||||
"managed-snapshot-page"));
|
||||
List<ItemRecord> result = new ArrayList<>(frozen.size());
|
||||
for (SnapshotItem entry : frozen) {
|
||||
ItemRecord item = entry.item();
|
||||
if (entry.body() == null) {
|
||||
result.add(item);
|
||||
} else {
|
||||
var parts =
|
||||
mapper.readValue(
|
||||
resource(
|
||||
entry.body(), tenant, session, "managed-message-parts"),
|
||||
new TypeReference<List<StoreModels.ItemPartRecord>>() {});
|
||||
result.add(
|
||||
new ItemRecord(
|
||||
item.tenantId(),
|
||||
item.sessionId(),
|
||||
item.itemId(),
|
||||
item.turnId(),
|
||||
item.type(),
|
||||
item.role(),
|
||||
item.status(),
|
||||
item.attributes(),
|
||||
item.firstSequence(),
|
||||
item.lastSequence(),
|
||||
item.createdAt(),
|
||||
item.updatedAt(),
|
||||
item.revision(),
|
||||
parts));
|
||||
}
|
||||
}
|
||||
return List.copyOf(result);
|
||||
} catch (java.io.IOException error) {
|
||||
throw new IllegalStateException("Snapshot page is invalid", error);
|
||||
}
|
||||
}
|
||||
|
||||
private byte[] resource(String key, String tenant, String session, String kind) {
|
||||
var rows =
|
||||
jdbc.query(
|
||||
"SELECT inline_bytes, sha256, byte_length FROM ma_resource WHERE"
|
||||
+ " resource_key = ? AND session_scope_key = ? AND tenant_id = ? AND"
|
||||
+ " session_id = ? AND kind = ? AND schema_version = 1 AND state ="
|
||||
+ " 'REFERENCED' AND deletion_operation_key IS NULL",
|
||||
(row, i) -> {
|
||||
byte[] bytes = row.getBytes("inline_bytes");
|
||||
if (bytes == null
|
||||
|| bytes.length != row.getLong("byte_length")
|
||||
|| !WorkspaceEvidenceResources.digest(bytes)
|
||||
.equals(row.getString("sha256"))) {
|
||||
throw new IllegalStateException("Snapshot resource digest differs");
|
||||
}
|
||||
return bytes;
|
||||
},
|
||||
key,
|
||||
ManagedSessionStore.sessionScopeKey(tenant, session),
|
||||
tenant,
|
||||
session,
|
||||
kind);
|
||||
if (rows.size() != 1) {
|
||||
throw new IllegalStateException("Snapshot resource is unavailable");
|
||||
}
|
||||
return rows.getFirst();
|
||||
}
|
||||
|
||||
private String writeResource(
|
||||
String tenant,
|
||||
String session,
|
||||
String scope,
|
||||
String owner,
|
||||
long version,
|
||||
String id,
|
||||
String kind,
|
||||
byte[] bytes) {
|
||||
return writeResource(tenant, session, scope, owner, version, id, kind, bytes, null);
|
||||
}
|
||||
|
||||
private String writeResource(
|
||||
String tenant,
|
||||
String session,
|
||||
String scope,
|
||||
String owner,
|
||||
long version,
|
||||
String id,
|
||||
String kind,
|
||||
byte[] bytes,
|
||||
PageLocation page) {
|
||||
String key = ManagedSessionStore.resourceKey(scope, id);
|
||||
jdbc.update(
|
||||
"INSERT INTO ma_resource (resource_key, session_scope_key, tenant_id, session_id, resource_id, kind, schema_version, byte_length, sha256, storage_kind, inline_bytes, state, gmt_create, last_verified_at, snapshot_id, snapshot_page_ordinal, snapshot_page_first_sequence, snapshot_page_last_sequence) VALUES (?, ?, ?, ?, ?, ?, 1, ?, ?, 'MYSQL_INLINE', ?, 'REFERENCED', " + ManagedSchemaTime.SQL_NOW + ", " + ManagedSchemaTime.SQL_NOW + ", ?, ?, ?, ?)",
|
||||
key,
|
||||
scope,
|
||||
tenant,
|
||||
session,
|
||||
id,
|
||||
kind,
|
||||
bytes.length,
|
||||
WorkspaceEvidenceResources.digest(bytes),
|
||||
bytes,
|
||||
page == null ? null : page.snapshot(),
|
||||
page == null ? null : page.ordinal(),
|
||||
page == null ? null : page.firstSequence(),
|
||||
page == null ? null : page.lastSequence());
|
||||
jdbc.update(
|
||||
"INSERT INTO ma_resource_ref (session_scope_key, owner_kind, owner_key, owner_revision, resource_key, tenant_id, session_id, resource_id, gmt_create) VALUES (?, 'SNAPSHOT', ?, ?, ?, ?, ?, ?, " + ManagedSchemaTime.SQL_NOW + ")",
|
||||
scope,
|
||||
owner,
|
||||
version,
|
||||
key,
|
||||
tenant,
|
||||
session,
|
||||
id);
|
||||
return key;
|
||||
}
|
||||
|
||||
private DraftPage draftPage(List<SnapshotItem> items) {
|
||||
return new DraftPage(
|
||||
json(items),
|
||||
items.getFirst().item().firstSequence(),
|
||||
items.getLast().item().firstSequence());
|
||||
}
|
||||
|
||||
private byte[] json(Object value) {
|
||||
try {
|
||||
return mapper.writeValueAsBytes(value);
|
||||
} catch (java.io.IOException error) {
|
||||
throw new IllegalStateException("Snapshot cannot be serialized", error);
|
||||
}
|
||||
}
|
||||
|
||||
private String encode(Cursor cursor, Manifest manifest) {
|
||||
byte[] payload = json(cursor);
|
||||
return Base64.getUrlEncoder().withoutPadding().encodeToString(payload)
|
||||
+ "."
|
||||
+ Base64.getUrlEncoder()
|
||||
.withoutPadding()
|
||||
.encodeToString(cursorMac(payload, manifest.cursorSecret()));
|
||||
}
|
||||
|
||||
private Cursor decode(String cursor) {
|
||||
try {
|
||||
if (cursor.length() > 4096 || cursor.indexOf('.') < 1) {
|
||||
throw invalidCursor();
|
||||
}
|
||||
return mapper.readValue(
|
||||
Base64.getUrlDecoder().decode(cursor.substring(0, cursor.indexOf('.'))),
|
||||
Cursor.class);
|
||||
} catch (java.io.IOException | IllegalArgumentException error) {
|
||||
throw invalidCursor();
|
||||
}
|
||||
}
|
||||
|
||||
private void verifyCursor(String cursor, Manifest manifest) {
|
||||
try {
|
||||
int separator = cursor.indexOf('.');
|
||||
byte[] payload = Base64.getUrlDecoder().decode(cursor.substring(0, separator));
|
||||
byte[] signature = Base64.getUrlDecoder().decode(cursor.substring(separator + 1));
|
||||
if (!java.security.MessageDigest.isEqual(
|
||||
signature, cursorMac(payload, manifest.cursorSecret()))) {
|
||||
throw invalidCursor();
|
||||
}
|
||||
} catch (IllegalArgumentException error) {
|
||||
throw invalidCursor();
|
||||
}
|
||||
}
|
||||
|
||||
private static byte[] cursorMac(byte[] payload, String secret) {
|
||||
try {
|
||||
return new HmacUtils(HmacAlgorithms.HMAC_SHA_256,
|
||||
Base64.getDecoder().decode(secret)).hmac(payload);
|
||||
} catch (IllegalArgumentException error) {
|
||||
if (error.getCause() instanceof GeneralSecurityException cause) {
|
||||
throw new IllegalStateException("Snapshot cursor signing is unavailable", cause);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
private static String newCursorSecret() {
|
||||
byte[] bytes = new byte[32];
|
||||
CURSOR_RANDOM.nextBytes(bytes);
|
||||
return Base64.getEncoder().encodeToString(bytes);
|
||||
}
|
||||
|
||||
private static String pageId(String snapshot, int page) {
|
||||
return "snapshot:" + snapshot + ":page:" + page;
|
||||
}
|
||||
|
||||
private static long number(Map<String, Object> row, String name) {
|
||||
return ((Number) row.get(name)).longValue();
|
||||
}
|
||||
|
||||
private static ManagedAgentException invalidCursor() {
|
||||
return new ManagedAgentException(
|
||||
HttpStatus.BAD_REQUEST, "invalid_item_cursor", "The item cursor is invalid.");
|
||||
}
|
||||
|
||||
private static ManagedAgentException expiredCursor() {
|
||||
return new ManagedAgentException(
|
||||
HttpStatus.CONFLICT, "cursor_expired", "The snapshot cursor has expired.");
|
||||
}
|
||||
|
||||
record Page(List<ItemRecord> items, boolean hasMore, String nextCursor, long coveredSequence) {}
|
||||
|
||||
private record Root(
|
||||
String id,
|
||||
long version,
|
||||
long sequence,
|
||||
int projection,
|
||||
String manifest,
|
||||
String digest,
|
||||
long created) {}
|
||||
|
||||
private record Draft(
|
||||
long sequence, int projection, int items, List<DraftPage> pages, List<String> bodies) {}
|
||||
|
||||
private record DraftPage(byte[] bytes, long firstSequence, long lastSequence) {}
|
||||
|
||||
private record PageLocation(
|
||||
String snapshot, int ordinal, long firstSequence, long lastSequence) {}
|
||||
|
||||
static Manifest decodeManifest(ObjectMapper mapper, byte[] bytes) throws java.io.IOException {
|
||||
return mapper.readValue(bytes, Manifest.class);
|
||||
}
|
||||
|
||||
static List<SnapshotItem> decodePage(ObjectMapper mapper, byte[] bytes) throws java.io.IOException {
|
||||
return mapper.readValue(bytes, ITEMS);
|
||||
}
|
||||
|
||||
record SnapshotItem(ItemRecord item, String body) {}
|
||||
|
||||
record Manifest(
|
||||
int version,
|
||||
String snapshot,
|
||||
long sequence,
|
||||
int projection,
|
||||
int items,
|
||||
int pages,
|
||||
String cursorSecret) {}
|
||||
|
||||
private record Cursor(
|
||||
int version,
|
||||
String scope,
|
||||
String snapshot,
|
||||
long snapshotVersion,
|
||||
int projection,
|
||||
int page,
|
||||
int offset,
|
||||
long expires,
|
||||
long afterSequence) {}
|
||||
}
|
||||
+28
-194
@@ -1,7 +1,6 @@
|
||||
package com.alibaba.qwen.code.managedagent.persistence;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.catalog.AgentDefinitionExtensions;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels;
|
||||
import com.alibaba.qwen.code.runtimebroker.CsiFilesRetirementProfile;
|
||||
import com.alibaba.qwen.code.runtimebroker.CsiNativeActivationProof;
|
||||
import com.alibaba.qwen.code.runtimebroker.CsiOwnershipRevision;
|
||||
@@ -89,7 +88,7 @@ final class WorkspaceCsiApplicationInventory {
|
||||
declaration("ma_event_stream", "tenant_id,session_id,stream_kind,stream_id",
|
||||
"tenant_id:VARBINARY,session_id:VARBINARY,stream_kind:VARCHAR,stream_id:VARBINARY,last_sequence:BIGINT,"
|
||||
+ "replay_floor_sequence:BIGINT,covered_sequence:BIGINT,wake_covered_sequence:BIGINT,"
|
||||
+ "projection_version:INT,snapshot_stale_since:BIGINT,current_snapshot_id:VARBINARY,"
|
||||
+ "projection_version:INT,"
|
||||
+ "artifact_refs_resource_key:CHAR,output_ranges_json:LONGTEXT,projection_pending:BOOLEAN,"
|
||||
+ "gmt_modified:BIGINT"),
|
||||
declaration("ma_extension_operation", "operation_key",
|
||||
@@ -168,8 +167,7 @@ final class WorkspaceCsiApplicationInventory {
|
||||
+ "session_id:VARBINARY,resource_id:VARBINARY,kind:VARBINARY,schema_version:INT,byte_length:BIGINT,"
|
||||
+ "sha256:CHAR,storage_kind:VARCHAR,inline_bytes:LONGBLOB,object_key:VARBINARY,"
|
||||
+ "object_version_id:VARBINARY,encryption_key_id:VARBINARY,publish_command_id:VARBINARY,state:VARCHAR,"
|
||||
+ "snapshot_id:VARBINARY,snapshot_page_ordinal:INT,snapshot_page_first_sequence:BIGINT,"
|
||||
+ "snapshot_page_last_sequence:BIGINT,scope_key:CHAR,publication_id:VARBINARY,slot_key:VARBINARY,"
|
||||
+ "scope_key:CHAR,publication_id:VARBINARY,slot_key:VARBINARY,"
|
||||
+ "capture_state:VARCHAR,capture_object_key:VARBINARY,operation_id:VARBINARY,quarantine_reason:VARCHAR,"
|
||||
+ "deletion_operation_key:CHAR,gmt_create:BIGINT,last_verified_at:BIGINT,retention_until:BIGINT"),
|
||||
declaration("ma_resource_ref", "session_scope_key,owner_kind,owner_key,owner_revision,resource_key",
|
||||
@@ -262,10 +260,6 @@ final class WorkspaceCsiApplicationInventory {
|
||||
+ "physical_key:CHAR,worktree_id:CHAR,owner_actor_key:VARBINARY,state:VARCHAR,revision:BIGINT,"
|
||||
+ "plan_json:LONGTEXT,identity_json:LONGTEXT,operation_id:VARBINARY,failure_code:VARCHAR,"
|
||||
+ "gmt_create:BIGINT,gmt_modified:BIGINT"),
|
||||
declaration("ma_snapshot", "tenant_id,session_id,snapshot_id",
|
||||
"tenant_id:VARBINARY,session_id:VARBINARY,snapshot_id:VARBINARY,snapshot_version:BIGINT,"
|
||||
+ "covered_sequence:BIGINT,projection_version:INT,manifest_resource_key:CHAR,manifest_sha256:CHAR,"
|
||||
+ "item_count:BIGINT,state:VARCHAR,gmt_create:BIGINT,published_at:BIGINT,retain_until:BIGINT"),
|
||||
declaration("ma_storage", "alias_key",
|
||||
"alias_key:CHAR,tenant_key:CHAR,storage_key:CHAR,tenant_id:VARBINARY,storage_id:VARBINARY,"
|
||||
+ "physical_key:CHAR,registration_state:VARCHAR,storage_kind:VARCHAR,registration_revision:BIGINT,"
|
||||
@@ -345,7 +339,6 @@ final class WorkspaceCsiApplicationInventory {
|
||||
"creation_key",
|
||||
"credential_key_id",
|
||||
"csi_provision_request_id",
|
||||
"current_snapshot_id",
|
||||
"decision_receipt_id",
|
||||
"default_tenant",
|
||||
"delivery_id",
|
||||
@@ -406,7 +399,6 @@ final class WorkspaceCsiApplicationInventory {
|
||||
"sender_id",
|
||||
"session_id",
|
||||
"slot_key",
|
||||
"snapshot_id",
|
||||
"source_key",
|
||||
"source_session_id",
|
||||
"storage_id",
|
||||
@@ -1183,13 +1175,12 @@ final class WorkspaceCsiApplicationInventory {
|
||||
var carriers = declaration.columns().stream().filter(column -> structured(column.name())
|
||||
|| column.kind() == Kind.MEDIUMBLOB || column.kind() == Kind.LONGBLOB
|
||||
|| "object_key".equals(column.name()) || "capture_object_key".equals(column.name())).map(column -> column.name() + " IS NOT NULL").toList();
|
||||
if (!carriers.isEmpty() || "ma_snapshot".equals(declaration.name())) {
|
||||
var candidates = read.query(declaration, carriers.isEmpty() ? "1 = 1" : String.join(" OR ", carriers), List.of());
|
||||
if (!carriers.isEmpty()) {
|
||||
var candidates = read.query(declaration, String.join(" OR ", carriers), List.of());
|
||||
// Manifests establish coverage before independently fetched byte parts are accepted.
|
||||
if ("ma_resource".equals(declaration.name())) {
|
||||
for (Row row : candidates) {
|
||||
if (!fixed.containsKey(key(row)) && (chunkManifest(row.text("kind"))
|
||||
|| "managed-snapshot-manifest".equals(row.text("kind")))) {
|
||||
if (!fixed.containsKey(key(row)) && chunkManifest(row.text("kind"))) {
|
||||
read.carrier(declaration.name(), row);
|
||||
}
|
||||
}
|
||||
@@ -1197,7 +1188,7 @@ final class WorkspaceCsiApplicationInventory {
|
||||
for (Row row : candidates) {
|
||||
if (!fixed.containsKey(key(row))) {
|
||||
if (!"ma_resource".equals(declaration.name())
|
||||
|| !chunkManifest(row.text("kind")) && !"managed-snapshot-manifest".equals(row.text("kind"))) {
|
||||
|| !chunkManifest(row.text("kind"))) {
|
||||
read.carrier(declaration.name(), row);
|
||||
}
|
||||
}
|
||||
@@ -1275,10 +1266,6 @@ final class WorkspaceCsiApplicationInventory {
|
||||
private final Map<String, Map<String, Row>> seen = new LinkedHashMap<>();
|
||||
private final List<Blocker> blockers = new ArrayList<>();
|
||||
private final Set<String> chunkParts = new LinkedHashSet<>();
|
||||
private final Set<String> snapshotRoots = new LinkedHashSet<>();
|
||||
private final Set<String> snapshotPages = new LinkedHashSet<>();
|
||||
private final Set<String> snapshotBodies = new LinkedHashSet<>();
|
||||
private final Set<String> snapshotReferences = new LinkedHashSet<>();
|
||||
private final Map<String, com.alibaba.qwen.code.runtimebroker.ToolExecutionRecord> nativeMembers = new LinkedHashMap<>();
|
||||
private final FrozenOwnerReader frozenReader;
|
||||
private int count;
|
||||
@@ -1405,15 +1392,7 @@ final class WorkspaceCsiApplicationInventory {
|
||||
block(table, row, "extra_historical_reference");
|
||||
return;
|
||||
}
|
||||
if ("ma_snapshot".equals(table)) {
|
||||
if (!snapshotRoots.contains(key(row))) {
|
||||
Row manifest = one("ma_resource", "resource_key = ?",
|
||||
List.of(new Cell(Kind.CHAR, utf8(row.text("manifest_resource_key")))));
|
||||
requireEncoded("managed-snapshot-manifest".equals(manifest.text("kind")));
|
||||
snapshot(manifest);
|
||||
requireEncoded(snapshotRoots.contains(key(row)));
|
||||
}
|
||||
} else if ("ma_operation".equals(table) && "CSI_RETIREMENT".equals(row.text("operation_kind"))) {
|
||||
if ("ma_operation".equals(table) && "CSI_RETIREMENT".equals(row.text("operation_kind"))) {
|
||||
if (frozenReader == null) {
|
||||
block(table, row, "unsupported_historical_payload");
|
||||
return;
|
||||
@@ -1476,14 +1455,6 @@ final class WorkspaceCsiApplicationInventory {
|
||||
agentResource(row);
|
||||
return;
|
||||
}
|
||||
if ("managed-snapshot-manifest".equals(kind)) {
|
||||
snapshot(row);
|
||||
return;
|
||||
}
|
||||
if ("managed-snapshot-page".equals(kind)) {
|
||||
requireEncoded(snapshotPages.contains(key(row)));
|
||||
return;
|
||||
}
|
||||
byte[] raw = payload(row, "inline_bytes", "sha256",
|
||||
"managed-message-parts".equals(kind) ? MAX_BYTES : 8 * 1024 * 1024);
|
||||
if (chunkPart(kind)) {
|
||||
@@ -1587,8 +1558,7 @@ final class WorkspaceCsiApplicationInventory {
|
||||
&& ((instructions ? "agent-instructions-" : "agent-skills-") + hash(raw)).equals(row.text("resource_id"))
|
||||
&& row.text("resource_id").equals(row.text("publish_command_id"))
|
||||
&& row.text("workspace_id") != null && !row.text("workspace_id").isEmpty());
|
||||
absent(row, "snapshot_id", "snapshot_page_ordinal", "snapshot_page_first_sequence", "snapshot_page_last_sequence",
|
||||
"scope_key", "publication_id", "slot_key", "capture_state", "capture_object_key", "operation_id",
|
||||
absent(row, "scope_key", "publication_id", "slot_key", "capture_state", "capture_object_key", "operation_id",
|
||||
"quarantine_reason", "deletion_operation_key");
|
||||
try {
|
||||
strictText(raw);
|
||||
@@ -1602,120 +1572,6 @@ final class WorkspaceCsiApplicationInventory {
|
||||
}
|
||||
}
|
||||
|
||||
private void snapshot(Row manifestRow) throws SQLException {
|
||||
publicResource(manifestRow);
|
||||
absent(manifestRow, "snapshot_id", "snapshot_page_ordinal", "snapshot_page_first_sequence", "snapshot_page_last_sequence");
|
||||
byte[] raw = payload(manifestRow, "inline_bytes", "sha256", MAX_BYTES);
|
||||
JsonNode value = publicJson(raw);
|
||||
WorkspaceCsiApplicationJson.closed(value, Set.of("version", "snapshot", "sequence", "projection", "items", "pages", "cursorSecret"));
|
||||
requireEncoded(integer(value, "version") == 1 && integer(value, "sequence") >= 0
|
||||
&& integer(value, "projection") > 0 && integer(value, "projection") <= Integer.MAX_VALUE
|
||||
&& integer(value, "items") >= 0 && integer(value, "items") <= MAX_ROWS * 100L
|
||||
&& integer(value, "pages") >= 0 && integer(value, "pages") <= MAX_ROWS
|
||||
&& value.path("snapshot").isTextual() && value.path("cursorSecret").isTextual());
|
||||
ManagedSnapshotStore.Manifest manifest;
|
||||
try {
|
||||
manifest = ManagedSnapshotStore.decodeManifest(PUBLIC_ARTIFACTS, raw);
|
||||
} catch (IOException malformed) {
|
||||
throw new IllegalArgumentException("Invalid snapshot manifest", malformed);
|
||||
}
|
||||
requireEncoded(java.util.UUID.fromString(manifest.snapshot()).toString().equals(manifest.snapshot())
|
||||
&& ("snapshot:" + manifest.snapshot() + ":manifest").equals(manifestRow.text("resource_id")));
|
||||
byte[] secret = Base64.getDecoder().decode(manifest.cursorSecret());
|
||||
requireEncoded(secret.length == 32 && Base64.getEncoder().encodeToString(secret).equals(manifest.cursorSecret()));
|
||||
Row root = one("ma_snapshot", "tenant_id = ? AND session_id = ? AND snapshot_id = ?",
|
||||
List.of(new Cell(Kind.VARBINARY, utf8(manifestRow.text("tenant_id"))),
|
||||
new Cell(Kind.VARBINARY, utf8(manifestRow.text("session_id"))), new Cell(Kind.VARBINARY, utf8(manifest.snapshot()))));
|
||||
requireEncoded("PUBLISHED".equals(root.text("state")) && root.number("snapshot_version") > 0
|
||||
&& Objects.equals(root.number("covered_sequence"), manifest.sequence())
|
||||
&& Objects.equals(root.number("projection_version"), (long) manifest.projection())
|
||||
&& Objects.equals(root.number("item_count"), (long) manifest.items())
|
||||
&& manifestRow.text("resource_key").equals(root.text("manifest_resource_key"))
|
||||
&& manifestRow.text("sha256").equals(root.text("manifest_sha256")) && root.number("published_at") != null);
|
||||
foreignArtifact("ma_snapshot", root, null);
|
||||
foreignArtifact("ma_resource", manifestRow, value);
|
||||
if (snapshotRoots.contains(key(root))) {
|
||||
return;
|
||||
}
|
||||
String scope = manifestRow.text("session_scope_key");
|
||||
var resources = new LinkedHashMap<String, Row>();
|
||||
resources.put(manifestRow.text("resource_key"), manifestRow);
|
||||
var bodies = new LinkedHashSet<String>();
|
||||
var pages = query(CATALOGUE.stream().filter(table -> table.name().equals("ma_resource")).findFirst().orElseThrow(),
|
||||
"session_scope_key = ? AND snapshot_id = ?",
|
||||
List.of(new Cell(Kind.CHAR, utf8(scope)), new Cell(Kind.VARBINARY, utf8(manifest.snapshot()))));
|
||||
requireEncoded(pages.size() == manifest.pages());
|
||||
pages.sort(java.util.Comparator.comparing(row -> row.number("snapshot_page_ordinal")));
|
||||
var ids = new LinkedHashSet<String>();
|
||||
StoreModels.ItemRecord previous = null;
|
||||
for (int ordinal = 0; ordinal < pages.size(); ordinal++) {
|
||||
historyRemaining(started);
|
||||
Row page = pages.get(ordinal);
|
||||
publicResource(page);
|
||||
requireEncoded("managed-snapshot-page".equals(page.text("kind")) && sameScope(manifestRow, page)
|
||||
&& ("snapshot:" + manifest.snapshot() + ":page:" + ordinal).equals(page.text("resource_id"))
|
||||
&& Objects.equals(page.number("snapshot_page_ordinal"), (long) ordinal));
|
||||
byte[] bytes = payload(page, "inline_bytes", "sha256", MAX_BYTES);
|
||||
JsonNode entries = publicJson(bytes);
|
||||
requireEncoded(entries.isArray() && !entries.isEmpty() && entries.size() <= 100);
|
||||
for (JsonNode entry : entries) {
|
||||
WorkspaceCsiApplicationJson.closed(entry, Set.of("item", "body"));
|
||||
validateItem(entry.path("item"), manifestRow, manifest.sequence());
|
||||
requireEncoded(entry.path("body").isNull() || entry.path("body").isTextual());
|
||||
}
|
||||
List<ManagedSnapshotStore.SnapshotItem> frozen;
|
||||
try {
|
||||
frozen = ManagedSnapshotStore.decodePage(PUBLIC_ARTIFACTS, bytes);
|
||||
} catch (IOException malformed) {
|
||||
throw new IllegalArgumentException("Invalid snapshot page", malformed);
|
||||
}
|
||||
requireEncoded(Objects.equals(page.number("snapshot_page_first_sequence"), frozen.getFirst().item().firstSequence())
|
||||
&& Objects.equals(page.number("snapshot_page_last_sequence"), frozen.getLast().item().firstSequence()));
|
||||
for (var entry : frozen) {
|
||||
var item = entry.item();
|
||||
requireEncoded(ids.add(item.itemId()) && (previous == null || previous.firstSequence() < item.firstSequence()
|
||||
|| previous.firstSequence() == item.firstSequence()
|
||||
&& Arrays.compareUnsigned(utf8(previous.itemId()), utf8(item.itemId())) < 0));
|
||||
previous = item;
|
||||
if (entry.body() != null) {
|
||||
requireEncoded(item.content().isEmpty() && !references.getOrDefault("resourceKey", Set.of()).contains(entry.body()));
|
||||
Row body = one("ma_resource", "resource_key = ?", List.of(new Cell(Kind.CHAR, utf8(entry.body()))));
|
||||
requireEncoded("managed-message-parts".equals(body.text("kind")) && sameScope(manifestRow, body));
|
||||
JsonNode parts = bodyParts(body, payload(body, "inline_bytes", "sha256", MAX_BYTES));
|
||||
bodyRevision(body, item.itemId(), item.revision());
|
||||
partCoverage(parts, item.firstSequence(), item.lastSequence(), item.createdAt(), item.updatedAt());
|
||||
foreignArtifact("ma_resource", body, parts);
|
||||
resources.put(body.text("resource_key"), body);
|
||||
bodies.add(key(body));
|
||||
}
|
||||
}
|
||||
foreignArtifact("ma_resource", page, entries);
|
||||
resources.put(page.text("resource_key"), page);
|
||||
}
|
||||
requireEncoded(ids.size() == manifest.items());
|
||||
String owner = com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaKeys.key(scope, manifest.snapshot());
|
||||
var refs = query(CATALOGUE.stream().filter(table -> table.name().equals("ma_resource_ref")).findFirst().orElseThrow(),
|
||||
"session_scope_key = ? AND owner_kind = 'SNAPSHOT' AND owner_key = ?",
|
||||
List.of(new Cell(Kind.CHAR, utf8(scope)), new Cell(Kind.CHAR, utf8(owner))));
|
||||
var referenced = new LinkedHashSet<String>();
|
||||
for (Row ref : refs) {
|
||||
Row resource = resources.get(ref.text("resource_key"));
|
||||
requireEncoded(resource != null && referenced.add(ref.text("resource_key"))
|
||||
&& Objects.equals(ref.number("owner_revision"), root.number("snapshot_version")));
|
||||
referenceScope(resource, ref);
|
||||
foreignArtifact("ma_resource_ref", ref, null);
|
||||
if (!bodies.contains(key(resource))) {
|
||||
var all = resourceReferences(resource);
|
||||
requireEncoded(all.size() == 1 && Arrays.equals(all.getFirst().bytes(), ref.bytes()));
|
||||
}
|
||||
}
|
||||
requireEncoded(referenced.equals(resources.keySet()));
|
||||
snapshotRoots.add(key(root));
|
||||
pages.forEach(page -> snapshotPages.add(key(page)));
|
||||
snapshotBodies.addAll(bodies);
|
||||
refs.forEach(ref -> snapshotReferences.add(key(ref)));
|
||||
}
|
||||
|
||||
private static boolean sameScope(Row left, Row right) {
|
||||
return Objects.equals(left.text("tenant_id"), right.text("tenant_id"))
|
||||
&& Objects.equals(left.text("session_id"), right.text("session_id"))
|
||||
@@ -1727,29 +1583,8 @@ final class WorkspaceCsiApplicationInventory {
|
||||
absent(ref, "workspace_id", "journal_revision");
|
||||
}
|
||||
|
||||
private static void validateItem(JsonNode item, Row manifest, long sequence) {
|
||||
WorkspaceCsiApplicationJson.closed(item, Set.of("tenantId", "sessionId", "itemId", "turnId", "type", "role", "status",
|
||||
"attributes", "firstSequence", "lastSequence", "createdAt", "updatedAt", "revision", "content"));
|
||||
requireEncoded(item.path("tenantId").isTextual() && manifest.text("tenant_id").equals(item.path("tenantId").textValue())
|
||||
&& item.path("sessionId").isTextual() && manifest.text("session_id").equals(item.path("sessionId").textValue())
|
||||
&& item.path("itemId").isTextual() && !item.path("itemId").textValue().isEmpty()
|
||||
&& StandardCharsets.UTF_8.newEncoder().canEncode(item.path("itemId").textValue())
|
||||
&& item.path("type").isTextual() && !item.path("type").textValue().isEmpty()
|
||||
&& (item.path("attributes").isNull() || item.path("attributes").isObject()));
|
||||
for (String field : List.of("turnId", "role", "status")) {
|
||||
requireEncoded(item.path(field).isNull() || item.path(field).isTextual());
|
||||
}
|
||||
long first = integer(item, "firstSequence"), last = integer(item, "lastSequence");
|
||||
long created = integer(item, "createdAt"), updated = integer(item, "updatedAt");
|
||||
requireEncoded(first >= 0 && last >= first && last <= sequence && created >= 0 && updated >= created && integer(item, "revision") > 0);
|
||||
JsonNode parts = item.path("content");
|
||||
validateParts(parts);
|
||||
partCoverage(parts, first, last, created, updated);
|
||||
}
|
||||
|
||||
private static JsonNode bodyParts(Row row, byte[] raw) {
|
||||
publicResource(row);
|
||||
absent(row, "snapshot_id", "snapshot_page_ordinal", "snapshot_page_first_sequence", "snapshot_page_last_sequence");
|
||||
requireEncoded(raw.length > 64 * 1024);
|
||||
JsonNode parts = publicJson(raw);
|
||||
validateParts(parts);
|
||||
@@ -1793,28 +1628,29 @@ final class WorkspaceCsiApplicationInventory {
|
||||
try {
|
||||
var messages = query(CATALOGUE.stream().filter(table -> table.name().equals("ma_message")).findFirst().orElseThrow(),
|
||||
"body_resource_key = ?", List.of(new Cell(Kind.CHAR, utf8(row.text("resource_key")))));
|
||||
requireEncoded(messages.size() <= 1 && (!messages.isEmpty() || snapshotBodies.contains(key(row))));
|
||||
long revision = 0;
|
||||
String owner = null;
|
||||
if (!messages.isEmpty()) {
|
||||
Row message = messages.getFirst();
|
||||
requireEncoded(Objects.equals(row.text("tenant_id"), message.text("tenant_id"))
|
||||
&& Objects.equals(row.text("session_id"), message.text("session_id")) && "[]".equals(message.text("parts_json")));
|
||||
revision = bodyRevision(row, message.text("item_id"), message.number("revision"));
|
||||
owner = com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaKeys.key(row.text("session_scope_key"), message.text("item_id"));
|
||||
partCoverage(parts, message.number("first_sequence"), message.number("last_sequence"),
|
||||
message.number("gmt_create"), message.number("gmt_modified"));
|
||||
requireEncoded(messages.size() <= 1);
|
||||
if (messages.isEmpty()) {
|
||||
requireEncoded(row.number("retention_until") != null && resourceReferences(row).isEmpty());
|
||||
String[] identity = row.text("resource_id").split(":", -1);
|
||||
requireEncoded(identity.length == 4 && "message".equals(identity[0])
|
||||
&& identity[1].matches("[0-9a-f]{64}") && identity[2].matches("[1-9][0-9]{0,18}")
|
||||
&& Long.parseLong(identity[2]) > 0 && row.text("sha256").equals(identity[3]));
|
||||
return parts;
|
||||
}
|
||||
Row message = messages.getFirst();
|
||||
requireEncoded(Objects.equals(row.text("tenant_id"), message.text("tenant_id"))
|
||||
&& Objects.equals(row.text("session_id"), message.text("session_id")) && "[]".equals(message.text("parts_json")));
|
||||
long revision = bodyRevision(row, message.text("item_id"), message.number("revision"));
|
||||
String owner = com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaKeys.key(row.text("session_scope_key"), message.text("item_id"));
|
||||
partCoverage(parts, message.number("first_sequence"), message.number("last_sequence"),
|
||||
message.number("gmt_create"), message.number("gmt_modified"));
|
||||
int current = 0;
|
||||
for (Row ref : resourceReferences(row)) {
|
||||
referenceScope(row, ref);
|
||||
foreignArtifact("ma_resource_ref", ref, null);
|
||||
if ("MESSAGE".equals(ref.text("owner_kind"))) {
|
||||
requireEncoded(owner != null && owner.equals(ref.text("owner_key")) && Objects.equals(ref.number("owner_revision"), revision));
|
||||
current++;
|
||||
} else {
|
||||
requireEncoded("SNAPSHOT".equals(ref.text("owner_kind")) && snapshotReferences.contains(key(ref)));
|
||||
}
|
||||
requireEncoded("MESSAGE".equals(ref.text("owner_kind"))
|
||||
&& owner.equals(ref.text("owner_key")) && Objects.equals(ref.number("owner_revision"), revision));
|
||||
current++;
|
||||
}
|
||||
requireEncoded(current == messages.size());
|
||||
return parts;
|
||||
@@ -2349,7 +2185,6 @@ final class WorkspaceCsiApplicationInventory {
|
||||
query(table, "tenant_id=T&session_id=sessions");
|
||||
case "ma_event_stream" -> query(table, "tenant_id=T&session_id=sessions", "artifact_refs_resource_key=resourceKeys");
|
||||
case "ma_message" -> query(table, "tenant_id=T&session_id=sessions", "body_resource_key=resourceKeys");
|
||||
case "ma_snapshot" -> query(table, "tenant_id=T&session_id=sessions", "manifest_resource_key=resourceKeys");
|
||||
case "ma_agent_definition" -> query(table, "tenant_id=T&agent_id=agent&revision=agentRevision");
|
||||
case "ma_workspace_access", "ma_channel_instance", "ma_extension_operation", "ma_extension_package",
|
||||
"ma_extension_settings" -> query(table, "tenant_id=T&workspace_id=W");
|
||||
@@ -2463,7 +2298,7 @@ final class WorkspaceCsiApplicationInventory {
|
||||
&& Objects.equals(binding.text("tenant_id"), row.text("tenant_id"))
|
||||
&& Objects.equals(binding.text("workspace_id"), row.text("workspace_id")));
|
||||
}
|
||||
case "ma_session", "ma_turn", "ma_public_event", "ma_event_stream", "ma_message", "ma_snapshot",
|
||||
case "ma_session", "ma_turn", "ma_public_event", "ma_event_stream", "ma_message",
|
||||
"ma_session_record", "ma_session_journal_tx", "ma_channel_binding", "ma_schedule", "ma_schedule_occurrence" ->
|
||||
otherSession(row.text("tenant_id"), row.text("workspace_id"), row.text("session_id"))
|
||||
&& (row.cell("session_scope_key") == null || Objects.equals(row.text("session_scope_key"),
|
||||
@@ -3172,8 +3007,7 @@ final class WorkspaceCsiApplicationInventory {
|
||||
expect(table, row, rawSession(row) && "SESSION".equals(row.text("stream_kind"))
|
||||
&& "".equals(row.text("stream_id")) && Objects.equals(1L, row.number("last_sequence"))
|
||||
&& Objects.equals(0L, row.number("covered_sequence")) && Objects.equals(0L, row.number("replay_floor_sequence"))
|
||||
&& row.cell("wake_covered_sequence").isNull() && row.cell("snapshot_stale_since").isNull()
|
||||
&& row.cell("current_snapshot_id").isNull() && row.cell("artifact_refs_resource_key").isNull()
|
||||
&& row.cell("wake_covered_sequence").isNull() && row.cell("artifact_refs_resource_key").isNull()
|
||||
&& row.cell("output_ranges_json").isNull() && Objects.equals(1L, row.number("projection_version"))
|
||||
&& Boolean.TRUE.equals(row.bool("projection_pending")) && parent != null
|
||||
&& Objects.equals(parent.number("gmt_create"), row.number("gmt_modified")));
|
||||
|
||||
+6
-12
@@ -30,7 +30,6 @@ import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionMutationComma
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionMutationKind;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionPage;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SnapshotRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.TurnPage;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.TurnRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.TurnSummary;
|
||||
@@ -838,28 +837,23 @@ public interface AgentStateStore {
|
||||
EventPage findTranscriptEvents(String tenantId, String sessionId,
|
||||
Long beforeSequence, int limit);
|
||||
|
||||
StoreModels.SnapshotPage findSnapshotPage(String tenantId, String sessionId, String cursor, long afterSequence, int limit);
|
||||
|
||||
StoreModels.SnapshotPage findRecentSnapshotPage(String tenantId, String sessionId, String cursor, int limit);
|
||||
|
||||
StoreModels.MessagePage findMessagePage(String tenantId, String sessionId,
|
||||
String cursor, int limit, boolean backward);
|
||||
|
||||
Optional<SnapshotRecord> findSnapshot(String tenantId,
|
||||
String sessionId);
|
||||
StoreModels.MessagePage findMessagePage(String tenantId, String sessionId,
|
||||
String cursor, int limit, boolean backward, long afterSequence);
|
||||
|
||||
/** The snapshot's covered sequence of each given Session, in one read. */
|
||||
Map<String, Long> findSnapshotCoveredSequences(String tenantId,
|
||||
/** The message projection coverage of each Session, in one read. */
|
||||
Map<String, Long> findMessageCoveredSequences(String tenantId,
|
||||
List<String> sessionIds);
|
||||
|
||||
ReplayWindow findReplayWindow(String tenantId, String sessionId);
|
||||
|
||||
/** The Sessions whose Snapshot covers more than their replay floor. */
|
||||
/** The Sessions whose message projection covers more than their replay floor. */
|
||||
List<ReplayFloorTarget> findReplayFloorTargets(int limit);
|
||||
|
||||
/**
|
||||
* Raises the Session's replay floor, never above the Snapshot's covered
|
||||
* sequence, so a client told to resync can resume from the Snapshot.
|
||||
* Raises the Session's replay floor only through readable message coverage.
|
||||
*/
|
||||
ReplayWindow advanceReplayFloor(String tenantId, String sessionId,
|
||||
long floorSequence);
|
||||
|
||||
+12
-100
@@ -40,7 +40,6 @@ import com.alibaba.qwen.code.managedagent.model.ApiModels.WebShellWorkspace;
|
||||
import com.alibaba.qwen.code.managedagent.model.RequestDigests;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.Admission;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.EventPage;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.EventRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.ItemPartRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.ItemRecord;
|
||||
@@ -49,10 +48,8 @@ import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionMutationComma
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionMutationKind;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionPage;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SnapshotRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.TurnPage;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.TurnSummary;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedAgentStore;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedArtifactReader;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedExtensionProjection;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedWorkspaceRegistry;
|
||||
@@ -1152,62 +1149,18 @@ public class ManagedAgentService {
|
||||
throw new ManagedAgentException(HttpStatus.BAD_REQUEST, "invalid_event_cursor", "Item sequence must be non-negative.");
|
||||
}
|
||||
requireReadableSession(tenantId, actorId, sessionId);
|
||||
var page = store.findSnapshotPage(tenantId, sessionId, cursor, afterSequence, limit(requestedLimit));
|
||||
var page = store.findMessagePage(tenantId, sessionId, cursor, limit(requestedLimit), false, afterSequence);
|
||||
return new PublicItemList("list", page.items().stream().map(this::publicItem).toList(), page.hasMore(),
|
||||
page.nextCursor(), page.coveredSequence());
|
||||
page.nextCursor(), page.throughSequence());
|
||||
}
|
||||
|
||||
public WebShellExportPage exportPage(String tenantId, String actorId,
|
||||
String sessionId, String cursor, int requestedLimit) {
|
||||
requireReadableSession(tenantId, actorId, sessionId);
|
||||
var page = store.findSnapshotPage(tenantId, sessionId, cursor, 0, limit(requestedLimit));
|
||||
return new WebShellExportPage(page.items().stream().map(this::webShellItem).toList(),
|
||||
page.nextCursor(), page.hasMore(), page.coveredSequence());
|
||||
return messageExportPage(tenantId, actorId, sessionId, cursor, requestedLimit, false);
|
||||
}
|
||||
|
||||
public WebShellTranscript transcript(String tenantId, String actorId,
|
||||
String sessionId, String cursor, int requestedLimit) {
|
||||
SessionRecord session = requireReadableSession(tenantId, actorId,
|
||||
sessionId);
|
||||
int limit = eventLimit(requestedLimit);
|
||||
if (StringUtils.isBlank(cursor)) {
|
||||
SnapshotRecord snapshot = store.findSnapshot(tenantId, sessionId)
|
||||
.orElse(null);
|
||||
if (snapshot != null) {
|
||||
long visibleSequence = Math.max(session.lastSequence(),
|
||||
snapshot.coveredSequence());
|
||||
List<EventRecord> events = new ArrayList<>(
|
||||
store.findControlEvents(tenantId, sessionId,
|
||||
snapshot.coveredSequence()));
|
||||
events.addAll(tailEvents(tenantId, sessionId,
|
||||
snapshot.coveredSequence(), visibleSequence));
|
||||
return new WebShellTranscript(snapshot.items().stream()
|
||||
.map(this::webShellItem).toList(),
|
||||
events.stream().map(this::webShellEvent).toList(),
|
||||
snapshot.coveredSequence(), null, false,
|
||||
visibleSequence);
|
||||
}
|
||||
}
|
||||
EventPage page = store.findTranscriptEvents(tenantId, sessionId,
|
||||
transcriptCursor(cursor), limit);
|
||||
List<WebShellEvent> events = page.events().stream()
|
||||
.map(this::webShellEvent).toList();
|
||||
String olderCursor = page.hasMore() && !events.isEmpty()
|
||||
? Long.toString(events.get(0).sequence()) : null;
|
||||
return new WebShellTranscript(List.of(), events, 0, olderCursor,
|
||||
page.hasMore(), session.lastSequence());
|
||||
}
|
||||
|
||||
public WebShellExportPage recentTranscriptItems(String tenantId, String actorId,
|
||||
String sessionId, String cursor, int requestedLimit) {
|
||||
requireReadableSession(tenantId, actorId, sessionId);
|
||||
var page = store.findRecentSnapshotPage(tenantId, sessionId, cursor, limit(requestedLimit));
|
||||
return new WebShellExportPage(page.items().stream().map(this::webShellItem).toList(),
|
||||
page.nextCursor(), page.hasMore(), page.coveredSequence());
|
||||
}
|
||||
|
||||
public WebShellTranscript messageTranscript(String tenantId, String actorId,
|
||||
String sessionId, String cursor, int requestedLimit) {
|
||||
requireReadableSession(tenantId, actorId, sessionId);
|
||||
var page = store.findMessagePage(tenantId, sessionId, cursor, requestedLimit, true);
|
||||
return new WebShellTranscript(page.items().stream().map(this::webShellItem).toList(),
|
||||
@@ -1215,7 +1168,12 @@ public class ManagedAgentService {
|
||||
page.nextCursor(), page.hasMore(), page.coveredSequence());
|
||||
}
|
||||
|
||||
public WebShellExportPage messageExportPage(String tenantId, String actorId,
|
||||
public WebShellExportPage recentTranscriptItems(String tenantId, String actorId,
|
||||
String sessionId, String cursor, int requestedLimit) {
|
||||
return messageExportPage(tenantId, actorId, sessionId, cursor, requestedLimit, true);
|
||||
}
|
||||
|
||||
private WebShellExportPage messageExportPage(String tenantId, String actorId,
|
||||
String sessionId, String cursor, int requestedLimit, boolean backward) {
|
||||
requireReadableSession(tenantId, actorId, sessionId);
|
||||
var page = store.findMessagePage(tenantId, sessionId, cursor, requestedLimit, backward);
|
||||
@@ -1223,14 +1181,7 @@ public class ManagedAgentService {
|
||||
page.nextCursor(), page.hasMore(), page.throughSequence());
|
||||
}
|
||||
|
||||
// Events are read before the floor: a floor that is not above the cursor
|
||||
// after the read was not above it during the read either, so no event
|
||||
// after the cursor had been pruned. A stream reconciliation discards the
|
||||
// Snapshot without lowering the floor, so the floor can exceed the
|
||||
// coverage while the Items rebuild; nothing prunes events yet, so such
|
||||
// a cursor is still served instead of looping 409/resync until the
|
||||
// rebuild finishes. The pruning work must revisit this condition once
|
||||
// rows below the floor can be gone.
|
||||
// Check the floor after reading events so a concurrent floor advance forces resync.
|
||||
private List<EventRecord> replayableEvents(SessionRecord session,
|
||||
long afterSequence, int limit) {
|
||||
List<EventRecord> events = store.findEvents(session.tenantId(),
|
||||
@@ -1269,7 +1220,7 @@ public class ManagedAgentService {
|
||||
Map<String, TurnSummary> activeTurns = store.findActiveTurns(tenantId,
|
||||
ids);
|
||||
Map<String, Long> coveredSequences = store
|
||||
.findSnapshotCoveredSequences(tenantId, ids);
|
||||
.findMessageCoveredSequences(tenantId, ids);
|
||||
Set<String> closed = completedWorkspaceCloses(tenantId, sessions);
|
||||
var compressions = store.contextCompressionStates(tenantId, sessions.stream()
|
||||
.filter(session -> session.workspace() != null).map(SessionRecord::sessionId).toList());
|
||||
@@ -1659,29 +1610,6 @@ public class ManagedAgentService {
|
||||
part.firstSequence(), part.lastSequence());
|
||||
}
|
||||
|
||||
private List<EventRecord> tailEvents(String tenantId, String sessionId,
|
||||
long afterSequence, long throughSequence) {
|
||||
List<EventRecord> result = new ArrayList<>();
|
||||
long cursor = afterSequence;
|
||||
while (cursor < throughSequence) {
|
||||
// Paged at the snapshot gate's lag bound: a session whose
|
||||
// snapshot lags within the gate is served in one round trip.
|
||||
List<EventRecord> page = store.findEvents(tenantId, sessionId,
|
||||
cursor, ManagedAgentStore.SNAPSHOT_REFRESH_EVENTS);
|
||||
if (page.isEmpty()) {
|
||||
break;
|
||||
}
|
||||
for (EventRecord event : page) {
|
||||
if (event.sequence() > throughSequence) {
|
||||
return List.copyOf(result);
|
||||
}
|
||||
result.add(event);
|
||||
cursor = event.sequence();
|
||||
}
|
||||
}
|
||||
return List.copyOf(result);
|
||||
}
|
||||
|
||||
private void dispatch(String tenantId, Admission admission) {
|
||||
dispatch(tenantId, admission, false);
|
||||
}
|
||||
@@ -2009,7 +1937,7 @@ public class ManagedAgentService {
|
||||
* Reads the next catch-up page of a stream.
|
||||
*
|
||||
* @throws ReplayCursorExpired when the cursor is below the replay floor
|
||||
* while the Snapshot backs it
|
||||
* when the message projection supplies the resynchronization state
|
||||
*/
|
||||
List<EventRecord> streamEvents(SessionRecord session, long afterSequence) {
|
||||
requireEventCursor(afterSequence);
|
||||
@@ -2415,22 +2343,6 @@ public class ManagedAgentService {
|
||||
"Turn cursor is invalid.");
|
||||
}
|
||||
|
||||
private static Long transcriptCursor(String cursor) {
|
||||
if (StringUtils.isBlank(cursor)) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
long value = Long.parseLong(cursor);
|
||||
if (value <= 0) {
|
||||
throw new NumberFormatException();
|
||||
}
|
||||
return value;
|
||||
} catch (NumberFormatException error) {
|
||||
throw new ManagedAgentException(HttpStatus.BAD_REQUEST,
|
||||
"invalid_cursor", "Transcript cursor is invalid.");
|
||||
}
|
||||
}
|
||||
|
||||
private static CommandAdmission response(Admission admission) {
|
||||
return new CommandAdmission(admission.sessionId(),
|
||||
admission.turnId(), "accepted", admission.replayed());
|
||||
|
||||
+3
-5
@@ -7,11 +7,9 @@ import org.springframework.http.HttpStatus;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* A replay cursor below the Session's replay floor while the Snapshot backs
|
||||
* the floor; a stream reconciliation discards the Snapshot without lowering
|
||||
* the floor, and during the rebuild such a cursor is still served. JSON reads
|
||||
* answer it with {@code 409 cursor_expired}; streams send one resync frame
|
||||
* instead.
|
||||
* A replay cursor below the durable floor requires reloading the live message
|
||||
* projection. JSON reads answer with {@code 409 cursor_expired}; streams send
|
||||
* one resync frame. A rebuilding projection can temporarily require a retry.
|
||||
*/
|
||||
public final class ReplayCursorExpired extends ManagedAgentException {
|
||||
private final transient ReplayWindow window;
|
||||
|
||||
+2
-11
@@ -11,15 +11,7 @@ import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
|
||||
/**
|
||||
* Raises each Session's replay floor as far as its Snapshot proves safe, the
|
||||
* first half of the Session event retention work (#12380): events at or below
|
||||
* the floor may be pruned later, and a replay cursor below it expires while
|
||||
* the Snapshot backs the floor.
|
||||
* Nothing deletes events here, and the pass is disabled until a deployment
|
||||
* opts in; the retention window and the reader-aware cleanup remain separate
|
||||
* decisions.
|
||||
*/
|
||||
/** Advances replay floors through message projection coverage when explicitly enabled. */
|
||||
@Component
|
||||
public class ReplayFloorAdvancer {
|
||||
private static final Logger LOG = LoggerFactory.getLogger(
|
||||
@@ -57,8 +49,7 @@ public class ReplayFloorAdvancer {
|
||||
}
|
||||
fresh = true;
|
||||
try {
|
||||
// MAX_VALUE asks for as far as the Snapshot proves safe;
|
||||
// the store clamps the floor to the covered sequence.
|
||||
// The store clamps the requested floor to readable projection coverage.
|
||||
store.advanceReplayFloor(target.tenantId(),
|
||||
target.sessionId(), Long.MAX_VALUE);
|
||||
} catch (RuntimeException error) {
|
||||
|
||||
+1
-10
@@ -418,16 +418,7 @@ final class ManagedSchemaCatalog {
|
||||
List<String> expectedTokens = numericBooleans(expressionTokens(expression));
|
||||
List<String> actualTokens =
|
||||
stored == null ? List.of() : numericBooleans(expressionTokens(stored));
|
||||
boolean same = expectedTokens.equals(actualTokens);
|
||||
if (!same && "projection_pending".equals(column.group(1))) {
|
||||
int split = expectedTokens.indexOf("or");
|
||||
List<String> commuted =
|
||||
new ArrayList<>(expectedTokens.subList(split + 1, expectedTokens.size()));
|
||||
commuted.add("or");
|
||||
commuted.addAll(expectedTokens.subList(0, split));
|
||||
same = commuted.equals(actualTokens);
|
||||
}
|
||||
if (!same) {
|
||||
if (!expectedTokens.equals(actualTokens)) {
|
||||
throw new SQLException(
|
||||
"Managed schema generated column differs: " + name + "." + column.group(1));
|
||||
}
|
||||
|
||||
-2
@@ -17,7 +17,6 @@ public final class ManagedSchemaContract {
|
||||
"ma_event_stream",
|
||||
"ma_public_event",
|
||||
"ma_message",
|
||||
"ma_snapshot",
|
||||
"ma_session_record",
|
||||
"ma_agent_definition",
|
||||
"ma_extension_package",
|
||||
@@ -112,7 +111,6 @@ public final class ManagedSchemaContract {
|
||||
|
||||
public enum ResourceOwnerKind {
|
||||
JOURNAL,
|
||||
SNAPSHOT,
|
||||
NATIVE_SNAPSHOT,
|
||||
CAPTURE,
|
||||
ARTIFACT,
|
||||
|
||||
+2
-7
@@ -22,7 +22,7 @@ public enum ManagedSchemaProjection {
|
||||
EVENT_STREAM(
|
||||
"tenant_id, session_id, stream_kind, stream_id, last_sequence, "
|
||||
+ "replay_floor_sequence, covered_sequence, wake_covered_sequence, projection_version, "
|
||||
+ "snapshot_stale_since, current_snapshot_id, artifact_refs_resource_key, "
|
||||
+ "artifact_refs_resource_key, "
|
||||
+ "output_ranges_json, projection_pending, gmt_modified"),
|
||||
MESSAGE(
|
||||
"tenant_id, session_id, item_id, turn_id, item_type, item_role, item_status, "
|
||||
@@ -74,8 +74,7 @@ public enum ManagedSchemaProjection {
|
||||
RESOURCE(
|
||||
"resource_key, session_scope_key, tenant_id, workspace_id, session_id, resource_id, "
|
||||
+ "kind, schema_version, byte_length, sha256, storage_kind, inline_bytes, object_key, "
|
||||
+ "object_version_id, encryption_key_id, publish_command_id, state, snapshot_id, "
|
||||
+ "snapshot_page_ordinal, snapshot_page_first_sequence, snapshot_page_last_sequence, "
|
||||
+ "object_version_id, encryption_key_id, publish_command_id, state, "
|
||||
+ "scope_key, publication_id, slot_key, capture_state, capture_object_key, "
|
||||
+ "operation_id, quarantine_reason, deletion_operation_key, gmt_create, "
|
||||
+ "last_verified_at, retention_until"),
|
||||
@@ -151,10 +150,6 @@ public enum ManagedSchemaProjection {
|
||||
+ "started_at, settled_at, first_sequence, hook_once_key_hash, hook_occurrence_hash, "
|
||||
+ "hook_ordinal, hook_definition_hash, record_kind, state, action_id, action_version, "
|
||||
+ "decision_receipt_id, decision_digest, prompt_id, turn_id, options_json"),
|
||||
SNAPSHOT(
|
||||
"tenant_id, session_id, snapshot_id, snapshot_version, covered_sequence, "
|
||||
+ "projection_version, manifest_resource_key, manifest_sha256, item_count, state, "
|
||||
+ "gmt_create, published_at, retain_until"),
|
||||
STORAGE(
|
||||
"alias_key, tenant_key, storage_key, tenant_id, storage_id, physical_key, "
|
||||
+ "registration_state, storage_kind, registration_revision, registration_json, "
|
||||
|
||||
-2
@@ -60,8 +60,6 @@ public final class ManagedSchemaRows {
|
||||
"artifact_id",
|
||||
"result_id",
|
||||
"occurrence_key",
|
||||
"snapshot_id",
|
||||
"current_snapshot_id",
|
||||
"turn_id",
|
||||
"prompt_id",
|
||||
"item_id",
|
||||
|
||||
+2
-41
@@ -255,11 +255,9 @@ CREATE TABLE ma_event_stream (
|
||||
covered_sequence BIGINT NOT NULL DEFAULT 0,
|
||||
wake_covered_sequence BIGINT,
|
||||
projection_version INT NOT NULL DEFAULT 1,
|
||||
snapshot_stale_since BIGINT UNSIGNED,
|
||||
current_snapshot_id VARBINARY(512),
|
||||
artifact_refs_resource_key CHAR(64),
|
||||
output_ranges_json LONGTEXT,
|
||||
projection_pending BOOLEAN GENERATED ALWAYS AS (last_sequence > covered_sequence OR snapshot_stale_since IS NOT NULL),
|
||||
projection_pending BOOLEAN GENERATED ALWAYS AS (last_sequence > covered_sequence),
|
||||
gmt_modified BIGINT UNSIGNED NOT NULL,
|
||||
PRIMARY KEY (tenant_id, session_id, stream_kind, stream_id),
|
||||
CONSTRAINT ck_ma_stream_kind CHECK ((CASE stream_kind WHEN 'SESSION' THEN 7 WHEN 'TASK' THEN 4 ELSE -1 END = CHAR_LENGTH(stream_kind))),
|
||||
@@ -338,29 +336,6 @@ CREATE INDEX ix_ma_message_history ON ma_message (tenant_id, session_id, first_s
|
||||
|
||||
CREATE INDEX ix_ma_message_execution ON ma_message (execution_key, item_id);
|
||||
|
||||
CREATE TABLE ma_snapshot (
|
||||
tenant_id VARBINARY(512) NOT NULL,
|
||||
session_id VARBINARY(512) NOT NULL,
|
||||
snapshot_id VARBINARY(512) NOT NULL,
|
||||
snapshot_version BIGINT NOT NULL,
|
||||
covered_sequence BIGINT NOT NULL,
|
||||
projection_version INT NOT NULL,
|
||||
manifest_resource_key CHAR(64) NOT NULL,
|
||||
manifest_sha256 CHAR(64) NOT NULL,
|
||||
item_count BIGINT NOT NULL,
|
||||
state VARCHAR(16) NOT NULL,
|
||||
gmt_create BIGINT UNSIGNED NOT NULL,
|
||||
published_at BIGINT UNSIGNED,
|
||||
retain_until BIGINT UNSIGNED NOT NULL,
|
||||
PRIMARY KEY (tenant_id, session_id, snapshot_id),
|
||||
CONSTRAINT uq_ma_snapshot_version UNIQUE (tenant_id, session_id, snapshot_version),
|
||||
CONSTRAINT ck_ma_snapshot_state CHECK ((CASE state WHEN 'BUILDING' THEN 8 WHEN 'PUBLISHED' THEN 9 WHEN 'RETIRED' THEN 7 ELSE -1 END = CHAR_LENGTH(state))),
|
||||
CONSTRAINT ck_ma_snapshot_revision CHECK (snapshot_version > 0 AND covered_sequence >= 0 AND item_count >= 0),
|
||||
CONSTRAINT fk_ma_snapshot_session FOREIGN KEY (tenant_id, session_id) REFERENCES ma_session (tenant_id, session_id)
|
||||
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
|
||||
|
||||
CREATE INDEX ix_ma_snapshot_expiry ON ma_snapshot (state, retain_until);
|
||||
|
||||
CREATE TABLE ma_session_record (
|
||||
session_scope_key CHAR(64) NOT NULL,
|
||||
record_key CHAR(64) NOT NULL,
|
||||
@@ -443,10 +418,6 @@ CREATE TABLE ma_resource (
|
||||
encryption_key_id VARBINARY(2048),
|
||||
publish_command_id VARBINARY(2048),
|
||||
state VARCHAR(32) NOT NULL,
|
||||
snapshot_id VARBINARY(512),
|
||||
snapshot_page_ordinal INT,
|
||||
snapshot_page_first_sequence BIGINT,
|
||||
snapshot_page_last_sequence BIGINT,
|
||||
scope_key CHAR(64),
|
||||
publication_id VARBINARY(512),
|
||||
slot_key VARBINARY(1024),
|
||||
@@ -461,8 +432,6 @@ CREATE TABLE ma_resource (
|
||||
PRIMARY KEY (resource_key),
|
||||
CONSTRAINT uq_ma_resource_id UNIQUE (session_scope_key, resource_id),
|
||||
CONSTRAINT uq_ma_capture_slot UNIQUE (scope_key, publication_id, slot_key),
|
||||
CONSTRAINT uq_ma_snapshot_page UNIQUE (session_scope_key, snapshot_id, snapshot_page_ordinal),
|
||||
CONSTRAINT ck_ma_resource_snapshot_page CHECK ((kind <> X'6d616e616765642d736e617073686f742d70616765' AND snapshot_id IS NULL AND snapshot_page_ordinal IS NULL AND snapshot_page_first_sequence IS NULL AND snapshot_page_last_sequence IS NULL) OR (snapshot_id IS NOT NULL AND snapshot_page_ordinal IS NOT NULL AND snapshot_page_ordinal >= 0 AND snapshot_page_first_sequence IS NOT NULL AND snapshot_page_first_sequence >= 0 AND snapshot_page_last_sequence IS NOT NULL AND snapshot_page_last_sequence >= snapshot_page_first_sequence AND kind = X'6d616e616765642d736e617073686f742d70616765' AND schema_version = 1 AND storage_kind = 'MYSQL_INLINE')),
|
||||
CONSTRAINT ck_ma_resource_length CHECK (byte_length >= 0),
|
||||
CONSTRAINT ck_ma_resource_capture CHECK ((publication_id IS NULL AND scope_key IS NULL AND slot_key IS NULL AND capture_state IS NULL) OR (publication_id IS NOT NULL AND scope_key IS NOT NULL AND slot_key IS NOT NULL AND capture_state IS NOT NULL))
|
||||
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
|
||||
@@ -475,8 +444,6 @@ CREATE INDEX ix_ma_resource_capture ON ma_resource (scope_key, publication_id, c
|
||||
|
||||
CREATE INDEX ix_ma_resource_delete_intent ON ma_resource (session_scope_key, state, deletion_operation_key, resource_key);
|
||||
|
||||
CREATE INDEX ix_ma_snapshot_page_lookup ON ma_resource (session_scope_key, snapshot_id, snapshot_page_last_sequence, snapshot_page_ordinal);
|
||||
|
||||
CREATE TABLE ma_resource_ref (
|
||||
session_scope_key CHAR(64) NOT NULL,
|
||||
owner_kind VARCHAR(32) NOT NULL,
|
||||
@@ -490,7 +457,7 @@ CREATE TABLE ma_resource_ref (
|
||||
resource_id VARBINARY(2048),
|
||||
gmt_create BIGINT UNSIGNED NOT NULL,
|
||||
PRIMARY KEY (session_scope_key, owner_kind, owner_key, owner_revision, resource_key),
|
||||
CONSTRAINT ck_ma_ref_owner CHECK ((CASE owner_kind WHEN 'JOURNAL' THEN 7 WHEN 'SNAPSHOT' THEN 8 WHEN 'NATIVE_SNAPSHOT' THEN 15 WHEN 'CAPTURE' THEN 7 WHEN 'ARTIFACT' THEN 8 WHEN 'MESSAGE' THEN 7 WHEN 'SESSION_RECORD' THEN 14 WHEN 'EVENT_STREAM' THEN 12 WHEN 'OPERATION' THEN 9 WHEN 'OPERATION_WORK' THEN 14 WHEN 'RECOVERY' THEN 8 WHEN 'HOSTED_CLEANUP' THEN 14 ELSE -1 END = CHAR_LENGTH(owner_kind))),
|
||||
CONSTRAINT ck_ma_ref_owner CHECK ((CASE owner_kind WHEN 'JOURNAL' THEN 7 WHEN 'NATIVE_SNAPSHOT' THEN 15 WHEN 'CAPTURE' THEN 7 WHEN 'ARTIFACT' THEN 8 WHEN 'MESSAGE' THEN 7 WHEN 'SESSION_RECORD' THEN 14 WHEN 'EVENT_STREAM' THEN 12 WHEN 'OPERATION' THEN 9 WHEN 'OPERATION_WORK' THEN 14 WHEN 'RECOVERY' THEN 8 WHEN 'HOSTED_CLEANUP' THEN 14 ELSE -1 END = CHAR_LENGTH(owner_kind))),
|
||||
CONSTRAINT fk_ma_ref_resource FOREIGN KEY (resource_key) REFERENCES ma_resource (resource_key),
|
||||
runtime_binding_key CHAR(64),
|
||||
CONSTRAINT ck_ma_ref_binding CHECK ((owner_kind = 'HOSTED_CLEANUP' AND runtime_binding_key IS NOT NULL AND owner_revision > 0) OR (owner_kind <> 'HOSTED_CLEANUP' AND runtime_binding_key IS NULL))
|
||||
@@ -1214,8 +1181,6 @@ CREATE INDEX ix_ma_occurrence_list ON ma_schedule_occurrence (tenant_id, schedul
|
||||
|
||||
CREATE INDEX ix_ma_occurrence_outcome ON ma_schedule_occurrence (tenant_id, schedule_id, outcome);
|
||||
|
||||
ALTER TABLE ma_snapshot ADD CONSTRAINT fk_ma_snapshot_manifest_resource_key FOREIGN KEY (manifest_resource_key) REFERENCES ma_resource (resource_key);
|
||||
|
||||
ALTER TABLE ma_message ADD CONSTRAINT fk_ma_message_body_resource_key FOREIGN KEY (body_resource_key) REFERENCES ma_resource (resource_key);
|
||||
|
||||
ALTER TABLE ma_output_capture ADD CONSTRAINT fk_ma_output_capture_manifest_resource_key FOREIGN KEY (manifest_resource_key) REFERENCES ma_resource (resource_key);
|
||||
@@ -1248,10 +1213,6 @@ CREATE INDEX ix_ma_operation_session_state ON ma_operation (session_scope_key, o
|
||||
|
||||
CREATE INDEX ix_ma_resource_scope_page ON ma_resource (session_scope_key, resource_key);
|
||||
|
||||
CREATE INDEX ix_ma_snapshot_session_expiry ON ma_snapshot (tenant_id, session_id, retain_until, snapshot_id);
|
||||
|
||||
ALTER TABLE ma_event_stream ADD CONSTRAINT fk_ma_stream_snapshot FOREIGN KEY (tenant_id, session_id, current_snapshot_id) REFERENCES ma_snapshot (tenant_id, session_id, snapshot_id);
|
||||
|
||||
ALTER TABLE ma_event_stream ADD CONSTRAINT fk_ma_stream_artifact_refs FOREIGN KEY (artifact_refs_resource_key) REFERENCES ma_resource (resource_key);
|
||||
|
||||
CREATE UNIQUE INDEX uq_ma_delivery_reconciliation ON ma_delivery (reconcile_operation_id);
|
||||
|
||||
+96
-73
@@ -2,8 +2,8 @@
|
||||
"openapi": "3.1.0",
|
||||
"info": {
|
||||
"title": "Qwen Managed Agent Public and WebShell API",
|
||||
"version": "1.97.0",
|
||||
"description": "Canonical first-release contract for the public Managed Agent resources and the WebShell adapter. Implemented routes and fields describe the current matching Server, Broker, Harness and WebShell build. Partial entries have remaining implementation or contract-test work; planned entries are not current server capabilities and must be excluded from production SDKs. Turn submission and cancellation always return the durable operation identifier and its persisted admission and delivery state. Session-list cursors are bound to the normalized archive state and title query. Earlier development API shapes, checkpoints and database catalogs are unsupported; initialize fresh state for this build. Runtime feature capabilities and deployment policy still govern optional execution modes."
|
||||
"version": "1.98.0",
|
||||
"description": "Canonical first-release contract for the public Managed Agent resources and the WebShell adapter. Implemented routes and fields describe the current matching Server, Broker, Harness and WebShell build. Partial entries have remaining implementation or contract-test work; planned entries are not current server capabilities and must be excluded from production SDKs. Turn submission and cancellation always return the durable operation identifier and its persisted admission and delivery state. Session-list cursors are bound to the normalized archive state and title query. Earlier development API shapes, checkpoints and database catalogs are unsupported; initialize fresh state for this build. Runtime feature capabilities and deployment policy still govern optional execution modes. Public conversation history reads directly from mutable message projections; persisted conversation snapshots and fixed-content paging are no longer provided."
|
||||
},
|
||||
"servers": [
|
||||
{
|
||||
@@ -506,7 +506,7 @@
|
||||
"tags": ["Public Sessions"],
|
||||
"operationId": "getSessionEvents",
|
||||
"x-qwen-implementation-status": "implemented",
|
||||
"description": "Returns JSON by default. Accept text/event-stream or stream=true to receive SSE. Last-Event-ID takes precedence over after. A cursor strictly below replay_floor_sequence gets 409 cursor_expired as JSON; SSE sends one agent.session.resync_required frame and closes. Projection rebuilds preserve the floor and current immutable Snapshot until a replacement is published, and do not make expired positions readable again.",
|
||||
"description": "Returns JSON by default. Accept text/event-stream or stream=true to receive SSE. Last-Event-ID takes precedence over after. A cursor strictly below replay_floor_sequence gets 409 cursor_expired as JSON; SSE sends one agent.session.resync_required frame and closes. Projection rebuilds preserve the floor and invalidate message-page cursors. Reload current Items after the projection catches up; expired event positions do not become readable again.",
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/AfterSequence"
|
||||
@@ -684,7 +684,7 @@
|
||||
],
|
||||
"default": 0
|
||||
},
|
||||
"description": "Initial sequence filter, or opaque next_cursor from the preceding Item page."
|
||||
"description": "Exclusive first_sequence boundary on the first page, or an opaque message cursor. Explicit cursor takes precedence."
|
||||
},
|
||||
{
|
||||
"name": "cursor",
|
||||
@@ -694,7 +694,7 @@
|
||||
"type": "string",
|
||||
"maxLength": 4096
|
||||
},
|
||||
"description": "Opaque next_cursor from the preceding Item page; when supplied, it takes precedence over after."
|
||||
"description": "Opaque current-message continuation cursor bound to scope, direction, membership range and projection generation. Old snapshot cursors are rejected."
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/ListLimit"
|
||||
@@ -702,7 +702,7 @@
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Materialized items and snapshot watermark.",
|
||||
"description": "Current message Items and the initial sequence membership upper bound.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
@@ -712,7 +712,7 @@
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"$ref": "#/components/responses/BadRequest"
|
||||
"$ref": "#/components/responses/InvalidMessagePage"
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
@@ -724,10 +724,16 @@
|
||||
"$ref": "#/components/responses/Forbidden"
|
||||
},
|
||||
"409": {
|
||||
"$ref": "#/components/responses/CursorExpired"
|
||||
"$ref": "#/components/responses/MessageProjectionChanged"
|
||||
},
|
||||
"413": {
|
||||
"$ref": "#/components/responses/MessagePageTooLarge"
|
||||
},
|
||||
"503": {
|
||||
"$ref": "#/components/responses/MessageProjectionPending"
|
||||
}
|
||||
},
|
||||
"description": "Items are read from one immutable Snapshot. Pass next_cursor as after or cursor to continue the same snapshot, even while new messages or revisions are committed. Positions are scoped to tenant and Session and expire after 30 minutes with 409 cursor_expired. A numeric after on the first page retains its sequence-filter meaning. snapshot_through_sequence remains fixed across that pagination."
|
||||
"description": "Requires current Session read authorization on every page. Read bounded current message-table pages in chronological order. The first request fixes a sequence membership upper bound; later pages retain that range but read current bodies and statuses. Numeric after excludes every Item with first_sequence at or below that value. Explicit cursor takes precedence; nonnumeric after is an existing cursor alias. Continuation uses the complete (first_sequence,item_id) position. Cursors bind tenant, Session, direction and projection generation, and are rejected after a projection rebuild. Old signed snapshot cursors are invalid. snapshot_through_sequence retains its wire name and denotes the initial membership bound, not immutable content. A lagging initial projection returns retryable message_projection_pending; canonical history and writer ownership are unchanged."
|
||||
}
|
||||
},
|
||||
"/v1/agents/sessions/{sessionId}/artifacts": {
|
||||
@@ -1232,7 +1238,7 @@
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"$ref": "#/components/responses/BadRequest"
|
||||
"$ref": "#/components/responses/InvalidMessagePage"
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
@@ -1244,23 +1250,16 @@
|
||||
"$ref": "#/components/responses/Forbidden"
|
||||
},
|
||||
"413": {
|
||||
"$ref": "#/components/responses/PayloadTooLarge"
|
||||
"$ref": "#/components/responses/MessagePageTooLarge"
|
||||
},
|
||||
"409": {
|
||||
"$ref": "#/components/responses/Conflict"
|
||||
"$ref": "#/components/responses/MessageProjectionChanged"
|
||||
},
|
||||
"503": {
|
||||
"description": "Message projection is not ready for a complete stream handoff; retry the read.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorEnvelope"
|
||||
}
|
||||
}
|
||||
}
|
||||
"$ref": "#/components/responses/MessageProjectionPending"
|
||||
}
|
||||
},
|
||||
"description": "Default view: Without a cursor, a Session that has a Snapshot returns all of its Items, the events up to the Snapshot other than turn.accepted, item.output_text.delta, item.reasoning.delta, item.tool_call.updated and item.tool_result.updated, which the Items already hold, and every event after it. Otherwise, and for an olderCursor, limit bounds the page of events. Explicit view=messages instead returns at most100 current ma_message Items using tenant/Session keyset pagination, independent of published snapshots. Bodies are live per page. coveredSequence and lastSequence are the projection sequence included by this read, so the client resumes SSE there and does not skip an unprojected tail. Only bounded lifecycle events for the selected Turns accompany the Items. An older cursor pins direction, scope, sequence membership and projection generation;409 message_projection_changed requires reloading history."
|
||||
"description": "Requires current Session read authorization. Read the most recent limit current message Items and selected bounded Turn control events; olderCursor pages toward older Items. Bodies and statuses remain live. The response coveredSequence and lastSequence both identify actual message projection coverage for SSE continuation, never an unmaterialized event tail. No persisted conversation snapshot or whole-history replay is read. Projection rebuild invalidates cursors. If projection coverage is below the replay floor, return retryable message_projection_pending."
|
||||
}
|
||||
},
|
||||
"/api/agent/web-shell/v1/events/stream": {
|
||||
@@ -1268,7 +1267,7 @@
|
||||
"tags": ["WebShell"],
|
||||
"operationId": "webShellStreamEvents",
|
||||
"x-qwen-implementation-status": "implemented",
|
||||
"description": "POST SSE consumed through fetch response.body. An afterSequence strictly below the durable replay floor ends the stream with one agent.session.resync_required frame. Projection rebuilds preserve that floor and the current immutable Snapshot. On the resync frame, reload the transcript and resume after its lastSequence.",
|
||||
"description": "POST SSE consumed through fetch response.body. An afterSequence strictly below the durable replay floor ends the stream with one agent.session.resync_required frame. Projection rebuilds preserve that floor and invalidate message-page cursors. On the resync frame, reload the transcript and resume after its lastSequence.",
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
@@ -5249,7 +5248,7 @@
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Chronological Item page: published snapshot by default, or live messages when explicitly selected.",
|
||||
"description": "Chronological page of current message Items within the initial sequence membership range; content remains live.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
@@ -5259,7 +5258,7 @@
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"$ref": "#/components/responses/BadRequest"
|
||||
"$ref": "#/components/responses/InvalidMessagePage"
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
@@ -5271,24 +5270,17 @@
|
||||
"$ref": "#/components/responses/Forbidden"
|
||||
},
|
||||
"413": {
|
||||
"$ref": "#/components/responses/PayloadTooLarge"
|
||||
"$ref": "#/components/responses/MessagePageTooLarge"
|
||||
},
|
||||
"409": {
|
||||
"$ref": "#/components/responses/CursorExpired"
|
||||
"$ref": "#/components/responses/MessageProjectionChanged"
|
||||
},
|
||||
"503": {
|
||||
"description": "Message projection has not caught up; retry the export.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorEnvelope"
|
||||
}
|
||||
}
|
||||
}
|
||||
"$ref": "#/components/responses/MessageProjectionPending"
|
||||
}
|
||||
},
|
||||
"description": "Default view: Requires current Session read authorization on every page. Reuses the public item snapshot cursor, whose subsequent pages retain the original snapshot version and watermark. No private journal, Runtime or storage credentials are returned. The client waits for a snapshot covering its captured Session watermark before using its first page. Explicit view=messages instead reads live message-table pages forward. The first read captures a finite membership bound; later body/status changes are allowed and no persistent snapshot is created or selected. snapshotThroughSequence then names that membership bound, not frozen content. If the initial message projection lags committed events,503 message_projection_pending asks the caller to retry instead of returning incomplete history. Projection-generation changes invalidate the cursor.",
|
||||
"summary": "Read one immutable public Session snapshot page for reading exports"
|
||||
"description": "Requires current Session read authorization on every page. Read current message bodies forward within the first request sequence membership bound. snapshotThroughSequence retains its wire name for that fixed bound and does not change between pages. Content and status may change; no frozen snapshot version or cursor retention is provided. An initial projection behind committed events returns retryable message_projection_pending instead of silently exporting incomplete history. No private journal, Runtime or storage credentials are returned. A projection-generation change returns 409 message_projection_changed; discard the cursor and restart this read.",
|
||||
"summary": "Read one current Session message page for exports"
|
||||
}
|
||||
},
|
||||
"/v1/agents/sessions/{sessionId}/user-messages": {
|
||||
@@ -7500,7 +7492,7 @@
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Bounded recent published Item page. Items are chronological within each page.",
|
||||
"description": "Bounded recent current message Item page. Items are chronological within each page.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
@@ -7510,7 +7502,7 @@
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"$ref": "#/components/responses/BadRequest"
|
||||
"$ref": "#/components/responses/InvalidMessagePage"
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
@@ -7522,24 +7514,17 @@
|
||||
"$ref": "#/components/responses/Forbidden"
|
||||
},
|
||||
"413": {
|
||||
"$ref": "#/components/responses/PayloadTooLarge"
|
||||
"$ref": "#/components/responses/MessagePageTooLarge"
|
||||
},
|
||||
"409": {
|
||||
"$ref": "#/components/responses/CursorExpired"
|
||||
"$ref": "#/components/responses/MessageProjectionChanged"
|
||||
},
|
||||
"503": {
|
||||
"description": "Message projection is not ready for a complete stream handoff; retry the read.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorEnvelope"
|
||||
}
|
||||
}
|
||||
}
|
||||
"$ref": "#/components/responses/MessageProjectionPending"
|
||||
}
|
||||
},
|
||||
"description": "Default view: Requires current Session read authorization on every page. The first page returns at most limit Items from the newest published snapshot, in chronological order. nextCursor pages toward older Items in that same signed, retained snapshot. snapshotThroughSequence identifies its coverage, which may lag the current Session head; an empty page with zero coverage does not certify an empty Session because no published snapshot may be available. This page does not contain live tail events or current approval/task state; use the existing transcript, event and state APIs for those. A newer snapshot does not change an existing cursor. Forward export cursors cannot be used here and backward cursors cannot be used for exports. No journal replay, writer acquisition or local cache is involved. Explicit view=messages instead reads current message-table Items backward without any published snapshot, using the same scoped keyset and projection-generation rules as messages-view transcript reads. snapshotThroughSequence then names the initial sequence membership bound; bodies can change between pages. This mode does not return lifecycle events or claim fixed snapshot content.",
|
||||
"summary": "Read recent published Session items and page toward older items"
|
||||
"description": "Requires current Session read authorization on every page. Read at most limit recent current message Items in chronological order; nextCursor continues toward older Items in the initial sequence membership range. snapshotThroughSequence names that fixed membership upper bound, while bodies and statuses remain live. Cursors bind scope, backward direction and projection generation. Rebuild invalidates the cursor. No snapshot copies are generated and no live tail is scanned; transcript reads provide actual projection coverage for SSE handoff. Projection coverage below the replay floor returns retryable message_projection_pending.",
|
||||
"summary": "Read recent current Session Items and page toward older Items"
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -7785,6 +7770,46 @@
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"MessageProjectionChanged": {
|
||||
"description": "409 message_projection_changed: the message projection generation changed. Discard the message cursor and restart this read; this is not replay-floor cursor expiration.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorEnvelope"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"MessagePageTooLarge": {
|
||||
"description": "413 message_page_too_large: selected message bodies or Turn control events exceed the response byte budget. Reducing limit can help a multi-Item page, but a single oversized Item cannot be returned even with limit=1. Signed POST requests may also return payload_too_large when their request body exceeds the configured signed-body limit.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorEnvelope"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"InvalidMessagePage": {
|
||||
"description": "400 invalid_message_cursor for malformed, wrong-scope, wrong-direction or out-of-range message cursors; invalid_limit outside 1..100; invalid_event_cursor for a negative numeric after.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorEnvelope"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"MessageProjectionPending": {
|
||||
"description": "503 message_projection_pending: projection coverage is below the replay floor, or the initial forward read is behind committed events. Retry after message projection catches up.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorEnvelope"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"schemas": {
|
||||
@@ -8228,13 +8253,14 @@
|
||||
"format": "int64",
|
||||
"minimum": 0,
|
||||
"default": 0,
|
||||
"description": "Events at or below this sequence may be pruned. A replay cursor strictly below this durable floor has expired, including after projection rebuild or removal of every retained event. The current immutable Snapshot remains available until replacement."
|
||||
"description": "Events at or below this sequence may be pruned. A replay cursor strictly below this durable floor has expired, including after projection rebuild or removal of every retained event. Reload current Items to resynchronize; projection coverage below the floor returns retryable message_projection_pending."
|
||||
},
|
||||
"snapshot_through_sequence": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"minimum": 0,
|
||||
"default": 0
|
||||
"default": 0,
|
||||
"description": "Retained wire name for current message projection coverage; no persisted conversation snapshot is implied."
|
||||
},
|
||||
"capabilities": {
|
||||
"$ref": "#/components/schemas/SessionCapabilities"
|
||||
@@ -8293,7 +8319,8 @@
|
||||
"type": "boolean"
|
||||
},
|
||||
"snapshots": {
|
||||
"type": "boolean"
|
||||
"type": "boolean",
|
||||
"description": "Retained capability name for reading the current message projection. It does not promise persisted fixed-version conversation snapshots."
|
||||
},
|
||||
"artifacts": {
|
||||
"type": "boolean"
|
||||
@@ -8426,7 +8453,7 @@
|
||||
},
|
||||
"PublicEvent": {
|
||||
"type": "object",
|
||||
"description": "A committed event, replayed with the versions and identity it was accepted with. A stream.reconciled event is the exception: it means that Harness recovery retracted earlier deltas of its Turn, which now have empty text and no item_id or content_part_id, and that later deltas may name other Parts. Reload the Items until the snapshot_through_sequence they return reaches the sequence of the stream.reconciled event, then resume after that snapshot_through_sequence, not after the stream.reconciled event; the Snapshot is rebuilt meanwhile.",
|
||||
"description": "A committed event, replayed with the versions and identity it was accepted with. A stream.reconciled event is the exception: it means that Harness recovery retracted earlier deltas of its Turn, which now have empty text and no item_id or content_part_id, and that later deltas may name other Parts. Start a fresh Items query and retry message_projection_pending until the returned snapshot_through_sequence reaches the stream.reconciled sequence. Resume after that returned sequence, which bounds the initial live-message membership range; content is not frozen.",
|
||||
"required": [
|
||||
"schema_version",
|
||||
"projection_version",
|
||||
@@ -8516,7 +8543,8 @@
|
||||
"snapshot_through_sequence": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"minimum": 0
|
||||
"minimum": 0,
|
||||
"description": "Retained wire name for current message projection coverage; no persisted conversation snapshot is implied."
|
||||
},
|
||||
"action": {
|
||||
"const": "reload_snapshot"
|
||||
@@ -8874,12 +8902,13 @@
|
||||
},
|
||||
"next_cursor": {
|
||||
"type": ["string", "null"],
|
||||
"description": "Opaque position in the same immutable Snapshot; pass it as after or cursor. Scoped to tenant and Session, with a fixed 30-minute expiry."
|
||||
"description": "Opaque current-message continuation position; pass it as after or cursor. Bound to tenant, Session, forward direction, initial membership range and projection generation. Rebuild invalidates the cursor; no fixed snapshot retention or expiry is promised."
|
||||
},
|
||||
"snapshot_through_sequence": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"minimum": 0
|
||||
"minimum": 0,
|
||||
"description": "Fixed first-request sequence membership upper bound. Bodies and statuses are read live on every page; this does not identify a retained content version."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -8959,13 +8988,8 @@
|
||||
"limit": {
|
||||
"type": ["integer", "null"],
|
||||
"minimum": 1,
|
||||
"maximum": 1000,
|
||||
"maximum": 100,
|
||||
"default": 100
|
||||
},
|
||||
"view": {
|
||||
"type": ["string", "null"],
|
||||
"enum": ["messages", null],
|
||||
"description": "Select direct live message-table pages. Omit to retain the existing snapshot/event view. Message view accepts limit1..100; cursors bind scope, direction, membership range and projection generation, not frozen bodies."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -9302,7 +9326,8 @@
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"minimum": 0,
|
||||
"x-qwen-implementation-status": "planned"
|
||||
"x-qwen-implementation-status": "planned",
|
||||
"description": "Retained wire name for current message projection coverage; no persisted conversation snapshot is implied."
|
||||
},
|
||||
"approvalMode": {
|
||||
"type": "string",
|
||||
@@ -9503,7 +9528,8 @@
|
||||
"coveredSequence": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"minimum": 0
|
||||
"minimum": 0,
|
||||
"description": "Actual message projection coverage of this read; resume SSE after this sequence."
|
||||
},
|
||||
"olderCursor": {
|
||||
"type": ["string", "null"]
|
||||
@@ -9514,7 +9540,8 @@
|
||||
"lastSequence": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"minimum": 0
|
||||
"minimum": 0,
|
||||
"description": "Actual message projection coverage for SSE handoff, equal to coveredSequence; unmaterialized tail events are delivered by SSE."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -9700,7 +9727,8 @@
|
||||
"snapshotThroughSequence": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"minimum": 0
|
||||
"minimum": 0,
|
||||
"description": "Retained wire name for current message projection coverage; no persisted conversation snapshot is implied."
|
||||
},
|
||||
"action": {
|
||||
"const": "reload_snapshot"
|
||||
@@ -15732,7 +15760,7 @@
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"minimum": 0,
|
||||
"description": " In explicit messages view, this is the fixed sequence membership upper bound; row bodies are live, not frozen snapshot content."
|
||||
"description": "Fixed first-request sequence membership upper bound. Bodies and statuses are read live on every page; this does not identify a retained content version."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -15754,11 +15782,6 @@
|
||||
"minimum": 1,
|
||||
"maximum": 100,
|
||||
"default": 100
|
||||
},
|
||||
"view": {
|
||||
"type": ["string", "null"],
|
||||
"enum": ["messages", null],
|
||||
"description": "Select direct live message-table pages. Omit to retain the existing snapshot/event view. Message view accepts limit1..100; cursors bind scope, direction, membership range and projection generation, not frozen bodies."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+1
-1
@@ -30,7 +30,7 @@ class BackfillWitnessesTest {
|
||||
+ " table_schema = 'public' AND table_type='BASE"
|
||||
+ " TABLE'",
|
||||
Integer.class))
|
||||
.isEqualTo(40);
|
||||
.isEqualTo(39);
|
||||
assertThat(
|
||||
new JdbcTemplate(source)
|
||||
.queryForObject(
|
||||
|
||||
+18
-11
@@ -560,14 +560,11 @@ class ManagedAgentApiContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void transcriptContractPromisesTheFullTailPastTheSnapshot() {
|
||||
// The cursor-less transcript serves every event after the Snapshot;
|
||||
// a server-side paging change that leaves this published sentence
|
||||
// stale must turn the suite red, so pin the sentence itself.
|
||||
void transcriptContractPromisesBoundedMessagesAndProjectedStreamHandoff() {
|
||||
String description = CONTRACT.operation("webShellTranscript").node()
|
||||
.path("description").asText();
|
||||
assertThat(description).contains("and every event after it");
|
||||
assertThat(description).contains("limit bounds the page of events");
|
||||
assertThat(description).contains("most recent limit current message Items");
|
||||
assertThat(description).contains("actual message projection coverage for SSE continuation");
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -1095,6 +1092,10 @@ class ManagedAgentApiContractTest {
|
||||
"agentId":"qwen-code","title":"web",
|
||||
"metadata":{"clientId":"contract"},"input":[]}
|
||||
""")).get("sessionId").asText();
|
||||
await().atMost(Duration.ofSeconds(5)).untilAsserted(() ->
|
||||
mvc.perform(get("/v1/agents/sessions/{id}/items", webSessionId)
|
||||
.header(TENANT, tenant))
|
||||
.andExpect(status().isOk()));
|
||||
exchange(drift, "webShellExportSession", 200,
|
||||
post(WEB_SHELL + "/sessions/export/query").header(TENANT, tenant),
|
||||
"{\"sessionId\":\"%s\",\"limit\":1}".formatted(webSessionId));
|
||||
@@ -1236,14 +1237,20 @@ class ManagedAgentApiContractTest {
|
||||
for (var operation : Map.of("webShellTranscript", "/transcript/query",
|
||||
"webShellExportSession", "/sessions/export/query",
|
||||
"webShellRecentTranscriptItems", "/transcript/items/query").entrySet()) {
|
||||
String liveRead = "{\"sessionId\":\"%s\",\"view\":\"messages\",\"limit\":1}".formatted(webSessionId);
|
||||
String liveRead = "{\"sessionId\":\"%s\",\"limit\":1}".formatted(webSessionId);
|
||||
exchange(drift, operation.getKey(), 200,
|
||||
post(WEB_SHELL + operation.getValue()).header(TENANT, tenant), liveRead);
|
||||
// The removed selector is tolerated by MVC, outside the current request schema.
|
||||
var legacy = mvc.perform(post(WEB_SHELL + operation.getValue()).header(TENANT, tenant)
|
||||
.contentType(MediaType.APPLICATION_JSON)
|
||||
.content(liveRead.replace("\"limit\":1", "\"view\":\"messages\",\"limit\":1")))
|
||||
.andReturn().getResponse();
|
||||
assertThat(legacy.getStatus()).isEqualTo(200);
|
||||
collect(drift, "legacy view response " + operation.getKey(), CONTRACT.validate(
|
||||
CONTRACT.responsePointer(CONTRACT.operation(operation.getKey()), 200),
|
||||
objectMapper.readTree(legacy.getContentAsString(StandardCharsets.UTF_8))));
|
||||
exchange(drift, operation.getKey(), 404,
|
||||
post(WEB_SHELL + operation.getValue()).header(TENANT, otherTenant), liveRead);
|
||||
exchange(drift, operation.getKey(), 400,
|
||||
post(WEB_SHELL + operation.getValue()).header(TENANT, tenant),
|
||||
liveRead.replace("\"messages\"", "\"unknown\""));
|
||||
exchange(drift, operation.getKey(), 400,
|
||||
post(WEB_SHELL + operation.getValue()).header(TENANT, tenant),
|
||||
liveRead.replace("\"limit\":1", "\"limit\":101"));
|
||||
@@ -1256,7 +1263,7 @@ class ManagedAgentApiContractTest {
|
||||
"{\"sessionId\":\"%s\"}".formatted(webSessionId));
|
||||
exchange(drift, "webShellTranscript", 200,
|
||||
post(WEB_SHELL + "/transcript/query").header(TENANT, tenant),
|
||||
"{\"sessionId\":\"%s\",\"limit\":1000}"
|
||||
"{\"sessionId\":\"%s\",\"limit\":100}"
|
||||
.formatted(webSessionId));
|
||||
|
||||
String webInputId = json(exchange(drift, "webShellCreateSession",
|
||||
|
||||
+1
-2
@@ -250,8 +250,7 @@ class ManagedAgentMySqlIT {
|
||||
.isEqualTo(new ReplayWindow(6, 6));
|
||||
assertThat(store.findReplayWindow(tenant, admission.sessionId()))
|
||||
.isEqualTo(new ReplayWindow(6, 6));
|
||||
assertThat(store.findSnapshot(tenant, admission.sessionId()))
|
||||
.get()
|
||||
assertThat(store.findMessagePage(tenant, admission.sessionId(), null, 100, false))
|
||||
.satisfies(
|
||||
snapshot -> {
|
||||
assertThat(snapshot.coveredSequence()).isEqualTo(6);
|
||||
|
||||
+31
-48
@@ -14,10 +14,10 @@ import com.alibaba.qwen.code.managedagent.model.StoreModels.DispatchTarget;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.EventRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.HarnessEvent;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.ItemRecord;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.MessagePage;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.OperationKind;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.ProjectedEvent;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SessionMutationKind;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.SnapshotRecord;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedAgentStore;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedSessionStoreModels;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedWorkspaceRegistry;
|
||||
@@ -834,9 +834,6 @@ class ManagedAgentServerIntegrationTest {
|
||||
.andExpect(jsonPath("$.environment.errorCode")
|
||||
.value("runtime_warm_failed"));
|
||||
|
||||
// The snapshot gate defers a caught-up rewrite until 5s after the
|
||||
// last one, so the trailing non-terminal event takes that long to
|
||||
// be covered.
|
||||
await().atMost(Duration.ofSeconds(15)).untilAsserted(() -> {
|
||||
MvcResult transcript = mvc.perform(post(
|
||||
"/api/agent/web-shell/v1/transcript/query")
|
||||
@@ -1063,7 +1060,7 @@ class ManagedAgentServerIntegrationTest {
|
||||
assertThat(store.findEvents(tenant, session.sessionId(), before, 100))
|
||||
.extracting(event -> event.sequence())
|
||||
.containsExactly(before + 1, before + 2, before + 3);
|
||||
assertThat(awaitSnapshotCovering(tenant, session.sessionId(),
|
||||
assertThat(awaitMessageCoverage(tenant, session.sessionId(),
|
||||
before + 3))
|
||||
.satisfies(snapshot -> {
|
||||
assertThat(snapshot.coveredSequence())
|
||||
@@ -1081,7 +1078,7 @@ class ManagedAgentServerIntegrationTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void preservesTextOrderAcrossToolsAndReasoningInSnapshots() {
|
||||
void preservesTextOrderAcrossToolsAndReasoningInMessages() {
|
||||
String tenant = "tenant-order-" + UUID.randomUUID();
|
||||
Admission session = store.insertSessionCommand(tenant,
|
||||
"CREATE_SESSION", "order-create", "digest-create",
|
||||
@@ -1099,16 +1096,12 @@ class ManagedAgentServerIntegrationTest {
|
||||
store.appendPublicEventIfAbsent(tenant, session.sessionId(), turnId,
|
||||
"item.reasoning.delta", Map.of("text", "thought"),
|
||||
false, "thought");
|
||||
// The trailing event is terminal so the explicit drain below always
|
||||
// rewrites the snapshot: the 10ms materializer may otherwise create
|
||||
// the snapshot mid-sequence, and a non-terminal catch-up inside the
|
||||
// 5s floor would legally leave the snapshot behind.
|
||||
store.appendPublicEventIfAbsent(tenant, session.sessionId(), turnId,
|
||||
"item.output_text.delta", Map.of("text", "final"),
|
||||
true, "final");
|
||||
store.materializeNextBatch(tenant, session.sessionId(), 100);
|
||||
assertThat(store.findSnapshot(tenant, session.sessionId()))
|
||||
.get().satisfies(snapshot -> assertThat(snapshot.items())
|
||||
assertThat(store.findMessagePage(tenant, session.sessionId(), null, 100, true))
|
||||
.satisfies(snapshot -> assertThat(snapshot.items())
|
||||
.filteredOn(item -> "message".equals(item.type()))
|
||||
.singleElement().satisfies(item ->
|
||||
assertThat(item.content())
|
||||
@@ -1145,8 +1138,8 @@ class ManagedAgentServerIntegrationTest {
|
||||
"item.output_text.delta", Map.of("text", "done"), true,
|
||||
"final");
|
||||
store.materializeNextBatch(tenant, session.sessionId(), 100);
|
||||
assertThat(store.findSnapshot(tenant, session.sessionId()))
|
||||
.get().satisfies(snapshot -> assertThat(snapshot.items())
|
||||
assertThat(store.findMessagePage(tenant, session.sessionId(), null, 100, true))
|
||||
.satisfies(snapshot -> assertThat(snapshot.items())
|
||||
.filteredOn(item -> "user".equals(item.role()))
|
||||
.singleElement().satisfies(item ->
|
||||
assertThat(item.content()).singleElement()
|
||||
@@ -1160,7 +1153,7 @@ class ManagedAgentServerIntegrationTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void eventsNameTheSnapshotItemsAndPartsTheyChange() {
|
||||
void eventsNameTheProjectedItemsAndPartsTheyChange() {
|
||||
String tenant = "tenant-identity-" + UUID.randomUUID();
|
||||
Admission session = store.insertSessionCommand(tenant,
|
||||
"CREATE_SESSION", "identity-create",
|
||||
@@ -1198,7 +1191,7 @@ class ManagedAgentServerIntegrationTest {
|
||||
true, "COMPLETED", null, null))));
|
||||
store.materializeNextBatch(tenant, sessionId, 200);
|
||||
|
||||
List<EventRecord> events = assertEventsNameTheSnapshot(tenant,
|
||||
List<EventRecord> events = assertEventsNameTheMessages(tenant,
|
||||
sessionId);
|
||||
List<EventRecord> deltas = events.stream()
|
||||
.filter(event -> event.type().endsWith(".delta")).toList();
|
||||
@@ -1229,16 +1222,12 @@ class ManagedAgentServerIntegrationTest {
|
||||
"item.output_text.delta", Map.of("text", text), false,
|
||||
"single:" + text);
|
||||
}
|
||||
// The trailing event is terminal so the explicit drain below always
|
||||
// rewrites the snapshot: the 10ms materializer may otherwise create
|
||||
// the snapshot mid-sequence, and a non-terminal catch-up inside the
|
||||
// 5s floor would legally leave the snapshot behind.
|
||||
store.appendPublicEventIfAbsent(tenant, sessionId, "turn-single",
|
||||
"item.reasoning.delta", Map.of("text", "c"), true,
|
||||
"single:c");
|
||||
store.materializeNextBatch(tenant, sessionId, 100);
|
||||
|
||||
List<EventRecord> deltas = assertEventsNameTheSnapshot(tenant,
|
||||
List<EventRecord> deltas = assertEventsNameTheMessages(tenant,
|
||||
sessionId).stream()
|
||||
.filter(event -> event.type().endsWith(".delta")).toList();
|
||||
assertThat(deltas).extracting(EventRecord::contentPartId)
|
||||
@@ -1293,7 +1282,7 @@ class ManagedAgentServerIntegrationTest {
|
||||
store.retractContinuationOutput(tenant, sessionId, turn.turnId(),
|
||||
owner, "boot_old", "epoch_old");
|
||||
|
||||
assertThat(assertEventsNameTheSnapshot(tenant, sessionId))
|
||||
assertThat(assertEventsNameTheMessages(tenant, sessionId))
|
||||
.filteredOn(event -> event.type().endsWith(".delta"))
|
||||
.extracting(EventRecord::itemId, EventRecord::contentPartId)
|
||||
.containsExactly(tuple(null, null), tuple(
|
||||
@@ -1302,28 +1291,22 @@ class ManagedAgentServerIntegrationTest {
|
||||
+ before.get(1).sequence()));
|
||||
}
|
||||
|
||||
// The Snapshot is published after the drain that produced it commits, and
|
||||
// a rewrite the refresh gate deferred stays behind until that floor ages
|
||||
// out, so a drain never leaves the published Snapshot current on its own.
|
||||
// Draining inside the wait keeps this independent of whether the 10ms
|
||||
// materializer or the caller's own drain consumed the batch.
|
||||
private SnapshotRecord awaitSnapshotCovering(String tenant, String sessionId,
|
||||
private MessagePage awaitMessageCoverage(String tenant, String sessionId,
|
||||
long sequence) {
|
||||
await().atMost(Duration.ofSeconds(30)).until(() -> {
|
||||
store.materializeNextBatch(tenant, sessionId, 200);
|
||||
return store.findSnapshot(tenant, sessionId)
|
||||
.filter(snapshot -> snapshot.coveredSequence() >= sequence)
|
||||
.isPresent();
|
||||
return store.findMessageCoveredSequences(tenant, List.of(sessionId))
|
||||
.getOrDefault(sessionId, 0L) >= sequence;
|
||||
});
|
||||
return store.findSnapshot(tenant, sessionId).orElseThrow();
|
||||
return store.findMessagePage(tenant, sessionId, null, 100, true);
|
||||
}
|
||||
|
||||
// Every event that names an Item or a Part names one of the Snapshot.
|
||||
private List<EventRecord> assertEventsNameTheSnapshot(String tenant,
|
||||
// Every event that names an Item or a Part names one of the projected messages.
|
||||
private List<EventRecord> assertEventsNameTheMessages(String tenant,
|
||||
String sessionId) {
|
||||
List<EventRecord> events = store.findEvents(tenant, sessionId, 0,
|
||||
100);
|
||||
List<ItemRecord> items = awaitSnapshotCovering(tenant, sessionId,
|
||||
List<ItemRecord> items = awaitMessageCoverage(tenant, sessionId,
|
||||
events.isEmpty() ? 0
|
||||
: events.get(events.size() - 1).sequence())
|
||||
.items();
|
||||
@@ -2054,8 +2037,8 @@ class ManagedAgentServerIntegrationTest {
|
||||
null, null))));
|
||||
|
||||
store.materializeNextBatch(tenant, session.sessionId(), 100);
|
||||
assertThat(store.findSnapshot(tenant, session.sessionId()))
|
||||
.isPresent();
|
||||
assertThat(store.findMessagePage(tenant, session.sessionId(), null, 100, true).items())
|
||||
.isNotEmpty();
|
||||
|
||||
store.retractContinuationOutput(tenant, session.sessionId(),
|
||||
turn.turnId(), owner, "boot_old", "epoch_old");
|
||||
@@ -2085,8 +2068,8 @@ class ManagedAgentServerIntegrationTest {
|
||||
.filteredOn(event -> "stream.reconciled".equals(event.type()))
|
||||
.hasSize(1);
|
||||
store.materializeNextBatch(tenant, session.sessionId(), 100);
|
||||
assertThat(store.findSnapshot(tenant, session.sessionId()))
|
||||
.get().satisfies(snapshot -> assertThat(snapshot.items())
|
||||
assertThat(store.findMessagePage(tenant, session.sessionId(), null, 100, true))
|
||||
.satisfies(snapshot -> assertThat(snapshot.items())
|
||||
.filteredOn(item -> "message".equals(item.type())
|
||||
&& "assistant".equals(item.role()))
|
||||
.singleElement().satisfies(item ->
|
||||
@@ -2094,7 +2077,7 @@ class ManagedAgentServerIntegrationTest {
|
||||
.satisfies(part -> assertThat(
|
||||
part.text())
|
||||
.isEqualTo("kept"))));
|
||||
assertEventsNameTheSnapshot(tenant, session.sessionId());
|
||||
assertEventsNameTheMessages(tenant, session.sessionId());
|
||||
}
|
||||
|
||||
// #13319: a restarted model attempt retracts the published prefix of the
|
||||
@@ -2137,8 +2120,8 @@ class ManagedAgentServerIntegrationTest {
|
||||
Map.of("text", "prefix"), false,
|
||||
null, null, null))));
|
||||
store.materializeNextBatch(tenant, session.sessionId(), 100);
|
||||
assertThat(store.findSnapshot(tenant, session.sessionId()))
|
||||
.isPresent();
|
||||
assertThat(store.findMessagePage(tenant, session.sessionId(), null, 100, true).items())
|
||||
.isNotEmpty();
|
||||
|
||||
store.retractHarnessTurnOutput(tenant, session.sessionId(),
|
||||
turn.turnId(), owner, "epoch_1", 3, 5);
|
||||
@@ -2171,15 +2154,15 @@ class ManagedAgentServerIntegrationTest {
|
||||
.get().satisfies(record -> assertThat(
|
||||
record.harnessLastEventId()).isEqualTo(5));
|
||||
store.materializeNextBatch(tenant, session.sessionId(), 100);
|
||||
assertThat(store.findSnapshot(tenant, session.sessionId()))
|
||||
.get().satisfies(snapshot -> assertThat(snapshot.items())
|
||||
assertThat(store.findMessagePage(tenant, session.sessionId(), null, 100, true))
|
||||
.satisfies(snapshot -> assertThat(snapshot.items())
|
||||
.filteredOn(item -> "message".equals(item.type())
|
||||
&& "assistant".equals(item.role()))
|
||||
.singleElement().satisfies(item ->
|
||||
assertThat(item.content()).singleElement()
|
||||
.satisfies(part -> assertThat(
|
||||
part.text()).isEqualTo("kept"))));
|
||||
assertEventsNameTheSnapshot(tenant, session.sessionId());
|
||||
assertEventsNameTheMessages(tenant, session.sessionId());
|
||||
|
||||
// A redelivered retraction does not retract again, but the cursor
|
||||
// still advances past it.
|
||||
@@ -2281,7 +2264,7 @@ class ManagedAgentServerIntegrationTest {
|
||||
assertThat(store.findTurn(tenant, session.sessionId(), turn.turnId())).get()
|
||||
.satisfies(record -> assertThat(record.harnessLastEventId()).isEqualTo(7));
|
||||
store.materializeNextBatch(tenant, session.sessionId(), 100);
|
||||
assertThat(store.findSnapshot(tenant, session.sessionId())).isPresent();
|
||||
assertThat(store.findMessagePage(tenant, session.sessionId(), null, 100, true).items()).isNotEmpty();
|
||||
store.recordHarnessEvents(tenant, session.sessionId(), turn.turnId(), owner, "epoch_new",
|
||||
List.of(new HarnessEvent(8, "boot_new:epoch_new:8", new ProjectedEvent("turn.completed",
|
||||
Map.of(), true, "COMPLETED", null, null))));
|
||||
@@ -2289,9 +2272,9 @@ class ManagedAgentServerIntegrationTest {
|
||||
.filteredOn(event -> "turn.completed".equals(event.type()) && turn.turnId().equals(event.turnId())).hasSize(1);
|
||||
store.materializeNextBatch(tenant, session.sessionId(), 100);
|
||||
await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->
|
||||
assertThat(store.findSnapshot(tenant, session.sessionId())).get().satisfies(snapshot ->
|
||||
assertThat(store.findMessagePage(tenant, session.sessionId(), null, 100, true)).satisfies(snapshot ->
|
||||
assertThat(snapshot.coveredSequence()).isEqualTo(store.requireSession(tenant, session.sessionId()).lastSequence())));
|
||||
assertEventsNameTheSnapshot(tenant, session.sessionId());
|
||||
assertEventsNameTheMessages(tenant, session.sessionId());
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
+7
-3
@@ -11,6 +11,7 @@ import com.alibaba.qwen.code.managedagent.artifact.ManagedArtifactPolicy;
|
||||
import com.alibaba.qwen.code.managedagent.artifact.ManagedArtifactService;
|
||||
import com.alibaba.qwen.code.managedagent.artifact.ManagedToolDetailsService;
|
||||
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector;
|
||||
import com.alibaba.qwen.code.managedagent.model.ApiModels.WebShellEvent;
|
||||
import com.alibaba.qwen.code.managedagent.model.RequestDigests;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedExtensionRecordStore;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedSessionStore;
|
||||
@@ -69,7 +70,7 @@ class ManagedArtifactApiIntegrationTest {
|
||||
var artifacts = fixture.results().listArtifacts("tenant-1", "session-1", null, null, null, 100).artifacts();
|
||||
stdout = artifacts.stream().filter(a -> "stdout".equals(a.streamId())).findFirst().orElseThrow().descriptor();
|
||||
stderr = artifacts.stream().filter(a -> "stderr".equals(a.streamId())).findFirst().orElseThrow().descriptor();
|
||||
itemId = fixture.sessions().findSnapshot("tenant-1", "session-1").orElseThrow()
|
||||
itemId = fixture.sessions().findMessagePage("tenant-1", "session-1", null, 100, false)
|
||||
.items().getFirst().itemId();
|
||||
}
|
||||
|
||||
@@ -580,8 +581,11 @@ class ManagedArtifactApiIntegrationTest {
|
||||
fixtureJson.set("toolResult", JSON.readTree(result));
|
||||
fixtureJson.set("artifactList", JSON.readTree(list));
|
||||
fixtureJson.set("transcript", JSON.valueToTree(sessions.transcript("tenant-1", "reader", "session-1", null, 100)));
|
||||
String before = Long.toString(fixture.sessions().requireSession("tenant-1", "session-1").lastSequence() + 1);
|
||||
fixtureJson.set("events", JSON.valueToTree(sessions.transcript("tenant-1", "reader", "session-1", before, 100).events()));
|
||||
var events = fixture.sessions().findEvents("tenant-1", "session-1", 0, 100);
|
||||
fixtureJson.set("events", JSON.valueToTree(events.stream().map(event -> new WebShellEvent(
|
||||
event.schemaVersion(), event.projectionVersion(), event.sequence(), event.eventId(),
|
||||
event.sessionId(), event.turnId(), event.itemId(), event.contentPartId(), event.type(),
|
||||
event.createdAt(), event.data(), event.terminal())).toList()));
|
||||
String output = System.getProperty("qwen.o3.fixture-output");
|
||||
if (output != null) {
|
||||
Files.writeString(Path.of(output), JSON.writerWithDefaultPrettyPrinter().writeValueAsString(fixtureJson));
|
||||
|
||||
+1
-2
@@ -71,7 +71,6 @@ class ManagedEventIdentityMigrationTest {
|
||||
advanced = transactions.execute(status -> store.materializeNextBatch(tenant, session, 200)).advanced();
|
||||
}
|
||||
while (advanced);
|
||||
jdbc.update("UPDATE ma_snapshot SET published_at = published_at - 6000 WHERE tenant_id = ? AND session_id = ?", tenant, session);
|
||||
transactions.executeWithoutResult(status -> store.materializeNextBatch(tenant, session, 200));
|
||||
var identities =
|
||||
jdbc.queryForList(
|
||||
@@ -82,7 +81,7 @@ class ManagedEventIdentityMigrationTest {
|
||||
session);
|
||||
assertThat(identities).hasSize(1);
|
||||
var identity = identities.getFirst();
|
||||
var item = store.findSnapshot(tenant, session).orElseThrow().items().getFirst();
|
||||
var item = store.findMessagePage(tenant, session, null, 100, false).items().getFirst();
|
||||
assertThat(item.itemId()).isEqualTo(identity.get("item_id"));
|
||||
assertThat(item.content()).hasSize(1);
|
||||
assertThat(item.content().getFirst().partId()).isEqualTo(identity.get("content_part_id"));
|
||||
|
||||
+5
-10
@@ -18,12 +18,10 @@ import java.util.UUID;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* Pins the deferral SQL on H2 in MySQL mode against the three columns the
|
||||
* Pins the deferral SQL on H2 in MySQL mode against the two columns the
|
||||
* target scan reads: {@code gmt_modified} moves so the target rotates behind
|
||||
* fresher rows, {@code covered_sequence} stays put so the gap guard in
|
||||
* materializeNextBatch still freezes on a failure, and
|
||||
* {@code snapshot_stale_since} stays put so a session whose snapshot
|
||||
* rewrite was deferred keeps being re-selected.
|
||||
* materializeNextBatch still freezes on a failure.
|
||||
*/
|
||||
class ManagedMaterializationDeferTest {
|
||||
@Test
|
||||
@@ -42,9 +40,8 @@ class ManagedMaterializationDeferTest {
|
||||
"tenant", "session", "qwen-code", "ACTIVE", 1L, 1L);
|
||||
jdbc.update("INSERT INTO ma_event_stream (tenant_id,"
|
||||
+ " session_id, stream_kind, stream_id, last_sequence, covered_sequence,"
|
||||
+ " gmt_modified, snapshot_stale_since) VALUES (?, ?,"
|
||||
+ " ?, ?, ?, ?, ?, ?)",
|
||||
"tenant", "session", "SESSION", "", 9L, 9L, 1L, 7L);
|
||||
+ " gmt_modified) VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||||
"tenant", "session", "SESSION", "", 9L, 9L, 1L);
|
||||
ManagedAgentStore store = new ManagedAgentStore(jdbc,
|
||||
new ObjectMapper(),
|
||||
Clock.fixed(Instant.ofEpochMilli(123456789L),
|
||||
@@ -54,15 +51,13 @@ class ManagedMaterializationDeferTest {
|
||||
|
||||
store.deferMaterializationTarget("tenant", "session");
|
||||
|
||||
var row = jdbc.queryForMap("SELECT covered_sequence, gmt_modified,"
|
||||
+ " snapshot_stale_since FROM"
|
||||
var row = jdbc.queryForMap("SELECT covered_sequence, gmt_modified FROM"
|
||||
+ " ma_event_stream WHERE"
|
||||
+ " tenant_id = ? AND session_id = ? AND"
|
||||
+ " stream_kind = ? AND stream_id = X''",
|
||||
"tenant", "session", "SESSION");
|
||||
assertThat(row.get("gmt_modified")).isEqualTo(123456789L);
|
||||
assertThat(row.get("covered_sequence")).isEqualTo(9L);
|
||||
assertThat(row.get("snapshot_stale_since")).isEqualTo(7L);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+1
-1
@@ -205,7 +205,7 @@ class SurfaceAdmissionAcceptanceTest {
|
||||
assertThat(artifacts).isNotEmpty();
|
||||
artifactId = artifacts.getFirst().descriptor().path("id").asText();
|
||||
artifactItemId = fixture.sessions()
|
||||
.findSnapshot(ARTIFACT_TENANT, ARTIFACT_SESSION).orElseThrow()
|
||||
.findMessagePage(ARTIFACT_TENANT, ARTIFACT_SESSION, null, 100, false)
|
||||
.items().getFirst().itemId();
|
||||
jdbc.update("INSERT INTO ma_workspace (tenant_id,"
|
||||
+ " workspace_id, workspace_generation, storage_id,"
|
||||
|
||||
+2
-2
@@ -3408,7 +3408,7 @@ class ToolPublicationStoreTest {
|
||||
publicSessions.appendPublicEventIfAbsent("tenant-1", "session-1", "turn_public_1", "item.tool_call.updated",
|
||||
Map.of("toolCallId", "model-1", "status", "in_progress"), false, "late-tool-update");
|
||||
new TransactionTemplate(manager).executeWithoutResult(status -> publicSessions.materializeNextBatch("tenant-1", "session-1", 100));
|
||||
assertThat(publicSessions.findSnapshot("tenant-1", "session-1").orElseThrow().items()).singleElement()
|
||||
assertThat(publicSessions.findMessagePage("tenant-1", "session-1", null, 100, false).items()).singleElement()
|
||||
.satisfies(item -> {
|
||||
assertThat(item.status()).isEqualTo("failed");
|
||||
assertThat(JSON.valueToTree(item.attributes()).path("result")).isEqualTo(result);
|
||||
@@ -3603,7 +3603,7 @@ class ToolPublicationStoreTest {
|
||||
.contains(descriptor);
|
||||
assertThat(publicResults.findResult("tenant-1", "other-session", descriptor.path("item_id").asText())).isEmpty();
|
||||
new TransactionTemplate(manager).executeWithoutResult(status -> publicSessions.materializeNextBatch("tenant-1", "session-1", 100));
|
||||
assertThat(publicSessions.findSnapshot("tenant-1", "session-1").orElseThrow().items()).singleElement()
|
||||
assertThat(publicSessions.findMessagePage("tenant-1", "session-1", null, 100, false).items()).singleElement()
|
||||
.satisfies(item -> assertThat(JSON.valueToTree(item.attributes()).path("result")).isEqualTo(descriptor));
|
||||
assertThat(publicSessions.findControlEvents("tenant-1", "session-1", 1)).isEmpty();
|
||||
assertThat(jdbc.queryForObject("SELECT status FROM ma_turn", String.class)).isEqualTo("COMPLETED");
|
||||
|
||||
+1
-1
@@ -81,7 +81,7 @@ class ManagedMessagePagingTest {
|
||||
assertThat(page.items()).extracting(StoreModels.ItemRecord::itemId).containsExactly(id(9999), id(10000));
|
||||
assertThat(page.hasMore()).isTrue();
|
||||
assertThat(jdbc.messageReads).isEqualTo(2);
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_snapshot", Long.class)).isZero();
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource", Long.class)).isZero();
|
||||
var older = read(page.nextCursor(), 2, true);
|
||||
assertThat(older.items()).extracting(StoreModels.ItemRecord::itemId).containsExactly(id(9997), id(9998));
|
||||
assertThat(jdbc.messageReads).isEqualTo(4);
|
||||
|
||||
+174
@@ -0,0 +1,174 @@
|
||||
package com.alibaba.qwen.code.managedagent.persistence;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchema;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaKeys;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.h2.jdbcx.JdbcDataSource;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.transaction.support.TransactionTemplate;
|
||||
|
||||
import java.sql.Connection;
|
||||
import java.sql.SQLException;
|
||||
import java.time.Clock;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class ManagedMessageSnapshotRemovalTest {
|
||||
private static final String TENANT = "snapshot-removal-tenant";
|
||||
private final ObjectMapper json = new ObjectMapper();
|
||||
private JdbcTemplate jdbc;
|
||||
private TransactionTemplate transaction;
|
||||
private DataSourceTransactionManager manager;
|
||||
private Connection keepAlive;
|
||||
private ManagedAgentStore store;
|
||||
private String session;
|
||||
|
||||
@BeforeEach
|
||||
void setup() throws SQLException {
|
||||
var source = new JdbcDataSource();
|
||||
source.setURL("jdbc:h2:mem:snapshot-removal-" + UUID.randomUUID()
|
||||
+ ";MODE=MySQL;DATABASE_TO_LOWER=TRUE;LOCK_TIMEOUT=10000");
|
||||
keepAlive = source.getConnection();
|
||||
ManagedSchema.initialize(source);
|
||||
jdbc = new ManagedSchemaJdbcTemplate(source);
|
||||
manager = new DataSourceTransactionManager(source);
|
||||
transaction = new TransactionTemplate(manager);
|
||||
store = new ManagedAgentStore(jdbc, json, Clock.systemUTC(), ignored -> {},
|
||||
new ManagedWorkspaceRegistry(jdbc), new ManagedAgentProperties());
|
||||
session = transaction.execute(status -> store.insertSessionCommand(TENANT, "actor",
|
||||
"CREATE_SESSION", "create", "digest", "qwen-code", null, "title", List.of(), null))
|
||||
.sessionId();
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
void close() throws SQLException {
|
||||
keepAlive.close();
|
||||
}
|
||||
|
||||
@Test
|
||||
void initializedSchemaOmitsOnlyPublicMessageSnapshotStorage() {
|
||||
assertThat(jdbc.queryForList("SELECT table_name FROM information_schema.tables"
|
||||
+ " WHERE table_schema = 'public'", String.class))
|
||||
.contains("ma_message", "ma_event_stream", "ma_resource", "ma_resource_ref",
|
||||
"ma_session_head", "ma_session_journal_tx")
|
||||
.doesNotContain("ma_snapshot");
|
||||
assertThat(columns("ma_event_stream"))
|
||||
.contains("covered_sequence", "projection_version", "replay_floor_sequence")
|
||||
.doesNotContain("current_snapshot_id", "snapshot_stale_since");
|
||||
assertThat(columns("ma_resource"))
|
||||
.doesNotContain("snapshot_id", "snapshot_page_ordinal",
|
||||
"snapshot_page_first_sequence", "snapshot_page_last_sequence");
|
||||
ManagedSchema.validate(jdbc.getDataSource());
|
||||
}
|
||||
|
||||
@Test
|
||||
void materializationAndReplayCoverageWorkWithoutPersistedSnapshotCopies() throws Exception {
|
||||
event("message", "before");
|
||||
materialize();
|
||||
long covered = store.requireSession(TENANT, session).lastSequence();
|
||||
event("message", "after");
|
||||
long head = store.requireSession(TENANT, session).lastSequence();
|
||||
|
||||
assertThat(head).isGreaterThan(covered);
|
||||
assertThat(store.findReplayWindow(TENANT, session).snapshotThroughSequence())
|
||||
.isEqualTo(covered);
|
||||
assertThat(store.findMessageCoveredSequences(TENANT, List.of(session)))
|
||||
.containsEntry(session, covered);
|
||||
|
||||
materialize();
|
||||
assertThat(store.findReplayWindow(TENANT, session).snapshotThroughSequence()).isEqualTo(head);
|
||||
assertThat(store.findMessageCoveredSequences(TENANT, List.of(session)))
|
||||
.containsEntry(session, head);
|
||||
var parts = json.readTree(jdbc.queryForObject("SELECT parts_json FROM ma_message"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND item_id = ?",
|
||||
String.class, TENANT, session, "message"));
|
||||
assertThat(parts.get(0).path("text").asText()).isEqualTo("beforeafter");
|
||||
assertThat(new ManagedProjectionResourceCollector(jdbc, manager).runOnce()).isFalse();
|
||||
assertNoPublicSnapshotCopies();
|
||||
}
|
||||
|
||||
@Test
|
||||
void collectorReclaimsReplacedMessageBodiesAndPreservesNativeSnapshotReferences() {
|
||||
event("large", "a".repeat(70_000));
|
||||
materialize();
|
||||
String oldBody = bodyKey();
|
||||
event("large", "b");
|
||||
materialize();
|
||||
String currentBody = bodyKey();
|
||||
assertThat(currentBody).isNotEqualTo(oldBody);
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource_ref WHERE resource_key = ?",
|
||||
Long.class, oldBody)).isZero();
|
||||
assertThat(jdbc.queryForObject("SELECT retention_until FROM ma_resource WHERE resource_key = ?",
|
||||
Long.class, oldBody)).isNotNull().isLessThanOrEqualTo(ToolPublicationRetentionStore.now(jdbc));
|
||||
|
||||
String scope = ManagedSessionStore.sessionScopeKey(TENANT, session);
|
||||
String nativeId = "native-snapshot-manifest";
|
||||
String nativeKey = ManagedSessionStore.resourceKey(scope, nativeId);
|
||||
byte[] nativeBytes = {123, 125};
|
||||
jdbc.update("INSERT INTO ma_resource (resource_key, session_scope_key, tenant_id, session_id,"
|
||||
+ " resource_id, kind, schema_version, byte_length, sha256, storage_kind,"
|
||||
+ " inline_bytes, state, gmt_create, retention_until)"
|
||||
+ " VALUES (?, ?, ?, ?, ?, ?, 1, ?, ?, 'MYSQL_INLINE', ?, 'REFERENCED', 0, 0)",
|
||||
nativeKey, scope, TENANT, session, nativeId, ManagedNativeJournalSnapshots.MANIFEST_KIND,
|
||||
nativeBytes.length, ToolPublicationContract.sha256(nativeBytes), nativeBytes);
|
||||
jdbc.update("INSERT INTO ma_resource_ref (session_scope_key, owner_kind, owner_key,"
|
||||
+ " owner_revision, resource_key, tenant_id, session_id, resource_id, gmt_create)"
|
||||
+ " VALUES (?, 'NATIVE_SNAPSHOT', ?, 1, ?, ?, ?, ?, 0)",
|
||||
scope, ManagedSchemaKeys.key(scope, nativeId), nativeKey, TENANT, session, nativeId);
|
||||
jdbc.update("UPDATE ma_resource SET retention_until = 0 WHERE resource_key = ?", currentBody);
|
||||
|
||||
assertThat(new ManagedProjectionResourceCollector(jdbc, manager).runOnce()).isTrue();
|
||||
assertThat(jdbc.queryForObject("SELECT state FROM ma_resource WHERE resource_key = ?",
|
||||
String.class, oldBody)).isEqualTo("COLLECTED");
|
||||
assertThat(jdbc.queryForObject("SELECT inline_bytes FROM ma_resource WHERE resource_key = ?",
|
||||
byte[].class, oldBody)).isNull();
|
||||
assertThat(jdbc.queryForObject("SELECT state FROM ma_resource WHERE resource_key = ?",
|
||||
String.class, currentBody)).isEqualTo("REFERENCED");
|
||||
assertThat(jdbc.queryForObject("SELECT retention_until FROM ma_resource WHERE resource_key = ?",
|
||||
Long.class, currentBody)).isNull();
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource_ref WHERE resource_key = ?",
|
||||
Long.class, currentBody)).isEqualTo(1);
|
||||
assertThat(jdbc.queryForObject("SELECT inline_bytes FROM ma_resource WHERE resource_key = ?",
|
||||
byte[].class, nativeKey)).containsExactly(nativeBytes);
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource_ref"
|
||||
+ " WHERE resource_key = ? AND owner_kind = 'NATIVE_SNAPSHOT'",
|
||||
Long.class, nativeKey)).isEqualTo(1);
|
||||
assertNoPublicSnapshotCopies();
|
||||
}
|
||||
|
||||
private List<String> columns(String table) {
|
||||
return jdbc.queryForList("SELECT column_name FROM information_schema.columns"
|
||||
+ " WHERE table_schema = 'public' AND table_name = ?", String.class, table);
|
||||
}
|
||||
|
||||
private void event(String item, String text) {
|
||||
transaction.executeWithoutResult(status -> store.appendPublicEventIfAbsent(TENANT, session,
|
||||
"turn", "item.output_text.delta", Map.of("itemId", item, "text", text),
|
||||
false, "event-" + UUID.randomUUID()));
|
||||
}
|
||||
|
||||
private void materialize() {
|
||||
transaction.execute(status -> store.materializeNextBatch(TENANT, session, 100));
|
||||
}
|
||||
|
||||
private String bodyKey() {
|
||||
return jdbc.queryForObject("SELECT body_resource_key FROM ma_message"
|
||||
+ " WHERE tenant_id = ? AND session_id = ? AND item_id = ?",
|
||||
String.class, TENANT, session, "large");
|
||||
}
|
||||
|
||||
private void assertNoPublicSnapshotCopies() {
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource WHERE kind IN (?, ?)",
|
||||
Long.class, "managed-snapshot-page", "managed-snapshot-manifest")).isZero();
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource_ref"
|
||||
+ " WHERE owner_kind = 'SNAPSHOT'", Long.class)).isZero();
|
||||
}
|
||||
}
|
||||
+3
-8
@@ -318,15 +318,10 @@ public class ManagedSchemaBusinessStoreTest {
|
||||
"INSERT INTO ma_message (tenant_id, session_id, item_id, turn_id, item_type, item_status, attributes_json, first_sequence, last_sequence, gmt_create, gmt_modified, parts_json, body_resource_key) VALUES (?, ?, 'item', 'turn', 'message', 'completed', '{}', 1, 7, 0, 0, '[]', ?)",
|
||||
tenant, session, resource);
|
||||
jdbc.update(
|
||||
"INSERT INTO ma_snapshot (tenant_id, session_id, snapshot_id, snapshot_version, covered_sequence, projection_version, manifest_resource_key, manifest_sha256, item_count, state, gmt_create, retain_until) VALUES (?, ?, 'snapshot', 1, 7, 1, ?, ?, 1, 'PUBLISHED', 0, 0)", tenant, session, resource, digest);
|
||||
for (String owner : List.of("MESSAGE", "SNAPSHOT")) {
|
||||
jdbc.update(
|
||||
"INSERT INTO ma_resource_ref (session_scope_key, owner_kind, owner_key, owner_revision, resource_key, tenant_id, session_id, resource_id, gmt_create) VALUES (?, ?, ?, 1, ?, ?, ?, ?, " + ManagedSchemaTime.SQL_NOW + ")",
|
||||
scope, owner, ManagedSchemaKeys.key(scope, owner.equals("MESSAGE") ? "item" : "snapshot"), resource,
|
||||
tenant, session, "projection-body");
|
||||
}
|
||||
"INSERT INTO ma_resource_ref (session_scope_key, owner_kind, owner_key, owner_revision, resource_key, tenant_id, session_id, resource_id, gmt_create) VALUES (?, 'MESSAGE', ?, 1, ?, ?, ?, ?, " + ManagedSchemaTime.SQL_NOW + ")",
|
||||
scope, ManagedSchemaKeys.key(scope, "item"), resource, tenant, session, "projection-body");
|
||||
jdbc.update(
|
||||
"INSERT INTO ma_event_stream (tenant_id, session_id, stream_kind, stream_id, last_sequence, replay_floor_sequence, covered_sequence, current_snapshot_id, gmt_modified) VALUES (?, ?, 'SESSION', ?, 7, 2, 5, 'snapshot', 0)",
|
||||
"INSERT INTO ma_event_stream (tenant_id, session_id, stream_kind, stream_id, last_sequence, replay_floor_sequence, covered_sequence, gmt_modified) VALUES (?, ?, 'SESSION', ?, 7, 2, 5, 0)",
|
||||
tenant, session, "");
|
||||
jdbc.update(
|
||||
"INSERT INTO ma_public_event (tenant_id, session_id, sequence_id, event_id, event_type, data_json, terminal, gmt_create) VALUES (?, ?, 7, ?, 'item.completed', '{}', FALSE, 0)", tenant, session, "event-" + tenant);
|
||||
|
||||
+5
-5
@@ -32,7 +32,7 @@ public class ManagedSchemaSessionMySqlIT extends ManagedSchemaSessionStoreTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void tenfoldHistoryKeepsDueClaimsAndSnapshotBodyReadsBounded() {
|
||||
void tenfoldHistoryKeepsDueClaimsAndMessageBodyReadsBounded() {
|
||||
var session =
|
||||
jdbc.queryForMap(
|
||||
"SELECT tenant_id, session_id FROM ma_session WHERE session_kind ="
|
||||
@@ -47,14 +47,14 @@ public class ManagedSchemaSessionMySqlIT extends ManagedSchemaSessionStoreTest {
|
||||
id,
|
||||
"turn",
|
||||
"item.output_text.delta",
|
||||
Map.of("itemId", "item", "text", "snapshot"),
|
||||
Map.of("itemId", "item", "text", "x".repeat(70_000)),
|
||||
false,
|
||||
UUID.randomUUID().toString());
|
||||
store.materializeNextBatch(tenant, id, 100);
|
||||
});
|
||||
String resource =
|
||||
jdbc.queryForObject(
|
||||
"SELECT manifest_resource_key FROM ma_snapshot WHERE tenant_id = ? AND"
|
||||
"SELECT body_resource_key FROM ma_message WHERE tenant_id = ? AND"
|
||||
+ " session_id = ?",
|
||||
String.class,
|
||||
tenant,
|
||||
@@ -74,8 +74,8 @@ public class ManagedSchemaSessionMySqlIT extends ManagedSchemaSessionStoreTest {
|
||||
assertThat(scanned(due, Long.MAX_VALUE).scannedRows()).isEqualTo(claims.scannedRows());
|
||||
assertThat(scanned(body, resource).scannedRows()).isEqualTo(reads.scannedRows());
|
||||
assertThat(store.deliverableRenames(Long.MAX_VALUE)).hasSize(1);
|
||||
assertThat(store.findSnapshotPage(tenant, id, null, 0, 10).items()).hasSize(1);
|
||||
var recent = store.findRecentSnapshotPage(tenant, id, null, 1);
|
||||
assertThat(tx.execute(status -> store.findMessagePage(tenant, id, null, 10, false)).items()).hasSize(1);
|
||||
var recent = tx.execute(status -> store.findMessagePage(tenant, id, null, 1, true));
|
||||
assertThat(recent.items()).hasSize(1);
|
||||
assertThat(recent.items().getFirst().itemId()).isEqualTo("item");
|
||||
assertThat(recent.hasMore()).isFalse();
|
||||
|
||||
+163
-291
@@ -44,6 +44,7 @@ public class ManagedSchemaSessionStoreTest {
|
||||
ManagedSchema.initialize(source);
|
||||
jdbc = new ManagedSchemaJdbcTemplate(source);
|
||||
tx = new TransactionTemplate(new DataSourceTransactionManager(source));
|
||||
tx.setIsolationLevel(org.springframework.transaction.TransactionDefinition.ISOLATION_REPEATABLE_READ);
|
||||
store =
|
||||
new ManagedAgentStore(
|
||||
jdbc,
|
||||
@@ -105,35 +106,28 @@ public class ManagedSchemaSessionStoreTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void immutablePaginationSurvivesNewMessagesAndExpires() {
|
||||
void messagePaginationKeepsItsRangeAndReadsCurrentContent() {
|
||||
for (int i = 0; i < 4; i++) {
|
||||
event("item-" + i, "text-" + i);
|
||||
}
|
||||
materialize();
|
||||
var first = store.findSnapshotPage(tenant, session, null, 0, 2);
|
||||
var first = page(null, 0, 2);
|
||||
assertThat(first.items()).hasSize(2);
|
||||
assertThat(first.hasMore()).isTrue();
|
||||
event("item-2", "changed");
|
||||
event("item-4", "new");
|
||||
clock.advance(Duration.ofSeconds(6));
|
||||
materialize();
|
||||
var next = store.findSnapshotPage(tenant, session, first.nextCursor(), 0, 2);
|
||||
assertThat(next.items())
|
||||
.extracting(StoreModels.ItemRecord::itemId)
|
||||
var next = page(first.nextCursor(), 0, 2);
|
||||
assertThat(next.items()).extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly("item-2", "item-3");
|
||||
assertThat(next.items().getFirst().content().getFirst().text()).isEqualTo("text-2");
|
||||
assertThat(next.coveredSequence()).isEqualTo(first.coveredSequence());
|
||||
assertThat(
|
||||
jdbc.queryForObject(
|
||||
"SELECT COUNT(*) FROM ma_snapshot WHERE tenant_id = ?",
|
||||
Integer.class,
|
||||
tenant))
|
||||
.isEqualTo(2);
|
||||
assertThat(store.findSnapshotPage(tenant, session, null, 0, 10).items()).hasSize(5);
|
||||
assertThat(next.items().getFirst().content().stream().map(StoreModels.ItemPartRecord::text)
|
||||
.collect(java.util.stream.Collectors.joining())).isEqualTo("text-2changed");
|
||||
assertThat(next.throughSequence()).isEqualTo(first.throughSequence());
|
||||
assertThat(next.coveredSequence()).isGreaterThan(first.coveredSequence());
|
||||
assertThat(page(null, 0, 10).items()).hasSize(5);
|
||||
clock.advance(Duration.ofMinutes(31));
|
||||
assertThatThrownBy(() -> store.findSnapshotPage(tenant, session, first.nextCursor(), 0, 2))
|
||||
.isInstanceOf(ManagedAgentException.class)
|
||||
.hasMessageContaining("expired");
|
||||
assertThat(page(first.nextCursor(), 0, 2).items()).extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly("item-2", "item-3");
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -163,8 +157,7 @@ public class ManagedSchemaSessionStoreTest {
|
||||
tenant))
|
||||
.isEqualTo("[]");
|
||||
assertThat(
|
||||
store.findSnapshot(tenant, session)
|
||||
.orElseThrow()
|
||||
messages()
|
||||
.items()
|
||||
.getFirst()
|
||||
.content()
|
||||
@@ -181,53 +174,46 @@ public class ManagedSchemaSessionStoreTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void expiredSnapshotBodiesAreCollectedWithoutRetiringAnIdleSession() {
|
||||
void releasedMessageBodiesAreCollectedOnlyWhenGcIsEnabled() {
|
||||
alignProjectionClock();
|
||||
for (int i = 0; i < 4; i++) {
|
||||
event("item-" + i, "text-" + i);
|
||||
materialize();
|
||||
clock.advance(Duration.ofSeconds(6));
|
||||
}
|
||||
jdbc.update("UPDATE ma_snapshot SET retain_until = 0 WHERE tenant_id = ?", tenant);
|
||||
event("big", "a".repeat(70_000));
|
||||
materialize();
|
||||
String original = messageBody("big");
|
||||
event("big", "b");
|
||||
materialize();
|
||||
var properties = new ManagedAgentProperties();
|
||||
var collector = new SessionResourceCollectionCollector(jdbc,
|
||||
new DataSourceTransactionManager(java.util.Objects.requireNonNull(jdbc.getDataSource())), properties);
|
||||
assertThat(collector.runOnce()).isFalse();
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_snapshot WHERE tenant_id = ?", Integer.class, tenant))
|
||||
.isEqualTo(4);
|
||||
assertThat(resourceState(original)).isEqualTo("REFERENCED");
|
||||
properties.getToolPublication().setGcEnabled(true);
|
||||
assertThat(collector.runOnce()).isTrue();
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_snapshot WHERE tenant_id = ?", Integer.class, tenant))
|
||||
.isEqualTo(1);
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource WHERE tenant_id = ? AND state = 'COLLECTED'"
|
||||
+ " AND inline_bytes IS NULL", Integer.class, tenant)).isEqualTo(6);
|
||||
assertThat(store.findSnapshot(tenant, session).orElseThrow().items()).hasSize(4);
|
||||
assertThat(resourceState(original)).isEqualTo("COLLECTED");
|
||||
assertThat(resourceState(messageBody("big"))).isEqualTo("REFERENCED");
|
||||
assertThat(messages().items()).singleElement()
|
||||
.satisfies(item -> assertThat(item.content().getFirst().text()).hasSize(70_001));
|
||||
assertThat(jdbc.queryForObject("SELECT retired_at FROM ma_session WHERE tenant_id = ? AND session_id = ?",
|
||||
Long.class, tenant, session)).isNull();
|
||||
}
|
||||
|
||||
@Test
|
||||
void validCursorRetainsOldLargeBodyUntilItsSnapshotExpires() {
|
||||
void messageCursorDoesNotPinReleasedBodiesAndReadsTheirReplacement() {
|
||||
alignProjectionClock();
|
||||
event("small", "first");
|
||||
event("big", "a".repeat(70_000));
|
||||
materialize();
|
||||
String original = messageBody("big");
|
||||
var cursor = store.findSnapshotPage(tenant, session, null, 0, 1).nextCursor();
|
||||
var first = page(null, 0, 1);
|
||||
event("big", "b".repeat(70_000));
|
||||
clock.advance(Duration.ofSeconds(6));
|
||||
materialize();
|
||||
projectionCollector().runOnce();
|
||||
assertThat(resourceState(original)).isEqualTo("REFERENCED");
|
||||
assertThat(store.findSnapshotPage(tenant, session, cursor, 0, 1).items().getFirst().content().getFirst().text())
|
||||
.isEqualTo("a".repeat(70_000));
|
||||
clock.advance(Duration.ofMinutes(31));
|
||||
jdbc.update("UPDATE ma_snapshot SET retain_until = 0 WHERE tenant_id = ?", tenant);
|
||||
projectionCollector().runOnce();
|
||||
assertThat(resourceState(original)).isEqualTo("COLLECTED");
|
||||
assertThat(resourceState(messageBody("big"))).isEqualTo("REFERENCED");
|
||||
assertThat(store.findSnapshot(tenant, session).orElseThrow().items().getLast().content().getFirst().text())
|
||||
var next = page(first.nextCursor(), 0, 1);
|
||||
assertThat(next.items().getFirst().content().getFirst().text())
|
||||
.isEqualTo("a".repeat(70_000) + "b".repeat(70_000));
|
||||
assertThat(next.throughSequence()).isEqualTo(first.throughSequence());
|
||||
assertThat(next.coveredSequence()).isGreaterThan(first.coveredSequence());
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -242,7 +228,6 @@ public class ManagedSchemaSessionStoreTest {
|
||||
var retention = new ToolPublicationRetentionStore(jdbc,
|
||||
new DataSourceTransactionManager(java.util.Objects.requireNonNull(jdbc.getDataSource())));
|
||||
try (var lease = retention.read(tenant, session)) {
|
||||
jdbc.update("UPDATE ma_snapshot SET retain_until = 0 WHERE tenant_id = ?", tenant);
|
||||
projectionCollector().runOnce();
|
||||
lease.check();
|
||||
assertThat(resourceState(original)).isEqualTo("REFERENCED");
|
||||
@@ -254,59 +239,6 @@ public class ManagedSchemaSessionStoreTest {
|
||||
assertThat(resourceState(original)).isEqualTo("COLLECTED");
|
||||
}
|
||||
|
||||
@Test
|
||||
void pausedRepeatableReadSnapshotBuildProtectsAReleasedMessageBody() throws Exception {
|
||||
try (var connection = jdbc.getDataSource().getConnection()) {
|
||||
org.junit.jupiter.api.Assumptions.assumeTrue(
|
||||
"MySQL".equals(connection.getMetaData().getDatabaseProductName()),
|
||||
"Cross-table consistent read is a real MySQL acceptance gate");
|
||||
}
|
||||
alignProjectionClock();
|
||||
event("big", "a".repeat(70_000));
|
||||
materialize();
|
||||
String body = messageBody("big");
|
||||
var original = store.findSnapshot(tenant, session).orElseThrow();
|
||||
jdbc.update("UPDATE ma_snapshot SET retain_until = 0 WHERE tenant_id = ?", tenant);
|
||||
var entered = new java.util.concurrent.CountDownLatch(1);
|
||||
var release = new java.util.concurrent.CountDownLatch(1);
|
||||
var builder = new ManagedSnapshotStore(jdbc, new ObjectMapper(), clock, (t, id, guard) -> {
|
||||
entered.countDown();
|
||||
try {
|
||||
if (!release.await(10, java.util.concurrent.TimeUnit.SECONDS)) {
|
||||
throw new IllegalStateException("projection did not commit");
|
||||
}
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException(interrupted);
|
||||
}
|
||||
guard.run();
|
||||
assertThat(jdbc.queryForObject("SELECT body_resource_key FROM ma_message WHERE tenant_id = ? AND session_id = ?",
|
||||
String.class, tenant, session)).isEqualTo(body);
|
||||
assertThat(jdbc.queryForObject("SELECT inline_bytes FROM ma_resource WHERE resource_key = ?", byte[].class, body))
|
||||
.isNotEmpty();
|
||||
return original.items();
|
||||
});
|
||||
try (var executor = java.util.concurrent.Executors.newSingleThreadExecutor()) {
|
||||
var build = executor.submit(() -> builder.build(tenant, session));
|
||||
try {
|
||||
assertThat(entered.await(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue();
|
||||
event("big", "b");
|
||||
clock.advance(Duration.ofSeconds(6));
|
||||
materialize();
|
||||
projectionCollector().runOnce();
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource_ref WHERE resource_key = ?", Integer.class, body))
|
||||
.isZero();
|
||||
assertThat(resourceState(body)).isEqualTo("REFERENCED");
|
||||
} finally {
|
||||
release.countDown();
|
||||
}
|
||||
build.get(10, java.util.concurrent.TimeUnit.SECONDS);
|
||||
}
|
||||
makeProjectionCandidatesDue();
|
||||
projectionCollector().runOnce();
|
||||
assertThat(resourceState(body)).isEqualTo("COLLECTED");
|
||||
}
|
||||
|
||||
@Test
|
||||
void projectionCleanupPreservesJournalReferencesAndRecoveryProtection() {
|
||||
alignProjectionClock();
|
||||
@@ -321,7 +253,6 @@ public class ManagedSchemaSessionStoreTest {
|
||||
event("big", "b");
|
||||
clock.advance(Duration.ofSeconds(6));
|
||||
materialize();
|
||||
jdbc.update("UPDATE ma_snapshot SET retain_until = 0 WHERE tenant_id = ?", tenant);
|
||||
projectionCollector().runOnce();
|
||||
assertThat(resourceState(original)).isEqualTo("REFERENCED");
|
||||
tx.executeWithoutResult(status -> {
|
||||
@@ -414,29 +345,25 @@ public class ManagedSchemaSessionStoreTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void modifiedCursorCannotExtendItsValidityOrChangeItsPosition() throws Exception {
|
||||
void messageCursorRejectsAnotherScopeAndInvalidUpperBound() throws Exception {
|
||||
event("a", "first");
|
||||
event("b", "second");
|
||||
materialize();
|
||||
String cursor = store.findSnapshotPage(tenant, session, null, 0, 1).nextCursor();
|
||||
String[] parts = cursor.split("\\.");
|
||||
String cursor = page(null, 0, 1).nextCursor();
|
||||
var mapper = new ObjectMapper();
|
||||
var data =
|
||||
(com.fasterxml.jackson.databind.node.ObjectNode)
|
||||
mapper.readTree(Base64.getUrlDecoder().decode(parts[0]));
|
||||
data.put("expires", clock.millis() + Duration.ofDays(365).toMillis());
|
||||
String changed =
|
||||
Base64.getUrlEncoder()
|
||||
.withoutPadding()
|
||||
.encodeToString(mapper.writeValueAsBytes(data))
|
||||
+ "."
|
||||
+ parts[1];
|
||||
assertThatThrownBy(() -> store.findSnapshotPage(tenant, session, changed, 0, 1))
|
||||
.isInstanceOf(ManagedAgentException.class)
|
||||
.hasMessageContaining("cursor");
|
||||
assertThat(store.findSnapshotPage(tenant, session, cursor, 0, 1).items())
|
||||
.extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly("b");
|
||||
for (String field : List.of("scope", "through")) {
|
||||
var data = (com.fasterxml.jackson.databind.node.ObjectNode)
|
||||
mapper.readTree(Base64.getUrlDecoder().decode(cursor));
|
||||
if (field.equals("scope")) {
|
||||
data.put(field, "0".repeat(64));
|
||||
} else {
|
||||
data.put(field, Long.MAX_VALUE);
|
||||
}
|
||||
String changed = Base64.getUrlEncoder().withoutPadding().encodeToString(mapper.writeValueAsBytes(data));
|
||||
assertThatThrownBy(() -> page(changed, 0, 1))
|
||||
.isInstanceOf(ManagedAgentException.class).hasMessageContaining("cursor");
|
||||
}
|
||||
assertThat(page(cursor, 0, 1).items()).extracting(StoreModels.ItemRecord::itemId).containsExactly("b");
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -449,9 +376,9 @@ public class ManagedSchemaSessionStoreTest {
|
||||
ManagedSessionStore.sessionScopeKey(tenant, "another-session"),
|
||||
tenant,
|
||||
"big");
|
||||
assertThatThrownBy(() -> store.findSnapshot(tenant, session))
|
||||
assertThatThrownBy(() -> messages())
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("Snapshot resource is unavailable");
|
||||
.hasMessageContaining("Message body resource is unavailable");
|
||||
event("big", "next");
|
||||
assertThatThrownBy(this::materialize)
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
@@ -459,188 +386,96 @@ public class ManagedSchemaSessionStoreTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void oldSnapshotPinsLargeMessageBodyAfterLatestMessageChanges() {
|
||||
void messageUpdatesReleaseOldBodyWithoutRetainingPageCopies() {
|
||||
event("big", "a".repeat(70_000));
|
||||
event("last", "last");
|
||||
materialize();
|
||||
var first = store.findSnapshotPage(tenant, session, null, 0, 1);
|
||||
String body =
|
||||
jdbc.queryForObject(
|
||||
"SELECT body_resource_key FROM ma_message WHERE tenant_id = ? AND item_id ="
|
||||
+ " ?",
|
||||
String.class,
|
||||
tenant,
|
||||
"big");
|
||||
var first = page(null, 0, 1);
|
||||
String body = messageBody("big");
|
||||
event("big", "b");
|
||||
clock.advance(Duration.ofSeconds(6));
|
||||
materialize();
|
||||
assertThat(
|
||||
jdbc.queryForObject(
|
||||
"SELECT COUNT(*) FROM ma_resource_ref WHERE owner_kind = 'MESSAGE'"
|
||||
+ " AND resource_key = ?",
|
||||
Integer.class,
|
||||
body))
|
||||
.isZero();
|
||||
assertThat(
|
||||
jdbc.queryForObject(
|
||||
"SELECT COUNT(*) FROM ma_resource_ref WHERE owner_kind = 'SNAPSHOT'"
|
||||
+ " AND resource_key = ?",
|
||||
Integer.class,
|
||||
body))
|
||||
.isEqualTo(1);
|
||||
assertThat(store.findSnapshotPage(tenant, session, first.nextCursor(), 0, 1).items())
|
||||
.extracting(StoreModels.ItemRecord::itemId)
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource_ref WHERE resource_key = ?",
|
||||
Integer.class, body)).isZero();
|
||||
assertThat(page(first.nextCursor(), 0, 1).items()).extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly("last");
|
||||
assertThat(
|
||||
store
|
||||
.findSnapshot(tenant, session)
|
||||
.orElseThrow()
|
||||
.items()
|
||||
.getFirst()
|
||||
.content()
|
||||
.stream()
|
||||
.map(StoreModels.ItemPartRecord::text)
|
||||
.collect(java.util.stream.Collectors.joining())
|
||||
.length())
|
||||
.isEqualTo(70_001);
|
||||
assertThat(messages().items().getFirst().content().stream().map(StoreModels.ItemPartRecord::text)
|
||||
.collect(java.util.stream.Collectors.joining())).hasSize(70_001);
|
||||
}
|
||||
|
||||
@Test
|
||||
void consistentViewPairsWatermarkAndMessagesWhileAnotherTransactionCommits() throws Exception {
|
||||
try (var connection = jdbc.getDataSource().getConnection()) {
|
||||
org.junit.jupiter.api.Assumptions.assumeTrue(
|
||||
connection.getMetaData().getDatabaseProductName().equals("MySQL"),
|
||||
"Cross-table consistent read is a real MySQL acceptance gate");
|
||||
}
|
||||
void repeatableReadPagePairsWatermarkAndMessagesWhileAnotherTransactionCommits() throws Exception {
|
||||
var readers = java.util.Arrays.stream(ManagedAgentStore.class.getMethods())
|
||||
.filter(method -> method.getName().equals("findMessagePage")).toList();
|
||||
assertThat(readers).hasSize(2).allSatisfy(method -> {
|
||||
var annotation = method.getAnnotation(org.springframework.transaction.annotation.Transactional.class);
|
||||
assertThat(annotation).isNotNull();
|
||||
assertThat(annotation.transactionManager()).isEqualTo("managedAgentTransactionManager");
|
||||
assertThat(annotation.readOnly()).isTrue();
|
||||
assertThat(annotation.isolation()).isEqualTo(org.springframework.transaction.annotation.Isolation.REPEATABLE_READ);
|
||||
});
|
||||
event("a", "before");
|
||||
materialize();
|
||||
var original = store.findSnapshot(tenant, session).orElseThrow();
|
||||
var entered = new java.util.concurrent.CountDownLatch(1);
|
||||
var release = new java.util.concurrent.CountDownLatch(1);
|
||||
var snapshot =
|
||||
new ManagedSnapshotStore(
|
||||
jdbc,
|
||||
new ObjectMapper(),
|
||||
clock,
|
||||
(t, id, guard) -> {
|
||||
entered.countDown();
|
||||
try {
|
||||
if (!release.await(10, java.util.concurrent.TimeUnit.SECONDS)) {
|
||||
throw new IllegalStateException("writer did not commit");
|
||||
}
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException(interrupted);
|
||||
}
|
||||
guard.run();
|
||||
assertThat(
|
||||
jdbc.queryForObject(
|
||||
"SELECT covered_sequence FROM ma_event_stream"
|
||||
+ " WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " AND stream_kind = 'SESSION'",
|
||||
Long.class,
|
||||
t,
|
||||
id))
|
||||
.isEqualTo(original.coveredSequence());
|
||||
assertThat(
|
||||
jdbc.queryForObject(
|
||||
"SELECT parts_json FROM ma_message WHERE"
|
||||
+ " tenant_id = ? AND session_id = ? AND"
|
||||
+ " item_id = ?",
|
||||
String.class,
|
||||
t,
|
||||
id,
|
||||
"a"))
|
||||
.contains("before")
|
||||
.doesNotContain("after");
|
||||
return original.items();
|
||||
});
|
||||
long original = messages().coveredSequence();
|
||||
var paused = new PausedMessageRead();
|
||||
try (var executor = java.util.concurrent.Executors.newSingleThreadExecutor()) {
|
||||
var build = executor.submit(() -> snapshot.build(tenant, session));
|
||||
var read = executor.submit(paused::read);
|
||||
try {
|
||||
assertThat(entered.await(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue();
|
||||
assertThat(paused.entered.await(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue();
|
||||
event("a", "after");
|
||||
materialize();
|
||||
assertThat(
|
||||
jdbc.queryForObject(
|
||||
"SELECT parts_json FROM ma_message WHERE tenant_id = ? AND"
|
||||
+ " session_id = ? AND item_id = ?",
|
||||
String.class,
|
||||
tenant,
|
||||
session,
|
||||
"a"))
|
||||
.contains("beforeafter");
|
||||
assertThat(messages().items().getFirst().content().getFirst().text()).isEqualTo("beforeafter");
|
||||
} finally {
|
||||
release.countDown();
|
||||
paused.release.countDown();
|
||||
}
|
||||
build.get(10, java.util.concurrent.TimeUnit.SECONDS);
|
||||
var page = read.get(10, java.util.concurrent.TimeUnit.SECONDS);
|
||||
assertThat(page.coveredSequence()).isEqualTo(original);
|
||||
assertThat(page.items().getFirst().content().getFirst().text()).isEqualTo("before");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void writerHeartbeatCompletesWhileSnapshotBodyConstructionIsPaused() throws Exception {
|
||||
void repeatableReadPageRetainsOldBodyViewWhileReleasedBytesAreCollected() throws Exception {
|
||||
alignProjectionClock();
|
||||
event("big", "a".repeat(70_000));
|
||||
materialize();
|
||||
String original = messageBody("big");
|
||||
var paused = new PausedMessageRead();
|
||||
try (var executor = java.util.concurrent.Executors.newSingleThreadExecutor()) {
|
||||
var read = executor.submit(paused::read);
|
||||
try {
|
||||
assertThat(paused.entered.await(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue();
|
||||
event("big", "b");
|
||||
materialize();
|
||||
projectionCollector().runOnce();
|
||||
assertThat(resourceState(original)).isEqualTo("COLLECTED");
|
||||
} finally {
|
||||
paused.release.countDown();
|
||||
}
|
||||
assertThat(read.get(10, java.util.concurrent.TimeUnit.SECONDS).items().getFirst().content().getFirst().text())
|
||||
.isEqualTo("a".repeat(70_000));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void writerHeartbeatCompletesWhileMessagePageReadIsPaused() throws Exception {
|
||||
var privateStore = new ManagedSessionStore(jdbc);
|
||||
String token = "a".repeat(32);
|
||||
var grant =
|
||||
tx.execute(
|
||||
status ->
|
||||
privateStore.acquireWriter(
|
||||
tenant,
|
||||
session,
|
||||
token,
|
||||
new ManagedSessionStoreModels.AcquireWriterRequest(
|
||||
"workspace", "writer", 60_000L)));
|
||||
var grant = tx.execute(status -> privateStore.acquireWriter(tenant, session, token,
|
||||
new ManagedSessionStoreModels.AcquireWriterRequest("workspace", "writer", 60_000L)));
|
||||
event("a", "before");
|
||||
materialize();
|
||||
var original = store.findSnapshot(tenant, session).orElseThrow();
|
||||
var entered = new java.util.concurrent.CountDownLatch(1);
|
||||
var release = new java.util.concurrent.CountDownLatch(1);
|
||||
var snapshot =
|
||||
new ManagedSnapshotStore(
|
||||
jdbc,
|
||||
new ObjectMapper(),
|
||||
clock,
|
||||
(t, id, guard) -> {
|
||||
entered.countDown();
|
||||
try {
|
||||
if (!release.await(10, java.util.concurrent.TimeUnit.SECONDS)) {
|
||||
throw new IllegalStateException("heartbeat did not complete");
|
||||
}
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException(interrupted);
|
||||
}
|
||||
guard.run();
|
||||
return original.items();
|
||||
});
|
||||
var paused = new PausedMessageRead();
|
||||
try (var executor = java.util.concurrent.Executors.newFixedThreadPool(2)) {
|
||||
var build = executor.submit(() -> snapshot.build(tenant, session));
|
||||
var read = executor.submit(paused::read);
|
||||
try {
|
||||
assertThat(entered.await(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue();
|
||||
var heartbeat =
|
||||
executor.submit(
|
||||
() ->
|
||||
tx.execute(
|
||||
status ->
|
||||
privateStore.renewWriter(
|
||||
tenant,
|
||||
session,
|
||||
token,
|
||||
new ManagedSessionStoreModels
|
||||
.RenewWriterRequest(
|
||||
"workspace",
|
||||
"writer",
|
||||
grant.writerGeneration(),
|
||||
60_000L))));
|
||||
assertThat(
|
||||
heartbeat
|
||||
.get(3, java.util.concurrent.TimeUnit.SECONDS)
|
||||
.writerGeneration())
|
||||
.isEqualTo(grant.writerGeneration());
|
||||
assertThat(paused.entered.await(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue();
|
||||
var heartbeat = executor.submit(() -> tx.execute(status -> privateStore.renewWriter(tenant, session, token,
|
||||
new ManagedSessionStoreModels.RenewWriterRequest("workspace", "writer", grant.writerGeneration(), 60_000L))));
|
||||
assertThat(heartbeat.get(3, java.util.concurrent.TimeUnit.SECONDS).writerGeneration()).isEqualTo(grant.writerGeneration());
|
||||
} finally {
|
||||
release.countDown();
|
||||
paused.release.countDown();
|
||||
}
|
||||
build.get(10, java.util.concurrent.TimeUnit.SECONDS);
|
||||
assertThat(read.get(10, java.util.concurrent.TimeUnit.SECONDS).items()).hasSize(1);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -713,7 +548,7 @@ public class ManagedSchemaSessionStoreTest {
|
||||
false,
|
||||
"boot:kept:1"));
|
||||
materialize();
|
||||
var frozen = store.findSnapshotPage(tenant, session, null, 0, 1);
|
||||
var frozen = page(null, 0, 1);
|
||||
String oldBody =
|
||||
jdbc.queryForObject(
|
||||
"SELECT body_resource_key FROM ma_message WHERE"
|
||||
@@ -723,7 +558,7 @@ public class ManagedSchemaSessionStoreTest {
|
||||
session,
|
||||
"withdrawn");
|
||||
long first =
|
||||
store.findSnapshot(tenant, session).orElseThrow().items().stream()
|
||||
messages().items().stream()
|
||||
.filter(item -> "withdrawn".equals(item.itemId()))
|
||||
.findFirst()
|
||||
.orElseThrow()
|
||||
@@ -734,7 +569,8 @@ public class ManagedSchemaSessionStoreTest {
|
||||
status ->
|
||||
store.retractContinuationOutput(
|
||||
tenant, session, turn, "owner", "boot", "old"));
|
||||
assertThat(store.findSnapshot(tenant, session).orElseThrow().items())
|
||||
materialize();
|
||||
assertThat(messages().items())
|
||||
.extracting(StoreModels.ItemRecord::itemId)
|
||||
.doesNotContain("withdrawn");
|
||||
assertThat(
|
||||
@@ -744,17 +580,11 @@ public class ManagedSchemaSessionStoreTest {
|
||||
Integer.class,
|
||||
oldBody))
|
||||
.isZero();
|
||||
assertThat(
|
||||
jdbc.queryForObject(
|
||||
"SELECT COUNT(*) FROM ma_resource_ref WHERE resource_key = ?"
|
||||
+ " AND owner_kind = 'SNAPSHOT'",
|
||||
Integer.class,
|
||||
oldBody))
|
||||
.isPositive();
|
||||
assertThat(store.findSnapshotPage(tenant, session, frozen.nextCursor(), 0, 10).items())
|
||||
.filteredOn(item -> "withdrawn".equals(item.itemId()))
|
||||
.singleElement()
|
||||
.satisfies(item -> assertThat(item.content().getFirst().text()).hasSize(70_000));
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource_ref WHERE resource_key = ?",
|
||||
Integer.class, oldBody)).isZero();
|
||||
assertThatThrownBy(() -> page(frozen.nextCursor(), 0, 10))
|
||||
.isInstanceOfSatisfying(ManagedAgentException.class,
|
||||
error -> assertThat(error.getCode()).isEqualTo("message_projection_changed"));
|
||||
tx.executeWithoutResult(
|
||||
status ->
|
||||
store.appendPublicEventIfAbsent(
|
||||
@@ -776,7 +606,7 @@ public class ManagedSchemaSessionStoreTest {
|
||||
true,
|
||||
"boot:new:2"));
|
||||
materialize();
|
||||
assertThat(store.findSnapshotPage(tenant, session, null, first, 10).items())
|
||||
assertThat(page(null, first, 10).items())
|
||||
.filteredOn(item -> "withdrawn".equals(item.itemId()))
|
||||
.singleElement()
|
||||
.satisfies(
|
||||
@@ -837,7 +667,7 @@ public class ManagedSchemaSessionStoreTest {
|
||||
"boot:kept:2"));
|
||||
materialize();
|
||||
var before =
|
||||
store.findSnapshot(tenant, session).orElseThrow().items().stream()
|
||||
messages().items().stream()
|
||||
.filter(item -> "same".equals(item.itemId()))
|
||||
.findFirst()
|
||||
.orElseThrow();
|
||||
@@ -845,7 +675,8 @@ public class ManagedSchemaSessionStoreTest {
|
||||
status ->
|
||||
store.retractContinuationOutput(
|
||||
tenant, session, turn, "owner", "boot", "old"));
|
||||
assertThat(store.findSnapshot(tenant, session).orElseThrow().items())
|
||||
materialize();
|
||||
assertThat(messages().items())
|
||||
.filteredOn(item -> "same".equals(item.itemId()))
|
||||
.singleElement()
|
||||
.satisfies(
|
||||
@@ -875,7 +706,7 @@ public class ManagedSchemaSessionStoreTest {
|
||||
event("big", "x".repeat(70_000));
|
||||
if ("none".equals(corruption)) {
|
||||
tx.executeWithoutResult(status -> subject.materializeNextBatch(tenant, session, 100));
|
||||
assertThat(subject.findSnapshot(tenant, session).orElseThrow().items())
|
||||
assertThat(tx.execute(status -> subject.findMessagePage(tenant, session, null, 100, false)).items())
|
||||
.singleElement()
|
||||
.satisfies(
|
||||
item -> assertThat(item.content().getFirst().text()).hasSize(70_000));
|
||||
@@ -948,6 +779,47 @@ public class ManagedSchemaSessionStoreTest {
|
||||
}
|
||||
}
|
||||
|
||||
private final class PausedMessageRead {
|
||||
private final java.util.concurrent.CountDownLatch entered = new java.util.concurrent.CountDownLatch(1);
|
||||
private final java.util.concurrent.CountDownLatch release = new java.util.concurrent.CountDownLatch(1);
|
||||
private final ManagedAgentStore subject;
|
||||
|
||||
PausedMessageRead() {
|
||||
var paused = new ManagedSchemaJdbcTemplate(java.util.Objects.requireNonNull(jdbc.getDataSource())) {
|
||||
@Override
|
||||
public <T> List<T> query(String sql, org.springframework.jdbc.core.RowMapper<T> mapper, Object... arguments) {
|
||||
var rows = super.query(sql, mapper, arguments);
|
||||
if (sql.startsWith("SELECT m.item_id, m.first_sequence,")) {
|
||||
entered.countDown();
|
||||
try {
|
||||
if (!release.await(10, java.util.concurrent.TimeUnit.SECONDS)) {
|
||||
throw new IllegalStateException("Message page read was not released");
|
||||
}
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException(interrupted);
|
||||
}
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
};
|
||||
subject = new ManagedAgentStore(paused, new ObjectMapper(), clock, ignored -> {},
|
||||
new ManagedWorkspaceRegistry(paused), new ManagedAgentProperties());
|
||||
}
|
||||
|
||||
StoreModels.MessagePage read() {
|
||||
return tx.execute(status -> subject.findMessagePage(tenant, session, null, 100, false));
|
||||
}
|
||||
}
|
||||
|
||||
private StoreModels.MessagePage page(String cursor, long afterSequence, int limit) {
|
||||
return tx.execute(status -> store.findMessagePage(tenant, session, cursor, limit, false, afterSequence));
|
||||
}
|
||||
|
||||
private StoreModels.MessagePage messages() {
|
||||
return page(null, 0, 100);
|
||||
}
|
||||
|
||||
private void event(String item, String text) {
|
||||
tx.executeWithoutResult(
|
||||
status ->
|
||||
|
||||
+61
-61
@@ -18,6 +18,7 @@ import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
class ManagedSnapshotNumericPagingTest {
|
||||
@Test
|
||||
@@ -25,26 +26,27 @@ class ManagedSnapshotNumericPagingTest {
|
||||
for (int count : List.of(1000, 10000)) {
|
||||
try (Fixture fixture = new Fixture(count, false)) {
|
||||
fixture.jdbc.bodyReads = 0;
|
||||
var page = fixture.store.findRecentSnapshotPage("tenant", "session", null, 2);
|
||||
var page = fixture.backward(null, 2);
|
||||
assertThat(page.items()).extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly(id(count - 1), id(count));
|
||||
assertThat(page.hasMore()).isTrue();
|
||||
assertThat(fixture.jdbc.bodyReads).isEqualTo(2);
|
||||
var older = fixture.store.findRecentSnapshotPage("tenant", "session", page.nextCursor(), 2);
|
||||
var older = fixture.backward(page.nextCursor(), 2);
|
||||
assertThat(older.items()).extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly(id(count - 3), id(count - 2));
|
||||
assertThat(older.coveredSequence()).isEqualTo(page.coveredSequence());
|
||||
assertThat(fixture.jdbc.bodyReads).isEqualTo(4);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void recentItemsCrossPhysicalPagesAndTerminateWithoutDuplicates() {
|
||||
void recentItemsCrossPagesAndTerminateWithoutDuplicates() {
|
||||
try (Fixture fixture = new Fixture(205, true)) {
|
||||
List<String> ids = new ArrayList<>();
|
||||
String cursor = null;
|
||||
do {
|
||||
var page = fixture.store.findRecentSnapshotPage("tenant", "session", cursor, 37);
|
||||
var page = fixture.backward(cursor, 37);
|
||||
assertThat(page.items()).hasSizeLessThanOrEqualTo(37);
|
||||
ids.addAll(0, page.items().stream().map(StoreModels.ItemRecord::itemId).toList());
|
||||
cursor = page.nextCursor();
|
||||
@@ -56,25 +58,20 @@ class ManagedSnapshotNumericPagingTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void recentCursorCannotBeUsedAsForwardCursorOrChanged() {
|
||||
void messageCursorRejectsAnotherDirectionAndMalformedData() {
|
||||
try (Fixture fixture = new Fixture(205, false)) {
|
||||
var page = fixture.store.findRecentSnapshotPage("tenant", "session", null, 1);
|
||||
org.assertj.core.api.Assertions.assertThatThrownBy(() -> fixture.store.findSnapshotPage(
|
||||
"tenant", "session", page.nextCursor(), 0, 1)).hasMessageContaining("cursor");
|
||||
var forward = fixture.store.findSnapshotPage("tenant", "session", null, 0, 1);
|
||||
org.assertj.core.api.Assertions.assertThatThrownBy(() -> fixture.store.findRecentSnapshotPage(
|
||||
"tenant", "session", forward.nextCursor(), 1)).hasMessageContaining("cursor");
|
||||
String cursor = page.nextCursor();
|
||||
String tampered = cursor.substring(0, cursor.indexOf('.') + 1) + "AAAA";
|
||||
org.assertj.core.api.Assertions.assertThatThrownBy(() -> fixture.store.findRecentSnapshotPage(
|
||||
"tenant", "session", tampered, 1)).hasMessageContaining("cursor");
|
||||
var page = fixture.backward(null, 1);
|
||||
assertThatThrownBy(() -> fixture.forward(page.nextCursor(), 0, 1)).hasMessageContaining("cursor");
|
||||
var forward = fixture.forward(null, 0, 1);
|
||||
assertThatThrownBy(() -> fixture.backward(forward.nextCursor(), 1)).hasMessageContaining("cursor");
|
||||
assertThatThrownBy(() -> fixture.backward("not-a-message-cursor", 1)).hasMessageContaining("cursor");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void recentCursorKeepsItsSnapshotWhenANewerProjectionIsPublished() {
|
||||
void recentCursorExpiresWhenTheMessageProjectionIsRebuilt() {
|
||||
try (Fixture fixture = new Fixture(205, false)) {
|
||||
var first = fixture.store.findRecentSnapshotPage("tenant", "session", null, 2);
|
||||
var first = fixture.backward(null, 2);
|
||||
fixture.transaction.executeWithoutResult(status -> {
|
||||
fixture.jdbc.update("UPDATE ma_message SET item_status='retracted'"
|
||||
+ " WHERE tenant_id=? AND session_id=? AND item_id=?", "tenant", "session", id(203));
|
||||
@@ -83,12 +80,10 @@ class ManagedSnapshotNumericPagingTest {
|
||||
"tenant", "session");
|
||||
fixture.project();
|
||||
});
|
||||
var latest = fixture.store.findRecentSnapshotPage("tenant", "session", null, 3);
|
||||
assertThat(latest.items().getFirst().status()).isEqualTo("retracted");
|
||||
var next = fixture.store.findRecentSnapshotPage("tenant", "session", first.nextCursor(), 2);
|
||||
assertThat(next.items()).extracting(StoreModels.ItemRecord::itemId).containsExactly(id(202), id(203));
|
||||
assertThat(next.items().getLast().status()).isEqualTo("completed");
|
||||
assertThat(next.coveredSequence()).isEqualTo(first.coveredSequence());
|
||||
assertThat(fixture.backward(null, 3).items().getFirst().status()).isEqualTo("retracted");
|
||||
assertThatThrownBy(() -> fixture.backward(first.nextCursor(), 2))
|
||||
.isInstanceOfSatisfying(com.alibaba.qwen.code.managedagent.exception.ManagedAgentException.class,
|
||||
error -> assertThat(error.getCode()).isEqualTo("message_projection_changed"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -98,7 +93,7 @@ class ManagedSnapshotNumericPagingTest {
|
||||
for (int count : List.of(1000, 10000)) {
|
||||
try (Fixture fixture = new Fixture(count, false)) {
|
||||
fixture.jdbc.bodyReads = 0;
|
||||
var page = fixture.store.findSnapshotPage("tenant", "session", null, count - 1, 1);
|
||||
var page = fixture.forward(null, count - 1, 1);
|
||||
assertThat(page.items())
|
||||
.extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly(id(count));
|
||||
@@ -108,69 +103,54 @@ class ManagedSnapshotNumericPagingTest {
|
||||
fixture.verifyDirectoryPlan();
|
||||
}
|
||||
}
|
||||
assertThat(reads).containsExactly(2, 2);
|
||||
assertThat(reads).containsExactly(1, 1);
|
||||
}
|
||||
|
||||
@Test
|
||||
void numericBoundarySkipsAllEqualSequencesAcrossPageBoundaries() {
|
||||
try (Fixture fixture = new Fixture(205, true)) {
|
||||
var ties = fixture.store.findSnapshotPage("tenant", "session", null, 99, 2);
|
||||
var ties = fixture.forward(null, 99, 2);
|
||||
assertThat(ties.items())
|
||||
.extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly(id(100), id(101));
|
||||
assertThat(ties.hasMore()).isTrue();
|
||||
var remainingTies =
|
||||
fixture.store.findSnapshotPage("tenant", "session", ties.nextCursor(), 0, 2);
|
||||
fixture.forward(ties.nextCursor(), 0, 2);
|
||||
assertThat(remainingTies.items())
|
||||
.extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly(id(102), id(103));
|
||||
var afterTies = fixture.store.findSnapshotPage("tenant", "session", null, 100, 2);
|
||||
var afterTies = fixture.forward(null, 100, 2);
|
||||
assertThat(afterTies.items())
|
||||
.extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly(id(106), id(107));
|
||||
fixture.jdbc.bodyReads = 0;
|
||||
var empty = fixture.store.findSnapshotPage("tenant", "session", null, 200, 2);
|
||||
var empty = fixture.forward(null, 200, 2);
|
||||
assertThat(empty.items()).isEmpty();
|
||||
assertThat(empty.hasMore()).isFalse();
|
||||
assertThat(empty.nextCursor()).isNull();
|
||||
assertThat(empty.coveredSequence()).isEqualTo(ties.coveredSequence());
|
||||
assertThat(fixture.jdbc.bodyReads).isEqualTo(1);
|
||||
assertThat(fixture.jdbc.bodyReads).isZero();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void numericFirstPageCursorRetainsOriginalSnapshotAfterProjectionChanges() {
|
||||
void numericFirstPageCursorExpiresAfterProjectionRebuild() {
|
||||
try (Fixture fixture = new Fixture(205, false)) {
|
||||
var first = fixture.store.findSnapshotPage("tenant", "session", null, 199, 2);
|
||||
fixture.transaction.executeWithoutResult(
|
||||
status -> {
|
||||
fixture.jdbc.update(
|
||||
"UPDATE ma_message SET item_status='retracted' WHERE tenant_id=?"
|
||||
+ " AND session_id=? AND item_id=?",
|
||||
"tenant",
|
||||
"session",
|
||||
id(203));
|
||||
fixture.jdbc.update(
|
||||
"UPDATE ma_event_stream SET projection_version=projection_version+1"
|
||||
+ " WHERE tenant_id=? AND session_id=? AND"
|
||||
+ " stream_kind='SESSION' AND stream_id=X''",
|
||||
"tenant",
|
||||
"session");
|
||||
fixture.project();
|
||||
});
|
||||
var changed = fixture.store.findSnapshotPage("tenant", "session", null, 201, 2);
|
||||
assertThat(changed.items())
|
||||
.extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly(id(202), id(203));
|
||||
var first = fixture.forward(null, 199, 2);
|
||||
fixture.transaction.executeWithoutResult(status -> {
|
||||
fixture.jdbc.update("UPDATE ma_message SET item_status='retracted' WHERE tenant_id=? AND session_id=? AND item_id=?",
|
||||
"tenant", "session", id(203));
|
||||
fixture.jdbc.update("UPDATE ma_event_stream SET projection_version=projection_version+1"
|
||||
+ " WHERE tenant_id=? AND session_id=? AND stream_kind='SESSION' AND stream_id=X''", "tenant", "session");
|
||||
fixture.project();
|
||||
});
|
||||
var changed = fixture.forward(null, 201, 2);
|
||||
assertThat(changed.items()).extracting(StoreModels.ItemRecord::itemId).containsExactly(id(202), id(203));
|
||||
assertThat(changed.items().getLast().status()).isEqualTo("retracted");
|
||||
assertThat(changed.coveredSequence()).isGreaterThan(first.coveredSequence());
|
||||
var next =
|
||||
fixture.store.findSnapshotPage("tenant", "session", first.nextCursor(), 0, 2);
|
||||
assertThat(next.items())
|
||||
.extracting(StoreModels.ItemRecord::itemId)
|
||||
.containsExactly(id(202), id(203));
|
||||
assertThat(next.items().getLast().status()).isEqualTo("completed");
|
||||
assertThat(next.coveredSequence()).isEqualTo(first.coveredSequence());
|
||||
assertThatThrownBy(() -> fixture.forward(first.nextCursor(), 0, 2))
|
||||
.isInstanceOfSatisfying(com.alibaba.qwen.code.managedagent.exception.ManagedAgentException.class,
|
||||
error -> assertThat(error.getCode()).isEqualTo("message_projection_changed"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -193,7 +173,7 @@ class ManagedSnapshotNumericPagingTest {
|
||||
if (sql.contains("FROM ma_resource WHERE") && sql.contains("inline_bytes")) {
|
||||
bodyReads++;
|
||||
}
|
||||
if (sql.startsWith("SELECT snapshot_page_ordinal FROM ma_resource")) {
|
||||
if (sql.startsWith("SELECT m.item_id, m.first_sequence,")) {
|
||||
directoryReads++;
|
||||
directorySql = sql;
|
||||
directoryArguments = args.clone();
|
||||
@@ -215,6 +195,7 @@ class ManagedSnapshotNumericPagingTest {
|
||||
jdbc = new CountingJdbc(database.source());
|
||||
transaction =
|
||||
new TransactionTemplate(new DataSourceTransactionManager(database.source()));
|
||||
transaction.setIsolationLevel(org.springframework.transaction.TransactionDefinition.ISOLATION_REPEATABLE_READ);
|
||||
store =
|
||||
new ManagedAgentStore(
|
||||
jdbc,
|
||||
@@ -254,9 +235,28 @@ class ManagedSnapshotNumericPagingTest {
|
||||
+ "first_sequence,last_sequence,gmt_create,gmt_modified,revision,parts_json)"
|
||||
+ " VALUES(?,?,?,?,'message','assistant','completed','{}',?,?,0,0,1,'[]')",
|
||||
rows);
|
||||
String scope = ManagedSessionStore.sessionScopeKey("tenant", "session");
|
||||
byte[] bytes = {91, 93};
|
||||
for (int index = Math.max(1, count - 3); index <= count; index++) {
|
||||
String resourceId = "numeric-parts-" + index;
|
||||
String resourceKey = ManagedSessionStore.resourceKey(scope, resourceId);
|
||||
jdbc.update("INSERT INTO ma_resource(resource_key,session_scope_key,tenant_id,session_id,"
|
||||
+ "resource_id,kind,schema_version,byte_length,sha256,storage_kind,inline_bytes,state,gmt_create)"
|
||||
+ " VALUES(?,?,'tenant','session',?,'managed-message-parts',1,2,?,'MYSQL_INLINE',?,'REFERENCED',0)",
|
||||
resourceKey, scope, resourceId, WorkspaceEvidenceResources.digest(bytes), bytes);
|
||||
jdbc.update("UPDATE ma_message SET body_resource_key=? WHERE item_id=?", resourceKey, id(index));
|
||||
}
|
||||
project();
|
||||
}
|
||||
|
||||
StoreModels.MessagePage forward(String cursor, long afterSequence, int limit) {
|
||||
return transaction.execute(status -> store.findMessagePage("tenant", "session", cursor, limit, false, afterSequence));
|
||||
}
|
||||
|
||||
StoreModels.MessagePage backward(String cursor, int limit) {
|
||||
return transaction.execute(status -> store.findMessagePage("tenant", "session", cursor, limit, true));
|
||||
}
|
||||
|
||||
void project() {
|
||||
transaction.executeWithoutResult(
|
||||
status -> {
|
||||
@@ -282,7 +282,7 @@ class ManagedSnapshotNumericPagingTest {
|
||||
String.class,
|
||||
jdbc.directoryArguments);
|
||||
assertThat(plan)
|
||||
.contains("ix_ma_snapshot_page_lookup")
|
||||
.contains("ix_ma_message_history")
|
||||
.doesNotContain("Table scan", "Sort:");
|
||||
var counts =
|
||||
java.util.regex.Pattern.compile(
|
||||
|
||||
+17
-19
@@ -3,7 +3,6 @@ package com.alibaba.qwen.code.managedagent.persistence;
|
||||
import com.alibaba.druid.pool.DruidDataSource;
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
|
||||
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchema;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaKeys;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaProjection;
|
||||
@@ -16,8 +15,6 @@ import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.transaction.support.TransactionTemplate;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
@@ -26,7 +23,6 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
class ManagedSnapshotPoolTest {
|
||||
private DruidDataSource source;
|
||||
private ManagedSchemaJdbcTemplate jdbc;
|
||||
private ManagedSnapshotStore snapshots;
|
||||
|
||||
@BeforeEach
|
||||
void setup() throws Exception {
|
||||
@@ -46,12 +42,6 @@ class ManagedSnapshotPoolTest {
|
||||
ManagedSchemaKeys.key("tenant"), ManagedSchemaKeys.key("session"),
|
||||
ManagedSessionStore.sessionScopeKey("tenant", "session"));
|
||||
jdbc.update("INSERT INTO ma_event_stream (tenant_id, session_id, stream_kind, stream_id, last_sequence, covered_sequence, gmt_modified) VALUES ('tenant', 'session', 'SESSION', X'', 1, 1, 0)");
|
||||
snapshots = new ManagedSnapshotStore(jdbc, new ObjectMapper(), Clock.systemUTC(),
|
||||
(tenant, session, guard) -> {
|
||||
guard.run();
|
||||
return List.of(new StoreModels.ItemRecord(tenant, session, "item", "turn", "message",
|
||||
"assistant", "completed", Map.of(), 1, 1, 0, 0, 1, List.of()));
|
||||
});
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
@@ -60,26 +50,34 @@ class ManagedSnapshotPoolTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void buildsAndReadsImmutablePagesWithOneConnection() {
|
||||
snapshots.build("tenant", "session");
|
||||
assertThat(snapshots.page("tenant", "session", null, 0, 20).items()).hasSize(1);
|
||||
assertThat(snapshots.latest("tenant", "session").orElseThrow().items()).hasSize(1);
|
||||
void readsMessagePagesWithOneConnection() {
|
||||
jdbc.update("INSERT INTO ma_message (tenant_id,session_id,item_id,turn_id,item_type,item_role,item_status,attributes_json,"
|
||||
+ "first_sequence,last_sequence,gmt_create,gmt_modified,parts_json)"
|
||||
+ " VALUES ('tenant','session','item','turn','message','assistant','completed','{}',1,1,0,0,'[]')");
|
||||
var store = new ManagedAgentStore(jdbc, new ObjectMapper(), Clock.systemUTC(), ignored -> {},
|
||||
new ManagedWorkspaceRegistry(jdbc), new ManagedAgentProperties());
|
||||
var view = new TransactionTemplate(new DataSourceTransactionManager(source));
|
||||
view.setReadOnly(true);
|
||||
view.setIsolationLevel(org.springframework.transaction.TransactionDefinition.ISOLATION_REPEATABLE_READ);
|
||||
assertThat(view.execute(status -> store.findMessagePage("tenant", "session", null, 20, true)).items())
|
||||
.singleElement().satisfies(item -> assertThat(item.itemId()).isEqualTo("item"));
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_read_lease", Integer.class)).isZero();
|
||||
}
|
||||
|
||||
@Test
|
||||
void retainedPrivatePinPreventsProjectionExpiryAndBodyCollection() {
|
||||
snapshots.build("tenant", "session");
|
||||
void retainedPrivatePinPreventsReleasedBodyCollection() {
|
||||
var manager = new DataSourceTransactionManager(source);
|
||||
jdbc.update("UPDATE ma_snapshot SET retain_until = 0");
|
||||
String scope = ManagedSessionStore.sessionScopeKey("tenant", "session");
|
||||
jdbc.update("INSERT INTO ma_resource (resource_key,session_scope_key,tenant_id,session_id,resource_id,kind,schema_version,"
|
||||
+ "byte_length,sha256,storage_kind,inline_bytes,state,gmt_create,retention_until)"
|
||||
+ " VALUES (?,?,'tenant','session','released','managed-message-parts',1,2,?,'MYSQL_INLINE',?,'REFERENCED',0,0)",
|
||||
ManagedSchemaKeys.key(scope, "released"), scope, "a".repeat(64), new byte[] {91, 93});
|
||||
jdbc.update("UPDATE ma_resource SET retention_until = " + ManagedSchemaTime.SQL_NOW);
|
||||
jdbc.update("UPDATE ma_session SET runtime_request_key = ?", "a".repeat(64));
|
||||
var roots = jdbc.queryForList("SELECT " + ManagedSchemaProjection.SNAPSHOT.columns() + " FROM ma_snapshot");
|
||||
var resources = jdbc.queryForList("SELECT " + ManagedSchemaProjection.RESOURCE.columns() + " FROM ma_resource");
|
||||
var refs = jdbc.queryForList("SELECT " + ManagedSchemaProjection.RESOURCE_REF.columns() + " FROM ma_resource_ref");
|
||||
var collector = new ManagedProjectionResourceCollector(jdbc, manager);
|
||||
assertThat(collector.runOnce()).isFalse();
|
||||
assertThat(jdbc.queryForList("SELECT " + ManagedSchemaProjection.SNAPSHOT.columns() + " FROM ma_snapshot")).isEqualTo(roots);
|
||||
assertThat(jdbc.queryForList("SELECT " + ManagedSchemaProjection.RESOURCE.columns() + " FROM ma_resource"))
|
||||
.usingRecursiveComparison()
|
||||
.isEqualTo(resources);
|
||||
|
||||
+4
-19
@@ -140,7 +140,7 @@ class SessionRowLockOrderTest {
|
||||
"ManagedProjectionResourceCollector.java",
|
||||
"ManagedRuntimeRetentionGuard.java",
|
||||
"ManagedSessionRetirementStore.java", "ManagedSessionStore.java",
|
||||
"ManagedSnapshotStore.java", "ManagedToolResultStore.java",
|
||||
"ManagedToolResultStore.java",
|
||||
"ManagedUserMessageStore.java",
|
||||
"ResidentWorkspaceRetirementStore.java", "ResidentWorkspaces.java",
|
||||
"ToolPublicationRetentionStore.java",
|
||||
@@ -229,11 +229,6 @@ class SessionRowLockOrderTest {
|
||||
"ManagedSessionStore.sha256/1",
|
||||
"ManagedSessionStore.storedTransaction/1",
|
||||
"ManagedSessionStore.usesDataSource/1")),
|
||||
Map.entry("ManagedSnapshotStore.java",
|
||||
Set.of("ItemReader.read/3",
|
||||
"ManagedSnapshotStore.decode/1",
|
||||
"SnapshotReadLease.check/0",
|
||||
"SnapshotReadLease.close/0")),
|
||||
Map.entry("ManagedTaskEventStore.java",
|
||||
Set.of("ManagedTaskEventStore.eventRow/2")),
|
||||
Map.entry("ManagedToolResultProjector.java",
|
||||
@@ -304,7 +299,7 @@ class SessionRowLockOrderTest {
|
||||
"Cell.isNull/0", "Cell.kind/0", "Cell.text/0",
|
||||
"Encoder.write/1",
|
||||
"FrozenOwnerReader.read/2|Inspection.read/2",
|
||||
"Inspection.context/2", "Inspection.snapshot/0",
|
||||
"Inspection.context/2",
|
||||
"Kind.accepts/1", "Kind.binary/0", "Kind.integral/0",
|
||||
"NativeReadEvidence.number/1|Row.number/1",
|
||||
"NativeReadEvidence.text/1|Row.text/1",
|
||||
@@ -360,16 +355,7 @@ class SessionRowLockOrderTest {
|
||||
Set.of("IdentityReader.read/1",
|
||||
"WorkspaceStorageGuard.verify/1")));
|
||||
|
||||
// Held bodies whose invocation the parse cannot see but whose
|
||||
// transaction it can: ManagedSnapshotStore.afterCommit registers a
|
||||
// post-commit synchronization, and the build it runs opens its own
|
||||
// REQUIRES_NEW transaction, so that body's session lock can never
|
||||
// order against a row its registering caller touched. A deferred body
|
||||
// is dropped rather than folded at its declaration, and the set is
|
||||
// asserted exactly per file, so a new held body reaching a side still
|
||||
// fails closed.
|
||||
private static final Set<String> DEFERRED_BODIES = Set.of(
|
||||
"ManagedSnapshotStore.afterCommit/2");
|
||||
private static final Set<String> DEFERRED_BODIES = Set.of();
|
||||
|
||||
private record Key(String owner, String name, List<String> paramTypes) {
|
||||
int arity() {
|
||||
@@ -626,8 +612,7 @@ class SessionRowLockOrderTest {
|
||||
entry.getValue().deferredBodies());
|
||||
}
|
||||
}
|
||||
assertThat(deferred).isEqualTo(Map.of("ManagedSnapshotStore.java",
|
||||
DEFERRED_BODIES));
|
||||
assertThat(deferred).isEmpty();
|
||||
}
|
||||
|
||||
// The derivation must report a writer that reaches the turn row through
|
||||
|
||||
+2
-5
@@ -106,9 +106,6 @@ class WorkspaceCsiApplicationInventoryTest {
|
||||
jdbc.update("INSERT INTO ma_message (tenant_id,session_id,item_id,turn_id,item_type,item_status,attributes_json,"
|
||||
+ "first_sequence,last_sequence,gmt_create,gmt_modified,parts_json,body_resource_key)"
|
||||
+ " VALUES ('foreign','foreign-session','item','turn','message','completed','{}',1,1,1,1,'[]',?)", resource);
|
||||
jdbc.update("INSERT INTO ma_snapshot (tenant_id,session_id,snapshot_id,snapshot_version,covered_sequence,"
|
||||
+ "projection_version,manifest_resource_key,manifest_sha256,item_count,state,gmt_create,retain_until)"
|
||||
+ " VALUES ('foreign','foreign-session','snapshot',1,1,1,?,?,0,'PUBLISHED',1,2)", resource, sha(bytes));
|
||||
jdbc.update("INSERT INTO ma_event_stream (tenant_id,session_id,stream_kind,stream_id,artifact_refs_resource_key,gmt_modified)"
|
||||
+ " VALUES ('foreign','foreign-session','SESSION','',?,1)", resource);
|
||||
String foreignScope = ManagedSessionStore.sessionScopeKey("foreign", "foreign-session");
|
||||
@@ -116,7 +113,7 @@ class WorkspaceCsiApplicationInventoryTest {
|
||||
+ "tenant_id,session_id,resource_id,gmt_create) VALUES (?,'MESSAGE',?,1,?,'foreign',"
|
||||
+ "'foreign-session','original-resource'," + ManagedSchemaTime.SQL_NOW + ")", foreignScope, "a".repeat(64), resource);
|
||||
var found = inspect();
|
||||
for (String table : List.of("ma_message", "ma_snapshot", "ma_event_stream", "ma_resource_ref")) {
|
||||
for (String table : List.of("ma_message", "ma_event_stream", "ma_resource_ref")) {
|
||||
assertThat(found.blockers()).as(table).anyMatch(blocker -> blocker.table().equals(table)
|
||||
&& blocker.code().equals("foreign_original_reference"));
|
||||
assertThat(found.table(table).rows()).anyMatch(row -> "foreign".equals(row.text("tenant_id")));
|
||||
@@ -785,7 +782,7 @@ class WorkspaceCsiApplicationInventoryTest {
|
||||
void inventoriesEveryDeclaredTableAndPreservesRealPrivateCreationFacts() throws Exception {
|
||||
var snapshot = inspect();
|
||||
assertThat(snapshot.tables()).hasSize(com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaContract.TABLES.size());
|
||||
assertThat(snapshot.tables().stream().map(table -> table.columns().size()).mapToInt(Integer::intValue).sum()).isEqualTo(824);
|
||||
assertThat(snapshot.tables().stream().map(table -> table.columns().size()).mapToInt(Integer::intValue).sum()).isEqualTo(805);
|
||||
assertThat(snapshot.blockers()).noneMatch(blocker -> blocker.code().equals("schema_drift")
|
||||
|| blocker.code().equals("query_unavailable"));
|
||||
for (String table : new String[] {"ma_session", "ma_public_event", "ma_event_stream", "ma_storage", "ma_operation",
|
||||
|
||||
+1
-1
@@ -291,7 +291,7 @@ class WorkspaceCsiNativeActivationGateTest {
|
||||
.map(row -> JSON.valueToTree(new TreeMap<>(row)).toString()).sorted().toList());
|
||||
}
|
||||
assertThat(rows).containsKeys("ma_delivery", "ma_session", "ma_session_head", "ma_request_receipt");
|
||||
assertThat(rows).hasSize(40);
|
||||
assertThat(rows).hasSize(39);
|
||||
return rows;
|
||||
}
|
||||
|
||||
|
||||
+110
-60
@@ -73,9 +73,9 @@ class WorkspaceCsiRetainedArtifactsTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void acceptsEmptySnapshotAndActualAgentResourcesIncludingEmptyAndLiteralText() throws Exception {
|
||||
void acceptsEmptyMessagesAndActualAgentResourcesIncludingEmptyAndLiteralText() throws Exception {
|
||||
materialize();
|
||||
assertThat(store.findSnapshot("foreign", session).orElseThrow().items()).isEmpty();
|
||||
assertThat(tx.execute(status -> store.findMessagePage("foreign", session, null, 100, true)).items()).isEmpty();
|
||||
var resources = new ManagedSessionStore(jdbc);
|
||||
String literal = "Unicode 指令 {\"sessionId\":\"" + expected.table("ma_session").rows().getFirst().text("session_id") + "\"}";
|
||||
for (String text : List.of("", literal)) {
|
||||
@@ -91,15 +91,21 @@ class WorkspaceCsiRetainedArtifactsTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void acceptsRealMultiplePagesAndAnOversizedSingleItem() throws Exception {
|
||||
void acceptsMultipleMessagePagesAndAnOversizedSingleItem() throws Exception {
|
||||
for (int index = 0; index < 101; index++) {
|
||||
append("item.tool_call.updated", Map.of("callId", "call-" + index, "name", "read_file", "status", "completed"));
|
||||
}
|
||||
append("item.tool_call.updated", Map.of("callId", "large", "name", "read_file", "status", "completed", "arguments", "x".repeat(150_000)));
|
||||
materialize();
|
||||
assertThat(store.findSnapshot("foreign", session).orElseThrow().items()).hasSize(102);
|
||||
assertThat(jdbc.queryForObject("SELECT MAX(byte_length) FROM ma_resource WHERE kind=?", Long.class, "managed-snapshot-page")).isGreaterThan(128 * 1024);
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource WHERE kind=?", Integer.class, "managed-snapshot-page")).isGreaterThan(1);
|
||||
var first = tx.execute(status -> store.findMessagePage("foreign", session, null, 100, true));
|
||||
var older = tx.execute(status -> store.findMessagePage("foreign", session, first.nextCursor(), 100, true));
|
||||
assertThat(first.items()).hasSize(100);
|
||||
assertThat(older.items()).hasSize(2);
|
||||
assertThat(older.hasMore()).isFalse();
|
||||
assertThat(java.util.stream.Stream.concat(first.items().stream(), older.items().stream())
|
||||
.map(StoreModels.ItemRecord::itemId).distinct().count()).isEqualTo(102);
|
||||
assertThat(jdbc.queryForObject("SELECT MAX(OCTET_LENGTH(attributes_json)) FROM ma_message WHERE tenant_id=?",
|
||||
Long.class, "foreign")).isGreaterThan(128 * 1024);
|
||||
accepted();
|
||||
}
|
||||
|
||||
@@ -112,25 +118,26 @@ class WorkspaceCsiRetainedArtifactsTest {
|
||||
jdbc.update("UPDATE ma_message SET first_sequence=1 WHERE tenant_id=?", "foreign");
|
||||
append("session.metadata.updated", Map.of("title", "ties"));
|
||||
materialize();
|
||||
var items = store.findSnapshot("foreign", session).orElseThrow().items();
|
||||
var items = tx.execute(status -> store.findMessagePage("foreign", session, null, 100, true)).items();
|
||||
assertThat(items).extracting(StoreModels.ItemRecord::firstSequence).containsOnly(1L);
|
||||
accepted();
|
||||
}
|
||||
|
||||
@Test
|
||||
void acceptsOlderBodyRevisionAndSeveralSnapshotsAfterTheCurrentBodyChanges() throws Exception {
|
||||
void acceptsOlderBodyRevisionAndReleasedBodiesWhileCollectionIsDisabled() throws Exception {
|
||||
largeBody(70_000);
|
||||
String oldBody = jdbc.queryForObject("SELECT body_resource_key FROM ma_message WHERE tenant_id=?", String.class, "foreign");
|
||||
append("turn.completed", Map.of());
|
||||
materialize();
|
||||
assertThat(jdbc.queryForObject("SELECT revision FROM ma_message WHERE tenant_id=?", Long.class, "foreign")).isGreaterThan(1);
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource_ref WHERE resource_key=?", Integer.class, oldBody)).isEqualTo(3);
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource_ref WHERE resource_key=?", Integer.class, oldBody)).isEqualTo(1);
|
||||
accepted();
|
||||
append("item.output_text.delta", Map.of("itemId", "large", "text", "later"));
|
||||
materialize();
|
||||
assertThat(jdbc.queryForObject("SELECT body_resource_key FROM ma_message WHERE tenant_id=?", String.class, "foreign")).isNotEqualTo(oldBody);
|
||||
assertThat(jdbc.queryForList("SELECT owner_kind FROM ma_resource_ref WHERE resource_key=?", String.class, oldBody)).containsOnly("SNAPSHOT");
|
||||
jdbc.update("UPDATE ma_snapshot SET retain_until=1 WHERE tenant_id=?", "foreign");
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource_ref WHERE resource_key=?", Integer.class, oldBody)).isZero();
|
||||
assertThat(jdbc.queryForMap("SELECT state, retention_until FROM ma_resource WHERE resource_key=?", oldBody))
|
||||
.containsEntry("state", "REFERENCED").doesNotContainValue(null);
|
||||
var before = artifactBytes();
|
||||
accepted();
|
||||
assertThat(artifactBytes()).containsExactlyElementsOf(before);
|
||||
@@ -144,40 +151,73 @@ class WorkspaceCsiRetainedArtifactsTest {
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"missing-root", "missing-page", "missing-ref", "extra-ref", "wrong-ref-revision",
|
||||
"wrong-ref-scope", "wrong-root-count", "wrong-root-version", "manifest-sequence", "manifest-type", "manifest-secret", "manifest-extra",
|
||||
"page-ordinal", "page-last", "page-owner", "page-extra", "page-order", "page-original-reference", "body-inline-content", "page-utf8"})
|
||||
void refusesIncompleteOrSubstitutedSnapshotGraphs(String fault) throws Exception {
|
||||
@ValueSource(strings = {"missing-ref", "extra-ref", "wrong-ref-revision", "wrong-ref-scope", "body-owner",
|
||||
"body-extra", "body-order", "body-original-reference", "body-inline-content", "body-utf8"})
|
||||
void refusesIncompleteOrSubstitutedMessageBodies(String fault) throws Exception {
|
||||
largeBody(70_000);
|
||||
append("item.tool_call.updated", Map.of("callId", "second", "name", "read_file", "status", "completed"));
|
||||
materialize();
|
||||
accepted();
|
||||
switch (fault) {
|
||||
case "missing-root" -> {
|
||||
jdbc.update("UPDATE ma_event_stream SET current_snapshot_id=NULL WHERE tenant_id=?", "foreign");
|
||||
jdbc.update("DELETE FROM ma_snapshot WHERE tenant_id=?", "foreign");
|
||||
}
|
||||
case "missing-page" -> removeResources("managed-snapshot-page");
|
||||
case "missing-ref" -> jdbc.update("DELETE FROM ma_resource_ref WHERE owner_kind='SNAPSHOT'");
|
||||
case "missing-ref" -> jdbc.update("DELETE FROM ma_resource_ref WHERE owner_kind='MESSAGE'");
|
||||
case "extra-ref" -> jdbc.update("UPDATE ma_resource_ref SET owner_kind='ARTIFACT' WHERE owner_kind='MESSAGE'");
|
||||
case "wrong-ref-revision" -> jdbc.update("UPDATE ma_resource_ref SET owner_revision=owner_revision+10 WHERE owner_kind='SNAPSHOT'");
|
||||
case "wrong-ref-scope" -> jdbc.update("UPDATE ma_resource_ref SET workspace_id='borrowed' WHERE owner_kind='SNAPSHOT'");
|
||||
case "wrong-root-count" -> jdbc.update("UPDATE ma_snapshot SET item_count=item_count+1 WHERE tenant_id=?", "foreign");
|
||||
case "wrong-root-version" -> jdbc.update("UPDATE ma_snapshot SET snapshot_version=snapshot_version+10 WHERE tenant_id=?", "foreign");
|
||||
case "manifest-sequence" -> edit("managed-snapshot-manifest", value -> ((ObjectNode) value).put("sequence", 9999));
|
||||
case "manifest-type" -> edit("managed-snapshot-manifest", value -> ((ObjectNode) value).put("version", "1"));
|
||||
case "manifest-secret" -> edit("managed-snapshot-manifest", value -> ((ObjectNode) value).put("cursorSecret", "AA=="));
|
||||
case "manifest-extra" -> edit("managed-snapshot-manifest", value -> ((ObjectNode) value).put("extra", true));
|
||||
case "page-ordinal" -> jdbc.update("UPDATE ma_resource SET snapshot_page_ordinal=9 WHERE kind=?", "managed-snapshot-page");
|
||||
case "page-last" -> jdbc.update("UPDATE ma_resource SET snapshot_page_last_sequence=9999 WHERE kind=?", "managed-snapshot-page");
|
||||
case "page-owner" -> edit("managed-snapshot-page", value -> ((ObjectNode) value.get(0).get("item")).put("tenantId", "borrowed"));
|
||||
case "page-extra" -> edit("managed-snapshot-page", value -> ((ObjectNode) value.get(0)).put("extra", true));
|
||||
case "page-order" -> edit("managed-snapshot-page", value -> ((ObjectNode) value.get(0).get("item")).put("firstSequence", 9999));
|
||||
case "page-original-reference" -> edit("managed-snapshot-page", value -> ((ObjectNode) value.get(0).get("item").get("attributes"))
|
||||
.put("bindingId", expected.table("ma_runtime_binding").rows().getFirst().text("binding_id")));
|
||||
case "body-inline-content" -> edit("managed-snapshot-page", value -> ((ObjectNode) value.get(0).get("item")).set("content", JSON.valueToTree(
|
||||
List.of(new StoreModels.ItemPartRecord("borrowed", "text", "hidden", 1, 1, 0, 0, 1)))));
|
||||
case "page-utf8" -> replace("managed-snapshot-page", new byte[] {(byte) 0xc3, 0x28});
|
||||
case "wrong-ref-revision" -> jdbc.update("UPDATE ma_resource_ref SET owner_revision=owner_revision+10 WHERE owner_kind='MESSAGE'");
|
||||
case "wrong-ref-scope" -> jdbc.update("UPDATE ma_resource_ref SET workspace_id='borrowed' WHERE owner_kind='MESSAGE'");
|
||||
case "body-owner" -> jdbc.update("UPDATE ma_resource SET tenant_id='borrowed' WHERE kind=?", "managed-message-parts");
|
||||
case "body-extra" -> editBody(value -> ((ObjectNode) value.get(0)).put("extra", true));
|
||||
case "body-order" -> editBody(value -> ((ObjectNode) value.get(0))
|
||||
.put("firstSequence", 9999).put("lastSequence", 9999));
|
||||
case "body-original-reference" -> jdbc.update("UPDATE ma_message SET attributes_json=? WHERE tenant_id=?",
|
||||
JSON.writeValueAsString(Map.of("bindingId", expected.table("ma_runtime_binding").rows().getFirst().text("binding_id"))), "foreign");
|
||||
case "body-inline-content" -> jdbc.update("UPDATE ma_message SET parts_json='[{}]' WHERE body_resource_key IS NOT NULL");
|
||||
case "body-utf8" -> {
|
||||
byte[] bytes = jdbc.queryForObject("SELECT inline_bytes FROM ma_resource WHERE kind=?",
|
||||
byte[].class, "managed-message-parts");
|
||||
String json = new String(bytes, StandardCharsets.UTF_8);
|
||||
String prefix = "\"text\":\"";
|
||||
int start = json.indexOf(prefix);
|
||||
assertThat(start).isGreaterThanOrEqualTo(0);
|
||||
int offset = json.substring(0, start + prefix.length()).getBytes(StandardCharsets.UTF_8).length;
|
||||
bytes[offset] = (byte) 0xc3;
|
||||
bytes[offset + 1] = 0x28;
|
||||
replaceBody(bytes);
|
||||
}
|
||||
default -> throw new AssertionError(fault);
|
||||
}
|
||||
assertThat(fixture.readReferences(expected).blockers()).as(fault).isNotEmpty();
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"missing-marker", "extra-owner", "wrong-id",
|
||||
"wrong-body-scope", "zero-body-ordinal", "wrong-body-digest"})
|
||||
void refusesUnprovenOrReferencedReleasedBodies(String fault) throws Exception {
|
||||
largeBody(70_000);
|
||||
String oldBody = jdbc.queryForObject("SELECT body_resource_key FROM ma_message WHERE tenant_id=?", String.class, "foreign");
|
||||
append("item.output_text.delta", Map.of("itemId", "large", "text", "later"));
|
||||
materialize();
|
||||
accepted();
|
||||
switch (fault) {
|
||||
case "missing-marker" -> jdbc.update("UPDATE ma_resource SET retention_until=NULL WHERE resource_key=?", oldBody);
|
||||
case "extra-owner" -> jdbc.update("INSERT INTO ma_resource_ref (session_scope_key, owner_kind, owner_key, owner_revision, resource_key, tenant_id, session_id, resource_id, gmt_create)"
|
||||
+ " SELECT session_scope_key,'ARTIFACT',?,1,resource_key,tenant_id,session_id,resource_id,0 FROM ma_resource WHERE resource_key=?",
|
||||
"a".repeat(64), oldBody);
|
||||
case "wrong-id" -> jdbc.update("UPDATE ma_resource SET resource_id='unproven' WHERE resource_key=?", oldBody);
|
||||
case "wrong-body-scope", "zero-body-ordinal", "wrong-body-digest" -> {
|
||||
var original = jdbc.queryForMap("SELECT resource_id,session_scope_key FROM ma_resource WHERE resource_key=?", oldBody);
|
||||
String[] identity = ((String) original.get("resource_id")).split(":", -1);
|
||||
assertThat(identity).hasSize(4);
|
||||
switch (fault) {
|
||||
case "wrong-body-scope" -> identity[1] = "g".repeat(64);
|
||||
case "zero-body-ordinal" -> identity[2] = "0";
|
||||
case "wrong-body-digest" -> identity[3] = (identity[3].startsWith("0") ? "1" : "0")
|
||||
+ identity[3].substring(1);
|
||||
default -> throw new AssertionError(fault);
|
||||
}
|
||||
String id = String.join(":", identity);
|
||||
String key = ManagedSessionStore.resourceKey((String) original.get("session_scope_key"), id);
|
||||
assertThat(jdbc.update("UPDATE ma_resource SET resource_key=?,resource_id=? WHERE resource_key=?",
|
||||
key, id, oldBody)).isEqualTo(1);
|
||||
}
|
||||
default -> throw new AssertionError(fault);
|
||||
}
|
||||
assertThat(fixture.readReferences(expected).blockers()).as(fault).isNotEmpty();
|
||||
@@ -220,32 +260,42 @@ class WorkspaceCsiRetainedArtifactsTest {
|
||||
private void largeBody(int length) {
|
||||
append("item.output_text.delta", Map.of("itemId", "large", "text", "a".repeat(length)));
|
||||
materialize();
|
||||
assertThat(store.findSnapshot("foreign", session).orElseThrow().items().getFirst().content().getFirst().text()).hasSize(length);
|
||||
}
|
||||
|
||||
private void edit(String kind, Consumer<JsonNode> change) throws Exception {
|
||||
var rows = jdbc.queryForList("SELECT inline_bytes FROM ma_resource WHERE kind=?", byte[].class, kind);
|
||||
for (byte[] bytes : rows) {
|
||||
JsonNode value = JSON.readTree(bytes);
|
||||
change.accept(value);
|
||||
byte[] updated = JSON.writeValueAsBytes(value);
|
||||
String digest = WorkspaceEvidenceResources.digest(updated);
|
||||
jdbc.update("UPDATE ma_resource SET inline_bytes=?,byte_length=?,sha256=? WHERE kind=? AND sha256=?", updated, updated.length, digest, kind, WorkspaceEvidenceResources.digest(bytes));
|
||||
if (kind.equals("managed-snapshot-manifest")) {
|
||||
jdbc.update("UPDATE ma_snapshot SET manifest_sha256=? WHERE manifest_sha256=?", digest, WorkspaceEvidenceResources.digest(bytes));
|
||||
}
|
||||
String key = jdbc.queryForObject("SELECT body_resource_key FROM ma_message WHERE tenant_id=? AND session_id=?",
|
||||
String.class, "foreign", session);
|
||||
byte[] bytes = jdbc.queryForObject("SELECT inline_bytes FROM ma_resource WHERE resource_key=?", byte[].class, key);
|
||||
try {
|
||||
assertThat(JSON.readTree(bytes).get(0).path("text").textValue()).hasSize(length);
|
||||
} catch (java.io.IOException malformed) {
|
||||
throw new AssertionError(malformed);
|
||||
}
|
||||
}
|
||||
|
||||
private void replace(String kind, byte[] bytes) {
|
||||
jdbc.update("UPDATE ma_resource SET inline_bytes=?,byte_length=?,sha256=? WHERE kind=?", bytes, bytes.length, WorkspaceEvidenceResources.digest(bytes), kind);
|
||||
private void editBody(Consumer<JsonNode> change) throws Exception {
|
||||
byte[] bytes = jdbc.queryForObject("SELECT inline_bytes FROM ma_resource WHERE kind=?",
|
||||
byte[].class, "managed-message-parts");
|
||||
JsonNode value = JSON.readTree(bytes);
|
||||
change.accept(value);
|
||||
replaceBody(JSON.writeValueAsBytes(value));
|
||||
}
|
||||
|
||||
private void removeResources(String kind) {
|
||||
for (String key : jdbc.queryForList("SELECT resource_key FROM ma_resource WHERE kind=?", String.class, kind)) {
|
||||
jdbc.update("DELETE FROM ma_resource_ref WHERE resource_key=?", key);
|
||||
jdbc.update("DELETE FROM ma_resource WHERE resource_key=?", key);
|
||||
}
|
||||
private void replaceBody(byte[] bytes) {
|
||||
tx.executeWithoutResult(status -> {
|
||||
var original = jdbc.queryForMap("SELECT resource_key,session_scope_key,resource_id FROM ma_resource WHERE kind=?",
|
||||
"managed-message-parts");
|
||||
String oldKey = (String) original.get("resource_key");
|
||||
String oldId = (String) original.get("resource_id");
|
||||
String digest = WorkspaceEvidenceResources.digest(bytes);
|
||||
String id = oldId.substring(0, oldId.lastIndexOf(':') + 1) + digest;
|
||||
String key = ManagedSessionStore.resourceKey((String) original.get("session_scope_key"), id);
|
||||
assertThat(jdbc.update("INSERT INTO ma_resource (resource_key,session_scope_key,tenant_id,workspace_id,session_id,resource_id,kind,"
|
||||
+ "schema_version,byte_length,sha256,storage_kind,inline_bytes,state,gmt_create,last_verified_at,retention_until)"
|
||||
+ " SELECT ?,session_scope_key,tenant_id,workspace_id,session_id,?,kind,schema_version,?,?,storage_kind,?,state,"
|
||||
+ "gmt_create,last_verified_at,retention_until FROM ma_resource WHERE resource_key=?",
|
||||
key, id, bytes.length, digest, bytes, oldKey)).isEqualTo(1);
|
||||
assertThat(jdbc.update("UPDATE ma_message SET body_resource_key=? WHERE body_resource_key=?", key, oldKey)).isEqualTo(1);
|
||||
assertThat(jdbc.update("UPDATE ma_resource_ref SET resource_key=?,resource_id=? WHERE resource_key=?", key, id, oldKey)).isEqualTo(1);
|
||||
assertThat(jdbc.update("DELETE FROM ma_resource WHERE resource_key=?", oldKey)).isEqualTo(1);
|
||||
});
|
||||
}
|
||||
|
||||
private void replaceAgent(String kind, byte[] bytes) {
|
||||
|
||||
+3
-22
@@ -14,7 +14,6 @@ import com.alibaba.qwen.code.runtimebroker.CsiFilesRetirementProfile;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.WorkspaceExecutionProfile;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaProjection;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaTime;
|
||||
import com.fasterxml.jackson.core.StreamReadFeature;
|
||||
import com.fasterxml.jackson.databind.DeserializationFeature;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
@@ -472,27 +471,9 @@ class WorkspaceCsiSessionMainTest {
|
||||
for (int index = 0; index < 3; index++) {
|
||||
var original = create(request("backlog-" + index));
|
||||
insertLegacyOperation(original.sessionId());
|
||||
String resourceKey =
|
||||
com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaKeys.key(
|
||||
ManagedSessionStore.sessionScopeKey("tenant", original.sessionId()),
|
||||
"private-manifest");
|
||||
jdbc.update(
|
||||
"INSERT INTO ma_resource (resource_key, session_scope_key, tenant_id, session_id, resource_id, kind, schema_version, byte_length, sha256, storage_kind, inline_bytes, state, publish_command_id, gmt_create) VALUES (?, ?, 'tenant', ?, 'private-manifest', 'snapshot-manifest', 1, 2, ?, 'MYSQL_INLINE', ?, 'REFERENCED', 'private-fixture', " + ManagedSchemaTime.SQL_NOW + ")",
|
||||
resourceKey,
|
||||
ManagedSessionStore.sessionScopeKey("tenant", original.sessionId()),
|
||||
original.sessionId(),
|
||||
com.alibaba.qwen.code.runtimebroker.CsiNativeActivationProof.sha256(
|
||||
new byte[] {91, 93}),
|
||||
new byte[] {91, 93});
|
||||
jdbc.update(
|
||||
"INSERT INTO ma_snapshot (tenant_id, session_id, snapshot_id, snapshot_version, covered_sequence, projection_version, manifest_resource_key, manifest_sha256, item_count, state, gmt_create, retain_until) VALUES ('tenant', ?, 'private-snapshot', 1, 1, 1, ?, ?, 0, 'PUBLISHED', 0, 0)",
|
||||
original.sessionId(),
|
||||
resourceKey,
|
||||
com.alibaba.qwen.code.runtimebroker.CsiNativeActivationProof.sha256(
|
||||
new byte[] {91, 93}));
|
||||
jdbc.update(
|
||||
"UPDATE ma_event_stream SET covered_sequence = 1, current_snapshot_id ="
|
||||
+ " 'private-snapshot' WHERE tenant_id = X'74656e616e74' AND session_id ="
|
||||
"UPDATE ma_event_stream SET covered_sequence = 1"
|
||||
+ " WHERE tenant_id = X'74656e616e74' AND session_id ="
|
||||
+ " ?",
|
||||
original.sessionId());
|
||||
}
|
||||
@@ -645,7 +626,7 @@ class WorkspaceCsiSessionMainTest {
|
||||
.sorted(Comparator.comparing(JsonNode::toString)).toList();
|
||||
result.put(table, json.valueToTree(rows));
|
||||
}
|
||||
assertThat(result).hasSize(40);
|
||||
assertThat(result).hasSize(39);
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
+1
-1
@@ -461,7 +461,7 @@ class WorkspaceCsiSqlWorkerMySqlIT {
|
||||
}
|
||||
}
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = ?"
|
||||
+ " AND table_type = 'BASE TABLE'", Integer.class, schema)).isEqualTo(40);
|
||||
+ " AND table_type = 'BASE TABLE'", Integer.class, schema)).isEqualTo(39);
|
||||
verify(provisioner).ensureResource(any(), any(), any());
|
||||
verify(provisioner).provision(any(), any());
|
||||
String evidence = System.getProperty("csi.acceptance.evidence");
|
||||
|
||||
+3
-3
@@ -201,11 +201,11 @@ class WorkspaceSchemaConsolidationTest {
|
||||
java.util.List<Object[]> resources = new java.util.ArrayList<>();
|
||||
java.util.List<Object[]> roots = new java.util.ArrayList<>();
|
||||
for (int index = 0; index < 4098; index++) {
|
||||
String id = index == 0 ? "checkpoint" : "public-page-" + index;
|
||||
String id = index == 0 ? "checkpoint" : "public-body-" + index;
|
||||
String resource = WorkspaceOperations.hash(scope, id);
|
||||
resources.add(new Object[] {resource, scope, id.getBytes(StandardCharsets.UTF_8),
|
||||
index == 0 ? "managed-checkpoint" : "managed-public-snapshot-page", WorkspaceEvidenceResources.digest(body), body});
|
||||
roots.add(new Object[] {scope, index == 0 ? "JOURNAL" : "SNAPSHOT", WorkspaceOperations.hash("owner", id), resource,
|
||||
index == 0 ? "managed-checkpoint" : "managed-message-parts", WorkspaceEvidenceResources.digest(body), body});
|
||||
roots.add(new Object[] {scope, index == 0 ? "JOURNAL" : "MESSAGE", WorkspaceOperations.hash("owner", id), resource,
|
||||
index == 0 ? 1L : null, id.getBytes(StandardCharsets.UTF_8)});
|
||||
}
|
||||
jdbc.batchUpdate("INSERT INTO ma_resource (resource_key, session_scope_key, tenant_id, workspace_id, session_id, resource_id, kind, schema_version, byte_length, sha256, storage_kind, inline_bytes, publish_command_id, state, gmt_create) VALUES (?, ?, 'tenant', 'root', 'session', ?, ?, 1, 2, ?, 'MYSQL_INLINE', ?, 'fixture', 'REFERENCED', " + ManagedSchemaTime.SQL_NOW + ")", resources);
|
||||
|
||||
+94
-318
@@ -5,7 +5,6 @@ import com.alibaba.qwen.code.managedagent.artifact.ToolPublicationObjectStore;
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
|
||||
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector;
|
||||
import com.alibaba.qwen.code.managedagent.model.RequestDigests;
|
||||
import com.alibaba.qwen.code.managedagent.model.StoreModels.MaterializationTarget;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedAgentStore;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedSessionStore;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedSessionStoreModels;
|
||||
@@ -40,7 +39,6 @@ import java.sql.ResultSet;
|
||||
import java.sql.Statement;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
@@ -72,11 +70,8 @@ import static org.mockito.Mockito.when;
|
||||
* production store/service classes exactly as the runtime wiring does.
|
||||
*
|
||||
* <ol>
|
||||
* <li>materializeNextBatch rewrites the snapshot only at creation, at
|
||||
* every SNAPSHOT_REFRESH_EVENTS covered events, on a batch carrying a
|
||||
* terminal event, or on a caught-up batch once the previous write is
|
||||
* SNAPSHOT_REFRESH_MILLIS old — not on every batch; a deferred write
|
||||
* converges via the snapshot_stale_since marker.</li>
|
||||
* <li>materializeNextBatch reads only the current batch and referenced message
|
||||
* parts; it never reads the complete history or writes public snapshots.</li>
|
||||
* <li>The SSE fan-out re-checks the workspace read grant once per recheck
|
||||
* window, not per delivered event.</li>
|
||||
* <li>listPublicSessions / listWebShellSessions assemble a page from a
|
||||
@@ -93,7 +88,7 @@ class Issue13181QueryBudgetTest {
|
||||
private ObjectNode checkpoint;
|
||||
|
||||
@Test
|
||||
void materializationRewritesTheSnapshotAtCreationAndTerminalEvents() {
|
||||
void materializationReadsOnlyTheCurrentBatchAndNeverWritesSnapshots() {
|
||||
Fixture fixture = new Fixture();
|
||||
String tenant = "tenant-" + UUID.randomUUID();
|
||||
String sessionId = fixture.store.insertSessionCommand(tenant,
|
||||
@@ -112,8 +107,7 @@ class Issue13181QueryBudgetTest {
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"item.output_text.delta", Map.of("text", "part"), false,
|
||||
"mat-part");
|
||||
// The burst ends with a terminal event, so the batch that covers it
|
||||
// rewrites the snapshot even though the snapshot is fresh.
|
||||
// The terminal event settles the message within the final bounded batch.
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"turn.completed", Map.of(), true, "mat-terminal");
|
||||
// 83 events with the session's own; the last batch covers the tail.
|
||||
@@ -143,221 +137,50 @@ class Issue13181QueryBudgetTest {
|
||||
+ " partReads=" + partReads
|
||||
+ " snapshotRewrites=" + snapshotWrites
|
||||
+ " totals=" + totals);
|
||||
// The first batch inserts the snapshot row; the intermediate batches
|
||||
// leave it alone; the batch carrying the terminal event rewrites it.
|
||||
assertThat(snapshotWrites)
|
||||
.containsExactly(1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L);
|
||||
// The whole item table is read exactly once per writing batch, and
|
||||
// the per-event attributes read fires once per tool event — one item
|
||||
// statement per covered event plus the page read, nothing more. The
|
||||
// parts read rides the same discipline.
|
||||
assertThat(snapshotWrites).containsOnly(0L);
|
||||
// Attributes are read per tool event; the last batch also reads its text
|
||||
// part and revision while flushing that one message.
|
||||
assertThat(itemReads)
|
||||
.containsExactly(11L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 5L);
|
||||
.containsExactly(9L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 3L);
|
||||
// The parts read rides the same discipline; the last batch also
|
||||
// projects the text delta, which reads the part it continues.
|
||||
assertThat(partReads)
|
||||
.containsExactly(0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L);
|
||||
// Hot path 1 is bounded the same way hot path 3 is: the whole
|
||||
// batch's statement count is pinned, so any added per-row read
|
||||
// shows here. Session and stream counters lock separately once per batch.
|
||||
// Five statements acquire Session/stream progress, read the batch and
|
||||
// commit coverage. Each new tool costs three; the final batch also
|
||||
// reads/inserts/flushes one text message and settles its items.
|
||||
assertThat(totals)
|
||||
.containsExactly(61L, 36L, 36L, 36L, 36L, 36L, 36L, 36L, 44L);
|
||||
// Batch 1 covers session.created plus nine tool events; the last
|
||||
// batch sees all 81 items (80 tool plus the text delta's assistant).
|
||||
assertThat(itemRowsRead)
|
||||
.containsExactly(9L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 81L);
|
||||
// The caught-up snapshot is complete and self-consistent.
|
||||
assertThat(fixture.store.findSnapshot(tenant, sessionId)).get()
|
||||
.satisfies(snapshot -> {
|
||||
assertThat(snapshot.coveredSequence()).isEqualTo(fixture
|
||||
.store.requireSession(tenant, sessionId)
|
||||
.lastSequence());
|
||||
assertThat(snapshot.items()).hasSize(81);
|
||||
});
|
||||
.containsExactly(32L, 35L, 35L, 35L, 35L, 35L, 35L, 35L, 15L);
|
||||
assertThat(itemRowsRead).containsOnly(0L);
|
||||
var page = fixture.tx.execute(status ->
|
||||
fixture.store.findMessagePage(tenant, sessionId, null, 100, false));
|
||||
assertThat(page.coveredSequence()).isEqualTo(fixture
|
||||
.store.requireSession(tenant, sessionId).lastSequence());
|
||||
assertThat(page.items()).hasSize(81);
|
||||
}
|
||||
|
||||
@Test
|
||||
void trickleTicksDoNotRewriteTheSnapshotPerTick() {
|
||||
// A frozen clock: the 5s age floor can never trip mid-loop, so the
|
||||
// exact per-tick write counts do not depend on the wall clock.
|
||||
Fixture fixture = new Fixture(
|
||||
Clock.fixed(Instant.now(), java.time.ZoneOffset.UTC));
|
||||
String tenant = "tenant-" + UUID.randomUUID();
|
||||
String sessionId = fixture.store.insertSessionCommand(tenant,
|
||||
"CREATE_SESSION", "trickle-" + UUID.randomUUID(), "digest",
|
||||
"qwen-code", null, null, List.of(), null).sessionId();
|
||||
// The workload the issue calls common: a trickle of events fully
|
||||
// drained by every scheduler tick (fewer than a batch per tick), so
|
||||
// catch-up alone must not rewrite the snapshot each tick.
|
||||
List<Long> snapshotWrites = new ArrayList<>();
|
||||
for (int tick = 0; tick < 12; tick++) {
|
||||
for (int index = 0; index < 5; index++) {
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId,
|
||||
"turn-1", "item.tool_call.updated",
|
||||
Map.of("callId", "call-" + tick + "-" + index,
|
||||
"status", "completed"),
|
||||
false, "trickle-" + tick + "-" + index);
|
||||
}
|
||||
fixture.ledger.reset();
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 200));
|
||||
snapshotWrites.add(fixture.ledger.count("insert into ma_snapshot"));
|
||||
}
|
||||
System.out.println("[issue-13181] trickle snapshotRewrites per tick: "
|
||||
+ snapshotWrites);
|
||||
assertThat(snapshotWrites)
|
||||
.containsExactly(1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L,
|
||||
0L);
|
||||
}
|
||||
|
||||
@Test
|
||||
void deferredSnapshotConvergesOnTheAgedOutReselection() {
|
||||
// A frozen clock: the deferral and the reselection both key off the
|
||||
// 5s age floor, which the test ages explicitly by SQL.
|
||||
Fixture fixture = new Fixture(
|
||||
Clock.fixed(Instant.now(), java.time.ZoneOffset.UTC));
|
||||
String tenant = "tenant-" + UUID.randomUUID();
|
||||
String sessionId = fixture.store.insertSessionCommand(tenant,
|
||||
"CREATE_SESSION", "converge-" + UUID.randomUUID(), "digest",
|
||||
"qwen-code", null, null, List.of(), null).sessionId();
|
||||
// The first drain creates the snapshot; the second drains below the
|
||||
// thresholds and defers the rewrite, leaving the snapshot behind.
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"item.tool_call.updated",
|
||||
Map.of("callId", "call-0", "status", "completed"),
|
||||
false, "converge-0");
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 200));
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"item.tool_call.updated",
|
||||
Map.of("callId", "call-1", "status", "completed"),
|
||||
false, "converge-1");
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 200));
|
||||
long snapshotCovered = fixture.store
|
||||
.findSnapshotCoveredSequences(tenant, List.of(sessionId))
|
||||
.getOrDefault(sessionId, 0L);
|
||||
long lastSequence = fixture.store.requireSession(tenant, sessionId)
|
||||
.lastSequence();
|
||||
assertThat(snapshotCovered).isLessThan(lastSequence);
|
||||
// A caught-up session is not a target while its snapshot is fresh...
|
||||
assertThat(fixture.store.findMaterializationTargets(32))
|
||||
.doesNotContain(new MaterializationTarget(tenant, sessionId));
|
||||
// ...but is re-selected once the deferral MARKER alone ages out —
|
||||
// the snapshot row stays fresh, so a regression to probing
|
||||
// ma_snapshot per row would not select it here.
|
||||
fixture.jdbc.update("UPDATE ma_event_stream SET"
|
||||
+ " snapshot_stale_since = snapshot_stale_since - 6000"
|
||||
+ " WHERE tenant_id = ? AND session_id = ?",
|
||||
tenant, sessionId);
|
||||
fixture.ledger.reset();
|
||||
assertThat(fixture.store.findMaterializationTargets(32))
|
||||
.contains(new MaterializationTarget(tenant, sessionId));
|
||||
assertThat(fixture.ledger.count("snapshot_stale_since"))
|
||||
.isGreaterThan(0);
|
||||
assertThat(fixture.ledger.count("ma_snapshot")).isZero();
|
||||
// The convergence tick still waits for the snapshot's own age.
|
||||
fixture.jdbc.update("UPDATE ma_snapshot SET published_at ="
|
||||
+ " published_at - 6000 WHERE tenant_id = ? AND"
|
||||
+ " session_id = ?", tenant, sessionId);
|
||||
// The re-selected tick has no new events; it converges the snapshot.
|
||||
fixture.ledger.reset();
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 200));
|
||||
assertThat(fixture.store.findSnapshotCoveredSequences(tenant,
|
||||
List.of(sessionId)).getOrDefault(sessionId, 0L))
|
||||
.isEqualTo(lastSequence);
|
||||
assertThat(fixture.ledger.count("insert into ma_snapshot"))
|
||||
.isEqualTo(1);
|
||||
}
|
||||
|
||||
@Test
|
||||
void caughtUpBatchRewritesAnAgedOutSnapshotInline() {
|
||||
void emptyBatchReadsOnlyProgressAndDoesNotScheduleCaughtUpSessions() {
|
||||
Fixture fixture = new Fixture();
|
||||
String tenant = "tenant-" + UUID.randomUUID();
|
||||
String sessionId = fixture.store.insertSessionCommand(tenant,
|
||||
"CREATE_SESSION", "inline-" + UUID.randomUUID(), "digest",
|
||||
"CREATE_SESSION", "empty-" + UUID.randomUUID(), "digest",
|
||||
"qwen-code", null, null, List.of(), null).sessionId();
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"item.tool_call.updated",
|
||||
Map.of("callId", "call-0", "status", "completed"),
|
||||
false, "inline-0");
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 200));
|
||||
// Age the snapshot past the 5s floor, then drain one more event:
|
||||
// the catching-up batch must rewrite inline instead of waiting for
|
||||
// an aged-out reselection tick.
|
||||
fixture.jdbc.update("UPDATE ma_snapshot SET published_at ="
|
||||
+ " published_at - 6000 WHERE tenant_id = ? AND"
|
||||
+ " session_id = ?", tenant, sessionId);
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"item.tool_call.updated",
|
||||
Map.of("callId", "call-1", "status", "completed"),
|
||||
false, "inline-1");
|
||||
fixture.ledger.reset();
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
var result = fixture.tx.execute(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 200));
|
||||
assertThat(fixture.store.findSnapshotCoveredSequences(tenant,
|
||||
List.of(sessionId)).getOrDefault(sessionId, 0L))
|
||||
.isEqualTo(fixture.store.requireSession(tenant, sessionId)
|
||||
.lastSequence());
|
||||
assertThat(fixture.ledger.count("insert into ma_snapshot"))
|
||||
.isEqualTo(1);
|
||||
assertThat(result.advanced()).isFalse();
|
||||
assertThat(result.coveredSequence()).isEqualTo(1);
|
||||
assertThat(fixture.ledger.total()).isEqualTo(4);
|
||||
assertThat(fixture.ledger.count("ma_snapshot")).isZero();
|
||||
assertThat(fixture.ledger.count("from ma_message")).isZero();
|
||||
assertThat(fixture.store.findMaterializationTargets(32)).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
void emptyBatchDoesNotRewriteAFreshLaggingSnapshot() {
|
||||
// A frozen clock: the debounce window never lapses mid-test.
|
||||
Fixture fixture = new Fixture(
|
||||
Clock.fixed(Instant.now(), java.time.ZoneOffset.UTC));
|
||||
String tenant = "tenant-" + UUID.randomUUID();
|
||||
String sessionId = fixture.store.insertSessionCommand(tenant,
|
||||
"CREATE_SESSION", "debounce-" + UUID.randomUUID(), "digest",
|
||||
"qwen-code", null, null, List.of(), null).sessionId();
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"item.tool_call.updated",
|
||||
Map.of("callId", "call-0", "status", "completed"),
|
||||
false, "debounce-0");
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 200));
|
||||
// A drained trickle batch leaves the fresh snapshot one event
|
||||
// behind; an empty tick arriving before the snapshot ages out must
|
||||
// not rewrite it.
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"item.tool_call.updated",
|
||||
Map.of("callId", "call-1", "status", "completed"),
|
||||
false, "debounce-1");
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 200));
|
||||
Long marker = fixture.jdbc.queryForObject(
|
||||
"SELECT snapshot_stale_since FROM"
|
||||
+ " ma_event_stream WHERE tenant_id = ?"
|
||||
+ " AND session_id = ?",
|
||||
Long.class, tenant, sessionId);
|
||||
assertThat(marker).isNotNull();
|
||||
fixture.ledger.reset();
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 200));
|
||||
assertThat(fixture.ledger.count("insert into ma_snapshot"))
|
||||
.isZero();
|
||||
// The empty tick — the scheduler's hottest path — is pinned whole:
|
||||
// the event read, the session lock, the stream counter lock, the
|
||||
// progress lock, and the snapshot read.
|
||||
assertThat(fixture.ledger.total()).isEqualTo(5);
|
||||
// The not-aged exit must leave the deferral marker untouched —
|
||||
// clearing or re-stamping it would stop the aged reselection from
|
||||
// ever converging the snapshot.
|
||||
assertThat(fixture.jdbc.queryForObject(
|
||||
"SELECT snapshot_stale_since FROM"
|
||||
+ " ma_event_stream WHERE tenant_id = ?"
|
||||
+ " AND session_id = ?",
|
||||
Long.class, tenant, sessionId)).isEqualTo(marker);
|
||||
assertThat(fixture.store.findMaterializationTargets(32))
|
||||
.doesNotContain(new MaterializationTarget(tenant, sessionId));
|
||||
}
|
||||
|
||||
@Test
|
||||
void replayFloorStaysClampedToALaggingSnapshot() {
|
||||
void replayFloorStaysClampedToTheLaggingMessageProjection() {
|
||||
Fixture fixture = new Fixture();
|
||||
String tenant = "tenant-" + UUID.randomUUID();
|
||||
String sessionId = fixture.store.insertSessionCommand(tenant,
|
||||
@@ -369,75 +192,26 @@ class Issue13181QueryBudgetTest {
|
||||
Map.of("callId", "call-" + index, "status", "completed"),
|
||||
false, "floor-" + index);
|
||||
}
|
||||
// Cover all five events so the snapshot exists, then append more:
|
||||
// the projection advances past the snapshot, which now lags.
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 100));
|
||||
long snapshotCovered = fixture.store
|
||||
.findSnapshotCoveredSequences(tenant, List.of(sessionId))
|
||||
long coveredBefore = fixture.store
|
||||
.findMessageCoveredSequences(tenant, List.of(sessionId))
|
||||
.getOrDefault(sessionId, 0L);
|
||||
for (int index = 0; index < 3; index++) {
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId,
|
||||
"turn-1", "item.tool_call.updated",
|
||||
Map.of("callId", "call-lag-" + index, "status",
|
||||
"completed"),
|
||||
Map.of("callId", "call-lag-" + index, "status", "completed"),
|
||||
false, "floor-lag-" + index);
|
||||
}
|
||||
// A partial batch advances the projection but, below the threshold,
|
||||
// leaves the snapshot behind.
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 2));
|
||||
assertThat(fixture.store.findSnapshotCoveredSequences(tenant,
|
||||
List.of(sessionId)).getOrDefault(sessionId, 0L))
|
||||
.isEqualTo(snapshotCovered);
|
||||
long lastSequence = fixture.store.requireSession(tenant, sessionId)
|
||||
.lastSequence();
|
||||
assertThat(lastSequence).isGreaterThan(snapshotCovered);
|
||||
// The floor never advances past what the snapshot covers.
|
||||
assertThat(fixture.store
|
||||
.advanceReplayFloor(tenant, sessionId, lastSequence)
|
||||
.floorSequence()).isEqualTo(snapshotCovered);
|
||||
}
|
||||
|
||||
@Test
|
||||
void materializationRewritesTheSnapshotOnTheEventThreshold() {
|
||||
Fixture fixture = new Fixture();
|
||||
String tenant = "tenant-" + UUID.randomUUID();
|
||||
String sessionId = fixture.store.insertSessionCommand(tenant,
|
||||
"CREATE_SESSION", "thr-" + UUID.randomUUID(), "digest",
|
||||
"qwen-code", null, null, List.of(), null).sessionId();
|
||||
// Establish the current snapshot with a small caught-up burst.
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"item.tool_call.updated",
|
||||
Map.of("callId", "call-first", "status", "completed"), false,
|
||||
"thr-first");
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 100));
|
||||
// A burst of 1500 events plus the turn's terminal event,
|
||||
// materialized in 500-event batches: the first batch stays 500
|
||||
// events below the threshold, the second crosses it (1000 covered
|
||||
// since the snapshot), the tail batch carries the terminal event.
|
||||
for (int index = 0; index < 1500; index++) {
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId,
|
||||
"turn-1", "item.tool_call.updated",
|
||||
Map.of("callId", "call-" + index, "status", "completed"),
|
||||
false, "thr-src-" + index);
|
||||
}
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"turn.completed", Map.of(), true, "thr-terminal");
|
||||
List<Long> snapshotWrites = new ArrayList<>();
|
||||
for (int batch = 0; batch < 4; batch++) {
|
||||
fixture.ledger.reset();
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 500));
|
||||
snapshotWrites.add(fixture.ledger.count("insert into ma_snapshot"));
|
||||
}
|
||||
assertThat(snapshotWrites).containsExactly(0L, 1L, 0L, 1L);
|
||||
assertThat(fixture.store.findSnapshot(tenant, sessionId)).get()
|
||||
.satisfies(snapshot -> assertThat(snapshot.coveredSequence())
|
||||
.isEqualTo(fixture.store
|
||||
.requireSession(tenant, sessionId)
|
||||
.lastSequence()));
|
||||
long covered = fixture.store.findMessageCoveredSequences(tenant,
|
||||
List.of(sessionId)).getOrDefault(sessionId, 0L);
|
||||
assertThat(covered).isEqualTo(coveredBefore + 2);
|
||||
long lastSequence = fixture.store.requireSession(tenant, sessionId).lastSequence();
|
||||
assertThat(covered).isLessThan(lastSequence);
|
||||
assertThat(fixture.store.advanceReplayFloor(tenant, sessionId, lastSequence)
|
||||
.floorSequence()).isEqualTo(covered);
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -560,7 +334,7 @@ class Issue13181QueryBudgetTest {
|
||||
plainTurnIds.add(turnId);
|
||||
}
|
||||
// Sessions 2 and 5 carry distinct environment events on their latest
|
||||
// turns; materializing then produces a real snapshot covered value
|
||||
// turns; materializing then produces a real message coverage value
|
||||
// per session.
|
||||
for (int index : new int[] {2, 5}) {
|
||||
fixture.store.appendPublicEventIfAbsent(plain, plainIds.get(index),
|
||||
@@ -583,7 +357,7 @@ class Issue13181QueryBudgetTest {
|
||||
assertThat(publicPage).hasSize(20);
|
||||
System.out.println("[issue-13181] listPublicSessions(20 rows): "
|
||||
+ fixture.ledger.summary());
|
||||
// Page + active turns + snapshot covered sequences.
|
||||
// Page + active turns + message coverage.
|
||||
assertThat(fixture.ledger.total()).isEqualTo(3);
|
||||
// The batch turn read projects only the summary columns; the page
|
||||
// never holds a parsed prompt graph per row.
|
||||
@@ -593,7 +367,7 @@ class Issue13181QueryBudgetTest {
|
||||
+ " gmt_create, completed_at, error_code"
|
||||
+ " from ma_turn")).isEqualTo(1);
|
||||
// Every row carries its own title, its own active turn, and its own
|
||||
// snapshot coverage: session 7 is the one without an active turn,
|
||||
// message coverage: session 7 is the one without an active turn,
|
||||
// and the two environment sessions were covered through one more
|
||||
// event.
|
||||
for (var row : publicPage) {
|
||||
@@ -996,21 +770,21 @@ class Issue13181QueryBudgetTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void transcriptServesEveryEventPastTheSnapshot() {
|
||||
void transcriptLeavesUnmaterializedTailForEventReplay() {
|
||||
Fixture fixture = new Fixture();
|
||||
String tenant = "tenant-" + UUID.randomUUID();
|
||||
String sessionId = fixture.store.insertSessionCommand(tenant,
|
||||
"CREATE_SESSION", "tail-" + UUID.randomUUID(), "digest",
|
||||
"qwen-code", null, null, List.of(), null).sessionId();
|
||||
// Five events materialize so a snapshot exists; 150 more are
|
||||
// appended and only partially materialized, leaving the snapshot
|
||||
// behind with a long tail.
|
||||
// A partial drain leaves committed events beyond the message projection.
|
||||
for (int index = 0; index < 5; index++) {
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId,
|
||||
"turn-1", "item.tool_call.updated",
|
||||
Map.of("callId", "call-" + index, "status", "completed"),
|
||||
false, "tail-" + index);
|
||||
}
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"turn.started", Map.of(), false, "tail-start");
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 100));
|
||||
for (int index = 0; index < 150; index++) {
|
||||
@@ -1022,34 +796,35 @@ class Issue13181QueryBudgetTest {
|
||||
}
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 100));
|
||||
long snapshotCovered = fixture.store
|
||||
.findSnapshotCoveredSequences(tenant, List.of(sessionId))
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"turn.completed", Map.of(), true, "tail-complete");
|
||||
long covered = fixture.store
|
||||
.findMessageCoveredSequences(tenant, List.of(sessionId))
|
||||
.getOrDefault(sessionId, 0L);
|
||||
long lastSequence = fixture.store.requireSession(tenant, sessionId)
|
||||
.lastSequence();
|
||||
assertThat(snapshotCovered).isLessThan(lastSequence);
|
||||
long lastSequence = fixture.store.requireSession(tenant, sessionId).lastSequence();
|
||||
assertThat(covered).isLessThan(lastSequence);
|
||||
fixture.ledger.reset();
|
||||
var transcript = fixture.service.transcript(tenant, null, sessionId,
|
||||
null, 10);
|
||||
// The published contract: the cursor-less page serves every event
|
||||
// after the snapshot, so the stream can resume at the reported
|
||||
// watermark with no band left unserved; the tail is one read sized
|
||||
// by the snapshot gate's lag bound.
|
||||
assertThat(transcript.events().stream()
|
||||
.filter(event -> event.sequence()
|
||||
> transcript.coveredSequence())
|
||||
.count()).isEqualTo(lastSequence - snapshotCovered);
|
||||
assertThat(transcript.events().getLast().sequence())
|
||||
.isEqualTo(lastSequence);
|
||||
assertThat(transcript.hasMore()).isFalse();
|
||||
assertThat(transcript.olderCursor()).isNull();
|
||||
assertThat(transcript.lastSequence()).isEqualTo(lastSequence);
|
||||
assertThat(fixture.ledger.count("from ma_public_event",
|
||||
"sequence_id >")).isEqualTo(1);
|
||||
var transcript = fixture.tx.execute(status ->
|
||||
fixture.service.transcript(tenant, null, sessionId, null, 10));
|
||||
assertThat(transcript.items()).hasSize(10);
|
||||
assertThat(transcript.hasMore()).isTrue();
|
||||
assertThat(transcript.olderCursor()).isNotNull();
|
||||
assertThat(transcript.coveredSequence()).isEqualTo(covered);
|
||||
assertThat(transcript.lastSequence()).isEqualTo(covered);
|
||||
assertThat(transcript.events()).extracting(event -> event.type()).containsExactly("turn.started");
|
||||
assertThat(transcript.events()).allSatisfy(event ->
|
||||
assertThat(event.sequence()).isLessThanOrEqualTo(covered));
|
||||
assertThat(fixture.ledger.count("from ma_public_event", "sequence_id >")).isZero();
|
||||
assertThat(fixture.ledger.rows("from ma_message m", "limit")).isEqualTo(11);
|
||||
var tail = fixture.service.publicEvents(tenant, null, sessionId,
|
||||
transcript.lastSequence(), 100).data();
|
||||
assertThat(tail).hasSize((int) (lastSequence - covered));
|
||||
assertThat(tail.getFirst().sequence()).isEqualTo(covered + 1);
|
||||
assertThat(tail.getLast().sequence()).isEqualTo(lastSequence);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aMaximallyLaggingSnapshotTailIsOneRead() {
|
||||
void transcriptReadBudgetDoesNotGrowWithUnmaterializedTail() {
|
||||
Fixture fixture = new Fixture();
|
||||
String tenant = "tenant-" + UUID.randomUUID();
|
||||
String sessionId = fixture.store.insertSessionCommand(tenant,
|
||||
@@ -1061,47 +836,48 @@ class Issue13181QueryBudgetTest {
|
||||
Map.of("callId", "call-" + index, "status", "completed"),
|
||||
false, "lag-" + index);
|
||||
}
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"turn.started", Map.of(), false, "lag-start");
|
||||
fixture.tx.executeWithoutResult(status -> fixture.store
|
||||
.materializeNextBatch(tenant, sessionId, 100));
|
||||
long covered = fixture.store
|
||||
.findSnapshotCoveredSequences(tenant, List.of(sessionId))
|
||||
long covered = fixture.store.findMessageCoveredSequences(tenant, List.of(sessionId))
|
||||
.getOrDefault(sessionId, 0L);
|
||||
// The gate's lag ceiling: 999 events appended, none materialized.
|
||||
for (int index = 0; index < 999; index++) {
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId,
|
||||
"turn-1", "item.tool_call.updated",
|
||||
Map.of("callId", "call-b-" + index, "status",
|
||||
"completed"),
|
||||
Map.of("callId", "call-b-" + index, "status", "completed"),
|
||||
false, "lag-b-" + index);
|
||||
}
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId, "turn-1",
|
||||
"turn.completed", Map.of(), true, "lag-complete");
|
||||
fixture.ledger.reset();
|
||||
var transcript = fixture.service.transcript(tenant, null, sessionId,
|
||||
null, 10);
|
||||
// The full 999-event tail is served, in one event read.
|
||||
assertThat(transcript.events().stream()
|
||||
.filter(event -> event.sequence() > covered).count())
|
||||
.isEqualTo(999);
|
||||
var transcript = fixture.tx.execute(status ->
|
||||
fixture.service.transcript(tenant, null, sessionId, null, 10));
|
||||
assertThat(transcript.items()).hasSize(5);
|
||||
assertThat(transcript.events()).extracting(event -> event.type()).containsExactly("turn.started");
|
||||
assertThat(transcript.coveredSequence()).isEqualTo(covered);
|
||||
assertThat(transcript.lastSequence()).isEqualTo(covered);
|
||||
assertThat(transcript.events()).allSatisfy(event ->
|
||||
assertThat(event.sequence()).isLessThanOrEqualTo(covered));
|
||||
assertThat(transcript.hasMore()).isFalse();
|
||||
assertThat(fixture.ledger.count("from ma_public_event",
|
||||
"sequence_id >")).isEqualTo(1);
|
||||
// A tail longer than one page (a session the materializer never
|
||||
// visited) is served completely, paging per 1000.
|
||||
assertThat(fixture.ledger.count("from ma_public_event", "sequence_id >")).isZero();
|
||||
long reads = fixture.ledger.total();
|
||||
for (int index = 0; index < 600; index++) {
|
||||
fixture.store.appendPublicEventIfAbsent(tenant, sessionId,
|
||||
"turn-1", "item.tool_call.updated",
|
||||
Map.of("callId", "call-c-" + index, "status",
|
||||
"completed"),
|
||||
Map.of("callId", "call-c-" + index, "status", "completed"),
|
||||
false, "lag-c-" + index);
|
||||
}
|
||||
fixture.ledger.reset();
|
||||
var longer = fixture.service.transcript(tenant, null, sessionId,
|
||||
null, 10);
|
||||
assertThat(longer.events().stream()
|
||||
.filter(event -> event.sequence() > covered).count())
|
||||
.isEqualTo(1599);
|
||||
var longer = fixture.tx.execute(status ->
|
||||
fixture.service.transcript(tenant, null, sessionId, null, 10));
|
||||
assertThat(longer.items()).isEqualTo(transcript.items());
|
||||
assertThat(longer.events()).isEqualTo(transcript.events());
|
||||
assertThat(longer.coveredSequence()).isEqualTo(covered);
|
||||
assertThat(longer.lastSequence()).isEqualTo(covered);
|
||||
assertThat(longer.hasMore()).isFalse();
|
||||
assertThat(fixture.ledger.count("from ma_public_event",
|
||||
"sequence_id >")).isEqualTo(2);
|
||||
assertThat(fixture.ledger.count("from ma_public_event", "sequence_id >")).isZero();
|
||||
assertThat(fixture.ledger.total()).isEqualTo(reads);
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
-4
@@ -344,10 +344,6 @@ class ManagedEventReplayTest {
|
||||
await().atMost(Duration.ofSeconds(10)).until(() ->
|
||||
store.findReplayWindow(tenant, sessionId)
|
||||
.snapshotThroughSequence() == last);
|
||||
// Older cursors have passed their thirty-minute retention window in this scenario.
|
||||
jdbc.update("UPDATE ma_snapshot SET retain_until = 0 WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " AND snapshot_id <> (SELECT current_snapshot_id FROM ma_event_stream WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " AND stream_kind = 'SESSION' AND stream_id = X'')", tenant, sessionId, tenant, sessionId);
|
||||
return store.advanceReplayFloor(tenant, sessionId, floor);
|
||||
}
|
||||
|
||||
|
||||
+4
-16
@@ -5,7 +5,6 @@ import com.alibaba.qwen.code.managedagent.model.StoreModels.ReplayFloorTarget;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedAgentStore;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
|
||||
import java.time.Duration;
|
||||
@@ -27,8 +26,8 @@ import static org.mockito.Mockito.verifyNoInteractions;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
/**
|
||||
* The retention pass raises a Session's replay floor as far as its Snapshot
|
||||
* proves safe, and only when it is enabled: the store is never touched while
|
||||
* The retention pass raises a Session's replay floor through its message projection
|
||||
* coverage, and only when it is enabled: the store is never touched while
|
||||
* {@code replay-floor-enabled} is false, a converged Session is never
|
||||
* selected again, one failing Session does not stop the rest, and a backlog
|
||||
* beyond one batch is drained within the pass without retrying the failure.
|
||||
@@ -48,12 +47,8 @@ class ReplayFloorAdvancerTest {
|
||||
@Autowired
|
||||
private ManagedAgentStore store;
|
||||
|
||||
@Autowired
|
||||
@Qualifier("managedJdbcTemplate")
|
||||
private org.springframework.jdbc.core.JdbcTemplate jdbc;
|
||||
|
||||
@Test
|
||||
void scheduledPassRaisesTheFloorToTheSnapshotCoverage() {
|
||||
void scheduledPassRaisesTheFloorToTheMessageCoverage() {
|
||||
String tenant = tenant();
|
||||
String sessionId = session(tenant);
|
||||
long last = append(tenant, sessionId, 5);
|
||||
@@ -61,10 +56,6 @@ class ReplayFloorAdvancerTest {
|
||||
await().atMost(Duration.ofSeconds(10)).until(() ->
|
||||
store.findReplayWindow(tenant, sessionId)
|
||||
.snapshotThroughSequence() == last);
|
||||
// This pass runs after any earlier page cursors have expired.
|
||||
jdbc.update("UPDATE ma_snapshot SET retain_until = 0 WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " AND snapshot_id <> (SELECT current_snapshot_id FROM ma_event_stream WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " AND stream_kind = 'SESSION' AND stream_id = X'')", tenant, sessionId, tenant, sessionId);
|
||||
await().atMost(Duration.ofSeconds(10)).until(() ->
|
||||
store.findReplayWindow(tenant, sessionId)
|
||||
.floorSequence() == last);
|
||||
@@ -76,9 +67,6 @@ class ReplayFloorAdvancerTest {
|
||||
String sessionId = session(tenant);
|
||||
long last = append(tenant, sessionId, 3);
|
||||
await().atMost(Duration.ofSeconds(10)).until(() -> store.findReplayWindow(tenant, sessionId).snapshotThroughSequence() == last);
|
||||
jdbc.update("UPDATE ma_snapshot SET retain_until = 0 WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " AND snapshot_id <> (SELECT current_snapshot_id FROM ma_event_stream WHERE tenant_id = ? AND session_id = ?"
|
||||
+ " AND stream_kind = 'SESSION' AND stream_id = X'')", tenant, sessionId, tenant, sessionId);
|
||||
await().atMost(Duration.ofSeconds(10)).until(() ->
|
||||
store.findReplayWindow(tenant, sessionId)
|
||||
.floorSequence() == last);
|
||||
@@ -98,7 +86,7 @@ class ReplayFloorAdvancerTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void enabledPassAdvancesEveryTargetAsFarAsTheSnapshotProves() {
|
||||
void enabledPassAdvancesEveryTargetThroughMessageCoverage() {
|
||||
AgentStateStore mockStore = mock(AgentStateStore.class);
|
||||
when(mockStore.findReplayFloorTargets(anyInt())).thenReturn(List.of(
|
||||
new ReplayFloorTarget("tenant-a", "session-a"),
|
||||
|
||||
+62
-16
@@ -1,5 +1,6 @@
|
||||
package com.alibaba.qwen.code.managedagent.session;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
|
||||
import com.alibaba.qwen.code.managedagent.model.ApiModels.SessionResyncRequired;
|
||||
import com.alibaba.qwen.code.managedagent.persistence.ManagedAgentStore;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
@@ -21,6 +22,7 @@ import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
@SpringBootTest(
|
||||
properties = {
|
||||
@@ -43,7 +45,7 @@ class ReplayFloorRetractionTest {
|
||||
@Autowired private ObjectMapper mapper;
|
||||
|
||||
@Test
|
||||
void retractionPreservesSnapshotCursorsAndStreamBoundaries() throws Exception {
|
||||
void retractionInvalidatesMessageCursorsAndPreservesStreamBoundaries() throws Exception {
|
||||
String tenant = "retraction-" + UUID.randomUUID();
|
||||
String session = session(tenant);
|
||||
var turn =
|
||||
@@ -69,7 +71,7 @@ class ReplayFloorRetractionTest {
|
||||
"boot:epoch:" + i);
|
||||
}
|
||||
store.materializeNextBatch(tenant, session, 100);
|
||||
var page = store.findSnapshotPage(tenant, session, null, 0, 1);
|
||||
var page = store.findMessagePage(tenant, session, null, 1, false);
|
||||
assertThat(page.nextCursor()).isNotNull();
|
||||
long covered = page.coveredSequence();
|
||||
long start =
|
||||
@@ -101,13 +103,12 @@ class ReplayFloorRetractionTest {
|
||||
tenant,
|
||||
session))
|
||||
.isEqualTo(projection + 1);
|
||||
var old = store.findSnapshotPage(tenant, session, page.nextCursor(), 0, 10);
|
||||
assertThat(old.items())
|
||||
.extracting(item -> item.content().getFirst().text())
|
||||
.containsExactly("text-0", "text-1");
|
||||
assertThat(old.coveredSequence()).isEqualTo(covered);
|
||||
assertThatThrownBy(() -> store.findMessagePage(tenant, session, page.nextCursor(), 10, false))
|
||||
.isInstanceOfSatisfying(ManagedAgentException.class,
|
||||
error -> assertThat(error.getCode()).isEqualTo("message_projection_changed"));
|
||||
store.materializeNextBatch(tenant, session, 100);
|
||||
assertThat(
|
||||
store.findSnapshot(tenant, session).orElseThrow().items().stream()
|
||||
store.findMessagePage(tenant, session, null, 100, false).items().stream()
|
||||
.flatMap(item -> item.content().stream())
|
||||
.map(part -> part.text())
|
||||
.toList())
|
||||
@@ -171,7 +172,7 @@ class ReplayFloorRetractionTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void itemsAfterWithdrawnSequenceIncludesReissuedMessageAndKeepsOldCursor() throws Exception {
|
||||
void itemsAfterWithdrawnSequenceIncludesReissuedMessageAndRejectsOldCursor() throws Exception {
|
||||
String tenant = "reissued-items-" + UUID.randomUUID();
|
||||
String session = session(tenant);
|
||||
String turn =
|
||||
@@ -194,14 +195,15 @@ class ReplayFloorRetractionTest {
|
||||
false,
|
||||
"boot:old:1");
|
||||
store.materializeNextBatch(tenant, session, 100);
|
||||
var frozen = store.findSnapshotPage(tenant, session, null, 0, 1);
|
||||
var frozen = store.findMessagePage(tenant, session, null, 1, false);
|
||||
long oldSequence =
|
||||
store.findSnapshot(tenant, session).orElseThrow().items().stream()
|
||||
store.findMessagePage(tenant, session, null, 100, false).items().stream()
|
||||
.filter(item -> "withdrawn".equals(item.itemId()))
|
||||
.findFirst()
|
||||
.orElseThrow()
|
||||
.firstSequence();
|
||||
store.retractContinuationOutput(tenant, session, turn, "owner", "boot", "old");
|
||||
store.materializeNextBatch(tenant, session, 100);
|
||||
var empty =
|
||||
mvc.perform(
|
||||
org.springframework.test.web.servlet.request.MockMvcRequestBuilders
|
||||
@@ -253,11 +255,55 @@ class ReplayFloorRetractionTest {
|
||||
.param("after", frozen.nextCursor()))
|
||||
.andReturn()
|
||||
.getResponse();
|
||||
assertThat(old.getStatus()).isEqualTo(200);
|
||||
JsonNode retained = EventStreams.json(mapper, old).get("data");
|
||||
assertThat(retained).hasSize(1);
|
||||
assertThat(retained.get(0).get("id").asText()).isEqualTo("withdrawn");
|
||||
assertThat(retained.get(0).get("content").get(0).get("text").asText()).isEqualTo("old");
|
||||
assertThat(old.getStatus()).isEqualTo(409);
|
||||
assertThat(EventStreams.json(mapper, old).path("error").path("code").asText())
|
||||
.isEqualTo("message_projection_changed");
|
||||
}
|
||||
|
||||
@Test
|
||||
void floorAdvanceAfterMessageReadResynchronizesThroughTheLiveTranscript() throws Exception {
|
||||
String tenant = "message-replay-race-" + UUID.randomUUID();
|
||||
String session = session(tenant);
|
||||
for (int index = 0; index < 3; index++) {
|
||||
store.appendPublicEventIfAbsent(tenant, session, "turn",
|
||||
"item.output_text.delta",
|
||||
Map.of("itemId", "message-" + index, "text", "text-" + index),
|
||||
false, "race-" + index);
|
||||
}
|
||||
store.materializeNextBatch(tenant, session, 100);
|
||||
var initial = service.transcript(tenant, null, session, null, 2);
|
||||
assertThat(initial.items()).hasSize(2);
|
||||
assertThat(initial.hasMore()).isTrue();
|
||||
store.appendPublicEventIfAbsent(tenant, session, "turn",
|
||||
"item.output_text.delta", Map.of("itemId", "latest", "text", "latest"),
|
||||
false, "race-latest");
|
||||
store.materializeNextBatch(tenant, session, 100);
|
||||
var window = store.advanceReplayFloor(tenant, session, Long.MAX_VALUE);
|
||||
assertThat(window.floorSequence()).isGreaterThan(initial.lastSequence());
|
||||
try (var executor = Executors.newCachedThreadPool()) {
|
||||
var expired = new EventStreams.RecordingEmitter();
|
||||
EventStreams.streams(service, hub, executor, expired)
|
||||
.publicStream(tenant, null, session, initial.lastSequence());
|
||||
assertThat(expired.completed.await(10, TimeUnit.SECONDS)).isTrue();
|
||||
assertThat(expired.failed).isEmpty();
|
||||
assertThat(expired.resync).containsExactly(new SessionResyncRequired(
|
||||
ManagedEventStreamService.RESYNC, session, window.floorSequence(),
|
||||
window.snapshotThroughSequence(), ManagedEventStreamService.RESYNC_ACTION));
|
||||
}
|
||||
var reloaded = service.transcript(tenant, null, session, null, 2);
|
||||
assertThat(reloaded.lastSequence()).isEqualTo(window.snapshotThroughSequence());
|
||||
assertThat(reloaded.coveredSequence()).isEqualTo(reloaded.lastSequence());
|
||||
assertThat(reloaded.items()).extracting(item -> item.itemId()).contains("latest");
|
||||
store.appendPublicEventIfAbsent(tenant, session, "turn",
|
||||
"item.output_text.delta", Map.of("itemId", "newer", "text", "newer"),
|
||||
false, "race-newer");
|
||||
store.materializeNextBatch(tenant, session, 100);
|
||||
long currentCoverage = store.findMessageCoveredSequences(tenant, List.of(session)).get(session);
|
||||
var older = service.transcript(tenant, null, session, reloaded.olderCursor(), 2);
|
||||
assertThat(currentCoverage).isGreaterThan(reloaded.coveredSequence());
|
||||
assertThat(older.coveredSequence()).isEqualTo(currentCoverage);
|
||||
assertThat(older.lastSequence()).isEqualTo(currentCoverage);
|
||||
assertThat(older.items()).extracting(item -> item.itemId()).doesNotContain("newer");
|
||||
}
|
||||
|
||||
private String session(String tenant) {
|
||||
|
||||
+18
-6
@@ -260,28 +260,40 @@ class ManagedSchemaMySqlIT {
|
||||
"ALTER TABLE ma_operation ALTER CHECK ck_ma_operation_kind ENFORCED");
|
||||
}
|
||||
}
|
||||
String pageCheck =
|
||||
String operationCheck = ManagedSchema.ddl().lines()
|
||||
.filter(line -> line.contains("CONSTRAINT ck_ma_operation_kind CHECK"))
|
||||
.findFirst().orElseThrow().trim().replaceFirst(",$", "");
|
||||
statement.execute("ALTER TABLE ma_operation " + dropCheck + "ck_ma_operation_kind");
|
||||
statement.execute("ALTER TABLE ma_operation ADD "
|
||||
+ operationCheck.replace("'CLOSE'", "'BROKE'"));
|
||||
try {
|
||||
assertThrows(IllegalStateException.class, () -> ManagedSchema.initialize(source));
|
||||
} finally {
|
||||
statement.execute("ALTER TABLE ma_operation " + dropCheck + "ck_ma_operation_kind");
|
||||
statement.execute("ALTER TABLE ma_operation ADD " + operationCheck);
|
||||
}
|
||||
String resourceCheck =
|
||||
ManagedSchema.ddl()
|
||||
.lines()
|
||||
.filter(
|
||||
line ->
|
||||
line.contains(
|
||||
"CONSTRAINT ck_ma_resource_snapshot_page"
|
||||
"CONSTRAINT ck_ma_resource_length"
|
||||
+ " CHECK"))
|
||||
.findFirst()
|
||||
.orElseThrow()
|
||||
.trim()
|
||||
.replaceFirst(",$", "");
|
||||
statement.execute("ALTER TABLE ma_resource " + dropCheck + "ck_ma_resource_snapshot_page");
|
||||
statement.execute("ALTER TABLE ma_resource " + dropCheck + "ck_ma_resource_length");
|
||||
statement.execute(
|
||||
"ALTER TABLE ma_resource ADD "
|
||||
+ pageCheck.replace("'MYSQL_INLINE'", "'MYSQL_BROKEN'"));
|
||||
+ resourceCheck.replace("byte_length >= 0", "byte_length >= -1"));
|
||||
try {
|
||||
assertThrows(IllegalStateException.class, () -> ManagedSchema.initialize(source));
|
||||
} finally {
|
||||
statement.execute(
|
||||
"ALTER TABLE ma_resource " + dropCheck + "ck_ma_resource_snapshot_page");
|
||||
statement.execute("ALTER TABLE ma_resource ADD " + pageCheck);
|
||||
"ALTER TABLE ma_resource " + dropCheck + "ck_ma_resource_length");
|
||||
statement.execute("ALTER TABLE ma_resource ADD " + resourceCheck);
|
||||
}
|
||||
statement.execute("ALTER TABLE ma_session_head " + dropCheck + "ck_ma_head_counters");
|
||||
statement.execute(
|
||||
|
||||
+22
-95
@@ -122,6 +122,28 @@ class ManagedSchemaTest {
|
||||
ManagedSchema.validate(source);
|
||||
}
|
||||
|
||||
@Test
|
||||
void reportsMissingAndChangedProjectionExpressionsAsSchemaErrors() throws Exception {
|
||||
DataSource source = source();
|
||||
try (var connection = source.getConnection();
|
||||
var statement = connection.createStatement()) {
|
||||
int variant = 0;
|
||||
for (String declaration : Set.of("BOOLEAN",
|
||||
"BOOLEAN GENERATED ALWAYS AS (last_sequence >= covered_sequence)")) {
|
||||
String table = "ma_generated_drift_" + variant++;
|
||||
statement.execute("CREATE TABLE " + table
|
||||
+ " (last_sequence BIGINT, covered_sequence BIGINT, projection_pending "
|
||||
+ declaration + ")");
|
||||
var error = assertThrows(SQLException.class,
|
||||
() -> ManagedSchemaCatalog.validateGeneratedColumns(connection, table,
|
||||
" projection_pending BOOLEAN GENERATED ALWAYS AS"
|
||||
+ " (last_sequence > covered_sequence)"));
|
||||
assertTrue(error.getMessage().contains(
|
||||
"Managed schema generated column differs: " + table + ".projection_pending"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void rejectsForeignLegacyAndPartialSchemasBeforeCreatingAnything() throws Exception {
|
||||
for (String table : Set.of("managed_agent_session", "business_data", "ma_session")) {
|
||||
@@ -320,70 +342,6 @@ class ManagedSchemaTest {
|
||||
ManagedSchema.validate(source);
|
||||
}
|
||||
|
||||
@Test
|
||||
void snapshotPageDirectoryRequiresCompleteTypedMetadataAndUniqueOrdinals() throws Exception {
|
||||
DataSource source = initialized();
|
||||
try (var connection = source.getConnection()) {
|
||||
snapshotPage(connection, "first", "managed-snapshot-page", "snapshot", 0, 1L, 100L);
|
||||
assertThrows(
|
||||
SQLException.class,
|
||||
() ->
|
||||
snapshotPage(
|
||||
connection,
|
||||
"duplicate",
|
||||
"managed-snapshot-page",
|
||||
"snapshot",
|
||||
0,
|
||||
1L,
|
||||
100L));
|
||||
assertThrows(
|
||||
SQLException.class,
|
||||
() ->
|
||||
snapshotPage(
|
||||
connection,
|
||||
"incomplete",
|
||||
"managed-snapshot-page",
|
||||
"snapshot",
|
||||
null,
|
||||
1L,
|
||||
100L));
|
||||
assertThrows(
|
||||
SQLException.class,
|
||||
() ->
|
||||
snapshotPage(
|
||||
connection,
|
||||
"missing",
|
||||
"managed-snapshot-page",
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null));
|
||||
assertThrows(
|
||||
SQLException.class,
|
||||
() ->
|
||||
snapshotPage(
|
||||
connection,
|
||||
"backward",
|
||||
"managed-snapshot-page",
|
||||
"snapshot",
|
||||
1,
|
||||
101L,
|
||||
100L));
|
||||
assertThrows(
|
||||
SQLException.class,
|
||||
() ->
|
||||
snapshotPage(
|
||||
connection,
|
||||
"wrong-kind",
|
||||
"managed-context",
|
||||
"snapshot",
|
||||
1,
|
||||
101L,
|
||||
200L));
|
||||
}
|
||||
ManagedSchema.validate(source);
|
||||
}
|
||||
|
||||
@Test
|
||||
void harnessBarrierHasNoInventedJournalAndStorageFenceHasNoInventedPhysicalOwner()
|
||||
throws Exception {
|
||||
@@ -598,37 +556,6 @@ class ManagedSchemaTest {
|
||||
assertThrows(IllegalArgumentException.class, ManagedSchemaKeys::key);
|
||||
}
|
||||
|
||||
private static void snapshotPage(
|
||||
Connection connection,
|
||||
String key,
|
||||
String kind,
|
||||
String snapshot,
|
||||
Integer ordinal,
|
||||
Long first,
|
||||
Long last)
|
||||
throws SQLException {
|
||||
try (var statement =
|
||||
connection.prepareStatement(
|
||||
"INSERT INTO ma_resource"
|
||||
+ " (resource_key,session_scope_key,tenant_id,session_id,resource_id,kind,schema_version,byte_length,"
|
||||
+ "sha256,storage_kind,inline_bytes,state,gmt_create,snapshot_id,snapshot_page_ordinal,snapshot_page_first_sequence,snapshot_page_last_sequence)"
|
||||
+ " VALUES"
|
||||
+ " (?,'scope',?,?,?,?,1,0,?,'MYSQL_INLINE',?,'REFERENCED'," + ManagedSchemaTime.SQL_NOW + ",?,?,?,?)")) {
|
||||
statement.setString(1, key);
|
||||
statement.setBytes(2, ManagedSchemaKeys.identity("tenant"));
|
||||
statement.setBytes(3, ManagedSchemaKeys.identity("session"));
|
||||
statement.setBytes(4, ManagedSchemaKeys.identity(key));
|
||||
statement.setBytes(5, ManagedSchemaKeys.identity(kind));
|
||||
statement.setString(6, "a".repeat(64));
|
||||
statement.setBytes(7, new byte[0]);
|
||||
statement.setBytes(8, snapshot == null ? null : ManagedSchemaKeys.identity(snapshot));
|
||||
statement.setObject(9, ordinal);
|
||||
statement.setObject(10, first);
|
||||
statement.setObject(11, last);
|
||||
statement.executeUpdate();
|
||||
}
|
||||
}
|
||||
|
||||
private static void session(Connection connection, String tenant, String session, String kind)
|
||||
throws SQLException {
|
||||
try (var statement =
|
||||
|
||||
+3
-4
@@ -25,10 +25,9 @@ class ManagedSchemaTimeContractTest {
|
||||
"ma_session_head=writer_lease_until,gmt_create,gmt_modified,"
|
||||
+ "activation_expires_at,claim_lease_until,wake_retry_at",
|
||||
"ma_session_journal_tx=gmt_create",
|
||||
"ma_event_stream=snapshot_stale_since,gmt_modified",
|
||||
"ma_event_stream=gmt_modified",
|
||||
"ma_public_event=gmt_create",
|
||||
"ma_message=gmt_create,gmt_modified",
|
||||
"ma_snapshot=gmt_create,published_at,retain_until",
|
||||
"ma_session_record=gmt_create,started_at,settled_at",
|
||||
"ma_agent_definition=gmt_create",
|
||||
"ma_resource=gmt_create,last_verified_at,retention_until",
|
||||
@@ -57,7 +56,7 @@ class ManagedSchemaTimeContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void all79PersistedTimestampsDeclareUnsignedIntegerMilliseconds() {
|
||||
void all75PersistedTimestampsDeclareUnsignedIntegerMilliseconds() {
|
||||
Map<String, String> declarations = declarations();
|
||||
List<Executable> checks = new ArrayList<>();
|
||||
int count = 0;
|
||||
@@ -74,7 +73,7 @@ class ManagedSchemaTimeContractTest {
|
||||
});
|
||||
}
|
||||
}
|
||||
assertEquals(79, count);
|
||||
assertEquals(75, count);
|
||||
assertAll(checks);
|
||||
}
|
||||
|
||||
|
||||
+1
-1
@@ -10,7 +10,7 @@ managed_workspace_create_command ma_request_receipt,ma_session
|
||||
managed_agent_event ma_public_event
|
||||
managed_agent_item ma_message
|
||||
managed_agent_item_part ma_message
|
||||
managed_agent_snapshot ma_snapshot
|
||||
managed_agent_snapshot ma_message,ma_event_stream
|
||||
managed_agent_consumer_progress ma_event_stream
|
||||
qwen_managed_session_journal_head ma_session_head
|
||||
qwen_managed_session_journal_tx ma_session_journal_tx
|
||||
|
||||
|
@@ -59,7 +59,7 @@ export interface paths {
|
||||
};
|
||||
get?: never;
|
||||
put?: never;
|
||||
/** @description Default view: Without a cursor, a Session that has a Snapshot returns all of its Items, the events up to the Snapshot other than turn.accepted, item.output_text.delta, item.reasoning.delta, item.tool_call.updated and item.tool_result.updated, which the Items already hold, and every event after it. Otherwise, and for an olderCursor, limit bounds the page of events. Explicit view=messages instead returns at most100 current ma_message Items using tenant/Session keyset pagination, independent of published snapshots. Bodies are live per page. coveredSequence and lastSequence are the projection sequence included by this read, so the client resumes SSE there and does not skip an unprojected tail. Only bounded lifecycle events for the selected Turns accompany the Items. An older cursor pins direction, scope, sequence membership and projection generation;409 message_projection_changed requires reloading history. */
|
||||
/** @description Requires current Session read authorization. Read the most recent limit current message Items and selected bounded Turn control events; olderCursor pages toward older Items. Bodies and statuses remain live. The response coveredSequence and lastSequence both identify actual message projection coverage for SSE continuation, never an unmaterialized event tail. No persisted conversation snapshot or whole-history replay is read. Projection rebuild invalidates cursors. If projection coverage is below the replay floor, return retryable message_projection_pending. */
|
||||
post: operations["webShellTranscript"];
|
||||
delete?: never;
|
||||
options?: never;
|
||||
@@ -76,7 +76,7 @@ export interface paths {
|
||||
};
|
||||
get?: never;
|
||||
put?: never;
|
||||
/** @description POST SSE consumed through fetch response.body. An afterSequence strictly below the durable replay floor ends the stream with one agent.session.resync_required frame. Projection rebuilds preserve that floor and the current immutable Snapshot. On the resync frame, reload the transcript and resume after its lastSequence. */
|
||||
/** @description POST SSE consumed through fetch response.body. An afterSequence strictly below the durable replay floor ends the stream with one agent.session.resync_required frame. Projection rebuilds preserve that floor and invalidate message-page cursors. On the resync frame, reload the transcript and resume after its lastSequence. */
|
||||
post: operations["webShellStreamEvents"];
|
||||
delete?: never;
|
||||
options?: never;
|
||||
@@ -805,8 +805,8 @@ export interface paths {
|
||||
get?: never;
|
||||
put?: never;
|
||||
/**
|
||||
* Read one immutable public Session snapshot page for reading exports
|
||||
* @description Default view: Requires current Session read authorization on every page. Reuses the public item snapshot cursor, whose subsequent pages retain the original snapshot version and watermark. No private journal, Runtime or storage credentials are returned. The client waits for a snapshot covering its captured Session watermark before using its first page. Explicit view=messages instead reads live message-table pages forward. The first read captures a finite membership bound; later body/status changes are allowed and no persistent snapshot is created or selected. snapshotThroughSequence then names that membership bound, not frozen content. If the initial message projection lags committed events,503 message_projection_pending asks the caller to retry instead of returning incomplete history. Projection-generation changes invalidate the cursor.
|
||||
* Read one current Session message page for exports
|
||||
* @description Requires current Session read authorization on every page. Read current message bodies forward within the first request sequence membership bound. snapshotThroughSequence retains its wire name for that fixed bound and does not change between pages. Content and status may change; no frozen snapshot version or cursor retention is provided. An initial projection behind committed events returns retryable message_projection_pending instead of silently exporting incomplete history. No private journal, Runtime or storage credentials are returned. A projection-generation change returns 409 message_projection_changed; discard the cursor and restart this read.
|
||||
*/
|
||||
post: operations["webShellExportSession"];
|
||||
delete?: never;
|
||||
@@ -1137,8 +1137,8 @@ export interface paths {
|
||||
get?: never;
|
||||
put?: never;
|
||||
/**
|
||||
* Read recent published Session items and page toward older items
|
||||
* @description Default view: Requires current Session read authorization on every page. The first page returns at most limit Items from the newest published snapshot, in chronological order. nextCursor pages toward older Items in that same signed, retained snapshot. snapshotThroughSequence identifies its coverage, which may lag the current Session head; an empty page with zero coverage does not certify an empty Session because no published snapshot may be available. This page does not contain live tail events or current approval/task state; use the existing transcript, event and state APIs for those. A newer snapshot does not change an existing cursor. Forward export cursors cannot be used here and backward cursors cannot be used for exports. No journal replay, writer acquisition or local cache is involved. Explicit view=messages instead reads current message-table Items backward without any published snapshot, using the same scoped keyset and projection-generation rules as messages-view transcript reads. snapshotThroughSequence then names the initial sequence membership bound; bodies can change between pages. This mode does not return lifecycle events or claim fixed snapshot content.
|
||||
* Read recent current Session Items and page toward older Items
|
||||
* @description Requires current Session read authorization on every page. Read at most limit recent current message Items in chronological order; nextCursor continues toward older Items in the initial sequence membership range. snapshotThroughSequence names that fixed membership upper bound, while bodies and statuses remain live. Cursors bind scope, backward direction and projection generation. Rebuild invalidates the cursor. No snapshot copies are generated and no live tail is scanned; transcript reads provide actual projection coverage for SSE handoff. Projection coverage below the replay floor returns retryable message_projection_pending.
|
||||
*/
|
||||
post: operations["webShellRecentTranscriptItems"];
|
||||
delete?: never;
|
||||
@@ -1244,11 +1244,6 @@ export interface components {
|
||||
cursor?: string | null;
|
||||
/** @default 100 */
|
||||
limit?: number | null;
|
||||
/**
|
||||
* @description Select direct live message-table pages. Omit to retain the existing snapshot/event view. Message view accepts limit1..100; cursors bind scope, direction, membership range and projection generation, not frozen bodies.
|
||||
* @enum {string|null}
|
||||
*/
|
||||
view?: "messages" | null;
|
||||
};
|
||||
WebShellStreamRequest: {
|
||||
/** Format: uuid */
|
||||
@@ -1459,11 +1454,17 @@ export interface components {
|
||||
WebShellTranscript: {
|
||||
items: components["schemas"]["WebShellItem"][];
|
||||
events: components["schemas"]["WebShellEvent"][];
|
||||
/** Format: int64 */
|
||||
/**
|
||||
* Format: int64
|
||||
* @description Actual message projection coverage of this read; resume SSE after this sequence.
|
||||
*/
|
||||
coveredSequence: number;
|
||||
olderCursor?: string | null;
|
||||
hasMore: boolean;
|
||||
/** Format: int64 */
|
||||
/**
|
||||
* Format: int64
|
||||
* @description Actual message projection coverage for SSE handoff, equal to coveredSequence; unmaterialized tail events are delivered by SSE.
|
||||
*/
|
||||
lastSequence: number;
|
||||
};
|
||||
WebShellContentPart: {
|
||||
@@ -1526,7 +1527,10 @@ export interface components {
|
||||
sessionId: string;
|
||||
/** Format: int64 */
|
||||
replayFloorSequence: number;
|
||||
/** Format: int64 */
|
||||
/**
|
||||
* Format: int64
|
||||
* @description Retained wire name for current message projection coverage; no persisted conversation snapshot is implied.
|
||||
*/
|
||||
snapshotThroughSequence: number;
|
||||
/** @constant */
|
||||
action: "reload_snapshot";
|
||||
@@ -2414,7 +2418,7 @@ export interface components {
|
||||
hasMore: boolean;
|
||||
/**
|
||||
* Format: int64
|
||||
* @description In explicit messages view, this is the fixed sequence membership upper bound; row bodies are live, not frozen snapshot content.
|
||||
* @description Fixed first-request sequence membership upper bound. Bodies and statuses are read live on every page; this does not identify a retained content version.
|
||||
*/
|
||||
snapshotThroughSequence: number;
|
||||
};
|
||||
@@ -2424,11 +2428,6 @@ export interface components {
|
||||
cursor?: string | null;
|
||||
/** @default 100 */
|
||||
limit?: number | null;
|
||||
/**
|
||||
* @description Select direct live message-table pages. Omit to retain the existing snapshot/event view. Message view accepts limit1..100; cursors bind scope, direction, membership range and projection generation, not frozen bodies.
|
||||
* @enum {string|null}
|
||||
*/
|
||||
view?: "messages" | null;
|
||||
};
|
||||
/** @description Durable Turn input or cancellation admission. The operation identifies the saved command and remains stable across idempotent retries. */
|
||||
WebShellTurnAdmission: {
|
||||
@@ -2736,15 +2735,6 @@ export interface components {
|
||||
"application/json": components["schemas"]["ErrorEnvelope"];
|
||||
};
|
||||
};
|
||||
/** @description Replay cursor is older than the retained replay floor. */
|
||||
CursorExpired: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["ErrorEnvelope"];
|
||||
};
|
||||
};
|
||||
/** @description Task event cursor is strictly below the task's durable retention floor, which survives an empty retained set; equality is valid. The envelope's replay_floor_sequence and snapshot_through_sequence stay absent because task cursors are opaque. */
|
||||
TaskCursorExpired: {
|
||||
headers: {
|
||||
@@ -2763,6 +2753,42 @@ export interface components {
|
||||
"application/json": components["schemas"]["ErrorEnvelope"];
|
||||
};
|
||||
};
|
||||
/** @description 409 message_projection_changed: the message projection generation changed. Discard the message cursor and restart this read; this is not replay-floor cursor expiration. */
|
||||
MessageProjectionChanged: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["ErrorEnvelope"];
|
||||
};
|
||||
};
|
||||
/** @description 413 message_page_too_large: selected message bodies or Turn control events exceed the response byte budget. Reducing limit can help a multi-Item page, but a single oversized Item cannot be returned even with limit=1. Signed POST requests may also return payload_too_large when their request body exceeds the configured signed-body limit. */
|
||||
MessagePageTooLarge: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["ErrorEnvelope"];
|
||||
};
|
||||
};
|
||||
/** @description 400 invalid_message_cursor for malformed, wrong-scope, wrong-direction or out-of-range message cursors; invalid_limit outside 1..100; invalid_event_cursor for a negative numeric after. */
|
||||
InvalidMessagePage: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["ErrorEnvelope"];
|
||||
};
|
||||
};
|
||||
/** @description 503 message_projection_pending: projection coverage is below the replay floor, or the initial forward read is behind committed events. Retry after message projection catches up. */
|
||||
MessageProjectionPending: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["ErrorEnvelope"];
|
||||
};
|
||||
};
|
||||
};
|
||||
parameters: {
|
||||
IdempotencyKey: string;
|
||||
@@ -2890,21 +2916,13 @@ export interface operations {
|
||||
"application/json": components["schemas"]["WebShellTranscript"];
|
||||
};
|
||||
};
|
||||
400: components["responses"]["BadRequest"];
|
||||
400: components["responses"]["InvalidMessagePage"];
|
||||
401: components["responses"]["Unauthorized"];
|
||||
403: components["responses"]["Forbidden"];
|
||||
404: components["responses"]["NotFound"];
|
||||
409: components["responses"]["Conflict"];
|
||||
413: components["responses"]["PayloadTooLarge"];
|
||||
/** @description Message projection is not ready for a complete stream handoff; retry the read. */
|
||||
503: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["ErrorEnvelope"];
|
||||
};
|
||||
};
|
||||
409: components["responses"]["MessageProjectionChanged"];
|
||||
413: components["responses"]["MessagePageTooLarge"];
|
||||
503: components["responses"]["MessageProjectionPending"];
|
||||
};
|
||||
};
|
||||
webShellStreamEvents: {
|
||||
@@ -4386,7 +4404,7 @@ export interface operations {
|
||||
};
|
||||
};
|
||||
responses: {
|
||||
/** @description Chronological Item page: published snapshot by default, or live messages when explicitly selected. */
|
||||
/** @description Chronological page of current message Items within the initial sequence membership range; content remains live. */
|
||||
200: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
@@ -4395,21 +4413,13 @@ export interface operations {
|
||||
"application/json": components["schemas"]["WebShellExportPage"];
|
||||
};
|
||||
};
|
||||
400: components["responses"]["BadRequest"];
|
||||
400: components["responses"]["InvalidMessagePage"];
|
||||
401: components["responses"]["Unauthorized"];
|
||||
403: components["responses"]["Forbidden"];
|
||||
404: components["responses"]["NotFound"];
|
||||
409: components["responses"]["CursorExpired"];
|
||||
413: components["responses"]["PayloadTooLarge"];
|
||||
/** @description Message projection has not caught up; retry the export. */
|
||||
503: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["ErrorEnvelope"];
|
||||
};
|
||||
};
|
||||
409: components["responses"]["MessageProjectionChanged"];
|
||||
413: components["responses"]["MessagePageTooLarge"];
|
||||
503: components["responses"]["MessageProjectionPending"];
|
||||
};
|
||||
};
|
||||
webShellQueryUserMessages: {
|
||||
@@ -5032,7 +5042,7 @@ export interface operations {
|
||||
};
|
||||
};
|
||||
responses: {
|
||||
/** @description Bounded recent published Item page. Items are chronological within each page. */
|
||||
/** @description Bounded recent current message Item page. Items are chronological within each page. */
|
||||
200: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
@@ -5041,21 +5051,13 @@ export interface operations {
|
||||
"application/json": components["schemas"]["WebShellExportPage"];
|
||||
};
|
||||
};
|
||||
400: components["responses"]["BadRequest"];
|
||||
400: components["responses"]["InvalidMessagePage"];
|
||||
401: components["responses"]["Unauthorized"];
|
||||
403: components["responses"]["Forbidden"];
|
||||
404: components["responses"]["NotFound"];
|
||||
409: components["responses"]["CursorExpired"];
|
||||
413: components["responses"]["PayloadTooLarge"];
|
||||
/** @description Message projection is not ready for a complete stream handoff; retry the read. */
|
||||
503: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["ErrorEnvelope"];
|
||||
};
|
||||
};
|
||||
409: components["responses"]["MessageProjectionChanged"];
|
||||
413: components["responses"]["MessagePageTooLarge"];
|
||||
503: components["responses"]["MessageProjectionPending"];
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -1432,7 +1432,6 @@ describe('createJavaManagedAgentProvider', () => {
|
||||
sessionId: 'session-1',
|
||||
cursor: 'before-1',
|
||||
limit: 25,
|
||||
view: 'messages',
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
@@ -210,7 +210,7 @@ export function createJavaManagedAgentProvider(
|
||||
},
|
||||
exportSnapshot: (sessionId, request) =>
|
||||
client.exportSnapshot(
|
||||
{ sessionId, cursor: request.cursor, limit: 100, view: 'messages' },
|
||||
{ sessionId, cursor: request.cursor, limit: 100 },
|
||||
request.signal,
|
||||
),
|
||||
exportConsistency: 'live',
|
||||
@@ -877,7 +877,6 @@ export function createJavaManagedAgentProvider(
|
||||
sessionId,
|
||||
cursor: request.before,
|
||||
limit: request.limit,
|
||||
view: 'messages',
|
||||
},
|
||||
request.signal,
|
||||
);
|
||||
|
||||
@@ -2,6 +2,7 @@ import { expect, test, type Page } from '@playwright/test';
|
||||
import type {
|
||||
JavaAgentEvent,
|
||||
JavaAgentSession,
|
||||
JavaAgentTranscript,
|
||||
} from '../components/managed/java-managed-agent-client';
|
||||
|
||||
async function installRecovery(page: Page, running = false) {
|
||||
@@ -94,6 +95,7 @@ async function installRecovery(page: Page, running = false) {
|
||||
sessionId: 'source',
|
||||
eventId: `event-${event.sequence}`,
|
||||
createdAt: event.sequence,
|
||||
terminal: event.type === 'turn.completed' || event.type === 'turn.failed',
|
||||
}));
|
||||
page.on('pageerror', (error) => errors.push(error.message));
|
||||
await page.route('**/api/agent/web-shell/v1/**', async (route) => {
|
||||
@@ -145,7 +147,13 @@ async function installRecovery(page: Page, running = false) {
|
||||
});
|
||||
if (path.endsWith('/transcript/query'))
|
||||
return route.fulfill({
|
||||
json: { items: [], events, lastSequence: 6, hasMore: false },
|
||||
json: {
|
||||
items: [],
|
||||
events,
|
||||
coveredSequence: 6,
|
||||
lastSequence: 6,
|
||||
hasMore: false,
|
||||
} satisfies JavaAgentTranscript,
|
||||
});
|
||||
if (path.endsWith('/events/stream'))
|
||||
return route.fulfill({ contentType: 'text/event-stream', body: '' });
|
||||
|
||||
Reference in new issue
Block a user