mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
feat(runtime): add resident workspace reclamation and reuse (#15729)
* feat(runtime): add resident workspace reclamation and reuse Add administrator-driven cleanup receipts, atomic resident release, and explicit same-tenant handoff with retained files. Preserve uncertain outcomes, prior operation history, and exact ownership revisions across reuse. Co-authored-by: Qwen-Coder <[email protected]> * test(runtime): align retirement alias fixture with tenant ownership Admit a same-tenant contender before sealing the original binding so the busy-volume assertion reaches reservation. Assert foreign-tenant registration is refused while an unrelated volume remains reservable. Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]>
This commit is contained in:
jinyeandQwen-Coder authored and GitHub committed
1 parent
0b3331b633
commit
414efe80a7
32 files changed
+1775
-64
No files matched your search
@@ -15,7 +15,7 @@ proposal #14761. Optional explicit idle retirement is implemented in #14765 and
|
||||
described below. This configuration needs no Go toolchain, custom containerd/shim,
|
||||
node collector or cold-bootstrap installation.
|
||||
|
||||
The [2026-10-24 scope decision](../../docs/design/2026-10-23-csi-evidence-source.md) confirms that complete CSI auditing is not a first-release prerequisite. Session handoff retains the original Pod/PVC; explicit idle retirement retains storage ownership. Platform-managed reclamation and cross-Workspace volume reuse remain follow-up work.
|
||||
The [2026-10-24 scope decision](../../docs/design/2026-10-23-csi-evidence-source.md) confirms that complete CSI auditing is not a first-release prerequisite. Session handoff retains the original Pod/PVC; deletion-only idle retirement retains storage ownership. The explicit S2/S3 reclamation and retained-data handoff implementation below requires separate native qualification.
|
||||
|
||||
## Trusted platform and supported storage
|
||||
|
||||
@@ -131,7 +131,7 @@ Background propagation. Credentials remain in the existing API/token/CA settings
|
||||
No PVC/PV, Secret or protection object is deleted. Same-version retries read the
|
||||
saved result without resending; a new version needs an explicit new invocation,
|
||||
with at most eight attempts in total. HTTP replies and Pod absence never release
|
||||
the storage or reopen the Workspace. Physical release and reuse are deferred.
|
||||
the storage or reopen the Workspace. Reclamation requires the preparation and completion protocol below.
|
||||
|
||||
A lost reply can leave `UNKNOWN` even when Kubernetes accepted the deletion.
|
||||
Once the original Pod has a `deletionTimestamp` or is missing, a different
|
||||
@@ -144,6 +144,22 @@ See the complete retirement design in [English](../../docs/design/2026-10-20-k2-
|
||||
and [简体中文](../../docs/design/2026-10-20-k2-physical-retirement-progress.zh-CN.md).
|
||||
Native ACK retirement qualification remains pending.
|
||||
|
||||
## Explicit reclamation and same-tenant data handoff
|
||||
|
||||
The [S2/S3 design](../../docs/design/2026-10-27-resident-workspace-reclamation.md) adds a healthy normal cleanup path. Native ACK qualification is pending; do not infer it from the existence of these commands or from the earlier S1 run. Keep the current single-control-plane trust boundary and use a dedicated runtime namespace/PVC. Admission/RBAC must leave only this Managed provisioner able to create Pods using the PVC, with no external controller, Pod creator, PV alias, or administrative storage operation during maintenance and handoff. The existing resource protection does not enforce Pod-creation exclusivity.
|
||||
|
||||
The maintenance credential additionally needs Pod `list` and `patch` in the runtime namespace, Node Lease `get` in `kube-node-lease`, and cluster PV/VolumeAttachment `list`. It never patches a node, attachment, PVC or PV. Keep these privileges away from the worker and public caller.
|
||||
|
||||
Using the same Java entry point, request and deployment configuration as above:
|
||||
|
||||
1. Close all Sessions, then run `begin /absolute/path/retirement.json`. The Workspace remains `DRAINING` throughout retirement and after release.
|
||||
2. Read the exact original Pod's current resourceVersion. Run `prepare-reclamation /absolute/path/retirement.json <resourceVersion>` before DELETE. It verifies the original running container, healthy node/boot, Node Lease, original attachment and exclusive physical-volume use, saves the baseline, then conditionally adds the operation's finalizer. Read a fresh resourceVersion afterward.
|
||||
3. Run `delete-pod /absolute/path/retirement.json <fresh-resourceVersion>`. It sends the same normal 30-second graceful DELETE and retains its actual response or `UNKNOWN`. Deletion-only operations without preparation cannot be retroactively qualified after the Pod disappears.
|
||||
4. Run `reclaim /absolute/path/retirement.json` to observe completion. It requires the retained original Pod and explicitly terminated original container, same node/boot with fresh Ready status and Lease, no remaining node volume use or attachment, and no physical alias or other Pod consumer. While any condition is missing, ownership remains held. After a durable cleanup receipt, it conditionally removes only its finalizer and waits for original Pod absence. It then atomically releases operation/binding/active slot/physical ownership. Run `inspect` after an unknown result; the command is bounded and may need another invocation while the platform finishes.
|
||||
5. Only after `RELEASED`, the administrator may replace the sole configured mapping with a new Workspace ID in the same tenant, provision its reviewed saved Workspace metadata and OWNER grant, and restart with the replacement configuration. Retain the original PVC/PV, registration, disk and files. The new request starts at runtime generation 1 and obtains the next physical reservation revision. The old Workspace/request stays fenced; its terminal receipt remains readable with its current OWNER grant even after configuration replacement and later reuse.
|
||||
|
||||
The data policy is `retain-same-tenant-admin-handoff/1`. Physical revisions advance through reserved/released cycles without resetting; retained tenant ownership rejects a foreign-tenant alias. There is no automatic assignment, data wipe, force-release override or reopening of the retired Workspace. A node/API partition, identity change, force deletion or missing Pod before a saved cleanup receipt retains ownership. Infrastructure repair alone cannot authorize product release. A receipt assumes the exclusive use boundary remains in force and does not authorize external remounts.
|
||||
|
||||
## Failure handling and acceptance
|
||||
|
||||
When an execution or release outcome is unknown, preserve its Session and
|
||||
|
||||
@@ -12,7 +12,7 @@ Read、Write 和 Edit。Session 关闭会结算并撤销原上下文，下一个
|
||||
可选的显式空闲退役由 #14765 实现，使用方式见下文。本配置不需要 Go 工具链、
|
||||
定制 containerd/shim、节点 collector 或冷启动安装。
|
||||
|
||||
[2026-10-24 范围决定](../../docs/design/2026-10-23-csi-evidence-source.zh-CN.md)明确完整 CSI 审计不再是首版前置条件。Session 交接保留原 Pod/PVC，显式空闲退役保留存储归属。平台托管回收和跨 Workspace 卷复用仍是后续工作。
|
||||
[2026-10-24 范围决定](../../docs/design/2026-10-23-csi-evidence-source.zh-CN.md)明确完整 CSI 审计不再是首版前置条件。Session 交接保留原 Pod/PVC，仅删除 Pod 的空闲退役保留存储归属。下文的显式 S2/S3 回收与保留数据交接实现需要独立原生验收。
|
||||
|
||||
## 可信平台与支持的存储
|
||||
|
||||
@@ -110,7 +110,7 @@ Kubernetes API 核验原资源，再记录带原 UID/resourceVersion 前置条
|
||||
及 Background propagation 的正常 Pod DELETE。凭据留在已有 API/token/CA 配置中。
|
||||
不删除 PVC/PV、Secret 或保护对象。同版本重试只读原结果，不重发；新版本需要
|
||||
显式再次调用，总计最多八次。HTTP 回复和 Pod 缺失不会释放存储或重新开放 Workspace。
|
||||
物理释放和复用延后。
|
||||
回收必须使用下文的准备与完成协议。
|
||||
|
||||
即使 Kubernetes 已接受删除，回复丢失仍可能留下 `UNKNOWN`。原 Pod 一旦具有
|
||||
`deletionTimestamp` 或已缺失，不同 `resourceVersion` 也不能授权再次 DELETE。
|
||||
@@ -121,6 +121,22 @@ Kubernetes API 核验原资源，再记录带原 UID/resourceVersion 前置条
|
||||
与[简体中文](../../docs/design/2026-10-20-k2-physical-retirement-progress.zh-CN.md)。
|
||||
原生 ACK 退役资格验证继续待完成。
|
||||
|
||||
## 显式回收与同租户数据交接
|
||||
|
||||
[S2/S3 设计](../../docs/design/2026-10-27-resident-workspace-reclamation.zh-CN.md) 增加健康正常清理路径。原生 ACK 验收仍待完成，不能由命令存在或先前 S1 运行推断通过。保留现有单控制面信任边界，使用专用 runtime namespace/PVC。准入/RBAC 必须保证只有该 Managed provisioner 能创建使用此 PVC 的 Pod；维护和交接期间不得有外部控制器、其他 Pod 创建者、PV 物理别名或管理员存储操作。现有资源保护不约束 Pod 创建的独占性。
|
||||
|
||||
维护凭据额外需要 runtime namespace 的 Pod `list`/`patch`、`kube-node-lease` 中 Node Lease 的 `get`，以及集群 PV/VolumeAttachment `list`。不修改节点、attachment、PVC 或 PV。不要向 worker 和公开调用者授予这些权限。
|
||||
|
||||
沿用上文的 Java 入口、请求与部署配置：
|
||||
|
||||
1. 关闭全部 Session，然后执行 `begin /absolute/path/retirement.json`。退役期间与释放后，Workspace 均保持 `DRAINING`。
|
||||
2. 读取精确原 Pod 当前 resourceVersion，在 DELETE 前执行 `prepare-reclamation /absolute/path/retirement.json <resourceVersion>`。它校验原运行中容器、健康节点/boot、Node Lease、原 attachment 及物理卷独占使用，保存基线，再条件添加操作的自有 finalizer。之后重新读取 resourceVersion。
|
||||
3. 执行 `delete-pod /absolute/path/retirement.json <fresh-resourceVersion>`，发送原有正常 30 秒优雅 DELETE，保留实际响应或 `UNKNOWN`。没有准备步骤的仅删除操作，不能在 Pod 消失后追认合格。
|
||||
4. 执行 `reclaim /absolute/path/retirement.json` 观察完成。要求保留的原 Pod 和明确终止的原容器、相同节点/boot 上新鲜的 Ready 状态和 Lease、节点不再使用卷且没有 attachment、物理别名或其他 Pod 消费者。任一条件缺失都保留归属。持久化清理回执后，仅条件移除自己的 finalizer，并等待原 Pod 消失，再原子释放操作/绑定/active slot/物理归属。结果未知时执行 `inspect`；命令有界，平台尚未完成时可能需要再次调用。
|
||||
5. 只有达到 `RELEASED` 后，管理员才可将唯一配置映射替换为同租户的新 Workspace ID，准备经审查的 saved Workspace 元数据和 OWNER 授权，并按新配置重启。保留原 PVC/PV、注册、磁盘和文件。新请求的 runtime generation 从 1 开始，取得下一个物理 reservation revision。旧 Workspace/请求保持围栏；配置替换和后续复用后，持有当前 OWNER 授权仍可读取其终态回执。
|
||||
|
||||
数据策略是 `retain-same-tenant-admin-handoff/1`。物理 revision 在占用/释放轮次间递增且不重置；保留的数据所属租户拒绝其他租户的别名。不提供自动分配、数据擦除、强制释放 override 或重新开放已退役 Workspace。节点/API 分区、身份变化、强制删除，或保存清理回执前原 Pod 消失，均保留归属。基础设施修复本身不授权产品释放。回执依赖独占使用边界持续成立，不授权外部重挂载。
|
||||
|
||||
## 故障处理与验收
|
||||
|
||||
执行或释放结果未知时，保留其 Session 和 reservation，并拒绝下一个 Session。
|
||||
|
||||
@@ -53,6 +53,8 @@ Regression coverage must include independent resident storage sharing a mount pa
|
||||
|
||||
## Follow-up platform-managed reclamation
|
||||
|
||||
The [S2/S3 reclamation design](2026-10-27-resident-workspace-reclamation.md) specifies the proposed completion, atomic release, and retained-data handoff contract. Its implementation and platform qualification are tracked separately from the historical S1 evidence below.
|
||||
|
||||
The next design delegates ordinary container termination and volume cleanup to supported ACK/Kubernetes lifecycle mechanisms. Complete CSI RPC history is no longer a required implementation choice. Before implementing release, that design must define:
|
||||
|
||||
1. Exact eligibility and immutable identity joins for the original Workspace, runtime generation, Pod/node, registered volume and current reservation.
|
||||
|
||||
@@ -53,6 +53,8 @@ resident 运行时词表仅包含 `read_file`、`write_file`、`edit`；模型
|
||||
|
||||
## 后续平台托管回收
|
||||
|
||||
[S2/S3 回收设计](2026-10-27-resident-workspace-reclamation.zh-CN.md) 规定拟议的完成条件、原子释放与保留数据交接契约。其实现与平台验收独立于下文的历史 S1 证据跟踪。
|
||||
|
||||
下一份设计把正常容器终止和卷清理交给受支持的 ACK/Kubernetes 生命周期机制，完整 CSI RPC 历史不再是必选实现方式。在实现释放前，设计必须明确：
|
||||
|
||||
1. 原 Workspace、runtime generation、Pod/node、注册卷和当前 reservation 的精确资格与不可变身份关联。
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
# Resident Workspace reclamation and retained-data handoff
|
||||
|
||||
[English](2026-10-27-resident-workspace-reclamation.md) | [简体中文](2026-10-27-resident-workspace-reclamation.zh-CN.md)
|
||||
|
||||
## Status
|
||||
|
||||
Combined S2 design and S3 implementation for [#13395](https://github.com/QwenLM/qwen-code/issues/13395). Implementation baseline: main `0539160f35f983ba4a6d4d369c0ae18941b2fdb2`. This document describes the implemented contract; complete platform acceptance remains pending.
|
||||
|
||||
On 2026-10-27 focused Java/Checkstyle, real MySQL/process checks and an isolated ACK healthy handoff passed in their recorded scopes. The native run used the implementation baseline plus the 31-file source manifest with SHA-256 `9edfca3f939c5e19b4cebdacbc13460b9c4b4590c2de2c9113b30d59fa299ee2`, ACK `1.36.2-aliyun.1` and Disk CSI `1.37.3`. One disk passed A→B→C→D with reserved revisions `1/3/5/7`, released revisions `2/4/6`, and 22 successful file-tool executions preserving prior bytes. The model endpoint was a controlled fixture. This evidence covers that source/artifact snapshot and healthy sequence; a rebase does not qualify a new integrated artifact. Full CI, the required native fault matrix and production least-privilege qualification remain pending.
|
||||
|
||||
On 2026-10-27 the operator selected preserving files and explicitly handing the disk to another Workspace in the same tenant. Reuse must preserve this decision through every ownership cycle. The [existing delivery boundary](2026-10-23-csi-evidence-source.md) and separate CSI LOST recovery issue [#12670](https://github.com/QwenLM/qwen-code/issues/12670) still apply.
|
||||
|
||||
## Problem and baseline state
|
||||
|
||||
Idle retirement fences the Workspace and original binding as `DRAINING`. It retains the active slot and `RESERVED` physical owner. Its normal Pod DELETE records an actual HTTP result or `UNKNOWN`; neither result releases anything. There is no durable platform-cleanup receipt or atomic physical release.
|
||||
|
||||
Four resident consumers assume reservation revision `1`, and the shared original-only CSI protocol allows only revisions `0`, `1`, and `2`. Releasing the current owner without changing these consumers cannot support repeated handoff. The CSI owner also lacks a populated tenant data owner. Clearing its reservation identity would otherwise let a foreign-tenant alias acquire retained files. Retirement inspection currently depends on the active configuration mapping, so changing that mapping would hide the old operation.
|
||||
|
||||
## Goals and boundaries
|
||||
|
||||
Provide explicit administrator maintenance for a healthy, idle, original resident runtime, with durable readback, atomic release, and repeated same-tenant handoff. Preserve original files, PVC/PV, disk, protection, and historical identity. Keep one configured Workspace per physical volume.
|
||||
|
||||
Do not add automatic expiry, active-Session forced drain, reopening of the retired Workspace, same-request replacement generations, cross-tenant reuse, data wiping, Shell/MCP/Hooks, CSI RPC auditing, node agents, or automatic LOST recovery. An uncertain shutdown stays fenced. External infrastructure cleanup does not itself change product ownership.
|
||||
|
||||
## Trust and platform completion
|
||||
|
||||
The supported boundary trusts the reviewed administrator, Kubernetes API and kubelet, host, and qualified ACK Disk CSI. The maintenance process uses its configured HTTPS origin, pinned CA, rotating credentials, and existing registration/protection verification. A request file supplies identity only; it cannot supply a completion receipt or claim that a node was fenced.
|
||||
|
||||
The runtime namespace and PVC are exclusive to this single Managed provisioner. Deployment admission/RBAC must exclude other Pod creators and external controllers using the PVC; administrators must not remount, alias, or force-modify the storage during maintenance or handoff. The existing resource protection alone does not enforce Pod-creation exclusivity. Complete bounded namespace Pod scans at preparation and completion reject every nonoriginal Pod referencing the PVC, including Pending and terminal Pods. Complete bounded PV scans require the registered name/UID to be the sole PV for the CSI driver/volumeHandle and reject physical aliases. Product-created consumers are fenced by Workspace and physical ownership. A receipt does not claim safety against subsequent external remount or cluster-admin actions outside this prerequisite.
|
||||
|
||||
Normal completion requires all of these observations, joined to the original resource handle:
|
||||
|
||||
1. Before DELETE, the exact original Pod is running with the original sole container ID, image ID, zero restarts, spec digest, Pod UID, and node UID. It has no owner references, deletion timestamp, or foreign finalizers. The original healthy node has a nonempty boot ID, the CSI unique volume in `status.volumesInUse`, and exactly one matching attached `VolumeAttachment` for the original PV, driver, and node. Record its UID. A current `kube-node-lease` Lease must belong to that Node UID and name.
|
||||
2. Install only the operation's deterministic Pod finalizer with JSON Patch tests for UID and resourceVersion, before sending the existing normal 30-second graceful DELETE. Preserve the Pod object while the kubelet and CSI perform normal cleanup. The finalizer is a metadata retention mechanism, not evidence of physical completion.
|
||||
3. Observe that same deleting Pod with the owned finalizer still present, unchanged spec and container identity, and the original container explicitly `terminated`, including `finishedAt`. A Pod phase alone is insufficient. Require no restart, replacement container, init container, or sidecar.
|
||||
4. Read the same Node UID and boot ID, `Ready=True`, no deleting/out-of-service state, and no occurrence of the original CSI unique volume in either `status.volumesInUse` or `status.volumesAttached`. Its Ready heartbeat must advance beyond the baseline and cover the container termination. Read the same Node Lease UID with the same Node owner and holder, a renewed lease after termination, and a current renewal within its declared duration, capped at 60 seconds. An old Ready condition or a fresh Lease alone is insufficient.
|
||||
5. A complete, bounded list of `storage.k8s.io/v1/volumeattachments` must contain no attachment for the original PV. Reject partial pages, malformed entries, duplicate identities, inconsistent pagination, overflow, and a replacement matching attachment. Recheck the original registration and protection. These are conjunctive observations, not independent release permissions.
|
||||
|
||||
The unique volume name is `kubernetes.io/csi/<driver>^<volumeHandle>`, matched exactly. Node and Lease timestamps are evaluated conservatively; clock skew or missing fields holds ownership. No force delete, force detach, node shutdown claim, cloud-disk status, or user-supplied boolean can substitute for these conditions.
|
||||
|
||||
`deletionTimestamp` is a grace deadline, not the start of deletion. The original container may finish before it, and normal kubelet finalization may change it. Require a valid non-null deletion timestamp and the original terminated-container evidence; do not compare `finishedAt` against the grace deadline or require that deadline to stay equal to the saved receipt. Fresh Node/Lease observations must still cover termination. See Kubernetes [graceful deletion implementation](https://github.com/kubernetes/kubernetes/blob/v1.36.2/staging/src/k8s.io/apiserver/pkg/registry/rest/delete.go).
|
||||
|
||||
Kubernetes documents [graceful Pod termination](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) and [finalizers](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/). The kubelet removes terminating Pods from its desired volume state after their containers stop; the attach/detach controller removes terminal Pods from its desired state. Therefore a retained terminal Pod should permit normal volume cleanup. This ordering is an implementation dependency to verify on the target ACK release. [Non-graceful node shutdown](https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/) can force detach without normal unmount; that path is excluded by the same-node, same-boot, terminated-container and fresh-status checks. A missing VolumeAttachment alone never proves unmount.
|
||||
|
||||
## Maintenance protocol and durable state
|
||||
|
||||
Extend the existing trusted maintenance entry point; expose no public or daemon route. The existing `inspect`, `begin`, and deletion-only `delete-pod` remain available. Add `prepare-reclamation <request-json> <resourceVersion>` and `reclaim <request-json>`.
|
||||
|
||||
| Stage | Durable contract | Infrastructure action |
|
||||
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ |
|
||||
| `begin` | Save a versioned immutable plan with the original mapping, request, binding/generation, handle identity and digest. Set Workspace/binding `DRAINING`; retain slot and owner. | None. |
|
||||
| `prepare-reclamation` | Save the original node boot/Lease/attachment baseline and a bounded conditional finalizer attempt before I/O. | Add the owned finalizer to the verified original Pod. |
|
||||
| `delete-pod` | Preserve the existing per-Pod-UID/resourceVersion attempt and `UNKNOWN` semantics. A prepared operation accepts exactly its owned finalizer. | One normal UID/RV-conditional DELETE per durable attempt. |
|
||||
| `reclaim`: observe | Revalidate the held original identity and idle state. Save a bounded immutable cleanup receipt and its digest; move operation to `PLATFORM_CLEANED`. | Read the conjunctive completion observations above. |
|
||||
| `reclaim`: finish | Save the conditional finalizer-removal intent before I/O. Keep all product ownership until the original Pod is absent. | Remove only the owned finalizer using UID/RV tests; read original Pod absence. |
|
||||
| `reclaim`: commit | In one SQL transaction mark the operation `RELEASED`, binding `RELEASED`, clear only its active slot, advance the physical revision to `RELEASED`, and retain tenant data ownership. | None. |
|
||||
|
||||
Every mutation is preceded by durable intent and uses a bounded response. Lost, incomplete, or unrecorded responses remain `UNKNOWN`. A DELETE attempt at the same Pod UID/resourceVersion is never resent. Metadata PATCH alone permits a bounded retry after a fresh GET proves the original UID, the pre-effect finalizer set, and the same or newer resourceVersion. Each PATCH tests UID, resourceVersion and the exact pre-effect finalizer set; a successful write advances the version and rejects stale copies. Preserve every old `UNKNOWN` attempt and SQL claim-generation fencing. This permits recovery from a crash before PATCH without waiting for an unrelated Pod change. Do not rewrite an earlier `UNKNOWN` DELETE into success when later observations prove cleanup.
|
||||
|
||||
Keep the receipt independent from transient HTTP outcomes. Once the exact original deleting Pod is terminal and its physical cleanup has been durably observed, that fact can survive a process restart and finalizer removal. A crash before receipt persistence requires observing completion again; Pod absence without a saved receipt cannot reconstruct one. A crash after receipt persistence resumes only the owned-finalizer cleanup and final SQL transition. Refuse a same-name replacement Pod even after recording cleanup.
|
||||
|
||||
An old version-1 deletion-only plan remains inspectable and conservative. It cannot retroactively qualify a Pod that was deleted without preparation. New plans freeze their mapping so terminal inspection after configuration changes needs no current mapping or current owner equality. Validate the original request actor's current OWNER grant, tenant, Workspace and generation, immutable plan and receipt digests, and terminal binding history. Keep the retired Workspace `DRAINING`; do not misuse the public Workspace removal lifecycle as a physical-release signal.
|
||||
|
||||
## Atomic release and lock order
|
||||
|
||||
Use the existing ten-second transaction bound and lock order: tenant placement guard, tenant publication guard, Workspace/access, original runtime slot, original binding, storage alias, physical owner, retirement operation, then Session/journal/RuntimeSession/execution idle checks. No Kubernetes HTTP call occurs inside this transaction.
|
||||
|
||||
At final commit, recheck the immutable plan and receipt, exact original binding and generation, sealed `DRAINING` state, no operation claim, original resource handle/lease identity, active slot, registration/protection, held reservation ID/revision/seed, and idle state. Require the terminal original-Pod readback associated with that receipt. Update the binding through a dedicated transaction-only repository primitive. Keep general binding CAS closed to sealed CSI transitions.
|
||||
|
||||
The physical CAS clears binding, reservation, registration and provision-seed references only for that exact holder. Preserve the tenant data owner. Advance revision without reset. Clear the exact active slot while preserving its last generation and original request identity. Mark retirement terminal in the same transaction. Any mismatch or failed update rolls everything back. A stale sender or lost SQL reply reads the terminal receipt and cannot modify a later holder.
|
||||
|
||||
Retain the original binding through `WORKSPACE_RETIREMENT` references, including terminal records. This is intentionally bounded by explicit maintenance history retention; this change adds no automatic deletion of those records. The receipt and original history must remain inspectable after another Workspace reserves the disk.
|
||||
|
||||
## Repeated same-tenant handoff
|
||||
|
||||
Populate and preserve `ma_storage_owner.tenant_id` on the first CSI reservation. Registration, inspection, and reservation refuse a different tenant once this owner exists. A previously reserved pre-release row may be backfilled only under the exact original binding/registration lock; never infer the tenant from the new claimant.
|
||||
|
||||
Resident ownership uses monotonically increasing even `RELEASED` and odd `RESERVED` revisions: `0 → 1 → 2 → 3 → 4 → 5`. Match the exact revision from the original handle at every execution, close, observation and retirement read. The private CSI retirement protocol retains its original-only `0/1/2` predicates and does not gain release or reuse authority from this extension. Later-cycle reservations require the resident profile; overflow or invalid phase/revision fails closed.
|
||||
|
||||
After `RELEASED`, the administrator replaces the sole configured mapping with a different Workspace in the same tenant and restarts with the new configuration. The existing PVC/PV, physical key, registration and files remain. Use a new Workspace ID and request key; its generation starts at 1 and its reservation gets the next physical revision. The old Workspace and request stay fenced. This configuration action is the explicit data handoff; there is no implicit assignment based on a disk becoming available. Concurrent aliases still serialize on the physical owner, and only one exact claimant can reserve it.
|
||||
|
||||
## Exceptional paths and readback
|
||||
|
||||
| Observation | Required outcome |
|
||||
| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
|
||||
| Busy Session, live writer, unsettled execution or RuntimeSession | Refuse maintenance; do not install a new fence or release anything. |
|
||||
| API outage, lost reply, stale Lease/heartbeat, missing node, changed boot ID, partition or out-of-service taint | Retain original slot/owner and closed admission; report incomplete reclamation. |
|
||||
| Forced Pod deletion or missing Pod before a saved cleanup receipt | Hold ownership; no receipt synthesis from absence. |
|
||||
| Replaced Pod/node/PVC/PV/protection/attachment or changed container | Hold ownership; never adopt the replacement. |
|
||||
| Crash before the release transaction commits | Recover the held original state and resume readback; no partial release. |
|
||||
| Crash or lost response after commit | Return the same terminal receipt, including after subsequent reuse; no new mutation. |
|
||||
| Foreign tenant or old Workspace tries to acquire the disk | Refuse, preserving both data tenancy and the new holder. |
|
||||
|
||||
An administrator may diagnose and repair infrastructure independently. Such repair does not authorize an exceptional product release. A future node/storage isolation procedure must have its own reviewed API contract and qualification; this change does not provide an override command. CSI LOST recovery and replay of unknown tools remain out of scope.
|
||||
|
||||
## Components affected
|
||||
|
||||
Extend the resident retirement store and maintenance entry point, bounded Kubernetes HTTP client, transaction-only binding repository, CSI reservation store, resident execution/close/observation predicates, operation state/work domains in the canonical V1 schema, and runtime retention guard. Update focused Java, MySQL and process-crash tests and the bilingual resident deployment instructions. No TypeScript, public API or OpenAPI changes are planned.
|
||||
|
||||
The canonical V1 work-kind domain changes. Follow the server's pre-release schema policy and recreate development catalogs; this change supplies no V2 migration or catalog upgrade path. Reading an old version-1 retirement plan is a protocol safeguard, not a promise that an older catalog can be upgraded in place. Preserve the separately retained volume files.
|
||||
|
||||
## Validation and acceptance
|
||||
|
||||
Use the detailed ignored E2E plan under `.qwen/e2e-tests/13395-s2-s3-20261027/`. Dry-run the global CLI first and explicitly record if the private maintenance path needs a Java/API fallback. Share one 600-second local Java verification budget across setup, builds, selected unit tests and selected real MySQL/process cases; leave full profiles to CI and keep uncovered checks explicit.
|
||||
|
||||
Required acceptance includes at least three reserve/release cycles with preserved file contents across distinct same-tenant Workspaces; execution and close at revisions greater than 1; foreign-tenant and old-Workspace refusal; two concurrent claimants; platform identity substitutions; API and node/network partition retention; duplicate and lost replies; crashes before/after receipt persistence and SQL commit; rollback at each release write; and terminal inspection after mapping replacement and later reuse. Verify task-owned resource cleanup independently.
|
||||
|
||||
Native ACK qualification must record Kubernetes/CSI versions and exact source/artifact identity. Prove the owned finalizer permits container termination, Node volume cleanup, attachment removal and final Pod deletion without a node-side helper. If that sequence fails, keep release disabled by its unmet predicates and revise the design; a unit fixture or Pod DELETE success cannot establish acceptance. Qualify healthy normal completion and fail-closed exceptional cases separately. Required CI profiles and full native acceptance remain pending until their evidence exists.
|
||||
|
||||
## Open qualification questions
|
||||
|
||||
Extend the recorded healthy retained-Pod cleanup result to the required native fault matrix and any different target Kubernetes/CSI versions. Verify production exclusive namespace/PVC use and the absence of external creators/controllers; the isolated run verified its own creator admission boundary. Check deployment RBAC grants only the additional Pod patch/list, Node Lease get, and PV/VolumeAttachment list permissions needed by this maintenance process; the isolated run used a trusted administrative control plane and did not qualify minimum RBAC. These are platform verification tasks, not permission to weaken completion conditions.
|
||||
@@ -0,0 +1,112 @@
|
||||
# 常驻 Workspace 回收与保留数据交接
|
||||
|
||||
[English](2026-10-27-resident-workspace-reclamation.md) | [简体中文](2026-10-27-resident-workspace-reclamation.zh-CN.md)
|
||||
|
||||
## 状态
|
||||
|
||||
这是 [#13395](https://github.com/QwenLM/qwen-code/issues/13395) 合并提交的 S2 设计与 S3 实现。实现基线为 main `0539160f35f983ba4a6d4d369c0ae18941b2fdb2`。本文描述已实现契约；完整平台验收仍待完成。
|
||||
|
||||
2026-10-27，定向 Java/Checkstyle、真实 MySQL/进程检查以及隔离 ACK 健康交接在各自记录的范围内通过。原生运行使用实现基线加 31 文件源码清单，其 SHA-256 为 `9edfca3f939c5e19b4cebdacbc13460b9c4b4590c2de2c9113b30d59fa299ee2`，平台为 ACK `1.36.2-aliyun.1` 与 Disk CSI `1.37.3`。同一磁盘完成 A→B→C→D，reserved revision 为 `1/3/5/7`、released revision 为 `2/4/6`，22 次文件工具执行成功并保留前序字节。模型端点使用受控 fixture。该证据仅覆盖上述源码/制品快照与健康流程；rebase 不会自动验收新集成制品。完整 CI、要求的原生故障矩阵与生产最小权限验收仍待完成。
|
||||
|
||||
2026-10-27，操作者选择保留文件，并由管理员显式将磁盘交接给同租户的另一个 Workspace。每一轮归属变化都必须遵守此决定。[现有交付边界](2026-10-23-csi-evidence-source.zh-CN.md) 及独立的 CSI LOST 恢复问题 [#12670](https://github.com/QwenLM/qwen-code/issues/12670) 继续有效。
|
||||
|
||||
## 问题与基线状态
|
||||
|
||||
空闲退役将 Workspace 与原绑定设为 `DRAINING`，保留 active slot 和 `RESERVED` 物理归属。正常 Pod DELETE 保存实际 HTTP 结果或 `UNKNOWN`，两者均不释放资源。目前没有持久化的平台清理回执或原子物理释放。
|
||||
|
||||
四处常驻消费者假定 reservation revision 为 `1`，共享的首次 CSI 协议仅允许 revision `0`、`1`、`2`。只释放当前 owner 而不调整这些消费者，无法支持重复交接。CSI owner 也未填充数据所属租户；如果清空 reservation 身份，其他租户的别名就可能取得保留文件。退役查询依赖当前配置映射，替换映射会导致旧操作无法回读。
|
||||
|
||||
## 目标与边界
|
||||
|
||||
为健康、空闲、身份未变的原常驻运行时提供管理员显式维护，支持持久化回读、原子释放和重复同租户交接。保留原文件、PVC/PV、磁盘、保护对象及历史身份。同一物理卷只配置一个 Workspace。
|
||||
|
||||
不增加自动过期、活跃 Session 强制 drain、重新开放已退役 Workspace、同一请求的替代 generation、跨租户复用、擦除数据、Shell/MCP/Hooks、CSI RPC 审计、节点代理或自动 LOST 恢复。不确定的停止结果继续保留围栏。外部基础设施清理本身不改变产品归属。
|
||||
|
||||
## 信任与平台完成条件
|
||||
|
||||
支持边界信任经过审查的管理员、Kubernetes API 与 kubelet、宿主机和已验收的 ACK Disk CSI。维护进程使用配置的 HTTPS origin、固定 CA、轮换凭证及现有注册/保护校验。请求文件只提供身份，不能提供完成回执或自行声称节点已被隔离。
|
||||
|
||||
runtime namespace 与 PVC 由此唯一 Managed provisioner 专用。部署准入/RBAC 必须排除使用该 PVC 的其他 Pod 创建者和外部控制器；维护或交接期间，管理员不得重新挂载、创建物理别名或强制修改存储。现有资源保护本身不约束 Pod 创建的独占性。准备与完成阶段都完整有界地扫描 namespace Pods，拒绝包括 Pending 和终态 Pod 在内的所有非原 Pod 对 PVC 的引用。完整有界地扫描 PV，要求注册 name/UID 是该 CSI driver/volumeHandle 的唯一 PV，拒绝物理别名。产品创建的消费者由 Workspace 和物理归属围栏约束。回执不声称能抵抗违反此前提的后续外部重挂载或 cluster-admin 操作。
|
||||
|
||||
正常完成要求同时满足以下观察，且全部关联到原 resource handle：
|
||||
|
||||
1. DELETE 前，精确的原 Pod 正在运行，唯一容器的 ID、image ID、零重启、spec digest、Pod UID 与 node UID 均匹配。不得有 owner reference、删除时间或其他 finalizer。原健康节点有非空 boot ID，其 `status.volumesInUse` 包含精确 CSI unique volume；原 PV、driver 和节点恰好对应一个已挂接的 `VolumeAttachment`，并保存其 UID。当前 `kube-node-lease` Lease 必须属于该 Node UID 与名称。
|
||||
2. 在现有正常 30 秒优雅 DELETE 前，用 JSON Patch 测试 UID 和 resourceVersion，只安装该操作的确定性 Pod finalizer。在 kubelet 与 CSI 正常清理期间保留 Pod 对象。finalizer 只用于保留元数据，本身不证明物理完成。
|
||||
3. 观察同一个正在删除的 Pod，其自有 finalizer 仍存在，spec 与容器身份未变，原容器明确为 `terminated`，且包含 `finishedAt`。单独的 Pod phase 不够。不得重启或替换容器，也不得有 init container 或 sidecar。
|
||||
4. 读取相同 Node UID 和 boot ID，要求 `Ready=True`，没有删除或 out-of-service 状态，且 `status.volumesInUse` 与 `status.volumesAttached` 均不再包含原 CSI unique volume。Ready heartbeat 必须晚于基线并覆盖容器终止时间。读取相同 Node Lease UID，Node owner 与 holder 相同，续租发生在终止之后，当前续租在声明的有效期内，且有效期最多 60 秒。旧 Ready condition 或单独的新 Lease 均不够。
|
||||
5. 完整且有界的 `storage.k8s.io/v1/volumeattachments` 列表不含原 PV 的任何 attachment。拒绝不完整分页、格式错误、身份重复、分页不一致、超限和替代的匹配 attachment。重新校验原注册与保护。上述观察必须同时成立，任何单项都不是释放授权。
|
||||
|
||||
unique volume 名称为 `kubernetes.io/csi/<driver>^<volumeHandle>`，使用精确匹配。保守判断 Node 和 Lease 时间；时钟偏差或字段缺失均保留归属。强制删除、强制 detach、节点已关机的声明、云盘状态或用户布尔值均不能代替这些条件。
|
||||
|
||||
`deletionTimestamp` 是宽限期截止时间，不是删除开始时间。原容器可以在它之前终止，正常 kubelet 收尾也可以改变它。要求有效的非空删除时间和原容器明确终止的证据，不将 `finishedAt` 与宽限期截止时间比较，也不要求该截止时间始终等于已存回执。新鲜 Node/Lease 观察仍须覆盖终止。参见 Kubernetes [优雅删除实现](https://github.com/kubernetes/kubernetes/blob/v1.36.2/staging/src/k8s.io/apiserver/pkg/registry/rest/delete.go)。
|
||||
|
||||
Kubernetes 文档说明了 [Pod 优雅终止](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) 和 [finalizer](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/)。kubelet 在容器停止后从期望卷状态中移除终止中的 Pod；attach/detach controller 从期望状态中移除终态 Pod。因此，保留终态 Pod 对象应允许正常卷清理。这一顺序是必须在目标 ACK 版本验证的实现依赖。[非优雅节点关闭](https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/) 可能跳过正常卸载而强制 detach；相同节点、相同启动、原容器终止和新鲜状态的组合检查排除该路径。单独的 VolumeAttachment 消失永远不证明卸载。
|
||||
|
||||
## 维护协议与持久化状态
|
||||
|
||||
扩展现有可信维护入口，不增加公开或 daemon 路由。保留现有 `inspect`、`begin` 和仅删除 Pod 的 `delete-pod`。增加 `prepare-reclamation <request-json> <resourceVersion>` 与 `reclaim <request-json>`。
|
||||
|
||||
| 阶段 | 持久化契约 | 基础设施动作 |
|
||||
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------- |
|
||||
| `begin` | 保存带版本的不可变计划，包含原映射、请求、绑定/generation、handle 身份与 digest。Workspace/绑定变为 `DRAINING`，保留 slot 和 owner。 | 无。 |
|
||||
| `prepare-reclamation` | 在 I/O 前保存原节点 boot/Lease/attachment 基线及有界条件 finalizer 尝试。 | 为校验过的原 Pod 添加自有 finalizer。 |
|
||||
| `delete-pod` | 保留现有按 Pod UID/resourceVersion 记录的尝试与 `UNKNOWN` 语义。已准备的操作只接受自己的 finalizer。 | 每个持久化尝试只发送一次正常 UID/RV 条件 DELETE。 |
|
||||
| `reclaim`：观察 | 重新校验持有的原身份与空闲状态，保存有界不可变清理回执及 digest，操作进入 `PLATFORM_CLEANED`。 | 读取上述同时成立的完成观察。 |
|
||||
| `reclaim`：收尾 | 在 I/O 前保存条件 finalizer 移除意图。原 Pod 消失前保留全部产品归属。 | 用 UID/RV 测试仅移除自己的 finalizer，再回读原 Pod 已不存在。 |
|
||||
| `reclaim`：提交 | 同一 SQL 事务中将操作设为 `RELEASED`、绑定设为 `RELEASED`、仅清空其 active slot、推进物理 revision 为 `RELEASED`，保留数据所属租户。 | 无。 |
|
||||
|
||||
每次修改前先持久化意图，响应有大小上限。丢失、不完整或未记录的响应保持 `UNKNOWN`。相同 Pod UID/resourceVersion 的 DELETE 尝试不得重发。只有 metadata PATCH 可以在新鲜 GET 确认原 UID、修改前 finalizer 集合和相同或更新的 resourceVersion 后有界重试。每次 PATCH 都测试 UID、resourceVersion 和精确的修改前 finalizer 集合；成功写入推进版本，拒绝旧副本。保留每条旧 `UNKNOWN` 和 SQL claim-generation 围栏。这样 PATCH 前崩溃可以恢复，不必等待无关的 Pod 变化。后续观察证明清理完成时，不得把早先 `UNKNOWN` DELETE 改写为成功。
|
||||
|
||||
回执独立于临时 HTTP 结果。只要精确的原删除中 Pod 已终止，且物理清理观察已经持久化，该事实可跨进程重启和 finalizer 移除保留。回执持久化前崩溃，需要重新观察；没有已存回执时，不能由 Pod 消失重建回执。持久化后崩溃，只恢复自有 finalizer 清理和最终 SQL 转换。即使已有清理记录，也拒绝同名替代 Pod。
|
||||
|
||||
旧 version-1 仅删除计划保持可查询和保守行为，不能追认没有准备步骤就被删除的 Pod。新计划冻结映射，终态查询不依赖当前映射或当前 owner 相等。校验原请求 actor 当前的 OWNER 授权、租户、Workspace 和 generation、不可变计划/回执 digest，以及终态绑定历史。已退役 Workspace 保持 `DRAINING`，不借用公开 Workspace 删除生命周期表示物理释放。
|
||||
|
||||
## 原子释放与锁顺序
|
||||
|
||||
沿用十秒事务上限与锁顺序：租户 placement guard、租户 publication guard、Workspace/access、原 runtime slot、原绑定、storage alias、物理 owner、退役操作，最后检查 Session/journal/RuntimeSession/execution 空闲。事务内不调用 Kubernetes HTTP。
|
||||
|
||||
最终提交时重新核对不可变计划与回执、精确原绑定和 generation、封存的 `DRAINING` 状态、没有 operation claim、原 handle/lease 身份、active slot、注册/保护、持有的 reservation ID/revision/seed 及空闲状态。要求与该回执关联的原 Pod 终态回读。通过专门的事务内 repository primitive 更新绑定，通用 binding CAS 继续禁止 sealed CSI 转换。
|
||||
|
||||
物理 CAS 仅为精确原 holder 清空绑定、reservation、注册与 provision seed 引用，保留数据所属租户，推进 revision 而不重置。仅清空精确 active slot，保留 last generation 和原请求身份。退役操作在同一事务中终结。任何身份不符或更新失败都整体回滚。过期发送者或 SQL 响应丢失后只读取终态回执，不能更改后续 holder。
|
||||
|
||||
通过 `WORKSPACE_RETIREMENT` 引用保留原绑定，包括终态记录。历史保留受显式维护历史策略约束；本改动不增加自动删除这些记录的行为。其他 Workspace 占用磁盘后，回执和原历史仍必须可读。
|
||||
|
||||
## 重复同租户交接
|
||||
|
||||
首次 CSI reservation 时填充并保留 `ma_storage_owner.tenant_id`。一旦数据 owner 存在，注册、查询和占用均拒绝其他租户。旧版本已占用行只能在精确原绑定/注册锁下回填，不得从新申请者推断原数据所属租户。
|
||||
|
||||
常驻归属使用单调递增的偶数 `RELEASED` 和奇数 `RESERVED` revision：`0 → 1 → 2 → 3 → 4 → 5`。每次执行、关闭、观察和退役均匹配原 handle 的精确 revision。私有 CSI 退役协议保留首次 `0/1/2` 判断，不能因本扩展而获得释放或复用权限。后续轮次只接受 resident profile；溢出或非法 phase/revision 均拒绝。
|
||||
|
||||
达到 `RELEASED` 后，管理员将唯一配置映射替换为同租户的另一个 Workspace，并按新配置重启。保留现有 PVC/PV、physical key、注册及文件。新 Workspace 使用新的 ID 和 request key，自己的 generation 从 1 开始，reservation 使用下一个物理 revision。旧 Workspace 和请求保持围栏。该配置动作就是显式数据交接，不因磁盘空闲而隐式分配。并发别名仍在物理 owner 上串行化，只有一个精确申请者能占用。
|
||||
|
||||
## 异常路径与回读
|
||||
|
||||
| 观察 | 必须结果 |
|
||||
| ----------------------------------------------------------------------------------------------- | ------------------------------------------------ |
|
||||
| 忙碌 Session、活跃 writer、未结算 execution 或 RuntimeSession | 拒绝维护，不增加新围栏，不释放。 |
|
||||
| API 不可用、响应丢失、Lease/heartbeat 过期、节点消失、boot ID 改变、分区或 out-of-service taint | 保留原 slot/owner 和关闭的准入，报告回收未完成。 |
|
||||
| 原 Pod 被强制删除，或保存清理回执前已消失 | 保留归属，不由缺失合成回执。 |
|
||||
| Pod/node/PVC/PV/保护/attachment 被替换，或容器身份改变 | 保留归属，不接管替代对象。 |
|
||||
| 释放事务提交前崩溃 | 恢复持有原归属的状态并继续回读，不允许部分释放。 |
|
||||
| 提交后崩溃或响应丢失 | 返回相同终态回执，后续复用后也如此，不再次修改。 |
|
||||
| 其他租户或旧 Workspace 尝试占用磁盘 | 拒绝，保留数据租户边界与新 holder。 |
|
||||
|
||||
管理员可独立诊断和修复基础设施，但不因此取得异常产品释放权限。未来节点/存储隔离流程必须有独立审查的 API 契约与验收；本改动不提供 override 命令。CSI LOST 恢复和未知工具重放继续不在范围内。
|
||||
|
||||
## 受影响组件
|
||||
|
||||
扩展常驻退役 store 与维护入口、有界 Kubernetes HTTP client、事务内 binding repository、CSI reservation store、常驻执行/关闭/观察判断、规范 V1 schema 中的 operation state/work domain，以及 runtime retention guard。更新相关 Java、MySQL、进程崩溃测试及双语常驻部署说明。计划不修改 TypeScript、公开 API 或 OpenAPI。
|
||||
|
||||
规范 V1 的 work-kind domain 有变化。遵循 server 首次正式发布前的 schema 策略，重建开发 catalog；本改动不提供 V2 migration 或 catalog 升级路径。读取旧 version-1 退役计划属于协议保护，不代表旧 catalog 可以原地升级。独立保留卷中的文件。
|
||||
|
||||
## 验证与验收
|
||||
|
||||
详细忽略目录 E2E 计划位于 `.qwen/e2e-tests/13395-s2-s3-20261027/`。先以全局 CLI dry-run，若私有维护路径需要 Java/API fallback 则明确记录。环境准备、构建、定向单测及定向真实 MySQL/进程用例共同使用一个 600 秒本地 Java 验证预算；全量 profile 交由 CI，未覆盖检查单独列明。
|
||||
|
||||
必须验收至少三轮 reserve/release，在同租户不同 Workspace 间保留文件内容；revision 大于 1 时执行与关闭；拒绝其他租户和旧 Workspace；两个并发申请者；平台身份替换；API 及节点/网络分区时保留归属；重复及丢失响应；回执持久化和 SQL 提交前后崩溃；各释放写入点回滚；映射替换与后续复用后的终态查询。独立复核任务资源清理。
|
||||
|
||||
原生 ACK 验收必须记录 Kubernetes/CSI 版本及精确源码/制品身份，证明自有 finalizer 不依赖节点辅助进程即可允许容器终止、Node 卷清理、attachment 移除及最终 Pod 删除。若顺序不成立，完成条件不满足，释放保持不可用，并修改设计；单测 fixture 或 Pod DELETE 成功不能证明验收。健康正常完成与异常拒绝分别验收。在证据实际存在前，必要 CI profile 与完整原生验收保持待完成。
|
||||
|
||||
## 待验证的平台问题
|
||||
|
||||
将已记录的健康保留 Pod 清理结果扩展到要求的原生故障矩阵，以及任何不同的目标 Kubernetes/CSI 版本。核验生产 namespace/PVC 的独占使用以及不存在其他创建者/控制器；隔离运行已验证其自身的创建者准入边界。检查部署 RBAC 只增加该维护进程需要的 Pod patch/list、Node Lease get 和 PV/VolumeAttachment list 权限；隔离运行使用受信任管理控制面，未验收最小 RBAC。这些是平台验证任务，不能用来放松完成条件。
|
||||
+1
-1
@@ -40,7 +40,7 @@ public final class ManagedRuntimeRetentionGuard implements JdbcRuntimeRetention.
|
||||
identity(binding.getBindingId()), binding.getGeneration())
|
||||
|| exists(
|
||||
connection,
|
||||
"SELECT 1 FROM ma_operation WHERE operation_kind = 'CSI_RETIREMENT' AND"
|
||||
"SELECT 1 FROM ma_operation WHERE operation_kind IN ('CSI_RETIREMENT', 'WORKSPACE_RETIREMENT') AND"
|
||||
+ " target_kind = 'RUNTIME_BINDING' AND target_id = ? AND"
|
||||
+ " target_revision = ? LIMIT 1",
|
||||
identity(binding.getBindingId()),
|
||||
|
||||
+205
@@ -0,0 +1,205 @@
|
||||
package com.alibaba.qwen.code.managedagent.persistence;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties.RuntimeBroker.ResidentWorkspace;
|
||||
import com.alibaba.qwen.code.runtimebroker.BrokerValues;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.provisioning.KubernetesHttpRuntimeClient;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import static com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity.digest;
|
||||
import static com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity.map;
|
||||
import static com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity.text;
|
||||
|
||||
/** Normal platform cleanup inside an exclusive Managed namespace; never an exceptional fence. */
|
||||
final class ResidentWorkspaceReclamation {
|
||||
static final String DATA_POLICY = "retain-same-tenant-admin-handoff/1";
|
||||
|
||||
record Baseline(String finalizer, String bootId, String nodeResourceVersion, String nodeHeartbeat,
|
||||
String leaseUid, String leaseResourceVersion, String leaseRenewTime, String attachmentUid) { }
|
||||
|
||||
record Receipt(String resourceIdentity, String baselineDigest, String podResourceVersion,
|
||||
String deletionTimestamp, String finishedAt, String nodeResourceVersion, String nodeHeartbeat,
|
||||
String leaseResourceVersion, String leaseRenewTime, String observedAt, String dataPolicy) { }
|
||||
|
||||
private record Node(Map<String, Object> value, String bootId, String resourceVersion, Instant heartbeat) { }
|
||||
private record Lease(String uid, String resourceVersion, Instant renewed) { }
|
||||
|
||||
private ResidentWorkspaceReclamation() { }
|
||||
|
||||
static Baseline prepare(RuntimeBindingRecord original, ResidentWorkspace selected,
|
||||
String finalizer, KubernetesHttpRuntimeClient api, Instant now) {
|
||||
var placement = map(original.getResourceHandle().getValue().get("placement"));
|
||||
var node = node(placement, api, now, false);
|
||||
var lease = lease(placement, api, now);
|
||||
requireVolumes(node.value(), uniqueVolume(selected), true);
|
||||
exclusive(original, selected, api);
|
||||
var attachments = attachments(selected, api);
|
||||
require(attachments.size() == 1);
|
||||
var attachment = attachments.getFirst();
|
||||
var metadata = map(attachment.get("metadata"));
|
||||
require(metadata.get("deletionTimestamp") == null
|
||||
&& placement.get("nodeName").equals(map(attachment.get("spec")).get("nodeName"))
|
||||
&& Boolean.TRUE.equals(map(attachment.get("status")).get("attached")));
|
||||
return new Baseline(finalizer, node.bootId(), node.resourceVersion(), node.heartbeat().toString(),
|
||||
lease.uid(), lease.resourceVersion(), lease.renewed().toString(), text(metadata, "uid", 128));
|
||||
}
|
||||
|
||||
static Receipt complete(RuntimeBindingRecord original, ResidentWorkspace selected, Baseline baseline,
|
||||
Map<String, Object> pod, KubernetesHttpRuntimeClient api, Instant now) {
|
||||
var handle = original.getResourceHandle().getValue();
|
||||
var placement = map(handle.get("placement"));
|
||||
require(pod != null);
|
||||
var metadata = map(pod.get("metadata"));
|
||||
require(placement.get("podUid").equals(metadata.get("uid"))
|
||||
&& placement.get("podName").equals(metadata.get("name"))
|
||||
&& placement.get("namespace").equals(metadata.get("namespace"))
|
||||
&& List.of(baseline.finalizer()).equals(metadata.get("finalizers"))
|
||||
&& empty(metadata.get("ownerReferences")) && digest(pod.get("spec")).equals(handle.get("podSpecDigest")));
|
||||
var spec = map(pod.get("spec"));
|
||||
require(empty(spec.get("initContainers")) && empty(spec.get("ephemeralContainers")));
|
||||
var status = map(pod.get("status"));
|
||||
require(List.of("Succeeded", "Failed").contains(status.get("phase"))
|
||||
&& status.get("containerStatuses") instanceof List<?> containers && containers.size() == 1);
|
||||
var container = map(((List<?>) status.get("containerStatuses")).getFirst());
|
||||
var state = map(container.get("state"));
|
||||
require(placement.get("containerName").equals(container.get("name"))
|
||||
&& placement.get("containerId").equals(container.get("containerID"))
|
||||
&& placement.get("imageId").equals(container.get("imageID"))
|
||||
&& Long.valueOf(0).equals(BrokerValues.exactLong(container.get("restartCount")))
|
||||
&& Boolean.FALSE.equals(container.get("ready")) && state.keySet().equals(java.util.Set.of("terminated")));
|
||||
var terminated = map(state.get("terminated"));
|
||||
require(placement.get("containerId").equals(terminated.get("containerID"))
|
||||
&& !"ContainerStatusUnknown".equals(terminated.get("reason")));
|
||||
Instant deletion = instant(metadata, "deletionTimestamp");
|
||||
Instant finished = instant(terminated, "finishedAt");
|
||||
// The API deletion timestamp is a grace deadline, not the start of termination.
|
||||
require(!finished.isAfter(now));
|
||||
var node = node(placement, api, now, true);
|
||||
var lease = lease(placement, api, now);
|
||||
require(node.bootId().equals(baseline.bootId()) && !node.resourceVersion().equals(baseline.nodeResourceVersion())
|
||||
&& node.heartbeat().isAfter(Instant.parse(baseline.nodeHeartbeat())) && !node.heartbeat().isBefore(finished)
|
||||
&& lease.uid().equals(baseline.leaseUid()) && !lease.resourceVersion().equals(baseline.leaseResourceVersion())
|
||||
&& lease.renewed().isAfter(Instant.parse(baseline.leaseRenewTime())) && !lease.renewed().isBefore(finished));
|
||||
requireVolumes(node.value(), uniqueVolume(selected), false);
|
||||
exclusive(original, selected, api);
|
||||
require(attachments(selected, api).isEmpty());
|
||||
return new Receipt((String) handle.get("identity"), digest(baseline), text(metadata, "resourceVersion", 256),
|
||||
deletion.toString(), finished.toString(), node.resourceVersion(), node.heartbeat().toString(),
|
||||
lease.resourceVersion(), lease.renewed().toString(), now.toString(), DATA_POLICY);
|
||||
}
|
||||
|
||||
static void verifyReceipt(RuntimeBindingRecord original, Baseline baseline, Receipt receipt) {
|
||||
require(receipt != null && receipt.resourceIdentity().equals(original.getResourceHandle().getValue().get("identity"))
|
||||
&& receipt.baselineDigest().equals(digest(baseline)) && DATA_POLICY.equals(receipt.dataPolicy())
|
||||
&& receipt.observedAt() != null && receipt.finishedAt() != null && receipt.deletionTimestamp() != null
|
||||
&& !Instant.parse(receipt.observedAt()).isBefore(Instant.parse(receipt.finishedAt())));
|
||||
Instant.parse(receipt.deletionTimestamp());
|
||||
}
|
||||
|
||||
private static Node node(Map<String, Object> placement, KubernetesHttpRuntimeClient api, Instant now, boolean fresh) {
|
||||
var node = api.getCluster("nodes", (String) placement.get("nodeName")).toCompletableFuture().join();
|
||||
require(node != null);
|
||||
var metadata = map(node.get("metadata"));
|
||||
require(placement.get("nodeUid").equals(metadata.get("uid")) && placement.get("nodeName").equals(metadata.get("name"))
|
||||
&& metadata.get("deletionTimestamp") == null);
|
||||
var spec = map(node.get("spec"));
|
||||
if (spec.get("taints") instanceof List<?> taints) {
|
||||
require(taints.stream().noneMatch(value -> List.of("node.kubernetes.io/out-of-service",
|
||||
"node.kubernetes.io/unreachable", "node.kubernetes.io/not-ready").contains(map(value).get("key"))));
|
||||
}
|
||||
var status = map(node.get("status"));
|
||||
require(status.get("conditions") instanceof List<?>);
|
||||
var conditions = ((List<?>) status.get("conditions")).stream().map(value -> map(value))
|
||||
.filter(value -> "Ready".equals(value.get("type"))).toList();
|
||||
require(conditions.size() == 1 && "True".equals(conditions.getFirst().get("status")));
|
||||
var heartbeat = instant(conditions.getFirst(), "lastHeartbeatTime");
|
||||
require(!heartbeat.isAfter(now) && (!fresh || !heartbeat.isBefore(now.minusSeconds(60))));
|
||||
return new Node(node, text(map(status.get("nodeInfo")), "bootID", 256),
|
||||
text(metadata, "resourceVersion", 256), heartbeat);
|
||||
}
|
||||
|
||||
private static Lease lease(Map<String, Object> placement, KubernetesHttpRuntimeClient api, Instant now) {
|
||||
var lease = api.getNodeLease((String) placement.get("nodeName")).toCompletableFuture().join();
|
||||
require(lease != null);
|
||||
var metadata = map(lease.get("metadata"));
|
||||
require(placement.get("nodeName").equals(metadata.get("name"))
|
||||
&& "kube-node-lease".equals(metadata.get("namespace")) && metadata.get("deletionTimestamp") == null
|
||||
&& metadata.get("ownerReferences") instanceof List<?> owners && owners.size() == 1);
|
||||
var owner = map(((List<?>) metadata.get("ownerReferences")).getFirst());
|
||||
require("v1".equals(owner.get("apiVersion")) && "Node".equals(owner.get("kind"))
|
||||
&& placement.get("nodeUid").equals(owner.get("uid")) && placement.get("nodeName").equals(owner.get("name")));
|
||||
var spec = map(lease.get("spec"));
|
||||
Long duration = BrokerValues.exactLong(spec.get("leaseDurationSeconds"));
|
||||
Instant renewed = instant(spec, "renewTime");
|
||||
require(placement.get("nodeName").equals(spec.get("holderIdentity")) && duration != null && duration > 0 && duration <= 60
|
||||
&& !renewed.isAfter(now) && renewed.plusSeconds(duration).isAfter(now));
|
||||
return new Lease(text(metadata, "uid", 128), text(metadata, "resourceVersion", 256), renewed);
|
||||
}
|
||||
|
||||
private static void exclusive(RuntimeBindingRecord original, ResidentWorkspace selected, KubernetesHttpRuntimeClient api) {
|
||||
var registration = selected.registration();
|
||||
var physical = api.listPersistentVolumes().toCompletableFuture().join().stream().filter(pv -> {
|
||||
var spec = map(pv.get("spec"));
|
||||
return spec.get("csi") instanceof Map<?, ?> csi && registration.driver().equals(csi.get("driver"))
|
||||
&& registration.volumeHandle().equals(csi.get("volumeHandle"));
|
||||
}).toList();
|
||||
require(physical.size() == 1 && registration.pvName().equals(map(physical.getFirst().get("metadata")).get("name"))
|
||||
&& registration.pvUid().equals(map(physical.getFirst().get("metadata")).get("uid")));
|
||||
var placement = map(original.getResourceHandle().getValue().get("placement"));
|
||||
var consumers = api.listPods(registration.namespace()).toCompletableFuture().join().stream().filter(pod -> {
|
||||
var volumes = map(pod.get("spec")).get("volumes");
|
||||
return volumes instanceof List<?> list && list.stream().anyMatch(value -> map(value).get("persistentVolumeClaim")
|
||||
instanceof Map<?, ?> pvc && registration.pvcName().equals(pvc.get("claimName")));
|
||||
}).toList();
|
||||
require(consumers.size() == 1 && placement.get("podUid").equals(map(consumers.getFirst().get("metadata")).get("uid"))
|
||||
&& placement.get("podName").equals(map(consumers.getFirst().get("metadata")).get("name")));
|
||||
}
|
||||
|
||||
private static List<Map<String, Object>> attachments(ResidentWorkspace selected, KubernetesHttpRuntimeClient api) {
|
||||
return api.listVolumeAttachments().toCompletableFuture().join().stream().filter(value -> {
|
||||
var spec = map(value.get("spec"));
|
||||
if (!selected.registration().pvName().equals(map(spec.get("source")).get("persistentVolumeName"))) {
|
||||
return false;
|
||||
}
|
||||
require(selected.registration().driver().equals(spec.get("attacher")));
|
||||
return true;
|
||||
}).toList();
|
||||
}
|
||||
|
||||
private static void requireVolumes(Map<String, Object> node, String unique, boolean mounted) {
|
||||
var status = map(node.get("status"));
|
||||
var inUse = status.get("volumesInUse");
|
||||
var attached = status.get("volumesAttached");
|
||||
require(inUse == null || inUse instanceof List<?>);
|
||||
require(attached == null || attached instanceof List<?>);
|
||||
boolean using = inUse instanceof List<?> list && list.contains(unique);
|
||||
boolean attaching = attached instanceof List<?> list && list.stream().anyMatch(value -> unique.equals(map(value).get("name")));
|
||||
require(using == mounted && attaching == mounted);
|
||||
}
|
||||
|
||||
private static String uniqueVolume(ResidentWorkspace selected) {
|
||||
return "kubernetes.io/csi/" + selected.registration().driver() + "^" + selected.registration().volumeHandle();
|
||||
}
|
||||
|
||||
private static Instant instant(Map<String, Object> value, String key) {
|
||||
try {
|
||||
return Instant.parse(text(value, key, 128));
|
||||
} catch (java.time.DateTimeException failure) {
|
||||
throw unavailable();
|
||||
}
|
||||
}
|
||||
|
||||
private static boolean empty(Object value) { return value == null || value instanceof List<?> list && list.isEmpty(); }
|
||||
private static void require(boolean valid) {
|
||||
if (!valid) {
|
||||
throw unavailable();
|
||||
}
|
||||
}
|
||||
private static RuntimeBrokerException unavailable() {
|
||||
return new RuntimeBrokerException(409, "resident_reclamation_incomplete", "Original platform cleanup is incomplete.", false);
|
||||
}
|
||||
}
|
||||
+9
-3
@@ -40,9 +40,9 @@ public final class ResidentWorkspaceRetirementMain {
|
||||
|
||||
static void run(String[] args) throws Exception {
|
||||
var command = new DefaultApplicationArguments(args).getNonOptionArgs();
|
||||
if (command.size() < 2 || !Set.of("inspect", "begin", "delete-pod").contains(command.getFirst())
|
||||
|| command.size() != ("delete-pod".equals(command.getFirst()) ? 3 : 2)) {
|
||||
throw new IllegalArgumentException("Expected inspect|begin <request-json> or delete-pod <request-json> <resourceVersion>.");
|
||||
if (command.size() < 2 || !Set.of("inspect", "begin", "delete-pod", "prepare-reclamation", "reclaim").contains(command.getFirst())
|
||||
|| command.size() != (Set.of("delete-pod", "prepare-reclamation").contains(command.getFirst()) ? 3 : 2)) {
|
||||
throw new IllegalArgumentException("Expected inspect|begin|reclaim <request-json> or delete-pod|prepare-reclamation <request-json> <resourceVersion>.");
|
||||
}
|
||||
var json = JsonMapper.builder().enable(StreamReadFeature.STRICT_DUPLICATE_DETECTION)
|
||||
.disable(DeserializationFeature.ACCEPT_FLOAT_AS_INT)
|
||||
@@ -66,6 +66,12 @@ public final class ResidentWorkspaceRetirementMain {
|
||||
case "delete-pod" -> store.deletePod(request, command.get(2), new KubernetesHttpRuntimeClient(
|
||||
URI.create(options.getResidentKubernetesApiUrl()), Path.of(options.getResidentKubernetesTokenFile()),
|
||||
Path.of(options.getResidentKubernetesCaFile())));
|
||||
case "prepare-reclamation" -> store.prepareReclamation(request, command.get(2), new KubernetesHttpRuntimeClient(
|
||||
URI.create(options.getResidentKubernetesApiUrl()), Path.of(options.getResidentKubernetesTokenFile()),
|
||||
Path.of(options.getResidentKubernetesCaFile())));
|
||||
case "reclaim" -> store.reclaim(request, new KubernetesHttpRuntimeClient(
|
||||
URI.create(options.getResidentKubernetesApiUrl()), Path.of(options.getResidentKubernetesTokenFile()),
|
||||
Path.of(options.getResidentKubernetesCaFile())));
|
||||
default -> throw new IllegalArgumentException("Unknown maintenance action");
|
||||
};
|
||||
System.out.println(json.writeValueAsString(result));
|
||||
|
||||
+296
-21
@@ -3,6 +3,7 @@ package com.alibaba.qwen.code.managedagent.persistence;
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties.RuntimeBroker.ResidentWorkspace;
|
||||
import com.alibaba.qwen.code.managedagent.workspace.runtime.WorkspaceCsiResourceGuard;
|
||||
import com.alibaba.qwen.code.runtimebroker.BrokerValues;
|
||||
import com.alibaba.qwen.code.runtimebroker.CsiOwnershipRevision;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
@@ -10,6 +11,7 @@ import com.alibaba.qwen.code.runtimebroker.RuntimeProvisionRequest;
|
||||
import com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaTime;
|
||||
import com.alibaba.qwen.code.runtimebroker.provisioning.KubernetesHttpRuntimeClient;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import com.fasterxml.jackson.core.JsonParser;
|
||||
import com.fasterxml.jackson.databind.DeserializationFeature;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
@@ -20,6 +22,7 @@ import org.springframework.transaction.support.TransactionTemplate;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Base64;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
@@ -32,9 +35,11 @@ import static com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity.di
|
||||
import static com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity.map;
|
||||
import static com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity.same;
|
||||
|
||||
/** Explicit idle maintenance; Pod deletion never releases physical storage. */
|
||||
/** Explicit idle maintenance with original platform cleanup and atomic ownership release. */
|
||||
public final class ResidentWorkspaceRetirementStore {
|
||||
private static final String WORK = "POD_DELETE_ATTEMPT";
|
||||
private static final String FINALIZER_WORK = "POD_FINALIZER_ATTEMPT";
|
||||
private static final int PLAN_LIMIT = 256 * 1024;
|
||||
private final JdbcTemplate jdbc;
|
||||
private final TransactionTemplate transaction;
|
||||
private final JdbcRuntimeBindingRepository bindings;
|
||||
@@ -53,14 +58,20 @@ public final class ResidentWorkspaceRetirementStore {
|
||||
}
|
||||
}
|
||||
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
private record Plan(int version, Request request, String bindingId, long runtimeGeneration,
|
||||
String resourceIdentity, String mappingDigest) { }
|
||||
String resourceIdentity, String mappingDigest, ResidentWorkspace mapping) { }
|
||||
|
||||
public record AttemptStatus(int ordinal, String resourceVersion, String state, Integer httpStatus,
|
||||
String responseSha256) { }
|
||||
|
||||
public record FinalizerAttemptStatus(int ordinal, String action, String resourceVersion, String state,
|
||||
Integer httpStatus, String responseSha256) { }
|
||||
|
||||
public record Status(String retirementId, String state, String bindingId, long runtimeGeneration,
|
||||
List<AttemptStatus> attempts) { }
|
||||
List<AttemptStatus> attempts, List<FinalizerAttemptStatus> finalizerAttempts, Completion completion) { }
|
||||
|
||||
public record Completion(String receiptSha256, String dataPolicy, Long releasedRevision) { }
|
||||
|
||||
private record Attempt(String workKey, String owner, long generation, byte[] body) { }
|
||||
|
||||
@@ -80,15 +91,28 @@ public final class ResidentWorkspaceRetirementStore {
|
||||
|
||||
public Status inspect(Request request) {
|
||||
return transaction.execute(status -> {
|
||||
var selected = authorize(request, false);
|
||||
var saved = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), false);
|
||||
if (saved == null) {
|
||||
return new Status(request.retirementId(), "NOT_STARTED", null, 0, List.of());
|
||||
authorize(request, false);
|
||||
return new Status(request.retirementId(), "NOT_STARTED", null, 0, List.of(), List.of(), null);
|
||||
}
|
||||
Plan plan = plan(request, selected, saved);
|
||||
Plan plan = readPlan(request, saved);
|
||||
boolean released = "RELEASED".equals(saved.get("state"));
|
||||
var selected = released ? plan.mapping() : selected(request);
|
||||
require(selected != null && digest(selected).equals(plan.mappingDigest()));
|
||||
authorize(request, selected, false);
|
||||
var original = bindings.findById(plan.bindingId());
|
||||
verifyPlan(plan, original);
|
||||
verifyOriginal(selected, original, false);
|
||||
if (released) {
|
||||
require(original.getState() == RuntimeBindingRecord.State.RELEASED);
|
||||
receipt(saved, original);
|
||||
} else {
|
||||
require(original.getState() == RuntimeBindingRecord.State.DRAINING);
|
||||
verifyOriginal(selected, original, false);
|
||||
if ("PLATFORM_CLEANED".equals(saved.get("state"))) {
|
||||
receipt(saved, original);
|
||||
}
|
||||
}
|
||||
return status(request, plan);
|
||||
});
|
||||
}
|
||||
@@ -134,9 +158,10 @@ public final class ResidentWorkspaceRetirementStore {
|
||||
require(jdbc.queryForList("SELECT operation_key FROM ma_operation WHERE operation_kind = 'WORKSPACE_RETIREMENT'"
|
||||
+ " AND target_key = ? AND target_revision = ? FOR UPDATE", String.class,
|
||||
WorkspaceOperations.hash(locked.getBindingId()), locked.getGeneration()).isEmpty());
|
||||
Plan plan = new Plan(1, request, locked.getBindingId(), locked.getGeneration(),
|
||||
(String) locked.getResourceHandle().getValue().get("identity"), digest(selected));
|
||||
Plan plan = new Plan(2, request, locked.getBindingId(), locked.getGeneration(),
|
||||
(String) locked.getResourceHandle().getValue().get("identity"), digest(selected), selected);
|
||||
String encoded = encode(plan);
|
||||
require(encoded.getBytes(StandardCharsets.UTF_8).length <= PLAN_LIMIT);
|
||||
operations.create(WORKSPACE_RETIREMENT, request.retirementId(), WorkspaceOperations.values(
|
||||
"binding_id", locked.getBindingId(), "runtime_generation", locked.getGeneration(),
|
||||
"storage_id", selected.registration().storageId(), "physical_key", selected.registration().physicalKey(),
|
||||
@@ -168,7 +193,9 @@ public final class ResidentWorkspaceRetirementStore {
|
||||
selected.protection()).verify().toCompletableFuture().join();
|
||||
String podName = (String) placement.get("podName");
|
||||
var pod = api.get("pods", selected.registration().namespace(), podName).toCompletableFuture().join();
|
||||
verifyPod(original, pod, resourceVersion);
|
||||
var saved = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), false);
|
||||
var baseline = saved.get("reclamationBaseline") == null ? null : baseline(saved);
|
||||
verifyPod(original, pod, resourceVersion, baseline == null ? List.of() : List.of(baseline.finalizer()));
|
||||
var node = api.getCluster("nodes", (String) placement.get("nodeName")).toCompletableFuture().join();
|
||||
require(node != null && placement.get("nodeUid").equals(map(node.get("metadata")).get("uid"))
|
||||
&& map(node.get("status")).get("conditions") instanceof List<?> conditions
|
||||
@@ -227,12 +254,227 @@ public final class ResidentWorkspaceRetirementStore {
|
||||
return inspect(request);
|
||||
}
|
||||
|
||||
public Status prepareReclamation(Request request, String resourceVersion, KubernetesHttpRuntimeClient api) {
|
||||
require(resourceVersion != null && resourceVersion.matches("[\\x21-\\x7e]{1,256}"));
|
||||
var prior = inspect(request);
|
||||
if (!"DRAINING".equals(prior.state())) {
|
||||
require(!"NOT_STARTED".equals(prior.state()));
|
||||
return prior;
|
||||
}
|
||||
var selected = selected(request);
|
||||
var original = transaction.execute(status -> lockedPlan(request, selected, true));
|
||||
var saved = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), false);
|
||||
require(readPlan(request, saved).version() == 2);
|
||||
verifyProtection(selected, api);
|
||||
var placement = map(original.getResourceHandle().getValue().get("placement"));
|
||||
var pod = api.get("pods", selected.registration().namespace(), (String) placement.get("podName")).toCompletableFuture().join();
|
||||
require(pod != null);
|
||||
String finalizer = finalizer(request, original);
|
||||
var retained = saved.get("reclamationBaseline") == null ? null : baseline(saved);
|
||||
if (retained != null && List.of(finalizer).equals(map(pod.get("metadata")).get("finalizers"))) {
|
||||
verifyPod(original, pod, resourceVersion, List.of(finalizer));
|
||||
return inspect(request);
|
||||
}
|
||||
verifyPod(original, pod, resourceVersion, List.of());
|
||||
var observed = ResidentWorkspaceReclamation.prepare(original, selected, finalizer, api, Instant.now());
|
||||
transaction.executeWithoutResult(status -> {
|
||||
lockedPlan(request, selected, true);
|
||||
idle(request);
|
||||
var current = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), true);
|
||||
require("DRAINING".equals(current.get("state")));
|
||||
if (current.get("reclamationBaseline") == null) {
|
||||
operations(request).update(WORKSPACE_RETIREMENT, request.retirementId(), WorkspaceOperations.values(
|
||||
"reclamationBaseline", json.valueToTree(observed), "reclamationBaselineDigest", digest(observed)));
|
||||
} else {
|
||||
var first = baseline(current);
|
||||
require(first.finalizer().equals(finalizer) && first.bootId().equals(observed.bootId())
|
||||
&& first.leaseUid().equals(observed.leaseUid()) && first.attachmentUid().equals(observed.attachmentUid()));
|
||||
}
|
||||
});
|
||||
patchFinalizer(request, selected, original, pod, "ADD", api);
|
||||
return inspect(request);
|
||||
}
|
||||
|
||||
public Status reclaim(Request request, KubernetesHttpRuntimeClient api) {
|
||||
var prior = inspect(request);
|
||||
if ("RELEASED".equals(prior.state())) {
|
||||
return prior;
|
||||
}
|
||||
var selected = selected(request);
|
||||
var original = transaction.execute(status -> lockedPlan(request, selected, true));
|
||||
var saved = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), false);
|
||||
require(readPlan(request, saved).version() == 2);
|
||||
var baseline = baseline(saved);
|
||||
require(baseline.finalizer().equals(finalizer(request, original)));
|
||||
var placement = map(original.getResourceHandle().getValue().get("placement"));
|
||||
String namespace = selected.registration().namespace();
|
||||
String name = (String) placement.get("podName");
|
||||
if ("DRAINING".equals(saved.get("state"))) {
|
||||
require(!attempts(request).isEmpty());
|
||||
verifyProtection(selected, api);
|
||||
var pod = api.get("pods", namespace, name).toCompletableFuture().join();
|
||||
var observed = ResidentWorkspaceReclamation.complete(original, selected, baseline, pod, api, Instant.now());
|
||||
transaction.executeWithoutResult(status -> {
|
||||
lockedPlan(request, selected, true);
|
||||
idle(request);
|
||||
var current = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), true);
|
||||
if ("DRAINING".equals(current.get("state"))) {
|
||||
require(digest(baseline).equals(current.get("reclamationBaselineDigest")));
|
||||
operations(request).update(WORKSPACE_RETIREMENT, request.retirementId(), WorkspaceOperations.values(
|
||||
"state", "PLATFORM_CLEANED", "cleanupReceipt", json.valueToTree(observed), "cleanupReceiptDigest", digest(observed)));
|
||||
} else {
|
||||
receipt(current, original);
|
||||
}
|
||||
});
|
||||
}
|
||||
saved = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), false);
|
||||
var proof = receipt(saved, original);
|
||||
var pod = api.get("pods", namespace, name).toCompletableFuture().join();
|
||||
if (pod != null) {
|
||||
var metadata = map(pod.get("metadata"));
|
||||
require(placement.get("podUid").equals(metadata.get("uid"))
|
||||
&& placement.get("podName").equals(metadata.get("name"))
|
||||
&& placement.get("namespace").equals(metadata.get("namespace"))
|
||||
&& metadata.get("deletionTimestamp") instanceof String
|
||||
&& digest(pod.get("spec")).equals(original.getResourceHandle().getValue().get("podSpecDigest")));
|
||||
Instant.parse((String) metadata.get("deletionTimestamp"));
|
||||
if (List.of(baseline.finalizer()).equals(metadata.get("finalizers"))) {
|
||||
patchFinalizer(request, selected, original, pod, "REMOVE", api);
|
||||
} else {
|
||||
require(empty(metadata.get("finalizers")));
|
||||
}
|
||||
if (api.get("pods", namespace, name).toCompletableFuture().join() != null) {
|
||||
return inspect(request);
|
||||
}
|
||||
}
|
||||
return transaction.execute(status -> {
|
||||
var current = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), false);
|
||||
if ("RELEASED".equals(current.get("state"))) {
|
||||
return inspect(request);
|
||||
}
|
||||
var locked = lockedPlan(request, selected, true);
|
||||
idle(request);
|
||||
current = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), true);
|
||||
require("PLATFORM_CLEANED".equals(current.get("state")) && digest(proof).equals(current.get("cleanupReceiptDigest")));
|
||||
receipt(current, locked);
|
||||
require(jdbc.queryForObject("SELECT COUNT(*) FROM ma_operation WHERE operation_key = ?"
|
||||
+ " AND (claim_until IS NULL OR claim_until <= " + ManagedSchemaTime.SQL_NOW + ")", Long.class, operationKey(request)) == 1);
|
||||
var storage = map(locked.getResourceHandle().getValue().get("storage"));
|
||||
long revision = Math.addExact(Long.parseLong((String) storage.get("reservationRevision")), 1);
|
||||
jdbc.execute((ConnectionCallback<Void>) connection -> {
|
||||
bindings.releaseResidentForRetirement(connection, locked);
|
||||
return null;
|
||||
});
|
||||
require(jdbc.update("UPDATE ma_storage_owner SET csi_phase = 'RELEASED', csi_revision = ?, tenant_id = ?,"
|
||||
+ " binding_id = NULL, runtime_generation = NULL, csi_reservation_id = NULL, csi_registration_key = NULL,"
|
||||
+ " csi_registration_revision = NULL, csi_provision_request_id = NULL"
|
||||
+ " WHERE storage_key = ? AND storage_kind = 'CSI' AND csi_phase = 'RESERVED' AND csi_revision = ?"
|
||||
+ " AND binding_id = ? AND runtime_generation = ? AND csi_reservation_id = ?"
|
||||
+ " AND (tenant_id IS NULL OR tenant_id = ?)", revision, request.tenantId(), selected.registration().physicalKey(),
|
||||
revision - 1, locked.getBindingId(), locked.getGeneration(), storage.get("reservationId"), request.tenantId()) == 1);
|
||||
operations(request).update(WORKSPACE_RETIREMENT, request.retirementId(), WorkspaceOperations.values(
|
||||
"state", "RELEASED", "releasedRevision", revision));
|
||||
require(jdbc.update("UPDATE ma_operation SET claim_owner = NULL, claim_until = NULL, claim_generation = claim_generation + 1"
|
||||
+ " WHERE operation_key = ?", operationKey(request)) == 1);
|
||||
return status(request, readPlan(request, operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), false)));
|
||||
});
|
||||
}
|
||||
|
||||
private void patchFinalizer(Request request, ResidentWorkspace selected, RuntimeBindingRecord original,
|
||||
Map<String, Object> pod, String action, KubernetesHttpRuntimeClient api) {
|
||||
var metadata = map(pod.get("metadata"));
|
||||
String resourceVersion = (String) metadata.get("resourceVersion");
|
||||
require(resourceVersion != null && resourceVersion.matches("[\\x21-\\x7e]{1,256}"));
|
||||
String finalizer = finalizer(request, original);
|
||||
var before = metadata.get("finalizers");
|
||||
require("ADD".equals(action) ? empty(before) : List.of(finalizer).equals(before));
|
||||
var patch = new java.util.ArrayList<Map<String, Object>>();
|
||||
patch.add(Map.of("op", "test", "path", "/metadata/uid", "value", metadata.get("uid")));
|
||||
patch.add(Map.of("op", "test", "path", "/metadata/resourceVersion", "value", resourceVersion));
|
||||
if (before != null) {
|
||||
patch.add(Map.of("op", "test", "path", "/metadata/finalizers", "value", before));
|
||||
}
|
||||
patch.add(Map.of("op", "add", "path", "/metadata/finalizers", "value", "ADD".equals(action) ? List.of(finalizer) : List.of()));
|
||||
byte[] body = WorkspaceCsiRuntimeIdentity.bytes(patch);
|
||||
var attempt = transaction.execute(status -> {
|
||||
lockedPlan(request, selected, true);
|
||||
idle(request);
|
||||
var saved = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), true);
|
||||
require(baseline(saved).finalizer().equals(finalizer)
|
||||
&& ("ADD".equals(action) ? "DRAINING".equals(saved.get("state")) : "PLATFORM_CLEANED".equals(saved.get("state"))));
|
||||
if ("REMOVE".equals(action)) {
|
||||
receipt(saved, original);
|
||||
}
|
||||
var previous = finalizerAttempts(request);
|
||||
require(previous.size() < 16 && previous.stream().filter(value -> action.equals(value.get("action"))).count() < 8);
|
||||
String owner = UUID.randomUUID().toString();
|
||||
require(jdbc.update("UPDATE ma_operation SET claim_owner = ?, claim_generation = claim_generation + 1,"
|
||||
+ " claim_until = " + ManagedSchemaTime.SQL_NOW + " + 60000 WHERE operation_key = ?"
|
||||
+ " AND (claim_until IS NULL OR claim_until <= " + ManagedSchemaTime.SQL_NOW + ")", owner, operationKey(request)) == 1);
|
||||
long generation = jdbc.queryForObject("SELECT claim_generation FROM ma_operation WHERE operation_key = ?", Long.class, operationKey(request));
|
||||
String workKey = WorkspaceOperations.hash(action, (String) metadata.get("uid"), resourceVersion, Long.toString(generation));
|
||||
operations(request).insertWork(WORKSPACE_RETIREMENT, request.retirementId(), FINALIZER_WORK, workKey, workKey, null,
|
||||
WorkspaceOperations.values("ordinal", previous.size() + 1, "action", action, "resourceVersion", resourceVersion,
|
||||
"requestBase64", Base64.getEncoder().encodeToString(body), "claimGeneration", generation), "UNKNOWN");
|
||||
return new Attempt(workKey, owner, generation, body);
|
||||
});
|
||||
KubernetesHttpRuntimeClient.DeleteResponse response = null;
|
||||
try {
|
||||
response = api.patchPodFinalizers(selected.registration().namespace(),
|
||||
(String) map(original.getResourceHandle().getValue().get("placement")).get("podName"), body).toCompletableFuture().join();
|
||||
} catch (java.util.concurrent.CompletionException failure) {
|
||||
// Independent platform readback can recover the effect; this attempt stays UNKNOWN.
|
||||
}
|
||||
var reply = response;
|
||||
transaction.executeWithoutResult(status -> {
|
||||
lockedPlan(request, selected, false);
|
||||
require(jdbc.queryForObject("SELECT COUNT(*) FROM ma_operation WHERE operation_key = ? AND claim_owner = ?"
|
||||
+ " AND claim_generation = ? AND claim_until > " + ManagedSchemaTime.SQL_NOW, Long.class,
|
||||
operationKey(request), attempt.owner(), attempt.generation()) == 1);
|
||||
if (reply != null) {
|
||||
require(reply.body().length <= 64 * 1024);
|
||||
operations(request).updateWork(WORKSPACE_RETIREMENT, request.retirementId(), FINALIZER_WORK, attempt.workKey(),
|
||||
WorkspaceOperations.values("state", reply.complete() ? "RETURNED" : "UNKNOWN", "httpStatus", reply.status(),
|
||||
"responseSha256", org.apache.commons.codec.digest.DigestUtils.sha256Hex(reply.body()), "complete", reply.complete()));
|
||||
}
|
||||
jdbc.update("UPDATE ma_operation SET claim_owner = NULL, claim_until = NULL WHERE operation_key = ?"
|
||||
+ " AND claim_owner = ? AND claim_generation = ?", operationKey(request), attempt.owner(), attempt.generation());
|
||||
});
|
||||
}
|
||||
|
||||
private ResidentWorkspaceReclamation.Baseline baseline(Map<String, Object> saved) {
|
||||
require(saved.get("reclamationBaseline") != null);
|
||||
var baseline = json.convertValue(saved.get("reclamationBaseline"), ResidentWorkspaceReclamation.Baseline.class);
|
||||
require(baseline != null && digest(baseline).equals(saved.get("reclamationBaselineDigest")));
|
||||
return baseline;
|
||||
}
|
||||
|
||||
private ResidentWorkspaceReclamation.Receipt receipt(Map<String, Object> saved, RuntimeBindingRecord original) {
|
||||
require(saved.get("cleanupReceipt") != null);
|
||||
var receipt = json.convertValue(saved.get("cleanupReceipt"), ResidentWorkspaceReclamation.Receipt.class);
|
||||
require(receipt != null && digest(receipt).equals(saved.get("cleanupReceiptDigest")));
|
||||
ResidentWorkspaceReclamation.verifyReceipt(original, baseline(saved), receipt);
|
||||
return receipt;
|
||||
}
|
||||
|
||||
private static String finalizer(Request request, RuntimeBindingRecord original) {
|
||||
return "qwen.ai/retirement-" + WorkspaceOperations.hash(request.tenantId(), request.retirementId(), original.getBindingId()).substring(0, 40);
|
||||
}
|
||||
|
||||
private static void verifyProtection(ResidentWorkspace selected, KubernetesHttpRuntimeClient api) {
|
||||
new WorkspaceCsiResourceGuard(api, selected.registration().clusterDomain(), selected.registration(),
|
||||
selected.protection()).verify().toCompletableFuture().join();
|
||||
}
|
||||
|
||||
private ResidentWorkspace authorize(Request request, boolean lock) {
|
||||
return authorize(request, selected(request), lock);
|
||||
}
|
||||
|
||||
private ResidentWorkspace authorize(Request request, ResidentWorkspace selected, boolean lock) {
|
||||
if (lock) {
|
||||
WorkspaceMigrationAdmission.lockTenant(jdbc, request.tenantId(), 10);
|
||||
ToolPublicationRetentionStore.lockTenant(jdbc, request.tenantId(), 10);
|
||||
}
|
||||
var selected = selected(request);
|
||||
var rows = jdbc.queryForList("SELECT state FROM ma_workspace WHERE tenant_id = ? AND workspace_id = ?"
|
||||
+ " AND workspace_generation = ? AND storage_id = ? AND config_ref = ? AND policy_ref = ?"
|
||||
+ (lock ? " FOR UPDATE" : ""), String.class, request.tenantId(), request.workspaceId(), request.workspaceGeneration(),
|
||||
@@ -284,6 +526,7 @@ public final class ResidentWorkspaceRetirementStore {
|
||||
var original = jdbc.execute((ConnectionCallback<RuntimeBindingRecord>) connection ->
|
||||
bindings.lockActiveForRetirement(connection, hint));
|
||||
verifyPlan(plan, original);
|
||||
require(original.getState() == RuntimeBindingRecord.State.DRAINING);
|
||||
verifyOriginal(selected, original, true);
|
||||
plan(request, selected, operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), true));
|
||||
require(jdbc.queryForObject("SELECT state FROM ma_workspace WHERE tenant_id = ? AND workspace_id = ?",
|
||||
@@ -308,8 +551,10 @@ public final class ResidentWorkspaceRetirementStore {
|
||||
"pvcUid", registration.pvcUid(), "pvUid", registration.pvUid(), "driver", registration.driver(),
|
||||
"volumeHandle", registration.volumeHandle(), "backendDomain", registration.backendDomain(),
|
||||
"diskSerial", registration.diskSerial(), "physicalKey", registration.physicalKey()));
|
||||
require(storage.get("reservationRevision") instanceof String revision && revision.matches("[1-9][0-9]{0,18}")
|
||||
&& CsiOwnershipRevision.residentReserved(Long.parseLong(revision)));
|
||||
expected.putAll(Map.of("registrationRevision", Long.toString(registration.revision()),
|
||||
"reservationId", storage.get("reservationId"), "reservationRevision", "1"));
|
||||
"reservationId", storage.get("reservationId"), "reservationRevision", storage.get("reservationRevision")));
|
||||
require(same(expected, storage));
|
||||
var protection = selected.protection();
|
||||
require(same(Map.of("namespaceUid", protection.namespaceUid(), "policyName", protection.policyName(), "policyUid", protection.policyUid(),
|
||||
@@ -326,12 +571,24 @@ public final class ResidentWorkspaceRetirementStore {
|
||||
}
|
||||
|
||||
private Plan plan(Request request, ResidentWorkspace selected, Map<String, Object> saved) {
|
||||
Plan plan = readPlan(request, saved);
|
||||
require(digest(selected).equals(plan.mappingDigest()) && !"RELEASED".equals(saved.get("state")));
|
||||
return plan;
|
||||
}
|
||||
|
||||
private Plan readPlan(Request request, Map<String, Object> saved) {
|
||||
try {
|
||||
String encoded = (String) saved.get("request_json");
|
||||
require(encoded != null && encoded.getBytes(StandardCharsets.UTF_8).length <= 16 * 1024);
|
||||
require(encoded != null && encoded.getBytes(StandardCharsets.UTF_8).length <= PLAN_LIMIT);
|
||||
Plan plan = json.readValue(encoded, Plan.class);
|
||||
require(plan.version() == 1 && request.equals(plan.request()) && digest(selected).equals(plan.mappingDigest())
|
||||
&& digest(plan).equals(saved.get("request_digest")) && "DRAINING".equals(saved.get("state")));
|
||||
require(request.equals(plan.request()) && digest(plan).equals(saved.get("request_digest"))
|
||||
&& List.of("DRAINING", "PLATFORM_CLEANED", "RELEASED").contains(saved.get("state"))
|
||||
&& (plan.version() == 1 && plan.mapping() == null && "DRAINING".equals(saved.get("state"))
|
||||
|| plan.version() == 2 && plan.mapping() != null
|
||||
&& digest(plan.mapping()).equals(plan.mappingDigest())
|
||||
&& request.tenantId().equals(plan.mapping().registration().tenantId())
|
||||
&& request.workspaceId().equals(plan.mapping().workspaceId())
|
||||
&& request.workspaceGeneration() == plan.mapping().workspaceGeneration()));
|
||||
return plan;
|
||||
} catch (java.io.IOException failure) {
|
||||
throw unavailable();
|
||||
@@ -340,18 +597,23 @@ public final class ResidentWorkspaceRetirementStore {
|
||||
|
||||
private static void verifyPlan(Plan plan, RuntimeBindingRecord original) {
|
||||
require(original != null && original.getGeneration() == plan.runtimeGeneration()
|
||||
&& original.getState() == RuntimeBindingRecord.State.DRAINING && original.isDrainRequested()
|
||||
&& List.of(RuntimeBindingRecord.State.DRAINING, RuntimeBindingRecord.State.RELEASED).contains(original.getState())
|
||||
&& original.isDrainRequested()
|
||||
&& original.getResourceHandle() != null && Objects.equals(plan.resourceIdentity(), original.getResourceHandle().getValue().get("identity")));
|
||||
WorkspaceCsiRuntimeIdentity.verify(original);
|
||||
require(plan.mapping() == null || original.getRequest().equals(runtimeRequest(plan.mapping())));
|
||||
}
|
||||
|
||||
private static void verifyPod(RuntimeBindingRecord original, Map<String, Object> pod, String resourceVersion) {
|
||||
private static void verifyPod(RuntimeBindingRecord original, Map<String, Object> pod, String resourceVersion,
|
||||
List<String> finalizers) {
|
||||
require(pod != null && "v1".equals(pod.get("apiVersion")) && "Pod".equals(pod.get("kind")));
|
||||
var placement = map(original.getResourceHandle().getValue().get("placement"));
|
||||
var metadata = map(pod.get("metadata"));
|
||||
require(placement.get("podName").equals(metadata.get("name")) && placement.get("namespace").equals(metadata.get("namespace"))
|
||||
&& placement.get("podUid").equals(metadata.get("uid")) && resourceVersion.equals(metadata.get("resourceVersion"))
|
||||
&& metadata.get("deletionTimestamp") == null && metadata.get("deletionGracePeriodSeconds") == null
|
||||
&& empty(metadata.get("ownerReferences")) && empty(metadata.get("finalizers"))
|
||||
&& empty(metadata.get("ownerReferences"))
|
||||
&& (finalizers.isEmpty() ? empty(metadata.get("finalizers")) : finalizers.equals(metadata.get("finalizers")))
|
||||
&& digest(pod.get("spec")).equals(original.getResourceHandle().getValue().get("podSpecDigest")));
|
||||
var status = map(pod.get("status"));
|
||||
require("Running".equals(status.get("phase")) && placement.get("podIp").equals(status.get("podIP"))
|
||||
@@ -368,13 +630,26 @@ public final class ResidentWorkspaceRetirementStore {
|
||||
(row, index) -> WorkspaceOperations.work(row), operationKey(request), WORK);
|
||||
}
|
||||
|
||||
private List<Map<String, Object>> finalizerAttempts(Request request) {
|
||||
return jdbc.query("SELECT * FROM ma_operation_work WHERE operation_key = ? AND work_kind = ? ORDER BY target_key LIMIT 17",
|
||||
(row, index) -> WorkspaceOperations.work(row), operationKey(request), FINALIZER_WORK);
|
||||
}
|
||||
|
||||
private Status status(Request request, Plan plan) {
|
||||
var attempts = attempts(request);
|
||||
require(attempts.size() <= 8);
|
||||
return new Status(request.retirementId(), "DRAINING", plan.bindingId(), plan.runtimeGeneration(), attempts.stream()
|
||||
var metadata = finalizerAttempts(request);
|
||||
var saved = operations(request).find(WORKSPACE_RETIREMENT, request.retirementId(), false);
|
||||
require(attempts.size() <= 8 && metadata.size() <= 16);
|
||||
return new Status(request.retirementId(), (String) saved.get("state"), plan.bindingId(), plan.runtimeGeneration(), attempts.stream()
|
||||
.map(value -> new AttemptStatus(((Number) value.get("ordinal")).intValue(), (String) value.get("resourceVersion"),
|
||||
(String) value.get("state"), value.get("httpStatus") instanceof Number code ? code.intValue() : null,
|
||||
(String) value.get("responseSha256"))).sorted(java.util.Comparator.comparingInt(AttemptStatus::ordinal)).toList());
|
||||
(String) value.get("responseSha256"))).sorted(java.util.Comparator.comparingInt(AttemptStatus::ordinal)).toList(),
|
||||
metadata.stream().map(value -> new FinalizerAttemptStatus(((Number) value.get("ordinal")).intValue(),
|
||||
(String) value.get("action"), (String) value.get("resourceVersion"), (String) value.get("state"),
|
||||
value.get("httpStatus") instanceof Number code ? code.intValue() : null, (String) value.get("responseSha256")))
|
||||
.sorted(java.util.Comparator.comparingInt(FinalizerAttemptStatus::ordinal)).toList(),
|
||||
saved.get("cleanupReceiptDigest") instanceof String sha ? new Completion(sha, ResidentWorkspaceReclamation.DATA_POLICY,
|
||||
saved.get("releasedRevision") instanceof Number revision ? revision.longValue() : null) : null);
|
||||
}
|
||||
|
||||
private WorkspaceOperations operations(Request request) { return new WorkspaceOperations(jdbc, request.tenantId()); }
|
||||
|
||||
+24
-4
@@ -133,7 +133,7 @@ public final class ResidentWorkspaces {
|
||||
throw WorkspaceExecutionStore.unavailable();
|
||||
}
|
||||
requireRegistration(jdbc, selected);
|
||||
var originals = jdbc.query("SELECT b.binding_id, b.runtime_generation FROM ma_runtime_binding b"
|
||||
var originals = jdbc.query("SELECT b.binding_id, b.runtime_generation, b.resource_handle_json FROM ma_runtime_binding b"
|
||||
+ " JOIN ma_runtime_slot slot ON slot.active_binding_id = b.binding_id AND slot.request_key = b.request_key"
|
||||
+ " WHERE b.request_key = ? AND b.tenant_id = ? AND b.workspace_id = ? AND b.workspace_generation = ?"
|
||||
+ " AND b.canonical_cwd = ? AND b.storage_id = ? AND b.capability_digest = ?"
|
||||
@@ -141,7 +141,7 @@ public final class ResidentWorkspaces {
|
||||
+ " AND b.provisioner_kind = 'kubernetes-workspace' AND b.binding_state = 'READY'"
|
||||
+ " AND b.drain_requested = FALSE AND b.resource_handle_json IS NOT NULL"
|
||||
+ " AND b.provision_seed_ciphertext IS NOT NULL AND b.attestation_generation > 0 FOR UPDATE",
|
||||
(row, index) -> new Original(row.getString(1), row.getLong(2)), request(binding).requestKey(), tenantId,
|
||||
(row, index) -> new Original(row.getString(1), row.getLong(2), row.getString(3)), request(binding).requestKey(), tenantId,
|
||||
binding.getWorkspaceId(), Long.toString(binding.getWorkspaceGeneration()), selected.registration().mountRoot(),
|
||||
binding.getStorageId(), ResidentWorkspaceProfile.CAPABILITY_DIGEST);
|
||||
var owners = jdbc.queryForList("SELECT binding_id FROM ma_runtime_session WHERE tenant_id = ?"
|
||||
@@ -157,6 +157,7 @@ public final class ResidentWorkspaces {
|
||||
throw WorkspaceExecutionStore.unavailable();
|
||||
}
|
||||
var original = originals.getFirst();
|
||||
var storage = heldStorage(original.handle());
|
||||
if (owners.stream().anyMatch(value -> !original.id().equals(value))
|
||||
|| !jdbc.queryForList("SELECT runtime_session_id FROM ma_runtime_session WHERE tenant_id = ?"
|
||||
+ " AND harness_session_id = ? AND (binding_id <> ? OR runtime_generation <> ?"
|
||||
@@ -169,9 +170,11 @@ public final class ResidentWorkspaces {
|
||||
+ " AND runtime_generation = ? AND session_state <> 'RELEASED' FOR UPDATE", String.class,
|
||||
original.id(), original.generation()).isEmpty()
|
||||
|| jdbc.queryForObject("SELECT COUNT(*) FROM ma_storage_owner WHERE storage_key = ? AND storage_kind = 'CSI'"
|
||||
+ " AND csi_phase = 'RESERVED' AND csi_revision = 1 AND binding_id = ? AND runtime_generation = ?"
|
||||
+ " AND csi_phase = 'RESERVED' AND csi_revision = ? AND csi_reservation_id = ?"
|
||||
+ " AND binding_id = ? AND runtime_generation = ?"
|
||||
+ " AND csi_registration_key = ? AND csi_registration_revision = ? AND holder_key IS NULL"
|
||||
+ " AND runtime_session_id IS NULL", Long.class, selected.registration().physicalKey(),
|
||||
Long.parseLong((String) storage.get("reservationRevision")), storage.get("reservationId"),
|
||||
original.id(), original.generation(), selected.registration().aliasKey(), selected.registration().revision()) != 1
|
||||
|| !jdbc.queryForList("SELECT execution_call_id FROM ma_tool_execution WHERE binding_id = ?"
|
||||
+ " AND runtime_generation = ? AND execution_state <> 'SETTLED' FOR UPDATE",
|
||||
@@ -181,5 +184,22 @@ public final class ResidentWorkspaces {
|
||||
return true;
|
||||
}
|
||||
|
||||
private record Original(String id, long generation) { }
|
||||
static java.util.Map<String, Object> heldStorage(String encoded) {
|
||||
try {
|
||||
var handle = com.alibaba.qwen.code.runtimebroker.RuntimeResourceHandle.fromJson("kubernetes-workspace", 1, encoded);
|
||||
var storage = com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity.map(handle.getValue().get("storage"));
|
||||
var revision = (String) storage.get("reservationRevision");
|
||||
if (revision == null || !revision.matches("[1-9][0-9]{0,18}")
|
||||
|| !com.alibaba.qwen.code.runtimebroker.CsiOwnershipRevision.residentReserved(Long.parseLong(revision))
|
||||
|| !(storage.get("reservationId") instanceof String reservation)
|
||||
|| !java.util.UUID.fromString(reservation).toString().equals(reservation)) {
|
||||
throw WorkspaceExecutionStore.unavailable();
|
||||
}
|
||||
return storage;
|
||||
} catch (IllegalArgumentException | ClassCastException failure) {
|
||||
throw WorkspaceExecutionStore.unavailable();
|
||||
}
|
||||
}
|
||||
|
||||
private record Original(String id, long generation, String handle) { }
|
||||
}
|
||||
+43
-6
@@ -4,6 +4,7 @@ import com.alibaba.qwen.code.runtimebroker.CsiOwnershipRevision;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcCsiApplicationProgress;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcCsiFilesRetirementGuard;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.ResidentWorkspaceProfile;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaProjection;
|
||||
@@ -88,13 +89,16 @@ public final class WorkspaceCsiReservationStore {
|
||||
+ " VALUES (?, 'CSI') ON DUPLICATE KEY UPDATE storage_key = storage_key",
|
||||
registration.physicalKey());
|
||||
readReservation(registration.physicalKey());
|
||||
requireDataTenant(registration);
|
||||
});
|
||||
}
|
||||
|
||||
public Reservation inspect(WorkspaceCsiRegistration registration) {
|
||||
return transaction.execute(status -> {
|
||||
verifyRegistration(registration);
|
||||
return readReservation(registration.physicalKey());
|
||||
var reservation = readReservation(registration.physicalKey());
|
||||
requireDataTenant(registration);
|
||||
return reservation;
|
||||
});
|
||||
}
|
||||
|
||||
@@ -139,15 +143,22 @@ public final class WorkspaceCsiReservationStore {
|
||||
if (!"RELEASED".equals(current.phase())) {
|
||||
throw busy();
|
||||
}
|
||||
long revision = nextRevision(current.revision());
|
||||
requireDataTenant(registration);
|
||||
long revision;
|
||||
try {
|
||||
revision = ResidentWorkspaceProfile.selects(original.getRequest().getScope())
|
||||
? Math.addExact(current.revision(), 1) : nextRevision(current.revision());
|
||||
} catch (ArithmeticException error) {
|
||||
throw unavailable();
|
||||
}
|
||||
int changed = jdbc.update("UPDATE ma_storage_owner"
|
||||
+ " SET csi_phase = 'RESERVED', csi_revision = ?,"
|
||||
+ " csi_reservation_id = ?, csi_registration_key = ?, csi_registration_revision = ?,"
|
||||
+ " binding_id = ?, runtime_generation = ?, csi_provision_request_id = ?"
|
||||
+ " binding_id = ?, runtime_generation = ?, csi_provision_request_id = ?, tenant_id = ?"
|
||||
+ " WHERE storage_key = ? AND storage_kind = 'CSI' AND csi_phase = 'RELEASED'"
|
||||
+ " AND csi_revision = ? AND holder_key IS NULL AND runtime_session_id IS NULL",
|
||||
revision, reservationId, registration.aliasKey(), registration.revision(), original.getBindingId(),
|
||||
original.getGeneration(), requestId, registration.physicalKey(), current.revision());
|
||||
original.getGeneration(), requestId, registration.tenantId(), registration.physicalKey(), current.revision());
|
||||
if (changed != 1) {
|
||||
throw unavailable();
|
||||
}
|
||||
@@ -525,11 +536,17 @@ public final class WorkspaceCsiReservationStore {
|
||||
com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaRows.identity(row, "csi_reservation_id"), com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaRows.identity(row, "csi_registration_key"),
|
||||
row.getObject("csi_registration_revision", Long.class), com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaRows.identity(row, "binding_id"),
|
||||
row.getObject("runtime_generation", Long.class), com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaRows.identity(row, "csi_provision_request_id"));
|
||||
boolean released = "RELEASED".equals(value.phase()) && CsiOwnershipRevision.released(value.revision())
|
||||
String tenant = row.getString("tenant_id");
|
||||
boolean cyclic = tenant != null && !tenant.isBlank();
|
||||
boolean released = "RELEASED".equals(value.phase())
|
||||
&& (CsiOwnershipRevision.released(value.revision())
|
||||
|| cyclic && CsiOwnershipRevision.residentReleased(value.revision()))
|
||||
&& value.reservationId() == null && value.registrationKey() == null
|
||||
&& value.registrationRevision() == null && value.bindingId() == null
|
||||
&& value.runtimeGeneration() == null && value.provisionRequestId() == null;
|
||||
boolean reserved = ("RESERVED".equals(value.phase()) && CsiOwnershipRevision.reserved(value.revision())
|
||||
boolean reserved = ("RESERVED".equals(value.phase())
|
||||
&& (CsiOwnershipRevision.reserved(value.revision())
|
||||
|| cyclic && CsiOwnershipRevision.residentReserved(value.revision()))
|
||||
|| "DRAINING".equals(value.phase()) && CsiOwnershipRevision.draining(value.revision()))
|
||||
&& uuid(value.reservationId()) && value.registrationKey() != null
|
||||
&& value.registrationRevision() != null && value.registrationRevision() > 0
|
||||
@@ -547,6 +564,26 @@ public final class WorkspaceCsiReservationStore {
|
||||
return rows.getFirst();
|
||||
}
|
||||
|
||||
private void requireDataTenant(WorkspaceCsiRegistration registration) {
|
||||
var owners = jdbc.query("SELECT tenant_id, csi_revision, csi_registration_key FROM ma_storage_owner"
|
||||
+ " WHERE storage_key = ? AND storage_kind = 'CSI' FOR UPDATE", (row, index) -> {
|
||||
String tenant = row.getString("tenant_id");
|
||||
if (tenant == null && row.getLong("csi_revision") > 0) {
|
||||
var retained = jdbc.queryForList("SELECT tenant_id FROM ma_storage WHERE alias_key = ?"
|
||||
+ " AND storage_kind = 'CSI' AND registration_state = 'REGISTERED'", String.class,
|
||||
row.getString("csi_registration_key"));
|
||||
if (retained.size() != 1) {
|
||||
throw unavailable();
|
||||
}
|
||||
tenant = retained.getFirst();
|
||||
}
|
||||
return tenant == null || tenant.equals(registration.tenantId());
|
||||
}, registration.physicalKey());
|
||||
if (!owners.equals(List.of(true))) {
|
||||
throw unavailable();
|
||||
}
|
||||
}
|
||||
|
||||
private static long nextRevision(long revision) {
|
||||
try {
|
||||
return CsiOwnershipRevision.next(revision);
|
||||
|
||||
+14
-4
@@ -69,7 +69,8 @@ public class WorkspaceExecutionStore {
|
||||
throw unavailable();
|
||||
}
|
||||
ResidentWorkspaces.requireRegistration(jdbc, selected);
|
||||
var rows = jdbc.queryForList("SELECT rs.runtime_session_id FROM ma_runtime_session rs"
|
||||
var rows = jdbc.query("SELECT rs.runtime_session_id, b.resource_handle_json, o.csi_revision, o.csi_reservation_id"
|
||||
+ " FROM ma_runtime_session rs"
|
||||
+ " JOIN ma_runtime_binding b ON b.binding_id = rs.binding_id AND b.runtime_generation = rs.runtime_generation"
|
||||
+ " JOIN ma_runtime_slot slot ON slot.active_binding_id = b.binding_id AND slot.request_key = b.request_key"
|
||||
+ " JOIN ma_storage_owner o ON o.binding_id = b.binding_id AND o.runtime_generation = b.runtime_generation"
|
||||
@@ -79,19 +80,28 @@ public class WorkspaceExecutionStore {
|
||||
? "'ACQUIRING', 'READY', 'RELEASING', 'RELEASED'" : "'ACQUIRING', 'READY'") + ")"
|
||||
+ " AND b.request_key = ? AND b.binding_state = 'READY' AND b.drain_requested = FALSE"
|
||||
+ " AND o.storage_key = ? AND o.storage_kind = 'CSI' AND o.csi_phase = 'RESERVED'"
|
||||
+ " AND o.csi_revision = 1 AND o.csi_registration_key = ? AND o.csi_registration_revision = ?"
|
||||
+ " AND o.csi_registration_key = ? AND o.csi_registration_revision = ?"
|
||||
+ " AND o.csi_provision_request_id = b.provision_request_id AND o.holder_key IS NULL"
|
||||
+ " AND o.runtime_session_id IS NULL", String.class, session.getBindingId(), session.getRuntimeGeneration(),
|
||||
+ " AND o.runtime_session_id IS NULL", (row, index) -> new ResidentHeld(row.getString(1), row.getString(2),
|
||||
row.getLong(3), row.getString(4)), session.getBindingId(), session.getRuntimeGeneration(),
|
||||
session.getRuntimeSessionId(), session.getSession().getHarnessSessionId(), residents.request(binding).requestKey(),
|
||||
selected.registration().physicalKey(), selected.registration().aliasKey(), selected.registration().revision());
|
||||
if (rows.size() != 1 || !session.getRuntimeSessionId().equals(rows.getFirst())
|
||||
if (rows.size() != 1 || !session.getRuntimeSessionId().equals(rows.getFirst().sessionId())
|
||||
|| !jdbc.queryForList("SELECT runtime_session_id FROM ma_runtime_session WHERE binding_id = ?"
|
||||
+ " AND runtime_generation = ? AND runtime_session_id <> ? AND session_state <> 'RELEASED'",
|
||||
String.class, session.getBindingId(), session.getRuntimeGeneration(), session.getRuntimeSessionId()).isEmpty()) {
|
||||
throw busy();
|
||||
}
|
||||
var held = rows.getFirst();
|
||||
var storage = ResidentWorkspaces.heldStorage(held.handle());
|
||||
if (!Long.toString(held.revision()).equals(storage.get("reservationRevision"))
|
||||
|| !held.reservationId().equals(storage.get("reservationId"))) {
|
||||
throw unavailable();
|
||||
}
|
||||
}
|
||||
|
||||
private record ResidentHeld(String sessionId, String handle, long revision, String reservationId) { }
|
||||
|
||||
public void authorize(SessionRecord session) {
|
||||
authorizePassiveAttachment(session);
|
||||
if (storageGuard != null && residents.find(session.workspace()) == null) {
|
||||
|
||||
+5
-2
@@ -173,7 +173,8 @@ final class WorkspaceOperations {
|
||||
case WORKSPACE_RECOVERY -> java.util.Set.of("CAPTURING", "VERIFYING", "SEALED", "VERIFIED", "INVALIDATED");
|
||||
case WORKSPACE_OPERATOR_RECOVERY -> java.util.Set.of("PREPARED", "ATTESTED", "COMPLETED");
|
||||
case STORAGE_MIGRATION -> java.util.Set.of("RETIRING", "RETIRED", "PREPARING", "PREPARED", "COMPLETED", "ABORTED", "INVALIDATED");
|
||||
case CSI_RETIREMENT, WORKSPACE_RETIREMENT -> java.util.Set.of("DRAINING");
|
||||
case CSI_RETIREMENT -> java.util.Set.of("DRAINING");
|
||||
case WORKSPACE_RETIREMENT -> java.util.Set.of("DRAINING", "PLATFORM_CLEANED", "RELEASED");
|
||||
case CHILD_WORKSPACE -> java.util.Set.of(ChildWorkspaceStore.PREPARING, ChildWorkspaceStore.READY,
|
||||
ChildWorkspaceStore.MERGING, ChildWorkspaceStore.APPLYING, ChildWorkspaceStore.APPLIED,
|
||||
ChildWorkspaceStore.MERGED, ChildWorkspaceStore.CONFLICTED, ChildWorkspaceStore.DISCARDING,
|
||||
@@ -198,7 +199,9 @@ final class WorkspaceOperations {
|
||||
|| "RETIRED".equals(from) && "PREPARING".equals(to)
|
||||
|| "PREPARING".equals(from) && "PREPARED".equals(to)
|
||||
|| "PREPARED".equals(from) && "COMPLETED".equals(to));
|
||||
case CSI_RETIREMENT, WORKSPACE_RETIREMENT, CHILD_WORKSPACE -> false;
|
||||
case WORKSPACE_RETIREMENT -> "DRAINING".equals(from) && "PLATFORM_CLEANED".equals(to)
|
||||
|| "PLATFORM_CLEANED".equals(from) && "RELEASED".equals(to);
|
||||
case CSI_RETIREMENT, CHILD_WORKSPACE -> false;
|
||||
};
|
||||
if (!allowed) {
|
||||
throw new IllegalStateException("Workspace operation transition conflicts");
|
||||
|
||||
+3
-1
@@ -325,7 +325,9 @@ public final class WorkspaceCsiRuntimeProvisioner implements RuntimeProvisioner
|
||||
RuntimeBindingRecord.State.RECOVERY_BLOCKED).contains(original.getState())
|
||||
&& (fresh && original.getResourceHandle() == null || handle.equals(original.getResourceHandle())));
|
||||
var expectedStorage = storageTuple(reserved);
|
||||
expectedStorage.put("reservationRevision", "1");
|
||||
require(com.alibaba.qwen.code.runtimebroker.ResidentWorkspaceProfile.selects(request.getScope())
|
||||
? com.alibaba.qwen.code.runtimebroker.CsiOwnershipRevision.residentReserved(reserved.revision())
|
||||
: com.alibaba.qwen.code.runtimebroker.CsiOwnershipRevision.reserved(reserved.revision()));
|
||||
require(same(expectedStorage, value.get("storage")));
|
||||
checkObservation(request, seed, handle);
|
||||
var receipt = attest(request, seed, placement, WorkspaceCsiRuntimeIdentity.boot(request, seed, handle));
|
||||
|
||||
+8
@@ -16,6 +16,14 @@ public final class CsiOwnershipRevision {
|
||||
return revision == 2;
|
||||
}
|
||||
|
||||
public static boolean residentReleased(long revision) {
|
||||
return revision >= 0 && revision % 2 == 0;
|
||||
}
|
||||
|
||||
public static boolean residentReserved(long revision) {
|
||||
return revision > 0 && revision % 2 == 1;
|
||||
}
|
||||
|
||||
public static long next(long revision) {
|
||||
if (revision < 0 || revision > 1) {
|
||||
throw new IllegalArgumentException("Only the original CSI ownership transition is available.");
|
||||
|
||||
+31
@@ -2137,6 +2137,37 @@ public final class JdbcRuntimeBindingRepository
|
||||
return selectById(connection, BrokerValues.requireId(bindingId, "bindingId"), true, 10);
|
||||
}
|
||||
|
||||
public RuntimeBindingRecord releaseResidentForRetirement(Connection connection,
|
||||
RuntimeBindingRecord expected) throws SQLException {
|
||||
RuntimeBindingRecord current = lockActiveForRetirement(connection, expected);
|
||||
if (!ResidentWorkspaceProfile.selects(current.getRequest().getScope())
|
||||
|| current.getState() != RuntimeBindingRecord.State.DRAINING || !current.isDrainRequested()
|
||||
|| current.getOperationOwner() != null || current.getOperationLeaseUntil() != null
|
||||
|| current.getVersion() != expected.getVersion() || !current.sameOperation(expected)
|
||||
|| !current.hasSameLease(expected.getLease())
|
||||
|| !Objects.equals(current.getResourceHandle(), expected.getResourceHandle())
|
||||
|| current.getAttestationGeneration() != expected.getAttestationGeneration()) {
|
||||
throw new IllegalStateException("Original resident retirement is unavailable");
|
||||
}
|
||||
WorkspaceCsiRuntimeIdentity.verify(current);
|
||||
RuntimeBindingRecord released = current.withState(RuntimeBindingRecord.State.RELEASED,
|
||||
current.getLease(), JdbcRepositorySupport.databaseNow(connection))
|
||||
.withOperation(null, null, current.getOperationGeneration() + 1)
|
||||
.withVersion(current.getVersion() + 1);
|
||||
updateBinding(connection, released);
|
||||
try (var statement = connection.prepareStatement("UPDATE ma_runtime_slot SET active_binding_id = NULL"
|
||||
+ " WHERE request_key = ? AND active_binding_id = ? AND last_generation = ?")) {
|
||||
statement.setQueryTimeout(10);
|
||||
statement.setString(1, JdbcRepositorySupport.requestKey(current.getRequest()));
|
||||
JdbcRepositorySupport.setStringParameter(statement, 2, current.getBindingId());
|
||||
statement.setLong(3, current.getGeneration());
|
||||
if (statement.executeUpdate() != 1) {
|
||||
throw new IllegalStateException("Original resident slot is unavailable");
|
||||
}
|
||||
}
|
||||
return released;
|
||||
}
|
||||
|
||||
public RuntimeBindingRecord findByIdInReadOnlySnapshot(Connection connection,
|
||||
String bindingId) throws SQLException {
|
||||
if (connection == null || connection.getAutoCommit() || !connection.isReadOnly()
|
||||
|
||||
+2
-1
@@ -104,7 +104,8 @@ public final class ManagedSchemaContract {
|
||||
RUNTIME_BINDING,
|
||||
CSI_WORKER_ACK,
|
||||
RECOVERY_EVIDENCE,
|
||||
POD_DELETE_ATTEMPT
|
||||
POD_DELETE_ATTEMPT,
|
||||
POD_FINALIZER_ATTEMPT
|
||||
}
|
||||
|
||||
public enum ResourceOwnerKind {
|
||||
|
||||
+69
-2
@@ -3,6 +3,7 @@ package com.alibaba.qwen.code.runtimebroker.provisioning;
|
||||
import com.alibaba.qwen.code.runtimebroker.BrokerValues;
|
||||
import com.alibaba.qwen.code.runtimebroker.JsonCodec;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity;
|
||||
import com.alibaba.qwen.code.runtimebroker.transport.HttpRuntimeTransport;
|
||||
|
||||
import javax.net.ssl.SSLContext;
|
||||
@@ -10,6 +11,7 @@ import javax.net.ssl.TrustManagerFactory;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.net.URLEncoder;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.nio.ByteBuffer;
|
||||
@@ -23,6 +25,9 @@ import java.security.KeyStore;
|
||||
import java.security.cert.CertificateFactory;
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.List;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashSet;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
@@ -95,6 +100,68 @@ public final class KubernetesHttpRuntimeClient implements KubernetesRuntimeClien
|
||||
}
|
||||
|
||||
public CompletionStage<DeleteResponse> deletePod(String namespace, String podName, byte[] deleteOptions) {
|
||||
return mutatePod(namespace, podName, "DELETE", "application/json", deleteOptions);
|
||||
}
|
||||
|
||||
public CompletionStage<DeleteResponse> patchPodFinalizers(String namespace, String podName, byte[] patch) {
|
||||
return mutatePod(namespace, podName, "PATCH", "application/json-patch+json", patch);
|
||||
}
|
||||
|
||||
public CompletionStage<Map<String, Object>> getNodeLease(String nodeName) {
|
||||
dnsSubdomain(nodeName);
|
||||
return exchange("/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/" + nodeName, null);
|
||||
}
|
||||
|
||||
public CompletionStage<List<Map<String, Object>>> listPersistentVolumes() {
|
||||
return list("/api/v1/persistentvolumes", "PersistentVolumeList");
|
||||
}
|
||||
|
||||
public CompletionStage<List<Map<String, Object>>> listVolumeAttachments() {
|
||||
return list("/apis/storage.k8s.io/v1/volumeattachments", "VolumeAttachmentList");
|
||||
}
|
||||
|
||||
public CompletionStage<List<Map<String, Object>>> listPods(String namespace) {
|
||||
dnsLabel(namespace);
|
||||
return list("/api/v1/namespaces/" + namespace + "/pods", "PodList");
|
||||
}
|
||||
|
||||
private CompletionStage<List<Map<String, Object>>> list(String path, String kind) {
|
||||
return listPage(path, kind, "", null, new ArrayList<>(), new HashSet<>(), new HashSet<>());
|
||||
}
|
||||
|
||||
private CompletionStage<List<Map<String, Object>>> listPage(String path, String kind, String token,
|
||||
String revision, List<Map<String, Object>> items, Set<String> identities, Set<String> tokens) {
|
||||
if (tokens.size() >= 8 || !tokens.add(token)) {
|
||||
return CompletableFuture.failedFuture(failure(502, false));
|
||||
}
|
||||
return exchange(path + "?limit=128&continue=" + URLEncoder.encode(token, StandardCharsets.UTF_8), null).thenCompose(page -> {
|
||||
if (page == null || !kind.equals(page.get("kind")) || !(page.get("metadata") instanceof Map<?, ?> metadata)
|
||||
|| !(metadata.get("resourceVersion") instanceof String observed) || observed.isBlank()
|
||||
|| revision != null && !revision.equals(observed) || !(page.get("items") instanceof List<?> values)) {
|
||||
throw failure(502, false);
|
||||
}
|
||||
for (Object value : values) {
|
||||
var item = WorkspaceCsiRuntimeIdentity.map(value);
|
||||
var identity = WorkspaceCsiRuntimeIdentity.map(item.get("metadata"));
|
||||
if (!(identity.get("uid") instanceof String uid) || uid.isBlank() || !identities.add(uid)
|
||||
|| items.size() >= 1024) {
|
||||
throw failure(502, false);
|
||||
}
|
||||
items.add(item);
|
||||
}
|
||||
Object next = metadata.get("continue");
|
||||
if (next == null || "".equals(next)) {
|
||||
return CompletableFuture.completedFuture(List.copyOf(items));
|
||||
}
|
||||
if (!(next instanceof String continuation) || continuation.length() > 16 * 1024) {
|
||||
throw failure(502, false);
|
||||
}
|
||||
return listPage(path, kind, continuation, observed, items, identities, tokens);
|
||||
});
|
||||
}
|
||||
|
||||
private CompletionStage<DeleteResponse> mutatePod(String namespace, String podName, String method,
|
||||
String contentType, byte[] deleteOptions) {
|
||||
dnsLabel(namespace);
|
||||
dnsSubdomain(podName);
|
||||
if (deleteOptions == null || deleteOptions.length == 0 || deleteOptions.length > DELETE_REQUEST_LIMIT) {
|
||||
@@ -103,8 +170,8 @@ public final class KubernetesHttpRuntimeClient implements KubernetesRuntimeClien
|
||||
HttpRequest request;
|
||||
try {
|
||||
request = requestBuilder("/api/v1/namespaces/" + namespace + "/pods/" + podName)
|
||||
.header("Content-Type", "application/json")
|
||||
.method("DELETE", HttpRequest.BodyPublishers.ofByteArray(deleteOptions.clone())).build();
|
||||
.header("Content-Type", contentType)
|
||||
.method(method, HttpRequest.BodyPublishers.ofByteArray(deleteOptions.clone())).build();
|
||||
} catch (IOException error) {
|
||||
return CompletableFuture.failedFuture(error);
|
||||
}
|
||||
|
||||
+1
-1
@@ -736,7 +736,7 @@ CREATE TABLE ma_operation_work (
|
||||
gmt_create BIGINT UNSIGNED NOT NULL,
|
||||
gmt_modified BIGINT UNSIGNED NOT NULL,
|
||||
PRIMARY KEY (operation_key, work_kind, target_key),
|
||||
CONSTRAINT ck_ma_work_kind CHECK ((CASE work_kind WHEN 'SESSION' THEN 7 WHEN 'RESOURCE' THEN 8 WHEN 'EXECUTION_ACK' THEN 13 WHEN 'RUNTIME_BINDING' THEN 15 WHEN 'CSI_WORKER_ACK' THEN 14 WHEN 'RECOVERY_EVIDENCE' THEN 17 WHEN 'POD_DELETE_ATTEMPT' THEN 18 ELSE -1 END = CHAR_LENGTH(work_kind))),
|
||||
CONSTRAINT ck_ma_work_kind CHECK ((CASE work_kind WHEN 'SESSION' THEN 7 WHEN 'RESOURCE' THEN 8 WHEN 'EXECUTION_ACK' THEN 13 WHEN 'RUNTIME_BINDING' THEN 15 WHEN 'CSI_WORKER_ACK' THEN 14 WHEN 'RECOVERY_EVIDENCE' THEN 17 WHEN 'POD_DELETE_ATTEMPT' THEN 18 WHEN 'POD_FINALIZER_ATTEMPT' THEN 21 ELSE -1 END = CHAR_LENGTH(work_kind))),
|
||||
CONSTRAINT fk_ma_work_operation FOREIGN KEY (operation_key) REFERENCES ma_operation (operation_key),
|
||||
CONSTRAINT fk_ma_work_evidence FOREIGN KEY (evidence_resource_key) REFERENCES ma_resource (resource_key)
|
||||
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
|
||||
|
||||
+72
-9
@@ -50,31 +50,36 @@ final class ResidentRetirementFixture {
|
||||
final ResidentWorkspaceRetirementStore.Request request;
|
||||
final RuntimeProvisionRequest runtimeRequest;
|
||||
final Map<String, Map<String, Object>> objects = new LinkedHashMap<>();
|
||||
final List<Map<String, Object>> attachments = new java.util.ArrayList<>();
|
||||
RuntimeBindingRecord original;
|
||||
private final ManagedAgentProperties properties = new ManagedAgentProperties();
|
||||
|
||||
ResidentRetirementFixture(DataSource source) {
|
||||
this(source, "workspace");
|
||||
}
|
||||
|
||||
ResidentRetirementFixture(DataSource source, String workspaceId) {
|
||||
this.source = source;
|
||||
ManagedSchema.initialize(source);
|
||||
jdbc = new ManagedSchemaJdbcTemplate(source);
|
||||
manager = new DataSourceTransactionManager(source);
|
||||
bindings = new JdbcRuntimeBindingRepository(source, new AesGcmSecretProtector("test", new byte[32]));
|
||||
mapping = new ResidentWorkspace("workspace", 1,
|
||||
mapping = new ResidentWorkspace(workspaceId, 1,
|
||||
new WorkspaceCsiRegistration("tenant", "storage", "cluster", "runtime", "claim", "pvc-uid",
|
||||
"volume", "pv-uid", "diskplugin.csi.alibabacloud.com", "handle", "backend", "serial", "/workspace", 1),
|
||||
new WorkspaceCsiResourceGuard.ProtectionIdentity(UID, "protect", UID, 1, "protect-binding", UID, 1),
|
||||
"worker@sha256:" + "a".repeat(64), List.of("node", "/worker/cli.js"), List.of());
|
||||
properties.getHarness().setWorkspaceFilesEnabled(true);
|
||||
properties.getRuntimeBroker().setResidentWorkspaces(List.of(mapping));
|
||||
request = new ResidentWorkspaceRetirementStore.Request("tenant", "workspace", 1, "owner", UUID.randomUUID().toString());
|
||||
request = new ResidentWorkspaceRetirementStore.Request("tenant", workspaceId, 1, "owner", UUID.randomUUID().toString());
|
||||
runtimeRequest = new RuntimeProvisionRequest(ResidentWorkspaces.scope(mapping), null, "kubernetes-workspace", "storage");
|
||||
storage = new WorkspaceCsiReservationStore(jdbc, manager, JSON);
|
||||
storage.register(mapping.registration());
|
||||
jdbc.update("INSERT INTO ma_workspace (tenant_id, workspace_id, workspace_generation, storage_id, display_name,"
|
||||
+ " config_ref, policy_ref, state) VALUES ('tenant', 'workspace', 1, 'storage', 'resident', ?, ?, 'ACTIVE')",
|
||||
WorkspaceExecutionProfile.CONFIG_REF, WorkspaceExecutionProfile.POLICY_REF);
|
||||
+ " config_ref, policy_ref, state) VALUES ('tenant', ?, 1, 'storage', 'resident', ?, ?, 'ACTIVE')",
|
||||
workspaceId, WorkspaceExecutionProfile.CONFIG_REF, WorkspaceExecutionProfile.POLICY_REF);
|
||||
jdbc.update("INSERT INTO ma_workspace_access (tenant_id, workspace_id, actor_id, role)"
|
||||
+ " VALUES ('tenant', 'workspace', ?, 'OWNER')", ManagedWorkspaceRegistry.actorKey("tenant", "owner"));
|
||||
+ " VALUES ('tenant', ?, ?, 'OWNER')", workspaceId, ManagedWorkspaceRegistry.actorKey("tenant", "owner"));
|
||||
}
|
||||
|
||||
ResidentWorkspaceRetirementStore store() {
|
||||
@@ -85,7 +90,7 @@ final class ResidentRetirementFixture {
|
||||
var store = new ManagedAgentStore(jdbc, JSON, Clock.systemUTC(), mock(CommittedEventPublisher.class),
|
||||
new ManagedWorkspaceRegistry(jdbc), properties);
|
||||
return new TransactionTemplate(manager).execute(status -> store.insertWorkspaceSessionCommand("tenant", "owner", key,
|
||||
"digest-" + key, "qwen-code", null, "Session", List.of(), null, new WorkspaceSelection("workspace", ".")));
|
||||
"digest-" + key, "qwen-code", null, "Session", List.of(), null, new WorkspaceSelection(mapping.workspaceId(), ".")));
|
||||
}
|
||||
|
||||
RuntimeBindingRecord ready() {
|
||||
@@ -102,11 +107,13 @@ final class ResidentRetirementFixture {
|
||||
"backendDomain", registration.backendDomain(), "driver", registration.driver(), "volumeHandle", registration.volumeHandle(),
|
||||
"diskSerial", registration.diskSerial(), "physicalKey", registration.physicalKey(), "pvcUid", registration.pvcUid(),
|
||||
"pvUid", registration.pvUid(), "registrationRevision", "1"));
|
||||
tuple.putAll(Map.of("reservationId", reserved.reservationId(), "reservationRevision", "1"));
|
||||
tuple.putAll(Map.of("reservationId", reserved.reservationId(), "reservationRevision", Long.toString(reserved.revision())));
|
||||
String name = WorkspaceCsiRuntimeIdentity.name(seed);
|
||||
String podUid = "workspace".equals(mapping.workspaceId()) ? UID
|
||||
: UUID.nameUUIDFromBytes(mapping.workspaceId().getBytes(java.nio.charset.StandardCharsets.UTF_8)).toString();
|
||||
var placement = new LinkedHashMap<String, Object>();
|
||||
placement.putAll(Map.of("clusterDomain", "cluster", "namespace", "runtime", "namespaceUid", UID,
|
||||
"podName", name, "podUid", UID, "nodeName", "node-a", "nodeUid", UID));
|
||||
"podName", name, "podUid", podUid, "nodeName", "node-a", "nodeUid", UID));
|
||||
placement.putAll(Map.of("containerName", "runtime", "containerId", "containerd://" + "b".repeat(64),
|
||||
"image", mapping.image(), "imageId", "sha256:" + "a".repeat(64), "podIp", "10.4.0.8",
|
||||
"secretName", name, "secretUid", UID));
|
||||
@@ -128,7 +135,7 @@ final class ResidentRetirementFixture {
|
||||
new RuntimeResourceHandle("kubernetes-workspace", 1, identity), Instant.now(), Instant.now()));
|
||||
original = bindings.releaseOperation(attested.getBindingId(), "setup", attested.getOperationGeneration());
|
||||
WorkspaceCsiRuntimeIdentity.verify(original);
|
||||
objects.put("pods", object("Pod", name, "runtime", UID, spec));
|
||||
objects.put("pods", object("Pod", name, "runtime", podUid, spec));
|
||||
objects.get("pods").put("status", copy(Map.of("phase", "Running", "podIP", "10.4.0.8", "containerStatuses",
|
||||
List.of(Map.of("name", "runtime", "containerID", placement.get("containerId"), "imageID", placement.get("imageId"),
|
||||
"restartCount", 0, "ready", true, "state", Map.of("running", Map.of()))))));
|
||||
@@ -146,9 +153,65 @@ final class ResidentRetirementFixture {
|
||||
CompletableFuture.completedFuture(objects.get(call.<String>getArgument(0))));
|
||||
when(api.deletePod(anyString(), anyString(), any(byte[].class))).thenReturn(
|
||||
CompletableFuture.completedFuture(new KubernetesHttpRuntimeClient.DeleteResponse(200, new byte[0], true)));
|
||||
when(api.getNodeLease(anyString())).thenAnswer(call -> CompletableFuture.completedFuture(objects.get("leases")));
|
||||
when(api.listPersistentVolumes()).thenAnswer(call -> CompletableFuture.completedFuture(List.of(objects.get("persistentvolumes"))));
|
||||
when(api.listVolumeAttachments()).thenAnswer(call -> CompletableFuture.completedFuture(List.copyOf(attachments)));
|
||||
when(api.listPods(anyString())).thenAnswer(call -> CompletableFuture.completedFuture(objects.containsKey("pods")
|
||||
? List.of(objects.get("pods")) : List.of()));
|
||||
when(api.patchPodFinalizers(anyString(), anyString(), any(byte[].class))).thenAnswer(call -> {
|
||||
var patch = JSON.readTree(call.<byte[]>getArgument(2));
|
||||
var finalizers = JSON.convertValue(patch.get(patch.size() - 1).get("value"), new TypeReference<List<String>>() { });
|
||||
var pod = objects.get("pods");
|
||||
var metadata = map(pod.get("metadata"));
|
||||
if (!patch.get(0).get("value").textValue().equals(metadata.get("uid"))
|
||||
|| !patch.get(1).get("value").textValue().equals(metadata.get("resourceVersion"))) {
|
||||
return CompletableFuture.completedFuture(new KubernetesHttpRuntimeClient.DeleteResponse(409, new byte[0], true));
|
||||
}
|
||||
metadata.put("finalizers", finalizers);
|
||||
metadata.put("resourceVersion", Long.toString(Long.parseLong((String) metadata.get("resourceVersion")) + 1));
|
||||
if (finalizers.isEmpty() && metadata.containsKey("deletionTimestamp")) {
|
||||
objects.remove("pods");
|
||||
}
|
||||
return CompletableFuture.completedFuture(new KubernetesHttpRuntimeClient.DeleteResponse(200, new byte[0], true));
|
||||
});
|
||||
return api;
|
||||
}
|
||||
|
||||
void preparePlatform() {
|
||||
var now = Instant.now();
|
||||
String volume = "kubernetes.io/csi/" + mapping.registration().driver() + "^handle";
|
||||
objects.get("nodes").put("status", copy(Map.of("nodeInfo", Map.of("bootID", "boot-a"),
|
||||
"conditions", List.of(Map.of("type", "Ready", "status", "True", "lastHeartbeatTime", now.minusSeconds(10).toString())),
|
||||
"volumesInUse", List.of(volume), "volumesAttached", List.of(Map.of("name", volume)))));
|
||||
objects.put("leases", object("Lease", "node-a", "kube-node-lease", "lease-uid", Map.of("holderIdentity", "node-a",
|
||||
"leaseDurationSeconds", 40, "renewTime", now.minusSeconds(2).toString())));
|
||||
map(objects.get("leases").get("metadata")).put("ownerReferences", List.of(Map.of("apiVersion", "v1", "kind", "Node", "name", "node-a", "uid", UID)));
|
||||
var attachment = object("VolumeAttachment", "attachment", null, "attachment-uid", Map.of("attacher", mapping.registration().driver(),
|
||||
"source", Map.of("persistentVolumeName", "volume"), "nodeName", "node-a"));
|
||||
attachment.put("status", Map.of("attached", true));
|
||||
attachments.add(attachment);
|
||||
}
|
||||
|
||||
void completePlatform() {
|
||||
var now = Instant.now();
|
||||
var pod = objects.get("pods");
|
||||
map(pod.get("metadata")).put("deletionTimestamp", now.minusSeconds(3).toString());
|
||||
var status = map(pod.get("status"));
|
||||
status.put("phase", "Failed");
|
||||
var container = map(((List<?>) status.get("containerStatuses")).getFirst());
|
||||
container.put("ready", false);
|
||||
container.put("state", Map.of("terminated", Map.of("containerID", container.get("containerID"),
|
||||
"finishedAt", now.minusSeconds(2).toString(), "reason", "Error", "exitCode", 143)));
|
||||
var nodeStatus = map(objects.get("nodes").get("status"));
|
||||
nodeStatus.put("volumesInUse", List.of());
|
||||
nodeStatus.put("volumesAttached", List.of());
|
||||
nodeStatus.put("conditions", List.of(Map.of("type", "Ready", "status", "True", "lastHeartbeatTime", now.minusSeconds(1).toString())));
|
||||
map(objects.get("nodes").get("metadata")).put("resourceVersion", "20");
|
||||
map(objects.get("leases").get("metadata")).put("resourceVersion", "20");
|
||||
map(objects.get("leases").get("spec")).put("renewTime", now.minusSeconds(1).toString());
|
||||
attachments.clear();
|
||||
}
|
||||
|
||||
void resourceVersion(String version) {
|
||||
map(objects.get("pods").get("metadata")).put("resourceVersion", version);
|
||||
}
|
||||
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
package com.alibaba.qwen.code.managedagent.persistence;
|
||||
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import static com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity.map;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.ArgumentMatchers.anyString;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class ResidentWorkspaceGraceDeadlineTest {
|
||||
private ResidentRetirementFixture fixture;
|
||||
|
||||
@BeforeEach
|
||||
void setUp() {
|
||||
fixture = new ResidentRetirementFixture(new DriverManagerDataSource(
|
||||
"jdbc:h2:mem:resident-grace-" + UUID.randomUUID()
|
||||
+ ";MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE", "sa", ""));
|
||||
fixture.ready();
|
||||
}
|
||||
|
||||
@Test
|
||||
void normalCleanupCanFinishBeforeTheGraceDeadline() {
|
||||
var api = prepared();
|
||||
fixture.completePlatform();
|
||||
var metadata = map(fixture.objects.get("pods").get("metadata"));
|
||||
metadata.put("deletionTimestamp", Instant.now().plusSeconds(27).toString());
|
||||
metadata.put("deletionGracePeriodSeconds", 30);
|
||||
RuntimeException refusal = null;
|
||||
try {
|
||||
fixture.store().reclaim(fixture.request, api);
|
||||
} catch (RuntimeException failure) {
|
||||
refusal = failure;
|
||||
}
|
||||
if (refusal != null) {
|
||||
assertThat(fixture.store().inspect(fixture.request).state()).isEqualTo("DRAINING");
|
||||
assertThat(fixture.storage.inspect(fixture.mapping.registration()).phase()).isEqualTo("RESERVED");
|
||||
}
|
||||
assertThat(refusal).as("Original container and volume cleanup completed before the future grace deadline").isNull();
|
||||
assertThat(fixture.store().inspect(fixture.request).state()).isEqualTo("RELEASED");
|
||||
}
|
||||
|
||||
@Test
|
||||
void savedCleanupAllowsTheApiToShortenTheDeletionTimestamp() {
|
||||
var api = prepared();
|
||||
fixture.completePlatform();
|
||||
var metadata = map(fixture.objects.get("pods").get("metadata"));
|
||||
metadata.put("deletionTimestamp", Instant.now().plusSeconds(27).toString());
|
||||
metadata.put("deletionGracePeriodSeconds", 30);
|
||||
var podReads = new AtomicInteger();
|
||||
when(api.get(eq("pods"), anyString(), anyString())).thenAnswer(call -> {
|
||||
if (podReads.incrementAndGet() == 2) {
|
||||
metadata.put("deletionTimestamp", Instant.now().toString());
|
||||
metadata.put("deletionGracePeriodSeconds", 0);
|
||||
metadata.put("resourceVersion", "12");
|
||||
}
|
||||
return CompletableFuture.completedFuture(fixture.objects.get("pods"));
|
||||
});
|
||||
RuntimeException refusal = null;
|
||||
try {
|
||||
fixture.store().reclaim(fixture.request, api);
|
||||
} catch (RuntimeException failure) {
|
||||
refusal = failure;
|
||||
}
|
||||
if (refusal != null) {
|
||||
assertThat(podReads).hasValue(2);
|
||||
assertThat(fixture.store().inspect(fixture.request).state()).isEqualTo("PLATFORM_CLEANED");
|
||||
assertThat(fixture.storage.inspect(fixture.mapping.registration()).phase()).isEqualTo("RESERVED");
|
||||
}
|
||||
assertThat(refusal).as("Saved cleanup receipt remains valid after same-Pod grace shortening").isNull();
|
||||
assertThat(fixture.store().inspect(fixture.request).state()).isEqualTo("RELEASED");
|
||||
}
|
||||
|
||||
private com.alibaba.qwen.code.runtimebroker.provisioning.KubernetesHttpRuntimeClient prepared() {
|
||||
fixture.preparePlatform();
|
||||
fixture.store().begin(fixture.request);
|
||||
var api = fixture.api();
|
||||
fixture.store().prepareReclamation(fixture.request, "10", api);
|
||||
fixture.store().deletePod(fixture.request, "11", api);
|
||||
return api;
|
||||
}
|
||||
}
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
package com.alibaba.qwen.code.managedagent.persistence;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties.RuntimeBroker.ResidentWorkspace;
|
||||
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.provisioning.KubernetesHttpRuntimeClient;
|
||||
import com.alibaba.qwen.code.runtimebroker.security.AesGcmSecretProtector;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.junit.jupiter.api.Timeout;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
import org.springframework.transaction.support.DefaultTransactionStatus;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.anyString;
|
||||
import static org.mockito.Mockito.doReturn;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.verifyNoInteractions;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class ResidentWorkspaceReclamationProcessMySqlIT {
|
||||
private static final ObjectMapper JSON = new ObjectMapper();
|
||||
@TempDir
|
||||
private Path directory;
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"before-receipt", "after-receipt", "before", "after"})
|
||||
@Timeout(45)
|
||||
void killedReleaseProcessReadsEitherWholeHeldStateOrWholeReleasedState(String cut) throws Exception {
|
||||
try (var database = new IsolatedManagedSchemaMySqlTestDatabase()) {
|
||||
var fixture = new ResidentRetirementFixture(database.source());
|
||||
fixture.ready();
|
||||
fixture.preparePlatform();
|
||||
fixture.store().begin(fixture.request);
|
||||
var api = fixture.api();
|
||||
fixture.store().prepareReclamation(fixture.request, "10", api);
|
||||
fixture.store().deletePod(fixture.request, "11", api);
|
||||
fixture.completePlatform();
|
||||
doReturn(CompletableFuture.completedFuture(new KubernetesHttpRuntimeClient.DeleteResponse(503, new byte[0], true)))
|
||||
.when(api).patchPodFinalizers(anyString(), anyString(), any(byte[].class));
|
||||
if (!cut.endsWith("receipt")) {
|
||||
assertThat(fixture.store().reclaim(fixture.request, api).state()).isEqualTo("PLATFORM_CLEANED");
|
||||
}
|
||||
Path input = directory.resolve("request.json");
|
||||
JSON.writeValue(input.toFile(), Map.of("mapping", fixture.mapping, "request", fixture.request, "cut", cut,
|
||||
"objects", fixture.objects));
|
||||
Path log = directory.resolve("child.log");
|
||||
var command = new ProcessBuilder(Path.of(System.getProperty("java.home"), "bin", "java").toString(), "-cp",
|
||||
System.getProperty("surefire.test.class.path", System.getProperty("java.class.path")),
|
||||
ResidentWorkspaceReclamationProcessMySqlIT.class.getName(), input.toString());
|
||||
command.environment().put("RECLAMATION_MYSQL_URL", database.url());
|
||||
command.environment().put("RECLAMATION_MYSQL_USER", System.getProperty("mysql.user"));
|
||||
command.environment().put("RECLAMATION_MYSQL_PASSWORD", System.getProperty("mysql.password", ""));
|
||||
var child = command.redirectErrorStream(true).redirectOutput(log.toFile()).start();
|
||||
try {
|
||||
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(20);
|
||||
while (!Files.exists(directory.resolve("cut")) && child.isAlive() && System.nanoTime() < deadline) {
|
||||
Thread.sleep(25);
|
||||
}
|
||||
assertThat(Files.exists(directory.resolve("cut"))).withFailMessage("Child did not reach commit cut: %s", Files.readString(log)).isTrue();
|
||||
child.destroyForcibly();
|
||||
assertThat(child.waitFor(5, TimeUnit.SECONDS)).isTrue();
|
||||
var state = fixture.store().inspect(fixture.request);
|
||||
assertThat(state.state()).isEqualTo(cut.equals("before-receipt") ? "DRAINING"
|
||||
: cut.equals("after") ? "RELEASED" : "PLATFORM_CLEANED");
|
||||
assertThat(fixture.storage.inspect(fixture.mapping.registration()).phase())
|
||||
.isEqualTo(cut.equals("after") ? "RELEASED" : "RESERVED");
|
||||
assertThat(fixture.bindings.findById(fixture.original.getBindingId()).getState())
|
||||
.isEqualTo(cut.equals("after") ? RuntimeBindingRecord.State.RELEASED : RuntimeBindingRecord.State.DRAINING);
|
||||
if (!cut.equals("after")) {
|
||||
assertThat(fixture.store().reclaim(fixture.request, fixture.api()).state()).isEqualTo("RELEASED");
|
||||
}
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT active_binding_id FROM ma_runtime_slot", String.class)).isNull();
|
||||
assertThat(fixture.storage.inspect(fixture.mapping.registration()).revision()).isEqualTo(2);
|
||||
var unused = mock(KubernetesHttpRuntimeClient.class);
|
||||
assertThat(fixture.store().reclaim(fixture.request, unused).state()).isEqualTo("RELEASED");
|
||||
verifyNoInteractions(unused);
|
||||
} finally {
|
||||
if (child.isAlive()) {
|
||||
child.destroyForcibly().waitFor(5, TimeUnit.SECONDS);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public static void main(String[] args) throws Exception {
|
||||
Path input = Path.of(args[0]);
|
||||
var document = JSON.readTree(input.toFile());
|
||||
var selected = JSON.treeToValue(document.get("mapping"), ResidentWorkspace.class);
|
||||
var request = JSON.treeToValue(document.get("request"), ResidentWorkspaceRetirementStore.Request.class);
|
||||
String cut = document.get("cut").textValue();
|
||||
var source = new DriverManagerDataSource(System.getenv("RECLAMATION_MYSQL_URL"),
|
||||
System.getenv("RECLAMATION_MYSQL_USER"), System.getenv("RECLAMATION_MYSQL_PASSWORD"));
|
||||
var jdbc = new ManagedSchemaJdbcTemplate(source);
|
||||
var manager = new DataSourceTransactionManager(source) {
|
||||
@Override
|
||||
protected void doCommit(DefaultTransactionStatus status) {
|
||||
String state = jdbc.queryForObject("SELECT state FROM ma_operation WHERE operation_kind = 'WORKSPACE_RETIREMENT'", String.class);
|
||||
boolean releasing = state.equals("RELEASED");
|
||||
boolean recording = state.equals("PLATFORM_CLEANED");
|
||||
if (releasing && cut.equals("before") || recording && cut.equals("before-receipt")) {
|
||||
awaitKill(input);
|
||||
}
|
||||
super.doCommit(status);
|
||||
if (releasing && cut.equals("after") || recording && cut.equals("after-receipt")) {
|
||||
awaitKill(input);
|
||||
}
|
||||
}
|
||||
};
|
||||
var bindings = new JdbcRuntimeBindingRepository(source, new AesGcmSecretProtector("test", new byte[32]));
|
||||
var api = mock(KubernetesHttpRuntimeClient.class);
|
||||
var objects = JSON.convertValue(document.get("objects"),
|
||||
new com.fasterxml.jackson.core.type.TypeReference<Map<String, Map<String, Object>>>() { });
|
||||
when(api.get(anyString(), anyString(), anyString())).thenAnswer(call -> CompletableFuture.completedFuture(
|
||||
cut.endsWith("receipt") ? objects.get(call.<String>getArgument(0)) : null));
|
||||
when(api.getCluster(anyString(), anyString())).thenAnswer(call -> CompletableFuture.completedFuture(objects.get(call.<String>getArgument(0))));
|
||||
when(api.getNodeLease(anyString())).thenReturn(CompletableFuture.completedFuture(objects.get("leases")));
|
||||
when(api.listPersistentVolumes()).thenReturn(CompletableFuture.completedFuture(List.of(objects.get("persistentvolumes"))));
|
||||
when(api.listPods(anyString())).thenReturn(CompletableFuture.completedFuture(List.of(objects.get("pods"))));
|
||||
when(api.listVolumeAttachments()).thenReturn(CompletableFuture.completedFuture(List.of()));
|
||||
new ResidentWorkspaceRetirementStore(jdbc, manager, bindings, List.of(selected), JSON).reclaim(request, api);
|
||||
}
|
||||
|
||||
private static void awaitKill(Path input) {
|
||||
try {
|
||||
Files.writeString(input.resolveSibling("cut"), "ready");
|
||||
Thread.sleep(30_000);
|
||||
throw new IllegalStateException("Parent did not kill maintenance process");
|
||||
} catch (java.io.IOException failure) {
|
||||
throw new java.io.UncheckedIOException(failure);
|
||||
} catch (InterruptedException failure) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException(failure);
|
||||
}
|
||||
}
|
||||
}
|
||||
+2
-1
@@ -57,7 +57,8 @@ class ResidentWorkspaceRetirementMainTest {
|
||||
@Test
|
||||
void invalidActionsRefuseBeforeBootstrappingOrReadingDeploymentConfiguration() {
|
||||
for (String[] args : new String[][] {{}, {"begin"}, {"release", "request"},
|
||||
{"delete-pod", "request"}, {"inspect", "request", "extra"}}) {
|
||||
{"delete-pod", "request"}, {"prepare-reclamation", "request"}, {"reclaim", "request", "extra"},
|
||||
{"prepare-reclamation", "request", "rv", "extra"}, {"inspect", "request", "extra"}}) {
|
||||
assertThatThrownBy(() -> ResidentWorkspaceRetirementMain.run(args)).isInstanceOf(IllegalArgumentException.class);
|
||||
}
|
||||
}
|
||||
|
||||
+176
@@ -13,6 +13,8 @@ import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.Timeout;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import org.mockito.ArgumentCaptor;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
@@ -347,6 +349,180 @@ class ResidentWorkspaceRetirementStoreTest {
|
||||
assertThat(original.getOperationOwner()).isNull();
|
||||
}
|
||||
|
||||
@Test
|
||||
void originalCleanupAtomicallyReleasesAndThreeHandoffsKeepTerminalReadback() {
|
||||
var current = fixture;
|
||||
var historical = new java.util.ArrayList<ResidentWorkspaceRetirementStore.Request>();
|
||||
var receipts = new java.util.ArrayList<ResidentWorkspaceRetirementStore.Completion>();
|
||||
for (int cycle = 0; cycle < 3; cycle++) {
|
||||
var cycleFixture = current;
|
||||
assertThat(current.storage.inspect(current.mapping.registration()).revision()).isEqualTo(cycle * 2L + 1);
|
||||
var api = prepared(current);
|
||||
assertThatThrownBy(() -> cycleFixture.store().reclaim(cycleFixture.request, api)).isInstanceOf(RuntimeBrokerException.class);
|
||||
current.completePlatform();
|
||||
var terminal = current.store().reclaim(current.request, api);
|
||||
assertThat(terminal.state()).isEqualTo("RELEASED");
|
||||
assertThat(terminal.completion().releasedRevision()).isEqualTo(cycle * 2L + 2);
|
||||
assertThat(terminal.completion().dataPolicy()).isEqualTo(ResidentWorkspaceReclamation.DATA_POLICY);
|
||||
historical.add(current.request);
|
||||
receipts.add(terminal.completion());
|
||||
assertThat(current.bindings.findById(current.original.getBindingId()).getState()).isEqualTo(RuntimeBindingRecord.State.RELEASED);
|
||||
assertThat(current.jdbc.queryForObject("SELECT active_binding_id FROM ma_runtime_slot WHERE request_key = ?",
|
||||
String.class, current.runtimeRequest.requestKey())).isNull();
|
||||
assertThat(current.jdbc.queryForObject("SELECT tenant_id FROM ma_storage_owner", String.class)).isEqualTo("tenant");
|
||||
assertThatThrownBy(() -> cycleFixture.bindings.findOrCreate(cycleFixture.runtimeRequest))
|
||||
.isInstanceOf(RuntimeBrokerException.class);
|
||||
var next = new ResidentRetirementFixture(fixture.source, "next-" + cycle);
|
||||
next.ready();
|
||||
var reader = new ResidentWorkspaceRetirementStore(next.jdbc, next.manager, next.bindings,
|
||||
List.of(next.mapping), new ObjectMapper());
|
||||
for (int index = 0; index < historical.size(); index++) {
|
||||
assertThat(reader.inspect(historical.get(index)).completion()).isEqualTo(receipts.get(index));
|
||||
}
|
||||
current = next;
|
||||
}
|
||||
assertThat(current.storage.inspect(current.mapping.registration()).revision()).isEqualTo(7);
|
||||
}
|
||||
|
||||
@Test
|
||||
void physicalReleasePreservesTenantAndRefusesAForeignAlias() {
|
||||
var api = prepared(fixture);
|
||||
fixture.completePlatform();
|
||||
fixture.store().reclaim(fixture.request, api);
|
||||
var original = fixture.mapping.registration();
|
||||
var foreign = new WorkspaceCsiRegistration("foreign", "foreign-storage", original.clusterDomain(), original.namespace(),
|
||||
original.pvcName(), original.pvcUid(), original.pvName(), original.pvUid(), original.driver(), original.volumeHandle(),
|
||||
original.backendDomain(), original.diskSerial(), original.mountRoot(), original.revision());
|
||||
var before = fixture.storage.inspect(original);
|
||||
assertThatThrownBy(() -> fixture.storage.register(foreign)).isInstanceOf(RuntimeBrokerException.class);
|
||||
assertThat(fixture.storage.inspect(original)).isEqualTo(before);
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM ma_storage", Long.class)).isEqualTo(1);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aLegacyDeletionOnlyPlanStaysReadableWithoutGainingReleaseAuthority() throws Exception {
|
||||
fixture.store().begin(fixture.request);
|
||||
var json = new ObjectMapper();
|
||||
var plan = (com.fasterxml.jackson.databind.node.ObjectNode) json.readTree(fixture.jdbc.queryForObject(
|
||||
"SELECT request_json FROM ma_operation WHERE operation_kind = 'WORKSPACE_RETIREMENT'", String.class));
|
||||
plan.put("version", 1);
|
||||
plan.remove("mapping");
|
||||
fixture.jdbc.update("UPDATE ma_operation SET request_json = ?, request_digest = ? WHERE operation_kind = 'WORKSPACE_RETIREMENT'",
|
||||
json.writeValueAsString(plan), com.alibaba.qwen.code.runtimebroker.WorkspaceCsiRuntimeIdentity.digest(plan));
|
||||
assertThat(fixture.store().inspect(fixture.request).state()).isEqualTo("DRAINING");
|
||||
assertThatThrownBy(() -> fixture.store().prepareReclamation(fixture.request, "10", fixture.api())).isInstanceOf(RuntimeBrokerException.class);
|
||||
assertThatThrownBy(() -> fixture.store().reclaim(fixture.request, fixture.api())).isInstanceOf(RuntimeBrokerException.class);
|
||||
assertThat(fixture.storage.inspect(fixture.mapping.registration()).phase()).isEqualTo("RESERVED");
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"missing-pod", "node-uid", "node-boot", "stale-lease", "lease-owner", "stale-heartbeat",
|
||||
"container", "attachment", "pv-alias", "other-pod", "api-unavailable", "foreign-finalizer"})
|
||||
void uncertainPlatformCompletionNeverReleases(String defect) {
|
||||
var api = prepared(fixture);
|
||||
var attachment = fixture.attachments.getFirst();
|
||||
fixture.completePlatform();
|
||||
switch (defect) {
|
||||
case "missing-pod" -> fixture.objects.remove("pods");
|
||||
case "node-uid" -> map(fixture.objects.get("nodes").get("metadata")).put("uid", "replacement");
|
||||
case "node-boot" -> map(map(fixture.objects.get("nodes").get("status")).get("nodeInfo")).put("bootID", "new-boot");
|
||||
case "stale-lease" -> map(fixture.objects.get("leases").get("spec")).put("renewTime", Instant.now().minusSeconds(120).toString());
|
||||
case "lease-owner" -> map(fixture.objects.get("leases").get("metadata")).put("ownerReferences", List.of());
|
||||
case "stale-heartbeat" -> map(fixture.objects.get("nodes").get("status")).put("conditions", List.of(
|
||||
Map.of("type", "Ready", "status", "True", "lastHeartbeatTime", Instant.now().minusSeconds(120).toString())));
|
||||
case "container" -> map(((List<?>) map(fixture.objects.get("pods").get("status")).get("containerStatuses")).getFirst())
|
||||
.put("containerID", "replacement");
|
||||
case "attachment" -> fixture.attachments.add(attachment);
|
||||
case "pv-alias" -> when(api.listPersistentVolumes()).thenReturn(CompletableFuture.completedFuture(
|
||||
List.of(fixture.objects.get("persistentvolumes"), fixture.objects.get("persistentvolumes"))));
|
||||
case "other-pod" -> when(api.listPods(anyString())).thenReturn(CompletableFuture.completedFuture(
|
||||
List.of(fixture.objects.get("pods"), fixture.objects.get("pods"))));
|
||||
case "api-unavailable" -> when(api.getCluster(org.mockito.ArgumentMatchers.eq("nodes"), anyString()))
|
||||
.thenReturn(CompletableFuture.failedFuture(new IOException("unavailable")));
|
||||
case "foreign-finalizer" -> map(fixture.objects.get("pods").get("metadata")).put("finalizers", List.of("foreign/finalizer"));
|
||||
default -> throw new AssertionError(defect);
|
||||
}
|
||||
var held = fixture.storage.inspect(fixture.mapping.registration());
|
||||
assertThatThrownBy(() -> fixture.store().reclaim(fixture.request, api)).isInstanceOf(RuntimeException.class);
|
||||
assertThat(fixture.storage.inspect(fixture.mapping.registration())).isEqualTo(held);
|
||||
assertThat(fixture.bindings.findById(fixture.original.getBindingId()).getState()).isEqualTo(RuntimeBindingRecord.State.DRAINING);
|
||||
assertThat(fixture.store().inspect(fixture.request).state()).isEqualTo("DRAINING");
|
||||
}
|
||||
|
||||
@Test
|
||||
void freshReadAllowsBoundedMetadataRetryWithoutErasingUnknown() {
|
||||
fixture.preparePlatform();
|
||||
fixture.store().begin(fixture.request);
|
||||
var api = fixture.api();
|
||||
var patches = new AtomicInteger();
|
||||
var delegate = fixture.api();
|
||||
when(api.patchPodFinalizers(anyString(), anyString(), any(byte[].class))).thenAnswer(call -> patches.incrementAndGet() == 1
|
||||
? CompletableFuture.failedFuture(new IOException("reply lost before effect"))
|
||||
: delegate.patchPodFinalizers(call.getArgument(0), call.getArgument(1), call.getArgument(2)));
|
||||
var first = fixture.store().prepareReclamation(fixture.request, "10", api);
|
||||
assertThat(first.finalizerAttempts().getFirst().state()).isEqualTo("UNKNOWN");
|
||||
var second = fixture.store().prepareReclamation(fixture.request, "10", api);
|
||||
assertThat(second.finalizerAttempts()).hasSize(2);
|
||||
assertThat(second.finalizerAttempts().getFirst().state()).isEqualTo("UNKNOWN");
|
||||
assertThat(second.finalizerAttempts().getLast().state()).isEqualTo("RETURNED");
|
||||
fixture.store().prepareReclamation(fixture.request, "11", api);
|
||||
assertThat(patches).hasValue(2);
|
||||
}
|
||||
|
||||
@Test
|
||||
void independentCleanupReceiptDoesNotRewriteALostDeleteResponse() {
|
||||
fixture.preparePlatform();
|
||||
fixture.store().begin(fixture.request);
|
||||
var api = fixture.api();
|
||||
fixture.store().prepareReclamation(fixture.request, "10", api);
|
||||
when(api.deletePod(anyString(), anyString(), any(byte[].class)))
|
||||
.thenReturn(CompletableFuture.failedFuture(new IOException("reply lost")));
|
||||
assertThat(fixture.store().deletePod(fixture.request, "11", api).attempts().getFirst().state()).isEqualTo("UNKNOWN");
|
||||
fixture.completePlatform();
|
||||
var result = fixture.store().reclaim(fixture.request, api);
|
||||
assertThat(result.state()).isEqualTo("RELEASED");
|
||||
assertThat(result.attempts().getFirst().state()).isEqualTo("UNKNOWN");
|
||||
verify(api, times(1)).deletePod(anyString(), anyString(), any(byte[].class));
|
||||
}
|
||||
|
||||
@Test
|
||||
void everyReleaseWriteRollsBackTogetherAndCanResumeAfterPodCleanup() {
|
||||
var api = prepared(fixture);
|
||||
fixture.completePlatform();
|
||||
var held = fixture.storage.inspect(fixture.mapping.registration());
|
||||
for (String cut : List.of("owner", "operation", "claim")) {
|
||||
var failing = new ManagedSchemaJdbcTemplate(fixture.source) {
|
||||
@Override
|
||||
public int update(String sql, Object... args) {
|
||||
if (cut.equals("owner") && sql.startsWith("UPDATE ma_storage_owner SET csi_phase = 'RELEASED'")
|
||||
|| cut.equals("operation") && sql.startsWith("UPDATE ma_operation SET state") && "RELEASED".equals(args[0])
|
||||
|| cut.equals("claim") && sql.startsWith("UPDATE ma_operation SET claim_owner = NULL, claim_until = NULL, claim_generation")) {
|
||||
throw new IllegalStateException("release cut");
|
||||
}
|
||||
return super.update(sql, args);
|
||||
}
|
||||
};
|
||||
var store = new ResidentWorkspaceRetirementStore(failing, fixture.manager, fixture.bindings,
|
||||
List.of(fixture.mapping), new ObjectMapper());
|
||||
assertThatThrownBy(() -> store.reclaim(fixture.request, api)).isInstanceOf(IllegalStateException.class).hasMessage("release cut");
|
||||
assertThat(fixture.storage.inspect(fixture.mapping.registration())).isEqualTo(held);
|
||||
assertThat(fixture.bindings.findById(fixture.original.getBindingId()).getState()).isEqualTo(RuntimeBindingRecord.State.DRAINING);
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT active_binding_id FROM ma_runtime_slot", String.class)).isEqualTo(fixture.original.getBindingId());
|
||||
assertThat(fixture.store().inspect(fixture.request).state()).isEqualTo("PLATFORM_CLEANED");
|
||||
}
|
||||
var released = fixture.store().reclaim(fixture.request, api);
|
||||
assertThat(fixture.store().reclaim(fixture.request, api)).isEqualTo(released);
|
||||
}
|
||||
|
||||
private static KubernetesHttpRuntimeClient prepared(ResidentRetirementFixture current) {
|
||||
current.preparePlatform();
|
||||
current.store().begin(current.request);
|
||||
var api = current.api();
|
||||
current.store().prepareReclamation(current.request, "10", api);
|
||||
current.store().deletePod(current.request, "11", api);
|
||||
return api;
|
||||
}
|
||||
|
||||
private static boolean race(CyclicBarrier barrier, Runnable operation) throws Exception {
|
||||
barrier.await(10, TimeUnit.SECONDS);
|
||||
try {
|
||||
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
package com.alibaba.qwen.code.managedagent.persistence;
|
||||
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
|
||||
import javax.sql.DataSource;
|
||||
|
||||
class ResidentWorkspaceReuseConcurrencyMySqlIT extends ResidentWorkspaceReuseConcurrencyTest {
|
||||
private IsolatedManagedSchemaMySqlTestDatabase database;
|
||||
|
||||
@Override
|
||||
protected DataSource dataSource() {
|
||||
database = new IsolatedManagedSchemaMySqlTestDatabase();
|
||||
return database.source();
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
void closeDatabase() {
|
||||
if (database != null) {
|
||||
database.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
+173
@@ -0,0 +1,173 @@
|
||||
package com.alibaba.qwen.code.managedagent.persistence;
|
||||
|
||||
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties.RuntimeBroker.ResidentWorkspace;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBindingRecord;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeBrokerException;
|
||||
import com.alibaba.qwen.code.runtimebroker.RuntimeProvisionRequest;
|
||||
import com.alibaba.qwen.code.runtimebroker.WorkspaceExecutionProfile;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.Timeout;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
|
||||
import javax.sql.DataSource;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.CyclicBarrier;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
class ResidentWorkspaceReuseConcurrencyTest {
|
||||
protected DataSource dataSource() {
|
||||
return new DriverManagerDataSource("jdbc:h2:mem:resident-reuse-race-" + UUID.randomUUID()
|
||||
+ ";MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE", "sa", "");
|
||||
}
|
||||
|
||||
@Test
|
||||
@Timeout(25)
|
||||
void retainedTenantRefusesAnAliasRegisteredBeforeTheFirstReservation() {
|
||||
var fixture = new ResidentRetirementFixture(dataSource());
|
||||
var original = fixture.mapping.registration();
|
||||
var foreign = new WorkspaceCsiRegistration("foreign", "foreign-storage", original.clusterDomain(), original.namespace(),
|
||||
original.pvcName(), original.pvcUid(), original.pvName(), original.pvUid(), original.driver(), original.volumeHandle(),
|
||||
original.backendDomain(), original.diskSerial(), original.mountRoot(), original.revision());
|
||||
fixture.storage.register(foreign);
|
||||
assertThat(fixture.storage.inspect(foreign).revision()).isZero();
|
||||
var mapping = new ResidentWorkspace("foreign-workspace", 1, foreign, fixture.mapping.protection(),
|
||||
fixture.mapping.image(), fixture.mapping.command(), fixture.mapping.artifacts());
|
||||
new ResidentWorkspaces(List.of(mapping));
|
||||
fixture.jdbc.update("INSERT INTO ma_workspace (tenant_id, workspace_id, workspace_generation, storage_id, display_name,"
|
||||
+ " config_ref, policy_ref, state) VALUES ('foreign', 'foreign-workspace', 1, 'foreign-storage', 'resident', ?, ?, 'ACTIVE')",
|
||||
WorkspaceExecutionProfile.CONFIG_REF, WorkspaceExecutionProfile.POLICY_REF);
|
||||
var request = new RuntimeProvisionRequest(ResidentWorkspaces.scope(mapping), null,
|
||||
WorkspaceCsiReservationStore.PROVISIONER_KIND, foreign.storageId());
|
||||
var created = fixture.bindings.findOrCreate(request);
|
||||
var foreignBinding = fixture.bindings.claimOperation(created.getBindingId(), "foreign-owner", Duration.ofMinutes(5));
|
||||
assertThat(foreignBinding).isNotNull();
|
||||
String reservationId = UUID.randomUUID().toString();
|
||||
fixture.ready();
|
||||
assertThatThrownBy(() -> fixture.storage.inspect(foreign)).isInstanceOf(RuntimeBrokerException.class);
|
||||
assertThatThrownBy(() -> fixture.storage.reserve(foreign, fixture.bindings, foreignBinding, reservationId))
|
||||
.isInstanceOfSatisfying(RuntimeBrokerException.class, failure -> assertThat(failure.getCode()).isEqualTo("workspace_csi_busy"));
|
||||
fixture.preparePlatform();
|
||||
fixture.store().begin(fixture.request);
|
||||
var api = fixture.api();
|
||||
fixture.store().prepareReclamation(fixture.request, "10", api);
|
||||
fixture.store().deletePod(fixture.request, "11", api);
|
||||
fixture.completePlatform();
|
||||
assertThat(fixture.store().reclaim(fixture.request, api).state()).isEqualTo("RELEASED");
|
||||
var released = fixture.storage.inspect(original);
|
||||
assertThat(released.revision()).isEqualTo(2);
|
||||
assertThatThrownBy(() -> fixture.storage.inspect(foreign)).isInstanceOf(RuntimeBrokerException.class);
|
||||
assertThatThrownBy(() -> fixture.storage.register(foreign)).isInstanceOf(RuntimeBrokerException.class);
|
||||
assertThatThrownBy(() -> fixture.storage.reserve(foreign, fixture.bindings, foreignBinding, reservationId))
|
||||
.isInstanceOfSatisfying(RuntimeBrokerException.class, failure -> assertThat(failure.getCode()).isEqualTo("workspace_csi_unavailable"));
|
||||
assertThat(fixture.storage.inspect(original)).isEqualTo(released);
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT tenant_id FROM ma_storage_owner", String.class)).isEqualTo("tenant");
|
||||
}
|
||||
|
||||
@Test
|
||||
@Timeout(25)
|
||||
void sameTenantAliasesCompeteAtRevisionThreeAfterOriginalRetirement() throws Exception {
|
||||
var fixture = new ResidentRetirementFixture(dataSource());
|
||||
fixture.ready();
|
||||
assertThat(fixture.storage.inspect(fixture.mapping.registration()).revision()).isEqualTo(1);
|
||||
fixture.preparePlatform();
|
||||
fixture.store().begin(fixture.request);
|
||||
var api = fixture.api();
|
||||
fixture.store().prepareReclamation(fixture.request, "10", api);
|
||||
fixture.store().deletePod(fixture.request, "11", api);
|
||||
fixture.completePlatform();
|
||||
var terminal = fixture.store().reclaim(fixture.request, api);
|
||||
assertThat(terminal.state()).isEqualTo("RELEASED");
|
||||
assertThat(terminal.completion().releasedRevision()).isEqualTo(2);
|
||||
assertThat(fixture.storage.inspect(fixture.mapping.registration()).phase()).isEqualTo("RELEASED");
|
||||
assertThat(fixture.storage.inspect(fixture.mapping.registration()).revision()).isEqualTo(2);
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT tenant_id FROM ma_storage_owner", String.class)).isEqualTo("tenant");
|
||||
|
||||
var first = contender(fixture, "reuse-a", "alias-a");
|
||||
var second = contender(fixture, "reuse-b", "alias-b");
|
||||
assertThat(first.registration().aliasKey()).isNotEqualTo(second.registration().aliasKey());
|
||||
assertThat(first.registration().physicalKey()).isEqualTo(second.registration().physicalKey());
|
||||
var barrier = new CyclicBarrier(2);
|
||||
boolean firstWon;
|
||||
boolean secondWon;
|
||||
try (var executor = Executors.newFixedThreadPool(2)) {
|
||||
var one = executor.submit(() -> contend(fixture, first, barrier));
|
||||
var two = executor.submit(() -> contend(fixture, second, barrier));
|
||||
firstWon = one.get(10, TimeUnit.SECONDS);
|
||||
secondWon = two.get(10, TimeUnit.SECONDS);
|
||||
}
|
||||
assertThat(firstWon ^ secondWon).isTrue();
|
||||
var winner = firstWon ? first : second;
|
||||
var loser = firstWon ? second : first;
|
||||
var held = fixture.storage.inspect(winner.registration());
|
||||
assertThat(held.phase()).isEqualTo("RESERVED");
|
||||
assertThat(held.revision()).isEqualTo(3);
|
||||
assertThat(held.reservationId()).isEqualTo(winner.reservationId());
|
||||
assertThat(held.registrationKey()).isEqualTo(winner.registration().aliasKey());
|
||||
assertThat(held.bindingId()).isEqualTo(winner.binding().getBindingId());
|
||||
assertThat(held.runtimeGeneration()).isEqualTo(winner.binding().getGeneration());
|
||||
assertThat(held.provisionRequestId()).isEqualTo(winner.binding().getProvisionSeed().getProvisionRequestId());
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT COUNT(*) FROM ma_storage_owner", Long.class)).isEqualTo(1);
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT tenant_id FROM ma_storage_owner", String.class)).isEqualTo("tenant");
|
||||
|
||||
assertThatThrownBy(() -> fixture.storage.reserve(loser.registration(), fixture.bindings,
|
||||
loser.binding(), loser.reservationId())).isInstanceOfSatisfying(RuntimeBrokerException.class,
|
||||
failure -> assertThat(failure.getCode()).isEqualTo("workspace_csi_busy"));
|
||||
assertThat(fixture.storage.inspect(winner.registration())).isEqualTo(held);
|
||||
assertThat(fixture.storage.reserve(winner.registration(), fixture.bindings,
|
||||
winner.binding(), winner.reservationId())).isEqualTo(held);
|
||||
assertThat(fixture.store().reclaim(fixture.request, api)).isEqualTo(terminal);
|
||||
assertThat(fixture.store().begin(fixture.request)).isEqualTo(terminal);
|
||||
assertThatThrownBy(() -> fixture.bindings.findOrCreate(fixture.runtimeRequest)).isInstanceOf(RuntimeBrokerException.class);
|
||||
assertThat(fixture.storage.inspect(winner.registration())).isEqualTo(held);
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT tenant_id FROM ma_storage_owner", String.class)).isEqualTo("tenant");
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT active_binding_id FROM ma_runtime_slot WHERE request_key = ?",
|
||||
String.class, fixture.runtimeRequest.requestKey())).isNull();
|
||||
assertThat(fixture.jdbc.queryForObject("SELECT active_binding_id FROM ma_runtime_slot WHERE request_key = ?",
|
||||
String.class, winner.binding().getRequest().requestKey())).isEqualTo(winner.binding().getBindingId());
|
||||
}
|
||||
|
||||
private static Contender contender(ResidentRetirementFixture fixture, String workspace, String storage) {
|
||||
var original = fixture.mapping.registration();
|
||||
var registration = new WorkspaceCsiRegistration(original.tenantId(), storage, original.clusterDomain(), original.namespace(),
|
||||
original.pvcName(), original.pvcUid(), original.pvName(), original.pvUid(), original.driver(), original.volumeHandle(),
|
||||
original.backendDomain(), original.diskSerial(), original.mountRoot(), original.revision());
|
||||
fixture.storage.register(registration);
|
||||
var mapping = new ResidentWorkspace(workspace, 1, registration, fixture.mapping.protection(),
|
||||
fixture.mapping.image(), fixture.mapping.command(), fixture.mapping.artifacts());
|
||||
var selected = new ResidentWorkspaces(List.of(mapping)).require(ResidentWorkspaces.scope(mapping));
|
||||
ResidentWorkspaces.requireRegistration(fixture.jdbc, selected);
|
||||
fixture.jdbc.update("INSERT INTO ma_workspace (tenant_id, workspace_id, workspace_generation, storage_id, display_name,"
|
||||
+ " config_ref, policy_ref, state) VALUES ('tenant', ?, 1, ?, 'resident', ?, ?, 'ACTIVE')", workspace, storage,
|
||||
WorkspaceExecutionProfile.CONFIG_REF, WorkspaceExecutionProfile.POLICY_REF);
|
||||
var request = new RuntimeProvisionRequest(ResidentWorkspaces.scope(selected), null,
|
||||
WorkspaceCsiReservationStore.PROVISIONER_KIND, storage);
|
||||
var created = fixture.bindings.findOrCreate(request);
|
||||
var claimed = fixture.bindings.claimOperation(created.getBindingId(), "reuse-" + workspace, Duration.ofMinutes(5));
|
||||
assertThat(claimed).isNotNull();
|
||||
return new Contender(registration, claimed, UUID.randomUUID().toString());
|
||||
}
|
||||
|
||||
private static boolean contend(ResidentRetirementFixture fixture, Contender contender, CyclicBarrier barrier) throws Exception {
|
||||
var store = new WorkspaceCsiReservationStore(new ManagedSchemaJdbcTemplate(fixture.source),
|
||||
new DataSourceTransactionManager(fixture.source), new ObjectMapper());
|
||||
barrier.await(5, TimeUnit.SECONDS);
|
||||
try {
|
||||
store.reserve(contender.registration(), fixture.bindings, contender.binding(), contender.reservationId());
|
||||
return true;
|
||||
} catch (RuntimeBrokerException failure) {
|
||||
assertThat(failure.getCode()).isEqualTo("workspace_csi_busy");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private record Contender(WorkspaceCsiRegistration registration, RuntimeBindingRecord binding, String reservationId) { }
|
||||
}
|
||||
+27
-2
@@ -239,6 +239,29 @@ class ResidentWorkspacesTest {
|
||||
assertThat(storage.inspect(mapping.registration()).phase()).isEqualTo("RESERVED");
|
||||
}
|
||||
|
||||
@Test
|
||||
void executionAndCloseUseTheExactLaterReservationInsteadOfTheFirstRevision() {
|
||||
jdbc.update("UPDATE ma_storage_owner SET csi_revision = 2, tenant_id = 'tenant' WHERE storage_kind = 'CSI'");
|
||||
var session = create("later-cycle").sessionId();
|
||||
var original = ready(residents.request(context));
|
||||
var record = active(original, session);
|
||||
assertThat(storage.inspect(mapping.registration()).revision()).isEqualTo(3);
|
||||
authority.claim(context, record);
|
||||
authority.assertHeld(context, record);
|
||||
jdbc.update("UPDATE ma_storage_owner SET csi_revision = 5");
|
||||
assertThatThrownBy(() -> authority.assertHeld(context, record)).isInstanceOf(RuntimeBrokerException.class);
|
||||
jdbc.update("UPDATE ma_storage_owner SET csi_revision = 3");
|
||||
var releasing = bindings.beginSessionRelease(runtimeSessions, executions, record);
|
||||
authority.release(context, releasing);
|
||||
bindings.completeSessionRelease(runtimeSessions, releasing);
|
||||
jdbc.update("UPDATE ma_session SET status = 'CLOSING' WHERE session_id = ?", session);
|
||||
Boolean closed = transaction.execute(status -> residents.verifyClosed(jdbc, "tenant", session));
|
||||
assertThat(closed).isTrue();
|
||||
jdbc.update("UPDATE ma_storage_owner SET csi_revision = 5");
|
||||
assertThatThrownBy(() -> transaction.execute(status -> residents.verifyClosed(jdbc, "tenant", session)))
|
||||
.isInstanceOf(RuntimeBrokerException.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPreparedDispatchCannotCrossTheDurableSessionCloseFence() {
|
||||
var session = create("original").sessionId();
|
||||
@@ -402,12 +425,14 @@ class ResidentWorkspacesTest {
|
||||
private RuntimeBindingRecord ready(RuntimeProvisionRequest request) {
|
||||
var created = bindings.findOrCreate(request);
|
||||
var claimed = bindings.claimOperation(created.getBindingId(), "owner", Duration.ofMinutes(5));
|
||||
storage.reserve(mapping.registration(), bindings, claimed, UUID.randomUUID().toString());
|
||||
var reservation = storage.reserve(mapping.registration(), bindings, claimed, UUID.randomUUID().toString());
|
||||
var seed = claimed.getProvisionSeed();
|
||||
var lease = new RuntimeLease(seed.getProvisionalRuntimeId(), URI.create("http://127.0.0.1:9"),
|
||||
seed.getToken(), seed.getLeaseId(), seed.getEpoch());
|
||||
var ready = bindings.compareAndSet(claimed, claimed.withAttestation(lease,
|
||||
new RuntimeResourceHandle("kubernetes-workspace", 1, Map.of("dataFixture", true)), Instant.now(), Instant.now()));
|
||||
new RuntimeResourceHandle("kubernetes-workspace", 1, Map.of("dataFixture", true, "storage",
|
||||
Map.of("reservationRevision", Long.toString(reservation.revision()), "reservationId", reservation.reservationId()))),
|
||||
Instant.now(), Instant.now()));
|
||||
bindings.releaseOperation(ready.getBindingId(), "owner", ready.getOperationGeneration());
|
||||
return bindings.findById(ready.getBindingId());
|
||||
}
|
||||
|
||||
+4
-2
@@ -294,15 +294,17 @@ class WorkspaceCsiRetirementStoreTest {
|
||||
|
||||
@Test
|
||||
void sameVolumeAliasIsStillBusyWhileUnrelatedVolumeCanReserve() {
|
||||
store.beginRetirement(registration, bindings, original, reservation, UUID.randomUUID().toString());
|
||||
var alias = WorkspaceCsiReservationStoreTest.registration("other", "alias", "backend", "handle", 1);
|
||||
var alias = WorkspaceCsiReservationStoreTest.registration("tenant", "alias", "backend", "handle", 1);
|
||||
store.register(alias);
|
||||
var contender = binding(alias);
|
||||
store.beginRetirement(registration, bindings, original, reservation, UUID.randomUUID().toString());
|
||||
assertThatThrownBy(() -> store.reserve(alias, bindings, contender, UUID.randomUUID().toString()))
|
||||
.isInstanceOfSatisfying(RuntimeBrokerException.class, error -> {
|
||||
assertThat(error.getCode()).isEqualTo("workspace_csi_busy");
|
||||
assertThat(error.isRetryable()).isTrue();
|
||||
});
|
||||
var foreign = WorkspaceCsiReservationStoreTest.registration("other", "foreign-alias", "backend", "handle", 1);
|
||||
assertUnavailable(() -> store.register(foreign));
|
||||
var unrelated = WorkspaceCsiReservationStoreTest.registration("other", "another", "backend", "another-handle", 1);
|
||||
store.register(unrelated);
|
||||
assertThat(store.reserve(unrelated, bindings, binding(unrelated), UUID.randomUUID().toString()).phase()).isEqualTo("RESERVED");
|
||||
|
||||
+27
@@ -143,6 +143,33 @@ class WorkspaceCsiRuntimeIdentityTest {
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void laterResidentReservationSurvivesAttestationAndReadonlyReconciliation() {
|
||||
bindings.compareAndSet(original, original.withState(RuntimeBindingRecord.State.RELEASED, null, Instant.now()));
|
||||
var jdbc = new ManagedSchemaJdbcTemplate(source);
|
||||
jdbc.update("UPDATE ma_storage_owner SET csi_revision = 2, tenant_id = 'tenant'");
|
||||
jdbc.update("INSERT INTO ma_workspace (tenant_id, workspace_id, workspace_generation, storage_id, display_name,"
|
||||
+ " config_ref, policy_ref, state) VALUES ('tenant', 'later-workspace', 1, 'storage', 'resident', ?, ?, 'ACTIVE')",
|
||||
WorkspaceExecutionProfile.CONFIG_REF, WorkspaceExecutionProfile.POLICY_REF);
|
||||
var request = new RuntimeProvisionRequest(new RuntimeScope("tenant", "later-workspace", "1", "/workspace",
|
||||
com.alibaba.qwen.code.runtimebroker.ResidentWorkspaceProfile.CAPABILITY_DIGEST, "workspace"),
|
||||
null, "kubernetes-workspace", "storage");
|
||||
original = bindings.claimOperation(bindings.findOrCreate(request).getBindingId(), "owner", Duration.ofMinutes(5));
|
||||
try (var provider = provider()) {
|
||||
provider.reserveResource(original);
|
||||
var handle = join(provider.ensureResource(request, original.getProvisionSeed(), null));
|
||||
assertThat(nested(handle.getValue(), "storage").get("reservationRevision")).isEqualTo("3");
|
||||
original = bindings.compareAndSet(original, original.withResourceHandle(handle, Instant.now()));
|
||||
var lease = join(provider.provision(request, original.getProvisionSeed()));
|
||||
original = bindings.compareAndSet(original, original.withAttestation(lease, handle, Instant.now(), Instant.now()));
|
||||
assertThat(join(provider.reconcile(request, original.getProvisionSeed(), handle, lease)).getOutcome())
|
||||
.isEqualTo(RuntimeObservation.Outcome.READY);
|
||||
join(provider.confirm(request, lease));
|
||||
assertThat(api.creates).isEqualTo(2);
|
||||
assertThat(storage.inspect(registration).revision()).isEqualTo(3);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void existingOrAmbiguousCreateNeverAdoptsOrCreatesAnotherPod() {
|
||||
for (String resource : List.of("secrets", "pods")) {
|
||||
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
package com.alibaba.qwen.code.runtimebroker;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
class CsiOwnershipRevisionTest {
|
||||
@Test
|
||||
void residentCyclesDoNotExpandTheOriginalPrivateRetirementProtocol() {
|
||||
for (long revision : new long[] {0, 2, 4, 6}) {
|
||||
assertThat(CsiOwnershipRevision.residentReleased(revision)).isTrue();
|
||||
assertThat(CsiOwnershipRevision.residentReserved(revision + 1)).isTrue();
|
||||
assertThat(CsiOwnershipRevision.reserved(revision + 1)).isEqualTo(revision == 0);
|
||||
assertThat(CsiOwnershipRevision.released(revision)).isEqualTo(revision == 0);
|
||||
}
|
||||
assertThat(CsiOwnershipRevision.draining(2)).isTrue();
|
||||
assertThatThrownBy(() -> CsiOwnershipRevision.next(2)).isInstanceOf(IllegalArgumentException.class);
|
||||
assertThat(CsiOwnershipRevision.residentReserved(-1)).isFalse();
|
||||
assertThat(CsiOwnershipRevision.residentReleased(-2)).isFalse();
|
||||
}
|
||||
}
|
||||
+39
@@ -21,6 +21,7 @@ import java.nio.file.Path;
|
||||
import java.time.Duration;
|
||||
import java.util.Arrays;
|
||||
import java.util.Map;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionException;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
@@ -117,6 +118,44 @@ class KubernetesHttpRuntimeClientTest {
|
||||
assertEquals(1, calls.get());
|
||||
}
|
||||
|
||||
@Test
|
||||
void reclamationReadsNodeLeaseAndPatchesOnlyTheExplicitPodPath() {
|
||||
client.getNodeLease("node-a").toCompletableFuture().join();
|
||||
assertEquals("/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/node-a", path.get());
|
||||
var patch = JsonCodec.encode(List.of(Map.of("op", "test", "path", "/metadata/uid", "value", "original")));
|
||||
status.set(409);
|
||||
var result = client.patchPodFinalizers("runtime", "original", patch).toCompletableFuture().join();
|
||||
assertEquals(409, result.status());
|
||||
assertTrue(result.complete());
|
||||
assertEquals("PATCH", method.get());
|
||||
assertEquals("application/json-patch+json", contentType.get());
|
||||
assertEquals("/api/v1/namespaces/runtime/pods/original", path.get());
|
||||
assertEquals(new String(patch, StandardCharsets.UTF_8), requestBody.get());
|
||||
}
|
||||
|
||||
@Test
|
||||
void completeEmptyAttachmentListIsDifferentFromMissingOrPartialList() {
|
||||
response.set(JsonCodec.encode(Map.of("kind", "VolumeAttachmentList", "metadata", Map.of("resourceVersion", "8"), "items", List.of())));
|
||||
assertEquals(List.of(), client.listVolumeAttachments().toCompletableFuture().join());
|
||||
assertEquals("/apis/storage.k8s.io/v1/volumeattachments?limit=128&continue=", path.get());
|
||||
status.set(404);
|
||||
assertEquals(502, failure(client.listVolumeAttachments().toCompletableFuture()).getStatusCode());
|
||||
status.set(200);
|
||||
response.set(JsonCodec.encode(Map.of("kind", "VolumeAttachmentList", "metadata",
|
||||
Map.of("resourceVersion", "8", "continue", "same"), "items", List.of())));
|
||||
assertEquals(502, failure(client.listVolumeAttachments().toCompletableFuture()).getStatusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void duplicateListIdentitiesAndMissingListRevisionCannotProveCleanup() {
|
||||
var item = Map.of("metadata", Map.of("name", "pv", "uid", "pv-uid"));
|
||||
response.set(JsonCodec.encode(Map.of("kind", "PersistentVolumeList", "metadata", Map.of("resourceVersion", "8"),
|
||||
"items", List.of(item, item))));
|
||||
assertEquals(502, failure(client.listPersistentVolumes().toCompletableFuture()).getStatusCode());
|
||||
response.set(JsonCodec.encode(Map.of("kind", "PodList", "metadata", Map.of(), "items", List.of())));
|
||||
assertEquals(502, failure(client.listPods("runtime").toCompletableFuture()).getStatusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void readsFullSizeArtifactChunksWithBoundedJsonEnvelopeHeadroom() {
|
||||
var object = Map.of("kind", "ConfigMap", "binaryData", Map.of("chunk", "A".repeat(1024 * 1024)));
|
||||
|
||||
Reference in new issue
Block a user