QwenLM/qwen-code
1
0 Fork 0
mirror of https://github.com/QwenLM/qwen-code.git synced 2026-11-07 06:19:43 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

feat(managed-agent): Add a shared Harness pool for Kubernetes (#15255)

Browse Source 
Route sessions to durable instance and boot placements with fenced writer grants, resource-idle ownership release, EndpointSlice discovery, and private mTLS deployment templates.

Co-authored-by: Qwen-Coder <[email protected]>
This commit is contained in:
jinyeandQwen-Coder authored and GitHub committed 2026-10-23 17:22:19 +00:00
1 parent 249ab50d84
commit a147c39712
97 files changed
+6933 -169
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/sdk-java.yml
+10 -6
View File
@@ -897,12 +897,15 @@ jobs:
# margin (11836 s of fork plus wrap); the FG12e Runtime connection
# replacement cell adds 28 s, moving the fork to 11264 s and leaving
# Verify at 198 minutes, wrapping fork + 600 s plus margin (11864 s of
# fork plus wrap, 16 s left);
# fork plus wrap, 16 s left). The pool fixture provisionally reserves
# 180 s (its 120 s method ceiling plus 60 s for process/context cleanup),
# pending CI measurement: fork 11444 s, Verify 201 minutes, and 12044 s
# of fork plus wrap with the same 16 s left;
# the Broker fault gates use 20,
# and admission plus O4 keep per-fork ceilings) plus ~13 minutes for the
# uncapped setup steps (Checkout.. cgroup delegation, the report
# checks): 288 + 13. The failover arms run in hosted-failover-e2e.
timeout-minutes: 301
# checks): 291 + 13. The failover arms run in hosted-failover-e2e.
timeout-minutes: 304
services:
mysql:
image: 'mysql:8.4.6'
@@ -1002,9 +1005,10 @@ jobs:
# fork leaves no per-test failure lines, so the main-CI failure
# analyzer can only file an undiagnosable per-commit issue (#13503).
# hosted-process-ci.test.js pins this ordering; the fork budget is
# 11264 s, so the step wraps to fork + 600 s plus margin (198 min,
# 16 s left before the step must move).
timeout-minutes: 198
# 11444 s, so the step wraps to fork + 600 s plus margin (201 min,
# 16 s left before the step must move), including the provisional
# 180 s pool-fixture reservation pending CI measurement.
timeout-minutes: 201
env:
MAVEN_ARGS: '--settings ${{ runner.temp }}/setup-java-m2/settings.xml --toolchains ${{ runner.temp }}/setup-java-m2/toolchains.xml'
MYSQL_PORT: "${{ job.services.mysql.ports['3306'] }}"
docs/design/2026-10-21-managed-harness-kubernetes.md
+227
View File
@@ -0,0 +1,227 @@
# Kubernetes shared Hosted Harness pool
[English](2026-10-21-managed-harness-kubernetes.md) | [简体中文](2026-10-21-managed-harness-kubernetes.zh-CN.md)
Status: Implementation candidate, 2026-10-23, integrated with main `32b4234b84` and its single-V1 first-release contract. Original source audit: `97567d5dd733ad905aacf61f1f61a1e71c6fcc72`. Historical ACK idle-pool acceptance predates this baseline; new-head CI, foreground/fault and automatic-scaling gates remain pending. This refines [shared multi-tenant Harness](2026-10-21-managed-shared-harness.md).
## Scope and decisions
Use one active Java control plane with a shared Harness Deployment. Each Harness Pod can serve multiple tenants and users. Java owns authorization, durable placement and dispatch; Kubernetes owns Pod lifecycle and replica count. Keep the existing Agent loop, public Session APIs and Runtime Broker boundary.
The first pool supports compatible instances, sticky Session ownership, manual scale-out and strict crash recovery. Add non-destructive release for resource-idle Sessions before enabling normal scale-in. Add application metrics and automatic scale-out afterward; automatic scale-in remains disabled until every enabled execution profile has passed arbitrary-victim termination tests.
Java high availability, a new Tool Runtime provisioner, universal live migration, scale-to-zero, tenant-provided model keys and dynamic configuration hot reload are outside this design. Shared-process tenant isolation from the parent design remains a prerequisite, not a property Kubernetes supplies.
## Verified starting point
| Finding at the audited commit | Source and implication |
| ------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Hosted requires loopback bind and bearer authentication. | [Profile:40](../../packages/cli/src/serve/hosted-harness-profile.ts), [resolved-bind check:3539](../../packages/cli/src/serve/run-qwen-serve.ts). A Pod cannot simply expose the Harness on `0.0.0.0`. |
| Loopback requests have a Host gate; remote Store/Broker calls require TLS by default. | [Host gate:340](../../packages/cli/src/serve/auth.ts), [Broker URL:26](../../packages/cli/src/serve/managed-runtime-broker-url.ts), [Store URL:844](../../packages/core/src/managed-runtime/http-managed-session-store.ts). A proxy must preserve these boundaries. |
| Java has one SDK client and global boot adoption. | [Connector:63,1557,1591](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/QwenHostedHarnessConnector.java). Random Service balancing would invalidate unrelated attachments. |
| Owner subscriptions use one writer identity. | [Subscriptions:130](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/session/OwnerSubscriptions.java), [connector contract:200](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/HarnessConnector.java). Per-attachment owner identity is required. |
| The bound writer token covers tenant, Workspace and Session, not placement. | [Credentials:57](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/security/WriterCredentialPolicy.java), [writer acquisition:670](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/persistence/ManagedSessionStore.java). A routing CAS alone cannot prevent old-owner reacquisition. |
| Ordinary Hosted registration returns no drain handle; SIGTERM closes the listener concurrently with host teardown. | [Registration:10816](../../packages/cli/src/serve/hosted-harness-session.ts), [close:193](../../packages/cli/src/serve/serve-app-lifecycle.ts). Its default listener timeout is five seconds, independently of Pod grace time. |
| Detach can cancel pending business inputs and drain Runtime resources. | [Detach:10269,10443](../../packages/cli/src/serve/hosted-harness-session.ts). It is not an ownership handoff operation. |
| Wake pumps can start accepted backlog independently of HTTP submission. | [Pump:213](../../packages/cli/src/serve/hosted-monitor-wake.ts). Rejecting new Session creation alone cannot quiesce a Pod. |
| Existing Session/queue gauges count ACP bridges, not the Hosted Session Map. | [Metrics:7313](../../packages/cli/src/serve/run-qwen-serve.ts), [Hosted Map:3663](../../packages/cli/src/serve/hosted-harness-session.ts). Existing daemon gauges cannot drive Harness HPA. |
| The current first-release schema uses one V1 baseline. | [Schema initialization](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/runtimebroker/managedschema/ManagedSchema.java), [Flyway strategy](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/config/ManagedSchemaConfiguration.java), [first-release policy](2026-10-23-managed-first-release-contract.md). Earlier development catalogs require a fresh installation rather than an upgrade. |
These are source observations. Existing tests were inspected, not executed for this research.
## Topology and transport
```mermaid
flowchart LR
U["Authenticated users"] --> J["Java: authorization, placement, dispatch"]
J --> D[("SQL / Session Store")]
J --> P1["Pod A: mTLS proxy"]
J --> P2["Pod B: mTLS proxy"]
P1 --> H1["127.0.0.1:4170 Harness A"]
P2 --> H2["127.0.0.1:4170 Harness B"]
H1 --> S["TLS Store / Broker origin"]
H2 --> S
S --> R["Independent Workspace Tool Runtimes"]
```
Keep the loopback-only Hosted profile. Use an operator-managed in-Pod mTLS reverse proxy, with Envoy as the reference packaging choice, to expose the private Pod endpoint. The proxy accepts approved Java service identities, retains Harness bearer authentication, rewrites upstream Host to `127.0.0.1:4170`, rejects browser Origin, and exposes only the approved private routes. It preserves protocol/boot headers, event IDs and epochs. Disable buffering, caching and automatic mutation retries; streaming timeouts must accommodate the existing 15-second SSE keepalive. See [Envoy TLS](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl).
Provide a genuine TLS endpoint for Java Store/Broker, or an authenticated local egress proxy. Do not enable non-loopback plaintext exceptions. Add an explicit optional `SSLContext` input to the Java Hosted SDK, consumed by Harness configuration for pool mTLS; retain ordinary JVM defaults for the local path. Keep hostname verification enabled. Load client key/trust material from mounted files and rotate by controlled replacement initially. See the [JDK HTTP client TLS contract](https://docs.oracle.com/en/java/javase/21/docs/api/java.net.http/java/net/http/HttpClient.Builder.html).
An individual endpoint must have a valid TLS authority: direct Pod-IP dialing requires a matching IP SAN; a DNS endpoint requires a matching certificate and verified mapping to the discovered Pod. Do not assume Deployment Pods have StatefulSet-style stable DNS, or turn off hostname verification to compensate. Certificate issuance and SAN lifecycle are explicit deployment inputs. A reverse proxy is an additional trusted platform component, not an untrusted execution sandbox.
## Discovery and instance identity
Keep two discovery modes: the current static single address by default, and namespace-scoped Kubernetes discovery when explicitly configured. Static multi-instance configuration is useful for deterministic tests. A single deployment mode chooses its source; failed Kubernetes discovery never falls back to localhost.
Use `discovery.k8s.io/v1` EndpointSlices for a configured Service. Verify namespace, Service name and UID, EndpointSlice ownership, Pod target UID and port. Resolve the concrete HTTPS endpoint and perform authenticated boot/protocol/capability negotiation before selecting it. Keep Pod UID, process boot ID and network address distinct. A reused IP or a new process in the same Pod is a different incarnation.
List first, then watch from the returned `resourceVersion`; deduplicate endpoints across slices and address families. Typed list members may omit `kind` and `apiVersion`; accept that representation while rejecting an explicitly wrong kind and preserving all ownership checks. Re-list on `410 Gone`, resume interrupted watches with bounded backoff, and publish a complete replacement snapshot after a successful list. During discovery uncertainty, pause new placement and preserve existing owner bindings; reachable, authenticated existing owners may continue. Endpoint disappearance is not proof of stopped execution. Terminating or not-ready endpoints are excluded from new assignments, while an existing draining owner can remain directly addressable. See [EndpointSlice conditions](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/) and [watch recovery](https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes).
Only Java receives the discovery ServiceAccount token. Grant namespace-scoped EndpointSlice list/watch and the necessary named-Service read; client label filtering is not an RBAC authorization boundary. Harness and proxy do not need Kubernetes API credentials. Enforce ingress/egress policy with a supporting CNI and keep user Tool Runtimes outside the Harness Pod. See [ServiceAccount mounting](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) and [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/).
## Durable placement and Store-enforced fencing
Add placement fields to the existing `ma_session` root rather than creating a general persisted Pod inventory:
| Proposed field | Meaning and reader |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `harness_instance_id` | Assigned static instance identity or trusted Kubernetes Pod identity; read by the router and Store credential enforcement. |
| `harness_owner_boot_id` | Intended process incarnation; checked by the SDK and Store. Existing `harness_boot_id` remains the last attached event-source observation. |
| `harness_placement_epoch` | Monotonic assignment version; used by CAS, caches, writer grants and release proof. |
| `harness_placement_state` | `UNASSIGNED`, `ACTIVE`, `QUIESCING` or terminal `RELEASED` ownership state. It is neither Pod readiness nor public Session status. |
Add a bounded owner/boot index for placement enumeration during drain and restart; do not poll-scan every Session. The in-memory discovery inventory remains disposable.
Unassigned legacy-local rows have no pool owner. In pool mode, Java authorizes the actor and canonical Session/Workspace, selects a compatible accepting instance, and commits placement before create/load. Every retry resolves the existing placement; a failed RPC does not select another Pod. Placement applies to Hosted-eligible roots, including any explicitly admitted CONTROL Session, under their kind-specific lifecycle rules; auxiliary CONTROL records without a Harness owner are not assigned. Public and CONTROL Session-ID factories must preserve global private-ID uniqueness, or explicitly reject colliding scopes before exposing a cached handle.
Use a new placement-bound writer grant, conceptually `qwp1`, authenticated with the existing HMAC machinery over tenant, Workspace, Session, instance identity, boot ID and placement epoch. Java is the issuer. Store checks the grant against durable current placement on acquire, renew, commit, activation and seal, including any new-work authorization path. Legacy scope-only `qwt1` tokens are rejected for pool-assigned rows. Token contents without a transactional current-placement check are insufficient. Ordinary data reads retain their existing authentication and scope checks; no public caller supplies placement authority.
Retain existing writer generation/lease, activation identity, Turn dispatch lease and Runtime execution fences. A placement epoch is not another independently renewed lease. Change placement only after a verified non-destructive release or database-proven writer expiry and the existing recovery prerequisites. A placement change must prevent the former boot from reacquiring even after `SEALED` or lease expiry. Old delayed requests cannot seal, renew or write a successor's state.
`QUIESCING` still permits the existing live writer to renew and commit pre-cutoff Turn progress, and permits release of its attachment activation. It does not authorize a new writer acquisition or new activation. Terminal revocation begins when release proof commits, or when placement changes through the fenced crash path; simply marking a Session quiescing must not prevent its admitted Turn from finishing.
All affected Store transactions need the same root-before-head rule: take existing tenant/admission guards where required, then the Session root and placement, then the journal head, retaining the established dependent-record order. Renew currently starts at the head and must be reconciled with this rule. Do not acquire the Session root after holding its head. Validate the complete lock graph against acquire, renew, commit, wake claim, release, lifecycle and Runtime fencing in real MySQL.
## Routing and downstream consumers
Keep one SDK client per concrete instance/boot, with independently scoped attachment, heartbeat and recovery caches. Resolve a placement once per operation and retain that target through the response; a boot change invalidates only that incarnation. Cache keys include tenant, Session and placement epoch. The router handles ownership, while the existing connector continues implementing the Hosted protocol.
| Consumer | Required placement behavior |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `HarnessCoordinator` | Create/load, submit, stream, continue, cancellation and takeover use one fenced target. |
| `ManagedAgentService`, `SessionRenameCoordinator` | Inline and background title operations preserve the same target and durable command identity. |
| `ActionResponseCoordinator` | Approval replies reach the current active Turn owner; never recreate the action on another Pod. |
| `ManagedAutomationService`, `AutomationScanner` | Fire, reconcile and cancel retain the automation's principal and owning Session. |
| `ManagedChannelService`, `SessionMessageRelay` | Receive/consume/delivery receipts route by each destination Session; replay existing command identities. |
| `ChildResultRelay`, `TaskCancelCoordinator` | Resolve parent and child separately; they can reside on different Pods. |
| `SessionLifecycleCoordinator` | Business close/delete/resource phases stay separate from Pod ownership release. |
| `WakeRecovery`, `TransportWakeActivator` | Consult placement and quiesce state before starting background work. |
| `OwnerSubscriptions` | Carry writer identity per attachment, group SQL checks by writer and tenant, and retain SQL proof of a live writer. |
MCP and Hook pins continue through Session create/load and the existing Runtime boundary; they are not a second routing authority. Existing ownerless `supports*` predicates use one homogeneous required capability contract for the pool. Every selected Pod must satisfy it. Bound Session operations check their actual owner; support on an arbitrary replica cannot authorize a request. Start with one engine digest and compatible recovery readers; heterogeneous engine routing is out of scope.
## New private lifecycle contract
The following are opt-in private endpoints implemented for pool instances. Classify the first two and metrics as process-global, and release as live-Session-owner scoped. Keep them Hosted-only, service-authenticated, versioned and unavailable through browser/public routes.
| Private route | Contract |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /internal/hosted-harness/v1/instance` | Returns trusted instance/boot identity, lifecycle state, capability identity and Hosted-owned capacity counters. It does not acquire a Session or start work. |
| `POST /internal/hosted-harness/v1/instance:drain` | Idempotent drain intent tied to operation ID and expected boot. Stops new placement eligibility and establishes the no-start cutoff. Does not close public Sessions. |
| `POST /session/:id/ownership:release` | Validates canonical scope, attachment, placement epoch and writer generation; releases only a proven resource-idle Session without cancelling pending inputs. |
| `GET /internal/hosted-harness/v1/metrics` | Bounded Hosted-specific instrumentation for authorized monitoring; contains no prompts, credentials or tenant/Session metric labels. |
Require explicit negotiated features `harness-pool-owner/1` and `harness-instance-lifecycle/1` before enabling pool placement. Add the new fields to private DTO/schema/client fixtures and the shared contract tests; do not reinterpret the existing protocol-v1 envelope. Old Harness processes remain valid for explicitly configured local single-address mode and are ineligible for the pool.
Authorization and exact committed-receipt replay precede refusal of fresh work. A drain refusal must unambiguously mean no fresh admission occurred. Status, streaming, original execution observation, approval and cancellation needed by admitted work remain available. The cutoff concerns new top-level work: an already running Turn retains the continuation permit defined below. Recovery without that existing permit cannot start model or tool work on the quiescing instance.
If admission closes while a recovery is awaiting durable reads or Runtime acquisition, its dispatch guard returns `409 hosted_harness_quiescing` after the usual lease cleanup. This remains a pool wait outside the execution retry budget. A resident writer's stopped state still takes precedence over quiescing. `RecoveryDeclined` remains reserved for deterministic durable refusals, and unrelated recovery failures retain their existing budget handling.
## Quiesce and release state machine
```mermaid
stateDiagram-v2
[*] --> STARTING
STARTING --> ACCEPTING: identity and dependencies ready
ACCEPTING --> DRAINING: stop new placement
DRAINING --> QUIESCING: atomic no-start cutoff
QUIESCING --> DRAINED: all Session ownership releases proven
QUIESCING --> RECOVERY_REQUIRED: deadline or unknown effects
DRAINED --> [*]
RECOVERY_REQUIRED --> [*]: bounded termination and fenced recovery
```
Java may retain newly authorized input in a bounded durable queue, but it cannot dispatch that input to a quiescing owner. At the atomic cutoff, each already running Turn retains an in-process continuation permit tied to its original prompt, activation and placement epoch. It may perform subsequent model rounds, foreground tool steps and approval continuation needed to settle that same Turn. Pending input, cold recovery, independently scheduled children and autonomous channel/message/automation work do not inherit that permit and cannot start; their durable identities and wake intents remain for the successor. A running Turn that needs a separately scheduled child may therefore remain non-releasable until the deadline; first release acceptance uses bounded foreground profiles. Merely returning no active prompt is not a quiescence proof.
Initially, release only Sessions with no executing Turn, in-flight mutation, unresolved tool/model effect, held Runtime operation, active Hook/MCP work, live background Shell/monitor/child work or pending publication. An idle attachment normally still has a live activation; release that attachment activation inside the no-start barrier as part of the protocol, rather than requiring it to be absent beforehand. Idle configured resources also require a proven reconnect/release contract; otherwise the Session is not eligible. Immutable resources needed for cold load must already be durable and readable. Static Skills do not by themselves require an execution process.
Persist the release operation's identity and proof through the existing private Session-record machinery, using a bounded `hosted_harness_release` domain record, scoped to the placement epoch, boot, writer generation and journal revision/sequence. Seal and release-proof publication must share the Store's fenced transaction after activation release and no-start proof. A committed release terminally revokes acquisition for that placement epoch immediately, even before Java changes placement; otherwise an old in-flight load could reopen a sealed head. This may require a dedicated Store operation built from existing primitives; do not assume `ManagedSession.close()` already implements it. Emit no business SessionEnd/Delete and cancel no accepted backlog.
Extend the closed Java/TypeScript domain and resource-schema registries, private projection/retention and CSI closure readers explicitly. This control receipt must not be forced into a task/run record shape or exposed as a public task. Its committed proof must survive the old process and remain available until placement reconciliation and safe retention complete; no new generic persistence subsystem is needed.
On lost ACK, Java reconciles the original release identity against SQL and the sealed head; it does not reacquire on the old Pod to inspect it. Successor CAS atomically rechecks the exact proof, sealed head/revision, current placement and canonical root's applicable admission/lifecycle rules under the root/head lock order. This applies to eligible PUBLIC and CONTROL roots. `OwnerSubscriptions` and cached attachments then follow the new SQL-proven writer. An RPC success or stopped local renewal alone is not release proof. `DRAINED` requires zero remaining resident owners and in-flight work; an ineligible Session keeps the process quiescing until release becomes safe or the deadline requires crash recovery.
If the deadline expires or an effect is unknown, do not manufacture release proof. Termination enters existing typed crash recovery after fencing; unsupported states remain blocked. Foreground model-request and approval recovery already cover some checkpoints without Hook/MCP, but live streams and process memory do not migrate. Harness loss and Runtime loss have different evidence boundaries.
## Kubernetes lifecycle, probes and scaling
The Harness image needs a dedicated launcher, mounted Secret handling, probe helper and preStop helper. The current root image is a generic CLI image; the Java image has no Node workers. Pin image digests, use a non-root account, fixed operator HOME/configuration and temporary writable paths, and avoid mounting user Workspaces into the Harness Pod. Load broker credentials without putting raw secrets in Deployment arguments; the current CLI-argv token path is a packaging gap requiring an explicit launcher/configuration solution.
Probe through authenticated loopback calls. Startup verifies Hosted capability registration; readiness combines compatible identity, dependency admission and `ACCEPTING`; liveness only detects a wedged process, not a slow model or a transient database outage. Ordinary `/health` can be healthy during Hosted bootstrap and cannot substitute for readiness. Private probes must not print bearer material.
Run application quiesce before SIGTERM: current listener teardown is not a drain implementation. The proxy must stay alive for existing RPC/SSE during that interval. Prefer native sidecar shutdown ordering where supported; ordinary sidecars need both preStop paths coordinated on the same drain barrier. Declare the supported Kubernetes minor versions and test their actual ordering rather than requiring an unverified cluster upgrade. Hook time is inside `terminationGracePeriodSeconds`; abrupt node loss still requires crash recovery. See [sidecars](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) and [Pod termination](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination-flow).
Use a Deployment and a discovery Service/EndpointSlices, not a load-balanced address for individual Session RPCs. Configure rollout replacement capacity and compatibility explicitly. A PDB helps the eviction path but does not govern Deployment rolling updates or make replica reduction choose a resource-idle Pod. HPA and manual scale-in must tolerate any selected victim; draining one chosen Pod does not guarantee a later replica decrease deletes that Pod. See [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) and [disruption limits](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/).
Add Hosted-owned attached/active/quiescing counts, startable input backlog, held resources and pending publications. Reuse measured process RSS, heap and event-loop signals. Java's runnable queue excludes approval waits, unknown outcomes and work blocked by model/Runtime quotas. Do not use the current ACP bridge gauges or high-cardinality tenant labels. Metrics loss is unknown capacity, not zero work.
Initially set HPA `behavior.scaleDown.selectPolicy=Disabled` and a nonzero minimum. Enable automatic scale-out only with a working custom/external metrics adapter, bounded maximum replicas and measured per-Pod capacity. Do not add another replica controller. Automatic scale-in requires the full enabled-profile drain/forced-loss matrix and a stable metrics window. Replica count never bypasses tenant/actor admission limits or increases model, SQL or Runtime capacity. See [HPA metrics and behavior](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/).
## Schema rollout and local compatibility
The implementation follows the [single-V1 first-release contract](2026-10-23-managed-first-release-contract.md). The four placement fields, exact ownership constraint and owner/boot index are part of `V1__managed_schema.sql`; the database version stays 1 and there is no V2–V18 runtime migration chain. Server Flyway and Broker initialization use the same baseline. Schema projection/identity decoding and CSI inventory/recovery validation include the placement fields, which default to `UNASSIGNED` local semantics. Validate fresh Server-first and Broker-first initialization, restart preservation and refusal of incomplete or incompatible catalogs.
Earlier development databases and state are not upgraded. Preserve historical access with their original artifacts and deployment, or start this build with a separate fresh database and state directory. Match Server, Broker and Harness builds before enabling pool admission; the old binaries do not enforce placement grants. Do not run an old validator against this baseline or use Flyway repair, out-of-order migrations or history edits to bypass refusal. Rollback retains the original deployment and its own state; it is not an in-place downgrade of the new database.
The default local base URL remains one static instance with no Kubernetes token, discovery, mTLS requirement or Pod metadata. Shared correctness fixes need local regression, but `LocalProcessRuntimeProvisioner` retains its launch/ready/execution/cancellation/recovery contract. A Kubernetes Harness with a local Runtime requires separate network/storage acceptance: the Harness reaches Java/Store/Broker, while a same-host Broker can still consume a local worker's loopback endpoint. No general hybrid-topology acceptance is inferred.
## Implementation slices
| Slice | Reviewable deliverable and exit gate |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| K0: baseline and contract | Reproduce single-endpoint assumptions; freeze lifecycle/grant schemas and local defaults. Add no Kubernetes deployment claim. |
| K1: static pool and fencing | Per-instance clients, placement persistence/grants, all consumer routing and owner subscriptions; two local Harness processes, real MySQL and stale-owner reacquisition proof. |
| K2: Kubernetes connectivity | Dedicated Harness image/launcher, mTLS proxy and TLS client configuration, EndpointSlice discovery, probes and manual scale-out. Test two Pods and IP/boot/certificate changes. |
| K3: non-destructive release | Full no-start gating, resource-idle release proof, bounded preStop and arbitrary-victim/manual scale-in/rolling replacement. Retain explicit blocked recovery for unsupported states. |
| K4: measured autoscaling | Hosted metrics and adapter-backed automatic scale-out; automatic scale-in only after its own enabled-profile acceptance. Publish limits and operational recovery procedures. |
The implementation provides a static pool and Kubernetes discovery, dedicated launcher/probes, private instance metrics and conservative ownership release. Distribution artifacts live in `packages/managed-agent-server/distribution/kubernetes/`. Automatic scale-down remains disabled; manifests require operator-supplied images, TLS material, reachable Store/Broker origins and a metrics adapter.
## Implementation decisions, 2026-10-22
The refreshed main adds context compression and non-acquiring recovery observation. Both routes use the placement router; observational paths never allocate or take over an owner. Every leaf call retains its tenant, Session, epoch, descriptor and writer grant through completion. `OwnerSubscriptions` groups SQL lease proof by the attachment writer and tenant.
Close/delete recovery uses a separate placement path that carries the canonical lifecycle operation. It verifies the pending Session status, operation kind/version, current claim owner/generation/expiry and journal lifecycle fence before applying the same writer-expiry or release-proof requirements. Ordinary placement remains restricted to `ACTIVE` Sessions. Pool capacity, ownership handoff and process quiesce waits are exempt from execution retry budgets in Turn and lifecycle dispatch; configuration and credential failures retain their existing handling.
The router records admission eligibility for each observed instance/boot, including successful drain intent. If that owner stops accepting before its first writer is acquired, an accepting successor can obtain the placement only through the existing SQL takeover checks. If that extra handoff attempt returns exactly `409 harness_owner_still_live` while the original owner remains reachable and its SQL placement is `ACTIVE`, completion, approval and cancellation calls retain the original descriptor and grant; process admission still rejects fresh work. Other refusals do not fall back. A known-unreachable owner with no eligible successor returns `503 harness_pool_unavailable` before another RPC, retaining its placement without charging an execution retry. Discovery uncertainty alone does not make a cached reachable owner quiescing or unreachable.
Client retirement distinguishes terminal business Sessions from active ownership. A closed, archived or deleted root retains its old client only while SQL proves an active live writer; pending lifecycle roots stay retained. This avoids accumulating old boot clients across rolling updates when ordinary business close seals a writer without producing an ownership-release receipt. Lifecycle writer checks explicitly lock the Session root before the claim/head query.
Pool mode requires an HMAC binding key and an explicit instance identity. Grants use `qwp1` with scope, owner digest and placement epoch; credential authentication does not query SQL before tenant guards, and mutation paths recheck the current root under a lock before taking the head. The V1 baseline defaults local rows to `UNASSIGNED`. Existing private CSI configurations are refused in the first pool implementation, and CSI closure inventory explicitly requires unassigned placement.
Automation gate probe releases count as held resources even after the fire request has returned an error. A successful retry clears its release debt before an otherwise idle owner can hand off. If the bounded release-key cache overflows, the Session stays non-releasable until it closes: forgetting a key does not prove its Runtime was released.
The process no-start cutoff precedes SQL quiescing. Busy resident Sessions may keep SQL `ACTIVE` so their existing Hook activation transitions are not interrupted; the process gate still refuses fresh work. Only after the conservative local resource-idle check does the Harness call private Store `writers:quiesce-ownership`, stop activation renewal, release its attachment activation and call `writers:release-ownership`. The latter atomically writes a bounded `hosted_harness_release` control record/resource, seals the exact head and marks placement `RELEASED`. This terminal marker fences every old-epoch mutation before successor CAS and supports indexed owner pruning. It does not advance the journal or emit a business lifecycle event. Generic domain commits cannot forge this reserved receipt. The receipt pins its resource through `ma_resource_ref` and exact retries work after successor placement without reacquiring the old writer.
The SDK retains Java 11 compatibility and accepts an optional `SSLContext`. Pool configuration loads mounted PKCS12 key/trust stores; EndpointSlice HTTPS dialing still verifies the endpoint SAN. Static loopback endpoints support deterministic local tests. Discovery watches are bounded, relist after uncertainty and never substitute the local base URL.
PreStop initiates instance drain through the local endpoint and attempts only eligible ownership release. Durable release still requires the Session Store to be reachable. Unsupported resident resources keep the instance quiescing until the bounded deadline; ordinary detach is never called. Temporary ACK acceptance verified real idle-Session release, manual scale-out, complete rolling replacement and scale-to-zero. In-flight continuation, arbitrary Pod loss and adapter-backed HPA still need their respective acceptance gates.
## Verification plan
| Gate | Required evidence |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Local compatibility | Existing startup and single-address settings work without Kubernetes; create, submit, events, approval, cancellation, close and restart recovery pass. |
| Tenant/user isolation | Two tenants and two users within one tenant share each Harness; cross-scope IDs/handles/events/commands/approvals are refused. Cover CONTROL IDs and Session close isolation. |
| Pool ownership | Two live instances keep independent boots/caches and SQL-proven event subscriptions; every consumer in the routing table reaches the intended owner. |
| Store fencing | Reanimate a former Pod with its old scope token/grant after release and expiry. Acquire, renew, activation, commit and seal must fail; the successor still works. |
| SQL concurrency/baseline | Real-MySQL renew/commit/wake/lifecycle/release/takeover contention and crash boundaries; fresh Server-first/Broker-first V1 initialization, restart preservation and incompatible-catalog refusal. |
| TLS/proxy | Verify Host rewrite, browser refusal, bearer/protocol headers, SSE reconnect/epoch, bounded bodies, no mutation retry and trusted SANs. Reject wrong CA/client/SAN and stale Pod identity. |
| Discovery | List/watch reconnect, `410 Gone`, duplicate/dual-stack endpoints, Service UID change, API outage, readiness/termination transitions and reused IP; no localhost fallback. |
| Quiescence | Race the cutoff against all start sources. Pre-cutoff running Turn permits can finish their own foreground/model steps, while accepted unattempted backlog and independent child/autonomous work remain pending and start once on the successor. |
| Effect boundaries | Positive idle attachment-activation release and negative Hook/MCP/background/child/publication cases; lost release ACK reconciles SQL. A delayed old load between seal and successor CAS cannot reacquire the released epoch. Detach/business close is never used as migration. |
| Kubernetes loss | Actual preStop, SIGTERM, grace expiry, OOM/node-loss simulation, rolling update and arbitrary scale-in victim; original executions are observed and unknown outcomes stay blocked. |
| Scaling | HPA reads Hosted rather than ACP counts; distinguish runnable backlog from external bottlenecks, missing metrics and approval waits. Prove tenant limits and controlled expansion. |
Reuse existing packaged Harness tests and the three root scripts `npm run test:e2e:managed-harness-restart-failover`, `npm run test:e2e:managed-harness-restart-inflight-failover` and `npm run test:e2e:managed-harness-restart-continuation-failover` as regression foundations, not proof of Kubernetes support. Include the Java Hosted process/Hook/MCP fault profiles and MySQL integration lane, affected package tests, build/typecheck/bundle and repeated diff audits. Future executable plans belong in `.qwen/e2e-tests/`, with disposable databases, namespaces, credentials, storage and test-owned process identities.
The original research ran documentation checks and source review. Local implementation evidence is recorded in `.qwen/e2e-tests/managed-harness-kubernetes.md`. A newly provisioned, isolated ACK `1.35.7-aliyun.1` cluster with Terway `1.17.7`, Envoy `1.39.4` and MySQL `8.4.11` subsequently passed the idle-pool path. Java exercised two tenants across two Pods, mTLS and scope negatives, sticky placement, durable drain receipts and stale-grant rejection, manual expansion to three Pods, complete rolling replacement with epoch advancement and scale-to-zero. The final independent SQL audit retained ten ACTIVE Sessions, ten RELEASED placements and sealed writer heads, nineteen release receipts, zero Turns and zero lifecycle operations; the Java driver exited zero. The local report `.qwen/e2e-tests/managed-harness-ack-20261022.md` records immutable images, test-only Pod-IP SAN issuance, deployment adaptations, failed attempts and resource cleanup. These synthetic idle Sessions do not establish foreground/model/tool execution, SSE continuity, the complete user-authorization matrix, crash recovery, production certificate issuance, HPA or measured production capacity. Full CI gates remain pending.
## Deployment inputs and residual limits
Before cluster validation, resolve the actual Kubernetes minor version and sidecar behavior, namespace/service identity, mTLS issuer and per-endpoint SAN issuance, discovery freshness and drain budgets, CNI enforcement, metric adapter and measured capacity. Confirm Java/Store/Broker reachability and the chosen Runtime persistence domain. These are deployment inputs to an otherwise specified protocol, not reasons to treat the design as implemented.
Resource-idle release is the first supported handoff boundary. Further Hook/MCP/background and mixed-version continuation support requires its own fault evidence. Child cleanup under lifecycle protocols 1/2 uses the lifecycle placement path; historical protocol-0 parents with child cleanup debt require separate reachability and recovery acceptance before admitting that combination to the pool. Shared Harness processes remain trusted infrastructure; an in-process compromise can affect resident tenants, and this design does not provide Java control-plane HA or arbitrary-code isolation inside Harness.
The current integration follows main `32b4234b84` and its single-V1 first-release policy, explicitly authorized on 2026-10-23. Earlier placement-V17 ACK evidence and later V18 CI results belong to previous schema lineages; neither establishes acceptance of this fresh V1 baseline or the current merged head. New-head Java, real-MySQL/Hosted and cross-language gates remain required. The first pool does not expose MCP management APIs or public child-agent capability; single-instance mode retains main behavior.
docs/design/2026-10-21-managed-harness-kubernetes.zh-CN.md
+227
View File
@@ -0,0 +1,227 @@
# Kubernetes 共享 Hosted Harness 服务池
[English](2026-10-21-managed-harness-kubernetes.md) | [简体中文](2026-10-21-managed-harness-kubernetes.zh-CN.md)
状态：实现候选，2026-10-23，已衔接 main `32b4234b84` 及其首发单 V1 契约。最初源码审计：`97567d5dd733ad905aacf61f1f61a1e71c6fcc72`。历史 ACK 空闲服务池验收早于本基线；当前提交 CI、前台/故障和自动扩缩容门禁仍待完成。本文细化[共享多租户 Harness](2026-10-21-managed-shared-harness.zh-CN.md)。
## 范围与决策
使用一个活跃的 Java 管控面和共享 Harness Deployment。每个 Harness Pod 可以服务多个租户和用户。Java 负责授权、持久归属和派发；Kubernetes 负责 Pod 生命周期和副本数量。复用已有 Agent loop、公共 Session API 和 Runtime Broker 边界。
首个服务池支持兼容实例、固定 Session 归属、手动扩容和严格的崩溃恢复。在开放正常缩容前，先增加资源空闲 Session 的非破坏性释放。随后增加应用指标和自动扩容；在所有已启用执行 profile 都通过任意目标 Pod 终止测试前，保持自动缩容关闭。
Java 高可用、新 Tool Runtime provisioner、通用热迁移、缩容到零、租户自带模型 Key 和动态配置热加载不在本文范围。上层方案中的同进程租户隔离仍是前提，不是 Kubernetes 自动提供的能力。
## 已验证的起点
| 审计提交中的发现 | 源码与含义 |
| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Hosted 要求回环监听和 bearer 认证。 | [Profile:40](../../packages/cli/src/serve/hosted-harness-profile.ts)、[解析后地址检查:3539](../../packages/cli/src/serve/run-qwen-serve.ts)。Pod 不能直接将 Harness 暴露在 `0.0.0.0`。 |
| 回环请求有 Host 校验；远程 Store/Broker 默认要求 TLS。 | [Host 校验:340](../../packages/cli/src/serve/auth.ts)、[Broker URL:26](../../packages/cli/src/serve/managed-runtime-broker-url.ts)、[Store URL:844](../../packages/core/src/managed-runtime/http-managed-session-store.ts)。代理必须保留这些边界。 |
| Java 只有一个 SDK 客户端，并全局处理 boot 更替。 | [连接器:63,1557,1591](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/QwenHostedHarnessConnector.java)。通过 Service 随机负载均衡，会使无关附件失效。 |
| 所有者订阅使用单个 writer 身份。 | [订阅:130](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/session/OwnerSubscriptions.java)、[连接器契约:200](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/HarnessConnector.java)。需要逐附件携带所有者身份。 |
| 绑定 writer token 只覆盖租户、Workspace 和 Session，不覆盖实例归属。 | [凭据:57](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/security/WriterCredentialPolicy.java)、[writer 获取:670](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/persistence/ManagedSessionStore.java)。只更新路由 CAS，不能防止旧所有者重新获取资格。 |
| 普通 Hosted 注册不返回排空句柄；SIGTERM 会在宿主清理的同时关闭监听。 | [注册:10816](../../packages/cli/src/serve/hosted-harness-session.ts)、[关闭:193](../../packages/cli/src/serve/serve-app-lifecycle.ts)。默认监听关闭超时为五秒，与 Pod 宽限期独立。 |
| Detach 会取消部分待处理业务输入，并排空 Runtime 资源。 | [Detach:10269,10443](../../packages/cli/src/serve/hosted-harness-session.ts)。它不是所有权移交操作。 |
| Wake pump 可以独立于 HTTP 提交，启动已接受的积压输入。 | [Pump:213](../../packages/cli/src/serve/hosted-monitor-wake.ts)。只拒绝新建 Session，不能让 Pod 停止开工。 |
| 已有 Session/队列指标统计 ACP bridge，不统计 Hosted Session Map。 | [指标:7313](../../packages/cli/src/serve/run-qwen-serve.ts)、[Hosted Map:3663](../../packages/cli/src/serve/hosted-harness-session.ts)。已有 daemon 指标不能直接驱动 Harness HPA。 |
| 当前首发 schema 使用单个 V1 基线。 | [Schema 初始化](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/runtimebroker/managedschema/ManagedSchema.java)、[Flyway 策略](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/config/ManagedSchemaConfiguration.java)、[首发策略](2026-10-23-managed-first-release-contract.zh-CN.md)。旧开发库需要全新安装，不提供升级路径。 |
以上是源码观察。本轮读取了已有测试定义，没有执行这些测试。
## 拓扑与传输
```mermaid
flowchart LR
U["已认证用户"] --> J["Java：授权、归属与派发"]
J --> D[("SQL / Session Store")]
J --> P1["Pod A：mTLS 代理"]
J --> P2["Pod B：mTLS 代理"]
P1 --> H1["127.0.0.1:4170 Harness A"]
P2 --> H2["127.0.0.1:4170 Harness B"]
H1 --> S["TLS Store / Broker 入口"]
H2 --> S
S --> R["独立的 Workspace Tool Runtime"]
```
保留 Hosted 仅回环监听的 profile。通过平台管理的 Pod 内 mTLS 反向代理暴露私有 Pod 端点，以 Envoy 作为参考打包选择。代理只接受获准的 Java 服务身份，保留 Harness bearer 认证，将上游 Host 改为 `127.0.0.1:4170`，拒绝浏览器 Origin，仅开放获准私有路由。保留协议/boot 请求头、事件 ID 和 epoch。禁用缓冲、缓存和变更请求自动重试；流式超时必须容纳已有的 15 秒 SSE keepalive。参见 [Envoy TLS](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl)。
为 Java Store/Broker 提供真实 TLS 端点，或使用经过认证的本地出站代理。不开启非回环明文例外。为 Java Hosted SDK 增加明确、可选的 `SSLContext` 输入，由 Harness 配置用于服务池 mTLS；本地路径继续使用普通 JVM 默认值。保留主机名验证。从挂载文件读取客户端密钥和信任材料，首版通过受控替换轮换。参见 [JDK HTTP 客户端 TLS 契约](https://docs.oracle.com/en/java/javase/21/docs/api/java.net.http/java/net/http/HttpClient.Builder.html)。
各实例端点必须有有效的 TLS authority：直接连接 Pod IP，需要匹配的 IP SAN；使用 DNS 端点，需要匹配的证书及经过验证的 Pod 映射。不能假设 Deployment Pod 有 StatefulSet 式稳定 DNS，也不能通过关闭主机名验证来弥补。证书签发与 SAN 生命周期是明确的部署输入。反向代理是新增的受信任平台组件，不是不可信代码执行沙箱。
## 服务发现与实例身份
保留两种发现模式：默认使用现有静态单地址，显式配置后使用 namespace 范围的 Kubernetes 发现。静态多实例配置用于确定性测试。每个部署模式选择一种来源；Kubernetes 发现失败时不能回退到 localhost。
对配置的 Service 使用 `discovery.k8s.io/v1` EndpointSlice。校验 namespace、Service 名称和 UID、EndpointSlice 所有权、Pod 目标 UID 和端口。解析具体 HTTPS 端点，在选用前完成带认证的 boot/协议/能力协商。区分 Pod UID、进程 boot ID 和网络地址。复用 IP 或同一 Pod 内的新进程，都属于不同实例代次。
先 list，再从返回的 `resourceVersion` 开始 watch；对不同 Slice 和地址族中的端点去重。带类型的列表项可能省略 `kind` 和 `apiVersion`；接受这种表示，同时拒绝显式错误的 kind，并保留全部所有权校验。遇到 `410 Gone` 重新 list，watch 中断后有界退避重连，成功 list 后再发布完整替换快照。发现状态不确定时暂停新归属，保留现有绑定；可访问且认证通过的已有所有者可以继续运行。端点消失不能证明执行已经停止。终止中或未 ready 的端点不参与新分配，已有 draining 所有者仍可被定向访问。参见 [EndpointSlice 条件](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/)与 [watch 恢复](https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes)。
只有 Java 获得服务发现的 ServiceAccount token。授予 namespace 范围的 EndpointSlice list/watch 和必要的指定 Service 读取；客户端标签过滤不是 RBAC 授权边界。Harness 和代理不需要 Kubernetes API 凭据。使用支持该能力的 CNI 落实入站/出站策略，将用户 Tool Runtime 放在 Harness Pod 之外。参见 [ServiceAccount 挂载](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/)与 [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)。
## 持久归属与 Store 强制隔离
在现有 `ma_session` 根记录增加归属字段，不新增通用的持久 Pod 清单：
| 建议字段 | 含义与读取方 |
| ------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `harness_instance_id` | 已分配的静态实例身份或可信 Kubernetes Pod 身份；由路由和 Store 凭据校验读取。 |
| `harness_owner_boot_id` | 目标进程代次；由 SDK 和 Store 校验。已有 `harness_boot_id` 保持最后一次附件事件源观察值的含义。 |
| `harness_placement_epoch` | 单调递增的归属版本；用于 CAS、缓存、writer grant 和释放证明。 |
| `harness_placement_state` | `UNASSIGNED`、`ACTIVE`、`QUIESCING` 或终态 `RELEASED` 归属状态，不代表 Pod readiness 或公共 Session 状态。 |
增加有界的 owner/boot 索引，用于排空和重启时枚举归属，不能轮询扫描全部 Session。内存中的服务发现清单仍可丢弃。
未分配的既有本地记录没有服务池所有者。服务池模式下，Java 授权 Actor 及规范 Session/Workspace，选择兼容且接收工作的实例，在 create/load 前提交归属。每次重试都解析已有归属；RPC 失败不能直接选择另一个 Pod。归属作用于具备 Hosted 资格的根记录，包括明确准入的 CONTROL Session，并遵守各自类别的生命周期规则；没有 Harness 所有者的辅助 CONTROL 记录不分配实例。公共和 CONTROL Session ID 工厂必须保持私有 ID 全局唯一，或者在暴露缓存句柄前明确拒绝范围冲突。
使用新的实例归属绑定 writer grant，概念名称为 `qwp1`，复用已有 HMAC 机制，对租户、Workspace、Session、实例身份、boot ID 和 placement epoch 认证，由 Java 签发。Store 在 acquire、renew、commit、activation 和 seal，以及所有新工作授权路径中，针对当前持久归属校验 grant。已分配到服务池的记录拒绝旧的仅范围绑定 `qwt1` token。只有 token 内容、没有事务内当前归属校验，不足以隔离。普通数据读取保留已有认证和范围校验；公共调用方不能提供归属权限。
保留已有 writer generation/lease、activation 身份、Turn 派发租约和 Runtime 执行隔离。Placement epoch 不是另一套独立续期的租约。只有取得经过验证的非破坏性释放证明，或数据库证明 writer 已过期且满足现有恢复前提，才改变归属。归属更换后，即使处于 `SEALED` 或租约过期状态，也必须阻止旧 boot 重新取得资格。延迟到达的旧请求不能 seal、续租或写入继任者状态。
`QUIESCING` 仍允许已有活跃 writer 续租并提交截止前 Turn 的进度，也允许释放其 attachment activation；它不授权获取新 writer 或创建新 activation。终结性撤销从释放证明提交时开始，或由带隔离的崩溃路径改变归属时开始；不能仅因为标记 quiescing，就阻止已准入 Turn 完成。
受影响的 Store 事务必须遵循相同的先根后 head 规则：按需先取得已有 tenant/admission guard，然后锁 Session 根与归属，再锁 journal head，并保留既有依赖记录的锁顺序。Renew 当前从 head 开始，必须协调到该规则；不能持有 head 后再锁 Session 根。在真实 MySQL 中，对 acquire、renew、commit、wake claim、release、生命周期和 Runtime 隔离验证完整锁图。
## 路由与下游消费者
每个具体实例/boot 使用独立 SDK 客户端，并分别维护附件、heartbeat 和恢复缓存。每次操作只解析一次归属，直到响应结束都保持相同目标；boot 变化只使该代次失效。缓存键包含租户、Session 和 placement epoch。路由负责所有权，已有连接器继续实现 Hosted 协议。
| 消费者 | 必须遵循的归属行为 |
| ------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `HarnessCoordinator` | Create/load、submit、stream、continue、取消和接管使用同一个受隔离目标。 |
| `ManagedAgentService`、`SessionRenameCoordinator` | 内联和后台标题操作保留相同目标与持久命令身份。 |
| `ActionResponseCoordinator` | 审批响应进入当前活跃 Turn 所有者，不能在另一个 Pod 重建 Action。 |
| `ManagedAutomationService`、`AutomationScanner` | Fire、reconcile 和 cancel 保留 Automation 自身的 principal 和所属 Session。 |
| `ManagedChannelService`、`SessionMessageRelay` | Receive/consume/delivery 回执按各目标 Session 路由，回放已有命令身份。 |
| `ChildResultRelay`、`TaskCancelCoordinator` | 分别解析父子 Session；两者可能位于不同 Pod。 |
| `SessionLifecycleCoordinator` | 业务 close/delete/resource 阶段与 Pod 所有权释放保持分离。 |
| `WakeRecovery`、`TransportWakeActivator` | 在启动后台工作前校验归属与 quiesce 状态。 |
| `OwnerSubscriptions` | 逐附件携带 writer 身份，按 writer 和租户分组查询 SQL，并保留活跃 writer 的 SQL 证明。 |
MCP 和 Hook pins 继续通过 Session create/load 及现有 Runtime 边界传递，不构成第二套路由权威。已有无 Session 参数的 `supports*` 判断，使用整个服务池统一要求的同构能力契约；每个被选 Pod 都必须满足。绑定 Session 的操作校验真实所有者，不能因某个副本支持某项能力就授权请求。首版采用单一 engine digest 和兼容的恢复读取器，不支持异构引擎路由。
## 新增私有生命周期契约
以下是 pool 实例通过显式配置启用的私有端点。前两个接口和指标属于进程全局范围，release 属于活跃 Session 所有者范围。它们只在 Hosted 模式启用，采用服务认证和版本化契约，不通过浏览器/公共路由开放。
| 建议路由 | 契约 |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `GET /internal/hosted-harness/v1/instance` | 返回可信实例/boot 身份、生命周期、能力身份及 Hosted 自身的容量计数。不获取 Session，不启动工作。 |
| `POST /internal/hosted-harness/v1/instance:drain` | 绑定操作 ID 和预期 boot 的幂等排空意图，停止参与新分配，并建立禁止新开工的截止点。不关闭公共 Session。 |
| `POST /session/:id/ownership:release` | 校验规范范围、附件、placement epoch 和 writer generation；仅释放已证明资源空闲的 Session，不取消待处理输入。 |
| `GET /internal/hosted-harness/v1/metrics` | 向获准监控提供有界的 Hosted 专属指标，不包含 Prompt、凭据或租户/Session 指标标签。 |
启用服务池归属前，必须协商得到明确的 `harness-pool-owner/1` 和 `harness-instance-lifecycle/1` 能力。新增字段进入私有 DTO/schema/客户端 fixture 和共享契约测试；不重新解释已有 protocol-v1 envelope。旧 Harness 进程仍可用于明确配置的本地单地址模式，但不能进入服务池。
授权和精确已提交回执的回放，先于对新工作的拒绝。Drain 拒绝必须明确表示没有发生新的准入。已准入工作需要的状态、事件流、原始执行观察、审批和取消继续可用。截止点限制的是新的顶层工作；已经运行的 Turn 保留下文定义的续跑许可。没有既有许可的恢复，不能在 quiesce 实例上启动模型或工具工作。
如果准入在恢复等待持久化读取或 Runtime 获取期间关闭，派发守卫会在既有租约清理后返回 `409 hosted_harness_quiescing`。这仍属于执行重试预算之外的 pool 等待。驻留 writer 的写入已停止状态仍优先于 quiescing。`RecoveryDeclined` 继续只表示确定性的持久状态拒绝，其他恢复失败保留原有预算处理。
## Quiesce 与释放状态机
```mermaid
stateDiagram-v2
[*] --> STARTING
STARTING --> ACCEPTING: 身份和依赖就绪
ACCEPTING --> DRAINING: 停止新分配
DRAINING --> QUIESCING: 原子禁止新开工
QUIESCING --> DRAINED: 全部 Session 所有权释放已证明
QUIESCING --> RECOVERY_REQUIRED: 超时或效果未知
DRAINED --> [*]
RECOVERY_REQUIRED --> [*]: 有界终止与带隔离恢复
```
Java 可以将新授权输入保存在有界持久队列中，但不能派发到正在 quiesce 的所有者。在原子截止点，每个已经运行的 Turn 保留进程内续跑许可，绑定原 Prompt、activation 和 placement epoch；它可以执行结算同一个 Turn 所需的后续模型轮次、前台工具步骤和审批续跑。待处理输入、冷恢复、独立调度的子任务及自主 Channel/消息/Automation 工作不能继承该许可，也不能开工；持久身份和 wake 意图留给继任者。因此，运行中的 Turn 若依赖独立调度的子任务，可能直到截止时间都无法释放；首个释放验收使用有界前台 profile。仅返回没有活跃 Prompt，不是停止开工的证明。
首版仅释放没有执行中 Turn、在途变更、未解决工具/模型效果、持有 Runtime 操作、活跃 Hook/MCP 工作、存活后台 Shell/Monitor/子任务或待发布输出的 Session。空闲附件通常仍持有活跃 activation；协议应在禁止新开工的屏障内释放该 attachment activation，而不是要求它事先不存在。空闲但已配置的资源，也需要已证明的重连/释放契约；否则该 Session 不具备释放资格。冷加载需要的不可变资源必须已持久化且可读。静态 Skill 本身不要求执行进程。
通过已有私有 Session-record 机制，使用有界 `hosted_harness_release` domain 记录，持久化释放操作身份与证明，绑定 placement epoch、boot、writer generation 及 journal revision/sequence。释放 activation 并证明禁止新开工后，seal 与释放证明发布必须共享 Store 的受隔离事务。已提交的释放立即、终结性地撤销该 placement epoch 的 acquire 资格，即使 Java 尚未改变归属；否则在途旧 load 仍可能重新打开 sealed head。可以基于已有原语增加专用 Store 操作，不能假设 `ManagedSession.close()` 已实现完整契约。不产生业务 SessionEnd/Delete，不取消已接受的积压输入。
明确扩展 Java/TypeScript 封闭的 domain 和 resource-schema 注册、私有投影/保留与 CSI 闭包读取器。不能将该控制回执硬套成 task/run 记录，也不能公开为业务任务。已提交证明必须在旧进程退出后保留，直到归属对账和安全保留流程完成；无需增加通用持久化子系统。
ACK 丢失时，Java 用 SQL 和 sealed head 对账原释放身份，不能为了观察而在旧 Pod 重新获取 writer。继任者 CAS 必须在先根后 head 的锁顺序下，原子重新校验精确证明、sealed head/revision、当前归属和规范根记录适用的准入/生命周期规则；适用于具备资格的 PUBLIC 和 CONTROL 根。随后 `OwnerSubscriptions` 和附件缓存跟随新的 SQL 已证明 writer。RPC 成功或本地停止续租，不是释放证明。`DRAINED` 要求剩余常驻所有者和在途工作均为零；不具备释放资格的 Session 会让进程继续 quiesce，直到安全释放或到期转入崩溃恢复。
超时或效果未知时，不伪造释放证明。终止后经过隔离，使用现有带类型的崩溃恢复；不支持的状态继续阻塞。已有前台模型请求和审批恢复，覆盖部分不含 Hook/MCP 的 Checkpoint，但实时流和进程内存不能迁移。Harness 丢失与 Runtime 丢失具有不同的证据边界。
## Kubernetes 生命周期、探针与扩缩容
Harness 镜像需要专用 launcher、挂载 Secret 读取、探针 helper 和 preStop helper。当前根镜像是通用 CLI 镜像；Java 镜像没有 Node worker。镜像按 digest 固定，使用非 root 账号、固定的平台 HOME/配置和临时可写目录，不将用户 Workspace 挂载到 Harness Pod。加载 Broker 凭据时不将原始 secret 写进 Deployment 参数；当前 CLI argv token 路径是打包缺口，需要明确的 launcher/配置方案。
探针通过带认证的回环调用实现。Startup 验证 Hosted 能力已注册；readiness 综合兼容身份、依赖准入和 `ACCEPTING` 状态；liveness 只检测进程卡死，不因模型缓慢或数据库短暂故障而失败。普通 `/health` 在 Hosted 启动阶段就可能正常，不能替代 readiness。私有探针不能打印 bearer 材料。
在 SIGTERM 前运行应用 quiesce：当前监听关闭不是排空实现。代理必须在此期间继续承接已有 RPC/SSE。集群支持时优先采用原生 sidecar 退出顺序；普通 sidecar 的两条 preStop 路径必须在相同 drain barrier 上协调。声明支持的 Kubernetes 次版本并验证实际顺序，不要求未经验证的集群升级。Hook 时间计入 `terminationGracePeriodSeconds`；节点突然丢失仍需要崩溃恢复。参见 [sidecar](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/)与 [Pod 终止](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination-flow)。
使用 Deployment 和发现用 Service/EndpointSlice，不能把单个 Session RPC 发往随机负载均衡地址。明确配置滚动发布的替代容量与兼容要求。PDB 有助于 eviction 路径，但不约束 Deployment 滚动更新，也不能让副本减少时自动选择资源空闲的 Pod。HPA 和手动缩容必须容忍任意被选目标；先排空某个指定 Pod，不保证随后减少副本数时删除的就是它。参见 [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)与 [中断限制](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)。
增加 Hosted 自身的 attached/active/quiescing 计数、可开工输入积压、持有资源和待发布输出指标。复用实测的进程 RSS、heap 和事件循环信号。Java 的可执行队列排除等待审批、结果未知和受模型/Runtime 配额阻塞的工作。不使用当前 ACP bridge 指标，也不添加高基数租户标签。指标丢失代表容量未知，不代表工作量为零。
首版将 HPA `behavior.scaleDown.selectPolicy=Disabled`，保持非零最小副本数。只有自定义/外部指标适配器可用、最大副本数有界、每 Pod 容量已实测，才开放自动扩容。不增加第二个副本控制器。自动缩容必须通过全部已启用 profile 的排空/强制丢失矩阵，并具备稳定指标窗口。副本数量不能绕过租户/Actor 准入上限，也不会增加模型、SQL 或 Runtime 容量。参见 [HPA 指标与行为](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/)。
## Schema 发布与本地兼容
实现遵循[首发单 V1 契约](2026-10-23-managed-first-release-contract.zh-CN.md)。四个 placement 字段、精确归属约束和 owner/boot 索引直接进入 `V1__managed_schema.sql`；数据库版本保持 1，不保留运行时 V2–V18 迁移链。Server 的 Flyway 和 Broker 初始化使用相同基线。Schema projection/身份解码及 CSI inventory/恢复校验包含 placement 字段，默认保留 `UNASSIGNED` 本地语义。验证 Server 先启动、Broker 先启动的全新初始化、重启保留，以及不完整或不兼容 catalog 的拒绝。
旧开发数据库和状态不升级。需要历史访问时保留原产物和部署，或为本构建使用独立的全新数据库和状态目录。启用 pool 准入前，Server、Broker 和 Harness 必须使用匹配构建；旧二进制不执行 placement grant 校验。不能在该基线上运行旧校验器，也不能使用 Flyway repair、乱序迁移或历史编辑绕过拒绝。回退保留原部署及其独立状态，不对新数据库原地降级。
默认本地 base URL 继续表示一个静态实例，无需 Kubernetes token、服务发现、mTLS 配置或 Pod 元数据。共用正确性修复需要本地回归，但 `LocalProcessRuntimeProvisioner` 保留启动/ready/执行/取消/恢复契约。Kubernetes Harness 配合本地 Runtime，需要单独验证网络和存储：Harness 访问 Java/Store/Broker，同机 Broker 仍可消费本地 worker 回环端点。不能由此推断通用混合拓扑已经验收。
## 实施切片
| 切片 | 可审查交付物与退出条件 |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| K0：基线与契约 | 复现单地址假设，冻结生命周期/grant schema 和本地默认行为。不宣称已支持 Kubernetes 部署。 |
| K1：静态池与隔离 | 独立实例客户端、持久归属/grant、全部消费者路由和所有者订阅；使用两个本地 Harness 进程、真实 MySQL，证明旧所有者无法重新取得资格。 |
| K2：Kubernetes 连通 | 专用 Harness 镜像/launcher、mTLS 代理和 TLS 客户端配置、EndpointSlice 发现、探针及手动扩容。验证两个 Pod 和 IP/boot/证书变化。 |
| K3：非破坏性释放 | 全来源禁止新开工、资源空闲释放证明、有界 preStop，以及任意目标/手动缩容/滚动替换。对不支持的状态继续明确阻塞恢复。 |
| K4：实测自动扩缩容 | Hosted 指标和适配器驱动的自动扩容；自动缩容必须另行通过自身已启用 profile 验收。发布容量上限和运维恢复流程。 |
实现提供静态池、Kubernetes 发现、专用 launcher/probe、私有实例指标以及保守的归属释放。分发产物位于 `packages/managed-agent-server/distribution/kubernetes/`。自动缩容保持关闭；部署文件需要运维提供镜像、TLS 材料、可达的 Store/Broker 地址和指标适配器。
## 实现决策，2026-10-22
刷新后的 main 增加了上下文压缩与不获取所有权的恢复观察。两类调用均通过归属路由；观察路径不会分配或接管所有者。每次 leaf 调用在完成前固定 tenant、Session、epoch、descriptor 和 writer grant。`OwnerSubscriptions` 按附件的 writer 和 tenant 分组验证 SQL 租约。
关闭/删除恢复使用携带规范 lifecycle operation 的独立归属路径。该路径先核验 Session pending 状态、operation kind/version、当前 claim owner/generation/expiry 和 journal lifecycle fence，再执行同样的 writer 过期或释放证明检查。普通归属仍只允许 `ACTIVE` Session。Turn 和 lifecycle 派发将 pool 容量、归属交接和进程 quiesce 等待排除在执行重试预算外；配置与凭证失败保留原有处理。
路由器按观测到的 instance/boot 记录准入资格，也记录成功发起的 drain 意图。如果原实例在首次获取 writer 前停止接纳，接受新工作的继任者仍必须通过既有 SQL takeover 校验才能获得归属。如果这次额外交接尝试明确返回 `409 harness_owner_still_live`，且原实例仍可达、SQL placement 仍为 `ACTIVE`，收尾、审批和取消调用继续携带原 descriptor 与 grant 访问原实例；进程准入仍拒绝新开工。其他拒绝不会回退。如果所有者已确认不可达且没有合格继任者，则在再次 RPC 前返回 `503 harness_pool_unavailable`，保留归属而不扣减执行重试预算。仅有发现不确定性，不会将缓存中仍可达的原所有者标为 quiescing 或不可达。
客户端回收区分业务终态 Session 与活动归属。关闭、归档或删除的根记录，仅在 SQL 证明仍有 active 且租约有效的 writer 时保留旧客户端；pending lifecycle 的根仍保留。这避免普通业务关闭只 seal writer、不产生归属释放回执时，在滚动更新中持续积累旧 boot 客户端。Lifecycle writer 校验在 claim/head 查询前显式锁住 Session 根。
Pool 模式要求 HMAC binding key 和显式实例身份。`qwp1` grant 绑定 scope、owner digest 和 placement epoch；凭证认证不会在 tenant guard 前查询 SQL，变更路径在锁住 root 后重新校验当前归属，再获取 head。V1 基线使本地行默认保持 `UNASSIGNED`。首版 pool 拒绝已有私有 CSI 配置，CSI closure inventory 也明确要求未分配归属。
自动化 gate 探测的释放债务计入持有资源，即使 fire 请求已经返回错误也不能交接。重试证明释放成功后，原本空闲的归属才可交接。如果有界释放键缓存溢出，该 Session 在关闭前始终不具备交接资格：遗忘键不代表对应 Runtime 已释放。
进程的禁止新开工截止点先于 SQL quiesce。忙碌的驻留 Session 可以保持 SQL `ACTIVE`，避免打断既有 Hook activation 转换；进程门禁仍拒绝新开工。只有通过保守的本地资源空闲检查后，Harness 才调用私有 Store `writers:quiesce-ownership`，停止 activation 续期、释放附件 activation，再调用 `writers:release-ownership`。后者在同一事务写入有界 `hosted_harness_release` 控制记录/资源、seal 精确 head，并将归属标为 `RELEASED`。这个终态标记在继任者 CAS 前就隔离所有旧 epoch 变更，也支持通过索引清理所有者客户端。不推进 journal、不产生业务生命周期事件。普通 domain commit 不能伪造这个保留回执。回执通过 `ma_resource_ref` 固定资源；即使继任者已经更新归属，精确重试也不会重新获取旧 writer。
SDK 保持 Java 11 兼容，并接受可选 `SSLContext`。Pool 配置从挂载文件读取 PKCS12 key/trust store；EndpointSlice 的 HTTPS 连接仍校验端点 SAN。静态回环端点支持确定性本地测试。发现 watch 有界，在不确定状态后重新 list，始终不会替换为本地默认地址。
PreStop 通过本地端点发起实例 drain，只尝试释放满足条件的归属。持久释放仍要求 Session Store 可达。不支持的驻留资源使实例保持 quiescing，直到有界期限；不会调用普通 detach。临时 ACK 验收已验证真实空闲 Session 释放、手动扩容、完整滚动替换及缩容至零。在途续跑、任意 Pod 丢失及接入指标适配器的 HPA 仍需各自的验收门禁。
## 验证计划
| 验收项 | 必须提供的证据 |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 本地兼容 | 无 Kubernetes 时，已有启动方式和单地址设置可用；创建、提交、事件、审批、取消、关闭和重启恢复通过。 |
| 租户/用户隔离 | 两个租户及同租户两个用户共享各 Harness；拒绝跨范围 ID/句柄/事件/命令/审批。覆盖 CONTROL ID 和关闭 Session 的隔离。 |
| 服务池归属 | 两个同时存活的实例保持独立 boot/缓存和 SQL 已证明的事件订阅；路由表中的全部消费者都进入正确所有者。 |
| Store 隔离 | 释放和租约过期后，恢复旧 Pod 并携带原 scope token/grant；acquire、renew、activation、commit 和 seal 全部失败，继任者仍正常。 |
| SQL 并发/基线 | 真实 MySQL 的 renew/commit/wake/生命周期/release/takeover 竞争及崩溃边界；Server 先启动、Broker 先启动的全新 V1 初始化、重启保留和不兼容 catalog 拒绝。 |
| TLS/代理 | 验证 Host 重写、浏览器拒绝、bearer/协议请求头、SSE 重连/epoch、有界请求体、不重试变更请求和可信 SAN。拒绝错误 CA/客户端/SAN 和过期 Pod 身份。 |
| 服务发现 | List/watch 重连、`410 Gone`、重复/双栈端点、Service UID 变化、API 中断、ready/终止变化和 IP 复用；不回退 localhost。 |
| 停止开工 | 将截止点与所有开工来源并发竞争。截止前运行的 Turn 许可允许完成自身前台/模型步骤；已接受但未尝试的输入和独立子任务/自主工作保持 pending，在继任者只启动一次。 |
| 效果边界 | 空闲附件 activation 释放正例与 Hook/MCP/后台/子任务/发布负例；释放 ACK 丢失通过 SQL 对账。在 seal 与继任者 CAS 之间延迟到达的旧 load，不能重新获取已释放 epoch。不使用 detach/业务关闭实现迁移。 |
| Kubernetes 丢失 | 真实 preStop、SIGTERM、宽限期超时、OOM/节点丢失模拟、滚动更新和任意缩容目标；观察原始执行，未知结果继续阻塞。 |
| 扩缩容 | HPA 读取 Hosted 而非 ACP 计数；区分可执行积压、外部瓶颈、指标缺失和等待审批。证明租户限制和受控扩容。 |
复用已有打包 Harness 测试及三个根脚本 `npm run test:e2e:managed-harness-restart-failover`、`npm run test:e2e:managed-harness-restart-inflight-failover` 和 `npm run test:e2e:managed-harness-restart-continuation-failover`，作为回归基础，不当作 Kubernetes 支持证明。纳入 Java Hosted process/Hook/MCP 故障 profile、MySQL 集成、受影响包测试、build/typecheck/bundle 和重复 diff 自审。未来可执行计划保存在 `.qwen/e2e-tests/`，使用一次性数据库、namespace、凭据、存储和测试拥有的进程身份。
最初调研执行了文档检查和源码审阅。本地实现证据记录在 `.qwen/e2e-tests/managed-harness-kubernetes.md`。随后在本次新建、隔离的 ACK `1.35.7-aliyun.1` 集群上，使用 Terway `1.17.7`、Envoy `1.39.4` 和 MySQL `8.4.11`，通过空闲服务池链路。Java 验证了两个租户跨两台 Pod、mTLS 与跨范围拒绝、粘性归属、持久 drain 回执与旧 grant 拒绝、手动扩至三副本、完整滚动替换及 epoch 递增、缩容至零。最终独立 SQL 审计确认十个 ACTIVE Session、十个 RELEASED placement 和已 seal 的 writer head、十九份释放回执、零 Turn 及零生命周期 operation；Java 验收入口以零退出码结束。本地报告 `.qwen/e2e-tests/managed-harness-ack-20261022.md` 记录不可变镜像、仅用于测试的 Pod IP SAN 签发、部署适配、失败尝试及资源清理。这些合成空闲 Session 不构成前台/模型/工具执行、SSE 连续性、完整用户授权矩阵、崩溃恢复、生产证书签发、HPA 或生产容量测量的证明。完整 CI 门禁仍待完成。
## 部署输入与剩余限制
集群验证前，确定实际 Kubernetes 次版本及 sidecar 行为、namespace/Service 身份、mTLS issuer 与逐端点 SAN 签发、发现新鲜度与排空预算、CNI 限制、指标适配器及实测容量。确认 Java/Store/Broker 可达性和所选 Runtime 持久化范围。这些是已明确协议的部署输入，不是将方案视为已实现的理由。
首个支持的移交边界为资源空闲释放。继续支持 Hook/MCP/后台和混合版本续接，分别需要故障证据。Lifecycle 协议 1/2 的子任务清理使用独立 lifecycle 归属路径；带有子任务清理债务的历史协议 0 父 Session，需要单独证明可达性和恢复路径后才能将这种组合纳入服务池。共享 Harness 进程仍是受信任基础设施，进程内被攻破可能影响常驻租户；本文不提供 Java 管控面高可用，也不提供 Harness 内任意代码隔离。
当前衔接遵循 main `32b4234b84` 的首发单 V1 策略，用户已于 2026-10-23 明确授权。早期 placement V17 的 ACK 证据及后续 V18 的 CI 结果属于此前的 schema 版本线，均不能证明本次全新 V1 基线或当前合并提交已验收。当前提交仍需 Java、真实 MySQL/Hosted 和跨语言门禁。首版 pool 不开放 MCP 管理 API 或公开子代理能力，单实例模式沿用 main 行为。
docs/design/2026-10-21-managed-shared-harness.md
+135
View File
@@ -0,0 +1,135 @@
# Shared Harness for multiple tenants and users
[English](2026-10-21-managed-shared-harness.md) | [简体中文](2026-10-21-managed-shared-harness.zh-CN.md)
Status: Implementation candidate, 2026-10-22. The Kubernetes design records the implemented boundaries and temporary ACK idle-pool acceptance. Original current-state observations refer to `97567d5dd7`; implementation is based on `74c67c9e94`. Production multi-tenant isolation, full CI and the remaining foreground/fault/scaling acceptance remain gated.
## Goal and scope
One Java control plane serves multiple tenants and users through a shared Hosted Harness service. The same Harness process may concurrently host tenant A's and tenant B's Sessions. Additional Harness replicas belong to the same shared pool; a tenant does not need a dedicated process or deployment.
Share the Agent implementation, service configuration and compute capacity. Keep authorization, conversation state, model policy, execution authority and resource accounting scoped to their owning Sessions and tenants. Multiple users within one tenant still require explicit access to Sessions and Workspaces.
The first delivery uses one Harness endpoint and operator-managed model credentials. Multi-replica scheduling follows after same-process isolation is demonstrated. Tenant-provided model credentials, a new identity provider, billing, and stronger isolation from a compromised Harness process are separate work.
## Current implementation
| Area | Observed behavior and remaining work |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Java identity | `AuthenticatedTenantActor` and `TenantContextFilter` bind an authenticated actor to a tenant. Public controllers pass both identities to services. Audit every exposed operation and replay path; a tenant header alone is insufficient authorization. |
| Harness Sessions | `hosted-harness-session.ts` keeps multiple resident Sessions in one process. Normal resident load compares tenant, Workspace and Store address. This is an existing foundation, not proof that every lifecycle and recovery path has the same checks. |
| Durable identity | Session Store identity contains `tenantId`, `workspaceId` and `sessionId`; Java attachment caches include tenant and Session. `WriterCredentialPolicy` can bind a writer credential to that scope. Writer lease and generation remain the execution fences. |
| Model configuration | Each Hosted model invocation constructs configuration from service settings and refreshes authentication. Skipping Workspace settings and environment loading helps, but reuse of general CLI configuration and authentication needs an audit for mutable process state and caches. |
| Harness routing | Java configures one Harness base URL and one client. Sequential process replacement exists; concurrent replica selection and durable Session routing require further work. |
| Runtime isolation | Broker scope includes tenant and Workspace, with Workspace or Session isolation classes. These logical scopes do not by themselves establish OS-level isolation between untrusted users. |
Source references: [identity filter](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/api/TenantContextFilter.java), [public controller](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/api/PublicAgentController.java), [Hosted Sessions](../../packages/cli/src/serve/hosted-harness-session.ts), [Hosted model](../../packages/cli/src/serve/hosted-harness-model.ts), [writer credentials](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/security/WriterCredentialPolicy.java), [connector](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/QwenHostedHarnessConnector.java), and [Runtime scope](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeScope.java).
## Deployment and trust boundary
```mermaid
flowchart TD
A["Tenant A: users A1 and A2"] --> J["Java API, authorization and admission"]
B["Tenant B: user B1"] --> J
J --> H["Shared Harness process: Sessions A1, A2 and B1"]
J --> S[("Session Store and events")]
H --> M["Operator-managed model service"]
H --> R["Runtime Broker"]
R --> RA["Tenant A Workspace Runtime"]
R --> RB["Tenant B Workspace Runtime"]
```
Java, Harness, Broker and persistence are trusted platform services. Users access Java's authenticated public surface; private Harness, Store and Broker routes use service authentication and scoped execution credentials. User-supplied code, command Hooks, MCP processes and filesystem tools execute in the resolved Runtime, not in the shared Harness host. Static Skills are Session-scoped instruction resources, not host-executed extensions.
An arbitrary-code or memory compromise of a shared Harness process can affect its resident tenants. Logical Session isolation does not provide process-compromise containment. This deployment model assumes trusted service code and excludes loading user-supplied in-process plugins. The Runtime deployment must separately demonstrate filesystem, process, credential and network confinement; trusted local-process or SSH deployment evidence does not establish that boundary for mutually untrusted users.
## Identity and authorization
Java derives `tenantId` and `actorId` from trusted authentication. It authorizes the requested Session, Workspace, operation and model before admission or replay. Lists, event streams, artifacts, approvals, cancellation and background operations must preserve the same access policy. Revoked access must stop subsequent reads and mutations; active streams follow the API's documented revalidation policy.
The existing canonical Session scope is `(tenantId, workspaceId, sessionId)`. Actor identity records who requested an action; it does not replace Session identity or make every user in the tenant an owner. Authorized collaboration may share a Workspace without sharing all conversation histories. Child Sessions, channels and automations retain their persisted execution principal and grants; background execution never inherits the most recent HTTP caller.
Java supplies the canonical scope and existing Store and Runtime descriptors. Harness attachment, lifecycle, continuation and recovery validate them before returning resident handles or performing work. A private client handle is scoped to its attachment and cannot substitute for public user authorization. Require bound writer credentials for the shared deployment and retain writer lease/generation checks; binding keys and unrestricted Broker credentials stay outside user Runtimes.
Keep existing globally generated private Session IDs where sufficient. An in-memory map keyed by that ID may remain if conflicting scope is rejected at every boundary. Do not add a user ID to every storage key or rewrite unrelated caches merely to change key shape. Any cache shared across Sessions must use a proven scope or contain only immutable, operator-owned data.
## Session configuration and credentials
History, instructions, Skills, model selection, approvals, pending executions and event cursors belong to the Session. Accepted Turns retain their durable model and policy snapshots across retries and recovery. A later user's request or configuration change cannot mutate an already admitted Turn.
Service HOME, `QWEN_HOME`, cwd, environment, proxy and TLS settings are fixed by the operator. A request must not switch them to impersonate a tenant. Audit the Hosted call chain through general CLI configuration, authentication, provider caches, logging and cleanup; fresh configuration objects alone do not prove the absence of shared mutable state. Session close only disposes that Session's resources.
The first delivery shares operator-managed model credentials. Java restricts the published model catalog per tenant, and model calls and usage retain Session/Turn attribution. No new credential store or model gateway is required for this scope. Tenant-provided keys remain unsupported until a separately validated request-scoped resolver or model gateway can supply them without mutating shared environment or credential files. Model policy must be enforceable for autonomous and recovered work as well as user Turns.
Runtime credentials remain scoped to the authorized tenant, Workspace, Session and operation as required by the existing protocol. Prompt text, user settings and model-generated arguments cannot select another tenant's Store, Runtime or credential. Logs retain correlation IDs while excluding raw credentials and unscoped conversation dumps.
## Capacity, ownership and replicas
Java enforces bounded queued work and active Turn limits globally, per tenant and per actor. Dispatch queued work round-robin among eligible tenants, respecting actor limits within each tenant; tenants do not reserve permanent compute. Account background work and Runtime capacity separately so parked Turns, MCP connections and command processes cannot consume unlimited resources. Admission and durable work ownership must survive restart; counters must reconcile with durable records. Shared Node memory and event-loop capacity have process-wide limits, not a fictional hard memory boundary for each Session. Choose numeric limits from load measurements.
For the first delivery, reuse the existing single Harness endpoint. One live writer owns each Session; losing a request or heartbeat does not authorize parallel execution. Existing recovery uses the original durable execution identities, and unknown tool outcomes remain blocked without replay.
For replicas, persist Session-to-instance routing before dispatch and maintain an independent client and boot identity per instance. New Sessions may select any compatible healthy replica regardless of tenant; all operations for a resident Session follow its owner. A draining instance stops accepting new Sessions while existing work follows its declared completion or fenced handoff policy. Preserve existing writer and dispatch fences during replacement. Pool routing is separate from same-process tenant isolation and must not be implemented as random per-request HTTP balancing.
## Kubernetes horizontal scaling
The detailed [Kubernetes pool design](2026-10-21-managed-harness-kubernetes.md) specifies the source-verified constraints, placement-bound Store grants, non-destructive release protocol and implementation gates. Current Hosted binding is loopback-only: the proposed first Pod uses an authenticated mTLS proxy to that loopback listener, preserving application bearer/Host/Origin checks. Kubernetes discovery and Deployment configuration alone are insufficient.
Deploy Java and Harness as independent workloads so Harness capacity can grow without adding a Java instance for every replica. Use a Harness Deployment with replaceable Pods, each serving Sessions from multiple tenants. Keep authoritative history, checkpoints and resource references in the configured external Session Store; local caches and temporary files are disposable. Workspace storage and Runtime ownership belong to the separate execution layer. This is our proposed topology; Kubernetes supplies [Deployment replica and rollout management](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/).
| Concern | Proposed responsibility |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Replica count | Kubernetes maintains the Harness Deployment; HPA can adjust its capacity after application lifecycle gates pass. |
| Instance discovery | Java discovers the selected Harness workload's endpoints and validates protocol and capability compatibility. Use EndpointSlice target identity and the Harness handshake to distinguish Pod UID, endpoint and process boot ID. A recycled IP is not the previous owner. |
| Session routing | Java persists the selected instance identity and routes submit, approval, cancel, events, lifecycle and recovery calls to that owner. Discovery does not change an existing Session's ownership. |
| Scaling signals | Publish runnable queue depth, active Turns and measured per-Pod load. Account for memory and event-loop pressure and for model/Runtime bottlenecks. Waiting for a model can leave CPU low; blocked or approval-waiting work must not cause unlimited scale-out. |
| Pod removal | The application drains ownership and preserves durable work; Kubernetes manages the bounded container shutdown. Removing a Pod does not close or delete the user's logical Sessions. |
Java needs an individually addressable owner endpoint. A headless Service can expose Pod addresses without platform load balancing; the application still needs instance identity, compatibility and ownership validation. Use narrowly scoped discovery permissions. Readiness controls eligibility for new assignments; a draining but reachable owner must remain addressable for its existing work. See [Kubernetes headless Services](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services).
Use the `autoscaling/v2` HPA API with an installed custom/external metrics adapter for application load. Start with automatic scale-out and `behavior.scaleDown.selectPolicy=Disabled`; enable automatic scale-in only after the drain and forced-termination gates pass. Set a nonzero minimum capacity for the first delivery. Application admission continues enforcing tenant and user limits regardless of Pod count. The metrics and autoscaler are proposed deployment work, not existing repository capabilities. See [HPA custom metrics and scaling behavior](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/).
A bounded application drain stops new assignments and establishes a cutoff for fresh top-level Turn/wake starts. Pre-cutoff running Turns retain scoped continuation permits; unattempted backlog and autonomous starts remain pending. Required status, cancellation and approval calls continue. Release the idle attachment activation and writer only after proving all relevant effects are safe, without cancelling pending inputs. A committed release revokes reacquisition for that placement epoch before successor CAS. Run this protocol through `preStop` before SIGTERM; current generic shutdown and detach are not safe handoff. `terminationGracePeriodSeconds` must cover the budget: Kubernetes counts hook execution within the grace period and can force termination afterward. A timed-out drain follows fenced crash recovery, preserving unknown outcomes. See [Pod termination](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination-flow).
Scale-out initially places new Sessions on added Pods; it does not migrate resident Sessions automatically. Rebalancing and normal scale-in use the same proven handoff protocol. Validate mixed-version compatibility during rolling updates and keep eligible replacement capacity available. Scaling Harness does not expand model-provider quotas, database throughput or Workspace Runtime capacity; those remain separately measured and controlled. This design does not establish multi-replica Java control-plane readiness.
## Local process compatibility
Harness placement and Tool Runtime provisioning are independent choices. Keep `QWEN_MANAGED_AGENT_HARNESS_BASE_URL` as the existing single-address local path. Shared routing can represent that path as one configured instance without Kubernetes discovery, API access or cluster credentials. Keep Pod metadata inside the Kubernetes discovery integration; generic ownership and recovery must also work for a local process with its Harness boot identity.
| Scenario | Required compatibility |
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Local Harness process | Reuse the same Hosted Agent loop and private HTTP protocol. Common fixes to Session scope, configuration, credentials and cleanup also affect this path and require regression coverage. Existing development and service startup must remain usable. |
| `local-process` Tool Runtime | Keep the existing Broker selection and `LocalProcessRuntimeProvisioner` launch, ready handshake, execution identity, cancellation and recovery contracts. Harness horizontal scaling does not require replacing this provisioner or installing Kubernetes locally. |
| Kubernetes Harness with a local-process Runtime | Validate network reachability and storage ownership for the chosen topology. Harness must reach Java's Store and Broker addresses. A worker's loopback endpoint can remain local when consumed by its same-host Broker; callbacks and resource URLs need validation in their actual caller's network namespace. This mixed topology needs its own acceptance evidence. |
Reference: [local Runtime provisioner](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/runtimebroker/provisioning/LocalProcessRuntimeProvisioner.java).
The shared multi-tenant deployment requires its stricter actor, writer-credential and capacity prerequisites. Preserve the existing explicitly trusted single-user local deployment's authentication and trust contract; locality must not silently exempt a deployment that actually serves multiple tenants. Any optional discovery and pool-lifecycle behavior stays outside the default single-address path. Source compatibility is a design requirement, not evidence that future common-code changes have already passed local regression or that local processes provide an untrusted-user sandbox.
## Delivery plan
| Slice | Deliverable and completion boundary |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1. Shared process | One Java instance and one real Harness process concurrently serve two tenants and two distinct actors within one tenant. Audit public/private authorization, configuration and authentication state, callbacks, caches and cleanup. Patch demonstrated gaps, require bound credentials, and enforce bounded admission. Keep the existing Agent loop and durable protocols wherever possible. |
| 2. Shared pool | Prove durable Session routing, per-instance clients, compatibility, health and bounded drain with two live Harness processes. Then discover the selected Kubernetes workload's endpoints while preserving instance identity and fenced replacement; tenants share the pool. |
| 3. Deployment acceptance | Demonstrate Runtime confinement, restart/recovery, noisy-neighbor limits and the packaged Kubernetes load envelope. Verify scale-out, rolling updates and forced termination before enabling automatic scale-in. Publish measured limits and outstanding gates. |
Before implementation, write the executable E2E plan and record the baseline through the repository's feature workflow. This design supplies acceptance cases, not a claim that the feature or test gates have run.
## Verification and acceptance
1. Run tenant A/users A1 and A2 and tenant B/user B1 concurrently through the same Java service and the same Harness PID/boot ID. Verify distinct histories, policies, events, checkpoints and Runtime owners. Process reuse is part of the evidence.
2. Attempt foreign Session, Workspace, artifact, approval, private handle, idempotency key and event cursor reuse. Include a deliberate private Session-ID collision and normal/lifecycle/recovery attachment paths. Unauthorized requests return no foreign data and cause no execution.
3. Verify permitted same-tenant collaboration and denied same-tenant access separately. Missing or mismatched authenticated actors fail before dispatch. Access revocation follows the documented read, stream and mutation semantics. Child, channel and automation work preserve their own grants.
4. Interleave distinct model selections and configuration under deterministic model fixtures. Confirm the right history and policy on every request, unchanged process environment/cwd and credential files, and correctly attributed usage. Closing one Session leaves the other tenant's requests, streams and resources intact.
5. Exercise configured per-actor, per-tenant and global admission limits. When multiple tenants have queued work, freed slots are dispatched fairly rather than monopolized by a flooding actor; rejected or queued requests do not start unaccounted work. Limits and ownership reconcile after restart, including parked and background work.
6. Restart the shared Harness while both tenants have durable work. Recover each Session only through its own writer and original execution identities. Unknown outcomes block; cancellation, lost replies and stale owners cannot replay effects or release another Session's resources. Repeat applicable cases with two live replicas for slice 2.
7. Demonstrate physical Runtime confinement with the selected deployment. A logical scope or separate cwd is insufficient evidence. Run affected package tests, Java verification and applicable MySQL/Hosted fault profiles, TypeScript build/typecheck, and full-diff audits before declaring implementation complete.
8. Scale the Harness Deployment from two to three Pods while both tenants execute. New Sessions can reach the new compatible Pod; existing Sessions retain their owner and one writer. Validate actual HPA metrics, identity after endpoint reuse, incompatible Pod exclusion, and pending work when model or Runtime capacity is the bottleneck.
9. Drain and remove a busy Pod through scale-in and rolling replacement, including pending approvals and autonomous wake inputs. Verify lossless durable ownership handoff where supported and explicit blocked recovery otherwise. Repeat with drain-budget expiry and abrupt Pod loss; prove that neither Kubernetes termination nor endpoint removal causes prompt/tool replay, logical Session deletion or release of a newer owner's Runtime.
10. With no Kubernetes API or credentials available, run the existing local startup path with one Harness address and a `local-process` Runtime. Verify create, submit, events, approval, cancel, restart/recovery and close; confirm no discovery dependency is introduced. Test local multiple-Harness routing separately when enabled, and record mixed-topology network/storage acceptance independently.
## Decisions still requiring deployment input
The trusted authentication adapter, Runtime confinement backend, metrics adapter, drain budget and numeric capacity limits must be supplied or measured for a concrete deployment. They do not change the decision to share Harness processes across tenants. Per-tenant model credentials, if later required, need a separate credential-lifecycle design before admission is enabled.
docs/design/2026-10-21-managed-shared-harness.zh-CN.md
+135
View File
@@ -0,0 +1,135 @@
# 多租户、多用户共享 Harness
[English](2026-10-21-managed-shared-harness.md) | [简体中文](2026-10-21-managed-shared-harness.zh-CN.md)
状态：实现候选，2026-10-22。Kubernetes 设计记录了实现边界和临时 ACK 空闲服务池验收。最初当前状态观察基于 `97567d5dd7`，实现基于 `74c67c9e94`。生产级多租户隔离、完整 CI 及剩余前台/故障/扩缩容验收仍需门禁。
## 目标与范围
一个 Java 管控面通过共享的 Hosted Harness 服务承接多个租户和用户。同一个 Harness 进程可以同时承接租户 A 和租户 B 的 Session。增加的 Harness 副本属于同一个共享池；每个租户无需独占进程或部署。
共享 Agent 实现、服务配置和计算容量。授权、对话状态、模型策略、执行权限和资源计量分别归属于相应的 Session 和租户。同一租户内的不同用户仍需要明确的 Session 和 Workspace 访问授权。
首个交付使用一个 Harness 地址和平台管理的模型凭据。在证明同进程隔离之后，再增加多副本调度。租户自带模型凭据、新身份提供方、计费以及共享 Harness 进程被攻破后的更强隔离，分别作为后续工作。
## 当前实现
| 范围 | 已观察行为与剩余工作 |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Java 身份 | `AuthenticatedTenantActor` 和 `TenantContextFilter` 将认证用户绑定到租户。公共 Controller 将两个身份传给服务。需要逐一核查所有开放操作和回放路径；单独的租户请求头不足以授权。 |
| Harness Session | `hosted-harness-session.ts` 在同一进程中保存多个常驻 Session。普通常驻加载会比较租户、Workspace 和 Store 地址。这是已有基础，不能证明每条生命周期和恢复路径都有相同校验。 |
| 持久身份 | Session Store 身份包含 `tenantId`、`workspaceId` 和 `sessionId`；Java 附件缓存包含租户和 Session。`WriterCredentialPolicy` 可以将 writer 凭据绑定到该范围。writer lease 和 generation 继续作为执行隔离机制。 |
| 模型配置 | 每次 Hosted 模型调用从服务设置构造配置并刷新认证。跳过 Workspace 设置和环境加载有帮助，但复用通用 CLI 配置和认证路径，仍需核查可变进程状态与缓存。 |
| Harness 路由 | Java 配置一个 Harness 地址和一个客户端。已有进程顺序替换能力；并发副本选择和持久 Session 路由仍需补齐。 |
| Runtime 隔离 | Broker 范围包含租户和 Workspace，并区分 Workspace 或 Session 隔离类别。这些逻辑范围本身不能证明不可信用户之间的操作系统隔离。 |
源码参考：[身份过滤器](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/api/TenantContextFilter.java)、[公共 Controller](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/api/PublicAgentController.java)、[Hosted Session](../../packages/cli/src/serve/hosted-harness-session.ts)、[Hosted 模型](../../packages/cli/src/serve/hosted-harness-model.ts)、[writer 凭据](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/security/WriterCredentialPolicy.java)、[连接器](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/QwenHostedHarnessConnector.java)和 [Runtime 范围](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeScope.java)。
## 部署与信任边界
```mermaid
flowchart TD
A["租户 A：用户 A1 和 A2"] --> J["Java API、授权与准入"]
B["租户 B：用户 B1"] --> J
J --> H["共享 Harness 进程：Session A1、A2 和 B1"]
J --> S[("Session Store 与事件")]
H --> M["平台管理的模型服务"]
H --> R["Runtime Broker"]
R --> RA["租户 A Workspace Runtime"]
R --> RB["租户 B Workspace Runtime"]
```
Java、Harness、Broker 和持久化服务属于受信任的平台服务。用户通过 Java 的认证公共入口访问；Harness、Store 和 Broker 私有接口使用服务认证和限定范围的执行凭据。用户提供的代码、命令 Hook、MCP 进程和文件工具在解析得到的 Runtime 内执行，不在共享 Harness 宿主上执行。静态 Skill 是限定到 Session 的指令资源，不是宿主内执行的扩展。
共享 Harness 进程发生任意代码执行或内存泄露时，其常驻租户可能同时受影响。Session 逻辑隔离不提供进程被攻破后的隔离保证。该部署模型信任服务代码，不加载用户提供的进程内插件。Runtime 部署必须单独证明文件、进程、凭据和网络限制；可信 local-process 或 SSH 部署的证据不能证明互不信任用户之间具备这些边界。
## 身份与授权
Java 从可信认证获得 `tenantId` 和 `actorId`，在准入或回放之前校验目标 Session、Workspace、操作和模型权限。列表、事件流、产物、审批、取消和后台操作必须遵循同一访问策略。撤销权限后，后续读取和修改必须停止；存量事件流遵循 API 已声明的重新鉴权策略。
已有规范 Session 范围为 `(tenantId, workspaceId, sessionId)`。Actor 身份记录谁发起了操作，不替代 Session 身份，也不使租户内每个用户自动成为所有者。获得授权的协作可以共享 Workspace，而不必共享全部对话历史。子 Session、Channel 和 Automation 保留已持久化的执行身份和授权；后台执行不能继承最近一次 HTTP 请求的用户身份。
Java 提供规范身份范围和现有 Store、Runtime 描述。Harness 在返回常驻句柄或执行工作之前，校验附件、生命周期、续接和恢复请求的身份。私有客户端句柄限定于其附件，不能代替公共用户授权。共享部署必须使用绑定范围的 writer 凭据，并保留 writer lease/generation 校验；绑定密钥和未限定范围的 Broker 凭据不得进入用户 Runtime。
已有全局生成的私有 Session ID 在足够满足边界时继续使用。按该 ID 索引的内存 Map 可以保留，前提是每个边界都拒绝范围冲突。无需给所有存储键增加用户 ID，也无需仅为改变键形状而重写无关缓存。跨 Session 缓存必须使用经过证明的范围，或仅保存平台拥有的不可变数据。
## Session 配置与凭据
历史、指令、Skill、模型选择、审批、待执行操作和事件游标归属于 Session。已接受的 Turn 在重试和恢复时保留持久模型与策略快照。后续用户请求或配置变化不能修改已准入的 Turn。
服务 HOME、`QWEN_HOME`、cwd、环境变量、代理和 TLS 设置由平台固定。请求不能通过切换这些进程状态来模拟租户身份。需要沿 Hosted 调用链核查通用 CLI 配置、认证、Provider 缓存、日志和清理；每次创建新的配置对象不能单独证明不存在共享可变状态。关闭 Session 只释放该 Session 的资源。
首个交付共享平台管理的模型凭据。Java 按租户限制已发布的模型目录，模型调用和用量保留 Session/Turn 归属。本范围无需新增凭据库或模型网关。租户自带 Key 暂不支持，直到单独验证的请求级解析器或模型网关能在不修改共享环境变量或凭据文件的前提下提供它们。模型策略必须能约束自主执行和恢复工作，而不只约束用户 Turn。
Runtime 凭据根据现有协议要求，限定到已授权的租户、Workspace、Session 和操作。Prompt 文本、用户设置和模型生成的参数不能选择另一个租户的 Store、Runtime 或凭据。日志保留关联 ID，同时排除原始凭据和未限定范围的对话转储。
## 容量、所有权与副本
Java 对全局、租户和 Actor 分别限制排队工作量和活跃 Turn 数。在具备执行条件的租户间轮询派发排队工作，同时遵守每个租户内的 Actor 上限；租户不永久预留计算资源。单独计量后台工作和 Runtime 容量，使等待中的 Turn、MCP 连接和命令进程不能无限占用资源。准入和持久任务所有权必须在重启后继续成立；计数必须能与持久记录对账。共享 Node 内存和事件循环容量使用进程级限制，不虚构每个 Session 都有硬内存边界。具体数值通过压测选择。
首个交付复用现有单 Harness 地址。每个 Session 只有一个有效 writer；请求或心跳丢失不能授权并行执行。现有恢复继续使用原始持久执行身份，工具结果未知时保持阻塞，不重放。
增加副本时，在派发之前持久化 Session 到实例的路由，并为每个实例维护独立客户端和启动身份。新 Session 可以选择任何兼容且健康的副本，不按租户独占；常驻 Session 的所有操作跟随其所有者。进入 draining 的实例停止接收新 Session，已有工作按声明的完成或带执行隔离的移交策略处理。实例替换保留现有 writer 和派发隔离机制。服务池路由与同进程租户隔离分别实现，不能用逐请求随机 HTTP 负载均衡代替。
## Kubernetes 水平扩展
[Kubernetes 服务池详细设计](2026-10-21-managed-harness-kubernetes.zh-CN.md)明确了源码已核实的限制、绑定归属的 Store grant、非破坏性释放协议和实施验收项。当前 Hosted 只允许回环监听；首个 Pod 方案通过带认证的 mTLS 代理访问该监听，并保留应用 bearer/Host/Origin 校验。只有 Kubernetes 发现和 Deployment 配置不足以完成接入。
Java 与 Harness 使用独立工作负载部署，扩展 Harness 时无需为每个副本增加 Java 实例。Harness 使用 Deployment，每个可替换 Pod 都承接多个租户的 Session。历史、Checkpoint 和资源引用的权威状态保存在配置的外部 Session Store；本地缓存与临时文件可丢弃。Workspace 存储和 Runtime 所有权属于独立执行层。这是本方案提出的拓扑；Kubernetes 提供 [Deployment 副本和滚动发布管理](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)。
| 范围 | 建议职责 |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 副本数量 | Kubernetes 维护 Harness Deployment；应用生命周期验收通过后，由 HPA 调整容量。 |
| 实例发现 | Java 发现指定 Harness 工作负载的端点并校验协议、能力兼容性。结合 EndpointSlice 目标身份和 Harness 握手，区分 Pod UID、端点和进程 boot ID。复用的 IP 不代表原所有者。 |
| Session 路由 | Java 持久化选定实例身份，向该所有者发送提交、审批、取消、事件、生命周期和恢复调用。发现新端点不改变已有 Session 的归属。 |
| 扩容信号 | 暴露可执行排队量、活跃 Turn 和实测 Pod 负载，考虑内存、事件循环压力及模型/Runtime 瓶颈。等待模型时 CPU 可能很低；阻塞或等待审批的工作不能导致无限扩容。 |
| Pod 移除 | 应用排空并移交所有权、保留持久任务；Kubernetes 管理有时间上限的容器退出。移除 Pod 不关闭或删除用户的逻辑 Session。 |
Java 需要能单独访问所有者端点。Headless Service 可以暴露 Pod 地址，不经过平台负载均衡；应用仍需校验实例身份、兼容性和所有权。服务发现权限限定到必要范围。Readiness 控制新分配资格；正在排空但仍可访问的所有者，必须继续承接已有工作的定向调用。参见 [Kubernetes Headless Service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services)。
使用 `autoscaling/v2` HPA API，通过已安装的自定义/外部指标适配器获取应用负载。先开放自动扩容，并设置 `behavior.scaleDown.selectPolicy=Disabled`；只有排空和强制终止验收通过后，才开放自动缩容。首个交付保持非零最小容量。无论 Pod 数量如何变化，应用准入始终执行租户和用户上限。指标与自动扩缩容属于计划中的部署工作，不是仓库已有能力。参见 [HPA 自定义指标与扩缩容行为](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/)。
有时间上限的应用排空流程停止新分配，并建立新的顶层 Turn/wake 开工截止点。截止前运行的 Turn 保留限定范围的续跑许可；未尝试的积压输入和自主开工保持 pending。必要的状态、取消和审批调用继续处理。只有证明全部相关效果安全，才释放空闲附件 activation 和 writer，不取消待处理输入。已提交释放必须在继任者 CAS 前撤销该 placement epoch 的重新获取资格。通过 `preStop` 在 SIGTERM 前执行此协议；当前通用关闭和 detach 不能安全移交。`terminationGracePeriodSeconds` 必须覆盖预算：Kubernetes 将 Hook 执行计入宽限期，超时后可能强制终止。排空超时转入带隔离的崩溃恢复，保留未知结果。参见 [Pod 终止流程](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination-flow)。
扩容初期只将新 Session 分配到新增 Pod，不自动迁移常驻 Session。重平衡和正常缩容使用同一套已证明的移交协议。滚动发布必须验证混合版本兼容性，并保留具备接管资格的替代容量。扩展 Harness 不会增加模型提供方配额、数据库吞吐量或 Workspace Runtime 容量；这些仍需分别测量和控制。本文不证明 Java 管控面已具备多副本部署条件。
## 本地进程兼容性
Harness 部署位置与 Tool Runtime 的创建方式是独立选择。保留 `QWEN_MANAGED_AGENT_HARNESS_BASE_URL`，作为已有的本地单地址路径。共享路由可以将其表示为一个配置实例，无需 Kubernetes 发现、API 访问或集群凭据。Pod 元数据只进入 Kubernetes 发现集成；通用所有权和恢复逻辑，也必须能处理具有 Harness 启动身份的本地进程。
| 场景 | 兼容要求 |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 本地 Harness 进程 | 复用同一套 Hosted Agent loop 和私有 HTTP 协议。Session 范围、配置、凭据和清理的共用修复同样影响此路径，必须回归验证。已有开发和服务启动方式继续可用。 |
| `local-process` Tool Runtime | 保留已有 Broker 选择和 `LocalProcessRuntimeProvisioner` 的启动、ready 握手、执行身份、取消与恢复契约。Harness 水平扩展不要求替换该 provisioner，也不要求本地安装 Kubernetes。 |
| Kubernetes Harness 配合本地进程 Runtime | 针对所选拓扑验证网络可达性和存储归属。Harness 必须能访问 Java 的 Store 和 Broker 地址。同机 Broker 消费的 worker 回环端点可以继续本地使用；回调和资源 URL 必须按实际调用方的网络命名空间验证。此混合拓扑需要独立验收证据。 |
参考：[本地 Runtime provisioner](../../packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/runtimebroker/provisioning/LocalProcessRuntimeProvisioner.java)。
共享多租户部署要求更严格的 Actor、writer 凭据和容量前提。保留已有明确受信任的单用户本地部署的认证与信任契约；不能仅因部署在本地，就悄然豁免实际服务多租户的部署。可选的发现和服务池生命周期行为不进入默认单地址路径。源码兼容是设计要求，不代表未来共用代码变更已经通过本地回归，也不代表本地进程提供了不可信用户沙箱。
## 交付计划
| 切片 | 交付物与完成边界 |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1. 共享进程 | 一个 Java 实例和一个真实 Harness 进程同时服务两个租户，以及同一租户中的两个不同 Actor。核查公共/私有授权、配置和认证状态、回调、缓存与清理。修复已证明的缺口，要求绑定凭据，并落实有界准入。尽可能复用现有 Agent loop 和持久协议。 |
| 2. 共享池 | 用两个同时存活的 Harness 进程证明持久 Session 路由、独立实例客户端、兼容性、健康和有时间上限的排空。随后接入指定 Kubernetes 工作负载的端点发现，保留实例身份和替换时的执行隔离；租户共享服务池。 |
| 3. 部署验收 | 证明 Runtime 限制、重启恢复、单租户过载限制和 Kubernetes 打包部署的负载范围。验证扩容、滚动发布和强制终止，再开放自动缩容。发布实测容量与未完成的验收项。 |
实现前，通过仓库的功能开发流程编写可执行 E2E 计划并记录基线。本文提供验收用例，不代表功能已经实现或测试已经运行。
## 验证与验收
1. 让租户 A 的用户 A1、A2 和租户 B 的用户 B1，通过同一个 Java 服务及相同 Harness PID/boot ID 并发执行。验证独立的历史、策略、事件、Checkpoint 和 Runtime 所有者。进程复用必须纳入证据。
2. 尝试复用其他范围的 Session、Workspace、产物、审批、私有句柄、幂等键和事件游标。覆盖刻意构造的私有 Session ID 冲突，以及普通、生命周期和恢复附件路径。未授权请求不能返回其他范围的数据或触发执行。
3. 分别验证同租户内允许的协作和被拒绝的访问。缺失或不匹配的认证用户必须在派发之前失败。权限撤销遵循已声明的读取、流和修改语义。子 Session、Channel 和 Automation 保留各自的授权。
4. 使用确定性模型 fixture 交错运行不同模型选择与配置。检查每次请求的历史和策略正确、进程环境/cwd 及凭据文件不变、用量归属正确。关闭一个 Session 不影响另一个租户的请求、事件流或资源。
5. 验证配置的 Actor、租户和全局准入上限。当多个租户有排队工作时，释放的执行名额公平派发，不能被高频请求的 Actor 独占；被拒绝或排队的请求不能启动未计量工作。重启后限制和所有权可对账，覆盖等待与后台工作。
6. 两个租户都有持久任务时重启共享 Harness。各 Session 只能通过自己的 writer 和原始执行身份恢复。未知结果保持阻塞；取消、响应丢失和过期所有者不能重放副作用或释放其他 Session 的资源。切片 2 使用两个同时存活的副本重复适用用例。
7. 在选定部署中证明 Runtime 的物理限制。逻辑范围或独立 cwd 不足以作为证据。在声明实现完成前，运行受影响包的测试、Java 验证及适用的 MySQL/Hosted 故障测试、TypeScript build/typecheck，以及完整 diff 自审。
8. 两个租户执行期间，将 Harness Deployment 从两个 Pod 扩为三个。新 Session 可以分配到兼容的新 Pod；已有 Session 保留原所有者和单一 writer。验证实际 HPA 指标、端点复用后的身份、不兼容 Pod 的排除，以及模型或 Runtime 容量成为瓶颈时的排队行为。
9. 通过缩容和滚动替换，排空并移除忙碌 Pod，覆盖待审批和自主 wake 输入。已支持的边界验证持久所有权无损移交，其余边界明确进入阻塞恢复。使用排空预算超时和 Pod 突然丢失重复测试；证明 Kubernetes 终止或端点移除不会造成 Prompt/工具重放、逻辑 Session 删除，或释放新所有者的 Runtime。
10. 在没有 Kubernetes API 或凭据的环境，使用已有本地启动路径、一个 Harness 地址和 `local-process` Runtime，验证创建、提交、事件、审批、取消、重启恢复和关闭，并确认未引入服务发现依赖。显式启用本地多 Harness 路由时单独测试；混合拓扑的网络/存储验收分别记录。
## 仍需部署输入的决策
具体部署必须提供可信认证适配器、Runtime 限制后端和指标适配器，并确定排空预算、测量容量上限。这些选择不改变跨租户共享 Harness 进程的决定。后续如果需要租户独立模型凭据，必须先完成凭据生命周期设计，再开放准入。
packages/cli/src/serve/hosted-automation-session.test.ts
+47
View File
@@ -1287,6 +1287,53 @@ function scriptedGate(results: HostedAutomationGateResult[]) {
const gated = { ...definition, gate: './probe.sh --gate' };
describe('hosted automation gates', () => {
it('retains unknown release debt after the release-key cache overflows', async () => {
const harness = await createHarness();
let failing = true;
const releaseGate = vi.fn(async () => {});
const occurrences = Array.from(
{ length: 1025 },
(_, index) =>
`schedule:${new Date(Date.UTC(2026, 2, 8 + index, 7))
.toISOString()
.replace('.000Z', 'Z')}`,
);
await withSession(
harness,
async (automations) => {
await automations.define({
scheduleId: SCHEDULE_ID,
operationId: nextOperationId(),
definition: gated,
});
expect(automations.hasPendingGateReleases).toBe(false);
for (const occurrence of occurrences) {
await expect(
automations.fire(fireParams(1, occurrence)),
).rejects.toThrow('unreleased Runtime');
}
failing = false;
for (const occurrence of occurrences.slice(1)) {
expect(
(await automations.fire(fireParams(1, occurrence))).gate,
).toEqual({
outcome: 'quiet',
durationMs: 0,
});
}
expect(releaseGate).toHaveBeenCalledTimes(1024);
expect(automations.hasPendingGateReleases).toBe(true);
},
{
gate: async () => {
if (failing) throw new Error('unreleased Runtime');
return { kind: 'quiet', durationMs: 0 };
},
releaseGate,
},
);
});
it('withholds a quiet fire with no claim, input or wake', async () => {
const harness = await createHarness();
const gate = scriptedGate([{ kind: 'quiet', durationMs: 12 }]);
packages/cli/src/serve/hosted-automation-session.ts
+7
View File
@@ -346,6 +346,7 @@ export class HostedAutomationSession {
* again to settle its entry.
*/
private readonly owedGateReleases = new Set<string>();
private gateReleaseDebtOverflowed = false;
constructor(
private readonly store: HostedAutomationStore,
@@ -353,6 +354,10 @@ export class HostedAutomationSession {
private readonly options: HostedAutomationSessionOptions = {},
) {}
get hasPendingGateReleases(): boolean {
return this.gateReleaseDebtOverflowed || this.owedGateReleases.size > 0;
}
schedule(scheduleId: string): Schedule | undefined {
const existing = this.store.authority.extensionRecord(
'schedule',
@@ -831,6 +836,8 @@ export class HostedAutomationSession {
this.owedGateReleases.delete(owed);
this.owedGateReleases.add(owed);
if (this.owedGateReleases.size > OWED_GATE_RELEASES_LIMIT) {
// Forgetting a release key cannot prove its Runtime was released.
this.gateReleaseDebtOverflowed = true;
this.owedGateReleases.delete(
this.owedGateReleases.values().next().value!,
);
packages/cli/src/serve/hosted-csi-session.ts
+1
View File
@@ -313,6 +313,7 @@ async function initialize(
activation,
releaseActivation: deny,
replaceActivation: deny,
prepareOwnershipRelease: deny,
close: deny,
};
owner.timer = setInterval(() => {
packages/cli/src/serve/hosted-harness-contract.ts
+9
View File
@@ -40,6 +40,9 @@ export function createHostedHarnessContract(
)
.split(',')
.filter(Boolean),
instanceId: string | undefined = process.env[
'QWEN_HOSTED_HARNESS_INSTANCE_ID'
],
): HostedHarnessContract {
if (
lifecycleV2Profiles.some(
@@ -60,7 +63,10 @@ export function createHostedHarnessContract(
if (!UUID_PATTERN.test(bootId)) {
throw new Error('Hosted Harness bootId must be an RFC UUID v1-v5.');
}
if (instanceId !== undefined && !/^[A-Za-z0-9._:-]{1,128}$/u.test(instanceId))
throw new Error('Invalid Hosted Harness instance identity.');
return Object.freeze({
...(instanceId === undefined ? {} : { instanceId }),
protocolVersions: Object.freeze({
current: HOSTED_HARNESS_PROTOCOL_VERSION,
supported: Object.freeze([HOSTED_HARNESS_PROTOCOL_VERSION] as const),
@@ -71,6 +77,9 @@ export function createHostedHarnessContract(
lifecycleV2Profiles: Object.freeze([...lifecycleV2Profiles]),
titleProtocolVersion: 1,
features: Object.freeze([
...(instanceId === undefined
? []
: ['harness-pool-owner/1', 'harness-instance-lifecycle/1']),
'hosted_approval_resume_v1',
'hosted_model_round_recovery_v1',
'model-selection/1',
packages/cli/src/serve/hosted-harness-instance.test.ts
+98
View File
@@ -0,0 +1,98 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/
import express from 'express';
import supertest from 'supertest';
import { describe, expect, it } from 'vitest';
import { createHostedHarnessContract } from './hosted-harness-contract.js';
import {
HostedHarnessInstance,
parseHostedHarnessPlacement,
} from './hosted-harness-instance.js';
const bootId = '11111111-1111-4111-8111-111111111111';
const operationId = '22222222-2222-4222-8222-222222222222';
const digest = `sha256:${'a'.repeat(64)}`;
function fixture() {
const app = express();
app.use(express.json());
const counters = {
attachedSessions: 1,
activeTurns: 0,
openingSessions: 0,
pendingInputs: 0,
heldResources: 0,
pendingPublications: 0,
};
const contract = {
...createHostedHarnessContract(digest, bootId, [], 'pod-a'),
instanceId: 'pod-a',
};
const instance = new HostedHarnessInstance(contract, () => counters);
instance.register(app);
return { app, counters, instance };
}
function headers(request: supertest.Test) {
return request
.set('X-Qwen-Harness-Protocol-Version', '1')
.set('X-Qwen-Harness-Boot-Id', bootId);
}
describe('Hosted Harness instance lifecycle', () => {
it('fences process identity and drains every owner before declaring DRAINED', async () => {
const { app, counters, instance } = fixture();
const route = '/internal/hosted-harness/v1';
expect((await supertest(app).get(`${route}/instance`)).status).toBe(426);
const drain = () =>
headers(supertest(app).post(`${route}/instance:drain`)).send({
operationId,
expectedBootId: bootId,
});
expect((await drain()).body.state).toBe('QUIESCING');
expect(instance.accepting).toBe(false);
expect((await drain()).status).toBe(202);
expect(
(
await headers(supertest(app).post(`${route}/instance:drain`)).send({
operationId: bootId,
expectedBootId: bootId,
})
).status,
).toBe(409);
counters.attachedSessions = 0;
counters.heldResources = 1;
expect(instance.snapshot().state).toBe('QUIESCING');
counters.heldResources = 0;
expect(instance.snapshot().state).toBe('DRAINED');
const metrics = await headers(supertest(app).get(`${route}/metrics`));
expect(metrics.headers['cache-control']).toBe('no-store');
expect(metrics.body.process.rssBytes).toBeGreaterThan(0);
expect(JSON.stringify(metrics.body)).not.toContain('tenantId');
});
it('requires a complete placement and leaves the local contract opt-in', () => {
expect(
createHostedHarnessContract(digest, bootId, [], undefined).features,
).not.toContain('harness-pool-owner/1');
expect(
parseHostedHarnessPlacement({
instanceId: 'pod-a',
bootId,
placementEpoch: 2,
}).placementEpoch,
).toBe(2);
for (const value of [
null,
{ instanceId: 'pod-a', bootId },
{ instanceId: '../pod', bootId, placementEpoch: 1 },
{ instanceId: 'pod-a', bootId, placementEpoch: 0 },
{ instanceId: 'pod-a', bootId, placementEpoch: 1, extra: true },
])
expect(() => parseHostedHarnessPlacement(value)).toThrow();
});
});
packages/cli/src/serve/hosted-harness-instance.ts
+150
View File
@@ -0,0 +1,150 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/
import { performance } from 'node:perf_hooks';
import type { Application, RequestHandler } from 'express';
import type { HostedHarnessPlacement } from '@qwen-code/qwen-code-core/managed-runtime/hosted-harness-release.js';
import {
hostedHarnessContractMiddleware,
type HostedHarnessContract,
} from './hosted-harness-contract.js';
const INSTANCE_ID = /^[A-Za-z0-9._:-]{1,128}$/u;
const UUID =
/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/iu;
export function parseHostedHarnessInstanceId(value: string): string {
if (!INSTANCE_ID.test(value))
throw new Error('Invalid Hosted Harness instance identity.');
return value;
}
export function parseHostedHarnessPlacement(
value: unknown,
): HostedHarnessPlacement {
if (!value || typeof value !== 'object' || Array.isArray(value))
throw new Error('Harness placement is required.');
const record = value as Record<string, unknown>;
if (
Object.keys(record).sort().join(',') !==
'bootId,instanceId,placementEpoch' ||
typeof record['instanceId'] !== 'string' ||
typeof record['bootId'] !== 'string' ||
!UUID.test(record['bootId']) ||
!Number.isSafeInteger(record['placementEpoch']) ||
(record['placementEpoch'] as number) < 1
)
throw new Error('Invalid Harness placement.');
return {
instanceId: parseHostedHarnessInstanceId(record['instanceId']),
bootId: record['bootId'],
placementEpoch: record['placementEpoch'] as number,
};
}
export interface HostedHarnessCounters {
attachedSessions: number;
activeTurns: number;
openingSessions: number;
pendingInputs: number;
heldResources: number;
pendingPublications: number;
}
export class HostedHarnessInstance {
private drainOperation?: string;
private operations = 0;
constructor(
readonly contract: HostedHarnessContract & { instanceId: string },
private readonly counters: () => HostedHarnessCounters,
private readonly onDrain?: () => void,
) {}
get accepting(): boolean {
return this.drainOperation === undefined;
}
get inFlightMutations(): number {
return this.operations;
}
track: RequestHandler = (req, res, next) => {
if (req.path.endsWith('/events') || req.path.endsWith('/ownership:release'))
return next();
this.operations += 1;
// A disconnected response cannot prove its asynchronous operation ended.
// Keep the owner non-releasable; the bounded shutdown takes the crash path.
res.once('finish', () => {
this.operations -= 1;
});
next();
};
snapshot() {
const counters = { ...this.counters(), inFlightMutations: this.operations };
const state = this.accepting
? 'ACCEPTING'
: counters.attachedSessions === 0 &&
counters.openingSessions === 0 &&
counters.activeTurns === 0 &&
counters.inFlightMutations === 0 &&
counters.heldResources === 0 &&
counters.pendingPublications === 0
? 'DRAINED'
: 'QUIESCING';
return {
format: 'qwen-hosted-harness-instance/1' as const,
instanceId: this.contract.instanceId,
bootId: this.contract.bootId,
capabilityDigest: this.contract.capabilityDigest,
state,
counters,
};
}
register(app: Application): void {
const prefix = '/internal/hosted-harness/v1';
app.use(prefix, (req, res, next) => {
res.setHeader('Cache-Control', 'no-store');
hostedHarnessContractMiddleware(this.contract)(req, res, next);
});
app.get(`${prefix}/instance`, (_req, res) => res.json(this.snapshot()));
app.post(`${prefix}/instance:drain`, (req, res) => {
const body = req.body as Record<string, unknown> | undefined;
if (
!body ||
Object.keys(body).sort().join(',') !== 'expectedBootId,operationId' ||
typeof body['operationId'] !== 'string' ||
!UUID.test(body['operationId']) ||
body['expectedBootId'] !== this.contract.bootId
) {
res.status(400).json({ code: 'invalid_hosted_harness_drain' });
return;
}
if (this.drainOperation && this.drainOperation !== body['operationId']) {
res.status(409).json({ code: 'hosted_harness_drain_conflict' });
return;
}
this.drainOperation = body['operationId'];
this.onDrain?.();
res
.status(202)
.json({ ...this.snapshot(), operationId: this.drainOperation });
});
app.get(`${prefix}/metrics`, (_req, res) => {
const memory = process.memoryUsage();
res.json({
...this.snapshot(),
process: {
rssBytes: memory.rss,
heapUsedBytes: memory.heapUsed,
eventLoopUtilization: performance.eventLoopUtilization().utilization,
},
});
});
}
}
packages/cli/src/serve/hosted-harness-profile.ts
+4
View File
@@ -90,6 +90,10 @@ export function validateHostedHarnessProfile(
);
}
if (opts.hostedCsiSessionStoreUrl !== undefined) {
if (process.env['QWEN_HOSTED_HARNESS_INSTANCE_ID'] !== undefined)
throw new Error(
'The shared Harness pool cannot host private CSI owners.',
);
if (
!opts.managedRuntimeBrokerUrl ||
!opts.managedRuntimeBrokerToken?.trim()
packages/cli/src/serve/hosted-harness-session.test.ts
+374 -10
View File
@@ -255,6 +255,7 @@ const state = vi.hoisted(() => ({
toolResults: null as DurableToolResultResourceStore | null,
publicationRequest: vi.fn(),
storeOptions: [] as Array<Record<string, unknown>>,
ownershipReleases: [] as Array<Record<string, unknown>>,
setLifecycleAuthority: vi.fn(),
authorizeLifecycle: vi.fn(async () => undefined),
authorizeOrdinary: vi.fn(async (_kind?: 'legacy-close') => undefined),
@@ -360,15 +361,20 @@ vi.mock(
}
return published;
};
const journalStore = new LocalJsonlManagedSessionJournalStore({
runtimeBaseDir: state.root,
sessionId: options.sessionKey.sessionId,
transcriptPath: path.join(
state.root,
`${options.sessionKey.sessionId}.jsonl`,
),
});
let journal: Awaited<ReturnType<typeof journalStore.open>> | undefined;
const openJournal = journalStore.open.bind(journalStore);
journalStore.open = async (request) =>
(journal = await openJournal(request));
return {
journalStore: new LocalJsonlManagedSessionJournalStore({
runtimeBaseDir: state.root,
sessionId: options.sessionKey.sessionId,
transcriptPath: path.join(
state.root,
`${options.sessionKey.sessionId}.jsonl`,
),
}),
journalStore,
resourceStore,
toolResultResources: state.toolResults ?? resourceStore,
assertWritable: state.assertWritable,
@@ -386,6 +392,28 @@ vi.mock(
rememberAdmission: () => undefined,
},
close: async () => undefined,
stopLocal: async () => undefined,
quiesceOwnership: async () => undefined,
releaseOwnership: async (
request: Record<string, unknown> & {
commitProof: {
lastCommitSequence: number;
committedPrefixHash: string;
};
},
) => {
state.ownershipReleases.push(request);
await journal!.seal(request.commitProof);
return {
format: 'qwen-hosted-harness-release/1',
state: 'RELEASED',
...options.sessionKey,
...request,
writerGeneration: 1,
journalRevision: 1,
committedSequence: request.commitProof.lastCommitSequence,
};
},
};
},
}),
@@ -512,7 +540,7 @@ afterEach(async () => {
listeners.clear();
});
async function app(withBroker = false) {
async function app(withBroker = false, instanceId?: string) {
const result = express();
result.use(express.json());
const contract = createHostedHarnessContract(
@@ -520,6 +548,7 @@ async function app(withBroker = false) {
BOOT_ID,
['hosted-workspace-shell/1', 'hosted-workspace-mcp/1'],
instanceId,
);
installHostedHarnessContractMiddleware(result, contract);
registerHostedHarnessSessionRoutes(
@@ -1337,6 +1366,7 @@ describe('Hosted Harness no-tool session', () => {
});
state.root = await mkdtemp(path.join(tmpdir(), 'hosted-harness-test-'));
state.toolResults = null;
state.ownershipReleases.length = 0;
state.assertWritable.mockReset();
state.assertWritable.mockResolvedValue(undefined);
state.authorizeLifecycle.mockReset();
@@ -2869,6 +2899,279 @@ describe('Hosted Harness no-tool session', () => {
firedAt: Date.parse('2026-03-08T07:00:05Z'),
});
it.each([false, true])(
'retains a completed gate release failure while the pool drains (retry: %s)',
async (retry) => {
const server = await app(true, 'pod-gate');
vi.spyOn(HostedWorkspaceBroker.prototype, 'warm').mockResolvedValue();
const acquire = vi
.spyOn(HostedWorkspaceBroker.prototype, 'acquire')
.mockResolvedValue();
const prepare = vi
.spyOn(HostedWorkspaceBroker.prototype, 'prepareReplayable')
.mockResolvedValue({
executionCallId: 'gate-execution',
settled: {
executionStatus: 'success',
responseParts: [{ text: '{"kind":"quiet","durationMs":3}' }],
},
});
const release = vi
.spyOn(HostedWorkspaceBroker.prototype, 'release')
.mockRejectedValue(
new HostedWorkspaceBrokerRejection(503, 'runtime_unavailable', {
message: 'fixture release refused',
}),
);
const created = await headers(supertest(server).post('/session')).send({
sessionId: SESSION_ID,
sessionScope: 'thread',
toolProfile: 'hosted-workspace-shell/1',
managedSessionStore: { ...store(), writerToken: 'a'.repeat(32) },
harnessPlacement: {
instanceId: 'pod-gate',
bootId: BOOT_ID,
placementEpoch: 1,
},
});
expect(created.status).toBe(200);
const authorized = (request: supertest.Test) =>
headers(request).set(
'X-Qwen-Client-Id',
created.body.clientId as string,
);
const operations = (body: Record<string, unknown>) =>
authorized(
supertest(server).post(
`/session/${SESSION_ID}/automations/operations`,
),
).send({ operationId: randomUUID(), ...body });
const snapshot = () =>
headers(supertest(server).get('/internal/hosted-harness/v1/instance'));
try {
expect(
(
await operations({
kind: 'define_schedule',
scheduleId: AUTOMATION_ID,
definition: { ...automationDefinition, gate: 'printf gate' },
})
).status,
).toBe(202);
const before = await snapshot();
const fired = await operations(fireBody());
expect(fired.status).toBe(503);
expect(fired.body.code).toBe('automation_operation_failed');
expect(release).toHaveBeenCalledTimes(1);
expect(prepare).toHaveBeenCalledTimes(1);
expect(acquire).not.toHaveBeenCalled();
expect(state.model).not.toHaveBeenCalled();
if (retry) {
release.mockResolvedValue();
const retried = await operations(fireBody());
expect(retried.status).toBe(202);
expect(retried.body.gate.outcome).toBe('quiet');
}
await vi.waitFor(
async () => {
const ended = await snapshot();
expect(ended.body.counters.inFlightMutations).toBe(0);
expect(ended.body.counters.openingSessions).toBe(0);
},
{ timeout: SETTLE_TIMEOUT_MS },
);
const drained = await headers(
supertest(server).post('/internal/hosted-harness/v1/instance:drain'),
).send({ operationId: randomUUID(), expectedBootId: BOOT_ID });
expect(drained.status).toBe(202);
await vi.waitFor(
async () => {
const after = await snapshot();
const observation = {
before: before.body,
fire: { status: fired.status, body: fired.body },
drain: { status: drained.status, body: drained.body },
after: after.body,
ownershipReleases: state.ownershipReleases,
};
expect(observation).toMatchObject({
after: {
state: retry ? 'DRAINED' : 'QUIESCING',
counters: {
inFlightMutations: 0,
attachedSessions: retry ? 0 : 1,
heldResources: retry ? 0 : 1,
},
},
ownershipReleases: retry ? [{ placementEpoch: 1 }] : [],
});
},
{ timeout: SETTLE_TIMEOUT_MS },
);
} finally {
release.mockResolvedValue();
await authorized(supertest(server).delete(`/session/${SESSION_ID}`));
}
},
);
it('keeps interrupted initialization unresolved after quiesce instead of claiming DRAINED', async () => {
const server = await app(false, 'pod-a');
let entered!: () => void;
let unblock!: () => void;
const started = new Promise<void>((resolve) => {
entered = resolve;
});
const gate = new Promise<void>((resolve) => {
unblock = resolve;
});
const original = LocalManagedSessionResourceStore.prototype.read;
let held = false;
const read = vi
.spyOn(LocalManagedSessionResourceStore.prototype, 'read')
.mockImplementation(async function (
this: LocalManagedSessionResourceStore,
ref,
) {
if (ref.kind === 'managed-definition' && !held) {
held = true;
entered();
await gate;
}
return original.call(this, ref);
});
const opening = headers(supertest(server).post('/session'))
.send({
sessionId: SESSION_ID,
sessionScope: 'thread',
managedSessionStore: { ...store(), writerToken: 'a'.repeat(32) },
harnessPlacement: {
instanceId: 'pod-a',
bootId: BOOT_ID,
placementEpoch: 1,
},
})
.then((response) => response);
try {
await started;
const draining = await headers(
supertest(server).post('/internal/hosted-harness/v1/instance:drain'),
).send({ operationId: randomUUID(), expectedBootId: BOOT_ID });
expect(draining.body.state).toBe('QUIESCING');
expect(draining.body.counters.openingSessions).toBeGreaterThan(0);
unblock();
expect((await opening).body.code).toBe('hosted_harness_quiescing');
const unresolved = await headers(
supertest(server).get('/internal/hosted-harness/v1/instance'),
);
expect(unresolved.body.state).toBe('QUIESCING');
expect(unresolved.body.counters.openingSessions).toBe(1);
expect(state.ownershipReleases).toHaveLength(0);
expect(state.model).not.toHaveBeenCalled();
} finally {
unblock();
await opening;
read.mockRestore();
}
});
it('quiesces after the running Turn and releases without starting or cancelling queued wake input', async () => {
const server = await app(false, 'pod-a');
const created = await headers(supertest(server).post('/session')).send({
sessionId: SESSION_ID,
sessionScope: 'thread',
managedSessionStore: { ...store(), writerToken: 'a'.repeat(32) },
harnessPlacement: {
instanceId: 'pod-a',
bootId: BOOT_ID,
placementEpoch: 1,
},
});
expect(created.status).toBe(200);
const authorized = (request: supertest.Test) =>
headers(request).set('X-Qwen-Client-Id', created.body.clientId as string);
const operations = (body: Record<string, unknown>) =>
authorized(
supertest(server).post(`/session/${SESSION_ID}/automations/operations`),
).send({ operationId: randomUUID(), ...body });
expect(
(
await operations({
kind: 'define_schedule',
scheduleId: AUTOMATION_ID,
definition: automationDefinition,
})
).status,
).toBe(202);
let finish!: () => void;
state.model.mockImplementationOnce(async () => {
await new Promise<void>((resolve) => {
finish = resolve;
});
return { text: 'finished original Turn', model: 'test-model' };
});
const prompt = [{ type: 'text', text: 'foreground before drain' }];
const body = {
promptId: PROMPT_ID,
prompt,
payloadDigest: `sha256:${createHash('sha256').update(JSON.stringify(prompt)).digest('hex')}`,
};
const submit = (payload: Record<string, unknown>) =>
authorized(supertest(server).post(`/session/${SESSION_ID}/prompt`)).send(
payload,
);
expect((await submit(body)).status).toBe(202);
await vi.waitFor(() => expect(finish).toBeTypeOf('function'));
const fired = await operations(fireBody());
expect(fired.status).toBe(202);
try {
const drained = await headers(
supertest(server).post('/internal/hosted-harness/v1/instance:drain'),
).send({ operationId: randomUUID(), expectedBootId: BOOT_ID });
expect(drained.body.state).toBe('QUIESCING');
expect((await submit(body)).status).toBe(202);
expect(
(await submit({ ...body, promptId: randomUUID() })).body.code,
).toBe('hosted_harness_quiescing');
expect(state.ownershipReleases).toHaveLength(0);
finish();
await vi.waitFor(
async () => {
const instance = await headers(
supertest(server).get('/internal/hosted-harness/v1/instance'),
);
expect(instance.body.state).toBe('DRAINED');
},
{ timeout: SETTLE_TIMEOUT_MS },
);
expect(state.model).toHaveBeenCalledTimes(1);
expect(state.ownershipReleases).toHaveLength(1);
const journal = await LocalJsonlManagedSessionJournalStore.read(
path.join(state.root, `${SESSION_ID}.jsonl`),
{ tenantId: 'tenant', workspaceId: 'workspace', sessionId: SESSION_ID },
);
expect(
journal.events.filter(
(event) =>
event.kind === 'input.accepted' &&
event.payload['turnId'] === fired.body.inputId,
),
).toHaveLength(1);
expect(
journal.events.filter(
(event) =>
event.kind === 'turn.settled' &&
event.payload['turnId'] === fired.body.inputId,
),
).toHaveLength(0);
expect(
journal.events.filter((event) => event.kind === 'lifecycle.changed'),
).toHaveLength(0);
} finally {
finish?.();
}
});
it('defines, fires and settles an automation run through the operations route', async () => {
const server = await app();
const created = await headers(supertest(server).post('/session')).send({
@@ -25922,12 +26225,14 @@ describe('Hosted Harness Runtime turn takeover', () => {
});
});
async function replacementApp() {
async function replacementApp(instanceId?: string) {
const result = express();
result.use(express.json());
const contract = createHostedHarnessContract(
`sha256:${'a'.repeat(64)}`,
BOOT_ID_2,
undefined,
instanceId,
);
installHostedHarnessContractMiddleware(result, contract);
registerHostedHarnessSessionRoutes(result, contract, state.root, {
@@ -26114,6 +26419,65 @@ describe('Hosted Harness Runtime turn takeover', () => {
return { server, loaded };
}
it.each([false, true])(
'keeps a quiescing recovery load in the pool-wait lane (resident: %s)',
async (resident) => {
await parkToolTurn();
const server = await replacementApp('recovery-pod');
const descriptor = {
managedSessionStore: {
...storeFor(BOOT_ID_2),
writerToken: 'a'.repeat(32),
},
harnessPlacement: {
instanceId: 'recovery-pod',
bootId: BOOT_ID_2,
placementEpoch: 1,
},
toolProfile: FILE_PROFILE,
};
vi.spyOn(HostedWorkspaceBroker.prototype, 'status').mockResolvedValue({
state: 'prepared',
});
if (resident) {
const attached = await replacementHeaders(
supertest(server).post(`/session/${SESSION_ID}/load`),
).send({ ...descriptor, passiveManagedRuntimeRecovery: true });
expect(attached.status).toBe(200);
}
try {
acquireSpy.mockClear();
const execute = vi.mocked(HostedWorkspaceBroker.prototype.execute);
execute.mockClear();
const release = vi.mocked(HostedWorkspaceBroker.prototype.release);
release.mockClear();
acquireSpy.mockImplementationOnce(async () => {
const draining = await replacementHeaders(
supertest(server).post(
'/internal/hosted-harness/v1/instance:drain',
),
).send({ operationId: randomUUID(), expectedBootId: BOOT_ID_2 });
expect(draining.status).toBe(202);
expect(draining.body.state).toBe('QUIESCING');
expect(draining.body.counters.inFlightMutations).toBeGreaterThan(0);
});
const loaded = await replacementHeaders(
supertest(server).post(`/session/${SESSION_ID}/load`),
).send({ ...descriptor, driveRuntimeRecovery: true });
expect(loaded.status).toBe(409);
expect(loaded.body.code).toBe('hosted_harness_quiescing');
expect(acquireSpy).toHaveBeenCalledOnce();
expect(execute).not.toHaveBeenCalled();
expect(release).toHaveBeenCalledTimes(resident ? 0 : 1);
} finally {
if (resident)
await replacementHeaders(
supertest(server).delete(`/session/${SESSION_ID}`),
).expect(204);
}
},
);
it.each(['cancel', 'close', 'continue-refusal'])(
'preserves the actual passive MCP owner through %s',
async (path) => {
packages/cli/src/serve/hosted-harness-session.ts
+382 -18
View File
@@ -52,6 +52,14 @@ import {
hasHostedCleanupDebt,
} from './hosted-turn-cleanup.js';
import { inspectHostedRecovery } from './hosted-recovery-status.js';
import {
HostedHarnessInstance,
parseHostedHarnessPlacement,
} from './hosted-harness-instance.js';
import type {
HostedHarnessPlacement,
HostedHarnessReleaseReceipt,
} from '@qwen-code/qwen-code-core/managed-runtime/hosted-harness-release.js';
import {
readHostedActionOptions,
hostedActionAllowed,
@@ -297,6 +305,7 @@ import {
answerResumedTurnCalls,
fillParkedRoundAgentGaps,
fillParkedRoundCommandGaps,
HostedRecoveryQuiescingError,
isDurableBlockedVerdict,
originalRuntimeBroker,
RecoveryDeclined,
@@ -508,6 +517,13 @@ function blockRecovery(
}
interface HostedSession {
placement?: HostedHarnessPlacement;
poolAccepting?: () => boolean;
ownershipRelease?: {
operationId: string;
proof?: HostedHarnessReleaseReceipt['commitProof'];
result?: Promise<HostedHarnessReleaseReceipt>;
};
compression?: HostedContextCompression;
streamOwner: { bootId: string; eventEpoch: string };
managed: ManagedSession;
@@ -3180,6 +3196,8 @@ async function startHostedRecoveredTurn(
resumeModel?: HostedModelRequest,
cancellationTakeover = false,
): Promise<void> {
if (session.poolAccepting?.() === false)
throw new HostedRecoveryQuiescingError('The Harness is quiescing.');
const abort = new AbortController();
const accepted = session.managed.authority
.eventsInSequenceRange(1, session.managed.authority.committedSequence)
@@ -3886,16 +3904,240 @@ export function registerHostedHarnessSessionRoutes(
// The owed mount handbacks of recovery-blocked turns (#13800) outlive
// their Session, whose close never frees a parked turn's mount; the
// daemon owns them and stops them at shutdown.
let ownershipDrainTimer: ReturnType<typeof setInterval> | undefined;
let ownershipDraining = false;
let ownershipDrainStopped = false;
const mountHandbacks = new Set<() => void>();
(
app.locals as { stopHostedMountHandbacks?: () => void }
).stopHostedMountHandbacks = () => {
ownershipDrainStopped = true;
if (ownershipDrainTimer) clearInterval(ownershipDrainTimer);
for (const stop of [...mountHandbacks]) stop();
};
if (brokerOptions) brokerOptions = { ...brokerOptions, mountHandbacks };
const sessions = new Map<string, HostedSession>();
const opening = new Set<string>();
const unresolvedOpenings = new Set<string>();
let initialReconciliations = 0;
const epoch = contract.bootId.replaceAll('-', '_');
const released = new Map<
string,
{ clientId: string; receipt: HostedHarnessReleaseReceipt }
>();
const heldResources = (session: HostedSession): number =>
Number(Boolean(session.hooks)) +
Number(Boolean(session.mcp)) +
Number(Boolean(session.automations?.hasPendingGateReleases)) +
Number(Boolean(session.shell?.publisher)) +
Number(Boolean(session.backgroundLane?.publisher)) +
session.heldRuntimeTurns.size +
Number(Boolean(session.runtimeLeaseHeld)) +
[...(session.shell?.monitorLoops?.values() ?? [])].length +
[...(session.backgroundLane?.monitorLoops?.values() ?? [])].length +
['child_run', 'monitor_run'].reduce(
(sum, domain) =>
sum +
session.managed.authority
.extensionRecordsInDomain(domain as 'child_run' | 'monitor_run')
.filter(
(entry) =>
!['settled', 'failed', 'cancelled'].includes(entry.run.state),
).length,
0,
);
const instance =
contract.instanceId === undefined
? undefined
: new HostedHarnessInstance(
{ ...contract, instanceId: contract.instanceId },
() => ({
attachedSessions: sessions.size,
activeTurns: [...sessions.values()].filter(
(session) => session.active,
).length,
openingSessions:
opening.size + unresolvedOpenings.size + initialReconciliations,
pendingInputs: [...sessions.values()].reduce(
(sum, session) =>
sum +
pendingSessionInputs(
session.managed.authority.eventsInSequenceRange(
1,
session.managed.authority.committedSequence,
),
).length,
0,
),
heldResources: [...sessions.values()].reduce(
(sum, session) => sum + heldResources(session),
mountHandbacks.size,
),
pendingPublications: [...sessions.values()].filter(
(session) =>
session.shell?.publisher || session.backgroundLane?.publisher,
).length,
}),
() => startOwnershipDrain(),
);
const acceptingWork = () => instance?.accepting !== false;
if (instance) {
instance.register(app);
app.use('/session', instance.track);
app.post('/session/:id/ownership:release', async (req, res) => {
res.setHeader('Cache-Control', 'no-store');
const body = object(req.body);
const operationId = body?.['operationId'];
const placementEpoch = body?.['placementEpoch'];
if (
!body ||
Object.keys(body).sort().join(',') !== 'operationId,placementEpoch' ||
typeof operationId !== 'string' ||
!HOSTED_UUID.test(operationId) ||
!Number.isSafeInteger(placementEpoch) ||
(placementEpoch as number) < 1
)
return error(res, 400, 'invalid_hosted_ownership_release');
const previous = released.get(req.params['id']);
if (
previous &&
previous.clientId === req.get('X-Qwen-Client-Id') &&
previous.receipt.operationId === operationId &&
previous.receipt.placementEpoch === placementEpoch
)
return res.json(previous.receipt);
const session = identity(req, sessions);
if (!session) return error(res, 404, 'hosted_session_not_found');
if (
instance.accepting ||
session.placement?.placementEpoch !== placementEpoch
)
return error(res, 409, 'hosted_ownership_release_conflict');
if (
session.ownershipRelease &&
session.ownershipRelease.operationId !== operationId
)
return error(res, 409, 'hosted_ownership_release_conflict');
try {
const receipt = await releaseIdleOwner(
req.params['id'],
session,
operationId,
);
if (!receipt) return error(res, 409, 'hosted_ownership_release_busy');
res.json(receipt);
} catch {
error(res, 503, 'hosted_ownership_release_unresolved');
}
});
}
async function releaseIdleOwner(
sessionId: string,
session: HostedSession,
operationId: string,
): Promise<HostedHarnessReleaseReceipt | undefined> {
if (!session.placement || !instance || instance.accepting) return undefined;
if (!session.ownershipRelease) {
if (
instance.inFlightMutations ||
opening.size ||
initialReconciliations ||
session.active ||
session.recoveredTurn ||
session.blocked ||
session.lifecycle ||
session.mcpBusy ||
session.hooksBusy ||
session.mcpRecovering ||
compressionBusy(session) ||
heldResources(session) ||
session.fileHistoryOwed ||
hasHostedCleanupDebt(session.managed) ||
session.childConsumption.size ||
session.messageOperations?.size ||
unsettledInputs(session).size
)
return undefined;
session.ownershipRelease = { operationId };
}
const release = session.ownershipRelease;
try {
release.result ??= (async () => {
if (!release.proof) {
const projected = await session.managed.sink.project();
const pending = pendingSessionInputs(
session.managed.authority.eventsInSequenceRange(
1,
session.managed.authority.committedSequence,
),
);
if (
pending.some(
(input) =>
!isWakeInputSource(input.source) ||
wakeHasPriorAttempt(projected, input.turnId),
)
)
throw new Error('An attempted input still requires recovery.');
await session.stores!.quiesceOwnership(session.placement!);
release.proof = await session.managed.prepareOwnershipRelease();
}
return session.stores!.releaseOwnership({
...session.placement!,
operationId: release.operationId,
commitProof: release.proof,
});
})();
const receipt = await release.result;
session.monitorWake?.close();
for (const stop of session.streams) stop();
await session.stores!.stopLocal();
released.set(sessionId, { clientId: session.clientId, receipt });
if (released.size > 256) released.delete(released.keys().next().value!);
sessions.delete(sessionId);
unresolvedOpenings.delete(sessionId);
return receipt;
} catch (cause) {
release.result = undefined;
writeStderrLineSafe(
`qwen serve: Hosted ownership release requires reconciliation: ${String(cause)}`,
);
throw cause;
}
}
function startOwnershipDrain(): void {
if (ownershipDrainTimer || ownershipDrainStopped) return;
const pass = async () => {
if (ownershipDraining || ownershipDrainStopped) return;
ownershipDraining = true;
try {
for (const [sessionId, session] of sessions) {
try {
await releaseIdleOwner(
sessionId,
session,
session.ownershipRelease?.operationId ?? randomUUID(),
);
} catch {
// Unknown work or a missing Store keeps this owner attached.
}
}
if (instance?.snapshot().state === 'DRAINED' && ownershipDrainTimer) {
clearInterval(ownershipDrainTimer);
ownershipDrainTimer = undefined;
}
} finally {
ownershipDraining = false;
}
};
ownershipDrainTimer = setInterval(() => {
void pass();
}, 500);
ownershipDrainTimer.unref();
void pass();
}
const sendAttachment = (
res: Response,
@@ -4253,6 +4495,25 @@ export function registerHostedHarnessSessionRoutes(
error(res, 409, 'hosted_harness_generation_mismatch');
return;
}
let placement: HostedHarnessPlacement | undefined;
try {
if (instance) {
placement = parseHostedHarnessPlacement(body?.['harnessPlacement']);
if (
placement.instanceId !== contract.instanceId ||
placement.bootId !== contract.bootId ||
store.allowInsecureHttp === true ||
!store.writerToken
)
throw new Error('Harness placement does not name this owner.');
} else if (body?.['harnessPlacement'] !== undefined) {
throw new Error('Harness pool mode is not enabled.');
}
} catch {
return error(res, 409, 'hosted_harness_placement_mismatch');
}
if (instance && !instance.accepting)
return error(res, 409, 'hosted_harness_quiescing');
let lifecycle: ManagedSessionLifecycleAuthority | undefined;
try {
lifecycle = lifecycleAuthority(body?.['lifecycleAuthority']);
@@ -4280,6 +4541,8 @@ export function registerHostedHarnessSessionRoutes(
return error(res, 409, 'hosted_lifecycle_profile_unavailable');
}
let resident = sessions.get(sessionId);
if (resident && !isDeepStrictEqual(resident.placement, placement))
return error(res, 409, 'hosted_harness_placement_mismatch');
// H4f × H4d-b: the load of a stopped run's child carries the stop, so
// its wake pump starts none of its message inputs, not even the first
// pass this load kicks before the stop itself arrives.
@@ -4494,6 +4757,7 @@ export function registerHostedHarnessSessionRoutes(
return;
}
opening.add(sessionId);
if (instance) unresolvedOpenings.add(sessionId);
resident.mcpClosing = true;
try {
await resident.managed.close();
@@ -4879,6 +5143,13 @@ export function registerHostedHarnessSessionRoutes(
return;
}
} catch (cause) {
if (
cause instanceof HostedRecoveryQuiescingError &&
!resident.managed.authority.writesStopped
) {
error(res, 409, 'hosted_harness_quiescing');
return;
}
blockRecovery(resident, namedStoreFailure(cause));
writeStderrLineSafe(
'qwen serve: Hosted recovery redrive failed: ' + String(cause),
@@ -4897,6 +5168,7 @@ export function registerHostedHarnessSessionRoutes(
try {
const outcome = await recoverHostedRuntimeTurn({
session: resident.managed,
canStart: resident.poolAccepting,
sessionId,
cwd,
promptId: parked,
@@ -4938,6 +5210,13 @@ export function registerHostedHarnessSessionRoutes(
writeStderrLineSafe(
`qwen serve: Hosted Harness recovery re-answer of session ${sessionId} failed: ${sanitizeDaemonLogLine(String(cause))}`,
);
if (
cause instanceof HostedRecoveryQuiescingError &&
!resident.managed.authority.writesStopped
) {
error(res, 409, 'hosted_harness_quiescing');
return;
}
// Retry-inviting, like the first load's recovery failure; the
// attached Session keeps its owed lease for the next redrive. A
// stopped writer outranks the takeover label, exactly as the
@@ -5031,6 +5310,7 @@ export function registerHostedHarnessSessionRoutes(
try {
const outcome = await recoverHostedRuntimeTurn({
session: resident.managed,
canStart: resident.poolAccepting,
sessionId,
cwd: resident.cwd,
promptId: parked,
@@ -5219,6 +5499,11 @@ export function registerHostedHarnessSessionRoutes(
? { create: refs, requireNew: true }
: { retainVerifiedResources: true }),
});
if (!acceptingWork()) {
await managed.close();
error(res, 409, 'hosted_harness_quiescing');
return;
}
if (typeof cancellationPromptId === 'string') {
const target = cancellationTargetState(managed, cancellationPromptId);
if (target === undefined) {
@@ -5364,7 +5649,13 @@ export function registerHostedHarnessSessionRoutes(
error(res, 409, 'hosted_tool_profile_conflict');
return;
}
if (!acceptingWork()) {
await managed.close();
error(res, 409, 'hosted_harness_quiescing');
return;
}
const session: HostedSession = {
...(placement ? { placement, poolAccepting: acceptingWork } : {}),
streamOwner: { bootId: contract.bootId, eventEpoch: epoch },
managed,
storeBaseUrl: store.baseUrl,
@@ -5685,6 +5976,8 @@ export function registerHostedHarnessSessionRoutes(
session.channels
) {
const wakeBusy = () =>
instance?.accepting === false ||
session.ownershipRelease !== undefined ||
compressionBusy(session) ||
session.active !== undefined ||
// A recovered-but-undriven Turn holds no live admission, yet its
@@ -6555,6 +6848,7 @@ export function registerHostedHarnessSessionRoutes(
try {
const outcome = await recoverHostedRuntimeTurn({
session: managed,
canStart: session.poolAccepting,
sessionId,
cwd,
promptId: unsettled,
@@ -6644,6 +6938,10 @@ export function registerHostedHarnessSessionRoutes(
writeStderrLineSafe(
`qwen serve: Hosted Harness recovery of session ${sessionId} failed: ${sanitizeDaemonLogLine(String(cause))}`,
);
if (cause instanceof HostedRecoveryQuiescingError) {
error(res, 409, 'hosted_harness_quiescing');
return;
}
// A failed takeover keeps the turn parked for the next attempt:
// refuse exactly like a plain recovery refusal so the coordinator
// retries instead of failing the Turn.
@@ -6884,26 +7182,33 @@ export function registerHostedHarnessSessionRoutes(
}
}
sessions.set(sessionId, session);
compressionFor(session).resume();
if (acceptingWork()) compressionFor(session).resume();
// H6c: a settle → settle-revision crash window closes here. It runs
// beside the pump's first turn: the funnel serializes its commits and
// a settle is idempotent, so the order between the two is immaterial.
void session.automations
?.reconcileRuns()
.catch((cause: unknown) =>
writeStderrLineSafe(
`qwen serve: Hosted automation runs of session ${sessionId} could not be reconciled: ${String(cause)}`,
),
);
// H5c: a settle → plan crash window closes here, before the pump
// can start another channel turn.
void session.channels
?.reconcileReplies()
.catch((cause: unknown) =>
writeStderrLineSafe(
`qwen serve: Hosted channel replies of session ${sessionId} could not be reconciled: ${String(cause)}`,
),
);
if (acceptingWork()) {
if (instance) initialReconciliations += 1;
void Promise.allSettled([
session.automations
?.reconcileRuns()
.catch((cause: unknown) =>
writeStderrLineSafe(
`qwen serve: Hosted automation runs of session ${sessionId} could not be reconciled: ${String(cause)}`,
),
),
// H5c: a settle → plan crash window closes here, before the pump
// can start another channel turn.
session.channels
?.reconcileReplies()
.catch((cause: unknown) =>
writeStderrLineSafe(
`qwen serve: Hosted channel replies of session ${sessionId} could not be reconciled: ${String(cause)}`,
),
),
]).finally(() => {
if (instance) initialReconciliations -= 1;
});
}
session.monitorWake?.kick();
// The registered Session now carries the owed lease itself; the
// refusal-time record is discharged.
@@ -6931,7 +7236,9 @@ export function registerHostedHarnessSessionRoutes(
}
const cause =
thrown instanceof ManagedSessionCloseError ? thrown.sealError : thrown;
if (isRetryableWorkspaceAcquisition(cause)) {
if (cause instanceof HostedRecoveryQuiescingError) {
error(res, 409, 'hosted_harness_quiescing');
} else if (isRetryableWorkspaceAcquisition(cause)) {
error(res, 409, cause.code);
} else if (
cause instanceof ManagedSessionStoreHttpError &&
@@ -6961,6 +7268,12 @@ export function registerHostedHarnessSessionRoutes(
}
} finally {
opening.delete(sessionId);
if (instance) {
// A failed open may have acquired a writer without a release receipt.
// Keep it unresolved until this Session attaches or the boot expires.
if (sessions.has(sessionId)) unresolvedOpenings.delete(sessionId);
else unresolvedOpenings.add(sessionId);
}
}
};
@@ -6978,6 +7291,50 @@ export function registerHostedHarnessSessionRoutes(
app.use('/session/:id', async (req, res, next) => {
const session = sessions.get(req.params['id']);
if (instance && session && req.method !== 'GET') {
if (!identity(req, sessions, req.method === 'DELETE'))
return error(res, 404, 'hosted_session_not_found');
if (session.ownershipRelease)
return error(res, 409, 'hosted_harness_quiescing');
if (!instance.accepting) {
const kind = object(req.body)?.['kind'];
const observed =
[
'/prompt',
'/heartbeat',
'/cancel',
'/managed-runtime/cancel',
'/lifecycle',
'/detach',
].includes(req.path) ||
req.method === 'DELETE' ||
/^\/actions\/[^/]+\/resolve$/u.test(req.path) ||
/\/(hooks|mcp)\/operations\/[^/]+\/cancel$/u.test(req.path) ||
(req.path === '/children/operations' &&
!['accept', 'mark_accepted', 'dispatch_started'].includes(
String(kind),
)) ||
(req.path === '/messages/operations' &&
[
'accepted',
'answered',
'consumed',
'cancelled',
'rejected',
'unknown',
'consume',
].includes(String(kind))) ||
(req.path === '/channels/operations' &&
['segment_receipt', 'settle_delivery', 'cancel_delivery'].includes(
String(kind),
)) ||
(req.path === '/automations/operations' &&
['cancel_run', 'reconcile_run', 'retire_schedule'].includes(
String(kind),
));
if (!observed) return error(res, 409, 'hosted_harness_quiescing');
}
}
if (
session?.lifecycle &&
req.method !== 'GET' &&
@@ -7475,6 +7832,11 @@ export function registerHostedHarnessSessionRoutes(
});
return;
}
if (
instance?.accepting === false &&
acceptedInputSequence(session, promptId) === undefined
)
return error(res, 409, 'hosted_harness_quiescing');
if (session.active) return error(res, 409, 'hosted_turn_active');
// A Hook recovery fence is not terminal: reconcile the pending records
// once — a non-cancelling poll, so no fenced occurrence re-dispatches —
@@ -7705,6 +8067,8 @@ export function registerHostedHarnessSessionRoutes(
// is what makes the check-then-claim above atomic again.
const relatched = turnSlotLatch();
if (relatched !== undefined) return refuseTurnSlot(relatched);
if (instance?.accepting === false)
return error(res, 409, 'hosted_harness_quiescing');
if (unsettledInputs(session).size !== 0)
return error(
res,
packages/cli/src/serve/hosted-runtime-recovery.test.ts
+43
View File
@@ -22,6 +22,7 @@ import {
settleParkedTurnCancelled,
answerCommittedTurnCalls,
recoverHostedRuntimeTurn,
HostedRecoveryQuiescingError,
RecoveryDeclined,
settleInterruptedTurnRuntime,
type HostedRecoveryTurn,
@@ -1636,6 +1637,48 @@ describe('recoverHostedRuntimeTurn', () => {
}
});
it.each([false, true])(
'keeps quiescing recovery retryable without executing tools (lease already held: %s)',
async (leaseAlreadyHeld) => {
await parkAtAwaitRuntime();
vi.spyOn(HostedWorkspaceBroker.prototype, 'acquire').mockResolvedValue();
const release = vi
.spyOn(HostedWorkspaceBroker.prototype, 'release')
.mockResolvedValue();
const execute = vi.spyOn(HostedWorkspaceBroker.prototype, 'execute');
const replacement = await open('boot-2', false);
try {
const checkpoint =
await replacement.authority.harnessRunAuthorization();
const sequence = replacement.authority.committedSequence;
const cause = await recoverHostedRuntimeTurn({
session: replacement,
sessionId: SESSION_ID,
cwd: root,
promptId: PROMPT_ID,
brokerOptions,
passive: false,
canStart: () => false,
leaseAlreadyHeld,
}).then(
() => undefined,
(error: unknown) => error,
);
expect(cause).toBeInstanceOf(HostedRecoveryQuiescingError);
expect(cause).not.toBeInstanceOf(RecoveryDeclined);
expect(cause).toHaveProperty('message', 'The Harness is quiescing.');
expect(execute).not.toHaveBeenCalled();
expect(release).toHaveBeenCalledTimes(leaseAlreadyHeld ? 0 : 1);
expect(replacement.authority.committedSequence).toBe(sequence);
expect(await replacement.authority.harnessRunAuthorization()).toEqual(
checkpoint,
);
} finally {
await replacement.close();
}
},
);
it('declines to re-dispatch a parked execution whose intent is missing', async () => {
// No tool.intent: the recovery cannot rebuild the original arguments, so
// it must decline rather than dispatch a tool with nothing behind it.
packages/cli/src/serve/hosted-runtime-recovery.ts
+6
View File
@@ -118,6 +118,9 @@ import {
/** The parked turn is not one this recovery can drive; the caller 409s. */
export class RecoveryDeclined extends Error {}
/** A temporary pool wait, distinct from a deterministic takeover decline. */
export class HostedRecoveryQuiescingError extends Error {}
/** The states a recovery report may advertise: exactly the vocabulary the
* hosted client's validator accepts (HostedHarnessClient's
* RUNTIME_EXECUTION_STATES). The broker's wider union folds into it at the
@@ -2254,6 +2257,7 @@ export async function recoverHostedRuntimeTurn(input: {
promptId: string;
brokerOptions: HostedWorkspaceBrokerOptions;
passive: boolean;
canStart?: () => boolean;
/** The evidence funnels of the interrupted-Turn settlement (#13847): a
* model-start park whose open round holds only journal-only calls
* answers them by what committed through these — the same shapes the
@@ -2587,6 +2591,8 @@ export async function recoverHostedRuntimeTurn(input: {
) as { payloadJson?: unknown };
if (typeof stored.payloadJson !== 'string')
throw new RecoveryDeclined();
if (input.canStart?.() === false)
throw new HostedRecoveryQuiescingError('The Harness is quiescing.');
const result = await broker
.execute(
item.executionCallId,
packages/cli/src/serve/hosted-workspace-tool-turn.test.ts
+1 -1
View File
@@ -5157,7 +5157,7 @@ it.each(
),
),
)(
'durably refuses malformed Shell Unicode before Hooks and acquisition ($publication/$version/$field/$value/last:$invalidLast)',
'durably refuses malformed Shell Unicode before Hooks and acquisition: %j',
async ({ publication, version, field, value, invalidLast }) => {
const owner = {
owner: vi.fn(),
packages/cli/src/serve/process-env-guard.test.ts
+23 -5
View File
@@ -41,8 +41,19 @@ const allowedProcessEnvAccesses = normalizeAllowances([
'packages/cli/src/serve/hosted-harness-contract.ts',
{
reason:
'The Hosted lifecycle v2 profile advertisement is a process-scoped deployment gate fixed when the daemon contract is created.',
accesses: { 'computed:HOSTED_LIFECYCLE_V2_PROFILES_ENV': 1 },
'The Hosted lifecycle v2 profile advertisement and pool instance identity are process-scoped deployment contracts fixed when the daemon contract is created.',
accesses: {
'computed:HOSTED_LIFECYCLE_V2_PROFILES_ENV': 1,
'key:QWEN_HOSTED_HARNESS_INSTANCE_ID': 1,
},
},
],
[
'packages/cli/src/serve/hosted-harness-profile.ts',
{
reason:
'Hosted profile validation refuses private CSI roots when the process-scoped pool instance identity is configured.',
accesses: { 'key:QWEN_HOSTED_HARNESS_INSTANCE_ID': 1 },
},
],
[
@@ -378,7 +389,7 @@ const allowedProcessEnvAccesses = normalizeAllowances([
'daemon environment into the TLS trust probe child. NODE_TLS_REJECT_UNAUTHORIZED is read to skip the ' +
'worker TLS trust check when it disables verification: workers inherit the variable unscrubbed and dial ' +
'via fetch, which honors it, so the strict probe would flag an outage that never happens. ' +
'The Hosted Harness capability digest and private CSI Session Store URL are process-scoped contracts fixed at daemon bootstrap.',
'The Hosted Harness capability digest, Broker URL/token and private CSI Session Store URL are process-scoped deployment contracts fixed at daemon bootstrap.',
accesses: {
'computed:EXTERNAL_TOOL_GUARD_TOKEN_ENV': 1,
'computed:HOSTED_HARNESS_CAPABILITY_DIGEST_ENV': 1,
@@ -392,6 +403,8 @@ const allowedProcessEnvAccesses = normalizeAllowances([
'key:NODE_EXTRA_CA_CERTS': 1,
'key:NODE_TLS_REJECT_UNAUTHORIZED': 1,
'key:QWEN_CODE_IDE_WORKSPACE_PATH': 1,
'key:QWEN_HOSTED_HARNESS_BROKER_URL': 1,
'key:QWEN_HOSTED_HARNESS_BROKER_TOKEN': 1,
'key:QWEN_SERVE_NO_MCP_POOL': 1,
'key:QWEN_SERVE_NO_PERSISTENT_REGISTRATION': 1,
'key:VITEST_WORKER_ID': 1,
@@ -518,8 +531,13 @@ const allowedProcessEnvAccesses = normalizeAllowances([
'Embedded server construction keeps a process-environment compatibility fallback. ' +
'The collaboration opt-in is read once at daemon startup and is process-scoped ' +
'by design: it governs work no session owns (the dispatch timer and Host ' +
'transport routes), so it cannot be a per-session setting.',
accesses: { whole: 1, 'key:QWEN_CODE_ENABLE_AGENT_COLLABORATION': 1 },
'transport routes), so it cannot be a per-session setting. ' +
'The Hosted pool instance setting is captured once when the server installs its process-scoped private routes.',
accesses: {
whole: 1,
'key:QWEN_CODE_ENABLE_AGENT_COLLABORATION': 1,
'key:QWEN_HOSTED_HARNESS_INSTANCE_ID': 1,
},
},
],
[
packages/cli/src/serve/run-qwen-serve.test.ts
+9
View File
@@ -6536,6 +6536,8 @@ describe('runQwenServe deployment profiles', () => {
});
it('does not restore channels or scheduled sessions for Hosted Harness', async () => {
vi.stubEnv('QWEN_HOSTED_HARNESS_BROKER_URL', 'https://broker.internal');
vi.stubEnv('QWEN_HOSTED_HARNESS_BROKER_TOKEN', 'file-fed-broker-secret');
const workspace = fs.realpathSync(
fs.mkdtempSync(path.join(os.tmpdir(), 'qws-hosted-profile-')),
);
@@ -6567,12 +6569,19 @@ describe('runQwenServe deployment profiles', () => {
},
);
expect(createApp.mock.calls[0]?.[0].channelSelection).toBeUndefined();
expect(createApp.mock.calls[0]?.[0].managedRuntimeBrokerUrl).toBe(
'https://broker.internal',
);
expect(createApp.mock.calls[0]?.[0].managedRuntimeBrokerToken).toBe(
'file-fed-broker-secret',
);
expect(createApp.mock.calls[0]?.[2]?.manageScheduledTaskSessions).toBe(
false,
);
} finally {
await handle?.close();
createApp.mockRestore();
vi.unstubAllEnvs();
fs.rmSync(workspace, { recursive: true, force: true });
}
});
packages/cli/src/serve/run-qwen-serve.ts
+14
View File
@@ -3466,6 +3466,16 @@ async function runQwenServeImpl(
const hostedCsiSessionStoreUrl =
optsIn.hostedCsiSessionStoreUrl ??
process.env[HOSTED_CSI_SESSION_STORE_URL_ENV];
const managedRuntimeBrokerUrl =
optsIn.managedRuntimeBrokerUrl ??
(optsIn.profile === 'hosted-harness'
? process.env['QWEN_HOSTED_HARNESS_BROKER_URL']
: undefined);
const managedRuntimeBrokerToken =
optsIn.managedRuntimeBrokerToken ??
(optsIn.profile === 'hosted-harness'
? process.env['QWEN_HOSTED_HARNESS_BROKER_TOKEN']
: undefined);
validateHostedHarnessProfile({
...optsIn,
token,
@@ -3473,6 +3483,8 @@ async function runQwenServeImpl(
optsIn.profile === 'hosted-harness' ? true : optsIn.requireAuth,
hostedHarnessCapabilityDigest,
hostedCsiSessionStoreUrl,
managedRuntimeBrokerUrl,
managedRuntimeBrokerToken,
});
const baseEnv: NodeJS.ProcessEnv = { ...process.env };
const launchMemoryProjectScopeValue =
@@ -3610,6 +3622,8 @@ async function runQwenServeImpl(
token,
hostedHarnessCapabilityDigest,
hostedCsiSessionStoreUrl,
managedRuntimeBrokerUrl,
managedRuntimeBrokerToken,
promptDeadlineMs,
writerIdleTimeoutMs,
workspace: rawWorkspace,
packages/cli/src/serve/server.ts
+3
View File
@@ -2310,11 +2310,14 @@ export function createServeApp(
);
app.use(allowOriginCors(originAllowlist));
if (opts.profile === 'hosted-harness') {
const poolEnabled =
process.env['QWEN_HOSTED_HARNESS_INSTANCE_ID'] !== undefined;
app.use((req, res, next) => {
if (
req.path === '/health' ||
req.path === '/capabilities' ||
req.path === '/session' ||
(poolEnabled && req.path.startsWith('/internal/hosted-harness/v1/')) ||
req.path.startsWith('/session/')
)
next();
packages/cli/src/serve/types.ts
+1
View File
@@ -452,6 +452,7 @@ export interface ServeOptions {
* `v` is the wire schema version; bumped only on breaking frame changes.
*/
export interface HostedHarnessCapabilities {
readonly instanceId?: string;
readonly features?: readonly string[];
readonly lifecycleProtocolVersion?: 1;
readonly lifecycleV2Profiles?: readonly string[];
packages/core/src/managed-runtime/contracts/managed-extension-record-v1.fixtures.json
+2 -1
View File
@@ -34,7 +34,8 @@
"session_command",
"session_metadata",
"file_history",
"session_source"
"session_source",
"hosted_harness_release"
],
"limits": {
"maxIdBytes": 512,
packages/core/src/managed-runtime/contracts/managed-extension-record-v1.schema.json
+4 -2
View File
@@ -65,7 +65,8 @@
"session_command",
"session_metadata",
"file_history",
"session_source"
"session_source",
"hosted_harness_release"
]
},
"limits": {
@@ -523,7 +524,8 @@
"session_command",
"session_metadata",
"file_history",
"session_source"
"session_source",
"hosted_harness_release"
]
},
"resourceScope": {
packages/core/src/managed-runtime/hosted-harness-release.ts
+29
View File
@@ -0,0 +1,29 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/
import type { ManagedSessionCommitProof } from './managed-session-storage.js';
import type { ManagedSessionKey } from './managed-session-records.js';
export interface HostedHarnessPlacement {
readonly instanceId: string;
readonly bootId: string;
readonly placementEpoch: number;
}
export interface HostedHarnessReleaseRequest extends HostedHarnessPlacement {
readonly operationId: string;
readonly commitProof: ManagedSessionCommitProof;
}
export interface HostedHarnessReleaseReceipt
extends HostedHarnessReleaseRequest,
ManagedSessionKey {
readonly format: 'qwen-hosted-harness-release/1';
readonly writerGeneration: number;
readonly journalRevision: number;
readonly committedSequence: number;
readonly state: 'RELEASED';
}
packages/core/src/managed-runtime/http-managed-session-store.test.ts
+88
View File
@@ -429,6 +429,94 @@ describe('HTTP Managed Session store', () => {
return { stores, authority };
}
it('retries the frozen ownership release after a lost acknowledgement without reacquiring', async () => {
const server = new FakeManagedSessionStore();
const attempts: Array<Record<string, unknown>> = [];
const quiesces: Array<Record<string, unknown>> = [];
const fetchFn = vi.fn<typeof fetch>(async (input, init) => {
if (requestUrl(input).endsWith('/writers:quiesce-ownership')) {
quiesces.push(
JSON.parse(String(init?.body)) as Record<string, unknown>,
);
return jsonResponse({ state: 'QUIESCING' });
}
if (!requestUrl(input).endsWith('/writers:release-ownership'))
return server.fetch(input, init);
const body = JSON.parse(String(init?.body)) as Record<string, unknown>;
attempts.push(body);
server.state = 'SEALED';
if (attempts.length === 1)
throw new TypeError('Lost release acknowledgement');
return jsonResponse({
format: 'qwen-hosted-harness-release/1',
state: 'RELEASED',
...SESSION_KEY,
operationId: body['operationId'],
instanceId: body['instanceId'],
bootId: body['bootId'],
placementEpoch: body['placementEpoch'],
writerGeneration: body['writerGeneration'],
journalRevision: body['journalRevision'],
commitProof: body['commitProof'],
committedSequence: (
body['commitProof'] as { lastCommitSequence: number }
).lastCommitSequence,
});
});
const { stores, authority } = await bootOriginalCsiAuthority(
server,
fetchFn,
);
try {
await stores.quiesceOwnership({
instanceId: 'pod-a',
bootId: 'harness-a',
placementEpoch: 1,
});
expect(quiesces).toEqual([
{
workspaceId: SESSION_KEY.workspaceId,
writerId: 'harness-a',
writerGeneration: 1,
instanceId: 'pod-a',
bootId: 'harness-a',
placementEpoch: 1,
},
]);
await authority.releaseActivation();
const request = {
instanceId: 'pod-a',
bootId: 'harness-a',
placementEpoch: 1,
operationId: randomUUID(),
commitProof: authority.commitProof,
};
const acquisitions = server.fetch.mock.calls.filter(([url]) =>
requestUrl(url).endsWith('/writers:acquire'),
).length;
await expect(stores.releaseOwnership(request)).rejects.toThrow(
'Lost release acknowledgement',
);
await expect(stores.assertWritable()).rejects.toThrow(
'ownership release',
);
const receipt = await stores.releaseOwnership(request);
expect(receipt.state).toBe('RELEASED');
expect(attempts[1]).toEqual(attempts[0]);
expect(
server.fetch.mock.calls.filter(([url]) =>
requestUrl(url).endsWith('/writers:acquire'),
).length,
).toBe(acquisitions);
expect(server.sealCount).toBe(0);
await expect(
stores.releaseOwnership({ ...request, operationId: randomUUID() }),
).rejects.toThrow('identity changed');
} finally {
await stores.stopLocal();
}
});
it('joins the queued original renewal and stages one exact release without HTTP or append', async () => {
const server = new FakeManagedSessionStore();
let enterRenewal!: () => void;
packages/core/src/managed-runtime/http-managed-session-store.ts
+144
View File
@@ -43,6 +43,11 @@ import {
parseHostedRecoveryResource,
} from './hosted-recovery-records.js';
import { MANAGED_EXTENSION_RECORD_BODIES } from './managed-extension-projection.js';
import type {
HostedHarnessPlacement,
HostedHarnessReleaseRequest,
HostedHarnessReleaseReceipt,
} from './hosted-harness-release.js';
import {
MANAGED_TOOL_RESULT_KINDS,
parseToolResultManifestBytes,
@@ -152,6 +157,10 @@ export interface HttpManagedSessionStores {
};
assertWritable(): Promise<void>;
stopLocal(): Promise<void>;
quiesceOwnership(placement: HostedHarnessPlacement): Promise<void>;
releaseOwnership(
request: HostedHarnessReleaseRequest,
): Promise<HostedHarnessReleaseReceipt>;
close(): Promise<void>;
setLifecycleAuthority(authority?: ManagedSessionLifecycleAuthority): void;
authorizeLifecycle(kind?: 'close' | 'delete'): Promise<void>;
@@ -246,6 +255,8 @@ export function createHttpManagedSessionStores(
},
assertWritable: () => client.assertWritable(),
stopLocal: () => client.stopLocal(),
quiesceOwnership: (placement) => client.quiesceOwnership(placement),
releaseOwnership: (request) => client.releaseOwnership(request),
close: () => client.seal(),
setLifecycleAuthority: (authority) =>
client.setLifecycleAuthority(authority),
@@ -731,6 +742,11 @@ class ManagedSessionStoreHttpClient {
private readonly fetchFn: typeof fetch;
private grant: WriterGrant | undefined;
private renewPromise: Promise<void> | undefined;
private ownershipRelease?: {
request: HostedHarnessReleaseRequest;
body: Readonly<Record<string, unknown>>;
receipt?: HostedHarnessReleaseReceipt;
};
/**
* The re-acquisition a lapsed lease started. Seal and local stop await it
* so they never read the grant-less window as "never held a writer", and a
@@ -883,6 +899,8 @@ class ManagedSessionStoreHttpClient {
}
async acquireWriter(): Promise<void> {
if (this.ownershipRelease)
throw corrupt('ownership release has fenced this writer.');
if (this.grant !== undefined || this.sealed || this.locallyStopped) {
throw new ManagedSessionRecordError(
'the HTTP Managed Session writer is already opened or sealed.',
@@ -1843,12 +1861,132 @@ class ManagedSessionStoreHttpClient {
await this.acquirePromise?.catch(() => undefined);
}
async releaseOwnership(
request: HostedHarnessReleaseRequest,
): Promise<HostedHarnessReleaseReceipt> {
assertManagedSessionStableId(request.operationId, 'release operationId');
assertManagedSessionStableId(request.instanceId, 'release instanceId');
if (
request.bootId !== this.writerId ||
!Number.isSafeInteger(request.placementEpoch) ||
request.placementEpoch < 1
)
throw corrupt('ownership release does not match the writer.');
if (!this.ownershipRelease) {
const grant = this.requireGrant();
if (
request.commitProof.lastCommitSequence !== grant.committedSequence ||
request.commitProof.committedPrefixHash !== grant.lastCommitDigest
)
throw corrupt('ownership release does not match the committed head.');
this.stopWriterRenewal();
this.ownershipRelease = {
request: { ...request, commitProof: { ...request.commitProof } },
body: {
workspaceId: this.sessionKey.workspaceId,
writerId: this.writerId,
writerGeneration: grant.writerGeneration,
journalRevision: grant.journalRevision,
...request,
commitProof: { ...request.commitProof },
},
};
}
const release = this.ownershipRelease;
if (
request.operationId !== release.request.operationId ||
request.instanceId !== release.request.instanceId ||
request.bootId !== release.request.bootId ||
request.placementEpoch !== release.request.placementEpoch ||
request.commitProof.lastCommitSequence !==
release.request.commitProof.lastCommitSequence ||
request.commitProof.committedPrefixHash !==
release.request.commitProof.committedPrefixHash
)
throw corrupt('ownership release identity changed.');
if (release.receipt) return release.receipt;
await this.renewPromise?.catch(() => undefined);
const value = asRecord(
await this.json('/writers:release-ownership', 'POST', release.body),
'ownership release receipt',
);
const proof = asRecord(value['commitProof'], 'release commit proof');
const receipt: HostedHarnessReleaseReceipt = {
format: 'qwen-hosted-harness-release/1',
state: 'RELEASED',
tenantId: string(value['tenantId'], 'tenantId'),
workspaceId: string(value['workspaceId'], 'workspaceId'),
sessionId: string(value['sessionId'], 'sessionId'),
operationId: string(value['operationId'], 'operationId'),
instanceId: string(value['instanceId'], 'instanceId'),
bootId: string(value['bootId'], 'bootId'),
placementEpoch: safeCounter(value['placementEpoch'], 'placementEpoch'),
writerGeneration: safeCounter(
value['writerGeneration'],
'writerGeneration',
),
journalRevision: safeCounter(value['journalRevision'], 'journalRevision'),
committedSequence: safeCounter(
value['committedSequence'],
'committedSequence',
),
commitProof: {
lastCommitSequence: safeCounter(
proof['lastCommitSequence'],
'lastCommitSequence',
),
committedPrefixHash: digest(
proof['committedPrefixHash'],
'committedPrefixHash',
),
},
};
if (
value['format'] !== receipt.format ||
value['state'] !== receipt.state ||
!managedSessionKeysEqual(receipt, this.sessionKey) ||
receipt.operationId !== request.operationId ||
receipt.instanceId !== request.instanceId ||
receipt.bootId !== request.bootId ||
receipt.placementEpoch !== request.placementEpoch ||
receipt.writerGeneration !== release.body['writerGeneration'] ||
receipt.journalRevision !== release.body['journalRevision'] ||
receipt.committedSequence !== request.commitProof.lastCommitSequence ||
receipt.commitProof.lastCommitSequence !==
request.commitProof.lastCommitSequence ||
receipt.commitProof.committedPrefixHash !==
request.commitProof.committedPrefixHash
)
throw corrupt('ownership release receipt does not match the request.');
release.receipt = receipt;
this.sealed = true;
this.resources.clear();
return receipt;
}
async quiesceOwnership(placement: HostedHarnessPlacement): Promise<void> {
const grant = this.requireGrant();
const response = asRecord(
await this.json('/writers:quiesce-ownership', 'POST', {
workspaceId: this.sessionKey.workspaceId,
writerId: this.writerId,
writerGeneration: grant.writerGeneration,
...placement,
}),
'ownership quiesce receipt',
);
if (response['state'] !== 'QUIESCING')
throw corrupt('ownership quiesce was not acknowledged.');
}
stopWriterRenewal(): void {
this.renewalStopped = true;
this.stopRenewal();
}
private async ensureWriter(): Promise<void> {
if (this.ownershipRelease)
throw corrupt('ownership release has fenced this writer.');
if (this.retirementJoin)
throw corrupt('original CSI writer admission is closed.');
// A concurrent re-acquisition owns the grant-less window; wait for it
@@ -1889,6 +2027,10 @@ class ManagedSessionStoreHttpClient {
}
private renewWriter(): Promise<void> {
if (this.ownershipRelease)
return Promise.reject(
corrupt('ownership release has fenced this writer.'),
);
if (this.retirementJoin)
return Promise.reject(corrupt('original CSI writer renewal is closed.'));
if (this.locallyStopped) {
@@ -2228,6 +2370,8 @@ class ManagedSessionStoreHttpClient {
timeoutMs = this.requestTimeoutMs,
accept = 'application/json',
): Promise<Response> {
if (this.ownershipRelease && path !== '/writers:release-ownership')
throw corrupt('ownership release has fenced this writer.');
if (this.locallyStopped) {
throw new ManagedSessionRecordError(
'the HTTP Managed Session writer is locally stopped.',
packages/core/src/managed-runtime/managed-extension-projection.test.ts
+12
View File
@@ -107,6 +107,18 @@ const fixtures = JSON.parse(
) as FixtureSuite;
describe('managed-extension-projection/1 fixtures', () => {
it('keeps Store-owned Harness release receipts unavailable to callers', () => {
expect(MANAGED_EXTENSION_RECORD_BODIES).not.toHaveProperty(
'hosted_harness_release',
);
expect(MANAGED_SESSION_ENABLED_DOMAINS).not.toContain(
'hosted_harness_release',
);
expect(MANAGED_SESSION_ENVELOPE_DOMAINS).not.toContain(
'hosted_harness_release',
);
});
it('pins the record bodies, task states and outbox states', () => {
expect(fixtures.contractVersion).toBe(1);
expect(
packages/core/src/managed-runtime/managed-session-assembly.test.ts
+21
View File
@@ -131,6 +131,27 @@ function record(overrides: Partial<ChatRecord>): ChatRecord {
}
describe('managed session assembly', () => {
it('prepares an ownership release without sealing or business close events', async () => {
const workspace = await createWorkspace();
const session = await open(workspace);
const before = session.authority.committedSequence;
const proof = await session.prepareOwnershipRelease();
expect(session.authority.currentActivation?.phase).toBe('released');
expect(session.authority.writerSealed).toBe(false);
expect(proof).toEqual(session.authority.commitProof);
expect(proof.lastCommitSequence).toBeGreaterThan(before);
expect(
session.authority
.eventsInSequenceRange(1, proof.lastCommitSequence)
.filter((event) =>
['turn.settled', 'domain.committed'].includes(event.kind),
),
).toEqual([]);
const prepared = session.authority.committedSequence;
await expect(session.prepareOwnershipRelease()).resolves.toEqual(proof);
expect(session.authority.committedSequence).toBe(prepared);
await session.close({ releaseActivation: false });
});
it('accepts injected journal and resource stores', async () => {
const workspace = await createWorkspace();
const journalStore = new LocalJsonlManagedSessionJournalStore({
packages/core/src/managed-runtime/managed-session-assembly.ts
+20 -5
View File
@@ -15,6 +15,7 @@ import type { SessionWriterLease } from '../services/session-writer-lease.js';
import { LocalManagedSessionResourceStore } from './managed-session-resources.js';
import {
ManagedSessionCommitRejectedError,
type ManagedSessionCommitProof,
type ManagedSessionJournalStore,
type ManagedSessionResourceStore,
} from './managed-session-storage.js';
@@ -55,15 +56,14 @@ const MAX_RENEWAL_FAILURES = 3;
function startActivationRenewal(
authority: LocalManagedSessionAuthority,
leaseDurationMs: number,
): { stop: () => void } {
): { stop: () => Promise<void> } {
const intervalMs = Math.max(Math.floor(leaseDurationMs / 3), 1);
let inFlight = false;
let inFlight: Promise<void> | undefined;
let failures = 0;
let stopped = false;
const timer = setInterval(() => {
if (inFlight || stopped) return;
inFlight = true;
authority
inFlight = authority
.renewActivation({ leaseDurationMs })
.then(() => {
if (!stopped) failures = 0;
@@ -104,7 +104,7 @@ function startActivationRenewal(
}
})
.finally(() => {
inFlight = false;
inFlight = undefined;
});
}, intervalMs);
timer.unref();
@@ -112,6 +112,7 @@ function startActivationRenewal(
stop: () => {
stopped = true;
clearInterval(timer);
return inFlight ?? Promise.resolve();
},
};
}
@@ -204,6 +205,7 @@ export interface ManagedSession {
* refused by the fence.
*/
releaseActivation(): Promise<void>;
prepareOwnershipRelease(): Promise<ManagedSessionCommitProof>;
/**
* Releases the current activation and installs a successor. The next Harness
* handle must present the returned identity; the sink names it automatically.
@@ -323,6 +325,19 @@ export async function openManagedSession(
return activation;
},
releaseActivation: () => authority.releaseActivation(),
prepareOwnershipRelease: async () => {
if (adopted)
throw new ManagedSessionRecordError(
'An adopted writer cannot release ownership.',
);
await renewal.stop();
await authority.releaseActivation();
if (authority.currentActivation?.phase !== 'released')
throw new ManagedSessionRecordError(
'Ownership release requires a released activation.',
);
return authority.commitProof;
},
async replaceActivation(subject, activationId) {
// Refuse before the release, or the current activation would be left
// released with no successor.
packages/core/src/managed-runtime/managed-session-records.test.ts
+2 -2
View File
@@ -840,8 +840,8 @@ describe('managed session per-kind rules', () => {
});
it('registers the v1 domains including file history and session source', () => {
expect(MANAGED_SESSION_DOMAINS).toHaveLength(34);
expect(new Set(MANAGED_SESSION_DOMAINS).size).toBe(34);
expect(MANAGED_SESSION_DOMAINS).toHaveLength(35);
expect(new Set(MANAGED_SESSION_DOMAINS).size).toBe(35);
expect(MANAGED_SESSION_DOMAINS).toContain('session_command');
});
packages/core/src/managed-runtime/managed-session-records.ts
+1
View File
@@ -116,6 +116,7 @@ export const MANAGED_SESSION_DOMAINS = [
'session_metadata',
'file_history',
'session_source',
'hosted_harness_release',
] as const;
export type ManagedSessionDomain = (typeof MANAGED_SESSION_DOMAINS)[number];
packages/managed-agent-server/distribution/kubernetes/Dockerfile
+22
View File
@@ -0,0 +1,22 @@
# Build from the repository root.
# ratchet:docker.io/library/node:22-slim
FROM docker.io/library/node:22-slim@sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5 AS builder
RUN apt-get update && apt-get install -y --no-install-recommends python3 make g++ git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY . .
RUN QWEN_SKIP_PREPARE=1 corepack pnpm install --frozen-lockfile --reporter=append-only \
&& npm run build && npm run bundle && npm run prepare:package \
&& cd dist && npm pack
# ratchet:docker.io/library/node:22-slim
FROM docker.io/library/node:22-slim@sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY --from=builder /src/dist/*.tgz /tmp/
RUN npm install --global /tmp/*.tgz && npm cache clean --force && rm /tmp/*.tgz \
&& mkdir -p /var/run/qwen && chown 10001:10001 /var/run/qwen
COPY packages/managed-agent-server/distribution/kubernetes/harness-*.mjs /opt/qwen/
USER 10001:10001
WORKDIR /var/run/qwen
ENTRYPOINT ["node", "/opt/qwen/harness-launcher.mjs"]
packages/managed-agent-server/distribution/kubernetes/Dockerfile.dockerignore
+18
View File
@@ -0,0 +1,18 @@
node_modules
**/node_modules
dist
**/dist
**/tsconfig.tsbuildinfo
target
**/target
.git
**/.git
.qwen
**/.qwen
.env
**/.env
.env.*
**/.env.*
**/*.p12
**/*.key
**/*.log
packages/managed-agent-server/distribution/kubernetes/README.md
+87
View File
@@ -0,0 +1,87 @@
# Shared Hosted Harness pool
[中文](README.zh-CN.md)
This opt-in package runs the Hosted Harness as a private, horizontally expandable pool. Java retains tenant/actor admission, durable placement and subscriptions. Session RPCs target the selected EndpointSlice Pod address and exact instance/boot; the Service is a discovery anchor. The existing local-process deployment does not enable the pool environment variable or Java pool configuration.
See the [design](../../../../docs/design/2026-10-21-managed-harness-kubernetes.md) for ownership, fencing and the acceptance matrix. These files are deployment templates. No cluster, certificate issuer, metrics adapter or production capacity has been validated by adding them.
## Prerequisites and inputs
- Kubernetes 1.33+ with native sidecars, NetworkPolicy enforcement, an existing Java control plane, durable Session Store and Runtime Broker. Use matching Server/Broker builds and a fresh single-V1 catalog containing the placement columns; earlier development catalogs require a separate fresh database and state directory rather than an upgrade. Harness Pods have no Kubernetes API token or tenant workspace mount.
- Build and publish an immutable Harness image, and pin both container images to verified digests in `harness-pool.yaml`. The example Envoy version is 1.39.4; validate the configuration with the selected image before rollout.
- Fill `settings.example.yaml` with the same capability digest required by Java, an HTTPS Broker URL and the approved model provider. Settings are operator-owned and shared; tenant files must never become operator HOME, settings or working directory.
- Create Secret `qwen-harness-secrets` from files named `harness-token`, `broker-token`, `model-api-key` and `outbound-ca.crt`. The first three contain only their respective credential. The CA file trusts the actual HTTPS Store/Broker/model gateways. The launcher reads files and passes credentials through the child environment; Deployment arguments contain no secrets.
- Create Secret `qwen-harness-tls` with `tls.crt`, `tls.key` and `client-ca.crt`. Envoy verifies the Java client CA and exact URI SAN `spiffe://qwen.internal/managed-agent-server`; change this allowlist to the actual approved identity. Java must present that client identity and trust the Harness server CA.
- **Endpoint TLS identity is a deployment prerequisite:** Java connects to concrete Pod IPs and retains normal HTTPS hostname verification. Every selected Pod IP must appear as an IP SAN in its presented certificate. Supply a per-Pod certificate projection/issuer, or an operator-managed shared certificate covering all actual endpoint IPs and their replacements. A Service DNS certificate alone cannot validate these connections. The Secret mount in the template is an input for that mechanism; it does not issue certificates. Do not deploy with hostname verification disabled.
- Bind `discovery-rbac.yaml` to the actual Java service account/namespace. Discovery reads the named Service and lists EndpointSlices only in the configured namespace, verifying the exact Service UID, named port, Pod UID and readiness. Harness processes have no cluster discovery role.
- The ingress NetworkPolicy assumes Java Pods share the Harness namespace. For a separate Java namespace, add the approved `namespaceSelector` alongside `podSelector` in the same ingress peer. Adapt egress destinations and ports to the actual HTTPS gateways. If a Service maps port `443` to target port `8443`, the CNI may evaluate the backend port; allow that target port only for the approved Store/Broker peers and verify connectivity from an actual Harness Pod. The tested Terway deployment required this explicit backend-port rule.
## Build and configure
Build from a clean source checkout at the repository root. The Dockerfile-specific ignore file excludes local settings, credentials, logs and build outputs:
```sh
docker build -f packages/managed-agent-server/distribution/kubernetes/Dockerfile -t YOUR_REGISTRY/qwen-harness:YOUR_VERSION .
```
Publish the image through the deployment system and replace `qwen-harness:replace-with-built-digest` with its immutable reference. Adapt resource sizes and HTTPS egress destinations to measured workload and actual ports; the supplied CPU/memory requests, limits, heap budget and backlog threshold are starting values for acceptance, not measured production sizing.
Create the Envoy ConfigMap from the reviewed configuration, then apply the filled settings, RBAC and pool manifests in the Harness namespace through your normal deployment workflow:
Each manifest contains one resource. Apply the Deployment in `harness-pool.yaml` together with `harness-service.yaml`, `harness-pdb.yaml` and `harness-network-policy.yaml`. Java discovery requires both `discovery-rbac.yaml` and `discovery-role-binding.yaml`.
```sh
kubectl create configmap qwen-harness-envoy --from-file=envoy.yaml=packages/managed-agent-server/distribution/kubernetes/envoy.yaml --dry-run=client -o yaml
```
The command renders the ConfigMap for review; the templates deliberately contain required image, model and identity inputs. Secrets must come from the operator's secret management system.
Configure the existing Java service with `qwen.managed-agent.harness.enabled=true`, its existing Harness token/capability digest, and these pool properties:
```yaml
qwen:
managed-agent:
harness:
pool:
enabled: true
discovery: kubernetes
namespace: YOUR_HARNESS_NAMESPACE
service-name: qwen-harness
service-uid: YOUR_ACTUAL_SERVICE_UID
port-name: https
key-store-file: /etc/java-tls/client.p12
key-store-password-file: /etc/java-tls/client-password
trust-store-file: /etc/java-tls/trust.p12
trust-store-password-file: /etc/java-tls/trust-password
```
Use HTTPS Store URLs with the pool writer grant supplied by Java. The pool refuses the plaintext Store opt-in and private CSI roots. Tool Runtime provisioning remains the Broker's responsibility. This image carries the Harness and CA certificates, without user workspaces or local tool installations.
## Lifecycle and operations
The Node process listens only on `127.0.0.1:4170`. Envoy exposes `8443` with mTLS, rejects every `Origin`, allows approved private routes, rewrites upstream Host to `127.0.0.1:4170`, and preserves bearer, protocol/boot, client, operation and SSE headers. It installs no buffering/cache filter and disables upstream retries. Its 60-second stream idle window accommodates the 15-second event keepalive. The existing application enforces its request body limits.
Exec startup/readiness probes authenticate and verify instance/boot and pool capabilities. Readiness requires `ACCEPTING`; liveness uses cheap `/health`, without SQL/model/Runtime dependency checks. Proxy listener readiness is also required. These probes do not prove mTLS interoperability; test Java's direct endpoint handshake separately.
`preStop` invokes the authenticated instance drain through loopback, retaining its operation ID for that boot. The native Envoy sidecar stays alive until the Harness stops. Drain immediately stops fresh Session admission, prompts, autonomous wakes and cold execution recovery. Existing foreground Turns may finish their continuation. The Harness itself quiesces and releases eligible resident owners through the Store, so Java unavailability does not prevent that attempt.
The first graceful boundary is resource-idle Sessions and foreground work that reaches that boundary. Unattempted durable wake backlog stays queued. Hooks, MCP, live publishers/background shells, child/monitor resources, cleanup debt, interrupted work and unknown effects keep the owner attached. Failed initialization remains unresolved until the same Session attaches successfully or the boot expires; a generic writer seal is not release proof. Release drops a live idle activation under the barrier, atomically obtains the Store's receipt for the exact placement/head/proof, then closes streams and local timers. It does not detach, cancel accepted inputs or emit SessionEnd. Only zero owners and zero initialization/mutations/resources/publications produce `DRAINED`.
The drain helper waits at most 90 seconds within the 120-second Pod termination grace period. A timeout, failed Store call or unsupported work does not prove release: Kubernetes eventually takes the crash path, and Java must wait for writer/placement lease fencing before recovering on another boot. SIGTERM, Pod absence, readiness loss and an HTTP release acknowledgement lost in transit are not durable release proof. A live stream reconnects on a new owner with its new event epoch and committed cursor.
Rolling updates use zero unavailable Pods, one surge Pod and a PDB minimum of one. The PDB does not select an idle victim or guarantee graceful release. Arbitrary deletion, eviction and forced termination must pass the design's failure matrix before enabling workloads beyond the first boundary.
## Metrics and optional scale-out
Authenticated `GET /internal/hosted-harness/v1/metrics` reports Hosted-owned attached/active/opening/input/resource/publication/mutation counters plus RSS, heap and event-loop utilization. The endpoint requires the same bearer and exact protocol/boot headers as instance administration; it has no tenant/session labels. Neither ACP gauges nor missing metrics represent empty Hosted capacity. `pendingInputs` includes durable waits and is not itself a runnable-backlog metric.
Apply `hpa.yaml` only after providing an external metrics adapter for `qwen_harness_runnable_backlog{pool=qwen-harness}`, derived from actual Java runnable admission and the Hosted counters. Exclude approval waits, unknown outcomes and work blocked by external quotas; preserve unknown/missing samples as unavailable metrics. The adapter is an operator integration and is not included here. Measure the per-Pod target before using the example value `4`. Remove the declarative Deployment `replicas` field from subsequent rollout manifests when handing replica control to HPA. The HPA expands at most one Pod per minute within 2..10 replicas and sets `scaleDown.selectPolicy: Disabled`. Automatic scale-in remains disabled pending its own resource/failure acceptance; no additional replica controller is introduced.
## Deployment acceptance
Validate the selected Envoy image configuration, Kubernetes server-side schema and secret mounts before rollout. In an isolated acceptance namespace, prove two instance/boot identities, real Java mTLS per endpoint, rejection of wrong CA/SAN/bearer/Origin/boot, Session placement affinity across two Pods and streamed event cursor/epoch continuity. Exercise an idle release, a foreground Turn plus queued wake, lost release ACK/retry, Store/Java outage, unknown tool effects, a busy arbitrary victim and the bounded crash recovery path. Verify local single-process regressions separately. A healthy listener or parsed YAML does not establish these outcomes.
References: [Kubernetes native sidecars](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/), [Envoy TLS validation](https://www.envoyproxy.io/docs/envoy/v1.39.4/api-v3/extensions/transport_sockets/tls/v3/common.proto), [HPA behavior](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/).
The first pool does not expose MCP management APIs or public child-agent delegation. Existing single-instance deployments retain those capabilities from main. Historical placement-V17 ACK acceptance and later V18 CI results predate the current single-V1 integration; fresh-baseline, Java/MySQL/Hosted and cross-language CI acceptance remain required. Preserve old artifacts and their own state for historical access; do not repair Flyway history to reuse them.
packages/managed-agent-server/distribution/kubernetes/README.zh-CN.md
+87
View File
@@ -0,0 +1,87 @@
# 共享 Hosted Harness 池
[English](README.md)
本部署包通过显式开启的方式，将 Hosted Harness 运行成可水平扩展的私有池。Java 继续负责租户和用户准入、持久化 placement 与订阅。Session RPC 绑定选定的 EndpointSlice Pod 地址和确切 instance/boot；Service 用作发现锚点。现有本地进程部署不设置池环境变量，也不启用 Java 池配置。
所有权、隔离栅栏与验收矩阵见[设计文档](../../../../docs/design/2026-10-21-managed-harness-kubernetes.zh-CN.md)。这些文件是部署模板，新增这些文件并不意味着已经验证集群、证书签发、指标适配器或生产容量。
## 前提与输入
- Kubernetes 1.33+，支持原生 sidecar 并执行 NetworkPolicy；已有 Java 控制面、持久化 Session Store 和 Runtime Broker。Server/Broker 使用匹配构建及包含 placement 列的全新单 V1 catalog；旧开发库需要独立的全新数据库和状态目录，不提供升级。Harness Pod 不持有 Kubernetes API token，也不挂载租户工作目录。
- 构建并发布不可变 Harness 镜像，在 `harness-pool.yaml` 中将两个镜像都固定到已验证的 digest。示例 Envoy 版本为 1.39.4；上线前用实际选定镜像验证配置。
- 填写 `settings.example.yaml`：使用 Java 同样要求的 capability digest、HTTPS Broker URL 和批准的模型配置。设置由运维持有并共享；租户文件不得成为运维 HOME、设置文件或工作目录。
- 从文件创建 `qwen-harness-secrets` Secret，键名为 `harness-token`、`broker-token`、`model-api-key` 和 `outbound-ca.crt`。前三个文件分别只包含对应凭据。CA 文件信任真实 HTTPS Store、Broker、模型网关。启动器读取文件并通过子进程环境传递凭据；Deployment 参数不包含秘密。
- 创建 `qwen-harness-tls` Secret，提供 `tls.crt`、`tls.key` 和 `client-ca.crt`。Envoy 同时验证 Java 客户端 CA 和精确 URI SAN `spiffe://qwen.internal/managed-agent-server`；将白名单改成实际批准的身份。Java 必须提供该客户端身份，并信任 Harness 服务端 CA。
- **端点 TLS 身份是部署前提：** Java 连接具体 Pod IP，并保留正常 HTTPS 主机名验证。每个选中 Pod IP 都必须出现在该端点证书的 IP SAN 中。请提供每个 Pod 的证书投影/签发机制，或由运维维护覆盖所有实际端点 IP 及替换 IP 的共享证书。只有 Service DNS 的证书不能验证这些连接。模板的 Secret 挂载只是该机制的输入，不负责签发证书。不得关闭主机名验证来部署。
- 将 `discovery-rbac.yaml` 绑定到真实 Java service account 和 namespace。发现逻辑只在配置的 namespace 读取指定 Service 并列出 EndpointSlice，核验精确 Service UID、命名端口、Pod UID 和就绪状态。Harness 进程不持有集群发现权限。
- 入口 NetworkPolicy 假设 Java Pod 与 Harness 位于同一 namespace。如果 Java 位于独立 namespace，在同一个 ingress peer 中为 `podSelector` 加上获准的 `namespaceSelector`。出口目标和端口也须适配真实 HTTPS 网关。如果 Service 将 `443` 映射到目标端口 `8443`，CNI 可能按后端端口执行策略；仅为批准的 Store/Broker peer 放行该目标端口，并从实际 Harness Pod 验证连通性。本次实测的 Terway 部署需要这条显式后端端口规则。
## 构建与配置
在干净的源码 checkout 中从仓库根目录构建。Dockerfile 专用 ignore 文件排除本地设置、凭据、日志与构建产物：
```sh
docker build -f packages/managed-agent-server/distribution/kubernetes/Dockerfile -t YOUR_REGISTRY/qwen-harness:YOUR_VERSION .
```
通过部署系统发布镜像，用其不可变引用替换 `qwen-harness:replace-with-built-digest`。根据实测工作负载和实际端口调整资源及 HTTPS 出站目标；示例 CPU/内存 requests、limits、堆预算和 backlog 阈值是验收起点，不是实测生产容量。
从审核过的配置生成 Envoy ConfigMap，再通过正常部署流程在 Harness namespace 应用填好的设置、RBAC 与池清单：
每份清单只包含一个资源。应用 `harness-pool.yaml` 中的 Deployment 时，同时应用 `harness-service.yaml`、`harness-pdb.yaml` 和 `harness-network-policy.yaml`。Java 发现同时需要 `discovery-rbac.yaml` 与 `discovery-role-binding.yaml`。
```sh
kubectl create configmap qwen-harness-envoy --from-file=envoy.yaml=packages/managed-agent-server/distribution/kubernetes/envoy.yaml --dry-run=client -o yaml
```
该命令生成待审核 ConfigMap；模板有意保留必须填写的镜像、模型和身份输入。秘密应来自运维的秘密管理系统。
在已有 Java 服务中配置 `qwen.managed-agent.harness.enabled=true`、现有 Harness token/capability digest，以及以下池属性：
```yaml
qwen:
managed-agent:
harness:
pool:
enabled: true
discovery: kubernetes
namespace: YOUR_HARNESS_NAMESPACE
service-name: qwen-harness
service-uid: YOUR_ACTUAL_SERVICE_UID
port-name: https
key-store-file: /etc/java-tls/client.p12
key-store-password-file: /etc/java-tls/client-password
trust-store-file: /etc/java-tls/trust.p12
trust-store-password-file: /etc/java-tls/trust-password
```
Store URL 使用 HTTPS，由 Java 提供池 writer grant。池拒绝显式放宽明文 Store 的开关及私有 CSI 根。Tool Runtime 仍由 Broker 提供。镜像携带 Harness 与 CA 证书，不包含用户工作目录或本地工具安装。
## 生命周期与运维
Node 进程只监听 `127.0.0.1:4170`。Envoy 在 `8443` 提供 mTLS，拒绝所有 `Origin`，只开放批准的私有路由，将上游 Host 改写为 `127.0.0.1:4170`，保留 bearer、protocol/boot、client、operation 和 SSE 头。代理不安装缓冲/缓存过滤器，并关闭上游重试。60 秒流空闲窗口容纳现有 15 秒事件 keepalive。现有应用执行请求体大小限制。
Exec startup/readiness 探针认证并核验 instance/boot 和池能力。Readiness 要求 `ACCEPTING`；liveness 使用轻量 `/health`，不检查 SQL、模型、Runtime 依赖。代理监听器也必须就绪。这些探针不能证明 mTLS 互通，仍须单独验证 Java 对具体端点的握手。
`preStop` 通过 loopback 调用认证后的 instance drain，并保留该 boot 的 operation ID。原生 Envoy sidecar 持续运行至 Harness 停止。Drain 立即拒绝新 Session、新 prompt、自主 wake 和冷恢复执行；已有前台 Turn 可以继续完成。Harness 自己通过 Store 对符合条件的驻留 owner 执行 quiesce 和释放，因此 Java 不可用不妨碍该尝试。
首个优雅迁移边界是无活跃资源的 Session，以及能到达该边界的前台工作。尚未尝试的持久化 wake backlog 保留排队状态。Hooks、MCP、活跃 publisher/后台 shell、child/monitor 资源、清理欠账、中断工作和未知副作用都会保留 owner。失败的初始化保持未决，直到同一 Session 成功附着或该 boot 过期；普通 writer seal 不是释放证明。释放在屏障下撤销空闲的活跃 activation，原子获取 Store 针对精确 placement、head 和 proof 的回执，再关闭流及本地计时器。它不 detach、不取消已接收输入、不发出 SessionEnd。只有 owner、初始化、变更请求、资源及 publication 全部为零时才是 `DRAINED`。
Drain helper 在 Pod 的 120 秒终止宽限内最多等待 90 秒。超时、Store 调用失败或不支持的工作均不构成释放证明：Kubernetes 最终进入崩溃路径，Java 必须等 writer/placement lease 隔离后才可在另一 boot 恢复。SIGTERM、Pod 消失、readiness 丢失或传输中丢失的 HTTP 释放响应都不是持久化释放证明。活跃流使用新事件 epoch 和已提交 cursor 在新 owner 重连。
滚动更新设置零不可用 Pod、一个额外 Pod，并使用至少一个可用 Pod 的 PDB。PDB 不会选择空闲 victim，也不保证优雅释放。启用超出首个边界的工作负载前，必须验证设计中的任意删除、驱逐和强制终止故障矩阵。
## 指标与可选扩容
认证后的 `GET /internal/hosted-harness/v1/metrics` 提供 Harness 自有 attached、active、opening、input、resource、publication、mutation 计数，以及 RSS、heap、event-loop utilization。端点要求与 instance 管理相同的 bearer 和精确 protocol/boot 头，不包含 tenant/session 标签。ACP 指标或缺失指标都不能表示 Hosted 空闲容量。`pendingInputs` 包含持久化等待，不能直接作为可运行 backlog 指标。
只有接入外部指标适配器 `qwen_harness_runnable_backlog{pool=qwen-harness}` 后才应用 `hpa.yaml`；指标必须依据 Java 真实可运行准入与 Hosted 计数构建。排除审批等待、未知结果及被外部配额阻塞的工作，缺失/未知样本保留为不可用指标。适配器是运维集成，本目录不提供实现。先实测每 Pod 目标再使用示例值 `4`。交由 HPA 管理副本时，从后续发布清单删除 Deployment 的声明式 `replicas` 字段。HPA 在 2..10 副本内每分钟最多新增一个 Pod，并配置 `scaleDown.selectPolicy: Disabled`。自动缩容需要独立的资源/故障验收，当前保持关闭；不增加其他副本控制器。
## 部署验收
上线前验证所选 Envoy 镜像配置、Kubernetes 服务端 schema 和 Secret 挂载。在隔离验收 namespace 证明两个 instance/boot 身份、Java 对每个具体端点的真实 mTLS、错误 CA/SAN/bearer/Origin/boot 拒绝、两个 Pod 间 Session placement 亲和，以及事件 cursor/epoch 的流式衔接。覆盖空闲释放、前台 Turn 加排队 wake、释放 ACK 丢失及重试、Store/Java 不可用、未知工具副作用、忙碌任意 victim 和有界崩溃恢复路径。单独验证本地单进程回归。监听器健康或 YAML 可解析都不能证明这些结果。
参考：[Kubernetes 原生 sidecar](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/)、[Envoy TLS 验证](https://www.envoyproxy.io/docs/envoy/v1.39.4/api-v3/extensions/transport_sockets/tls/v3/common.proto)、[HPA 行为](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/)。
首版 pool 不开放 MCP 管理 API 或公开子代理委派；现有单实例部署保留 main 中的这些能力。历史 placement V17 的 ACK 验收及后续 V18 的 CI 结果均早于当前单 V1 衔接；仍需全新基线、Java/MySQL/Hosted 及跨语言 CI 验收。需要历史访问时保留原产物及其独立状态，不得通过修复 Flyway 历史复用旧库。
packages/managed-agent-server/distribution/kubernetes/discovery-rbac.yaml
+21
View File
@@ -0,0 +1,21 @@
# Apply in the Harness namespace; bind the existing Java service account.
apiVersion: 'rbac.authorization.k8s.io/v1'
kind: 'Role'
metadata:
name: 'qwen-harness-discovery'
rules:
- apiGroups:
- ''
resources:
- 'services'
resourceNames:
- 'qwen-harness'
verbs:
- 'get'
- apiGroups:
- 'discovery.k8s.io'
resources:
- 'endpointslices'
verbs:
- 'list'
- 'watch'
packages/managed-agent-server/distribution/kubernetes/discovery-role-binding.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: 'rbac.authorization.k8s.io/v1'
kind: 'RoleBinding'
metadata:
name: 'qwen-harness-discovery'
roleRef:
apiGroup: 'rbac.authorization.k8s.io'
kind: 'Role'
name: 'qwen-harness-discovery'
subjects:
- kind: 'ServiceAccount'
name: 'managed-agent-server'
# Set the actual Java service account namespace before applying.
namespace: 'default'
packages/managed-agent-server/distribution/kubernetes/envoy.yaml
+102
View File
@@ -0,0 +1,102 @@
static_resources:
listeners:
- name: 'harness_mtls'
address:
socket_address:
address: '0.0.0.0'
port_value: 8443
filter_chains:
- transport_socket:
name: 'envoy.transport_sockets.tls'
typed_config:
'@type': 'type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext'
require_client_certificate: true
common_tls_context:
tls_params:
tls_minimum_protocol_version: 'TLSv1_2'
tls_certificates:
- certificate_chain:
filename: '/etc/qwen/tls/tls.crt'
private_key:
filename: '/etc/qwen/tls/tls.key'
validation_context:
trusted_ca:
filename: '/etc/qwen/tls/client-ca.crt'
match_typed_subject_alt_names:
- san_type: 'URI'
matcher:
exact: 'spiffe://qwen.internal/managed-agent-server'
filters:
- name: 'envoy.filters.network.http_connection_manager'
typed_config:
'@type': 'type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager'
stat_prefix: 'hosted_harness'
use_remote_address: true
normalize_path: true
merge_slashes: true
path_with_escaped_slashes_action: 'REJECT_REQUEST'
stream_idle_timeout: '60s'
request_headers_timeout: '10s'
max_request_headers_kb: 32
route_config:
name: 'harness_private'
virtual_hosts:
- name: 'harness'
domains:
- '*'
request_headers_to_remove:
- 'x-envoy-retry-on'
- 'x-envoy-retry-grpc-on'
- 'x-envoy-max-retries'
- 'x-envoy-upstream-rq-timeout-ms'
- 'x-envoy-upstream-rq-per-try-timeout-ms'
routes:
- match:
prefix: '/'
headers:
- name: 'origin'
present_match: true
direct_response:
status: 403
- match:
path: '/capabilities'
route: &upstream
cluster: 'harness_loopback'
host_rewrite_literal: '127.0.0.1:4170'
timeout: '0s'
retry_policy:
retry_on: ''
num_retries: 0
- match:
path: '/health'
route: *upstream
- match:
path: '/session'
route: *upstream
- match:
prefix: '/session/'
route: *upstream
- match:
prefix: '/internal/hosted-harness/v1/'
route: *upstream
- match:
prefix: '/'
direct_response:
status: 404
http_filters:
- name: 'envoy.filters.http.router'
typed_config:
'@type': 'type.googleapis.com/envoy.extensions.filters.http.router.v3.Router'
clusters:
- name: 'harness_loopback'
type: 'STATIC'
connect_timeout: '2s'
load_assignment:
cluster_name: 'harness_loopback'
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: '127.0.0.1'
port_value: 4170
packages/managed-agent-server/distribution/kubernetes/harness-launcher.mjs
+96
View File
@@ -0,0 +1,96 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/
import { spawn } from 'node:child_process';
import console from 'node:console';
import { mkdir, readFile } from 'node:fs/promises';
import process from 'node:process';
import { URL } from 'node:url';
const secrets = '/etc/qwen/secrets';
const runtime = '/var/run/qwen';
async function secret(name) {
const value = (await readFile(`${secrets}/${name}`, 'utf8')).trim();
if (!value || /[\r\n\0]/u.test(value))
throw new Error(`Invalid ${name} secret file.`);
return value;
}
try {
if (
!/^[A-Za-z0-9._:-]{1,128}$/u.test(
process.env.QWEN_HOSTED_HARNESS_INSTANCE_ID ?? '',
)
)
throw new Error('A Pod instance identity is required.');
if (
!/^sha256:[0-9a-f]{64}$/u.test(
process.env.QWEN_HOSTED_HARNESS_CAPABILITY_DIGEST ?? '',
)
)
throw new Error('A verified capability digest is required.');
const brokerUrl = new URL(process.env.QWEN_HOSTED_HARNESS_BROKER_URL);
if (
brokerUrl.protocol !== 'https:' ||
brokerUrl.username ||
brokerUrl.password ||
brokerUrl.search ||
brokerUrl.hash
)
throw new Error('The pool requires an HTTPS Broker URL.');
await Promise.all(
['home', 'state', 'workspace'].map((name) =>
mkdir(`${runtime}/${name}`, { recursive: true, mode: 0o700 }),
),
);
const child = spawn(
'/usr/local/bin/qwen',
[
'serve',
'--profile',
'hosted-harness',
'--http-bridge',
'--no-web',
'--hostname',
'127.0.0.1',
'--port',
'4170',
'--workspace',
`${runtime}/workspace`,
],
{
cwd: `${runtime}/workspace`,
stdio: 'inherit',
env: {
...process.env,
HOME: `${runtime}/home`,
QWEN_HOME: `${runtime}/state`,
QWEN_CODE_SYSTEM_SETTINGS_PATH: '/etc/qwen/config/settings.json',
QWEN_SERVER_TOKEN: await secret('harness-token'),
QWEN_HOSTED_HARNESS_BROKER_TOKEN: await secret('broker-token'),
OPENAI_API_KEY: await secret('model-api-key'),
},
},
);
for (const signal of ['SIGTERM', 'SIGINT'])
process.on(signal, () => child.kill(signal));
child.once('error', () => {
console.error('Hosted Harness child could not start.');
process.exitCode = 1;
});
child.once('exit', (code, signal) => {
process.exitCode = code ?? (signal ? 1 : 0);
});
} catch (error) {
// Never print child environment or secret file contents.
console.error(
error instanceof Error && error.message.startsWith('The pool')
? error.message
: 'Hosted Harness configuration or secret files are invalid.',
);
process.exitCode = 1;
}
packages/managed-agent-server/distribution/kubernetes/harness-network-policy.yaml
+36
View File
@@ -0,0 +1,36 @@
apiVersion: 'networking.k8s.io/v1'
kind: 'NetworkPolicy'
metadata:
name: 'qwen-harness'
spec:
podSelector:
matchLabels:
app: 'qwen-harness'
policyTypes:
- 'Ingress'
- 'Egress'
ingress:
- from:
- podSelector:
matchLabels:
app: 'managed-agent-server'
ports:
- protocol: 'TCP'
port: 8443
# Narrow HTTPS destinations to the actual Store, Broker and model egress gateway.
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: 'kube-system'
podSelector:
matchLabels:
k8s-app: 'kube-dns'
ports:
- protocol: 'UDP'
port: 53
- protocol: 'TCP'
port: 53
- ports:
- protocol: 'TCP'
port: 443
packages/managed-agent-server/distribution/kubernetes/harness-pdb.yaml
+9
View File
@@ -0,0 +1,9 @@
apiVersion: 'policy/v1'
kind: 'PodDisruptionBudget'
metadata:
name: 'qwen-harness'
spec:
minAvailable: 1
selector:
matchLabels:
app: 'qwen-harness'
packages/managed-agent-server/distribution/kubernetes/harness-pool.yaml
+178
View File
@@ -0,0 +1,178 @@
apiVersion: 'apps/v1'
kind: 'Deployment'
metadata:
name: 'qwen-harness'
spec:
replicas: 2
selector:
matchLabels:
app: 'qwen-harness'
strategy:
type: 'RollingUpdate'
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
metadata:
labels:
app: 'qwen-harness'
spec:
automountServiceAccountToken: false
terminationGracePeriodSeconds: 120
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: 'RuntimeDefault'
# A native sidecar remains available until the Harness preStop completes.
initContainers:
- name: 'mtls'
restartPolicy: 'Always'
image: 'docker.io/envoyproxy/envoy:v1.39.4'
command:
- '/usr/local/bin/envoy'
args:
- '-c'
- '/etc/envoy/envoy.yaml'
- '--log-level'
- 'warning'
- '--concurrency'
- '1'
ports:
- name: 'https'
containerPort: 8443
startupProbe:
tcpSocket:
port: 'https'
periodSeconds: 2
failureThreshold: 30
readinessProbe:
tcpSocket:
port: 'https'
periodSeconds: 5
resources:
requests:
cpu: '100m'
memory: '128Mi'
limits:
cpu: '1'
memory: '256Mi'
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- 'ALL'
volumeMounts:
- name: 'envoy'
mountPath: '/etc/envoy'
readOnly: true
- name: 'tls'
mountPath: '/etc/qwen/tls'
readOnly: true
containers:
- name: 'harness'
# Replace with the immutable image built from this directory's Dockerfile.
image: 'qwen-harness:replace-with-built-digest'
env:
- name: 'QWEN_HOSTED_HARNESS_INSTANCE_ID'
valueFrom:
fieldRef:
fieldPath: 'metadata.uid'
- name: 'QWEN_HOSTED_HARNESS_CAPABILITY_DIGEST'
valueFrom:
configMapKeyRef:
name: 'qwen-harness-settings'
key: 'capability-digest'
- name: 'QWEN_HOSTED_HARNESS_BROKER_URL'
valueFrom:
configMapKeyRef:
name: 'qwen-harness-settings'
key: 'broker-url'
- name: 'NODE_EXTRA_CA_CERTS'
value: '/etc/qwen/secrets/outbound-ca.crt'
- name: 'NODE_OPTIONS'
value: '--max-old-space-size=1536'
startupProbe:
exec:
command:
- 'node'
- '/opt/qwen/harness-probe.mjs'
- 'startup'
timeoutSeconds: 5
periodSeconds: 5
failureThreshold: 30
readinessProbe:
exec:
command:
- 'node'
- '/opt/qwen/harness-probe.mjs'
- 'ready'
timeoutSeconds: 5
periodSeconds: 5
livenessProbe:
exec:
command:
- 'node'
- '/opt/qwen/harness-probe.mjs'
- 'live'
timeoutSeconds: 5
periodSeconds: 10
failureThreshold: 6
lifecycle:
preStop:
exec:
command:
- 'node'
- '/opt/qwen/harness-probe.mjs'
- 'drain'
resources:
requests:
cpu: '1'
memory: '1Gi'
limits:
cpu: '2'
memory: '3Gi'
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- 'ALL'
volumeMounts:
- name: 'runtime'
mountPath: '/var/run/qwen'
- name: 'tmp'
mountPath: '/tmp'
- name: 'settings'
mountPath: '/etc/qwen/config'
readOnly: true
- name: 'secrets'
mountPath: '/etc/qwen/secrets'
readOnly: true
volumes:
- name: 'runtime'
emptyDir:
sizeLimit: '1Gi'
- name: 'tmp'
emptyDir:
sizeLimit: '256Mi'
- name: 'settings'
configMap:
name: 'qwen-harness-settings'
items:
- key: 'settings.json'
path: 'settings.json'
- name: 'envoy'
configMap:
name: 'qwen-harness-envoy'
- name: 'secrets'
secret:
secretName: 'qwen-harness-secrets'
defaultMode: 288
- name: 'tls'
secret:
secretName: 'qwen-harness-tls'
defaultMode: 288
packages/managed-agent-server/distribution/kubernetes/harness-probe.mjs
+118
View File
@@ -0,0 +1,118 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/
import { randomUUID } from 'node:crypto';
import console from 'node:console';
import { readFile, writeFile } from 'node:fs/promises';
import process from 'node:process';
import { setTimeout } from 'node:timers/promises';
const { fetch, AbortSignal } = globalThis;
const origin = 'http://127.0.0.1:4170';
const prefix = '/internal/hosted-harness/v1';
const mode = process.argv[2];
const deadline = Date.now() + (mode === 'drain' ? 90_000 : 3_000);
try {
const token = (
await readFile('/etc/qwen/secrets/harness-token', 'utf8')
).trim();
if (!token || /[\r\n\0]/u.test(token)) throw new Error('Invalid token.');
const request = async (path, bootId, body) => {
const response = await fetch(`${origin}${path}`, {
method: body === undefined ? 'GET' : 'POST',
headers: {
Authorization: `Bearer ${token}`,
...(bootId
? {
'X-Qwen-Harness-Protocol-Version': '1',
'X-Qwen-Harness-Boot-Id': bootId,
}
: {}),
...(body ? { 'Content-Type': 'application/json' } : {}),
},
body: body === undefined ? undefined : JSON.stringify(body),
signal: AbortSignal.timeout(
Math.max(1, Math.min(2500, deadline - Date.now())),
),
});
const value = await response.json();
return { response, value };
};
if (mode === 'live') {
const { response, value } = await request('/health');
if (!response.ok || value.status !== 'ok')
throw new Error('Health refused.');
} else {
if (!['startup', 'ready', 'drain'].includes(mode))
throw new Error('Unknown probe.');
const capabilities = await request('/capabilities');
const contract = capabilities.value.hostedHarness;
if (
!capabilities.response.ok ||
!contract ||
contract.instanceId !== process.env.QWEN_HOSTED_HARNESS_INSTANCE_ID ||
contract.capabilityDigest !==
process.env.QWEN_HOSTED_HARNESS_CAPABILITY_DIGEST ||
contract.protocolVersions?.current !== 1 ||
!/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/iu.test(
contract.bootId ?? '',
) ||
!contract.features?.includes('harness-pool-owner/1') ||
!contract.features?.includes('harness-instance-lifecycle/1')
)
throw new Error('Pool contract unavailable.');
const readInstance = async () => {
const result = await request(`${prefix}/instance`, contract.bootId);
if (
!result.response.ok ||
result.value.format !== 'qwen-hosted-harness-instance/1' ||
result.value.instanceId !== contract.instanceId ||
result.value.bootId !== contract.bootId
)
throw new Error('Instance generation unavailable.');
return result.value;
};
let instance = await readInstance();
if (mode === 'ready' && instance.state !== 'ACCEPTING')
throw new Error('Quiescing.');
if (mode === 'drain') {
if (instance.state === 'ACCEPTING') {
const path = '/var/run/qwen/drain-operation.json';
let body;
try {
const saved = JSON.parse(await readFile(path, 'utf8'));
if (saved.expectedBootId === contract.bootId) body = saved;
} catch {
/* A new boot creates its own drain identity. */
}
body ??= { operationId: randomUUID(), expectedBootId: contract.bootId };
await writeFile(path, JSON.stringify(body), { mode: 0o600 });
const result = await request(
`${prefix}/instance:drain`,
contract.bootId,
body,
);
// Another authenticated controller can already own this boot's drain.
if (
result.response.status !== 202 &&
result.value.code !== 'hosted_harness_drain_conflict'
)
throw new Error('Drain refused.');
}
while (Date.now() < deadline) {
instance = await readInstance();
if (instance.state === 'DRAINED') break;
await setTimeout(Math.min(500, Math.max(1, deadline - Date.now())));
}
if (instance.state !== 'DRAINED')
throw new Error('Drain deadline elapsed.');
}
}
} catch {
console.error(`Hosted Harness ${mode ?? 'unknown'} probe did not pass.`);
process.exitCode = 1;
}
packages/managed-agent-server/distribution/kubernetes/harness-service.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: 'v1'
kind: 'Service'
metadata:
name: 'qwen-harness'
spec:
selector:
app: 'qwen-harness'
ports:
- name: 'https'
port: 8443
targetPort: 'https'
packages/managed-agent-server/distribution/kubernetes/hpa.yaml
+32
View File
@@ -0,0 +1,32 @@
# Apply only after the external metrics adapter and capacity threshold are verified.
apiVersion: 'autoscaling/v2'
kind: 'HorizontalPodAutoscaler'
metadata:
name: 'qwen-harness'
spec:
scaleTargetRef:
apiVersion: 'apps/v1'
kind: 'Deployment'
name: 'qwen-harness'
minReplicas: 2
maxReplicas: 10
behavior:
scaleUp:
stabilizationWindowSeconds: 30
policies:
- type: 'Pods'
value: 1
periodSeconds: 60
scaleDown:
selectPolicy: 'Disabled'
metrics:
- type: 'External'
external:
metric:
name: 'qwen_harness_runnable_backlog'
selector:
matchLabels:
pool: 'qwen-harness'
target:
type: 'AverageValue'
averageValue: '4'
packages/managed-agent-server/distribution/kubernetes/settings.example.yaml
+21
View File
@@ -0,0 +1,21 @@
apiVersion: 'v1'
kind: 'ConfigMap'
metadata:
name: 'qwen-harness-settings'
data:
# Replace with the digest selected and verified by Java and this Harness build.
capability-digest: 'sha256:REPLACE_WITH_64_LOWERCASE_HEX_CHARACTERS'
broker-url: 'https://runtime-broker.example.internal'
settings.json: |
{
"security": { "auth": { "selectedType": "openai" } },
"model": { "name": "REPLACE_WITH_APPROVED_MODEL_ID" },
"telemetry": { "enabled": false },
"modelProviders": {
"openai": [{
"id": "REPLACE_WITH_APPROVED_MODEL_ID",
"envKey": "OPENAI_API_KEY",
"baseUrl": "https://model-gateway.example.internal/v1"
}]
}
}
packages/managed-agent-server/pom.xml
+5 -2
View File
@@ -714,8 +714,11 @@
adds 13 s (2x its measured 6.5 s): healthy 10636 s, fork 11236 s,
Verify 198 minutes. The FG12e Runtime connection replacement cell
adds 28 s (2x its measured 13.8 s): healthy 10664 s, fork 11264 s,
Verify 198 minutes. -->
<forkedProcessTimeoutInSeconds>11264</forkedProcessTimeoutInSeconds>
Verify 198 minutes. The pool fixture provisionally reserves 180 s
(its 120 s method ceiling plus 60 s for process/context cleanup),
pending CI measurement: healthy 10844 s, fork 11444 s,
Verify 201 minutes. -->
<forkedProcessTimeoutInSeconds>11444</forkedProcessTimeoutInSeconds>
</configuration>
<executions>
<execution>
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/api/ManagedSessionStoreController.java
+17
View File
@@ -114,6 +114,23 @@ public class ManagedSessionStoreController {
writerToken, request, lifecycleAuthority(headers));
}
@PostMapping("/writers:release-ownership")
public com.fasterxml.jackson.databind.JsonNode releaseOwnership(TenantContext tenant,
@PathVariable String sessionId,
@RequestHeader(ManagedSessionStoreModels.WRITER_TOKEN_HEADER) String writerToken,
@Valid @RequestBody ManagedSessionStoreModels.ReleaseOwnershipRequest request) {
return store.releaseOwnership(tenant.tenantId(), sessionId, writerToken, request);
}
@PostMapping("/writers:quiesce-ownership")
public java.util.Map<String, String> quiesceOwnership(TenantContext tenant,
@PathVariable String sessionId,
@RequestHeader(ManagedSessionStoreModels.WRITER_TOKEN_HEADER) String writerToken,
@Valid @RequestBody ManagedSessionStoreModels.QuiesceOwnershipRequest request) {
store.quiesceOwnership(tenant.tenantId(), sessionId, writerToken, request);
return java.util.Map.of("state", "QUIESCING");
}
@PostMapping("/recovery:block")
public RecoveryStateReceipt blockRecovery(TenantContext tenant,
@PathVariable String sessionId,
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/config/BrokerSecurity.java
+13
View File
@@ -279,6 +279,19 @@ public class BrokerSecurity {
if (!properties.getHarness().isEnabled()) {
return;
}
if (properties.getHarness().getPool().isEnabled()) {
if (!properties.getSessionStore().isEnabled() || StringUtils.isBlank(properties.getSessionStore().getBindingKey())) {
throw new IllegalStateException("Harness pool requires an enabled Session Store and binding key");
}
for (var instance : properties.getHarness().getPool().getInstances()) {
java.net.URI endpoint = java.net.URI.create(instance.getBaseUrl());
if (!"https".equals(endpoint.getScheme())
&& !("http".equals(endpoint.getScheme()) && isLoopback(hostOf(endpoint)))) {
throw new IllegalStateException("Harness pool requires TLS for remote endpoints");
}
}
return;
}
String baseUrl = properties.getHarness().getBaseUrl();
if (StringUtils.isBlank(baseUrl)) {
return;
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/config/HarnessConfiguration.java
+46 -1
View File
@@ -1,19 +1,28 @@
package com.alibaba.qwen.code.managedagent.config;
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector;
import com.alibaba.qwen.code.managedagent.harness.HarnessDiscovery;
import com.alibaba.qwen.code.managedagent.harness.HarnessPoolConnector;
import com.alibaba.qwen.code.managedagent.harness.KubernetesHarnessDiscovery;
import com.alibaba.qwen.code.managedagent.harness.QwenHostedHarnessConnector;
import com.alibaba.qwen.code.managedagent.harness.UnavailableHarnessConnector;
import com.alibaba.qwen.code.managedagent.persistence.HarnessPlacementStore;
import com.alibaba.qwen.code.managedagent.persistence.ManagedActionStore;
import com.alibaba.qwen.code.managedagent.persistence.ManagedAgentDefinitionStore;
import com.alibaba.qwen.code.managedagent.persistence.ManagedSessionSkills;
import com.alibaba.qwen.code.managedagent.persistence.ManagedSessionStore;
import com.alibaba.qwen.code.managedagent.persistence.SshWorkspaceStore;
import com.alibaba.qwen.code.managedagent.persistence.WorkspaceExecutionStore;
import com.alibaba.qwen.code.managedagent.session.AgentStateStore;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.net.URI;
import java.util.HashSet;
@Configuration
public class HarnessConfiguration {
@Bean(destroyMethod = "close")
@@ -26,7 +35,43 @@ public class HarnessConfiguration {
AgentStateStore sessions,
WorkspaceExecutionStore workspaceExecution,
ManagedActionStore actions, ManagedAgentDefinitionStore definitions,
ManagedSessionStore resources, SshWorkspaceStore ssh) {
ManagedSessionStore resources, SshWorkspaceStore ssh, HarnessPlacementStore placements,
ManagedSessionSkills skills, ObjectMapper json) {
var pool = properties.getHarness().getPool();
if (pool.isEnabled()) {
if ("kubernetes-workspace".equals(properties.getRuntimeBroker().getProvisioner())) {
throw new IllegalStateException("Harness pool does not admit CSI resident Runtime ownership");
}
HarnessDiscovery discovery;
if ("kubernetes".equals(pool.getDiscovery())) {
if (!pool.getInstances().isEmpty()) {
throw new IllegalStateException("Kubernetes Harness discovery cannot combine with static instances");
}
discovery = new KubernetesHarnessDiscovery(pool, json);
} else if ("static".equals(pool.getDiscovery())) {
if (pool.getInstances().isEmpty() || pool.getInstances().size() > 256) {
throw new IllegalStateException("Static Harness pool requires 1..256 instances");
}
var identities = new HashSet<String>();
var endpoints = pool.getInstances().stream().map(instance -> {
if (!identities.add(instance.getInstanceId())) {
throw new IllegalStateException("Duplicate Harness instance identity");
}
return new HarnessDiscovery.Endpoint(instance.getInstanceId(), URI.create(instance.getBaseUrl()), true);
}).toList();
discovery = () -> new HarnessDiscovery.Snapshot(endpoints, true);
} else {
throw new IllegalStateException("Harness pool discovery must be static or kubernetes");
}
try {
return new HarnessPoolConnector(properties, placements, discovery,
endpoint -> new QwenHostedHarnessConnector(properties, sessions, workspaceExecution,
actions, definitions, resources, ssh), skills);
} catch (RuntimeException error) {
discovery.close();
throw error;
}
}
return new QwenHostedHarnessConnector(properties, sessions, workspaceExecution, actions, definitions, resources, ssh);
}
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/config/ManagedAgentProperties.java
+156
View File
@@ -24,6 +24,7 @@ import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
@@ -604,6 +605,12 @@ public class ManagedAgentProperties implements EnvironmentAware {
}
public static class Harness {
private final HarnessPool pool = new HarnessPool();
public HarnessPool getPool() {
return pool;
}
private boolean enabled;
private boolean workspaceFilesEnabled;
private boolean workspaceShellEnabled;
@@ -752,6 +759,155 @@ public class ManagedAgentProperties implements EnvironmentAware {
}
}
public static class HarnessPool {
private boolean enabled;
private String discovery = "static";
private List<HarnessInstance> instances = new ArrayList<>();
private String keyStoreFile = "";
private String keyStorePasswordFile = "";
private String trustStoreFile = "";
private String trustStorePasswordFile = "";
private String kubernetesApiUrl = "https://kubernetes.default.svc";
private String kubernetesTokenFile = "/var/run/secrets/kubernetes.io/serviceaccount/token";
private String kubernetesCaFile = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt";
private String namespace = "";
private String serviceName = "";
private String serviceUid = "";
private String portName = "https";
public boolean isEnabled() {
return enabled;
}
public void setEnabled(boolean value) {
enabled = value;
}
public String getDiscovery() {
return discovery;
}
public void setDiscovery(String value) {
discovery = value;
}
public List<HarnessInstance> getInstances() {
return instances;
}
public void setInstances(List<HarnessInstance> value) {
instances = value;
}
public String getKeyStoreFile() {
return keyStoreFile;
}
public void setKeyStoreFile(String value) {
keyStoreFile = value;
}
public String getKeyStorePasswordFile() {
return keyStorePasswordFile;
}
public void setKeyStorePasswordFile(String value) {
keyStorePasswordFile = value;
}
public String getTrustStoreFile() {
return trustStoreFile;
}
public void setTrustStoreFile(String value) {
trustStoreFile = value;
}
public String getTrustStorePasswordFile() {
return trustStorePasswordFile;
}
public void setTrustStorePasswordFile(String value) {
trustStorePasswordFile = value;
}
public String getKubernetesApiUrl() {
return kubernetesApiUrl;
}
public void setKubernetesApiUrl(String value) {
kubernetesApiUrl = value;
}
public String getKubernetesTokenFile() {
return kubernetesTokenFile;
}
public void setKubernetesTokenFile(String value) {
kubernetesTokenFile = value;
}
public String getKubernetesCaFile() {
return kubernetesCaFile;
}
public void setKubernetesCaFile(String value) {
kubernetesCaFile = value;
}
public String getNamespace() {
return namespace;
}
public void setNamespace(String value) {
namespace = value;
}
public String getServiceName() {
return serviceName;
}
public void setServiceName(String value) {
serviceName = value;
}
public String getServiceUid() {
return serviceUid;
}
public void setServiceUid(String value) {
serviceUid = value;
}
public String getPortName() {
return portName;
}
public void setPortName(String value) {
portName = value;
}
}
public static class HarnessInstance {
private String instanceId = "";
private String baseUrl = "";
public String getInstanceId() {
return instanceId;
}
public void setInstanceId(String value) {
instanceId = value;
}
public String getBaseUrl() {
return baseUrl;
}
public void setBaseUrl(String value) {
baseUrl = value;
}
}
public static class SessionStore {
private boolean enabled;
private String baseUrl = "";
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/HarnessConnector.java
+22 -1
View File
@@ -1,6 +1,7 @@
package com.alibaba.qwen.code.managedagent.harness;
import com.alibaba.qwen.code.daemon.HarnessRuntimeRecovery;
import com.alibaba.qwen.code.managedagent.model.StoreModels.OperationRecord;
import com.fasterxml.jackson.databind.JsonNode;
import java.util.List;
@@ -16,6 +17,23 @@ public interface HarnessConnector extends AutoCloseable {
default boolean supportsPublicChildAgents() { return false; }
default String closeSession(OperationRecord operation) {
return closeSession(operation.tenantId(), operation.sessionId());
}
default void runLifecycleChildOperation(OperationRecord operation, Map<String, Object> body) {
runChildOperation(operation.tenantId(), operation.sessionId(), body);
}
default Map<String, Object> drainInstance(String instanceId, String operationId) {
throw new UnsupportedOperationException("Harness drain is unavailable");
}
default Map<String, Object> releaseOwnership(String tenantId, String sessionId,
String operationId, long expectedPlacementEpoch) {
throw new UnsupportedOperationException("Harness ownership release is unavailable");
}
default boolean supportsApprovalSelection() { return false; }
default boolean supportsLiveApprovalSelection() { return false; }
@@ -314,7 +332,10 @@ public interface HarnessConnector extends AutoCloseable {
default void close() {
}
record AttachedSession(String tenantId, String sessionId) {
record AttachedSession(String tenantId, String sessionId, String writerIdentity, long placementEpoch) {
public AttachedSession(String tenantId, String sessionId) {
this(tenantId, sessionId, null, 0);
}
}
record Attachment(String bootId, HarnessRuntimeRecovery runtimeRecovery,
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/HarnessDiscovery.java
+21
View File
@@ -0,0 +1,21 @@
package com.alibaba.qwen.code.managedagent.harness;
import java.net.URI;
import java.util.List;
public interface HarnessDiscovery extends AutoCloseable {
Snapshot snapshot();
@Override
default void close() {
}
record Endpoint(String instanceId, URI baseUri, boolean accepting) {
}
record Snapshot(List<Endpoint> endpoints, boolean current) {
public Snapshot {
endpoints = List.copyOf(endpoints);
}
}
}
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/HarnessPoolConnector.java
+676
View File
@@ -0,0 +1,676 @@
package com.alibaba.qwen.code.managedagent.harness;
import com.alibaba.qwen.code.daemon.DaemonTransportException;
import com.alibaba.qwen.code.daemon.HostedHarnessCapabilities;
import com.alibaba.qwen.code.daemon.HostedHarnessClient;
import com.alibaba.qwen.code.daemon.HostedHarnessGenerationException;
import com.alibaba.qwen.code.daemon.HostedHarnessPlacement;
import com.alibaba.qwen.code.daemon.MutationOutcomeUnknownException;
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
import com.alibaba.qwen.code.managedagent.model.StoreModels.OperationRecord;
import com.alibaba.qwen.code.managedagent.persistence.HarnessPlacementStore;
import com.alibaba.qwen.code.managedagent.persistence.HarnessPlacementStore.Placement;
import com.alibaba.qwen.code.managedagent.persistence.ManagedSessionSkills;
import com.alibaba.qwen.code.managedagent.security.WriterCredentialPolicy;
import com.alibaba.qwen.code.managedagent.workspace.WorkspaceToolProfiles;
import com.fasterxml.jackson.databind.JsonNode;
import org.springframework.http.HttpStatus;
import javax.net.ssl.SSLContext;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Function;
/** Sticky routing over independently fenced Hosted process incarnations. */
public class HarnessPoolConnector implements HarnessConnector {
private final ManagedAgentProperties properties;
private final HarnessPlacementStore placements;
private final WriterCredentialPolicy credentials;
private final HarnessDiscovery discovery;
private final Function<HarnessDiscovery.Endpoint, QwenHostedHarnessConnector> connectors;
private final Function<HarnessDiscovery.Endpoint, HostedHarnessClient> clients;
private final ManagedSessionSkills skills;
private final SSLContext tls;
private final Map<Owner, Member> members = new ConcurrentHashMap<>();
private final AtomicInteger next = new AtomicInteger();
private final ScheduledExecutorService refreshes = Executors.newSingleThreadScheduledExecutor(runnable -> {
Thread thread = new Thread(runnable, "managed-harness-pool");
thread.setDaemon(true);
return thread;
});
private volatile List<Member> accepting = List.of();
private volatile Contract contract;
private volatile boolean closed;
private record Owner(String instanceId, String bootId) {
}
private record Contract(Set<String> features, int title, int lifecycle, Set<String> profiles) {
static Contract of(HostedHarnessCapabilities capabilities) {
return new Contract(Set.copyOf(capabilities.getFeatures()), capabilities.getTitleProtocolVersion(),
capabilities.getLifecycleProtocolVersion(), Set.copyOf(capabilities.getLifecycleV2Profiles()));
}
}
private static final class Member {
private final HarnessDiscovery.Endpoint endpoint;
private final HostedHarnessClient client;
private final QwenHostedHarnessConnector connector;
private volatile boolean reachable;
private volatile boolean quiescing;
Member(HarnessDiscovery.Endpoint endpoint, HostedHarnessClient client, QwenHostedHarnessConnector connector) {
this.endpoint = endpoint;
this.client = client;
this.connector = connector;
}
Owner owner() {
return new Owner(endpoint.instanceId(), client.capabilities().getBootId());
}
}
private record Selected(Placement placement, Member member, QwenHostedHarnessConnector.PoolCall call) {
}
public HarnessPoolConnector(ManagedAgentProperties properties, HarnessPlacementStore placements,
HarnessDiscovery discovery, Function<HarnessDiscovery.Endpoint, QwenHostedHarnessConnector> connectors,
ManagedSessionSkills skills) {
this(properties, placements, discovery, connectors, skills, HarnessTls.poolContext(properties.getHarness().getPool()), null);
}
HarnessPoolConnector(ManagedAgentProperties properties, HarnessPlacementStore placements,
HarnessDiscovery discovery, Function<HarnessDiscovery.Endpoint, QwenHostedHarnessConnector> connectors,
ManagedSessionSkills skills, SSLContext tls,
Function<HarnessDiscovery.Endpoint, HostedHarnessClient> clients) {
this.properties = properties;
this.placements = placements;
this.credentials = new WriterCredentialPolicy(properties);
this.discovery = discovery;
this.connectors = connectors;
this.skills = skills;
this.tls = tls;
if (!properties.getSessionStore().isEnabled() || !credentials.isBound()) {
throw new IllegalStateException("Harness pool requires an enabled Session Store and binding key");
}
var settings = properties.getHarness();
if (settings.getToken().isBlank() || !settings.getCapabilityDigest().matches("sha256:[0-9a-f]{64}")) {
throw new IllegalStateException("Harness pool requires a token and capability digest");
}
if ("kubernetes".equals(settings.getPool().getDiscovery()) && tls == null) {
throw new IllegalStateException("Kubernetes Harness pool requires configured mTLS");
}
for (var endpoint : discovery.snapshot().endpoints()) {
validateEndpoint(endpoint, tls);
}
this.clients = clients == null ? this::connect : clients;
if (clients == null) {
refreshes.scheduleWithFixedDelay(() -> {
try {
refresh();
} catch (RuntimeException unavailable) {
accepting = List.of();
}
}, 0, 5, TimeUnit.SECONDS);
}
}
private HostedHarnessClient connect(HarnessDiscovery.Endpoint endpoint) {
validateEndpoint(endpoint, tls);
var settings = properties.getHarness();
return HostedHarnessClient.builder().baseUri(endpoint.baseUri()).sslContext(tls)
.bearerToken(settings.getToken()).capabilityDigest(settings.getCapabilityDigest())
.connectTimeout(settings.getConnectTimeout()).requestTimeout(settings.getRequestTimeout())
.loadTimeout(settings.getLoadTimeout()).heartbeatInterval(settings.getHeartbeatInterval()).build();
}
static void validateEndpoint(HarnessDiscovery.Endpoint endpoint, SSLContext tls) {
var uri = endpoint.baseUri();
if (endpoint.instanceId() == null || !endpoint.instanceId().matches("[A-Za-z0-9._:-]{1,128}")
|| uri == null || uri.getHost() == null || uri.getUserInfo() != null || uri.getQuery() != null
|| uri.getFragment() != null || uri.getPath() != null && !uri.getPath().isEmpty()
&& !"/".equals(uri.getPath())) {
throw new IllegalArgumentException("Invalid Harness pool endpoint");
}
boolean local = Set.of("127.0.0.1", "localhost", "[::1]", "::1").contains(uri.getHost());
if (!("https".equals(uri.getScheme()) || local && "http".equals(uri.getScheme()))
|| !local && tls == null) {
throw new IllegalArgumentException("Remote Harness pool endpoints require configured mTLS");
}
}
synchronized void refresh() {
if (closed) {
return;
}
var snapshot = discovery.snapshot();
if (!snapshot.current()) {
accepting = List.of();
return;
}
List<Member> ready = new ArrayList<>();
for (var endpoint : snapshot.endpoints()) {
if (closed) {
return;
}
Member existing = members.values().stream()
.filter(member -> member.endpoint.instanceId().equals(endpoint.instanceId())
&& member.endpoint.baseUri().equals(endpoint.baseUri()))
.filter(member -> member.reachable).findFirst().orElse(null);
Member member = existing;
HostedHarnessClient created = null;
try {
if (member == null) {
created = clients.apply(endpoint);
if (closed) {
created.close();
return;
}
var capabilities = created.capabilities();
if (!endpoint.instanceId().equals(capabilities.getInstanceId())
|| !capabilities.getFeatures().containsAll(
List.of("harness-pool-owner/1", "harness-instance-lifecycle/1"))) {
throw new IllegalStateException("Harness pool identity or capabilities differ");
}
Contract negotiated = Contract.of(capabilities);
if (contract != null && !contract.equals(negotiated)) {
throw new IllegalStateException("Harness pool capabilities are not homogeneous");
}
var leaf = connectors.apply(endpoint);
leaf.configurePool(endpoint, created, tls, skills);
member = new Member(endpoint, created, leaf);
Owner registeredOwner = member.owner();
var previous = members.putIfAbsent(registeredOwner, member);
if (previous != null) {
leaf.close();
member = previous;
}
created = null;
if (closed) {
members.remove(registeredOwner, member);
member.connector.close();
return;
}
contract = negotiated;
}
var status = member.client.readInstance();
member.quiescing = !endpoint.accepting() || !"ACCEPTING".equals(status.get("state"));
member.reachable = true;
if (!member.quiescing) {
ready.add(member);
}
} catch (RuntimeException unavailable) {
if (member != null) {
member.reachable = false;
}
if (created != null) {
created.close();
}
}
}
ready.sort(Comparator.comparing(member -> member.endpoint.instanceId()));
accepting = List.copyOf(ready);
members.entrySet().removeIf(entry -> {
if (ready.contains(entry.getValue())) {
return false;
}
Member member = entry.getValue();
boolean owns = placements.hasUnreleasedOwnership(member.owner().instanceId(), member.owner().bootId());
if (!owns) {
member.connector.close();
return true;
}
return false;
});
}
private static boolean matches(Placement placement, Member member) {
return placement != null && placement.instanceId().equals(member.endpoint.instanceId())
&& placement.bootId().equals(member.client.capabilities().getBootId());
}
private Selected select(String tenant, String session, boolean mayAcquire) {
return select(tenant, session, mayAcquire, null);
}
private Selected select(String tenant, String session, boolean mayAcquire, OperationRecord operation) {
if (closed) {
throw unavailable();
}
Placement placement = placements.get(tenant, session);
Member member = placement == null ? null : members.get(new Owner(placement.instanceId(), placement.bootId()));
boolean quiescingOwner = member != null && member.reachable && member.quiescing && "ACTIVE".equals(placement.state());
if (mayAcquire && (placement == null || member == null || !member.reachable
|| quiescingOwner || !"ACTIVE".equals(placement.state()))) {
List<Member> candidates = accepting.stream()
.filter(value -> value.reachable && !value.quiescing).toList();
if (candidates.isEmpty()) {
if (member == null || !member.reachable) {
throw unavailable();
}
} else {
Member candidate = candidates.get(Math.floorMod(next.getAndIncrement(), candidates.size()));
try {
if (operation == null) {
placement = placement == null
? placements.assign(tenant, session, candidate.endpoint.instanceId(), candidate.owner().bootId())
: placements.takeover(tenant, session, placement, candidate.endpoint.instanceId(), candidate.owner().bootId());
} else {
placement = placement == null
? placements.assignForLifecycle(operation, candidate.endpoint.instanceId(), candidate.owner().bootId())
: placements.takeoverForLifecycle(operation, placement, candidate.endpoint.instanceId(), candidate.owner().bootId());
}
} catch (ManagedAgentException refusal) {
if (!quiescingOwner || !member.reachable || refusal.getStatus() != HttpStatus.CONFLICT
|| !"harness_owner_still_live".equals(refusal.getCode())) {
throw refusal;
}
// Keep completion RPCs on the live owner while its SQL fence
// refuses a handoff; its process gate still rejects new work.
}
member = members.get(new Owner(placement.instanceId(), placement.bootId()));
}
}
if (placement == null || member == null) {
if (mayAcquire) {
throw unavailable();
}
return null;
}
String workspace = placement.workspaceId() == null
? properties.getSessionStore().getWorkspaceId() : placement.workspaceId();
var descriptor = new HostedHarnessPlacement(placement.instanceId(), placement.bootId(), placement.epoch());
String token = credentials.issue(tenant, workspace, session, placement.instanceId(), placement.bootId(), placement.epoch());
return new Selected(placement, member, new QwenHostedHarnessConnector.PoolCall(tenant, session, descriptor, token));
}
private static ManagedAgentException unavailable() {
return new ManagedAgentException(HttpStatus.SERVICE_UNAVAILABLE, "harness_pool_unavailable", "No compatible Harness owner is available");
}
private <T> T call(String tenant, String session, Function<QwenHostedHarnessConnector, T> action) {
return call(select(tenant, session, true), action);
}
private <T> T lifecycleCall(OperationRecord operation, Function<QwenHostedHarnessConnector, T> action) {
return call(select(operation.tenantId(), operation.sessionId(), true, operation), action);
}
private <T> T call(Selected selected, Function<QwenHostedHarnessConnector, T> action) {
try {
return selected.member().connector.withPlacement(selected.call(), () -> action.apply(selected.member().connector));
} catch (DaemonTransportException | HostedHarnessGenerationException | MutationOutcomeUnknownException error) {
selected.member().reachable = false;
throw error;
}
}
private <T> T observe(String tenant, String session, T absent, Function<QwenHostedHarnessConnector, T> action) {
Selected selected = select(tenant, session, false);
return selected == null ? absent : selected.member().connector.withPlacement(selected.call(),
() -> action.apply(selected.member().connector));
}
@Override
public boolean isAvailable() {
return !closed;
}
private boolean feature(String name) {
return contract != null && contract.features().contains(name);
}
@Override
public boolean supportsApprovalSelection() {
return feature("approval-selection/1");
}
@Override
public boolean supportsLiveApprovalSelection() {
return feature("live-approval-selection/1") && supportsApprovalSelection();
}
@Override
public boolean supportsAgentDefinitionApprovalSelection() {
return feature("agent-definition-approval-selection/1") && supportsApprovalSelection();
}
@Override
public boolean supportsModelSelection() {
return feature("model-selection/1");
}
@Override
public boolean supportsContextCompression() {
return feature("context-compression/1");
}
@Override
public boolean supportsPlanMode() {
return feature("plan-mode/1");
}
@Override
public boolean supportsInputAttachments() {
return feature("input-attachments/1");
}
@Override
public boolean supportsSessionSkills() {
return feature("session-skills/2");
}
@Override
public boolean isWorkspaceFilesAvailable() {
return properties.getHarness().isWorkspaceFilesEnabled();
}
@Override
public boolean supportsFencedTitles() {
return contract != null && contract.title() == 1;
}
@Override
public boolean supportsLifecycle() {
return supportsLifecycle(WorkspaceToolProfiles.FILES);
}
@Override
public boolean supportsLifecycle(String profile) {
return contract != null && (WorkspaceToolProfiles.FILES.equals(profile)
? isWorkspaceFilesAvailable() && contract.lifecycle() == 1
: WorkspaceToolProfiles.lifecycleProtocol(profile) == 2 && contract.profiles().contains(profile));
}
@Override
public List<AttachedSession> attachedSessions() {
List<AttachedSession> attached = new ArrayList<>();
for (Member member : members.values()) {
for (var session : member.connector.attachedSessions()) {
Placement placement = placements.get(session.tenantId(), session.sessionId());
if (matches(placement, member) && placement.epoch() == session.placementEpoch()) {
attached.add(session);
}
}
}
return List.copyOf(attached);
}
@Override
public Map<String, Object> drainInstance(String instance, String operationId) {
Member member = members.values().stream().filter(value -> value.endpoint.instanceId().equals(instance)
&& value.reachable).findFirst().orElseThrow(HarnessPoolConnector::unavailable);
var receipt = member.client.drainInstance(operationId);
member.quiescing = true;
accepting = accepting.stream().filter(value -> value != member).toList();
return receipt;
}
@Override
public ContextCompressionSubmission prepareContextCompression(String tenant, String session, Map<String, Object> body) {
Selected selected = select(tenant, session, true);
var submission = selected.member().connector.withPlacement(selected.call(),
() -> selected.member().connector.prepareContextCompression(tenant, session, body));
return () -> selected.member().connector.withPlacement(selected.call(), submission::submit);
}
@Override
public void close() {
closed = true;
refreshes.shutdownNow();
discovery.close();
members.values().forEach(member -> member.connector.close());
members.clear();
accepting = List.of();
}
@Override
public void recoverForWake(String tenant, String session) {
call(tenant, session, leaf -> {
leaf.recoverForWake(tenant, session);
return null;
});
}
@Override
public Attachment createOrLoad(String tenant, String session, boolean existing) {
return call(tenant, session, leaf -> leaf.createOrLoad(tenant, session, existing));
}
@Override
public Attachment createOrLoad(String tenant, String session, boolean existing, boolean passive) {
return call(tenant, session, leaf -> leaf.createOrLoad(tenant, session, existing, passive));
}
@Override
public Attachment recoverManagedRuntime(String tenant, String session, boolean cancellation) {
return call(tenant, session, leaf -> leaf.recoverManagedRuntime(tenant, session, cancellation));
}
@Override
public Attachment recoverManagedRuntime(String tenant, String session, boolean cancellation, String prompt) {
return call(tenant, session, leaf -> leaf.recoverManagedRuntime(tenant, session, cancellation, prompt));
}
@Override
public Attachment recoverManagedCancellation(String tenant, String session) {
return call(tenant, session, leaf -> leaf.recoverManagedCancellation(tenant, session));
}
@Override
public Attachment recoverManagedCancellation(String tenant, String session, String prompt) {
return call(tenant, session, leaf -> leaf.recoverManagedCancellation(tenant, session, prompt));
}
@Override
public Attachment recoverManagedFailureCancellation(String tenant, String session, String turn, String owner) {
return call(tenant, session, leaf -> leaf.recoverManagedFailureCancellation(tenant, session, turn, owner));
}
@Override
public Attachment recoverManagedFailureCancellation(String tenant, String session, String turn, String owner, String prompt) {
return call(tenant, session, leaf -> leaf.recoverManagedFailureCancellation(tenant, session, turn, owner, prompt));
}
@Override
public Admission submit(String tenant, String session, String prompt, List<Map<String, Object>> input, String digest) {
return call(tenant, session, leaf -> leaf.submit(tenant, session, prompt, input, digest));
}
@Override
public Admission continueManagedRuntime(String tenant, String session, String prompt, String checkpoint, String activation) {
return call(tenant, session, leaf -> leaf.continueManagedRuntime(tenant, session, prompt, checkpoint, activation));
}
@Override
public Admission cancelManagedRuntime(String tenant, String session, String prompt, String checkpoint, String activation) {
return call(tenant, session, leaf -> leaf.cancelManagedRuntime(tenant, session, prompt, checkpoint, activation));
}
@Override
public SourceStream stream(String tenant, String session, long last, String epoch) {
return call(tenant, session, leaf -> leaf.stream(tenant, session, last, epoch));
}
@Override
public void resolveAction(String tenant, String session, String action, JsonNode response) {
call(tenant, session, leaf -> {
leaf.resolveAction(tenant, session, action, response);
return null;
});
}
@Override
public void cancel(String tenant, String session) {
call(tenant, session, leaf -> {
leaf.cancel(tenant, session);
return null;
});
}
@Override
public void cancelOriginalTurn(String tenant, String session, String prompt) {
call(tenant, session, leaf -> {
leaf.cancelOriginalTurn(tenant, session, prompt);
return null;
});
}
@Override
public Map<String, Object> runAutomationOperation(String tenant, String session, Map<String, Object> body) {
return call(tenant, session, leaf -> leaf.runAutomationOperation(tenant, session, body));
}
@Override
public void runChildOperation(String tenant, String session, Map<String, Object> body) {
call(tenant, session, leaf -> {
leaf.runChildOperation(tenant, session, body);
return null;
});
}
@Override
public Map<String, Object> runMessageOperation(String tenant, String session, Map<String, Object> body) {
return call(tenant, session, leaf -> leaf.runMessageOperation(tenant, session, body));
}
@Override
public Map<String, Object> runChannelOperation(String tenant, String session, Map<String, Object> body) {
return call(tenant, session, leaf -> leaf.runChannelOperation(tenant, session, body));
}
@Override
public void rename(String tenant, String session, String title) {
call(tenant, session, leaf -> {
leaf.rename(tenant, session, title);
return null;
});
}
@Override
public void rename(String tenant, String session, String title, long revision) {
call(tenant, session, leaf -> {
leaf.rename(tenant, session, title, revision);
return null;
});
}
@Override
public String closeSession(OperationRecord operation) {
return lifecycleCall(operation, leaf -> leaf.closeSession(operation.tenantId(), operation.sessionId()));
}
@Override
public void runLifecycleChildOperation(OperationRecord operation, Map<String, Object> body) {
lifecycleCall(operation, leaf -> {
leaf.runLifecycleChildOperation(operation, body);
return null;
});
}
@Override
public String closeSession(String tenant, String session) {
return call(tenant, session, leaf -> leaf.closeSession(tenant, session));
}
@Override
public boolean activateForWake(String tenant, String session) {
return call(tenant, session, leaf -> leaf.activateForWake(tenant, session));
}
@Override
public Map<String, Object> runContextCompression(String tenant, String session, Map<String, Object> body) {
return call(tenant, session, leaf -> leaf.runContextCompression(tenant, session, body));
}
@Override
public Map<String, Object> readContextCompression(String tenant, String session, String operation) {
return call(tenant, session, leaf -> leaf.readContextCompression(tenant, session, operation));
}
@Override
public Map<String, Object> releaseOwnership(String tenant, String session,
String operation, long expectedPlacementEpoch) {
Map<String, Object> committed = placements.releaseReceipt(tenant, session, expectedPlacementEpoch, operation);
if (committed != null) {
return committed;
}
Selected selected = select(tenant, session, false);
if (selected == null || selected.placement().epoch() != expectedPlacementEpoch) {
throw new IllegalStateException("Original Harness placement is unavailable");
}
return selected.member().connector.withPlacement(selected.call(),
() -> selected.member().connector.releaseOwnership(
tenant, session, operation, expectedPlacementEpoch));
}
@Override
public boolean recheckRecoveryCleanup(String tenant, String session, String prompt) {
return call(tenant, session, leaf -> leaf.recheckRecoveryCleanup(tenant, session, prompt));
}
@Override
public JsonNode settleLifecycle(OperationRecord operation) {
return lifecycleCall(operation, leaf -> leaf.settleLifecycle(operation));
}
@Override
public void prepareLifecycleResources(OperationRecord operation) {
lifecycleCall(operation, leaf -> {
leaf.prepareLifecycleResources(operation);
return null;
});
}
@Override
public void settleLifecycleResources(OperationRecord operation) {
lifecycleCall(operation, leaf -> {
leaf.settleLifecycleResources(operation);
return null;
});
}
@Override
public void detachLifecycle(OperationRecord operation) {
lifecycleCall(operation, leaf -> {
leaf.detachLifecycle(operation);
return null;
});
}
@Override
public void discardManagedRecovery(String tenant, String session) {
observe(tenant, session, null, leaf -> {
leaf.discardManagedRecovery(tenant, session);
return null;
});
}
@Override
public RecoveryStatus inspectRecovery(String tenant, String session, String prompt) {
return observe(tenant, session, null, leaf -> leaf.inspectRecovery(tenant, session, prompt));
}
@Override
public Attachment cachedRecoveryAttachment(String tenant, String session) {
return observe(tenant, session, null, leaf -> leaf.cachedRecoveryAttachment(tenant, session));
}
@Override
public SourceStream observeStream(String tenant, String session, String boot, long last, String epoch) {
SourceStream stream = observe(tenant, session, null, leaf -> leaf.observeStream(tenant, session, boot, last, epoch));
if (stream == null) {
throw unavailable();
}
return stream;
}
}
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/HarnessTls.java
+91
View File
@@ -0,0 +1,91 @@
package com.alibaba.qwen.code.managedagent.harness;
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.GeneralSecurityException;
import java.security.KeyStore;
import java.security.cert.CertificateFactory;
import java.util.Arrays;
public final class HarnessTls {
private HarnessTls() {
}
public static SSLContext poolContext(ManagedAgentProperties.HarnessPool properties) {
if (properties.getKeyStoreFile().isBlank() && properties.getTrustStoreFile().isBlank()) {
return null;
}
if (properties.getKeyStoreFile().isBlank() || properties.getTrustStoreFile().isBlank()) {
throw new IllegalArgumentException("Harness mTLS requires both PKCS12 key and trust stores");
}
char[] keyPassword = password(properties.getKeyStorePasswordFile());
char[] trustPassword = password(properties.getTrustStorePasswordFile());
try {
KeyStore keys = load(properties.getKeyStoreFile(), keyPassword);
KeyStore trust = load(properties.getTrustStoreFile(), trustPassword);
var keyManagers = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
keyManagers.init(keys, keyPassword);
var trustManagers = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagers.init(trust);
SSLContext context = SSLContext.getInstance("TLS");
context.init(keyManagers.getKeyManagers(), trustManagers.getTrustManagers(), null);
return context;
} catch (IOException | GeneralSecurityException error) {
throw new IllegalArgumentException("Harness mTLS material could not be loaded");
} finally {
Arrays.fill(keyPassword, '\0');
Arrays.fill(trustPassword, '\0');
}
}
static SSLContext kubernetesContext(Path caFile) {
try (var input = Files.newInputStream(caFile)) {
var certificates = CertificateFactory.getInstance("X.509").generateCertificates(input);
if (certificates.isEmpty()) {
throw new GeneralSecurityException("Empty Kubernetes CA");
}
KeyStore trust = KeyStore.getInstance(KeyStore.getDefaultType());
trust.load(null, null);
int index = 0;
for (var certificate : certificates) {
trust.setCertificateEntry("ca-" + index++, certificate);
}
var managers = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
managers.init(trust);
SSLContext context = SSLContext.getInstance("TLS");
context.init(null, managers.getTrustManagers(), null);
return context;
} catch (IOException | GeneralSecurityException error) {
throw new IllegalArgumentException("Kubernetes discovery CA could not be loaded");
}
}
private static KeyStore load(String file, char[] password) throws IOException, GeneralSecurityException {
KeyStore store = KeyStore.getInstance("PKCS12");
try (var input = Files.newInputStream(Path.of(file))) {
store.load(input, password);
}
return store;
}
private static char[] password(String file) {
if (file.isBlank()) {
return new char[0];
}
try {
if (Files.size(Path.of(file)) > 4096) {
throw new IOException("Oversized password file");
}
return Files.readString(Path.of(file)).stripTrailing().toCharArray();
} catch (IOException error) {
throw new IllegalArgumentException("Harness TLS password file could not be loaded");
}
}
}
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/KubernetesHarnessDiscovery.java
+289
View File
@@ -0,0 +1,289 @@
package com.alibaba.qwen.code.managedagent.harness;
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
import com.alibaba.qwen.code.runtimebroker.transport.HttpRuntimeTransport;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
/** A bounded namespace-scoped list/watch; uncertainty never creates placement candidates. */
public final class KubernetesHarnessDiscovery implements HarnessDiscovery {
private static final int RESPONSE_LIMIT = 2 * 1024 * 1024;
private final ManagedAgentProperties.HarnessPool properties;
private final ObjectMapper json;
private final URI origin;
private final HttpClient client;
private final Path tokenFile;
private final ExecutorService watcher = Executors.newSingleThreadExecutor(runnable -> {
Thread thread = new Thread(runnable, "managed-harness-discovery");
thread.setDaemon(true);
return thread;
});
private final AtomicReference<CompletableFuture<?>> request = new AtomicReference<>();
private volatile Snapshot snapshot = new Snapshot(List.of(), false);
private volatile boolean closed;
public KubernetesHarnessDiscovery(ManagedAgentProperties.HarnessPool properties, ObjectMapper json) {
this(properties, json, secureOrigin(properties.getKubernetesApiUrl()),
HttpClient.newBuilder().sslContext(HarnessTls.kubernetesContext(Path.of(properties.getKubernetesCaFile())))
.connectTimeout(Duration.ofSeconds(5)).followRedirects(HttpClient.Redirect.NEVER).build());
}
KubernetesHarnessDiscovery(ManagedAgentProperties.HarnessPool properties, ObjectMapper json, URI origin, HttpClient client) {
label(properties.getNamespace());
label(properties.getServiceName());
label(properties.getPortName());
if (properties.getServiceUid().isBlank() || properties.getServiceUid().length() > 256
|| client.followRedirects() != HttpClient.Redirect.NEVER) {
throw new IllegalArgumentException("Kubernetes Harness discovery requires pinned Service UID and non-redirecting TLS");
}
this.properties = properties;
this.json = json;
this.origin = origin;
this.client = client;
this.tokenFile = Path.of(properties.getKubernetesTokenFile());
watcher.execute(this::watch);
}
private static URI secureOrigin(String value) {
URI uri = URI.create(value);
if (!"https".equals(uri.getScheme()) || uri.getHost() == null || uri.getUserInfo() != null
|| uri.getQuery() != null || uri.getFragment() != null
|| uri.getPath() != null && !uri.getPath().isEmpty() && !"/".equals(uri.getPath())) {
throw new IllegalArgumentException("Kubernetes discovery requires an HTTPS origin");
}
return uri;
}
private static void label(String value) {
if (value == null || !value.matches("[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?")) {
throw new IllegalArgumentException("Invalid Kubernetes discovery name");
}
}
@Override
public Snapshot snapshot() {
return snapshot;
}
private void watch() {
String version = null;
Map<String, JsonNode> slices = new HashMap<>();
long backoff = 500;
while (!closed) {
try {
String path = "/apis/discovery.k8s.io/v1/namespaces/" + properties.getNamespace() + "/endpointslices";
String selector = "labelSelector=" + encode("kubernetes.io/service-name=" + properties.getServiceName());
if (version == null) {
JsonNode service = exchange("/api/v1/namespaces/" + properties.getNamespace()
+ "/services/" + properties.getServiceName());
if (!"Service".equals(service.path("kind").asText())
|| !properties.getServiceUid().equals(service.path("metadata").path("uid").asText())
|| !properties.getNamespace().equals(service.path("metadata").path("namespace").asText())
|| !properties.getServiceName().equals(service.path("metadata").path("name").asText())) {
throw new IOException("Kubernetes Service identity differs");
}
JsonNode listed = exchange(path + "?" + selector);
if (!"EndpointSliceList".equals(listed.path("kind").asText()) || !listed.path("items").isArray()
|| listed.path("items").size() > 256 || !listed.path("metadata").path("continue").asText().isEmpty()) {
throw new IOException("Invalid or oversized EndpointSlice list");
}
Map<String, JsonNode> replacement = new HashMap<>();
for (JsonNode slice : listed.path("items")) {
replacement.put(identity(slice.path("metadata").path("uid")), slice);
}
version = identity(listed.path("metadata").path("resourceVersion"));
slices = replacement;
snapshot = new Snapshot(endpoints(properties, slices.values()), true);
}
String body = exchangeBytes(path + "?" + selector + "&watch=true&allowWatchBookmarks=true&timeoutSeconds=5&resourceVersion="
+ encode(version));
for (String line : body.split("\\n")) {
if (line.isBlank()) {
continue;
}
JsonNode event = json.readTree(line);
JsonNode object = event.path("object");
String type = event.path("type").asText();
if ("ERROR".equals(type)) {
throw new IOException("Kubernetes watch requires a new list");
}
String nextVersion = identity(object.path("metadata").path("resourceVersion"));
if ("BOOKMARK".equals(type)) {
version = nextVersion;
continue;
}
String uid = identity(object.path("metadata").path("uid"));
switch (type) {
case "ADDED", "MODIFIED" -> slices.put(uid, object);
case "DELETED" -> slices.remove(uid);
default -> throw new IOException("Unknown Kubernetes watch event");
}
if (slices.size() > 256) {
throw new IOException("Too many EndpointSlices");
}
version = nextVersion;
snapshot = new Snapshot(endpoints(properties, slices.values()), true);
}
backoff = 500;
} catch (Exception unavailable) {
if (unavailable instanceof InterruptedException) {
Thread.currentThread().interrupt();
return;
}
snapshot = new Snapshot(snapshot.endpoints(), false);
version = null;
try {
Thread.sleep(backoff);
} catch (InterruptedException stopped) {
Thread.currentThread().interrupt();
return;
}
backoff = Math.min(10_000, backoff * 2);
}
}
}
private JsonNode exchange(String path) throws Exception {
return json.readTree(exchangeBytes(path));
}
private String exchangeBytes(String path) throws Exception {
if (Files.size(tokenFile) > 64 * 1024) {
throw new IOException("Invalid discovery token file");
}
String token = Files.readString(tokenFile).strip();
if (token.isEmpty() || token.indexOf('\n') >= 0 || token.indexOf('\r') >= 0) {
throw new IOException("Invalid discovery token");
}
HttpRequest sent = HttpRequest.newBuilder(origin.resolve(path)).timeout(Duration.ofSeconds(15))
.header("Authorization", "Bearer " + token).header("Accept", "application/json").GET().build();
var pending = client.sendAsync(sent, ignored -> new HttpRuntimeTransport.BoundedBodySubscriber(RESPONSE_LIMIT));
request.set(pending);
try {
var response = pending.get(20, TimeUnit.SECONDS);
if (response.statusCode() != 200 || response.body().overflow()) {
throw new IOException("Kubernetes discovery failed or exceeded its bound");
}
return new String(response.body().bytes(), StandardCharsets.UTF_8);
} finally {
request.compareAndSet(pending, null);
pending.cancel(true);
}
}
static List<Endpoint> endpoints(ManagedAgentProperties.HarnessPool properties, Iterable<JsonNode> slices) throws IOException {
Map<String, Endpoint> result = new LinkedHashMap<>();
for (JsonNode slice : slices) {
JsonNode metadata = slice.path("metadata");
if (slice.has("kind") && !"EndpointSlice".equals(slice.path("kind").asText())
|| !properties.getNamespace().equals(metadata.path("namespace").asText())
|| !properties.getServiceName().equals(metadata.path("labels").path("kubernetes.io/service-name").asText())) {
throw new IOException("EndpointSlice scope differs");
}
boolean owned = false;
for (JsonNode owner : metadata.path("ownerReferences")) {
owned |= "Service".equals(owner.path("kind").asText()) && "v1".equals(owner.path("apiVersion").asText())
&& properties.getServiceName().equals(owner.path("name").asText())
&& properties.getServiceUid().equals(owner.path("uid").asText())
&& owner.path("controller").asBoolean();
}
if (!owned) {
throw new IOException("EndpointSlice owner differs");
}
int port = 0;
for (JsonNode candidate : slice.path("ports")) {
if (properties.getPortName().equals(candidate.path("name").asText())
&& "TCP".equals(candidate.path("protocol").asText("TCP"))) {
if (port != 0 || !candidate.path("port").canConvertToInt()) {
throw new IOException("Ambiguous Harness port");
}
port = candidate.path("port").intValue();
}
}
if (port < 1 || port > 65535) {
throw new IOException("Harness port is absent");
}
for (JsonNode endpoint : slice.path("endpoints")) {
JsonNode target = endpoint.path("targetRef");
String targetVersion = target.path("apiVersion").asText();
if (!"Pod".equals(target.path("kind").asText()) || !targetVersion.isEmpty() && !"v1".equals(targetVersion)
|| !properties.getNamespace().equals(target.path("namespace").asText())
|| target.path("name").asText().isEmpty()) {
throw new IOException("Harness Pod target identity is absent");
}
String uid = identity(target.path("uid"));
boolean accepting = endpoint.path("conditions").path("ready").asBoolean(false)
&& !endpoint.path("conditions").path("terminating").asBoolean(false);
for (JsonNode address : endpoint.path("addresses")) {
String host = address.asText();
String addressType = slice.path("addressType").asText();
if (!("IPv4".equals(addressType) && host.matches("[0-9.]{7,15}")
|| "IPv6".equals(addressType) && host.matches("[0-9a-fA-F:]{2,39}"))) {
throw new IOException("Harness endpoint address is not a literal IP");
}
URI uri = URI.create("https://" + (host.contains(":") ? "[" + host + "]" : host) + ":" + port);
Endpoint candidate = new Endpoint(uid, uri, accepting);
Endpoint previous = result.get(uid);
if (previous == null) {
result.put(uid, candidate);
} else {
if (previous.baseUri().getPort() != port) {
throw new IOException("Conflicting Harness port for one Pod");
}
URI preferred = candidate.baseUri().toString().compareTo(previous.baseUri().toString()) < 0
? candidate.baseUri() : previous.baseUri();
result.put(uid, new Endpoint(uid, preferred, previous.accepting() && accepting));
}
}
if (result.size() > 256) {
throw new IOException("Too many Harness endpoints");
}
}
}
List<Endpoint> ordered = new ArrayList<>(result.values());
ordered.sort(Comparator.comparing(Endpoint::instanceId));
return List.copyOf(ordered);
}
private static String identity(JsonNode node) throws IOException {
if (!node.isTextual() || node.textValue().isEmpty() || node.textValue().length() > 256) {
throw new IOException("Kubernetes identity is missing or oversized");
}
return node.textValue();
}
private static String encode(String value) {
return URLEncoder.encode(value, StandardCharsets.UTF_8);
}
@Override
public void close() {
closed = true;
var pending = request.getAndSet(null);
if (pending != null) {
pending.cancel(true);
}
watcher.shutdownNow();
}
}
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/harness/QwenHostedHarnessConnector.java
+169 -52
View File
@@ -10,6 +10,7 @@ import com.alibaba.qwen.code.daemon.HarnessEventStream;
import com.alibaba.qwen.code.daemon.HarnessSessionRef;
import com.alibaba.qwen.code.daemon.HostedHarnessClient;
import com.alibaba.qwen.code.daemon.HostedHarnessGenerationException;
import com.alibaba.qwen.code.daemon.HostedHarnessPlacement;
import com.alibaba.qwen.code.daemon.LoadHarnessSession;
import com.alibaba.qwen.code.daemon.ManagedSessionStoreConnection;
import com.alibaba.qwen.code.daemon.PromptReceipt;
@@ -33,6 +34,8 @@ import com.fasterxml.jackson.databind.JsonNode;
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.lang3.StringUtils;
import javax.net.ssl.SSLContext;
import java.net.URI;
import java.time.Duration;
import java.util.LinkedHashMap;
@@ -42,6 +45,7 @@ import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Supplier;
public class QwenHostedHarnessConnector implements HarnessConnector {
private final ManagedAgentProperties settings;
@@ -65,6 +69,60 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
private final boolean childWorkspaces;
private final boolean childSnapshots;
private volatile HostedHarnessClient client;
private HarnessDiscovery.Endpoint poolEndpoint;
private String poolBootId;
private SSLContext sslContext;
private final ThreadLocal<PoolCall> poolCall = new ThreadLocal<>();
record PoolCall(String tenantId, String sessionId, HostedHarnessPlacement placement, String writerToken) {
}
void configurePool(HarnessDiscovery.Endpoint endpoint, HostedHarnessClient negotiated,
SSLContext tls, ManagedSessionSkills skills) {
this.poolEndpoint = endpoint;
this.poolBootId = negotiated.capabilities().getBootId();
this.sslContext = tls;
this.sessionSkills = skills;
this.client = negotiated;
}
<T> T withPlacement(PoolCall call, Supplier<T> action) {
PoolCall previous = poolCall.get();
poolCall.set(call);
try {
return action.get();
} finally {
if (previous == null) {
poolCall.remove();
} else {
poolCall.set(previous);
}
}
}
private PoolCall placementCall(String tenantId, String sessionId) {
if (poolEndpoint == null) {
return null;
}
PoolCall call = poolCall.get();
if (call == null || !tenantId.equals(call.tenantId()) || !sessionId.equals(call.sessionId())
|| !poolEndpoint.instanceId().equals(call.placement().getInstanceId())
|| !poolBootId.equals(call.placement().getBootId())) {
throw new IllegalStateException("Harness operation has no matching placement");
}
return call;
}
private HostedHarnessPlacement placementDescriptor(SessionRecord session) {
PoolCall call = placementCall(session.tenantId(), session.sessionId());
return call == null ? null : call.placement();
}
private AttachmentKey attachmentKey(String tenantId, String sessionId) {
PoolCall call = placementCall(tenantId, sessionId);
return new AttachmentKey(tenantId, sessionId, call == null ? 0 : call.placement().getPlacementEpoch());
}
private volatile boolean closed;
private final ReentrantLock clientLock = new ReentrantLock();
// Sessions whose takeover load reported parked Runtime work that no
@@ -186,6 +244,24 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
this.properties.getApprovalMode());
}
@Override
public Map<String, Object> releaseOwnership(String tenantId, String sessionId,
String operationId, long expectedPlacementEpoch) {
PoolCall call = placementCall(tenantId, sessionId);
if (call == null || call.placement().getPlacementEpoch() != expectedPlacementEpoch) {
throw new IllegalStateException("Ownership release requires pool placement");
}
AttachmentKey key = attachmentKey(tenantId, sessionId);
HarnessSessionRef ref = attachments.get(key);
if (ref == null) {
throw new IllegalStateException("Original Harness attachment is unavailable");
}
var receipt = client().releaseOwnership(ref, operationId, call.placement().getPlacementEpoch());
attachments.remove(key, ref);
pendingRecovery.remove(key);
return receipt;
}
@Override
public boolean isAvailable() {
return true;
@@ -239,23 +315,8 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
if (operation.lifecycleProtocolVersion() != 2) {
requireShellApproval(session);
}
AttachmentKey key = new AttachmentKey(operation.tenantId(), operation.sessionId());
HarnessSessionRef ref = attachments.get(key);
if (ref == null) {
ref = client().loadSession(new LoadHarnessSession(session.sessionId(), managedSessionStore(session),
false, toolProfile(session), false, false,
WorkspaceToolProfiles.isPublicShell(toolProfile(session)) && !publicChildLaunchAvailable(session))
.withAgentDefinition(agentDefinition(session, false))
.withSessionSkillsRef(sessionSkillsRef(session, false))
.withMcpServers(mcpServers(session))
.withHookCatalog(hookCatalog(session))
.withPublicChildAgentsVersion(session.publicChildAgentsVersion())
.withPublicChildModelRoute(publicChildModelRoute(session))
.withForegroundOnly(foregroundOnly(session))
.withChildWorkspaces(childWorkspaces && session.publicChildAgentsVersion() == 0 && !foregroundOnly(session), childSnapshots && session.publicChildAgentsVersion() == 0 && !foregroundOnly(session))
.forLifecycle(operation.operationId(), operation.claimGeneration(), operation.lifecycleProtocolVersion() == 2 ? 2 : 1));
attachments.put(key, ref);
}
AttachmentKey key = attachmentKey(operation.tenantId(), operation.sessionId());
HarnessSessionRef ref = lifecycleAttachment(operation, session);
var authority = Map.<String, Object>of("operationId", operation.operationId(), "claimGeneration", operation.claimGeneration());
var request = new LinkedHashMap<String, Object>(Map.<String, Object>of("sessionKey", Map.of("tenantId", session.tenantId(),
"workspaceId", session.workspace().getWorkspaceId(), "sessionId", session.sessionId()),
@@ -279,6 +340,44 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
}
}
private HarnessSessionRef lifecycleAttachment(
com.alibaba.qwen.code.managedagent.model.StoreModels.OperationRecord operation, SessionRecord session) {
AttachmentKey key = attachmentKey(operation.tenantId(), operation.sessionId());
HarnessSessionRef ref = attachments.get(key);
if (ref == null) {
ref = client().loadSession(new LoadHarnessSession(session.sessionId(), managedSessionStore(session),
false, toolProfile(session), false, false,
WorkspaceToolProfiles.isPublicShell(toolProfile(session)) && !publicChildLaunchAvailable(session))
.withHarnessPlacement(placementDescriptor(session))
.withAgentDefinition(agentDefinition(session, false))
.withSessionSkillsRef(sessionSkillsRef(session, false))
.withMcpServers(mcpServers(session))
.withHookCatalog(hookCatalog(session))
.withPublicChildAgentsVersion(session.publicChildAgentsVersion())
.withPublicChildModelRoute(publicChildModelRoute(session))
.withForegroundOnly(foregroundOnly(session))
.withChildWorkspaces(childWorkspaces && session.publicChildAgentsVersion() == 0 && !foregroundOnly(session), childSnapshots && session.publicChildAgentsVersion() == 0 && !foregroundOnly(session))
.forLifecycle(operation.operationId(), operation.claimGeneration(), operation.lifecycleProtocolVersion() == 2 ? 2 : 1));
attachments.put(key, ref);
}
return ref;
}
@Override
public void runLifecycleChildOperation(com.alibaba.qwen.code.managedagent.model.StoreModels.OperationRecord operation,
Map<String, Object> body) {
try {
SessionRecord session = sessions.requireSession(operation.tenantId(), operation.sessionId());
if (operation.lifecycleProtocolVersion() != 2) {
requireShellApproval(session);
}
client().runChildOperation(lifecycleAttachment(operation, session), body);
} catch (HostedHarnessGenerationException error) {
adoptGeneration(error);
throw error;
}
}
@Override
public void prepareLifecycleResources(com.alibaba.qwen.code.managedagent.model.StoreModels.OperationRecord operation) {
runResourcePhase(operation, "prepare_resources");
@@ -309,7 +408,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
}
private void doDetachLifecycle(com.alibaba.qwen.code.managedagent.model.StoreModels.OperationRecord operation) {
AttachmentKey key = new AttachmentKey(operation.tenantId(), operation.sessionId());
AttachmentKey key = attachmentKey(operation.tenantId(), operation.sessionId());
HarnessSessionRef ref = attachments.get(key);
try {
var authority = Map.<String, Object>of("operationId", operation.operationId(), "claimGeneration", operation.claimGeneration());
@@ -385,7 +484,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
}
}
requireApprovalSelectionSupport(session);
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
if (!mcpManagementOnly && !loadExisting && !attachments.containsKey(key)
&& mcpManagementAttachments.containsKey(key)) {
return doCreateOrLoad(tenantId, sessionId, true, true, actionResponse, false);
@@ -501,7 +600,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
@Override
public boolean supportsPublicChildAgents() {
try {
return client().capabilities().getFeatures().contains("public-child-agents/1")
return poolEndpoint == null && client().capabilities().getFeatures().contains("public-child-agents/1")
&& supportsApprovalSelection() && supportsLifecycle(WorkspaceToolProfiles.SHELL);
} catch (RuntimeException unavailable) {
return false;
@@ -738,7 +837,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
HarnessSessionRef ref = attachment(tenantId, sessionId, true);
PromptReceipt receipt = client().continueManagedRuntime(
ref, promptId, checkpointId, activationId);
pendingRecovery.remove(new AttachmentKey(tenantId, sessionId));
pendingRecovery.remove(attachmentKey(tenantId, sessionId));
return new Admission(receipt.getLastEventId(),
receipt.getEventEpoch());
}
@@ -764,7 +863,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
PromptReceipt receipt = client().cancelManagedRuntime(
new CancelManagedRuntime(ref,
promptId, checkpointId, activationId));
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
attachments.computeIfPresent(key, (ignored, held) -> {
var recovery = ref.getRuntimeRecovery();
if (held == ref && recovery != null
@@ -800,7 +899,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
public SourceStream observeStream(String tenantId, String sessionId, String bootId,
long lastEventId, String eventEpoch) {
requireShellApproval(sessions.requireSession(tenantId, sessionId));
HarnessSessionRef ref = attachments.get(new AttachmentKey(tenantId, sessionId));
HarnessSessionRef ref = attachments.get(attachmentKey(tenantId, sessionId));
if (bootId == null || ref == null || !bootId.equals(ref.getHarnessBootId())) {
throw new IllegalStateException("Original Harness attachment is unavailable");
}
@@ -874,7 +973,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
String actionId,
JsonNode response) {
SessionRecord session = requireReadyForNewWork(tenantId, sessionId, true);
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
HarnessSessionRef ref = attachments.get(key);
if (ref == null) {
// A resident Harness refuses an ordinary load after the dispatcher loses its cache.
@@ -908,7 +1007,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
}
HarnessSessionRef ref = cancellationAttachment(tenantId, sessionId);
client().cancelTurn(ref);
pendingRecovery.remove(new AttachmentKey(tenantId, sessionId));
pendingRecovery.remove(attachmentKey(tenantId, sessionId));
}
@Override
@@ -981,7 +1080,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
public void runChildOperation(String tenantId, String sessionId,
Map<String, Object> body) {
try {
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
if (!attachments.containsKey(key)
&& sessions.requireSession(tenantId, sessionId)
.harnessBootId() != null) {
@@ -1051,7 +1150,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
String sessionId, boolean stopMessages) {
SessionRecord session = sessions.requireSession(tenantId, sessionId);
requireShellApproval(session);
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
HarnessSessionRef cached = attachments.get(key);
if (cached != null) {
return cached;
@@ -1071,6 +1170,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
session.harnessBootId() != null, toolProfile(session),
false, false,
WorkspaceToolProfiles.isPublicShell(toolProfile(session)) && !publicChildLaunchAvailable(session))
.withHarnessPlacement(placementDescriptor(session))
.withAgentDefinition(agentDefinition(session, false))
.withSessionSkillsRef(sessionSkillsRef(session, false))
.withMcpServers(mcpServers(session))
@@ -1102,7 +1202,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
* (HarnessCoordinator.runClaimed).
*/
private void reattachTakenOver(String tenantId, String sessionId) {
if (!attachments.containsKey(new AttachmentKey(tenantId, sessionId))
if (!attachments.containsKey(attachmentKey(tenantId, sessionId))
&& sessions.requireSession(tenantId, sessionId)
.harnessBootId() != null) {
recoverManagedRuntime(tenantId, sessionId, false);
@@ -1164,7 +1264,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
// An attachment held here is not this verb's to evict: a wake
// only accelerates, and a stale entry is repaired by the paths
// that own it.
if (attachments.containsKey(new AttachmentKey(tenantId,
if (attachments.containsKey(attachmentKey(tenantId,
sessionId))) {
return false;
}
@@ -1184,15 +1284,17 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
@Override
public List<AttachedSession> attachedSessions() {
return attachments.keySet().stream()
.map(key -> new AttachedSession(key.tenantId(),
key.sessionId()))
.map(key -> poolEndpoint == null
? new AttachedSession(key.tenantId(), key.sessionId())
: new AttachedSession(key.tenantId(), key.sessionId(),
writerIdentity(), key.placementEpoch()))
.toList();
}
private String doCloseSession(String tenantId, String sessionId) {
attachments.remove(new AttachmentKey(tenantId, sessionId));
mcpManagementAttachments.remove(new AttachmentKey(tenantId, sessionId));
pendingRecovery.remove(new AttachmentKey(tenantId, sessionId));
attachments.remove(attachmentKey(tenantId, sessionId));
mcpManagementAttachments.remove(attachmentKey(tenantId, sessionId));
pendingRecovery.remove(attachmentKey(tenantId, sessionId));
HostedHarnessClient current = client();
current.closeSession(sessionId);
// The client rejects an answer from any other boot.
@@ -1231,7 +1333,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
*/
private HarnessSessionRef channelAttachment(String tenantId,
String sessionId) {
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
HarnessSessionRef attachment = attachments.get(key);
if (attachment == null) {
SessionRecord session = sessions.requireSession(tenantId,
@@ -1254,7 +1356,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
}
private HarnessSessionRef mcpManagementAttachment(String tenantId, String sessionId, boolean initialize) {
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
HarnessSessionRef attached = attachments.get(key);
if (attached == null) {
SessionRecord session = sessions.requireSession(tenantId, sessionId);
@@ -1266,7 +1368,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
private HarnessSessionRef attachment(String tenantId, String sessionId, boolean newWork) {
requireShellApproval(sessions.requireSession(tenantId, sessionId));
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
HarnessSessionRef attachment = attachments.get(key);
if (attachment == null || newWork && attachment.isRecoveryCleanupOnly()) {
createOrLoad(tenantId, sessionId, true,
@@ -1278,7 +1380,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
private HarnessSessionRef cancellationAttachment(String tenantId, String sessionId) {
requireShellApproval(sessions.requireSession(tenantId, sessionId));
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
HarnessSessionRef attached = attachments.get(key);
if (attached == null) {
recoverManagedRuntime(tenantId, sessionId, true);
@@ -1291,7 +1393,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
SessionRecord session = sessions.requireSession(tenantId, sessionId);
requireShellApproval(session);
requireTargetedCancellation(promptId);
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
HarnessSessionRef attached = attachments.get(key);
if (attached == null || attached.isRecoveryCleanupOnly()) {
recoverManagedRuntime(tenantId, sessionId, true, promptId);
@@ -1334,6 +1436,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
CreateHarnessSession.Builder builder =
CreateHarnessSession.builder()
.harnessSessionId(session.sessionId())
.harnessPlacement(placementDescriptor(session))
.approvalMode(
session.workspace() == null
&& session.agentDefinitionDigest() == null
@@ -1425,6 +1528,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
passiveManagedRuntimeRecovery, profile,
driveRuntimeRecovery, cancellationTakeover,
WorkspaceToolProfiles.isPublicShell(profile) && !publicChildLaunchAvailable(session))
.withHarnessPlacement(placementDescriptor(session))
.withAgentDefinition(agentDefinition(session, false))
.withSessionSkillsRef(sessionSkillsRef(session, false))
.withMcpServers(mcpServers(session))
@@ -1535,7 +1639,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
// The recheck must ask the Broker again, not replay the snapshot the
// park was decided on. Dropping the cached attachment forces the
// next load to re-read; a concurrent stream keeps its own ref.
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
pendingRecovery.remove(key);
attachments.remove(key);
}
@@ -1543,7 +1647,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
@Override
public RecoveryStatus inspectRecovery(String tenantId, String sessionId, String promptId) {
var current = client;
var ref = attachments.get(new AttachmentKey(tenantId, sessionId));
var ref = attachments.get(attachmentKey(tenantId, sessionId));
if (current == null || ref == null) {
return null;
}
@@ -1555,7 +1659,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
@Override
public Attachment cachedRecoveryAttachment(String tenantId, String sessionId) {
var ref = attachments.get(new AttachmentKey(tenantId, sessionId));
var ref = attachments.get(attachmentKey(tenantId, sessionId));
return ref == null ? null : new Attachment(ref.getHarnessBootId(), null,
ref.getHarnessLastEventId(), ref.getHarnessEventEpoch());
}
@@ -1572,7 +1676,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
private boolean doRecheckRecoveryCleanup(String tenantId, String sessionId, String promptId) {
var current = client();
var key = new AttachmentKey(tenantId, sessionId);
var key = attachmentKey(tenantId, sessionId);
var ref = attachments.get(key);
if (ref == null) {
var session = sessions.requireSession(tenantId, sessionId);
@@ -1584,6 +1688,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
workspaceExecution.authorizePassiveAttachment(session);
var loaded = current.loadSession(new LoadHarnessSession(sessionId, managedSessionStore(session), false,
toolProfile(session), false, false, WorkspaceToolProfiles.isPublicShell(session.toolProfile()) && !publicChildLaunchAvailable(session))
.withHarnessPlacement(placementDescriptor(session))
.withAgentDefinition(agentDefinition(session, false))
.withSessionSkillsRef(sessionSkillsRef(session, false))
.withPublicChildAgentsVersion(session.publicChildAgentsVersion())
@@ -1629,7 +1734,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
}
private Attachment loadCancellationTakeover(SessionRecord session, String promptId) {
AttachmentKey key = new AttachmentKey(session.tenantId(), session.sessionId());
AttachmentKey key = attachmentKey(session.tenantId(), session.sessionId());
// A Session THIS Harness already serves can still park a dead
// generation's Turn: the plain cancel route settles nothing there
// and answered the coded refusal, so the cancellation load always
@@ -1666,7 +1771,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
*/
private boolean approvalModeConfirmed(SessionRecord session,
HarnessSessionRef ref, boolean minted) {
AttachmentKey key = new AttachmentKey(session.tenantId(),
AttachmentKey key = attachmentKey(session.tenantId(),
session.sessionId());
try {
return actions.approvalMode(session.tenantId(), session.sessionId())
@@ -1710,7 +1815,7 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
if (!cancellation) {
requireApprovalSelectionSupport(session);
}
AttachmentKey key = new AttachmentKey(tenantId, sessionId);
AttachmentKey key = attachmentKey(tenantId, sessionId);
HarnessSessionRef cached = attachments.get(key);
boolean minted = cached == null || cached.isRecoveryCleanupOnly();
if (cached == null || cached.isRecoveryCleanupOnly()) {
@@ -1935,9 +2040,11 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
sessionStore.getWriterLeaseDuration())
.allowInsecureHttp(
sessionStore.isAllowInsecureHttp());
if (credentials.isBound()) {
builder.writerToken(credentials.issue(session.tenantId(),
workspace, session.sessionId()));
PoolCall call = placementCall(session.tenantId(), session.sessionId());
if (call != null) {
builder.writerToken(call.writerToken());
} else if (credentials.isBound()) {
builder.writerToken(credentials.issue(session.tenantId(), workspace, session.sessionId()));
}
return builder.build();
}
@@ -1973,8 +2080,9 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
// Package-private so the adoption race test substitutes the
// replacement instead of standing up a live /capabilities call.
HostedHarnessClient createClient() {
return HostedHarnessClient.builder()
.baseUri(URI.create(properties.getBaseUrl()))
HostedHarnessClient created = HostedHarnessClient.builder()
.baseUri(poolEndpoint == null ? URI.create(properties.getBaseUrl()) : poolEndpoint.baseUri())
.sslContext(sslContext)
.bearerToken(properties.getToken())
.capabilityDigest(properties.getCapabilityDigest())
.connectTimeout(properties.getConnectTimeout())
@@ -1982,6 +2090,12 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
.loadTimeout(properties.getLoadTimeout())
.heartbeatInterval(properties.getHeartbeatInterval())
.build();
if (poolEndpoint != null && (!poolBootId.equals(created.capabilities().getBootId())
|| !poolEndpoint.instanceId().equals(created.capabilities().getInstanceId()))) {
created.close();
throw new IllegalStateException("Assigned Harness incarnation changed");
}
return created;
}
/**
@@ -2007,6 +2121,9 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
* null report.
*/
private void adoptGeneration(HostedHarnessGenerationException error) {
if (poolEndpoint != null) {
return;
}
HostedHarnessClient stale = null;
boolean dropped = false;
synchronized (this) {
@@ -2058,6 +2175,6 @@ public class QwenHostedHarnessConnector implements HarnessConnector {
private int holders;
}
private record AttachmentKey(String tenantId, String sessionId) {
private record AttachmentKey(String tenantId, String sessionId, long placementEpoch) {
}
}
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/persistence/HarnessPlacementStore.java
+304
View File
@@ -0,0 +1,304 @@
package com.alibaba.qwen.code.managedagent.persistence;
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
import com.alibaba.qwen.code.managedagent.model.StoreModels.OperationRecord;
import com.alibaba.qwen.code.managedagent.security.WriterCredentialPolicy;
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchemaTime;
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.lang3.StringUtils;
import org.springframework.http.HttpStatus;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Repository;
import org.springframework.transaction.annotation.Transactional;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
/** Durable ownership; an endpoint disappearing never authorizes another writer. */
@Repository
public class HarnessPlacementStore {
public static final String RELEASE_DOMAIN = "hosted_harness_release";
private static final String COLUMNS = "tenant_id, session_id, workspace_id, harness_instance_id, "
+ "harness_owner_boot_id, harness_placement_epoch, harness_placement_state";
private final JdbcTemplate jdbc;
public HarnessPlacementStore(JdbcTemplate jdbc) {
this.jdbc = jdbc;
}
public record Placement(String tenantId, String sessionId, String workspaceId,
String instanceId, String bootId, long epoch, String state) {
}
public Placement get(String tenant, String session) {
var rows = placements(" WHERE tenant_id = ? AND session_id = ?", tenant, session);
return rows.isEmpty() || rows.getFirst().instanceId() == null ? null : rows.getFirst();
}
public boolean hasUnreleasedOwnership(String instance, String boot) {
return !jdbc.query("SELECT m.session_id FROM ma_session m WHERE m.harness_instance_id = ?"
+ " AND m.harness_owner_boot_id = ? AND m.resource_gc_completed = FALSE"
+ " AND m.harness_placement_state <> 'RELEASED'"
+ " AND (m.status NOT IN ('CLOSED', 'ARCHIVED', 'DELETED') OR EXISTS ("
+ " SELECT 1 FROM ma_session_head h WHERE h.tenant_id = m.tenant_id AND h.session_id = m.session_id"
+ " AND h.head_kind = 'MANAGED_SESSION' AND h.state = 'ACTIVE' AND h.writer_lease_until > "
+ ManagedSchemaTime.SQL_NOW + ")) LIMIT 1",
(row, index) -> row.getString(1), instance, boot).isEmpty();
}
public Map<String, Object> releaseReceipt(String tenant, String session, long epoch, String operationId) {
String scope = ManagedSessionStore.sessionScopeKey(tenant, session);
String key = ManagedExtensionProjection.recordKey(session, RELEASE_DOMAIN, releaseId(epoch));
var saved = jdbc.query("SELECT b.inline_bytes, b.sha256 FROM ma_session_record r JOIN ma_resource b"
+ " ON b.session_scope_key = r.session_scope_key AND b.resource_id = r.record_resource_id"
+ " AND b.tenant_id = r.tenant_id AND b.workspace_id = r.workspace_id AND b.session_id = r.session_id"
+ " WHERE r.session_scope_key = ? AND r.record_key = ? AND r.tenant_id = ? AND r.session_id = ?"
+ " AND r.domain = 'hosted_harness_release' AND r.record_kind = 'DOMAIN'"
+ " AND b.kind = ? AND b.schema_version = 1 AND b.state = 'REFERENCED'"
+ " AND b.storage_kind = 'MYSQL_INLINE' AND b.byte_length BETWEEN 1 AND 16384 AND LENGTH(b.inline_bytes) = b.byte_length",
(row, index) -> {
byte[] bytes = row.getBytes(1);
if (!DigestUtils.sha256Hex(bytes).equals(row.getString(2))) {
throw blocked("harness_release_receipt_corrupt");
}
return ManagedExtensionRecordStore.parse(new String(bytes, StandardCharsets.UTF_8));
}, scope, key, tenant, session, "managed-hosted_harness_release");
if (saved.isEmpty()) {
return null;
}
var receipt = saved.getFirst();
ManagedHarnessReleaseRecords.require(receipt);
if (!tenant.equals(receipt.path("tenantId").asText()) || !session.equals(receipt.path("sessionId").asText())
|| receipt.path("placementEpoch").asLong() != epoch || !operationId.equals(receipt.path("operationId").asText())) {
throw blocked("harness_release_operation_conflict");
}
Map<String, Object> result = new LinkedHashMap<>();
receipt.fields().forEachRemaining(field -> {
var value = field.getValue();
result.put(field.getKey(), value.isTextual() ? value.textValue() : value.isNumber() ? value.longValue()
: Map.of("lastCommitSequence", value.path("lastCommitSequence").asLong(),
"committedPrefixHash", value.path("committedPrefixHash").asText()));
});
return Map.copyOf(result);
}
@Transactional
public Placement assign(String tenant, String session, String instance, String boot) {
return assign(tenant, session, instance, boot, null);
}
@Transactional
public Placement assignForLifecycle(OperationRecord operation, String instance, String boot) {
return assign(operation.tenantId(), operation.sessionId(), instance, boot, operation);
}
private Placement assign(String tenant, String session, String instance, String boot, OperationRecord operation) {
identity(instance);
identity(boot);
Placement current = lockEligible(tenant, session, operation);
if (current.instanceId() != null) {
return current;
}
// The private daemon indexes Sessions by id, independent of tenant.
// Reject an ambiguous CONTROL id before any handle is exposed.
Integer collisions = jdbc.queryForObject("SELECT COUNT(*) FROM ma_session WHERE session_id = ?",
Integer.class, session);
if (collisions == null || collisions != 1) {
throw blocked("harness_session_scope_ambiguous");
}
jdbc.update("UPDATE ma_session SET harness_instance_id = ?, harness_owner_boot_id = ?,"
+ " harness_placement_epoch = 1, harness_placement_state = 'ACTIVE'"
+ " WHERE tenant_id = ? AND session_id = ?", instance, boot, tenant, session);
return new Placement(tenant, session, current.workspaceId(), instance, boot, 1, "ACTIVE");
}
@Transactional
public boolean quiesce(String tenant, String session, Placement expected) {
var current = lock(tenant, session);
if (!sameOwner(current, expected) || "RELEASED".equals(current.state())) {
return false;
}
jdbc.update("UPDATE ma_session SET harness_placement_state = 'QUIESCING'"
+ " WHERE tenant_id = ? AND session_id = ?", tenant, session);
return true;
}
@Transactional
public Placement takeover(String tenant, String session, Placement expected, String instance, String boot) {
return takeover(tenant, session, expected, instance, boot, null);
}
@Transactional
public Placement takeoverForLifecycle(OperationRecord operation, Placement expected, String instance, String boot) {
return takeover(operation.tenantId(), operation.sessionId(), expected, instance, boot, operation);
}
private Placement takeover(String tenant, String session, Placement expected, String instance, String boot, OperationRecord operation) {
identity(instance);
identity(boot);
Placement current = lockEligible(tenant, session, operation);
if (!sameOwner(current, expected)) {
throw blocked("harness_placement_changed");
}
var heads = jdbc.query("SELECT state, writer_lease_until, activation_phase, activation_expires_at,"
+ " activation_head_revision, journal_revision, activation_epoch"
+ " FROM ma_session_head WHERE tenant_key = ? AND harness_key = ? AND tenant_id = ?"
+ " AND session_id = ? AND head_kind = 'MANAGED_SESSION' FOR UPDATE",
(row, index) -> new Object[] {row.getString(1), row.getObject(2, Long.class),
row.getString(3), row.getObject(4, Long.class), row.getObject(5, Long.class),
row.getLong(6), row.getLong(7)},
JdbcRuntimeBindingRepository.harnessDrainKey(tenant),
JdbcRuntimeBindingRepository.harnessDrainKey(session), tenant, session);
if (!heads.isEmpty()) {
Object[] head = heads.getFirst();
long now = now(jdbc);
boolean released = "RELEASED".equals(current.state()) && "SEALED".equals(head[0])
&& released(jdbc, tenant, session, current.epoch());
boolean activationKnown = Objects.equals(head[4], head[5]) || Objects.equals(head[6], 0L);
boolean expired = head[1] != null && (Long) head[1] <= now && activationKnown
&& (Objects.equals(head[6], 0L) || "released".equals(head[2]) || "revoked".equals(head[2])
|| "active".equals(head[2]) && head[3] != null && (Long) head[3] <= now);
if (!released && !expired) {
throw blocked("harness_owner_still_live");
}
}
if (current.epoch() >= ManagedSessionStoreModels.MAX_SAFE_COUNTER) {
throw blocked("harness_placement_epoch_exhausted");
}
long epoch = current.epoch() + 1;
jdbc.update("UPDATE ma_session SET harness_instance_id = ?, harness_owner_boot_id = ?,"
+ " harness_placement_epoch = ?, harness_placement_state = 'ACTIVE'"
+ " WHERE tenant_id = ? AND session_id = ?", instance, boot, epoch, tenant, session);
return new Placement(tenant, session, current.workspaceId(), instance, boot, epoch, "ACTIVE");
}
static Placement requireCurrent(JdbcTemplate jdbc, WriterCredentialPolicy credentials,
String tenant, String workspace, String session, String token, String writer, boolean newWork) {
var rows = jdbc.query("SELECT " + COLUMNS + " FROM ma_session WHERE tenant_id = ? AND session_id = ? FOR UPDATE",
(row, index) -> new Placement(row.getString(1), row.getString(2), row.getString(3),
row.getString(4), row.getString(5), row.getLong(6), row.getString(7)), tenant, session);
Placement current = rows.isEmpty() ? null : rows.getFirst();
if (current == null || current.instanceId() == null) {
if (WriterCredentialPolicy.isPlacementToken(token)) {
throw blocked("harness_placement_changed");
}
return null;
}
credentials.requirePlacement(tenant, workspace, session, current.instanceId(), current.bootId(), current.epoch(), token);
if (writer != null && !current.bootId().equals(writer)) {
throw blocked("harness_placement_changed");
}
if ("RELEASED".equals(current.state())) {
throw blocked("harness_placement_released");
}
if (newWork && !"ACTIVE".equals(current.state())) {
throw blocked("harness_placement_quiescing");
}
return current;
}
static String releaseId(long epoch) {
return "placement:" + epoch;
}
static boolean released(JdbcTemplate jdbc, String tenant, String session, long epoch) {
return !jdbc.query("SELECT record_id FROM ma_session_record WHERE session_scope_key = ?"
+ " AND record_key = ? AND tenant_id = ? AND session_id = ? AND domain = ? AND record_kind = 'DOMAIN'",
(row, index) -> row.getString(1), ManagedSessionStore.sessionScopeKey(tenant, session),
ManagedExtensionProjection.recordKey(session, RELEASE_DOMAIN, releaseId(epoch)),
tenant, session, RELEASE_DOMAIN).isEmpty();
}
private Placement lockEligible(String tenant, String session, OperationRecord operation) {
if (operation != null) {
WorkspaceLifecycleStore.lockPlacement(jdbc, tenant);
ToolPublicationRetentionStore.lockTenant(jdbc, tenant);
}
var current = lock(tenant, session);
if (operation != null) {
requireLifecycleClaim(operation);
}
String pendingStatus = operation == null ? "ACTIVE" : switch (operation.kind()) {
case DELETE -> "DELETING";
case ARCHIVE -> "ARCHIVING";
default -> "CLOSING";
};
var admission = jdbc.query("SELECT status, session_kind, workspace_id, csi_guard FROM ma_session"
+ " WHERE tenant_id = ? AND session_id = ? FOR UPDATE", (row, index) ->
pendingStatus.equals(row.getString(1))
&& ("PUBLIC".equals(row.getString(2)) || row.getString(3) != null)
&& !row.getBoolean(4), tenant, session);
if (admission.isEmpty() || !admission.getFirst()) {
throw blocked("harness_placement_not_admitted");
}
return current;
}
private void requireLifecycleClaim(OperationRecord operation) {
if (!List.of("CLOSE", "ARCHIVE", "DELETE").contains(operation.kind().name())
|| operation.lifecycleProtocolVersion() < 0 || operation.lifecycleProtocolVersion() > 2
|| operation.leaseOwner() == null) {
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_claim_fenced");
}
var rows = jdbc.query("SELECT o.claim_generation, o.claim_until, o.claim_owner, o.delivery_state,"
+ " o.lifecycle_protocol_version, f.phase, f.operation_id FROM ma_operation o JOIN ma_session_head f"
+ " ON f.tenant_id = o.tenant_id AND f.harness_session_id = o.session_id"
+ " WHERE o.tenant_id = ? AND o.session_id = ? AND o.operation_id = ? AND o.operation_kind = ?"
+ " AND f.tenant_key = ? AND f.harness_key = ? FOR UPDATE",
(row, index) -> new Object[] {row.getLong(1), row.getLong(2), row.getString(3), row.getString(4),
row.getInt(5), row.getString(6), row.getString(7)},
operation.tenantId(), operation.sessionId(), operation.operationId(), operation.kind().name(),
JdbcRuntimeBindingRepository.harnessDrainKey(operation.tenantId()),
JdbcRuntimeBindingRepository.harnessDrainKey(operation.sessionId()));
if (rows.size() != 1) {
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_claim_fenced");
}
Object[] row = rows.getFirst();
if (!Objects.equals(row[0], operation.claimGeneration()) || (Long) row[1] <= now(jdbc)
|| !operation.leaseOwner().equals(row[2]) || !"LEASED".equals(row[3])
|| !Objects.equals(row[4], operation.lifecycleProtocolVersion())
|| !(operation.lifecycleProtocolVersion() == 0
? "DRAINING".equals(row[5]) && row[6] == null
: List.of("DRAINING", "LIFECYCLE_ONLY").contains(row[5]) && operation.operationId().equals(row[6]))) {
throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_claim_fenced");
}
}
private Placement lock(String tenant, String session) {
var rows = placements(" WHERE tenant_id = ? AND session_id = ? FOR UPDATE", tenant, session);
if (rows.isEmpty()) {
throw new ManagedAgentException(HttpStatus.NOT_FOUND, "session_not_found", "The Session does not exist.");
}
return rows.getFirst();
}
private List<Placement> placements(String suffix, Object... args) {
return jdbc.query("SELECT " + COLUMNS + " FROM ma_session" + suffix,
(row, index) -> new Placement(row.getString(1), row.getString(2), row.getString(3),
row.getString(4), row.getString(5), row.getLong(6), row.getString(7)), args);
}
private static boolean sameOwner(Placement left, Placement right) {
return left != null && right != null && left.epoch() == right.epoch()
&& Objects.equals(left.tenantId(), right.tenantId()) && Objects.equals(left.sessionId(), right.sessionId())
&& Objects.equals(left.instanceId(), right.instanceId()) && Objects.equals(left.bootId(), right.bootId());
}
static long now(JdbcTemplate jdbc) {
return jdbc.queryForObject("SELECT " + ManagedSchemaTime.SQL_NOW, Long.class);
}
private static void identity(String id) {
if (StringUtils.isBlank(id) || id.indexOf('\0') >= 0 || id.getBytes(StandardCharsets.UTF_8).length > 144) {
throw new IllegalArgumentException("Harness instance and boot identities must contain 1 to 144 UTF-8 bytes.");
}
}
static ManagedAgentException blocked(String code) {
return new ManagedAgentException(HttpStatus.CONFLICT, code, "The Harness ownership boundary refused this operation.");
}
}
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/persistence/ManagedExtensionProjection.java
+4
View File
@@ -24,6 +24,10 @@ import static java.util.Map.entry;
public final class ManagedExtensionProjection {
/** The record bodies defined so far; a slice adds its body here. ofEntries, since Map.of caps ten pairs. */
public static final Map<String, Body> RECORD_BODIES = Map.ofEntries(
entry("hosted_harness_release", new Body(record -> null,
ManagedHarnessReleaseRecords::require,
body -> HarnessPlacementStore.releaseId(body.path("placementEpoch").asLong()),
body -> true, (previous, next) -> false)),
entry("history_maintenance", new Body(record -> null,
ManagedHistoryMaintenanceRecords::requireRecord,
body -> body.get("operationId").textValue(),
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/persistence/ManagedExtensionRecordStore.java
+2
View File
@@ -1045,6 +1045,8 @@ public class ManagedExtensionRecordStore {
*/
private static void requireDomainCommitted(JsonNode payload,
String domain, boolean stageH) {
require(!HarnessPlacementStore.RELEASE_DOMAIN.equals(domain),
"Harness release receipts can only be written by the ownership release transaction.");
try {
ManagedExtensionRecords.closed(payload, PAYLOAD_FIELDS,
"event.payload");
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/persistence/ManagedExtensionRecords.java
+2 -2
View File
@@ -35,8 +35,8 @@ public final class ManagedExtensionRecords {
"memory_job", "monitor_run", "goal_state", "todo_state",
"plan_mode", "team_state", "team_task", "team_message",
"team_plan", "session_message", "session_command",
"session_metadata",
"file_history", "session_source");
"session_metadata", "file_history", "session_source",
"hosted_harness_release");
/** The event kinds of managed-session/1, as the authority writes them. */
public static final List<String> EVENT_KINDS = List.of("input.accepted",
"wake.requested", "activation.changed", "model.attempt",
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/persistence/ManagedHarnessReleaseRecords.java
+36
View File
@@ -0,0 +1,36 @@
package com.alibaba.qwen.code.managedagent.persistence;
import com.fasterxml.jackson.databind.JsonNode;
import java.util.List;
import java.util.Set;
/** A private control receipt, not a task, run, or business Session boundary. */
public final class ManagedHarnessReleaseRecords {
private ManagedHarnessReleaseRecords() {
}
public static void require(JsonNode record) {
ManagedExtensionRecords.closed(record, Set.of("format", "tenantId", "workspaceId", "sessionId",
"operationId", "instanceId", "bootId", "placementEpoch", "writerGeneration", "journalRevision",
"committedSequence", "commitProof", "state"), "Harness release");
ManagedExtensionRecords.oneOf(record.get("format"), List.of("qwen-hosted-harness-release/1"), "format");
ManagedExtensionRecords.oneOf(record.get("state"), List.of("RELEASED"), "state");
for (String field : List.of("tenantId", "workspaceId", "sessionId", "operationId", "instanceId", "bootId")) {
ManagedExtensionRecords.id(record.get(field), "Harness release " + field);
}
for (String field : List.of("placementEpoch", "writerGeneration", "journalRevision")) {
ManagedExtensionRecords.count(record.get(field), 1, ManagedSessionStoreModels.MAX_SAFE_COUNTER, field);
}
ManagedExtensionRecords.count(record.get("committedSequence"), 0, ManagedSessionStoreModels.MAX_SAFE_COUNTER,
"committedSequence");
JsonNode proof = record.get("commitProof");
ManagedExtensionRecords.closed(proof, Set.of("lastCommitSequence", "committedPrefixHash"), "commitProof");
ManagedExtensionRecords.count(proof.get("lastCommitSequence"), 0, ManagedSessionStoreModels.MAX_SAFE_COUNTER,
"lastCommitSequence");
ManagedExtensionRecords.digest(proof.get("committedPrefixHash"), "committedPrefixHash");
if (proof.path("lastCommitSequence").asLong() != record.path("committedSequence").asLong()) {
throw new ManagedExtensionRecords.InvalidRecordException("Harness release prefix is invalid.");
}
}
}
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/persistence/ManagedSessionStore.java
+135
View File
@@ -197,6 +197,7 @@ public class ManagedSessionStore {
lockBounded(() -> ToolPublicationRetentionStore.lockTenant(jdbc, tenantId,
GUARD_ACQUIRE_TIMEOUT_SECONDS));
ensureSessionRoot(tenantId, sessionId);
requirePlacement(tenantId, request.workspaceId(), sessionId, writerToken, request.writerId(), false);
jdbc.queryForList(
"SELECT session_id FROM ma_session WHERE tenant_id = ? AND session_id = ? FOR"
+ " UPDATE",
@@ -260,6 +261,7 @@ public class ManagedSessionStore {
GUARD_ACQUIRE_TIMEOUT_SECONDS));
lockBounded(() -> ToolPublicationRetentionStore.lockTenant(jdbc, tenantId,
GUARD_ACQUIRE_TIMEOUT_SECONDS));
ToolPublicationRetentionStore.lockSessionRoot(jdbc, tenantId, sessionId);
WorkspaceLifecycleStore.requireClaim(jdbc, tenantId, sessionId, authority, true);
}
@@ -378,6 +380,7 @@ public class ManagedSessionStore {
// Session root is locked before the drain admission reads the head
// row, keeping the root-then-head order of every other writer.
ToolPublicationRetentionStore.lockSessionRoot(jdbc, tenantId, sessionId);
requirePlacement(tenantId, request.workspaceId(), sessionId, writerToken, request.writerId(), false);
if (!"legacy-close".equals(request.kind()) || !WorkspaceLifecycleStore.legacyClose(jdbc, tenantId, sessionId)) {
requireOrdinaryAdmission(tenantId, sessionId);
}
@@ -515,6 +518,11 @@ public class ManagedSessionStore {
credentials.require(tenantId, workspaceId, sessionId, writerToken);
}
private HarnessPlacementStore.Placement requirePlacement(String tenant, String workspace,
String session, String token, String writer, boolean newWork) {
return HarnessPlacementStore.requireCurrent(jdbc, credentials, tenant, workspace, session, token, writer, newWork);
}
record PublicationWriter(long now, long leaseUntil, long journalRevision,
long committedSequence, long activationEpoch, String checkpointId, String recoveryStatus,
String activationId, String activationPhase, Long activationEventEpoch,
@@ -533,6 +541,7 @@ public class ManagedSessionStore {
// before #13800 decoupled ordinary traffic.
lockBounded(() -> ToolPublicationRetentionStore.lockTenant(jdbc, tenant,
GUARD_ACQUIRE_TIMEOUT_SECONDS));
requirePlacement(tenant, workspace, session, token, writer, false);
var continuation = lockCsiContinuation(tenant, session);
var csiOriginal = continuation == null ? null : continuation.original();
if (csiOriginal != null) {
@@ -588,6 +597,7 @@ public class ManagedSessionStore {
}
WorkspaceMigrationAdmission.requireSessionOpen(jdbc, tenantId, sessionId, request.workspaceId());
ensureSessionRoot(tenantId, sessionId);
requirePlacement(tenantId, request.workspaceId(), sessionId, writerToken, request.writerId(), true);
bindControlSessionStorage(tenantId, request.workspaceId(), sessionId);
// Materialize the key before any locking read: missing-key RR locks can
// deadlock independent tenants when both first writers insert a head.
@@ -743,6 +753,7 @@ public class ManagedSessionStore {
validateStableId(request.writerId(), "writerId");
validateCounter(request.writerGeneration(), "writerGeneration", 1);
validateLeaseMillis(request.leaseMillis());
requirePlacement(tenantId, request.workspaceId(), sessionId, writerToken, request.writerId(), false);
var continuation = lockCsiContinuation(tenantId, sessionId);
var csiOriginal = continuation == null ? null : continuation.original();
if (csiOriginal != null) {
@@ -789,6 +800,7 @@ public class ManagedSessionStore {
checkLifecycleWriter(tenantId, sessionId, authority);
validateStableId(request.writerId(), "writerId");
validateCounter(request.writerGeneration(), "writerGeneration", 1);
requirePlacement(tenantId, request.workspaceId(), sessionId, writerToken, request.writerId(), false);
if (lockCsiOriginal(tenantId, sessionId) != null) {
throw conflict(
"csi_finalize_required",
@@ -820,6 +832,119 @@ public class ManagedSessionStore {
return new SealReceipt(head.writerGeneration(), "SEALED", false);
}
@Transactional
public void quiesceOwnership(String tenant, String session, String token,
ManagedSessionStoreModels.QuiesceOwnershipRequest request) {
validateScope(tenant, request.workspaceId(), session);
requireCredential(tenant, request.workspaceId(), session, token);
credentials.requirePlacement(tenant, request.workspaceId(), session, request.instanceId(), request.bootId(),
request.placementEpoch(), token);
requirePlacement(tenant, request.workspaceId(), session, token, request.writerId(), false);
HeadRow head = requireHeadForUpdate(tenant, session);
requireHeadScope(head, tenant, request.workspaceId(), session);
requireWriter(head, request.writerId(), request.writerGeneration(), token, databaseNow(), true);
jdbc.update("UPDATE ma_session SET harness_placement_state = 'QUIESCING' WHERE tenant_id = ? AND session_id = ?", tenant, session);
}
@Transactional
public JsonNode releaseOwnership(String tenantId, String sessionId, String writerToken,
ManagedSessionStoreModels.ReleaseOwnershipRequest request) {
validateScope(tenantId, request.workspaceId(), sessionId);
requireCredential(tenantId, request.workspaceId(), sessionId, writerToken);
validateStableId(request.operationId(), "operationId");
validateStableId(request.writerId(), "writerId");
validateCounter(request.writerGeneration(), "writerGeneration", 1);
validateCounter(request.journalRevision(), "journalRevision", 1);
validateCounter(request.placementEpoch(), "placementEpoch", 1);
if (request.commitProof() == null || !request.bootId().equals(request.writerId())) {
throw invalid("Ownership release requires the original boot and committed prefix.");
}
validateDigest(request.commitProof().committedPrefixHash(), "committedPrefixHash", false);
credentials.requirePlacement(tenantId, request.workspaceId(), sessionId,
request.instanceId(), request.bootId(), request.placementEpoch(), writerToken);
lockBounded(() -> WorkspaceLifecycleStore.lockPlacement(jdbc, tenantId, GUARD_ACQUIRE_TIMEOUT_SECONDS));
lockBounded(() -> ToolPublicationRetentionStore.lockTenant(jdbc, tenantId, GUARD_ACQUIRE_TIMEOUT_SECONDS));
ToolPublicationRetentionStore.lockSessionRoot(jdbc, tenantId, sessionId);
String recordId = HarnessPlacementStore.releaseId(request.placementEpoch());
String scope = sessionScopeKey(tenantId, sessionId);
String recordKey = ManagedExtensionProjection.recordKey(sessionId, HarnessPlacementStore.RELEASE_DOMAIN, recordId);
ObjectNode receipt = CSI_JSON.createObjectNode();
receipt.put("format", "qwen-hosted-harness-release/1");
receipt.put("tenantId", tenantId);
receipt.put("workspaceId", request.workspaceId());
receipt.put("sessionId", sessionId);
receipt.put("operationId", request.operationId());
receipt.put("instanceId", request.instanceId());
receipt.put("bootId", request.bootId());
receipt.put("placementEpoch", request.placementEpoch());
receipt.put("writerGeneration", request.writerGeneration());
receipt.put("journalRevision", request.journalRevision());
receipt.put("committedSequence", request.commitProof().lastCommitSequence());
receipt.set("commitProof", CSI_JSON.valueToTree(request.commitProof()));
receipt.put("state", "RELEASED");
// Normalize integer nodes to the representation read back from the stored JSON.
receipt = (ObjectNode) ManagedExtensionRecordStore.parse(receipt.toString());
var saved = jdbc.query("SELECT record_resource_id FROM ma_session_record WHERE session_scope_key = ?"
+ " AND record_key = ? AND tenant_id = ? AND session_id = ? AND domain = ? AND record_kind = 'DOMAIN'",
(row, index) -> row.getString(1), scope, recordKey, tenantId, sessionId, HarnessPlacementStore.RELEASE_DOMAIN);
if (!saved.isEmpty()) {
var resource = findResource(scope, saved.getFirst());
if (resource == null) {
throw HarnessPlacementStore.blocked("harness_release_receipt_missing");
}
verifyStoredResource(resource);
JsonNode original = ManagedExtensionRecordStore.parse(new String(resource.bytes(), StandardCharsets.UTF_8));
if (!receipt.equals(original)) {
throw HarnessPlacementStore.blocked("harness_release_operation_conflict");
}
return original;
}
var placement = requirePlacement(tenantId, request.workspaceId(), sessionId, writerToken, request.writerId(), false);
if (placement == null || !"QUIESCING".equals(placement.state())) {
throw HarnessPlacementStore.blocked("harness_placement_not_quiescing");
}
requireNoLifecycleClaim(tenantId, sessionId);
HeadRow head = requireHeadForUpdate(tenantId, sessionId);
requireHeadScope(head, tenantId, request.workspaceId(), sessionId);
Timestamp now = databaseNow();
requireWriter(head, request.writerId(), request.writerGeneration(), writerToken, now, true);
if (head.journalRevision() != request.journalRevision()
|| head.committedSequence() != request.commitProof().lastCommitSequence()
|| !Objects.equals(head.lastCommitDigest(), request.commitProof().committedPrefixHash())
|| !"released".equals(head.activationPhase())
|| !Objects.equals(head.activationHeadRevision(), head.journalRevision())) {
throw HarnessPlacementStore.blocked("harness_release_proof_conflict");
}
String resourceId = "harness-release:" + request.placementEpoch();
byte[] bytes;
try {
bytes = CSI_JSON.writeValueAsBytes(receipt);
} catch (com.fasterxml.jackson.core.JsonProcessingException failure) {
throw new IllegalStateException("Ownership release receipt cannot be encoded.", failure);
}
ManagedHarnessReleaseRecords.require(receipt);
String resourceKey = resourceKey(scope, resourceId);
jdbc.update("INSERT INTO ma_resource (resource_key, session_scope_key, tenant_id, workspace_id, session_id,"
+ " resource_id, kind, schema_version, byte_length, sha256, storage_kind, inline_bytes, state, gmt_create, last_verified_at)"
+ " VALUES (?, ?, ?, ?, ?, ?, 'managed-hosted_harness_release', 1, ?, ?, 'MYSQL_INLINE', ?, 'REFERENCED', ?, ?)",
resourceKey, scope, tenantId, request.workspaceId(), sessionId, resourceId,
bytes.length, sha256(bytes), bytes, now.getTime(), now.getTime());
jdbc.update("INSERT INTO ma_session_record (session_scope_key, record_key, tenant_id, workspace_id, session_id,"
+ " domain, record_id, operation_hash, revision, record_resource_id, gmt_create, settled_at, first_sequence)"
+ " VALUES (?, ?, ?, ?, ?, ?, ?, ?, 1, ?, ?, ?, ?)", scope, recordKey, tenantId,
request.workspaceId(), sessionId, HarnessPlacementStore.RELEASE_DOMAIN, recordId, sha256(request.operationId().getBytes(StandardCharsets.UTF_8)),
resourceId, now.getTime(), now.getTime(), head.committedSequence());
jdbc.update("INSERT INTO ma_resource_ref (session_scope_key, owner_kind, owner_key, owner_revision, resource_key,"
+ " tenant_id, workspace_id, session_id, resource_id, gmt_create) VALUES (?, 'SESSION_RECORD', ?, 1, ?, ?, ?, ?, ?, ?)",
scope, recordKey, resourceKey, tenantId, request.workspaceId(), sessionId, resourceId, now.getTime());
jdbc.update("UPDATE ma_session_head SET state = 'SEALED', gmt_modified = ? WHERE tenant_key = ? AND harness_key = ?"
+ " AND tenant_id = ? AND session_id = ? AND head_kind = 'MANAGED_SESSION'", now.getTime(),
JdbcRuntimeBindingRepository.harnessDrainKey(tenantId), JdbcRuntimeBindingRepository.harnessDrainKey(sessionId), tenantId, sessionId);
jdbc.update("UPDATE ma_session SET harness_placement_state = 'RELEASED' WHERE tenant_id = ? AND session_id = ?",
tenantId, sessionId);
return receipt;
}
@Transactional
public RecoveryStateReceipt blockRecovery(String tenantId,
String sessionId, String writerToken,
@@ -838,6 +963,7 @@ public class ManagedSessionStore {
// would reject non-ASCII codes that legitimately fit VARCHAR(128).
validateText(request.recoveryDetailCode(), "recoveryDetailCode",
MAX_TEXT_BYTES);
requirePlacement(tenantId, request.workspaceId(), sessionId, writerToken, request.writerId(), false);
var continuation = lockCsiContinuation(tenantId, sessionId);
var csiOriginal = continuation == null ? null : continuation.original();
if (csiOriginal != null) {
@@ -902,6 +1028,8 @@ public class ManagedSessionStore {
GUARD_ACQUIRE_TIMEOUT_SECONDS));
validateStableId(request.writerId(), "writerId");
ValidatedCommit validated = validateCommit(request);
HarnessPlacementStore.Placement placement = requirePlacement(tenantId, request.workspaceId(),
sessionId, writerToken, request.writerId(), false);
boolean promotes = ManagedHostedRecoveryRecords.requiresVersionTwo(validated.recordBytes(), request.resources())
|| validated.resources().stream().anyMatch(resource -> {
if (!"managed-action-options".equals(resource.resource().kind()) || resource.bytes() == null) {
@@ -1176,6 +1304,12 @@ public class ManagedSessionStore {
validated.recordDigest(), now.getTime());
ActivationChange activation = activationChange(
applied.lastActivation());
if (placement != null && "QUIESCING".equals(placement.state()) && activation != null
&& "active".equals(activation.phase())
&& (!"active".equals(head.activationPhase()) || !Objects.equals(activation.id(), head.activationId())
|| !Objects.equals(activation.epoch(), head.activationEventEpoch()))) {
throw HarnessPlacementStore.blocked("harness_placement_quiescing");
}
// Never certify a skewed activation projection as current.
Long activationHeadRevision = head.activationHeadRevision();
if (activation != null || (activationHeadRevision != null
@@ -2001,6 +2135,7 @@ public class ManagedSessionStore {
GUARD_ACQUIRE_TIMEOUT_SECONDS));
// After both bounded waits, so the CSI check's read does not pin this
// transaction's snapshot ahead of them.
requirePlacement(tenantId, request.workspaceId(), sessionId, writerToken, request.writerId(), false);
var continuation = lockCsiContinuation(tenantId, sessionId);
var csiOriginal = continuation == null ? null : continuation.original();
if (csiOriginal != null) {
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/persistence/ManagedSessionStoreModels.java
+25
View File
@@ -108,6 +108,31 @@ public final class ManagedSessionStoreModels {
@Min(1) @Max(MAX_SAFE_COUNTER) long writerGeneration) {
}
public record OwnershipCommitProof(@Min(0) @Max(MAX_SAFE_COUNTER) long lastCommitSequence,
@NotBlank @Pattern(regexp = "[0-9a-f]{64}") String committedPrefixHash) {
}
public record QuiesceOwnershipRequest(
@NotBlank @Size(max = 512) String workspaceId,
@NotBlank @Size(max = 512) String writerId,
@Min(1) @Max(MAX_SAFE_COUNTER) long writerGeneration,
@Min(1) @Max(MAX_SAFE_COUNTER) long placementEpoch,
@NotBlank @Size(max = 144) String instanceId,
@NotBlank @Size(max = 144) String bootId) {
}
public record ReleaseOwnershipRequest(
@NotBlank @Size(max = 512) String workspaceId,
@NotBlank @Size(max = 512) String writerId,
@Min(1) @Max(MAX_SAFE_COUNTER) long writerGeneration,
@Min(1) @Max(MAX_SAFE_COUNTER) long journalRevision,
@Min(1) @Max(MAX_SAFE_COUNTER) long placementEpoch,
@NotBlank @Size(max = 144) String instanceId,
@NotBlank @Size(max = 144) String bootId,
@NotBlank @Size(max = 512) String operationId,
@NotNull @jakarta.validation.Valid OwnershipCommitProof commitProof) {
}
@JsonInclude(JsonInclude.Include.NON_NULL)
public record WriterGrant(long writerGeneration, long leaseUntil,
long journalRevision, long committedSequence,
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/persistence/WorkspaceCsiApplicationInventory.java
+5 -1
View File
@@ -217,7 +217,8 @@ final class WorkspaceCsiApplicationInventory {
+ "retirement_generation:BIGINT,retired_at:BIGINT,recovery_protected:BOOLEAN,"
+ "resource_gc_completed:BOOLEAN,runtime_request_key:CHAR,csi_guard:BOOLEAN,"
+ "model_id:VARBINARY,model_revision:BIGINT,selected_approval_mode:VARCHAR,approval_revision:BIGINT,"
+ "plan_enabled:BOOLEAN,public_child_agents_version:INT"),
+ "plan_enabled:BOOLEAN,public_child_agents_version:INT,harness_instance_id:VARBINARY,harness_owner_boot_id:VARBINARY,"
+ "harness_placement_epoch:BIGINT,harness_placement_state:VARCHAR"),
declaration("ma_session_head", "tenant_key,harness_key",
"tenant_id:VARBINARY,workspace_id:VARBINARY,session_id:VARBINARY,storage_version:INT,state:VARCHAR,"
+ "writer_generation:BIGINT,writer_id:VARBINARY,writer_lease_until:BIGINT,lease_token_hash:CHAR,"
@@ -2975,6 +2976,9 @@ final class WorkspaceCsiApplicationInventory {
&& row.cell("selected_approval_mode").isNull() && Objects.equals(0L, row.number("approval_revision"))
&& Boolean.FALSE.equals(row.bool("plan_enabled"))
&& Objects.equals(0L, row.number("public_child_agents_version"))
&& row.cell("harness_instance_id").isNull() && row.cell("harness_owner_boot_id").isNull()
&& Objects.equals(0L, row.number("harness_placement_epoch"))
&& "UNASSIGNED".equals(row.text("harness_placement_state"))
&& original.request().requestKey().equals(row.text("runtime_request_key"))
&& workspace.equals(row.text("workspace_id")) && storage.equals(row.text("workspace_storage_id"))
&& original.request().getScope().getWorkspaceGeneration().equals(String.valueOf(row.number("workspace_generation")))
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/preflight/ManagedAgentPreflight.java
+7
View File
@@ -435,6 +435,13 @@ public final class ManagedAgentPreflight {
: "QWEN_HOSTED_HARNESS_CAPABILITY_DIGEST in "
+ file));
}
if (harness.getPool().isEnabled()) {
if (!properties.getSessionStore().isEnabled() || StringUtils.isBlank(properties.getSessionStore().getBindingKey())) {
return new Check("harness", Status.FAIL, "pool Session Store or binding key is missing", "enable session-store with a binding key");
}
return new Check("harness", Status.SKIP, "pool endpoints negotiate identity and TLS through the configured discovery source",
"verify individual Harness pool readiness after startup");
}
String baseUrl = harness.getBaseUrl();
if (StringUtils.isBlank(baseUrl)) {
return new Check("harness", Status.FAIL, "base URL is empty",
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/security/WriterCredentialPolicy.java
+58 -7
View File
@@ -3,6 +3,7 @@ package com.alibaba.qwen.code.managedagent.security;
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
import com.alibaba.qwen.code.managedagent.persistence.ManagedSessionStoreModels;
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.codec.digest.HmacAlgorithms;
import org.apache.commons.codec.digest.HmacUtils;
import org.apache.commons.lang3.StringUtils;
@@ -23,6 +24,7 @@ import java.util.Base64;
@Component
public class WriterCredentialPolicy {
private static final String TOKEN_PREFIX = "qwt1_";
private static final String PLACEMENT_PREFIX = "qwp1_";
private static final int MIN_BINDING_KEY_BYTES = 32;
private static final WriterCredentialPolicy UNBOUND =
new WriterCredentialPolicy((String) null);
@@ -65,23 +67,72 @@ public class WriterCredentialPolicy {
.encodeToString(mac(tenantId, workspaceId, sessionId));
}
public String issue(String tenantId, String workspaceId, String sessionId,
String instanceId, String bootId, long epoch) {
if (!isBound() || StringUtils.isBlank(instanceId) || StringUtils.isBlank(bootId) || epoch < 1) {
throw new IllegalStateException("A placement credential requires a binding key and a concrete owner.");
}
String owner = DigestUtils.sha256Hex(instanceId + "\0" + bootId);
String claims = epoch + "_" + owner;
return PLACEMENT_PREFIX + claims + "_" + Base64.getUrlEncoder().withoutPadding()
.encodeToString(placementMac(tenantId, workspaceId, sessionId, claims));
}
/** Rejects a presented token that is not the Session's credential. */
public void require(String tenantId, String workspaceId, String sessionId,
String writerToken) {
if (isPlacementToken(writerToken)) {
int signatureAt = writerToken.length() - 44;
if (isBound() && signatureAt > PLACEMENT_PREFIX.length()
&& writerToken.charAt(signatureAt) == '_') {
String claims = writerToken.substring(PLACEMENT_PREFIX.length(), signatureAt);
if (claims.matches("[1-9][0-9]{0,18}_[0-9a-f]{64}")) {
String expected = PLACEMENT_PREFIX + claims + "_" + Base64.getUrlEncoder().withoutPadding()
.encodeToString(placementMac(tenantId, workspaceId, sessionId, claims));
if (same(expected, writerToken)) {
return;
}
}
}
throw invalidCredential();
}
if (!isBound()) {
return;
}
boolean valid = writerToken != null && MessageDigest.isEqual(
issue(tenantId, workspaceId, sessionId)
.getBytes(StandardCharsets.US_ASCII),
writerToken.getBytes(StandardCharsets.US_ASCII));
boolean valid = same(issue(tenantId, workspaceId, sessionId), writerToken);
if (!valid) {
throw new ManagedAgentException(HttpStatus.FORBIDDEN,
ManagedSessionStoreModels.ERROR_WRITER_CREDENTIAL_INVALID,
"The writer credential is not valid for this Session.");
throw invalidCredential();
}
}
public void requirePlacement(String tenantId, String workspaceId, String sessionId,
String instanceId, String bootId, long epoch, String token) {
if (!isBound() || !same(issue(tenantId, workspaceId, sessionId, instanceId, bootId, epoch), token)) {
throw invalidCredential();
}
}
public static boolean isPlacementToken(String token) {
return token != null && token.startsWith(PLACEMENT_PREFIX);
}
private byte[] placementMac(String tenant, String workspace, String session, String claims) {
return new HmacUtils(HmacAlgorithms.HMAC_SHA_256, bindingKey)
.hmac(("qwen-managed-writer/v2\0" + tenant + "\0" + workspace + "\0" + session + "\0" + claims)
.getBytes(StandardCharsets.UTF_8));
}
private static boolean same(String expected, String presented) {
return presented != null && MessageDigest.isEqual(expected.getBytes(StandardCharsets.US_ASCII),
presented.getBytes(StandardCharsets.US_ASCII));
}
private static ManagedAgentException invalidCredential() {
return new ManagedAgentException(HttpStatus.FORBIDDEN,
ManagedSessionStoreModels.ERROR_WRITER_CREDENTIAL_INVALID,
"The writer credential is not valid for this Session.");
}
private byte[] mac(String tenantId, String workspaceId,
String sessionId) {
String canonical = "qwen-managed-writer/v1\0" + tenantId + "\0"
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/session/HarnessCoordinator.java
+19 -1
View File
@@ -8,6 +8,7 @@ import com.alibaba.qwen.code.daemon.HarnessSessionRefusedException;
import com.alibaba.qwen.code.daemon.HostedHarnessCapabilityMismatchException;
import com.alibaba.qwen.code.daemon.HostedHarnessGenerationException;
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector;
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector.Admission;
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector.Attachment;
@@ -26,6 +27,7 @@ import jakarta.annotation.PreDestroy;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@@ -66,6 +68,11 @@ public class HarnessCoordinator {
// anyway, so an entry could never change an outcome.
private static final Set<String> LEASE_BOUNDED_409_CODES =
Set.of("managed_session_writer_conflict");
// Capacity discovery and ownership handoff postpone execution; a
// waiting Turn has no failed execution attempt to charge.
private static final Set<String> POOL_OWNERSHIP_WAIT_CODES = Set.of(
"harness_owner_still_live", "harness_placement_changed",
"harness_placement_quiescing", "harness_placement_released");
// `hosted_turn_recovery_in_progress` is the ONLY refusal the daemon
// provably clears under its own recovery channel: the wake pump
// settles the park the prompt route still sees. A user Turn arriving
@@ -1554,7 +1561,7 @@ public class HarnessCoordinator {
&& current.consecutiveFailures()
>= (long) Math.max(1, budget)
* GENERATION_CHURN_BUDGET_FACTOR;
boolean exempt = exemptFromRetryBudget && !generationChurn;
boolean exempt = (exemptFromRetryBudget || isPoolWait(error)) && !generationChurn;
if (!submissionAttempted
&& !exempt
&& (chargedFailures >= maxPreAdmissionRetries
@@ -1671,6 +1678,17 @@ public class HarnessCoordinator {
return error instanceof HarnessDisabledException;
}
static boolean isPoolWait(RuntimeException error) {
if (error instanceof ManagedAgentException managed) {
return managed.getStatus() == HttpStatus.SERVICE_UNAVAILABLE
&& "harness_pool_unavailable".equals(managed.getCode())
|| managed.getStatus() == HttpStatus.CONFLICT && managed.getCode() != null
&& POOL_OWNERSHIP_WAIT_CODES.contains(managed.getCode());
}
return error instanceof DaemonHttpException http && http.getStatusCode() == 409
&& "hosted_harness_quiescing".equals(http.getErrorCode());
}
// #14387: refuse the workflow Turn's unwitnessed re-execution — mark
// it durably (the child result relay reads the code off the Turn line
// and settles the record blocked) and end the drive. The Turn itself
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/session/OwnerSubscriptions.java
+9 -10
View File
@@ -128,20 +128,19 @@ public class OwnerSubscriptions {
}
private Set<String> held() {
String writer = harness.writerIdentity();
if (writer == null) {
// No Harness client: this node writes no journal.
return Set.of();
record Owner(String writer, String tenant) {
}
Map<String, List<String>> byTenant = new HashMap<>();
Map<Owner, List<String>> groups = new HashMap<>();
for (AttachedSession session : harness.attachedSessions()) {
byTenant.computeIfAbsent(session.tenantId(),
ignored -> new ArrayList<>()).add(session.sessionId());
String writer = session.writerIdentity() == null ? harness.writerIdentity() : session.writerIdentity();
if (writer != null) {
groups.computeIfAbsent(new Owner(writer, session.tenantId()),
ignored -> new ArrayList<>()).add(session.sessionId());
}
}
Set<String> held = new HashSet<>();
for (Map.Entry<String, List<String>> tenant : byTenant.entrySet()) {
held.addAll(store.heldBy(writer, tenant.getKey(),
tenant.getValue()));
for (var group : groups.entrySet()) {
held.addAll(store.heldBy(group.getKey().writer(), group.getKey().tenant(), group.getValue()));
}
return held;
}
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/session/SessionLifecycleCoordinator.java
+12 -4
View File
@@ -317,6 +317,12 @@ public class SessionLifecycleCoordinator {
? httpError.getErrorCode() : "workspace_lifecycle_hooks_unsettled";
}
}
if (claimed.kind() != OperationKind.CWD_CHANGE && cause instanceof RuntimeException wait
&& HarnessCoordinator.isPoolWait(wait)) {
blocked = null;
daemonWaitCode = wait instanceof com.alibaba.qwen.code.managedagent.exception.ManagedAgentException apiError
? apiError.getCode() : ((DaemonHttpException) wait).getErrorCode();
}
if (blocked != null) {
failureCode = blocked;
}
@@ -808,8 +814,7 @@ public class SessionLifecycleCoordinator {
String holder = store.requireSession(operation.tenantId(),
operation.sessionId()).harnessBootId();
if (harness.isAvailable()) {
String answered = harness.closeSession(operation.tenantId(),
operation.sessionId());
String answered = harness.closeSession(operation);
// The Harness was asked and answered: even when another
// server's Harness replied (confirming nothing about the
// Session), the close was delivered — a later failure of
@@ -1266,8 +1271,11 @@ public class SessionLifecycleCoordinator {
"kind", parent.kind() == OperationKind.CLOSE
? "close" : "delete"));
}
harness.runChildOperation(parent.tenantId(), parent.sessionId(),
childBody);
if (operationIsLifecycle(parent)) {
harness.runLifecycleChildOperation(parent, childBody);
} else {
harness.runChildOperation(parent.tenantId(), parent.sessionId(), childBody);
}
}
/** The newest binding row proving a child's dispatch identity, in
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/runtimebroker/managedschema/ManagedSchemaProjection.java
+2 -1
View File
@@ -125,7 +125,8 @@ public enum ManagedSchemaProjection {
+ "session_kind, tenant_key, session_key, session_scope_key, title_revision, "
+ "retirement_operation_id, retirement_generation, retired_at, recovery_protected, "
+ "resource_gc_completed, runtime_request_key, csi_guard, model_id, model_revision, "
+ "selected_approval_mode, approval_revision, plan_enabled, public_child_agents_version"),
+ "selected_approval_mode, approval_revision, plan_enabled, public_child_agents_version, harness_instance_id, "
+ "harness_owner_boot_id, harness_placement_epoch, harness_placement_state"),
SESSION_HEAD(
"tenant_id, workspace_id, session_id, storage_version, state, writer_generation, "
+ "writer_id, writer_lease_until, lease_token_hash, journal_revision, "
packages/managed-agent-server/src/main/java/com/alibaba/qwen/code/runtimebroker/managedschema/ManagedSchemaRows.java
+2
View File
@@ -102,6 +102,8 @@ public final class ManagedSchemaRows {
"encryption_key_id",
"publish_command_id",
"harness_boot_id",
"harness_instance_id",
"harness_owner_boot_id",
"harness_event_epoch",
"workspace_storage_id",
"context_config_ref",
packages/managed-agent-server/src/main/resources/db/managed-schema/V1__managed_schema.sql
+7 -1
View File
@@ -65,9 +65,15 @@ CREATE TABLE ma_session (
plan_enabled BOOLEAN NOT NULL DEFAULT FALSE,
CONSTRAINT ck_ma_session_plan CHECK (plan_enabled = FALSE OR (selected_approval_mode IS NOT NULL AND approval_revision >= 1)),
public_child_agents_version INT NOT NULL DEFAULT 0,
CONSTRAINT ck_ma_public_child_agents CHECK (public_child_agents_version >= 0 AND public_child_agents_version <= 1)
CONSTRAINT ck_ma_public_child_agents CHECK (public_child_agents_version >= 0 AND public_child_agents_version <= 1),
harness_instance_id VARBINARY(144),
harness_owner_boot_id VARBINARY(144),
harness_placement_epoch BIGINT NOT NULL DEFAULT 0,
harness_placement_state VARCHAR(16) NOT NULL DEFAULT 'UNASSIGNED',
CONSTRAINT ck_ma_harness_placement CHECK ((harness_placement_state = 'UNASSIGNED' AND harness_instance_id IS NULL AND harness_owner_boot_id IS NULL AND harness_placement_epoch = 0) OR (harness_instance_id IS NOT NULL AND harness_owner_boot_id IS NOT NULL AND harness_placement_epoch > 0 AND (CASE harness_placement_state WHEN 'ACTIVE' THEN 6 WHEN 'QUIESCING' THEN 9 WHEN 'RELEASED' THEN 8 ELSE -1 END = CHAR_LENGTH(harness_placement_state))))
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
CREATE INDEX ix_ma_harness_owner ON ma_session (harness_instance_id, harness_owner_boot_id, harness_placement_state, tenant_id, session_id);
CREATE INDEX ix_ma_session_retirement ON ma_session (retired_at, recovery_protected);
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedExtensionProjectionContractTest.java
+2
View File
@@ -70,6 +70,8 @@ class ManagedExtensionProjectionContractTest {
// The bodies H6 and H4d added to the projection contract after H4.
expected.putAll(JSON.convertValue(
fixtures.required("additionalRecordBodies"), TreeMap.class));
// Release receipts are Store-owned; a Harness cannot submit them.
expected.put("hosted_harness_release", null);
assertEquals(JSON.convertValue(expected, TreeMap.class),
JSON.convertValue(bodies, TreeMap.class));
assertEquals(JSON.convertValue(fixtures.required("taskStates"),
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/SurfaceAdmissionAcceptanceTest.java
+8
View File
@@ -1558,6 +1558,14 @@ class SurfaceAdmissionAcceptanceTest {
private String bodyFor(SurfaceRegistry entry, String tenant,
String session) {
if (entry == SurfaceRegistry.INTERNAL_WRITER_QUIESCE_OWNERSHIP
|| entry == SurfaceRegistry.INTERNAL_WRITER_RELEASE_OWNERSHIP) {
String scope = "\"workspaceId\":\"ws\",\"writerId\":\"boot-probe\",\"writerGeneration\":1,"
+ "\"instanceId\":\"pod-probe\",\"bootId\":\"boot-probe\",\"placementEpoch\":1";
return "{" + scope + (entry == SurfaceRegistry.INTERNAL_WRITER_RELEASE_OWNERSHIP
? ",\"operationId\":\"release-probe\",\"journalRevision\":1,\"commitProof\":{\"lastCommitSequence\":1,"
+ "\"committedPrefixHash\":\"" + "a".repeat(64) + "\"}" : "") + "}";
}
Capability capability = entry.capabilities().iterator().next();
boolean publicSurface = entry.surface() == Surface.PUBLIC;
return switch (capability) {
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/api/SurfaceRegistry.java
+6
View File
@@ -411,6 +411,12 @@ public enum SurfaceRegistry {
+ "/writers:seal",
Surface.INTERNAL, RuleClass.INTERNAL_WRITER,
EnumSet.of(Capability.STORE_WRITER_SEAL)),
INTERNAL_WRITER_QUIESCE_OWNERSHIP(Method.POST,
"/internal/managed-session-store/v1/sessions/{sessionId}/writers:quiesce-ownership",
Surface.INTERNAL, RuleClass.INTERNAL_WRITER, EnumSet.of(Capability.STORE_WRITER_SEAL)),
INTERNAL_WRITER_RELEASE_OWNERSHIP(Method.POST,
"/internal/managed-session-store/v1/sessions/{sessionId}/writers:release-ownership",
Surface.INTERNAL, RuleClass.INTERNAL_WRITER, EnumSet.of(Capability.STORE_WRITER_SEAL)),
INTERNAL_RECOVERY_BLOCK(Method.POST,
"/internal/managed-session-store/v1/sessions/{sessionId}"
+ "/recovery:block",
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/config/ManagedSchemaConfigurationTest.java
+5 -1
View File
@@ -50,7 +50,11 @@ class ManagedSchemaConfigurationTest {
.containsEntry("model_id", null)
.containsEntry("model_revision", 0L)
.containsEntry("selected_approval_mode", null)
.containsEntry("approval_revision", 0L);
.containsEntry("approval_revision", 0L)
.containsEntry("harness_instance_id", null)
.containsEntry("harness_owner_boot_id", null)
.containsEntry("harness_placement_epoch", 0L)
.containsEntry("harness_placement_state", "UNASSIGNED");
assertThat(jdbc.queryForList("SELECT * FROM \"flyway_schema_history\" ORDER BY \"installed_rank\""))
.isEqualTo(history);
assertThat(jdbc.queryForList("SELECT \"version\" FROM \"flyway_schema_history\""
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/harness/HarnessPoolConnectorTest.java
+467
View File
@@ -0,0 +1,467 @@
package com.alibaba.qwen.code.managedagent.harness;
import com.alibaba.qwen.code.daemon.DaemonHttpException;
import com.alibaba.qwen.code.daemon.DaemonTransportException;
import com.alibaba.qwen.code.daemon.HostedHarnessCapabilities;
import com.alibaba.qwen.code.daemon.HostedHarnessClient;
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
import com.alibaba.qwen.code.managedagent.model.StoreModels.OperationKind;
import com.alibaba.qwen.code.managedagent.model.StoreModels.OperationRecord;
import com.alibaba.qwen.code.managedagent.persistence.HarnessPlacementStore;
import com.alibaba.qwen.code.managedagent.persistence.HarnessPlacementStore.Placement;
import com.alibaba.qwen.code.managedagent.security.WriterCredentialPolicy;
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.ValueSource;
import org.mockito.ArgumentCaptor;
import org.springframework.http.HttpStatus;
import java.net.URI;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Supplier;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
class HarnessPoolConnectorTest {
private static final String DIGEST = "sha256:" + "a".repeat(64);
@Test
void everyVerbKeepsItsPlacementAndObservationNeverCreatesOne() {
var settings = settings();
var placements = mock(HarnessPlacementStore.class);
var saved = new AtomicReference<Placement>();
when(placements.get("tenant", "session")).thenAnswer(call -> saved.get());
when(placements.assign(eq("tenant"), eq("session"), anyString(), anyString())).thenAnswer(call -> {
var placement = new Placement("tenant", "session", "workspace", call.getArgument(2), call.getArgument(3), 1, "ACTIVE");
saved.set(placement);
return placement;
});
var a = endpoint("a", 4101);
var b = endpoint("b", 4102);
var leafA = leaf();
var leafB = leaf();
var clientA = client("a");
var clientB = client("b");
HarnessDiscovery discovery = () -> new HarnessDiscovery.Snapshot(List.of(a, b), true);
try (var pool = new HarnessPoolConnector(settings, placements, discovery,
endpoint -> endpoint.instanceId().equals("a") ? leafA : leafB, null, null,
endpoint -> endpoint.instanceId().equals("a") ? clientA : clientB)) {
pool.refresh();
assertThat(pool.inspectRecovery("tenant", "session", "prompt")).isNull();
assertThat(pool.cachedRecoveryAttachment("tenant", "session")).isNull();
verify(placements, never()).assign(anyString(), anyString(), anyString(), anyString());
pool.createOrLoad("tenant", "session", false);
pool.submit("tenant", "session", "prompt", List.of(), DIGEST);
pool.cancelOriginalTurn("tenant", "session", "prompt");
pool.runAutomationOperation("tenant", "session", Map.of("kind", "fire_run"));
pool.runMessageOperation("tenant", "session", Map.of("kind", "receive"));
pool.runChannelOperation("tenant", "session", Map.of("kind", "submit_input"));
pool.runChildOperation("tenant", "session", Map.of("kind", "dispatch"));
pool.readContextCompression("tenant", "session", "operation");
verify(leafA).createOrLoad("tenant", "session", false);
verify(leafA).submit("tenant", "session", "prompt", List.of(), DIGEST);
verify(leafA).cancelOriginalTurn("tenant", "session", "prompt");
verify(leafA).runAutomationOperation("tenant", "session", Map.of("kind", "fire_run"));
verify(leafA).runMessageOperation("tenant", "session", Map.of("kind", "receive"));
verify(leafA).runChannelOperation("tenant", "session", Map.of("kind", "submit_input"));
verify(leafA).runChildOperation("tenant", "session", Map.of("kind", "dispatch"));
verify(leafA).readContextCompression("tenant", "session", "operation");
verify(leafB, never()).createOrLoad(anyString(), anyString(), eq(false));
verify(placements, never()).takeover(anyString(), anyString(), any(), anyString(), anyString());
}
}
@Test
void discoveryUncertaintyPausesPlacementWhileOriginalCachedOwnerRemainsUsable() {
var placements = mock(HarnessPlacementStore.class);
var endpoint = endpoint("a", 4101);
var leaf = leaf();
var client = client("a");
var snapshot = new AtomicReference<>(new HarnessDiscovery.Snapshot(List.of(endpoint), true));
String boot = client.capabilities().getBootId();
when(placements.get("tenant", "held")).thenReturn(new Placement("tenant", "held", "workspace", "a", boot, 2, "ACTIVE"));
try (var pool = new HarnessPoolConnector(settings(), placements, snapshot::get, ignored -> leaf,
null, null, ignored -> client)) {
pool.refresh();
snapshot.set(new HarnessDiscovery.Snapshot(List.of(endpoint), false));
pool.refresh();
pool.cancel("tenant", "held");
verify(leaf).cancel("tenant", "held");
assertThatThrownBy(() -> pool.createOrLoad("tenant", "new", false))
.hasMessageContaining("No compatible Harness owner");
verify(placements, never()).assign(anyString(), anyString(), anyString(), anyString());
}
}
@ParameterizedTest
@ValueSource(strings = {"QUIESCING", "DRAINED"})
void observedDrainMovesANeverAcquiredPlacementThroughTheStore(String state) {
var settings = settings();
var placements = mock(HarnessPlacementStore.class);
var a = endpoint("a", 4101);
var b = endpoint("b", 4102);
var leafA = leaf();
var leafB = leaf();
var clientA = client("a");
var clientB = client("b");
var first = new Placement("tenant", "session", "workspace", "a", clientA.capabilities().getBootId(), 1, "ACTIVE");
var successor = new Placement("tenant", "session", "workspace", "b", clientB.capabilities().getBootId(), 2, "ACTIVE");
var saved = new AtomicReference<>(first);
when(placements.get("tenant", "session")).thenAnswer(call -> saved.get());
when(placements.hasUnreleasedOwnership(first.instanceId(), first.bootId())).thenReturn(true);
when(placements.takeover("tenant", "session", first, "b", successor.bootId())).thenAnswer(call -> {
saved.set(successor);
return successor;
});
var quiescing = mock(DaemonHttpException.class);
when(quiescing.getStatusCode()).thenReturn(409);
when(quiescing.getErrorCode()).thenReturn("hosted_harness_quiescing");
when(leafA.createOrLoad("tenant", "session", false)).thenThrow(quiescing);
try (var pool = new HarnessPoolConnector(settings, placements,
() -> new HarnessDiscovery.Snapshot(List.of(a, b), true),
endpoint -> endpoint.instanceId().equals("a") ? leafA : leafB, null, null,
endpoint -> endpoint.instanceId().equals("a") ? clientA : clientB)) {
pool.refresh();
assertThatThrownBy(() -> pool.createOrLoad("tenant", "session", false)).isSameAs(quiescing);
when(clientA.readInstance()).thenReturn(Map.of("state", state));
pool.refresh();
for (int retry = 0; retry < 6; retry++) {
pool.createOrLoad("tenant", "session", false);
}
assertThat(saved.get()).isEqualTo(successor);
verify(placements).takeover("tenant", "session", first, "b", successor.bootId());
verify(placements, never()).assign(anyString(), anyString(), anyString(), anyString());
verify(leafA).createOrLoad("tenant", "session", false);
verify(leafB, times(6)).createOrLoad("tenant", "session", false);
var routed = ArgumentCaptor.forClass(QwenHostedHarnessConnector.PoolCall.class);
verify(leafB, times(6)).withPlacement(routed.capture(), any());
assertThat(routed.getAllValues()).allSatisfy(call -> {
assertThat(call.placement().getInstanceId()).isEqualTo("b");
assertThat(call.placement().getBootId()).isEqualTo(successor.bootId());
assertThat(call.placement().getPlacementEpoch()).isEqualTo(2);
new WriterCredentialPolicy(settings).requirePlacement("tenant", "workspace", "session",
"b", successor.bootId(), 2, call.writerToken());
});
}
}
@Test
void explicitDrainChangesPlacementEligibilityBeforeTheNextRefresh() {
var placements = mock(HarnessPlacementStore.class);
var a = endpoint("a", 4101);
var b = endpoint("b", 4102);
var leafA = leaf();
var leafB = leaf();
var clientA = client("a");
var clientB = client("b");
var first = new Placement("tenant", "session", "workspace", "a", clientA.capabilities().getBootId(), 1, "ACTIVE");
var successor = new Placement("tenant", "session", "workspace", "b", clientB.capabilities().getBootId(), 2, "ACTIVE");
when(placements.get("tenant", "session")).thenReturn(first);
when(placements.takeover("tenant", "session", first, "b", successor.bootId())).thenReturn(successor);
String operation = UUID.randomUUID().toString();
when(clientA.drainInstance(operation)).thenReturn(Map.of("state", "DRAINED"));
try (var pool = new HarnessPoolConnector(settings(), placements,
() -> new HarnessDiscovery.Snapshot(List.of(a, b), true),
endpoint -> endpoint.instanceId().equals("a") ? leafA : leafB, null, null,
endpoint -> endpoint.instanceId().equals("a") ? clientA : clientB)) {
pool.refresh();
pool.drainInstance("a", operation);
pool.createOrLoad("tenant", "session", false);
verify(placements).takeover("tenant", "session", first, "b", successor.bootId());
verify(leafA, never()).createOrLoad(anyString(), anyString(), eq(false));
verify(leafB).createOrLoad("tenant", "session", false);
}
}
@ParameterizedTest
@CsvSource({"ACTIVE,true,harness_owner_still_live,true", "ACTIVE,true,harness_placement_changed,false",
"QUIESCING,true,harness_owner_still_live,false", "ACTIVE,false,harness_owner_still_live,false"})
void onlyAReachableActiveLiveOwnerKeepsItsCompletionCallsDuringDrain(
String state, boolean reachable, String code, boolean keepsOwner) {
var placements = mock(HarnessPlacementStore.class);
var a = endpoint("a", 4101);
var b = endpoint("b", 4102);
var leafA = leaf();
var leafB = leaf();
var clientA = client("a");
var clientB = client("b");
var owner = new Placement("tenant", "session", "workspace", "a", clientA.capabilities().getBootId(), 1, state);
when(placements.get("tenant", "session")).thenReturn(owner);
when(placements.hasUnreleasedOwnership(owner.instanceId(), owner.bootId())).thenReturn(true);
var refusal = new ManagedAgentException(HttpStatus.CONFLICT, code, "Writer fence refused takeover");
when(placements.takeover("tenant", "session", owner, "b", clientB.capabilities().getBootId())).thenThrow(refusal);
try (var pool = new HarnessPoolConnector(settings(), placements,
() -> new HarnessDiscovery.Snapshot(List.of(a, b), true),
endpoint -> endpoint.instanceId().equals("a") ? leafA : leafB, null, null,
endpoint -> endpoint.instanceId().equals("a") ? clientA : clientB)) {
pool.refresh();
if (reachable) {
when(clientA.readInstance()).thenReturn(Map.of("state", "QUIESCING"));
} else {
when(clientA.readInstance()).thenThrow(mock(DaemonTransportException.class));
}
pool.refresh();
if (keepsOwner) {
var response = JsonNodeFactory.instance.objectNode();
pool.stream("tenant", "session", 0, "cursor");
pool.resolveAction("tenant", "session", "action", response);
pool.cancel("tenant", "session");
verify(leafA).stream("tenant", "session", 0, "cursor");
verify(leafA).resolveAction("tenant", "session", "action", response);
verify(leafA).cancel("tenant", "session");
var routed = ArgumentCaptor.forClass(QwenHostedHarnessConnector.PoolCall.class);
verify(leafA, times(3)).withPlacement(routed.capture(), any());
assertThat(routed.getAllValues()).allSatisfy(call -> {
assertThat(call.placement().getInstanceId()).isEqualTo("a");
assertThat(call.placement().getBootId()).isEqualTo(owner.bootId());
assertThat(call.placement().getPlacementEpoch()).isEqualTo(owner.epoch());
});
} else {
assertThatThrownBy(() -> pool.stream("tenant", "session", 0, "cursor")).isSameAs(refusal);
verify(leafA, never()).stream(anyString(), anyString(), eq(0L), anyString());
}
verify(leafB, never()).stream(anyString(), anyString(), eq(0L), anyString());
verify(leafB, never()).resolveAction(anyString(), anyString(), anyString(), any());
verify(leafB, never()).cancel(anyString(), anyString());
verify(placements, never()).assign(anyString(), anyString(), anyString(), anyString());
}
}
@ParameterizedTest
@ValueSource(booleans = {true, false})
void aKnownUnreachableOwnerWaitsForCapacityWithoutFurtherRpcs(boolean refreshFailure) {
var placements = mock(HarnessPlacementStore.class);
var a = endpoint("a", 4101);
var b = endpoint("b", 4102);
var leafA = leaf();
var leafB = leaf();
var clientA = client("a");
var clientB = client("b");
var first = new Placement("tenant", "session", "workspace", "a", clientA.capabilities().getBootId(), 1, "ACTIVE");
var successor = new Placement("tenant", "session", "workspace", "b", clientB.capabilities().getBootId(), 2, "ACTIVE");
var saved = new AtomicReference<>(first);
var snapshot = new AtomicReference<>(new HarnessDiscovery.Snapshot(List.of(a), true));
when(placements.get("tenant", "session")).thenAnswer(call -> saved.get());
when(placements.hasUnreleasedOwnership(first.instanceId(), first.bootId())).thenReturn(true);
when(placements.takeover("tenant", "session", first, "b", successor.bootId())).thenAnswer(call -> {
saved.set(successor);
return successor;
});
var transport = mock(DaemonTransportException.class);
try (var pool = new HarnessPoolConnector(settings(), placements, snapshot::get,
endpoint -> endpoint.instanceId().equals("a") ? leafA : leafB, null, null,
endpoint -> endpoint.instanceId().equals("a") ? clientA : clientB)) {
pool.refresh();
if (refreshFailure) {
when(clientA.readInstance()).thenThrow(transport);
pool.refresh();
} else {
when(leafA.createOrLoad("tenant", "session", false)).thenThrow(transport);
assertThatThrownBy(() -> pool.createOrLoad("tenant", "session", false)).isSameAs(transport);
}
for (int retry = 0; retry < 6; retry++) {
assertThatThrownBy(() -> pool.createOrLoad("tenant", "session", false))
.isInstanceOfSatisfying(ManagedAgentException.class, error -> {
assertThat(error.getStatus()).isEqualTo(HttpStatus.SERVICE_UNAVAILABLE);
assertThat(error.getCode()).isEqualTo("harness_pool_unavailable");
});
}
assertThat(saved.get()).isEqualTo(first);
verify(leafA, times(refreshFailure ? 0 : 1)).createOrLoad("tenant", "session", false);
verify(placements, never()).assign(anyString(), anyString(), anyString(), anyString());
verify(placements, never()).takeover(anyString(), anyString(), any(), anyString(), anyString());
doThrow(transport).when(clientA).readInstance();
snapshot.set(new HarnessDiscovery.Snapshot(List.of(a, b), true));
pool.refresh();
pool.createOrLoad("tenant", "session", false);
assertThat(saved.get()).isEqualTo(successor);
verify(placements).takeover("tenant", "session", first, "b", successor.bootId());
verify(leafB).createOrLoad("tenant", "session", false);
}
}
@Test
void ownershipReleaseNeverReassignsOrTargetsASuccessorEpoch() {
var placements = mock(HarnessPlacementStore.class);
var endpoint = endpoint("a", 4101);
var leaf = leaf();
var client = client("a");
String boot = client.capabilities().getBootId();
var saved = new AtomicReference<>(new Placement("tenant", "session", "workspace", "a", boot, 2, "QUIESCING"));
when(placements.get("tenant", "session")).thenAnswer(call -> saved.get());
when(placements.releaseReceipt("tenant", "session", 2, "operation")).thenReturn(null);
try (var pool = new HarnessPoolConnector(settings(), placements,
() -> new HarnessDiscovery.Snapshot(List.of(endpoint), true), ignored -> leaf,
null, null, ignored -> client)) {
pool.refresh();
pool.releaseOwnership("tenant", "session", "operation", 2);
verify(leaf).releaseOwnership("tenant", "session", "operation", 2);
saved.set(new Placement("tenant", "session", "workspace", "a", boot, 3, "ACTIVE"));
assertThatThrownBy(() -> pool.releaseOwnership("tenant", "session", "operation", 2))
.hasMessageContaining("Original Harness placement");
var receipt = Map.<String, Object>of("operationId", "operation", "placementEpoch", 2L, "state", "RELEASED");
when(placements.releaseReceipt("tenant", "session", 2, "operation")).thenReturn(receipt);
assertThat(pool.releaseOwnership("tenant", "session", "operation", 2)).isEqualTo(receipt);
verify(placements, never()).assign(anyString(), anyString(), anyString(), anyString());
verify(placements, never()).takeover(anyString(), anyString(), any(), anyString(), anyString());
verify(leaf, never()).releaseOwnership("tenant", "session", "operation", 3);
}
}
@Test
void releasedPlacementRequiresStoreCasBeforeSameBootReuse() {
var placements = mock(HarnessPlacementStore.class);
var endpoint = endpoint("a", 4101);
var leaf = leaf();
var client = client("a");
String boot = client.capabilities().getBootId();
var released = new Placement("tenant", "session", "workspace", "a", boot, 2, "RELEASED");
var successor = new Placement("tenant", "session", "workspace", "a", boot, 3, "ACTIVE");
when(placements.get("tenant", "session")).thenReturn(released);
when(placements.takeover("tenant", "session", released, "a", boot)).thenReturn(successor);
try (var pool = new HarnessPoolConnector(settings(), placements,
() -> new HarnessDiscovery.Snapshot(List.of(endpoint), true), ignored -> leaf,
null, null, ignored -> client)) {
pool.refresh();
pool.createOrLoad("tenant", "session", true);
verify(placements).takeover("tenant", "session", released, "a", boot);
verify(leaf).createOrLoad("tenant", "session", true);
verify(placements, never()).assign(anyString(), anyString(), anyString(), anyString());
}
}
@Test
void lifecycleHandoffUsesCanonicalOperationInsteadOfOrdinaryPlacementAdmission() {
var placements = mock(HarnessPlacementStore.class);
var endpoint = endpoint("a", 4101);
var leaf = leaf();
var client = client("a");
String boot = client.capabilities().getBootId();
var released = new Placement("tenant", "session", "workspace", "a", boot, 2, "RELEASED");
var successor = new Placement("tenant", "session", "workspace", "a", boot, 3, "ACTIVE");
var operation = new OperationRecord("tenant", "session", "close", OperationKind.CLOSE, "request", "PENDING",
"ADMITTED", "LEASED", "ACTIVE", null, "worker", 7, 0, null, 1);
when(placements.get("tenant", "session")).thenReturn(released);
when(placements.takeoverForLifecycle(operation, released, "a", boot)).thenReturn(successor);
try (var pool = new HarnessPoolConnector(settings(), placements,
() -> new HarnessDiscovery.Snapshot(List.of(endpoint), true), ignored -> leaf,
null, null, ignored -> client)) {
pool.refresh();
pool.settleLifecycle(operation);
verify(placements).takeoverForLifecycle(operation, released, "a", boot);
verify(leaf).settleLifecycle(operation);
verify(placements, never()).takeover(anyString(), anyString(), any(), anyString(), anyString());
verify(placements, never()).assign(anyString(), anyString(), anyString(), anyString());
}
}
@Test
void shutdownClosesAClientRegisteredAfterItsInitialClosedCheck() throws Exception {
var placements = mock(HarnessPlacementStore.class);
var endpoint = endpoint("a", 4101);
var leaf = leaf();
var entered = new CountDownLatch(1);
var continueRegistration = new CountDownLatch(1);
var pool = new HarnessPoolConnector(settings(), placements,
() -> new HarnessDiscovery.Snapshot(List.of(endpoint), true), ignored -> {
entered.countDown();
try {
if (!continueRegistration.await(5, TimeUnit.SECONDS)) {
throw new IllegalStateException("Timed out waiting for shutdown");
}
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
throw new IllegalStateException(interrupted);
}
return leaf;
}, null, null, ignored -> client("a"));
var refresh = new Thread(pool::refresh);
refresh.start();
try {
assertThat(entered.await(5, TimeUnit.SECONDS)).isTrue();
pool.close();
continueRegistration.countDown();
refresh.join(5_000);
assertThat(refresh.isAlive()).isFalse();
verify(leaf).close();
assertThatThrownBy(() -> pool.createOrLoad("tenant", "session", false))
.hasMessageContaining("No compatible Harness owner");
verify(placements, never()).assign(anyString(), anyString(), anyString(), anyString());
} finally {
continueRegistration.countDown();
pool.close();
}
}
@Test
void rejectsUnboundPoolAndRemotePlaintextEvenWhenLegacyOverrideIsAllowed() {
var settings = settings();
settings.getSessionStore().setBindingKey("");
assertThatThrownBy(() -> new HarnessPoolConnector(settings, mock(HarnessPlacementStore.class),
() -> new HarnessDiscovery.Snapshot(List.of(), true), ignored -> leaf(), null, null, ignored -> client("a")))
.hasMessageContaining("binding key");
settings.getSessionStore().setBindingKey("binding-key-".repeat(4));
settings.getSessionStore().setEnabled(false);
assertThatThrownBy(() -> new HarnessPoolConnector(settings, mock(HarnessPlacementStore.class),
() -> new HarnessDiscovery.Snapshot(List.of(), true), ignored -> leaf(), null, null, ignored -> client("a")))
.hasMessageContaining("enabled Session Store");
assertThatThrownBy(() -> HarnessPoolConnector.validateEndpoint(
new HarnessDiscovery.Endpoint("remote", URI.create("http://10.0.0.2:4170"), true), null))
.hasMessageContaining("mTLS");
assertThatThrownBy(() -> HarnessPoolConnector.validateEndpoint(
new HarnessDiscovery.Endpoint("remote", URI.create("https://10.0.0.2:4170"), true), null))
.hasMessageContaining("mTLS");
}
private static QwenHostedHarnessConnector leaf() {
var leaf = mock(QwenHostedHarnessConnector.class);
doAnswer(call -> ((Supplier<?>) call.getArgument(1)).get()).when(leaf).withPlacement(any(), any());
when(leaf.attachedSessions()).thenReturn(List.of());
return leaf;
}
private static HarnessDiscovery.Endpoint endpoint(String id, int port) {
return new HarnessDiscovery.Endpoint(id, URI.create("http://127.0.0.1:" + port), true);
}
private static HostedHarnessClient client(String id) {
var client = mock(HostedHarnessClient.class);
var capabilities = mock(HostedHarnessCapabilities.class);
when(client.capabilities()).thenReturn(capabilities);
when(capabilities.getBootId()).thenReturn(UUID.randomUUID().toString());
when(capabilities.getInstanceId()).thenReturn(id);
when(capabilities.getFeatures()).thenReturn(List.of("harness-pool-owner/1", "harness-instance-lifecycle/1"));
when(capabilities.getLifecycleV2Profiles()).thenReturn(List.of());
when(client.readInstance()).thenReturn(Map.of("state", "ACCEPTING"));
return client;
}
private static ManagedAgentProperties settings() {
var properties = new ManagedAgentProperties();
properties.getHarness().setToken("test-token");
properties.getHarness().setCapabilityDigest(DIGEST);
properties.getSessionStore().setBindingKey("binding-key-".repeat(4));
properties.getSessionStore().setEnabled(true);
properties.getSessionStore().setWorkspaceId("workspace");
return properties;
}
}
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/harness/HostedHarnessPoolMySqlIT.java
+256
View File
@@ -0,0 +1,256 @@
package com.alibaba.qwen.code.managedagent.harness;
import com.alibaba.qwen.code.daemon.HostedHarnessClient;
import com.alibaba.qwen.code.managedagent.ManagedAgentServerApplication;
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
import com.alibaba.qwen.code.managedagent.persistence.HarnessPlacementStore;
import com.alibaba.qwen.code.managedagent.persistence.ManagedSessionStore;
import com.alibaba.qwen.code.managedagent.security.WriterCredentialPolicy;
import com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Timeout;
import org.junit.jupiter.api.io.TempDir;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext;
import org.springframework.jdbc.core.JdbcTemplate;
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
import java.util.function.BooleanSupplier;
import java.util.regex.Pattern;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class HostedHarnessPoolMySqlIT {
private static final String TOKEN = "hosted-pool-local-fixture-token";
private static final String BINDING_KEY = "pool-binding-key-0123456789abcdef0123456789abcdef";
private static final String WORKSPACE = "hosted-pool-fixture-workspace";
private static final String DIGEST = "sha256:" + "a".repeat(64);
private static final Pattern LISTENING = Pattern.compile("listening on http://127\\.0\\.0\\.1:(\\d+)");
private final ObjectMapper json = new ObjectMapper();
private final HttpClient http = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(3)).build();
private final List<Process> children = new ArrayList<>();
private final List<HostedHarnessClient> clients = new ArrayList<>();
@TempDir
private Path temporary;
private ServletWebServerApplicationContext spring;
private HarnessPoolConnector pool;
private JdbcTemplate jdbc;
@Test
@Timeout(120)
void realProcessesKeepTenantPlacementAndReleaseIdleOwnersWithoutCancellingSessions() throws Exception {
Path cli = Path.of(required("qwen.cli.entry")).toAbsolutePath();
assertThat(cli).isRegularFile();
URI first = startHarness(cli, "pool-a");
URI second = startHarness(cli, "pool-b");
spring = (ServletWebServerApplicationContext) new SpringApplicationBuilder(ManagedAgentServerApplication.class).run(
"--server.address=127.0.0.1", "--server.port=0",
"--spring.datasource.url=" + required("mysql.url"),
"--spring.datasource.username=" + required("mysql.user"),
"--spring.datasource.password=" + System.getProperty("mysql.password", ""),
"--spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver",
"--spring.datasource.druid.max-active=4", "--spring.datasource.druid.max-wait=5000",
"--spring.datasource.druid.connect-properties.connectTimeout=5000",
"--spring.datasource.druid.connect-properties.socketTimeout=10000",
"--qwen.managed-agent.session-store.enabled=true",
"--qwen.managed-agent.session-store.workspace-id=" + WORKSPACE,
"--qwen.managed-agent.session-store.binding-key=" + BINDING_KEY,
"--qwen.managed-agent.harness.enabled=true", "--qwen.managed-agent.runtime-broker.enabled=false",
"--qwen.managed-agent.harness.token=" + TOKEN,
"--qwen.managed-agent.harness.capability-digest=" + DIGEST,
"--qwen.managed-agent.harness.heartbeat-interval=PT0S",
"--qwen.managed-agent.harness.pool.enabled=true",
"--qwen.managed-agent.harness.pool.instances[0].instance-id=pool-a",
"--qwen.managed-agent.harness.pool.instances[0].base-url=" + first,
"--qwen.managed-agent.harness.pool.instances[1].instance-id=pool-b",
"--qwen.managed-agent.harness.pool.instances[1].base-url=" + second);
jdbc = spring.getBean(JdbcTemplate.class);
assertThat(jdbc.queryForObject("SELECT VERSION()", String.class)).startsWith("8.4.");
var settings = spring.getBean(ManagedAgentProperties.class);
settings.getSessionStore().setBaseUrl("http://127.0.0.1:" + spring.getWebServer().getPort());
var placements = spring.getBean(HarnessPlacementStore.class);
pool = spring.getBean(HarnessPoolConnector.class);
pool.refresh();
assertThat(pool.supportsFencedTitles()).isTrue();
String tenantA = "pool-a-" + UUID.randomUUID();
String tenantB = "pool-b-" + UUID.randomUUID();
String a1 = root(tenantA);
String b1 = root(tenantB);
String b2 = root(tenantB);
String a2 = root(tenantA);
var attachedA = pool.createOrLoad(tenantA, a1, false);
pool.createOrLoad(tenantB, b1, false);
pool.createOrLoad(tenantB, b2, false);
pool.createOrLoad(tenantA, a2, false);
var original = placements.get(tenantA, a1);
assertThat(original.instanceId()).isEqualTo("pool-a");
assertThat(Set.of(placements.get(tenantA, a1).instanceId(), placements.get(tenantA, a2).instanceId()))
.containsExactlyInAnyOrder("pool-a", "pool-b");
assertThat(Set.of(placements.get(tenantB, b1).instanceId(), placements.get(tenantB, b2).instanceId()))
.containsExactlyInAnyOrder("pool-a", "pool-b");
assertThat(pool.createOrLoad(tenantA, a1, true)).isEqualTo(attachedA);
assertThat(placements.get(tenantA, a1)).isEqualTo(original);
assertThat(pool.attachedSessions()).hasSize(4);
assertThatThrownBy(() -> pool.createOrLoad(tenantB, a1, true)).hasMessageContaining("Session does not exist");
assertThat(placements.get(tenantB, a1)).isNull();
var credentials = new WriterCredentialPolicy(settings);
String oldGrant = credentials.issue(tenantA, WORKSPACE, a1, original.instanceId(), original.bootId(), original.epoch());
HostedHarnessClient firstClient = client(first);
assertThat(firstClient.readInstance().get("state")).isEqualTo("ACCEPTING");
pool.drainInstance("pool-a", UUID.randomUUID().toString());
await(() -> "DRAINED".equals(firstClient.readInstance().get("state")), 30);
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_session_record WHERE tenant_id IN (?, ?)"
+ " AND domain = 'hosted_harness_release'", Integer.class, tenantA, tenantB)).isEqualTo(2);
assertThat(jdbc.queryForObject("SELECT state FROM ma_session_head WHERE tenant_id = ? AND session_id = ?"
+ " AND head_kind = 'MANAGED_SESSION'", String.class, tenantA, a1)).isEqualTo("SEALED");
assertThat(staleAcquire(tenantA, a1, original.bootId(), oldGrant).path("error").path("code").asText())
.isEqualTo("harness_placement_released");
pool.refresh();
var recovered = pool.createOrLoad(tenantA, a1, true);
var successor = placements.get(tenantA, a1);
assertThat(successor.epoch()).isEqualTo(original.epoch() + 1);
assertThat(successor.instanceId()).isEqualTo("pool-b");
assertThat(recovered.bootId()).isNotEqualTo(original.bootId());
assertThat(staleAcquire(tenantA, a1, original.bootId(), oldGrant).path("error").path("code").asText())
.isEqualTo("writer_credential_invalid");
assertThat(client(second).readInstance().get("state")).isEqualTo("ACCEPTING");
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_session WHERE tenant_id IN (?, ?) AND status = 'ACTIVE'",
Integer.class, tenantA, tenantB)).isEqualTo(4);
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_operation WHERE tenant_id IN (?, ?)",
Integer.class, tenantA, tenantB)).isZero();
}
private String root(String tenant) {
String id = UUID.randomUUID().toString();
jdbc.update("INSERT INTO ma_session (tenant_id, session_id, agent_id, tenant_key, session_key, session_scope_key,"
+ " gmt_create, gmt_modified) VALUES (?, ?, 'qwen-code', ?, ?, ?, 1, 1)", tenant, id,
JdbcRuntimeBindingRepository.harnessDrainKey(tenant), JdbcRuntimeBindingRepository.harnessDrainKey(id),
ManagedSessionStore.sessionScopeKey(tenant, id));
jdbc.update("INSERT INTO ma_event_stream (tenant_id, session_id, stream_kind, stream_id, gmt_modified)"
+ " VALUES (?, ?, 'SESSION', X'', 1)", tenant, id);
return id;
}
private JsonNode staleAcquire(String tenant, String session, String boot, String grant) throws Exception {
var request = HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + spring.getWebServer().getPort()
+ "/internal/managed-session-store/v1/sessions/" + session + "/writers:acquire"))
.timeout(Duration.ofSeconds(10)).header("Content-Type", "application/json")
.header("X-Qwen-Tenant-Id", tenant).header("X-Qwen-Managed-Writer-Token", grant)
.POST(HttpRequest.BodyPublishers.ofString(json.writeValueAsString(
Map.of("workspaceId", WORKSPACE, "writerId", boot, "leaseMillis", 60_000)))).build();
var response = http.send(request, HttpResponse.BodyHandlers.ofString());
assertThat(response.statusCode()).as("Old writer grant must be rejected").isIn(403, 409);
return json.readTree(response.body());
}
private HostedHarnessClient client(URI base) {
var client = HostedHarnessClient.builder().baseUri(base).bearerToken(TOKEN).capabilityDigest(DIGEST)
.connectTimeout(Duration.ofSeconds(3)).requestTimeout(Duration.ofSeconds(10))
.heartbeatInterval(Duration.ZERO).build();
clients.add(client);
return client;
}
private URI startHarness(Path cli, String id) throws Exception {
Path dir = Files.createDirectories(temporary.resolve(id));
Path config = Files.createDirectory(dir.resolve(".qwen"));
String model = "pool-idle-fixture";
Files.writeString(config.resolve("settings.json"), json.writeValueAsString(Map.of(
"security", Map.of("auth", Map.of("selectedType", "openai")),
"model", Map.of("name", model), "telemetry", Map.of("enabled", false),
"modelProviders", Map.of("openai", List.of(Map.of("id", model,
"envKey", "OPENAI_API_KEY", "baseUrl", "http://127.0.0.1:9/v1"))))));
Path log = dir.resolve("process.log");
var builder = new ProcessBuilder(System.getProperty("qwen.node", "node"), cli.toString(), "serve",
"--profile", "hosted-harness", "--http-bridge", "--no-web", "--hostname", "127.0.0.1",
"--port", "0", "--workspace", dir.toString()).directory(dir.toFile()).redirectErrorStream(true)
.redirectOutput(log.toFile());
var environment = builder.environment();
environment.clear();
if (System.getenv("PATH") != null) {
environment.put("PATH", System.getenv("PATH"));
}
environment.putAll(Map.of("HOME", dir.toString(), "QWEN_HOME", config.toString(),
"QWEN_RUNTIME_DIR", dir.resolve("runtime").toString(), "OPENAI_API_KEY", "test-fixture-key",
"OPENAI_MODEL", model, "OPENAI_BASE_URL", "http://127.0.0.1:9/v1", "QWEN_SERVER_TOKEN", TOKEN,
"QWEN_HOSTED_HARNESS_CAPABILITY_DIGEST", DIGEST, "QWEN_HOSTED_HARNESS_INSTANCE_ID", id,
"QWEN_CODE_MODELS_DEV_REFRESH", "off"));
environment.put("QWEN_CODE_SYSTEM_SETTINGS_PATH", dir.resolve("system-settings.json").toString());
environment.put("QWEN_CODE_SYSTEM_DEFAULTS_PATH", dir.resolve("system-defaults.json").toString());
Process process = builder.start();
children.add(process);
URI[] base = new URI[1];
await(() -> {
try {
String logs = Files.readString(log);
assertThat(process.isAlive()).as("Harness exited: %s", logs).isTrue();
var match = LISTENING.matcher(logs);
if (!match.find()) {
return false;
}
base[0] = URI.create("http://127.0.0.1:" + match.group(1));
return http.send(HttpRequest.newBuilder(base[0].resolve("/capabilities"))
.timeout(Duration.ofSeconds(2)).header("Authorization", "Bearer " + TOKEN).build(),
HttpResponse.BodyHandlers.discarding()).statusCode() == 200;
} catch (IOException unavailable) {
return false;
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
throw new IllegalStateException(interrupted);
}
}, 30);
return base[0];
}
private static void await(BooleanSupplier done, int seconds) throws InterruptedException {
long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(seconds);
while (!done.getAsBoolean()) {
assertThat(System.nanoTime()).as("Timed out waiting for the real process").isLessThan(end);
Thread.sleep(100);
}
}
@AfterEach
void cleanup() throws Exception {
if (pool != null) {
pool.close();
}
clients.forEach(HostedHarnessClient::close);
for (var child : children) {
child.descendants().forEach(ProcessHandle::destroyForcibly);
child.destroy();
if (!child.waitFor(5, TimeUnit.SECONDS)) {
child.destroyForcibly();
}
assertThat(child.waitFor(5, TimeUnit.SECONDS)).isTrue();
}
if (spring != null) {
spring.close();
}
}
private static String required(String key) {
String value = System.getProperty(key);
if (value == null || value.isBlank()) {
throw new IllegalStateException("Required integration prerequisite: -D" + key);
}
return value;
}
}
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/harness/KubernetesHarnessDiscoveryTest.java
+147
View File
@@ -0,0 +1,147 @@
package com.alibaba.qwen.code.managedagent.harness;
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.sun.net.httpserver.HttpServer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class KubernetesHarnessDiscoveryTest {
private final ObjectMapper json = new ObjectMapper();
@Test
void typedListMembersMayOmitTypeMetadata() throws Exception {
var listed = (ObjectNode) json.readTree(slice("slice", "10.0.0.2", true));
listed.remove(List.of("apiVersion", "kind"));
var endpoints = KubernetesHarnessDiscovery.endpoints(settings(), List.of(listed));
assertThat(endpoints).hasSize(1);
assertThat(endpoints.getFirst().instanceId()).isEqualTo("pod-uid");
assertThat(endpoints.getFirst().accepting()).isTrue();
listed.put("kind", "Endpoints");
assertThatThrownBy(() -> KubernetesHarnessDiscovery.endpoints(settings(), List.of(listed)))
.hasMessageContaining("scope differs");
}
@Test
void deduplicatesPodIdentityAndRetainsNonReadyOwnersForDirectRouting() throws Exception {
var settings = settings();
var first = json.readTree(slice("uid-a", "10.0.0.2", true));
var duplicate = json.readTree(slice("uid-b", "10.0.0.2", true));
var endpoints = KubernetesHarnessDiscovery.endpoints(settings, List.of(first, duplicate));
assertThat(endpoints).hasSize(1);
assertThat(endpoints.getFirst().instanceId()).isEqualTo("pod-uid");
assertThat(endpoints.getFirst().baseUri()).isEqualTo(URI.create("https://10.0.0.2:8443"));
assertThat(endpoints.getFirst().accepting()).isTrue();
var terminating = json.readTree(slice("uid-a", "10.0.0.2", false));
endpoints = KubernetesHarnessDiscovery.endpoints(settings, List.of(terminating));
assertThat(endpoints).hasSize(1);
assertThat(endpoints.getFirst().accepting()).isFalse();
endpoints = KubernetesHarnessDiscovery.endpoints(settings, List.of(first, terminating));
assertThat(endpoints.getFirst().accepting()).isFalse();
var conflictingPort = json.readTree(slice("uid-b", "10.0.0.3", true).replace("8443", "9443"));
assertThatThrownBy(() -> KubernetesHarnessDiscovery.endpoints(settings, List.of(first, conflictingPort)))
.hasMessageContaining("Conflicting Harness port");
}
@Test
void refusesForeignServiceAndMissingPodTargetRatherThanSelectingAnAddress() throws Exception {
var foreign = json.readTree(slice("slice", "10.0.0.2", true).replace("service-uid", "foreign-uid"));
assertThatThrownBy(() -> KubernetesHarnessDiscovery.endpoints(settings(), List.of(foreign)))
.hasMessageContaining("owner differs");
var targetless = json.readTree(slice("slice", "10.0.0.2", true).replace("\"Pod\"", "\"Node\""));
assertThatThrownBy(() -> KubernetesHarnessDiscovery.endpoints(settings(), List.of(targetless)))
.hasMessageContaining("Pod target");
var futureVersion = json.readTree(slice("slice", "10.0.0.2", true)
.replace("\"kind\":\"Pod\"", "\"apiVersion\":\"v2\",\"kind\":\"Pod\""));
assertThatThrownBy(() -> KubernetesHarnessDiscovery.endpoints(settings(), List.of(futureVersion)))
.hasMessageContaining("Pod target");
}
@Test
void expiredWatchRelistsAndServiceRecreationPausesNewAssignments(@TempDir Path directory) throws Exception {
var settings = settings();
Path token = directory.resolve("token");
Files.writeString(token, "first-token");
settings.setKubernetesTokenFile(token.toString());
var lists = new AtomicInteger();
var serviceReads = new AtomicInteger();
var watchStarted = new CountDownLatch(1);
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/api/v1/namespaces/test/services/harness", exchange -> {
String uid = serviceReads.incrementAndGet() == 1 ? "service-uid" : "recreated-service";
byte[] response = ("{\"kind\":\"Service\",\"metadata\":{\"namespace\":\"test\",\"name\":\"harness\",\"uid\":\"" + uid + "\"}}")
.getBytes(StandardCharsets.UTF_8);
exchange.sendResponseHeaders(200, response.length);
exchange.getResponseBody().write(response);
exchange.close();
});
server.createContext("/apis/discovery.k8s.io/v1/namespaces/test/endpointslices", exchange -> {
boolean watch = exchange.getRequestURI().getRawQuery().contains("watch=true");
String body;
if (watch) {
watchStarted.countDown();
body = "{\"kind\":\"Status\",\"code\":410}";
} else {
lists.incrementAndGet();
body = "{\"kind\":\"EndpointSliceList\",\"metadata\":{\"resourceVersion\":\"10\"},\"items\":["
+ slice("slice", "10.0.0.2", true) + "]}";
}
byte[] response = body.getBytes(StandardCharsets.UTF_8);
exchange.sendResponseHeaders(watch ? 410 : 200, response.length);
exchange.getResponseBody().write(response);
exchange.close();
});
server.start();
try (var discovery = new KubernetesHarnessDiscovery(settings, json,
URI.create("http://127.0.0.1:" + server.getAddress().getPort()),
HttpClient.newBuilder().followRedirects(HttpClient.Redirect.NEVER).build())) {
assertThat(watchStarted.await(3, TimeUnit.SECONDS)).isTrue();
long until = System.nanoTime() + Duration.ofSeconds(3).toNanos();
while ((serviceReads.get() < 2 || discovery.snapshot().current()) && System.nanoTime() < until) {
Thread.sleep(5);
}
assertThat(serviceReads.get()).isGreaterThanOrEqualTo(2);
assertThat(lists.get()).isEqualTo(1);
assertThat(discovery.snapshot().current()).isFalse();
assertThat(discovery.snapshot().endpoints()).hasSize(1);
} finally {
server.stop(0);
}
}
private static ManagedAgentProperties.HarnessPool settings() {
var properties = new ManagedAgentProperties.HarnessPool();
properties.setNamespace("test");
properties.setServiceName("harness");
properties.setServiceUid("service-uid");
return properties;
}
private static String slice(String id, String address, boolean ready) {
return """
{"apiVersion":"discovery.k8s.io/v1","kind":"EndpointSlice","addressType":"IPv4",
"metadata":{"uid":"%s","resourceVersion":"10","namespace":"test",
"labels":{"kubernetes.io/service-name":"harness"},"ownerReferences":[
{"apiVersion":"v1","kind":"Service","name":"harness","uid":"service-uid","controller":true}]},
"ports":[{"name":"https","protocol":"TCP","port":8443}],
"endpoints":[{"addresses":["%s"],"conditions":{"ready":%s},
"targetRef":{"kind":"Pod","namespace":"test","name":"harness-a","uid":"pod-uid"}}]}
""".formatted(id, address, ready);
}
}
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/persistence/HarnessPlacementMySqlIT.java
+23
View File
@@ -0,0 +1,23 @@
package com.alibaba.qwen.code.managedagent.persistence;
import org.junit.jupiter.api.AfterEach;
import javax.sql.DataSource;
/** Runs the placement/grant/release cases against the real canonical MySQL schema. */
public class HarnessPlacementMySqlIT extends HarnessPlacementStoreTest {
private IsolatedManagedSchemaMySqlTestDatabase database;
@Override
protected DataSource source() {
database = new IsolatedManagedSchemaMySqlTestDatabase();
return database.source();
}
@AfterEach
void cleanup() {
if (database != null) {
database.close();
}
}
}
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/persistence/HarnessPlacementStoreTest.java
+319
View File
@@ -0,0 +1,319 @@
package com.alibaba.qwen.code.managedagent.persistence;
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
import com.alibaba.qwen.code.managedagent.model.StoreModels.OperationKind;
import com.alibaba.qwen.code.managedagent.model.StoreModels.OperationRecord;
import com.alibaba.qwen.code.managedagent.security.WriterCredentialPolicy;
import com.alibaba.qwen.code.runtimebroker.managedschema.ManagedSchema;
import org.apache.commons.codec.digest.DigestUtils;
import org.h2.jdbcx.JdbcDataSource;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import org.springframework.dao.DataAccessException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
import javax.sql.DataSource;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
public class HarnessPlacementStoreTest {
protected JdbcTemplate jdbc;
protected TransactionTemplate tx;
protected HarnessPlacementStore placements;
protected ManagedSessionStore store;
protected WriterCredentialPolicy credentials;
private static final String TENANT = "tenant";
private static final String SESSION = "session";
private static final String WORKSPACE = "workspace";
protected DataSource source() {
var source = new JdbcDataSource();
source.setURL("jdbc:h2:mem:placement-" + UUID.randomUUID()
+ ";MODE=MySQL;DATABASE_TO_LOWER=TRUE;DB_CLOSE_DELAY=-1");
return source;
}
@BeforeEach
void setup() {
var source = source();
ManagedSchema.initialize(source);
jdbc = new ReleaseFaultJdbc(source);
tx = new TransactionTemplate(new DataSourceTransactionManager(source));
placements = new HarnessPlacementStore(jdbc);
store = new ManagedSessionStore(jdbc);
var properties = new ManagedAgentProperties();
properties.getSessionStore().setBindingKey("0123456789abcdef0123456789abcdef");
credentials = new WriterCredentialPolicy(properties);
store.setCredentials(credentials);
jdbc.update("INSERT INTO ma_session (tenant_id, session_id, agent_id, gmt_create, gmt_modified)"
+ " VALUES (?, ?, 'qwen-code', 1, 1)", TENANT, SESSION);
}
@Test
void freshBaselineRefusesAnAssignedPlacementWithoutItsOwner() {
assertThat(jdbc.queryForMap("SELECT harness_instance_id, harness_owner_boot_id,"
+ " harness_placement_epoch, harness_placement_state FROM ma_session"))
.containsEntry("harness_instance_id", null)
.containsEntry("harness_owner_boot_id", null)
.containsEntry("harness_placement_epoch", 0L)
.containsEntry("harness_placement_state", "UNASSIGNED");
assertThatThrownBy(() -> jdbc.update("UPDATE ma_session SET harness_instance_id = 'pod-a',"
+ " harness_placement_epoch = 1, harness_placement_state = 'ACTIVE'"))
.isInstanceOfSatisfying(DataAccessException.class, error ->
assertThat(error.getMostSpecificCause().getMessage())
.containsIgnoringCase("ck_ma_harness_placement"));
assertThat(placements.get(TENANT, SESSION)).isNull();
}
@ParameterizedTest
@CsvSource({"CLOSE,CLOSING", "ARCHIVE,ARCHIVING", "DELETE,DELETING"})
void pendingLifecycleCanMoveOnlyUnderItsCurrentClaimAndWriterFence(OperationKind kind, String pendingStatus) {
var owner = tx.execute(status -> placements.assign(TENANT, SESSION, "pod-a", "boot-a"));
tx.executeWithoutResult(status -> store.acquireWriter(TENANT, SESSION, token(owner),
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L)));
jdbc.update("UPDATE ma_session SET status = ? WHERE tenant_id = ? AND session_id = ?", pendingStatus, TENANT, SESSION);
jdbc.update("UPDATE ma_session_head SET phase = 'LIFECYCLE_ONLY', operation_id = 'close'"
+ " WHERE tenant_id = ? AND session_id = ?", TENANT, SESSION);
jdbc.update("INSERT INTO ma_operation (operation_key, scope_key, operation_id, operation_kind, tenant_id, session_id,"
+ " target_kind, target_key, request_digest, request_json, state, phase, delivery_state, claim_owner,"
+ " claim_generation, claim_until, lifecycle_protocol_version, gmt_create, gmt_modified)"
+ " VALUES (?, ?, 'close', ?, ?, ?, 'SESSION', ?, ?, '{}', 'PENDING', 'LIFECYCLE_ONLY',"
+ " 'LEASED', 'worker', 7, ?, 1, 1, 1)", "c".repeat(64), "d".repeat(64), kind.name(), TENANT, SESSION,
"e".repeat(64), "f".repeat(64), HarnessPlacementStore.now(jdbc) + 60_000);
var operation = new OperationRecord(TENANT, SESSION, "close", kind, "request", "PENDING",
"ADMITTED", "LEASED", "ACTIVE", null, "worker", 7, 0, null, 1);
assertThatThrownBy(() -> tx.execute(status -> placements.takeover(TENANT, SESSION, owner, "pod-b", "boot-b")))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("harness_placement_not_admitted");
assertThatThrownBy(() -> tx.execute(status -> placements.takeoverForLifecycle(operation, owner, "pod-b", "boot-b")))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("harness_owner_still_live");
jdbc.update("UPDATE ma_session_head SET writer_lease_until = 1 WHERE tenant_id = ? AND session_id = ?", TENANT, SESSION);
var successor = tx.execute(status -> placements.takeoverForLifecycle(operation, owner, "pod-b", "boot-b"));
assertThat(successor.epoch()).isEqualTo(2);
jdbc.update("UPDATE ma_operation SET claim_generation = 8 WHERE operation_id = ?", "close");
assertThatThrownBy(() -> tx.execute(status -> placements.assignForLifecycle(operation, "pod-c", "boot-c")))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("workspace_lifecycle_claim_fenced");
}
@Test
void terminalSessionsDoNotRetainAnOldClientAfterTheirWriterStops() {
var owner = tx.execute(status -> placements.assign(TENANT, SESSION, "pod-a", "boot-a"));
tx.executeWithoutResult(status -> store.acquireWriter(TENANT, SESSION, token(owner),
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L)));
jdbc.update("UPDATE ma_session SET status = 'CLOSED' WHERE tenant_id = ? AND session_id = ?", TENANT, SESSION);
assertThat(placements.hasUnreleasedOwnership("pod-a", "boot-a")).isTrue();
jdbc.update("UPDATE ma_session_head SET state = 'SEALED' WHERE tenant_id = ? AND session_id = ?", TENANT, SESSION);
assertThat(placements.hasUnreleasedOwnership("pod-a", "boot-a")).isFalse();
jdbc.update("UPDATE ma_session SET status = 'ARCHIVED' WHERE tenant_id = ? AND session_id = ?", TENANT, SESSION);
assertThat(placements.hasUnreleasedOwnership("pod-a", "boot-a")).isFalse();
jdbc.update("UPDATE ma_session SET status = 'CLOSING' WHERE tenant_id = ? AND session_id = ?", TENANT, SESSION);
assertThat(placements.hasUnreleasedOwnership("pod-a", "boot-a")).isTrue();
}
@Test
void neverAcquiredPlacementCanMoveAndRejectsTheOriginalGrant() {
var first = tx.execute(status -> placements.assign(TENANT, SESSION, "pod-a", "boot-a"));
String originalToken = token(first);
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_session_head WHERE tenant_id = ? AND session_id = ?",
Integer.class, TENANT, SESSION)).isZero();
var successor = tx.execute(status -> placements.takeover(TENANT, SESSION, first, "pod-b", "boot-b"));
assertThat(successor.epoch()).isEqualTo(2);
assertThat(placements.get(TENANT, SESSION)).isEqualTo(successor);
assertThatThrownBy(() -> tx.execute(status -> store.acquireWriter(TENANT, SESSION, originalToken,
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L))))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("writer_credential_invalid");
var acquired = tx.execute(status -> store.acquireWriter(TENANT, SESSION, token(successor),
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-b", 60_000L)));
assertThat(acquired.writerGeneration()).isPositive();
}
@Test
void placementIsStickyAndAFormerOwnerCannotReacquireAfterExpiry() {
var first = tx.execute(status -> placements.assign(TENANT, SESSION, "pod-a", "boot-a"));
var repeated = tx.execute(status -> placements.assign(TENANT, SESSION, "pod-b", "boot-b"));
assertThat(repeated).isEqualTo(first);
String token = token(first);
var grant = tx.execute(status -> store.acquireWriter(TENANT, SESSION, token,
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L)));
assertThatThrownBy(() -> tx.execute(status -> placements.takeover(TENANT, SESSION, first, "pod-b", "boot-b")))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("harness_owner_still_live");
jdbc.update("UPDATE ma_session_head SET writer_lease_until = 1 WHERE tenant_id = ? AND session_id = ?"
+ " AND head_kind = 'MANAGED_SESSION'", TENANT, SESSION);
var next = tx.execute(status -> placements.takeover(TENANT, SESSION, first, "pod-b", "boot-b"));
assertThat(next.epoch()).isEqualTo(2);
assertThatThrownBy(() -> tx.execute(status -> store.acquireWriter(TENANT, SESSION, token,
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L))))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("writer_credential_invalid");
var successor = tx.execute(status -> store.acquireWriter(TENANT, SESSION, token(next),
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-b", 60_000L)));
assertThat(successor.writerGeneration()).isGreaterThan(grant.writerGeneration());
assertThatThrownBy(() -> tx.execute(status -> store.sealWriter(TENANT, SESSION, token,
new ManagedSessionStoreModels.SealWriterRequest(WORKSPACE, "boot-a", grant.writerGeneration()))))
.isInstanceOf(ManagedAgentException.class);
var renewed = tx.execute(status -> store.renewWriter(TENANT, SESSION, token(next),
new ManagedSessionStoreModels.RenewWriterRequest(WORKSPACE, "boot-b", successor.writerGeneration(), 60_000L)));
assertThat(renewed)
.extracting(ManagedSessionStoreModels.WriterGrant::writerGeneration).isEqualTo(successor.writerGeneration());
}
@Test
void releaseRevokesTheEpochBeforeSuccessorCasAndReplaysOnlyTheExactReceipt() {
var owner = tx.execute(status -> placements.assign(TENANT, SESSION, "pod-a", "boot-a"));
String token = token(owner);
var grant = tx.execute(status -> store.acquireWriter(TENANT, SESSION, token,
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L)));
tx.executeWithoutResult(status -> placements.quiesce(TENANT, SESSION, owner));
assertThatThrownBy(() -> tx.execute(status -> store.acquireWriter(TENANT, SESSION, token,
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L))))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("harness_placement_quiescing");
tx.executeWithoutResult(status -> store.renewWriter(TENANT, SESSION, token,
new ManagedSessionStoreModels.RenewWriterRequest(WORKSPACE, "boot-a", grant.writerGeneration(), 60_000L)));
var release = new ManagedSessionStoreModels.ReleaseOwnershipRequest(WORKSPACE, "boot-a", grant.writerGeneration(),
1, 1, "pod-a", "boot-a", "release-one", new ManagedSessionStoreModels.OwnershipCommitProof(3, "a".repeat(64)));
assertThatThrownBy(() -> tx.execute(status -> store.releaseOwnership(TENANT, SESSION, token, release)))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("harness_release_proof_conflict");
jdbc.update("UPDATE ma_session_head SET journal_revision = 1, committed_sequence = 3, last_commit_digest = ?,"
+ " activation_epoch = 1, activation_phase = 'released', activation_head_revision = 1"
+ " WHERE tenant_id = ? AND session_id = ? AND head_kind = 'MANAGED_SESSION'", "a".repeat(64), TENANT, SESSION);
var receipt = tx.execute(status -> store.releaseOwnership(TENANT, SESSION, token, release));
assertThat(receipt.path("state").asText()).isEqualTo("RELEASED");
assertThat(placements.get(TENANT, SESSION).state()).isEqualTo("RELEASED");
assertThat(tx.<Boolean>execute(status -> placements.quiesce(TENANT, SESSION, owner))).isFalse();
assertThat(placements.hasUnreleasedOwnership("pod-a", "boot-a")).isFalse();
assertThat(placements.releaseReceipt(TENANT, SESSION, 1, "release-one")).containsEntry("state", "RELEASED");
var repeated = tx.execute(status -> store.releaseOwnership(TENANT, SESSION, token, release));
assertThat(repeated).isEqualTo(receipt);
for (Runnable stale : List.<Runnable>of(
() -> store.acquireWriter(TENANT, SESSION, token, new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L)),
() -> store.renewWriter(TENANT, SESSION, token, new ManagedSessionStoreModels.RenewWriterRequest(WORKSPACE, "boot-a", grant.writerGeneration(), 60_000L)),
() -> store.sealWriter(TENANT, SESSION, token, new ManagedSessionStoreModels.SealWriterRequest(WORKSPACE, "boot-a", grant.writerGeneration())),
() -> store.authorizeOrdinary(TENANT, SESSION, token, new ManagedSessionStoreModels.AuthorizeLifecycleRequest(WORKSPACE, "boot-a", grant.writerGeneration())),
() -> store.commit(TENANT, SESSION, token, emptyCommit(grant.writerGeneration())))) {
assertThatThrownBy(() -> tx.executeWithoutResult(status -> stale.run())).isInstanceOf(ManagedAgentException.class)
.extracting("code").isEqualTo("harness_placement_released");
}
var next = tx.execute(status -> placements.takeover(TENANT, SESSION, placements.get(TENANT, SESSION), "pod-b", "boot-b"));
assertThat(next.epoch()).isEqualTo(2);
var afterTakeover = tx.execute(status -> store.releaseOwnership(TENANT, SESSION, token, release));
assertThat(afterTakeover).isEqualTo(receipt);
assertThat(placements.hasUnreleasedOwnership("pod-b", "boot-b")).isTrue();
assertThatThrownBy(() -> tx.execute(status -> store.releaseOwnership(TENANT, SESSION, token,
new ManagedSessionStoreModels.ReleaseOwnershipRequest(WORKSPACE, "boot-a", grant.writerGeneration(), 1, 1,
"pod-a", "boot-a", "different-operation", release.commitProof()))))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("harness_release_operation_conflict");
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_session_record WHERE task_kind IS NOT NULL", Integer.class)).isZero();
assertThatThrownBy(() -> store.restore(TENANT, WORKSPACE, SESSION, token)).isInstanceOf(ManagedAgentException.class);
tx.executeWithoutResult(status -> store.acquireWriter(TENANT, SESSION, token(next),
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-b", 60_000L)));
assertThat(store.restore(TENANT, WORKSPACE, SESSION, token(next)).journalRevision()).isEqualTo(1);
}
@Test
void releaseReceiptAndSealRollBackTogetherAndFenceAWaitingRenewal() throws Exception {
var owner = tx.execute(status -> placements.assign(TENANT, SESSION, "pod-a", "boot-a"));
String token = token(owner);
var grant = tx.execute(status -> store.acquireWriter(TENANT, SESSION, token,
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L)));
tx.executeWithoutResult(status -> store.quiesceOwnership(TENANT, SESSION, token,
new ManagedSessionStoreModels.QuiesceOwnershipRequest(WORKSPACE, "boot-a", grant.writerGeneration(), 1, "pod-a", "boot-a")));
jdbc.update("UPDATE ma_session_head SET journal_revision = 1, committed_sequence = 3, last_commit_digest = ?,"
+ " activation_epoch = 1, activation_phase = 'released', activation_head_revision = 1"
+ " WHERE tenant_id = ? AND session_id = ? AND head_kind = 'MANAGED_SESSION'", "a".repeat(64), TENANT, SESSION);
var release = new ManagedSessionStoreModels.ReleaseOwnershipRequest(WORKSPACE, "boot-a", grant.writerGeneration(), 1, 1,
"pod-a", "boot-a", "release-atomic", new ManagedSessionStoreModels.OwnershipCommitProof(3, "a".repeat(64)));
((ReleaseFaultJdbc) jdbc).failSeal = true;
assertThatThrownBy(() -> tx.execute(status -> store.releaseOwnership(TENANT, SESSION, token, release)))
.isInstanceOf(IllegalStateException.class).hasMessage("injected release seal failure");
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_session_record WHERE domain = 'hosted_harness_release'", Integer.class)).isZero();
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM ma_resource WHERE kind = ?", Integer.class,
"managed-hosted_harness_release")).isZero();
((ReleaseFaultJdbc) jdbc).failSeal = false;
var started = new CountDownLatch(1);
var pending = new AtomicReference<Future<String>>();
try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
tx.executeWithoutResult(status -> {
jdbc.queryForList("SELECT session_id FROM ma_session WHERE tenant_id = ? AND session_id = ? FOR UPDATE", TENANT, SESSION);
pending.set(executor.submit(() -> {
started.countDown();
try {
tx.executeWithoutResult(inner -> store.renewWriter(TENANT, SESSION, token,
new ManagedSessionStoreModels.RenewWriterRequest(WORKSPACE, "boot-a", grant.writerGeneration(), 60_000L)));
return "renewed";
} catch (ManagedAgentException refused) {
return refused.getCode();
}
}));
try {
assertThat(started.await(5, TimeUnit.SECONDS)).isTrue();
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
throw new IllegalStateException(interrupted);
}
store.releaseOwnership(TENANT, SESSION, token, release);
});
assertThat(pending.get().get(10, TimeUnit.SECONDS)).isEqualTo("harness_placement_released");
}
assertThat(jdbc.queryForObject("SELECT state FROM ma_session_head WHERE head_kind = 'MANAGED_SESSION' AND tenant_id = ? AND session_id = ?",
String.class, TENANT, SESSION)).isEqualTo("SEALED");
}
@Test
void scopeCredentialsAndAmbiguousControlIdsNeverEnterThePool() {
var owner = tx.execute(status -> placements.assign(TENANT, SESSION, "pod-a", "boot-a"));
assertThatThrownBy(() -> tx.execute(status -> store.acquireWriter(TENANT, SESSION,
credentials.issue(TENANT, WORKSPACE, SESSION),
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L))))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("writer_credential_invalid");
assertThatThrownBy(() -> tx.execute(status -> store.acquireWriter("other", SESSION, token(owner),
new ManagedSessionStoreModels.AcquireWriterRequest(WORKSPACE, "boot-a", 60_000L))))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("writer_credential_invalid");
for (String tenant : List.of("one", "two")) {
jdbc.update("INSERT INTO ma_session (tenant_id, session_id, session_kind, workspace_id, gmt_create, gmt_modified)"
+ " VALUES (?, 'control', 'CONTROL', 'workspace', 1, 1)", tenant);
}
assertThatThrownBy(() -> tx.execute(status -> placements.assign("one", "control", "pod-a", "boot-a")))
.isInstanceOf(ManagedAgentException.class).extracting("code").isEqualTo("harness_session_scope_ambiguous");
}
private String token(HarnessPlacementStore.Placement placement) {
return credentials.issue(TENANT, WORKSPACE, SESSION, placement.instanceId(), placement.bootId(), placement.epoch());
}
private ManagedSessionStoreModels.CommitTransactionRequest emptyCommit(long generation) {
String records = "{\"subtype\":\"session_execution_engine\"}\n{\"subtype\":\"managed_session_header_v1\"}\n";
return new ManagedSessionStoreModels.CommitTransactionRequest(WORKSPACE, "boot-a", generation, 1, 3,
"transaction", "session.create", "stale-command", "0".repeat(64), 0, 0, 0, null, null, null, 0, null,
2, Base64.getEncoder().encodeToString(records.getBytes(StandardCharsets.UTF_8)), DigestUtils.sha256Hex(records), List.of());
}
private static final class ReleaseFaultJdbc extends ManagedSchemaJdbcTemplate {
private boolean failSeal;
private ReleaseFaultJdbc(DataSource source) {
super(source);
}
@Override
public int update(String sql, Object... args) {
if (failSeal && sql.startsWith("UPDATE ma_session_head SET state = 'SEALED'")) {
throw new IllegalStateException("injected release seal failure");
}
return super.update(sql, args);
}
}
}
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/persistence/SessionRowLockOrderTest.java
+5
View File
@@ -134,6 +134,7 @@ class SessionRowLockOrderTest {
// new sibling turn-row access — fails here instead of passing
// unaudited.
private static final Set<String> SIBLING_SESSION_LOCKERS = Set.of(
"HarnessPlacementStore.java",
"ManagedActionStore.java", "ManagedExtensionRecordStore.java",
"ManagedLegacySessionGuard.java",
"ManagedProjectionResourceCollector.java",
@@ -180,6 +181,10 @@ class SessionRowLockOrderTest {
Set.of("ChildWorkspaceStore.find/3",
"ChildWorkspaceStore.row/2",
"Row.snapshot/0")),
// Same-package lifecycle refusals share this local name;
// both factories only construct an exception.
Map.entry("HarnessPlacementStore.java",
Set.of("HarnessPlacementStore.blocked/1")),
Map.entry("JdbcChannelRouteRepository.java",
Set.of("JdbcChannelRouteRepository.row/2")),
Map.entry("ManagedAgentDefinitionStore.java",
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/persistence/WorkspaceCsiApplicationInventoryTest.java
+1 -1
View File
@@ -770,7 +770,7 @@ class WorkspaceCsiApplicationInventoryTest {
void inventoriesEveryDeclaredTableAndPreservesRealPrivateCreationFacts() throws Exception {
var snapshot = inspect();
assertThat(snapshot.tables()).hasSize(32);
assertThat(snapshot.tables().stream().map(table -> table.columns().size()).mapToInt(Integer::intValue).sum()).isEqualTo(743);
assertThat(snapshot.tables().stream().map(table -> table.columns().size()).mapToInt(Integer::intValue).sum()).isEqualTo(747);
assertThat(snapshot.blockers()).noneMatch(blocker -> blocker.code().equals("schema_drift")
|| blocker.code().equals("query_unavailable"));
for (String table : new String[] {"ma_session", "ma_public_event", "ma_event_stream", "ma_storage", "ma_operation",
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/security/WriterCredentialPolicyTest.java
+19
View File
@@ -51,6 +51,25 @@ class WriterCredentialPolicyTest {
.hasMessageContaining("binding-key");
}
@Test
void placementCredentialsBindScopeOwnerBootAndEpoch() {
var policy = policy(KEY);
String token = policy.issue("tenant", "workspace", "session", "pod-a", "boot-a", 1);
assertThat(token).startsWith("qwp1_").matches("[A-Za-z0-9_-]{32,512}");
assertThatCode(() -> policy.require("tenant", "workspace", "session", token)).doesNotThrowAnyException();
assertThatCode(() -> policy.requirePlacement("tenant", "workspace", "session", "pod-a", "boot-a", 1, token))
.doesNotThrowAnyException();
assertThatThrownBy(() -> policy.require("other", "workspace", "session", token)).isInstanceOf(ManagedAgentException.class);
assertThatThrownBy(() -> policy.requirePlacement("tenant", "workspace", "session", "pod-b", "boot-a", 1, token))
.isInstanceOf(ManagedAgentException.class);
assertThatThrownBy(() -> policy.requirePlacement("tenant", "workspace", "session", "pod-a", "boot-b", 1, token))
.isInstanceOf(ManagedAgentException.class);
assertThatThrownBy(() -> policy.requirePlacement("tenant", "workspace", "session", "pod-a", "boot-a", 2, token))
.isInstanceOf(ManagedAgentException.class);
assertThatThrownBy(() -> WriterCredentialPolicy.unbound().require("tenant", "workspace", "session", token))
.isInstanceOf(ManagedAgentException.class);
}
@Test
void boundRejectsForeignAndSelfMintedTokens() {
WriterCredentialPolicy policy = policy(KEY);
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/session/HarnessCoordinatorTest.java
+41
View File
@@ -11,6 +11,7 @@ import com.alibaba.qwen.code.daemon.HarnessSessionRefusedException;
import com.alibaba.qwen.code.daemon.HostedHarnessGenerationException;
import com.alibaba.qwen.code.daemon.HostedHarnessStreamIdleException;
import com.alibaba.qwen.code.managedagent.config.ManagedAgentProperties;
import com.alibaba.qwen.code.managedagent.exception.ManagedAgentException;
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector;
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector.Admission;
import com.alibaba.qwen.code.managedagent.harness.HarnessConnector.Attachment;
@@ -33,6 +34,7 @@ import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.ValueSource;
import org.mockito.InOrder;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import java.time.Clock;
import java.time.Duration;
@@ -586,6 +588,45 @@ class HarnessCoordinatorTest {
anyString(), anyString(), anyString(), anyString());
}
@ParameterizedTest
@CsvSource({"503,harness_pool_unavailable", "409,harness_owner_still_live",
"409,harness_placement_changed", "409,harness_placement_quiescing", "409,harness_placement_released"})
void poolCapacityAndOwnershipWaitsOutliveBothAdmissionBudgets(int status, String code) {
for (boolean submitted : List.of(false, true)) {
var failure = new ManagedAgentException(HttpStatus.valueOf(status), code, "Waiting for a Harness owner");
AgentStateStore store = dispatchWithCreateOrLoadFailure(failure, submitted, 100, unboundSessionRecord());
verify(store).scheduleTurnRetry(eq("tenant"), eq("session"), eq("turn"),
anyString(), anyLong(), eq(true));
verify(store, never()).failTurn(anyString(), anyString(), anyString(), anyString(), anyString(), anyString());
verify(store, never()).markTurnFailureRecovery(anyString(), anyString(), anyString(), anyString(), anyString());
}
}
@Test
void hostedDrainCutoffOutlivesBothAdmissionBudgets() {
for (boolean submitted : List.of(false, true)) {
var failure = mock(DaemonHttpException.class);
when(failure.getStatusCode()).thenReturn(409);
when(failure.getErrorCode()).thenReturn("hosted_harness_quiescing");
AgentStateStore store = dispatchWithCreateOrLoadFailure(failure, submitted, 100, unboundSessionRecord());
verify(store).scheduleTurnRetry(eq("tenant"), eq("session"), eq("turn"),
anyString(), anyLong(), eq(true));
verify(store, never()).failTurn(anyString(), anyString(), anyString(), anyString(), anyString(), anyString());
verify(store, never()).markTurnFailureRecovery(anyString(), anyString(), anyString(), anyString(), anyString());
}
}
@ParameterizedTest
@CsvSource({"503,other_unavailable", "403,writer_credential_invalid", "409,harness_session_scope_ambiguous",
"409,harness_placement_not_admitted", "409,harness_placement_epoch_exhausted", "400,harness_pool_unavailable"})
void poolConfigurationAndUnclassifiedFailuresStillMeetTheAdmissionBudget(int status, String code) {
var failure = new ManagedAgentException(HttpStatus.valueOf(status), code, "Cannot admit this Harness owner");
AgentStateStore store = dispatchWithCreateOrLoadFailure(failure, false, 100, unboundSessionRecord());
verify(store).failTurn(eq("tenant"), eq("session"), eq("turn"),
anyString(), eq("hosted_harness_unavailable"), anyString());
verify(store, never()).scheduleTurnRetry(anyString(), anyString(), anyString(), anyString(), anyLong(), anyBoolean());
}
// A permanent transport failure meets the pre-admission budget even on
// a bound Session — only the writer lease's own code use the
// lease-window exemption, never any other failure kind (its takeover
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/session/OperationRetryTerminalStateTest.java
+23 -23
View File
@@ -99,7 +99,7 @@ class OperationRetryTerminalStateTest {
1, lifecycleProtocol == 1 ? BOUND_WORKSPACE : null,
"yolo", TOOL_PROFILE));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenReturn("boot-1");
when(harness.closeSession(any(OperationRecord.class))).thenReturn("boot-1");
// Another live Harness keeps the Session's journal writer, so the
// close cannot settle yet: settle() throws on every attempt.
when(sessionStore.hasLiveWriter("tenant", "session"))
@@ -161,7 +161,7 @@ class OperationRetryTerminalStateTest {
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
// The writer is gone, but the Harness call itself still fails.
when(harness.closeSession("tenant", "session"))
when(harness.closeSession(any(OperationRecord.class)))
.thenThrow(new IllegalStateException("harness unreachable"));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
@@ -218,7 +218,7 @@ class OperationRetryTerminalStateTest {
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenThrow(
when(harness.closeSession(any(OperationRecord.class))).thenThrow(
mock(HostedHarnessGenerationException.class));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
@@ -275,7 +275,7 @@ class OperationRetryTerminalStateTest {
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenThrow(
when(harness.closeSession(any(OperationRecord.class))).thenThrow(
mock(HostedHarnessGenerationException.class));
// The old boot's writer lease has not expired yet.
when(sessionStore.hasLiveWriter("tenant", "session"))
@@ -328,7 +328,7 @@ class OperationRetryTerminalStateTest {
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenThrow(
when(harness.closeSession(any(OperationRecord.class))).thenThrow(
mock(HostedHarnessGenerationException.class));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
@@ -383,7 +383,7 @@ class OperationRetryTerminalStateTest {
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenReturn("boot-1");
when(harness.closeSession(any(OperationRecord.class))).thenReturn("boot-1");
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(true);
@@ -441,7 +441,7 @@ class OperationRetryTerminalStateTest {
when(harness.isAvailable()).thenReturn(true);
// The writer is already gone and the Harness call itself keeps
// failing — a settle the operation can never complete.
when(harness.closeSession("tenant", "session"))
when(harness.closeSession(any(OperationRecord.class)))
.thenThrow(new IllegalStateException("harness unreachable"));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
@@ -608,7 +608,7 @@ class OperationRetryTerminalStateTest {
verify(store, never()).retryOperation(anyString(), anyString(),
anyString(), anyString(), anyLong(), anyLong());
// The refused close never reached the Harness.
verify(harness, never()).closeSession(anyString(), anyString());
verify(harness, never()).closeSession(any(OperationRecord.class));
} finally {
coordinator.stopRenewals();
}
@@ -641,7 +641,7 @@ class OperationRetryTerminalStateTest {
HostedHarnessCapabilityMismatchException mismatch =
mock(HostedHarnessCapabilityMismatchException.class);
when(mismatch.getCode()).thenReturn("managed_capability_mismatch");
when(harness.closeSession("tenant", "session")).thenThrow(mismatch);
when(harness.closeSession(any(OperationRecord.class))).thenThrow(mismatch);
// The Harness was never asked to stop, so it keeps the writer.
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(true);
@@ -702,7 +702,7 @@ class OperationRetryTerminalStateTest {
when(runtimeWarmer.closeWorkspace("tenant", "session"))
.thenReturn(CompletableFuture.completedFuture(null));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session"))
when(harness.closeSession(any(OperationRecord.class)))
.thenThrow(mock(DaemonProtocolException.class));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(true);
@@ -760,7 +760,7 @@ class OperationRetryTerminalStateTest {
HostedHarnessCapabilityMismatchException mismatch =
mock(HostedHarnessCapabilityMismatchException.class);
when(mismatch.getCode()).thenReturn("managed_capability_mismatch");
when(harness.closeSession("tenant", "session")).thenThrow(mismatch);
when(harness.closeSession(any(OperationRecord.class))).thenThrow(mismatch);
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(true);
@@ -814,7 +814,7 @@ class OperationRetryTerminalStateTest {
null, "ACTIVE", "boot-1", null, 0, 0, 0, 1, 1, null,
1, BOUND_WORKSPACE, "yolo", TOOL_PROFILE));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session"))
when(harness.closeSession(any(OperationRecord.class)))
.thenThrow(new IllegalStateException("harness unreachable"));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
@@ -938,7 +938,7 @@ class OperationRetryTerminalStateTest {
null, "ACTIVE", "boot-1", null, 0, 0, 0, 1, 1, null,
1, BOUND_WORKSPACE, "yolo", TOOL_PROFILE));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenReturn("boot-1");
when(harness.closeSession(any(OperationRecord.class))).thenReturn("boot-1");
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
when(runtimeWarmer.supportsWorkspaceClose(BOUND_WORKSPACE)).thenReturn(true);
@@ -1068,7 +1068,7 @@ class OperationRetryTerminalStateTest {
null, "ACTIVE", "boot-1", null, 0, 0, 0, 1, 1, null,
1, BOUND_WORKSPACE, "yolo", TOOL_PROFILE));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenReturn("boot-1");
when(harness.closeSession(any(OperationRecord.class))).thenReturn("boot-1");
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
when(runtimeWarmer.supportsWorkspaceClose(BOUND_WORKSPACE)).thenReturn(true);
@@ -1129,7 +1129,7 @@ class OperationRetryTerminalStateTest {
null, "ACTIVE", "boot-1", null, 0, 0, 0, 1, 1, null,
1, BOUND_WORKSPACE, "yolo", TOOL_PROFILE));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenReturn("boot-1");
when(harness.closeSession(any(OperationRecord.class))).thenReturn("boot-1");
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
when(runtimeWarmer.supportsWorkspaceClose(BOUND_WORKSPACE)).thenReturn(true);
@@ -1505,7 +1505,7 @@ class OperationRetryTerminalStateTest {
HostedHarnessCapabilityMismatchException mismatch =
mock(HostedHarnessCapabilityMismatchException.class);
when(mismatch.getCode()).thenReturn("managed_capability_mismatch");
when(harness.closeSession("tenant", "session")).thenThrow(mismatch);
when(harness.closeSession(any(OperationRecord.class))).thenThrow(mismatch);
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenThrow(new IllegalStateException("lock wait timeout"));
@@ -1802,7 +1802,7 @@ class OperationRetryTerminalStateTest {
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session"))
when(harness.closeSession(any(OperationRecord.class)))
.thenThrow(new IllegalStateException("harness unreachable"));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
@@ -1853,7 +1853,7 @@ class OperationRetryTerminalStateTest {
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session"))
when(harness.closeSession(any(OperationRecord.class)))
.thenThrow(new IllegalStateException("harness unreachable"));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
@@ -2093,7 +2093,7 @@ class OperationRetryTerminalStateTest {
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session"))
when(harness.closeSession(any(OperationRecord.class)))
.thenThrow(new IllegalStateException("harness unreachable"));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenThrow(new IllegalStateException("lock wait timeout"));
@@ -2147,7 +2147,7 @@ class OperationRetryTerminalStateTest {
null, "ACTIVE", "boot-1", null, 0, 0, 0, 1, 1, null,
1, BOUND_WORKSPACE, "yolo", TOOL_PROFILE));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenReturn("boot-1");
when(harness.closeSession(any(OperationRecord.class))).thenReturn("boot-1");
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
when(runtimeWarmer.supportsWorkspaceClose(BOUND_WORKSPACE)).thenReturn(true);
@@ -2267,7 +2267,7 @@ class OperationRetryTerminalStateTest {
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenReturn("boot-1");
when(harness.closeSession(any(OperationRecord.class))).thenReturn("boot-1");
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(true);
@@ -2700,7 +2700,7 @@ class OperationRetryTerminalStateTest {
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session"))
when(harness.closeSession(any(OperationRecord.class)))
.thenThrow(new IllegalStateException("harness unreachable"));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
@@ -2761,7 +2761,7 @@ class OperationRetryTerminalStateTest {
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session"))
when(harness.closeSession(any(OperationRecord.class)))
.thenThrow(new IllegalStateException("harness unreachable"));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/session/OwnerSubscriptionsTest.java
+14
View File
@@ -39,6 +39,20 @@ class OwnerSubscriptionsTest {
owners.bind(consumer);
}
@Test
void sharedPoolVerifiesEachAttachmentsOwnWriterInsteadOfOneGlobalIdentity() throws Exception {
when(harness.writerIdentity()).thenReturn(null);
when(harness.attachedSessions()).thenReturn(List.of(
new AttachedSession("tenant", "first", "boot-a", 1),
new AttachedSession("tenant", "second", "boot-b", 2)));
when(consumer.getLiteTopicSet()).thenReturn(Set.of());
when(store.heldBy("boot-a", "tenant", List.of("first"))).thenReturn(Set.of("first"));
when(store.heldBy("boot-b", "tenant", List.of("second"))).thenReturn(Set.of("second"));
owners.reconcile();
verify(consumer).subscribeLite("first");
verify(consumer).subscribeLite("second");
}
@Test
void followsTheSessionsThisNodesHarnessWrites() throws Exception {
when(consumer.getLiteTopicSet()).thenReturn(Set.of("held", "released"));
packages/managed-agent-server/src/test/resources/capability-parity/managed-surface.txt
+3 -1
View File
@@ -1,5 +1,5 @@
# Generated by ManagedSurfaceCaptureTest (capture).
# sha256 of the route lines below: 899354d4f81879927d1ff97dcab89faca076ae6ee8025cf26832d9d7f0b35eaa
# sha256 of the route lines below: a3f85909e0a5162bdcc8c8902d5d1de890e548c67d374261d16dddb2dcab44b8
AUTOMATION_RETIRE DELETE /v1/agent-automations/{automationId} public
SESSION_DELETE DELETE /v1/agents/sessions/{sessionId} public
STORE_RESOURCE_GET GET /internal/managed-session-store/v1/sessions/{sessionId}/resources/{resourceId} internal
@@ -95,6 +95,8 @@ STORE_TOOL_RESULT_PUBLISH POST /internal/managed-session-store/v1/sessions/{sess
STORE_TRANSACTION_COMMIT POST /internal/managed-session-store/v1/sessions/{sessionId}/transactions:commit internal
STORE_TRANSACTION_COMMIT POST /internal/managed-session-store/v1/sessions/{sessionId}/transactions:commit-v2 internal
STORE_WRITER_ACQUIRE POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:acquire internal
STORE_WRITER_SEAL POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:quiesce-ownership internal
STORE_WRITER_SEAL POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:release-ownership internal
STORE_WRITER_RENEW POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:renew internal
STORE_WRITER_SEAL POST /internal/managed-session-store/v1/sessions/{sessionId}/writers:seal internal
PUB_GRANT POST /internal/managed-tool-publications/v1/sessions/{sessionId}/grants internal
packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/CreateHarnessSession.java
+11
View File
@@ -8,6 +8,7 @@ import java.util.stream.Collectors;
/** Input for creating a Hosted Harness session with a caller-owned UUID. */
public final class CreateHarnessSession {
private final String harnessSessionId;
private final HostedHarnessPlacement harnessPlacement;
private final String approvalMode;
private final ManagedSessionStoreConnection managedSessionStore;
private final String toolProfile;
@@ -29,6 +30,7 @@ public final class CreateHarnessSession {
this.harnessSessionId = HostedHarnessClient.requireUuid(
builder.harnessSessionId, "harnessSessionId");
this.approvalMode = builder.approvalMode;
this.harnessPlacement = builder.harnessPlacement;
this.managedSessionStore = builder.managedSessionStore;
this.toolProfile = builder.toolProfile;
this.approvalTimeoutMs = builder.approvalTimeoutMs;
@@ -58,6 +60,9 @@ public final class CreateHarnessSession {
Map<String, Object> result = new LinkedHashMap<>();
result.put("sessionId", harnessSessionId);
result.put("sessionScope", "thread");
if (harnessPlacement != null) {
result.put("harnessPlacement", harnessPlacement.toJson());
}
if (approvalMode != null) {
result.put("approvalMode", approvalMode);
}
@@ -111,6 +116,7 @@ public final class CreateHarnessSession {
public static final class Builder {
private String harnessSessionId;
private HostedHarnessPlacement harnessPlacement;
private String approvalMode;
private ManagedSessionStoreConnection managedSessionStore;
private String toolProfile;
@@ -191,6 +197,11 @@ public final class CreateHarnessSession {
private Builder() {
}
public Builder harnessPlacement(HostedHarnessPlacement value) {
this.harnessPlacement = value;
return this;
}
public Builder harnessSessionId(String harnessSessionId) {
this.harnessSessionId = harnessSessionId;
return this;
packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessCapabilities.java
+13
View File
@@ -9,6 +9,7 @@ public final class HostedHarnessCapabilities {
private final int currentProtocolVersion;
private final List<Integer> supportedProtocolVersions;
private final String bootId;
private final String instanceId;
private final String capabilityDigest;
private final int lifecycleProtocolVersion;
private final List<String> features;
@@ -47,6 +48,14 @@ public final class HostedHarnessCapabilities {
HostedHarnessCapabilities(int currentProtocolVersion, List<Integer> supportedProtocolVersions,
String bootId, String capabilityDigest, int lifecycleProtocolVersion, int titleProtocolVersion,
List<String> features, List<String> lifecycleV2Profiles) {
this(currentProtocolVersion, supportedProtocolVersions, bootId, capabilityDigest,
lifecycleProtocolVersion, titleProtocolVersion, features, lifecycleV2Profiles, null);
}
HostedHarnessCapabilities(int currentProtocolVersion, List<Integer> supportedProtocolVersions,
String bootId, String capabilityDigest, int lifecycleProtocolVersion, int titleProtocolVersion,
List<String> features, List<String> lifecycleV2Profiles, String instanceId) {
this.instanceId = instanceId;
this.features = List.copyOf(features);
this.lifecycleV2Profiles = List.copyOf(lifecycleV2Profiles);
this.titleProtocolVersion = titleProtocolVersion;
@@ -68,6 +77,10 @@ public final class HostedHarnessCapabilities {
return supportedProtocolVersions;
}
public String getInstanceId() {
return instanceId;
}
public String getBootId() {
return bootId;
}
packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessClient.java
+122 -4
View File
@@ -1,5 +1,7 @@
package com.alibaba.qwen.code.daemon;
import javax.net.ssl.SSLContext;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
@@ -137,12 +139,15 @@ public final class HostedHarnessClient implements AutoCloseable {
daemonThreadFactory(
"qwencode-hosted-harness-" + number + "-timer-"));
this.scheduler.setRemoveOnCancelPolicy(true);
this.httpClient = HttpClient.newBuilder()
HttpClient.Builder transport = HttpClient.newBuilder()
.connectTimeout(builder.connectTimeout)
.executor(httpExecutor)
.followRedirects(HttpClient.Redirect.NEVER)
.version(HttpClient.Version.HTTP_1_1)
.build();
.version(HttpClient.Version.HTTP_1_1);
if (builder.sslContext != null) {
transport.sslContext(builder.sslContext);
}
this.httpClient = transport.build();
try {
this.capabilities = negotiateCapabilities(expectedDigest);
} catch (RuntimeException e) {
@@ -162,6 +167,111 @@ public final class HostedHarnessClient implements AutoCloseable {
return capabilities;
}
/** Observes process-global admission without attaching or acquiring a Session. */
public Map<String, Object> readInstance() {
ensurePoolLifecycle();
String operation = "GET /internal/hosted-harness/v1/instance";
try {
var raw = send("/internal/hosted-harness/v1/instance", "GET", null, null);
validateGeneration(raw.headers(), raw.statusCode());
var response = HttpSupport.consume(raw, operation);
DaemonClient.requireStatus(response, 200, operation);
return instanceResponse(response.getBody());
} catch (IOException error) {
throw new DaemonTransportException(operation + " transport failed", error);
} catch (InterruptedException error) {
Thread.currentThread().interrupt();
throw new DaemonTransportException(operation + " was interrupted", error);
}
}
/** Installs a process-wide no-start cutoff; this does not close business Sessions. */
public Map<String, Object> drainInstance(String operationId) {
ensurePoolLifecycle();
String operation = "POST /internal/hosted-harness/v1/instance:drain";
var response = sendMutation("/internal/hosted-harness/v1/instance:drain",
Map.of("operationId", requireUuid(operationId, "operationId"),
"expectedBootId", capabilities.getBootId()), capabilities.getBootId(), null, operation);
requireMutationStatus(response, 202, operation);
try {
return instanceResponse(response.getBody());
} catch (DaemonProtocolException error) {
throw new MutationOutcomeUnknownException(operation, error);
}
}
/** Releases an idle attachment without cancelling its durable backlog. */
public Map<String, Object> releaseOwnership(HarnessSessionRef session,
String operationId, long placementEpoch) {
ensurePoolLifecycle();
HarnessSessionRef ref = requireSessionRef(session);
if (placementEpoch < 1 || placementEpoch > 9_007_199_254_740_991L) {
throw new IllegalArgumentException("Invalid placement epoch");
}
String id = requireUuid(operationId, "operationId");
String operation = "POST /session/:id/ownership:release";
var response = sendMutation(sessionPath(ref.getHarnessSessionId()) + "/ownership:release",
Map.of("operationId", id, "placementEpoch", placementEpoch),
ref.getHarnessSessionId(), ref.getHarnessClientId(), operation, loadTimeout);
requireMutationStatus(response, 200, operation);
try {
var body = JsonSupport.parseObject(response.getBody(), operation);
if (!"qwen-hosted-harness-release/1".equals(body.get("format"))
|| !id.equals(body.get("operationId"))
|| !capabilities.getInstanceId().equals(body.get("instanceId"))
|| !capabilities.getBootId().equals(body.get("bootId"))
|| !"RELEASED".equals(body.get("state"))
|| JsonSupport.requiredNonNegativeLong(body, "placementEpoch", operation) != placementEpoch
|| body.containsKey("sessionId") && !ref.getHarnessSessionId().equals(body.get("sessionId"))) {
throw new DaemonProtocolException("Harness release receipt identity differs");
}
long generation = JsonSupport.requiredNonNegativeLong(body, "writerGeneration", operation);
long revision = JsonSupport.requiredNonNegativeLong(body, "journalRevision", operation);
long committed = JsonSupport.requiredNonNegativeLong(body, "committedSequence", operation);
var proof = JsonSupport.requiredObject(body, "commitProof", operation);
if (generation < 1 || generation > 9_007_199_254_740_991L
|| revision < 1 || revision > 9_007_199_254_740_991L
|| committed > 9_007_199_254_740_991L
|| JsonSupport.requiredNonNegativeLong(proof, "lastCommitSequence", operation) != committed
|| !JsonSupport.requiredString(proof, "committedPrefixHash", operation).matches("[0-9a-f]{64}")) {
throw new DaemonProtocolException("Harness release committed prefix differs");
}
removeAttachment(ref);
retireStaleLedger(ref.getHarnessSessionId());
return body;
} catch (DaemonProtocolException error) {
throw new MutationOutcomeUnknownException(operation, error);
}
}
private void ensurePoolLifecycle() {
ensureOpen();
if (capabilities.getInstanceId() == null
|| !capabilities.getFeatures().contains("harness-pool-owner/1")
|| !capabilities.getFeatures().contains("harness-instance-lifecycle/1")) {
throw new DaemonProtocolException("Harness pool lifecycle is unavailable");
}
}
private Map<String, Object> instanceResponse(String json) {
var body = JsonSupport.parseObject(json, "Harness instance response");
Object state = body.get("state");
if (!"qwen-hosted-harness-instance/1".equals(body.get("format"))
|| !capabilities.getInstanceId().equals(body.get("instanceId"))
|| !capabilities.getBootId().equals(body.get("bootId"))
|| !capabilities.getCapabilityDigest().equals(body.get("capabilityDigest"))
|| !(state instanceof String)
|| !Set.of("ACCEPTING", "QUIESCING", "DRAINED").contains(state)) {
throw new DaemonProtocolException("Harness instance identity differs");
}
var counters = JsonSupport.requiredObject(body, "counters", "Harness instance response");
for (String name : List.of("attachedSessions", "activeTurns", "openingSessions", "pendingInputs",
"heldResources", "pendingPublications", "inFlightMutations")) {
JsonSupport.requiredNonNegativeLong(counters, name, "Harness instance counters");
}
return body;
}
public HarnessSessionRef createSession(CreateHarnessSession request) {
if (request == null) {
throw new IllegalArgumentException("request must not be null");
@@ -1912,7 +2022,8 @@ public final class HostedHarnessClient implements AutoCloseable {
? JsonSupport.requiredInt(hosted, "lifecycleProtocolVersion", "capabilities.hostedHarness") : 0,
hosted.containsKey("titleProtocolVersion")
? JsonSupport.requiredInt(hosted, "titleProtocolVersion", "capabilities.hostedHarness") : 0,
hostedFeatures, parseLifecycleV2Profiles(hosted));
hostedFeatures, parseLifecycleV2Profiles(hosted),
JsonSupport.optionalString(hosted, "instanceId"));
}
private static List<String> parseLifecycleV2Profiles(Map<String, Object> hosted) {
@@ -2673,6 +2784,7 @@ public final class HostedHarnessClient implements AutoCloseable {
private Duration sseIdleTimeout = Duration.ofSeconds(45);
private int maximumSseFrameBytes = 16 * 1024 * 1024;
private ExecutorService httpExecutorOverride;
private SSLContext sslContext;
private Builder() {
}
@@ -2684,6 +2796,12 @@ public final class HostedHarnessClient implements AutoCloseable {
return this;
}
/** Uses explicit client identity and trust while retaining HTTPS hostname verification. */
public Builder sslContext(SSLContext value) {
this.sslContext = value;
return this;
}
public Builder baseUri(URI baseUri) {
this.baseUri = baseUri;
return this;
packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessPlacement.java
+38
View File
@@ -0,0 +1,38 @@
package com.alibaba.qwen.code.daemon;
import java.util.Map;
/** The control plane's durable assignment of one Session to a process incarnation. */
public final class HostedHarnessPlacement {
private final String instanceId;
private final String bootId;
private final long placementEpoch;
public HostedHarnessPlacement(String instanceId, String bootId, long placementEpoch) {
if (instanceId == null || !instanceId.matches("[A-Za-z0-9._:-]{1,128}")) {
throw new IllegalArgumentException("Invalid Harness instance identity");
}
this.instanceId = instanceId;
this.bootId = HostedHarnessClient.requireUuid(bootId, "bootId");
if (placementEpoch < 1 || placementEpoch > 9_007_199_254_740_991L) {
throw new IllegalArgumentException("Invalid Harness placement epoch");
}
this.placementEpoch = placementEpoch;
}
public String getInstanceId() {
return instanceId;
}
public String getBootId() {
return bootId;
}
public long getPlacementEpoch() {
return placementEpoch;
}
Map<String, Object> toJson() {
return Map.of("instanceId", instanceId, "bootId", bootId, "placementEpoch", placementEpoch);
}
}
packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/LoadHarnessSession.java
+13
View File
@@ -8,6 +8,7 @@ import java.util.stream.Collectors;
/** Input for attaching Java to an existing Hosted Harness session. */
public final class LoadHarnessSession {
private final String harnessSessionId;
private HostedHarnessPlacement harnessPlacement;
private final ManagedSessionStoreConnection managedSessionStore;
private final boolean passiveManagedRuntimeRecovery;
private final String toolProfile;
@@ -76,6 +77,11 @@ public final class LoadHarnessSession {
return this;
}
public LoadHarnessSession withHarnessPlacement(HostedHarnessPlacement value) {
this.harnessPlacement = value;
return this;
}
public LoadHarnessSession withAgentDefinition(Map<String, Object> value) {
this.agentDefinition = value == null ? null : Map.copyOf(value);
return this;
@@ -113,6 +119,7 @@ public final class LoadHarnessSession {
LoadHarnessSession copy = new LoadHarnessSession(harnessSessionId, managedSessionStore,
passiveManagedRuntimeRecovery, toolProfile, driveRuntimeRecovery, cancellationTakeover,
suppressChildAgents);
copy.harnessPlacement = harnessPlacement;
copy.agentDefinition = agentDefinition;
copy.sessionSkillsRef = sessionSkillsRef;
copy.mcpServers = mcpServers;
@@ -135,6 +142,7 @@ public final class LoadHarnessSession {
LoadHarnessSession copy = new LoadHarnessSession(harnessSessionId, managedSessionStore,
passiveManagedRuntimeRecovery, toolProfile, driveRuntimeRecovery, cancellationTakeover,
suppressChildAgents);
copy.harnessPlacement = harnessPlacement;
copy.agentDefinition = agentDefinition;
copy.sessionSkillsRef = sessionSkillsRef;
copy.mcpServers = mcpServers;
@@ -169,6 +177,7 @@ public final class LoadHarnessSession {
LoadHarnessSession copy = new LoadHarnessSession(harnessSessionId, managedSessionStore,
passiveManagedRuntimeRecovery, toolProfile, driveRuntimeRecovery, cancellationTakeover,
suppressChildAgents);
copy.harnessPlacement = harnessPlacement;
copy.agentDefinition = agentDefinition;
copy.sessionSkillsRef = sessionSkillsRef;
copy.mcpServers = mcpServers;
@@ -192,6 +201,7 @@ public final class LoadHarnessSession {
LoadHarnessSession copy = new LoadHarnessSession(harnessSessionId, managedSessionStore,
passiveManagedRuntimeRecovery, toolProfile, driveRuntimeRecovery, cancellationTakeover,
suppressChildAgents);
copy.harnessPlacement = harnessPlacement;
copy.agentDefinition = agentDefinition;
copy.sessionSkillsRef = sessionSkillsRef;
copy.mcpServers = mcpServers;
@@ -278,6 +288,9 @@ public final class LoadHarnessSession {
Map<String, Object> toJson() {
Map<String, Object> result = new LinkedHashMap<>();
if (harnessPlacement != null) {
result.put("harnessPlacement", harnessPlacement.toJson());
}
if (recoveryCleanupPromptId != null) {
result.put("recoveryCleanupPromptId", recoveryCleanupPromptId);
}
packages/sdk-java/qwencode/src/test/java/com/alibaba/qwen/code/daemon/HostedHarnessPoolClientTest.java
+172
View File
@@ -0,0 +1,172 @@
package com.alibaba.qwen.code.daemon;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import com.sun.net.httpserver.HttpsConfigurator;
import com.sun.net.httpserver.HttpsServer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.KeyStore;
import java.time.Duration;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
class HostedHarnessPoolClientTest {
private static final String BOOT = UUID.randomUUID().toString();
private static final String SESSION = UUID.randomUUID().toString();
private static final String OPERATION = UUID.randomUUID().toString();
private static final String DIGEST = "sha256:" + "a".repeat(64);
@Test
void placementSurvivesEveryLoadCopyWithoutChangingLegacyRequests() {
var placement = new HostedHarnessPlacement("instance-a", BOOT, 7);
assertEquals(placement.toJson(), CreateHarnessSession.builder().harnessSessionId(SESSION)
.harnessPlacement(placement).build().toJson().get("harnessPlacement"));
assertFalse(CreateHarnessSession.builder().harnessSessionId(SESSION).build().toJson().containsKey("harnessPlacement"));
var loaded = new LoadHarnessSession(SESSION).withHarnessPlacement(placement)
.withChildWorkspaces(true).withForegroundOnly(true).withStoppedMessages().forLifecycle("close", 1);
assertEquals(placement.toJson(), loaded.toJson().get("harnessPlacement"));
assertFalse(new LoadHarnessSession(SESSION).toJson().containsKey("harnessPlacement"));
assertThrows(IllegalArgumentException.class, () -> new HostedHarnessPlacement("instance", BOOT, 0));
}
@Test
void instanceAndDrainUsePinnedBootAndDoNotAcquireSessions() throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
var drainBody = new AtomicReference<Map<String, Object>>();
server.createContext("/capabilities", exchange -> send(exchange, capabilities()));
server.createContext("/internal/hosted-harness/v1/instance", exchange -> send(exchange, instance("ACCEPTING")));
server.createContext("/internal/hosted-harness/v1/instance:drain", exchange -> {
assertEquals(BOOT, exchange.getRequestHeaders().getFirst("X-Qwen-Harness-Boot-Id"));
drainBody.set(JsonSupport.parseObject(new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8), "drain"));
send(exchange, instance("QUIESCING"));
});
server.start();
try (var client = builder(URI.create("http://127.0.0.1:" + server.getAddress().getPort())).build()) {
assertEquals("instance-a", client.capabilities().getInstanceId());
assertEquals("ACCEPTING", client.readInstance().get("state"));
assertEquals("QUIESCING", client.drainInstance(OPERATION).get("state"));
assertEquals(Map.of("operationId", OPERATION, "expectedBootId", BOOT), drainBody.get());
} finally {
server.stop(0);
}
}
@Test
void releaseChecksItsEpochAndCommittedPrefix() throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
var response = new AtomicReference<>(release(7, "a".repeat(64)));
server.createContext("/capabilities", exchange -> send(exchange, capabilities()));
server.createContext("/session/" + SESSION + "/ownership:release", exchange -> send(exchange, response.get()));
server.start();
try (var client = builder(URI.create("http://127.0.0.1:" + server.getAddress().getPort())).build()) {
var ref = new HarnessSessionRef(SESSION, "client-a", BOOT, "/tmp", null, 0L, null);
assertEquals("RELEASED", client.releaseOwnership(ref, OPERATION, 7).get("state"));
response.set(release(8, "a".repeat(64)));
assertThrows(MutationOutcomeUnknownException.class, () -> client.releaseOwnership(ref, OPERATION, 7));
response.set(release(7, "invalid-hash"));
assertThrows(MutationOutcomeUnknownException.class, () -> client.releaseOwnership(ref, OPERATION, 7));
} finally {
server.stop(0);
}
}
@Test
void explicitTlsContextAuthenticatesPeerAndKeepsHostnameVerification(@TempDir Path directory) throws Exception {
Path storeFile = directory.resolve("identity.p12");
var process = new ProcessBuilder(Path.of(System.getProperty("java.home"), "bin", "keytool").toString(),
"-genkeypair", "-alias", "test", "-keyalg", "RSA", "-keysize", "2048", "-validity", "2",
"-dname", "CN=Harness test", "-ext", "SAN=IP:127.0.0.1", "-storetype", "PKCS12",
"-keystore", storeFile.toString(), "-storepass", "test-password", "-keypass", "test-password", "-noprompt")
.redirectErrorStream(true).start();
assertTrue(process.waitFor(20, TimeUnit.SECONDS));
assertEquals(0, process.exitValue());
KeyStore store = KeyStore.getInstance("PKCS12");
try (var input = Files.newInputStream(storeFile)) {
store.load(input, "test-password".toCharArray());
}
var keys = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
keys.init(store, "test-password".toCharArray());
var trust = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trust.init(store);
SSLContext identity = SSLContext.getInstance("TLS");
identity.init(keys.getKeyManagers(), trust.getTrustManagers(), null);
SSLContext trustOnly = SSLContext.getInstance("TLS");
trustOnly.init(null, trust.getTrustManagers(), null);
HttpsServer server = HttpsServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.setHttpsConfigurator(new HttpsConfigurator(identity) {
@Override
public void configure(com.sun.net.httpserver.HttpsParameters parameters) {
var ssl = getSSLContext().getDefaultSSLParameters();
ssl.setNeedClientAuth(true);
parameters.setSSLParameters(ssl);
}
});
server.createContext("/capabilities", exchange -> send(exchange, capabilities()));
server.start();
URI valid = URI.create("https://127.0.0.1:" + server.getAddress().getPort());
try {
try (var client = builder(valid).sslContext(identity).build()) {
assertEquals(BOOT, client.capabilities().getBootId());
}
assertThrows(DaemonTransportException.class, () -> builder(valid).sslContext(trustOnly).build());
assertThrows(DaemonTransportException.class,
() -> builder(URI.create("https://localhost:" + server.getAddress().getPort())).sslContext(identity).build());
} finally {
server.stop(0);
}
}
private static HostedHarnessClient.Builder builder(URI uri) {
return HostedHarnessClient.builder().baseUri(uri).bearerToken("test-token").capabilityDigest(DIGEST)
.connectTimeout(Duration.ofSeconds(2)).requestTimeout(Duration.ofSeconds(3)).heartbeatInterval(Duration.ZERO);
}
private static String capabilities() {
return "{\"v\":1,\"features\":[\"hosted_harness_private_v1\",\"managed_session_journal_delta_v1\"],"
+ "\"transports\":[\"rest\"],\"hostedHarness\":{\"protocolVersions\":{\"current\":1,\"supported\":[1]},"
+ "\"bootId\":\"" + BOOT + "\",\"instanceId\":\"instance-a\",\"capabilityDigest\":\"" + DIGEST
+ "\",\"features\":[\"harness-pool-owner/1\",\"harness-instance-lifecycle/1\"]}}";
}
private static String instance(String state) {
return "{\"format\":\"qwen-hosted-harness-instance/1\",\"instanceId\":\"instance-a\",\"bootId\":\"" + BOOT
+ "\",\"capabilityDigest\":\"" + DIGEST + "\",\"state\":\"" + state + "\",\"counters\":{"
+ "\"attachedSessions\":0,\"activeTurns\":0,\"openingSessions\":0,\"pendingInputs\":0,"
+ "\"heldResources\":0,\"pendingPublications\":0,\"inFlightMutations\":0}}";
}
private static String release(long epoch, String hash) {
return "{\"format\":\"qwen-hosted-harness-release/1\",\"instanceId\":\"instance-a\",\"bootId\":\"" + BOOT
+ "\",\"operationId\":\"" + OPERATION + "\",\"placementEpoch\":" + epoch
+ ",\"sessionId\":\"" + SESSION + "\",\"writerGeneration\":1,\"journalRevision\":2,\"committedSequence\":3,"
+ "\"commitProof\":{\"lastCommitSequence\":3,\"committedPrefixHash\":\"" + hash + "\"},\"state\":\"RELEASED\"}";
}
private static void send(HttpExchange exchange, String body) throws IOException {
byte[] bytes = body.getBytes(StandardCharsets.UTF_8);
exchange.getResponseHeaders().set("Content-Type", "application/json");
exchange.getResponseHeaders().set("X-Qwen-Harness-Boot-Id", BOOT);
exchange.sendResponseHeaders(exchange.getRequestURI().getPath().endsWith(":drain") ? 202 : 200, bytes.length);
exchange.getResponseBody().write(bytes);
exchange.close();
}
}
scripts/tests/hosted-process-ci.test.js
+11 -6
View File
@@ -580,9 +580,9 @@ describe('Hosted real-process gates', () => {
});
it.each([
// Step ceilings today (198 + 20 + 50 + 20): Verify, the Broker fault
// Step ceilings today (201 + 20 + 50 + 20): Verify, the Broker fault
// gates, admission and O4. The failover arms moved to their own job.
['hosted-harness-mysql', 288, 301],
['hosted-harness-mysql', 291, 304],
// Twenty failover arms at 10 each.
['hosted-failover-e2e', 200, 215],
])(
@@ -834,12 +834,14 @@ describe('Hosted real-process gates', () => {
// (2x its measured 6.5 s), moving the fork to 11236 and leaving the step at
// 198 minutes; the FG12e Runtime connection replacement cell adds 28 s
// (2x its measured 13.8 s), moving the fork to 11264 and leaving the step
// at 198 minutes):
// at 198 minutes). The pool fixture provisionally adds 180 s (its
// 120 s method ceiling plus 60 s for process/context cleanup), moving
// the fork to 11444 and the step to 201 minutes until CI measures it:
// the relational check above tolerates
// anything up to step*60, so the value is pinned literally here —
// an unnoticed raise has no other red to land on.
if (profile === 'hosted-harness-mysql') {
expect(forkSeconds).toBe(11264);
expect(forkSeconds).toBe(11444);
}
},
);
@@ -935,13 +937,15 @@ describe('Hosted real-process gates', () => {
// measured 12.3s), so it becomes 10612s; the cancel-isolation cell adds 11s (2x its measured
// 5.1s), so it becomes 10623s; the queue-one cell adds 13s (2x its measured
// 6.5s), so it becomes 10636s; the FG12e Runtime connection replacement cell
// adds 28s (2x its measured 13.8s), so it becomes 10664s. The
// adds 28s (2x its measured 13.8s), so it becomes 10664s. The pool fixture
// provisionally reserves 180s (its 120s method ceiling plus 60s for
// process/context cleanup), so it becomes 10844s until CI measures it. The
// largest
// ceiling (600s) plus
// that healthy sum still fits the fork. A larger ceiling lets failsafe
// kill the fork mid-run, and the classes left unrun never report, failing
// the gate undiagnosably (#13780).
const hostedForkHealthySeconds = 10664;
const hostedForkHealthySeconds = 10844;
const hostedForkSeconds = () =>
Number(
read('packages/managed-agent-server/pom.xml')
@@ -1010,6 +1014,7 @@ describe('Hosted real-process gates', () => {
'HostedEventStreamLivenessIT',
'HostedExtensionsIT',
'HostedHarnessMySqlIT',
'HostedHarnessPoolMySqlIT',
'HostedHookCancelFaultGatesIT',
'HostedHookFaultGatesIT',
'HostedHookMessageDisplayReloadIT',