Shaojin WenandQwen-Coder
9123d5ee04
* fix(managed-agent): refuse every URL fetch refuses and say why a health probe failed start and verify refused a listener URL only when URL.canParse rejected it, so a --server-url without a scheme (proxy.test:8443), with a user name or password, or on a port the Fetch standard blocks (6000, 10080) passed the check and then failed every probe until --timeout. They now refuse each of those before anything runs, and a test checks that the runtime's fetch rejects every URL the check refuses. A probe that is sent and still fails now says what it got: start's timeout names the last probe's answer and verify's health row names its own, such as "HTTP 503, status DOWN", "self-signed certificate" or ECONNREFUSED, where both said only "not UP". The listeners() comment names the reachesHarness option verify passes. * fix(managed-agent): refuse a Harness URL that is not http or https in upgrade start start checked the Harness URL with new URL() alone, so an opaque value such as proxy.test:8443 parsed with no host and no port: start then waited out --timeout for the Harness to listen on port 80 of an empty host. It now refuses any Harness URL that is not http or https before it starts a unit, as the preflight would fail it later anyway. The probe-answer test also runs the runtime's own fetch against a port nothing listens on, so the ECONNREFUSED wording comes from a real error rather than a constructed one. * fix(managed-agent): keep the endpoint's answer when the deadline cuts start's last probe short start's last health probe gets the 1 ms the deadline leaves it, so against a real endpoint it times out before any answer arrives, and the timeout message reported that abort instead of what the endpoint had been answering, such as HTTP 503, status DOWN. A probe that times out on a budget the deadline cut below the per-probe cap no longer replaces an earlier answer; one that runs out of its own 5 seconds still does, since the endpoint stopped answering. The test rig's fake fetch now awaits a response function, so a test can model an endpoint that answers late and honours the abort signal. Also pins the IPv6-zone diagnosis to a percent inside the brackets. * fix(managed-agent): treat a health body the deadline stalls as a cut-short probe too A health endpoint that sends its headers and then stalls the body made response.json() reject with the signal's TimeoutError. That branch did not report timedOut, so start's guard took the probe for an answer, and a probe the deadline cut short replaced an earlier HTTP 503, status DOWN with "HTTP 200, The operation was aborted due to timeout". The body-read branch now reports timedOut as well. The last-answer test gains two real-socket cases: a body that stalls after its headers keeps the earlier answer, and a refused connection after an answer replaces it, which pins the timedOut half of the guard. The design doc says when the 5-second limit counts as the probe's own, and that any other failure replaces an earlier answer. * fix(managed-agent): sanitize health text and align upgrade refusals Address review R1-1..R1-5 on the upgrade URL/health change: - R1-1: neutralize terminal control bytes (C0/DEL/C1 -> U+FFFD) in the endpoint-derived health answer before it is interpolated into start's timeout refusal and verify's health row, so a queryable endpoint cannot move the cursor, split a row, or write the clipboard (OSC-52). - R1-2: the Harness URL refusal reuses fetchRefusal when the value does not parse, naming an IPv6 zone exactly as --server-url does. - R1-3: start's timeout labels the retained value 'the last answer', matching the design doc, since a deadline-cut probe keeps an earlier answer. - R1-4: pin verify's URL refusal (listeners runs it with reachesHarness:false) with a test asserting it refuses before any probe. - R1-5: tear down the 'closes' test server via t.after so a failing assertion cannot hang node --test to the job ceiling. Tests: 63 pass; R1-1/R1-2/R1-4 mutation-witnessed (printable no-op, refusal gated on reachesHarness, zone reason reverted each redden their pin). Prettier and ESLint clean. Co-authored-by: Qwen-Coder <[email protected]> * test(managed-agent): exercise the health sanitization over a real socket Add a real-scenario pin for R1-1: a local HTTP server serves a status field carrying cursor moves, a newline and an OSC-52 clipboard write, and verify runs against it through the runtime's own fetch — the real health() -> printable() -> row() -> out() path, not the rig's fake. Asserts the printed health row keeps no control byte while the readable text survives. 64 tests pass. Co-authored-by: Qwen-Coder <[email protected]> * fix(managed-agent): mask a refused URL's password and name why verify's API reads failed Found on a real systemd deployment (real control plane, Harness, MySQL, TLS and nginx): - A refused --server-url was printed as typed, so a URL with a password, which fetch refuses, put that password in the output. The refusal now prints it as ***. - verify's session and turn rows still said only "fetch failed" where the health row above them named the cause, such as "self-signed certificate in certificate chain". API reads now take the same cause. - Text the API hands back (an error reason, a body that fails to parse, a Session ID, an event type) reached the session and turn rows with its control bytes; it is now neutralized as the health answer is. The design doc says the probe answer's control characters show as U+FFFD. * fix(managed-agent): show a health status that is no string as its JSON printable() ran String() on what the endpoint returned, and String() throws on a valid JSON object whose own toString is no function, such as {"status":{"toString":1}}. health() is not inside a try, so on a real deployment verify exited with "Cannot convert object to primitive value" before it printed its health row, and start's probe loop stopped at the first probe. A value that is no string is now shown as its JSON, which is also what the endpoint actually said; an API error reason gets the same. * fix(managed-agent): make the upgrade reports total and bounded, and refuse only what a phase dials Review round 5 traced its findings to two shared causes, fixed at the source rather than one site at a time. Endpoint text in the reports (printable, withoutCredentials): - printable() no longer throws on any parsed JSON value: a status nested deeper than JSON.stringify() recurses showed a bare "Maximum call stack size exceeded" from verify and left start's server running without its guidance; it is now shown as [unprintable]. - It also stops at 200 code points, cut after the control bytes are gone and never inside a surrogate pair, so a long status cannot scroll the report away. The item count on the session row goes through it too. - withoutCredentials() masks the whole user information, so a token in the user-name position is masked, and masks it textually when the URL does not parse. The Harness refusal masks its URL as well. URL checks (listeners, the Harness guard): - A phase is refused only over a URL it dials: start and verify probe the health endpoint, and only verify --tenant reads the API. An API on a port fetch refuses behind a separate management port no longer stops start. - The Harness URL is checked as the preflight checks it, an http(s) origin without user information, a query or a fragment, so start refuses it before it restarts a running Harness rather than after. * fix(managed-agent): take the Harness verdict from the preflight and check the API URL only when verify reads it - verify decides whether its session row reads the API before it checks the listeners, so a signed deployment, or --actor without a trusted actor header, is no longer refused over an API URL it never dials. - The configuration resolver now prints the preflight's own verdict on the Harness URL, read untrimmed as java.net.URI reads it, and start refuses on it before restarting the Harness: a host with an underscore, an empty user name or a padded value no longer restart the Harness only to fail the preflight. - withoutCredentials masks everything up to the last @ of a value the parser rejects, so a password holding / ? or # stays hidden. - printable counts code points in one pass and keeps only the head, so a huge endpoint answer costs no copy of itself. - The request an API error names is bounded like the rest of the row. * fix(managed-agent): print the Harness URL untrimmed, as the server parses it A padded qwen.managed-agent.harness.base-url fails the preflight, and the start refusal now shows the padding instead of a value that looks valid. * fix(managed-agent): judge the Harness URL only where the server reads it The resolver passed on the preflight's verdict on the Harness base URL with no regard to the Harness's state, so start refused a value that a disabled or pooled Harness leaves unread: the preflight skips its harness row before reading it, and the server never dials it. The verdict is now "skip" there. start refuses on the preflight's grounds (user information, a query, a fragment, or anything java.net.URI reads differently) only on a "false" verdict, and its own reasons only word that verdict. It still refuses a URL that is not http or https, since it waits for the Harness there. Without a verdict, as when the configuration does not resolve, the preflight judges the URL once the Harness is up. verify now checks the type of the values the API chose before using them. Building a route from a Session id, matching an event type, or reading an event sequence coerced the value first, and that throws on an object such as {"toString":1} or on an array nested deeper than the stack, so printable() never got to show what came back. A Session without a string id fails its row with that value, an event type that is not a string fails the turn row, and a sequence counts only as a number or a numeric string. The start tests now check every run's output for the password, not only the last run's. * fix(managed-agent): judge a pooled Harness URL, and read only lists from the API The previous commit gave "skip" for a pooled Harness as well as a disabled one, since the preflight skips its harness row in both before reading the base URL. A pooled server still reads that URL, though: each member's QwenHostedHarnessConnector is built from it and refuses it by the same rule as the preflight. The pool's refresh drops a member whose connector refuses it, so such a pool accepts no Session while the server reports UP. Only a disabled Harness gets "skip" now. A test pins the verdict to whether the connector builds, for a single Harness and for a pool. Since the verdict is now the server's rather than only the preflight's, the refusal says "the server does not take". verify reads Sessions, items and events only as lists. Data that is not a list fails its row and says what came back: iterating it would throw, or walk a string, and its length could be any value. A listed entry of 0, "", false or null fails the session row instead of being reported as no Session. An event sequence moves the cursor only as a whole number, and only forward: Number() reads "Infinity", "0x10" or " 4 " as numbers. The user guide now says that, when the configuration does not resolve, the preflight judges the Harness URL after the Harness starts. * fix(managed-agent): keep start's own Harness URL rules without a verdict When the configuration does not resolve there is no verdict. The previous commits then let start accept any http(s) Harness URL and left the URL to the preflight. A pooled preflight never reads it, though, and every pool member's connector refuses user information, a query or a fragment, so the pool would start and accept no Session. Without a verdict, start refuses those itself again, since the server refuses them wherever it reads the value. A "true" or "skip" verdict still asks nothing more, and the generic wording is kept for a "false" verdict. HarnessConfigurationTest now builds a pool through the real member factory and shows that the single base URL decides whether a member is accepted. The pooled half of the agreement test checked nothing the single half did not, so it is gone. verify now requires every page to carry its data as a list: a body that is no object, or one whose data is null or missing, fails the row instead of reading as no Session. A Session id of "", "." or ".." fails, since fetch resolves dot segments and an empty id reads the list. Only terminal: true ends a turn. The docs no longer say the server takes only an origin, since a path is fine. They say which component refuses the URL, and they word the sequence rule as the code applies it. * fix(managed-agent): sort the test's static imports, and refuse a --server-url query The Mockito static import sat between two JUnit ones, so the verification module's ImportOrder check failed. The static block is now sorted. start and verify append /actuator/health and the API paths to --server-url, so a query or a fragment there took the paths in. With http://proxy.test/?a the command probed /?a/actuator/health, and fetch dropped a fragment along with the path after it. Such a URL is now refused up front. A listed Session entry with no id is now shown as itself rather than as "undefined". A health answer whose JSON parse error quotes control bytes is pinned to come out sanitized. --------- Co-authored-by: Qwen-Coder <[email protected]>
42 KiB
Managed Agent Reference Deployment
The Managed Agent reference distribution is experimental. Its accepted initial topology is one Java control plane and its embedded Runtime Broker on one trusted Linux host, one MySQL database, and one compatible Qwen Hosted Harness. The compatibility manifest in each release is authoritative for that release.
Choose a tier
Each tier adds to the one before it:
| Tier | What runs | What it adds |
|---|---|---|
| 1. Text-only Hosted Sessions | The Java control plane, one Hosted Harness and MySQL on one trusted Linux host | — |
| 2. Local-process tools | Hosted tools on the Workspace, such as file access and, when enabled, a foreground Shell | The embedded Runtime Broker, Runtime workers on the same host, the Runtime credential key, a Runtime state directory and a registered Workspace |
| 3. Kubernetes | The API, the Harness and MySQL in a cluster | Container images built from source, installed with a Helm chart, outside this distribution's acceptance scope |
Start at Tier 1 even when tools are the goal. A Session that completes a no-tool Turn proves the control plane, the Harness, the model and the database together, and Tier 2 then enables tools on the same deployment.
Tier 1: text-only Hosted Sessions
One control plane, one Harness and MySQL on one host. The steps run in order: verify the archive, generate the credentials, start the Harness, check the configuration, start the control plane, and complete a first Session.
Verify and extract
Download the archive, its detached manifest, and SHA256SUMS.txt from the same GitHub prerelease. Verify them before extraction:
sha256sum --check SHA256SUMS.txt
tar -xzf qwen-managed-agent-reference-<version>.tar.gz
cd qwen-managed-agent-reference-<version>
sha256sum --check SHA256SUMS
Inspect manifest.json. Confirm that source.revision is the reviewed revision, compatibility.qwenCodeVersion matches the CLI/Harness installation, and compatibility.databaseSchema.latestMigrationVersion is acceptable for the target database. The release is invalid if any checksum or compatibility check fails.
Prerequisites
- Linux on the trusted application host.
- JDK 21 or later.
- Node.js matching
compatibility.nodeand Qwen Code matchingcompatibility.qwenCodeVersion. - MySQL 8 with an empty database or a database previously managed by an older compatible reference distribution.
- A model endpoint for the Harness, such as an OpenAI-compatible one with its model ID and API key.
- A private, persistent Workspace directory, and for Tier 2 a Runtime state directory.
- Stable database credentials and Harness token, and for Tier 2 the Runtime credential key ring, stored outside the archive.
The Java archive does not contain the npm CLI. Install or build the exact compatible Qwen Code revision separately: the Harness runs it as qwen serve, and Tier 2's Runtime workers run its dist/cli.js.
Bootstrap credentials
bin/qwen-managed-agent-bootstrap generates the shared Harness token and capability digest, the broker token, the Runtime credential key and the database password. It writes config/harness.env, config/server.env, config/application.yml and config/mysql-setup.sql with mode 0600, refuses to overwrite existing files unless --force is given, and prints paths only:
bin/qwen-managed-agent-bootstrap
The token and digest, and the Broker URL and token for the tool path, are written only to harness.env: server.env names that file through QWEN_MANAGED_AGENT_HARNESS_CREDENTIALS_FILE, and the control plane reads them from it. It refuses to start when the file is set together with QWEN_MANAGED_AGENT_HARNESS_TOKEN or QWEN_MANAGED_AGENT_CAPABILITY_DIGEST, or when the file's Broker token is set together with QWEN_MANAGED_AGENT_RUNTIME_BROKER_TOKEN, so a rotation changes one file. The generated harness.env and server.env are shell-sourceable and also valid systemd EnvironmentFiles.
server.env names /srv/qwen-workspace as the Workspace directory in QWEN_MANAGED_AGENT_WORKSPACE_CWD; create it, or point the variable at an existing directory. The Session Store derives its workspace ID from that directory, so keep it once Sessions exist.
mysql-setup.sql creates the qwen_managed_agent database and the qwen account that server.env names, for both localhost and 127.0.0.1, with the performance_schema grant Flyway needs. Run it once as a MySQL administrator, for example mysql -u root < config/mysql-setup.sql, then delete it; server.env keeps the password. It stops at the first statement that fails, so it never changes an existing database or account. A MySQL in a container sees the gateway address rather than 127.0.0.1; create the account for that host instead.
For the systemd layout, run it as root with --directory /etc/qwen-managed-agent --service-user qwen-managed-agent. The account then owns harness.env and application.yml, server.env is 0640 root:qwen-managed-agent, a new directory is 0750 root:qwen-managed-agent, and mysql-setup.sql stays root's. The bootstrap refuses an account that does not exist, or a directory the account cannot enter. server.env also records QWEN_MANAGED_AGENT_SERVICE_USER, so the preflight checks file access as that account even when it runs as root.
Start the Hosted Harness
Start the Harness with config/harness.env, which holds its token, capability digest and Broker pair, and with the model it calls. Qwen Code reads an OpenAI-compatible provider from OPENAI_API_KEY, OPENAI_BASE_URL and OPENAI_MODEL; the authentication guide covers the others.
set -a; . config/harness.env; set +a
export OPENAI_API_KEY='<model-api-key>'
export OPENAI_BASE_URL='https://<openai-compatible-endpoint>/v1'
export OPENAI_MODEL='<model-id>'
qwen serve --profile hosted-harness --port 4170 --hostname 127.0.0.1 --no-web
On the systemd layout, put the three model variables in harness.env as plain KEY=value lines instead.
Keep the Harness on loopback. The Hosted Harness profile does not move to another port: when --port is in use, the Harness fails to start, naming the port.
Without the bootstrap's files, choose one token and capability digest and give the same values to the Harness and the Java control plane:
export QWEN_MANAGED_AGENT_HARNESS_TOKEN='<private-random-token>'
export QWEN_MANAGED_AGENT_CAPABILITY_DIGEST='sha256:<64-hex-digest>'
QWEN_SERVER_TOKEN="$QWEN_MANAGED_AGENT_HARNESS_TOKEN" \
QWEN_HOSTED_HARNESS_CAPABILITY_DIGEST="$QWEN_MANAGED_AGENT_CAPABILITY_DIGEST" \
qwen serve --profile hosted-harness --port 4170 --hostname 127.0.0.1 --no-web
Preflight check
In a second terminal, load config/server.env, or without the bootstrap's files export the variables listed under the control plane below. With the Harness running, run the bundled launcher with --check before the first start and after installing each new artifact. It binds the same configuration (config/application.yml plus the environment), starts no listener, and prints one row per input:
set -a; . config/server.env; set +a
bin/qwen-managed-agent-server --check
It covers the JDK, the service account, the configuration file and its permissions, the credential key ring, the startup property validation, the auth/bind combination, the Harness, the Workspace files and Shell cross-settings (including the mount, creation-root or SSH-node requirement), broker credentials and directories, the Workspace cwd, free ports, Runtime workers an earlier run left alive, and MySQL connectivity, the performance_schema grant and the Flyway version against manifest.json. WARN and SKIP do not fail; any FAIL exits 1.
The harness row negotiates with a single configured Harness as the first Session does; with a Harness pool it skips the pool's endpoints once the Session Store is enabled with a binding key, and fails until then. It fails when the capability digest differs, the protocol or a required feature is missing, or the Harness lacks the lifecycle a configured Workspace profile needs. It warns when the Harness runs another Qwen Code version than manifest.json names, or does not say which. The port checks bind the configured addresses the way the server does, so a port the stopped server left in TIME_WAIT counts as free, and the workers row warns about workers still running in the Runtime state directory, so stop the service first. While it runs, add --online: the port rows then check that something listens, and the workers row reports only workers a stopped server left behind, or is skipped for a durable Broker, which adopts them. The command does not verify the native Workspace storage registration.
Before the first start, the database is empty: the database row reports its schema as not migrated yet, and the capacity row warns that its tables do not exist. The first start creates them, and both rows then pass.
The database row warns when the database's Flyway version is older than manifest.json names: that is an upgraded database before the first start of the new release, which migrates it. It fails when the database is newer. At the version manifest.json names, it validates the schema as the server does at startup, and fails when the database does not match, as when an experimental release changed its schema baseline in place.
Start the Java control plane
In the same terminal, start the control plane:
bin/qwen-managed-agent-server
Without the bootstrap's files, configure the same Harness, the database and an existing Workspace directory:
export SPRING_DATASOURCE_URL='jdbc:mysql://127.0.0.1:3306/qwen_managed_agent'
export SPRING_DATASOURCE_USERNAME='qwen'
export SPRING_DATASOURCE_PASSWORD='<private-password>'
export QWEN_MANAGED_AGENT_HARNESS_ENABLED='true'
export QWEN_MANAGED_AGENT_HARNESS_BASE_URL='http://127.0.0.1:4170'
export QWEN_MANAGED_AGENT_HARNESS_TOKEN='<private-random-token>'
export QWEN_MANAGED_AGENT_CAPABILITY_DIGEST='sha256:<64-hex-digest>'
export QWEN_MANAGED_AGENT_WORKSPACE_CWD='/srv/qwen-workspace'
bin/qwen-managed-agent-server
Settings that cannot be expressed as environment variables, such as the Runtime credential key ring, belong in config/application.yml. The bootstrap writes it; otherwise start from the bundled template and keep the file private:
cp config/application.example.yml config/application.yml
chmod 600 config/application.yml
Both launchers load that file automatically when it exists. The launcher uses the server JAR in this archive. Spring Flyway applies forward database migrations before the service becomes ready. Wait for:
curl --fail http://127.0.0.1:8080/actuator/health
The enabled Harness turns on the Managed Session Store. The Store's base URL defaults to this server's plaintext loopback listener. Its workspace ID defaults to the one the Runtime Broker derives from QWEN_MANAGED_AGENT_WORKSPACE_CWD, so the identity stays the same when Tier 2 enables the tool path. A deployment that already set QWEN_MANAGED_AGENT_WORKSPACE_ID must keep it: unbound Sessions resolve their Store scope from that ID. Set QWEN_MANAGED_AGENT_SESSION_STORE_BASE_URL only when the Harness reaches this server through another address, or when the listener serves TLS.
This base configuration supports text-only Hosted Sessions. Keep the listener on loopback unless a signed or authenticated ingress is configured. Every setting, with its variable and default, is listed in the configuration reference, bundled as docs/configuration.md. The complete authentication, API, and capability configuration is bundled as docs/server-reference.md.
Run a first Session
With the default loopback listener, the API takes the tenant from the X-Qwen-Tenant-Id header. Create a Session whose first Turn asks the model a question:
key="first-session-$(date +%s)"
curl --fail -X POST http://127.0.0.1:8080/v1/agents/sessions \
-H 'X-Qwen-Tenant-Id: default' -H 'Content-Type: application/json' \
-H "Idempotency-Key: $key" \
-d '{"agent_id":"qwen-code","input":[{"type":"input_text","text":"Say hello."}]}'
The Idempotency-Key header is required. Use a new key for each check, and send the same one again only to retry that request: the same key returns the same Session instead of creating another, so a later check with an old key replays the old Session and its old result. The response carries the Session's id. Read its events, starting with after=0:
curl --fail -H 'X-Qwen-Tenant-Id: default' \
'http://127.0.0.1:8080/v1/agents/sessions/<id>/events?after=<after>&limit=100'
A page holds at most limit events, in sequence order. Set after to the sequence of the last event received, keep it when a page comes back empty, and poll again until an event has "terminal": true.
A Turn that succeeds ends with turn.completed, and the model's reply is the text of the item.output_text.delta events before it. A terminal turn.failed, or none before the Turn deadline, points at the Harness or the model: the Harness log names the cause. This Session is the deployment's first proof; repeat it after every upgrade, as verify --turn does on the systemd layout.
systemd
The distribution includes systemd/ unit and environment templates for the accepted single-host topology. The control-plane unit uses KillMode=process so Runtime workers survive a Broker restart; the Harness unit uses KillMode=mixed so its tool processes terminate with it. Follow systemd/README.md to install them.
Tier 2: local-process tools
Tier 2 adds Hosted tools to a working Tier 1 deployment. The embedded Runtime Broker places Runtime workers, Node.js processes on the same host that run the tools on the Workspace. Tool execution requires the existing fail-closed Runtime and Workspace setup. On top of the Tier 1 configuration, at minimum, configure:
export QWEN_MANAGED_AGENT_RUNTIME_BROKER_ENABLED='true'
# Omit when harness.env carries QWEN_MANAGED_RUNTIME_BROKER_TOKEN, as the bootstrap's does.
export QWEN_MANAGED_AGENT_RUNTIME_BROKER_TOKEN='<private-random-token>'
export QWEN_MANAGED_AGENT_RUNTIME_CREDENTIAL_KEY_ID='reference-v1'
export QWEN_MANAGED_AGENT_RUNTIME_CREDENTIAL_KEY='<base64-encoded-32-byte-key>'
export QWEN_MANAGED_AGENT_RUNTIME_STATE_DIRECTORY='/var/lib/qwen-managed-agent/runtime'
export QWEN_MANAGED_AGENT_NODE_EXECUTABLE='/usr/bin/node'
export QWEN_MANAGED_AGENT_RUNTIME_WORKER_ENTRY='/opt/qwen-code/dist/cli.js'
export QWEN_MANAGED_AGENT_RUNTIME_VERIFIED_WORKSPACE_RECOVERY_ENABLED='true'
The Hosted Harness reaches the same Broker through its own environment, not its command line and not the control plane's variables above: set QWEN_MANAGED_RUNTIME_BROKER_URL='http://127.0.0.1:4182' and QWEN_MANAGED_RUNTIME_BROKER_TOKEN to the Broker token in harness.env, or in whatever environment starts the Harness. qwen serve removes both from its environment once read, so no process the Harness starts inherits the token, and the control plane keeps the QWEN_MANAGED_RUNTIME_BROKER_ variables out of Runtime workers. With the bootstrap's files, harness.env already holds both, and the control plane takes its Broker token from that file, so leave QWEN_MANAGED_AGENT_RUNTIME_BROKER_TOKEN unset; setting both copies refuses startup. Set the two Harness lines together or not at all, each as a plain KEY=value line: startup refuses a file with only one of them or with either one empty, which the Harness also refuses, and a line such as export KEY=value, which systemd ignores. The preflight broker-url row reads the file's URL the way the Harness's URL parser does, so http://127.1:4182 counts as http://127.0.0.1:4182. It passes when that URL names the Broker's own listener over http. It fails when the Harness would refuse the URL, which must be an http or https origin and may use plain http only on the loopback interface, and when it names the listener over https, which the plain-HTTP Broker cannot accept. It warns when the URL names another address or port, which only a proxy could forward; a Broker bound to a specific address outside loopback, which needs QWEN_MANAGED_AGENT_RUNTIME_BROKER_ALLOW_NON_LOOPBACK=true, is always reached that way. A Broker bound to 0.0.0.0 needs that setting too, but still serves loopback, where the Harness dials it directly at http://127.0.0.1:4182 and the row passes. An IPv6 URL such as http://[::1]:4182 warns there, because that bind takes IPv6 only when the control plane's JVM listens on both families, which it does not under -Djava.net.preferIPv4Stack=true. It also warns when the file has no Broker URL, which is what a Harness given the pair through its own environment or command line looks like to preflight; there the warning is expected, and moving the pair into the file means deleting QWEN_MANAGED_AGENT_RUNTIME_BROKER_TOKEN from server.env.
Before admitting file, Shell, or child Git work, complete the native storage registration and mount-identity verification described under “Verified physical Workspace recovery” in docs/server-reference.md. An unregistered or ambiguous mount must remain unavailable. Public Shell execution remains disabled by default. Shell lifecycle v2 is advertised by default; an explicit lifecycle list replaces the default, and an empty list disables it. MCP lifecycle still requires explicit inclusion. Capability advertisement does not establish deployment qualification.
Tier 3: Kubernetes
The Helm chart installs the signed Managed Agent API and a private Hosted Harness in one Pod, with MySQL 8.4 on a persistent volume, from images you build at a reviewed source revision. Its default runs text Sessions; optional settings add trusted single-node Workspace files and a foreground Shell, and a Managed WebShell gateway. It is installed from source rather than from this archive, follows its own guide, and does not extend this distribution's acceptance scope. K2 CSI resident Workspaces are a separate deployment, described in deploy/k2-resident.
Operate the deployment
Maintenance commands
bin/qwen-managed-agent-admin <command> [arguments] runs the offline maintenance entry points, such as storage-registration, workspace-migration and the CSI commands, from the server jar through Spring Boot's PropertiesLauncher. Run it without arguments to list the commands, and run a command without arguments for its usage. Like the server launcher, it uses config/application.yml when that file exists and SPRING_CONFIG_ADDITIONAL_LOCATION is not set. Most commands that open the database take their own credentials rather than server.env's SPRING_DATASOURCE_*: the Workspace commands read W1_JDBC_URL, W1_JDBC_USER and W1_JDBC_PASSWORD, and the CSI commands read K2_JDBC_URL, K2_JDBC_USER and K2_JDBC_PASSWORD. Two are exceptions: operator-recovery and resident-retirement start the server's Spring configuration and use its datasource, so they read SPRING_DATASOURCE_URL, SPRING_DATASOURCE_USERNAME and SPRING_DATASOURCE_PASSWORD, together with the other control-plane settings, from the environment and config/application.yml; run them with server.env loaded. csi-protection opens no database, and neither does resolved-configuration, which prints the Runtime state directory, auth mode and trusted actor header that the server's configuration resolves to in the environment it runs in, and the URLs of the control plane, its health endpoint and the Hosted Harness; the upgrade command runs it the same way to find the Runtime state and those URLs. Each command is an offline procedure with its own preconditions, documented with it; meet them before running it. bin/qwen-managed-agent-operator-recovery remains the entry for the Hosted Shell recovery procedure, and bin/qwen-managed-agent-admin operator-recovery runs the same command.
Upgrade
On the systemd layout, the new distribution's bin/qwen-managed-agent-upgrade runs the upgrade in phases. Run each as root from the extracted and verified archive:
cd qwen-managed-agent-reference-<new-version>
# Pass check the options backup gets, such as --skip-database.
bin/qwen-managed-agent-upgrade check
# Stop new ingress and let admitted work settle.
bin/qwen-managed-agent-upgrade stop
bin/qwen-managed-agent-upgrade backup --to /var/backups/qwen-managed-agent/<date>
bin/qwen-managed-agent-upgrade install --backup /var/backups/qwen-managed-agent/<date>
# The manual steps check printed, such as installing Qwen Code for the Harness.
bin/qwen-managed-agent-upgrade start
bin/qwen-managed-agent-upgrade verify --tenant <tenant> --turn
# Reopen ingress.
Each phase reads the units, their environment files and the installed release again, and refuses unless the previous phase left them as it expects, so an interrupted upgrade resumes at the phase that failed. --server-unit and --harness-unit name units other than qwen-managed-agent-server and qwen-hosted-harness; the command finds the installed release, the environment files and the service account through them.
checkchanges nothing. It verifies the new distribution against itsSHA256SUMS, compares both releases (version, the Flyway scripts in the server jars, Java, Node.js and Qwen Code), and reads the Harness Broker settings, the unit templates, operator files in the installedconfig/and the release layout. Itsbackup-toolsandruntime-staterows tell whetherbackupwill run with the options given, so passcheckthe backup options too; the plan it prints repeats them. It prints one row per input and a plan with the manual steps where they fall, and exits 1 on any FAIL. Thedatabaserow fails when the new release changed or removed a script the installed database ran: until the first formal release, a release may rewrite its schema baseline in place, and such a release cannot start on the existing database. It also fails when the release adds a versioned script that does not sort after the newest one the installed database ran, which Flyway refuses to apply.stoprepeats the checks abovebackup-toolsand stops nothing unless they pass. Thebackup-toolsandruntime-staterows gate onlybackup, so settle them withcheckbeforestop. It stops the control plane, stops the Runtime workers thatKillMode=processleaves in its cgroup and waits for them (--timeout, 60 seconds by default), then stops the Harness. It counts the workers in the cgroup systemd names, in the unified hierarchy or on cgroup v1 in systemd's own, and refuses when it cannot read that cgroup. Once the new release is installed,stopskips the checks, so it also stops a control plane thatstartleft running. The JVM exits 143 onSIGTERM, so a control-plane unit installed without the template'sSuccessExitStatus=143shows asfailedafter the stop;stopsays so, and once the template's change is carried over,systemctl reset-failedclears the state.backup --to <new empty directory>needs both units stopped and no worker left, and refuses once the new release is installed, because itsfrom.releasewould then name the new release. A directory that already exists must be empty and belong to root. It writesdatabase.sqlwithmysqldump, passing the unit'sSPRING_DATASOURCE_*credentials through an option file in a temporary directory outside the backup rather than on the command line;mysqldumpreads no other option file, so root's~/.my.cnfcannot replace them. It writesruntime-state.tarof the Runtime state directory,configuration.tarof the unit files, their environment files, the Harness credentials file and theSPRING_CONFIG_ADDITIONAL_LOCATIONfiles, andrelease-config.tarof the installedconfig/and any of those files that live in the release, recorded by their path in the release. A file that is a symbolic link goes in as the link and as the file it names. Every file is0600in a0700directory, andbackup.json, written last, records their checksums, the database the dump is of, and whereinstallleaves the previous release.--mysqldump <path>names another client, and--skip-databaseskips the dump when the database is backed up another way. A backup that fails, or thatSIGINT,SIGTERMorSIGHUPends, removes what it wrote and the option file. ASIGKILLremoves nothing; remove the${TMPDIR:-/tmp}/qwen-managed-agent-upgrade-*directory it leaves, which holds the credentials.checkandbackuptake the Runtime state directory from the server's own configuration. They run the new release's server jar as the control-plane unit runs the server: with its account, working directory and environment, and the launcher's defaultconfig/application.ymlwhenSPRING_CONFIG_ADDITIONAL_LOCATIONis unset or empty. The jar prepares Spring's environment the way the server does, so every configuration file, import and profile counts, as doSPRING_APPLICATION_JSON, system properties fromJAVA_TOOL_OPTIONSand relaxed names, and a relative directory is resolved in the unit's working directory. It starts nothing and opens no connection. When it cannot resolve the configuration, or the unit passes the launcher arguments, whichsystemctldoes not print exactly,backuprefuses until--runtime-state <directory>names the directory the server uses, or--runtime-state nonesays it keeps none. Without a configured directory, as on a deployment without the tool path,backuparchives no Runtime state and says so.
install --backup <directory>refuses a backup of another release, a backup file that changed, a backup the control plane has started since, and a backup of another database than the unit'sSPRING_DATASOURCE_URLnames. It copies the release to<distribution>-releases/<version>, such as/opt/qwen-managed-agent-releases/<version>(--releasesnames another directory; pass the same one to every phase), copies operator files from the installedconfig/with their owner and mode, and points the distribution path at the new release with one rename. The copy's own files, thoseSHA256SUMSlists, are made readable by every account, so the service account can run a release extracted with a restrictive umask. The first run moves the installed directory to<releases>/<old version>and puts a link in its place, so the units keep their paths. The previous release stays where it is. An existing<releases>/<version>, such as the archive extracted there, is used only when itsSHA256SUMSis the new distribution's and its files match it, and its listed files are made readable the same way; a copy that fails is removed. A crash between the first run's two renames leaves<distribution>.upgrade-<suffix>beside the missing distribution path; every phase then names themv -Tthat finishes the install. An install killed at another step can leave a partial copy.<version>.partial-<suffix>in the releases directory, an<distribution>.upgrade-<suffix>link beside the distribution path, or an operator file's.tmp-<suffix>copy in the new release'sconfig/; thelayoutrow lists them, and the nextinstallremoves them. The suffix is the eight hexadecimal digitsinstallgenerates, so an operator's own<distribution>.upgrade-backupis left alone. Run again once the release is installed, as after an interrupted session,installsays so and points atstart.startrefuses beforeinstall, and while the Harness Broker settings still need step 5 below. It starts the Harness, or restarts one that an earlierstartleft running so that it reads what was fixed since, runsbin/qwen-managed-agent-server --checkas the service account with the unit's environment, starts the control plane only when that passes, and waits for a healthy actuator (--timeout, 300 seconds by default; one probe waits at most 5 seconds). Flyway migrates the schema during this start. A control plane that exits before it is healthy is stopped and reported, even one that systemd'sRestart=on-failurewould bring up when it runs it again: an exit during the start that migrates the schema is worth reading in the journal beforestartruns again. One that is still not healthy at the timeout keeps running; read its journal, then runverify, orstopandstartagain.startandverifyfind the Harness, the control plane and its health endpoint the same way, so a port or address set whereserver.envdoes not show it counts: in a configuration file, on the command line or inSPRING_APPLICATION_JSON. The Harness is whereqwen.managed-agent.harness.base-urlsays. The control plane is whereserver.port,server.addressandserver.sslbind it, with a wildcard address reached on the loopback interface, and the health endpoint follows Spring Boot's management settings: amanagement.server.portof its own moves it to that port, onmanagement.server.addressand undermanagement.server.base-path.--server-url <url>names the control plane instead, for example behind a proxy, and the health endpoint is then<url>/actuator/health. A phase checks only the URLs it dials: both probe the health endpoint, andverify --tenantalso reads the API when the auth mode isopen. A port of 0 or below gives no URL to wait on, so the phase refuses until--server-urlnames one; it also refuses a URLfetchrefuses to dial: a link-local IPv6 address with its zone, a port the Fetch standard blocks (6000, for one), or a--server-urlwithouthttp://orhttps://, with a user name or password, which the refusal prints as***, or with a query or a fragment, which the paths appended to it would land in;startlikewise refuses a Harness URL that is nothttporhttps, and, while the Harness is enabled, one the server's Harness connector would refuse: anything but anhttporhttpsURL with a host and without credentials, a query or a fragment, as the server reads it withjava.net.URI, so a host name with an underscore is refused too; when the configuration does not resolve,startitself refuses credentials, a query or a fragment and leaves the rest to the preflight. When the health endpoint does not report UP,startandverifysay what it answered last; for a certificate this host does not trust, pointNODE_EXTRA_CA_CERTSat its CA when you run the command. When the configuration does not resolve, or the unit passes the launcher arguments, they use the URLsserver.envnames and print a line that says so.
verifyruns--check --onlinethe same way and reads the actuator, waiting up to--timeout(300 seconds by default) for its answer. With--tenant <id>, it reads the tenant's most recent Session and its Items, before--turncreates one;--turncreates a Session and waits for one Turn, which calls the model, polling past a5xxanswer or a dropped connection; when the Session read is skipped, theturnrow says so. No request outlasts--timeout. A request without an actor lists only Sessions outside Workspaces;--actor <id>sends one through the headerQWEN_MANAGED_AGENT_TRUSTED_ACTOR_HEADERnames.verifyreads the auth mode the waybackupreads the Runtime state directory, and signed authentication skips both reads; read a Session through the ingress instead.
The command starts and stops units only. It never stops ingress, installs Qwen Code, edits environment or unit files, or deletes a release. When check warns that a unit template changed, carry the change over to the installed unit and run systemctl daemon-reload before start.
Manual procedure
Other layouts follow these steps. On the systemd layout the command runs stop, backup, install, start and verify, and check prints the steps it leaves to the operator where they fall: stopping and reopening ingress, installing Qwen Code and editing harness.env in step 5, and carrying changed unit templates over.
- Download and verify the new archive and read both manifests. Confirm the new Qwen Code/Harness, Java, protocol, and database requirements before touching the running deployment.
- Stop new ingress. Wait for admitted work to settle according to the deployment's existing drain procedure.
- Back up MySQL, the Runtime state directory, the complete Runtime credential key ring, the operator-owned
server.envandharness.env, and the exact old distribution. Do not rotate the credential key during the distribution upgrade. Withserver.envloaded,bin/qwen-managed-agent-admin resolved-configurationprints the Runtime state directory the configuration resolves to. - Stop the Java service and Hosted Harness. Preserve the Workspace and durable Runtime state. The shipped control-plane unit uses
KillMode=process, sosystemctl stopleaves Runtime workers running; reap them withsystemctl kill --kill-whom=all -s TERM qwen-managed-agent-serverbefore starting the new version. - Install the manifest-compatible Qwen Code/Harness and start it with the existing token and capability digest. On a tool-path host, first replace any
QWEN_HOSTED_HARNESS_BROKER_ARGSline inharness.envwithQWEN_MANAGED_RUNTIME_BROKER_URLandQWEN_MANAGED_RUNTIME_BROKER_TOKEN: nothing reads that line any more, so without them the Harness starts with no Broker. Whenserver.envenables the Harness and namesharness.envas its credentials file, as the bootstrap's files do, also deleteQWEN_MANAGED_AGENT_RUNTIME_BROKER_TOKENfromserver.env; the control plane then takes the token fromharness.env, and keeping both copies refuses startup. The previous distribution reads neither new line and refuses to start its Broker without the deleted token, so a rollback to it restores both files from the step 3 backup. The new distribution's--checknow lists any surviving worker in itsworkersrow; run it here, offline, with the service still stopped and the Harness up, because online a durable Broker makes that row skip. - Start the new
bin/qwen-managed-agent-serverwith the same database and durable settings. Flyway migrates the schema forward. - Require a healthy actuator response, read an existing Session and history, then complete one new no-tool Turn before reopening ingress. Requalify any optional tool capability before enabling it.
Database downgrade is unsupported. If startup or validation fails before traffic resumes, correct the configuration and retry the same artifact. To return to a version whose manifest has an older schema range, restore the pre-upgrade database and Runtime-state backup together before accepting any new write; never run an older server against a migrated database by assumption. Any return to the previous distribution also restores server.env and harness.env from the same backup, because step 5 rewrites them for the new version.
With the command's backup, return to the previous release with both units stopped. The names in angle brackets are fields of <backup>/backup.json. Run the mysql lines as the administrator that applied config/mysql-setup.sql: on the database host as shown, or for a remote database with -h <datasource.host> -P <datasource.port> and that account's credentials. Skip the line of an archive that its files list does not name, and both mysql lines when its database is skipped; then restore the database the way it was backed up.
mysql -u root -e 'DROP DATABASE <datasource.database>'
mysql -u root < <backup>/database.sql # recreates the database
tar -C <runtimeState> --numeric-owner -xpf <backup>/runtime-state.tar
tar -C / --numeric-owner -xpf <backup>/configuration.tar
tar -C <from.release> --numeric-owner -xpf <backup>/release-config.tar
ln -sfn <from.release> <distribution>
systemctl daemon-reload
systemctl start <harness.unit> <server.unit>
Empty the Runtime state directory before extracting into it. release-config.tar goes into the previous release itself rather than through <distribution>, which still links to the new release until the ln line. When the new control plane refused to start because the database did not match its schema, it migrated nothing, and the link and the files alone go back.
Rotate the Runtime credential key
Rotate the credential key separately from a distribution upgrade. Back up the database and current keys, then configure the new active pair and retain the old pair under previous-credential-keys:
qwen:
managed-agent:
runtime-broker:
credential-key-id: reference-v2
credential-key: <base64-encoded-32-byte-v2-key>
previous-credential-keys:
'[reference-v1]': <base64-encoded-32-byte-v1-key>
Wrap every map key in [ ... ]. Spring's relaxed binder normalises plain keys — an ID such as legacy/v1, legacy:v1 or legacy v1 binds as legacyv1 whether it is quoted in the YAML or not — so the ring then never matches the persisted key ID and rows written under that ID stay unreadable even though the key is configured. The bracket form preserves the exact ID.
Restart the single Java control plane, verify that existing bindings load, and verify that new binding rows record reference-v2. New secrets use only the active key; old rows remain encrypted under their recorded key ID. The server does not re-encrypt old rows on read, and removing a still-referenced key makes those rows unreadable.
Retire reference-v1 only when no persisted row references it. Runtime bindings,
complete Session model configurations, and extension settings retain key IDs.
Compare them as bytes so case or accent differences cannot hide a still-required
key:
SELECT DISTINCT credential_key_id AS key_id
FROM ma_runtime_binding WHERE credential_key_id IS NOT NULL
UNION
SELECT DISTINCT runtime_credential_key_id AS key_id
FROM ma_runtime_binding WHERE runtime_credential_key_id IS NOT NULL
UNION
SELECT DISTINCT CAST(model_config_key_id AS BINARY) AS key_id
FROM ma_session WHERE model_config_key_id IS NOT NULL
UNION
SELECT DISTINCT CAST(encryption_key_id AS BINARY) AS key_id
FROM ma_extension_settings WHERE encryption_key_id IS NOT NULL;
The result lists the key IDs that persisted rows still require. Keep every returned key configured; reference-v1 can be retired only when it is absent from the result.
Session model configurations are written once; closing or deleting a Session retains their rows. Extension settings retain earlier revisions. Rotating the active key does not re-encrypt or remove these records, so their old key IDs stay in the result while the records remain. This procedure provides no re-encryption or physical-purge step; keep the required keys configured and do not delete rows just to clear the check.
The key ring is a map and cannot be supplied through a QWEN_MANAGED_AGENT_* environment variable: relaxed binding silently resolves an env-var form to an empty or mis-keyed ring, with no startup error. Configure previous-credential-keys in config/application.yml. The distribution ships config/application.example.yml, and both launchers export SPRING_CONFIG_ADDITIONAL_LOCATION=file:<distribution-root>/config/application.yml when that file exists, so the location no longer depends on the process working directory and no option is added to the command's positional arguments. When the variable is already set, the launchers leave it in effect instead. A deployment that relies on environment variables alone starts with an empty ring, so rotating credential-key-id without configuration of the previous key makes rows written under it unreadable. An explicit --spring.config.additional-location command-line argument outranks this environment default; when both are present, the command-line value wins and the bundled config/application.yml is not loaded.
The operator-recovery entry reads the same active pair and previous-credential-keys map as the server. Its launcher loads config/application.yml the same way; when the file lives elsewhere, set SPRING_CONFIG_ADDITIONAL_LOCATION=file:<absolute-path>/application.yml. Keep the new key active while retaining every key still referenced by a persisted row. The CSI/migration maintenance mains, including the CSI Hosted probe, still read a single credential key pair; launch those entries with the key that encrypted their target binding as the active pair.
After successful key-ring configuration, the server and recovery entry log the resolved active ID and configured IDs once per protector instance. These IDs are non-secret identifiers already stored beside ciphertext; diagnostics JSON-escape controls and non-ASCII characters, preserve spaces and case, and never include key material, ciphertext or tokens. Compare the configured IDs with the exact IDs returned by the retirement query before using the rotated deployment. An unavailable-key error names the requested, active and configured IDs, so a silently normalised or mistyped mapping can be located without exposing secrets. This diagnostic does not prove that every stored row can be decrypted.
Operator recovery
bin/qwen-managed-agent-operator-recovery starts the separately packaged recovery entry point. It is disabled unless the recovery-specific environment is set and does not start the HTTP service or worker. Follow docs/hosted-workspace-recovery.md; never infer physical stop or delete durable rows to clear a safety fence.
Scope boundary
This distribution does not certify multiple Java control planes, cross-host takeover, public CSI, Kubernetes retirement, public Shell, v2 advertisement, or stable production support. Those capabilities require separate release evidence.