qwen-code/docs/users/managed-agent-reference-deployment.md

Managed Agent Reference Deployment

English | 简体中文

The Managed Agent reference distribution is experimental. Its accepted initial topology is one Java control plane and its embedded Runtime Broker on one trusted Linux host, one MySQL database, and one compatible Qwen Hosted Harness. The compatibility manifest in each release is authoritative for that release.

Choose a tier

Each tier adds to the one before it:

Tier	What runs	What it adds
1. Text-only Hosted Sessions	The Java control plane, one Hosted Harness and MySQL on one trusted Linux host	—
2. Local-process tools	Hosted tools on the Workspace, such as file access and, when enabled, a foreground Shell	The embedded Runtime Broker, Runtime workers on the same host, the Runtime credential key, a Runtime state directory and a registered Workspace
3. Kubernetes	The API, the Harness and MySQL in a cluster	Container images built from source, installed with a Helm chart, outside this distribution's acceptance scope

Start at Tier 1 even when tools are the goal. A Session that completes a no-tool Turn proves the control plane, the Harness, the model and the database together, and Tier 2 then enables tools on the same deployment.

Tier 1: text-only Hosted Sessions

One control plane, one Harness and MySQL on one host. The steps run in order: verify the archive, generate the credentials, start the Harness, check the configuration, start the control plane, and complete a first Session.

Verify and extract

Download the archive, its detached manifest, and SHA256SUMS.txt from the same GitHub prerelease. Verify them before extraction:

sha256sum --check SHA256SUMS.txt
tar -xzf qwen-managed-agent-reference-<version>.tar.gz
cd qwen-managed-agent-reference-<version>
sha256sum --check SHA256SUMS

Inspect manifest.json. Confirm that source.revision is the reviewed revision, compatibility.qwenCodeVersion matches the CLI/Harness installation, and compatibility.databaseSchema.latestMigrationVersion is acceptable for the target database. The release is invalid if any checksum or compatibility check fails.

Prerequisites

• Linux on the trusted application host.
• JDK 21 or later.
• Node.js matching compatibility.node and Qwen Code matching compatibility.qwenCodeVersion.
• MySQL 8 with an empty database or a database previously managed by an older compatible reference distribution.
• A model endpoint for the Harness, such as an OpenAI-compatible one with its model ID and API key.
• A private, persistent Workspace directory, and for Tier 2 a Runtime state directory.
• Stable database credentials and Harness token, and for Tier 2 the Runtime credential key ring, stored outside the archive.

The Java archive does not contain the npm CLI. Install or build the exact compatible Qwen Code revision separately: the Harness runs it as qwen serve, and Tier 2's Runtime workers run its dist/cli.js.

Bootstrap credentials

bin/qwen-managed-agent-bootstrap generates the shared Harness token and capability digest, the broker token, the Runtime credential key and the database password. It writes config/harness.env, config/server.env, config/application.yml and config/mysql-setup.sql with mode 0600, refuses to overwrite existing files unless --force is given, and prints paths only:

bin/qwen-managed-agent-bootstrap

The token and digest, and the Broker URL and token for the tool path, are written only to harness.env: server.env names that file through QWEN_MANAGED_AGENT_HARNESS_CREDENTIALS_FILE, and the control plane reads them from it. It refuses to start when the file is set together with QWEN_MANAGED_AGENT_HARNESS_TOKEN or QWEN_MANAGED_AGENT_CAPABILITY_DIGEST, or when the file's Broker token is set together with QWEN_MANAGED_AGENT_RUNTIME_BROKER_TOKEN, so a rotation changes one file. The generated harness.env and server.env are shell-sourceable and also valid systemd EnvironmentFiles.

server.env names /srv/qwen-workspace as the Workspace directory in QWEN_MANAGED_AGENT_WORKSPACE_CWD; create it, or point the variable at an existing directory. The Session Store derives its workspace ID from that directory, so keep it once Sessions exist.

mysql-setup.sql creates the qwen_managed_agent database and the qwen account that server.env names, for both localhost and 127.0.0.1, with the performance_schema grant Flyway needs. Run it once as a MySQL administrator, for example mysql -u root < config/mysql-setup.sql, then delete it; server.env keeps the password. It stops at the first statement that fails, so it never changes an existing database or account. A MySQL in a container sees the gateway address rather than 127.0.0.1; create the account for that host instead.

For the systemd layout, run it as root with --directory /etc/qwen-managed-agent --service-user qwen-managed-agent. The account then owns harness.env and application.yml, server.env is 0640 root:qwen-managed-agent, a new directory is 0750 root:qwen-managed-agent, and mysql-setup.sql stays root's. The bootstrap refuses an account that does not exist, or a directory the account cannot enter. server.env also records QWEN_MANAGED_AGENT_SERVICE_USER, so the preflight checks file access as that account even when it runs as root.

Start the Hosted Harness

Start the Harness with config/harness.env, which holds its token, capability digest and Broker pair, and with the model it calls. Qwen Code reads an OpenAI-compatible provider from OPENAI_API_KEY, OPENAI_BASE_URL and OPENAI_MODEL; the authentication guide covers the others.

set -a; . config/harness.env; set +a
export OPENAI_API_KEY='<model-api-key>'
export OPENAI_BASE_URL='https://<openai-compatible-endpoint>/v1'
export OPENAI_MODEL='<model-id>'
qwen serve --profile hosted-harness --port 4170 --hostname 127.0.0.1 --no-web

On the systemd layout, put the three model variables in harness.env as plain KEY=value lines instead.

Keep the Harness on loopback. The Hosted Harness profile does not move to another port: when --port is in use, the Harness fails to start, naming the port.

Without the bootstrap's files, choose one token and capability digest and give the same values to the Harness and the Java control plane:

export QWEN_MANAGED_AGENT_HARNESS_TOKEN='<private-random-token>'
export QWEN_MANAGED_AGENT_CAPABILITY_DIGEST='sha256:<64-hex-digest>'

QWEN_SERVER_TOKEN="$QWEN_MANAGED_AGENT_HARNESS_TOKEN" \
QWEN_HOSTED_HARNESS_CAPABILITY_DIGEST="$QWEN_MANAGED_AGENT_CAPABILITY_DIGEST" \
qwen serve --profile hosted-harness --port 4170 --hostname 127.0.0.1 --no-web

Preflight check

In a second terminal, load config/server.env, or without the bootstrap's files export the variables listed under the control plane below. With the Harness running, run the bundled launcher with --check before the first start and after installing each new artifact. It binds the same configuration (config/application.yml plus the environment), starts no listener, and prints one row per input:

set -a; . config/server.env; set +a
bin/qwen-managed-agent-server --check

It covers the JDK, the service account, the configuration file and its permissions, the credential key ring, the startup property validation, the auth/bind combination, the Harness, the Workspace files and Shell cross-settings (including the mount, creation-root or SSH-node requirement), broker credentials and directories, the Workspace cwd, free ports, Runtime workers an earlier run left alive, and MySQL connectivity, the performance_schema grant and the Flyway version against manifest.json. WARN and SKIP do not fail; any FAIL exits 1.

The harness row negotiates with a single configured Harness as the first Session does; with a Harness pool it skips the pool's endpoints once the Session Store is enabled with a binding key, and fails until then. It fails when the capability digest differs, the protocol or a required feature is missing, or the Harness lacks the lifecycle a configured Workspace profile needs. It warns when the Harness runs another Qwen Code version than manifest.json names, or does not say which. The port checks bind the configured addresses the way the server does, so a port the stopped server left in TIME_WAIT counts as free, and the workers row warns about workers still running in the Runtime state directory, so stop the service first. While it runs, add --online: the port rows then check that something listens, and the workers row reports only workers a stopped server left behind, or is skipped for a durable Broker, which adopts them. The command does not verify the native Workspace storage registration.

Before the first start, the database is empty: the database row reports its schema as not migrated yet, and the capacity row warns that its tables do not exist. The first start creates them, and both rows then pass.

The database row warns when the database's Flyway version is older than manifest.json names: that is an upgraded database before the first start of the new release, which migrates it. It fails when the database is newer. At the version manifest.json names, it validates the schema as the server does at startup, and fails when the database does not match, as when an experimental release changed its schema baseline in place.

Start the Java control plane

In the same terminal, start the control plane:

bin/qwen-managed-agent-server

Without the bootstrap's files, configure the same Harness, the database and an existing Workspace directory:

export SPRING_DATASOURCE_URL='jdbc:mysql://127.0.0.1:3306/qwen_managed_agent'
export SPRING_DATASOURCE_USERNAME='qwen'
export SPRING_DATASOURCE_PASSWORD='<private-password>'
export QWEN_MANAGED_AGENT_HARNESS_ENABLED='true'
export QWEN_MANAGED_AGENT_HARNESS_BASE_URL='http://127.0.0.1:4170'
export QWEN_MANAGED_AGENT_HARNESS_TOKEN='<private-random-token>'
export QWEN_MANAGED_AGENT_CAPABILITY_DIGEST='sha256:<64-hex-digest>'
export QWEN_MANAGED_AGENT_WORKSPACE_CWD='/srv/qwen-workspace'

bin/qwen-managed-agent-server

Settings that cannot be expressed as environment variables, such as the Runtime credential key ring, belong in config/application.yml. The bootstrap writes it; otherwise start from the bundled template and keep the file private:

cp config/application.example.yml config/application.yml
chmod 600 config/application.yml

Both launchers load that file automatically when it exists. The launcher uses the server JAR in this archive. Spring Flyway applies forward database migrations before the service becomes ready. Wait for:

curl --fail http://127.0.0.1:8080/actuator/health

The enabled Harness turns on the Managed Session Store. The Store's base URL defaults to this server's plaintext loopback listener. Its workspace ID defaults to the one the Runtime Broker derives from QWEN_MANAGED_AGENT_WORKSPACE_CWD, so the identity stays the same when Tier 2 enables the tool path. A deployment that already set QWEN_MANAGED_AGENT_WORKSPACE_ID must keep it: unbound Sessions resolve their Store scope from that ID. Set QWEN_MANAGED_AGENT_SESSION_STORE_BASE_URL only when the Harness reaches this server through another address, or when the listener serves TLS.

This base configuration supports text-only Hosted Sessions. Keep the listener on loopback unless a signed or authenticated ingress is configured. Every setting, with its variable and default, is listed in the configuration reference, bundled as docs/configuration.md. The complete authentication, API, and capability configuration is bundled as docs/server-reference.md.

Run a first Session

With the default loopback listener, the API takes the tenant from the X-Qwen-Tenant-Id header. Create a Session whose first Turn asks the model a question:

key="first-session-$(date +%s)"
curl --fail -X POST http://127.0.0.1:8080/v1/agents/sessions \
  -H 'X-Qwen-Tenant-Id: default' -H 'Content-Type: application/json' \
  -H "Idempotency-Key: $key" \
  -d '{"agent_id":"qwen-code","input":[{"type":"input_text","text":"Say hello."}]}'

The Idempotency-Key header is required. Use a new key for each check, and send the same one again only to retry that request: the same key returns the same Session instead of creating another, so a later check with an old key replays the old Session and its old result. The response carries the Session's id. Read its events, starting with after=0:

curl --fail -H 'X-Qwen-Tenant-Id: default' \
  'http://127.0.0.1:8080/v1/agents/sessions/<id>/events?after=<after>&limit=100'

A page holds at most limit events, in sequence order. Set after to the sequence of the last event received, keep it when a page comes back empty, and poll again until an event has "terminal": true.

A Turn that succeeds ends with turn.completed, and the model's reply is the text of the item.output_text.delta events before it. A terminal turn.failed, or none before the Turn deadline, points at the Harness or the model: the Harness log names the cause. This Session is the deployment's first proof; repeat it after every upgrade, as verify --turn does on the systemd layout.

systemd

The distribution includes systemd/ unit and environment templates for the accepted single-host topology. The control-plane unit uses KillMode=process so Runtime workers survive a Broker restart; the Harness unit uses KillMode=mixed so its tool processes terminate with it. Follow systemd/README.md to install them.

Tier 2: local-process tools

Tier 2 adds Hosted tools to a working Tier 1 deployment. The embedded Runtime Broker places Runtime workers, Node.js processes on the same host that run the tools on the Workspace. Tool execution requires the existing fail-closed Runtime and Workspace setup. On top of the Tier 1 configuration, at minimum, configure:

export QWEN_MANAGED_AGENT_RUNTIME_BROKER_ENABLED='true'
# Omit when harness.env carries QWEN_MANAGED_RUNTIME_BROKER_TOKEN, as the bootstrap's does.
export QWEN_MANAGED_AGENT_RUNTIME_BROKER_TOKEN='<private-random-token>'
export QWEN_MANAGED_AGENT_RUNTIME_CREDENTIAL_KEY_ID='reference-v1'
export QWEN_MANAGED_AGENT_RUNTIME_CREDENTIAL_KEY='<base64-encoded-32-byte-key>'
export QWEN_MANAGED_AGENT_RUNTIME_STATE_DIRECTORY='/var/lib/qwen-managed-agent/runtime'
export QWEN_MANAGED_AGENT_NODE_EXECUTABLE='/usr/bin/node'
export QWEN_MANAGED_AGENT_RUNTIME_WORKER_ENTRY='/opt/qwen-code/dist/cli.js'
export QWEN_MANAGED_AGENT_RUNTIME_VERIFIED_WORKSPACE_RECOVERY_ENABLED='true'

The Hosted Harness reaches the same Broker through its own environment, not its command line and not the control plane's variables above: set QWEN_MANAGED_RUNTIME_BROKER_URL='http://127.0.0.1:4182' and QWEN_MANAGED_RUNTIME_BROKER_TOKEN to the Broker token in harness.env, or in whatever environment starts the Harness. qwen serve removes both from its environment once read, so no process the Harness starts inherits the token, and the control plane keeps the QWEN_MANAGED_RUNTIME_BROKER_ variables out of Runtime workers. With the bootstrap's files, harness.env already holds both, and the control plane takes its Broker token from that file, so leave QWEN_MANAGED_AGENT_RUNTIME_BROKER_TOKEN unset; setting both copies refuses startup. Set the two Harness lines together or not at all, each as a plain KEY=value line: startup refuses a file with only one of them or with either one empty, which the Harness also refuses, and a line such as export KEY=value, which systemd ignores. The preflight broker-url row reads the file's URL the way the Harness's URL parser does, so http://127.1:4182 counts as http://127.0.0.1:4182. It passes when that URL names the Broker's own listener over http. It fails when the Harness would refuse the URL, which must be an http or https origin and may use plain http only on the loopback interface, and when it names the listener over https, which the plain-HTTP Broker cannot accept. It warns when the URL names another address or port, which only a proxy could forward; a Broker bound to a specific address outside loopback, which needs QWEN_MANAGED_AGENT_RUNTIME_BROKER_ALLOW_NON_LOOPBACK=true, is always reached that way. A Broker bound to 0.0.0.0 needs that setting too, but still serves loopback, where the Harness dials it directly at http://127.0.0.1:4182 and the row passes. An IPv6 URL such as http://[::1]:4182 warns there, because that bind takes IPv6 only when the control plane's JVM listens on both families, which it does not under -Djava.net.preferIPv4Stack=true. It also warns when the file has no Broker URL, which is what a Harness given the pair through its own environment or command line looks like to preflight; there the warning is expected, and moving the pair into the file means deleting QWEN_MANAGED_AGENT_RUNTIME_BROKER_TOKEN from server.env.

Before admitting file, Shell, or child Git work, complete the native storage registration and mount-identity verification described under “Verified physical Workspace recovery” in docs/server-reference.md. An unregistered or ambiguous mount must remain unavailable. Public Shell execution remains disabled by default. Shell lifecycle v2 is advertised by default; an explicit lifecycle list replaces the default, and an empty list disables it. MCP lifecycle still requires explicit inclusion. Capability advertisement does not establish deployment qualification.

Tier 3: Kubernetes

The Helm chart installs the signed Managed Agent API and a private Hosted Harness in one Pod, with MySQL 8.4 on a persistent volume, from images you build at a reviewed source revision. Its default runs text Sessions; optional settings add trusted single-node Workspace files and a foreground Shell, and a Managed WebShell gateway. It is installed from source rather than from this archive, follows its own guide, and does not extend this distribution's acceptance scope. K2 CSI resident Workspaces are a separate deployment, described in deploy/k2-resident.

Operate the deployment

Maintenance commands

bin/qwen-managed-agent-admin <command> [arguments] runs the offline maintenance entry points, such as storage-registration, workspace-migration and the CSI commands, from the server jar through Spring Boot's PropertiesLauncher. Run it without arguments to list the commands, and run a command without arguments for its usage. Like the server launcher, it uses config/application.yml when that file exists and SPRING_CONFIG_ADDITIONAL_LOCATION is not set. Most commands that open the database take their own credentials rather than server.env's SPRING_DATASOURCE_*: the Workspace commands read W1_JDBC_URL, W1_JDBC_USER and W1_JDBC_PASSWORD, and the CSI commands read K2_JDBC_URL, K2_JDBC_USER and K2_JDBC_PASSWORD. Two are exceptions: operator-recovery and resident-retirement start the server's Spring configuration and use its datasource, so they read SPRING_DATASOURCE_URL, SPRING_DATASOURCE_USERNAME and SPRING_DATASOURCE_PASSWORD, together with the other control-plane settings, from the environment and config/application.yml; run them with server.env loaded. csi-protection opens no database, and neither does resolved-configuration, which prints the Runtime state directory, auth mode and trusted actor header that the server's configuration resolves to in the environment it runs in, and the URLs of the control plane, its health endpoint and the Hosted Harness; the upgrade command runs it the same way to find the Runtime state and those URLs. Each command is an offline procedure with its own preconditions, documented with it; meet them before running it. bin/qwen-managed-agent-operator-recovery remains the entry for the Hosted Shell recovery procedure, and bin/qwen-managed-agent-admin operator-recovery runs the same command.

Upgrade

On the systemd layout, the new distribution's bin/qwen-managed-agent-upgrade runs the upgrade in phases. Run each as root from the extracted and verified archive:

cd qwen-managed-agent-reference-<new-version>
# Pass check the options backup gets, such as --skip-database.
bin/qwen-managed-agent-upgrade check
# Stop new ingress and let admitted work settle.
bin/qwen-managed-agent-upgrade stop
bin/qwen-managed-agent-upgrade backup --to /var/backups/qwen-managed-agent/<date>
bin/qwen-managed-agent-upgrade install --backup /var/backups/qwen-managed-agent/<date>
# The manual steps check printed, such as installing Qwen Code for the Harness.
bin/qwen-managed-agent-upgrade start
bin/qwen-managed-agent-upgrade verify --tenant <tenant> --turn
# Reopen ingress.

Each phase reads the units, their environment files and the installed release again, and refuses unless the previous phase left them as it expects, so an interrupted upgrade resumes at the phase that failed. --server-unit and --harness-unit name units other than qwen-managed-agent-server and qwen-hosted-harness; the command finds the installed release, the environment files and the service account through them.

• check changes nothing. It verifies the new distribution against its SHA256SUMS, compares both releases (version, the Flyway scripts in the server jars, Java, Node.js and Qwen Code), and reads the Harness Broker settings, the unit templates, operator files in the installed config/ and the release layout. Its backup-tools and runtime-state rows tell whether backup will run with the options given, so pass check the backup options too; the plan it prints repeats them. It prints one row per input and a plan with the manual steps where they fall, and exits 1 on any FAIL. The database row fails when the new release changed or removed a script the installed database ran: until the first formal release, a release may rewrite its schema baseline in place, and such a release cannot start on the existing database. It also fails when the release adds a versioned script that does not sort after the newest one the installed database ran, which Flyway refuses to apply.
• stop repeats the checks above backup-tools and stops nothing unless they pass. The backup-tools and runtime-state rows gate only backup, so settle them with check before stop. It stops the control plane, stops the Runtime workers that KillMode=process leaves in its cgroup and waits for them (--timeout, 60 seconds by default), then stops the Harness. It counts the workers in the cgroup systemd names, in the unified hierarchy or on cgroup v1 in systemd's own, and refuses when it cannot read that cgroup. Once the new release is installed, stop skips the checks, so it also stops a control plane that start left running. The JVM exits 143 on SIGTERM, so a control-plane unit installed without the template's SuccessExitStatus=143 shows as failed after the stop; stop says so, and once the template's change is carried over, systemctl reset-failed clears the state.
• backup --to <new empty directory> needs both units stopped and no worker left, and refuses once the new release is installed, because its from.release would then name the new release. A directory that already exists must be empty and belong to root. It writes database.sql with mysqldump, passing the unit's SPRING_DATASOURCE_* credentials through an option file in a temporary directory outside the backup rather than on the command line; mysqldump reads no other option file, so root's ~/.my.cnf cannot replace them. It writes runtime-state.tar of the Runtime state directory, configuration.tar of the unit files, their environment files, the Harness credentials file and the SPRING_CONFIG_ADDITIONAL_LOCATION files, and release-config.tar of the installed config/ and any of those files that live in the release, recorded by their path in the release. A file that is a symbolic link goes in as the link and as the file it names. Every file is 0600 in a 0700 directory, and backup.json, written last, records their checksums, the database the dump is of, and where install leaves the previous release. --mysqldump <path> names another client, and --skip-database skips the dump when the database is backed up another way. A backup that fails, or that SIGINT, SIGTERM or SIGHUP ends, removes what it wrote and the option file. A SIGKILL removes nothing; remove the ${TMPDIR:-/tmp}/qwen-managed-agent-upgrade-* directory it leaves, which holds the credentials.
  • check and backup take the Runtime state directory from the server's own configuration. They run the new release's server jar as the control-plane unit runs the server: with its account, working directory and environment, and the launcher's default config/application.yml when SPRING_CONFIG_ADDITIONAL_LOCATION is unset or empty. The jar prepares Spring's environment the way the server does, so every configuration file, import and profile counts, as do SPRING_APPLICATION_JSON, system properties from JAVA_TOOL_OPTIONS and relaxed names, and a relative directory is resolved in the unit's working directory. It starts nothing and opens no connection. When it cannot resolve the configuration, or the unit passes the launcher arguments, which systemctl does not print exactly, backup refuses until --runtime-state <directory> names the directory the server uses, or --runtime-state none says it keeps none. Without a configured directory, as on a deployment without the tool path, backup archives no Runtime state and says so.
• install --backup <directory> refuses a backup of another release, a backup file that changed, a backup the control plane has started since, and a backup of another database than the unit's SPRING_DATASOURCE_URL names. It copies the release to <distribution>-releases/<version>, such as /opt/qwen-managed-agent-releases/<version> (--releases names another directory; pass the same one to every phase), copies operator files from the installed config/ with their owner and mode, and points the distribution path at the new release with one rename. The copy's own files, those SHA256SUMS lists, are made readable by every account, so the service account can run a release extracted with a restrictive umask. The first run moves the installed directory to <releases>/<old version> and puts a link in its place, so the units keep their paths. The previous release stays where it is. An existing <releases>/<version>, such as the archive extracted there, is used only when its SHA256SUMS is the new distribution's and its files match it, and its listed files are made readable the same way; a copy that fails is removed. A crash between the first run's two renames leaves <distribution>.upgrade-<suffix> beside the missing distribution path; every phase then names the mv -T that finishes the install. An install killed at another step can leave a partial copy .<version>.partial-<suffix> in the releases directory, an <distribution>.upgrade-<suffix> link beside the distribution path, or an operator file's .tmp-<suffix> copy in the new release's config/; the layout row lists them, and the next install removes them. The suffix is the eight hexadecimal digits install generates, so an operator's own <distribution>.upgrade-backup is left alone. Run again once the release is installed, as after an interrupted session, install says so and points at start.
• start refuses before install, and while the Harness Broker settings still need step 5 below. It starts the Harness, or restarts one that an earlier start left running so that it reads what was fixed since, runs bin/qwen-managed-agent-server --check as the service account with the unit's environment, starts the control plane only when that passes, and waits for a healthy actuator (--timeout, 300 seconds by default; one probe waits at most 5 seconds). Flyway migrates the schema during this start. A control plane that exits before it is healthy is stopped and reported, even one that systemd's Restart=on-failure would bring up when it runs it again: an exit during the start that migrates the schema is worth reading in the journal before start runs again. One that is still not healthy at the timeout keeps running; read its journal, then run verify, or stop and start again.
  • start and verify find the Harness, the control plane and its health endpoint the same way, so a port or address set where server.env does not show it counts: in a configuration file, on the command line or in SPRING_APPLICATION_JSON. The Harness is where qwen.managed-agent.harness.base-url says. The control plane is where server.port, server.address and server.ssl bind it, with a wildcard address reached on the loopback interface, and the health endpoint follows Spring Boot's management settings: a management.server.port of its own moves it to that port, on management.server.address and under management.server.base-path. --server-url <url> names the control plane instead, for example behind a proxy, and the health endpoint is then <url>/actuator/health. A phase checks only the URLs it dials: both probe the health endpoint, and verify --tenant also reads the API when the auth mode is open and, with --actor, the server names a trusted actor header. A port of 0 or below gives no URL to wait on, so the phase refuses until --server-url names one; it also refuses a URL fetch refuses to dial: a link-local IPv6 address with its zone, a port the Fetch standard blocks (6000, for one), or a --server-url without http:// or https://, with a user name or password, which the refusal prints as *** up to the last @, or with a query or a fragment, which the paths appended to it would land in; check judges --server-url the same way, and an @ in its path stays valid path text, with printed copies masked up to the last @ and their control characters replaced; start likewise refuses a Harness URL that is not http or https, and, while the Harness is enabled, one the server's Harness connector would refuse: anything but an http or https URL with a host and without credentials, a query or a fragment, as the server reads it with java.net.URI, so a host name with an underscore is refused too; when the configuration does not resolve, start takes only a plain http or https URL, a host and port with a path, and leaves the rest to the preflight. When the health endpoint does not report UP, start and verify say what it answered last; for a certificate this host does not trust, point NODE_EXTRA_CA_CERTS at its CA when you run the command. When the configuration does not resolve, or the unit passes the launcher arguments, they use the URLs server.env names and print a line that says so.
• verify runs --check --online the same way and reads the actuator, waiting up to --timeout (300 seconds by default) for its answer. With --tenant <id>, it reads the tenant's most recent Session and its Items, before --turn creates one; --turn creates a Session and waits for one Turn, which calls the model, polling past a 5xx answer or a dropped connection; when the Session read is skipped, the turn row says so. No request outlasts --timeout. A request without an actor lists only Sessions outside Workspaces; --actor <id> sends one through the header QWEN_MANAGED_AGENT_TRUSTED_ACTOR_HEADER names. verify reads the auth mode the way backup reads the Runtime state directory, and signed authentication skips both reads; read a Session through the ingress instead.

The command starts and stops units only. It never stops ingress, installs Qwen Code, edits environment or unit files, or deletes a release. When check warns that a unit template changed, carry the change over to the installed unit and run systemctl daemon-reload before start.

Manual procedure

Other layouts follow these steps. On the systemd layout the command runs stop, backup, install, start and verify, and check prints the steps it leaves to the operator where they fall: stopping and reopening ingress, installing Qwen Code and editing harness.env in step 5, and carrying changed unit templates over.

1. Download and verify the new archive and read both manifests. Confirm the new Qwen Code/Harness, Java, protocol, and database requirements before touching the running deployment.
2. Stop new ingress. Wait for admitted work to settle according to the deployment's existing drain procedure.
3. Back up MySQL, the Runtime state directory, the complete Runtime credential key ring, the operator-owned server.env and harness.env, and the exact old distribution. Do not rotate the credential key during the distribution upgrade. With server.env loaded, bin/qwen-managed-agent-admin resolved-configuration prints the Runtime state directory the configuration resolves to.
4. Stop the Java service and Hosted Harness. Preserve the Workspace and durable Runtime state. The shipped control-plane unit uses KillMode=process, so systemctl stop leaves Runtime workers running; reap them with systemctl kill --kill-whom=all -s TERM qwen-managed-agent-server before starting the new version.
5. Install the manifest-compatible Qwen Code/Harness and start it with the existing token and capability digest. On a tool-path host, first replace any QWEN_HOSTED_HARNESS_BROKER_ARGS line in harness.env with QWEN_MANAGED_RUNTIME_BROKER_URL and QWEN_MANAGED_RUNTIME_BROKER_TOKEN: nothing reads that line any more, so without them the Harness starts with no Broker. When server.env enables the Harness and names harness.env as its credentials file, as the bootstrap's files do, also delete QWEN_MANAGED_AGENT_RUNTIME_BROKER_TOKEN from server.env; the control plane then takes the token from harness.env, and keeping both copies refuses startup. The previous distribution reads neither new line and refuses to start its Broker without the deleted token, so a rollback to it restores both files from the step 3 backup. The new distribution's --check now lists any surviving worker in its workers row; run it here, offline, with the service still stopped and the Harness up, because online a durable Broker makes that row skip.
6. Start the new bin/qwen-managed-agent-server with the same database and durable settings. Flyway migrates the schema forward.
7. Require a healthy actuator response, read an existing Session and history, then complete one new no-tool Turn before reopening ingress. Requalify any optional tool capability before enabling it.

Database downgrade is unsupported. If startup or validation fails before traffic resumes, correct the configuration and retry the same artifact. To return to a version whose manifest has an older schema range, restore the pre-upgrade database and Runtime-state backup together before accepting any new write; never run an older server against a migrated database by assumption. Any return to the previous distribution also restores server.env and harness.env from the same backup, because step 5 rewrites them for the new version.

With the command's backup, return to the previous release with both units stopped. The names in angle brackets are fields of <backup>/backup.json. Run the mysql lines as the administrator that applied config/mysql-setup.sql: on the database host as shown, or for a remote database with -h <datasource.host> -P <datasource.port> and that account's credentials. Skip the line of an archive that its files list does not name, and both mysql lines when its database is skipped; then restore the database the way it was backed up.

mysql -u root -e 'DROP DATABASE <datasource.database>'
mysql -u root < <backup>/database.sql   # recreates the database
tar -C <runtimeState> --numeric-owner -xpf <backup>/runtime-state.tar
tar -C / --numeric-owner -xpf <backup>/configuration.tar
tar -C <from.release> --numeric-owner -xpf <backup>/release-config.tar
ln -sfn <from.release> <distribution>
systemctl daemon-reload
systemctl start <harness.unit> <server.unit>

Empty the Runtime state directory before extracting into it. release-config.tar goes into the previous release itself rather than through <distribution>, which still links to the new release until the ln line. When the new control plane refused to start because the database did not match its schema, it migrated nothing, and the link and the files alone go back.

Rotate the Runtime credential key

Rotate the credential key separately from a distribution upgrade. Back up the database and current keys, then configure the new active pair and retain the old pair under previous-credential-keys:

qwen:
  managed-agent:
    runtime-broker:
      credential-key-id: reference-v2
      credential-key: <base64-encoded-32-byte-v2-key>
      previous-credential-keys:
        '[reference-v1]': <base64-encoded-32-byte-v1-key>

Wrap every map key in [ ... ]. Spring's relaxed binder normalises plain keys — an ID such as legacy/v1, legacy:v1 or legacy v1 binds as legacyv1 whether it is quoted in the YAML or not — so the ring then never matches the persisted key ID and rows written under that ID stay unreadable even though the key is configured. The bracket form preserves the exact ID.

Restart the single Java control plane, verify that existing bindings load, and verify that new binding rows record reference-v2. New secrets use only the active key; old rows remain encrypted under their recorded key ID. The server does not re-encrypt old rows on read, and removing a still-referenced key makes those rows unreadable.

Retire reference-v1 only when no persisted row references it. Runtime bindings, complete Session model configurations, and extension settings retain key IDs. Compare them as bytes so case or accent differences cannot hide a still-required key:

SELECT DISTINCT credential_key_id AS key_id
FROM ma_runtime_binding WHERE credential_key_id IS NOT NULL
UNION
SELECT DISTINCT runtime_credential_key_id AS key_id
FROM ma_runtime_binding WHERE runtime_credential_key_id IS NOT NULL
UNION
SELECT DISTINCT CAST(model_config_key_id AS BINARY) AS key_id
FROM ma_session WHERE model_config_key_id IS NOT NULL
UNION
SELECT DISTINCT CAST(encryption_key_id AS BINARY) AS key_id
FROM ma_extension_settings WHERE encryption_key_id IS NOT NULL;

The result lists the key IDs that persisted rows still require. Keep every returned key configured; reference-v1 can be retired only when it is absent from the result.

Session model configurations are written once; closing or deleting a Session retains their rows. Extension settings retain earlier revisions. Rotating the active key does not re-encrypt or remove these records, so their old key IDs stay in the result while the records remain. This procedure provides no re-encryption or physical-purge step; keep the required keys configured and do not delete rows just to clear the check.

The key ring is a map and cannot be supplied through a QWEN_MANAGED_AGENT_* environment variable: relaxed binding silently resolves an env-var form to an empty or mis-keyed ring, with no startup error. Configure previous-credential-keys in config/application.yml. The distribution ships config/application.example.yml, and both launchers export SPRING_CONFIG_ADDITIONAL_LOCATION=file:<distribution-root>/config/application.yml when that file exists, so the location no longer depends on the process working directory and no option is added to the command's positional arguments. When the variable is already set, the launchers leave it in effect instead. A deployment that relies on environment variables alone starts with an empty ring, so rotating credential-key-id without configuration of the previous key makes rows written under it unreadable. An explicit --spring.config.additional-location command-line argument outranks this environment default; when both are present, the command-line value wins and the bundled config/application.yml is not loaded.

The operator-recovery entry reads the same active pair and previous-credential-keys map as the server. Its launcher loads config/application.yml the same way; when the file lives elsewhere, set SPRING_CONFIG_ADDITIONAL_LOCATION=file:<absolute-path>/application.yml. Keep the new key active while retaining every key still referenced by a persisted row. The CSI/migration maintenance mains, including the CSI Hosted probe, still read a single credential key pair; launch those entries with the key that encrypted their target binding as the active pair.

After successful key-ring configuration, the server and recovery entry log the resolved active ID and configured IDs once per protector instance. These IDs are non-secret identifiers already stored beside ciphertext; diagnostics JSON-escape controls and non-ASCII characters, preserve spaces and case, and never include key material, ciphertext or tokens. Compare the configured IDs with the exact IDs returned by the retirement query before using the rotated deployment. An unavailable-key error names the requested, active and configured IDs, so a silently normalised or mistyped mapping can be located without exposing secrets. This diagnostic does not prove that every stored row can be decrypted.

Operator recovery

bin/qwen-managed-agent-operator-recovery starts the separately packaged recovery entry point. It is disabled unless the recovery-specific environment is set and does not start the HTTP service or worker. Follow docs/hosted-workspace-recovery.md; never infer physical stop or delete durable rows to clear a safety fence.

Scope boundary

This distribution does not certify multiple Java control planes, cross-host takeover, public CSI, Kubernetes retirement, public Shell, v2 advertisement, or stable production support. Those capabilities require separate release evidence.