horizon/docs/release-flow.md

Release flow

How a change gets from a pull request to players: the checks a pull request passes, how it merges, and the automatic releases that follow. Releases are automatic and frequent so the latest Horizon is always playable. Since 2026-10-28 Horizon ships only the Nightly (Ludovic's decision; the Team proposed it): the batch train builds a Nightly instead of a vX.Y.Z release, it is published once its smoke test passed and no release-blocker is open, and every launcher and horizon-server update install it (Nightly only).

Pull request checks

The checks workflow (.github/workflows/ci.yml) runs two jobs on every pull request (opened, pushed to or reopened; labels never start it):

Job	Runs on	What	Time
verify	Our checks runners (label horizon-checks, the desktop's) for the pull requests whose number ends in 0 or 1, forks included; GitHub ubuntu-latest for the others and main's pushes. SELF_HOSTED_CI moves them: all all to ours, light or off all to GitHub (CI runners)	Public-tree check, Python tests, rustfmt, clippy with warnings denied, Rust tests, one after another in one job. Dependencies and the workspace build come from a cache saved by main's runs (GitHub) or the runner's own build folder (ours)	~8 min
parity / parity	A runner with the label horizon-game: the Mac mini's and playhorizon's (below)	Only for a change to a harness, an oracle, a probe or the runner (or the run-parity label): the vanilla harnesses the change can affect (tools/parity.py --changed <base>). Other pull requests skip it: parity-main checks them on main	seconds when nothing is affected; minutes per affected harness

Pushes to main run verify only; the nightly audit covers main's harnesses.

verify is the check main requires, so it has no condition and runs in every run of checks: GitHub counts a skipped required check as passed, and labels used to start runs whose verify was skipped, which let auto-merge merge before the real verify finished. A pull request that changes no Rust (docs, tools) runs only its quick steps and finishes in about a minute. A pull request's public-tree check reads only its own commits (--range base..head); main's pushes read every ref's history.

The parity job

The job is .github/workflows/parity.yml, a reusable workflow: checks calls it for pull requests from this repository, run-parity.yml for a fork's pull request a maintainer labels run-parity.

A pull request every harness reaches (a crate split moves files every probe links: #2160 ran all 352 harnesses, 71 minutes on a Mac mini runner, while five other pull requests' jobs queued behind it) carries the full-parity label: its job runs on playhorizon's runner only (4 harnesses at a time), and waits for it while that runner is busy instead of taking the Mac mini's. Labels don't start checks: add it when opening the pull request (gh pr create --label full-parity), or before the next push.

A pull request that must leave the picture alone (HDR, shader packs) carries the frame-identity label: its job also runs frame_identity (parity-workflow.md), which no automatic selection picks, since it fails any change that moves the picture on purpose. Like full-parity, it is read when the job starts.

Horizon never downloads the game, so harnesses that compare with vanilla run only where the maintainer's own copy is: the Mac mini and playhorizon (below). Their runners run as the maintainer's account and keep their checkout between runs (clean: false), so the probes build incrementally into .local/parity-target and earlier passes stand.

The job tests the pull request's head merged into main's current tip, which is what merging it would land: harness and game fixes that reached main after the branch was made apply to it too, and an old branch runs main's runner scripts and selection unless it changes them. The base for the selection is that main commit, so only the pull request's own changes count. The job summary names the main commit. A head that conflicts with main is tested alone, with a warning. The check is still reported on the head.

1. tools/ci/runner-env.sh puts the machine's Python 3.13 and the JDK research-lock.json pins on the PATH (Homebrew and java_home on the Mac; on Linux what linux-runner.sh pinned, else nixpkgs), and sccache when the machine has it.
2. tools/ci/vanilla-data.sh links .local/ to the runner's copy of the vanilla data, cloned once from ~/code/horizon-parity/shared-local into ~/.actions-runners/horizon/_work/horizon/shared-local, and prepares .local/horizon-client with the pull request's own horizon-prepare once per preparer version (its sources, the exporters in mods/horizon/src/test/java and research-lock.json). The game's sounds, which depend only on the Minecraft version, are kept once in shared-local/sounds/<version>/ (copied from a folder horizon-prepare sounds made from the maintainer's own game; the Mac mini has no official launcher of its own) and cloned into each preparation (not linked: vanilla's packs refuse symbolic links), for the audio harnesses. tools/prepare_output.py check then compares what was prepared with crates/horizon-prepare/output-versions.txt: launchers keep a player's prepared resources across releases whose preparer declares the same OUTPUT_VERSION (crates/horizon-prepare/src/lib.rs), so a change to the prepared files fails here until it bumps the number and appends the new digest (the error gives both lines).
3. tools/parity.py --changed <base sha> picks the harnesses whose traced dependencies (their script, Python modules, Java oracles and every Rust source their probes compile) changed since the merge base, plus any harness nothing is recorded about yet, and runs those that have no pass for their current inputs. A change to Cargo.lock or research-lock.json affects every harness. Each harness has 25 minutes, the job 45.

Failed harnesses' logs are uploaded as the parity-logs artifact.

Platform-specific harnesses. A harness listed under linux_only in tools/parity_hosts.json gives different results on macOS for a reason that is not a Horizon bug (the reason is printed). parity.py skips it on other platforms unless it is named on the command line. Today it holds only save_roundtrip (it needs vscreen): options, chunk_generation, world_border_render and mob_ai, which once differed on macOS through the platform's libm, pass on the Mac mini since horizon_core::jmath computes what HotSpot computes (checked 2026-10-14).

Pull requests from forks never run code on the Mac mini by themselves. A maintainer who has read the change adds the run-parity label: that runs the job once, on the commit the pull request had then. To run it again after new pushes, remove the label and add it again.

Two runners. The Mac mini has two runners for this repository, ludovics-mac-mini (~/.actions-runners/horizon, CARGO_BUILD_JOBS=10) and ludovics-mac-mini-2 (~/.actions-runners/horizon-2, CARGO_BUILD_JOBS=6), with the same labels, so the 30-minute nightly audit or a long parity job never holds a release's macOS build back. Each keeps its own checkout, target/ and Rust install; the second's _work/horizon/shared-local links to the first's, so they share the vanilla data, the preparations (one at a time, under a lock) and the parity passes and slots.

The desktop's runner. Since 2026-11-03 (the maintainer's decision) ludodesktop-horizon takes pull requests' parity jobs too, beside the Mac mini and playhorizon: labels self-hosted, X64, horizon-game, ludodesktop, and no Linux, so ci.yml's by-hand [self-hosted, linux, horizon-game] jobs stay on playhorizon. It is declared in the maintainer's NixOS configuration (services.github-runners, his private fleet repo), not installed by tools/ci/linux-runner.sh. A fork's job reaches it only through run-parity, so it runs as a system user, gha, with no sudo, SSH keys or logins; its service sees no /home and no device (no GPU: Vulkan is Mesa's lavapipe, on an Xvfb of its own), the system is read-only to it, and it runs in a slice at CPU and I/O weight 20, at most 16 of the 24 threads, 20 GB of memory, nice 10 (CARGO_BUILD_JOBS=16, 8 harness slots). It runs only while the maintainer isn't at the machine: a timer stops it within a minute when he is at the screen (an unlocked session) and starts it 2 minutes after he leaves. A job it stops fails: rerun it, and it goes to whichever runner is free. Its vanilla data is his own copy, kept in the runner's folder; a clean-up before each start keeps its build folders under 40 GB and its 2 newest preparations. The nightly audit and parity-main stay on the Mac mini's runners: the audit compares with the previous night, so it stays on one platform, and a parity-main job stopped because he came back would bisect and comment on merges for nothing. The desktop is no parity host for tools/parity.py --hosts any more (its tools/parity_hosts.json entry points at the 10-26 setup, whose pause file stays set, so it refuses work).

playhorizon's runner. playhorizon-horizon runs on a server a hosting partner lends the project (Ubuntu, 8 AMD EPYC threads, 30 GB, in Detroit; reachable only through the maintainer's tailnet). Nobody sits at it, so the watcher never pauses it except by hand. It gets half the machine: RUNNER_MEMORY=24, CARGO_BUILD_JOBS and the harness slots at 4, and a drop-in (~/.config/systemd/user/horizon-runner.service.d/cpus.conf) that starts it under taskset -c 4-7 (the user slice can't set AllowedCPUs: systemd doesn't delegate the cpuset controller to it). Agents' tools/remote.sh runs and benchmarks share CPUs 0-3, so they never slow a pull request's job. Without Nix, linux-runner.sh takes the build libraries, the JDK, sccache, Xvfb and Mesa's software Vulkan (lavapipe; there is no GPU) from apt, and Python 3.13 from uv. The partner can reach the machine, so gh is not logged in there: the registration token comes from another machine (RUNNER_TOKEN), and no job with secrets may run there (parity has none). It is also a parity host (playhorizon in tools/parity_hosts.json, 3 jobs on CPUs 0-3). Since 2026-10-26 it has the horizon-game label: pull requests' parity jobs run there, on the Mac mini or on the desktop, and full-parity ones only there.

Merging

Pull requests merge with a merge commit, by hand: never gh pr merge --auto. Branch protection on main requires verify for everyone, and Ludovic's review for the files .github/CODEOWNERS lists (what builds, signs and ships releases). Branches don't have to be up to date with main: main's own checks run catches the rare semantic conflict.

Who merges. The Horizon Team (GitHub team horizon-team: kacpherek, levzzz5154, rdbtCVS, Online5277) and Ludovic. The orchestrator's agents prove only the risky pull requests (below), so a contributor's ordinary pull request doesn't wait for them.

An ordinary pull request merges on CI. A Team member:

1. reads the whole diff (for a fork, before adding run-parity: that runs the fork's code on a maintainer's machine);
2. adds run-parity to a fork's pull request; our own branches run parity by themselves. Since #2448 it tests the pull request merged into main;
3. merges once verify and parity pass, nobody's change request is open, and every comment has been read (a "don't merge" from anyone holds it).

Never merge your own pull request: another Team member reviews and merges it. Never force-push or rebase a contributor's branch; push a fixup commit, or merge main into it.

A risky pull request gets the needs-proof label and waits for an agent's local proof (both single-player paths, lan_e2e, a screenshot for anything visible) before it merges. Risky means any of:

• it changes what is saved or loaded (save.rs, entity_nbt.rs, block_entity_nbt.rs, region.rs, datafix/): a bug there damages players' worlds;
• the network or the server thread: horizon-protocol, net.rs, remote*.rs, lan/, lan*.rs, integrated.rs, levels.rs, dedicated*;
• world generation (worldgen/): seeds must keep generating the same terrain;
• the launcher's installs and updates, or the mod runtime;
• the harnesses themselves (tools/test_*_parity.py, the Java oracles): a wrong harness hides regressions;
• over ~500 lines, or parity failed, was skipped or ran no harness for gameplay code.

Unsure? Add needs-proof: it costs a few hours, a broken release costs more.

parity becomes required once it has run green on pull requests for a while (a week without a false failure):

gh api -X POST repos/horizon-rust/horizon/branches/main/protection/required_status_checks/contexts -f 'contexts[]=parity / parity'

A skipped parity (a fork's pull request without the label) counts as passed, so requiring it never blocks a fork's pull request: the Team member decides whether to run it, and a skipped run makes the pull request risky.

Parity on main

Most pull requests merge on verify alone and leave the vanilla harnesses to .github/workflows/parity-main.yml, which runs them on main a batch of merges at a time, on the Mac mini's runner:

• It starts on every push to main, one run at a time (concurrency: parity-main). A run that started always finishes; merges that land while it goes on wait in one pending run: GitHub keeps only the newest and cancels the others, so the next run takes them all. The many cancelled runs in the list are those pending ones, plus runs whose bisect outgrew the job's 240 minutes.
• tools/parity_main.py base gives the newest commit main contains whose parity-main passed (else the last merge's parent), and the run is tools/parity.py --changed <that commit>: the union of the harnesses every merge since then can affect, once, with every earlier pass reused.
• On a failure, tools/parity_main.py bisect checks out the batch's first-parent merges by halves and runs only the failing harnesses, names the first merge they fail on in the run's summary and comments on its pull request (🤖 Agent: parity-main found that this merge breaks ...). Fix it with a pull request as usual; the next run checks from the last green commit again.
• Nightlies build only on a commit both checks and parity-main passed on (tools/release_train.py, tools/ci/nightly.py, see below), so a broken merge holds them instead of reaching players. They take the newest such commit even when a later batch failed: that failure is in merges the green commit doesn't contain, so a broken or flaky harness holds only what landed after the last green run, never the whole Nightly (before 2026-11-03, any later failure held every Nightly until a run passed again: 30 h without one). release.yml also starts when a parity-main run finishes. A forced run (force=true) doesn't wait for parity-main.

A pull request still runs its own parity job when it changes a harness script, an oracle (mods/horizon/src/test/), a probe (crates/*/examples/, game/*_fixture.rs, game/*_harness.rs), the runner (tools/parity*.py, tools/lab.py, tools/parity_hosts.json, parity*.yml), or carries the run-parity label (run-parity.yml runs it when the label is added).

Game releases

.github/workflows/release.yml is the batch train: it decides when a new Nightly goes out and starts nightly build (.github/workflows/nightly-build.yml, Nightly channel) on the commit it decided, which builds, signs and publishes it. It starts after every green checks or parity-main run of a push to main, every 10 minutes (so a batch waiting for merges to settle goes out without another merge), and by hand.

flowchart LR
    pr(["pull request"]) --> checks{"verify<br/>parity"}
    checks -- green --> merge["merge<br/>(merge commit)"]
    merge --> main(["main"])
    main --> mainchecks{"checks and<br/>parity-main"}
    mainchecks -- green --> decide{"decide<br/>tools/release_train.py"}
    cron(["every 10 minutes"]) --> decide
    decide -- "5 player-facing PRs settled 5 min,<br/>the oldest waited 6 h, or release-now" --> nightly["nightly build:<br/>that commit + the reviewers' picks"]
    decide -- otherwise --> wait(["wait"])
    nightly --> gate{"smoke test drew a world,<br/>no release-blocker is open"}
    gate -- no --> nothing(["nothing is published"])
    gate -- yes --> publish["publish the nightly pre-release<br/>notes: what merged since the last one"]
    publish --> launcher(["every launcher and server installs it"])
    publish --> kira(["Kira posts it; the first after<br/>20:00 UTC pings, once a day"])

When it releases

tools/release_train.py decide --kind game looks at the pull requests merged into main (first-parent merge commits) since the commit the published Nightly was built on:

• A pull request counts when it changed what players run: crates/ (not examples/, tests/, benches/), xtask/, packaging/, Cargo.toml, Cargo.lock, research-lock.json, or an exporter horizon-prepare compiles (the mods/horizon/src/test/java/dev/horizonlab/parity/*.java files its sources name). Markdown never counts.
• It goes in batches, so players get a few Nightlies a day rather than one per merge: once 5 pull requests count and the newest of them merged at least 5 minutes ago (a wave of merges becomes one Nightly), or once the oldest has waited 6 hours, however few they are.
• A pull request labelled release-now (a crash or a blocker) goes out at once.
• A pull request labelled skip-release (work players don't see) doesn't count and isn't listed in the notes.
• An open issue labelled release-blocker holds every Nightly, forced ones too (held by release-blocker #N); nightly build checks again right before publishing.
• One Nightly build at a time: while one runs or waits, the train starts none. At most 12 builds a UTC day (the reviewers' /nightly builds count): past that the train waits for the next day, except for release-now and force=true.
• A scheduled run takes main's newest commit where checks and parity-main passed (a later failed parity-main batch doesn't change it); after a run of either, the commit it passed on, once both passed there.

Nightly only

Ludovic decided on 2026-10-28 that Horizon ships only the Nightly. No vX.Y.Z game release is made any more: the last one stays GitHub's latest, so launchers and servers from before the switch (which don't read Nightlies) keep a working build until they update themselves. .github/release-line.txt still numbers the launcher's releases ([email protected]).

• Launchers (#3342) install the current nightly pre-release as their update, as the version nightly-<date>-<commit>, after checking its checksums signed for the Nightly channel. A player on the newest release, on a Nightly or on nothing moves to it at the next check; a version picked by hand or pinned by an instance stays. Worlds are in the game's folder, never in a version's, so nothing touches them. The "Nightly builds" setting is gone. The last week's Nightlies (nightly-<date>) and the old releases stay on the Versions page.
• Servers (#3331): horizon-server update, install.sh and install.ps1 follow the Nightly the same way.
• Kira posts each Nightly in #releases; the bug template asks for the version the game shows.
• Everyone gets the reviewers' picks. A Nightly is main plus the pull requests the trusted reviewers picked (below), so a pick reaches every player with the next Nightly. /nightly clear makes it main alone.

Labels

Label	Effect
feature	Listed under New in the Nightly's and the launcher's notes
release-now	A Nightly as soon as main's checks and parity-main pass, past the batch rule and the day's cap
skip-release	Doesn't trigger a Nightly or a launcher release and isn't listed in the notes
run-parity	Runs the parity job once on a fork's pull request
full-parity	Runs the pull request's parity job on playhorizon's runner only (a change every harness reaches)
parity-regression	Opened by the nightly audit for a harness that fails on main
release-blocker	On an issue: no Nightly is published while it is open (When it releases)

Pull requests without feature are listed under Fixed. Titles are written for players, since they become the notes.

Builds

Archive	Runs on	Checks
horizon-linux-x86_64.tar.gz	GitHub ubuntu-22.04 (an older glibc, so it runs on any recent distribution)	the standard loader and libraries only (tools/check_imports.py)
horizon-linux-musl-x86_64.tar.gz	Alpine 3.22 in Docker on GitHub ubuntu-22.04, targeting x86_64-unknown-linux-musl	musl loader and libraries only (tools/check_imports.py --musl), CPU baseline and every program's startup checked inside Alpine
horizon-server-linux-x86_64.tar.gz	the same Linux job, after the client archive: horizon-server, horizon-prepare and the game built with --no-default-features as horizon-engine (docs/server.md)	as above, and neither libasound.so.2 nor libudev.so.1 (tools/check_imports.py --server)
horizon-windows-x64.zip	GitHub windows-latest, C runtime linked in	system DLLs only (tools/check_imports.py); PE machine is x64
horizon-windows-arm64.zip	GitHub windows-11-vs2026-arm, built for aarch64-pc-windows-msvc, C runtime linked in	system DLLs only (tools/check_imports.py); PE machine is ARM64 and binaries start on the native runner
horizon-macos-arm64.zip	The Mac mini's runner (vars.MAC_RUNNER=macos-latest sends it to GitHub)	system libraries only (otool -L); a 1500-frame offscreen run of a new world (about 12 s; 300 frames end before the world has loaded) with resources prepared from the runner's own Minecraft (--offscreen --world smoke --frames 1500 --smoke-report: frames drawn, chunks generated and meshed, a player, no error)
horizon-macos-x64.zip	The same runner, cross-built for x86_64-apple-darwin (#1161)	as horizon-macos-arm64.zip, every program and the 1500-frame run under Rosetta 2; only the arm64 build's smoke-test.json is published

Each builds horizon-client, horizon-ctl, horizon-prepare and horizon-server in release mode, checks they exist, verifies Windows PE architectures, checks the --help and --version entry points, packages the folder with cargo run -p xtask -- package and archives it. The GitHub jobs keep their Cargo caches (Swatinem/rust-cache); the Mac mini keeps its target/. Only the Mac runs the game: GitHub's runners have neither a GPU worth the name nor the game files.

The musl archive is an optional manual download for musl-based Linux desktops; see installation and updates. The launcher still selects the standard glibc archive. The musl client and tools keep audio and controllers enabled and link musl dynamically (-C target-feature=-crt-static) to load graphics and window libraries at runtime. A failed musl build does not block other platforms: signing and publication omit the missing archive with a warning. The dedicated server archive remains glibc; the bundled musl horizon-server refuses automatic updates to glibc archives.

Musl gameplay parity is not yet validated. On x86-64, horizon-core::jmath uses the platform libm for several functions, so musl may round differently from glibc or HotSpot. Values near a simulation threshold could therefore change world generation or mob behaviour for the same seed. This build has no musl parity results; CLI startup checks do not establish gameplay parity. Before merging the platform support, record a successful Nightly dry run and a musl-desktop launch covering the title screen, a new world, sound and a controller.

Building Windows ARM64 locally: Windows ARM64.

Notes

tools/ci/nightly.py pick writes them: what reached main since the previous Nightly (### New, label feature, and ### Fixed, one line per pull request: its title, its number, the issues it closes; release_train.bullet), then the picked pull requests (### Not merged yet), ### Known issues (the bullets of docs/known-issues.md), thanks to outside contributors, the source commit and the disclaimer. The title stays Nightly <date> · <commit>, which launchers and servers read the version from. Pull request titles are written for players, since they become the notes.

Kira posts every Nightly once in Discord's #releases. It pings the releases role at most once a day: the first Nightly published after 20:00 UTC, with a short list of what came out since its last ping. Every other post is silent (no ping and Discord's @silent flag). Launcher releases post without a ping.

One channel

Before 2026-10-28 game releases were vX.Y.Z (numbering: versioning.md), each published by this workflow as the release and GitHub's latest. From 2026-10-23 there was one channel (the Alpha pre-releases and the daily promotion at 20:00 UTC ended: promote.yml and tools/release_channels.py are gone), and from 2026-10-27 the Nightly was opt-in beside it. Since 2026-10-28 the one channel is the Nightly (Nightly only).

Signed checksums

Every game and launcher release carries SHA256SUMS (the archives' SHA-256, sha256sum's format) and SHA256SUMS.sig: one base64 Ed25519 signature per line over horizon release <tag>\n followed by SHA256SUMS, so a signed list can't be reused under another tag.

• A sign job of its own makes them: tools/ci/sign_release.sh runs the runner's sha256sum and OpenSSL and nothing else. It is the only job that holds the private key, the RELEASE_SIGNING_KEY secret of the repository's release environment, which only real releases from main enter (environment: is empty in dry runs, which sign with a throwaway key). No code from the repository runs beside the key, so no pull request can read it by changing what a build compiles; the key is in no file, no log and no password vault, and nobody needs to read it.
• The publish job then checks the signature the way launchers do (horizon-release-sign verify --publishing, crates/horizon-release-sign), against the public keys built in from crates/horizon-release-sign/release-keys.txt. While that file lists no key, releases go out with SHA256SUMS alone and a warning (today's releases keep working). Once it lists one, a missing secret or a key it doesn't list fails the release before anything is published, instead of publishing something launchers would refuse.
• Making the key: tools/ci/new_release_key.sh pipes a new key from horizon-release-sign keygen straight into gh secret set RELEASE_SIGNING_KEY --env release and prints only the public key, which then goes into release-keys.txt by pull request.

Releases from before signing. The launcher and horizon-server update install only signed releases, with one exception, so a player or a server can still go back to an older version: every release published before signing began has its archives' SHA-256 pinned in crates/horizon-release-sign/legacy-releases.txt, built into both. Those releases install only with exactly those files, whatever GitHub serves for them later. The file was written once by tools/ci/legacy_releases.py from the SHA-256s GitHub computed at upload; it lists no signed release and never changes again (--check says whether it is complete).

Rotating the key (a new key every year, or at once if the old one may have leaked). The publish job signs with every key the environment holds, RELEASE_SIGNING_KEY and RELEASE_SIGNING_KEY_NEXT, and the two names take turns:

1. tools/ci/new_release_key.sh <the empty name> makes the next key; add its public key to release-keys.txt beside the old one. From then on every release is signed with both keys (one line each).
2. Wait until launchers and servers built with both keys are in use (a few weeks: launchers update themselves).
3. Delete the old key's secret (gh secret delete <its name> --env release) and its line in release-keys.txt. The next rotation makes its new key under the name just freed.

If the old key may have leaked, skip the wait: remove its public key from release-keys.txt at once and delete its secret. Launchers built before the new key cannot check releases signed only with it: they refuse to install them with a clear message, and players download the launcher again from the website once.

Starting one by hand

gh workflow run release --ref <branch>                         # dry run: decides and says why, starts nothing
gh workflow run release --ref main -f dry_run=false            # the usual decision, now
gh workflow run release --ref main -f dry_run=false -f force=true   # a Nightly now, whatever the rules (even while paused)

Pausing

gh variable set RELEASES_PAUSED --body 1     # no automatic Nightly or launcher releases
gh variable delete RELEASES_PAUSED           # back on; the next run catches up

While paused, runs still decide and say why in their summary; force=true still starts a Nightly, but never past an open release-blocker. The reviewers' /nightly and the 20:17 UTC build still run: a line paused in .github/trusted-reviewers.txt stops the reviewers' commands.

Rolling back

To stop the next Nightlies while a problem is fixed, label an issue about it release-blocker. The way out is forward: revert the pull request (or /nightly remove a pick), label the revert release-now, and a fixed Nightly is out as soon as main's checks and parity-main pass. Meanwhile a player can pick an earlier build on the launcher's Versions page (the last week's nightly-<date>, or a vX.Y.Z), and horizon-server update --version <tag> does the same for a server.

Launcher releases

.github/workflows/launcher.yml releases the launcher the same way. It starts on a push to main that changes launcher/, crates/horizon-prepare/ or crates/horizon-theme/ (what the launcher builds in), every hour at :47, and by hand; pushes to launcher and launcher-* branches only build.

• tools/release_train.py decide --kind launcher counts the pull requests merged since the last published launcher release that changed those paths (not launcher/links.json, which the launcher reads from main, nor tests or Markdown), with the same batching rule (5 pull requests settled 5 minutes, or the oldest 6 hours), release-now, skip-release and RELEASES_PAUSED.
• The tag is [email protected] on the release line ([email protected] first; the game and the launcher count their patches separately). Without .github/release-line.txt, launcher-vX.Y.Z: the patch after the last launcher tag, or the next minor (X.Y+1.0) when one of those pull requests is labelled feature. launcher-v0.2.0 was taken long before 0.2, hence the new prefix.
• No version-bump commits: the build gets the version in HORIZON_LAUNCHER_VERSION and the tag in HORIZON_LAUNCHER_TAG, which launcher/src/backend.rs reads with option_env! (falling back to Cargo.toml's), so --version, the window, the User-Agent and the self-update comparison all use them. Each build checks --version says it.
• Linux on GitHub ubuntu-22.04, Windows x64 on windows-latest, Windows ARM64 on windows-11-vs2026-arm, and macOS on the Mac mini's runner; each runs the launcher's tests and checks its libraries. The smoke test installs the newest game build for that platform; Intel macOS skips only the install step while no matching build is available yet, and Windows ARM64 installs the x64 build (under emulation) from a Nightly without an ARM64 one.
• Published with --latest=false: the repository's latest release stays a game release. Notes list the pull requests under New and Fixed; Kira posts launcher releases without pinging.
• Windows also publishes horizon-launcher-setup.exe (x64) and horizon-launcher-setup-arm64.exe, the launcher's binary under another name: with no arguments it opens the per-user setup (launcher/src/installer.rs) instead of Home, and horizon-launcher.exe uninstall removes it again. Both files are in the release's signed SHA256SUMS.

Making a launcher release GitHub's latest instead would break servers: every horizon-server already installed, install.sh and install.ps1 ask for releases/latest and expect the game's archives there, and a server that can't update can't fetch the fix either. So the game release stays the latest, and it sends people to the launcher (#1647): its notes open with a "Want to play? Download the Horizon launcher" block (PLAYERS in tools/release_train.py: raw HTML, which GitHub shows and launchers' notes skip; the site's Releases page leaves it out too), which PLAYERS_FOOTER repeats above the disclaimer at the end, and its archives carry labels ("Game files the launcher installs: Linux (…)") that GitHub shows in place of the file names. Everything that downloads matches the asset's name, which is unchanged.

Starting by hand, pausing and rolling back work as for the game (gh workflow run launcher …, RELEASES_PAUSED, gh release edit [email protected] --draft). Launchers older than the withdrawn release then see the previous one as the newest; launchers that already updated keep running.

With several game releases a day, the newest launcher release soon falls out of the newest 100 releases the launcher and the website read: both then look the newest launcher tags up one by one (git/matching-refs/tags/launcher@, then launcher-v if there is none, then releases/tags/<tag>).

Nightly channel

.github/workflows/nightly-build.yml builds and publishes the Nightly, the only build players get since 2026-10-28: when the batch train decides (Game releases, with the commit as base), once a night (20:17 UTC) and when a trusted reviewer asks (#2681). It builds main's newest commit where checks and parity-main passed, as a release does (or the base input), plus the pull requests the trusted reviewers picked, merged in their order; with none picked, that commit alone (tools/ci/nightly.py pick). No green commit (main's newest checks or parity-main run failed) builds nothing. A night with nothing new publishes nothing. A pull request goes in without a review: a trusted reviewer or Ludovic can pick any open pull request to main, reviewed or not (Ludovic, 2026-10-30, at the Team's request, because they were often blocked on the launcher; from 2026-10-28 it needed an approving review at the head that is built). The guards stay: the pull request must be open and target main, it must leave the CODEOWNERS-guarded files alone, the sets and builds stay capped, paused stops the reviewers, Ludovic's remove sticks, and the build runs without secrets. Each pull request is built at the head it was picked at; a push after the pick waits until someone picks it again (/nightly add).

Every Nightly ships its exact source. Its notes name the commit; horizon-source.tar.gz on the release is that commit's git archive, signed with the builds in SHA256SUMS; and nightly.py keep-source tags the commit nightly-source/<date>-<short>. That tag is never pruned, unlike the nightly-<date> release and the nightly/build-* branch (7 days each), because native mods pin a Nightly's commit and the launcher downloads that commit's source to build them (native-mods.md).

Trusted reviewers

One list, .github/trusted-reviewers.txt, governs two things for the people on it (kacpherek, rdbtCVS and Online5277 since 2026-10-27, levzzz5154 since 2026-10-28) and for Ludovic, with no repository role: /nightly (below) and triage by comment.

/close                    closes an issue as completed; a pull request is closed without merging
/close not planned        closes an issue as not planned
/close duplicate of #123  comments "Duplicate of #123", closes as a duplicate
/reopen
/label bug, good first issue    existing labels only; never creates one
/unlabel bug
/unclaim @login           releases someone else's claim on an issue (anyone can `/claim` and `/unclaim` their own)

• Where. On the first line of a comment on the issue or pull request (.github/workflows/triage-command.yml, tools/ci/triage_command.py). The workflow answers with one comment, 🤖 Triage: <action>, on behalf of @login, linking the comment that asked; a refusal answers too. Claims are their own workflow (.github/workflows/claim-command.yml, tools/ci/claim_command.py), open to everyone for their own claim.
• Refused. Any label with release in its name (release-now, release-blocker, skip-release) and full-parity; anything on Ludovic's own issues and pull requests; a merged pull request; a locked thread; more than 60 actions per reviewer in 24 hours (counted from the workflow's own comments).
• Safety. Same as /nightly: main's copy of the workflow runs, no pull request's code is checked out or run, the comment reaches the script as env vars, and the token has issues and pull-requests write only.
• Ludovic stays in charge. A one-line pull request removing a reviewer's line revokes it as soon as it merges (the file is his under CODEOWNERS). A line paused stops /nightly, the triage commands and the reviewers' claim powers for every reviewer; his own still work.
• Accounts, not names. Each line is <numeric GitHub ID> <login>, and every command decides by the comment author's ID (AUTHOR_ID, from github.event.comment.user.id), never by the login: GitHub hands a renamed or deleted account's login to whoever claims it next, while an ID is never reused. Ludovic is OWNER_ID in tools/ci/nightly_set.py. A line in any other form, a bare login included, grants nothing. python3 tools/ci/resolve_reviewer_ids.py fills in IDs for new lines; --check reports a listed ID whose login changed (a rename to look at) or whose account is gone.

Trusted reviewers pick the Nightly's pull requests

The Nightly is where contributors test pull requests before the gate. The trusted reviewers listed in .github/trusted-reviewers.txt (kacpherek, rdbtCVS and Online5277 since 2026-10-27, levzzz5154 since 2026-10-28) and Ludovic pick them with a comment on any issue or pull request (.github/workflows/nightly-command.yml, tools/ci/nightly_set.py):

/nightly 2640 2642     the set becomes these pull requests, merged onto main in this order; builds now
/nightly add 2650      adds one, or takes its newest head again; builds now
/nightly remove 2640   drops one; builds now
/nightly clear         empties the set: back to main alone; builds now
/nightly status        what the set holds
/nightly dry-run 2640  builds main + these and signs with a throwaway key; publishes nothing, the set stays

• The answer. The workflow answers in the thread with one comment starting 🤖 Nightly:: the set, the build's run, and later the published Nightly (or why it failed).

• The set is set.json on the nightly-state branch: who asked, when, and each pull request at the head it had then. A head is taken again only when someone asks again, never silently in between. The scheduled run builds every pull request at that head and drops the merged, closed and refused ones; the notes say so. builds.jsonl there records each build's set (for nightly_stats.py).

• Assembly. pick (main's code) fetches each head and merges them onto main with git merge-tree --write-tree: git objects only, no working tree, so no file, hook or filter of theirs reaches the runner. The result is pushed as nightly/build-<date>-<n> and build compiles that, without secrets, as for a release. Equal inputs give the same commit, so an unchanged set and main publish nothing new. On a conflict the build stops and answers which two pull requests conflict and where; a reviewer can then open an integration pull request from their fork that combines them, and pick that one instead.

• Old builds. The scheduled run deletes the nightly/build-* branches older than 7 days (Ludovic, 2026-10-27), and lists each in its summary. It never deletes the newest build, a build a published nightly or nightly-<date> release points to, or any other branch (nightly.py prune-builds).

• Refused. A pull request that changes a path CODEOWNERS gives to Ludovic (.github/, tools/ci/, the signing and update code, launcher/src/backend.rs, launcher/links.json…), more than 30 in a set, and more than 12 builds a day (the scheduled one always runs; 6 until 2026-10-28).

• No repository role. Reviewers need no permission on the repository: the list file and the workflow are the whole of it.

• Ludovic stays in charge.

  • Ludovic's /nightly clear or /nightly remove always applies. A pull request he removed can only come back when he adds it.
  • A one-line pull request removing a login from .github/trusted-reviewers.txt revokes it as soon as it merges (the file is his under CODEOWNERS).
  • A line paused in that file stops every reviewer's /nightly (and triage commands); his own still work.
  • Main and the gate don't change, but a pick reaches every player with the next Nightly.

• Jobs. pick assembles the commit and writes the notes: what reached main since the previous Nightly (New and Fixed, as a release's notes), then each picked pull request with its title, author and who picked it, the known issues and the thanks. build is game-build.yml, the same builds and smoke test as a release, with the version nightly-<date> and HORIZON_NIGHTLY="<date> <short sha>" built in, macOS signed ad hoc. sign signs SHA256SUMS for the channel (the message says horizon release nightly, so a nightly's signature never passes for a release's). publish checks the signature, the smoke test and that no release-blocker is open (as a release; a dry run only warns, so /nightly dry-run <pr> still tries a blocker's fix), renames the previous nightly pre-release to nightly-<its date>, publishes the new one as the nightly pre-release (never GitHub's latest), and deletes the nightly-<date> pre-releases and tags older than 7 days, nothing else.

• The signing key never meets the pull requests' code. Every workflow file and script runs from main's copy (a schedule and issue_comment run the default branch; uses: ./… takes the caller's commit), never a pull request's .github/, and a pull request that changes .github/ or tools/ci/ is refused. build compiles the merge with a read-only token that isn't kept in the checkout and no secrets; the runner's Developer ID keychain stays locked. sign is the only job with the key, and checks out only main's tools/ci/sign_release.sh, which hashes the archives with the runner's sha256sum and signs with its OpenSSL: it never runs what build made. publish runs main's horizon-release-sign, never the archives. Nothing a nightly's build leaves reaches a release either: game-build.yml's trusted input is true only for release.yml. An untrusted build runs its macOS jobs on the Mac mini's nightly runner (horizon-nightly, its own so a nightly never waits behind pull requests' parity queue; same machine and trust as the horizon-game runner), never the release runner (MAC_RUNNER), and its Rust cache has its own key and is never saved. A trusted build doesn't use the machine's shared sccache, which pull requests' jobs write (HORIZON_NO_SCCACHE, tools/ci/runner-env.sh).

• Players. Every launcher and horizon-server update install the nightly pre-release (Nightly only). It is never GitHub's latest: the last vX.Y.Z stays that, for launchers, servers and install.sh from before the switch.

gh workflow run "nightly build"                  # now; skips when the pick is the commit already published
gh workflow run "nightly build" -f force=true    # build and publish anyway
gh workflow run "nightly build" -f dry_run=true  # build and sign with a throwaway key, publish nothing

Nightly audit

.github/workflows/nightly.yml runs every harness from scratch (tools/parity.py --all) on main at 03:00 Paris time, on the Mac mini's runner, after preparing the resources again if the preparer or its exporters changed. tools/ci/parity_issues.py then compares the results with the previous audit (.local/parity-runner/audit.json in the runner's checkout):

• a harness that passed last night and fails now gets an issue labelled parity-regression, titled Parity regression: <harness>, with the commit, the run and the end of its log (one issue per harness: an open one gets a comment instead);
• a harness with an open issue that passes again gets its issue closed.

The bugs session picks the issues up. The audit's logs are the run's nightly-parity-logs artifact; harnesses listed linux_only are skipped there (see above). By hand: gh workflow run "nightly audit" (add -f dry_run=true to leave issues alone).

Disk

Releases ship often, so nothing they leave behind may grow without bound.

• Artifacts: every upload keeps 3 days (nightly parity logs 7); a release's assets live on the release itself.
• Mac runners: tools/ci/runner-gc.sh runs every 6 hours from launchd on the Mac mini, as a copy taken from main (~/.actions-runners/runner-gc.sh, plist dev.horizon.runner-gc), never from a pull request's checkout. It skips a runner in a job, removes a checkout's cargo build folders past 12 GB (MAX_TARGET_GB) and harness work folders under .local/ untouched for 3 days (STALE_DAYS), and keeps the 2 newest shared preparations (KEEP_PREPARED) while no runner works. It also stops harness JVMs (vanilla servers, oracles) left orphaned for 3 hours (ORPHAN_HOURS): parent gone, command line under a runner's checkout or a worktree's .local/. tools/parity.py stops its own harnesses on SIGTERM and SIGHUP (a cancelled job), so these come only from a SIGKILL or a harness run by hand. Runner builds go through the machine's sccache (tools/ci/runner-env.sh), so what it removes comes back from the cache. Update the copy after changing the script: ssh macmini 'curl -fsSL https://raw.githubusercontent.com/horizon-rust/horizon/main/tools/ci/runner-gc.sh -o ~/.actions-runners/runner-gc.sh'.