Compare commits

...
1 Commits
Author SHA1 Message Date
Horizon agentandClaude Opus 5.5 42ee9ad644 CI: run verify in the merge queue (merge_group)
The council decided to enable GitHub's merge queue on main (poll P43, decision D17). Main requires
`verify`, and no workflow listened to merge_group, so every queued pull request would have waited forever.

- ci.yml triggers on merge_group (checks_requested). verify stays unconditional and scopes the queue's
  commit against merge_group.base_sha/head_sha.
- Queue runs always use GitHub's runners: the commit can hold a fork's code, which job-started.sh
  refuses on ours, so SELF_HOSTED_CI=all must not send them there.
- Each entry gets its own concurrency group (merge_group.head_ref), never cancelled by the workflow.
- launcher and parity are skipped in the queue; main's push and parity-main still run them.
- docs/release-flow.md: what runs in the queue and the owner's steps to turn it on.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-12 10:43:04 +01:00
4 changed files with 93 additions and 21 deletions

No files matched your search

+32 -18
View File
@@ -1,5 +1,5 @@
name: checks
# verify: every pull request and every push to main, on GitHub's runners (the repository is public). It is the
# verify: every pull request, every merge queue entry and every push to main (runners: see the job). It is the
# check main requires, so it never has a condition: a skipped job counts as passed, and every run of this workflow
# must report a real verify (a pull request that changes no Rust finishes it in seconds instead).
# parity: the vanilla harnesses a pull request's changes can affect, on the Mac mini's runner, which holds the
@@ -7,11 +7,15 @@ name: checks
# request that changes a harness, oracle, probe or the runner: the rest are checked on main by parity-main. A fork's pull
# request runs it only when a maintainer adds the run-parity label (.github/workflows/run-parity.yml), so labels
# never start this workflow. See docs/release-flow.md.
# merge_group: the merge queue's commit (main's tip, the pull requests ahead in the queue and this one) runs verify
# before it lands, on GitHub's runners; parity and launcher wait for main's push (docs/release-flow.md#merge-queue).
on:
push:
branches: [main]
pull_request:
types: [opened, synchronize, reopened]
merge_group:
types: [checks_requested]
workflow_dispatch:
permissions:
contents: read
@@ -19,7 +23,8 @@ concurrency:
# A new push to a pull request replaces its running checks. On main, the running check finishes and only the
# newest push waits behind it: GitHub cancels a waiting run when a newer one joins the group. A burst of merges
# queued ~100 main runs on 2026-11-09; the release follows the newest finished main run (docs/release-flow.md).
group: checks-${{ github.event.pull_request.number || (github.event_name == 'push' && 'main') || github.sha }}
# Each merge queue entry has a group of its own (its temporary branch), and nothing cancels it: the queue does.
group: checks-${{ github.event.pull_request.number || github.event.merge_group.head_ref || (github.event_name == 'push' && 'main') || github.sha }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
# Test runs never reach the telemetry server: no live-count ping, no uploads (docs/telemetry.md).
@@ -34,11 +39,14 @@ jobs:
# forks' pull requests and main's pushes (which also save the Rust cache GitHub's pull request runs read), so both
# pools work at once. The repository variable SELF_HOSTED_CI moves them: `all` every job to ours, `light` (the
# desktop paused) or `off` every job to GitHub's. A fork's pull request sees no repository variable, so no switch
# could move it off a machine that is off: it always runs on GitHub's. On ours the build folder persists
# (CARGO_TARGET_DIR, outside the checkout), so the apt packages, Python and the Rust cache steps are GitHub's only.
# could move it off a machine that is off: it always runs on GitHub's. So does a merge queue's run, whatever
# SELF_HOSTED_CI says: its commit can hold a fork's code, which job-started.sh refuses on ours. On ours the build
# folder persists (CARGO_TARGET_DIR, outside the checkout), so the apt packages, Python and the Rust cache steps are
# GitHub's only.
verify:
runs-on: >-
${{ vars.SELF_HOSTED_CI != 'off' && vars.SELF_HOSTED_CI != 'light' && (vars.SELF_HOSTED_CI == 'all' ||
${{ vars.SELF_HOSTED_CI != 'off' && vars.SELF_HOSTED_CI != 'light' && github.event_name != 'merge_group' &&
(vars.SELF_HOSTED_CI == 'all' ||
github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository &&
(endsWith(format('{0}', github.event.pull_request.number), '0') ||
endsWith(format('{0}', github.event.pull_request.number), '1'))) &&
@@ -52,22 +60,24 @@ jobs:
if: runner.environment == 'github-hosted'
with:
python-version: '3.12'
# A merge queue's run reads its commit against main's tip (merge_group's base and head).
- name: Does this pull request need Rust checks?
if: github.event_name == 'pull_request'
if: github.event_name == 'pull_request' || github.event_name == 'merge_group'
id: rust
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.event.merge_group.head_sha }}
run: python3 tools/ci/rust_changes.py "$BASE_SHA" "$HEAD_SHA" >> "$GITHUB_OUTPUT"
# A pull request adds only its own commits to main, so it reads those; main's pushes read every ref's history.
# A pull request (or a merge queue entry) adds only its own commits to main, so it reads those; main's pushes
# read every ref's history.
- name: Keep game artifacts out of Git
env:
EVENT: ${{ github.event_name }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.event.merge_group.head_sha }}
run: |
python tools/check_public_tree.py
if [ "$EVENT" = pull_request ]; then
if [ "$EVENT" = pull_request ] || [ "$EVENT" = merge_group ]; then
python tools/check_public_tree.py --range "$BASE_SHA..$HEAD_SHA"
else
python tools/check_public_tree.py --history
@@ -84,21 +94,22 @@ jobs:
echo "WGPU_BACKEND=dx12" >> "$GITHUB_ENV"
# On a pull request, the next three checks compare with HEAD^1: the main that GitHub's merge commit puts the pull
# request on. pull_request.base.sha is main at the pull request's last push and falls behind, so main's own
# changes since then would count as the pull request's (#13874).
# changes since then would count as the pull request's (#13874). A merge queue's entry compares with main's tip
# (merge_group.base_sha), whatever the queue's merge method makes of HEAD^1.
# docs/layers.md: a crate depends only on its own layer or a lower one; today's exceptions only shrink.
- name: Crates follow the layers
env:
BASE: ${{ github.event_name == 'pull_request' && 'HEAD^1' || '' }}
BASE: ${{ github.event_name == 'pull_request' && 'HEAD^1' || github.event.merge_group.base_sha || '' }}
run: python3 tools/check_layers.py ${BASE:+--base "$BASE"}
# docs/layers.md#built-in-addresses: every built-in outside address in the Horizon layer's home; the rest only shrinks.
- name: Built-in addresses stay in their home
env:
BASE: ${{ github.event_name == 'pull_request' && 'HEAD^1' || '' }}
BASE: ${{ github.event_name == 'pull_request' && 'HEAD^1' || github.event.merge_group.base_sha || '' }}
run: python3 tools/check_addresses.py ${BASE:+--base "$BASE"}
# docs/layers.md#file-size: a Rust file over 3000 lines may not grow (pull requests; main has no base to compare).
- name: Big files don't grow
env:
BASE: ${{ github.event_name == 'pull_request' && 'HEAD^1' || '' }}
BASE: ${{ github.event_name == 'pull_request' && 'HEAD^1' || github.event.merge_group.base_sha || '' }}
run: python3 tools/check_file_size.py ${BASE:+--base "$BASE"}
- name: Test offline research tooling
run: python -m unittest discover -s tests -v
@@ -139,10 +150,13 @@ jobs:
cargo +stable test --locked --workspace --doc
# The launcher is its own workspace (launcher/Cargo.toml), so verify's cargo runs never build it. A pull request that
# changes it or a crate it builds is checked here, and so is every push to main: two pull requests can each pass
# alone and break it together (#2059 and the registry ratings did, 2026-10-26).
# alone and break it together (#2059 and the registry ratings did, 2026-10-26). Not in the merge queue: no check of
# its is required, so the queue wouldn't wait for it, and main's push runs it anyway.
launcher:
if: github.event_name != 'merge_group'
runs-on: >-
${{ vars.SELF_HOSTED_CI != 'off' && vars.SELF_HOSTED_CI != 'light' && (vars.SELF_HOSTED_CI == 'all' ||
${{ vars.SELF_HOSTED_CI != 'off' && vars.SELF_HOSTED_CI != 'light' && github.event_name != 'merge_group' &&
(vars.SELF_HOSTED_CI == 'all' ||
github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository &&
(endsWith(format('{0}', github.event.pull_request.number), '0') ||
endsWith(format('{0}', github.event.pull_request.number), '1'))) &&
+3 -1
View File
@@ -35,6 +35,8 @@ from a `ACTIONS_RUNNER_HOOK_JOB_STARTED` hook in the NixOS configuration.
minutes of work on GitHub, 11–13 on a desktop runner with a cold build folder) and `launcher` (1–11). Three
runners give ~180 runner-minutes an hour: about the fifth of the pull requests they take. Main's pushes also save
the Rust cache GitHub's runs of pull requests read. `SELF_HOSTED_CI=all` sends them all to ours.
- **The merge queue's `verify`**, whatever `SELF_HOSTED_CI` says: the queue's commit can hold a fork's code, which
`job-started.sh` refuses on ours ([release flow](release-flow.md#merge-queue)).
## Which jobs, and when
@@ -44,7 +46,7 @@ for):
| `SELF_HOSTED_CI` | Bots (`horizon-light`) | `checks`, `mod-registry`, `telemetry-server`, `pack load bench` (`horizon-checks`) |
| --- | --- | --- |
| unset (default) | ours | `verify`, `launcher`: ours for this repository's own pull requests ending in 0 or 1, GitHub's for the others, forks' and main's pushes; the rest ours, forks' pull requests excepted |
| `all` | ours | ours, every pull request |
| `all` | ours | ours, every pull request and main's pushes (not the merge queue's runs) |
| `light` | ours | GitHub's: the desktop is paused |
| `off` | GitHub's | GitHub's: everything back, in one switch |
+39 -2
View File
@@ -16,7 +16,8 @@ reopened; labels never start it):
| `verify` | Our checks runners (label `horizon-checks`, the desktop's) for this repository's own pull requests whose number ends in 0 or 1; GitHub `ubuntu-latest` for the others, forks' pull requests (they see no repository variable, so no switch could move them off a machine that is off) and main's pushes. `SELF_HOSTED_CI` moves them: `all` all to ours, `light` or `off` all to GitHub ([CI runners](ci-runners.md)) | Public-tree check, Python tests, rustfmt, clippy with warnings denied, Rust tests, one after another in one job. Dependencies and the workspace build come from a cache saved by main's runs (GitHub) or the runner's own build folder (ours) | ~8 min |
| `parity / parity` | A runner with the label `horizon-game`: the Mac mini's and playhorizon's (below) | Only for a change to a harness, an oracle, a probe or the runner (or the `run-parity` label): the vanilla harnesses the change can affect (`tools/parity.py --changed <base>`). Other pull requests skip it: [parity-main](#parity-on-main) checks them on main | seconds when nothing is affected; minutes per affected harness |
Pushes to main run `verify` only; the nightly audit covers main's harnesses.
Pushes to main run `verify` and `launcher`; parity-main covers main's harnesses. The merge queue, once an owner
turns it on, runs `verify` alone ([merge queue](#merge-queue)).
`verify` is the check main requires, so it has no condition and runs in every run of `checks`: GitHub counts a
*skipped* required check as passed, and labels used to start runs whose `verify` was skipped, which let
@@ -123,7 +124,7 @@ run there (parity has none). It is also a parity host (`playhorizon` in `tools/p
Pull requests merge with a merge commit, by hand: never `gh pr merge --auto`. Branch protection on main
requires `verify` for everyone, and Ludovic's review for the files `.github/CODEOWNERS` lists (what builds, signs
and ships releases). Branches don't have to be up to date with main: main's own `checks` run catches the rare
semantic conflict.
semantic conflict, and the [merge queue](#merge-queue), once on, before it lands.
**Who merges.** The Horizon Team (GitHub team `horizon-team`: kacpherek, levzzz5154, rdbtCVS, Online5277, dezer-x) and
Ludovic. The orchestrator's agents prove only the risky pull requests (below), so a contributor's ordinary pull
@@ -154,6 +155,42 @@ paths, `lan_e2e`, a screenshot for anything visible) before it merges. Risky mea
Unsure? Add `needs-proof`: it costs a few hours, a broken release costs more.
### Merge queue
The Council decided on a merge queue (poll P43, decision D17, 2026-11-12). With it, merging a pull request adds it to
main's queue; GitHub builds a temporary branch (`gh-readonly-queue/main/...`) of main's tip, the entries ahead of it
and this one, starts `checks` on it (`merge_group`), and moves main there only once the required checks pass on it.
Two pull requests that each pass alone but break main together no longer both land.
In a queue run (`.github/workflows/ci.yml`):
- `verify` runs as on a pull request, on the queue's commit against main's tip (`merge_group.base_sha`): the
public-tree check reads only the new commits, and a change with no Rust runs only the quick steps.
- It always runs on GitHub's runners, whatever `SELF_HOSTED_CI` says: the commit can hold a fork's code, which
`job-started.sh` refuses on ours ([CI runners](ci-runners.md)).
- Each entry has its own concurrency group and nothing in the workflow cancels it; the queue does when it drops
or rebuilds an entry.
- `parity` and `launcher` are skipped (a skipped job counts as passed): parity-main and `launcher` run on main's push
that follows, as today. Main's push still runs `checks` and parity-main, so releases change nothing.
Turning it on is an organisation owner's step; the workflow side is in place:
1. Settings → Rules → Rulesets → New branch ruleset: target `main` (Default branch), enforcement Active, rule
**Require merge queue** with merge method **Merge commit** (the release train and parity-main read `Merge pull
request #N` first-parent merges), build concurrency 5, minimum group size 1 (no wait), maximum group size 5,
"Require all queue entries to pass required checks" on, status check timeout 60 minutes (`verify` takes ~15 at
most). Add **Require status checks to pass** with `verify` (source GitHub Actions) in the same ruleset. Bypass
list: only whoever may still merge past the queue (`gh pr merge --admin`).
2. Classic branch protection on main keeps requiring `verify`: the queue's `merge_group` run reports `verify` on the
queue's commit, which satisfies it. Before requiring any other check in either place (`parity / parity` below
included), make sure queue runs report it under that name, or every entry waits for it until the timeout: a
skipped reusable-workflow call reports only `parity`.
After that, `gh pr merge N --match-head-commit SHA` queues the pull request instead of merging it: the command
returns at once and the pull request is merged later, or dropped from the queue (its timeline says why). Scripts that
treat its success as "merged" have to check the pull request's state afterwards. `--auto` stays forbidden;
`--admin` merges directly, past the queue, only for someone on the ruleset's bypass list.
`parity` becomes required once it has run green on pull requests for a while (a week without a false failure):
```sh
+19
View File
@@ -127,6 +127,25 @@ class OwnRunnerTests(unittest.TestCase):
if "horizon-checks" in value:
self.assertIn(FORK_GUARD, value, f"{path.name} {name}")
def test_the_merge_queue_runs_the_required_verify_on_githubs_runners(self):
# main requires verify; with a merge queue on, every entry waits for a verify of its merge_group run. A skipped
# job counts as passed, so verify has no condition. The queue's commit can hold a fork's code, which
# job-started.sh refuses on ours: SELF_HOSTED_CI=all must not send it there.
ci = (WORKFLOWS / "ci.yml").read_text()
self.assertIn("merge_group", triggers(ci))
verify = dict(jobs(ci))["verify"]
self.assertFalse([l for l in verify.splitlines() if l.startswith(" if:")])
for path in sorted(WORKFLOWS.glob("*.yml")):
text = path.read_text()
if "merge_group" not in triggers(text):
continue
for name, block in jobs(text):
value = runs_on(block)
job_if = next((l for l in block.splitlines() if l.startswith(" if:")), "")
# A job that runs only for another event never sees the queue.
if "self-hosted" in value and not re.search(r"github\.event_name (== '(?!merge_group)|!= 'merge_group')", job_if):
self.assertIn("github.event_name != 'merge_group'", value, f"{path.name} {name}")
def test_the_light_runners_refuse_forks_themselves(self):
script = (ROOT / "tools/ci/checks-runners.sh").read_text()
self.assertIn("ACTIONS_RUNNER_HOOK_JOB_STARTED=$shared/job-started.sh", script)