mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
feat(ci): add disposable external PR runners (#16513)
* feat(ci): add disposable external PR runners * test(ci): align platform lanes with disposable runner routing --------- Co-authored-by: yiliang114 <[email protected]>
This commit is contained in:
易良andyiliang114 authored and GitHub committed
1 parent
d85e911c78
commit
708ba347d3
17 files changed
+2427
-59
No files matched your search
@@ -0,0 +1,39 @@
|
||||
ARG BASE_IMAGE=node:22-bookworm-slim@sha256:5ec23b7c622b00547753bc6ffd78085dd868a68aeb41c8e0a1418f033e13d96d
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
ARG TARGETARCH
|
||||
ARG RUNNER_VERSION=2.338.2
|
||||
ARG RUNNER_RELEASE_URL=https://github.com/actions/runner/releases/download
|
||||
ARG DEBIAN_MIRROR=deb.debian.org
|
||||
ARG PIP_INDEX_URL=https://pypi.org/simple
|
||||
|
||||
RUN sed -i "s|deb.debian.org|${DEBIAN_MIRROR}|g" /etc/apt/sources.list.d/debian.sources \
|
||||
&& apt-get update -o Acquire::http::Timeout=30 \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
build-essential ca-certificates curl file gh git git-lfs jq libicu72 \
|
||||
libkrb5-3 libssl3 libunwind8 procps python3 python3-pip python3-venv \
|
||||
sudo tmux unzip zip zlib1g \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& python3 -m venv /opt/ci-tools \
|
||||
&& /opt/ci-tools/bin/pip install --no-cache-dir 'yamllint==1.35.1' \
|
||||
&& echo 'node ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/runner
|
||||
|
||||
ENV PATH="/opt/ci-tools/bin:${PATH}"
|
||||
|
||||
WORKDIR /home/node/actions-runner
|
||||
RUN case "${TARGETARCH}" in \
|
||||
amd64) runner_arch=x64; runner_sha=baebf204307bac6193480852732f0aba2ae97b6c5597c244f4d5558390147c40 ;; \
|
||||
arm64) runner_arch=arm64; runner_sha=670adb2f268fe21dcbb58b3db0e0aa9d5b3bddfe2c651481336e561ff86de945 ;; \
|
||||
*) echo "Unsupported architecture: ${TARGETARCH}" >&2; exit 1 ;; \
|
||||
esac \
|
||||
&& curl -fsSL --retry 3 --connect-timeout 15 --max-time 180 -o /tmp/runner.tar.gz \
|
||||
"${RUNNER_RELEASE_URL}/v${RUNNER_VERSION}/actions-runner-linux-${runner_arch}-${RUNNER_VERSION}.tar.gz" \
|
||||
&& echo "${runner_sha} /tmp/runner.tar.gz" | sha256sum -c - \
|
||||
&& tar -xzf /tmp/runner.tar.gz \
|
||||
&& rm /tmp/runner.tar.gz \
|
||||
&& chown -R node:node /home/node/actions-runner
|
||||
|
||||
COPY entrypoint.sh /usr/local/bin/ephemeral-runner
|
||||
RUN chmod 755 /usr/local/bin/ephemeral-runner
|
||||
USER node
|
||||
ENTRYPOINT ["/usr/local/bin/ephemeral-runner"]
|
||||
@@ -0,0 +1,140 @@
|
||||
# Ephemeral GitHub Actions runners
|
||||
|
||||
Each slot registers one just-in-time runner and runs it in a new container.
|
||||
After at most one job, the manager removes the container and runner registration
|
||||
before creating the next one. While the manager waits on a container it also
|
||||
enforces a per-slot deadline and removes a container whose runner does not exit;
|
||||
a manager crash is covered by reconciling leftover resources on the next start.
|
||||
|
||||
Each job has a separate Docker bridge with inter-container communication
|
||||
disabled. Deleting the network is part of cleanup and a failure blocks slot
|
||||
replacement. This does not itself prevent access to the host or private
|
||||
networks: configure and verify the host's external egress controls separately.
|
||||
|
||||
Use a dedicated host for untrusted pull requests. Docker shares the host kernel;
|
||||
jobs must not receive the host Docker socket, host directories, cloud instance
|
||||
credentials, internal network access, or long-lived secrets. A runner label is a
|
||||
scheduling selector, not an access-control boundary. Before enabling external
|
||||
pull requests, verify they cannot select other sensitive runner pools.
|
||||
|
||||
## Build the image
|
||||
|
||||
The image supports Linux x64 and ARM64. Its Node base is pinned to an OCI index
|
||||
digest and the runner downloads are checked against pinned SHA-256 digests.
|
||||
Use BuildKit to select both the base image and runner architecture:
|
||||
|
||||
```sh
|
||||
docker buildx build --platform linux/arm64 --load -t qwen-ephemeral-runner:trial scripts/ci/ephemeral-runner
|
||||
```
|
||||
|
||||
BuildKit supplies `TARGETARCH`. With Docker's legacy builder, build on the target
|
||||
architecture and pass its matching `TARGETARCH` explicitly (`amd64` or `arm64`).
|
||||
That argument selects only the runner archive; it does not change the base image
|
||||
architecture.
|
||||
|
||||
The runner image supports ARM64, but full PR CI on this image has not yet been
|
||||
verified. The repository also selects ARM64 linters; verify the complete CI
|
||||
toolchain before routing production work to either architecture.
|
||||
|
||||
Hosts with restricted upstream connectivity can set `DEBIAN_MIRROR`,
|
||||
`PIP_INDEX_URL` or `RUNNER_RELEASE_URL` at build time. The runner archive is
|
||||
still verified against its pinned digest when served from a mirror.
|
||||
|
||||
For a real deployment, publish the image to a registry and configure its
|
||||
immutable `repository@sha256:...` digest. A local `sha256:...` image ID may be
|
||||
used for a trial. Runner and base versions need routine security updates.
|
||||
|
||||
## Configure the manager
|
||||
|
||||
Run one manager on each host, managing only that host's Docker socket. Give each
|
||||
host a unique `poolId`; matching runner labels let GitHub share work across the
|
||||
hosts without a central scheduler. `slots` is a per-host limit, not a fleet total.
|
||||
New runner names start with `ecs-qwen-` so the existing CI worker caps and test
|
||||
timeouts apply. Restart recovery also accepts the earlier `qwen-` identities.
|
||||
|
||||
Install Docker, Node 22, `gh`, and `flock` on the host. The manager uses a
|
||||
dedicated PAT supplied as `GH_TOKEN`. The account must administer the target
|
||||
repository. Prefer a fine-grained PAT restricted to that repository with
|
||||
Administration: write and a short expiry. GitHub currently requires this broader
|
||||
permission for runner registration; it is not a runner-only credential. Some
|
||||
organization or outside-collaborator policies prevent fine-grained access. Do
|
||||
not silently replace it with a classic `repo` token, which can cover other
|
||||
repositories the account can access.
|
||||
|
||||
Store `GH_TOKEN=<PAT>` in `/etc/qwen-ephemeral-runner/credentials.env`, owned by
|
||||
root with mode `600`, inside the root-only configuration directory. The supplied
|
||||
systemd unit loads this file; do not put it in shell history, config JSON, job
|
||||
environments or the repository. Replace it and restart the service to rotate or
|
||||
renew the token. The service fails startup if `GH_TOKEN` is missing. Docker child
|
||||
environments are allowlisted, and only the one-time JIT configuration enters a
|
||||
job container. Host root and Docker administrators are trusted and can access
|
||||
host credentials; ordinary container isolation does not change that boundary.
|
||||
|
||||
Copy `config.example.json` to `/etc/qwen-ephemeral-runner/config.json` and
|
||||
replace the placeholders: repository, actual runner group ID, unique host/pool
|
||||
ID, immutable image reference, and matching architecture and labels. Use a
|
||||
unique test label and a controlled test repository for a trial. The supplied
|
||||
manual smoke workflow selects only `ecs-qwen-ephemeral-smoke`; a trial runner
|
||||
can register with just that label to avoid matching generic Linux jobs. The example runs
|
||||
one slot; increase only after measuring the host. Restrict configuration and
|
||||
credential directories to the manager account.
|
||||
|
||||
The `ttlSeconds` limit starts when the container starts, including idle time.
|
||||
A job assigned near the deadline can be interrupted. The manager removes and
|
||||
replaces the container; GitHub remains authoritative for job status. The
|
||||
manager logs cleanup but does not distinguish expiry from normal completion.
|
||||
A nonzero or unknown container exit status delays replenishment by 60 seconds;
|
||||
a successful short job is replenished immediately. GitHub remains authoritative
|
||||
for job success. Diagnostic export is best effort: stderr is not archived, and
|
||||
output exceeding the capture limits is discarded.
|
||||
`minDiskBytes` applies to the Docker data filesystem. Configure and verify
|
||||
disk quotas or a dedicated data volume as appropriate for the host.
|
||||
|
||||
## Install and operate
|
||||
|
||||
Install `manager.mjs` in `/opt/qwen-ephemeral-runner/`, install the supplied
|
||||
unit in `/etc/systemd/system/`, and create the configured state/log directories
|
||||
with permissions restricted to the manager account. From the trusted checkout:
|
||||
|
||||
```sh
|
||||
sudo install -d /opt/qwen-ephemeral-runner
|
||||
sudo install -d -m 700 /etc/qwen-ephemeral-runner /var/lib/qwen-ephemeral-runner /var/log/qwen-ephemeral-runner
|
||||
sudo install -m 644 scripts/ci/ephemeral-runner/manager.mjs /opt/qwen-ephemeral-runner/manager.mjs
|
||||
sudo install -m 644 scripts/ci/ephemeral-runner/qwen-ephemeral-runner.service /etc/systemd/system/qwen-ephemeral-runner.service
|
||||
```
|
||||
|
||||
Finish the configuration and PAT setup above before starting. A Docker-authorized user
|
||||
has host-level power; the unit is intended for a trusted dedicated host.
|
||||
|
||||
```sh
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now qwen-ephemeral-runner.service
|
||||
sudo systemctl status qwen-ephemeral-runner.service
|
||||
sudo journalctl -u qwen-ephemeral-runner.service -f
|
||||
```
|
||||
|
||||
systemd restarts an unexpectedly exited manager after ten seconds, including a
|
||||
clean exit. An explicit `systemctl stop` leaves it stopped. This is process
|
||||
supervision, not hang detection; inspect the journal and runner availability if
|
||||
slots are not replenishing.
|
||||
|
||||
Stopping the service stops replenishment, waits up to the configured grace
|
||||
period, and cleans up its containers. The next start reconciles abandoned
|
||||
resources first. A cleanup or ambiguous registration error blocks that slot;
|
||||
inspect its state and GitHub runner list before manual reconciliation. Never
|
||||
remove unrelated runners or run a global Docker prune.
|
||||
|
||||
```sh
|
||||
sudo systemctl stop qwen-ephemeral-runner.service
|
||||
```
|
||||
|
||||
After validating a single runner and its recovery paths, set the repository
|
||||
variable `EXTERNAL_PR_EPHEMERAL_RUNNER_ENABLED=true` to route untrusted fork PR
|
||||
build, test, lint and no-AK integration jobs to this pool. Unset, `false`, or any
|
||||
other value keeps those jobs on `ubuntu-latest`. Trusted and same-repository PRs
|
||||
retain their existing routing. The global kill-switch and hosted-fleet override
|
||||
still win. Leave the switch off until the host and GitHub access policies have
|
||||
been verified. Changing it affects newly evaluated jobs; cancel queued work and
|
||||
start a new workflow run when rolling back. It does not detect idle runners or
|
||||
migrate a queued job automatically.
|
||||
Set `EXTERNAL_PR_EPHEMERAL_ARCH=arm64` for an ARM64 pool; the default is x64.
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"repo": "QwenLM/qwen-code",
|
||||
"runnerGroupId": 1,
|
||||
"poolId": "external-ci-host-1",
|
||||
"image": "registry.example.invalid/qwen-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"labels": ["self-hosted", "linux", "x64", "ecs-qwen-ephemeral"],
|
||||
"slots": 1,
|
||||
"cpus": 4,
|
||||
"memory": "8g",
|
||||
"pidsLimit": 1024,
|
||||
"ttlSeconds": 10800,
|
||||
"shutdownGraceSeconds": 60,
|
||||
"minDiskBytes": 10737418240,
|
||||
"stateDir": "/var/lib/qwen-ephemeral-runner",
|
||||
"logDir": "/var/log/qwen-ephemeral-runner"
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
: "${RUNNER_JIT_CONFIG:?RUNNER_JIT_CONFIG is required}"
|
||||
jit_config="$RUNNER_JIT_CONFIG"
|
||||
unset RUNNER_JIT_CONFIG
|
||||
cd /home/node/actions-runner
|
||||
exec ./run.sh --jitconfig "$jit_config"
|
||||
@@ -0,0 +1,620 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { realpathSync } from 'node:fs';
|
||||
import {
|
||||
lstat,
|
||||
mkdir,
|
||||
readFile,
|
||||
readdir,
|
||||
rename,
|
||||
rm,
|
||||
statfs,
|
||||
writeFile,
|
||||
} from 'node:fs/promises';
|
||||
import { isAbsolute, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const sleep = (ms) =>
|
||||
new Promise((resolveSleep) => setTimeout(resolveSleep, ms));
|
||||
|
||||
export function command(
|
||||
program,
|
||||
args,
|
||||
{ env = process.env, input, timeout = 30000, binary = false } = {},
|
||||
) {
|
||||
return new Promise((resolveCommand, reject) => {
|
||||
const child = spawn(program, args, {
|
||||
env,
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
});
|
||||
const output = [];
|
||||
let outputSize = 0;
|
||||
let stderr = '';
|
||||
const timer = setTimeout(() => child.kill('SIGKILL'), timeout);
|
||||
child.stdout.on('data', (chunk) => {
|
||||
outputSize += chunk.length;
|
||||
if (outputSize > 4 * 1024 * 1024) child.kill('SIGKILL');
|
||||
else output.push(chunk);
|
||||
});
|
||||
child.stderr.on('data', (chunk) => {
|
||||
stderr += chunk;
|
||||
if (stderr.length > 1024 * 1024) child.kill('SIGKILL');
|
||||
});
|
||||
child.on('error', (error) => {
|
||||
clearTimeout(timer);
|
||||
reject(error);
|
||||
});
|
||||
child.on('close', (code) => {
|
||||
clearTimeout(timer);
|
||||
if (code === 0 && outputSize <= 4 * 1024 * 1024)
|
||||
resolveCommand(
|
||||
binary ? Buffer.concat(output) : Buffer.concat(output).toString(),
|
||||
);
|
||||
else {
|
||||
const status = Number(stderr.match(/HTTP (\d{3})/)?.[1] || 0);
|
||||
const error = new Error(
|
||||
`${program} failed (exit ${code}${status ? `, HTTP ${status}` : ''})`,
|
||||
);
|
||||
error.status = status;
|
||||
error.exitCode = code;
|
||||
reject(error);
|
||||
}
|
||||
});
|
||||
child.stdin.on('error', () => {});
|
||||
child.stdin.end(input);
|
||||
});
|
||||
}
|
||||
|
||||
function validate(c) {
|
||||
if (
|
||||
!/^[\w.-]+\/[\w.-]+$/.test(c.repo || '') ||
|
||||
!/^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,39}$/.test(c.poolId || '') ||
|
||||
!/^(?:[a-zA-Z0-9][a-zA-Z0-9._:/-]*@)?sha256:[a-f0-9]{64}$/.test(
|
||||
c.image || '',
|
||||
) ||
|
||||
!Number.isInteger(c.slots) ||
|
||||
c.slots < 1 ||
|
||||
c.slots > 10 ||
|
||||
!Number.isInteger(c.runnerGroupId) ||
|
||||
c.runnerGroupId < 1 ||
|
||||
!Array.isArray(c.labels) ||
|
||||
!c.labels.length ||
|
||||
!c.labels.every(
|
||||
(label) =>
|
||||
typeof label === 'string' && /^[a-zA-Z0-9_-]{1,63}$/.test(label),
|
||||
) ||
|
||||
typeof c.stateDir !== 'string' ||
|
||||
!isAbsolute(c.stateDir) ||
|
||||
typeof c.logDir !== 'string' ||
|
||||
!isAbsolute(c.logDir) ||
|
||||
(c.dockerSocket !== undefined &&
|
||||
(typeof c.dockerSocket !== 'string' ||
|
||||
!isAbsolute(c.dockerSocket) ||
|
||||
c.dockerSocket.includes('\n'))) ||
|
||||
!Number.isFinite(c.cpus) ||
|
||||
!(c.cpus > 0) ||
|
||||
!/^[1-9]\d*[mMgG]$/.test(c.memory || '') ||
|
||||
!Number.isInteger(c.pidsLimit) ||
|
||||
c.pidsLimit < 1 ||
|
||||
!Number.isFinite(c.ttlSeconds) ||
|
||||
!(c.ttlSeconds > 0) ||
|
||||
!Number.isFinite(c.shutdownGraceSeconds) ||
|
||||
!(c.shutdownGraceSeconds >= 0) ||
|
||||
!Number.isSafeInteger(c.minDiskBytes) ||
|
||||
c.minDiskBytes < 0
|
||||
)
|
||||
throw new Error('Invalid manager configuration');
|
||||
}
|
||||
|
||||
export function createManager(config, dependencies = {}) {
|
||||
validate(config);
|
||||
const execute = dependencies.command || command;
|
||||
const run = (program, args, options = {}) => {
|
||||
if (program !== 'docker') return execute(program, args, options);
|
||||
const source = options.env || process.env;
|
||||
const env = {};
|
||||
for (const key of ['PATH', 'HOME', 'DOCKER_CONFIG', 'XDG_RUNTIME_DIR']) {
|
||||
if (source[key] !== undefined) env[key] = source[key];
|
||||
}
|
||||
env.DOCKER_HOST = `unix://${config.dockerSocket || '/var/run/docker.sock'}`;
|
||||
if (args[0] === 'create' && options.env?.RUNNER_JIT_CONFIG)
|
||||
env.RUNNER_JIT_CONFIG = options.env.RUNNER_JIT_CONFIG;
|
||||
return execute(program, args, { ...options, env });
|
||||
};
|
||||
const wait = dependencies.sleep || sleep;
|
||||
const diskStats = dependencies.statfs || statfs;
|
||||
let stopping = false;
|
||||
let stopAt = Infinity;
|
||||
const stateDir = resolve(config.stateDir);
|
||||
const logDir = resolve(config.logDir);
|
||||
const lockPath = join(stateDir, 'manager.lock');
|
||||
const slotPath = (slot) => join(stateDir, `slot-${slot}.json`);
|
||||
const log = (message) => console.log(`[${config.poolId}] ${message}`);
|
||||
function parseJson(text) {
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch {
|
||||
throw new Error('Invalid JSON response from GitHub');
|
||||
}
|
||||
}
|
||||
|
||||
async function gh(args, input) {
|
||||
return run('gh', ['api', ...args, ...(input ? ['--input', '-'] : [])], {
|
||||
env: process.env,
|
||||
input: input && JSON.stringify(input),
|
||||
});
|
||||
}
|
||||
|
||||
async function save(slot, state) {
|
||||
const path = slotPath(slot);
|
||||
const temp = `${path}.${randomUUID()}.tmp`;
|
||||
await writeFile(temp, JSON.stringify(state), { mode: 0o600 });
|
||||
await rename(temp, path);
|
||||
}
|
||||
|
||||
async function load(slot) {
|
||||
try {
|
||||
const raw = await readFile(slotPath(slot), 'utf8');
|
||||
let state;
|
||||
try {
|
||||
state = JSON.parse(raw);
|
||||
} catch {
|
||||
throw new Error('Invalid slot state; manual reconciliation required');
|
||||
}
|
||||
// Reconcile identities created before the ECS-compatible name prefix.
|
||||
const name =
|
||||
typeof state?.name === 'string' ? state.name.replace(/^ecs-/, '') : '';
|
||||
const prefix = `qwen-${config.poolId}-${slot}-`;
|
||||
if (
|
||||
!name.startsWith(prefix) ||
|
||||
!/^[0-9a-f-]{36}$/.test(name.slice(prefix.length)) ||
|
||||
typeof state.uncertain !== 'boolean' ||
|
||||
(!state.uncertain &&
|
||||
(!Number.isSafeInteger(state.runnerId) || state.runnerId < 1)) ||
|
||||
(state.runnerId !== undefined &&
|
||||
(!Number.isSafeInteger(state.runnerId) || state.runnerId < 1))
|
||||
)
|
||||
throw new Error('Invalid slot state; manual reconciliation required');
|
||||
return state;
|
||||
} catch (error) {
|
||||
if (error.code === 'ENOENT') return null;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function listContainers(slot) {
|
||||
const output = await run('docker', [
|
||||
'ps',
|
||||
'-aq',
|
||||
'--no-trunc',
|
||||
'--filter',
|
||||
`label=qwen.pool=${config.poolId}`,
|
||||
'--filter',
|
||||
`label=qwen.slot=${slot}`,
|
||||
]);
|
||||
return output.trim().split('\n').filter(Boolean);
|
||||
}
|
||||
|
||||
async function listNetworks(slot) {
|
||||
const args = [
|
||||
'network',
|
||||
'ls',
|
||||
'-q',
|
||||
'--no-trunc',
|
||||
'--filter',
|
||||
`label=qwen.pool=${config.poolId}`,
|
||||
];
|
||||
if (slot !== undefined) args.push('--filter', `label=qwen.slot=${slot}`);
|
||||
return (await run('docker', args)).trim().split('\n').filter(Boolean);
|
||||
}
|
||||
|
||||
async function listRunners() {
|
||||
const output = await gh([
|
||||
'--paginate',
|
||||
'--jq',
|
||||
'.runners | @json',
|
||||
`/repos/${config.repo}/actions/runners?per_page=100`,
|
||||
]);
|
||||
const pages = output.trim().split('\n').map(parseJson);
|
||||
if (
|
||||
!pages.every(
|
||||
(runners) =>
|
||||
Array.isArray(runners) &&
|
||||
runners.every(
|
||||
(runner) =>
|
||||
typeof runner?.name === 'string' &&
|
||||
Number.isSafeInteger(runner.id) &&
|
||||
runner.id > 0,
|
||||
),
|
||||
)
|
||||
)
|
||||
throw new Error('Invalid GitHub runner list');
|
||||
return pages.flat();
|
||||
}
|
||||
|
||||
async function pruneLogs() {
|
||||
const files = [];
|
||||
for (const entry of await readdir(logDir, { withFileTypes: true })) {
|
||||
if (
|
||||
!entry.isFile() ||
|
||||
!entry.name.replace(/^ecs-/, '').startsWith(`qwen-${config.poolId}-`) ||
|
||||
!/\.(log|tar)$/.test(entry.name)
|
||||
)
|
||||
continue;
|
||||
const path = join(logDir, entry.name);
|
||||
const info = await lstat(path);
|
||||
files.push({ path, size: info.size, time: info.mtimeMs });
|
||||
}
|
||||
files.sort((a, b) => b.time - a.time);
|
||||
let size = 0;
|
||||
for (const file of files) {
|
||||
size += file.size;
|
||||
if (file.time < Date.now() - 7 * 86400000 || size > 100 * 1024 * 1024)
|
||||
await rm(file.path, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function cleanup(slot, state) {
|
||||
const containers = await listContainers(slot);
|
||||
const networks = await listNetworks(slot);
|
||||
if ((containers.length || networks.length) && !state?.name)
|
||||
throw new Error(
|
||||
'Container has no recorded runner identity; manual reconciliation required',
|
||||
);
|
||||
for (const id of containers) {
|
||||
try {
|
||||
const logs = await run('docker', ['logs', '--tail', '10000', id]);
|
||||
await writeFile(
|
||||
join(
|
||||
logDir,
|
||||
`${state?.name || `slot-${slot}`}-${id.slice(0, 12)}.log`,
|
||||
),
|
||||
logs.slice(-1024 * 1024),
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
await pruneLogs();
|
||||
} catch {
|
||||
log(`slot ${slot}: container logs unavailable`);
|
||||
}
|
||||
try {
|
||||
const archive = await run(
|
||||
'docker',
|
||||
['cp', `${id}:/home/node/actions-runner/_diag`, '-'],
|
||||
{ binary: true },
|
||||
);
|
||||
await writeFile(
|
||||
join(logDir, `${state.name}-${id.slice(0, 12)}.tar`),
|
||||
archive,
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
await pruneLogs();
|
||||
} catch {
|
||||
log(
|
||||
`slot ${slot}: runner diagnostics unavailable or larger than 4 MiB`,
|
||||
);
|
||||
}
|
||||
await run('docker', ['rm', '-f', '-v', id]);
|
||||
}
|
||||
if ((await listContainers(slot)).length)
|
||||
throw new Error('Container cleanup failed');
|
||||
for (const id of networks) await run('docker', ['network', 'rm', id]);
|
||||
if ((await listNetworks(slot)).length)
|
||||
throw new Error('Network cleanup failed');
|
||||
if (state?.name) {
|
||||
const all = await listRunners();
|
||||
const runners = all.filter((runner) => runner.name === state.name);
|
||||
if (
|
||||
state.runnerId &&
|
||||
(runners.some((runner) => runner.id !== state.runnerId) ||
|
||||
all.some(
|
||||
(runner) =>
|
||||
runner.id === state.runnerId && runner.name !== state.name,
|
||||
))
|
||||
)
|
||||
throw new Error(
|
||||
'Runner identity mismatch; manual reconciliation required',
|
||||
);
|
||||
if (
|
||||
runners.length > 1 ||
|
||||
runners.some(
|
||||
(runner) => !Number.isSafeInteger(runner.id) || runner.id < 1,
|
||||
)
|
||||
)
|
||||
throw new Error(
|
||||
'Ambiguous runner identity; manual reconciliation required',
|
||||
);
|
||||
if (state.uncertain && !runners.length)
|
||||
throw new Error(
|
||||
'JIT response was lost; runner not found, manual reconciliation required',
|
||||
);
|
||||
for (const runner of runners) {
|
||||
try {
|
||||
await gh([
|
||||
'-X',
|
||||
'DELETE',
|
||||
`/repos/${config.repo}/actions/runners/${runner.id}`,
|
||||
]);
|
||||
} catch (error) {
|
||||
if (error.status !== 404) throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
await rm(slotPath(slot), { force: true });
|
||||
log(`slot ${slot}: cleaned`);
|
||||
}
|
||||
|
||||
async function cycle(slot) {
|
||||
let earlyExit = false;
|
||||
let state = await load(slot);
|
||||
if (
|
||||
state ||
|
||||
(await listContainers(slot)).length ||
|
||||
(await listNetworks(slot)).length
|
||||
)
|
||||
await cleanup(slot, state);
|
||||
if (stopping) return;
|
||||
const dockerRoot = (
|
||||
await run('docker', ['info', '--format', '{{.DockerRootDir}}'])
|
||||
).trim();
|
||||
if (!isAbsolute(dockerRoot) || dockerRoot.includes('\n'))
|
||||
throw new Error('Invalid Docker Root Dir');
|
||||
const disk = await diskStats(dockerRoot);
|
||||
if (disk.bavail * disk.bsize < config.minDiskBytes)
|
||||
throw new Error(
|
||||
`Insufficient disk capacity on Docker Root Dir ${dockerRoot}`,
|
||||
);
|
||||
state = {
|
||||
name: `ecs-qwen-${config.poolId}-${slot}-${randomUUID()}`,
|
||||
createdAt: new Date().toISOString(),
|
||||
uncertain: true,
|
||||
};
|
||||
await save(slot, state);
|
||||
let result;
|
||||
try {
|
||||
result = parseJson(
|
||||
await gh(
|
||||
[
|
||||
'-X',
|
||||
'POST',
|
||||
`/repos/${config.repo}/actions/runners/generate-jitconfig`,
|
||||
],
|
||||
{
|
||||
name: state.name,
|
||||
runner_group_id: config.runnerGroupId,
|
||||
labels: config.labels,
|
||||
work_folder: '_work',
|
||||
},
|
||||
),
|
||||
);
|
||||
} catch (error) {
|
||||
if (error.status >= 400 && error.status < 500 && error.status !== 408) {
|
||||
await rm(slotPath(slot), { force: true });
|
||||
} else {
|
||||
await cleanup(slot, state);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
try {
|
||||
if (
|
||||
result?.runner?.name !== state.name ||
|
||||
!Number.isSafeInteger(result.runner.id) ||
|
||||
result.runner.id < 1
|
||||
)
|
||||
throw new Error('Invalid JIT runner identity');
|
||||
state.runnerId = result.runner.id;
|
||||
state.uncertain = false;
|
||||
await save(slot, state);
|
||||
if (!result.encoded_jit_config)
|
||||
throw new Error('GitHub returned no JIT configuration');
|
||||
const networkId = (
|
||||
await run('docker', [
|
||||
'network',
|
||||
'create',
|
||||
'--driver',
|
||||
'bridge',
|
||||
'--opt',
|
||||
'com.docker.network.bridge.enable_icc=false',
|
||||
'--label',
|
||||
`qwen.pool=${config.poolId}`,
|
||||
'--label',
|
||||
`qwen.slot=${slot}`,
|
||||
'--label',
|
||||
`qwen.instance=${state.name}`,
|
||||
state.name,
|
||||
])
|
||||
).trim();
|
||||
if (!networkId) throw new Error('Docker returned no network ID');
|
||||
state.networkId = networkId;
|
||||
await save(slot, state);
|
||||
const env = {
|
||||
...process.env,
|
||||
RUNNER_JIT_CONFIG: result.encoded_jit_config,
|
||||
};
|
||||
delete env.GH_TOKEN;
|
||||
delete env.GITHUB_TOKEN;
|
||||
const id = (
|
||||
await run(
|
||||
'docker',
|
||||
[
|
||||
'create',
|
||||
'--init',
|
||||
'--network',
|
||||
networkId,
|
||||
'--label',
|
||||
`qwen.pool=${config.poolId}`,
|
||||
'--label',
|
||||
`qwen.slot=${slot}`,
|
||||
'--label',
|
||||
`qwen.instance=${state.name}`,
|
||||
'--cpus',
|
||||
String(config.cpus),
|
||||
'--memory',
|
||||
config.memory,
|
||||
'--pids-limit',
|
||||
String(config.pidsLimit),
|
||||
'--log-driver',
|
||||
'json-file',
|
||||
'--log-opt',
|
||||
'max-size=10m',
|
||||
'--log-opt',
|
||||
'max-file=3',
|
||||
'--env',
|
||||
'RUNNER_JIT_CONFIG',
|
||||
config.image,
|
||||
],
|
||||
{ env },
|
||||
)
|
||||
).trim();
|
||||
state.containerId = id;
|
||||
await save(slot, state);
|
||||
if (!id) throw new Error('Docker returned no container ID');
|
||||
if (!stopping) {
|
||||
await run('docker', ['start', id]);
|
||||
log(
|
||||
`slot ${slot}: started runner ${state.name}, id ${state.runnerId}, container ${id}`,
|
||||
);
|
||||
const deadline = Date.now() + config.ttlSeconds * 1000;
|
||||
let exitCode;
|
||||
const watching = run('docker', ['wait', id], {
|
||||
timeout: (config.ttlSeconds + 10) * 1000,
|
||||
}).then(
|
||||
(output) => {
|
||||
const status = output.trim();
|
||||
exitCode =
|
||||
/^\d+$/.test(status) && Number(status) <= 255
|
||||
? Number(status)
|
||||
: null;
|
||||
return true;
|
||||
},
|
||||
() => {
|
||||
exitCode = null;
|
||||
return true;
|
||||
},
|
||||
);
|
||||
while (Date.now() < Math.min(deadline, stopAt)) {
|
||||
if (await Promise.race([watching, wait(1000).then(() => false)]))
|
||||
break;
|
||||
}
|
||||
if (exitCode !== undefined && exitCode !== 0) {
|
||||
earlyExit = true;
|
||||
log(
|
||||
`slot ${slot}: runner exit status ${exitCode ?? 'unknown'}; next JIT request is delayed`,
|
||||
);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await cleanup(slot, state);
|
||||
}
|
||||
return { earlyExit };
|
||||
}
|
||||
|
||||
async function acquireLock() {
|
||||
await mkdir(stateDir, { recursive: true, mode: 0o700 });
|
||||
await mkdir(logDir, { recursive: true, mode: 0o700 });
|
||||
const child = spawn('flock', ['-n', lockPath, 'cat'], {
|
||||
stdio: ['pipe', 'pipe', 'ignore'],
|
||||
});
|
||||
await new Promise((resolveLock, reject) => {
|
||||
child.once('error', reject);
|
||||
child.once('exit', () =>
|
||||
reject(new Error('Could not acquire manager lock')),
|
||||
);
|
||||
child.stdout.once('data', resolveLock);
|
||||
child.stdin.on('error', () => {});
|
||||
child.stdin.write('locked\n');
|
||||
});
|
||||
return child;
|
||||
}
|
||||
|
||||
async function start() {
|
||||
const lock = await acquireLock();
|
||||
try {
|
||||
const slots = (
|
||||
await run('docker', [
|
||||
'ps',
|
||||
'-a',
|
||||
'--filter',
|
||||
`label=qwen.pool=${config.poolId}`,
|
||||
'--format',
|
||||
'{{.Label "qwen.slot"}}',
|
||||
])
|
||||
)
|
||||
.trim()
|
||||
.split('\n')
|
||||
.filter(Boolean);
|
||||
const states = (await readdir(stateDir))
|
||||
.map((name) => name.match(/^slot-(\d+)\.json$/)?.[1])
|
||||
.filter(Boolean);
|
||||
const networkSlots = await Promise.all(
|
||||
(await listNetworks()).map(async (id) =>
|
||||
(
|
||||
await run('docker', [
|
||||
'network',
|
||||
'inspect',
|
||||
'--format',
|
||||
'{{index .Labels "qwen.slot"}}',
|
||||
id,
|
||||
])
|
||||
).trim(),
|
||||
),
|
||||
);
|
||||
if (
|
||||
[...slots, ...states, ...networkSlots].some(
|
||||
(slot) =>
|
||||
!/^\d+$/.test(slot) ||
|
||||
Number(slot) < 1 ||
|
||||
Number(slot) > config.slots,
|
||||
)
|
||||
) {
|
||||
throw new Error(
|
||||
'Pool contains resources outside configured slots; reconcile before starting',
|
||||
);
|
||||
}
|
||||
await Promise.all(
|
||||
Array.from({ length: config.slots }, async (_, i) => {
|
||||
const slot = i + 1;
|
||||
while (!stopping) {
|
||||
try {
|
||||
const result = await cycle(slot);
|
||||
if (result?.earlyExit && !stopping) {
|
||||
log(`slot ${slot}: backing off 60 s before replenishing`);
|
||||
await wait(60000);
|
||||
}
|
||||
} catch (error) {
|
||||
log(
|
||||
`slot ${slot}: ${error.message}; retrying without replenishing`,
|
||||
);
|
||||
await wait(10000);
|
||||
}
|
||||
}
|
||||
}),
|
||||
);
|
||||
} finally {
|
||||
lock.stdin.end();
|
||||
}
|
||||
}
|
||||
function stop() {
|
||||
stopping = true;
|
||||
stopAt = Math.min(stopAt, Date.now() + config.shutdownGraceSeconds * 1000);
|
||||
}
|
||||
return { start, stop, cycle, cleanup };
|
||||
}
|
||||
|
||||
if (
|
||||
process.argv[1] &&
|
||||
realpathSync(process.argv[1]) === fileURLToPath(import.meta.url)
|
||||
) {
|
||||
try {
|
||||
if (!process.env.GH_TOKEN?.trim()) throw new Error('GH_TOKEN is required');
|
||||
const config = JSON.parse(
|
||||
await readFile(resolve(process.argv[2] || ''), 'utf8'),
|
||||
);
|
||||
const manager = createManager(config);
|
||||
process.on('SIGTERM', manager.stop);
|
||||
process.on('SIGINT', manager.stop);
|
||||
await manager.start();
|
||||
} catch (error) {
|
||||
console.error(`manager: ${error.message}`);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,822 @@
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import {
|
||||
mkdtemp,
|
||||
mkdir,
|
||||
readFile,
|
||||
rm,
|
||||
symlink,
|
||||
writeFile,
|
||||
} from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { command as runCommand, createManager } from './manager.mjs';
|
||||
|
||||
async function fixture(t) {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'runner-manager-'));
|
||||
t.after(() => rm(dir, { recursive: true, force: true }));
|
||||
await mkdir(join(dir, 'state'));
|
||||
await mkdir(join(dir, 'logs'));
|
||||
const config = {
|
||||
repo: 'owner/repo',
|
||||
poolId: 'trial',
|
||||
image: `image@sha256:${'a'.repeat(64)}`,
|
||||
slots: 1,
|
||||
runnerGroupId: 1,
|
||||
labels: ['self-hosted', 'linux', 'ARM64', 'trial-only'],
|
||||
cpus: 1,
|
||||
memory: '1g',
|
||||
pidsLimit: 100,
|
||||
ttlSeconds: 2,
|
||||
shutdownGraceSeconds: 0,
|
||||
minDiskBytes: 0,
|
||||
stateDir: join(dir, 'state'),
|
||||
logDir: join(dir, 'logs'),
|
||||
};
|
||||
const calls = [];
|
||||
const containers = new Map();
|
||||
const networks = new Map();
|
||||
let runner;
|
||||
let counter = 0;
|
||||
let foreignCounter = 0;
|
||||
let failRemove = false;
|
||||
let failNetworkRemove = false;
|
||||
let keepContainerAfterRemove = false;
|
||||
let loseJit = false;
|
||||
const labelsFromArgs = (args) => {
|
||||
const labels = {};
|
||||
for (let i = 0; i < args.length; i += 1) {
|
||||
if (args[i] !== '--label' || typeof args[i + 1] !== 'string') continue;
|
||||
const at = args[i + 1].indexOf('=');
|
||||
if (at > 0) labels[args[i + 1].slice(0, at)] = args[i + 1].slice(at + 1);
|
||||
}
|
||||
return labels;
|
||||
};
|
||||
const filtersFromArgs = (args) => {
|
||||
const filters = [];
|
||||
for (let i = 0; i < args.length; i += 1) {
|
||||
if (args[i] === '--filter' && typeof args[i + 1] === 'string')
|
||||
filters.push(args[i + 1].replace(/^label=/, ''));
|
||||
}
|
||||
return filters;
|
||||
};
|
||||
const matchesFilters = (entry, args) =>
|
||||
filtersFromArgs(args).every((filter) => {
|
||||
const at = filter.indexOf('=');
|
||||
if (at < 0) return true;
|
||||
return entry.labels[filter.slice(0, at)] === filter.slice(at + 1);
|
||||
});
|
||||
async function command(program, args, options = {}) {
|
||||
calls.push({ program, args, options });
|
||||
if (program === 'gh') {
|
||||
const path = args.find((v) => v.startsWith('/'));
|
||||
if (path.includes('generate-jitconfig')) {
|
||||
const body = JSON.parse(options.input);
|
||||
runner = { id: ++counter, name: body.name };
|
||||
if (loseJit) throw new Error('network failure');
|
||||
return JSON.stringify({ runner, encoded_jit_config: 'jit-secret' });
|
||||
}
|
||||
if (args.includes('DELETE')) {
|
||||
runner = undefined;
|
||||
return '';
|
||||
}
|
||||
return JSON.stringify(runner ? [runner] : []);
|
||||
}
|
||||
if (args[0] === 'ps') {
|
||||
const matched = [...containers.entries()].filter(([, entry]) =>
|
||||
matchesFilters(entry, args),
|
||||
);
|
||||
if (!matched.length) return '';
|
||||
if (args.includes('--format'))
|
||||
return `${matched.map(([, entry]) => entry.labels['qwen.slot'] ?? '').join('\n')}\n`;
|
||||
return `${matched.map(([id]) => id).join('\n')}\n`;
|
||||
}
|
||||
if (args[0] === 'network' && args[1] === 'ls') {
|
||||
const matched = [...networks.entries()].filter(([, entry]) =>
|
||||
matchesFilters(entry, args),
|
||||
);
|
||||
return matched.length ? `${matched.map(([id]) => id).join('\n')}\n` : '';
|
||||
}
|
||||
if (args[0] === 'network' && args[1] === 'create') {
|
||||
const id = `network-${counter}`;
|
||||
networks.set(id, { labels: labelsFromArgs(args) });
|
||||
return id;
|
||||
}
|
||||
if (args[0] === 'network' && args[1] === 'rm') {
|
||||
if (failNetworkRemove) throw new Error('network removal failed');
|
||||
networks.delete(args[args.length - 1]);
|
||||
return '';
|
||||
}
|
||||
if (args[0] === 'network' && args[1] === 'inspect') {
|
||||
const entry = networks.get(args[args.length - 1]);
|
||||
return `${entry?.labels['qwen.slot'] ?? ''}\n`;
|
||||
}
|
||||
if (args[0] === 'info') return `${dir}\n`;
|
||||
if (args[0] === 'create') {
|
||||
const id = `container-${counter}`;
|
||||
containers.set(id, { labels: labelsFromArgs(args) });
|
||||
return id;
|
||||
}
|
||||
if (args[0] === 'logs') return 'runner log';
|
||||
if (args[0] === 'rm') {
|
||||
if (failRemove) throw new Error('docker unavailable');
|
||||
if (!keepContainerAfterRemove) containers.delete(args[args.length - 1]);
|
||||
return '';
|
||||
}
|
||||
if (args[0] === 'wait') return '0\n';
|
||||
if (args[0] === 'start') return '';
|
||||
return '';
|
||||
}
|
||||
return {
|
||||
config,
|
||||
calls,
|
||||
command,
|
||||
sleep: async () => {},
|
||||
addForeignNetwork(labels) {
|
||||
const id = `network-foreign-${++foreignCounter}`;
|
||||
networks.set(id, { labels });
|
||||
return id;
|
||||
},
|
||||
networkIds: () => [...networks.keys()],
|
||||
setFailRemove: (v) => {
|
||||
failRemove = v;
|
||||
},
|
||||
setLoseJit: (v) => {
|
||||
loseJit = v;
|
||||
},
|
||||
setFailNetworkRemove: (v) => {
|
||||
failNetworkRemove = v;
|
||||
},
|
||||
setKeepContainerAfterRemove: (v) => {
|
||||
keepContainerAfterRemove = v;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
test('creates a new runner and container per job and keeps GitHub token out of Docker', async (t) => {
|
||||
const previousToken = process.env.GH_TOKEN;
|
||||
process.env.GH_TOKEN = 'host-only-pat';
|
||||
t.after(() => {
|
||||
if (previousToken === undefined) delete process.env.GH_TOKEN;
|
||||
else process.env.GH_TOKEN = previousToken;
|
||||
});
|
||||
const f = await fixture(t);
|
||||
const manager = createManager(f.config, { command: f.command });
|
||||
await manager.cycle(1);
|
||||
await manager.cycle(1);
|
||||
const creates = f.calls.filter(
|
||||
(c) => c.program === 'docker' && c.args[0] === 'create',
|
||||
);
|
||||
const requests = f.calls.filter(
|
||||
(c) =>
|
||||
c.program === 'gh' &&
|
||||
c.args.includes('/repos/owner/repo/actions/runners/generate-jitconfig'),
|
||||
);
|
||||
assert.ok(
|
||||
requests.every((c) =>
|
||||
JSON.parse(c.options.input).name.startsWith('ecs-qwen-'),
|
||||
),
|
||||
);
|
||||
assert.ok(requests.every((c) => c.options.env.GH_TOKEN === 'host-only-pat'));
|
||||
assert.equal(creates.length, 2);
|
||||
assert.notEqual(
|
||||
JSON.parse(requests[0].options.input).name,
|
||||
JSON.parse(requests[1].options.input).name,
|
||||
);
|
||||
assert.deepEqual(
|
||||
JSON.parse(requests[0].options.input).labels,
|
||||
f.config.labels,
|
||||
);
|
||||
assert.ok(
|
||||
creates.every(
|
||||
(c) =>
|
||||
!c.options.env.GH_TOKEN &&
|
||||
c.options.env.RUNNER_JIT_CONFIG === 'jit-secret',
|
||||
),
|
||||
);
|
||||
assert.deepEqual(
|
||||
creates.map((c) => c.args[c.args.indexOf('--network') + 1]),
|
||||
['network-1', 'network-2'],
|
||||
);
|
||||
const networks = f.calls.filter(
|
||||
(c) =>
|
||||
c.program === 'docker' &&
|
||||
c.args[0] === 'network' &&
|
||||
c.args[1] === 'create',
|
||||
);
|
||||
assert.equal(networks.length, 2);
|
||||
assert.ok(
|
||||
networks.every((c) =>
|
||||
c.args.includes('com.docker.network.bridge.enable_icc=false'),
|
||||
),
|
||||
);
|
||||
assert.equal(
|
||||
f.calls.filter((c) => c.program === 'docker' && c.args[0] === 'rm').length,
|
||||
2,
|
||||
);
|
||||
});
|
||||
|
||||
test('failed cleanup blocks replacement until recovery cleans the old container', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const manager = createManager(f.config, { command: f.command });
|
||||
f.setFailRemove(true);
|
||||
await assert.rejects(manager.cycle(1), /docker unavailable/);
|
||||
await assert.rejects(manager.cycle(1), /docker unavailable/);
|
||||
assert.equal(
|
||||
f.calls.filter((c) => c.program === 'docker' && c.args[0] === 'create')
|
||||
.length,
|
||||
1,
|
||||
);
|
||||
f.setFailRemove(false);
|
||||
await manager.cycle(1);
|
||||
const ops = f.calls
|
||||
.filter((c) => c.program === 'docker')
|
||||
.map((c) => c.args[0]);
|
||||
assert.ok(ops.lastIndexOf('rm') > ops.lastIndexOf('create'));
|
||||
assert.equal(ops.filter((op) => op === 'create').length, 2);
|
||||
});
|
||||
|
||||
test('lost JIT response reconciles the recorded name before requesting another', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const manager = createManager(f.config, { command: f.command });
|
||||
f.setLoseJit(true);
|
||||
await assert.rejects(manager.cycle(1), /network failure/);
|
||||
await assert.rejects(readFile(join(f.config.stateDir, 'slot-1.json')), {
|
||||
code: 'ENOENT',
|
||||
});
|
||||
assert.ok(
|
||||
f.calls.some((c) => c.program === 'gh' && c.args.includes('DELETE')),
|
||||
);
|
||||
f.setLoseJit(false);
|
||||
await manager.cycle(1);
|
||||
});
|
||||
|
||||
test('an unconfirmed JIT request blocks replacement when no runner can be found', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const command = (program, args, options) => {
|
||||
if (
|
||||
program === 'gh' &&
|
||||
args.includes('/repos/owner/repo/actions/runners/generate-jitconfig')
|
||||
) {
|
||||
return Promise.reject(new Error('connection lost'));
|
||||
}
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
const manager = createManager(f.config, { command });
|
||||
await assert.rejects(manager.cycle(1), /manual reconciliation/);
|
||||
await assert.rejects(manager.cycle(1), /manual reconciliation/);
|
||||
const state = JSON.parse(
|
||||
await readFile(join(f.config.stateDir, 'slot-1.json'), 'utf8'),
|
||||
);
|
||||
assert.equal(state.uncertain, true);
|
||||
assert.equal(
|
||||
f.calls.filter((c) => c.program === 'docker' && c.args[0] === 'create')
|
||||
.length,
|
||||
0,
|
||||
);
|
||||
});
|
||||
|
||||
test('a failed container start is cleaned up before another cycle', async (t) => {
|
||||
const f = await fixture(t);
|
||||
let fail = true;
|
||||
const command = (program, args, options) => {
|
||||
if (program === 'docker' && args[0] === 'start' && fail) {
|
||||
fail = false;
|
||||
return Promise.reject(new Error('start failed'));
|
||||
}
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
const manager = createManager(f.config, { command });
|
||||
await assert.rejects(manager.cycle(1), /start failed/);
|
||||
assert.equal(
|
||||
f.calls.filter((c) => c.program === 'docker' && c.args[0] === 'rm').length,
|
||||
1,
|
||||
);
|
||||
await manager.cycle(1);
|
||||
});
|
||||
|
||||
test('watchdog removes a container that does not exit', async (t) => {
|
||||
const f = await fixture(t);
|
||||
f.config.ttlSeconds = 0.01;
|
||||
const command = (program, args, options) =>
|
||||
program === 'docker' && args[0] === 'wait'
|
||||
? new Promise(() => {})
|
||||
: f.command(program, args, options);
|
||||
await createManager(f.config, { command }).cycle(1);
|
||||
assert.equal(
|
||||
f.calls.filter((c) => c.program === 'docker' && c.args[0] === 'rm').length,
|
||||
1,
|
||||
);
|
||||
});
|
||||
|
||||
test('checks Docker root disk rather than the state directory', async (t) => {
|
||||
const f = await fixture(t);
|
||||
let checked;
|
||||
const manager = createManager(f.config, {
|
||||
command: f.command,
|
||||
statfs: async (path) => {
|
||||
checked = path;
|
||||
return { bavail: 0, bsize: 4096 };
|
||||
},
|
||||
});
|
||||
f.config.minDiskBytes = 1;
|
||||
await assert.rejects(manager.cycle(1), /Insufficient disk capacity/);
|
||||
assert.equal(checked, join(f.config.stateDir, '..'));
|
||||
assert.equal(f.calls.filter((c) => c.program === 'gh').length, 0);
|
||||
const info = f.calls.find(
|
||||
(c) => c.program === 'docker' && c.args[0] === 'info',
|
||||
);
|
||||
assert.equal(info.options.env.DOCKER_HOST, 'unix:///var/run/docker.sock');
|
||||
});
|
||||
|
||||
test('rejects same-name runner with a different ID without deleting it', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const state = {
|
||||
name: 'qwen-trial-1-aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa',
|
||||
runnerId: 42,
|
||||
uncertain: false,
|
||||
};
|
||||
await writeFile(
|
||||
join(f.config.stateDir, 'slot-1.json'),
|
||||
JSON.stringify(state),
|
||||
);
|
||||
const command = (program, args, options) => {
|
||||
if (program === 'gh' && args.includes('--paginate'))
|
||||
return Promise.resolve(JSON.stringify([{ name: state.name, id: 43 }]));
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
await assert.rejects(
|
||||
createManager(f.config, { command }).cycle(1),
|
||||
/identity mismatch/,
|
||||
);
|
||||
assert.ok(
|
||||
!f.calls.some((c) => c.program === 'gh' && c.args.includes('DELETE')),
|
||||
);
|
||||
assert.equal(
|
||||
JSON.parse(await readFile(join(f.config.stateDir, 'slot-1.json'), 'utf8'))
|
||||
.runnerId,
|
||||
42,
|
||||
);
|
||||
});
|
||||
|
||||
test('reconciles a legacy identity on a later page before registering its replacement', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const name = 'qwen-trial-1-aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa';
|
||||
await writeFile(
|
||||
join(f.config.stateDir, 'slot-1.json'),
|
||||
JSON.stringify({
|
||||
name,
|
||||
runnerId: 42,
|
||||
uncertain: false,
|
||||
}),
|
||||
);
|
||||
let deleted = false;
|
||||
const command = (program, args, options) => {
|
||||
if (program === 'gh' && args.includes('--paginate') && !deleted) {
|
||||
assert.ok(!args.includes('--slurp'));
|
||||
assert.equal(args[args.indexOf('--jq') + 1], '.runners | @json');
|
||||
return Promise.resolve('[]\n' + JSON.stringify([{ name, id: 42 }]));
|
||||
}
|
||||
if (
|
||||
program === 'gh' &&
|
||||
args.includes('/repos/owner/repo/actions/runners/42')
|
||||
) {
|
||||
deleted = true;
|
||||
}
|
||||
if (
|
||||
program === 'gh' &&
|
||||
args.includes('/repos/owner/repo/actions/runners/generate-jitconfig')
|
||||
) {
|
||||
assert.ok(
|
||||
deleted,
|
||||
'legacy registration must be removed before replacement',
|
||||
);
|
||||
}
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
await createManager(f.config, { command }).cycle(1);
|
||||
assert.ok(deleted);
|
||||
});
|
||||
|
||||
test('startup reclaims a live container before replenishing after a crash', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const previousPath = process.env.PATH;
|
||||
await writeFile(join(f.config.stateDir, 'flock'), '#!/bin/sh\nexec cat\n', {
|
||||
mode: 0o755,
|
||||
});
|
||||
process.env.PATH = `${f.config.stateDir}:${previousPath}`;
|
||||
t.after(() => {
|
||||
process.env.PATH = previousPath;
|
||||
});
|
||||
await writeFile(
|
||||
join(f.config.stateDir, 'slot-1.json'),
|
||||
JSON.stringify({
|
||||
name: 'ecs-qwen-trial-1-aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa',
|
||||
runnerId: 42,
|
||||
uncertain: false,
|
||||
}),
|
||||
);
|
||||
let oldContainer = true;
|
||||
let manager;
|
||||
const command = async (program, args, options) => {
|
||||
if (program === 'docker' && args[0] === 'ps') {
|
||||
if (args.includes('--format')) {
|
||||
// Docker quiet mode overrides --format, as on the trial host.
|
||||
return oldContainer
|
||||
? args.includes('-aq')
|
||||
? 'old-container'
|
||||
: '1'
|
||||
: '';
|
||||
}
|
||||
if (oldContainer) return 'old-container';
|
||||
}
|
||||
if (
|
||||
program === 'docker' &&
|
||||
args[0] === 'rm' &&
|
||||
args.includes('old-container')
|
||||
) {
|
||||
oldContainer = false;
|
||||
return '';
|
||||
}
|
||||
if (
|
||||
program === 'gh' &&
|
||||
args.includes('/repos/owner/repo/actions/runners/generate-jitconfig')
|
||||
) {
|
||||
assert.equal(
|
||||
oldContainer,
|
||||
false,
|
||||
'remove old container before registration',
|
||||
);
|
||||
}
|
||||
if (program === 'docker' && args[0] === 'start') manager.stop();
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
manager = createManager(f.config, { command });
|
||||
await manager.start();
|
||||
assert.equal(oldContainer, false);
|
||||
assert.equal(
|
||||
f.calls.filter((c) => c.program === 'docker' && c.args[0] === 'create')
|
||||
.length,
|
||||
1,
|
||||
);
|
||||
});
|
||||
|
||||
test('accepts immutable local image IDs and rejects mutable tags and invalid configuration', async (t) => {
|
||||
const f = await fixture(t);
|
||||
assert.doesNotThrow(() =>
|
||||
createManager({ ...f.config, image: `sha256:${'b'.repeat(64)}` }),
|
||||
);
|
||||
assert.throws(
|
||||
() => createManager({ ...f.config, image: 'image:latest' }),
|
||||
/Invalid manager configuration/,
|
||||
);
|
||||
assert.throws(
|
||||
() =>
|
||||
createManager({ ...f.config, dockerSocket: 'tcp://example.com:2375' }),
|
||||
/Invalid manager configuration/,
|
||||
);
|
||||
assert.throws(
|
||||
() => createManager({ ...f.config, cpus: Infinity }),
|
||||
/Invalid manager configuration/,
|
||||
);
|
||||
});
|
||||
|
||||
test('does not expose malformed GitHub responses in errors', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const command = (program, args, options) => {
|
||||
if (
|
||||
program === 'gh' &&
|
||||
args.some((arg) => arg.includes('generate-jitconfig'))
|
||||
) {
|
||||
return f
|
||||
.command(program, args, options)
|
||||
.then(() => 'malformed-secret-token');
|
||||
}
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
await assert.rejects(
|
||||
createManager(f.config, { command }).cycle(1),
|
||||
(error) => {
|
||||
assert.match(error.message, /Invalid JSON response/);
|
||||
assert.doesNotMatch(error.message, /malformed-secret-token/);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('HTTP 429 rejection does not leave uncertain state and the next cycle can retry', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const fakeGh = join(f.config.stateDir, 'gh');
|
||||
await writeFile(
|
||||
fakeGh,
|
||||
'#!/bin/sh\necho "gh: API rate limit exceeded (HTTP 429)" >&2\nexit 1\n',
|
||||
{ mode: 0o755 },
|
||||
);
|
||||
let rejected = false;
|
||||
const command = (program, args, options) => {
|
||||
if (
|
||||
program === 'gh' &&
|
||||
args.includes('/repos/owner/repo/actions/runners/generate-jitconfig') &&
|
||||
!rejected
|
||||
) {
|
||||
rejected = true;
|
||||
return runCommand(fakeGh, [], { env: {} });
|
||||
}
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
const manager = createManager(f.config, { command });
|
||||
await assert.rejects(manager.cycle(1), (error) => error.status === 429);
|
||||
await assert.rejects(readFile(join(f.config.stateDir, 'slot-1.json')), {
|
||||
code: 'ENOENT',
|
||||
});
|
||||
await manager.cycle(1);
|
||||
assert.equal(
|
||||
f.calls.filter((c) => c.program === 'docker' && c.args[0] === 'create')
|
||||
.length,
|
||||
1,
|
||||
);
|
||||
});
|
||||
|
||||
test('network cleanup failure prevents new runner and is reconciled on the next cycle', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const manager = createManager(f.config, { command: f.command });
|
||||
f.setFailNetworkRemove(true);
|
||||
await assert.rejects(manager.cycle(1), /network removal failed/);
|
||||
await assert.rejects(manager.cycle(1), /network removal failed/);
|
||||
assert.equal(
|
||||
f.calls.filter((c) => c.program === 'docker' && c.args[0] === 'create')
|
||||
.length,
|
||||
1,
|
||||
);
|
||||
f.setFailNetworkRemove(false);
|
||||
await manager.cycle(1);
|
||||
assert.equal(
|
||||
f.calls.filter((c) => c.program === 'docker' && c.args[0] === 'create')
|
||||
.length,
|
||||
2,
|
||||
);
|
||||
});
|
||||
|
||||
test('a lost network create response is recovered using labels', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const command = async (program, args, options) => {
|
||||
const result = await f.command(program, args, options);
|
||||
if (program === 'docker' && args[0] === 'network' && args[1] === 'create')
|
||||
throw new Error('network create response lost');
|
||||
return result;
|
||||
};
|
||||
await assert.rejects(
|
||||
createManager(f.config, { command }).cycle(1),
|
||||
/network create response lost/,
|
||||
);
|
||||
assert.equal(
|
||||
f.calls.filter(
|
||||
(c) =>
|
||||
c.program === 'docker' && c.args[0] === 'network' && c.args[1] === 'rm',
|
||||
).length,
|
||||
1,
|
||||
);
|
||||
await assert.rejects(readFile(join(f.config.stateDir, 'slot-1.json')), {
|
||||
code: 'ENOENT',
|
||||
});
|
||||
});
|
||||
|
||||
test('network deletion is verified even when docker reports success', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const command = (program, args, options) => {
|
||||
if (program === 'docker' && args[0] === 'network' && args[1] === 'rm')
|
||||
return Promise.resolve('');
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
const manager = createManager(f.config, { command });
|
||||
await assert.rejects(manager.cycle(1), /Network cleanup failed/);
|
||||
await assert.rejects(manager.cycle(1), /Network cleanup failed/);
|
||||
assert.equal(
|
||||
f.calls.filter((c) => c.program === 'docker' && c.args[0] === 'create')
|
||||
.length,
|
||||
1,
|
||||
);
|
||||
});
|
||||
|
||||
test('passes only the docker environment allowlist and the JIT config by name', async (t) => {
|
||||
const sentinels = {
|
||||
AWS_SECRET_ACCESS_KEY: 'sentinel-secret',
|
||||
SOME_DEPLOY_TOKEN: 'sentinel-token',
|
||||
};
|
||||
const previous = {};
|
||||
for (const key of Object.keys(sentinels)) {
|
||||
previous[key] = process.env[key];
|
||||
process.env[key] = sentinels[key];
|
||||
}
|
||||
t.after(() => {
|
||||
for (const key of Object.keys(sentinels)) {
|
||||
if (previous[key] === undefined) delete process.env[key];
|
||||
else process.env[key] = previous[key];
|
||||
}
|
||||
});
|
||||
const f = await fixture(t);
|
||||
await createManager(f.config, { command: f.command }).cycle(1);
|
||||
const create = f.calls.find(
|
||||
(c) => c.program === 'docker' && c.args[0] === 'create',
|
||||
);
|
||||
const allowed = [
|
||||
'DOCKER_CONFIG',
|
||||
'DOCKER_HOST',
|
||||
'HOME',
|
||||
'PATH',
|
||||
'RUNNER_JIT_CONFIG',
|
||||
'XDG_RUNTIME_DIR',
|
||||
];
|
||||
assert.deepEqual(
|
||||
Object.keys(create.options.env).filter((key) => !allowed.includes(key)),
|
||||
[],
|
||||
);
|
||||
assert.ok(!JSON.stringify(create.options.env).includes('sentinel'));
|
||||
assert.ok(!JSON.stringify(create.args).includes('sentinel'));
|
||||
assert.equal(
|
||||
create.args[create.args.indexOf('--log-driver') + 1],
|
||||
'json-file',
|
||||
);
|
||||
const at = create.args.indexOf('--env');
|
||||
assert.equal(create.args[at + 1], 'RUNNER_JIT_CONFIG');
|
||||
});
|
||||
|
||||
test('labels pool and slot on create and honours recovery filters', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const foreign = f.addForeignNetwork({
|
||||
'qwen.pool': 'other',
|
||||
'qwen.slot': '9',
|
||||
});
|
||||
const command = async (program, args, options) => {
|
||||
const result = await f.command(program, args, options);
|
||||
if (program === 'docker' && args[0] === 'network' && args[1] === 'create')
|
||||
throw new Error('network create response lost');
|
||||
return result;
|
||||
};
|
||||
await assert.rejects(
|
||||
createManager(f.config, { command }).cycle(1),
|
||||
/network create response lost/,
|
||||
);
|
||||
const created = f.calls.find(
|
||||
(c) =>
|
||||
c.program === 'docker' &&
|
||||
c.args[0] === 'network' &&
|
||||
c.args[1] === 'create',
|
||||
);
|
||||
assert.ok(created.args.includes('qwen.pool=trial'));
|
||||
assert.ok(created.args.includes('qwen.slot=1'));
|
||||
const removed = f.calls
|
||||
.filter(
|
||||
(c) =>
|
||||
c.program === 'docker' && c.args[0] === 'network' && c.args[1] === 'rm',
|
||||
)
|
||||
.map((c) => c.args[c.args.length - 1]);
|
||||
assert.deepEqual(removed, ['network-1']);
|
||||
assert.ok(f.networkIds().includes(foreign));
|
||||
});
|
||||
|
||||
test('container cleanup failure blocks replenishment even when docker reports success', async (t) => {
|
||||
const f = await fixture(t);
|
||||
f.setKeepContainerAfterRemove(true);
|
||||
const manager = createManager(f.config, { command: f.command });
|
||||
await assert.rejects(manager.cycle(1), /Container cleanup failed/);
|
||||
const state = JSON.parse(
|
||||
await readFile(join(f.config.stateDir, 'slot-1.json'), 'utf8'),
|
||||
);
|
||||
assert.ok(state.name);
|
||||
assert.equal(state.runnerId, 1);
|
||||
});
|
||||
|
||||
test('rejects corrupted slot state without deleting a runner', async (t) => {
|
||||
const validName = 'qwen-trial-1-aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa';
|
||||
const cases = [
|
||||
{
|
||||
label: 'foreign prefix',
|
||||
state: {
|
||||
name: 'qwen-other-1-aaaaaaaa-aaaa-4aaa-aaaa-aaaaaaaaaaaa',
|
||||
runnerId: 42,
|
||||
uncertain: false,
|
||||
},
|
||||
},
|
||||
{
|
||||
label: 'non-UUID suffix',
|
||||
state: {
|
||||
name: 'qwen-trial-1-not-a-uuid',
|
||||
runnerId: 42,
|
||||
uncertain: false,
|
||||
},
|
||||
},
|
||||
{
|
||||
label: 'non-boolean uncertain',
|
||||
state: { name: validName, runnerId: 42, uncertain: 'no' },
|
||||
},
|
||||
{
|
||||
label: 'zero runner id',
|
||||
state: { name: validName, runnerId: 0, uncertain: false },
|
||||
},
|
||||
];
|
||||
for (const item of cases) {
|
||||
const f = await fixture(t);
|
||||
await writeFile(
|
||||
join(f.config.stateDir, 'slot-1.json'),
|
||||
JSON.stringify(item.state),
|
||||
);
|
||||
await assert.rejects(
|
||||
createManager(f.config, { command: f.command }).cycle(1),
|
||||
/Invalid slot state/,
|
||||
item.label,
|
||||
);
|
||||
assert.ok(
|
||||
!f.calls.some((c) => c.program === 'gh' && c.args.includes('DELETE')),
|
||||
item.label,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('backs off on a failed container even after the startup window', async (t) => {
|
||||
const f = await fixture(t);
|
||||
f.config.ttlSeconds = 120;
|
||||
let now = Date.now();
|
||||
t.mock.method(Date, 'now', () => now);
|
||||
const command = (program, args, options) => {
|
||||
if (program === 'docker' && args[0] === 'wait') {
|
||||
now += 31000;
|
||||
return Promise.resolve('126\n');
|
||||
}
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
const result = await createManager(f.config, { command }).cycle(1);
|
||||
assert.equal(result.earlyExit, true);
|
||||
});
|
||||
|
||||
test('backs off before the next JIT request after a failed container', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const previousPath = process.env.PATH;
|
||||
await writeFile(join(f.config.stateDir, 'flock'), '#!/bin/sh\nexec cat\n', {
|
||||
mode: 0o755,
|
||||
});
|
||||
process.env.PATH = `${f.config.stateDir}:${previousPath}`;
|
||||
t.after(() => {
|
||||
process.env.PATH = previousPath;
|
||||
});
|
||||
const sleeps = [];
|
||||
let manager;
|
||||
let jitRequests = 0;
|
||||
const command = async (program, args, options) => {
|
||||
if (
|
||||
program === 'gh' &&
|
||||
args.includes('/repos/owner/repo/actions/runners/generate-jitconfig')
|
||||
) {
|
||||
jitRequests += 1;
|
||||
if (jitRequests > 1) manager.stop();
|
||||
}
|
||||
if (program === 'docker' && args[0] === 'wait') return '127\n';
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
manager = createManager(f.config, {
|
||||
command,
|
||||
sleep: async (ms) => {
|
||||
sleeps.push(ms);
|
||||
},
|
||||
});
|
||||
await manager.start();
|
||||
// The 1 s poll that races `docker wait` also goes through the injected sleep;
|
||||
// the replenishment backoff is the 60 s entry.
|
||||
assert.equal(sleeps.filter((ms) => ms === 60000).length, 1);
|
||||
assert.equal(sleeps.at(-1), 60000);
|
||||
assert.equal(jitRequests, 2);
|
||||
});
|
||||
|
||||
test('does not penalize a fast successful container', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const result = await createManager(f.config, { command: f.command }).cycle(1);
|
||||
assert.equal(result.earlyExit, false);
|
||||
});
|
||||
|
||||
test('backs off when docker wait cannot determine container status', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const command = (program, args, options) => {
|
||||
if (program === 'docker' && args[0] === 'wait')
|
||||
return Promise.reject(new Error('Docker disconnected'));
|
||||
return f.command(program, args, options);
|
||||
};
|
||||
const result = await createManager(f.config, { command }).cycle(1);
|
||||
assert.equal(result.earlyExit, true);
|
||||
});
|
||||
|
||||
test('starts through a symlink and reports invalid configuration', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const link = join(f.config.stateDir, 'manager.mjs');
|
||||
await symlink(new URL('./manager.mjs', import.meta.url), link);
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[link, join(f.config.stateDir, 'missing.json')],
|
||||
{
|
||||
env: { ...process.env, GH_TOKEN: 'test-only' },
|
||||
encoding: 'utf8',
|
||||
},
|
||||
);
|
||||
assert.equal(result.status, 1);
|
||||
assert.match(result.stderr, /^manager: /);
|
||||
});
|
||||
@@ -0,0 +1,18 @@
|
||||
[Unit]
|
||||
Description=Qwen Code ephemeral GitHub Actions runners
|
||||
After=docker.service network-online.target
|
||||
Requires=docker.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
EnvironmentFile=/etc/qwen-ephemeral-runner/credentials.env
|
||||
WorkingDirectory=/opt/qwen-ephemeral-runner
|
||||
ExecStart=/usr/bin/env node /opt/qwen-ephemeral-runner/manager.mjs /etc/qwen-ephemeral-runner/config.json
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
KillMode=mixed
|
||||
TimeoutStopSec=300
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -116,8 +116,7 @@ describe('complete macOS test groups', () => {
|
||||
// The extended ceilings below, except the test lane's, are scoped to the
|
||||
// self-hosted pool: `runs-on` falls back to ubuntu-latest when
|
||||
// MAINTAINER_ECS_RUNNER_DISABLED=true (and, for the association-gated lanes,
|
||||
// for fork PRs from untrusted authors — lint_and_static reads lint_runner,
|
||||
// which routes every PR to the pool), so those hosted paths keep the
|
||||
// for untrusted fork PRs with the disposable pool disabled), so hosted paths keep the
|
||||
// pre-contention ceiling. The ARM64 fleet renders
|
||||
// `["self-hosted", "linux", "arm64"]` — no `ecs-qwen` label — so the scope
|
||||
// cannot key on the x64 fleet's name. Evaluate the real timeout expression for
|
||||
@@ -391,8 +390,11 @@ it('reports the required Linux check only when the build and every shard pass',
|
||||
const gate = ci.jobs.test_ubuntu;
|
||||
expect(gate.name).toBe('Test (ubuntu-latest, Node 22.x)');
|
||||
expect(gate.needs).toEqual(['classify_pr', 'test_build', 'test']);
|
||||
// On the shards' runners, not a hosted one that can queue it for minutes.
|
||||
expect(gate['runs-on']).toBe(ci.jobs.test['runs-on']);
|
||||
// The gate remains on the original association-gated runner, while test
|
||||
// shards can opt into the disposable pool for untrusted forks.
|
||||
expect(gate['runs-on']).toBe(
|
||||
'${{ fromJSON(needs.classify_pr.outputs.ubuntu_runner || \'["ubuntu-latest"]\') }}',
|
||||
);
|
||||
expect(condOf('test_ubuntu')).toBe(condOf('test'));
|
||||
expect(gate.steps[0].run).toBe(
|
||||
'test "$BUILD_RESULT" = success && test "$SHARDS_RESULT" = success',
|
||||
@@ -862,15 +864,11 @@ describe('post-merge push lane', () => {
|
||||
).toContain('classify_pr');
|
||||
});
|
||||
|
||||
it('keeps test on the association-gated output', () => {
|
||||
// Nothing else reads test's runs-on: switching it to lint_runner would
|
||||
// hand every untrusted fork the pool while the timeout pins above stayed
|
||||
// green — timeoutMinutesOn derives each lane's output from its own
|
||||
// runs-on, so it cannot catch a lane pointed at the wrong output. Pin the
|
||||
// expression, not an equality with lint_and_static — the two are meant
|
||||
// to differ.
|
||||
it('keeps test on the opt-in disposable output', () => {
|
||||
// The classifier defaults this output to ubuntu_runner, but selects the
|
||||
// disposable pool for untrusted forks only when explicitly enabled.
|
||||
expect(String(ci.jobs.test['runs-on'])).toBe(
|
||||
'${{ fromJSON(needs.classify_pr.outputs.ubuntu_runner || \'["ubuntu-latest"]\') }}',
|
||||
'${{ fromJSON(needs.classify_pr.outputs.pr_test_runner || \'["ubuntu-latest"]\') }}',
|
||||
);
|
||||
});
|
||||
|
||||
|
||||
@@ -55,16 +55,18 @@ const ciYaml = parse(readFileSync('.github/workflows/ci.yml', 'utf8'));
|
||||
// possibly dirty workspace. Match the pool itself, not just the output
|
||||
// references that usually name it: jobs can also hard-code the shared label
|
||||
// array, and a checkout on either form inherits the same leftovers.
|
||||
// lint_and_static reads lint_runner (every PR reaches the pool), the other
|
||||
// lanes ubuntu_runner — both name the same shared workspace.
|
||||
// lint_and_static reads lint_runner (every PR reaches the pool), the test
|
||||
// lanes read pr_test_runner, the other lanes ubuntu_runner — all name the same
|
||||
// shared workspace. pr_test_runner is listed explicitly because jobs that read
|
||||
// it are also kept in scope by their own sweep step, which would otherwise hide
|
||||
// a future job that adds the pool and forgets the sweep.
|
||||
// Include retained recovery steps on hosted jobs too, so rerouting cannot
|
||||
// silently restore an untested sweep. Select by properties, not job name.
|
||||
const sharedPoolSelector = /ubuntu_runner|lint_runner|pr_test_runner|ecs-qwen/;
|
||||
const ciCleanSteps = Object.entries(ciYaml.jobs)
|
||||
.filter(
|
||||
([, job]) =>
|
||||
(/ubuntu_runner|lint_runner|ecs-qwen/.test(
|
||||
JSON.stringify(job['runs-on'] ?? ''),
|
||||
) ||
|
||||
(sharedPoolSelector.test(JSON.stringify(job['runs-on'] ?? '')) ||
|
||||
(job.steps ?? []).some(
|
||||
(s) => s.name === 'Clean stale .qwen before checkout',
|
||||
)) &&
|
||||
@@ -381,10 +383,24 @@ describe('review worktree cleanup steps', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('matches every pr_test_runner job with the shared-pool selector itself', () => {
|
||||
const poolJobs = Object.entries(ciYaml.jobs).filter(([, job]) =>
|
||||
JSON.stringify(job['runs-on'] ?? '').includes('pr_test_runner'),
|
||||
);
|
||||
expect(poolJobs.length).toBeGreaterThan(0);
|
||||
for (const [id, job] of poolJobs) {
|
||||
expect(
|
||||
sharedPoolSelector.test(JSON.stringify(job['runs-on'] ?? '')),
|
||||
`job "${id}" runs on pr_test_runner and must match the pool selector`,
|
||||
).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('keeps every shared-pool ci.yml checkout sweep pinned to paths.ts', () => {
|
||||
expect(ciCleanSteps.map(({ id }) => id)).toEqual(
|
||||
expect.arrayContaining([
|
||||
'test',
|
||||
'test_build',
|
||||
'web_shell_e2e_smoke',
|
||||
'integration_cli',
|
||||
]),
|
||||
|
||||
Reference in new issue
Block a user