perf(runner-e2e): cut the Daytona image build critical path (#19545)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner Full-Stack E2E workflow tests the Runner in Daytona
sandboxes from a signed, content-addressed image
> - Every change to an audited Runner input needs a new image before a
Daytona cell can test it
> - The image job took 10 to 16.5 minutes today, and a restart chaos
campaign waits on it after each runner-side fix
> - Measurement shows two serial waits: a full Rust dependency compile
(about 5 of 6.6 minutes) and a 3.5 to 4 minute download of cached base
layers that starts only after the builder stages finish
> - There is also no way to build only the image: every dispatch reaches
a paid gate that the operator must reject
> - This pull request removes both serial waits and adds an `image_only`
dispatch input
> - The benefit is a shorter fix-to-Daytona-test loop, with every image
security check unchanged

## Linked Issues or Issue Description

**What happened?**

The `Publish verified Daytona image` job is on the critical path of the
control-plane restart chaos campaign. Builds today took 16.5 min (run
42240434011), 15 min (42253920727), 10.5 min (42259118276) and 16 min
(42265637650). Run 42264611573 failed: its RunsOn runner lost
communication with GitHub 7.6 minutes into the build step, during the
Rust compile. GitHub has no log for that job.

**Expected behavior**

A Runner source change should not recompile the locked third-party Rust
graph. Cached image layers should download while the builder stages
compile. An operator should be able to publish only the image, without a
paid gate.

**Steps to reproduce**

Read the BuildKit timeline in the logs of job 127970125495 (run
42240434011). See the stage table below.

## What Changed

- `docker/daytona-runner/Dockerfile`: compile the locked Rust dependency
graph in its own layer against placeholder targets. The layer depends
only on `Cargo.toml`, `Cargo.lock` and the crate manifest. The step
deletes the placeholder outputs and fingerprints, so the next step
always compiles the real Runner sources. The final stage still receives
only the stripped executable.
- `docker/daytona-runner/Dockerfile`: move the content-ID /
source-revision check above the two `COPY --from` steps. It does not
depend on either builder stage, so BuildKit now downloads the cached
base and CLI layers in parallel with the compile. Both copies use `COPY
--link`.
- `.github/workflows/runner-full-stack-e2e.yml`: add the boolean
`image_only` dispatch input. `authorize`, `target_lock`, `catalog` and
`daytona_image` run unchanged. Every other campaign job gets `&&
inputs.image_only != true`. `authorize` rejects `image_only` together
with TestOps inputs.
- `tests/runner-e2e/workflow-image-only.test.ts` (new): pin the job
classification, the top-level guard on each campaign job, the guard on
every protected environment, and the TestOps rejection.
- `tests/runner-e2e/README.md`: document the layer layout and the
`image_only` command.

## Verification

Measured on the same RunsOn `m7i-flex.xlarge` (4 vCPU, 16 GiB, native
x64) runner class. "Before" is master job 127970125495 (run
42240434011). "After (cold)" is this branch, job 128077687214 (run
42271236140), with no dependency layer in the registry cache yet. Its
paid gate was rejected.

| Stage | Before | After (cold dependency cache) |
|---|---|---|
| Whole image job | 16.5 min | **11.6 min** |
| Rust dependency graph | 311 s (inside one 399 s step) | 395 s (own
layer; reused from cache after a master build) |
| `paperclip-runner-core` compile | ~87 s | 65 s |
| Cached base and CLI layer download | 236 s, **after** cargo finished |
257 s, **in parallel** with cargo |
| Copy runnerd + provider pack | 20 s + 50 s | 9 s (`--link`) |
| Provider pack stage | 143 s (parallel) | 152 s (parallel) |
| Export + push | 82 s + 10 s | 81 s + 11 s |
| Dependency layer size | n/a | `target` 852 MB + registry 333 MB
(builder stage only) |

With a warm dependency cache, a Runner source change compiles only the
65 s crate step. The critical path then becomes the ~4 min cached-layer
download or the provider pack. The expected job time is about 7 to 8
min. I measure this after merge with an `image_only` dispatch (master
first, to write the cache, then a Rust-only branch change).

Disk: the instance starts with 38.76 GiB free. The dependency layer adds
about 1.2 GB uncompressed to builder storage. The final image does not
change.


- `vitest run tests/runner-e2e/workflow-image-only.test.ts
tests/runner-e2e/workflow-security.test.ts
tests/runner-e2e/daytona-image.test.ts`: the new test and all image
tests pass. Three tests in `workflow-security.test.ts` (`gates every
provider-secret job with stable actor IDs`, `keeps provider credentials
inside explicitly gated paid workflows`, `isolates private archive trust
from workflows that execute target-branch code`) fail on `origin/master`
without this change, because of `chat-catalog-testops-case.yml`. This
pull request does not change them.
- `node --test tests/runner-e2e/product-regression-testops.test.mjs`:
15/15 pass.
- No local Docker build. The Dockerfile is measured with a
branch-targeted CI dispatch whose paid gate was rejected.

## Risks

- The dependency layer keeps the Rust `target` and registry trees in a
builder-stage layer, and default-branch builds export it to the registry
cache. This adds builder disk and cache upload when `Cargo.lock`
changes. The build prints the size (`du -sk target
/usr/local/cargo/registry`).
- The dependency cache helps only after a default-branch build writes
it. Branch builds read it but never write it. This is the existing cache
policy.
- `image_only` changes the workflow, so it works only after merge to
master.
- No check is removed. Lockfile verification, cosign sign and verify,
the anonymous pull check, the provider-pack manifest match, the
default-branch-only cache write and the actor allowlist are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, GitHub CLI). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-06 14:15:16 -06:00
1 parent 5a5bfe689f
commit 840838476a
4 files changed
+146 -18

No files matched your search

+22 -9
View File
@@ -60,6 +60,10 @@ on:
description: "Allure TestOps service input"
type: string
required: false
image_only:
description: "Only build, sign and verify the Daytona image for the selected cells; skip every campaign job and paid gate"
type: boolean
default: false
permissions:
contents: read
@@ -139,8 +143,14 @@ jobs:
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
EVENT_NAME: ${{ github.event_name }}
EVAL_CADENCE: ${{ inputs.cadence || 'manual' }}
IMAGE_ONLY: ${{ inputs.image_only == true }}
TESTOPS_REQUESTED: ${{ inputs.ALLURE_JOB_RUN_ID != '' || inputs.testops_case != '' }}
run: |
set -euo pipefail
if [ "$IMAGE_ONLY" = true ] && [ "$TESTOPS_REQUESTED" = true ]; then
echo "image_only builds publish no results; it cannot bind a TestOps selection." >&2
exit 1
fi
if [ -z "$TARGET_BRANCH" ] || [[ "$TARGET_BRANCH" == refs/* ]]; then
echo "target_branch must name a branch in this repository without a refs/ prefix." >&2
exit 1
@@ -692,7 +702,7 @@ jobs:
name: Build reusable runner campaign artifacts
needs: [authorize, target_lock, catalog]
if: >-
!cancelled() && needs.catalog.result == 'success' &&
!cancelled() && needs.catalog.result == 'success' && inputs.image_only != true &&
(github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true'))
# Compile native binaries on the same reviewed image used to execute them,
# avoiding libc/architecture drift between GitHub-hosted and AWS lanes.
@@ -823,7 +833,7 @@ jobs:
[authorize, target_lock, catalog, daytona_image, build_runner_artifacts]
if: >-
!cancelled() && needs.catalog.result == 'success' && needs.daytona_image.result == 'success' &&
needs.build_runner_artifacts.result == 'success' &&
needs.build_runner_artifacts.result == 'success' && inputs.image_only != true &&
(github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true'))
runs-on: ubuntu-latest
timeout-minutes: 15
@@ -963,7 +973,8 @@ jobs:
if: >-
!cancelled() && needs.catalog.result == 'success' && needs.daytona_image.result == 'success' &&
needs.build_runner_artifacts.result == 'success' && needs.build_remote_provider_pack.result == 'success' &&
needs.catalog.outputs.operations_cadence == '' && needs.authorize.outputs.evidence_retention == 'actions'
needs.catalog.outputs.operations_cadence == '' && needs.authorize.outputs.evidence_retention == 'actions' &&
inputs.image_only != true
needs:
[
authorize,
@@ -1458,7 +1469,8 @@ jobs:
if: >-
!cancelled() && needs.catalog.result == 'success' && needs.daytona_image.result == 'success' &&
needs.build_runner_artifacts.result == 'success' && needs.build_remote_provider_pack.result == 'success' &&
needs.catalog.outputs.operations_cadence != '' && needs.authorize.outputs.target_sha == github.sha && github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
needs.catalog.outputs.operations_cadence != '' && needs.authorize.outputs.target_sha == github.sha && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) &&
inputs.image_only != true
needs: [authorize, target_lock, catalog, daytona_image, build_runner_artifacts, build_remote_provider_pack]
runs-on: ${{ needs.authorize.outputs.test_runner }}
timeout-minutes: ${{ matrix.timeoutMinutes }}
@@ -1478,7 +1490,8 @@ jobs:
if: >-
!cancelled() && needs.catalog.result == 'success' && needs.daytona_image.result == 'success' &&
needs.build_runner_artifacts.result == 'success' && needs.build_remote_provider_pack.result == 'success' &&
needs.catalog.outputs.operations_cadence == '' && needs.authorize.outputs.evidence_retention == 'private' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
needs.catalog.outputs.operations_cadence == '' && needs.authorize.outputs.evidence_retention == 'private' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) &&
inputs.image_only != true
needs: [authorize, target_lock, catalog, daytona_image, build_runner_artifacts, build_remote_provider_pack]
runs-on: ${{ needs.authorize.outputs.test_runner }}
timeout-minutes: ${{ matrix.timeoutMinutes }}
@@ -1495,7 +1508,7 @@ jobs:
report:
name: Merge and enforce campaign result
if: always() && !cancelled() && needs.catalog.result == 'success' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
if: always() && !cancelled() && needs.catalog.result == 'success' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) && inputs.image_only != true
needs: [authorize, catalog, daytona_image, test, test_recurring, test_manual_private]
outputs:
history_source_ready: ${{ steps.history_source_ready.outputs.ready }}
@@ -2036,7 +2049,7 @@ jobs:
publish_testops:
name: Publish bound Product regression results to TestOps
needs: [authorize, testops_selection, catalog, report]
if: always() && needs.testops_selection.outputs.job_run_started == 'true'
if: always() && needs.testops_selection.outputs.job_run_started == 'true' && inputs.image_only != true
runs-on: ubuntu-latest
timeout-minutes: 10
environment: runner-e2e-testops-reporting
@@ -2099,7 +2112,7 @@ jobs:
publish_history:
name: Publish S3 history and Pages bundle with declared screenshots
needs: [authorize, catalog, report]
if: always() && needs.catalog.result == 'success' && needs.catalog.outputs.operations_cadence == '' && needs.authorize.outputs.evidence_retention == 'actions' && needs.report.outputs.history_source_ready == 'true'
if: always() && needs.catalog.result == 'success' && needs.catalog.outputs.operations_cadence == '' && needs.authorize.outputs.evidence_retention == 'actions' && needs.report.outputs.history_source_ready == 'true' && inputs.image_only != true
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
@@ -2177,7 +2190,7 @@ jobs:
pages:
name: Publish latest dashboard with declared screenshots
needs: [report, publish_history]
if: always() && needs.report.outputs.history_source_ready == 'true' && needs.publish_history.result == 'success' && vars.RUNNER_FULL_STACK_E2E_PUBLISH_PAGES == 'true'
if: always() && needs.report.outputs.history_source_ready == 'true' && needs.publish_history.result == 'success' && vars.RUNNER_FULL_STACK_E2E_PUBLISH_PAGES == 'true' && inputs.image_only != true
runs-on: ubuntu-latest
permissions:
pages: write
+35 -9
View File
@@ -7,11 +7,32 @@ FROM rust:1.97-bookworm@sha256:408fe88047cef61a2087653b0c5255fa51c0f2d6d94ddedd7
WORKDIR /workspace/packages/paperclip-runner/runner
COPY packages/paperclip-runner/runner/Cargo.toml packages/paperclip-runner/runner/Cargo.lock ./
COPY packages/paperclip-runner/runner/crates/runner-core/Cargo.toml ./crates/runner-core/Cargo.toml
# Compile the locked third-party dependency graph against placeholder targets
# so this layer depends only on the manifests and lockfile. Runner source
# changes then reuse it from the registry cache and recompile one crate instead
# of the whole AWS/TLS graph (about five of the build's six and a half minutes).
# The placeholder targets' outputs and fingerprints are deleted, so the next
# step always compiles the real runner sources against the same lockfile.
RUN set -eu; \
mkdir -p crates/runner-core/src; \
: > crates/runner-core/src/lib.rs; \
for target in $(grep -Eo 'src/bin/[A-Za-z0-9_-]+\.rs' crates/runner-core/Cargo.toml); do \
mkdir -p "crates/runner-core/$(dirname "$target")"; \
printf 'fn main() {}\n' > "crates/runner-core/$target"; \
done; \
cargo build --locked --release -j 2 -p paperclip-runner-core --bin paperclip-runnerd; \
rm -rf crates/runner-core/src \
target/release/.fingerprint/paperclip-runner-core-* \
target/release/deps/*paperclip_runner* \
target/release/paperclip-runnerd target/release/paperclip-runnerd.d; \
du -sk target /usr/local/cargo/registry
COPY packages/paperclip-runner/runner/crates ./crates
COPY packages/paperclip-runner/protocol ../protocol
COPY packages/paperclip-runner/acpx-profiles.json packages/paperclip-runner/antigravity-distributions.json ../
# Retain only the finished executable in this layer. Keeping Cargo's target and
# download trees makes BuildKit run out of disk while exporting the final image.
# Only the finished executable leaves this stage. The final image never carries
# Cargo's target or download trees; the dependency layer above keeps them only
# in this builder stage and the trusted build cache.
RUN cargo build --locked --release -j 2 -p paperclip-runner-core --bin paperclip-runnerd \
&& strip /workspace/packages/paperclip-runner/runner/target/release/paperclip-runnerd \
&& install -m 0755 target/release/paperclip-runnerd /usr/local/bin/paperclip-runnerd \
@@ -122,21 +143,26 @@ COPY packages/paperclip-runner/scripts/provision-grok.mjs /tmp/grok-provision/sc
COPY packages/paperclip-runner/src/providers/grok/platforms.json /tmp/grok-provision/src/providers/grok/platforms.json
RUN node /tmp/grok-provision/scripts/provision-grok.mjs /opt/paperclip/providers/grok/1.0.13/grok && rm -rf /tmp/grok-provision
COPY --from=runnerd-build /usr/local/bin/paperclip-runnerd /usr/local/bin/paperclip-runnerd
COPY --chown=0:0 --from=provider-pack-build /provider-pack /opt/paperclip-runner/provider-pack
RUN set -eu; for cli in codex claude opencode; do \
printf '#!/bin/sh\nexec /opt/paperclip-runner/provider-pack/node_modules/.bin/%s "$@"\n' "$cli" > "/usr/local/bin/$cli"; \
chmod 755 "/usr/local/bin/$cli"; \
done
# Keep revision-dependent metadata below the stable agent CLI installation
# layers. A source-only image miss can then reuse those expensive layers from
# the trusted registry cache.
# This step must stay ABOVE the COPY --from steps. It does not wait for either
# builder stage, so BuildKit downloads the cached base and CLI layers (about
# four minutes) while runnerd and the provider pack compile, instead of after.
ARG PAPERCLIP_RUNNER_CONTENT_ID
ARG PAPERCLIP_RUNNER_SOURCE_REVISION
RUN test -n "${PAPERCLIP_RUNNER_CONTENT_ID}" \
&& test -n "${PAPERCLIP_RUNNER_SOURCE_REVISION}"
# --link builds each copied tree as an independent layer, so producing it does
# not wait for the cached layers below to finish downloading.
COPY --link --from=runnerd-build /usr/local/bin/paperclip-runnerd /usr/local/bin/paperclip-runnerd
COPY --link --chown=0:0 --from=provider-pack-build /provider-pack /opt/paperclip-runner/provider-pack
RUN set -eu; for cli in codex claude opencode; do \
printf '#!/bin/sh\nexec /opt/paperclip-runner/provider-pack/node_modules/.bin/%s "$@"\n' "$cli" > "/usr/local/bin/$cli"; \
chmod 755 "/usr/local/bin/$cli"; \
done
RUN set -eu; \
printf '%s\n' 'export PATH=/opt/paperclip-runner/provider-pack/node_modules/.bin:$PATH' \
> /etc/profile.d/01-paperclip-runner-provider-pack.sh; \
+21
View File
@@ -1208,6 +1208,27 @@ changes can repeat the dependency install; this bounds retained builder disk
at the cost of install-cache granularity. The build reports workspace and pack
sizes, and final runtime and CLI checks still execute in the published image.
Build speed comes from three layout rules. The Rust third-party dependency
graph compiles in its own layer, keyed only by the Cargo manifests and lock, so
a Runner source change recompiles one crate from the default-branch registry
cache. The revision-metadata step sits above both `COPY --from` steps, so the
cached base and CLI layers download while the builder stages compile. Both
copies use `--link`, so they do not wait for that download.
To publish only the image for a branch, without a paid gate, dispatch with
`image_only`. Authorization, target resolution, catalog validation and the
signed, verified image publication run unchanged; every campaign job is
skipped. Keep a selector that includes a Daytona cell:
```bash
gh workflow run runner-full-stack-e2e.yml \
--ref master \
-f target_branch=fix/example \
-f all=false \
-f id=core-compatibility.runner-codex.daytona.message-marker \
-f image_only=true
```
The ephemeral Daytona fixture requests a 60-minute wall-clock lifetime at creation.
The provider destroys each sandbox at that expiry in any state. This bound includes
room for each declared attempt and the acquisition budget. It supplements the
@@ -0,0 +1,68 @@
import { readFile } from "node:fs/promises";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { parse as parseYaml } from "yaml";
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
describe("runner full-stack image_only dispatch", () => {
it("lets image_only dispatches build the Daytona image without reaching any campaign job", async () => {
const workflowText = await readFile(
path.join(repositoryRoot, ".github/workflows/runner-full-stack-e2e.yml"),
"utf8",
);
const workflow = parseYaml(workflowText) as {
on: { workflow_dispatch: { inputs: Record<string, { type?: string; default?: unknown }> } };
jobs: Record<string, { if?: string; environment?: unknown; steps?: Array<{ name?: string; env?: Record<string, string>; run?: string }> }>;
};
expect(workflow.on.workflow_dispatch.inputs.image_only).toMatchObject({ type: "boolean", default: false });
// Authorization, target resolution, catalog validation and the signed,
// verified image publication run exactly as in a campaign.
const imageJobs = ["authorize", "target_lock", "catalog", "daytona_image"];
// TestOps binding is reachable only through inputs that authorize rejects
// for image_only dispatches.
const rejectedByAuthorize = ["testops_selection"];
const campaignJobs = [
"build_runner_artifacts",
"build_remote_provider_pack",
"test",
"test_recurring",
"test_manual_private",
"report",
"publish_testops",
"publish_history",
"pages",
];
// A new job must be classified here before it can run in an image-only dispatch.
expect(Object.keys(workflow.jobs).sort()).toEqual(
[...imageJobs, ...rejectedByAuthorize, ...campaignJobs].sort(),
);
for (const name of [...imageJobs, ...rejectedByAuthorize]) {
expect(String(workflow.jobs[name]!.if ?? ""), name).not.toContain("image_only");
}
for (const name of campaignJobs) {
const condition = String(workflow.jobs[name]!.if ?? "");
expect(condition, `${name} must skip image_only dispatches`).toMatch(/&& inputs\.image_only != true(?:\s|$)/u);
// The guard must be a top-level conjunct: no unparenthesized || may bypass it.
let topLevel = condition;
while (/\([^()]*\)/u.test(topLevel)) topLevel = topLevel.replace(/\([^()]*\)/gu, "_");
expect(topLevel, name).not.toContain("||");
}
// Every protected paid environment is below the image-only guard.
for (const [name, job] of Object.entries(workflow.jobs)) {
if (job.environment !== undefined && name !== "testops_selection") {
expect(campaignJobs, `${name} has a protected environment`).toContain(name);
}
}
const targetStep = workflow.jobs.authorize!.steps!.find(
step => step.name === "Resolve requested repository branch to an immutable commit",
)!;
expect(targetStep.env).toMatchObject({
IMAGE_ONLY: "${{ inputs.image_only == true }}",
TESTOPS_REQUESTED: "${{ inputs.ALLURE_JOB_RUN_ID != '' || inputs.testops_case != '' }}",
});
expect(targetStep.run).toContain('if [ "$IMAGE_ONLY" = true ] && [ "$TESTOPS_REQUESTED" = true ]; then');
});
});