From fb1b4584092557502eb92a79c2b53a35fef82bc3 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 17:49:39 -0600 Subject: [PATCH] fix(runner-e2e): drop the Rust dependency layer that filled the image builder disk (#19669) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Runner Full-Stack E2E runs Daytona cells from a signed, content-addressed runner image > - #19545 made the image build faster. One part of it added a Rust dependency layer that keeps Cargo's target and registry trees in the builder > - The first default-branch build after the merge ran out of builder disk while it exported the registry cache > - That run pushed the image tag before the failure, so the tag has no signature, and campaigns that reuse it fail `cosign verify` > - This pull request removes only the dependency layer and keeps the parts of #19545 that add no disk > - The benefit is a working default-branch image build again, a new content ID that replaces the unsigned tag, and the measured 16.5 to 11.6 minute improvement ## Linked Issues or Issue Description Refs #19545. Related #18494 (the same builder disk limit). **What happened?** The default-branch `image_only` dispatch, run 42286903290 (job 128202722275), built and pushed `ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-7154c0dfc9abb297d8d638a982326c1435dc4440c25112afacc559712d4430b3` (`sha256:2767e91c5279...`). The registry cache export then failed with `ResourceExhausted: ... no space left on device` (exit 102). `preparing build cache for export` took 182 s, against 87 s before #19545. The job stopped before `cosign sign`. The dependency layer retained `target` 852 MB and Cargo registry 333 MB (measured in run 42271236140). **Expected behavior** Default-branch image builds publish the image, write the cache, and sign the digest within the builder disk. **Steps to reproduce** Dispatch `runner-full-stack-e2e.yml` from master with `-f image_only=true` and a Daytona selector on `840838476a`. Read the `Reuse or publish immutable image` step log. ## What Changed - `docker/daytona-runner/Dockerfile`: remove the placeholder-target dependency layer. The runnerd stage is again the single build layer from before #19545, so its cache key matches the earlier default-branch cache. Restore and extend the disk comment. - Keep the content-ID check above the `COPY --from` steps and keep `COPY --link`. They add no retained data. - `tests/runner-e2e/README.md`: describe the two remaining layout rules and why there is no dependency layer. - The Dockerfile is a content-ID input, so the next build gets a new tag. No campaign reuses the unsigned tag. ## Verification - `vitest run tests/runner-e2e/daytona-image.test.ts tests/runner-e2e/workflow-image-only.test.ts`: 20/20 pass. - Branch evidence for the kept parts: job 128077687214 took 11.6 min, against 16.5 min before. The cached-layer download overlapped the compile. - After merge, an `image_only` dispatch from master must publish, write the cache and sign. I did not use a dispatch before merge, because only a default-branch build writes the cache, and the cache export is what failed. ## Risks - Runner source changes again recompile the full Rust dependency graph (about 5 min). To get that saving back, the builder needs more free disk first (for example, remove unused preinstalled toolchains in the image job, or use a larger RunsOn volume). That is a separate change. - The unsigned tag stays in GHCR. Nothing selects it after this change, because the content ID changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, GitHub CLI). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- docker/daytona-runner/Dockerfile | 28 +++++----------------------- tests/runner-e2e/README.md | 12 ++++++------ 2 files changed, 11 insertions(+), 29 deletions(-) diff --git a/docker/daytona-runner/Dockerfile b/docker/daytona-runner/Dockerfile index dd0ba18828..fcbe31de75 100644 --- a/docker/daytona-runner/Dockerfile +++ b/docker/daytona-runner/Dockerfile @@ -7,32 +7,14 @@ FROM rust:1.97-bookworm@sha256:408fe88047cef61a2087653b0c5255fa51c0f2d6d94ddedd7 WORKDIR /workspace/packages/paperclip-runner/runner COPY packages/paperclip-runner/runner/Cargo.toml packages/paperclip-runner/runner/Cargo.lock ./ -COPY packages/paperclip-runner/runner/crates/runner-core/Cargo.toml ./crates/runner-core/Cargo.toml -# Compile the locked third-party dependency graph against placeholder targets -# so this layer depends only on the manifests and lockfile. Runner source -# changes then reuse it from the registry cache and recompile one crate instead -# of the whole AWS/TLS graph (about five of the build's six and a half minutes). -# The placeholder targets' outputs and fingerprints are deleted, so the next -# step always compiles the real runner sources against the same lockfile. -RUN set -eu; \ - mkdir -p crates/runner-core/src; \ - : > crates/runner-core/src/lib.rs; \ - for target in $(grep -Eo 'src/bin/[A-Za-z0-9_-]+\.rs' crates/runner-core/Cargo.toml); do \ - mkdir -p "crates/runner-core/$(dirname "$target")"; \ - printf 'fn main() {}\n' > "crates/runner-core/$target"; \ - done; \ - cargo build --locked --release -j 2 -p paperclip-runner-core --bin paperclip-runnerd; \ - rm -rf crates/runner-core/src \ - target/release/.fingerprint/paperclip-runner-core-* \ - target/release/deps/*paperclip_runner* \ - target/release/paperclip-runnerd target/release/paperclip-runnerd.d; \ - du -sk target /usr/local/cargo/registry COPY packages/paperclip-runner/runner/crates ./crates COPY packages/paperclip-runner/protocol ../protocol COPY packages/paperclip-runner/acpx-profiles.json packages/paperclip-runner/antigravity-distributions.json ../ -# Only the finished executable leaves this stage. The final image never carries -# Cargo's target or download trees; the dependency layer above keeps them only -# in this builder stage and the trusted build cache. +# Retain only the finished executable in this layer. Keeping Cargo's target and +# download trees makes BuildKit run out of disk while exporting the final image +# and the default-branch build cache. A separate dependency-precompile layer +# (#19545) did exactly that: about 1.2 GB of retained Cargo trees exhausted the +# builder disk during the registry cache export. RUN cargo build --locked --release -j 2 -p paperclip-runner-core --bin paperclip-runnerd \ && strip /workspace/packages/paperclip-runner/runner/target/release/paperclip-runnerd \ && install -m 0755 target/release/paperclip-runnerd /usr/local/bin/paperclip-runnerd \ diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 85f1e3a042..3a02c6e83e 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1208,12 +1208,12 @@ changes can repeat the dependency install; this bounds retained builder disk at the cost of install-cache granularity. The build reports workspace and pack sizes, and final runtime and CLI checks still execute in the published image. -Build speed comes from three layout rules. The Rust third-party dependency -graph compiles in its own layer, keyed only by the Cargo manifests and lock, so -a Runner source change recompiles one crate from the default-branch registry -cache. The revision-metadata step sits above both `COPY --from` steps, so the -cached base and CLI layers download while the builder stages compile. Both -copies use `--link`, so they do not wait for that download. +Build speed comes from two layout rules. The revision-metadata step sits above +both `COPY --from` steps, so the cached base and CLI layers download while the +builder stages compile. Both copies use `--link`, so they do not wait for that +download. The Rust build keeps no dependency layer: retaining Cargo's target +and registry trees (about 1.2 GB) exhausted the builder disk during the +default-branch cache export. To publish only the image for a branch, without a paid gate, dispatch with `image_only`. Authorization, target resolution, catalog validation and the