mirror of
https://github.com/Significant-Gravitas/AutoGPT.git
synced 2026-11-09 04:49:50 -10:00
dev
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
c23b67a643 |
ci(platform): let single-container PR builds read the GHCR layer cache (#15013)
### Why / What / How
**Why:** Single-container PR builds run cold every time: 386-462 s for
the image build on recent PR runs, against 151-177 s for a warm `dev`
push. #14969 moved the dependency layers (apt, Node, RabbitMQ/Erlang,
FalkorDB, frontend) under the code layers, so a PR that only touches
code could reuse almost all of them. PR runs just never read the cache.
**What:** PR runs now read the shared `dev` GHCR layer cache
(`v4-{arch}`), the same trick as the e2e job in #14892. They still never
write it.
**How:** Stacked on #14969, which has now merged into `dev`.
- `single_container_cache.py` no longer returns early for
`pull_request`. PR runs get the three `cache-from` lines
(`backend-server-base`, `backend-server`, `single-container`) for
`v4-{arch}` and no `cache-to`. Only a push to `dev` and
`workflow_dispatch` export, as before.
- The workflow's "Authenticate to GHCR build cache" step now runs on PRs
too, with `continue-on-error: ${{ github.event_name == 'pull_request'
}}`. On a PR, a failed login (for example a fork's read-only token
without package access) gives a cold build, not a red job. On a `dev`
push or a dispatch, a failed login still fails loudly.
### Changes 🏗️
- `.github/scripts/single_container_cache.py`: drop the `pull_request`
early return.
- `.github/workflows/platform-single-container-docker.yml`: the GHCR
login step drops `if: github.event_name != 'pull_request'` and gains
PR-only `continue-on-error`.
- `autogpt_platform/single-container/tests/test_ci_registry_cache.py`:
- `test_pull_requests_never_read_or_write_registry_cache` becomes
`test_pull_requests_read_but_never_write_registry_cache`: exactly the
three `cache-from ...:v4-amd64` lines, and nothing with `cache-to`
(arm64 too).
- New `test_only_pull_requests_tolerate_a_failed_cache_login`: the login
step has no `if:`, and its `continue-on-error` is PR-only.
- `docs/platform/contributing/tests.md`: the Caching section said PRs
don't read or write registry caches. It now says they read but never
write, and that on a PR a failed login gives a cold build.
No `.env`, compose or permission changes. The job keeps `packages:
write`: PR runs never get a `cache-to` line, and fork PRs get a
read-only token anyway, the same shape as the e2e job.
### Evidence
Collected on a tester VM against `dev`
|
||
|
Nicholas TindleandClaude Opus 5.5
|
6b97e95ffd |
refactor(platform): put the single-container image's code layers last (#14969)
### Why / What / How **Why:** The single-container image was built on top of the finished backend `server` image. Everything the appliance adds (Postgres, Valkey, nginx, supervisor, Node, RabbitMQ/Erlang, FalkorDB and the frontend, about 0.8 GB) sat above AutoGPT's code. So every commit re-ran those install steps and stored all of it again. On a self-hosted box that builds branch images, a one-line backend change cost 878 MB of Docker disk. **What:** Reorder the image so dependencies sit below the code. The cloud `server` target still runs the same steps in the same order. **How:** - `backend/Dockerfile` splits the `server` stage where the code starts: - `server-base`: packages, Python, Node, Prisma engines, Chromium and the dependency venv. - `server`: `FROM server-base` plus the unchanged code steps. - The appliance builds `FROM server-base`, adds its own dependencies, then the frontend, then AutoGPT's code. - The code is copied from the `server` image with its venv dropped, so it is exactly the files the backend image ships. The appliance then re-runs `poetry install --only-root` to create the entry points. - `IMAGE_VERSION`/`VCS_REF` and the OCI labels move below the last layer, so the per-build arguments no longer invalidate cached `RUN` steps. **Side effect on the image scans:** Trivy treats every layer below the last `CMD` in the image history as a base image and skips secret scanning there. The `server` stage's `CMD ["rest"]` used to sit in the appliance's history, so Trivy never scanned the backend's dependencies or code for secrets. It now does, and two fixes follow: - One pre-existing false positive: yt-dlp's Shahid extractor embeds that site's public client key. `autogpt_platform/single-container/.trivyignore.yaml` ignores that one rule in that one file, in all four scan steps. - The report verifier required a `Type` on every Trivy result, but Trivy never writes one for secret results. It now accepts clean secret results without `Type`, and any secret finding still fails. ### Changes 🏗️ - `autogpt_platform/backend/Dockerfile`: split `server` into `server-base` and `server`. No steps changed. - `autogpt_platform/single-container/Dockerfile`: - Builds on `server-base`, with a new `backend-code` stage. - The frontend, backend code and entry points go last; the build arguments and labels go at the end. - Two cleanup commands (the `config.json` and `.next/cache` symlinks) moved next to the files they touch. No other command changed. - `autogpt_platform/single-container/docker-bake.hcl`: new `backend-server-base` target, passed in as the `autogpt-backend-base` context. - `.github/scripts/single_container_cache.py`: the new target reads the registry cache like the other two. - `autogpt_platform/single-container/.trivyignore.yaml` (new) and `.github/workflows/platform-single-container-docker.yml`: one ignore entry (rule `aws-access-key-id` in `yt_dlp/extractor/shahid.py`), used by the CI scans and the release digest scans. - `.github/scripts/verify_single_container_reports.py`: secret results need only `Target` and `Class`. - Tests: - `test_dockerfile_layers.py` (new): `server-base` holds only dependencies; the appliance copies everything the `server` target adds; the build arguments come after every layer; the bake wiring is in place. - `test_ci_scan_ignores.py` (new): every Trivy step uses the ignore file. - `test_ci_registry_cache.py`: every bake target reads the validation cache. - `test_ci_report_verifier.py`: a clean secret result passes without `Type`; a secret finding fails; a package result without `Type` still fails. **CI caching:** - The e2e GHCR cache from #14892 and #14877 is keyed by compose target (`backend-server`, `backend-migrate`). Those targets are unchanged and every existing step keeps its cache key. - The single-container registry cache rebuilds the moved appliance layers once after merge. - PR builds don't use the registry cache. **Measured** on the same builder, building `dev` and this branch, then each again with a one-line backend change: | | `dev` | this PR | |---|---|---| | New Docker disk per commit | 877.7 MB | 45.5 MB | | Rebuild after a one-line change | 72 s | 38 s | | Image size | 3.52 GB | 3.52 GB | ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Built `dev` and this branch with the single-container bake file on the same builder, then each again with a one-line backend change, and compared new data per commit and rebuild time (table above) - [x] Compared every file in the old and new single-container images (path, type, mode, owner, size, symlink target, sha256) and their config. They are identical apart from the edited `backend/Dockerfile` and timestamps in the apt/dpkg logs and the ldconfig cache, which also differ between two builds of unchanged `dev` - [x] Compared the backend `server` target before and after: the first 14 of 17 layers are reused, and every file is identical except the edited `Dockerfile` it ships - [x] Reproduced the CI secret finding with Trivy 0.70.0 and fresh caches: `dev`'s image has 0 findings, this image has 1 (`shahid.py`, in the venv layer) - [x] Checked the ignore entry: with the real entry both CI scans exit 0; a decoy entry for a different rule still exits 1 - [x] Ran the verifier on those real reports: it passes with the ignore entry and fails with `Secrets=1` without it - [x] Ran the single-container unit suite on Linux: 172 passed (163 existing plus 9 new). The layer-order and scan-ignore tests fail on deliberate regressions - [x] Ran `.github/scripts/platform-single-container-smoke.sh` against the new image: all phases passed (boot, contracts, 7 s clean stop, replacement, FalkorDB recovery, watchdog restart) - [x] Ran pre-commit on the changed files <details> <summary>Example test plan</summary> - [ ] Create from scratch and execute an agent with at least 3 blocks - [ ] Import an agent from file upload, and confirm it executes correctly - [ ] Upload agent to marketplace - [ ] Import an agent from marketplace and confirm it executes correctly - [ ] Edit an agent from monitor, and confirm it executes correctly </details> #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) <details> <summary>Examples of configuration changes</summary> - Changing ports - Adding new services that need to communicate with each other - Secrets or environment variable changes - New or infrastructure changes such as databases </details> 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]> |
||
|
|
43a7bc38d2 |
ci(platform): speed up and harden single-container validation (#14282)
### Why / What / How
The appliance build was exporting a large GHA cache that was evicted
before reuse, while smoke tests spent substantial time polling boot
state. This workflow layer uses architecture-scoped GHCR validation
caches, overlaps immutable-image scans with runtime smoke tests, and
preserves fail-closed reports for both architectures.
Stack base: #14281; next: #14284. The maintainer-closed overlap layer
and already-landed CodeQL change are excluded from the remaining stack.
Historical timings and approvals do not validate this rewritten head.
Pwuts's earlier comparison measured appliance wall time at 20m46s ->
14m15s (-31%) and runner-minutes at 37.57 -> 25.40 (-32%). These were
single samples with cache-sensitive build time, not a new benchmark on
this head. Every-dev-push validation also adds runs for pushes that
previously missed the broad source filter; that cost is intentional.
### Changes 🏗️
- Replace GHA buildx caching with GHCR `mode=max` per architecture. Dev
pushes refresh a shared validation tag; dispatches refresh a
branch-scoped tag. PRs neither read nor write these registry caches.
- Make cache export best-effort (`ignore-error=true`) without making
image builds, smoke tests, report validation, or security scans
optional.
- Build release digests cold: no writable CI-cache imports, no GHCR
login, no package-read permission in the publication job. Smoke-test and
scan the exact pushed digest before the multi-platform manifest is
published.
- Run appliance validation on every dev push, release, and dispatch;
restrict PR triggering to the explicitly listed packaging inputs.
Source/schema/migration-only PRs no longer trigger this workflow;
full-stack CI still validates the composed platform, and appliance
validation runs on the resulting dev push. This coverage tradeoff was
explicitly approved by the evaluator.
- Manual dispatch validates but does not publish Docker Hub test images.
Publication is release-only; this is an intentional externally visible
behavior change, not just a performance change.
- Add `SCHEDULER_STARTUP_EMBEDDING_BACKFILL`, defaulting to **true**.
Existing production startup behavior remains enabled; isolated
full-stack tests may opt out. Regression tests pin the actual Config
default as well as enabled/disabled scheduling behavior.
- Poll initial health every second, retain the 30-second steady-state
watchdog cadence and three-real-failure restart threshold, and permit
smoke tests to advance cadence via USR1.
- Fix the smoke race by snapshotting all failure ordinals before
stopping nginx and accepting new scheduled or forced evidence for each
ordinal. A scheduled check can no longer cause the smoke test to wait
forever for a skipped forced ordinal. Missing, stale, or wrong-ordinal
evidence still fails.
- Record boot/phase timings and preserve original failing reports and
`validated: false` status on validation failures.
### Agents and large language models used
Codex: unknown runtime model; gpt-daybreak-blue-latest contributed to
earlier review/fixes. Earlier contributions and human reviewer
attribution remain in the commit trailers.
### Checklist 📋
- [x] Changes, publication policy, path-filter tradeoff, and test plan
listed.
- [x] Linux appliance helper suite through the actual CI JUnit reporter:
162 tests, 161 passed and the one pre-existing allowed Bash 3 skip.
- [x] Scheduler service-free unit suite on the rebased head: 115 passed.
In-memory default=False mutation correctly fails the new default-pinning
regression.
- [x] Changed backend files pass formatting/import-order and
Linux-targeted type checking.
- [x] Publication self-tests and smoke-script Bash syntax pass.
- [x] Same-head GHCR cold/warm dispatch pair passed both architectures
on pre-restack `1be28e2d`: builds 10m49s -> 2m39s amd64 and 9m20s ->
2m25s arm64, 35/98 warm cached vertices each. Full evidence and phase
timings are in the September 22 review follow-up; the upper layer is
unchanged, but new-head checks below remain required.
- [x] Predecessor `77d09c54ce32f6424adf54426abd3c20b8e4130c`: [amd64 and
arm64 builds, full smoke tests, scans, and
reports](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35781920373)
passed; publication skipped. [Backend
CI](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35781920605)
passed all 18 jobs, including the [original duplicate suite
rerun](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35781919712/attempts/2).
- [x] Current `628a444a79080fe77ef07fb852a70b19219a6beb`: [appliance
CI](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35793258786)
passed both architecture builds, complete smoke tests, scans and report
gates; publication skipped. [Backend
CI](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35793258641)
passed all 18 jobs. Range-diff confirms this PR's own layer is
unchanged.
- [x] Inherited full-stack [35793258577 attempt
2](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35793258577/attempts/2)
passed all 28 tests with zero failures/skips/flaky results on unchanged
`628a444a79`; aggregate status is green. First warm attempt failed a
Copilot session-creation HTTP 504 (27/28 passed), despite successful SQL
restore and all eight fresh-key credential checks. The rerun missed the
evicted cache, so it is a cold success, not proof that the first warm
failure was fixed. Root #14281 independently passed same-head warm E2E.
Failed report/trace retained; no failure waiver or timeout/retry
changes, and no concrete link to this layer was found in Daybreak's
review.
- [ ] Fresh review, resolved threads, and two quiet post-CI polls.
- [x] `.env.default` and Compose are compatible: production default
remains enabled without an environment override.
Local limitation: the broad Windows backend format/type command reaches
six errors in unchanged files (including POSIX-only APIs and missing
optional Playwright fixtures). No unrelated fixes or type suppressions
are included. Hosted Linux checks now pass as linked above. Earlier
comments about this layer's backend tree being unchanged predate the
scheduler setting and are no longer the validation argument for this
head.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes release-only Docker Hub publishing, CI gating, and scheduler
startup embedding behavior; mistakes could block releases or alter
production cold-start search indexing.
>
> **Overview**
> **Hardens and speeds up single-container CI** by switching validation
builds to architecture-scoped GHCR registry caches, overlapping image
build, smoke tests, and Trivy scans, and requiring machine-readable
JUnit/Trivy reports with a verifier before the job succeeds.
>
> **Publication policy changes:** Docker Hub publishing is
**release-only**; manual `workflow_dispatch` no longer publishes
(validation-only). Release digest builds stay cold (no writable CI
cache). The publish script drops dispatch/`PUBLISH_REQUESTED` paths and
tightens release-tag and digest-enumeration error handling.
>
> **Smoke and watchdog:** The smoke script polls health every second,
logs boot/phase timings (sanitized milestones), syncs on concurrent
scans, hardens cleanup and negative probes, and exercises watchdog
restart via USR1 with race-safe failure ordinals. `watchdog.sh` adds
forced checks on USR1 and a 1s initial health poll while keeping the 30s
steady-state cadence and three-failure limit.
>
> **Scheduler:** Adds `scheduler_startup_embedding_backfill` (default
**true**); when false, the first embedding coverage job waits six hours
instead of running at startup.
>
> **Triggers:** Every `dev` push runs full appliance validation; PRs
only run when listed packaging paths change.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|