alexmi256/rea-agent-docker
1
0 Fork 0
mirror of https://github.com/alexmi256/rea-agent-docker.git synced 2026-11-07 09:45:59 -10:00
Code Issues Packages Projects Releases Wiki Activity

add a version-drift checker and the update procedure

Browse Source 
scripts/check-updates.py answers 'is this repo behind?' in ~5s, and encodes the
distinction that actually matters here: REA validates engines and helper tools by
digest or version banner, so upstream 'latest' is frequently the WRONG target.
The checker compares every pin against two references - the newest thing upstream
ships, and what the pinned REA release ADMITS - and reads those admissions live
from morluto/rea@rea-agents-<REA_VERSION> (GhidraInstallationPolicy,
LinuxHopper SUPPORTED_RELEASES/HOPPER_SHA256, CapabilityInventory remediation
strings, runtimeVersion). Bumping REA therefore re-derives the expectations
instead of relying on documentation memory.

33 rows: rea + Node (checked against REA's engines range), engines, every
REA-located helper tool, our own unpinned tools, the base image and the action
majors. Statuses separate the two failure modes - behind-REA (a breakage: REA
refuses the banner) from outdated (optional: upstream newer, not admitted) - and
'docker compose pull'-style checks are not confused with engine validation.
Modes: table, --quiet, --json, --markdown; exit 1 when actionable; --only/--skip
by key or group. Every row carries the URL it came from, and a failed lookup is
reported as unknown-with-reason, never as up to date.

Findings on day one: 6 actionable (Node 24.21->24.22, unicorn, uv, frida,
supergateway patches; Wakaru MISSING - REA 6.5.0 admits ^1.13.0 for
recover_javascript_sources and we do not install it), plus apktool 3.x and
ilspycmd 11.x as explicitly optional (REA's apktool requirement is a floor of
2.7.0, and REA only needs a runnable ilspycmd).

Wiring: make check-updates / check-updates-json; a weekly 'updates' workflow
(Mondays 06:17 UTC + dispatch) that needs no Docker or package permissions, puts
the report in the run summary, uploads the JSON, and fails when something is
actionable; validate now compiles the script, proves --help works offline and
requires scripts/*.py to be executable.

docs/UPDATES.md is the procedure: which reference governs which component, the
general loop, per-engine procedures (Ghidra's exact build, Hopper's digest gate),
the helper-tool table with locator variables and install method, the release
string that must agree across five files, how to add a tool REA advertises but we
do not ship (worked example: Wakaru), accepted gaps, how to extend the checker,
and the traps worth remembering.
This commit is contained in:
Hermes Agent committed 2026-10-15 20:30:28 +00:00
1 parent e23e8bbcc8
commit 02f9c3adaa
6 files changed
+1376 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/build.yml
+10 -1
View File
@@ -128,6 +128,15 @@ jobs:
done
echo "shell scripts parse"
- name: Python syntax (the update checker)
run: |
set -e
# No network here: --help must work offline, and the module must
# import cleanly, or the weekly `updates` workflow fails at 6am.
python3 -m py_compile scripts/check-updates.py
scripts/check-updates.py --help > /dev/null
echo "check-updates.py imports and starts"
- name: YAML syntax
run: |
python3 -c "import yaml" 2>/dev/null || sudo apt-get install -y -qq python3-yaml
@@ -164,7 +173,7 @@ jobs:
set -e
# Entry points are invoked by path (Makefile, docs, each other), so
# they must be executable in git - not just readable by `bash <path>`.
for f in scripts/*.sh docker/*.sh docker/verify/*.sh; do
for f in scripts/*.sh scripts/*.py docker/*.sh docker/verify/*.sh; do
[ -x "$f" ] || { echo "not executable in git: $f"; exit 1; }
done
echo "every script is executable"
.github/workflows/updates.yml
+55
View File
@@ -0,0 +1,55 @@
name: updates
# Weekly check that every pinned version is still current, plus manual dispatch.
#
# Deliberately separate from build.yml: this job needs no Docker, no build and no
# package permissions, and a red run means "a version moved upstream", not "the
# image is broken". The report lands in the run summary (Actions -> updates ->
# the run -> Summary) and the job fails when something is actionable, so the
# failure itself is the notification.
#
# A false red is cheap to read (it names the component and the two candidates);
# a stale pin is not, because REA refuses engine builds and helper-tool banners
# that it has not verified.
on:
schedule:
- cron: "17 6 * * 1" # Mondays, 06:17 UTC
workflow_dispatch:
permissions:
contents: read
concurrency:
group: updates
cancel-in-progress: false
jobs:
pins:
name: are the pins still current?
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- name: Check pins against upstream and against what REA admits
run: |
set -euo pipefail
# Two passes over the same data: one for the machine, one for the
# human-readable summary. The script is stdlib-only and takes ~5s.
scripts/check-updates.py --json --no-fail > "$RUNNER_TEMP/updates.json"
scripts/check-updates.py --markdown --no-fail > "$RUNNER_TEMP/updates.md"
cat "$RUNNER_TEMP/updates.md" >> "$GITHUB_STEP_SUMMARY"
actionable="$(python3 -c "import json;print(json.load(open('$RUNNER_TEMP/updates.json'))['summary']['actionable'])")"
if [ "$actionable" != "0" ]; then
echo "::warning::$actionable pinned version(s) are behind - see docs/UPDATES.md for the procedure"
exit 1
fi
echo "all pins current"
- name: Upload the JSON report
if: always()
uses: actions/upload-artifact@v7
with:
name: update-check
path: ${{ runner.temp }}/updates.json
if-no-files-found: warn
Makefile
+8
View File
@@ -130,6 +130,14 @@ test-host: ## run the harness in a throwaway container against ./work
-v $(PWD)/work:/work -v $(PWD)/config:/etc/rea:ro $(IMAGE) \
/opt/rea/verify/verify-all.sh
.PHONY: check-updates
check-updates: ## are the pins still current? (upstream + what REA admits)
scripts/check-updates.py $(ARGS)
.PHONY: check-updates-json
check-updates-json: ## the same check as JSON (for scripts)
scripts/check-updates.py --json --no-fail
.PHONY: sizes
sizes: ## report the size of every engine image
@for e in all ghidra hopper ida; do \
README.md
+24 -2
View File
@@ -221,15 +221,37 @@ docker/verify/ the in-image, engine-aware verification harness
scripts/build.sh build one engine image (path-mapped for DooD)
scripts/build-all.sh build all four, then report sizes
scripts/verify.sh run the harness in a throwaway container
scripts/check-updates.py are the pins still current? (upstream vs what REA admits)
scripts/image-name.sh resolve an engine to its image (one source of truth)
scripts/setup-ida-mcp.sh wire IDA into REA (6 modes)
scripts/hermes-mcp.yaml paste-ready Hermes MCP block
docs/ images, architecture, BYO licences, IDA editions,
multi-arch, MCP, troubleshooting, verification
.github/workflows/ CI: build + verify per engine, full on demand
multi-arch, MCP, troubleshooting, updates, verification
.github/workflows/ CI: validate + publish per engine; weekly pin check
```
---
## Keeping up to date
```bash
make check-updates # ~5 s: every pin, what upstream ships, what REA admits
```
One row per component, exit code 1 when something needs action. The distinction
it encodes is the whole game: **REA validates engines and helper tools by digest
or version banner, so upstream "latest" is frequently the wrong target** — it
refuses a Hopper build whose launcher digest it has not verified and a Ghidra
release it was not built against. The checker reads REA's admitted versions live
from the pinned release, so bumping REA re-derives them.
A weekly CI job runs the same check and fails when something moves; the report
lands in the run summary. Per-component procedures, the release-string files that
must agree, the Wakaru gap, and the traps worth remembering are in
**[docs/UPDATES.md](docs/UPDATES.md)**.
---
## Verification
Everything below is run *inside the built image*, against fixtures it builds
docs/UPDATES.md
+352
View File
@@ -0,0 +1,352 @@
# Keeping rea-agent-docker up to date
Everything in this image is pinned on purpose, so "is it out of date?" is a real
question with a mechanical answer:
```bash
make check-updates # or: scripts/check-updates.py
```
It reads every pin out of `Dockerfile`, asks each upstream what it currently
ships, and prints one row per component in about five seconds. Exit code is 1
when something needs action, so it works as a gate as well as a report.
| mode | use |
| --- | --- |
| `scripts/check-updates.py` | human table, grouped, with the reason on every row |
| `--quiet` | only the actionable rows |
| `--json` | stable shape: `summary`, `rea_admissions`, `rows[]` (each with `source`) |
| `--markdown` | a table for a CI step summary |
| `--only rea,hopper` / `--skip actions` | narrow by key or group (run `--json` to list keys) |
| `--no-fail` | report, never exit non-zero |
A weekly job runs it and fails when something moves — see
[Automation](#automation). The full procedure for each kind of bump is below.
---
## The rule that matters: upstream latest is not the target
Two references exist for nearly every row, and they usually disagree:
1. **what upstream ships** — the newest release anywhere, and
2. **what REA admits** — the builds and version banners the pinned REA release
was verified against.
REA *refuses* what it does not admit, rather than guessing:
* Hopper is validated by the **SHA-256 of its launcher binary**; a build outside
the admitted set fails engine validation.
* Ghidra is validated against **one exact release**; a newer one reports
`config_drift` while looking correctly installed.
* Helper tools are validated by **version banner** (`REA_WABT_BIN_DIRECTORY`,
`REA_BINWALK_COMMAND`, `REA_UNBLOB_COMMAND`, `REA_TSHARK_COMMAND`,
`REA_SIGROK_COMMAND`, `REA_JADX_MCP_JAR`, `REA_PWNTOOLS_PYTHON`, …). A newer
tool can be rejected; an older one is a capability gap.
So for those rows, **chasing the newest release is the wrong move and falling
behind REA's admitted version is the breakage**. The checker enforces exactly
that distinction, and it reads the admitted versions *live out of REA's source at
the pinned release*
(`raw.githubusercontent.com/morluto/rea/rea-agents-<REA_VERSION>/…`), so bumping
REA re-derives the expectations instead of relying on this document's memory.
| which reference governs | components |
| --- | --- |
| **REA admits (exact)** | Ghidra, Hopper package + launcher digest, JDK major, binwalk, unblob, WABT, TShark, sigrok-cli, jadx-headless-mcp, pwntools, Wakaru |
| **REA floor only** | apktool (≥ 2.7.0 — we hold 2.x deliberately: 3.x changes the output layout) |
| **no REA constraint** | ilspycmd (only has to be runnable), mitmproxy, pyelftools, unicorn, uv, supergateway, frida, playwright, Chrome, .NET, adb |
| **npm/GitHub latest** | rea-agents itself, Node.js (within REA's `engines` range), GitHub Actions majors |
| **nothing to pin** | `ubuntu:24.04`, apt-installed tshark/capinfos/sigrok-cli/adb/openjdk/clang-cross — a rebuild picks up the current packages |
---
## The general loop
1. `make check-updates` — see what moved and why it matters.
2. Bump the pin in the `Dockerfile` (`ARG …`), and in `.env.example` if it is
mirrored there.
3. `make check-updates` again — the row should go green.
4. Build the *affected* engine image and run the harness. Do not skip this: the
pins are exactly the inputs the checks assert on.
```bash
make build-ghidra # or build / build-hopper / build-ida
scripts/verify.sh --engine ghidra # full harness for that image
```
5. Update docs that *name* versions (README, `docs/IMAGES.md`,
`docs/VERIFICATION.md`) and the size table if it changed (`make sizes`).
6. Commit with the upstream URL and the check you ran; push. CI re-validates and
republishes under the new tag.
Bumping more than one component at a time is how you lose track of which change
broke a check — the harness names the phase, so keep the commit small and the
message specific.
---
## A. REA itself (`rea-agents`)
The only bump that touches everything, because REA re-defines what the image
must contain.
**What to check before bumping:**
* the release's own `engines.node` (the checker prints it) — our `NODE_VERSION`
must satisfy it, or REA refuses to run at all;
* the release's admitted engine builds (**the checker's `REA admits` column now
changes to match the new release** — that is the point of reading it live);
* whether the release adds or renames tool families. REA's docs and `main` run
ahead of the published package, so a family in the repo may not exist in the
package you install, and vice versa. The harness reports absent families as
explicit *skips*, which is how you tell "not in this release" from "broken";
* the bundled skill (`skills/reverse-engineer-anything` inside the package) —
its `tool_count` and version identify which release it describes.
**Where the release string lives** (all of it must agree; CI `validate` fails if
it does not):
| file | what |
| --- | --- |
| `Dockerfile` | `ARG REA_VERSION` |
| `.env.example` | `REA_VERSION`, `REA_IMAGE`, `REA_GHIDRA_IMAGE`, `REA_HOPPER_IMAGE`, `REA_IDA_IMAGE` (tag suffix `<version>-linux-amd64`) |
| `docker-compose.yml` | the four `image:` fallbacks |
| `compose.http.yml` | the sidecar's image fallback |
| docs | README quick start + table, `docs/IMAGES.md`, `docs/VERIFICATION.md` |
**Procedure:**
```bash
make check-updates --only rea # confirm the new release, read engines.node
# bump REA_VERSION in Dockerfile + .env.example, and the tag suffix in the four
# image names (both compose files + .env.example)
scripts/image-name.sh ghidra # tags now resolve with the new suffix
make build-all # all four, shared layers cached
scripts/verify-all-engines.sh # 4 × full harness
```
Then push: GHCR gets `ghcr.io/alexmi256/rea-agent[-ghidra|-hopper|-ida]:<new>-linux-amd64`
plus a per-image `latest`. Old tags stay, so pinned users are unaffected —
`latest` is per image and always the newest build of *that* image, which is why
anything reproducible should pin the version tag (see `docs/IMAGES.md`).
**Watch for on a REA bump:**
* a **new admitted engine version** → the engine pins must move too (section B);
* a **new helper requirement** → a row appears as `MISSING` (section D);
* **renamed/removed tools** → the harness's tool inventory checks change; update
`docker/verify/*` assertions rather than loosening them.
---
## B. Engines
### Ghidra (exact version)
REA's adapter is verified against one build (`SUPPORTED_GHIDRA_VERSION` in
`src/ghidra/GhidraInstallationPolicy.ts`), and a different one reports
`config_drift` while everything still looks installed.
```bash
make check-updates --only ghidra # compares pin, upstream and REA's version
# bump GHIDRA_VERSION *and* GHIDRA_ZIP together (the zip name carries both the
# version and the build date), leave GHIDRA_SHA256 empty to verify against the
# release, then:
make build --no-cache # the engine layer must rebuild
docker compose exec rea-agent rea doctor --provider ghidra --json
```
The JDK matters too: `SUPPORTED_GHIDRA_JAVA_MAJOR` (21 today) is the major Ghidra
declares bounds for. We install `openjdk-21-jdk-headless` from apt, so a Java
*patch* needs nothing but a rebuild.
### Hopper (a digest, not a version)
The gate is the **launcher's SHA-256**, and the package is pinned by **SHA-1 and
byte length**:
```
Dockerfile: HOPPER_DEB_SHA1 / HOPPER_DEB_BYTES / HOPPER_DEB_URL (empty = derived)
REA source: src/application/LinuxHopper.ts → SUPPORTED_RELEASES (deb sha1 + bytes)
→ SUPPORTED_HOPPER_SHA256 (launcher digests)
```
The checker prints how many launcher digests REA admits and whether our package
matches `SUPPORTED_RELEASES` exactly. **A licensed Hopper build can legitimately
report drift while working** — that is a vendor-licence difference, not a broken
install. Never download a licensed build: Hopper ships as the vendor's **demo
package** only (see `docs/BYO-LICENSES.md`).
### IDA / JEB (never installed)
Commercial engines are bring-your-own registrations (`REA_IDA_MCP_CONFIG`,
`REA_JEB_MCP_URL`) and are never downloaded, by design — see
`docs/BYO-LICENSES.md` and `docs/ida-variants.md`. The checker tracks
`mrexodia/ida-pro-mcp` as information only, because that is the upstream that
users install *outside* the image. Headless IDA needs `idalib`, so IDA Free
cannot do it.
---
## C. REA-admitted helper tools
One `ARG` each, one locator variable each. Bump the `ARG`; the tool must report
the version REA admits.
| tool | pin | installed by | REA locator | harness phase |
| --- | --- | --- | --- | --- |
| binwalk | `BINWALK_VERSION` | `cargo install --locked --version` (crates.io) | `REA_BINWALK_COMMAND` | `static` |
| unblob | `UNBLOB_VERSION` | `pip install "unblob==…"` into `/opt/venvs/unblob` | `REA_UNBLOB_COMMAND` | `static` |
| WABT | `WABT_VERSION` | GitHub release tarball + sha256, `/opt/wabt` | `REA_WABT_BIN_DIRECTORY` | `static` |
| jadx-headless-mcp | `JADX_MCP_VERSION` | GitHub release jar, `/opt/jadx` | `REA_JADX_MCP_JAR` | `static` |
| apktool | `APKTOOL_VERSION` | GitHub release jar + wrapper script | `REA_APKTOOL_COMMAND` | `static` |
| pwntools / pyelftools / unicorn | `PWTOOLS_VERSION` / `PYELFTOOLS_VERSION` / `UNICORN_VERSION` | one venv, `/opt/venvs/pwntools` | `REA_PWNTOOLS_PYTHON` | `static` |
| TShark / capinfos | — (apt) | `apt-get install tshark` | `REA_TSHARK_COMMAND`, `REA_CAPINFOS_COMMAND` | `capture` |
| sigrok-cli | — (apt) | `apt-get install sigrok-cli` | `REA_SIGROK_COMMAND` | `capture` |
| adb | — (apt) | `apt-get install adb` | `REA_ADB_PATH` | `tools` |
| ilspycmd | `ILSPYCMD_VERSION` | `dotnet tool install --tool-path` | `REA_ILSPY_CMD_PATH` | `static` |
Notes:
* **`REA_WABT_BIN_DIRECTORY` points at a directory, not a binary** — the whole
WABT release has to be extracted, so a WABT bump changes the tarball name too
(`wabt-<ver>-linux-x64.tar.gz`, verified against its published `.sha256`).
* **Apktool is deliberately held at 2.x.** REA's floor is 2.7.0 and 3.x changes
the output layout; the checker reports 3.x as optional rather than required.
* **ilspycmd has no admitted version** — REA only requires a runnable binary, so
the checker reports a newer NuGet release as information. Every `.NET` fixture
in the harness exercises it, so bumping is cheap but not free.
* **apt tools are not pinned.** A rebuild picks up Ubuntu's security updates;
that is the intended update path for tshark, sigrok-cli, adb and the JDK.
## D. Adding a tool REA advertises but we do not ship
This is the `MISSING` status, and it is the most valuable row the checker
produces: REA advertises a capability whose tool is absent, so the tool reports
`provider_missing` at call time.
**Worked example — Wakaru (as of REA 6.5.0).** REA's own remediation string says:
> On Linux x64, provide an absolute `REA_WAKARU_COMMAND` for Wakaru ^1.13.0
> (verified with 1.14.0) and util-linux prlimit;
Procedure:
1. read the remediation text for the exact requirement — never guess it, it is
in `src/application/CapabilityInventory.ts` of the pinned release;
2. install it the same way as the other Node-based tools (`npm install --global`
in the build stage), then locate it with the `ENV` REA expects
(`REA_WAKARU_COMMAND=/opt/…/wakaru`);
3. add a harness assertion to the right phase (this one belongs in `static`,
next to the other JavaScript checks);
4. `make build && scripts/verify.sh` — the skip should turn into a pass;
5. document it in `docs/ARCHITECTURE.md` (what the image ships) and here.
If you deliberately do **not** want it, add it to the accepted-gaps list below
with the reason, so the row is expected rather than surprising.
### Accepted gaps
| gap | why |
| --- | --- |
| Wakaru (`recover_javascript_sources`) | not installed yet — REA admits ^1.13.0; the procedure above is the fix |
| IDA / JEB | licensable vendor software: bring-your-own by design, never downloaded |
| Hopper licensed builds | same; only the vendor demo package is installed |
| Apple composite providers (plist etc.) | provider-scoped, not a host capability |
---
## E. Our own tools (no REA gate)
`uv`, `supergateway`, `frida`, `mitmproxy`, `playwright`, Chrome, the .NET SDK
and the cross toolchains are ours, not REA's — bump them freely, then run the
phase that exercises them:
| tool | pin | proves it |
| --- | --- | --- |
| frida | `FRIDA_VERSION` (npm `frida`, `--allow-scripts=frida`) | `clients` phase (`getLocalDevice()`) |
| supergateway | `SUPERGATEWAY_VERSION` | `mcp` phase (HTTP bridge handshake) |
| uv | `UV_VERSION` | build stage (`uv --version`) |
| mitmproxy | `MITMPROXY_VERSION` | `capture` phase |
| playwright / Chrome | unpinned | `clients` phase (Chromium launch) |
| .NET SDK | apt `dotnet-sdk-8.0` | `static` phase (`ilspycmd`) |
A frida bump deserves a second look at the harness rather than a glance at the
version: frida's install scripts are skipped by npm unless
`--allow-scripts=frida` is passed, and the version is referenced from
`rea-agents/node_modules` by symlink.
## F. Base image and CI plumbing
**`ubuntu:24.04`** is deliberately unpinned: a rebuild picks up the current point
release. The checker prints the tag's current digest and date so you can see
whether a rebuild would move anything. Ubuntu 24.04 is not arbitary — REA's
Hopper installer supports macOS 12+/Ubuntu 24.04+/Fedora 41+/Arch, and 24.04
resolves every pinned helper from apt (`docs/MULTIARCH.md`).
**GitHub Actions majors** are checked too, because a stale major is what triggers
the Node-runtime deprecation warning on runners:
| action | current | note |
| --- | --- | --- |
| `actions/checkout` | `@v7` | Node 24 from v5 |
| `actions/upload-artifact` | `@v7` | Node 24 only from **v6** |
| `docker/login-action` | `@v4` | Node 24 only from **v4** |
The trap is that these boundaries differ per action — pinning everything to the
same major is exactly how you end up with one warning left.
---
## Automation
`.github/workflows/updates.yml` runs the checker **weekly (Mondays 06:17 UTC)**
and on manual dispatch. It needs no Docker, no build and no package permissions,
and it is separate from the image build on purpose: a red `updates` run means
"a version moved", not "the image is broken".
* the markdown report lands in the run summary
(Actions → *updates* → the run → **Summary**);
* the JSON report is uploaded as the `update-check` artifact;
* the job **fails when anything is actionable** — the failure is the reminder,
and it names the component plus both candidates.
Locally the same check is `make check-updates` (or `make check-updates-json`).
CI's `validate` job additionally syntax-checks the script and proves it starts
without network, so a broken checker can never be the reason the weekly job is
quiet.
### Extending the checker
Everything lives in `scripts/check-updates.py`:
* **a new upstream version to watch** → add a row in `check_tools`
(pypi/github/npm/nuget dicts) or `check_misc`, and pass through `tool_row`
when REA has an admission for it;
* **a new REA admission pattern** → add a regex to the `patterns` dict in
`scrape_rea` (keys match the checker's own component keys) and point the row at
it via `admissions.tool("<key>")`;
* **a new REA source file** → add it to `REA_SOURCES`;
* **a new local consistency rule** → add it to the coherence row in `check_rea`
(that is where "the release string must agree everywhere" lives).
Keep two habits: every row carries the URL it came from, and a failed lookup is
reported as `unknown` + reason — never as "up to date". A check that silently
passes when it cannot see upstream is worse than no check.
---
## Traps worth remembering
* **`config_drift` from `rea doctor` is not an install failure.** It usually
means the engine build is newer than the admitted one.
* **A Hopper version number proves nothing** — the gate is the launcher digest.
* **"Newer" and "admitted" can point in opposite directions.** jadx-headless-mcp
is the live example: we pin 0.7.1 because REA 6.5.0 admits 0.7.1, while npm
still only publishes 0.7.0. The checker reports that as fine, and it is.
* **GHCR tags are per image.** There is no repo-wide `latest`; bumps add tags and
leave old ones in place.
* **The image size table drifts** with dependency changes — refresh it with
`make sizes` when you touch the pins that carry weight (Ghidra ~900 MB, Hopper
~92 MB deltas are documented in `docs/IMAGES.md`).
* **Evidence beats intent**: after any bump, the harness has to run and its
skips have to be read.
scripts/check-updates.py
Executable
+927
View File
@@ -0,0 +1,927 @@
#!/usr/bin/env python3
# =============================================================================
# check-updates.py - is this repo behind its upstreams?
#
# scripts/check-updates.py # table for humans
# scripts/check-updates.py --json # machine-readable
# scripts/check-updates.py --markdown # for a CI step summary
# scripts/check-updates.py --only rea,hopper,wakaru
# scripts/check-updates.py --no-fail # report, never exit non-zero
#
# What it does: reads every pinned version out of the Dockerfile, asks each
# upstream what it currently ships, and compares that against TWO references:
#
# 1. the newest thing upstream ships, and
# 2. what the pinned REA release actually ADMITS.
#
# (2) is the one that matters for the engine and helper tools. REA 6.5.0 admits
# Ghidra 12.1.4, Hopper 6.4.2-by-digest, Binwalk 3.1.x, Unblob 26.6.x, WABT
# 1.0.42, sigrok-cli 0.7.2, TShark 4.x, jadx-headless-mcp 0.7.1, pwntools
# 4.15.0, Wakaru ^1.13.0 - and it refuses builds that do not match (Hopper by
# launcher SHA-256, the rest by version banner). So "newest upstream" is
# frequently the WRONG target for those rows: chasing apktool 3.x or ilspycmd 11
# is a judgement call, while falling behind REA's admitted WABT is a breakage.
#
# Those expectations are read live from REA's own source at the pinned release
# tag, so bumping REA_VERSION automatically re-derives them - which is what
# makes docs/UPDATES.md a procedure rather than a memory test.
#
# Every row carries the URL it was derived from, and a failed lookup is reported
# as unknown-with-reason rather than being silently treated as up to date.
# No credentials are used or printed; stdlib only (uses `gh` if present, for the
# higher authenticated GitHub rate limit).
# =============================================================================
from __future__ import annotations
import argparse
import json
import os
import re
import shutil
import subprocess
import sys
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
PROJECT_DIR = Path(__file__).resolve().parent.parent
DOCKERFILE = PROJECT_DIR / "Dockerfile"
ENV_EXAMPLE = PROJECT_DIR / ".env.example"
WORKFLOW_DIR = PROJECT_DIR / ".github" / "workflows"
REA_REPO = "morluto/rea"
REA_RAW = "https://raw.githubusercontent.com/{repo}/rea-agents-{version}/{path}"
# REA source files that carry the admitted-version tables.
REA_SOURCES = {
"node": "src/domain/runtimeVersion.ts",
"ghidra": "src/ghidra/GhidraInstallationPolicy.ts",
"hopper": "src/application/LinuxHopper.ts",
"tools": "src/application/CapabilityInventory.ts",
}
# statuses that mean "do something"
ACTIONABLE = {"outdated", "behind-rea", "missing", "mismatch"}
STATUS_LABEL = {
"ok": "ok",
"outdated": "OUTDATED",
"behind-rea": "BEHIND-REA",
"ahead": "ahead",
"missing": "MISSING",
"mismatch": "MISMATCH",
"info": "info",
"unknown": "unknown",
}
# --------------------------------------------------------------------------- #
# tiny version helpers
# --------------------------------------------------------------------------- #
def vparse(value: str | None) -> tuple[int, ...] | None:
"""'v1.2.3', 'Ghidra_12.1.4_build', '8.2.0.7535' -> comparable tuple."""
if not value:
return None
match = re.search(r"(\d+(?:\.\d+)+)", value)
if not match:
return None
return tuple(int(part) for part in match.group(1).split("."))
def vpad(a: tuple[int, ...], b: tuple[int, ...]) -> tuple[tuple[int, ...], tuple[int, ...]]:
width = max(len(a), len(b))
return a + (0,) * (width - len(a)), b + (0,) * (width - len(b))
def compare(ours: str | None, theirs: str | None) -> int | None:
"""-1 ours older, 0 equal, 1 ours newer, None incomparable."""
a, b = vparse(ours), vparse(theirs)
if a is None or b is None:
if ours and theirs and ours.strip() == theirs.strip():
return 0
return None
a, b = vpad(a, b)
return (a > b) - (a < b)
def major(value: str | None) -> int | None:
parsed = vparse(value)
return parsed[0] if parsed else None
# --------------------------------------------------------------------------- #
# fetching
# --------------------------------------------------------------------------- #
class Fetcher:
def __init__(self, timeout: float, jobs: int) -> None:
self.timeout = timeout
self.jobs = jobs
self._use_gh = shutil.which("gh") is not None
self._cache: dict[str, object] = {}
def json(self, url: str) -> object:
if url in self._cache:
cached = self._cache[url]
if isinstance(cached, Exception):
raise cached
return cached
try:
result = self._json_uncached(url)
except Exception as exc: # noqa: BLE001 - reported per row
self._cache[url] = exc
raise
self._cache[url] = result
return result
def _json_uncached(self, url: str) -> object:
if self._use_gh and "api.github.com/repos/" in url:
path = url.split("api.github.com/", 1)[1]
proc = subprocess.run(
["gh", "api", path],
capture_output=True,
text=True,
timeout=self.timeout + 10,
check=False,
)
if proc.returncode == 0 and proc.stdout.strip():
return json.loads(proc.stdout)
raise RuntimeError(f"gh api {path}: {proc.stderr.strip()[:120] or 'no output'}")
request = urllib.request.Request(
url,
headers={
"User-Agent": "rea-agent-docker-update-check",
"Accept": "application/vnd.github+json, application/json",
},
)
with urllib.request.urlopen(request, timeout=self.timeout) as response:
return json.loads(response.read().decode("utf-8", "replace"))
def text(self, url: str) -> str:
if url in self._cache:
cached = self._cache[url]
if isinstance(cached, Exception):
raise cached
return str(cached)
try:
request = urllib.request.Request(
url, headers={"User-Agent": "rea-agent-docker-update-check"}
)
with urllib.request.urlopen(request, timeout=self.timeout) as response:
value = response.read().decode("utf-8", "replace")
except Exception as exc: # noqa: BLE001
self._cache[url] = exc
raise
self._cache[url] = value
return value
# --------------------------------------------------------------------------- #
# pins
# --------------------------------------------------------------------------- #
def read_pins() -> tuple[dict[str, str], dict[str, str]]:
def parse(path: Path, prefix: str) -> dict[str, str]:
pins: dict[str, str] = {}
if not path.exists():
return pins
pattern = re.compile(rf"^{prefix}([A-Z0-9_]+)=(.*)$")
for line in path.read_text().splitlines():
match = pattern.match(line.strip())
if match:
pins[match.group(1)] = match.group(2).strip().strip('"')
return pins
return parse(DOCKERFILE, r"ARG "), parse(ENV_EXAMPLE, r"")
# --------------------------------------------------------------------------- #
# rows
# --------------------------------------------------------------------------- #
@dataclass
class Row:
key: str
label: str
group: str
status: str
pinned: str | None = None
upstream: str | None = None
admits: str | None = None
source: str = ""
note: str = ""
hint: str = ""
def as_dict(self) -> dict[str, object]:
return {
"key": self.key,
"label": self.label,
"group": self.group,
"status": self.status,
"actionable": self.status in ACTIONABLE,
"pinned": self.pinned,
"upstream": self.upstream,
"admits": self.admits,
"source": self.source,
"note": self.note,
"hint": self.hint,
}
@dataclass
class ReaAdmissions:
"""What the pinned REA release says it accepts."""
version: str
node_range: str | None = None
ghidra: str | None = None
ghidra_java_major: str | None = None
hopper_deb: dict[str, str] = field(default_factory=dict)
hopper_hashes: list[str] = field(default_factory=list)
tools: dict[str, dict[str, str]] = field(default_factory=dict)
errors: dict[str, str] = field(default_factory=dict)
def tool(self, name: str) -> dict[str, str] | None:
return self.tools.get(name)
def scrape_rea(fetcher: Fetcher, version: str) -> ReaAdmissions:
"""Read REA's admitted-version tables out of its source at the pinned tag."""
admissions = ReaAdmissions(version=version)
def raw(name: str) -> str | None:
url = REA_RAW.format(repo=REA_REPO, version=version, path=REA_SOURCES[name])
try:
return fetcher.text(url)
except Exception as exc: # noqa: BLE001
admissions.errors[name] = f"{url}: {exc}"
return None
node = raw("node")
if node:
match = re.search(r'SUPPORTED_NODE_VERSION_RANGE\s*=\s*"([^"]+)"', node)
if match:
admissions.node_range = match.group(1)
ghidra = raw("ghidra")
if ghidra:
match = re.search(r'SUPPORTED_GHIDRA_VERSION\s*=\s*"([^"]+)"', ghidra)
if match:
admissions.ghidra = match.group(1)
match = re.search(r"SUPPORTED_GHIDRA_JAVA_MAJOR\s*=\s*(\d+)", ghidra)
if match:
admissions.ghidra_java_major = match.group(1)
hopper = raw("hopper")
if hopper:
deb = re.search(
r"deb:\s*\{\s*filename:\s*`[^`]*?(Hopper-[\w.\-]+)`,"
r"\s*file_length:\s*\"(\d+)\",\s*file_hash:\s*\"(\w+)\"",
hopper,
re.S,
)
if deb:
admissions.hopper_deb = {
"filename": deb.group(1),
"bytes": deb.group(2),
"sha1": deb.group(3),
}
hashes = re.search(r"SUPPORTED_HOPPER_SHA256\s*=\s*new Set\(\[(.*?)\]\)", hopper, re.S)
if hashes:
admissions.hopper_hashes = re.findall(r'"([0-9a-f]{64})"', hashes.group(1))
tools = raw("tools")
if tools:
# Ordered: first pattern that matches wins, so keep specific ones first.
patterns = {
"pwntools": r"pwntools (\d+(?:\.\d+)*)",
"wabt": r"WABT (\d+(?:\.\d+)*)",
"binwalk": r"Binwalk (\d+(?:\.\d+)*)x[^\n]*?verified with (\d+(?:\.\d+)*)",
"unblob": r"Unblob (\d+(?:\.\d+)*)x[^\n]*?verified with (\d+(?:\.\d+)*)",
"tshark": r"TShark (\d+)\.x",
"sigrok-cli": r"sigrok-cli (\d+(?:\.\d+)*)",
"jadx-headless-mcp": r"jadx-headless-mcp (\d+(?:\.\d+)*)",
"apktool": r"Apktool (\d+(?:\.\d+)*) or newer",
"wakaru": r"Wakaru \^?(\d+(?:\.\d+)*) \(verified with (\d+(?:\.\d+)*)\)",
"ilspycmd": r"ilspycmd (\d+(?:\.\d+)*)",
}
for name, pattern in patterns.items():
match = re.search(pattern, tools)
if not match:
continue
groups = [g for g in match.groups() if g]
admissions.tools[name] = {
"required": groups[0],
"verified": groups[1] if len(groups) > 1 else groups[0],
}
return admissions
# --------------------------------------------------------------------------- #
# individual checks
# --------------------------------------------------------------------------- #
def npm_latest(fetcher: Fetcher, package: str) -> dict[str, object]:
return fetcher.json(f"https://registry.npmjs.org/{package}/latest") # type: ignore[return-value]
def pypi_latest(fetcher: Fetcher, package: str) -> str:
data = fetcher.json(f"https://pypi.org/pypi/{package}/json")
return data["info"]["version"] # type: ignore[index]
def github_latest(fetcher: Fetcher, repo: str) -> str:
data = fetcher.json(f"https://api.github.com/repos/{repo}/releases/latest")
return data["tag_name"] # type: ignore[index]
def launchpad_noble(fetcher: Fetcher, binary: str) -> str:
series = "https%3A%2F%2Fapi.launchpad.net%2F1.0%2Fubuntu%2Fnoble%2Famd64"
url = (
"https://api.launchpad.net/1.0/ubuntu/+archive/primary"
f"?ws.op=getPublishedBinaries&binary_name={binary}&exact_match=true"
f"&distro_arch_series={series}&status=Published"
)
data = fetcher.json(url)
entries = data["entries"] # type: ignore[index]
if not entries:
raise RuntimeError("no published noble binary")
return sorted({e["binary_package_version"] for e in entries})[-1]
def unknowable(row: Row, exc: Exception) -> Row:
row.status = "unknown"
reason = str(exc).strip() or exc.__class__.__name__
row.note = f"lookup failed: {reason[:160]}"
return row
def check_rea(fetcher: Fetcher, pins: dict[str, str], admissions: ReaAdmissions) -> list[Row]:
rows: list[Row] = []
pinned = pins.get("REA_VERSION", "")
row = Row("rea", "rea-agents (npm)", "REA core", "ok", pinned=pinned,
source="https://registry.npmjs.org/rea-agents/latest")
try:
data = npm_latest(fetcher, "rea-agents")
latest = str(data["version"])
row.upstream = latest
order = compare(pinned, latest)
if order is None:
row.status = "unknown"
row.note = "could not compare versions"
elif order < 0:
row.status = "outdated"
row.hint = "bump REA_VERSION, then rebuild and run the harness (see docs/UPDATES.md)"
node_engines = str((data.get("engines") or {}).get("node", ""))
if node_engines:
row.note = f"engines.node: {node_engines}"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
rows.append(row)
# the release string has to agree across every file that carries it
_, env_pins = read_pins()
env_text = ENV_EXAMPLE.read_text() if ENV_EXAMPLE.exists() else ""
compose_path = PROJECT_DIR / "docker-compose.yml"
compose = compose_path.read_text() if compose_path.exists() else ""
http_compose = PROJECT_DIR / "compose.http.yml"
http_text = http_compose.read_text() if http_compose.exists() else ""
expected_tag = f"{pinned}-linux-amd64"
probe = Row("rea-coherence", "release string in repo files", "REA core", "ok", pinned=pinned)
problems = []
if env_pins.get("REA_VERSION") and env_pins["REA_VERSION"] != pinned:
problems.append(f".env.example REA_VERSION={env_pins['REA_VERSION']}")
for path, text in ((Path(".env.example"), env_text),
(Path("docker-compose.yml"), compose),
(Path("compose.http.yml"), http_text)):
if text and expected_tag not in text:
problems.append(f"{path} does not use :{expected_tag}")
if compose and pinned not in env_text:
problems.append(f".env.example does not mention {pinned}")
if problems:
probe.status = "mismatch"
probe.note = "; ".join(problems)
probe.hint = "make the release string agree everywhere (CI validate checks this too)"
rows.append(probe)
return rows
def check_node(fetcher: Fetcher, pins: dict[str, str], admissions: ReaAdmissions) -> list[Row]:
pinned = pins.get("NODE_VERSION", "")
row = Row("node", "Node.js", "REA core", "ok", pinned=pinned,
admits=admissions.node_range,
source="https://nodejs.org/dist/index.json")
try:
index = fetcher.json("https://nodejs.org/dist/index.json")
line_major = str(major(pinned) or "")
same_line = [d for d in index if str(d["version"]).startswith(f"v{line_major}.")]
lts = [d for d in index if d.get("lts")]
if same_line:
newest = str(same_line[0]["version"])
row.upstream = newest.lstrip("v")
order = compare(pinned, newest)
if order is not None and order < 0:
row.status = "outdated"
row.hint = "bump NODE_VERSION; keep within REA's engines range"
if lts:
row.note = f"latest LTS line: {lts[0]['version']} ({lts[0].get('lts')})"
if str(lts[0]["version"]).lstrip("v") != row.upstream:
row.note += f"; {line_major}.x is not the newest LTS line"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
# REA's range is an admission gate: our runtime must satisfy it
if admissions.node_range and vparse(pinned):
clauses = [c.strip() for c in admissions.node_range.split("||")]
satisfied = False
for clause in clauses:
match = re.match(r"\^?(\d+)\.(\d+)\.(\d+)$", clause)
if match:
line, minor, patch = (int(g) for g in match.groups())
if major(pinned) == line and (vparse(pinned) or (0,)) >= (line, minor, patch):
satisfied = True
elif clause.startswith(">="):
floor = vparse(clause)
if floor and (vparse(pinned) or (0,)) >= floor:
satisfied = True
if not satisfied:
row.status = "behind-rea"
row.note = f"Node {pinned} does not satisfy REA {admissions.version} range {admissions.node_range}"
row.hint = "REA refuses unsupported runtimes; move to an admitted line"
return [row]
def check_ghidra(fetcher: Fetcher, pins: dict[str, str], admissions: ReaAdmissions) -> list[Row]:
rows: list[Row] = []
pinned = pins.get("GHIDRA_VERSION", "")
row = Row("ghidra", "Ghidra", "Engines", "ok", pinned=pinned, admits=admissions.ghidra,
source="https://api.github.com/repos/NationalSecurityAgency/ghidra/releases/latest")
try:
latest = github_latest(fetcher, "NationalSecurityAgency/ghidra")
row.upstream = latest
order = compare(pinned, latest)
if order is not None and order < 0:
row.status = "outdated"
row.hint = "bump GHIDRA_VERSION and GHIDRA_ZIP together (GhidraInstallationPolicy.ts admits one build)"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
if admissions.ghidra:
order = compare(pinned, admissions.ghidra)
if order is not None and order != 0:
row.status = "behind-rea"
row.note = f"REA {admissions.version} admits Ghidra {admissions.ghidra}"
row.hint = "install exactly the admitted build; a newer Ghidra reports config_drift"
rows.append(row)
zip_pin = pins.get("GHIDRA_ZIP", "")
consistency = Row("ghidra-zip", "Ghidra zip filename", "Engines", "ok", pinned=zip_pin)
if pinned and pinned not in zip_pin:
consistency.status = "mismatch"
consistency.note = f"GHIDRA_ZIP '{zip_pin}' does not name version {pinned}"
consistency.hint = "bump both together"
rows.append(consistency)
return rows
def check_hopper(fetcher: Fetcher, pins: dict[str, str], admissions: ReaAdmissions) -> list[Row]:
rows: list[Row] = []
sha1 = pins.get("HOPPER_DEB_SHA1", "")
expected = admissions.hopper_deb or {}
label_version = ""
if expected.get("filename"):
match = re.search(r"Hopper-([\w.]+)-Linux", expected["filename"])
label_version = match.group(1) if match else ""
row = Row("hopper", f"Hopper{(' ' + label_version) if label_version else ''}", "Engines", "ok",
pinned=sha1[:12] + "…" if sha1 else None,
admits=(expected.get("sha1", "")[:12] + "…") if expected.get("sha1") else None,
source=REA_RAW.format(repo=REA_REPO, version=admissions.version, path=REA_SOURCES["hopper"]))
if expected.get("sha1"):
if expected["sha1"] == sha1:
row.note = f"matches the REA-admitted demo package ({expected.get('bytes')} bytes)"
else:
row.status = "behind-rea"
row.note = f"REA admits {expected.get('filename')} sha1 {expected['sha1']}"
row.hint = "re-pin HOPPER_DEB_SHA1/HOPPER_DEB_BYTES to the admitted package"
elif admissions.errors.get("hopper"):
unknowable(row, RuntimeError(admissions.errors["hopper"]))
if pins.get("HOPPER_DEB_BYTES") and expected.get("bytes") and pins["HOPPER_DEB_BYTES"] != expected["bytes"]:
row.status = "behind-rea"
row.note = (row.note + "; " if row.note else "") + (
f"HOPPER_DEB_BYTES={pins['HOPPER_DEB_BYTES']} but REA expects {expected['bytes']}"
)
rows.append(row)
launchers = Row("hopper-launcher", "Hopper admitted launcher hashes", "Engines", "info",
admits=f"{len(admissions.hopper_hashes)} admitted launcher digest(s)")
launchers.note = "a Hopper build outside this set fails engine validation"
if admissions.errors.get("hopper"):
unknowable(launchers, RuntimeError(admissions.errors["hopper"]))
rows.append(launchers)
return rows
def check_java(fetcher: Fetcher, pins: dict[str, str], admissions: ReaAdmissions) -> list[Row]:
row = Row("java", "openjdk (Ghidra's JDK)", "Engines", "info", pinned="openjdk-21",
admits=(f"major {admissions.ghidra_java_major}" if admissions.ghidra_java_major else None),
source="Launchpad noble/amd64")
try:
row.upstream = launchpad_noble(fetcher, "openjdk-21-jdk-headless")
if admissions.ghidra_java_major and "21" not in row.pinned:
row.status = "behind-rea"
row.hint = "Ghidra's bound JDK major changed"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
return [row]
def tool_row(
key: str,
label: str,
pinned: str,
admitted: dict[str, str] | None,
upstream_url: str,
upstream: str | None,
fetcher_error: Exception | None,
*,
floor_semantics: bool = False,
pinned_present: bool = True,
) -> Row:
"""Shared shape for the REA-admitted helper tools."""
row = Row(key, label, "REA-admitted tools", "ok",
pinned=pinned if pinned_present else None,
upstream=upstream,
admits=(admitted or {}).get("required"),
source=upstream_url)
if admitted and admitted.get("verified") and admitted["verified"] != admitted["required"]:
row.note = f"REA {admitted['required']} (verified with {admitted['verified']})"
if fetcher_error is not None:
unknowable(row, fetcher_error)
if not pinned_present:
row.status = "missing"
row.upstream = upstream
row.note = (row.note + "; " if row.note else "") + "not installed in the image"
row.hint = "REA advertises this capability but the tool is absent - it will report provider_missing"
return row
if admitted:
target = admitted.get("verified") or admitted.get("required", "")
order = compare(pinned, target)
if order is None:
if not pinned.startswith(tuple(str(ch) for ch in "0123456789")):
return row
elif order < 0:
row.status = "behind-rea"
row.note = (row.note + "; " if row.note else "") + (
f"pinned {pinned} is older than what REA admits ({target})"
)
row.hint = "REA checks tool version banners; bump the pin to the admitted version"
return row
elif order > 0 and not floor_semantics:
row.status = "ahead"
row.note = (row.note + "; " if row.note else "") + (
f"pinned {pinned} is NEWER than REA's verified {target}"
)
row.hint = "unsupported newer banner - verify with the harness before trusting it"
order = compare(pinned, row.upstream)
if order is not None and order < 0 and row.status == "ok":
if floor_semantics and admitted:
# REA states a minimum, not a target: anything at or above the floor
# is admitted, so a newer upstream is optional rather than required.
row.status = "info"
row.note = ((row.note + "; ") if row.note else "") + (
f"REA's floor is {admitted.get('required')}; {row.upstream} is optional"
)
elif admitted and compare(pinned, admitted.get("verified", "")) == 0:
row.status = "info"
row.note = (row.note + "; " if row.note else "") + "upstream newer, but not admitted by REA"
else:
row.status = "outdated"
row.hint = row.hint or "newer upstream release available"
return row
def check_tools(fetcher: Fetcher, pins: dict[str, str], admissions: ReaAdmissions) -> list[Row]:
rows: list[Row] = []
pypi = {
"unblob": ("UNBLOB_VERSION", "unblob"),
"mitmproxy": ("MITMPROXY_VERSION", "mitmproxy"),
"pwntools": ("PWTOOLS_VERSION", "pwntools"),
"pyelftools": ("PYELFTOOLS_VERSION", "pyelftools"),
"unicorn": ("UNICORN_VERSION", "unicorn"),
"uv": ("UV_VERSION", "uv"),
}
github = {
"binwalk": ("BINWALK_VERSION", "ReFirmLabs/binwalk", False),
"wabt": ("WABT_VERSION", "WebAssembly/wabt", False),
"apktool": ("APKTOOL_VERSION", "iBotPeaches/Apktool", True),
"pwndbg": ("PWNDBG_TAG", "pwndbg/pwndbg", False),
}
npm = {
"frida": ("FRIDA_VERSION", "frida"),
"supergateway": ("SUPERGATEWAY_VERSION", "supergateway"),
"jadx-headless-mcp": ("JADX_MCP_VERSION", "jadx-headless-mcp"),
}
for key, (pin, package) in pypi.items():
pinned = pins.get(pin, "")
url = f"https://pypi.org/pypi/{package}/json"
upstream, error = None, None
try:
upstream = pypi_latest(fetcher, package)
except Exception as exc: # noqa: BLE001
error = exc
rows.append(tool_row(key, f"{package} (PyPI)", pinned, admissions.tool(key), url, upstream, error))
for key, (pin, repo, floor) in github.items():
pinned = pins.get(pin, "")
url = f"https://api.github.com/repos/{repo}/releases/latest"
upstream, error = None, None
try:
upstream = github_latest(fetcher, repo)
except Exception as exc: # noqa: BLE001
error = exc
rows.append(tool_row(key, f"{repo} (GitHub)", pinned, admissions.tool(key), url, upstream, error,
floor_semantics=floor))
for key, (pin, package) in npm.items():
pinned = pins.get(pin, "")
url = f"https://registry.npmjs.org/{package}/latest"
upstream, error = None, None
try:
upstream = str(npm_latest(fetcher, package)["version"])
except Exception as exc: # noqa: BLE001
error = exc
rows.append(tool_row(key, f"{package} (npm)", pinned, admissions.tool(key), url, upstream, error))
# ilspycmd: NuGet, and REA imposes no version admission
pinned = pins.get("ILSPYCMD_VERSION", "")
url = "https://api.nuget.org/v3-flatcontainer/ilspycmd/index.json"
upstream, error = None, None
try:
data = fetcher.json(url)
stable = [v for v in data["versions"] if "-" not in v] # type: ignore[index]
upstream = sorted(stable, key=lambda v: vparse(v) or (0,))[-1]
except Exception as exc: # noqa: BLE001
error = exc
row = tool_row("ilspycmd", "ilspycmd (NuGet)", pinned, admissions.tool("ilspycmd"), url, upstream, error)
if not admissions.tool("ilspycmd"):
if row.status == "outdated":
row.status = "info"
row.note = ((row.note + "; ") if row.note else "") + "REA only requires a runnable ilspycmd"
row.admits = None
rows.append(row)
# Wakaru: REA advertises javascript recovery, we do not install it
admitted = admissions.tool("wakaru")
upstream, error = None, None
try:
upstream = str(npm_latest(fetcher, "wakaru")["version"])
except Exception as exc: # noqa: BLE001
error = exc
rows.append(tool_row("wakaru", "wakaru (npm)", "", admitted,
"https://registry.npmjs.org/wakaru/latest", upstream, error,
pinned_present=False))
return rows
def check_misc(fetcher: Fetcher, pins: dict[str, str], admissions: ReaAdmissions) -> list[Row]:
rows: list[Row] = []
row = Row("ubuntu", "ubuntu:24.04 base", "Base & CI", "info", pinned="24.04",
source="https://hub.docker.com/v2/repositories/library/ubuntu/tags/24.04")
try:
data = fetcher.json("https://hub.docker.com/v2/repositories/library/ubuntu/tags/24.04")
row.upstream = str(data.get("last_updated", ""))[:10] # type: ignore[union-attr]
digest = str(data.get("digest", "")) # type: ignore[union-attr]
row.note = f"current digest {digest[:19]}…"
row.hint = "no pin to bump: a rebuild picks up the current image"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
rows.append(row)
for key, label, binary, url in [
("tshark", "tshark/capinfos (apt, REA wants 4.x)", "tshark", "https://api.github.com/repos/wireshark/wireshark/releases/latest"),
("sigrok", "sigrok-cli (apt, REA wants 0.7.2)", "sigrok-cli", "https://api.github.com/repos/sigrokproject/sigrok-cli/releases/latest"),
]:
row = Row(key, label, "REA-admitted tools", "info",
admits=(admissions.tool(key.split("-")[0]) or {}).get("required"),
source="Launchpad noble/amd64")
try:
row.pinned = launchpad_noble(fetcher, binary)
row.note = "installed from apt at build time (no version pin)"
row.hint = "a rebuild picks up Ubuntu security updates"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
rows.append(row)
row = Row("dotnet", "dotnet-sdk-8.0 (apt)", "Base & CI", "info", pinned="8.0.x",
source="https://dotnetcli.blob.core.windows.net/dotnet/release-metadata/8.0/releases.json")
try:
data = fetcher.json("https://dotnetcli.blob.core.windows.net/dotnet/release-metadata/8.0/releases.json")
row.upstream = str(data["latest-release"]) # type: ignore[index]
row.note = "apt-provided; rebuild to pick up patches"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
rows.append(row)
row = Row("chrome", "Google Chrome (unpinned deb)", "Base & CI", "info",
pinned="google-chrome-stable_current",
source="https://googlechromelabs.github.io/chrome-for-testing/last-known-good-versions.json")
try:
data = fetcher.json("https://googlechromelabs.github.io/chrome-for-testing/last-known-good-versions.json")
row.upstream = str(data["channels"]["Stable"]["version"]) # type: ignore[index]
row.note = "always the current stable at build time"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
rows.append(row)
row = Row("playwright", "playwright (npm)", "Base & CI", "info",
source="https://registry.npmjs.org/playwright/latest")
try:
row.upstream = str(npm_latest(fetcher, "playwright")["version"])
row.note = "installed unpinned; browsers land in /opt/ms-playwright"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
rows.append(row)
row = Row("ida-pro-mcp", "mrexodia/ida-pro-mcp (BYO IDA path)", "Engines", "info",
source="https://api.github.com/repos/mrexodia/ida-pro-mcp/releases/latest")
try:
row.upstream = github_latest(fetcher, "mrexodia/ida-pro-mcp")
row.note = "only used when a user brings their own IDA"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
rows.append(row)
return rows
def check_actions(fetcher: Fetcher) -> list[Row]:
"""Compare pinned action majors with the newest release major."""
rows: list[Row] = []
seen: dict[str, str] = {}
if not WORKFLOW_DIR.exists():
return rows
for workflow in sorted(WORKFLOW_DIR.glob("*.yml")):
for match in re.finditer(r"uses:\s*([\w.\-]+/[\w.\-]+)@(\S+)", workflow.read_text()):
seen.setdefault(match.group(1), match.group(2))
for action, pin in sorted(seen.items()):
row = Row(f"action:{action}", action, "Base & CI", "ok", pinned=pin,
source=f"https://api.github.com/repos/{action}/releases/latest")
try:
latest = github_latest(fetcher, action)
row.upstream = latest
if major(latest) and major(pin) and major(latest) > major(pin): # type: ignore[operator]
row.status = "outdated"
row.hint = "bump the action major (a stale major is what triggers the Node runtime deprecation warning)"
elif major(latest) and major(pin) and major(latest) == major(pin):
row.status = "ok"
except Exception as exc: # noqa: BLE001
unknowable(row, exc)
rows.append(row)
return rows
# --------------------------------------------------------------------------- #
# main
# --------------------------------------------------------------------------- #
def main() -> int:
parser = argparse.ArgumentParser(
description="Check whether rea-agent-docker's pins are still current.",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument("--json", action="store_true", help="emit JSON")
parser.add_argument("--markdown", action="store_true", help="emit a markdown table (CI step summary)")
parser.add_argument("--only", default="", help="comma-separated keys/groups to include (keys: run --json to list)")
parser.add_argument("--skip", default="", help="comma-separated keys/groups to skip")
parser.add_argument("--jobs", type=int, default=8, help="concurrent lookups (default 8)")
parser.add_argument("--timeout", type=float, default=12.0, help="per-request timeout in seconds")
parser.add_argument("--no-fail", action="store_true", help="always exit 0")
parser.add_argument("--quiet", action="store_true", help="only print actionable rows")
args = parser.parse_args()
docker_pins, env_pins = read_pins()
pins = {**docker_pins, **{k: v for k, v in env_pins.items() if k not in docker_pins}}
fetcher = Fetcher(args.timeout, args.jobs)
rea_version = pins.get("REA_VERSION", "")
admissions = scrape_rea(fetcher, rea_version) if rea_version else ReaAdmissions(version="")
checks = [
lambda: check_rea(fetcher, pins, admissions),
lambda: check_node(fetcher, pins, admissions),
lambda: check_ghidra(fetcher, pins, admissions),
lambda: check_hopper(fetcher, pins, admissions),
lambda: check_java(fetcher, pins, admissions),
lambda: check_tools(fetcher, pins, admissions),
lambda: check_misc(fetcher, pins, admissions),
lambda: check_actions(fetcher),
]
rows: list[Row] = []
with ThreadPoolExecutor(max_workers=min(args.jobs, len(checks) * 2)) as pool:
for result in pool.map(lambda fn: fn(), checks):
rows.extend(result)
if args.only:
needles = [n.strip().lower() for n in args.only.split(",") if n.strip()]
rows = [r for r in rows if any(n in r.key.lower() or n in r.group.lower() for n in needles)]
if args.skip:
needles = [n.strip().lower() for n in args.skip.split(",") if n.strip()]
rows = [r for r in rows if not any(n in r.key.lower() or n in r.group.lower() for n in needles)]
actionable = [r for r in rows if r.status in ACTIONABLE]
unknowns = [r for r in rows if r.status == "unknown"]
stamp = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
if args.json:
print(json.dumps(
{
"generated_at": stamp,
"rea_version": rea_version,
"rea_admissions": {
"node_range": admissions.node_range,
"ghidra": admissions.ghidra,
"ghidra_java_major": admissions.ghidra_java_major,
"hopper_deb": admissions.hopper_deb,
"hopper_launcher_hashes": len(admissions.hopper_hashes),
"tools": admissions.tools,
"errors": admissions.errors,
},
"summary": {
"rows": len(rows),
"actionable": len(actionable),
"unknown": len(unknowns),
},
"rows": [r.as_dict() for r in rows],
},
indent=2,
))
return 1 if actionable and not args.no_fail else 0
if args.markdown:
print(f"### rea-agent-docker update check ({stamp})")
print()
print(f"Pins read from `Dockerfile`; REA admission tables read from "
f"`morluto/rea@rea-agents-{rea_version}`. "
f"**{len(actionable)} actionable**, {len(unknowns)} unverifiable, {len(rows)} rows.")
print()
if actionable:
print("| status | component | pinned | upstream | REA admits |")
print("| --- | --- | --- | --- | --- |")
for row in actionable:
print(f"| **{STATUS_LABEL[row.status]}** | {row.label} | {row.pinned or '-'} | "
f"{row.upstream or '-'} | {row.admits or '-'} |")
else:
print("Nothing actionable: every pin matches what upstream and REA currently expect.")
return 1 if actionable and not args.no_fail else 0
print(f"rea-agent-docker update check — {stamp}")
print(f"pins: Dockerfile ARGs; REA admissions: morluto/rea@rea-agents-{rea_version}")
if admissions.errors:
for name, error in admissions.errors.items():
print(f" ! could not read REA {name} table: {error}")
print()
# an engine that REA does not admit is the failure mode worth shouting about
for row in rows:
if row.status == "behind-rea":
print(f" !! {row.label}: {row.note}")
if row.hint:
print(f" -> {row.hint}")
if any(row.status == "behind-rea" for row in rows):
print()
groups: dict[str, list[Row]] = {}
for row in rows:
if args.quiet and row.status not in ACTIONABLE:
continue
groups.setdefault(row.group, []).append(row)
width = max((len(r.label) for r in rows), default=20) + 2
for group, group_rows in groups.items():
print(f"{group}")
for row in group_rows:
label = STATUS_LABEL[row.status]
pin = row.pinned or "-"
bits = [f"pin {pin}"]
if row.upstream and row.upstream != row.pinned:
bits.append(f"upstream {row.upstream}")
if row.admits and row.admits != row.pinned:
bits.append(f"REA {row.admits}")
line = f" [{label:^10}] {row.label:<{width}} " + " ".join(bits)
print(line)
if row.note:
print(f" {'':<{width}} {row.note}")
if row.status in ACTIONABLE and row.hint:
print(f" {'':<{width}} -> {row.hint}")
print()
print(f"{len(rows)} rows: {len(actionable)} actionable, "
f"{len(unknowns)} unverifiable, {len(rows) - len(actionable) - len(unknowns)} fine")
if actionable:
print("See docs/UPDATES.md for what to check and how to apply each bump.")
return 1 if actionable and not args.no_fail else 0
if __name__ == "__main__":
sys.exit(main())