libretro/virtualjaguar-libretro
1
0 Fork 0
mirror of https://github.com/libretro/virtualjaguar-libretro.git synced 2026-11-07 04:36:00 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

feat(test): add a GitHub release-asset backend to the ROM store

Browse Source 
At 22 MB a private GitHub repo is simpler and free, and the earlier "avoid
GitHub" note did not apply to it: that was about LFS, whose bandwidth is sold
in $5/50 GB packs. Regular git operations and release assets are not metered
that way, and nothing here is near a limit (100 MB/file, 2 GB/asset).

A release ASSET rather than files committed to a repo: git keeps every version
of a binary forever, so re-dumping a 4 MB ROM would add 4 MB to every clone
permanently, while an asset is replaced in place. The whole Tier-1 set travels
as one zip -- nine assets would be nine round trips and nine chances to end up
half-updated, and at this size partial transfers buy nothing.

Backend is inferred: VJ_ROM_GH_REPO set -> github, else s3. Both stay
supported, because Tier 2 (GB-scale CD images) will want R2's economics where
asset and repo ceilings bite first.

CI passes both secret groups; whichever is configured wins. VJ_ROM_GH_TOKEN
must be a fine-grained PAT with read-only Contents on the ROM repo -- the
built-in GITHUB_TOKEN is scoped to THIS repository and cannot read another
private one, which surfaces as a 404 rather than a permission error, so it is
worth stating in the workflow rather than leaving someone to debug it.

Docs also record the non-cost risk: these are commercial ROMs plus a BIOS
image, a takedown strike lands on the account or org rather than the file, and
Atari SA does act on Jaguar IP -- so prefer a personal account over the org
that hosts the emulator itself.

Verified against a stubbed `gh` implementing the release verbs over a
directory: publish creates the release and uploads a 17.5 MB zip; cold fetch
unpacks and verifies all 9; warm fetch never contacts the release; a missing
token exits 77 naming the GITHUB variables (the first version printed the S3
ones, which sends a GitHub user hunting a credential that has no bearing on
the skip); and a tampered archive is rejected with "fetched archive does not
match the manifest", exit 1, rather than being left where the next run's fast
path would trust it.

That tamper case initially "passed" because the packed ROMs carry mode 444
from the read-only corpus, so the dd that was meant to corrupt one failed with
Permission denied and the test proved nothing. Chasing the wrong result to its
cause is what turned it into a real check.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Joseph MattielloandClaude Opus 5 committed 2026-08-20 20:48:15 -04:00
1 parent 28326f0ad6
commit 4fa775d5cb
3 files changed
+196 -30
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/c-cpp.yml
+10
View File
@@ -255,6 +255,16 @@ jobs:
if: ${{ !matrix.config.emscripten && !matrix.config.android && !matrix.config.cross && runner.os != 'Windows' }}
continue-on-error: true
env:
# Two backends; rom-store.sh picks whichever is configured (GitHub
# wins when VJ_ROM_GH_REPO is set). Set one group of secrets, not both.
#
# GitHub backend: VJ_ROM_GH_TOKEN must be a fine-grained PAT with
# read-only Contents on the ROM repo. The built-in GITHUB_TOKEN cannot
# be used -- it is scoped to THIS repository and cannot read another
# private one, which fails as a 404 rather than a permission error.
VJ_ROM_GH_REPO: ${{ secrets.VJ_ROM_GH_REPO }}
GH_TOKEN: ${{ secrets.VJ_ROM_GH_TOKEN }}
# S3 backend (Cloudflare R2 etc.)
VJ_ROM_ENDPOINT: ${{ secrets.VJ_ROM_ENDPOINT }}
VJ_ROM_BUCKET: ${{ secrets.VJ_ROM_BUCKET }}
VJ_ROM_KEY: ${{ secrets.VJ_ROM_KEY }}
docs/test-rom-store.md
+71 -24
View File
@@ -36,41 +36,84 @@ just grows the download.
22 MB fits GitHub's 10 GB Actions cache with four orders of magnitude spare, so
steady-state CI egress is effectively zero — only a manifest change re-downloads.
## Why R2
## Choosing a backend
Any S3-compatible store works; nothing about the tooling is R2-specific.
Cloudflare R2 is the recommended default for one reason: **zero egress fees**.
CI is an egress-shaped workload — it downloads far more than it stores — which
is exactly where per-GB bandwidth pricing bills you for a corpus that costs
cents to store. 22 MB on R2 rounds to free; the full 24 GB would be ~$0.36/mo.
Two are supported. `rom-store.sh` picks GitHub when `VJ_ROM_GH_REPO` is set,
otherwise S3.
Avoid GitHub LFS for this. Its bandwidth is sold in $5 / 50 GB packs, and a
CI-shaped workload burns packs fast.
### GitHub release asset (simplest, free)
**The bucket must be private.** These are commercial ROMs and a BIOS image.
`rom-store.sh` never creates a bucket, never sets an ACL, and never prints a
credential.
One zip attached to a release in a **private** repo, moved with `gh`. Free,
one secret, and `gh` is already on every runner. At 22 MB nothing is near a
limit (100 MB/file, 2 GB/asset).
A release **asset**, not files committed to the repo: git keeps every version
of a binary forever, so re-dumping a 4 MB ROM would add 4 MB to every clone
permanently, while an asset is replaced in place.
This is **not** the GitHub LFS path, whose bandwidth is sold in $5/50 GB packs.
Regular git operations and release assets are not metered that way.
### S3-compatible (Cloudflare R2 etc.)
Zero egress on R2, ~$0.36/mo for the entire 24 GB corpus. Worth it when the
corpus outgrows what belongs on GitHub — Tier 2 CD images are GB-scale, where
asset and repo ceilings bite long before R2's economics do.
### The risk that is not about cost
These are commercial ROMs plus a BIOS image. Hosting them on GitHub is a
takedown surface, and **a strike lands on the account or org, not just the
file**. Atari SA holds the Jaguar IP and does act on it. If you use the GitHub
backend, prefer a **personal account over the org that hosts the emulator
itself** — losing a scratch ROM repo is an inconvenience; a strike against the
project's org is not.
Either way the repo/bucket must be **private**. `rom-store.sh` never creates a
bucket, never sets an ACL, and never prints a credential.
## One-time setup
1. Create a **private** R2 bucket (Cloudflare dashboard → R2 → Create bucket),
e.g. `vj-test-roms`.
2. Create an R2 API token scoped to **that bucket only**, with Object
Read & Write. Note the Access Key ID, Secret Access Key, and your account
ID — the endpoint is `https://<account-id>.r2.cloudflarestorage.com`.
3. Publish from a machine that has the full corpus:
### GitHub backend
1. Create a **private** repo, e.g. `JoeMatt/vj-test-roms`. It needs no content;
the ROMs live in a release asset.
2. Publish from a machine that has the full corpus:
```bash
export VJ_ROM_GH_REPO=JoeMatt/vj-test-roms
export GH_TOKEN=... # or just be logged in: gh auth login
make roms-manifest # regenerate from your local corpus
make roms-publish # creates the release, uploads tier1.zip
```
3. Add two repository secrets. `VJ_ROM_GH_TOKEN` must be a **fine-grained PAT
with read-only Contents** on the ROM repo — the built-in `GITHUB_TOKEN` is
scoped to this repository and cannot read another private one (it fails as
a 404, not a permission error):
```bash
gh secret set VJ_ROM_GH_REPO --repo libretro/virtualjaguar-libretro
gh secret set VJ_ROM_GH_TOKEN --repo libretro/virtualjaguar-libretro
```
### S3 / R2 backend
1. Create a **private** R2 bucket (Cloudflare dashboard → R2), e.g.
`vj-test-roms`.
2. Create an R2 API token scoped to that bucket, Object Read & Write. The
endpoint is `https://<account-id>.r2.cloudflarestorage.com`.
3. Publish:
```bash
export VJ_ROM_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
export VJ_ROM_BUCKET=vj-test-roms
export VJ_ROM_KEY=... # R2 access key id
export VJ_ROM_SECRET=... # R2 secret access key
make roms-manifest # regenerate from your local corpus
make roms-publish
export VJ_ROM_KEY=... VJ_ROM_SECRET=...
make roms-manifest && make roms-publish
```
4. Add four repository secrets so CI can read the bucket. Use a **read-only**
token here, not the read-write one used to publish:
4. Add four secrets, using a **read-only** token for CI rather than the one
used to publish:
```bash
gh secret set VJ_ROM_ENDPOINT --repo libretro/virtualjaguar-libretro
@@ -80,7 +123,7 @@ credential.
```
Secrets are unavailable to fork PRs by design. The fetch step is best-effort
and those runs simply skip the ROM-gated checks, exactly as they do today.
and those runs skip the ROM-gated checks, exactly as they do today.
## Daily use
@@ -104,6 +147,10 @@ ROMS_PRIVATE_ROOT=test/roms/tier1 make test
never be picked up by a later run as a valid cached copy.
- **Fetch is idempotent and offline-clean.** If every file is already present
and hashes match, the store is not contacted at all.
- **The GitHub backend verifies the archive after unpacking, into a staging
tree first.** A half-extracted or tampered zip is rejected (`fetched archive
does not match the manifest`, exit 1) rather than being left in place where
the next run's fast path would treat it as a verified corpus.
- **`manifest` and `publish` share one resolver.** They both walk
`tier1.patterns`; neither re-derives a source path by globbing a manifest
basename. That is not hypothetical tidiness — the first version did re-glob,
scripts/rom-store.sh
+115 -6
View File
@@ -33,7 +33,30 @@
# scripts/rom-store.sh fetch # download + verify into $VJ_ROM_DEST
# scripts/rom-store.sh verify # hash-check what is already on disk
#
# ENV
# BACKENDS
# VJ_ROM_BACKEND=github (default when VJ_ROM_GH_REPO is set)
# One zip attached to a release in a PRIVATE GitHub repo, moved with the
# `gh` CLI. Free, one secret, and `gh` is already on every runner.
# A release ASSET rather than files committed to the repo: git keeps
# every version of a binary forever, so re-dumping a 4 MB ROM would add
# 4 MB to every clone permanently, while an asset is replaced in place.
# Right for Tier 1 (22 MB). Wrong for Tier 2: CD images are GB-scale,
# and asset/repo ceilings bite long before R2's economics do.
#
# VJ_ROM_BACKEND=s3 (default otherwise)
# Any S3-compatible store. R2 recommended for zero egress.
#
# Hosting commercial ROMs plus a BIOS image on GitHub is a takedown surface,
# and a strike lands on the ACCOUNT, not the file. Prefer a personal
# account over an org that hosts the emulator itself.
#
# ENV (github backend)
# VJ_ROM_GH_REPO owner/repo of the private ROM repo
# VJ_ROM_GH_TAG release tag holding the asset (default: tier1)
# VJ_ROM_GH_ASSET asset filename (default: tier1.zip)
# GH_TOKEN token with read access (write, to publish)
#
# ENV (s3 backend)
# VJ_ROM_ENDPOINT S3 endpoint URL. R2: https://<account>.r2.cloudflarestorage.com
# VJ_ROM_BUCKET bucket name
# VJ_ROM_KEY access key id (GitHub secret)
@@ -58,6 +81,15 @@ DEST="${VJ_ROM_DEST:-$REPO/test/roms/tier1}"
case "$DEST" in /*) ;; *) DEST="$REPO/$DEST" ;; esac
PREFIX="${VJ_ROM_PREFIX:-tier1}"
REGION="${VJ_ROM_REGION:-auto}"
GH_REPO="${VJ_ROM_GH_REPO:-}"
GH_TAG="${VJ_ROM_GH_TAG:-tier1}"
GH_ASSET="${VJ_ROM_GH_ASSET:-tier1.zip}"
# Infer the backend so a caller that sets only the GitHub vars does not also
# have to remember to say so.
BACKEND="${VJ_ROM_BACKEND:-}"
if [ -z "$BACKEND" ]; then
if [ -n "$GH_REPO" ]; then BACKEND=github; else BACKEND=s3; fi
fi
die() { echo "rom-store: $*" >&2; exit 1; }
note() { echo "rom-store: $*"; }
@@ -75,8 +107,67 @@ filesize() { wc -c < "$1" | tr -d ' '; }
# ------------------------------------------------------------------ store ---
store_configured() {
[ -n "${VJ_ROM_ENDPOINT:-}" ] && [ -n "${VJ_ROM_BUCKET:-}" ] &&
[ -n "${VJ_ROM_KEY:-}" ] && [ -n "${VJ_ROM_SECRET:-}" ]
case "$BACKEND" in
github) [ -n "$GH_REPO" ] && [ -n "${GH_TOKEN:-${GITHUB_TOKEN:-}}" ] ;;
*) [ -n "${VJ_ROM_ENDPOINT:-}" ] && [ -n "${VJ_ROM_BUCKET:-}" ] &&
[ -n "${VJ_ROM_KEY:-}" ] && [ -n "${VJ_ROM_SECRET:-}" ] ;;
esac
}
gh_cli() {
command -v gh >/dev/null 2>&1 || die "gh CLI not found (brew install gh)"
GH_TOKEN="${GH_TOKEN:-${GITHUB_TOKEN:-}}" gh "$@"
}
# The whole Tier-1 set travels as ONE zip: nine separate release assets would
# be nine round trips and nine chances to end up half-updated, and at 22 MB
# there is nothing to gain from partial transfers.
gh_publish() {
local staging zip src rel n=0
staging=$(mktemp -d "${TMPDIR:-/tmp}/vjroms.XXXXXX")
zip="$staging/$GH_ASSET"
mkdir -p "$staging/pack"
while IFS=$'\t' read -r src rel; do
[ -n "${rel:-}" ] || continue
mkdir -p "$staging/pack/$(dirname "$rel")"
cp "$src" "$staging/pack/$rel"
n=$((n + 1))
done < <(resolve_patterns) || { command rm -rf "$staging"; die "local corpus is incomplete; cannot publish"; }
( cd "$staging/pack" && zip -q -r -X "$zip" . ) || { command rm -rf "$staging"; die "zip failed"; }
note "packed $n file(s) -> $GH_ASSET ($(filesize "$zip") bytes)"
# Create the release on first publish; --clobber replaces the asset after.
gh_cli release view "$GH_TAG" --repo "$GH_REPO" >/dev/null 2>&1 || {
note "creating release $GH_TAG in $GH_REPO"
gh_cli release create "$GH_TAG" --repo "$GH_REPO" \
--title "Tier-1 test ROMs" \
--notes "Tier-1 private test corpus. Contents and hashes: test/roms/tier1.manifest" \
>/dev/null || { command rm -rf "$staging"; die "could not create release $GH_TAG"; }
}
gh_cli release upload "$GH_TAG" "$zip" --repo "$GH_REPO" --clobber >/dev/null \
|| { command rm -rf "$staging"; die "asset upload failed"; }
command rm -rf "$staging"
note "published $n file(s) to $GH_REPO release $GH_TAG ($GH_ASSET)"
}
gh_fetch() {
local staging
staging=$(mktemp -d "${TMPDIR:-/tmp}/vjroms.XXXXXX")
note "get $GH_REPO release $GH_TAG :: $GH_ASSET"
gh_cli release download "$GH_TAG" --repo "$GH_REPO" --pattern "$GH_ASSET" \
--dir "$staging" --clobber >/dev/null \
|| { command rm -rf "$staging"; die "release download failed ($GH_REPO $GH_TAG)"; }
command -v unzip >/dev/null 2>&1 || { command rm -rf "$staging"; die "unzip not found"; }
# Unpack to a staging tree, NOT straight over DEST: the per-file hash check
# in cmd_verify below is what decides these are good, and a half-extracted
# archive must never be able to masquerade as a verified corpus.
unzip -qo "$staging/$GH_ASSET" -d "$staging/pack" \
|| { command rm -rf "$staging"; die "unzip failed"; }
mkdir -p "$DEST"
( cd "$staging/pack" && find . -type f -print0 | while IFS= read -r -d '' f; do
mkdir -p "$DEST/$(dirname "${f#./}")"
mv -f "$f" "$DEST/${f#./}"
done )
command rm -rf "$staging"
}
# aws CLI is the transport: it is the one S3 client already present on GitHub
@@ -140,6 +231,7 @@ cmd_manifest() {
cmd_publish() {
[ -f "$MANIFEST" ] || die "no manifest -- run: $0 manifest"
store_configured || { note "SKIP: store not configured (see ENV in $0)"; exit 77; }
if [ "$BACKEND" = github ]; then gh_publish; return; fi
local src rel n=0 want
while IFS=$'\t' read -r src rel; do
[ -n "${rel:-}" ] || continue
@@ -167,11 +259,28 @@ cmd_fetch() {
return 0
fi
store_configured || {
note "SKIP: store not configured -- set VJ_ROM_ENDPOINT / VJ_ROM_BUCKET /"
note " VJ_ROM_KEY / VJ_ROM_SECRET. Local devs with their own corpus"
note " should point ROMS_PRIVATE_ROOT at it and skip this entirely."
# Name the variables for the backend actually in play. Telling a GitHub
# user to set VJ_ROM_KEY sends them looking for a credential that has no
# bearing on why their fetch skipped.
if [ "$BACKEND" = github ]; then
note "SKIP: store not configured -- set VJ_ROM_GH_REPO and GH_TOKEN."
else
note "SKIP: store not configured -- set VJ_ROM_ENDPOINT / VJ_ROM_BUCKET /"
note " VJ_ROM_KEY / VJ_ROM_SECRET."
fi
note " Local devs with their own corpus should point"
note " ROMS_PRIVATE_ROOT at it and skip this entirely."
exit 77
}
if [ "$BACKEND" = github ]; then
gh_fetch
if cmd_verify; then
note "fetch complete -> $DEST"
note "point the suite at it with: ROMS_PRIVATE_ROOT=$DEST"
return 0
fi
die "fetched archive does not match the manifest"
fi
while read -r hash size rel; do
[ -n "${rel:-}" ] || continue
dst="$DEST/$rel"