52 Commits
Author SHA1 Message Date
Georges-Antoine AssiandClaude Opus 5.5 ad0146a88e chore(trunk): fix the scheduled trunk check-all failures (#5141)
* chore(trunk): fix the scheduled trunk check-all failures

- Ignore hadolint DL3008 on the runtime stage apt-get, matching the
  emulator-download stage
- Build alias-search test ROMs with tests.factories.make_rom instead of
  the banned add_rom
- Drop a bandit B405 ignore that suppresses nothing
- Bump fast-uri to 3.1.8 for GHSA-hrr3-gc8f-f4qj

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(tests): type the alias-search rom helper kwargs as Any

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
2026-10-05 08:05:38 -04:00
Georges-Antoine AssiandClaude Opus 5.5 6f2a361481 Play RPG Maker 2000/2003 games in the browser with EasyRPG (#5077)
* feat(player): play RPG Maker 2000/2003 games in the browser with EasyRPG

Embed the EasyRPG web player for extracted game folders on the rpg-maker
platform. The backend generates the player's index.json from the scanned
files and merges in the free EasyRPG RTP under every name the official
and fan RTPs give its assets, so games that lean on the RTP find them.
In-game saves sync with the user's RomM saves through the player's
IndexedDB storage.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(easyrpg): read the free RTP from a fixed path instead of an env var

The full image always keeps the RTP at /var/www/html/assets/easyrpg/rtp,
so the path is a constant beside the web path nginx serves it under. The
dev container links its copy there at startup.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(easyrpg): code-review fixes for the save sync and asset endpoint

- Wait for an in-flight push before the quit push, so the newest save uploads
- Keep Play disabled until the fresh rom (and its saves) is fetched
- Stop onPlay and the rom fetch from acting after the view unmounts
- Leave through usePlayerExit like the other players
- Handle a rejected IDB commit when the run itself fails
- Track uploads by the browser's file name
- Load only visibility and files per asset request, not the full rom
- Serve DEV_MODE files as octet-stream, matching nginx
- Make the traversal test actually reach the endpoint

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(easyrpg): simplify pass over the player and asset endpoint

- Load only file paths per asset request, and look up RTP folders in O(1)
- Reuse RomFile.file_name_for_download and usePlayerFullscreen
- Share one dev/prod serve helper for game and RTP files
- Derive the save sync's game key from the rom
- Have the RTP table tool reuse normalize_name
- Make save-module internals private and tidy the prepare/read loops

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* feat(easyrpg): flag RPG Maker game folders on the rom schemas

Gallery lists carry no files, so the frontend guessed that every multi-file
rom on the rpg-maker platform was playable, including RPG Maker XP/VX/MV
folders. Rom now has a deferred `is_easyrpg_game` subquery (an RPG_RT.ldb at
the folder root), loaded with the other gallery file stats and derived from
loaded files on the with_files path, and the frontend reads it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(easyrpg): drop server save sync, keep saves per user in the browser

Server sync moves to a follow-up that puts the web players on device sync.
Saves now stay in the player's IndexedDB, as js-dos does, with the same note
on the start page. The player is launched as `<rom id>-<user id>` so each
RomM account sharing a browser profile keeps its own saves; nginx and Vite
drop the user id when routing to the backend.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(easyrpg): address Greptile review

- Start timing the play session once the player page loads
- Use object-form props in the view test's button mock
- Keep the RTP table tool's docstring to one sentence

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(easyrpg): keep the RTP table in a fixtures folder

Matches handler/metadata/fixtures, where the other bundled data files live.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
2026-10-03 17:51:38 -04:00
Sam Dornan 655976faa9 feat(soundtrack): play SPC, VGM, VGZ and GYM soundtracks in the browser (#5079) 2026-10-03 13:12:28 -04:00
Yukine f0928e4f53 chore(converto): bump rom-converto to v0.23.2 in both images (#5041) 2026-10-02 19:00:42 -04:00
640a783f3f feat(converto): convert the library in place and serve downloads in a requested format (#4999)
* feat(converto): integrate rom-converto for title id extraction and download conversion

* fix(docker): ship rom-converto in the production image

Only the dev Dockerfile installed the binary, so the integration could
never enable on a released image. Fetch the same pinned, checksummed
musl build in a dedicated stage and copy it next to RAHasher.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(converto): skip the update check and batch info calls

The CLI asks api.github.com for a newer release on every run unless
told not to, so pass --no-update-check on each invocation. Inspect a
rom's files in one `info --paths-file` run instead of one subprocess
per file, and read the inspectable extensions from the capabilities
manifest so scans skip files info cannot recognize.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(scan): keep sigil authoritative for title ids

rom-converto ran first and overrode the ids sigil read. Sigil now runs
first and records its per-file ids; rom-converto only fills files sigil
left unset, on new or changed files, in one batch. Restore the base-id
selection for sigil extractions on every platform, as on master.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(downloads): make conversion opt-in and bound the cache

The download endpoint returned a converted file to every client once a
platform had a target, including the browser player and apps that check
hashes against RomFile. Conversion now needs ?converted=true.

A pre-warmed copy is served whatever its size; the in-request size cap
only gates starting a conversion. ROM_CONVERTO_CACHE_MAX_SIZE_GB
(default 20, 0 unbounded) caps the cache: conversions and the pre-warm
task stop when it is full, and cleanup evicts the least recently served
copies.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* chore(types): regenerate frontend types for the converto schema

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(converto): address code-review findings (pr-ready step 2)

- Fall back to rom-converto's stored ids only when it ran this pass, in
  disc order, so a rescan that skipped extraction keeps sigil's save target
- Persist per-file title ids that extraction writes onto reused rows
- Only authenticated callers may start an in-request conversion; past a
  240s deadline serve the original and let the conversion finish into the
  cache instead of nginx returning 504
- Treat a probe OSError or timeout as a failed probe; kill the subprocess
  when the caller is cancelled
- Give the pre-warm task the scan timeout and report progress in the
  ConversionStats shape the task UI reads
- Reuse zip_cache constants, walk the cache off the event loop, tolerate
  files vanishing mid-cleanup, and trim long comments

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(converto): simplify conversion code (pr-ready step 3)

- Move the converted-download policy (target lookup, cached-first, sync
  size cap, deadline) from the endpoint into utils/conversion_cache and
  share conversion_target with the pre-warm task
- Share one path lookup between get_cached_converted and get_or_convert;
  the pre-warm walks the cache once and tracks a running byte budget
  instead of walking it twice per rom
- One normalize_platform_formats beside TARGETS_BY_PLATFORM for the
  config loader (via _validated_platform_map) and the settings endpoint;
  update_converto_settings takes a ConvertoConfig and is written back
  with dataclasses.asdict
- read_infos owns its never-raise contract, so the scan drops its blanket
  except; drop the unused RomConvertoInfo.kind and error attributes
- Sort files for the rom-converto identity only when it is used; stat
  each cached file once during cleanup

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* chore(converto): trim comments (pr-ready step 4)

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(converto): convert on first download instead of pre-warming

A library pre-warm fought the TTL: cleanup dropped every copy nobody
downloaded within a day. Remove the Convert library task and convert on
demand only. Files over ROM_CONVERTO_MAX_SYNC_SIZE_MB now start their
conversion in the background while the original is served, so the next
download within the TTL gets the converted copy at any size.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(converto): drop decrypt and encrypt download targets

Downloads now only change a rom's container or compression (CHD, RVZ,
CSO, NSZ, ...); serving decrypted files is left for a separate decision.
NDS and PS3 have no remaining targets. Also note that the conversion
concurrency limit applies per process.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(converto): move the cache limits into converto.*

cache_max_size_gb and max_sync_size_mb join cache_ttl_hours in
config.yml and PUT /api/config/converto_settings, replacing the
ROM_CONVERTO_CACHE_MAX_SIZE_GB and ROM_CONVERTO_MAX_SYNC_SIZE_MB env
vars, so every eviction knob is edited in one place at runtime. The
loader takes its defaults from ConvertoConfig and one table holds each
integer's minimum for both the loader and the payload.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(converto): lock conversions with flock instead of a sentinel

A conversion now holds a non-blocking flock on its key dir, which the
kernel releases if the worker dies. That replaces the .partial sentinel,
its 6-hour staleness guess (which a long ROM_CONVERTO_TIMEOUT could
outlive), and the rename-aside takeover. Cleanup skips locked dirs, and
partial output gets a hidden name so it never counts as a served copy.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(converto): parametrize repeated test cases

Collapse the per-platform info parsing, operation resolution and
converto_active cases into parametrized tests, so a new platform or
target is one row.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(scan): run rom-converto first with sigil as the fallback

rom-converto reads each rom's new or changed files first, in one batch,
and its title ids win. sigil still reads what it can, since it alone
provides save targets and Switch content types, and fills the ids
rom-converto left unset.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* feat(converto): store per-file metadata read by rom-converto during scans

* fix(scan): keep the stored save target when sigil reads nothing

* fix(converto): read Xbox 360 disc ids and Wii title versions, store content type as text

* feat(converto): save file icons, banners and backgrounds read by rom-converto

* chore(converto): bump rom-converto to v0.23.0

* fix(converto): address review findings in the conversion cache, scan metadata and settings

* refactor(converto): move per-file metadata out of the conversion PR

Scan-time metadata (title ids, the 17 rom_files columns, artwork and
converto.scan_metadata) moves to its own PR so its columns land with the
UI that reads them. This PR keeps conversion only.

The conversion cache no longer has uncompressed_size_bytes to budget
with, so a conversion to an uncompressed target reserves a fixed
multiple of its input and the cleanup pass after it evicts any overshoot.

Co-Authored-By: DevYukine <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ

* feat(converto): let the client pick the download format with ?format=

`?format=zso,iso` lists the formats a client can read. The stored file is
served whenever its format is listed, even with conversion off, so most
emulator clients never trigger a conversion. Otherwise the first listed
format with a cached copy is served, then the first one rom-converto can
produce.

A conversion that outlasts a short wait answers 202 with Retry-After
instead of falling back to the original, so a URL never serves two
different files and a client never gets bytes it can't read. 406 means
no listed format can be served. HEAD reports the same without starting
a conversion. This replaces `?converted=true`, the per-platform download
target and `max_sync_size_mb`; `cache_ttl_hours` stays config.yml-only.

Co-Authored-By: DevYukine <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ

* feat(converto): convert the library in place with a manual task

`converto.platform_formats` now names the format a platform's library is
stored in, and the new "Convert library" task brings every matched rom
to it. Each output is staged in a `.romm_tmp_` directory beside the
original, moved into place, and only then are the originals (and a cue's
tracks) deleted. Single-file roms take the new name, folder roms keep
theirs and get their .m3u entries rewritten, and the file rows refresh,
so a rom keeps its id, saves and collections.

Only lossless conversions qualify, so xbox (xiso trims the dump) has no
library format. Unmatched roms are skipped because a converted file no
longer hash-matches a DAT. `/api/config` now carries the library targets
and, per input extension, the formats a download can be converted to.

Co-Authored-By: DevYukine <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ

* fix(converto): address Greptile and Copilot review findings

- Publish a converted library file with a no-replace link, so a file that
  appears during conversion is never overwritten.
- Delete originals only after the rom row or its playlists point at the new
  file, and keep them if that update fails. Playlists are rewritten atomically.
- Refuse a cue conversion whose tracks another rom or another cue holds.
- Remove stage dirs a killed conversion left behind.
- Try later ?format= targets when a conversion fails, treat any locked key dir
  as converting, and cap how many conversions downloads can start.
- Drop the unsupported iso target for CD platforms from the example config and
  list the ROM_CONVERTO_* variables in env.template.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(converto): address code-review findings (pr-ready step 2)

Serve converted outputs at 0644, read cue sheets and playlists with a BOM,
stage folder rom conversions outside the rom folder, keep one rom's failure
from stopping the convert library task, accept alias extensions as the stored
format, probe running conversions with a shared lock, and bound the timeout.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(converto): simplify conversion code (pr-ready step 3)

Check and chmod rom-converto output once in the adapter, canonicalize
requested format aliases, reuse LINK_FALLBACK_ERRNOS and the handler's atomic
write, make the download format table a constant, skip the probe lock once
cached, and fold duplicated name checks and cache stat passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* chore(converto): trim comments (pr-ready step 4)

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(converto): find rom-converto on PATH instead of ROM_CONVERTO_PATH

Both images install the binary on PATH, the variable was undocumented,
and RAHasher is already resolved the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Yukine <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: DevYukine <[email protected]>
2026-10-01 19:02:37 -04:00
Sam Dornan 864a0769fc chore(deps): bump uv to 0.12.19 and relax required-version for Dependabot (#4832) 2026-09-27 10:23:19 -04:00
Sam Dornan 6002b8fdaa chore(docker): verify downloads with ADD --checksum and pin RAHasher by commit (#4781) 2026-09-26 11:35:02 -04:00
nendo a10a62a29a fix(sigil): bump the pin so gamecube save targets name a gci on disk 2026-09-22 08:01:57 +09:00
nendo 33c3011ef6 fix(docker): clone sigil from its rommapp home 2026-09-17 07:07:07 +09:00
nendo ca17bfe7dd fix(docker): bump sigil to read lowercase and irregular PSX/PS2 boot serials 2026-09-16 21:22:47 +09:00
Sam DornanandClaude Opus 5 e371a80ffa fix(uv): pin uv exactly and cover the devcontainer image
The devcontainer image (./Dockerfile, built as the romm-dev compose
service) copies uv from a tag of its own and runs uv sync over the same
pyproject.toml, so the new required-version rejected its 0.11.2 and broke
dependency installation there. Bring it to the same version.

A floor also left the drift it was meant to close: setup-uv could resolve
a newer uv than either image runs, and a lock written by that uv would
pass CI and then fail an image build, which is label-gated and so not
part of an ordinary run. Pin the version exactly instead.

The test now covers both images and asserts the pin stays exact.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HDXxq3vAxJVGhm1pQuK8V7
2026-09-12 00:56:05 +00:00
Sam DornanandClaude Opus 5 a1e12fb6ac fix(docker): bump sigil past the zstd asm link failure
The pinned sigil revision globs zstd's decompress sources as C only, so
huf_decompress_amd64.S is never assembled while huf_decompress.c still calls
into it. Building the cffi bindings fails at the shared-object link:

  undefined reference to `HUF_decompress4X2_usingDTable_internal_fast_asm_loop'
  relocation R_X86_64_PC32 against undefined hidden symbol ... can not be
  used when making a shared object

argosy-sigil fixed this on main by selecting zstd's C fallback with
ZSTD_DISABLE_ASM. Bump the pin to pick it up.

Verified on linux/amd64 against 4b0f138: cmake build, shared-object link, the
cffi binding build under Python 3.14, and importing the module. sigil.extract
keeps the signature adapters/services/sigil.py calls.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NtfAbk9LrxmwFnBktP3NkY
2026-09-09 16:11:43 -05:00
Georges-Antoine AssiandClaude Opus 5 a44c3bbce9 docs(docker): drop the rationale from the DL4006 ignore
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-09 08:17:47 -04:00
Georges-Antoine AssiandClaude Opus 5 609f1bcb3d chore(dev): provision the browser emulator runtimes in the dev image
Fetch EmulatorJS, Ruffle, js-dos and FAKE-08 in the dev Dockerfile, on the
same pins and checksums as the emulator stage of docker/Dockerfile, into
/opt/romm/emulators. The ./frontend bind mount hides anything the build
writes under /app/frontend, so entrypoint.sh links each runtime into the
assets tree the dev server serves, leaving names held by checked-in art
alone. The runtime directories join the dev watcher's ignore list so
chokidar does not crawl them.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-08 21:27:04 -04:00
Georges-Antoine AssiandClaude Opus 5 a3aecdf045 chore(python): restore the C accelerators lost on 3.14
The runtime moved to 3.14 but the pinned versions did not, so packages
without cp314 wheels silently resolved to their pure-Python universal
wheels. Bump each to the first release that ships cp314 wheels, all
within the existing pyproject specs:

- SQLAlchemy 2.0.41 -> 2.0.52 (cyextension, and greenlet is back so
  sqlalchemy.ext.asyncio is functional rather than merely importable)
- multidict, propcache, frozenlist (aiohttp/yarl accelerators)
- wrapt 1.17.2 -> 1.17.3 (OpenTelemetry instrumentation proxies)
- coverage 7.9.2 -> 7.16.0 (CTracer, and CI runs the suite under --cov)
- mysql-connector-python 9.3.0 -> 9.7.0 (C extension protocol)
- debugpy 1.8.14 -> 1.8.21 (pydevd accelerators)
- pyinstrument 5.0.3 -> 5.1.3 (no longer compiled from sdist on every
  `uv sync --all-extras`, which broke the no-toolchain dev setup)
- anyio 4.9.0 -> 4.14.2, dropping a PEP 765 SyntaxWarning printed on
  every process start

Run pytest and the migrations workflow when the runtime or the lock
changes: nothing in the 3.14 bump lives under `backend/`, so neither
workflow triggered on it. `e2e.yml` already listed those paths.

Make `.python-version` the single home for the version: `uv python
install` and `actions/setup-python` both read it. In the dev Dockerfile
that also fixes the interpreter layer not being invalidated when the pin
changes, and the sigil install now derives site-packages from the venv.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-05 21:48:52 -04:00
Sam DornanandClaude Fable 5.1 3c547dfbd0 chore(python): move the backend to Python 3.14
Bumps the runtime everywhere it is pinned: requires-python, the
.python-version uv reads, both Dockerfiles (with the python:3.14-alpine3.23
and uv 3.14 image digests), the six workflows that install Python, and the
contributor docs.

Several locked C-extension versions had no cp314 wheels, so the Alpine
image fell back to building them from source and failed on the missing
C++ and Rust toolchains. Those packages are upgraded within their existing
pyproject ranges: pydantic 2.11.7 -> 2.13.5 (pydantic-core 2.46.5), uvloop
0.22.1, grpcio 1.83.1, httptools 0.8.0, inflate64 1.0.4, markupsafe 3.0.3,
pyyaml 6.0.3 and streaming-form-data 1.20.0. The lockfile also drops the
3.13-only resolution branches.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-05 08:57:10 -05:00
Georges-Antoine AssiandClaude Opus 5 97a16e2c79 refactor: tighten comments and tests on the sigil branch
Reverts the FileRow hash-chip loop. It was left over from the per-file
version chip that no longer exists, so it is unrelated to title ids.

Cuts the comments that restate the code or defend an obvious choice, and
the `Args:` block on `get_matching_missing_rom` that repeated its own
parameter names. The CNMT content-type rationale now lives only on
`CONTENT_TYPE_CATEGORIES`.

Tests exercise the production path: the title id regex is checked with
`fullmatch`, the way the caller uses it, and `test_identity` builds its
auth header the way every other endpoint test does.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-02 17:13:20 -04:00
Georges-Antoine AssiandClaude Opus 5 bcafc3d504 refactor(scan): tighten the sigil title id plumbing
Rename embedding now goes through the handler's own rename primitive
(`rename_fs_rom`), so it inherits the library-root path validation and
per-path locking every other write in that class gets, and builds the new
name with the repo's multi-part extension helpers.

Extraction no longer reads files it can never parse: archives, manuals,
soundtracks and screenshots are skipped, and on non-Switch platforms it
stops after the first result, since only Switch looks at a per-file
content type.

Also collapses the two rom-level branches (dropping a closure), gives the
Switch family one home in RomM's own terms instead of deriving it from
sigil's slug string, logs an unrecognized save target layout instead of
dropping it silently, deletes the `ScannedFSRomValues` mirror of `FSRom`,
drops a dead `rom.version` locale key, and slims the sigil Docker builds
to one layer with a filtered clone.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-02 16:34:36 -04:00
nendo a057c886ae feat(scan): extract PS3, Xbox and Xbox 360 title ids via sigil
sigil gained an Xbox extractor and learned to read PS3, PS Vita, Xbox 360
and Wii U out of their containers, so the three platforms it could
already name are now reachable from a scan. All three are non-hashable
or partly so, which is where a binary id earns the most: it is the only
identity those libraries carry. The extractors are flagged experimental
upstream.
2026-08-28 10:54:58 +09:00
nendo b9261cdea6 fix(scan): store 3DS, Wii and Wii U save ids in the emulator's case
save_id is the literal on-disk save location, and azahar, dolphin and
cemu all build those directories with lowercase hex. On case-sensitive
storage an uppercase id addressed a second directory beside the
emulator's own, so a game's saves split across the two. Sigil now emits
the path in the case the emulator writes; the pin moves with it, and
0111 recases the ids already stored so an existing library does not have
to be rescanned. title_id keeps its canonical uppercase form.
2026-08-26 19:27:15 +09:00
nendo 0cf06f2200 Bump sigil pin for 3DS z3ds extraction fix 2026-08-17 03:21:41 +09:00
nendo 0c88dfdfe0 Extract 3DS save location via sigil folder-split usage
The 3DS nests its save under two directory levels, so sigil reports it as
a '/'-separated save_id (00040000/00033500) flagged with usage=folder-split
rather than a flat id. Add SaveUsage.FOLDER_SPLIT, migrate the saveusage
enum (0106), regenerate the frontend SaveUsage type, and bump the sigil
pin to 98f3c62.
2026-07-30 15:27:28 +09:00
nendo 6042a1c56b Merge upstream master into feature/sigil-title-ids
Re-integrate the sigil title-id feature on top of upstream's scan
reconciliation refactor and migration additions.

Scan reconciliation:
- _identify_rom now persists rom files via db_rom_handler.sync_rom_files
  instead of purge_rom_files + per-file add_rom_file. The obsolete
  RomFile copy-list (title_id/save_id/title_version) is removed; those
  fields now ride along through ROM_FILE_SCANNED_COLUMNS.
- Add title_id, save_id, title_version to ROM_FILE_SCANNED_COLUMNS so
  sync_rom_files copies them onto reconciled rows. Without this the
  extracted ids/version silently drop to NULL on rescan.
- Single-file embedding rename (renamed_rom_fs_name) preserved in both
  the new-entry and update paths of _identify_rom.

Migrations renumbered to chain after the new upstream head:
- 0102_sigil_title_ids -> 0104_sigil_title_ids (down: 0103_roms_facets_provider_ids)
- 0103_sigil_title_version -> 0105_sigil_title_version (down: 0104_sigil_title_ids)

Tests:
- Reworked TestIdentifyRomPersistsFileTitleVersion to capture
  sync_rom_files and assert the scanned RomFile carries title_id and
  title_version; added a direct assertion that ROM_FILE_SCANNED_COLUMNS
  includes the sigil columns.
2026-07-24 07:20:10 +09:00
nendo fe0b4de169 Bump sigil pin for Switch CNMT content type and version 2026-07-24 05:20:09 +09:00
nendo c90260500e Bump sigil pin for PS2 folder-prefix save_id 2026-07-24 04:23:41 +09:00
nendo 8fa9bc53e4 Bump sigil pin for the Linux large-CHD fix 2026-07-24 03:45:00 +09:00
nendo 2f1d69f834 Build sigil into Docker images as an optional dependency
docker/Dockerfile gains a sigil-build stage (python:3.13-alpine base so
the cffi extension matches the venv's cp313/musl ABI) whose staged
package is copied into the slim image's site-packages; the Ubuntu dev
image builds it inline after uv sync. Pinned by commit ARG. The
backend treats the package as optional, so images built without it
simply skip extraction. DEVELOPER_SETUP documents the optional local
build.
2026-07-24 02:54:37 +09:00
Georges-Antoine AssiandClaude Opus 4.8 dc981f51c8 fix(archives): read RAR archives through bsdtar so they can be hashed
The bundled 7zz is built without the RAR codec (unRAR license), so it can
neither list nor extract RAR members. Every content-based match silently
came up empty for .rar: no RetroAchievements hash and no CRC/MD5/SHA-1 for
hash-database providers.

RAR listing and extraction now go through libarchive's bsdtar, added to the
images as libarchive-tools. Listing uses an mtree rewrite (libarchive's only
machine-readable listing) and member names are escaped before being handed
back to bsdtar, which matches them as globs.

Fixes #3884

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-07-22 10:13:50 -04:00
Georges-Antoine Assi 52163633e5 Merge branch 'master' into claude/server-side-rom-patching-EaQck 2026-06-23 07:01:07 -04:00
zurdi 8f879d0b59 chore: update Node.js version to 24.16 across workflows and Dockerfile, and enhance package.json with npm version specification 2026-05-26 09:23:47 +00:00
Georges-Antoine AssiandClaude Opus 4.6 1ee4682845 feat: drive frontend patcher via server-side endpoint, isolate Node deps
- Rewrite frontend/src/views/Patcher.vue around library ROM/patch pickers
  (v-autocomplete with debounced search) and call POST /api/roms/{id}/patch
  with responseType: blob; download and/or re-upload the patched output.
- Drop the rom-patcher npm dep and vite-plugin-static-copy from the frontend;
  remove the now-unused web worker, type decls, and viteStaticCopy plugin.
- Move the Node patcher into a self-contained npm project at
  backend/utils/rom_patcher/ (dir uses underscore so it's a valid Python
  package). Patcher.js loads RomPatcher.js from its sibling node_modules.
- Extract the heavy subprocess logic into backend/utils/rom_patcher/patcher.py
  (apply_patch, PatcherError, SUPPORTED_PATCH_EXTENSIONS); slim patch.py to
  HTTP/DB plumbing and fix a stale .parent.parent path.
- Wire up Docker: dev Dockerfile installs the new npm project; prod
  Dockerfile adds a backend-node-build stage and copies rom-patcher-js into
  the production image; install nodejs at runtime.
- Add new patcher i18n keys to en_US (other locales fall back).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-13 07:39:31 -04:00
Georges-Antoine Assi ec8583016b mega ton of fixes for 4.8 2026-04-03 10:54:31 -04:00
zurdi 56bc8c58dd feat: update RAHasher to version 1.8.3 and add Wii platform support 2026-03-20 13:24:19 +00:00
Georges-Antoine Assi 695a12a21a upgrade some base images 2026-02-16 22:11:47 -05:00
Georges-Antoine Assi d005dba90b fix a bunch of shit 2025-08-09 11:16:12 -04:00
Georges-Antoine Assi b247a5c5bb small tweaks 2025-08-07 17:39:57 -04:00
Georges-Antoine Assi 6b307afcd3 Run worker as a native process 2025-08-06 21:57:32 -04:00
Georges-Antoine AssiandMichael Manganiello 44131e6973 Apply suggestion from @adamantike
Co-authored-by: Michael Manganiello <[email protected]>
2025-07-04 21:25:27 -04:00
Georges-Antoine AssiandMichael Manganiello 80baaccf6d Apply suggestion from @adamantike
Co-authored-by: Michael Manganiello <[email protected]>
2025-07-04 21:25:09 -04:00
Georges-Antoine Assi 543238fb40 install uv to correct dir 2025-07-04 10:03:44 -04:00
Georges-Antoine Assi e8f8f1621c complete migration to uv 2025-07-04 09:54:04 -04:00
Michael Manganiello f96adeeaee misc: Upgrade to Python 3.13
Small changes to upgrade Python to version 3.13, and fixes based on a
`pyupgrade` run using the `--py313-plus` flag.
2025-07-03 23:37:00 -03:00
Michael Manganiello ab6ccc0999 misc: Upgrade RALibretro to v1.8.1
The previous attempt to upgrade RALibretro before a new tagged version
was available (#1970) needed to be reverted, as the RAHasher tool failed
with segmentation faults in some scenarios.
2025-06-28 13:10:37 -03:00
Georges-Antoine Assi 393c886a80 final tweaks 2025-06-19 09:54:00 -04:00
Georges-Antoine Assi cfe958f6d4 changes from copilot 2025-06-18 23:50:36 -04:00
Georges-Antoine Assi f4caf4c7da running app with HMR 2025-06-18 23:23:09 -04:00
Georges-Antoine Assi 55afea70ba more progress on docekrfile 2025-06-09 17:14:40 -04:00
Georges-Antoine Assi c5f2f59855 Start working on dev dockefile 2025-06-09 13:41:27 -04:00
zurdi zurdo 278195bfd5 roms backend functionality added 2023-03-12 03:20:42 +01:00
zurdi zurdo c473fed6d0 resources structure refactor 2023-03-11 12:37:39 +01:00