* fix(config): migrate retired filesystem folder keys instead of exiting
5.2 wrote filesystem.roms_folder and filesystem.firmware_folder into
config.yml on every settings save, so instances that never chose a layout
crash-looped on 5.3.1. Appliance installs (TrueNAS, Unraid, CasaOS) have
no way to edit the file.
Translate the keys at load time into the filesystem.structure templates
that reproduce the layout 5.2 resolved them to, using the same disk check
5.2 used to choose between `<folder>/{platform}` and `{platform}/<folder>`.
A declared structure template still wins. The migrated config is written
back when config.yml is writable; otherwise it applies in memory and logs
the replacement once.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(config): refuse folder names a template can't express before migrating
Addresses Greptile review on #5211: a retired folder name containing braces
produced an invalid template after config.yml had already been rewritten.
The migrated templates are now parsed first and startup exits with the
reason, leaving the file untouched. Also trims a test docstring.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(config): code-review fixes for the retired folder key migration
Keep an undetectable layout in memory, accept a '.' folder, honour a
differently cased declared key, stop retrying a failed write, and share
the layout detection with check_library_layout.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(config): simplify pass over the retired folder key migration
Build the templates in one pass with a parser per key, reuse get_nested
and the library fixture, skip the layout string on the in-memory path,
and memoize a detected layout across get_config() calls.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(config): review-polish, drop a docstring that restates the test name
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(config): migrate retired folder keys at startup and write config.yml atomically
Loading the config only translates the retired keys in memory now;
startup writes the migration once, before any worker runs, so processes
no longer race to rewrite config.yml.
config.yml is written to a sibling temp file and renamed over the
original, keeping a symlink and the file mode. When the rename fails
(a config.yml bind-mounted on its own), the in-place write puts the
previous content back if it fails, so a full disk no longer empties it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(config): drop the atomic config write and flatten the folder key migration
The temp-file rename, symlink/mode handling and in-place restore are a
separate concern from migrating the retired keys, so config.yml goes back
to the plain write. The translation validates its two templates directly
instead of through a table of parsers.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(streaming): let the backend pick the resume picker's states
The player decided which of a ROM's states the resume picker offers from
a core rule the config shipped, and so repeated the rules the claim
checks a pick against.
The player now asks `GET /api/streaming/states/{rom_id}` which states
load natively and which the broker can import, so the rules live only in
the backend. A native state whose filename carries no slot number is
offered only when the broker can import it, since it can't resume as is.
The player asks again only when the ROM's states, the emulator or the
import kinds change, keeps its last answer if a refresh fails, and holds
Play until the first answer for the ROM arrives, so a quick click can't
skip the state about to be preselected.
`StreamingContainerSchema.state_core` and `StateCoreSchema` are removed,
and the config no longer asks each broker for its core.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
* fix(streaming): address Greptile review on the state picks
Refetch the picks when a state is renamed, since the filename decides whether it resumes as it is, and fold the new docstrings to one sentence.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
---------
Co-authored-by: Zach Clendenen <[email protected]>
Co-authored-by: Claude Fable 5.1 <[email protected]>
* fix(streaming): keep a stream capture Cloud Sync serves from its slot
A stream capture filed in a core's folder is the state Cloud Sync serves
for its slot while it is the newest there. Pruning the stream's state
history past its limit would delete it, and a device holding it as that
slot would lose the slot.
The prune now skips a capture a slot serves, until a newer state takes
that slot or the capture's file is gone from disk. History can go over
the limit by at most one capture per slot.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
* docs(streaming): fold the prune_state_history docstring to one sentence
Greptile review on #5209.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
---------
Co-authored-by: Zach Clendenen <[email protected]>
Co-authored-by: Claude Fable 5.1 <[email protected]>
* fix(sync): file a screenshot sent ahead of its state past a stream capture
A Cloud Sync client sends a slot's screenshot before the state it belongs
to. With a stream capture serving the slot, the screenshot was filed under
the capture's name and wrote over the capture's own thumbnail.
The screenshot now goes to the state the slot's next upload writes to: the
slot's newest non-capture state, or the canonical name when the upload
will create one. A copy of the capture's own screenshot sent back is left
alone.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
* refactor(sync): address Greptile review on the stream capture checks
Move the capture copy checks for a state and a screenshot upload out of the endpoint into the sync handler, and fold the core_library_name, prune_state_history and Stream.vue comments to two lines.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
---------
Co-authored-by: Zach Clendenen <[email protected]>
Co-authored-by: Claude Fable 5.1 <[email protected]>
* fix(oidc): match any of the comma-separated groups in OIDC_ROLE_*
OIDC_ROLE_ADMIN/EDITOR/VIEWER were read as one raw string and checked
for membership in the roles claim, so a value like
"romm-admin,platform-admins" never matched and the login was rejected
with 403. Parse each variable into a set of stripped, non-empty values
and grant the role when it intersects the claim. A single-valued string
claim is matched whole rather than by substring.
Fixes#4302
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(oidc): reject a string role claim that only contains a mapped group
Addresses Greptile's review on #5203: the string-claim case only covered a
match, which the old substring check would also have passed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(config): parse every comma-separated env var with one helper
Add _get_env_list and build _get_env_set on it, with an optional fallback,
so ROMM_CORS_ALLOWED_ORIGINS and DEVICE_INSTALL_EXCLUDED_PLATFORM_SLUGS stop
hand-rolling the split/strip/drop-empty parse.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
RetroArch Cloud Sync now serves a ROM and core's newest autosave .srm
and hides an unslotted .srm behind it, so the upload dialog moving a
save picked with a core to "No slot" left it invisible to RetroArch once
RetroArch had synced that game. The dialog keeps the autosave default,
and the slot hint says RetroArch reads autosave.
Refs #5158
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(oidc): match any of the comma-separated groups in OIDC_ROLE_*
OIDC_ROLE_ADMIN/EDITOR/VIEWER were read as one raw string and checked
for membership in the roles claim, so a value like
"romm-admin,platform-admins" never matched and the login was rejected
with 403. Parse each variable into a set of stripped, non-empty values
and grant the role when it intersects the claim. A single-valued string
claim is matched whole rather than by substring.
Fixes#4302
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(oidc): reject a string role claim that only contains a mapped group
Addresses Greptile's review on #5203: the string-claim case only covered a
match, which the old substring check would also have passed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(saves): make a deduplicated slot upload the slot head
A slotted upload whose bytes match an older version in the slot returned
that version without touching its updated_at, so the slot's newest save
stayed one the device had moved away from. A client re-downloading the
slot head then restored the wrong save. Bump the matched version before
pruning so it becomes the head and survives the cap. A retry of the
current head is left alone so other devices don't see a spurious update.
Fixes#5200
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(saves): promote a deduplicated save in the handler, past same-second heads
Addresses Greptile review on #5204: the head promotion moves out of the
endpoint into db_save_handler.promote_to_slot_head, and the promoted
timestamp is at least a second past the current head's, so a head written
in the same second can no longer win the id tiebreak.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Reassociation matched a moved file to its missing entry by CRC, MD5 and
SHA1, and only consulted the binary title id when the hashes were absent.
Moving a GameCube iso into its own folder next to a dumpinfo.txt makes it
a folder ROM whose hash spans both files, so the hashes no longer matched,
the title id was never tried, and the scan created a new entry that showed
up under Recently added.
Fall back to the title id when the hashes match no missing entry. Hashes
still win when they match, and ambiguous hashes still return no match
rather than letting the title id break the tie.
Fixes#5046
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(security): check collection ownership in the ROM filters
The collection endpoints refuse a private collection to anyone but its
owner, but the ROM listing endpoints accepted the same id as a
collection_id or smart_collection_id filter without checking it. Any
user (or a kiosk visitor) could list, randomly pick from and bulk
download another user's private collections and Favourites, and re-share
one through a public smart collection scoped to it.
The membership filters now match only collections that are public or
owned by the caller. A collection_id inside smart collection criteria is
checked against the smart collection's owner, both when the criteria are
saved and when they are evaluated.
Reported by Takaya Sato (dagadjno).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(security): clear stale smart collection caches across owners
Addresses Greptile's review on #5201. Smart collections cache their
membership on the row, so one scoped to another user's private
collection kept serving the members it cached before the ownership
check. Migration 0152 empties those caches on upgrade, and sharing or
unsharing a collection now refreshes the smart collections scoped to
it. Also covers a second user listing a public smart collection scoped
to its owner's private collection.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(collections): harden smart collection scope refreshes (code-review)
Allow re-saving a smart collection whose stored scope was since lost,
reject non-object criteria in from_stored_criteria, guard the scoped
refresh, and refresh smart collections when a collection's ROMs are
replaced or the collection is deleted.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(collections): share the refresh wrapper and trim scope tests (simplify)
Move the scoped smart collection refresh next to the existing wrapper,
match the scope on the raw criteria before refreshing, batch the
migration update, and fold the duplicate unshare test into helpers.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* docs(migrations): describe the cleared smart scopes by current behavior (review-polish)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* docs(roms): state the collection filter comment plainly (humanizer)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): resolve RetroArch paths across emulators sharing a core folder
Several RomM emulator ids map to one RetroArch folder (mednafen_psx_hw,
beetle_psx_hw and the raw "Beetle PSX HW" all sync through
saves/Beetle PSX HW/). The manifest built paths from whichever id an
asset carried, but GET parsed the folder back to the canonical id and
matched it exactly, so an asset filed under another alias was advertised
and then 404'd. Uploads also started a second autosave history beside
the alias's.
The manifest now lists the newest asset per folder, and GET, PUT and
DELETE resolve a path across every alias. Rewrites of an existing
unslotted save or state stay under that row's own emulator.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): prefer on-disk alias saves and share newest-per-key grouping
pr-ready step 2 (code-review --fix).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(sync): one alias helper and a lineage computed once
pr-ready step 3 (simplify).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(sync): type the alias fixture's asset kind
pr-ready step 4 (review-polish).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(sync): isolate the asset dir per RetroArch sync test
The alias tests write different bytes to the same asset paths, and
user and ROM ids repeat across xdist worker databases, so parallel
runs raced on those files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): serve the newest on-disk RetroArch state in a slot
Missing states are dropped before slot grouping for the manifest and reads, so a newer missing state (or alias) no longer hides an older one still on disk. A PUT still matches the missing row and revives it in place.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): fall back past a RetroArch file that vanished unflagged
Addresses Greptile's review on #5193: a state whose file was deleted before
`missing_from_fs` was set still won its slot. The manifest now rebuilds when
hashing flags an asset, and GET flags a missing file and resolves again, so an
older asset at the path is listed and served.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): resolve RetroArch paths across emulators sharing a core folder
Several RomM emulator ids map to one RetroArch folder (mednafen_psx_hw,
beetle_psx_hw and the raw "Beetle PSX HW" all sync through
saves/Beetle PSX HW/). The manifest built paths from whichever id an
asset carried, but GET parsed the folder back to the canonical id and
matched it exactly, so an asset filed under another alias was advertised
and then 404'd. Uploads also started a second autosave history beside
the alias's.
The manifest now lists the newest asset per folder, and GET, PUT and
DELETE resolve a path across every alias. Rewrites of an existing
unslotted save or state stay under that row's own emulator.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): prefer on-disk alias saves and share newest-per-key grouping
pr-ready step 2 (code-review --fix).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(sync): one alias helper and a lineage computed once
pr-ready step 3 (simplify).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(sync): type the alias fixture's asset kind
pr-ready step 4 (review-polish).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(sync): isolate the asset dir per RetroArch sync test
The alias tests write different bytes to the same asset paths, and
user and ROM ids repeat across xdist worker databases, so parallel
runs raced on those files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* perf(sync): resolve RetroArch core aliases in one query
The saves handler now filters on a collection of lineages and looks up a file name across several folders, so the RetroArch alias resolvers issue one query each instead of one per alias.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* perf(sync): filter a RetroArch delete to autosave and unslotted saves in SQL
Addresses Greptile's review on #5192: saves_at_path loaded every slot of
the matching lineages and dropped the named slots in Python.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): resolve RetroArch paths across emulators sharing a core folder
Several RomM emulator ids map to one RetroArch folder (mednafen_psx_hw,
beetle_psx_hw and the raw "Beetle PSX HW" all sync through
saves/Beetle PSX HW/). The manifest built paths from whichever id an
asset carried, but GET parsed the folder back to the canonical id and
matched it exactly, so an asset filed under another alias was advertised
and then 404'd. Uploads also started a second autosave history beside
the alias's.
The manifest now lists the newest asset per folder, and GET, PUT and
DELETE resolve a path across every alias. Rewrites of an existing
unslotted save or state stay under that row's own emulator.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): prefer on-disk alias saves and share newest-per-key grouping
pr-ready step 2 (code-review --fix).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(sync): one alias helper and a lineage computed once
pr-ready step 3 (simplify).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(sync): type the alias fixture's asset kind
pr-ready step 4 (review-polish).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(sync): isolate the asset dir per RetroArch sync test
The alias tests write different bytes to the same asset paths, and
user and ROM ids repeat across xdist worker databases, so parallel
runs raced on those files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): delete every aliased state in a slot and prefer one on disk
Addresses Greptile review on #5191: a slot's state is picked live-first
across aliases, so a newer missing alias no longer hides an older one on
disk, and DELETE removes every state in the slot so an older alias can't
resurface at the same path.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(docker): boot when the container runs as a non-default UID
Unraid runs the container as 99:100, which can't write to the image's
/romm (owned by 1000, mode 755), only to the mounted subfolders. The
PSP pending dir at /romm/cache/retroarch_sync was created at import
time, so every gunicorn worker failed to boot.
- Buffer pending PSP save files under the user's assets folder
(assets/users/<user>/retroarch/psp_pending/), beside the other
RetroArch Cloud Sync files and outside the client-reachable blob
categories.
- Move the zip and rom-converto download caches to /tmp/romm/cache,
and point nginx's internal /cache/ alias there.
- Put gunicorn's control socket in /tmp/gunicorn instead of
$HOME/.gunicorn, since HOME is / for that UID.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(docker): honor ROMM_TMP_PATH for caches, explain an unwritable sync folder
- The download caches live under $ROMM_TMP_PATH/cache when it's set,
so installs with a small Docker disk (Unraid's docker.img) can point
them at a share. The entrypoint exports the same path for nginx's
/cache/ alias.
- With ENABLE_SYNC_FOLDER_WATCHER on and /romm/sync not writable, /init
warns and skips the watcher instead of dying on mkdir under errexit.
- The compose example lists the /romm/sync mount the sync features need.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(sync): cover reading back and clearing buffered PSP files
Addresses Greptile's review on #5189.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): apply code-review findings to PSP pending buffer
Remove the emptied psp_pending folder after a merge, define the sync
folder path once in init, update the architecture doc's PSP and blob
locations, and drop a stale .gitignore entry.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(sync): apply simplify findings to PSP pending cleanup
Clear the buffered folder with remove_directory, let the startup check
own creating the sync folder, and reword the test fixture docstring.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): cap buffered PSP files and drop them on delete
Buffered files for an unresolved save folder now count against the
bundle limits, so a folder can't fill the assets share and then fail
every merge once it resolves. A DELETE also removes the buffered copy,
so a deleted file isn't merged back when the folder resolves.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(docker): check the sync folder for push-pull sync too
ENABLE_SYNC_PUSH_PULL keeps SSH keys and device saves under /romm/sync
as well, so an unwritable folder now disables it with the watcher and
warns at boot instead of failing when the task runs.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(docker): strip whitespace from env vars like the backend does
The backend strips every variable it reads, while the init script and
nginx templates used raw values, so a padded ROMM_TMP_PATH sent nginx's
/cache/ alias somewhere the backend never writes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(config): drop the CACHE_BASE_PATH comment
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* docs(docker): say which backend config strips env whitespace
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* revert(docker): drop the entrypoint env whitespace trimming
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): apply pr-ready code-review findings
Gate push-pull on its SSH keys path rather than a writable sync folder,
accept the same truthy values as the backend, and clear only the merged
PSP pending files so another worker's buffered file survives.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor: apply pr-ready simplify findings
Share one truthy pattern across init's flag checks (the watcher gates
now accept the same values as the backend), disable gunicorn's unused
control socket instead of placing it, derive the converts redirect from
its own cache path, and factor the unknown PSP folder in tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* docs(docker): apply pr-ready humanizer pass to the sync mount note
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(fs): create handler base folders on first write, skip leftover temp files
FSHandler no longer mkdirs its base path when constructed, so a module-level
handler rooted at an unmounted or unwritable folder can't take down boot;
every write path already creates its parent folders. Blob listings skip
.romm_tmp_ files a crashed write left behind, so they are never merged into
a PSP bundle or advertised in the sync manifest. The prefix now lives in one
TEMP_FILE_PREFIX constant.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
A client that drains installs by polling claim (Argosy) never holds the
/devices socket, so the install dialog showed it offline while awake.
Each claim now sets a presence key that lapses after the same 90 seconds
as a socket's.
Also add TZ to env.template, since save version names and log
timestamps follow the container's time zone.
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(platforms): give platforms missing from IGDB a family
Platform families only came from the IGDB platform list, so platforms
IGDB doesn't list (win9x, win3x, Sega arcade boards, TI computers, ...)
landed in "Other" when grouping the platforms page by family.
Platforms IGDB doesn't list now either share the family and category of
an IGDB sibling (win9x -> Windows), or get an explicit family, which can
override the sibling's (Sega Model 2 -> Sega, Arcade). 108 of the 242
unlisted platforms get a family; the rest are engines, generic groups,
single-member makers or ambiguous.
Fixes#5178
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(platforms): satisfy mypy when applying a platform family
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(platforms): check every unlisted platform mapping resolves a family
Addresses Greptile review on #5187.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* perf(tasks): batch RetroAchievements progress sync reads
Add get_roms_by_ra_ids and get_rom_users_by_rom_ids (single index-backed
IN queries, no eager joins) and fold the per-user feed to {ra_id: status}
so each user costs 2 reads plus writes instead of 2 reads per game.
* fix(tasks): skip joined eager-loads in RA sync batch reads
* style(tasks): trim batch-read comments to house style
* fix(tasks): sync every ROM sharing an ra_id, skip empty rom set early
* fix(tasks): raiseload instead of deprecated noload, fix CI failures
* fix(tasks): apply code-review fixes to RA progress sync batching
pr-ready step 2 (code-review xhigh --fix): load only id/ra_id for the
batched ROM read, flatten the sync loop, isolate per-user status sync
failures, log the ROM id, add a real-row test, and move the batched
RomUser getter next to its siblings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(tasks): simplify RA progress sync batching and its tests
pr-ready step 3 (simplify): select (id, ra_id) columns instead of Rom
entities, drop early returns the getters already short-circuit, and
fold the mocked sync tests into shared helpers and one parametrized
status-transition test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* style(tasks): trim batched RomUser getter comment
pr-ready step 4 (review-polish).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* docs(tasks): tighten unknown-award test docstring
pr-ready step 5 (humanizer).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* perf(tasks): batch RA status writes and make add_rom_user race-safe
set_rom_user_statuses writes one user's statuses in a single
transaction: it inserts missing RomUser rows, then runs one conditional
UPDATE per distinct status. The RETIRED/NEVER_PLAYING guard lives in the
WHERE clause, so a status the user sets mid-sync is never overwritten,
and the per-user cache bumps are queued once.
add_rom_user now inserts inside a savepoint and returns the existing row
when a concurrent writer won the unique (rom_id, user_id) insert, so the
sync no longer needs a catch-all around each user.
A rolled-back savepoint fires after_rollback, which drops every queued
user cache bump, so bumps are queued only after the savepoints finish.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tasks): skip existing rom_user rows in the batch insert
Addresses Greptile review on #5179. The batch read existing rows before
inserting, so under MariaDB's REPEATABLE READ a row another writer
committed in between raised IntegrityError, and the fallback SELECT (or a
FOR UPDATE read, which MariaDB's snapshot isolation refuses) could not see
it. The insert now skips duplicates itself (ON CONFLICT DO NOTHING /
INSERT IGNORE) with no prior read. The race test commits the competing row
right before the INSERT instead of mocking the lookup. Also trims the
set_rom_user_statuses docstring to one sentence.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tasks): apply code-review fixes to batched RA status writes
Batch set_rom_user_statuses to stay under PostgreSQL's bind parameter
cap, keep queued cache bumps across a rolled-back savepoint, fold the
single-caller insert helper into add_rom_user, and fix stale comments.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(tasks): apply simplify pass to batched RA status writes
Group by status before batching to drop a loop level, name the
user-owned statuses, split the dialect insert out of the execute call,
and share a status lookup across the write tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tasks): type the dialect insert for mypy (review-polish)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Shreeniwas <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(sync): sync RetroArch Cloud Sync saves through save slots
RetroArch saves were unslotted and overwritten in place, so they never
reached the slot-based sync the native clients use, and a bad upload
replaced the only copy.
A save path now serves the newest slotted version for its ROM, core and
extension, from any slot, under RetroArch's own file name. An upload adds
a version to that slot (or autosave), skips unchanged bytes, and prunes to
MAX_SAVES_PER_SLOT. A delete removes every save the path covers, so none
resurfaces on the next sync. Unslotted saves keep serving until the first
slotted upload; PSP bundles and states are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): lock RetroArch save uploads and prune per core and extension
Addresses Greptile's review on #5170:
- Move the RetroArch save upload into sync_handler.store_save_version,
leaving the endpoint to validate and map the result to a status.
- Hold a Redis lock per (user, rom, core, extension) across the lookup,
write, scan and row update, so concurrent same-second uploads never file
two rows over one tagged file. A lock timeout answers 503.
- Prune only the uploaded path's core and extension in the slot, so
another core's or a .rtc's current save is never dropped.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): pr-ready code-review fixes for RetroArch save slots
- Scope an extensionless prune to dotless file names, since every name
ends in an empty extension.
- Treat only NULL slots as unslotted in the manifest, matching the query
it replaced and the PSP bundle lookup.
- Share one helper for the slot versions a RetroArch path covers.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(sync): pr-ready simplify pass for RetroArch save slots
- Scope a slot prune by save ids, so the versions a RetroArch path covers
are decided once in Python instead of again in SQL.
- One newest-per-key helper for states and save versions, and
resolve_save_by_slot reuses it.
- A canonical_save_file_name helper next to the state one.
- The upload builds its tagged name once, checks its length in the
handler, and finds a same-second version in the list it already loaded.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(sync): pr-ready review-polish pass
Drop a comment the canonical_save_file_name helper now says.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): map RetroArch saves onto autosave lineages with unique version names
- Only RetroArch's .srm goes through slots, as the autosave versions of
its ROM and core. Companion files such as .rtc stay single unslotted
saves, so a slot never offers a clock file as the save.
- RetroArch reads, writes and deletes in autosave only, leaving named
slots alone.
- Every slot prune, web uploads included, is scoped to the uploaded save's
lineage (emulator and extension), so one core's uploads never drop
another's only save.
- The manifest lists the newest version under the client's spelling of
the game when it still names the ROM.
- reserve_version_name moves a taken tag on a second and claims it in
Redis, so same-second, concurrent or repeated-hour uploads from either
endpoint never share a file. The RetroArch lock and its 503 go.
- The manifest reads only the newest version per lineage, through one
window query.
- An upload is hashed before it is written, so an unchanged re-send
writes nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(sync): seed the client-spelled save instead of uploading it
Uploading TEST_ROM.srm only resolves to the test_rom ROM under MariaDB's
case-insensitive collation, so the test 409'd on PostgreSQL. Seeding the
version directly still checks that the manifest keeps the client's
spelling of the game.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): resolve RetroArch saves to ROMs case-insensitively on PostgreSQL
Cloud Sync matches a save's game name to a ROM's fs_name_no_ext. MariaDB's
collation ignores case, but PostgreSQL compared exactly, so TEST_ROM.srm
never reached the test_rom ROM. PostgreSQL now compares lowercased names,
served by a new expression index (0150), and the endpoint test uploads the
differently-cased name again.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): address Greptile review on RetroArch save slots
- add_save derives the prune lineage from the tagged name it stores, so a
dotted upload such as game.rev.srm is counted against its own versions.
- Deleting a RetroArch .srm path also removes unslotted saves in that
lineage spelled in another case, which the manifest hid behind the
autosave head and PostgreSQL's exact match left behind.
- reserve_version_name looks up existing versions off the event loop.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* ci: bound apt network timeouts for Playwright's dependency install
`npx playwright install --with-deps` runs its own `apt-get update`, so the
`-o` flags from #5181 could not reach it, and a stalled Ubuntu mirror hung
the frontend test job until the 15 minute limit cancelled it. Move the
retry/timeout options into an apt.conf.d file written by a small composite
action, used by setup-backend and the frontend test job.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* ci: install Playwright's headless shell without apt
The apt.conf.d timeouts did not stop the hang: on a runner whose Azure
mirror is unreachable, apt falls back to archive.ubuntu.com and stalls
while downloading the package indexes, slowly enough that no timeout
fires. The ubuntu-latest image already ships Chrome's system libraries,
so drop --with-deps from the frontend and e2e jobs and keep apt out of
their path. Revert the bound-apt-timeouts action, and run the frontend
checks when frontend.yml itself changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* ci: install setup-backend's apt packages from a cache
The MariaDB connector .debs (plus each job's extra apt packages) were
fetched from an Ubuntu mirror on every backend job, so a stalled mirror
could hang any of them until the 15 minute limit, the same way it hung
Playwright's dependency install. Cache the downloaded .debs keyed on the
runner image OS and package list, saved from master only, and install
from them without apt-get update on a hit. A miss still downloads them
with the #5181 retry and timeout options.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* ci: install cached apt packages without re-downloading them
With package lists present, apt fetched the mirror's copy of each local
.deb instead of installing the file, so even the cached path reached the
mirror. Point apt's archives dir at the .deb directory and pass
--no-download. Also read ImageOS in a shell step: it is a runner shell
variable, not in the `env` context, so the cache key came out empty.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* ci: pin workflow runners to ubuntu-24.04
The apt .deb cache in setup-backend only holds packages built for one
Ubuntu release. Pin every job to ubuntu-24.04 so a GitHub move of
ubuntu-latest can't restore noble packages onto a newer image, and
hardcode the release in the cache key instead of reading ImageOS.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): keep RetroArch Cloud Sync from stalling the event loop
Basic auth ran its bcrypt check (~250ms) on the event loop for every
WebDAV request, and the manifest ran its queries there too, so parallel
RetroArch requests queued behind each other. Both now run in a thread.
Cloud Sync resolves every save and state to a ROM by fs_name_no_ext,
which no index led with; 0149 adds one so unmatched uploads fail fast.
The upload dialog now offers a core for saves as well as states, so a
manually uploaded save lands in the core folder RetroArch reads, and its
slot hint says only unslotted saves reach RetroArch.
Refs #5158
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(sync): cover manifest queries running off the event loop
Addresses Greptile review on #5171: trim the 0149 migration docstring to
its summary, annotate the new MockConnection constructor, and add
regression tests that build_manifest and list_manifest_paths query from a
worker thread.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(auth): mark Basic-auth users active off the event loop
pr-ready step 2 (code-review): run the user lookup, password check and
set_last_active in one worker-thread hop, and test that SaveDataTab offers
save cores and forwards the picked core on upload.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(v2): build SaveDataTab test saves and states from shared fixtures
pr-ready step 3 (simplify): replace the hand-cast UserSaveSchema and
UserStateSchema stubs with saveFixture/stateFixture plus toUserSave/toUserState.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(sync): drop restating index comment and type the test spy
pr-ready step 4 (review-polish): remove the comment that only restated
idx_roms_fs_name_no_ext, and type the manifest test's get_saves spy so mypy
passes across the backend.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(auth): check the password grant's credentials off the event loop
The OAuth password grant ran bcrypt (~250ms) on the event loop, stalling
every other request the worker was serving.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(sync): run RetroArch WebDAV database work off the event loop
GET, PUT, DELETE and PROPFIND resolved ROMs, looked up and wrote save,
state and screenshot rows, and listed whole platforms synchronously inside
async handlers; the PSP bundle helpers did the same. Each now runs that
work in a worker thread.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(saves): keep another core's same-named save on upload
A slot-less upload matched an existing save by file name alone, so
uploading game.srm for a second core rewrote the first core's row to the
new folder and deleted its file. The lookup now only matches a save made
with the requested core. Clients that send no core still match any, so
device conflict detection is unchanged, preferring a save filed under no
core.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(v2): file a save picked with a core out of the slots
RetroArch Cloud Sync only lists unslotted saves, so picking a core in the
upload dialog moves an untouched slot from autosave to "No slot", and
clearing the core moves it back. A slot the user picked is kept.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* revert(i18n): restore the original upload slot hint
Picking a core now files a save out of the slots, so the hint no longer
needs to explain RetroArch Cloud Sync.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
apt-get has no network timeout by default, so a stalled Ubuntu mirror
blocked a migrate-mariadb job in `apt-get update` until the 15 minute
job limit cancelled it. Retry up to 3 times with a 30s timeout instead.
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Add RomMulus, RetroVault, Gameflow Deck and RomMix, drop SwitchRomM
(no push in over 180 days), and point DeckyRommSync at its new name,
Tender. Regroup the list into Mobile and TV, Desktop, Handhelds and
Steam Deck, Sync, and Integrations.
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(v2): show a top-bar indicator while a "Download as" conversion runs
A format download that needs a conversion only flashed a toast, then left the
user waiting with nothing on screen until the browser download began. Pending
conversions now live in a store, and a "Converting" pill in the top bar lists
them until each download starts. The pill shares its look with the scan
indicator through a new NavStatusPill.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(v2): shorten converting indicator comments
Addresses Greptile review on #5157.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(v2): address code-review findings on the converting indicator
Give the Czech converting-downloads string the four plural forms the
cs_CZ rule picks from, use a count-agnostic Russian wording, wrap the
status pills in an element so their Transition animates, and tighten
the format conversions store and its tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(v2): simplify the nav status pills
Move the pill enter/leave transition into a NavStatusTransition wrapper
with scoped styles, show the conversion format with RTag, pass a null
scan progress for the indeterminate bar, and trim the format
conversions store to what downloadAs uses.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(v2): review-polish pass on the converting indicator
Drop CSS comments that restate their rules and use the object props
form in the indicator's component mocks.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* ci: check that PR titles follow Conventional Commits
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* ci: address Greptile review on the PR title check
Give the HLTB updater a Conventional Commits title so its PRs pass the
check, and strip line breaks from the title before echoing it so it
cannot start a workflow command.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
The config/, thumbnails/ and system/ blobs lived at /romm/retroarch_sync,
which no documented mount covers, so they sat in the image's anonymous
/romm volume and were lost on `compose down`/`up`, `-V`, or a
docker rm/run upgrade. RetroArch can then treat the missing files as
deleted on the server and drop its local copies.
They now live at assets/users/<user>/retroarch/<category>/, beside the
user's saves and states. The old path never shipped in a release, so
no data migration is needed.
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* chore(trunk): fix the scheduled trunk check-all failures
- Ignore hadolint DL3008 on the runtime stage apt-get, matching the
emulator-download stage
- Build alias-search test ROMs with tests.factories.make_rom instead of
the banned add_rom
- Drop a bandit B405 ignore that suppresses nothing
- Bump fast-uri to 3.1.8 for GHSA-hrr3-gc8f-f4qj
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tests): type the alias-search rom helper kwargs as Any
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
The Trunk launcher verifies the CLI tarball against cli.sha256 in
.trunk/trunk.yaml, but only when those hashes are present. Add the
published 1.25.0 hashes so every launcher path (trunk-action, the
Copilot setup workflow, dev-setup.sh) refuses a tampered download.
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat: searchable alternative titles for ROMs
Add an editable "Alternative titles" field to manual metadata, folded
into the search titles alongside provider aliases. IGDB localization
titles now join the ROM's alternative names, and the canonical IGDB
name is kept as one when a regional title is displayed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix: list hand-added titles in alternative_names, test the field
Address Greptile review on #5111: the response's alternative_names now
appends the hand-added titles to the first provider's, and the Alternative
titles field gets a component test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test: type the alternative titles field lookup
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix: code-review pass on alternative titles
Validate title lists from client-written metadata in one helper, list
hand-added titles first so the search text cap never drops them, and
let the titles combobox keep commas inside a title.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor: simplify pass on alternative titles
Dedupe IGDB titles once in build_igdb_rom, inline the manual title
source, and reword the combobox comma comment.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* docs: review-polish pass on alternative titles
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* docs: humanizer pass on alternative titles
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix: hand-set alternative titles override, providers join
A non-empty manual list replaces every provider's titles, in the list
and in search; otherwise every provider's titles are joined. IGDB and
Hasheous store every title they know, the canonical name and regional
localizations included, and the ROM's displayed name is left out only
when listing them.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix: validate manual metadata on ROM update
A malformed raw_manual_metadata is rejected with a 422 instead of being
stored and failing every later response for that ROM.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix: list each alternative title once however it is spaced
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(player): play RPG Maker 2000/2003 games in the browser with EasyRPG
Embed the EasyRPG web player for extracted game folders on the rpg-maker
platform. The backend generates the player's index.json from the scanned
files and merges in the free EasyRPG RTP under every name the official
and fan RTPs give its assets, so games that lean on the RTP find them.
In-game saves sync with the user's RomM saves through the player's
IndexedDB storage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(easyrpg): read the free RTP from a fixed path instead of an env var
The full image always keeps the RTP at /var/www/html/assets/easyrpg/rtp,
so the path is a constant beside the web path nginx serves it under. The
dev container links its copy there at startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(easyrpg): code-review fixes for the save sync and asset endpoint
- Wait for an in-flight push before the quit push, so the newest save uploads
- Keep Play disabled until the fresh rom (and its saves) is fetched
- Stop onPlay and the rom fetch from acting after the view unmounts
- Leave through usePlayerExit like the other players
- Handle a rejected IDB commit when the run itself fails
- Track uploads by the browser's file name
- Load only visibility and files per asset request, not the full rom
- Serve DEV_MODE files as octet-stream, matching nginx
- Make the traversal test actually reach the endpoint
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(easyrpg): simplify pass over the player and asset endpoint
- Load only file paths per asset request, and look up RTP folders in O(1)
- Reuse RomFile.file_name_for_download and usePlayerFullscreen
- Share one dev/prod serve helper for game and RTP files
- Derive the save sync's game key from the rom
- Have the RTP table tool reuse normalize_name
- Make save-module internals private and tidy the prepare/read loops
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(easyrpg): flag RPG Maker game folders on the rom schemas
Gallery lists carry no files, so the frontend guessed that every multi-file
rom on the rpg-maker platform was playable, including RPG Maker XP/VX/MV
folders. Rom now has a deferred `is_easyrpg_game` subquery (an RPG_RT.ldb at
the folder root), loaded with the other gallery file stats and derived from
loaded files on the with_files path, and the frontend reads it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(easyrpg): drop server save sync, keep saves per user in the browser
Server sync moves to a follow-up that puts the web players on device sync.
Saves now stay in the player's IndexedDB, as js-dos does, with the same note
on the start page. The player is launched as `<rom id>-<user id>` so each
RomM account sharing a browser profile keeps its own saves; nginx and Vite
drop the user id when routing to the backend.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(sync): let a client ask negotiate to restore the saves it lost
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(saves): sync web player saves as a browser device
A shared layer for the in-browser players: each browser profile registers
as a sync device, keeps every account's saves in its own IndexedDB store,
negotiates them with the server at launch (restoring any it lost), and
pushes changes with the device's conflict guard. Hashes match the server's,
including its zip-entry hash.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(emulatorjs): save as the browser's sync device, with the slot guard
EmulatorJS now uploads and downloads as this browser's registered device
instead of the IP-based login device, confirms downloads with their hash, and
lets a write that continues the slot's head be refused when another device
wrote the slot since; that progress is archived and the session keeps
writing to the archive.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(settings): list, rename, pause and remove your devices
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(player): sync EasyRPG, js-dos, PICO-8 and Ruffle saves with RomM
Each player's own storage is restored from the browser's synced copy before
boot and captured back during play and on leave:
- EasyRPG: its IDBFS saves, one slot per in-game slot
- js-dos: its changes zip through fsChanges pull/push, migrating saves it
kept in OPFS before
- PICO-8: cartdata files, flushed by loading a blank cart on leave
- Ruffle: the game's SharedObjects zipped into one save, kept out of
localStorage between plays so accounts don't share them
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(easyrpg): address Greptile review
- Start timing the play session once the player page loads
- Use object-form props in the view test's button mock
- Keep the RTP table tool's docstring to one sentence
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(saves): address Greptile review on web device save sync
- Remove EasyRPG save files that sync deleted, so the game cannot load them
- Keep the stale-device guard on device sync's page-close uploads
- Restore only the Ruffle SharedObjects the current SWF can reach
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(saves): code-review pass on web device save sync
Restore master's EasyRPG RTP table path lost in the merge, serialize
DeviceSaveSync pushes, only mark a no_op synced when the server hash
matches, forget the browser device when it is removed, close a late
EasyRPG IndexedDB connection, send content_hash on unload updates, and
share the IndexedDB helpers and slot-conflict check.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(saves): simplify pass on web device save sync
Archive conflicts through uploadArchivedSave and isSlotConflict, skip
hashing unchanged bytes, share bytesEqual from saveSync/hash, pass
restore_unlisted from DeviceSaveSync, use newest() for the EmulatorJS
slot head, and trim one-off Ruffle and PICO-8 helpers.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(saves): review-polish pass on web device save sync
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* docs(saves): humanizer pass on web device save sync comments
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(saves): open every player IndexedDB through openDb
The save sync store now rejects when another tab blocks its upgrade
instead of waiting forever, as the EasyRPG and pending-asset stores
already did.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(sync): let negotiate scope server saves to the client's emulators
A browser player's saves load only in that player, so a native save in
the same slot must not pair with it or download into it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(saves): register the browser device on each launch
Registration returns the existing device, so calling it every time
brings back a device removed from any browser, which EmulatorJS
otherwise kept sending as a dead id. The cached id now serves only when
offline. DeviceSaveSync also negotiates within its own emulator.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(player): share web player save sync through useDeviceSaveSync
The coalesced polled push, the flush before leaving, the page-close
capture and the discard prompt live in one composable, and the player
suites share one DeviceSaveSync fake from test-utils.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(saves): update one archive per slot after another device takes it
A slot refused mid-session archived every later write as a new save.
Later writes now update that archive in place, page-close sends
included, and the next launch settles the slot.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* perf(ruffle): skip unchanged polls and upload saves every 5 minutes
Polls compare the stored SharedObjects with the last read before zipping
them, and keep what changed in the browser, uploading it at most every
5 minutes. Leaving and page close still upload at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* perf(ruffle): sync saves like EmulatorJS, polling every second
Polls compare the raw stored strings, so an idle second decodes,
zips and hashes nothing. A change uploads once two polls agree on it,
and a failing upload backs off from 2 to 30 seconds through the retry
backoff EmulatorJS uses, now shared from saveSync. Leaving still
uploads at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(v2): scope stored preferences to the signed-in user
Preferences in localStorage were shared by everyone using the same
browser, so one person's gallery layout, scan providers, chosen cores
and so on carried over to the next account. useUserLocalStorage stores
them under `user:<id>:<key>` while someone is signed in, and under the
bare key while signed out. The first user to sign in after this change
claims any unscoped value, so existing preferences carry over once.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(v2): address Greptile review on user-scoped storage
- Scope the core, BIOS and disc keys the shared EmulatorJS player writes
(and v1's matching reads), so a game start no longer leaves bare keys
for the next user to claim.
- Build the test user with userFixture.
- Trim the helper's header comment.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(v2): store the jukebox session under the user's storage key
The resume session now goes through userStorage like the other stored
preferences, so each user gets their own key instead of one shared key
carrying a userId to check on restore.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(v2): lint against storage that isn't scoped to the user
Adds romm/no-unscoped-local-storage, which rejects raw localStorage and
vueuse's useLocalStorage/useStorage outside frozen v1, so new stored
preferences go through useUserLocalStorage/userStorage. The locale read
at module load stays unscoped with an inline disable.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(v2): resume jukebox music after a reload
An opt-in "Resume music after reload" setting (Settings > User Interface >
Jukebox) saves the soundtrack player's track, queue, shuffle state and
position on the device and restores them on the next load.
Browsers block audio before the first user interaction, so a restored
session loads paused at its position and, if it was playing, starts on the
first click, tap or key press. The session is keyed to the signed-in user,
and a restored track that no longer loads is dropped without an error toast.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(v2): address Greptile review on jukebox resume
- Reset the soundtrack store on sign-out and restore on every sign-in,
so the next user neither inherits the queue nor saves it as theirs.
- Start a resumed session after the first press is handled, so a first
press on Play no longer pauses it, and keep waiting while a game
blocks the music instead of consuming the press.
- Validate the saved session's meta and queue entries before restoring.
- Build the test user with userFixture.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(v2): code-review fixes for jukebox resume
- Re-check the game block when the deferred start runs, and re-arm the
first-interaction listener when the press is blocked or play() is refused
- Disarm the listener when playback starts some other way, on track change
and on unmount, including its pending timeout
- Validate isShuffled, activePlaylistRomId and wasPlaying in stored sessions
- Skip deep watching the stored session and only watch it while the
setting is on
- Make the smAndDown test mock reactive
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(v2): simplify pass on jukebox resume
- Keep a single pendingResume in the store in place of the one-shot
takePendingResume, awaitingAutoplay and the component's resumedTrack
- Leave the first-interaction listener armed until the sink plays instead
of re-arming it by recursion
- Read and write the saved session with localStorage directly, so other
tabs don't re-parse the queue on every save
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(v2): review-polish comments on jukebox resume
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(v2): humanizer pass on jukebox resume comments
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(v2): don't autoplay a restored jukebox session another tab is playing
A new tab picked up the saved session of a tab still playing and started
the same track a second time on the first click. On restore, the tab now
asks the others on a BroadcastChannel whether one is playing and waits
200ms for a reply. A playing tab answers, so the session loads paused;
with no answer (a reload, or every tab closed) it starts on the first
interaction as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(v2): only save the jukebox session a tab changed itself
Every open tab wrote its own session on pagehide and after a restore, so
closing an idle tab, or a tab that restored the session paused, replaced
the session another tab was playing. A tab now saves only after its own
player changed, and not while a restored session is waiting to start.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(v2): fade in a resumed jukebox session
A session restored after a reload now ramps from silence to the user's
volume over a second instead of starting at full volume.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(player): play RPG Maker 2000/2003 games in the browser with EasyRPG
Embed the EasyRPG web player for extracted game folders on the rpg-maker
platform. The backend generates the player's index.json from the scanned
files and merges in the free EasyRPG RTP under every name the official
and fan RTPs give its assets, so games that lean on the RTP find them.
In-game saves sync with the user's RomM saves through the player's
IndexedDB storage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(easyrpg): read the free RTP from a fixed path instead of an env var
The full image always keeps the RTP at /var/www/html/assets/easyrpg/rtp,
so the path is a constant beside the web path nginx serves it under. The
dev container links its copy there at startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(easyrpg): code-review fixes for the save sync and asset endpoint
- Wait for an in-flight push before the quit push, so the newest save uploads
- Keep Play disabled until the fresh rom (and its saves) is fetched
- Stop onPlay and the rom fetch from acting after the view unmounts
- Leave through usePlayerExit like the other players
- Handle a rejected IDB commit when the run itself fails
- Track uploads by the browser's file name
- Load only visibility and files per asset request, not the full rom
- Serve DEV_MODE files as octet-stream, matching nginx
- Make the traversal test actually reach the endpoint
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(easyrpg): simplify pass over the player and asset endpoint
- Load only file paths per asset request, and look up RTP folders in O(1)
- Reuse RomFile.file_name_for_download and usePlayerFullscreen
- Share one dev/prod serve helper for game and RTP files
- Derive the save sync's game key from the rom
- Have the RTP table tool reuse normalize_name
- Make save-module internals private and tidy the prepare/read loops
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(easyrpg): flag RPG Maker game folders on the rom schemas
Gallery lists carry no files, so the frontend guessed that every multi-file
rom on the rpg-maker platform was playable, including RPG Maker XP/VX/MV
folders. Rom now has a deferred `is_easyrpg_game` subquery (an RPG_RT.ldb at
the folder root), loaded with the other gallery file stats and derived from
loaded files on the with_files path, and the frontend reads it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(easyrpg): drop server save sync, keep saves per user in the browser
Server sync moves to a follow-up that puts the web players on device sync.
Saves now stay in the player's IndexedDB, as js-dos does, with the same note
on the start page. The player is launched as `<rom id>-<user id>` so each
RomM account sharing a browser profile keeps its own saves; nginx and Vite
drop the user id when routing to the backend.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(easyrpg): address Greptile review
- Start timing the play session once the player page loads
- Use object-form props in the view test's button mock
- Keep the RTP table tool's docstring to one sentence
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(easyrpg): keep the RTP table in a fixtures folder
Matches handler/metadata/fixtures, where the other bundled data files live.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* test(db): give worktree runs their own temporary test database
Checkouts share romm_test, so a worktree on another branch can migrate it
to a revision the current branch lacks, and every test then errors in
setup with "Can't locate revision". ROMM_TEST_DB_TAG gives a run its own
romm_test_<tag> databases (one per xdist worker) that are dropped when
the session ends. AGENTS.md tells worktree sessions to use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(db): address Greptile review on temporary test databases
- Prefix tagged databases with tmp_ so they can never share a name with
an untagged xdist worker database that teardown would then drop.
- Normalize the tag to [A-Za-z0-9_] so tags like my-worktree work.
- Drop the temporary database even when the migration fails.
- Shorten the new comment to two lines.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* ci(e2e): copy frontend/assets into the preview build
The CI e2e run serves `vite build` output through `vite preview`, but the
build leaves out frontend/assets, which only the Docker image copies on top
of dist. Platform icons, logos, fonts and default covers came back as the
SPA fallback HTML. Copy the folder into dist the same way the image does.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* Address Greptile review: shorten the asset-copy comment
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(scan): keep an uploaded manual when the rom had no scraped one
Update, unmatched and hashes scans pin a stored manual url, but a manual
uploaded to a rom without a scraped url has nothing to pin. The provider's
url then won, differed from the empty stored one, and its download
overwrote the upload at the shared manual path. A locked manual, or one
on disk with no source url, now keeps its empty url through the scan.
Fixes#4101
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(scan): drop a scraped manual's path on unmatch, test the download stage
Addresses Greptile's review on #5030:
- Unmatch cleared url_manual but kept path_manual, so the next scan read the
old scraped manual as an upload and never fetched the new match's one. It
now clears path_manual too, unless url_manual is locked (an upload).
- Trim the url_manual comment to two lines.
- Carry the scan result through download_rom_resources in the upload test and
check the manual is never stored over.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(scan): refresh screenshots when only their order changes
Stored screenshots are named by position (0.jpg, 1.jpg, ...), but the
re-download check compared the provider's urls as an unordered set, so a
reorder was never picked up. Compare the lists directly, in both the scan
download step and update_rom.
Fixes#4103
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(scan): cover the screenshot reorder in update_rom
Address Greptile review on #5029: add an endpoint test for update_rom's
own reorder check, and narrow await_args before reading its kwargs.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(scan): drop the old screenshot when its replacement fails
With overwrite on, a failed download left the previous image at that
index. The next scan saw matching urls and a file on disk, so it kept
the stale image for good. Remove it instead, so the next scan refetches.
Addresses Greptile review on #5029.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(converto): read per-file title ids with rom-converto during scans
The title-id half of #5002: rom-converto's info runs fill rom_files.title_id
and title_version, sigil fills what rom-converto can't read and keeps owning
save targets, and roms in a scan batch share info runs of up to 64 files.
A converto_read_at marker lets quick scans backfill files rom-converto has
not read yet. converto.scan_metadata (config.yml only) turns it off.
The other per-file fields and embedded images stay out until a UI reads them.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): name the same-content check that keeps unread title ids
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): queue unread files in one pass over the built rows
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(converto): address Greptile review on #5031
- A file refresh now persists converto_read_at when rom-converto's read
confirms a reused row's stored title id, so the file isn't re-read on
every refresh.
- RomConvertoService.read_infos splits its paths into info runs of at
most INFO_BATCH_SIZE, so a rom with many files can't send them all
through one run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(converto): apply code-review fixes
- A stored file title id counts as rom-converto's only once rom-converto
has read the row, so a stale sigil id can't override a fresh extraction.
- Fresh hashes decide whether unread title ids still describe a file,
ahead of a size and mtime a copy can preserve.
- One rom's failed candidate listing no longer aborts its scan batch.
- The info paths file's cleanup can't raise out of read_infos.
- _rom_file_order sorts by Path like _TitleIdSource.order.
- Restored the in-place reconcile test for _identify_rom.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): apply simplify pass
- sync_rom_files reads 'rom-converto read this pass' off converto_read_at
instead of an inspected set threaded through ParsedRomFiles, FSRom, the
scan socket and the refresh.
- One _converto_may_read check serves the scan and the batch candidates.
- converto_candidates reuses _list_rom_dir, validates paths off the event
loop and is best-effort itself, so the scan socket gathers it plainly.
- get_rom_files builds rom-converto paths from the validated rom dir
instead of validating each file again.
- _rom_level_identity returns early on sigil's extraction.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(converto): apply review-polish pass
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): drop the shared info batch
Sharing rom-converto info runs across a scan batch saved about 2 ms per
rom (one process start) in exchange for a second directory walk per
rebuilt rom, the RomConvertoInfoBatch bookkeeping and the scan socket
wiring. Each rom now reads its own files; read_infos still caps a run
at INFO_RUN_MAX_FILES.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): name the rom-converto columns inline
Drops ROM_FILE_CONVERTO_COLUMNS and ROM_FILE_CONVERTO_KEPT_COLUMNS, so
_copy_scanned_columns keeps master's frozenset keep_when_unset.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(streaming): release a failed claim from one place
claim_session released a failed claim at each refusal inside its helpers
and re-raised any HTTPException untouched from its catch-all, trusting the
raiser to have released it. A new HTTPException that skipped its own
abort_claim would have left the container reserved until the claim went
stale.
The helpers now only raise. A _ClaimUndo carries the session and the id
of any blank card _settle_memory_card creates, and the catch-all calls
abort_claim once for every failure, HTTPException included. The probe and
_hydrate_saves no longer take the session or blank id, which they only
needed to abort.
Closes#5004
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* Trim the _settle_memory_card docstring per Greptile review
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
At 11px, detailed flag emoji like the US flag collapse into unreadable
glyphs on Firefox/Windows (Twemoji). 14px keeps them recognizable while
the chip stays compact.
Fixes#4952
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(converto): integrate rom-converto for title id extraction and download conversion
* feat(converto): add conversion settings UI with per-platform format policies
* fix(converto): read conversion targets from the backend and translate the settings page
* fix(docker): ship rom-converto in the production image
Only the dev Dockerfile installed the binary, so the integration could
never enable on a released image. Fetch the same pinned, checksummed
musl build in a dedicated stage and copy it next to RAHasher.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(converto): skip the update check and batch info calls
The CLI asks api.github.com for a newer release on every run unless
told not to, so pass --no-update-check on each invocation. Inspect a
rom's files in one `info --paths-file` run instead of one subprocess
per file, and read the inspectable extensions from the capabilities
manifest so scans skip files info cannot recognize.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(scan): keep sigil authoritative for title ids
rom-converto ran first and overrode the ids sigil read. Sigil now runs
first and records its per-file ids; rom-converto only fills files sigil
left unset, on new or changed files, in one batch. Restore the base-id
selection for sigil extractions on every platform, as on master.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(downloads): make conversion opt-in and bound the cache
The download endpoint returned a converted file to every client once a
platform had a target, including the browser player and apps that check
hashes against RomFile. Conversion now needs ?converted=true.
A pre-warmed copy is served whatever its size; the in-request size cap
only gates starting a conversion. ROM_CONVERTO_CACHE_MAX_SIZE_GB
(default 20, 0 unbounded) caps the cache: conversions and the pre-warm
task stop when it is full, and cleanup evicts the least recently served
copies.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(types): regenerate frontend types for the converto schema
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(converto): address code-review findings (pr-ready step 2)
- Fall back to rom-converto's stored ids only when it ran this pass, in
disc order, so a rescan that skipped extraction keeps sigil's save target
- Persist per-file title ids that extraction writes onto reused rows
- Only authenticated callers may start an in-request conversion; past a
240s deadline serve the original and let the conversion finish into the
cache instead of nginx returning 504
- Treat a probe OSError or timeout as a failed probe; kill the subprocess
when the caller is cancelled
- Give the pre-warm task the scan timeout and report progress in the
ConversionStats shape the task UI reads
- Reuse zip_cache constants, walk the cache off the event loop, tolerate
files vanishing mid-cleanup, and trim long comments
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): simplify conversion code (pr-ready step 3)
- Move the converted-download policy (target lookup, cached-first, sync
size cap, deadline) from the endpoint into utils/conversion_cache and
share conversion_target with the pre-warm task
- Share one path lookup between get_cached_converted and get_or_convert;
the pre-warm walks the cache once and tracks a running byte budget
instead of walking it twice per rom
- One normalize_platform_formats beside TARGETS_BY_PLATFORM for the
config loader (via _validated_platform_map) and the settings endpoint;
update_converto_settings takes a ConvertoConfig and is written back
with dataclasses.asdict
- read_infos owns its never-raise contract, so the scan drops its blanket
except; drop the unused RomConvertoInfo.kind and error attributes
- Sort files for the rom-converto identity only when it is used; stat
each cached file once during cleanup
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(converto): trim comments (pr-ready step 4)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): convert on first download instead of pre-warming
A library pre-warm fought the TTL: cleanup dropped every copy nobody
downloaded within a day. Remove the Convert library task and convert on
demand only. Files over ROM_CONVERTO_MAX_SYNC_SIZE_MB now start their
conversion in the background while the original is served, so the next
download within the TTL gets the converted copy at any size.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): drop decrypt and encrypt download targets
Downloads now only change a rom's container or compression (CHD, RVZ,
CSO, NSZ, ...); serving decrypted files is left for a separate decision.
NDS and PS3 have no remaining targets. Also note that the conversion
concurrency limit applies per process.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): move the cache limits into converto.*
cache_max_size_gb and max_sync_size_mb join cache_ttl_hours in
config.yml and PUT /api/config/converto_settings, replacing the
ROM_CONVERTO_CACHE_MAX_SIZE_GB and ROM_CONVERTO_MAX_SYNC_SIZE_MB env
vars, so every eviction knob is edited in one place at runtime. The
loader takes its defaults from ConvertoConfig and one table holds each
integer's minimum for both the loader and the payload.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): lock conversions with flock instead of a sentinel
A conversion now holds a non-blocking flock on its key dir, which the
kernel releases if the worker dies. That replaces the .partial sentinel,
its 6-hour staleness guess (which a long ROM_CONVERTO_TIMEOUT could
outlive), and the rename-aside takeover. Cleanup skips locked dirs, and
partial output gets a hidden name so it never counts as a served copy.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(converto): parametrize repeated test cases
Collapse the per-platform info parsing, operation resolution and
converto_active cases into parametrized tests, so a new platform or
target is one row.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(scan): run rom-converto first with sigil as the fallback
rom-converto reads each rom's new or changed files first, in one batch,
and its title ids win. sigil still reads what it can, since it alone
provides save targets and Switch content types, and fills the ids
rom-converto left unset.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(converto): edit the cache size limits on the settings page
* feat(converto): store per-file metadata read by rom-converto during scans
* fix(converto): show the labels of the cache settings fields
* fix(scan): keep the stored save target when sigil reads nothing
* fix(converto): read Xbox 360 disc ids and Wii title versions, store content type as text
* feat(converto): save file icons, banners and backgrounds read by rom-converto
* chore(converto): bump rom-converto to v0.23.0
* fix(converto): address review findings in the conversion cache, scan metadata and settings
* refactor(converto): move per-file metadata out of the conversion PR
Scan-time metadata (title ids, the 17 rom_files columns, artwork and
converto.scan_metadata) moves to its own PR so its columns land with the
UI that reads them. This PR keeps conversion only.
The conversion cache no longer has uncompressed_size_bytes to budget
with, so a conversion to an uncompressed target reserves a fixed
multiple of its input and the cleanup pass after it evicts any overshoot.
Co-Authored-By: DevYukine <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ
* feat(converto): let the client pick the download format with ?format=
`?format=zso,iso` lists the formats a client can read. The stored file is
served whenever its format is listed, even with conversion off, so most
emulator clients never trigger a conversion. Otherwise the first listed
format with a cached copy is served, then the first one rom-converto can
produce.
A conversion that outlasts a short wait answers 202 with Retry-After
instead of falling back to the original, so a URL never serves two
different files and a client never gets bytes it can't read. 406 means
no listed format can be served. HEAD reports the same without starting
a conversion. This replaces `?converted=true`, the per-platform download
target and `max_sync_size_mb`; `cache_ttl_hours` stays config.yml-only.
Co-Authored-By: DevYukine <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ
* feat(converto): convert the library in place with a manual task
`converto.platform_formats` now names the format a platform's library is
stored in, and the new "Convert library" task brings every matched rom
to it. Each output is staged in a `.romm_tmp_` directory beside the
original, moved into place, and only then are the originals (and a cue's
tracks) deleted. Single-file roms take the new name, folder roms keep
theirs and get their .m3u entries rewritten, and the file rows refresh,
so a rom keeps its id, saves and collections.
Only lossless conversions qualify, so xbox (xiso trims the dump) has no
library format. Unmatched roms are skipped because a converted file no
longer hash-matches a DAT. `/api/config` now carries the library targets
and, per input extension, the formats a download can be converted to.
Co-Authored-By: DevYukine <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ
* feat(converto): library format settings and "Download as" on the game page
The settings page follows the reworked backend: per-platform formats are
now the library format the "Convert library" task stores matched games
in ("Leave as is" keeps a platform untouched), the scan metadata, cache
TTL and max-sync fields are gone, and a "Convert library now" button
starts the task behind a typed confirmation, since it deletes the
originals.
The game's More menu offers "Download as <format>" for a single-file rom
when download conversion is on, from the per-extension table
`/api/config` exposes. A one-byte ranged GET asks for the format, the UI
polls through 202s while it converts, then downloads it; a 406 shows an
error.
Co-Authored-By: DevYukine <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ
* fix(converto): address code-review findings (pr-ready step 2)
Guard the conversion settings route, add it to the user menu, ignore a
non-positive Retry-After, drop duplicate Download as clicks, name the game
in the persisted error, and tidy the settings page (display-name sort,
useUnloadGuard, decimal cache sizes, save snapshot).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(settings): share the config alerts and save bar (pr-ready step 3)
Extract ConfigFileAlerts and SettingsSaveBar from the three settings
views that copied them, move ScanSettings onto useUnloadGuard, align the
retry slot, and flatten downloadFormats, getDownloadFormats and the
format poll loop.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(converto): trim comments (pr-ready step 4)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(converto): guard destructive tasks and serve download formats per rom
- TaskSpec gains destructive and single_instance. The Tasks list asks for a
typed confirmation before a destructive task, and POST /tasks/run answers
409 while a single-instance task is queued or running.
- convert_library is both, and only runnable with ROM_CONVERTO_ENABLED.
- The heartbeat reports CONVERTO.ENABLED, which hides Conversion settings
from the sidebar and user menu and warns on the page.
- DetailedRomSchema.download_formats lists what the caller can convert the
rom's single file to, resolved by the backend's own operation table and
empty for visitors who can't start a conversion. It replaces
CONVERTO_DOWNLOAD_FORMATS in /api/config and the frontend's copy of the
extension matching.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Yukine <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: DevYukine <[email protected]>
* fix(converto): refuse cue sheets that reach outside their folder
rom-converto joins each cue FILE name onto the cue's folder without
rejecting `..` or absolute paths, so a crafted cue made `chd compress`
and `cue to-iso` pack any file the backend can read into the output.
The library task already refused such cues; the on-demand download
path did not. The check now lives in RomConvertoService.convert and
mirrors how the CLI reads FILE lines (first to last quote).
Also run the CLI from an empty private dir with an empty --config and
--no-cache, so it no longer picks up .env, rom-converto.toml or its
hash cache from the backend's cwd or home.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(converto): remove a sandbox made after its run was cancelled
Addresses Greptile review on #5008: a run cancelled while the sandbox
thread was still creating the dir never reached the cleanup, leaving
the dir behind.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(converto): integrate rom-converto for title id extraction and download conversion
* fix(docker): ship rom-converto in the production image
Only the dev Dockerfile installed the binary, so the integration could
never enable on a released image. Fetch the same pinned, checksummed
musl build in a dedicated stage and copy it next to RAHasher.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(converto): skip the update check and batch info calls
The CLI asks api.github.com for a newer release on every run unless
told not to, so pass --no-update-check on each invocation. Inspect a
rom's files in one `info --paths-file` run instead of one subprocess
per file, and read the inspectable extensions from the capabilities
manifest so scans skip files info cannot recognize.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(scan): keep sigil authoritative for title ids
rom-converto ran first and overrode the ids sigil read. Sigil now runs
first and records its per-file ids; rom-converto only fills files sigil
left unset, on new or changed files, in one batch. Restore the base-id
selection for sigil extractions on every platform, as on master.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(downloads): make conversion opt-in and bound the cache
The download endpoint returned a converted file to every client once a
platform had a target, including the browser player and apps that check
hashes against RomFile. Conversion now needs ?converted=true.
A pre-warmed copy is served whatever its size; the in-request size cap
only gates starting a conversion. ROM_CONVERTO_CACHE_MAX_SIZE_GB
(default 20, 0 unbounded) caps the cache: conversions and the pre-warm
task stop when it is full, and cleanup evicts the least recently served
copies.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(types): regenerate frontend types for the converto schema
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(converto): address code-review findings (pr-ready step 2)
- Fall back to rom-converto's stored ids only when it ran this pass, in
disc order, so a rescan that skipped extraction keeps sigil's save target
- Persist per-file title ids that extraction writes onto reused rows
- Only authenticated callers may start an in-request conversion; past a
240s deadline serve the original and let the conversion finish into the
cache instead of nginx returning 504
- Treat a probe OSError or timeout as a failed probe; kill the subprocess
when the caller is cancelled
- Give the pre-warm task the scan timeout and report progress in the
ConversionStats shape the task UI reads
- Reuse zip_cache constants, walk the cache off the event loop, tolerate
files vanishing mid-cleanup, and trim long comments
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): simplify conversion code (pr-ready step 3)
- Move the converted-download policy (target lookup, cached-first, sync
size cap, deadline) from the endpoint into utils/conversion_cache and
share conversion_target with the pre-warm task
- Share one path lookup between get_cached_converted and get_or_convert;
the pre-warm walks the cache once and tracks a running byte budget
instead of walking it twice per rom
- One normalize_platform_formats beside TARGETS_BY_PLATFORM for the
config loader (via _validated_platform_map) and the settings endpoint;
update_converto_settings takes a ConvertoConfig and is written back
with dataclasses.asdict
- read_infos owns its never-raise contract, so the scan drops its blanket
except; drop the unused RomConvertoInfo.kind and error attributes
- Sort files for the rom-converto identity only when it is used; stat
each cached file once during cleanup
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(converto): trim comments (pr-ready step 4)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): convert on first download instead of pre-warming
A library pre-warm fought the TTL: cleanup dropped every copy nobody
downloaded within a day. Remove the Convert library task and convert on
demand only. Files over ROM_CONVERTO_MAX_SYNC_SIZE_MB now start their
conversion in the background while the original is served, so the next
download within the TTL gets the converted copy at any size.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): drop decrypt and encrypt download targets
Downloads now only change a rom's container or compression (CHD, RVZ,
CSO, NSZ, ...); serving decrypted files is left for a separate decision.
NDS and PS3 have no remaining targets. Also note that the conversion
concurrency limit applies per process.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): move the cache limits into converto.*
cache_max_size_gb and max_sync_size_mb join cache_ttl_hours in
config.yml and PUT /api/config/converto_settings, replacing the
ROM_CONVERTO_CACHE_MAX_SIZE_GB and ROM_CONVERTO_MAX_SYNC_SIZE_MB env
vars, so every eviction knob is edited in one place at runtime. The
loader takes its defaults from ConvertoConfig and one table holds each
integer's minimum for both the loader and the payload.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): lock conversions with flock instead of a sentinel
A conversion now holds a non-blocking flock on its key dir, which the
kernel releases if the worker dies. That replaces the .partial sentinel,
its 6-hour staleness guess (which a long ROM_CONVERTO_TIMEOUT could
outlive), and the rename-aside takeover. Cleanup skips locked dirs, and
partial output gets a hidden name so it never counts as a served copy.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(converto): parametrize repeated test cases
Collapse the per-platform info parsing, operation resolution and
converto_active cases into parametrized tests, so a new platform or
target is one row.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(scan): run rom-converto first with sigil as the fallback
rom-converto reads each rom's new or changed files first, in one batch,
and its title ids win. sigil still reads what it can, since it alone
provides save targets and Switch content types, and fills the ids
rom-converto left unset.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(converto): store per-file metadata read by rom-converto during scans
* fix(scan): keep the stored save target when sigil reads nothing
* fix(converto): read Xbox 360 disc ids and Wii title versions, store content type as text
* feat(converto): save file icons, banners and backgrounds read by rom-converto
* chore(converto): bump rom-converto to v0.23.0
* fix(converto): address review findings in the conversion cache, scan metadata and settings
* refactor(converto): move per-file metadata out of the conversion PR
Scan-time metadata (title ids, the 17 rom_files columns, artwork and
converto.scan_metadata) moves to its own PR so its columns land with the
UI that reads them. This PR keeps conversion only.
The conversion cache no longer has uncompressed_size_bytes to budget
with, so a conversion to an uncompressed target reserves a fixed
multiple of its input and the cleanup pass after it evicts any overshoot.
Co-Authored-By: DevYukine <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ
* feat(converto): let the client pick the download format with ?format=
`?format=zso,iso` lists the formats a client can read. The stored file is
served whenever its format is listed, even with conversion off, so most
emulator clients never trigger a conversion. Otherwise the first listed
format with a cached copy is served, then the first one rom-converto can
produce.
A conversion that outlasts a short wait answers 202 with Retry-After
instead of falling back to the original, so a URL never serves two
different files and a client never gets bytes it can't read. 406 means
no listed format can be served. HEAD reports the same without starting
a conversion. This replaces `?converted=true`, the per-platform download
target and `max_sync_size_mb`; `cache_ttl_hours` stays config.yml-only.
Co-Authored-By: DevYukine <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ
* feat(converto): convert the library in place with a manual task
`converto.platform_formats` now names the format a platform's library is
stored in, and the new "Convert library" task brings every matched rom
to it. Each output is staged in a `.romm_tmp_` directory beside the
original, moved into place, and only then are the originals (and a cue's
tracks) deleted. Single-file roms take the new name, folder roms keep
theirs and get their .m3u entries rewritten, and the file rows refresh,
so a rom keeps its id, saves and collections.
Only lossless conversions qualify, so xbox (xiso trims the dump) has no
library format. Unmatched roms are skipped because a converted file no
longer hash-matches a DAT. `/api/config` now carries the library targets
and, per input extension, the formats a download can be converted to.
Co-Authored-By: DevYukine <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ
* fix(converto): address Greptile and Copilot review findings
- Publish a converted library file with a no-replace link, so a file that
appears during conversion is never overwritten.
- Delete originals only after the rom row or its playlists point at the new
file, and keep them if that update fails. Playlists are rewritten atomically.
- Refuse a cue conversion whose tracks another rom or another cue holds.
- Remove stage dirs a killed conversion left behind.
- Try later ?format= targets when a conversion fails, treat any locked key dir
as converting, and cap how many conversions downloads can start.
- Drop the unsupported iso target for CD platforms from the example config and
list the ROM_CONVERTO_* variables in env.template.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(converto): address code-review findings (pr-ready step 2)
Serve converted outputs at 0644, read cue sheets and playlists with a BOM,
stage folder rom conversions outside the rom folder, keep one rom's failure
from stopping the convert library task, accept alias extensions as the stored
format, probe running conversions with a shared lock, and bound the timeout.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): simplify conversion code (pr-ready step 3)
Check and chmod rom-converto output once in the adapter, canonicalize
requested format aliases, reuse LINK_FALLBACK_ERRNOS and the handler's atomic
write, make the download format table a constant, skip the probe lock once
cached, and fold duplicated name checks and cache stat passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(converto): trim comments (pr-ready step 4)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* refactor(converto): find rom-converto on PATH instead of ROM_CONVERTO_PATH
Both images install the binary on PATH, the variable was undocumented,
and RAHasher is already resolved the same way.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Yukine <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: DevYukine <[email protected]>
