mirror of
https://github.com/rommapp/romm.git
synced 2026-10-12 02:21:58 +00:00
640a783f3f
* feat(converto): integrate rom-converto for title id extraction and download conversion * fix(docker): ship rom-converto in the production image Only the dev Dockerfile installed the binary, so the integration could never enable on a released image. Fetch the same pinned, checksummed musl build in a dedicated stage and copy it next to RAHasher. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(converto): skip the update check and batch info calls The CLI asks api.github.com for a newer release on every run unless told not to, so pass --no-update-check on each invocation. Inspect a rom's files in one `info --paths-file` run instead of one subprocess per file, and read the inspectable extensions from the capabilities manifest so scans skip files info cannot recognize. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(scan): keep sigil authoritative for title ids rom-converto ran first and overrode the ids sigil read. Sigil now runs first and records its per-file ids; rom-converto only fills files sigil left unset, on new or changed files, in one batch. Restore the base-id selection for sigil extractions on every platform, as on master. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(downloads): make conversion opt-in and bound the cache The download endpoint returned a converted file to every client once a platform had a target, including the browser player and apps that check hashes against RomFile. Conversion now needs ?converted=true. A pre-warmed copy is served whatever its size; the in-request size cap only gates starting a conversion. ROM_CONVERTO_CACHE_MAX_SIZE_GB (default 20, 0 unbounded) caps the cache: conversions and the pre-warm task stop when it is full, and cleanup evicts the least recently served copies. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * chore(types): regenerate frontend types for the converto schema Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(converto): address code-review findings (pr-ready step 2) - Fall back to rom-converto's stored ids only when it ran this pass, in disc order, so a rescan that skipped extraction keeps sigil's save target - Persist per-file title ids that extraction writes onto reused rows - Only authenticated callers may start an in-request conversion; past a 240s deadline serve the original and let the conversion finish into the cache instead of nginx returning 504 - Treat a probe OSError or timeout as a failed probe; kill the subprocess when the caller is cancelled - Give the pre-warm task the scan timeout and report progress in the ConversionStats shape the task UI reads - Reuse zip_cache constants, walk the cache off the event loop, tolerate files vanishing mid-cleanup, and trim long comments Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): simplify conversion code (pr-ready step 3) - Move the converted-download policy (target lookup, cached-first, sync size cap, deadline) from the endpoint into utils/conversion_cache and share conversion_target with the pre-warm task - Share one path lookup between get_cached_converted and get_or_convert; the pre-warm walks the cache once and tracks a running byte budget instead of walking it twice per rom - One normalize_platform_formats beside TARGETS_BY_PLATFORM for the config loader (via _validated_platform_map) and the settings endpoint; update_converto_settings takes a ConvertoConfig and is written back with dataclasses.asdict - read_infos owns its never-raise contract, so the scan drops its blanket except; drop the unused RomConvertoInfo.kind and error attributes - Sort files for the rom-converto identity only when it is used; stat each cached file once during cleanup Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * chore(converto): trim comments (pr-ready step 4) Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): convert on first download instead of pre-warming A library pre-warm fought the TTL: cleanup dropped every copy nobody downloaded within a day. Remove the Convert library task and convert on demand only. Files over ROM_CONVERTO_MAX_SYNC_SIZE_MB now start their conversion in the background while the original is served, so the next download within the TTL gets the converted copy at any size. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): drop decrypt and encrypt download targets Downloads now only change a rom's container or compression (CHD, RVZ, CSO, NSZ, ...); serving decrypted files is left for a separate decision. NDS and PS3 have no remaining targets. Also note that the conversion concurrency limit applies per process. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): move the cache limits into converto.* cache_max_size_gb and max_sync_size_mb join cache_ttl_hours in config.yml and PUT /api/config/converto_settings, replacing the ROM_CONVERTO_CACHE_MAX_SIZE_GB and ROM_CONVERTO_MAX_SYNC_SIZE_MB env vars, so every eviction knob is edited in one place at runtime. The loader takes its defaults from ConvertoConfig and one table holds each integer's minimum for both the loader and the payload. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): lock conversions with flock instead of a sentinel A conversion now holds a non-blocking flock on its key dir, which the kernel releases if the worker dies. That replaces the .partial sentinel, its 6-hour staleness guess (which a long ROM_CONVERTO_TIMEOUT could outlive), and the rename-aside takeover. Cleanup skips locked dirs, and partial output gets a hidden name so it never counts as a served copy. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * test(converto): parametrize repeated test cases Collapse the per-platform info parsing, operation resolution and converto_active cases into parametrized tests, so a new platform or target is one row. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(scan): run rom-converto first with sigil as the fallback rom-converto reads each rom's new or changed files first, in one batch, and its title ids win. sigil still reads what it can, since it alone provides save targets and Switch content types, and fills the ids rom-converto left unset. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(converto): store per-file metadata read by rom-converto during scans * fix(scan): keep the stored save target when sigil reads nothing * fix(converto): read Xbox 360 disc ids and Wii title versions, store content type as text * feat(converto): save file icons, banners and backgrounds read by rom-converto * chore(converto): bump rom-converto to v0.23.0 * fix(converto): address review findings in the conversion cache, scan metadata and settings * refactor(converto): move per-file metadata out of the conversion PR Scan-time metadata (title ids, the 17 rom_files columns, artwork and converto.scan_metadata) moves to its own PR so its columns land with the UI that reads them. This PR keeps conversion only. The conversion cache no longer has uncompressed_size_bytes to budget with, so a conversion to an uncompressed target reserves a fixed multiple of its input and the cleanup pass after it evicts any overshoot. Co-Authored-By: DevYukine <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ * feat(converto): let the client pick the download format with ?format= `?format=zso,iso` lists the formats a client can read. The stored file is served whenever its format is listed, even with conversion off, so most emulator clients never trigger a conversion. Otherwise the first listed format with a cached copy is served, then the first one rom-converto can produce. A conversion that outlasts a short wait answers 202 with Retry-After instead of falling back to the original, so a URL never serves two different files and a client never gets bytes it can't read. 406 means no listed format can be served. HEAD reports the same without starting a conversion. This replaces `?converted=true`, the per-platform download target and `max_sync_size_mb`; `cache_ttl_hours` stays config.yml-only. Co-Authored-By: DevYukine <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ * feat(converto): convert the library in place with a manual task `converto.platform_formats` now names the format a platform's library is stored in, and the new "Convert library" task brings every matched rom to it. Each output is staged in a `.romm_tmp_` directory beside the original, moved into place, and only then are the originals (and a cue's tracks) deleted. Single-file roms take the new name, folder roms keep theirs and get their .m3u entries rewritten, and the file rows refresh, so a rom keeps its id, saves and collections. Only lossless conversions qualify, so xbox (xiso trims the dump) has no library format. Unmatched roms are skipped because a converted file no longer hash-matches a DAT. `/api/config` now carries the library targets and, per input extension, the formats a download can be converted to. Co-Authored-By: DevYukine <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ * fix(converto): address Greptile and Copilot review findings - Publish a converted library file with a no-replace link, so a file that appears during conversion is never overwritten. - Delete originals only after the rom row or its playlists point at the new file, and keep them if that update fails. Playlists are rewritten atomically. - Refuse a cue conversion whose tracks another rom or another cue holds. - Remove stage dirs a killed conversion left behind. - Try later ?format= targets when a conversion fails, treat any locked key dir as converting, and cap how many conversions downloads can start. - Drop the unsupported iso target for CD platforms from the example config and list the ROM_CONVERTO_* variables in env.template. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(converto): address code-review findings (pr-ready step 2) Serve converted outputs at 0644, read cue sheets and playlists with a BOM, stage folder rom conversions outside the rom folder, keep one rom's failure from stopping the convert library task, accept alias extensions as the stored format, probe running conversions with a shared lock, and bound the timeout. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): simplify conversion code (pr-ready step 3) Check and chmod rom-converto output once in the adapter, canonicalize requested format aliases, reuse LINK_FALLBACK_ERRNOS and the handler's atomic write, make the download format table a constant, skip the probe lock once cached, and fold duplicated name checks and cache stat passes. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * chore(converto): trim comments (pr-ready step 4) Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): find rom-converto on PATH instead of ROM_CONVERTO_PATH Both images install the binary on PATH, the variable was undocumented, and RAHasher is already resolved the same way. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Yukine <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: DevYukine <[email protected]>