mirror of
https://github.com/rommapp/romm.git
synced 2026-10-12 02:21:58 +00:00
feat(converto): convert the library in place and serve downloads in a requested format (#4999)
* feat(converto): integrate rom-converto for title id extraction and download conversion * fix(docker): ship rom-converto in the production image Only the dev Dockerfile installed the binary, so the integration could never enable on a released image. Fetch the same pinned, checksummed musl build in a dedicated stage and copy it next to RAHasher. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(converto): skip the update check and batch info calls The CLI asks api.github.com for a newer release on every run unless told not to, so pass --no-update-check on each invocation. Inspect a rom's files in one `info --paths-file` run instead of one subprocess per file, and read the inspectable extensions from the capabilities manifest so scans skip files info cannot recognize. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(scan): keep sigil authoritative for title ids rom-converto ran first and overrode the ids sigil read. Sigil now runs first and records its per-file ids; rom-converto only fills files sigil left unset, on new or changed files, in one batch. Restore the base-id selection for sigil extractions on every platform, as on master. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(downloads): make conversion opt-in and bound the cache The download endpoint returned a converted file to every client once a platform had a target, including the browser player and apps that check hashes against RomFile. Conversion now needs ?converted=true. A pre-warmed copy is served whatever its size; the in-request size cap only gates starting a conversion. ROM_CONVERTO_CACHE_MAX_SIZE_GB (default 20, 0 unbounded) caps the cache: conversions and the pre-warm task stop when it is full, and cleanup evicts the least recently served copies. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * chore(types): regenerate frontend types for the converto schema Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(converto): address code-review findings (pr-ready step 2) - Fall back to rom-converto's stored ids only when it ran this pass, in disc order, so a rescan that skipped extraction keeps sigil's save target - Persist per-file title ids that extraction writes onto reused rows - Only authenticated callers may start an in-request conversion; past a 240s deadline serve the original and let the conversion finish into the cache instead of nginx returning 504 - Treat a probe OSError or timeout as a failed probe; kill the subprocess when the caller is cancelled - Give the pre-warm task the scan timeout and report progress in the ConversionStats shape the task UI reads - Reuse zip_cache constants, walk the cache off the event loop, tolerate files vanishing mid-cleanup, and trim long comments Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): simplify conversion code (pr-ready step 3) - Move the converted-download policy (target lookup, cached-first, sync size cap, deadline) from the endpoint into utils/conversion_cache and share conversion_target with the pre-warm task - Share one path lookup between get_cached_converted and get_or_convert; the pre-warm walks the cache once and tracks a running byte budget instead of walking it twice per rom - One normalize_platform_formats beside TARGETS_BY_PLATFORM for the config loader (via _validated_platform_map) and the settings endpoint; update_converto_settings takes a ConvertoConfig and is written back with dataclasses.asdict - read_infos owns its never-raise contract, so the scan drops its blanket except; drop the unused RomConvertoInfo.kind and error attributes - Sort files for the rom-converto identity only when it is used; stat each cached file once during cleanup Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * chore(converto): trim comments (pr-ready step 4) Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): convert on first download instead of pre-warming A library pre-warm fought the TTL: cleanup dropped every copy nobody downloaded within a day. Remove the Convert library task and convert on demand only. Files over ROM_CONVERTO_MAX_SYNC_SIZE_MB now start their conversion in the background while the original is served, so the next download within the TTL gets the converted copy at any size. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): drop decrypt and encrypt download targets Downloads now only change a rom's container or compression (CHD, RVZ, CSO, NSZ, ...); serving decrypted files is left for a separate decision. NDS and PS3 have no remaining targets. Also note that the conversion concurrency limit applies per process. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): move the cache limits into converto.* cache_max_size_gb and max_sync_size_mb join cache_ttl_hours in config.yml and PUT /api/config/converto_settings, replacing the ROM_CONVERTO_CACHE_MAX_SIZE_GB and ROM_CONVERTO_MAX_SYNC_SIZE_MB env vars, so every eviction knob is edited in one place at runtime. The loader takes its defaults from ConvertoConfig and one table holds each integer's minimum for both the loader and the payload. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): lock conversions with flock instead of a sentinel A conversion now holds a non-blocking flock on its key dir, which the kernel releases if the worker dies. That replaces the .partial sentinel, its 6-hour staleness guess (which a long ROM_CONVERTO_TIMEOUT could outlive), and the rename-aside takeover. Cleanup skips locked dirs, and partial output gets a hidden name so it never counts as a served copy. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * test(converto): parametrize repeated test cases Collapse the per-platform info parsing, operation resolution and converto_active cases into parametrized tests, so a new platform or target is one row. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(scan): run rom-converto first with sigil as the fallback rom-converto reads each rom's new or changed files first, in one batch, and its title ids win. sigil still reads what it can, since it alone provides save targets and Switch content types, and fills the ids rom-converto left unset. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(converto): store per-file metadata read by rom-converto during scans * fix(scan): keep the stored save target when sigil reads nothing * fix(converto): read Xbox 360 disc ids and Wii title versions, store content type as text * feat(converto): save file icons, banners and backgrounds read by rom-converto * chore(converto): bump rom-converto to v0.23.0 * fix(converto): address review findings in the conversion cache, scan metadata and settings * refactor(converto): move per-file metadata out of the conversion PR Scan-time metadata (title ids, the 17 rom_files columns, artwork and converto.scan_metadata) moves to its own PR so its columns land with the UI that reads them. This PR keeps conversion only. The conversion cache no longer has uncompressed_size_bytes to budget with, so a conversion to an uncompressed target reserves a fixed multiple of its input and the cleanup pass after it evicts any overshoot. Co-Authored-By: DevYukine <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ * feat(converto): let the client pick the download format with ?format= `?format=zso,iso` lists the formats a client can read. The stored file is served whenever its format is listed, even with conversion off, so most emulator clients never trigger a conversion. Otherwise the first listed format with a cached copy is served, then the first one rom-converto can produce. A conversion that outlasts a short wait answers 202 with Retry-After instead of falling back to the original, so a URL never serves two different files and a client never gets bytes it can't read. 406 means no listed format can be served. HEAD reports the same without starting a conversion. This replaces `?converted=true`, the per-platform download target and `max_sync_size_mb`; `cache_ttl_hours` stays config.yml-only. Co-Authored-By: DevYukine <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ * feat(converto): convert the library in place with a manual task `converto.platform_formats` now names the format a platform's library is stored in, and the new "Convert library" task brings every matched rom to it. Each output is staged in a `.romm_tmp_` directory beside the original, moved into place, and only then are the originals (and a cue's tracks) deleted. Single-file roms take the new name, folder roms keep theirs and get their .m3u entries rewritten, and the file rows refresh, so a rom keeps its id, saves and collections. Only lossless conversions qualify, so xbox (xiso trims the dump) has no library format. Unmatched roms are skipped because a converted file no longer hash-matches a DAT. `/api/config` now carries the library targets and, per input extension, the formats a download can be converted to. Co-Authored-By: DevYukine <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_017BuuZVVtVPNLRaCjTN9bgQ * fix(converto): address Greptile and Copilot review findings - Publish a converted library file with a no-replace link, so a file that appears during conversion is never overwritten. - Delete originals only after the rom row or its playlists point at the new file, and keep them if that update fails. Playlists are rewritten atomically. - Refuse a cue conversion whose tracks another rom or another cue holds. - Remove stage dirs a killed conversion left behind. - Try later ?format= targets when a conversion fails, treat any locked key dir as converting, and cap how many conversions downloads can start. - Drop the unsupported iso target for CD platforms from the example config and list the ROM_CONVERTO_* variables in env.template. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(converto): address code-review findings (pr-ready step 2) Serve converted outputs at 0644, read cue sheets and playlists with a BOM, stage folder rom conversions outside the rom folder, keep one rom's failure from stopping the convert library task, accept alias extensions as the stored format, probe running conversions with a shared lock, and bound the timeout. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): simplify conversion code (pr-ready step 3) Check and chmod rom-converto output once in the adapter, canonicalize requested format aliases, reuse LINK_FALLBACK_ERRNOS and the handler's atomic write, make the download format table a constant, skip the probe lock once cached, and fold duplicated name checks and cache stat passes. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * chore(converto): trim comments (pr-ready step 4) Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(converto): find rom-converto on PATH instead of ROM_CONVERTO_PATH Both images install the binary on PATH, the variable was undocumented, and RAHasher is already resolved the same way. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Yukine <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: DevYukine <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5.5
Yukine
DevYukine
parent
77da7144d1
commit
640a783f3f
+16
@@ -111,6 +111,22 @@ RUN make HAVE_CHD=1 -f ./Makefile.RAHasher \
|
||||
&& cp ./bin64/RAHasher /usr/bin/RAHasher
|
||||
RUN rm -rf /tmp/RALibretro
|
||||
|
||||
# Install rom-converto (optional); its static musl build runs on this glibc image.
|
||||
# Keep the version and both sums in sync with docker/Dockerfile.
|
||||
ARG TARGETARCH
|
||||
RUN ROM_CONVERTO_VERSION=v0.23.0 \
|
||||
&& arch="${TARGETARCH:-$(dpkg --print-architecture)}" \
|
||||
&& case "${arch}" in \
|
||||
amd64) rc="linux-x64-musl"; sum="37d03a67ad97fc02768890e763b4c80072fdfd15f2c5b97e44f9a2b50fc40876" ;; \
|
||||
arm64) rc="linux-arm64-musl"; sum="8abe37ca2d1a0e3d719fb8e795e011c672e3f1c088521cf0bb1e467dc658dc3c" ;; \
|
||||
*) echo "unsupported architecture: ${arch}" && exit 1 ;; \
|
||||
esac \
|
||||
&& curl -fsSL -o "/tmp/rom-converto-cli-${rc}" \
|
||||
"https://github.com/DevYukine/rom-converto/releases/download/${ROM_CONVERTO_VERSION}/rom-converto-cli-${rc}" \
|
||||
&& echo "${sum} /tmp/rom-converto-cli-${rc}" | sha256sum -c --status \
|
||||
&& install -m 0755 "/tmp/rom-converto-cli-${rc}" /usr/bin/rom-converto \
|
||||
&& rm -f "/tmp/rom-converto-cli-${rc}"
|
||||
|
||||
# Install frontend dependencies
|
||||
COPY frontend/package.json /app/frontend/
|
||||
WORKDIR /app/frontend
|
||||
|
||||
@@ -0,0 +1,341 @@
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import shutil
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Final
|
||||
|
||||
from config import ROM_CONVERTO_ENABLED, ROM_CONVERTO_TIMEOUT
|
||||
from logger.formatter import LIGHTMAGENTA
|
||||
from logger.formatter import highlight as hl
|
||||
from logger.logger import log
|
||||
from utils.filesystem import SERVED_FILE_MODE
|
||||
from utils.platform_slugs import UniversalPlatformSlug as UPS
|
||||
|
||||
# The capabilities probe must never hang download paths; a real
|
||||
# manifest print is instant.
|
||||
_PROBE_TIMEOUT_SECONDS = 30
|
||||
|
||||
_STDERR_TAIL_BYTES = 400
|
||||
|
||||
_BINARY: Final = "rom-converto"
|
||||
|
||||
|
||||
class RomConvertoError(Exception): ...
|
||||
|
||||
|
||||
class RomConvertoBinaryNotFoundError(RomConvertoError): ...
|
||||
|
||||
|
||||
class RomConvertoTimeoutError(RomConvertoError): ...
|
||||
|
||||
|
||||
class RomConvertoOperationError(RomConvertoError):
|
||||
"""A conversion command exited nonzero."""
|
||||
|
||||
|
||||
# `xbox convert` keeps only a dump's game partition, so a library never stores its output.
|
||||
_LOSSY_ARGV: Final[frozenset[tuple[str, ...]]] = frozenset({("xbox", "convert")})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Operation:
|
||||
"""A subcommand bringing `input_exts` files to `target`; source and output are appended to `argv`."""
|
||||
|
||||
target: str
|
||||
platforms: frozenset[str]
|
||||
argv: tuple[str, ...]
|
||||
input_exts: frozenset[str]
|
||||
output_ext: str
|
||||
|
||||
@property
|
||||
def lossless(self) -> bool:
|
||||
return self.argv not in _LOSSY_ARGV
|
||||
|
||||
def output_name(self, src: Path, input_ext: str) -> str:
|
||||
"""`src` renamed from the matched lowercase `input_ext` to the output extension."""
|
||||
return f"{src.name[: len(src.name) - len(input_ext)]}{self.output_ext}"
|
||||
|
||||
|
||||
def _op(
|
||||
target: str,
|
||||
platforms: set[UPS],
|
||||
argv: str,
|
||||
input_exts: str,
|
||||
output_ext: str,
|
||||
) -> Operation:
|
||||
return Operation(
|
||||
target=target,
|
||||
platforms=frozenset(platforms),
|
||||
argv=tuple(argv.split()),
|
||||
input_exts=frozenset(input_exts.split()),
|
||||
output_ext=output_ext,
|
||||
)
|
||||
|
||||
|
||||
_DVD_PLATFORMS = {UPS.PSP, UPS.PS2}
|
||||
_CD_PLATFORMS = {UPS.PSX, UPS.SATURN, UPS.SEGACD, UPS.DC}
|
||||
|
||||
# The CLI's single-file format conversions. Decrypt/encrypt and directory-shaped
|
||||
# operations (Wii U packs, Switch merge/split, `extract`, Xbox 360 GoD) are left out.
|
||||
OPERATIONS: Final[tuple[Operation, ...]] = (
|
||||
# 3DS
|
||||
_op("z3ds", {UPS.N3DS}, "ctr compress", ".cia", ".zcia"),
|
||||
_op("z3ds", {UPS.N3DS}, "ctr compress", ".cci .3ds", ".zcci"),
|
||||
_op("z3ds", {UPS.N3DS}, "ctr compress", ".cxi", ".zcxi"),
|
||||
_op("cia", {UPS.N3DS}, "ctr decompress", ".zcia", ".cia"),
|
||||
_op("cia", {UPS.N3DS}, "ctr convert", ".3ds .cci", ".cia"),
|
||||
_op("cci", {UPS.N3DS}, "ctr decompress", ".zcci", ".cci"),
|
||||
_op("cci", {UPS.N3DS}, "ctr convert", ".cia", ".cci"),
|
||||
# Sony discs
|
||||
_op("iso", _DVD_PLATFORMS, "cso decompress", ".cso .zso .dax", ".iso"),
|
||||
# CD-mode CHDs extract to .bin/.cue, so only DVD platforms get this.
|
||||
_op("iso", _DVD_PLATFORMS, "chd extract", ".chd", ".iso"),
|
||||
_op("iso", _DVD_PLATFORMS, "cue to-iso", ".cue", ".iso"),
|
||||
_op("iso", {UPS.PSP}, "psp to-iso", ".pbp", ".iso"),
|
||||
_op("cso", _DVD_PLATFORMS, "cso compress --format cso", ".iso", ".cso"),
|
||||
_op("cso", _DVD_PLATFORMS, "chd to-cso --format cso", ".chd", ".cso"),
|
||||
_op("zso", _DVD_PLATFORMS, "cso compress --format zso", ".iso", ".zso"),
|
||||
_op("zso", _DVD_PLATFORMS, "chd to-cso --format zso", ".chd", ".zso"),
|
||||
_op("chd", _DVD_PLATFORMS | _CD_PLATFORMS, "chd compress", ".cue .iso", ".chd"),
|
||||
_op("chd", _DVD_PLATFORMS, "cso to-chd", ".cso .zso .dax", ".chd"),
|
||||
# GameCube / Wii
|
||||
_op("rvz", {UPS.NGC}, "dol compress", ".iso .gcm", ".rvz"),
|
||||
_op("rvz", {UPS.NGC}, "dol migrate", ".gcz .nkit.iso .nkit.gcz", ".rvz"),
|
||||
_op("iso", {UPS.NGC}, "dol decompress", ".rvz", ".iso"),
|
||||
_op("rvz", {UPS.WII}, "rvl compress", ".iso .wbfs", ".rvz"),
|
||||
_op("rvz", {UPS.WII}, "rvl migrate", ".wia .gcz .nkit.iso .nkit.gcz", ".rvz"),
|
||||
_op("iso", {UPS.WII}, "rvl decompress", ".rvz", ".iso"),
|
||||
_op("wbfs", {UPS.WII}, "rvl decompress", ".rvz", ".wbfs"),
|
||||
# Switch (prod.keys resolved by the CLI itself)
|
||||
_op("nsz", {UPS.SWITCH, UPS.SWITCH_2}, "nx compress", ".nsp", ".nsz"),
|
||||
_op("xcz", {UPS.SWITCH, UPS.SWITCH_2}, "nx compress", ".xci", ".xcz"),
|
||||
_op("nsp", {UPS.SWITCH, UPS.SWITCH_2}, "nx decompress", ".nsz", ".nsp"),
|
||||
_op("xci", {UPS.SWITCH, UPS.SWITCH_2}, "nx decompress", ".xcz", ".xci"),
|
||||
# Xbox
|
||||
_op("xiso", {UPS.XBOX}, "xbox convert", ".iso", ".xiso"),
|
||||
_op("zar", {UPS.XBOX360}, "xenon compress", ".iso", ".zar"),
|
||||
)
|
||||
|
||||
# Targets that store the image as-is, so converting to one can grow the file.
|
||||
UNCOMPRESSED_TARGETS: Final[frozenset[str]] = frozenset(
|
||||
{"cci", "cia", "iso", "nsp", "wbfs", "xci", "xiso"}
|
||||
)
|
||||
|
||||
|
||||
# Platform slug -> the targets a library can be stored in.
|
||||
LIBRARY_TARGETS_BY_PLATFORM: Final[dict[str, frozenset[str]]] = {
|
||||
slug: frozenset(
|
||||
op.target for op in OPERATIONS if slug in op.platforms and op.lossless
|
||||
)
|
||||
for slug in sorted(
|
||||
{slug for op in OPERATIONS if op.lossless for slug in op.platforms}
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
def _download_formats() -> dict[str, dict[str, list[str]]]:
|
||||
table: dict[str, dict[str, set[str]]] = {}
|
||||
for op in OPERATIONS:
|
||||
for slug in op.platforms:
|
||||
for ext in op.input_exts:
|
||||
table.setdefault(slug, {}).setdefault(ext, set()).add(op.target)
|
||||
return {
|
||||
slug: {ext: sorted(targets) for ext, targets in sorted(exts.items())}
|
||||
for slug, exts in sorted(table.items())
|
||||
}
|
||||
|
||||
|
||||
# Platform slug -> input extension -> the targets a download of it can be converted to.
|
||||
DOWNLOAD_FORMATS: Final[dict[str, dict[str, list[str]]]] = _download_formats()
|
||||
|
||||
|
||||
# Extensions whose format goes by another name than the extension itself.
|
||||
_FORMAT_ALIASES: Final[dict[str, str]] = {
|
||||
".3ds": "cci",
|
||||
".gcm": "iso",
|
||||
".zcci": "z3ds",
|
||||
".zcia": "z3ds",
|
||||
".zcxi": "z3ds",
|
||||
}
|
||||
_KNOWN_EXTS: Final[frozenset[str]] = frozenset(
|
||||
{ext for op in OPERATIONS for ext in (*op.input_exts, op.output_ext)}
|
||||
| _FORMAT_ALIASES.keys()
|
||||
)
|
||||
|
||||
|
||||
def file_format(file_name: str) -> str:
|
||||
"""The format `file_name` is in, as a target name or its extension (`nkit-iso`)."""
|
||||
name = file_name.lower()
|
||||
ext = max((e for e in _KNOWN_EXTS if name.endswith(e)), key=len, default=None)
|
||||
if ext is None:
|
||||
return Path(name).suffix.lstrip(".")
|
||||
return _FORMAT_ALIASES.get(ext, ext.lstrip(".").replace(".", "-"))
|
||||
|
||||
|
||||
def canonical_format(name: str) -> str:
|
||||
"""The target name a client means by `name`, so `3ds` reads as `cci`."""
|
||||
return _FORMAT_ALIASES.get(f".{name}", name)
|
||||
|
||||
|
||||
def normalize_platform_formats(raw: dict[str, str]) -> dict[str, str]:
|
||||
"""`raw` with slugs and library targets trimmed and lowercased.
|
||||
|
||||
Raises:
|
||||
ValueError: A platform has no library targets, or a target isn't one of them.
|
||||
"""
|
||||
cleaned = {
|
||||
str(slug).strip().lower(): str(target).strip().lower()
|
||||
for slug, target in raw.items()
|
||||
}
|
||||
for slug, target in cleaned.items():
|
||||
targets = LIBRARY_TARGETS_BY_PLATFORM.get(slug)
|
||||
if targets is None:
|
||||
raise ValueError(
|
||||
f"rom-converto has no library formats for {slug!r}. "
|
||||
f"Supported: {sorted(LIBRARY_TARGETS_BY_PLATFORM)}."
|
||||
)
|
||||
if target not in targets:
|
||||
raise ValueError(
|
||||
f"{target!r} is not a conversion target for {slug}. "
|
||||
f"Valid options: {sorted(targets)}."
|
||||
)
|
||||
return cleaned
|
||||
|
||||
|
||||
def resolve_operation(
|
||||
platform_slug: str, target: str, file_name: str, *, lossless: bool = False
|
||||
) -> tuple[Operation, str] | None:
|
||||
"""The operation and matched extension bringing `file_name` to `target`, or None if none applies."""
|
||||
name = file_name.lower()
|
||||
best: tuple[Operation, str] | None = None
|
||||
for op in OPERATIONS:
|
||||
if op.target != target or platform_slug not in op.platforms:
|
||||
continue
|
||||
if lossless and not op.lossless:
|
||||
continue
|
||||
for ext in op.input_exts:
|
||||
# Prefer the longer extension so `.nkit.iso` is not read as `.iso`.
|
||||
if name.endswith(ext) and (best is None or len(ext) > len(best[1])):
|
||||
best = (op, ext)
|
||||
return best
|
||||
|
||||
|
||||
def _tail(text: str) -> str:
|
||||
return text.strip()[-_STDERR_TAIL_BYTES:]
|
||||
|
||||
|
||||
async def _run(argv: list[str], timeout_seconds: float) -> tuple[int, str, str]:
|
||||
"""Run a rom-converto subcommand and return (returncode, stdout, stderr)."""
|
||||
binary = await asyncio.to_thread(shutil.which, _BINARY)
|
||||
if binary is None:
|
||||
raise RomConvertoBinaryNotFoundError(f"{_BINARY} binary not found on PATH")
|
||||
# The CLI otherwise asks api.github.com for a newer release on every run.
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
binary,
|
||||
"--no-update-check",
|
||||
*argv,
|
||||
stdin=asyncio.subprocess.DEVNULL,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
try:
|
||||
stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout_seconds)
|
||||
except TimeoutError as err:
|
||||
_kill(proc)
|
||||
# Drain the pipes so the transport closes instead of leaking its fds.
|
||||
await proc.communicate()
|
||||
raise RomConvertoTimeoutError(
|
||||
f"rom-converto {' '.join(argv[:2])} timed out after {timeout_seconds}s"
|
||||
) from err
|
||||
except BaseException:
|
||||
# A cancelled caller must not leave a conversion running unowned.
|
||||
_kill(proc)
|
||||
raise
|
||||
|
||||
return (
|
||||
proc.returncode or 0,
|
||||
stdout.decode("utf-8", errors="replace"),
|
||||
stderr.decode("utf-8", errors="replace"),
|
||||
)
|
||||
|
||||
|
||||
def _settle_output(operation: Operation, out: Path) -> None:
|
||||
"""Check `out` is all the command wrote beside it, and make it servable."""
|
||||
if [p.name for p in out.parent.iterdir()] != [out.name]:
|
||||
raise RomConvertoOperationError(
|
||||
f"rom-converto {' '.join(operation.argv)} did not write exactly {out.name}"
|
||||
)
|
||||
# rom-converto may write an owner-only file, which nginx can't serve.
|
||||
out.chmod(SERVED_FILE_MODE)
|
||||
|
||||
|
||||
def _kill(proc: asyncio.subprocess.Process) -> None:
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
proc.kill()
|
||||
|
||||
|
||||
class RomConvertoService:
|
||||
"""Service to convert ROMs using the rom-converto CLI."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._available: bool | None = None
|
||||
self._probe_lock = asyncio.Lock()
|
||||
|
||||
async def is_enabled(self) -> bool:
|
||||
if not ROM_CONVERTO_ENABLED:
|
||||
return False
|
||||
if self._available is not None:
|
||||
return self._available
|
||||
async with self._probe_lock:
|
||||
if self._available is not None:
|
||||
return self._available
|
||||
# A corrupt or wrong-arch binary passes which(); a missing one
|
||||
# stays uncached so installing it needs no restart.
|
||||
failure: str | None
|
||||
try:
|
||||
code, stdout, _ = await _run(["capabilities"], _PROBE_TIMEOUT_SECONDS)
|
||||
except RomConvertoBinaryNotFoundError:
|
||||
return False
|
||||
except (RomConvertoTimeoutError, OSError) as exc:
|
||||
failure = str(exc)
|
||||
else:
|
||||
failure = f"code {code}" if code != 0 else None
|
||||
if failure is not None:
|
||||
log.warning(
|
||||
f"{hl(_BINARY)} failed its capability "
|
||||
f"probe ({failure}); disabling integration until restart"
|
||||
)
|
||||
self._available = False
|
||||
return False
|
||||
try:
|
||||
manifest = json.loads(stdout)
|
||||
except json.JSONDecodeError:
|
||||
manifest = None
|
||||
version = (
|
||||
manifest.get("version") if isinstance(manifest, dict) else None
|
||||
) or "unknown version"
|
||||
log.info(
|
||||
f"Detected {hl('rom-converto', color=LIGHTMAGENTA)} {hl(str(version))}"
|
||||
)
|
||||
self._available = True
|
||||
return True
|
||||
|
||||
async def convert(self, operation: Operation, src: Path, out: Path) -> None:
|
||||
"""Run `operation` on `src`, writing `out`, which must be alone in its directory.
|
||||
|
||||
Raises:
|
||||
RomConvertoOperationError: The command failed or wrote more than `out`.
|
||||
"""
|
||||
argv = [*operation.argv, str(src), str(out)]
|
||||
code, stdout, stderr = await _run(argv, ROM_CONVERTO_TIMEOUT)
|
||||
if code != 0:
|
||||
diagnostic = _tail(stderr) or _tail(stdout)
|
||||
raise RomConvertoOperationError(
|
||||
f"rom-converto {' '.join(operation.argv)} failed with code {code}: {diagnostic}"
|
||||
)
|
||||
await asyncio.to_thread(_settle_output, operation, out)
|
||||
|
||||
|
||||
rom_converto_service = RomConvertoService()
|
||||
@@ -65,6 +65,19 @@ ROM_PATCHER_MAX_CONCURRENCY: Final[int] = max(
|
||||
1, safe_int(_get_env("ROM_PATCHER_MAX_CONCURRENCY"), 2)
|
||||
)
|
||||
|
||||
# ROM CONVERTO
|
||||
ROM_CONVERTO_ENABLED: Final[bool] = safe_str_to_bool(_get_env("ROM_CONVERTO_ENABLED"))
|
||||
# Seconds per rom-converto CLI operation.
|
||||
ROM_CONVERTO_TIMEOUT: Final[int] = max(
|
||||
1, safe_int(_get_env("ROM_CONVERTO_TIMEOUT"), 600)
|
||||
)
|
||||
# Concurrent conversion subprocesses per process (each web and RQ worker).
|
||||
ROM_CONVERTO_MAX_CONCURRENCY: Final[int] = max(
|
||||
1, safe_int(_get_env("ROM_CONVERTO_MAX_CONCURRENCY"), 2)
|
||||
)
|
||||
# Disk cache for converted downloads, under the tree nginx serves at /cache/.
|
||||
ROM_CONVERTO_CACHE_PATH: Final[str] = f"{ROMM_BASE_PATH}/cache/converts"
|
||||
|
||||
# DATABASE
|
||||
DB_HOST: Final[str | None] = _get_env("DB_HOST")
|
||||
DB_PORT: Final[int] = safe_int(_get_env("DB_PORT"), 3306)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import dataclasses
|
||||
import enum
|
||||
import functools
|
||||
import glob
|
||||
@@ -14,6 +15,7 @@ import yaml
|
||||
from sqlalchemy import URL
|
||||
from yaml.loader import SafeLoader
|
||||
|
||||
from adapters.services.rom_converto import normalize_platform_formats
|
||||
from config import (
|
||||
DB_HOST,
|
||||
DB_NAME,
|
||||
@@ -401,6 +403,22 @@ VALID_SCAN_PRIORITY_SOURCES = frozenset(
|
||||
VALID_SCAN_REGION_MODES = frozenset({"prefer_rom_tags", "prefer_config"})
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class ConvertoConfig:
|
||||
download_conversion_enabled: bool = False
|
||||
cache_ttl_hours: int = 24
|
||||
# 0 leaves the cache unbounded.
|
||||
cache_max_size_gb: int = 20
|
||||
platform_formats: dict[str, str] = dataclasses.field(default_factory=dict)
|
||||
|
||||
|
||||
# The smallest value each integer converto.* setting accepts.
|
||||
CONVERTO_INT_MINIMUMS: Final[dict[str, int]] = {
|
||||
"cache_ttl_hours": 1,
|
||||
"cache_max_size_gb": 0,
|
||||
}
|
||||
|
||||
|
||||
class EjsControls(TypedDict):
|
||||
_0: dict[int, EjsControlsButton] # button_number -> EjsControlsButton
|
||||
_1: dict[int, EjsControlsButton]
|
||||
@@ -506,6 +524,7 @@ class Config:
|
||||
GAMELIST_MEDIA_IMAGE: MetadataMediaType
|
||||
STREAMING_ENABLED: bool
|
||||
STREAMING_CONTAINERS: list[StreamingContainer]
|
||||
CONVERTO: ConvertoConfig
|
||||
|
||||
def __init__(self, **entries: Any) -> None:
|
||||
self.__dict__.update(entries)
|
||||
@@ -879,6 +898,12 @@ class ConfigManager:
|
||||
STREAMING_CONTAINERS=pydash.get(
|
||||
self._raw_config, "streaming.containers", []
|
||||
),
|
||||
CONVERTO=ConvertoConfig(
|
||||
**{
|
||||
key: pydash.get(self._raw_config, f"converto.{key}", default)
|
||||
for key, default in dataclasses.asdict(ConvertoConfig()).items()
|
||||
}
|
||||
),
|
||||
STRUCTURE_TEMPLATES=pydash.get(
|
||||
self._raw_config, "filesystem.structure", {}
|
||||
),
|
||||
@@ -1281,6 +1306,31 @@ class ConfigManager:
|
||||
len(legacy_containers),
|
||||
)
|
||||
|
||||
if not isinstance(self.config.CONVERTO.download_conversion_enabled, bool):
|
||||
log.critical(
|
||||
"Invalid config.yml: converto.download_conversion_enabled must be a boolean"
|
||||
)
|
||||
sys.exit(3)
|
||||
|
||||
for key, minimum in CONVERTO_INT_MINIMUMS.items():
|
||||
value = getattr(self.config.CONVERTO, key)
|
||||
if isinstance(value, bool) or not isinstance(value, int) or value < minimum:
|
||||
log.critical(
|
||||
f"Invalid config.yml: converto.{key} must be an integer >= {minimum}"
|
||||
)
|
||||
sys.exit(3)
|
||||
|
||||
try:
|
||||
self.config.CONVERTO.platform_formats = normalize_platform_formats(
|
||||
self._validated_platform_map(
|
||||
self.config.CONVERTO.platform_formats or {},
|
||||
"converto.platform_formats",
|
||||
)
|
||||
)
|
||||
except ValueError as exc:
|
||||
log.critical(f"Invalid config.yml: converto.platform_formats: {exc}")
|
||||
sys.exit(3)
|
||||
|
||||
def get_config(self) -> Config:
|
||||
try:
|
||||
with open(self.config_file, "r") as config_file:
|
||||
@@ -1366,6 +1416,7 @@ class ConfigManager:
|
||||
"export": self.config.PEGASUS_AUTO_EXPORT_ON_SCAN,
|
||||
},
|
||||
},
|
||||
"converto": dataclasses.asdict(self.config.CONVERTO),
|
||||
}
|
||||
|
||||
# The streaming section isn't editable at runtime, but it must survive
|
||||
@@ -1500,5 +1551,10 @@ class ConfigManager:
|
||||
self.config.PEGASUS_AUTO_EXPORT_ON_SCAN = pegasus_export
|
||||
self._update_config_file()
|
||||
|
||||
def update_converto_settings(self, converto: ConvertoConfig) -> None:
|
||||
"""Replace the whole converto.* section and persist it to config.yml."""
|
||||
self.config.CONVERTO = converto
|
||||
self._update_config_file()
|
||||
|
||||
|
||||
config_manager = ConfigManager()
|
||||
|
||||
@@ -1,7 +1,13 @@
|
||||
from fastapi import HTTPException, Request, status
|
||||
from pydantic import BaseModel, field_validator
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from adapters.services.rom_converto import (
|
||||
DOWNLOAD_FORMATS,
|
||||
LIBRARY_TARGETS_BY_PLATFORM,
|
||||
normalize_platform_formats,
|
||||
)
|
||||
from config.config_manager import (
|
||||
CONVERTO_INT_MINIMUMS,
|
||||
DEFAULT_EXCLUDED_EXTENSIONS,
|
||||
DEFAULT_EXCLUDED_FILES,
|
||||
DEFAULT_EXCLUDED_MULTI_FILE_DIRS,
|
||||
@@ -9,6 +15,7 @@ from config.config_manager import (
|
||||
VALID_GAMELIST_IMAGE_TYPES,
|
||||
VALID_GAMELIST_THUMBNAIL_TYPES,
|
||||
VALID_SCAN_PRIORITY_SOURCES,
|
||||
ConvertoConfig,
|
||||
ExclusionType,
|
||||
MetadataMediaType,
|
||||
)
|
||||
@@ -117,6 +124,19 @@ class ScanSettingsPayload(BaseModel):
|
||||
return value
|
||||
|
||||
|
||||
class ConvertoSettingsPayload(BaseModel):
|
||||
"""The converto.* settings editable at runtime; `cache_ttl_hours` stays config.yml-only."""
|
||||
|
||||
download_conversion_enabled: bool
|
||||
cache_max_size_gb: int = Field(ge=CONVERTO_INT_MINIMUMS["cache_max_size_gb"])
|
||||
platform_formats: dict[str, str]
|
||||
|
||||
@field_validator("platform_formats")
|
||||
@classmethod
|
||||
def validate_platform_formats(cls, value: dict[str, str]) -> dict[str, str]:
|
||||
return normalize_platform_formats(value)
|
||||
|
||||
|
||||
@router.get("")
|
||||
def get_config(request: Request) -> ConfigResponse:
|
||||
"""Get config endpoint
|
||||
@@ -169,6 +189,12 @@ def get_config(request: Request) -> ConfigResponse:
|
||||
GAMELIST_MEDIA_THUMBNAIL=cfg.GAMELIST_MEDIA_THUMBNAIL,
|
||||
GAMELIST_MEDIA_IMAGE=cfg.GAMELIST_MEDIA_IMAGE,
|
||||
PEGASUS_AUTO_EXPORT_ON_SCAN=cfg.PEGASUS_AUTO_EXPORT_ON_SCAN,
|
||||
CONVERTO=cfg.CONVERTO,
|
||||
CONVERTO_LIBRARY_TARGETS={
|
||||
slug: sorted(targets)
|
||||
for slug, targets in LIBRARY_TARGETS_BY_PLATFORM.items()
|
||||
},
|
||||
CONVERTO_DOWNLOAD_FORMATS=DOWNLOAD_FORMATS,
|
||||
)
|
||||
|
||||
|
||||
@@ -338,3 +364,24 @@ async def update_scan_settings(request: Request, payload: ScanSettingsPayload) -
|
||||
# cached gallery sidecar is stale the moment the order changes.
|
||||
if region_priority_changed:
|
||||
db_rom_handler.invalidate_filter_values_cache()
|
||||
|
||||
|
||||
@protected_route(router.put, "/converto_settings", [Scope.PLATFORMS_WRITE])
|
||||
async def update_converto_settings(
|
||||
request: Request, payload: ConvertoSettingsPayload
|
||||
) -> None:
|
||||
"""Update the runtime-editable converto.* settings"""
|
||||
|
||||
current = cm.get_config().CONVERTO
|
||||
try:
|
||||
cm.update_converto_settings(
|
||||
ConvertoConfig(
|
||||
**payload.model_dump(), cache_ttl_hours=current.cache_ttl_hours
|
||||
)
|
||||
)
|
||||
except ConfigNotWritableException as exc:
|
||||
log.critical(exc.message)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=exc.message
|
||||
) from exc
|
||||
_record_config(request, "converto_settings", "update")
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
from typing import TypedDict
|
||||
|
||||
from config.config_manager import EjsControls, MetadataMediaType, NetplayICEServer
|
||||
from config.config_manager import (
|
||||
ConvertoConfig,
|
||||
EjsControls,
|
||||
MetadataMediaType,
|
||||
NetplayICEServer,
|
||||
)
|
||||
|
||||
|
||||
class ConfigResponse(TypedDict):
|
||||
@@ -40,3 +45,8 @@ class ConfigResponse(TypedDict):
|
||||
GAMELIST_MEDIA_THUMBNAIL: MetadataMediaType
|
||||
GAMELIST_MEDIA_IMAGE: MetadataMediaType
|
||||
PEGASUS_AUTO_EXPORT_ON_SCAN: bool
|
||||
CONVERTO: ConvertoConfig
|
||||
# Platform slug -> the formats its library can be stored in.
|
||||
CONVERTO_LIBRARY_TARGETS: dict[str, list[str]]
|
||||
# Platform slug -> input extension -> the formats a download can be converted to.
|
||||
CONVERTO_DOWNLOAD_FORMATS: dict[str, dict[str, list[str]]]
|
||||
|
||||
@@ -122,6 +122,7 @@ from models.rom import (
|
||||
PINNED_MEDIA_MAX_ITEMS,
|
||||
TITLE_ID_MAX_LENGTH,
|
||||
Rom,
|
||||
RomFile,
|
||||
RomIdentity,
|
||||
RomUserStatus,
|
||||
SaveTargetLayout,
|
||||
@@ -130,6 +131,13 @@ from models.rom import (
|
||||
)
|
||||
from utils import switch
|
||||
from utils.background_tasks import fire_and_forget
|
||||
from utils.conversion_cache import (
|
||||
RETRY_AFTER_SECONDS,
|
||||
FormatOutcome,
|
||||
get_redirect_path,
|
||||
parse_formats,
|
||||
resolve_format_download,
|
||||
)
|
||||
from utils.database import safe_int, safe_str_to_bool
|
||||
from utils.filesystem import check_filename_length, sanitize_filename
|
||||
from utils.hashing import crc32_to_hex
|
||||
@@ -1312,11 +1320,86 @@ def get_rom(
|
||||
return DetailedRomSchema.from_orm_with_request(rom, request)
|
||||
|
||||
|
||||
CONTENT_RESPONSES: Final[dict[int | str, dict[str, Any]]] = {
|
||||
status.HTTP_202_ACCEPTED: {
|
||||
"description": "The requested format is converting; retry after Retry-After."
|
||||
},
|
||||
status.HTTP_404_NOT_FOUND: {},
|
||||
status.HTTP_406_NOT_ACCEPTABLE: {
|
||||
"description": "No requested format can be served for this download."
|
||||
},
|
||||
}
|
||||
|
||||
FormatQuery = Annotated[
|
||||
str | None,
|
||||
Query(
|
||||
alias="format",
|
||||
description="Comma-separated formats the client can read, such as "
|
||||
"`zso,iso`. The stored file is served when its format is listed, else "
|
||||
"a converted copy: 202 with Retry-After while it converts, 406 when no "
|
||||
"listed format can be produced. Single-file downloads only.",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
async def _negotiate_format(
|
||||
request: Request,
|
||||
rom: Rom,
|
||||
files: list[RomFile],
|
||||
raw_formats: str | None,
|
||||
*,
|
||||
start: bool,
|
||||
) -> Response | None:
|
||||
"""The response a `?format=` download gets, or None to serve the stored file."""
|
||||
formats = parse_formats(raw_formats or "")
|
||||
if not formats:
|
||||
return None
|
||||
if len(files) != 1:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_406_NOT_ACCEPTABLE,
|
||||
detail="A format can only be requested for a single file",
|
||||
)
|
||||
|
||||
file = files[0]
|
||||
# DISABLE_DOWNLOAD_ENDPOINT_AUTH and KIOSK_MODE open this route to visitors,
|
||||
# and a conversion is minutes of CPU.
|
||||
allowed = request.user.is_authenticated and not request.user.is_kiosk_guest
|
||||
resolution = await resolve_format_download(
|
||||
rom, file, formats, allowed=allowed, start=start, touch=start
|
||||
)
|
||||
if resolution.outcome == FormatOutcome.ORIGINAL:
|
||||
return None
|
||||
if resolution.outcome == FormatOutcome.PENDING:
|
||||
return Response(
|
||||
status_code=status.HTTP_202_ACCEPTED,
|
||||
headers={"Retry-After": str(RETRY_AFTER_SECONDS)},
|
||||
)
|
||||
if resolution.outcome == FormatOutcome.UNAVAILABLE or resolution.path is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_406_NOT_ACCEPTABLE,
|
||||
detail=f"{file.file_name} can't be served as {', '.join(formats)}",
|
||||
)
|
||||
|
||||
converted = resolution.path
|
||||
if DEV_MODE:
|
||||
return FileResponse(
|
||||
path=converted,
|
||||
filename=converted.name,
|
||||
headers={
|
||||
"Content-Disposition": content_disposition(converted.name),
|
||||
"Content-Type": "application/octet-stream",
|
||||
},
|
||||
)
|
||||
return FileRedirectResponse(
|
||||
download_path=get_redirect_path(converted), filename=converted.name
|
||||
)
|
||||
|
||||
|
||||
@protected_route(
|
||||
router.head,
|
||||
"/{id}/content/{file_name}",
|
||||
[] if DISABLE_DOWNLOAD_ENDPOINT_AUTH else [Scope.ROMS_READ],
|
||||
responses={status.HTTP_404_NOT_FOUND: {}},
|
||||
responses=CONTENT_RESPONSES,
|
||||
)
|
||||
async def head_rom_content(
|
||||
request: Request,
|
||||
@@ -1328,6 +1411,7 @@ async def head_rom_content(
|
||||
description="Comma-separated list of file ids to download for multi-part roms."
|
||||
),
|
||||
] = None,
|
||||
formats: FormatQuery = None,
|
||||
) -> Response:
|
||||
"""Retrieve head information for a rom file download."""
|
||||
|
||||
@@ -1350,6 +1434,10 @@ async def head_rom_content(
|
||||
detail=f"No files found for ROM {id}",
|
||||
)
|
||||
|
||||
# Report what a GET would serve, but never start a conversion.
|
||||
if negotiated := await _negotiate_format(request, rom, files, formats, start=False):
|
||||
return negotiated
|
||||
|
||||
# Serve the file directly in development mode for emulatorjs
|
||||
if DEV_MODE:
|
||||
if len(files) == 1:
|
||||
@@ -1410,7 +1498,7 @@ async def head_rom_content(
|
||||
router.get,
|
||||
"/{id}/content/{file_name}",
|
||||
[] if DISABLE_DOWNLOAD_ENDPOINT_AUTH else [Scope.ROMS_READ],
|
||||
responses={status.HTTP_404_NOT_FOUND: {}},
|
||||
responses=CONTENT_RESPONSES,
|
||||
)
|
||||
async def get_rom_content(
|
||||
request: Request,
|
||||
@@ -1429,6 +1517,7 @@ async def get_rom_content(
|
||||
"recorded as a player load rather than a download."
|
||||
),
|
||||
] = "download",
|
||||
formats: FormatQuery = None,
|
||||
) -> Response:
|
||||
"""Download a rom.
|
||||
|
||||
@@ -1494,6 +1583,13 @@ async def get_rom_content(
|
||||
)
|
||||
return response
|
||||
|
||||
if negotiated := await _negotiate_format(request, rom, files, formats, start=True):
|
||||
return (
|
||||
served(negotiated)
|
||||
if negotiated.status_code == status.HTTP_200_OK
|
||||
else negotiated
|
||||
)
|
||||
|
||||
m3u_files = playlist_files(files)
|
||||
|
||||
# Serve the file directly in development mode for emulatorjs
|
||||
|
||||
@@ -56,6 +56,7 @@ VISIBLE_SCHEDULED_TASKS: Final[dict[str, TaskSpec]] = {
|
||||
"build_recommendations",
|
||||
"convert_images_to_webp",
|
||||
"cleanup_zip_cache",
|
||||
"cleanup_conversion_cache",
|
||||
"cleanup_orphaned_resources",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -7,7 +7,8 @@ from logger.formatter import highlight as hl
|
||||
from logger.logger import log
|
||||
from models.rom import Rom
|
||||
|
||||
_STAGE_PREFIX = ".romm_tmp_"
|
||||
# Scans and the watcher skip dot-prefixed entries, so staged files stay unseen.
|
||||
STAGE_PREFIX = ".romm_tmp_"
|
||||
|
||||
# Parallel uploads each promote the ROM, maybe on different gunicorn workers.
|
||||
# Keyed by the target folder, since lone files sharing a stem promote into one.
|
||||
@@ -39,7 +40,7 @@ async def _promote(rom: Rom) -> Rom:
|
||||
origin = f"{fs_path}/{fs_name}"
|
||||
dest_dir = f"{fs_path}/{folder}"
|
||||
final = f"{dest_dir}/{fs_name}"
|
||||
staged = f"{fs_path}/{_STAGE_PREFIX}{fs_name}"
|
||||
staged = f"{fs_path}/{STAGE_PREFIX}{fs_name}"
|
||||
extensionless = folder == fs_name
|
||||
|
||||
# Extensionless dest_dir is the file's own path; only a directory collides.
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
"""Convert each matched ROM to its platform's library format, in place."""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Final
|
||||
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
|
||||
from adapters.services.rom_converto import (
|
||||
Operation,
|
||||
RomConvertoError,
|
||||
file_format,
|
||||
resolve_operation,
|
||||
rom_converto_service,
|
||||
)
|
||||
from config.config_manager import config_manager as cm
|
||||
from handler.database import db_platform_handler, db_rom_handler
|
||||
from handler.filesystem import fs_rom_handler
|
||||
from handler.rom_conversion import STAGE_PREFIX
|
||||
from handler.rom_files import refresh_rom_files
|
||||
from logger.formatter import highlight as hl
|
||||
from logger.logger import log
|
||||
from models.rom import Rom, RomFile, RomFileCategory
|
||||
from tasks.registry import CONVERT_LIBRARY_SPEC
|
||||
from tasks.scheduled.convert_images_to_webp import ConversionStats
|
||||
from tasks.tasks import Task
|
||||
from utils.context import initialize_context
|
||||
from utils.filesystem import LINK_FALLBACK_ERRNOS
|
||||
|
||||
_CUE_FILE_LINE: Final = re.compile(r'^\s*FILE\s+(?:"([^"]+)"|(\S+))', re.IGNORECASE)
|
||||
_CUE_EXT: Final = ".cue"
|
||||
_PLAYLIST_EXT: Final = ".m3u"
|
||||
_CONVERT_STAGE_PREFIX: Final = f"{STAGE_PREFIX}convert_"
|
||||
|
||||
|
||||
@dataclass
|
||||
class ConvertLibraryStats:
|
||||
converted: int = 0
|
||||
already_converted: int = 0
|
||||
# Converted files no longer hash-match a DAT, so only matched roms convert.
|
||||
unmatched: int = 0
|
||||
unsupported: int = 0
|
||||
failed: int = 0
|
||||
bytes_saved: int = 0
|
||||
|
||||
|
||||
def _cue_tracks(cue: Path) -> list[Path]:
|
||||
"""The track files a cue sheet references, all beside it."""
|
||||
tracks: list[Path] = []
|
||||
for line in cue.read_text(encoding="utf-8-sig", errors="replace").splitlines():
|
||||
match = _CUE_FILE_LINE.match(line)
|
||||
if match is None:
|
||||
continue
|
||||
name = match.group(1) or match.group(2)
|
||||
# A track outside the cue's folder isn't this rom's to delete.
|
||||
if Path(name).name != name:
|
||||
raise ValueError(f"{cue.name} references a track outside its folder")
|
||||
tracks.append(cue.with_name(name))
|
||||
return tracks
|
||||
|
||||
|
||||
def _shared_tracks(cue: Path, tracks: list[Path]) -> set[Path]:
|
||||
"""The tracks of `cue` that another cue beside it also references."""
|
||||
others: set[Path] = set()
|
||||
for other in cue.parent.iterdir():
|
||||
if other == cue or other.suffix.lower() != _CUE_EXT or not other.is_file():
|
||||
continue
|
||||
with contextlib.suppress(OSError, ValueError):
|
||||
others.update(_cue_tracks(other))
|
||||
return set(tracks) & others
|
||||
|
||||
|
||||
def _renamed_entry(line: str, renames: dict[str, str]) -> str:
|
||||
entry = line.strip()
|
||||
return renames.get(os.path.normpath(entry), line) if entry else line
|
||||
|
||||
|
||||
def _rewritten_playlists(folder: Path, renames: dict[str, str]) -> dict[Path, str]:
|
||||
"""The playlists in `folder` that name a renamed file, with their new text."""
|
||||
rewrites: dict[Path, str] = {}
|
||||
for playlist in folder.iterdir():
|
||||
if playlist.suffix.lower() != _PLAYLIST_EXT or not playlist.is_file():
|
||||
continue
|
||||
lines = playlist.read_text(encoding="utf-8-sig", errors="replace").splitlines()
|
||||
rewritten = [_renamed_entry(line, renames) for line in lines]
|
||||
if rewritten != lines:
|
||||
rewrites[playlist] = "\n".join(rewritten) + "\n"
|
||||
return rewrites
|
||||
|
||||
|
||||
async def _rewrite_playlists(folder: Path, renames: dict[str, str]) -> None:
|
||||
"""Point each playlist entry for a converted file at its new name."""
|
||||
rewrites = await asyncio.to_thread(_rewritten_playlists, folder, renames)
|
||||
for playlist, text in rewrites.items():
|
||||
async with fs_rom_handler._atomic_write(playlist) as staged:
|
||||
await asyncio.to_thread(staged.write_text, text)
|
||||
|
||||
|
||||
def _sources(src: Path, final: Path, input_ext: str) -> list[Path]:
|
||||
"""The files a conversion of `src` replaces, refusing one that would overwrite."""
|
||||
if final.exists():
|
||||
raise FileExistsError(f"{final.name} already exists")
|
||||
if input_ext != _CUE_EXT:
|
||||
return [src]
|
||||
tracks = _cue_tracks(src)
|
||||
if shared := _shared_tracks(src, tracks):
|
||||
raise ValueError(
|
||||
f"{src.name} shares {sorted(p.name for p in shared)} with another cue"
|
||||
)
|
||||
return [src, *tracks]
|
||||
|
||||
|
||||
def _drop_stale_stages(folder: Path) -> None:
|
||||
"""Remove stage dirs a killed conversion left, none recent enough to be live."""
|
||||
cutoff = time.time() - CONVERT_LIBRARY_SPEC.timeout
|
||||
for stage in folder.glob(f"{_CONVERT_STAGE_PREFIX}*"):
|
||||
with contextlib.suppress(OSError):
|
||||
if stage.is_dir() and stage.stat().st_mtime < cutoff:
|
||||
shutil.rmtree(stage, ignore_errors=True)
|
||||
|
||||
|
||||
def _publish(stage_dir: Path, final: Path) -> None:
|
||||
"""Move the staged output to `final`, never replacing a file that appeared there."""
|
||||
staged = stage_dir / final.name
|
||||
try:
|
||||
os.link(staged, final)
|
||||
except OSError as exc:
|
||||
if exc.errno not in LINK_FALLBACK_ERRNOS:
|
||||
raise
|
||||
if final.exists():
|
||||
raise FileExistsError(f"{final.name} already exists") from None
|
||||
os.replace(staged, final)
|
||||
|
||||
|
||||
def _replace_sources(sources: list[Path], final: Path) -> int:
|
||||
"""Delete the converted files and return the bytes the conversion saved."""
|
||||
saved = sum(p.stat().st_size for p in sources if p.exists()) - final.stat().st_size
|
||||
for path in sources:
|
||||
try:
|
||||
path.unlink(missing_ok=True)
|
||||
except OSError as exc:
|
||||
log.warning(
|
||||
f"Converted to {hl(final.name)} but could not delete {path}: {exc}"
|
||||
)
|
||||
return saved
|
||||
|
||||
|
||||
async def _convert_beside(
|
||||
src: Path, final: Path, operation: Operation, stage_parent: Path
|
||||
) -> None:
|
||||
"""Convert `src` to `final` beside it, staging in `stage_parent` and leaving `src` in place."""
|
||||
stage_dir = Path(
|
||||
await asyncio.to_thread(
|
||||
tempfile.mkdtemp, prefix=_CONVERT_STAGE_PREFIX, dir=stage_parent
|
||||
)
|
||||
)
|
||||
try:
|
||||
await rom_converto_service.convert(
|
||||
operation, src=src, out=stage_dir / final.name
|
||||
)
|
||||
await asyncio.to_thread(_publish, stage_dir, final)
|
||||
finally:
|
||||
await asyncio.to_thread(shutil.rmtree, stage_dir, True)
|
||||
|
||||
|
||||
def _names_taken(rom: Rom, names: list[str]) -> list[str]:
|
||||
"""The `names` another rom row beside `rom` already holds."""
|
||||
if not names:
|
||||
return []
|
||||
owned = db_rom_handler.get_roms_by_fs_name(rom.platform_id, names)
|
||||
return sorted(n for n in names if f"{rom.fs_path}/{n}" in owned)
|
||||
|
||||
|
||||
def _game_files(rom: Rom) -> list[RomFile]:
|
||||
return [
|
||||
f
|
||||
for f in rom.files
|
||||
if f.is_top_level
|
||||
and not f.missing_from_fs
|
||||
and f.category in (None, RomFileCategory.GAME)
|
||||
]
|
||||
|
||||
|
||||
class ConvertLibraryTask(Task):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(CONVERT_LIBRARY_SPEC)
|
||||
|
||||
async def _convert_rom(
|
||||
self, rom: Rom, target: str, stats: ConvertLibraryStats
|
||||
) -> None:
|
||||
single_file = rom.has_simple_single_file
|
||||
planned: list[tuple[RomFile, Operation, str]] = []
|
||||
for rom_file in _game_files(rom):
|
||||
if file_format(rom_file.file_name) == target:
|
||||
stats.already_converted += 1
|
||||
continue
|
||||
resolved = resolve_operation(
|
||||
rom.platform_slug, target, rom_file.file_name, lossless=True
|
||||
)
|
||||
if resolved is None:
|
||||
stats.unsupported += 1
|
||||
continue
|
||||
name = resolved[0].output_name(Path(rom.fs_name), resolved[1])
|
||||
if single_file and _names_taken(rom, [name]):
|
||||
log.warning(
|
||||
f"Not converting {hl(rom.fs_name)}: a rom named {name} exists"
|
||||
)
|
||||
stats.failed += 1
|
||||
continue
|
||||
planned.append((rom_file, *resolved))
|
||||
if not planned:
|
||||
return
|
||||
if not rom.is_identified:
|
||||
stats.unmatched += 1
|
||||
return
|
||||
|
||||
# A rom folder lists dot-prefixed subfolders as its files, so stage beside it.
|
||||
stage_parent = fs_rom_handler.validate_path(rom.fs_path)
|
||||
await asyncio.to_thread(_drop_stale_stages, stage_parent)
|
||||
# Originals go only once the rom and its playlists point at the new files.
|
||||
published: list[tuple[Path, Path, list[Path]]] = []
|
||||
for rom_file, operation, input_ext in planned:
|
||||
src = fs_rom_handler.validate_path(rom_file.full_path)
|
||||
final = src.with_name(operation.output_name(src, input_ext))
|
||||
try:
|
||||
sources = await asyncio.to_thread(_sources, src, final, input_ext)
|
||||
if single_file and (
|
||||
taken := _names_taken(rom, [t.name for t in sources[1:]])
|
||||
):
|
||||
raise ValueError(f"{taken} belong to other roms")
|
||||
await _convert_beside(src, final, operation, stage_parent)
|
||||
except (RomConvertoError, OSError, ValueError) as exc:
|
||||
log.warning(f"Could not convert {hl(rom_file.file_name)}: {exc}")
|
||||
stats.failed += 1
|
||||
continue
|
||||
published.append((src, final, sources))
|
||||
|
||||
if not published:
|
||||
return
|
||||
renames = {src.name: final.name for src, final, _ in published}
|
||||
try:
|
||||
if single_file:
|
||||
db_rom_handler.update_rom(rom.id, {"fs_name": renames[rom.fs_name]})
|
||||
else:
|
||||
await _rewrite_playlists(
|
||||
fs_rom_handler.validate_path(rom.full_path), renames
|
||||
)
|
||||
except (OSError, SQLAlchemyError) as exc:
|
||||
log.warning(f"Kept the originals of {hl(rom.fs_name)}: {exc}")
|
||||
stats.failed += len(published)
|
||||
if single_file:
|
||||
for _, final, _ in published:
|
||||
final.unlink(missing_ok=True)
|
||||
return
|
||||
|
||||
for src, final, sources in published:
|
||||
stats.bytes_saved += await asyncio.to_thread(
|
||||
_replace_sources, sources, final
|
||||
)
|
||||
stats.converted += 1
|
||||
log.info(f"Converted {hl(src.name)} to {hl(final.name)}")
|
||||
refreshed = db_rom_handler.get_rom(rom.id)
|
||||
if refreshed is not None:
|
||||
await refresh_rom_files(refreshed)
|
||||
|
||||
@initialize_context()
|
||||
async def run(self, platform_id: int | None = None) -> dict[str, int]:
|
||||
"""Convert every matched ROM on platforms that have a library format."""
|
||||
log.info(f"Starting {self.spec.title} task...")
|
||||
stats = ConvertLibraryStats()
|
||||
if not await rom_converto_service.is_enabled():
|
||||
log.warning(
|
||||
f"{self.spec.title} needs rom-converto; set ROM_CONVERTO_ENABLED"
|
||||
)
|
||||
return asdict(stats)
|
||||
|
||||
formats = cm.get_config().CONVERTO.platform_formats
|
||||
rom_ids: list[tuple[int, str]] = []
|
||||
for platform in db_platform_handler.get_platforms():
|
||||
if platform_id is not None and platform.id != platform_id:
|
||||
continue
|
||||
target = formats.get(platform.slug)
|
||||
if target:
|
||||
rom_ids.extend(
|
||||
(rom_id, target)
|
||||
for rom_id in db_rom_handler.get_rom_ids(platform_ids=[platform.id])
|
||||
)
|
||||
|
||||
progress = ConversionStats()
|
||||
progress.update(total=len(rom_ids))
|
||||
for index, (rom_id, target) in enumerate(rom_ids):
|
||||
rom = db_rom_handler.get_rom(rom_id)
|
||||
if rom is not None:
|
||||
try:
|
||||
await self._convert_rom(rom, target, stats)
|
||||
except Exception as exc:
|
||||
log.exception(f"Could not convert {hl(rom.fs_name)}: {exc}")
|
||||
stats.failed += 1
|
||||
progress.update(processed=index + 1, errors=stats.failed)
|
||||
|
||||
log.info(
|
||||
f"{self.spec.title} completed: {stats.converted} converted, "
|
||||
f"{stats.failed} failed, {stats.bytes_saved} bytes saved"
|
||||
)
|
||||
return asdict(stats)
|
||||
|
||||
|
||||
convert_library_task = ConvertLibraryTask()
|
||||
@@ -113,6 +113,15 @@ CLEANUP_ZIP_CACHE_SPEC: Final = TaskSpec(
|
||||
cron_string=SCHEDULED_CLEANUP_ZIP_CACHE_CRON,
|
||||
)
|
||||
|
||||
CLEANUP_CONVERSION_CACHE_SPEC: Final = TaskSpec(
|
||||
implementation="tasks.scheduled.cleanup_conversion_cache.cleanup_conversion_cache_task",
|
||||
title="Scheduled conversion cache cleanup",
|
||||
description="Removes stale converted download files based on TTL",
|
||||
task_type=TaskType.CLEANUP,
|
||||
enabled=True,
|
||||
cron_string="0 4 * * *",
|
||||
)
|
||||
|
||||
CLEANUP_ORPHANED_RESOURCES_SPEC: Final = TaskSpec(
|
||||
implementation="tasks.scheduled.cleanup_orphaned_resources.cleanup_orphaned_resources_task",
|
||||
title="Cleanup orphaned resources",
|
||||
@@ -232,6 +241,20 @@ RECOMPUTE_SAVE_CONTENT_HASHES_SPEC: Final = TaskSpec(
|
||||
manual_run=True,
|
||||
)
|
||||
|
||||
CONVERT_LIBRARY_SPEC: Final = TaskSpec(
|
||||
implementation="tasks.manual.convert_library.convert_library_task",
|
||||
title="Convert library",
|
||||
description=(
|
||||
"Convert each matched ROM to its platform's library format, "
|
||||
"replacing the original files"
|
||||
),
|
||||
task_type=TaskType.CONVERSION,
|
||||
enabled=True,
|
||||
manual_run=True,
|
||||
# One conversion after another, each up to ROM_CONVERTO_TIMEOUT.
|
||||
timeout=SCAN_TIMEOUT,
|
||||
)
|
||||
|
||||
# The keys are the names the API and the cron schedule address a task by, and
|
||||
# they end up in the job payload, so they outlive any given release. Every task
|
||||
# that runs on a schedule belongs here; which of them the API surfaces is the
|
||||
@@ -243,6 +266,7 @@ SCHEDULED_TASKS: Final[dict[str, TaskSpec]] = {
|
||||
"build_recommendations": BUILD_RECOMMENDATIONS_SPEC,
|
||||
"convert_images_to_webp": CONVERT_IMAGES_TO_WEBP_SPEC,
|
||||
"cleanup_zip_cache": CLEANUP_ZIP_CACHE_SPEC,
|
||||
"cleanup_conversion_cache": CLEANUP_CONVERSION_CACHE_SPEC,
|
||||
"cleanup_orphaned_resources": CLEANUP_ORPHANED_RESOURCES_SPEC,
|
||||
"cleanup_netplay": CLEANUP_NETPLAY_SPEC,
|
||||
"cleanup_upload_tmp": CLEANUP_UPLOAD_TMP_SPEC,
|
||||
@@ -258,6 +282,7 @@ MANUAL_TASKS: Final[dict[str, TaskSpec]] = {
|
||||
"cleanup_missing_firmware": CLEANUP_MISSING_FIRMWARE_SPEC,
|
||||
"sync_folder_scan": SYNC_FOLDER_SCAN_SPEC,
|
||||
"recompute_save_content_hashes": RECOMPUTE_SAVE_CONTENT_HASHES_SPEC,
|
||||
"convert_library": CONVERT_LIBRARY_SPEC,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
from logger.logger import log
|
||||
from tasks.registry import CLEANUP_CONVERSION_CACHE_SPEC
|
||||
from tasks.tasks import PeriodicTask
|
||||
from utils.conversion_cache import cleanup_stale_conversions
|
||||
|
||||
|
||||
class CleanupConversionCacheTask(PeriodicTask):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(CLEANUP_CONVERSION_CACHE_SPEC)
|
||||
|
||||
async def run(self) -> None:
|
||||
if not self.spec.enabled:
|
||||
return
|
||||
|
||||
deleted = cleanup_stale_conversions()
|
||||
if deleted:
|
||||
log.info(f"Cleaned up {deleted} stale converted download caches")
|
||||
|
||||
|
||||
cleanup_conversion_cache_task = CleanupConversionCacheTask()
|
||||
@@ -0,0 +1,392 @@
|
||||
import asyncio
|
||||
import stat
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from adapters.services import rom_converto
|
||||
from adapters.services.rom_converto import (
|
||||
DOWNLOAD_FORMATS,
|
||||
LIBRARY_TARGETS_BY_PLATFORM,
|
||||
Operation,
|
||||
RomConvertoBinaryNotFoundError,
|
||||
RomConvertoOperationError,
|
||||
RomConvertoService,
|
||||
RomConvertoTimeoutError,
|
||||
canonical_format,
|
||||
file_format,
|
||||
resolve_operation,
|
||||
)
|
||||
from utils.filesystem import SERVED_FILE_MODE
|
||||
|
||||
|
||||
class FakeProc:
|
||||
def __init__(
|
||||
self,
|
||||
returncode: int = 0,
|
||||
stdout: bytes = b"",
|
||||
stderr: bytes = b"",
|
||||
delay: float = 0.0,
|
||||
):
|
||||
self._stdout = stdout
|
||||
self._stderr = stderr
|
||||
self._delay = delay
|
||||
self.killed = False
|
||||
self.returncode = returncode
|
||||
|
||||
async def communicate(self) -> tuple[bytes, bytes]:
|
||||
if self._delay and not self.killed:
|
||||
await asyncio.sleep(self._delay)
|
||||
return self._stdout, self._stderr
|
||||
|
||||
def kill(self) -> None:
|
||||
self.killed = True
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def service() -> RomConvertoService:
|
||||
return RomConvertoService()
|
||||
|
||||
|
||||
class TestRun:
|
||||
async def test_binary_missing_raises(self) -> None:
|
||||
with (
|
||||
patch("shutil.which", return_value=None),
|
||||
pytest.raises(RomConvertoBinaryNotFoundError),
|
||||
):
|
||||
await rom_converto._run(["info", "--json", "x"], timeout_seconds=1)
|
||||
|
||||
async def test_timeout_kills_process(self) -> None:
|
||||
proc = FakeProc(delay=5.0)
|
||||
with (
|
||||
patch("shutil.which", return_value="/usr/bin/rom-converto"),
|
||||
patch("asyncio.create_subprocess_exec", return_value=proc),
|
||||
pytest.raises(RomConvertoTimeoutError),
|
||||
):
|
||||
await rom_converto._run(["info", "--json", "x"], timeout_seconds=0.01)
|
||||
assert proc.killed is True
|
||||
|
||||
async def test_cancellation_kills_process(self) -> None:
|
||||
proc = FakeProc(delay=5.0)
|
||||
with (
|
||||
patch("shutil.which", return_value="/usr/bin/rom-converto"),
|
||||
patch("asyncio.create_subprocess_exec", return_value=proc),
|
||||
):
|
||||
run = asyncio.create_task(
|
||||
rom_converto._run(["info", "--json", "x"], timeout_seconds=10)
|
||||
)
|
||||
await asyncio.sleep(0.01)
|
||||
run.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await run
|
||||
assert proc.killed is True
|
||||
|
||||
|
||||
class TestIsEnabled:
|
||||
async def test_disabled_when_config_disabled(self, service: RomConvertoService):
|
||||
with (
|
||||
patch.object(rom_converto, "ROM_CONVERTO_ENABLED", False),
|
||||
patch("shutil.which", return_value="/usr/bin/rom-converto"),
|
||||
):
|
||||
assert await service.is_enabled() is False
|
||||
|
||||
async def test_probe_fail_cached_false(self, service: RomConvertoService):
|
||||
proc = FakeProc(returncode=1)
|
||||
spawn_count = 0
|
||||
|
||||
async def spawn(*args, **kwargs):
|
||||
nonlocal spawn_count
|
||||
spawn_count += 1
|
||||
return proc
|
||||
|
||||
with (
|
||||
patch.object(rom_converto, "ROM_CONVERTO_ENABLED", True),
|
||||
patch("shutil.which", return_value="/usr/bin/rom-converto"),
|
||||
patch("asyncio.create_subprocess_exec", spawn),
|
||||
):
|
||||
assert await service.is_enabled() is False
|
||||
assert await service.is_enabled() is False
|
||||
assert spawn_count == 1
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"error",
|
||||
[OSError(8, "Exec format error"), RomConvertoTimeoutError("timed out")],
|
||||
)
|
||||
async def test_probe_that_cannot_run_is_cached_false(
|
||||
self, service: RomConvertoService, error: Exception
|
||||
):
|
||||
spawn_count = 0
|
||||
|
||||
async def spawn(*args, **kwargs):
|
||||
nonlocal spawn_count
|
||||
spawn_count += 1
|
||||
raise error
|
||||
|
||||
with (
|
||||
patch.object(rom_converto, "ROM_CONVERTO_ENABLED", True),
|
||||
patch("shutil.which", return_value="/usr/bin/rom-converto"),
|
||||
patch("asyncio.create_subprocess_exec", spawn),
|
||||
):
|
||||
assert await service.is_enabled() is False
|
||||
assert await service.is_enabled() is False
|
||||
assert spawn_count == 1
|
||||
|
||||
async def test_probe_ok_logs_version(self, service: RomConvertoService, mocker):
|
||||
log_info = mocker.patch("adapters.services.rom_converto.log.info")
|
||||
proc = FakeProc(stdout=b'{"version": "0.21.0"}')
|
||||
with (
|
||||
patch.object(rom_converto, "ROM_CONVERTO_ENABLED", True),
|
||||
patch("shutil.which", return_value="/usr/bin/rom-converto"),
|
||||
patch("asyncio.create_subprocess_exec", return_value=proc),
|
||||
):
|
||||
assert await service.is_enabled() is True
|
||||
assert log_info.call_count == 1
|
||||
|
||||
async def test_missing_binary_not_cached(self, service: RomConvertoService):
|
||||
with (
|
||||
patch.object(rom_converto, "ROM_CONVERTO_ENABLED", True),
|
||||
patch("shutil.which", return_value=None),
|
||||
):
|
||||
assert await service.is_enabled() is False
|
||||
|
||||
proc = FakeProc(stdout=b'{"version": "0.21.0"}')
|
||||
with (
|
||||
patch.object(rom_converto, "ROM_CONVERTO_ENABLED", True),
|
||||
patch("shutil.which", return_value="/usr/bin/rom-converto"),
|
||||
patch("asyncio.create_subprocess_exec", return_value=proc),
|
||||
):
|
||||
assert await service.is_enabled() is True
|
||||
|
||||
|
||||
def _capture_spawn(proc: FakeProc, calls: list[tuple[str, ...]]):
|
||||
async def spawn(*args, **kwargs):
|
||||
calls.append(args)
|
||||
return proc
|
||||
|
||||
return spawn
|
||||
|
||||
|
||||
class TestUpdateCheck:
|
||||
async def test_every_run_disables_the_update_check(self):
|
||||
calls: list[tuple[str, ...]] = []
|
||||
with (
|
||||
patch("shutil.which", return_value="/usr/bin/rom-converto"),
|
||||
patch("asyncio.create_subprocess_exec", _capture_spawn(FakeProc(), calls)),
|
||||
):
|
||||
await rom_converto._run(["capabilities"], timeout_seconds=1)
|
||||
|
||||
assert calls[0][:3] == (
|
||||
"/usr/bin/rom-converto",
|
||||
"--no-update-check",
|
||||
"capabilities",
|
||||
)
|
||||
|
||||
|
||||
class TestConvert:
|
||||
def _op(self) -> Operation:
|
||||
return Operation(
|
||||
target="chd",
|
||||
platforms=frozenset({"psp"}),
|
||||
argv=("chd", "compress"),
|
||||
input_exts=frozenset({".iso"}),
|
||||
output_ext=".chd",
|
||||
)
|
||||
|
||||
async def test_argv_is_operation_argv_plus_src_and_out(
|
||||
self, service: RomConvertoService, tmp_path: Path
|
||||
):
|
||||
recorded: list[list[str]] = []
|
||||
|
||||
src = tmp_path / "game.iso"
|
||||
out = tmp_path / "stage" / "game.chd"
|
||||
out.parent.mkdir()
|
||||
|
||||
async def fake_run(argv: list[str], timeout_seconds: float):
|
||||
recorded.append(argv)
|
||||
out.write_bytes(b"converted")
|
||||
return 0, "", ""
|
||||
|
||||
with patch.object(rom_converto, "_run", fake_run):
|
||||
await service.convert(self._op(), src, out)
|
||||
|
||||
assert recorded == [["chd", "compress", str(src), str(out)]]
|
||||
|
||||
async def test_an_owner_only_output_is_made_readable_for_nginx(
|
||||
self, service: RomConvertoService, tmp_path: Path
|
||||
):
|
||||
out = tmp_path / "game.chd"
|
||||
|
||||
async def fake_run(argv: list[str], timeout_seconds: float):
|
||||
out.write_bytes(b"converted")
|
||||
out.chmod(0o600)
|
||||
return 0, "", ""
|
||||
|
||||
with patch.object(rom_converto, "_run", fake_run):
|
||||
await service.convert(self._op(), tmp_path / "src" / "game.iso", out)
|
||||
|
||||
assert stat.S_IMODE(out.stat().st_mode) == SERVED_FILE_MODE
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"written",
|
||||
[
|
||||
pytest.param(["game.chd", "game.bin"], id="split"),
|
||||
pytest.param([], id="nothing"),
|
||||
],
|
||||
)
|
||||
async def test_output_other_than_out_raises_operation_error(
|
||||
self, service: RomConvertoService, tmp_path: Path, written: list[str]
|
||||
):
|
||||
async def fake_run(argv: list[str], timeout_seconds: float):
|
||||
for name in written:
|
||||
(tmp_path / name).write_bytes(b"x")
|
||||
return 0, "", ""
|
||||
|
||||
with (
|
||||
patch.object(rom_converto, "_run", fake_run),
|
||||
pytest.raises(RomConvertoOperationError, match="did not write exactly"),
|
||||
):
|
||||
await service.convert(self._op(), tmp_path / "x.iso", tmp_path / "game.chd")
|
||||
|
||||
async def test_nonzero_raises_operation_error_with_diagnostic(
|
||||
self, service: RomConvertoService, tmp_path: Path
|
||||
):
|
||||
async def fake_run(argv: list[str], timeout_seconds: float):
|
||||
return 1, "", "error: bad disc key"
|
||||
|
||||
src = tmp_path / "game.iso"
|
||||
out = tmp_path / "game.chd"
|
||||
with (
|
||||
patch.object(rom_converto, "_run", fake_run),
|
||||
pytest.raises(RomConvertoOperationError) as exc_info,
|
||||
):
|
||||
await service.convert(self._op(), src, out)
|
||||
|
||||
assert "bad disc key" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestFileFormat:
|
||||
@pytest.mark.parametrize(
|
||||
("file_name", "expected"),
|
||||
[
|
||||
pytest.param("Game.ISO", "iso", id="target-extension"),
|
||||
pytest.param("game.zcia", "z3ds", id="compressed-3ds"),
|
||||
pytest.param("game.3ds", "cci", id="alias"),
|
||||
pytest.param("game.nkit.iso", "nkit-iso", id="compound-extension"),
|
||||
pytest.param("game.pbp", "pbp", id="input-only-extension"),
|
||||
pytest.param("game.zip", "zip", id="unknown-extension"),
|
||||
pytest.param("README", "", id="no-extension"),
|
||||
],
|
||||
)
|
||||
def test_names_the_format(self, file_name: str, expected: str):
|
||||
assert file_format(file_name) == expected
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "expected"),
|
||||
[
|
||||
pytest.param("iso", "iso", id="plain"),
|
||||
pytest.param("3ds", "cci", id="alias"),
|
||||
pytest.param("zcia", "z3ds", id="compressed-alias"),
|
||||
pytest.param("nkit-iso", "nkit-iso", id="compound-not-iso"),
|
||||
],
|
||||
)
|
||||
def test_canonical_format_reads_aliases(self, name: str, expected: str):
|
||||
assert canonical_format(name) == expected
|
||||
|
||||
|
||||
class TestResolveOperation:
|
||||
@pytest.mark.parametrize(
|
||||
("platform", "target", "file_name", "argv", "input_ext", "output_name"),
|
||||
[
|
||||
(
|
||||
"ngc",
|
||||
"rvz",
|
||||
"Game.nkit.iso",
|
||||
("dol", "migrate"),
|
||||
".nkit.iso",
|
||||
"Game.rvz",
|
||||
),
|
||||
("ngc", "rvz", "Game.iso", ("dol", "compress"), ".iso", "Game.rvz"),
|
||||
("wii", "wbfs", "Game.rvz", ("rvl", "decompress"), ".rvz", "Game.wbfs"),
|
||||
("3ds", "z3ds", "Game.3ds", ("ctr", "compress"), ".3ds", "Game.zcci"),
|
||||
("3ds", "cci", "My Game.CIA", ("ctr", "convert"), ".cia", "My Game.cci"),
|
||||
("psp", "chd", "Game.ISO", ("chd", "compress"), ".iso", "Game.chd"),
|
||||
],
|
||||
)
|
||||
def test_resolves_operation_and_output_name(
|
||||
self,
|
||||
platform: str,
|
||||
target: str,
|
||||
file_name: str,
|
||||
argv: tuple[str, ...],
|
||||
input_ext: str,
|
||||
output_name: str,
|
||||
):
|
||||
resolved = resolve_operation(platform, target, file_name)
|
||||
|
||||
assert resolved is not None
|
||||
op, ext = resolved
|
||||
assert (op.argv, ext) == (argv, input_ext)
|
||||
assert op.output_name(Path(file_name), ext) == output_name
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("platform", "target", "file_name"),
|
||||
[
|
||||
pytest.param("psp", "iso", "Game.iso", id="already-in-target-format"),
|
||||
pytest.param("psx", "iso", "Game.chd", id="cd-chd-has-no-extract"),
|
||||
pytest.param("psx", "iso", "Game.cue", id="psx-cue-can-have-audio"),
|
||||
pytest.param("saturn", "iso", "Game.cue", id="saturn-cue-can-have-audio"),
|
||||
pytest.param("segacd", "iso", "Game.cue", id="segacd-cue-can-have-audio"),
|
||||
pytest.param("dc", "iso", "Game.cue", id="dc-cue-can-have-audio"),
|
||||
],
|
||||
)
|
||||
def test_returns_none_when_nothing_applies(
|
||||
self, platform: str, target: str, file_name: str
|
||||
):
|
||||
assert resolve_operation(platform, target, file_name) is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"platform",
|
||||
[
|
||||
pytest.param("psp", id="psp"),
|
||||
pytest.param("ps2", id="ps2"),
|
||||
],
|
||||
)
|
||||
def test_dvd_cue_still_converts_to_iso(self, platform: str):
|
||||
resolved = resolve_operation(platform, "iso", "Game.cue")
|
||||
|
||||
assert resolved is not None
|
||||
op, ext = resolved
|
||||
assert op.argv == ("cue", "to-iso")
|
||||
assert op.output_name(Path("Game.cue"), ext) == "Game.iso"
|
||||
|
||||
|
||||
class TestLibraryTargetsByPlatform:
|
||||
def test_psp_targets(self):
|
||||
assert LIBRARY_TARGETS_BY_PLATFORM["psp"] == {"chd", "cso", "iso", "zso"}
|
||||
|
||||
def test_platforms_without_format_conversions(self):
|
||||
for slug in ("wiiu", "psvita", "nds", "ps3"):
|
||||
assert slug not in LIBRARY_TARGETS_BY_PLATFORM
|
||||
|
||||
def test_a_lossy_conversion_is_no_library_target(self):
|
||||
assert "xbox" not in LIBRARY_TARGETS_BY_PLATFORM
|
||||
assert resolve_operation("xbox", "xiso", "game.iso") is not None
|
||||
assert resolve_operation("xbox", "xiso", "game.iso", lossless=True) is None
|
||||
|
||||
|
||||
class TestDownloadFormats:
|
||||
def test_maps_each_input_extension_to_its_targets(self):
|
||||
formats = DOWNLOAD_FORMATS
|
||||
|
||||
assert formats["psp"][".chd"] == ["cso", "iso", "zso"]
|
||||
assert formats["xbox"][".iso"] == ["xiso"]
|
||||
|
||||
def test_no_decrypt_or_encrypt_targets(self):
|
||||
targets = {
|
||||
target
|
||||
for exts in DOWNLOAD_FORMATS.values()
|
||||
for ext_targets in exts.values()
|
||||
for target in ext_targets
|
||||
}
|
||||
assert not targets & {"decrypted", "encrypted"}
|
||||
@@ -11,6 +11,7 @@ from config.config_manager import (
|
||||
DEFAULT_EXCLUDED_MULTI_FILE_DIRS,
|
||||
DEFAULT_EXCLUDED_PLATFORM_DIRS,
|
||||
ConfigManager,
|
||||
ConvertoConfig,
|
||||
parse_firmware_template,
|
||||
parse_platform_templates,
|
||||
parse_structure_template,
|
||||
@@ -188,6 +189,9 @@ def test_empty_config_loader():
|
||||
assert loader.config.SCAN_REGION_MODE == "prefer_rom_tags"
|
||||
assert loader.config.GAMELIST_MEDIA_THUMBNAIL == "box2d"
|
||||
assert loader.config.GAMELIST_MEDIA_IMAGE == "screenshot"
|
||||
assert not loader.config.CONVERTO.download_conversion_enabled
|
||||
assert loader.config.CONVERTO.cache_ttl_hours == 24
|
||||
assert loader.config.CONVERTO.platform_formats == {}
|
||||
assert loader.config.STRUCTURE_TEMPLATES == {}
|
||||
|
||||
|
||||
@@ -677,6 +681,72 @@ def test_reparenting_a_folder_replaces_the_existing_version(tmp_path):
|
||||
assert reloaded.config.PLATFORMS_VERSIONS == {"naomi": "dc"}
|
||||
|
||||
|
||||
def test_converto_config_from_yaml(tmp_path):
|
||||
config_file = tmp_path / "config.yml"
|
||||
config_file.write_text(
|
||||
"converto:\n"
|
||||
" download_conversion_enabled: true\n"
|
||||
" cache_ttl_hours: 48\n"
|
||||
" cache_max_size_gb: 0\n"
|
||||
" platform_formats:\n"
|
||||
" PSP: iso\n"
|
||||
" ngc: rvz\n"
|
||||
)
|
||||
loader = ConfigManager(str(config_file))
|
||||
|
||||
# Slugs and targets are normalized to lowercase.
|
||||
assert loader.config.CONVERTO == ConvertoConfig(
|
||||
download_conversion_enabled=True,
|
||||
cache_ttl_hours=48,
|
||||
cache_max_size_gb=0,
|
||||
platform_formats={"psp": "iso", "ngc": "rvz"},
|
||||
)
|
||||
|
||||
|
||||
def test_null_converto_platform_formats_means_empty(tmp_path):
|
||||
config_file = tmp_path / "config.yml"
|
||||
config_file.write_text("converto:\n platform_formats:\n")
|
||||
|
||||
assert ConfigManager(str(config_file)).config.CONVERTO.platform_formats == {}
|
||||
|
||||
|
||||
def test_update_converto_settings_round_trip(tmp_path):
|
||||
config_file = tmp_path / "config.yml"
|
||||
config_file.write_text("converto:\n cache_ttl_hours: 12\n")
|
||||
loader = ConfigManager(str(config_file))
|
||||
converto = ConvertoConfig(
|
||||
download_conversion_enabled=True,
|
||||
cache_ttl_hours=72,
|
||||
cache_max_size_gb=50,
|
||||
platform_formats={"psp": "iso", "ngc": "rvz"},
|
||||
)
|
||||
|
||||
loader.update_converto_settings(converto)
|
||||
|
||||
assert ConfigManager(str(config_file)).config.CONVERTO == converto
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"block",
|
||||
[
|
||||
" platform_formats:\n psvita: iso\n",
|
||||
" platform_formats:\n psp: rvz\n",
|
||||
" platform_formats:\n xbox: xiso\n",
|
||||
" cache_ttl_hours: 0\n",
|
||||
" cache_max_size_gb: -1\n",
|
||||
" cache_max_size_gb: true\n",
|
||||
],
|
||||
)
|
||||
def test_invalid_converto_setting_exits(tmp_path, block):
|
||||
config_file = tmp_path / "config.yml"
|
||||
config_file.write_text(f"converto:\n{block}")
|
||||
|
||||
with pytest.raises(SystemExit) as excinfo:
|
||||
ConfigManager(str(config_file))
|
||||
|
||||
assert excinfo.value.code == 3
|
||||
|
||||
|
||||
def _write_emulatorjs_config(tmp_path: Path, emulatorjs_block: str) -> ConfigManager:
|
||||
config_file = tmp_path / "config.yml"
|
||||
config_file.write_text(f"emulatorjs:\n{emulatorjs_block}")
|
||||
|
||||
@@ -0,0 +1,311 @@
|
||||
import asyncio
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
from urllib.parse import unquote
|
||||
|
||||
import pytest
|
||||
from fastapi import status
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from adapters.services.rom_converto import resolve_operation
|
||||
from config.config_manager import ConvertoConfig
|
||||
from models.rom import Rom
|
||||
from utils import conversion_cache
|
||||
from utils.conversion_cache import FAILED_FILE, RETRY_AFTER_SECONDS
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def conversion(tmp_path: Path, mocker):
|
||||
"""Enable download conversion with every rom-converto step mocked.
|
||||
|
||||
Any listed target resolves to a CHD under a temporary cache, and a
|
||||
conversion writes it there.
|
||||
"""
|
||||
converto = ConvertoConfig(download_conversion_enabled=True)
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.cm.get_config",
|
||||
return_value=SimpleNamespace(CONVERTO=converto),
|
||||
)
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.is_enabled",
|
||||
AsyncMock(return_value=True),
|
||||
)
|
||||
final = tmp_path / "cache/converts/1-abc/test_rom.chd"
|
||||
final.parent.mkdir(parents=True)
|
||||
resolved = resolve_operation("psp", "chd", "test_rom.iso")
|
||||
assert resolved is not None
|
||||
lookup = mocker.patch.object(
|
||||
conversion_cache, "_lookup", return_value=(resolved[0], final)
|
||||
)
|
||||
mocker.patch.object(conversion_cache, "ROMM_BASE_PATH", str(tmp_path))
|
||||
mocker.patch.object(
|
||||
conversion_cache, "ROM_CONVERTO_CACHE_PATH", str(final.parent.parent)
|
||||
)
|
||||
|
||||
async def convert(*args, **kwargs) -> Path:
|
||||
final.write_bytes(b"x" * 120)
|
||||
return final
|
||||
|
||||
return SimpleNamespace(
|
||||
converto=converto,
|
||||
final=final,
|
||||
lookup=lookup,
|
||||
convert=mocker.patch.object(
|
||||
conversion_cache, "get_or_convert", AsyncMock(side_effect=convert)
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def cached_copy(conversion) -> Path:
|
||||
final: Path = conversion.final
|
||||
final.write_bytes(b"x" * 120)
|
||||
served = time.time() - 3600
|
||||
os.utime(final, (served, served))
|
||||
return final
|
||||
|
||||
|
||||
def _redirect(response) -> str:
|
||||
return unquote(response.headers["X-Accel-Redirect"])
|
||||
|
||||
|
||||
def _download(client: TestClient, rom: Rom, *, token: str | None, **params):
|
||||
return client.get(
|
||||
f"/api/roms/{rom.id}/content/test_rom.zip",
|
||||
headers={"Authorization": f"Bearer {token}"} if token else {},
|
||||
params=params,
|
||||
follow_redirects=False,
|
||||
)
|
||||
|
||||
|
||||
def _as_kiosk(mocker) -> None:
|
||||
mocker.patch("handler.auth.hybrid_auth.KIOSK_MODE", True)
|
||||
mocker.patch("handler.auth.permissions.KIOSK_MODE", True)
|
||||
|
||||
|
||||
def test_download_is_untouched_without_a_format(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion
|
||||
):
|
||||
response = _download(client, rom, token=access_token)
|
||||
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
assert _redirect(response) == f"/library/{rom_file.full_path}"
|
||||
conversion.lookup.assert_not_called()
|
||||
conversion.convert.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("formats", ["zip", "chd,zip", " ZIP "])
|
||||
def test_a_listed_stored_format_serves_the_stored_file(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion, formats
|
||||
):
|
||||
response = _download(client, rom, token=access_token, format=formats)
|
||||
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
assert _redirect(response) == f"/library/{rom_file.full_path}"
|
||||
conversion.convert.assert_not_called()
|
||||
|
||||
|
||||
def test_a_listed_stored_format_is_served_with_conversion_disabled(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion
|
||||
):
|
||||
conversion.converto.download_conversion_enabled = False
|
||||
|
||||
response = _download(client, rom, token=access_token, format="zip")
|
||||
|
||||
assert _redirect(response) == f"/library/{rom_file.full_path}"
|
||||
|
||||
|
||||
def test_a_cached_copy_is_served_without_converting(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, cached_copy, conversion
|
||||
):
|
||||
response = _download(client, rom, token=access_token, format="chd")
|
||||
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
assert _redirect(response) == "/cache/converts/1-abc/test_rom.chd"
|
||||
assert "test_rom.chd" in response.headers["Content-Disposition"]
|
||||
conversion.convert.assert_not_called()
|
||||
|
||||
|
||||
def test_a_quick_conversion_is_served_by_the_same_request(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion
|
||||
):
|
||||
response = _download(client, rom, token=access_token, format="chd")
|
||||
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
assert _redirect(response) == "/cache/converts/1-abc/test_rom.chd"
|
||||
conversion.convert.assert_awaited_once()
|
||||
|
||||
|
||||
def test_a_slow_conversion_answers_202_and_keeps_running(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion, mocker
|
||||
):
|
||||
mocker.patch.object(conversion_cache, "CONVERSION_WAIT_SECONDS", 0.01)
|
||||
|
||||
final: Path = conversion.final
|
||||
|
||||
async def slow_convert(*args, **kwargs) -> Path:
|
||||
await asyncio.sleep(0.2)
|
||||
return final
|
||||
|
||||
conversion.convert.side_effect = slow_convert
|
||||
|
||||
response = _download(client, rom, token=access_token, format="chd")
|
||||
|
||||
assert response.status_code == status.HTTP_202_ACCEPTED
|
||||
assert response.headers["Retry-After"] == str(RETRY_AFTER_SECONDS)
|
||||
assert "X-Accel-Redirect" not in response.headers
|
||||
conversion.convert.assert_called_once()
|
||||
|
||||
|
||||
def test_a_running_conversion_answers_202_without_starting_another(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion, mocker
|
||||
):
|
||||
mocker.patch.object(conversion_cache, "_is_converting", return_value=True)
|
||||
|
||||
response = _download(client, rom, token=access_token, format="chd")
|
||||
|
||||
assert response.status_code == status.HTTP_202_ACCEPTED
|
||||
conversion.convert.assert_not_called()
|
||||
|
||||
|
||||
def test_head_reports_202_or_the_cached_copy_but_never_converts(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion
|
||||
):
|
||||
headers = {"Authorization": f"Bearer {access_token}"}
|
||||
url = f"/api/roms/{rom.id}/content/test_rom.zip"
|
||||
|
||||
uncached = client.head(url, headers=headers, params={"format": "chd"})
|
||||
conversion.final.write_bytes(b"x")
|
||||
cached = client.head(url, headers=headers, params={"format": "chd"})
|
||||
|
||||
assert uncached.status_code == status.HTTP_202_ACCEPTED
|
||||
assert _redirect(cached) == "/cache/converts/1-abc/test_rom.chd"
|
||||
conversion.convert.assert_not_called()
|
||||
|
||||
|
||||
def test_a_kiosk_visitor_never_starts_a_conversion(
|
||||
client: TestClient, rom: Rom, rom_file, conversion, mocker
|
||||
):
|
||||
_as_kiosk(mocker)
|
||||
|
||||
response = _download(client, rom, token=None, format="chd")
|
||||
|
||||
assert response.status_code == status.HTTP_406_NOT_ACCEPTABLE
|
||||
conversion.convert.assert_not_called()
|
||||
|
||||
|
||||
def test_a_kiosk_visitor_still_gets_a_cached_copy(
|
||||
client: TestClient, rom: Rom, rom_file, cached_copy, conversion, mocker
|
||||
):
|
||||
_as_kiosk(mocker)
|
||||
|
||||
response = _download(client, rom, token=None, format="chd")
|
||||
|
||||
assert _redirect(response) == "/cache/converts/1-abc/test_rom.chd"
|
||||
|
||||
|
||||
def test_no_producible_format_answers_406(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion
|
||||
):
|
||||
conversion.lookup.return_value = None
|
||||
|
||||
response = _download(client, rom, token=access_token, format="rvz,wbfs")
|
||||
|
||||
assert response.status_code == status.HTTP_406_NOT_ACCEPTABLE
|
||||
assert "rvz, wbfs" in response.json()["detail"]
|
||||
conversion.convert.assert_not_called()
|
||||
|
||||
|
||||
def test_disabled_conversion_answers_406_for_another_format(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion
|
||||
):
|
||||
conversion.converto.download_conversion_enabled = False
|
||||
|
||||
response = _download(client, rom, token=access_token, format="chd")
|
||||
|
||||
assert response.status_code == status.HTTP_406_NOT_ACCEPTABLE
|
||||
conversion.lookup.assert_not_called()
|
||||
|
||||
|
||||
def test_a_failed_conversion_is_not_retried(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion
|
||||
):
|
||||
(conversion.final.parent / FAILED_FILE).touch()
|
||||
|
||||
response = _download(client, rom, token=access_token, format="chd")
|
||||
|
||||
assert response.status_code == status.HTTP_406_NOT_ACCEPTABLE
|
||||
conversion.convert.assert_not_called()
|
||||
|
||||
|
||||
def test_a_conversion_that_fails_answers_406(
|
||||
client: TestClient, access_token: str, rom: Rom, rom_file, conversion
|
||||
):
|
||||
conversion.convert.side_effect = None
|
||||
conversion.convert.return_value = None
|
||||
|
||||
response = _download(client, rom, token=access_token, format="chd")
|
||||
|
||||
assert response.status_code == status.HTTP_406_NOT_ACCEPTABLE
|
||||
|
||||
|
||||
def test_a_format_needs_a_single_file(
|
||||
client: TestClient, access_token: str, multi_file_rom: Rom, conversion
|
||||
):
|
||||
response = client.head(
|
||||
f"/api/roms/{multi_file_rom.id}/content/test_multi_file_rom.zip",
|
||||
headers={"Authorization": f"Bearer {access_token}"},
|
||||
params={"format": "chd"},
|
||||
)
|
||||
|
||||
assert response.status_code == status.HTTP_406_NOT_ACCEPTABLE
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "visitor"),
|
||||
[
|
||||
pytest.param("get", "authenticated", id="authenticated-download"),
|
||||
pytest.param("get", "kiosk", id="kiosk-download"),
|
||||
pytest.param("get", "public", id="download-auth-disabled"),
|
||||
pytest.param("head", "authenticated", id="metadata"),
|
||||
],
|
||||
)
|
||||
def test_get_refreshes_a_cached_copy_but_head_does_not(
|
||||
client: TestClient,
|
||||
access_token: str,
|
||||
rom: Rom,
|
||||
rom_file,
|
||||
conversion,
|
||||
cached_copy: Path,
|
||||
mocker,
|
||||
method,
|
||||
visitor,
|
||||
):
|
||||
headers = {"Authorization": f"Bearer {access_token}"}
|
||||
if visitor == "kiosk":
|
||||
_as_kiosk(mocker)
|
||||
headers = {}
|
||||
elif visitor == "public":
|
||||
# Download scopes are fixed when the route decorators are evaluated.
|
||||
mocker.patch("decorators.auth.has_required_scope", return_value=True)
|
||||
headers = {}
|
||||
conversion.converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
|
||||
response = client.request(
|
||||
method,
|
||||
f"/api/roms/{rom.id}/content/test_rom.zip",
|
||||
headers=headers,
|
||||
params={"format": "chd"},
|
||||
follow_redirects=False,
|
||||
)
|
||||
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
assert _redirect(response) == "/cache/converts/1-abc/test_rom.chd"
|
||||
assert (cached_copy.stat().st_mtime > time.time() - 60) is (method == "get")
|
||||
assert conversion_cache.cleanup_stale_conversions() == (0 if method == "get" else 1)
|
||||
assert cached_copy.exists() is (method == "get")
|
||||
conversion.convert.assert_not_called()
|
||||
@@ -1,5 +1,7 @@
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
import pytest
|
||||
from fastapi import status
|
||||
|
||||
from config.config_manager import (
|
||||
@@ -7,6 +9,7 @@ from config.config_manager import (
|
||||
DEFAULT_EXCLUDED_FILES,
|
||||
DEFAULT_EXCLUDED_MULTI_FILE_DIRS,
|
||||
DEFAULT_EXCLUDED_PLATFORM_DIRS,
|
||||
ConvertoConfig,
|
||||
ExclusionType,
|
||||
)
|
||||
from config.config_manager import config_manager as cm
|
||||
@@ -261,3 +264,66 @@ def test_update_scan_settings_normalizes_codes(client, access_token: str):
|
||||
_, kwargs = update_scan_settings.call_args
|
||||
assert kwargs["region_priority"] == ["us", "eu"]
|
||||
assert kwargs["language_priority"] == ["en"]
|
||||
|
||||
|
||||
def _converto_payload(**overrides):
|
||||
payload = {
|
||||
"download_conversion_enabled": True,
|
||||
"cache_max_size_gb": 20,
|
||||
"platform_formats": {"psp": "cso"},
|
||||
}
|
||||
payload.update(overrides)
|
||||
return payload
|
||||
|
||||
|
||||
def test_update_converto_settings_normalizes_formats_and_keeps_the_ttl(
|
||||
client, access_token: str
|
||||
):
|
||||
current = SimpleNamespace(CONVERTO=ConvertoConfig(cache_ttl_hours=72))
|
||||
with (
|
||||
patch.object(cm, "get_config", return_value=current),
|
||||
patch.object(cm, "update_converto_settings") as update_converto_settings,
|
||||
):
|
||||
response = client.put(
|
||||
"/api/config/converto_settings",
|
||||
headers={"Authorization": f"Bearer {access_token}"},
|
||||
json=_converto_payload(platform_formats={" PSP ": "CSO"}),
|
||||
)
|
||||
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
update_converto_settings.assert_called_once_with(
|
||||
ConvertoConfig(
|
||||
download_conversion_enabled=True,
|
||||
cache_ttl_hours=72,
|
||||
cache_max_size_gb=20,
|
||||
platform_formats={"psp": "cso"},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_update_converto_settings_requires_auth(client):
|
||||
response = client.put("/api/config/converto_settings", json=_converto_payload())
|
||||
assert response.status_code == status.HTTP_401_UNAUTHORIZED
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"overrides",
|
||||
[
|
||||
{"platform_formats": {"psp": "rvz"}},
|
||||
{"platform_formats": {"psvita": "iso"}},
|
||||
{"platform_formats": {"xbox": "xiso"}},
|
||||
{"cache_max_size_gb": -1},
|
||||
],
|
||||
)
|
||||
def test_update_converto_settings_rejects_invalid_values(
|
||||
client, access_token: str, overrides
|
||||
):
|
||||
with patch.object(cm, "update_converto_settings") as update_converto_settings:
|
||||
response = client.put(
|
||||
"/api/config/converto_settings",
|
||||
headers={"Authorization": f"Bearer {access_token}"},
|
||||
json=_converto_payload(**overrides),
|
||||
)
|
||||
|
||||
assert response.status_code == status.HTTP_422_UNPROCESSABLE_CONTENT
|
||||
update_converto_settings.assert_not_called()
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
from dataclasses import replace
|
||||
|
||||
from tasks.registry import CLEANUP_CONVERSION_CACHE_SPEC
|
||||
from tasks.scheduled.cleanup_conversion_cache import CleanupConversionCacheTask
|
||||
|
||||
|
||||
class TestCleanupConversionCacheTask:
|
||||
def test_configuration(self):
|
||||
assert CLEANUP_CONVERSION_CACHE_SPEC.enabled is True
|
||||
assert CLEANUP_CONVERSION_CACHE_SPEC.cron_string == "0 4 * * *"
|
||||
|
||||
async def test_run_calls_cleanup(self, mocker):
|
||||
task = CleanupConversionCacheTask()
|
||||
mock_cleanup = mocker.patch(
|
||||
"tasks.scheduled.cleanup_conversion_cache.cleanup_stale_conversions",
|
||||
return_value=2,
|
||||
)
|
||||
await task.run()
|
||||
mock_cleanup.assert_called_once_with()
|
||||
|
||||
async def test_run_disabled_skips_the_cleanup(self, mocker):
|
||||
task = CleanupConversionCacheTask()
|
||||
task.spec = replace(task.spec, enabled=False)
|
||||
mock_cleanup = mocker.patch(
|
||||
"tasks.scheduled.cleanup_conversion_cache.cleanup_stale_conversions",
|
||||
)
|
||||
await task.run()
|
||||
mock_cleanup.assert_not_called()
|
||||
@@ -0,0 +1,400 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
from tests.factories import make_rom
|
||||
|
||||
from adapters.services.rom_converto import (
|
||||
RomConvertoOperationError,
|
||||
rom_converto_service,
|
||||
)
|
||||
from config.config_manager import ConvertoConfig
|
||||
from config.config_manager import config_manager as cm
|
||||
from handler.database import db_platform_handler, db_rom_handler
|
||||
from handler.filesystem import fs_rom_handler
|
||||
from models.platform import Platform
|
||||
from models.rom import Rom, RomFile, RomFileCategory
|
||||
from models.user import User
|
||||
from tasks.manual.convert_library import ConvertLibraryTask
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def library(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
||||
lib = tmp_path / "library"
|
||||
lib.mkdir()
|
||||
monkeypatch.setattr(fs_rom_handler, "base_path", lib.resolve())
|
||||
return lib
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def converto(mocker) -> ConvertoConfig:
|
||||
config = ConvertoConfig(platform_formats={"psp": "chd", "psx": "chd"})
|
||||
# The file refresh after a conversion reads the rest of the same config.
|
||||
real = cm.get_config()
|
||||
mocker.patch.object(real, "CONVERTO", config)
|
||||
mocker.patch.object(cm, "get_config", return_value=real)
|
||||
return config
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def converted(mocker) -> list[tuple[str, Path]]:
|
||||
"""Fake rom-converto: each conversion writes a small output and is recorded."""
|
||||
calls: list[tuple[str, Path]] = []
|
||||
|
||||
def convert(operation, src: Path, out: Path) -> None:
|
||||
calls.append((src.name, out))
|
||||
out.write_bytes(b"c")
|
||||
|
||||
mocker.patch.object(
|
||||
rom_converto_service, "is_enabled", AsyncMock(return_value=True)
|
||||
)
|
||||
mocker.patch.object(rom_converto_service, "convert", AsyncMock(side_effect=convert))
|
||||
return calls
|
||||
|
||||
|
||||
def _platform(slug: str) -> Platform:
|
||||
return db_platform_handler.add_platform(
|
||||
Platform(name=slug, slug=slug, fs_slug=slug)
|
||||
)
|
||||
|
||||
|
||||
def _rom(
|
||||
library: Path,
|
||||
admin_user: User,
|
||||
platform: Platform,
|
||||
files: dict[str, bytes],
|
||||
*,
|
||||
folder: bool = False,
|
||||
matched: bool = True,
|
||||
) -> Rom:
|
||||
"""A rom with `files` on disk: a single file, or a folder holding them all."""
|
||||
name = "game"
|
||||
rom = make_rom(
|
||||
platform,
|
||||
name,
|
||||
fs_extension="" if folder else Path(next(iter(files))).suffix.lstrip("."),
|
||||
igdb_id=1 if matched else None,
|
||||
)
|
||||
db_rom_handler.add_rom_user(rom_id=rom.id, user_id=admin_user.id)
|
||||
file_path = f"{rom.fs_path}/{rom.fs_name}" if folder else rom.fs_path
|
||||
for file_name, data in files.items():
|
||||
disk = library / file_path / file_name
|
||||
disk.parent.mkdir(parents=True, exist_ok=True)
|
||||
disk.write_bytes(data)
|
||||
db_rom_handler.add_rom_file(
|
||||
RomFile(
|
||||
rom_id=rom.id,
|
||||
file_name=file_name,
|
||||
file_path=file_path,
|
||||
file_size_bytes=len(data),
|
||||
last_modified=disk.stat().st_mtime,
|
||||
category=RomFileCategory.GAME,
|
||||
)
|
||||
)
|
||||
refreshed = db_rom_handler.get_rom(rom.id)
|
||||
assert refreshed is not None
|
||||
return refreshed
|
||||
|
||||
|
||||
def _reload(rom: Rom) -> Rom:
|
||||
refreshed = db_rom_handler.get_rom(rom.id)
|
||||
assert refreshed is not None
|
||||
return refreshed
|
||||
|
||||
|
||||
async def _run(platform: Platform | None = None) -> dict[str, int]:
|
||||
return await ConvertLibraryTask().run(platform.id if platform else None)
|
||||
|
||||
|
||||
async def test_a_single_file_rom_is_replaced_and_keeps_its_id(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
psp = _platform("psp")
|
||||
rom = _rom(library, admin_user, psp, {"game.iso": b"x" * 100})
|
||||
folder = library / rom.fs_path
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["converted"] == 1
|
||||
assert stats["bytes_saved"] == 99
|
||||
assert sorted(p.name for p in folder.iterdir()) == ["game.chd"]
|
||||
after = _reload(rom)
|
||||
assert after.fs_name == "game.chd"
|
||||
assert after.fs_extension == "chd"
|
||||
assert [f.file_name for f in after.files] == ["game.chd"]
|
||||
|
||||
|
||||
async def test_a_cue_folder_becomes_one_chd_and_its_playlist_follows(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
psx = _platform("psx")
|
||||
cue = 'FILE "game (Track 1).bin" BINARY\nFILE "game (Track 2).bin" BINARY\n'
|
||||
rom = _rom(
|
||||
library,
|
||||
admin_user,
|
||||
psx,
|
||||
{
|
||||
"game.cue": cue.encode(),
|
||||
"game (Track 1).bin": b"x" * 50,
|
||||
"game (Track 2).bin": b"x" * 50,
|
||||
"game.m3u": b"game.cue\n",
|
||||
},
|
||||
folder=True,
|
||||
)
|
||||
folder = library / rom.full_path
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["converted"] == 1
|
||||
assert sorted(p.name for p in folder.iterdir()) == ["game.chd", "game.m3u"]
|
||||
assert (folder / "game.m3u").read_text() == "game.chd\n"
|
||||
after = _reload(rom)
|
||||
assert after.fs_name == rom.fs_name
|
||||
assert sorted(f.file_name for f in after.files) == ["game.chd", "game.m3u"]
|
||||
|
||||
|
||||
async def test_a_bom_cue_and_playlist_still_follow_the_conversion(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
bom = "\ufeff".encode()
|
||||
rom = _rom(
|
||||
library,
|
||||
admin_user,
|
||||
_platform("psx"),
|
||||
{
|
||||
"game.cue": bom + b'FILE "game.bin" BINARY\n',
|
||||
"game.bin": b"x" * 50,
|
||||
"game.m3u": bom + b"./game.cue\n",
|
||||
},
|
||||
folder=True,
|
||||
)
|
||||
folder = library / rom.full_path
|
||||
|
||||
await _run()
|
||||
|
||||
assert sorted(p.name for p in folder.iterdir()) == ["game.chd", "game.m3u"]
|
||||
assert (folder / "game.m3u").read_text() == "game.chd\n"
|
||||
# Staged beside the rom folder, whose dot-prefixed subfolders count as its files.
|
||||
assert converted[0][1].parent.parent == library / rom.fs_path
|
||||
|
||||
|
||||
async def test_one_roms_failure_does_not_stop_the_run(
|
||||
library: Path, admin_user: User, converto, converted, mocker
|
||||
):
|
||||
_rom(library, admin_user, _platform("psp"), {"game.iso": b"x"})
|
||||
_rom(library, admin_user, _platform("psx"), {"game.cue": b""}, folder=True)
|
||||
mocker.patch(
|
||||
"tasks.manual.convert_library.refresh_rom_files",
|
||||
AsyncMock(side_effect=[RuntimeError("lock timeout"), None]),
|
||||
)
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["failed"] == 1
|
||||
assert sorted(src for src, _ in converted) == ["game.cue", "game.iso"]
|
||||
|
||||
|
||||
async def test_a_file_already_in_the_library_format_is_left_alone(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
_rom(library, admin_user, _platform("psp"), {"game.chd": b"x"})
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["already_converted"] == 1
|
||||
assert converted == []
|
||||
|
||||
|
||||
async def test_an_unmatched_rom_is_not_converted(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
rom = _rom(library, admin_user, _platform("psp"), {"game.iso": b"x"}, matched=False)
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["unmatched"] == 1
|
||||
assert converted == []
|
||||
assert (library / rom.full_path).exists()
|
||||
|
||||
|
||||
async def test_a_platform_without_a_library_format_is_skipped(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
converto.platform_formats = {}
|
||||
_rom(library, admin_user, _platform("psp"), {"game.iso": b"x"})
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["converted"] == 0
|
||||
assert converted == []
|
||||
|
||||
|
||||
async def test_only_the_requested_platform_is_converted(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
psp = _platform("psp")
|
||||
_rom(library, admin_user, psp, {"game.iso": b"x" * 10})
|
||||
_rom(library, admin_user, _platform("psx"), {"game.cue": b""}, folder=True)
|
||||
|
||||
await _run(psp)
|
||||
|
||||
assert [src for src, _ in converted] == ["game.iso"]
|
||||
|
||||
|
||||
async def test_a_failed_conversion_leaves_the_original_and_no_staged_output(
|
||||
library: Path, admin_user: User, converto, converted, mocker
|
||||
):
|
||||
rom = _rom(library, admin_user, _platform("psp"), {"game.iso": b"x"})
|
||||
mocker.patch.object(
|
||||
rom_converto_service,
|
||||
"convert",
|
||||
AsyncMock(side_effect=RomConvertoOperationError("boom")),
|
||||
)
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["failed"] == 1
|
||||
assert sorted(p.name for p in (library / rom.fs_path).iterdir()) == ["game.iso"]
|
||||
assert _reload(rom).fs_name == "game.iso"
|
||||
|
||||
|
||||
async def test_an_existing_output_is_never_overwritten(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
rom = _rom(library, admin_user, _platform("psp"), {"game.iso": b"x"})
|
||||
(library / rom.fs_path / "game.chd").write_bytes(b"theirs")
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["failed"] == 1
|
||||
assert converted == []
|
||||
assert (library / rom.fs_path / "game.chd").read_bytes() == b"theirs"
|
||||
assert (library / rom.full_path).exists()
|
||||
|
||||
|
||||
async def test_an_output_appearing_during_conversion_is_never_overwritten(
|
||||
library: Path, admin_user: User, converto, converted, mocker
|
||||
):
|
||||
rom = _rom(library, admin_user, _platform("psp"), {"game.iso": b"x"})
|
||||
theirs = library / rom.fs_path / "game.chd"
|
||||
|
||||
def convert(operation, src: Path, out: Path) -> None:
|
||||
theirs.write_bytes(b"theirs")
|
||||
out.write_bytes(b"c")
|
||||
|
||||
mocker.patch.object(rom_converto_service, "convert", AsyncMock(side_effect=convert))
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["failed"] == 1
|
||||
assert theirs.read_bytes() == b"theirs"
|
||||
assert (library / rom.full_path).exists()
|
||||
|
||||
|
||||
async def test_a_failed_rom_update_keeps_the_original(
|
||||
library: Path, admin_user: User, converto, converted, mocker
|
||||
):
|
||||
rom = _rom(library, admin_user, _platform("psp"), {"game.iso": b"x"})
|
||||
mocker.patch.object(
|
||||
db_rom_handler, "update_rom", side_effect=SQLAlchemyError("conflict")
|
||||
)
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["failed"] == 1
|
||||
assert sorted(p.name for p in (library / rom.fs_path).iterdir()) == ["game.iso"]
|
||||
|
||||
|
||||
async def test_a_cue_track_held_by_another_rom_is_not_deleted(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
psx = _platform("psx")
|
||||
rom = _rom(library, admin_user, psx, {"game.cue": b'FILE "game.bin" BINARY\n'})
|
||||
(library / rom.fs_path / "game.bin").write_bytes(b"x")
|
||||
make_rom(psx, "track", fs_stem="game", fs_extension="bin")
|
||||
|
||||
stats = await _run(psx)
|
||||
|
||||
assert stats["failed"] == 1
|
||||
assert converted == []
|
||||
assert (library / rom.fs_path / "game.bin").exists()
|
||||
|
||||
|
||||
async def test_a_track_another_cue_shares_is_refused(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
track = b'FILE "game.bin" BINARY\n'
|
||||
rom = _rom(
|
||||
library,
|
||||
admin_user,
|
||||
_platform("psx"),
|
||||
{"disc 1.cue": track, "disc 2.cue": track, "game.bin": b"x"},
|
||||
folder=True,
|
||||
)
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["failed"] == 2
|
||||
assert converted == []
|
||||
assert (library / rom.full_path / "game.bin").exists()
|
||||
|
||||
|
||||
async def test_a_stage_left_by_a_killed_conversion_is_removed(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
rom = _rom(library, admin_user, _platform("psp"), {"game.iso": b"x"})
|
||||
stale = library / rom.fs_path / ".romm_tmp_convert_old"
|
||||
stale.mkdir()
|
||||
os.utime(stale, (0, 0))
|
||||
|
||||
await _run()
|
||||
|
||||
assert not stale.exists()
|
||||
|
||||
|
||||
async def test_a_rom_row_already_holding_the_output_name_is_left_alone(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
psp = _platform("psp")
|
||||
rom = _rom(library, admin_user, psp, {"game.iso": b"x"})
|
||||
make_rom(psp, "other", fs_stem="game", fs_extension="chd")
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["failed"] == 1
|
||||
assert converted == []
|
||||
assert (library / rom.full_path).exists()
|
||||
|
||||
|
||||
async def test_a_cue_pointing_outside_its_folder_is_refused(
|
||||
library: Path, admin_user: User, converto, converted
|
||||
):
|
||||
rom = _rom(
|
||||
library,
|
||||
admin_user,
|
||||
_platform("psx"),
|
||||
{"game.cue": b'FILE "../other.bin" BINARY\n'},
|
||||
folder=True,
|
||||
)
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["failed"] == 1
|
||||
assert converted == []
|
||||
assert (library / rom.full_path / "game.cue").exists()
|
||||
|
||||
|
||||
async def test_nothing_runs_without_rom_converto(
|
||||
library: Path, admin_user: User, converto, converted, mocker
|
||||
):
|
||||
mocker.patch.object(
|
||||
rom_converto_service, "is_enabled", AsyncMock(return_value=False)
|
||||
)
|
||||
_rom(library, admin_user, _platform("psp"), {"game.iso": b"x"})
|
||||
|
||||
stats = await _run()
|
||||
|
||||
assert stats["converted"] == 0
|
||||
assert converted == []
|
||||
@@ -0,0 +1,986 @@
|
||||
import asyncio
|
||||
import fcntl
|
||||
import os
|
||||
import shutil
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from contextlib import ExitStack, contextmanager, nullcontext
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import Any, cast
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
|
||||
from adapters.services.rom_converto import (
|
||||
RomConvertoBinaryNotFoundError,
|
||||
RomConvertoOperationError,
|
||||
RomConvertoTimeoutError,
|
||||
resolve_operation,
|
||||
)
|
||||
from config import ROMM_BASE_PATH
|
||||
from config.config_manager import ConvertoConfig
|
||||
from models.rom import Rom, RomFile
|
||||
from utils import conversion_cache
|
||||
from utils.conversion_cache import (
|
||||
FAILED_FILE,
|
||||
PARTIAL_DIR,
|
||||
SERVE_GRACE_SECONDS,
|
||||
FormatOutcome,
|
||||
FormatResolution,
|
||||
converted_file_path,
|
||||
get_cached_converted,
|
||||
get_or_convert,
|
||||
get_redirect_path,
|
||||
parse_formats,
|
||||
resolve_format_download,
|
||||
)
|
||||
|
||||
|
||||
def _rom_file(**overrides: Any) -> RomFile:
|
||||
defaults: dict[str, Any] = {
|
||||
"rom_id": 1,
|
||||
"file_name": "game.iso",
|
||||
"file_path": "psp/roms",
|
||||
"file_size_bytes": 1000,
|
||||
"last_modified": 1700000000.0,
|
||||
}
|
||||
defaults.update(overrides)
|
||||
return RomFile(**defaults)
|
||||
|
||||
|
||||
def _final_path(rom_file: RomFile) -> Path:
|
||||
resolved = resolve_operation("psp", "chd", rom_file.file_name)
|
||||
assert resolved is not None
|
||||
return converted_file_path(1, rom_file, *resolved)
|
||||
|
||||
|
||||
def _partial_path(final: Path) -> Path:
|
||||
return final.parent / PARTIAL_DIR / final.name
|
||||
|
||||
|
||||
def _seed_cached(rom_file: RomFile, *, age_seconds: float = 0) -> Path:
|
||||
"""Write a converted copy for `rom_file`, last served `age_seconds` ago."""
|
||||
final = _final_path(rom_file)
|
||||
final.parent.mkdir(parents=True, exist_ok=True)
|
||||
final.write_bytes(b"cached")
|
||||
served = time.time() - age_seconds
|
||||
os.utime(final, (served, served))
|
||||
return final
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _held_lock(key_dir: Path, *, shared: bool = False) -> Iterator[None]:
|
||||
"""Hold a key dir lock while a request or worker uses it."""
|
||||
fd = os.open(key_dir, os.O_RDONLY)
|
||||
fcntl.flock(fd, fcntl.LOCK_SH if shared else fcntl.LOCK_EX)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def converto(mocker) -> ConvertoConfig:
|
||||
"""The converto.* settings every cache function reads, editable per test."""
|
||||
config = ConvertoConfig()
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.cm.get_config",
|
||||
return_value=SimpleNamespace(CONVERTO=config),
|
||||
)
|
||||
return config
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def cache_root(tmp_path, mocker):
|
||||
mocker.patch.object(conversion_cache, "ROM_CONVERTO_CACHE_PATH", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def convert_outputs(mocker) -> list[Path]:
|
||||
"""Fake the adapter: write each requested output and record its path."""
|
||||
outputs: list[Path] = []
|
||||
|
||||
async def convert(operation, src, out) -> None:
|
||||
outputs.append(out)
|
||||
out.write_bytes(b"converted")
|
||||
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.convert", side_effect=convert
|
||||
)
|
||||
return outputs
|
||||
|
||||
|
||||
class TestGetOrConvert:
|
||||
async def test_unresolvable_returns_none_without_touching_disk(
|
||||
self, cache_root, convert_outputs
|
||||
):
|
||||
f = _rom_file(file_name="game.zip")
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "chd") is None
|
||||
assert convert_outputs == []
|
||||
assert not any(cache_root.iterdir())
|
||||
|
||||
async def test_cache_hit_returns_path_and_refreshes_mtime(
|
||||
self, cache_root, convert_outputs
|
||||
):
|
||||
f = _rom_file()
|
||||
final = _seed_cached(f, age_seconds=3600)
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "chd") == final
|
||||
assert final.stat().st_mtime > time.time() - 60
|
||||
assert convert_outputs == []
|
||||
|
||||
async def test_oversized_conversion_returns_none_without_converting(
|
||||
self, cache_root, converto, convert_outputs, mocker
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
|
||||
assert (
|
||||
await get_or_convert(1, _rom_file(file_size_bytes=101), "psp", "chd")
|
||||
is None
|
||||
)
|
||||
assert convert_outputs == []
|
||||
|
||||
async def test_admission_reserves_room_for_a_decompression(
|
||||
self, cache_root, converto, convert_outputs, mocker
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
f = _rom_file(file_name="game.chd", file_size_bytes=30)
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "iso") is None
|
||||
assert convert_outputs == []
|
||||
|
||||
async def test_admission_evicts_an_older_copy_to_make_room(
|
||||
self, cache_root, converto, convert_outputs, mocker
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
older = _seed_cached(_rom_file(file_name="older.iso"))
|
||||
older.write_bytes(b"x" * 40)
|
||||
served = time.time() - 300
|
||||
os.utime(older, (served, served))
|
||||
f = _rom_file(file_size_bytes=70)
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "chd") == _final_path(f)
|
||||
assert not older.parent.exists()
|
||||
assert _final_path(f).read_bytes() == b"converted"
|
||||
|
||||
async def test_admission_does_not_evict_a_recently_served_copy(
|
||||
self, cache_root, converto, convert_outputs, mocker
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
recent = _seed_cached(_rom_file(file_name="recent.iso"))
|
||||
recent.write_bytes(b"x" * 80)
|
||||
f = _rom_file(file_size_bytes=50)
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "chd") is None
|
||||
assert recent.read_bytes() == b"x" * 80
|
||||
assert convert_outputs == []
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("size_bytes", "blocked"),
|
||||
[
|
||||
pytest.param(101, "none", id="larger-than-cap"),
|
||||
pytest.param(70, "grace", id="insufficient-outside-grace"),
|
||||
pytest.param(70, "exclusive", id="conversion-locked"),
|
||||
pytest.param(70, "shared", id="download-locked"),
|
||||
],
|
||||
)
|
||||
async def test_impossible_admission_preserves_other_copies(
|
||||
self, cache_root, converto, convert_outputs, mocker, size_bytes, blocked
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
older = _seed_cached(_rom_file(file_name="older.iso"))
|
||||
protected = _seed_cached(_rom_file(file_name="protected.iso"))
|
||||
older.write_bytes(b"x" * 40)
|
||||
protected.write_bytes(b"x" * 50)
|
||||
now = time.time()
|
||||
os.utime(older, (now - 300, now - 300))
|
||||
protected_age = 0 if blocked == "grace" else SERVE_GRACE_SECONDS * 2
|
||||
os.utime(protected, (now - protected_age, now - protected_age))
|
||||
holder = (
|
||||
_held_lock(protected.parent, shared=blocked == "shared")
|
||||
if blocked in ("exclusive", "shared")
|
||||
else nullcontext()
|
||||
)
|
||||
|
||||
with holder:
|
||||
assert (
|
||||
await get_or_convert(
|
||||
1, _rom_file(file_size_bytes=size_bytes), "psp", "chd"
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
assert older.read_bytes() == b"x" * 40
|
||||
assert protected.read_bytes() == b"x" * 50
|
||||
assert convert_outputs == []
|
||||
|
||||
async def test_conversion_evicts_the_least_recently_served_other_copy(
|
||||
self, cache_root, converto, mocker
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
oldest = _seed_cached(_rom_file(file_name="old.iso"), age_seconds=300)
|
||||
newer = _seed_cached(_rom_file(file_name="newer.iso"), age_seconds=200)
|
||||
oldest.write_bytes(b"x" * 40)
|
||||
newer.write_bytes(b"x" * 40)
|
||||
now = time.time()
|
||||
os.utime(oldest, (now - 300, now - 300))
|
||||
os.utime(newer, (now - 200, now - 200))
|
||||
f = _rom_file(file_size_bytes=10)
|
||||
|
||||
async def convert(op, src, out) -> None:
|
||||
out.write_bytes(b"converted" * 6)
|
||||
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.convert", side_effect=convert
|
||||
)
|
||||
|
||||
final = await get_or_convert(1, f, "psp", "chd")
|
||||
|
||||
assert final == _final_path(f)
|
||||
assert final is not None
|
||||
assert final.read_bytes() == b"converted" * 6
|
||||
assert not oldest.parent.exists()
|
||||
assert newer.exists()
|
||||
assert conversion_cache.cache_size_bytes() <= 100
|
||||
|
||||
async def test_post_conversion_cleanup_failure_still_serves_the_copy(
|
||||
self, cache_root, convert_outputs, mocker
|
||||
):
|
||||
f = _rom_file()
|
||||
final = _final_path(f)
|
||||
cleanup = conversion_cache.cleanup_stale_conversions
|
||||
|
||||
def fail_after_publication(*args, **kwargs):
|
||||
if final.exists():
|
||||
raise RuntimeError("cleanup failed")
|
||||
return cleanup(*args, **kwargs)
|
||||
|
||||
mocker.patch.object(
|
||||
conversion_cache,
|
||||
"cleanup_stale_conversions",
|
||||
side_effect=fail_after_publication,
|
||||
)
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "chd") == final
|
||||
assert final.read_bytes() == b"converted"
|
||||
|
||||
async def test_locked_key_dir_returns_none(self, cache_root, convert_outputs):
|
||||
f = _rom_file()
|
||||
key_dir = _final_path(f).parent
|
||||
key_dir.mkdir(parents=True)
|
||||
|
||||
with _held_lock(key_dir):
|
||||
assert await get_or_convert(1, f, "psp", "chd") is None
|
||||
assert convert_outputs == []
|
||||
|
||||
async def test_concurrent_request_returns_none_while_one_converts(
|
||||
self, cache_root, mocker
|
||||
):
|
||||
f = _rom_file()
|
||||
started = asyncio.Event()
|
||||
release = asyncio.Event()
|
||||
|
||||
async def convert(op, src, out) -> None:
|
||||
started.set()
|
||||
await release.wait()
|
||||
out.write_bytes(b"converted")
|
||||
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.convert", side_effect=convert
|
||||
)
|
||||
|
||||
first = asyncio.create_task(get_or_convert(1, f, "psp", "chd"))
|
||||
await started.wait()
|
||||
assert await get_or_convert(1, f, "psp", "chd") is None
|
||||
release.set()
|
||||
assert await first == _final_path(f)
|
||||
|
||||
async def test_queued_conversion_checks_budget_after_a_slot_opens(
|
||||
self, cache_root, converto, mocker
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
mocker.patch.object(
|
||||
conversion_cache, "_convert_semaphore", asyncio.Semaphore(1)
|
||||
)
|
||||
first_file = _rom_file(file_size_bytes=10)
|
||||
second_file = _rom_file(file_name="second.iso", file_size_bytes=10)
|
||||
second_dir = _final_path(second_file).parent
|
||||
started = asyncio.Event()
|
||||
release = asyncio.Event()
|
||||
|
||||
async def convert(op, src, out) -> None:
|
||||
started.set()
|
||||
await release.wait()
|
||||
out.write_bytes(b"x" * 100)
|
||||
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.convert", side_effect=convert
|
||||
)
|
||||
|
||||
first = asyncio.create_task(get_or_convert(1, first_file, "psp", "chd"))
|
||||
await started.wait()
|
||||
second = asyncio.create_task(get_or_convert(1, second_file, "psp", "chd"))
|
||||
try:
|
||||
while not second.done():
|
||||
if second_dir.exists():
|
||||
with conversion_cache._try_lock(second_dir) as locked:
|
||||
if not locked:
|
||||
break
|
||||
await asyncio.sleep(0)
|
||||
queued = not second.done()
|
||||
finally:
|
||||
release.set()
|
||||
results = await asyncio.gather(first, second)
|
||||
|
||||
assert queued
|
||||
assert list(results) == [_final_path(first_file), None]
|
||||
assert _final_path(first_file).read_bytes() == b"x" * 100
|
||||
assert not _final_path(second_file).exists()
|
||||
|
||||
async def test_writes_in_partial_dir_then_renames_into_place(
|
||||
self, cache_root, convert_outputs
|
||||
):
|
||||
f = _rom_file()
|
||||
final = _final_path(f)
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "chd") == final
|
||||
assert convert_outputs == [_partial_path(final)]
|
||||
assert final.read_bytes() == b"converted"
|
||||
assert [p.name for p in final.parent.iterdir()] == [final.name]
|
||||
|
||||
async def test_leading_dot_copy_survives_cleanup(self, cache_root, convert_outputs):
|
||||
f = _rom_file(file_name=".hack - Infection (USA).iso")
|
||||
final = _final_path(f)
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "chd") == final
|
||||
assert final.name == ".hack - Infection (USA).chd"
|
||||
assert conversion_cache.cleanup_stale_conversions() == 0
|
||||
assert final.read_bytes() == b"converted"
|
||||
assert get_cached_converted(1, f, "psp", "chd") == final
|
||||
|
||||
async def test_removes_all_a_crashed_runs_partial_outputs(
|
||||
self, cache_root, convert_outputs
|
||||
):
|
||||
f = _rom_file()
|
||||
final = _final_path(f)
|
||||
final.parent.mkdir(parents=True)
|
||||
partial = _partial_path(final)
|
||||
partial.parent.mkdir()
|
||||
for path in (
|
||||
partial,
|
||||
partial.with_name(f".{partial.name}.ABCDEF.tmp"),
|
||||
partial.with_suffix(".bin"),
|
||||
):
|
||||
path.write_bytes(b"leftover")
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "chd") == final
|
||||
assert final.read_bytes() == b"converted"
|
||||
assert list(final.parent.iterdir()) == [final]
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("error", "writes_partial"),
|
||||
[
|
||||
pytest.param(RomConvertoOperationError, False, id="nonzero-exit-no-output"),
|
||||
pytest.param(RomConvertoOperationError, True, id="nonzero-exit-partials"),
|
||||
pytest.param(RomConvertoTimeoutError, True, id="timeout-partials"),
|
||||
],
|
||||
)
|
||||
async def test_failed_conversion_retries_only_after_marker_expires(
|
||||
self, cache_root, converto, mocker, error, writes_partial
|
||||
):
|
||||
f = _rom_file()
|
||||
final = _final_path(f)
|
||||
marker = final.parent / FAILED_FILE
|
||||
|
||||
async def convert(op, src, out) -> None:
|
||||
if writes_partial:
|
||||
out.write_bytes(b"partial")
|
||||
out.with_name(f".{out.name}.ABCDEF.tmp").write_bytes(b"scratch")
|
||||
out.with_suffix(".bin").write_bytes(b"data")
|
||||
raise error("boom")
|
||||
|
||||
convert_mock = mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.convert", side_effect=convert
|
||||
)
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "chd") is None
|
||||
assert list(final.parent.iterdir()) == [marker]
|
||||
assert await get_or_convert(1, f, "psp", "chd") is None
|
||||
convert_mock.assert_awaited_once()
|
||||
expired = time.time() - converto.cache_ttl_hours * 3600 - 1
|
||||
os.utime(marker, (expired, expired))
|
||||
assert conversion_cache.cleanup_stale_conversions() == 1
|
||||
|
||||
async def succeed(op, src, out) -> None:
|
||||
out.write_bytes(b"converted")
|
||||
|
||||
convert_mock.side_effect = succeed
|
||||
assert await get_or_convert(1, f, "psp", "chd") == final
|
||||
assert final.read_bytes() == b"converted"
|
||||
assert not marker.exists()
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"stage",
|
||||
[
|
||||
pytest.param("mkdir", id="partial-dir-error"),
|
||||
pytest.param("replace", id="publication-error"),
|
||||
pytest.param("convert-os-error", id="subprocess-os-error"),
|
||||
pytest.param("missing-binary", id="binary-not-found"),
|
||||
],
|
||||
)
|
||||
async def test_transient_failure_retries_the_next_download(
|
||||
self, cache_root, mocker, stage
|
||||
):
|
||||
f = _rom_file()
|
||||
final = _final_path(f)
|
||||
failed = False
|
||||
|
||||
async def convert(op, src, out) -> None:
|
||||
nonlocal failed
|
||||
if not failed and stage in ("convert-os-error", "missing-binary"):
|
||||
failed = True
|
||||
error = (
|
||||
RomConvertoBinaryNotFoundError
|
||||
if stage == "missing-binary"
|
||||
else OSError
|
||||
)
|
||||
raise error("temporary failure")
|
||||
out.write_bytes(b"converted")
|
||||
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.convert", side_effect=convert
|
||||
)
|
||||
if stage in ("mkdir", "replace"):
|
||||
write = Path.mkdir if stage == "mkdir" else os.replace
|
||||
|
||||
def fail_write_once(*args, **kwargs):
|
||||
nonlocal failed
|
||||
if not failed and (stage == "replace" or args[0].name == PARTIAL_DIR):
|
||||
failed = True
|
||||
raise OSError("temporary filesystem failure")
|
||||
return write(*args, **kwargs)
|
||||
|
||||
mocker.patch(
|
||||
"pathlib.Path.mkdir" if stage == "mkdir" else "os.replace",
|
||||
side_effect=fail_write_once,
|
||||
autospec=True,
|
||||
)
|
||||
|
||||
assert await get_or_convert(1, f, "psp", "chd") is None
|
||||
assert not (final.parent / FAILED_FILE).exists()
|
||||
assert not (final.parent / PARTIAL_DIR).exists()
|
||||
assert await get_or_convert(1, f, "psp", "chd") == final
|
||||
assert final.read_bytes() == b"converted"
|
||||
|
||||
async def test_cancellation_removes_every_partial_output(self, cache_root, mocker):
|
||||
f = _rom_file()
|
||||
final = _final_path(f)
|
||||
started = asyncio.Event()
|
||||
|
||||
async def convert(op, src, out) -> None:
|
||||
out.write_bytes(b"partial")
|
||||
out.with_name(f".{out.name}.ABCDEF.tmp").write_bytes(b"scratch")
|
||||
out.with_suffix(".bin").write_bytes(b"data")
|
||||
started.set()
|
||||
await asyncio.Event().wait()
|
||||
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.convert", side_effect=convert
|
||||
)
|
||||
|
||||
conversion = asyncio.create_task(get_or_convert(1, f, "psp", "chd"))
|
||||
await started.wait()
|
||||
conversion.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await conversion
|
||||
|
||||
assert not final.exists()
|
||||
assert list(final.parent.iterdir()) == []
|
||||
|
||||
|
||||
class TestGetCachedConverted:
|
||||
@pytest.mark.parametrize("file_name", ["game.zip", "game.iso"])
|
||||
def test_returns_none_when_unresolvable_or_not_cached(self, cache_root, file_name):
|
||||
f = _rom_file(file_name=file_name)
|
||||
|
||||
assert get_cached_converted(1, f, "psp", "chd") is None
|
||||
|
||||
@pytest.mark.parametrize("touch", [False, True])
|
||||
def test_returns_the_copy_refreshing_mtime_only_when_asked(self, cache_root, touch):
|
||||
f = _rom_file()
|
||||
final = _seed_cached(f, age_seconds=3600)
|
||||
|
||||
assert get_cached_converted(1, f, "psp", "chd", touch=touch) == final
|
||||
assert (final.stat().st_mtime > time.time() - 60) is touch
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"touch",
|
||||
[
|
||||
pytest.param(False, id="lookup"),
|
||||
pytest.param(True, id="download"),
|
||||
],
|
||||
)
|
||||
def test_returns_none_while_key_dir_is_locked(self, cache_root, touch):
|
||||
f = _rom_file()
|
||||
final = _seed_cached(f)
|
||||
|
||||
with _held_lock(final.parent):
|
||||
assert get_cached_converted(1, f, "psp", "chd", touch=touch) is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"error",
|
||||
[
|
||||
pytest.param(FileNotFoundError, id="touch-not-found"),
|
||||
pytest.param(PermissionError, id="permission-denied"),
|
||||
],
|
||||
)
|
||||
def test_touch_failure_still_serves_the_copy(self, cache_root, mocker, error):
|
||||
f = _rom_file()
|
||||
final = _seed_cached(f)
|
||||
mocker.patch("os.utime", side_effect=error)
|
||||
|
||||
assert get_cached_converted(1, f, "psp", "chd", touch=True) == final
|
||||
assert final.read_bytes() == b"cached"
|
||||
|
||||
def test_lock_open_failure_returns_none(self, cache_root, mocker):
|
||||
f = _rom_file()
|
||||
_seed_cached(f)
|
||||
mocker.patch("os.open", side_effect=PermissionError)
|
||||
|
||||
assert get_cached_converted(1, f, "psp", "chd") is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"recreate",
|
||||
[
|
||||
pytest.param(False, id="removed"),
|
||||
pytest.param(True, id="recreated"),
|
||||
],
|
||||
)
|
||||
def test_does_not_serve_a_replaced_key_dir(self, cache_root, mocker, recreate):
|
||||
f = _rom_file()
|
||||
final = _seed_cached(f)
|
||||
flock = fcntl.flock
|
||||
|
||||
def replace_dir(fd, mode):
|
||||
flock(fd, mode)
|
||||
shutil.rmtree(final.parent)
|
||||
if recreate:
|
||||
final.parent.mkdir()
|
||||
final.write_bytes(b"replacement")
|
||||
|
||||
mocker.patch("fcntl.flock", side_effect=replace_dir)
|
||||
|
||||
assert get_cached_converted(1, f, "psp", "chd") is None
|
||||
|
||||
|
||||
class TestHasRoomFor:
|
||||
def test_unbounded_when_cap_is_zero(self, cache_root, converto):
|
||||
converto.cache_max_size_gb = 0
|
||||
|
||||
assert conversion_cache.has_room_for(10 * conversion_cache.BYTES_PER_GB)
|
||||
|
||||
def test_counts_partial_output_against_the_cap(self, cache_root, converto, mocker):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
key_dir = cache_root / "1-a"
|
||||
key_dir.mkdir()
|
||||
(key_dir / "game.chd").write_bytes(b"x" * 40)
|
||||
partial_dir = key_dir / PARTIAL_DIR
|
||||
partial_dir.mkdir()
|
||||
(partial_dir / "other.chd").write_bytes(b"x" * 20)
|
||||
|
||||
assert conversion_cache.has_room_for(40)
|
||||
assert not conversion_cache.has_room_for(41)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"vanishes",
|
||||
[
|
||||
pytest.param("file", id="vanishing-file"),
|
||||
pytest.param("dir", id="vanishing-dir"),
|
||||
],
|
||||
)
|
||||
def test_size_walk_tolerates_vanishing_entries(self, cache_root, mocker, vanishes):
|
||||
vanished = _seed_cached(_rom_file(file_name="vanished.iso"))
|
||||
survivor = _seed_cached(_rom_file(file_name="survivor.iso"))
|
||||
stat = os.stat
|
||||
scandir = os.scandir
|
||||
|
||||
if vanishes == "file":
|
||||
|
||||
def stat_after_removal(path, *args, **kwargs):
|
||||
if path == str(vanished):
|
||||
vanished.unlink(missing_ok=True)
|
||||
return stat(path, *args, **kwargs)
|
||||
|
||||
mocker.patch("os.stat", side_effect=stat_after_removal)
|
||||
else:
|
||||
|
||||
def scan_after_removal(path):
|
||||
if path == str(vanished.parent):
|
||||
shutil.rmtree(vanished.parent)
|
||||
return scandir(path)
|
||||
|
||||
mocker.patch("os.scandir", side_effect=scan_after_removal)
|
||||
|
||||
assert conversion_cache.cache_size_bytes() == len(b"cached")
|
||||
assert survivor.read_bytes() == b"cached"
|
||||
|
||||
|
||||
class TestGetRedirectPath:
|
||||
def test_relative_to_romm_base_path(self):
|
||||
converted_path = Path(ROMM_BASE_PATH) / "cache/converts/1-abc/Game.chd"
|
||||
assert get_redirect_path(converted_path) == Path(
|
||||
"/cache/converts/1-abc/Game.chd"
|
||||
)
|
||||
|
||||
|
||||
class TestCleanupStaleConversions:
|
||||
def test_missing_root_is_noop(self, tmp_path, mocker):
|
||||
mocker.patch.object(
|
||||
conversion_cache, "ROM_CONVERTO_CACHE_PATH", str(tmp_path / "nope")
|
||||
)
|
||||
assert conversion_cache.cleanup_stale_conversions() == 0
|
||||
|
||||
def test_deletes_expired_and_keeps_fresh(self, cache_root):
|
||||
expired = _seed_cached(_rom_file(file_name="old.iso"), age_seconds=25 * 3600)
|
||||
fresh = _seed_cached(_rom_file(file_name="new.iso"))
|
||||
|
||||
assert conversion_cache.cleanup_stale_conversions() == 1
|
||||
assert not expired.parent.exists()
|
||||
assert fresh.exists()
|
||||
|
||||
def test_removes_partial_outputs_from_a_kept_dir(self, cache_root):
|
||||
final = _seed_cached(_rom_file())
|
||||
partial = _partial_path(final)
|
||||
partial.parent.mkdir()
|
||||
for path in (
|
||||
partial,
|
||||
partial.with_name(f".{partial.name}.ABCDEF.tmp"),
|
||||
partial.with_suffix(".bin"),
|
||||
):
|
||||
path.write_bytes(b"leftover")
|
||||
|
||||
assert conversion_cache.cleanup_stale_conversions() == 0
|
||||
assert final.read_bytes() == b"cached"
|
||||
assert list(final.parent.iterdir()) == [final]
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"leftover",
|
||||
[
|
||||
pytest.param(False, id="empty"),
|
||||
pytest.param(True, id="partial"),
|
||||
],
|
||||
)
|
||||
def test_deletes_dirs_without_a_served_copy(self, cache_root, leftover):
|
||||
key_dir = cache_root / "1-e"
|
||||
key_dir.mkdir()
|
||||
if leftover:
|
||||
partial_dir = key_dir / PARTIAL_DIR
|
||||
partial_dir.mkdir()
|
||||
(partial_dir / "game.chd").write_bytes(b"partial")
|
||||
|
||||
assert conversion_cache.cleanup_stale_conversions() == 1
|
||||
assert not key_dir.exists()
|
||||
|
||||
def test_skips_a_dir_whose_conversion_is_running(self, cache_root):
|
||||
key_dir = cache_root / "1-busy"
|
||||
key_dir.mkdir()
|
||||
partial_dir = key_dir / PARTIAL_DIR
|
||||
partial_dir.mkdir()
|
||||
(partial_dir / "game.chd").write_bytes(b"partial")
|
||||
|
||||
with _held_lock(key_dir):
|
||||
assert conversion_cache.cleanup_stale_conversions() == 0
|
||||
assert key_dir.exists()
|
||||
|
||||
def test_kept_copy_remains_available_during_cleanup(self, cache_root, mocker):
|
||||
f = _rom_file()
|
||||
final = _seed_cached(f)
|
||||
files = conversion_cache._files
|
||||
|
||||
def serve_during_listing(key_dir):
|
||||
listed = files(key_dir)
|
||||
assert get_cached_converted(1, f, "psp", "chd") == final
|
||||
return listed
|
||||
|
||||
mocker.patch.object(
|
||||
conversion_cache, "_files", side_effect=serve_during_listing
|
||||
)
|
||||
|
||||
assert conversion_cache.cleanup_stale_conversions() == 0
|
||||
assert final.read_bytes() == b"cached"
|
||||
|
||||
def test_evicts_least_recently_served_over_the_cap(
|
||||
self, cache_root, converto, mocker
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
dirs = {}
|
||||
for name, age in (("old", 300), ("mid", 200), ("new", 100)):
|
||||
final = _seed_cached(_rom_file(file_name=f"{name}.iso"))
|
||||
final.write_bytes(b"x" * 40)
|
||||
os.utime(final, (time.time() - age, time.time() - age))
|
||||
dirs[name] = final.parent
|
||||
|
||||
assert conversion_cache.cleanup_stale_conversions() == 1
|
||||
assert not dirs["old"].exists()
|
||||
assert dirs["mid"].exists()
|
||||
assert dirs["new"].exists()
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"reserve_bytes",
|
||||
[
|
||||
pytest.param(0, id="post-conversion"),
|
||||
pytest.param(70, id="admission"),
|
||||
],
|
||||
)
|
||||
def test_size_eviction_preserves_failure_markers(
|
||||
self, cache_root, converto, mocker, reserve_bytes
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
marker_dir = cache_root / "1-failed"
|
||||
marker_dir.mkdir()
|
||||
marker = marker_dir / FAILED_FILE
|
||||
marker.touch()
|
||||
served = time.time() - 3600
|
||||
os.utime(marker, (served, served))
|
||||
recent = _seed_cached(_rom_file())
|
||||
recent.write_bytes(b"x" * 120)
|
||||
|
||||
assert conversion_cache.cleanup_stale_conversions(reserve_bytes) == 0
|
||||
assert marker.exists()
|
||||
assert recent.read_bytes() == b"x" * 120
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("age_seconds", "kept"),
|
||||
[
|
||||
pytest.param(SERVE_GRACE_SECONDS // 2, True, id="within-grace"),
|
||||
pytest.param(SERVE_GRACE_SECONDS * 2, False, id="after-grace"),
|
||||
],
|
||||
)
|
||||
def test_size_eviction_respects_serve_grace(
|
||||
self, cache_root, converto, mocker, age_seconds, kept
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
final = _seed_cached(_rom_file())
|
||||
final.write_bytes(b"x" * 120)
|
||||
served = time.time() - age_seconds
|
||||
os.utime(final, (served, served))
|
||||
|
||||
assert conversion_cache.cleanup_stale_conversions() == (0 if kept else 1)
|
||||
assert final.exists() is kept
|
||||
|
||||
def test_size_eviction_rechecks_a_copy_served_after_the_ttl_pass(
|
||||
self, cache_root, converto, mocker
|
||||
):
|
||||
converto.cache_max_size_gb = 1
|
||||
mocker.patch.object(conversion_cache, "BYTES_PER_GB", 100)
|
||||
f = _rom_file()
|
||||
final = _seed_cached(f)
|
||||
final.write_bytes(b"x" * 120)
|
||||
served = time.time() - 120
|
||||
os.utime(final, (served, served))
|
||||
iterdir = Path.iterdir
|
||||
|
||||
def serve_after_listing(path):
|
||||
yield from iterdir(path)
|
||||
if path == cache_root:
|
||||
assert get_cached_converted(1, f, "psp", "chd", touch=True) == final
|
||||
|
||||
mocker.patch.object(
|
||||
Path, "iterdir", side_effect=serve_after_listing, autospec=True
|
||||
)
|
||||
|
||||
assert conversion_cache.cleanup_stale_conversions() == 0
|
||||
assert final.read_bytes() == b"x" * 120
|
||||
|
||||
|
||||
class TestParseFormats:
|
||||
@pytest.mark.parametrize(
|
||||
("raw", "expected"),
|
||||
[
|
||||
pytest.param("zso,iso", ("zso", "iso"), id="in-order"),
|
||||
pytest.param(" ZSO , iso,zso ", ("zso", "iso"), id="trimmed-deduplicated"),
|
||||
pytest.param(",,", (), id="empty"),
|
||||
pytest.param("3ds,GCM", ("cci", "iso"), id="aliases"),
|
||||
],
|
||||
)
|
||||
def test_parses_the_list(self, raw: str, expected: tuple[str, ...]):
|
||||
assert parse_formats(raw) == expected
|
||||
|
||||
|
||||
class TestResolveFormatDownload:
|
||||
@pytest.fixture(autouse=True)
|
||||
def enabled(self, converto, mocker):
|
||||
converto.download_conversion_enabled = True
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.is_enabled",
|
||||
AsyncMock(return_value=True),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _rom() -> Rom:
|
||||
return cast(Rom, SimpleNamespace(id=1, platform_slug="psp"))
|
||||
|
||||
async def test_a_listed_stored_format_wins_over_a_cached_copy(self, cache_root):
|
||||
f = _rom_file()
|
||||
_seed_cached(f)
|
||||
|
||||
resolution = await resolve_format_download(
|
||||
self._rom(), f, ("chd", "iso"), allowed=True, start=True, touch=True
|
||||
)
|
||||
|
||||
assert resolution == FormatResolution(FormatOutcome.ORIGINAL)
|
||||
|
||||
async def test_an_aliased_files_own_extension_serves_the_original(
|
||||
self, cache_root, convert_outputs
|
||||
):
|
||||
rom = cast(Rom, SimpleNamespace(id=1, platform_slug="3ds"))
|
||||
|
||||
resolution = await resolve_format_download(
|
||||
rom,
|
||||
_rom_file(file_name="game.3ds"),
|
||||
parse_formats("3ds,cia"),
|
||||
allowed=True,
|
||||
start=True,
|
||||
touch=True,
|
||||
)
|
||||
|
||||
assert resolution == FormatResolution(FormatOutcome.ORIGINAL)
|
||||
assert convert_outputs == []
|
||||
|
||||
async def test_a_cached_copy_wins_over_an_earlier_uncached_format(
|
||||
self, cache_root, convert_outputs
|
||||
):
|
||||
f = _rom_file()
|
||||
final = _seed_cached(f)
|
||||
|
||||
resolution = await resolve_format_download(
|
||||
self._rom(), f, ("cso", "chd"), allowed=True, start=True, touch=True
|
||||
)
|
||||
|
||||
assert resolution == FormatResolution(FormatOutcome.CONVERTED, final)
|
||||
assert convert_outputs == []
|
||||
|
||||
async def test_converts_to_the_first_listed_format_it_can_produce(
|
||||
self, cache_root, convert_outputs
|
||||
):
|
||||
f = _rom_file()
|
||||
|
||||
resolution = await resolve_format_download(
|
||||
self._rom(), f, ("rvz", "cso", "chd"), allowed=True, start=True, touch=True
|
||||
)
|
||||
|
||||
assert resolution.outcome == FormatOutcome.CONVERTED
|
||||
assert resolution.path is not None and resolution.path.suffix == ".cso"
|
||||
assert [out.suffix for out in convert_outputs] == [".cso"]
|
||||
|
||||
async def test_skips_a_format_whose_conversion_failed(
|
||||
self, cache_root, convert_outputs
|
||||
):
|
||||
f = _rom_file()
|
||||
failed = _final_path(f).parent
|
||||
failed.mkdir(parents=True)
|
||||
(failed / FAILED_FILE).touch()
|
||||
|
||||
resolution = await resolve_format_download(
|
||||
self._rom(), f, ("chd", "cso"), allowed=True, start=True, touch=True
|
||||
)
|
||||
|
||||
assert resolution.path is not None and resolution.path.suffix == ".cso"
|
||||
|
||||
async def test_reports_pending_without_starting(self, cache_root, convert_outputs):
|
||||
resolution = await resolve_format_download(
|
||||
self._rom(), _rom_file(), ("chd",), allowed=True, start=False, touch=False
|
||||
)
|
||||
|
||||
assert resolution.outcome == FormatOutcome.PENDING
|
||||
assert convert_outputs == []
|
||||
|
||||
async def test_falls_back_to_a_later_format_when_a_conversion_fails(
|
||||
self, cache_root, mocker
|
||||
):
|
||||
async def convert(operation, src, out) -> None:
|
||||
if out.suffix == ".cso":
|
||||
raise RomConvertoOperationError("bad image")
|
||||
out.write_bytes(b"converted")
|
||||
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.convert", side_effect=convert
|
||||
)
|
||||
|
||||
resolution = await resolve_format_download(
|
||||
self._rom(),
|
||||
_rom_file(),
|
||||
("cso", "chd"),
|
||||
allowed=True,
|
||||
start=True,
|
||||
touch=True,
|
||||
)
|
||||
|
||||
assert resolution.path is not None and resolution.path.suffix == ".chd"
|
||||
|
||||
async def test_a_locked_key_dir_without_partial_output_is_pending(
|
||||
self, cache_root, convert_outputs
|
||||
):
|
||||
key_dir = _final_path(_rom_file()).parent
|
||||
key_dir.mkdir(parents=True)
|
||||
|
||||
with _held_lock(key_dir):
|
||||
resolution = await resolve_format_download(
|
||||
self._rom(), _rom_file(), ("chd",), allowed=True, start=True, touch=True
|
||||
)
|
||||
|
||||
assert resolution.outcome == FormatOutcome.PENDING
|
||||
assert convert_outputs == []
|
||||
|
||||
async def test_a_finished_conversion_is_served_while_a_download_holds_its_dir(
|
||||
self, cache_root, mocker
|
||||
):
|
||||
f = _rom_file()
|
||||
final = _final_path(f)
|
||||
|
||||
async def convert(operation, src, out) -> None:
|
||||
out.write_bytes(b"converted")
|
||||
|
||||
async def convert_then_serve(*args) -> Path | None:
|
||||
converted = await get_or_convert(*args)
|
||||
stack.enter_context(_held_lock(final.parent, shared=True))
|
||||
return converted
|
||||
|
||||
mocker.patch(
|
||||
"utils.conversion_cache.rom_converto_service.convert", side_effect=convert
|
||||
)
|
||||
mocker.patch.object(
|
||||
conversion_cache, "get_or_convert", side_effect=convert_then_serve
|
||||
)
|
||||
with ExitStack() as stack:
|
||||
resolution = await resolve_format_download(
|
||||
self._rom(), f, ("chd",), allowed=True, start=True, touch=True
|
||||
)
|
||||
|
||||
assert resolution == FormatResolution(FormatOutcome.CONVERTED, final)
|
||||
|
||||
async def test_is_pending_without_starting_once_too_many_conversions_run(
|
||||
self, cache_root, convert_outputs, mocker
|
||||
):
|
||||
mocker.patch.object(conversion_cache, "MAX_STARTED_CONVERSIONS", 0)
|
||||
|
||||
resolution = await resolve_format_download(
|
||||
self._rom(), _rom_file(), ("chd",), allowed=True, start=True, touch=True
|
||||
)
|
||||
|
||||
assert resolution.outcome == FormatOutcome.PENDING
|
||||
assert convert_outputs == []
|
||||
@@ -0,0 +1,415 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import enum
|
||||
import fcntl
|
||||
import hashlib
|
||||
import os
|
||||
import shutil
|
||||
import time
|
||||
from collections.abc import Generator, Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Final
|
||||
|
||||
from adapters.services.rom_converto import (
|
||||
UNCOMPRESSED_TARGETS,
|
||||
Operation,
|
||||
RomConvertoOperationError,
|
||||
RomConvertoTimeoutError,
|
||||
canonical_format,
|
||||
file_format,
|
||||
resolve_operation,
|
||||
rom_converto_service,
|
||||
)
|
||||
from config import (
|
||||
LIBRARY_BASE_PATH,
|
||||
ROM_CONVERTO_CACHE_PATH,
|
||||
ROM_CONVERTO_MAX_CONCURRENCY,
|
||||
ROMM_BASE_PATH,
|
||||
)
|
||||
from config.config_manager import config_manager as cm
|
||||
from logger.formatter import highlight as hl
|
||||
from logger.logger import log
|
||||
from utils.background_tasks import fire_and_forget
|
||||
from utils.zip_cache import CACHE_KEY_LENGTH, SECONDS_PER_HOUR
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from models.rom import Rom, RomFile
|
||||
|
||||
BYTES_PER_GB = 1024**3
|
||||
# How long a download waits on a conversion it started before answering 202.
|
||||
CONVERSION_WAIT_SECONDS: Final = 15
|
||||
RETRY_AFTER_SECONDS: Final = 30
|
||||
PARTIAL_DIR: Final = ".partial"
|
||||
FAILED_FILE: Final = ".failed"
|
||||
# Give nginx time to open a copy after the X-Accel-Redirect response.
|
||||
SERVE_GRACE_SECONDS: Final = 60
|
||||
# Room a conversion to an uncompressed target reserves per input byte; the
|
||||
# cleanup pass after it evicts any overshoot.
|
||||
EXPANSION_RESERVE_FACTOR: Final = 4
|
||||
# Downloads may start conversions until this many wait for or hold the semaphore.
|
||||
MAX_STARTED_CONVERSIONS: Final = ROM_CONVERTO_MAX_CONCURRENCY * 4
|
||||
|
||||
# Each conversion reads and writes whole disc images.
|
||||
_convert_semaphore = asyncio.Semaphore(ROM_CONVERTO_MAX_CONCURRENCY)
|
||||
_started_conversions: set[asyncio.Task[Path | None]] = set()
|
||||
|
||||
|
||||
def converted_file_path(
|
||||
rom_id: int, rom_file: RomFile, operation: Operation, input_ext: str
|
||||
) -> Path:
|
||||
"""Deterministic cache path for a converted ROM file."""
|
||||
key = hashlib.sha1(
|
||||
f"{rom_file.file_path}/{rom_file.file_name}|{rom_file.last_modified}|{rom_file.file_size_bytes}|{operation.target}".encode(),
|
||||
usedforsecurity=False,
|
||||
).hexdigest()[:CACHE_KEY_LENGTH]
|
||||
key_dir = Path(ROM_CONVERTO_CACHE_PATH) / f"{rom_id}-{key}"
|
||||
return key_dir / operation.output_name(Path(rom_file.file_name), input_ext)
|
||||
|
||||
|
||||
def get_redirect_path(converted_path: Path) -> Path:
|
||||
"""The nginx-internal path for a converted file (`/cache/` aliases `${ROMM_BASE_PATH}/cache/`)."""
|
||||
return Path("/") / converted_path.relative_to(ROMM_BASE_PATH)
|
||||
|
||||
|
||||
def _dir_size(path: Path) -> int:
|
||||
total = 0
|
||||
for root, _, files in os.walk(path):
|
||||
for name in files:
|
||||
with contextlib.suppress(OSError):
|
||||
total += os.stat(os.path.join(root, name)).st_size
|
||||
return total
|
||||
|
||||
|
||||
def _files(key_dir: Path) -> list[Path]:
|
||||
return [p for p in key_dir.iterdir() if p.is_file()]
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _try_lock(key_dir: Path, *, shared: bool = False) -> Generator[bool]:
|
||||
"""Lock the key dir without blocking, yielding False if it is busy, removed or replaced."""
|
||||
fd = os.open(key_dir, os.O_RDONLY)
|
||||
try:
|
||||
try:
|
||||
mode = fcntl.LOCK_SH if shared else fcntl.LOCK_EX
|
||||
fcntl.flock(fd, mode | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
yield False
|
||||
return
|
||||
try:
|
||||
current_dir = os.path.samestat(os.fstat(fd), os.stat(key_dir))
|
||||
except FileNotFoundError:
|
||||
current_dir = False
|
||||
yield current_dir
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def _stat_files(key_dir: Path) -> tuple[dict[str, os.stat_result], float]:
|
||||
"""The stats of `key_dir`'s files and the newest mtime among them."""
|
||||
stats = {p.name: p.stat() for p in _files(key_dir)}
|
||||
return stats, max((st.st_mtime for st in stats.values()), default=0.0)
|
||||
|
||||
|
||||
def cache_size_bytes() -> int:
|
||||
"""Bytes every key dir holds on disk, partial output included."""
|
||||
return _dir_size(Path(ROM_CONVERTO_CACHE_PATH))
|
||||
|
||||
|
||||
def has_room_for(size_bytes: int) -> bool:
|
||||
"""Whether adding `size_bytes` keeps the cache under `converto.cache_max_size_gb`."""
|
||||
max_bytes = cm.get_config().CONVERTO.cache_max_size_gb * BYTES_PER_GB
|
||||
return not max_bytes or cache_size_bytes() + size_bytes <= max_bytes
|
||||
|
||||
|
||||
def _lookup(
|
||||
rom_id: int, rom_file: RomFile, platform_slug: str, target: str
|
||||
) -> tuple[Operation, Path] | None:
|
||||
"""The operation bringing `rom_file` to `target` and its cache path."""
|
||||
resolved = resolve_operation(platform_slug, target, rom_file.file_name)
|
||||
if resolved is None:
|
||||
return None
|
||||
operation, input_ext = resolved
|
||||
return operation, converted_file_path(rom_id, rom_file, operation, input_ext)
|
||||
|
||||
|
||||
def _serve_cached(final_path: Path, *, touch: bool) -> Path | None:
|
||||
"""`final_path` if it is cached, or None while cleanup or a conversion holds its dir."""
|
||||
with (
|
||||
contextlib.suppress(OSError),
|
||||
_try_lock(final_path.parent, shared=True) as locked,
|
||||
):
|
||||
if locked and final_path.exists():
|
||||
if touch:
|
||||
# Keep a served file fresh so TTL cleanup measures demand.
|
||||
with contextlib.suppress(OSError):
|
||||
os.utime(final_path)
|
||||
return final_path
|
||||
return None
|
||||
|
||||
|
||||
def get_cached_converted(
|
||||
rom_id: int,
|
||||
rom_file: RomFile,
|
||||
platform_slug: str,
|
||||
target: str,
|
||||
*,
|
||||
touch: bool = False,
|
||||
) -> Path | None:
|
||||
"""The converted file if it is already cached, without converting.
|
||||
|
||||
Args:
|
||||
touch: Refresh its mtime so TTL cleanup measures demand.
|
||||
"""
|
||||
found = _lookup(rom_id, rom_file, platform_slug, target)
|
||||
return _serve_cached(found[1], touch=touch) if found else None
|
||||
|
||||
|
||||
class FormatOutcome(enum.StrEnum):
|
||||
ORIGINAL = "original"
|
||||
CONVERTED = "converted"
|
||||
PENDING = "pending"
|
||||
UNAVAILABLE = "unavailable"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FormatResolution:
|
||||
outcome: FormatOutcome
|
||||
# The cached copy to serve when `outcome` is CONVERTED.
|
||||
path: Path | None = None
|
||||
|
||||
|
||||
_PENDING: Final = FormatResolution(FormatOutcome.PENDING)
|
||||
_UNAVAILABLE: Final = FormatResolution(FormatOutcome.UNAVAILABLE)
|
||||
|
||||
|
||||
def parse_formats(raw: str) -> tuple[str, ...]:
|
||||
"""The comma-separated `?format=` list, canonical, lowercased and deduplicated in order."""
|
||||
formats = (canonical_format(f.strip().lower()) for f in raw.split(","))
|
||||
return tuple(dict.fromkeys(f for f in formats if f))
|
||||
|
||||
|
||||
def _is_converting(final_path: Path) -> bool:
|
||||
"""Whether a worker holds `final_path`'s key dir, which it locks before converting."""
|
||||
# A shared probe leaves downloads serving a cached copy undisturbed.
|
||||
with (
|
||||
contextlib.suppress(OSError),
|
||||
_try_lock(final_path.parent, shared=True) as locked,
|
||||
):
|
||||
return not locked
|
||||
return False
|
||||
|
||||
|
||||
async def resolve_format_download(
|
||||
rom: Rom,
|
||||
file: RomFile,
|
||||
formats: Sequence[str],
|
||||
*,
|
||||
allowed: bool,
|
||||
start: bool,
|
||||
touch: bool,
|
||||
) -> FormatResolution:
|
||||
"""What a `?format=` download serves: the stored file if listed, else the first cached, then the first convertible format.
|
||||
|
||||
Args:
|
||||
allowed: The caller may start a conversion.
|
||||
start: Start it now, rather than only report that one would run.
|
||||
touch: Refresh a served copy's mtime so TTL cleanup measures demand.
|
||||
"""
|
||||
if file_format(file.file_name) in formats:
|
||||
return FormatResolution(FormatOutcome.ORIGINAL)
|
||||
|
||||
converto = cm.get_config().CONVERTO
|
||||
if (
|
||||
not converto.download_conversion_enabled
|
||||
or not await rom_converto_service.is_enabled()
|
||||
):
|
||||
return _UNAVAILABLE
|
||||
|
||||
candidates = [
|
||||
(target, found[1])
|
||||
for target in formats
|
||||
if (found := _lookup(rom.id, file, rom.platform_slug, target))
|
||||
]
|
||||
for _, final_path in candidates:
|
||||
if cached := _serve_cached(final_path, touch=touch):
|
||||
return FormatResolution(FormatOutcome.CONVERTED, cached)
|
||||
if not allowed:
|
||||
return _UNAVAILABLE
|
||||
|
||||
for target, final_path in candidates:
|
||||
if (final_path.parent / FAILED_FILE).exists():
|
||||
continue
|
||||
if (
|
||||
not start
|
||||
or _is_converting(final_path)
|
||||
or len(_started_conversions) >= MAX_STARTED_CONVERSIONS
|
||||
):
|
||||
return _PENDING
|
||||
# The conversion finishes into the cache even when this request stops waiting.
|
||||
conversion = fire_and_forget(
|
||||
get_or_convert(rom.id, file, rom.platform_slug, target)
|
||||
)
|
||||
_started_conversions.add(conversion)
|
||||
conversion.add_done_callback(_started_conversions.discard)
|
||||
done, _ = await asyncio.wait({conversion}, timeout=CONVERSION_WAIT_SECONDS)
|
||||
if done and (converted := conversion.result()):
|
||||
return FormatResolution(FormatOutcome.CONVERTED, converted)
|
||||
if not done or _is_converting(final_path):
|
||||
return _PENDING
|
||||
return _UNAVAILABLE
|
||||
|
||||
|
||||
async def get_or_convert(
|
||||
rom_id: int,
|
||||
rom_file: RomFile,
|
||||
platform_slug: str,
|
||||
target: str,
|
||||
) -> Path | None:
|
||||
"""The converted file, converting it under a lock on its key dir, or None when it can't be."""
|
||||
found = _lookup(rom_id, rom_file, platform_slug, target)
|
||||
if found is None:
|
||||
return None
|
||||
operation, final_path = found
|
||||
|
||||
try:
|
||||
if cached := _serve_cached(final_path, touch=True):
|
||||
return cached
|
||||
|
||||
final_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with _try_lock(final_path.parent) as locked:
|
||||
if not locked:
|
||||
# Another worker is converting it.
|
||||
return None
|
||||
# A conversion may have finished between the check above and the lock.
|
||||
if final_path.exists():
|
||||
with contextlib.suppress(OSError):
|
||||
os.utime(final_path)
|
||||
return final_path
|
||||
if (final_path.parent / FAILED_FILE).exists():
|
||||
return None
|
||||
async with _convert_semaphore:
|
||||
await asyncio.to_thread(
|
||||
shutil.rmtree, final_path.parent / PARTIAL_DIR, True
|
||||
)
|
||||
size_bytes = (rom_file.file_size_bytes or 0) * (
|
||||
EXPANSION_RESERVE_FACTOR
|
||||
if operation.target in UNCOMPRESSED_TARGETS
|
||||
else 1
|
||||
)
|
||||
await asyncio.to_thread(cleanup_stale_conversions, size_bytes)
|
||||
if not await asyncio.to_thread(has_room_for, size_bytes):
|
||||
log.info(
|
||||
f"Conversion cache is full, not converting ROM {rom_id} (target {hl(target)})"
|
||||
)
|
||||
return None
|
||||
converted = await _convert(
|
||||
rom_id, rom_file, target, operation, final_path
|
||||
)
|
||||
if converted:
|
||||
try:
|
||||
await asyncio.to_thread(cleanup_stale_conversions)
|
||||
except Exception as e:
|
||||
log.warning(
|
||||
f"Conversion cache cleanup failed for ROM {rom_id}: {e}; serving converted copy"
|
||||
)
|
||||
return converted
|
||||
except Exception as e:
|
||||
log.warning(
|
||||
f"Conversion cache unavailable for ROM {rom_id} (target {hl(target)}): {e}"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
async def _convert(
|
||||
rom_id: int, rom_file: RomFile, target: str, operation: Operation, final_path: Path
|
||||
) -> Path | None:
|
||||
"""Convert into a partial file and rename it into place; the caller holds the key dir lock."""
|
||||
partial_dir = final_path.parent / PARTIAL_DIR
|
||||
produced = partial_dir / final_path.name
|
||||
try:
|
||||
partial_dir.mkdir()
|
||||
await rom_converto_service.convert(
|
||||
operation, src=Path(LIBRARY_BASE_PATH) / rom_file.full_path, out=produced
|
||||
)
|
||||
os.replace(produced, final_path)
|
||||
except Exception as e:
|
||||
log.warning(f"Conversion failed for ROM {rom_id} (target {hl(target)}): {e}")
|
||||
if isinstance(e, (RomConvertoOperationError, RomConvertoTimeoutError)):
|
||||
(final_path.parent / FAILED_FILE).touch()
|
||||
return None
|
||||
finally:
|
||||
await asyncio.to_thread(shutil.rmtree, partial_dir, True)
|
||||
return final_path
|
||||
|
||||
|
||||
def cleanup_stale_conversions(reserve_bytes: int = 0) -> int:
|
||||
"""Remove expired or leaked key dirs, then evict least recently served copies to make room."""
|
||||
cache_root = Path(ROM_CONVERTO_CACHE_PATH)
|
||||
if not cache_root.exists():
|
||||
return 0
|
||||
|
||||
converto = cm.get_config().CONVERTO
|
||||
now = time.time()
|
||||
expired_before = now - converto.cache_ttl_hours * SECONDS_PER_HOUR
|
||||
deleted = 0
|
||||
# (last served, size) of the final copies in each dir that survives the TTL pass.
|
||||
kept: list[tuple[float, int, Path]] = []
|
||||
|
||||
for key_dir in cache_root.iterdir():
|
||||
if not key_dir.is_dir():
|
||||
continue
|
||||
with contextlib.suppress(FileNotFoundError):
|
||||
stats, last_served = _stat_files(key_dir)
|
||||
partial_dir = key_dir / PARTIAL_DIR
|
||||
if last_served < expired_before or partial_dir.exists():
|
||||
with _try_lock(key_dir) as locked:
|
||||
if locked:
|
||||
stats, last_served = _stat_files(key_dir)
|
||||
if last_served < expired_before:
|
||||
shutil.rmtree(key_dir, ignore_errors=True)
|
||||
deleted += 1
|
||||
continue
|
||||
shutil.rmtree(partial_dir, ignore_errors=True)
|
||||
# A failure marker frees nothing, and evicting it would retry the conversion.
|
||||
stats.pop(FAILED_FILE, None)
|
||||
if stats:
|
||||
kept.append(
|
||||
(
|
||||
max(st.st_mtime for st in stats.values()),
|
||||
sum(st.st_size for st in stats.values()),
|
||||
key_dir,
|
||||
)
|
||||
)
|
||||
|
||||
max_bytes = converto.cache_max_size_gb * BYTES_PER_GB
|
||||
if not max_bytes:
|
||||
return deleted
|
||||
used = cache_size_bytes()
|
||||
candidates = sorted(c for c in kept if c[0] < now - SERVE_GRACE_SECONDS)
|
||||
if reserve_bytes and used + reserve_bytes > max_bytes:
|
||||
evictable = 0
|
||||
for _, size, key_dir in candidates:
|
||||
# A dir a download or conversion holds right now can't be evicted.
|
||||
with contextlib.suppress(FileNotFoundError), _try_lock(key_dir) as locked:
|
||||
evictable += size if locked else 0
|
||||
if used + reserve_bytes - evictable > max_bytes:
|
||||
# Evicting every candidate would still not make room, so keep them.
|
||||
return deleted
|
||||
for _, _, key_dir in candidates:
|
||||
if used + reserve_bytes <= max_bytes:
|
||||
break
|
||||
with contextlib.suppress(FileNotFoundError), _try_lock(key_dir) as locked:
|
||||
# A download may have touched the copy since the pass above.
|
||||
if not locked or any(
|
||||
p.stat().st_mtime >= now - SERVE_GRACE_SECONDS for p in _files(key_dir)
|
||||
):
|
||||
continue
|
||||
size = _dir_size(key_dir)
|
||||
shutil.rmtree(key_dir, ignore_errors=True)
|
||||
used -= size
|
||||
deleted += 1
|
||||
|
||||
return deleted
|
||||
@@ -5,6 +5,7 @@
|
||||
# - backend-dev-build: Similar to `backend-build`, but also compiles and installs development dependencies
|
||||
# - rahasher-build: Build RAHasher
|
||||
# - sigil-build: Build sigil Python bindings (title ID extraction)
|
||||
# - rom-converto-fetch: Fetch the rom-converto CLI (ROM conversion and metadata)
|
||||
# - emulator-stage: Fetch and extract emulators
|
||||
# - nginx-build: Build nginx modules
|
||||
# - production-stage: Setup frontend and backend
|
||||
@@ -144,6 +145,22 @@ RUN git clone --filter=blob:none https://github.com/rommapp/argosy-sigil.git &&
|
||||
cp ./sigil/*.py ./sigil/_sigil.*.so /sigil-pkg/
|
||||
|
||||
|
||||
# ROM-CONVERTO CLI
|
||||
# Keep the version and both sums in sync with the dev image's copy in ../Dockerfile.
|
||||
FROM alpine:${ALPINE_VERSION}@sha256:${ALPINE_SHA256} AS rom-converto-fetch
|
||||
ARG TARGETARCH
|
||||
ARG ROM_CONVERTO_VERSION=v0.23.0
|
||||
RUN case "${TARGETARCH}" in \
|
||||
amd64) rc="linux-x64-musl"; sum="37d03a67ad97fc02768890e763b4c80072fdfd15f2c5b97e44f9a2b50fc40876" ;; \
|
||||
arm64) rc="linux-arm64-musl"; sum="8abe37ca2d1a0e3d719fb8e795e011c672e3f1c088521cf0bb1e467dc658dc3c" ;; \
|
||||
*) echo "unsupported TARGETARCH: ${TARGETARCH}" && exit 1 ;; \
|
||||
esac && \
|
||||
wget -qO /rom-converto \
|
||||
"https://github.com/DevYukine/rom-converto/releases/download/${ROM_CONVERTO_VERSION}/rom-converto-cli-${rc}" && \
|
||||
echo "${sum} /rom-converto" | sha256sum -c - && \
|
||||
chmod 0755 /rom-converto
|
||||
|
||||
|
||||
# FETCH EMULATORJS, RUFFLE, JS-DOS AND FAKE-08
|
||||
# Keep the pins below in sync with the dev image's copies in ../Dockerfile.
|
||||
# checkov:skip=CKV_DOCKER_4:every ADD below is a remote URL pinned by --checksum
|
||||
@@ -258,6 +275,7 @@ COPY --from=python-runtime /usr/local/lib/libpython* /usr/local/lib/
|
||||
COPY --from=python-runtime /usr/local/lib/python${PYTHON_VERSION} /usr/local/lib/python${PYTHON_VERSION}
|
||||
|
||||
COPY --from=rahasher-build /RALibretro/bin64/RAHasher /usr/bin/RAHasher
|
||||
COPY --from=rom-converto-fetch /rom-converto /usr/bin/rom-converto
|
||||
COPY --from=nginx-build ./nginx/objs/ngx_http_zip_module.so /usr/lib/nginx/modules/
|
||||
COPY --from=frontend-build /front/dist ${WEBSERVER_FOLDER}
|
||||
COPY --from=backend-node-build /backend-node/rom_patcher/node_modules/rom-patcher/rom-patcher-js /backend/utils/rom_patcher/rom-patcher-js
|
||||
|
||||
@@ -321,6 +321,7 @@ backend/
|
||||
│ │ └── reap_streaming_sessions.py # Stop abandoned streaming sessions
|
||||
│ └── manual/ # On-demand tasks
|
||||
│ ├── cleanup_missing_roms.py # Drop DB entries for missing files
|
||||
│ ├── convert_library.py # Store matched ROMs in each platform's library format
|
||||
│ ├── cleanup_orphaned_resources.py # Remove unreferenced artwork
|
||||
│ └── sync_folder_scan.py # Scan sync folder for new saves
|
||||
│
|
||||
@@ -908,8 +909,11 @@ A reset link is emailed when SMTP is set up, the user has an address and `ROMM_B
|
||||
| DELETE | `/{id}` | ROMS_WRITE | Delete ROM |
|
||||
| POST | `/delete` | ROMS_WRITE | Bulk delete |
|
||||
| POST | `/download/{id}/{file_name}` | ROMS_READ | Download ROM |
|
||||
| GET | `/{id}/content/{file_name}` | ROMS_READ | Download ROM, in a listed `?format=` when asked |
|
||||
| POST | `/unidentified` | ROMS_READ | Get unidentified ROMs |
|
||||
|
||||
With rom-converto enabled (`ROM_CONVERTO_ENABLED`, `converto.download_conversion_enabled`), `?format=zso,iso` lists the formats a client can read. A single-file download whose stored format is listed is served as-is. Otherwise the first listed format with a copy cached under `/romm/cache/converts` is served, then the first one rom-converto can produce: answered by the same request if it converts within a few seconds, else `202` with `Retry-After` while it finishes. `406` means no listed format can be served, and `HEAD` reports the same without starting a conversion. Only signed-in, non-kiosk users start conversions. `CONVERTO_DOWNLOAD_FORMATS` in `/api/config` lists the formats each input extension converts to.
|
||||
|
||||
#### ROM Upload (Chunked)
|
||||
|
||||
| Method | Path | Scope | Description |
|
||||
@@ -1679,29 +1683,33 @@ failure callback and a scan needs one to report a worker that died mid-scan.
|
||||
|
||||
Toggled via environment variables:
|
||||
|
||||
| Task | Env Toggle | Default Cron | Description |
|
||||
| --------------------------------- | ----------------------------------------------------- | ------------------ | ---------------------- |
|
||||
| `scan_library` | `ENABLE_SCHEDULED_RESCAN` | `0 3 * * *` (3 AM) | Full library rescan |
|
||||
| `update_switch_titledb` | `ENABLE_SCHEDULED_UPDATE_SWITCH_TITLEDB` | `0 4 * * *` | Update Switch game DB |
|
||||
| `update_launchbox_metadata` | `ENABLE_SCHEDULED_UPDATE_LAUNCHBOX_METADATA` | `0 4 * * *` | Refresh LaunchBox data |
|
||||
| `convert_images_to_webp` | `ENABLE_SCHEDULED_CONVERT_IMAGES_TO_WEBP` | `0 4 * * *` | Image optimization |
|
||||
| `sync_retroachievements_progress` | `ENABLE_SCHEDULED_RETROACHIEVEMENTS_PROGRESS_SYNC` | `0 4 * * *` | Sync RA user progress |
|
||||
| `cleanup_orphaned_resources` | `ENABLE_SCHEDULED_CLEANUP_ORPHANED_RESOURCES` | `0 5 * * *` | Remove unused artwork |
|
||||
| `cleanup_netplay` | `ENABLE_SCHEDULED_CLEANUP_NETPLAY` (default on) | `*/30 * * * *` | Clean empty rooms |
|
||||
| `cleanup_upload_tmp` | `ENABLE_SCHEDULED_CLEANUP_UPLOAD_TMP` (default on) | `0 * * * *` | Drop stale uploads |
|
||||
| `cleanup_zip_cache` | `ENABLE_SCHEDULED_CLEANUP_ZIP_CACHE` (default on) | `0 4 * * *` | Drop stale cached ZIPs |
|
||||
| `cleanup_sync_sessions` | `ENABLE_SCHEDULED_CLEANUP_SYNC_SESSIONS` (default on) | `23 * * * *` | Fail abandoned syncs |
|
||||
| `reap_streaming_sessions` | `streaming.enabled` in config, read at startup | `* * * * *` | Stop abandoned streams |
|
||||
| `cleanup_audit_log` | `AUDIT_LOG_RETENTION_DAYS` above 0 (default 90) | `30 4 * * *` | Prune old audit events |
|
||||
| Task | Env Toggle | Default Cron | Description |
|
||||
| --------------------------------- | ----------------------------------------------------- | ------------------ | ------------------------------ |
|
||||
| `scan_library` | `ENABLE_SCHEDULED_RESCAN` | `0 3 * * *` (3 AM) | Full library rescan |
|
||||
| `update_switch_titledb` | `ENABLE_SCHEDULED_UPDATE_SWITCH_TITLEDB` | `0 4 * * *` | Update Switch game DB |
|
||||
| `update_launchbox_metadata` | `ENABLE_SCHEDULED_UPDATE_LAUNCHBOX_METADATA` | `0 4 * * *` | Refresh LaunchBox data |
|
||||
| `convert_images_to_webp` | `ENABLE_SCHEDULED_CONVERT_IMAGES_TO_WEBP` | `0 4 * * *` | Image optimization |
|
||||
| `sync_retroachievements_progress` | `ENABLE_SCHEDULED_RETROACHIEVEMENTS_PROGRESS_SYNC` | `0 4 * * *` | Sync RA user progress |
|
||||
| `cleanup_orphaned_resources` | `ENABLE_SCHEDULED_CLEANUP_ORPHANED_RESOURCES` | `0 5 * * *` | Remove unused artwork |
|
||||
| `cleanup_netplay` | `ENABLE_SCHEDULED_CLEANUP_NETPLAY` (default on) | `*/30 * * * *` | Clean empty rooms |
|
||||
| `cleanup_upload_tmp` | `ENABLE_SCHEDULED_CLEANUP_UPLOAD_TMP` (default on) | `0 * * * *` | Drop stale uploads |
|
||||
| `cleanup_zip_cache` | `ENABLE_SCHEDULED_CLEANUP_ZIP_CACHE` (default on) | `0 4 * * *` | Drop stale cached ZIPs |
|
||||
| `cleanup_conversion_cache` | Always on | `0 4 * * *` | Drop stale converted downloads |
|
||||
| `cleanup_sync_sessions` | `ENABLE_SCHEDULED_CLEANUP_SYNC_SESSIONS` (default on) | `23 * * * *` | Fail abandoned syncs |
|
||||
| `reap_streaming_sessions` | `streaming.enabled` in config, read at startup | `* * * * *` | Stop abandoned streams |
|
||||
| `cleanup_audit_log` | `AUDIT_LOG_RETENTION_DAYS` above 0 (default 90) | `30 4 * * *` | Prune old audit events |
|
||||
|
||||
### Manual Tasks
|
||||
|
||||
Triggered via `POST /api/tasks/run/{task_name}`, which enqueues on `low_prio_queue` and answers 503 when no live worker is listening on it:
|
||||
|
||||
| Task | Description |
|
||||
| ---------------------- | --------------------------------------------- |
|
||||
| `cleanup_missing_roms` | Remove DB entries for files no longer on disk |
|
||||
| `sync_folder_scan` | Scan sync folder for new device saves |
|
||||
| Task | Description |
|
||||
| ---------------------- | ------------------------------------------------------------------------------------- |
|
||||
| `cleanup_missing_roms` | Remove DB entries for files no longer on disk |
|
||||
| `sync_folder_scan` | Scan sync folder for new device saves |
|
||||
| `convert_library` | Convert matched ROMs to `converto.platform_formats` in place, replacing the originals |
|
||||
|
||||
`convert_library` only converts losslessly (no xiso) and only identified ROMs, since a converted file no longer hash-matches a DAT. It stages each output under a `.romm_tmp_` directory beside the ROM, deletes the originals (and a cue's tracks) once the output is in place, rewrites `.m3u` entries in folder ROMs, and refreshes the ROM's files, so the ROM keeps its id, saves and collections.
|
||||
|
||||
`cleanup_orphaned_resources` is also runnable this way; it is listed under
|
||||
Scheduled Tasks because it additionally supports an opt-in cron schedule. It
|
||||
|
||||
@@ -150,6 +150,11 @@ DISABLE_RUFFLE_RS=false # Disable in-browser Flash playback via RuffleRS
|
||||
DISABLE_JSDOS=false # Disable in-browser Win3.x and Win9.x playback via js-dos
|
||||
DISABLE_PICO8=false # Disable in-browser PICO-8 playback via FAKE-08
|
||||
|
||||
# ROM conversion
|
||||
ROM_CONVERTO_ENABLED=false # Enable rom-converto for the "Convert library" task and ?format= download conversions
|
||||
ROM_CONVERTO_TIMEOUT=600 # Timeout for each rom-converto operation in seconds
|
||||
ROM_CONVERTO_MAX_CONCURRENCY=2 # Concurrent conversions per web or task worker
|
||||
|
||||
# Integrations
|
||||
YOUTUBE_BASE_URL=https://www.youtube.com # Base URL for alternate YouTube frontends (Piped, Invidious, etc.)
|
||||
TINFOIL_WELCOME_MESSAGE="RomM Switch Library" # Welcome message shown in Tinfoil Switch clients
|
||||
|
||||
@@ -253,3 +253,24 @@
|
||||
# core: parallel_n64
|
||||
# # Lets a core listed as broken by the broker run anyway.
|
||||
# # experimental_cores: true
|
||||
|
||||
# converto:
|
||||
# # Convert a single-file download into a format the client lists with
|
||||
# # ?format=, e.g. ?format=zso,iso (needs ROM_CONVERTO_ENABLED)
|
||||
# download_conversion_enabled: false
|
||||
# # Hours a converted download stays cached after it was last downloaded
|
||||
# cache_ttl_hours: 24
|
||||
# # Cache size before the least recently downloaded copies are evicted (0: unbounded)
|
||||
# cache_max_size_gb: 20
|
||||
# # Platform slug -> the format the "Convert library" task stores its matched
|
||||
# # roms in, replacing the original files. Formats per platform:
|
||||
# # 3ds: cci, cia, z3ds
|
||||
# # psp, ps2: chd, cso, iso, zso
|
||||
# # psx, saturn, segacd, dc: chd
|
||||
# # ngc: iso, rvz
|
||||
# # wii: iso, rvz, wbfs
|
||||
# # switch, switch-2: nsp, nsz, xci, xcz
|
||||
# # xbox360: zar
|
||||
# platform_formats:
|
||||
# psx: chd
|
||||
# ngc: rvz
|
||||
|
||||
Generated
+2
@@ -82,6 +82,8 @@ export type { ContainerSessionSchema } from './models/ContainerSessionSchema';
|
||||
export type { ConversionStats } from './models/ConversionStats';
|
||||
export type { ConversionTaskMeta } from './models/ConversionTaskMeta';
|
||||
export type { ConversionTaskStatusResponse } from './models/ConversionTaskStatusResponse';
|
||||
export type { ConvertoConfig } from './models/ConvertoConfig';
|
||||
export type { ConvertoSettingsPayload } from './models/ConvertoSettingsPayload';
|
||||
export type { CoverResource } from './models/CoverResource';
|
||||
export type { CsdbCredit } from './models/CsdbCredit';
|
||||
export type { CustomLimitOffsetPage_SimpleRomSchema_ } from './models/CustomLimitOffsetPage_SimpleRomSchema_';
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
/* istanbul ignore file */
|
||||
/* tslint:disable */
|
||||
/* eslint-disable */
|
||||
import type { ConvertoConfig } from './ConvertoConfig';
|
||||
import type { EjsControls } from './EjsControls';
|
||||
import type { MetadataMediaType } from './MetadataMediaType';
|
||||
import type { NetplayICEServer } from './NetplayICEServer';
|
||||
@@ -42,5 +43,8 @@ export type ConfigResponse = {
|
||||
GAMELIST_MEDIA_THUMBNAIL: MetadataMediaType;
|
||||
GAMELIST_MEDIA_IMAGE: MetadataMediaType;
|
||||
PEGASUS_AUTO_EXPORT_ON_SCAN: boolean;
|
||||
CONVERTO: ConvertoConfig;
|
||||
CONVERTO_LIBRARY_TARGETS: Record<string, Array<string>>;
|
||||
CONVERTO_DOWNLOAD_FORMATS: Record<string, Record<string, Array<string>>>;
|
||||
};
|
||||
|
||||
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
/* generated using openapi-typescript-codegen -- do not edit */
|
||||
/* istanbul ignore file */
|
||||
/* tslint:disable */
|
||||
/* eslint-disable */
|
||||
export type ConvertoConfig = {
|
||||
download_conversion_enabled?: boolean;
|
||||
cache_ttl_hours?: number;
|
||||
cache_max_size_gb?: number;
|
||||
platform_formats?: Record<string, string>;
|
||||
};
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
/* generated using openapi-typescript-codegen -- do not edit */
|
||||
/* istanbul ignore file */
|
||||
/* tslint:disable */
|
||||
/* eslint-disable */
|
||||
/**
|
||||
* The converto.* settings editable at runtime; `cache_ttl_hours` stays config.yml-only.
|
||||
*/
|
||||
export type ConvertoSettingsPayload = {
|
||||
download_conversion_enabled: boolean;
|
||||
cache_max_size_gb: number;
|
||||
platform_formats: Record<string, string>;
|
||||
};
|
||||
|
||||
@@ -51,6 +51,14 @@ const defaultConfig = {
|
||||
GAMELIST_MEDIA_THUMBNAIL: "box2d",
|
||||
GAMELIST_MEDIA_IMAGE: "screenshot",
|
||||
PEGASUS_AUTO_EXPORT_ON_SCAN: false,
|
||||
CONVERTO: {
|
||||
download_conversion_enabled: false,
|
||||
cache_ttl_hours: 24,
|
||||
cache_max_size_gb: 20,
|
||||
platform_formats: {},
|
||||
},
|
||||
CONVERTO_LIBRARY_TARGETS: {},
|
||||
CONVERTO_DOWNLOAD_FORMATS: {},
|
||||
} as ConfigResponse;
|
||||
|
||||
export default defineStore("config", {
|
||||
|
||||
Reference in New Issue
Block a user