mirror of
https://github.com/rommapp/romm.git
synced 2026-10-11 22:40:04 +00:00
fix(docker): boot when the container runs as a non-default UID (#5189)
* fix(docker): boot when the container runs as a non-default UID Unraid runs the container as 99:100, which can't write to the image's /romm (owned by 1000, mode 755), only to the mounted subfolders. The PSP pending dir at /romm/cache/retroarch_sync was created at import time, so every gunicorn worker failed to boot. - Buffer pending PSP save files under the user's assets folder (assets/users/<user>/retroarch/psp_pending/), beside the other RetroArch Cloud Sync files and outside the client-reachable blob categories. - Move the zip and rom-converto download caches to /tmp/romm/cache, and point nginx's internal /cache/ alias there. - Put gunicorn's control socket in /tmp/gunicorn instead of $HOME/.gunicorn, since HOME is / for that UID. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(docker): honor ROMM_TMP_PATH for caches, explain an unwritable sync folder - The download caches live under $ROMM_TMP_PATH/cache when it's set, so installs with a small Docker disk (Unraid's docker.img) can point them at a share. The entrypoint exports the same path for nginx's /cache/ alias. - With ENABLE_SYNC_FOLDER_WATCHER on and /romm/sync not writable, /init warns and skips the watcher instead of dying on mkdir under errexit. - The compose example lists the /romm/sync mount the sync features need. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * test(sync): cover reading back and clearing buffered PSP files Addresses Greptile's review on #5189. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(sync): apply code-review findings to PSP pending buffer Remove the emptied psp_pending folder after a merge, define the sync folder path once in init, update the architecture doc's PSP and blob locations, and drop a stale .gitignore entry. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(sync): apply simplify findings to PSP pending cleanup Clear the buffered folder with remove_directory, let the startup check own creating the sync folder, and reword the test fixture docstring. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(sync): cap buffered PSP files and drop them on delete Buffered files for an unresolved save folder now count against the bundle limits, so a folder can't fill the assets share and then fail every merge once it resolves. A DELETE also removes the buffered copy, so a deleted file isn't merged back when the folder resolves. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(docker): check the sync folder for push-pull sync too ENABLE_SYNC_PUSH_PULL keeps SSH keys and device saves under /romm/sync as well, so an unwritable folder now disables it with the watcher and warns at boot instead of failing when the task runs. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(docker): strip whitespace from env vars like the backend does The backend strips every variable it reads, while the init script and nginx templates used raw values, so a padded ROMM_TMP_PATH sent nginx's /cache/ alias somewhere the backend never writes. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * chore(config): drop the CACHE_BASE_PATH comment Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(docker): say which backend config strips env whitespace Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * revert(docker): drop the entrypoint env whitespace trimming Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(sync): apply pr-ready code-review findings Gate push-pull on its SSH keys path rather than a writable sync folder, accept the same truthy values as the backend, and clear only the merged PSP pending files so another worker's buffered file survives. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor: apply pr-ready simplify findings Share one truthy pattern across init's flag checks (the watcher gates now accept the same values as the backend), disable gunicorn's unused control socket instead of placing it, derive the converts redirect from its own cache path, and factor the unknown PSP folder in tests. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(docker): apply pr-ready humanizer pass to the sync mount note Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(fs): create handler base folders on first write, skip leftover temp files FSHandler no longer mkdirs its base path when constructed, so a module-level handler rooted at an unmounted or unwritable folder can't take down boot; every write path already creates its parent folders. Blob listings skip .romm_tmp_ files a crashed write left behind, so they are never merged into a PSP bundle or advertised in the sync manifest. The prefix now lives in one TEMP_FILE_PREFIX constant. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
Georges-Antoine Assi
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
1060afc1af
commit
e232a52656
@@ -65,8 +65,6 @@ frontend/assets/easyrpg
|
||||
backend/romm_test/resources
|
||||
backend/romm_test/logs
|
||||
backend/romm_test/assets
|
||||
backend/romm_test/retroarch_sync
|
||||
backend/romm_test/cache/retroarch_sync
|
||||
.pytest_cache
|
||||
.hypothesis
|
||||
/romm_test
|
||||
|
||||
@@ -38,7 +38,10 @@ ROMM_TMP_PATH: Final[str | None] = _get_env("ROMM_TMP_PATH")
|
||||
LIBRARY_BASE_PATH: Final[str] = f"{ROMM_BASE_PATH}/library"
|
||||
RESOURCES_BASE_PATH: Final[str] = f"{ROMM_BASE_PATH}/resources"
|
||||
ASSETS_BASE_PATH: Final[str] = f"{ROMM_BASE_PATH}/assets"
|
||||
ZIP_CACHE_PATH: Final[str] = f"{ROMM_BASE_PATH}/cache/zips"
|
||||
CACHE_BASE_PATH: Final[str] = (
|
||||
f"{ROMM_TMP_PATH or '/tmp/romm'}/cache" # trunk-ignore(bandit/B108)
|
||||
)
|
||||
ZIP_CACHE_PATH: Final[str] = f"{CACHE_BASE_PATH}/zips"
|
||||
FRONTEND_RESOURCES_PATH: Final[str] = "/assets/romm/resources"
|
||||
|
||||
# ROM UPLOADS
|
||||
@@ -76,7 +79,7 @@ ROM_CONVERTO_MAX_CONCURRENCY: Final[int] = max(
|
||||
1, safe_int(_get_env("ROM_CONVERTO_MAX_CONCURRENCY"), 2)
|
||||
)
|
||||
# Disk cache for converted downloads, under the tree nginx serves at /cache/.
|
||||
ROM_CONVERTO_CACHE_PATH: Final[str] = f"{ROMM_BASE_PATH}/cache/converts"
|
||||
ROM_CONVERTO_CACHE_PATH: Final[str] = f"{CACHE_BASE_PATH}/converts"
|
||||
|
||||
# DATABASE
|
||||
DB_HOST: Final[str | None] = _get_env("DB_HOST")
|
||||
@@ -384,8 +387,6 @@ SYNC_SSH_KEYS_PATH: Final[str] = _get_env(
|
||||
SYNC_SSH_KNOWN_HOSTS_PATH: Final[str] = _get_env(
|
||||
"SYNC_SSH_KNOWN_HOSTS_PATH", f"{SYNC_BASE_PATH}/known_hosts"
|
||||
)
|
||||
# PSP save folder files buffered until the folder resolves to a rom.
|
||||
SYNC_RETROARCH_PSP_PENDING_PATH: Final[str] = f"{ROMM_BASE_PATH}/cache/retroarch_sync"
|
||||
# JSON map of PSP serial to extensionless rom file name, for saves whose title
|
||||
# matches no rom, e.g. {"ULUS10336": "Crisis Core - Final Fantasy VII (USA)"}.
|
||||
SYNC_RETROARCH_PSP_SERIAL_MAP: Final[dict[str, str]] = json.loads(
|
||||
|
||||
@@ -27,6 +27,7 @@ from models.base import (
|
||||
from utils.filesystem import (
|
||||
LINK_FALLBACK_ERRNOS,
|
||||
SERVED_FILE_MODE,
|
||||
TEMP_FILE_PREFIX,
|
||||
check_filename_length,
|
||||
iter_directories,
|
||||
iter_files,
|
||||
@@ -375,7 +376,6 @@ class FSHandler:
|
||||
self.base_path = Path(base_path).resolve()
|
||||
self._locks: dict[str, asyncio.Lock] = {}
|
||||
self._lock_mutex = asyncio.Lock()
|
||||
self.base_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
async def _get_file_lock(self, file_path: str) -> asyncio.Lock:
|
||||
"""Get or create a lock for a specific file path."""
|
||||
@@ -497,7 +497,7 @@ class FSHandler:
|
||||
final os.replace() occurs on the same filesystem.
|
||||
"""
|
||||
fd, temp_path_str = tempfile.mkstemp(
|
||||
dir=str(target_path.parent), prefix=".romm_tmp_"
|
||||
dir=str(target_path.parent), prefix=TEMP_FILE_PREFIX
|
||||
)
|
||||
temp_path = Path(temp_path_str)
|
||||
os.close(fd)
|
||||
|
||||
@@ -11,9 +11,5 @@ class FSLaunchboxHandler(FSHandler):
|
||||
|
||||
@functools.cache
|
||||
def get_fs_launchbox_handler() -> FSLaunchboxHandler:
|
||||
"""Lazily instantiate the LaunchBox handler on first use.
|
||||
|
||||
Deferred so that startup doesn't fail when the LaunchBox feature is
|
||||
unconfigured or its base path is not writable.
|
||||
"""
|
||||
"""Lazily instantiate the LaunchBox handler on first use."""
|
||||
return FSLaunchboxHandler()
|
||||
|
||||
@@ -7,7 +7,7 @@ from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from config import ASSETS_BASE_PATH
|
||||
from utils.filesystem import iter_files
|
||||
from utils.filesystem import TEMP_FILE_PREFIX, iter_files
|
||||
|
||||
from .base_handler import FSHandler
|
||||
|
||||
@@ -25,6 +25,9 @@ def _walk_files(root: Path) -> list[BlobFile]:
|
||||
"""Every file under `root`, without descending into symlinked directories."""
|
||||
files: list[BlobFile] = []
|
||||
for directory, name in iter_files(str(root), recursive=True):
|
||||
# A write interrupted by a crash leaves its temp file behind.
|
||||
if name.startswith(TEMP_FILE_PREFIX):
|
||||
continue
|
||||
path = directory / name
|
||||
try:
|
||||
file_stat = path.stat()
|
||||
|
||||
@@ -112,9 +112,5 @@ class FSSyncHandler(FSHandler):
|
||||
|
||||
@functools.cache
|
||||
def get_fs_sync_handler() -> FSSyncHandler:
|
||||
"""Lazily instantiate the sync folder handler on first use.
|
||||
|
||||
Deferred so that startup doesn't fail when sync is unconfigured or its
|
||||
base path is not writable.
|
||||
"""
|
||||
"""Lazily instantiate the sync folder handler on first use."""
|
||||
return FSSyncHandler()
|
||||
|
||||
@@ -11,11 +11,9 @@ from logger.formatter import BLUE
|
||||
from logger.formatter import highlight as hl
|
||||
from logger.logger import log
|
||||
from models.rom import Rom
|
||||
from utils.filesystem import TEMP_FILE_PREFIX
|
||||
from utils.m3u import listing_playlist
|
||||
|
||||
# Scans and the watcher skip dot-prefixed entries, so staged files stay unseen.
|
||||
STAGE_PREFIX = ".romm_tmp_"
|
||||
|
||||
# Parallel uploads each promote the ROM, maybe on different gunicorn workers.
|
||||
# Keyed by the target folder, since lone files sharing a stem promote into one.
|
||||
PROMOTION_LOCK_TIMEOUT_SECONDS = 600
|
||||
@@ -60,7 +58,7 @@ async def _promote(rom: Rom) -> Rom:
|
||||
origin = f"{fs_path}/{fs_name}"
|
||||
dest_dir = f"{fs_path}/{folder}"
|
||||
final = f"{dest_dir}/{fs_name}"
|
||||
staged = f"{fs_path}/{STAGE_PREFIX}{fs_name}"
|
||||
staged = f"{fs_path}/{TEMP_FILE_PREFIX}{fs_name}"
|
||||
extensionless = folder == fs_name
|
||||
|
||||
# Extensionless dest_dir is the file's own path; only a directory collides.
|
||||
|
||||
@@ -22,10 +22,9 @@ from typing import Literal
|
||||
|
||||
from redis.exceptions import RedisError
|
||||
|
||||
from config import SYNC_RETROARCH_PSP_PENDING_PATH, SYNC_RETROARCH_PSP_SERIAL_MAP
|
||||
from config import SYNC_RETROARCH_PSP_SERIAL_MAP
|
||||
from handler.database import db_platform_handler, db_rom_handler, db_save_handler
|
||||
from handler.filesystem import fs_asset_handler
|
||||
from handler.filesystem.base_handler import FSHandler
|
||||
from handler.filesystem import fs_asset_handler, fs_retroarch_sync_handler
|
||||
from handler.redis_handler import async_cache
|
||||
from handler.sync.retroarch import sync_handler
|
||||
from logger.formatter import highlight as hl
|
||||
@@ -38,6 +37,8 @@ from utils.zip_cache import ensure_zipfile_writable
|
||||
|
||||
_IGNORED_CATEGORY = "SYSTEM"
|
||||
_SAVEDATA_CATEGORY = "SAVEDATA"
|
||||
# Outside BLOB_CATEGORIES, so clients can never read or write it.
|
||||
_PENDING_CATEGORY = "psp_pending"
|
||||
|
||||
# Real PSP save folders hold a handful of small files; anything past these is
|
||||
# not one, and inflating it on every manifest build would exhaust memory.
|
||||
@@ -46,8 +47,6 @@ BUNDLE_MAX_UNCOMPRESSED_BYTES = 64 * 1024 * 1024
|
||||
|
||||
_BUNDLE_FOLDER_PATTERN = re.compile(r"^PSP-(.+?)(?: \[.*])?\.zip$")
|
||||
|
||||
fs_psp_pending_handler = FSHandler(base_path=SYNC_RETROARCH_PSP_PENDING_PATH)
|
||||
|
||||
|
||||
class PspFolderUnresolved(Exception):
|
||||
"""A save folder no ROM matches yet; its files wait on disk until one does."""
|
||||
@@ -331,6 +330,27 @@ async def _load_bundle(
|
||||
_folder_locks: defaultdict[str, asyncio.Lock] = defaultdict(asyncio.Lock)
|
||||
|
||||
|
||||
def _pending_dir(user: User, save_folder: str) -> str:
|
||||
return sync_handler.user_blob_path(user, f"{_PENDING_CATEGORY}/{save_folder}")
|
||||
|
||||
|
||||
async def _list_pending(pending_dir: str) -> dict[str, int]:
|
||||
"""Each buffered file's size, by name."""
|
||||
return {
|
||||
blob.relative_path: blob.size
|
||||
for blob in await fs_retroarch_sync_handler.list_blob_files(pending_dir)
|
||||
}
|
||||
|
||||
|
||||
async def _drop_pending(pending_dir: str, names: Iterable[str]) -> None:
|
||||
for name in names:
|
||||
with suppress(FileNotFoundError):
|
||||
await fs_retroarch_sync_handler.remove_file(f"{pending_dir}/{name}")
|
||||
# rmdir keeps a folder another worker buffered a file into meanwhile.
|
||||
with suppress(OSError):
|
||||
fs_retroarch_sync_handler.validate_path(pending_dir).rmdir()
|
||||
|
||||
|
||||
async def _resolve_folder_rom(
|
||||
info: PspFilePath,
|
||||
content: bytes,
|
||||
@@ -355,7 +375,12 @@ async def _resolve_folder_rom(
|
||||
if rom:
|
||||
return rom
|
||||
|
||||
await fs_psp_pending_handler.write_file(
|
||||
pending_sizes = await _list_pending(pending_dir)
|
||||
pending_sizes[info.file_name] = len(content)
|
||||
if _exceeds_bundle_limits(pending_sizes.values()):
|
||||
raise PspBundleInvalid(info.save_folder)
|
||||
|
||||
await fs_retroarch_sync_handler.write_file(
|
||||
file=content, path=pending_dir, filename=info.file_name
|
||||
)
|
||||
log.warning(
|
||||
@@ -408,7 +433,7 @@ async def put_psp_file(
|
||||
# PPSSPP writes a folder as a burst of PUTs, so the bundle is rewritten in
|
||||
# place rather than keeping each partial merge as save history.
|
||||
async with _folder_locks[f"{user.id}:{info.save_folder}"]:
|
||||
pending_dir = f"{user.id}/{info.save_folder}"
|
||||
pending_dir = _pending_dir(user, info.save_folder)
|
||||
existing = await asyncio.to_thread(
|
||||
_find_bundle_by_folder, user, info.save_folder, can_see
|
||||
)
|
||||
@@ -424,12 +449,9 @@ async def put_psp_file(
|
||||
else:
|
||||
rom = await _resolve_folder_rom(info, content, pending_dir, can_see)
|
||||
|
||||
try:
|
||||
pending_names = await fs_psp_pending_handler.list_files(pending_dir)
|
||||
except FileNotFoundError:
|
||||
pending_names = []
|
||||
pending_names = await _list_pending(pending_dir)
|
||||
for name in pending_names:
|
||||
merged[name] = await fs_psp_pending_handler.read_file(
|
||||
merged[name] = await fs_retroarch_sync_handler.read_file(
|
||||
f"{pending_dir}/{name}"
|
||||
)
|
||||
merged[info.file_name] = content
|
||||
@@ -441,9 +463,7 @@ async def put_psp_file(
|
||||
else:
|
||||
await _add_bundle(user, info, rom, merged)
|
||||
|
||||
for name in pending_names:
|
||||
with suppress(FileNotFoundError):
|
||||
await fs_psp_pending_handler.remove_file(f"{pending_dir}/{name}")
|
||||
await _drop_pending(pending_dir, pending_names)
|
||||
|
||||
|
||||
async def _rewrite_bundle(bundle: Save, entries: dict[str, bytes]) -> None:
|
||||
@@ -498,8 +518,10 @@ async def get_psp_file(
|
||||
async def delete_psp_file(
|
||||
user: User, info: PspFilePath, can_see: Callable[[Rom], bool]
|
||||
) -> None:
|
||||
"""Drop one member from its folder's bundle, and the bundle once empty."""
|
||||
"""Drop one member from its folder's bundle and buffer, and either once empty."""
|
||||
async with _folder_locks[f"{user.id}:{info.save_folder}"]:
|
||||
await _drop_pending(_pending_dir(user, info.save_folder), [info.file_name])
|
||||
|
||||
bundle = await asyncio.to_thread(
|
||||
_find_bundle_by_folder, user, info.save_folder, can_see
|
||||
)
|
||||
|
||||
@@ -25,7 +25,6 @@ from adapters.services.rom_converto import (
|
||||
from config.config_manager import config_manager as cm
|
||||
from handler.database import db_platform_handler, db_rom_handler
|
||||
from handler.filesystem import fs_rom_handler
|
||||
from handler.rom_conversion import STAGE_PREFIX
|
||||
from handler.rom_files import refresh_rom_files
|
||||
from logger.formatter import highlight as hl
|
||||
from logger.logger import log
|
||||
@@ -34,10 +33,10 @@ from tasks.registry import CONVERT_LIBRARY_SPEC
|
||||
from tasks.scheduled.convert_images_to_webp import ConversionStats
|
||||
from tasks.tasks import Task
|
||||
from utils.context import initialize_context
|
||||
from utils.filesystem import LINK_FALLBACK_ERRNOS
|
||||
from utils.filesystem import LINK_FALLBACK_ERRNOS, TEMP_FILE_PREFIX
|
||||
|
||||
_PLAYLIST_EXT: Final = ".m3u"
|
||||
_CONVERT_STAGE_PREFIX: Final = f"{STAGE_PREFIX}convert_"
|
||||
_CONVERT_STAGE_PREFIX: Final = f"{TEMP_FILE_PREFIX}convert_"
|
||||
|
||||
|
||||
@dataclass
|
||||
|
||||
@@ -33,14 +33,13 @@ def conversion(tmp_path: Path, mocker):
|
||||
"utils.conversion_cache.rom_converto_service.is_enabled",
|
||||
AsyncMock(return_value=True),
|
||||
)
|
||||
final = tmp_path / "cache/converts/1-abc/test_rom.chd"
|
||||
final = tmp_path / "converts/1-abc/test_rom.chd"
|
||||
final.parent.mkdir(parents=True)
|
||||
resolved = resolve_operation("psp", "chd", "test_rom.iso")
|
||||
assert resolved is not None
|
||||
lookup = mocker.patch.object(
|
||||
conversion_cache, "_lookup", return_value=(resolved[0], final)
|
||||
)
|
||||
mocker.patch.object(conversion_cache, "ROMM_BASE_PATH", str(tmp_path))
|
||||
mocker.patch.object(
|
||||
conversion_cache, "ROM_CONVERTO_CACHE_PATH", str(final.parent.parent)
|
||||
)
|
||||
|
||||
@@ -71,14 +71,10 @@ def _retroarch_upload_cap(client: TestClient, max_size: int):
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolated_sync_dirs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
|
||||
"""Keep blobs and pending PSP files per test, since user ids repeat across test databases."""
|
||||
for handler, name in (
|
||||
(fs_retroarch_sync_handler, "retroarch_sync"),
|
||||
(psp.fs_psp_pending_handler, "psp_pending"),
|
||||
):
|
||||
base = (tmp_path / name).resolve()
|
||||
base.mkdir()
|
||||
monkeypatch.setattr(handler, "base_path", base)
|
||||
"""Keep sync files per test, since user ids repeat across test databases."""
|
||||
base = (tmp_path / "retroarch_sync").resolve()
|
||||
base.mkdir()
|
||||
monkeypatch.setattr(fs_retroarch_sync_handler, "base_path", base)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -1289,6 +1285,15 @@ class TestRetroArchSyncDelete:
|
||||
assert response.status_code == status.HTTP_404_NOT_FOUND
|
||||
|
||||
|
||||
UNKNOWN_FOLDER_URL = "/api/sync/retroarch/saves/PPSSPP/PSP/SAVEDATA/UNKNOWN99999DATA0"
|
||||
|
||||
|
||||
def _unknown_folder_pending_dir(user: User) -> Path:
|
||||
return fs_retroarch_sync_handler.validate_path(
|
||||
psp._pending_dir(user, "UNKNOWN99999DATA0")
|
||||
)
|
||||
|
||||
|
||||
class TestRetroArchSyncPsp:
|
||||
"""PSP save-folder bundling, with roms resolved via the serial map rather than search."""
|
||||
|
||||
@@ -1553,18 +1558,98 @@ class TestRetroArchSyncPsp:
|
||||
"saves/PSP/SAVEDATA/TEST12345DATA0/SAVE.BIN"
|
||||
]
|
||||
|
||||
def test_unresolved_folder_is_buffered_and_conflicts(
|
||||
self, client, admin_user: User, monkeypatch: pytest.MonkeyPatch
|
||||
def test_unresolved_folder_is_buffered_until_it_resolves(
|
||||
self, client, admin_user: User, rom: Rom, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
monkeypatch.setattr(psp, "SYNC_RETROARCH_PSP_SERIAL_MAP", {})
|
||||
|
||||
response = client.put(
|
||||
"/api/sync/retroarch/saves/PPSSPP/PSP/SAVEDATA/UNKNOWN99999DATA0/SAVE.BIN",
|
||||
f"{UNKNOWN_FOLDER_URL}/SAVE.BIN",
|
||||
content=b"orphaned save data",
|
||||
auth=ADMIN_AUTH,
|
||||
)
|
||||
|
||||
assert response.status_code == status.HTTP_409_CONFLICT
|
||||
pending_file = _unknown_folder_pending_dir(admin_user) / "SAVE.BIN"
|
||||
assert pending_file.read_bytes() == b"orphaned save data"
|
||||
|
||||
response = client.get(
|
||||
"/api/sync/retroarch/psp_pending/UNKNOWN99999DATA0/SAVE.BIN",
|
||||
auth=ADMIN_AUTH,
|
||||
)
|
||||
assert response.status_code == status.HTTP_404_NOT_FOUND
|
||||
|
||||
monkeypatch.setattr(
|
||||
psp, "SYNC_RETROARCH_PSP_SERIAL_MAP", {"UNKNOWN99999": rom.fs_name_no_ext}
|
||||
)
|
||||
response = client.put(
|
||||
f"{UNKNOWN_FOLDER_URL}/PARAM.SFO",
|
||||
content=b"sfo",
|
||||
auth=ADMIN_AUTH,
|
||||
)
|
||||
assert response.status_code == status.HTTP_201_CREATED
|
||||
assert not pending_file.parent.exists()
|
||||
|
||||
response = client.get(
|
||||
f"{UNKNOWN_FOLDER_URL}/SAVE.BIN",
|
||||
auth=ADMIN_AUTH,
|
||||
)
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
assert response.content == b"orphaned save data"
|
||||
|
||||
def test_buffer_past_the_bundle_limits_conflicts(
|
||||
self, client, admin_user: User, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
monkeypatch.setattr(psp, "SYNC_RETROARCH_PSP_SERIAL_MAP", {})
|
||||
monkeypatch.setattr(psp, "_BUNDLE_MAX_MEMBERS", 1)
|
||||
client.put(
|
||||
f"{UNKNOWN_FOLDER_URL}/SAVE.BIN",
|
||||
content=b"data",
|
||||
auth=ADMIN_AUTH,
|
||||
)
|
||||
|
||||
response = client.put(
|
||||
f"{UNKNOWN_FOLDER_URL}/ICON0.PNG",
|
||||
content=b"icon",
|
||||
auth=ADMIN_AUTH,
|
||||
)
|
||||
|
||||
assert response.status_code == status.HTTP_409_CONFLICT
|
||||
pending_dir = _unknown_folder_pending_dir(admin_user)
|
||||
assert [path.name for path in pending_dir.iterdir()] == ["SAVE.BIN"]
|
||||
|
||||
def test_delete_drops_the_buffered_copy(
|
||||
self, client, admin_user: User, rom: Rom, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
monkeypatch.setattr(psp, "SYNC_RETROARCH_PSP_SERIAL_MAP", {})
|
||||
client.put(
|
||||
f"{UNKNOWN_FOLDER_URL}/SAVE.BIN",
|
||||
content=b"deleted save data",
|
||||
auth=ADMIN_AUTH,
|
||||
)
|
||||
|
||||
response = client.delete(
|
||||
f"{UNKNOWN_FOLDER_URL}/SAVE.BIN",
|
||||
auth=ADMIN_AUTH,
|
||||
)
|
||||
|
||||
assert response.status_code == status.HTTP_204_NO_CONTENT
|
||||
pending_dir = _unknown_folder_pending_dir(admin_user)
|
||||
assert not pending_dir.exists()
|
||||
|
||||
monkeypatch.setattr(
|
||||
psp, "SYNC_RETROARCH_PSP_SERIAL_MAP", {"UNKNOWN99999": rom.fs_name_no_ext}
|
||||
)
|
||||
client.put(
|
||||
f"{UNKNOWN_FOLDER_URL}/PARAM.SFO",
|
||||
content=b"sfo",
|
||||
auth=ADMIN_AUTH,
|
||||
)
|
||||
response = client.get(
|
||||
f"{UNKNOWN_FOLDER_URL}/SAVE.BIN",
|
||||
auth=ADMIN_AUTH,
|
||||
)
|
||||
assert response.status_code == status.HTTP_404_NOT_FOUND
|
||||
|
||||
def test_delete_keeps_the_rest_of_the_bundle(
|
||||
self, client, admin_user: User, rom: Rom
|
||||
|
||||
@@ -52,15 +52,15 @@ class TestFSHandler:
|
||||
mock_file.file = BytesIO(sample_file_content)
|
||||
return mock_file
|
||||
|
||||
def test_init_creates_base_directory(self, temp_dir):
|
||||
"""Test that FSHandler creates base directory on initialization"""
|
||||
# Remove the directory to test creation
|
||||
async def test_base_directory_is_created_on_first_write(self, temp_dir):
|
||||
shutil.rmtree(temp_dir)
|
||||
|
||||
handler = FSHandler(temp_dir)
|
||||
assert not handler.base_path.exists()
|
||||
|
||||
assert handler.base_path.exists()
|
||||
assert handler.base_path.is_dir()
|
||||
await handler.write_file(b"data", path="", filename="test.txt")
|
||||
|
||||
assert (handler.base_path / "test.txt").read_bytes() == b"data"
|
||||
|
||||
def test_init_resolves_path(self, temp_dir):
|
||||
"""Test that FSHandler resolves the base path"""
|
||||
@@ -770,15 +770,14 @@ class TestFSHandler:
|
||||
|
||||
|
||||
class TestFSHandlerInit:
|
||||
def test_raises_on_mkdir_failure(self):
|
||||
"""OSError from mkdir propagates, so misconfigured paths fail loudly
|
||||
at construction time. Optional features should defer construction
|
||||
(e.g. via a lazy factory) rather than swallow the error."""
|
||||
def test_does_not_touch_an_unwritable_base_path(self):
|
||||
"""Handlers are built at import, so an unmounted base path can't block boot."""
|
||||
with patch.object(
|
||||
Path, "mkdir", side_effect=PermissionError(errno.EACCES, "denied")
|
||||
):
|
||||
with pytest.raises(PermissionError):
|
||||
FSHandler("/some/unwritable/path")
|
||||
) as mkdir:
|
||||
FSHandler("/some/unwritable/path")
|
||||
|
||||
mkdir.assert_not_called()
|
||||
|
||||
|
||||
class TestRegionRanksForPriority:
|
||||
|
||||
@@ -1,15 +1,13 @@
|
||||
"""Tests for filesystem sync handler."""
|
||||
|
||||
import errno
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from handler.filesystem.sync_handler import FSSyncHandler, get_fs_sync_handler
|
||||
from handler.filesystem.sync_handler import FSSyncHandler
|
||||
|
||||
|
||||
class TestFSSyncHandler:
|
||||
@@ -136,22 +134,3 @@ class TestFSSyncHandler:
|
||||
def test_remove_incoming_file_nonexistent(self, handler: FSSyncHandler):
|
||||
# Should not raise for nonexistent files
|
||||
handler.remove_incoming_file("/nonexistent/path/file.sav")
|
||||
|
||||
|
||||
class TestFSSyncHandlerLazyFactory:
|
||||
"""Sync is an optional feature: if /romm/sync isn't writable (bad mount,
|
||||
wrong ownership), the app must still boot. The factory defers construction
|
||||
until first use so the error surfaces at the call site, not at startup."""
|
||||
|
||||
def test_factory_raises_at_call_time_when_unwritable(self):
|
||||
"""Regression for the PermissionError on /romm/sync at boot: the
|
||||
failure must surface from the factory call, not from module import."""
|
||||
get_fs_sync_handler.cache_clear()
|
||||
try:
|
||||
with patch.object(
|
||||
Path, "mkdir", side_effect=PermissionError(errno.EACCES, "denied")
|
||||
):
|
||||
with pytest.raises(PermissionError):
|
||||
get_fs_sync_handler()
|
||||
finally:
|
||||
get_fs_sync_handler.cache_clear()
|
||||
|
||||
@@ -22,6 +22,7 @@ from handler.sync.retroarch.sync_handler import (
|
||||
)
|
||||
from models.assets import Save, State
|
||||
from models.user import User
|
||||
from utils.filesystem import TEMP_FILE_PREFIX
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
@@ -158,6 +159,14 @@ class TestListBlobFiles:
|
||||
|
||||
assert await blob_handler.list_blob_files("config") == []
|
||||
|
||||
async def test_skips_interrupted_write_temp_files(
|
||||
self, blob_handler: FSRetroArchSyncHandler, tmp_path: Path
|
||||
):
|
||||
(tmp_path / "config").mkdir()
|
||||
(tmp_path / "config" / f"{TEMP_FILE_PREFIX}ab12").write_bytes(b"partial")
|
||||
|
||||
assert await blob_handler.list_blob_files("config") == []
|
||||
|
||||
async def test_missing_prefix_is_empty(self, blob_handler: FSRetroArchSyncHandler):
|
||||
assert await blob_handler.list_blob_files("config") == []
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ from adapters.services.rom_converto import (
|
||||
RomConvertoTimeoutError,
|
||||
resolve_operation,
|
||||
)
|
||||
from config import ROMM_BASE_PATH
|
||||
from config import ROM_CONVERTO_CACHE_PATH
|
||||
from config.config_manager import ConvertoConfig
|
||||
from models.rom import Rom, RomFile
|
||||
from utils import conversion_cache
|
||||
@@ -689,8 +689,8 @@ class TestHasRoomFor:
|
||||
|
||||
|
||||
class TestGetRedirectPath:
|
||||
def test_relative_to_romm_base_path(self):
|
||||
converted_path = Path(ROMM_BASE_PATH) / "cache/converts/1-abc/Game.chd"
|
||||
def test_relative_to_converts_cache_path(self):
|
||||
converted_path = Path(ROM_CONVERTO_CACHE_PATH) / "1-abc/Game.chd"
|
||||
assert get_redirect_path(converted_path) == Path(
|
||||
"/cache/converts/1-abc/Game.chd"
|
||||
)
|
||||
|
||||
@@ -30,7 +30,6 @@ from config import (
|
||||
ROM_CONVERTO_CACHE_PATH,
|
||||
ROM_CONVERTO_ENABLED,
|
||||
ROM_CONVERTO_MAX_CONCURRENCY,
|
||||
ROMM_BASE_PATH,
|
||||
)
|
||||
from config.config_manager import config_manager as cm
|
||||
from logger.formatter import highlight as hl
|
||||
@@ -75,8 +74,8 @@ def converted_file_path(
|
||||
|
||||
|
||||
def get_redirect_path(converted_path: Path) -> Path:
|
||||
"""The nginx-internal path for a converted file (`/cache/` aliases `${ROMM_BASE_PATH}/cache/`)."""
|
||||
return Path("/") / converted_path.relative_to(ROMM_BASE_PATH)
|
||||
"""The nginx-internal path for a converted file (`/cache/` aliases CACHE_BASE_PATH)."""
|
||||
return Path("/cache/converts") / converted_path.relative_to(ROM_CONVERTO_CACHE_PATH)
|
||||
|
||||
|
||||
def _dir_size(path: Path) -> int:
|
||||
|
||||
@@ -20,6 +20,10 @@ COMPRESSED_FILE_SUFFIXES: tuple[str, ...] = tuple(COMPRESSED_FILE_EXTENSIONS)
|
||||
# tempfile.mkstemp creates files 0600, too narrow for the nginx user to read.
|
||||
SERVED_FILE_MODE = 0o644
|
||||
|
||||
# Prefix of in-progress writes and staged conversions, renamed into place once
|
||||
# complete. Scans skip dot-prefixed names, so these stay unseen.
|
||||
TEMP_FILE_PREFIX = ".romm_tmp_"
|
||||
|
||||
|
||||
def iter_files(path: str, recursive: bool = False) -> Iterator[tuple[Path, str]]:
|
||||
"""List files in a directory.
|
||||
|
||||
+2
-1
@@ -25,6 +25,7 @@ from handler.scan_jobs import get_pending_scan_jobs
|
||||
from logger.formatter import CYAN
|
||||
from logger.formatter import highlight as hl
|
||||
from logger.logger import log
|
||||
from utils.filesystem import TEMP_FILE_PREFIX
|
||||
from utils.sentry import init_sentry
|
||||
|
||||
init_sentry()
|
||||
@@ -120,7 +121,7 @@ def process_changes(changes: Sequence[Change]) -> None:
|
||||
def _is_excluded(path: str) -> bool:
|
||||
parts = path.strip("/").split("/")
|
||||
for part in parts:
|
||||
if part.startswith(".romm_tmp_"):
|
||||
if part.startswith(TEMP_FILE_PREFIX):
|
||||
return True
|
||||
if any(
|
||||
part == pat or fnmatch.fnmatch(part, pat) for pat in excluded_patterns
|
||||
|
||||
@@ -38,6 +38,8 @@ done
|
||||
# support the default value syntax `${VAR:-default}`.
|
||||
export ROMM_BASE_PATH=${ROMM_BASE_PATH:-/romm}
|
||||
export ROMM_PORT=${ROMM_PORT:-8080}
|
||||
# Matches CACHE_BASE_PATH in the backend config, for nginx's /cache/ alias.
|
||||
export ROMM_CACHE_BASE_PATH="${ROMM_TMP_PATH:-/tmp/romm}/cache"
|
||||
# Keep the nginx upstream port in sync with the port gunicorn binds to.
|
||||
export DEV_PORT=${DEV_PORT:-5000}
|
||||
|
||||
|
||||
@@ -14,6 +14,10 @@ ENABLE_SCHEDULED_RESCAN="${ENABLE_SCHEDULED_RESCAN:="false"}"
|
||||
ENABLE_SCHEDULED_UPDATE_LAUNCHBOX_METADATA="${ENABLE_SCHEDULED_UPDATE_LAUNCHBOX_METADATA:="false"}"
|
||||
ENABLE_SCHEDULED_UPDATE_SWITCH_TITLEDB="${ENABLE_SCHEDULED_UPDATE_SWITCH_TITLEDB:="false"}"
|
||||
ENABLE_SYNC_FOLDER_WATCHER="${ENABLE_SYNC_FOLDER_WATCHER:="false"}"
|
||||
ENABLE_SYNC_PUSH_PULL="${ENABLE_SYNC_PUSH_PULL:="false"}"
|
||||
sync_base_path="${ROMM_BASE_PATH:-/romm}/sync"
|
||||
# The values the backend's safe_str_to_bool reads as true, matched lowercased.
|
||||
TRUTHY_PATTERN='^(1|true|yes|on)$'
|
||||
|
||||
# if REDIS_HOST is set, we assume that an external redis is used
|
||||
REDIS_HOST="${REDIS_HOST:=""}"
|
||||
@@ -138,6 +142,7 @@ start_bin_gunicorn() {
|
||||
"${wrap[@]}" gunicorn \
|
||||
--bind=0.0.0.0:"${DEV_PORT:-5000}" \
|
||||
--pid=/tmp/gunicorn.pid \
|
||||
--no-control-socket \
|
||||
--forwarded-allow-ips="${FORWARDED_ALLOW_IPS:-127.0.0.1,::1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,100.64.0.0/10,fc00::/7}" \
|
||||
--worker-class uvicorn_worker.UvicornWorker \
|
||||
--workers "${WEB_SERVER_CONCURRENCY:-4}" \
|
||||
@@ -227,12 +232,11 @@ start_bin_valkey-server() {
|
||||
# --url, which would put it on a world-readable command line. REDIS_SSL is a
|
||||
# boolean to the app, so "false" and "0" mean plaintext.
|
||||
build_redis_url() {
|
||||
local scheme userinfo="" ssl="${REDIS_SSL-}"
|
||||
local scheme="redis" userinfo="" ssl="${REDIS_SSL-}"
|
||||
|
||||
case "${ssl,,}" in
|
||||
1 | true | yes | on) scheme="rediss" ;;
|
||||
*) scheme="redis" ;;
|
||||
esac
|
||||
if [[ ${ssl,,} =~ ${TRUTHY_PATTERN} ]]; then
|
||||
scheme="rediss"
|
||||
fi
|
||||
|
||||
if [[ -n ${REDIS_PASSWORD-} ]]; then
|
||||
userinfo="${REDIS_USERNAME-}:${REDIS_PASSWORD}@"
|
||||
@@ -322,8 +326,6 @@ start_bin_watcher() {
|
||||
|
||||
start_bin_sync_watcher() {
|
||||
info_log "Starting sync folder watcher"
|
||||
sync_base_path="${ROMM_BASE_PATH:-/romm}/sync"
|
||||
mkdir -p "${sync_base_path}"
|
||||
local -a wrap=()
|
||||
otel_prefix wrap sync_watcher
|
||||
watchfiles \
|
||||
@@ -418,6 +420,21 @@ fi
|
||||
# Startup process requires database and cache to be already available
|
||||
run_startup
|
||||
|
||||
# The image's /romm is only writable by UID 1000, so the sync folder needs its own mount.
|
||||
if [[ ${ENABLE_SYNC_FOLDER_WATCHER,,} =~ ${TRUTHY_PATTERN} ]]; then
|
||||
if ! mkdir -p "${sync_base_path}" 2>/dev/null || [[ ! -w ${sync_base_path} ]]; then
|
||||
warn_log "${sync_base_path} is not writable, so the sync folder watcher and sync folder scan are disabled. Mount a host folder there to use them."
|
||||
ENABLE_SYNC_FOLDER_WATCHER="false"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Push-pull only reads its SSH keys, so a read-only mount is enough.
|
||||
sync_ssh_keys_path="${SYNC_SSH_KEYS_PATH:-${sync_base_path}/keys}"
|
||||
if [[ ${ENABLE_SYNC_PUSH_PULL,,} =~ ${TRUTHY_PATTERN} ]] && ! mkdir -p "${sync_ssh_keys_path}" 2>/dev/null; then
|
||||
warn_log "${sync_ssh_keys_path} doesn't exist and can't be created, so push-pull sync is disabled. Mount your SSH keys there, or set SYNC_SSH_KEYS_PATH."
|
||||
ENABLE_SYNC_PUSH_PULL="false"
|
||||
fi
|
||||
|
||||
# main loop
|
||||
while ! ((exited)); do
|
||||
watchdog_process_pid gunicorn
|
||||
@@ -432,12 +449,12 @@ while ! ((exited)); do
|
||||
watchdog_process_pid rq_streaming_worker
|
||||
|
||||
# only start the watcher if enabled
|
||||
if [[ ${ENABLE_RESCAN_ON_FILESYSTEM_CHANGE} == "true" ]]; then
|
||||
if [[ ${ENABLE_RESCAN_ON_FILESYSTEM_CHANGE,,} =~ ${TRUTHY_PATTERN} ]]; then
|
||||
watchdog_process_pid watcher
|
||||
fi
|
||||
|
||||
# only start the sync folder watcher if enabled
|
||||
if [[ ${ENABLE_SYNC_FOLDER_WATCHER} == "true" ]]; then
|
||||
if [[ ${ENABLE_SYNC_FOLDER_WATCHER,,} =~ ${TRUTHY_PATTERN} ]]; then
|
||||
watchdog_process_pid sync_watcher
|
||||
fi
|
||||
|
||||
|
||||
@@ -131,7 +131,7 @@ server {
|
||||
# Internally redirect cached zip file requests (Range-resumable downloads)
|
||||
location /cache/ {
|
||||
internal;
|
||||
alias "${ROMM_BASE_PATH}/cache/";
|
||||
alias "${ROMM_CACHE_BASE_PATH}/";
|
||||
}
|
||||
|
||||
# Internal decoding endpoint, used to decode base64 encoded data
|
||||
|
||||
@@ -914,7 +914,7 @@ A reset link is emailed when SMTP is set up, the user has an address and `ROMM_B
|
||||
| GET | `/{id}/content/{file_name}` | ROMS_READ | Download ROM, in a listed `?format=` when asked |
|
||||
| POST | `/unidentified` | ROMS_READ | Get unidentified ROMs |
|
||||
|
||||
With rom-converto enabled (`ROM_CONVERTO_ENABLED`, `converto.download_conversion_enabled`), `?format=zso,iso` lists the formats a client can read. A single-file download whose stored format is listed is served as-is. Otherwise the first listed format with a copy cached under `/romm/cache/converts` is served, then the first one rom-converto can produce: answered by the same request if it converts within a few seconds, else `202` with `Retry-After` while it finishes. `406` means no listed format can be served, and `HEAD` reports the same without starting a conversion. Only signed-in, non-kiosk users start conversions. A rom's detail response lists the formats its single file can be converted to in `download_formats`, empty when the caller can't start a conversion.
|
||||
With rom-converto enabled (`ROM_CONVERTO_ENABLED`, `converto.download_conversion_enabled`), `?format=zso,iso` lists the formats a client can read. A single-file download whose stored format is listed is served as-is. Otherwise the first listed format with a copy cached under `/tmp/romm/cache/converts` (`$ROMM_TMP_PATH/cache/converts` when set) is served, then the first one rom-converto can produce: answered by the same request if it converts within a few seconds, else `202` with `Retry-After` while it finishes. `406` means no listed format can be served, and `HEAD` reports the same without starting a conversion. Only signed-in, non-kiosk users start conversions. A rom's detail response lists the formats its single file can be converted to in `download_formats`, empty when the caller can't start a conversion.
|
||||
|
||||
#### ROM Upload (Chunked)
|
||||
|
||||
@@ -1031,10 +1031,11 @@ World`), so a name shared across platforms resolves to the lowest visible ROM id
|
||||
- **Manifest:** assets whose file is gone are flagged `missing_from_fs`. Hashes
|
||||
are MD5s of the bytes on disk, Redis-cached by path, size and mtime.
|
||||
- **PSP:** PPSSPP's `PSP/SAVEDATA/<folder>/` files are stored as one zipped
|
||||
`Save` per folder. A folder whose title matches no ROM is buffered under
|
||||
`SYNC_RETROARCH_PSP_PENDING_PATH`, or mapped through `SYNC_RETROARCH_PSP_SERIAL_MAP`.
|
||||
`Save` per folder. A folder whose title matches no ROM is buffered in the
|
||||
user's `retroarch/psp_pending/` folder until it resolves, or mapped through
|
||||
`SYNC_RETROARCH_PSP_SERIAL_MAP`. Clients can't reach `psp_pending/`.
|
||||
- **Blobs:** `config/`, `thumbnails/` and `system/` belong to no ROM, so they
|
||||
are stored per user under `SYNC_RETROARCH_BASE_PATH` (`FSRetroArchSyncHandler`).
|
||||
are stored per user under `assets/users/<user>/retroarch/` (`FSRetroArchSyncHandler`).
|
||||
- **Auth:** the router gates itself so it can answer a 401 Basic challenge,
|
||||
and sends body-less errors, which RetroArch's client needs. The kiosk guest
|
||||
may browse `roms/` but is challenged everywhere else. Uploads are capped at
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
# Core Application
|
||||
ROMM_BASE_PATH=/romm # Base folder path for library, resources and assets
|
||||
ROMM_TMP_PATH= # Custom temporary directory path
|
||||
ROMM_TMP_PATH= # Custom temporary directory path (upload staging and download caches)
|
||||
ROMM_BASE_URL=http://0.0.0.0 # Public URL of this instance
|
||||
ROMM_PORT=8080 # Port on which the application listens
|
||||
KIOSK_MODE=false # Let visitors browse without logging in (read-only)
|
||||
|
||||
@@ -27,6 +27,7 @@ services:
|
||||
- /path/to/library:/romm/library # Your game library. Check https://docs.romm.app/latest/Getting-Started/Folder-Structure/ for more details.
|
||||
- /path/to/assets:/romm/assets # Uploaded saves, states, etc.
|
||||
- /path/to/config:/romm/config # (Optional) Path where config.yml is stored
|
||||
# - /path/to/sync:/romm/sync # Required for ENABLE_SYNC_FOLDER_WATCHER, and holds the SSH keys for ENABLE_SYNC_PUSH_PULL
|
||||
ports:
|
||||
- 80:8080
|
||||
depends_on:
|
||||
|
||||
Reference in New Issue
Block a user