mirror of
https://github.com/rommapp/romm.git
synced 2026-10-12 02:21:58 +00:00
* fix(docker): boot when the container runs as a non-default UID Unraid runs the container as 99:100, which can't write to the image's /romm (owned by 1000, mode 755), only to the mounted subfolders. The PSP pending dir at /romm/cache/retroarch_sync was created at import time, so every gunicorn worker failed to boot. - Buffer pending PSP save files under the user's assets folder (assets/users/<user>/retroarch/psp_pending/), beside the other RetroArch Cloud Sync files and outside the client-reachable blob categories. - Move the zip and rom-converto download caches to /tmp/romm/cache, and point nginx's internal /cache/ alias there. - Put gunicorn's control socket in /tmp/gunicorn instead of $HOME/.gunicorn, since HOME is / for that UID. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(docker): honor ROMM_TMP_PATH for caches, explain an unwritable sync folder - The download caches live under $ROMM_TMP_PATH/cache when it's set, so installs with a small Docker disk (Unraid's docker.img) can point them at a share. The entrypoint exports the same path for nginx's /cache/ alias. - With ENABLE_SYNC_FOLDER_WATCHER on and /romm/sync not writable, /init warns and skips the watcher instead of dying on mkdir under errexit. - The compose example lists the /romm/sync mount the sync features need. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * test(sync): cover reading back and clearing buffered PSP files Addresses Greptile's review on #5189. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(sync): apply code-review findings to PSP pending buffer Remove the emptied psp_pending folder after a merge, define the sync folder path once in init, update the architecture doc's PSP and blob locations, and drop a stale .gitignore entry. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(sync): apply simplify findings to PSP pending cleanup Clear the buffered folder with remove_directory, let the startup check own creating the sync folder, and reword the test fixture docstring. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(sync): cap buffered PSP files and drop them on delete Buffered files for an unresolved save folder now count against the bundle limits, so a folder can't fill the assets share and then fail every merge once it resolves. A DELETE also removes the buffered copy, so a deleted file isn't merged back when the folder resolves. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(docker): check the sync folder for push-pull sync too ENABLE_SYNC_PUSH_PULL keeps SSH keys and device saves under /romm/sync as well, so an unwritable folder now disables it with the watcher and warns at boot instead of failing when the task runs. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(docker): strip whitespace from env vars like the backend does The backend strips every variable it reads, while the init script and nginx templates used raw values, so a padded ROMM_TMP_PATH sent nginx's /cache/ alias somewhere the backend never writes. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * chore(config): drop the CACHE_BASE_PATH comment Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(docker): say which backend config strips env whitespace Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * revert(docker): drop the entrypoint env whitespace trimming Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(sync): apply pr-ready code-review findings Gate push-pull on its SSH keys path rather than a writable sync folder, accept the same truthy values as the backend, and clear only the merged PSP pending files so another worker's buffered file survives. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor: apply pr-ready simplify findings Share one truthy pattern across init's flag checks (the watcher gates now accept the same values as the backend), disable gunicorn's unused control socket instead of placing it, derive the converts redirect from its own cache path, and factor the unknown PSP folder in tests. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(docker): apply pr-ready humanizer pass to the sync mount note Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(fs): create handler base folders on first write, skip leftover temp files FSHandler no longer mkdirs its base path when constructed, so a module-level handler rooted at an unmounted or unwritable folder can't take down boot; every write path already creates its parent folders. Blob listings skip .romm_tmp_ files a crashed write left behind, so they are never merged into a PSP bundle or advertised in the sync manifest. The prefix now lives in one TEMP_FILE_PREFIX constant. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1072 lines
36 KiB
Python
1072 lines
36 KiB
Python
import asyncio
|
|
import fnmatch
|
|
import hashlib
|
|
import os
|
|
import re
|
|
import shutil
|
|
import tempfile
|
|
from collections.abc import AsyncIterator, Callable, Iterable, Sequence
|
|
from contextlib import asynccontextmanager
|
|
from enum import Enum
|
|
from io import BytesIO
|
|
from pathlib import Path
|
|
from tempfile import SpooledTemporaryFile
|
|
from typing import BinaryIO, Final
|
|
|
|
from anyio import AsyncFile
|
|
from anyio import Path as AnyioPath
|
|
from anyio import open_file
|
|
from starlette.datastructures import UploadFile
|
|
|
|
from config.config_manager import config_manager as cm
|
|
from models.base import (
|
|
compute_file_extension,
|
|
compute_file_name_no_ext,
|
|
compute_file_name_no_tags,
|
|
)
|
|
from utils.filesystem import (
|
|
LINK_FALLBACK_ERRNOS,
|
|
SERVED_FILE_MODE,
|
|
TEMP_FILE_PREFIX,
|
|
check_filename_length,
|
|
iter_directories,
|
|
iter_files,
|
|
link_or_copy_file,
|
|
)
|
|
|
|
UUID_V4_REGEX = re.compile(
|
|
r"[0-9a-f]{8}-[0-9a-f]{4}-[0-5][0-9a-f]{3}-[089ab][0-9a-f]{3}-[0-9a-f]{12}",
|
|
re.IGNORECASE,
|
|
)
|
|
|
|
|
|
LANGUAGES = (
|
|
("Ar", "Arabic"),
|
|
("Da", "Danish"),
|
|
("De", "German"),
|
|
("El", "Greek"),
|
|
("En", "English"),
|
|
("Es", "Spanish"),
|
|
("Fi", "Finnish"),
|
|
("Fr", "French"),
|
|
("It", "Italian"),
|
|
("Ja", "Japanese"),
|
|
("Ko", "Korean"),
|
|
("Nl", "Dutch"),
|
|
("No", "Norwegian"),
|
|
("Pl", "Polish"),
|
|
("Pt", "Portuguese"),
|
|
("Ru", "Russian"),
|
|
("Sr", "Serbian"),
|
|
("Sv", "Swedish"),
|
|
("Zh", "Chinese"),
|
|
("nolang", "No Language"),
|
|
)
|
|
|
|
REGIONS = (
|
|
("A", "Australia"),
|
|
("AS", "Asia"),
|
|
("B", "Brazil"),
|
|
("C", "Canada"),
|
|
("CH", "China"),
|
|
("E", "Europe"),
|
|
("F", "France"),
|
|
("FN", "Finland"),
|
|
("G", "Germany"),
|
|
("GR", "Greece"),
|
|
("H", "Holland"),
|
|
("HK", "Hong Kong"),
|
|
("I", "Italy"),
|
|
("J", "Japan"),
|
|
("K", "Korea"),
|
|
("NL", "Netherlands"),
|
|
("NO", "Norway"),
|
|
("PD", "Public Domain"),
|
|
("R", "Russia"),
|
|
("S", "Spain"),
|
|
("SW", "Sweden"),
|
|
("T", "Taiwan"),
|
|
("U", "USA"),
|
|
("UK", "England"),
|
|
("UNK", "Unknown"),
|
|
("UNL", "Unlicensed"),
|
|
("W", "World"),
|
|
)
|
|
|
|
REGIONS_BY_SHORTCODE = {region[0]: region[1] for region in REGIONS}
|
|
|
|
# Every accepted region spelling, lowercased, mapped to its canonical name.
|
|
_REGION_BY_ALIAS = {
|
|
**{name.lower(): name for _, name in REGIONS},
|
|
**{code.lower(): name for code, name in REGIONS},
|
|
}
|
|
|
|
|
|
def normalize_region(tag: str) -> str | None:
|
|
"""Resolve a filename region tag to its canonical REGIONS name.
|
|
|
|
Case-insensitive, so "usa", "USA" and "Usa" collapse to one facet value
|
|
instead of three. Returns None for tags that name no known region.
|
|
"""
|
|
return _REGION_BY_ALIAS.get(tag.strip().lower())
|
|
|
|
|
|
# Maps full REGIONS names to lowercase shortcodes used by metadata providers
|
|
REGION_NAME_TO_PROVIDER_SHORTCODE: dict[str, str] = {
|
|
"Australia": "au",
|
|
"Asia": "asi",
|
|
"Brazil": "br",
|
|
"Canada": "ca",
|
|
"China": "cn",
|
|
"England": "uk",
|
|
"Europe": "eu",
|
|
"Finland": "fi",
|
|
"France": "fr",
|
|
"Germany": "de",
|
|
"Greece": "gr",
|
|
"Holland": "nl",
|
|
"Hong Kong": "hk",
|
|
"Italy": "it",
|
|
"Japan": "jp",
|
|
"Korea": "kr",
|
|
"Netherlands": "nl",
|
|
"Norway": "no",
|
|
"Russia": "ru",
|
|
"Spain": "sp",
|
|
"Sweden": "se",
|
|
"Taiwan": "tw",
|
|
"USA": "us",
|
|
"World": "wor",
|
|
# ScreenScraper regions with no filename shortcode of their own. Providers
|
|
# send these codes bare, so without a name here they land in a facet as
|
|
# "pl" beside the "Poland" a filename or Hasheous produces.
|
|
"Americas": "ame",
|
|
"Bulgaria": "bg",
|
|
"Chile": "cl",
|
|
"Czech Republic": "cz",
|
|
"Denmark": "dk",
|
|
"Hungary": "hu",
|
|
"Israel": "il",
|
|
"Kuwait": "kw",
|
|
"Middle East": "mor",
|
|
"New Zealand": "nz",
|
|
"Oceania": "oce",
|
|
"Peru": "pe",
|
|
"Poland": "pl",
|
|
"Portugal": "pt",
|
|
"Slovakia": "sk",
|
|
"Turkey": "tr",
|
|
}
|
|
|
|
_REGION_NAME_TO_PROVIDER_SHORTCODE_CI = {
|
|
k.lower(): v for k, v in REGION_NAME_TO_PROVIDER_SHORTCODE.items()
|
|
}
|
|
|
|
|
|
def region_name_to_provider_shortcode(region_name: str | None) -> str | None:
|
|
"""Look up a provider shortcode for a region name (case-insensitive).
|
|
|
|
ROM filename parsing can leave region names in their raw casing
|
|
(e.g. "europe", "EUROPE"), so callers must normalize before lookup.
|
|
"""
|
|
if not region_name:
|
|
return None
|
|
return _REGION_NAME_TO_PROVIDER_SHORTCODE_CI.get(region_name.lower())
|
|
|
|
|
|
# Reverse of REGION_NAME_TO_PROVIDER_SHORTCODE. A list per code because two
|
|
# names can claim one code ("nl" for both Holland and Netherlands), and a rom
|
|
# tagged with either has to rank the same.
|
|
_REGION_NAMES_BY_PROVIDER_SHORTCODE: dict[str, list[str]] = {}
|
|
for _name, _code in REGION_NAME_TO_PROVIDER_SHORTCODE.items():
|
|
_REGION_NAMES_BY_PROVIDER_SHORTCODE.setdefault(_code, []).append(_name)
|
|
|
|
|
|
def region_ranks_for_priority(shortcodes: Sequence[str]) -> dict[str, int]:
|
|
"""Map canonical region names to their rank in a shortcode priority list.
|
|
|
|
The rank is the shortcode's position, not the name's, so two names claiming
|
|
one shortcode ("nl" for both Holland and Netherlands) rank equally and fall
|
|
through to whatever tiebreak follows.
|
|
|
|
A shortcode naming no known region (ScreenScraper's "ss", or a typo the
|
|
open-ended settings input allowed) contributes nothing rather than shifting
|
|
the ranks of everything after it. Nor does a repeated shortcode.
|
|
"""
|
|
ranks: dict[str, int] = {}
|
|
next_rank = 0
|
|
for shortcode in dict.fromkeys(code.lower() for code in shortcodes):
|
|
names = _REGION_NAMES_BY_PROVIDER_SHORTCODE.get(shortcode)
|
|
if not names:
|
|
continue
|
|
for name in names:
|
|
ranks[name] = next_rank
|
|
next_rank += 1
|
|
return ranks
|
|
|
|
|
|
# Provider spellings kept out of the filename aliases, where several of these
|
|
# codes ("de", "fr", "nl", "ru") read as a language tag instead (issue #3026).
|
|
_REGION_BY_PROVIDER_ALIAS = {
|
|
**{name.lower(): name for name in REGION_NAME_TO_PROVIDER_SHORTCODE},
|
|
**{code: names[0] for code, names in _REGION_NAMES_BY_PROVIDER_SHORTCODE.items()},
|
|
}
|
|
|
|
|
|
def provider_region_name(value: str) -> str | None:
|
|
"""Resolve a metadata provider's region spelling to its canonical name."""
|
|
return normalize_region(value) or _REGION_BY_PROVIDER_ALIAS.get(
|
|
value.strip().lower()
|
|
)
|
|
|
|
|
|
LANGUAGES_BY_SHORTCODE = {lang[0]: lang[1] for lang in LANGUAGES}
|
|
|
|
# Every accepted language spelling, lowercased, mapped to its canonical name.
|
|
_LANGUAGE_BY_ALIAS = {
|
|
**{name.lower(): name for _, name in LANGUAGES},
|
|
**{code.lower(): name for code, name in LANGUAGES},
|
|
}
|
|
|
|
|
|
def normalize_language(tag: str) -> str | None:
|
|
"""Resolve a filename language tag to its canonical LANGUAGES name.
|
|
|
|
Case-insensitive, so "english", "English" and "ENGLISH" collapse to one
|
|
facet value. Returns None for tags that name no known language.
|
|
"""
|
|
return _LANGUAGE_BY_ALIAS.get(tag.strip().lower())
|
|
|
|
|
|
# Codes only a translation tag uses for its target language ("[T+Eng]",
|
|
# "[T-Ge]"). Two-letter ISO codes are absent: normalize_language has those.
|
|
_TRANSLATION_LANGUAGE_ALIASES = {
|
|
"ara": "Arabic",
|
|
"chi": "Chinese",
|
|
"dan": "Danish",
|
|
"du": "Dutch",
|
|
"dut": "Dutch",
|
|
"eng": "English",
|
|
"fin": "Finnish",
|
|
"fre": "French",
|
|
"ge": "German",
|
|
"ger": "German",
|
|
"gr": "Greek",
|
|
"gre": "Greek",
|
|
"ita": "Italian",
|
|
"jap": "Japanese",
|
|
"jp": "Japanese",
|
|
"kor": "Korean",
|
|
"nor": "Norwegian",
|
|
"pol": "Polish",
|
|
"por": "Portuguese",
|
|
"rus": "Russian",
|
|
"ser": "Serbian",
|
|
"sp": "Spanish",
|
|
"spa": "Spanish",
|
|
"swe": "Swedish",
|
|
}
|
|
|
|
|
|
# The tag a translated dump carries, in place of the group and patch version
|
|
# the raw tag encodes, which would give every translation its own facet value.
|
|
TRANSLATION_TAG = "Translation"
|
|
|
|
|
|
def translation_language(code: str) -> str | None:
|
|
"""Resolve the language a translation tag targets, or None."""
|
|
tag = code.strip().lower()
|
|
return normalize_language(tag) or _TRANSLATION_LANGUAGE_ALIASES.get(tag)
|
|
|
|
|
|
# ISO-639-1 languages with no filename shortcode of their own. Provider-only,
|
|
# because a filename tag reads by dumper convention instead: "(Tr)" marks a
|
|
# translation, not Turkish.
|
|
PROVIDER_LANGUAGES: Final = (
|
|
("af", "Afrikaans"),
|
|
("be", "Belarusian"),
|
|
("bg", "Bulgarian"),
|
|
("ca", "Catalan"),
|
|
("cs", "Czech"),
|
|
("et", "Estonian"),
|
|
("he", "Hebrew"),
|
|
("hi", "Hindi"),
|
|
("hr", "Croatian"),
|
|
("hu", "Hungarian"),
|
|
("hy", "Armenian"),
|
|
("id", "Indonesian"),
|
|
("is", "Icelandic"),
|
|
("la", "Latin"),
|
|
("lt", "Lithuanian"),
|
|
("lv", "Latvian"),
|
|
("mk", "Macedonian"),
|
|
("ro", "Romanian"),
|
|
("sk", "Slovak"),
|
|
("sl", "Slovenian"),
|
|
("sq", "Albanian"),
|
|
("th", "Thai"),
|
|
("tr", "Turkish"),
|
|
("uk", "Ukrainian"),
|
|
("vi", "Vietnamese"),
|
|
)
|
|
|
|
_LANGUAGE_BY_PROVIDER_ALIAS = {
|
|
**{name.lower(): name for _, name in PROVIDER_LANGUAGES},
|
|
**{code: name for code, name in PROVIDER_LANGUAGES},
|
|
}
|
|
|
|
# Region shortcodes a filename can also carry, as TOSEC and similar sets write
|
|
# them ("(US)", "(JP)"). Added once both language vocabularies are known, and
|
|
# only for a code neither claims: "(De)" is German, and "(Tr)" marks a
|
|
# translation rather than Turkey (issue #3026).
|
|
_REGION_BY_ALIAS.update(
|
|
{
|
|
code: names[0]
|
|
for code, names in _REGION_NAMES_BY_PROVIDER_SHORTCODE.items()
|
|
if code not in _LANGUAGE_BY_ALIAS and code not in _LANGUAGE_BY_PROVIDER_ALIAS
|
|
}
|
|
)
|
|
|
|
|
|
def provider_language_name(value: str) -> str | None:
|
|
"""Resolve a metadata provider's language spelling to its canonical name."""
|
|
return normalize_language(value) or _LANGUAGE_BY_PROVIDER_ALIAS.get(
|
|
value.strip().lower()
|
|
)
|
|
|
|
|
|
def normalize_provider_values(
|
|
values: Iterable[str], resolve: Callable[[str], str | None]
|
|
) -> list[str]:
|
|
"""Canonicalize provider spellings, dropping blanks and duplicates.
|
|
|
|
An unrecognized value is kept as given, the way filename parsing keeps a
|
|
tag it does not know rather than dropping it.
|
|
"""
|
|
return list(
|
|
dict.fromkeys(
|
|
resolve(value) or value.strip()
|
|
for value in values
|
|
if value and value.strip()
|
|
)
|
|
)
|
|
|
|
|
|
def normalize_provider_regions(values: Iterable[str]) -> list[str]:
|
|
return normalize_provider_values(values, provider_region_name)
|
|
|
|
|
|
def normalize_provider_languages(values: Iterable[str]) -> list[str]:
|
|
return normalize_provider_values(values, provider_language_name)
|
|
|
|
|
|
class CoverSize(Enum):
|
|
SMALL = "small"
|
|
BIG = "big"
|
|
|
|
|
|
class Asset(Enum):
|
|
SAVES = "saves"
|
|
STATES = "states"
|
|
SCREENSHOTS = "screenshots"
|
|
|
|
|
|
class FSHandler:
|
|
def __init__(self, base_path: str) -> None:
|
|
self.base_path = Path(base_path).resolve()
|
|
self._locks: dict[str, asyncio.Lock] = {}
|
|
self._lock_mutex = asyncio.Lock()
|
|
|
|
async def _get_file_lock(self, file_path: str) -> asyncio.Lock:
|
|
"""Get or create a lock for a specific file path."""
|
|
async with self._lock_mutex:
|
|
if file_path not in self._locks:
|
|
self._locks[file_path] = asyncio.Lock()
|
|
return self._locks[file_path]
|
|
|
|
def _sanitize_filename(self, filename: str) -> str:
|
|
"""Sanitize filename to prevent path traversal and other attacks."""
|
|
if not filename:
|
|
raise ValueError("Empty filename")
|
|
|
|
# Remove path components and get basename only
|
|
filename = os.path.basename(filename)
|
|
|
|
check_filename_length(filename)
|
|
|
|
# Ensure we have a valid filename
|
|
if not filename or filename == "." or filename == "..":
|
|
raise ValueError(f"Invalid filename {filename}")
|
|
|
|
return filename
|
|
|
|
def validate_path(self, path: str) -> Path:
|
|
"""Validate and normalize path to prevent directory traversal."""
|
|
path_path = Path(path)
|
|
|
|
# Check for explicit parent directory references
|
|
if ".." in path_path.parts:
|
|
raise ValueError(
|
|
f"Path {path} contains invalid parent directory references"
|
|
)
|
|
|
|
# Check for absolute paths
|
|
if path_path.is_absolute():
|
|
raise ValueError(f"Path {path} must be relative, not absolute")
|
|
|
|
# Normalize path without resolving the full path yet
|
|
base_path_obj = Path(self.base_path).resolve()
|
|
base_path_str = str(base_path_obj)
|
|
normalized_path = os.path.normpath(os.path.join(base_path_str, path_path))
|
|
if normalized_path == base_path_str:
|
|
return base_path_obj
|
|
# A bare startswith guard, which CodeQL recognizes as a path sanitizer.
|
|
if not normalized_path.startswith(base_path_str + os.sep):
|
|
raise ValueError(
|
|
f"Path {path} is outside the base directory {self.base_path}"
|
|
)
|
|
full_path = Path(normalized_path)
|
|
|
|
try:
|
|
# Detect a symlink anywhere in the path, not just at the leaf:
|
|
# users may symlink an intermediate directory (e.g. the library
|
|
# root) to point at storage on another filesystem.
|
|
has_symlink_in_path = full_path.is_symlink()
|
|
if not has_symlink_in_path:
|
|
for parent in full_path.parents:
|
|
if parent == base_path_obj:
|
|
break
|
|
if parent.is_symlink():
|
|
has_symlink_in_path = True
|
|
break
|
|
|
|
# A symlinked path already passed the lexical check above, so its
|
|
# target is reachable only via an intentionally-configured link.
|
|
if not has_symlink_in_path:
|
|
full_path.resolve().relative_to(base_path_obj)
|
|
except ValueError as exc:
|
|
raise ValueError(
|
|
f"Path {path} is outside the base directory {self.base_path}"
|
|
) from exc
|
|
|
|
return full_path
|
|
|
|
def cached_path_validator(self) -> Callable[[str], Path]:
|
|
"""Return a `validate_path` that checks each parent directory once per callable."""
|
|
validated_dirs: dict[Path, Path] = {}
|
|
|
|
def validate(path: str) -> Path:
|
|
rel_path = Path(path)
|
|
if ".." in rel_path.parts or rel_path.is_absolute() or not rel_path.name:
|
|
return self.validate_path(path)
|
|
|
|
parent = rel_path.parent
|
|
full_dir = validated_dirs.get(parent)
|
|
if full_dir is None:
|
|
full_dir = validated_dirs[parent] = self.validate_path(str(parent))
|
|
# validate_path accepts a symlinked leaf, and a plain leaf resolves
|
|
# inside its already checked directory, so the leaf needs no syscall.
|
|
return full_dir / rel_path.name
|
|
|
|
return validate
|
|
|
|
async def _compute_file_hash(self, file_path: str) -> str:
|
|
full_path = self.validate_path(file_path)
|
|
|
|
def digest() -> str:
|
|
with open(full_path, "rb") as f:
|
|
return hashlib.file_digest(
|
|
f, lambda: hashlib.md5(usedforsecurity=False)
|
|
).hexdigest()
|
|
|
|
return await asyncio.to_thread(digest)
|
|
|
|
async def compute_file_md5(self, file_path: str) -> str:
|
|
"""MD5 of the bytes on disk, unlike zip-aware `compute_content_hash`.
|
|
|
|
Raises:
|
|
OSError: The file is missing or unreadable.
|
|
"""
|
|
return await self._compute_file_hash(file_path)
|
|
|
|
@asynccontextmanager
|
|
async def _atomic_write(self, target_path: Path) -> AsyncIterator[Path]:
|
|
"""Context manager for atomic file writing.
|
|
|
|
Creates the temp file in the same directory as the target so the
|
|
final os.replace() occurs on the same filesystem.
|
|
"""
|
|
fd, temp_path_str = tempfile.mkstemp(
|
|
dir=str(target_path.parent), prefix=TEMP_FILE_PREFIX
|
|
)
|
|
temp_path = Path(temp_path_str)
|
|
os.close(fd)
|
|
|
|
try:
|
|
yield temp_path
|
|
os.chmod(temp_path, SERVED_FILE_MODE)
|
|
os.replace(str(temp_path), str(target_path))
|
|
|
|
# BaseException, not Exception: a cancelled scan raises CancelledError,
|
|
# which would otherwise skip cleanup and strand a temp file per cancel.
|
|
except BaseException:
|
|
async_temp = AnyioPath(temp_path)
|
|
if await async_temp.exists():
|
|
await async_temp.unlink()
|
|
raise
|
|
|
|
def get_file_name_with_no_extension(self, file_name: str) -> str:
|
|
return compute_file_name_no_ext(file_name)
|
|
|
|
def get_file_name_with_no_tags(self, file_name: str) -> str:
|
|
return compute_file_name_no_tags(file_name)
|
|
|
|
def parse_file_extension(self, file_name: str) -> str:
|
|
return compute_file_extension(file_name)
|
|
|
|
def extract_uuid_v4_from_filename(self, file_name: str) -> str:
|
|
match = UUID_V4_REGEX.search(file_name)
|
|
return match.group(0) if match else ""
|
|
|
|
def exclude_single_files(self, files: list[str]) -> list[str]:
|
|
cnfg = cm.get_config()
|
|
excluded_extensions = cnfg.EXCLUDED_SINGLE_EXT
|
|
excluded_names = cnfg.EXCLUDED_SINGLE_FILES
|
|
|
|
# Built once rather than per file, and endswith takes the whole tuple.
|
|
excluded_suffixes = tuple(f".{ext}" for ext in excluded_extensions)
|
|
|
|
def is_excluded(file_name: str) -> bool:
|
|
# Check whether the filename ends with any excluded extension entry.
|
|
if file_name.lower().endswith(excluded_suffixes):
|
|
return True
|
|
|
|
# Check if the file name matches a pattern in the excluded list.
|
|
return file_name in excluded_names or any(
|
|
fnmatch.fnmatch(file_name, name) for name in excluded_names
|
|
)
|
|
|
|
# Deciding per file keeps this linear. Collecting the exclusions first and
|
|
# then filtering against that list rescans it once per file, which gets
|
|
# expensive on platforms holding tens of thousands of files.
|
|
return [f for f in files if not is_excluded(f)]
|
|
|
|
async def make_directory(self, path: str) -> None:
|
|
"""
|
|
Create a directory at the specified path.
|
|
Args:
|
|
path: Relative path within base directory
|
|
|
|
Raises:
|
|
ValueError: If path is invalid
|
|
FileNotFoundError: If path is not a directory
|
|
"""
|
|
target_directory = self.validate_path(path)
|
|
|
|
# Async thread-safe directory creation
|
|
lock = await self._get_file_lock(str(target_directory))
|
|
async with lock:
|
|
if not target_directory.exists():
|
|
target_directory.mkdir(parents=True, exist_ok=True)
|
|
elif not target_directory.is_dir():
|
|
raise FileNotFoundError(
|
|
f"Path already exists and is not a directory: {str(target_directory)}"
|
|
)
|
|
|
|
async def list_directories(self, path: str) -> list[str]:
|
|
"""
|
|
List all directories in a given path.
|
|
|
|
Args:
|
|
path: Relative path within base directory
|
|
|
|
Returns:
|
|
List of directory names in the specified path
|
|
|
|
Raises:
|
|
FileNotFoundError: If path is invalid or not a directory
|
|
"""
|
|
target_directory = self.validate_path(path)
|
|
|
|
# Async thread-safe directory listing
|
|
lock = await self._get_file_lock(str(target_directory))
|
|
async with lock:
|
|
if not target_directory.is_dir():
|
|
raise FileNotFoundError(
|
|
f"Path does not exist or is not a directory: {str(target_directory)}"
|
|
)
|
|
|
|
return [
|
|
d for _, d in iter_directories(str(target_directory), recursive=False)
|
|
]
|
|
|
|
async def remove_directory(self, path: str) -> None:
|
|
"""
|
|
Remove a directory and all its contents.
|
|
|
|
Args:
|
|
path: Relative path within base directory
|
|
|
|
Raises:
|
|
FileNotFoundError: If path is invalid or not a directory
|
|
"""
|
|
target_directory = self.validate_path(path)
|
|
|
|
# Async thread-safe directory removal
|
|
lock = await self._get_file_lock(str(target_directory))
|
|
async with lock:
|
|
if not target_directory.is_dir():
|
|
raise FileNotFoundError(
|
|
f"Path does not exist or is not a directory: {str(target_directory)}"
|
|
)
|
|
|
|
shutil.rmtree(target_directory, ignore_errors=False)
|
|
|
|
async def write_file(
|
|
self,
|
|
file: UploadFile | BinaryIO | BytesIO | bytes | SpooledTemporaryFile[bytes],
|
|
path: str,
|
|
filename: str | None = None,
|
|
) -> None:
|
|
"""
|
|
Securely write file to filesystem.
|
|
|
|
Args:
|
|
file: File-like object to write
|
|
path: Relative path within base directory
|
|
filename: Optional filename override
|
|
|
|
Returns:
|
|
Dictionary with operation result and file info
|
|
"""
|
|
|
|
original_filename = filename or getattr(file, "filename", None)
|
|
if not original_filename:
|
|
raise ValueError("Filename cannot be empty")
|
|
|
|
# Validate and sanitize inputs
|
|
sanitized_filename = self._sanitize_filename(original_filename)
|
|
final_file_path = self.validate_path(os.path.join(path, sanitized_filename))
|
|
target_directory = final_file_path.parent
|
|
|
|
# Async thread-safe file operations
|
|
lock = await self._get_file_lock(str(final_file_path))
|
|
async with lock:
|
|
# Ensure target directory exists
|
|
target_directory.mkdir(parents=True, exist_ok=True)
|
|
|
|
# Write file atomically
|
|
async with self._atomic_write(final_file_path) as temp_path:
|
|
async with await open_file(temp_path, "wb") as temp_file:
|
|
if isinstance(file, UploadFile):
|
|
while chunk := file.file.read(8192):
|
|
await temp_file.write(chunk)
|
|
elif isinstance(file, BinaryIO) or isinstance(
|
|
file, SpooledTemporaryFile
|
|
):
|
|
file.seek(0)
|
|
while chunk := file.read(8192):
|
|
await temp_file.write(chunk)
|
|
elif isinstance(file, BytesIO):
|
|
await temp_file.write(file.getvalue())
|
|
elif isinstance(file, bytes):
|
|
await temp_file.write(file)
|
|
else:
|
|
raise ValueError("Unsupported file type for writing")
|
|
|
|
@asynccontextmanager
|
|
async def write_file_streamed(
|
|
self, path: str, filename: str
|
|
) -> AsyncIterator[AsyncFile[bytes]]:
|
|
"""
|
|
Write file to filesystem using a streamed approach.
|
|
|
|
The stream lands in a temporary file that is renamed over the target
|
|
once the caller's block completes, so a download killed mid-stream
|
|
leaves any existing file intact instead of truncating it in place. A
|
|
truncated file is the worst outcome, since it still satisfies the
|
|
`*_exists` checks and every later scan skips it.
|
|
|
|
Args:
|
|
path: Relative path within base directory
|
|
filename: Name of the file to write
|
|
|
|
Yields:
|
|
File object for writing
|
|
|
|
Raises:
|
|
ValueError: If path or filename is invalid
|
|
"""
|
|
if not path or not filename:
|
|
raise ValueError("Path and filename cannot be empty")
|
|
|
|
# Validate and sanitize inputs
|
|
sanitized_filename = self._sanitize_filename(filename)
|
|
final_file_path = self.validate_path(os.path.join(path, sanitized_filename))
|
|
target_directory = final_file_path.parent
|
|
|
|
# Async thread-safe file operations
|
|
lock = await self._get_file_lock(str(final_file_path))
|
|
async with lock:
|
|
# Ensure target directory exists
|
|
target_directory.mkdir(parents=True, exist_ok=True)
|
|
|
|
# The handle closes before _atomic_write renames, so the target
|
|
# never receives a partially flushed file.
|
|
async with self._atomic_write(final_file_path) as temp_path:
|
|
async with await open_file(temp_path, "wb") as f:
|
|
yield f
|
|
|
|
async def read_file(self, file_path: str) -> bytes:
|
|
"""
|
|
Read file from filesystem.
|
|
|
|
Args:
|
|
file_path: Relative path to the file
|
|
|
|
Returns:
|
|
File content as bytes
|
|
|
|
Raises:
|
|
FileNotFoundError: If file does not exist
|
|
"""
|
|
if not file_path:
|
|
raise ValueError("File path cannot be empty")
|
|
|
|
# Validate and normalize path
|
|
full_path = self.validate_path(file_path)
|
|
|
|
# Async thread-safe file read
|
|
lock = await self._get_file_lock(str(full_path))
|
|
async with lock:
|
|
if not full_path.is_file():
|
|
raise FileNotFoundError(f"File not found: {full_path}")
|
|
|
|
async with await open_file(full_path, "rb") as f:
|
|
return await f.read()
|
|
|
|
async def stream_file(self, file_path: str) -> AsyncFile[bytes]:
|
|
"""
|
|
Stream file from filesystem.
|
|
|
|
Args:
|
|
file_path: Relative path to the file
|
|
|
|
Returns:
|
|
File content as a stream
|
|
|
|
Raises:
|
|
FileNotFoundError: If file does not exist
|
|
"""
|
|
if not file_path:
|
|
raise ValueError("File path cannot be empty")
|
|
|
|
# Validate and normalize path
|
|
full_path = self.validate_path(file_path)
|
|
|
|
# Async thread-safe file stream
|
|
lock = await self._get_file_lock(str(full_path))
|
|
async with lock:
|
|
if not full_path.is_file():
|
|
raise FileNotFoundError(f"File not found: {full_path}")
|
|
|
|
return await open_file(full_path, "rb")
|
|
|
|
async def copy_file(
|
|
self,
|
|
source_full_path: Path,
|
|
dest_path: str,
|
|
allow_link: bool = False,
|
|
) -> None:
|
|
"""
|
|
Copy a file from source to destination.
|
|
|
|
Args:
|
|
source_full_path: Absolute path to the source file
|
|
dest_path: Relative path to the destination file
|
|
allow_link: Try a hardlink first and fall back to
|
|
a copy when the link isn't possible (cross-device,
|
|
unsupported filesystem, etc.)
|
|
|
|
Raises:
|
|
FileNotFoundError: If source file does not exist
|
|
ValueError: If destination path is invalid
|
|
"""
|
|
if not source_full_path or not dest_path:
|
|
raise ValueError("Source and destination paths cannot be empty")
|
|
|
|
# Validate and normalize path
|
|
dest_full_path = self.validate_path(dest_path)
|
|
|
|
# Use locks for both source and destination
|
|
source_lock = await self._get_file_lock(str(source_full_path))
|
|
dest_lock = await self._get_file_lock(str(dest_full_path))
|
|
|
|
# Async thread-safe file copy
|
|
async with source_lock, dest_lock:
|
|
source_anyio_path = AnyioPath(str(source_full_path))
|
|
if not await source_anyio_path.is_file():
|
|
raise FileNotFoundError(f"Source file not found: {source_full_path}")
|
|
|
|
# Create destination directory if needed
|
|
dest_parent_anyio_path = AnyioPath(str(dest_full_path.parent))
|
|
await dest_parent_anyio_path.mkdir(parents=True, exist_ok=True)
|
|
if allow_link:
|
|
link_or_copy_file(source_full_path, dest_full_path)
|
|
else:
|
|
shutil.copy2(str(source_full_path), str(dest_full_path))
|
|
|
|
async def move_file_or_folder(self, source_path: str, dest_path: str) -> None:
|
|
"""
|
|
Move a file from source to destination.
|
|
|
|
Args:
|
|
source_path: Relative path to the source file
|
|
dest_path: Relative path to the destination file
|
|
|
|
Raises:
|
|
FileNotFoundError: If source file does not exist
|
|
ValueError: If destination path is invalid
|
|
"""
|
|
if not source_path or not dest_path:
|
|
raise ValueError("Source and destination paths cannot be empty")
|
|
|
|
# Validate and normalize paths
|
|
source_full_path = self.validate_path(source_path)
|
|
dest_full_path = self.validate_path(dest_path)
|
|
|
|
# Use locks for both source and destination
|
|
source_lock = await self._get_file_lock(str(source_full_path))
|
|
dest_lock = await self._get_file_lock(str(dest_full_path))
|
|
|
|
# Async thread-safe file move
|
|
async with source_lock, dest_lock:
|
|
if not source_full_path.exists():
|
|
raise FileNotFoundError(
|
|
f"Source file or folder not found: {source_full_path}"
|
|
)
|
|
|
|
# Create destination directory if needed
|
|
dest_full_path.parent.mkdir(parents=True, exist_ok=True)
|
|
shutil.move(str(source_full_path), str(dest_full_path))
|
|
|
|
def is_same_file(self, path: str, other_path: str) -> bool:
|
|
"""Whether two relative paths name one file, as names differing only in
|
|
case do on a case-insensitive filesystem."""
|
|
full_path = self.validate_path(path)
|
|
other_full_path = self.validate_path(other_path)
|
|
if full_path == other_full_path:
|
|
return True
|
|
try:
|
|
return full_path.samefile(other_full_path)
|
|
except FileNotFoundError:
|
|
return False
|
|
|
|
async def copy_to_new_file(self, source_path: str, dest_path: str) -> None:
|
|
"""
|
|
Copy a file to a path nothing holds yet, never replacing another file.
|
|
|
|
Args:
|
|
source_path: Relative path to the file to copy
|
|
dest_path: Relative path of the copy
|
|
|
|
Raises:
|
|
FileNotFoundError: If the source does not exist
|
|
FileExistsError: If the destination already exists
|
|
"""
|
|
source_full_path = self.validate_path(source_path)
|
|
dest_full_path = self.validate_path(dest_path)
|
|
if source_full_path == dest_full_path:
|
|
raise FileExistsError(f"File already exists: {dest_full_path}")
|
|
|
|
source_lock = await self._get_file_lock(str(source_full_path))
|
|
dest_lock = await self._get_file_lock(str(dest_full_path))
|
|
|
|
async with source_lock, dest_lock:
|
|
if not source_full_path.is_file():
|
|
raise FileNotFoundError(f"File not found: {source_full_path}")
|
|
|
|
# Both a link and an exclusive create refuse a name that exists.
|
|
try:
|
|
os.link(source_full_path, dest_full_path)
|
|
except OSError as exc:
|
|
if exc.errno not in LINK_FALLBACK_ERRNOS:
|
|
raise
|
|
with (
|
|
source_full_path.open("rb") as source,
|
|
dest_full_path.open("xb") as dest,
|
|
):
|
|
try:
|
|
shutil.copyfileobj(source, dest)
|
|
except BaseException:
|
|
# Only a file this copy created, never one it refused.
|
|
dest_full_path.unlink()
|
|
raise
|
|
shutil.copymode(source_full_path, dest_full_path)
|
|
|
|
async def rename_file(self, file_path: str, new_name: str) -> None:
|
|
"""
|
|
Rename a file within its directory, never replacing another file.
|
|
|
|
Args:
|
|
file_path: Relative path to the file to rename
|
|
new_name: New file name
|
|
|
|
Raises:
|
|
FileNotFoundError: If the file does not exist
|
|
FileExistsError: If another file already holds the new name
|
|
"""
|
|
source_full_path = self.validate_path(file_path)
|
|
dest_full_path = self.validate_path(
|
|
str(Path(file_path).with_name(self._sanitize_filename(new_name)))
|
|
)
|
|
if source_full_path == dest_full_path:
|
|
return
|
|
|
|
source_lock = await self._get_file_lock(str(source_full_path))
|
|
dest_lock = await self._get_file_lock(str(dest_full_path))
|
|
|
|
async with source_lock, dest_lock:
|
|
if not source_full_path.is_file():
|
|
raise FileNotFoundError(f"File not found: {source_full_path}")
|
|
|
|
if dest_full_path.exists():
|
|
# A case-only rename on a case-insensitive filesystem finds itself.
|
|
if not dest_full_path.samefile(source_full_path):
|
|
raise FileExistsError(f"File already exists: {dest_full_path}")
|
|
source_full_path.rename(dest_full_path)
|
|
return
|
|
|
|
# A link refuses a name another worker took since the check, which
|
|
# a rename would silently replace.
|
|
try:
|
|
os.link(source_full_path, dest_full_path)
|
|
except FileExistsError as exc:
|
|
raise FileExistsError(f"File already exists: {dest_full_path}") from exc
|
|
except OSError as exc:
|
|
if exc.errno not in LINK_FALLBACK_ERRNOS:
|
|
raise
|
|
source_full_path.rename(dest_full_path)
|
|
return
|
|
source_full_path.unlink()
|
|
|
|
async def remove_file(self, file_path: str) -> None:
|
|
"""
|
|
Remove a file from the filesystem.
|
|
|
|
Args:
|
|
file_path: Relative path to the file to remove
|
|
|
|
Raises:
|
|
FileNotFoundError: If file does not exist
|
|
"""
|
|
if not file_path:
|
|
raise ValueError("File path cannot be empty")
|
|
|
|
# Validate and normalize path
|
|
full_path = self.validate_path(file_path)
|
|
|
|
# Async thread-safe file removal
|
|
lock = await self._get_file_lock(str(full_path))
|
|
async with lock:
|
|
if not full_path.exists():
|
|
raise FileNotFoundError(f"File not found: {full_path}")
|
|
|
|
full_path.unlink()
|
|
|
|
async def list_files(self, path: str) -> list[str]:
|
|
"""
|
|
List all files in a directory.
|
|
|
|
Args:
|
|
directory: Relative path to the directory
|
|
|
|
Returns:
|
|
List of file names in the directory
|
|
|
|
Raises:
|
|
FileNotFoundError: If directory does not exist
|
|
"""
|
|
if not path:
|
|
raise ValueError("Directory cannot be empty")
|
|
|
|
# Validate and normalize path
|
|
full_path = self.validate_path(path)
|
|
|
|
# Async thread-safe directory listing
|
|
lock = await self._get_file_lock(str(full_path))
|
|
async with lock:
|
|
if not full_path.is_dir():
|
|
raise FileNotFoundError(f"Directory not found: {full_path}")
|
|
|
|
return [f for _, f in iter_files(str(full_path), recursive=False)]
|
|
|
|
async def file_exists(self, file_path: str) -> bool:
|
|
"""
|
|
Check if a file exists.
|
|
|
|
Args:
|
|
file_path: Relative path to the file
|
|
|
|
Returns:
|
|
True if file exists, False otherwise
|
|
"""
|
|
if not file_path:
|
|
raise ValueError("File path cannot be empty")
|
|
|
|
# Validate and normalize path
|
|
full_path = self.validate_path(file_path)
|
|
|
|
# Async thread-safe existence check
|
|
lock = await self._get_file_lock(str(full_path))
|
|
async with lock:
|
|
return full_path.is_file()
|
|
|
|
async def directory_exists(self, path: str) -> bool:
|
|
"""
|
|
Check if a directory exists.
|
|
|
|
Args:
|
|
path: Relative path to the directory
|
|
|
|
Returns:
|
|
True if directory exists, False otherwise
|
|
"""
|
|
if not path:
|
|
raise ValueError("Directory path cannot be empty")
|
|
|
|
# Validate and normalize path
|
|
full_path = self.validate_path(path)
|
|
|
|
# Async thread-safe existence check
|
|
lock = await self._get_file_lock(str(full_path))
|
|
async with lock:
|
|
return full_path.is_dir()
|
|
|
|
async def get_file_size(self, file_path: str) -> int:
|
|
"""
|
|
Get the size of a file.
|
|
|
|
Args:
|
|
file_path: Relative path to the file
|
|
|
|
Returns:
|
|
Size of the file in bytes
|
|
|
|
Raises:
|
|
FileNotFoundError: If file does not exist
|
|
"""
|
|
if not file_path:
|
|
raise ValueError("File path cannot be empty")
|
|
|
|
# Validate and normalize path
|
|
full_path = self.validate_path(file_path)
|
|
|
|
# Async thread-safe file size retrieval
|
|
lock = await self._get_file_lock(str(full_path))
|
|
async with lock:
|
|
if not full_path.is_file():
|
|
raise FileNotFoundError(f"File not found: {full_path}")
|
|
|
|
return full_path.stat().st_size
|