Files
romm/backend/handler/filesystem/base_handler.py
T
Georges-Antoine AssiandClaude Opus 5.5 e232a52656 fix(docker): boot when the container runs as a non-default UID (#5189)
* fix(docker): boot when the container runs as a non-default UID

Unraid runs the container as 99:100, which can't write to the image's
/romm (owned by 1000, mode 755), only to the mounted subfolders. The
PSP pending dir at /romm/cache/retroarch_sync was created at import
time, so every gunicorn worker failed to boot.

- Buffer pending PSP save files under the user's assets folder
  (assets/users/<user>/retroarch/psp_pending/), beside the other
  RetroArch Cloud Sync files and outside the client-reachable blob
  categories.
- Move the zip and rom-converto download caches to /tmp/romm/cache,
  and point nginx's internal /cache/ alias there.
- Put gunicorn's control socket in /tmp/gunicorn instead of
  $HOME/.gunicorn, since HOME is / for that UID.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(docker): honor ROMM_TMP_PATH for caches, explain an unwritable sync folder

- The download caches live under $ROMM_TMP_PATH/cache when it's set,
  so installs with a small Docker disk (Unraid's docker.img) can point
  them at a share. The entrypoint exports the same path for nginx's
  /cache/ alias.
- With ENABLE_SYNC_FOLDER_WATCHER on and /romm/sync not writable, /init
  warns and skips the watcher instead of dying on mkdir under errexit.
- The compose example lists the /romm/sync mount the sync features need.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(sync): cover reading back and clearing buffered PSP files

Addresses Greptile's review on #5189.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(sync): apply code-review findings to PSP pending buffer

Remove the emptied psp_pending folder after a merge, define the sync
folder path once in init, update the architecture doc's PSP and blob
locations, and drop a stale .gitignore entry.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(sync): apply simplify findings to PSP pending cleanup

Clear the buffered folder with remove_directory, let the startup check
own creating the sync folder, and reword the test fixture docstring.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(sync): cap buffered PSP files and drop them on delete

Buffered files for an unresolved save folder now count against the
bundle limits, so a folder can't fill the assets share and then fail
every merge once it resolves. A DELETE also removes the buffered copy,
so a deleted file isn't merged back when the folder resolves.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(docker): check the sync folder for push-pull sync too

ENABLE_SYNC_PUSH_PULL keeps SSH keys and device saves under /romm/sync
as well, so an unwritable folder now disables it with the watcher and
warns at boot instead of failing when the task runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(docker): strip whitespace from env vars like the backend does

The backend strips every variable it reads, while the init script and
nginx templates used raw values, so a padded ROMM_TMP_PATH sent nginx's
/cache/ alias somewhere the backend never writes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* chore(config): drop the CACHE_BASE_PATH comment

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs(docker): say which backend config strips env whitespace

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* revert(docker): drop the entrypoint env whitespace trimming

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(sync): apply pr-ready code-review findings

Gate push-pull on its SSH keys path rather than a writable sync folder,
accept the same truthy values as the backend, and clear only the merged
PSP pending files so another worker's buffered file survives.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor: apply pr-ready simplify findings

Share one truthy pattern across init's flag checks (the watcher gates
now accept the same values as the backend), disable gunicorn's unused
control socket instead of placing it, derive the converts redirect from
its own cache path, and factor the unknown PSP folder in tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs(docker): apply pr-ready humanizer pass to the sync mount note

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(fs): create handler base folders on first write, skip leftover temp files

FSHandler no longer mkdirs its base path when constructed, so a module-level
handler rooted at an unmounted or unwritable folder can't take down boot;
every write path already creates its parent folders. Blob listings skip
.romm_tmp_ files a crashed write left behind, so they are never merged into
a PSP bundle or advertised in the sync manifest. The prefix now lives in one
TEMP_FILE_PREFIX constant.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
2026-10-08 16:48:50 -04:00

1072 lines
36 KiB
Python

import asyncio
import fnmatch
import hashlib
import os
import re
import shutil
import tempfile
from collections.abc import AsyncIterator, Callable, Iterable, Sequence
from contextlib import asynccontextmanager
from enum import Enum
from io import BytesIO
from pathlib import Path
from tempfile import SpooledTemporaryFile
from typing import BinaryIO, Final
from anyio import AsyncFile
from anyio import Path as AnyioPath
from anyio import open_file
from starlette.datastructures import UploadFile
from config.config_manager import config_manager as cm
from models.base import (
compute_file_extension,
compute_file_name_no_ext,
compute_file_name_no_tags,
)
from utils.filesystem import (
LINK_FALLBACK_ERRNOS,
SERVED_FILE_MODE,
TEMP_FILE_PREFIX,
check_filename_length,
iter_directories,
iter_files,
link_or_copy_file,
)
UUID_V4_REGEX = re.compile(
r"[0-9a-f]{8}-[0-9a-f]{4}-[0-5][0-9a-f]{3}-[089ab][0-9a-f]{3}-[0-9a-f]{12}",
re.IGNORECASE,
)
LANGUAGES = (
("Ar", "Arabic"),
("Da", "Danish"),
("De", "German"),
("El", "Greek"),
("En", "English"),
("Es", "Spanish"),
("Fi", "Finnish"),
("Fr", "French"),
("It", "Italian"),
("Ja", "Japanese"),
("Ko", "Korean"),
("Nl", "Dutch"),
("No", "Norwegian"),
("Pl", "Polish"),
("Pt", "Portuguese"),
("Ru", "Russian"),
("Sr", "Serbian"),
("Sv", "Swedish"),
("Zh", "Chinese"),
("nolang", "No Language"),
)
REGIONS = (
("A", "Australia"),
("AS", "Asia"),
("B", "Brazil"),
("C", "Canada"),
("CH", "China"),
("E", "Europe"),
("F", "France"),
("FN", "Finland"),
("G", "Germany"),
("GR", "Greece"),
("H", "Holland"),
("HK", "Hong Kong"),
("I", "Italy"),
("J", "Japan"),
("K", "Korea"),
("NL", "Netherlands"),
("NO", "Norway"),
("PD", "Public Domain"),
("R", "Russia"),
("S", "Spain"),
("SW", "Sweden"),
("T", "Taiwan"),
("U", "USA"),
("UK", "England"),
("UNK", "Unknown"),
("UNL", "Unlicensed"),
("W", "World"),
)
REGIONS_BY_SHORTCODE = {region[0]: region[1] for region in REGIONS}
# Every accepted region spelling, lowercased, mapped to its canonical name.
_REGION_BY_ALIAS = {
**{name.lower(): name for _, name in REGIONS},
**{code.lower(): name for code, name in REGIONS},
}
def normalize_region(tag: str) -> str | None:
"""Resolve a filename region tag to its canonical REGIONS name.
Case-insensitive, so "usa", "USA" and "Usa" collapse to one facet value
instead of three. Returns None for tags that name no known region.
"""
return _REGION_BY_ALIAS.get(tag.strip().lower())
# Maps full REGIONS names to lowercase shortcodes used by metadata providers
REGION_NAME_TO_PROVIDER_SHORTCODE: dict[str, str] = {
"Australia": "au",
"Asia": "asi",
"Brazil": "br",
"Canada": "ca",
"China": "cn",
"England": "uk",
"Europe": "eu",
"Finland": "fi",
"France": "fr",
"Germany": "de",
"Greece": "gr",
"Holland": "nl",
"Hong Kong": "hk",
"Italy": "it",
"Japan": "jp",
"Korea": "kr",
"Netherlands": "nl",
"Norway": "no",
"Russia": "ru",
"Spain": "sp",
"Sweden": "se",
"Taiwan": "tw",
"USA": "us",
"World": "wor",
# ScreenScraper regions with no filename shortcode of their own. Providers
# send these codes bare, so without a name here they land in a facet as
# "pl" beside the "Poland" a filename or Hasheous produces.
"Americas": "ame",
"Bulgaria": "bg",
"Chile": "cl",
"Czech Republic": "cz",
"Denmark": "dk",
"Hungary": "hu",
"Israel": "il",
"Kuwait": "kw",
"Middle East": "mor",
"New Zealand": "nz",
"Oceania": "oce",
"Peru": "pe",
"Poland": "pl",
"Portugal": "pt",
"Slovakia": "sk",
"Turkey": "tr",
}
_REGION_NAME_TO_PROVIDER_SHORTCODE_CI = {
k.lower(): v for k, v in REGION_NAME_TO_PROVIDER_SHORTCODE.items()
}
def region_name_to_provider_shortcode(region_name: str | None) -> str | None:
"""Look up a provider shortcode for a region name (case-insensitive).
ROM filename parsing can leave region names in their raw casing
(e.g. "europe", "EUROPE"), so callers must normalize before lookup.
"""
if not region_name:
return None
return _REGION_NAME_TO_PROVIDER_SHORTCODE_CI.get(region_name.lower())
# Reverse of REGION_NAME_TO_PROVIDER_SHORTCODE. A list per code because two
# names can claim one code ("nl" for both Holland and Netherlands), and a rom
# tagged with either has to rank the same.
_REGION_NAMES_BY_PROVIDER_SHORTCODE: dict[str, list[str]] = {}
for _name, _code in REGION_NAME_TO_PROVIDER_SHORTCODE.items():
_REGION_NAMES_BY_PROVIDER_SHORTCODE.setdefault(_code, []).append(_name)
def region_ranks_for_priority(shortcodes: Sequence[str]) -> dict[str, int]:
"""Map canonical region names to their rank in a shortcode priority list.
The rank is the shortcode's position, not the name's, so two names claiming
one shortcode ("nl" for both Holland and Netherlands) rank equally and fall
through to whatever tiebreak follows.
A shortcode naming no known region (ScreenScraper's "ss", or a typo the
open-ended settings input allowed) contributes nothing rather than shifting
the ranks of everything after it. Nor does a repeated shortcode.
"""
ranks: dict[str, int] = {}
next_rank = 0
for shortcode in dict.fromkeys(code.lower() for code in shortcodes):
names = _REGION_NAMES_BY_PROVIDER_SHORTCODE.get(shortcode)
if not names:
continue
for name in names:
ranks[name] = next_rank
next_rank += 1
return ranks
# Provider spellings kept out of the filename aliases, where several of these
# codes ("de", "fr", "nl", "ru") read as a language tag instead (issue #3026).
_REGION_BY_PROVIDER_ALIAS = {
**{name.lower(): name for name in REGION_NAME_TO_PROVIDER_SHORTCODE},
**{code: names[0] for code, names in _REGION_NAMES_BY_PROVIDER_SHORTCODE.items()},
}
def provider_region_name(value: str) -> str | None:
"""Resolve a metadata provider's region spelling to its canonical name."""
return normalize_region(value) or _REGION_BY_PROVIDER_ALIAS.get(
value.strip().lower()
)
LANGUAGES_BY_SHORTCODE = {lang[0]: lang[1] for lang in LANGUAGES}
# Every accepted language spelling, lowercased, mapped to its canonical name.
_LANGUAGE_BY_ALIAS = {
**{name.lower(): name for _, name in LANGUAGES},
**{code.lower(): name for code, name in LANGUAGES},
}
def normalize_language(tag: str) -> str | None:
"""Resolve a filename language tag to its canonical LANGUAGES name.
Case-insensitive, so "english", "English" and "ENGLISH" collapse to one
facet value. Returns None for tags that name no known language.
"""
return _LANGUAGE_BY_ALIAS.get(tag.strip().lower())
# Codes only a translation tag uses for its target language ("[T+Eng]",
# "[T-Ge]"). Two-letter ISO codes are absent: normalize_language has those.
_TRANSLATION_LANGUAGE_ALIASES = {
"ara": "Arabic",
"chi": "Chinese",
"dan": "Danish",
"du": "Dutch",
"dut": "Dutch",
"eng": "English",
"fin": "Finnish",
"fre": "French",
"ge": "German",
"ger": "German",
"gr": "Greek",
"gre": "Greek",
"ita": "Italian",
"jap": "Japanese",
"jp": "Japanese",
"kor": "Korean",
"nor": "Norwegian",
"pol": "Polish",
"por": "Portuguese",
"rus": "Russian",
"ser": "Serbian",
"sp": "Spanish",
"spa": "Spanish",
"swe": "Swedish",
}
# The tag a translated dump carries, in place of the group and patch version
# the raw tag encodes, which would give every translation its own facet value.
TRANSLATION_TAG = "Translation"
def translation_language(code: str) -> str | None:
"""Resolve the language a translation tag targets, or None."""
tag = code.strip().lower()
return normalize_language(tag) or _TRANSLATION_LANGUAGE_ALIASES.get(tag)
# ISO-639-1 languages with no filename shortcode of their own. Provider-only,
# because a filename tag reads by dumper convention instead: "(Tr)" marks a
# translation, not Turkish.
PROVIDER_LANGUAGES: Final = (
("af", "Afrikaans"),
("be", "Belarusian"),
("bg", "Bulgarian"),
("ca", "Catalan"),
("cs", "Czech"),
("et", "Estonian"),
("he", "Hebrew"),
("hi", "Hindi"),
("hr", "Croatian"),
("hu", "Hungarian"),
("hy", "Armenian"),
("id", "Indonesian"),
("is", "Icelandic"),
("la", "Latin"),
("lt", "Lithuanian"),
("lv", "Latvian"),
("mk", "Macedonian"),
("ro", "Romanian"),
("sk", "Slovak"),
("sl", "Slovenian"),
("sq", "Albanian"),
("th", "Thai"),
("tr", "Turkish"),
("uk", "Ukrainian"),
("vi", "Vietnamese"),
)
_LANGUAGE_BY_PROVIDER_ALIAS = {
**{name.lower(): name for _, name in PROVIDER_LANGUAGES},
**{code: name for code, name in PROVIDER_LANGUAGES},
}
# Region shortcodes a filename can also carry, as TOSEC and similar sets write
# them ("(US)", "(JP)"). Added once both language vocabularies are known, and
# only for a code neither claims: "(De)" is German, and "(Tr)" marks a
# translation rather than Turkey (issue #3026).
_REGION_BY_ALIAS.update(
{
code: names[0]
for code, names in _REGION_NAMES_BY_PROVIDER_SHORTCODE.items()
if code not in _LANGUAGE_BY_ALIAS and code not in _LANGUAGE_BY_PROVIDER_ALIAS
}
)
def provider_language_name(value: str) -> str | None:
"""Resolve a metadata provider's language spelling to its canonical name."""
return normalize_language(value) or _LANGUAGE_BY_PROVIDER_ALIAS.get(
value.strip().lower()
)
def normalize_provider_values(
values: Iterable[str], resolve: Callable[[str], str | None]
) -> list[str]:
"""Canonicalize provider spellings, dropping blanks and duplicates.
An unrecognized value is kept as given, the way filename parsing keeps a
tag it does not know rather than dropping it.
"""
return list(
dict.fromkeys(
resolve(value) or value.strip()
for value in values
if value and value.strip()
)
)
def normalize_provider_regions(values: Iterable[str]) -> list[str]:
return normalize_provider_values(values, provider_region_name)
def normalize_provider_languages(values: Iterable[str]) -> list[str]:
return normalize_provider_values(values, provider_language_name)
class CoverSize(Enum):
SMALL = "small"
BIG = "big"
class Asset(Enum):
SAVES = "saves"
STATES = "states"
SCREENSHOTS = "screenshots"
class FSHandler:
def __init__(self, base_path: str) -> None:
self.base_path = Path(base_path).resolve()
self._locks: dict[str, asyncio.Lock] = {}
self._lock_mutex = asyncio.Lock()
async def _get_file_lock(self, file_path: str) -> asyncio.Lock:
"""Get or create a lock for a specific file path."""
async with self._lock_mutex:
if file_path not in self._locks:
self._locks[file_path] = asyncio.Lock()
return self._locks[file_path]
def _sanitize_filename(self, filename: str) -> str:
"""Sanitize filename to prevent path traversal and other attacks."""
if not filename:
raise ValueError("Empty filename")
# Remove path components and get basename only
filename = os.path.basename(filename)
check_filename_length(filename)
# Ensure we have a valid filename
if not filename or filename == "." or filename == "..":
raise ValueError(f"Invalid filename {filename}")
return filename
def validate_path(self, path: str) -> Path:
"""Validate and normalize path to prevent directory traversal."""
path_path = Path(path)
# Check for explicit parent directory references
if ".." in path_path.parts:
raise ValueError(
f"Path {path} contains invalid parent directory references"
)
# Check for absolute paths
if path_path.is_absolute():
raise ValueError(f"Path {path} must be relative, not absolute")
# Normalize path without resolving the full path yet
base_path_obj = Path(self.base_path).resolve()
base_path_str = str(base_path_obj)
normalized_path = os.path.normpath(os.path.join(base_path_str, path_path))
if normalized_path == base_path_str:
return base_path_obj
# A bare startswith guard, which CodeQL recognizes as a path sanitizer.
if not normalized_path.startswith(base_path_str + os.sep):
raise ValueError(
f"Path {path} is outside the base directory {self.base_path}"
)
full_path = Path(normalized_path)
try:
# Detect a symlink anywhere in the path, not just at the leaf:
# users may symlink an intermediate directory (e.g. the library
# root) to point at storage on another filesystem.
has_symlink_in_path = full_path.is_symlink()
if not has_symlink_in_path:
for parent in full_path.parents:
if parent == base_path_obj:
break
if parent.is_symlink():
has_symlink_in_path = True
break
# A symlinked path already passed the lexical check above, so its
# target is reachable only via an intentionally-configured link.
if not has_symlink_in_path:
full_path.resolve().relative_to(base_path_obj)
except ValueError as exc:
raise ValueError(
f"Path {path} is outside the base directory {self.base_path}"
) from exc
return full_path
def cached_path_validator(self) -> Callable[[str], Path]:
"""Return a `validate_path` that checks each parent directory once per callable."""
validated_dirs: dict[Path, Path] = {}
def validate(path: str) -> Path:
rel_path = Path(path)
if ".." in rel_path.parts or rel_path.is_absolute() or not rel_path.name:
return self.validate_path(path)
parent = rel_path.parent
full_dir = validated_dirs.get(parent)
if full_dir is None:
full_dir = validated_dirs[parent] = self.validate_path(str(parent))
# validate_path accepts a symlinked leaf, and a plain leaf resolves
# inside its already checked directory, so the leaf needs no syscall.
return full_dir / rel_path.name
return validate
async def _compute_file_hash(self, file_path: str) -> str:
full_path = self.validate_path(file_path)
def digest() -> str:
with open(full_path, "rb") as f:
return hashlib.file_digest(
f, lambda: hashlib.md5(usedforsecurity=False)
).hexdigest()
return await asyncio.to_thread(digest)
async def compute_file_md5(self, file_path: str) -> str:
"""MD5 of the bytes on disk, unlike zip-aware `compute_content_hash`.
Raises:
OSError: The file is missing or unreadable.
"""
return await self._compute_file_hash(file_path)
@asynccontextmanager
async def _atomic_write(self, target_path: Path) -> AsyncIterator[Path]:
"""Context manager for atomic file writing.
Creates the temp file in the same directory as the target so the
final os.replace() occurs on the same filesystem.
"""
fd, temp_path_str = tempfile.mkstemp(
dir=str(target_path.parent), prefix=TEMP_FILE_PREFIX
)
temp_path = Path(temp_path_str)
os.close(fd)
try:
yield temp_path
os.chmod(temp_path, SERVED_FILE_MODE)
os.replace(str(temp_path), str(target_path))
# BaseException, not Exception: a cancelled scan raises CancelledError,
# which would otherwise skip cleanup and strand a temp file per cancel.
except BaseException:
async_temp = AnyioPath(temp_path)
if await async_temp.exists():
await async_temp.unlink()
raise
def get_file_name_with_no_extension(self, file_name: str) -> str:
return compute_file_name_no_ext(file_name)
def get_file_name_with_no_tags(self, file_name: str) -> str:
return compute_file_name_no_tags(file_name)
def parse_file_extension(self, file_name: str) -> str:
return compute_file_extension(file_name)
def extract_uuid_v4_from_filename(self, file_name: str) -> str:
match = UUID_V4_REGEX.search(file_name)
return match.group(0) if match else ""
def exclude_single_files(self, files: list[str]) -> list[str]:
cnfg = cm.get_config()
excluded_extensions = cnfg.EXCLUDED_SINGLE_EXT
excluded_names = cnfg.EXCLUDED_SINGLE_FILES
# Built once rather than per file, and endswith takes the whole tuple.
excluded_suffixes = tuple(f".{ext}" for ext in excluded_extensions)
def is_excluded(file_name: str) -> bool:
# Check whether the filename ends with any excluded extension entry.
if file_name.lower().endswith(excluded_suffixes):
return True
# Check if the file name matches a pattern in the excluded list.
return file_name in excluded_names or any(
fnmatch.fnmatch(file_name, name) for name in excluded_names
)
# Deciding per file keeps this linear. Collecting the exclusions first and
# then filtering against that list rescans it once per file, which gets
# expensive on platforms holding tens of thousands of files.
return [f for f in files if not is_excluded(f)]
async def make_directory(self, path: str) -> None:
"""
Create a directory at the specified path.
Args:
path: Relative path within base directory
Raises:
ValueError: If path is invalid
FileNotFoundError: If path is not a directory
"""
target_directory = self.validate_path(path)
# Async thread-safe directory creation
lock = await self._get_file_lock(str(target_directory))
async with lock:
if not target_directory.exists():
target_directory.mkdir(parents=True, exist_ok=True)
elif not target_directory.is_dir():
raise FileNotFoundError(
f"Path already exists and is not a directory: {str(target_directory)}"
)
async def list_directories(self, path: str) -> list[str]:
"""
List all directories in a given path.
Args:
path: Relative path within base directory
Returns:
List of directory names in the specified path
Raises:
FileNotFoundError: If path is invalid or not a directory
"""
target_directory = self.validate_path(path)
# Async thread-safe directory listing
lock = await self._get_file_lock(str(target_directory))
async with lock:
if not target_directory.is_dir():
raise FileNotFoundError(
f"Path does not exist or is not a directory: {str(target_directory)}"
)
return [
d for _, d in iter_directories(str(target_directory), recursive=False)
]
async def remove_directory(self, path: str) -> None:
"""
Remove a directory and all its contents.
Args:
path: Relative path within base directory
Raises:
FileNotFoundError: If path is invalid or not a directory
"""
target_directory = self.validate_path(path)
# Async thread-safe directory removal
lock = await self._get_file_lock(str(target_directory))
async with lock:
if not target_directory.is_dir():
raise FileNotFoundError(
f"Path does not exist or is not a directory: {str(target_directory)}"
)
shutil.rmtree(target_directory, ignore_errors=False)
async def write_file(
self,
file: UploadFile | BinaryIO | BytesIO | bytes | SpooledTemporaryFile[bytes],
path: str,
filename: str | None = None,
) -> None:
"""
Securely write file to filesystem.
Args:
file: File-like object to write
path: Relative path within base directory
filename: Optional filename override
Returns:
Dictionary with operation result and file info
"""
original_filename = filename or getattr(file, "filename", None)
if not original_filename:
raise ValueError("Filename cannot be empty")
# Validate and sanitize inputs
sanitized_filename = self._sanitize_filename(original_filename)
final_file_path = self.validate_path(os.path.join(path, sanitized_filename))
target_directory = final_file_path.parent
# Async thread-safe file operations
lock = await self._get_file_lock(str(final_file_path))
async with lock:
# Ensure target directory exists
target_directory.mkdir(parents=True, exist_ok=True)
# Write file atomically
async with self._atomic_write(final_file_path) as temp_path:
async with await open_file(temp_path, "wb") as temp_file:
if isinstance(file, UploadFile):
while chunk := file.file.read(8192):
await temp_file.write(chunk)
elif isinstance(file, BinaryIO) or isinstance(
file, SpooledTemporaryFile
):
file.seek(0)
while chunk := file.read(8192):
await temp_file.write(chunk)
elif isinstance(file, BytesIO):
await temp_file.write(file.getvalue())
elif isinstance(file, bytes):
await temp_file.write(file)
else:
raise ValueError("Unsupported file type for writing")
@asynccontextmanager
async def write_file_streamed(
self, path: str, filename: str
) -> AsyncIterator[AsyncFile[bytes]]:
"""
Write file to filesystem using a streamed approach.
The stream lands in a temporary file that is renamed over the target
once the caller's block completes, so a download killed mid-stream
leaves any existing file intact instead of truncating it in place. A
truncated file is the worst outcome, since it still satisfies the
`*_exists` checks and every later scan skips it.
Args:
path: Relative path within base directory
filename: Name of the file to write
Yields:
File object for writing
Raises:
ValueError: If path or filename is invalid
"""
if not path or not filename:
raise ValueError("Path and filename cannot be empty")
# Validate and sanitize inputs
sanitized_filename = self._sanitize_filename(filename)
final_file_path = self.validate_path(os.path.join(path, sanitized_filename))
target_directory = final_file_path.parent
# Async thread-safe file operations
lock = await self._get_file_lock(str(final_file_path))
async with lock:
# Ensure target directory exists
target_directory.mkdir(parents=True, exist_ok=True)
# The handle closes before _atomic_write renames, so the target
# never receives a partially flushed file.
async with self._atomic_write(final_file_path) as temp_path:
async with await open_file(temp_path, "wb") as f:
yield f
async def read_file(self, file_path: str) -> bytes:
"""
Read file from filesystem.
Args:
file_path: Relative path to the file
Returns:
File content as bytes
Raises:
FileNotFoundError: If file does not exist
"""
if not file_path:
raise ValueError("File path cannot be empty")
# Validate and normalize path
full_path = self.validate_path(file_path)
# Async thread-safe file read
lock = await self._get_file_lock(str(full_path))
async with lock:
if not full_path.is_file():
raise FileNotFoundError(f"File not found: {full_path}")
async with await open_file(full_path, "rb") as f:
return await f.read()
async def stream_file(self, file_path: str) -> AsyncFile[bytes]:
"""
Stream file from filesystem.
Args:
file_path: Relative path to the file
Returns:
File content as a stream
Raises:
FileNotFoundError: If file does not exist
"""
if not file_path:
raise ValueError("File path cannot be empty")
# Validate and normalize path
full_path = self.validate_path(file_path)
# Async thread-safe file stream
lock = await self._get_file_lock(str(full_path))
async with lock:
if not full_path.is_file():
raise FileNotFoundError(f"File not found: {full_path}")
return await open_file(full_path, "rb")
async def copy_file(
self,
source_full_path: Path,
dest_path: str,
allow_link: bool = False,
) -> None:
"""
Copy a file from source to destination.
Args:
source_full_path: Absolute path to the source file
dest_path: Relative path to the destination file
allow_link: Try a hardlink first and fall back to
a copy when the link isn't possible (cross-device,
unsupported filesystem, etc.)
Raises:
FileNotFoundError: If source file does not exist
ValueError: If destination path is invalid
"""
if not source_full_path or not dest_path:
raise ValueError("Source and destination paths cannot be empty")
# Validate and normalize path
dest_full_path = self.validate_path(dest_path)
# Use locks for both source and destination
source_lock = await self._get_file_lock(str(source_full_path))
dest_lock = await self._get_file_lock(str(dest_full_path))
# Async thread-safe file copy
async with source_lock, dest_lock:
source_anyio_path = AnyioPath(str(source_full_path))
if not await source_anyio_path.is_file():
raise FileNotFoundError(f"Source file not found: {source_full_path}")
# Create destination directory if needed
dest_parent_anyio_path = AnyioPath(str(dest_full_path.parent))
await dest_parent_anyio_path.mkdir(parents=True, exist_ok=True)
if allow_link:
link_or_copy_file(source_full_path, dest_full_path)
else:
shutil.copy2(str(source_full_path), str(dest_full_path))
async def move_file_or_folder(self, source_path: str, dest_path: str) -> None:
"""
Move a file from source to destination.
Args:
source_path: Relative path to the source file
dest_path: Relative path to the destination file
Raises:
FileNotFoundError: If source file does not exist
ValueError: If destination path is invalid
"""
if not source_path or not dest_path:
raise ValueError("Source and destination paths cannot be empty")
# Validate and normalize paths
source_full_path = self.validate_path(source_path)
dest_full_path = self.validate_path(dest_path)
# Use locks for both source and destination
source_lock = await self._get_file_lock(str(source_full_path))
dest_lock = await self._get_file_lock(str(dest_full_path))
# Async thread-safe file move
async with source_lock, dest_lock:
if not source_full_path.exists():
raise FileNotFoundError(
f"Source file or folder not found: {source_full_path}"
)
# Create destination directory if needed
dest_full_path.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(source_full_path), str(dest_full_path))
def is_same_file(self, path: str, other_path: str) -> bool:
"""Whether two relative paths name one file, as names differing only in
case do on a case-insensitive filesystem."""
full_path = self.validate_path(path)
other_full_path = self.validate_path(other_path)
if full_path == other_full_path:
return True
try:
return full_path.samefile(other_full_path)
except FileNotFoundError:
return False
async def copy_to_new_file(self, source_path: str, dest_path: str) -> None:
"""
Copy a file to a path nothing holds yet, never replacing another file.
Args:
source_path: Relative path to the file to copy
dest_path: Relative path of the copy
Raises:
FileNotFoundError: If the source does not exist
FileExistsError: If the destination already exists
"""
source_full_path = self.validate_path(source_path)
dest_full_path = self.validate_path(dest_path)
if source_full_path == dest_full_path:
raise FileExistsError(f"File already exists: {dest_full_path}")
source_lock = await self._get_file_lock(str(source_full_path))
dest_lock = await self._get_file_lock(str(dest_full_path))
async with source_lock, dest_lock:
if not source_full_path.is_file():
raise FileNotFoundError(f"File not found: {source_full_path}")
# Both a link and an exclusive create refuse a name that exists.
try:
os.link(source_full_path, dest_full_path)
except OSError as exc:
if exc.errno not in LINK_FALLBACK_ERRNOS:
raise
with (
source_full_path.open("rb") as source,
dest_full_path.open("xb") as dest,
):
try:
shutil.copyfileobj(source, dest)
except BaseException:
# Only a file this copy created, never one it refused.
dest_full_path.unlink()
raise
shutil.copymode(source_full_path, dest_full_path)
async def rename_file(self, file_path: str, new_name: str) -> None:
"""
Rename a file within its directory, never replacing another file.
Args:
file_path: Relative path to the file to rename
new_name: New file name
Raises:
FileNotFoundError: If the file does not exist
FileExistsError: If another file already holds the new name
"""
source_full_path = self.validate_path(file_path)
dest_full_path = self.validate_path(
str(Path(file_path).with_name(self._sanitize_filename(new_name)))
)
if source_full_path == dest_full_path:
return
source_lock = await self._get_file_lock(str(source_full_path))
dest_lock = await self._get_file_lock(str(dest_full_path))
async with source_lock, dest_lock:
if not source_full_path.is_file():
raise FileNotFoundError(f"File not found: {source_full_path}")
if dest_full_path.exists():
# A case-only rename on a case-insensitive filesystem finds itself.
if not dest_full_path.samefile(source_full_path):
raise FileExistsError(f"File already exists: {dest_full_path}")
source_full_path.rename(dest_full_path)
return
# A link refuses a name another worker took since the check, which
# a rename would silently replace.
try:
os.link(source_full_path, dest_full_path)
except FileExistsError as exc:
raise FileExistsError(f"File already exists: {dest_full_path}") from exc
except OSError as exc:
if exc.errno not in LINK_FALLBACK_ERRNOS:
raise
source_full_path.rename(dest_full_path)
return
source_full_path.unlink()
async def remove_file(self, file_path: str) -> None:
"""
Remove a file from the filesystem.
Args:
file_path: Relative path to the file to remove
Raises:
FileNotFoundError: If file does not exist
"""
if not file_path:
raise ValueError("File path cannot be empty")
# Validate and normalize path
full_path = self.validate_path(file_path)
# Async thread-safe file removal
lock = await self._get_file_lock(str(full_path))
async with lock:
if not full_path.exists():
raise FileNotFoundError(f"File not found: {full_path}")
full_path.unlink()
async def list_files(self, path: str) -> list[str]:
"""
List all files in a directory.
Args:
directory: Relative path to the directory
Returns:
List of file names in the directory
Raises:
FileNotFoundError: If directory does not exist
"""
if not path:
raise ValueError("Directory cannot be empty")
# Validate and normalize path
full_path = self.validate_path(path)
# Async thread-safe directory listing
lock = await self._get_file_lock(str(full_path))
async with lock:
if not full_path.is_dir():
raise FileNotFoundError(f"Directory not found: {full_path}")
return [f for _, f in iter_files(str(full_path), recursive=False)]
async def file_exists(self, file_path: str) -> bool:
"""
Check if a file exists.
Args:
file_path: Relative path to the file
Returns:
True if file exists, False otherwise
"""
if not file_path:
raise ValueError("File path cannot be empty")
# Validate and normalize path
full_path = self.validate_path(file_path)
# Async thread-safe existence check
lock = await self._get_file_lock(str(full_path))
async with lock:
return full_path.is_file()
async def directory_exists(self, path: str) -> bool:
"""
Check if a directory exists.
Args:
path: Relative path to the directory
Returns:
True if directory exists, False otherwise
"""
if not path:
raise ValueError("Directory path cannot be empty")
# Validate and normalize path
full_path = self.validate_path(path)
# Async thread-safe existence check
lock = await self._get_file_lock(str(full_path))
async with lock:
return full_path.is_dir()
async def get_file_size(self, file_path: str) -> int:
"""
Get the size of a file.
Args:
file_path: Relative path to the file
Returns:
Size of the file in bytes
Raises:
FileNotFoundError: If file does not exist
"""
if not file_path:
raise ValueError("File path cannot be empty")
# Validate and normalize path
full_path = self.validate_path(file_path)
# Async thread-safe file size retrieval
lock = await self._get_file_lock(str(full_path))
async with lock:
if not full_path.is_file():
raise FileNotFoundError(f"File not found: {full_path}")
return full_path.stat().st_size