rommapp/romm
1
0
Fork 0
mirror of https://github.com/rommapp/romm.git synced 2026-10-12 02:21:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into claude/server-side-rom-patching-EaQck

Browse Source
This commit is contained in:
Georges-Antoine Assi
2026-06-23 07:01:07 -04:00
parent 316895ccdb ed7604d288
commit 52163633e5
1042 changed files with 148477 additions and 8879 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/backend-development/SKILL.md
+93
View File
@@ -0,0 +1,93 @@
---
name: backend-development
description: Working on the RomM Python backend (backend/) — a FastAPI app with SQLAlchemy 2.0, Alembic, RQ/Redis, and Socket.IO. Use when adding or changing API endpoints, handlers, ORM models, response schemas, metadata-provider adapters, background tasks, database migrations, or backend tests. Covers the layered architecture, conventions, auth/scopes, the OpenAPI→frontend type pipeline, and the uv/pytest/alembic/trunk workflow. Trigger on any work under backend/.
---
# RomM Backend — FastAPI / SQLAlchemy
Python 3.13+, FastAPI, SQLAlchemy 2.0 (MariaDB default; MySQL/PostgreSQL supported), Alembic, Redis + RQ for jobs/cache/sessions, Socket.IO for real-time. Managed with **uv**.
Full reference: **`docs/BACKEND_ARCHITECTURE.md`** (directory map, ER diagram, every endpoint, auth flows). Read it before non-trivial changes.
---
## Layered architecture — where code goes
```txt
endpoints/ FastAPI routers: request validation, response schemas, @protected_route scopes
endpoints/responses/ Pydantic response schemas (these shape the OpenAPI → frontend types)
endpoints/sockets/ Socket.IO event handlers
handler/ Business logic, decoupled from HTTP
├ auth/ HybridAuthBackend (session/basic/bearer/OIDC/client-token), scopes, CSRF/session middleware
├ database/ Per-entity CRUD handlers (db_rom_handler, db_user_handler, …), engine/session factory
├ metadata/ One handler per provider; normalizes + ranks by priority
└ filesystem/ ROM/asset/firmware file I/O, hashing, archive extraction
adapters/services/ Typed external API clients (igdb.py + igdb_types.py, screenscraper.py, …)
models/ SQLAlchemy ORM models (BaseModel adds created_at/updated_at)
tasks/ RQ jobs — scheduled/ (cron) and manual/ (on-demand); base classes in tasks.py
config/ Env-var loading (__init__.py) + YAML config manager (singleton)
decorators/ @begin_session (DB session), @protected_route (auth + scopes)
exceptions/ Custom exception hierarchy
utils/ logger/ Shared helpers, structured logging
alembic/ Migrations (env.py + versions/)
```
**Endpoint → handler → (database | metadata | filesystem) → models/adapters.** Endpoints stay thin: validate, enforce scopes, call handlers, serialize via a response schema. Don't put business logic or raw queries in endpoints.
## Conventions
- **Naming:** Classes `PascalCase`; functions/vars `snake_case`; constants `UPPER_SNAKE_CASE`; private `_prefixed`.
- **DB sessions:** decorate handler methods with `@begin_session`; it injects and manages the SQLAlchemy session/transaction. Don't open sessions ad hoc.
- **Async:** I/O-bound endpoints and tasks use `async/await`. Per-request `httpx`/`aiohttp` clients come from context vars (`utils/context.py`), not new clients per call.
- **Imports:** stdlib → third-party → local; explicit (no wildcards); `TYPE_CHECKING` blocks to break circular imports.
- **Errors:** raise the custom exceptions in `exceptions/` (e.g. `RomNotFoundInDatabaseException`), not bare `HTTPException`, where a typed one exists.
- **Validation/SSRF:** sanitize filenames/paths before filesystem use (`utils/`); paths are rooted at `LIBRARY_BASE_PATH`/`RESOURCES_BASE_PATH`/`ASSETS_BASE_PATH` from config.
## Auth & scopes
- Roles: `VIEWER` (read), `EDITOR` (+write roms/platforms/assets), `ADMIN` (+users/tasks/logs). Defined on `models/user.py`; scope tiers in `handler/auth/constants.py`.
- Granular scopes: `me.read/write`, `roms.read/write`, `platforms.*`, `assets.*`, `devices.*`, `firmware.*`, `collections.*`, `users.*`, `tasks.run`, `logs.read`.
- Protect routes with `@protected_route(router.<method>, "<path>", [Scope.X])`. The frontend mirrors these scopes — keep them aligned.
## Adding things
- **Endpoint:** add the route in the right `endpoints/*` router, a response schema in `endpoints/responses/`, enforce scopes, delegate to a handler. If the response shape changes, the frontend must regenerate types (below).
- **Model / schema change:** edit `models/`, then create a migration (below). Update the matching response schema so OpenAPI stays accurate.
- **Metadata provider:** add a typed client in `adapters/services/<name>.py` (+ `<name>_types.py`) and a `handler/metadata/<name>_handler.py` that normalizes into the common shape and slots into the priority order.
- **Background job:** subclass `Task`/`PeriodicTask` in `tasks/scheduled/` or `tasks/manual/`; register scheduled jobs in `startup.py`.
## Database migrations (Alembic)
Migrations must work on **MariaDB, MySQL, and PostgreSQL** (CI runs `alembic upgrade head` on Postgres and MariaDB — `.github/workflows/migrations.yml`). Use batch mode / DB-specific SQL where needed; mirror existing migrations in `alembic/versions/`.
```bash
cd backend
uv run alembic revision --autogenerate -m "short description" # generate, then HAND-REVIEW the file
uv run alembic upgrade head # apply
uv run alembic downgrade -1 # verify the downgrade works
```
Always review autogenerated migrations — they miss server-default/enum/index nuances and cross-dialect differences. The `virtual_collections` DB view is excluded from migrations.
## OpenAPI → frontend types
FastAPI serves the schema at `GET /openapi.json`. The frontend regenerates its TypeScript types from it:
```bash
# backend running on :3000, then in frontend/
npm run generate # writes src/__generated__/ via openapi-typescript-codegen
```
**Any change to a response schema or route signature should be followed by `npm run generate` + a frontend typecheck.**
## Run, test, lint
```bash
cd backend
uv run python3 main.py # run (migrations auto-apply on startup)
uv run pytest [path/file] # tests (subset by path); uv run pytest -vv for all
```
- Tests: pytest + pytest-asyncio, isolated per `pytest-xdist` worker (per-worker DBs); `fakeredis`; `pytest-recording` VCR cassettes mock external APIs; Hypothesis for property tests. Mirror the `backend/<area>/` layout under `backend/tests/`. First-time test DB setup: `docker exec -i romm-db-dev mariadb -uroot -p<pw> < backend/romm_test/setup.sql`.
- **Lint / format / type-check run through Trunk** (ruff, black, isort, mypy, bandit): `trunk fmt && trunk check`. CI enforces Trunk on every PR. Never bypass with `--no-verify`.
- New/changed logic needs a test; new endpoints need endpoint tests.
.claude/skills/frontend-i18n/SKILL.md
+32
View File
@@ -0,0 +1,32 @@
---
name: frontend-i18n
description: Internationalization for the RomM frontend (both v1 and v2). Use whenever adding, renaming, or removing any user-visible string / translation key under frontend/src/locales/. Covers the en_US-is-source rule, the requirement to add every new key to ALL locale directories in the same change, namespace layout, and the check_i18n_locales.py validator enforced in CI. Trigger on any change to frontend/src/locales/**.
---
# RomM Frontend — i18n / Localization
User-visible strings are **never hard-coded** in components — they come from locale files via `vue-i18n` (`$t(...)` in templates/composites; utils may call `i18n.global.t(...)`; **v2 lib primitives must not call `$t` at all** — text passes via props/slots).
## Structure
- Locales live in `frontend/src/locales/<locale>/<namespace>.json`, loaded by dynamic glob import in `src/locales/index.ts`.
- **18 locales**: `en_US` (default + fallback), `en_GB`, `bg_BG`, `cs_CZ`, `de_DE`, `es_ES`, `fr_FR`, `hu_HU`, `it_IT`, `ja_JP`, `ko_KR`, `pl_PL`, `pt_BR`, `ro_RO`, `ru_RU`, `zh_CN`, `zh_TW`.
- Namespaces are per-feature files (e.g. `collection`, `common`, `console`, `detail`, `emulator`, `gallery`, `home`, `library`, `login`, `navigation`, `patcher`, `platform`, `scan`, `settings`, `task`).
## The rule (enforced in CI)
- **`en_US` is the source of truth**, but **every key added to `en_US` must be added to all other locale directories in the same change.** Never leave a key English-only.
- Translate where you can; otherwise copy the English value as a placeholder so the key exists.
- Removing or renaming a key means doing it across **every** locale.
## Verify before handoff
```bash
python3 frontend/src/locales/check_i18n_locales.py
```
It compares every non-English locale against `en_US` and **fails on any missing file, missing key, or extra key**. CI runs the same script (`.github/workflows/i18n.yml`) on any change under `frontend/src/locales/**`. It must pass with zero missing/extra keys.
## Adding a new language
Create a new folder under `frontend/src/locales/` mirroring `en_US/`'s files, then translate. Open the PR against `master` (see the docs/Contributing flow). This is the one i18n change where a new locale directory is expected.
.claude/skills/frontend-v2-components/SKILL.md
+124
View File
@@ -0,0 +1,124 @@
---
name: frontend-v2-components
description: Building or modifying components in the RomM v2 frontend (frontend/src/v2/). Use when creating/editing v2 primitives (R* components in src/v2/lib/), shared composites, or feature composites — covers the three-tier model, file/folder conventions, SFC structure, import order, barrels, Storybook requirements, and v2 anti-patterns. Trigger on any work under frontend/src/v2/.
---
# RomM Frontend v2 — Component Constitution
This governs work inside `frontend/src/v2/`. **v1 is frozen** (`src/views/`, `src/components/`, `src/console/`, `src/layouts/`) — never refactor it; it will be deleted wholesale in a final wave. v2 is gated by `user.ui_settings.uiVersion`.
> Official language for all code, comments, identifiers, `.md`, and commit/PR messages: **English**.
Related skills: `frontend-v2-theming` (tokens/colors), `frontend-v2-input` (focus/gamepad/responsive), `frontend-v2-patterns` (errors/loading/forms/permissions/confirmations), `frontend-i18n`, `pre-pr-verification`.
---
## Premises (stable)
1. **v1 is frozen.** Don't touch `src/views/`, `src/components/`, `src/console/`, `src/layouts/`. When coexistence forces a v2 fork of a store/composable/util, annotate the v1 export with `@deprecated` pointing at the v2 replacement.
2. **Three component tiers** (below).
3. **Shared resources are canonical.** Pinia stores, API services, OpenAPI types (`src/__generated__/`), locales, utils — v2 _imports_ them, never forks them. Additive changes to shared resources are allowed; changing a shared store API to work around a v2 call-site issue is not.
4. **TypeScript strict.** Zero `any` (justify with a comment if unavoidable). No `as unknown as ...`; fix the source or define an intermediate type.
5. **Universal substitution.** When an `R*` primitive exists, use it. If it doesn't, create or extend it. Never drop to raw HTML or raw Vuetify when a primitive applies.
6. **Wrapper contract.** Wrappers around Vuetify use `defineOptions({ inheritAttrs: false })` + `v-bind="$attrs"` + slot passthrough, and accept every prop/slot of the wrapped component.
7. **Layout via Vuetify utility classes + our wrappers, not plain CSS.** Use `d-flex`, `pa-4`, `align-center` directly. Vuetify layout components (`v-row`, `v-col`, `v-container`, `v-spacer`, `v-app`, `v-main`) get wrapped as `R*` on first use (lazy).
8. **Accessibility & performance** are requirements: semantic HTML, focus management, contrast, ARIA on icon-only controls; lazy-load heavy views, virtualize large lists, stable `:key` on every `v-for`.
---
## The three tiers
| Tier | Path | Prefix | Stores/services/router/emitter/i18n | Story | Domain knowledge |
| --------------------- | ------------------------------ | -------------- | ----------------------------------- | --------- | --------------------------------- |
| **Primitive** | `src/v2/lib/` | `R*` mandatory | **No** | Mandatory | None |
| **Shared composite** | `src/v2/components/shared/` | no prefix | Yes | Optional | Cross-feature, no specific domain |
| **Feature composite** | `src/v2/components/<feature>/` | no prefix | Yes | Optional | Feature-specific |
### A component is a primitive only if all three hold
1. Does not depend on stores, services, router, or emitter.
2. No knowledge of a product domain (ROM, Platform, Collection, User…). `RAvatar` yes, `UserAvatar` no.
3. Its API can be described without naming features — generic props/slots/events.
If any fails: **shared composite** if generic across features, **feature composite** if owned by one feature. Edge cases get raised to the user, not auto-decided. Consumer count never demotes a primitive.
### Primitive boundaries
- **Can use**: tokens, other primitives, Vue/Vuetify, generic composables (`useInput*`, `useFocus*`).
- **Cannot use**: Pinia stores, API services, `emitter`, `router` (a `RouterLink` may be accepted as a prop), `i18n` directly. **No `$t()` in primitives** — text comes via props or slots.
---
## File & folder conventions
- **Primitive**: one per folder — `RFoo/RFoo.vue`, `RFoo/RFoo.stories.ts`, `RFoo/index.ts`, optional `RFoo/types.ts`.
- **Composite**: flat `.vue` if one file suffices; a folder with the same internal structure (no story required) if it has sub-pieces.
- **Barrel**: `src/v2/lib/index.ts` re-exports every primitive — update it when a new primitive ships. Composites are imported directly by path; no barrel. No single-file `index.ts` that just re-exports to shorten a path.
### SFC structure
- `<script setup lang="ts">` always.
- `defineOptions({ inheritAttrs: false })` on every wrapper, paired with `v-bind="$attrs"` and slot passthrough (without the bind, attrs vanish silently).
- Props via `defineProps<Props>()` (interface), never runtime declarations. Emits via `defineEmits<{...}>()`. Slots with payload via `defineSlots<{}>()`.
- Order: `<script setup>` → `<template>` → `<style scoped>`. Unscoped `<style>` (teleport overrides only) goes after the scoped block.
### Import order & aliases
```ts
// 1. External
// 2. v2 primitives
import { RBtn, RDialog } from "@v2/lib";
import { computed, ref } from "vue";
import type { SimpleRom } from "@/__generated__";
// 5. Canonical shared resources
import storeAuth from "@/stores/auth";
// 4. v2 feature siblings
import GameCard from "@/v2/components/GameCard.vue";
// 3. v2 composables / shared
import { useCan } from "@/v2/composables/useCan";
```
- `@v2/lib` — primitives barrel. `@/v2/...` — anything else under v2. `@/...` — canonical shared resources. Never relative paths (`../../foo`) when an alias exists.
- Shared v2 types live in `src/v2/types/`; backend types come from `src/__generated__/`; not `src/types/` (legacy).
### Composables
- `use` prefix; single named export from `composables/useFoo/index.ts`; fully typed args/return; no side effects on module load (init on first call). Creating a v2-only composable when a v1 equivalent exists is allowed.
### Console logging
- `console.error` allowed for production-visible errors. `console.log`/`console.warn` must not ship. `console.debug` is dev-only — remove before PR.
---
## Storybook (mandatory for `/lib`)
- Every primitive ships at least one story with controls and at least one variant per theme.
- A new interactive primitive that warrants gamepad navigation ships a `play()` interaction.
- Modified primitive: existing story must still render and its interactions still pass.
- `npm run test` runs Vitest **and** every `/lib` story's `play()` via `composeStories`. Don't duplicate coverage between Vitest (pure logic) and Storybook `play()` (components).
---
## Anti-patterns (beyond what the premises already say)
1. Don't change shared store APIs to work around a v2 call-site issue. (Fix the call site; the Gallery lesson was calling `romsStore.reset()` from the view, not adding `_fetchSeq` to the store.)
2. Don't drop to inline role checks — always go through `useCan` (see `frontend-v2-patterns`).
3. Don't reinvent a surface — dialog/menu/popover/card all go through their primitive; special cases become a new prop, not a parallel surface.
4. Don't use `v-form` directly — use `RForm`.
5. Don't add backwards-compat shims inside v2: delete removed code; no `// removed`, no renamed-but-unused exports, no deprecated wrappers that just call the new function.
6. Don't write redundant tests; don't touch v1; never `--no-verify` on commits.
**Allowed (often misread):** modifying shared stores/services/utils _additively_; creating v2-only composables; importing from `src/__generated__/`.
---
## Known debt (focused follow-ups)
- **Virtualisation migration** — `RVirtualScroller` (`src/v2/lib/structural/`) needs to absorb `GameGrid`/`LetterGroupedGrid` (structural refactor of `Platform.vue`/`Search.vue`/`Collection.vue`); `useLetterGroups` must become index-based for AlphaStrip scroll-spy.
- **`useGalleryFilterUrl`** — sync `galleryFilter` store fields to URL query params for bookmarkable links; mark v1 store usage `@deprecated`.
- **Vue Router scroll restoration** — galleries scroll custom containers (`.r-v2-plat__scroll`), not window; add a Pinia `routeFullPath → offsetTop` map with per-view hooks. Bundle with the virtualisation migration.
- **`useSocketEvent` composable** — typed socket subscriptions with mount/unmount cleanup (consumers currently wire `socket.on/off` by hand).
- **When v1 dies**: move `uiVersion` into `UI_SETTINGS_KEYS`; drop `.r-v2-*` scope classes (tokens move to `:root`); simplify `useUISettings` sync; delete `useGameAnimation`; drop the color-string→tone collapser in `NotificationHost`; remove the Vuetify rule arrays in `stores/users.ts`.
Full reference: `docs/FRONTEND_ARCHITECTURE.md`.
.claude/skills/frontend-v2-input/SKILL.md
+60
View File
@@ -0,0 +1,60 @@
---
name: frontend-v2-input
description: Universal input (mouse, touch, keyboard, gamepad) and responsive/universal-viewport layout in the RomM v2 frontend. Use when adding interactive v2 components, focus management, spatial navigation, gamepad/keyboard handling, modality-gated focus rings, breakpoints, or responsive layout. Covers useInput, focus geometry primitives, the overlay scope stack, useBreakpoint, and the data-bp/data-input attributes. Trigger on interactive or responsive work under frontend/src/v2/.
---
# RomM v2 — Universal Input & Universal Viewport
**Premise:** all v2 UI works with mouse, touch, keyboard, **and** gamepad; and every surface reads cleanly from a 320px phone to a 4K display. Both mechanisms are fixed — don't invent a parallel one.
The input system lives in `src/v2/composables/useInput/` (bus, keyboard, gamepad, actions, scope). It generalises the original gamepad-only `src/console/` system. There are **no `/console/*` routes in v2**.
---
## Modality
`useInputModality` sets `data-input="mouse|touch|key|pad"` on `<html>` from the most recent input.
- **Focus rings appear only with `key` and `pad`** (CSS in `global.css`). **Never use bare `:focus` in styles** — use the modality-gated selectors, or focus rings flash on mouse click.
- Hit targets may scale up for `touch` and `pad`.
## Coverage — every interactive primitive participates
Buttons, list items, tabs, menu items, focusable cards, toggleable chips — all participate in spatial navigation (not optional). A new interactive primitive must:
- be focusable (proper `tabindex`, or already so via a wrapped Vuetify component);
- react to logical actions (confirm/cancel) from `useInput`, in addition to native click;
- show a modality-gated focus state.
Storybook `play()` covering gamepad input is required **only when applicable** (the primitive is interactive enough that gamepad navigation matters).
## Focus geometry
Each view declares its layout with focus primitives: `RFocusZone`, `RFocusGrid`, `RFocusRow`, `RFocusColumn`. Multiple regions = multiple zones. Predictable up/down/left/right movement is the view's responsibility.
## Element-level global shortcuts
Above per-view geometry, some elements bind globally regardless of focus location:
- `UserMenu` opens on **Start**.
- Navbar tabs cycle with **LB/RB** (plus D-pad).
- Context menu opens on **X or Y**.
## Scope (overlay stack)
When a dialog opens, push a scope; when it closes, pop. This stops Escape from closing two things at once and stops `confirm` leaking to controls beneath an overlay. `RDialog` and `RMenu` manage their scope automatically — custom overlays are an anti-pattern; go through the primitives.
---
## Responsive layout (universal viewport)
- **Single breakpoint source: `useBreakpoint`** (`src/v2/composables/useBreakpoint/`). Material thresholds — `xs <600`, `sm 600–959`, `md 960–1279`, `lg 1280–1919`, `xl ≥1920`. `installBreakpointAttribute()` (mounted once in `AppLayout`) mirrors the active set onto `<html data-bp="…">`.
- **Layout switches live in CSS** via the attribute selector: `html[data-bp~="xs"] .foo { … }`, `html[data-bp~="sm-and-down"] .foo { … }`. **No raw `@media` for layout** — the only allowed `@media` are `prefers-reduced-motion` and print. The attribute is on `<html>` so it reaches teleported overlays.
- **Conditional rendering** (mount/unmount a different component per tier) uses the `useBreakpoint()` refs in `<script>` — e.g. `v-if="xs"`. Prefer mount-gating over `display:none` for focusable chrome so hidden controls never sit in the tab/spatial-nav order.
- **`--r-row-pad` is the global horizontal gutter**, already re-scoped responsive in `global.css` (36 → 20 → 14px). Consume `var(--r-row-pad)`; don't hard-code a smaller `xs` padding per component.
- **Touch targets** ≥ `--r-touch-target` (44px) on `xs`/touch. Gate the bump to touch/pad where desktop sizing would bloat.
- **Overlays go full-bleed on `xs`.** `RDialog` renders full-screen / bottom-sheet on phones (`fullscreenOnMobile`, default on); `RMenu` bottom-sheets large menus. Never float a 600px dialog on a 360px screen.
- **Label→icon collapse** (the AppNav precedent) is the canonical way to compress chrome; the four primary destinations relocate to `BottomNav` on `sm-and-down`.
- **Grids** size via `useResponsiveColumns` (ResizeObserver), never a fixed column count.
Verification adds a breakpoint sweep — see `pre-pr-verification`.
.claude/skills/frontend-v2-patterns/SKILL.md
+89
View File
@@ -0,0 +1,89 @@
---
name: frontend-v2-patterns
description: Cross-cutting feature patterns for the RomM v2 frontend — error/snackbar handling, loading & skeleton states, real-time Socket.IO updates, UI state persistence (URL vs localStorage vs ephemeral), pagination/infinite scroll, forms & validation, permissions (useCan), and destructive confirmations. Use when wiring up a v2 feature's behavior (not just its markup). Trigger when implementing data flows, dialogs, forms, toggles, or permission gating under frontend/src/v2/.
---
# RomM v2 — Architecture Patterns
How v2 features behave. Each pattern has one canonical mechanism — don't invent a parallel one.
---
## A. Errors & snackbars
- Single channel: `useSnackbar()` (`src/v2/composables/useSnackbar/`) with `success | error | warning | info` methods. It emits `snackbarShow`; `NotificationHost` stacks toasts.
- The **call site** decides what's significant — no global "wrap-every-promise" magic.
- Field validation errors render **in-place**, never as a snackbar.
- Auth (401/403) is handled by the axios interceptor; no per-call-site checks.
- Successful critical actions → `success` snackbar. Routine optimistic toggles → silent on success, `error` on failure.
- Don't snackbar every rejected promise.
## B. Loading states
- **Skeleton** (`RSkeletonBlock`) for first load of a view with known layout — mimic the real shape so the layout doesn't jump.
- **Inline `:loading` on the control itself** for in-flight actions (`RBtn`, `RTextField`, `RSelect`). Never put an external `RSpinner` next to a button that has its own `loading`.
- **`RSpinner` inline** when what's loading isn't a control with native `loading`.
- **Determinate progress (%)**: use `RProgressLinear` — no raw `v-progress-linear`.
- **Empty state ≠ loading state.** Zero items is its own UX (message, illustration, optional CTA).
- **Optimistic toggles show no spinner**: flip immediately; on failure, revert + snackbar.
- `RBtn` ships `loadingDebounce={200}` — actions resolving under 200ms never paint a spinner; loading→not-loading is immediate.
## C. Real-time updates (Socket.IO)
- One instance: `src/services/socket.ts`. Never `new io()`.
- New consumers go through (or build) a `useSocketEvent(event, handler)` composable for typed subscriptions with automatic mount/unmount cleanup (this composable is still debt — today consumers wire `socket.on/off` by hand).
- **Ownership rule:** state living only while a view is open → subscribe in the view; state that must outlive a view (e.g. scan badge in navbar) → a Pinia store subscribes globally and views just read.
- Reconnection is socket.io's job — don't roll your own.
## D. UI state persistence — three layers
1. **Persistent preferences** (theme, language, gallery defaults like `groupRoms`/`boxartStyle`, Home panels) → `useUISettings` (localStorage + backend `user.ui_settings` two-way sync). Add a key to `UI_SETTINGS_KEYS`.
2. **Bookmarkable session state** (active filters, search query, sort, current tab in detail views) → **URL query params**. Anyone copying the link reproduces what they see. **Active gallery filter must be in URL.**
3. **Ephemeral session state** (open dialog, hover, expansion) → `ref` if local, Pinia store if cross-component within the session.
Don't push state into `useUISettings` "so it persists" — follow the rule above.
## E. Pagination & infinite scroll
- `LoadMore` (`RBtn` + `RSpinner` + IntersectionObserver) is the canonical fallback when virtualization stalls.
- `RVirtualScroller` (`src/v2/lib/structural/`, wrapping `v-virtual-scroll`) is the substrate for large lists/grids.
- Page size lives in the store (`fetchLimit`); not user-configurable for now.
- **Scroll restoration** on back-nav: Vue Router `scrollBehavior` + Pinia in-session offset. URL holds filters/sort/search but **not** scroll offset.
## F. Forms & validation
- Use the **`RForm` primitive** (wraps `v-form`: Enter-to-submit when valid, scroll-to-first-error after a failed `validate()`). **Never use `v-form` directly.**
- **Native Vuetify rules** — no Zod/Yup. Rules are arrays of `(v) => true | string`.
- **Reusable rules** in `src/v2/utils/validation.ts` (`required(msg?)`, `email`, `asciiOnly`, `lengthBetween`, `usernameLength/Chars`, `passwordLength`). Utility code _may_ call `i18n.global.t(...)` (the no-i18n rule covers lib primitives, not utils).
- **Submit pattern:** `await formRef.value?.validate()` before the API call; submit button uses `:loading="submitting"`; errors → snackbar; field errors stay in-place via `:error-messages`.
## G. Permissions
- Action vocabulary `domain.action` (`rom.upload`, `rom.delete`, `library.scan`, `user.create`, `app.admin`) in `src/v2/composables/useCan/actions.ts`.
- Scope vocabulary:
```ts
type PermissionScope =
| { kind: "global" }
| { kind: "platform"; id: number }
| { kind: "collection"; id: number }
| { kind: "rom"; id: number };
```
- **`useCan(action, scope?)`** returns `ComputedRef<boolean>`, reactive to `permissionsStore.grants`. Without scope: "can do this anywhere."
- `stores/permissions.ts` holds normalised grants, hydrated from `authStore.user.role` via the role-map (`installPermissionsHydration()` in `AppLayout`); a future `/permissions/me` will replace it.
- **`v-if`** to hide options a user shouldn't see; **`:disabled`** with tooltip when the option must be visible but blocked.
- **Backend is source of truth** — frontend is a UX hint. Never bypass with inline `user.role === "..."`. All grants are pre-loaded (no `useCanAsync`).
## H. Destructive confirmations
Three friction levels:
- **Low / High** → shared composite `ConfirmDialog` (`components/shared/`) opened via `useConfirm({ title, body, confirmText, tone, requireTyped }) => Promise<boolean>` (mounted once in `GlobalDialogs`).
- **Medium** → a feature composite when the flow needs extra options (e.g. `DeleteRomDialog` with per-item filesystem checkboxes).
Common rules:
- All destruction goes through a dialog — no silent destructive action.
- Confirm button is danger-toned; **focus starts on Cancel**; Enter cancels.
- Success → success snackbar or navigate away, dialog closes. Error → error snackbar, dialog stays open. During action → confirm shows `:loading`, cancel disabled.
- The destructive control respects `useCan(action, scope)`.
- **No "don't ask again."** **Type-to-confirm (`requireTyped`) is required when the action affects the filesystem.**
.claude/skills/frontend-v2-theming/SKILL.md
+79
View File
@@ -0,0 +1,79 @@
---
name: frontend-v2-theming
description: Theming, design tokens, colors, and visual language in the RomM v2 frontend. Use when styling v2 components, picking colors, adding/using CSS variables, working with light/dark themes, or whenever you'd reach for a hex/rgba literal. Covers the token pipeline (src/v2/tokens/index.ts → build:tokens → tokens.css), the .r-v2 scope classes, the zero-hex-literal policy, and shared state semantics. Trigger on any color/theme/token work under frontend/src/v2/.
---
# RomM v2 — Tokens, Theming & Visual Language
**Tokens are the only source of truth for theming. Zero hex/`rgba()` literals in v2 components.** If a value is missing, add a token. Every component must work in both `v2-dark` and `v2-light`.
---
## Token pipeline
`src/v2/tokens/index.ts` is the source. It feeds two consumers:
- **`src/v2/styles/tokens.css`** — _generated_ by `scripts/build-tokens.ts` (`npm run build:tokens`, hooked into `predev`/`prebuild`). **Do not hand-edit.** This is how the vast majority of tokens are consumed: `var(--r-color-...)` in CSS.
- **Direct JS/TS imports** of named exports (`colorCanvas`, `colorCoverArt`, `layout`, …) for the few cases needing a token value in JavaScript — baking colors into an SVG string (`utils/covers`), canvas/QR backgrounds (`Player/Ruffle.vue`, `ShowQRCodeDialog`), and the virtualiser's pixel math (`Gallery/listColumns` reading `layout`).
v2 has **no Vuetify theme of its own.** `tokens.css` emits a palette block per theme under `.r-v2.r-v2-dark` / `.r-v2.r-v2-light`; `RomM.vue` toggles those classes on `<html>`. v2 surfaces never read Vuetify's runtime theme.
> Caveat: a wrapped Vuetify component still resolves `color="primary"` against Vuetify's _own_ registered themes (`src/plugins/vuetify.ts`, sourced from v1's `@/styles/themes`). They mirror the brand tokens by hand (both `#8B74E8`), so they line up, but it's a parallel source. **Prefer `var(--r-...)` over the `color` prop.**
### Adding a new token
1. Add it to `src/v2/tokens/index.ts` with a **semantic, role-based** name (`--r-color-danger`, not `--r-color-red`).
2. Provide **both** dark and light values.
3. Consume via `var(--r-...)` (CSS) or the named export (JS).
4. Run `npm run build:tokens` to regenerate `tokens.css`.
5. If the JS→CSS variable name needs an exception (e.g. `--r-nav-h`), add an entry to `NAME_OVERRIDES` in the generator.
### Where the scope classes live — and why `<html>`
`.r-v2`, `.r-v2-dark`, `.r-v2-light` go on `<html>` (`RomM.vue` toggles them whenever `uiVersion` or the active theme changes). Vuetify teleports overlays (`VDialog`, `VMenu`, `VTooltip`) to `<body> > .v-overlay-container`, **outside** `<v-app>`. Only `<html>` covers both the regular tree and the teleports — without it, overlays lose their tokens.
### Diagnostics — when `var(--r-color-...)` resolves to nothing on an overlay
1. Check `RomM.vue`'s watch on `documentElement.classList` (load-bearing).
2. Check that the teleported component carries a `content-class` tying it back to scope (e.g. `RDialog` uses `content-class="r-dialog"`).
3. **Never** "fix" it by swapping the token for a hex literal — that hides the bug and breaks the dual theme.
---
## Visual language
1. **Single visual vocabulary.** Every surface (dialog, menu, popover, card, toolbar) reads as a sibling — same blur, curvature, depth. No standalone "dialog look" vs "menu look".
2. **Canonical references when designing**: ask the user before consulting `https://mockup.thebirdcage.tv/`. They decide whether the mockup or existing primitives take priority.
3. **State semantics are shared across primitives** (don't reinvent per component):
- hover (neutral, or brand-tinted on selected rows)
- selected/checked (`--r-color-brand-primary`)
- active/favorite (`--r-color-fav`)
- focus (modality-gated; visible only on `key`/`pad` — see `frontend-v2-input`)
- busy/pending · disabled
4. **Implementation gotchas:**
- `RDialog` ships unscoped `<style>` at the bottom that strips Vuetify defaults from `.v-overlay__content`. Load-bearing — without it the `--r-radius-card` corners disappear.
- Every dialog goes through `RDialog`; every menu through `RMenu`.
---
## Color-literal policy: zero exceptions
Outside `src/v2/tokens/index.ts` (the source-of-truth TS module) and the generated `src/v2/styles/tokens.css`, **no hex or `rgba()` literals exist anywhere in v2.** Everything previously "excepted" is now a token or a `color-mix`:
- Cover-overlay glass → `--r-color-overlay-*` (fixed dark glass; never theme-flips).
- Cover artwork placeholder & shimmer → `--r-color-cover-placeholder`, `--r-color-cover-placeholder-bright`.
- Panel / tooltip / shimmer-sweep → `--r-color-panel`, `--r-color-panel-border`, `--r-color-tooltip-bg`, `--r-color-shimmer-sweep`.
- Backdrop scrims (`global.css`) → `color-mix(in srgb, var(--r-color-bg) X%, transparent)`.
- Status tints → `color-mix(in srgb, var(--r-color-status-base-{success,warning,danger,info}) X%, transparent)`.
- Brand-tinted backgrounds (selected rows, focus rings) → `color-mix(in srgb, var(--r-color-brand-primary) X%, transparent)`.
- Black/white shadows → `color-mix(in srgb, black X%, transparent)` (CSS named color, not a hex literal).
- Metadata-provider chips → `--r-color-provider-*`. Player canvas → `--r-color-canvas-bg`, `--r-color-canvas-bg-deep`.
- Emphasis pill (always-white-on-dark "Play" CTA over cover art) → `--r-color-overlay-emphasis-bg/-fg/-bg-hover`.
If a literal would otherwise be needed, the answer is: **add a token** (steps above), then consume via `var(--r-color-...)` or the named export.
## Style conventions
- Scoped `<style>` by default; unscoped only for teleport overrides.
- BEM-ish class names: `.feature__element--modifier`. Prefixes: `.r-v2-...` for app-shell surfaces outside components; `.r-...` for globally shared utilities/tokens.
- No plain CSS where a Vuetify utility class covers the case (`d-flex`, `pa-4`, `align-center`).
.claude/skills/pre-pr-verification/SKILL.md
+65
View File
@@ -0,0 +1,65 @@
---
name: pre-pr-verification
description: The before-handoff / before-PR verification gate for RomM, covering both stacks. Use right before committing, opening a PR, or telling the user a change is done — to run the right static checks, tests, and (for UI) manual browser/theme/input/Storybook checks so CI stays green. Covers frontend (typecheck/lint/test/build/i18n/tokens), backend (pytest/alembic/trunk), and the OpenAPI regen step. Trigger when wrapping up any change.
---
# RomM — Verification Before Handoff
Run the checks that match what you touched. **Static checks don't prove a feature works** — when UI changed, also test it in the browser. Mirror the CI gates so review isn't the first place a failure shows up. **Never `--no-verify`.**
---
## Frontend (`frontend/`)
Run from `frontend/`:
1. `npm run typecheck` — zero errors (`vue-tsc --noEmit`).
2. `npm run lint` _(if present)_ / ESLint clean. Trunk also runs ESLint + Prettier in CI.
3. `npm run test` — zero failures (Vitest + happy-dom; runs unit tests **and** every `/lib` story's `play()` via `composeStories`).
4. `npm run build` — zero failures (CI sanity check).
**If you touched the backend API:** start the backend, run `npm run generate`, then re-`typecheck`.
**If you touched tokens** (`src/v2/tokens/index.ts`): `npm run build:tokens` (also auto-runs on `predev`/`prebuild`) and confirm `tokens.css` regenerated.
**If you touched locales** (`src/locales/**`): `python3 frontend/src/locales/check_i18n_locales.py` must pass with zero missing/extra keys. See the `frontend-i18n` skill.
### UI manual pass (when changes are visible) — v2
With `uiVersion = "v2"`:
- **Golden path + edge cases:** empty, error, loading, no-permission, extreme data; plus nearby regressions.
- **Both themes:** `v2-dark` and `v2-light`.
- **All four input modalities:** mouse, touch, keyboard, gamepad — focus ring only on `key`/`pad`.
- **Responsive sweep:** 320px → 4K across the `useBreakpoint` tiers; overlays full-bleed on `xs`.
- **Accessibility:** contrast, keyboard reachability with no traps, aria-labels on icon-only controls.
- **Performance:** lists/grids of 1000+ items stay smooth; every `v-for` has a stable `:key`.
### Storybook (for `/lib`)
- New primitive → mandatory story with controls + at least one variant per theme; interactive ones get a `play()`.
- Modified primitive → existing story still renders and interactions still pass.
- Don't duplicate coverage between Vitest (pure logic) and Storybook `play()` (components).
---
## Backend (`backend/`)
Run from `backend/`:
1. `uv run pytest [path/file]` — zero failures (run the affected subset, or all with `-vv`).
2. `trunk fmt && trunk check` — ruff/black/isort/mypy/bandit clean (CI enforces Trunk).
3. **If you added a migration:** `uv run alembic upgrade head` then `uv run alembic downgrade -1` to prove both directions; it must work on MariaDB **and** PostgreSQL (CI runs both).
4. **If a response schema or route signature changed:** regenerate frontend types (`npm run generate`) and typecheck the frontend.
---
## CI gates this mirrors
`typecheck.yml` (vue-tsc + lockfile lint), `frontend.yml` (vitest + build), `i18n.yml` (locale check), `pytest.yml` (pytest on MariaDB + PostgreSQL), `migrations.yml` (alembic on both DBs), `trunk-check.yml` (Trunk across the repo). Green locally → green in CI.
## Don't
- Open a PR without manually testing the UI when UI was touched.
- `--no-verify` on commits.
- Leave a locale key English-only, a token un-generated, or a migration one-directional.
.gitattributes
+10
View File
@@ -1 +1,11 @@
**/tests/cassettes/*.yaml linguist-generated
# Keep this file itself in LF so git can parse it reliably on all platforms.
.gitattributes text eol=lf
# Ensure shell scripts always use LF line endings in the repository,
# regardless of the developer's OS or core.autocrlf setting.
# Without this, Windows checkouts produce CRLF scripts that fail to
# execute inside Linux containers with "no such file or directory".
*.sh text eol=lf
docker/init_scripts/init text eol=lf
.github/workflows/copilot-setup-steps.yml
+1 -1
View File
@@ -41,7 +41,7 @@ jobs:
- name: Set up Node.js
uses: actions/[email protected]
with:
node-version: "18"
node-version-file: frontend/.nvmrc
- name: Install frontend dependencies
run: npm install
.github/workflows/frontend.yml
+56
View File
@@ -0,0 +1,56 @@
name: Frontend Checks
on:
pull_request:
paths:
- "frontend/**"
push:
branches:
- "master"
paths:
- "frontend/**"
permissions: read-all
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/[email protected]
- name: Set up Node.js
uses: actions/[email protected]
with:
node-version-file: frontend/.nvmrc
cache: "npm"
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
run: npm ci
working-directory: frontend
- name: Run tests (vitest, includes Storybook play() tests)
run: npm test
working-directory: frontend
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/[email protected]
- name: Set up Node.js
uses: actions/[email protected]
with:
node-version-file: frontend/.nvmrc
cache: "npm"
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
run: npm ci
working-directory: frontend
- name: Build
run: npm run build
working-directory: frontend
.github/workflows/pytest.yml
+8 -2
View File
@@ -71,7 +71,11 @@ jobs:
- name: Initiate MariaDB database
if: matrix.db == 'mariadb'
run: |
mysql --host 127.0.0.1 --port ${{ job.services.mariadb.ports['3306'] }} -uroot -ppasswd -e "GRANT ALL PRIVILEGES ON romm_test.* TO 'romm_test'@'%' WITH GRANT OPTION; FLUSH PRIVILEGES;"
# Grant on the `romm_test%` namespace so the test user can create the
# per-worker databases (romm_test_gw0, ...) used under pytest-xdist,
# without granting it global privileges. The backticks are escaped so
# the shell doesn't treat them as command substitution.
mysql --host 127.0.0.1 --port ${{ job.services.mariadb.ports['3306'] }} -uroot -ppasswd -e "GRANT ALL PRIVILEGES ON \`romm\_test%\`.* TO 'romm_test'@'%' WITH GRANT OPTION; FLUSH PRIVILEGES;"
- name: Run python tests
env:
@@ -80,9 +84,11 @@ jobs:
ROMM_DB_DRIVER: ${{ matrix.db }}
REDIS_HOST: 127.0.0.1
REDIS_PORT: ${{ job.services.valkey.ports['6379'] }}
HYPOTHESIS_PROFILE: ci
run: |
cd backend
uv run pytest -vv --maxfail=10 --junitxml=pytest-report.xml --cov --cov-report xml:coverage.xml --cov-config=.coveragerc .
# GitHub-hosted Linux runners have 4 vCPUs; run one worker per core.
uv run pytest -n 4 -vv --maxfail=10 --junitxml=pytest-report.xml --cov --cov-report xml:coverage.xml --cov-config=.coveragerc .
- name: Publish test results
uses: EnricoMi/publish-unit-test-result-action/[email protected]
.github/workflows/typecheck.yml
+4 -2
View File
@@ -25,10 +25,12 @@ jobs:
- name: Set up Node.js
uses: actions/[email protected]
with:
node-version: "18"
node-version-file: frontend/.nvmrc
cache: "npm"
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
run: npm install
run: npm ci
working-directory: frontend
- name: Run typecheck
.gitignore
+7 -1
View File
@@ -54,6 +54,7 @@ romm_mock
backend/romm_test/resources
backend/romm_test/logs
.pytest_cache
.hypothesis
/romm_test
.coverage
pytest-report.xml
@@ -62,9 +63,14 @@ coverage.xml
# service worker
frontend/dev-dist
# storybook
frontend/storybook-static
# outside data
switch_titledb.json
switch_product_ids.json
# AI tools
.claude
# Keep shared agent skills tracked; ignore personal/local Claude config.
.claude/*
!.claude/skills/
.trunk/setup-ci/action.yaml
+1 -1
View File
@@ -7,7 +7,7 @@ runs:
- name: Setup node
uses: actions/[email protected]
with:
node-version: 18
node-version-file: frontend/.nvmrc
- name: Install dependencies
shell: bash
.trunk/trunk.yaml
+2
View File
@@ -51,6 +51,8 @@ lint:
- frontend/src/__generated__/**
- docker/Dockerfile
- docker/nginx/js/**
# Intentionally malformed YAML, used as a test fixture
- backend/tests/config/fixtures/config/malformed_config.yml
files:
- name: vue
extensions: [vue]
.vscode/tasks.json
Vendored
+9 -3
View File
@@ -3,9 +3,8 @@
"tasks": [
{
"label": "Launch frontend",
"type": "npm",
"path": "frontend",
"script": "dev",
"type": "shell",
"command": "cd frontend && npm run dev",
"problemMatcher": []
},
{
@@ -37,6 +36,13 @@
"dependsOn": ["Launch frontend", "Launch backend"],
"problemMatcher": [],
"dependsOrder": "parallel"
},
{
"label": "Launch Storybook",
"type": "npm",
"path": "frontend",
"script": "storybook",
"problemMatcher": []
}
]
}
CLAUDE.md
+98
View File
@@ -0,0 +1,98 @@
# RomM - Repository Guide for Contributors & Agents
RomM is a self-hosted ROM manager and player: scan a game library off disk, enrich it with metadata from 10+ providers, browse it in a web UI, and play in the browser.
---
## The stack at a glance
| | Backend | Frontend |
| --------- | ------------------------------------------- | ----------------------------------------- |
| Path | `backend/` | `frontend/` |
| Language | Python 3.13+ | TypeScript 5.7 (Vue 3) |
| Framework | FastAPI, SQLAlchemy 2.0, Alembic | Vue 3 + Vite, Vuetify, Pinia, Vue Router |
| Infra | Redis + RQ (jobs/cache/sessions), Socket.IO | vue-i18n, Socket.IO client |
| DB | MariaDB (default), MySQL, PostgreSQL | - |
| Tooling | `uv`, pytest, Trunk (ruff/black/isort/mypy) | `npm`, vue-tsc, ESLint, Vitest, Storybook |
The frontend talks to the backend over `/api/*` (REST) and `/ws` (Socket.IO). TypeScript types are **generated** from the backend's OpenAPI schema into `frontend/src/__generated__/` - the backend is the single source of truth for API shapes.
### Deep-dive references
- **`docs/BACKEND_ARCHITECTURE.md`** - directory map, ER diagram, every endpoint, auth/scopes, tasks.
- **`docs/FRONTEND_ARCHITECTURE.md`** - routing, stores, services, theming, build tooling.
- **`DEVELOPER_SETUP.md`** - Docker and manual local setup (mock library, `.env`, services).
- **`CONTRIBUTING.md`** - contribution flow, **AI-assistance disclosure**, translations.
---
## The frontend has two UIs - know which you're in
- **v1 is frozen.** Everything under `frontend/src/views/`, `src/components/`, `src/console/`, `src/layouts/` is legacy and will be deleted wholesale in a final wave. **Do not refactor v1.** Only touch it for a critical bug, and when a v2 fork exists, mark the v1 export `@deprecated`.
- **v2 is the active rewrite** under `frontend/src/v2/`, gated by `user.ui_settings.uiVersion`. It has its own design system (tokens), primitive library (`R*` components in `src/v2/lib/`), universal input (mouse/touch/keyboard/gamepad), and responsive system. New frontend work goes in v2.
v2 has a detailed constitution, split across focused skills (below). **Read the relevant skill before editing v2 code.**
---
## Skills - load the focused guide for your task
These live in `.claude/skills/` and carry the detailed rules. Invoke the one that matches what you're doing:
| Skill | When |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ |
| `frontend-v2-components` | Building/editing any v2 component - tiers (lib/shared/feature), file & SFC conventions, barrels, anti-patterns. |
| `frontend-v2-theming` | Colors, tokens, light/dark themes, visual language - and the **zero-hex-literal** policy. |
| `frontend-v2-input` | Interactive components, focus/spatial navigation, gamepad/keyboard, breakpoints & responsive layout. |
| `frontend-v2-patterns` | Feature behavior - errors/snackbars, loading, sockets, state persistence, pagination, forms, permissions, destructive confirmations. |
| `frontend-i18n` | Any user-visible string or change under `frontend/src/locales/**`. |
| `backend-development` | Endpoints, handlers, models, schemas, metadata adapters, tasks, migrations under `backend/`. |
| `pre-pr-verification` | Before committing / opening a PR / declaring done - the checks that keep CI green. |
---
## Repo-wide rules
**Disclose AI assistance in the PR.** RomM requires it (see `CONTRIBUTING.md`): state that AI was used and to what extent. This is mandatory and non-negotiable for agent-written contributions.
**Branch off `master`; open PRs against `master`.** Fork → feature branch → PR. Don't push to `master`.
**Linting is via [Trunk](https://trunk.io)** (`trunk fmt && trunk check`) — it wraps ruff, black, isort, mypy, ESLint, Prettier, and more, and runs in CI on every PR. **Never commit with `--no-verify`.**
**The backend owns the API contract.** Changed a response schema or route? Regenerate frontend types (`npm run generate`) and re-typecheck.
**Tests travel with code.** New logic gets a test; new endpoints get endpoint tests; new v2 primitives get a Storybook story (+ `play()` if interactive).
**Verify before handoff.** Don't say "done" on UI work without testing it in the browser in both themes and all input modalities. See `pre-pr-verification`.
**English first.** Outside of language files, all code, comments, identifiers, `.md` files, and commit/PR messages are in English.
**No em-dashes.** Never use em-dashes (—) when writing comments or text. Use commas, parentheses, or separate sentences instead.
**Keep comments short.** Comments should be concise, and focus on the "why" rather than the "what" (the code itself is the "what"). Avoid long paragraphs; break them into multiple lines or sentences.
**Never commit secrets.** Never commit secrets (API keys, passwords, tokens, etc.) to the repo. Use environment variables or secret management tools instead.
**Don't explain a change.** Avoid comments that explain why a change was made to the code. Focus instead on the current behaviour of the code and how it works.
---
## Quick command reference
**Setup:** see `DEVELOPER_SETUP.md`. Docker path is `cp env.template .env` → `docker compose build` → `docker compose up -d` (app at `http://localhost:3000`).
**Backend** (`cd backend`):
```bash
uv sync --all-extras --dev # install
uv run main.py # run (migrations auto-apply)
uv run pytest [path/file] # test (subset) - or -vv for all
uv run alembic revision --autogenerate -m "msg" # new migration (then HAND-REVIEW)
uv run alembic upgrade head # apply migrations
```
**Frontend** (`cd frontend`):
```bash
npm install # install (Node 24)
npm run dev # dev server :3000
npm run typecheck # vue-tsc
npm run test # vitest (+ Storybook play() tests)
npm run build # production build
npm run generate # regenerate types from backend OpenAPI (backend must be running)
npm run build:tokens # regenerate v2 tokens.css (auto on predev/prebuild)
npm run storybook # component library on :6006
python3 src/locales/check_i18n_locales.py # i18n parity check
```
**Lint (both stacks):** `trunk fmt && trunk check`.
Dockerfile
+4 -4
View File
@@ -37,10 +37,10 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
ENV NVM_DIR="/root/.nvm"
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash \
&& . "$NVM_DIR/nvm.sh" \
&& nvm install 24.13.1 \
&& nvm use 24.13.1 \
&& nvm alias default 24.13.1
ENV PATH="$NVM_DIR/versions/node/v24.13.1/bin:$PATH"
&& nvm install 24.16.0 \
&& nvm use 24.16.0 \
&& nvm alias default 24.16.0
ENV PATH="$NVM_DIR/versions/node/v24.16.0/bin:$PATH"
# Build and install RAHasher (optional for RA hashes)
RUN git clone --recursive --branch 1.8.3 --depth 1 https://github.com/RetroAchievements/RALibretro.git /tmp/RALibretro
backend/adapters/services/igdb.py
+4379 -1
View File
File diff suppressed because it is too large Load Diff
backend/adapters/services/libretro_thumbnails.py
+1 -1
View File
@@ -34,7 +34,7 @@ class _AnchorHrefParser(HTMLParser):
# Skip sort/parent/query links exposed by Apache autoindex.
if value.startswith(("?", "/", "#")) or ".." in value:
return
# Only keep image files — libretro stores PNGs.
# Only keep image files, since libretro stores PNGs.
decoded = unquote(value)
if decoded.lower().endswith((".png", ".jpg", ".jpeg", ".webp")):
self.filenames.append(decoded)
backend/adapters/services/mobygames.py
+8 -1
View File
@@ -2,7 +2,7 @@ import asyncio
import http
import json
from collections.abc import Collection
from typing import Literal, overload
from typing import Final, Literal, overload
import aiohttp
import yarl
@@ -14,6 +14,11 @@ from config import MOBYGAMES_API_KEY
from logger.logger import log
from utils import get_version
from utils.context import ctx_aiohttp_session
from utils.rate_limiter import RateLimiter
# MobyGames caps the free/non-commercial tier at 1 request per second.
MOBYGAMES_MAX_REQUESTS_PER_SECOND: Final[float] = 1
_rate_limiter = RateLimiter(MOBYGAMES_MAX_REQUESTS_PER_SECOND)
async def auth_middleware(
@@ -45,6 +50,7 @@ class MobyGamesService:
)
try:
await _rate_limiter.acquire()
res = await aiohttp_session.get(
url,
headers={"user-agent": f"RomM/{get_version()}"},
@@ -85,6 +91,7 @@ class MobyGamesService:
url,
request_timeout,
)
await _rate_limiter.acquire()
res = await aiohttp_session.get(
url,
headers={"user-agent": f"RomM/{get_version()}"},
backend/adapters/services/rahasher.py
+87 -1
View File
@@ -1,5 +1,6 @@
import asyncio
import re
from pathlib import Path
from handler.metadata.base_handler import UniversalPlatformSlug as UPS
from handler.metadata.ra_handler import RAGamesPlatform
@@ -10,6 +11,48 @@ from utils.filesystem import COMPRESSED_FILE_EXTENSIONS
RAHASHER_VALID_HASH_REGEX = re.compile(r"[0-9a-f]{32}")
# Disc descriptor / playlist files that point RAHasher at the real data
# tracks. Handing RAHasher one of these reproduces the hash a shell-expanded
# glob would, in priority order: a per-disc descriptor (.cue/.gdi) wins over
# an .m3u playlist so single-disc folders resolve to their own disc.
RA_DISC_DESCRIPTOR_EXTENSIONS: tuple[str, ...] = (".cue", ".gdi", ".m3u")
def _pick_ra_file(folder: Path) -> Path | None:
"""Pick a single real file to hand RAHasher for a folder-based ROM.
RAHasher is launched without a shell, so it never expands a ``/*`` glob —
it must be given a concrete file. Prefer a disc descriptor
(``.cue``/``.gdi``/``.m3u``), which RAHasher follows to the referenced
tracks; otherwise fall back to the largest file in the folder (a raw
``.iso``/``.bin`` image, or the main file of a multi-file cartridge set).
Returns ``None`` when the folder is missing or holds no files.
"""
if not folder.is_dir():
return None
try:
files = [f for f in folder.iterdir() if f.is_file()]
except (OSError, PermissionError):
return None
if not files:
return None
def _size(p: Path) -> int:
try:
return p.stat().st_size
except (OSError, PermissionError):
return -1
for ext in RA_DISC_DESCRIPTOR_EXTENSIONS:
matches = [f for f in files if f.name.lower().endswith(ext)]
if matches:
picked = max(matches, key=_size)
return picked if _size(picked) >= 0 else None
picked = max(files, key=_size)
return picked if _size(picked) >= 0 else None
# Platforms whose hash algorithm requires an on-disk disc image
# (ISO9660/bin+cue/CHD). When the source file is an archive, RAHasher falls
# back to "buffer hash" mode which these consoles don't support, failing
@@ -109,7 +152,7 @@ class RAHasherService:
# Skip the subprocess entirely when the file is an archive and the
# RA platform needs an on-disk disc image. RAHasher would just spawn,
# fail with "Unsupported console for buffer hash: {id}", and return
# nothing — paying process-spawn overhead per ROM for no result.
# nothing, so we'd pay process-spawn overhead per ROM for no result.
if file_path.lower().endswith(tuple(COMPRESSED_FILE_EXTENSIONS)):
if platform["ra_id"] in RA_BUFFER_HASH_UNSUPPORTED_IDS:
log.debug(
@@ -119,6 +162,49 @@ class RAHasherService:
)
return ""
# For folder-based multi-file ROMs the path ends with /*. RAHasher is
# launched without a shell (create_subprocess_exec), so it never expands
# the glob — it receives the literal "*" and fails ("Could not open
# track/file"). Resolve "/*" to a single real file: hash the largest
# archive directly when the folder holds archives (or skip for disc
# platforms that can't buffer-hash them), otherwise pick a disc
# descriptor / track via _pick_ra_file.
if file_path.endswith("/*"):
folder = Path(file_path[:-2])
def _find_archives() -> list[Path]:
if not folder.is_dir():
return []
return [
f
for f in folder.iterdir()
if f.is_file()
and f.name.lower().endswith(tuple(COMPRESSED_FILE_EXTENSIONS))
]
archive_files = await asyncio.to_thread(_find_archives)
if archive_files:
if platform["ra_id"] in RA_BUFFER_HASH_UNSUPPORTED_IDS:
log.debug(
f"Skipping {hl('RAHasher', color=LIGHTMAGENTA)} for folder "
f"{hl(file_path)}: contains compressed files on disc-based platform"
)
return ""
# Cartridge platform: buffer-hash the largest archive directly
largest = await asyncio.to_thread(
max, archive_files, key=lambda f: f.stat().st_size
)
file_path = str(largest)
else:
# Folder of uncompressed disc tracks (the standard Redump
# .cue + .bin layout, .gdi sets, multi-bin) or a multi-file
# cartridge set. RAHasher never expands the "/*" glob itself,
# so resolve it to a single real file — the disc descriptor
# when present, otherwise the largest track.
resolved = await asyncio.to_thread(_pick_ra_file, folder)
if resolved is not None:
file_path = str(resolved)
log.debug(
f"Executing {hl('RAHasher', color=LIGHTMAGENTA)} for platform: {hl(platform['slug'], color=LIGHTMAGENTA)} - file: {hl(file_path)}"
)
backend/adapters/services/retroachievements.py
+9 -1
View File
@@ -2,7 +2,7 @@ import asyncio
import http
import json
from collections.abc import AsyncIterator
from typing import cast
from typing import Final, cast
import aiohttp
import yarl
@@ -20,6 +20,12 @@ from config import RETROACHIEVEMENTS_API_KEY
from logger.logger import log
from utils import get_version
from utils.context import ctx_aiohttp_session
from utils.rate_limiter import RateLimiter
# RetroAchievements does not publish a fixed limit, try to stay
# within the "fair burst" allowance the API documents
RA_MAX_REQUESTS_PER_SECOND: Final[float] = 4
_rate_limiter = RateLimiter(RA_MAX_REQUESTS_PER_SECOND)
async def auth_middleware(
@@ -53,6 +59,7 @@ class RetroAchievementsService:
request_timeout,
)
try:
await _rate_limiter.acquire()
res = await aiohttp_session.get(
url,
headers={"user-agent": f"RomM/{get_version()}"},
@@ -90,6 +97,7 @@ class RetroAchievementsService:
url,
request_timeout,
)
await _rate_limiter.acquire()
res = await aiohttp_session.get(
url,
headers={"user-agent": f"RomM/{get_version()}"},
backend/adapters/services/screenscraper.py
+114 -37
View File
@@ -14,11 +14,37 @@ from config import SCREENSCRAPER_PASSWORD, SCREENSCRAPER_USER
from logger.logger import log
from utils import get_version
from utils.context import ctx_aiohttp_session
from utils.rate_limiter import ConcurrencyLimiter
SS_DEV_ID: Final = base64.b64decode("enVyZGkxNQ==").decode()
SS_DEV_PASSWORD: Final = base64.b64decode("eFRKd29PRmpPUUc=").decode()
LOGIN_ERROR_CHECK: Final = "Erreur de login"
# ScreenScraper enforces a per-account *thread* (concurrency) cap rather than a
# request rate. Because a request can take several seconds, spacing out request
# starts is not enough, as overlapping requests would exceed the cap and get
# rejected. We instead bound simultaneous in-flight requests.
SS_DEFAULT_MAX_THREADS: Final[int] = 1
_concurrency_limiter = ConcurrencyLimiter(SS_DEFAULT_MAX_THREADS)
def _update_thread_allowance(response: dict) -> None:
"""Raise (or lower) the concurrency cap to the account's advertised threads.
ScreenScraper reports the per-account thread allowance in
``response.ssuser.maxthreads`` (higher for contributors/donors).
"""
try:
max_threads = int(response["response"]["ssuser"]["maxthreads"])
except (AttributeError, KeyError, TypeError, ValueError):
return
if max_threads < 1 or max_threads == _concurrency_limiter.max_concurrency:
return
log.info("ScreenScraper: setting thread allowance to %d", max_threads)
_concurrency_limiter.set_max_concurrency(max_threads)
async def auth_middleware(
req: aiohttp.ClientRequest, handler: aiohttp.ClientHandlerType
@@ -30,8 +56,8 @@ async def auth_middleware(
"devpassword": SS_DEV_PASSWORD,
"output": "json",
"softname": "romm",
"ssid": SCREENSCRAPER_USER,
"sspassword": SCREENSCRAPER_PASSWORD,
"ssid": SCREENSCRAPER_USER or "",
"sspassword": SCREENSCRAPER_PASSWORD or "",
},
)
return await handler(req)
@@ -57,21 +83,24 @@ class ScreenScraperService:
request_timeout,
)
try:
res = await aiohttp_session.get(
url,
headers={"user-agent": f"RomM/{get_version()}"},
middlewares=(auth_middleware,),
timeout=ClientTimeout(total=request_timeout),
)
res.raise_for_status()
res_text = await res.text()
if LOGIN_ERROR_CHECK in res_text:
log.error("Invalid ScreenScraper credentials")
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid ScreenScraper credentials",
async with _concurrency_limiter:
res = await aiohttp_session.get(
url,
headers={"user-agent": f"RomM/{get_version()}"},
middlewares=(auth_middleware,),
timeout=ClientTimeout(total=request_timeout),
)
return await res.json()
res.raise_for_status()
res_text = await res.text()
if LOGIN_ERROR_CHECK in res_text:
log.error("Invalid ScreenScraper credentials")
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid ScreenScraper credentials",
)
data = await res.json()
_update_thread_allowance(data)
return data
except aiohttp.ServerTimeoutError:
# Retry the request once if it times out
pass
@@ -85,10 +114,34 @@ class ScreenScraperService:
) from exc
except aiohttp.ClientResponseError as err:
if err.status == http.HTTPStatus.TOO_MANY_REQUESTS:
# Retry after 2 seconds if rate limit hit
log.warning("ScreenScraper: rate limit hit, retrying after 2s")
await asyncio.sleep(2)
elif err.status == 426:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="ScreenScraper has blacklisted this application version. Please update RomM.",
) from err
elif err.status == 430:
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail="ScreenScraper daily scrape quota exhausted. Try again tomorrow.",
) from err
elif err.status == 431:
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail="ScreenScraper daily unrecognized-ROM quota exhausted. Try again tomorrow.",
) from err
elif err.status == 423:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="ScreenScraper API is currently offline.",
) from err
elif err.status == http.HTTPStatus.UNAUTHORIZED:
log.warning(
"ScreenScraper API is temporarily unavailable (server CPU >60%)"
)
return {}
else:
# Log the error and return an empty dict if the request fails with a different code
log.error(err)
return {}
except json.JSONDecodeError as exc:
@@ -101,27 +154,51 @@ class ScreenScraperService:
url,
request_timeout,
)
res = await aiohttp_session.get(
url,
headers={"user-agent": f"RomM/{get_version()}"},
middlewares=(auth_middleware,),
timeout=ClientTimeout(total=request_timeout),
)
res.raise_for_status()
res_text = await res.text()
if LOGIN_ERROR_CHECK in res_text:
log.error("Invalid ScreenScraper credentials")
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid ScreenScraper credentials",
async with _concurrency_limiter:
res = await aiohttp_session.get(
url,
headers={"user-agent": f"RomM/{get_version()}"},
middlewares=(auth_middleware,),
timeout=ClientTimeout(total=request_timeout),
)
return await res.json()
res.raise_for_status()
res_text = await res.text()
if LOGIN_ERROR_CHECK in res_text:
log.error("Invalid ScreenScraper credentials")
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid ScreenScraper credentials",
)
data = await res.json()
_update_thread_allowance(data)
return data
except (aiohttp.ClientResponseError, aiohttp.ServerTimeoutError) as err:
if (
isinstance(err, aiohttp.ClientResponseError)
and err.status == http.HTTPStatus.UNAUTHORIZED
):
return {}
if isinstance(err, aiohttp.ClientResponseError):
if err.status == http.HTTPStatus.UNAUTHORIZED:
log.warning(
"ScreenScraper API is temporarily unavailable (server CPU >60%)"
)
return {}
elif err.status == 426:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="ScreenScraper has blacklisted this application version. Please update RomM.",
) from err
elif err.status == 430:
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail="ScreenScraper daily scrape quota exhausted. Try again tomorrow.",
) from err
elif err.status == 431:
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail="ScreenScraper daily unrecognized-ROM quota exhausted. Try again tomorrow.",
) from err
elif err.status == 423:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="ScreenScraper API is currently offline.",
) from err
log.error(err)
return {}
backend/adapters/services/screenscraper_types.py
+12 -1
View File
@@ -1,4 +1,15 @@
from typing import Literal, TypedDict
from typing import Literal, NotRequired, TypedDict
# Per-account limits returned in `response.ssuser`
# Reference: https://api.screenscraper.fr/webapi2.php
class SSUser(TypedDict):
id: NotRequired[str]
niveau: NotRequired[str]
maxthreads: NotRequired[str]
maxrequestspermin: NotRequired[str]
maxrequestsperday: NotRequired[str]
requeststoday: NotRequired[str]
class SSText(TypedDict):
backend/alembic/env.py
+9
View File
@@ -41,6 +41,15 @@ def include_object(object, name, type_, reflected, compare_to):
]: # Virtual table
return False
# Skip DB-specific search indexes in autogenerate
# to avoid false drop/create operations
if type_ == "index" and name in (
"idx_roms_name_fs_name_fulltext",
"idx_roms_name_trgm",
"idx_roms_fs_name_trgm",
):
return False
return True
backend/alembic/versions/0080_add_chd_sha1_hash.py
+28
View File
@@ -0,0 +1,28 @@
"""Add chd_sha1_hash to rom_files
Revision ID: 0080_add_chd_sha1_hash
Revises: 0079_add_rom_files_rom_id_index
Create Date: 2026-04-24 00:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
revision = "0080_add_chd_sha1_hash"
down_revision = "0079_add_rom_files_rom_id_index"
branch_labels = None
depends_on = None
def upgrade() -> None:
with op.batch_alter_table("rom_files", schema=None) as batch_op:
batch_op.add_column(
sa.Column("chd_sha1_hash", sa.String(length=100), nullable=True),
if_not_exists=True,
)
def downgrade() -> None:
with op.batch_alter_table("rom_files", schema=None) as batch_op:
batch_op.drop_column("chd_sha1_hash", if_exists=True)
backend/alembic/versions/0081_add_archive_members.py
+30
View File
@@ -0,0 +1,30 @@
"""Add archive_members JSON column to rom_files
Revision ID: 0081_add_archive_members
Revises: 0080_add_chd_sha1_hash
Create Date: 2026-05-28 00:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
from utils.database import CustomJSON
revision = "0081_add_archive_members"
down_revision = "0080_add_chd_sha1_hash"
branch_labels = None
depends_on = None
def upgrade() -> None:
with op.batch_alter_table("rom_files", schema=None) as batch_op:
batch_op.add_column(
sa.Column("archive_members", CustomJSON(), nullable=True),
if_not_exists=True,
)
def downgrade() -> None:
with op.batch_alter_table("rom_files", schema=None) as batch_op:
batch_op.drop_column("archive_members", if_exists=True)
backend/alembic/versions/0082_save_origin_device.py
+43
View File
@@ -0,0 +1,43 @@
"""Track the device that created a save
Revision ID: 0082_save_origin_device
Revises: 0081_add_archive_members
Create Date: 2026-06-05 00:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
revision = "0082_save_origin_device"
down_revision = "0081_add_archive_members"
branch_labels = None
depends_on = None
def upgrade() -> None:
# Order matters on MariaDB: the index must exist before the FK (it backs the
# constraint), so create column -> index -> FK.
with op.batch_alter_table("saves", schema=None) as batch_op:
batch_op.add_column(
sa.Column("origin_device_id", sa.String(length=255), nullable=True),
if_not_exists=True,
)
batch_op.create_index(
"ix_saves_origin_device_id", ["origin_device_id"], if_not_exists=True
)
batch_op.create_foreign_key(
"fk_saves_origin_device_id",
"devices",
["origin_device_id"],
["id"],
ondelete="SET NULL",
)
def downgrade() -> None:
# Drop the FK before the index it backs, then the column.
with op.batch_alter_table("saves", schema=None) as batch_op:
batch_op.drop_constraint("fk_saves_origin_device_id", type_="foreignkey")
batch_op.drop_index("ix_saves_origin_device_id", if_exists=True)
batch_op.drop_column("origin_device_id", if_exists=True)
backend/alembic/versions/0083_rom_category_soundtrack.py
+72
View File
@@ -0,0 +1,72 @@
"""Update rom_file.category column enum to include soundtracks, and add
audio_meta JSON column for tag/duration/cover metadata.
Revision ID: 0083_rom_category_soundtrack
Revises: 0082_save_origin_device
Create Date: 2026-04-17 00:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
from utils.database import CustomJSON, is_postgresql
# revision identifiers, used by Alembic.
revision = "0083_rom_category_soundtrack"
down_revision = "0082_save_origin_device"
branch_labels = None
depends_on = None
def upgrade() -> None:
connection = op.get_bind()
if is_postgresql(connection):
# `ALTER TYPE ... ADD VALUE` must run outside a transaction in PostgreSQL.
# autocommit_block breaks out of alembic's wrapping transaction for
# exactly this operation.
with op.get_context().autocommit_block():
op.execute(
"ALTER TYPE romfilecategory ADD VALUE IF NOT EXISTS 'SOUNDTRACK'"
)
# audio_meta column (added separately so the enum change sticks even
# if the JSON column already exists from a partial prior run).
op.add_column(
"rom_files",
sa.Column("audio_meta", CustomJSON(), nullable=True),
if_not_exists=True,
)
else:
rom_file_category_enum = sa.Enum(
"GAME",
"DLC",
"HACK",
"MANUAL",
"PATCH",
"UPDATE",
"MOD",
"DEMO",
"TRANSLATION",
"PROTOTYPE",
"CHEAT",
"SOUNDTRACK",
name="romfilecategory",
)
with op.batch_alter_table("rom_files", schema=None) as batch_op:
batch_op.alter_column(
"category", type_=rom_file_category_enum, nullable=True
)
batch_op.add_column(
sa.Column("audio_meta", CustomJSON(), nullable=True),
if_not_exists=True,
)
def downgrade() -> None:
# PostgreSQL cannot remove enum values, so on downgrade we only drop the
# audio_meta column and leave SOUNDTRACK in the enum. Any rows that still
# hold the new value keep working; a re-upgrade is a no-op.
with op.batch_alter_table("rom_files", schema=None) as batch_op:
batch_op.drop_column("audio_meta", if_exists=True)
backend/alembic/versions/0084_add_roms_search_index.py
+137
View File
@@ -0,0 +1,137 @@
"""Speed up gallery search and name ordering on the roms table
This migration adds the database structures the gallery relies on to search
and sort large libraries without full table scans:
- Search indexes on roms.name and roms.fs_name, tailored per backend so the
search query stays index-backed: a FULLTEXT index on MySQL/MariaDB, and
pg_trgm GIN indexes on PostgreSQL (installing the pg_trgm extension first).
- A plain idx_roms_name index on roms.name to accelerate name lookups and
range scans.
- A precomputed, indexed name_sort_key column for natural-sort ordering.
Sorting by name previously ran a per-row regexp (strip leading articles,
zero-pad numbers) that no index could cover, forcing a full sort on every
page. The key is now stored on write and backfilled here, so ordering by
name — including deep-offset pages — uses idx_roms_name_sort_key.
downgrade() drops every object created here in reverse order, leaving the
pg_trgm extension in place since other objects may depend on it.
Revision ID: 0084_add_roms_search_index
Revises: 0083_rom_category_soundtrack
Create Date: 2026-06-16 00:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
from models.rom import NAME_SORT_KEY_MAX_LENGTH, compute_name_sort_key
from utils.database import is_mariadb, is_mysql, is_postgresql
# revision identifiers, used by Alembic.
revision = "0084_add_roms_search_index"
down_revision = "0083_rom_category_soundtrack"
branch_labels = None
depends_on = None
FULLTEXT_INDEX_NAME = "idx_roms_name_fs_name_fulltext"
PG_NAME_INDEX = "idx_roms_name_trgm"
PG_FS_NAME_INDEX = "idx_roms_fs_name_trgm"
_BACKFILL_BATCH = 1000
def upgrade() -> None:
bind = op.get_bind()
# 1. DB-specific search indexes on roms.name and roms.fs_name.
if is_mysql(bind) or is_mariadb(bind):
op.execute(
sa.text(
f"CREATE FULLTEXT INDEX {FULLTEXT_INDEX_NAME} "
"ON roms (name, fs_name)"
)
)
elif is_postgresql(bind):
# pg_trgm is a trusted extension since PostgreSQL 13, so a non-superuser
# with CREATE on the database can install it.
op.execute(sa.text("CREATE EXTENSION IF NOT EXISTS pg_trgm"))
op.execute(
sa.text(
f"CREATE INDEX IF NOT EXISTS {PG_NAME_INDEX} "
"ON roms USING gin (name gin_trgm_ops)"
)
)
op.execute(
sa.text(
f"CREATE INDEX IF NOT EXISTS {PG_FS_NAME_INDEX} "
"ON roms USING gin (fs_name gin_trgm_ops)"
)
)
# 2. Plain index on roms.name.
with op.batch_alter_table("roms", schema=None) as batch_op:
batch_op.create_index(
"idx_roms_name",
["name"],
unique=False,
if_not_exists=True,
)
# 3. Precomputed name_sort_key column for natural-sort ordering.
op.add_column(
"roms",
sa.Column(
"name_sort_key",
sa.String(length=NAME_SORT_KEY_MAX_LENGTH),
nullable=True,
),
if_not_exists=True,
)
roms = sa.table(
"roms",
sa.column("id", sa.Integer),
sa.column("name", sa.String),
sa.column("name_sort_key", sa.String),
)
result = bind.execute(sa.select(roms.c.id, roms.c.name))
update_stmt = (
roms.update()
.where(roms.c.id == sa.bindparam("_id"))
.values(name_sort_key=sa.bindparam("_key"))
)
while True:
batch = result.fetchmany(_BACKFILL_BATCH)
if not batch:
break
bind.execute(
update_stmt,
[{"_id": row.id, "_key": compute_name_sort_key(row.name)} for row in batch],
)
op.create_index(
"idx_roms_name_sort_key", "roms", ["name_sort_key"], if_not_exists=True
)
def downgrade() -> None:
bind = op.get_bind()
# 3. name_sort_key column and its index.
op.drop_index("idx_roms_name_sort_key", table_name="roms", if_exists=True)
op.drop_column("roms", "name_sort_key", if_exists=True)
# 2. Plain index on roms.name.
with op.batch_alter_table("roms", schema=None) as batch_op:
batch_op.drop_index("idx_roms_name", if_exists=True)
# 1. DB-specific search indexes.
if is_mysql(bind) or is_mariadb(bind):
op.execute(sa.text(f"DROP INDEX {FULLTEXT_INDEX_NAME} ON roms"))
elif is_postgresql(bind):
# Leave the pg_trgm extension in place; other objects may depend on it.
op.execute(sa.text(f"DROP INDEX IF EXISTS {PG_FS_NAME_INDEX}"))
op.execute(sa.text(f"DROP INDEX IF EXISTS {PG_NAME_INDEX}"))
backend/alembic/versions/0085_rom_category_screenshot.py
+81
View File
@@ -0,0 +1,81 @@
"""Update rom_file.category column enum to include screenshots.
Revision ID: 0085_rom_category_screenshot
Revises: 0084_add_roms_search_index
Create Date: 2026-06-17 00:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
from utils.database import is_postgresql
# revision identifiers, used by Alembic.
revision = "0085_rom_category_screenshot"
down_revision = "0084_add_roms_search_index"
branch_labels = None
depends_on = None
def upgrade() -> None:
connection = op.get_bind()
if is_postgresql(connection):
# `ALTER TYPE ... ADD VALUE` must run outside a transaction in PostgreSQL.
# autocommit_block breaks out of alembic's wrapping transaction for
# exactly this operation.
with op.get_context().autocommit_block():
op.execute(
"ALTER TYPE romfilecategory ADD VALUE IF NOT EXISTS 'SCREENSHOT'"
)
else:
rom_file_category_enum = sa.Enum(
"GAME",
"DLC",
"HACK",
"MANUAL",
"PATCH",
"UPDATE",
"MOD",
"DEMO",
"TRANSLATION",
"PROTOTYPE",
"CHEAT",
"SOUNDTRACK",
"SCREENSHOT",
name="romfilecategory",
)
with op.batch_alter_table("rom_files", schema=None) as batch_op:
batch_op.alter_column(
"category", type_=rom_file_category_enum, nullable=True
)
def downgrade() -> None:
# PostgreSQL cannot remove enum values, so the downgrade is a no-op there.
# On other backends, narrow the enum back to its pre-screenshot set. Any
# rows still holding the new value would block this, but the upload flow
# is additive and a re-upgrade is a no-op.
connection = op.get_bind()
if is_postgresql(connection):
return
rom_file_category_enum = sa.Enum(
"GAME",
"DLC",
"HACK",
"MANUAL",
"PATCH",
"UPDATE",
"MOD",
"DEMO",
"TRANSLATION",
"PROTOTYPE",
"CHEAT",
"SOUNDTRACK",
name="romfilecategory",
)
with op.batch_alter_table("rom_files", schema=None) as batch_op:
batch_op.alter_column("category", type_=rom_file_category_enum, nullable=True)
backend/alembic/versions/0086_screenshot_visibility.py
+49
View File
@@ -0,0 +1,49 @@
"""Add is_gallery + is_public flags to screenshots for per-user community
gallery screenshots (distinct from auto-captured save/state thumbnails).
Revision ID: 0086_screenshot_visibility
Revises: 0085_rom_category_screenshot
Create Date: 2026-06-18 00:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = "0086_screenshot_visibility"
down_revision = "0085_rom_category_screenshot"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column(
"screenshots",
sa.Column(
"is_gallery",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
),
if_not_exists=True,
)
op.add_column(
"screenshots",
sa.Column(
"is_public",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
),
if_not_exists=True,
)
op.create_index(
"idx_screenshots_public", "screenshots", ["is_public"], if_not_exists=True
)
def downgrade() -> None:
op.drop_index("idx_screenshots_public", table_name="screenshots", if_exists=True)
op.drop_column("screenshots", "is_public", if_exists=True)
op.drop_column("screenshots", "is_gallery", if_exists=True)
backend/alembic/versions/0087_save_state_visibility.py
+38
View File
@@ -0,0 +1,38 @@
"""Add is_public flag to saves and states for per-user community sharing
(mirrors screenshots/notes visibility).
Revision ID: 0087_save_state_visibility
Revises: 0086_screenshot_visibility
Create Date: 2026-06-20 00:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = "0087_save_state_visibility"
down_revision = "0086_screenshot_visibility"
branch_labels = None
depends_on = None
def upgrade() -> None:
for table in ("saves", "states"):
op.add_column(
table,
sa.Column(
"is_public",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
),
if_not_exists=True,
)
op.create_index(f"idx_{table}_public", table, ["is_public"], if_not_exists=True)
def downgrade() -> None:
for table in ("saves", "states"):
op.drop_index(f"idx_{table}_public", table_name=table, if_exists=True)
op.drop_column(table, "is_public", if_exists=True)
backend/alembic/versions/0088_devices_client_identifier.py
+36
View File
@@ -0,0 +1,36 @@
"""Add client_device_identifier to devices
Revision ID: 0088_devices_client_identifier
Revises: 0087_save_state_visibility
Create Date: 2026-04-24 00:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = "0088_devices_client_identifier"
down_revision = "0087_save_state_visibility"
branch_labels = None
depends_on = None
def upgrade() -> None:
with op.batch_alter_table("devices", schema=None) as batch_op:
batch_op.add_column(
sa.Column("client_device_identifier", sa.String(length=255), nullable=True),
if_not_exists=True,
)
batch_op.create_index(
"ix_devices_user_client_identifier",
["user_id", "client_device_identifier"],
unique=True,
if_not_exists=True,
)
def downgrade() -> None:
with op.batch_alter_table("devices", schema=None) as batch_op:
batch_op.drop_index("ix_devices_user_client_identifier", if_exists=True)
batch_op.drop_column("client_device_identifier", if_exists=True)
backend/alembic/versions/0089_client_tokens_device_id.py
+45
View File
@@ -0,0 +1,45 @@
"""Add device_id FK to client_tokens
Revision ID: 0089_client_tokens_device_id
Revises: 0088_devices_client_identifier
Create Date: 2026-04-24 00:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = "0089_client_tokens_device_id"
down_revision = "0088_devices_client_identifier"
branch_labels = None
depends_on = None
def upgrade() -> None:
with op.batch_alter_table("client_tokens", schema=None) as batch_op:
batch_op.add_column(
sa.Column("device_id", sa.String(length=255), nullable=True),
if_not_exists=True,
)
batch_op.create_foreign_key(
"fk_client_tokens_device_id",
"devices",
["device_id"],
["id"],
ondelete="SET NULL",
)
batch_op.create_index(
"ix_client_tokens_device_id",
["device_id"],
unique=False,
if_not_exists=True,
)
def downgrade() -> None:
with op.batch_alter_table("client_tokens", schema=None) as batch_op:
# Drop FK before the backing index -- MariaDB refuses otherwise
batch_op.drop_constraint("fk_client_tokens_device_id", type_="foreignkey")
batch_op.drop_index("ix_client_tokens_device_id", if_exists=True)
batch_op.drop_column("device_id", if_exists=True)
backend/config/__init__.py
+8 -3
View File
@@ -89,6 +89,7 @@ REFRESH_RETROACHIEVEMENTS_CACHE_DAYS: Final[int] = safe_int(
)
# LAUNCHBOX
LAUNCHBOX_BASE_PATH: Final[str] = f"{ROMM_BASE_PATH}/launchbox"
LAUNCHBOX_API_ENABLED: Final[bool] = safe_str_to_bool(_get_env("LAUNCHBOX_API_ENABLED"))
# PLAYMATCH
@@ -141,6 +142,9 @@ INVITE_TOKEN_EXPIRY_SECONDS: Final[int] = safe_int(
# OIDC
OIDC_ENABLED: Final[bool] = safe_str_to_bool(_get_env("OIDC_ENABLED"))
OIDC_AUTOLOGIN: Final[bool] = safe_str_to_bool(_get_env("OIDC_AUTOLOGIN"))
OIDC_ALLOW_REGISTRATION: Final[bool] = safe_str_to_bool(
_get_env("OIDC_ALLOW_REGISTRATION", "true")
)
OIDC_PROVIDER: Final[str] = _get_env("OIDC_PROVIDER", "")
OIDC_CLIENT_ID: Final[str] = _get_env("OIDC_CLIENT_ID", "")
OIDC_CLIENT_SECRET: Final[str] = _get_env("OIDC_CLIENT_SECRET", "")
@@ -213,7 +217,7 @@ SCHEDULED_RETROACHIEVEMENTS_PROGRESS_SYNC_CRON: Final[str] = _get_env(
)
# SYNC
SYNC_BASE_PATH: Final[str] = _get_env("SYNC_BASE_PATH", f"{ROMM_BASE_PATH}/sync")
SYNC_BASE_PATH: Final[str] = f"{ROMM_BASE_PATH}/sync"
ENABLE_SYNC_FOLDER_WATCHER: Final[bool] = safe_str_to_bool(
_get_env("ENABLE_SYNC_FOLDER_WATCHER")
)
@@ -226,10 +230,10 @@ SYNC_PUSH_PULL_CRON: Final[str] = _get_env(
"*/30 * * * *", # Every 30 minutes
)
SYNC_SSH_KEYS_PATH: Final[str] = _get_env(
"SYNC_SSH_KEYS_PATH", f"{ROMM_BASE_PATH}/sync/keys"
"SYNC_SSH_KEYS_PATH", f"{SYNC_BASE_PATH}/keys"
)
SYNC_SSH_KNOWN_HOSTS_PATH: Final[str] = _get_env(
"SYNC_SSH_KNOWN_HOSTS_PATH", f"{ROMM_BASE_PATH}/sync/known_hosts"
"SYNC_SSH_KNOWN_HOSTS_PATH", f"{SYNC_BASE_PATH}/known_hosts"
)
# EMULATION
@@ -238,6 +242,7 @@ DISABLE_RUFFLE_RS: Final[bool] = safe_str_to_bool(_get_env("DISABLE_RUFFLE_RS"))
# FRONTEND
KIOSK_MODE: Final[bool] = safe_str_to_bool(_get_env("KIOSK_MODE"))
DISABLE_LOGS_VIEWER: Final[bool] = safe_str_to_bool(_get_env("DISABLE_LOGS_VIEWER"))
# LOGGING
LOGLEVEL: Final[str] = _get_env("LOGLEVEL", "INFO").upper()
backend/config/config_manager.py
+94 -21
View File
@@ -1,7 +1,10 @@
import enum
import functools
import glob
import json
import os
import sys
from pathlib import Path
from typing import Final, NotRequired, TypedDict
import pydash
@@ -49,6 +52,7 @@ DEFAULT_EXCLUDED_FILES: Final = [
]
DEFAULT_EXCLUDED_DIRS: Final = [
"@eaDir",
"assets",
"__MACOSX",
"$RECYCLE.BIN",
".Trash-*",
@@ -69,8 +73,10 @@ class MetadataMediaType(enum.StrEnum):
BEZEL = "bezel"
BOX2D = "box2d"
BOX2D_BACK = "box2d_back"
BOX2D_SIDE = "box2d_side"
BOX3D = "box3d"
MIXIMAGE = "miximage"
MIXIMAGE_V2 = "miximage_v2"
PHYSICAL = "physical"
SCREENSHOT = "screenshot"
TITLE_SCREEN = "title_screen"
@@ -114,7 +120,6 @@ class Config:
ROMS_FOLDER_NAME: str
FIRMWARE_FOLDER_NAME: str
SKIP_HASH_CALCULATION: bool
HIGH_PRIO_STRUCTURE_PATH: str
EJS_DEBUG: bool
EJS_CACHE_LIMIT: int | None
EJS_DISABLE_AUTO_UNLOAD: bool
@@ -133,7 +138,28 @@ class Config:
def __init__(self, **entries):
self.__dict__.update(entries)
self.HIGH_PRIO_STRUCTURE_PATH = f"{LIBRARY_BASE_PATH}/{self.ROMS_FOLDER_NAME}"
@functools.cached_property
def has_structure_path_a(self) -> bool:
# Structure A ({roms_folder}/{platform}) takes priority: if the top-level roms
# folder exists, claim Structure A even if some platform dirs happen to
# contain a {roms_folder} sub-folder.
roms_path = os.path.join(LIBRARY_BASE_PATH, self.ROMS_FOLDER_NAME)
return os.path.isdir(roms_path)
@functools.cached_property
def has_structure_path_b(self) -> bool:
if self.has_structure_path_a:
return False
pattern = os.path.join(
LIBRARY_BASE_PATH, "*", glob.escape(self.ROMS_FOLDER_NAME)
)
for match in glob.iglob(pattern):
if os.path.isdir(match):
return True
return False
class ConfigManager:
@@ -163,14 +189,12 @@ class ConfigManager:
# Check if the config file is mounted
with open(self.config_file, "r") as cf:
self._config_file_mounted = True
self._raw_config = yaml.load(cf, Loader=SafeLoader) or {}
self._raw_config = self._safe_load_yaml(cf)
# Also check if the config file is writable
self._config_file_writable = os.access(self.config_file, os.W_OK)
except FileNotFoundError:
log.critical(
"Config file not found! Any changes made to the configuration will not persist after the application restarts."
)
self._create_missing_config_file()
except PermissionError:
log.warning(
"Config file not writable! Any changes made to the configuration will not persist after the application restarts."
@@ -180,6 +204,41 @@ class ConfigManager:
self._parse_config()
self._validate_config()
def _safe_load_yaml(self, cf) -> dict:
"""Load YAML, falling back to an empty config on syntax errors so the
app can still boot with defaults rather than crashing."""
try:
return yaml.load(cf, Loader=SafeLoader) or {}
except yaml.YAMLError as exc:
log.critical(
f"Failed to parse {hl(self.config_file, BLUE)}: {exc}. "
"Falling back to default configuration, fix the YAML "
"syntax to apply your settings."
)
return {}
def _create_missing_config_file(self) -> None:
log.warning(
f"Config file not found, creating an empty config at {hl(self.config_file, BLUE)}"
)
try:
config_file = Path(self.config_file)
config_file.parent.mkdir(parents=True, exist_ok=True)
config_file.touch(exist_ok=True)
# Reset any previously loaded singleton state so parsing reflects
# the newly created empty config file.
self._raw_config = {}
self._config_file_mounted = True
self._config_file_writable = os.access(self.config_file, os.W_OK)
except PermissionError:
self._config_file_mounted = False
self._config_file_writable = False
log.critical(
"Config file not found and could not be created! Any changes made to the configuration will not persist after the application restarts."
)
@staticmethod
def get_db_engine() -> URL:
"""Builds the database connection string using environment variables
@@ -602,17 +661,25 @@ class ConfigManager:
log.critical("Invalid config.yml: scan.media must be a list")
sys.exit(3)
for media in self.config.SCAN_MEDIA:
if media not in MetadataMediaType:
log.critical(
f"Invalid config.yml: scan.media.{media} is not a valid media type"
)
sys.exit(3)
# Drop unknown media types rather than exiting, since a newer release
# may ship sample configs referencing media types this version doesn't know.
unknown_media = [
m for m in self.config.SCAN_MEDIA if m not in MetadataMediaType
]
if unknown_media:
log.warning(
f"Ignoring unknown values in scan.media: {unknown_media}. "
"These may be from a newer RomM version; update to use them."
)
self.config.SCAN_MEDIA = [
m for m in self.config.SCAN_MEDIA if m in MetadataMediaType
]
valid_thumbnail_options = {
MetadataMediaType.BOX2D,
MetadataMediaType.BOX3D,
MetadataMediaType.MIXIMAGE,
MetadataMediaType.MIXIMAGE_V2,
MetadataMediaType.PHYSICAL,
}
if not isinstance(self.config.GAMELIST_MEDIA_THUMBNAIL, str):
@@ -621,14 +688,17 @@ class ConfigManager:
)
sys.exit(3)
if self.config.GAMELIST_MEDIA_THUMBNAIL not in valid_thumbnail_options:
log.critical(
f"Invalid config.yml: scan.gamelist.media.thumbnail must be one of {valid_thumbnail_options}"
log.warning(
f"Unknown scan.gamelist.media.thumbnail value "
f"{self.config.GAMELIST_MEDIA_THUMBNAIL!r}; falling back to "
f"{MetadataMediaType.BOX2D.value!r}. Valid options: {sorted(o.value for o in valid_thumbnail_options)}."
)
sys.exit(3)
self.config.GAMELIST_MEDIA_THUMBNAIL = MetadataMediaType.BOX2D
valid_image_options = {
MetadataMediaType.TITLE_SCREEN,
MetadataMediaType.MIXIMAGE,
MetadataMediaType.MIXIMAGE_V2,
MetadataMediaType.BOX2D,
MetadataMediaType.SCREENSHOT,
}
@@ -640,15 +710,17 @@ class ConfigManager:
sys.exit(3)
if self.config.GAMELIST_MEDIA_IMAGE not in valid_image_options:
log.critical(
f"Invalid config.yml: scan.gamelist.media.image must be one of {valid_image_options}"
log.warning(
f"Unknown scan.gamelist.media.image value "
f"{self.config.GAMELIST_MEDIA_IMAGE!r}; falling back to "
f"{MetadataMediaType.SCREENSHOT.value!r}. Valid options: {sorted(o.value for o in valid_image_options)}."
)
sys.exit(3)
self.config.GAMELIST_MEDIA_IMAGE = MetadataMediaType.SCREENSHOT
def get_config(self) -> Config:
try:
with open(self.config_file, "r") as config_file:
self._raw_config = yaml.load(config_file, Loader=SafeLoader) or {}
self._raw_config = self._safe_load_yaml(config_file)
except FileNotFoundError:
log.debug("Config file not found!")
@@ -682,6 +754,7 @@ class ConfigManager:
"filesystem": {
"roms_folder": self.config.ROMS_FOLDER_NAME,
"firmware_folder": self.config.FIRMWARE_FOLDER_NAME,
"skip_hash_calculation": self.config.SKIP_HASH_CALCULATION,
},
"system": {
"platforms": self.config.PLATFORMS_BINDING,
@@ -710,8 +783,8 @@ class ConfigManager:
"gamelist": {
"export": self.config.GAMELIST_AUTO_EXPORT_ON_SCAN,
"media": {
"thumbnail": self.config.GAMELIST_MEDIA_THUMBNAIL,
"image": self.config.GAMELIST_MEDIA_IMAGE,
"thumbnail": str(self.config.GAMELIST_MEDIA_THUMBNAIL),
"image": str(self.config.GAMELIST_MEDIA_IMAGE),
},
},
"pegasus": {
backend/conftest.py
+15
View File
@@ -0,0 +1,15 @@
import os
# When running under pytest-xdist, give each worker its own database so the
# autouse `clear_database` fixture in one worker can't wipe rows another worker
# is mid-test with. This must run before any application module (config /
# database handlers) is imported, so the engine built at import time binds to
# the per-worker name. As the rootdir conftest, this file is imported before
# `tests/conftest.py` (which imports those modules).
#
# The Redis cache needs no equivalent handling: under pytest it is an in-process
# FakeRedis, so each worker process is already isolated.
_xdist_worker = os.environ.get("PYTEST_XDIST_WORKER")
if _xdist_worker:
_base_db_name = os.environ.get("DB_NAME", "romm_test")
os.environ["DB_NAME"] = f"{_base_db_name}_{_xdist_worker}"
backend/decorators/database.py
+2 -1
View File
@@ -10,7 +10,8 @@ from logger.logger import log
def begin_session(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
if hasattr(kwargs, "session"):
# Reuse a caller-provided session so the handler can join an existing unit of work
if kwargs.get("session") is not None:
return func(*args, **kwargs)
try:
backend/docker-compose.test.yml
+37
View File
@@ -0,0 +1,37 @@
# Local test infrastructure matching .github/workflows/pytest.yml.
# Usage: docker compose -f docker-compose.test.yml up -d
# uv run pytest
services:
mariadb:
image: mariadb:10.11
ports:
- "3306:3306"
environment:
MYSQL_USER: romm_test
MYSQL_PASSWORD: passwd
MYSQL_DATABASE: romm_test
MYSQL_ROOT_PASSWORD: passwd
healthcheck:
test:
[
"CMD",
"mariadb-admin",
"ping",
"-h",
"127.0.0.1",
"-uroot",
"-ppasswd",
]
interval: 5s
timeout: 2s
retries: 10
valkey:
image: valkey/valkey:7.2
ports:
- "6379:6379"
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 2s
retries: 10
backend/endpoints/activity.py
+137
View File
@@ -0,0 +1,137 @@
from datetime import datetime, timezone
from fastapi import HTTPException, Request, status
from pydantic import BaseModel, Field
from decorators.auth import protected_route
from endpoints.responses.activity import ActivityClearSchema, ActivityEntrySchema
from handler.activity_handler import ActivityEntry, activity_handler
from handler.auth.constants import Scope
from handler.database import db_device_handler, db_rom_handler, db_save_handler
from handler.socket_handler import socket_handler
from logger.logger import log
from utils.router import APIRouter
from utils.screenshots import continue_playing_screenshot
router = APIRouter(
prefix="/activity",
tags=["activity"],
)
class DeviceHeartbeatPayload(BaseModel):
rom_id: int = Field(ge=1)
device_id: str = Field(min_length=1, max_length=255)
@protected_route(router.get, "", [Scope.ROMS_USER_READ])
async def get_all_activity(request: Request) -> list[ActivityEntrySchema]:
"""Return every currently active play session across all users."""
entries = await activity_handler.get_all_active()
return [ActivityEntrySchema(**e) for e in entries]
@protected_route(router.get, "/rom/{rom_id}", [Scope.ROMS_USER_READ])
async def get_rom_activity(request: Request, rom_id: int) -> list[ActivityEntrySchema]:
"""Return all active play sessions for a specific ROM."""
entries = await activity_handler.get_active_for_rom(rom_id)
return [ActivityEntrySchema(**e) for e in entries]
@protected_route(router.post, "/heartbeat", [Scope.ROMS_USER_WRITE])
async def device_heartbeat(
request: Request, payload: DeviceHeartbeatPayload
) -> ActivityEntrySchema:
"""Heartbeat endpoint for external devices (muOS, Android, etc.).
Called periodically by devices while the user is playing a game. Writes
activity state to Redis and broadcasts an ``activity:update`` event over
the main Socket.IO namespace.
"""
rom = db_rom_handler.get_rom(payload.rom_id)
if rom is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"ROM {payload.rom_id} not found",
)
device = db_device_handler.get_device(
device_id=payload.device_id, user_id=request.user.id
)
if device is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Device {payload.device_id} not found for this user",
)
# Preserve the started_at from the existing entry if we are refreshing.
existing = await activity_handler.get_active(request.user.id, device.id)
started_at = (
existing["started_at"] if existing else datetime.now(timezone.utc).isoformat()
)
latest_save = db_save_handler.get_latest_saves_for_roms(
user_id=request.user.id, rom_ids=[rom.id]
).get(rom.id)
screenshot_path = continue_playing_screenshot(rom, latest_save) or ""
platform = rom.platform
entry = ActivityEntry(
user_id=request.user.id,
username=request.user.username,
avatar_path=request.user.avatar_path or "",
rom_id=rom.id,
rom_name=rom.name or rom.fs_name,
rom_cover_path=rom.path_cover_s or "",
screenshot_path=screenshot_path,
platform_slug=platform.slug if platform else "",
platform_name=(platform.custom_name or platform.name) if platform else "",
device_id=device.id,
device_type=device.client or "unknown",
started_at=started_at,
)
await activity_handler.set_active(entry)
# Update the device last_seen as a side-effect (mirrors play session ingest).
db_device_handler.update_last_seen(device_id=device.id, user_id=request.user.id)
# Broadcast to all connected sockets. The REST app shares this process with
# the Socket.IO server, so emit through the already-initialised, Redis-backed
# server (it fans out across workers) rather than opening a manager per call.
try:
await socket_handler.socket_server.emit("activity:update", dict(entry))
except Exception as e: # noqa: BLE001
log.warning(
f"Failed to broadcast activity:update for user {request.user.id}: {e}"
)
return ActivityEntrySchema(**entry)
@protected_route(
router.delete,
"/heartbeat",
[Scope.ROMS_USER_WRITE],
status_code=status.HTTP_204_NO_CONTENT,
)
async def clear_device_activity(request: Request, device_id: str) -> None:
"""Immediately clear an active session for a device (e.g. on graceful exit)."""
rom_id = await activity_handler.clear_active(request.user.id, device_id)
if rom_id is None:
return None
try:
await socket_handler.socket_server.emit(
"activity:clear",
ActivityClearSchema(
user_id=request.user.id,
device_id=device_id,
rom_id=rom_id,
).model_dump(),
)
except Exception as e: # noqa: BLE001
log.warning(
f"Failed to broadcast activity:clear for user {request.user.id}: {e}"
)
return None
backend/endpoints/collections.py
+66 -2
View File
@@ -6,6 +6,7 @@ from typing import Annotated
from fastapi import File, Form, HTTPException
from fastapi import Path as PathVar
from fastapi import Query, Request, UploadFile, status
from pydantic import BaseModel as PydanticBaseModel
from decorators.auth import protected_route
from endpoints.responses.collection import (
@@ -21,6 +22,7 @@ from exceptions.endpoint_exceptions import (
from handler.auth.constants import Scope
from handler.database import db_collection_handler
from handler.filesystem import fs_resource_handler
from handler.filesystem.assets_handler import validate_image_upload
from handler.filesystem.base_handler import CoverSize
from logger.formatter import BLUE
from logger.formatter import highlight as hl
@@ -81,7 +83,7 @@ async def add_collection(
try:
if artwork is not None and artwork.filename is not None:
file_ext = artwork.filename.split(".")[-1]
file_ext = validate_image_upload(artwork, label="Artwork")
artwork_content = BytesIO(await artwork.read())
(
path_cover_l,
@@ -426,7 +428,7 @@ async def update_collection(
cleaned_data.update({"url_cover": ""})
else:
if artwork is not None and artwork.filename is not None:
file_ext = artwork.filename.split(".")[-1]
file_ext = validate_image_upload(artwork, label="Artwork")
artwork_content = BytesIO(await artwork.read())
(
path_cover_l,
@@ -474,6 +476,68 @@ async def update_collection(
return CollectionSchema.model_validate(updated_collection)
class CollectionRomsPayload(PydanticBaseModel):
rom_ids: list[int]
@protected_route(router.post, "/{id}/roms", [Scope.COLLECTIONS_WRITE])
async def add_roms_to_collection(
request: Request,
id: int,
payload: CollectionRomsPayload,
) -> CollectionSchema:
"""Atomically add ROMs to a collection without replacing the full list.
Args:
request (Request): Fastapi Request object
id (int): Collection id
payload (CollectionRomsPayload): ROM IDs to add
Returns:
CollectionSchema: Updated collection
"""
collection = db_collection_handler.get_collection(id)
if not collection:
raise CollectionNotFoundInDatabaseException(id)
if collection.user_id != request.user.id:
raise CollectionPermissionError(id)
updated_collection = db_collection_handler.add_roms_to_collection(
id, payload.rom_ids
)
return CollectionSchema.model_validate(updated_collection)
@protected_route(router.delete, "/{id}/roms", [Scope.COLLECTIONS_WRITE])
async def remove_roms_from_collection(
request: Request,
id: int,
payload: CollectionRomsPayload,
) -> CollectionSchema:
"""Atomically remove ROMs from a collection without replacing the full list.
Args:
request (Request): Fastapi Request object
id (int): Collection id
payload (CollectionRomsPayload): ROM IDs to remove
Returns:
CollectionSchema: Updated collection
"""
collection = db_collection_handler.get_collection(id)
if not collection:
raise CollectionNotFoundInDatabaseException(id)
if collection.user_id != request.user.id:
raise CollectionPermissionError(id)
updated_collection = db_collection_handler.remove_roms_from_collection(
id, payload.rom_ids
)
return CollectionSchema.model_validate(updated_collection)
@protected_route(router.put, "/smart/{id}", [Scope.COLLECTIONS_WRITE])
async def update_smart_collection(
request: Request,
backend/endpoints/device.py
+2 -2
View File
@@ -8,7 +8,7 @@ from decorators.auth import protected_route
from endpoints.responses.device import DeviceCreateResponse, DeviceSchema
from handler.auth.constants import Scope
from handler.database import db_device_handler, db_device_save_sync_handler
from handler.filesystem import fs_sync_handler
from handler.filesystem import get_fs_sync_handler
from logger.logger import log
from models.device import Device, SyncMode
from utils.router import APIRouter
@@ -123,7 +123,7 @@ def register_device(
# Auto-create sync folders for file_transfer devices
if payload.sync_mode == SyncMode.FILE_TRANSFER:
try:
fs_sync_handler.ensure_device_directories(device_id)
get_fs_sync_handler().ensure_device_directories(device_id)
except Exception:
log.warning(f"Failed to create sync directories for device {device_id}")
backend/endpoints/device_auth.py
+344
View File
@@ -0,0 +1,344 @@
"""Device authorization flow endpoints (RFC 8628-style, tailored for RomM).
A device POSTs to /authorize with its metadata and requested scopes; the server
returns a device_code (secret, for polling) and a user_code (short, for QR).
The user scans the QR on their phone, lands on /pair/device in the web UI,
approves (possibly editing scopes and device name), and the device's next poll
on /token returns a ClientToken bound 1:1 to a Device record.
The flow is unauthenticated on /authorize and /token (the whole point — the
device has no credentials yet). State lives exclusively in Redis with a hard
10-minute TTL ceiling.
"""
import uuid
from datetime import datetime, timezone
from fastapi import HTTPException, Request, status
from decorators.auth import protected_route
from endpoints.responses.device_auth import (
DeviceAuthApprovePayload,
DeviceAuthApproveResponse,
DeviceAuthDenyPayload,
DeviceAuthInitPayload,
DeviceAuthInitResponse,
DeviceAuthPendingSchema,
DeviceAuthTokenPayload,
DeviceAuthTokenResponse,
)
from handler.auth import auth_handler
from handler.auth.constants import Scope
from handler.database import db_client_token_handler, db_device_handler
from handler.database.base_handler import sync_session
from logger.logger import log
from models.client_token import ClientToken
from models.device import Device, SyncMode
from utils.client_tokens import parse_expiry
from utils.device_auth import (
PENDING_TTL_SECONDS,
POLL_DEFAULT_INTERVAL_SECONDS,
FlowStatus,
build_verification_paths,
check_authorize_rate_limit,
check_token_poll_rate_limit,
consume_approved,
generate_device_code,
generate_user_code,
load_pending,
mark_approved,
mark_denied,
normalize_user_code,
pending_expires_at,
polled_too_fast,
resolve_device_code_from_user_code,
store_pending,
)
from utils.router import APIRouter
router = APIRouter(prefix="/auth/device", tags=["device-auth"])
def _device_code_prefix(device_code: str) -> str:
"""Short prefix for log lines — never log the full secret."""
return device_code[:8]
@router.post("/init", status_code=status.HTTP_201_CREATED)
def device_auth_init(
request: Request, payload: DeviceAuthInitPayload
) -> DeviceAuthInitResponse:
"""Device-initiated: start a new pairing flow. Open endpoint, rate-limited."""
check_authorize_rate_limit(request)
device_code = generate_device_code()
user_code = generate_user_code()
store_pending(
device_code,
user_code,
{
"client_device_identifier": payload.client_device_identifier,
"name": payload.name,
"client": payload.client,
"platform": payload.platform,
"client_version": payload.client_version,
"requested_scopes": payload.requested_scopes,
"interval": POLL_DEFAULT_INTERVAL_SECONDS,
},
)
verification_path, verification_path_complete = build_verification_paths(user_code)
log.info(
f"device_auth.init client={payload.client} "
f"identifier={payload.client_device_identifier} "
f"device_code_prefix={_device_code_prefix(device_code)}"
)
return DeviceAuthInitResponse(
device_code=device_code,
user_code=user_code,
verification_path=verification_path,
verification_path_complete=verification_path_complete,
expires_in=PENDING_TTL_SECONDS,
interval=POLL_DEFAULT_INTERVAL_SECONDS,
)
@protected_route(router.get, "/pending/{user_code}", [Scope.ME_READ])
def get_pending(request: Request, user_code: str) -> DeviceAuthPendingSchema:
"""Fetch a pending request's metadata for the approval screen."""
normalized = normalize_user_code(user_code)
device_code = resolve_device_code_from_user_code(normalized)
if device_code is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Unknown or expired code",
)
data = load_pending(device_code)
if data is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Unknown or expired code",
)
if data.get("status") != FlowStatus.PENDING:
raise HTTPException(
status_code=status.HTTP_410_GONE,
detail=f"Code already {data.get('status')}",
)
user_scopes = {str(s) for s in request.user.oauth_scopes}
requested = list(data.get("requested_scopes", []))
allowed = sorted(set(requested) & user_scopes)
return DeviceAuthPendingSchema(
client_device_identifier=data["client_device_identifier"],
name=data["name"],
client=data["client"],
platform=data.get("platform"),
client_version=data.get("client_version"),
requested_scopes=sorted(requested),
allowed_scopes=allowed,
expires_at=pending_expires_at(device_code),
)
@protected_route(router.post, "/approve", [Scope.ME_WRITE])
def approve(
request: Request, payload: DeviceAuthApprovePayload
) -> DeviceAuthApproveResponse:
"""Create the Device + bound ClientToken and mark the request approved."""
normalized = normalize_user_code(payload.user_code)
device_code = resolve_device_code_from_user_code(normalized)
if device_code is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Unknown or expired code",
)
data = load_pending(device_code)
if data is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Unknown or expired code",
)
if data.get("status") != FlowStatus.PENDING:
raise HTTPException(
status_code=status.HTTP_410_GONE,
detail=f"Code already {data.get('status')}",
)
user_scopes = {str(s) for s in request.user.oauth_scopes}
requested = set(data.get("requested_scopes", []))
allowed = requested & user_scopes
approved_set = set(payload.approved_scopes)
if not approved_set or not approved_set.issubset(allowed):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Approved scopes exceed what's allowed for this user",
)
expires_at = parse_expiry(payload.expires_in)
now = datetime.now(timezone.utc)
device_name = payload.device_name or data["name"]
raw_token = auth_handler.generate_client_token()
# Create/update the Device and its bound ClientToken in a single unit of work
with sync_session.begin() as session:
existing = db_device_handler.get_device_by_client_identifier(
user_id=request.user.id,
client_device_identifier=data["client_device_identifier"],
session=session,
)
if existing is not None:
update_data = {
"name": device_name,
"last_seen": now,
}
client_version = data.get("client_version")
if client_version is not None:
update_data["client_version"] = client_version
device = (
db_device_handler.update_device(
device_id=existing.id,
user_id=request.user.id,
data=update_data,
session=session,
)
or existing
)
else:
device = db_device_handler.add_device(
Device(
id=str(uuid.uuid4()),
user_id=request.user.id,
name=device_name,
client=data.get("client"),
platform=data.get("platform"),
client_version=data.get("client_version"),
client_device_identifier=data["client_device_identifier"],
sync_mode=SyncMode.API,
last_seen=now,
),
session=session,
)
token = db_client_token_handler.add_token(
ClientToken(
user_id=request.user.id,
name=device_name,
hashed_token=auth_handler.hash_client_token(raw_token),
scopes=" ".join(sorted(approved_set)),
expires_at=expires_at,
device_id=device.id,
),
session=session,
)
mark_approved(
device_code,
raw_token=raw_token,
device_id=device.id,
scopes=sorted(approved_set),
expires_at=token.expires_at,
)
log.info(
f"device_auth.approve user_id={request.user.id} "
f"device_id={device.id} token_id={token.id} "
f"device_code_prefix={_device_code_prefix(device_code)} "
f"scopes={' '.join(sorted(approved_set))}"
)
return DeviceAuthApproveResponse(device_id=device.id, device_name=device_name)
@protected_route(router.post, "/deny", [Scope.ME_WRITE])
def deny(request: Request, payload: DeviceAuthDenyPayload) -> None:
normalized = normalize_user_code(payload.user_code)
device_code = resolve_device_code_from_user_code(normalized)
if device_code is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Unknown or expired code",
)
data = load_pending(device_code)
if data is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Unknown or expired code",
)
if data.get("status") != FlowStatus.PENDING:
raise HTTPException(
status_code=status.HTTP_410_GONE,
detail=f"Code already {data.get('status')}",
)
mark_denied(device_code)
log.info(
f"device_auth.deny user_id={request.user.id} "
f"device_code_prefix={_device_code_prefix(device_code)}"
)
@router.post("/token")
def token(request: Request, payload: DeviceAuthTokenPayload) -> DeviceAuthTokenResponse:
"""Device-facing polling endpoint. Open, rate-limited per-IP and per-code."""
check_token_poll_rate_limit(request)
data = load_pending(payload.device_code)
if data is None:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="expired_token",
)
state = data.get("status")
if state == FlowStatus.PENDING:
interval = int(data.get("interval", POLL_DEFAULT_INTERVAL_SECONDS))
if polled_too_fast(payload.device_code, interval):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="slow_down",
)
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="authorization_pending",
)
if state == FlowStatus.DENIED:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="access_denied",
)
if state == FlowStatus.APPROVED:
approved = consume_approved(payload.device_code)
if approved is None:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="expired_token",
)
expires_at_raw = approved.get("expires_at")
expires_at = datetime.fromisoformat(expires_at_raw) if expires_at_raw else None
log.info(
f"device_auth.token_issued device_id={approved['device_id']} "
f"device_code_prefix={_device_code_prefix(payload.device_code)}"
)
return DeviceAuthTokenResponse(
access_token=approved["raw_token"],
device_id=approved["device_id"],
scopes=list(approved.get("scopes", [])),
expires_at=expires_at,
)
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="expired_token",
)
backend/endpoints/feeds.py
+26 -10
View File
@@ -40,7 +40,6 @@ from endpoints.responses.feeds import (
from handler.auth.constants import Scope
from handler.database import db_platform_handler, db_rom_handler
from handler.filesystem import fs_rom_handler
from handler.filesystem.roms_handler import is_compressed_file
from handler.metadata import meta_igdb_handler
from handler.metadata.base_handler import (
SONY_SERIAL_REGEX,
@@ -49,6 +48,7 @@ from handler.metadata.base_handler import (
)
from handler.metadata.base_handler import UniversalPlatformSlug as UPS
from models.rom import Rom, RomFile, RomFileCategory
from utils.archives import is_compressed_file
from utils.router import APIRouter
@@ -213,7 +213,7 @@ async def tinfoil_index_feed(
return titledb
roms = db_rom_handler.get_roms_scalar(platform_ids=[switch.id])
roms = db_rom_handler.get_roms_scalar(platform_ids=[switch.id], include_files=True)
return TinfoilFeedSchema(
files=[
@@ -329,7 +329,9 @@ def pkgi_ps3_feed(
status_code=400, detail=f"Invalid content type: {content_type}"
) from e
roms = db_rom_handler.get_roms_scalar(platform_ids=[ps3_platform.id])
roms = db_rom_handler.get_roms_scalar(
platform_ids=[ps3_platform.id], include_files=True
)
txt_lines = []
for rom in roms:
@@ -405,7 +407,9 @@ def pkgi_psvita_feed(
status_code=400, detail=f"Invalid content type: {content_type}"
) from e
roms = db_rom_handler.get_roms_scalar(platform_ids=[psvita_platform.id])
roms = db_rom_handler.get_roms_scalar(
platform_ids=[psvita_platform.id], include_files=True
)
txt_lines = []
for rom in roms:
@@ -480,7 +484,9 @@ def pkgi_psp_feed(
status_code=400, detail=f"Invalid content type: {content_type}"
) from e
roms = db_rom_handler.get_roms_scalar(platform_ids=[psp_platform.id])
roms = db_rom_handler.get_roms_scalar(
platform_ids=[psp_platform.id], include_files=True
)
txt_lines = []
for rom in roms:
@@ -663,7 +669,9 @@ def pkgj_psp_games_feed(request: Request) -> Response:
status_code=404, detail="PlayStation Portable platform not found"
)
roms = db_rom_handler.get_roms_scalar(platform_ids=[platform.id])
roms = db_rom_handler.get_roms_scalar(
platform_ids=[platform.id], include_files=True
)
txt_lines = []
txt_lines.append(
"Title ID\tRegion\tType\tName\tPKG direct link\tContent ID\tLast Modification Date\tRAP\tDownload .RAP file\tFile Size\tSHA256"
@@ -727,7 +735,9 @@ def pkgj_psp_dlcs_feed(request: Request) -> Response:
status_code=404, detail="PlayStation Portable platform not found"
)
roms = db_rom_handler.get_roms_scalar(platform_ids=[platform.id])
roms = db_rom_handler.get_roms_scalar(
platform_ids=[platform.id], include_files=True
)
txt_lines = []
txt_lines.append(
"Title ID\tRegion\tName\tPKG direct link\tContent ID\tLast Modification Date\tRAP\tDownload .RAP file\tFile Size\tSHA256"
@@ -789,7 +799,9 @@ def pkgj_psv_games_feed(request: Request) -> Response:
status_code=404, detail="PlayStation Vita platform not found"
)
roms = db_rom_handler.get_roms_scalar(platform_ids=[platform.id])
roms = db_rom_handler.get_roms_scalar(
platform_ids=[platform.id], include_files=True
)
txt_lines = []
txt_lines.append(
"Title ID\tRegion\tName\tPKG direct link\tzRIF\tContent ID\tLast Modification Date\tOriginal Name\tFile Size\tSHA256\tRequired FW\tApp Version"
@@ -854,7 +866,9 @@ def pkgj_psv_dlcs_feed(request: Request) -> Response:
status_code=404, detail="PlayStation Vita platform not found"
)
roms = db_rom_handler.get_roms_scalar(platform_ids=[platform.id])
roms = db_rom_handler.get_roms_scalar(
platform_ids=[platform.id], include_files=True
)
txt_lines = []
txt_lines.append(
"Title ID\tRegion\tName\tPKG direct link\tzRIF\tContent ID\tLast Modification Date\tFile Size\tSHA256"
@@ -911,7 +925,9 @@ def pkgj_psx_games_feed(request: Request) -> Response:
if not platform:
raise HTTPException(status_code=404, detail="PlayStation platform not found")
roms = db_rom_handler.get_roms_scalar(platform_ids=[platform.id])
roms = db_rom_handler.get_roms_scalar(
platform_ids=[platform.id], include_files=True
)
txt_lines = []
txt_lines.append(
"Title ID\tRegion\tName\tPKG direct link\tContent ID\tLast Modification Date\tOriginal Name\tFile Size\tSHA256"
backend/endpoints/heartbeat.py
+6 -1
View File
@@ -5,6 +5,7 @@ from fastapi import HTTPException, Request, status
from config import (
DISABLE_EMULATOR_JS,
DISABLE_LOGS_VIEWER,
DISABLE_RUFFLE_RS,
DISABLE_SETUP_WIZARD,
DISABLE_USERPASS_LOGIN,
@@ -30,6 +31,7 @@ from exceptions.fs_exceptions import PlatformAlreadyExistsException
from handler.auth.constants import Scope
from handler.database import db_user_handler
from handler.filesystem import fs_platform_handler
from handler.filesystem.base_handler import LibraryStructure
from handler.metadata import (
meta_flashpoint_handler,
meta_gamelist_handler,
@@ -117,6 +119,7 @@ async def heartbeat() -> HeartbeatResponse:
},
"FRONTEND": {
"DISABLE_USERPASS_LOGIN": DISABLE_USERPASS_LOGIN,
"DISABLE_LOGS_VIEWER": DISABLE_LOGS_VIEWER,
"YOUTUBE_BASE_URL": YOUTUBE_BASE_URL,
},
"OIDC": {
@@ -159,6 +162,8 @@ async def metadata_heartbeat(source: str) -> bool:
return await meta_launchbox_handler.heartbeat()
case MetadataSource.HASHEOUS:
return await meta_hasheous_handler.heartbeat()
case MetadataSource.PLAYMATCH:
return await meta_playmatch_handler.heartbeat()
case MetadataSource.TGDB:
return await meta_tgdb_handler.heartbeat()
case MetadataSource.SGDB:
@@ -224,7 +229,7 @@ async def get_setup_library_info(request: Request):
rom_count = 0
try:
# Determine the roms directory based on structure
if detected_structure == "struct_a":
if detected_structure == LibraryStructure.A:
roms_path = os.path.join(
LIBRARY_BASE_PATH, cnfg.ROMS_FOLDER_NAME, fs_slug
)
backend/endpoints/logs.py
+41
View File
@@ -0,0 +1,41 @@
from fastapi import HTTPException, Request, status
from config import DISABLE_LOGS_VIEWER
from decorators.auth import protected_route
from endpoints.responses.log import LogEntrySchema
from endpoints.sockets.logs import get_recent_logs
from handler.auth.constants import Scope
from logger.log_stream_handler import LOG_BUFFER_SIZE
from utils.router import APIRouter
router = APIRouter(
prefix="/logs",
tags=["logs"],
)
# Largest backfill the buffer holds — derived from the producer's ring buffer
# so the two never drift.
MAX_LOG_LIMIT = LOG_BUFFER_SIZE
@protected_route(router.get, "", [Scope.LOGS_READ])
async def get_logs(
request: Request, limit: int = MAX_LOG_LIMIT
) -> list[LogEntrySchema]:
"""Return the most recent backend log lines for backfill (admin only).
Args:
request (Request): FastAPI Request object.
limit (int): Maximum number of recent lines to return.
Returns:
list[LogEntrySchema]: Recent log lines, oldest-first.
"""
if DISABLE_LOGS_VIEWER:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="Logs viewer is disabled"
)
bounded = max(1, min(limit, MAX_LOG_LIMIT))
entries = await get_recent_logs(bounded)
return [LogEntrySchema(**entry) for entry in entries]
backend/endpoints/platform.py
-3
View File
@@ -113,7 +113,6 @@ def get_platform(
async def update_platform(
request: Request,
id: Annotated[int, PathVar(description="Platform id.", ge=1)],
aspect_ratio: Annotated[str | None, Body(description="Cover aspect ratio.")] = None,
custom_name: Annotated[
str | None, Body(description="Custom platform name.")
] = None,
@@ -124,8 +123,6 @@ async def update_platform(
if not platform_db:
raise PlatformNotFoundInDatabaseException(id)
if aspect_ratio is not None:
platform_db.aspect_ratio = aspect_ratio
if custom_name is not None:
platform_db.custom_name = custom_name
platform_db = db_platform_handler.add_platform(platform_db)
backend/endpoints/play_sessions.py
+5 -2
View File
@@ -62,6 +62,8 @@ def ingest_play_sessions(
detail=f"Batch size exceeds maximum of {MAX_BATCH_SIZE}",
)
device_id = payload.device_id or getattr(request.state, "device_id", None)
summary = _ingest(
user_id=request.user.id,
username=request.user.username,
@@ -75,7 +77,7 @@ def ingest_play_sessions(
}
for s in payload.sessions
],
device_id=payload.device_id,
device_id=device_id,
)
return PlaySessionIngestResponse(
@@ -103,10 +105,11 @@ def get_play_sessions(
limit: int = 50,
offset: int = 0,
) -> list[PlaySessionSchema]:
effective_device_id = device_id or getattr(request.state, "device_id", None)
sessions = db_play_session_handler.get_sessions(
user_id=request.user.id,
rom_id=rom_id,
device_id=device_id,
device_id=effective_device_id,
start_after=start_after,
end_before=end_before,
limit=limit if start_after is None and end_before is None else None,
backend/endpoints/raw.py
+25 -6
View File
@@ -1,9 +1,13 @@
from mimetypes import guess_type
from pathlib import Path
from fastapi import HTTPException, Request
from fastapi.responses import FileResponse
from decorators.auth import protected_route
from handler.auth.constants import Scope
from handler.filesystem import fs_asset_handler
from handler.filesystem.assets_handler import SAFE_IMAGE_MIME_TYPES
from utils.router import APIRouter
router = APIRouter(
@@ -12,6 +16,24 @@ router = APIRouter(
)
def _build_asset_response(resolved_path: Path) -> FileResponse:
guessed_type, _ = guess_type(resolved_path.name)
if guessed_type in SAFE_IMAGE_MIME_TYPES:
return FileResponse(
path=str(resolved_path),
filename=resolved_path.name,
media_type=guessed_type,
content_disposition_type="inline",
)
return FileResponse(
path=str(resolved_path),
filename=resolved_path.name,
media_type="application/octet-stream",
content_disposition_type="attachment",
)
@protected_route(router.head, "/assets/{path:path}", [Scope.ASSETS_READ])
def head_raw_asset(request: Request, path: str):
try:
@@ -21,9 +43,9 @@ def head_raw_asset(request: Request, path: str):
# Check if file exists and is a file (not directory)
if not resolved_path.exists() or not resolved_path.is_file():
return HTTPException(status_code=404, detail="Asset not found")
raise HTTPException(status_code=404, detail="Asset not found")
return FileResponse(path=str(resolved_path), filename=resolved_path.name)
return _build_asset_response(resolved_path)
@protected_route(router.get, "/assets/{path:path}", [Scope.ASSETS_READ])
@@ -47,9 +69,6 @@ def get_raw_asset(request: Request, path: str):
# Check if file exists and is a file (not directory)
if not resolved_path.exists() or not resolved_path.is_file():
return HTTPException(status_code=404, detail="Asset not found")
if not resolved_path:
raise HTTPException(status_code=404, detail="Asset not found")
return FileResponse(path=str(resolved_path), filename=resolved_path.name)
return _build_asset_response(resolved_path)
backend/endpoints/responses/activity.py
+22
View File
@@ -0,0 +1,22 @@
from .base import BaseModel
class ActivityEntrySchema(BaseModel):
user_id: int
username: str
avatar_path: str
rom_id: int
rom_name: str
rom_cover_path: str = ""
screenshot_path: str = ""
platform_slug: str
platform_name: str
device_id: str
device_type: str
started_at: str
class ActivityClearSchema(BaseModel):
user_id: int
device_id: str
rom_id: int
backend/endpoints/responses/assets.py
+35 -1
View File
@@ -30,14 +30,27 @@ class BaseAsset(BaseModel):
class ScreenshotSchema(BaseAsset):
pass
is_gallery: bool = False
is_public: bool = False
class UserScreenshotSchema(ScreenshotSchema):
"""A gallery screenshot enriched with its owner's username, for the
community (My / Community) view. Mirrors UserNoteSchema."""
username: str
# Author identity for rendering an avatar next to community screenshots.
user_avatar_path: str = ""
user_updated_at: UTCDatetime | None = None
class SaveSchema(BaseAsset):
emulator: str | None
slot: str | None = None
content_hash: str | None = None
is_public: bool = False
screenshot: ScreenshotSchema | None
origin_device_id: str | None = None
device_syncs: list[DeviceSyncSchema] = []
@model_validator(mode="before")
@@ -60,6 +73,16 @@ class SaveSchema(BaseAsset):
return result
class UserSaveSchema(SaveSchema):
"""A save enriched with its owner's username, for the community (My /
Community) view. Mirrors UserScreenshotSchema."""
username: str
# Author identity for rendering an avatar next to community saves.
user_avatar_path: str = ""
user_updated_at: UTCDatetime | None = None
class SlotSummarySchema(BaseModel):
slot: str | None
count: int
@@ -73,4 +96,15 @@ class SaveSummarySchema(BaseModel):
class StateSchema(BaseAsset):
emulator: str | None
is_public: bool = False
screenshot: ScreenshotSchema | None
class UserStateSchema(StateSchema):
"""A state enriched with its owner's username, for the community (My /
Community) view. Mirrors UserScreenshotSchema."""
username: str
# Author identity for rendering an avatar next to community states.
user_avatar_path: str = ""
user_updated_at: UTCDatetime | None = None
backend/endpoints/responses/client_token.py
+3
View File
@@ -13,6 +13,7 @@ class ClientTokenSchema(BaseModel):
last_used_at: UTCDatetime | None
created_at: UTCDatetime
user_id: int
device_id: str | None = None
class ClientTokenCreateSchema(ClientTokenSchema):
@@ -21,6 +22,8 @@ class ClientTokenCreateSchema(ClientTokenSchema):
class ClientTokenAdminSchema(ClientTokenSchema):
username: str
user_avatar_path: str
user_updated_at: UTCDatetime
class ClientTokenPairSchema(BaseModel):
backend/endpoints/responses/device.py
+1
View File
@@ -31,6 +31,7 @@ class DeviceSchema(BaseModel):
ip_address: str | None
mac_address: str | None
hostname: str | None
client_device_identifier: str | None
sync_mode: SyncMode
sync_enabled: bool
sync_config: dict | None
backend/endpoints/responses/device_auth.py
+81
View File
@@ -0,0 +1,81 @@
from pydantic import Field, field_validator
from handler.auth.constants import Scope
from .base import BaseModel, UTCDatetime
# A request can never legitimately ask for more than every known scope.
_MAX_REQUESTED_SCOPES = len(Scope)
_VALID_SCOPES = {str(s) for s in Scope}
class DeviceAuthInitPayload(BaseModel):
client_device_identifier: str = Field(min_length=1, max_length=255)
name: str = Field(min_length=1, max_length=255)
client: str = Field(min_length=1, max_length=50)
platform: str | None = Field(default=None, max_length=50)
client_version: str | None = Field(default=None, max_length=50)
requested_scopes: list[str] = Field(min_length=1, max_length=_MAX_REQUESTED_SCOPES)
@field_validator("requested_scopes")
@classmethod
def _validate_requested_scopes(cls, value: list[str]) -> list[str]:
# Reject invalid scopes at the boundary
invalid = [s for s in value if s not in _VALID_SCOPES]
if invalid:
raise ValueError(f"Unknown scopes: {', '.join(sorted(set(invalid)))}")
return value
class DeviceAuthInitResponse(BaseModel):
device_code: str
user_code: str
verification_path: str = Field(
description=(
"Relative web-UI path (/pair/device). The client joins it with the "
"origin it was configured to reach; the server is origin-agnostic."
)
)
verification_path_complete: str = Field(
description="Same path with ?user_code= appended, for QR display."
)
expires_in: int
interval: int
class DeviceAuthPendingSchema(BaseModel):
client_device_identifier: str
name: str
client: str
platform: str | None
client_version: str | None
requested_scopes: list[str]
allowed_scopes: list[str]
expires_at: UTCDatetime
class DeviceAuthApprovePayload(BaseModel):
user_code: str = Field(min_length=1, max_length=32)
approved_scopes: list[str] = Field(min_length=1)
device_name: str | None = Field(default=None, max_length=255)
expires_in: str | None = None
class DeviceAuthApproveResponse(BaseModel):
device_id: str
device_name: str | None
class DeviceAuthDenyPayload(BaseModel):
user_code: str = Field(min_length=1, max_length=32)
class DeviceAuthTokenPayload(BaseModel):
device_code: str = Field(min_length=1, max_length=128)
class DeviceAuthTokenResponse(BaseModel):
access_token: str
device_id: str
scopes: list[str]
expires_at: UTCDatetime | None
backend/endpoints/responses/heartbeat.py
+1
View File
@@ -33,6 +33,7 @@ class EmulationDict(TypedDict):
class FrontendDict(TypedDict):
DISABLE_USERPASS_LOGIN: bool
DISABLE_LOGS_VIEWER: bool
YOUTUBE_BASE_URL: str
backend/endpoints/responses/identity.py
+3 -1
View File
@@ -43,7 +43,9 @@ class UserSchema(BaseModel):
return None
schema = cls.model_validate(db_user)
schema.current_device_id = request.session.get("device_id")
schema.current_device_id = getattr(
request.state, "device_id", None
) or request.session.get("device_id")
return schema
backend/endpoints/responses/log.py
+10
View File
@@ -0,0 +1,10 @@
from endpoints.responses.base import BaseModel
class LogEntrySchema(BaseModel):
"""A single backend log line streamed to the admin log viewer."""
ts: int # epoch milliseconds
level: str
module: str
message: str
backend/endpoints/responses/platform.py
-3
View File
@@ -1,7 +1,5 @@
from pydantic import ConfigDict, Field, computed_field, field_validator
from models.platform import DEFAULT_COVER_ASPECT_RATIO
from .base import BaseModel, UTCDatetime
from .firmware import FirmwareSchema
@@ -35,7 +33,6 @@ class PlatformSchema(BaseModel):
url: str | None = None
url_logo: str | None = None
firmware: list[FirmwareSchema] = Field(default_factory=list)
aspect_ratio: str = DEFAULT_COVER_ASPECT_RATIO
created_at: UTCDatetime
updated_at: UTCDatetime
fs_size_bytes: int
backend/endpoints/responses/rom.py
+199 -33
View File
@@ -1,13 +1,21 @@
from __future__ import annotations
import re
from collections.abc import Sequence
from datetime import datetime, timezone
from typing import NotRequired, TypedDict, get_type_hints
from fastapi import Request
from pydantic import ConfigDict, computed_field, field_validator, model_validator
from pydantic import ConfigDict, Field, computed_field, field_validator, model_validator
from endpoints.responses.assets import SaveSchema, ScreenshotSchema, StateSchema
from endpoints.responses.assets import (
SaveSchema,
ScreenshotSchema,
StateSchema,
UserSaveSchema,
UserScreenshotSchema,
UserStateSchema,
)
from handler.metadata.flashpoint_handler import FlashpointMetadata
from handler.metadata.gamelist_handler import GamelistMetadata
from handler.metadata.hasheous_handler import HasheousMetadata
@@ -18,7 +26,7 @@ from handler.metadata.moby_handler import MobyMetadata
from handler.metadata.ra_handler import RAMetadata
from handler.metadata.ss_handler import SSMetadata
from models.collection import Collection
from models.rom import Rom, RomFileCategory, RomUserStatus
from models.rom import Rom, RomArchiveMember, RomFile, RomFileCategory, RomUserStatus
from .base import BaseModel, UTCDatetime
@@ -37,6 +45,9 @@ class UserNoteSchema(BaseModel):
updated_at: UTCDatetime
user_id: int
username: str
# Author identity for rendering an avatar next to community notes.
user_avatar_path: str = ""
user_updated_at: UTCDatetime | None = None
RomIGDBMetadata = TypedDict( # type: ignore[misc]
@@ -147,6 +158,21 @@ class RomUserSchema(BaseModel):
return rom_user_schema_factory()
class RomFileAudioMetaSchema(BaseModel):
model_config = ConfigDict(from_attributes=True)
title: str | None = None
artist: str | None = None
album: str | None = None
year: str | None = None
genre: str | None = None
track: str | None = None
disc: str | None = None
duration_seconds: float | None = None
has_embedded_cover: bool = False
cover_path: str | None = None
class RomFileSchema(BaseModel):
model_config = ConfigDict(from_attributes=True)
@@ -164,7 +190,19 @@ class RomFileSchema(BaseModel):
md5_hash: str | None
sha1_hash: str | None
ra_hash: str | None
chd_sha1_hash: str | None
archive_members: list[RomArchiveMember] | None
category: RomFileCategory | None
audio_meta: RomFileAudioMetaSchema | None = None
class SoundtrackTrackMetaSchema(BaseModel):
model_config = ConfigDict(from_attributes=True)
file_id: int
file_name: str
file_size_bytes: int
audio_meta: RomFileAudioMetaSchema | None = None
@model_validator(mode="after")
def default_category_for_non_nested(self) -> RomFileSchema:
@@ -251,6 +289,7 @@ class RomSchema(BaseModel):
fs_size_bytes: int
name: str | None
name_sort_key: str | None
slug: str | None
summary: str | None
@@ -273,6 +312,8 @@ class RomSchema(BaseModel):
url_cover: str | None
has_manual: bool
has_manual_files: bool
has_soundtrack: bool
path_manual: str | None
url_manual: str | None
@@ -295,31 +336,33 @@ class RomSchema(BaseModel):
has_simple_single_file: bool
has_nested_single_file: bool
has_multiple_files: bool
files: list[RomFileSchema]
full_path: str
created_at: UTCDatetime
updated_at: UTCDatetime
missing_from_fs: bool
has_notes: bool
siblings: list[SiblingRomSchema]
rom_user: RomUserSchema
merged_screenshots: list[str]
merged_ra_metadata: RomRAMetadata | None
files: list[RomFileSchema] = Field(validation_alias="included_files")
sibling_roms: list[SiblingRomSchema] = Field(
validation_alias="included_sibling_roms"
)
@field_validator("files")
def sort_files(cls, v: list[RomFileSchema]) -> list[RomFileSchema]:
return sorted(v, key=lambda x: x.file_name)
@field_validator("sibling_roms")
def sort_sibling_roms(cls, v: list[SiblingRomSchema]) -> list[SiblingRomSchema]:
return sorted(v, key=lambda x: x.sort_comparator)
@classmethod
def populate_properties(cls, db_rom: Rom, request: Request) -> Rom:
db_rom.rom_user = RomUserSchema.for_user(request.user.id, db_rom) # type: ignore
db_rom.siblings = [ # type: ignore
SiblingRomSchema(
id=s.id,
name=s.name,
fs_name_no_tags=s.fs_name_no_tags,
fs_name_no_ext=s.fs_name_no_ext,
)
for s in db_rom.sibling_roms
]
db_rom.has_notes = any( # type: ignore
db_rom.rom_user = RomUserSchema.for_user(request.user.id, db_rom) # type: ignore[assignment]
db_rom.has_notes = any( # type: ignore[assignment]
note.is_public or note.user_id == request.user.id for note in db_rom.notes
)
return db_rom
@@ -332,20 +375,13 @@ class RomSchema(BaseModel):
def sort_alternative_names(cls, v: list[str]) -> list[str]:
return sorted(v)
@field_validator("files")
def sort_files(cls, v: list[RomFileSchema]) -> list[RomFileSchema]:
return sorted(v, key=lambda x: x.file_name)
@field_validator("siblings")
def sort_siblings(cls, v: list[SiblingRomSchema]) -> list[SiblingRomSchema]:
return sorted(v, key=lambda x: x.sort_comparator)
class SiblingRomSchema(BaseModel):
id: int
name: str | None
fs_name_no_tags: str
fs_name_no_ext: str
is_main_sibling: bool
@computed_field # type: ignore
@property
@@ -359,18 +395,68 @@ class SiblingRomSchema(BaseModel):
.lower()
)
@classmethod
def from_rom(cls, rom: Rom, *, is_main_sibling: bool = False) -> SiblingRomSchema:
return cls(
id=rom.id,
name=rom.name,
fs_name_no_tags=rom.fs_name_no_tags,
fs_name_no_ext=rom.fs_name_no_ext,
is_main_sibling=is_main_sibling,
)
class SimpleRomSchema(RomSchema):
screenshot_path: str | None = None
@classmethod
def from_orm_with_request(cls, db_rom: Rom, request: Request) -> SimpleRomSchema:
def from_orm_with_request(
cls,
db_rom: Rom,
request: Request,
files: Sequence[RomFile] | None = None,
siblings: Sequence[tuple[Rom, bool]] | None = None,
screenshot_path: str | None = None,
) -> SimpleRomSchema:
db_rom = cls.populate_properties(db_rom, request)
db_rom.screenshot_path = screenshot_path # type: ignore[assignment]
# The list endpoint passes pre-fetched `files`/`siblings` (batched via
# get_files_for_roms / get_siblings_for_roms, no per-row hydration).
# Single-rom endpoints (e.g. `/{id}/simple`, loaded via the
# `with_details` decorator) pass neither and fall back to the
# eager-loaded relationships. `None` (not provided) is distinct from an
# explicit empty list (e.g. the gallery list, which intentionally omits
# files unless `with_files` is set).
if files is None:
files = db_rom.files
if siblings is None:
user_id = request.user.id
siblings = [
(
s,
any(
ru.user_id == user_id and ru.is_main_sibling
for ru in s.rom_users
),
)
for s in db_rom.sibling_roms
]
db_rom.included_files = list(files) # type: ignore[assignment]
db_rom.included_sibling_roms = [ # type: ignore[assignment]
SiblingRomSchema.from_rom(s, is_main_sibling=is_main)
for s, is_main in siblings
]
return cls.model_validate(db_rom)
@classmethod
def from_orm_with_factory(cls, db_rom: Rom) -> SimpleRomSchema:
db_rom.rom_user = rom_user_schema_factory() # type: ignore
db_rom.siblings = [] # type: ignore
db_rom.has_notes = False # type: ignore
db_rom.rom_user = rom_user_schema_factory() # type: ignore[assignment]
db_rom.included_files = [] # type: ignore[assignment]
db_rom.included_sibling_roms = [] # type: ignore[assignment]
db_rom.has_notes = False # type: ignore[assignment]
db_rom.screenshot_path = None # type: ignore[assignment]
return cls.model_validate(db_rom)
@@ -395,7 +481,10 @@ class UserCollectionSchema(BaseModel):
class DetailedRomSchema(RomSchema):
user_saves: list[SaveSchema]
user_states: list[StateSchema]
all_user_saves: list[UserSaveSchema]
all_user_states: list[UserStateSchema]
user_screenshots: list[ScreenshotSchema]
all_user_screenshots: list[UserScreenshotSchema]
user_collections: list[UserCollectionSchema]
all_user_notes: list[UserNoteSchema]
@@ -404,18 +493,34 @@ class DetailedRomSchema(RomSchema):
user_id = request.user.id
db_rom = cls.populate_properties(db_rom, request)
db_rom.user_saves = [ # type: ignore
sorted_siblings = sorted(
(
SiblingRomSchema.from_rom(
s,
is_main_sibling=any(
ru.user_id == user_id and ru.is_main_sibling
for ru in s.rom_users
),
)
for s in db_rom.sibling_roms
),
key=lambda x: x.sort_comparator,
)
db_rom.included_sibling_roms = sorted_siblings # type: ignore[assignment]
db_rom.included_files = sorted(db_rom.files, key=lambda x: x.file_name) # type: ignore[assignment]
db_rom.user_saves = [ # type: ignore[assignment]
SaveSchema.model_validate(s) for s in db_rom.saves if s.user_id == user_id
]
db_rom.user_states = [ # type: ignore
db_rom.user_states = [ # type: ignore[assignment]
StateSchema.model_validate(s) for s in db_rom.states if s.user_id == user_id
]
db_rom.user_screenshots = [ # type: ignore
db_rom.user_screenshots = [ # type: ignore[assignment]
ScreenshotSchema.model_validate(s)
for s in db_rom.screenshots
if s.user_id == user_id
]
db_rom.user_collections = UserCollectionSchema.for_user( # type: ignore
db_rom.user_collections = UserCollectionSchema.for_user( # type: ignore[assignment]
user_id, db_rom.collections
)
@@ -437,12 +542,73 @@ class DetailedRomSchema(RomSchema):
"updated_at": note.updated_at,
"user_id": note.user_id,
"username": note.user.username,
"user_avatar_path": note.user.avatar_path,
"user_updated_at": note.user.updated_at,
}
all_notes.append(UserNoteSchema.model_validate(note_dict))
# Sort notes by updated_at (most recent first)
all_notes.sort(key=lambda x: x.updated_at, reverse=True)
db_rom.all_user_notes = all_notes # type: ignore
db_rom.all_user_notes = all_notes # type: ignore[assignment]
# Gallery screenshots visible to this user: own (public + private) plus
# other users' public ones. Mirrors the notes flow above. Excludes the
# auto-captured save/state thumbnails (is_gallery == False).
from handler.database import db_screenshot_handler
gallery_screenshots = db_screenshot_handler.get_rom_gallery_screenshots(
rom_id=db_rom.id, user_id=user_id
)
db_rom.all_user_screenshots = [ # type: ignore[assignment]
UserScreenshotSchema.model_validate(
{
**{
field: getattr(s, field)
for field in ScreenshotSchema.model_fields
},
"username": s.user.username,
"user_avatar_path": s.user.avatar_path,
"user_updated_at": s.user.updated_at,
}
)
for s in gallery_screenshots
]
# Saves/states visible to this user: own (public + private) plus other
# users' public ones. Mirrors the screenshots flow above.
from handler.database import db_save_handler, db_state_handler
# SaveSchema.model_validate handles the lazy `device_syncs` relationship
# (skips it when unloaded); reuse it rather than getattr-ing raw fields.
shared_saves = db_save_handler.get_rom_shared_saves(
rom_id=db_rom.id, user_id=user_id
)
db_rom.all_user_saves = [ # type: ignore[assignment]
UserSaveSchema.model_validate(
{
**SaveSchema.model_validate(s).model_dump(),
"username": s.user.username,
"user_avatar_path": s.user.avatar_path,
"user_updated_at": s.user.updated_at,
}
)
for s in shared_saves
]
shared_states = db_state_handler.get_rom_shared_states(
rom_id=db_rom.id, user_id=user_id
)
db_rom.all_user_states = [ # type: ignore[assignment]
UserStateSchema.model_validate(
{
**StateSchema.model_validate(s).model_dump(),
"username": s.user.username,
"user_avatar_path": s.user.avatar_path,
"user_updated_at": s.user.updated_at,
}
)
for s in shared_states
]
return cls.model_validate(db_rom)
backend/endpoints/roms/__init__.py
+208 -43
View File
@@ -4,7 +4,7 @@ from base64 import b64encode
from datetime import datetime, timezone
from io import BytesIO
from stat import S_IFREG
from typing import Annotated, Any
from typing import Annotated, Any, Sequence
from urllib.parse import quote
from zipfile import ZIP_DEFLATED, ZIP_STORED, ZipFile, ZipInfo
@@ -25,7 +25,7 @@ from fastapi import (
status,
)
from fastapi.responses import Response
from fastapi_pagination.ext.sqlalchemy import paginate
from fastapi_pagination import resolve_params
from fastapi_pagination.limit_offset import LimitOffsetPage, LimitOffsetParams
from pydantic import BaseModel, Field
from starlette.responses import FileResponse
@@ -46,33 +46,39 @@ from endpoints.responses.rom import (
from exceptions.endpoint_exceptions import RomNotFoundInDatabaseException
from exceptions.fs_exceptions import RomAlreadyExistsException
from handler.auth.constants import Scope
from handler.database import db_rom_handler
from handler.database import db_rom_handler, db_save_handler
from handler.database.base_handler import sync_session
from handler.filesystem import fs_resource_handler, fs_rom_handler
from handler.filesystem.assets_handler import validate_image_upload
from handler.metadata import (
meta_flashpoint_handler,
meta_igdb_handler,
meta_launchbox_handler,
meta_moby_handler,
meta_playmatch_handler,
meta_ra_handler,
meta_ss_handler,
)
from handler.metadata.ss_handler import get_preferred_media_types
from handler.metadata.ss_handler import add_ss_auth_to_url, get_preferred_media_types
from logger.formatter import BLUE
from logger.formatter import highlight as hl
from logger.logger import log
from models.rom import Rom, RomUserStatus
from models.rom import Rom, RomUserStatus, compute_name_sort_key
from utils.background_tasks import fire_and_forget
from utils.database import safe_int, safe_str_to_bool
from utils.filesystem import sanitize_filename
from utils.hashing import crc32_to_hex
from utils.nginx import FileRedirectResponse, ZipContentLine, ZipResponse
from utils.router import APIRouter
from utils.screenshots import continue_playing_screenshot
from utils.validation import ValidationError
from .files import router as files_router
from .manual import router as manual_router
from .notes import router as notes_router
from .patch import router as patch_router
from .screenshot import router as screenshot_router
from .soundtrack import router as soundtrack_router
from .upload import router as upload_router
router = APIRouter(
@@ -82,6 +88,8 @@ router = APIRouter(
router.include_router(upload_router)
router.include_router(files_router)
router.include_router(manual_router)
router.include_router(soundtrack_router)
router.include_router(screenshot_router)
router.include_router(notes_router)
router.include_router(patch_router)
@@ -130,6 +138,7 @@ class RomUpdateForm(BaseModel):
default=None, description="Raw manual metadata as JSON string."
)
name: str | None = None
name_sort_key: str | None = None
summary: str | None = None
fs_name: str | None = None
url_cover: str | None = None
@@ -189,6 +198,7 @@ async def parse_rom_update_form(
raw_hltb_metadata: str | None = Form(default=None),
raw_manual_metadata: str | None = Form(default=None),
name: str | None = Form(default=None),
name_sort_key: str | None = Form(default=None),
summary: str | None = Form(default=None),
fs_name: str | None = Form(default=None),
url_cover: str | None = Form(default=None),
@@ -217,6 +227,7 @@ async def parse_rom_update_form(
"raw_hltb_metadata": raw_hltb_metadata,
"raw_manual_metadata": raw_manual_metadata,
"name": name,
"name_sort_key": name_sort_key,
"summary": summary,
"fs_name": fs_name,
"url_cover": url_cover,
@@ -469,8 +480,14 @@ def get_roms(
] = "any",
order_by: Annotated[
str,
Query(description="Field to order results by."),
] = "name",
Query(
description=(
"Field to order results by. Leave empty to order by search "
"relevance when a search term is given on MySQL/MariaDB; other "
"databases fall back to name."
),
),
] = "",
order_dir: Annotated[
str,
Query(description="Order direction, either 'asc' or 'desc'."),
@@ -481,12 +498,17 @@ def get_roms(
description="Filter roms updated after this datetime (ISO 8601 format with timezone information)."
),
] = None,
with_files: Annotated[
bool,
Query(description="Whether to include each rom's file entries."),
] = False,
) -> CustomLimitOffsetPage[SimpleRomSchema]:
"""Retrieve roms."""
unfiltered_query, order_by_attr = db_rom_handler.get_roms_query(
user_id=request.user.id,
order_by=order_by.lower(),
order_dir=order_dir.lower(),
search_term=search_term,
)
# Filter down the query
@@ -527,13 +549,36 @@ def get_roms(
player_counts_logic=player_counts_logic,
group_by_meta_id=group_by_meta_id,
updated_after=updated_after,
include_file_stats=True,
)
# Cache only the unscoped library scan; scoped/searched sets are narrower and computed live.
is_unscoped = not (
search_term
or platform_ids
or collection_id
or virtual_collection_id
or smart_collection_id
)
# Get the char index for the roms
char_index_dict = {}
if with_char_index:
# The computed positions depend on ordering and grouping, so those
# must be part of the cache key. Otherwise switching sort
# direction/column (or toggling grouping) reuses a stale index and
# the AlphaStrip highlights the wrong letters.
char_index_cache_key = (
f"all:u{request.user.id}"
f":o{order_by.lower()}:d{order_dir.lower()}:g{int(group_by_meta_id)}"
if is_unscoped
else None
)
char_index = db_rom_handler.with_char_index(
query=query, order_by_attr=order_by_attr
query=query,
order_by_attr=order_by_attr,
order_dir=order_dir.lower(),
cache_key=char_index_cache_key,
)
char_index_dict = {char: index for (char, index) in char_index}
@@ -560,7 +605,16 @@ def get_roms(
smart_collection_id=smart_collection_id,
search_term=search_term,
)
query_filters = db_rom_handler.with_filter_values(query=filter_query)
cache_key = (
f"all:u{request.user.id}"
f":o{order_by.lower()}:d{order_dir.lower()}:g{int(group_by_meta_id)}"
if is_unscoped
else None
)
query_filters = db_rom_handler.with_filter_values(
query=filter_query,
cache_key=cache_key,
)
# trunk-ignore(mypy/typeddict-item)
filter_values = RomFiltersDict(**query_filters)
@@ -568,17 +622,56 @@ def get_roms(
with sync_session.begin() as session:
rom_id_index = session.scalars(query.with_only_columns(Rom.id)).all() # type: ignore
return paginate(
session,
query,
transformer=lambda items: [
SimpleRomSchema.from_orm_with_request(i, request) for i in items
],
additional_data={
"char_index": char_index_dict,
"rom_id_index": rom_id_index,
"filter_values": filter_values,
},
def _transform(items: Sequence[Rom]) -> list[SimpleRomSchema]:
rom_ids = [i.id for i in items]
files_by_rom = (
db_rom_handler.get_files_for_roms(rom_ids, session=session)
if with_files
else {}
)
siblings_by_rom = db_rom_handler.get_siblings_for_roms(
rom_ids, user_id=request.user.id, session=session
)
# Continue-playing rail
screenshot_by_rom: dict[int, str | None] = {}
if last_played:
latest_saves = db_save_handler.get_latest_saves_for_roms(
user_id=request.user.id, rom_ids=rom_ids, session=session
)
for item in items:
screenshot_by_rom[item.id] = continue_playing_screenshot(
item, latest_saves.get(item.id)
)
return [
SimpleRomSchema.from_orm_with_request(
db_rom=item,
request=request,
files=files_by_rom.get(item.id, []),
siblings=siblings_by_rom.get(item.id, []),
screenshot_path=screenshot_by_rom.get(item.id),
)
for item in items
]
params = resolve_params()
total = len(rom_id_index)
page_ids = list(rom_id_index[params.offset : params.offset + params.limit])
if page_ids:
page_rows = session.scalars(query.where(Rom.id.in_(page_ids))).all()
rows_by_id = {rom.id: rom for rom in page_rows}
page_items = [rows_by_id[i] for i in page_ids if i in rows_by_id]
else:
page_items = []
return CustomLimitOffsetPage.create(
_transform(page_items),
params,
total=total,
char_index=char_index_dict,
rom_id_index=list(rom_id_index),
filter_values=filter_values,
)
@@ -804,6 +897,32 @@ async def get_rom_filters(request: Request) -> RomFiltersDict:
return RomFiltersDict(**filters)
@protected_route(
router.get,
"/{id}/simple",
[] if DISABLE_DOWNLOAD_ENDPOINT_AUTH else [Scope.ROMS_READ],
responses={status.HTTP_404_NOT_FOUND: {}},
)
def get_rom_simple(
request: Request,
id: Annotated[int, PathVar(description="Rom internal id.", ge=1)],
) -> SimpleRomSchema:
"""Retrieve a rom by ID with the lightweight schema — no eager-loaded
`user_saves` / `user_states` / `user_screenshots` / `user_collections` /
`all_user_notes` arrays. Designed for the v2 gallery card which only
needs the indicator flags (`has_notes`, `ra_id`, status, etc.) already
present on `SimpleRomSchema`. The full `DetailedRomSchema` is only
fetched on user-driven detail interactions (game details page, quick-
note dialog open, achievements panel)."""
rom = db_rom_handler.get_rom(id)
if not rom:
raise RomNotFoundInDatabaseException(id)
return SimpleRomSchema.from_orm_with_request(rom, request)
@protected_route(
router.get,
"/{id}",
@@ -854,6 +973,12 @@ async def head_rom_content(
files = [f for f in files if f.id in file_id_values]
files.sort(key=lambda x: x.file_name)
if not files:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"No files found for ROM {id}",
)
# Serve the file directly in development mode for emulatorjs
if DEV_MODE:
if len(files) == 1:
@@ -931,6 +1056,12 @@ async def get_rom_content(
files = [f for f in files if f.id in file_id_values]
files.sort(key=lambda x: x.file_name)
if not files:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"No files found for ROM {id}",
)
log.info(
f"User {hl(current_username, color=BLUE)} is downloading {hl(rom.fs_name)}"
)
@@ -1094,6 +1225,7 @@ async def update_rom(
"hltb_id": None,
"libretro_id": None,
"name": rom.fs_name,
"name_sort_key": compute_name_sort_key(rom.fs_name),
"summary": "",
"url_screenshots": [],
"path_screenshots": [],
@@ -1119,6 +1251,7 @@ async def update_rom(
if not rom:
raise RomNotFoundInDatabaseException(id)
db_rom_handler.invalidate_filter_values_cache()
return DetailedRomSchema.from_orm_with_request(rom, request)
provided_fields = form_data.model_fields_set
@@ -1255,7 +1388,7 @@ async def update_rom(
cleaned_data.update({"ss_id": None, "ss_metadata": {}})
if cleaned_data["igdb_id"] and int(cleaned_data["igdb_id"]) != rom.igdb_id:
igdb_rom = await meta_igdb_handler.get_rom_by_id(cleaned_data["igdb_id"])
igdb_rom = await meta_igdb_handler.get_rom_by_id(rom, cleaned_data["igdb_id"])
if igdb_rom.get("igdb_id"):
cleaned_data.update(igdb_rom)
elif rom.igdb_id and not cleaned_data["igdb_id"]:
@@ -1268,7 +1401,7 @@ async def update_rom(
path_screenshots = await fs_resource_handler.get_rom_screenshots(
rom=rom,
overwrite=bool(screenshots_changed),
url_screenshots=cleaned_data.get("url_screenshots", []),
url_screenshots=[add_ss_auth_to_url(u) for u in url_screenshots],
)
cleaned_data.update(
{"path_screenshots": path_screenshots, "url_screenshots": []}
@@ -1277,33 +1410,51 @@ async def update_rom(
log.error(f"Invalid screenshot URL in update_rom: {str(e)}")
raise HTTPException(status_code=400, detail=str(e)) from e
name_value = form_data.name if "name" in provided_fields else rom.name
cleaned_data.update(
{
"name": form_data.name if "name" in provided_fields else rom.name,
"name": name_value,
"summary": (
form_data.summary if "summary" in provided_fields else rom.summary
),
}
)
if "name_sort_key" in provided_fields:
# The edit form always echoes the current key back, so only act when the
# user actually changed it: a value is a custom override, an empty value
# reverts to deriving from the (possibly new) name. When unchanged, leave
# it out so update_rom re-derives only if the stored key isn't custom.
submitted = (form_data.name_sort_key or "").strip()
if submitted != (rom.name_sort_key or "").strip():
cleaned_data["name_sort_key"] = compute_name_sort_key(
submitted or name_value
)
new_fs_name = str(form_data.fs_name or rom.fs_name)
new_fs_name = sanitize_filename(new_fs_name)
cleaned_data.update(
{
"fs_name": new_fs_name,
"fs_name_no_tags": fs_rom_handler.get_file_name_with_no_tags(new_fs_name),
"fs_name_no_ext": fs_rom_handler.get_file_name_with_no_extension(
new_fs_name
),
}
)
cleaned_data.update({"fs_name": new_fs_name})
# Re-parse tags from the filename so region/language/revision/version/tags
# stay in sync whenever the fs_name changes.
if new_fs_name != rom.fs_name:
parsed_tags = fs_rom_handler.parse_tags(new_fs_name)
cleaned_data.update(
{
"regions": parsed_tags.regions,
"languages": parsed_tags.languages,
"tags": parsed_tags.other_tags,
"revision": parsed_tags.revision,
"version": parsed_tags.version,
}
)
if remove_cover:
cleaned_data.update(await fs_resource_handler.remove_cover(rom))
cleaned_data.update({"url_cover": ""})
else:
if artwork is not None and artwork.filename is not None:
file_ext = artwork.filename.split(".")[-1]
file_ext = validate_image_upload(artwork, label="Artwork")
artwork_content = BytesIO(await artwork.read())
(
path_cover_l,
@@ -1325,7 +1476,7 @@ async def update_rom(
path_cover_s, path_cover_l = await fs_resource_handler.get_cover(
entity=rom,
overwrite=url_cover != rom.url_cover,
url_cover=str(url_cover),
url_cover=add_ss_auth_to_url(url_cover),
)
cleaned_data.update(
{
@@ -1345,7 +1496,7 @@ async def update_rom(
path_manual = await fs_resource_handler.get_manual(
rom=rom,
overwrite=url_manual != rom.url_manual,
url_manual=str(url_manual) if url_manual else None,
url_manual=add_ss_auth_to_url(url_manual),
)
cleaned_data.update(
{
@@ -1385,10 +1536,16 @@ async def update_rom(
media_type,
)
if cleaned_data.get("ss_metadata", {}).get(f"{media_type.value}_path"):
media_path = cleaned_data.get("ss_metadata", {}).get(
f"{media_type.value}_path"
)
media_url = cleaned_data.get("ss_metadata", {}).get(
f"{media_type.value}_url"
)
if media_path and media_url:
await fs_resource_handler.store_media_file(
cleaned_data["ss_metadata"][f"{media_type.value}_url"],
cleaned_data["ss_metadata"][f"{media_type.value}_path"],
add_ss_auth_to_url(media_url),
media_path,
)
log.debug(
@@ -1428,6 +1585,10 @@ async def update_rom(
if not rom:
raise RomNotFoundInDatabaseException(id)
if meta_playmatch_handler.is_manual_match(form_data.model_fields_set):
fire_and_forget(meta_playmatch_handler.submit_manual_match_suggestion(rom))
db_rom_handler.invalidate_filter_values_cache()
return DetailedRomSchema.from_orm_with_request(rom, request)
@@ -1495,11 +1656,9 @@ async def delete_roms(
f"Couldn't clean up empty parent directory for {hl(rom.fs_name)}: {dir_err}"
)
except FileNotFoundError:
error = f"Rom file {hl(rom.fs_name)} not found for platform {hl(rom.platform_display_name, color=BLUE)}[{hl(rom.platform_slug)}]"
log.error(error)
errors.append(error)
failed_items += 1
continue
log.warning(
f"Rom file {hl(rom.fs_name)} not found for platform {hl(rom.platform_display_name, color=BLUE)}[{hl(rom.platform_slug)}], deleting database entry only"
)
log.info(
f"Deleting {hl(str(rom.name or 'ROM'), color=BLUE)} [{hl(rom.fs_name)}] from database"
@@ -1518,6 +1677,9 @@ async def delete_roms(
failed_items += 1
errors.append(f"Failed to delete ROM {id}: {str(e)}")
if successful_items:
db_rom_handler.invalidate_filter_values_cache()
return {
"successful_items": successful_items,
"failed_items": failed_items,
@@ -1567,4 +1729,7 @@ async def update_rom_user(
rom_user = db_rom_handler.update_rom_user(db_rom_user.id, cleaned_data)
if "hidden" in cleaned_data:
db_rom_handler.invalidate_filter_values_cache()
return RomUserSchema.model_validate(rom_user)
backend/endpoints/roms/files.py
+25 -9
View File
@@ -1,5 +1,4 @@
from typing import Annotated
from urllib.parse import quote
from anyio import Path
from fastapi import HTTPException
@@ -16,6 +15,8 @@ from handler.filesystem import fs_rom_handler
from logger.formatter import BLUE
from logger.formatter import highlight as hl
from logger.logger import log
from models.rom import RomFileCategory
from utils.audio_tags import guess_audio_media_type
from utils.nginx import FileRedirectResponse
from utils.router import APIRouter
@@ -68,22 +69,37 @@ async def get_romfile_content(
detail="File not found",
)
log.info(f"User {hl(current_username, color=BLUE)} is downloading {hl(file_name)}")
log.info(
f"User {hl(current_username, color=BLUE)} is downloading {hl(file.file_name)}"
)
# Derive content type / disposition / download name from the trusted DB
# record, never from the client-supplied file_name path param — otherwise a
# caller could request the same bytes with an arbitrary extension to force a
# mismatched Content-Type while served inline (content-sniffing/XSS).
is_audio = file.category == RomFileCategory.SOUNDTRACK
media_type = (
guess_audio_media_type(file.file_name)
if is_audio
else "application/octet-stream"
)
disposition = "inline" if is_audio else "attachment"
# Serve the file directly in development mode for emulatorjs
if DEV_MODE:
rom_path = fs_rom_handler.validate_path(file.full_path)
# Starlette sets Content-Length and honors Range natively — inline
# disposition lets <audio> seek via Range requests.
return FileResponse(
path=rom_path,
filename=file_name,
headers={
"Content-Disposition": f"attachment; filename*=UTF-8''{quote(file_name)}; filename=\"{quote(file_name)}\"",
"Content-Type": "application/octet-stream",
"Content-Length": str(file.file_size_bytes),
},
filename=file.file_name,
media_type=media_type,
content_disposition_type=disposition,
)
# Otherwise proxy through nginx
# Otherwise proxy through nginx (which parses Range itself via X-Accel-Redirect)
return FileRedirectResponse(
download_path=Path(f"/library/{file.full_path}"),
disposition=disposition,
media_type=media_type,
)
backend/endpoints/roms/manual.py
+245 -9
View File
@@ -1,3 +1,4 @@
import os
from typing import Annotated
from fastapi import Header, HTTPException
@@ -12,14 +13,23 @@ from decorators.auth import protected_route
from exceptions.endpoint_exceptions import RomNotFoundInDatabaseException
from handler.auth.constants import Scope
from handler.database import db_rom_handler
from handler.filesystem import fs_resource_handler
from handler.filesystem import fs_resource_handler, fs_rom_handler
from logger.formatter import BLUE
from logger.formatter import highlight as hl
from logger.logger import log
from models.rom import RomFile, RomFileCategory
from utils.router import APIRouter
router = APIRouter()
MANUAL_FOLDER = "manual"
ALLOWED_MANUAL_EXTENSIONS = frozenset({".pdf", ".md"})
def _is_allowed_manual_file(file_name: str) -> bool:
_, ext = os.path.splitext(file_name)
return ext.lower() in ALLOWED_MANUAL_EXTENSIONS
@protected_route(
router.post,
@@ -45,6 +55,16 @@ async def add_rom_manuals(
if not rom:
raise RomNotFoundInDatabaseException(id)
# The stored filename is always `{rom.id}.pdf`; we only use `filename` as
# the form-field key, but normalise it to a safe basename first.
try:
safe_field_name = fs_resource_handler._sanitize_filename(filename)
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Invalid upload filename: {exc}",
) from exc
manuals_path = f"{rom.fs_resources_path}/manual"
file_location = fs_resource_handler.validate_path(f"{manuals_path}/{rom.id}.pdf")
log.info(f"Uploading manual to {hl(str(file_location))}")
@@ -53,7 +73,7 @@ async def add_rom_manuals(
parser = StreamingFormDataParser(headers=request.headers)
parser.register("x-upload-platform", NullTarget())
parser.register(filename, FileTarget(str(file_location)))
parser.register(safe_field_name, FileTarget(str(file_location)))
def cleanup_partial_file():
if file_location.exists():
@@ -62,16 +82,10 @@ async def add_rom_manuals(
try:
async for chunk in request.stream():
parser.data_received(chunk)
db_rom_handler.update_rom(
id,
{
"path_manual": f"{manuals_path}/{rom.id}.pdf",
},
)
except ClientDisconnect:
log.error("Client disconnected during upload")
cleanup_partial_file()
raise
except Exception as exc:
log.error("Error uploading files", exc_info=exc)
cleanup_partial_file()
@@ -80,6 +94,228 @@ async def add_rom_manuals(
detail="There was an error uploading the manual",
) from exc
db_rom_handler.update_rom(
id,
{
"path_manual": f"{manuals_path}/{rom.id}.pdf",
},
)
return Response(status_code=status.HTTP_201_CREATED)
@protected_route(
router.post,
"/{id}/manuals/redownload",
[Scope.ROMS_WRITE],
responses={
status.HTTP_404_NOT_FOUND: {},
status.HTTP_400_BAD_REQUEST: {},
},
)
async def redownload_rom_manual(
request: Request,
id: Annotated[int, PathVar(description="Rom internal id.", ge=1)],
) -> Response:
"""Re-download a rom's manual from its scraped URL (e.g. screenscraper.fr)."""
rom = db_rom_handler.get_rom(id)
if not rom:
raise RomNotFoundInDatabaseException(id)
if not rom.url_manual:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="No scraped manual URL available for this ROM",
)
try:
path_manual = await fs_resource_handler.get_manual(
rom=rom,
overwrite=True,
url_manual=str(rom.url_manual),
)
db_rom_handler.update_rom(id, {"path_manual": path_manual})
log.info(
f"Re-downloaded manual for {hl(rom.name or 'ROM', color=BLUE)} "
f"[{hl(rom.fs_name)}]"
)
except Exception as exc:
log.error("Error re-downloading manual", exc_info=exc)
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="There was an error re-downloading the manual",
) from exc
return Response()
@protected_route(
router.post,
"/{id}/manuals/files",
[Scope.ROMS_WRITE],
status_code=status.HTTP_201_CREATED,
responses={
status.HTTP_404_NOT_FOUND: {},
status.HTTP_400_BAD_REQUEST: {},
},
)
async def add_rom_manual_file(
request: Request,
id: Annotated[int, PathVar(description="Rom internal id.", ge=1)],
filename: Annotated[
str,
Header(
description="The name of the file being uploaded.",
alias="x-upload-filename",
),
],
) -> Response:
"""Upload a manual PDF into the ROM's own manual/ subfolder."""
rom = db_rom_handler.get_rom(id)
if not rom:
raise RomNotFoundInDatabaseException(id)
if rom.has_simple_single_file:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="Manual files can only be uploaded to folder-based ROMs",
)
try:
safe_filename = fs_rom_handler._sanitize_filename(filename)
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Invalid upload filename: {exc}",
) from exc
if safe_filename != filename:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="Upload filename must be a plain file name, not a path",
)
if not _is_allowed_manual_file(safe_filename):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=(
f"Unsupported manual file type. Allowed: "
f"{', '.join(sorted(ALLOWED_MANUAL_EXTENSIONS))}"
),
)
manual_dir_rel = f"{rom.full_path}/{MANUAL_FOLDER}"
file_rel_path = f"{manual_dir_rel}/{safe_filename}"
file_location = fs_rom_handler.validate_path(file_rel_path)
log.info(f"Uploading manual file to {hl(str(file_location))}")
await fs_rom_handler.make_directory(manual_dir_rel)
parser = StreamingFormDataParser(headers=request.headers)
parser.register("x-upload-platform", NullTarget())
parser.register(safe_filename, FileTarget(str(file_location)))
def cleanup_partial_file():
if file_location.exists():
file_location.unlink()
try:
async for chunk in request.stream():
parser.data_received(chunk)
except ClientDisconnect:
log.error("Client disconnected during upload")
cleanup_partial_file()
raise
except Exception as exc:
log.error("Error uploading manual file", exc_info=exc)
cleanup_partial_file()
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="There was an error uploading the manual file",
) from exc
stat = os.stat(file_location)
existing = db_rom_handler.get_rom_file_by_path(
rom_id=rom.id, file_path=manual_dir_rel, file_name=safe_filename
)
if existing:
db_rom_handler.update_rom_file(
existing.id,
{
"file_size_bytes": stat.st_size,
"last_modified": stat.st_mtime,
"category": RomFileCategory.MANUAL,
"missing_from_fs": False,
},
)
else:
db_rom_handler.add_rom_file(
RomFile(
rom_id=rom.id,
file_name=safe_filename,
file_path=manual_dir_rel,
file_size_bytes=stat.st_size,
last_modified=stat.st_mtime,
category=RomFileCategory.MANUAL,
)
)
return Response(status_code=status.HTTP_201_CREATED)
@protected_route(
router.delete,
"/{id}/manuals/files/{file_id}",
[Scope.ROMS_WRITE],
responses={status.HTTP_404_NOT_FOUND: {}},
)
async def delete_rom_manual_file(
request: Request,
id: Annotated[int, PathVar(description="Rom internal id.", ge=1)],
file_id: Annotated[int, PathVar(description="Rom file internal id.", ge=1)],
) -> Response:
"""Delete a single manual file from a ROM's manual/ subfolder."""
rom = db_rom_handler.get_rom(id)
if not rom:
raise RomNotFoundInDatabaseException(id)
rom_file = db_rom_handler.get_rom_file_by_id(file_id)
if (
not rom_file
or rom_file.rom_id != rom.id
or rom_file.category != RomFileCategory.MANUAL
):
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Manual file not found",
)
file_rel_path = rom_file.full_path
try:
await fs_rom_handler.remove_file(file_rel_path)
except FileNotFoundError:
log.warning(
f"Manual file {hl(file_rel_path)} not found on disk; "
f"removing DB row anyway"
)
except Exception as exc:
log.error(f"Error deleting manual file {hl(file_rel_path)}", exc_info=exc)
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="There was an error deleting the manual file",
) from exc
db_rom_handler.delete_rom_file(file_id)
log.info(
f"Deleted manual file {hl(rom_file.file_name)} from "
f"{hl(rom.name or 'ROM', color=BLUE)} [{hl(rom.fs_name)}]"
)
return Response()
backend/endpoints/roms/notes.py
+6 -2
View File
@@ -100,8 +100,10 @@ async def create_rom_note(
tags=note_data.get("tags", []),
)
# Add username to the note data
# Add author identity to the note data
note["username"] = request.user.username
note["user_avatar_path"] = request.user.avatar_path
note["user_updated_at"] = request.user.updated_at
return UserNoteSchema.model_validate(note)
@@ -133,8 +135,10 @@ async def update_rom_note(
status_code=404, detail="Note not found or not owned by user"
)
# Add username to the note data
# Add author identity to the note data
note["username"] = request.user.username
note["user_avatar_path"] = request.user.avatar_path
note["user_updated_at"] = request.user.updated_at
return UserNoteSchema.model_validate(note)
backend/endpoints/roms/screenshot.py
+200
View File
@@ -0,0 +1,200 @@
import os
from typing import Annotated
from fastapi import Header, HTTPException
from fastapi import Path as PathVar
from fastapi import Request, status
from fastapi.responses import Response
from starlette.requests import ClientDisconnect
from streaming_form_data import StreamingFormDataParser
from streaming_form_data.targets import FileTarget, NullTarget
from decorators.auth import protected_route
from exceptions.endpoint_exceptions import RomNotFoundInDatabaseException
from handler.auth.constants import Scope
from handler.database import db_rom_handler
from handler.filesystem import fs_rom_handler
from logger.formatter import BLUE
from logger.formatter import highlight as hl
from logger.logger import log
from models.rom import RomFile, RomFileCategory
from utils.router import APIRouter
from utils.screenshots import (
ALLOWED_SCREENSHOT_EXTENSIONS,
is_allowed_screenshot_file,
)
router = APIRouter()
SCREENSHOT_FOLDER = "screenshots"
@protected_route(
router.post,
"/{id}/screenshots",
[Scope.ROMS_WRITE],
status_code=status.HTTP_201_CREATED,
responses={status.HTTP_404_NOT_FOUND: {}},
)
async def add_rom_screenshots(
request: Request,
id: Annotated[int, PathVar(description="Rom internal id.", ge=1)],
filename: Annotated[
str,
Header(
description="The name of the file being uploaded.",
alias="x-upload-filename",
),
],
) -> Response:
"""Upload a screenshot image for a multi-file ROM."""
rom = db_rom_handler.get_rom(id)
if not rom:
raise RomNotFoundInDatabaseException(id)
if rom.has_simple_single_file:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="Screenshots can only be uploaded to folder-based ROMs",
)
try:
safe_filename = fs_rom_handler._sanitize_filename(filename)
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Invalid upload filename: {exc}",
) from exc
# Reject rather than silently strip — otherwise the client's form-field
# name won't match what we register with the parser below.
if safe_filename != filename:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="Upload filename must be a plain file name, not a path",
)
if not is_allowed_screenshot_file(safe_filename):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=(
f"Unsupported image file type. Allowed: "
f"{', '.join(sorted(ALLOWED_SCREENSHOT_EXTENSIONS))}"
),
)
screenshot_dir_rel = f"{rom.full_path}/{SCREENSHOT_FOLDER}"
file_rel_path = f"{screenshot_dir_rel}/{safe_filename}"
file_location = fs_rom_handler.validate_path(file_rel_path)
log.info(f"Uploading screenshot to {hl(str(file_location))}")
await fs_rom_handler.make_directory(screenshot_dir_rel)
parser = StreamingFormDataParser(headers=request.headers)
parser.register("x-upload-platform", NullTarget())
parser.register(safe_filename, FileTarget(str(file_location)))
def cleanup_partial_file():
if file_location.exists():
file_location.unlink()
try:
async for chunk in request.stream():
parser.data_received(chunk)
except ClientDisconnect:
log.error("Client disconnected during upload")
cleanup_partial_file()
raise
except Exception as exc:
log.error("Error uploading screenshot", exc_info=exc)
cleanup_partial_file()
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="There was an error uploading the screenshot",
) from exc
stat = os.stat(file_location)
existing = db_rom_handler.get_rom_file_by_path(
rom_id=rom.id, file_path=screenshot_dir_rel, file_name=safe_filename
)
if existing:
db_rom_handler.update_rom_file(
existing.id,
{
"file_size_bytes": stat.st_size,
"last_modified": stat.st_mtime,
"category": RomFileCategory.SCREENSHOT,
"missing_from_fs": False,
},
)
else:
db_rom_handler.add_rom_file(
RomFile(
rom_id=rom.id,
file_name=safe_filename,
file_path=screenshot_dir_rel,
file_size_bytes=stat.st_size,
last_modified=stat.st_mtime,
category=RomFileCategory.SCREENSHOT,
)
)
return Response(status_code=status.HTTP_201_CREATED)
@protected_route(
router.delete,
"/{id}/screenshots/{file_id}",
[Scope.ROMS_WRITE],
responses={status.HTTP_404_NOT_FOUND: {}},
)
async def delete_rom_screenshot(
request: Request,
id: Annotated[int, PathVar(description="Rom internal id.", ge=1)],
file_id: Annotated[int, PathVar(description="Rom file internal id.", ge=1)],
) -> Response:
"""Delete a single screenshot file from a ROM."""
rom = db_rom_handler.get_rom(id)
if not rom:
raise RomNotFoundInDatabaseException(id)
rom_file = db_rom_handler.get_rom_file_by_id(file_id)
if (
not rom_file
or rom_file.rom_id != rom.id
or rom_file.category != RomFileCategory.SCREENSHOT
):
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Screenshot file not found",
)
file_rel_path = rom_file.full_path
try:
await fs_rom_handler.remove_file(file_rel_path)
except FileNotFoundError:
log.warning(
f"Screenshot file {hl(file_rel_path)} not found on disk; "
f"removing DB row anyway"
)
except Exception as exc:
log.error(
f"Error deleting screenshot {hl(file_rel_path)}",
exc_info=exc,
)
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="There was an error deleting the screenshot",
) from exc
db_rom_handler.delete_rom_file(file_id)
log.info(
f"Deleted screenshot {hl(rom_file.file_name)} from "
f"{hl(rom.name or 'ROM', color=BLUE)} [{hl(rom.fs_name)}]"
)
return Response()
backend/endpoints/roms/soundtrack.py
+259
View File
@@ -0,0 +1,259 @@
import os
from typing import Annotated
from fastapi import Header, HTTPException
from fastapi import Path as PathVar
from fastapi import Request, status
from fastapi.responses import Response
from starlette.requests import ClientDisconnect
from streaming_form_data import StreamingFormDataParser
from streaming_form_data.targets import FileTarget, NullTarget
from decorators.auth import protected_route
from endpoints.responses.rom import RomFileAudioMetaSchema, SoundtrackTrackMetaSchema
from exceptions.endpoint_exceptions import RomNotFoundInDatabaseException
from handler.auth.constants import Scope
from handler.database import db_rom_handler
from handler.filesystem import fs_rom_handler
from logger.formatter import BLUE
from logger.formatter import highlight as hl
from logger.logger import log
from models.rom import RomFile, RomFileCategory
from utils.audio_tags import (
ALLOWED_AUDIO_EXTENSIONS,
extract_audio_meta,
is_allowed_audio_file,
persist_cover_and_build_meta,
remove_persisted_cover,
)
from utils.router import APIRouter
router = APIRouter()
SOUNDTRACK_FOLDER = "soundtrack"
@protected_route(
router.get,
"/{id}/soundtracks/metadata",
[Scope.ROMS_READ],
responses={status.HTTP_404_NOT_FOUND: {}},
)
async def get_rom_soundtrack_metadata(
request: Request,
id: Annotated[int, PathVar(description="Rom internal id.", ge=1)],
) -> list[SoundtrackTrackMetaSchema]:
"""Return compact audio metadata for every soundtrack file of a ROM."""
rom = db_rom_handler.get_rom(id)
if not rom:
raise RomNotFoundInDatabaseException(id)
tracks = db_rom_handler.get_rom_files_by_category(
rom_id=rom.id, category=RomFileCategory.SOUNDTRACK
)
return [
SoundtrackTrackMetaSchema(
file_id=f.id,
file_name=f.file_name,
file_size_bytes=f.file_size_bytes,
audio_meta=(
RomFileAudioMetaSchema.model_validate(f.audio_meta)
if f.audio_meta
else None
),
)
for f in tracks
]
@protected_route(
router.post,
"/{id}/soundtracks",
[Scope.ROMS_WRITE],
status_code=status.HTTP_201_CREATED,
responses={status.HTTP_404_NOT_FOUND: {}},
)
async def add_rom_soundtracks(
request: Request,
id: Annotated[int, PathVar(description="Rom internal id.", ge=1)],
filename: Annotated[
str,
Header(
description="The name of the file being uploaded.",
alias="x-upload-filename",
),
],
) -> Response:
"""Upload a soundtrack audio file for a multi-file ROM."""
rom = db_rom_handler.get_rom(id)
if not rom:
raise RomNotFoundInDatabaseException(id)
if rom.has_simple_single_file:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="Soundtracks can only be uploaded to folder-based ROMs",
)
try:
safe_filename = fs_rom_handler._sanitize_filename(filename)
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Invalid upload filename: {exc}",
) from exc
# Reject rather than silently strip — otherwise the client's form-field
# name won't match what we register with the parser below.
if safe_filename != filename:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="Upload filename must be a plain file name, not a path",
)
if not is_allowed_audio_file(safe_filename):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=(
f"Unsupported audio file type. Allowed: "
f"{', '.join(sorted(ALLOWED_AUDIO_EXTENSIONS))}"
),
)
soundtrack_dir_rel = f"{rom.full_path}/{SOUNDTRACK_FOLDER}"
file_rel_path = f"{soundtrack_dir_rel}/{safe_filename}"
file_location = fs_rom_handler.validate_path(file_rel_path)
log.info(f"Uploading soundtrack to {hl(str(file_location))}")
await fs_rom_handler.make_directory(soundtrack_dir_rel)
parser = StreamingFormDataParser(headers=request.headers)
parser.register("x-upload-platform", NullTarget())
parser.register(safe_filename, FileTarget(str(file_location)))
def cleanup_partial_file():
if file_location.exists():
file_location.unlink()
try:
async for chunk in request.stream():
parser.data_received(chunk)
except ClientDisconnect:
log.error("Client disconnected during upload")
cleanup_partial_file()
raise
except Exception as exc:
log.error("Error uploading soundtrack", exc_info=exc)
cleanup_partial_file()
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="There was an error uploading the soundtrack",
) from exc
stat = os.stat(file_location)
audio_meta = extract_audio_meta(str(file_location))
existing = db_rom_handler.get_rom_file_by_path(
rom_id=rom.id, file_path=soundtrack_dir_rel, file_name=safe_filename
)
if existing:
# Reuploading: drop stale cover first so a new id-based path can replace it.
if existing.audio_meta and existing.audio_meta.get("cover_path"):
remove_persisted_cover(existing.audio_meta["cover_path"])
saved = db_rom_handler.update_rom_file(
existing.id,
{
"file_size_bytes": stat.st_size,
"last_modified": stat.st_mtime,
"category": RomFileCategory.SOUNDTRACK,
"audio_meta": audio_meta,
"missing_from_fs": False,
},
)
else:
saved = db_rom_handler.add_rom_file(
RomFile(
rom_id=rom.id,
file_name=safe_filename,
file_path=soundtrack_dir_rel,
file_size_bytes=stat.st_size,
last_modified=stat.st_mtime,
category=RomFileCategory.SOUNDTRACK,
audio_meta=audio_meta,
)
)
if saved and audio_meta and audio_meta.get("has_embedded_cover"):
persisted_meta = persist_cover_and_build_meta(
audio_full_path=str(file_location),
platform_id=rom.platform_id,
rom_id=rom.id,
file_id=saved.id,
audio_meta=audio_meta,
)
if persisted_meta:
db_rom_handler.update_rom_file(saved.id, {"audio_meta": persisted_meta})
return Response(status_code=status.HTTP_201_CREATED)
@protected_route(
router.delete,
"/{id}/soundtracks/{file_id}",
[Scope.ROMS_WRITE],
responses={status.HTTP_404_NOT_FOUND: {}},
)
async def delete_rom_soundtrack(
request: Request,
id: Annotated[int, PathVar(description="Rom internal id.", ge=1)],
file_id: Annotated[int, PathVar(description="Rom file internal id.", ge=1)],
) -> Response:
"""Delete a single soundtrack file from a ROM."""
rom = db_rom_handler.get_rom(id)
if not rom:
raise RomNotFoundInDatabaseException(id)
rom_file = db_rom_handler.get_rom_file_by_id(file_id)
if (
not rom_file
or rom_file.rom_id != rom.id
or rom_file.category != RomFileCategory.SOUNDTRACK
):
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Soundtrack file not found",
)
file_rel_path = rom_file.full_path
try:
await fs_rom_handler.remove_file(file_rel_path)
except FileNotFoundError:
log.warning(
f"Soundtrack file {hl(file_rel_path)} not found on disk; "
f"removing DB row anyway"
)
except Exception as exc:
log.error(
f"Error deleting soundtrack {hl(file_rel_path)}",
exc_info=exc,
)
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="There was an error deleting the soundtrack",
) from exc
if rom_file.audio_meta and rom_file.audio_meta.get("cover_path"):
remove_persisted_cover(rom_file.audio_meta["cover_path"])
db_rom_handler.delete_rom_file(file_id)
log.info(
f"Deleted soundtrack {hl(rom_file.file_name)} from "
f"{hl(rom.name or 'ROM', color=BLUE)} [{hl(rom.fs_name)}]"
)
return Response()
backend/endpoints/roms/upload.py
+5 -2
View File
@@ -1,6 +1,5 @@
import json
import shutil
import tempfile
from pathlib import Path
from typing import Annotated
from uuid import UUID, uuid4
@@ -9,6 +8,7 @@ from anyio import open_file
from fastapi import Header, HTTPException, Request, status
from starlette.responses import Response
from config import ROMM_BASE_PATH, ROMM_TMP_PATH
from decorators.auth import protected_route
from handler.auth.constants import Scope
from handler.database import db_platform_handler
@@ -22,7 +22,10 @@ router = APIRouter(
tags=["upload"],
)
ROM_UPLOAD_TMP_BASE = Path(tempfile.gettempdir()) / "romm" / "uploads"
# Store upload chunks under ROMM_BASE_PATH (disk-backed) by default.
# Users can override the tmp location with the ROMM_TMP_PATH env variable.
_tmp_root = Path(ROMM_TMP_PATH) if ROMM_TMP_PATH else Path(ROMM_BASE_PATH)
ROM_UPLOAD_TMP_BASE = _tmp_root / "tmp" / "uploads"
ROM_UPLOAD_TTL = 86400 # 24 hours
ROM_ASSEMBLY_CHUNK_SIZE = 8192 # 8KB read buffer during assembly
ROM_UPLOAD_MAX_CHUNK_SIZE = 64 * 1024 * 1024 # 64MB hard cap per chunk
backend/endpoints/saves.py
+95 -52
View File
@@ -1,5 +1,6 @@
import os
import re
from collections.abc import Sequence
from datetime import datetime, timezone
from typing import Annotated
@@ -34,34 +35,66 @@ from utils.router import APIRouter
def _build_save_schema(
save: Save,
syncs: Sequence[tuple[DeviceSaveSync, str | None]] = (),
device: Device | None = None,
sync: DeviceSaveSync | None = None,
) -> SaveSchema:
"""Attach one ``DeviceSyncSchema`` per device that has synced this save.
``syncs`` is the full list of sync rows (paired with device name) for this
save across every device, so clients can attribute the save to its creator.
``device`` is the caller's device, when supplied: its entry is emitted first
for stable ordering and old-client compatibility, and a placeholder entry is
synthesized when the caller has not yet synced this save.
"""
save_schema = SaveSchema.model_validate(save)
if device:
if sync:
is_current = to_utc(sync.last_synced_at) >= to_utc(save.updated_at)
last_synced = sync.last_synced_at
is_untracked = sync.is_untracked
else:
is_current = False
last_synced = save.updated_at
is_untracked = False
save_updated = to_utc(save.updated_at)
caller_present = False
entries: list[DeviceSyncSchema] = []
for sync, device_name in syncs:
if device and sync.device_id == device.id:
caller_present = True
entries.append(
DeviceSyncSchema(
device_id=sync.device_id,
device_name=device_name,
last_synced_at=sync.last_synced_at,
is_untracked=sync.is_untracked,
is_current=to_utc(sync.last_synced_at) >= save_updated,
)
)
save_schema.device_syncs = [
if device and not caller_present:
entries.append(
DeviceSyncSchema(
device_id=device.id,
device_name=device.name,
last_synced_at=last_synced,
is_untracked=is_untracked,
is_current=is_current,
last_synced_at=save.updated_at,
is_untracked=False,
is_current=False,
)
]
)
if device:
entries.sort(key=lambda entry: entry.device_id != device.id)
save_schema.device_syncs = entries
return save_schema
def _syncs_for_save(
save_id: int, device: Device | None
) -> list[tuple[DeviceSaveSync, str | None]]:
"""Fetch every device sync for a single save when a device is in context.
Device attribution is only meaningful to a device-scoped caller, so callers
without a device get an empty list and no query is issued.
"""
if not device:
return []
return db_device_save_sync_handler.get_syncs_for_saves([save_id]).get(save_id, [])
DATETIME_TAG_PATTERN = re.compile(r" \[\d{4}-\d{2}-\d{2}_\d{2}-\d{2}-\d{2}\]")
@@ -162,7 +195,7 @@ async def add_save(
actual_filename = _apply_datetime_tag(sanitized_save_filename)
db_save = db_save_handler.get_save_by_filename(
user_id=request.user.id, rom_id=rom.id, file_name=actual_filename
user_id=request.user.id, rom_id=rom.id, file_name=actual_filename, slot=slot
)
if device and slot and not overwrite:
@@ -222,18 +255,16 @@ async def add_save(
user_id=request.user.id,
rom_id=rom.id,
content_hash=scanned_save.content_hash,
slot=slot,
)
if existing_by_hash:
try:
await fs_asset_handler.remove_file(f"{saves_path}/{actual_filename}")
except FileNotFoundError:
pass
sync = None
if device:
sync = db_device_save_sync_handler.get_sync(
device_id=device.id, save_id=existing_by_hash.id
)
return _build_save_schema(existing_by_hash, device, sync)
return _build_save_schema(
existing_by_hash, _syncs_for_save(existing_by_hash.id, device), device
)
if db_save:
update_data: dict = {
@@ -248,6 +279,7 @@ async def add_save(
scanned_save.user_id = request.user.id
scanned_save.emulator = emulator
scanned_save.slot = slot
scanned_save.origin_device_id = device.id if device else None
db_save = db_save_handler.add_save(save=scanned_save)
if device:
@@ -321,12 +353,7 @@ async def add_save(
rom_user.id, {"last_played": datetime.now(timezone.utc)}
)
sync = None
if device:
sync = db_device_save_sync_handler.get_sync(
device_id=device.id, save_id=db_save.id
)
return _build_save_schema(db_save, device, sync)
return _build_save_schema(db_save, _syncs_for_save(db_save.id, device), device)
@protected_route(router.get, "", [Scope.ASSETS_READ])
@@ -349,13 +376,13 @@ def get_saves(
if not device:
return [_build_save_schema(save) for save in saves]
syncs = db_device_save_sync_handler.get_syncs_for_device_and_saves(
device_id=device.id, save_ids=[s.id for s in saves]
syncs_by_save_id = db_device_save_sync_handler.get_syncs_for_saves(
[s.id for s in saves]
)
sync_by_save_id = {s.save_id: s for s in syncs}
return [
_build_save_schema(save, device, sync_by_save_id.get(save.id)) for save in saves
_build_save_schema(save, syncs_by_save_id.get(save.id, []), device)
for save in saves
]
@@ -403,12 +430,7 @@ def get_save(request: Request, id: int, device_id: str | None = None) -> SaveSch
detail=f"Save with ID {id} not found",
)
sync = None
if device:
sync = db_device_save_sync_handler.get_sync(
device_id=device.id, save_id=save.id
)
return _build_save_schema(save, device, sync)
return _build_save_schema(save, _syncs_for_save(save.id, device), device)
@protected_route(router.get, "/{id}/content", [Scope.ASSETS_READ])
@@ -424,12 +446,14 @@ def download_save(
device_id, request.user.id, request.auth.scopes, Scope.DEVICES_READ
)
save = db_save_handler.get_save(user_id=request.user.id, id=id)
if not save:
# Owner can download any of their saves; everyone else only public ones.
save = db_save_handler.get_save_by_id(id)
if not save or (save.user_id != request.user.id and not save.is_public):
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Save with ID {id} not found",
)
is_owner = save.user_id == request.user.id
try:
file_path = fs_asset_handler.validate_path(save.full_path)
@@ -445,7 +469,8 @@ def download_save(
detail="Save file not found on disk",
)
if device and optimistic:
# Sync bookkeeping only makes sense for the owner's own saves.
if device and optimistic and is_owner:
db_device_save_sync_handler.upsert_sync(
device_id=device.id,
save_id=save.id,
@@ -474,14 +499,14 @@ def confirm_download(
)
device = _resolve_device(device_id, request.user.id)
sync = db_device_save_sync_handler.upsert_sync(
db_device_save_sync_handler.upsert_sync(
device_id=device_id,
save_id=save.id,
synced_at=save.updated_at,
)
db_device_handler.update_last_seen(device_id=device_id, user_id=request.user.id)
return _build_save_schema(save, device, sync)
return _build_save_schema(save, _syncs_for_save(save.id, device), device)
@protected_route(router.put, "/{id}", [Scope.ASSETS_WRITE])
@@ -580,12 +605,30 @@ async def update_save(
)
db_device_handler.update_last_seen(device_id=device.id, user_id=request.user.id)
sync = None
if device:
sync = db_device_save_sync_handler.get_sync(
device_id=device.id, save_id=db_save.id
return _build_save_schema(db_save, _syncs_for_save(db_save.id, device), device)
@protected_route(
router.put,
"/{id}/visibility",
[Scope.ASSETS_WRITE],
responses={status.HTTP_404_NOT_FOUND: {}},
)
def update_save_visibility(
request: Request,
id: int,
is_public: Annotated[bool, Body(embed=True)],
) -> SaveSchema:
"""Toggle a save's public/private visibility (owner only)."""
save = db_save_handler.get_save_by_id(id)
if not save or save.user_id != request.user.id:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Save with ID {id} not found",
)
return _build_save_schema(db_save, device, sync)
updated = db_save_handler.update_save(id, {"is_public": is_public})
return _build_save_schema(updated)
@protected_route(
@@ -660,11 +703,11 @@ def track_save(
)
device = _resolve_device(device_id, request.user.id)
sync = db_device_save_sync_handler.set_untracked(
db_device_save_sync_handler.set_untracked(
device_id=device_id, save_id=id, untracked=False
)
return _build_save_schema(save, device, sync)
return _build_save_schema(save, _syncs_for_save(save.id, device), device)
@protected_route(router.post, "/{id}/untrack", [Scope.DEVICES_WRITE])
@@ -682,8 +725,8 @@ def untrack_save(
)
device = _resolve_device(device_id, request.user.id)
sync = db_device_save_sync_handler.set_untracked(
db_device_save_sync_handler.set_untracked(
device_id=device_id, save_id=id, untracked=True
)
return _build_save_schema(save, device, sync)
return _build_save_schema(save, _syncs_for_save(save.id, device), device)
backend/endpoints/screenshots.py
+87 -7
View File
@@ -1,4 +1,9 @@
from fastapi import File, HTTPException, Request, UploadFile, status
from typing import Annotated
from fastapi import Body, File, HTTPException
from fastapi import Path as PathVar
from fastapi import Request, UploadFile, status
from fastapi.responses import Response
from decorators.auth import protected_route
from endpoints.responses.assets import ScreenshotSchema
@@ -12,6 +17,10 @@ from logger.formatter import highlight as hl
from logger.logger import log
from utils.filesystem import sanitize_filename
from utils.router import APIRouter
from utils.screenshots import (
ALLOWED_SCREENSHOT_EXTENSIONS,
is_allowed_screenshot_file,
)
router = APIRouter(
prefix="/screenshots",
@@ -27,12 +36,18 @@ async def add_screenshot(
rom_id: int,
screenshotFile: UploadFile = SCREENSHOT_FILE_UPLOAD,
) -> ScreenshotSchema:
"""Upload a per-user gallery screenshot for a ROM.
Stored under the user's asset folder (not the ROM folder) and flagged
`is_gallery=True` so it surfaces in the gallery; `is_public=False` so it
stays private until the owner shares it.
"""
rom = db_rom_handler.get_rom(id=rom_id)
if not rom:
raise RomNotFoundInDatabaseException(rom_id)
current_user = request.user
log.info(f"Uploading screenshots to {hl(str(rom.name), color=BLUE)}")
log.info(f"Uploading screenshot to {hl(str(rom.name), color=BLUE)}")
screenshots_path = fs_asset_handler.build_screenshots_file_path(
user=request.user, platform_fs_slug=rom.platform_slug, rom_id=rom.id
@@ -53,6 +68,15 @@ async def add_screenshot(
detail=f"Invalid screenshot filename: {str(exc)}",
) from exc
if not is_allowed_screenshot_file(sanitized_screenshot_filename):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=(
f"Unsupported image file type. Allowed: "
f"{', '.join(sorted(ALLOWED_SCREENSHOT_EXTENSIONS))}"
),
)
await fs_asset_handler.write_file(
file=screenshotFile,
path=screenshots_path,
@@ -74,17 +98,73 @@ async def add_screenshot(
if db_screenshot:
db_screenshot = db_screenshot_handler.update_screenshot(
db_screenshot.id,
{"file_size_bytes": scanned_screenshot.file_size_bytes},
{
"file_size_bytes": scanned_screenshot.file_size_bytes,
"is_gallery": True,
"missing_from_fs": False,
},
)
else:
scanned_screenshot.rom_id = rom.id
scanned_screenshot.user_id = current_user.id
scanned_screenshot.is_gallery = True
db_screenshot = db_screenshot_handler.add_screenshot(
screenshot=scanned_screenshot
)
rom = db_rom_handler.get_rom(id=rom_id)
if not rom:
raise RomNotFoundInDatabaseException(rom_id)
return ScreenshotSchema.model_validate(db_screenshot)
@protected_route(
router.put,
"/{id}",
[Scope.ASSETS_WRITE],
responses={status.HTTP_404_NOT_FOUND: {}},
)
async def update_screenshot(
request: Request,
id: Annotated[int, PathVar(description="Screenshot internal id.", ge=1)],
is_public: Annotated[bool, Body(embed=True)],
) -> ScreenshotSchema:
"""Toggle a gallery screenshot's public/private visibility (owner only)."""
screenshot = db_screenshot_handler.get_screenshot_by_id(id)
if not screenshot or screenshot.user_id != request.user.id:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Screenshot not found",
)
updated = db_screenshot_handler.update_screenshot(id, {"is_public": is_public})
return ScreenshotSchema.model_validate(updated)
@protected_route(
router.delete,
"/{id}",
[Scope.ASSETS_WRITE],
responses={status.HTTP_404_NOT_FOUND: {}},
)
async def delete_screenshot(
request: Request,
id: Annotated[int, PathVar(description="Screenshot internal id.", ge=1)],
) -> Response:
"""Delete a gallery screenshot — its file and DB row (owner only)."""
screenshot = db_screenshot_handler.get_screenshot_by_id(id)
if not screenshot or screenshot.user_id != request.user.id:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Screenshot not found",
)
try:
await fs_asset_handler.remove_file(file_path=screenshot.full_path)
except FileNotFoundError:
log.warning(
f"Screenshot file {hl(screenshot.full_path)} not found on disk; "
f"removing DB row anyway"
)
db_screenshot_handler.delete_screenshot(screenshot.id)
log.info(f"Deleted screenshot {hl(screenshot.file_name)}")
return Response()
backend/endpoints/search.py
+2 -2
View File
@@ -100,7 +100,7 @@ async def search_rom(
if search_by.lower() == "id":
try:
igdb_rom, moby_rom, ss_rom, lb_rom = await asyncio.gather(
meta_igdb_handler.get_matched_rom_by_id(int(search_term)),
meta_igdb_handler.get_matched_rom_by_id(rom, int(search_term)),
meta_moby_handler.get_matched_rom_by_id(int(search_term)),
meta_ss_handler.get_matched_rom_by_id(rom, int(search_term)),
meta_launchbox_handler.get_matched_rom_by_id(int(search_term)),
@@ -125,7 +125,7 @@ async def search_rom(
launchbox_matched_roms,
) = await asyncio.gather(
meta_igdb_handler.get_matched_roms_by_name(
search_term, get_main_platform_igdb_id(rom.platform)
rom, search_term, get_main_platform_igdb_id(rom.platform)
),
meta_moby_handler.get_matched_roms_by_name(
search_term, rom.platform.moby_id
backend/endpoints/sockets/activity.py
+231
View File
@@ -0,0 +1,231 @@
"""Socket.IO events for real-time user game activity.
Handles:
- activity:start - client reports starting a game (emits activity:update)
- activity:heartbeat - client refreshes TTL while playing (emits activity:update)
- activity:stop - client reports stopping (emits activity:clear)
- disconnect - safety net: clears any activity registered for the socket
The acting user is never taken from the client payload — it is resolved from
the authenticated socket session (stored on connect, see ``endpoints.sockets``)
so a client cannot broadcast a "now playing" session on behalf of another user.
Only ``rom_id`` / ``device_id`` come from the client.
All events broadcast to every connected client on the main `/ws` namespace.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any, TypedDict
from endpoints.responses.activity import ActivityClearSchema
from handler.activity_handler import ActivityEntry, activity_handler
from handler.database import (
db_device_handler,
db_rom_handler,
db_save_handler,
db_user_handler,
)
from handler.socket_handler import socket_handler
from logger.logger import log
from utils.screenshots import continue_playing_screenshot
# Socket-session key holding the authenticated user id, written by the connect
# handler. Identity for every activity event is derived from this, not payload.
AUTH_USER_SESSION_KEY = "activity_auth_user_id"
class ActivityEventPayload(TypedDict, total=False):
rom_id: int
device_id: str
async def _session(sid: str) -> dict[str, Any]:
try:
return await socket_handler.socket_server.get_session(sid) or {}
except KeyError:
return {}
async def store_authenticated_user(sid: str, user_id: int) -> None:
"""Record the authenticated user for a socket so activity events trust it.
Called from the connect handler once the session has been resolved. Without
this, activity events have no identity to act on and are ignored.
"""
existing = await _session(sid)
existing[AUTH_USER_SESSION_KEY] = user_id
await socket_handler.socket_server.save_session(sid, existing)
async def _authenticated_user_id(sid: str) -> int | None:
"""Return the user id resolved at connect time, or ``None`` if unauthenticated."""
user_id = (await _session(sid)).get(AUTH_USER_SESSION_KEY)
return int(user_id) if user_id is not None else None
async def _store_session(sid: str, user_id: int, device_id: str) -> None:
"""Remember the user/device associated with a socket for disconnect cleanup."""
existing = await _session(sid)
existing["activity_user_id"] = user_id
existing["activity_device_id"] = device_id
await socket_handler.socket_server.save_session(sid, existing)
async def _build_entry(
*, user_id: int, device_id: str, rom_id: int, preserve_started_at: bool
) -> ActivityEntry | None:
"""Look up DB info and assemble an ActivityEntry. Returns None if invalid."""
user = db_user_handler.get_user(user_id)
if user is None:
log.debug(f"activity: unknown user_id {user_id}")
return None
rom = db_rom_handler.get_rom(rom_id)
if rom is None:
log.debug(f"activity: unknown rom_id {rom_id}")
return None
platform = rom.platform
started_at = datetime.now(timezone.utc).isoformat()
if preserve_started_at:
existing = await activity_handler.get_active(user_id, device_id)
if existing:
started_at = existing["started_at"]
device = db_device_handler.get_device(device_id=device_id, user_id=user_id)
device_type = device.client if device else None
# "Where they are" image — the player's latest save screenshot, else the
# title screen / first gameplay screenshot (frontend falls back to cover).
latest_save = db_save_handler.get_latest_saves_for_roms(
user_id=user_id, rom_ids=[rom_id]
).get(rom_id)
screenshot_path = continue_playing_screenshot(rom, latest_save) or ""
return ActivityEntry(
user_id=user.id,
username=user.username,
avatar_path=user.avatar_path or "",
rom_id=rom.id,
rom_name=rom.name or rom.fs_name,
rom_cover_path=rom.path_cover_s or "",
screenshot_path=screenshot_path,
platform_slug=platform.slug if platform else "",
platform_name=((platform.custom_name or platform.name) if platform else ""),
device_id=device_id,
device_type=device_type or "web",
started_at=started_at,
)
def _extract_payload(data: object) -> tuple[str | None, int | None]:
"""Return ``(device_id, rom_id)`` parsed from an event payload.
``user_id`` is deliberately not read from the client — identity comes from
the authenticated socket session, never the payload.
"""
if not isinstance(data, dict):
return None, None
device_id = data.get("device_id")
if not isinstance(device_id, str) or not device_id:
device_id = None
try:
data_rom_id = data.get("rom_id")
rom_id = int(data_rom_id) if data_rom_id else None
except (TypeError, ValueError):
rom_id = None
return device_id, rom_id
@socket_handler.socket_server.on("activity:start") # type: ignore
async def activity_start(sid: str, data: ActivityEventPayload) -> None:
user_id = await _authenticated_user_id(sid)
device_id, rom_id = _extract_payload(data)
if user_id is None or device_id is None or rom_id is None:
log.debug(f"activity:start ignored (unauthenticated or invalid): {data}")
return
entry = await _build_entry(
user_id=user_id,
device_id=device_id,
rom_id=rom_id,
preserve_started_at=False,
)
if entry is None:
return
await activity_handler.set_active(entry)
await _store_session(sid, user_id, device_id)
await socket_handler.socket_server.emit("activity:update", dict(entry))
@socket_handler.socket_server.on("activity:heartbeat") # type: ignore
async def activity_heartbeat(sid: str, data: ActivityEventPayload) -> None:
user_id = await _authenticated_user_id(sid)
device_id, rom_id = _extract_payload(data)
if user_id is None or device_id is None or rom_id is None:
return
entry = await _build_entry(
user_id=user_id,
device_id=device_id,
rom_id=rom_id,
preserve_started_at=True,
)
if entry is None:
return
await activity_handler.set_active(entry)
await _store_session(sid, user_id, device_id)
await socket_handler.socket_server.emit("activity:update", dict(entry))
@socket_handler.socket_server.on("activity:stop") # type: ignore
async def activity_stop(sid: str, data: ActivityEventPayload | None = None) -> None:
user_id = await _authenticated_user_id(sid)
device_id: str | None = None
if data:
device_id, _ = _extract_payload(data)
# Fall back to the device stored at start time if the payload omits it.
if not device_id:
device_id = (await _session(sid)).get("activity_device_id")
if user_id is None or not device_id:
return
rom_id = await activity_handler.clear_active(int(user_id), device_id)
if rom_id is None:
return
await socket_handler.socket_server.emit(
"activity:clear",
ActivityClearSchema(
user_id=int(user_id), device_id=device_id, rom_id=rom_id
).model_dump(),
)
@socket_handler.socket_server.on("disconnect") # type: ignore
async def activity_on_disconnect(sid: str) -> None:
"""Safety net: clear any activity tied to a disconnecting socket."""
session = await _session(sid)
user_id = session.get("activity_user_id")
device_id = session.get("activity_device_id")
if user_id is None or not device_id:
return
rom_id = await activity_handler.clear_active(int(user_id), device_id)
if rom_id is None:
return
await socket_handler.socket_server.emit(
"activity:clear",
ActivityClearSchema(
user_id=int(user_id), device_id=device_id, rom_id=rom_id
).model_dump(),
)
backend/endpoints/sockets/logs.py
+163
View File
@@ -0,0 +1,163 @@
"""Real-time backend log streaming over Socket.IO (admin only).
Pieces:
- ``connect`` handler on the main socket server: resolves the session user and,
if they are an admin, joins them to the ``admin`` room. It never rejects a
connection, so the existing scan/sync sockets keep working for everyone.
- ``start_log_forwarder``: a single background task (Redis-lock guarded) that
subscribes to the ``romm:logs`` pub/sub channel — fed by ``LogStreamHandler``
in every backend process — and relays each line to the ``admin`` room.
- ``get_recent_logs``: reads the capped ring buffer for backfill on view open.
"""
import asyncio
import uuid
from typing import Any, Final
import socketio # type: ignore
from config import DISABLE_LOGS_VIEWER, REDIS_URL
from endpoints.sockets.activity import store_authenticated_user
from handler.database import db_user_handler
from handler.redis_handler import async_cache
from handler.socket_handler import socket_handler
from logger.log_stream_handler import LOG_BUFFER_KEY, LOG_CHANNEL
from logger.logger import log
from models.user import Role
from utils import json_module
from utils.auth import get_session_from_environ
ADMIN_ROOM: Final = "admin"
FORWARDER_LOCK_KEY: Final = "romm:logs:forwarder"
FORWARDER_LOCK_TTL: Final = 30 # seconds
@socket_handler.socket_server.on("connect") # type: ignore
async def connect(sid: str, environ: dict[str, Any], auth: Any = None) -> None:
"""Resolve the authenticated user on socket connect.
Stores the user id in the socket session so activity events can trust the
server-resolved identity instead of a client-supplied ``user_id``, and joins
admin users to the log-streaming room. Always returns ``None`` (accepts the
connection) — only identity storage and room membership are gated, so the
existing scan/sync sockets keep working for everyone.
"""
try:
session = await get_session_from_environ(environ)
if session.get("iss") != "romm:auth":
return
username = session.get("sub")
if not username:
return
user = db_user_handler.get_user_by_username(username)
if not user or not user.enabled:
return
await store_authenticated_user(sid, user.id)
if not DISABLE_LOGS_VIEWER and user.role == Role.ADMIN:
await socket_handler.socket_server.enter_room(sid, ADMIN_ROOM)
except Exception: # noqa: BLE001 - never let auth resolution refuse a socket
log.exception("Failed to resolve user on socket connect")
async def get_recent_logs(limit: int) -> list[dict[str, Any]]:
"""Return the most recent buffered log lines in chronological order."""
raw = await async_cache.lrange(LOG_BUFFER_KEY, 0, max(0, limit - 1))
entries: list[dict[str, Any]] = []
for item in raw:
try:
entries.append(json_module.loads(item))
except Exception: # noqa: BLE001 - skip malformed entry # nosec B112
continue
# The buffer is newest-first (LPUSH); callers want oldest-first.
entries.reverse()
return entries
async def start_log_forwarder() -> None:
"""Relay log lines from Redis pub/sub to admin Socket.IO clients.
Guarded by a Redis lock so that, when more than one web worker is running
(WEB_SERVER_CONCURRENCY > 1), exactly one forwards and clients don't receive
duplicate lines.
"""
if DISABLE_LOGS_VIEWER:
return
lock_id = str(uuid.uuid4())
pubsub = None
# Write-only manager for emitting — the same proven path scan/sync use. It
# publishes the room emit to Redis; the main socket server's read-side
# manager resolves `admin` room membership and delivers. Created inside the
# running loop (like sync.py) rather than at import time.
socket_manager = socketio.AsyncRedisManager(REDIS_URL, write_only=True)
try:
while True:
got_lock = await async_cache.set(
FORWARDER_LOCK_KEY, lock_id, nx=True, ex=FORWARDER_LOCK_TTL
)
if not got_lock:
# Another worker owns the forwarder; check back periodically in
# case it dies and the lock expires.
await asyncio.sleep(FORWARDER_LOCK_TTL / 2)
continue
pubsub = async_cache.pubsub()
await pubsub.subscribe(LOG_CHANNEL)
log.info("Log stream forwarder started")
try:
while True:
# Heartbeat the lock, but only while we still own it. If
# another worker took it (after a stall / Redis hiccup),
# stop forwarding to avoid duplicate lines — the outer loop
# will then contend for the lock again.
if await async_cache.get(FORWARDER_LOCK_KEY) != lock_id:
break
await async_cache.set(
FORWARDER_LOCK_KEY, lock_id, ex=FORWARDER_LOCK_TTL
)
message = await pubsub.get_message(
ignore_subscribe_messages=True,
timeout=FORWARDER_LOCK_TTL / 3,
)
if not message:
continue
data = message.get("data")
if not data:
continue
# One bad message or emit must not kill the forwarder —
# swallow per-line and keep relaying.
try:
payload = json_module.loads(data)
await socket_manager.emit(
"logs:entry", payload, room=ADMIN_ROOM
)
except Exception: # noqa: BLE001 - keep forwarding # nosec B112
continue
finally:
await pubsub.unsubscribe(LOG_CHANNEL)
await pubsub.aclose() # type: ignore[attr-defined]
pubsub = None
except asyncio.CancelledError:
if pubsub is not None:
await pubsub.aclose() # type: ignore[attr-defined]
# Release the lock on shutdown so a restart (e.g. uvicorn --reload)
# resumes forwarding immediately instead of waiting out the TTL.
await _release_lock(lock_id)
raise
except Exception: # noqa: BLE001 - never let the forwarder kill the app
log.exception("Log forwarder crashed")
await _release_lock(lock_id)
async def _release_lock(lock_id: str) -> None:
"""Delete the forwarder lock only if we still own it."""
try:
current = await async_cache.get(FORWARDER_LOCK_KEY)
if current == lock_id:
await async_cache.delete(FORWARDER_LOCK_KEY)
except Exception: # noqa: BLE001 - best-effort cleanup # nosec B110
pass
backend/endpoints/sockets/netplay.py
+1 -1
View File
@@ -172,7 +172,7 @@ async def webrtc_signal(sid: str, data: WebRTCSignalData):
)
else:
if not target:
return # drop message—no recipient
return # drop message, no recipient
await netplay_socket_handler.socket_server.emit(
"webrtc-signal",
{
backend/endpoints/sockets/scan.py
+88 -35
View File
@@ -11,6 +11,7 @@ from rq import Worker
from rq.job import Job
from config import DEV_MODE, REDIS_URL, SCAN_TIMEOUT, SCAN_WORKERS, TASK_RESULT_TTL
from config.config_manager import MetadataMediaType
from config.config_manager import config_manager as cm
from endpoints.responses import TaskType
from endpoints.responses.platform import PlatformSchema
@@ -31,11 +32,12 @@ from handler.filesystem import (
)
from handler.filesystem.roms_handler import FSRom
from handler.metadata import meta_gamelist_handler, meta_hltb_handler
from handler.metadata.ss_handler import get_preferred_media_types
from handler.metadata.ss_handler import add_ss_auth_to_url, get_preferred_media_types
from handler.redis_handler import get_job_func_name, high_prio_queue, redis_client
from handler.scan_handler import (
MetadataSource,
ScanType,
persist_soundtrack_cover,
scan_firmware,
scan_platform,
scan_rom,
@@ -159,6 +161,10 @@ def _should_scan_rom(
metadata_sources (list[str]): List of metadata sources to be used.
"""
# When roms_ids is provided, the scan is scoped to those roms only
if roms_ids:
return bool(rom and rom.id in roms_ids)
# This logic is tricky so only touch it if you know what you're doing"""
should_scan = bool(
# Any new roms should be scanned
@@ -170,19 +176,21 @@ def _should_scan_rom(
or (
rom
and (
# Selected ROMs are always scanned
(rom.id in roms_ids)
# Update scan should scan ROMs identified by the selected metadata sources
or (
(
scan_type == ScanType.UPDATE
and rom.is_identified
and any(getattr(rom, f"{source}_id") for source in metadata_sources)
and any(
getattr(rom, f"{source}_id", None)
for source in metadata_sources
)
)
# Unmatched scan should scan ROMs that are not identified by the selected metadata sources
or (
scan_type == ScanType.UNMATCHED
and any(
not getattr(rom, f"{source}_id") for source in metadata_sources
not getattr(rom, f"{source}_id", None)
for source in metadata_sources
)
)
)
@@ -206,12 +214,6 @@ def _should_get_rom_files(
newly_added (bool): Whether the rom is newly added.
roms_ids (list[int]): List of selected roms to be scanned.
"""
# Get hash calculation setting from config
calculate_hashes = not cm.get_config().SKIP_HASH_CALCULATION
# Skip file processing entirely if hashes are disabled (except for HASHES scan type)
if not calculate_hashes and scan_type != ScanType.HASHES:
return False
return bool(
newly_added
@@ -235,9 +237,9 @@ async def _identify_rom(
roms_ids: list[int],
metadata_sources: list[str],
launchbox_remote_enabled: bool,
playmatch_enabled: bool,
socket_manager: socketio.AsyncRedisManager,
scan_stats: ScanStats,
calculate_hashes: bool = True,
) -> None:
# Break early if the flag is set
if redis_client.get(STOP_SCAN_FLAG):
@@ -254,13 +256,6 @@ async def _identify_rom(
Rom(
fs_name=fs_rom["fs_name"],
fs_path=roms_path,
fs_name_no_tags=fs_rom_handler.get_file_name_with_no_tags(
fs_rom["fs_name"]
),
fs_name_no_ext=fs_rom_handler.get_file_name_with_no_extension(
fs_rom["fs_name"]
),
fs_extension=fs_rom_handler.parse_file_extension(fs_rom["fs_name"]),
regions=parsed_tags.regions,
revision=parsed_tags.revision,
version=parsed_tags.version,
@@ -300,6 +295,36 @@ async def _identify_rom(
}
)
# For a COMPLETE rescan, wipe all downloaded resources before re-fetching so
# stale files (e.g. a cover from the wrong region) can't be reused. The
# post-scan download steps below skip downloads when a file already exists or
# when the source URL is unchanged, so the on-disk files must be removed here.
if not newly_added and scan_type == ScanType.COMPLETE:
try:
await fs_resource_handler.remove_cover(rom)
except FileNotFoundError:
pass
try:
await fs_resource_handler.remove_manual(rom)
except FileNotFoundError:
pass
try:
await fs_resource_handler.remove_directory(
f"{rom.fs_resources_path}/screenshots"
)
except FileNotFoundError:
pass
for media_type in MetadataMediaType:
try:
await fs_resource_handler.remove_media_resources_path(
platform.id, rom.id, media_type
)
except FileNotFoundError:
pass
log.debug(f"Scanning {rom.fs_name}...")
scanned_rom = await scan_rom(
scan_type=scan_type,
@@ -309,6 +334,7 @@ async def _identify_rom(
metadata_sources=metadata_sources,
newly_added=newly_added,
launchbox_remote_enabled=launchbox_remote_enabled,
playmatch_enabled=playmatch_enabled,
socket_manager=socket_manager,
)
@@ -331,6 +357,7 @@ async def _identify_rom(
"rom_user",
"last_modified",
"files",
"sibling_roms",
}
),
)
@@ -348,15 +375,18 @@ async def _identify_rom(
file_size_bytes=file.file_size_bytes,
last_modified=file.last_modified,
category=file.category,
audio_meta=file.audio_meta,
crc_hash=file.crc_hash,
md5_hash=file.md5_hash,
sha1_hash=file.sha1_hash,
ra_hash=file.ra_hash,
chd_sha1_hash=file.chd_sha1_hash,
)
for file in fs_rom["files"]
]
for new_rom_file in new_rom_files:
db_rom_handler.add_rom_file(new_rom_file)
saved = db_rom_handler.add_rom_file(new_rom_file)
persist_soundtrack_cover(saved, _added_rom)
# Short circuit if the scan type is hashes
if scan_type == ScanType.HASHES:
@@ -365,22 +395,23 @@ async def _identify_rom(
path_cover_s, path_cover_l = await fs_resource_handler.get_cover(
entity=_added_rom,
overwrite=_added_rom.url_cover != rom.url_cover,
url_cover=_added_rom.url_cover,
url_cover=add_ss_auth_to_url(_added_rom.url_cover),
)
path_manual = await fs_resource_handler.get_manual(
rom=_added_rom,
overwrite=_added_rom.url_manual != rom.url_manual,
url_manual=_added_rom.url_manual,
url_manual=add_ss_auth_to_url(_added_rom.url_manual),
)
screenshots_changed = pydash.xor(
_added_rom.url_screenshots or [], rom.url_screenshots or []
)
url_screenshots = _added_rom.url_screenshots or []
path_screenshots = await fs_resource_handler.get_rom_screenshots(
rom=_added_rom,
overwrite=bool(screenshots_changed),
url_screenshots=_added_rom.url_screenshots,
url_screenshots=[add_ss_auth_to_url(u) for u in url_screenshots],
)
_added_rom.path_cover_s = path_cover_s
@@ -403,10 +434,12 @@ async def _identify_rom(
if _added_rom.ss_metadata and MetadataSource.SS in metadata_sources:
preferred_media_types = get_preferred_media_types()
for media_type in preferred_media_types:
if _added_rom.ss_metadata.get(f"{media_type.value}_path"):
media_path = _added_rom.ss_metadata.get(f"{media_type.value}_path")
media_url = _added_rom.ss_metadata.get(f"{media_type.value}_url")
if media_path and media_url:
await fs_resource_handler.store_media_file(
_added_rom.ss_metadata[f"{media_type.value}_url"],
_added_rom.ss_metadata[f"{media_type.value}_path"],
add_ss_auth_to_url(media_url),
media_path,
)
# Handle special media files from ES-DE gamelist.xml
@@ -419,6 +452,16 @@ async def _identify_rom(
_added_rom.gamelist_metadata[f"{media_type.value}_path"],
)
# Handle special media files from LaunchBox
if _added_rom.launchbox_metadata and MetadataSource.LAUNCHBOX in metadata_sources:
preferred_media_types = get_preferred_media_types()
for media_type in preferred_media_types:
if _added_rom.launchbox_metadata.get(f"{media_type.value}_path"):
await fs_resource_handler.store_media_file(
_added_rom.launchbox_metadata[f"{media_type.value}_url"],
_added_rom.launchbox_metadata[f"{media_type.value}_path"],
)
# Store normal and locked badges
if _added_rom.ra_metadata and MetadataSource.RA in metadata_sources:
for ach in _added_rom.ra_metadata.get("achievements", []):
@@ -436,7 +479,14 @@ async def _identify_rom(
await socket_manager.emit(
"scan:scanning_rom",
SimpleRomSchema.from_orm_with_factory(_added_rom).model_dump(
exclude={"created_at", "updated_at", "rom_user", "last_modified", "files"}
exclude={
"created_at",
"updated_at",
"rom_user",
"last_modified",
"files",
"sibling_roms",
}
),
)
@@ -448,9 +498,9 @@ async def _identify_platform(
roms_ids: list[int],
metadata_sources: list[str],
launchbox_remote_enabled: bool,
playmatch_enabled: bool,
socket_manager: socketio.AsyncRedisManager,
scan_stats: ScanStats,
calculate_hashes: bool = True,
) -> ScanStats:
# Stop the scan if the flag is set
if redis_client.get(STOP_SCAN_FLAG):
@@ -546,9 +596,9 @@ async def _identify_platform(
roms_ids=roms_ids,
metadata_sources=metadata_sources,
launchbox_remote_enabled=launchbox_remote_enabled,
playmatch_enabled=playmatch_enabled,
socket_manager=socket_manager,
scan_stats=scan_stats,
calculate_hashes=calculate_hashes,
)
for fs_roms_batch in batched(fs_roms, 200, strict=False):
@@ -619,6 +669,7 @@ async def scan_platforms(
scan_type: ScanType = ScanType.QUICK,
roms_ids: list[int] | None = None,
launchbox_remote_enabled: bool = True,
playmatch_enabled: bool = True,
) -> ScanStats:
"""Scan all the listed platforms and fetch metadata from different sources
@@ -628,10 +679,6 @@ async def scan_platforms(
scan_type (ScanType): Type of scan to be performed.
roms_ids (list[int], optional): List of selected roms to be scanned.
"""
# Get hash calculation setting from config
calculate_hashes = not cm.get_config().SKIP_HASH_CALCULATION
if not roms_ids:
roms_ids = []
@@ -698,9 +745,9 @@ async def scan_platforms(
roms_ids=roms_ids,
metadata_sources=metadata_sources,
launchbox_remote_enabled=launchbox_remote_enabled,
playmatch_enabled=playmatch_enabled,
socket_manager=socket_manager,
scan_stats=scan_stats,
calculate_hashes=calculate_hashes,
)
missed_platforms = db_platform_handler.mark_missing_platforms(fs_platforms)
@@ -711,6 +758,9 @@ async def scan_platforms(
log.info(f"{emoji.EMOJI_CHECK_MARK} Scan completed")
# The library changed; drop cached filter values.
db_rom_handler.invalidate_filter_values_cache()
# Export metadata files if enabled in config
config = cm.get_config()
platforms_by_slug = {p.fs_slug: p for p in db_platform_handler.get_platforms()}
@@ -783,6 +833,7 @@ async def scan_handler(_sid: str, options: dict[str, Any]):
roms_ids = options.get("roms_ids", [])
metadata_sources = options.get("apis", [])
launchbox_remote_enabled = bool(options.get("launchbox_remote_enabled", True))
playmatch_enabled = bool(options.get("playmatch_enabled", True))
if DEV_MODE:
return await scan_platforms(
@@ -791,6 +842,7 @@ async def scan_handler(_sid: str, options: dict[str, Any]):
scan_type=scan_type,
roms_ids=roms_ids,
launchbox_remote_enabled=launchbox_remote_enabled,
playmatch_enabled=playmatch_enabled,
)
return high_prio_queue.enqueue(
@@ -800,6 +852,7 @@ async def scan_handler(_sid: str, options: dict[str, Any]):
scan_type=scan_type,
roms_ids=roms_ids,
launchbox_remote_enabled=launchbox_remote_enabled,
playmatch_enabled=playmatch_enabled,
job_timeout=SCAN_TIMEOUT, # Timeout (default of 4 hours)
result_ttl=TASK_RESULT_TTL,
meta={
backend/endpoints/states.py
+23
View File
@@ -285,6 +285,29 @@ async def update_state(
return StateSchema.model_validate(db_state)
@protected_route(
router.put,
"/{id}/visibility",
[Scope.ASSETS_WRITE],
responses={status.HTTP_404_NOT_FOUND: {}},
)
def update_state_visibility(
request: Request,
id: int,
is_public: Annotated[bool, Body(embed=True)],
) -> StateSchema:
"""Toggle a state's public/private visibility (owner only)."""
state = db_state_handler.get_state_by_id(id)
if not state or state.user_id != request.user.id:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"State with ID {id} not found",
)
updated = db_state_handler.update_state(id, {"is_public": is_public})
return StateSchema.model_validate(updated)
@protected_route(
router.post,
"/delete",
backend/endpoints/sync.py
+26 -12
View File
@@ -50,7 +50,7 @@ class ClientSaveState(BaseModel):
class SyncNegotiatePayload(BaseModel):
device_id: str
device_id: str | None = None
saves: list[ClientSaveState]
@@ -86,13 +86,23 @@ def negotiate_sync(
The client sends its current save state, and the server returns a list of
operations (upload, download, conflict, no_op) to bring both sides in sync.
"""
device = db_device_handler.get_device(
device_id=payload.device_id, user_id=request.user.id
device_id: str | None = payload.device_id or getattr(
request.state, "device_id", None
)
if not device_id:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=(
"device_id is required (either in the request payload or "
"implicit via a device-bound client token)"
),
)
device = db_device_handler.get_device(device_id=device_id, user_id=request.user.id)
if not device:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Device with ID {payload.device_id} not found",
detail=f"Device with ID {device_id} not found",
)
if not device.sync_enabled:
@@ -115,12 +125,16 @@ def negotiate_sync(
operations: list[SyncOperationSchema] = []
# Build a set of server saves for this user, keyed by (rom_id, file_name)
# We'll also track which server saves were mentioned by the client
server_saves = db_save_handler.get_saves(user_id=request.user.id)
server_save_map: dict[tuple[int, str], Save] = {}
# Pair on (rom_id, slot), keeping the newest row per slot: slot uploads are datetime-tagged (spec) so tagged filenames never equal the client's untagged name, and a slot accrues many rows over time. Null-slot rows stay archival-only.
server_saves = db_save_handler.get_saves(
user_id=request.user.id, slot_not_null=True
)
server_save_map: dict[tuple[int, str | None], Save] = {}
for save in server_saves:
server_save_map[(save.rom_id, save.file_name)] = save
key = (save.rom_id, save.slot)
current = server_save_map.get(key)
if current is None or to_utc(save.updated_at) > to_utc(current.updated_at):
server_save_map[key] = save
# Get all sync records for this device
all_save_ids = [s.id for s in server_saves]
@@ -134,7 +148,7 @@ def negotiate_sync(
# Process each client save
for client_save in payload.saves:
key = (client_save.rom_id, client_save.file_name)
key = (client_save.rom_id, client_save.slot)
server_save = server_save_map.get(key)
if server_save is None:
@@ -192,8 +206,8 @@ def negotiate_sync(
)
)
# Check for server saves the client didn't mention
for save in server_saves:
# Check for current saves the client didn't mention (superseded older rows per slot are history, not downloads)
for save in server_save_map.values():
if save.id in matched_server_save_ids:
continue
backend/endpoints/tasks.py
+12 -2
View File
@@ -36,6 +36,9 @@ from handler.redis_handler import (
)
from tasks.manual.cleanup_missing_roms import cleanup_missing_roms_task
from tasks.manual.cleanup_orphaned_resources import cleanup_orphaned_resources_task
from tasks.manual.recompute_save_content_hashes import (
recompute_save_content_hashes_task,
)
from tasks.manual.sync_folder_scan import sync_folder_scan_task
from tasks.scheduled.convert_images_to_webp import convert_images_to_webp_task
from tasks.scheduled.scan_library import scan_library_task
@@ -116,6 +119,13 @@ manual_tasks: list[ManualTask] = [
"task": sync_folder_scan_task,
}
),
ManualTask(
{
"name": "recompute_save_content_hashes",
"type": TaskType.CLEANUP,
"task": recompute_save_content_hashes_task,
}
),
]
@@ -147,7 +157,7 @@ def _build_task_status_response(
common_data = {
"task_name": task_name,
"task_id": job.get_id(),
"task_id": job.id,
"status": job.get_status(),
"created_at": created_at,
"enqueued_at": enqueued_at,
@@ -386,7 +396,7 @@ async def run_single_task(
return {
"task_name": task_instance.title,
"task_id": job.get_id(),
"task_id": job.id,
"status": job.get_status() or JobStatus.QUEUED,
"created_at": (
job.created_at.isoformat()
backend/endpoints/user.py
+4 -2
View File
@@ -12,6 +12,7 @@ from handler.auth import auth_handler
from handler.auth.constants import Scope
from handler.database import db_user_handler
from handler.filesystem import fs_asset_handler
from handler.filesystem.assets_handler import validate_image_upload
from handler.metadata import meta_ra_handler
from handler.metadata.ra_handler import RAUserProgression
from logger.logger import log
@@ -387,9 +388,10 @@ async def update_user(
) from exc
if form_data.avatar is not None and form_data.avatar.filename is not None:
safe_extension = validate_image_upload(form_data.avatar, label="Avatar")
user_avatar_path = fs_asset_handler.build_avatar_path(user=db_user)
file_extension = form_data.avatar.filename.split(".")[-1]
file_name = f"avatar.{file_extension}"
file_name = f"avatar.{safe_extension}"
await fs_asset_handler.write_file(
file=form_data.avatar.file, path=user_avatar_path, filename=file_name
backend/handler/activity_handler.py
+144
View File
@@ -0,0 +1,144 @@
"""Real-time user game activity tracking.
Stores ephemeral "currently playing" state in Redis. Each active session is a
Redis key with a short TTL, refreshed by periodic heartbeats from the client
(browser) or the device. When the TTL expires (no heartbeat received), the
session is considered ended automatically.
"""
from __future__ import annotations
import json
from typing import TypedDict
from handler.redis_handler import async_cache
from logger.logger import log
class ActivityEntry(TypedDict):
user_id: int
username: str
avatar_path: str
rom_id: int
rom_name: str
rom_cover_path: str # small cover path, may be empty
screenshot_path: str # "where they are" image (save/title/screenshot), may be empty
platform_slug: str
platform_name: str
device_id: str
device_type: str # "web", "grout", "argosy-launcher", etc.
started_at: str # ISO 8601 timestamp
class ActivityHandler:
"""Redis-backed store for currently active game play sessions."""
ACTIVITY_TTL = 90 # seconds; refreshed by heartbeats
ROM_INDEX_TTL = 120 # slightly longer than ACTIVITY_TTL
KEY_PREFIX = "activity:user:"
ROM_INDEX_PREFIX = "activity:rom:"
def _activity_key(self, user_id: int, device_id: str) -> str:
return f"{self.KEY_PREFIX}{user_id}:{device_id}"
def _rom_index_key(self, rom_id: int) -> str:
return f"{self.ROM_INDEX_PREFIX}{rom_id}"
def _member(self, user_id: int, device_id: str) -> str:
return f"{user_id}:{device_id}"
async def set_active(self, entry: ActivityEntry) -> None:
"""Store or refresh a user's active play session."""
key = self._activity_key(entry["user_id"], entry["device_id"])
rom_key = self._rom_index_key(entry["rom_id"])
member = self._member(entry["user_id"], entry["device_id"])
async with async_cache.pipeline() as pipe:
await pipe.set(key, json.dumps(entry), ex=self.ACTIVITY_TTL)
await pipe.sadd(rom_key, member)
await pipe.expire(rom_key, self.ROM_INDEX_TTL)
await pipe.execute()
async def clear_active(self, user_id: int, device_id: str) -> int | None:
"""Clear a user's active play session. Returns the rom_id that was cleared, or None."""
key = self._activity_key(user_id, device_id)
raw = await async_cache.get(key)
if not raw:
return None
try:
entry = json.loads(raw)
rom_id = int(entry["rom_id"])
except (ValueError, KeyError, TypeError) as e:
log.warning(f"Failed to parse activity entry for cleanup: {e}")
await async_cache.delete(key)
return None
member = self._member(user_id, device_id)
async with async_cache.pipeline() as pipe:
await pipe.delete(key)
await pipe.srem(self._rom_index_key(rom_id), member)
await pipe.execute()
return rom_id
async def get_active(self, user_id: int, device_id: str) -> ActivityEntry | None:
"""Get a single active session by user and device."""
key = self._activity_key(user_id, device_id)
raw = await async_cache.get(key)
if not raw:
return None
try:
return json.loads(raw)
except ValueError:
return None
async def get_all_active(self) -> list[ActivityEntry]:
"""Get all currently active play sessions across all users."""
pattern = f"{self.KEY_PREFIX}*"
keys = [key async for key in async_cache.scan_iter(match=pattern)]
if not keys:
return []
# Single round-trip for every value instead of a GET per key.
entries: list[ActivityEntry] = []
for raw in await async_cache.mget(keys):
if not raw:
continue
try:
entries.append(json.loads(raw))
except ValueError:
continue
return entries
async def get_active_for_rom(self, rom_id: int) -> list[ActivityEntry]:
"""Get all active play sessions for a specific ROM."""
rom_key = self._rom_index_key(rom_id)
members = await async_cache.smembers(rom_key)
entries: list[ActivityEntry] = []
stale_members: list[str] = []
for member in members:
try:
user_id_str, device_id = member.rsplit(":", 1)
user_id = int(user_id_str)
except (ValueError, AttributeError):
stale_members.append(member)
continue
raw = await async_cache.get(self._activity_key(user_id, device_id))
if not raw:
# Key expired; clean up the stale set member.
stale_members.append(member)
continue
try:
entries.append(json.loads(raw))
except ValueError:
stale_members.append(member)
if stale_members:
await async_cache.srem(rom_key, *stale_members)
return entries
activity_handler = ActivityHandler()
backend/handler/auth/base_handler.py
+10
View File
@@ -13,6 +13,7 @@ from starlette.requests import HTTPConnection
from config import (
INVITE_TOKEN_EXPIRY_SECONDS,
OIDC_ALLOW_REGISTRATION,
OIDC_CLAIM_ROLES,
OIDC_ENABLED,
OIDC_ROLE_ADMIN,
@@ -436,6 +437,15 @@ class OpenIDHandler:
user = db_user_handler.get_user_by_email(email)
if user is None:
if not OIDC_ALLOW_REGISTRATION:
log.error(
"User with email '%s' not found and OIDC registration is disabled",
hl(email, color=CYAN),
)
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="User registration is disabled. Please contact an administrator to create an account.",
)
log.info(
"User with email '%s' not found, creating new user",
hl(email, color=CYAN),
backend/handler/auth/constants.py
+5
View File
@@ -4,6 +4,9 @@ from typing import Final
ALGORITHM: Final = "HS256"
DEFAULT_OAUTH_TOKEN_EXPIRY: Final = timedelta(minutes=15)
# Name of the session cookie set by RedisSessionMiddleware (see main.py). Shared
# so the socket log-stream handshake reads the same cookie the middleware writes.
SESSION_COOKIE_NAME: Final = "romm_session"
class Scope(enum.StrEnum):
@@ -26,6 +29,7 @@ class Scope(enum.StrEnum):
USERS_READ = "users.read"
USERS_WRITE = "users.write"
TASKS_RUN = "tasks.run"
LOGS_READ = "logs.read"
READ_SCOPES_MAP: Final = {
@@ -57,6 +61,7 @@ FULL_SCOPES_MAP: Final = {
Scope.USERS_READ: "View users",
Scope.USERS_WRITE: "Modify users",
Scope.TASKS_RUN: "Run tasks",
Scope.LOGS_READ: "View backend logs",
}
READ_SCOPES: Final = list(READ_SCOPES_MAP.keys())
backend/handler/auth/hybrid_auth.py
+15 -2
View File
@@ -6,7 +6,11 @@ from starlette.requests import HTTPConnection
from config import KIOSK_MODE
from handler.auth import auth_handler, oauth_handler
from handler.database import db_client_token_handler, db_user_handler
from handler.database import (
db_client_token_handler,
db_device_handler,
db_user_handler,
)
from models.user import User
from utils.datetime import to_utc
@@ -25,7 +29,11 @@ class HybridAuthBackend(AuthenticationBackend):
# Check if Authorization header exists
if "Authorization" in conn.headers:
scheme, token = conn.headers["Authorization"].split()
auth_header_parts = conn.headers["Authorization"].split()
if len(auth_header_parts) != 2:
return None
scheme, token = auth_header_parts
# Check if basic auth header is valid
if scheme.lower() == "basic":
@@ -66,6 +74,11 @@ class HybridAuthBackend(AuthenticationBackend):
db_client_token_handler.update_last_used(client_token.id)
user.set_last_active()
conn.state.client_token_id = client_token.id
conn.state.device_id = client_token.device_id
if client_token.device_id:
db_device_handler.update_last_seen_debounced(
client_token.device_id
)
return (AuthCredentials(effective_scopes), user)
# OAuth JWT bearer tokens
backend/handler/auth/middleware/csrf_middleware.py
+20 -4
View File
@@ -90,18 +90,21 @@ class CSRFMiddleware:
await self.app(scope, receive, send)
async def send(self, message: Message, send: Send, scope: Scope) -> None:
if message["type"] != "http.response.start":
await send(message)
return
request = Request(scope)
csrf_cookie = request.cookies.get(self.cookie_name)
current_user_id = request.user.id if request.user.is_authenticated else None
if csrf_cookie is None:
if not self._csrf_cookie_has_user(csrf_cookie, current_user_id):
message.setdefault("headers", [])
headers = MutableHeaders(scope=message)
cookie: http.cookies.BaseCookie = http.cookies.SimpleCookie()
cookie_name = self.cookie_name
cookie[cookie_name] = self._generate_csrf_token(
request.user.id if request.user.is_authenticated else None
)
cookie[cookie_name] = self._generate_csrf_token(current_user_id)
cookie[cookie_name]["path"] = self.cookie_path
cookie[cookie_name]["secure"] = self.cookie_secure
cookie[cookie_name]["httponly"] = self.cookie_httponly
@@ -143,6 +146,19 @@ class CSRFMiddleware:
obj = {"token": secrets.token_urlsafe(128), "user_id": user_id}
return cast(str, self.serializer.dumps(obj))
def _csrf_cookie_has_user(
self, csrf_cookie: str | None, user_id: int | None
) -> bool:
if csrf_cookie is None:
return False
try:
decoded_csrf_cookie = self.serializer.loads(csrf_cookie)
except (TypeError, BadSignature):
return False
return decoded_csrf_cookie.get("user_id") == user_id
def _csrf_tokens_match(
self, document_cookie: str, header_cookie: str, user_id: int | None
) -> bool:
backend/handler/database/collections_handler.py
+412 -342
View File
@@ -1,342 +1,412 @@
import functools
from collections.abc import Sequence
from datetime import datetime
from typing import Any
from sqlalchemy import delete, insert, literal, or_, select, update
from sqlalchemy.orm import (
Query,
QueryableAttribute,
Session,
load_only,
noload,
selectinload,
)
from decorators.database import begin_session
from models.collection import (
Collection,
CollectionRom,
SmartCollection,
VirtualCollection,
)
from models.rom import Rom
from .base_handler import DBBaseHandler
def with_roms(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
kwargs["query"] = select(Collection).options(
selectinload(Collection.roms)
.load_only(
Rom.id,
Rom.path_cover_s,
Rom.path_cover_l,
)
.options(noload(Rom.platform), noload(Rom.metadatum))
)
return func(*args, **kwargs)
return wrapper
class DBCollectionsHandler(DBBaseHandler):
@begin_session
@with_roms
def add_collection(
self,
collection: Collection,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection:
collection = session.merge(collection)
session.flush()
return session.scalar(query.filter_by(id=collection.id).limit(1))
@begin_session
@with_roms
def get_collection(
self,
id: int,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection | None:
return session.scalar(query.filter_by(id=id).limit(1))
@begin_session
@with_roms
def get_collection_by_name(
self,
name: str,
user_id: int,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection | None:
return session.scalar(query.filter_by(name=name, user_id=user_id).limit(1))
@begin_session
@with_roms
def get_favorite_collection(
self,
user_id: int,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection | None:
return session.scalar(
query.filter_by(is_favorite=True, user_id=user_id).limit(1)
)
@begin_session
@with_roms
def get_collections(
self,
updated_after: datetime | None = None,
only_fields: Sequence[QueryableAttribute] | None = None,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Sequence[Collection]:
if updated_after:
query = query.filter(Collection.updated_at > updated_after)
if only_fields:
query = query.options(load_only(*only_fields))
return session.scalars(query.order_by(Collection.name.asc())).unique().all()
@begin_session
@with_roms
def update_collection(
self,
id: int,
data: dict,
rom_ids: list[int] | None = None,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection:
session.execute(
update(Collection)
.where(Collection.id == id)
.values(**data)
.execution_options(synchronize_session="evaluate")
)
if rom_ids is not None:
# Delete all existing CollectionRom entries for this collection
session.execute(
delete(CollectionRom).where(CollectionRom.collection_id == id)
)
# Insert new CollectionRom entries for this collection
if rom_ids:
# Filter out rom_ids that no longer exist in the roms table to
# avoid foreign key constraint violations (e.g. after a rescan)
valid_rom_ids = set(
session.scalars(select(Rom.id).where(Rom.id.in_(rom_ids))).all()
)
if valid_rom_ids:
session.execute(
insert(CollectionRom),
[
{"collection_id": id, "rom_id": rom_id}
for rom_id in valid_rom_ids
],
)
return session.scalar(query.filter_by(id=id).limit(1))
@begin_session
def delete_collection(
self,
id: int,
session: Session = None, # type: ignore
) -> None:
session.execute(
delete(Collection)
.where(Collection.id == id)
.execution_options(synchronize_session="evaluate")
)
# Virtual collections
@begin_session
def get_virtual_collection(
self,
id: str,
session: Session = None, # type: ignore
) -> VirtualCollection | None:
name, type = VirtualCollection.from_id(id)
return session.scalar(
select(VirtualCollection).filter_by(name=name, type=type).limit(1)
)
@begin_session
def get_virtual_collections(
self,
type: str,
limit: int | None = None,
only_fields: Sequence[QueryableAttribute] | None = None,
session: Session = None, # type: ignore
) -> Sequence[VirtualCollection]:
query = (
select(VirtualCollection)
.filter(or_(VirtualCollection.type == type, literal(type == "all")))
.limit(limit)
.order_by(VirtualCollection.name.asc())
)
if only_fields:
query = query.options(load_only(*only_fields))
return session.scalars(query).unique().all()
# Smart collections
@begin_session
def add_smart_collection(
self,
smart_collection: SmartCollection,
session: Session = None, # type: ignore
) -> SmartCollection:
smart_collection = session.merge(smart_collection)
session.flush()
return session.query(SmartCollection).filter_by(id=smart_collection.id).one()
@begin_session
def get_smart_collection(
self,
id: int,
session: Session = None, # type: ignore
) -> SmartCollection | None:
return session.scalar(select(SmartCollection).filter_by(id=id).limit(1))
@begin_session
def get_smart_collection_by_name(
self,
name: str,
user_id: int,
session: Session = None, # type: ignore
) -> SmartCollection | None:
return session.scalar(
select(SmartCollection).filter_by(name=name, user_id=user_id).limit(1)
)
@begin_session
def get_smart_collections(
self,
user_id: int | None = None,
updated_after: datetime | None = None,
only_fields: Sequence[QueryableAttribute] | None = None,
session: Session = None, # type: ignore
) -> Sequence[SmartCollection]:
query = select(SmartCollection).order_by(SmartCollection.name.asc())
if user_id is not None:
# Get user's smart collections and public ones
query = query.filter(
(SmartCollection.user_id == user_id) | SmartCollection.is_public
)
if updated_after:
query = query.filter(SmartCollection.updated_at > updated_after)
if only_fields:
query = query.options(load_only(*only_fields))
return session.scalars(query).unique().all()
@begin_session
def update_smart_collection(
self,
id: int,
data: dict[str, Any],
session: Session = None, # type: ignore
) -> SmartCollection:
session.execute(
update(SmartCollection)
.where(SmartCollection.id == id)
.values(**data)
.execution_options(synchronize_session="evaluate")
)
return session.query(SmartCollection).filter_by(id=id).one()
@begin_session
def delete_smart_collection(
self,
id: int,
session: Session = None, # type: ignore
) -> None:
session.execute(
delete(SmartCollection)
.where(SmartCollection.id == id)
.execution_options(synchronize_session="evaluate")
)
def get_smart_collection_roms(
self, smart_collection: SmartCollection, user_id: int | None = None
) -> Sequence["Rom"]:
"""Get ROMs that match the smart collection's filter criteria."""
from handler.database import db_rom_handler
# Extract filter criteria
criteria = smart_collection.filter_criteria
# Convert legacy single-value criteria to arrays for backward compatibility
def convert_legacy_filter(new_key: str, old_key: str) -> list[str] | None:
"""Convert legacy single-value filter to array format."""
if new_value := criteria.get(new_key):
return new_value if isinstance(new_value, list) else [new_value]
if old_value := criteria.get(old_key):
return [old_value]
return None
# Apply conversions
genres = convert_legacy_filter("genres", "selected_genre")
franchises = convert_legacy_filter("franchises", "selected_franchise")
collections = convert_legacy_filter("collections", "selected_collection")
companies = convert_legacy_filter("companies", "selected_company")
age_ratings = convert_legacy_filter("age_ratings", "selected_age_rating")
regions = convert_legacy_filter("regions", "selected_region")
languages = convert_legacy_filter("languages", "selected_language")
statuses = convert_legacy_filter("statuses", "selected_status")
# Use the existing filter_roms method with the stored criteria
platform_ids = criteria.get("platform_ids")
if platform_ids is None:
if platform_id := criteria.get("platform_id"):
platform_ids = [platform_id]
return db_rom_handler.get_roms_scalar(
platform_ids=platform_ids,
collection_id=criteria.get("collection_id"),
virtual_collection_id=criteria.get("virtual_collection_id"),
search_term=criteria.get("search_term"),
matched=criteria.get("matched"),
favorite=criteria.get("favorite"),
duplicate=criteria.get("duplicate"),
playable=criteria.get("playable"),
has_ra=criteria.get("has_ra"),
missing=criteria.get("missing"),
verified=criteria.get("verified"),
genres=genres,
franchises=franchises,
collections=collections,
companies=companies,
age_ratings=age_ratings,
statuses=statuses,
regions=regions,
languages=languages,
# Logic operators for multi-value filters
genres_logic=criteria.get("genres_logic", "any"),
franchises_logic=criteria.get("franchises_logic", "any"),
collections_logic=criteria.get("collections_logic", "any"),
companies_logic=criteria.get("companies_logic", "any"),
age_ratings_logic=criteria.get("age_ratings_logic", "any"),
regions_logic=criteria.get("regions_logic", "any"),
languages_logic=criteria.get("languages_logic", "any"),
statuses_logic=criteria.get("statuses_logic", "any"),
user_id=user_id,
order_by=criteria.get("order_by", "name"),
order_dir=criteria.get("order_dir", "asc"),
)
import functools
from collections.abc import Sequence
from datetime import datetime, timezone
from typing import Any
from sqlalchemy import delete, insert, literal, or_, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import (
Query,
QueryableAttribute,
Session,
load_only,
noload,
selectinload,
)
from decorators.database import begin_session
from models.collection import (
Collection,
CollectionRom,
SmartCollection,
VirtualCollection,
)
from models.rom import Rom
from .base_handler import DBBaseHandler
def with_roms(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
kwargs["query"] = select(Collection).options(
selectinload(Collection.roms)
.load_only(
Rom.id,
Rom.path_cover_s,
Rom.path_cover_l,
)
.options(noload(Rom.platform), noload(Rom.metadatum))
)
return func(*args, **kwargs)
return wrapper
class DBCollectionsHandler(DBBaseHandler):
@begin_session
@with_roms
def add_collection(
self,
collection: Collection,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection:
collection = session.merge(collection)
session.flush()
return session.scalar(query.filter_by(id=collection.id).limit(1))
@begin_session
@with_roms
def get_collection(
self,
id: int,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection | None:
return session.scalar(query.filter_by(id=id).limit(1))
@begin_session
@with_roms
def get_collection_by_name(
self,
name: str,
user_id: int,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection | None:
return session.scalar(query.filter_by(name=name, user_id=user_id).limit(1))
@begin_session
@with_roms
def get_favorite_collection(
self,
user_id: int,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection | None:
return session.scalar(
query.filter_by(is_favorite=True, user_id=user_id).limit(1)
)
@begin_session
@with_roms
def get_collections(
self,
updated_after: datetime | None = None,
only_fields: Sequence[QueryableAttribute] | None = None,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Sequence[Collection]:
if updated_after:
query = query.filter(Collection.updated_at > updated_after)
if only_fields:
query = query.options(load_only(*only_fields))
return session.scalars(query.order_by(Collection.name.asc())).unique().all()
@begin_session
@with_roms
def update_collection(
self,
id: int,
data: dict,
rom_ids: list[int] | None = None,
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection:
session.execute(
update(Collection)
.where(Collection.id == id)
.values(**data)
.execution_options(synchronize_session="evaluate")
)
if rom_ids is not None:
# Delete all existing CollectionRom entries for this collection
session.execute(
delete(CollectionRom).where(CollectionRom.collection_id == id)
)
# Insert new CollectionRom entries for this collection
if rom_ids:
# Filter out rom_ids that no longer exist in the roms table to
# avoid foreign key constraint violations (e.g. after a rescan)
valid_rom_ids = set(
session.scalars(select(Rom.id).where(Rom.id.in_(rom_ids))).all()
)
if valid_rom_ids:
session.execute(
insert(CollectionRom),
[
{"collection_id": id, "rom_id": rom_id}
for rom_id in valid_rom_ids
],
)
return session.scalar(query.filter_by(id=id).limit(1))
@begin_session
@with_roms
def add_roms_to_collection(
self,
id: int,
rom_ids: list[int],
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection:
if rom_ids:
valid_rom_ids = set(
session.scalars(select(Rom.id).where(Rom.id.in_(rom_ids))).all()
)
existing_ids = set(
session.scalars(
select(CollectionRom.rom_id).where(
CollectionRom.collection_id == id
)
).all()
)
new_ids = valid_rom_ids - existing_ids
if new_ids:
try:
with session.begin_nested():
session.execute(
insert(CollectionRom),
[
{"collection_id": id, "rom_id": rom_id}
for rom_id in new_ids
],
)
except IntegrityError:
# Concurrent request inserted the same rows; data is consistent
pass
session.execute(
update(Collection)
.where(Collection.id == id)
.values(updated_at=datetime.now(timezone.utc))
.execution_options(synchronize_session="evaluate")
)
return session.scalar(query.filter_by(id=id).limit(1))
@begin_session
@with_roms
def remove_roms_from_collection(
self,
id: int,
rom_ids: list[int],
query: Query = None, # type: ignore
session: Session = None, # type: ignore
) -> Collection:
if rom_ids:
result = session.execute(
delete(CollectionRom).where(
CollectionRom.collection_id == id,
CollectionRom.rom_id.in_(rom_ids),
)
)
if result.rowcount > 0:
session.execute(
update(Collection)
.where(Collection.id == id)
.values(updated_at=datetime.now(timezone.utc))
.execution_options(synchronize_session="evaluate")
)
return session.scalar(query.filter_by(id=id).limit(1))
@begin_session
def delete_collection(
self,
id: int,
session: Session = None, # type: ignore
) -> None:
session.execute(
delete(Collection)
.where(Collection.id == id)
.execution_options(synchronize_session="evaluate")
)
# Virtual collections
@begin_session
def get_virtual_collection(
self,
id: str,
session: Session = None, # type: ignore
) -> VirtualCollection | None:
name, type = VirtualCollection.from_id(id)
return session.scalar(
select(VirtualCollection).filter_by(name=name, type=type).limit(1)
)
@begin_session
def get_virtual_collections(
self,
type: str,
limit: int | None = None,
only_fields: Sequence[QueryableAttribute] | None = None,
session: Session = None, # type: ignore
) -> Sequence[VirtualCollection]:
query = (
select(VirtualCollection)
.filter(or_(VirtualCollection.type == type, literal(type == "all")))
.limit(limit)
.order_by(VirtualCollection.name.asc())
)
if only_fields:
query = query.options(load_only(*only_fields))
return session.scalars(query).unique().all()
# Smart collections
@begin_session
def add_smart_collection(
self,
smart_collection: SmartCollection,
session: Session = None, # type: ignore
) -> SmartCollection:
smart_collection = session.merge(smart_collection)
session.flush()
return session.query(SmartCollection).filter_by(id=smart_collection.id).one()
@begin_session
def get_smart_collection(
self,
id: int,
session: Session = None, # type: ignore
) -> SmartCollection | None:
return session.scalar(select(SmartCollection).filter_by(id=id).limit(1))
@begin_session
def get_smart_collection_by_name(
self,
name: str,
user_id: int,
session: Session = None, # type: ignore
) -> SmartCollection | None:
return session.scalar(
select(SmartCollection).filter_by(name=name, user_id=user_id).limit(1)
)
@begin_session
def get_smart_collections(
self,
user_id: int | None = None,
updated_after: datetime | None = None,
only_fields: Sequence[QueryableAttribute] | None = None,
session: Session = None, # type: ignore
) -> Sequence[SmartCollection]:
query = select(SmartCollection).order_by(SmartCollection.name.asc())
if user_id is not None:
# Get user's smart collections and public ones
query = query.filter(
(SmartCollection.user_id == user_id) | SmartCollection.is_public
)
if updated_after:
query = query.filter(SmartCollection.updated_at > updated_after)
if only_fields:
query = query.options(load_only(*only_fields))
return session.scalars(query).unique().all()
@begin_session
def update_smart_collection(
self,
id: int,
data: dict[str, Any],
session: Session = None, # type: ignore
) -> SmartCollection:
session.execute(
update(SmartCollection)
.where(SmartCollection.id == id)
.values(**data)
.execution_options(synchronize_session="evaluate")
)
return session.query(SmartCollection).filter_by(id=id).one()
@begin_session
def delete_smart_collection(
self,
id: int,
session: Session = None, # type: ignore
) -> None:
session.execute(
delete(SmartCollection)
.where(SmartCollection.id == id)
.execution_options(synchronize_session="evaluate")
)
def get_smart_collection_roms(
self, smart_collection: SmartCollection, user_id: int | None = None
) -> Sequence["Rom"]:
"""Get ROMs that match the smart collection's filter criteria."""
from handler.database import db_rom_handler
# Extract filter criteria
criteria = smart_collection.filter_criteria
# Convert legacy single-value criteria to arrays for backward compatibility
def convert_legacy_filter(new_key: str, old_key: str) -> list[str] | None:
"""Convert legacy single-value filter to array format."""
if new_value := criteria.get(new_key):
return new_value if isinstance(new_value, list) else [new_value]
if old_value := criteria.get(old_key):
return old_value if isinstance(old_value, list) else [old_value]
return None
# Apply conversions
genres = convert_legacy_filter("genres", "selected_genre")
franchises = convert_legacy_filter("franchises", "selected_franchise")
collections = convert_legacy_filter("collections", "selected_collection")
companies = convert_legacy_filter("companies", "selected_company")
age_ratings = convert_legacy_filter("age_ratings", "selected_age_rating")
regions = convert_legacy_filter("regions", "selected_region")
languages = convert_legacy_filter("languages", "selected_language")
statuses = convert_legacy_filter("statuses", "selected_status")
# Use the existing filter_roms method with the stored criteria
platform_ids = criteria.get("platform_ids")
if platform_ids is None:
if platform_id := criteria.get("platform_id"):
platform_ids = [platform_id]
return db_rom_handler.get_roms_scalar(
platform_ids=platform_ids,
collection_id=criteria.get("collection_id"),
virtual_collection_id=criteria.get("virtual_collection_id"),
search_term=criteria.get("search_term"),
matched=criteria.get("matched"),
favorite=criteria.get("favorite"),
duplicate=criteria.get("duplicate"),
playable=criteria.get("playable"),
has_ra=criteria.get("has_ra"),
missing=criteria.get("missing"),
verified=criteria.get("verified"),
genres=genres,
franchises=franchises,
collections=collections,
companies=companies,
age_ratings=age_ratings,
statuses=statuses,
regions=regions,
languages=languages,
# Logic operators for multi-value filters
genres_logic=criteria.get("genres_logic", "any"),
franchises_logic=criteria.get("franchises_logic", "any"),
collections_logic=criteria.get("collections_logic", "any"),
companies_logic=criteria.get("companies_logic", "any"),
age_ratings_logic=criteria.get("age_ratings_logic", "any"),
regions_logic=criteria.get("regions_logic", "any"),
languages_logic=criteria.get("languages_logic", "any"),
statuses_logic=criteria.get("statuses_logic", "any"),
user_id=user_id,
order_by=criteria.get("order_by", "name"),
order_dir=criteria.get("order_dir", "asc"),
)
backend/handler/database/device_save_sync_handler.py
+26
View File
@@ -5,6 +5,7 @@ from sqlalchemy import delete, select, update
from sqlalchemy.orm import Session
from decorators.database import begin_session
from models.device import Device
from models.device_save_sync import DeviceSaveSync
from .base_handler import DBBaseHandler
@@ -40,6 +41,31 @@ class DBDeviceSaveSyncHandler(DBBaseHandler):
)
).all()
@begin_session
def get_syncs_for_saves(
self,
save_ids: list[int],
session: Session = None, # type: ignore
) -> dict[int, list[tuple[DeviceSaveSync, str | None]]]:
"""Fetch every device sync row for the given saves, grouped by save id.
Each row is paired with its device name so callers can attribute a save
to the device that created it without triggering the lazy-raise
``DeviceSaveSync.device`` relationship.
"""
if not save_ids:
return {}
rows = session.execute(
select(DeviceSaveSync, Device.name)
.join(Device, DeviceSaveSync.device_id == Device.id)
.filter(DeviceSaveSync.save_id.in_(save_ids))
.order_by(DeviceSaveSync.last_synced_at.desc(), DeviceSaveSync.device_id)
).all()
grouped: dict[int, list[tuple[DeviceSaveSync, str | None]]] = {}
for sync, device_name in rows:
grouped.setdefault(sync.save_id, []).append((sync, device_name))
return grouped
@begin_session
def upsert_sync(
self,
backend/handler/database/devices_handler.py
+49 -1
View File
@@ -1,14 +1,17 @@
from collections.abc import Sequence
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete, select, update
from sqlalchemy.orm import Session
from decorators.database import begin_session
from models.device import Device, SyncMode
from utils.datetime import to_utc
from .base_handler import DBBaseHandler
LAST_SEEN_DEBOUNCE = timedelta(minutes=5)
class DBDevicesHandler(DBBaseHandler):
@begin_session
@@ -74,6 +77,25 @@ class DBDevicesHandler(DBBaseHandler):
"""Get a device by ID without user filtering (for server-side operations)."""
return session.scalar(select(Device).filter_by(id=device_id).limit(1))
@begin_session
def get_device_by_client_identifier(
self,
user_id: int,
client_device_identifier: str,
session: Session = None, # type: ignore
) -> Device | None:
"""Find a device by its client-supplied stable identifier, scoped to a user."""
if not client_device_identifier:
return None
return session.scalar(
select(Device)
.filter_by(
user_id=user_id,
client_device_identifier=client_device_identifier,
)
.limit(1)
)
@begin_session
def get_devices(
self,
@@ -123,6 +145,32 @@ class DBDevicesHandler(DBBaseHandler):
.execution_options(synchronize_session="evaluate")
)
@begin_session
def update_last_seen_debounced(
self,
device_id: str,
session: Session = None, # type: ignore
) -> None:
"""Bump last_seen on the device, skipping if updated within the debounce window.
Intended for the auth hot path -- called on every authenticated client-token
request. Mirrors the debounce used by ClientToken.update_last_used.
"""
now = datetime.now(timezone.utc)
device = session.get(Device, device_id)
if device is None:
return
if device.last_seen and (now - to_utc(device.last_seen)) < LAST_SEEN_DEBOUNCE:
return
session.execute(
update(Device)
.where(Device.id == device_id)
.values(last_seen=now)
.execution_options(synchronize_session="evaluate")
)
@begin_session
def delete_device(
self,
backend/handler/database/roms_handler.py
+412 -61
View File
@@ -1,11 +1,13 @@
import functools
import json
import re
from collections.abc import Iterable, Sequence
from datetime import datetime
from typing import Any
from redis.exceptions import WatchError
from sqlalchemy import (
Integer,
Row,
String,
Text,
and_,
@@ -29,6 +31,7 @@ from sqlalchemy.orm import (
load_only,
noload,
selectinload,
undefer,
)
from sqlalchemy.sql.elements import ColumnElement
from sqlalchemy.sql.selectable import Select
@@ -36,9 +39,20 @@ from sqlalchemy.sql.selectable import Select
from config import ROMM_DB_DRIVER
from decorators.database import begin_session
from handler.metadata.base_handler import UniversalPlatformSlug as UPS
from handler.redis_handler import sync_cache
from models.assets import Save, Screenshot, State
from models.base import compute_file_name_parts
from models.platform import Platform
from models.rom import Rom, RomFile, RomMetadata, RomNote, RomUser
from models.rom import (
Rom,
RomFile,
RomFileCategory,
RomMetadata,
RomNote,
RomUser,
SiblingRom,
compute_name_sort_key,
)
from utils.database import (
json_array_contains_all,
json_array_contains_any,
@@ -101,7 +115,53 @@ RUFFLE_SUPPORTED_PLATFORMS = [
UPS.BROWSER,
]
STRIP_ARTICLES_REGEX = r"^(the|a|an)\s+"
# Used to remove native full-text SQL operators
FULLTEXT_BOOLEAN_OPERATORS_REGEX = re.compile(r'[+\-~<>()"@*]')
# 3 is the default minimum size in InnoDB
FULLTEXT_MIN_TOKEN_SIZE = 3
# Cached ROM filter values (genres/franchises/etc.) so it doesn't get
# recomputed on every call to /api/roms
ROM_FILTERS_CACHE_VERSION_KEY = "filter_values:ver"
ROM_FILTERS_CACHE_KEYS_PREFIX = "filter_values:keys"
ROM_FILTERS_CACHE_TTL = 60 * 60 * 24 * 7 # 7 days
def _cache_value_to_str(value: Any) -> str | None:
if value is None:
return None
if isinstance(value, bytes):
return value.decode()
return str(value)
def _filter_values_cache_version() -> str:
return _cache_value_to_str(sync_cache.get(ROM_FILTERS_CACHE_VERSION_KEY)) or "0"
def _filter_values_cache_keys_key(version: str) -> str:
return f"{ROM_FILTERS_CACHE_KEYS_PREFIX}:v{version}"
def _store_versioned_cache(redis_key: str, version: str, result: Any) -> None:
version_keys_set = _filter_values_cache_keys_key(version)
with sync_cache.pipeline() as pipe:
try:
pipe.watch(ROM_FILTERS_CACHE_VERSION_KEY)
current_version = (
_cache_value_to_str(pipe.get(ROM_FILTERS_CACHE_VERSION_KEY)) or "0"
)
if current_version != version:
pipe.unwatch()
else:
pipe.multi()
pipe.set(redis_key, json.dumps(result), ex=ROM_FILTERS_CACHE_TTL)
pipe.sadd(version_keys_set, redis_key)
pipe.expire(version_keys_set, ROM_FILTERS_CACHE_TTL)
pipe.execute()
except WatchError:
pass
def _create_metadata_id_case(
@@ -142,14 +202,31 @@ def with_details(func):
),
selectinload(Rom.rom_users).options(noload(RomUser.rom)),
selectinload(Rom.metadatum).options(noload(RomMetadata.rom)),
# Multi-file downloads, 3DS QR codes, and metadata matching
selectinload(Rom.files).options(
joinedload(RomFile.rom).load_only(Rom.fs_path, Rom.fs_name)
),
selectinload(Rom.sibling_roms).options(
noload(Rom.platform), noload(Rom.metadatum)
noload(Rom.platform),
noload(Rom.metadatum),
# Per-sibling is_main_sibling resolution for the
# SiblingRomSchema needs each sibling's RomUser for the
# request user — the relationship is `lazy="raise"`, so
# it has to be eager-loaded here.
selectinload(Rom.rom_users).options(noload(RomUser.rom)),
load_only(
Rom.id,
Rom.name,
Rom.fs_name_no_tags,
Rom.fs_name_no_ext,
),
),
selectinload(Rom.collections),
selectinload(Rom.notes),
undefer(Rom.multi_file),
undefer(Rom.top_level_file_count),
undefer(Rom.has_manual_files),
undefer(Rom.has_soundtrack),
)
return func(*args, **kwargs)
@@ -195,6 +272,85 @@ class DBRomsHandler(DBBaseHandler):
return []
return session.scalars(query.filter(Rom.id.in_(ids))).all()
def get_files_for_roms(
self,
rom_ids: list[int],
*,
session: Session,
) -> dict[int, list[RomFile]]:
"""Return {rom_id: [RomFile, ...]} for the given rom IDs in a single query.
Used by the list endpoint to serialize files without relying on the
query's relationship eager-load surviving pagination.
"""
if not rom_ids:
return {}
files = session.scalars(
select(RomFile).where(RomFile.rom_id.in_(rom_ids))
).all()
buckets: dict[int, list[RomFile]] = {rom_id: [] for rom_id in rom_ids}
for file in files:
buckets[file.rom_id].append(file)
return buckets
def get_siblings_for_roms(
self,
rom_ids: list[int],
user_id: int,
*,
session: Session,
) -> dict[int, list[tuple[Rom, bool]]]:
"""Return {rom_id: [(sibling Rom, is_main_sibling), ...]} in a single query.
Joins sibling_roms → roms (only the columns SiblingRomSchema needs) and
left-joins rom_user for the requesting user, so the per-user
`is_main_sibling` flag is resolved without hydrating the wide roms table
or its JSON metadata on every page.
"""
if not rom_ids:
return {}
rows = session.execute(
select(
SiblingRom.rom_id,
Rom,
func.coalesce(RomUser.is_main_sibling, false()).label(
"is_main_sibling"
),
)
.join(Rom, Rom.id == SiblingRom.sibling_rom_id)
.outerjoin(
RomUser,
and_(
RomUser.rom_id == SiblingRom.sibling_rom_id,
RomUser.user_id == user_id,
),
)
.where(SiblingRom.rom_id.in_(rom_ids))
.options(
load_only(
Rom.name,
Rom.fs_name_no_tags,
Rom.fs_name_no_ext,
)
)
).all()
# Dedupe by (parent rom, sibling id) so a duplicate join row doesn't
# surface the same sibling twice on the wire.
seen: dict[int, set[int]] = {rom_id: set() for rom_id in rom_ids}
buckets: dict[int, list[tuple[Rom, bool]]] = {rom_id: [] for rom_id in rom_ids}
for rom_id, sibling, is_main in rows:
if sibling.id in seen[rom_id]:
continue
seen[rom_id].add(sibling.id)
buckets[rom_id].append((sibling, bool(is_main)))
return buckets
def filter_by_platform_id(self, query: Query, platform_id: int):
return query.filter(Rom.platform_id == platform_id)
@@ -242,19 +398,53 @@ class DBRomsHandler(DBBaseHandler):
return query.filter(Rom.id.in_(smart_collection.rom_ids))
return query
def _build_fulltext_boolean_query(self, term: str) -> str | None:
words = FULLTEXT_BOOLEAN_OPERATORS_REGEX.sub(" ", term).split()
if not words or any(len(word) < FULLTEXT_MIN_TOKEN_SIZE for word in words):
return None
return " ".join(f"+{word}*" for word in words)
def _build_fulltext_relevance(self, search_term: str) -> str | None:
parts: list[str] = []
for term in search_term.split("|"):
words = FULLTEXT_BOOLEAN_OPERATORS_REGEX.sub(" ", term).split()
if len(words) > 1:
parts.append('"' + " ".join(words) + '"')
return " ".join(parts) if parts else None
def _filter_by_search_term(self, query: Query, search_term: str):
terms = [term.strip() for term in search_term.split("|")]
conditions = [
condition
for term in terms
for condition in (
Rom.fs_name.ilike(f"%{term}%"),
Rom.name.ilike(f"%{term}%"),
)
if term
]
terms = [term for term in terms if term]
if not terms:
return query
return query.filter(or_(*conditions))
if ROMM_DB_DRIVER in ("mariadb", "mysql"):
match_clauses: list[Any] = []
for idx, term in enumerate(terms):
boolean_query = self._build_fulltext_boolean_query(term)
if boolean_query is None:
match_clauses = []
break
param = f"fulltext_search_{idx}"
match_clauses.append(
text(
f"MATCH(roms.name, roms.fs_name) "
f"AGAINST(:{param} IN BOOLEAN MODE)"
).bindparams(**{param: boolean_query})
)
if match_clauses:
return query.filter(or_(*match_clauses))
# psql and full-text fallback
term_conditions = []
for term in terms:
word_conditions = [
or_(Rom.fs_name.ilike(f"%{word}%"), Rom.name.ilike(f"%{word}%"))
for word in term.split()
]
if word_conditions:
term_conditions.append(and_(*word_conditions))
return query.filter(or_(*term_conditions))
def _filter_by_matched(self, query: Query, value: bool) -> Query:
"""Filter based on whether the rom is matched to a metadata provider.
@@ -549,6 +739,8 @@ class DBRomsHandler(DBBaseHandler):
player_counts_logic: str = "any",
user_id: int | None = None,
updated_after: datetime | None = None,
include_file_stats: bool = False,
include_files: bool = False,
session: Session = None, # type: ignore
) -> Query[Rom]:
from handler.scan_handler import MetadataSource
@@ -560,18 +752,35 @@ class DBRomsHandler(DBBaseHandler):
selectinload(Rom.rom_users).options(noload(RomUser.rom)),
# Sort table by metadata (first_release_date)
selectinload(Rom.metadatum).options(noload(RomMetadata.rom)),
# Required for multi-file ROM actions and 3DS QR code
selectinload(Rom.files).options(
joinedload(RomFile.rom).load_only(Rom.fs_path, Rom.fs_name)
),
# Show sibling rom badges on cards
selectinload(Rom.sibling_roms).options(
noload(Rom.platform), noload(Rom.metadatum)
),
# Show notes indicator on cards
# Notes indicator on cards
selectinload(Rom.notes),
)
# Only load files (and the RomFile.rom backref needed by `is_top_level` /
# `file_name_for_download`) when the caller iterates them — e.g. the
# feed endpoints. The gallery/list and filter-value paths serialize
# SimpleRomSchema without files, so they skip this entirely.
if include_files:
query = query.options(
selectinload(Rom.files).options(
joinedload(RomFile.rom).load_only(Rom.fs_path, Rom.fs_name)
)
)
# Correlated subqueries and only undefer when the caller serializes the
# gallery-card flags. Feeds and filter-value lookups don't need them.
if include_file_stats:
query = query.options(
undefer(Rom.multi_file),
undefer(Rom.top_level_file_count),
undefer(Rom.has_manual_files),
undefer(Rom.has_soundtrack),
)
# Handle platform filtering - platform filtering always uses OR logic since ROMs belong to only one platform
if platform_ids:
query = self._filter_by_platform_ids(query, platform_ids)
@@ -776,8 +985,9 @@ class DBRomsHandler(DBBaseHandler):
def get_roms_query(
self,
*,
order_by: str = "name",
order_by: str = "",
order_dir: str = "asc",
search_term: str | None = None,
user_id: int | None = None,
session: Session = None, # type: ignore
) -> tuple[Query[Rom], Any]:
@@ -799,26 +1009,38 @@ class DBRomsHandler(DBBaseHandler):
else:
order_attr = Rom.name
# Use indexed `name_sort_key` to have fast access to names without
# articles (the, a, an) and leading digits. The key is derived from
# `name` at write time, or holds a custom override when one is set.
if order_attr is Rom.name:
order_attr = Rom.name_sort_key
order_attr_column = order_attr
# Ignore case when the order attribute is a number
if isinstance(order_attr.type, (String, Text)):
# Remove any leading articles
order_attr = func.trim(
func.lower(order_attr).regexp_replace(STRIP_ARTICLES_REGEX, "")
)
# Pad numbers with leading zeros to ensure natural sorting
order_attr = order_attr.regexp_replace(
r"(\d+)", r"00000000000\1"
).regexp_replace(r"0*(\d{12})", r"\1")
if order_dir.lower() == "desc":
order_attr = order_attr.desc()
else:
order_attr = order_attr.asc()
return query.order_by(order_attr), order_attr_column # type: ignore
relevance_clause = None
if search_term and ROMM_DB_DRIVER in ("mariadb", "mysql"):
relevance = self._build_fulltext_relevance(search_term)
if relevance:
relevance_clause = text(
"MATCH(roms.name, roms.fs_name) "
"AGAINST(:relevance IN BOOLEAN MODE) DESC"
).bindparams(relevance=relevance)
if order_by: # explicit sort wins, relevance breaks ties
order_clauses = [order_attr]
if relevance_clause is not None:
order_clauses.append(relevance_clause)
else: # no sort selected: relevance leads, name is the tiebreaker
order_clauses = [order_attr]
if relevance_clause is not None:
order_clauses.insert(0, relevance_clause)
return query.order_by(*order_clauses), order_attr_column # type: ignore
@begin_session
def get_roms_scalar(
@@ -829,8 +1051,9 @@ class DBRomsHandler(DBBaseHandler):
**kwargs,
) -> Sequence[Rom]:
query, _ = self.get_roms_query(
order_by=kwargs.get("order_by", "name"),
order_by=kwargs.get("order_by", ""),
order_dir=kwargs.get("order_dir", "asc"),
search_term=kwargs.get("search_term", None),
user_id=kwargs.get("user_id", None),
)
@@ -872,6 +1095,7 @@ class DBRomsHandler(DBBaseHandler):
player_counts_logic=kwargs.get("player_counts_logic", "any"),
user_id=kwargs.get("user_id", None),
group_by_meta_id=kwargs.get("group_by_meta_id", False),
include_files=kwargs.get("include_files", False),
)
return session.scalars(roms).all()
@@ -880,22 +1104,35 @@ class DBRomsHandler(DBBaseHandler):
self,
query: Query,
order_by_attr: Any,
*,
cache_key: str | None = None,
order_dir: str = "asc",
session: Session = None, # type: ignore
) -> list[Row[tuple[str, int]]]:
if isinstance(order_by_attr.type, (String, Text)):
# Remove any leading articles
order_by_attr = func.trim(
func.lower(order_by_attr).regexp_replace(STRIP_ARTICLES_REGEX, "")
)
else:
order_by_attr = func.trim(
func.lower(Rom.name).regexp_replace(STRIP_ARTICLES_REGEX, "")
)
) -> list[tuple[str, int]]:
redis_key: str | None = None
version: str | None = None
if cache_key:
version = _filter_values_cache_version()
redis_key = f"char_index:{cache_key}:v{version}"
cached = sync_cache.get(redis_key)
if cached is not None:
return json.loads(cached)
# Pad numbers with leading zeros to ensure natural sorting
order_by_attr = order_by_attr.regexp_replace(
r"(\d+)", r"00000000000\1"
).regexp_replace(r"0*(\d{12})", r"\1")
# Drop any ordering carried over from the main query (e.g. search relevance).
# This builds its own positional ordering below.
query = query.order_by(None)
if not isinstance(order_by_attr.type, (String, Text)):
order_by_attr = Rom.name_sort_key
# Apply the same direction the main query uses so the position
# numbers we emit (and the per-letter min position downstream)
# match the actual order the client paginates over. Without this
# the frontend AlphaStrip would highlight the wrong letter when
# order_dir=desc.
order_window = (
order_by_attr.desc() if order_dir.lower() == "desc" else order_by_attr.asc()
)
# Get the row number and first letter for each item
subquery = (
@@ -906,21 +1143,27 @@ class DBRomsHandler(DBBaseHandler):
1,
1,
).label("letter"),
func.row_number().over(order_by=order_by_attr).label("position"),
func.row_number().over(order_by=order_window).label("position"),
)
.subquery()
)
# Get the minimum position for each letter
return (
rows = (
session.query(
subquery.c.letter, func.min(subquery.c.position - 1).label("position")
)
.filter(subquery.c.letter.isnot(None))
.group_by(subquery.c.letter)
.order_by(subquery.c.letter)
.all()
)
result = [(letter, int(position)) for letter, position in rows]
if redis_key is not None and version is not None:
_store_versioned_cache(redis_key, version, result)
return result
@begin_session
def get_roms_by_fs_name(
self,
@@ -931,8 +1174,8 @@ class DBRomsHandler(DBBaseHandler):
"""Retrieve a dictionary of roms by their filesystem names.
Eager-loads only `platform` (used downstream by the scan loop via
`rom.platform_slug` / `rom.platform.fs_slug`) — deliberately
avoiding `with_details`, whose full relationship eager-load is
`rom.platform_slug` / `rom.platform.fs_slug`). This deliberately
avoids `with_details`, whose full relationship eager-load is
wasted work for the scan-skip decision on large platforms.
"""
roms = (
@@ -940,9 +1183,6 @@ class DBRomsHandler(DBBaseHandler):
select(Rom)
.options(
selectinload(Rom.platform),
selectinload(Rom.files).options(
joinedload(RomFile.rom).load_only(Rom.fs_path, Rom.fs_name)
),
)
.where(
and_(
@@ -964,6 +1204,26 @@ class DBRomsHandler(DBBaseHandler):
data: dict,
session: Session = None, # type: ignore
) -> Rom:
if "name" in data and "name_sort_key" not in data:
# Re-derive the key from the new name, but only when the stored key
# is still the derived value (i.e. not a manual override). Mirrors
# the `@validates` logic, which the bulk update() bypasses.
existing = session.query(Rom).filter_by(id=id).one()
if (
existing.name_sort_key is None
or existing.name_sort_key == compute_name_sort_key(existing.name)
):
data = {**data, "name_sort_key": compute_name_sort_key(data["name"])}
if "fs_name" in data:
parts = compute_file_name_parts(data["fs_name"])
data = {
**data,
"fs_name_no_tags": parts.no_tags,
"fs_name_no_ext": parts.no_ext,
"fs_extension": parts.extension,
}
session.execute(
update(Rom)
.where(Rom.id == id)
@@ -1066,7 +1326,9 @@ class DBRomsHandler(DBBaseHandler):
user_id: int,
session: Session = None, # type: ignore
) -> RomUser:
return session.merge(RomUser(rom_id=rom_id, user_id=user_id))
rom_user = session.merge(RomUser(rom_id=rom_id, user_id=user_id))
session.flush()
return rom_user
@begin_session
def get_rom_user(
@@ -1131,7 +1393,9 @@ class DBRomsHandler(DBBaseHandler):
rom_file: RomFile,
session: Session = None, # type: ignore
) -> RomFile:
return session.merge(rom_file)
merged = session.merge(rom_file)
session.flush()
return merged
@begin_session
def get_rom_file_by_id(
@@ -1141,13 +1405,62 @@ class DBRomsHandler(DBBaseHandler):
) -> RomFile | None:
return session.scalar(select(RomFile).filter_by(id=id).limit(1))
@begin_session
def get_rom_file_by_path(
self,
rom_id: int,
file_path: str,
file_name: str,
session: Session = None, # type: ignore
) -> RomFile | None:
return session.scalar(
select(RomFile)
.filter_by(rom_id=rom_id, file_path=file_path, file_name=file_name)
.limit(1)
)
@begin_session
def get_rom_files_by_category(
self,
rom_id: int,
category: RomFileCategory,
session: Session = None, # type: ignore
) -> Sequence[RomFile]:
"""Return the ROM's files for a single category, ordered by file_name."""
return (
session.scalars(
select(RomFile)
.filter_by(rom_id=rom_id, category=category)
.order_by(RomFile.file_name.asc())
)
.unique()
.all()
)
@begin_session
def rom_files_for_rom_id(
self,
rom_id: int,
session: Session = None, # type: ignore
) -> list[RomFile]:
"""Fetch a ROM's files on demand, with the `RomFile.rom` backref loaded."""
return list(
session.scalars(
select(RomFile)
.filter_by(rom_id=rom_id)
.options(joinedload(RomFile.rom).load_only(Rom.fs_path, Rom.fs_name))
)
.unique()
.all()
)
@begin_session
def update_rom_file(
self,
id: int,
data: dict,
session: Session = None, # type: ignore
) -> RomFile:
) -> RomFile | None:
session.execute(
update(RomFile)
.where(RomFile.id == id)
@@ -1155,7 +1468,7 @@ class DBRomsHandler(DBBaseHandler):
.execution_options(synchronize_session="evaluate")
)
return session.query(RomFile).filter_by(id=id).one()
return session.query(RomFile).filter_by(id=id).one_or_none()
@begin_session
def purge_rom_files(
@@ -1173,6 +1486,18 @@ class DBRomsHandler(DBBaseHandler):
)
return purged_rom_files
@begin_session
def delete_rom_file(
self,
id: int,
session: Session = None, # type: ignore
) -> None:
session.execute(
delete(RomFile)
.where(RomFile.id == id)
.execution_options(synchronize_session="evaluate")
)
# Note management methods
@begin_session
def get_rom_notes(
@@ -1429,16 +1754,39 @@ class DBRomsHandler(DBBaseHandler):
"platforms": sorted(platforms),
}
def invalidate_filter_values_cache(self) -> None:
old_version = str(int(sync_cache.incr(ROM_FILTERS_CACHE_VERSION_KEY)) - 1)
old_keys_set = _filter_values_cache_keys_key(old_version)
old_cache_keys = [
key
for raw_key in sync_cache.smembers(old_keys_set)
if (key := _cache_value_to_str(raw_key)) is not None
]
if old_cache_keys:
sync_cache.delete(*old_cache_keys)
sync_cache.delete(old_keys_set)
@begin_session
def with_filter_values(
self,
query: Query,
*,
cache_key: str | None = None,
session: Session = None, # type: ignore
) -> dict:
"""
Returns the list of filters given the current subset of ROMs in the query
"""
ids_subq = query.with_only_columns(Rom.id).scalar_subquery() # type: ignore
redis_key: str | None = None
version: str | None = None
if cache_key:
version = _filter_values_cache_version()
redis_key = f"filter_values:{cache_key}:v{version}"
cached = sync_cache.get(redis_key)
if cached is not None:
return json.loads(cached)
ids_subq = query.order_by(None).with_only_columns(Rom.id).scalar_subquery() # type: ignore
statement = (
select(
@@ -1458,7 +1806,10 @@ class DBRomsHandler(DBBaseHandler):
.where(Rom.id.in_(ids_subq))
)
return self._collect_filter_values(session, statement)
result = self._collect_filter_values(session, statement)
if redis_key is not None and version is not None:
_store_versioned_cache(redis_key, version, result)
return result
@begin_session
def get_rom_filters(
backend/handler/database/saves_handler.py
+108 -10
View File
@@ -1,7 +1,7 @@
from collections.abc import Sequence
from typing import Literal
from sqlalchemy import and_, asc, delete, desc, select, update
from sqlalchemy import and_, asc, delete, desc, func, or_, select, update
from sqlalchemy.orm import QueryableAttribute, Session, load_only
from decorators.database import begin_session
@@ -35,13 +35,15 @@ class DBSavesHandler(DBBaseHandler):
user_id: int,
rom_id: int,
file_name: str,
slot: str | None = None,
session: Session = None, # type: ignore
) -> Save | None:
return session.scalars(
select(Save)
.filter_by(rom_id=rom_id, user_id=user_id, file_name=file_name)
.limit(1)
).first()
query = select(Save).filter_by(
rom_id=rom_id, user_id=user_id, file_name=file_name
)
if slot is not None:
query = query.filter(Save.slot == slot)
return session.scalars(query.limit(1)).first()
@begin_session
def get_save_by_content_hash(
@@ -49,13 +51,15 @@ class DBSavesHandler(DBBaseHandler):
user_id: int,
rom_id: int,
content_hash: str,
slot: str | None = None,
session: Session = None, # type: ignore
) -> Save | None:
return session.scalar(
select(Save)
.filter_by(rom_id=rom_id, user_id=user_id, content_hash=content_hash)
.limit(1)
query = select(Save).filter_by(
rom_id=rom_id, user_id=user_id, content_hash=content_hash
)
if slot is not None:
query = query.filter(Save.slot == slot)
return session.scalar(query.limit(1))
@begin_session
def get_saves(
@@ -64,6 +68,7 @@ class DBSavesHandler(DBBaseHandler):
rom_id: int | None = None,
platform_id: int | None = None,
slot: str | None = None,
slot_not_null: bool = False,
order_by: Literal["updated_at", "created_at"] | None = None,
order_dir: Literal["asc", "desc"] = "desc",
only_fields: Sequence[QueryableAttribute] | None = None,
@@ -82,6 +87,9 @@ class DBSavesHandler(DBBaseHandler):
if slot is not None:
query = query.filter(Save.slot == slot)
if slot_not_null:
query = query.filter(Save.slot.is_not(None))
if order_by:
order_col = getattr(Save, order_by)
order_fn = asc if order_dir == "asc" else desc
@@ -92,6 +100,65 @@ class DBSavesHandler(DBBaseHandler):
return session.scalars(query).all()
@begin_session
def get_save_by_id(
self,
id: int,
session: Session = None, # type: ignore
) -> Save | None:
"""Fetch a save by id without scoping to an owner. Used for the
visibility toggle and community downloads, where the caller may not own
the save. Mirrors db_screenshot_handler.get_screenshot_by_id."""
return session.get(Save, id)
@begin_session
def get_rom_shared_saves(
self,
rom_id: int,
user_id: int,
public_only: bool = False,
session: Session = None, # type: ignore
) -> Sequence[Save]:
"""Saves for a ROM visible to the requesting user: own (public +
private) plus other users' public ones. Mirrors
db_screenshot_handler.get_rom_gallery_screenshots."""
query = select(Save).filter(Save.rom_id == rom_id)
if public_only:
query = query.filter(Save.is_public)
else:
query = query.filter(or_(Save.user_id == user_id, Save.is_public))
query = query.order_by(desc(Save.updated_at))
return session.scalars(query).all()
@begin_session
def get_latest_saves_for_roms(
self,
user_id: int,
rom_ids: Sequence[int],
session: Session = None, # type: ignore
) -> dict[int, Save]:
"""The most recent save per ROM for a user, keyed by `rom_id`.
Batched for the continue-playing rail, which enriches each card with
the in-game screenshot captured alongside the user's latest save.
"""
if not rom_ids:
return {}
saves = session.scalars(
select(Save)
.filter(Save.user_id == user_id, Save.rom_id.in_(rom_ids))
.order_by(desc(Save.updated_at))
).all()
latest: dict[int, Save] = {}
for save in saves:
# Saves come newest-first, so the first one seen per ROM wins.
latest.setdefault(save.rom_id, save)
return latest
@begin_session
def update_save(
self,
@@ -176,3 +243,34 @@ class DBSavesHandler(DBBaseHandler):
"total_count": len(saves),
"slots": list(slots_data.values()),
}
@begin_session
def count_saves_missing_content_hash(
self,
session: Session = None, # type: ignore
) -> int:
"""Number of Save rows whose content_hash is NULL. Used at startup to
decide whether the one-shot recompute task needs to be enqueued."""
return (
session.scalar(
select(func.count(Save.id)).where(Save.content_hash.is_(None))
)
or 0
)
@begin_session
def get_saves_after_id(
self,
after_id: int,
limit: int,
session: Session = None, # type: ignore
) -> Sequence[Save]:
"""Page Save rows by primary key. Returns up to ``limit`` rows with
``id > after_id``, ordered by id. Used by the
recompute_save_content_hashes maintenance task to walk every row in
bounded-memory batches: streaming via ``yield_per`` is incompatible
with the per-call session lifetime that ``@begin_session`` enforces,
so the caller drives pagination with this method instead."""
return session.scalars(
select(Save).where(Save.id > after_id).order_by(asc(Save.id)).limit(limit)
).all()
backend/handler/database/screenshots_handler.py
+27
View File
@@ -82,6 +82,33 @@ class DBScreenshotsHandler(DBBaseHandler):
) -> Screenshot | None:
return session.get(Screenshot, id)
@begin_session
def get_rom_gallery_screenshots(
self,
rom_id: int,
user_id: int,
public_only: bool = False,
session: Session = None, # type: ignore
) -> Sequence[Screenshot]:
"""Gallery (intentionally-uploaded) screenshots for a ROM, visible to
the requesting user. Mirrors `db_rom_handler.get_rom_notes`: own
screenshots (public + private) plus other users' public ones. Excludes
the auto-captured save/state thumbnails (`is_gallery == False`)."""
query = select(Screenshot).filter(
Screenshot.rom_id == rom_id,
Screenshot.is_gallery,
)
if public_only:
query = query.filter(Screenshot.is_public)
else:
query = query.filter(
or_(Screenshot.user_id == user_id, Screenshot.is_public)
)
query = query.order_by(Screenshot.created_at.desc())
return session.scalars(query).all()
@begin_session
def update_screenshot(
self,
backend/handler/database/states_handler.py
+33 -1
View File
@@ -1,6 +1,6 @@
from collections.abc import Sequence
from sqlalchemy import and_, delete, select, update
from sqlalchemy import and_, delete, desc, or_, select, update
from sqlalchemy.orm import QueryableAttribute, Session, load_only
from decorators.database import begin_session
@@ -66,6 +66,38 @@ class DBStatesHandler(DBBaseHandler):
return session.scalars(query).all()
@begin_session
def get_state_by_id(
self,
id: int,
session: Session = None, # type: ignore
) -> State | None:
"""Fetch a state by id without scoping to an owner. Used for the
visibility toggle and community downloads, where the caller may not own
the state. Mirrors db_screenshot_handler.get_screenshot_by_id."""
return session.get(State, id)
@begin_session
def get_rom_shared_states(
self,
rom_id: int,
user_id: int,
public_only: bool = False,
session: Session = None, # type: ignore
) -> Sequence[State]:
"""States for a ROM visible to the requesting user: own (public +
private) plus other users' public ones. Mirrors
db_screenshot_handler.get_rom_gallery_screenshots."""
query = select(State).filter(State.rom_id == rom_id)
if public_only:
query = query.filter(State.is_public)
else:
query = query.filter(or_(State.user_id == user_id, State.is_public))
query = query.order_by(desc(State.updated_at))
return session.scalars(query).all()
@begin_session
def update_state(
self,
backend/handler/filesystem/__init__.py
+19 -2
View File
@@ -1,13 +1,30 @@
from .assets_handler import FSAssetsHandler
from .firmware_handler import FSFirmwareHandler
from .launchbox_handler import FSLaunchboxHandler, get_fs_launchbox_handler
from .platforms_handler import FSPlatformsHandler
from .resources_handler import FSResourcesHandler
from .roms_handler import FSRomsHandler
from .sync_handler import FSSyncHandler
from .sync_handler import FSSyncHandler, get_fs_sync_handler
fs_asset_handler = FSAssetsHandler()
fs_firmware_handler = FSFirmwareHandler()
fs_platform_handler = FSPlatformsHandler()
fs_rom_handler = FSRomsHandler()
fs_resource_handler = FSResourcesHandler()
fs_sync_handler = FSSyncHandler()
__all__ = [
"FSAssetsHandler",
"FSFirmwareHandler",
"FSLaunchboxHandler",
"FSPlatformsHandler",
"FSResourcesHandler",
"FSRomsHandler",
"FSSyncHandler",
"fs_asset_handler",
"fs_firmware_handler",
"fs_platform_handler",
"fs_resource_handler",
"fs_rom_handler",
"get_fs_launchbox_handler",
"get_fs_sync_handler",
]
backend/handler/filesystem/assets_handler.py
+59 -2
View File
@@ -1,6 +1,11 @@
import hashlib
import os
import threading
import zipfile
from typing import Final
import magic
from fastapi import HTTPException, UploadFile, status
from config import ASSETS_BASE_PATH
from logger.logger import log
@@ -8,6 +13,57 @@ from models.user import User
from .base_handler import FSHandler
# Image MIME types we trust to (a) accept as avatar uploads and (b) serve
# inline from the raw asset endpoint
SAFE_IMAGE_MIME_TYPES: Final[dict[str, str]] = {
"image/png": "png",
"image/jpeg": "jpg",
"image/webp": "webp",
"image/gif": "gif",
}
# libmagic loads its database on construction (~few MB read from disk), so we
# share a single Magic instance across requests. The underlying magic_t handle
# is not thread-safe, so guard from_buffer with a lock. Endpoints that call
# this validator may execute in worker threads under sync routes.
_MIME_DETECTOR = magic.Magic(mime=True)
_MIME_DETECTOR_LOCK = threading.Lock()
def validate_image_upload(upload: UploadFile, *, label: str = "Image") -> str:
"""Validate that an uploaded file is one of the safe image types.
Sniffs the leading bytes with libmagic and returns the trusted extension
matching the detected MIME type. Raises HTTPException(400) if the file
is not a recognized PNG/JPEG/WebP/GIF, or if MIME sniffing fails.
Leaves the file cursor at 0.
"""
upload.file.seek(0)
header = upload.file.read(4096)
upload.file.seek(0)
try:
with _MIME_DETECTOR_LOCK:
detected_mime = _MIME_DETECTOR.from_buffer(header)
except magic.MagicException as exc:
log.error(f"libmagic failed to sniff uploaded {label.lower()}: {exc}")
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Could not determine {label.lower()} file type",
) from exc
safe_extension = SAFE_IMAGE_MIME_TYPES.get(detected_mime)
if not safe_extension:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=(
f"{label} must be a PNG, JPEG, WebP, or GIF image "
f"(detected {detected_mime or 'unknown type'})"
),
)
return safe_extension
class FSAssetsHandler(FSHandler):
def __init__(self) -> None:
@@ -76,7 +132,7 @@ class FSAssetsHandler(FSHandler):
return hash_obj.hexdigest()
async def _compute_zip_hash(self, zip_path: str) -> str:
with zipfile.ZipFile(f"{self.base_path}/{zip_path}", "r") as zf:
with zipfile.ZipFile(self.base_path / zip_path, "r") as zf:
file_hashes = []
for name in sorted(zf.namelist()):
if not name.endswith("/"):
@@ -88,7 +144,8 @@ class FSAssetsHandler(FSHandler):
async def compute_content_hash(self, file_path: str) -> str | None:
try:
if zipfile.is_zipfile(file_path):
full_path = self.base_path / file_path
if zipfile.is_zipfile(full_path):
return await self._compute_zip_hash(file_path)
return await self._compute_file_hash(file_path)
except Exception as e:
backend/handler/filesystem/base_handler.py
+95 -26
View File
@@ -16,11 +16,14 @@ from anyio import open_file
from starlette.datastructures import UploadFile
from config.config_manager import config_manager as cm
from models.base import FILE_NAME_MAX_LENGTH
from utils.filesystem import iter_directories, iter_files
from models.base import (
FILE_NAME_MAX_LENGTH,
compute_file_extension,
compute_file_name_no_ext,
compute_file_name_no_tags,
)
from utils.filesystem import iter_directories, iter_files, link_or_copy_file
TAG_REGEX = re.compile(r"\(([^)]+)\)|\[([^]]+)\]")
EXTENSION_REGEX = re.compile(r"\.(([a-z]+\.)*\w+)$")
UUID_V4_REGEX = re.compile(
r"[0-9a-f]{8}-[0-9a-f]{4}-[0-5][0-9a-f]{3}-[089ab][0-9a-f]{3}-[0-9a-f]{12}",
re.IGNORECASE,
@@ -88,6 +91,50 @@ REGIONS = (
REGIONS_BY_SHORTCODE = {region[0]: region[1] for region in REGIONS}
REGIONS_NAME_KEYS = frozenset(region[1].lower() for region in REGIONS)
# Maps full REGIONS names to lowercase shortcodes used by metadata providers
REGION_NAME_TO_PROVIDER_SHORTCODE: dict[str, str] = {
"Australia": "au",
"Asia": "asi",
"Brazil": "br",
"Canada": "ca",
"China": "cn",
"England": "uk",
"Europe": "eu",
"Finland": "fi",
"France": "fr",
"Germany": "de",
"Greece": "gr",
"Holland": "nl",
"Hong Kong": "hk",
"Italy": "it",
"Japan": "jp",
"Korea": "kr",
"Netherlands": "nl",
"Norway": "no",
"Russia": "ru",
"Spain": "sp",
"Sweden": "se",
"Taiwan": "tw",
"USA": "us",
"World": "wor",
}
_REGION_NAME_TO_PROVIDER_SHORTCODE_CI = {
k.lower(): v for k, v in REGION_NAME_TO_PROVIDER_SHORTCODE.items()
}
def region_name_to_provider_shortcode(region_name: str | None) -> str | None:
"""Look up a provider shortcode for a region name (case-insensitive).
ROM filename parsing can leave region names in their raw casing
(e.g. "europe", "EUROPE"), so callers must normalize before lookup.
"""
if not region_name:
return None
return _REGION_NAME_TO_PROVIDER_SHORTCODE_CI.get(region_name.lower())
LANGUAGES_BY_SHORTCODE = {lang[0]: lang[1] for lang in LANGUAGES}
LANGUAGES_NAME_KEYS = frozenset(lang[1].lower() for lang in LANGUAGES)
@@ -108,8 +155,6 @@ class FSHandler:
self.base_path = Path(base_path).resolve()
self._locks: dict[str, asyncio.Lock] = {}
self._lock_mutex = asyncio.Lock()
# Create base directory synchronously during initialization
self.base_path.mkdir(parents=True, exist_ok=True)
async def _get_file_lock(self, file_path: str) -> asyncio.Lock:
@@ -158,11 +203,24 @@ class FSHandler:
full_path = base_path_obj / path_path
try:
if full_path.is_symlink():
# For symlinks, ensure the symlink itself is within base directory
# Detect a symlink anywhere in the path, not just at the leaf —
# users may symlink an intermediate directory (e.g. the library
# root) to point at storage on another filesystem.
has_symlink_in_path = full_path.is_symlink()
if not has_symlink_in_path:
for parent in full_path.parents:
if parent == base_path_obj:
break
if parent.is_symlink():
has_symlink_in_path = True
break
if has_symlink_in_path:
# Validate lexically — `..` and absolute paths are already
# rejected above, so the symlink target is reachable only via
# an intentionally-configured link.
full_path.relative_to(base_path_obj)
else:
# For regular files/dirs, ensure resolved path is within base directory
full_path.resolve().relative_to(base_path_obj)
except ValueError as exc:
raise ValueError(
@@ -195,37 +253,37 @@ class FSHandler:
raise
def get_file_name_with_no_extension(self, file_name: str) -> str:
return EXTENSION_REGEX.sub("", file_name).strip()
return compute_file_name_no_ext(file_name)
def get_file_name_with_no_tags(self, file_name: str) -> str:
file_name_no_extension = self.get_file_name_with_no_extension(file_name)
return TAG_REGEX.split(file_name_no_extension)[0].strip()
return compute_file_name_no_tags(file_name)
def parse_file_extension(self, file_name: str) -> str:
match = EXTENSION_REGEX.search(file_name)
return match.group(1) if match else ""
return compute_file_extension(file_name)
def extract_uuid_v4_from_filename(self, file_name: str) -> str:
match = UUID_V4_REGEX.search(file_name)
return match.group(0) if match else ""
def exclude_single_files(self, files: list[str]) -> list[str]:
excluded_extensions = cm.get_config().EXCLUDED_SINGLE_EXT
excluded_names = cm.get_config().EXCLUDED_SINGLE_FILES
cnfg = cm.get_config()
excluded_extensions = cnfg.EXCLUDED_SINGLE_EXT
excluded_names = cnfg.EXCLUDED_SINGLE_FILES
excluded_files: list[str] = []
for file_name in files:
# Split the file name to get the extension.
ext = self.parse_file_extension(file_name)
file_name_lower = file_name.lower()
# Exclude the file if it has no extension or the extension is in the excluded list.
if ext and ext.lower() in excluded_extensions:
# Check whether the filename ends with any excluded extension entry.
if any(file_name_lower.endswith("." + ext) for ext in excluded_extensions):
excluded_files.append(file_name)
continue
# Additionally, check if the file name mathes a pattern in the excluded list.
for name in excluded_names:
if file_name == name or fnmatch.fnmatch(file_name, name):
excluded_files.append(file_name)
# Check if the file name matches a pattern in the excluded list.
if file_name in excluded_names or any(
fnmatch.fnmatch(file_name, name) for name in excluded_names
):
excluded_files.append(file_name)
# Return files that are not in the filtered list.
return [f for f in files if f not in excluded_files]
@@ -441,13 +499,21 @@ class FSHandler:
return await open_file(full_path, "rb")
async def copy_file(self, source_full_path: Path, dest_path: str) -> None:
async def copy_file(
self,
source_full_path: Path,
dest_path: str,
allow_link: bool = False,
) -> None:
"""
Copy a file from source to destination.
Args:
source_full_path: Absolute path to the source file
dest_path: Relative path to the destination file
allow_link: Try a hardlink first and fall back to
a copy when the link isn't possible (cross-device,
unsupported filesystem, etc.)
Raises:
FileNotFoundError: If source file does not exist
@@ -472,7 +538,10 @@ class FSHandler:
# Create destination directory if needed
dest_parent_anyio_path = AnyioPath(str(dest_full_path.parent))
await dest_parent_anyio_path.mkdir(parents=True, exist_ok=True)
shutil.copy2(str(source_full_path), str(dest_full_path))
if allow_link:
link_or_copy_file(source_full_path, dest_full_path)
else:
shutil.copy2(str(source_full_path), str(dest_full_path))
async def move_file_or_folder(self, source_path: str, dest_path: str) -> None:
"""
backend/handler/filesystem/firmware_handler.py
+3 -4
View File
@@ -1,6 +1,5 @@
import binascii
import hashlib
import os
from config import LIBRARY_BASE_PATH
from config.config_manager import config_manager as cm
@@ -17,9 +16,9 @@ class FSFirmwareHandler(FSHandler):
def get_firmware_fs_structure(self, fs_slug: str) -> str:
cnfg = cm.get_config()
return (
f"{cnfg.FIRMWARE_FOLDER_NAME}/{fs_slug}"
if os.path.exists(cnfg.HIGH_PRIO_STRUCTURE_PATH)
else f"{fs_slug}/{cnfg.FIRMWARE_FOLDER_NAME}"
f"{fs_slug}/{cnfg.FIRMWARE_FOLDER_NAME}"
if cnfg.has_structure_path_b
else f"{cnfg.FIRMWARE_FOLDER_NAME}/{fs_slug}"
)
async def get_firmware(self, platform_fs_slug: str):
backend/handler/filesystem/launchbox_handler.py
+19
View File
@@ -0,0 +1,19 @@
import functools
from config import LAUNCHBOX_BASE_PATH
from handler.filesystem.base_handler import FSHandler
class FSLaunchboxHandler(FSHandler):
def __init__(self) -> None:
super().__init__(base_path=LAUNCHBOX_BASE_PATH)
@functools.cache
def get_fs_launchbox_handler() -> FSLaunchboxHandler:
"""Lazily instantiate the LaunchBox handler on first use.
Deferred so that startup doesn't fail when the LaunchBox feature is
unconfigured or its base path is not writable.
"""
return FSLaunchboxHandler()
backend/handler/filesystem/platforms_handler.py
+41 -31
View File
@@ -4,10 +4,8 @@ from anyio import Path as AnyioPath
from config import LIBRARY_BASE_PATH
from config.config_manager import config_manager as cm
from exceptions.fs_exceptions import (
FolderStructureNotMatchException,
PlatformAlreadyExistsException,
)
from exceptions.fs_exceptions import PlatformAlreadyExistsException
from logger.logger import log
from .base_handler import FSHandler, LibraryStructure
@@ -33,49 +31,45 @@ class FSPlatformsHandler(FSHandler):
def detect_library_structure(self) -> LibraryStructure | None:
"""Detects the library structure type.
Structure A ({roms_folder}/{platform}) takes priority over Structure B
({platform}/{roms_folder}) so that existing libraries are not broken when a
stray {platform}/{roms_folder} directory happens to exist alongside them.
Returns:
"LibraryStructure.A" for Structure A (roms/{platform})
"LibraryStructure.B" for Structure B ({platform}/roms)
None if no structure detected
"LibraryStructure.A" for Structure A (roms/{platform}) when the
top-level roms folder exists.
"LibraryStructure.B" for Structure B ({platform}/roms) when no
top-level roms folder exists but at least one platform has a
roms subfolder.
None if no structure detected.
"""
cnfg = cm.get_config()
# Check if the roms folder exists (Structure A indicator)
roms_path = os.path.join(LIBRARY_BASE_PATH, cnfg.ROMS_FOLDER_NAME)
if os.path.exists(roms_path):
if cnfg.has_structure_path_a:
return LibraryStructure.A
# Check if any platform folders with roms subfolders exist (Structure B)
try:
library_contents = os.listdir(LIBRARY_BASE_PATH)
for item in library_contents:
item_path = os.path.join(LIBRARY_BASE_PATH, item)
roms_subfolder = os.path.join(item_path, cnfg.ROMS_FOLDER_NAME)
if os.path.isdir(item_path) and os.path.exists(roms_subfolder):
return LibraryStructure.B
except (OSError, FileNotFoundError):
pass
if cnfg.has_structure_path_b:
return LibraryStructure.B
return None
def get_platforms_directory(self) -> str:
cnfg = cm.get_config()
# Fallback to config hint when detection is inconclusive
return (
cnfg.ROMS_FOLDER_NAME
if os.path.exists(cnfg.HIGH_PRIO_STRUCTURE_PATH)
else ""
)
# Fallback to config hint when detection is inconclusive: default to
# Structure A (roms/{platform}) so the bare library root is not treated
# as a flat list of platforms. When the roms folder is missing entirely,
# get_platforms() bootstraps it instead of failing.
return "" if cnfg.has_structure_path_b else cnfg.ROMS_FOLDER_NAME
def get_platform_fs_structure(self, fs_slug: str) -> str:
cnfg = cm.get_config()
# Fallback to config hint when detection is inconclusive
return (
f"{cnfg.ROMS_FOLDER_NAME}/{fs_slug}"
if os.path.exists(cnfg.HIGH_PRIO_STRUCTURE_PATH)
else f"{fs_slug}/{cnfg.ROMS_FOLDER_NAME}"
f"{fs_slug}/{cnfg.ROMS_FOLDER_NAME}"
if cnfg.has_structure_path_b
else f"{cnfg.ROMS_FOLDER_NAME}/{fs_slug}"
)
async def add_platform(self, fs_slug: str) -> None:
@@ -94,6 +88,11 @@ class FSPlatformsHandler(FSHandler):
async def get_platforms(self) -> list[str]:
"""Retrieves all platforms from the filesystem.
If no library structure exists yet (neither Structure A's top-level roms
folder nor a Structure B {platform}/roms folder), defaults to Structure A
by creating the roms folder and returns an empty list, so RomM starts
cleanly with an empty library instead of failing.
Returns:
List of platform slugs.
"""
@@ -101,8 +100,19 @@ class FSPlatformsHandler(FSHandler):
try:
platforms = await self.list_directories(path=self.get_platforms_directory())
except FileNotFoundError as e:
raise FolderStructureNotMatchException() from e
except FileNotFoundError:
# The platforms directory does not exist, which means no library
# structure has been set up yet. Bootstrap Structure A so the
# filesystem is in a valid state and report an empty library.
log.warning(
"No library structure found; creating default Structure A "
"(roms folder) and starting with an empty library."
)
try:
self.create_library_structure()
except OSError:
log.error("Failed to create default library structure", exc_info=True)
return []
# For Structure B, only include directories that have a roms subfolder
structure = self.detect_library_structure()
backend/handler/filesystem/resources_handler.py
+64 -64
View File
@@ -15,10 +15,37 @@ from models.collection import Collection
from models.rom import Rom
from tasks.scheduled.convert_images_to_webp import ImageConverter
from utils.context import ctx_httpx_client
from utils.validation import validate_url_for_http_request
from .base_handler import CoverSize, FSHandler
LOCAL_FILE_SCHEMES = ("file://", "launchbox-file://")
def _resolve_local_file_uri(uri: str) -> Path | None:
"""Resolve a local-file URI to an absolute Path, or None if unsafe/unknown.
`file://` resolves under the ROM library root. `launchbox-file://` resolves
under the LaunchBox data root, since LaunchBox metadata produces paths
relative to `/romm/launchbox`, which is not the same as the library root.
"""
from handler.filesystem import fs_rom_handler, get_fs_launchbox_handler
if uri.startswith("launchbox-file://"):
try:
return get_fs_launchbox_handler().validate_path(
uri[len("launchbox-file://") :]
)
except ValueError:
return None
if uri.startswith("file://"):
try:
return fs_rom_handler.validate_path(uri[len("file://") :])
except ValueError:
return None
return None
def _content_type_essence(header_value: str) -> str:
"""Return the MIME type token (before parameters), lowercased."""
@@ -95,34 +122,28 @@ class FSResourcesHandler(FSHandler):
cover_file = f"{entity.fs_resources_path}/cover"
await self.make_directory(cover_file)
# Handle file:// URLs for gamelist.xml
if url_cover.startswith("file://"):
# Handle local-file URIs from metadata handlers (gamelist, LaunchBox)
if url_cover.startswith(LOCAL_FILE_SCHEMES):
try:
from handler.filesystem import fs_rom_handler
validated = fs_rom_handler.validate_path(
url_cover[7:] # Remove "file://" prefix
)
if await AnyioPath(validated).exists():
# Copy the file to the resources directory
dest_path = f"{cover_file}/{size.value}.png"
await self.copy_file(validated, dest_path)
if ENABLE_SCHEDULED_CONVERT_IMAGES_TO_WEBP:
self.image_converter.convert_to_webp(
self.validate_path(f"{cover_file}/{size.value}.png"),
force=True,
)
else:
log.warning(f"Cover file not found: {str(validated)}")
resolved = _resolve_local_file_uri(url_cover)
if resolved is None or not await AnyioPath(resolved).exists():
log.warning(f"Cover file not found: {url_cover}")
return None
dest_path = f"{cover_file}/{size.value}.png"
# Small-size covers get resized in place, which would mutate
# the user's source image if the destination were a hardlink.
await self.copy_file(resolved, dest_path, allow_link=False)
if ENABLE_SCHEDULED_CONVERT_IMAGES_TO_WEBP:
self.image_converter.convert_to_webp(
self.validate_path(f"{cover_file}/{size.value}.png"),
force=True,
)
except Exception as exc:
log.error(f"Unable to copy cover file {url_cover}: {str(exc)}")
return None
else:
# Handle HTTP URLs
validate_url_for_http_request(url_cover, "url_cover")
httpx_client = ctx_httpx_client.get()
try:
async with httpx_client.stream(
@@ -275,27 +296,21 @@ class FSResourcesHandler(FSHandler):
screenshot_path = f"{rom.fs_resources_path}/screenshots"
await self.make_directory(screenshot_path)
# Handle file:// URLs for gamelist.xml
if url_screenhot.startswith("file://"):
# Handle local-file URIs from metadata handlers (gamelist, LaunchBox)
if url_screenhot.startswith(LOCAL_FILE_SCHEMES):
try:
from handler.filesystem import fs_rom_handler
validated = fs_rom_handler.validate_path(
url_screenhot[7:] # Remove "file://" prefix
)
if await AnyioPath(validated).exists():
# Copy the file to the resources directory
await self.copy_file(validated, f"{screenshot_path}/{idx}.jpg")
else:
log.warning(f"Screenshot file not found: {str(validated)}")
resolved = _resolve_local_file_uri(url_screenhot)
if resolved is None or not await AnyioPath(resolved).exists():
log.warning(f"Screenshot file not found: {url_screenhot}")
return None
await self.copy_file(
resolved, f"{screenshot_path}/{idx}.jpg", allow_link=True
)
except Exception as exc:
log.error(f"Unable to copy screenshot file {url_screenhot}: {str(exc)}")
return None
else:
# Handle HTTP URLs
validate_url_for_http_request(url_screenhot, "url_screenshot")
httpx_client = ctx_httpx_client.get()
try:
async with httpx_client.stream(
@@ -395,27 +410,21 @@ class FSResourcesHandler(FSHandler):
manual_path = f"{rom.fs_resources_path}/manual"
await self.make_directory(manual_path)
# Handle file:// URLs for gamelist.xml
if url_manual.startswith("file://"):
# Handle local-file URIs from metadata handlers (gamelist, LaunchBox)
if url_manual.startswith(LOCAL_FILE_SCHEMES):
try:
from handler.filesystem import fs_rom_handler
validated = fs_rom_handler.validate_path(
url_manual[7:] # Remove "file://" prefix
)
if await AnyioPath(validated).exists():
# Copy the file to the resources directory
await self.copy_file(validated, f"{manual_path}/{rom.id}.pdf")
else:
log.warning(f"Manual file not found: {str(validated)}")
resolved = _resolve_local_file_uri(url_manual)
if resolved is None or not await AnyioPath(resolved).exists():
log.warning(f"Manual file not found: {url_manual}")
return None
await self.copy_file(
resolved, f"{manual_path}/{rom.id}.pdf", allow_link=True
)
except Exception as exc:
log.error(f"Unable to copy manual file {url_manual}: {str(exc)}")
return None
else:
# Handle HTTP URL
validate_url_for_http_request(url_manual, "url_manual")
httpx_client = ctx_httpx_client.get()
try:
async with httpx_client.stream(
@@ -485,8 +494,6 @@ class FSResourcesHandler(FSHandler):
# Retroachievements
async def store_ra_badge(self, url: str, path: str) -> None:
validate_url_for_http_request(url, "url_badge")
httpx_client = ctx_httpx_client.get()
directory, filename = os.path.split(path)
@@ -542,24 +549,17 @@ class FSResourcesHandler(FSHandler):
# Ensure destination directory exists
await self.make_directory(directory)
# Handle file:// URLs for gamelist.xml
if url_media.startswith("file://"):
# Handle local-file URIs from metadata handlers (gamelist, LaunchBox)
if url_media.startswith(LOCAL_FILE_SCHEMES):
try:
from handler.filesystem import fs_rom_handler
validated = fs_rom_handler.validate_path(
url_media[7:] # Remove "file://" prefix
)
file_path = AnyioPath(validated)
if await file_path.exists():
await self.copy_file(Path(str(file_path)), dest_path)
resolved = _resolve_local_file_uri(url_media)
if resolved is not None and await AnyioPath(resolved).exists():
await self.copy_file(resolved, dest_path, allow_link=True)
except Exception as exc:
log.error(f"Unable to copy media file {url_media}: {str(exc)}")
return None
else:
# Handle HTTP URLs
validate_url_for_http_request(url_media, "url_media")
httpx_client = ctx_httpx_client.get()
try:
async with httpx_client.stream(
backend/handler/filesystem/roms_handler.py
+243 -248
View File
@@ -1,22 +1,21 @@
import asyncio
import binascii
import bz2
import fnmatch
import hashlib
import os
import re
import tarfile
import zipfile
import zlib
from collections.abc import Callable, Iterator
from dataclasses import dataclass
from pathlib import Path
from typing import IO, Any, Final, Literal, TypedDict, cast
from typing import Any, TypedDict
import magic
import zipfile_inflate64 # trunk-ignore(ruff/F401): Patches zipfile to support Enhanced Deflate
from anyio import Path as AnyioPath
from config import LIBRARY_BASE_PATH
from config.config_manager import (
DEFAULT_EXCLUDED_EXTENSIONS,
DEFAULT_EXCLUDED_FILES,
)
from config.config_manager import config_manager as cm
from exceptions.fs_exceptions import (
RomAlreadyExistsException,
@@ -25,8 +24,22 @@ from exceptions.fs_exceptions import (
from handler.metadata.base_handler import UniversalPlatformSlug as UPS
from models.platform import Platform
from models.rom import Rom, RomFile, RomFileCategory
from utils.archive_7zip import process_file_7z
from utils.filesystem import COMPRESSED_FILE_EXTENSIONS, iter_files
from utils.archives import (
detect_mime_type,
extract_chd_hash,
is_chd_file,
process_7z_file,
read_7z_archive_files,
read_basic_file,
read_bz2_file,
read_gz_file,
read_rar_archive_files,
read_tar_archive_files,
read_tar_file,
read_zip_archive_files,
read_zip_file,
)
from utils.filesystem import iter_files
from utils.hashing import crc32_to_hex
from .base_handler import (
@@ -34,37 +47,13 @@ from .base_handler import (
LANGUAGES_NAME_KEYS,
REGIONS_BY_SHORTCODE,
REGIONS_NAME_KEYS,
TAG_REGEX,
FSHandler,
)
# Known compressed file MIME types
COMPRESSED_MIME_TYPES: Final = frozenset(
(
"application/x-7z-compressed",
"application/x-bzip2",
"application/x-gzip",
"application/x-tar",
"application/zip",
)
)
# PICO-8 cartridges are often stored as PNG files
PICO8_CARTRIDGE_EXTENSION = ".p8.png"
# CHD (Compressed Hunks of Data) v5 format constants
# See: https://github.com/mamedev/mame/blob/master/src/lib/util/chd.h
CHD_SIGNATURE: Final = b"MComprHD"
CHD_SIGNATURE_LENGTH: Final = 8
CHD_MIN_HEADER_LENGTH: Final = 16 # Minimum to read signature and version
CHD_V5_HEADER_LENGTH: Final = 124 # Total v5 header size
CHD_VERSION_OFFSET: Final = 12 # Bytes offset for version field
CHD_VERSION_LENGTH: Final = 4 # Version is a uint32
CHD_V5_SHA1_OFFSET: Final = 84 # Combined raw+meta SHA1 offset in v5
CHD_V5_SHA1_LENGTH: Final = 20 # SHA1 is 20 bytes
CHD_V5_VERSION: Final = 5 # CHD v5 identifier
NON_HASHABLE_PLATFORMS = frozenset(
(
UPS.AMAZON_ALEXA,
@@ -96,8 +85,6 @@ NON_HASHABLE_PLATFORMS = frozenset(
)
)
FILE_READ_CHUNK_SIZE = 1024 * 8
class FSRom(TypedDict):
fs_name: str
@@ -114,165 +101,7 @@ class FileHash(TypedDict):
crc_hash: str
md5_hash: str
sha1_hash: str
def is_compressed_file(file_path: str) -> bool:
mime = magic.Magic(mime=True)
file_type = mime.from_file(file_path)
return file_type in COMPRESSED_MIME_TYPES or file_path.lower().endswith(
tuple(COMPRESSED_FILE_EXTENSIONS)
)
def read_basic_file(file_path: os.PathLike[str]) -> Iterator[bytes]:
with open(file_path, "rb") as f:
while chunk := f.read(FILE_READ_CHUNK_SIZE):
yield chunk
def read_zip_file(file: str | os.PathLike[str] | IO[bytes]) -> Iterator[bytes]:
try:
with zipfile.ZipFile(file, "r") as z:
# Find the biggest file in the archive
largest_file = max(z.infolist(), key=lambda x: x.file_size)
with z.open(largest_file, "r") as f:
while chunk := f.read(FILE_READ_CHUNK_SIZE):
yield chunk
except zipfile.BadZipFile:
if isinstance(file, Path):
for chunk in read_basic_file(file):
yield chunk
def read_tar_file(
file_path: Path, mode: Literal["r", "r:*", "r:", "r:gz", "r:bz2", "r:xz"] = "r"
) -> Iterator[bytes]:
try:
with tarfile.open(file_path, mode) as f:
regular_files = [member for member in f.getmembers() if member.isfile()]
# Find the largest file among regular files only
largest_file = max(regular_files, key=lambda x: x.size)
with f.extractfile(largest_file) as ef: # type: ignore
while chunk := ef.read(FILE_READ_CHUNK_SIZE):
yield chunk
except tarfile.ReadError:
for chunk in read_basic_file(file_path):
yield chunk
def read_gz_file(file_path: Path) -> Iterator[bytes]:
return read_tar_file(file_path, "r:gz")
def process_7z_file(
file_path: Path,
fn_hash_update: Callable[[bytes | bytearray], None],
) -> None:
processed = process_file_7z(
file_path=file_path,
fn_hash_update=fn_hash_update,
)
if not processed:
for chunk in read_basic_file(file_path):
fn_hash_update(chunk)
def read_bz2_file(file_path: Path) -> Iterator[bytes]:
try:
with bz2.BZ2File(file_path, "rb") as f:
while chunk := f.read(FILE_READ_CHUNK_SIZE):
yield chunk
except EOFError:
for chunk in read_basic_file(file_path):
yield chunk
def extract_chd_hash(file_path: Path) -> str | None:
"""
Extract the embedded SHA1 hash from a CHD (Compressed Hunks of Data) v5 file header.
Only CHD v5 files are supported, matching MAMERedump's database.
CHD v5 files store the combined raw+meta SHA1 hash in the header.
This hash is what ROM databases use for CHD identification, since it includes
metadata like CD track layouts which are essential for proper disc image
identification.
For reference, check out "chd.h" in the MAME source tree.
---------------------------------- Why? ----------------------------------
CHDMAN does not produce nor guarantee stable, byte-for-byte identical
outputs for a given disc image. (Including HD images.)
For this reason, the CHD format embeds the original source data hash in
its header, allowing different CHD files to be verified as equivalent
even when their compressed representations differ.
--------------------------------------------------------------------------
Args:
file_path: Path to the CHD file
Returns:
SHA1 hash as hex string, or None if file is not a valid CHD v5 file or parsing fails
"""
try:
with open(file_path, "rb") as f:
# Read the v5 header and extract the embedded SHA1
header = f.read(CHD_V5_HEADER_LENGTH)
# Check for "MComprHD" signature
if (
len(header) < CHD_MIN_HEADER_LENGTH
or header[:CHD_SIGNATURE_LENGTH] != CHD_SIGNATURE
):
return None
# Extract and verify version (big-endian uint32)
version_end = CHD_VERSION_OFFSET + CHD_VERSION_LENGTH
version = int.from_bytes(header[CHD_VERSION_OFFSET:version_end], "big")
# Only support v5 CHD files
if version != CHD_V5_VERSION:
return None
# Extract combined raw+meta SHA1 from v5 header
sha1_end = CHD_V5_SHA1_OFFSET + CHD_V5_SHA1_LENGTH
if len(header) < sha1_end:
return None
sha1_bytes = header[CHD_V5_SHA1_OFFSET:sha1_end]
return sha1_bytes.hex()
except OSError:
return None
class CHDHashWrapper:
"""
Wrapper class that mimics hashlib hash objects but returns a pre-computed hash.
This class provides a hashlib-compatible interface for pre-computed hashes
extracted from CHD v5 file headers. It implements the same methods and attributes
as hashlib hash objects (digest(), hexdigest(), update(), and name).
"""
def __init__(self, hash_hex: str, name: str):
self.hash_hex = hash_hex
self.name = name
# Store the digest as bytes
self._digest = bytes.fromhex(hash_hex)
def hexdigest(self) -> str:
"""Return the hash as a hexadecimal string."""
return self.hash_hex
def digest(self) -> bytes:
"""Return the hash as bytes."""
return self._digest
def update(self, data: bytes | bytearray) -> None:
"""No-op update method for compatibility with hashlib interface."""
pass
chd_sha1_hash: str
def category_matches(category: str, path_parts: list[str]):
@@ -283,6 +112,35 @@ DEFAULT_CRC_C = 0
DEFAULT_MD5_H_DIGEST = hashlib.md5(usedforsecurity=False).digest()
DEFAULT_SHA1_H_DIGEST = hashlib.sha1(usedforsecurity=False).digest()
ARCHIVE_READERS = {
".zip": read_zip_archive_files,
".tar": read_tar_archive_files,
".tar.gz": read_tar_archive_files,
".tgz": read_tar_archive_files,
".tar.bz2": read_tar_archive_files,
".tbz2": read_tar_archive_files,
".tar.xz": read_tar_archive_files,
".txz": read_tar_archive_files,
".7z": read_7z_archive_files,
".rar": read_rar_archive_files,
}
def _make_file_hash(
crc_c: int, md5_h: Any, sha1_h: Any, chd_sha1_hash: str = ""
) -> FileHash:
"""Build a FileHash, blanking each field whose hasher state is still the default."""
return FileHash(
crc_hash=crc32_to_hex(crc_c) if crc_c != DEFAULT_CRC_C else "",
md5_hash=md5_h.hexdigest() if md5_h.digest() != DEFAULT_MD5_H_DIGEST else "",
sha1_hash=(
sha1_h.hexdigest() if sha1_h.digest() != DEFAULT_SHA1_H_DIGEST else ""
),
chd_sha1_hash=chd_sha1_hash,
)
GENERIC_TAG_REGEX = re.compile(r"\(([^)]+)\)|\[([^]]+)\]")
VERSION_TAG_REGEX = re.compile(r"^(?:version|ver|v)[\s_-]?(.*)", re.I)
REGION_TAG_REGEX = re.compile(r"^reg[\s|-](.*)$", re.I)
REVISION_TAG_REGEX = re.compile(r"^rev[\s|-](.*)$", re.I)
@@ -313,15 +171,15 @@ class FSRomsHandler(FSHandler):
def get_roms_fs_structure(self, fs_slug: str) -> str:
cnfg = cm.get_config()
return (
f"{cnfg.ROMS_FOLDER_NAME}/{fs_slug}"
if os.path.exists(cnfg.HIGH_PRIO_STRUCTURE_PATH)
else f"{fs_slug}/{cnfg.ROMS_FOLDER_NAME}"
f"{fs_slug}/{cnfg.ROMS_FOLDER_NAME}"
if cnfg.has_structure_path_b
else f"{cnfg.ROMS_FOLDER_NAME}/{fs_slug}"
)
def parse_tags(self, fs_name: str) -> ParsedTags:
tags = [
chunk.strip()
for tag in (m[0] or m[1] for m in TAG_REGEX.findall(fs_name))
for tag in (m[0] or m[1] for m in GENERIC_TAG_REGEX.findall(fs_name))
for chunk in tag.split(",")
]
@@ -379,18 +237,36 @@ class FSRomsHandler(FSHandler):
def exclude_multi_roms(self, roms: list[str]) -> list[str]:
excluded_names = cm.get_config().EXCLUDED_MULTI_FILES
filtered_files: list = []
normalized_patterns = [
excluded_name.lower().strip() for excluded_name in excluded_names
]
kept_roms: list[str] = []
for rom in roms:
if rom in excluded_names:
filtered_files.append(rom)
normalized_rom_name = rom.strip().lower()
if normalized_rom_name in normalized_patterns:
continue
return [f for f in roms if f not in filtered_files]
if any(
fnmatch.fnmatch(normalized_rom_name, pattern)
for pattern in normalized_patterns
):
continue
kept_roms.append(rom)
return kept_roms
def _build_rom_file(
self, rom: Rom, rom_path: Path, file_name: str, file_hash: FileHash
self,
rom: Rom,
rom_path: Path,
file_name: str,
file_hash: FileHash,
file_size_bytes: int | None = None,
last_modified: float | None = None,
archive_members: list[dict[str, Any]] | None = None,
) -> RomFile:
# Absolute path to roms
abs_file_path = Path(self.base_path, rom_path, file_name)
path_parts_lower = list(map(str.lower, rom_path.parts))
@@ -403,17 +279,35 @@ class FSRomsHandler(FSHandler):
None,
)
audio_meta = None
if matching_category == RomFileCategory.SOUNDTRACK:
from utils.audio_tags import extract_audio_meta, is_allowed_audio_file
if is_allowed_audio_file(file_name):
audio_meta = extract_audio_meta(str(abs_file_path))
return RomFile(
rom=rom,
rom_id=rom.id,
file_name=file_name,
file_path=str(rom_path),
file_size_bytes=os.stat(abs_file_path).st_size,
last_modified=os.path.getmtime(abs_file_path),
file_size_bytes=(
file_size_bytes
if file_size_bytes is not None
else os.stat(abs_file_path).st_size
),
last_modified=(
last_modified
if last_modified is not None
else os.path.getmtime(abs_file_path)
),
category=matching_category,
audio_meta=audio_meta,
crc_hash=file_hash["crc_hash"],
md5_hash=file_hash["md5_hash"],
sha1_hash=file_hash["sha1_hash"],
chd_sha1_hash=file_hash["chd_sha1_hash"],
archive_members=archive_members,
)
async def get_rom_files(
@@ -433,33 +327,58 @@ class FSRomsHandler(FSHandler):
rom.platform_slug not in NON_HASHABLE_PLATFORMS and calculate_hashes
)
excluded_file_names = cm.get_config().EXCLUDED_MULTI_PARTS_FILES
excluded_file_exts = cm.get_config().EXCLUDED_MULTI_PARTS_EXT
cnfg = cm.get_config()
excluded_file_names = cnfg.EXCLUDED_MULTI_PARTS_FILES
excluded_file_exts = cnfg.EXCLUDED_MULTI_PARTS_EXT
rom_crc_c = 0
rom_md5_h = hashlib.md5(usedforsecurity=False) if calculate_hashes else None
rom_sha1_h = hashlib.sha1(usedforsecurity=False) if calculate_hashes else None
rom_ra_h = ""
rom_dir = Path(abs_fs_path, rom.fs_name)
rom_ext = f".{rom.fs_extension.lower()}" if rom.fs_extension else ""
# Check if rom is a multi-part rom
if await AnyioPath(f"{abs_fs_path}/{rom.fs_name}").is_dir():
# Calculate the RA hash if the platform has a slug that matches a known RA slug
if calculate_hashes:
ra_platform = meta_ra_handler.get_platform(rom.platform_slug)
if ra_platform and ra_platform["ra_id"]:
# RAHasher can't process CHD files via the /* wildcard and instead expects
# track files (bin/cue/etc.). For CHD-only folders, find the largest
# CHD and pass it directly, matching single-file CHD behaviour.
def _largest_chd_file() -> Path | None:
chds = [f for f in rom_dir.iterdir() if is_chd_file(f)]
sorted_chds = sorted(
chds, key=lambda f: f.stat().st_size, reverse=True
)
return sorted_chds[0] if sorted_chds else None
chd_file = await asyncio.to_thread(_largest_chd_file)
ra_path = (
str(chd_file)
if chd_file and chd_file.is_file()
else f"{abs_fs_path}/{rom.fs_name}/*"
)
rom_ra_h = await RAHasherService().calculate_hash(
ra_platform,
f"{abs_fs_path}/{rom.fs_name}/*",
ra_path,
)
for f_path, file_name in iter_files(
f"{abs_fs_path}/{rom.fs_name}", recursive=True
):
# Check if file is excluded
ext = self.parse_file_extension(file_name)
if ext in excluded_file_exts:
# Check if file is excluded by extension.
f_rom_dir = Path(f_path, rom.fs_name)
file_name_lower = file_name.lower()
if any(
file_name_lower.endswith("." + ext) for ext in excluded_file_exts
):
continue
# Check if the file name matches a pattern in the excluded list.
if any(
file_name == exc_name or fnmatch.fnmatch(file_name, exc_name)
for exc_name in excluded_file_names
@@ -474,7 +393,8 @@ class FSRomsHandler(FSHandler):
if is_top_level:
# Include this file in the main ROM hash calculation
crc_c, rom_crc_c, md5_h, rom_md5_h, sha1_h, rom_sha1_h = (
self._calculate_rom_hashes(
await asyncio.to_thread(
self._calculate_rom_hashes,
Path(f_path, file_name),
rom_crc_c,
rom_md5_h,
@@ -483,7 +403,8 @@ class FSRomsHandler(FSHandler):
)
else:
# Calculate individual file hash only
crc_c, _, md5_h, _, sha1_h, _ = self._calculate_rom_hashes(
crc_c, _, md5_h, _, sha1_h, _ = await asyncio.to_thread(
self._calculate_rom_hashes,
Path(f_path, file_name),
0,
hashlib.md5(usedforsecurity=False),
@@ -494,16 +415,13 @@ class FSRomsHandler(FSHandler):
md5_h = hashlib.md5(usedforsecurity=False)
sha1_h = hashlib.sha1(usedforsecurity=False)
file_hash = FileHash(
crc_hash=crc32_to_hex(crc_c) if crc_c != DEFAULT_CRC_C else "",
md5_hash=(
md5_h.hexdigest()
if md5_h.digest() != DEFAULT_MD5_H_DIGEST
else ""
),
sha1_hash=(
sha1_h.hexdigest()
if sha1_h.digest() != DEFAULT_SHA1_H_DIGEST
file_hash = _make_file_hash(
crc_c,
md5_h,
sha1_h,
chd_sha1_hash=(
extract_chd_hash(f_rom_dir)
if is_chd_file(f_rom_dir)
else ""
),
)
@@ -512,6 +430,7 @@ class FSRomsHandler(FSHandler):
crc_hash="",
md5_hash="",
sha1_hash="",
chd_sha1_hash="",
)
rom_files.append(
@@ -522,11 +441,101 @@ class FSRomsHandler(FSHandler):
file_hash=file_hash,
)
)
elif hashable_platform and rom_ext in ARCHIVE_READERS:
# Multi-file archive: compute a composite hash across all
# internal entries (in ASCII path order) for hash-database
# matching, while still emitting a single RomFile for the
# archive file itself. Per-member hashes are stored on that
# RomFile in `archive_members` so consumers can identify each
# internal file without us inventing RomFile rows whose
# full_path would point inside the archive and break downloads.
assert rom_md5_h is not None and rom_sha1_h is not None
def _hash_archive_entries(
crc: int, md5_h: Any, sha1_h: Any
) -> tuple[list[dict[str, Any]], int]:
members: list[dict[str, Any]] = []
for name, size, chunks in ARCHIVE_READERS[rom_ext](
rom_dir,
DEFAULT_EXCLUDED_FILES,
DEFAULT_EXCLUDED_EXTENSIONS,
):
member_crc = 0
member_md5 = hashlib.md5(usedforsecurity=False)
member_sha1 = hashlib.sha1(usedforsecurity=False)
for chunk in chunks:
crc = binascii.crc32(chunk, crc)
md5_h.update(chunk)
sha1_h.update(chunk)
member_crc = binascii.crc32(chunk, member_crc)
member_md5.update(chunk)
member_sha1.update(chunk)
members.append(
{
"name": name,
"size": size,
"crc_hash": crc32_to_hex(member_crc),
"md5_hash": member_md5.hexdigest(),
"sha1_hash": member_sha1.hexdigest(),
}
)
return members, crc
members, rom_crc_c = await asyncio.to_thread(
_hash_archive_entries, rom_crc_c, rom_md5_h, rom_sha1_h
)
if members:
if calculate_hashes:
ra_platform = meta_ra_handler.get_platform(rom.platform_slug)
if ra_platform and ra_platform["ra_id"]:
rom_ra_h = await RAHasherService().calculate_hash(
ra_platform,
f"{abs_fs_path}/{rom.fs_name}",
)
rom_files.append(
self._build_rom_file(
rom=rom,
rom_path=Path(rel_roms_path),
file_name=rom.fs_name,
file_hash=_make_file_hash(rom_crc_c, rom_md5_h, rom_sha1_h),
archive_members=members,
)
)
else:
# Empty, malformed, or all-excluded archive: hash the archive
# file's raw bytes. We avoid `_calculate_rom_hashes` here because
# it would decompress based on extension and end up hashing the
# largest internal member, not the archive itself — and would
# crash on an empty zip. `archive_members` stays None.
def _hash_raw_archive(crc: int) -> int:
for chunk in read_basic_file(rom_dir):
crc = binascii.crc32(chunk, crc)
if rom_md5_h:
rom_md5_h.update(chunk)
if rom_sha1_h:
rom_sha1_h.update(chunk)
return crc
rom_crc_c = await asyncio.to_thread(_hash_raw_archive, rom_crc_c)
rom_files.append(
self._build_rom_file(
rom=rom,
rom_path=Path(rel_roms_path),
file_name=rom.fs_name,
file_hash=_make_file_hash(rom_crc_c, rom_md5_h, rom_sha1_h),
)
)
elif hashable_platform:
try:
crc_c, rom_crc_c, md5_h, rom_md5_h, sha1_h, rom_sha1_h = (
self._calculate_rom_hashes(
Path(abs_fs_path, rom.fs_name), rom_crc_c, rom_md5_h, rom_sha1_h
await asyncio.to_thread(
self._calculate_rom_hashes,
Path(abs_fs_path, rom.fs_name),
rom_crc_c,
rom_md5_h,
rom_sha1_h,
)
)
except zlib.error:
@@ -543,15 +552,12 @@ class FSRomsHandler(FSHandler):
f"{abs_fs_path}/{rom.fs_name}",
)
file_hash = FileHash(
crc_hash=crc32_to_hex(crc_c) if crc_c != DEFAULT_CRC_C else "",
md5_hash=(
md5_h.hexdigest() if md5_h.digest() != DEFAULT_MD5_H_DIGEST else ""
),
sha1_hash=(
sha1_h.hexdigest()
if sha1_h.digest() != DEFAULT_SHA1_H_DIGEST
else ""
file_hash = _make_file_hash(
crc_c,
md5_h,
sha1_h,
chd_sha1_hash=(
extract_chd_hash(rom_dir) if is_chd_file(rom_dir) else ""
),
)
rom_files.append(
@@ -567,6 +573,7 @@ class FSRomsHandler(FSHandler):
crc_hash="",
md5_hash="",
sha1_hash="",
chd_sha1_hash="",
)
rom_files.append(
self._build_rom_file(
@@ -601,9 +608,8 @@ class FSRomsHandler(FSHandler):
rom_sha1_h: Any,
) -> tuple[int, int, Any, Any, Any, Any]:
extension = Path(file_path).suffix.lower()
mime = magic.Magic(mime=True)
try:
file_type = mime.from_file(file_path)
file_type = detect_mime_type(file_path)
crc_c = 0
md5_h = hashlib.md5(usedforsecurity=False)
@@ -643,17 +649,6 @@ class FSRomsHandler(FSHandler):
for chunk in read_bz2_file(file_path):
update_hashes(chunk)
elif extension == ".chd" or file_type == "application/x-mame-chd":
chd_hash = extract_chd_hash(file_path)
if chd_hash:
sha1_h = cast(Any, CHDHashWrapper(chd_hash, name="sha1"))
rom_sha1_h = cast(Any, CHDHashWrapper(chd_hash, name="sha1"))
else:
# Not a valid v5 CHD, treat as basic file
# This ensures CRC32 and MD5 are still calculated for non-v5 CHDs
for chunk in read_basic_file(file_path):
update_hashes(chunk)
else:
for chunk in read_basic_file(file_path):
update_hashes(chunk)
backend/handler/filesystem/sync_handler.py
+11
View File
@@ -1,3 +1,4 @@
import functools
import hashlib
import os
from pathlib import Path
@@ -110,3 +111,13 @@ class FSSyncHandler(FSHandler):
f"Path {full_path} is outside the sync base directory"
) from e
path.unlink()
@functools.cache
def get_fs_sync_handler() -> FSSyncHandler:
"""Lazily instantiate the sync folder handler on first use.
Deferred so that startup doesn't fail when sync is unconfigured or its
base path is not writable.
"""
return FSSyncHandler()

Some files were not shown because too many files have changed in this diff Show More