diff --git a/.hermes/plans/2026-10-25_164406-remote-godot-library.md b/.hermes/plans/2026-10-25_164406-remote-godot-library.md new file mode 100644 index 0000000..5329c19 --- /dev/null +++ b/.hermes/plans/2026-10-25_164406-remote-godot-library.md @@ -0,0 +1,174 @@ +# PLAN: Self-hosted remote game-asset library with Godot on-demand import (v0.3.0) + +## Goal + +Turn Asset-Manager into a self-hosted central asset library that a Godot editor +client browses remotely, downloading only chosen assets into the local project — +so devs import what they need without storing the whole library on device. + +## Current context / assumptions + +- Repo: `Heretek-AI/Asset-Manager`, branch `main`, tag `v0.2.0`, tree clean. +- Layout (all paths relative to repo root): + - `assetmgr/` — Python host: `actions.py` (manifest discovery), `host.py` + (dispatch), `store.py` (`*.approj` markers, `.meta.json` sidecars), + `scan.py` (project scan), `godot.py` (ConfigFile parser, `.import` + read/`set_import_params`, `IMPORTER_PARAMS`, `validate_import_params`), + `cli.py` (every command has `--json`, single-document stdout). + - `anchorpoint/` + `apsync/` — automation API + local engine + (`generate_thumbnail` shells to ffmpeg/ImageMagick). + - `native/` — Go binary `assetmgrd` (stdlib only): shells out to + `python3 -m assetmgr.cli … --json`, serves embedded `native/web/`, JSON API + (`GET /api/info|actions|validate|status|scan`, `POST /api/project-init|run|thumbnail`). + - `Dockerfile` (Go build → `trixie-slim` + python3 + git + headless Godot 4.3 + + ffmpeg), `docker-compose.yml`, `actions/godot/*`, `tests/`. +- Runtimes available in this dev container: system `python3` (project venv at + `/home/hermes/.hermes/cache/scratch/assetmgr-test-venv`), `go 1.24.4`, + `godot` 4.3 at `/tmp/godot-bin/godot` (host) and `/usr/local/bin/godot` + (image), `ffmpeg` 7.1.5, Docker engine 26.1.5. +- Anchorpoint (reference, anchorpoint.app): Git-based VC for game teams, + **selective checkout** ("download only what you need"), asset pipeline + *before* the engine, per-DCC integrations. We copy the *selective* idea, + not the vendor: our library is a dumb content store + metadata, NOT git. +- Out of scope: vendor cloud, OAuth secrets, PySide6 shell, multiplayer + locking (existing `ServerRequiredError` boundary stays). + +## Architecture / proposed approach + +Add a **library server** (new `library/` Python package: content-addressed +blob store + SQLite metadata + token auth) next to the existing local host; +the Go binary gains a reverse-proxy route plus a **Godot editor plugin** +(new `godot-plugin/addons/asset_library/`) that browses the remote catalog +from inside the editor and downloads chosen assets into `res://`, then +triggers a reimport. Artists keep terabytes server-side; devs pull kilobytes. + +## Step-by-step tasks + +Conventions for every task: test venv python is +`/home/hermes/.hermes/cache/scratch/assetmgr-test-venv/bin/python`. +TDD each task: write failing test → run → implement → run → `git commit`. +Never commit red. Paths are repo-relative. + +### Phase A — library server core + +**A1. Blob store module (`library/blobs.py`).** +Write `tests/test_blobs.py` first: +```python +from library.blobs import put, get_path +def test_roundtrip(tmp_path): + digest = put(tmp_path / "store", b"abc") + assert digest == "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad" + assert get_path(tmp_path / "store", digest).read_bytes() == b"abc" +``` +Run (expect FAIL, module missing): +`assetmgr-test-venv/bin/python -m pytest tests/test_blobs.py -q` +Implement `library/blobs.py`: `put(root, data)->sha256 hex` writing to +`root/ab/cd/rest`, `get_path(root, digest)->Path` (raise +`FileNotFoundError` if absent). Re-run: expect `1 passed`. +Commit: `git commit -qm "library: content-addressed blob store"`. + +**A2. Metadata DB (`library/db.py`).** +Test first (`tests/test_library_db.py`): open sqlite in tmp, `add_asset()` +returns id, `get_asset()` round-trips +`{name, blob_sha256, size_bytes, kind, godot_importer, params_json, tags}`. +Run: expect FAIL. Implement with `sqlite3` stdlib, schema: +```sql +CREATE TABLE IF NOT EXISTS assets( + id INTEGER PRIMARY KEY, name TEXT NOT NULL, blob_sha256 TEXT NOT NULL, + size_bytes INTEGER NOT NULL, kind TEXT NOT NULL DEFAULT 'raw', + godot_importer TEXT, params_json TEXT NOT NULL DEFAULT '{}', + tags_json TEXT NOT NULL DEFAULT '[]', + created_at TEXT NOT NULL DEFAULT (datetime('now'))); +``` +Re-run: expect pass. Commit. + +**A3. HTTP API (`library/server.py`, stdlib `http.server`, no new deps).** +Endpoints (JSON; `Authorization: Bearer ` on all but `/health`): +`GET /health` → `{"ok":true}`; `GET /api/v1/assets?q=&tag=&kind=` → +`{"assets":[...]}`; `POST /api/v1/assets` (multipart: file + fields +name/kind/importer/params/tags) → `{"id":…}`; `GET /api/v1/assets//blob` +→ bytes. Token from env `ASSETMGR_LIBRARY_TOKEN`, compared with +`hmac.compare_digest`. Test with `urllib` in `tests/test_library_server.py` +spinning the server on port 0 in a thread. Run full suite: +`assetmgr-test-venv/bin/python -m pytest -q` → all pass. Commit. + +**A4. CLI thin wrappers (`assetmgr library-*`).** +Add `library-init `, `library-push --server URL --token T +[--name N --kind K --importer I --params JSON --tags a,b]`, +`library-pull --server URL --token T --out PATH` in `assetmgr/cli.py` +following the existing `_cmd_*` + `--json` pattern. Tests in +`tests/test_cli_json.py` style against a live test server (reuse A3 helper). +Full suite green. Commit. + +### Phase B — publish from existing projects + +**B1. `godot::publish-to-library` action** (`actions/godot/publish/` + +`publish.py`, mirror `import_asset` structure): resolves project, uploads +selection via `library-push` equivalent (subprocess the CLI), toasts the new +asset id. Test dispatch with a stub server (A3 helper) asserting the blob +arrives byte-identical. Commit. + +### Phase C — Godot editor plugin (the remote-import UX) + +**C1. Plugin skeleton** (`godot-plugin/addons/asset_library/plugin.cfg`, +`plugin.gd` with `_enter_tree` adding a dock, `_exit_tree` removing it). +Verify headless: `godot --headless --path /tmp/gproj --quit` exits 0 with +the addon present (no script errors in stderr). Commit. + +**C2. Dock panel** (`panel.tscn` + `panel.gd`): server URL + token fields +(persisted to `user://asset_library.cfg` via `ConfigFile`), search box, +`ItemList` of results from `GET /api/v1/assets`. Test: GDScript is +headless-testable only by running — assert via `--headless --script` probe +that search URL construction is correct (print + grep). Commit. + +**C3. Download + import** (`download.gd`): `HTTPRequest` GET blob → +`FileAccess` write to `res://` → `EditorFileSystem.scan()` (or +`--import` fallback) → toast with importer badge. Live test: real Godot 4.3 +opens the demo project, downloads `hero.png` from the test server, asserts +`.godot/imported/hero.png-*.ctex` appears. Commit. + +### Phase D — self-hosting + +**D1. Compose service**: extend `docker-compose.yml` with `library:` reusing +the `assetmgr` image, `CMD ["assetmgr-library", ...]` (or `python -m +library.server`), named volume `library-data`, token via env file +(`.env.example` with `ASSETMGR_LIBRARY_TOKEN=changeme`). Verify: +`docker compose up -d --build library && curl -s localhost:8081/health` +→ `{"ok":true}`, then `docker compose down`. Commit. + +**D2. Docs + README**: `docs/REMOTE_LIBRARY.md` (setup, token, plugin +install, selective-download workflow) + README section with the same two +screenshots pattern as v0.2.0. Commit. + +### Phase E — release + +**E1.** Bump `pyproject.toml` + `assetmgr/__init__.py` to `0.3.0`, full +suite + `go test ./...` green, commit, push, `gh release create v0.3.0` +with notes + rebuilt `assetmgrd` binary. Verify via +`gh release view v0.3.0 --json tagName,assets`. + +## Tests / validation + +- Per-task TDD as above; no task is done without its test green. +- Gates before push: `pytest -q` (expect 55+ new-count passing, skips only + environmental), `gofmt -l .` empty, `go vet ./...` clean, + `go test -count=1 ./...` ok. +- Live proofs required (not optional): C3 Godot download→import test, D1 + compose health check, E1 release asset presence. + +## Risks, tradeoffs, and open questions + +- **Godot plugin GDScript has no pytest**: live headless runs are the test + harness; keep plugin logic thin, push decisions (validation, paths) into + tested Python. +- **Binary assets in git**: screenshots only; never commit `.godot/`, + `*.ctex`, or test fixtures >100KB (generate at test time, as today). +- **Auth is bearer-token minimal**: fine for self-hosted single-team use; + multi-user ACLs are explicitly out of scope. +- **Large uploads**: stdlib server buffers to disk via `tempfile`; set a + documented 2 GiB cap (`MAX_UPLOAD_BYTES`) rather than building chunked + upload now (YAGNI). +- **Open question for the user**: one shared library per team, or + per-project libraries? Plan assumes one server, many projects (token per + team). If per-project isolation is needed, add a `project` column — ask + before Phase A if unsure. diff --git a/library/blobs.py b/library/blobs.py new file mode 100644 index 0000000..d455bfa --- /dev/null +++ b/library/blobs.py @@ -0,0 +1,30 @@ +"""Content-addressed blob store (SHA-256, `ab/cd/rest` fan-out).""" + +from __future__ import annotations + +import hashlib +from pathlib import Path + +__all__ = ["put", "get_path"] + + +def _shard(root: Path, digest: str) -> Path: + return root / digest[:2] / digest[2:4] / digest[4:] + + +def put(root: str | Path, data: bytes) -> str: + """Store *data*, returning its SHA-256 hex digest (idempotent).""" + digest = hashlib.sha256(data).hexdigest() + target = _shard(Path(root), digest) + if not target.exists(): + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(data) + return digest + + +def get_path(root: str | Path, digest: str) -> Path: + """Filesystem path of a stored blob; raises FileNotFoundError if absent.""" + target = _shard(Path(root), digest) + if not target.is_file(): + raise FileNotFoundError(f"blob {digest} not in store") + return target diff --git a/library/db.py b/library/db.py new file mode 100644 index 0000000..8e9694e --- /dev/null +++ b/library/db.py @@ -0,0 +1,93 @@ +"""SQLite metadata for library assets (stdlib only).""" + +from __future__ import annotations + +import json +import sqlite3 +from pathlib import Path +from typing import Any + +__all__ = ["open_db", "add_asset", "get_asset", "list_assets"] + +SCHEMA = """\ +CREATE TABLE IF NOT EXISTS assets( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL, + blob_sha256 TEXT NOT NULL, + size_bytes INTEGER NOT NULL, + kind TEXT NOT NULL DEFAULT 'raw', + godot_importer TEXT, + params_json TEXT NOT NULL DEFAULT '{}', + tags_json TEXT NOT NULL DEFAULT '[]', + created_at TEXT NOT NULL DEFAULT (datetime('now')) +); +CREATE INDEX IF NOT EXISTS idx_assets_kind ON assets(kind); +""" + + +def open_db(path: str | Path) -> sqlite3.Connection: + """Open (creating) the library database at *path*.""" + conn = sqlite3.connect(str(path)) + conn.row_factory = sqlite3.Row + conn.executescript(SCHEMA) + return conn + + +def _row_to_dict(row: sqlite3.Row) -> dict[str, Any]: + return { + "id": row["id"], + "name": row["name"], + "blob_sha256": row["blob_sha256"], + "size_bytes": row["size_bytes"], + "kind": row["kind"], + "godot_importer": row["godot_importer"], + "params": json.loads(row["params_json"]), + "tags": json.loads(row["tags_json"]), + "created_at": row["created_at"], + } + + +def add_asset(conn: sqlite3.Connection, *, name: str, blob_sha256: str, + size_bytes: int, kind: str = "raw", + godot_importer: str | None = None, + params: dict | None = None, + tags: list | None = None) -> int: + """Insert one asset row; returns its id.""" + cur = conn.execute( + "INSERT INTO assets(name, blob_sha256, size_bytes, kind," + " godot_importer, params_json, tags_json)" + " VALUES(?,?,?,?,?,?,)", + (name, blob_sha256, size_bytes, kind, godot_importer, + json.dumps(params or {}), json.dumps(tags or []))) + conn.commit() + return int(cur.lastrowid) + + +def get_asset(conn: sqlite3.Connection, asset_id: int) -> dict[str, Any]: + """Fetch one asset by id; raises KeyError when absent.""" + row = conn.execute("SELECT * FROM assets WHERE id=?", + (asset_id,)).fetchone() + if row is None: + raise KeyError(f"asset {asset_id} not found") + return _row_to_dict(row) + + +def list_assets(conn: sqlite3.Connection, *, q: str = "", + tag: str = "", kind: str = "") -> list[dict[str, Any]]: + """List assets, newest first, with optional filters.""" + where: list[str] = [] + args: list = [] + if q: + where.append("name LIKE ?") + args.append(f"%{q}%") + if kind: + where.append("kind = ?") + args.append(kind) + sql = "SELECT * FROM assets" + if where: + sql += " WHERE " + " AND ".join(where) + sql += " ORDER BY id DESC" + rows = [_row_to_dict(r) for r in conn.execute(sql, args)] + if tag: + rows = [r for r in rows if tag in r["tags"]] + return rows diff --git a/tests/test_blobs.py b/tests/test_blobs.py new file mode 100644 index 0000000..bcfead2 --- /dev/null +++ b/tests/test_blobs.py @@ -0,0 +1,23 @@ +"""Content-addressed blob store tests.""" + +from library.blobs import get_path, put + +SHA_ABCD = "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad" + + +def test_roundtrip(tmp_path): + digest = put(tmp_path / "store", b"abc") + assert digest == SHA_ABCD + assert get_path(tmp_path / "store", digest).read_bytes() == b"abc" + + +def test_missing_blob_raises(tmp_path): + import pytest + with pytest.raises(FileNotFoundError): + get_path(tmp_path / "store", "0" * 64) + + +def test_dedupe_same_digest(tmp_path): + root = tmp_path / "store" + assert put(root, b"abc") == put(root, b"abc") + assert len(list((root).rglob("*"))) == 3 # ab/cd/rest chain minus dirs diff --git a/tests/test_library_db.py b/tests/test_library_db.py new file mode 100644 index 0000000..a9735da --- /dev/null +++ b/tests/test_library_db.py @@ -0,0 +1,30 @@ +"""Library metadata DB tests.""" + +from library.db import add_asset, get_asset, open_db + + +def test_add_get_roundtrip(tmp_path): + db = open_db(tmp_path / "library.db") + asset_id = add_asset(db, name="hero.png", blob_sha256="ab" * 32, + size_bytes=70, kind="texture", + godot_importer="texture", + params={"compress/mode": 0}, tags=["hero"]) + row = get_asset(db, asset_id) + assert row["name"] == "hero.png" + assert row["blob_sha256"] == "ab" * 32 + assert row["godot_importer"] == "texture" + assert row["params"] == {"compress/mode": 0} + assert row["tags"] == ["hero"] + + +def test_list_filter(tmp_path): + from library.db import list_assets + db = open_db(tmp_path / "library.db") + add_asset(db, name="hero.png", blob_sha256="aa" * 32, size_bytes=1, + tags=["game"]) + add_asset(db, name="click.wav", blob_sha256="bb" * 32, size_bytes=2, + kind="audio", tags=["ui"]) + assert len(list_assets(db)) == 2 + assert [a["name"] for a in list_assets(db, tag="game")] == ["hero.png"] + assert [a["name"] for a in list_assets(db, kind="audio")] == ["click.wav"] + assert list_assets(db, q="her") and not list_assets(db, q="zzz")