## Summary
alpha.2 has been released. Changing `main` to alpha.2-post.
## Related Issues / Discussions
<!--WHEN APPLICABLE: List any related issues or discussions on github or
discord. If this PR closes an issue, please use the "Closes #1234"
format, so that the issue will be automatically closed when the PR
merges.-->
## QA Instructions
<!-- List actual commands/results and relevant reproduction steps. State
skipped, failing, or unavailable checks honestly. Include
screenshots/video for changed UI and before/after measurements for
material performance claims. -->
## Review
<!-- Summarize material findings resolved and remaining risks or
limitations. If there were no material findings, say so briefly. Omit
subagent counts, review tiers, and review-process narration. Do not
attach review transcripts or work logs. -->
## Compatibility / Rollout
<!-- When applicable: persisted-data migrations, API/generated-artifact
compatibility, deployment ordering, and recovery considerations. Omit
this section when unnecessary. -->
## Checklist
- [ ] _The PR has a short but descriptive title, suitable for a
changelog_
- [ ] _Meaningful regression coverage added / updated where needed;
obsolete tests/code removed_
- [ ] _Persisted-state and API changes include required migrations /
compatibility validation_
- [ ] _Relevant performance/efficiency opportunities considered;
material claims have evidence_
- [ ] _Material review findings resolved and relevant checks rerun_
- [ ] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
## Summary
Every board used to live in one flat list, with each project owning a
single board. This PR gives boards a home: a board belongs to exactly
one of its owner's projects, or to the **Library**, the tier every
project can see. Every project has an **Inbox**, where its results land
by default.
- **Backend.** `boards.project_id` records membership (`NULL` is the
Library). `POST /boards/?project_id=` creates in a project, `PATCH
/boards/{id}` with `project_id` moves a board (`null` for the Library),
and `DELETE /projects/{id}?boards=release|delete` either releases a
project's other boards to the Library (the default) or deletes them with
it. `BoardDTO.is_inbox` marks the inbox, which cannot be renamed, moved,
archived or deleted on its own.
- **Gallery.** The board panel groups boards by **the open project**
(Inbox pinned first), the **Library** (Uncategorized pinned first),
**Other projects** when that filter is on, then By Date and Archived. A
typed name no board has is offered as a create row in both tiers, and
each tier's "+" opens a create dialog started from whatever is typed.
Board menus, the selection bar and the image context menu offer **Move
to…** with the same grouping. Deleting a project asks what to do with
its other boards.
- **Transfer.** Duplicate, export and import carry every board of a
project. The board snapshot lists each board, inbox first, with its
archived state; `board.json` is version 2 with one entry per board, and
version 1 still reads as a lone inbox. Import and duplicate stage one
Library board per board, in source order and on the rollback ledger as
each is created, claim the inbox on the project create and move the
others in afterwards. A board that cannot be moved is reported and stays
in the Library with everything on it.
## QA Instructions
Frontend (`invokeai/frontend/webv2`, Node 22, pnpm 10):
```
pnpm lint # format, oxlint, tsc, architecture:check
pnpm test # unit
pnpm test:browser src/workbench/projects src/features/gallery src/workbench/image-actions \
src/workbench/launchpad/projects src/workbench/shell/topbar src/app
pnpm test:project-files # Playwright journey against the mock backend: export, import, duplicate
pnpm test:performance:architecture
pnpm test:accessibility
node --test scripts/mock-backend-fixtures.test.mjs
```
Backend (repo root):
```
uv tool run [email protected].2 check invokeai tests && uv tool run [email protected].2 format --check invokeai tests
uv run --no-sync python -m pytest tests/app/routers/test_projects.py tests/app/routers/test_boards_projects.py \
tests/app/routers/test_boards_multiuser.py tests/app/services/project_records tests/app/services/boards \
tests/app/services/shared/sqlite_migrator
```
All of the above pass on the branch head. The project-file journey
exports a project with an inbox and a member board, imports it and
duplicates it, and checks both copies end up with the inbox plus the
member board and its two items.
Manual, against the mock backend (`node scripts/mock-backend.mjs` and
`pnpm dev`): board panel sections and hints, create in each tier, Move
to… in both directions, both delete-project choices, the Other projects
section, and export/import/duplicate of a multi-board project. The
browser baseline was re-recorded for the locale growth.
Not covered: the journey does not exercise an archived or empty member
board against the mock backend; unit tests cover those.
## Review
Material findings resolved:
- Staging boards created before the rollback ledger existed were
orphaned when a later staging create failed. The ledger is now created
first and records each board as it is made, with regression tests on
both the import and duplicate paths.
- The duplicate path never rolled back on a failure before the create
attempt. It now mirrors the import path.
- The import toast did not say where an unplaced board went. It and the
user guide now say the board is in the Library.
- Exporting a board with a blank name produced a file the importer
refused. The parser keeps blank member names.
- A board.json from a newer app was reported as damaged. It is now
reported as unsupported.
- The Launchpad route statically pulled the gallery data layer through
the delete dialog, tripping the performance gate. Both hosts load the
dialog lazily.
- A failed board create from the new dialog would have closed it and
discarded the name. The create action now reports its outcome, and the
dialog keeps the name on failure.
- `board.json` bounded items but not boards, so a few kilobytes of empty
boards could make an import issue an unbounded number of creates and
moves. An archive now names at most 1000 boards, refused on read, on
export and on duplicate before any request.
- The lazily loaded delete-project dialog rendered nothing while its
module downloaded, leaving application shortcuts active with a deletion
pending. Both hosts now use the pending dialog stand-in, so the
confirmation is modal from the keypress that asked for it.
- Board names were measured in UTF-16 units by the archive reader and
the staging create, while the server counts characters, so a name with
emoji could be refused on import or cut through an emoji when staged.
Both now count code points and share one limit.
- The board snapshot query sorted every row before applying its limit,
so a project far over the item ceiling paid a full sort before the 413.
The query no longer orders in SQL; rows are sorted per board once they
are known to fit.
No remaining limitations beyond those in QA.
## Compatibility / Rollout
- Migration `migration_2026_10_08_add_board_project_membership` adds the
nullable, indexed `boards.project_id` column and claims each project's
existing board as its inbox. It is forward only.
- The OpenAPI contract and generated `frontend/api` artifacts are
updated: `BoardDTO.is_inbox`, `BoardChanges.project_id`, the
`project_id` query on board creation, the `boards` query on project
deletion, and the board snapshot's `boards` shape.
- `.invk` archives written by this branch carry `board.json` version 2.
Older archives (version 1 or no `board.json`) still import. Archives
from this branch are not readable by builds before it, which is the same
policy as every earlier format change here.
- Persisted gallery settings with the old `boards` section id are
dropped by the existing filter; no user action needed.
## Checklist
- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [x] _Meaningful regression coverage added / updated where needed;
obsolete tests/code removed_
- [x] _Persisted-state and API changes include required migrations /
compatibility validation_
- [x] _Relevant performance/efficiency opportunities considered;
material claims have evidence_
- [x] _Material review findings resolved and relevant checks rerun_
- [x] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
Move to… and board dragging are limited to the user's own private boards, and only the user's projects take drops.
The deleted-board check asks the server afresh whenever a project loads instead of trusting earlier answers.
## Summary
Fixes the open High and Critical npm Dependabot alerts (66 of 67) in the
four pnpm packages. The one left is braces, which has no patched
release.
| Package | Change | Alerts fixed |
|---|---|---|
| `invokeai/frontend/api` | lockfile only: fast-uri, js-yaml, minimatch,
brace-expansion | 12 / 12 |
| `invokeai/frontend/webv2` | lockfile only: source-map-js 1.2.2 | 1 / 1
|
| `invokeai/frontend/webv1` | vite ^8.3.4, nanoid ^5.1.16, jsondiffpatch
^0.7.6, react-router-dom ^7.18.4, react-use ^17.6.1 (moves its js-cookie
to 3.x), concurrently ^9.2.5; a scoped `rollup@>=4.0.0 <4.59.0` override
in `pnpm-workspace.yaml`; transitive refreshes (ws, socket.io-parser,
engine.io-client, shell-quote, postcss, lodash, flatted, browserslist,
picomatch, …) | 25 / 25 |
| `docs` | astro 6 → ^7.3.8, @astrojs/starlight 0.39 → ^0.42.6,
starlight-links-validator ^0.27, starlight-llms-txt ^0.12, sharp
^0.35.5; adds `@astrojs/markdown-remark`, which astro 7 no longer
installs but the docs' remark/rehype plugins need; tar and undici
refreshed under node-gyp | 28 / 29 |
Notes:
- rollup 4 was only an optional peer of Storybook and rollup plugins,
and vite 8 bundles with rolldown. With the override, pnpm no longer
installs rollup 4 at all. The production build, Storybook build and
`storybook dev` all still work.
- `pnpm audit --audit-level high` now reports only braces in each
package. That includes tar (Critical), undici and picomatch, which
Dependabot had not flagged yet.
## Related Issues / Discussions
- Dependabot alerts:
https://github.com/invoke-ai/InvokeAI/security/dependabot
- Python counterpart: #9723
## QA Instructions
All checks ran on Node 22 with pnpm 10.12.4, and every package installs
with `--frozen-lockfile`.
- **api:** `pnpm lint`, `pnpm test` pass. `pnpm typegen < openapi.json`
regenerates `schema.ts` byte-identical.
- **webv2:** `pnpm build` (lint + build) passes, and `pnpm test` passes
10,066 tests.
- **webv1:**
- `pnpm test:no-watch` passes 2,369 tests; `pnpm vite build` and `pnpm
build-storybook` pass; knip, eslint, prettier and dpdm are clean.
- `lint:tsc` reports 16 errors and
`src/features/nodes/types/common.test-d.ts` fails. Both are identical on
main; they come from drift between webv1's model types and the generated
schema, not from this change.
- The dev server proxies `/api`, `/openapi.json` and `/ws/socket.io`
(polling and websocket) to a live backend.
- socket.io behaves identically with the old and new clients against a
live python-socketio server.
- **docs:**
- `pnpm build`, `pnpm test`, `check-deploy-output` and `check-redirects`
pass.
- Compared with main, both the `custom` and `ghpages` builds produce the
same page set and redirects, and all 338,045 links are the same, in the
same order. That covers sidebar order, the base path and prev/next
links. The search index and `llms.txt` match too.
- The only visible text changes are Starlight's new "Menu" button label
and opening curly quotes in headings, which astro 6 rendered backwards.
- The links validator still fails the build on a broken link or anchor.
Not done: a click-through of webv1 or the docs site in a browser.
## Review
Review found three additional High/Critical advisories (tar, undici,
picomatch) that only `pnpm audit` reported; they are now fixed. It also
noted that pnpm 11 ignores overrides in `package.json`, so the rollup
override now lives in webv1's `pnpm-workspace.yaml`.
Remaining:
- **braces 3.0.3** (GHSA-vfj7-8cjw-p6xm) has no patched release. It is a
stack-exhaustion DoS on deeply nested glob patterns, reached only from
build tooling with patterns from our own config.
- `starlight-contextual-menu` 0.1.5 still declares peer `astro ^5`, as
it already did against astro 6. The menu's script and markdown routes
are present and unchanged in the build.
## Checklist
- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [ ] _Meaningful regression coverage added / updated where needed;
obsolete tests/code removed_
- [ ] _Persisted-state and API changes include required migrations /
compatibility validation_
- [ ] _Relevant performance/efficiency opportunities considered;
material claims have evidence_
- [x] _Material review findings resolved and relevant checks rerun_
- [ ] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
Projects loaded into the editor drop selections, auto-add and queued targets naming boards the server reports gone.
GET /boards/{id} answers 404 only for a missing board; placement reads a board back before reporting it unmoved.
Board lists refresh after a rollback, the project cover skips empty boards, and focus follows a moved board's row.
## Summary
Float PCM extraction let FFmpeg select the default audio stream while
`_probe_audio_sample_rate()` read the first stream. In a multi-track
clip, this labeled decoded samples with the wrong rate and changed their
interpreted duration.
Select the first audio stream for both float and int16 extraction,
matching the probe and upload ingest. This also addresses the review
suggestion so VideoConcat and Frame Extract Range choose the same
soundtrack as the float-based paths.
## QA Instructions
- Real two-track FFmpeg fixture: the first track is a 44,100 Hz, 440 Hz
tone; the second is a default 48,000 Hz, 220 Hz tone. Before the int16
follow-up, the new int16 case failed because it returned 48,000 Hz.
After the change, both formats return the first track's rate, duration,
and tone.
- Focused `tests/app/util/test_video_audio_float.py`: 6 passed with
Python 3.12.14 and the locked NumPy 1.26.4, imageio 2.37.4,
imageio-ffmpeg 0.6.0, and pytest 9.1.1 dependencies. Tests ran in
isolation with `--confcutdir=tests/app/util`; the full application suite
was not run locally.
- Coverage includes both formats on multi-track clips, no-audio clips
returning `None`, single-track int16 extraction, and float precision
preservation.
- Ruff 0.11.2 check and format check passed for the two changed files;
`git diff --check` passed.
## Review
No material findings. Stream selection adds no extra process or decoding
pass. The remote application checks need to pass on the updated head.
## Compatibility / Rollout
For multi-track inputs, int16 extraction now selects the first audio
stream instead of FFmpeg's automatically selected default stream.
Single-track inputs and no-audio handling are unchanged.
## Checklist
- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [x] _Meaningful regression coverage added / updated where needed;
obsolete tests/code removed_
- [ ] _Persisted-state and API changes include required migrations /
compatibility validation_ (not applicable)
- [x] _Relevant performance/efficiency opportunities considered;
material claims have evidence_
- [x] _Material review findings resolved and relevant checks rerun_
- [x] _Documentation added / updated (if applicable)_ (helper docstring)
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
(not a release)
> **Stacked on #9709, which is stacked on #9708.** Until those merge,
this PR also shows their commits. The change here is the last commit,
`feat(ernie-image): load GGUF transformers`.
## Summary
ERNIE-Image already loads Diffusers pipelines and Comfy-Org's
safetensors single files. GGUF transformers did not register. This PR
adds them.
- **Identification.** `Main_GGUF_ErnieImage_Config` identifies ERNIE
GGUFs by the same four-key fingerprint as the safetensors config.
unsloth's GGUFs (ERNIE-Image and ERNIE-Image Turbo, the most downloaded)
carry the diffusers key layout unchanged: all 409 tensors match
`ErnieImageTransformer2DModel`, checked programmatically against the
header. Their `general.architecture` names another model, though: `wan`
for unsloth, `flux` for vantagewithai. Identification therefore rests on
the keys alone, and the tests run the whole factory over both headers.
- **Loader.** `ErnieImageGGUFModel` builds the diffusers transformer and
loads the GGUF strictly.
- Quantized Linears stay packed.
- Everything the forward reads outside a Linear matmul is dequantized
once at load, as is everything that is not actually quantized. That
covers the RMSNorms, the F32 biases, the BF16 stem weights, and the 4-D
BF16 patch convolution, which `aten.convolution` cannot run packed.
- The helper doing this, `unpack_ggml_at_load`, moves from the Ideogram
4 loader (#9708) into `quantization/gguf/loaders.py`, since ERNIE is its
second caller.
- **Node and denoise.** The ERNIE loader node treats GGUF as a single
file. The denoise passes the GGUF dequantization transient from #9709.
webv2 needs no change, because its ERNIE component slots already key on
`format !== 'diffusers'`.
- **Starters.** unsloth ERNIE-Image and ERNIE-Image Turbo at Q4_K_M and
Q8_0, each pinned to a commit. Each installs the Ministral 3B encoder
and the FLUX.2 VAE.
Measured on an RTX 4090 with ERNIE-Image Turbo, 1024², 8 steps, warm (3
runs after a warm-up):
| | Time per image | GPU peak | Transformer resident |
|---|---|---|---|
| Diffusers pipeline | 11.5–11.7 s | 23.6 GB | 15.3 GB |
| GGUF Q4_K_M | 11.0–11.1 s | 17.8 GB | 4.8 GB |
The GGUF image is visually indistinguishable from the Diffusers one,
with the text rendered cleanly. The record received the Turbo defaults
(8 steps, CFG 1.0) from its name.
## Related Issues / Discussions
Closes#9499. Ideogram 4 GGUF is #9708; ERNIE-Image GGUF is this PR.
## QA Instructions
1. Install the starter **ERNIE-Image Turbo (GGUF, Q4_K_M)**. It also
pulls the Ministral 3B encoder and the FLUX.2 VAE. The transformer
installs as `gguf_quantized` with 8 steps and CFG 1.0.
2. Select it in Generate. The Components section asks for the Mistral
encoder and the VAE, as it does for the safetensors single file. Invoke
and check that the image is clean.
Checks run:
- `uv run pytest tests/app/invocations tests/backend/quantization
tests/backend/model_manager -n 4`: 4585 passed.
- New coverage:
- Factory identification over real tiny GGUFs with `wan` and `flux`
headers, including the Turbo/base defaults.
- A safetensors file is not claimed as GGUF.
- The forward matches a dense reference for Q8_0, Q5_1 and Q4_0, using
the published BF16/F32 layout including the 4-D conv. It fails with a
convolution dispatch error when the at-load unpacking is removed.
- The absolute load reservation.
- The denoise passes the transient.
- The loader node accepts GGUF as a single file.
- `ruff check` and `ruff format --check`. `openapi.json` and `schema.ts`
regenerated; API package lint passes.
## Review
Material findings, resolved:
- The test fixture had the BF16 layout backwards: it made `text_proj`
BF16 and the conv F32. The real files keep the 4-D conv in BF16 and
quantize `text_proj`. The fixture now follows the published headers, so
the 4-D BF16 unpack path is under test.
- Starter copy called Q4_K_M "the smallest build". The repos also hold
Q2_K/Q3_K. Sizes now use decimal GB consistently.
Out of scope:
- **GGUF Ministral 3B encoder.** It is still refused by the Mistral GGUF
config, which builds a `MistralModel` without Ministral's YaRN RoPE. The
encoder stays the 7.7 GB safetensors file.
- **Wan's own at-load unwrap.** `_unwrap_unquantized_to_compute_dtype`
covers F32/F16 only. Switching it to `unpack_ggml_at_load` is a
follow-up.
## Compatibility / Rollout
No migration; this adds a config class and a loader. The generated
`openapi.json` and `schema.ts` gain `Main_GGUF_ErnieImage_Config`.
Existing Diffusers and safetensors installs behave as before.
## Checklist
- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [x] _Meaningful regression coverage added / updated where needed;
obsolete tests/code removed_
- [x] _Persisted-state and API changes include required migrations /
compatibility validation_
- [x] _Relevant performance/efficiency opportunities considered;
material claims have evidence_
- [x] _Material review findings resolved and relevant checks rerun_
- [x] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
Board rows drag onto the open project, the Library or another project; project counts include the inbox.
Deleting a project clears open projects' references to its boards; imports refresh board lists once boards are placed.
An unanswered project create is settled by reading it back; restores pool uploads across boards and report placement.
## Summary
Fix compatibility scheduling for a nested If graph with a literal outer
condition. When the selected branch source is unresolved and has no loop
ancestor, infer the root frame so the outer If can activate its branch.
Preserve existing loop-scoped behavior and avoid fallback for completed
empty sources.
## Related Issues / Discussions
<!--WHEN APPLICABLE: List any related issues or discussions on github or
discord. If this PR closes an issue, please use the "Closes #1234"
format, so that the issue will be automatically closed when the PR
merges.-->
## QA Instructions
- `pytest -q tests/test_graph_execution_state.py -k
test_nested_if_literal_condition_activates_selected_branch` - 1 passed.
- `pytest -q tests/app/services/shared/test_graph_if_dependencies.py` -
35 passed.
- `pytest -q
tests/app/services/shared/test_execution_engine_differential.py -k
nested_if` - 101 passed.
- Ruff check, Ruff format check, and `git diff --check` passed.
## Review
Self-review found no remaining material issues.
## Compatibility / Rollout
<!-- When applicable: persisted-data migrations, API/generated-artifact
compatibility, deployment ordering, and recovery considerations. Omit
this section when unnecessary. -->
## Checklist
- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [x] _Meaningful regression coverage added / updated where needed;
obsolete tests/code removed_
- [ ] _Persisted-state and API changes include required migrations /
compatibility validation_
- [x] _Relevant performance/efficiency opportunities considered;
material claims have evidence_
- [x] _Material review findings resolved and relevant checks rerun_
- [ ] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
## Summary
Which file format loads for which model was spread over the Model
Families prose and the individual family pages, and went stale whenever
a format landed (Ideogram 4 and ERNIE-Image GGUF most recently). This
adds one **Model Format Support** page under Users Guide → Models with
four tables:
- **Main models.** Diffusers, bf16 single file, fp8 scaled, int8, nvfp4,
MXFP8, GGUF, SDNQ and NF4 for all 14 families.
- **Text encoders.** T5, CLIP, Qwen3, Qwen3-VL (4B/8B and the truncated
32B), Qwen3.5, Qwen2.5-VL, Mistral, Ministral, Gemma-2, Gemma-4 and
UMT5.
- **VAEs.** Single file and Diffusers, plus which family uses which VAE.
- **LoRAs and adapters per family.** LoRA formats, ControlNet,
IP-/T2I-Adapter, Redux, PiD.
Cells are ✓ (loads), ✗ (not supported), **R** (refused at install with a
message) or – (no such build). Model Families links to the page from its
quantized-formats section. The new-model integration checklist asks to
update it.
The tables reflect `main` as of #9708/#9709. ERNIE-Image GGUF is still ✗
here; #9725 flips that cell when it merges.
## Related Issues / Discussions
Follow-up to #9708, #9709 and #9725.
## QA Instructions
- `pnpm -C docs build`: completes.
- Every internal link on the new page resolves to a built page (16
checked against `docs/dist`), and the `#quantized-formats` anchor
exists.
- The cells come from the loader registrations, the config classes and
their explicit refusals in `invokeai/backend/model_manager/`.
## Review
Docs only. No material findings.
## Checklist
- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [ ] _Meaningful regression coverage added / updated where needed;
obsolete tests/code removed_
- [ ] _Persisted-state and API changes include required migrations /
compatibility validation_
- [ ] _Relevant performance/efficiency opportunities considered;
material claims have evidence_
- [x] _Material review findings resolved and relevant checks rerun_
- [x] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
Only a Diffusers folder is expected to carry scheduler/, so single files of any format no longer warn.
The GGUF forward test compared all-zero outputs (diffusers zero-inits adaLN and the final layer); it now fills
those in and also covers Q4_K, the starters' type, on a 256-wide geometry.
## Summary
Add a simple boolean logic node that performs boolean AND, OR, XOR, NOT,
NAND, NOR, or XNOR operations on inputs A and B. NOT uses only A.
## Related Issues / Discussions
<!--WHEN APPLICABLE: List any related issues or discussions on github or
discord. If this PR closes an issue, please use the "Closes #1234"
format, so that the issue will be automatically closed when the PR
merges.-->
## QA Instructions
`uv run --no-sync pytest tests/app/invocations/test_boolean_logic.py -q`
passes. This node has no effect on other components.
## Review
No findings.
## Compatibility / Rollout
<!-- When applicable: persisted-data migrations, API/generated-artifact
compatibility, deployment ordering, and recovery considerations. Omit
this section when unnecessary. -->
## Checklist
- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [x] _Meaningful regression coverage added / updated where needed;
obsolete tests/code removed_
- [ ] _Persisted-state and API changes include required migrations /
compatibility validation_
- [ ] _Relevant performance/efficiency opportunities considered;
material claims have evidence_
- [x] _Material review findings resolved and relevant checks rerun_
- [ ] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
## Summary
A model load could read its record and then wait for the model-load
lock. Meanwhile a load-affecting edit (e.g. `cpu_only`, `fp8_storage`)
could commit and invalidate the cache. The load then built and admitted
the pre-edit model, and later requests reused its stale weights.
**What a cold load does now:**
- On a cache miss, the load checks the caller's config against the
stored record.
- If an edit has superseded it, the load raises `StaleModelConfigError`
instead of building. Nothing is reserved or admitted.
- `ModelLoadService.load_model` then waits for the edit to finish,
re-reads the record and retries once. Every caller gets this:
invocations, the prompt utility routes, the image index and convert.
- A load whose files are missing because the model was moved and its
record updated to the new path is retried the same way. A missing file
under an unchanged record still raises.
- `model_load_complete` is sent however the load ends, so UIs never keep
showing a load that has finished or failed.
**No database access under `MODEL_LOAD_LOCK`:** a long DB transaction
(e.g. admin VACUUM) never stalls unrelated model builds or VRAM moves.
- Record writers that can change how a model loads (`PATCH /i/{key}` and
reidentify) wrap their write and any cache invalidation in
`ModelLoadService.record_edit(key)`. They report on the yielded
`RecordEdit` whether the write changed anything that loads.
- Before taking the lock, a cold load waits until no edit of its model
is in progress, then reads the record.
- Under the lock, it only consults in-memory edit state. It is rejected
if an edit of that model is still in progress, or if a load-affecting
edit ended since its read. Edits that change nothing that loads, such as
a rename, don't reject it.
**Other details:**
- The load-affecting predicate (#9669's config fingerprint) lives in
`services/model_load/model_load_common.py`, so cache invalidation and
the load check share one definition. The load check resolves `path`
against the models directory, because the loader rewrites `config.path`
to absolute in place.
- `ModelLoader.config_check` is an optional instance attribute that
`ModelLoadService` sets. It is not a constructor argument, so registered
loaders that override `__init__` are unaffected.
- Model loaders must subclass `ModelLoader`, as the docs already
required, because that is where the check runs.
`ModelLoaderRegistry.register` rejects any other class with a
`TypeError`, and `ModelLoadService` refuses one supplied by a custom
registry.
- Loads that start after an edit has finished, and edits that don't
affect loading, behave as before.
## Related Issues / Discussions
Closes#9674. Follow-up to #9669. Remaining pre-existing race tracked in
#9711.
## QA Instructions
- `uv run --no-sync pytest
tests/app/services/model_load/test_stale_model_load_fence.py` passes 14
tests. A fixture records, for every record read made by a loading
thread, whether that thread held `MODEL_LOAD_LOCK`; the overlap tests
assert none did. The tests cover:
- a load whose record was superseded before it reached the lock loads
the updated record, built once, with one started/complete event pair;
- a load queued on the lock while an edit commits and invalidates (the
#9674 ordering) loads the updated record;
- an edit that starts while the load is queued and invalidates after the
load holds the lock: the load is not built from the old record, whether
the edit commits before the rejection or while the rejected load waits;
- a rename landing while a load is queued doesn't reject it (built once,
no retry);
- a load overlapping an edit that is stuck behind a DB transaction waits
without holding `MODEL_LOAD_LOCK`, so another thread can take it, and
then loads the edit;
- a DB transaction starting while a cold load is queued doesn't leave
the load holding `MODEL_LOAD_LOCK`;
- a second supersession during the retry raises after exactly two checks
and still sends `model_load_complete`;
- a model moved with its record loads from its new path, while missing
files under an unchanged record still fail;
- a config object reused after eviction, whose path the loader made
absolute, still counts as current.
- per-model edit tracking is dropped once no edit or load needs it, but
kept while a queued load still watches it (a rename ending must not hide
a load-affecting edit that follows).
- a loader built directly on `ModelLoaderBase` is refused before it can
cache anything, with the started/complete pair still sent.
-
`tests/backend/model_manager/load/test_model_loader_registry_entries.py`
checks that `register` rejects a function and a direct `ModelLoaderBase`
subclass without changing the registry.
- `tests/app/routers/test_model_manager.py` and
`test_update_model_record_cache_invalidation.py` check that `PATCH
/i/{key}` commits and drops inside `record_edit`, that reidentify
replaces inside it, and that the edit is flagged load-affecting exactly
for fields in the fingerprint.
- Mutation checks. Each of these makes at least one test fail:
- removing the retry, or retrying through a second started/complete
pair;
- re-reading the record under the lock;
- not waiting for edits before the check, or before the retry's read;
- counting every edit as load-affecting;
- ignoring in-progress edits at the final check;
- routes never clearing the flag, or the flag defaulting to not
load-affecting;
- removing either route's bracket;
- no terminal event on failure;
- no moved-file retry;
- comparing paths without the models directory.
- removing the registry's class check, or the service's.
- `OMP_NUM_THREADS=4 uv run --no-sync pytest -n 4
tests/app/services/model_load tests/backend/model_manager
tests/app/routers tests/app/services/model_records
tests/app/services/model_install` gave 3424 passed and 113 skipped, plus
one failure in
`test_model_install.py::test_restore_metadata_timeout_preserves_partial_download_for_retry`.
That test fails intermittently on its own as well and exercises code
this PR doesn't touch.
- `ruff check` and `ruff format --check` are clean.
- I did not do a manual reproduction against a running server.
## Review
Findings resolved during review:
- The check is an attribute rather than a constructor argument, so it
doesn't break third-party loaders.
- The retry moved from `context.models.load()` into the load service.
Direct callers such as `/expand-prompt` and `/image-to-prompt` no longer
return 500, and the loading indicator no longer sticks after a retry.
- Nothing touches the database under `MODEL_LOAD_LOCK`, including when a
load overlaps an edit of the same model. Reidentify writes are bracketed
like edits.
- Rename-only edits no longer reject overlapping loads.
- A load whose model was moved with its record retries instead of
failing, and failed loads send `model_load_complete`.
- Per-model edit tracking no longer grows with every model ever edited.
Remaining limitations, which predate this change:
- An edit commits its record before taking the lock. A load that passes
its final check just before that commit is admitted, and then dropped by
the invalidation. If that entry is held, the drop only marks it stale,
and `ModelCache.get` keeps serving it to same-cache requests until the
hold is released. Tracked in #9711.
- A cold load of a model waits while that same model is being edited.
That is usually brief, but if the edit is itself waiting on a long DB
transaction, the load waits as long. Loads of other models are
unaffected.
## Compatibility / Rollout
Any writer of load-affecting model-record fields must use `record_edit`
and report `load_affecting` (see the `ModelLoadServiceBase.record_edit`
docstring). `model_load_complete` now also marks a failed load's end;
both web UIs only use it to clear the loading indicator, and webv2 also
refreshes its model list.
## Checklist
- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [x] _Meaningful regression coverage added / updated where needed;
obsolete tests/code removed_
- [ ] _Persisted-state and API changes include required migrations /
compatibility validation_
- [x] _Relevant performance/efficiency opportunities considered;
material claims have evidence_
- [x] _Material review findings resolved and relevant checks rerun_
- [x] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
`test_workflow_last_run_at.py::test_update_last_run_at_sets_timestamp`
fails intermittently, mostly on Windows CI.
- The fixture stores the 43 bundled workflows, then the test creates one
workflow and expects it in the first 10 results of a listing sorted
newest first by `created_at`.
- `created_at` has millisecond resolution. On Windows' ~15 ms clock the
new workflow usually gets the same timestamp as the bundled ones.
- Ties sort by `workflow_id` descending. A random UUID sorts above the
bundled `default_...` ids only about 1 time in 7 or 8, so the new
workflow usually falls off the page and the test's `next(...)` raises
`StopIteration`.
The listing now includes only user workflows, which is what the check is
about. Nothing else changes.
## Related Issues / Discussions
Seen as the `windows-cpu` failure on #9694. The test came in with #9702.
## QA Instructions
- Forced tie (a throwaway test setting every row's `created_at` to the
same value), 20 runs each: the old query lost the workflow in 19, the
new one in none.
- `uv run --no-sync pytest tests/app/services/workflow_records`: 137
passed.
- `ruff check` and `ruff format --check` are clean.
## Review
No material findings. This is a one-line test change.
## Checklist
- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [x] _Meaningful regression coverage added / updated where needed;
obsolete tests/code removed_
- [ ] _Persisted-state and API changes include required migrations /
compatibility validation_
- [ ] _Relevant performance/efficiency opportunities considered;
material claims have evidence_
- [x] _Material review findings resolved and relevant checks rerun_
- [ ] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
The listing check now pages user workflows only. On a coarse clock the new workflow
ties on created_at with the 43 bundled workflows, and the workflow_id tie-break
usually put it past the first 10.
Only ModelLoader checks a cold load against edits of its record before caching it.
The registry now rejects any other class at registration, and the load service
refuses one supplied by a custom registry.
Mark formats that install but fail on load as L, keep R for install-time refusals, and mark unrecognized files ✗.
Gemma-4 installs only as a folder with its config and tokenizer; quantized VAEs are not refused everywhere.
blacksmith-sh[bot]