feat: Video generation (#9163)

* feat(model): add Wan 2.2 image generation support (Phases 0-2)

Foundation + TI2V-5B MVP + A14B dual-expert MoE for Wan 2.2 image
generation. Wan was trained on video but is competitive with leading
open-source image models when run at num_frames=1; this commit wires
that path into InvokeAI.

Phase 0 — Foundation:
- BaseModelType.Wan + WanVariantType {T2V_A14B, TI2V_5B}
- SubModelType.Transformer2 for the dual-expert MoE
- MainModelDefaultSettings per variant
- step_callback Wan branch (16-channel preview; 48-channel TI2V-5B
  falls back to slicing first 16 channels until proper factors land)
- Frontend enums + node colour

Phase 1 — TI2V-5B Diffusers MVP:
- Main_Diffusers_Wan_Config probe (variant from transformer_2/ +
  vae/config.json::z_dim, with filename heuristic fallback)
- WanDiffusersModel loader (subclasses GenericDiffusersLoader)
- WanT5EncoderField, WanTransformerField (with dual-expert slots),
  WanConditioningField, WanConditioningInfo
- New invocations: wan_model_loader, wan_text_encoder, wan_denoise,
  wan_image_to_latents, wan_latents_to_image
- FlowMatchEulerDiscreteScheduler integration with on-disk config load
- RectifiedFlowInpaintExtension reused for inpaint
- 5D <-> 4D shape juggling: latents stay 4D in InvokeAI's pipeline,
  re-add T=1 only inside the transformer call / VAE encode-decode

Phase 2 — A14B dual-expert MoE:
- Probe reads boundary_ratio from model_index.json
- Loader emits both transformer (high-noise) and transformer_low_noise
  (low-noise expert at transformer_2/) for A14B
- _ExpertSwapper in wan_denoise drives GPU residency between experts:
  high-noise for t >= boundary_ratio * num_train_timesteps, low-noise
  below. Only one expert locked at a time so the cache can evict the
  other - relies on existing CachedModelWithPartialLoad to handle
  oversized models on lower-VRAM GPUs.
- guidance_scale_low_noise field for separate low-noise CFG override

Tests:
- 24 passing tests covering probe variant detection, default settings,
  noise sampling, end-to-end denoise on a synthetic transformer (CPU),
  dual-expert boundary swap, CFG branch
- 1 heavy-test placeholder gated by INVOKEAI_HEAVY_TESTS=1 for the
  real-weights smoke test

Phase 3+ deferred: standalone VAE/encoder configs, GGUF, LoRA,
ControlNet, ref image, inpaint UI, frontend wiring, starter models.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(model): Wan 2.2 Phase 3 + tokenizer-load fix

Phase 3 adds standalone VAE and UMT5-XXL encoder configs so users can run
GGUF-quantized Wan transformers (Phase 4) without installing the full
~30 GB Diffusers pipeline.

VAE configs:
- VAE_Checkpoint_Wan_Config + VAE_Diffusers_Wan_Config (16-channel A14B
  vs 48-channel TI2V-5B, distinguished by decoder.conv_in z_dim).
- 16-channel files share the AutoencoderKLWan architecture with Qwen
  Image; disambiguated via filename heuristic ("wan" in name -> Wan,
  otherwise -> Qwen Image). Mirror exclusion in QwenImage's probe.
- VAELoader gets a Wan branch that builds AutoencoderKLWan(z_dim=...)
  via init_empty_weights, mirroring the QwenImage single-file pattern.
- Existing standard VAE probe excludes both QwenImage- and Wan-style
  state dicts.

UMT5-XXL encoder:
- New ModelType.WanT5Encoder + ModelFormat.WanT5Encoder.
- WanT5Encoder_WanT5Encoder_Config probes the diffusers folder layout
  (text_encoder/config.json with model_type=umt5, or flat layout with
  config.json at root). Refuses full Wan pipelines.
- WanT5EncoderLoader handles both layouts and loads UMT5EncoderModel +
  AutoTokenizer.

Component-source plumbing:
- WanModelLoaderInvocation now exposes wan_t5_encoder_model and
  component_source pickers (mirrors QwenImage pattern). Resolution
  order: standalone > main (if Diffusers) > component_source. Required
  when the main model is a single-file format in Phase 4.

Bug fix in wan_text_encoder:
- Tokenizer was loading via AutoTokenizer.from_pretrained(<root>)
  directly, which fails for nested layouts where files live in
  <root>/tokenizer/. Now routed through the model cache so the
  registered loaders handle layout differences correctly.

Frontend:
- New type guards (isWanVAEModelConfig, isWanT5EncoderModelConfig,
  isWanMainModelConfig, isWanDiffusersMainModelConfig) and hooks/
  selectors (useWanVAEModels, useWanT5EncoderModels,
  useWanDiffusersModels). New zSubModelType / zModelType / zModelFormat
  enum entries for transformer_2 and wan_t5_encoder.

Tests:
- 16 new tests covering z_dim detection, VAE checkpoint/diffusers
  probes, the bidirectional Qwen-vs-Wan filename deferral, and the
  UMT5 encoder probe (nested + flat + T5 + full-pipeline rejection).
- Total Wan test count: 41 passing, 1 heavy-test placeholder skipped.
- Full config test suite (63 tests) still passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): unbreak frontend lint after Wan additions

Five issues turned up running `make frontend-lint`:

1. wan_denoise.py used `from __future__ import annotations`, which made
   the `invoke()` return annotation a string ('LatentsOutput'). The
   InvocationRegistry's `get_output_annotation()` returns the raw
   annotation, so OpenAPI generation crashed with
   `'str' object has no attribute '__name__'`. Removed the future-import
   and added `Any` to the typing imports.

2. ModelRecordChanges.variant didn't list WanVariantType, so the
   generated schema's install/update endpoints rejected `t2v_a14b` and
   `ti2v_5b`. Added it.

3. Regenerated frontend/web/src/services/api/schema.ts from the live
   backend so it now includes BaseModelType.wan, ModelType.wan_t5_encoder,
   SubModelType.transformer_2, ModelFormat.wan_t5_encoder, the Wan
   variants, all Wan invocation types and their conditioning/transformer
   field types.

4. modelManagerV2/models.ts: added `wan_t5_encoder` to the category map,
   `wan` to the base color/long-name/short-name maps, the two Wan
   variants to the variant-name map, and `wan_t5_encoder` to the
   format-name map.

5. ModelManagerPanel/ModelFormatBadge.tsx: added `wan_t5_encoder` to
   FORMAT_NAME_MAP and FORMAT_COLOR_MAP.

`make frontend-lint` now passes cleanly (tsc, dpdm, eslint, prettier).
All 41 Wan Python tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

chore(wan): drop unused FE exports flagged by knip

These were forward-compatibility wiring for Phase 9 (the FE graph
builder) that has no consumers yet; knip rightly flagged them. Removed
or de-exported. They'll come back when the graph builder lands and
needs them.

- common.ts: zWanVariantType drops `export` (still used internally by
  zAnyModelVariant).
- types.ts: drop isWanMainModelConfig, isWanDiffusersMainModelConfig,
  isWanVAEModelConfig (no callers). The remaining
  isWanT5EncoderModelConfig is used by models.ts. WanT5EncoderModelConfig
  type drops `export` (still used as the type guard's narrowing target).
- modelsByType.ts: drop the six unused useWan*/selectWan* hooks +
  selectors and their type-guard imports.

`make frontend-lint` (tsc + dpdm + eslint + prettier + knip) now green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

docs(wan): use *-Diffusers HF repo names in plan

The Wan-AI org publishes two flavours of each release:
  * Wan-AI/Wan2.2-{TI2V-5B,T2V-A14B,I2V-A14B}            ← upstream native
  * Wan-AI/Wan2.2-{TI2V-5B,T2V-A14B,I2V-A14B}-Diffusers  ← convertible

The native release has _class_name=WanModel in config.json and ships
weights flat at the repo root with no transformer/, vae/, text_encoder/
subdirs. It is not loadable by Diffusers' WanPipeline.from_pretrained.

Update plan doc to reference the -Diffusers repos throughout (probe
notes, starter-model entries) so the plumbing path matches what the
Diffusers loader actually expects.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): accept 0 as 'unset' sentinel for guidance_scale_low_noise

The frontend renders Optional[float] inputs with default 0 in the
numeric input rather than passing null/unset. Combined with ge=1.0,
this caused every wan_denoise invocation to fail Pydantic validation
with "Input should be greater than or equal to 1" until the user
manually entered a value (or knew to leave the field disconnected).

The validation error was rejected before invocation logging, so it
never showed up in the server log either - making the failure hard to
diagnose.

Relaxing the constraint to ge=0.0 and treating values below 1.0 as the
"fall back to primary Guidance Scale" sentinel. The user's natural FE
default (0) now works as expected.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): correct preview dimensions and colors for TI2V-5B

Two bugs in the Wan branch of the diffusion step callback:

1. Wrong dimensions. The reported preview size hardcoded `* 8` for the
   spatial downscale ratio, but TI2V-5B's Wan2.2-VAE uses 16x. A
   1024x1024 target was being announced to the FE as 512x512.

2. Wrong colors. The previous fallback for 48-channel TI2V-5B latents
   sliced the first 16 channels and applied the standard 16-channel
   Wan-VAE projection. Those channel layouts are unrelated, so the
   projection produced meaningless colors.

Adding the proper Wan2.2-VAE 48-channel RGB projection matrix (and
bias) from ComfyUI's Wan22 latent format, and selecting the right
matrix + spatial-scale by latent channel count: 16 → A14B (Wan VAE,
8x), 48 → TI2V-5B (Wan2.2-VAE, 16x).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): honor model's _class_name when building scheduler

TI2V-5B's scheduler_config.json declares _class_name=UniPCMultistepScheduler
with flow_shift=5.0. The previous code hardcoded
FlowMatchEulerDiscreteScheduler.from_pretrained(...), which silently
constructed a default-config FlowMatch instead of the UniPC the model
expects. The mismatched noise schedule manifests as soft / under-denoised
faces and global graininess in the final images.

Now: read scheduler_config.json, look up the named class on the diffusers
module, and instantiate that class via from_pretrained. UniPC and
FlowMatch share the same step()/set_timesteps()/sigmas/num_train_timesteps
interfaces, so the denoise loop works transparently for either.

A14B continues to use FlowMatchEulerDiscreteScheduler when its scheduler
config says so (its reference is FlowMatchEuler with shift=8.0). Falls
back to FlowMatchEulerDiscreteScheduler defaults when no on-disk config
is available.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): match diffusers WanPipeline tokenizer length and latent dtype

Two divergences from the Diffusers reference that were hurting image
quality (soft / grainy / distorted faces at default settings):

1. Tokenizer max_sequence_length was 226 in wan_text_encoder, but the
   model was trained with 512-token sequences. The upstream native
   config.json has text_len: 512, and Diffusers' WanPipeline.__call__
   default is 512 (overriding _get_t5_prompt_embeds's stale 226 default).
   Wan's cross-attention sees padded zeros past the prompt's actual
   length but expects to be looking at a 512-position context window.

2. Latents were stored in bf16 throughout the denoise loop. Diffusers'
   WanPipeline.prepare_latents explicitly uses dtype=torch.float32 and
   only casts to the transformer's dtype right at the forward call:
       latent_model_input = latents.to(transformer_dtype)
   Storing in bf16 between steps accumulates ~40 steps of bf16
   quantization on the scheduler's small per-step deltas. Now
   latent_dtype = torch.float32 throughout, with a per-step cast for
   the transformer forward pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

chore(wan): add diffusers reference comparison script

scripts/wan_diffusers_reference.py runs a Diffusers-format Wan 2.2
checkpoint directly via WanPipeline.from_pretrained, with the same
arguments InvokeAI's wan_denoise uses. Use to A/B against InvokeAI
output when image quality is questionable.

Defaults to enable_model_cpu_offload so the script fits on 16 GB cards
where the full pipeline (transformer + UMT5-XXL + VAE) would otherwise
OOM. --offload {model,sequential,none} controls the strategy.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(model): Wan 2.2 Phase 4 - GGUF transformer support

Adds single-file GGUF support for Wan 2.2 transformers, the path that
makes A14B usable on consumer GPUs (~7 GB/expert at Q4_K_M instead of
~28 GB at bf16).

Probe (configs/main.py):
- New helpers: _has_wan_keys (Wan vs Qwen/FLUX/Z-Image fingerprint via
  condition_embedder.text_embedder.linear_1 + patch_embedding);
  _detect_wan_gguf_variant (16ch -> A14B, 48ch -> TI2V-5B from
  patch_embedding.weight.shape[1]); _detect_wan_gguf_expert (filename
  heuristic for high_noise / low_noise / none).
- Main_GGUF_Wan_Config(base=Wan, format=GGUFQuantized, variant, expert).
  Tolerates the ComfyUI 'model.diffusion_model.' / 'diffusion_model.'
  prefixes via _has_wan_keys' multi-prefix scan.
- Registered in factory.py.

Loader (model_loaders/wan.py):
- WanGGUFCheckpointModel mirrors the QwenImage GGUF pattern:
  gguf_sd_loader -> strip ComfyUI prefix -> auto-detect arch from state
  dict shapes (num_layers, inner_dim, ffn_dim, text_dim, in_channels,
  num_heads = inner_dim/128) -> init_empty_weights +
  load_state_dict(strict=False, assign=True).

Loader invocation (wan_model_loader.py):
- New 'Transformer (Low Noise)' picker: optional second GGUF for the
  A14B dual-expert MoE. Auto-swaps if the user wired the experts in
  the wrong order. Warns when an A14B GGUF is loaded without a paired
  low-noise expert (single-expert run, degraded quality).
- GGUF mains require either a standalone VAE+encoder or a Diffusers
  Component Source (which can also supply boundary_ratio).
- Diffusers main path unchanged (still pulls both experts from
  transformer/ + transformer_2/).

Tests (tests/.../test_wan_gguf_config.py):
- 14 tests across key fingerprint, variant detection, expert filename
  heuristic, and the full probe (A14B high/low, TI2V-5B, GGUF rejection,
  unrecognised state-dict rejection, explicit override).

Total Wan tests: 55 passing (no regressions). FE lint clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): support QuantStack-style GGUFs and standalone Diffusers VAE

The city96 Wan 2.2 GGUF repos have been removed from Hugging Face,
leaving QuantStack as the surviving distributor. QuantStack ships the
native upstream Wan key layout (text_embedding.0/2, self_attn/cross_attn,
ffn.0/2, head.head, head.modulation, ...) rather than the diffusers
naming city96 used; biases are stored as F16 rather than BF16; and the
standalone Wan VAE installs as a flat AutoencoderKLWan folder which the
generic loader rejects. Three fixes:

1. Probe now recognises both diffusers and native key layouts via a new
   _is_native_wan_layout helper; _has_wan_keys accepts either text-proj
   fingerprint.

2. GGUF loader converts native -> diffusers keys (mirroring diffusers'
   convert_wan_transformer_to_diffusers) and unwraps non-quantized
   GGMLTensors to plain tensors at compute_dtype. The unwrap is needed
   because conv3d isn't in GGMLTensor's dispatch table, so the F16
   patch_embedding bias would otherwise hit conv3d against bf16 latents.

3. VAELoader gains a VAE_Diffusers_Wan_Config branch that loads
   AutoencoderKLWan directly; the generic path can't handle a flat
   single-class folder when a submodel_type is provided.

Adds 12 tests covering the native layout (probe + converter + unwrap).
Verified end-to-end against Wan2.2-T2V-A14B-Q4_K_M from QuantStack:
1095 tensors round-trip key-for-key against WanTransformer3DModel.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(model): Wan 2.2 Phase 5 - LoRA support

Probe + config (LoRA_LyCORIS_Wan_Config):
  - Detects Wan LoRAs in three layouts: diffusers PEFT, native upstream PEFT
    (ComfyUI), and Kohya (both naming variants).
  - Anti-pattern guards prevent collisions with Anima (Cosmos DiT q_proj
    convention), QwenImage (transformer_blocks), Flux (double/single blocks),
    and Z-Image (diffusion_model.layers).
  - Optional ``expert: "high" | "low" | None`` field; auto-detected from
    filename (high_noise / low_noise / hyphenated / concatenated variants).

Key conversion (wan_lora_conversion_utils):
  - Native upstream keys (self_attn/cross_attn, ffn.0/2) -> diffusers
    (attn1/attn2, ffn.net.0.proj / ffn.net.2).
  - Strips ``transformer.``, ``diffusion_model.``, ``base_model.model.transformer.``
    prefixes from PEFT-style keys.
  - Kohya layer names mapped through an explicit longest-match table.
  - Output paths use diffusers naming so the LayerPatcher can resolve them
    against WanTransformer3DModel parameter paths.

Loader integration:
  - Adds BaseModelType.Wan branch to LoRALoader._load_model.

Invocation nodes (wan_lora_loader.py):
  - WanLoRALoaderInvocation: single LoRA with auto/both/high/low target field.
  - WanLoRACollectionLoader: list of LoRAs, auto-routed by each LoRA's
    recorded expert tag.
  - Output WanLoRALoaderOutput carries the WanTransformerField with updated
    ``loras`` / ``loras_low_noise`` lists.

Denoise integration:
  - _ExpertSwapper now manages both the model_on_device context and the
    LayerPatcher.apply_smart_model_patches context per expert. LoRA patches
    are entered after device load and exited before device release, with
    fresh iterators per swap.
  - GGUF (quantized) experts request sidecar patching so GGMLTensor weights
    aren't touched directly.
  - Low-noise expert falls back to the primary loras list when
    ``loras_low_noise`` is empty (matches WanTransformerField semantics).

Tests: 81 new tests covering probe accept/reject across formats, anti-pattern
guards on competing architectures, converter round-trips for all three
layouts, invocation target resolution + routing + duplicate guards, and the
_ExpertSwapper lifecycle (lora context opens/closes in the right order
around the device swap, quantized flag forwards, no-LoRA path skips the
patch context, re-entering the same label is a no-op).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): probe Wan LoRA before Anima in the config union

Native-PEFT Wan LoRAs (lightx2v's Lightning, most ComfyUI-trained Wan
LoRAs) carry keys like ``diffusion_model.blocks.X.cross_attn.k.lora_A.weight``.
Anima's probe matches on the bare ``cross_attn``/``self_attn`` substring —
it does not require the Anima-specific ``_proj`` suffix nor any of the
``mlp``/``adaln_modulation`` Cosmos DiT markers — so these Wan LoRAs were
classified as ``BaseModelType.Anima`` because Anima happened to run first.

Reorder the LyCORIS section of ``AnyModelConfig`` so Wan probes first.
Wan's probe is strictly more restrictive (it rejects Anima's ``_proj``
attention suffix via the anti-pattern guard added in the previous commit),
so Anima LoRAs are still correctly classified after this reorder.

Existing users with mis-tagged installs need to delete the affected LoRA
records and reinstall.

Adds two regression tests: a union-ordering assertion, and a sanity check
that demonstrates Anima's probe *would* match Wan native keys if asked
directly — pinning the constraint that motivates the ordering.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

chore(i18n): add Wan2.2 T5 Encoder model-manager label

The frontend source already references ``modelManager.wanT5Encoder``;
the locale key was added with a casing typo (``want5Encoder``). Fix
the key so the Wan T5 Encoder model type renders its display name
correctly in the model manager UI.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(model): Wan 2.2 Phase 7 - reference-image (I2V) conditioning

Re-implementation after the first attempt — which used CLIP-vision
conditioning — was reverted. Wan 2.2 I2V-A14B does NOT use a CLIP-vision
encoder (the Diffusers repo ships ``image_encoder: [null, null]`` in
``model_index.json``); instead it conditions on a reference image by
VAE-encoding it and concatenating the resulting latents (plus a
first-frame mask) to the noise latents along the channel dim. The I2V
transformer therefore has ``in_channels=36`` (16 noise + 16 ref-image
latents + 4 mask) vs ``in_channels=16`` for T2V.

Taxonomy:
  - Re-adds ``WanVariantType.I2V_A14B``.

Probes:
  - Diffusers: ``_detect_wan_variant`` reads ``transformer/config.json::in_channels``;
    36 → I2V_A14B, 16 → T2V_A14B (both share the dual-expert layout).
  - GGUF: ``_detect_wan_gguf_variant`` recognises ``in_channels=36`` from the
    patch_embedding tensor shape and emits I2V_A14B.

Backend extension (``backend/wan/extensions/wan_ref_image_extension.py``):
  - ``preprocess_reference_image`` resizes + normalises to a 5D pixel tensor.
  - ``encode_reference_image_to_condition`` VAE-encodes the image and stacks
    a 4-channel first-frame mask on top, producing the
    ``[1, 20, 1, H/8, W/8]`` condition tensor the denoise loop consumes.
  - Mirrors diffusers ``WanImageToVideoPipeline.prepare_latents`` with
    ``num_frames=1`` and ``expand_timesteps=False``.

Invocation node (``wan_ref_image_encoder.py``):
  - "Reference Image - Wan 2.2": image + VAE + width/height pickers.
  - Output ``WanRefImageConditioningField`` carries the condition tensor
    name plus the dimensions used (so the denoise step can validate dim
    parity).

Denoise integration:
  - ``WanDenoiseInvocation`` gains an optional ``ref_image`` field.
  - Variant gate: rejects ref_image on T2V_A14B and TI2V-5B with a clear
    error before doing any work.
  - Dimension gate: rejects ref-image width/height mismatch vs denoise.
  - At every transformer call, concatenates the 20-channel condition
    tensor to the 16-channel noise latents along the channel dim before
    passing to the transformer (giving the 36-channel input I2V expects).

Tests: 14 new across the probe, the extension, and the denoise loop.
The synthetic ``_ZeroTransformer`` test stand-in now mirrors the real
I2V transformer's ``in_channels=36, out_channels=16`` asymmetry by
slicing its zero output back to 16 channels when the input is 36-wide.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): derive GGUF out_channels from proj_out shape (I2V support)

The GGUF loader was setting ``out_channels = in_channels`` which is wrong for
Wan 2.2 I2V-A14B: that variant has ``in_channels=36`` (16 noise + 16 ref-image
latents + 4 first-frame mask, concatenated by the denoise loop) but
``out_channels=16`` since the transformer only predicts the noise component
back. Loading an I2V GGUF would build a transformer with the wrong proj_out
shape and crash:

  RuntimeError: Error(s) in loading state_dict for WanTransformer3DModel:
    size mismatch for proj_out.weight: copying a param with shape
    torch.Size([64, 5120]) from checkpoint, the shape in current model is
    torch.Size([144, 5120]).

(144 = 36 * 4, 64 = 16 * 4 — patch_size=(1, 2, 2) → prod=4)

Read out_channels directly from the ``proj_out.weight`` shape in the state
dict. This is correct for all three Wan 2.2 variants without needing to know
the variant in advance.

Also tighten the num_layers fallback: T2V_A14B and I2V_A14B share 40 layers;
only TI2V-5B has 30. The fallback is rarely hit in practice (the per-block
count comes from the state dict scan), but the previous code would have
defaulted I2V_A14B to 30 layers.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(model): make Anima LoRA probe mutually exclusive with Wan

InvokeAI's ``Config_Base.CONFIG_CLASSES`` is a Python ``set``, so iteration
order during model probing is non-deterministic across process restarts.
First-match-wins ordering in ``AnyModelConfig`` is documentation only — it
has no effect on which config is iterated first.

Anima's previous probe accepted any state dict containing the substring
``cross_attn`` or ``self_attn``, which collides with Wan's native LoRA key
layout (``diffusion_model.blocks.X.cross_attn.q.lora_down.weight``). Both
probes accepted Wan native LoRAs (including lightx2v's Lightning T2V and I2V
distillations), and the ``matches.sort_key`` tiebreaker only disambiguates
by ModelType, not within LoRA configs. So which config "won" depended on
dict hash order — sometimes Wan, sometimes Anima.

The previous mitigation reordered the AnyModelConfig union to put Wan
before Anima. That worked by luck and was inherently fragile.

Tighten Anima's probe to require Cosmos-DiT-exclusive subcomponents:
``mlp``, ``adaln_modulation``, or ``_proj``-suffixed attention names
(``q_proj``/``k_proj``/``v_proj``/``output_proj``) — none of which appear
in any Wan LoRA. Wan native uses bare ``.q``/``.k``/``.v``/``.o`` on
``self_attn``/``cross_attn``, and ``ffn.N``/``ffn.net.N`` instead of ``mlp``.

The new strict detectors live alongside the original loose ones so the
Anima conversion utility (which runs after probing) still works.

Regression tests in ``test_wan_lora_probe_independence.py`` cover:
- I2V Lightning V1 (the bug-triggering LoRA), T2V Lightning V2, Wan Kohya
  and Wan diffusers PEFT layouts — Wan probe accepts, Anima probe rejects.
- Anima PEFT and Kohya layouts — Anima accepts, Wan rejects.
- A meta-test that runs every LoRA config in CONFIG_CLASSES against the
  Lightning state dicts and asserts exactly one accepts — this catches
  ANY future probe collision, not just Wan vs Anima.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): defer expert model loading in _ExpertSwapper to avoid cache thrash

The swapper used to take pre-loaded ``LoadedModel`` handles at construction:

    high_info = context.models.load(self.transformer.transformer)
    low_info  = context.models.load(self.transformer.transformer_low_noise)
    swapper = _ExpertSwapper(high_info=high_info, low_info=low_info, ...)

With dual ~9 GB A14B GGUF experts plus the ~10 GB UMT5-XXL encoder competing
for the same RAM cache, the LRU policy frequently dropped one expert by the
time the denoise loop swapped into it. The model manager then emitted

    [MODEL CACHE] Locking model cache entry ... but it has already been
    dropped from the RAM cache. This is a sign that the model loading
    order is non-optimal in the invocation code (See ... #7513).

and reloaded the weights from disk (~1.2s extra per swap).

Refactor the swapper to take the ``ModelIdentifierField`` plus the
``InvocationContext`` and call ``context.models.load(model_id)`` lazily
inside ``get()``. Each swap obtains a fresh handle, the LRU window is
small, and the warning goes away.

Config metadata (used to compute ``is_quantized``) is read upfront via
``context.models.get_config()`` — that's metadata, not weights, so it
doesn't put pressure on the cache.

Tests: existing swapper lifecycle tests refactored to use a fake context
whose ``models.load`` is logged. A new ``test_lazy_load_per_swap_not_upfront``
pins the regression — it asserts ``models.load`` is NOT called at swapper
construction, only at first get() per expert.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test(wan): add Phase 8 inpaint regression tests

The denoise_mask wiring + RectifiedFlowInpaintExtension integration in
wan_denoise.py was put in place during Phase 2/3 alongside the rest of
the denoise loop. Phase 8 of the plan was about ensuring this path
worked and is locked in by tests.

Three new tests under TestWanDenoiseInpaint:

1. test_preserved_region_matches_init_exactly: builds a half/half mask
   (left = preserve, right = regenerate in user-side convention), runs
   full denoise with the synthetic zero-output transformer, and asserts
   the preserved half of the final latents equals the init exactly while
   the regenerated half does not. Pins the mask-inversion + per-step
   merge behavior.

2. test_inpaint_requires_init_latents: a mask without init latents must
   raise a clear ValueError — the merge has nothing to weld back to.

3. test_no_mask_path_is_unchanged: regression that adding the inpaint
   extension didn't perturb the non-inpaint codepath (with init latents
   + denoising_start=0.5 but no mask, the loop just runs img2img).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

feat(frontend): add I2V_A14B to Wan variant zod enum + manager label

Phase 7 added the I2V_A14B backend variant. The frontend's zod enum
(features/nodes/types/common.ts:zWanVariantType) and the model manager's
variant-label map (features/modelManagerV2/models.ts) were still on the
two-variant list, so:

  - ModelIdentifierField inputs with ui_model_variant filters on Wan
    couldn't list I2V models.
  - The model manager UI showed a raw 'i2v_a14b' string instead of the
    human label.

Phase 9 (full linear-view wiring — type guards, hooks, params slice,
graph builder, tab UI) is in progress on a follow-up commit; this lands
the two small enum fixes first so the I2V probe / install paths work
correctly end-to-end with the existing FE.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): Phase 9 piece #1 - linear-view T2V txt2img graph builder

Adds the minimum frontend wiring needed to generate Wan 2.2 images from
the linear view:

  - buildWanGraph.ts (new): text-to-image graph (model_loader →
    text_encoder × 2 → denoise → l2i). Diffusers main model only —
    transformer, VAE and UMT5 encoder all resolve from the same repo, so
    no Wan-specific params slice fields are required yet. CFG-skip
    branch when guidance_scale ≤ 1.0.
  - useEnqueueGenerate / useEnqueueCanvas dispatchers: route
    base === 'wan' to buildWanGraph.
  - graph/types.ts: add wan_l2i / wan_i2l / wan_denoise / wan_model_loader
    to the relevant node-type unions.
  - addTextToImage / addImageToImage: include wan_denoise / wan_l2i so
    width/height are wired correctly and the txt2img helper accepts the
    Wan l2i node.
  - isMainModelWithoutUnet: include wan_model_loader (Wan has no UNet,
    same as the other modern bases).
  - metadata.py: add wan_txt2img / wan_img2img / wan_inpaint to the
    generation_mode enum (img2img / inpaint pieces land next).
  - schema.ts: regenerated to pick up the metadata enum + new
    Wan invocations.

Pieces left in Phase 9: params slice (standalone VAE / T5 / GGUF
low-noise / LoRA / ref-image fields + selectors), img2img + I2V + inpaint
branches in the graph builder, and Wan-specific UI components.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

feat(wan): Phase 9 piece #2 - GGUF support and CFG-Low control in linear view

Adds the three Wan-specific params + UI controls that gate GGUF workflows
plus a separate low-noise CFG slider for A14B users.

Params slice:
  - wanTransformerLowNoise (the second-expert GGUF for A14B)
  - wanComponentSource (Diffusers Wan model providing VAE + UMT5-XXL
    when the main is a GGUF)
  - wanGuidanceScaleLowNoise (optional separate CFG for the low-noise
    expert; null = fall back to the primary CFG)

Plus a `selectIsWan` selector for accordion gating.

UI components:
  - ParamWanModelSelects.tsx (Advanced accordion): two model pickers —
    Transformer (Low Noise) filtered to Wan GGUF mains, and VAE/Encoder
    Source filtered to Wan Diffusers mains. Mirrors the
    ParamQwenImageComponentSourceSelect structure.
  - ParamWanGuidanceScaleLowNoise.tsx (Generation accordion): slider +
    number input with an "auto" indicator when cleared. Default 3.5
    matches the diffusers reference 4.0 / 3.0 split.

Wiring:
  - Generation accordion: ParamWanGuidanceScaleLowNoise shown when base
    is wan, scheduler excluded for wan (same pattern as Anima/Qwen).
  - Advanced accordion: ParamWanModelSelects shown when base is wan, and
    Wan excluded from the SD-family VAE/CFG-rescale blocks.
  - buildWanGraph.ts: forwards the three new params to the model loader
    and denoise nodes (transformer_low_noise_model, component_source,
    guidance_scale_low_noise) and adds them to the graph metadata.

Hooks/types:
  - useWanDiffusersModels + useWanGGUFModels in modelsByType.ts.
  - isWanDiffusersMainModelConfig + isWanGGUFMainModelConfig type guards.
  - Three new locale strings (wanComponentSource, wanTransformerLowNoise,
    wanGuidanceScaleLowNoise[Auto]).

GGUF workflow now works end-to-end in the linear view: pick a Wan GGUF
main, set Transformer (Low Noise) to the paired second-expert GGUF, set
VAE/Encoder Source to any Diffusers Wan repo (TI2V-5B is convenient at
~12 GB) — generate produces an image.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): UX polish on the Wan linear-view controls

Bundles four small fixes applied during a usability review of the Wan
linear-view section (piece #2):

1. **Filter Main vs Transformer (Low Noise) dropdowns by expert tag.**
   The Wan GGUF probe records each file's ``expert`` field
   (``"high"`` / ``"low"`` / ``"none"``) via filename heuristic.
   - ``MainModelPicker``: hides ``expert === 'low'`` Wan GGUFs so users
     can't accidentally wire a low-noise expert as the primary main.
   - Transformer (Low Noise) picker (``useWanGGUFLowNoiseModels``):
     shows ``expert === 'low'`` Wan GGUFs only.

   Diffusers Wan mains and TI2V-5B aren't affected — they don't carry
   the ``expert`` field on their config schema. The backend's auto-swap
   safety net stays in place.

2. **Match the primary CFG slider's range.** The Wan low-noise CFG
   slider was constrained to 1–10 while the primary CFG ranges 1–20.
   With the diffusers reference 4/3 split, the low-noise slider thumb
   sat noticeably further right than the primary — visually misleading.
   Both sliders now share the 1–20 range with marks at [1, 10, 20].

3. **Label fits the form column.** "CFG (Low Noise)" → "CFG (Low)" so
   the slider fits cleanly next to its label instead of overlapping.

4. **Indicator state for the low-noise CFG slider.** Replaced the inline
   "(auto)" / "(same as cfg)" text — which kept overlapping the slider
   regardless of how short the label got — with an X-only reset button
   that's only visible when the user has set an explicit value. Absence
   of the X conveys auto/fallback state without any text overhang.

5. **Friendlier Transformer (Low Noise) placeholder.** "Second-expert
   GGUF for A14B (pair with the high-noise main)" → "Add for full
   detail" — concise nudge for users who haven't paired the second
   expert yet.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

feat(wan): Phase 9 piece #3 - linear-view img2img branch

Adds Wan 2.2 image-to-image to the linear view, mirroring the Qwen Image
pattern. The mode switches on the canvas state — pure-prompt runs go
through addTextToImage as before; canvas runs with an init image go
through addImageToImage which wires a fresh wan_i2l (Image to Latents -
Wan 2.2) node between the init image and the denoise's `latents` input,
honoring the existing denoise_start slider.

buildWanGraph:
  - Drops the txt2img-only guard, branches on generationMode.
  - img2img: spins up a wan_i2l node and hands it to addImageToImage
    alongside the existing denoise / l2i / modelLoader (as vaeSource).
  - inpaint / outpaint still fail loudly — pieces #4-#6.

graphBuilderUtils.getDenoisingStartAndEnd:
  - Adds 'wan' to the simple-linear case (denoising_start = 1 -
    denoisingStrength). Note: Wan's flow-matching schedule is "sticky"
    on the init compared to SDXL — users will likely need denoisingStrength
    ≥ 0.7 to see substantial change, matching the user-found 0.15-0.3
    denoising_start sweet spot from earlier img2img testing. We may
    revisit this with an exponent rescale (like FLUX uses) if the
    response curve feels off.

addImageToImage:
  - Adds 'wan_i2l' to the i2l-node-type union so the Wan i2l can be
    threaded through the shared helper.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): add wan_denoise to addImageToImage/addInpaint/addOutpaint type checks

Three sibling graph-helper utilities had the same modern-base list as
addTextToImage did, and the buildWanGraph img2img branch tripped one of
them at canvas-Generate time:

    error  [generation]: Failed to build graph
    {name: 'Error', message: 'Wrong assertion encountered'}

The else-branch in each helper assumes 'denoise_latents' (the SD1.5/SDXL
legacy path) and asserts that — failing for any modern base not listed
above the branch. addTextToImage was already updated in Phase 9 piece #1;
this catches the parallel cases that the img2img/inpaint/outpaint flows
go through.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

feat(wan): Phase 9 piece #4 - linear-view inpaint and outpaint branches

Wires Wan 2.2 inpaint and outpaint through the existing addInpaint /
addOutpaint helpers. The backend's RectifiedFlowInpaintExtension was
plumbed into wan_denoise.py back in Phase 8 (commit ab54617173); this
just connects the FE.

buildWanGraph:
  - generationMode === 'inpaint' → spin up a wan_i2l, call addInpaint
    with denoise + l2i + modelLoader (used as both vaeSource and
    modelLoader since the Wan model loader carries the VAE).
  - generationMode === 'outpaint' → parallel branch with addOutpaint.

addInpaint:
  - i2l-node-type union now includes 'wan_i2l' (the addImageToImage and
    addOutpaint type unions already do — different union shapes).

metadata.py:
  - generation_mode literal adds "wan_outpaint" alongside the existing
    wan_txt2img / wan_img2img / wan_inpaint entries.

isMainModelWithoutUnet already includes wan_model_loader (Phase 9 piece
create_gradient_mask when Wan is the main.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

feat(wan): Phase 9 piece #5 - linear-view I2V branch (raster as reference image)

Wan 2.2 I2V-A14B models condition on a reference image whose VAE-encoded
latents are concatenated to the noise along the channel dim each step
(in_channels=36 on the I2V transformer). In the linear view this maps
cleanly onto the existing canvas raster layer: pick an I2V model, drag
an image to raster, generate.

buildWanGraph:
  - Fetch the modelConfig early so the variant gate (i2v_a14b vs the
    rest) can drive the branch shape instead of being a post-hoc check.
  - I2V + txt2img: fail loudly ("Switch to the canvas tab and drag an
    image to the raster layer"). I2V models won't produce useful output
    without a reference, and the backend would crash trying to
    concatenate a missing condition tensor.
  - I2V + img2img: pull the raster image via the canvas compositor,
    wire it through a wan_ref_image_encoder (which VAE-encodes it and
    builds the 4-mask + 16-latent condition tensor backend-side), then
    feed the result into denoise.ref_image. Denoise runs from fresh
    noise (denoising_start=0, no init_latents) — the ref image is
    cross-attention/concat conditioning, not a noise-trajectory anchor.
  - I2V + inpaint/outpaint: fail clearly. Combining ref-image
    conditioning with a denoise mask is conceptually possible but the
    backend interaction hasn't been validated end-to-end.

metadata.py:
  - Adds "wan_i2v" to the generation_mode literal so the metadata field
    on I2V renders correctly.

T2V flows (txt2img / img2img / inpaint / outpaint) are unchanged for
non-I2V Wan variants (T2V-A14B and TI2V-5B).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): enforce multiple-of-16 dimensions to match transformer patch grid

Wan 2.2's transformer has ``patch_size=(1, 2, 2)``: it patch-embeds with
stride 2 then un-patches by 2. Combined with the VAE's 8x spatial scale,
canvas H/W must be a multiple of ``8 * 2 = 16`` — not just 8 — for the
patch round-trip to land exactly. Otherwise the latents and noise
prediction disagree by one in the spatial dim and the scheduler step
fails:

    RuntimeError: The size of tensor a (147) must match the size of
    tensor b (146) at non-singleton dimension 3

(here latent_w=147 → patch_w=73 → un-patched_w=146 ≠ 147)

This was silent for T2V at 1024x1024 (already a multiple of 16) but
fired for I2V at non-multiple-of-16 canvas sizes.

Fixes:

- ``optimalDimension.getGridSize``: Wan moves from the default 8 case to
  the multiple-of-16 case (alongside flux / sd-3 / qwen-image / z-image
  which have the same patch arithmetic). The canvas bbox UI now snaps
  Wan dimensions to multiples of 16.

- ``wan_denoise.py`` and ``wan_ref_image_encoder.py``: bump width/height
  ``multiple_of`` from 8 to 16. Defense-in-depth — workflow-editor
  users won't be able to send a non-16-aligned dim either.

Existing backend tests (23 passing) still hold — 1024 is divisible by 16
so the test fixtures didn't exercise the off-by-one path.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): show negative prompt box in Wan linear-view

Wan was missing from SUPPORTS_NEGATIVE_PROMPT_BASE_MODELS, so the
linear-view negative-prompt input was hidden even though the Wan denoise
node already wires negative conditioning when CFG > 1
(buildWanGraph.ts:67-75). Adds 'wan' to the list.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

feat(wan): Phase 9 piece #6 - Wan LoRA collection in linear view

Adds Wan LoRA wiring to buildWanGraph, mirroring the Qwen Image pattern.
The shared LoRASelect / LoRAList UI in the linear view already filters
LoRAs by the selected main model's base, so Wan LoRAs surface
automatically when a Wan main is picked — no UI changes needed.

addWanLoRAs (new):
  - Filters state.loras.loras to enabled Wan LoRAs.
  - For each LoRA: spawns a ``lora_selector`` node and threads it
    through a single ``collect`` collector.
  - Routes the collector into a ``wan_lora_collection_loader`` which
    sits between modelLoader and denoise — modelLoader.transformer →
    loader, then loader.transformer → denoise (rerouting the original
    modelLoader → denoise edge).
  - Emits per-LoRA metadata so PNG metadata + workflow restore work.

The dual-expert routing (high-noise vs low-noise vs untagged) is
handled entirely on the backend by ``WanLoRACollectionLoader`` based on
each LoRA's recorded ``expert`` tag (set by the probe from the filename
heuristic in piece #5 of Phase 5). The FE just hands over the bag of
LoRAs; no per-list FE plumbing needed.

buildWanGraph:
  - Calls addWanLoRAs(state, g, denoise, modelLoader) after the base
    transformer edge is in place. The helper is a no-op when no Wan
    LoRAs are enabled, so it's safe to call unconditionally.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(wan): detect LoRA variant and filter by main model

Wan 2.2 A14B (inner_dim=5120) and TI2V-5B (inner_dim=3072) LoRAs are not
interchangeable — applying one against the wrong main model crashes the
layer patcher with a tensor-shape error (e.g. A14B Lightning on TI2V-5B
mains produced ``shape '[3072, 3072]' is invalid for input of size 26214400``).

Probe Wan LoRAs' inner-dim at install time and record the family on a new
``variant`` field (``a14b`` / ``5b`` / null). The LoRA picker in the linear
view hides incompatible variants when the user selects a main, and the
graph builder filters any still-enabled mismatches at submit time with a
warning. Untagged LoRAs (probe couldn't identify) pass through so they
aren't silently hidden.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

feat(wan): ref-image panel, GGUF readiness, and auto-default sources

Wan 2.2 I2V now uses the global Reference Images panel (same UX as Qwen
Image Edit and FLUX.2 Klein) instead of pulling the conditioning image
from a canvas raster layer. Adds:

  - WanReferenceImageConfig zod type + isWanReferenceImageConfig guard;
    integrated into the ref-image discriminated union, settings panel,
    layer hooks, and validators.
  - 'wan' added to SUPPORTS_REF_IMAGES_BASE_MODELS, but the panel only
    shows for the i2v_a14b variant (T2V and TI2V-5B don't consume ref
    images, so the panel is hidden for them).
  - buildWanGraph I2V branch reads the first enabled wan_reference_image
    from refImagesSlice; the canvas-raster-as-ref path is removed. I2V
    now only supports txt2img mode (canvas img2img/inpaint/outpaint
    assert with a clear message).

GGUF Wan readiness check: GGUF mains carry only the transformer, so the
loader needs a Diffusers Component Source (or standalone VAE + UMT5-XXL
encoder) to resolve the VAE and text encoder. Without one, enqueue is
now blocked with a clear reason. The low-noise A14B partner expert
remains optional (loader falls back to the high-noise expert when it's
missing).

Adds standalone Wan VAE and Wan T5 Encoder selectors to the Advanced
accordion (Qwen pattern). Wires them as vae_model / wan_t5_encoder_model
on the wan_model_loader node — backend priority is standalone > diffusers
main > component source.

Auto-default on Wan selection (so GGUF users don't have to fiddle with
Advanced): when the new main is a Wan GGUF, fill the Component Source,
standalone VAE, and standalone T5 encoder with first available matches
if not already set. Component Source is matched by variant family
(A14B GGUF prefers an A14B Diffusers; TI2V-5B prefers a TI2V-5B
Diffusers) since the two families use different VAE channel counts
(16 vs 48); within A14B, T2V and I2V share VAE/encoder so they're
interchangeable as a source. Runs on every Wan selection (including
Diffusers -> GGUF switches), only fills empty slots.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): add Wan 2.2 starter models and bundle

Wan 2.2 starter pack (selected when the user picks the Wan 2.2 bundle)
brings up the minimal-cost path to running A14B T2V end-to-end:

  - Standalone UMT5-XXL encoder and A14B VAE (so GGUF mains don't need
    a full Diffusers download for their VAE/encoder sources).
  - T2V A14B Q4_K_M and Q8_0 GGUF expert pairs (high + low noise).
  - T2V Lightning V1.1 Seko rank-64 LoRA pair (4-step inference).

Additional Wan 2.2 starter models browseable from the model manager:

  - Full Diffusers T2V A14B, I2V A14B, and TI2V-5B.
  - I2V A14B Q4_K_M and Q8_0 GGUF expert pairs + Lightning V1 LoRA pair.
  - TI2V-5B Q4_K_M and Q8_0 GGUFs + the 48-channel TI2V-5B VAE.

Each "high noise" GGUF lists its low-noise partner plus the shared VAE
and UMT5-XXL encoder as dependencies, so installing one of them pulls
in everything the loader needs. QuantStack's HighNoise/LowNoise file
naming and lightx2v's high_noise_model/low_noise_model.safetensors are
both picked up by the existing filename heuristic in the GGUF probe.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

docs(wan): add Wan 2.2 hardware requirements

Adds Wan 2.2 A14B (T2V/I2V) and TI2V-5B rows to the hardware
requirements table with rough VRAM/RAM guidance per quantization.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(wan): recall low-noise transformer, component source, and standalone VAE/T5

Wan-specific metadata fields embedded by the graph builder
(wan_transformer_low_noise, wan_component_source, wan_vae_model,
wan_t5_encoder_model, wan_guidance_scale_low_noise) had no recall
handlers in features/metadata/parsing.tsx, so recalling an image's
parameters would leave these fields empty. Adds a handler for each
that dispatches the matching paramsSlice action and renders a row in
the metadata viewer.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): add default Wan 2.2 T2V and I2V workflows

Ships two default workflows in the library, tagged so they appear in
"Browse Workflows" under the wan2.2 / text to image / image to image
tags:

  - Text to Image - Wan 2.2: full T2V/TI2V-5B graph (model loader,
    positive + negative encoders, denoise, l2i). Exposes the five
    model slots, prompts, steps, dual CFG, and dimensions.
  - Image to Image - Wan 2.2: I2V A14B graph that adds a
    wan_ref_image_encoder. Exposes the reference image input plus
    the standard fields.

Both follow default-workflow rules: IDs prefixed with default_,
meta.category = "default", and no references to user-installed
resources.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(videos): Phase 1 - backend video storage, records, REST API

Adds a parallel video pipeline alongside the existing image pipeline so the
gallery can host MP4 alongside PNGs. Implements:

- New service modules (parallel to image equivalents):
    video_records/    record store + sqlite impl
    video_files/      disk file store (mp4 + first-frame webp thumb)
    videos/           orchestrating service
    board_video_records/   board <-> video association
- migration_32 creates `videos` and `board_videos` tables
- /api/v1/videos/ router: upload, list, get DTO, /full (with HTTP Range
  so HTML5 <video> seek/scrub works), /thumbnail, /metadata, star/unstar,
  delete, batch delete, board add/remove
- LocalUrlService.get_video_url and SimpleNameService.create_video_name
- imageio[ffmpeg] dep for video encode (used in later phases)
- Wires all four new services into InvocationServices, dependencies.py,
  api_app.py, and three test fixtures

Verified end-to-end against an in-memory db + tmp output dir: upload,
probe, save (file + thumbnail + record), DTO build, list, delete.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(videos): Phase 2 - polymorphic gallery list endpoint

Adds /api/v1/gallery/items/ and /api/v1/gallery/items/names returning a
unified time-sorted stream of images + videos so the frontend can render
them interleaved with a single virtualized query.

- gallery_common: GalleryItem discriminated union (kind + name + shared
  fields + nullable video duration/fps), GalleryItemRef, names result
- gallery_default: SqliteGalleryService implements UNION ALL across the
  images and videos tables, applying identical filters (origin/category/
  is_intermediate/board_id/search) to each half; pagination via outer
  ORDER BY + LIMIT/OFFSET; counts are summed across the two halves
- URLs are resolved at row -> DTO conversion time so each item routes to
  the correct /api/v1/images or /api/v1/videos endpoint
- Wired into InvocationServices, dependencies.py, api_app.py, and the
  three test fixtures

Existing /api/v1/images endpoints are unchanged so any non-gallery
consumers (queue, recall, metadata workflows) continue to work as-is.

Verified e2e: 2 images + 2 videos inserted in alternating order, both
list_items and list_item_names return the correct interleaved order;
category filter narrows to a single kind; starring an item bumps it to
the top when starred_first=True.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(videos): Phase 3 - frontend RTK endpoints + MP4 upload routing

Adds the typed API surface and upload integration so videos can be
uploaded through the same gallery upload button that handles images.

Schema: re-ran pnpm typegen against the running backend to pick up
VideoDTO, VideoRecordChanges, GalleryItem, GalleryItemKind,
GalleryItemRef, GalleryItemNamesResult and the two new paginated
result types.

RTK Query (services/api/endpoints/videos.ts) - parallel to images.ts:
listVideos, getVideoDTO, getVideoMetadata, getVideoNames, uploadVideo,
deleteVideo / deleteVideos, changeVideoIsIntermediate, starVideos /
unstarVideos, addVideoToBoard / removeVideoFromBoard. Imperative helpers
(getVideoDTO, getVideoDTOSafe, uploadVideo, uploadVideos) and the
useVideoDTO convenience hook ride alongside, mirroring the image side.

Tag types and invalidation: added Video / VideoList / VideoMetadata /
VideoNameList / BoardVideosTotal / GalleryItemList / GalleryItemNameList
to the api root. Board-affecting mutations now invalidate the polymorphic
gallery list/name caches so videos and images stay coherent once the
gallery wiring lands in Phase 4. Added a sibling
getTagsToInvalidateForVideoMutation helper.

Upload UX: useImageUploadButton.tsx's dropzone now accepts video/mp4,
video/webm, video/quicktime alongside the existing image MIMEs. The
drop handler splits files into image/video sets and routes each through
its own mutation; a new onUploadVideo callback parallels the existing
onUpload. Existing image-only callers pass through unchanged.

Polymorphic gallery query endpoints + the useGalleryItemDTO hook will
land with Phase 4 where they have actual consumers; the schema types
they'll need are already in place under @knipignore tags.

Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass; live curl against the running dev server
uploads an MP4 and serves both the webp thumbnail and the MP4 with
a working HTTP Range response (206 + Content-Range).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(videos): Phase 4 - mixed gallery grid with video play badge

Videos now appear in the same gallery grid as images, interleaved by
created_at. Video thumbnails get a centered play-button badge so they
read as videos at a glance; everything else (selection, virtualization,
search, paged/virtual gallery views, keyboard nav) is unchanged.

Approach: selection state stays `string[]` of names. The kind is
recovered from the filename extension (.mp4 = video, anything else =
image), which is reliable because the backend's SimpleNameService
always emits `<uuid>.png` for images and `<uuid>.mp4` for videos. This
sidesteps a 32-file cross-cut from changing the selection shape to a
discriminated union, and selection is persist-denylisted so no
migration is needed.

Frontend:
- new isVideoName helper in features/gallery/store/types
- new endpoints/gallery.ts (deferred from Phase 3): useGetGalleryItemNamesQuery
- new ImageGrid/GalleryItemPlayBadge: centered triangular badge over thumbnail
- new ImageGrid/GalleryItemVideoStarIconButton: video-typed star toggle
- new ImageGrid/GalleryVideoItem: counterpart to GalleryImage; reuses
  galleryItemContainerSX, GalleryItemSizeBadge (width/height-only stand-in),
  selection handling (single/shift/ctrl/cmd); alt-click falls through to a
  normal select since comparison is image-only
- use-gallery-image-names now calls the polymorphic gallery names endpoint
  and exposes a mixed flat name list (existing callers - paged grid, search,
  navigation hotkeys - get the same shape)
- useRangeBasedImageFetching partitions visible names by extension; images
  bulk-fetch via the existing getImageDTOsByNames mutation, videos dispatch
  individual getVideoDTO queries (no batch endpoint yet)
- GalleryImageGrid's ImageAtPosition dispatches on isVideoName to render
  GalleryImage or GalleryVideoItem; star hotkey dispatches to the right
  star/unstar mutation based on kind
- pruned the now-unused useGetImageNamesQuery / isImageName exports

Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass; live curl of /api/v1/gallery/items returns
57 polymorphic items with video duration populated and image duration
null, /api/v1/gallery/items/names returns matching {kind, name} refs.

The useGalleryItemDTO hook is intentionally deferred to Phase 5 where
the polymorphic viewer is its first real consumer.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(videos): Phase 5 - inline video player in the image viewer

Selecting a video now renders a polymorphic preview inside the existing
viewer panel: thumbnail with a centered play button by default; clicking
play swaps in an HTML5 <video controls autoplay>. Switching to a
different item drops the video element back to idle (auto-pauses) and
selecting an image again returns to the normal image preview.

New components (features/gallery/components/ImageViewer/):
- VideoPlayButtonOverlay: large centered play button with hover/shadow,
  used over the thumbnail in the idle state.
- CurrentVideoPreview: idle/playing state machine. Resets on
  video_name change. The <video> src points at /api/v1/videos/i/.../full
  which supports HTTP Range, so seek/scrub work natively in the browser.

New hook:
- common/hooks/useGalleryItemDTO: polymorphic DTO resolver that
  dispatches between useImageDTO and useVideoDTO based on filename
  extension (isVideoName). Centralizes the kind-dispatch the viewer
  and toolbar both need.

Wiring:
- ImageViewer dispatches on galleryItem.kind to render CurrentImagePreview
  or CurrentVideoPreview. The compare-image DnD drop target is hidden when
  a video is selected (comparison is image-only).
- ImageViewerToolbar hides the image-specific action row
  (CurrentImageButtons - load workflow, recall metadata, edit, etc.) and
  the metadata viewer toggle when a video is selected. The general-purpose
  ToggleProgressButton stays.

Out of scope (per the plan): video deletion from the viewer (use gallery
hover icons), video-specific metadata viewer, comparison-mode support
for videos.

Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(videos): accept MP4 (and other video) drops on the fullscreen dropzone

The gallery-wide drag-and-drop target lives in FullscreenDropzone, not
in useImageUploadButton (which only powers the upload button). It had
its own hardcoded image-only zod allowlist that rejected MP4 files
with "File type / extension is not supported".

- Broaden the zod refines to accept video/mp4, video/webm,
  video/quicktime, video/x-matroska and the matching extensions
- Add isVideoFile helper, split dropped files into image/video sets,
  and route each set through its own uploader (uploadImages /
  uploadVideos). Both update their respective RTK caches and
  invalidate the polymorphic gallery list/names.
- Skip the canvas-paste fast-path for single-video drops — the canvas
  doesn't host videos as layers.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(videos): right-click context menu on video items

Adds a three-item context menu (delete, change board, download) on
right-click / long-press of any gallery video item. Mirrors the image
context menu's singleton-portal architecture so re-renders stay cheap.

New files:
- features/gallery/contexts/VideoDTOContext: small React context that
  scopes the active video DTO to the menu items (parallels
  ImageDTOContext).
- features/gallery/components/ContextMenu/MenuItems/
    ContextMenuItemDeleteVideo: window.confirm + deleteVideo mutation.
      Videos can't be referenced from canvas/nodes/refs, so the image
      modal's usage analysis is unnecessary; a one-step confirm matches
      the "minimal" scope.
    ContextMenuItemDownloadVideo: reuses the existing useDownloadItem
      hook against videoDTO.video_url / video_name.
    ContextMenuItemChangeBoardVideo: dispatches videosToChangeSelected
      and opens the (now polymorphic) ChangeBoardModal.
- features/gallery/components/ContextMenu/VideoContextMenu: singleton
  pattern lifted from ImageContextMenu — registers gallery video
  elements via a Map; right-click looks up the target node and opens
  the menu at the cursor.

Extended files:
- features/changeBoardModal/store/slice: added video_names alongside
  image_names plus a videosToChangeSelected action. The two arrays are
  mutually exclusive — setting one clears the other.
- features/changeBoardModal/components/ChangeBoardModal: now dispatches
  the matching video board mutations (add/removeVideoToBoard, plural
  endpoints don't exist yet so videos move one at a time — the menu
  acts on a single selection so this is a one-iteration loop).
- features/gallery/components/ImageGrid/GalleryVideoItem: registers
  itself with useVideoContextMenu.
- app/components/GlobalModalIsolator: mounts the singleton.

Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): Phase 6 - Wan 2.2 T2V/I2V workflow nodes

Adds two new invocation nodes that produce MP4 videos from a Wan 2.2
A14B transformer + VAE, plus the supporting plumbing.

New invocations:
- WanVideoDenoise (wan_video_denoise) — multi-frame counterpart to
  WanDenoise. Same per-step logic (CFG, MoE expert swap at the
  boundary timestep, LoRA patching, scheduler dispatch) — reuses
  _ExpertSwapper, _resolve_variant, and the scheduler/LoRA helpers
  from wan_denoise. Difference: the noise tensor has a real temporal
  dim built from num_frames, and the I2V condition is built across
  all latent frames (frame 0 conditioned, rest zero). Defaults match
  the Wan 2.2 reference: 832x480 / 81 frames / 40 steps / CFG 5.0
  (high) + 4.0 (low). Inpaint / img2img are out of scope for this
  first cut. TI2V-5B is rejected; T2V/I2V A14B only.
- WanLatentsToVideo (wan_l2v) — VAE-decodes 5D latents to RGB frames
  via AutoencoderKLWan (T_pixel = (T_lat - 1) * 4 + 1), then encodes
  an MP4 with imageio[ffmpeg] (libx264, yuv420p for browser
  compatibility). The temp file is moved into outputs/videos/ via
  context.videos.save().

Backend shared pieces:
- make_noise gains num_latent_frames (default 1, backward compatible).
- Added num_latent_frames_for(num_frames, scale=4) helper.
- New encode_reference_image_to_video_condition mirrors diffusers'
  WanImageToVideoPipeline.prepare_latents with last_image=None and
  expand_timesteps=False: pads the reference image with zero
  pixel-frames, VAE-encodes the full pseudo-video, normalises, and
  builds the 4-channel temporal-rearranged first-frame mask. Verified
  numerically: 21 latent frames for num_frames=81, first latent
  frame's 4 mask channels = 1, rest = 0.
- The existing single-frame encoder is left untouched.

Schema / context:
- New VideoField primitive (parallel to ImageField) and VideoOutput
  invocation output (width/height/num_frames/fps/duration/video).
- New VideosInterface on InvocationContext with .save(source_path,
  width, height, duration, fps, ...) returning VideoDTO. Mirrors
  ImagesInterface — falls back to WithBoard / WithMetadata mixins
  and embeds the queue item's workflow/graph as a JSON sidecar.
- WanRefImageConditioningField now carries num_frames so the denoise
  nodes can sanity-check the I2V condition. WanRefImageEncoder bumps
  to v1.1.0 and gains num_frames=1 input (use 81+ for video I2V; the
  encoder dispatches between the single- and multi-frame helpers).
- Image WanDenoise now rejects multi-frame conditions with a clear
  message pointing at WanVideoDenoise.

Verified: pnpm lint (5/5) green; pnpm tests (multiuser auth 122/122
+ broader suite via prior runs); numerical shape checks for noise
and ref-image condition; end-to-end smoke via VideoService.create.

A restart of the InvokeAI server is required to pick up the new
invocations in the workflow editor.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): add Wan 2.2 T2V and I2V starter video workflows

Two new default workflows for the workflow editor 'Browse' modal:

- 'Text to Video - Wan 2.2' — model loader -> two text encoders ->
  wan_video_denoise -> wan_l2v. Exposes prompt, model picks, CFG
  (high + low), dimensions, frames, fps, and steps.
- 'Image to Video - Wan 2.2' — same shape plus a wan_ref_image_encoder
  feeding the denoise node's ref_image input. Exposes the reference
  image and the frames field on the ref-image node (must match the
  denoise node's frames — there is a clear validation error if they
  diverge, but the starter has them in sync at 81).

Both default to the Wan 2.2 reference settings: 832x480, 81 frames @
16 FPS (~5 s), 40 steps, CFG 5.0 (high expert) + 4.0 (low expert),
seeded by a rand_int. Pass the existing _sync_default_workflows
validator (id starts with default_, meta.category=default).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(wan): startup crash from stringified VideoOutput annotation

run_app.py validates every invocation's return-type annotation against
the output-class registry. wan_latents_to_video.py had a stray
'from __future__ import annotations' which made the `invoke()` return
annotation a string ('VideoOutput') at runtime. The registry mismatch
triggered the unregistered-output warning path, which itself crashed
on output_annotation.__name__ because the annotation was a str:

  AttributeError: 'str' object has no attribute '__name__'

The other Wan invocations don't use future annotations — drop the
import to match. Verified post-fix: api_app import populates 95
output classes, wan_l2v annotation resolves to the real VideoOutput
class and is in the registry.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): add Wan 2.2 Lightning T2V starter workflow

Same graph as 'Text to Video - Wan 2.2' but with two Apply LoRA - Wan
2.2 nodes chained between the model loader and the denoise node, and
defaults retuned for the Lightning distillation: 4 steps and CFG 1.0
on both experts (CFG=1 skips the negative-conditioning forward pass
entirely, ~20x faster than the 40-step / CFG-5.0 baseline at similar
quality).

Adapted from a user-saved workflow; cleaned for distribution by
stripping the install-specific model/LoRA key bindings (defaults
should not bake in local UUIDs), bumping to a fresh default_-prefixed
id with meta.category=default, exposing the two LoRA fields (lora +
weight) so users can swap LoRAs without diving into the canvas, and
flagging the negative-prompt node as unused at CFG=1.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): add Wan 2.2 Lightning T2V and I2V starter workflows

Two new default workflows that wire the Lightning LoRA pair into the
T2V and I2V video pipelines for a ~20x speedup:

- 'Text to Video - Wan 2.2 Lightning' — model loader -> apply LoRA
  (high) -> apply LoRA (low) -> text encoders -> wan_video_denoise
  -> wan_l2v. Defaults to 4 steps and CFG 1.0 (no negative branch).
  Cleaned-up version of Lincoln's saved Lightning workflow: stripped
  per-install model/LoRA keys, switched meta.category to 'default'
  with a default_ id, and exposed both LoRA loaders' lora/weight/
  target fields so users can swap LoRAs without diving into the
  canvas.
- 'Image to Video - Wan 2.2 Lightning' — same chain plus a
  wan_ref_image_encoder (v1.1.0 with num_frames) feeding the denoise
  ref_image input. Defaults match the non-Lightning I2V starter
  (832x480, 81 frames @ 16 FPS) but with 4 steps / CFG 1.0.

LoRA target defaults to 'auto' so properly-tagged Lightning LoRAs
route themselves; both workflow descriptions tell users to set
explicit 'high'/'low' targets if their LoRAs are untagged.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(wan): use FFMPEG plugin (not pyav) for MP4 encode

wan_latents_to_video was passing plugin='pyav' to iio.imwrite, but the
runtime only has imageio-ffmpeg installed (no PyAV). The encode step
at the very end of generation crashed with:

  ImportError: The `pyav` plugin is not installed.
  Use `pip install imageio[pyav]` to install it

Switch to plugin='FFMPEG' — backed by the bundled imageio-ffmpeg
binary that pyproject already requires via imageio[ffmpeg]. libx264
yuv420p is the FFMPEG plugin's default for .mp4, so the explicit
pixel_format is dropped (specifying it just produced a "Multiple
-pix_fmt options" warning).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): log VAE decode and MP4 encode milestones in wan_l2v

The video VAE decode + MP4 encode tail can take 30-90s on top of the
denoise loop, and the toast-style signal_progress() messages don't
land in the server log. Add context.logger.info() at:

- VAE decode start: latent frame count -> pixel frame count + resolution
- MP4 encode start: frames, fps, duration, dimensions
- MP4 encode complete: encoded file size
- Video saved: final video_name

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(wan): switch video thumbnail/probe to imageio[ffmpeg] backend

After wan_l2v wrote a successful libx264 MP4 to disk, the invocation
would hang in DiskVideoFileStorage.save() during the cv2.VideoCapture
thumbnail-extraction step. cv2 wheels on this build can't reliably
decode our libx264/yuv420p output (most often the wheel was compiled
without an h264 decoder, but the failure mode is silent hang rather
than a clear error). The net effect: the MP4 ends up in
outputs/videos but the queue item never completes, so the frontend
spinner spins forever and the gallery doesn't pick up the new entry.

Fix: rewrite extract_video_frame and probe_video to try imageio's
FFMPEG plugin first (same backend that did the encoding — so reading
our own output is guaranteed to work), with cv2 retained only as a
fallback for uploaded videos in formats imageio can't decode.

Also add fine-grained log lines + exception guards inside
DiskVideoFileStorage.save() so a future thumbnail failure can no
longer hang the whole save — it now logs a warning and continues,
leaving the video record in place even if the thumbnail step
errored. With logging at each step (video written, thumbnail
written, sidecar written) any future hang will be obvious from the
last log line.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(videos): handle VideoField outputs in invocation_complete

After wan_l2v wrote its MP4 successfully, the gallery and viewer were
never updated: the new video didn't appear and the viewer stayed stuck
on the previous "Saving video" progress spinner indefinitely.

Root cause: onInvocationComplete.tsx only inspected results for
isImageField / isImageFieldCollection. VideoField outputs were silently
dropped, so the polymorphic gallery list never invalidated and no
auto-switch happened. The viewer therefore kept rendering
CurrentImagePreview, whose ImageViewerContext-local $progressEvent /
$progressImage atoms intentionally aren't cleared on queue completion
when autoSwitch is on — they rely on the new image's DndImage onLoad
to clear them, which never fires for a video.

Fix: add isVideoField (mirrors isImageField against {video_name}) and
plumb video outputs through onInvocationComplete:
- getResultVideoDTOs pulls VideoDTOs via getVideoDTOSafe
- addVideosToGallery invalidates GalleryItemNameList / GalleryItemList
  so the polymorphic gallery refetches and the new video shows up
- auto-switch dispatches the video name into selection (selection is a
  polymorphic string[]; useGalleryItemDTO already discriminates by
  filename extension)

The selection change swaps CurrentImagePreview for CurrentVideoPreview,
which unmounts the stale progress overlay along with it — so the stuck
spinner clears as a side-effect of the auto-switch.

Also drops the now-stale @knipignore on getVideoDTOSafe, which has a
real consumer now.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(video): add 'Frame from Video' invocation

Extracts a single frame from a VideoField input and saves it as a
regular ImageDTO via context.images.save, so it appears in the gallery
like any other generated image.

Primary use case is I2V "shot extension": take the last frame of a
Wan-generated clip (default frame_index=-1) and feed it back as the
reference image for the next clip, then stitch the MP4s to get videos
longer than the model's single-shot frame budget at a given VRAM.

Negative frame_index is resolved against the actual decoded frame count
via probe_video() rather than passed through to imageio — not all
imageio plugins handle index=-1 uniformly, and being explicit lets us
emit a precise out-of-range error.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(video): add 'Concatenate Videos' invocation

Joins two or more videos into a single MP4 with one of three transition
modes between consecutive clips:

- cut: hard splice, no blending. Total length = sum of inputs.
- crossfade: linear A→B dissolve over transition_frames. Each boundary
  consumes N frames from both surrounding clips, shrinking total length
  by N per boundary.
- fade_through_black: A fades to black, then B fades in. Each boundary
  consumes N/2 from each side and emits N output frames — total length
  is preserved.

Implementation decodes via imageio's FFMPEG plugin (matching wan_l2v on
the encode side) and runs the blends in numpy. All decoded frames are
kept in memory at once; fine for the few-hundred-frame I2V chains that
motivated this, would want streaming if anyone ever feeds in hour-long
uploads.

Up-front validation enforces matching dimensions across inputs and
checks that each clip has enough frames to spare from its head and tail
for the requested transitions — saves a wasted decode pass when the
transition window is too wide for one of the clips.

Pairs with 'Frame from Video' for I2V shot extension: generate N clips
chained via last-frame-as-ref-image, then glue them with a crossfade.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(videos): show full-resolution first frame in viewer

The viewer used a chakra <Image src={thumbnail_url}> in the idle (not-
playing) state, so once a clip auto-selected after generation the
preview snapped from the full-resolution denoise progress image to the
small WebP gallery thumbnail upscaled to fit — visibly soft compared to
what the user was watching seconds earlier.

Switch to a single <video> element that spans both states:

- idle: muted, no controls, preload="metadata". With no `poster` attr
  the browser decodes and shows the video's actual first frame at full
  resolution (this is the documented HTMLVideoElement default).
- playing: same DOM node with controls+audio toggled on, kicked off via
  ref.play(). No reload between states — the decoded buffer carries
  over.

`key={videoName}` swaps the element cleanly when the user moves to a
different clip, dropping any in-progress playback state.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(nodes): show 'Save in gallery' on video-output nodes

The footer checkbox was gated on useNodeHasImageOutput, which only
matched ImageField outputs. wan_l2v and video_concat produce VideoField
and so had no toggle — users had no UI path to flip is_intermediate on
them, even though VideoOutput goes through context.videos.save and
lands in the gallery the same way ImageOutput does.

Rename the hook to useNodeHasGalleryOutput and extend it to match
VideoField as well. Update the three call sites (the hook itself,
the checkbox, and the footer wrapper) so the toggle and the footer
render whenever a node produces something destined for the gallery.

The image primitive ('image' type) is still excluded since it doesn't
save a new image; no equivalent video primitive exists yet, so no
analogous exclusion is needed for VideoField.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore: remove unwanted planning documents

* chore: fix ruff I001 import-order violations

Auto-fix from `ruff check --select I001 --fix`. Touches 10 files across
the Wan and videos changes where added imports landed out of order.
No behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(videos): restrict uploads to MP4 only

The upload allowlist previously included .mov/.webm/.mkv, but the
names service (create_video_name) unconditionally emits {uuid}.mp4
and we don't transcode on upload. The result: non-MP4 containers were
stored under a .mp4 name and served with the .mp4 MIME type, which
silently broke playback in browsers when the container didn't match.

Drop the non-MP4 extensions from ACCEPTED_VIDEO_EXTENSIONS and tighten
the accepted MIME prefix to "video/mp4". Wan-generated output is MP4
anyway, so this matches current reality. If we want to support more
containers later, the right move is to extend the names service to
preserve the source extension, then re-add the formats here.

Also drops the now-dead suffix-detection block in upload_video and
the os import it required.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(videos): clean up stale @knipignore on consumed hooks

useDeleteVideoMutation, useAddVideoToBoardMutation, and
useRemoveVideoFromBoardMutation are now consumed by Phase 4 components
(context menu, change-board modal) but were still annotated with the
multi-phase @knipignore tag — that generated false-positive knip
warnings and misrepresented the implementation status.

Move those three into the unconditional export block. The remaining
five hooks (useListVideosQuery, useGetVideoMetadataQuery,
useGetVideoNamesQuery, useDeleteVideosMutation,
useChangeVideoIsIntermediateMutation) are still unused in the current
codebase and stay under a narrower @knipignore.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs(videos): document invalidate/select race in addVideosToGallery

The video gallery path uses tag invalidation rather than an optimistic
insert (the image path's `insertImageIntoNamesResult` doesn't have a
polymorphic equivalent yet). Because invalidation kicks off an async
refetch, the `imageSelected` dispatch below it fires before the new
video name is in `imageNames`, so the gallery grid's
`useKeepSelectedImageInView` no-ops on its first pass. The scroll
self-corrects on the next pass when the refetch lands and the
`imageNames` dep updates.

The user-visible effect is just a small lag on gallery scroll-to-
selection — the viewer selection applies immediately — so this is a
documented limitation rather than a bug. Worth a follow-up if the
lag becomes noticeable in practice.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(video): add Video Primitive invocation and VideoField input UI

Mirrors the Image Primitive flow end-to-end for videos. Users can now
drag a video from the gallery onto a "Video Primitive" node and feed
its output into downstream nodes like Frame from Video or Concatenate
Videos — exactly the way Image Primitive feeds the rest of the image
pipeline.

Backend (invokeai/app/invocations/primitives.py):
- New VideoInvocation, declared *after* VideoOutput so the return
  annotation is a real class (not a forward-ref string). Stringified
  output annotations crashed startup before — see cac366229a — so the
  ordering matters.

Frontend:
- Register VideoField as a stateful field type in types/field.ts:
  zVideoFieldType, zVideoFieldValue, zVideoFieldInputInstance/Template,
  output template + type guards, plus entries in the four stateful
  unions (FieldType, FieldValue, InputInstance, InputTemplate).
- buildFieldInputTemplate / buildFieldInputInstance gain VideoField
  branches so OpenAPI-derived templates resolve correctly.
- nodesSlice: fieldVideoValueChanged reducer + export.
- imageActions/actions.ts: setNodeVideoFieldVideo helper.
- dnd.ts: singleVideoDndSource + setNodeVideoFieldVideoDndTarget,
  wired into the dndTargets array.
- GalleryVideoItem: register itself as a drag source so videos in the
  gallery actually drag (previously they were click-only).
- VideoFieldInputComponent: parallel to ImageFieldInputComponent —
  shows the WebP thumbnail with a dimensions badge, accepts video DnD,
  drops stale references on reconnect if the underlying video was
  deleted.
- InputFieldRenderer: dispatch VideoField templates to the new
  component (placed right after the ImageField branch).
- useNodeHasGalleryOutput: also exclude the new `video` primitive type
  so the "Save in gallery" toggle does not render on the pass-through
  node (same treatment the `image` primitive already gets).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

fix(video): allow video drops to reach DnD target handlers

useDndMonitor is the global drop monitor that actually invokes each
target's handler() — DndDropTarget only does enter/leave bookkeeping.
Its canMonitor gate explicitly allowlists source types and only listed
singleImageDndSource + multipleImageDndSource. So when a video was
dragged from the gallery onto a VideoField input, the drop was visible
to the DOM but the monitor silently filtered it out, the handler never
ran, and fieldVideoValueChanged was never dispatched.

Add singleVideoDndSource to the allowlist. Dropping a video onto a
Video Primitive (or any other VideoField input) now wires the asset
into the field as intended.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(backend): ruff

* chore(frontend): typegen

* feat(wan): split Wan 2.2 starter bundle into T2V and I2V

Replaces the single ~63 GB Wan 2.2 bundle with two smaller bundles so
users only pay for the capability they need. T2V (~36 GB) covers
text-to-video plus a low-VRAM image-to-video option via TI2V-5B; I2V
(~32 GB) adds the heavier I2V-A14B path. Drops the Q8 T2V pair from the
default bundle — both Q8 variants and full Diffusers builds remain
available as a-la-carte starters.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(videos): tighten multiuser isolation in list and board-move endpoints

Three related fixes flagged in code review (PR #9163, JPPhoto):

1. Video and gallery list/name SQL paths only filtered by user_id when
   board_id was the literal "none" sentinel. When the URL parameter was
   omitted entirely, no user filter applied and non-admin callers could
   enumerate every user's videos / mixed gallery items. Added an
   explicit per-user isolation branch for the omitted case.

2. /v1/videos/ and /v1/videos/names accepted explicit board IDs with
   no read-access check; the route now mirrors the images and gallery
   routers and calls _assert_board_read_access for non-"none" values.

3. add_video_to_board and remove_video_from_board only validated video
   ownership, not destination/source board write access — a caller
   could move their video into someone else's private board. Added
   _assert_board_write_access and a strict _assert_video_direct_owner
   helper (no board-owner / public-board fallback) for board-move ops,
   mirroring _assert_image_direct_owner.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(boards): cascade video deletion when deleting a board with media

Previously delete_board only handled images. With include_images=true the
backend would delete images on the board but the videos would silently
cascade out of board_videos and survive as uncategorized records — almost
certainly not what the caller intended. Without include_images, the same
mismatch meant the response could not report affected videos.

Now:
- include_images=true also calls videos.delete_videos_on_board
- include_images=false collects the soon-to-be-uncategorized video names
- DeleteBoardResult gains deleted_board_videos and deleted_videos fields
  (default empty so existing clients are unaffected)

Frontend deleteBoard / deleteBoardAndImages mutations gain the matching
VideoList / VideoNameList / GalleryItem* tag invalidations so the
polymorphic gallery and video list views refresh.

Reported in code review (PR #9163, JPPhoto).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(videos): return affected_boards from board move/remove endpoints

removeVideoFromBoard previously returned VideoDTO; the frontend then read
result.board_id (null after removal) and only invalidated the 'none'
board cache — the previous board's list stayed stale until refetch.
addVideoToBoard had the same problem (the route never knew the source
board, so the old-board cache was never invalidated).

Mirror the image equivalents (board_images.py): the routes now return
AddVideosToBoardResult / RemoveVideosFromBoardResult with the moved
video name(s) and the full set of affected board IDs. Both old and new
boards get invalidated atomically. Frontend mutations updated to
consume the new shape via getTagsToInvalidateForBoardAffectingMutation
on result.affected_boards.

The auto-generated schema.ts will need a typegen pass after the dev
server restart to pick up the new response types.

Reported in code review (PR #9163, JPPhoto).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(videos): stream uploads, FileResponse for full video + thumbnail

Three perf items from the code review (PR #9163, JPPhoto):

- upload_video read the entire UploadFile into a Python bytes object
  before writing to the temp file. Multi-GB videos allocated multi-GB
  buffers. Now chunk-stream into the temp file with a 1 GB per-upload
  cap (HTTP 413 on overflow). Cap is intentionally generous — the goal
  is RAM-exhaustion protection, not content policy.

- get_video_full read the whole MP4 into RAM when no Range header was
  present. Browsers usually send Range, but curl / direct downloads /
  CDN edge fetches do not, and a multi-GB load per such request is a
  trivial DoS vector. Replaced with FileResponse (sendfile).

- get_video_thumbnail similarly buffered the WebP. Thumbnails are tiny
  so this was minor, but FileResponse is idiomatic and shaves the
  unnecessary copy.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(videos): localize video UI strings

- Add gallery.deleteVideo_one / deleteVideo_other, deleteVideoConfirmation,
  and playVideo to en.json
- ContextMenuItemDeleteVideo: drop the inline English defaultValue (the
  translation key now exists) and use gallery.deleteVideo for aria/tooltip
  (was reusing gallery.deleteImage so it rendered "Delete Image")
- VideoPlayButtonOverlay: replace the hardcoded "Play video" aria with
  t('gallery.playVideo')

Reported in code review (PR #9163, JPPhoto).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(video-invocations): exact frame counts and odd-tf transitions

video_frame_extract: resolving frame_index=-1 previously computed
n_frames as round(duration * fps). For VFR uploads or containers with
approximate metadata that can overshoot the actual decoded frame count,
making the last-frame extraction fail. Use iio.improps(plugin='FFMPEG')
for the exact decoder count when available; fall back to the duration
* fps estimate only if the props query fails.

video_concat fade_through_black: with an odd transition_frames the
symmetric half = tf // 2 split emitted tf - 1 frames per boundary,
violating the documented "emits transition_frames" contract. Split
asymmetrically (tail_half = tf // 2, head_half = tf - tail_half) so
the emitted count equals tf exactly for both even and odd values.
Validation and docstring updated to match.

Verified with manual cases: tf=1, tf=4, tf=5 all emit the documented
total length for two 10-frame inputs.

Reported in code review (PR #9163, JPPhoto).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs(gallery): describe video gallery items, upload, and deletion

Update the Gallery Panel docs to reflect the polymorphic gallery added
in the Wan 2.2 video feature branch:

- Gallery intro now mentions images + videos coexist on boards.
- Board deletion warning clarified to cover both kinds of media.
- New "Videos in the Gallery" section covering: how video items appear
  (first-frame thumbnail + play badge), MP4-only upload constraint with
  the typical re-encode command, the video context menu, and that
  videos count toward board totals.

Reported in code review (PR #9163, JPPhoto).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test(videos): regression coverage for PR #9163 review fixes

Adds tests that pin the behaviour fixed in the JPPhoto review and would
catch a recurrence:

- tests/app/services/video_records/test_video_records_sqlite.py
  get_many / get_video_names: non-admin callers only see their own
  videos when board_id is omitted; admins see all; the "none" branch
  still filters by user.

- tests/app/services/gallery/test_gallery_default.py
  Same multiuser isolation guarantee through the polymorphic gallery
  union for both images and videos.

- tests/app/routers/test_videos_multiuser.py
  /v1/videos/ and /v1/videos/names: 403 when a non-owner passes an
  explicit private board_id; 200 for owners, admins, "none", and omitted
  board_id (the auth-required smoke tests pin the 401 paths too).

- tests/app/routers/test_boards_multiuser.py
  Adds two delete-board cases proving the video cascade: include_images
  invokes delete_videos_on_board and reports deleted_videos; the
  no-include path reports deleted_board_videos without calling the
  destructive service. Existing fixture extended to stub the video
  services that the new route logic now touches.

- tests/app/invocations/test_video_concat.py
  Parametric coverage that fade_through_black emits exactly tf frames
  for both even and odd tf, plus three-clip chains, plus the crossfade
  and cut/zero-tf cases as guards.

- tests/app/invocations/test_video_frame_extract.py
  _decoder_frame_count returns the exact count via the cv2 fallback for
  several clip lengths and gracefully returns None for missing /
  non-video inputs (caller falls back to duration * fps).

Bug found during test authoring: _decoder_frame_count over-flowed int()
on iio's "inf" nframes for libx264 streams, and improps never returns a
real count for that codec anyway. Helper now ignores non-finite shapes
and falls back to cv2's CAP_PROP_FRAME_COUNT, which gives the exact
value for libx264.

schema.ts regenerated to pick up the AddVideosToBoardResult /
RemoveVideosFromBoardResult / extended DeleteBoardResult types added
in earlier commits in this series.

All 70 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): add 'Wan 2.2 I2V Ideal Dimensions' invocation

Computes Wan I2V-compatible (width, height) for a source W×H at a target
short-side resolution (e.g. 720 for "720p"), snapping each output to a
multiple of 16 (Wan's transformer patch_size × VAE 8x pixel-grid
constraint enforced by wan_ref_image_encoder).

Replaces the 6-node math chain (Float Math × 4 + Float To Integer × 2)
that was otherwise required to compute these dimensions from an arbitrary
input image. Wire the Image Primitive's width/height outputs into this
node, and feed its (width, height) outputs into both
wan_ref_image_encoder and wan_denoise (they must match).

Three rounding modes:
- nearest (default): minimizes aspect-ratio drift
- floor: guaranteed not to exceed unsnapped target (safer for VRAM)
- ceiling: rounds up

Output schema reuses IdealSizeOutput so it slots into existing pipes that
already consume Ideal Size — SD1.5, SDXL.

Includes regression tests covering the documented common-case table,
all three rounding modes, postcondition invariants (multiple of 16,
aspect ratio within 1.2%, never zero), and input validation.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): swap target_short_side int for 480p/720p/1080p preset dropdown

Wan 2.2 was trained at 480p and 720p; a free integer encouraged users
to pick noncanonical short sides that the model handles poorly. Replace
the int field with a Literal dropdown of "480p" / "720p" / "1080p"
(via ui_choice_labels) so the UI surfaces the canonical choices.

1080p is included with a label noting it's extrapolated from training
(not a Wan native size) — useful for users with VRAM headroom but
shouldn't be the default.

Version bumped to 1.1.0 since the field schema changed (the node was
only committed locally; no published workflow needs migrating).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(wan): empty_cache around the I2V reference-image VAE encode

Two-sided fix to avoid VRAM allocator fragmentation that was causing the
subsequent denoise-transformer partial load to OOM:

- Before vae.encode(): clears blocks left over from earlier nodes (the
  denoise expert swap especially leaves the cache fragmented).
- After the condition tensor is on CPU: returns the VAE encode's
  intermediates so the next partial_load_to_vram sees a real free
  contiguous range.

Mirrors the same pattern in wan_latents_to_image.py and
wan_latents_to_video.py — those are the existing precedent. The cost is
a handful of microseconds per encoder invocation and only the cache
state is touched; model weights stay resident.

Observed-by symptom from a workflow review: at encoder=480x720 and a
source image of 880x1184, the encoder ran fine but the I2V high-noise
expert failed to partial-load with a cryptic CUDA OOM at
_load_state_dict_with_fast_device_conversion. Pre-resizing the source
to 80% incidentally cleared the allocator state and let the run
succeed; this fix removes the incidental dependency on source size.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): support TI2V-5B in the video denoise node (T2V mode)

The video denoise node previously hard-errored on TI2V-5B with "not
supported." Most of the surrounding machinery (variant-aware spatial
scale, variant-aware scheduler, single-expert ExpertSwapper path) was
already in place — the gate just needed lifting and the hard-coded
A14B latent channel count needed to follow the variant.

Changes:
- Drop the upfront "TI2V-5B is not supported" raise.
- Use get_default_latent_channels(variant) so latents are 48-channel for
  TI2V-5B and 16-channel for the A14B family (matches the image denoise
  node's existing logic).
- For TI2V-5B with a Reference Image input, raise a sharper, accurate
  error that explains TI2V-5B's I2V uses diffusers' expand_timesteps
  path (first-frame-mask blend + per-position timestep gating) which
  this node does not implement yet — pointing the user at the working
  T2V path or the I2V-A14B model.
- Update the transformer field description to reflect what's now
  supported.

Image-to-video with TI2V-5B remains a follow-up; the conditioning math
is genuinely different from A14B (no 36-channel concat) and warrants a
separate code path rather than parameterising this one.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(wan): instantiate TI2V-5B VAE with the right architectural config

The single-file Wan VAE loader was always calling
``AutoencoderKLWan(z_dim=config.latent_channels)`` and relying on
diffusers' constructor defaults for every other parameter — but those
defaults match the Wan 2.1 / A14B VAE (base_dim=96, in/out=3, 8x
spatial, no patchify). For TI2V-5B's Wan 2.2-VAE the architecture is
materially different:

- base_dim=160, decoder_base_dim=256
- in_channels=12, out_channels=12 (3 RGB x 2x2 patch)
- patch_size=2
- scale_factor_spatial=16
- is_residual=True
- 48-vector latents_mean / latents_std (required for the model's
  encode/decode normalisation to produce non-garbage outputs)

Loading the TI2V-5B VAE state_dict into the default-constructed model
failed with shape mismatches throughout the encoder + decoder, surfaced
in wan_l2v as "Error(s) in loading state_dict for AutoencoderKLWan."

This commit routes z_dim=48 to a verbatim copy of the TI2V-5B VAE config
(from vae/config.json in Wan-AI/Wan2.2-TI2V-5B-Diffusers); z_dim=16
keeps the previous A14B / Wan 2.1 default behaviour.

Verified end-to-end: both kwargs construct cleanly and produce the
expected layer shapes (decoder.conv_out emits 12 channels for TI2V-5B,
3 channels for A14B).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(wan): variant-aware default scheduler for standalone installs

When the main model has no on-disk ``scheduler/`` directory (every
standalone GGUF / single-file install), ``_build_scheduler`` previously
fell back to ``FlowMatchEulerDiscreteScheduler()`` for every variant.
That's correct for the A14B family but wrong for TI2V-5B, which ships
``UniPCMultistepScheduler`` with ``flow_shift=5.0`` +
``prediction_type="flow_prediction"`` + ``use_flow_sigmas=True``. The
mismatch produces drifty samples on TI2V-5B.

Add a ``_default_scheduler_for_variant`` helper that reconstructs the
right scheduler from the variant tag (values verbatim from each
variant's ``scheduler/scheduler_config.json`` in the matching
Wan-AI/Wan2.2-*-Diffusers repo). The on-disk-config-present path is
unchanged — if the model ships a scheduler dir, that wins.

Full scheduler-selection UI is deferred to a future PR per discussion;
this special-case keeps the standalone TI2V-5B path producing the right
sampler without surfacing a new field.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(wan): TI2V-5B image-to-video support

TI2V-5B I2V uses a fundamentally different conditioning scheme from
A14B I2V. Implement diffusers' ``expand_timesteps`` path so the same
``Reference Image - Wan 2.2`` node and ``Denoise Video - Wan 2.2`` node
work for both variants, dispatched by VAE z_dim / transformer variant.

Encoder side (wan_ref_image_extension.py / wan_ref_image_encoder.py)
- Add ``encode_reference_image_to_ti2v_condition`` that VAE-encodes a
  single image frame to ``[1, 48, 1, H/16, W/16]`` with the Wan2.2-VAE
  normalisation, no mask channels.
- ``WanRefImageEncoderInvocation`` dispatches on ``vae.config.z_dim``:
  z_dim=48 → TI2V-5B path, z_dim=16 → existing A14B path.
- Enforce ``multiple_of=32`` for width/height in the TI2V-5B case
  (16x VAE * 2 transformer patch = pixel dims must divide by 32) with
  a clear error message pointing at the constraint.

Denoise side (wan_video_denoise.py)
- Replace the "TI2V-5B I2V not supported" raise with a variant-aware
  dispatch on ``ref_condition.shape`` and ``variant``.
- For TI2V-5B I2V build a ``first_frame_mask`` once (0 at frame 0, 1
  elsewhere). At each step:
    latent_model_input = (1 - mask) * condition + mask * latents
    temp_ts            = (mask[0,0,:,::2,::2] * t).flatten()
    timestep           = temp_ts.unsqueeze(0).expand(B, -1)
  Per-token timesteps gate the model: frame 0 sees t=0 (locked to
  condition), other frames see t (normal denoise).
- After the denoise loop, re-clamp frame 0 to the clean condition so
  the locked first frame doesn't show scheduler drift in the final VAE
  decode. Mirrors WanImageToVideoPipeline:813-814.
- Skip the encoder-num_frames-must-match check for TI2V-5B (its
  condition is always single-frame regardless of output length).

Tests
- Three new tests on encode_reference_image_to_ti2v_condition covering
  output shape at small and Wan-realistic dims plus the no-mask-channels
  invariant. Full video-denoise integration tests would need a new
  fixture stack (none exist for wan_video_denoise yet) — deferred.

A14B I2V is unchanged. TI2V-5B T2V (added in the previous commit) is
unchanged. Verified at the import + encoder-shape level; end-to-end
verification requires a TI2V-5B I2V workflow.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(videos): show denoise progress overlay over the video viewer

CurrentVideoPreview rendered only the <video> element, so when the
last-selected gallery item was a video, a freshly-started render's
denoise preview images had nowhere to display — the user saw the
static first-frame still of the previously-loaded video until the new
render's final video swapped in.

Mirror CurrentImagePreview's progress-overlay pattern: subscribe to
$progressImage / $progressEvent, gate on selectShouldShowProgressInViewer,
and render a ProgressImage stack on top of the video when a render is
in progress. Hide the play-button overlay while progress is showing so
it doesn't sit on top of the preview.

Reported by Lincoln during TI2V-5B testing: previews started working
after restarting the server only because there was no video loaded at
that point; once a video was selected, the previews silently dropped.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(lint): apply ruff format + isort across recent Wan video work

ruff check found one I001 (import order) in
``invokeai/backend/model_manager/load/model_loaders/vae.py`` and ruff
format flagged five files. All cosmetic; no behaviour changes.

- vae.py: import reorder
- video_concat.py: minor reflow
- test_wan_ideal_dimensions.py / test_boards_multiuser.py /
  test_videos_multiuser.py: prettier-style wrapping

Verified: full ruff check + ruff format --check clean, 141 backend
tests pass, and ``pnpm lint`` (knip + dpdm + eslint + prettier + tsc)
all green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs(features): add user guide for Wan 2.2 video generation

Comprehensive guide covering:
- The three Wan 2.2 variants (T2V-A14B, I2V-A14B, TI2V-5B), their
  conditioning differences, and the dual-expert MoE explanation
- Lightning LoRA distillation for 4-step A14B inference
- Starter bundles (Text-to-Video and Image-to-Video splits)
- Workflow setup for T2V and I2V with the constraint matrix:
  * frame count: (num_frames - 1) % 4 == 0
  * pixel dims: multiple of 16 for A14B, 32 for TI2V-5B
  * encoder + denoise must agree on width/height
- The chain-and-concat trick for making longer videos, with the
  bridge-frame degradation mitigations
- Troubleshooting: OOM, late-frame artifacts, dim mismatches, VAE
  load errors, scheduler issues, preview-not-appearing, MP4 glitches

Lands under Features → Video Generation (experimental). Astro
auto-generates the sidebar from features/ so no nav config change
needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(nodes): skip image-DTO fetch for videos in Current Image node

CurrentImageNode unconditionally called useImageDTO(lastSelectedItem)
even when the selected gallery item was a video, firing
GET /api/v1/images/i/<uuid>.mp4 on every video thumbnail click. The
endpoint 404s and the backend logged "Image record not found" each
time — benign but noisy.

Apply the same null-skip pattern useGalleryItemDTO uses: pass the name
only when it's not a video, so RTK Query skips the request for video
selections. Current Image is image-only by design, so videos rendering
the empty fallback matches existing behaviour.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(videos): clear stale progress overlay + force first-frame paint

Two viewer bugs after auto-switching to a freshly-rendered video:

- The denoise progress overlay never cleared. CurrentImagePreview clears
  the ImageViewerContext $progressImage/$progressEvent atoms via DndImage's
  onLoad callback; the video viewer had no analog, so the last progress
  still sat on top of the new video forever — clicking other video
  thumbnails did nothing visible, and only selecting an image (which
  fires onLoadImage via DndImage) cleared it.
- Even with the overlay gone, the <video> element rendered its black
  background instead of the first frame. preload="metadata" loads
  dimensions/duration but doesn't guarantee a decoded first frame on all
  browsers; an explicit seek is needed to force a paint.

Wire onLoadedMetadata to (1) call onLoadImage() — mirroring DndImage's
onLoad — and (2) nudge currentTime to 0.0001 so the decoder paints the
first frame without measurably advancing playback.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(hotkeys): skip image-DTO fetch for videos in GlobalImageHotkeys

Companion to a3bdc3304e (CurrentImageNode). GlobalImageHotkeys is a
mounted-everywhere singleton that wires recall hotkeys (seed, prompts,
remix, etc.) to whatever item is currently selected. It was passing the
raw selection name through to useImageDTO unconditionally, so every video
thumbnail click fired GET /api/v1/images/i/<uuid>.mp4 → 404 and the
"Image record not found" log line.

Gate on isVideoName(), mirroring the polymorphic null-skip pattern in
useGalleryItemDTO. Recall hotkeys don't apply to videos anyway, so this
just suppresses the noise.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(wan): empty CUDA cache between A14B expert swaps

The dual-expert swapper releases the active expert via its context
manager exit, but PyTorch's caching allocator retains the freed blocks
as reserved-not-yet-claimable space until empty_cache runs. The next
partial_load_to_vram for the incoming expert then sees a fragmented
free pool and offloads layers it could otherwise have kept on device.

Users running A14B observed the low-noise expert ending up far more
CPU-resident than the high-noise one on otherwise identical settings —
that was the leftover reservation from the high-noise expert masking
real free VRAM.

Call TorchDevice.empty_cache() between the release and the next load.
Same pattern as the VAE-encode fix earlier in this branch.

Regression test in test_wan_expert_swapper.py mocks empty_cache and
asserts it fires on every actual swap but not on a same-label re-get.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(videos): allow drag-and-drop to change a video's board

Dropping a video thumbnail onto a board in the boards list was a no-op
(the dnd target only accepted image sources). Extend
addImageToBoardDndTarget and removeImageFromBoardDndTarget to also accept
SingleVideoDndSourceData and dispatch the corresponding video mutations.

Permission UX mirrors the image path:
- Same canMoveFromSourceBoard gate (owner / public source board)
- Same "do nothing if dropping on the current board" early-out

Backend enforcement on /api/v1/videos/board already mirrors the image
endpoints — _assert_board_write_access on the destination plus
_assert_video_direct_owner on the video. The frontend gate intentionally
mirrors only the source-board part of that, leaving the direct-owner
check to surface as a 403 on attempt (same compromise as images, where
the client doesn't have per-item owner info to gate cleanly).

Multi-video drag is not supported yet (the gallery only registers a
single-video draggable per item, no multi-select bundle), so this only
wires the SingleVideoDndSourceData path.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(wan): force outgoing A14B expert off GPU on swap

The previous empty_cache() fix (53b2f4d4c7) was insufficient. unlock()
only decrements the cache record's lock counter — the weights stay on
GPU until the cache's automatic offload decides to free them on the
next lock(). That heuristic uses
``torch.cuda.memory_allocated() - working_mem`` to estimate free space,
which under-frees when the previous denoise step's workspace
activations are still allocated alongside the just-unlocked expert.
The user-visible symptom was a log line like

  Loaded model '...:transformer' onto cuda device in 0.37s.
  Total model size: 9203.13MB, VRAM: 2381.18MB (25.9%)

for the incoming low-noise expert, while the high-noise expert
continued to hold ~9 GB of VRAM.

The swapper now stashes the LoadedModel info handle and, on each swap,
explicitly invokes ``cached_model.full_unload_from_vram()`` on the
outgoing expert before locking the incoming one. This sidesteps the
heuristic and guarantees the previous expert's weights leave GPU
before partial_load_to_vram measures available room.

The access path ``info._cache_record.cached_model`` reaches into a
private attribute — there is no public LoadedModel API for "unload
from VRAM but keep in RAM" today, and a broader backend refactor felt
out of scope. The call is wrapped in getattr/try-except and pinned by
a regression test so a future refactor breaks the test, not the swap.

Tests:
- Updated existing dual-expert lifecycle test to expect the new
  full-unload step in the swap log sequence.
- New test_outgoing_expert_force_unloaded_from_vram covers the
  per-swap behavior (outgoing only, no initial unload).
- New test_force_unload_failure_does_not_break_swap pins the
  defensive fallback so swap reliability survives a future LoadedModel
  refactor.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(gallery): restore shift/ctrl-click range selection in image grid

GalleryImage's modifier-key click handler was reading the legacy
imagesApi getImageNames cache to compute range-selection indices, but
the gallery grid was switched to the polymorphic galleryApi
getGalleryItemNames endpoint (the only source that includes videos).
The legacy cache is no longer populated for the grid, so the
ordered-name list came back empty and the handler fell into its
"no names cached" early-return:

  if (imageNames.length === 0) {
      if (!shiftKey && !ctrlKey && !metaKey && !altKey) {
          dispatch(selectionChanged([imageName]));
      }
      return;
  }

making shift- and ctrl-click no-ops.

GalleryVideoItem already had the correct reader inlined as a private
helper. Hoist it to a shared module
(features/gallery/store/selectCachedGalleryItemNames) so both grids
use the polymorphic cache, and update GalleryImage to call it.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(typegen): regenerate schema.ts

Refresh of the OpenAPI-derived TypeScript bindings against the current
backend. No hand edits — this is the output of the typegen step
re-run against the Wan video routes and recent backend changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(startup): silence HF tokenizers fork-after-parallelism warning

Set TOKENIZERS_PARALLELISM=false at startup (via os.environ.setdefault
so users can override) before any HF library is imported.

The Rust ``tokenizers`` library warms a thread pool the first time a
tokenizer runs — for us that's UMT5 / T5 text encoding during Wan /
FLUX / SD3 conditioning. Every subsequent fork() then logs

  huggingface/tokenizers: The current process just got forked, after
  parallelism has already been used. Disabling parallelism to avoid
  deadlocks...

In video generation we fork on every MP4 encode (imageio's FFMPEG
plugin uses subprocess.Popen → fork+exec), so this warning lands once
per generation in the server log. The advisory is benign — the child
correctly falls back to single-threaded tokenization before exec(),
and the parent's thread pool is unaffected — but the noise obscures
real warnings.

Setting the env var before any HF import prevents the thread pool
from warming up at all, so the fork detector stays quiet without
sacrificing anything: tokenization happens once per generation and
isn't a hot path for us.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(startup): hoist TOKENIZERS_PARALLELISM=false to module level

Follow-up to 2106f10ec4 — the previous attempt set the env var inside
``run_app()``, which races against any transitive HF import triggered
by the console-script's

  from invokeai.app.run_app import run_app

If ``tokenizers`` is imported anywhere in that import chain (directly
or via diffusers/transformers re-exports), the library's fork
detector registers before our setdefault runs and the warning still
fires.

Move the setdefault to module level so it executes the instant
``run_app.py`` is loaded — i.e. before the function defs are even
parsed, and well before any HF library has a chance to import.

Note for testing: jurigged hot-reload only re-runs function bodies,
so picking up this fix requires a full server restart, not just a
file save under ``--dev-reload``.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(videos): replace window.confirm with ConfirmationAlertDialog

Use the in-app delete-confirmation dialog (the same Chakra
ConfirmationAlertDialog the image flow uses) instead of the browser's
window.confirm() prompt. Matches the visual + interaction language of
the rest of the gallery and picks up the shared
``shouldConfirmOnDelete`` system preference — flipping the "Don't ask
me again" toggle now silences the prompt for both images and videos.

Implementation mirrors features/deleteImageModal/ but trimmed: the
image dialog computes "usage" (canvas layers, node fields, reference
images, upscale source) so the user knows what they'll break. Videos
have no analogous attachment points, so the video state machine is a
straight confirm-then-delete with no usage analysis.

- features/deleteVideoModal/store/state.ts — nanostores atom + an
  awaitable ``deleteVideosWithDialog`` that opens the dialog and
  resolves/rejects on confirm/cancel. Skips the dialog entirely when
  shouldConfirmOnDelete is off.
- features/deleteVideoModal/components/DeleteVideoModal.tsx —
  ConfirmationAlertDialog with the new deleteVideoPermanent message
  and the shared "Don't ask me again" switch.
- GlobalModalIsolator.tsx — mount the new modal alongside
  DeleteImageModal.
- ContextMenuItemDeleteVideo.tsx — call useDeleteVideoModalApi().delete
  instead of window.confirm + useDeleteVideoMutation.
- en.json — added gallery.deleteVideoPermanent, dropped the now-unused
  gallery.deleteVideoConfirmation.
- videos.ts — useDeleteVideoMutation moves into the @knipignore export
  group since the only call site now uses
  videosApi.endpoints.deleteVideo.initiate via the modal.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(gallery): refetch polymorphic gallery cache on image completion

The gallery grid subscribes to the polymorphic ``getGalleryItemNames``
RTK Query endpoint (so images and videos interleave by created_at). But
``onInvocationComplete``'s image path only did an optimistic insert
into the image-only ``getImageNames`` cache, leaving the polymorphic
cache stale — a freshly-generated image landed correctly in board
totals and the per-DTO cache, but never showed up in the grid until
the user reloaded the page.

Mirror the videos path (which has invalidated these tags since the
polymorphic endpoint was introduced) and dispatch
``galleryApi.util.invalidateTags(['GalleryItemNameList',
'GalleryItemList'])`` after image outputs are processed. The cost is
one extra HTTP round-trip per generation; a future optimization could
optimistically splice the new entry into the polymorphic shape, but
that requires a different ``insertImageIntoNamesResult`` for the
``GetGalleryItemNamesResult`` shape and is a bigger change.

Regression test in onInvocationComplete.test.ts pins the behavior:
verifies the invalidation fires on a fake image complete event, and
verifies it does NOT fire for denylisted passthrough node types
(load_image, image). Confirmed test correctly fails when the fix is
reverted via git stash.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* review: address 2nd-pass code review findings

Self-review pass before re-pinging external reviewers. Five fixes; the
three medium ones have user-visible consequences, the two low ones are
guard + docstring.

1. videos.py: delete_video no longer swallows service errors into a
   misleading HTTP 200. Missing DTO -> 404, delete failure -> 500. The
   prior shape returned 200 with an empty deleted_videos list, which
   the frontend treated as success, dropped from cache, and left the
   video on disk — silent data-consistency failure visible only on
   next page reload.

2. videos.ts: starVideos / unstarVideos invalidate the LIST_TAG-scoped
   { type: 'VideoList' } entry alongside the per-video and
   board-affecting tags. Without this, starred_first=true gallery
   queries kept the just-starred video in its old position until the
   next list-affecting mutation. Mirrors the delete + upload pattern.

3. wan_denoise.py: _ExpertSwapper.get() stashes _active_device_ctx
   right after device_ctx.__enter__() succeeds, before attempting the
   LoRA patcher's __enter__. If the LoRA enter raises, _release() can
   now actually find the device context and exit it — previously the
   ctx was unreachable and 8-9 GB of GGUF expert weights stayed pinned
   to GPU until the model cache LRU evicted them.

4. wan_ideal_dimensions.py: reject sources whose longer side is below
   the 16-px Wan grid. The downstream max(w, 16) clamp would otherwise
   silently disconnect the output from the requested aspect ratio
   (returning 16×16 regardless of the source's actual shape).

6. wan_video_denoise.py: docstring now explains the deliberate absence
   of denoising_start / denoising_end / initial-latents inputs (video
   i2v uses reference-frame conditioning, not noise injection; the
   image denoise node still handles still-image img2img).

Tests:
- test_device_context_released_when_lora_enter_raises pins #3.
- test_input_smaller_than_pixel_grid_rejected pins #4.
- test_output_dims_never_zero renamed to
  test_smallest_valid_input_still_snaps_to_16_grid (now exercises 16×16
  rather than 8×8 since the latter is now correctly rejected).

All 58 affected backend tests pass, frontend lint clean.

Audit note for the PR description (NOT a fix): delete_video's
_assert_video_owner permits write access on public boards (mirroring
the image router's _assert_image_owner — intentional symmetry). The
stricter _assert_video_direct_owner is reserved for board-move ops.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(frontend): typegen

* fix(gallery): multi-select context menu actions for videos

The video gallery context menu only operated on the single right-clicked
item, so selecting multiple videos and hitting the trash icon deleted just
the first one. Adds a video-side multi-selection menu mirroring the image
one for star/unstar/download/change-board/delete, switched in on
selectionCount > 1.

Each menu now filters the polymorphic selection to its own kind and labels
the action with an explicit count + kind (e.g. "Delete 3 Videos", "Move 2
Images to Board"). The destructive items disable when the kind-filtered
subset is empty, so a video-only selection greys out the image menu and
vice versa.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(multiuser): address Pfannkuchensack PR #9163 review findings

Finding 1 (Medium): delete_board cascade ignored per-video / per-image
ownership, letting a board owner destroy other users' contributions to a
public/shared board just by deleting the board with include_images=true.
Adds user_id filtering through get_all_board_*_names_for_board and
delete_*_on_board (base + sqlite + image wrapper). Non-admin requests pass
the requester's id so the SQL WHERE clause narrows the cascade to that
user's rows; admins still pass None for the unrestricted path. Other users'
content cascades to "uncategorized" via the existing FK on board_videos /
board_images.

Finding 2 (Low, i18n): GalleryItemStarIconButton and
GalleryItemVideoStarIconButton shipped raw English "Star"/"Unstar" tooltips.
Both now use the gallery.starImage / starVideo translation keys.

Finding 3 (Low): delete_videos_from_list and delete_images_from_list
re-raised HTTPException mid-loop, throwing away the response payload for
items already deleted before the foreign name was hit. The frontend cache
never learned about those partial successes, so deleted records reappeared
in the UI until the next manual refresh. Both routes now skip auth-failed
items in-loop and return 200 with the partial-success list.

Residual: adds a test that an upload with an .mp4 extension but
non-decodable bytes (a) reaches probe_video, (b) surfaces 415, (c) unlinks
the streamed-to-disk temp file so the server doesn't leak storage on
garbage uploads.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(openapi): regenerate openapi.json

The committed schema was stale relative to the current server (missing the
utilities/expand-prompt and utilities/image-to-prompt endpoints, the
ModelRecordOrderBy / SQLiteDirection list params, and the Wan / QwenImage /
QwenVLEncoder config variants this branch adds). Regenerated via the same
command the new openapi-checks workflow uses so the diff CI is empty.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(workflows): add "Image to Video - Add Frames" starter workflow

Extends an existing video by extracting its penultimate frame, running it
through Wan 2.2 I2V A14B + the Lightning LoRA pair to generate a new clip,
and concatenating the result onto the source with a short crossfade.

Cleaned per the default-workflows README: stripped value references on the
four model loader fields and both Lightning LoRA fields so the workflow
ships without keys/hashes for user-installed resources, gave the LoRA nodes
"Apply LoRA (High)" / "(Low)" labels matching the existing Lightning
default, remapped six stale exposedFields entries that pointed to template
LoRA IDs no longer present in the graph, and synced the wan_video_denoise
num_frames default to the value driven by the connected integer node.
Tagged with both Text to Video and Image to Video so it surfaces under
either filter in the Workflow Library.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(gallery): video viewer polish and selection regressions

- Restore auto-select-on-startup and on-board-switch: the polymorphic
  getGalleryItemNames endpoint replaced getImageNames as the grid's
  source of truth, so appStarted and boardIdSelected now wait on / read
  that cache instead of timing out forever.
- Delete-then-select: video delete used to clear selection to null;
  image delete read a cache that's no longer warmed. Both now snapshot
  the gallery list before deletion and advance to the adjacent surviving
  item (prev > next > null) via a shared pickSelectionAfterDelete helper.
- Video Viewer: right-aligned action bar with Open in new tab, Copy
  frame, Download, Delete, and a labelled Close video player button that
  only appears while playback is active. Copy uses canvas + ClipboardItem
  since video MIME types aren't supported cross-browser.
- Next/prev arrows + galleryNav hotkeys now work when a video is in the
  Viewer (previously image-only).
- Video context menu uses full-width text MenuItems instead of the
  cramped icon group, and gains an Open in new tab entry.

* fix(gallery): bulk video drag-to-board and shift-click range selection

- Bulk video drag: introduced multipleVideoDndSource so a multi-selection
  dragged from a video thumbnail moves every selected video, not just the
  first. The whitelist in useDndMonitor.ts also needed updating — without
  it the monitor's canMonitor gate silently dropped the new source type.
- Mixed selections: both the multi-image and multi-video drag payloads now
  carry image_names + video_names side-by-side, so dragging from either
  kind in a mixed selection dispatches addImagesToBoard + addVideosToBoard
  together. Previously the image side leaked video names into image_names
  and the image router 404'd on each one.
- Bulk video helpers: added addVideosToBoard / removeVideosFromBoard that
  fan out over the existing singular video router endpoint (no batch
  endpoint exists yet) — mirrors the change-board modal's existing loop.
- Shift-click range selection: selectCachedGalleryItemNames now looks up
  the cache entry matching the gallery's current query args instead of
  taking the first entry from selectInvalidatedBy. RTK Query keeps unused
  entries warm for 60s after a board switch, and the old "first wins"
  behavior frequently landed on a stale board's name list, making
  shift-click silently no-op until a delete/move forced a refetch.

* fix(scripts): force generate_openapi_schema.py to resolve invokeai from the repo root

When the script was invoked as ``python scripts/generate_openapi_schema.py``,
Python placed the script's directory at ``sys.path[0]`` rather than the repo
root. ``import invokeai`` then resolved via the venv's site-packages, which on
multi-worktree editable installs ends up importing ``invokeai`` as a PEP 420
namespace package that aggregates every worktree's ``invokeai/`` directory.
Side-effect imports driven by submodule discovery silently miss whichever
worktree isn't first on the namespace path, so the registry came up short by
the invocations declared only in this worktree (the wan/video set, 15
classes). Running the same imports via ``python -c`` worked because
``sys.path[0]`` defaulted to the cwd and ``invokeai/__init__.py`` resolved
cleanly to the worktree.

Prepend the resolved repo root to ``sys.path`` before importing ``invokeai.*``
so the script always picks up the local sources regardless of how it was
launched.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(video): add Frame Range from Video invocation with scrubbable preview

New ``extract_video_range`` node trims a source video to a contiguous frame
range and re-encodes it as MP4, slotting in naturally between a generated
clip and Concatenate Videos for I2V chain shaping. Bounds are inclusive and
support negative indices (``end_frame=-1`` keeps the final frame), matching
Frame from Video. Output fps inherits from the input unless overridden.

The node renders a per-type preview inside the workflow editor: two
``<video>`` tiles side by side, each driven by a CompositeSlider that scrubs
the corresponding integer field. The tile uses ``currentTime = frame / fps``
so browsers display the seeked frame natively without a canvas roundtrip.
Negative-index entries in the standard integer input are resolved against
the source frame count for display only; the underlying field value is
preserved verbatim.

The custom UI is wired in via a ``CustomNodeBody`` dispatcher in
``InvocationNode.tsx`` rather than a registry — small enough to be explicit.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(video): emit resolved frame indices and move preview to per-field renderer

Three changes to the ``extract_video_range`` invocation:

1. New ``ExtractVideoRangeOutput`` mirrors ``VideoOutput`` and additionally
   emits the resolved (positive, 0-based) ``start_frame`` and ``end_frame``
   indices. Chained workflows can feed those back into a downstream
   Frame from Video to extract the same boundary frame the trim landed on.

2. ``fps`` is now a plain ``int`` defaulting to 16 (was ``Optional[int]``
   with an "inherit from input" fallback). Matches the default used by
   wan_l2v and the other Wan video producers, so chained workflows agree
   on framerate without each node guessing.

3. The frame preview is now a per-field widget driven by a new
   ``UIComponent.VideoFrameIndex`` hint. ``start_frame`` and ``end_frame``
   are tagged with it; the new ``VideoFrameIndexFieldInput`` renders a
   number input plus a live <video> thumbnail and a scrubber slider, all
   writing to the same Redux field. Negative indices entered in the
   number input are still resolved against the source frame count for
   display only — the backend re-resolves at invoke time.

The widget reads its companion ``VideoField`` (by convention, the sibling
field named ``video`` on the same node) via direct Redux selectors, so it
works wherever ``InputFieldRenderer`` is used — the workflow editor's node
body AND the Form Builder's view/edit modes. The previous node-body
``ExtractVideoRangePreview`` and its ``CustomNodeBody`` dispatcher in
``InvocationNode.tsx`` are removed; the per-field widget supersedes both.

In the workflow editor, side-by-side framing is lost in exchange for
Form Builder support; users wanting the side-by-side layout in a form can
group the two frame fields in a row container.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore: ruff

* fix(video): address PR #9163 review follow-ups

- delete_board: include_images query description and OpenAPI schema now mention
  videos alongside images
- get_video_thumbnail: check path existence before returning FileResponse so a
  missing thumbnail produces the documented 404 instead of an after-route error
- delete_videos_on_board: stop deleting records for videos whose files failed to
  delete, so a transient FS error no longer orphans the file with no record
  pointing at it
- DeleteBoardModal: destructive button and warning copy now mention videos

* fix(video): address PR #9163 May-22 review and failing CI

- remove_video_from_board now accepts either the direct video owner or a
  board write-access holder, so videos uploaded to a board that later
  flipped Public -> Shared/Private aren't stranded.
- VideoService.create rolls back the DB record and board association if
  the underlying file save fails, preventing ghost records whose file
  endpoints 404.
- delete_videos_on_board returns the actually-deleted names; delete_board
  uses that list so the response can't claim a video was destroyed when
  its record was preserved due to a file-delete failure.
- Local test_videos_multiuser fixture now patches
  invokeai.app.api.routers._access so list/names route 403 checks work.
- Regenerate schema.ts to pick up the CacheStats description.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(frontend): rebuild openapi

* fix(video): register Viewer <video> as drag source

Drag-and-drop from the Viewer pane now produces the same singleVideoDndSource
(and multipleVideoDndSource for active multi-selection) as the gallery
thumbnail, so a video can be dropped onto a Video Primitive's "Starting Video"
field directly from the Viewer.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(video): add frame preview to Frame from Video node

Tag frame_index with ui_component=VideoFrameIndex so the node renders the
same live frame thumbnail + scrubber as Frame Range from Video. The widget
keys off the sibling 'video' field, which this node already has, so no
frontend changes are needed. Bump node version 1.0.0 -> 1.1.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* feat(video): derive Frame Range fps from source video by default

Make the fps field optional (default None). When unset, the output frame
rate is inherited from the probed source video so a trimmed clip plays back
at the same speed as its source, falling back to 16 fps when the source rate
can't be probed. An explicit fps still overrides. Bump node version
1.0.0 -> 1.1.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix(video): use fps=0 sentinel for source-derived Frame Range rate

The previous Optional[int]/None design had no natural way to express
'unset' in the node's number input, and the ge=1 constraint rejected the
intuitive fps=0 with a validation error. Make fps a plain int defaulting to
0, allow ge=0, and treat 0 as 'match the source video's frame rate'. Keeps
in-progress workflows (already saved with fps=0) working without a version
bump.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* feat(wan): add Wan 2.2 TI2V Ideal Dimensions node

TI2V-5B uses the 16x Wan 2.2-VAE plus a 2x transformer patch, so pixel dims
must be multiples of 32 (the existing I2V node snaps to 16, which the TI2V-5B
patchify step rejects). Add a wan_ti2v_ideal_dimensions node that snaps to 32.

Factor the shared scale-and-snap math into _scale_and_snap(multiple=...) so
both nodes derive from one implementation; the I2V node is unchanged
behaviorally (its existing tests still pass). Add a mirrored TI2V test suite.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* docs(wan): add A14B/5B model hints to ideal-dimensions node titles

Suffix the node titles with the target model family (A14B / 5B) so they're
distinguishable in the add-node search and node header, and rewrite both
docstrings to lead with which Wan 2.2 model they're for and cross-reference
the other node. Purely UI metadata — no behavior or schema change.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* feat(workflows): replace bundled Wan 2.2 video workflows with curated set

Remove the 6 previously-bundled Wan 2.2 *video* default workflows (Text to
Video, Text to Video Lightning x2, Image to Video, Image to Video Lightning,
Image to Video - Add Frames) and replace them with the 8 curated starter
workflows: Text/Image to Video Lightning (+ Concept LoRA variants), Extend
Video Lightning (+ Concept LoRA variant), and the TI2V-5B text/image-to-video
low-quality variants.

Each is assigned a stable default_ id and meta.category=default. Model fields
are intentionally blanked (per-install keys don't resolve cross-instance) with
the required models listed in each workflow's Notes. The two Wan 2.2 *image*
workflows (Image to Image, Text to Image) are retained.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* docs(video): add beginner Video Workflows guide for the 8 starter workflows

New Features-section page (sibling to Video Generation) describing the eight
bundled Wan 2.2 video workflows in plain language: how to choose between the
Text/Image/Extend families and their Lightning / Concept-LoRA / TI2V-5B
variants, how to select models from each workflow's Notes, how to run one, and
a quick per-GPU guide. Cross-linked both ways with the Video Generation
technical reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* docs(video): fix Concept LoRA slot guidance (slots are required)

The w/ Concept LoRAs workflows wire required lora fields (lora_selector /
wan_lora_loader, no default) into the graph, so an empty slot blocks
invocation. Correct the earlier claim that empty concept slots behave like the
base workflow: every LoRA slot must be filled, and users without concept LoRAs
should use the plain variant.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* docs(video): drop '(experimental)' from Video Generation title

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* chore: refresh uv.lock

Routine lock refresh (transitive dev deps: docutils, idna, platformdirs,
python_discovery, tornado).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* feat(wan): add first-last-frame interpolation (FLF2V) to I2V-A14B

The Reference Image - Wan 2.2 node gains an optional End Image input: when set,
encode_reference_image_to_video_condition places the end image in the final
temporal slot and anchors the mask at both the first and last latent frames, so
I2V-A14B interpolates a clip from the start image to the end image. Mirrors
diffusers WanImageToVideoPipeline.prepare_latents with last_image set.

The denoise loop is unchanged - for A14B it just concatenates the 20-channel
condition, which is agnostic to one vs two anchors. FLF2V is A14B video only
(num_frames > 1); the encoder raises a clear error for TI2V-5B or single-frame.
Bump wan_ref_image_encoder to 1.2.0; add mask-anchoring unit tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* feat(workflows): add 'Interpolate 2 Images to Video' starter + FLF2V docs

Ship a default workflow that wires the new FLF2V End Image input end to end
(I2V-A14B + Lightning, two image inputs interpolated). Model fields blanked with
the required models listed in Notes, default_ id + category=default. Document
FLF2V in the Video Generation reference and add the workflow to the Video
Workflows guide.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* feat(workflows): add Text/Image/Video to Video library filters + fix video tags

Add 'Text to Video', 'Image to Video', and 'Video to Video' to the Common Tasks
filter list in the Workflow Library browser. Fix the tags on the nine bundled
Wan 2.2 video workflows, which were all copy-pasted as 'text to video':
- Text to Video: the three T2V workflows
- Image to Video: the I2V workflows + Interpolate (two-image)
- Video to Video: the two Extend Video workflows
The TI2V-5B variants also drop the spurious lightning/lora tags (they have no LoRAs).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* feat(workflows): add 'Extend Video to Image' FLF2V starter + docs

Ship a default workflow that extends a video toward a user-provided target
image: the new segment interpolates (FLF2V) from the source video's last frame
to the destination image, then concatenates onto the original with a cross-fade.
Model fields blanked, default_ id + category=default, tagged 'video to video'.
Document it (card + usage instructions) in the Video Workflows guide.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* docs(video): reorganize Video Workflows guide sections

Group the Interpolate section and the Concept-LoRA / TI2V-5B asides with the
image workflows, keep the Extend family (including Extend Video to Image) at the
end, and retitle the section to 'Bundled video workflows'.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* chore(frontend): regenerate openapi and typegen

* chore(backend): ruff

* fix(future): make the WAN LoRA loader compatible with LoRA picker node PR #9259

* chore(frontend): remove unused selectT5EncoderModels import

* chore(frontend): remove unused export

* fix(gallery): count videos and pick video covers for board tiles

Gallery boards previously joined only the `images` table for their
headline count and cover thumbnail, so a board containing nothing but
videos rendered as empty with no preview. BoardDTO now exposes
`video_count` and an optional `cover_video_name`; the boards service
picks the best cover across both tables using the same
(starred DESC, created_at DESC) tie-break the image path already used,
and the gallery list renders `image_count + video_count` everywhere it
previously rendered just images (real boards, no-board pseudo-board,
and the tooltip). Adds `getBoardVideosTotal` to round out the no-board
counts (the BoardVideosTotal tag was already wired into invalidation).

* test(boards): wire video record storage into multiuser test fixtures

After the board cover/count fix started reading from `video_records` and
`board_video_records`, the multiuser test fixtures that still set both
to `None` started erroring out — the boards router's catch-all turned the
AttributeError into a 404, cascading through every test that PATCHes or
GETs a board (auth, workflows, data-isolation suites). Swap the `None`
placeholders for real SqliteVideoRecordStorage / SqliteBoardVideoRecordStorage
instances (paralleling the existing image storage setup), and pin sane
defaults on the MagicMocks in `test_videos_multiuser.py` so the
get_dto cover/count lookups don't trip Pydantic validation.

* fix(ui): widen useVideoContextMenu ref type to allow null

The ref param was typed RefObject<HTMLElement>, but useRef produces
RefObject<HTMLElement | null>, breaking lint:tsc in GalleryVideoItem.
Match the sibling useImageContextMenu signature.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* chore(backend): ruff

* fix(tests): pass video/gallery services to InvocationServices in workflow-call router tests

The workflow-call router tests from main construct InvocationServices
directly and predate the video/gallery services added on this branch, so
every test in the file errored with missing positional arguments. Mirror
tests/conftest.py: real sqlite stores for video_records and
board_video_records, None for the services the tests never touch.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): refresh board caches when a generated video completes

Video completion previously invalidated only the polymorphic gallery
list tags, so the new video appeared in the grid while the board's
video_count, cover thumbnail (Board tag / listAllBoards), and
BoardVideosTotal stayed stale until an unrelated mutation refetched
them. Use the shared getTagsToInvalidateForBoardAffectingMutation
helper over the affected boards, matching the video mutation endpoints.

Reported by @JPPhoto in PR #9163 review.

Co-Authored-By: Claude Fable 5 <[email protected]>

* feat: include videos in date-based virtual boards

Date virtual boards were image-only even though the gallery grid is now
polymorphic: video-only dates never appeared, and mixed dates omitted videos
from counts/contents/covers.

- SqliteGalleryService owns virtual-board dates now: get_dates() unions
  images+videos per date (video_count added to VirtualSubBoardDTO, cover is
  the newest item of either kind via cover_image_name/cover_video_name), and
  list_item_names() gained a created_date filter.
- New GET /api/v1/virtual_boards/by_date/{date}/item_names returns the same
  polymorphic (kind, name) refs as the gallery names endpoint; the legacy
  image_names route is kept for API compatibility.
- Frontend virtual-board selection consumes the new endpoint, so videos show
  up in virtual date boards; VirtualBoardItem shows video counts (localized
  tooltip) and falls back to the video thumbnail for video covers.
- tests/conftest.py wires a real SqliteGalleryService so router tests exercise
  the filter SQL; service + router tests cover video-only dates, mixed dates,
  cover selection, and per-user isolation.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): don't advance gallery selection for videos whose delete failed

handleDeletions treated every requested video as deleted when picking the
post-delete selection, so a 403/500 on deleteVideo could jump the Viewer away
from a video that still exists, and a surviving neighbour was skipped as a
replacement candidate. Only successfully deleted names now count: a failed
displayed video keeps its selection, and failed neighbours remain valid
replacements. Covered by state.test.ts with rejected deleteVideo dispatches.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): count uncategorized videos when deciding the gallery has content

useHasImages only looked at boards and the uncategorized *image* total, so a
gallery whose only content was an uncategorized video rendered the
new-user/get-started view instead of the normal no-selection state. The hook
now also reads the uncategorized video total (getBoardVideosTotal('none'));
the decision logic is extracted as getHasGalleryContent and unit-tested.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): stop labeling board video counts as images in tooltips

Board tooltips folded video_count into image_count and rendered
boards.imagesWithCount, so a video-only board read e.g. '1 image, 0 assets'.
Tooltips now show split image/video/asset counts using the new
boards.videosWithCount translation; the compact unlabeled headline count in
the boards list stays combined so video-only boards don't read as empty.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): restrict client-side video upload acceptance to MP4 only

The dropzone accept map advertised .webm/.mov and isVideoFile treated
.webm/.mov/.mkv as videos, but the upload router accepts MP4 only, so
those files were accepted client-side and then rejected with 415 after
the bytes were uploaded. Consolidate the accepted-media lists into
common/util/uploadMediaAccept.ts (single source of truth shared by
useImageUploadButton and FullscreenDropzone) and pin them to the
backend contract with a regression test.

Also split the accept map: image-only upload fields (board covers,
style presets, model images, workflow thumbnails) no longer advertise
video/mp4, which they had inherited when video entries were added to
the shared map.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(api): bulk video star/unstar returns partial successes instead of 403 mid-batch

star_videos_in_list and unstar_videos_in_list re-raised the ownership
HTTPException mid-loop, so a batch containing one foreign (or stale)
name mutated the earlier owned videos and then returned 403 with no
payload — the client never invalidated caches for the videos that did
change. Skip unauthorized names and return 200 with the actually
starred/unstarred videos, mirroring delete_videos_from_list. Router
tests cover the mixed-ownership batch for both routes.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): localize virtual board section header and toggle

The 'By Date' header and the Collapse/Expand aria-label in
VirtualBoardSection were hardcoded English. Add boards.byDate and
common.collapse/common.expand translation keys and use them.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(app): failed video saves no longer orphan files on disk

DiskVideoFileStorage.save() moves the source MP4 into permanent storage before
writing the thumbnail and sidecar, so a failure in either later step used to
leave the moved MP4 (and partial artifacts) on disk with no DB record through
which they could be managed. save() now removes its destination files before
raising, and VideoService.create()'s rollback also deletes files to cover
failures after a successful file save (e.g. building the DTO).

Also documents why board attachment during create is best-effort (mirroring
ImageService.create: a board deleted mid-generation must not destroy the
render) and pins the explicit fallback — DTO reports the actual missing board
association and a warning is logged — with a service test.

Co-Authored-By: Claude Fable 5 <[email protected]>

* docs(app): document videos.user_id lifecycle and pin user-deletion behavior

videos.user_id deliberately has no FK to users, matching images/boards/
workflows (migration_27 adds those user_id columns index-only): deleting a
user leaves their media in place for admin review/cleanup rather than
cascading a row delete that would strand files on disk. A migration comment
now states the parallel, and a migration-backed test creates a user and a
video, deletes the user, and asserts the record survives, stays attributed to
the deleted owner, and remains visible only to admins.

Co-Authored-By: Claude Fable 5 <[email protected]>

* feat(app): support VideoField values in session queue batch data

Batch.data previously allowed ImageField but not VideoField, so submitting
multiple VideoField values through the generic batching capability failed
Pydantic validation before enqueueing. VideoField now joins the
BatchScalarDataType union; a test asserts a VideoField batch validates and
expands into separate sessions. schema.ts/openapi.json regenerated.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): video uploads are opt-in per consumer; upload validation fixes

- useImageUploadButton gains an allowVideos opt-in (default off). Only the
  gallery uploader accepts videos; image-only consumers (ref images, board
  covers, launchpad buttons, image-to-prompt, etc.) no longer let a selected
  MP4 upload into the gallery while the requested image action goes nowhere.
  Videos are excluded from their accept map and rejected at runtime if the
  file dialog bypasses it, with tests via partitionUploadFiles.
- The hook's loading state now covers both the image and video mutations, so
  an in-flight MP4 upload shows a loading button and blocks resubmission.
- The fullscreen drag-drop/paste validator accepts a file when either its
  MIME type or its extension is recognized — a clip.mp4 with an empty
  File.type used to be rejected even though the backend accepts it. The
  validator moved to a pure module with tests.
- Failed video uploads no longer toast "Image Upload Failed": video-only
  batches use a new toast.videoUploadFailed key, mixed batches the neutral
  toast.uploadFailed.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(app): bound untrusted video decoding with a killable subprocess timeout

probe_video / extract_video_frame / decoder_frame_count now run in a
short-lived child process (video_decode_worker.py) killed after a hard
timeout. Previously a crafted MP4 that failed the imageio probe and then
hung inside cv2.VideoCapture()/read() would pin the FastAPI request
worker that called it forever; repeated uploads could exhaust the pool.

The worker is run by file path (not -m) and imports only imageio/PIL/cv2
so it starts without pulling in the invokeai package or torch. Tests
substitute a never-returning worker command and assert the helpers fail
within a bounded interval, plus happy-path tests against a real
synthetic MP4 to validate the subprocess plumbing end to end.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(app): stream frames through the video concat/trim nodes

video_concat and extract_video_range fully decoded their inputs into
lists of uncompressed frames before encoding; with the 1 GB upload cap a
long 1080p source can expand to tens of gigabytes of RAM, so any user
able to enqueue these nodes could exhaust server memory.

Frames now stream from the decoder straight into an incremental FFMPEG
writer. The concat node buffers only the transition windows (bounded by
transition_frames), and the range node holds one frame at a time and
stops decoding at the end of the requested range. Tests use lazy frame
iterators to pin that encoding begins before the inputs are exhausted
and that look-ahead stays bounded.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): video upload feedback parity with images

- Add uploadVideo matchFulfilled/matchRejected listeners mirroring the
  image upload listeners: success toasts name the destination board and
  navigate the gallery on the first upload of a batch; failure toasts
  name the failed file, which is what makes partially failed
  Promise.allSettled batches attributable (uploadVideos and the
  fullscreen dropzone aggregate without rethrowing, same as images).
- GalleryUploadButton now uses the hook's combined isUploading, so an
  in-flight MP4 shows a spinner and blocks resubmission.
- Media-neutral labels on the two video-enabled surfaces: gallery
  uploader aria/tooltip says Upload Media, the fullscreen overlay says
  uploaded items (not images) will be added, and its invalid-file toast
  mentions MP4.

Co-Authored-By: Claude Fable 5 <[email protected]>

* docs: include videos in the user-deletion data-loss note

The admin guide's user-deletion warning enumerated boards, images,
workflows, queue items, and style presets but not videos. State that
video records survive with the deleted user_id, that files remain under
outputs/videos, and that administrators keep gallery visibility of the
orphaned media for review/cleanup — matching the behavior pinned by the
video_records user-deletion lifecycle test.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix video processing resource bounds

* fix video lifecycle edge cases

* stabilize decoder inactivity test

* chore(deps): declare psutil as a direct dependency

video_thumbnails.py now imports psutil for decode-worker process-tree
termination, but it was only present transitively (via accelerate and
friends). Declare it so the import can't silently break when an
upstream package drops it.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix video security and lifecycle regressions

* chore: regenerate OpenAPI schema

* fix: preserve exact frame dimensions in video encoders

imageio's FFMPEG writer defaults to macro_block_size=16, which makes ffmpeg
silently rescale frames to the next multiple of 16 — a 1920x1080 upload
trimmed by Frame Range from Video came back as 1920x1088 while the DTO
recorded 1080, so concatenating the trim with its own source failed the
same-dimensions check.

- New invokeai/app/util/video_encoding.make_mp4_writer single-sources the
  encoder settings (libx264, macro_block_size=1) for wan_latents_to_video,
  video_concat, and video_frame_extract_range.
- yuv420p requires even dimensions, so concat and extract-range now reject
  odd-dimension sources up front with a clear error instead of an opaque
  ffmpeg failure mid-encode.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: correct A14B fallback scheduler and stop LoRA leakage to low-noise expert

Two silent-wrong-output bugs on the GGUF A14B path:

- The no-scheduler-dir fallback returned FlowMatchEulerDiscreteScheduler for
  A14B, but both Wan-AI/Wan2.2-{T2V,I2V}-A14B-Diffusers repos ship
  UniPCMultistepScheduler with flow_shift=3.0 / flow_prediction /
  use_flow_sigmas (verified against the upstream scheduler_config.json).
  Every A14B GGUF render ran an unshifted first-order Euler schedule,
  degrading output and skewing how many steps land above the MoE boundary.
  An unreadable on-disk config now also falls back to the variant default
  instead of bare FlowMatchEuler.

- low_loras fell back to the primary list when loras_low_noise was empty,
  but the Wan LoRA loader deliberately routes expert-tagged LoRAs to exactly
  one list — so a high-noise-only LoRA (e.g. a Lightning high-noise distill)
  was silently applied to the low-noise expert too, and high-only targeting
  was impossible. An empty low list now means no LoRAs on the low expert.

Also (here and in the previous commit): the Wan VAE decode nodes now raise a
clear latent-channel mismatch error (16-channel A14B latents vs 48-channel
TI2V-5B VAE and vice versa) instead of an opaque tensor-size RuntimeError
when the wrong VAE is selected.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: keep viewer selection on surviving item after image deletion

The image-side handleDeletions cleared the gallery selection
(imageSelected(null)) whenever the deletion intersected the multi-selection
but the displayed item was not among the deleted names — e.g. a video
displayed while only images were deleted from a mixed selection, or a
hover-delete of a non-displayed selected image. It also treated every
requested name as deleted, ignoring the server's deleted_images response,
so a partial failure could jump the selection away from an image that still
exists.

Port the deleteVideoModal logic: only server-confirmed deletions count, a
surviving displayed item stays selected, and the usage-reset sweep
(nodes/canvas/ref-image layers) runs only for actually-deleted images.
Regression tests mirror deleteVideoModal/store/state.test.ts.

Co-Authored-By: Claude Fable 5 <[email protected]>

* perf: ~48x faster Wan VAE decode on ROCm via conv2d decomposition

MIOpen has no implicit-GEMM 3D-convolution kernels for the Wan VAE's shapes
on RDNA3 and falls back to Im3d2Col (61% of decode GPU time in a torch
profile). An 81-frame 832x480 decode took 730s on a W7900 vs 13s on an RTX
5060; dtype changes and cudnn.benchmark kernel search were all within +/-7%.

A stride-1 kTxkHxkW conv3d is exactly the sum of kT conv2d taps over shifted
temporal slices, and MIOpen's conv2d kernels are well optimized. This rebinds
WanCausalConv3d.forward (class-level, idempotent, ROCm builds only) to that
decomposition:

- same 3-latent-frame decode: 81.6s -> 1.71s (~48x), extrapolating to ~12s
  for the 81-frame workload — matching NVIDIA wall-clock
- numerically equivalent up to summation order: ~1e-6 max error vs F.conv3d
  in fp32; full bf16 decode differs by <=3/255 in pixel space (0.1% of pixels
  by more than 1/255)
- strided encoder downsample convs keep the stock F.conv3d path (temporal
  taps couple under stride)
- applied from every AutoencoderKLWan load site (Wan checkpoint/diffusers
  VAE loaders, Wan main-model VAE submodel, Anima VAE), so decode, encode,
  and ref-image conditioning all benefit; CUDA builds are untouched

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: self-heal the media cookie for sessions that predate or outlive it

Video playback authenticates via an HttpOnly cookie (media elements can't
send Authorization headers) that was only issued at login. A session restored
from localStorage can hold a valid JWT without the cookie — the session may
predate the cookie's introduction, or the cookie may have been cleared while
the JWT survived. Every API call works, but each <video> request 401s and the
player silently renders black with 0:00 duration (hit during PR #9163
functional testing).

- New POST /api/v1/auth/media-cookie re-issues the cookie from a valid Bearer
  token: same live-user check as get_current_user, cookie lifetime clamped to
  the token's remaining validity, successful no-op in single-user mode.
  Cookie attributes are shared with login via _set_media_cookie so they can't
  drift.
- Frontend calls it once per app load when an authenticated session exists
  (useMediaCookieRefresh in GlobalHookIsolator); failures are left to the
  existing global 401 handling.

Co-Authored-By: Claude Fable 5 <[email protected]>

* feat: use Apply LoRA Collection node in Wan concept-LoRA workflow templates

Replace the per-slot concept-LoRA plumbing in the three 'w/ Concept LoRAs'
video templates (Text to Video, Image to Video, Extend Video) with the single
wan_lora_collection_loader node: users now add any number of concept LoRAs
through one multi-LoRA form field instead of two fixed slots (T2V/Extend) or
the lora_selector + collect chain (I2V).

Chain in all three: model loader -> Lightning high-noise LoRA -> Lightning
low-noise LoRA -> Apply LoRA Collection (concept LoRAs, ships empty) ->
denoise. Also prunes exposedFields entries that referenced nodes deleted in
an earlier revision of these templates (pre-existing; the frontend ignored
them, but they were dead weight).

Validated: backend WorkflowValidator + default-sync asserts, node versions
current, every edge/form/exposedFields reference resolves, frontend
parseAndMigrateWorkflow accepts all three, no machine-specific model
identifiers ship.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: enforce multiuser image authorization

* fix: address remaining Wan video review findings

* test: give closed-stream decoder test headroom for slow Windows spawn

The 0.2s decode timeout raced against Python subprocess startup on the
Windows CI runner: the inactivity deadline fired before the worker could
close its stdout, raising the generic decode timeout instead of the
expected 'decoder worker' one. A generous timeout makes the EOF path
deterministic; proc.wait still bounds the test at ~1s.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: address adversarial Wan video review findings

* fix: resolve remaining Wan video review issues

* feat(ui): rename gallery/board strings from Images to Images/Videos

The gallery grid, selections, board operations, and related settings now
operate on both images and videos, so the user-facing strings that
describe them say so. Image-only surfaces (compare, reference images,
progress previews, image storage maintenance, upload-format errors) are
unchanged, as are unused legacy keys.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(api): harden video/media API per PR review

- Scope the media cookie (set + delete) and the sliding-token middleware's
  auth-route exclusions to the reverse-proxy root_path, so media auth works
  behind sub-path proxies and a proxied logout can't mint a replacement token.
- Video upload: run filesystem writes, MP4 validation, ffmpeg probing, and
  create() in the thread pool; add VideoUploadLimitASGIMiddleware to bound
  request size before multipart spooling and cap concurrent uploads.
- Add GET /videos/i/{name}/workflow (mirrors the image route) so persisted
  video workflows/graphs are retrievable, with read-access checks.
- Add DELETE /videos/uncategorized so the "Delete All Uncategorized
  Images/Videos" action can cover both media kinds.
- Make polymorphic gallery ordering deterministic on created_at ties with
  kind+name tie-breakers, and pick virtual-board covers via ROW_NUMBER
  instead of a bare-column MAX() aggregate.
- Add cpu_only to WanT5Encoder_WanT5Encoder_Config (parity with the other
  standalone text-encoder configs; the loader already honors the field).
- Regenerate schema.ts/openapi.json.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(backend): Wan VAE effective device + generalized working-memory estimation

- Move VAE inputs to get_effective_device(vae) instead of the globally
  selected device — a cpu_only Wan VAE previously crashed every Wan VAE
  invocation on GPU hosts.
- Add estimate_vae_working_memory_wan (per-frame conv working set + resident
  RGB clip, config-driven spatial scale for TI2V's 16x compression) and
  reserve working memory in all four Wan VAE paths, replacing the Flux
  estimator / missing reservations.
- Fall back to spatial tiling for video decodes whose full-frame working set
  exceeds the execution device's VRAM, and move the decoded clip to the CPU
  before MP4 encoding so VRAM isn't held for the encode's duration.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): video gallery/deletion/workflow fixes per PR review

- Video deletion: use the batch endpoint (one request per invocation), clear
  workflow-node VideoField inputs only for server-confirmed deletions, and
  invalidate per-video DTO/metadata/workflow caches on delete (including
  board-cascade deletions in the board mutations).
- VideoFieldInputComponent resets its value only on a confirmed 404, not on
  transient auth/server/network errors.
- Global Delete hotkey partitions the polymorphic selection and routes videos
  through the video delete flow.
- "Delete All Uncategorized Images/Videos" now deletes both media kinds;
  "Download Board" relabeled "Download Board Images" (image-only endpoint).
- Translation splits: image-only multi-select actions revert to "Images";
  polymorphic gallery search + star hotkey become media-neutral; the multi-
  drag preview counts the whole mixed selection.
- Expose video metadata + workflow in the viewer: new video details overlay
  (metadata/workflow/graph tabs), a Load Workflow toolbar action for videos,
  and a 'video' source for the load-workflow dialog.
- Model Manager: wan_t5_encoder gets the encoder settings panel (Run on CPU).

Co-Authored-By: Claude Fable 5 <[email protected]>

* docs: make gallery docs video-aware; fix video workflow count

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: address remaining video review findings

* fix(backend): Wan inference fixes from full-PR self-review

- Force bfloat16 in the standalone Wan VAE checkpoint loader: `precision: auto`
  resolves to fp16 on CUDA, and fp16 is unstable on the Wan VAE (the diffusers
  folder path already forced bf16). Both starter VAEs route through this loader.
- Count the decoded RGB clip twice in the Wan working-memory estimator:
  diffusers' frame accumulation transiently holds ~2x the clip at peak, which
  the tiled-decode fallback previously undercounted by up to ~2 GB.
- Ignore a wired 'Transformer (Low Noise)' for TI2V-5B (warn instead of raising
  a misleading A14B error), matching the field's documented behavior.
- Release the expert swapper's device context even when LoRA weight-restore
  raises, so a failed unwind can't pin an 8-9 GB expert in VRAM.
- Validate LoRA variant (A14B vs 5B) against the wired transformer in both Wan
  LoRA loaders — a mismatch previously crashed mid-denoise with an opaque
  layer-patcher shape error.
- Fix the WanDiffusersModel exception ladder: the old-diffusers torch_dtype
  retry now also gets the missing-variant OSError fallback, with the matching
  dtype kwarg.
- Mark both Wan ideal-dimensions nodes Prototype like every other Wan node;
  correct the text-encoder docstring (seq_len 512, not 226).
- Add CPU tests for the multi-frame WanVideoDenoise loop (T_lat>1 shapes,
  zero-velocity invariant, A14B I2V 36-channel concat across frames, TI2V-5B
  expand-timesteps mask blend incl. per-token timesteps and frame-0 restore).

Node version bumps: wan_model_loader, wan_lora_loader,
wan_lora_collection_loader -> 1.0.1.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(api): video service fixes from full-PR self-review

- Purge cached invocation outputs on video deletion: the memory invocation
  cache registered images/tensors/conditioning on_deleted hooks but not videos,
  so re-running an identical graph after deleting its output "succeeded" with a
  cached VideoOutput naming a 404 video.
- Add the single-user early-return to VideosInterface's read-access and
  board-save checks, matching ImagesInterface — after a multiuser->single-user
  switch, video workflows no longer fail with PermissionError where identical
  image operations succeed.
- Restructure staged-delete recovery to match the image side: video_records
  .get() raises rather than returning None, so the explicit commit branch was
  unreachable and recovery semantics lived in the exception handler by luck.
- Return 416 (not 206 with "bytes 0--1/0") for any Range request against a
  zero-length video file; add tests for the whole Range-parser matrix.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): video UX fixes from full-PR self-review

- Add 'video' to the invocation-complete passthrough denylist: a Video
  Primitive completing mid-run invalidated gallery caches and auto-switched the
  user's selection/board to the node's *input* video.
- Show the multi-selection context menu only when the clicked item is part of
  the selection (both image and video menus): right-clicking a video with 2+
  images selected previously produced a menu with every action disabled.
- Clear workflow VideoField references when videos are cascade-deleted via
  board deletion or delete-uncategorized, matching the direct-delete flow.
- Toast on total video-delete-batch failure (the untracked mutation was
  otherwise silent) and on failed logout (the button previously did nothing
  when the server was unreachable).
- Check resp.ok in useDownloadItem so an expired media cookie can't save error
  bodies as .mp4/.png files.
- Provide the LIST_TAG-scoped VideoList tag from listVideos so the star/board
  invalidations that reference it actually match; fix the misleading comment;
  dedupe the doubled BoardVideosTotal tag type.
- Validate VideoField access on workflow load (checkVideoAccess), resetting
  stale refs with a warning like image fields.
- Wire middle-click-open-in-new-tab for gallery videos (the setting label
  already promised it).
- Show the effective fallback (primary CFG) in the low-noise guidance slider
  when unset, instead of a constant the run never uses.
- "Moving 1 image/video to board:" singular form for mixed-media moves.
- Regenerate schema.ts/openapi.json (node version bumps, classification,
  docstring fixes).

Co-Authored-By: Claude Fable 5 <[email protected]>

* chore: workflows/docs/deps fixes from full-PR self-review

- Update all 12 bundled Wan workflows to current node versions (wan_model_loader
  / wan_lora_loader / wan_lora_collection_loader 1.0.1, wan_ref_image_encoder
  1.2.0, backfilling the optional end_image/num_frames inputs) so fresh installs
  don't open with "node needs update" badges; add a registry-consistency test
  over the bundled Wan/video workflows so stale embeds can't recur.
- Docs: the A14B auto scheduler is UniPC (not FlowMatchEuler); note that the
  bundled TI2V-5B workflows ship 20 steps as a speed compromise vs the 40-50
  quality recommendation.
- Pin imageio[ffmpeg]>=2.37 and psutil>=6 (imageio encode behavior is
  version-sensitive enough that we carry a regression test for it); relock.
- De-flake the thumbnail worker descendant-kill test (0.5s was the only tight
  ceiling in the file; a loaded runner could kill the worker before the child
  pid file existed).

Co-Authored-By: Claude Fable 5 <[email protected]>

* ci: raise python-tests job timeout to 30 minutes

Main already runs 9-11 min per platform and this PR pushed py3.11 windows-cpu
past the 15-minute cap (cancelled mid-pytest at 15m10s on the last run).

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: resolve CI failures from main's route-auth audit and knip

The route-authorization audit merged from main (#9367) only recognized the
two Bearer-token dependencies, so it flagged the video media routes (which
authenticate via get_current_media_user_or_default) and the media-cookie
endpoint (which validated its Bearer token inline). Teach the audit about
the media dependency, drop the image media routes from PUBLIC_ROUTES (they
now carry cookie auth on this branch), and give refresh_media_cookie a
CurrentUserOrDefault dependency in place of its duplicated inline
validation. The media-cookie tests now patch auth_dependencies'
ApiDependencies like every other auth-dependent route test.

knip: getDeletedVideosFromDeleteBoardAction was exported but only used
in-module; cover it in the listener unit tests like its image twin.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(backend): Wan invocation fixes from JPPhoto's 2026-07-21 review

- Both Wan LoRA loaders now validate the *resolved* config (type=LoRA,
  base=Wan) instead of trusting the client-supplied identifier fields; a
  mislabeled Flux/SDXL/main key is rejected up front instead of reaching
  the layer patcher.
- The collection loader rejects LoRAs already applied upstream on either
  expert list (same invariant as the single loader) instead of silently
  doubling their effective weight.
- A LoRA routed only to the low-noise list of a TI2V-5B main now logs a
  warning — the single-transformer path never consumes that list, so the
  routing was a silent no-op.
- _ExpertSwapper._release clears its slots in a nested finally, so a
  device-context exit failure can no longer leave stale contexts that a
  later close() would double-exit.
- WanLatentsToImage rejects multi-frame (T>1) video latents with a clear
  error pointing at wan_l2v, before the VAE is even loaded — previously it
  ran the full multi-frame decode and died in an opaque einops rank error.
- wan_ref_image_encoder docstrings now describe both the 20-channel A14B
  and 48-channel TI2V-5B condition paths (they claimed A14B-only and told
  users to omit the node for TI2V-5B, contradicting the implementation).

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(api): per-user upload slots, probe validation, stream decode fallback

- VideoUploadLimitASGIMiddleware now accounts upload slots per user (cap 2)
  on top of the global cap, so one tenant's slow chunked uploads can no
  longer hold all four slots and starve other users into 429s. Single-user
  mode keeps the whole global capacity (no per-user quota).
- probe_video validates decoder-reported metadata: non-positive or
  over-limit dimensions (> 64 MP) and non-finite/negative durations are
  rejected before the upload path persists them; garbage fps degrades to
  None (unknown). The decode worker refuses to decode frames from files
  whose probed dimensions exceed the bound — a small crafted container
  claiming 100k x 100k would otherwise trigger a ~30 GB allocation.
- The worker's stream command falls back to cv2 like probe/frame/count do,
  so an MP4 accepted at upload via the cv2 path now also works in the
  frame-range and concat nodes. The fallback only engages before the first
  emitted frame; a mid-stream decoder death still surfaces as an error.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(ui): media resilience fixes from JPPhoto's 2026-07-21 review

- Partial board deletion: the boardAndImagesDeleted listener now invalidates
  the per-item Image*/Video* tags for the confirmed-deleted names it parses
  out of the 500 detail — the rejected mutation runs invalidatesTags with no
  result, so those caches previously stayed readable.
- Media-cookie refresh retries transient failures on a bounded backoff
  (2s, 10s) instead of latching before the request and giving up forever;
  401 still bails (session genuinely expired).
- Thumbnail 404s degrade gracefully: BoardTooltip, GalleryBoard,
  VirtualBoardItem, and VideoFieldInputComponent show an icon fallback via
  fallbackStrategy="onError" (thumbnail generation is best-effort server-
  side), and GalleryVideoThumbnail's <video> fallback does the near-zero
  seek on loadedmetadata so browsers that don't auto-paint the first frame
  no longer show a black tile.
- CurrentVideoPreview handles play() rejection (rolls isPlaying back) and
  media element errors (drops back to the play overlay) instead of hiding
  the overlay over a dead element with an unhandled promise rejection.
- Hardening from the disputed items: changeVideoIsIntermediate also
  invalidates the VideoList LIST_TAG (covers a future omitted-board_id
  list); logout clears gallery.selection and the logout mutation documents
  that resetApiState in store.ts is what actually clears cross-user caches.
- Typegen regenerated for the wan_lora_loader / wan_ref_image_encoder
  docstring updates.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: address final video review findings

* fix: harden video workflows and auth refresh

* chore: regenerate OpenAPI schema

* fix: close video workflow review gaps

* fix: address self-review findings on video workflows and auth refresh

Fixes the confirmed findings from the 2026-07-22 self-review round
(github.com/invoke-ai/InvokeAI/pull/9163#issuecomment-5051225515):

Frontend:
- DeleteVideoModal: detach the dialog promise callbacks before the async
  deletion so the accept path's synchronous onClose (bound to cancel) no
  longer rejects a confirmed deletion as "User canceled"; dismissal still
  rejects. Adds behavioral tests for both paths.
- Sliding-window refresh: bound the media-cookie sync fetch with a 10s
  AbortSignal timeout so a stalled request can't hold the exclusive
  cross-tab media-auth lock (shared with login/logout) forever; commit the
  refreshed token even when the cookie sync fails with a 5xx/network error
  (only a 401/403 rejection of the token blocks the commit); throttle
  acceptance to once per minute so bulk mutations don't pay a serialized
  cookie round trip per request.
- Fallback media-auth lock: waiters renew their ticket lease while queueing
  so a >30s wait no longer lets a later ticket enter concurrently.
- useMediaCookieRefresh: a pause() abort now resumes the same attempt
  instead of consuming a retry slot (and no longer permanently disables
  self-heal when the final attempt was paused); effect cleanup aborts
  in-flight refreshes so every logout path (sessionExpiredLogout, direct
  logout) stops a pending refresh from re-minting the cookie post-logout.
- CurrentVideoPreview: a benign AbortError from play() rolls back silently,
  and load errors during the pending media-cookie self-heal window no
  longer raise a spurious "Unable to Play Video" toast.

Backend:
- Decode-worker memory bounds resized for legal near-cap frames: worker
  RLIMIT_AS headroom 1->4 GiB, parent RSS kill threshold 1->3 GiB (with
  keep-in-sync cross-references), monitor poll 50->250 ms.
- Upload probe: a decode-worker timeout is now inconclusive (upload
  proceeds) instead of a 415; the probe's decoded first frame is reused as
  the thumbnail source, dropping one worker subprocess per upload.
- delete_images_on_board / delete_videos_on_board return (deleted, failed)
  and delete_board reports the services' ground truth instead of a racy
  router-side listing diff (which also doubled the DB work).
- Video list/uncategorized delete endpoints skip HTTPException (ownership
  skips, 404 races) silently instead of reporting them as failures,
  matching the image endpoints; delete_images_from_list now populates
  failed_images for genuine failures, matching the video path.
- video_concat: an unknown probed fps mixed with agreeing known rates uses
  the known rate again instead of hard-erroring; disagreeing known rates
  still require an explicit Output FPS.
- SlidingWindowTokenMiddleware runs its synchronous SQLite user lookup via
  run_in_threadpool so a contended DB lock can't stall the event loop.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(tests): deflake drip-feed upload timeout test on Windows

The test gave the middleware a 20 ms absolute upload deadline and delivered a
chunk every 5 ms — but Windows event-loop timers have ~15.6 ms granularity, so
the deadline could expire before the first chunk was ever delivered. The
request then ended at receive_calls == 1 and the `receive_calls > 1` assertion
failed (py3.12 windows-cpu CI).

Widen the margins so the scenario the test describes actually occurs on coarse
timers: 250 ms absolute deadline (many chunks flow first on every platform)
with a 1 s idle timeout that never fires between 5 ms chunks.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(tests): sort imports in test_video_upload_limits (ruff I001)

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: address in-PR items from JPPhoto's non-merge-blocker list

Fixes the subset of JPPhoto's 2026-07-22 "Still Open, Non-Merge Blockers"
that are small, self-contained, and scoped to surfaces this PR introduced;
the rest are deferred to a follow-on PR (triage rationale posted on the PR).

- video_thumbnails._run_worker: an unexpected exception from communicate()
  (e.g. OSError) now terminates the worker process tree unconditionally —
  previously the finally stopped the RSS-monitor backstop while the except
  path left the worker and its ffmpeg child running forever. Adds the
  injected-OSError test JPPhoto asked for.
- reduxRemember driver: client-state persistence POSTs now commit
  X-Refreshed-Token via the same acceptance flow as dynamicBaseQuery
  (extracted as acceptRefreshedToken, sharing the cross-tab lock, cookie
  sync, throttle, and generation guards), so persistence-only sessions no
  longer hard-expire mid-activity.
- delete_videos_from_list / delete_images_from_list: dedup request names —
  a repeated name was processed twice and landed in both deleted_* and
  failed_* under the admin ownership bypass, toasting a spurious partial
  failure. Regression test added.
- gallery + videos list endpoints: bound offset (ge=0) and limit
  (ge=0, le=MAX_PAGE_SIZE=1000) — these flowed verbatim into SQL, where a
  negative LIMIT means unlimited in SQLite, so one request could
  materialize the entire gallery. openapi.json regenerated (schema.ts is
  unchanged — constraints don't alter the generated types).
- get_video_full: open the file once and serve HEAD/range/full from the fd
  (full downloads now stream chunked from the handle instead of
  FileResponse's lazy path-based open), eliminating the delete-race 500;
  deletion's atomic rename can no longer invalidate a path between check
  and open.
- upload_video: close the multipart spool immediately after the body copy,
  shrinking the double-temp-disk window (2 x 1 GiB x 4 concurrent worst
  case) to the copy loop itself.
- docs/gallery.mdx: document shared-board deletion semantics (only your
  own media is permanently deleted; admins delete everything) and the
  kept-on-failure -> Uncategorized behavior with its UI warning.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix video gallery review findings

* fix Windows video thumbnail handling

* test: cover remaining video review findings

* test: cover adversarial video and Wan findings

* fix: address remaining video and Wan review findings

* test: call now-sync star/unstar routes directly

11b38696bf converted the video batch routes from async def to sync def
(so FastAPI offloads them to its threadpool), but the star/unstar
dedupe test still drove them through asyncio.run(), which requires a
coroutine.

Co-Authored-By: Claude Fable 5 <[email protected]>

* test: cover Wan conditioning and video link regressions

* fix: validate Wan conditions and video links

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Co-authored-by: Jonathan <[email protected]>
Co-authored-by: JPPhoto <[email protected]>
This commit is contained in:
Lincoln Stein
2026-07-27 20:42:26 -04:00
committed by GitHub
co-authored by Claude Fable 5 Jonathan JPPhoto
parent 527d984b05
commit eb9a951248
349 changed files with 54271 additions and 931 deletions
@@ -219,10 +219,13 @@ If no arguments are provided, the script will run in interactive mode.
:::danger[Data Loss]
Deleting a user removes the user record and cascades to their sessions, board shares, sent
invitations, and per-user client state. It does **not** delete the boards, images, workflows, queue
items, or style presets they created — those rows remain in the database, owned by a user_id that no
longer exists, and will not appear in any user's gallery. Physical image files in `outputs/images`
are also left in place until a gallery maintenance script is run to remove orphan images.
invitations, and per-user client state. It does **not** delete the boards, images, videos,
workflows, queue items, or style presets they created — those rows remain in the database, owned by
a user_id that no longer exists, and will not appear in other (non-admin) users' galleries.
Administrators keep gallery visibility of the orphaned images and videos and can review or delete
them from there. Physical media files in `outputs/images` and `outputs/videos` are also left in
place: orphan image files can be removed with a gallery maintenance script, while video files should
be deleted through the gallery (which removes the file along with the record) or cleaned up manually.
If you want their content gone as well, reassign or delete it before deleting the user. Back up the
database first if recovery might be needed.
+40 -11
View File
@@ -1,14 +1,14 @@
---
title: Gallery Panel
description: Learn how to manage, organize, and use your generated images and assets with the Gallery Panel in InvokeAI.
lastUpdated: 2026-02-19
description: Learn how to manage, organize, and use your generated images, videos, and assets with the Gallery Panel in InvokeAI.
lastUpdated: 2026-07-19
sidebar:
order: 1
---
import { Card, CardGrid, Steps } from '@astrojs/starlight/components';
The Gallery Panel is a fast way to review, find, and make use of images you've generated and loaded. The Gallery is divided into **Boards**. The *Uncategorized* board is always present, but you can create your own for better organization.
The Gallery Panel is a fast way to review, find, and make use of images and videos you've generated and loaded. The Gallery is divided into **Boards**. The *Uncategorized* board is always present, but you can create your own for better organization. Boards are polymorphic — images and videos coexist on the same board and appear together in the gallery, sorted by creation time.
![Gallery Panel Overview](./assets/gallery.png)
@@ -50,11 +50,17 @@ Each board has a context menu accessible via right-click (or Ctrl+click).
![Thumbnail Menu](./assets/thumbnail_menu.png)
- **Auto-add to this Board:** If *Auto-Assign Board on Click* is disabled in settings, use this option to quickly set the selected board as the default destination for new images.
- **Download Board:** Packages all images within the board into a `.zip` file. A notification link will be provided when the download is ready.
- **Delete Board:** Permanently removes the board and all of its contents.
- **Download Board Images:** Packages all images within the board into a `.zip` file. A notification link will be provided when the download is ready. Videos are not included; download them individually from the gallery or viewer.
- **Delete Board:** Opens a confirmation dialog. **Delete Board Only** removes the board and leaves its contents uncategorized; **Delete Board and Assets** permanently removes the board, its images, and its videos.
:::danger
Deleting a board will **permanently delete all images** contained within it. Proceed with caution!
Only **Delete Board and Assets** permanently deletes the board's images and videos. This action cannot be undone.
:::
:::note
In multi-user deployments, **Delete Board and Assets** only permanently deletes *your own* media on a shared board — other users' contributions are preserved and move to Uncategorized. (Administrators delete everything on the board.)
If any item's files cannot be removed (for example, due to a transient storage error), that item is kept rather than left orphaned on disk: its record moves to Uncategorized, and the app shows a warning with the number of items that could not be deleted.
:::
### Image Storage Maintenance
@@ -67,22 +73,22 @@ Every board is organized into two distinct tabs:
![Board Tabs](./assets/board_tabs.png)
1. **Images:** Images generated directly within InvokeAI.
1. **Images/Videos:** Images and videos generated directly within InvokeAI, shown interleaved by creation time.
2. **Assets:** External images you have uploaded to use as an [Image Prompt](https://support.invoke.ai/support/solutions/articles/151000159340-using-the-image-prompt-adapter-ip-adapter-) or within the Image to Image tab.
---
## Virtual Boards
Virtual boards are read-only board groupings that Invoke computes on-the-fly from your image metadata rather than storing in the database. The first available type groups images **By Date**, creating one sub-board per day on which you generated images.
Virtual boards are read-only board groupings that Invoke computes on-the-fly from your gallery records rather than storing in the database. The first available type groups media **By Date**, creating one sub-board per day on which you generated images or videos.
Virtual boards are **off by default**. To enable them:
1. Open the **board settings** (gear icon at the top of the Gallery).
2. Toggle **Virtual Boards** on.
3. A collapsible **By Date** section appears in the board list, with a sub-board for each day that has images. Each sub-board shows the date, image / asset counts, and a cover thumbnail.
3. A collapsible **By Date** section appears in the board list, with a sub-board for each day that has images or videos. Each sub-board shows the date, image / asset / video counts, and a cover thumbnail (the newest item of that day, image or video).
Selecting a date sub-board filters the gallery to just the images from that day. The collapse state of the By Date section persists across reloads.
Selecting a date sub-board filters the gallery to just the images and videos from that day. The collapse state of the By Date section persists across reloads.
### Limits
@@ -90,7 +96,7 @@ Because virtual boards are derived, not stored:
- They are **read-only**: no drag-and-drop, no context menu, no auto-add destination.
- You cannot rename or delete them.
- Generating a new image updates the counts immediately, but the image is still saved to your regular auto-add board — virtual boards are a *view*, not a destination.
- Generating a new image or video updates the counts immediately, but the item is still saved to your regular auto-add board — virtual boards are a *view*, not a destination.
- Disabling the **Virtual Boards** toggle hides the section and resets the selection to *Uncategorized* if you were viewing a virtual sub-board.
---
@@ -134,6 +140,29 @@ Additionally, each image has a context menu (right-click or Ctrl+click) with pow
---
## Videos in the Gallery
Videos generated by InvokeAI (currently from the Wan 2.2 model family) appear alongside images in the same gallery view. Each video item displays a first-frame still as its thumbnail with a play badge in the corner; selecting it opens the video in the viewer where you can play it back inline.
### Uploading Videos
You can upload existing videos to a board via the standard drop-or-upload affordance. The upload pipeline accepts **MP4 files encoded with H.264/AVC only** so uploaded videos can play in supported browsers. Other containers (`.mov`, `.webm`, `.mkv`) and codecs such as HEVC/H.265 are not transcoded on upload and are rejected at the API boundary. Re-encode them before uploading, for example with `ffmpeg -i input.mov -c:v libx264 output.mp4`.
The **Concatenate Videos** and **Extract Video Range** nodes encode video frames only; audio tracks are not preserved. Concatenation requires matching source frame rates unless an explicit output FPS is selected, which retimes the source frames.
### Video Context Menu
Each video has a context menu with the same organization actions as images, plus video-appropriate variants:
- **Open in New Tab / Download:** Opens or saves the raw MP4 file.
- **Star Video:** Pins the video to the top of the gallery.
- **Change Board:** Moves the video to a different board. *(Drag-and-drop onto board thumbnails also works.)*
- **Delete Video:** Permanently deletes the video and its thumbnail.
Videos count toward board contents: a board with two images and three videos shows five items in the polymorphic gallery list and reports both totals in its stats.
---
## Summary
This walkthrough covers the Gallery interface and Boards. For guidance on prompting and generation workflows, please refer to the [Prompting Guide](/concepts/prompting-guide/) and [AI Image Generation](/concepts/image-generation/).
@@ -0,0 +1,267 @@
---
title: Video Generation
description: Generate short videos with the Wan 2.2 model family — text-to-video, image-to-video, and the trick for stitching longer sequences.
lastUpdated: 2026-07-19
---
import { Card, CardGrid, Steps } from '@astrojs/starlight/components';
InvokeAI ships **experimental support for the Wan 2.2 model family**, which lets you generate short MP4 clips from a text prompt, an image, or both. Output ranges from a few-second loop (the model's training distribution) up to longer sequences assembled with the [concat trick](#making-longer-videos) below.
:::caution[Experimental]
Video generation is a prototype feature. Workflows, node fields, and starter-model packaging may change between releases. The underlying models are also new — expect rough edges in coherence and artifacts at longer durations.
:::
:::tip[Just want to make a video?]
Start with the [**Video Workflows** guide](/features/video-workflows/) — a plain-language tour of the ten ready-made workflows that ship with InvokeAI and how to run them. This page is the deeper technical reference behind them.
:::
---
## Models
Wan 2.2 ships three transformer variants, plus a shared text encoder and two VAEs. All share the same diffusion-style sampling but differ in size, conditioning, and intended task.
### The variants
| Variant | Task | Params | VAE | Conditioning |
|---|---|---|---|---|
| **T2V-A14B** | Text → Video | 14B × 2 experts | A14B VAE (16-ch, 8× spatial) | Text only |
| **I2V-A14B** | Image+Text → Video | 14B × 2 experts | A14B VAE (16-ch, 8× spatial) | Text + reference image (36-channel concat) |
| **TI2V-5B** | Text → Video OR Image+Text → Video | 5B (single) | Wan 2.2-VAE (48-ch, 16× spatial) | Text, optionally with reference image (first-frame mask blend) |
* **T2V-A14B** generates videos from a text prompt alone. Best motion coherence and prompt-following of the three.
* **I2V-A14B** locks the first frame to a reference image you supply. Best subject-preservation; the image is concatenated to the noise latents at every step so the model "sees" the reference throughout denoising.
* **TI2V-5B** is the small single-expert variant. It can do **both** text-to-video and image-to-video with the same checkpoint, at substantially lower VRAM, but with somewhat less stable long-range coherence than the A14B variants.
### High-noise and low-noise transformers (A14B variants only)
The A14B models are a **mixture-of-experts (MoE)** pair. There are actually *two* 14B transformers on disk per variant — a "high-noise" expert and a "low-noise" expert — and the denoise loop swaps between them at a model-defined boundary timestep:
* **High-noise expert** runs early in denoising, when the latents are still mostly noise. It's responsible for composition, layout, and broad motion.
* **Low-noise expert** runs later, when the latents are close to clean. It refines detail and texture.
InvokeAI handles the swap automatically — both experts have to be installed (the starter bundle handles this), but the workflow only references the "high-noise" model as the main and the "low-noise" model is wired alongside it via the loader node. You don't manage the boundary yourself.
**TI2V-5B is single-expert** — no swap, no boundary, just one model that runs every step. Workflows for TI2V-5B are correspondingly simpler.
### Lightning LoRAs (4-step inference for A14B)
The default A14B variants need ~40–50 denoise steps for clean output. The Wan team also released **Lightning distillation LoRAs** that collapse that to 4 steps with minimal quality loss — about a 10× speedup. There's a pair per variant (one LoRA for the high-noise expert, one for the low-noise), wired through the LoRA loader nodes in the starter workflows.
:::tip
**TI2V-5B doesn't have a Lightning LoRA.** Its smaller size means each step is cheap; you typically run it at 40–50 steps and end up in a similar wall-clock ballpark as A14B + Lightning. (The bundled TI2V-5B workflows ship with 20 steps as a speed/quality compromise — raise the step count if you want cleaner output.)
:::
### Installing models
The model manager ships two **starter bundles** for video work:
* **Wan 2.2 Text-to-Video** (~36 GB) — UMT5-XXL text encoder, both VAEs, TI2V-5B Q4_K_M, T2V-A14B Q4_K_M (high + low), T2V Lightning (high + low).
* **Wan 2.2 Image-to-Video** (~32 GB) — UMT5-XXL, A14B VAE, I2V-A14B Q4_K_M (high + low), I2V Lightning (high + low).
The bundles are independent. Installing both ends up at ~56 GB total (shared components — UMT5-XXL and the A14B VAE — are deduplicated on the second install). A 12 GB VRAM card can install only the Text-to-Video bundle and have **TI2V-5B available for both T2V and image-to-video** without ever touching the I2V bundle.
Higher-quality Q8_0 quantizations of every transformer, plus full Diffusers builds of all three variants, are available as a-la-carte installs in the starter models list.
:::tip
**On a 12 GB VRAM card**: install just the Text-to-Video bundle and use TI2V-5B. The A14B variants will technically run via aggressive offloading but are slow and prone to OOM. TI2V-5B Q4_K_M fits comfortably and is what we recommend for that tier.
:::
---
## Workflow setup
The shipped starter workflows ("Text to Video - Wan 2.2 Lightning", "Image to Video - Wan 2.2 Lightning") are the easiest starting point — load them from the workflow library, pick your models, set a prompt, and Invoke. The [Video Workflows guide](/features/video-workflows/) walks through all ten and how to choose between them; the sections below describe what's happening inside so you can build your own.
### Constraints that apply to every video workflow
**Frame count**: `num_frames - 1` must be divisible by **4**. This is dictated by the Wan VAE's temporal compression (4 pixel-frames → 1 latent-frame). Valid values: 5, 9, 13, … **81** (the training default, 5 seconds at 16 fps), 85, 89, etc.
**Pixel dimensions**: must be a multiple of **16** for T2V-A14B and I2V-A14B, and a multiple of **32** for TI2V-5B. The constraint comes from the VAE's spatial downsample × the transformer's 2×2 patch size:
| Variant | VAE spatial | Pixel multiple of |
|---|---|---|
| T2V-A14B, I2V-A14B | 8× | **16** |
| TI2V-5B | 16× | **32** |
Reference values that work: 832×480 (480p), 1280×720 (720p, A14B only — TI2V-5B needs 1280×704 instead since 720 isn't divisible by 32).
**Encoder and denoise dimensions must match**: the `Reference Image - Wan 2.2` encoder and the `Denoise Video - Wan 2.2` node both have their own `width` and `height` fields. They have to be identical or the denoise loop will reject the condition tensor.
:::tip[Use Wan 2.2 I2V Ideal Dimensions]
The **Wan 2.2 I2V Ideal Dimensions** node takes a source image's W×H and a target preset (480p / 720p / 1080p) and outputs valid (width, height) for the encoder + denoise inputs. Wire it once and feed its outputs into both nodes. Saves the manual snap-to-16/snap-to-32 math.
:::
### Text-to-video workflow
The minimum node chain for T2V:
```
Wan Main Model Loader ──┐
│
Wan T5 Text Encoder ────┤
▼
Wan Compel Conditioning (positive)
│
▼
Denoise Video - Wan 2.2 ──→ Latents to Video - Wan 2.2 ──→ MP4
▲
Wan Compel Conditioning (negative) ─┘
```
For **TI2V-5B T2V** this is the entire graph — load the TI2V-5B model and the TI2V-5B VAE, set width/height/num_frames, and run.
For **T2V-A14B** the main model loader also exposes the low-noise expert slot, and you typically add the **Lightning LoRA pair** (one for each expert) to bring step count down to 4. Recommended:
* Steps: **4** (with Lightning) or **40–50** (without)
* CFG: **5.0** high-noise / **4.0** low-noise (the dedicated `Guidance Scale (Low Noise)` field on the denoise node)
* Width × Height: 832×480 (faster, default) or 1280×720 (sharper, 4× the memory)
### Image-to-video workflow
I2V adds a **Reference Image** branch alongside the denoise. The reference image gets VAE-encoded into a conditioning tensor that the denoise loop uses to anchor the video's content:
```
Wan Main Model Loader ──┐
Wan T5 Text Encoder ────┤
Wan Compel Conditioning ┤
│
Image Primitive ──→ Reference Image - Wan 2.2 ──┐
│ │
▼ ▼
Denoise Video - Wan 2.2 ──→ Latents to Video - Wan 2.2 ──→ MP4
```
For **I2V-A14B**, both the reference encoder and the denoise node need to use the same width/height. The encoder also takes a `num_frames` parameter that must match the denoise's `num_frames` — set both to 81 by default.
For **TI2V-5B image-to-video**, the conditioning math is different (the model uses a first-frame-mask blend rather than channel concatenation), but the workflow shape is the same. The encoder auto-detects TI2V-5B from the VAE's 48 latent channels and emits the right condition tensor.
:::caution[TI2V-5B I2V dimensional constraint]
TI2V-5B image-to-video requires **width and height divisible by 32** (not just 16). The encoder will refuse the workflow with a clear error if not. 832×480 works; 1280×720 does not (720 is not divisible by 32). Use 1280×704 for 720p-ish on TI2V-5B.
:::
### First-last-frame interpolation (FLF2V)
The **Reference Image - Wan 2.2** node has an optional **End Image** input. Provide a start image on the regular `image` field and an end image on `End Image (FLF2V)`, and the model interpolates a clip that **begins on the first image and animates to the second** — handy for morphing between two stills or bridging two shots.
```
Image Primitive (start) ──→ image ────┐
├─→ Reference Image - Wan 2.2 ──→ Denoise Video ──→ MP4
Image Primitive (end) ──→ End Image ───┘
```
This is **I2V-A14B only** (`num_frames > 1`); it is not supported on TI2V-5B or single-frame I2V (the node raises a clear error). Wan 2.2 has no dedicated first-last-frame checkpoint — that was a Wan 2.1 model — so the stock I2V-A14B model accepts the end-frame anchor through its existing conditioning channels. Quality is good in practice but not guaranteed for every image pair, so eyeball your results. The shipped **"Interpolate 2 Images to Video - Wan 2.2 Lightning"** workflow wires this up end to end.
### Recommended starting parameters
| | T2V-A14B + Lightning | T2V-A14B | I2V-A14B + Lightning | TI2V-5B (T2V or I2V) |
|---|---|---|---|---|
| Steps | 4 | 40–50 | 4 | 40–50 (bundled workflows use 20 for speed) |
| CFG (high) | 1.0 | 5.0 | 1.0 | 5.0–5.5 |
| CFG (low) | 1.0 | 4.0 | 1.0 | n/a (single expert) |
| Num frames | 81 | 81 | 81 | 81 |
| Width × Height | 832×480 | 832×480 | 832×480 | 832×480 |
| Scheduler | Auto (UniPC) | Auto (UniPC) | Auto (UniPC) | Auto (UniPC) |
---
## Making longer videos
The Wan 2.2 models were trained on **81-frame** clips (5 seconds at 16 fps). Outputs much longer than that suffer rapidly degrading coherence — the temporal positional encoding goes out of distribution and the model loses track of scene content. So instead of asking for `num_frames=200`, the recommended pattern is **chaining**: render a sequence of 81-frame clips where each one's first frame matches the previous clip's last frame, then concatenate them with the `Concatenate Videos` node.
### The basic chain
<Steps>
1. **Render the first clip** with I2V or T2V, ending on whatever subject/scene you want to continue.
2. **Extract the last frame** of clip 1 using the `Frame from Video` node. Use `frame_index = -1` for the literal last frame, or `-3` / `-5` to step back a few frames (last frames sometimes have boundary artifacts — see the [troubleshooting](#late-frame-artifacts-text-color-blobs) note).
3. **Feed that frame as the reference image** for an I2V run that becomes clip 2. Adjust the prompt for whatever motion you want next.
4. **Repeat** as many times as you want clips.
5. **Concatenate** all the clips into a single MP4 with `Concatenate Videos`. Pick a transition mode based on whether you want a seamless join (`cut` if the bridge frame matches perfectly), a smooth blend (`crossfade`), or a punctuated scene change (`fade_through_black`).
</Steps>
### Transition modes
The `Concatenate Videos` node offers three:
* **`cut`** — hard splice. Fastest. Total length = sum of inputs. Use this when the bridge frame is genuinely shared (clip 2's first frame = clip 1's last frame) — the seam is invisible.
* **`crossfade`** — linear A→B dissolve over `transition_frames`. Consumes `transition_frames` from both sides of each boundary. Total length = `sum(inputs) - transition_frames × (n-1)`. Use this when bridge frames don't quite match.
* **`fade_through_black`** — A fades to black, then B fades in from black. Total length is preserved. Use this for explicit scene changes.
### Quality degradation across iterations
A real failure mode of long chains: each iteration's reference image is itself a *generation output*, so artifacts compound. The model treats codec artifacts and VAE softness in the bridge frame as "style" and reproduces them in the next clip. By the 4th or 5th iteration you can see noticeable softening or color drift.
**Mitigations**:
1. **Pick a bridge frame a few back from the end** (e.g., `frame_index = -3` or `-5`). The very last frame is often the worst frame of a clip due to boundary effects in the temporal attention.
2. **Refresh the bridge frame with a low-strength img2img pass** before feeding it into the next I2V. An SDXL or FLUX img2img at strength ~0.2 with a quality-focused negative prompt (`blurry, low quality, compression artifacts`) noticeably suppresses the cumulative drift.
3. **Don't chain more than 4–5 clips** unless you're explicitly doing img2img refinement between each.
---
## Troubleshooting
### OOM errors
Video denoise is memory-intensive — attention scales roughly as `(T_lat × H/16 × W/16)²`, so resolution and frame count both quadratically affect peak VRAM.
* **Drop resolution before frame count.** Going from 1280×720 to 832×480 is a ~2.4× memory drop and visually subtle in most content. Going from 81 frames to 65 only saves ~20%.
* **TI2V-5B before A14B.** TI2V-5B Q4_K_M peaks around ~6–8 GB at 832×480, versus ~12–14 GB for A14B Q4_K_M. If you're at the OOM edge, switch model family.
* **OOM at the *reference image encoder* step** is usually allocator fragmentation from a previous run rather than absolute memory pressure. Restart the dev server and try again; if it recurs reproducibly, file an issue.
### Late-frame artifacts (text, color blobs)
If your video looks great for most of its duration but the last ~20% develops Asian text, watermarks, or floating colored shapes, **that's the model's training-data prior leaking through** as temporal coherence weakens at long temporal distance. It's particularly common on TI2V-5B (smaller model, less capacity to hold scene).
**Mitigations**:
* Add to the negative prompt: `text, watermark, logo, subtitles, chinese characters, kanji, ticker, banner`
* Use a more specific prompt — describe the action you want to *happen* through the clip, not just the static scene
* Bump CFG to 5.5 (TI2V-5B tolerates this)
* Stay at `num_frames=81`; values above push temporal RoPE out of distribution and artifacts accelerate
### "Dimensions must be multiples of 16/32"
The encoder and denoise nodes enforce these at runtime. Either:
* Use the **Wan 2.2 I2V Ideal Dimensions** node to compute valid (W, H) automatically from a source image, or
* Manually round to the right multiple (16 for A14B variants, 32 for TI2V-5B)
### Reference image / denoise dimension mismatch
If the denoise refuses with `Reference-image dimensions … must match denoise dimensions`, both nodes have their own width/height fields and they need to agree. Wire the same values (or the same Ideal Dimensions output) into both.
### TI2V-5B VAE state-dict load error
If `Latents to Video - Wan 2.2` fails with `Error(s) in loading state_dict for AutoencoderKLWan: ... size mismatch for ...`, you have the **wrong VAE installed** for the chosen transformer. TI2V-5B needs the **Wan 2.2 TI2V-5B VAE** (48-channel, Wan 2.2-VAE), not the A14B VAE (16-channel). Both are in the model manager — check the VAE field on the loader and the latents-to-video node.
### Sampler drift on the standalone TI2V-5B GGUF
Standalone GGUF installs don't ship a `scheduler/` config directory. InvokeAI now defaults to `UniPCMultistepScheduler` with the correct Wan-flow params when the model is TI2V-5B and there's no on-disk scheduler — but if you have an older install behaving oddly, the safer alternative is the **full Diffusers TI2V-5B** install (which includes the scheduler config).
### Preview images don't appear
Two known causes:
1. **A video is already loaded in the viewer.** If the last-selected gallery item is a video, the viewer renders the video element. The progress preview overlays on top of it. If you don't see it at all, **hard-refresh the browser** (`Ctrl+Shift+R` / `Cmd+Shift+R`) — Vite's bundle cache occasionally serves a stale build.
2. **`Show progress in viewer` is disabled.** Check the gallery settings (gear icon at the top of the gallery panel).
### Pipeline runs but the final MP4 is glitchy
This is almost always a **VAE mismatch** or a **scheduler mismatch** — both surface as garbage at the very end of the pipeline. Check that:
* The VAE matches the transformer family (16-ch for A14B, 48-ch for TI2V-5B)
* You're using the default (auto-selected) scheduler — manually overriding it is currently not supported
---
## Acknowledgements
Wan 2.2 model family by the Alibaba Wan-AI team. Lightning distillation LoRAs by [lightx2v](https://huggingface.co/lightx2v/Wan2.2-Lightning). GGUF quantizations by [QuantStack](https://huggingface.co/QuantStack).
@@ -0,0 +1,166 @@
---
title: Video Workflows
description: A plain-language guide to the ready-made Wan 2.2 video workflows that ship with InvokeAI — which one to pick, and how to run it.
lastUpdated: 2026-05-30
---
import { Card, CardGrid, Steps, LinkButton } from '@astrojs/starlight/components';
InvokeAI ships **ten ready-made video workflows** so you can start generating video without wiring up nodes yourself. Each one is a complete recipe — open it from the **Workflows** library, pick your models, type a prompt (and/or drop in an image), and press **Invoke**.
This page helps you choose the right workflow and run it. For how the models work under the hood, VRAM details, and troubleshooting, see the [Video Generation guide](/features/video-generation/).
:::caution[Experimental]
Video generation is a prototype feature. These workflows and the models behind them are new — expect rough edges, and check the [Video Generation guide](/features/video-generation/) if something misbehaves.
:::
---
## Before you start
**1. Install the video models.** Open the **Model Manager** and install a Wan 2.2 starter bundle (Text-to-Video and/or Image-to-Video). The [Video Generation guide explains the bundles and what fits your GPU](/features/video-generation/#installing-models).
**2. You'll select models when you open a workflow.** The shipped workflows come with their model slots **left empty on purpose** — every InvokeAI install stores models a little differently, so a workflow can't point at yours automatically. When you open one, look at its **Notes** panel (the note icon in the workflow editor): it lists exactly which model to pick in each slot. Drop those in once and you're set.
:::tip
Whenever a workflow won't run because a model field is empty, open its **Notes** — the shopping list of models to select is right there.
:::
---
## Which workflow should I use?
The workflows come in three families — **Text to Video**, **Image to Video**, and **Extend Video** — each with a high-quality option, a "concept LoRA" variant, and a lighter low-VRAM option — plus an **Interpolate** workflow that bridges two images and an **Extend Video to Image** variant that steers a continuation toward a target frame.
| I want to… | Use | Notes |
|---|---|---|
| Make a clip from a text description | **Text to Video - Wan 2.2 Lightning** | Best quality, fast. Start here. |
| Animate an existing image | **Image to Video - Wan 2.2 Lightning** | Your image becomes the first frame. |
| Make a video between two images | **Interpolate 2 Images to Video - Wan 2.2 Lightning** | Provide a start and end image; the model interpolates between them. |
| Make a video longer than ~5 seconds | **Extend Video - Wan 2.2 Lightning** | Continues an existing video and stitches the pieces together. |
| Extend a video toward a specific end frame | **Extend Video to Image - Wan 2.2 Lightning** | Continues a video and interpolates the new part to a target image. |
| Add my own style/subject LoRAs | the **w/ Concept LoRAs** variant of any of the above | Same as the base workflow, plus slots for your LoRAs. |
| Run on a smaller GPU (≈12–16 GB) | a **TI2V-5B (Low Quality)** workflow | Smaller model, lower memory, lower quality. |
If you're not sure, start with **Text to Video - Wan 2.2 Lightning** to get a feel for it, then move to Image-to-Video and Extend.
---
## Bundled video workflows
### Text to Video
<CardGrid>
<Card title="Text to Video - Wan 2.2 Lightning" icon="pencil">
Type a prompt, get a ~5-second clip. Uses the high-quality A14B model with the Lightning speed-up (just 4 steps). **The recommended starting point.**
</Card>
<Card title="Text to Video - …w/ Concept LoRAs" icon="pencil">
The same workflow, plus extra slots for your own concept LoRAs (a style or character you've trained or downloaded). Every LoRA slot must be filled — if you don't have concept LoRAs to add, use the plain version instead.
</Card>
<Card title="Text to Video - Wan 2.2 TI2V-5B (Low Quality)" icon="pencil">
Uses the smaller TI2V-5B model. Lower quality and slower per result, but fits comfortably on 12–16 GB GPUs. Good for quick drafts or smaller cards.
</Card>
</CardGrid>
### Image to Video
<CardGrid>
<Card title="Image to Video - Wan 2.2 Lightning" icon="star">
Drop in a starting image and a prompt; the model animates outward from your image as the first frame. Uses the A14B model + Lightning. **The recommended image-to-video starting point.**
</Card>
<Card title="Image to Video - …w/ Concept LoRAs" icon="star">
Same as above, with slots for your own concept LoRAs.
</Card>
<Card title="Image to Video - Wan 2.2 TI2V-5B (Low Quality)" icon="star">
Image-to-video on the smaller TI2V-5B model. Lower quality, lower VRAM — the low-memory option for animating an image.
</Card>
</CardGrid>
### Interpolate Between Two Images
<CardGrid>
<Card title="Interpolate 2 Images to Video - Wan 2.2 Lightning" icon="random">
Provide a **start image** and an **end image**; the model generates a clip that begins on the first and animates smoothly to the second (first-last-frame interpolation). Great for morphing between two stills or bridging two shots. Uses I2V-A14B + Lightning (4-step).
</Card>
</CardGrid>
:::caution[The "Concept LoRAs" variants need a LoRA in every slot]
A **concept LoRA** is an add-on that teaches the model a specific style, character, or subject. The *w/ Concept LoRAs* workflows are the same as their plain counterparts but add extra LoRA slots. **Every LoRA slot is required** — if any slot, including the extra concept ones, is left empty, the workflow won't run. Only use these variants when you actually have concept LoRAs to apply; otherwise use the plain version.
:::
:::note[About the "TI2V-5B (Low Quality)" variants]
"Low quality" here means *lower than the A14B Lightning workflows*, in exchange for running on much less video memory. On a 12–16 GB GPU they may be your best (or only) option, and they're handy everywhere for fast drafts.
:::
### Extend Video
The Wan models are trained on short (~5 second) clips, so the way to make something longer is to **continue** a video and stitch the pieces together. These workflows do that for you.
<CardGrid>
<Card title="Extend Video - Wan 2.2 Lightning" icon="clock">
Provide an existing video and use the sliders to choose where to continue from. The workflow generates a new clip starting from that point and joins it onto the original with a smooth transition. Run it repeatedly to keep growing the video.
</Card>
<Card title="Extend Video - …w/ Concept LoRAs" icon="clock">
Same as above, with slots for your own concept LoRAs so the continuation keeps your style or subject.
</Card>
<Card title="Extend Video to Image - Wan 2.2 Lightning" icon="clock">
Continue a video **toward a target image**. Provide a starting video and a destination image; the new segment interpolates from the video's last frame to your image (FLF2V), then joins onto the original with a smooth transition. Combines the Extend and Interpolate ideas — useful for steering where a continuation ends up.
</Card>
</CardGrid>
:::tip[Using "Extend Video to Image"]
This variant takes **two inputs** — the source video and a destination image:
1. Drop your **video** into the video input and use the **start/end frame sliders** to pick the slice to continue from.
2. Drop a **destination image** into the image input. The generated continuation interpolates from the video's last selected frame to this image, so the extended clip *ends on your image*.
3. **Invoke.** The new segment is concatenated onto the source with a cross-fade — tune the join (transition type and frame count) on the **Concatenate Videos** node.
Because the endpoint is pinned to your image, it's ideal for bridging a clip to a known next shot or landing on a specific final frame.
:::
---
## Running a workflow
<Steps>
1. **Open the Workflows tab** and load one of the workflows from the library.
2. **Read the Notes** (note icon) and select the models it lists — the main model, VAE, and text encoder, plus the Lightning LoRA pair on the A14B workflows.
3. **Set your inputs:**
- *Text to Video* — type a **positive prompt** describing the scene and motion.
- *Image to Video* — drop a starting **image** into the image field, and optionally add a prompt.
- *Extend Video* — provide a **starting video** and use the sliders to pick where the continuation begins.
4. **(Optional) adjust resolution.** The defaults are chosen to balance speed and memory. If you hit an out-of-memory error, lower the resolution preset — see the [VRAM tips](#which-one-fits-my-gpu) below.
5. **Press Invoke.** The finished MP4 appears in the gallery and plays inline in the viewer.
</Steps>
:::tip
Leave the **frame count** and **steps** at their defaults to start. They're already tuned for each workflow (for example, the Lightning workflows use only 4 steps, which is what makes them fast).
:::
---
## Which one fits my GPU?
A rough guide — exact numbers depend on resolution and your system. See [Video Generation → OOM errors](/features/video-generation/#oom-errors) for the full picture.
| Your GPU | Best choice |
|---|---|
| 24 GB and up | Any workflow at 720p. |
| 16–24 GB | A14B Lightning workflows at 480p; TI2V-5B at up to 720p. |
| 12–16 GB | The **TI2V-5B (Low Quality)** workflows. |
If you get an **out-of-memory** error, lower the resolution first (it helps far more than reducing the frame count), and switch to a TI2V-5B workflow if you're still over budget.
---
## Going further
These workflows are starting points. Once you're comfortable, the [Video Generation guide](/features/video-generation/) covers what's happening inside them — the model variants, how image conditioning works, the trick for chaining clips into longer videos, recommended parameters, and troubleshooting.
<LinkButton href="/features/video-generation/" variant="secondary">Read the Video Generation guide</LinkButton>
@@ -29,6 +29,8 @@ The requirements below are rough guidelines for best performance. GPUs with less
| FLUX.2 Klein 4B | 1024x1024 | Nvidia 30xx+ | 12GB | 16GB | FP8 works with 8GB+; Diffusers + encoder |
| FLUX.2 Klein 9B | 1024x1024 | Nvidia 40xx | 24GB | 32GB | FP8 works with 12GB+; Diffusers + encoder |
| Z-Image Turbo | 1024x1024 | Nvidia 20xx+ | 8GB | 16GB | Q4_K 8GB; Q8/BF16 16GB+ |
| Wan 2.2 A14B (T2V/I2V) | 1280x720 | Nvidia 30xx+ | 12GB | 32GB | Dual-expert MoE; Q4_K_M 12GB; Q8 18GB+; Diffusers requires 32GB+ |
| Wan 2.2 TI2V-5B | 1280x720 | Nvidia 20xx+ | 8GB | 16GB | Single transformer; Q4_K_M 6GB+; Q8 8GB+; Diffusers 12GB+ |
:::tip[`tmpfs` on Linux]
If your temporary directory is mounted as a `tmpfs`, ensure it has sufficient space.
+27 -1
View File
@@ -2,7 +2,7 @@
from typing import Annotated
from fastapi import Depends, HTTPException, status
from fastapi import Cookie, Depends, HTTPException, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from invokeai.app.api.dependencies import ApiDependencies
@@ -13,6 +13,18 @@ logger = logging.getLogger(__name__)
# HTTP Bearer token security scheme
security = HTTPBearer(auto_error=False)
MEDIA_TOKEN_COOKIE = "invokeai_media_token"
def _validate_token(token: str, invalid_detail: str) -> TokenData:
token_data = verify_token(token)
if token_data is None:
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail=invalid_detail)
user = ApiDependencies.invoker.services.users.get(token_data.user_id)
if user is None or not user.is_active:
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="User not found or inactive")
return token_data
async def get_current_user(
@@ -116,6 +128,19 @@ async def get_current_user_or_default(
return token_data
async def get_current_media_user_or_default(
credentials: Annotated[HTTPAuthorizationCredentials | None, Depends(security)],
media_token: Annotated[str | None, Cookie(alias=MEDIA_TOKEN_COOKIE)] = None,
) -> TokenData:
"""Authenticate media requests with a Bearer token or the path-scoped login cookie."""
if not ApiDependencies.invoker.services.configuration.multiuser:
return TokenData(user_id="system", email="[email protected]", is_admin=True)
token = credentials.credentials if credentials is not None else media_token
if token is None:
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Authentication required")
return _validate_token(token, "Invalid or expired token")
async def require_admin(
current_user: Annotated[TokenData, Depends(get_current_user)],
) -> TokenData:
@@ -162,5 +187,6 @@ async def require_admin_or_default(
# Type aliases for convenient use in route dependencies
CurrentUser = Annotated[TokenData, Depends(get_current_user)]
CurrentUserOrDefault = Annotated[TokenData, Depends(get_current_user_or_default)]
CurrentMediaUserOrDefault = Annotated[TokenData, Depends(get_current_media_user_or_default)]
AdminUser = Annotated[TokenData, Depends(require_admin)]
AdminUserOrDefault = Annotated[TokenData, Depends(require_admin_or_default)]
+17
View File
@@ -10,6 +10,7 @@ from invokeai.app.services.auth.token_service import set_jwt_secret
from invokeai.app.services.board_image_records.board_image_records_sqlite import SqliteBoardImageRecordStorage
from invokeai.app.services.board_images.board_images_default import BoardImagesService
from invokeai.app.services.board_records.board_records_sqlite import SqliteBoardRecordStorage
from invokeai.app.services.board_video_records.board_video_records_sqlite import SqliteBoardVideoRecordStorage
from invokeai.app.services.boards.boards_default import BoardService
from invokeai.app.services.bulk_download.bulk_download_default import BulkDownloadService
from invokeai.app.services.client_state_persistence.client_state_persistence_sqlite import ClientStatePersistenceSqlite
@@ -24,6 +25,7 @@ from invokeai.app.services.external_generation.providers import (
SeedreamProvider,
)
from invokeai.app.services.external_generation.startup import sync_configured_external_starter_models
from invokeai.app.services.gallery.gallery_default import SqliteGalleryService
from invokeai.app.services.image_files.image_files_disk import DiskImageFileStorage
from invokeai.app.services.image_moves.image_moves_default import ImageMoveService
from invokeai.app.services.image_records.image_records_sqlite import SqliteImageRecordStorage
@@ -52,6 +54,9 @@ from invokeai.app.services.style_preset_images.style_preset_images_disk import S
from invokeai.app.services.style_preset_records.style_preset_records_sqlite import SqliteStylePresetRecordsStorage
from invokeai.app.services.urls.urls_default import LocalUrlService
from invokeai.app.services.users.users_default import UserService
from invokeai.app.services.video_files.video_files_disk import DiskVideoFileStorage
from invokeai.app.services.video_records.video_records_sqlite import SqliteVideoRecordStorage
from invokeai.app.services.videos.videos_default import VideoService
from invokeai.app.services.workflow_records.workflow_records_sqlite import SqliteWorkflowRecordsStorage
from invokeai.app.services.workflow_thumbnails.workflow_thumbnails_disk import WorkflowThumbnailFileStorageDisk
from invokeai.backend.stable_diffusion.diffusion.conditioning_data import (
@@ -64,6 +69,7 @@ from invokeai.backend.stable_diffusion.diffusion.conditioning_data import (
QwenImageConditioningInfo,
SD3ConditioningInfo,
SDXLConditioningInfo,
WanConditioningInfo,
ZImageConditioningInfo,
)
from invokeai.backend.util.logging import InvokeAILogger
@@ -109,6 +115,7 @@ class ApiDependencies:
raise ValueError("Output folder is not set")
image_files = DiskImageFileStorage(f"{output_folder}/images")
video_files = DiskVideoFileStorage(f"{output_folder}/videos")
model_images_folder = config.models_path
style_presets_folder = config.style_presets_path
@@ -134,6 +141,10 @@ class ApiDependencies:
image_records = SqliteImageRecordStorage(db=db)
image_moves = ImageMoveService(db=db, image_files=image_files, config=configuration, logger=logger)
images = ImageService()
video_records = SqliteVideoRecordStorage(db=db)
videos = VideoService()
board_video_records = SqliteBoardVideoRecordStorage(db=db)
gallery = SqliteGalleryService(db=db)
invocation_cache = MemoryInvocationCache(max_cache_size=config.node_cache_size)
tensors = ObjectSerializerForwardCache(
ObjectSerializerDisk[torch.Tensor](
@@ -156,6 +167,7 @@ class ApiDependencies:
Ideogram4ConditioningInfo,
QwenImageConditioningInfo,
AnimaConditioningInfo,
WanConditioningInfo,
],
ephemeral=True,
),
@@ -226,6 +238,11 @@ class ApiDependencies:
workflow_thumbnails=workflow_thumbnails,
client_state_persistence=client_state_persistence,
users=users,
videos=videos,
video_files=video_files,
video_records=video_records,
board_video_records=board_video_records,
gallery=gallery,
)
ApiDependencies.invoker = Invoker(services)
+107 -7
View File
@@ -5,12 +5,23 @@ import string
from datetime import timedelta
from typing import Annotated
from fastapi import APIRouter, Body, HTTPException, Path, status
from fastapi import APIRouter, Body, Depends, HTTPException, Path, Request, Response, status
from fastapi.security import HTTPAuthorizationCredentials
from pydantic import BaseModel, Field, field_validator
from invokeai.app.api.auth_dependencies import AdminUser, CurrentUser
from invokeai.app.api.auth_dependencies import (
MEDIA_TOKEN_COOKIE,
AdminUser,
CurrentUser,
CurrentUserOrDefault,
security,
)
from invokeai.app.api.dependencies import ApiDependencies
from invokeai.app.services.auth.token_service import TokenData, create_access_token
from invokeai.app.services.auth.token_service import (
TokenData,
create_access_token,
get_token_remaining_seconds,
)
from invokeai.app.services.users.users_common import (
UserCreateRequest,
UserDTO,
@@ -74,6 +85,40 @@ class LogoutResponse(BaseModel):
success: bool = Field(description="Whether logout was successful")
class MediaCookieResponse(BaseModel):
"""Response from refreshing the media cookie."""
success: bool = Field(description="Whether the media cookie was set (always true in single-user mode)")
def _media_cookie_path(request: Request) -> str:
"""Public path prefix the media cookie is scoped to.
Behind a sub-path reverse proxy the browser sees routes under e.g. `/invoke/api/v1`,
so the cookie path must include the proxy prefix (advertised via `root_path` by
SubPathASGIMiddleware) or browsers will omit the cookie from media requests.
"""
return request.scope.get("root_path", "") + "/api/v1"
def _set_media_cookie(request: Request, response: Response, token: str, max_age_seconds: int) -> None:
"""Set the HttpOnly cookie that authenticates <video>/<img> media requests.
Media elements can't send Authorization headers, so the image and video media routes
accept the JWT via this path-scoped cookie instead. Shared by login and
/media-cookie so the cookie attributes can't drift between the two.
"""
response.set_cookie(
MEDIA_TOKEN_COOKIE,
token,
max_age=max_age_seconds,
httponly=True,
secure=request.url.scheme == "https",
samesite="lax",
path=_media_cookie_path(request),
)
class SetupStatusResponse(BaseModel):
"""Response for setup status check."""
@@ -119,7 +164,9 @@ async def get_setup_status() -> SetupStatusResponse:
@auth_router.post("/login", response_model=LoginResponse)
async def login(
request: Annotated[LoginRequest, Body(description="Login credentials")],
login_request: Annotated[LoginRequest, Body(description="Login credentials")],
request: Request,
response: Response,
) -> LoginResponse:
"""Authenticate user and return access token.
@@ -143,7 +190,7 @@ async def login(
)
user_service = ApiDependencies.invoker.services.users
user = user_service.authenticate(request.email, request.password)
user = user_service.authenticate(login_request.email, login_request.password)
if user is None:
raise HTTPException(
@@ -156,14 +203,17 @@ async def login(
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="User account is disabled")
# Create token with appropriate expiration
expires_delta = timedelta(days=TOKEN_EXPIRATION_REMEMBER_ME if request.remember_me else TOKEN_EXPIRATION_NORMAL)
expires_delta = timedelta(
days=TOKEN_EXPIRATION_REMEMBER_ME if login_request.remember_me else TOKEN_EXPIRATION_NORMAL
)
token_data = TokenData(
user_id=user.user_id,
email=user.email,
is_admin=user.is_admin,
remember_me=request.remember_me,
remember_me=login_request.remember_me,
)
token = create_access_token(token_data, expires_delta)
_set_media_cookie(request, response, token, int(expires_delta.total_seconds()))
return LoginResponse(
token=token,
@@ -175,6 +225,8 @@ async def login(
@auth_router.post("/logout", response_model=LogoutResponse)
async def logout(
current_user: CurrentUser,
request: Request,
response: Response,
) -> LogoutResponse:
"""Logout current user.
@@ -193,9 +245,57 @@ async def logout(
"""
# TODO: Implement token invalidation when server-side session management is added
# For now, this is a no-op since we use stateless JWT tokens
response.delete_cookie(MEDIA_TOKEN_COOKIE, path=_media_cookie_path(request))
return LogoutResponse(success=True)
@auth_router.post("/media-cookie", response_model=MediaCookieResponse)
async def refresh_media_cookie(
request: Request,
response: Response,
_current_user: CurrentUserOrDefault,
credentials: Annotated[HTTPAuthorizationCredentials | None, Depends(security)],
) -> MediaCookieResponse:
"""Re-issue the media cookie from a valid Bearer token.
The media cookie is normally set at login, but a session can hold a valid JWT
without it — the session may predate the cookie's introduction, or the cookie
may have been cleared while the JWT (in client storage) survived. Media
elements can't send Authorization headers, so such sessions silently fail to
load videos (black player) while every other API call works. The frontend
calls this on app load so an existing session self-heals without re-login.
The cookie's lifetime is clamped to the presented token's remaining validity —
it is the same JWT, so a longer-lived cookie would just yield 401s after
expiry anyway.
Returns:
MediaCookieResponse indicating the cookie was set. In single-user mode the
media routes don't require authentication, so this is a successful no-op.
Raises:
HTTPException: 401 if the Bearer token is missing, invalid, or expired, or
the user no longer exists or is inactive (raised by the auth dependency).
"""
config = ApiDependencies.invoker.services.configuration
if not config.multiuser:
return MediaCookieResponse(success=True)
# CurrentUserOrDefault has already validated the Bearer token (signature, expiry,
# user exists and is active) — in multiuser mode it 401s otherwise, so credentials
# cannot be None here. The raw token is still needed as the cookie value.
if credentials is None:
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Authentication required")
token = credentials.credentials
remaining = get_token_remaining_seconds(token)
if remaining is None:
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid or expired token")
_set_media_cookie(request, response, token, remaining)
return MediaCookieResponse(success=True)
@auth_router.get("/me", response_model=UserDTO)
async def get_current_user_info(
current_user: CurrentUser,
+61 -7
View File
@@ -22,6 +22,22 @@ class DeleteBoardResult(BaseModel):
description="The image names of the board-images relationships that were deleted."
)
deleted_images: list[str] = Field(description="The names of the images that were deleted.")
deleted_board_videos: list[str] = Field(
default_factory=list,
description="The video names of the board-videos relationships that were deleted.",
)
deleted_videos: list[str] = Field(
default_factory=list,
description="The names of the videos that were deleted.",
)
failed_images: list[str] = Field(
default_factory=list,
description="The names of images that could not be deleted and became uncategorized.",
)
failed_videos: list[str] = Field(
default_factory=list,
description="The names of videos that could not be deleted and became uncategorized.",
)
@boards_router.post(
@@ -103,10 +119,12 @@ async def update_board(
@boards_router.delete("/{board_id}", operation_id="delete_board", response_model=DeleteBoardResult)
async def delete_board(
def delete_board(
current_user: CurrentUserOrDefault,
board_id: str = Path(description="The id of board to delete"),
include_images: Optional[bool] = Query(description="Permanently delete all images on the board", default=False),
include_images: Optional[bool] = Query(
description="Permanently delete all images and videos on the board", default=False
),
) -> DeleteBoardResult:
"""Deletes a board (user must have access to it)"""
try:
@@ -117,20 +135,37 @@ async def delete_board(
if not current_user.is_admin and board.user_id != current_user.user_id:
raise HTTPException(status_code=403, detail="Not authorized to delete this board")
# Admins delete everything on the board; regular owners only delete their own
# contributions so that contributions from other users to a public/shared board
# are preserved (they cascade to "uncategorized" via FK on board_videos / board_images).
cascade_user_id: Optional[str] = None if current_user.is_admin else current_user.user_id
deleted_images: list[str] = []
deleted_videos: list[str] = []
try:
if include_images is True:
assert_image_move_maintenance_inactive()
deleted_images = ApiDependencies.invoker.services.board_images.get_all_board_image_names_for_board(
board_id=board_id,
categories=None,
is_intermediate=None,
# The services report both outcomes: records whose file delete failed are
# preserved (they cascade to "uncategorized" via the board FKs when the
# board is deleted below) and returned as failures. This is the ground
# truth — reconstructing failures by diffing a router-side board listing
# against the deleted names would double the DB work and misreport items
# moved or deleted concurrently between the two queries.
deleted_images, failed_images = ApiDependencies.invoker.services.images.delete_images_on_board(
board_id=board_id, user_id=cascade_user_id
)
deleted_videos, failed_videos = ApiDependencies.invoker.services.videos.delete_videos_on_board(
board_id=board_id, user_id=cascade_user_id
)
ApiDependencies.invoker.services.images.delete_images_on_board(board_id=board_id)
ApiDependencies.invoker.services.boards.delete(board_id=board_id)
return DeleteBoardResult(
board_id=board_id,
deleted_board_images=[],
deleted_images=deleted_images,
deleted_board_videos=[],
deleted_videos=deleted_videos,
failed_images=failed_images,
failed_videos=failed_videos,
)
else:
deleted_board_images = ApiDependencies.invoker.services.board_images.get_all_board_image_names_for_board(
@@ -138,15 +173,34 @@ async def delete_board(
categories=None,
is_intermediate=None,
)
deleted_board_videos = (
ApiDependencies.invoker.services.board_video_records.get_all_board_video_names_for_board(
board_id=board_id,
categories=None,
is_intermediate=None,
)
)
ApiDependencies.invoker.services.boards.delete(board_id=board_id)
return DeleteBoardResult(
board_id=board_id,
deleted_board_images=deleted_board_images,
deleted_images=[],
deleted_board_videos=deleted_board_videos,
deleted_videos=[],
)
except HTTPException:
raise
except Exception:
if include_images is True:
raise HTTPException(
status_code=500,
detail={
"message": "Failed to delete board after partially deleting media",
"deleted_images": deleted_images,
"deleted_videos": deleted_videos,
"board_deleted": False,
},
)
raise HTTPException(status_code=500, detail="Failed to delete board")
+99
View File
@@ -0,0 +1,99 @@
from typing import Optional
from fastapi import HTTPException, Query
from fastapi.routing import APIRouter
from invokeai.app.api.auth_dependencies import CurrentUserOrDefault
from invokeai.app.api.dependencies import ApiDependencies
from invokeai.app.api.routers.images import _assert_board_read_access
from invokeai.app.services.gallery.gallery_common import GalleryItem, GalleryItemNamesResult
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
from invokeai.app.services.shared.pagination import MAX_PAGE_SIZE, OffsetPaginatedResults
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
gallery_router = APIRouter(prefix="/v1/gallery", tags=["gallery"])
@gallery_router.get(
"/items/",
operation_id="list_gallery_items",
response_model=OffsetPaginatedResults[GalleryItem],
)
async def list_gallery_items(
current_user: CurrentUserOrDefault,
origin: Optional[ResourceOrigin] = Query(default=None, description="The origin of items to list."),
categories: Optional[list[ImageCategory]] = Query(
default=None,
description="The categories to include. Shared between images and videos.",
),
is_intermediate: Optional[bool] = Query(default=None, description="Whether to list intermediate items."),
board_id: Optional[str] = Query(
default=None,
description="The board id to filter by. Use 'none' to find items without a board.",
),
# Bounds matter: these flow verbatim into SQL, and a negative LIMIT means
# *unlimited* in SQLite — one request would materialize the entire gallery.
offset: int = Query(default=0, ge=0, description="The page offset"),
limit: int = Query(default=10, ge=0, le=MAX_PAGE_SIZE, description="The number of items per page"),
order_dir: SQLiteDirection = Query(default=SQLiteDirection.Descending, description="The order of sort"),
starred_first: bool = Query(default=True, description="Whether to sort by starred items first"),
search_term: Optional[str] = Query(default=None, description="The term to search for"),
) -> OffsetPaginatedResults[GalleryItem]:
"""Returns a paginated, time-sorted stream of polymorphic gallery items (images + videos)."""
if board_id is not None and board_id != "none":
_assert_board_read_access(board_id, current_user)
return ApiDependencies.invoker.services.gallery.list_items(
offset=offset,
limit=limit,
starred_first=starred_first,
order_dir=order_dir,
origin=origin,
categories=categories,
is_intermediate=is_intermediate,
board_id=board_id,
search_term=search_term,
user_id=current_user.user_id,
is_admin=current_user.is_admin,
)
@gallery_router.get(
"/items/names",
operation_id="get_gallery_item_names",
response_model=GalleryItemNamesResult,
)
async def get_gallery_item_names(
current_user: CurrentUserOrDefault,
origin: Optional[ResourceOrigin] = Query(default=None, description="The origin of items to list."),
categories: Optional[list[ImageCategory]] = Query(
default=None,
description="The categories to include. Shared between images and videos.",
),
is_intermediate: Optional[bool] = Query(default=None, description="Whether to list intermediate items."),
board_id: Optional[str] = Query(
default=None,
description="The board id to filter by. Use 'none' to find items without a board.",
),
order_dir: SQLiteDirection = Query(default=SQLiteDirection.Descending, description="The order of sort"),
starred_first: bool = Query(default=True, description="Whether to sort by starred items first"),
search_term: Optional[str] = Query(default=None, description="The term to search for"),
) -> GalleryItemNamesResult:
"""Returns an ordered (kind, name) list — used to drive virtualized gallery selection."""
if board_id is not None and board_id != "none":
_assert_board_read_access(board_id, current_user)
try:
return ApiDependencies.invoker.services.gallery.list_item_names(
starred_first=starred_first,
order_dir=order_dir,
origin=origin,
categories=categories,
is_intermediate=is_intermediate,
board_id=board_id,
search_term=search_term,
user_id=current_user.user_id,
is_admin=current_user.is_admin,
)
except Exception:
raise HTTPException(status_code=500, detail="Failed to get gallery item names")
+34 -15
View File
@@ -9,7 +9,7 @@ from fastapi.routing import APIRouter
from PIL import Image
from pydantic import BaseModel, Field, model_validator
from invokeai.app.api.auth_dependencies import CurrentUserOrDefault
from invokeai.app.api.auth_dependencies import CurrentMediaUserOrDefault, CurrentUserOrDefault
from invokeai.app.api.dependencies import ApiDependencies
from invokeai.app.api.extract_metadata_from_image import extract_metadata_from_image
from invokeai.app.api.routers._access import (
@@ -48,6 +48,12 @@ images_router = APIRouter(prefix="/v1/images", tags=["images"])
IMAGE_MAX_AGE = 31536000
def _get_image_cache_control() -> str:
if ApiDependencies.invoker.services.configuration.multiuser:
return "private, no-store"
return f"max-age={IMAGE_MAX_AGE}"
class ResizeToDimensions(BaseModel):
width: int = Field(..., gt=0)
height: int = Field(..., gt=0)
@@ -353,15 +359,15 @@ async def get_image_workflow(
},
)
async def get_image_full(
current_user: CurrentMediaUserOrDefault,
image_name: str = Path(description="The name of full-resolution image file to get"),
) -> Response:
"""Gets a full-resolution image file.
This endpoint is intentionally unauthenticated because browsers load images
via <img src> tags which cannot send Bearer tokens. Image names are UUIDs,
providing security through unguessability. Returns 409 while image storage
maintenance is active.
Browser media requests authenticate with the path-scoped HttpOnly cookie set at login.
Returns 409 while image storage maintenance is active.
"""
_assert_image_read_access(image_name, current_user)
assert_image_move_maintenance_inactive()
try:
@@ -369,7 +375,7 @@ async def get_image_full(
with open(path, "rb") as f:
content = f.read()
response = Response(content, media_type="image/png")
response.headers["Cache-Control"] = f"max-age={IMAGE_MAX_AGE}"
response.headers["Cache-Control"] = _get_image_cache_control()
response.headers["Content-Disposition"] = f'inline; filename="{image_name}"'
return response
except Exception:
@@ -389,15 +395,15 @@ async def get_image_full(
},
)
async def get_image_thumbnail(
current_user: CurrentMediaUserOrDefault,
image_name: str = Path(description="The name of thumbnail image file to get"),
) -> Response:
"""Gets a thumbnail image file.
This endpoint is intentionally unauthenticated because browsers load images
via <img src> tags which cannot send Bearer tokens. Image names are UUIDs,
providing security through unguessability. Returns 409 while image storage
maintenance is active.
Browser media requests authenticate with the path-scoped HttpOnly cookie set at login.
Returns 409 while image storage maintenance is active.
"""
_assert_image_read_access(image_name, current_user)
assert_image_move_maintenance_inactive()
try:
@@ -405,7 +411,7 @@ async def get_image_thumbnail(
with open(path, "rb") as f:
content = f.read()
response = Response(content, media_type="image/webp")
response.headers["Cache-Control"] = f"max-age={IMAGE_MAX_AGE}"
response.headers["Cache-Control"] = _get_image_cache_control()
return response
except Exception:
raise HTTPException(status_code=404)
@@ -492,9 +498,17 @@ async def delete_images_from_list(
raise
try:
# Skip — but do not re-raise — auth failures so a foreign name mid-batch doesn't
# discard the response payload for items the caller had already legitimately deleted.
# Without this, the client cache never learns about the partial successes and the
# already-deleted records reappear in the UI until the next full refresh.
deleted_images: set[str] = set()
failed_images: set[str] = set()
affected_boards: set[str] = set()
for image_name in image_names:
# Dedup while preserving order: a name repeated in the request would otherwise
# be processed twice, and the second pass's not-found error would land the same
# name in both deleted_images and failed_images.
for image_name in dict.fromkeys(image_names):
try:
_assert_image_owner(image_name, current_user)
image_dto = ApiDependencies.invoker.services.images.get_dto(image_name)
@@ -503,11 +517,14 @@ async def delete_images_from_list(
deleted_images.add(image_name)
affected_boards.add(board_id)
except HTTPException:
raise
continue
except Exception:
pass
# A genuine deletion failure (not an auth/404 skip) — report it so the
# client can surface a partial-failure warning, matching the video path.
failed_images.add(image_name)
return DeleteImagesResult(
deleted_images=list(deleted_images),
failed_images=list(failed_images),
affected_boards=list(affected_boards),
)
except HTTPException:
@@ -529,6 +546,7 @@ async def delete_uncategorized_images(
try:
deleted_images: set[str] = set()
failed_images: set[str] = set()
affected_boards: set[str] = set()
for image_name in image_names:
try:
@@ -540,9 +558,10 @@ async def delete_uncategorized_images(
# Skip images not owned by the current user
pass
except Exception:
pass
failed_images.add(image_name)
return DeleteImagesResult(
deleted_images=list(deleted_images),
failed_images=list(failed_images),
affected_boards=list(affected_boards),
)
except Exception:
+904
View File
@@ -0,0 +1,904 @@
import os
import re
import tempfile
import traceback
from collections.abc import Iterator
from pathlib import Path
from typing import Annotated, BinaryIO, Optional
from fastapi import Body, HTTPException, Query, Request, Response, UploadFile
from fastapi import Path as PathParam
from fastapi.responses import StreamingResponse
from fastapi.routing import APIRouter
from PIL import Image as PILImage
from pydantic import BaseModel, Field, StringConstraints, ValidationError
from starlette.concurrency import run_in_threadpool
from invokeai.app.api.auth_dependencies import CurrentMediaUserOrDefault, CurrentUserOrDefault
from invokeai.app.api.dependencies import ApiDependencies
from invokeai.app.api.routers.images import WorkflowAndGraphResponse, _assert_board_read_access
from invokeai.app.invocations.fields import MetadataField, MetadataFieldValidator
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
from invokeai.app.services.shared.pagination import MAX_PAGE_SIZE, OffsetPaginatedResults
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
from invokeai.app.services.video_records.video_records_common import VideoNamesResult, VideoRecordChanges
from invokeai.app.services.videos.videos_common import (
AddVideosToBoardResult,
DeleteVideosResult,
RemoveVideosFromBoardResult,
StarredVideosResult,
UnstarredVideosResult,
VideoDTO,
VideoUrlsDTO,
)
from invokeai.app.util.video_thumbnails import VideoDecodeTimeoutError, extract_video_frame, probe_video_with_codec
videos_router = APIRouter(prefix="/v1/videos", tags=["videos"])
# Videos are immutable; set a high max-age (1 year)
VIDEO_MAX_AGE = 31536000
# MP4 only — the names service emits `{uuid}.mp4` unconditionally and we don't transcode on
# upload. Accepting .mov/.webm/.mkv here previously caused those containers to be stored
# under a .mp4 name and served with the .mp4 MIME type, which silently broke playback in
# browsers when the container did not match.
ACCEPTED_VIDEO_MIME_PREFIXES = ("video/mp4",)
ACCEPTED_VIDEO_EXTENSIONS = (".mp4",)
# Per-chunk size for HTTP Range responses (1 MB)
RANGE_CHUNK_SIZE = 1024 * 1024
# Upload streaming chunk size (1 MB) and a coarse per-upload size cap. The cap is generous
# because Wan-generated MP4s for long sequences can run into the hundreds of megabytes;
# the goal is to prevent a single client from exhausting RAM, not to be a content policy.
UPLOAD_CHUNK_SIZE = 1024 * 1024
MAX_UPLOAD_SIZE = 1024 * 1024 * 1024 # 1 GB
# Pre-parse ingress cap enforced by VideoUploadLimitASGIMiddleware, applied to the whole
# request body *before* the multipart parser spools it to temp storage. Slightly larger
# than MAX_UPLOAD_SIZE to allow for multipart framing and the metadata form field.
MAX_UPLOAD_REQUEST_SIZE = MAX_UPLOAD_SIZE + 10 * 1024 * 1024
# Global bound on concurrent video uploads — each in-flight upload can hold up to two
# copies of the file in temp storage (the multipart spool + the route's own tmp file).
MAX_CONCURRENT_VIDEO_UPLOADS = 2
# Per-user bound (multiuser mode only): keeps one tenant's slow uploads from holding
# every global slot and starving the other users into 429s.
MAX_CONCURRENT_VIDEO_UPLOADS_PER_USER = 1
MAX_VIDEO_BATCH_SIZE = 1000
VideoName = Annotated[str, StringConstraints(max_length=255)]
class VideoNamesBatch(BaseModel):
video_names: list[VideoName] = Field(
max_length=MAX_VIDEO_BATCH_SIZE,
description="The list of video names to process",
)
def _get_video_cache_control() -> str:
if ApiDependencies.invoker.services.configuration.multiuser:
return "private, no-store"
return f"max-age={VIDEO_MAX_AGE}"
def _assert_video_owner(video_name: str, current_user: CurrentUserOrDefault) -> None:
"""Raise 403 if the current user does not own the video and is not an admin."""
from invokeai.app.services.board_records.board_records_common import BoardVisibility
if current_user.is_admin:
return
owner = ApiDependencies.invoker.services.video_records.get_user_id(video_name)
if owner is not None and owner == current_user.user_id:
return
board_id = ApiDependencies.invoker.services.board_video_records.get_board_for_video(video_name)
if board_id is not None:
try:
board = ApiDependencies.invoker.services.boards.get_dto(board_id=board_id)
if board.user_id == current_user.user_id:
return
if board.board_visibility == BoardVisibility.Public:
return
except Exception:
pass
raise HTTPException(status_code=403, detail="Not authorized to modify this video")
def _assert_video_direct_owner(video_name: str, current_user: CurrentUserOrDefault) -> None:
"""Raise 403 if the current user is not the direct owner of the video.
Intentionally stricter than _assert_video_owner: board-ownership and public-board
fallbacks are NOT honored. Mirrors _assert_image_direct_owner in board_images.py —
board-move operations need to verify the *original* owner, otherwise a user could
move someone else's video onto their own board via the board-owner branch.
"""
if current_user.is_admin:
return
owner = ApiDependencies.invoker.services.video_records.get_user_id(video_name)
if owner is not None and owner == current_user.user_id:
return
raise HTTPException(status_code=403, detail="Not authorized to move this video")
def _assert_board_write_access(board_id: str, current_user: CurrentUserOrDefault) -> None:
"""Raise 403 if the current user may not mutate the given board.
Mirrors _assert_board_write_access in board_images.py: admins and the board owner
may write; public boards accept contributions from any user.
"""
from invokeai.app.services.board_records.board_records_common import BoardVisibility
try:
board = ApiDependencies.invoker.services.boards.get_dto(board_id=board_id)
except Exception:
raise HTTPException(status_code=404, detail="Board not found")
if current_user.is_admin:
return
if board.user_id == current_user.user_id:
return
if board.board_visibility == BoardVisibility.Public:
return
raise HTTPException(status_code=403, detail="Not authorized to modify this board")
def _assert_video_read_access(video_name: str, current_user: CurrentUserOrDefault) -> None:
"""Raise 403 if the current user may not view the video."""
from invokeai.app.services.board_records.board_records_common import BoardVisibility
if current_user.is_admin:
return
owner = ApiDependencies.invoker.services.video_records.get_user_id(video_name)
if owner is not None and owner == current_user.user_id:
return
board_id = ApiDependencies.invoker.services.board_video_records.get_board_for_video(video_name)
if board_id is not None:
try:
board = ApiDependencies.invoker.services.boards.get_dto(board_id=board_id)
if board.board_visibility in (BoardVisibility.Shared, BoardVisibility.Public):
return
except Exception:
pass
raise HTTPException(status_code=403, detail="Not authorized to access this video")
def _is_accepted_video_upload(file: UploadFile) -> bool:
if file.content_type and file.content_type.startswith(ACCEPTED_VIDEO_MIME_PREFIXES):
return True
if file.filename:
return file.filename.lower().endswith(ACCEPTED_VIDEO_EXTENSIONS)
return False
def _is_mp4_file(path: Path) -> bool:
try:
with open(path, "rb") as video_file:
search_limit = min(path.stat().st_size, 64 * 1024)
position = 0
while position + 8 <= search_limit:
video_file.seek(position)
header = video_file.read(8)
box_size = int.from_bytes(header[:4], byteorder="big")
box_type = header[4:8]
header_size = 8
if box_size == 1:
extended_size = video_file.read(8)
if len(extended_size) != 8:
return False
box_size = int.from_bytes(extended_size, byteorder="big")
header_size = 16
if box_size < header_size:
return False
if box_type == b"ftyp":
major_brand = video_file.read(4)
return len(major_brand) == 4 and major_brand != b"qt "
position += box_size
except OSError:
return False
return False
def _probe_decodable_video(path: Path) -> tuple[tuple[int, int, float, Optional[float]], Optional[PILImage.Image]]:
"""Probes metadata and proves the video has a decodable first frame.
Returns the metadata plus the decoded frame so the save path can reuse it as the
thumbnail source instead of spawning another decode worker. A decode timeout is
contention on a loaded server, not evidence the video is bad — probe_video already
succeeded — so it yields (metadata, None) and the upload proceeds, with save-time
thumbnail extraction as the backstop.
"""
width, height, duration, fps, codec = probe_video_with_codec(path)
if codec is None or codec.lower() not in {"h264", "avc", "avc1", "libx264"}:
raise ValueError("Video must use a browser-compatible H.264/AVC codec")
metadata = (width, height, duration, fps)
try:
first_frame = extract_video_frame(path, frame_index=0, raise_on_timeout=True)
except VideoDecodeTimeoutError:
return metadata, None
if first_frame is None:
raise ValueError("Video has no decodable frame")
return metadata, first_frame
@videos_router.post(
"/upload",
operation_id="upload_video",
responses={
201: {"description": "The video was uploaded successfully"},
415: {"description": "Video upload failed"},
},
status_code=201,
response_model=VideoDTO,
)
async def upload_video(
current_user: CurrentUserOrDefault,
file: UploadFile,
request: Request,
response: Response,
video_category: ImageCategory = Query(description="The category of the video"),
is_intermediate: bool = Query(description="Whether this is an intermediate video"),
board_id: Optional[str] = Query(default=None, description="The board to add this video to, if any"),
session_id: Optional[str] = Query(default=None, description="The session ID associated with this upload, if any"),
metadata: Optional[str] = Body(
default=None,
description="The metadata to associate with the video, must be a stringified JSON dict",
embed=True,
),
) -> VideoDTO:
"""Uploads a video for the current user."""
if metadata is not None:
try:
MetadataFieldValidator.validate_json(metadata)
except ValidationError as e:
raise HTTPException(status_code=422, detail="Metadata must be a JSON object") from e
# Check board access for uploads to a specific board.
if board_id is not None:
from invokeai.app.services.board_records.board_records_common import BoardVisibility
try:
board = ApiDependencies.invoker.services.boards.get_dto(board_id=board_id)
except Exception:
raise HTTPException(status_code=404, detail="Board not found")
if (
not current_user.is_admin
and board.user_id != current_user.user_id
and board.board_visibility != BoardVisibility.Public
):
raise HTTPException(status_code=403, detail="Not authorized to upload to this board")
if not _is_accepted_video_upload(file):
raise HTTPException(status_code=415, detail="Not a supported video file")
# Stream the upload to a tmp file so we can probe and then hand its path to the service.
# Reading the full body into memory first risked exhausting RAM on multi-GB uploads;
# chunk-stream instead and enforce a hard size cap. Filesystem writes, container
# validation, ffmpeg probing, and thumbnail extraction are all blocking — run them in
# the thread pool so a slow (or hostile) file can't stall the event loop and every
# other API request with it.
tmp = tempfile.NamedTemporaryFile(prefix="invokeai_upload_", suffix=".mp4", delete=False)
tmp_path = Path(tmp.name)
try:
total = 0
while chunk := await file.read(UPLOAD_CHUNK_SIZE):
total += len(chunk)
if total > MAX_UPLOAD_SIZE:
tmp.close()
raise HTTPException(
status_code=413,
detail=f"Video upload exceeds maximum size ({MAX_UPLOAD_SIZE} bytes)",
)
await run_in_threadpool(tmp.write, chunk)
tmp.close()
# Release the multipart spool now that the body is copied: each in-flight
# upload otherwise holds TWO on-disk copies (Starlette's spool + our tmp file)
# through the probe/thumbnail/create phase — up to 2 x MAX_UPLOAD_SIZE x
# MAX_CONCURRENT_VIDEO_UPLOADS of temp disk. Closing shrinks the double-copy
# window to the copy loop itself.
await file.close()
if not await run_in_threadpool(_is_mp4_file, tmp_path):
raise HTTPException(status_code=415, detail="Not an MP4 video file")
try:
(width, height, duration, fps), first_frame = await run_in_threadpool(_probe_decodable_video, tmp_path)
except Exception:
ApiDependencies.invoker.services.logger.error(traceback.format_exc())
raise HTTPException(status_code=415, detail="Failed to read video")
try:
video_dto = await run_in_threadpool(
lambda: ApiDependencies.invoker.services.videos.create(
source_path=tmp_path,
width=width,
height=height,
duration=duration,
fps=fps,
first_frame=first_frame,
video_origin=ResourceOrigin.EXTERNAL,
video_category=video_category,
session_id=session_id,
board_id=board_id,
metadata=metadata,
workflow=None,
graph=None,
is_intermediate=is_intermediate,
user_id=current_user.user_id,
)
)
response.status_code = 201
response.headers["Location"] = video_dto.video_url
return video_dto
except Exception:
ApiDependencies.invoker.services.logger.error(traceback.format_exc())
raise HTTPException(status_code=500, detail="Failed to create video")
finally:
# If create() succeeded the file was moved; this unlink is a no-op then.
try:
tmp.close()
except Exception:
pass
try:
tmp_path.unlink(missing_ok=True)
except Exception:
pass
@videos_router.delete("/i/{video_name}", operation_id="delete_video", response_model=DeleteVideosResult)
async def delete_video(
current_user: CurrentUserOrDefault,
video_name: str = PathParam(description="The name of the video to delete"),
) -> DeleteVideosResult:
_assert_video_owner(video_name, current_user)
# Let service-level failures surface as 500s rather than swallowing them and returning a
# success-shaped response. A previous version of this handler caught everything and
# returned an empty ``deleted_videos`` list with HTTP 200; the frontend treated that as
# success, dropped the item from its cache, and the video stayed on disk — a silent
# data-consistency failure that only became visible on the next page reload.
try:
video_dto = ApiDependencies.invoker.services.videos.get_dto(video_name)
except Exception:
raise HTTPException(status_code=404, detail="Video not found")
board_id = video_dto.board_id or "none"
try:
ApiDependencies.invoker.services.videos.delete(video_name)
except Exception:
raise HTTPException(status_code=500, detail="Failed to delete video")
return DeleteVideosResult(
deleted_videos=[video_name],
failed_videos=[],
affected_boards=[board_id],
)
@videos_router.post("/delete", operation_id="delete_videos_from_list", response_model=DeleteVideosResult)
def delete_videos_from_list(
current_user: CurrentUserOrDefault,
batch: VideoNamesBatch,
) -> DeleteVideosResult:
# Skip — but do not re-raise — auth failures so a foreign name mid-batch doesn't
# discard the response payload for items the caller had already legitimately deleted.
# Without this, the client cache never learns about the partial successes and the
# already-deleted records reappear in the UI until the next full refresh.
#
# HTTPException here means an intentional skip (a foreign name, or a 404 because a
# concurrent session deleted it first) — not a failed deletion. Reporting those in
# failed_videos would toast a spurious "could not be deleted" warning; the images
# endpoints skip them silently, and mixed selections must behave consistently.
deleted_videos: set[str] = set()
failed_videos: set[str] = set()
affected_boards: set[str] = set()
# Dedup while preserving order: a name repeated in the request would otherwise be
# processed twice, and the second pass's not-found error would land the same name
# in both deleted_videos and failed_videos.
for video_name in dict.fromkeys(batch.video_names):
try:
_assert_video_owner(video_name, current_user)
video_dto = ApiDependencies.invoker.services.videos.get_dto(video_name)
board_id = video_dto.board_id or "none"
ApiDependencies.invoker.services.videos.delete(video_name)
deleted_videos.add(video_name)
affected_boards.add(board_id)
except HTTPException:
continue
except Exception:
failed_videos.add(video_name)
return DeleteVideosResult(
deleted_videos=list(deleted_videos),
failed_videos=list(failed_videos),
affected_boards=list(affected_boards),
)
@videos_router.delete("/uncategorized", operation_id="delete_uncategorized_videos", response_model=DeleteVideosResult)
def delete_uncategorized_videos(
current_user: CurrentUserOrDefault,
) -> DeleteVideosResult:
"""Deletes all uncategorized videos owned by the current user (or all if admin).
Mirrors ``delete_uncategorized_images`` so the "Delete All Uncategorized
Images/Videos" board action covers both media kinds.
"""
names_result = ApiDependencies.invoker.services.videos.get_video_names(
board_id="none",
user_id=current_user.user_id,
is_admin=current_user.is_admin,
)
deleted_videos: set[str] = set()
failed_videos: set[str] = set()
affected_boards: set[str] = set()
for video_name in names_result.video_names:
try:
_assert_video_owner(video_name, current_user)
ApiDependencies.invoker.services.videos.delete(video_name)
deleted_videos.add(video_name)
affected_boards.add("none")
except HTTPException:
# Skip videos not owned by the current user — an intentional skip, not a
# failed deletion, so it must not be reported (and toasted) as one.
continue
except Exception:
failed_videos.add(video_name)
return DeleteVideosResult(
deleted_videos=list(deleted_videos),
failed_videos=list(failed_videos),
affected_boards=list(affected_boards),
)
@videos_router.patch("/i/{video_name}", operation_id="update_video", response_model=VideoDTO)
async def update_video(
current_user: CurrentUserOrDefault,
video_name: str = PathParam(description="The name of the video to update"),
video_changes: VideoRecordChanges = Body(description="The changes to apply to the video"),
) -> VideoDTO:
_assert_video_owner(video_name, current_user)
try:
return ApiDependencies.invoker.services.videos.update(video_name, video_changes)
except Exception:
raise HTTPException(status_code=400, detail="Failed to update video")
@videos_router.get("/i/{video_name}", operation_id="get_video_dto", response_model=VideoDTO)
async def get_video_dto(
current_user: CurrentUserOrDefault,
video_name: str = PathParam(description="The name of video to get"),
) -> VideoDTO:
_assert_video_read_access(video_name, current_user)
try:
return ApiDependencies.invoker.services.videos.get_dto(video_name)
except Exception:
raise HTTPException(status_code=404)
@videos_router.get(
"/i/{video_name}/metadata", operation_id="get_video_metadata", response_model=Optional[MetadataField]
)
async def get_video_metadata(
current_user: CurrentUserOrDefault,
video_name: str = PathParam(description="The name of video to get"),
) -> Optional[MetadataField]:
_assert_video_read_access(video_name, current_user)
try:
return ApiDependencies.invoker.services.videos.get_metadata(video_name)
except Exception:
raise HTTPException(status_code=404)
@videos_router.get(
"/i/{video_name}/workflow", operation_id="get_video_workflow", response_model=WorkflowAndGraphResponse
)
async def get_video_workflow(
current_user: CurrentUserOrDefault,
video_name: str = PathParam(description="The name of video whose workflow to get"),
) -> WorkflowAndGraphResponse:
"""Gets the workflow and graph saved with a generated video (mirrors the image route)."""
_assert_video_read_access(video_name, current_user)
try:
workflow = ApiDependencies.invoker.services.videos.get_workflow(video_name)
graph = ApiDependencies.invoker.services.videos.get_graph(video_name)
return WorkflowAndGraphResponse(workflow=workflow, graph=graph)
except Exception:
raise HTTPException(status_code=404)
def _parse_range_header(range_header: str, file_size: int) -> Optional[tuple[int, int]]:
"""Parses an HTTP Range header of the form `bytes=START-END`. Returns inclusive (start, end)
byte offsets, or None if the header is malformed or unsatisfiable."""
match = re.match(r"^bytes=(\d*)-(\d*)$", range_header.strip())
if match is None:
return None
if file_size <= 0:
# No byte range is satisfiable against an empty file (a suffix range would
# otherwise "satisfy" with the invalid pair (0, -1)).
return None
start_str, end_str = match.group(1), match.group(2)
if start_str == "" and end_str == "":
return None
if start_str == "":
# suffix range: last N bytes
try:
suffix_len = int(end_str)
except ValueError:
return None
if suffix_len == 0:
return None
start = max(file_size - suffix_len, 0)
end = file_size - 1
else:
try:
start = int(start_str)
except ValueError:
return None
if end_str == "":
end = file_size - 1
else:
try:
end = int(end_str)
except ValueError:
return None
if start > end or start >= file_size:
return None
end = min(end, file_size - 1)
return start, end
@videos_router.get(
"/i/{video_name}/full",
operation_id="get_video_full",
response_class=Response,
responses={
200: {"description": "Return the full video file", "content": {"video/mp4": {}}},
206: {"description": "Return a byte-range of the video file", "content": {"video/mp4": {}}},
404: {"description": "Video not found"},
},
)
@videos_router.head(
"/i/{video_name}/full",
operation_id="get_video_full_head",
response_class=Response,
responses={
200: {"description": "Return the full video file", "content": {"video/mp4": {}}},
404: {"description": "Video not found"},
},
)
async def get_video_full(
request: Request,
current_user: CurrentMediaUserOrDefault,
video_name: str = PathParam(description="The name of video file to get"),
) -> Response:
"""Serves the video file with HTTP Range support so HTML5 <video> seek/scrub works.
Browser media requests authenticate with the path-scoped HttpOnly cookie set at login.
"""
_assert_video_read_access(video_name, current_user)
try:
path_str = ApiDependencies.invoker.services.videos.get_path(video_name, thumbnail=False)
except Exception:
raise HTTPException(status_code=404)
# Open once and serve every branch from the fd. Deletion stages files away via an
# atomic rename, so any later path-based stat/open — including FileResponse's lazy
# open after the route returns — races with a concurrent delete and surfaces as an
# uncontrolled 500. An open fd is immune: the data stays readable until the handle
# closes, even after the path is gone.
video_file: Optional[BinaryIO] = None
try:
video_file = open(path_str, "rb")
except OSError:
raise HTTPException(status_code=404)
try:
file_size = os.fstat(video_file.fileno()).st_size
range_header = request.headers.get("range") or request.headers.get("Range")
common_headers = {
"Accept-Ranges": "bytes",
"Cache-Control": _get_video_cache_control(),
"Content-Disposition": f'inline; filename="{video_name}"',
}
# HEAD: respond with metadata only.
if request.method == "HEAD":
return Response(
status_code=200,
media_type="video/mp4",
headers={**common_headers, "Content-Length": str(file_size)},
)
if range_header is None:
# Stream from the already-open fd in bounded chunks rather than reading the
# file into RAM — multi-GB MP4 downloads (clients without Range, CLI tools,
# CDN edge fetches) would otherwise allocate a multi-GB bytes object. This
# trades FileResponse's sendfile() for delete-race immunity.
handle = video_file
video_file = None # ownership moves to the streaming iterator
def iter_video() -> Iterator[bytes]:
try:
while chunk := handle.read(RANGE_CHUNK_SIZE):
yield chunk
finally:
handle.close()
return StreamingResponse(
iter_video(),
media_type="video/mp4",
headers={**common_headers, "Content-Length": str(file_size)},
)
parsed = _parse_range_header(range_header, file_size)
if parsed is None:
# Unsatisfiable range.
return Response(
status_code=416,
headers={**common_headers, "Content-Range": f"bytes */{file_size}"},
)
start, end = parsed
length = end - start + 1
video_file.seek(start)
# Read at most one chunk; clients ask for more via subsequent ranges.
read_length = min(length, RANGE_CHUNK_SIZE)
chunk = video_file.read(read_length)
actual_end = start + len(chunk) - 1
return Response(
chunk,
status_code=206,
media_type="video/mp4",
headers={
**common_headers,
"Content-Range": f"bytes {start}-{actual_end}/{file_size}",
"Content-Length": str(len(chunk)),
},
)
finally:
if video_file is not None:
video_file.close()
@videos_router.get(
"/i/{video_name}/thumbnail",
operation_id="get_video_thumbnail",
response_class=Response,
responses={
200: {"description": "Return the video thumbnail", "content": {"image/webp": {}}},
404: {"description": "Video not found"},
},
)
async def get_video_thumbnail(
current_user: CurrentMediaUserOrDefault,
video_name: str = PathParam(description="The name of thumbnail file to get"),
) -> Response:
"""Returns the first-frame WebP thumbnail of an authorized video."""
_assert_video_read_access(video_name, current_user)
try:
path = ApiDependencies.invoker.services.videos.get_path(video_name, thumbnail=True)
except Exception:
raise HTTPException(status_code=404)
try:
with open(path, "rb") as thumbnail_file:
thumbnail = thumbnail_file.read()
except OSError:
raise HTTPException(status_code=404)
return Response(
thumbnail,
media_type="image/webp",
headers={"Cache-Control": _get_video_cache_control()},
)
@videos_router.get("/i/{video_name}/urls", operation_id="get_video_urls", response_model=VideoUrlsDTO)
async def get_video_urls(
current_user: CurrentUserOrDefault,
video_name: str = PathParam(description="The name of the video whose URL to get"),
) -> VideoUrlsDTO:
_assert_video_read_access(video_name, current_user)
try:
video_url = ApiDependencies.invoker.services.videos.get_url(video_name)
thumbnail_url = ApiDependencies.invoker.services.videos.get_url(video_name, thumbnail=True)
return VideoUrlsDTO(video_name=video_name, video_url=video_url, thumbnail_url=thumbnail_url)
except Exception:
raise HTTPException(status_code=404)
@videos_router.get("/", operation_id="list_video_dtos", response_model=OffsetPaginatedResults[VideoDTO])
async def list_video_dtos(
current_user: CurrentUserOrDefault,
video_origin: Optional[ResourceOrigin] = Query(default=None, description="The origin of videos to list."),
categories: Optional[list[ImageCategory]] = Query(default=None, description="The categories of video to include."),
is_intermediate: Optional[bool] = Query(default=None, description="Whether to list intermediate videos."),
board_id: Optional[str] = Query(
default=None,
description="The board id to filter by. Use 'none' to find videos without a board.",
),
# Bounds matter: these flow verbatim into SQL, and a negative LIMIT means
# *unlimited* in SQLite — one request would materialize every video row.
offset: int = Query(default=0, ge=0, description="The page offset"),
limit: int = Query(default=10, ge=0, le=MAX_PAGE_SIZE, description="The number of videos per page"),
order_dir: SQLiteDirection = Query(default=SQLiteDirection.Descending, description="The order of sort"),
starred_first: bool = Query(default=True, description="Whether to sort by starred videos first"),
search_term: Optional[str] = Query(default=None, description="The term to search for"),
) -> OffsetPaginatedResults[VideoDTO]:
"""Gets a list of video DTOs for the current user."""
# Validate that the caller can read from this board. "none" is handled by the SQL layer.
if board_id is not None and board_id != "none":
_assert_board_read_access(board_id, current_user)
return ApiDependencies.invoker.services.videos.get_many(
offset,
limit,
starred_first,
order_dir,
video_origin,
categories,
is_intermediate,
board_id,
search_term,
current_user.user_id,
current_user.is_admin,
)
@videos_router.get("/names", operation_id="get_video_names")
async def get_video_names(
current_user: CurrentUserOrDefault,
video_origin: Optional[ResourceOrigin] = Query(default=None, description="The origin of videos to list."),
categories: Optional[list[ImageCategory]] = Query(default=None, description="The categories of video to include."),
is_intermediate: Optional[bool] = Query(default=None, description="Whether to list intermediate videos."),
board_id: Optional[str] = Query(
default=None,
description="The board id to filter by. Use 'none' to find videos without a board.",
),
order_dir: SQLiteDirection = Query(default=SQLiteDirection.Descending, description="The order of sort"),
starred_first: bool = Query(default=True, description="Whether to sort by starred videos first"),
search_term: Optional[str] = Query(default=None, description="The term to search for"),
) -> VideoNamesResult:
"""Gets ordered list of video names with metadata for optimistic updates."""
# Validate that the caller can read from this board. "none" is handled by the SQL layer.
if board_id is not None and board_id != "none":
_assert_board_read_access(board_id, current_user)
try:
return ApiDependencies.invoker.services.videos.get_video_names(
starred_first=starred_first,
order_dir=order_dir,
video_origin=video_origin,
categories=categories,
is_intermediate=is_intermediate,
board_id=board_id,
search_term=search_term,
user_id=current_user.user_id,
is_admin=current_user.is_admin,
)
except Exception:
raise HTTPException(status_code=500, detail="Failed to get video names")
@videos_router.post("/star", operation_id="star_videos_in_list", response_model=StarredVideosResult)
def star_videos_in_list(
current_user: CurrentUserOrDefault,
batch: VideoNamesBatch,
) -> StarredVideosResult:
# Skip — but do not re-raise — auth failures so a foreign name mid-batch doesn't
# discard the response payload for items that were already starred. Mirrors
# delete_videos_from_list: re-raising turned partial successes into an error-shaped
# response, so the client never invalidated caches for the videos that did change.
starred_videos: set[str] = set()
failed_videos: set[str] = set()
affected_boards: set[str] = set()
for video_name in dict.fromkeys(batch.video_names):
try:
_assert_video_owner(video_name, current_user)
updated = ApiDependencies.invoker.services.videos.update(
video_name, changes=VideoRecordChanges(starred=True)
)
starred_videos.add(video_name)
affected_boards.add(updated.board_id or "none")
except HTTPException:
continue
except Exception:
failed_videos.add(video_name)
return StarredVideosResult(
starred_videos=list(starred_videos),
failed_videos=list(failed_videos),
affected_boards=list(affected_boards),
)
@videos_router.post("/unstar", operation_id="unstar_videos_in_list", response_model=UnstarredVideosResult)
def unstar_videos_in_list(
current_user: CurrentUserOrDefault,
batch: VideoNamesBatch,
) -> UnstarredVideosResult:
# See star_videos_in_list: skip foreign names instead of re-raising mid-batch.
unstarred_videos: set[str] = set()
failed_videos: set[str] = set()
affected_boards: set[str] = set()
for video_name in dict.fromkeys(batch.video_names):
try:
_assert_video_owner(video_name, current_user)
updated = ApiDependencies.invoker.services.videos.update(
video_name, changes=VideoRecordChanges(starred=False)
)
unstarred_videos.add(video_name)
affected_boards.add(updated.board_id or "none")
except HTTPException:
continue
except Exception:
failed_videos.add(video_name)
return UnstarredVideosResult(
unstarred_videos=list(unstarred_videos),
failed_videos=list(failed_videos),
affected_boards=list(affected_boards),
)
class VideoBoardArg(BaseModel):
board_id: str = Field(description="The id of the board to add or remove the video from")
video_name: str = Field(description="The name of the video to add to / remove from the board")
@videos_router.post(
"/board",
operation_id="add_video_to_board",
response_model=AddVideosToBoardResult,
)
async def add_video_to_board(
current_user: CurrentUserOrDefault,
arg: VideoBoardArg = Body(),
) -> AddVideosToBoardResult:
_assert_board_write_access(arg.board_id, current_user)
_assert_video_direct_owner(arg.video_name, current_user)
try:
# Capture the source board BEFORE mutating so the frontend can invalidate both
# the old and new board caches. Mirrors add_image_to_board.
old_board_id = (
ApiDependencies.invoker.services.board_video_records.get_board_for_video(arg.video_name) or "none"
)
ApiDependencies.invoker.services.board_video_records.add_video_to_board(
board_id=arg.board_id, video_name=arg.video_name
)
return AddVideosToBoardResult(
added_videos=[arg.video_name],
affected_boards=list({arg.board_id, old_board_id}),
)
except Exception:
raise HTTPException(status_code=500, detail="Failed to add video to board")
@videos_router.delete(
"/board",
operation_id="remove_video_from_board",
response_model=RemoveVideosFromBoardResult,
)
async def remove_video_from_board(
current_user: CurrentUserOrDefault,
video_name: str = Body(description="The name of the video to remove from its board", embed=True),
) -> RemoveVideosFromBoardResult:
# A video association can be removed by EITHER the direct video owner OR a user with
# write access to the destination board (admin, board owner, or any contributor when the
# board is Public). This mirrors remove_image_from_board and prevents a video from being
# stranded when a non-owner uploads into a Public board that is later made Shared/Private:
# without the board-write fallback, neither the uploader nor the board owner could
# detach the video. See PR #9163 review.
old_board_id = ApiDependencies.invoker.services.board_video_records.get_board_for_video(video_name)
try:
_assert_video_direct_owner(video_name, current_user)
except HTTPException:
if old_board_id is None:
raise
_assert_board_write_access(old_board_id, current_user)
try:
ApiDependencies.invoker.services.board_video_records.remove_video_from_board(video_name=video_name)
return RemoveVideosFromBoardResult(
removed_videos=[video_name],
affected_boards=list({old_board_id or "none", "none"}),
)
except Exception:
raise HTTPException(status_code=500, detail="Failed to remove video from board")
+34 -3
View File
@@ -3,6 +3,7 @@ from fastapi.routing import APIRouter
from invokeai.app.api.auth_dependencies import CurrentUserOrDefault
from invokeai.app.api.dependencies import ApiDependencies
from invokeai.app.services.gallery.gallery_common import GalleryItemNamesResult
from invokeai.app.services.image_records.image_records_common import ImageCategory, ImageNamesResult
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
from invokeai.app.services.virtual_boards.virtual_boards_common import VirtualSubBoardDTO
@@ -18,9 +19,9 @@ virtual_boards_router = APIRouter(prefix="/v1/virtual_boards", tags=["virtual_bo
async def list_virtual_boards_by_date(
current_user: CurrentUserOrDefault,
) -> list[VirtualSubBoardDTO]:
"""Gets a list of virtual sub-boards grouped by date."""
"""Gets a list of virtual sub-boards grouped by date. Covers both images and videos."""
try:
return ApiDependencies.invoker.services.image_records.get_image_dates(
return ApiDependencies.invoker.services.gallery.get_dates(
user_id=current_user.user_id,
is_admin=current_user.is_admin,
)
@@ -41,7 +42,8 @@ async def list_virtual_board_image_names_by_date(
categories: list[ImageCategory] | None = Query(default=None, description="The categories of images to include"),
search_term: str | None = Query(default=None, description="Search term to filter images"),
) -> ImageNamesResult:
"""Gets ordered image names for a specific date."""
"""Gets ordered image names for a specific date. Image-only; kept for API compatibility —
the UI uses the polymorphic `/by_date/{date}/item_names` endpoint."""
try:
return ApiDependencies.invoker.services.image_records.get_image_names_by_date(
date=date,
@@ -54,3 +56,32 @@ async def list_virtual_board_image_names_by_date(
)
except Exception:
raise HTTPException(status_code=500, detail="Failed to get image names for date")
@virtual_boards_router.get(
"/by_date/{date}/item_names",
operation_id="list_virtual_board_item_names_by_date",
response_model=GalleryItemNamesResult,
)
async def list_virtual_board_item_names_by_date(
current_user: CurrentUserOrDefault,
date: str = Path(description="The ISO date string, e.g. '2026-03-18'"),
starred_first: bool = Query(default=True, description="Whether to sort starred items first"),
order_dir: SQLiteDirection = Query(default=SQLiteDirection.Descending, description="The sort direction"),
categories: list[ImageCategory] | None = Query(default=None, description="The categories of items to include"),
search_term: str | None = Query(default=None, description="Search term to filter items"),
) -> GalleryItemNamesResult:
"""Gets ordered polymorphic (image + video) item refs for a specific date."""
try:
return ApiDependencies.invoker.services.gallery.list_item_names(
starred_first=starred_first,
order_dir=order_dir,
categories=categories,
is_intermediate=False,
search_term=search_term,
user_id=current_user.user_id,
is_admin=current_user.is_admin,
created_date=date,
)
except Exception:
raise HTTPException(status_code=500, detail="Failed to get gallery item names for date")
+195 -6
View File
@@ -1,5 +1,7 @@
import asyncio
import inspect
import logging
from collections.abc import Awaitable, Callable
from contextlib import asynccontextmanager
from pathlib import Path
@@ -7,11 +9,14 @@ from fastapi import FastAPI, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.middleware.gzip import GZipMiddleware
from fastapi.openapi.docs import get_redoc_html, get_swagger_ui_html
from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse
from fastapi.security.utils import get_authorization_scheme_param
from fastapi_events.handlers.local import local_handler
from fastapi_events.middleware import EventHandlerASGIMiddleware
from starlette.concurrency import run_in_threadpool
from starlette.datastructures import Headers
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.types import ASGIApp, Receive, Scope, Send
from starlette.types import ASGIApp, Message, Receive, Scope, Send
import invokeai.frontend.web as web_dir
from invokeai.app.api.dependencies import ApiDependencies
@@ -24,6 +29,7 @@ from invokeai.app.api.routers import (
client_state,
custom_nodes,
download_queue,
gallery,
image_moves,
images,
model_manager,
@@ -32,6 +38,7 @@ from invokeai.app.api.routers import (
session_queue,
style_presets,
utilities,
videos,
virtual_boards,
workflows,
)
@@ -99,18 +106,36 @@ class SlidingWindowTokenMiddleware(BaseHTTPMiddleware):
# genuine user activity. GET requests are often background fetches (RTK Query
# cache revalidation, refetch-on-focus, etc.) and should not reset the
# inactivity timer.
if response.status_code < 400 and request.method in ("POST", "PUT", "PATCH", "DELETE"):
# Behind a sub-path proxy the public path carries the prefix (see SubPathASGIMiddleware),
# so strip `root_path` before matching the auth-route exclusions — otherwise a proxied
# logout would mint a replacement token/cookie right after the route deleted them.
route_path = request.url.path.removeprefix(request.scope.get("root_path", ""))
if (
response.status_code < 400
and request.method in ("POST", "PUT", "PATCH", "DELETE")
and route_path not in ("/api/v1/auth/login", "/api/v1/auth/logout", "/api/v1/auth/media-cookie")
):
auth_header = request.headers.get("authorization", "")
if auth_header.startswith("Bearer "):
token = auth_header[7:]
try:
from datetime import timedelta
from invokeai.app.api.routers.auth import TOKEN_EXPIRATION_NORMAL, TOKEN_EXPIRATION_REMEMBER_ME
from invokeai.app.services.auth.token_service import create_access_token, verify_token
from invokeai.app.api.routers.auth import (
TOKEN_EXPIRATION_NORMAL,
TOKEN_EXPIRATION_REMEMBER_ME,
)
from invokeai.app.services.auth.token_service import TokenData, create_access_token, verify_token
token_data = verify_token(token)
if token_data is not None:
# The user lookup is a synchronous SQLite query behind a
# process-wide lock; run it off the event loop so a contended
# lock (e.g. generation-result writes) can't stall every
# concurrent request from inside this per-mutation middleware.
user = await run_in_threadpool(ApiDependencies.invoker.services.users.get, token_data.user_id)
if user is None or not user.is_active:
return response
# Use the remember_me claim from the token to determine the
# correct refresh duration. This avoids the bug where a 7-day
# token with <24h remaining would be silently downgraded to 1 day.
@@ -119,7 +144,13 @@ class SlidingWindowTokenMiddleware(BaseHTTPMiddleware):
else:
expires_delta = timedelta(days=TOKEN_EXPIRATION_NORMAL)
new_token = create_access_token(token_data, expires_delta)
refreshed_data = TokenData(
user_id=user.user_id,
email=user.email,
is_admin=user.is_admin,
remember_me=token_data.remember_me,
)
new_token = create_access_token(refreshed_data, expires_delta)
response.headers["X-Refreshed-Token"] = new_token
except Exception:
pass # Don't fail the request if token refresh fails
@@ -148,6 +179,155 @@ class RedirectRootWithQueryStringMiddleware(BaseHTTPMiddleware):
return response
def _identify_video_upload_user(scope: Scope) -> tuple[bool, str | None]:
"""Identify the uploader for auth and per-user slot accounting.
Returns (authenticated, per_user_key). In single-user mode there is only one
tenant, so per_user_key is None — the whole global capacity is theirs and no
per-user quota applies.
"""
if not ApiDependencies.invoker.services.configuration.multiuser:
return True, None
scheme, token = get_authorization_scheme_param(Headers(scope=scope).get("authorization"))
if scheme.lower() != "bearer":
return False, None
from invokeai.app.services.auth.token_service import verify_token
token_data = verify_token(token)
if token_data is None:
return False, None
user = ApiDependencies.invoker.services.users.get(token_data.user_id)
if user is None or not user.is_active:
return False, None
return True, token_data.user_id
async def _identify_video_upload_user_async(scope: Scope) -> tuple[bool, str | None]:
return await run_in_threadpool(_identify_video_upload_user, scope)
class VideoUploadLimitASGIMiddleware:
"""Bound video-upload ingress *before* FastAPI's multipart parser runs.
The upload route's own MAX_UPLOAD_SIZE check only fires after the multipart body has
been fully parsed (and spooled to temp storage), so oversized, chunked, or many
concurrent uploads could exhaust temp space before ever being rejected. This
middleware rejects oversized requests from the Content-Length header, aborts
chunked bodies that exceed the cap mid-stream, and bounds concurrent uploads
both globally and per user (so one tenant's slow uploads cannot starve the
others into 429s).
"""
def __init__(
self,
app: ASGIApp,
max_body_bytes: int,
max_concurrent: int,
max_concurrent_per_user: int | None = None,
identify_user: Callable[[Scope], tuple[bool, str | None] | Awaitable[tuple[bool, str | None]]] | None = None,
idle_timeout_seconds: float = 120.0,
max_upload_duration_seconds: float = 30 * 60,
) -> None:
self.app = app
self.max_body_bytes = max_body_bytes
self.max_concurrent = max_concurrent
self.max_concurrent_per_user = max_concurrent_per_user
self.identify_user = identify_user
self.idle_timeout_seconds = idle_timeout_seconds
self.max_upload_duration_seconds = max_upload_duration_seconds
self._active = 0
self._active_by_user: dict[str, int] = {}
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
return await self.app(scope, receive, send)
# Behind a sub-path proxy the public path carries the prefix (see SubPathASGIMiddleware).
route_path: str = scope.get("path", "").removeprefix(scope.get("root_path", ""))
if not (scope.get("method") == "POST" and route_path == "/api/v1/videos/upload"):
return await self.app(scope, receive, send)
per_user_key: str | None = None
if self.identify_user is not None:
identity = self.identify_user(scope)
if inspect.isawaitable(identity):
identity = await identity
authenticated, per_user_key = identity
if not authenticated:
response = JSONResponse(
{"detail": "Authentication required"},
status_code=401,
headers={"WWW-Authenticate": "Bearer"},
)
return await response(scope, receive, send)
content_length = Headers(scope=scope).get("content-length")
if content_length is not None and content_length.isdigit() and int(content_length) > self.max_body_bytes:
response = JSONResponse(
{"detail": f"Video upload exceeds maximum request size ({self.max_body_bytes} bytes)"},
status_code=413,
)
return await response(scope, receive, send)
if self._active >= self.max_concurrent:
response = JSONResponse(
{"detail": "Too many concurrent video uploads; try again shortly"},
status_code=429,
headers={"Retry-After": "5"},
)
return await response(scope, receive, send)
if (
per_user_key is not None
and self.max_concurrent_per_user is not None
and self._active_by_user.get(per_user_key, 0) >= self.max_concurrent_per_user
):
response = JSONResponse(
{"detail": "Too many concurrent video uploads for this user; try again shortly"},
status_code=429,
headers={"Retry-After": "5"},
)
return await response(scope, receive, send)
self._active += 1
if per_user_key is not None:
self._active_by_user[per_user_key] = self._active_by_user.get(per_user_key, 0) + 1
received = 0
upload_started_at = asyncio.get_running_loop().time()
async def limited_receive() -> Message:
# Backstop for clients that omit Content-Length (chunked transfer): count the
# streamed body and abort the request once it exceeds the cap, so the multipart
# parser stops spooling. A clean 413 isn't possible mid-parse; the aborted
# request surfaces to the client as a dropped connection.
nonlocal received
remaining_duration = self.max_upload_duration_seconds - (
asyncio.get_running_loop().time() - upload_started_at
)
if remaining_duration <= 0:
return {"type": "http.disconnect"}
try:
message = await asyncio.wait_for(receive(), timeout=min(self.idle_timeout_seconds, remaining_duration))
except TimeoutError:
return {"type": "http.disconnect"}
if message["type"] == "http.request":
received += len(message.get("body", b""))
if received > self.max_body_bytes:
return {"type": "http.disconnect"}
return message
try:
await self.app(scope, limited_receive, send)
finally:
self._active -= 1
if per_user_key is not None:
remaining = self._active_by_user.get(per_user_key, 0) - 1
if remaining > 0:
self._active_by_user[per_user_key] = remaining
else:
self._active_by_user.pop(per_user_key, None)
class SubPathASGIMiddleware:
"""Make the app work behind a reverse proxy that serves it under a sub-path (e.g. `/invoke`).
@@ -191,6 +371,13 @@ class SubPathASGIMiddleware:
# Add the middleware
app.add_middleware(RedirectRootWithQueryStringMiddleware)
app.add_middleware(SlidingWindowTokenMiddleware)
app.add_middleware(
VideoUploadLimitASGIMiddleware,
max_body_bytes=videos.MAX_UPLOAD_REQUEST_SIZE,
max_concurrent=videos.MAX_CONCURRENT_VIDEO_UPLOADS,
max_concurrent_per_user=videos.MAX_CONCURRENT_VIDEO_UPLOADS_PER_USER,
identify_user=_identify_video_upload_user_async,
)
# Add event handler
@@ -223,6 +410,8 @@ app.include_router(model_manager.model_manager_router, prefix="/api")
app.include_router(download_queue.download_queue_router, prefix="/api")
app.include_router(image_moves.image_moves_router, prefix="/api")
app.include_router(images.images_router, prefix="/api")
app.include_router(videos.videos_router, prefix="/api")
app.include_router(gallery.gallery_router, prefix="/api")
app.include_router(boards.boards_router, prefix="/api")
app.include_router(board_images.board_images_router, prefix="/api")
app.include_router(virtual_boards.virtual_boards_router, prefix="/api")
+43
View File
@@ -141,6 +141,7 @@ class UIComponent(str, Enum, metaclass=MetaEnum):
None_ = "none"
Textarea = "textarea"
Slider = "slider"
VideoFrameIndex = "video-frame-index"
class FieldDescriptions:
@@ -174,6 +175,9 @@ class FieldDescriptions:
z_image_model = "Z-Image model (Transformer) to load"
qwen_image_model = "Qwen Image Edit model (Transformer) to load"
qwen_vl_encoder = "Qwen2.5-VL tokenizer, processor and text/vision encoder"
wan_model = "Wan 2.2 model (Transformer) to load"
wan_t5_encoder = "UMT5-XXL tokenizer and text encoder for Wan 2.2"
wan_ref_image = "Reference-image (VAE-latent) conditioning for Wan 2.2 I2V."
sdxl_main_model = "SDXL Main model (UNet, VAE, CLIP1, CLIP2) to load"
sdxl_refiner_model = "SDXL Refiner Main Modde (UNet, VAE, CLIP2) to load"
onnx_main_model = "ONNX Main model (UNet, VAE, CLIP) to load"
@@ -241,6 +245,12 @@ class ImageField(BaseModel):
image_name: str = Field(description="The name of the image")
class VideoField(BaseModel):
"""A video primitive field"""
video_name: str = Field(description="The name of the video")
class BoardField(BaseModel):
"""A board primitive field"""
@@ -371,6 +381,39 @@ class AnimaConditioningField(BaseModel):
)
class WanConditioningField(BaseModel):
"""A Wan 2.2 conditioning tensor primitive value.
Wan conditioning is the UMT5-XXL hidden state for the prompt plus an attention
mask marking valid (non-padding) tokens.
"""
conditioning_name: str = Field(description="The name of conditioning tensor")
class WanRefImageConditioningField(BaseModel):
"""Reference-image conditioning for Wan 2.2 I2V.
Carries the 20-channel VAE-latent condition tensor (4-channel first-frame
mask + 16-channel ref-image latents). The denoise loop concatenates this
to the 16-channel noise latents along the channel dim each step, producing
the 36-channel input the I2V-A14B transformer expects.
Also carries the spatial dims and frame count used to encode the image so
the denoise node can sanity-check the user's width/height/num_frames — a
latent temporal-dim mismatch is hard to debug from the downstream error.
"""
condition_tensor_name: str = Field(description="Name of the saved [1, 20, T_lat, H/8, W/8] condition tensor.")
width: int = Field(description="Image width used during VAE encoding (matches denoise width).")
height: int = Field(description="Image height used during VAE encoding (matches denoise height).")
num_frames: int = Field(
default=1,
description="Pixel-frame count the condition was built for. 1 for single-frame I2V "
"(image output), 81+ for video.",
)
class ConditioningField(BaseModel):
"""A conditioning tensor primitive value"""
+5
View File
@@ -175,6 +175,11 @@ GENERATION_MODES = Literal[
"anima_img2img",
"anima_inpaint",
"anima_outpaint",
"wan_txt2img",
"wan_img2img",
"wan_inpaint",
"wan_outpaint",
"wan_i2v",
]
+49
View File
@@ -87,6 +87,14 @@ class Qwen3EncoderField(BaseModel):
loras: List[LoRAField] = Field(default_factory=list, description="LoRAs to apply on model loading")
class WanT5EncoderField(BaseModel):
"""Field for the UMT5-XXL text encoder used by Wan 2.2 models."""
tokenizer: ModelIdentifierField = Field(description="Info to load tokenizer submodel")
text_encoder: ModelIdentifierField = Field(description="Info to load text_encoder submodel")
loras: List[LoRAField] = Field(default_factory=list, description="LoRAs to apply on model loading")
class VAEField(BaseModel):
vae: ModelIdentifierField = Field(description="Info to load vae submodel")
seamless_axes: List[str] = Field(default_factory=list, description='Axes("x" and "y") to which apply seamless')
@@ -101,6 +109,47 @@ class TransformerField(BaseModel):
loras: List[LoRAField] = Field(description="LoRAs to apply on model loading")
class WanTransformerField(BaseModel):
"""Transformer field for Wan 2.2 models.
Wan 2.2 A14B is a Mixture-of-Experts model with two transformer experts:
a high-noise expert (active at large timesteps) and a low-noise expert
(active at small timesteps). TI2V-5B is a single-transformer model and only
populates ``transformer``.
``boundary_ratio`` matches Diffusers' ``WanPipeline`` semantics: it's the
boundary timestep as a fraction of ``num_train_timesteps`` (typically 1000),
so ``boundary_ratio=0.875`` means the high-noise expert handles t >= 875 and
the low-noise expert handles t < 875.
"""
transformer: ModelIdentifierField = Field(
description="Primary transformer submodel. For A14B this is the high-noise expert."
)
transformer_low_noise: ModelIdentifierField | None = Field(
default=None,
description="Low-noise transformer expert (Wan 2.2 A14B only). None for TI2V-5B.",
)
loras: List[LoRAField] = Field(
default_factory=list,
description="LoRAs to apply to the primary transformer. For A14B applied to the high-noise expert.",
)
loras_low_noise: List[LoRAField] = Field(
default_factory=list,
description="LoRAs to apply to the low-noise expert (Wan 2.2 A14B). The Wan LoRA loader "
"routes 'both'- and 'low'-targeted LoRAs here; if empty, no LoRAs are applied to the "
"low-noise expert (a 'high'-targeted LoRA must not leak onto it).",
)
boundary_ratio: float = Field(
default=0.875,
ge=0.0,
le=1.0,
description="Boundary timestep as a fraction of num_train_timesteps (Wan 2.2 A14B only). "
"High-noise expert: t >= boundary_ratio * num_train_timesteps. Low-noise expert: t below. "
"Ignored for TI2V-5B.",
)
@invocation_output("unet_output")
class UNetOutput(BaseInvocationOutput):
"""Base class for invocations that output a UNet field."""
+93
View File
@@ -30,10 +30,14 @@ from invokeai.app.invocations.fields import (
SD3ConditioningField,
TensorField,
UIComponent,
VideoField,
WanConditioningField,
WanRefImageConditioningField,
ZImageConditioningField,
)
from invokeai.app.services.images.images_common import ImageDTO
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.app.services.videos.videos_common import VideoDTO
"""
Primitives: Boolean, Integer, Float, String, Image, Latents, Conditioning, Color
@@ -522,6 +526,44 @@ class AnimaConditioningOutput(BaseInvocationOutput):
return cls(conditioning=AnimaConditioningField(conditioning_name=conditioning_name))
@invocation_output("wan_conditioning_output")
class WanConditioningOutput(BaseInvocationOutput):
"""Base class for nodes that output a Wan 2.2 text conditioning tensor."""
conditioning: WanConditioningField = OutputField(description=FieldDescriptions.cond)
@classmethod
def build(cls, conditioning_name: str) -> "WanConditioningOutput":
return cls(conditioning=WanConditioningField(conditioning_name=conditioning_name))
@invocation_output("wan_ref_image_output")
class WanRefImageOutput(BaseInvocationOutput):
"""Output of a Wan 2.2 reference-image VAE-encoder."""
ref_image: WanRefImageConditioningField = OutputField(
description="VAE-latent reference-image conditioning for Wan 2.2 I2V.",
title="Reference Image",
)
@classmethod
def build(
cls,
condition_tensor_name: str,
width: int,
height: int,
num_frames: int = 1,
) -> "WanRefImageOutput":
return cls(
ref_image=WanRefImageConditioningField(
condition_tensor_name=condition_tensor_name,
width=width,
height=height,
num_frames=num_frames,
)
)
@invocation_output("conditioning_output")
class ConditioningOutput(BaseInvocationOutput):
"""Base class for nodes that output a single conditioning tensor"""
@@ -533,6 +575,57 @@ class ConditioningOutput(BaseInvocationOutput):
return cls(conditioning=ConditioningField(conditioning_name=conditioning_name))
@invocation_output("video_output")
class VideoOutput(BaseInvocationOutput):
"""Output of a node that produces a video file (e.g. Wan 2.2 latents-to-video)."""
video: VideoField = OutputField(description="The output video")
width: int = OutputField(description="The width of the video in pixels")
height: int = OutputField(description="The height of the video in pixels")
num_frames: int = OutputField(description="The number of frames in the video")
fps: float = OutputField(description="The frames-per-second of the video")
duration: float = OutputField(description="The duration of the video in seconds")
@classmethod
def build(cls, video_dto: VideoDTO, num_frames: Optional[int] = None) -> "VideoOutput":
# Frame count isn't stored on the DTO; derive it from duration * fps when fps is known.
fps = video_dto.fps or 0.0
if num_frames is None:
num_frames = int(round(video_dto.duration * fps)) if fps > 0 else 0
return cls(
video=VideoField(video_name=video_dto.video_name),
width=video_dto.width,
height=video_dto.height,
num_frames=num_frames,
fps=fps,
duration=video_dto.duration,
)
@invocation(
"video",
title="Video Primitive",
tags=["primitives", "video"],
category="primitives",
version="1.0.0",
)
class VideoInvocation(BaseInvocation):
"""A video primitive value. Drop a video onto the field to make it available as an input
to downstream nodes (e.g. Frame from Video, Concatenate Videos)."""
video: VideoField = InputField(description="The video to load")
# Return annotation is a real class (not a forward-ref string) because a previous
# `from __future__ import annotations` left wan_l2v with a stringified annotation
# and crashed the output-class registry on startup (commit cac366229a).
def invoke(self, context: InvocationContext) -> VideoOutput:
from invokeai.app.util.video_thumbnails import decoder_frame_count
video_dto = context.videos.get_dto(self.video.video_name)
num_frames = decoder_frame_count(context.videos.get_path(self.video.video_name))
return VideoOutput.build(video_dto=video_dto, num_frames=num_frames)
@invocation_output("conditioning_collection_output")
class ConditioningCollectionOutput(BaseInvocationOutput):
"""Base class for nodes that output a collection of conditioning tensors"""
+286
View File
@@ -0,0 +1,286 @@
"""Concatenate two or more videos with an optional transition.
Pairs naturally with the I2V chaining workflow: feed several Wan-generated
clips into this node to glue them into one longer video. The transition
options hide the seam between independently-denoised clips.
Implementation uses imageio (FFMPEG plugin) for both decode and encode, matching
``wan_latents_to_video`` and ``video_thumbnails`` — so we can read our own
output without surprises. Frames stream from the decoders into the encoder one
at a time, buffering only the transition windows, so peak memory stays
O(transition_frames) even when the inputs are long uploads (the upload cap
admits files whose decoded frames would run to tens of gigabytes).
"""
import math
import tempfile
from collections import deque
from pathlib import Path
from typing import Callable, Iterable, Iterator, Literal, Optional
import numpy as np
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
from invokeai.app.invocations.fields import (
InputField,
VideoField,
WithBoard,
WithMetadata,
)
from invokeai.app.invocations.primitives import VideoOutput
from invokeai.app.services.session_processor.session_processor_common import CanceledException
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.app.util.video_encoding import make_mp4_writer
from invokeai.app.util.video_thumbnails import iter_video_frames, probe_video
TransitionMode = Literal["cut", "crossfade", "fade_through_black"]
MAX_TRANSITION_MEMORY_BYTES = 512 * 1024 * 1024
_BLEND_WORKING_FRAMES = 13
def _crossfade(a_tail: list[np.ndarray], b_head: list[np.ndarray]) -> Iterator[np.ndarray]:
"""Yields a linear A to B cross-dissolve without retaining the blended frames."""
n = len(a_tail)
for i in range(n):
alpha = (i + 1) / (n + 1)
blended = a_tail[i].astype(np.float32) * (1.0 - alpha) + b_head[i].astype(np.float32) * alpha
yield np.clip(blended, 0, 255).astype(np.uint8)
def _fade_through_black(a_tail: list[np.ndarray], b_head: list[np.ndarray]) -> Iterator[np.ndarray]:
"""A fades to black, then black fades to B. Consumes N/2 frames from each side and returns N output frames.
Asymmetric framing: the first ``len(a_tail)`` output frames are the trailing A frames scaled
toward zero brightness; the next ``len(b_head)`` are the leading B frames scaled up from zero.
"""
n_a = len(a_tail)
for i, fa in enumerate(a_tail):
# 1.0 at i=0 (fully visible) → near 0 at i=n_a-1 (essentially black).
alpha = 1.0 - (i + 1) / (n_a + 1)
yield np.clip(fa.astype(np.float32) * alpha, 0, 255).astype(np.uint8)
n_b = len(b_head)
for j, fb in enumerate(b_head):
alpha = (j + 1) / (n_b + 1)
yield np.clip(fb.astype(np.float32) * alpha, 0, 255).astype(np.uint8)
@invocation(
"video_concat",
title="Concatenate Videos",
tags=["video", "concat", "transition"],
category="video",
version="1.0.0",
classification=Classification.Prototype,
)
class VideoConcatInvocation(BaseInvocation, WithMetadata, WithBoard):
"""Join two or more videos into a single MP4.
Transitions:
* ``cut`` — hard splice, no blending. Fastest; total length is the sum of inputs.
* ``crossfade`` — linear A→B cross-dissolve over ``transition_frames``. Each boundary
consumes ``transition_frames`` from both adjacent clips, so total length is
``sum(inputs) - transition_frames * (n - 1)``.
* ``fade_through_black`` — A fades to black, then B fades in from black. Each boundary
consumes ``transition_frames // 2`` frames from the preceding clip's tail and the
remainder (``transition_frames - transition_frames // 2``) from the next clip's head,
so the total emitted is exactly ``transition_frames`` per boundary — even for odd
``transition_frames`` — and the overall length equals the sum of inputs.
All inputs must share the same pixel dimensions. Output frame rate defaults to the
first input's fps; override with ``fps`` to force a specific rate (the frames are not
resampled, only the container is encoded at the new rate).
"""
videos: list[VideoField] = InputField(
min_length=2,
description="Videos to concatenate, in order. At least two are required.",
)
transition: TransitionMode = InputField(
default="cut",
description="Transition between consecutive clips.",
)
transition_frames: int = InputField(
default=8,
ge=0,
le=240,
description="Length of each transition in frames. Ignored when transition is 'cut'.",
)
fps: Optional[int] = InputField(
default=None,
ge=1,
le=120,
description="Output frame rate. Defaults to the first input's fps.",
)
def invoke(self, context: InvocationContext) -> VideoOutput:
if len(self.videos) < 2:
raise ValueError("video_concat requires at least two input videos.")
paths: list[Path] = [context.videos.get_path(v.video_name) for v in self.videos]
# Probe inputs up front: enforce matching dims and pick the default output fps.
probes = [probe_video(p) for p in paths]
widths = {(w, h) for (w, h, _, _) in probes}
if len(widths) > 1:
raise ValueError(
f"All inputs must share the same dimensions. Got: "
f"{sorted(widths)}. Re-render at a single resolution before concatenating."
)
width, height, _, _first_fps = probes[0]
# libx264 + yuv420p needs even dimensions; we encode with macro_block_size=1 to
# preserve the source dimensions exactly, so reject odd sources with a clear error.
if width % 2 or height % 2:
raise ValueError(
f"Input videos are {width}x{height}; H.264 encoding requires even dimensions. "
"Re-encode or crop the sources to even width and height first."
)
self._validate_transition_memory(width, height)
output_fps = self._resolve_output_fps([probe[3] for probe in probes])
context.util.signal_progress(f"Joining {len(self.videos)} clip(s) ({self.transition}) @ {output_fps:.2f} fps")
tmp = tempfile.NamedTemporaryFile(prefix="invokeai_video_concat_", suffix=".mp4", delete=False)
tmp.close()
tmp_path = Path(tmp.name)
try:
# Frames stream from the decoders straight into the encoder; only the
# transition windows are buffered. See _iter_joined_frames.
writer = make_mp4_writer(tmp_path, output_fps)
num_frames = 0
try:
clip_iters = [iter_video_frames(p, is_canceled=context.util.is_canceled) for p in paths]
for frame in self._iter_joined_frames(clip_iters, is_canceled=context.util.is_canceled):
writer.append_data(frame)
num_frames += 1
finally:
writer.close()
if num_frames == 0:
raise ValueError("Concatenation produced zero output frames.")
duration = num_frames / output_fps
context.logger.info(
f"Encoded concatenated MP4: {num_frames} frames @ {output_fps:.2f} fps "
f"({duration:.2f}s) at {width}x{height}"
)
video_dto = context.videos.save(
source_path=tmp_path,
width=width,
height=height,
duration=duration,
fps=output_fps,
)
context.logger.info(f"Saved concatenated video: {video_dto.video_name}")
return VideoOutput.build(video_dto)
finally:
try:
tmp_path.unlink(missing_ok=True)
except Exception:
pass
def _estimate_transition_memory(self, width: int, height: int) -> int:
if self.transition == "cut" or self.transition_frames == 0:
return 0
buffered_frames = self.transition_frames * (2 if self.transition == "crossfade" else 1)
frame_bytes = width * height * 3
return frame_bytes * (buffered_frames + _BLEND_WORKING_FRAMES)
def _resolve_output_fps(self, source_rates: list[Optional[float]]) -> float:
if self.fps is not None:
return float(self.fps)
known_rates = [rate for rate in source_rates if rate is not None and rate > 0]
if not known_rates:
return 16.0
if any(not math.isclose(rate, known_rates[0], rel_tol=1e-3) for rate in known_rates[1:]):
raise ValueError("Input videos have different frame rates; set Output FPS to retime them.")
# An unknown rate mixed with agreeing known ones is not an error: probe_video
# deliberately reports None for metadata-poor containers (VFR flags, missing
# avg_frame_rate) whose real rate is usually the same as their neighbours'.
# Erroring here would break previously-working concat workflows over a metadata
# quirk; disagreement between *known* rates is the case that silently retimes.
return known_rates[0]
def _validate_transition_memory(self, width: int, height: int) -> None:
estimated_bytes = self._estimate_transition_memory(width, height)
if estimated_bytes > MAX_TRANSITION_MEMORY_BYTES:
estimated_mib = estimated_bytes / (1024 * 1024)
limit_mib = MAX_TRANSITION_MEMORY_BYTES / (1024 * 1024)
raise ValueError(
f"The requested transition needs an estimated {estimated_mib:.0f} MiB, "
f"which exceeds the {limit_mib:.0f} MiB transition memory budget. "
"Lower transition_frames or use lower-resolution clips."
)
def _iter_joined_frames(
self,
clips: list[Iterable[np.ndarray]],
is_canceled: Optional[Callable[[], bool]] = None,
) -> Iterator[np.ndarray]:
"""Yields the joined output frames, pulling lazily from each clip's frame iterator.
A frame is emitted as soon as it can no longer participate in a transition, so at
most one transition window (the previous clip's tail plus the current clip's head,
each bounded by ``transition_frames``) is buffered at a time — never a whole clip.
Transition layout matches the class docstring:
* ``crossfade`` consumes ``tf`` frames from both sides of each boundary and emits
``tf`` blended frames in their place.
* ``fade_through_black`` splits ``tf`` asymmetrically (``tf // 2`` from the previous
clip's tail, the remainder from the next clip's head) so an odd ``tf`` still emits
exactly ``tf`` frames per boundary.
Raises ValueError if a clip decodes to zero frames or is too short to supply its
transition windows. Because frames stream straight into the encoder, that error can
surface mid-encode; the caller discards the partial output file.
"""
if self.transition == "crossfade" and self.transition_frames > 0:
tail_need = head_need = self.transition_frames
blend = _crossfade
elif self.transition == "fade_through_black" and self.transition_frames > 0:
tail_need = self.transition_frames // 2
head_need = self.transition_frames - tail_need
blend = _fade_through_black
else:
# "cut", or a zero-length transition: plain concatenation.
tail_need = head_need = 0
blend = None
a_tail: list[np.ndarray] = []
for i, clip in enumerate(clips):
head_want = 0 if i == 0 else head_need
tail_keep = 0 if i == len(clips) - 1 else tail_need
b_head: list[np.ndarray] = []
head_complete = head_want == 0
tail_buf: deque[np.ndarray] = deque()
n_frames = 0
for frame in clip:
if is_canceled is not None and is_canceled():
raise CanceledException
frame = np.ascontiguousarray(frame)
n_frames += 1
# The clip's first head_want frames are consumed into the boundary blend
# with the previous clip's tail rather than emitted directly.
if not head_complete:
b_head.append(frame)
if len(b_head) == head_want and blend is not None:
yield from blend(a_tail, b_head)
a_tail = []
b_head = []
head_complete = True
continue
# Hold back the last tail_keep frames seen so far; anything older is
# guaranteed not to be part of the next boundary and can be emitted.
tail_buf.append(frame)
if len(tail_buf) > tail_keep:
yield tail_buf.popleft()
if n_frames == 0:
raise ValueError(f"Input video {i} ({self.videos[i].video_name}) decoded to zero frames.")
if n_frames < head_want + tail_keep:
raise ValueError(
f"Clip {i} has {n_frames} frames but the requested transitions need "
f"{head_want} from its head + {tail_keep} from its tail. Lower "
f"transition_frames or use longer clips."
)
a_tail = list(tail_buf)
@@ -0,0 +1,75 @@
"""Extract a single frame from a video as an image.
Enables I2V "shot extension": take the last frame of one clip and feed it back
in as the reference image for the next clip, then concatenate the MP4s
externally to get a video longer than the model's single-shot frame budget.
Also useful as a general-purpose video-to-image step.
"""
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
from invokeai.app.invocations.fields import (
InputField,
UIComponent,
VideoField,
WithBoard,
WithMetadata,
)
from invokeai.app.invocations.primitives import ImageOutput
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.app.util.video_thumbnails import decoder_frame_count, extract_video_frame, probe_video
@invocation(
"video_frame_extract",
title="Frame from Video",
tags=["video", "image", "frame"],
category="image",
version="1.1.0",
classification=Classification.Prototype,
)
class VideoFrameExtractInvocation(BaseInvocation, WithMetadata, WithBoard):
"""Extract a single frame from a video and save it as an image.
``frame_index`` is 0-based. Negative indices count from the end, so the
default of -1 returns the final frame — the typical setup for chaining
I2V clips into a longer sequence.
"""
video: VideoField = InputField(description="The video to extract a frame from.")
frame_index: int = InputField(
default=-1,
description="Index of the frame to extract. 0 = first frame, -1 = last frame, -2 = second-to-last, etc.",
ui_component=UIComponent.VideoFrameIndex,
)
def invoke(self, context: InvocationContext) -> ImageOutput:
video_path = context.videos.get_path(self.video.video_name)
# Resolve negative indices against the actual frame count rather than
# trusting imageio plugins to accept index=-1 uniformly. Use the decoder's
# frame count when available — duration*fps can be off-by-one for VFR
# uploads or containers with approximate metadata, causing frame_index=-1
# to point past the final frame.
index = self.frame_index
if index < 0:
n_frames = decoder_frame_count(video_path)
if n_frames is None:
_, _, duration, fps = probe_video(video_path)
if not fps or duration <= 0:
raise ValueError(
f"Cannot resolve negative frame index for video {self.video.video_name}: "
f"probe returned duration={duration}, fps={fps}."
)
n_frames = int(round(duration * fps))
if n_frames <= 0:
raise ValueError(f"Video {self.video.video_name} has no decodable frames (probed {n_frames}).")
index = n_frames + index
if index < 0:
raise ValueError(f"frame_index {self.frame_index} is out of range for a {n_frames}-frame video.")
frame = extract_video_frame(video_path, frame_index=index)
if frame is None:
raise ValueError(f"Failed to extract frame {index} from {self.video.video_name}.")
image_dto = context.images.save(image=frame)
return ImageOutput.build(image_dto=image_dto)
@@ -0,0 +1,244 @@
"""Extract a contiguous range of frames from a video and re-encode as MP4.
Companion to ``video_frame_extract`` (single frame → image) and
``video_concat`` (many videos → one). This node takes a slice of an input
video and emits a new MP4, so the output can be fed straight into
Concatenate Videos to splice clips together — e.g. trim a generated clip
to a usable middle section before chaining it to another shot.
"""
import tempfile
from itertools import islice
from pathlib import Path
from typing import Callable, Iterator, Optional, Protocol
import numpy as np
from invokeai.app.invocations.baseinvocation import (
BaseInvocation,
BaseInvocationOutput,
Classification,
invocation,
invocation_output,
)
from invokeai.app.invocations.fields import (
InputField,
OutputField,
UIComponent,
VideoField,
WithBoard,
WithMetadata,
)
from invokeai.app.invocations.primitives import VideoOutput
from invokeai.app.services.session_processor.session_processor_common import CanceledException
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.app.util.video_encoding import make_mp4_writer
from invokeai.app.util.video_thumbnails import decoder_frame_count, iter_video_frames, probe_video
class _FrameWriter(Protocol):
def append_data(self, frame: np.ndarray) -> None: ...
def _validate_even_dimensions(width: int, height: int, video_name: str) -> None:
"""Raises if the source dimensions can't be encoded as-is with libx264 + yuv420p.
We encode with ``macro_block_size=1`` to preserve the source dimensions exactly
(imageio's default of 16 silently rescales), which requires even width and height.
"""
if width % 2 or height % 2:
raise ValueError(
f"Video {video_name} is {width}x{height}; H.264 encoding requires even dimensions. "
"Re-encode or crop the source to even width and height first."
)
def _write_frame_range(
frames: Iterator[np.ndarray],
writer: _FrameWriter,
start: int,
end: int,
is_canceled: Optional[Callable[[], bool]] = None,
) -> int:
"""Streams frames[start..end] (inclusive) from a lazy decoder into the writer.
Frames are appended one at a time as they stream past — the full range is never
materialized in RAM. The upload cap admits files whose decoded frames would run to
tens of gigabytes, so peak memory here must stay one-frame-sized regardless of the
requested range. Decoding stops as soon as ``end`` has been written. Returns the
number of frames written.
"""
written = 0
for idx, frame in enumerate(islice(frames, end + 1)):
if is_canceled is not None and is_canceled():
raise CanceledException
if idx < start:
continue
writer.append_data(np.ascontiguousarray(frame))
written += 1
return written
@invocation_output("extract_video_range_output")
class ExtractVideoRangeOutput(BaseInvocationOutput):
"""Output of ``extract_video_range``: a trimmed video plus the resolved frame indices.
Mirrors ``VideoOutput`` so the video can be piped directly into Concatenate Videos or
any other ``VideoField``-consuming node, and additionally exposes the resolved
(positive, clamped) start and end indices so chained workflows can feed them back in
— e.g. drive a downstream Frame from Video to pull the same boundary frame.
"""
video: VideoField = OutputField(description="The trimmed video")
width: int = OutputField(description="The width of the video in pixels")
height: int = OutputField(description="The height of the video in pixels")
num_frames: int = OutputField(description="The number of frames in the trimmed video")
fps: float = OutputField(description="The frames-per-second of the trimmed video")
duration: float = OutputField(description="The duration of the trimmed video in seconds")
start_frame: int = OutputField(description="The resolved (positive, 0-based) start frame index in the source video")
end_frame: int = OutputField(description="The resolved (positive, 0-based) end frame index in the source video")
@invocation(
"extract_video_range",
title="Frame Range from Video",
tags=["video", "trim", "range", "frames"],
category="video",
version="1.1.0",
classification=Classification.Prototype,
)
class ExtractVideoRangeInvocation(BaseInvocation, WithMetadata, WithBoard):
"""Trim a video to a contiguous frame range and re-encode as MP4.
Both bounds are inclusive and 0-based — ``start_frame=10, end_frame=50``
emits 41 frames. Negative indices count from the end (``end_frame=-1``
is the final frame), matching ``video_frame_extract``. The output frame
rate defaults to the source video's frame rate; set ``fps=0`` to inherit
it (or 16 fps if the source rate can't be probed).
The resolved (positive) ``start_frame`` and ``end_frame`` are also emitted as
outputs, so chained workflows can re-use the boundary indices — e.g. feeding
them into a downstream Frame from Video to extract the same boundary frame.
"""
video: VideoField = InputField(description="The video to extract a frame range from.")
start_frame: int = InputField(
default=0,
description=("First frame to keep, inclusive. 0 = first frame. Negative indices count from the end."),
ui_component=UIComponent.VideoFrameIndex,
)
end_frame: int = InputField(
default=-1,
description=("Last frame to keep, inclusive. -1 = last frame. Negative indices count from the end."),
ui_component=UIComponent.VideoFrameIndex,
)
fps: int = InputField(
default=0,
ge=0,
le=120,
description="Output frame rate. 0 = match the source video's frame rate "
"(falls back to 16 fps if the source rate can't be probed).",
)
def invoke(self, context: InvocationContext) -> ExtractVideoRangeOutput:
video_path = context.videos.get_path(self.video.video_name)
width, height, duration, source_fps = probe_video(video_path)
n_frames = decoder_frame_count(video_path)
if n_frames is None:
if not source_fps or duration <= 0:
raise ValueError(
f"Cannot determine frame count for {self.video.video_name}: "
f"probe returned duration={duration}, fps={source_fps}."
)
n_frames = int(round(duration * source_fps))
if n_frames <= 0:
raise ValueError(f"Video {self.video.video_name} has no decodable frames (probed {n_frames}).")
start = self._resolve_index(self.start_frame, n_frames, "start_frame")
end = self._resolve_index(self.end_frame, n_frames, "end_frame")
if end < start:
raise ValueError(
f"end_frame ({self.end_frame} → {end}) must be >= start_frame "
f"({self.start_frame} → {start}) after resolving negative indices."
)
# Derive the output frame rate from the source video when ``fps`` is 0
# (the default), so a trimmed clip plays back at the same speed as its
# source. Fall back to 16 fps (the Wan video default) when the source
# rate couldn't be probed.
if self.fps > 0:
output_fps = float(self.fps)
elif source_fps and source_fps > 0:
output_fps = float(source_fps)
else:
output_fps = 16.0
# libx264 + yuv420p needs even dimensions. Reject odd sources up front with a
# clear message instead of surfacing an opaque ffmpeg error mid-encode.
_validate_even_dimensions(width, height, self.video.video_name)
context.util.signal_progress(f"Transcoding frames {start}-{end} of {n_frames} @ {output_fps:.2f} fps")
tmp = tempfile.NamedTemporaryFile(prefix="invokeai_video_range_", suffix=".mp4", delete=False)
tmp.close()
tmp_path = Path(tmp.name)
try:
# imageio's iter_index isn't exposed by iio.imiter, so we enumerate and skip.
# Frames stream straight from the decoder into the encoder; see _write_frame_range.
writer = make_mp4_writer(tmp_path, output_fps)
try:
num_frames = _write_frame_range(
iter_video_frames(video_path, is_canceled=context.util.is_canceled),
writer,
start,
end,
is_canceled=context.util.is_canceled,
)
finally:
writer.close()
expected_frames = end - start + 1
if num_frames != expected_frames:
raise ValueError(
f"Decoded only {num_frames} of {expected_frames} requested frames for range {start}-{end} "
f"of {self.video.video_name} "
f"(probed {n_frames} frames). The container's metadata may be inaccurate."
)
out_duration = num_frames / output_fps
context.logger.info(
f"Encoded trimmed MP4: {num_frames} frames @ {output_fps:.2f} fps "
f"({out_duration:.2f}s) at {width}x{height}"
)
video_dto = context.videos.save(
source_path=tmp_path,
width=width,
height=height,
duration=out_duration,
fps=output_fps,
)
context.logger.info(f"Saved trimmed video: {video_dto.video_name}")
base = VideoOutput.build(video_dto)
return ExtractVideoRangeOutput(
video=base.video,
width=base.width,
height=base.height,
num_frames=base.num_frames,
fps=base.fps,
duration=base.duration,
start_frame=start,
end_frame=end,
)
finally:
try:
tmp_path.unlink(missing_ok=True)
except Exception:
pass
@staticmethod
def _resolve_index(value: int, n_frames: int, field_name: str) -> int:
resolved = value + n_frames if value < 0 else value
if resolved < 0 or resolved >= n_frames:
raise ValueError(f"{field_name}={value} is out of range for a {n_frames}-frame video.")
return resolved
+777
View File
@@ -0,0 +1,777 @@
"""Wan 2.2 denoise invocation.
Supports both single-transformer (TI2V-5B) and dual-expert MoE (A14B) denoising.
For A14B the high-noise expert handles timesteps ``t >= boundary_timestep`` and
the low-noise expert handles ``t < boundary_timestep``, where
``boundary_timestep = boundary_ratio * num_train_timesteps`` (typically 1000).
To keep VRAM usage manageable both experts are pinned in the model cache
(system RAM) but only one is GPU-resident at a time. The boundary is normally
crossed once per denoise, so the swap incurs a single CPU→GPU transfer.
Phase 8 will add inpaint via :class:`RectifiedFlowInpaintExtension`.
The transformer call signature mirrors Diffusers' ``WanPipeline``:
transformer(
hidden_states=latents_5d, # [B, C, 1, H/s, W/s]
timestep=t.expand(B), # scheduler-time
encoder_hidden_states=prompt_embeds, # [B, seq_len, 4096]
attention_kwargs=None,
return_dict=False,
)[0]
"""
from contextlib import ExitStack
from pathlib import Path
from typing import Any, Callable, Iterable, Iterator, Optional, Tuple
import torch
import torchvision.transforms as tv_transforms
from torchvision.transforms.functional import resize as tv_resize
from tqdm import tqdm
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
from invokeai.app.invocations.fields import (
DenoiseMaskField,
FieldDescriptions,
Input,
InputField,
LatentsField,
WanConditioningField,
WanRefImageConditioningField,
)
from invokeai.app.invocations.model import LoRAField, WanTransformerField
from invokeai.app.invocations.primitives import LatentsOutput
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.backend.model_manager.taxonomy import BaseModelType, ModelFormat, WanVariantType
from invokeai.backend.patches.layer_patcher import LayerPatcher
from invokeai.backend.patches.lora_conversions.wan_lora_constants import WAN_LORA_TRANSFORMER_PREFIX
from invokeai.backend.patches.model_patch_raw import ModelPatchRaw
from invokeai.backend.rectified_flow.rectified_flow_inpaint_extension import RectifiedFlowInpaintExtension
from invokeai.backend.stable_diffusion.diffusers_pipeline import PipelineIntermediateState
from invokeai.backend.stable_diffusion.diffusion.conditioning_data import WanConditioningInfo
from invokeai.backend.util.devices import TorchDevice
from invokeai.backend.wan.sampling_utils import get_spatial_scale_factor, make_noise
# Type alias: a factory that produces a fresh iterator of (LoRA patch, weight)
# pairs each time it is called. We need fresh iterators because the patcher
# consumes the iterator once per ``apply_smart_model_patches`` invocation, and
# the expert may be swapped (and re-entered) multiple times in a render.
LoRAIteratorFactory = Callable[[], Iterable[Tuple[ModelPatchRaw, float]]]
def _resolve_variant(context: InvocationContext, transformer_field: WanTransformerField) -> WanVariantType:
"""Look up the Wan variant from the main model config that produced this transformer."""
config = context.models.get_config(transformer_field.transformer)
variant = getattr(config, "variant", None)
if not isinstance(variant, WanVariantType):
raise ValueError(f"Could not determine Wan variant from model {config.name!r}: variant is {variant!r}.")
return variant
def _validate_spatial_dimensions(variant: WanVariantType, width: int, height: int) -> None:
if variant == WanVariantType.TI2V_5B and (width % 32 or height % 32):
raise ValueError(
f"TI2V-5B requires width and height to be multiples of 32 (got {width}x{height}). "
"Wan 2.2-VAE 16x spatial * transformer patch_size 2 = pixel dims must divide by 32."
)
def _validate_ref_condition_shape(
condition: torch.Tensor,
*,
channels: int,
frames: int,
height: int,
width: int,
) -> None:
if condition.ndim != 5:
raise ValueError(f"Wan reference condition must be a 5D tensor; got shape {tuple(condition.shape)}.")
if condition.shape[0] != 1:
raise ValueError(f"Wan reference condition requires batch size 1; got {condition.shape[0]}.")
if condition.shape[1] != channels:
raise ValueError(f"Wan reference condition requires {channels} channels; got {condition.shape[1]}.")
if condition.shape[2] != frames:
expected = "a single latent frame" if frames == 1 else f"{frames} latent frames"
raise ValueError(f"Wan reference condition requires {expected}; got {condition.shape[2]}.")
if condition.shape[3:] != (height, width):
raise ValueError(
f"Wan reference condition requires {width}x{height} latent spatial dimensions; "
f"got {condition.shape[4]}x{condition.shape[3]}."
)
def _scheduler_path_for_transformer(context: InvocationContext, transformer_field: WanTransformerField) -> Path | None:
"""Return the on-disk ``scheduler/`` directory for the main model, or None."""
config = context.models.get_config(transformer_field.transformer)
model_root = context.models.get_absolute_path(config)
if model_root.is_file():
return None
candidate = model_root / "scheduler"
if (candidate / "scheduler_config.json").exists():
return candidate
return None
def _default_scheduler_for_variant(variant: WanVariantType):
"""Build a variant-appropriate scheduler when no on-disk config is available.
Standalone GGUF / single-file installs don't ship a ``scheduler/`` directory,
so we have to reconstruct the scheduler from variant knowledge. Values are
verbatim from each variant's ``scheduler/scheduler_config.json`` in the
matching ``Wan-AI/Wan2.2-*-Diffusers`` repo.
"""
from diffusers import UniPCMultistepScheduler
# Both Wan 2.2 families ship UniPCMultistepScheduler with flow_prediction +
# use_flow_sigmas; only the exponential flow_shift differs (5.0 for TI2V-5B,
# 3.0 for the A14B pair). Without these settings samples drift — an unshifted
# schedule also skews how many A14B steps land above the MoE expert boundary.
flow_shift = 5.0 if variant == WanVariantType.TI2V_5B else 3.0
return UniPCMultistepScheduler(
num_train_timesteps=1000,
solver_order=2,
prediction_type="flow_prediction",
flow_shift=flow_shift,
use_flow_sigmas=True,
solver_type="bh2",
final_sigmas_type="zero",
)
class _ExpertSwapper:
"""Manages GPU residency and LoRA patching of one or two Wan transformer experts.
Both experts are kept in the model cache (system RAM); only one is on
device at a time. ``get(label)`` returns the model for the requested label,
swapping GPU residency when the label changes and applying that expert's
LoRA patches via ``LayerPatcher.apply_smart_model_patches``.
Ordering on swap: exit the active expert's LoRA context (restores weights)
-> exit ``model_on_device`` (returns expert to RAM) -> load the new expert
(fresh handle) -> enter its device context -> apply its LoRAs. This
mirrors the pattern used by ``flux_denoise``/``anima_denoise`` but adds
the extra context layer needed for dual experts.
Model handles are obtained lazily inside ``get()`` rather than cached at
construction. With dual ~9 GB GGUF experts plus a UMT5-XXL encoder
competing for the RAM cache, holding both ``LoadedModel`` handles upfront
can leave one of them stale by the time the swap happens — InvokeAI's
model cache emits a ``has already been dropped from the RAM cache``
warning and reloads from disk per swap. See issue #7513 for the broader
pattern.
"""
HIGH = "high"
LOW = "low"
def __init__(
self,
context: InvocationContext,
high_model: Any,
low_model: Any | None,
inference_dtype: torch.dtype,
high_lora_factory: LoRAIteratorFactory | None = None,
low_lora_factory: LoRAIteratorFactory | None = None,
high_is_quantized: bool = False,
low_is_quantized: bool = False,
) -> None:
self._context = context
self._high_model = high_model
self._low_model = low_model
self._inference_dtype = inference_dtype
self._high_lora_factory = high_lora_factory
self._low_lora_factory = low_lora_factory
self._high_is_quantized = high_is_quantized
self._low_is_quantized = low_is_quantized
self._active_label: str | None = None
self._active_info: Any | None = None
self._active_device_ctx: Any | None = None
self._active_lora_ctx: Any | None = None
self._active_model: Any | None = None
def get(self, label: str) -> Any:
if label not in (self.HIGH, self.LOW):
raise ValueError(f"Unknown expert label: {label!r}")
if label == self.LOW and self._low_model is None:
raise ValueError("Low-noise expert was requested but is not available.")
if label == self._active_label:
assert self._active_model is not None
return self._active_model
# Capture the outgoing expert's cache record before _release() drops our handle.
# We need it to force-unload below.
outgoing_cached_model = None
if self._active_info is not None:
# ``LoadedModel`` exposes its cache_record only via a private attribute. There
# is no public ``unload_from_vram`` on the LoadedModel today, and we don't want
# to take on a broader backend refactor in this fix; tolerate AttributeError
# so a future refactor doesn't break the swap.
outgoing_cached_model = getattr(self._active_info, "_cache_record", None)
if outgoing_cached_model is not None:
outgoing_cached_model = getattr(outgoing_cached_model, "cached_model", None)
# Release current GPU residency before bringing the other expert on device.
self._release()
# Force the outgoing expert off GPU. The model cache's automatic offload
# (inside lock() -> _offload_unlocked_models) decides how much to free based on
# ``torch.cuda.memory_allocated()`` minus a 3 GB working-memory budget. With Wan
# 81-frame video the intermediate activations from the previous denoise step are
# still allocated alongside the just-unlocked high-noise expert, so the cache
# underestimates how much room the new expert really needs and partial-loads
# most of its layers to CPU. The user-visible symptom: log line "Loaded model
# ... VRAM: 2381 MB (25.9%)" instead of ~100% for the incoming expert.
#
# Sidestep the heuristic by explicitly unloading every weight of the outgoing
# expert to RAM. This is safe even if the cache evicted the entry between unlock
# and now — the cached_model object still owns the tensors.
if outgoing_cached_model is not None:
try:
outgoing_cached_model.full_unload_from_vram()
except Exception:
pass
# Hand the PyTorch allocator a clean slate before partial_load_to_vram measures
# free space — the freed blocks stay pinned in the caching allocator until
# empty_cache is called.
TorchDevice.empty_cache()
# Load the requested expert lazily so its ``LoadedModel`` handle is
# always fresh — see class docstring for the cache-eviction reasoning.
model_id = self._high_model if label == self.HIGH else self._low_model
info = self._context.models.load(model_id)
device_ctx = info.model_on_device()
cached_weights, model = device_ctx.__enter__()
# Stash the device-context state immediately. If anything below fails (most
# likely the LoRA patcher), the surrounding ExitStack will eventually call
# ``swapper.close() -> _release()`` to clean up, and ``_release`` needs to
# see the device context to exit it. Without this early stash the expert's
# weights stay GPU-resident (8-9 GB for GGUF experts) until the cache's LRU
# finally evicts them, masquerading as a memory leak.
self._active_label = label
self._active_info = info
self._active_device_ctx = device_ctx
self._active_model = model
# Apply LoRA patches for this expert. GGUF transformers need sidecar
# patching since direct patching of GGMLTensors isn't supported.
lora_factory = self._high_lora_factory if label == self.HIGH else self._low_lora_factory
is_quantized = self._high_is_quantized if label == self.HIGH else self._low_is_quantized
lora_ctx: Any | None = None
if lora_factory is not None:
lora_ctx = LayerPatcher.apply_smart_model_patches(
model=model,
patches=lora_factory(),
prefix=WAN_LORA_TRANSFORMER_PREFIX,
dtype=self._inference_dtype,
cached_weights=cached_weights,
force_sidecar_patching=is_quantized,
)
lora_ctx.__enter__()
self._active_lora_ctx = lora_ctx
return model
def _release(self) -> None:
# LoRA context first so weights are restored before the model leaves GPU. The
# device context must exit even if weight-restore raises — otherwise the cache
# record stays locked and the expert is pinned in VRAM for the whole process.
# The field-clearing sits in its own finally so that a device-exit failure
# cannot leave stale contexts behind: a later close() would double-exit the
# already-exited LoRA context, and get() could hand back a stale model.
try:
if self._active_lora_ctx is not None:
self._active_lora_ctx.__exit__(None, None, None)
finally:
try:
if self._active_device_ctx is not None:
self._active_device_ctx.__exit__(None, None, None)
finally:
self._active_label = None
self._active_info = None
self._active_device_ctx = None
self._active_lora_ctx = None
self._active_model = None
def close(self) -> None:
self._release()
@invocation(
"wan_denoise",
title="Denoise - Wan 2.2",
tags=["image", "wan"],
category="image",
version="1.0.0",
classification=Classification.Prototype,
)
class WanDenoiseInvocation(BaseInvocation):
"""Run the denoising process with a Wan 2.2 model.
Drives the flow-matching schedule the model ships with (UniPC with flow
sigmas for the Wan 2.2 reference repos; see ``_build_scheduler``). CFG is
supported when negative conditioning is provided and ``guidance_scale != 1.0``.
For Wan 2.2 A14B the high-noise expert handles timesteps at and above
``boundary_ratio * num_train_timesteps``; the low-noise expert handles
timesteps below. Both experts share the model cache; only the active one is
GPU-resident at any time.
"""
transformer: WanTransformerField = InputField(
description="Wan transformer field (transformer + optional dual-expert metadata).",
input=Input.Connection,
title="Transformer",
)
positive_conditioning: WanConditioningField = InputField(
description=FieldDescriptions.positive_cond, input=Input.Connection
)
negative_conditioning: Optional[WanConditioningField] = InputField(
default=None, description=FieldDescriptions.negative_cond, input=Input.Connection
)
ref_image: Optional[WanRefImageConditioningField] = InputField(
default=None,
description=FieldDescriptions.wan_ref_image,
input=Input.Connection,
title="Reference Image",
)
latents: Optional[LatentsField] = InputField(
default=None,
description=FieldDescriptions.latents,
input=Input.Connection,
)
denoise_mask: Optional[DenoiseMaskField] = InputField(
default=None,
description=FieldDescriptions.denoise_mask,
input=Input.Connection,
)
denoising_start: float = InputField(default=0.0, ge=0, le=1, description=FieldDescriptions.denoising_start)
denoising_end: float = InputField(default=1.0, ge=0, le=1, description=FieldDescriptions.denoising_end)
add_noise: bool = InputField(default=True, description="Add noise based on denoising start.")
guidance_scale: float = InputField(
default=4.0,
ge=1.0,
description="Classifier-free guidance scale. 4.0 is the Wan 2.2 default for A14B; "
"TI2V-5B can tolerate higher values up to ~5.5.",
title="Guidance Scale",
)
guidance_scale_low_noise: Optional[float] = InputField(
default=None,
ge=0.0,
description="Optional separate CFG scale for the low-noise expert (Wan 2.2 A14B only). "
"Values below 1.0 (including 0) fall back to the primary 'Guidance Scale'. "
"Ignored for TI2V-5B.",
title="Guidance Scale (Low Noise)",
)
# Wan transformer has ``patch_size=(1, 2, 2)``: combined with the VAE's
# 8x spatial scale, generated H/W must be a multiple of 16 (not just 8)
# or the patch round-trip lands off-by-one and the scheduler step fails
# with a spatial-dim mismatch.
width: int = InputField(default=1024, gt=0, multiple_of=16, description="Width of the generated image.")
height: int = InputField(default=1024, gt=0, multiple_of=16, description="Height of the generated image.")
steps: int = InputField(default=40, gt=0, description="Number of denoising steps.")
seed: int = InputField(default=0, description="Randomness seed for reproducibility.")
@torch.no_grad()
def invoke(self, context: InvocationContext) -> LatentsOutput:
latents = self._run_diffusion(context)
latents = latents.detach().to("cpu")
name = context.tensors.save(tensor=latents)
return LatentsOutput.build(latents_name=name, latents=latents, seed=None)
def _run_diffusion(self, context: InvocationContext) -> torch.Tensor:
if self.denoising_start >= self.denoising_end:
raise ValueError(
f"denoising_start ({self.denoising_start}) must be less than denoising_end ({self.denoising_end})."
)
device = TorchDevice.choose_torch_device()
inference_dtype = TorchDevice.choose_bfloat16_safe_dtype(device)
variant = _resolve_variant(context, self.transformer)
_validate_spatial_dimensions(variant, self.width, self.height)
spatial_scale = get_spatial_scale_factor(variant)
scheduler = self._build_scheduler(context, device)
pos_cond = self._load_conditioning(context, self.positive_conditioning, device=device, dtype=inference_dtype)
low_cfg_enabled = (
self.transformer.transformer_low_noise is not None
and self.guidance_scale_low_noise is not None
and self.guidance_scale_low_noise >= 1.0
and self.guidance_scale_low_noise != 1.0
)
do_cfg = self.negative_conditioning is not None and (self.guidance_scale != 1.0 or low_cfg_enabled)
neg_cond: WanConditioningInfo | None = None
if do_cfg:
assert self.negative_conditioning is not None
neg_cond = self._load_conditioning(
context, self.negative_conditioning, device=device, dtype=inference_dtype
)
# Reference-image conditioning (Wan 2.2 I2V-A14B only). The condition
# tensor is 20 channels (4 mask + 16 VAE-encoded image latents); it
# gets concatenated to the 16-channel noise latents each step,
# yielding the 36-channel input the I2V transformer expects.
ref_condition: torch.Tensor | None = None
if self.ref_image is not None:
if variant != WanVariantType.I2V_A14B:
raise ValueError(
f"Reference-image conditioning is only supported by the Wan 2.2 I2V variant. "
f"The selected transformer is {variant.value!r}. Remove the Reference Image input "
"or load an I2V model."
)
if self.ref_image.width != self.width or self.ref_image.height != self.height:
raise ValueError(
f"Reference-image dimensions ({self.ref_image.width}x{self.ref_image.height}) must "
f"match denoise dimensions ({self.width}x{self.height})."
)
if self.ref_image.num_frames > 1:
# The image denoise produces single-frame output; concatenating a multi-frame
# condition to a single-frame noise tensor mismatches the temporal dim and the
# downstream tensor-shape error would be unhelpful.
raise ValueError(
f"This denoise node produces a single-frame image but the reference image was "
f"encoded for {self.ref_image.num_frames} frames. Use the Denoise Video - Wan 2.2 "
"node for video I2V, or set num_frames=1 on the Reference Image node."
)
ref_condition = context.tensors.load(self.ref_image.condition_tensor_name).to(
device=device, dtype=inference_dtype
)
_validate_ref_condition_shape(
ref_condition,
channels=20,
frames=1,
height=self.height // spatial_scale,
width=self.width // spatial_scale,
)
# Schedule timesteps. set_timesteps populates scheduler.timesteps and
# scheduler.sigmas (where sigmas is in [0, 1] flow-matching space).
scheduler.set_timesteps(num_inference_steps=self.steps, device=device)
timesteps = scheduler.timesteps
# sigmas has length steps + 1.
sigmas = scheduler.sigmas
# Apply denoising_start / denoising_end clipping.
if self.denoising_start > 0 or self.denoising_end < 1:
start_idx = int(self.denoising_start * self.steps)
end_idx = int(self.denoising_end * self.steps)
timesteps = timesteps[start_idx:end_idx]
sigmas = sigmas[start_idx : end_idx + 1]
total_steps = len(timesteps)
# Latents stay in fp32 throughout the denoise loop to avoid accumulating
# bf16 quantization across the scheduler's small per-step deltas. We
# cast to bf16 only when calling the transformer, matching Diffusers'
# WanPipeline (which calls ``prepare_latents(..., dtype=torch.float32)``
# then ``latent_model_input = latents.to(transformer_dtype)``).
latent_dtype = torch.float32
# Load init latents (img2img) and convert 4D → 5D.
init_latents_5d: torch.Tensor | None = None
if self.latents is not None:
loaded = context.tensors.load(self.latents.latents_name).to(device=device, dtype=latent_dtype)
if loaded.ndim not in (4, 5):
raise ValueError(
f"Wan image denoise expects a 4D or 5D initial latent tensor; got {tuple(loaded.shape)}."
)
if loaded.shape[0] != 1:
raise ValueError(
f"Wan image denoise requires initial latents with batch size 1; got {loaded.shape[0]}."
)
if loaded.ndim == 5 and loaded.shape[2] != 1:
raise ValueError(
f"Wan image denoise requires single-frame initial latents; got {loaded.shape[2]} frames."
)
if loaded.ndim == 4:
loaded = loaded.unsqueeze(2)
expected_channels = 48 if variant == WanVariantType.TI2V_5B else 16
if loaded.shape[1] != expected_channels:
raise ValueError(
f"Wan {variant.value} image denoise expects {expected_channels} channels in initial latents; "
f"got {loaded.shape[1]}."
)
expected_height = self.height // spatial_scale
expected_width = self.width // spatial_scale
if loaded.shape[-2:] != (expected_height, expected_width):
raise ValueError(
f"Wan image denoise expects initial latent dimensions {expected_height}x{expected_width}; "
f"got {loaded.shape[-2]}x{loaded.shape[-1]}."
)
init_latents_5d = loaded
# Determine the latent channel count. Prefer init_latents shape; otherwise
# fall back to the variant default. (We avoid loading the transformer just
# to read .config.in_channels; the variant gives us the right answer.)
latent_channels = (
init_latents_5d.shape[1]
if init_latents_5d is not None
else (48 if variant == WanVariantType.TI2V_5B else 16)
)
noise = make_noise(
batch_size=1,
latent_channels=latent_channels,
height=self.height,
width=self.width,
spatial_scale_factor=spatial_scale,
device=device,
dtype=latent_dtype,
seed=self.seed,
)
# Combine init latents + noise per the schedule's starting sigma.
if init_latents_5d is not None:
if self.add_noise:
s_0 = float(sigmas[0])
latents = s_0 * noise + (1.0 - s_0) * init_latents_5d
else:
latents = init_latents_5d
else:
if self.denoising_start > 1e-5:
raise ValueError("denoising_start should be 0 when initial latents are not provided.")
latents = noise
if total_steps <= 0:
return latents.squeeze(2)
# Inpaint extension (4D space — the existing extension is shape-agnostic
# but operates on the squeezed-T shape we use for masks).
inpaint_mask = self._prep_inpaint_mask(context, latents.squeeze(2))
inpaint_extension: RectifiedFlowInpaintExtension | None = None
if inpaint_mask is not None:
if init_latents_5d is None:
raise ValueError("Initial latents are required when using an inpaint mask (img2img inpainting).")
inpaint_extension = RectifiedFlowInpaintExtension(
init_latents=init_latents_5d.squeeze(2),
inpaint_mask=inpaint_mask,
noise=noise.squeeze(2),
)
step_callback = self._build_step_callback(context)
# Resolve experts and the boundary timestep that triggers the MoE swap.
#
# We deliberately do NOT call ``context.models.load(...)`` for the
# transformer experts here — that would put both ~9 GB GGUF handles
# in the model cache concurrently. With UMT5-XXL (~10 GB) competing
# for the same cache, the LRU policy can drop one of them by the
# time the denoise loop swaps in, producing the
# "has already been dropped from the RAM cache" warning and forcing
# a disk reload per swap. The swapper calls ``models.load`` lazily
# inside each ``get()`` instead, so handles are always fresh.
#
# The config metadata (variant / format) is fine to read upfront —
# ``get_config`` doesn't touch the weights cache.
high_model = self.transformer.transformer
low_model = self.transformer.transformer_low_noise
low_config = context.models.get_config(low_model) if low_model is not None else None
# FlowMatchEulerDiscreteScheduler stores num_train_timesteps in its config
# (default 1000). Diffusers' WanPipeline computes:
# boundary_timestep = boundary_ratio * num_train_timesteps
num_train_timesteps = int(scheduler.config.num_train_timesteps)
boundary_timestep = self.transformer.boundary_ratio * num_train_timesteps if low_model is not None else None
# LoRA wiring. The high-noise expert uses ``transformer.loras``; the
# low-noise expert uses ``transformer.loras_low_noise``. No fallback
# between the lists: the Wan LoRA loader routes untagged LoRAs to both
# and expert-tagged LoRAs to exactly one, so an empty low list means
# "no LoRAs on the low expert" — reusing the primary list here would
# silently apply high-expert-only LoRAs (e.g. Lightning high-noise
# distills) to the wrong expert.
# Quantized (GGUF) experts force sidecar patching so GGMLTensor weights
# aren't touched directly.
high_loras = self.transformer.loras
low_loras = self.transformer.loras_low_noise
high_config = context.models.get_config(high_model)
high_is_quantized = high_config.format == ModelFormat.GGUFQuantized
low_is_quantized = low_config.format == ModelFormat.GGUFQuantized if low_config is not None else False
def high_lora_factory() -> Iterable[Tuple[ModelPatchRaw, float]]:
return self._lora_iterator(context, high_loras)
def low_lora_factory() -> Iterable[Tuple[ModelPatchRaw, float]]:
return self._lora_iterator(context, low_loras)
with ExitStack() as exit_stack:
swapper = _ExpertSwapper(
context=context,
high_model=high_model,
low_model=low_model,
inference_dtype=inference_dtype,
high_lora_factory=high_lora_factory if high_loras else None,
low_lora_factory=low_lora_factory if low_loras else None,
high_is_quantized=high_is_quantized,
low_is_quantized=low_is_quantized,
)
exit_stack.callback(swapper.close)
for step_idx, t in enumerate(tqdm(timesteps, desc="Denoising (Wan 2.2)", total=total_steps)):
timestep = t.expand(latents.shape[0])
# Pick the active expert: high-noise for t >= boundary_timestep,
# low-noise below. Single-transformer models always use HIGH.
if low_model is not None and float(t) < float(boundary_timestep):
active_label = _ExpertSwapper.LOW
# Treat None or values below 1.0 (incl. the FE's default 0)
# as "use the primary guidance_scale".
low_cfg = self.guidance_scale_low_noise
active_cfg = low_cfg if (low_cfg is not None and low_cfg >= 1.0) else self.guidance_scale
else:
active_label = _ExpertSwapper.HIGH
active_cfg = self.guidance_scale
transformer = swapper.get(active_label)
# Cast latents to the transformer's dtype only for the forward
# pass; keep the scheduler-level latents in fp32.
latent_model_input = latents.to(dtype=inference_dtype)
# For I2V, concatenate the ref-image condition (4-ch mask + 16-ch
# image latents) along the channel dim, producing the 36-channel
# input the I2V transformer's patch_embedding expects.
if ref_condition is not None:
latent_model_input = torch.cat([latent_model_input, ref_condition], dim=1)
noise_pred_cond = transformer(
hidden_states=latent_model_input,
timestep=timestep,
encoder_hidden_states=pos_cond.prompt_embeds.unsqueeze(0),
attention_kwargs=None,
return_dict=False,
)[0]
if neg_cond is not None and active_cfg != 1.0:
noise_pred_uncond = transformer(
hidden_states=latent_model_input,
timestep=timestep,
encoder_hidden_states=neg_cond.prompt_embeds.unsqueeze(0),
attention_kwargs=None,
return_dict=False,
)[0]
noise_pred = noise_pred_uncond + active_cfg * (noise_pred_cond - noise_pred_uncond)
else:
noise_pred = noise_pred_cond
latents = scheduler.step(noise_pred, t, latents, return_dict=False)[0]
if inpaint_extension is not None:
sigma_prev = float(sigmas[step_idx + 1])
latents_4d = latents.squeeze(2)
latents_4d = inpaint_extension.merge_intermediate_latents_with_init_latents(latents_4d, sigma_prev)
latents = latents_4d.unsqueeze(2)
step_callback(
PipelineIntermediateState(
step=step_idx + 1,
order=1,
total_steps=total_steps,
timestep=int(t.item()),
latents=latents.squeeze(2),
)
)
# Squeeze T for downstream 4D consumers.
return latents.squeeze(2)
def _build_scheduler(self, context: InvocationContext, device: torch.device):
"""Construct the scheduler matching the model's on-disk ``scheduler_config.json``.
All Wan 2.2 variants ship ``UniPCMultistepScheduler`` with flow-matching
settings, but with different exponential shifts (TI2V-5B: ``flow_shift=5.0``,
A14B: ``3.0``) — and future variants may differ further, so when a
``scheduler/`` directory exists we dispatch on its ``_class_name`` and load
it verbatim. When no on-disk config is available (standalone GGUF /
single-file installs don't ship one) — or the config is unreadable — fall
back to the variant-aware default so the noise schedule still matches what
the model was trained against.
"""
import json
import diffusers
scheduler_dir = _scheduler_path_for_transformer(context, self.transformer)
scheduler_cls = None
if scheduler_dir is not None:
# Read the on-disk class name and instantiate that class. Diffusers'
# SchedulerMixin.from_pretrained does class dispatch internally, but
# only when called from the abstract base; calling a concrete subclass
# silently builds the wrong type. Resolve it explicitly.
config_path = scheduler_dir / "scheduler_config.json"
try:
with config_path.open("r", encoding="utf-8") as f:
cfg = json.load(f)
class_name = cfg.get("_class_name")
scheduler_cls = getattr(diffusers, class_name, None) if class_name else None
except (OSError, json.JSONDecodeError):
scheduler_cls = None
if scheduler_dir is None or scheduler_cls is None:
variant = _resolve_variant(context, self.transformer)
return _default_scheduler_for_variant(variant)
return scheduler_cls.from_pretrained(str(scheduler_dir), local_files_only=True)
def _load_conditioning(
self,
context: InvocationContext,
cond_field: WanConditioningField,
*,
device: torch.device,
dtype: torch.dtype,
) -> WanConditioningInfo:
cond_data = context.conditioning.load(cond_field.conditioning_name)
assert len(cond_data.conditionings) == 1
cond_info = cond_data.conditionings[0]
assert isinstance(cond_info, WanConditioningInfo)
return cond_info.to(device=device, dtype=dtype)
def _prep_inpaint_mask(self, context: InvocationContext, latents_4d: torch.Tensor) -> torch.Tensor | None:
"""Resize the user-supplied mask down to latent resolution.
Convention matches Anima/FLUX: the original mask has 0 = preserve and
1 = denoise; the extension expects the inverted form.
"""
if self.denoise_mask is None:
return None
mask = context.tensors.load(self.denoise_mask.mask_name)
mask = 1.0 - mask
_, _, latent_h, latent_w = latents_4d.shape
mask = tv_resize(
img=mask,
size=[latent_h, latent_w],
interpolation=tv_transforms.InterpolationMode.BILINEAR,
antialias=False,
)
return mask.to(device=latents_4d.device, dtype=latents_4d.dtype)
def _build_step_callback(self, context: InvocationContext) -> Callable[[PipelineIntermediateState], None]:
def step_callback(state: PipelineIntermediateState) -> None:
context.util.sd_step_callback(state, BaseModelType.Wan)
return step_callback
def _lora_iterator(
self, context: InvocationContext, loras: list[LoRAField]
) -> Iterator[Tuple[ModelPatchRaw, float]]:
"""Yield (ModelPatchRaw, weight) pairs for the given LoRA list.
The caller passes either ``transformer.loras`` (high-noise expert) or
``transformer.loras_low_noise`` (low-noise expert) — the fallback to
the primary list when low-noise is empty is handled at the call site.
"""
for lora_field in loras:
lora_info = context.models.load(lora_field.lora)
assert isinstance(lora_info.model, ModelPatchRaw), (
f"Wan LoRA model must be ModelPatchRaw, got {type(lora_info.model).__name__}"
)
yield (lora_info.model, lora_field.weight)
del lora_info
@@ -0,0 +1,209 @@
"""Compute Wan 2.2-compatible pixel dimensions for a target short-side resolution.
Wan's transformer ``patch_size=(1, 2, 2)`` adds a 2x patchify on top of the VAE's
spatial compression, so pixel dimensions must be a multiple of (2 × VAE scale):
- I2V-A14B / T2V (8x VAE) → multiples of 16
- TI2V-5B (Wan 2.2-VAE, 16x VAE) → multiples of 32
This module exposes one node per family. Each takes a source image's W×H and a
target short-side preset (480p / 720p / 1080p) and returns the scaled, snapped
(width, height) that can be fed directly into the matching ``wan_ref_image_encoder``
/ ``wan_denoise`` inputs. Both nodes share :func:`_scale_and_snap`; they differ
only in the pixel multiple they snap to.
"""
import math
from typing import Literal
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
from invokeai.app.invocations.fields import InputField
from invokeai.app.invocations.ideal_size import IdealSizeOutput
from invokeai.app.services.shared.invocation_context import InvocationContext
WanTargetResolution = Literal["480p", "720p", "1080p"]
WanRounding = Literal["nearest", "floor", "ceiling"]
# Pixel-grid multiple = 2 (transformer patch) × VAE spatial scale. The 8x-VAE
# I2V/T2V models need multiples of 16; the 16x-VAE TI2V-5B needs multiples of 32.
WAN_I2V_PIXEL_MULTIPLE = 16
WAN_TI2V_PIXEL_MULTIPLE = 32
# Short-side pixel count for each preset. "p" notation is by convention the *short*
# dimension in modern video (so a portrait 720p video is 720 wide × 1280 tall).
WAN_TARGET_RESOLUTION_PX: dict[str, int] = {
"480p": 480,
"720p": 720,
"1080p": 1080,
}
WAN_TARGET_RESOLUTION_LABELS: dict[str, str] = {
"480p": "480p (Wan native)",
"720p": "720p (Wan native, default)",
"1080p": "1080p (extrapolated — not a Wan training size)",
}
def _scale_and_snap(
width: int,
height: int,
target_short_side: int,
rounding: WanRounding,
multiple: int,
) -> tuple[int, int]:
"""Scale a source W×H so its shorter side equals ``target_short_side``, then
snap each dimension to ``multiple`` using the requested rounding mode.
``multiple`` is the Wan pixel-grid constraint (16 for the 8x-VAE I2V/T2V
models, 32 for the 16x-VAE TI2V-5B). Shared by both ideal-dimensions nodes.
"""
short = min(width, height)
if short <= 0:
raise ValueError("Source dimensions must be positive.")
# Reject sources so narrow that the scaled long side is still under one Wan
# pixel grid. The downstream clamp to ``max(w, multiple)`` would otherwise
# silently return multiple×multiple, which has no relation to the requested
# aspect ratio — better to fail fast and have the workflow author fix inputs.
long_side = max(width, height)
if long_side < multiple:
raise ValueError(
f"Source longer side ({long_side}px) is smaller than the Wan pixel grid ({multiple}px). "
f"Use an input image at least {multiple}px on its longer side."
)
scale = target_short_side / short
raw_w = width * scale
raw_h = height * scale
if rounding == "floor":
w = int(raw_w // multiple) * multiple
h = int(raw_h // multiple) * multiple
elif rounding == "ceiling":
w = int(math.ceil(raw_w / multiple)) * multiple
h = int(math.ceil(raw_h / multiple)) * multiple
else: # nearest
w = round(raw_w / multiple) * multiple
h = round(raw_h / multiple) * multiple
# Belt-and-suspenders clamp against floor-of-<multiple — unreachable now that
# the long_side guard runs first, but keeps the contract that returned
# dimensions are always valid Wan inputs.
w = max(w, multiple)
h = max(h, multiple)
return w, h
@invocation(
"wan_i2v_ideal_dimensions",
title="Wan 2.2 I2V Ideal Dimensions (A14B)",
tags=["wan", "video", "dimensions", "math"],
category="video",
version="1.1.0",
classification=Classification.Prototype,
)
class WanI2VIdealDimensionsInvocation(BaseInvocation):
"""Ideal dimensions for the Wan 2.2 A14B models (I2V-A14B and T2V-A14B).
Use this node for the A14B family. For the TI2V-5B model use "Wan 2.2 TI2V
Ideal Dimensions" instead — TI2V-5B requires multiples of 32, and feeding it
these multiples-of-16 dims fails the patchify step.
Scales the input W×H so the shorter side equals the chosen preset (480 / 720 /
1080 px), then snaps each dimension to a multiple of 16 (the A14B pixel-grid
constraint). Wire from ``Image Primitive``'s width/height outputs and into
``wan_ref_image_encoder`` / ``wan_denoise``.
"""
width: int = InputField(
default=1024,
gt=0,
description="Source image width in pixels.",
)
height: int = InputField(
default=1024,
gt=0,
description="Source image height in pixels.",
)
target_resolution: WanTargetResolution = InputField(
default="720p",
description=(
"Short-side resolution preset. 480p and 720p are Wan 2.2's native training "
"resolutions; 1080p works but is extrapolation and costs ~2.25x the memory "
"of 720p."
),
ui_choice_labels=WAN_TARGET_RESOLUTION_LABELS,
)
rounding: WanRounding = InputField(
default="nearest",
description=(
"How to snap each dimension to a multiple of 16. 'floor' rounds down — "
"safest for VRAM, guaranteed not to exceed the unsnapped target. "
"'ceiling' rounds up. 'nearest' minimizes aspect-ratio drift (default)."
),
)
def invoke(self, context: InvocationContext) -> IdealSizeOutput:
target_short_side = WAN_TARGET_RESOLUTION_PX[self.target_resolution]
w, h = _scale_and_snap(
self.width, self.height, target_short_side, self.rounding, multiple=WAN_I2V_PIXEL_MULTIPLE
)
return IdealSizeOutput(width=w, height=h)
@invocation(
"wan_ti2v_ideal_dimensions",
title="Wan 2.2 TI2V Ideal Dimensions (5B)",
tags=["wan", "video", "dimensions", "math"],
category="video",
version="1.0.0",
classification=Classification.Prototype,
)
class WanTI2VIdealDimensionsInvocation(BaseInvocation):
"""Ideal dimensions for the Wan 2.2 TI2V-5B model.
Use this node for TI2V-5B only. For the A14B models (I2V-A14B / T2V-A14B) use
"Wan 2.2 I2V Ideal Dimensions" instead — those need multiples of 16, and this
node's multiples-of-32 dims would overshoot their pixel grid.
Identical to the A14B node but snaps each dimension to a multiple of 32 instead
of 16: the Wan 2.2-VAE used by TI2V-5B applies 16x spatial compression and the
transformer adds a 2x patch on top, so pixel dims must divide by 32 for the
patchify step. Wire from ``Image Primitive``'s width/height outputs and into
the matching ``wan_denoise`` inputs.
"""
width: int = InputField(
default=1024,
gt=0,
description="Source image width in pixels.",
)
height: int = InputField(
default=1024,
gt=0,
description="Source image height in pixels.",
)
target_resolution: WanTargetResolution = InputField(
default="720p",
description=(
"Short-side resolution preset. 480p and 720p are Wan 2.2's native training "
"resolutions; 1080p works but is extrapolation and costs ~2.25x the memory "
"of 720p."
),
ui_choice_labels=WAN_TARGET_RESOLUTION_LABELS,
)
rounding: WanRounding = InputField(
default="nearest",
description=(
"How to snap each dimension to a multiple of 32. 'floor' rounds down — "
"safest for VRAM, guaranteed not to exceed the unsnapped target. "
"'ceiling' rounds up. 'nearest' minimizes aspect-ratio drift (default)."
),
)
def invoke(self, context: InvocationContext) -> IdealSizeOutput:
target_short_side = WAN_TARGET_RESOLUTION_PX[self.target_resolution]
w, h = _scale_and_snap(
self.width, self.height, target_short_side, self.rounding, multiple=WAN_TI2V_PIXEL_MULTIPLE
)
return IdealSizeOutput(width=w, height=h)
@@ -0,0 +1,106 @@
"""Wan 2.2 image-to-latents invocation.
Encodes an image to latent space using the Wan VAE (AutoencoderKLWan). The Wan
VAE expects 5D ``[B, C, T, H, W]`` input with ``T=1`` for single images. After
encoding, latents are normalised against the per-channel ``latents_mean`` and
``latents_std`` stored in the VAE config — this matches the Diffusers
``WanPipeline`` reference and is the inverse of the denormalisation in
``wan_latents_to_image.py``.
"""
import einops
import torch
from diffusers.models.autoencoders import AutoencoderKLWan
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
from invokeai.app.invocations.fields import (
FieldDescriptions,
ImageField,
Input,
InputField,
WithBoard,
WithMetadata,
)
from invokeai.app.invocations.model import VAEField
from invokeai.app.invocations.primitives import LatentsOutput
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.backend.model_manager.load.load_base import LoadedModel
from invokeai.backend.model_manager.load.model_cache.utils import get_effective_device
from invokeai.backend.stable_diffusion.diffusers_pipeline import image_resized_to_grid_as_tensor
from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_wan
@invocation(
"wan_i2l",
title="Image to Latents - Wan 2.2",
tags=["image", "latents", "vae", "i2l", "wan"],
category="image",
version="1.0.0",
classification=Classification.Prototype,
)
class WanImageToLatentsInvocation(BaseInvocation, WithMetadata, WithBoard):
"""Encodes an image with the Wan VAE (AutoencoderKLWan).
The output latents have the temporal dimension squeezed out, so downstream
nodes see 4D ``[B, C, H, W]``. The denoise loop re-adds ``T=1`` before
feeding the transformer.
"""
image: ImageField = InputField(description="The image to encode.")
vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection)
@staticmethod
def vae_encode(vae_info: LoadedModel, image_tensor: torch.Tensor) -> torch.Tensor:
if not isinstance(vae_info.model, AutoencoderKLWan):
raise TypeError(f"Expected AutoencoderKLWan for Wan VAE, got {type(vae_info.model).__name__}.")
estimated_working_memory = estimate_vae_working_memory_wan(
operation="encode",
vae=vae_info.model,
pixel_height=image_tensor.shape[-2],
pixel_width=image_tensor.shape[-1],
pixel_frames=image_tensor.shape[2] if image_tensor.ndim == 5 else 1,
)
with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae):
assert isinstance(vae, AutoencoderKLWan)
vae_dtype = next(iter(vae.parameters())).dtype
image_tensor = image_tensor.to(device=get_effective_device(vae), dtype=vae_dtype)
with torch.inference_mode():
# Wan VAE expects 5D [B, C, T, H, W].
if image_tensor.ndim == 4:
image_tensor = image_tensor.unsqueeze(2) # [B, C, H, W] -> [B, C, 1, H, W]
encoded = vae.encode(image_tensor, return_dict=False)[0]
latents = encoded.sample().to(dtype=vae_dtype)
# Normalise to the denoiser's expected zero-centred space:
# (latents - mean) / std
latents_mean = torch.tensor(vae.config.latents_mean).view(1, -1, 1, 1, 1).to(latents)
latents_std = torch.tensor(vae.config.latents_std).view(1, -1, 1, 1, 1).to(latents)
latents = (latents - latents_mean) / latents_std
# Drop the temporal dim to keep the rest of the InvokeAI pipeline 4D.
if latents.ndim == 5:
latents = latents.squeeze(2)
return latents
@torch.no_grad()
def invoke(self, context: InvocationContext) -> LatentsOutput:
image = context.images.get_pil(self.image.image_name)
image_tensor = image_resized_to_grid_as_tensor(image.convert("RGB"))
if image_tensor.dim() == 3:
image_tensor = einops.rearrange(image_tensor, "c h w -> 1 c h w")
vae_info = context.models.load(self.vae.vae)
context.util.signal_progress("Running Wan VAE encode")
latents = self.vae_encode(vae_info=vae_info, image_tensor=image_tensor)
latents = latents.to("cpu")
name = context.tensors.save(tensor=latents)
return LatentsOutput.build(latents_name=name, latents=latents, seed=None)
@@ -0,0 +1,121 @@
"""Wan 2.2 latents-to-image invocation.
Decodes Wan latents using the Wan VAE (AutoencoderKLWan).
Latents from the denoise loop are in normalised space (zero-centred). Before
VAE decode they are denormalised using the VAE config's per-channel
``latents_mean`` / ``latents_std`` (matching Diffusers ``WanPipeline``).
The VAE expects 5D ``[B, C, T, H, W]``; downstream nodes work with 4D, so this
node re-adds ``T=1`` before decode and squeezes it back out afterwards.
"""
import torch
from diffusers.models.autoencoders import AutoencoderKLWan
from einops import rearrange
from PIL import Image
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
from invokeai.app.invocations.fields import (
FieldDescriptions,
Input,
InputField,
LatentsField,
WithBoard,
WithMetadata,
)
from invokeai.app.invocations.model import VAEField
from invokeai.app.invocations.primitives import ImageOutput
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.backend.model_manager.load.model_cache.utils import get_effective_device
from invokeai.backend.util.devices import TorchDevice
from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_wan
@invocation(
"wan_l2i",
title="Latents to Image - Wan 2.2",
tags=["latents", "image", "vae", "l2i", "wan"],
category="latents",
version="1.0.0",
classification=Classification.Prototype,
)
class WanLatentsToImageInvocation(BaseInvocation, WithMetadata, WithBoard):
"""Decodes Wan latents back to RGB."""
latents: LatentsField = InputField(description=FieldDescriptions.latents, input=Input.Connection)
vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection)
@torch.no_grad()
def invoke(self, context: InvocationContext) -> ImageOutput:
latents = context.tensors.load(self.latents.latents_name)
if latents.ndim not in (4, 5):
raise ValueError(
f"Wan latents-to-image expects a 4D or 5D latent tensor [B, C, (T), H, W]; got {tuple(latents.shape)}."
)
if latents.shape[0] != 1:
raise ValueError(f"Wan latents-to-image requires batch size 1; got {latents.shape[0]}.")
# This node decodes exactly one image. Multi-frame video latents would otherwise
# run the full (expensive) multi-frame VAE decode — under a working-memory
# estimate that assumed one frame — and then die in an opaque einops rank error
# at the final rearrange. Checked before the VAE is even loaded.
if latents.ndim == 5 and latents.shape[2] != 1:
raise ValueError(
f"These latents hold {latents.shape[2]} frames of video; this node decodes a single "
"image. Use 'Latents to Video - Wan 2.2' (wan_l2v) for video latents."
)
vae_info = context.models.load(self.vae.vae)
if not isinstance(vae_info.model, AutoencoderKLWan):
raise TypeError(f"Expected AutoencoderKLWan for Wan VAE, got {type(vae_info.model).__name__}.")
spatial_scale = getattr(vae_info.model.config, "scale_factor_spatial", None) or 8
estimated_working_memory = estimate_vae_working_memory_wan(
operation="decode",
vae=vae_info.model,
pixel_height=latents.shape[-2] * spatial_scale,
pixel_width=latents.shape[-1] * spatial_scale,
pixel_frames=1,
)
with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae):
context.util.signal_progress("Running Wan VAE decode")
assert isinstance(vae, AutoencoderKLWan)
vae_dtype = next(iter(vae.parameters())).dtype
latents = latents.to(device=get_effective_device(vae), dtype=vae_dtype)
TorchDevice.empty_cache()
with torch.inference_mode():
# Re-add the temporal dim if upstream squeezed it out.
if latents.ndim == 4:
latents = latents.unsqueeze(2)
if latents.shape[1] != vae.config.z_dim:
raise ValueError(
f"Latent channel mismatch: these latents have {latents.shape[1]} channels but the "
f"selected VAE expects {vae.config.z_dim}. A14B models need the 16-channel Wan 2.1 "
"VAE; TI2V-5B needs the 48-channel Wan 2.2 VAE."
)
# Denormalise from denoiser space back to raw VAE space.
latents_mean = torch.tensor(vae.config.latents_mean).view(1, -1, 1, 1, 1).to(latents)
latents_std = torch.tensor(vae.config.latents_std).view(1, -1, 1, 1, 1).to(latents)
latents = latents * latents_std + latents_mean
decoded = vae.decode(latents, return_dict=False)[0]
if decoded.ndim == 5:
decoded = decoded.squeeze(2)
img = decoded.clamp(-1, 1)
img = rearrange(img[0], "c h w -> h w c")
img_pil = Image.fromarray((127.5 * (img + 1.0)).byte().cpu().numpy())
TorchDevice.empty_cache()
image_dto = context.images.save(image=img_pil)
return ImageOutput.build(image_dto)
@@ -0,0 +1,230 @@
"""Wan 2.2 latents-to-video invocation.
Decodes multi-frame Wan latents with the Wan VAE and encodes the result to an
MP4 file via :mod:`imageio` (backed by the bundled FFmpeg binary from
``imageio-ffmpeg``). The video is then persisted through ``context.videos.save``,
which moves the temp file into ``outputs/videos/`` and records the DTO.
Latent shape on input is 5D ``[B, C, T_lat, H_lat, W_lat]`` (typically B=1).
The VAE expands the temporal dim by 4× during decode minus the initial offset:
``T_pixel = (T_lat - 1) * 4 + 1`` (e.g. T_lat=21 → 81 pixel frames).
"""
import tempfile
from collections.abc import Iterable, Iterator
from pathlib import Path
from typing import Callable, Protocol
import numpy as np
import torch
from diffusers.models.autoencoders import AutoencoderKLWan
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
from invokeai.app.invocations.fields import (
FieldDescriptions,
Input,
InputField,
LatentsField,
WithBoard,
WithMetadata,
)
from invokeai.app.invocations.model import VAEField
from invokeai.app.invocations.primitives import VideoOutput
from invokeai.app.services.session_processor.session_processor_common import CanceledException
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.app.util.video_encoding import make_mp4_writer
from invokeai.backend.model_manager.load.model_cache.utils import get_effective_device
from invokeai.backend.util.devices import TorchDevice
from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_wan
class _FrameWriter(Protocol):
def append_data(self, frame: np.ndarray) -> None: ...
def _validate_video_latent_batch(latents: torch.Tensor) -> None:
if latents.ndim in (4, 5) and latents.shape[0] != 1:
raise ValueError(f"Wan latents-to-video requires batch size 1; got {latents.shape[0]}.")
def _iter_decoded_frames(decoded: torch.Tensor) -> Iterator[np.ndarray]:
for index in range(decoded.shape[1]):
frame = decoded[:, index]
frame = frame.clamp(-1, 1).permute(1, 2, 0).cpu().float()
yield (127.5 * (frame + 1.0)).round().clamp(0, 255).byte().numpy()
def _write_video_frames(writer: _FrameWriter, frames: Iterable[np.ndarray], is_canceled: Callable[[], bool]) -> None:
frames_iter = iter(frames)
while True:
if is_canceled():
raise CanceledException
try:
frame = next(frames_iter)
except StopIteration:
return
writer.append_data(np.ascontiguousarray(frame))
@invocation(
"wan_l2v",
title="Latents to Video - Wan 2.2",
tags=["latents", "video", "vae", "l2v", "wan"],
category="latents",
version="1.0.0",
classification=Classification.Prototype,
)
class WanLatentsToVideoInvocation(BaseInvocation, WithMetadata, WithBoard):
"""Decode 5D Wan latents to RGB frames and encode an MP4."""
latents: LatentsField = InputField(description=FieldDescriptions.latents, input=Input.Connection)
vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection)
fps: int = InputField(
default=16,
ge=1,
le=120,
description="Frames-per-second for the encoded MP4. Wan 2.2 was trained at 16 FPS.",
)
@torch.no_grad()
def invoke(self, context: InvocationContext) -> VideoOutput:
latents = context.tensors.load(self.latents.latents_name)
_validate_video_latent_batch(latents)
if latents.ndim == 4:
# Promote 4D (single-frame) to 5D so this node can also serve as a
# one-frame "video" encode if someone wires it that way.
latents = latents.unsqueeze(2)
if latents.ndim != 5:
raise ValueError(
f"Wan latents-to-video expects a 5D latent tensor [B, C, T, H, W]; got {tuple(latents.shape)}."
)
if any(size == 0 for size in latents.shape[2:]):
raise ValueError("Wan latents-to-video requires non-empty temporal and spatial dimensions.")
vae_info = context.models.load(self.vae.vae)
if not isinstance(vae_info.model, AutoencoderKLWan):
raise TypeError(f"Expected AutoencoderKLWan for Wan VAE, got {type(vae_info.model).__name__}.")
if latents.shape[1] != vae_info.model.config.z_dim:
raise ValueError(
f"Latent channel mismatch: these latents have {latents.shape[1]} channels but the "
f"selected VAE expects {vae_info.model.config.z_dim}. A14B models need the 16-channel Wan 2.1 VAE; "
"TI2V-5B needs the 48-channel Wan 2.2 VAE."
)
_, _, t_lat, h_lat, w_lat = latents.shape
spatial_scale = getattr(vae_info.model.config, "scale_factor_spatial", None) or 8
temporal_scale = getattr(vae_info.model.config, "scale_factor_temporal", None) or 4
t_pixel = (t_lat - 1) * temporal_scale + 1
h_pixel, w_pixel = h_lat * spatial_scale, w_lat * spatial_scale
estimated_working_memory = estimate_vae_working_memory_wan(
operation="decode",
vae=vae_info.model,
pixel_height=h_pixel,
pixel_width=w_pixel,
pixel_frames=t_pixel,
)
# Long/high-res clips can need a working set no card fits. When the full-frame
# estimate exceeds the execution device's total VRAM, fall back to spatial tiling
# and budget for the tiled working set instead. (A cpu_only VAE runs in system
# RAM, where the working set is not the constraint.)
use_tiling = False
if not getattr(vae_info.config, "cpu_only", None):
exec_device = TorchDevice.choose_torch_device()
if exec_device.type == "cuda":
total_vram = torch.cuda.get_device_properties(exec_device).total_memory
if estimated_working_memory > 0.9 * total_vram:
use_tiling = True
tile_size = int(getattr(vae_info.model, "tile_sample_min_height", 256))
estimated_working_memory = estimate_vae_working_memory_wan(
operation="decode",
vae=vae_info.model,
pixel_height=h_pixel,
pixel_width=w_pixel,
pixel_frames=t_pixel,
tile_size=tile_size,
)
with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae):
assert isinstance(vae, AutoencoderKLWan)
context.logger.info(
f"Running Wan VAE decode: {t_lat} latent frames -> {t_pixel} pixel frames at {w_pixel}x{h_pixel}"
+ (" (tiled)" if use_tiling else "")
)
context.util.signal_progress("Running Wan VAE decode (video)")
vae_dtype = next(iter(vae.parameters())).dtype
latents = latents.to(device=get_effective_device(vae), dtype=vae_dtype)
TorchDevice.empty_cache()
if use_tiling:
vae.enable_tiling()
try:
with torch.inference_mode():
# Denormalise from denoiser space back to VAE space.
latents_mean = torch.tensor(vae.config.latents_mean).view(1, -1, 1, 1, 1).to(latents)
latents_std = torch.tensor(vae.config.latents_std).view(1, -1, 1, 1, 1).to(latents)
latents = latents * latents_std + latents_mean
# [B, C=3, T_pixel, H, W] in [-1, 1] (roughly).
decoded = vae.decode(latents, return_dict=False)[0]
del latents, latents_mean, latents_std
finally:
if use_tiling:
# The VAE instance is cached and shared; don't leak tiling into other nodes.
vae.disable_tiling()
# Take batch 0 (we generate one video at a time) and move the clip off the
# accelerator now — MP4 encoding can take a while, and holding the full
# decoded clip in VRAM for its duration starves the next node's model load.
decoded = decoded[0].cpu() # [C, T, H, W]
TorchDevice.empty_cache()
if context.util.is_canceled():
raise CanceledException
num_frames = decoded.shape[1]
if num_frames == 0:
raise ValueError("Wan VAE decode produced zero frames.")
height, width = decoded.shape[2:]
duration = num_frames / float(self.fps)
# Encode to a temporary MP4 (libx264 + yuv420p, exact frame dimensions —
# see make_mp4_writer for why macro_block_size matters).
tmp = tempfile.NamedTemporaryFile(prefix="invokeai_wan_video_", suffix=".mp4", delete=False)
tmp.close()
tmp_path = Path(tmp.name)
try:
context.logger.info(
f"Encoding MP4: {num_frames} frames @ {self.fps} fps ({duration:.2f}s) at {width}x{height} via libx264"
)
context.util.signal_progress(f"Encoding MP4 ({num_frames} frames @ {self.fps} fps)")
writer = make_mp4_writer(tmp_path, self.fps)
try:
_write_video_frames(writer, _iter_decoded_frames(decoded), context.util.is_canceled)
finally:
writer.close()
del decoded
TorchDevice.empty_cache()
encoded_bytes = tmp_path.stat().st_size
context.logger.info(f"MP4 encode complete: {encoded_bytes / 1024:.1f} KB")
video_dto = context.videos.save(
source_path=tmp_path,
width=width,
height=height,
duration=duration,
fps=float(self.fps),
)
context.logger.info(f"Saved video: {video_dto.video_name}")
return VideoOutput.build(video_dto)
finally:
# If save() moved the file this is a no-op; if it failed earlier, we
# don't want a lingering temp file.
try:
tmp_path.unlink(missing_ok=True)
except Exception:
pass
+265
View File
@@ -0,0 +1,265 @@
from typing import Literal, Optional
from invokeai.app.invocations.baseinvocation import (
BaseInvocation,
BaseInvocationOutput,
Classification,
invocation,
invocation_output,
)
from invokeai.app.invocations.fields import FieldDescriptions, Input, InputField, OutputField
from invokeai.app.invocations.model import LoRAField, ModelIdentifierField, WanTransformerField
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.backend.model_manager.taxonomy import (
BaseModelType,
ModelType,
WanLoRAVariantType,
WanVariantType,
)
# Target option for routing a LoRA to one or both Wan A14B expert lists.
#
# - ``auto``: read the LoRA config's ``expert`` field (set by the probe / from
# filename). ``"high"`` -> primary list only, ``"low"`` -> low-noise list
# only, ``None`` -> both lists.
# - ``both``: append to both lists regardless of the config.
# - ``high``: append only to the primary list (high-noise expert).
# - ``low``: append only to the low-noise list (low-noise expert).
WanLoRATarget = Literal["auto", "both", "high", "low"]
def _assert_is_wan_lora(lora_config: object, lora_key: str) -> None:
"""Reject an identifier whose *resolved* config is not a Wan LoRA.
The identifier's own ``base``/``type`` fields are client-supplied and cannot be
trusted: a hand-authored workflow can label any existing model key as a Wan LoRA
and reach model patching (or fail minutes in, after expensive loading). Only the
config the key actually resolves to is authoritative.
"""
config_type = getattr(lora_config, "type", None)
config_base = getattr(lora_config, "base", None)
if config_type is not ModelType.LoRA or config_base is not BaseModelType.Wan:
raise ValueError(
f"Model '{lora_key}' is not a Wan LoRA (resolved to "
f"type={getattr(config_type, 'value', config_type)}, "
f"base={getattr(config_base, 'value', config_base)})."
)
def _assert_lora_variant_matches_main(lora_config: object, main_config: object, lora_key: str) -> None:
"""Reject an A14B LoRA wired against a 5B main (and vice versa).
A mismatch otherwise crashes deep in the layer patcher mid-denoise with an opaque
tensor-shape error, after minutes of model loading. Skips silently when either
variant is unrecorded (e.g. a LoRA whose targeted layers don't pin the inner dim).
"""
lora_variant = getattr(lora_config, "variant", None)
main_variant = getattr(main_config, "variant", None)
if lora_variant is None or main_variant is None:
return
lora_is_5b = lora_variant == WanLoRAVariantType.Wan5B
main_is_5b = main_variant == WanVariantType.TI2V_5B
if lora_is_5b != main_is_5b:
raise ValueError(
f"LoRA '{lora_key}' targets Wan {lora_variant.value.upper()} models, but the "
f"transformer is a {main_variant.value} model. A14B and 5B LoRAs are not interchangeable."
)
def _warn_if_low_routing_is_inert(
context: InvocationContext, main_config: object, lora_key: str, to_primary: bool, to_low_noise: bool
) -> None:
"""Warn when a LoRA is routed only to the low-noise list of a TI2V-5B main.
The single-transformer TI2V-5B denoise path consumes only the primary list, so
such a LoRA silently has no effect — the node would otherwise report success
while doing nothing.
"""
if to_primary or not to_low_noise:
return
if getattr(main_config, "variant", None) == WanVariantType.TI2V_5B:
context.logger.warning(
f"LoRA '{lora_key}' is routed only to the low-noise expert, which the single-transformer "
"TI2V-5B variant never uses — the LoRA will have no effect."
)
def _resolve_target(target: WanLoRATarget, lora_expert: str | None) -> tuple[bool, bool]:
"""Return (apply_to_primary, apply_to_low_noise) based on the requested
target and the LoRA's recorded expert tag."""
if target == "both":
return True, True
if target == "high":
return True, False
if target == "low":
return False, True
# auto
if lora_expert == "high":
return True, False
if lora_expert == "low":
return False, True
return True, True
@invocation_output("wan_lora_loader_output")
class WanLoRALoaderOutput(BaseInvocationOutput):
"""Wan 2.2 LoRA loader output."""
transformer: Optional[WanTransformerField] = OutputField(
default=None, description=FieldDescriptions.transformer, title="Wan Transformer"
)
@invocation(
"wan_lora_loader",
title="Apply LoRA - Wan 2.2",
tags=["lora", "model", "wan"],
category="model",
version="1.0.1",
classification=Classification.Prototype,
)
class WanLoRALoaderInvocation(BaseInvocation):
"""Apply a LoRA to the Wan 2.2 transformer(s).
For A14B (dual expert) the LoRA's recorded ``expert`` field determines
which expert list it lands in: ``"high"`` -> primary list, ``"low"`` ->
low-noise list, ``None`` (untagged) -> both lists. Use the ``target``
field to override.
For TI2V-5B (single transformer) only the primary list is used at denoise
time; a LoRA routed only to the low-noise list would be inert, so that
routing logs a warning.
"""
lora: ModelIdentifierField = InputField(
description=FieldDescriptions.lora_model,
title="LoRA",
ui_model_base=BaseModelType.Wan,
ui_model_type=ModelType.LoRA,
)
weight: float = InputField(default=0.75, description=FieldDescriptions.lora_weight)
target: WanLoRATarget = InputField(
default="auto",
description="Which expert(s) to apply this LoRA to. 'auto' uses the LoRA's "
"recorded expert tag (or both if untagged); 'both'/'high'/'low' override it.",
)
transformer: WanTransformerField | None = InputField(
default=None,
description=FieldDescriptions.transformer,
input=Input.Connection,
title="Wan Transformer",
)
def invoke(self, context: InvocationContext) -> WanLoRALoaderOutput:
lora_key = self.lora.key
if not context.models.exists(lora_key):
raise ValueError(f"Unknown lora: {lora_key}!")
lora_config = context.models.get_config(self.lora)
_assert_is_wan_lora(lora_config, lora_key)
output = WanLoRALoaderOutput()
if self.transformer is None:
return output
main_config = context.models.get_config(self.transformer.transformer)
_assert_lora_variant_matches_main(lora_config, main_config, lora_key)
lora_expert = getattr(lora_config, "expert", None)
to_primary, to_low_noise = _resolve_target(self.target, lora_expert)
_warn_if_low_routing_is_inert(context, main_config, lora_key, to_primary, to_low_noise)
# Reject duplicates on whichever list(s) we're about to append to.
if to_primary and any(item.lora.key == lora_key for item in self.transformer.loras):
raise ValueError(f'LoRA "{lora_key}" already applied to primary transformer list.')
if to_low_noise and any(item.lora.key == lora_key for item in self.transformer.loras_low_noise):
raise ValueError(f'LoRA "{lora_key}" already applied to low-noise transformer list.')
output.transformer = self.transformer.model_copy(deep=True)
new_lora = LoRAField(lora=self.lora, weight=self.weight)
if to_primary:
output.transformer.loras.append(new_lora)
if to_low_noise:
output.transformer.loras_low_noise.append(new_lora)
return output
@invocation(
"wan_lora_collection_loader",
title="Apply LoRA Collection - Wan 2.2",
tags=["lora", "model", "wan"],
category="model",
version="1.0.1",
classification=Classification.Prototype,
)
class WanLoRACollectionLoader(BaseInvocation):
"""Apply a collection of LoRAs to the Wan 2.2 transformer(s).
Each LoRA is routed to the primary and/or low-noise list based on its
recorded ``expert`` tag (set by the probe from the filename). Untagged
LoRAs go to both lists.
"""
loras: Optional[LoRAField | list[LoRAField]] = InputField(
default=None,
description="LoRAs to apply. May be a single LoRA or a collection.",
title="LoRAs",
ui_model_base=[BaseModelType.Wan],
ui_model_type=ModelType.LoRA,
)
transformer: Optional[WanTransformerField] = InputField(
default=None,
description=FieldDescriptions.transformer,
input=Input.Connection,
title="Wan Transformer",
)
def invoke(self, context: InvocationContext) -> WanLoRALoaderOutput:
output = WanLoRALoaderOutput()
if self.transformer is None:
return output
output.transformer = self.transformer.model_copy(deep=True)
if self.loras is None:
return output
loras = self.loras if isinstance(self.loras, list) else [self.loras]
added: set[str] = set()
main_config = context.models.get_config(self.transformer.transformer)
for lora in loras:
if lora is None or lora.lora.key in added:
continue
lora_key = lora.lora.key
if not context.models.exists(lora_key):
raise ValueError(f"Unknown lora: {lora_key}!")
lora_config = context.models.get_config(lora.lora)
_assert_is_wan_lora(lora_config, lora_key)
_assert_lora_variant_matches_main(lora_config, main_config, lora_key)
lora_expert = getattr(lora_config, "expert", None)
to_primary, to_low_noise = _resolve_target("auto", lora_expert)
_warn_if_low_routing_is_inert(context, main_config, lora_key, to_primary, to_low_noise)
# Reject LoRAs already applied upstream (same invariant the single loader
# enforces) — re-appending would silently double the effective weight.
# Intra-collection duplicates are skipped via `added` before reaching here.
if to_primary and any(item.lora.key == lora_key for item in output.transformer.loras):
raise ValueError(f'LoRA "{lora_key}" already applied to primary transformer list.')
if to_low_noise and any(item.lora.key == lora_key for item in output.transformer.loras_low_noise):
raise ValueError(f'LoRA "{lora_key}" already applied to low-noise transformer list.')
added.add(lora_key)
if to_primary:
output.transformer.loras.append(lora)
if to_low_noise:
output.transformer.loras_low_noise.append(lora)
return output
@@ -0,0 +1,300 @@
from typing import Optional
from invokeai.app.invocations.baseinvocation import (
BaseInvocation,
BaseInvocationOutput,
Classification,
invocation,
invocation_output,
)
from invokeai.app.invocations.fields import FieldDescriptions, Input, InputField, OutputField
from invokeai.app.invocations.model import (
ModelIdentifierField,
VAEField,
WanT5EncoderField,
WanTransformerField,
)
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.backend.model_manager.taxonomy import BaseModelType, ModelFormat, ModelType, SubModelType, WanVariantType
@invocation_output("wan_model_loader_output")
class WanModelLoaderOutput(BaseInvocationOutput):
"""Wan 2.2 model loader output."""
transformer: WanTransformerField = OutputField(
description="Wan transformer (one or two experts depending on the variant)",
title="Transformer",
)
wan_t5_encoder: WanT5EncoderField = OutputField(
description=FieldDescriptions.wan_t5_encoder,
title="UMT5-XXL Encoder",
)
vae: VAEField = OutputField(description=FieldDescriptions.vae, title="VAE")
@invocation(
"wan_model_loader",
title="Main Model - Wan 2.2",
tags=["model", "wan"],
category="model",
version="1.0.1",
classification=Classification.Prototype,
)
class WanModelLoaderInvocation(BaseInvocation):
"""Loads a Wan 2.2 model, outputting its submodels.
Components can be mixed and matched, mirroring the Qwen Image loader pattern:
- Transformer(s):
* Diffusers main: emits ``transformer/`` and (for A14B) ``transformer_2/``
from the same model record.
* GGUF main: emits the single GGUF as the primary transformer; for A14B
the second-expert GGUF must be wired to ``Transformer (Low Noise)``.
- VAE: standalone Wan VAE > main (if Diffusers) > Component Source (Diffusers).
- UMT5-XXL encoder: standalone Wan T5 encoder > main (if Diffusers) >
Component Source (Diffusers).
The Component Source slot lets users supply a Diffusers Wan main model purely
for VAE / encoder extraction when the actual transformer is in a single-file
format. Together, the standalone VAE + standalone encoder let a GGUF
transformer run without a full ~30 GB Diffusers install.
"""
model: ModelIdentifierField = InputField(
description=FieldDescriptions.wan_model,
input=Input.Direct,
ui_model_base=BaseModelType.Wan,
ui_model_type=ModelType.Main,
title="Transformer",
)
transformer_low_noise_model: Optional[ModelIdentifierField] = InputField(
default=None,
description="Optional second GGUF transformer for the A14B low-noise expert. "
"Only relevant when the main model is a single-file GGUF and the variant is A14B; "
"ignored when the main is a Diffusers A14B (both experts are pulled from "
"transformer/ and transformer_2/ already) or when the variant is TI2V-5B.",
input=Input.Direct,
ui_model_base=BaseModelType.Wan,
ui_model_type=ModelType.Main,
ui_model_format=ModelFormat.GGUFQuantized,
title="Transformer (Low Noise)",
)
vae_model: Optional[ModelIdentifierField] = InputField(
default=None,
description="Standalone Wan VAE model. If not set, the VAE is loaded from the main model "
"(when in Diffusers format) or from the Component Source.",
input=Input.Direct,
ui_model_base=BaseModelType.Wan,
ui_model_type=ModelType.VAE,
title="VAE",
)
wan_t5_encoder_model: Optional[ModelIdentifierField] = InputField(
default=None,
description="Standalone Wan UMT5-XXL encoder. If not set, the encoder is loaded from the main "
"model (when in Diffusers format) or from the Component Source.",
input=Input.Direct,
ui_model_type=ModelType.WanT5Encoder,
title="Wan T5 Encoder",
)
component_source: Optional[ModelIdentifierField] = InputField(
default=None,
description="Diffusers Wan main model to extract VAE and/or encoder from. "
"Use this if you don't have separate VAE/encoder models. "
"Ignored for any submodel that is provided separately.",
input=Input.Direct,
ui_model_base=BaseModelType.Wan,
ui_model_type=ModelType.Main,
ui_model_format=ModelFormat.Diffusers,
title="Component Source (Diffusers)",
)
def invoke(self, context: InvocationContext) -> WanModelLoaderOutput:
main_config = context.models.get_config(self.model)
self._validate_main_config(main_config, "Wan main")
main_format = main_config.format
main_is_diffusers = main_format == ModelFormat.Diffusers
main_is_gguf = main_format == ModelFormat.GGUFQuantized
main_variant = getattr(main_config, "variant", None)
if main_is_gguf and self.component_source is not None:
self._validate_component_source_format(context, self.component_source)
# Resolve transformer + dual-expert wiring + boundary_ratio.
#
# Diffusers main: transformer/ is the primary, transformer_2/ is the
# low-noise expert (A14B only). boundary_ratio comes from the probed
# model_index.json.
#
# GGUF main: the file itself is one expert (high or low). For A14B,
# the user wires the other expert to transformer_low_noise_model.
# We swap so the *high*-noise expert is always the primary if needed.
# boundary_ratio falls back to 0.875 unless a Diffusers component_source
# provides a recorded value.
boundary_ratio = 0.9 if main_variant == WanVariantType.I2V_A14B else 0.875
transformer_low_noise: Optional[ModelIdentifierField] = None
if main_is_diffusers:
transformer = self.model.model_copy(update={"submodel_type": SubModelType.Transformer})
if getattr(main_config, "has_dual_expert", False):
transformer_low_noise = self.model.model_copy(update={"submodel_type": SubModelType.Transformer2})
recorded = getattr(main_config, "boundary_ratio", None)
if recorded is not None:
boundary_ratio = float(recorded)
elif main_is_gguf:
primary_expert = getattr(main_config, "expert", "none")
primary_id = self.model.model_copy(update={"submodel_type": SubModelType.Transformer})
if self.transformer_low_noise_model is not None and main_variant == WanVariantType.TI2V_5B:
# The field's own docs promise this input is ignored for the
# single-expert TI2V-5B — e.g. a leftover wire from an A14B session.
context.logger.warning("'Transformer (Low Noise)' is ignored for the single-expert TI2V-5B variant.")
if self.transformer_low_noise_model is not None and main_variant != WanVariantType.TI2V_5B:
low_config = context.models.get_config(self.transformer_low_noise_model)
self._validate_main_config(low_config, "Transformer (Low Noise)")
if low_config.format != ModelFormat.GGUFQuantized:
raise ValueError(
f"'Transformer (Low Noise)' must be a GGUF-format Wan model. "
f"'{low_config.name}' is in {low_config.format.value} format."
)
low_id = self.transformer_low_noise_model.model_copy(update={"submodel_type": SubModelType.Transformer})
low_expert = getattr(low_config, "expert", "none")
if getattr(low_config, "variant", None) != main_variant:
raise ValueError("The high-noise and low-noise GGUF models must use the same Wan variant.")
if {primary_expert, low_expert} != {"high", "low"}:
raise ValueError("A Wan A14B GGUF expert pair must contain one high and one low expert.")
# Make sure 'transformer' is the high-noise expert and
# 'transformer_low_noise' is the low-noise expert. If the user
# accidentally swapped them, swap back.
if primary_expert == "low" and low_expert == "high":
transformer = low_id
transformer_low_noise = primary_id
else:
transformer = primary_id
transformer_low_noise = low_id
else:
transformer = primary_id
if main_variant in (WanVariantType.T2V_A14B, WanVariantType.I2V_A14B) and primary_expert != "high":
raise ValueError("An unpaired Wan A14B GGUF model must be the high-noise expert.")
# A14B without a paired low-noise GGUF will produce degraded
# quality (only the high-noise expert runs). Warn but don't
# abort — TI2V-5B GGUFs are single-expert and totally fine.
if main_variant in (WanVariantType.T2V_A14B, WanVariantType.I2V_A14B):
context.logger.warning(
"A14B GGUF main was provided without a paired 'Transformer (Low Noise)'. "
"Only the high-noise expert will run; image quality will be reduced."
)
# Borrow the boundary_ratio recorded on the optional Diffusers
# component_source, when one is wired.
if self.component_source is not None:
src_cfg = context.models.get_config(self.component_source)
src_boundary = getattr(src_cfg, "boundary_ratio", None)
if (
src_cfg.format == ModelFormat.Diffusers
and getattr(src_cfg, "variant", None) == main_variant
and src_boundary is not None
):
boundary_ratio = float(src_boundary)
else:
raise ValueError(
f"Unsupported main model format for Wan: {main_format.value}. "
"Use a Diffusers folder or a GGUF single-file checkpoint."
)
# VAE: standalone override > main (if Diffusers) > component source.
if self.vae_model is not None:
self._validate_standalone_vae(context, self.vae_model, main_variant)
vae = self.vae_model.model_copy(update={"submodel_type": SubModelType.VAE})
elif main_is_diffusers:
vae = self.model.model_copy(update={"submodel_type": SubModelType.VAE})
elif self.component_source is not None:
self._validate_component_source_vae(context, self.component_source, main_variant)
vae = self.component_source.model_copy(update={"submodel_type": SubModelType.VAE})
else:
raise ValueError(
"No source for VAE. Either set 'VAE' to a standalone Wan VAE, "
"or set 'Component Source' to a Diffusers Wan main model."
)
# Tokenizer + text encoder: standalone override > main (if Diffusers) > component source.
if self.wan_t5_encoder_model is not None:
t5_config = context.models.get_config(self.wan_t5_encoder_model)
if t5_config.type != ModelType.WanT5Encoder or t5_config.format != ModelFormat.WanT5Encoder:
raise ValueError("The Wan T5 Encoder must resolve to a standalone Wan T5 encoder model.")
tokenizer = self.wan_t5_encoder_model.model_copy(update={"submodel_type": SubModelType.Tokenizer})
text_encoder = self.wan_t5_encoder_model.model_copy(update={"submodel_type": SubModelType.TextEncoder})
elif main_is_diffusers:
tokenizer = self.model.model_copy(update={"submodel_type": SubModelType.Tokenizer})
text_encoder = self.model.model_copy(update={"submodel_type": SubModelType.TextEncoder})
elif self.component_source is not None:
self._validate_component_source_format(context, self.component_source)
tokenizer = self.component_source.model_copy(update={"submodel_type": SubModelType.Tokenizer})
text_encoder = self.component_source.model_copy(update={"submodel_type": SubModelType.TextEncoder})
else:
raise ValueError(
"No source for Wan T5 encoder. "
"Either set 'Wan T5 Encoder' to a standalone UMT5-XXL encoder, "
"or set 'Component Source' to a Diffusers Wan main model."
)
return WanModelLoaderOutput(
transformer=WanTransformerField(
transformer=transformer,
transformer_low_noise=transformer_low_noise,
boundary_ratio=boundary_ratio,
),
wan_t5_encoder=WanT5EncoderField(tokenizer=tokenizer, text_encoder=text_encoder),
vae=VAEField(vae=vae),
)
@staticmethod
def _validate_main_config(config: object, label: str) -> None:
if getattr(config, "base", None) != BaseModelType.Wan or getattr(config, "type", None) != ModelType.Main:
raise ValueError(f"The {label} model must resolve to a Wan main model.")
@staticmethod
def _validate_component_source_format(context: InvocationContext, model: ModelIdentifierField) -> None:
source_config = context.models.get_config(model)
if source_config.base != BaseModelType.Wan or source_config.type != ModelType.Main:
raise ValueError("The Component Source model must resolve to a Wan main model.")
if source_config.format != ModelFormat.Diffusers:
raise ValueError(
f"The Component Source model must be in Diffusers format. "
f"The selected model '{source_config.name}' is in {source_config.format.value} format."
)
@staticmethod
def _validate_component_source_vae(
context: InvocationContext, model: ModelIdentifierField, main_variant: WanVariantType
) -> None:
WanModelLoaderInvocation._validate_component_source_format(context, model)
source_config = context.models.get_config(model)
source_variant = getattr(source_config, "variant", None)
main_is_ti2v = main_variant == WanVariantType.TI2V_5B
source_is_ti2v = source_variant == WanVariantType.TI2V_5B
if main_is_ti2v != source_is_ti2v:
raise ValueError(
"The Component Source VAE is incompatible with the selected transformer. "
"TI2V-5B requires the 48-channel Wan 2.2 VAE; A14B models require the 16-channel Wan 2.1 VAE."
)
@staticmethod
def _validate_standalone_vae(
context: InvocationContext, model: ModelIdentifierField, main_variant: WanVariantType
) -> None:
vae_config = context.models.get_config(model)
if vae_config.base != BaseModelType.Wan or vae_config.type != ModelType.VAE:
raise ValueError("The VAE must resolve to a standalone Wan VAE model.")
expected_channels = 48 if main_variant == WanVariantType.TI2V_5B else 16
if vae_config.latent_channels != expected_channels:
raise ValueError(
"The standalone VAE is incompatible with the selected transformer. "
"TI2V-5B requires the 48-channel Wan 2.2 VAE; A14B models require the 16-channel Wan 2.1 VAE."
)
@@ -0,0 +1,209 @@
"""Reference-image (VAE-latent) encoder for Wan 2.2 image-to-video conditioning.
Wan 2.2 I2V conditions on a reference image by VAE-encoding it. The condition
shape depends on the wired VAE (which selects the model family):
- **A14B** (16-channel Wan 2.1 VAE): a 20-channel condition tensor (4-ch
first-frame mask + 16-ch image latents) that the denoise loop concatenates
to the noise latents along the channel dim each step.
- **TI2V-5B** (48-channel Wan 2.2 VAE): a single-frame 48-channel condition
that the denoise loop blends with the noisy latents at every step
(expand_timesteps path).
Supports both single-frame (image I2V, ``num_frames=1``) and multi-frame
(video I2V, e.g. ``num_frames=81``) condition tensors; multi-frame and FLF2V
(end-image interpolation) are A14B-only.
"""
from typing import Optional
import torch
from diffusers.models.autoencoders import AutoencoderKLWan
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
from invokeai.app.invocations.fields import (
FieldDescriptions,
ImageField,
Input,
InputField,
)
from invokeai.app.invocations.model import VAEField
from invokeai.app.invocations.primitives import WanRefImageOutput
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.backend.model_manager.load.model_cache.utils import get_effective_device
from invokeai.backend.util.devices import TorchDevice
from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_wan
from invokeai.backend.wan.extensions.wan_ref_image_extension import (
encode_reference_image_to_condition,
encode_reference_image_to_ti2v_condition,
encode_reference_image_to_video_condition,
)
@invocation(
"wan_ref_image_encoder",
title="Reference Image - Wan 2.2",
tags=["image", "conditioning", "wan", "i2v"],
category="conditioning",
version="1.2.0",
classification=Classification.Prototype,
)
class WanRefImageEncoderInvocation(BaseInvocation):
"""VAE-encode a reference image into Wan 2.2 I2V conditioning.
The wired VAE selects the output shape. With the 16-channel A14B VAE the
output is a ``[1, 20, T_lat, height // 8, width // 8]`` condition (4-ch
first-frame mask + 16-ch image latents) that the denoise loop concatenates
to the noise latents each step, producing the 36-channel input the
I2V-A14B transformer expects. With the 48-channel TI2V-5B VAE the output
is a ``[1, 48, 1, height // 16, width // 16]`` condition that the denoise
loop blends into the noisy latents at every step.
For image (single-frame) I2V leave ``num_frames=1`` (T_lat=1). For A14B
video I2V set ``num_frames`` to match the value on the video-denoise node
(e.g. 81 for the Wan 2.2 reference defaults); TI2V-5B always uses the
single-frame condition.
Supply an optional ``end_image`` for **first-last-frame interpolation
(FLF2V)** — the model then interpolates the motion from ``image`` (first
frame) to ``end_image`` (final frame). FLF2V is I2V-A14B video only
(``num_frames > 1``); it is not supported for TI2V-5B or single-frame I2V.
For T2V (no reference image), omit this node entirely.
"""
image: ImageField = InputField(description="Reference image to condition on (the first frame).")
vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection, title="VAE")
# Must match wan_denoise's width/height. multiple_of=16 (not 8) because
# Wan's transformer patch_size=(1, 2, 2) needs latent H/W to be even.
width: int = InputField(
default=1024,
gt=0,
multiple_of=16,
description="Width to resize the reference image to (must match denoise width).",
)
height: int = InputField(
default=1024,
gt=0,
multiple_of=16,
description="Height to resize the reference image to (must match denoise height).",
)
num_frames: int = InputField(
default=1,
ge=1,
description="Pixel-frame count to build the condition for. Use 1 for single-frame image "
"I2V. For video I2V, set this to match the video-denoise node's num_frames (and ensure "
"(num_frames - 1) %% 4 == 0, e.g. 81).",
title="Number of Frames",
)
end_image: Optional[ImageField] = InputField(
default=None,
description="Optional end frame for first-last-frame interpolation (FLF2V). When set, the "
"video interpolates from the reference image (first frame) to this image (final frame). "
"I2V-A14B video only (num_frames > 1); not supported for TI2V-5B or single-frame I2V.",
title="End Image (FLF2V)",
)
@torch.no_grad()
def invoke(self, context: InvocationContext) -> WanRefImageOutput:
if self.num_frames > 1 and (self.num_frames - 1) % 4 != 0:
raise ValueError(
f"num_frames must satisfy (num_frames - 1) %% 4 == 0 for the Wan VAE's temporal "
f"compression (got {self.num_frames}). Try 5, 9, 13, ..., 81, 85, ..."
)
pil_image = context.images.get_pil(self.image.image_name, "RGB")
end_pil_image = context.images.get_pil(self.end_image.image_name, "RGB") if self.end_image is not None else None
vae_info = context.models.load(self.vae.vae)
if not isinstance(vae_info.model, AutoencoderKLWan):
raise TypeError(f"Reference-image encoder requires AutoencoderKLWan, got {type(vae_info.model).__name__}.")
estimated_working_memory = estimate_vae_working_memory_wan(
operation="encode",
vae=vae_info.model,
pixel_height=self.height,
pixel_width=self.width,
pixel_frames=self.num_frames,
)
with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae):
assert isinstance(vae, AutoencoderKLWan)
# A cpu_only VAE stays in system RAM even when an accelerator is selected —
# run the encode where the weights actually live.
device = get_effective_device(vae)
target_dtype = TorchDevice.choose_bfloat16_safe_dtype(device)
context.util.signal_progress(
("VAE-encoding FLF2V start+end images" if end_pil_image is not None else "VAE-encoding reference image")
+ (f" ({self.num_frames} frames)" if self.num_frames > 1 else "")
)
# Free cached allocator blocks left over from earlier nodes (denoise expert
# swaps in particular can leave the cache fragmented in ways that look like
# free VRAM but fail a single large contiguous request). Mirrors the
# pattern used in wan_latents_to_image.py / wan_latents_to_video.py.
TorchDevice.empty_cache()
# Pick the encoder path by VAE z_dim: 48 means the Wan 2.2-VAE (TI2V-5B),
# which uses a single-frame 48-channel condition that the denoise loop
# blends with the noisy latents at every step (expand_timesteps path).
# 16 means the standard Wan VAE (A14B), which uses the 20-channel
# mask + latent condition concatenated to noise along the channel dim.
is_ti2v_5b = getattr(vae.config, "z_dim", 16) == 48
if end_pil_image is not None and (is_ti2v_5b or self.num_frames <= 1):
raise ValueError(
"End-image (FLF2V) interpolation is only supported for I2V-A14B video "
f"(num_frames > 1). Got {'TI2V-5B' if is_ti2v_5b else 'single-frame I2V'}. "
"Remove the End Image input, or use an A14B VAE with num_frames > 1."
)
if is_ti2v_5b:
# TI2V-5B I2V needs latent H/W to be even for the transformer
# patch_size=(1,2,2), so pixel dims must be multiples of 32
# (16x VAE * 2 transformer patch). A14B's 8x VAE only needed
# multiples of 16.
if self.width % 32 != 0 or self.height % 32 != 0:
raise ValueError(
f"TI2V-5B I2V requires width and height to be multiples of 32 "
f"(got {self.width}x{self.height}). The Wan 2.2-VAE uses 16x "
f"spatial compression and the transformer adds a 2x patch on "
f"top, so pixel dims must divide by 32 for the patchify step."
)
condition = encode_reference_image_to_ti2v_condition(
image=pil_image,
vae=vae,
width=self.width,
height=self.height,
device=device,
dtype=target_dtype,
)
elif self.num_frames <= 1:
condition = encode_reference_image_to_condition(
image=pil_image,
vae=vae,
width=self.width,
height=self.height,
device=device,
dtype=target_dtype,
)
else:
condition = encode_reference_image_to_video_condition(
image=pil_image,
vae=vae,
width=self.width,
height=self.height,
num_frames=self.num_frames,
device=device,
dtype=target_dtype,
last_image=end_pil_image,
)
condition = condition.detach().to("cpu")
# Release this node's VAE-encode intermediates before the next node tries to
# partial-load the denoise transformer — the OOM we saw in PR #9163 review
# was the I2V expert load racing against still-cached encode activations.
TorchDevice.empty_cache()
name = context.tensors.save(tensor=condition)
return WanRefImageOutput.build(
condition_tensor_name=name,
width=self.width,
height=self.height,
num_frames=self.num_frames,
)
@@ -0,0 +1,112 @@
import torch
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
from invokeai.app.invocations.fields import (
FieldDescriptions,
Input,
InputField,
UIComponent,
)
from invokeai.app.invocations.model import WanT5EncoderField
from invokeai.app.invocations.primitives import WanConditioningOutput
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.backend.model_manager.load.model_cache.utils import get_effective_device
from invokeai.backend.stable_diffusion.diffusion.conditioning_data import (
ConditioningFieldData,
WanConditioningInfo,
)
# Wan models are trained with 512-token text sequences (matches the
# upstream config.json's ``text_len: 512`` and the WanPipeline.__call__
# default). Diffusers' ``_get_t5_prompt_embeds`` has a stale 226 default
# that gets overridden by ``__call__``; using 512 here matches the actual
# pipeline behaviour.
WAN_T5_MAX_SEQ_LEN = 512
@invocation(
"wan_text_encoder",
title="Prompt - Wan 2.2",
tags=["prompt", "conditioning", "wan"],
category="conditioning",
version="1.0.0",
classification=Classification.Prototype,
)
class WanTextEncoderInvocation(BaseInvocation):
"""Encodes a text prompt for Wan 2.2 using the UMT5-XXL encoder.
Output is the encoder's last hidden state (shape: [seq_len=512, 4096]) plus
an attention mask marking valid (non-padding) tokens. The Wan transformer
consumes these directly as ``encoder_hidden_states``.
"""
prompt: str = InputField(description="Text prompt for Wan 2.2.", ui_component=UIComponent.Textarea)
wan_t5_encoder: WanT5EncoderField = InputField(
title="UMT5-XXL Encoder",
description=FieldDescriptions.wan_t5_encoder,
input=Input.Connection,
)
@torch.no_grad()
def invoke(self, context: InvocationContext) -> WanConditioningOutput:
prompt_embeds, attention_mask = self._encode(context)
# Persist on CPU; the denoise loop will move to device as needed.
prompt_embeds = prompt_embeds.detach().to("cpu")
attention_mask = attention_mask.detach().to("cpu") if attention_mask is not None else None
conditioning_data = ConditioningFieldData(
conditionings=[WanConditioningInfo(prompt_embeds=prompt_embeds, prompt_attention_mask=attention_mask)]
)
conditioning_name = context.conditioning.save(conditioning_data)
return WanConditioningOutput.build(conditioning_name)
def _encode(self, context: InvocationContext) -> tuple[torch.Tensor, torch.Tensor | None]:
from diffusers.pipelines.wan.pipeline_wan import prompt_clean
from transformers import UMT5EncoderModel
cleaned = prompt_clean(self.prompt)
# Tokenizer + text encoder both routed through the model cache so the
# registered loaders handle the nested-vs-flat directory layout for us
# (main-model layout: <root>/tokenizer/ + <root>/text_encoder/;
# standalone WanT5Encoder layout may also be flat).
tokenizer_info = context.models.load(self.wan_t5_encoder.tokenizer)
with tokenizer_info.model_on_device() as (_, tokenizer):
text_inputs = tokenizer(
[cleaned],
padding="max_length",
max_length=WAN_T5_MAX_SEQ_LEN,
truncation=True,
add_special_tokens=True,
return_attention_mask=True,
return_tensors="pt",
)
text_encoder_info = context.models.load(self.wan_t5_encoder.text_encoder)
with text_encoder_info.model_on_device() as (_, text_encoder):
assert isinstance(text_encoder, UMT5EncoderModel)
device = get_effective_device(text_encoder)
input_ids = text_inputs.input_ids.to(device)
attention_mask = text_inputs.attention_mask.to(device)
context.util.signal_progress("Running UMT5-XXL text encoder")
outputs = text_encoder(input_ids, attention_mask)
# Drop the batch dim (we always encode one prompt at a time).
prompt_embeds = outputs.last_hidden_state.squeeze(0)
attention_mask_out = attention_mask.squeeze(0)
# Match the Diffusers reference: zero out the embeddings past the valid
# token count so the transformer sees clean padding.
valid_len = int(attention_mask_out.sum().item())
if valid_len < prompt_embeds.shape[0]:
prompt_embeds = prompt_embeds.clone()
prompt_embeds[valid_len:] = 0
# If every token is valid we don't need the mask downstream.
mask_out: torch.Tensor | None = attention_mask_out
if attention_mask_out.all():
mask_out = None
return prompt_embeds.to(dtype=torch.bfloat16), mask_out
@@ -0,0 +1,400 @@
"""Wan 2.2 video denoise invocation (T2V / I2V).
Multi-frame counterpart to :mod:`wan_denoise`. Drives the same flow-matching
schedule + expert-swap MoE logic, but the noise tensor has a real temporal
dimension (``T_lat = (num_frames - 1) // 4 + 1``) and the I2V conditioning is
built across all latent frames (first frame conditioned, rest zero).
Kept as a separate file rather than parameterizing ``WanDenoiseInvocation``
so the working single-frame T2I path is not risked by the video work; the
shared bits (expert swapper, scheduler construction, conditioning loading,
LoRA iteration) live in ``wan_denoise`` and are imported here.
"""
from contextlib import ExitStack
from typing import Callable, Iterable, Optional, Tuple
import torch
from tqdm import tqdm
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
from invokeai.app.invocations.fields import (
FieldDescriptions,
Input,
InputField,
WanConditioningField,
WanRefImageConditioningField,
)
from invokeai.app.invocations.model import WanTransformerField
from invokeai.app.invocations.primitives import LatentsOutput
from invokeai.app.invocations.wan_denoise import (
WanDenoiseInvocation,
_ExpertSwapper,
_resolve_variant,
_validate_ref_condition_shape,
_validate_spatial_dimensions,
)
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.backend.model_manager.taxonomy import BaseModelType, ModelFormat, WanVariantType
from invokeai.backend.patches.model_patch_raw import ModelPatchRaw
from invokeai.backend.stable_diffusion.diffusers_pipeline import PipelineIntermediateState
from invokeai.backend.stable_diffusion.diffusion.conditioning_data import WanConditioningInfo
from invokeai.backend.util.devices import TorchDevice
from invokeai.backend.wan.sampling_utils import (
get_default_latent_channels,
get_spatial_scale_factor,
make_noise,
num_latent_frames_for,
)
@invocation(
"wan_video_denoise",
title="Denoise Video - Wan 2.2",
tags=["video", "wan"],
category="latents",
version="1.0.0",
classification=Classification.Prototype,
)
class WanVideoDenoiseInvocation(BaseInvocation):
"""Run the Wan 2.2 denoising loop on a multi-frame latent tensor.
The output is a 5D ``[1, C, T_lat, H/8, W/8]`` latent tensor ready for
:class:`WanLatentsToVideoInvocation` to VAE-decode and encode as MP4.
Mirrors :class:`WanDenoiseInvocation` for the per-step logic (CFG, MoE
expert swap at the boundary timestep, LoRA patching, scheduler selection).
Differences from the image denoise:
* The noise tensor has a real temporal dim built from ``num_frames``.
* The I2V condition is built across all latent frames (frame 0
conditioned, rest zero) via
:func:`encode_reference_image_to_video_condition` upstream — the
``ref_image`` field on this node carries a tensor of shape
``[1, 20, T_lat, H_lat, W_lat]`` instead of ``[1, 20, 1, ...]``.
* No ``denoising_start`` / ``denoising_end`` / initial-latents inputs.
The image denoise node uses those for img2img (noise injection on an
existing latent), but image-conditioned video generation flows through
the reference-frame conditioning mechanism instead — the first frame
drives subsequent frames at every step, so a partial-schedule run from
an initial latent has no analogue here. Run the schedule from t=1
to t=0 every time. The base ``WanDenoiseInvocation`` still handles
the img2img case for stills.
"""
transformer: WanTransformerField = InputField(
description=(
"Wan transformer field. Supported: T2V-A14B / I2V-A14B (dual-expert) and "
"TI2V-5B (single-expert, handles both T2V and I2V). All three accept a "
"Reference Image input for image-to-video; A14B uses the 36-channel concat "
"scheme while TI2V-5B uses the expand_timesteps first-frame-mask blend."
),
input=Input.Connection,
title="Transformer",
)
positive_conditioning: WanConditioningField = InputField(
description=FieldDescriptions.positive_cond, input=Input.Connection
)
negative_conditioning: Optional[WanConditioningField] = InputField(
default=None, description=FieldDescriptions.negative_cond, input=Input.Connection
)
ref_image: Optional[WanRefImageConditioningField] = InputField(
default=None,
description=FieldDescriptions.wan_ref_image,
input=Input.Connection,
title="Reference Image",
)
guidance_scale: float = InputField(
default=5.0,
ge=1.0,
description="Classifier-free guidance scale. Wan 2.2 video reference uses 5.0 for the "
"high-noise expert and 4.0 for the low-noise expert.",
title="Guidance Scale",
)
guidance_scale_low_noise: Optional[float] = InputField(
default=4.0,
ge=0.0,
description="Optional separate CFG scale for the low-noise expert (Wan 2.2 A14B only). "
"Values below 1.0 fall back to the primary 'Guidance Scale'.",
title="Guidance Scale (Low Noise)",
)
# Wan transformer patch_size=(1, 2, 2) × VAE spatial 8x => H/W multiple of 16.
width: int = InputField(default=832, gt=0, multiple_of=16, description="Width of the generated video.")
height: int = InputField(default=480, gt=0, multiple_of=16, description="Height of the generated video.")
num_frames: int = InputField(
default=81,
ge=5,
description="Number of output frames. Must satisfy (num_frames - 1) %% 4 == 0 so the latent "
"temporal dim divides cleanly. Wan 2.2 was trained at 81 frames @ 16 FPS (~5 s).",
title="Number of Frames",
)
steps: int = InputField(default=40, gt=0, description="Number of denoising steps.")
seed: int = InputField(default=0, description="Randomness seed for reproducibility.")
@torch.no_grad()
def invoke(self, context: InvocationContext) -> LatentsOutput:
latents = self._run_diffusion(context)
# Keep the 5D shape (B, C, T, H, W) — wan_latents_to_video expects it.
latents = latents.detach().to("cpu")
name = context.tensors.save(tensor=latents)
# LatentsOutput.build uses latents.size()[3] / [2] for width / height.
# For 5D the spatial dims are at indices 4 / 3 instead of 3 / 2, so we
# call the constructor directly with the actual H/W from the inputs.
from invokeai.app.invocations.fields import LatentsField
return LatentsOutput(
latents=LatentsField(latents_name=name, seed=self.seed),
width=self.width,
height=self.height,
)
def _run_diffusion(self, context: InvocationContext) -> torch.Tensor:
if (self.num_frames - 1) % 4 != 0:
raise ValueError(
f"num_frames must satisfy (num_frames - 1) %% 4 == 0 for the Wan VAE's temporal "
f"compression (got {self.num_frames}). Try 5, 9, 13, ..., 81, 85, ..."
)
device = TorchDevice.choose_torch_device()
inference_dtype = TorchDevice.choose_bfloat16_safe_dtype(device)
variant = _resolve_variant(context, self.transformer)
_validate_spatial_dimensions(variant, self.width, self.height)
spatial_scale = get_spatial_scale_factor(variant)
# Reuse the image denoise's scheduler construction so we pick up whatever
# scheduler the variant ships with (UniPCMultistepScheduler with the
# variant's flow_shift, or whatever an on-disk config specifies).
scheduler_builder = WanDenoiseInvocation._build_scheduler # bound on instance below
# Bind a minimal instance to call _build_scheduler — it only reads
# self.transformer, which is shape-compatible.
proxy = WanDenoiseInvocation.model_construct(
transformer=self.transformer,
positive_conditioning=self.positive_conditioning,
)
scheduler = scheduler_builder(proxy, context, device)
pos_cond = self._load_conditioning(context, self.positive_conditioning, device=device, dtype=inference_dtype)
low_cfg_enabled = (
self.transformer.transformer_low_noise is not None
and self.guidance_scale_low_noise is not None
and self.guidance_scale_low_noise >= 1.0
and self.guidance_scale_low_noise != 1.0
)
do_cfg = self.negative_conditioning is not None and (self.guidance_scale != 1.0 or low_cfg_enabled)
neg_cond: WanConditioningInfo | None = None
if do_cfg:
assert self.negative_conditioning is not None
neg_cond = self._load_conditioning(
context, self.negative_conditioning, device=device, dtype=inference_dtype
)
# I2V condition tensor. Two flavours:
# * A14B I2V — [1, 20, T_lat, H_lat, W_lat] (4 mask + 16 latent channels).
# Concatenated to noise latents along the channel dim each step → 36ch.
# * TI2V-5B I2V — [1, 48, 1, H_lat, W_lat] (single latent frame, same
# channel count as the noise latents). Blended with noise via a
# first_frame_mask at every step (expand_timesteps path).
# Variant dispatch happens via the condition tensor's channel count below.
ref_condition: torch.Tensor | None = None
if self.ref_image is not None:
if variant not in (WanVariantType.I2V_A14B, WanVariantType.TI2V_5B):
raise ValueError(
f"Reference-image conditioning is only supported by Wan 2.2 I2V variants "
f"(I2V-A14B or TI2V-5B). The selected transformer is {variant.value!r}. "
"Remove the Reference Image input or load an I2V variant."
)
if self.ref_image.width != self.width or self.ref_image.height != self.height:
raise ValueError(
f"Reference-image dimensions ({self.ref_image.width}x{self.ref_image.height}) must "
f"match denoise dimensions ({self.width}x{self.height})."
)
# A14B encodes one condition tensor per pixel-frame count, so the
# encoder's num_frames must match. TI2V-5B's condition is always
# single-frame regardless of the output length, so the field's
# num_frames is informational only and we skip this check.
if variant == WanVariantType.I2V_A14B and self.ref_image.num_frames != self.num_frames:
raise ValueError(
f"Reference-image num_frames ({self.ref_image.num_frames}) must match denoise "
f"num_frames ({self.num_frames}). Re-run the Reference Image - Wan 2.2 node with "
f"num_frames={self.num_frames}."
)
ref_condition = context.tensors.load(self.ref_image.condition_tensor_name).to(
device=device, dtype=inference_dtype
)
_validate_ref_condition_shape(
ref_condition,
channels=48 if variant == WanVariantType.TI2V_5B else 20,
frames=1 if variant == WanVariantType.TI2V_5B else num_latent_frames_for(self.num_frames),
height=self.height // spatial_scale,
width=self.width // spatial_scale,
)
scheduler.set_timesteps(num_inference_steps=self.steps, device=device)
timesteps = scheduler.timesteps
total_steps = len(timesteps)
# fp32 latents through the loop; cast to inference_dtype only when
# calling the transformer (same as wan_denoise).
latent_dtype = torch.float32
# 48 for TI2V-5B (Wan 2.2-VAE z_dim=48), 16 for A14B variants.
latent_channels = get_default_latent_channels(variant)
t_lat = num_latent_frames_for(self.num_frames)
latents = make_noise(
batch_size=1,
latent_channels=latent_channels,
height=self.height,
width=self.width,
spatial_scale_factor=spatial_scale,
device=device,
dtype=latent_dtype,
seed=self.seed,
num_latent_frames=t_lat,
)
if total_steps <= 0:
return latents
# Build the TI2V-5B first-frame mask once: 0 at frame 0 (locked to the
# condition), 1 elsewhere (free to denoise). Broadcasts across channel dim.
first_frame_mask: torch.Tensor | None = None
if ref_condition is not None and variant == WanVariantType.TI2V_5B:
_, _, _, h_lat, w_lat = latents.shape
first_frame_mask = torch.ones(1, 1, t_lat, h_lat, w_lat, device=device, dtype=inference_dtype)
first_frame_mask[:, :, 0] = 0
step_callback = self._build_step_callback(context)
high_model = self.transformer.transformer
low_model = self.transformer.transformer_low_noise
low_config = context.models.get_config(low_model) if low_model is not None else None
num_train_timesteps = int(scheduler.config.num_train_timesteps)
boundary_timestep = self.transformer.boundary_ratio * num_train_timesteps if low_model is not None else None
# No fallback between the LoRA lists — see the matching comment in
# wan_denoise: an empty low list means "no LoRAs on the low expert".
high_loras = self.transformer.loras
low_loras = self.transformer.loras_low_noise
high_config = context.models.get_config(high_model)
high_is_quantized = high_config.format == ModelFormat.GGUFQuantized
low_is_quantized = low_config.format == ModelFormat.GGUFQuantized if low_config is not None else False
def high_lora_factory() -> Iterable[Tuple[ModelPatchRaw, float]]:
return proxy._lora_iterator(context, high_loras)
def low_lora_factory() -> Iterable[Tuple[ModelPatchRaw, float]]:
return proxy._lora_iterator(context, low_loras)
with ExitStack() as exit_stack:
swapper = _ExpertSwapper(
context=context,
high_model=high_model,
low_model=low_model,
inference_dtype=inference_dtype,
high_lora_factory=high_lora_factory if high_loras else None,
low_lora_factory=low_lora_factory if low_loras else None,
high_is_quantized=high_is_quantized,
low_is_quantized=low_is_quantized,
)
exit_stack.callback(swapper.close)
for step_idx, t in enumerate(
tqdm(timesteps, desc=f"Denoising Wan 2.2 video ({self.num_frames} frames)", total=total_steps)
):
if low_model is not None and float(t) < float(boundary_timestep):
active_label = _ExpertSwapper.LOW
low_cfg = self.guidance_scale_low_noise
active_cfg = low_cfg if (low_cfg is not None and low_cfg >= 1.0) else self.guidance_scale
else:
active_label = _ExpertSwapper.HIGH
active_cfg = self.guidance_scale
transformer = swapper.get(active_label)
latent_model_input = latents.to(dtype=inference_dtype)
# Per-variant conditioning. Two distinct mechanisms:
if first_frame_mask is not None:
# TI2V-5B I2V (expand_timesteps): blend the condition into frame 0
# and the noisy latents elsewhere. Per-token timestep tensor
# gates the model so it sees timestep=0 at frame-0 positions
# (nothing to denoise) and the normal `t` everywhere else.
assert ref_condition is not None
latent_model_input = (1 - first_frame_mask) * ref_condition + first_frame_mask * latent_model_input
# Strided slice matches the transformer's spatial patch_size=2;
# flatten gives per-token timesteps. Shape: [1, T_lat * H_lat//2 * W_lat//2].
temp_ts = (first_frame_mask[0, 0, :, ::2, ::2] * t).flatten()
timestep = temp_ts.unsqueeze(0).expand(latents.shape[0], -1).to(dtype=inference_dtype)
elif ref_condition is not None:
# A14B I2V: concat 20-ch condition along channel dim → 36-ch input.
latent_model_input = torch.cat([latent_model_input, ref_condition], dim=1)
timestep = t.expand(latents.shape[0])
else:
# T2V (any variant): scalar timestep per batch.
timestep = t.expand(latents.shape[0])
noise_pred_cond = transformer(
hidden_states=latent_model_input,
timestep=timestep,
encoder_hidden_states=pos_cond.prompt_embeds.unsqueeze(0),
attention_kwargs=None,
return_dict=False,
)[0]
if neg_cond is not None and active_cfg != 1.0:
noise_pred_uncond = transformer(
hidden_states=latent_model_input,
timestep=timestep,
encoder_hidden_states=neg_cond.prompt_embeds.unsqueeze(0),
attention_kwargs=None,
return_dict=False,
)[0]
noise_pred = noise_pred_uncond + active_cfg * (noise_pred_cond - noise_pred_uncond)
else:
noise_pred = noise_pred_cond
latents = scheduler.step(noise_pred, t, latents, return_dict=False)[0]
step_callback(
PipelineIntermediateState(
step=step_idx + 1,
order=1,
total_steps=total_steps,
timestep=int(t.item()),
# Preview shows the middle frame for video.
latents=latents[:, :, t_lat // 2],
)
)
# TI2V-5B: frame 0's latents drifted through the scheduler step at each
# iteration; restore them to the clean condition before VAE-decoding so
# the first output frame matches the reference image. Mirrors
# ``WanImageToVideoPipeline`` (pipeline_wan_i2v.py:813-814).
if first_frame_mask is not None:
assert ref_condition is not None
latents = (1 - first_frame_mask) * ref_condition.to(dtype=latents.dtype) + first_frame_mask * latents
return latents
def _load_conditioning(
self,
context: InvocationContext,
cond_field: WanConditioningField,
*,
device: torch.device,
dtype: torch.dtype,
) -> WanConditioningInfo:
cond_data = context.conditioning.load(cond_field.conditioning_name)
assert len(cond_data.conditionings) == 1
cond_info = cond_data.conditionings[0]
assert isinstance(cond_info, WanConditioningInfo)
return cond_info.to(device=device, dtype=dtype)
def _build_step_callback(self, context: InvocationContext) -> Callable[[PipelineIntermediateState], None]:
def step_callback(state: PipelineIntermediateState) -> None:
context.util.sd_step_callback(state, BaseModelType.Wan)
return step_callback
+17
View File
@@ -1,5 +1,22 @@
import os
from typing import Any
# Suppress the HuggingFace tokenizers fork-after-parallelism warning. The Rust
# ``tokenizers`` library warms a thread pool the first time a tokenizer is used
# (e.g. the UMT5 / T5 text encoder during Wan / FLUX / SD3 conditioning), then
# complains every time we fork() afterwards — which we do, on every MP4 encode,
# because imageio's FFMPEG plugin shells out to ffmpeg via subprocess.Popen.
# The warning is harmless (the child correctly falls back to single-threaded
# tokenization before exec()) but it spams the log on every video generation.
#
# This MUST execute before any HF library is imported. The pyproject console-script
# (``invokeai-web = invokeai.app.run_app:run_app``) reaches this module first via
# ``from invokeai.app.run_app import run_app``, so setting the env var at module
# level — not inside ``run_app()`` — guarantees it lands before any transitive HF
# import. Use ``setdefault`` so anyone who explicitly exports ``true`` upstream
# keeps their value.
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
def get_app():
"""Import the app and event loop. We wrap this in a function to more explicitly control when it happens, because
@@ -104,3 +104,23 @@ def verify_token(token: str) -> TokenData | None:
except Exception:
# Catch any other exceptions (e.g., Pydantic validation errors)
return None
def get_token_remaining_seconds(token: str) -> int | None:
"""Return the number of seconds until a *valid* token expires.
Verifies the token first (signature + expiry + payload shape); returns None if
it fails verification. A valid token without an ``exp`` claim gets the default
expiration window, matching what ``create_access_token`` would have assigned.
"""
if verify_token(token) is None:
return None
try:
claims = jwt.get_unverified_claims(token)
except JWTError:
return None
exp = claims.get("exp")
if exp is None:
return int(timedelta(hours=DEFAULT_EXPIRATION_HOURS).total_seconds())
remaining = int(cast(float, exp) - datetime.now(timezone.utc).timestamp())
return remaining if remaining > 0 else None
@@ -30,8 +30,13 @@ class BoardImageRecordStorageBase(ABC):
board_id: str,
categories: list[ImageCategory] | None,
is_intermediate: bool | None,
user_id: Optional[str] = None,
) -> list[str]:
"""Gets all board images for a board, as a list of the image names."""
"""Gets all board images for a board, as a list of the image names.
When ``user_id`` is provided, results are restricted to images owned by that user;
pass ``None`` for the admin path (no per-user restriction).
"""
pass
@abstractmethod
@@ -80,6 +80,7 @@ class SqliteBoardImageRecordStorage(BoardImageRecordStorageBase):
board_id: str,
categories: list[ImageCategory] | None,
is_intermediate: bool | None,
user_id: Optional[str] = None,
) -> list[str]:
with self._db.transaction() as cursor:
params: list[str | bool] = []
@@ -124,6 +125,13 @@ class SqliteBoardImageRecordStorage(BoardImageRecordStorageBase):
"""
params.append(is_intermediate)
# Per-user filter — admins pass user_id=None to skip this clause.
if user_id is not None:
stmt += """--sql
AND images.user_id = ?
"""
params.append(user_id)
# Put a ring on it
stmt += ";"
@@ -30,8 +30,13 @@ class BoardImagesServiceABC(ABC):
board_id: str,
categories: list[ImageCategory] | None,
is_intermediate: bool | None,
user_id: Optional[str] = None,
) -> list[str]:
"""Gets all board images for a board, as a list of the image names."""
"""Gets all board images for a board, as a list of the image names.
When ``user_id`` is provided, results are restricted to images owned by that user;
pass ``None`` for the admin path (no per-user restriction).
"""
pass
@abstractmethod
@@ -29,11 +29,13 @@ class BoardImagesService(BoardImagesServiceABC):
board_id: str,
categories: list[ImageCategory] | None,
is_intermediate: bool | None,
user_id: Optional[str] = None,
) -> list[str]:
return self.__invoker.services.board_image_records.get_all_board_image_names_for_board(
board_id,
categories,
is_intermediate,
user_id=user_id,
)
def get_board_for_image(
@@ -0,0 +1,43 @@
from abc import ABC, abstractmethod
from typing import Optional
from invokeai.app.services.image_records.image_records_common import ImageCategory
class BoardVideoRecordStorageBase(ABC):
"""Abstract base class for the one-to-many board-video relationship record storage."""
@abstractmethod
def add_video_to_board(self, board_id: str, video_name: str) -> None:
"""Adds a video to a board."""
pass
@abstractmethod
def remove_video_from_board(self, video_name: str) -> None:
"""Removes a video from a board."""
pass
@abstractmethod
def get_all_board_video_names_for_board(
self,
board_id: str,
categories: list[ImageCategory] | None,
is_intermediate: bool | None,
user_id: Optional[str] = None,
) -> list[str]:
"""Gets all board videos for a board, as a list of the video names.
When ``user_id`` is provided, results are restricted to videos owned by that user;
pass ``None`` for the admin path (no per-user restriction).
"""
pass
@abstractmethod
def get_board_for_video(self, video_name: str) -> Optional[str]:
"""Gets a video's board id, if it has one."""
pass
@abstractmethod
def get_video_count_for_board(self, board_id: str) -> int:
"""Gets the number of videos for a board."""
pass
@@ -0,0 +1,97 @@
import sqlite3
from typing import Optional, cast
from invokeai.app.services.board_video_records.board_video_records_base import BoardVideoRecordStorageBase
from invokeai.app.services.image_records.image_records_common import ImageCategory
from invokeai.app.services.shared.sqlite.sqlite_database import SqliteDatabase
class SqliteBoardVideoRecordStorage(BoardVideoRecordStorageBase):
def __init__(self, db: SqliteDatabase) -> None:
super().__init__()
self._db = db
def add_video_to_board(self, board_id: str, video_name: str) -> None:
with self._db.transaction() as cursor:
cursor.execute(
"""--sql
INSERT INTO board_videos (board_id, video_name)
VALUES (?, ?)
ON CONFLICT (video_name) DO UPDATE SET board_id = ?;
""",
(board_id, video_name, board_id),
)
def remove_video_from_board(self, video_name: str) -> None:
with self._db.transaction() as cursor:
cursor.execute(
"DELETE FROM board_videos WHERE video_name = ?;",
(video_name,),
)
def get_all_board_video_names_for_board(
self,
board_id: str,
categories: list[ImageCategory] | None,
is_intermediate: bool | None,
user_id: Optional[str] = None,
) -> list[str]:
with self._db.transaction() as cursor:
params: list[str | bool] = []
stmt = """
SELECT videos.video_name
FROM videos
LEFT JOIN board_videos ON board_videos.video_name = videos.video_name
WHERE 1=1
"""
if board_id == "none":
stmt += " AND board_videos.board_id IS NULL "
else:
stmt += " AND board_videos.board_id = ? "
params.append(board_id)
if categories is not None:
category_strings = [c.value for c in set(categories)]
placeholders = ",".join("?" * len(category_strings))
stmt += f" AND videos.video_category IN ( {placeholders} ) "
for c in category_strings:
params.append(c)
if is_intermediate is not None:
stmt += " AND videos.is_intermediate = ? "
params.append(is_intermediate)
if user_id is not None:
stmt += " AND videos.user_id = ? "
params.append(user_id)
stmt += ";"
cursor.execute(stmt, params)
result = cast(list[sqlite3.Row], cursor.fetchall())
return [r[0] for r in result]
def get_board_for_video(self, video_name: str) -> Optional[str]:
with self._db.transaction() as cursor:
cursor.execute(
"SELECT board_id FROM board_videos WHERE video_name = ?;",
(video_name,),
)
result = cursor.fetchone()
if result is None:
return None
return cast(str, result[0])
def get_video_count_for_board(self, board_id: str) -> int:
with self._db.transaction() as cursor:
cursor.execute(
"""--sql
SELECT COUNT(*)
FROM board_videos
INNER JOIN videos ON board_videos.video_name = videos.video_name
WHERE videos.is_intermediate = FALSE
AND board_videos.board_id = ?;
""",
(board_id,),
)
count = cast(int, cursor.fetchone()[0])
return count
@@ -10,8 +10,14 @@ class BoardDTO(BoardRecord):
cover_image_name: Optional[str] = Field(description="The name of the board's cover image.")
"""The URL of the thumbnail of the most recent image in the board."""
cover_video_name: Optional[str] = Field(
default=None, description="The name of the board's cover video, when the most recent item is a video."
)
"""The name of the cover video, set when the most-recent item on the board is a video rather than an image."""
image_count: int = Field(description="The number of images in the board.")
"""The number of images in the board."""
video_count: int = Field(default=0, description="The number of videos in the board.")
"""The number of videos in the board."""
asset_count: int = Field(description="The number of assets in the board.")
"""The number of assets in the board."""
owner_username: Optional[str] = Field(default=None, description="The username of the board owner (for admin view).")
@@ -24,12 +30,16 @@ def board_record_to_dto(
image_count: int,
asset_count: int,
owner_username: Optional[str] = None,
cover_video_name: Optional[str] = None,
video_count: int = 0,
) -> BoardDTO:
"""Converts a board record to a board DTO."""
return BoardDTO(
**board_record.model_dump(exclude={"cover_image_name"}),
cover_image_name=cover_image_name,
cover_video_name=cover_video_name,
image_count=image_count,
video_count=video_count,
asset_count=asset_count,
owner_username=owner_username,
)
+84 -35
View File
@@ -1,3 +1,5 @@
from typing import Optional
from invokeai.app.services.board_records.board_records_common import BoardChanges, BoardRecordOrderBy
from invokeai.app.services.boards.boards_base import BoardServiceABC
from invokeai.app.services.boards.boards_common import BoardDTO, board_record_to_dto
@@ -12,6 +14,38 @@ class BoardService(BoardServiceABC):
def start(self, invoker: Invoker) -> None:
self.__invoker = invoker
def _resolve_cover(self, board_id: str) -> tuple[Optional[str], Optional[str]]:
"""Pick the cover item for a board, considering both images and videos.
Returns ``(cover_image_name, cover_video_name)`` — at most one is set.
The winner is chosen by ``(starred DESC, created_at DESC)`` across both
tables so a recent video can supersede an older image (and vice versa).
"""
cover_image = self.__invoker.services.image_records.get_most_recent_image_for_board(board_id)
cover_video = self.__invoker.services.video_records.get_most_recent_video_for_board(board_id)
if cover_image is None and cover_video is None:
return None, None
if cover_video is None:
assert cover_image is not None
return cover_image.image_name, None
if cover_image is None:
return None, cover_video.video_name
# Match the gallery's deterministic (starred, created_at, kind, name) ordering.
image_key = (cover_image.starred, cover_image.created_at, "image", cover_image.image_name)
video_key = (cover_video.starred, cover_video.created_at, "video", cover_video.video_name)
if video_key > image_key:
return None, cover_video.video_name
return cover_image.image_name, None
def _get_counts(self, board_id: str) -> tuple[int, int, int]:
"""Return ``(image_count, video_count, asset_count)`` for a board."""
image_count = self.__invoker.services.board_image_records.get_image_count_for_board(board_id)
asset_count = self.__invoker.services.board_image_records.get_asset_count_for_board(board_id)
video_count = self.__invoker.services.board_video_records.get_video_count_for_board(board_id)
return image_count, video_count, asset_count
def create(
self,
board_name: str,
@@ -22,14 +56,16 @@ class BoardService(BoardServiceABC):
def get_dto(self, board_id: str) -> BoardDTO:
board_record = self.__invoker.services.board_records.get(board_id)
cover_image = self.__invoker.services.image_records.get_most_recent_image_for_board(board_record.board_id)
if cover_image:
cover_image_name = cover_image.image_name
else:
cover_image_name = None
image_count = self.__invoker.services.board_image_records.get_image_count_for_board(board_id)
asset_count = self.__invoker.services.board_image_records.get_asset_count_for_board(board_id)
return board_record_to_dto(board_record, cover_image_name, image_count, asset_count)
cover_image_name, cover_video_name = self._resolve_cover(board_record.board_id)
image_count, video_count, asset_count = self._get_counts(board_id)
return board_record_to_dto(
board_record,
cover_image_name,
image_count,
asset_count,
cover_video_name=cover_video_name,
video_count=video_count,
)
def update(
self,
@@ -37,15 +73,16 @@ class BoardService(BoardServiceABC):
changes: BoardChanges,
) -> BoardDTO:
board_record = self.__invoker.services.board_records.update(board_id, changes)
cover_image = self.__invoker.services.image_records.get_most_recent_image_for_board(board_record.board_id)
if cover_image:
cover_image_name = cover_image.image_name
else:
cover_image_name = None
image_count = self.__invoker.services.board_image_records.get_image_count_for_board(board_id)
asset_count = self.__invoker.services.board_image_records.get_asset_count_for_board(board_id)
return board_record_to_dto(board_record, cover_image_name, image_count, asset_count)
cover_image_name, cover_video_name = self._resolve_cover(board_record.board_id)
image_count, video_count, asset_count = self._get_counts(board_id)
return board_record_to_dto(
board_record,
cover_image_name,
image_count,
asset_count,
cover_video_name=cover_video_name,
video_count=video_count,
)
def delete(self, board_id: str) -> None:
self.__invoker.services.board_records.delete(board_id)
@@ -63,16 +100,12 @@ class BoardService(BoardServiceABC):
board_records = self.__invoker.services.board_records.get_many(
user_id, is_admin, order_by, direction, offset, limit, include_archived
)
summaries = self.__invoker.services.gallery.get_board_media_summaries(
[record.board_id for record in board_records.items]
)
board_dtos = []
for r in board_records.items:
cover_image = self.__invoker.services.image_records.get_most_recent_image_for_board(r.board_id)
if cover_image:
cover_image_name = cover_image.image_name
else:
cover_image_name = None
image_count = self.__invoker.services.board_image_records.get_image_count_for_board(r.board_id)
asset_count = self.__invoker.services.board_image_records.get_asset_count_for_board(r.board_id)
summary = summaries[r.board_id]
# For admin users, include owner username
owner_username = None
@@ -81,7 +114,17 @@ class BoardService(BoardServiceABC):
if owner:
owner_username = owner.display_name or owner.email
board_dtos.append(board_record_to_dto(r, cover_image_name, image_count, asset_count, owner_username))
board_dtos.append(
board_record_to_dto(
r,
summary.cover_image_name,
summary.image_count,
summary.asset_count,
owner_username,
cover_video_name=summary.cover_video_name,
video_count=summary.video_count,
)
)
return OffsetPaginatedResults[BoardDTO](items=board_dtos, offset=offset, limit=limit, total=len(board_dtos))
@@ -96,16 +139,12 @@ class BoardService(BoardServiceABC):
board_records = self.__invoker.services.board_records.get_all(
user_id, is_admin, order_by, direction, include_archived
)
summaries = self.__invoker.services.gallery.get_board_media_summaries(
[record.board_id for record in board_records]
)
board_dtos = []
for r in board_records:
cover_image = self.__invoker.services.image_records.get_most_recent_image_for_board(r.board_id)
if cover_image:
cover_image_name = cover_image.image_name
else:
cover_image_name = None
image_count = self.__invoker.services.board_image_records.get_image_count_for_board(r.board_id)
asset_count = self.__invoker.services.board_image_records.get_asset_count_for_board(r.board_id)
summary = summaries[r.board_id]
# For admin users, include owner username
owner_username = None
@@ -114,6 +153,16 @@ class BoardService(BoardServiceABC):
if owner:
owner_username = owner.display_name or owner.email
board_dtos.append(board_record_to_dto(r, cover_image_name, image_count, asset_count, owner_username))
board_dtos.append(
board_record_to_dto(
r,
summary.cover_image_name,
summary.image_count,
summary.asset_count,
owner_username,
cover_video_name=summary.cover_video_name,
video_count=summary.video_count,
)
)
return board_dtos
@@ -0,0 +1,65 @@
from abc import ABC, abstractmethod
from typing import Optional
from invokeai.app.services.gallery.gallery_common import BoardMediaSummary, GalleryItem, GalleryItemNamesResult
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
from invokeai.app.services.virtual_boards.virtual_boards_common import VirtualSubBoardDTO
class GalleryServiceABC(ABC):
"""High-level service producing a polymorphic stream of images and videos."""
@abstractmethod
def list_items(
self,
offset: int = 0,
limit: int = 10,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> OffsetPaginatedResults[GalleryItem]:
"""Lists a paginated, time-sorted stream of image + video items."""
pass
@abstractmethod
def list_item_names(
self,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
created_date: Optional[str] = None,
) -> GalleryItemNamesResult:
"""Returns ordered (kind, name) refs for optimistic UI / virtualized lists.
`created_date` restricts the result to items created on the given ISO date — used by
date-based virtual boards.
"""
pass
@abstractmethod
def get_dates(
self,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> list[VirtualSubBoardDTO]:
"""Returns date-based virtual sub-boards covering both images and videos."""
pass
@abstractmethod
def get_board_media_summaries(self, board_ids: list[str]) -> dict[str, BoardMediaSummary]:
"""Returns covers and media counts for each requested board."""
pass
@@ -0,0 +1,65 @@
"""Polymorphic gallery types: images and videos appearing in a single time-sorted stream."""
import datetime
from dataclasses import dataclass
from enum import Enum
from typing import Optional, Union
from pydantic import BaseModel, Field
from invokeai.app.services.image_records.image_records_common import ImageCategory
from invokeai.app.util.metaenum import MetaEnum
from invokeai.app.util.model_exclude_null import BaseModelExcludeNull
class GalleryItemKind(str, Enum, metaclass=MetaEnum):
"""Discriminator for polymorphic gallery items."""
IMAGE = "image"
VIDEO = "video"
class GalleryItemRef(BaseModel):
"""A thin reference to a gallery item — used for ordered name lists."""
kind: GalleryItemKind = Field(description="Whether the item is an image or video.")
name: str = Field(description="The unique name of the image or video.")
class GalleryItem(BaseModelExcludeNull):
"""A gallery item — either an image or a video, with shared fields and a discriminator.
Frontend code should dispatch on `kind` to render image- vs video-specific UI.
"""
kind: GalleryItemKind = Field(description="Whether the item is an image or video.")
name: str = Field(description="The unique name of the image or video.")
full_url: str = Field(description="URL to the full-resolution image PNG or the full-quality video MP4.")
thumbnail_url: str = Field(description="URL to the static (WebP) thumbnail.")
width: int = Field(description="The width of the item in pixels.")
height: int = Field(description="The height of the item in pixels.")
category: ImageCategory = Field(description="The category of the item (images and videos share the same enum).")
starred: bool = Field(description="Whether the item is starred.")
is_intermediate: bool = Field(description="Whether the item is an intermediate output.")
board_id: Optional[str] = Field(default=None, description="Owning board id, if any.")
created_at: Union[datetime.datetime, str] = Field(description="The created timestamp of the item.")
# Video-only fields. None for images.
duration: Optional[float] = Field(default=None, description="Video duration in seconds. None for images.")
fps: Optional[float] = Field(default=None, description="Video frames per second. None for images.")
class GalleryItemNamesResult(BaseModel):
"""Ordered list of gallery item references plus counts for optimistic UI."""
items: list[GalleryItemRef] = Field(description="Ordered list of (kind, name) references.")
starred_count: int = Field(description="Number of starred items (when starred_first=True).")
total_count: int = Field(description="Total number of items matching the query.")
@dataclass(frozen=True)
class BoardMediaSummary:
cover_image_name: Optional[str] = None
cover_video_name: Optional[str] = None
image_count: int = 0
video_count: int = 0
asset_count: int = 0
@@ -0,0 +1,471 @@
import sqlite3
from typing import Optional, Union, cast
from invokeai.app.services.gallery.gallery_base import GalleryServiceABC
from invokeai.app.services.gallery.gallery_common import (
BoardMediaSummary,
GalleryItem,
GalleryItemKind,
GalleryItemNamesResult,
GalleryItemRef,
)
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
from invokeai.app.services.invoker import Invoker
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
from invokeai.app.services.shared.sqlite.sqlite_database import SqliteDatabase
from invokeai.app.services.virtual_boards.virtual_boards_common import VirtualSubBoardDTO
class SqliteGalleryService(GalleryServiceABC):
"""Implements a polymorphic gallery via UNION ALL across the `images` and `videos` tables.
Filters are applied identically on each half. The two halves expose a common column set so
the result is shape-compatible (a literal `kind` discriminator + a `name` alias + duration/fps
that are NULL for images).
"""
__invoker: Invoker
def __init__(self, db: SqliteDatabase) -> None:
super().__init__()
self._db = db
def start(self, invoker: Invoker) -> None:
self.__invoker = invoker
def list_items(
self,
offset: int = 0,
limit: int = 10,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> OffsetPaginatedResults[GalleryItem]:
image_half, image_params, image_count_query = self._build_half(
kind="image",
origin=origin,
categories=categories,
is_intermediate=is_intermediate,
board_id=board_id,
search_term=search_term,
user_id=user_id,
is_admin=is_admin,
)
video_half, video_params, video_count_query = self._build_half(
kind="video",
origin=origin,
categories=categories,
is_intermediate=is_intermediate,
board_id=board_id,
search_term=search_term,
user_id=user_id,
is_admin=is_admin,
)
order_clause = self._build_order_clause(starred_first, order_dir)
union_query = f"""--sql
SELECT * FROM (
{image_half}
UNION ALL
{video_half}
)
{order_clause}
LIMIT ? OFFSET ?
;
"""
with self._db.transaction() as cursor:
cursor.execute(union_query, image_params + video_params + [limit, offset])
rows = cast(list[sqlite3.Row], cursor.fetchall())
cursor.execute(image_count_query, image_params)
image_count = cast(int, cursor.fetchone()[0])
cursor.execute(video_count_query, video_params)
video_count = cast(int, cursor.fetchone()[0])
urls = self.__invoker.services.urls
items = [self._row_to_item(row, urls) for row in rows]
return OffsetPaginatedResults[GalleryItem](
items=items,
offset=offset,
limit=limit,
total=image_count + video_count,
)
def list_item_names(
self,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
created_date: Optional[str] = None,
) -> GalleryItemNamesResult:
image_half, image_params, _ = self._build_half(
kind="image",
origin=origin,
categories=categories,
is_intermediate=is_intermediate,
board_id=board_id,
search_term=search_term,
user_id=user_id,
is_admin=is_admin,
names_only=True,
created_date=created_date,
)
video_half, video_params, _ = self._build_half(
kind="video",
origin=origin,
categories=categories,
is_intermediate=is_intermediate,
board_id=board_id,
search_term=search_term,
user_id=user_id,
is_admin=is_admin,
names_only=True,
created_date=created_date,
)
order_clause = self._build_order_clause(starred_first, order_dir)
union_query = f"""--sql
SELECT * FROM (
{image_half}
UNION ALL
{video_half}
)
{order_clause}
;
"""
with self._db.transaction() as cursor:
cursor.execute(union_query, image_params + video_params)
rows = cast(list[sqlite3.Row], cursor.fetchall())
starred_count = 0
if starred_first:
starred_count = sum(1 for r in rows if r["starred"])
refs = [GalleryItemRef(kind=GalleryItemKind(row["kind"]), name=row["name"]) for row in rows]
return GalleryItemNamesResult(items=refs, starred_count=starred_count, total_count=len(refs))
def get_dates(
self,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> list[VirtualSubBoardDTO]:
image_conditions = " AND images.is_intermediate = 0 "
video_conditions = " AND videos.is_intermediate = 0 "
image_params: list[Union[int, str, bool]] = []
video_params: list[Union[int, str, bool]] = []
if user_id is not None and not is_admin:
image_conditions += " AND images.user_id = ? "
image_params.append(user_id)
video_conditions += " AND videos.user_id = ? "
video_params.append(user_id)
union = f"""--sql
SELECT
images.created_at AS created_at,
'image' AS kind,
images.image_name AS name,
images.image_category AS category
FROM images
WHERE 1=1 {image_conditions}
UNION ALL
SELECT
videos.created_at AS created_at,
'video' AS kind,
videos.video_name AS name,
videos.video_category AS category
FROM videos
WHERE 1=1 {video_conditions}
"""
counts_query = f"""--sql
SELECT
DATE(created_at) AS date,
SUM(CASE WHEN kind = 'image' AND category = 'general' THEN 1 ELSE 0 END) AS image_count,
SUM(CASE WHEN kind = 'image' AND category != 'general' THEN 1 ELSE 0 END) AS asset_count,
SUM(CASE WHEN kind = 'video' THEN 1 ELSE 0 END) AS video_count
FROM ({union})
GROUP BY DATE(created_at)
ORDER BY date DESC;
"""
# The cover is the newest item of each date. ROW_NUMBER with kind/name tie-breakers
# (rather than a bare-column MAX() aggregate) keeps the choice deterministic when
# several items share the maximum timestamp — otherwise the cover could flicker
# between equally-new items across refetches.
covers_query = f"""--sql
SELECT date, kind, name FROM (
SELECT
DATE(created_at) AS date,
kind,
name,
ROW_NUMBER() OVER (
PARTITION BY DATE(created_at)
ORDER BY created_at DESC, kind DESC, name DESC
) AS rn
FROM ({union})
)
WHERE rn = 1;
"""
with self._db.transaction() as cursor:
cursor.execute(counts_query, image_params + video_params)
count_rows = cast(list[sqlite3.Row], cursor.fetchall())
cursor.execute(covers_query, image_params + video_params)
cover_rows = cast(list[sqlite3.Row], cursor.fetchall())
covers = {row["date"]: (row["kind"], row["name"]) for row in cover_rows}
boards: list[VirtualSubBoardDTO] = []
for row in count_rows:
date = row["date"]
cover_kind, cover_name = covers.get(date, (None, None))
boards.append(
VirtualSubBoardDTO(
virtual_board_id=f"by_date:{date}",
board_name=date,
date=date,
image_count=row["image_count"],
asset_count=row["asset_count"],
video_count=row["video_count"],
cover_image_name=cover_name if cover_kind == "image" else None,
cover_video_name=cover_name if cover_kind == "video" else None,
)
)
return boards
def get_board_media_summaries(self, board_ids: list[str]) -> dict[str, BoardMediaSummary]:
summaries = {board_id: BoardMediaSummary() for board_id in board_ids}
if not board_ids:
return summaries
placeholders = ",".join("?" for _ in board_ids)
query = f"""--sql
SELECT
board_id,
SUM(CASE WHEN kind = 'image' AND category = 'general' THEN 1 ELSE 0 END) AS image_count,
SUM(CASE WHEN kind = 'image' AND category != 'general' THEN 1 ELSE 0 END) AS asset_count,
SUM(CASE WHEN kind = 'video' THEN 1 ELSE 0 END) AS video_count,
MAX(CASE WHEN rank = 1 AND kind = 'image' THEN name END) AS cover_image_name,
MAX(CASE WHEN rank = 1 AND kind = 'video' THEN name END) AS cover_video_name
FROM (
SELECT
board_id,
kind,
name,
category,
ROW_NUMBER() OVER (
PARTITION BY board_id
ORDER BY starred DESC, created_at DESC, kind DESC, name DESC
) AS rank
FROM (
SELECT
board_images.board_id AS board_id,
'image' AS kind,
images.image_name AS name,
images.image_category AS category,
images.starred AS starred,
images.created_at AS created_at
FROM board_images
INNER JOIN images ON board_images.image_name = images.image_name
WHERE images.is_intermediate = FALSE
AND board_images.board_id IN ({placeholders})
UNION ALL
SELECT
board_videos.board_id AS board_id,
'video' AS kind,
videos.video_name AS name,
videos.video_category AS category,
videos.starred AS starred,
videos.created_at AS created_at
FROM board_videos
INNER JOIN videos ON board_videos.video_name = videos.video_name
WHERE videos.is_intermediate = FALSE
AND board_videos.board_id IN ({placeholders})
)
)
GROUP BY board_id;
"""
with self._db.transaction() as cursor:
cursor.execute(query, [*board_ids, *board_ids])
rows = cast(list[sqlite3.Row], cursor.fetchall())
for row in rows:
summaries[row["board_id"]] = BoardMediaSummary(
cover_image_name=row["cover_image_name"],
cover_video_name=row["cover_video_name"],
image_count=row["image_count"],
video_count=row["video_count"],
asset_count=row["asset_count"],
)
return summaries
@staticmethod
def _build_order_clause(starred_first: bool, order_dir: SQLiteDirection) -> str:
"""Ordering with `kind` + `name` tie-breakers.
Images and videos created within the same timestamp granularity would otherwise
have no defined relative order — rows could reorder across refetches or shift
between offset pages. The tie-breakers follow `order_dir` so ascending and
descending remain exact mirror images.
"""
direction = order_dir.value
tie_breakers = f"kind {direction}, name {direction}"
if starred_first:
return f"ORDER BY starred DESC, created_at {direction}, {tie_breakers}"
return f"ORDER BY created_at {direction}, {tie_breakers}"
def _build_half(
self,
kind: str,
origin: Optional[ResourceOrigin],
categories: Optional[list[ImageCategory]],
is_intermediate: Optional[bool],
board_id: Optional[str],
search_term: Optional[str],
user_id: Optional[str],
is_admin: bool,
names_only: bool = False,
created_date: Optional[str] = None,
) -> tuple[str, list[Union[int, str, bool]], str]:
"""Builds one half of the union (either `images` or `videos`).
Returns `(query_with_select, params, count_query)`. Both halves emit the same columns so
UNION ALL is shape-compatible: `kind`, `name`, `width`, `height`, `category`, `starred`,
`is_intermediate`, `board_id`, `created_at`, `duration`, `fps`.
`names_only=True` selects only `kind`, `name`, `starred`, `created_at` (the minimum needed
for ordering + the counts result).
"""
if kind == "image":
base_table = "images"
join_table = "board_images"
name_col = "image_name"
category_col = "image_category"
origin_col = "image_origin"
duration_expr = "NULL"
fps_expr = "NULL"
elif kind == "video":
base_table = "videos"
join_table = "board_videos"
name_col = "video_name"
category_col = "video_category"
origin_col = "video_origin"
duration_expr = f"{base_table}.duration"
fps_expr = f"{base_table}.fps"
else:
raise ValueError(f"Unknown kind: {kind}")
if names_only:
select_cols = (
f"'{kind}' AS kind, "
f"{base_table}.{name_col} AS name, "
f"{base_table}.starred AS starred, "
f"{base_table}.created_at AS created_at"
)
else:
select_cols = (
f"'{kind}' AS kind, "
f"{base_table}.{name_col} AS name, "
f"{base_table}.width AS width, "
f"{base_table}.height AS height, "
f"{base_table}.{category_col} AS category, "
f"{base_table}.starred AS starred, "
f"{base_table}.is_intermediate AS is_intermediate, "
f"{join_table}.board_id AS board_id, "
f"{base_table}.created_at AS created_at, "
f"{duration_expr} AS duration, "
f"{fps_expr} AS fps"
)
from_clause = f"FROM {base_table} LEFT JOIN {join_table} ON {join_table}.{name_col} = {base_table}.{name_col}"
conditions = ""
params: list[Union[int, str, bool]] = []
if origin is not None:
conditions += f" AND {base_table}.{origin_col} = ? "
params.append(origin.value)
if categories is not None:
category_strings = [c.value for c in set(categories)]
placeholders = ",".join("?" * len(category_strings))
conditions += f" AND {base_table}.{category_col} IN ( {placeholders} ) "
for c in category_strings:
params.append(c)
if is_intermediate is not None:
conditions += f" AND {base_table}.is_intermediate = ? "
params.append(is_intermediate)
if created_date is not None:
conditions += f" AND DATE({base_table}.created_at) = ? "
params.append(created_date)
if board_id == "none":
conditions += f" AND {join_table}.board_id IS NULL "
if user_id is not None and not is_admin:
conditions += f" AND {base_table}.user_id = ? "
params.append(user_id)
elif board_id is not None:
conditions += f" AND {join_table}.board_id = ? "
params.append(board_id)
elif user_id is not None and not is_admin:
# No board_id supplied — still enforce per-user isolation so
# non-admin callers cannot enumerate other users' items.
conditions += f" AND {base_table}.user_id = ? "
params.append(user_id)
if search_term:
conditions += f" AND ({base_table}.metadata LIKE ? OR {base_table}.created_at LIKE ?) "
params.append(f"%{search_term.lower()}%")
params.append(f"%{search_term.lower()}%")
half_query = f"SELECT {select_cols} {from_clause} WHERE 1=1 {conditions}"
count_query = f"SELECT COUNT(*) {from_clause} WHERE 1=1 {conditions}"
return half_query, params, count_query
def _row_to_item(self, row: sqlite3.Row, urls) -> GalleryItem:
kind = GalleryItemKind(row["kind"])
name = row["name"]
if kind == GalleryItemKind.IMAGE:
full_url = urls.get_image_url(name)
thumbnail_url = urls.get_image_url(name, thumbnail=True)
duration = None
fps = None
else:
full_url = urls.get_video_url(name)
thumbnail_url = urls.get_video_url(name, thumbnail=True)
duration = row["duration"]
fps = row["fps"]
return GalleryItem(
kind=kind,
name=name,
full_url=full_url,
thumbnail_url=thumbnail_url,
width=row["width"],
height=row["height"],
category=ImageCategory(row["category"]),
starred=bool(row["starred"]),
is_intermediate=bool(row["is_intermediate"]),
board_id=row["board_id"],
created_at=row["created_at"],
duration=duration,
fps=fps,
)
@@ -61,6 +61,21 @@ class ImageFileStorageBase(ABC):
"""Deletes an image and its thumbnail (if one exists)."""
pass
@abstractmethod
def stage_delete(self, image_name: str, image_subfolder: str = "") -> object:
"""Moves an image's files out of service and returns a rollback token."""
pass
@abstractmethod
def commit_delete(self, token: object) -> None:
"""Permanently removes files represented by a staged-delete token."""
pass
@abstractmethod
def rollback_delete(self, token: object) -> None:
"""Restores files represented by a staged-delete token."""
pass
@abstractmethod
def get_workflow(self, image_name: str, image_subfolder: str = "") -> Optional[str]:
"""Gets the workflow of an image."""
@@ -1,6 +1,11 @@
# Copyright (c) 2022 Kyle Schouviller (https://github.com/kyle0654) and the InvokeAI Team
import io
import json
import os
import shutil
import tempfile
import zlib
from dataclasses import dataclass
from pathlib import Path
from queue import Queue
from typing import Optional, Union
@@ -16,6 +21,7 @@ from invokeai.app.services.image_files.image_files_common import (
)
from invokeai.app.services.invoker import Invoker
from invokeai.app.util.thumbnails import get_thumbnail_name, make_thumbnail
from invokeai.backend.util.logging import InvokeAILogger
_PNG_RLE_MIN_PIXELS = 512 * 512
_PNG_RLE_SAMPLE_TILE = 32
@@ -23,6 +29,12 @@ _PNG_RLE_MIN_RAW_SIZE_PERCENT = 30
_PNG_RLE_MAX_SAMPLE_SIZE_PERCENT = 102
@dataclass
class _StagedDelete:
directory: Path
files: list[tuple[Path, Path]]
def _get_png_size(image: PILImageType, compress_type: Optional[int] = None) -> int:
output = io.BytesIO()
options = {"compress_level": 1}
@@ -73,6 +85,7 @@ class DiskImageFileStorage(ImageFileStorageBase):
def start(self, invoker: Invoker) -> None:
self.__invoker = invoker
self.__recover_staged_deletes()
@property
def image_root(self) -> Path:
@@ -157,20 +170,53 @@ class DiskImageFileStorage(ImageFileStorageBase):
raise ImageFileSaveException from e
def delete(self, image_name: str, image_subfolder: str = "") -> None:
token = self.stage_delete(image_name, image_subfolder)
self.commit_delete(token)
def stage_delete(self, image_name: str, image_subfolder: str = "") -> _StagedDelete:
candidates = [
self.get_path(image_name, image_subfolder=image_subfolder),
self.get_path(image_name, thumbnail=True, image_subfolder=image_subfolder),
]
staging_dir = Path(tempfile.mkdtemp(prefix=".delete_", dir=self.__output_folder))
staged: list[tuple[Path, Path]] = []
try:
image_path = self.get_path(image_name, image_subfolder=image_subfolder)
with open(staging_dir / "manifest.json", "w", encoding="utf-8") as manifest:
manifest.write(json.dumps({"image_name": image_name, "image_subfolder": image_subfolder}))
manifest.flush()
os.fsync(manifest.fileno())
for index, source in enumerate(candidates):
self.__cache.pop(source, None)
if source.exists():
destination = staging_dir / str(index)
source.replace(destination)
staged.append((source, destination))
return _StagedDelete(directory=staging_dir, files=staged)
except Exception as e:
for source, destination in reversed(staged):
if destination.exists():
source.parent.mkdir(parents=True, exist_ok=True)
destination.replace(source)
shutil.rmtree(staging_dir, ignore_errors=True)
raise ImageFileDeleteException from e
if image_path.exists():
image_path.unlink()
if image_path in self.__cache:
del self.__cache[image_path]
def commit_delete(self, token: object) -> None:
if not isinstance(token, _StagedDelete):
raise ImageFileDeleteException("Invalid staged-delete token")
try:
shutil.rmtree(token.directory)
except Exception as e:
raise ImageFileDeleteException from e
thumbnail_path = self.get_path(image_name, True, image_subfolder=image_subfolder)
if thumbnail_path.exists():
thumbnail_path.unlink()
if thumbnail_path in self.__cache:
del self.__cache[thumbnail_path]
def rollback_delete(self, token: object) -> None:
if not isinstance(token, _StagedDelete):
raise ImageFileDeleteException("Invalid staged-delete token")
try:
for source, destination in reversed(token.files):
if destination.exists():
source.parent.mkdir(parents=True, exist_ok=True)
destination.replace(source)
shutil.rmtree(token.directory, ignore_errors=True)
except Exception as e:
raise ImageFileDeleteException from e
@@ -240,6 +286,37 @@ class DiskImageFileStorage(ImageFileStorageBase):
for folder in folders:
folder.mkdir(parents=True, exist_ok=True)
def __recover_staged_deletes(self) -> None:
logger = InvokeAILogger.get_logger()
for staging_dir in self.__output_folder.glob(".delete_*"):
manifest_path = staging_dir / "manifest.json"
if not manifest_path.is_file():
if not any(staging_dir.iterdir()):
staging_dir.rmdir()
continue
try:
with open(manifest_path, encoding="utf-8") as manifest:
data = json.load(manifest)
image_name = data["image_name"]
image_subfolder = data.get("image_subfolder", "")
candidates = [
self.get_path(image_name, image_subfolder=image_subfolder),
self.get_path(image_name, thumbnail=True, image_subfolder=image_subfolder),
]
token = _StagedDelete(
directory=staging_dir,
files=[(source, staging_dir / str(index)) for index, source in enumerate(candidates)],
)
self.__invoker.services.image_records.get(image_name)
self.rollback_delete(token)
except Exception as error:
from invokeai.app.services.image_records.image_records_common import ImageRecordNotFoundException
if isinstance(error, ImageRecordNotFoundException):
shutil.rmtree(staging_dir, ignore_errors=True)
else:
logger.error(f"Failed to recover staged image deletion {staging_dir}: {error}")
def __get_cache(self, image_name: Path) -> Optional[PILImageType]:
return None if image_name not in self.__cache else self.__cache[image_name]
+11 -2
View File
@@ -148,8 +148,17 @@ class ImageServiceABC(ABC):
pass
@abstractmethod
def delete_images_on_board(self, board_id: str):
"""Deletes all images on a board."""
def delete_images_on_board(self, board_id: str, user_id: Optional[str] = None) -> tuple[list[str], list[str]]:
"""Deletes all images on a board; returns ``(deleted_names, failed_names)``.
When ``user_id`` is provided, only images owned by that user are deleted (other users'
contributions to a public/shared board are preserved). Pass ``None`` for the admin
path to delete every image on the board regardless of uploader.
``failed_names`` is the service's ground truth for per-image failures — callers must
not reconstruct it by diffing their own board listing against ``deleted_names``, which
races with concurrent moves/deletes.
"""
pass
@abstractmethod
@@ -47,6 +47,10 @@ class ResultWithAffectedBoards(BaseModel):
class DeleteImagesResult(ResultWithAffectedBoards):
deleted_images: list[str] = Field(description="The names of the images that were deleted")
failed_images: list[str] = Field(
default_factory=list,
description="The names of authorized images that could not be deleted",
)
class StarredImagesResult(ResultWithAffectedBoards):
+35 -6
View File
@@ -291,22 +291,51 @@ class ImageService(ImageServiceABC):
self.__invoker.services.logger.error("Problem deleting image record and file")
raise e
def delete_images_on_board(self, board_id: str):
def delete_images_on_board(self, board_id: str, user_id: Optional[str] = None) -> tuple[list[str], list[str]]:
try:
# When ``user_id`` is set the lookup filters to images owned by that user so the
# cascade doesn't destroy other users' contributions to a public/shared board.
image_names = self.__invoker.services.board_image_records.get_all_board_image_names_for_board(
board_id,
categories=None,
is_intermediate=None,
user_id=user_id,
)
deleted_image_names: list[str] = []
failed_image_names: list[str] = []
staged_deletes: list[tuple[str, object]] = []
for image_name in image_names:
try:
record = self.__invoker.services.image_records.get(image_name)
self.__invoker.services.image_files.delete(image_name, image_subfolder=record.image_subfolder)
except Exception:
pass
self.__invoker.services.image_records.delete_many(image_names)
for image_name in image_names:
token = self.__invoker.services.image_files.stage_delete(
image_name, image_subfolder=record.image_subfolder
)
staged_deletes.append((image_name, token))
deleted_image_names.append(image_name)
except Exception as e:
failed_image_names.append(image_name)
self.__invoker.services.logger.error(
f"Failed to delete image file {image_name}; keeping record: {str(e)}"
)
try:
self.__invoker.services.image_records.delete_many(deleted_image_names)
except Exception:
for image_name, token in staged_deletes:
try:
self.__invoker.services.image_files.rollback_delete(token)
except Exception as rollback_error:
self.__invoker.services.logger.error(
f"Failed to restore staged image files for {image_name}: {rollback_error}"
)
raise
for _, token in staged_deletes:
try:
self.__invoker.services.image_files.commit_delete(token)
except Exception as cleanup_error:
self.__invoker.services.logger.error(f"Failed to purge staged image files: {cleanup_error}")
for image_name in deleted_image_names:
self._on_deleted(image_name)
return deleted_image_names, failed_image_names
except ImageRecordDeleteException:
self.__invoker.services.logger.error("Failed to delete image records")
raise
@@ -37,6 +37,7 @@ class MemoryInvocationCache(InvocationCacheBase):
if self._max_cache_size == 0:
return
self._invoker.services.images.on_deleted(self._delete_by_match)
self._invoker.services.videos.on_deleted(self._delete_by_match)
self._invoker.services.tensors.on_deleted(self._delete_by_match)
self._invoker.services.conditioning.on_deleted(self._delete_by_match)
@@ -15,6 +15,7 @@ if TYPE_CHECKING:
from invokeai.app.services.board_image_records.board_image_records_base import BoardImageRecordStorageBase
from invokeai.app.services.board_images.board_images_base import BoardImagesServiceABC
from invokeai.app.services.board_records.board_records_base import BoardRecordStorageBase
from invokeai.app.services.board_video_records.board_video_records_base import BoardVideoRecordStorageBase
from invokeai.app.services.boards.boards_base import BoardServiceABC
from invokeai.app.services.bulk_download.bulk_download_base import BulkDownloadBase
from invokeai.app.services.client_state_persistence.client_state_persistence_base import ClientStatePersistenceABC
@@ -22,6 +23,7 @@ if TYPE_CHECKING:
from invokeai.app.services.download import DownloadQueueServiceBase
from invokeai.app.services.events.events_base import EventServiceBase
from invokeai.app.services.external_generation.external_generation_base import ExternalGenerationServiceBase
from invokeai.app.services.gallery.gallery_base import GalleryServiceABC
from invokeai.app.services.image_files.image_files_base import ImageFileStorageBase
from invokeai.app.services.image_moves.image_moves_default import ImageMoveService
from invokeai.app.services.image_records.image_records_base import ImageRecordStorageBase
@@ -39,6 +41,9 @@ if TYPE_CHECKING:
from invokeai.app.services.session_queue.session_queue_base import SessionQueueBase
from invokeai.app.services.urls.urls_base import UrlServiceBase
from invokeai.app.services.users.users_base import UserServiceBase
from invokeai.app.services.video_files.video_files_base import VideoFileStorageBase
from invokeai.app.services.video_records.video_records_base import VideoRecordStorageBase
from invokeai.app.services.videos.videos_base import VideoServiceABC
from invokeai.app.services.workflow_records.workflow_records_base import WorkflowRecordsStorageBase
from invokeai.app.services.workflow_thumbnails.workflow_thumbnails_base import WorkflowThumbnailServiceBase
from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ConditioningFieldData
@@ -80,6 +85,11 @@ class InvocationServices:
workflow_thumbnails: "WorkflowThumbnailServiceBase",
client_state_persistence: "ClientStatePersistenceABC",
users: "UserServiceBase",
videos: "VideoServiceABC",
video_files: "VideoFileStorageBase",
video_records: "VideoRecordStorageBase",
board_video_records: "BoardVideoRecordStorageBase",
gallery: "GalleryServiceABC",
image_moves: "ImageMoveService | None" = None,
):
self.board_images = board_images
@@ -114,3 +124,8 @@ class InvocationServices:
self.workflow_thumbnails = workflow_thumbnails
self.client_state_persistence = client_state_persistence
self.users = users
self.videos = videos
self.video_files = video_files
self.video_records = video_records
self.board_video_records = board_video_records
self.gallery = gallery
@@ -33,6 +33,8 @@ from invokeai.backend.model_manager.taxonomy import (
Qwen3VariantType,
QwenImageVariantType,
SchedulerPredictionType,
WanLoRAVariantType,
WanVariantType,
ZImageVariantType,
)
@@ -134,6 +136,8 @@ class ModelRecordChanges(BaseModelExcludeNull):
| Flux2VariantType
| ZImageVariantType
| QwenImageVariantType
| WanVariantType
| WanLoRAVariantType
| Qwen3VariantType
] = Field(description="The variant of the model.", default=None)
prediction_type: Optional[SchedulerPredictionType] = Field(
@@ -9,3 +9,8 @@ class NameServiceBase(ABC):
def create_image_name(self) -> str:
"""Creates a name for an image."""
pass
@abstractmethod
def create_video_name(self) -> str:
"""Creates a name for a video."""
pass
@@ -10,3 +10,8 @@ class SimpleNameService(NameServiceBase):
uuid_str = uuid_string()
filename = f"{uuid_str}.png"
return filename
def create_video_name(self) -> str:
uuid_str = uuid_string()
filename = f"{uuid_str}.mp4"
return filename
@@ -15,7 +15,7 @@ from pydantic import (
)
from pydantic_core import to_jsonable_python
from invokeai.app.invocations.fields import ImageField
from invokeai.app.invocations.fields import ImageField, VideoField
from invokeai.app.services.shared.graph import Graph, GraphExecutionState, NodeNotFoundError
from invokeai.app.services.workflow_records.workflow_records_common import (
WorkflowWithoutID,
@@ -51,7 +51,7 @@ class SessionQueueItemNotFoundError(ValueError):
# region Batch
BatchScalarDataType = Union[StrictStr, float, int, ImageField]
BatchScalarDataType = Union[StrictStr, float, int, ImageField, VideoField]
BatchDataType = Union[BatchScalarDataType, list[BatchScalarDataType]]
@@ -9,7 +9,7 @@ from torch import Tensor
from invokeai.app.invocations.constants import IMAGE_MODES
from invokeai.app.invocations.fields import MetadataField, WithBoard, WithMetadata
from invokeai.app.services.board_records.board_records_common import BoardRecordOrderBy
from invokeai.app.services.board_records.board_records_common import BoardRecordOrderBy, BoardVisibility
from invokeai.app.services.boards.boards_common import BoardDTO
from invokeai.app.services.config.config_default import InvokeAIAppConfig
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
@@ -18,6 +18,7 @@ from invokeai.app.services.invocation_services import InvocationServices
from invokeai.app.services.model_records.model_records_base import UnknownModelException
from invokeai.app.services.session_processor.session_processor_common import ProgressImage
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
from invokeai.app.services.videos.videos_common import VideoDTO
from invokeai.app.util.step_callback import diffusion_step_callback
from invokeai.backend.model_manager.configs.base import Config_Base
from invokeai.backend.model_manager.configs.factory import AnyModelConfig
@@ -169,6 +170,22 @@ class ImagesInterface(InvocationContextInterface):
super().__init__(services, data)
self._util = util
def _assert_read_access(self, image_name: str) -> None:
if not self._services.configuration.multiuser:
return
user_id = self._data.queue_item.user_id
user = self._services.users.get(user_id)
if user is None:
raise PermissionError("Queue user is not authorized to access this image")
if user.is_admin or self._services.image_records.get_user_id(image_name) == user_id:
return
board_id = self._services.board_image_records.get_board_for_image(image_name)
if board_id is not None:
board = self._services.boards.get_dto(board_id)
if board.board_visibility in (BoardVisibility.Shared, BoardVisibility.Public):
return
raise PermissionError("Queue user is not authorized to access this image")
def save(
self,
image: Image,
@@ -211,6 +228,16 @@ class ImagesInterface(InvocationContextInterface):
elif isinstance(self._data.invocation, WithBoard) and self._data.invocation.board:
board_id_ = self._data.invocation.board.board_id
if board_id_ is not None and self._services.configuration.multiuser:
board = self._services.boards.get_dto(board_id_)
user = self._services.users.get(self._data.queue_item.user_id)
if user is None or (
not user.is_admin
and board.user_id != self._data.queue_item.user_id
and board.board_visibility != BoardVisibility.Public
):
raise PermissionError("Queue user is not authorized to save images to this board")
workflow_ = None
if self._data.queue_item.workflow:
workflow_ = self._data.queue_item.workflow.model_dump_json()
@@ -243,6 +270,7 @@ class ImagesInterface(InvocationContextInterface):
Returns:
The image as a PIL Image object.
"""
self._assert_read_access(image_name)
image = self._services.images.get_pil_image(image_name)
if mode and mode != image.mode:
try:
@@ -266,6 +294,7 @@ class ImagesInterface(InvocationContextInterface):
Returns:
The image's metadata, if it has any.
"""
self._assert_read_access(image_name)
return self._services.images.get_metadata(image_name)
def get_dto(self, image_name: str) -> ImageDTO:
@@ -277,6 +306,7 @@ class ImagesInterface(InvocationContextInterface):
Returns:
The image as an ImageDTO object.
"""
self._assert_read_access(image_name)
return self._services.images.get_dto(image_name)
def get_path(self, image_name: str, thumbnail: bool = False) -> Path:
@@ -289,9 +319,121 @@ class ImagesInterface(InvocationContextInterface):
Returns:
The local path of the image or thumbnail.
"""
self._assert_read_access(image_name)
return Path(self._services.images.get_path(image_name, thumbnail))
class VideosInterface(InvocationContextInterface):
"""Save and look up videos produced by invocations.
Mirrors :class:`ImagesInterface` but consumes a path to an already-encoded
MP4 (or other supported container) rather than an in-memory PIL image —
video encoding is the caller's responsibility (e.g. the
``wan_latents_to_video`` node uses ``imageio[ffmpeg]``).
"""
def __init__(self, services: InvocationServices, data: InvocationContextData, util: "UtilInterface") -> None:
super().__init__(services, data)
self._util = util
def _assert_read_access(self, video_name: str) -> None:
if not self._services.configuration.multiuser:
return
user_id = self._data.queue_item.user_id
user = self._services.users.get(user_id)
if user is None:
raise PermissionError("Queue user is not authorized to access this video")
if user.is_admin or self._services.video_records.get_user_id(video_name) == user_id:
return
board_id = self._services.board_video_records.get_board_for_video(video_name)
if board_id is not None:
board = self._services.boards.get_dto(board_id)
if board.board_visibility in (BoardVisibility.Shared, BoardVisibility.Public):
return
raise PermissionError("Queue user is not authorized to access this video")
def save(
self,
source_path: Path,
width: int,
height: int,
duration: float,
fps: Optional[float] = None,
board_id: Optional[str] = None,
image_category: ImageCategory = ImageCategory.GENERAL,
metadata: Optional[MetadataField] = None,
) -> VideoDTO:
"""Save a video produced by an invocation. The file at ``source_path`` is moved into
the videos output folder; the caller should treat the path as consumed after this
returns.
``board_id`` falls back to the invocation's :class:`WithBoard` mixin if unset, and
``metadata`` falls back to the :class:`WithMetadata` mixin. Both can be overridden
explicitly. ``image_category`` reuses the image enum since the gallery's category
filter is shared between kinds.
"""
self._util.signal_progress("Saving video")
metadata_ = None
if metadata:
metadata_ = metadata.model_dump_json()
elif isinstance(self._data.invocation, WithMetadata) and self._data.invocation.metadata:
metadata_ = self._data.invocation.metadata.model_dump_json()
board_id_ = None
if board_id:
board_id_ = board_id
elif isinstance(self._data.invocation, WithBoard) and self._data.invocation.board:
board_id_ = self._data.invocation.board.board_id
if board_id_ is not None and self._services.configuration.multiuser:
board = self._services.boards.get_dto(board_id_)
user = self._services.users.get(self._data.queue_item.user_id)
if user is None or (
not user.is_admin
and board.user_id != self._data.queue_item.user_id
and board.board_visibility != BoardVisibility.Public
):
raise PermissionError("Queue user is not authorized to save videos to this board")
workflow_ = None
if self._data.queue_item.workflow:
workflow_ = self._data.queue_item.workflow.model_dump_json()
graph_ = None
if self._data.queue_item.session.graph:
graph_ = self._data.queue_item.session.graph.model_dump_json()
return self._services.videos.create(
source_path=source_path,
width=width,
height=height,
duration=duration,
fps=fps,
is_intermediate=self._data.invocation.is_intermediate,
video_category=image_category,
board_id=board_id_,
metadata=metadata_,
video_origin=ResourceOrigin.INTERNAL,
workflow=workflow_,
graph=graph_,
session_id=self._data.queue_item.session_id,
node_id=self._data.invocation.id,
user_id=self._data.queue_item.user_id,
)
def get_dto(self, video_name: str) -> VideoDTO:
"""Get a video DTO by name."""
self._assert_read_access(video_name)
return self._services.videos.get_dto(video_name)
def get_path(self, video_name: str, thumbnail: bool = False) -> Path:
"""Get the on-disk path to a video file or its WebP thumbnail."""
self._assert_read_access(video_name)
return Path(self._services.videos.get_path(video_name, thumbnail=thumbnail))
class TensorsInterface(InvocationContextInterface):
def save(self, tensor: Tensor) -> str:
"""Saves a tensor, returning its name.
@@ -738,6 +880,7 @@ class InvocationContext:
def __init__(
self,
images: ImagesInterface,
videos: VideosInterface,
tensors: TensorsInterface,
conditioning: ConditioningInterface,
models: ModelsInterface,
@@ -750,6 +893,8 @@ class InvocationContext:
) -> None:
self.images = images
"""Methods to save, get and update images and their metadata."""
self.videos = videos
"""Methods to save and get videos produced by invocations."""
self.tensors = tensors
"""Methods to save and get tensors, including image, noise, masks, and masked images."""
self.conditioning = conditioning
@@ -792,10 +937,12 @@ def build_invocation_context(
conditioning = ConditioningInterface(services=services, data=data)
models = ModelsInterface(services=services, data=data, util=util)
images = ImagesInterface(services=services, data=data, util=util)
videos = VideosInterface(services=services, data=data, util=util)
boards = BoardsInterface(services=services, data=data)
ctx = InvocationContext(
images=images,
videos=videos,
logger=logger,
config=config,
tensors=tensors,
@@ -4,6 +4,11 @@ from pydantic import BaseModel, Field
GenericBaseModel = TypeVar("GenericBaseModel", bound=BaseModel)
# Upper bound for offset-paginated `limit` query params. Values flow verbatim into
# SQL LIMIT clauses, where SQLite treats a negative limit as unlimited and a huge one
# materializes every row into DTOs — routes must clamp with ge=0/le=MAX_PAGE_SIZE.
MAX_PAGE_SIZE = 1000
class CursorPaginatedResults(BaseModel, Generic[GenericBaseModel]):
"""
@@ -0,0 +1,124 @@
"""Migration: Add `videos` and `board_videos` tables for minimal video support.
The `videos` table parallels `images` but with extra `duration` and `fps` columns.
The `board_videos` table parallels `board_images`, providing one-to-many board↔video association.
Foreign-key cascades from `boards` mirror the image side, so deleting a board also removes its videos' associations.
"""
import sqlite3
from invokeai.app.services.shared.sqlite_migrator.sqlite_migrator_common import Migration
class AddVideosTablesCallback:
def __call__(self, cursor: sqlite3.Cursor) -> None:
self._create_videos(cursor)
self._create_board_videos(cursor)
def _create_videos(self, cursor: sqlite3.Cursor) -> None:
tables = [
"""--sql
CREATE TABLE IF NOT EXISTS videos (
video_name TEXT NOT NULL PRIMARY KEY,
video_origin TEXT NOT NULL,
video_category TEXT NOT NULL,
width INTEGER NOT NULL,
height INTEGER NOT NULL,
duration REAL NOT NULL DEFAULT 0.0,
fps REAL,
session_id TEXT,
node_id TEXT,
metadata TEXT,
is_intermediate BOOLEAN DEFAULT FALSE,
starred BOOLEAN DEFAULT FALSE,
has_workflow BOOLEAN DEFAULT FALSE,
-- Deliberately no FK to users(user_id), matching images/boards/workflows
-- (migration_27 adds those user_id columns with an index only): deleting a
-- user leaves their media in place for admin review/cleanup rather than
-- cascading a row delete that would strand the files on disk.
user_id TEXT NOT NULL DEFAULT 'system',
video_subfolder TEXT NOT NULL DEFAULT '',
created_at DATETIME NOT NULL DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
-- Updated via trigger
updated_at DATETIME NOT NULL DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
-- Soft delete, currently unused
deleted_at DATETIME
);
"""
]
indices = [
"CREATE UNIQUE INDEX IF NOT EXISTS idx_videos_video_name ON videos(video_name);",
"CREATE INDEX IF NOT EXISTS idx_videos_video_origin ON videos(video_origin);",
"CREATE INDEX IF NOT EXISTS idx_videos_video_category ON videos(video_category);",
"CREATE INDEX IF NOT EXISTS idx_videos_created_at ON videos(created_at);",
"CREATE INDEX IF NOT EXISTS idx_videos_starred ON videos(starred);",
"CREATE INDEX IF NOT EXISTS idx_videos_user_id ON videos(user_id);",
]
triggers = [
"""--sql
CREATE TRIGGER IF NOT EXISTS tg_videos_updated_at
AFTER UPDATE
ON videos FOR EACH ROW
BEGIN
UPDATE videos SET updated_at = STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')
WHERE video_name = old.video_name;
END;
"""
]
for stmt in tables + indices + triggers:
cursor.execute(stmt)
def _create_board_videos(self, cursor: sqlite3.Cursor) -> None:
tables = [
"""--sql
CREATE TABLE IF NOT EXISTS board_videos (
board_id TEXT NOT NULL,
video_name TEXT NOT NULL,
created_at DATETIME NOT NULL DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
-- updated via trigger
updated_at DATETIME NOT NULL DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
-- Soft delete, currently unused
deleted_at DATETIME,
-- enforce one-to-many board↔video using PK on video_name
PRIMARY KEY (video_name),
FOREIGN KEY (board_id) REFERENCES boards (board_id) ON DELETE CASCADE,
FOREIGN KEY (video_name) REFERENCES videos (video_name) ON DELETE CASCADE
);
"""
]
indices = [
"CREATE INDEX IF NOT EXISTS idx_board_videos_board_id ON board_videos (board_id);",
"CREATE INDEX IF NOT EXISTS idx_board_videos_board_id_created_at ON board_videos (board_id, created_at);",
]
triggers = [
"""--sql
CREATE TRIGGER IF NOT EXISTS tg_board_videos_updated_at
AFTER UPDATE
ON board_videos FOR EACH ROW
BEGIN
UPDATE board_videos SET updated_at = STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')
WHERE board_id = new.board_id AND video_name = old.video_name;
END;
"""
]
for stmt in tables + indices + triggers:
cursor.execute(stmt)
def build_migration() -> Migration:
"""Builds the migration that adds the videos and board_videos tables.
Depends on migration_27, which last reshaped the boards table and created the 'system' user
that videos.user_id defaults to.
"""
return Migration(
id="2026_07_01_add_videos_tables",
depends_on="migration_27",
callback=AddVideosTablesCallback(),
)
+5
View File
@@ -9,6 +9,11 @@ class UrlServiceBase(ABC):
"""Gets the URL for an image or thumbnail."""
pass
@abstractmethod
def get_video_url(self, video_name: str, thumbnail: bool = False) -> str:
"""Gets the URL for a video or its first-frame thumbnail."""
pass
@abstractmethod
def get_model_image_url(self, model_key: str) -> str:
"""Gets the URL for a model image"""
@@ -17,6 +17,15 @@ class LocalUrlService(UrlServiceBase):
return f"{self._base_url}/images/i/{image_basename}/full"
def get_video_url(self, video_name: str, thumbnail: bool = False) -> str:
video_basename = os.path.basename(video_name)
# These paths are determined by the routes in invokeai/app/api/routers/videos.py
if thumbnail:
return f"{self._base_url}/videos/i/{video_basename}/thumbnail"
return f"{self._base_url}/videos/i/{video_basename}/full"
def get_model_image_url(self, model_key: str) -> str:
return f"{self._base_url_v2}/models/i/{model_key}/image"
@@ -0,0 +1,67 @@
from abc import ABC, abstractmethod
from pathlib import Path
from typing import Optional
from PIL import Image
class VideoFileStorageBase(ABC):
"""Low-level service responsible for storing and retrieving video files."""
@abstractmethod
def get_path(self, video_name: str, thumbnail: bool = False, video_subfolder: str = "") -> Path:
"""Gets the internal path to a video or its thumbnail."""
pass
@abstractmethod
def save(
self,
source_path: Path,
video_name: str,
thumbnail_size: int = 256,
video_subfolder: str = "",
metadata: Optional[str] = None,
workflow: Optional[str] = None,
graph: Optional[str] = None,
first_frame: Optional[Image.Image] = None,
) -> None:
"""Saves a video by moving/copying the file at `source_path` into storage, then writes a sibling
WEBP thumbnail extracted from the first frame, plus an optional sidecar JSON of metadata/workflow/graph.
A caller that already decoded frame 0 can pass it as `first_frame` to skip the extraction.
"""
pass
@abstractmethod
def delete(self, video_name: str, video_subfolder: str = "") -> None:
"""Deletes a video file and its thumbnail (if one exists)."""
pass
@abstractmethod
def stage_delete(self, video_name: str, video_subfolder: str = "") -> object:
"""Moves a video's files out of service and returns a rollback token."""
pass
@abstractmethod
def commit_delete(self, token: object) -> None:
"""Permanently removes files represented by a staged-delete token."""
pass
@abstractmethod
def rollback_delete(self, token: object) -> None:
"""Restores files represented by a staged-delete token."""
pass
@abstractmethod
def get_workflow(self, video_name: str, video_subfolder: str = "") -> Optional[str]:
"""Gets the workflow JSON sidecar of a video, if any."""
pass
@abstractmethod
def get_graph(self, video_name: str, video_subfolder: str = "") -> Optional[str]:
"""Gets the graph JSON sidecar of a video, if any."""
pass
@abstractmethod
def validate_path(self, path: str) -> bool:
"""Validates the path given for a video or thumbnail."""
pass
@@ -0,0 +1,19 @@
class VideoFileNotFoundException(Exception):
"""Raised when a video file is not found in storage."""
def __init__(self, message="Video file not found"):
super().__init__(message)
class VideoFileSaveException(Exception):
"""Raised when a video file cannot be saved."""
def __init__(self, message="Video file not saved"):
super().__init__(message)
class VideoFileDeleteException(Exception):
"""Raised when a video file cannot be deleted."""
def __init__(self, message="Video file not deleted"):
super().__init__(message)
@@ -0,0 +1,281 @@
import json
import os
import shutil
import tempfile
from dataclasses import dataclass
from pathlib import Path
from typing import Optional, Union
from PIL import Image
from invokeai.app.services.invoker import Invoker
from invokeai.app.services.video_files.video_files_base import VideoFileStorageBase
from invokeai.app.services.video_files.video_files_common import (
VideoFileDeleteException,
VideoFileNotFoundException,
VideoFileSaveException,
)
from invokeai.app.util.thumbnails import make_thumbnail
from invokeai.app.util.video_thumbnails import extract_video_frame, get_video_thumbnail_name
from invokeai.backend.util.logging import InvokeAILogger
@dataclass
class _StagedDelete:
directory: Path
files: list[tuple[Path, Path]]
class DiskVideoFileStorage(VideoFileStorageBase):
"""Stores video files on disk under {outputs}/videos/, with first-frame WebP thumbnails under
{outputs}/videos/thumbnails/ and optional JSON sidecars for metadata/workflow/graph under
{outputs}/videos/sidecars/."""
def __init__(self, output_folder: Union[str, Path]):
self.__output_folder = output_folder if isinstance(output_folder, Path) else Path(output_folder)
self.__thumbnails_folder = self.__output_folder / "thumbnails"
self.__sidecars_folder = self.__output_folder / "sidecars"
self.__validate_storage_folders()
def start(self, invoker: Invoker) -> None:
self.__invoker = invoker
self.__recover_staged_deletes()
def save(
self,
source_path: Path,
video_name: str,
thumbnail_size: int = 256,
video_subfolder: str = "",
metadata: Optional[str] = None,
workflow: Optional[str] = None,
graph: Optional[str] = None,
first_frame: Optional[Image.Image] = None,
) -> None:
logger = InvokeAILogger.get_logger()
try:
self.__validate_storage_folders()
video_path = self.get_path(video_name, video_subfolder=video_subfolder)
video_path.parent.mkdir(parents=True, exist_ok=True)
# Move if the source is on the same filesystem; otherwise copy then unlink.
try:
shutil.move(str(source_path), str(video_path))
except Exception:
shutil.copy2(str(source_path), str(video_path))
try:
Path(source_path).unlink(missing_ok=True)
except Exception:
pass
logger.info(f"Video file written: {video_path}")
thumbnail_name = get_video_thumbnail_name(video_name)
thumbnail_path = self.get_path(thumbnail_name, thumbnail=True, video_subfolder=video_subfolder)
thumbnail_path.parent.mkdir(parents=True, exist_ok=True)
# Thumbnail extraction is best-effort — if both imageio and cv2 fail, we still want
# the video record + file in place and the invocation to complete. A missing
# thumbnail leaves the gallery with a broken-image placeholder for that item, which
# is annoying but not fatal. The upload path already decoded frame 0 to prove
# decodability and passes it in, saving a decode-worker subprocess per upload.
frame = first_frame
if frame is None:
try:
frame = extract_video_frame(video_path, frame_index=0)
except Exception as e:
logger.warning(f"Thumbnail extraction raised for {video_name}: {e}")
frame = None
if frame is not None:
thumbnail = make_thumbnail(frame, thumbnail_size)
thumbnail.save(thumbnail_path, "WEBP")
logger.info(f"Thumbnail written: {thumbnail_path}")
else:
logger.warning(
f"Could not extract a thumbnail frame for {video_name}; gallery thumbnail will be missing."
)
if metadata is not None or workflow is not None or graph is not None:
sidecar_path = self.__get_sidecar_path(video_name, video_subfolder=video_subfolder)
sidecar_path.parent.mkdir(parents=True, exist_ok=True)
sidecar = {
"invokeai_metadata": metadata,
"invokeai_workflow": workflow,
"invokeai_graph": graph,
}
with open(sidecar_path, "w", encoding="utf-8") as f:
json.dump(sidecar, f)
logger.info(f"Sidecar written: {sidecar_path}")
except Exception as e:
# By this point the source MP4 has usually already been moved into permanent
# storage, so bailing out without cleanup would orphan the video (and any
# partially written thumbnail/sidecar) on disk with no DB record through which
# it can be managed — the caller rolls the record back on this exception.
try:
self.delete(video_name, video_subfolder=video_subfolder)
except Exception as cleanup_err:
logger.error(f"Failed to clean up partially saved files for {video_name}: {cleanup_err}")
raise VideoFileSaveException from e
def delete(self, video_name: str, video_subfolder: str = "") -> None:
token = self.stage_delete(video_name, video_subfolder)
self.commit_delete(token)
def stage_delete(self, video_name: str, video_subfolder: str = "") -> _StagedDelete:
candidates = [
self.get_path(video_name, video_subfolder=video_subfolder),
self.get_path(video_name, thumbnail=True, video_subfolder=video_subfolder),
self.__get_sidecar_path(video_name, video_subfolder=video_subfolder),
]
staging_dir = Path(tempfile.mkdtemp(prefix=".delete_", dir=self.__output_folder))
staged: list[tuple[Path, Path]] = []
try:
with open(staging_dir / "manifest.json", "w", encoding="utf-8") as manifest:
manifest.write(json.dumps({"video_name": video_name, "video_subfolder": video_subfolder}))
manifest.flush()
os.fsync(manifest.fileno())
for index, source in enumerate(candidates):
if source.exists():
destination = staging_dir / str(index)
source.replace(destination)
staged.append((source, destination))
return _StagedDelete(directory=staging_dir, files=staged)
except Exception as e:
for source, destination in reversed(staged):
if destination.exists():
source.parent.mkdir(parents=True, exist_ok=True)
destination.replace(source)
shutil.rmtree(staging_dir, ignore_errors=True)
raise VideoFileDeleteException from e
def commit_delete(self, token: object) -> None:
if not isinstance(token, _StagedDelete):
raise VideoFileDeleteException("Invalid staged-delete token")
shutil.rmtree(token.directory)
def rollback_delete(self, token: object) -> None:
if not isinstance(token, _StagedDelete):
raise VideoFileDeleteException("Invalid staged-delete token")
try:
for source, destination in reversed(token.files):
if destination.exists():
source.parent.mkdir(parents=True, exist_ok=True)
destination.replace(source)
shutil.rmtree(token.directory, ignore_errors=True)
except Exception as e:
raise VideoFileDeleteException from e
def get_path(self, video_name: str, thumbnail: bool = False, video_subfolder: str = "") -> Path:
base_folder = self.__thumbnails_folder if thumbnail else self.__output_folder
filename = get_video_thumbnail_name(video_name) if thumbnail else video_name
basename = Path(filename).name
if basename != filename:
raise ValueError("Invalid video name, potential directory traversal detected")
if video_subfolder:
self._validate_subfolder(video_subfolder)
video_path = base_folder / video_subfolder / basename
else:
video_path = base_folder / basename
resolved_base = base_folder.resolve()
resolved_video_path = video_path.resolve()
if not resolved_video_path.is_relative_to(resolved_base):
raise ValueError("Video path outside outputs folder, potential directory traversal detected")
return resolved_video_path
def get_workflow(self, video_name: str, video_subfolder: str = "") -> Optional[str]:
sidecar = self.__read_sidecar(video_name, video_subfolder)
if sidecar is None:
return None
workflow = sidecar.get("invokeai_workflow")
return workflow if isinstance(workflow, str) else None
def get_graph(self, video_name: str, video_subfolder: str = "") -> Optional[str]:
sidecar = self.__read_sidecar(video_name, video_subfolder)
if sidecar is None:
return None
graph = sidecar.get("invokeai_graph")
return graph if isinstance(graph, str) else None
def validate_path(self, path: Union[str, Path]) -> bool:
path = path if isinstance(path, Path) else Path(path)
return path.exists()
@staticmethod
def _validate_subfolder(subfolder: str) -> None:
"""Validates a subfolder path to prevent directory traversal."""
if not subfolder:
return
if "\\" in subfolder:
raise ValueError("Backslashes not allowed in subfolder path")
if subfolder.startswith("/"):
raise ValueError("Absolute paths not allowed in subfolder path")
for part in subfolder.split("/"):
if part == "..":
raise ValueError("Parent directory references not allowed in subfolder path")
if part == "":
raise ValueError("Empty path segments not allowed in subfolder path")
def __get_sidecar_path(self, video_name: str, video_subfolder: str = "") -> Path:
sidecar_name = Path(video_name).stem + ".json"
if video_subfolder:
self._validate_subfolder(video_subfolder)
sidecar_path = self.__sidecars_folder / video_subfolder / sidecar_name
else:
sidecar_path = self.__sidecars_folder / sidecar_name
resolved_base = self.__sidecars_folder.resolve()
resolved_sidecar_path = sidecar_path.resolve()
if not resolved_sidecar_path.is_relative_to(resolved_base):
raise ValueError("Sidecar path outside outputs folder, potential directory traversal detected")
return resolved_sidecar_path
def __read_sidecar(self, video_name: str, video_subfolder: str = "") -> Optional[dict]:
path = self.__get_sidecar_path(video_name, video_subfolder=video_subfolder)
if not path.exists():
return None
try:
with open(path, encoding="utf-8") as f:
return json.load(f)
except Exception as e:
raise VideoFileNotFoundException from e
def __validate_storage_folders(self) -> None:
for folder in (self.__output_folder, self.__thumbnails_folder, self.__sidecars_folder):
folder.mkdir(parents=True, exist_ok=True)
def __recover_staged_deletes(self) -> None:
logger = InvokeAILogger.get_logger()
for staging_dir in self.__output_folder.glob(".delete_*"):
manifest_path = staging_dir / "manifest.json"
if not manifest_path.is_file():
if not any(staging_dir.iterdir()):
staging_dir.rmdir()
continue
try:
with open(manifest_path, encoding="utf-8") as manifest:
data = json.load(manifest)
video_name = data["video_name"]
video_subfolder = data.get("video_subfolder", "")
candidates = [
self.get_path(video_name, video_subfolder=video_subfolder),
self.get_path(video_name, thumbnail=True, video_subfolder=video_subfolder),
self.__get_sidecar_path(video_name, video_subfolder=video_subfolder),
]
token = _StagedDelete(
directory=staging_dir,
files=[(source, staging_dir / str(index)) for index, source in enumerate(candidates)],
)
# get() raises VideoRecordNotFoundException when the record is gone —
# handled below by purging the staging dir (the delete had committed in
# the DB, so finish it). A surviving record means the delete never
# committed: restore the files.
self.__invoker.services.video_records.get(video_name)
self.rollback_delete(token)
except Exception as error:
from invokeai.app.services.video_records.video_records_common import VideoRecordNotFoundException
if isinstance(error, VideoRecordNotFoundException):
shutil.rmtree(staging_dir, ignore_errors=True)
else:
logger.error(f"Failed to recover staged video deletion {staging_dir}: {error}")
@@ -0,0 +1,108 @@
from abc import ABC, abstractmethod
from datetime import datetime
from typing import Optional
from invokeai.app.invocations.fields import MetadataField
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
from invokeai.app.services.video_records.video_records_common import (
VideoNamesResult,
VideoRecord,
VideoRecordChanges,
)
class VideoRecordStorageBase(ABC):
"""Low-level service responsible for interfacing with the video record store."""
@abstractmethod
def get(self, video_name: str) -> VideoRecord:
"""Gets a video record."""
pass
@abstractmethod
def get_metadata(self, video_name: str) -> Optional[MetadataField]:
"""Gets a video's metadata."""
pass
@abstractmethod
def update(self, video_name: str, changes: VideoRecordChanges) -> None:
"""Updates a video record."""
pass
@abstractmethod
def get_many(
self,
offset: int = 0,
limit: int = 10,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
video_origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> OffsetPaginatedResults[VideoRecord]:
"""Gets a page of video records."""
pass
@abstractmethod
def delete(self, video_name: str) -> None:
"""Deletes a video record."""
pass
@abstractmethod
def delete_many(self, video_names: list[str]) -> None:
"""Deletes many video records."""
pass
@abstractmethod
def save(
self,
video_name: str,
video_origin: ResourceOrigin,
video_category: ImageCategory,
width: int,
height: int,
duration: float,
fps: Optional[float],
has_workflow: bool,
is_intermediate: Optional[bool] = False,
starred: Optional[bool] = False,
session_id: Optional[str] = None,
node_id: Optional[str] = None,
metadata: Optional[str] = None,
user_id: Optional[str] = None,
video_subfolder: str = "",
) -> datetime:
"""Saves a video record."""
pass
@abstractmethod
def get_user_id(self, video_name: str) -> Optional[str]:
"""Gets the user_id of the video owner. Returns None if video not found."""
pass
@abstractmethod
def get_most_recent_video_for_board(self, board_id: str) -> Optional[VideoRecord]:
"""Gets the most recent non-intermediate video on a board, preferring starred videos."""
pass
@abstractmethod
def get_video_names(
self,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
video_origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> VideoNamesResult:
"""Gets ordered list of video names with metadata for optimistic updates."""
pass
@@ -0,0 +1,137 @@
import datetime
from typing import Optional, Union
from pydantic import BaseModel, Field, StrictBool, StrictStr
from invokeai.app.services.image_records.image_records_common import (
ImageCategory,
ResourceOrigin,
)
from invokeai.app.util.misc import get_iso_timestamp
from invokeai.app.util.model_exclude_null import BaseModelExcludeNull
class VideoRecordNotFoundException(Exception):
"""Raised when a video record is not found."""
def __init__(self, message="Video record not found"):
super().__init__(message)
class VideoRecordSaveException(Exception):
"""Raised when a video record cannot be saved."""
def __init__(self, message="Video record not saved"):
super().__init__(message)
class VideoRecordDeleteException(Exception):
"""Raised when a video record cannot be deleted."""
def __init__(self, message="Video record not deleted"):
super().__init__(message)
VIDEO_DTO_COLS = ", ".join(
[
"videos." + c
for c in [
"video_name",
"video_origin",
"video_category",
"width",
"height",
"duration",
"fps",
"session_id",
"node_id",
"has_workflow",
"is_intermediate",
"created_at",
"updated_at",
"deleted_at",
"starred",
"video_subfolder",
]
]
)
class VideoRecord(BaseModelExcludeNull):
"""Deserialized video record without metadata."""
video_name: str = Field(description="The unique name of the video.")
video_origin: ResourceOrigin = Field(description="The origin of the video.")
video_category: ImageCategory = Field(description="The category of the video (reuses ImageCategory).")
width: int = Field(description="The pixel width of the video.")
height: int = Field(description="The pixel height of the video.")
duration: float = Field(description="The duration of the video in seconds.")
fps: Optional[float] = Field(default=None, description="The frames-per-second of the video, if known.")
created_at: Union[datetime.datetime, str] = Field(description="The created timestamp of the video.")
updated_at: Union[datetime.datetime, str] = Field(description="The updated timestamp of the video.")
deleted_at: Optional[Union[datetime.datetime, str]] = Field(
default=None, description="The deleted timestamp of the video."
)
is_intermediate: bool = Field(description="Whether this is an intermediate video.")
session_id: Optional[str] = Field(default=None, description="The session ID that produced this video, if any.")
node_id: Optional[str] = Field(default=None, description="The node ID that produced this video, if any.")
starred: bool = Field(description="Whether this video is starred.")
has_workflow: bool = Field(description="Whether this video has a workflow associated.")
video_subfolder: str = Field(default="", description="The subfolder where the video is stored on disk.")
class VideoRecordChanges(BaseModelExcludeNull, extra="allow"):
"""Allowed mutations on a video record."""
video_category: Optional[ImageCategory] = Field(default=None, description="The video's new category.")
session_id: Optional[StrictStr] = Field(default=None, description="The video's new session ID.")
is_intermediate: Optional[StrictBool] = Field(default=None, description="The video's new `is_intermediate` flag.")
starred: Optional[StrictBool] = Field(default=None, description="The video's new `starred` state.")
def deserialize_video_record(video_dict: dict) -> VideoRecord:
"""Deserializes a video record from a sqlite row dict."""
video_name = video_dict.get("video_name", "unknown")
video_origin = ResourceOrigin(video_dict.get("video_origin", ResourceOrigin.INTERNAL.value))
video_category = ImageCategory(video_dict.get("video_category", ImageCategory.GENERAL.value))
width = video_dict.get("width", 0)
height = video_dict.get("height", 0)
duration = video_dict.get("duration", 0.0)
fps_raw = video_dict.get("fps", None)
fps = float(fps_raw) if fps_raw is not None else None
session_id = video_dict.get("session_id", None)
node_id = video_dict.get("node_id", None)
created_at = video_dict.get("created_at", get_iso_timestamp())
updated_at = video_dict.get("updated_at", get_iso_timestamp())
deleted_at = video_dict.get("deleted_at", None)
is_intermediate = video_dict.get("is_intermediate", False)
starred = video_dict.get("starred", False)
has_workflow = video_dict.get("has_workflow", False)
video_subfolder = video_dict.get("video_subfolder", "")
return VideoRecord(
video_name=video_name,
video_origin=video_origin,
video_category=video_category,
width=width,
height=height,
duration=float(duration),
fps=fps,
session_id=session_id,
node_id=node_id,
created_at=created_at,
updated_at=updated_at,
deleted_at=deleted_at,
is_intermediate=is_intermediate,
starred=starred,
has_workflow=has_workflow,
video_subfolder=video_subfolder,
)
class VideoNamesResult(BaseModel):
"""Response containing ordered video names with metadata for optimistic updates."""
video_names: list[str] = Field(description="Ordered list of video names")
starred_count: int = Field(description="Number of starred videos (when starred_first=True)")
total_count: int = Field(description="Total number of videos matching the query")
@@ -0,0 +1,380 @@
import sqlite3
from datetime import datetime
from typing import Optional, Union, cast
from invokeai.app.invocations.fields import MetadataField, MetadataFieldValidator
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
from invokeai.app.services.shared.sqlite.sqlite_database import SqliteDatabase
from invokeai.app.services.video_records.video_records_base import VideoRecordStorageBase
from invokeai.app.services.video_records.video_records_common import (
VIDEO_DTO_COLS,
VideoNamesResult,
VideoRecord,
VideoRecordChanges,
VideoRecordDeleteException,
VideoRecordNotFoundException,
VideoRecordSaveException,
deserialize_video_record,
)
class SqliteVideoRecordStorage(VideoRecordStorageBase):
def __init__(self, db: SqliteDatabase) -> None:
super().__init__()
self._db = db
def get(self, video_name: str) -> VideoRecord:
with self._db.transaction() as cursor:
try:
cursor.execute(
f"""--sql
SELECT {VIDEO_DTO_COLS} FROM videos
WHERE video_name = ?;
""",
(video_name,),
)
result = cast(Optional[sqlite3.Row], cursor.fetchone())
except sqlite3.Error as e:
raise VideoRecordNotFoundException from e
if not result:
raise VideoRecordNotFoundException
return deserialize_video_record(dict(result))
def get_user_id(self, video_name: str) -> Optional[str]:
with self._db.transaction() as cursor:
cursor.execute(
"""--sql
SELECT user_id FROM videos
WHERE video_name = ?;
""",
(video_name,),
)
result = cast(Optional[sqlite3.Row], cursor.fetchone())
if not result:
return None
return cast(Optional[str], dict(result).get("user_id"))
def get_most_recent_video_for_board(self, board_id: str) -> Optional[VideoRecord]:
with self._db.transaction() as cursor:
cursor.execute(
f"""--sql
SELECT {VIDEO_DTO_COLS}
FROM videos
JOIN board_videos ON videos.video_name = board_videos.video_name
WHERE board_videos.board_id = ?
AND videos.is_intermediate = FALSE
ORDER BY videos.starred DESC, videos.created_at DESC, videos.video_name DESC
LIMIT 1;
""",
(board_id,),
)
result = cast(Optional[sqlite3.Row], cursor.fetchone())
if result is None:
return None
return deserialize_video_record(dict(result))
def get_metadata(self, video_name: str) -> Optional[MetadataField]:
with self._db.transaction() as cursor:
try:
cursor.execute(
"""--sql
SELECT metadata FROM videos
WHERE video_name = ?;
""",
(video_name,),
)
result = cast(Optional[sqlite3.Row], cursor.fetchone())
except sqlite3.Error as e:
raise VideoRecordNotFoundException from e
if not result:
raise VideoRecordNotFoundException
as_dict = dict(result)
metadata_raw = cast(Optional[str], as_dict.get("metadata", None))
return MetadataFieldValidator.validate_json(metadata_raw) if metadata_raw is not None else None
def update(self, video_name: str, changes: VideoRecordChanges) -> None:
with self._db.transaction() as cursor:
try:
if changes.video_category is not None:
cursor.execute(
"UPDATE videos SET video_category = ? WHERE video_name = ?;",
(changes.video_category.value, video_name),
)
if changes.session_id is not None:
cursor.execute(
"UPDATE videos SET session_id = ? WHERE video_name = ?;",
(changes.session_id, video_name),
)
if changes.is_intermediate is not None:
cursor.execute(
"UPDATE videos SET is_intermediate = ? WHERE video_name = ?;",
(changes.is_intermediate, video_name),
)
if changes.starred is not None:
cursor.execute(
"UPDATE videos SET starred = ? WHERE video_name = ?;",
(changes.starred, video_name),
)
except sqlite3.Error as e:
raise VideoRecordSaveException from e
def get_many(
self,
offset: int = 0,
limit: int = 10,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
video_origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> OffsetPaginatedResults[VideoRecord]:
with self._db.transaction() as cursor:
count_query = """--sql
SELECT COUNT(*)
FROM videos
LEFT JOIN board_videos ON board_videos.video_name = videos.video_name
WHERE 1=1
"""
videos_query = f"""--sql
SELECT {VIDEO_DTO_COLS}
FROM videos
LEFT JOIN board_videos ON board_videos.video_name = videos.video_name
WHERE 1=1
"""
query_conditions = ""
query_params: list[Union[int, str, bool]] = []
if video_origin is not None:
query_conditions += " AND videos.video_origin = ? "
query_params.append(video_origin.value)
if categories is not None:
category_strings = [c.value for c in set(categories)]
placeholders = ",".join("?" * len(category_strings))
query_conditions += f" AND videos.video_category IN ( {placeholders} ) "
for c in category_strings:
query_params.append(c)
if is_intermediate is not None:
query_conditions += " AND videos.is_intermediate = ? "
query_params.append(is_intermediate)
if board_id == "none":
query_conditions += " AND board_videos.board_id IS NULL "
if user_id is not None and not is_admin:
query_conditions += " AND videos.user_id = ? "
query_params.append(user_id)
elif board_id is not None:
query_conditions += " AND board_videos.board_id = ? "
query_params.append(board_id)
elif user_id is not None and not is_admin:
# No board_id supplied — still enforce per-user isolation so
# non-admin callers cannot enumerate other users' videos.
query_conditions += " AND videos.user_id = ? "
query_params.append(user_id)
if search_term:
query_conditions += " AND (videos.metadata LIKE ? OR videos.created_at LIKE ?) "
query_params.append(f"%{search_term.lower()}%")
query_params.append(f"%{search_term.lower()}%")
if starred_first:
query_pagination = (
f" ORDER BY videos.starred DESC, videos.created_at {order_dir.value}, "
f"videos.video_name {order_dir.value} LIMIT ? OFFSET ? "
)
else:
query_pagination = (
f" ORDER BY videos.created_at {order_dir.value}, videos.video_name {order_dir.value} "
"LIMIT ? OFFSET ? "
)
videos_query += query_conditions + query_pagination + ";"
videos_params = query_params.copy()
videos_params.extend([limit, offset])
cursor.execute(videos_query, videos_params)
result = cast(list[sqlite3.Row], cursor.fetchall())
videos = [deserialize_video_record(dict(r)) for r in result]
count_query += query_conditions + ";"
cursor.execute(count_query, query_params.copy())
count = cast(int, cursor.fetchone()[0])
return OffsetPaginatedResults(items=videos, offset=offset, limit=limit, total=count)
def delete(self, video_name: str) -> None:
with self._db.transaction() as cursor:
try:
cursor.execute("DELETE FROM videos WHERE video_name = ?;", (video_name,))
except sqlite3.Error as e:
raise VideoRecordDeleteException from e
def delete_many(self, video_names: list[str]) -> None:
with self._db.transaction() as cursor:
try:
placeholders = ",".join("?" for _ in video_names)
cursor.execute(f"DELETE FROM videos WHERE video_name IN ({placeholders})", video_names)
except sqlite3.Error as e:
raise VideoRecordDeleteException from e
def save(
self,
video_name: str,
video_origin: ResourceOrigin,
video_category: ImageCategory,
width: int,
height: int,
duration: float,
fps: Optional[float],
has_workflow: bool,
is_intermediate: Optional[bool] = False,
starred: Optional[bool] = False,
session_id: Optional[str] = None,
node_id: Optional[str] = None,
metadata: Optional[str] = None,
user_id: Optional[str] = None,
video_subfolder: str = "",
) -> datetime:
with self._db.transaction() as cursor:
try:
cursor.execute(
"""--sql
INSERT OR IGNORE INTO videos (
video_name,
video_origin,
video_category,
width,
height,
duration,
fps,
node_id,
session_id,
metadata,
is_intermediate,
starred,
has_workflow,
user_id,
video_subfolder
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
""",
(
video_name,
video_origin.value,
video_category.value,
width,
height,
float(duration),
float(fps) if fps is not None else None,
node_id,
session_id,
metadata,
is_intermediate,
starred,
has_workflow,
user_id or "system",
video_subfolder,
),
)
cursor.execute(
"SELECT created_at FROM videos WHERE video_name = ?;",
(video_name,),
)
created_at = datetime.fromisoformat(cursor.fetchone()[0])
except sqlite3.Error as e:
raise VideoRecordSaveException from e
return created_at
def get_video_names(
self,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
video_origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> VideoNamesResult:
with self._db.transaction() as cursor:
query_conditions = ""
query_params: list[Union[int, str, bool]] = []
if video_origin is not None:
query_conditions += " AND videos.video_origin = ? "
query_params.append(video_origin.value)
if categories is not None:
category_strings = [c.value for c in set(categories)]
placeholders = ",".join("?" * len(category_strings))
query_conditions += f" AND videos.video_category IN ( {placeholders} ) "
for c in category_strings:
query_params.append(c)
if is_intermediate is not None:
query_conditions += " AND videos.is_intermediate = ? "
query_params.append(is_intermediate)
if board_id == "none":
query_conditions += " AND board_videos.board_id IS NULL "
if user_id is not None and not is_admin:
query_conditions += " AND videos.user_id = ? "
query_params.append(user_id)
elif board_id is not None:
query_conditions += " AND board_videos.board_id = ? "
query_params.append(board_id)
elif user_id is not None and not is_admin:
# No board_id supplied — still enforce per-user isolation so
# non-admin callers cannot enumerate other users' videos.
query_conditions += " AND videos.user_id = ? "
query_params.append(user_id)
if search_term:
query_conditions += " AND (videos.metadata LIKE ? OR videos.created_at LIKE ?) "
query_params.append(f"%{search_term.lower()}%")
query_params.append(f"%{search_term.lower()}%")
starred_count = 0
if starred_first:
cursor.execute(
f"""--sql
SELECT COUNT(*)
FROM videos
LEFT JOIN board_videos ON board_videos.video_name = videos.video_name
WHERE videos.starred = TRUE AND (1=1{query_conditions})
""",
query_params,
)
starred_count = cast(int, cursor.fetchone()[0])
order_clause = (
f" ORDER BY videos.starred DESC, videos.created_at {order_dir.value}, "
f"videos.video_name {order_dir.value} "
if starred_first
else f" ORDER BY videos.created_at {order_dir.value}, videos.video_name {order_dir.value} "
)
cursor.execute(
f"""--sql
SELECT videos.video_name
FROM videos
LEFT JOIN board_videos ON board_videos.video_name = videos.video_name
WHERE 1=1{query_conditions}
{order_clause}
""",
query_params,
)
result = cast(list[sqlite3.Row], cursor.fetchall())
video_names = [row[0] for row in result]
return VideoNamesResult(video_names=video_names, starred_count=starred_count, total_count=len(video_names))
+166
View File
@@ -0,0 +1,166 @@
from abc import ABC, abstractmethod
from pathlib import Path
from typing import Callable, Optional
from PIL import Image
from invokeai.app.invocations.fields import MetadataField
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
from invokeai.app.services.video_records.video_records_common import (
VideoNamesResult,
VideoRecord,
VideoRecordChanges,
)
from invokeai.app.services.videos.videos_common import VideoDTO
class VideoServiceABC(ABC):
"""High-level service for video management."""
_on_changed_callbacks: list[Callable[[VideoDTO], None]]
_on_deleted_callbacks: list[Callable[[str], None]]
def __init__(self) -> None:
self._on_changed_callbacks = []
self._on_deleted_callbacks = []
def on_changed(self, on_changed: Callable[[VideoDTO], None]) -> None:
"""Register a callback for when a video is changed."""
self._on_changed_callbacks.append(on_changed)
def on_deleted(self, on_deleted: Callable[[str], None]) -> None:
"""Register a callback for when a video is deleted."""
self._on_deleted_callbacks.append(on_deleted)
def _on_changed(self, item: VideoDTO) -> None:
for callback in self._on_changed_callbacks:
callback(item)
def _on_deleted(self, item_id: str) -> None:
for callback in self._on_deleted_callbacks:
callback(item_id)
@abstractmethod
def create(
self,
source_path: Path,
width: int,
height: int,
duration: float,
fps: Optional[float],
video_origin: ResourceOrigin,
video_category: ImageCategory,
node_id: Optional[str] = None,
session_id: Optional[str] = None,
board_id: Optional[str] = None,
is_intermediate: Optional[bool] = False,
metadata: Optional[str] = None,
workflow: Optional[str] = None,
graph: Optional[str] = None,
user_id: Optional[str] = None,
first_frame: Optional[Image.Image] = None,
) -> VideoDTO:
"""Creates a video by moving/copying the file at `source_path` into storage and recording it.
``first_frame``, when provided (e.g. the upload path already decoded frame 0 to
prove decodability), is used as the thumbnail source instead of spawning another
decode worker.
"""
pass
@abstractmethod
def update(self, video_name: str, changes: VideoRecordChanges) -> VideoDTO:
"""Updates a video."""
pass
@abstractmethod
def get_record(self, video_name: str) -> VideoRecord:
"""Gets a video record."""
pass
@abstractmethod
def get_dto(self, video_name: str) -> VideoDTO:
"""Gets a video DTO."""
pass
@abstractmethod
def get_metadata(self, video_name: str) -> Optional[MetadataField]:
"""Gets a video's metadata."""
pass
@abstractmethod
def get_workflow(self, video_name: str) -> Optional[str]:
"""Gets a video's workflow."""
pass
@abstractmethod
def get_graph(self, video_name: str) -> Optional[str]:
"""Gets a video's graph."""
pass
@abstractmethod
def get_path(self, video_name: str, thumbnail: bool = False) -> str:
"""Gets a video's on-disk path."""
pass
@abstractmethod
def get_url(self, video_name: str, thumbnail: bool = False) -> str:
"""Gets a video's URL."""
pass
@abstractmethod
def get_many(
self,
offset: int = 0,
limit: int = 10,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
video_origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> OffsetPaginatedResults[VideoDTO]:
"""Gets a paginated list of video DTOs."""
pass
@abstractmethod
def delete(self, video_name: str) -> None:
"""Deletes a video."""
pass
@abstractmethod
def delete_videos_on_board(self, board_id: str, user_id: Optional[str] = None) -> tuple[list[str], list[str]]:
"""Deletes all videos on a board; returns ``(deleted_names, failed_names)``.
When ``user_id`` is provided, only videos owned by that user are deleted (other users'
contributions to a public/shared board are preserved). Pass ``None`` for the admin
path to delete every video on the board regardless of uploader.
Videos whose backing file deletion fails are intentionally retained (their DB records
survive and cascade to "uncategorized" via the board_videos FK), so the returned lists
are the authoritative ``deleted_videos``/``failed_videos`` for the caller's response —
callers must not reconstruct failures by diffing their own board listing, which races
with concurrent moves/deletes.
"""
pass
@abstractmethod
def get_video_names(
self,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
video_origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> VideoNamesResult:
"""Gets ordered list of video names."""
pass
@@ -0,0 +1,64 @@
from typing import Optional
from pydantic import BaseModel, Field
from invokeai.app.services.video_records.video_records_common import VideoRecord
from invokeai.app.util.model_exclude_null import BaseModelExcludeNull
class VideoUrlsDTO(BaseModelExcludeNull):
"""The URLs for a video and its thumbnail."""
video_name: str = Field(description="The unique name of the video.")
video_url: str = Field(description="The URL of the video file (MP4).")
thumbnail_url: str = Field(description="The URL of the video's first-frame thumbnail (WebP).")
class VideoDTO(VideoRecord, VideoUrlsDTO):
"""Deserialized video record, enriched for the frontend."""
board_id: Optional[str] = Field(
default=None, description="The id of the board the video belongs to, if one exists."
)
def video_record_to_dto(
video_record: VideoRecord,
video_url: str,
thumbnail_url: str,
board_id: Optional[str],
) -> VideoDTO:
"""Converts a video record to a video DTO."""
return VideoDTO(
**video_record.model_dump(),
video_url=video_url,
thumbnail_url=thumbnail_url,
board_id=board_id,
)
class VideoResultWithAffectedBoards(BaseModel):
affected_boards: list[str] = Field(description="The ids of boards affected by the operation")
class DeleteVideosResult(VideoResultWithAffectedBoards):
deleted_videos: list[str] = Field(description="The names of the videos that were deleted")
failed_videos: list[str] = Field(description="The names of videos that were not deleted")
class StarredVideosResult(VideoResultWithAffectedBoards):
starred_videos: list[str] = Field(description="The names of the videos that were starred")
failed_videos: list[str] = Field(description="The names of videos that were not starred")
class UnstarredVideosResult(VideoResultWithAffectedBoards):
unstarred_videos: list[str] = Field(description="The names of the videos that were unstarred")
failed_videos: list[str] = Field(description="The names of videos that were not unstarred")
class AddVideosToBoardResult(VideoResultWithAffectedBoards):
added_videos: list[str] = Field(description="The video names that were added to the board")
class RemoveVideosFromBoardResult(VideoResultWithAffectedBoards):
removed_videos: list[str] = Field(description="The video names that were removed from their board")
@@ -0,0 +1,406 @@
from pathlib import Path
from typing import Optional
from PIL import Image
from invokeai.app.invocations.fields import MetadataField
from invokeai.app.services.image_records.image_records_common import (
ImageCategory,
InvalidImageCategoryException,
InvalidOriginException,
ResourceOrigin,
)
from invokeai.app.services.invoker import Invoker
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
from invokeai.app.services.video_files.video_files_common import (
VideoFileDeleteException,
VideoFileNotFoundException,
VideoFileSaveException,
)
from invokeai.app.services.video_records.video_records_common import (
VideoNamesResult,
VideoRecord,
VideoRecordChanges,
VideoRecordDeleteException,
VideoRecordNotFoundException,
VideoRecordSaveException,
)
from invokeai.app.services.videos.videos_base import VideoServiceABC
from invokeai.app.services.videos.videos_common import VideoDTO, video_record_to_dto
class VideoService(VideoServiceABC):
__invoker: Invoker
def start(self, invoker: Invoker) -> None:
self.__invoker = invoker
def create(
self,
source_path: Path,
width: int,
height: int,
duration: float,
fps: Optional[float],
video_origin: ResourceOrigin,
video_category: ImageCategory,
node_id: Optional[str] = None,
session_id: Optional[str] = None,
board_id: Optional[str] = None,
is_intermediate: Optional[bool] = False,
metadata: Optional[str] = None,
workflow: Optional[str] = None,
graph: Optional[str] = None,
user_id: Optional[str] = None,
first_frame: Optional[Image.Image] = None,
) -> VideoDTO:
if video_origin not in ResourceOrigin:
raise InvalidOriginException
if video_category not in ImageCategory:
raise InvalidImageCategoryException
video_name = self.__invoker.services.names.create_video_name()
# Reuse the image subfolder strategy for video organization.
from invokeai.app.services.image_files.image_subfolder_strategy import create_subfolder_strategy
strategy_name = self.__invoker.services.configuration.image_subfolder_strategy
strategy = create_subfolder_strategy(strategy_name)
video_subfolder = strategy.get_subfolder(video_name, video_category, is_intermediate or False)
record_saved = False
board_attached = False
try:
self.__invoker.services.video_records.save(
video_name=video_name,
video_origin=video_origin,
video_category=video_category,
width=width,
height=height,
duration=duration,
fps=fps,
has_workflow=workflow is not None or graph is not None,
is_intermediate=is_intermediate,
node_id=node_id,
metadata=metadata,
session_id=session_id,
user_id=user_id,
video_subfolder=video_subfolder,
)
record_saved = True
if board_id is not None:
# Board attachment is deliberately best-effort, mirroring ImageService.create:
# this is reachable when the board is deleted between the caller's access
# check and this insert, and failing the whole create here would destroy a
# just-generated video over a cosmetic categorization problem. The returned
# DTO reports the video's *actual* board (None on fallback), so callers are
# not told the attachment succeeded.
try:
self.__invoker.services.board_video_records.add_video_to_board(
board_id=board_id, video_name=video_name
)
board_attached = True
except Exception as e:
self.__invoker.services.logger.warning(f"Failed to add video to board {board_id}: {str(e)}")
self.__invoker.services.video_files.save(
source_path=source_path,
video_name=video_name,
video_subfolder=video_subfolder,
metadata=metadata,
workflow=workflow,
graph=graph,
first_frame=first_frame,
)
video_dto = self.get_dto(video_name)
self._on_changed(video_dto)
return video_dto
except VideoRecordSaveException:
self.__invoker.services.logger.error("Failed to save video record")
raise
except Exception as e:
# Roll back any DB-side state we created so the gallery doesn't end up with a
# ghost record whose file endpoints 404. Most commonly triggered by
# VideoFileSaveException (disk save or sidecar write failure), but we also
# need to unwind on any unexpected post-record failure.
if board_attached:
try:
self.__invoker.services.board_video_records.remove_video_from_board(video_name=video_name)
except Exception as rollback_err:
self.__invoker.services.logger.error(
f"Failed to roll back board attachment for {video_name}: {str(rollback_err)}"
)
if record_saved:
try:
self.__invoker.services.video_records.delete(video_name)
except Exception as rollback_err:
self.__invoker.services.logger.error(
f"Failed to roll back video record for {video_name}: {str(rollback_err)}"
)
# The disk layer cleans up after itself when the save fails, but a failure
# after a successful file save (e.g. building the DTO) would still leave the
# files on disk with no record pointing at them. delete() skips files that
# don't exist, so this is a no-op when nothing was written.
try:
self.__invoker.services.video_files.delete(video_name, video_subfolder=video_subfolder)
except Exception as rollback_err:
self.__invoker.services.logger.error(
f"Failed to roll back video files for {video_name}: {str(rollback_err)}"
)
if isinstance(e, VideoFileSaveException):
self.__invoker.services.logger.error("Failed to save video file")
else:
self.__invoker.services.logger.error(f"Problem saving video record and file: {str(e)}")
raise
def update(self, video_name: str, changes: VideoRecordChanges) -> VideoDTO:
try:
self.__invoker.services.video_records.update(video_name, changes)
video_dto = self.get_dto(video_name)
self._on_changed(video_dto)
return video_dto
except VideoRecordSaveException:
self.__invoker.services.logger.error("Failed to update video record")
raise
except Exception as e:
self.__invoker.services.logger.error("Problem updating video record")
raise e
def get_record(self, video_name: str) -> VideoRecord:
try:
return self.__invoker.services.video_records.get(video_name)
except VideoRecordNotFoundException:
self.__invoker.services.logger.error("Video record not found")
raise
except Exception as e:
self.__invoker.services.logger.error("Problem getting video record")
raise e
def get_dto(self, video_name: str) -> VideoDTO:
try:
video_record = self.__invoker.services.video_records.get(video_name)
return video_record_to_dto(
video_record=video_record,
video_url=self.__invoker.services.urls.get_video_url(video_name),
thumbnail_url=self.__invoker.services.urls.get_video_url(video_name, thumbnail=True),
board_id=self.__invoker.services.board_video_records.get_board_for_video(video_name),
)
except VideoRecordNotFoundException:
self.__invoker.services.logger.error("Video record not found")
raise
except Exception as e:
self.__invoker.services.logger.error("Problem getting video DTO")
raise e
def get_metadata(self, video_name: str) -> Optional[MetadataField]:
try:
return self.__invoker.services.video_records.get_metadata(video_name)
except VideoRecordNotFoundException:
self.__invoker.services.logger.error("Video record not found")
raise
except Exception as e:
self.__invoker.services.logger.error("Problem getting video metadata")
raise e
def get_workflow(self, video_name: str) -> Optional[str]:
try:
record = self.__invoker.services.video_records.get(video_name)
return self.__invoker.services.video_files.get_workflow(video_name, video_subfolder=record.video_subfolder)
except VideoFileNotFoundException:
self.__invoker.services.logger.error("Video file not found")
raise
except Exception:
self.__invoker.services.logger.error("Problem getting video workflow")
raise
def get_graph(self, video_name: str) -> Optional[str]:
try:
record = self.__invoker.services.video_records.get(video_name)
return self.__invoker.services.video_files.get_graph(video_name, video_subfolder=record.video_subfolder)
except VideoFileNotFoundException:
self.__invoker.services.logger.error("Video file not found")
raise
except Exception:
self.__invoker.services.logger.error("Problem getting video graph")
raise
def get_path(self, video_name: str, thumbnail: bool = False) -> str:
try:
record = self.__invoker.services.video_records.get(video_name)
return str(
self.__invoker.services.video_files.get_path(
video_name, thumbnail=thumbnail, video_subfolder=record.video_subfolder
)
)
except Exception as e:
self.__invoker.services.logger.error("Problem getting video path")
raise e
def get_url(self, video_name: str, thumbnail: bool = False) -> str:
try:
return self.__invoker.services.urls.get_video_url(video_name, thumbnail=thumbnail)
except Exception as e:
self.__invoker.services.logger.error("Problem getting video URL")
raise e
def get_many(
self,
offset: int = 0,
limit: int = 10,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
video_origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> OffsetPaginatedResults[VideoDTO]:
try:
results = self.__invoker.services.video_records.get_many(
offset,
limit,
starred_first,
order_dir,
video_origin,
categories,
is_intermediate,
board_id,
search_term,
user_id,
is_admin,
)
video_dtos = [
video_record_to_dto(
video_record=r,
video_url=self.__invoker.services.urls.get_video_url(r.video_name),
thumbnail_url=self.__invoker.services.urls.get_video_url(r.video_name, thumbnail=True),
board_id=self.__invoker.services.board_video_records.get_board_for_video(r.video_name),
)
for r in results.items
]
return OffsetPaginatedResults[VideoDTO](
items=video_dtos, offset=results.offset, limit=results.limit, total=results.total
)
except Exception as e:
self.__invoker.services.logger.error("Problem getting paginated video DTOs")
raise e
def delete(self, video_name: str) -> None:
token: object | None = None
record_deleted = False
try:
record = self.__invoker.services.video_records.get(video_name)
token = self.__invoker.services.video_files.stage_delete(video_name, video_subfolder=record.video_subfolder)
self.__invoker.services.video_records.delete(video_name)
record_deleted = True
try:
self.__invoker.services.video_files.commit_delete(token)
except Exception as cleanup_error:
self.__invoker.services.logger.error(f"Failed to purge staged video files: {cleanup_error}")
self._on_deleted(video_name)
except VideoRecordDeleteException:
if token is not None:
self.__invoker.services.video_files.rollback_delete(token)
self.__invoker.services.logger.error("Failed to delete video record")
raise
except VideoFileDeleteException:
self.__invoker.services.logger.error("Failed to delete video file")
raise
except Exception as e:
if token is not None and not record_deleted:
try:
self.__invoker.services.video_files.rollback_delete(token)
except Exception as rollback_error:
self.__invoker.services.logger.error(f"Failed to restore video files: {rollback_error}")
self.__invoker.services.logger.error("Problem deleting video record and file")
raise e
def delete_videos_on_board(self, board_id: str, user_id: Optional[str] = None) -> tuple[list[str], list[str]]:
try:
# When ``user_id`` is set the lookup filters to videos owned by that user so the
# cascade doesn't destroy other users' contributions to a public/shared board.
video_names = self.__invoker.services.board_video_records.get_all_board_video_names_for_board(
board_id, categories=None, is_intermediate=None, user_id=user_id
)
# Only delete records for files we actually managed to remove. Otherwise a
# transient FS error would leave the file orphaned on disk with no record
# pointing at it — the API would report success and the user would have no
# way to clean up the leak. The board itself will still be deleted by the
# caller, so any preserved records cascade to "uncategorized" via the
# board_videos FK.
deleted_video_names: list[str] = []
failed_video_names: list[str] = []
staged_deletes: list[tuple[str, object]] = []
for video_name in video_names:
try:
record = self.__invoker.services.video_records.get(video_name)
token = self.__invoker.services.video_files.stage_delete(
video_name, video_subfolder=record.video_subfolder
)
staged_deletes.append((video_name, token))
deleted_video_names.append(video_name)
except Exception as e:
failed_video_names.append(video_name)
self.__invoker.services.logger.error(
f"Failed to delete video file {video_name}; keeping record: {str(e)}"
)
try:
self.__invoker.services.video_records.delete_many(deleted_video_names)
except Exception:
for video_name, token in staged_deletes:
try:
self.__invoker.services.video_files.rollback_delete(token)
except Exception as rollback_error:
self.__invoker.services.logger.error(
f"Failed to restore staged video files for {video_name}: {rollback_error}"
)
raise
for _, token in staged_deletes:
try:
self.__invoker.services.video_files.commit_delete(token)
except Exception as cleanup_error:
self.__invoker.services.logger.error(f"Failed to purge staged video files: {cleanup_error}")
for video_name in deleted_video_names:
self._on_deleted(video_name)
return deleted_video_names, failed_video_names
except VideoRecordDeleteException:
self.__invoker.services.logger.error("Failed to delete video records")
raise
except VideoFileDeleteException:
self.__invoker.services.logger.error("Failed to delete video files")
raise
except Exception as e:
self.__invoker.services.logger.error(f"Problem deleting video records and files: {str(e)}")
raise e
def get_video_names(
self,
starred_first: bool = True,
order_dir: SQLiteDirection = SQLiteDirection.Descending,
video_origin: Optional[ResourceOrigin] = None,
categories: Optional[list[ImageCategory]] = None,
is_intermediate: Optional[bool] = None,
board_id: Optional[str] = None,
search_term: Optional[str] = None,
user_id: Optional[str] = None,
is_admin: bool = False,
) -> VideoNamesResult:
try:
return self.__invoker.services.video_records.get_video_names(
starred_first=starred_first,
order_dir=order_dir,
video_origin=video_origin,
categories=categories,
is_intermediate=is_intermediate,
board_id=board_id,
search_term=search_term,
user_id=user_id,
is_admin=is_admin,
)
except Exception as e:
self.__invoker.services.logger.error("Problem getting video names")
raise e
@@ -4,11 +4,16 @@ from pydantic import BaseModel, Field
class VirtualSubBoardDTO(BaseModel):
"""A virtual sub-board computed from image metadata, not stored in the database."""
"""A virtual sub-board computed from image/video metadata, not stored in the database."""
virtual_board_id: str = Field(description="The virtual board ID, e.g. 'by_date:2026-03-18'.")
board_name: str = Field(description="The display name of the virtual sub-board, e.g. '2026-03-18'.")
date: str = Field(description="The ISO date string, e.g. '2026-03-18'.")
image_count: int = Field(description="The number of general images for this date.")
asset_count: int = Field(description="The number of asset images for this date.")
video_count: int = Field(default=0, description="The number of videos for this date.")
cover_image_name: Optional[str] = Field(default=None, description="The most recent image name for this date.")
cover_video_name: Optional[str] = Field(
default=None,
description="The most recent video name for this date. Set instead of cover_image_name when the newest item for the date is a video.",
)
@@ -0,0 +1,872 @@
{
"name": "Image to Video - Wan 2.2 TI2V (Low Quality)",
"author": "InvokeAI",
"description": "Low quality but fast image-to-video generation with Wan 2.2 TI2V 5B model. Provide an initial image and a target resolution for the video.",
"version": "1.0.0",
"contact": "",
"tags": "wan2.2, image to video, video",
"notes": "This workflow needs the Wan 2.2 TI2V-5B model. Each InvokeAI install assigns its own internal model IDs, so the model fields are left blank -- select these in the form after loading:\n\nMain Model node:\n- Main Model: Wan 2.2 TI2V-5B (Q4_K_M GGUF recommended for <=16 GB VRAM)\n- Transformer (Low Noise): leave empty (TI2V-5B is a single-expert model)\n- VAE: Wan 2.2 TI2V-5B VAE\n- T5 Encoder: Wan T5 Encoder (UMT5-XXL)\n\n720p should work on GPUs with 16 GB VRAM and up.",
"exposedFields": [
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "transformer_low_noise_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "component_source"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "vae_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "wan_t5_encoder_model"
},
{
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
"fieldName": "lora"
},
{
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
"fieldName": "weight"
},
{
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
"fieldName": "target"
},
{
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
"fieldName": "lora"
},
{
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
"fieldName": "weight"
},
{
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
"fieldName": "target"
},
{
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"fieldName": "prompt"
},
{
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"fieldName": "prompt"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "steps"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale_low_noise"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "width"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "height"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "num_frames"
},
{
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"fieldName": "fps"
}
],
"meta": {
"version": "4.0.0",
"category": "default"
},
"nodes": [
{
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"type": "invocation",
"data": {
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_model_loader",
"inputs": {
"model": {
"name": "model",
"label": "Transformer (High Noise)",
"description": "",
"value": null
},
"transformer_low_noise_model": {
"name": "transformer_low_noise_model",
"label": "",
"description": ""
},
"vae_model": {
"name": "vae_model",
"label": "",
"description": "",
"value": null
},
"wan_t5_encoder_model": {
"name": "wan_t5_encoder_model",
"label": "",
"description": "",
"value": null
},
"component_source": {
"name": "component_source",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": false
},
"position": {
"x": 0,
"y": 0
}
},
{
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"type": "invocation",
"data": {
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "Positive Prompt",
"notes": "",
"type": "wan_text_encoder",
"inputs": {
"prompt": {
"name": "prompt",
"label": "",
"description": "",
"value": ""
},
"wan_t5_encoder": {
"name": "wan_t5_encoder",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 686.9799980678728,
"y": 210.13999972398182
}
},
{
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"type": "invocation",
"data": {
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "Negative Prompt (unused at CFG=1.0)",
"notes": "",
"type": "wan_text_encoder",
"inputs": {
"prompt": {
"name": "prompt",
"label": "",
"description": "",
"value": " "
},
"wan_t5_encoder": {
"name": "wan_t5_encoder",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 700,
"y": 392
}
},
{
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"type": "invocation",
"data": {
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "rand_int",
"inputs": {
"low": {
"name": "low",
"label": "",
"description": "",
"value": 0
},
"high": {
"name": "high",
"label": "",
"description": "",
"value": 2147483647
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": false
},
"position": {
"x": 700,
"y": 572
}
},
{
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"type": "invocation",
"data": {
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_video_denoise",
"inputs": {
"transformer": {
"name": "transformer",
"label": "",
"description": ""
},
"positive_conditioning": {
"name": "positive_conditioning",
"label": "",
"description": ""
},
"negative_conditioning": {
"name": "negative_conditioning",
"label": "",
"description": ""
},
"ref_image": {
"name": "ref_image",
"label": "",
"description": ""
},
"guidance_scale": {
"name": "guidance_scale",
"label": "CFG (High Noise)",
"description": "",
"value": 1
},
"guidance_scale_low_noise": {
"name": "guidance_scale_low_noise",
"label": "CFG (Low Noise)",
"description": "",
"value": 1
},
"width": {
"name": "width",
"label": "",
"description": "",
"value": 832
},
"height": {
"name": "height",
"label": "",
"description": "",
"value": 480
},
"num_frames": {
"name": "num_frames",
"label": "Frames",
"description": "",
"value": 81
},
"steps": {
"name": "steps",
"label": "",
"description": "",
"value": 20
},
"seed": {
"name": "seed",
"label": "",
"description": "",
"value": 0
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 1050,
"y": 230
}
},
{
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"type": "invocation",
"data": {
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_l2v",
"inputs": {
"board": {
"name": "board",
"label": "",
"description": ""
},
"metadata": {
"name": "metadata",
"label": "",
"description": ""
},
"latents": {
"name": "latents",
"label": "",
"description": ""
},
"vae": {
"name": "vae",
"label": "",
"description": ""
},
"fps": {
"name": "fps",
"label": "FPS",
"description": "",
"value": 16
}
},
"isOpen": true,
"isIntermediate": false,
"useCache": true
},
"position": {
"x": 1400,
"y": 33
}
},
{
"id": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
"type": "invocation",
"data": {
"id": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
"version": "1.2.0",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_ref_image_encoder",
"inputs": {
"image": {
"name": "image",
"label": "",
"description": ""
},
"end_image": { "name": "end_image", "label": "" },
"vae": {
"name": "vae",
"label": "",
"description": ""
},
"width": {
"name": "width",
"label": "",
"description": "",
"value": 1024
},
"height": {
"name": "height",
"label": "",
"description": "",
"value": 1024
},
"num_frames": {
"name": "num_frames",
"label": "",
"description": "",
"value": 1
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 669.4858247907592,
"y": -383.75896955550337
}
},
{
"id": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
"type": "invocation",
"data": {
"id": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
"version": "1.0.2",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "image",
"inputs": {
"image": {
"name": "image",
"label": "Initial Image",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": -165.5308813883526,
"y": -364.6304014144227
}
},
{
"id": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
"type": "invocation",
"data": {
"id": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_ti2v_ideal_dimensions",
"inputs": {
"width": {
"name": "width",
"label": "",
"description": "",
"value": 1024
},
"height": {
"name": "height",
"label": "",
"description": "",
"value": 1024
},
"target_resolution": {
"name": "target_resolution",
"label": "",
"description": "",
"value": "720p"
},
"rounding": {
"name": "rounding",
"label": "",
"description": "",
"value": "nearest"
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 261.9930762062073,
"y": -602.4138188363532
}
}
],
"edges": [
{
"id": "edge-loader-t5-pos",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-t5-neg",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-vae-l2v",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "vae",
"targetHandle": "vae"
},
{
"id": "edge-pos-cond-denoise",
"type": "default",
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "positive_conditioning"
},
{
"id": "edge-neg-cond-denoise",
"type": "default",
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "negative_conditioning"
},
{
"id": "edge-rand-seed-denoise",
"type": "default",
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "value",
"targetHandle": "seed"
},
{
"id": "edge-denoise-latents-l2v",
"type": "default",
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "latents",
"targetHandle": "latents"
},
{
"id": "reactflow__edge-1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7etransformer-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabtransformer",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "transformer",
"targetHandle": "transformer"
},
{
"id": "reactflow__edge-1473db93-45eb-4468-bba1-d0f9d1c21aedref_image-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabref_image",
"type": "default",
"source": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "ref_image",
"targetHandle": "ref_image"
},
{
"id": "reactflow__edge-1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7evae-1473db93-45eb-4468-bba1-d0f9d1c21aedvae",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
"sourceHandle": "vae",
"targetHandle": "vae"
},
{
"id": "reactflow__edge-0771d3a4-7f0b-45bd-9c73-a99b40e24b7bimage-1473db93-45eb-4468-bba1-d0f9d1c21aedimage",
"type": "default",
"source": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
"target": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
"sourceHandle": "image",
"targetHandle": "image"
},
{
"id": "reactflow__edge-0771d3a4-7f0b-45bd-9c73-a99b40e24b7bwidth-55cb8b4f-b7fc-4d37-9861-d5c84a57353bwidth",
"type": "default",
"source": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
"target": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
"sourceHandle": "width",
"targetHandle": "width"
},
{
"id": "reactflow__edge-0771d3a4-7f0b-45bd-9c73-a99b40e24b7bheight-55cb8b4f-b7fc-4d37-9861-d5c84a57353bheight",
"type": "default",
"source": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
"target": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
"sourceHandle": "height",
"targetHandle": "height"
},
{
"id": "reactflow__edge-55cb8b4f-b7fc-4d37-9861-d5c84a57353bwidth-1473db93-45eb-4468-bba1-d0f9d1c21aedwidth",
"type": "default",
"source": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
"target": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
"sourceHandle": "width",
"targetHandle": "width"
},
{
"id": "reactflow__edge-55cb8b4f-b7fc-4d37-9861-d5c84a57353bheight-1473db93-45eb-4468-bba1-d0f9d1c21aedheight",
"type": "default",
"source": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
"target": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
"sourceHandle": "height",
"targetHandle": "height"
},
{
"id": "reactflow__edge-55cb8b4f-b7fc-4d37-9861-d5c84a57353bwidth-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabwidth",
"type": "default",
"source": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "width",
"targetHandle": "width"
},
{
"id": "reactflow__edge-55cb8b4f-b7fc-4d37-9861-d5c84a57353bheight-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabheight",
"type": "default",
"source": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "height",
"targetHandle": "height"
}
],
"form": {
"elements": {
"root": {
"id": "root",
"data": {
"layout": "column",
"children": [
"heading-XDPaQofdEE",
"node-field-KcT0glAFDK",
"node-field-e3hxuqgTgS",
"container-mUeBcw2tYf",
"heading-wuPtY8B4vR",
"container-AkpUH1WtID",
"heading-10PuWMKJg9",
"container-C0z6ru33to"
]
},
"type": "container"
},
"node-field-zQA9XuRZRW": {
"id": "node-field-zQA9XuRZRW",
"parentId": "container-mUeBcw2tYf",
"data": {
"fieldIdentifier": {
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"fieldName": "fps"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-ok9GStueKC": {
"id": "node-field-ok9GStueKC",
"parentId": "container-mUeBcw2tYf",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "num_frames"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-PdU0Sz2m5z": {
"id": "node-field-PdU0Sz2m5z",
"parentId": "container-y0w7TfJC7e",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale"
},
"showDescription": false,
"settings": {
"type": "float-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-UiQ7MaseDR": {
"id": "node-field-UiQ7MaseDR",
"parentId": "container-mUeBcw2tYf",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "steps"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-KcT0glAFDK": {
"id": "node-field-KcT0glAFDK",
"parentId": "root",
"data": {
"fieldIdentifier": {
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"fieldName": "prompt"
},
"showDescription": false,
"settings": {
"type": "string-field-config",
"component": "textarea"
}
},
"type": "node-field"
},
"node-field-N9mexEKkYg": {
"id": "node-field-N9mexEKkYg",
"parentId": "container-C0z6ru33to",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "wan_t5_encoder_model"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-2u9lAbMBnD": {
"id": "node-field-2u9lAbMBnD",
"parentId": "container-C0z6ru33to",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "vae_model"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-CTGqrO3x8o": {
"id": "node-field-CTGqrO3x8o",
"parentId": "container-C0z6ru33to",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "component_source"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-Q345WyNzc4": {
"id": "node-field-Q345WyNzc4",
"parentId": "container-y0w7TfJC7e",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "model"
},
"showDescription": false
},
"type": "node-field"
},
"container-mUeBcw2tYf": {
"id": "container-mUeBcw2tYf",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"node-field-UiQ7MaseDR",
"node-field-ok9GStueKC",
"node-field-zQA9XuRZRW"
]
},
"type": "container"
},
"heading-XDPaQofdEE": {
"id": "heading-XDPaQofdEE",
"parentId": "root",
"data": {
"content": "Generation"
},
"type": "heading"
},
"heading-wuPtY8B4vR": {
"id": "heading-wuPtY8B4vR",
"parentId": "root",
"data": {
"content": "Model"
},
"type": "heading"
},
"container-AkpUH1WtID": {
"id": "container-AkpUH1WtID",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"container-y0w7TfJC7e"
]
},
"type": "container"
},
"container-y0w7TfJC7e": {
"id": "container-y0w7TfJC7e",
"parentId": "container-AkpUH1WtID",
"data": {
"layout": "column",
"children": [
"node-field-Q345WyNzc4",
"node-field-PdU0Sz2m5z"
]
},
"type": "container"
},
"container-C0z6ru33to": {
"id": "container-C0z6ru33to",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"node-field-N9mexEKkYg",
"node-field-2u9lAbMBnD",
"node-field-CTGqrO3x8o"
]
},
"type": "container"
},
"heading-10PuWMKJg9": {
"id": "heading-10PuWMKJg9",
"parentId": "root",
"data": {
"content": "Advanced"
},
"type": "heading"
},
"node-field-e3hxuqgTgS": {
"id": "node-field-e3hxuqgTgS",
"parentId": "root",
"data": {
"fieldIdentifier": {
"nodeId": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
"fieldName": "image"
},
"showDescription": true
},
"type": "node-field"
}
},
"rootElementId": "root"
},
"id": "default_wan22_i2v_ti2v5b"
}
@@ -0,0 +1,991 @@
{
"name": "Text to Video - Wan 2.2 Lightning w/ Concept LoRAs",
"author": "InvokeAI",
"description": "Fast text-to-video generation with Wan 2.2 T2V A14B + the Lightning LoRA pair. Distillation lets the model converge in 4 steps with CFG 1.0 (no negative branch), roughly 20x faster than the standard T2V workflow at the cost of some image quality. Pick the high-noise Lightning LoRA on 'Apply LoRA (High)' and the low-noise one on 'Apply LoRA (Low)' — the 'auto' target routes each to the right expert when the LoRAs are tagged. Defaults: 832x480, 81 frames @ 16 FPS (~5 s).",
"version": "1.0.0",
"contact": "",
"tags": "wan2.2, text to video, video, lightning, lora",
"notes": "Each InvokeAI install assigns its own internal model IDs, so the model fields are left blank -- select these in the form after loading:\n\nMain Model node:\n- Main Model: Wan 2.2 T2V A14B High Noise (Q4_K_M GGUF recommended for <=16 GB VRAM)\n- Transformer (Low Noise): Wan 2.2 T2V A14B Low Noise (Q4_K_M)\n- VAE: Wan 2.2 A14B VAE\n- T5 Encoder: Wan T5 Encoder (UMT5-XXL)\n\nLightning LoRAs (4-step distillation - use the T2V variant, NOT I2V):\n- Apply Lightning LoRA (High Noise): Wan 2.2 T2V Lightning High Noise (4-step)\n- Apply Lightning LoRA (Low Noise): Wan 2.2 T2V Lightning Low Noise (4-step)\n\nThis workflow also has two optional concept-LoRA loaders (routed via 'target' = high/low) -- leave them empty or select your own high/low-noise concept LoRA pair.\n\nThe Lightning LoRAs are A14B-specific and distinct for T2V vs I2V. If your LoRAs are untagged (no 'expert' field), use each LoRA loader's 'target' dropdown to route the high-noise LoRA to 'high' and the low-noise to 'low'. CFG=1.0 skips the negative-conditioning branch, so the Negative Prompt is ignored at runtime.",
"exposedFields": [
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "transformer_low_noise_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "component_source"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "vae_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "wan_t5_encoder_model"
},
{
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"fieldName": "prompt"
},
{
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"fieldName": "prompt"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "steps"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale_low_noise"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "width"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "height"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "num_frames"
},
{
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"fieldName": "fps"
}
],
"meta": {
"version": "4.0.0",
"category": "default"
},
"nodes": [
{
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"type": "invocation",
"data": {
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_model_loader",
"inputs": {
"model": {
"name": "model",
"label": "Transformer (High Noise)",
"description": ""
},
"transformer_low_noise_model": {
"name": "transformer_low_noise_model",
"label": "",
"description": ""
},
"vae_model": {
"name": "vae_model",
"label": "",
"description": ""
},
"wan_t5_encoder_model": {
"name": "wan_t5_encoder_model",
"label": "",
"description": ""
},
"component_source": {
"name": "component_source",
"label": "",
"description": ""
}
},
"dynamicInputTemplates": {},
"isOpen": true,
"isIntermediate": true,
"useCache": false
},
"position": {
"x": 0,
"y": 0
}
},
{
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"type": "invocation",
"data": {
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "Positive Prompt",
"notes": "",
"type": "wan_text_encoder",
"inputs": {
"prompt": {
"name": "prompt",
"label": "",
"description": "",
"value": ""
},
"wan_t5_encoder": {
"name": "wan_t5_encoder",
"label": "",
"description": ""
}
},
"dynamicInputTemplates": {},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 1400,
"y": 212
}
},
{
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"type": "invocation",
"data": {
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "Negative Prompt (unused at CFG=1.0)",
"notes": "",
"type": "wan_text_encoder",
"inputs": {
"prompt": {
"name": "prompt",
"label": "",
"description": "",
"value": " "
},
"wan_t5_encoder": {
"name": "wan_t5_encoder",
"label": "",
"description": ""
}
},
"dynamicInputTemplates": {},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 1400,
"y": 392
}
},
{
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"type": "invocation",
"data": {
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "rand_int",
"inputs": {
"low": {
"name": "low",
"label": "",
"description": "",
"value": 0
},
"high": {
"name": "high",
"label": "",
"description": "",
"value": 2147483647
}
},
"dynamicInputTemplates": {},
"isOpen": true,
"isIntermediate": true,
"useCache": false
},
"position": {
"x": 1400,
"y": 572
}
},
{
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"type": "invocation",
"data": {
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_video_denoise",
"inputs": {
"transformer": {
"name": "transformer",
"label": "",
"description": ""
},
"positive_conditioning": {
"name": "positive_conditioning",
"label": "",
"description": ""
},
"negative_conditioning": {
"name": "negative_conditioning",
"label": "",
"description": ""
},
"ref_image": {
"name": "ref_image",
"label": "",
"description": ""
},
"guidance_scale": {
"name": "guidance_scale",
"label": "CFG (High Noise)",
"description": "",
"value": 1
},
"guidance_scale_low_noise": {
"name": "guidance_scale_low_noise",
"label": "CFG (Low Noise)",
"description": "",
"value": 1
},
"width": {
"name": "width",
"label": "",
"description": "",
"value": 832
},
"height": {
"name": "height",
"label": "",
"description": "",
"value": 480
},
"num_frames": {
"name": "num_frames",
"label": "Frames",
"description": "",
"value": 81
},
"steps": {
"name": "steps",
"label": "",
"description": "",
"value": 4
},
"seed": {
"name": "seed",
"label": "",
"description": "",
"value": 0
}
},
"dynamicInputTemplates": {},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 1750,
"y": 230
}
},
{
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"type": "invocation",
"data": {
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_l2v",
"inputs": {
"board": {
"name": "board",
"label": "",
"description": ""
},
"metadata": {
"name": "metadata",
"label": "",
"description": ""
},
"latents": {
"name": "latents",
"label": "",
"description": ""
},
"vae": {
"name": "vae",
"label": "",
"description": ""
},
"fps": {
"name": "fps",
"label": "FPS",
"description": "",
"value": 16
}
},
"dynamicInputTemplates": {},
"isOpen": true,
"isIntermediate": false,
"useCache": true
},
"position": {
"x": 2100,
"y": 33
}
},
{
"id": "5d18cf53-905d-4eb6-afd4-6c19f790b374",
"type": "invocation",
"data": {
"id": "5d18cf53-905d-4eb6-afd4-6c19f790b374",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_lora_loader",
"inputs": {
"lora": {
"name": "lora",
"label": "Lightning LoRA (High Noise)",
"description": ""
},
"weight": {
"name": "weight",
"label": "",
"description": "",
"value": 1
},
"target": {
"name": "target",
"label": "",
"description": "",
"value": "high"
},
"transformer": {
"name": "transformer",
"label": "",
"description": ""
}
},
"dynamicInputTemplates": {},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 350,
"y": 772
}
},
{
"id": "eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75",
"type": "invocation",
"data": {
"id": "eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_lora_loader",
"inputs": {
"lora": {
"name": "lora",
"label": "Lightning LoRA (Low Noise)",
"description": ""
},
"weight": {
"name": "weight",
"label": "",
"description": "",
"value": 1
},
"target": {
"name": "target",
"label": "",
"description": "",
"value": "low"
},
"transformer": {
"name": "transformer",
"label": "",
"description": ""
}
},
"dynamicInputTemplates": {},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 700,
"y": 772
}
},
{
"id": "f37bd1f0-f0d8-4105-8e07-5ec484298d36",
"type": "invocation",
"data": {
"id": "f37bd1f0-f0d8-4105-8e07-5ec484298d36",
"type": "wan_lora_collection_loader",
"version": "1.0.1",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"nodePack": "invokeai",
"inputs": {
"loras": {
"name": "loras",
"label": "",
"description": "",
"value": []
},
"transformer": {
"name": "transformer",
"label": "",
"description": ""
}
},
"dynamicInputTemplates": {}
},
"position": {
"x": 1053.289904257177,
"y": 770.7485394743496
}
}
],
"edges": [
{
"id": "edge-loader-t5-pos",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-t5-neg",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-vae-l2v",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "vae",
"targetHandle": "vae"
},
{
"id": "edge-pos-cond-denoise",
"type": "default",
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "positive_conditioning"
},
{
"id": "edge-neg-cond-denoise",
"type": "default",
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "negative_conditioning"
},
{
"id": "edge-rand-seed-denoise",
"type": "default",
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "value",
"targetHandle": "seed"
},
{
"id": "edge-denoise-latents-l2v",
"type": "default",
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "latents",
"targetHandle": "latents"
},
{
"id": "reactflow__edge-1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7etransformer-5d18cf53-905d-4eb6-afd4-6c19f790b374transformer",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "5d18cf53-905d-4eb6-afd4-6c19f790b374",
"sourceHandle": "transformer",
"targetHandle": "transformer"
},
{
"id": "reactflow__edge-5d18cf53-905d-4eb6-afd4-6c19f790b374transformer-eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75transformer",
"type": "default",
"source": "5d18cf53-905d-4eb6-afd4-6c19f790b374",
"target": "eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75",
"sourceHandle": "transformer",
"targetHandle": "transformer"
},
{
"id": "reactflow__edge-eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75transformer-f37bd1f0-f0d8-4105-8e07-5ec484298d36transformer",
"type": "default",
"source": "eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75",
"target": "f37bd1f0-f0d8-4105-8e07-5ec484298d36",
"sourceHandle": "transformer",
"targetHandle": "transformer"
},
{
"id": "reactflow__edge-f37bd1f0-f0d8-4105-8e07-5ec484298d36transformer-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabtransformer",
"type": "default",
"source": "f37bd1f0-f0d8-4105-8e07-5ec484298d36",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "transformer",
"targetHandle": "transformer"
}
],
"form": {
"elements": {
"root": {
"id": "root",
"data": {
"layout": "column",
"children": [
"heading-XDPaQofdEE",
"node-field-KcT0glAFDK",
"container-i8V1oJeQCP",
"container-mUeBcw2tYf",
"heading-wuPtY8B4vR",
"container-AkpUH1WtID",
"heading-TwsoE3wQi9",
"container-0u7wyxEdNa",
"heading-vSOD5q5gvL",
"node-field-EbS4ZMqDPZ",
"heading-10PuWMKJg9",
"container-C0z6ru33to"
]
},
"type": "container"
},
"node-field-zQA9XuRZRW": {
"id": "node-field-zQA9XuRZRW",
"parentId": "container-mUeBcw2tYf",
"data": {
"fieldIdentifier": {
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"fieldName": "fps"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-ok9GStueKC": {
"id": "node-field-ok9GStueKC",
"parentId": "container-mUeBcw2tYf",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "num_frames"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-58JRDH0dfF": {
"id": "node-field-58JRDH0dfF",
"parentId": "container-i8V1oJeQCP",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "height"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-jIfJ3RUDyT": {
"id": "node-field-jIfJ3RUDyT",
"parentId": "container-i8V1oJeQCP",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "width"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-QDqQ4XOKvg": {
"id": "node-field-QDqQ4XOKvg",
"parentId": "container-7g1Xedgg8T",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale_low_noise"
},
"showDescription": false,
"settings": {
"type": "float-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-PdU0Sz2m5z": {
"id": "node-field-PdU0Sz2m5z",
"parentId": "container-y0w7TfJC7e",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale"
},
"showDescription": false,
"settings": {
"type": "float-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-UiQ7MaseDR": {
"id": "node-field-UiQ7MaseDR",
"parentId": "container-mUeBcw2tYf",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "steps"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-KcT0glAFDK": {
"id": "node-field-KcT0glAFDK",
"parentId": "root",
"data": {
"fieldIdentifier": {
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"fieldName": "prompt"
},
"showDescription": false,
"settings": {
"type": "string-field-config",
"component": "textarea"
}
},
"type": "node-field"
},
"node-field-N9mexEKkYg": {
"id": "node-field-N9mexEKkYg",
"parentId": "container-C0z6ru33to",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "wan_t5_encoder_model"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-2u9lAbMBnD": {
"id": "node-field-2u9lAbMBnD",
"parentId": "container-C0z6ru33to",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "vae_model"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-CTGqrO3x8o": {
"id": "node-field-CTGqrO3x8o",
"parentId": "container-C0z6ru33to",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "component_source"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-7G9srQT2Uw": {
"id": "node-field-7G9srQT2Uw",
"parentId": "container-7g1Xedgg8T",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "transformer_low_noise_model"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-Q345WyNzc4": {
"id": "node-field-Q345WyNzc4",
"parentId": "container-y0w7TfJC7e",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "model"
},
"showDescription": false
},
"type": "node-field"
},
"container-mUeBcw2tYf": {
"id": "container-mUeBcw2tYf",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"node-field-UiQ7MaseDR",
"node-field-ok9GStueKC",
"node-field-zQA9XuRZRW"
]
},
"type": "container"
},
"heading-XDPaQofdEE": {
"id": "heading-XDPaQofdEE",
"parentId": "root",
"data": {
"content": "Generation"
},
"type": "heading"
},
"container-i8V1oJeQCP": {
"id": "container-i8V1oJeQCP",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"node-field-jIfJ3RUDyT",
"node-field-58JRDH0dfF"
]
},
"type": "container"
},
"heading-wuPtY8B4vR": {
"id": "heading-wuPtY8B4vR",
"parentId": "root",
"data": {
"content": "Model"
},
"type": "heading"
},
"container-AkpUH1WtID": {
"id": "container-AkpUH1WtID",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"container-y0w7TfJC7e",
"container-7g1Xedgg8T"
]
},
"type": "container"
},
"container-y0w7TfJC7e": {
"id": "container-y0w7TfJC7e",
"parentId": "container-AkpUH1WtID",
"data": {
"layout": "column",
"children": [
"node-field-Q345WyNzc4",
"node-field-PdU0Sz2m5z"
]
},
"type": "container"
},
"container-7g1Xedgg8T": {
"id": "container-7g1Xedgg8T",
"parentId": "container-AkpUH1WtID",
"data": {
"layout": "column",
"children": [
"node-field-7G9srQT2Uw",
"node-field-QDqQ4XOKvg"
]
},
"type": "container"
},
"container-C0z6ru33to": {
"id": "container-C0z6ru33to",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"node-field-N9mexEKkYg",
"node-field-2u9lAbMBnD",
"node-field-CTGqrO3x8o"
]
},
"type": "container"
},
"heading-10PuWMKJg9": {
"id": "heading-10PuWMKJg9",
"parentId": "root",
"data": {
"content": "Advanced"
},
"type": "heading"
},
"container-0u7wyxEdNa": {
"id": "container-0u7wyxEdNa",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"container-Um9gcFv0lo",
"container-4owWd0thWk"
]
},
"type": "container"
},
"container-Um9gcFv0lo": {
"id": "container-Um9gcFv0lo",
"parentId": "container-0u7wyxEdNa",
"data": {
"layout": "row",
"children": [
"container-lF8o60KuVb"
]
},
"type": "container"
},
"container-4owWd0thWk": {
"id": "container-4owWd0thWk",
"parentId": "container-0u7wyxEdNa",
"data": {
"layout": "row",
"children": [
"container-rvE0oy63qL"
]
},
"type": "container"
},
"container-lF8o60KuVb": {
"id": "container-lF8o60KuVb",
"parentId": "container-Um9gcFv0lo",
"data": {
"layout": "column",
"children": [
"node-field-tpqKlLmMtv"
]
},
"type": "container"
},
"container-rvE0oy63qL": {
"id": "container-rvE0oy63qL",
"parentId": "container-4owWd0thWk",
"data": {
"layout": "column",
"children": [
"node-field-yuWwWH1ucK"
]
},
"type": "container"
},
"heading-TwsoE3wQi9": {
"id": "heading-TwsoE3wQi9",
"parentId": "root",
"data": {
"content": "Lightning LoRAs"
},
"type": "heading"
},
"heading-vSOD5q5gvL": {
"id": "heading-vSOD5q5gvL",
"parentId": "root",
"data": {
"content": "Concept LoRAs"
},
"type": "heading"
},
"node-field-tpqKlLmMtv": {
"id": "node-field-tpqKlLmMtv",
"parentId": "container-lF8o60KuVb",
"data": {
"fieldIdentifier": {
"nodeId": "5d18cf53-905d-4eb6-afd4-6c19f790b374",
"fieldName": "lora"
},
"showDescription": true
},
"type": "node-field"
},
"node-field-yuWwWH1ucK": {
"id": "node-field-yuWwWH1ucK",
"parentId": "container-rvE0oy63qL",
"data": {
"fieldIdentifier": {
"nodeId": "eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75",
"fieldName": "lora"
},
"showDescription": true
},
"type": "node-field"
},
"node-field-EbS4ZMqDPZ": {
"id": "node-field-EbS4ZMqDPZ",
"type": "node-field",
"parentId": "root",
"data": {
"fieldIdentifier": {
"nodeId": "f37bd1f0-f0d8-4105-8e07-5ec484298d36",
"fieldName": "loras"
},
"showDescription": true
}
}
},
"rootElementId": "root"
},
"id": "default_wan22_t2v_lightning_concept"
}
@@ -0,0 +1,986 @@
{
"name": "Text to Video - Wan 2.2 Lightning",
"author": "InvokeAI",
"description": "Fast text-to-video generation with Wan 2.2 T2V A14B + the Lightning LoRA pair. Distillation lets the model converge in 4 steps with CFG 1.0 (no negative branch), roughly 20x faster than the standard T2V workflow at the cost of some image quality. Pick the high-noise Lightning LoRA on 'Apply LoRA (High)' and the low-noise one on 'Apply LoRA (Low)' — the 'auto' target routes each to the right expert when the LoRAs are tagged. Defaults: 832x480, 81 frames @ 16 FPS (~5 s).",
"version": "1.0.0",
"contact": "",
"tags": "wan2.2, text to video, video, lightning, lora",
"notes": "Each InvokeAI install assigns its own internal model IDs, so the model fields are left blank -- select these in the form after loading:\n\nMain Model node:\n- Main Model: Wan 2.2 T2V A14B High Noise (Q4_K_M GGUF recommended for <=16 GB VRAM)\n- Transformer (Low Noise): Wan 2.2 T2V A14B Low Noise (Q4_K_M)\n- VAE: Wan 2.2 A14B VAE\n- T5 Encoder: Wan T5 Encoder (UMT5-XXL)\n\nLightning LoRAs (4-step distillation - use the T2V variant, NOT I2V):\n- Apply Lightning LoRA (High Noise): Wan 2.2 T2V Lightning High Noise (4-step)\n- Apply Lightning LoRA (Low Noise): Wan 2.2 T2V Lightning Low Noise (4-step)\n\nThe Lightning LoRAs are A14B-specific and distinct for T2V vs I2V. If your LoRAs are untagged (no 'expert' field), use each LoRA loader's 'target' dropdown to route the first to 'high' and the second to 'low'. CFG=1.0 skips the negative-conditioning branch, so the Negative Prompt is ignored at runtime.",
"exposedFields": [
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "transformer_low_noise_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "component_source"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "vae_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "wan_t5_encoder_model"
},
{
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
"fieldName": "lora"
},
{
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
"fieldName": "weight"
},
{
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
"fieldName": "target"
},
{
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
"fieldName": "lora"
},
{
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
"fieldName": "weight"
},
{
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
"fieldName": "target"
},
{
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"fieldName": "prompt"
},
{
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"fieldName": "prompt"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "steps"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale_low_noise"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "width"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "height"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "num_frames"
},
{
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"fieldName": "fps"
}
],
"meta": {
"version": "4.0.0",
"category": "default"
},
"nodes": [
{
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"type": "invocation",
"data": {
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_model_loader",
"inputs": {
"model": {
"name": "model",
"label": "Transformer (High Noise)",
"description": "",
"value": null
},
"transformer_low_noise_model": {
"name": "transformer_low_noise_model",
"label": "",
"description": "",
"value": null
},
"vae_model": {
"name": "vae_model",
"label": "",
"description": "",
"value": null
},
"wan_t5_encoder_model": {
"name": "wan_t5_encoder_model",
"label": "",
"description": "",
"value": null
},
"component_source": {
"name": "component_source",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": false
},
"position": {
"x": 0,
"y": 0
}
},
{
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"type": "invocation",
"data": {
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "Positive Prompt",
"notes": "",
"type": "wan_text_encoder",
"inputs": {
"prompt": {
"name": "prompt",
"label": "",
"description": "",
"value": ""
},
"wan_t5_encoder": {
"name": "wan_t5_encoder",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 700,
"y": 212
}
},
{
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"type": "invocation",
"data": {
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "Negative Prompt (unused at CFG=1.0)",
"notes": "",
"type": "wan_text_encoder",
"inputs": {
"prompt": {
"name": "prompt",
"label": "",
"description": "",
"value": " "
},
"wan_t5_encoder": {
"name": "wan_t5_encoder",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 700,
"y": 392
}
},
{
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"type": "invocation",
"data": {
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "rand_int",
"inputs": {
"low": {
"name": "low",
"label": "",
"description": "",
"value": 0
},
"high": {
"name": "high",
"label": "",
"description": "",
"value": 2147483647
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": false
},
"position": {
"x": 700,
"y": 572
}
},
{
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"type": "invocation",
"data": {
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_video_denoise",
"inputs": {
"transformer": {
"name": "transformer",
"label": "",
"description": ""
},
"positive_conditioning": {
"name": "positive_conditioning",
"label": "",
"description": ""
},
"negative_conditioning": {
"name": "negative_conditioning",
"label": "",
"description": ""
},
"ref_image": {
"name": "ref_image",
"label": "",
"description": ""
},
"guidance_scale": {
"name": "guidance_scale",
"label": "CFG (High Noise)",
"description": "",
"value": 1
},
"guidance_scale_low_noise": {
"name": "guidance_scale_low_noise",
"label": "CFG (Low Noise)",
"description": "",
"value": 1
},
"width": {
"name": "width",
"label": "",
"description": "",
"value": 832
},
"height": {
"name": "height",
"label": "",
"description": "",
"value": 480
},
"num_frames": {
"name": "num_frames",
"label": "Frames",
"description": "",
"value": 81
},
"steps": {
"name": "steps",
"label": "",
"description": "",
"value": 4
},
"seed": {
"name": "seed",
"label": "",
"description": "",
"value": 0
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 1050,
"y": 230
}
},
{
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"type": "invocation",
"data": {
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_l2v",
"inputs": {
"board": {
"name": "board",
"label": "",
"description": ""
},
"metadata": {
"name": "metadata",
"label": "",
"description": ""
},
"latents": {
"name": "latents",
"label": "",
"description": ""
},
"vae": {
"name": "vae",
"label": "",
"description": ""
},
"fps": {
"name": "fps",
"label": "FPS",
"description": "",
"value": 16
}
},
"isOpen": true,
"isIntermediate": false,
"useCache": true
},
"position": {
"x": 1400,
"y": 33
}
},
{
"id": "291abec8-b996-4c7b-959c-360f56a3ed60",
"type": "invocation",
"data": {
"id": "291abec8-b996-4c7b-959c-360f56a3ed60",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "Apply Lightning LoRA (High Noise)",
"notes": "",
"type": "wan_lora_loader",
"inputs": {
"lora": {
"name": "lora",
"label": "Lightning LoRA (High Noise)",
"description": "",
"value": null
},
"weight": {
"name": "weight",
"label": "",
"description": "",
"value": 1
},
"target": {
"name": "target",
"label": "",
"description": "",
"value": "high"
},
"transformer": {
"name": "transformer",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 350,
"y": 772
}
},
{
"id": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
"type": "invocation",
"data": {
"id": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_lora_loader",
"inputs": {
"lora": {
"name": "lora",
"label": "Lightning LoRA (Low Noise)",
"description": "",
"value": null
},
"weight": {
"name": "weight",
"label": "",
"description": "",
"value": 1
},
"target": {
"name": "target",
"label": "",
"description": "",
"value": "low"
},
"transformer": {
"name": "transformer",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 700,
"y": 772
}
}
],
"edges": [
{
"id": "edge-loader-t5-pos",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-t5-neg",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-vae-l2v",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "vae",
"targetHandle": "vae"
},
{
"id": "edge-pos-cond-denoise",
"type": "default",
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "positive_conditioning"
},
{
"id": "edge-neg-cond-denoise",
"type": "default",
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "negative_conditioning"
},
{
"id": "edge-rand-seed-denoise",
"type": "default",
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "value",
"targetHandle": "seed"
},
{
"id": "edge-denoise-latents-l2v",
"type": "default",
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "latents",
"targetHandle": "latents"
},
{
"id": "reactflow__edge-1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7etransformer-291abec8-b996-4c7b-959c-360f56a3ed60transformer",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "291abec8-b996-4c7b-959c-360f56a3ed60",
"sourceHandle": "transformer",
"targetHandle": "transformer"
},
{
"id": "reactflow__edge-291abec8-b996-4c7b-959c-360f56a3ed60transformer-ae8f115d-a3b8-4f76-b280-7313366a94adtransformer",
"type": "default",
"source": "291abec8-b996-4c7b-959c-360f56a3ed60",
"target": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
"sourceHandle": "transformer",
"targetHandle": "transformer"
},
{
"id": "reactflow__edge-ae8f115d-a3b8-4f76-b280-7313366a94adtransformer-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabtransformer",
"type": "default",
"source": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "transformer",
"targetHandle": "transformer"
}
],
"form": {
"elements": {
"root": {
"id": "root",
"data": {
"layout": "column",
"children": [
"heading-XDPaQofdEE",
"node-field-KcT0glAFDK",
"container-i8V1oJeQCP",
"container-mUeBcw2tYf",
"heading-wuPtY8B4vR",
"container-AkpUH1WtID",
"heading-TwsoE3wQi9",
"container-0u7wyxEdNa",
"heading-10PuWMKJg9",
"container-C0z6ru33to"
]
},
"type": "container"
},
"node-field-zQA9XuRZRW": {
"id": "node-field-zQA9XuRZRW",
"parentId": "container-mUeBcw2tYf",
"data": {
"fieldIdentifier": {
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"fieldName": "fps"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-ok9GStueKC": {
"id": "node-field-ok9GStueKC",
"parentId": "container-mUeBcw2tYf",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "num_frames"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-58JRDH0dfF": {
"id": "node-field-58JRDH0dfF",
"parentId": "container-i8V1oJeQCP",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "height"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-jIfJ3RUDyT": {
"id": "node-field-jIfJ3RUDyT",
"parentId": "container-i8V1oJeQCP",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "width"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-QDqQ4XOKvg": {
"id": "node-field-QDqQ4XOKvg",
"parentId": "container-7g1Xedgg8T",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale_low_noise"
},
"showDescription": false,
"settings": {
"type": "float-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-PdU0Sz2m5z": {
"id": "node-field-PdU0Sz2m5z",
"parentId": "container-y0w7TfJC7e",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale"
},
"showDescription": false,
"settings": {
"type": "float-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-UiQ7MaseDR": {
"id": "node-field-UiQ7MaseDR",
"parentId": "container-mUeBcw2tYf",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "steps"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-KcT0glAFDK": {
"id": "node-field-KcT0glAFDK",
"parentId": "root",
"data": {
"fieldIdentifier": {
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"fieldName": "prompt"
},
"showDescription": false,
"settings": {
"type": "string-field-config",
"component": "textarea"
}
},
"type": "node-field"
},
"node-field-N9mexEKkYg": {
"id": "node-field-N9mexEKkYg",
"parentId": "container-C0z6ru33to",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "wan_t5_encoder_model"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-2u9lAbMBnD": {
"id": "node-field-2u9lAbMBnD",
"parentId": "container-C0z6ru33to",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "vae_model"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-CTGqrO3x8o": {
"id": "node-field-CTGqrO3x8o",
"parentId": "container-C0z6ru33to",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "component_source"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-7G9srQT2Uw": {
"id": "node-field-7G9srQT2Uw",
"parentId": "container-7g1Xedgg8T",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "transformer_low_noise_model"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-Q345WyNzc4": {
"id": "node-field-Q345WyNzc4",
"parentId": "container-y0w7TfJC7e",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "model"
},
"showDescription": false
},
"type": "node-field"
},
"container-mUeBcw2tYf": {
"id": "container-mUeBcw2tYf",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"node-field-UiQ7MaseDR",
"node-field-ok9GStueKC",
"node-field-zQA9XuRZRW"
]
},
"type": "container"
},
"heading-XDPaQofdEE": {
"id": "heading-XDPaQofdEE",
"parentId": "root",
"data": {
"content": "Generation"
},
"type": "heading"
},
"container-i8V1oJeQCP": {
"id": "container-i8V1oJeQCP",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"node-field-jIfJ3RUDyT",
"node-field-58JRDH0dfF"
]
},
"type": "container"
},
"heading-wuPtY8B4vR": {
"id": "heading-wuPtY8B4vR",
"parentId": "root",
"data": {
"content": "Model"
},
"type": "heading"
},
"container-AkpUH1WtID": {
"id": "container-AkpUH1WtID",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"container-y0w7TfJC7e",
"container-7g1Xedgg8T"
]
},
"type": "container"
},
"container-y0w7TfJC7e": {
"id": "container-y0w7TfJC7e",
"parentId": "container-AkpUH1WtID",
"data": {
"layout": "column",
"children": [
"node-field-Q345WyNzc4",
"node-field-PdU0Sz2m5z"
]
},
"type": "container"
},
"container-7g1Xedgg8T": {
"id": "container-7g1Xedgg8T",
"parentId": "container-AkpUH1WtID",
"data": {
"layout": "column",
"children": [
"node-field-7G9srQT2Uw",
"node-field-QDqQ4XOKvg"
]
},
"type": "container"
},
"container-C0z6ru33to": {
"id": "container-C0z6ru33to",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"node-field-N9mexEKkYg",
"node-field-2u9lAbMBnD",
"node-field-CTGqrO3x8o"
]
},
"type": "container"
},
"heading-10PuWMKJg9": {
"id": "heading-10PuWMKJg9",
"parentId": "root",
"data": {
"content": "Advanced"
},
"type": "heading"
},
"container-0u7wyxEdNa": {
"id": "container-0u7wyxEdNa",
"parentId": "root",
"data": {
"layout": "row",
"children": [
"container-Um9gcFv0lo",
"container-4owWd0thWk"
]
},
"type": "container"
},
"container-Um9gcFv0lo": {
"id": "container-Um9gcFv0lo",
"parentId": "container-0u7wyxEdNa",
"data": {
"layout": "row",
"children": [
"container-lF8o60KuVb"
]
},
"type": "container"
},
"container-4owWd0thWk": {
"id": "container-4owWd0thWk",
"parentId": "container-0u7wyxEdNa",
"data": {
"layout": "row",
"children": [
"container-rvE0oy63qL"
]
},
"type": "container"
},
"node-field-a8nZxgz9u9": {
"id": "node-field-a8nZxgz9u9",
"parentId": "container-lF8o60KuVb",
"data": {
"fieldIdentifier": {
"nodeId": "291abec8-b996-4c7b-959c-360f56a3ed60",
"fieldName": "lora"
},
"showDescription": true
},
"type": "node-field"
},
"node-field-NwR9yOugEz": {
"id": "node-field-NwR9yOugEz",
"parentId": "container-rvE0oy63qL",
"data": {
"fieldIdentifier": {
"nodeId": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
"fieldName": "lora"
},
"showDescription": true
},
"type": "node-field"
},
"container-lF8o60KuVb": {
"id": "container-lF8o60KuVb",
"parentId": "container-Um9gcFv0lo",
"data": {
"layout": "column",
"children": [
"node-field-a8nZxgz9u9",
"node-field-0al3LtpcGU"
]
},
"type": "container"
},
"container-rvE0oy63qL": {
"id": "container-rvE0oy63qL",
"parentId": "container-4owWd0thWk",
"data": {
"layout": "column",
"children": [
"node-field-NwR9yOugEz",
"node-field-lgLAjS4nks"
]
},
"type": "container"
},
"node-field-0al3LtpcGU": {
"id": "node-field-0al3LtpcGU",
"parentId": "container-lF8o60KuVb",
"data": {
"fieldIdentifier": {
"nodeId": "291abec8-b996-4c7b-959c-360f56a3ed60",
"fieldName": "weight"
},
"showDescription": true,
"settings": {
"type": "float-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-lgLAjS4nks": {
"id": "node-field-lgLAjS4nks",
"parentId": "container-rvE0oy63qL",
"data": {
"fieldIdentifier": {
"nodeId": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
"fieldName": "weight"
},
"showDescription": true,
"settings": {
"type": "float-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"heading-TwsoE3wQi9": {
"id": "heading-TwsoE3wQi9",
"parentId": "root",
"data": {
"content": "Lightning LoRAs"
},
"type": "heading"
}
},
"rootElementId": "root"
},
"id": "default_wan22_t2v_lightning"
}
@@ -0,0 +1,869 @@
{
"name": "Text to Video - Wan 2.2 TI2V-5B (Low Quality)",
"author": "InvokeAI",
"description": "Text-to-video generation with Wan 2.2 TI2V-5B. Produces an MP4 of `num_frames` frames at `fps` frames-per-second (defaults: 81 frames @ 16 FPS = ~5 s, 720p ~1248x704). The output lands in the gallery alongside images and plays inline in the viewer.",
"version": "1.0.0",
"contact": "",
"tags": "wan2.2, text to video, video",
"notes": "This workflow needs the Wan 2.2 TI2V-5B model. Each InvokeAI install assigns its own internal model IDs, so the model fields are left blank -- select these in the form after loading:\n\nMain Model node:\n- Main Model: Wan 2.2 TI2V-5B (Q4_K_M GGUF recommended for <=16 GB VRAM)\n- Transformer (Low Noise): leave empty (TI2V-5B is a single-expert model)\n- VAE: Wan 2.2 TI2V-5B VAE\n- T5 Encoder: Wan T5 Encoder (UMT5-XXL)\n\nDefault output is 720p (~1248x704); reduce the \"Ideal Dimensions\" preset to 480p for less VRAM.",
"exposedFields": [
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "transformer_low_noise_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "component_source"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "vae_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "wan_t5_encoder_model"
},
{
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"fieldName": "prompt"
},
{
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"fieldName": "prompt"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "steps"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale_low_noise"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "width"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "height"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "num_frames"
},
{
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"fieldName": "fps"
}
],
"meta": {
"version": "4.0.0",
"category": "default"
},
"nodes": [
{
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"type": "invocation",
"data": {
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_model_loader",
"inputs": {
"model": {
"name": "model",
"label": "",
"description": "",
"value": null
},
"transformer_low_noise_model": {
"name": "transformer_low_noise_model",
"label": "",
"description": ""
},
"vae_model": {
"name": "vae_model",
"label": "",
"description": "",
"value": null
},
"wan_t5_encoder_model": {
"name": "wan_t5_encoder_model",
"label": "",
"description": "",
"value": null
},
"component_source": {
"name": "component_source",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": false
},
"position": {
"x": 200,
"y": 0
}
},
{
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"type": "invocation",
"data": {
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "Positive Prompt",
"notes": "",
"type": "wan_text_encoder",
"inputs": {
"prompt": {
"name": "prompt",
"label": "Positive Prompt",
"description": "",
"value": ""
},
"wan_t5_encoder": {
"name": "wan_t5_encoder",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 700,
"y": -200
}
},
{
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"type": "invocation",
"data": {
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "Negative Prompt",
"notes": "",
"type": "wan_text_encoder",
"inputs": {
"prompt": {
"name": "prompt",
"label": "Negative Prompt",
"description": "",
"value": ""
},
"wan_t5_encoder": {
"name": "wan_t5_encoder",
"label": "",
"description": ""
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 683.3585813389434,
"y": 19.353124950263965
}
},
{
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"type": "invocation",
"data": {
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"version": "1.0.1",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "rand_int",
"inputs": {
"low": {
"name": "low",
"label": "",
"description": "",
"value": 0
},
"high": {
"name": "high",
"label": "",
"description": "",
"value": 2147483647
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": false
},
"position": {
"x": 382.5329363121502,
"y": 709.7864089212082
}
},
{
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"type": "invocation",
"data": {
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_video_denoise",
"inputs": {
"transformer": {
"name": "transformer",
"label": "",
"description": ""
},
"positive_conditioning": {
"name": "positive_conditioning",
"label": "",
"description": ""
},
"negative_conditioning": {
"name": "negative_conditioning",
"label": "",
"description": ""
},
"ref_image": {
"name": "ref_image",
"label": "",
"description": ""
},
"guidance_scale": {
"name": "guidance_scale",
"label": "CFG",
"description": "",
"value": 5
},
"guidance_scale_low_noise": {
"name": "guidance_scale_low_noise",
"label": "CFG (Low)",
"description": "",
"value": 4
},
"width": {
"name": "width",
"label": "",
"description": "",
"value": 720
},
"height": {
"name": "height",
"label": "",
"description": "",
"value": 480
},
"num_frames": {
"name": "num_frames",
"label": "Frames",
"description": "",
"value": 81
},
"steps": {
"name": "steps",
"label": "",
"description": "",
"value": 20
},
"seed": {
"name": "seed",
"label": "",
"description": "",
"value": 0
}
},
"isOpen": true,
"isIntermediate": true,
"useCache": true
},
"position": {
"x": 1165.285565516453,
"y": -39.7591269778113
}
},
{
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"type": "invocation",
"data": {
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"version": "1.0.0",
"nodePack": "invokeai",
"label": "",
"notes": "",
"type": "wan_l2v",
"inputs": {
"board": {
"name": "board",
"label": "",
"description": ""
},
"metadata": {
"name": "metadata",
"label": "",
"description": ""
},
"latents": {
"name": "latents",
"label": "",
"description": ""
},
"vae": {
"name": "vae",
"label": "",
"description": ""
},
"fps": {
"name": "fps",
"label": "FPS",
"description": "",
"value": 16
}
},
"isOpen": true,
"isIntermediate": false,
"useCache": true
},
"position": {
"x": 1531.0370931112793,
"y": -63.90613171839515
}
},
{
"id": "0cc328ea-b3bf-4ac1-98f3-844de0e072c8",
"type": "invocation",
"data": {
"id": "0cc328ea-b3bf-4ac1-98f3-844de0e072c8",
"type": "integer",
"version": "1.0.1",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"nodePack": "invokeai",
"inputs": {
"value": {
"name": "value",
"label": "Width",
"description": "Width of the video. Will be adjusted to meet the constraints of the selected target resolution.",
"value": 832
}
}
},
"position": {
"x": 370.6719737658655,
"y": 436.76341584866606
}
},
{
"id": "e20a326b-a312-4cc4-89cf-cfc13887cd4a",
"type": "invocation",
"data": {
"id": "e20a326b-a312-4cc4-89cf-cfc13887cd4a",
"type": "integer",
"version": "1.0.1",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"nodePack": "invokeai",
"inputs": {
"value": {
"name": "value",
"label": "Height",
"description": "Height of the video. Will be adjusted to meet the constraints of the selected target resolution.",
"value": 480
}
}
},
"position": {
"x": 373.58106252303446,
"y": 567.1240792266004
}
},
{
"id": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
"type": "invocation",
"data": {
"id": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
"type": "wan_ti2v_ideal_dimensions",
"version": "1.0.0",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"nodePack": "invokeai",
"inputs": {
"width": {
"name": "width",
"label": "",
"description": "",
"value": 1024
},
"height": {
"name": "height",
"label": "",
"description": "",
"value": 480
},
"target_resolution": {
"name": "target_resolution",
"label": "",
"description": "",
"value": "720p"
},
"rounding": {
"name": "rounding",
"label": "",
"description": "",
"value": "nearest"
}
}
},
"position": {
"x": 773.8083794848407,
"y": 420.1537431373451
}
}
],
"edges": [
{
"id": "edge-loader-transformer-denoise",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "transformer",
"targetHandle": "transformer"
},
{
"id": "edge-loader-t5-pos",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-t5-neg",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-vae-l2v",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "vae",
"targetHandle": "vae"
},
{
"id": "edge-pos-cond-denoise",
"type": "default",
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "positive_conditioning"
},
{
"id": "edge-neg-cond-denoise",
"type": "default",
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "negative_conditioning"
},
{
"id": "edge-rand-seed-denoise",
"type": "default",
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "value",
"targetHandle": "seed"
},
{
"id": "edge-denoise-latents-l2v",
"type": "default",
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "latents",
"targetHandle": "latents"
},
{
"id": "reactflow__edge-0cc328ea-b3bf-4ac1-98f3-844de0e072c8value-0f05c4e0-f92f-4b23-9a57-c69ccc50c2cdwidth",
"type": "default",
"source": "0cc328ea-b3bf-4ac1-98f3-844de0e072c8",
"target": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
"sourceHandle": "value",
"targetHandle": "width"
},
{
"id": "reactflow__edge-0f05c4e0-f92f-4b23-9a57-c69ccc50c2cdwidth-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabwidth",
"type": "default",
"source": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "width",
"targetHandle": "width"
},
{
"id": "reactflow__edge-e20a326b-a312-4cc4-89cf-cfc13887cd4avalue-0f05c4e0-f92f-4b23-9a57-c69ccc50c2cdheight",
"type": "default",
"source": "e20a326b-a312-4cc4-89cf-cfc13887cd4a",
"target": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
"sourceHandle": "value",
"targetHandle": "height"
},
{
"id": "reactflow__edge-0f05c4e0-f92f-4b23-9a57-c69ccc50c2cdheight-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabheight",
"type": "default",
"source": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "height",
"targetHandle": "height"
}
],
"form": {
"elements": {
"root": {
"id": "root",
"data": {
"layout": "column",
"children": [
"heading-NVsTvi64mE",
"node-field-LCSk79Cykw",
"node-field-8eaiK9h1E6",
"node-field-z8AjtazZ1C",
"container-yTCxecjcVW",
"container-tL8WPBUtAl",
"container-WfDQ0EqgcT",
"divider-dWeIfMmqLs",
"heading-4KzzU0571E",
"container-ZPG1DkfsBF",
"container-LtEu7q4UHX"
]
},
"type": "container"
},
"node-field-mz64Xfk59R": {
"id": "node-field-mz64Xfk59R",
"parentId": "container-tL8WPBUtAl",
"data": {
"fieldIdentifier": {
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"fieldName": "fps"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-dADhnNIONx": {
"id": "node-field-dADhnNIONx",
"parentId": "container-tL8WPBUtAl",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "num_frames"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-GM02XkVswM": {
"id": "node-field-GM02XkVswM",
"parentId": "container-WfDQ0EqgcT",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale"
},
"showDescription": false,
"settings": {
"type": "float-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-RN6aZuubTt": {
"id": "node-field-RN6aZuubTt",
"parentId": "container-WfDQ0EqgcT",
"data": {
"fieldIdentifier": {
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "steps"
},
"showDescription": false,
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
}
},
"type": "node-field"
},
"node-field-8eaiK9h1E6": {
"id": "node-field-8eaiK9h1E6",
"parentId": "root",
"data": {
"fieldIdentifier": {
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"fieldName": "prompt"
},
"showDescription": false,
"settings": {
"type": "string-field-config",
"component": "textarea"
}
},
"type": "node-field"
},
"node-field-LCSk79Cykw": {
"id": "node-field-LCSk79Cykw",
"parentId": "root",
"data": {
"fieldIdentifier": {
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"fieldName": "prompt"
},
"showDescription": false,
"settings": {
"type": "string-field-config",
"component": "textarea"
}
},
"type": "node-field"
},
"node-field-tmweV5koO8": {
"id": "node-field-tmweV5koO8",
"parentId": "container-YUjkEQOaZH",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "wan_t5_encoder_model"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-Ny6XBv2KCR": {
"id": "node-field-Ny6XBv2KCR",
"parentId": "container-LtEu7q4UHX",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "vae_model"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-zmxixZsriN": {
"id": "node-field-zmxixZsriN",
"parentId": "container-LtEu7q4UHX",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "component_source"
},
"showDescription": false
},
"type": "node-field"
},
"node-field-FnSN39ojJs": {
"id": "node-field-FnSN39ojJs",
"parentId": "container-YUjkEQOaZH",
"data": {
"fieldIdentifier": {
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "model"
},
"showDescription": false
},
"type": "node-field"
},
"heading-NVsTvi64mE": {
"id": "heading-NVsTvi64mE",
"parentId": "root",
"type": "heading",
"data": {
"content": "Generation"
}
},
"container-WfDQ0EqgcT": {
"id": "container-WfDQ0EqgcT",
"parentId": "root",
"type": "container",
"data": {
"layout": "row",
"children": [
"node-field-RN6aZuubTt",
"node-field-GM02XkVswM"
]
}
},
"container-bOWWWaSIsY": {
"id": "container-bOWWWaSIsY",
"parentId": "container-yTCxecjcVW",
"type": "container",
"data": {
"layout": "row",
"children": [
"node-field-41iwUAOdbd",
"node-field-C8Yua39ccI"
]
}
},
"container-yTCxecjcVW": {
"id": "container-yTCxecjcVW",
"parentId": "root",
"type": "container",
"data": {
"layout": "row",
"children": [
"container-bOWWWaSIsY"
]
}
},
"container-tL8WPBUtAl": {
"id": "container-tL8WPBUtAl",
"parentId": "root",
"type": "container",
"data": {
"layout": "row",
"children": [
"node-field-dADhnNIONx",
"node-field-mz64Xfk59R"
]
}
},
"heading-4KzzU0571E": {
"id": "heading-4KzzU0571E",
"parentId": "root",
"type": "heading",
"data": {
"content": "Model"
}
},
"divider-dWeIfMmqLs": {
"id": "divider-dWeIfMmqLs",
"parentId": "root",
"type": "divider"
},
"container-YUjkEQOaZH": {
"id": "container-YUjkEQOaZH",
"parentId": "container-ZPG1DkfsBF",
"type": "container",
"data": {
"layout": "row",
"children": [
"node-field-FnSN39ojJs",
"node-field-tmweV5koO8"
]
}
},
"container-ZPG1DkfsBF": {
"id": "container-ZPG1DkfsBF",
"parentId": "root",
"type": "container",
"data": {
"layout": "row",
"children": [
"container-YUjkEQOaZH"
]
}
},
"container-LtEu7q4UHX": {
"id": "container-LtEu7q4UHX",
"parentId": "root",
"type": "container",
"data": {
"layout": "row",
"children": [
"node-field-Ny6XBv2KCR",
"node-field-zmxixZsriN"
]
}
},
"node-field-z8AjtazZ1C": {
"id": "node-field-z8AjtazZ1C",
"type": "node-field",
"parentId": "root",
"data": {
"fieldIdentifier": {
"nodeId": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
"fieldName": "target_resolution"
},
"showDescription": true
}
},
"node-field-41iwUAOdbd": {
"id": "node-field-41iwUAOdbd",
"type": "node-field",
"parentId": "container-bOWWWaSIsY",
"data": {
"fieldIdentifier": {
"nodeId": "0cc328ea-b3bf-4ac1-98f3-844de0e072c8",
"fieldName": "value"
},
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
},
"showDescription": true
}
},
"node-field-C8Yua39ccI": {
"id": "node-field-C8Yua39ccI",
"type": "node-field",
"parentId": "container-bOWWWaSIsY",
"data": {
"fieldIdentifier": {
"nodeId": "e20a326b-a312-4cc4-89cf-cfc13887cd4a",
"fieldName": "value"
},
"settings": {
"type": "integer-field-config",
"showShuffle": false,
"component": "number-input"
},
"showDescription": true
}
}
},
"rootElementId": "root"
},
"id": "default_wan22_t2v_ti2v5b"
}
@@ -0,0 +1,307 @@
{
"id": "default_wan22_i2v_c2d5e1b3-3e4f-5b6c-af7d-8e9f0a1b2c3e",
"name": "Image to Image - Wan 2.2",
"author": "InvokeAI",
"description": "Image-to-image generation with Wan 2.2 I2V A14B. The reference image is VAE-encoded and concatenated to the noise latents each step (the I2V transformer has in_channels=36). Drop a reference image into the 'Reference Image' input, then invoke. Only the I2V A14B variant is supported — T2V and TI2V-5B don't consume reference images.",
"version": "1.0.0",
"contact": "",
"tags": "wan2.2, image to image",
"notes": "Prerequisite model downloads: a Wan 2.2 I2V A14B main (Diffusers or GGUF expert pair). For GGUF mains, also install the Component Source (Diffusers Wan I2V) OR the standalone Wan VAE + UMT5-XXL encoder. Wan 2.2 I2V was trained for video — at single-frame inference it tends to anchor strongly to the reference. Recommended settings: 30-40 steps and CFG 5-7 (or 4 steps and CFG 1 with the Wan I2V Lightning LoRA pair).",
"exposedFields": [
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "transformer_low_noise_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "component_source"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "vae_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "wan_t5_encoder_model"
},
{
"nodeId": "7a6edc2d-f38e-a0c1-e27a-8f9dcbb20fce",
"fieldName": "image"
},
{
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"fieldName": "prompt"
},
{
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"fieldName": "prompt"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "steps"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale_low_noise"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "width"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "height"
}
],
"meta": {
"version": "3.0.0",
"category": "default"
},
"nodes": [
{
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"type": "invocation",
"data": {
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"type": "wan_model_loader",
"version": "1.0.1",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": false,
"inputs": {
"model": { "name": "model", "label": "" },
"transformer_low_noise_model": { "name": "transformer_low_noise_model", "label": "" },
"vae_model": { "name": "vae_model", "label": "" },
"wan_t5_encoder_model": { "name": "wan_t5_encoder_model", "label": "" },
"component_source": { "name": "component_source", "label": "" }
}
},
"position": { "x": 200, "y": 0 }
},
{
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"type": "invocation",
"data": {
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"type": "wan_text_encoder",
"version": "1.0.0",
"label": "Positive Prompt",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"inputs": {
"prompt": { "name": "prompt", "label": "", "value": "" },
"wan_t5_encoder": { "name": "wan_t5_encoder", "label": "" }
}
},
"position": { "x": 700, "y": -200 }
},
{
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"type": "invocation",
"data": {
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"type": "wan_text_encoder",
"version": "1.0.0",
"label": "Negative Prompt",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"inputs": {
"prompt": { "name": "prompt", "label": "", "value": " " },
"wan_t5_encoder": { "name": "wan_t5_encoder", "label": "" }
}
},
"position": { "x": 700, "y": 100 }
},
{
"id": "7a6edc2d-f38e-a0c1-e27a-8f9dcbb20fce",
"type": "invocation",
"data": {
"id": "7a6edc2d-f38e-a0c1-e27a-8f9dcbb20fce",
"type": "wan_ref_image_encoder",
"version": "1.2.0",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"inputs": {
"image": { "name": "image", "label": "Reference Image" },
"end_image": { "name": "end_image", "label": "" },
"num_frames": { "name": "num_frames", "label": "", "value": 1 },
"vae": { "name": "vae", "label": "" },
"width": { "name": "width", "label": "", "value": 1024 },
"height": { "name": "height", "label": "", "value": 1024 }
}
},
"position": { "x": 700, "y": 300 }
},
{
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"type": "invocation",
"data": {
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"type": "rand_int",
"version": "1.0.1",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": false,
"inputs": {
"low": { "name": "low", "label": "", "value": 0 },
"high": { "name": "high", "label": "", "value": 2147483647 }
}
},
"position": { "x": 700, "y": 550 }
},
{
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"type": "invocation",
"data": {
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"type": "wan_denoise",
"version": "1.0.0",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"inputs": {
"transformer": { "name": "transformer", "label": "" },
"positive_conditioning": { "name": "positive_conditioning", "label": "" },
"negative_conditioning": { "name": "negative_conditioning", "label": "" },
"ref_image": { "name": "ref_image", "label": "" },
"latents": { "name": "latents", "label": "" },
"denoise_mask": { "name": "denoise_mask", "label": "" },
"denoising_start": { "name": "denoising_start", "label": "", "value": 0 },
"denoising_end": { "name": "denoising_end", "label": "", "value": 1 },
"add_noise": { "name": "add_noise", "label": "", "value": true },
"guidance_scale": { "name": "guidance_scale", "label": "CFG", "value": 5.0 },
"guidance_scale_low_noise": { "name": "guidance_scale_low_noise", "label": "CFG (Low)" },
"width": { "name": "width", "label": "", "value": 1024 },
"height": { "name": "height", "label": "", "value": 1024 },
"steps": { "name": "steps", "label": "", "value": 30 },
"seed": { "name": "seed", "label": "", "value": 0 }
}
},
"position": { "x": 1100, "y": -50 }
},
{
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"type": "invocation",
"data": {
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"type": "wan_l2i",
"version": "1.0.0",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": false,
"useCache": true,
"inputs": {
"board": { "name": "board", "label": "" },
"metadata": { "name": "metadata", "label": "" },
"latents": { "name": "latents", "label": "" },
"vae": { "name": "vae", "label": "" }
}
},
"position": { "x": 1550, "y": -50 }
}
],
"edges": [
{
"id": "edge-loader-transformer-denoise",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "transformer",
"targetHandle": "transformer"
},
{
"id": "edge-loader-t5-pos",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-t5-neg",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-vae-l2i",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "vae",
"targetHandle": "vae"
},
{
"id": "edge-loader-vae-refenc",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "7a6edc2d-f38e-a0c1-e27a-8f9dcbb20fce",
"sourceHandle": "vae",
"targetHandle": "vae"
},
{
"id": "edge-pos-cond-denoise",
"type": "default",
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "positive_conditioning"
},
{
"id": "edge-neg-cond-denoise",
"type": "default",
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "negative_conditioning"
},
{
"id": "edge-refenc-refimage-denoise",
"type": "default",
"source": "7a6edc2d-f38e-a0c1-e27a-8f9dcbb20fce",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "ref_image",
"targetHandle": "ref_image"
},
{
"id": "edge-rand-seed-denoise",
"type": "default",
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "value",
"targetHandle": "seed"
},
{
"id": "edge-denoise-latents-l2i",
"type": "default",
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "latents",
"targetHandle": "latents"
}
]
}
@@ -0,0 +1,264 @@
{
"id": "default_wan22_t2v_b1c4f0a2-2d3e-4a5b-9f6c-7d8e0a1b2c3d",
"name": "Text to Image - Wan 2.2",
"author": "InvokeAI",
"description": "Text-to-image generation with Wan 2.2 (T2V A14B or TI2V-5B). For A14B GGUFs, wire the second-expert transformer into 'Transformer (Low Noise)' and pick a Diffusers Wan as the Component Source (or use standalone VAE + UMT5-XXL encoder). TI2V-5B is a single-transformer model — leave the low-noise slot empty.",
"version": "1.0.0",
"contact": "",
"tags": "wan2.2, text to image",
"notes": "Prerequisite model downloads: a Wan 2.2 main model (Diffusers or GGUF). For GGUF mains, also install the Component Source (Diffusers Wan) OR the standalone Wan VAE + UMT5-XXL encoder. The Wan 2.2 starter bundle in the Model Manager pulls everything you need for T2V A14B Q4_K_M/Q8_0. Recommended settings: 30-40 steps and CFG 5-7 (or 4 steps and CFG 1 with the Wan Lightning LoRA pair).",
"exposedFields": [
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "transformer_low_noise_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "component_source"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "vae_model"
},
{
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"fieldName": "wan_t5_encoder_model"
},
{
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"fieldName": "prompt"
},
{
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"fieldName": "prompt"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "steps"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "guidance_scale_low_noise"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "width"
},
{
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"fieldName": "height"
}
],
"meta": {
"version": "3.0.0",
"category": "default"
},
"nodes": [
{
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"type": "invocation",
"data": {
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"type": "wan_model_loader",
"version": "1.0.1",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": false,
"inputs": {
"model": { "name": "model", "label": "" },
"transformer_low_noise_model": { "name": "transformer_low_noise_model", "label": "" },
"vae_model": { "name": "vae_model", "label": "" },
"wan_t5_encoder_model": { "name": "wan_t5_encoder_model", "label": "" },
"component_source": { "name": "component_source", "label": "" }
}
},
"position": { "x": 200, "y": 0 }
},
{
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"type": "invocation",
"data": {
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"type": "wan_text_encoder",
"version": "1.0.0",
"label": "Positive Prompt",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"inputs": {
"prompt": { "name": "prompt", "label": "", "value": "a cat" },
"wan_t5_encoder": { "name": "wan_t5_encoder", "label": "" }
}
},
"position": { "x": 700, "y": -200 }
},
{
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"type": "invocation",
"data": {
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"type": "wan_text_encoder",
"version": "1.0.0",
"label": "Negative Prompt",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"inputs": {
"prompt": { "name": "prompt", "label": "", "value": " " },
"wan_t5_encoder": { "name": "wan_t5_encoder", "label": "" }
}
},
"position": { "x": 700, "y": 100 }
},
{
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"type": "invocation",
"data": {
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"type": "rand_int",
"version": "1.0.1",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": false,
"inputs": {
"low": { "name": "low", "label": "", "value": 0 },
"high": { "name": "high", "label": "", "value": 2147483647 }
}
},
"position": { "x": 700, "y": 400 }
},
{
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"type": "invocation",
"data": {
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"type": "wan_denoise",
"version": "1.0.0",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": true,
"useCache": true,
"inputs": {
"transformer": { "name": "transformer", "label": "" },
"positive_conditioning": { "name": "positive_conditioning", "label": "" },
"negative_conditioning": { "name": "negative_conditioning", "label": "" },
"ref_image": { "name": "ref_image", "label": "" },
"latents": { "name": "latents", "label": "" },
"denoise_mask": { "name": "denoise_mask", "label": "" },
"denoising_start": { "name": "denoising_start", "label": "", "value": 0 },
"denoising_end": { "name": "denoising_end", "label": "", "value": 1 },
"add_noise": { "name": "add_noise", "label": "", "value": true },
"guidance_scale": { "name": "guidance_scale", "label": "CFG", "value": 5.0 },
"guidance_scale_low_noise": { "name": "guidance_scale_low_noise", "label": "CFG (Low)" },
"width": { "name": "width", "label": "", "value": 1024 },
"height": { "name": "height", "label": "", "value": 1024 },
"steps": { "name": "steps", "label": "", "value": 30 },
"seed": { "name": "seed", "label": "", "value": 0 }
}
},
"position": { "x": 1100, "y": -50 }
},
{
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"type": "invocation",
"data": {
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"type": "wan_l2i",
"version": "1.0.0",
"label": "",
"notes": "",
"isOpen": true,
"isIntermediate": false,
"useCache": true,
"inputs": {
"board": { "name": "board", "label": "" },
"metadata": { "name": "metadata", "label": "" },
"latents": { "name": "latents", "label": "" },
"vae": { "name": "vae", "label": "" }
}
},
"position": { "x": 1550, "y": -50 }
}
],
"edges": [
{
"id": "edge-loader-transformer-denoise",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "transformer",
"targetHandle": "transformer"
},
{
"id": "edge-loader-t5-pos",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-t5-neg",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"sourceHandle": "wan_t5_encoder",
"targetHandle": "wan_t5_encoder"
},
{
"id": "edge-loader-vae-l2i",
"type": "default",
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "vae",
"targetHandle": "vae"
},
{
"id": "edge-pos-cond-denoise",
"type": "default",
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "positive_conditioning"
},
{
"id": "edge-neg-cond-denoise",
"type": "default",
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "conditioning",
"targetHandle": "negative_conditioning"
},
{
"id": "edge-rand-seed-denoise",
"type": "default",
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"sourceHandle": "value",
"targetHandle": "seed"
},
{
"id": "edge-denoise-latents-l2i",
"type": "default",
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
"sourceHandle": "latents",
"targetHandle": "latents"
}
]
}
+94 -2
View File
@@ -179,6 +179,84 @@ ANIMA_LATENT_RGB_FACTORS = [
ANIMA_LATENT_RGB_BIAS = [-0.1835, -0.0868, -0.3360]
# Wan 2.2 A14B uses the standard 16-channel Wan VAE.
# Factors come from ComfyUI's Wan21 latent_format (same VAE as A14B).
WAN_LATENT_RGB_FACTORS = [
[-0.1299, -0.1692, 0.2932],
[0.0671, 0.0406, 0.0442],
[0.3568, 0.2548, 0.1747],
[0.0372, 0.2344, 0.1420],
[0.0313, 0.0189, -0.0328],
[0.0296, -0.0956, -0.0665],
[-0.3477, -0.4059, -0.2925],
[0.0166, 0.1902, 0.1975],
[-0.0412, 0.0267, -0.1364],
[-0.1293, 0.0740, 0.1636],
[0.0680, 0.3019, 0.1128],
[0.0032, 0.0581, 0.0639],
[-0.1251, 0.0927, 0.1699],
[0.0060, -0.0633, 0.0005],
[0.3477, 0.2275, 0.2950],
[0.1984, 0.0913, 0.1861],
]
WAN_LATENT_RGB_BIAS = [-0.1835, -0.0868, -0.3360]
# Wan 2.2 TI2V-5B uses Wan2.2-VAE with 48 latent channels and 16x spatial downscale.
# Factors come from ComfyUI's Wan22 latent_format.
WAN22_LATENT_RGB_FACTORS = [
[0.0119, 0.0103, 0.0046],
[-0.1062, -0.0504, 0.0165],
[0.0140, 0.0409, 0.0491],
[-0.0813, -0.0677, 0.0607],
[0.0656, 0.0851, 0.0808],
[0.0264, 0.0463, 0.0912],
[0.0295, 0.0326, 0.0590],
[-0.0244, -0.0270, 0.0025],
[0.0443, -0.0102, 0.0288],
[-0.0465, -0.0090, -0.0205],
[0.0359, 0.0236, 0.0082],
[-0.0776, 0.0854, 0.1048],
[0.0564, 0.0264, 0.0561],
[0.0006, 0.0594, 0.0418],
[-0.0319, -0.0542, -0.0637],
[-0.0268, 0.0024, 0.0260],
[0.0539, 0.0265, 0.0358],
[-0.0359, -0.0312, -0.0287],
[-0.0285, -0.1032, -0.1237],
[0.1041, 0.0537, 0.0622],
[-0.0086, -0.0374, -0.0051],
[0.0390, 0.0670, 0.2863],
[0.0069, 0.0144, 0.0082],
[0.0006, -0.0167, 0.0079],
[0.0313, -0.0574, -0.0232],
[-0.1454, -0.0902, -0.0481],
[0.0714, 0.0827, 0.0447],
[-0.0304, -0.0574, -0.0196],
[0.0401, 0.0384, 0.0204],
[-0.0758, -0.0297, -0.0014],
[0.0568, 0.1307, 0.1372],
[-0.0055, -0.0310, -0.0380],
[0.0239, -0.0305, 0.0325],
[-0.0663, -0.0673, -0.0140],
[-0.0416, -0.0047, -0.0023],
[0.0166, 0.0112, -0.0093],
[-0.0211, 0.0011, 0.0331],
[0.1833, 0.1466, 0.2250],
[-0.0368, 0.0370, 0.0295],
[-0.3441, -0.3543, -0.2008],
[-0.0479, -0.0489, -0.0420],
[-0.0660, -0.0153, 0.0800],
[-0.0101, 0.0068, 0.0156],
[-0.0690, -0.0452, -0.0927],
[-0.0145, 0.0041, 0.0015],
[0.0421, 0.0451, 0.0373],
[0.0504, -0.0483, -0.0356],
[-0.0837, 0.0168, 0.0055],
]
WAN22_LATENT_RGB_BIAS = [0.0317, -0.0878, -0.1388]
def sample_to_lowres_estimated_image(
samples: torch.Tensor,
@@ -270,6 +348,15 @@ def diffusion_step_callback(
# Anima uses Wan 2.1 VAE with 16 latent channels
latent_rgb_factors = ANIMA_LATENT_RGB_FACTORS
latent_rgb_bias = ANIMA_LATENT_RGB_BIAS
elif base_model == BaseModelType.Wan:
# A14B (16-ch standard Wan VAE, 8x spatial) vs TI2V-5B (48-ch Wan2.2-VAE,
# 16x spatial). The latent channel count uniquely identifies the variant.
if sample.shape[-3] == 48:
latent_rgb_factors = WAN22_LATENT_RGB_FACTORS
latent_rgb_bias = WAN22_LATENT_RGB_BIAS
else:
latent_rgb_factors = WAN_LATENT_RGB_FACTORS
latent_rgb_bias = WAN_LATENT_RGB_BIAS
else:
raise ValueError(f"Unsupported base model: {base_model}")
@@ -287,8 +374,13 @@ def diffusion_step_callback(
latent_rgb_bias=latent_rgb_bias_torch,
)
width = image.width * 8
height = image.height * 8
# Spatial downscale ratio: 8x is the SD/SDXL/FLUX/Wan-A14B default;
# Wan TI2V-5B's Wan2.2-VAE uses 16x.
spatial_scale = 8
if base_model == BaseModelType.Wan and sample.shape[-3] == 48:
spatial_scale = 16
width = image.width * spatial_scale
height = image.height * spatial_scale
percentage = calc_percentage(intermediate_state)
signal_progress("Denoising", percentage, image, (width, height))
+288
View File
@@ -0,0 +1,288 @@
"""Standalone decode worker for untrusted video files.
This script is executed as a short-lived child process by
``invokeai.app.util.video_thumbnails`` — never imported by the server at runtime — so
that a decoder hang on a crafted or malformed container can be bounded by a timeout and
killed. cv2 wheels have historically hung on certain codec/container combinations, and
a hung *thread* cannot be killed from Python, so process isolation is the only reliable
bound. It is run by file path (not ``-m``) and deliberately imports only the stdlib plus
imageio/PIL/cv2, so it starts quickly without pulling in the invokeai package or torch.
Protocol: ``python video_decode_worker.py <command> <args...>``. On success a single
JSON object is written to stdout and the exit code is 0; on any failure the exit code is
non-zero with a message on stderr.
Commands:
probe <video_path> -> {"width", "height", "duration", "fps", "codec"}
frame <video_path> <index> <out_path> -> {"ok": true}; the frame is written to out_path as PNG
count <video_path> -> {"count": <int or null>}
stream <video_path> -> consecutive numpy arrays on stdout
"""
import io
import json
import math
import struct
import sys
from pathlib import Path
from typing import Optional
import imageio.v3 as iio
import numpy as np
from PIL import Image
# Upper bound on the decoded frame size we are willing to allocate (~8K video). A small
# crafted container can claim absurd dimensions in cheap metadata while the full frame
# is only allocated at decode time, so dimensions are checked before any decode.
# Keep in sync with MAX_VIDEO_FRAME_PIXELS in video_thumbnails.py (this script must not
# import the invokeai package).
MAX_FRAME_PIXELS = 64 * 1024 * 1024
# Address-space headroom for everything allocated after startup. Must accommodate a
# legal max-size decode, not just a typical one: a MAX_FRAME_PIXELS RGB frame is
# 192 MiB and the emit path holds ~4 frame-sized copies at once (~768 MiB), on top of
# which RLIMIT_AS also counts the lazily-imported cv2's library mappings, glibc
# per-thread arenas, and the codec's reference-frame buffers — address space routinely
# exceeds resident memory by a wide margin. Keep in sync with
# MAX_VIDEO_DECODE_WORKER_RSS_BYTES in video_thumbnails.py (the parent-side resident
# bound; this script must not import the invokeai package).
WORKER_MEMORY_HEADROOM_BYTES = 4 * 1024 * 1024 * 1024
def _limit_worker_memory() -> None:
"""Bound allocations made after worker startup on Linux."""
if sys.platform != "linux":
return
try:
import os
import resource
pages = int(Path("/proc/self/statm").read_text().split()[0])
limit = pages * os.sysconf("SC_PAGE_SIZE") + WORKER_MEMORY_HEADROOM_BYTES
_soft, hard = resource.getrlimit(resource.RLIMIT_AS)
if hard != resource.RLIM_INFINITY:
limit = min(limit, hard)
resource.setrlimit(resource.RLIMIT_AS, (limit, hard))
except (OSError, ValueError):
pass
def _validate_decoded_frame(frame: np.ndarray) -> None:
if frame.ndim != 3 or frame.shape[2] != 3:
raise ValueError(f"Decoded frame must be RGB; got shape {frame.shape}")
height, width = frame.shape[:2]
if height <= 0 or width <= 0 or height * width > MAX_FRAME_PIXELS:
raise ValueError(f"Decoded frame dimensions {width}x{height} exceed the maximum decodable size")
def _assert_decodable_dims(video_path: Path) -> None:
"""Refuses to decode frames from a video whose reported dimensions exceed the bound.
If the dimensions cannot be probed, decoding is refused because the parent's frame
record bound and PIL checks run only after the decoder has allocated the frame.
"""
try:
width, height, _duration, _fps = _probe(video_path)[:4]
except Exception as error:
raise ValueError(f"Unable to validate video dimensions for {video_path}") from error
if width <= 0 or height <= 0:
raise ValueError(f"Video reports invalid dimensions {width}x{height}")
if width * height > MAX_FRAME_PIXELS:
raise ValueError(f"Video dimensions {width}x{height} exceed the maximum decodable size")
def _extract_frame(video_path: Path, frame_index: int) -> Optional[Image.Image]:
"""Extracts a single frame from a video file as a PIL Image. Returns None on failure.
Tries imageio's FFMPEG plugin first since it's the same encoder we use for output,
then falls back to cv2 — uploaded videos with unusual codecs may need that path.
"""
try:
# iio.imread with index=N seeks to that frame directly. Returns RGB HxWxC uint8.
frame = iio.imread(video_path, plugin="FFMPEG", index=frame_index)
_validate_decoded_frame(frame)
return Image.fromarray(frame)
except Exception:
pass
try:
import cv2 # local import so the imageio-only path doesn't pay the cv2 import cost
capture = cv2.VideoCapture(str(video_path))
if not capture.isOpened():
capture.release()
return None
try:
if frame_index > 0:
capture.set(cv2.CAP_PROP_POS_FRAMES, frame_index)
ok, frame_bgr = capture.read()
if not ok or frame_bgr is None:
return None
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
_validate_decoded_frame(frame_rgb)
return Image.fromarray(frame_rgb)
finally:
capture.release()
except Exception:
return None
def _probe(video_path: Path) -> tuple[int, int, float, Optional[float], Optional[str]]:
"""Returns (width, height, duration_seconds, fps_or_none, codec_or_none) for a video file.
Tries imageio's FFMPEG plugin first; falls back to cv2.VideoCapture. Raises if
neither backend can read the file.
"""
try:
meta = iio.immeta(video_path, plugin="FFMPEG")
fps_raw = meta.get("fps")
duration = float(meta.get("duration", 0.0)) if meta.get("duration") is not None else 0.0
size = meta.get("size")
if size is None:
# Fall through to cv2 — imageio didn't give us dimensions.
raise ValueError("imageio probe missing 'size'")
width, height = int(size[0]), int(size[1])
fps: Optional[float] = float(fps_raw) if fps_raw and fps_raw > 0 else None
codec_raw = meta.get("codec")
codec = str(codec_raw).lower() if codec_raw else None
return width, height, duration, fps, codec
except Exception:
pass
import cv2
capture = cv2.VideoCapture(str(video_path))
if not capture.isOpened():
capture.release()
raise FileNotFoundError(f"Unable to open video at {video_path}")
try:
width = int(capture.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(capture.get(cv2.CAP_PROP_FRAME_HEIGHT))
frame_count = int(capture.get(cv2.CAP_PROP_FRAME_COUNT))
fps_raw = capture.get(cv2.CAP_PROP_FPS)
fps_v2: Optional[float] = float(fps_raw) if fps_raw and fps_raw > 0 else None
duration = (frame_count / fps_v2) if (fps_v2 and frame_count > 0) else 0.0
fourcc = int(capture.get(cv2.CAP_PROP_FOURCC))
codec = "".join(chr((fourcc >> (8 * index)) & 0xFF) for index in range(4)).strip().lower() or None
finally:
capture.release()
return width, height, duration, fps_v2, codec
def _count(video_path: Path) -> Optional[int]:
"""Return the exact decoded frame count, or None if neither backend can determine it.
Tries imageio's improps first (works for a handful of codecs that expose nframes in
container metadata). For libx264 streams imageio reports ``inf``, so we fall through
to cv2's ``CAP_PROP_FRAME_COUNT`` which reads the actual packet count. Both sources
are preferred over a ``duration * fps`` estimate, which can overshoot by one on VFR
uploads or containers with imprecise metadata.
"""
try:
props = iio.improps(video_path, plugin="FFMPEG")
except Exception:
props = None
shape = getattr(props, "shape", None) if props is not None else None
if shape:
n = shape[0]
if not (isinstance(n, float) and not math.isfinite(n)):
try:
return int(n)
except (TypeError, ValueError, OverflowError):
pass
try:
import cv2
capture = cv2.VideoCapture(str(video_path))
if not capture.isOpened():
capture.release()
return None
try:
count = int(capture.get(cv2.CAP_PROP_FRAME_COUNT))
finally:
capture.release()
return count if count > 0 else None
except Exception:
return None
def _emit_stream_frame(frame: np.ndarray) -> None:
_validate_decoded_frame(frame)
record = io.BytesIO()
np.save(record, np.ascontiguousarray(frame), allow_pickle=False)
payload = record.getvalue()
sys.stdout.buffer.write(struct.pack(">Q", len(payload)))
sys.stdout.buffer.write(payload)
sys.stdout.buffer.flush()
def _stream(video_path: Path) -> None:
"""Writes decoded frames as length-prefixed, non-pickled numpy records.
Tries imageio's FFMPEG plugin first and falls back to cv2, mirroring
``_extract_frame``/``_probe``/``_count`` — a file accepted at upload via the cv2
fallback must also stream in the frame-range and concat nodes. The fallback only
engages when no frame has been emitted yet: a mid-stream decoder failure must
surface as an error, not restart the stream from frame 0.
"""
emitted = False
try:
for frame in iio.imiter(video_path, plugin="FFMPEG"):
_emit_stream_frame(frame)
emitted = True
except Exception:
if emitted:
raise
if emitted:
return
import cv2
capture = cv2.VideoCapture(str(video_path))
if not capture.isOpened():
capture.release()
raise FileNotFoundError(f"Unable to open video at {video_path}")
try:
while True:
ok, frame_bgr = capture.read()
if not ok or frame_bgr is None:
break
_emit_stream_frame(cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB))
emitted = True
finally:
capture.release()
if not emitted:
raise ValueError(f"No frames decoded from {video_path}")
def main(argv: list[str]) -> int:
try:
_limit_worker_memory()
command = argv[1]
if command == "stream":
_assert_decodable_dims(Path(argv[2]))
_stream(Path(argv[2]))
elif command == "probe":
width, height, duration, fps, codec = _probe(Path(argv[2]))
print(json.dumps({"width": width, "height": height, "duration": duration, "fps": fps, "codec": codec}))
elif command == "frame":
_assert_decodable_dims(Path(argv[2]))
image = _extract_frame(Path(argv[2]), int(argv[3]))
if image is None:
print("no frame decoded", file=sys.stderr)
return 1
image.save(argv[4], format="PNG")
print(json.dumps({"ok": True}))
elif command == "count":
print(json.dumps({"count": _count(Path(argv[2]))}))
else:
print(f"unknown command {command!r}", file=sys.stderr)
return 1
except Exception as e:
print(str(e) or repr(e), file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
+21
View File
@@ -0,0 +1,21 @@
"""Shared MP4 writer construction for video-producing invocations.
All video nodes encode through this helper so the encoder settings stay in one
place. libx264 + yuv420p (imageio's defaults for the FFMPEG plugin) give
broadly-compatible browser playback. ``macro_block_size=1`` is load-bearing:
imageio's default of 16 makes ffmpeg silently *rescale* frames to the next
multiple of 16 (e.g. 1920x1080 -> 1920x1088), which desynchronizes the encoded
file from the dimensions recorded in the video DTO and breaks same-dimension
checks downstream (e.g. concatenating a trimmed clip with its source).
yuv420p requires even dimensions; callers validate that before encoding.
"""
from pathlib import Path
import imageio.v2 as iio2
def make_mp4_writer(path: Path | str, fps: float):
"""Returns an imageio FFMPEG writer that preserves frame dimensions exactly."""
return iio2.get_writer(str(path), format="FFMPEG", mode="I", fps=fps, codec="libx264", macro_block_size=1)
+442
View File
@@ -0,0 +1,442 @@
"""Video frame/probe helpers used by the video file store, upload router, and video nodes.
Decoding runs in a short-lived child process (``video_decode_worker.py``) with a hard
timeout. Files reaching these helpers are user uploads, and both decode backends can
hang indefinitely on crafted or malformed containers (cv2 wheels historically, ffmpeg in
degenerate cases). An in-process hang would tie up the FastAPI request worker that
called us — repeated crafted uploads could exhaust the worker pool — and a hung thread
cannot be killed from Python, so process isolation is the only reliable bound.
The parent explicitly terminates the worker and its descendants when the timeout
expires, so a hostile file costs at most ``timeout`` seconds and cannot leak a stuck
FFmpeg process.
"""
import io
import json
import math
import os
import queue
import signal
import struct
import subprocess
import sys
import tempfile
import threading
import time
from pathlib import Path
from typing import Any, Callable, Iterator, Optional
import numpy as np
import psutil
from PIL import Image
from invokeai.app.services.session_processor.session_processor_common import CanceledException
# Generous — a healthy decode of a single frame or of container metadata takes well
# under a second even for large files. The timeout exists to bound adversarial or hung
# decodes, not to police slow ones.
VIDEO_DECODE_TIMEOUT_SECONDS = 30.0
MAX_DECODED_FRAME_RECORD_BYTES = 256 * 1024 * 1024
MAX_DECODE_STDERR_BYTES = 16 * 1024
# Resident-memory kill threshold for a decode worker *tree* (the worker plus its
# ffmpeg child). Sized for a legal max-size decode, not a typical one: a
# MAX_VIDEO_FRAME_PIXELS RGB frame is 192 MiB, the worker's emit path holds ~4
# frame-sized copies at once (~768 MiB), and the ffmpeg child keeps its own output
# frame plus YUV reference frames (several hundred MiB more) — so a 1 GiB bound would
# kill decodes of videos the frame-size validators explicitly accept. This is an
# anti-runaway bound, not a budget. Keep in sync with WORKER_MEMORY_HEADROOM_BYTES in
# video_decode_worker.py (the worker-side address-space bound).
MAX_VIDEO_DECODE_WORKER_RSS_BYTES = 3 * 1024 * 1024 * 1024
# A bound this size doesn't need 50 ms granularity; each poll walks the process tree
# via psutil, so a coarser period keeps sustained streaming decodes cheap.
WORKER_MEMORY_POLL_SECONDS = 0.25
# Upper bound on decoded frame size (~8K video). Decoder-reported dimensions are
# untrusted metadata: a tiny crafted container can claim absurd dimensions whose full
# frame is only allocated at decode time. probe_video rejects such files before the
# upload path persists them, and the worker refuses to decode frames from them.
# Keep in sync with MAX_FRAME_PIXELS in video_decode_worker.py.
MAX_VIDEO_FRAME_PIXELS = 64 * 1024 * 1024
MAX_CONCURRENT_VIDEO_DECODERS = 2
_VIDEO_DECODER_SLOTS = threading.BoundedSemaphore(MAX_CONCURRENT_VIDEO_DECODERS)
_VIDEO_STREAM_SLOTS = threading.BoundedSemaphore(1)
_WORKER_PATH = Path(__file__).parent / "video_decode_worker.py"
class VideoDecodeTimeoutError(TimeoutError):
"""The decode worker did not finish within its time budget.
A timeout is contention (or an adversarial file), not evidence the video is
undecodable — callers that gate acceptance on decodability must treat it as
inconclusive rather than as a failed decode.
"""
def get_video_thumbnail_name(video_name: str) -> str:
"""Given a video file name (e.g. <uuid>.mp4), returns the matching thumbnail name (e.g. <uuid>.webp)."""
return os.path.splitext(video_name)[0] + ".webp"
def _worker_command(*args: str) -> list[str]:
"""Command line for one decode-worker invocation (patchable in tests).
The worker is run by file path rather than ``-m`` so the child process doesn't
import the invokeai package (and transitively torch) just to decode a frame.
"""
return [sys.executable, str(_WORKER_PATH), *args]
def _spawn_worker(*args: str, **kwargs: Any) -> subprocess.Popen[Any]:
"""Starts a worker in an independently killable process group on POSIX."""
if os.name != "nt":
kwargs["start_new_session"] = True
return subprocess.Popen(_worker_command(*args), **kwargs)
def _is_process_running(pid: int) -> bool:
"""Returns whether a process exists and is not a zombie."""
try:
process = psutil.Process(pid)
return process.is_running() and process.status() != psutil.STATUS_ZOMBIE
except psutil.Error:
return False
def _terminate_process_tree(proc: subprocess.Popen[Any]) -> None:
"""Kills a worker and every descendant it spawned."""
if os.name != "nt":
try:
os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
proc.wait(timeout=5)
return
except (OSError, subprocess.TimeoutExpired):
pass
try:
parent = psutil.Process(proc.pid)
processes = parent.children(recursive=True)
processes.append(parent)
for process in processes:
try:
process.kill()
except psutil.Error:
pass
psutil.wait_procs(processes, timeout=5)
except psutil.Error:
try:
proc.kill()
except OSError:
pass
def _worker_tree_rss(proc: subprocess.Popen[Any]) -> int:
"""Returns resident memory used by the worker and its decoder children."""
try:
worker = psutil.Process(proc.pid)
processes = [worker, *worker.children(recursive=True)]
except psutil.Error:
return 0
total = 0
for process in processes:
try:
total += process.memory_info().rss
except psutil.Error:
pass
return total
def _start_worker_memory_monitor(
proc: subprocess.Popen[Any],
) -> tuple[threading.Event, threading.Event, threading.Thread]:
"""Kills a decoder process tree when its cross-platform RSS bound is exceeded."""
stopped = threading.Event()
exceeded = threading.Event()
def monitor() -> None:
while not stopped.is_set():
if proc.poll() is not None:
return
if _worker_tree_rss(proc) > MAX_VIDEO_DECODE_WORKER_RSS_BYTES:
exceeded.set()
_terminate_process_tree(proc)
return
stopped.wait(WORKER_MEMORY_POLL_SECONDS)
thread = threading.Thread(target=monitor, name="video-memory-monitor", daemon=True)
thread.start()
return stopped, exceeded, thread
def _run_worker_unbounded(args: list[str], timeout: float, raise_on_timeout: bool = False) -> Optional[dict[str, Any]]:
"""Runs the decode worker; returns its parsed JSON output, or None on failure or timeout.
With ``raise_on_timeout``, a timeout raises VideoDecodeTimeoutError instead of
returning None, letting callers distinguish "could not decode" from "ran out of
time on a loaded machine".
"""
monitor_stop: threading.Event | None = None
monitor: threading.Thread | None = None
try:
proc = _spawn_worker(*args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
except Exception:
return None
try:
monitor_stop, _memory_exceeded, monitor = _start_worker_memory_monitor(proc)
stdout, _ = proc.communicate(timeout=timeout)
except subprocess.TimeoutExpired as error:
_terminate_process_tree(proc)
proc.communicate()
if raise_on_timeout:
raise VideoDecodeTimeoutError(f"Video decode worker timed out after {timeout}s") from error
return None
except Exception:
# An unexpected failure (e.g. OSError from communicate()) must not leak the
# worker tree: the finally below stops the RSS-monitor backstop, so nothing
# else would ever reap a still-running worker and its ffmpeg child.
_terminate_process_tree(proc)
return None
finally:
if monitor_stop is not None and monitor is not None:
monitor_stop.set()
monitor.join(timeout=1)
if proc.returncode != 0:
return None
try:
result = json.loads(stdout)
except ValueError:
return None
return result if isinstance(result, dict) else None
def _run_worker(args: list[str], timeout: float, raise_on_timeout: bool = False) -> Optional[dict[str, Any]]:
deadline = time.monotonic() + timeout
if not _VIDEO_DECODER_SLOTS.acquire(timeout=timeout):
if raise_on_timeout:
raise VideoDecodeTimeoutError(f"Video decode worker timed out after {timeout}s")
return None
try:
remaining = max(0.0, deadline - time.monotonic())
return _run_worker_unbounded(args, remaining, raise_on_timeout)
finally:
_VIDEO_DECODER_SLOTS.release()
def _iter_video_frames_unbounded(
video_path: Path,
timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS,
is_canceled: Optional[Callable[[], bool]] = None,
) -> Iterator[np.ndarray]:
"""Streams decoded frames from an isolated worker with bounded memory and wait time."""
proc = _spawn_worker(
"stream",
str(video_path),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
if proc.stdout is None or proc.stderr is None:
_terminate_process_tree(proc)
raise RuntimeError("Unable to open video decoder output stream")
memory_monitor_stop, _memory_exceeded, memory_monitor = _start_worker_memory_monitor(proc)
results: queue.Queue[tuple[str, object]] = queue.Queue(maxsize=1)
stopped = threading.Event()
stderr_tail = bytearray()
def read_stderr() -> str:
return bytes(stderr_tail).decode(errors="replace").strip()
def drain_stderr() -> None:
while chunk := proc.stderr.read(4096):
stderr_tail.extend(chunk)
if len(stderr_tail) > MAX_DECODE_STDERR_BYTES:
del stderr_tail[:-MAX_DECODE_STDERR_BYTES]
def read_exactly(size: int) -> bytes:
chunks: list[bytes] = []
remaining = size
while remaining > 0:
chunk = proc.stdout.read(remaining)
if not chunk:
if remaining == size:
raise EOFError
raise OSError("Truncated frame record from video decoder")
chunks.append(chunk)
remaining -= len(chunk)
return b"".join(chunks)
def put_result(result: tuple[str, object]) -> None:
while not stopped.is_set():
try:
results.put(result, timeout=0.1)
return
except queue.Full:
continue
def read_frames() -> None:
try:
while not stopped.is_set():
record_size = struct.unpack(">Q", read_exactly(8))[0]
if record_size > MAX_DECODED_FRAME_RECORD_BYTES:
raise ValueError(f"Decoded frame record exceeds {MAX_DECODED_FRAME_RECORD_BYTES} bytes")
payload = read_exactly(record_size)
put_result(("frame", np.load(io.BytesIO(payload), allow_pickle=False)))
except (EOFError, ValueError, OSError) as error:
put_result(("done", error))
reader = threading.Thread(target=read_frames, name="video-frame-reader", daemon=True)
stderr_reader = threading.Thread(target=drain_stderr, name="video-stderr-reader", daemon=True)
reader.start()
stderr_reader.start()
deadline = time.monotonic() + timeout
try:
while True:
if is_canceled is not None and is_canceled():
raise CanceledException
remaining = deadline - time.monotonic()
if remaining <= 0:
raise TimeoutError(f"Timed out decoding frames from {video_path}")
try:
kind, value = results.get(timeout=min(0.1, remaining))
except queue.Empty:
continue
if kind == "frame":
if not isinstance(value, np.ndarray):
raise ValueError(f"Decoder returned an invalid frame for {video_path}")
yield value
deadline = time.monotonic() + timeout
continue
try:
return_code = proc.wait(timeout=min(1, timeout))
except subprocess.TimeoutExpired as error:
_terminate_process_tree(proc)
stderr_reader.join(timeout=1)
detail = read_stderr()
message = f"Timed out waiting for video decoder worker for {video_path}"
raise TimeoutError(f"{message}: {detail}" if detail else message) from error
if return_code != 0:
stderr_reader.join(timeout=1)
detail = read_stderr()
message = f"Unable to decode video at {video_path}"
raise ValueError(f"{message}: {detail}" if detail else message) from value
return
finally:
memory_monitor_stop.set()
memory_monitor.join(timeout=1)
stopped.set()
if proc.poll() is None:
_terminate_process_tree(proc)
proc.stdout.close()
proc.wait()
reader.join(timeout=1)
stderr_reader.join(timeout=1)
proc.stderr.close()
def iter_video_frames(
video_path: Path,
timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS,
is_canceled: Optional[Callable[[], bool]] = None,
) -> Iterator[np.ndarray]:
acquired: list[threading.BoundedSemaphore] = []
capacity_deadline = time.monotonic() + timeout
try:
for slot in (_VIDEO_STREAM_SLOTS, _VIDEO_DECODER_SLOTS):
while True:
if is_canceled is not None and is_canceled():
raise CanceledException
remaining = capacity_deadline - time.monotonic()
if remaining <= 0:
raise TimeoutError(f"Timed out waiting to decode frames from {video_path}")
if slot.acquire(timeout=min(0.1, remaining)):
acquired.append(slot)
break
yield from _iter_video_frames_unbounded(video_path, timeout, is_canceled)
finally:
for slot in reversed(acquired):
slot.release()
def extract_video_frame(
video_path: Path,
frame_index: int = 0,
timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS,
raise_on_timeout: bool = False,
) -> Optional[Image.Image]:
"""Extracts a single frame from a video file as a PIL Image. Returns None on failure or timeout.
With ``raise_on_timeout``, a timeout raises VideoDecodeTimeoutError instead of
returning None.
"""
fd, tmp_name = tempfile.mkstemp(prefix="invokeai_frame_", suffix=".png")
os.close(fd)
try:
result = _run_worker(["frame", str(video_path), str(frame_index), tmp_name], timeout, raise_on_timeout)
if result is None:
return None
with Image.open(tmp_name) as image:
image.load()
return image
except VideoDecodeTimeoutError:
raise
except Exception:
return None
finally:
Path(tmp_name).unlink(missing_ok=True)
def probe_video_with_codec(
video_path: Path, timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS
) -> tuple[int, int, float, Optional[float], Optional[str]]:
"""Returns (width, height, duration_seconds, fps_or_none, codec_or_none) for a video file.
Raises FileNotFoundError if the file cannot be read — including when the decode
times out, since a file we cannot probe within the bound is treated as unreadable —
or when the decoder reports metadata no sane video has (non-positive or over-limit
dimensions, non-finite or negative duration). Decoder-reported values are untrusted:
they come from the uploaded container, and the upload path persists them and sizes
thumbnail decoding by them. A non-finite or non-positive fps is coerced to None
(unknown) rather than rejected, matching the decoder's own unknown-fps behavior.
"""
result = _run_worker(["probe", str(video_path)], timeout)
if result is None:
raise FileNotFoundError(f"Unable to open video at {video_path}")
try:
width = int(result["width"])
height = int(result["height"])
duration = float(result["duration"])
fps_raw = result.get("fps")
fps: Optional[float] = float(fps_raw) if fps_raw else None
codec_raw = result.get("codec")
codec = str(codec_raw).lower() if codec_raw else None
except (KeyError, TypeError, ValueError, OverflowError) as e:
raise FileNotFoundError(f"Unable to open video at {video_path}") from e
if width <= 0 or height <= 0 or width * height > MAX_VIDEO_FRAME_PIXELS:
raise FileNotFoundError(f"Video at {video_path} reports invalid dimensions {width}x{height}")
if not math.isfinite(duration) or duration < 0:
raise FileNotFoundError(f"Video at {video_path} reports an invalid duration {duration}")
if fps is not None and (not math.isfinite(fps) or fps <= 0):
fps = None
return width, height, duration, fps, codec
def probe_video(
video_path: Path, timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS
) -> tuple[int, int, float, Optional[float]]:
"""Returns validated video metadata without the codec."""
width, height, duration, fps, _codec = probe_video_with_codec(video_path, timeout)
return width, height, duration, fps
def decoder_frame_count(video_path: Path, timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS) -> Optional[int]:
"""Returns the exact decoded frame count, or None if it cannot be determined in time.
Preferred over a ``duration * fps`` estimate, which can overshoot by one on VFR
uploads or containers with imprecise metadata; callers fall back to that estimate
when this returns None.
"""
result = _run_worker(["count", str(video_path)], timeout)
if result is None:
return None
count = result.get("count")
if isinstance(count, bool) or not isinstance(count, (int, float)):
return None
return int(count) if count > 0 else None
@@ -55,6 +55,7 @@ from invokeai.backend.model_manager.configs.lora import (
LoRA_LyCORIS_SD1_Config,
LoRA_LyCORIS_SD2_Config,
LoRA_LyCORIS_SDXL_Config,
LoRA_LyCORIS_Wan_Config,
LoRA_LyCORIS_ZImage_Config,
LoRA_OMI_FLUX_Config,
LoRA_OMI_SDXL_Config,
@@ -81,10 +82,12 @@ from invokeai.backend.model_manager.configs.main import (
Main_Diffusers_SD3_Config,
Main_Diffusers_SDXL_Config,
Main_Diffusers_SDXLRefiner_Config,
Main_Diffusers_Wan_Config,
Main_Diffusers_ZImage_Config,
Main_GGUF_Flux2_Config,
Main_GGUF_FLUX_Config,
Main_GGUF_QwenImage_Config,
Main_GGUF_Wan_Config,
Main_GGUF_ZImage_Config,
MainModelDefaultSettings,
)
@@ -126,10 +129,13 @@ from invokeai.backend.model_manager.configs.vae import (
VAE_Checkpoint_SD1_Config,
VAE_Checkpoint_SD2_Config,
VAE_Checkpoint_SDXL_Config,
VAE_Checkpoint_Wan_Config,
VAE_Diffusers_Flux2_Config,
VAE_Diffusers_SD1_Config,
VAE_Diffusers_SDXL_Config,
VAE_Diffusers_Wan_Config,
)
from invokeai.backend.model_manager.configs.wan_t5_encoder import WanT5Encoder_WanT5Encoder_Config
from invokeai.backend.model_manager.model_on_disk import ModelOnDisk
from invokeai.backend.model_manager.taxonomy import (
BaseModelType,
@@ -180,6 +186,7 @@ AnyModelConfig = Annotated[
Annotated[Main_Diffusers_Flux2_Config, Main_Diffusers_Flux2_Config.get_tag()],
Annotated[Main_Diffusers_CogView4_Config, Main_Diffusers_CogView4_Config.get_tag()],
Annotated[Main_Diffusers_QwenImage_Config, Main_Diffusers_QwenImage_Config.get_tag()],
Annotated[Main_Diffusers_Wan_Config, Main_Diffusers_Wan_Config.get_tag()],
Annotated[Main_Diffusers_ZImage_Config, Main_Diffusers_ZImage_Config.get_tag()],
Annotated[Main_Diffusers_Ideogram4_Config, Main_Diffusers_Ideogram4_Config.get_tag()],
# Main (Pipeline) - checkpoint format
@@ -201,6 +208,7 @@ AnyModelConfig = Annotated[
Annotated[Main_GGUF_Flux2_Config, Main_GGUF_Flux2_Config.get_tag()],
Annotated[Main_GGUF_FLUX_Config, Main_GGUF_FLUX_Config.get_tag()],
Annotated[Main_GGUF_QwenImage_Config, Main_GGUF_QwenImage_Config.get_tag()],
Annotated[Main_GGUF_Wan_Config, Main_GGUF_Wan_Config.get_tag()],
Annotated[Main_GGUF_ZImage_Config, Main_GGUF_ZImage_Config.get_tag()],
# VAE - checkpoint format
Annotated[VAE_Checkpoint_SD1_Config, VAE_Checkpoint_SD1_Config.get_tag()],
@@ -208,12 +216,18 @@ AnyModelConfig = Annotated[
Annotated[VAE_Checkpoint_SDXL_Config, VAE_Checkpoint_SDXL_Config.get_tag()],
Annotated[VAE_Checkpoint_FLUX_Config, VAE_Checkpoint_FLUX_Config.get_tag()],
Annotated[VAE_Checkpoint_Flux2_Config, VAE_Checkpoint_Flux2_Config.get_tag()],
# IMPORTANT: VAE_Checkpoint_Wan_Config must be checked BEFORE QwenImage —
# both share the AutoencoderKLWan architecture and the Wan config relies
# on a filename heuristic to claim 16-channel files; ordering here lets
# Wan win when the filename suggests it.
Annotated[VAE_Checkpoint_Wan_Config, VAE_Checkpoint_Wan_Config.get_tag()],
Annotated[VAE_Checkpoint_QwenImage_Config, VAE_Checkpoint_QwenImage_Config.get_tag()],
Annotated[VAE_Checkpoint_Anima_Config, VAE_Checkpoint_Anima_Config.get_tag()],
# VAE - diffusers format
Annotated[VAE_Diffusers_SD1_Config, VAE_Diffusers_SD1_Config.get_tag()],
Annotated[VAE_Diffusers_SDXL_Config, VAE_Diffusers_SDXL_Config.get_tag()],
Annotated[VAE_Diffusers_Flux2_Config, VAE_Diffusers_Flux2_Config.get_tag()],
Annotated[VAE_Diffusers_Wan_Config, VAE_Diffusers_Wan_Config.get_tag()],
# ControlNet - checkpoint format
Annotated[ControlNet_Checkpoint_SD1_Config, ControlNet_Checkpoint_SD1_Config.get_tag()],
Annotated[ControlNet_Checkpoint_SD2_Config, ControlNet_Checkpoint_SD2_Config.get_tag()],
@@ -236,6 +250,13 @@ AnyModelConfig = Annotated[
Annotated[LoRA_LyCORIS_FLUX_Config, LoRA_LyCORIS_FLUX_Config.get_tag()],
Annotated[LoRA_LyCORIS_ZImage_Config, LoRA_LyCORIS_ZImage_Config.get_tag()],
Annotated[LoRA_LyCORIS_QwenImage_Config, LoRA_LyCORIS_QwenImage_Config.get_tag()],
# Wan and Anima both target ``blocks.X`` shapes; their LoRA probes are
# mutually exclusive — Wan rejects Anima's ``_proj``/``mlp``/
# ``adaln_modulation`` markers, Anima requires at least one of those
# markers (see ``has_cosmos_dit_*_keys_strict``). Order between these
# two doesn't affect correctness; mutual exclusivity is locked in by
# ``test_wan_lora_probe_independence.py``.
Annotated[LoRA_LyCORIS_Wan_Config, LoRA_LyCORIS_Wan_Config.get_tag()],
Annotated[LoRA_LyCORIS_Anima_Config, LoRA_LyCORIS_Anima_Config.get_tag()],
# LoRA - OMI format
Annotated[LoRA_OMI_SDXL_Config, LoRA_OMI_SDXL_Config.get_tag()],
@@ -262,6 +283,8 @@ AnyModelConfig = Annotated[
# Qwen VL Encoder (Qwen2.5-VL multimodal encoder for Qwen Image)
Annotated[QwenVLEncoder_Diffusers_Config, QwenVLEncoder_Diffusers_Config.get_tag()],
Annotated[QwenVLEncoder_Checkpoint_Config, QwenVLEncoder_Checkpoint_Config.get_tag()],
# Wan T5 Encoder (UMT5-XXL for Wan 2.2)
Annotated[WanT5Encoder_WanT5Encoder_Config, WanT5Encoder_WanT5Encoder_Config.get_tag()],
# TI - file format
Annotated[TI_File_SD1_Config, TI_File_SD1_Config.get_tag()],
Annotated[TI_File_SD2_Config, TI_File_SD2_Config.get_tag()],
+114 -8
View File
@@ -28,14 +28,23 @@ from invokeai.backend.model_manager.taxonomy import (
FluxLoRAFormat,
ModelFormat,
ModelType,
WanLoRAVariantType,
ZImageVariantType,
)
from invokeai.backend.model_manager.util.model_util import lora_token_vector_length
from invokeai.backend.patches.lora_conversions.anima_lora_constants import (
has_cosmos_dit_kohya_keys,
has_cosmos_dit_kohya_keys_strict,
has_cosmos_dit_peft_keys,
has_cosmos_dit_peft_keys_strict,
)
from invokeai.backend.patches.lora_conversions.flux_control_lora_utils import is_state_dict_likely_flux_control
from invokeai.backend.patches.lora_conversions.wan_lora_constants import (
detect_wan_lora_variant,
has_non_wan_architecture_keys,
has_wan_kohya_keys,
has_wan_peft_keys,
)
# Defaults used to compute the effective slider range when one or both bounds
# are unset. These intentionally mirror the frontend's DEFAULT_LORA_WEIGHT_CONFIG
@@ -909,16 +918,20 @@ class LoRA_LyCORIS_Anima_Config(LoRA_LyCORIS_Config_Base, Config_Base):
Anima LoRAs have keys like:
- lora_unet_blocks_0_cross_attn_k_proj.lora_down.weight (Kohya format)
- diffusion_model.blocks.0.cross_attn.k_proj.lora_A.weight (diffusers PEFT format)
- transformer.blocks.0.cross_attn.k_proj.lora_A.weight (diffusers PEFT format)
- transformer.blocks.0.mlp.layer_0.lora_A.weight (Anima-only MLP layer)
Detection requires Cosmos DiT-specific subcomponent names (cross_attn,
self_attn, mlp, adaln_modulation) to avoid false-positives on other
architectures that also use ``blocks`` in their paths.
Uses the **strict** Cosmos-DiT detectors, which require an
Anima-exclusive subcomponent name (``mlp``, ``adaln_modulation``, or
``_proj``-suffixed attention). The loose detectors would also accept
Wan-native LoRAs (which use ``cross_attn``/``self_attn`` too but with
bare ``.q``/``.k``/``.v``/``.o`` rather than ``_proj``), so they're not
safe for first-match-wins probing — see the regression tests in
``test_wan_lora_probe_independence.py``.
"""
state_dict = mod.load_state_dict()
str_keys = [k for k in state_dict.keys() if isinstance(k, str)]
has_cosmos_keys = has_cosmos_dit_kohya_keys(str_keys) or has_cosmos_dit_peft_keys(str_keys)
has_cosmos_keys = has_cosmos_dit_kohya_keys_strict(str_keys) or has_cosmos_dit_peft_keys_strict(str_keys)
# Also check for LoRA/LoKR weight suffixes
has_lora_suffix = state_dict_has_any_keys_ending_with(
@@ -941,19 +954,112 @@ class LoRA_LyCORIS_Anima_Config(LoRA_LyCORIS_Config_Base, Config_Base):
@classmethod
def _get_base_or_raise(cls, mod: ModelOnDisk) -> BaseModelType:
"""Anima LoRAs target Cosmos DiT blocks (blocks.X.cross_attn, blocks.X.self_attn, etc.).
"""Anima LoRAs target Cosmos DiT blocks (blocks.X.mlp, blocks.X.adaln_modulation,
blocks.X.cross_attn.q_proj, etc.).
Uses Cosmos DiT-specific subcomponent names to avoid false-positives.
Uses the strict Cosmos-DiT detectors to be mutually exclusive with
Wan-LoRA detection — see ``_validate_looks_like_lora`` for rationale.
"""
state_dict = mod.load_state_dict()
str_keys = [k for k in state_dict.keys() if isinstance(k, str)]
if has_cosmos_dit_kohya_keys(str_keys) or has_cosmos_dit_peft_keys(str_keys):
if has_cosmos_dit_kohya_keys_strict(str_keys) or has_cosmos_dit_peft_keys_strict(str_keys):
return BaseModelType.Anima
raise NotAMatchError("model does not look like an Anima LoRA")
class LoRA_LyCORIS_Wan_Config(LoRA_LyCORIS_Config_Base, Config_Base):
"""Model config for Wan 2.2 LoRA models in LyCORIS format.
Wan LoRAs target ``WanTransformer3DModel`` blocks. The Wan 2.2 A14B family
is dual-expert (high-noise + low-noise) — LoRAs are typically trained
against one expert. ``expert`` records which one so the model loader
invocation can wire it to the correct ``loras`` / ``loras_low_noise`` list.
Many LoRAs are expert-agnostic (TI2V-5B family, or community LoRAs that
just don't tag the expert) — these get ``expert=None`` and are applied to
both experts by default.
"""
base: Literal[BaseModelType.Wan] = Field(default=BaseModelType.Wan)
expert: Literal["high", "low"] | None = Field(
default=None,
description="For Wan 2.2 A14B dual-expert LoRAs: 'high' targets the high-noise expert, "
"'low' targets the low-noise expert. None means the LoRA is expert-agnostic "
"(TI2V-5B, or community LoRAs without explicit tagging) and is applied to both.",
)
variant: WanLoRAVariantType | None = Field(
default=None,
description="The Wan model family this LoRA targets, detected from its inner-dim "
"(5120 -> A14B, 3072 -> TI2V-5B). A14B LoRAs are incompatible with TI2V-5B mains "
"(and vice versa) — they crash with a shape mismatch in the layer patcher. The "
"linear-view graph builder filters LoRAs on variant when building the LoRA "
"collection. None means the LoRA's inner-dim couldn't be identified.",
)
@classmethod
def _validate_looks_like_lora(cls, mod: ModelOnDisk) -> None:
"""Wan LoRAs target attn1/attn2/ffn.net (diffusers form) or self_attn/cross_attn/ffn.N (native form)."""
state_dict = mod.load_state_dict()
str_keys = [k for k in state_dict.keys() if isinstance(k, str)]
has_wan_keys = has_wan_kohya_keys(str_keys) or has_wan_peft_keys(str_keys)
has_lora_suffix = state_dict_has_any_keys_ending_with(
state_dict,
{
"lora_A.weight",
"lora_B.weight",
"lora_down.weight",
"lora_up.weight",
"dora_scale",
".lokr_w1",
".lokr_w2",
},
)
# Reject if any non-Wan architecture signature is present. Without this
# guard a Wan LoRA could be falsely identified by Anima (cross_attn /
# self_attn name collision) or vice versa.
if has_wan_keys and has_lora_suffix and not has_non_wan_architecture_keys(str_keys):
return
raise NotAMatchError("model does not match Wan LoRA heuristics")
@classmethod
def _get_base_or_raise(cls, mod: ModelOnDisk) -> BaseModelType:
state_dict = mod.load_state_dict()
str_keys = [k for k in state_dict.keys() if isinstance(k, str)]
if (has_wan_kohya_keys(str_keys) or has_wan_peft_keys(str_keys)) and not has_non_wan_architecture_keys(
str_keys
):
return BaseModelType.Wan
raise NotAMatchError("model does not look like a Wan LoRA")
@classmethod
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
# Run the base-class probe (file-check, lora-suffix, base detection).
instance = super().from_model_on_disk(mod, override_fields)
# Auto-detect the expert tag from the filename if the user didn't
# override it. ``high_noise`` / ``low_noise`` / hyphenated / concatenated
# variants — mirrors the GGUF transformer probe's heuristic.
if instance.expert is None:
name = mod.path.stem.lower()
if any(s in name for s in ("high_noise", "high-noise", "highnoise")):
instance.expert = "high"
elif any(s in name for s in ("low_noise", "low-noise", "lownoise")):
instance.expert = "low"
# Auto-detect the model-family variant from inner_dim in the state
# dict. The override field skips this if the user has set it.
if instance.variant is None:
instance.variant = detect_wan_lora_variant(mod.load_state_dict())
return instance
class ControlAdapter_Config_Base(ABC, BaseModel):
default_settings: ControlAdapterDefaultSettings | None = Field(None)
+284 -1
View File
@@ -33,6 +33,7 @@ from invokeai.backend.model_manager.taxonomy import (
QwenImageVariantType,
SchedulerPredictionType,
SubModelType,
WanVariantType,
ZImageVariantType,
)
from invokeai.backend.quantization.gguf.ggml_tensor import GGMLTensor
@@ -65,7 +66,12 @@ class MainModelDefaultSettings(BaseModel):
def from_base(
cls,
base: BaseModelType,
variant: Flux2VariantType | FluxVariantType | ModelVariantType | ZImageVariantType | None = None,
variant: Flux2VariantType
| FluxVariantType
| ModelVariantType
| WanVariantType
| ZImageVariantType
| None = None,
) -> Self | None:
match base:
case BaseModelType.StableDiffusion1:
@@ -99,6 +105,12 @@ class MainModelDefaultSettings(BaseModel):
return cls(steps=4, cfg_scale=1.0, width=1024, height=1024)
case BaseModelType.QwenImage:
return cls(steps=40, cfg_scale=4.0, width=1024, height=1024)
case BaseModelType.Wan:
# Wan 2.2 recommended defaults differ by variant.
if variant == WanVariantType.TI2V_5B:
return cls(steps=30, cfg_scale=5.0, width=1024, height=1024)
# Default to A14B settings (also used when variant is unknown).
return cls(steps=40, cfg_scale=4.0, width=1024, height=1024)
case _:
# TODO(psyche): Do we want defaults for other base types?
return None
@@ -1474,6 +1486,277 @@ class Main_GGUF_QwenImage_Config(Checkpoint_Config_Base, Main_Config_Base, Confi
return cls(**override_fields, variant=explicit_variant)
def _has_wan_keys(state_dict: dict[str | int, Any]) -> bool:
"""Check if state dict contains Wan 2.2 transformer keys.
Two layouts are accepted:
* **Diffusers** (city96-style GGUF, Wan-AI/*-Diffusers safetensors): the text
projection is named ``condition_embedder.text_embedder.linear_1``.
* **Native upstream** (QuantStack-style GGUF, ComfyUI, Wan-AI's non-Diffusers
releases): the text projection is named ``text_embedding.0``.
Both layouts share ``patch_embedding.weight`` as the input conv. Combined with
the text-projection fingerprint, this won't collide with FLUX
(``double_blocks/single_blocks``), Qwen Image (``txt_in/img_in``), Z-Image
(``cap_embedder``), or Anima (``llm_adapter``).
Tolerates both bare keys and the ComfyUI ``model.diffusion_model.`` /
``diffusion_model.`` prefixes.
"""
text_proj_options = (
"condition_embedder.text_embedder.linear_1.weight",
"text_embedding.0.weight",
)
prefixes = ("", "model.diffusion_model.", "diffusion_model.")
keys = state_dict.keys()
if not any((p + "patch_embedding.weight") in keys for p in prefixes):
return False
return any((p + needle) in keys for p in prefixes for needle in text_proj_options)
def _is_native_wan_layout(state_dict: dict[str | int, Any]) -> bool:
"""True if the state dict uses the native upstream Wan key layout.
Native layout uses ``text_embedding.0/2``, ``self_attn``/``cross_attn``,
``ffn.0/2``, ``head.head``, ``head.modulation``, etc. — what ComfyUI and
QuantStack ship. Diffusers layout uses ``condition_embedder.*``, ``attn1``/
``attn2``, ``ffn.net.*``, ``proj_out``, ``scale_shift_table``.
"""
prefixes = ("", "model.diffusion_model.", "diffusion_model.")
keys = state_dict.keys()
return any((p + "text_embedding.0.weight") in keys for p in prefixes)
def _detect_wan_gguf_variant(state_dict: dict[str | int, Any]) -> WanVariantType | None:
"""Determine A14B (T2V vs I2V) vs TI2V-5B from the GGUF state dict.
``patch_embedding.weight`` has shape ``[inner_dim, in_channels, T, H, W]``;
``in_channels`` uniquely identifies the Wan 2.2 variant:
- 16 → T2V-A14B (noise latents only).
- 36 → I2V-A14B (16 noise + 16 ref-image latents + 4 first-frame mask,
concatenated along the channel dim — see diffusers
``WanImageToVideoPipeline.prepare_latents``).
- 48 → TI2V-5B (Wan2.2-VAE z_dim=48).
Returns None if the tensor is missing or the channel count is unrecognised.
"""
candidates = (
"patch_embedding.weight",
"model.diffusion_model.patch_embedding.weight",
"diffusion_model.patch_embedding.weight",
)
for key in candidates:
if key in state_dict:
tensor = state_dict[key]
shape = getattr(tensor, "tensor_shape", None) or getattr(tensor, "shape", None)
if shape is None or len(shape) < 2:
return None
in_channels = int(shape[1])
if in_channels == 16:
return WanVariantType.T2V_A14B
if in_channels == 36:
return WanVariantType.I2V_A14B
if in_channels == 48:
return WanVariantType.TI2V_5B
return None
return None
def _detect_wan_gguf_expert(filename: str) -> Literal["high", "low", "none"]:
"""Filename heuristic for the A14B dual-expert MoE.
Community releases tag each expert in the filename — typically
``high_noise`` / ``low_noise`` (or hyphenated/concatenated variants).
Returns 'none' when neither marker is present (single-expert model or
ambiguous filename).
"""
name = filename.lower()
if any(s in name for s in ("high_noise", "high-noise", "highnoise")):
return "high"
if any(s in name for s in ("low_noise", "low-noise", "lownoise")):
return "low"
return "none"
class Main_GGUF_Wan_Config(Checkpoint_Config_Base, Main_Config_Base, Config_Base):
"""Model config for GGUF-quantized Wan 2.2 transformer models.
A14B's MoE ships as two GGUF files (one per expert); ``expert`` records
which one this is so the model loader invocation can pair them. TI2V-5B
is a single-transformer model and stores ``expert='none'``.
"""
base: Literal[BaseModelType.Wan] = Field(default=BaseModelType.Wan)
format: Literal[ModelFormat.GGUFQuantized] = Field(default=ModelFormat.GGUFQuantized)
variant: WanVariantType = Field()
expert: Literal["high", "low", "none"] = Field(
default="none",
description="For Wan 2.2 A14B's dual-expert MoE: 'high' for the high-noise expert, "
"'low' for the low-noise expert. 'none' for single-transformer models (TI2V-5B).",
)
@classmethod
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
raise_if_not_file(mod)
raise_for_override_fields(cls, override_fields)
sd = mod.load_state_dict()
if not _has_ggml_tensors(sd):
raise NotAMatchError("state dict does not look like GGUF quantized")
if not _has_wan_keys(sd):
raise NotAMatchError("state dict does not look like a Wan transformer")
gguf_name = mod.metadata().get("general.name", "")
normalized_identity = "".join(
character for character in f"{mod.path.stem} {gguf_name}".lower() if character.isalnum()
)
if "wan21" in normalized_identity:
raise NotAMatchError("Wan 2.1 GGUF models are not supported by the Wan 2.2 loader")
explicit_variant = override_fields.pop("variant", None)
variant = explicit_variant or _detect_wan_gguf_variant(sd)
if variant is None:
raise NotAMatchError("could not determine Wan variant from state dict")
if variant in (WanVariantType.T2V_A14B, WanVariantType.I2V_A14B) and "wan22" not in normalized_identity:
raise NotAMatchError("Wan A14B GGUF filename or metadata must identify the model as Wan 2.2")
explicit_expert = override_fields.pop("expert", None)
expert = explicit_expert or _detect_wan_gguf_expert(mod.path.stem)
return cls(**override_fields, variant=variant, expert=expert)
class Main_Diffusers_Wan_Config(Diffusers_Config_Base, Main_Config_Base, Config_Base):
"""Model config for Wan 2.2 diffusers models.
Covers both the dual-expert T2V-A14B family and the single-transformer TI2V-5B
family. Variant is detected from the on-disk transformer config (latent channel
count) plus the presence of a sibling ``transformer_2/`` directory.
"""
base: Literal[BaseModelType.Wan] = Field(default=BaseModelType.Wan)
variant: WanVariantType = Field()
has_dual_expert: bool = Field(
default=False,
description="Whether this model ships two transformer experts (Wan 2.2 A14B MoE). False for TI2V-5B.",
)
boundary_ratio: float | None = Field(
default=None,
description="MoE expert switch point as a fraction of num_train_timesteps (typically 1000). "
"None for single-transformer models. Read from model_index.json by Diffusers' WanPipeline.",
)
@classmethod
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
raise_if_not_dir(mod)
raise_for_override_fields(cls, override_fields)
# Wan repos ship with WanPipeline (T2V) or WanImageToVideoPipeline (I2V/TI2V).
# Either class name is sufficient to identify a Wan diffusers model.
raise_for_class_name(
common_config_paths(mod.path),
{
"WanPipeline",
"WanImageToVideoPipeline",
},
)
repo_variant = override_fields.pop("repo_variant", None) or cls._get_repo_variant_or_raise(mod)
explicit_variant = override_fields.pop("variant", None)
has_dual_expert = (mod.path / "transformer_2" / "config.json").exists()
variant = explicit_variant or cls._detect_wan_variant(mod, has_dual_expert)
boundary_ratio = override_fields.pop("boundary_ratio", None)
if boundary_ratio is None:
boundary_ratio = cls._read_boundary_ratio(mod)
return cls(
**override_fields,
repo_variant=repo_variant,
variant=variant,
has_dual_expert=has_dual_expert,
boundary_ratio=boundary_ratio,
)
@classmethod
def _read_boundary_ratio(cls, mod: ModelOnDisk) -> float | None:
"""Pull ``boundary_ratio`` from ``model_index.json`` if present.
Diffusers' ``WanPipeline.__init__`` registers it via ``register_to_config``,
which persists it as a top-level key in the saved pipeline config.
"""
try:
model_index = get_config_dict_or_raise(mod.path / "model_index.json")
except NotAMatchError:
return None
value = model_index.get("boundary_ratio")
if value is None:
return None
try:
return float(value)
except (TypeError, ValueError):
return None
@classmethod
def _detect_wan_variant(cls, mod: ModelOnDisk, has_dual_expert: bool) -> WanVariantType:
"""Detect Wan variant from transformer + VAE config.
- T2V-A14B: dual transformer experts, standard Wan VAE (z_dim=16),
transformer ``in_channels=16`` (text-only conditioning).
- I2V-A14B: dual transformer experts, standard Wan VAE,
transformer ``in_channels=36`` (text + VAE-encoded reference image
+ first-frame mask concatenated along the channel dim).
- TI2V-5B: single transformer, Wan2.2-VAE (z_dim=48).
"""
if has_dual_expert:
# Disambiguate T2V vs I2V via the transformer's input channel count.
# Wan 2.2 I2V uses VAE-latent concatenation: 16 noise + 16 ref-image
# latents + 4 first-frame mask = 36. (Wan 2.1 I2V used CLIP-vision
# via ``image_dim``; that mechanism is absent in Wan 2.2.)
in_channels = cls._transformer_in_channels(mod)
if in_channels == 36:
return WanVariantType.I2V_A14B
return WanVariantType.T2V_A14B
# Single-transformer model: distinguish TI2V-5B from any future single-expert
# A14B-derived release by inspecting the VAE latent dimension.
try:
vae_config = get_config_dict_or_raise(mod.path / "vae" / "config.json")
z_dim = vae_config.get("z_dim")
if z_dim is not None and int(z_dim) >= 32:
return WanVariantType.TI2V_5B
except NotAMatchError:
# No VAE config to inspect — fall through to the heuristic path below.
pass
# Filename / repo-name heuristic as a last resort.
name = mod.path.name.lower()
if "5b" in name or "ti2v" in name:
return WanVariantType.TI2V_5B
return WanVariantType.T2V_A14B
@staticmethod
def _transformer_in_channels(mod: ModelOnDisk) -> int | None:
"""Read ``in_channels`` from ``transformer/config.json``.
For Wan 2.2 A14B, this is the canonical discriminator between T2V
(``in_channels=16``) and I2V (``in_channels=36``). Returns None if the
config can't be read.
"""
try:
transformer_config = get_config_dict_or_raise(mod.path / "transformer" / "config.json")
except NotAMatchError:
return None
value = transformer_config.get("in_channels")
try:
return int(value) if value is not None else None
except (TypeError, ValueError):
return None
class Main_Checkpoint_Anima_Config(Checkpoint_Config_Base, Main_Config_Base, Config_Base):
"""Model config for Anima single-file checkpoint models (safetensors).
+126 -3
View File
@@ -40,6 +40,11 @@ def _is_qwen_image_vae(state_dict: dict[str | int, Any]) -> bool:
1. Diffusers-format encoder/decoder keys (`encoder.conv_in`, `decoder.conv_in`)
2. 5-dimensional convolution weights (3D causal convolutions vs. standard 2D conv in SD/SDXL/FLUX VAEs)
3. 16-dimensional latent space (z_dim=16)
Note: Wan 2.2 A14B reuses the same architecture (AutoencoderKLWan with z_dim=16),
so this function returns True for both. Disambiguation between the two for
standalone files relies on the filename heuristic in :func:`_is_wan_vae` and
config registration order.
"""
decoder_conv_in_key = "decoder.conv_in.weight"
if decoder_conv_in_key not in state_dict:
@@ -52,6 +57,34 @@ def _is_qwen_image_vae(state_dict: dict[str | int, Any]) -> bool:
return shape[1] == 16
def _wan_vae_z_dim(state_dict: dict[str | int, Any]) -> int | None:
"""Return ``z_dim`` for a Wan-family VAE state dict, or ``None`` if it isn't one.
Wan-family VAEs (AutoencoderKLWan) have 5D convolution weights and a
decoder.conv_in input channel count of 16 (Wan 2.1 / A14B / Qwen Image) or
48 (Wan 2.2 TI2V-5B's Wan2.2-VAE).
"""
decoder_conv_in_key = "decoder.conv_in.weight"
if decoder_conv_in_key not in state_dict:
return None
weight = state_dict[decoder_conv_in_key]
shape = getattr(weight, "shape", None)
if shape is None or len(shape) != 5:
return None
z = int(shape[1])
return z if z in (16, 48) else None
def _filename_suggests_wan(mod: ModelOnDisk) -> bool:
"""Filename heuristic to distinguish standalone Wan VAE files from Qwen Image VAEs.
Both use the same ``AutoencoderKLWan`` architecture for 16-channel files, so the
state dict alone can't tell them apart. Filenames in the wild (community ports,
ComfyUI repacks) typically include ``wan`` for Wan releases.
"""
return "wan" in mod.path.name.lower()
def _is_flux2_vae(state_dict: dict[str | int, Any]) -> bool:
"""Check if state dict is a FLUX.2 VAE (AutoencoderKLFlux2).
@@ -114,9 +147,10 @@ class VAE_Checkpoint_Config_Base(Checkpoint_Config_Base):
if _is_flux2_vae(state_dict):
raise NotAMatchError("model is a FLUX.2 VAE, not a standard VAE")
# Exclude Qwen Image VAEs - they have their own config class
if _is_qwen_image_vae(state_dict):
raise NotAMatchError("model is a Qwen Image VAE, not a standard VAE")
# Exclude Qwen Image / Wan VAEs - they share the AutoencoderKLWan
# architecture and each has its own config class.
if _is_qwen_image_vae(state_dict) or _wan_vae_z_dim(state_dict) is not None:
raise NotAMatchError("model is a Wan-family VAE, not a standard VAE")
@classmethod
def _get_base_or_raise(cls, mod: ModelOnDisk) -> BaseModelType:
@@ -218,9 +252,98 @@ class VAE_Checkpoint_QwenImage_Config(Checkpoint_Config_Base, Config_Base):
if not _is_qwen_image_vae(state_dict):
raise NotAMatchError("state dict does not look like a Qwen Image VAE")
# Defer to VAE_Checkpoint_Wan_Config for files whose names indicate Wan
# (both architectures are 16-channel AutoencoderKLWan and otherwise
# indistinguishable from the state dict alone).
if _filename_suggests_wan(mod):
raise NotAMatchError("filename suggests a Wan VAE, not Qwen Image")
return cls(**override_fields)
class VAE_Checkpoint_Wan_Config(Checkpoint_Config_Base, Config_Base):
"""Model config for Wan 2.2 VAE checkpoint models (AutoencoderKLWan).
Distinguishes A14B (z_dim=16, standard Wan VAE) from TI2V-5B (z_dim=48,
Wan2.2-VAE) via the input channel count of ``decoder.conv_in.weight``.
"""
type: Literal[ModelType.VAE] = Field(default=ModelType.VAE)
format: Literal[ModelFormat.Checkpoint] = Field(default=ModelFormat.Checkpoint)
base: Literal[BaseModelType.Wan] = Field(default=BaseModelType.Wan)
cpu_only: bool | None = Field(default=None, description="Whether this model should run on CPU only")
latent_channels: Literal[16, 48] = Field(
description="VAE latent channel count: 16 for A14B (standard Wan VAE) or 48 for TI2V-5B (Wan2.2-VAE)."
)
@classmethod
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
raise_if_not_file(mod)
raise_for_override_fields(cls, override_fields)
state_dict = mod.load_state_dict()
z_dim = _wan_vae_z_dim(state_dict)
if z_dim is None:
raise NotAMatchError("state dict does not look like a Wan VAE")
# 48-channel files are unambiguously Wan2.2-VAE (TI2V-5B). 16-channel
# files are architecturally identical to Qwen Image's VAE; require the
# filename to suggest Wan to claim them, otherwise let the QwenImage
# config win.
latent_channels: int = z_dim
if latent_channels == 16 and not _filename_suggests_wan(mod):
raise NotAMatchError(
"16-channel AutoencoderKLWan VAE without 'wan' in filename — deferring to Qwen Image VAE config."
)
explicit = override_fields.pop("latent_channels", None)
if explicit is not None:
latent_channels = int(explicit)
return cls(**override_fields, latent_channels=latent_channels)
class VAE_Diffusers_Wan_Config(Diffusers_Config_Base, Config_Base):
"""Model config for Wan 2.2 VAE in diffusers folder layout (AutoencoderKLWan)."""
type: Literal[ModelType.VAE] = Field(default=ModelType.VAE)
format: Literal[ModelFormat.Diffusers] = Field(default=ModelFormat.Diffusers)
base: Literal[BaseModelType.Wan] = Field(default=BaseModelType.Wan)
cpu_only: bool | None = Field(default=None, description="Whether this model should run on CPU only")
latent_channels: Literal[16, 48] = Field(
default=16,
description="VAE latent channel count: 16 for A14B or 48 for TI2V-5B's Wan2.2-VAE.",
)
@classmethod
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
raise_if_not_dir(mod)
raise_for_override_fields(cls, override_fields)
raise_for_class_name(
common_config_paths(mod.path),
{"AutoencoderKLWan"},
)
# Read z_dim from the diffusers config to set latent_channels.
latent_channels: int = 16
try:
config = get_config_dict_or_raise(common_config_paths(mod.path))
z = config.get("z_dim")
if z is not None and int(z) in (16, 48):
latent_channels = int(z)
except NotAMatchError:
pass
explicit = override_fields.pop("latent_channels", None)
if explicit is not None:
latent_channels = int(explicit)
return cls(**override_fields, latent_channels=latent_channels)
def _has_anima_vae_keys(state_dict: dict[str | int, Any]) -> bool:
"""Check if state dict looks like an Anima QwenImage VAE (AutoencoderKLQwenImage).
@@ -0,0 +1,86 @@
"""Configurations for the UMT5-XXL text encoder used by Wan 2.2.
Wan ships a UMT5-XXL encoder (not the more common T5-XXL). The two are not
weight-compatible — UMT5 has a different vocabulary and ``model_type``. We
register a dedicated config + ModelType so users can't accidentally wire a
FLUX/SD3-style T5-XXL into a Wan slot.
For Phase 3 we accept the diffusers-folder layout only. Single-file UMT5
checkpoints are uncommon; if they show up later, a checkpoint config can be
added alongside this one.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any, Literal, Self
from pydantic import Field
from invokeai.backend.model_manager.configs.base import Config_Base
from invokeai.backend.model_manager.configs.identification_utils import (
NotAMatchError,
raise_for_override_fields,
raise_if_not_dir,
)
from invokeai.backend.model_manager.model_on_disk import ModelOnDisk
from invokeai.backend.model_manager.taxonomy import BaseModelType, ModelFormat, ModelType
def _read_text_encoder_model_type(mod: ModelOnDisk) -> str | None:
"""Return ``model_type`` from the encoder's ``config.json``.
Diffusers encoder folders may live at the root (``config.json``) or under a
``text_encoder/`` subdirectory. UMT5-XXL sets ``model_type`` to ``"umt5"``;
a regular T5-XXL would be ``"t5"``.
"""
candidates: list[Path] = [mod.path / "text_encoder" / "config.json", mod.path / "config.json"]
for path in candidates:
if path.exists():
try:
with path.open("r", encoding="utf-8") as f:
config = json.load(f)
except (json.JSONDecodeError, OSError):
continue
mt = config.get("model_type")
if isinstance(mt, str):
return mt.lower()
return None
class WanT5Encoder_WanT5Encoder_Config(Config_Base):
"""UMT5-XXL encoder in diffusers folder layout.
Accepts either:
- A directory containing ``text_encoder/`` (and typically ``tokenizer/``) ─ the
shape produced by ``Wan-AI/Wan2.2-T2V-A14B::text_encoder+tokenizer``.
- A bare ``text_encoder/`` directory whose own ``config.json`` declares
``model_type: umt5``.
"""
base: Literal[BaseModelType.Any] = Field(default=BaseModelType.Any)
type: Literal[ModelType.WanT5Encoder] = Field(default=ModelType.WanT5Encoder)
format: Literal[ModelFormat.WanT5Encoder] = Field(default=ModelFormat.WanT5Encoder)
cpu_only: bool | None = Field(default=None, description="Whether this model should run on CPU only")
@classmethod
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
raise_if_not_dir(mod)
raise_for_override_fields(cls, override_fields)
# Refuse to claim full Wan pipelines — they should match Main_Diffusers_Wan_Config.
if (mod.path / "model_index.json").exists() or (mod.path / "transformer").exists():
raise NotAMatchError(
"directory looks like a full Wan pipeline (model_index.json or transformer/), "
"not a standalone Wan T5 encoder"
)
model_type = _read_text_encoder_model_type(mod)
if model_type is None:
raise NotAMatchError("no encoder config.json found at root or text_encoder/")
if model_type != "umt5":
raise NotAMatchError(f"encoder model_type is {model_type!r}, not 'umt5'")
return cls(**override_fields)
@@ -63,6 +63,7 @@ from invokeai.backend.patches.lora_conversions.qwen_image_lora_conversion_utils
)
from invokeai.backend.patches.lora_conversions.sd_lora_conversion_utils import lora_model_from_sd_state_dict
from invokeai.backend.patches.lora_conversions.sdxl_lora_conversion_utils import convert_sdxl_keys_to_diffusers_format
from invokeai.backend.patches.lora_conversions.wan_lora_conversion_utils import lora_model_from_wan_state_dict
from invokeai.backend.patches.lora_conversions.z_image_lora_conversion_utils import lora_model_from_z_image_state_dict
@@ -175,6 +176,10 @@ class LoRALoader(ModelLoader):
elif self._model_base == BaseModelType.Anima:
# Anima LoRAs use Kohya-style or diffusers PEFT format targeting Cosmos DiT blocks.
model = lora_model_from_anima_state_dict(state_dict=state_dict, alpha=None)
elif self._model_base == BaseModelType.Wan:
# Wan LoRAs use Kohya / diffusers PEFT / native PEFT formats targeting
# WanTransformer3DModel attention (attn1/attn2) and FFN blocks.
model = lora_model_from_wan_state_dict(state_dict=state_dict, alpha=None)
else:
raise ValueError(f"Unsupported LoRA base model: {self._model_base}")
@@ -10,6 +10,8 @@ from invokeai.backend.model_manager.configs.vae import (
VAE_Checkpoint_Anima_Config,
VAE_Checkpoint_Config_Base,
VAE_Checkpoint_QwenImage_Config,
VAE_Checkpoint_Wan_Config,
VAE_Diffusers_Wan_Config,
)
from invokeai.backend.model_manager.load.model_loader_registry import ModelLoaderRegistry
from invokeai.backend.model_manager.load.model_loaders.generic_diffusers import GenericDiffusersLoader
@@ -21,6 +23,133 @@ from invokeai.backend.model_manager.taxonomy import (
SubModelType,
)
# Architectural defaults for the Wan 2.2-VAE (TI2V-5B). Verbatim from the
# vae/config.json shipped with Wan-AI/Wan2.2-TI2V-5B-Diffusers — only the
# values that differ from diffusers' AutoencoderKLWan defaults are listed.
# latents_mean / latents_std are required because the model normalises latents
# against them at encode/decode time; the wrong arrays produce silent garbage.
_WAN_TI2V_5B_VAE_CONFIG: dict = {
"base_dim": 160,
"decoder_base_dim": 256,
"z_dim": 48,
"in_channels": 12,
"out_channels": 12,
"patch_size": 2,
"scale_factor_spatial": 16,
"is_residual": True,
"latents_mean": [
-0.2289,
-0.0052,
-0.1323,
-0.2339,
-0.2799,
0.0174,
0.1838,
0.1557,
-0.1382,
0.0542,
0.2813,
0.0891,
0.1570,
-0.0098,
0.0375,
-0.1825,
-0.2246,
-0.1207,
-0.0698,
0.5109,
0.2665,
-0.2108,
-0.2158,
0.2502,
-0.2055,
-0.0322,
0.1109,
0.1567,
-0.0729,
0.0899,
-0.2799,
-0.1230,
-0.0313,
-0.1649,
0.0117,
0.0723,
-0.2839,
-0.2083,
-0.0520,
0.3748,
0.0152,
0.1957,
0.1433,
-0.2944,
0.3573,
-0.0548,
-0.1681,
-0.0667,
],
"latents_std": [
0.4765,
1.0364,
0.4514,
1.1677,
0.5313,
0.4990,
0.4818,
0.5013,
0.8158,
1.0344,
0.5894,
1.0901,
0.6885,
0.6165,
0.8454,
0.4978,
0.5759,
0.3523,
0.7135,
0.6804,
0.5833,
1.4146,
0.8986,
0.5659,
0.7069,
0.5338,
0.4889,
0.4917,
0.4069,
0.4999,
0.6866,
0.4093,
0.5709,
0.6065,
0.6415,
0.4944,
0.5726,
1.2042,
0.5458,
1.6887,
0.3971,
1.0600,
0.3943,
0.5537,
0.5444,
0.4089,
0.7468,
0.7744,
],
}
def _wan_vae_init_kwargs_for(latent_channels: int) -> dict:
"""Return the AutoencoderKLWan constructor kwargs for a given z_dim.
z_dim=48 means TI2V-5B's Wan 2.2-VAE (different base dim, patchified IO,
16x spatial). Anything else falls back to the A14B / Wan 2.1 defaults.
"""
if latent_channels == 48:
return dict(_WAN_TI2V_5B_VAE_CONFIG)
return {"z_dim": latent_channels}
@ModelLoaderRegistry.register(base=BaseModelType.Any, type=ModelType.VAE, format=ModelFormat.Diffusers)
@ModelLoaderRegistry.register(base=BaseModelType.Any, type=ModelType.VAE, format=ModelFormat.Checkpoint)
@@ -35,10 +164,17 @@ class VAELoader(GenericDiffusersLoader):
if isinstance(config, VAE_Checkpoint_Anima_Config):
from diffusers.models.autoencoders import AutoencoderKLWan
from invokeai.backend.wan.rocm_causal_conv3d import patch_wan_causal_conv3d_for_rocm
patch_wan_causal_conv3d_for_rocm()
return AutoencoderKLWan.from_single_file(
config.path,
torch_dtype=self._torch_dtype,
)
elif isinstance(config, VAE_Checkpoint_Wan_Config):
return self._load_wan_vae(config)
elif isinstance(config, VAE_Diffusers_Wan_Config):
return self._load_wan_vae_diffusers(config)
elif isinstance(config, VAE_Checkpoint_QwenImage_Config):
return self._load_qwen_image_vae(config)
elif isinstance(config, VAE_Checkpoint_Config_Base):
@@ -49,6 +185,77 @@ class VAELoader(GenericDiffusersLoader):
else:
return super()._load_model(config, submodel_type)
def _load_wan_vae(self, config: VAE_Checkpoint_Wan_Config) -> AnyModel:
"""Load a Wan 2.2 VAE from a single safetensors file.
Picks the correct ``AutoencoderKLWan`` config based on ``z_dim``. The Wan
ecosystem ships two distinct VAE architectures:
* ``z_dim=16`` — the Wan 2.1 / Wan 2.2 A14B VAE. Diffusers' defaults match
this one (base_dim=96, 8x spatial, no patchify, 3 in/out channels).
* ``z_dim=48`` — the Wan 2.2-VAE used by TI2V-5B. Larger (base_dim=160,
decoder_base_dim=256), 16x spatial, patchify with patch_size=2 (so
in/out channels are 12 = 3 RGB x 2x2 patch), residual blocks, and
its own latents_mean / latents_std.
Without overriding those params at construction time, the state dict
from the TI2V-5B VAE checkpoint won't load (channel and shape mismatches
throughout the encoder + decoder).
Forces bfloat16 (same as ``WanDiffusersModel`` and ``_load_wan_vae_diffusers``) —
fp16 is unstable on the Wan VAE, and the default ``precision: auto`` resolves to
float16 on CUDA.
"""
import accelerate
import torch
from diffusers.models.autoencoders.autoencoder_kl_wan import AutoencoderKLWan
from safetensors.torch import load_file
from invokeai.backend.wan.rocm_causal_conv3d import patch_wan_causal_conv3d_for_rocm
patch_wan_causal_conv3d_for_rocm()
sd = load_file(config.path)
for k in list(sd.keys()):
if sd[k].is_floating_point():
sd[k] = sd[k].to(torch.bfloat16)
new_sd_size = sum(t.nelement() * t.element_size() for t in sd.values())
self._ram_cache.make_room(new_sd_size)
init_kwargs = _wan_vae_init_kwargs_for(config.latent_channels)
with accelerate.init_empty_weights():
model = AutoencoderKLWan(**init_kwargs)
model.load_state_dict(sd, strict=True, assign=True)
model.eval()
return model
def _load_wan_vae_diffusers(self, config: VAE_Diffusers_Wan_Config) -> AnyModel:
"""Load a Wan 2.2 VAE from a flat diffusers folder (AutoencoderKLWan).
The standalone install ``Wan-AI/Wan2.2-T2V-A14B-Diffusers::vae`` lands as a
single-class folder (``config.json`` + ``diffusion_pytorch_model.safetensors``,
no ``model_index.json``). The generic loader rejects this when a
``submodel_type`` is requested — we always pass ``SubModelType.VAE`` from
the model loader invocation since that's how cached entries are keyed.
Loading ``AutoencoderKLWan`` directly here sidesteps the submodel check.
Forces bfloat16 (same as ``WanDiffusersModel``) — fp16 is unstable on the
Wan VAE.
"""
import torch
from diffusers.models.autoencoders.autoencoder_kl_wan import AutoencoderKLWan
from invokeai.backend.wan.rocm_causal_conv3d import patch_wan_causal_conv3d_for_rocm
patch_wan_causal_conv3d_for_rocm()
return AutoencoderKLWan.from_pretrained(
config.path,
torch_dtype=torch.bfloat16,
local_files_only=True,
)
def _load_qwen_image_vae(self, config: VAE_Checkpoint_QwenImage_Config) -> AnyModel:
"""Load a Qwen Image VAE from a single safetensors file.
@@ -0,0 +1,357 @@
"""Loader registrations for Wan 2.2 image-generation models.
Currently covers:
- Main: Diffusers format (T2V-A14B with dual experts via Transformer +
Transformer2 submodels, plus TI2V-5B). Phase 4 will add a GGUFQuantized loader.
- WanT5Encoder: standalone UMT5-XXL encoder folder (``text_encoder/`` +
``tokenizer/`` subdirs, or a flat ``text_encoder/`` folder).
- VAE: handled in ``vae.py`` (registered for type=VAE generically).
"""
from pathlib import Path
from typing import Optional
import torch
from invokeai.backend.model_manager.configs.base import Checkpoint_Config_Base, Diffusers_Config_Base
from invokeai.backend.model_manager.configs.factory import AnyModelConfig
from invokeai.backend.model_manager.configs.main import Main_GGUF_Wan_Config, _is_native_wan_layout
from invokeai.backend.model_manager.load.load_default import ModelLoader
from invokeai.backend.model_manager.load.model_loader_registry import ModelLoaderRegistry
from invokeai.backend.model_manager.load.model_loaders.generic_diffusers import GenericDiffusersLoader
from invokeai.backend.model_manager.taxonomy import (
AnyModel,
BaseModelType,
ModelFormat,
ModelType,
SubModelType,
WanVariantType,
)
from invokeai.backend.quantization.gguf.ggml_tensor import GGMLTensor
from invokeai.backend.quantization.gguf.loaders import gguf_sd_loader
from invokeai.backend.quantization.gguf.utils import TORCH_COMPATIBLE_QTYPES
from invokeai.backend.util.devices import TorchDevice
@ModelLoaderRegistry.register(base=BaseModelType.Wan, type=ModelType.Main, format=ModelFormat.Diffusers)
class WanDiffusersModel(GenericDiffusersLoader):
"""Loader for Wan 2.2 diffusers-format models (T2V-A14B and TI2V-5B).
Forces bfloat16 for the transformer and VAE — fp16 is unstable on Wan VAE
(same issue affects the Flux VAE). Resolves the appropriate Hugging Face
class for each submodel via the parent loader's ``get_hf_load_class``.
"""
def _load_model(
self,
config: AnyModelConfig,
submodel_type: Optional[SubModelType] = None,
) -> AnyModel:
if isinstance(config, Checkpoint_Config_Base):
raise NotImplementedError("Single-file checkpoint format is not yet supported for Wan models.")
if submodel_type is None:
raise Exception("A submodel type must be provided when loading Wan main pipelines.")
if submodel_type is SubModelType.VAE:
from invokeai.backend.wan.rocm_causal_conv3d import patch_wan_causal_conv3d_for_rocm
patch_wan_causal_conv3d_for_rocm()
model_path = Path(config.path)
load_class = self.get_hf_load_class(model_path, submodel_type)
repo_variant = config.repo_variant if isinstance(config, Diffusers_Config_Base) else None
variant = repo_variant.value if repo_variant else None
model_path = model_path / submodel_type.value
def _load_with_variant_fallback(dtype_kwarg: dict[str, torch.dtype]) -> AnyModel:
# Some Wan repos ship without a fp16 variant suffix on every submodel.
# If the requested variant isn't on disk, fall back to the default weights.
try:
return load_class.from_pretrained(
model_path,
**dtype_kwarg,
variant=variant,
local_files_only=True,
)
except OSError as e:
if variant and "no file named" in str(e):
return load_class.from_pretrained(model_path, **dtype_kwarg, local_files_only=True)
raise
# bfloat16 across the board: matches Diffusers WanPipeline reference and
# avoids the fp16 instability seen in the Wan VAE.
try:
result: AnyModel = _load_with_variant_fallback({"dtype": torch.bfloat16})
except TypeError:
# Older diffusers releases use torch_dtype instead of dtype.
result = _load_with_variant_fallback({"torch_dtype": torch.bfloat16})
return result
# Native (upstream) -> Diffusers key rename rules.
#
# Mirrors diffusers.loaders.single_file_utils.convert_wan_transformer_to_diffusers
# (T2V subset; we don't ship VACE / motion / face-adapter conversion). Order
# matters — `cross_attn`/`self_attn` must come before `.q. .k. .v. .o.` so the
# attention blocks are renamed before the projection suffix swap. The norm2/3
# swap uses a placeholder to avoid collisions during the substring rewrite.
_WAN_NATIVE_TO_DIFFUSERS_RENAMES: tuple[tuple[str, str], ...] = (
("time_embedding.0", "condition_embedder.time_embedder.linear_1"),
("time_embedding.2", "condition_embedder.time_embedder.linear_2"),
("text_embedding.0", "condition_embedder.text_embedder.linear_1"),
("text_embedding.2", "condition_embedder.text_embedder.linear_2"),
("time_projection.1", "condition_embedder.time_proj"),
("cross_attn", "attn2"),
("self_attn", "attn1"),
(".o.", ".to_out.0."),
(".q.", ".to_q."),
(".k.", ".to_k."),
(".v.", ".to_v."),
(".k_img.", ".add_k_proj."),
(".v_img.", ".add_v_proj."),
(".norm_k_img.", ".norm_added_k."),
("head.modulation", "scale_shift_table"),
("head.head", "proj_out"),
("modulation", "scale_shift_table"),
("ffn.0", "ffn.net.0.proj"),
("ffn.2", "ffn.net.2"),
# norm2 <-> norm3 swap via placeholder
("norm2", "norm__placeholder"),
("norm3", "norm2"),
("norm__placeholder", "norm3"),
# I2V-only keys (harmless on T2V)
("img_emb.proj.0", "condition_embedder.image_embedder.norm1"),
("img_emb.proj.1", "condition_embedder.image_embedder.ff.net.0.proj"),
("img_emb.proj.3", "condition_embedder.image_embedder.ff.net.2"),
("img_emb.proj.4", "condition_embedder.image_embedder.norm2"),
)
def _convert_wan_native_to_diffusers(state_dict: dict) -> dict:
"""Rename native upstream Wan keys (ComfyUI / QuantStack) to diffusers names.
Pure substring replacement — no tensor manipulation — so it's safe to apply
to a dict of GGMLTensors. Returns a new dict; the input is not mutated.
"""
converted: dict = {}
for key, value in state_dict.items():
if not isinstance(key, str):
converted[key] = value
continue
new_key = key
for needle, replacement in _WAN_NATIVE_TO_DIFFUSERS_RENAMES:
new_key = new_key.replace(needle, replacement)
converted[new_key] = value
return converted
def _unwrap_unquantized_to_compute_dtype(state_dict: dict) -> dict:
"""Replace non-quantized GGMLTensor entries with plain tensors at compute_dtype.
Why: QuantStack-style GGUFs store biases (and other small tensors) as F16,
while Wan's ``patch_embedding`` is an ``nn.Conv3d``. ``conv3d`` isn't in
GGMLTensor's dispatch table, so PyTorch reads the wrapper's underlying F16
storage directly and crashes against bf16 latents
(``Input type (c10::BFloat16) and bias type (c10::Half) should be the same``).
For compatible qtypes (F16/F32/BF16) we just pre-cast to compute_dtype here —
they're not quantized, there's no benefit to keeping them wrapped, and
unwrapping them sidesteps the missing-op problem entirely. Genuinely
quantized tensors (Q4_K, Q6_K, etc.) stay wrapped — their on-demand
dequantization through the linear/addmm dispatch path still works.
"""
unwrapped: dict = {}
for key, value in state_dict.items():
if isinstance(value, GGMLTensor) and value._ggml_quantization_type in TORCH_COMPATIBLE_QTYPES:
# GGMLTensor.get_dequantized_tensor() already casts to compute_dtype.
unwrapped[key] = value.get_dequantized_tensor()
else:
unwrapped[key] = value
return unwrapped
@ModelLoaderRegistry.register(base=BaseModelType.Wan, type=ModelType.Main, format=ModelFormat.GGUFQuantized)
class WanGGUFCheckpointModel(ModelLoader):
"""Loader for GGUF-quantized Wan 2.2 transformer models.
The community typically distributes Wan A14B as two files (one per expert
— high-noise + low-noise). Each file is loaded independently here; the
pairing happens at the WanModelLoaderInvocation layer. TI2V-5B ships as a
single file.
Mirrors the QwenImage GGUF loader pattern: ``gguf_sd_loader`` -> strip the
ComfyUI ``model.diffusion_model.`` / ``diffusion_model.`` prefix if present
-> auto-detect arch from state-dict shapes -> ``init_empty_weights`` +
``load_state_dict(strict=False, assign=True)``.
"""
def _load_model(
self,
config: AnyModelConfig,
submodel_type: Optional[SubModelType] = None,
) -> AnyModel:
if not isinstance(config, Main_GGUF_Wan_Config):
raise TypeError(f"Expected Main_GGUF_Wan_Config, got {type(config).__name__}.")
if submodel_type != SubModelType.Transformer:
raise ValueError(
"Only the Transformer submodel is available from a GGUF Wan checkpoint. "
"Pair with a standalone Wan VAE and Wan T5 encoder for the other components."
)
return self._load_from_singlefile(config)
def _load_from_singlefile(self, config: Main_GGUF_Wan_Config) -> AnyModel:
import accelerate
from diffusers import WanTransformer3DModel
model_path = Path(config.path)
target_device = TorchDevice.choose_torch_device()
compute_dtype = TorchDevice.choose_bfloat16_safe_dtype(target_device)
sd = gguf_sd_loader(model_path, compute_dtype=compute_dtype)
# Strip ComfyUI-style prefixes if present.
for prefix in ("model.diffusion_model.", "diffusion_model."):
if any(isinstance(k, str) and k.startswith(prefix) for k in sd.keys()):
sd = {
(k[len(prefix) :] if isinstance(k, str) and k.startswith(prefix) else k): v for k, v in sd.items()
}
break
# QuantStack and other community releases ship the native upstream Wan key
# layout (text_embedding.0, self_attn/cross_attn, ffn.0/2, head.head, ...);
# diffusers' WanTransformer3DModel expects condition_embedder.*, attn1/attn2,
# ffn.net.*, proj_out. Convert in place if needed.
if _is_native_wan_layout(sd):
sd = _convert_wan_native_to_diffusers(sd)
# Pre-cast non-quantized tensors (F16/F32/BF16 biases, scale_shift_table,
# patch_embedding.weight, etc.) to compute_dtype. This avoids dtype
# mismatches in conv3d at the input (patch_embedding is the only Conv3d
# in WanTransformer3DModel; conv3d isn't in GGMLTensor's dispatch table
# so the wrapper's underlying storage dtype reaches PyTorch directly).
sd = _unwrap_unquantized_to_compute_dtype(sd)
# Auto-detect architecture from the state dict.
num_layers = 0
for key in sd.keys():
if isinstance(key, str) and key.startswith("blocks."):
parts = key.split(".")
if len(parts) >= 2:
try:
num_layers = max(num_layers, int(parts[1]) + 1)
except ValueError:
pass
# Patch embedding gives us in_channels (16=A14B, 48=TI2V-5B) and inner dim.
patch_w = sd.get("patch_embedding.weight")
if patch_w is None:
raise RuntimeError("GGUF state dict missing patch_embedding.weight after prefix strip")
patch_shape = patch_w.tensor_shape if isinstance(patch_w, GGMLTensor) else patch_w.shape
inner_dim = int(patch_shape[0])
in_channels = int(patch_shape[1])
# Wan uses head_dim=128 throughout the family; num_heads = inner_dim / 128.
attention_head_dim = 128
num_attention_heads = inner_dim // attention_head_dim
ffn_w = sd.get("blocks.0.ffn.net.0.proj.weight")
if ffn_w is None:
raise RuntimeError("GGUF state dict missing blocks.0.ffn.net.0.proj.weight after prefix strip")
ffn_shape = ffn_w.tensor_shape if isinstance(ffn_w, GGMLTensor) else ffn_w.shape
ffn_dim = int(ffn_shape[0])
text_w = sd.get("condition_embedder.text_embedder.linear_1.weight")
text_dim = 4096
if text_w is not None:
text_shape = text_w.tensor_shape if isinstance(text_w, GGMLTensor) else text_w.shape
text_dim = int(text_shape[1])
# out_channels is read from proj_out.weight directly rather than assumed
# equal to in_channels: I2V-A14B has in_channels=36 (16 noise + 16
# ref-image latents + 4 mask, concatenated by the denoise loop) but
# out_channels=16 (only the noise prediction comes back). proj_out is
# ``nn.Linear(inner_dim, out_channels * prod(patch_size))`` and
# patch_size is (1, 2, 2) → prod = 4 for the Wan 2.2 family.
proj_out_w = sd.get("proj_out.weight")
if proj_out_w is None:
raise RuntimeError("GGUF state dict missing proj_out.weight after prefix strip")
proj_out_shape = proj_out_w.tensor_shape if isinstance(proj_out_w, GGMLTensor) else proj_out_w.shape
out_channels = int(proj_out_shape[0]) // 4
# Layer count fallback (only triggers if the auto-count loop above
# found zero blocks, which shouldn't happen for a valid GGUF). T2V/I2V
# A14B have 40 layers; TI2V-5B has 30.
layer_count_fallback = 30 if config.variant == WanVariantType.TI2V_5B else 40
model_config: dict = {
"patch_size": (1, 2, 2),
"in_channels": in_channels,
"out_channels": out_channels,
"num_layers": num_layers if num_layers > 0 else layer_count_fallback,
"attention_head_dim": attention_head_dim,
"num_attention_heads": num_attention_heads,
"ffn_dim": ffn_dim,
"text_dim": text_dim,
}
with accelerate.init_empty_weights():
model = WanTransformer3DModel(**model_config)
incompatible_keys = model.load_state_dict(sd, strict=False, assign=True)
if incompatible_keys.missing_keys:
raise RuntimeError(f"GGUF state dict is missing model parameters: {incompatible_keys.missing_keys}")
return model
@ModelLoaderRegistry.register(base=BaseModelType.Any, type=ModelType.WanT5Encoder, format=ModelFormat.WanT5Encoder)
class WanT5EncoderLoader(ModelLoader):
"""Loader for the standalone Wan UMT5-XXL encoder.
Accepts two on-disk layouts:
1. Parent dir with ``text_encoder/`` (and typically ``tokenizer/``) subdirs —
what ``Wan-AI/Wan2.2-T2V-A14B::text_encoder+tokenizer`` produces.
2. A flat ``text_encoder/`` folder with ``config.json`` declaring
``model_type: umt5`` directly at the root. In this case the tokenizer
is loaded from the same folder via ``AutoTokenizer.from_pretrained``.
"""
def _load_model(
self,
config: AnyModelConfig,
submodel_type: Optional[SubModelType] = None,
) -> AnyModel:
if submodel_type is None:
raise ValueError("A submodel type (Tokenizer or TextEncoder) must be provided.")
root = Path(config.path)
nested_text_encoder = root / "text_encoder"
nested_tokenizer = root / "tokenizer"
if submodel_type == SubModelType.TextEncoder:
from transformers import UMT5EncoderModel
target = nested_text_encoder if nested_text_encoder.exists() else root
return UMT5EncoderModel.from_pretrained(
str(target),
torch_dtype=torch.bfloat16,
local_files_only=True,
)
if submodel_type == SubModelType.Tokenizer:
from transformers import AutoTokenizer
# Prefer a sibling tokenizer/ directory; fall back to the encoder dir
# itself, which is normal for "flat" downloads.
target = (
nested_tokenizer
if nested_tokenizer.exists()
else (nested_text_encoder if nested_text_encoder.exists() else root)
)
return AutoTokenizer.from_pretrained(str(target), local_files_only=True)
raise ValueError(
f"Unsupported submodel type for WanT5Encoder: {submodel_type.value if submodel_type else 'None'}"
)
@@ -3,13 +3,14 @@ from typing import Any, Optional, TypeAlias
import safetensors.torch
import torch
from gguf import GGUFValueType
from picklescan.scanner import scan_file_path
from safetensors import safe_open
from invokeai.app.services.config.config_default import get_config
from invokeai.backend.model_hash.model_hash import HASHING_ALGORITHMS, ModelHash
from invokeai.backend.model_manager.taxonomy import ModelRepoVariant
from invokeai.backend.quantization.gguf.loaders import gguf_sd_loader
from invokeai.backend.quantization.gguf.loaders import WrappedGGUFReader, gguf_sd_loader
from invokeai.backend.util.logging import InvokeAILogger
from invokeai.backend.util.silence_warnings import SilenceWarnings
@@ -52,9 +53,19 @@ class ModelOnDisk:
if path in self._metadata_cache:
return self._metadata_cache[path]
try:
with safe_open(self.path, framework="pt", device="cpu") as f:
metadata = f.metadata()
assert isinstance(metadata, dict)
if path.suffix == ".gguf":
with WrappedGGUFReader(path) as reader:
metadata = {
name: value
for name, field in reader.fields.items()
if field.types
and field.types[0] == GGUFValueType.STRING
and isinstance(value := field.contents(), str)
}
else:
with safe_open(path, framework="pt", device="cpu") as f:
metadata = f.metadata()
assert isinstance(metadata, dict)
except Exception:
metadata = {}

Some files were not shown because too many files have changed in this diff Show More