mirror of
https://github.com/invoke-ai/InvokeAI.git
synced 2026-10-12 02:16:43 +00:00
feat: Video generation (#9163)
* feat(model): add Wan 2.2 image generation support (Phases 0-2)
Foundation + TI2V-5B MVP + A14B dual-expert MoE for Wan 2.2 image
generation. Wan was trained on video but is competitive with leading
open-source image models when run at num_frames=1; this commit wires
that path into InvokeAI.
Phase 0 — Foundation:
- BaseModelType.Wan + WanVariantType {T2V_A14B, TI2V_5B}
- SubModelType.Transformer2 for the dual-expert MoE
- MainModelDefaultSettings per variant
- step_callback Wan branch (16-channel preview; 48-channel TI2V-5B
falls back to slicing first 16 channels until proper factors land)
- Frontend enums + node colour
Phase 1 — TI2V-5B Diffusers MVP:
- Main_Diffusers_Wan_Config probe (variant from transformer_2/ +
vae/config.json::z_dim, with filename heuristic fallback)
- WanDiffusersModel loader (subclasses GenericDiffusersLoader)
- WanT5EncoderField, WanTransformerField (with dual-expert slots),
WanConditioningField, WanConditioningInfo
- New invocations: wan_model_loader, wan_text_encoder, wan_denoise,
wan_image_to_latents, wan_latents_to_image
- FlowMatchEulerDiscreteScheduler integration with on-disk config load
- RectifiedFlowInpaintExtension reused for inpaint
- 5D <-> 4D shape juggling: latents stay 4D in InvokeAI's pipeline,
re-add T=1 only inside the transformer call / VAE encode-decode
Phase 2 — A14B dual-expert MoE:
- Probe reads boundary_ratio from model_index.json
- Loader emits both transformer (high-noise) and transformer_low_noise
(low-noise expert at transformer_2/) for A14B
- _ExpertSwapper in wan_denoise drives GPU residency between experts:
high-noise for t >= boundary_ratio * num_train_timesteps, low-noise
below. Only one expert locked at a time so the cache can evict the
other - relies on existing CachedModelWithPartialLoad to handle
oversized models on lower-VRAM GPUs.
- guidance_scale_low_noise field for separate low-noise CFG override
Tests:
- 24 passing tests covering probe variant detection, default settings,
noise sampling, end-to-end denoise on a synthetic transformer (CPU),
dual-expert boundary swap, CFG branch
- 1 heavy-test placeholder gated by INVOKEAI_HEAVY_TESTS=1 for the
real-weights smoke test
Phase 3+ deferred: standalone VAE/encoder configs, GGUF, LoRA,
ControlNet, ref image, inpaint UI, frontend wiring, starter models.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(model): Wan 2.2 Phase 3 + tokenizer-load fix
Phase 3 adds standalone VAE and UMT5-XXL encoder configs so users can run
GGUF-quantized Wan transformers (Phase 4) without installing the full
~30 GB Diffusers pipeline.
VAE configs:
- VAE_Checkpoint_Wan_Config + VAE_Diffusers_Wan_Config (16-channel A14B
vs 48-channel TI2V-5B, distinguished by decoder.conv_in z_dim).
- 16-channel files share the AutoencoderKLWan architecture with Qwen
Image; disambiguated via filename heuristic ("wan" in name -> Wan,
otherwise -> Qwen Image). Mirror exclusion in QwenImage's probe.
- VAELoader gets a Wan branch that builds AutoencoderKLWan(z_dim=...)
via init_empty_weights, mirroring the QwenImage single-file pattern.
- Existing standard VAE probe excludes both QwenImage- and Wan-style
state dicts.
UMT5-XXL encoder:
- New ModelType.WanT5Encoder + ModelFormat.WanT5Encoder.
- WanT5Encoder_WanT5Encoder_Config probes the diffusers folder layout
(text_encoder/config.json with model_type=umt5, or flat layout with
config.json at root). Refuses full Wan pipelines.
- WanT5EncoderLoader handles both layouts and loads UMT5EncoderModel +
AutoTokenizer.
Component-source plumbing:
- WanModelLoaderInvocation now exposes wan_t5_encoder_model and
component_source pickers (mirrors QwenImage pattern). Resolution
order: standalone > main (if Diffusers) > component_source. Required
when the main model is a single-file format in Phase 4.
Bug fix in wan_text_encoder:
- Tokenizer was loading via AutoTokenizer.from_pretrained(<root>)
directly, which fails for nested layouts where files live in
<root>/tokenizer/. Now routed through the model cache so the
registered loaders handle layout differences correctly.
Frontend:
- New type guards (isWanVAEModelConfig, isWanT5EncoderModelConfig,
isWanMainModelConfig, isWanDiffusersMainModelConfig) and hooks/
selectors (useWanVAEModels, useWanT5EncoderModels,
useWanDiffusersModels). New zSubModelType / zModelType / zModelFormat
enum entries for transformer_2 and wan_t5_encoder.
Tests:
- 16 new tests covering z_dim detection, VAE checkpoint/diffusers
probes, the bidirectional Qwen-vs-Wan filename deferral, and the
UMT5 encoder probe (nested + flat + T5 + full-pipeline rejection).
- Total Wan test count: 41 passing, 1 heavy-test placeholder skipped.
- Full config test suite (63 tests) still passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): unbreak frontend lint after Wan additions
Five issues turned up running `make frontend-lint`:
1. wan_denoise.py used `from __future__ import annotations`, which made
the `invoke()` return annotation a string ('LatentsOutput'). The
InvocationRegistry's `get_output_annotation()` returns the raw
annotation, so OpenAPI generation crashed with
`'str' object has no attribute '__name__'`. Removed the future-import
and added `Any` to the typing imports.
2. ModelRecordChanges.variant didn't list WanVariantType, so the
generated schema's install/update endpoints rejected `t2v_a14b` and
`ti2v_5b`. Added it.
3. Regenerated frontend/web/src/services/api/schema.ts from the live
backend so it now includes BaseModelType.wan, ModelType.wan_t5_encoder,
SubModelType.transformer_2, ModelFormat.wan_t5_encoder, the Wan
variants, all Wan invocation types and their conditioning/transformer
field types.
4. modelManagerV2/models.ts: added `wan_t5_encoder` to the category map,
`wan` to the base color/long-name/short-name maps, the two Wan
variants to the variant-name map, and `wan_t5_encoder` to the
format-name map.
5. ModelManagerPanel/ModelFormatBadge.tsx: added `wan_t5_encoder` to
FORMAT_NAME_MAP and FORMAT_COLOR_MAP.
`make frontend-lint` now passes cleanly (tsc, dpdm, eslint, prettier).
All 41 Wan Python tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
chore(wan): drop unused FE exports flagged by knip
These were forward-compatibility wiring for Phase 9 (the FE graph
builder) that has no consumers yet; knip rightly flagged them. Removed
or de-exported. They'll come back when the graph builder lands and
needs them.
- common.ts: zWanVariantType drops `export` (still used internally by
zAnyModelVariant).
- types.ts: drop isWanMainModelConfig, isWanDiffusersMainModelConfig,
isWanVAEModelConfig (no callers). The remaining
isWanT5EncoderModelConfig is used by models.ts. WanT5EncoderModelConfig
type drops `export` (still used as the type guard's narrowing target).
- modelsByType.ts: drop the six unused useWan*/selectWan* hooks +
selectors and their type-guard imports.
`make frontend-lint` (tsc + dpdm + eslint + prettier + knip) now green.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
docs(wan): use *-Diffusers HF repo names in plan
The Wan-AI org publishes two flavours of each release:
* Wan-AI/Wan2.2-{TI2V-5B,T2V-A14B,I2V-A14B} ← upstream native
* Wan-AI/Wan2.2-{TI2V-5B,T2V-A14B,I2V-A14B}-Diffusers ← convertible
The native release has _class_name=WanModel in config.json and ships
weights flat at the repo root with no transformer/, vae/, text_encoder/
subdirs. It is not loadable by Diffusers' WanPipeline.from_pretrained.
Update plan doc to reference the -Diffusers repos throughout (probe
notes, starter-model entries) so the plumbing path matches what the
Diffusers loader actually expects.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): accept 0 as 'unset' sentinel for guidance_scale_low_noise
The frontend renders Optional[float] inputs with default 0 in the
numeric input rather than passing null/unset. Combined with ge=1.0,
this caused every wan_denoise invocation to fail Pydantic validation
with "Input should be greater than or equal to 1" until the user
manually entered a value (or knew to leave the field disconnected).
The validation error was rejected before invocation logging, so it
never showed up in the server log either - making the failure hard to
diagnose.
Relaxing the constraint to ge=0.0 and treating values below 1.0 as the
"fall back to primary Guidance Scale" sentinel. The user's natural FE
default (0) now works as expected.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): correct preview dimensions and colors for TI2V-5B
Two bugs in the Wan branch of the diffusion step callback:
1. Wrong dimensions. The reported preview size hardcoded `* 8` for the
spatial downscale ratio, but TI2V-5B's Wan2.2-VAE uses 16x. A
1024x1024 target was being announced to the FE as 512x512.
2. Wrong colors. The previous fallback for 48-channel TI2V-5B latents
sliced the first 16 channels and applied the standard 16-channel
Wan-VAE projection. Those channel layouts are unrelated, so the
projection produced meaningless colors.
Adding the proper Wan2.2-VAE 48-channel RGB projection matrix (and
bias) from ComfyUI's Wan22 latent format, and selecting the right
matrix + spatial-scale by latent channel count: 16 → A14B (Wan VAE,
8x), 48 → TI2V-5B (Wan2.2-VAE, 16x).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): honor model's _class_name when building scheduler
TI2V-5B's scheduler_config.json declares _class_name=UniPCMultistepScheduler
with flow_shift=5.0. The previous code hardcoded
FlowMatchEulerDiscreteScheduler.from_pretrained(...), which silently
constructed a default-config FlowMatch instead of the UniPC the model
expects. The mismatched noise schedule manifests as soft / under-denoised
faces and global graininess in the final images.
Now: read scheduler_config.json, look up the named class on the diffusers
module, and instantiate that class via from_pretrained. UniPC and
FlowMatch share the same step()/set_timesteps()/sigmas/num_train_timesteps
interfaces, so the denoise loop works transparently for either.
A14B continues to use FlowMatchEulerDiscreteScheduler when its scheduler
config says so (its reference is FlowMatchEuler with shift=8.0). Falls
back to FlowMatchEulerDiscreteScheduler defaults when no on-disk config
is available.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): match diffusers WanPipeline tokenizer length and latent dtype
Two divergences from the Diffusers reference that were hurting image
quality (soft / grainy / distorted faces at default settings):
1. Tokenizer max_sequence_length was 226 in wan_text_encoder, but the
model was trained with 512-token sequences. The upstream native
config.json has text_len: 512, and Diffusers' WanPipeline.__call__
default is 512 (overriding _get_t5_prompt_embeds's stale 226 default).
Wan's cross-attention sees padded zeros past the prompt's actual
length but expects to be looking at a 512-position context window.
2. Latents were stored in bf16 throughout the denoise loop. Diffusers'
WanPipeline.prepare_latents explicitly uses dtype=torch.float32 and
only casts to the transformer's dtype right at the forward call:
latent_model_input = latents.to(transformer_dtype)
Storing in bf16 between steps accumulates ~40 steps of bf16
quantization on the scheduler's small per-step deltas. Now
latent_dtype = torch.float32 throughout, with a per-step cast for
the transformer forward pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
chore(wan): add diffusers reference comparison script
scripts/wan_diffusers_reference.py runs a Diffusers-format Wan 2.2
checkpoint directly via WanPipeline.from_pretrained, with the same
arguments InvokeAI's wan_denoise uses. Use to A/B against InvokeAI
output when image quality is questionable.
Defaults to enable_model_cpu_offload so the script fits on 16 GB cards
where the full pipeline (transformer + UMT5-XXL + VAE) would otherwise
OOM. --offload {model,sequential,none} controls the strategy.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(model): Wan 2.2 Phase 4 - GGUF transformer support
Adds single-file GGUF support for Wan 2.2 transformers, the path that
makes A14B usable on consumer GPUs (~7 GB/expert at Q4_K_M instead of
~28 GB at bf16).
Probe (configs/main.py):
- New helpers: _has_wan_keys (Wan vs Qwen/FLUX/Z-Image fingerprint via
condition_embedder.text_embedder.linear_1 + patch_embedding);
_detect_wan_gguf_variant (16ch -> A14B, 48ch -> TI2V-5B from
patch_embedding.weight.shape[1]); _detect_wan_gguf_expert (filename
heuristic for high_noise / low_noise / none).
- Main_GGUF_Wan_Config(base=Wan, format=GGUFQuantized, variant, expert).
Tolerates the ComfyUI 'model.diffusion_model.' / 'diffusion_model.'
prefixes via _has_wan_keys' multi-prefix scan.
- Registered in factory.py.
Loader (model_loaders/wan.py):
- WanGGUFCheckpointModel mirrors the QwenImage GGUF pattern:
gguf_sd_loader -> strip ComfyUI prefix -> auto-detect arch from state
dict shapes (num_layers, inner_dim, ffn_dim, text_dim, in_channels,
num_heads = inner_dim/128) -> init_empty_weights +
load_state_dict(strict=False, assign=True).
Loader invocation (wan_model_loader.py):
- New 'Transformer (Low Noise)' picker: optional second GGUF for the
A14B dual-expert MoE. Auto-swaps if the user wired the experts in
the wrong order. Warns when an A14B GGUF is loaded without a paired
low-noise expert (single-expert run, degraded quality).
- GGUF mains require either a standalone VAE+encoder or a Diffusers
Component Source (which can also supply boundary_ratio).
- Diffusers main path unchanged (still pulls both experts from
transformer/ + transformer_2/).
Tests (tests/.../test_wan_gguf_config.py):
- 14 tests across key fingerprint, variant detection, expert filename
heuristic, and the full probe (A14B high/low, TI2V-5B, GGUF rejection,
unrecognised state-dict rejection, explicit override).
Total Wan tests: 55 passing (no regressions). FE lint clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): support QuantStack-style GGUFs and standalone Diffusers VAE
The city96 Wan 2.2 GGUF repos have been removed from Hugging Face,
leaving QuantStack as the surviving distributor. QuantStack ships the
native upstream Wan key layout (text_embedding.0/2, self_attn/cross_attn,
ffn.0/2, head.head, head.modulation, ...) rather than the diffusers
naming city96 used; biases are stored as F16 rather than BF16; and the
standalone Wan VAE installs as a flat AutoencoderKLWan folder which the
generic loader rejects. Three fixes:
1. Probe now recognises both diffusers and native key layouts via a new
_is_native_wan_layout helper; _has_wan_keys accepts either text-proj
fingerprint.
2. GGUF loader converts native -> diffusers keys (mirroring diffusers'
convert_wan_transformer_to_diffusers) and unwraps non-quantized
GGMLTensors to plain tensors at compute_dtype. The unwrap is needed
because conv3d isn't in GGMLTensor's dispatch table, so the F16
patch_embedding bias would otherwise hit conv3d against bf16 latents.
3. VAELoader gains a VAE_Diffusers_Wan_Config branch that loads
AutoencoderKLWan directly; the generic path can't handle a flat
single-class folder when a submodel_type is provided.
Adds 12 tests covering the native layout (probe + converter + unwrap).
Verified end-to-end against Wan2.2-T2V-A14B-Q4_K_M from QuantStack:
1095 tensors round-trip key-for-key against WanTransformer3DModel.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(model): Wan 2.2 Phase 5 - LoRA support
Probe + config (LoRA_LyCORIS_Wan_Config):
- Detects Wan LoRAs in three layouts: diffusers PEFT, native upstream PEFT
(ComfyUI), and Kohya (both naming variants).
- Anti-pattern guards prevent collisions with Anima (Cosmos DiT q_proj
convention), QwenImage (transformer_blocks), Flux (double/single blocks),
and Z-Image (diffusion_model.layers).
- Optional ``expert: "high" | "low" | None`` field; auto-detected from
filename (high_noise / low_noise / hyphenated / concatenated variants).
Key conversion (wan_lora_conversion_utils):
- Native upstream keys (self_attn/cross_attn, ffn.0/2) -> diffusers
(attn1/attn2, ffn.net.0.proj / ffn.net.2).
- Strips ``transformer.``, ``diffusion_model.``, ``base_model.model.transformer.``
prefixes from PEFT-style keys.
- Kohya layer names mapped through an explicit longest-match table.
- Output paths use diffusers naming so the LayerPatcher can resolve them
against WanTransformer3DModel parameter paths.
Loader integration:
- Adds BaseModelType.Wan branch to LoRALoader._load_model.
Invocation nodes (wan_lora_loader.py):
- WanLoRALoaderInvocation: single LoRA with auto/both/high/low target field.
- WanLoRACollectionLoader: list of LoRAs, auto-routed by each LoRA's
recorded expert tag.
- Output WanLoRALoaderOutput carries the WanTransformerField with updated
``loras`` / ``loras_low_noise`` lists.
Denoise integration:
- _ExpertSwapper now manages both the model_on_device context and the
LayerPatcher.apply_smart_model_patches context per expert. LoRA patches
are entered after device load and exited before device release, with
fresh iterators per swap.
- GGUF (quantized) experts request sidecar patching so GGMLTensor weights
aren't touched directly.
- Low-noise expert falls back to the primary loras list when
``loras_low_noise`` is empty (matches WanTransformerField semantics).
Tests: 81 new tests covering probe accept/reject across formats, anti-pattern
guards on competing architectures, converter round-trips for all three
layouts, invocation target resolution + routing + duplicate guards, and the
_ExpertSwapper lifecycle (lora context opens/closes in the right order
around the device swap, quantized flag forwards, no-LoRA path skips the
patch context, re-entering the same label is a no-op).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): probe Wan LoRA before Anima in the config union
Native-PEFT Wan LoRAs (lightx2v's Lightning, most ComfyUI-trained Wan
LoRAs) carry keys like ``diffusion_model.blocks.X.cross_attn.k.lora_A.weight``.
Anima's probe matches on the bare ``cross_attn``/``self_attn`` substring —
it does not require the Anima-specific ``_proj`` suffix nor any of the
``mlp``/``adaln_modulation`` Cosmos DiT markers — so these Wan LoRAs were
classified as ``BaseModelType.Anima`` because Anima happened to run first.
Reorder the LyCORIS section of ``AnyModelConfig`` so Wan probes first.
Wan's probe is strictly more restrictive (it rejects Anima's ``_proj``
attention suffix via the anti-pattern guard added in the previous commit),
so Anima LoRAs are still correctly classified after this reorder.
Existing users with mis-tagged installs need to delete the affected LoRA
records and reinstall.
Adds two regression tests: a union-ordering assertion, and a sanity check
that demonstrates Anima's probe *would* match Wan native keys if asked
directly — pinning the constraint that motivates the ordering.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
chore(i18n): add Wan2.2 T5 Encoder model-manager label
The frontend source already references ``modelManager.wanT5Encoder``;
the locale key was added with a casing typo (``want5Encoder``). Fix
the key so the Wan T5 Encoder model type renders its display name
correctly in the model manager UI.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(model): Wan 2.2 Phase 7 - reference-image (I2V) conditioning
Re-implementation after the first attempt — which used CLIP-vision
conditioning — was reverted. Wan 2.2 I2V-A14B does NOT use a CLIP-vision
encoder (the Diffusers repo ships ``image_encoder: [null, null]`` in
``model_index.json``); instead it conditions on a reference image by
VAE-encoding it and concatenating the resulting latents (plus a
first-frame mask) to the noise latents along the channel dim. The I2V
transformer therefore has ``in_channels=36`` (16 noise + 16 ref-image
latents + 4 mask) vs ``in_channels=16`` for T2V.
Taxonomy:
- Re-adds ``WanVariantType.I2V_A14B``.
Probes:
- Diffusers: ``_detect_wan_variant`` reads ``transformer/config.json::in_channels``;
36 → I2V_A14B, 16 → T2V_A14B (both share the dual-expert layout).
- GGUF: ``_detect_wan_gguf_variant`` recognises ``in_channels=36`` from the
patch_embedding tensor shape and emits I2V_A14B.
Backend extension (``backend/wan/extensions/wan_ref_image_extension.py``):
- ``preprocess_reference_image`` resizes + normalises to a 5D pixel tensor.
- ``encode_reference_image_to_condition`` VAE-encodes the image and stacks
a 4-channel first-frame mask on top, producing the
``[1, 20, 1, H/8, W/8]`` condition tensor the denoise loop consumes.
- Mirrors diffusers ``WanImageToVideoPipeline.prepare_latents`` with
``num_frames=1`` and ``expand_timesteps=False``.
Invocation node (``wan_ref_image_encoder.py``):
- "Reference Image - Wan 2.2": image + VAE + width/height pickers.
- Output ``WanRefImageConditioningField`` carries the condition tensor
name plus the dimensions used (so the denoise step can validate dim
parity).
Denoise integration:
- ``WanDenoiseInvocation`` gains an optional ``ref_image`` field.
- Variant gate: rejects ref_image on T2V_A14B and TI2V-5B with a clear
error before doing any work.
- Dimension gate: rejects ref-image width/height mismatch vs denoise.
- At every transformer call, concatenates the 20-channel condition
tensor to the 16-channel noise latents along the channel dim before
passing to the transformer (giving the 36-channel input I2V expects).
Tests: 14 new across the probe, the extension, and the denoise loop.
The synthetic ``_ZeroTransformer`` test stand-in now mirrors the real
I2V transformer's ``in_channels=36, out_channels=16`` asymmetry by
slicing its zero output back to 16 channels when the input is 36-wide.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): derive GGUF out_channels from proj_out shape (I2V support)
The GGUF loader was setting ``out_channels = in_channels`` which is wrong for
Wan 2.2 I2V-A14B: that variant has ``in_channels=36`` (16 noise + 16 ref-image
latents + 4 first-frame mask, concatenated by the denoise loop) but
``out_channels=16`` since the transformer only predicts the noise component
back. Loading an I2V GGUF would build a transformer with the wrong proj_out
shape and crash:
RuntimeError: Error(s) in loading state_dict for WanTransformer3DModel:
size mismatch for proj_out.weight: copying a param with shape
torch.Size([64, 5120]) from checkpoint, the shape in current model is
torch.Size([144, 5120]).
(144 = 36 * 4, 64 = 16 * 4 — patch_size=(1, 2, 2) → prod=4)
Read out_channels directly from the ``proj_out.weight`` shape in the state
dict. This is correct for all three Wan 2.2 variants without needing to know
the variant in advance.
Also tighten the num_layers fallback: T2V_A14B and I2V_A14B share 40 layers;
only TI2V-5B has 30. The fallback is rarely hit in practice (the per-block
count comes from the state dict scan), but the previous code would have
defaulted I2V_A14B to 30 layers.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(model): make Anima LoRA probe mutually exclusive with Wan
InvokeAI's ``Config_Base.CONFIG_CLASSES`` is a Python ``set``, so iteration
order during model probing is non-deterministic across process restarts.
First-match-wins ordering in ``AnyModelConfig`` is documentation only — it
has no effect on which config is iterated first.
Anima's previous probe accepted any state dict containing the substring
``cross_attn`` or ``self_attn``, which collides with Wan's native LoRA key
layout (``diffusion_model.blocks.X.cross_attn.q.lora_down.weight``). Both
probes accepted Wan native LoRAs (including lightx2v's Lightning T2V and I2V
distillations), and the ``matches.sort_key`` tiebreaker only disambiguates
by ModelType, not within LoRA configs. So which config "won" depended on
dict hash order — sometimes Wan, sometimes Anima.
The previous mitigation reordered the AnyModelConfig union to put Wan
before Anima. That worked by luck and was inherently fragile.
Tighten Anima's probe to require Cosmos-DiT-exclusive subcomponents:
``mlp``, ``adaln_modulation``, or ``_proj``-suffixed attention names
(``q_proj``/``k_proj``/``v_proj``/``output_proj``) — none of which appear
in any Wan LoRA. Wan native uses bare ``.q``/``.k``/``.v``/``.o`` on
``self_attn``/``cross_attn``, and ``ffn.N``/``ffn.net.N`` instead of ``mlp``.
The new strict detectors live alongside the original loose ones so the
Anima conversion utility (which runs after probing) still works.
Regression tests in ``test_wan_lora_probe_independence.py`` cover:
- I2V Lightning V1 (the bug-triggering LoRA), T2V Lightning V2, Wan Kohya
and Wan diffusers PEFT layouts — Wan probe accepts, Anima probe rejects.
- Anima PEFT and Kohya layouts — Anima accepts, Wan rejects.
- A meta-test that runs every LoRA config in CONFIG_CLASSES against the
Lightning state dicts and asserts exactly one accepts — this catches
ANY future probe collision, not just Wan vs Anima.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): defer expert model loading in _ExpertSwapper to avoid cache thrash
The swapper used to take pre-loaded ``LoadedModel`` handles at construction:
high_info = context.models.load(self.transformer.transformer)
low_info = context.models.load(self.transformer.transformer_low_noise)
swapper = _ExpertSwapper(high_info=high_info, low_info=low_info, ...)
With dual ~9 GB A14B GGUF experts plus the ~10 GB UMT5-XXL encoder competing
for the same RAM cache, the LRU policy frequently dropped one expert by the
time the denoise loop swapped into it. The model manager then emitted
[MODEL CACHE] Locking model cache entry ... but it has already been
dropped from the RAM cache. This is a sign that the model loading
order is non-optimal in the invocation code (See ... #7513).
and reloaded the weights from disk (~1.2s extra per swap).
Refactor the swapper to take the ``ModelIdentifierField`` plus the
``InvocationContext`` and call ``context.models.load(model_id)`` lazily
inside ``get()``. Each swap obtains a fresh handle, the LRU window is
small, and the warning goes away.
Config metadata (used to compute ``is_quantized``) is read upfront via
``context.models.get_config()`` — that's metadata, not weights, so it
doesn't put pressure on the cache.
Tests: existing swapper lifecycle tests refactored to use a fake context
whose ``models.load`` is logged. A new ``test_lazy_load_per_swap_not_upfront``
pins the regression — it asserts ``models.load`` is NOT called at swapper
construction, only at first get() per expert.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* test(wan): add Phase 8 inpaint regression tests
The denoise_mask wiring + RectifiedFlowInpaintExtension integration in
wan_denoise.py was put in place during Phase 2/3 alongside the rest of
the denoise loop. Phase 8 of the plan was about ensuring this path
worked and is locked in by tests.
Three new tests under TestWanDenoiseInpaint:
1. test_preserved_region_matches_init_exactly: builds a half/half mask
(left = preserve, right = regenerate in user-side convention), runs
full denoise with the synthetic zero-output transformer, and asserts
the preserved half of the final latents equals the init exactly while
the regenerated half does not. Pins the mask-inversion + per-step
merge behavior.
2. test_inpaint_requires_init_latents: a mask without init latents must
raise a clear ValueError — the merge has nothing to weld back to.
3. test_no_mask_path_is_unchanged: regression that adding the inpaint
extension didn't perturb the non-inpaint codepath (with init latents
+ denoising_start=0.5 but no mask, the loop just runs img2img).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
feat(frontend): add I2V_A14B to Wan variant zod enum + manager label
Phase 7 added the I2V_A14B backend variant. The frontend's zod enum
(features/nodes/types/common.ts:zWanVariantType) and the model manager's
variant-label map (features/modelManagerV2/models.ts) were still on the
two-variant list, so:
- ModelIdentifierField inputs with ui_model_variant filters on Wan
couldn't list I2V models.
- The model manager UI showed a raw 'i2v_a14b' string instead of the
human label.
Phase 9 (full linear-view wiring — type guards, hooks, params slice,
graph builder, tab UI) is in progress on a follow-up commit; this lands
the two small enum fixes first so the I2V probe / install paths work
correctly end-to-end with the existing FE.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): Phase 9 piece #1 - linear-view T2V txt2img graph builder
Adds the minimum frontend wiring needed to generate Wan 2.2 images from
the linear view:
- buildWanGraph.ts (new): text-to-image graph (model_loader →
text_encoder × 2 → denoise → l2i). Diffusers main model only —
transformer, VAE and UMT5 encoder all resolve from the same repo, so
no Wan-specific params slice fields are required yet. CFG-skip
branch when guidance_scale ≤ 1.0.
- useEnqueueGenerate / useEnqueueCanvas dispatchers: route
base === 'wan' to buildWanGraph.
- graph/types.ts: add wan_l2i / wan_i2l / wan_denoise / wan_model_loader
to the relevant node-type unions.
- addTextToImage / addImageToImage: include wan_denoise / wan_l2i so
width/height are wired correctly and the txt2img helper accepts the
Wan l2i node.
- isMainModelWithoutUnet: include wan_model_loader (Wan has no UNet,
same as the other modern bases).
- metadata.py: add wan_txt2img / wan_img2img / wan_inpaint to the
generation_mode enum (img2img / inpaint pieces land next).
- schema.ts: regenerated to pick up the metadata enum + new
Wan invocations.
Pieces left in Phase 9: params slice (standalone VAE / T5 / GGUF
low-noise / LoRA / ref-image fields + selectors), img2img + I2V + inpaint
branches in the graph builder, and Wan-specific UI components.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
feat(wan): Phase 9 piece #2 - GGUF support and CFG-Low control in linear view
Adds the three Wan-specific params + UI controls that gate GGUF workflows
plus a separate low-noise CFG slider for A14B users.
Params slice:
- wanTransformerLowNoise (the second-expert GGUF for A14B)
- wanComponentSource (Diffusers Wan model providing VAE + UMT5-XXL
when the main is a GGUF)
- wanGuidanceScaleLowNoise (optional separate CFG for the low-noise
expert; null = fall back to the primary CFG)
Plus a `selectIsWan` selector for accordion gating.
UI components:
- ParamWanModelSelects.tsx (Advanced accordion): two model pickers —
Transformer (Low Noise) filtered to Wan GGUF mains, and VAE/Encoder
Source filtered to Wan Diffusers mains. Mirrors the
ParamQwenImageComponentSourceSelect structure.
- ParamWanGuidanceScaleLowNoise.tsx (Generation accordion): slider +
number input with an "auto" indicator when cleared. Default 3.5
matches the diffusers reference 4.0 / 3.0 split.
Wiring:
- Generation accordion: ParamWanGuidanceScaleLowNoise shown when base
is wan, scheduler excluded for wan (same pattern as Anima/Qwen).
- Advanced accordion: ParamWanModelSelects shown when base is wan, and
Wan excluded from the SD-family VAE/CFG-rescale blocks.
- buildWanGraph.ts: forwards the three new params to the model loader
and denoise nodes (transformer_low_noise_model, component_source,
guidance_scale_low_noise) and adds them to the graph metadata.
Hooks/types:
- useWanDiffusersModels + useWanGGUFModels in modelsByType.ts.
- isWanDiffusersMainModelConfig + isWanGGUFMainModelConfig type guards.
- Three new locale strings (wanComponentSource, wanTransformerLowNoise,
wanGuidanceScaleLowNoise[Auto]).
GGUF workflow now works end-to-end in the linear view: pick a Wan GGUF
main, set Transformer (Low Noise) to the paired second-expert GGUF, set
VAE/Encoder Source to any Diffusers Wan repo (TI2V-5B is convenient at
~12 GB) — generate produces an image.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): UX polish on the Wan linear-view controls
Bundles four small fixes applied during a usability review of the Wan
linear-view section (piece #2):
1. **Filter Main vs Transformer (Low Noise) dropdowns by expert tag.**
The Wan GGUF probe records each file's ``expert`` field
(``"high"`` / ``"low"`` / ``"none"``) via filename heuristic.
- ``MainModelPicker``: hides ``expert === 'low'`` Wan GGUFs so users
can't accidentally wire a low-noise expert as the primary main.
- Transformer (Low Noise) picker (``useWanGGUFLowNoiseModels``):
shows ``expert === 'low'`` Wan GGUFs only.
Diffusers Wan mains and TI2V-5B aren't affected — they don't carry
the ``expert`` field on their config schema. The backend's auto-swap
safety net stays in place.
2. **Match the primary CFG slider's range.** The Wan low-noise CFG
slider was constrained to 1–10 while the primary CFG ranges 1–20.
With the diffusers reference 4/3 split, the low-noise slider thumb
sat noticeably further right than the primary — visually misleading.
Both sliders now share the 1–20 range with marks at [1, 10, 20].
3. **Label fits the form column.** "CFG (Low Noise)" → "CFG (Low)" so
the slider fits cleanly next to its label instead of overlapping.
4. **Indicator state for the low-noise CFG slider.** Replaced the inline
"(auto)" / "(same as cfg)" text — which kept overlapping the slider
regardless of how short the label got — with an X-only reset button
that's only visible when the user has set an explicit value. Absence
of the X conveys auto/fallback state without any text overhang.
5. **Friendlier Transformer (Low Noise) placeholder.** "Second-expert
GGUF for A14B (pair with the high-noise main)" → "Add for full
detail" — concise nudge for users who haven't paired the second
expert yet.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
feat(wan): Phase 9 piece #3 - linear-view img2img branch
Adds Wan 2.2 image-to-image to the linear view, mirroring the Qwen Image
pattern. The mode switches on the canvas state — pure-prompt runs go
through addTextToImage as before; canvas runs with an init image go
through addImageToImage which wires a fresh wan_i2l (Image to Latents -
Wan 2.2) node between the init image and the denoise's `latents` input,
honoring the existing denoise_start slider.
buildWanGraph:
- Drops the txt2img-only guard, branches on generationMode.
- img2img: spins up a wan_i2l node and hands it to addImageToImage
alongside the existing denoise / l2i / modelLoader (as vaeSource).
- inpaint / outpaint still fail loudly — pieces #4-#6.
graphBuilderUtils.getDenoisingStartAndEnd:
- Adds 'wan' to the simple-linear case (denoising_start = 1 -
denoisingStrength). Note: Wan's flow-matching schedule is "sticky"
on the init compared to SDXL — users will likely need denoisingStrength
≥ 0.7 to see substantial change, matching the user-found 0.15-0.3
denoising_start sweet spot from earlier img2img testing. We may
revisit this with an exponent rescale (like FLUX uses) if the
response curve feels off.
addImageToImage:
- Adds 'wan_i2l' to the i2l-node-type union so the Wan i2l can be
threaded through the shared helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): add wan_denoise to addImageToImage/addInpaint/addOutpaint type checks
Three sibling graph-helper utilities had the same modern-base list as
addTextToImage did, and the buildWanGraph img2img branch tripped one of
them at canvas-Generate time:
error [generation]: Failed to build graph
{name: 'Error', message: 'Wrong assertion encountered'}
The else-branch in each helper assumes 'denoise_latents' (the SD1.5/SDXL
legacy path) and asserts that — failing for any modern base not listed
above the branch. addTextToImage was already updated in Phase 9 piece #1;
this catches the parallel cases that the img2img/inpaint/outpaint flows
go through.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
feat(wan): Phase 9 piece #4 - linear-view inpaint and outpaint branches
Wires Wan 2.2 inpaint and outpaint through the existing addInpaint /
addOutpaint helpers. The backend's RectifiedFlowInpaintExtension was
plumbed into wan_denoise.py back in Phase 8 (commit ab54617173); this
just connects the FE.
buildWanGraph:
- generationMode === 'inpaint' → spin up a wan_i2l, call addInpaint
with denoise + l2i + modelLoader (used as both vaeSource and
modelLoader since the Wan model loader carries the VAE).
- generationMode === 'outpaint' → parallel branch with addOutpaint.
addInpaint:
- i2l-node-type union now includes 'wan_i2l' (the addImageToImage and
addOutpaint type unions already do — different union shapes).
metadata.py:
- generation_mode literal adds "wan_outpaint" alongside the existing
wan_txt2img / wan_img2img / wan_inpaint entries.
isMainModelWithoutUnet already includes wan_model_loader (Phase 9 piece
create_gradient_mask when Wan is the main.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
feat(wan): Phase 9 piece #5 - linear-view I2V branch (raster as reference image)
Wan 2.2 I2V-A14B models condition on a reference image whose VAE-encoded
latents are concatenated to the noise along the channel dim each step
(in_channels=36 on the I2V transformer). In the linear view this maps
cleanly onto the existing canvas raster layer: pick an I2V model, drag
an image to raster, generate.
buildWanGraph:
- Fetch the modelConfig early so the variant gate (i2v_a14b vs the
rest) can drive the branch shape instead of being a post-hoc check.
- I2V + txt2img: fail loudly ("Switch to the canvas tab and drag an
image to the raster layer"). I2V models won't produce useful output
without a reference, and the backend would crash trying to
concatenate a missing condition tensor.
- I2V + img2img: pull the raster image via the canvas compositor,
wire it through a wan_ref_image_encoder (which VAE-encodes it and
builds the 4-mask + 16-latent condition tensor backend-side), then
feed the result into denoise.ref_image. Denoise runs from fresh
noise (denoising_start=0, no init_latents) — the ref image is
cross-attention/concat conditioning, not a noise-trajectory anchor.
- I2V + inpaint/outpaint: fail clearly. Combining ref-image
conditioning with a denoise mask is conceptually possible but the
backend interaction hasn't been validated end-to-end.
metadata.py:
- Adds "wan_i2v" to the generation_mode literal so the metadata field
on I2V renders correctly.
T2V flows (txt2img / img2img / inpaint / outpaint) are unchanged for
non-I2V Wan variants (T2V-A14B and TI2V-5B).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): enforce multiple-of-16 dimensions to match transformer patch grid
Wan 2.2's transformer has ``patch_size=(1, 2, 2)``: it patch-embeds with
stride 2 then un-patches by 2. Combined with the VAE's 8x spatial scale,
canvas H/W must be a multiple of ``8 * 2 = 16`` — not just 8 — for the
patch round-trip to land exactly. Otherwise the latents and noise
prediction disagree by one in the spatial dim and the scheduler step
fails:
RuntimeError: The size of tensor a (147) must match the size of
tensor b (146) at non-singleton dimension 3
(here latent_w=147 → patch_w=73 → un-patched_w=146 ≠ 147)
This was silent for T2V at 1024x1024 (already a multiple of 16) but
fired for I2V at non-multiple-of-16 canvas sizes.
Fixes:
- ``optimalDimension.getGridSize``: Wan moves from the default 8 case to
the multiple-of-16 case (alongside flux / sd-3 / qwen-image / z-image
which have the same patch arithmetic). The canvas bbox UI now snaps
Wan dimensions to multiples of 16.
- ``wan_denoise.py`` and ``wan_ref_image_encoder.py``: bump width/height
``multiple_of`` from 8 to 16. Defense-in-depth — workflow-editor
users won't be able to send a non-16-aligned dim either.
Existing backend tests (23 passing) still hold — 1024 is divisible by 16
so the test fixtures didn't exercise the off-by-one path.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): show negative prompt box in Wan linear-view
Wan was missing from SUPPORTS_NEGATIVE_PROMPT_BASE_MODELS, so the
linear-view negative-prompt input was hidden even though the Wan denoise
node already wires negative conditioning when CFG > 1
(buildWanGraph.ts:67-75). Adds 'wan' to the list.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
feat(wan): Phase 9 piece #6 - Wan LoRA collection in linear view
Adds Wan LoRA wiring to buildWanGraph, mirroring the Qwen Image pattern.
The shared LoRASelect / LoRAList UI in the linear view already filters
LoRAs by the selected main model's base, so Wan LoRAs surface
automatically when a Wan main is picked — no UI changes needed.
addWanLoRAs (new):
- Filters state.loras.loras to enabled Wan LoRAs.
- For each LoRA: spawns a ``lora_selector`` node and threads it
through a single ``collect`` collector.
- Routes the collector into a ``wan_lora_collection_loader`` which
sits between modelLoader and denoise — modelLoader.transformer →
loader, then loader.transformer → denoise (rerouting the original
modelLoader → denoise edge).
- Emits per-LoRA metadata so PNG metadata + workflow restore work.
The dual-expert routing (high-noise vs low-noise vs untagged) is
handled entirely on the backend by ``WanLoRACollectionLoader`` based on
each LoRA's recorded ``expert`` tag (set by the probe from the filename
heuristic in piece #5 of Phase 5). The FE just hands over the bag of
LoRAs; no per-list FE plumbing needed.
buildWanGraph:
- Calls addWanLoRAs(state, g, denoise, modelLoader) after the base
transformer edge is in place. The helper is a no-op when no Wan
LoRAs are enabled, so it's safe to call unconditionally.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(wan): detect LoRA variant and filter by main model
Wan 2.2 A14B (inner_dim=5120) and TI2V-5B (inner_dim=3072) LoRAs are not
interchangeable — applying one against the wrong main model crashes the
layer patcher with a tensor-shape error (e.g. A14B Lightning on TI2V-5B
mains produced ``shape '[3072, 3072]' is invalid for input of size 26214400``).
Probe Wan LoRAs' inner-dim at install time and record the family on a new
``variant`` field (``a14b`` / ``5b`` / null). The LoRA picker in the linear
view hides incompatible variants when the user selects a main, and the
graph builder filters any still-enabled mismatches at submit time with a
warning. Untagged LoRAs (probe couldn't identify) pass through so they
aren't silently hidden.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
feat(wan): ref-image panel, GGUF readiness, and auto-default sources
Wan 2.2 I2V now uses the global Reference Images panel (same UX as Qwen
Image Edit and FLUX.2 Klein) instead of pulling the conditioning image
from a canvas raster layer. Adds:
- WanReferenceImageConfig zod type + isWanReferenceImageConfig guard;
integrated into the ref-image discriminated union, settings panel,
layer hooks, and validators.
- 'wan' added to SUPPORTS_REF_IMAGES_BASE_MODELS, but the panel only
shows for the i2v_a14b variant (T2V and TI2V-5B don't consume ref
images, so the panel is hidden for them).
- buildWanGraph I2V branch reads the first enabled wan_reference_image
from refImagesSlice; the canvas-raster-as-ref path is removed. I2V
now only supports txt2img mode (canvas img2img/inpaint/outpaint
assert with a clear message).
GGUF Wan readiness check: GGUF mains carry only the transformer, so the
loader needs a Diffusers Component Source (or standalone VAE + UMT5-XXL
encoder) to resolve the VAE and text encoder. Without one, enqueue is
now blocked with a clear reason. The low-noise A14B partner expert
remains optional (loader falls back to the high-noise expert when it's
missing).
Adds standalone Wan VAE and Wan T5 Encoder selectors to the Advanced
accordion (Qwen pattern). Wires them as vae_model / wan_t5_encoder_model
on the wan_model_loader node — backend priority is standalone > diffusers
main > component source.
Auto-default on Wan selection (so GGUF users don't have to fiddle with
Advanced): when the new main is a Wan GGUF, fill the Component Source,
standalone VAE, and standalone T5 encoder with first available matches
if not already set. Component Source is matched by variant family
(A14B GGUF prefers an A14B Diffusers; TI2V-5B prefers a TI2V-5B
Diffusers) since the two families use different VAE channel counts
(16 vs 48); within A14B, T2V and I2V share VAE/encoder so they're
interchangeable as a source. Runs on every Wan selection (including
Diffusers -> GGUF switches), only fills empty slots.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): add Wan 2.2 starter models and bundle
Wan 2.2 starter pack (selected when the user picks the Wan 2.2 bundle)
brings up the minimal-cost path to running A14B T2V end-to-end:
- Standalone UMT5-XXL encoder and A14B VAE (so GGUF mains don't need
a full Diffusers download for their VAE/encoder sources).
- T2V A14B Q4_K_M and Q8_0 GGUF expert pairs (high + low noise).
- T2V Lightning V1.1 Seko rank-64 LoRA pair (4-step inference).
Additional Wan 2.2 starter models browseable from the model manager:
- Full Diffusers T2V A14B, I2V A14B, and TI2V-5B.
- I2V A14B Q4_K_M and Q8_0 GGUF expert pairs + Lightning V1 LoRA pair.
- TI2V-5B Q4_K_M and Q8_0 GGUFs + the 48-channel TI2V-5B VAE.
Each "high noise" GGUF lists its low-noise partner plus the shared VAE
and UMT5-XXL encoder as dependencies, so installing one of them pulls
in everything the loader needs. QuantStack's HighNoise/LowNoise file
naming and lightx2v's high_noise_model/low_noise_model.safetensors are
both picked up by the existing filename heuristic in the GGUF probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
docs(wan): add Wan 2.2 hardware requirements
Adds Wan 2.2 A14B (T2V/I2V) and TI2V-5B rows to the hardware
requirements table with rough VRAM/RAM guidance per quantization.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(wan): recall low-noise transformer, component source, and standalone VAE/T5
Wan-specific metadata fields embedded by the graph builder
(wan_transformer_low_noise, wan_component_source, wan_vae_model,
wan_t5_encoder_model, wan_guidance_scale_low_noise) had no recall
handlers in features/metadata/parsing.tsx, so recalling an image's
parameters would leave these fields empty. Adds a handler for each
that dispatches the matching paramsSlice action and renders a row in
the metadata viewer.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): add default Wan 2.2 T2V and I2V workflows
Ships two default workflows in the library, tagged so they appear in
"Browse Workflows" under the wan2.2 / text to image / image to image
tags:
- Text to Image - Wan 2.2: full T2V/TI2V-5B graph (model loader,
positive + negative encoders, denoise, l2i). Exposes the five
model slots, prompts, steps, dual CFG, and dimensions.
- Image to Image - Wan 2.2: I2V A14B graph that adds a
wan_ref_image_encoder. Exposes the reference image input plus
the standard fields.
Both follow default-workflow rules: IDs prefixed with default_,
meta.category = "default", and no references to user-installed
resources.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(videos): Phase 1 - backend video storage, records, REST API
Adds a parallel video pipeline alongside the existing image pipeline so the
gallery can host MP4 alongside PNGs. Implements:
- New service modules (parallel to image equivalents):
video_records/ record store + sqlite impl
video_files/ disk file store (mp4 + first-frame webp thumb)
videos/ orchestrating service
board_video_records/ board <-> video association
- migration_32 creates `videos` and `board_videos` tables
- /api/v1/videos/ router: upload, list, get DTO, /full (with HTTP Range
so HTML5 <video> seek/scrub works), /thumbnail, /metadata, star/unstar,
delete, batch delete, board add/remove
- LocalUrlService.get_video_url and SimpleNameService.create_video_name
- imageio[ffmpeg] dep for video encode (used in later phases)
- Wires all four new services into InvocationServices, dependencies.py,
api_app.py, and three test fixtures
Verified end-to-end against an in-memory db + tmp output dir: upload,
probe, save (file + thumbnail + record), DTO build, list, delete.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(videos): Phase 2 - polymorphic gallery list endpoint
Adds /api/v1/gallery/items/ and /api/v1/gallery/items/names returning a
unified time-sorted stream of images + videos so the frontend can render
them interleaved with a single virtualized query.
- gallery_common: GalleryItem discriminated union (kind + name + shared
fields + nullable video duration/fps), GalleryItemRef, names result
- gallery_default: SqliteGalleryService implements UNION ALL across the
images and videos tables, applying identical filters (origin/category/
is_intermediate/board_id/search) to each half; pagination via outer
ORDER BY + LIMIT/OFFSET; counts are summed across the two halves
- URLs are resolved at row -> DTO conversion time so each item routes to
the correct /api/v1/images or /api/v1/videos endpoint
- Wired into InvocationServices, dependencies.py, api_app.py, and the
three test fixtures
Existing /api/v1/images endpoints are unchanged so any non-gallery
consumers (queue, recall, metadata workflows) continue to work as-is.
Verified e2e: 2 images + 2 videos inserted in alternating order, both
list_items and list_item_names return the correct interleaved order;
category filter narrows to a single kind; starring an item bumps it to
the top when starred_first=True.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(videos): Phase 3 - frontend RTK endpoints + MP4 upload routing
Adds the typed API surface and upload integration so videos can be
uploaded through the same gallery upload button that handles images.
Schema: re-ran pnpm typegen against the running backend to pick up
VideoDTO, VideoRecordChanges, GalleryItem, GalleryItemKind,
GalleryItemRef, GalleryItemNamesResult and the two new paginated
result types.
RTK Query (services/api/endpoints/videos.ts) - parallel to images.ts:
listVideos, getVideoDTO, getVideoMetadata, getVideoNames, uploadVideo,
deleteVideo / deleteVideos, changeVideoIsIntermediate, starVideos /
unstarVideos, addVideoToBoard / removeVideoFromBoard. Imperative helpers
(getVideoDTO, getVideoDTOSafe, uploadVideo, uploadVideos) and the
useVideoDTO convenience hook ride alongside, mirroring the image side.
Tag types and invalidation: added Video / VideoList / VideoMetadata /
VideoNameList / BoardVideosTotal / GalleryItemList / GalleryItemNameList
to the api root. Board-affecting mutations now invalidate the polymorphic
gallery list/name caches so videos and images stay coherent once the
gallery wiring lands in Phase 4. Added a sibling
getTagsToInvalidateForVideoMutation helper.
Upload UX: useImageUploadButton.tsx's dropzone now accepts video/mp4,
video/webm, video/quicktime alongside the existing image MIMEs. The
drop handler splits files into image/video sets and routes each through
its own mutation; a new onUploadVideo callback parallels the existing
onUpload. Existing image-only callers pass through unchanged.
Polymorphic gallery query endpoints + the useGalleryItemDTO hook will
land with Phase 4 where they have actual consumers; the schema types
they'll need are already in place under @knipignore tags.
Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass; live curl against the running dev server
uploads an MP4 and serves both the webp thumbnail and the MP4 with
a working HTTP Range response (206 + Content-Range).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(videos): Phase 4 - mixed gallery grid with video play badge
Videos now appear in the same gallery grid as images, interleaved by
created_at. Video thumbnails get a centered play-button badge so they
read as videos at a glance; everything else (selection, virtualization,
search, paged/virtual gallery views, keyboard nav) is unchanged.
Approach: selection state stays `string[]` of names. The kind is
recovered from the filename extension (.mp4 = video, anything else =
image), which is reliable because the backend's SimpleNameService
always emits `<uuid>.png` for images and `<uuid>.mp4` for videos. This
sidesteps a 32-file cross-cut from changing the selection shape to a
discriminated union, and selection is persist-denylisted so no
migration is needed.
Frontend:
- new isVideoName helper in features/gallery/store/types
- new endpoints/gallery.ts (deferred from Phase 3): useGetGalleryItemNamesQuery
- new ImageGrid/GalleryItemPlayBadge: centered triangular badge over thumbnail
- new ImageGrid/GalleryItemVideoStarIconButton: video-typed star toggle
- new ImageGrid/GalleryVideoItem: counterpart to GalleryImage; reuses
galleryItemContainerSX, GalleryItemSizeBadge (width/height-only stand-in),
selection handling (single/shift/ctrl/cmd); alt-click falls through to a
normal select since comparison is image-only
- use-gallery-image-names now calls the polymorphic gallery names endpoint
and exposes a mixed flat name list (existing callers - paged grid, search,
navigation hotkeys - get the same shape)
- useRangeBasedImageFetching partitions visible names by extension; images
bulk-fetch via the existing getImageDTOsByNames mutation, videos dispatch
individual getVideoDTO queries (no batch endpoint yet)
- GalleryImageGrid's ImageAtPosition dispatches on isVideoName to render
GalleryImage or GalleryVideoItem; star hotkey dispatches to the right
star/unstar mutation based on kind
- pruned the now-unused useGetImageNamesQuery / isImageName exports
Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass; live curl of /api/v1/gallery/items returns
57 polymorphic items with video duration populated and image duration
null, /api/v1/gallery/items/names returns matching {kind, name} refs.
The useGalleryItemDTO hook is intentionally deferred to Phase 5 where
the polymorphic viewer is its first real consumer.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(videos): Phase 5 - inline video player in the image viewer
Selecting a video now renders a polymorphic preview inside the existing
viewer panel: thumbnail with a centered play button by default; clicking
play swaps in an HTML5 <video controls autoplay>. Switching to a
different item drops the video element back to idle (auto-pauses) and
selecting an image again returns to the normal image preview.
New components (features/gallery/components/ImageViewer/):
- VideoPlayButtonOverlay: large centered play button with hover/shadow,
used over the thumbnail in the idle state.
- CurrentVideoPreview: idle/playing state machine. Resets on
video_name change. The <video> src points at /api/v1/videos/i/.../full
which supports HTTP Range, so seek/scrub work natively in the browser.
New hook:
- common/hooks/useGalleryItemDTO: polymorphic DTO resolver that
dispatches between useImageDTO and useVideoDTO based on filename
extension (isVideoName). Centralizes the kind-dispatch the viewer
and toolbar both need.
Wiring:
- ImageViewer dispatches on galleryItem.kind to render CurrentImagePreview
or CurrentVideoPreview. The compare-image DnD drop target is hidden when
a video is selected (comparison is image-only).
- ImageViewerToolbar hides the image-specific action row
(CurrentImageButtons - load workflow, recall metadata, edit, etc.) and
the metadata viewer toggle when a video is selected. The general-purpose
ToggleProgressButton stays.
Out of scope (per the plan): video deletion from the viewer (use gallery
hover icons), video-specific metadata viewer, comparison-mode support
for videos.
Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(videos): accept MP4 (and other video) drops on the fullscreen dropzone
The gallery-wide drag-and-drop target lives in FullscreenDropzone, not
in useImageUploadButton (which only powers the upload button). It had
its own hardcoded image-only zod allowlist that rejected MP4 files
with "File type / extension is not supported".
- Broaden the zod refines to accept video/mp4, video/webm,
video/quicktime, video/x-matroska and the matching extensions
- Add isVideoFile helper, split dropped files into image/video sets,
and route each set through its own uploader (uploadImages /
uploadVideos). Both update their respective RTK caches and
invalidate the polymorphic gallery list/names.
- Skip the canvas-paste fast-path for single-video drops — the canvas
doesn't host videos as layers.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(videos): right-click context menu on video items
Adds a three-item context menu (delete, change board, download) on
right-click / long-press of any gallery video item. Mirrors the image
context menu's singleton-portal architecture so re-renders stay cheap.
New files:
- features/gallery/contexts/VideoDTOContext: small React context that
scopes the active video DTO to the menu items (parallels
ImageDTOContext).
- features/gallery/components/ContextMenu/MenuItems/
ContextMenuItemDeleteVideo: window.confirm + deleteVideo mutation.
Videos can't be referenced from canvas/nodes/refs, so the image
modal's usage analysis is unnecessary; a one-step confirm matches
the "minimal" scope.
ContextMenuItemDownloadVideo: reuses the existing useDownloadItem
hook against videoDTO.video_url / video_name.
ContextMenuItemChangeBoardVideo: dispatches videosToChangeSelected
and opens the (now polymorphic) ChangeBoardModal.
- features/gallery/components/ContextMenu/VideoContextMenu: singleton
pattern lifted from ImageContextMenu — registers gallery video
elements via a Map; right-click looks up the target node and opens
the menu at the cursor.
Extended files:
- features/changeBoardModal/store/slice: added video_names alongside
image_names plus a videosToChangeSelected action. The two arrays are
mutually exclusive — setting one clears the other.
- features/changeBoardModal/components/ChangeBoardModal: now dispatches
the matching video board mutations (add/removeVideoToBoard, plural
endpoints don't exist yet so videos move one at a time — the menu
acts on a single selection so this is a one-iteration loop).
- features/gallery/components/ImageGrid/GalleryVideoItem: registers
itself with useVideoContextMenu.
- app/components/GlobalModalIsolator: mounts the singleton.
Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): Phase 6 - Wan 2.2 T2V/I2V workflow nodes
Adds two new invocation nodes that produce MP4 videos from a Wan 2.2
A14B transformer + VAE, plus the supporting plumbing.
New invocations:
- WanVideoDenoise (wan_video_denoise) — multi-frame counterpart to
WanDenoise. Same per-step logic (CFG, MoE expert swap at the
boundary timestep, LoRA patching, scheduler dispatch) — reuses
_ExpertSwapper, _resolve_variant, and the scheduler/LoRA helpers
from wan_denoise. Difference: the noise tensor has a real temporal
dim built from num_frames, and the I2V condition is built across
all latent frames (frame 0 conditioned, rest zero). Defaults match
the Wan 2.2 reference: 832x480 / 81 frames / 40 steps / CFG 5.0
(high) + 4.0 (low). Inpaint / img2img are out of scope for this
first cut. TI2V-5B is rejected; T2V/I2V A14B only.
- WanLatentsToVideo (wan_l2v) — VAE-decodes 5D latents to RGB frames
via AutoencoderKLWan (T_pixel = (T_lat - 1) * 4 + 1), then encodes
an MP4 with imageio[ffmpeg] (libx264, yuv420p for browser
compatibility). The temp file is moved into outputs/videos/ via
context.videos.save().
Backend shared pieces:
- make_noise gains num_latent_frames (default 1, backward compatible).
- Added num_latent_frames_for(num_frames, scale=4) helper.
- New encode_reference_image_to_video_condition mirrors diffusers'
WanImageToVideoPipeline.prepare_latents with last_image=None and
expand_timesteps=False: pads the reference image with zero
pixel-frames, VAE-encodes the full pseudo-video, normalises, and
builds the 4-channel temporal-rearranged first-frame mask. Verified
numerically: 21 latent frames for num_frames=81, first latent
frame's 4 mask channels = 1, rest = 0.
- The existing single-frame encoder is left untouched.
Schema / context:
- New VideoField primitive (parallel to ImageField) and VideoOutput
invocation output (width/height/num_frames/fps/duration/video).
- New VideosInterface on InvocationContext with .save(source_path,
width, height, duration, fps, ...) returning VideoDTO. Mirrors
ImagesInterface — falls back to WithBoard / WithMetadata mixins
and embeds the queue item's workflow/graph as a JSON sidecar.
- WanRefImageConditioningField now carries num_frames so the denoise
nodes can sanity-check the I2V condition. WanRefImageEncoder bumps
to v1.1.0 and gains num_frames=1 input (use 81+ for video I2V; the
encoder dispatches between the single- and multi-frame helpers).
- Image WanDenoise now rejects multi-frame conditions with a clear
message pointing at WanVideoDenoise.
Verified: pnpm lint (5/5) green; pnpm tests (multiuser auth 122/122
+ broader suite via prior runs); numerical shape checks for noise
and ref-image condition; end-to-end smoke via VideoService.create.
A restart of the InvokeAI server is required to pick up the new
invocations in the workflow editor.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): add Wan 2.2 T2V and I2V starter video workflows
Two new default workflows for the workflow editor 'Browse' modal:
- 'Text to Video - Wan 2.2' — model loader -> two text encoders ->
wan_video_denoise -> wan_l2v. Exposes prompt, model picks, CFG
(high + low), dimensions, frames, fps, and steps.
- 'Image to Video - Wan 2.2' — same shape plus a wan_ref_image_encoder
feeding the denoise node's ref_image input. Exposes the reference
image and the frames field on the ref-image node (must match the
denoise node's frames — there is a clear validation error if they
diverge, but the starter has them in sync at 81).
Both default to the Wan 2.2 reference settings: 832x480, 81 frames @
16 FPS (~5 s), 40 steps, CFG 5.0 (high expert) + 4.0 (low expert),
seeded by a rand_int. Pass the existing _sync_default_workflows
validator (id starts with default_, meta.category=default).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(wan): startup crash from stringified VideoOutput annotation
run_app.py validates every invocation's return-type annotation against
the output-class registry. wan_latents_to_video.py had a stray
'from __future__ import annotations' which made the `invoke()` return
annotation a string ('VideoOutput') at runtime. The registry mismatch
triggered the unregistered-output warning path, which itself crashed
on output_annotation.__name__ because the annotation was a str:
AttributeError: 'str' object has no attribute '__name__'
The other Wan invocations don't use future annotations — drop the
import to match. Verified post-fix: api_app import populates 95
output classes, wan_l2v annotation resolves to the real VideoOutput
class and is in the registry.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): add Wan 2.2 Lightning T2V starter workflow
Same graph as 'Text to Video - Wan 2.2' but with two Apply LoRA - Wan
2.2 nodes chained between the model loader and the denoise node, and
defaults retuned for the Lightning distillation: 4 steps and CFG 1.0
on both experts (CFG=1 skips the negative-conditioning forward pass
entirely, ~20x faster than the 40-step / CFG-5.0 baseline at similar
quality).
Adapted from a user-saved workflow; cleaned for distribution by
stripping the install-specific model/LoRA key bindings (defaults
should not bake in local UUIDs), bumping to a fresh default_-prefixed
id with meta.category=default, exposing the two LoRA fields (lora +
weight) so users can swap LoRAs without diving into the canvas, and
flagging the negative-prompt node as unused at CFG=1.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): add Wan 2.2 Lightning T2V and I2V starter workflows
Two new default workflows that wire the Lightning LoRA pair into the
T2V and I2V video pipelines for a ~20x speedup:
- 'Text to Video - Wan 2.2 Lightning' — model loader -> apply LoRA
(high) -> apply LoRA (low) -> text encoders -> wan_video_denoise
-> wan_l2v. Defaults to 4 steps and CFG 1.0 (no negative branch).
Cleaned-up version of Lincoln's saved Lightning workflow: stripped
per-install model/LoRA keys, switched meta.category to 'default'
with a default_ id, and exposed both LoRA loaders' lora/weight/
target fields so users can swap LoRAs without diving into the
canvas.
- 'Image to Video - Wan 2.2 Lightning' — same chain plus a
wan_ref_image_encoder (v1.1.0 with num_frames) feeding the denoise
ref_image input. Defaults match the non-Lightning I2V starter
(832x480, 81 frames @ 16 FPS) but with 4 steps / CFG 1.0.
LoRA target defaults to 'auto' so properly-tagged Lightning LoRAs
route themselves; both workflow descriptions tell users to set
explicit 'high'/'low' targets if their LoRAs are untagged.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(wan): use FFMPEG plugin (not pyav) for MP4 encode
wan_latents_to_video was passing plugin='pyav' to iio.imwrite, but the
runtime only has imageio-ffmpeg installed (no PyAV). The encode step
at the very end of generation crashed with:
ImportError: The `pyav` plugin is not installed.
Use `pip install imageio[pyav]` to install it
Switch to plugin='FFMPEG' — backed by the bundled imageio-ffmpeg
binary that pyproject already requires via imageio[ffmpeg]. libx264
yuv420p is the FFMPEG plugin's default for .mp4, so the explicit
pixel_format is dropped (specifying it just produced a "Multiple
-pix_fmt options" warning).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): log VAE decode and MP4 encode milestones in wan_l2v
The video VAE decode + MP4 encode tail can take 30-90s on top of the
denoise loop, and the toast-style signal_progress() messages don't
land in the server log. Add context.logger.info() at:
- VAE decode start: latent frame count -> pixel frame count + resolution
- MP4 encode start: frames, fps, duration, dimensions
- MP4 encode complete: encoded file size
- Video saved: final video_name
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(wan): switch video thumbnail/probe to imageio[ffmpeg] backend
After wan_l2v wrote a successful libx264 MP4 to disk, the invocation
would hang in DiskVideoFileStorage.save() during the cv2.VideoCapture
thumbnail-extraction step. cv2 wheels on this build can't reliably
decode our libx264/yuv420p output (most often the wheel was compiled
without an h264 decoder, but the failure mode is silent hang rather
than a clear error). The net effect: the MP4 ends up in
outputs/videos but the queue item never completes, so the frontend
spinner spins forever and the gallery doesn't pick up the new entry.
Fix: rewrite extract_video_frame and probe_video to try imageio's
FFMPEG plugin first (same backend that did the encoding — so reading
our own output is guaranteed to work), with cv2 retained only as a
fallback for uploaded videos in formats imageio can't decode.
Also add fine-grained log lines + exception guards inside
DiskVideoFileStorage.save() so a future thumbnail failure can no
longer hang the whole save — it now logs a warning and continues,
leaving the video record in place even if the thumbnail step
errored. With logging at each step (video written, thumbnail
written, sidecar written) any future hang will be obvious from the
last log line.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(videos): handle VideoField outputs in invocation_complete
After wan_l2v wrote its MP4 successfully, the gallery and viewer were
never updated: the new video didn't appear and the viewer stayed stuck
on the previous "Saving video" progress spinner indefinitely.
Root cause: onInvocationComplete.tsx only inspected results for
isImageField / isImageFieldCollection. VideoField outputs were silently
dropped, so the polymorphic gallery list never invalidated and no
auto-switch happened. The viewer therefore kept rendering
CurrentImagePreview, whose ImageViewerContext-local $progressEvent /
$progressImage atoms intentionally aren't cleared on queue completion
when autoSwitch is on — they rely on the new image's DndImage onLoad
to clear them, which never fires for a video.
Fix: add isVideoField (mirrors isImageField against {video_name}) and
plumb video outputs through onInvocationComplete:
- getResultVideoDTOs pulls VideoDTOs via getVideoDTOSafe
- addVideosToGallery invalidates GalleryItemNameList / GalleryItemList
so the polymorphic gallery refetches and the new video shows up
- auto-switch dispatches the video name into selection (selection is a
polymorphic string[]; useGalleryItemDTO already discriminates by
filename extension)
The selection change swaps CurrentImagePreview for CurrentVideoPreview,
which unmounts the stale progress overlay along with it — so the stuck
spinner clears as a side-effect of the auto-switch.
Also drops the now-stale @knipignore on getVideoDTOSafe, which has a
real consumer now.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(video): add 'Frame from Video' invocation
Extracts a single frame from a VideoField input and saves it as a
regular ImageDTO via context.images.save, so it appears in the gallery
like any other generated image.
Primary use case is I2V "shot extension": take the last frame of a
Wan-generated clip (default frame_index=-1) and feed it back as the
reference image for the next clip, then stitch the MP4s to get videos
longer than the model's single-shot frame budget at a given VRAM.
Negative frame_index is resolved against the actual decoded frame count
via probe_video() rather than passed through to imageio — not all
imageio plugins handle index=-1 uniformly, and being explicit lets us
emit a precise out-of-range error.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(video): add 'Concatenate Videos' invocation
Joins two or more videos into a single MP4 with one of three transition
modes between consecutive clips:
- cut: hard splice, no blending. Total length = sum of inputs.
- crossfade: linear A→B dissolve over transition_frames. Each boundary
consumes N frames from both surrounding clips, shrinking total length
by N per boundary.
- fade_through_black: A fades to black, then B fades in. Each boundary
consumes N/2 from each side and emits N output frames — total length
is preserved.
Implementation decodes via imageio's FFMPEG plugin (matching wan_l2v on
the encode side) and runs the blends in numpy. All decoded frames are
kept in memory at once; fine for the few-hundred-frame I2V chains that
motivated this, would want streaming if anyone ever feeds in hour-long
uploads.
Up-front validation enforces matching dimensions across inputs and
checks that each clip has enough frames to spare from its head and tail
for the requested transitions — saves a wasted decode pass when the
transition window is too wide for one of the clips.
Pairs with 'Frame from Video' for I2V shot extension: generate N clips
chained via last-frame-as-ref-image, then glue them with a crossfade.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(videos): show full-resolution first frame in viewer
The viewer used a chakra <Image src={thumbnail_url}> in the idle (not-
playing) state, so once a clip auto-selected after generation the
preview snapped from the full-resolution denoise progress image to the
small WebP gallery thumbnail upscaled to fit — visibly soft compared to
what the user was watching seconds earlier.
Switch to a single <video> element that spans both states:
- idle: muted, no controls, preload="metadata". With no `poster` attr
the browser decodes and shows the video's actual first frame at full
resolution (this is the documented HTMLVideoElement default).
- playing: same DOM node with controls+audio toggled on, kicked off via
ref.play(). No reload between states — the decoded buffer carries
over.
`key={videoName}` swaps the element cleanly when the user moves to a
different clip, dropping any in-progress playback state.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(nodes): show 'Save in gallery' on video-output nodes
The footer checkbox was gated on useNodeHasImageOutput, which only
matched ImageField outputs. wan_l2v and video_concat produce VideoField
and so had no toggle — users had no UI path to flip is_intermediate on
them, even though VideoOutput goes through context.videos.save and
lands in the gallery the same way ImageOutput does.
Rename the hook to useNodeHasGalleryOutput and extend it to match
VideoField as well. Update the three call sites (the hook itself,
the checkbox, and the footer wrapper) so the toggle and the footer
render whenever a node produces something destined for the gallery.
The image primitive ('image' type) is still excluded since it doesn't
save a new image; no equivalent video primitive exists yet, so no
analogous exclusion is needed for VideoField.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore: remove unwanted planning documents
* chore: fix ruff I001 import-order violations
Auto-fix from `ruff check --select I001 --fix`. Touches 10 files across
the Wan and videos changes where added imports landed out of order.
No behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(videos): restrict uploads to MP4 only
The upload allowlist previously included .mov/.webm/.mkv, but the
names service (create_video_name) unconditionally emits {uuid}.mp4
and we don't transcode on upload. The result: non-MP4 containers were
stored under a .mp4 name and served with the .mp4 MIME type, which
silently broke playback in browsers when the container didn't match.
Drop the non-MP4 extensions from ACCEPTED_VIDEO_EXTENSIONS and tighten
the accepted MIME prefix to "video/mp4". Wan-generated output is MP4
anyway, so this matches current reality. If we want to support more
containers later, the right move is to extend the names service to
preserve the source extension, then re-add the formats here.
Also drops the now-dead suffix-detection block in upload_video and
the os import it required.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore(videos): clean up stale @knipignore on consumed hooks
useDeleteVideoMutation, useAddVideoToBoardMutation, and
useRemoveVideoFromBoardMutation are now consumed by Phase 4 components
(context menu, change-board modal) but were still annotated with the
multi-phase @knipignore tag — that generated false-positive knip
warnings and misrepresented the implementation status.
Move those three into the unconditional export block. The remaining
five hooks (useListVideosQuery, useGetVideoMetadataQuery,
useGetVideoNamesQuery, useDeleteVideosMutation,
useChangeVideoIsIntermediateMutation) are still unused in the current
codebase and stay under a narrower @knipignore.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* docs(videos): document invalidate/select race in addVideosToGallery
The video gallery path uses tag invalidation rather than an optimistic
insert (the image path's `insertImageIntoNamesResult` doesn't have a
polymorphic equivalent yet). Because invalidation kicks off an async
refetch, the `imageSelected` dispatch below it fires before the new
video name is in `imageNames`, so the gallery grid's
`useKeepSelectedImageInView` no-ops on its first pass. The scroll
self-corrects on the next pass when the refetch lands and the
`imageNames` dep updates.
The user-visible effect is just a small lag on gallery scroll-to-
selection — the viewer selection applies immediately — so this is a
documented limitation rather than a bug. Worth a follow-up if the
lag becomes noticeable in practice.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(video): add Video Primitive invocation and VideoField input UI
Mirrors the Image Primitive flow end-to-end for videos. Users can now
drag a video from the gallery onto a "Video Primitive" node and feed
its output into downstream nodes like Frame from Video or Concatenate
Videos — exactly the way Image Primitive feeds the rest of the image
pipeline.
Backend (invokeai/app/invocations/primitives.py):
- New VideoInvocation, declared *after* VideoOutput so the return
annotation is a real class (not a forward-ref string). Stringified
output annotations crashed startup before — see cac366229a — so the
ordering matters.
Frontend:
- Register VideoField as a stateful field type in types/field.ts:
zVideoFieldType, zVideoFieldValue, zVideoFieldInputInstance/Template,
output template + type guards, plus entries in the four stateful
unions (FieldType, FieldValue, InputInstance, InputTemplate).
- buildFieldInputTemplate / buildFieldInputInstance gain VideoField
branches so OpenAPI-derived templates resolve correctly.
- nodesSlice: fieldVideoValueChanged reducer + export.
- imageActions/actions.ts: setNodeVideoFieldVideo helper.
- dnd.ts: singleVideoDndSource + setNodeVideoFieldVideoDndTarget,
wired into the dndTargets array.
- GalleryVideoItem: register itself as a drag source so videos in the
gallery actually drag (previously they were click-only).
- VideoFieldInputComponent: parallel to ImageFieldInputComponent —
shows the WebP thumbnail with a dimensions badge, accepts video DnD,
drops stale references on reconnect if the underlying video was
deleted.
- InputFieldRenderer: dispatch VideoField templates to the new
component (placed right after the ImageField branch).
- useNodeHasGalleryOutput: also exclude the new `video` primitive type
so the "Save in gallery" toggle does not render on the pass-through
node (same treatment the `image` primitive already gets).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
fix(video): allow video drops to reach DnD target handlers
useDndMonitor is the global drop monitor that actually invokes each
target's handler() — DndDropTarget only does enter/leave bookkeeping.
Its canMonitor gate explicitly allowlists source types and only listed
singleImageDndSource + multipleImageDndSource. So when a video was
dragged from the gallery onto a VideoField input, the drop was visible
to the DOM but the monitor silently filtered it out, the handler never
ran, and fieldVideoValueChanged was never dispatched.
Add singleVideoDndSource to the allowlist. Dropping a video onto a
Video Primitive (or any other VideoField input) now wires the asset
into the field as intended.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore(backend): ruff
* chore(frontend): typegen
* feat(wan): split Wan 2.2 starter bundle into T2V and I2V
Replaces the single ~63 GB Wan 2.2 bundle with two smaller bundles so
users only pay for the capability they need. T2V (~36 GB) covers
text-to-video plus a low-VRAM image-to-video option via TI2V-5B; I2V
(~32 GB) adds the heavier I2V-A14B path. Drops the Q8 T2V pair from the
default bundle — both Q8 variants and full Diffusers builds remain
available as a-la-carte starters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(videos): tighten multiuser isolation in list and board-move endpoints
Three related fixes flagged in code review (PR #9163, JPPhoto):
1. Video and gallery list/name SQL paths only filtered by user_id when
board_id was the literal "none" sentinel. When the URL parameter was
omitted entirely, no user filter applied and non-admin callers could
enumerate every user's videos / mixed gallery items. Added an
explicit per-user isolation branch for the omitted case.
2. /v1/videos/ and /v1/videos/names accepted explicit board IDs with
no read-access check; the route now mirrors the images and gallery
routers and calls _assert_board_read_access for non-"none" values.
3. add_video_to_board and remove_video_from_board only validated video
ownership, not destination/source board write access — a caller
could move their video into someone else's private board. Added
_assert_board_write_access and a strict _assert_video_direct_owner
helper (no board-owner / public-board fallback) for board-move ops,
mirroring _assert_image_direct_owner.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(boards): cascade video deletion when deleting a board with media
Previously delete_board only handled images. With include_images=true the
backend would delete images on the board but the videos would silently
cascade out of board_videos and survive as uncategorized records — almost
certainly not what the caller intended. Without include_images, the same
mismatch meant the response could not report affected videos.
Now:
- include_images=true also calls videos.delete_videos_on_board
- include_images=false collects the soon-to-be-uncategorized video names
- DeleteBoardResult gains deleted_board_videos and deleted_videos fields
(default empty so existing clients are unaffected)
Frontend deleteBoard / deleteBoardAndImages mutations gain the matching
VideoList / VideoNameList / GalleryItem* tag invalidations so the
polymorphic gallery and video list views refresh.
Reported in code review (PR #9163, JPPhoto).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(videos): return affected_boards from board move/remove endpoints
removeVideoFromBoard previously returned VideoDTO; the frontend then read
result.board_id (null after removal) and only invalidated the 'none'
board cache — the previous board's list stayed stale until refetch.
addVideoToBoard had the same problem (the route never knew the source
board, so the old-board cache was never invalidated).
Mirror the image equivalents (board_images.py): the routes now return
AddVideosToBoardResult / RemoveVideosFromBoardResult with the moved
video name(s) and the full set of affected board IDs. Both old and new
boards get invalidated atomically. Frontend mutations updated to
consume the new shape via getTagsToInvalidateForBoardAffectingMutation
on result.affected_boards.
The auto-generated schema.ts will need a typegen pass after the dev
server restart to pick up the new response types.
Reported in code review (PR #9163, JPPhoto).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(videos): stream uploads, FileResponse for full video + thumbnail
Three perf items from the code review (PR #9163, JPPhoto):
- upload_video read the entire UploadFile into a Python bytes object
before writing to the temp file. Multi-GB videos allocated multi-GB
buffers. Now chunk-stream into the temp file with a 1 GB per-upload
cap (HTTP 413 on overflow). Cap is intentionally generous — the goal
is RAM-exhaustion protection, not content policy.
- get_video_full read the whole MP4 into RAM when no Range header was
present. Browsers usually send Range, but curl / direct downloads /
CDN edge fetches do not, and a multi-GB load per such request is a
trivial DoS vector. Replaced with FileResponse (sendfile).
- get_video_thumbnail similarly buffered the WebP. Thumbnails are tiny
so this was minor, but FileResponse is idiomatic and shaves the
unnecessary copy.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(videos): localize video UI strings
- Add gallery.deleteVideo_one / deleteVideo_other, deleteVideoConfirmation,
and playVideo to en.json
- ContextMenuItemDeleteVideo: drop the inline English defaultValue (the
translation key now exists) and use gallery.deleteVideo for aria/tooltip
(was reusing gallery.deleteImage so it rendered "Delete Image")
- VideoPlayButtonOverlay: replace the hardcoded "Play video" aria with
t('gallery.playVideo')
Reported in code review (PR #9163, JPPhoto).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(video-invocations): exact frame counts and odd-tf transitions
video_frame_extract: resolving frame_index=-1 previously computed
n_frames as round(duration * fps). For VFR uploads or containers with
approximate metadata that can overshoot the actual decoded frame count,
making the last-frame extraction fail. Use iio.improps(plugin='FFMPEG')
for the exact decoder count when available; fall back to the duration
* fps estimate only if the props query fails.
video_concat fade_through_black: with an odd transition_frames the
symmetric half = tf // 2 split emitted tf - 1 frames per boundary,
violating the documented "emits transition_frames" contract. Split
asymmetrically (tail_half = tf // 2, head_half = tf - tail_half) so
the emitted count equals tf exactly for both even and odd values.
Validation and docstring updated to match.
Verified with manual cases: tf=1, tf=4, tf=5 all emit the documented
total length for two 10-frame inputs.
Reported in code review (PR #9163, JPPhoto).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* docs(gallery): describe video gallery items, upload, and deletion
Update the Gallery Panel docs to reflect the polymorphic gallery added
in the Wan 2.2 video feature branch:
- Gallery intro now mentions images + videos coexist on boards.
- Board deletion warning clarified to cover both kinds of media.
- New "Videos in the Gallery" section covering: how video items appear
(first-frame thumbnail + play badge), MP4-only upload constraint with
the typical re-encode command, the video context menu, and that
videos count toward board totals.
Reported in code review (PR #9163, JPPhoto).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* test(videos): regression coverage for PR #9163 review fixes
Adds tests that pin the behaviour fixed in the JPPhoto review and would
catch a recurrence:
- tests/app/services/video_records/test_video_records_sqlite.py
get_many / get_video_names: non-admin callers only see their own
videos when board_id is omitted; admins see all; the "none" branch
still filters by user.
- tests/app/services/gallery/test_gallery_default.py
Same multiuser isolation guarantee through the polymorphic gallery
union for both images and videos.
- tests/app/routers/test_videos_multiuser.py
/v1/videos/ and /v1/videos/names: 403 when a non-owner passes an
explicit private board_id; 200 for owners, admins, "none", and omitted
board_id (the auth-required smoke tests pin the 401 paths too).
- tests/app/routers/test_boards_multiuser.py
Adds two delete-board cases proving the video cascade: include_images
invokes delete_videos_on_board and reports deleted_videos; the
no-include path reports deleted_board_videos without calling the
destructive service. Existing fixture extended to stub the video
services that the new route logic now touches.
- tests/app/invocations/test_video_concat.py
Parametric coverage that fade_through_black emits exactly tf frames
for both even and odd tf, plus three-clip chains, plus the crossfade
and cut/zero-tf cases as guards.
- tests/app/invocations/test_video_frame_extract.py
_decoder_frame_count returns the exact count via the cv2 fallback for
several clip lengths and gracefully returns None for missing /
non-video inputs (caller falls back to duration * fps).
Bug found during test authoring: _decoder_frame_count over-flowed int()
on iio's "inf" nframes for libx264 streams, and improps never returns a
real count for that codec anyway. Helper now ignores non-finite shapes
and falls back to cv2's CAP_PROP_FRAME_COUNT, which gives the exact
value for libx264.
schema.ts regenerated to pick up the AddVideosToBoardResult /
RemoveVideosFromBoardResult / extended DeleteBoardResult types added
in earlier commits in this series.
All 70 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): add 'Wan 2.2 I2V Ideal Dimensions' invocation
Computes Wan I2V-compatible (width, height) for a source W×H at a target
short-side resolution (e.g. 720 for "720p"), snapping each output to a
multiple of 16 (Wan's transformer patch_size × VAE 8x pixel-grid
constraint enforced by wan_ref_image_encoder).
Replaces the 6-node math chain (Float Math × 4 + Float To Integer × 2)
that was otherwise required to compute these dimensions from an arbitrary
input image. Wire the Image Primitive's width/height outputs into this
node, and feed its (width, height) outputs into both
wan_ref_image_encoder and wan_denoise (they must match).
Three rounding modes:
- nearest (default): minimizes aspect-ratio drift
- floor: guaranteed not to exceed unsnapped target (safer for VRAM)
- ceiling: rounds up
Output schema reuses IdealSizeOutput so it slots into existing pipes that
already consume Ideal Size — SD1.5, SDXL.
Includes regression tests covering the documented common-case table,
all three rounding modes, postcondition invariants (multiple of 16,
aspect ratio within 1.2%, never zero), and input validation.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): swap target_short_side int for 480p/720p/1080p preset dropdown
Wan 2.2 was trained at 480p and 720p; a free integer encouraged users
to pick noncanonical short sides that the model handles poorly. Replace
the int field with a Literal dropdown of "480p" / "720p" / "1080p"
(via ui_choice_labels) so the UI surfaces the canonical choices.
1080p is included with a label noting it's extrapolated from training
(not a Wan native size) — useful for users with VRAM headroom but
shouldn't be the default.
Version bumped to 1.1.0 since the field schema changed (the node was
only committed locally; no published workflow needs migrating).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(wan): empty_cache around the I2V reference-image VAE encode
Two-sided fix to avoid VRAM allocator fragmentation that was causing the
subsequent denoise-transformer partial load to OOM:
- Before vae.encode(): clears blocks left over from earlier nodes (the
denoise expert swap especially leaves the cache fragmented).
- After the condition tensor is on CPU: returns the VAE encode's
intermediates so the next partial_load_to_vram sees a real free
contiguous range.
Mirrors the same pattern in wan_latents_to_image.py and
wan_latents_to_video.py — those are the existing precedent. The cost is
a handful of microseconds per encoder invocation and only the cache
state is touched; model weights stay resident.
Observed-by symptom from a workflow review: at encoder=480x720 and a
source image of 880x1184, the encoder ran fine but the I2V high-noise
expert failed to partial-load with a cryptic CUDA OOM at
_load_state_dict_with_fast_device_conversion. Pre-resizing the source
to 80% incidentally cleared the allocator state and let the run
succeed; this fix removes the incidental dependency on source size.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): support TI2V-5B in the video denoise node (T2V mode)
The video denoise node previously hard-errored on TI2V-5B with "not
supported." Most of the surrounding machinery (variant-aware spatial
scale, variant-aware scheduler, single-expert ExpertSwapper path) was
already in place — the gate just needed lifting and the hard-coded
A14B latent channel count needed to follow the variant.
Changes:
- Drop the upfront "TI2V-5B is not supported" raise.
- Use get_default_latent_channels(variant) so latents are 48-channel for
TI2V-5B and 16-channel for the A14B family (matches the image denoise
node's existing logic).
- For TI2V-5B with a Reference Image input, raise a sharper, accurate
error that explains TI2V-5B's I2V uses diffusers' expand_timesteps
path (first-frame-mask blend + per-position timestep gating) which
this node does not implement yet — pointing the user at the working
T2V path or the I2V-A14B model.
- Update the transformer field description to reflect what's now
supported.
Image-to-video with TI2V-5B remains a follow-up; the conditioning math
is genuinely different from A14B (no 36-channel concat) and warrants a
separate code path rather than parameterising this one.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(wan): instantiate TI2V-5B VAE with the right architectural config
The single-file Wan VAE loader was always calling
``AutoencoderKLWan(z_dim=config.latent_channels)`` and relying on
diffusers' constructor defaults for every other parameter — but those
defaults match the Wan 2.1 / A14B VAE (base_dim=96, in/out=3, 8x
spatial, no patchify). For TI2V-5B's Wan 2.2-VAE the architecture is
materially different:
- base_dim=160, decoder_base_dim=256
- in_channels=12, out_channels=12 (3 RGB x 2x2 patch)
- patch_size=2
- scale_factor_spatial=16
- is_residual=True
- 48-vector latents_mean / latents_std (required for the model's
encode/decode normalisation to produce non-garbage outputs)
Loading the TI2V-5B VAE state_dict into the default-constructed model
failed with shape mismatches throughout the encoder + decoder, surfaced
in wan_l2v as "Error(s) in loading state_dict for AutoencoderKLWan."
This commit routes z_dim=48 to a verbatim copy of the TI2V-5B VAE config
(from vae/config.json in Wan-AI/Wan2.2-TI2V-5B-Diffusers); z_dim=16
keeps the previous A14B / Wan 2.1 default behaviour.
Verified end-to-end: both kwargs construct cleanly and produce the
expected layer shapes (decoder.conv_out emits 12 channels for TI2V-5B,
3 channels for A14B).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(wan): variant-aware default scheduler for standalone installs
When the main model has no on-disk ``scheduler/`` directory (every
standalone GGUF / single-file install), ``_build_scheduler`` previously
fell back to ``FlowMatchEulerDiscreteScheduler()`` for every variant.
That's correct for the A14B family but wrong for TI2V-5B, which ships
``UniPCMultistepScheduler`` with ``flow_shift=5.0`` +
``prediction_type="flow_prediction"`` + ``use_flow_sigmas=True``. The
mismatch produces drifty samples on TI2V-5B.
Add a ``_default_scheduler_for_variant`` helper that reconstructs the
right scheduler from the variant tag (values verbatim from each
variant's ``scheduler/scheduler_config.json`` in the matching
Wan-AI/Wan2.2-*-Diffusers repo). The on-disk-config-present path is
unchanged — if the model ships a scheduler dir, that wins.
Full scheduler-selection UI is deferred to a future PR per discussion;
this special-case keeps the standalone TI2V-5B path producing the right
sampler without surfacing a new field.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(wan): TI2V-5B image-to-video support
TI2V-5B I2V uses a fundamentally different conditioning scheme from
A14B I2V. Implement diffusers' ``expand_timesteps`` path so the same
``Reference Image - Wan 2.2`` node and ``Denoise Video - Wan 2.2`` node
work for both variants, dispatched by VAE z_dim / transformer variant.
Encoder side (wan_ref_image_extension.py / wan_ref_image_encoder.py)
- Add ``encode_reference_image_to_ti2v_condition`` that VAE-encodes a
single image frame to ``[1, 48, 1, H/16, W/16]`` with the Wan2.2-VAE
normalisation, no mask channels.
- ``WanRefImageEncoderInvocation`` dispatches on ``vae.config.z_dim``:
z_dim=48 → TI2V-5B path, z_dim=16 → existing A14B path.
- Enforce ``multiple_of=32`` for width/height in the TI2V-5B case
(16x VAE * 2 transformer patch = pixel dims must divide by 32) with
a clear error message pointing at the constraint.
Denoise side (wan_video_denoise.py)
- Replace the "TI2V-5B I2V not supported" raise with a variant-aware
dispatch on ``ref_condition.shape`` and ``variant``.
- For TI2V-5B I2V build a ``first_frame_mask`` once (0 at frame 0, 1
elsewhere). At each step:
latent_model_input = (1 - mask) * condition + mask * latents
temp_ts = (mask[0,0,:,::2,::2] * t).flatten()
timestep = temp_ts.unsqueeze(0).expand(B, -1)
Per-token timesteps gate the model: frame 0 sees t=0 (locked to
condition), other frames see t (normal denoise).
- After the denoise loop, re-clamp frame 0 to the clean condition so
the locked first frame doesn't show scheduler drift in the final VAE
decode. Mirrors WanImageToVideoPipeline:813-814.
- Skip the encoder-num_frames-must-match check for TI2V-5B (its
condition is always single-frame regardless of output length).
Tests
- Three new tests on encode_reference_image_to_ti2v_condition covering
output shape at small and Wan-realistic dims plus the no-mask-channels
invariant. Full video-denoise integration tests would need a new
fixture stack (none exist for wan_video_denoise yet) — deferred.
A14B I2V is unchanged. TI2V-5B T2V (added in the previous commit) is
unchanged. Verified at the import + encoder-shape level; end-to-end
verification requires a TI2V-5B I2V workflow.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(videos): show denoise progress overlay over the video viewer
CurrentVideoPreview rendered only the <video> element, so when the
last-selected gallery item was a video, a freshly-started render's
denoise preview images had nowhere to display — the user saw the
static first-frame still of the previously-loaded video until the new
render's final video swapped in.
Mirror CurrentImagePreview's progress-overlay pattern: subscribe to
$progressImage / $progressEvent, gate on selectShouldShowProgressInViewer,
and render a ProgressImage stack on top of the video when a render is
in progress. Hide the play-button overlay while progress is showing so
it doesn't sit on top of the preview.
Reported by Lincoln during TI2V-5B testing: previews started working
after restarting the server only because there was no video loaded at
that point; once a video was selected, the previews silently dropped.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore(lint): apply ruff format + isort across recent Wan video work
ruff check found one I001 (import order) in
``invokeai/backend/model_manager/load/model_loaders/vae.py`` and ruff
format flagged five files. All cosmetic; no behaviour changes.
- vae.py: import reorder
- video_concat.py: minor reflow
- test_wan_ideal_dimensions.py / test_boards_multiuser.py /
test_videos_multiuser.py: prettier-style wrapping
Verified: full ruff check + ruff format --check clean, 141 backend
tests pass, and ``pnpm lint`` (knip + dpdm + eslint + prettier + tsc)
all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* docs(features): add user guide for Wan 2.2 video generation
Comprehensive guide covering:
- The three Wan 2.2 variants (T2V-A14B, I2V-A14B, TI2V-5B), their
conditioning differences, and the dual-expert MoE explanation
- Lightning LoRA distillation for 4-step A14B inference
- Starter bundles (Text-to-Video and Image-to-Video splits)
- Workflow setup for T2V and I2V with the constraint matrix:
* frame count: (num_frames - 1) % 4 == 0
* pixel dims: multiple of 16 for A14B, 32 for TI2V-5B
* encoder + denoise must agree on width/height
- The chain-and-concat trick for making longer videos, with the
bridge-frame degradation mitigations
- Troubleshooting: OOM, late-frame artifacts, dim mismatches, VAE
load errors, scheduler issues, preview-not-appearing, MP4 glitches
Lands under Features → Video Generation (experimental). Astro
auto-generates the sidebar from features/ so no nav config change
needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(nodes): skip image-DTO fetch for videos in Current Image node
CurrentImageNode unconditionally called useImageDTO(lastSelectedItem)
even when the selected gallery item was a video, firing
GET /api/v1/images/i/<uuid>.mp4 on every video thumbnail click. The
endpoint 404s and the backend logged "Image record not found" each
time — benign but noisy.
Apply the same null-skip pattern useGalleryItemDTO uses: pass the name
only when it's not a video, so RTK Query skips the request for video
selections. Current Image is image-only by design, so videos rendering
the empty fallback matches existing behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(videos): clear stale progress overlay + force first-frame paint
Two viewer bugs after auto-switching to a freshly-rendered video:
- The denoise progress overlay never cleared. CurrentImagePreview clears
the ImageViewerContext $progressImage/$progressEvent atoms via DndImage's
onLoad callback; the video viewer had no analog, so the last progress
still sat on top of the new video forever — clicking other video
thumbnails did nothing visible, and only selecting an image (which
fires onLoadImage via DndImage) cleared it.
- Even with the overlay gone, the <video> element rendered its black
background instead of the first frame. preload="metadata" loads
dimensions/duration but doesn't guarantee a decoded first frame on all
browsers; an explicit seek is needed to force a paint.
Wire onLoadedMetadata to (1) call onLoadImage() — mirroring DndImage's
onLoad — and (2) nudge currentTime to 0.0001 so the decoder paints the
first frame without measurably advancing playback.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(hotkeys): skip image-DTO fetch for videos in GlobalImageHotkeys
Companion to a3bdc3304e (CurrentImageNode). GlobalImageHotkeys is a
mounted-everywhere singleton that wires recall hotkeys (seed, prompts,
remix, etc.) to whatever item is currently selected. It was passing the
raw selection name through to useImageDTO unconditionally, so every video
thumbnail click fired GET /api/v1/images/i/<uuid>.mp4 → 404 and the
"Image record not found" log line.
Gate on isVideoName(), mirroring the polymorphic null-skip pattern in
useGalleryItemDTO. Recall hotkeys don't apply to videos anyway, so this
just suppresses the noise.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(wan): empty CUDA cache between A14B expert swaps
The dual-expert swapper releases the active expert via its context
manager exit, but PyTorch's caching allocator retains the freed blocks
as reserved-not-yet-claimable space until empty_cache runs. The next
partial_load_to_vram for the incoming expert then sees a fragmented
free pool and offloads layers it could otherwise have kept on device.
Users running A14B observed the low-noise expert ending up far more
CPU-resident than the high-noise one on otherwise identical settings —
that was the leftover reservation from the high-noise expert masking
real free VRAM.
Call TorchDevice.empty_cache() between the release and the next load.
Same pattern as the VAE-encode fix earlier in this branch.
Regression test in test_wan_expert_swapper.py mocks empty_cache and
asserts it fires on every actual swap but not on a same-label re-get.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(videos): allow drag-and-drop to change a video's board
Dropping a video thumbnail onto a board in the boards list was a no-op
(the dnd target only accepted image sources). Extend
addImageToBoardDndTarget and removeImageFromBoardDndTarget to also accept
SingleVideoDndSourceData and dispatch the corresponding video mutations.
Permission UX mirrors the image path:
- Same canMoveFromSourceBoard gate (owner / public source board)
- Same "do nothing if dropping on the current board" early-out
Backend enforcement on /api/v1/videos/board already mirrors the image
endpoints — _assert_board_write_access on the destination plus
_assert_video_direct_owner on the video. The frontend gate intentionally
mirrors only the source-board part of that, leaving the direct-owner
check to surface as a 403 on attempt (same compromise as images, where
the client doesn't have per-item owner info to gate cleanly).
Multi-video drag is not supported yet (the gallery only registers a
single-video draggable per item, no multi-select bundle), so this only
wires the SingleVideoDndSourceData path.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(wan): force outgoing A14B expert off GPU on swap
The previous empty_cache() fix (53b2f4d4c7) was insufficient. unlock()
only decrements the cache record's lock counter — the weights stay on
GPU until the cache's automatic offload decides to free them on the
next lock(). That heuristic uses
``torch.cuda.memory_allocated() - working_mem`` to estimate free space,
which under-frees when the previous denoise step's workspace
activations are still allocated alongside the just-unlocked expert.
The user-visible symptom was a log line like
Loaded model '...:transformer' onto cuda device in 0.37s.
Total model size: 9203.13MB, VRAM: 2381.18MB (25.9%)
for the incoming low-noise expert, while the high-noise expert
continued to hold ~9 GB of VRAM.
The swapper now stashes the LoadedModel info handle and, on each swap,
explicitly invokes ``cached_model.full_unload_from_vram()`` on the
outgoing expert before locking the incoming one. This sidesteps the
heuristic and guarantees the previous expert's weights leave GPU
before partial_load_to_vram measures available room.
The access path ``info._cache_record.cached_model`` reaches into a
private attribute — there is no public LoadedModel API for "unload
from VRAM but keep in RAM" today, and a broader backend refactor felt
out of scope. The call is wrapped in getattr/try-except and pinned by
a regression test so a future refactor breaks the test, not the swap.
Tests:
- Updated existing dual-expert lifecycle test to expect the new
full-unload step in the swap log sequence.
- New test_outgoing_expert_force_unloaded_from_vram covers the
per-swap behavior (outgoing only, no initial unload).
- New test_force_unload_failure_does_not_break_swap pins the
defensive fallback so swap reliability survives a future LoadedModel
refactor.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(gallery): restore shift/ctrl-click range selection in image grid
GalleryImage's modifier-key click handler was reading the legacy
imagesApi getImageNames cache to compute range-selection indices, but
the gallery grid was switched to the polymorphic galleryApi
getGalleryItemNames endpoint (the only source that includes videos).
The legacy cache is no longer populated for the grid, so the
ordered-name list came back empty and the handler fell into its
"no names cached" early-return:
if (imageNames.length === 0) {
if (!shiftKey && !ctrlKey && !metaKey && !altKey) {
dispatch(selectionChanged([imageName]));
}
return;
}
making shift- and ctrl-click no-ops.
GalleryVideoItem already had the correct reader inlined as a private
helper. Hoist it to a shared module
(features/gallery/store/selectCachedGalleryItemNames) so both grids
use the polymorphic cache, and update GalleryImage to call it.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore(typegen): regenerate schema.ts
Refresh of the OpenAPI-derived TypeScript bindings against the current
backend. No hand edits — this is the output of the typegen step
re-run against the Wan video routes and recent backend changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore(startup): silence HF tokenizers fork-after-parallelism warning
Set TOKENIZERS_PARALLELISM=false at startup (via os.environ.setdefault
so users can override) before any HF library is imported.
The Rust ``tokenizers`` library warms a thread pool the first time a
tokenizer runs — for us that's UMT5 / T5 text encoding during Wan /
FLUX / SD3 conditioning. Every subsequent fork() then logs
huggingface/tokenizers: The current process just got forked, after
parallelism has already been used. Disabling parallelism to avoid
deadlocks...
In video generation we fork on every MP4 encode (imageio's FFMPEG
plugin uses subprocess.Popen → fork+exec), so this warning lands once
per generation in the server log. The advisory is benign — the child
correctly falls back to single-threaded tokenization before exec(),
and the parent's thread pool is unaffected — but the noise obscures
real warnings.
Setting the env var before any HF import prevents the thread pool
from warming up at all, so the fork detector stays quiet without
sacrificing anything: tokenization happens once per generation and
isn't a hot path for us.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore(startup): hoist TOKENIZERS_PARALLELISM=false to module level
Follow-up to 2106f10ec4 — the previous attempt set the env var inside
``run_app()``, which races against any transitive HF import triggered
by the console-script's
from invokeai.app.run_app import run_app
If ``tokenizers`` is imported anywhere in that import chain (directly
or via diffusers/transformers re-exports), the library's fork
detector registers before our setdefault runs and the warning still
fires.
Move the setdefault to module level so it executes the instant
``run_app.py`` is loaded — i.e. before the function defs are even
parsed, and well before any HF library has a chance to import.
Note for testing: jurigged hot-reload only re-runs function bodies,
so picking up this fix requires a full server restart, not just a
file save under ``--dev-reload``.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(videos): replace window.confirm with ConfirmationAlertDialog
Use the in-app delete-confirmation dialog (the same Chakra
ConfirmationAlertDialog the image flow uses) instead of the browser's
window.confirm() prompt. Matches the visual + interaction language of
the rest of the gallery and picks up the shared
``shouldConfirmOnDelete`` system preference — flipping the "Don't ask
me again" toggle now silences the prompt for both images and videos.
Implementation mirrors features/deleteImageModal/ but trimmed: the
image dialog computes "usage" (canvas layers, node fields, reference
images, upscale source) so the user knows what they'll break. Videos
have no analogous attachment points, so the video state machine is a
straight confirm-then-delete with no usage analysis.
- features/deleteVideoModal/store/state.ts — nanostores atom + an
awaitable ``deleteVideosWithDialog`` that opens the dialog and
resolves/rejects on confirm/cancel. Skips the dialog entirely when
shouldConfirmOnDelete is off.
- features/deleteVideoModal/components/DeleteVideoModal.tsx —
ConfirmationAlertDialog with the new deleteVideoPermanent message
and the shared "Don't ask me again" switch.
- GlobalModalIsolator.tsx — mount the new modal alongside
DeleteImageModal.
- ContextMenuItemDeleteVideo.tsx — call useDeleteVideoModalApi().delete
instead of window.confirm + useDeleteVideoMutation.
- en.json — added gallery.deleteVideoPermanent, dropped the now-unused
gallery.deleteVideoConfirmation.
- videos.ts — useDeleteVideoMutation moves into the @knipignore export
group since the only call site now uses
videosApi.endpoints.deleteVideo.initiate via the modal.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(gallery): refetch polymorphic gallery cache on image completion
The gallery grid subscribes to the polymorphic ``getGalleryItemNames``
RTK Query endpoint (so images and videos interleave by created_at). But
``onInvocationComplete``'s image path only did an optimistic insert
into the image-only ``getImageNames`` cache, leaving the polymorphic
cache stale — a freshly-generated image landed correctly in board
totals and the per-DTO cache, but never showed up in the grid until
the user reloaded the page.
Mirror the videos path (which has invalidated these tags since the
polymorphic endpoint was introduced) and dispatch
``galleryApi.util.invalidateTags(['GalleryItemNameList',
'GalleryItemList'])`` after image outputs are processed. The cost is
one extra HTTP round-trip per generation; a future optimization could
optimistically splice the new entry into the polymorphic shape, but
that requires a different ``insertImageIntoNamesResult`` for the
``GetGalleryItemNamesResult`` shape and is a bigger change.
Regression test in onInvocationComplete.test.ts pins the behavior:
verifies the invalidation fires on a fake image complete event, and
verifies it does NOT fire for denylisted passthrough node types
(load_image, image). Confirmed test correctly fails when the fix is
reverted via git stash.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* review: address 2nd-pass code review findings
Self-review pass before re-pinging external reviewers. Five fixes; the
three medium ones have user-visible consequences, the two low ones are
guard + docstring.
1. videos.py: delete_video no longer swallows service errors into a
misleading HTTP 200. Missing DTO -> 404, delete failure -> 500. The
prior shape returned 200 with an empty deleted_videos list, which
the frontend treated as success, dropped from cache, and left the
video on disk — silent data-consistency failure visible only on
next page reload.
2. videos.ts: starVideos / unstarVideos invalidate the LIST_TAG-scoped
{ type: 'VideoList' } entry alongside the per-video and
board-affecting tags. Without this, starred_first=true gallery
queries kept the just-starred video in its old position until the
next list-affecting mutation. Mirrors the delete + upload pattern.
3. wan_denoise.py: _ExpertSwapper.get() stashes _active_device_ctx
right after device_ctx.__enter__() succeeds, before attempting the
LoRA patcher's __enter__. If the LoRA enter raises, _release() can
now actually find the device context and exit it — previously the
ctx was unreachable and 8-9 GB of GGUF expert weights stayed pinned
to GPU until the model cache LRU evicted them.
4. wan_ideal_dimensions.py: reject sources whose longer side is below
the 16-px Wan grid. The downstream max(w, 16) clamp would otherwise
silently disconnect the output from the requested aspect ratio
(returning 16×16 regardless of the source's actual shape).
6. wan_video_denoise.py: docstring now explains the deliberate absence
of denoising_start / denoising_end / initial-latents inputs (video
i2v uses reference-frame conditioning, not noise injection; the
image denoise node still handles still-image img2img).
Tests:
- test_device_context_released_when_lora_enter_raises pins #3.
- test_input_smaller_than_pixel_grid_rejected pins #4.
- test_output_dims_never_zero renamed to
test_smallest_valid_input_still_snaps_to_16_grid (now exercises 16×16
rather than 8×8 since the latter is now correctly rejected).
All 58 affected backend tests pass, frontend lint clean.
Audit note for the PR description (NOT a fix): delete_video's
_assert_video_owner permits write access on public boards (mirroring
the image router's _assert_image_owner — intentional symmetry). The
stricter _assert_video_direct_owner is reserved for board-move ops.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore(frontend): typegen
* fix(gallery): multi-select context menu actions for videos
The video gallery context menu only operated on the single right-clicked
item, so selecting multiple videos and hitting the trash icon deleted just
the first one. Adds a video-side multi-selection menu mirroring the image
one for star/unstar/download/change-board/delete, switched in on
selectionCount > 1.
Each menu now filters the polymorphic selection to its own kind and labels
the action with an explicit count + kind (e.g. "Delete 3 Videos", "Move 2
Images to Board"). The destructive items disable when the kind-filtered
subset is empty, so a video-only selection greys out the image menu and
vice versa.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(multiuser): address Pfannkuchensack PR #9163 review findings
Finding 1 (Medium): delete_board cascade ignored per-video / per-image
ownership, letting a board owner destroy other users' contributions to a
public/shared board just by deleting the board with include_images=true.
Adds user_id filtering through get_all_board_*_names_for_board and
delete_*_on_board (base + sqlite + image wrapper). Non-admin requests pass
the requester's id so the SQL WHERE clause narrows the cascade to that
user's rows; admins still pass None for the unrestricted path. Other users'
content cascades to "uncategorized" via the existing FK on board_videos /
board_images.
Finding 2 (Low, i18n): GalleryItemStarIconButton and
GalleryItemVideoStarIconButton shipped raw English "Star"/"Unstar" tooltips.
Both now use the gallery.starImage / starVideo translation keys.
Finding 3 (Low): delete_videos_from_list and delete_images_from_list
re-raised HTTPException mid-loop, throwing away the response payload for
items already deleted before the foreign name was hit. The frontend cache
never learned about those partial successes, so deleted records reappeared
in the UI until the next manual refresh. Both routes now skip auth-failed
items in-loop and return 200 with the partial-success list.
Residual: adds a test that an upload with an .mp4 extension but
non-decodable bytes (a) reaches probe_video, (b) surfaces 415, (c) unlinks
the streamed-to-disk temp file so the server doesn't leak storage on
garbage uploads.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore(openapi): regenerate openapi.json
The committed schema was stale relative to the current server (missing the
utilities/expand-prompt and utilities/image-to-prompt endpoints, the
ModelRecordOrderBy / SQLiteDirection list params, and the Wan / QwenImage /
QwenVLEncoder config variants this branch adds). Regenerated via the same
command the new openapi-checks workflow uses so the diff CI is empty.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(workflows): add "Image to Video - Add Frames" starter workflow
Extends an existing video by extracting its penultimate frame, running it
through Wan 2.2 I2V A14B + the Lightning LoRA pair to generate a new clip,
and concatenating the result onto the source with a short crossfade.
Cleaned per the default-workflows README: stripped value references on the
four model loader fields and both Lightning LoRA fields so the workflow
ships without keys/hashes for user-installed resources, gave the LoRA nodes
"Apply LoRA (High)" / "(Low)" labels matching the existing Lightning
default, remapped six stale exposedFields entries that pointed to template
LoRA IDs no longer present in the graph, and synced the wan_video_denoise
num_frames default to the value driven by the connected integer node.
Tagged with both Text to Video and Image to Video so it surfaces under
either filter in the Workflow Library.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(gallery): video viewer polish and selection regressions
- Restore auto-select-on-startup and on-board-switch: the polymorphic
getGalleryItemNames endpoint replaced getImageNames as the grid's
source of truth, so appStarted and boardIdSelected now wait on / read
that cache instead of timing out forever.
- Delete-then-select: video delete used to clear selection to null;
image delete read a cache that's no longer warmed. Both now snapshot
the gallery list before deletion and advance to the adjacent surviving
item (prev > next > null) via a shared pickSelectionAfterDelete helper.
- Video Viewer: right-aligned action bar with Open in new tab, Copy
frame, Download, Delete, and a labelled Close video player button that
only appears while playback is active. Copy uses canvas + ClipboardItem
since video MIME types aren't supported cross-browser.
- Next/prev arrows + galleryNav hotkeys now work when a video is in the
Viewer (previously image-only).
- Video context menu uses full-width text MenuItems instead of the
cramped icon group, and gains an Open in new tab entry.
* fix(gallery): bulk video drag-to-board and shift-click range selection
- Bulk video drag: introduced multipleVideoDndSource so a multi-selection
dragged from a video thumbnail moves every selected video, not just the
first. The whitelist in useDndMonitor.ts also needed updating — without
it the monitor's canMonitor gate silently dropped the new source type.
- Mixed selections: both the multi-image and multi-video drag payloads now
carry image_names + video_names side-by-side, so dragging from either
kind in a mixed selection dispatches addImagesToBoard + addVideosToBoard
together. Previously the image side leaked video names into image_names
and the image router 404'd on each one.
- Bulk video helpers: added addVideosToBoard / removeVideosFromBoard that
fan out over the existing singular video router endpoint (no batch
endpoint exists yet) — mirrors the change-board modal's existing loop.
- Shift-click range selection: selectCachedGalleryItemNames now looks up
the cache entry matching the gallery's current query args instead of
taking the first entry from selectInvalidatedBy. RTK Query keeps unused
entries warm for 60s after a board switch, and the old "first wins"
behavior frequently landed on a stale board's name list, making
shift-click silently no-op until a delete/move forced a refetch.
* fix(scripts): force generate_openapi_schema.py to resolve invokeai from the repo root
When the script was invoked as ``python scripts/generate_openapi_schema.py``,
Python placed the script's directory at ``sys.path[0]`` rather than the repo
root. ``import invokeai`` then resolved via the venv's site-packages, which on
multi-worktree editable installs ends up importing ``invokeai`` as a PEP 420
namespace package that aggregates every worktree's ``invokeai/`` directory.
Side-effect imports driven by submodule discovery silently miss whichever
worktree isn't first on the namespace path, so the registry came up short by
the invocations declared only in this worktree (the wan/video set, 15
classes). Running the same imports via ``python -c`` worked because
``sys.path[0]`` defaulted to the cwd and ``invokeai/__init__.py`` resolved
cleanly to the worktree.
Prepend the resolved repo root to ``sys.path`` before importing ``invokeai.*``
so the script always picks up the local sources regardless of how it was
launched.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(video): add Frame Range from Video invocation with scrubbable preview
New ``extract_video_range`` node trims a source video to a contiguous frame
range and re-encodes it as MP4, slotting in naturally between a generated
clip and Concatenate Videos for I2V chain shaping. Bounds are inclusive and
support negative indices (``end_frame=-1`` keeps the final frame), matching
Frame from Video. Output fps inherits from the input unless overridden.
The node renders a per-type preview inside the workflow editor: two
``<video>`` tiles side by side, each driven by a CompositeSlider that scrubs
the corresponding integer field. The tile uses ``currentTime = frame / fps``
so browsers display the seeked frame natively without a canvas roundtrip.
Negative-index entries in the standard integer input are resolved against
the source frame count for display only; the underlying field value is
preserved verbatim.
The custom UI is wired in via a ``CustomNodeBody`` dispatcher in
``InvocationNode.tsx`` rather than a registry — small enough to be explicit.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(video): emit resolved frame indices and move preview to per-field renderer
Three changes to the ``extract_video_range`` invocation:
1. New ``ExtractVideoRangeOutput`` mirrors ``VideoOutput`` and additionally
emits the resolved (positive, 0-based) ``start_frame`` and ``end_frame``
indices. Chained workflows can feed those back into a downstream
Frame from Video to extract the same boundary frame the trim landed on.
2. ``fps`` is now a plain ``int`` defaulting to 16 (was ``Optional[int]``
with an "inherit from input" fallback). Matches the default used by
wan_l2v and the other Wan video producers, so chained workflows agree
on framerate without each node guessing.
3. The frame preview is now a per-field widget driven by a new
``UIComponent.VideoFrameIndex`` hint. ``start_frame`` and ``end_frame``
are tagged with it; the new ``VideoFrameIndexFieldInput`` renders a
number input plus a live <video> thumbnail and a scrubber slider, all
writing to the same Redux field. Negative indices entered in the
number input are still resolved against the source frame count for
display only — the backend re-resolves at invoke time.
The widget reads its companion ``VideoField`` (by convention, the sibling
field named ``video`` on the same node) via direct Redux selectors, so it
works wherever ``InputFieldRenderer`` is used — the workflow editor's node
body AND the Form Builder's view/edit modes. The previous node-body
``ExtractVideoRangePreview`` and its ``CustomNodeBody`` dispatcher in
``InvocationNode.tsx`` are removed; the per-field widget supersedes both.
In the workflow editor, side-by-side framing is lost in exchange for
Form Builder support; users wanting the side-by-side layout in a form can
group the two frame fields in a row container.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore: ruff
* fix(video): address PR #9163 review follow-ups
- delete_board: include_images query description and OpenAPI schema now mention
videos alongside images
- get_video_thumbnail: check path existence before returning FileResponse so a
missing thumbnail produces the documented 404 instead of an after-route error
- delete_videos_on_board: stop deleting records for videos whose files failed to
delete, so a transient FS error no longer orphans the file with no record
pointing at it
- DeleteBoardModal: destructive button and warning copy now mention videos
* fix(video): address PR #9163 May-22 review and failing CI
- remove_video_from_board now accepts either the direct video owner or a
board write-access holder, so videos uploaded to a board that later
flipped Public -> Shared/Private aren't stranded.
- VideoService.create rolls back the DB record and board association if
the underlying file save fails, preventing ghost records whose file
endpoints 404.
- delete_videos_on_board returns the actually-deleted names; delete_board
uses that list so the response can't claim a video was destroyed when
its record was preserved due to a file-delete failure.
- Local test_videos_multiuser fixture now patches
invokeai.app.api.routers._access so list/names route 403 checks work.
- Regenerate schema.ts to pick up the CacheStats description.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* chore(frontend): rebuild openapi
* fix(video): register Viewer <video> as drag source
Drag-and-drop from the Viewer pane now produces the same singleVideoDndSource
(and multipleVideoDndSource for active multi-selection) as the gallery
thumbnail, so a video can be dropped onto a Video Primitive's "Starting Video"
field directly from the Viewer.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat(video): add frame preview to Frame from Video node
Tag frame_index with ui_component=VideoFrameIndex so the node renders the
same live frame thumbnail + scrubber as Frame Range from Video. The widget
keys off the sibling 'video' field, which this node already has, so no
frontend changes are needed. Bump node version 1.0.0 -> 1.1.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* feat(video): derive Frame Range fps from source video by default
Make the fps field optional (default None). When unset, the output frame
rate is inherited from the probed source video so a trimmed clip plays back
at the same speed as its source, falling back to 16 fps when the source rate
can't be probed. An explicit fps still overrides. Bump node version
1.0.0 -> 1.1.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* fix(video): use fps=0 sentinel for source-derived Frame Range rate
The previous Optional[int]/None design had no natural way to express
'unset' in the node's number input, and the ge=1 constraint rejected the
intuitive fps=0 with a validation error. Make fps a plain int defaulting to
0, allow ge=0, and treat 0 as 'match the source video's frame rate'. Keeps
in-progress workflows (already saved with fps=0) working without a version
bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* feat(wan): add Wan 2.2 TI2V Ideal Dimensions node
TI2V-5B uses the 16x Wan 2.2-VAE plus a 2x transformer patch, so pixel dims
must be multiples of 32 (the existing I2V node snaps to 16, which the TI2V-5B
patchify step rejects). Add a wan_ti2v_ideal_dimensions node that snaps to 32.
Factor the shared scale-and-snap math into _scale_and_snap(multiple=...) so
both nodes derive from one implementation; the I2V node is unchanged
behaviorally (its existing tests still pass). Add a mirrored TI2V test suite.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs(wan): add A14B/5B model hints to ideal-dimensions node titles
Suffix the node titles with the target model family (A14B / 5B) so they're
distinguishable in the add-node search and node header, and rewrite both
docstrings to lead with which Wan 2.2 model they're for and cross-reference
the other node. Purely UI metadata — no behavior or schema change.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* feat(workflows): replace bundled Wan 2.2 video workflows with curated set
Remove the 6 previously-bundled Wan 2.2 *video* default workflows (Text to
Video, Text to Video Lightning x2, Image to Video, Image to Video Lightning,
Image to Video - Add Frames) and replace them with the 8 curated starter
workflows: Text/Image to Video Lightning (+ Concept LoRA variants), Extend
Video Lightning (+ Concept LoRA variant), and the TI2V-5B text/image-to-video
low-quality variants.
Each is assigned a stable default_ id and meta.category=default. Model fields
are intentionally blanked (per-install keys don't resolve cross-instance) with
the required models listed in each workflow's Notes. The two Wan 2.2 *image*
workflows (Image to Image, Text to Image) are retained.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs(video): add beginner Video Workflows guide for the 8 starter workflows
New Features-section page (sibling to Video Generation) describing the eight
bundled Wan 2.2 video workflows in plain language: how to choose between the
Text/Image/Extend families and their Lightning / Concept-LoRA / TI2V-5B
variants, how to select models from each workflow's Notes, how to run one, and
a quick per-GPU guide. Cross-linked both ways with the Video Generation
technical reference.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs(video): fix Concept LoRA slot guidance (slots are required)
The w/ Concept LoRAs workflows wire required lora fields (lora_selector /
wan_lora_loader, no default) into the graph, so an empty slot blocks
invocation. Correct the earlier claim that empty concept slots behave like the
base workflow: every LoRA slot must be filled, and users without concept LoRAs
should use the plain variant.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs(video): drop '(experimental)' from Video Generation title
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* chore: refresh uv.lock
Routine lock refresh (transitive dev deps: docutils, idna, platformdirs,
python_discovery, tornado).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* feat(wan): add first-last-frame interpolation (FLF2V) to I2V-A14B
The Reference Image - Wan 2.2 node gains an optional End Image input: when set,
encode_reference_image_to_video_condition places the end image in the final
temporal slot and anchors the mask at both the first and last latent frames, so
I2V-A14B interpolates a clip from the start image to the end image. Mirrors
diffusers WanImageToVideoPipeline.prepare_latents with last_image set.
The denoise loop is unchanged - for A14B it just concatenates the 20-channel
condition, which is agnostic to one vs two anchors. FLF2V is A14B video only
(num_frames > 1); the encoder raises a clear error for TI2V-5B or single-frame.
Bump wan_ref_image_encoder to 1.2.0; add mask-anchoring unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* feat(workflows): add 'Interpolate 2 Images to Video' starter + FLF2V docs
Ship a default workflow that wires the new FLF2V End Image input end to end
(I2V-A14B + Lightning, two image inputs interpolated). Model fields blanked with
the required models listed in Notes, default_ id + category=default. Document
FLF2V in the Video Generation reference and add the workflow to the Video
Workflows guide.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* feat(workflows): add Text/Image/Video to Video library filters + fix video tags
Add 'Text to Video', 'Image to Video', and 'Video to Video' to the Common Tasks
filter list in the Workflow Library browser. Fix the tags on the nine bundled
Wan 2.2 video workflows, which were all copy-pasted as 'text to video':
- Text to Video: the three T2V workflows
- Image to Video: the I2V workflows + Interpolate (two-image)
- Video to Video: the two Extend Video workflows
The TI2V-5B variants also drop the spurious lightning/lora tags (they have no LoRAs).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* feat(workflows): add 'Extend Video to Image' FLF2V starter + docs
Ship a default workflow that extends a video toward a user-provided target
image: the new segment interpolates (FLF2V) from the source video's last frame
to the destination image, then concatenates onto the original with a cross-fade.
Model fields blanked, default_ id + category=default, tagged 'video to video'.
Document it (card + usage instructions) in the Video Workflows guide.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs(video): reorganize Video Workflows guide sections
Group the Interpolate section and the Concept-LoRA / TI2V-5B asides with the
image workflows, keep the Extend family (including Extend Video to Image) at the
end, and retitle the section to 'Bundled video workflows'.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* chore(frontend): regenerate openapi and typegen
* chore(backend): ruff
* fix(future): make the WAN LoRA loader compatible with LoRA picker node PR #9259
* chore(frontend): remove unused selectT5EncoderModels import
* chore(frontend): remove unused export
* fix(gallery): count videos and pick video covers for board tiles
Gallery boards previously joined only the `images` table for their
headline count and cover thumbnail, so a board containing nothing but
videos rendered as empty with no preview. BoardDTO now exposes
`video_count` and an optional `cover_video_name`; the boards service
picks the best cover across both tables using the same
(starred DESC, created_at DESC) tie-break the image path already used,
and the gallery list renders `image_count + video_count` everywhere it
previously rendered just images (real boards, no-board pseudo-board,
and the tooltip). Adds `getBoardVideosTotal` to round out the no-board
counts (the BoardVideosTotal tag was already wired into invalidation).
* test(boards): wire video record storage into multiuser test fixtures
After the board cover/count fix started reading from `video_records` and
`board_video_records`, the multiuser test fixtures that still set both
to `None` started erroring out — the boards router's catch-all turned the
AttributeError into a 404, cascading through every test that PATCHes or
GETs a board (auth, workflows, data-isolation suites). Swap the `None`
placeholders for real SqliteVideoRecordStorage / SqliteBoardVideoRecordStorage
instances (paralleling the existing image storage setup), and pin sane
defaults on the MagicMocks in `test_videos_multiuser.py` so the
get_dto cover/count lookups don't trip Pydantic validation.
* fix(ui): widen useVideoContextMenu ref type to allow null
The ref param was typed RefObject<HTMLElement>, but useRef produces
RefObject<HTMLElement | null>, breaking lint:tsc in GalleryVideoItem.
Match the sibling useImageContextMenu signature.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* chore(backend): ruff
* fix(tests): pass video/gallery services to InvocationServices in workflow-call router tests
The workflow-call router tests from main construct InvocationServices
directly and predate the video/gallery services added on this branch, so
every test in the file errored with missing positional arguments. Mirror
tests/conftest.py: real sqlite stores for video_records and
board_video_records, None for the services the tests never touch.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): refresh board caches when a generated video completes
Video completion previously invalidated only the polymorphic gallery
list tags, so the new video appeared in the grid while the board's
video_count, cover thumbnail (Board tag / listAllBoards), and
BoardVideosTotal stayed stale until an unrelated mutation refetched
them. Use the shared getTagsToInvalidateForBoardAffectingMutation
helper over the affected boards, matching the video mutation endpoints.
Reported by @JPPhoto in PR #9163 review.
Co-Authored-By: Claude Fable 5 <[email protected]>
* feat: include videos in date-based virtual boards
Date virtual boards were image-only even though the gallery grid is now
polymorphic: video-only dates never appeared, and mixed dates omitted videos
from counts/contents/covers.
- SqliteGalleryService owns virtual-board dates now: get_dates() unions
images+videos per date (video_count added to VirtualSubBoardDTO, cover is
the newest item of either kind via cover_image_name/cover_video_name), and
list_item_names() gained a created_date filter.
- New GET /api/v1/virtual_boards/by_date/{date}/item_names returns the same
polymorphic (kind, name) refs as the gallery names endpoint; the legacy
image_names route is kept for API compatibility.
- Frontend virtual-board selection consumes the new endpoint, so videos show
up in virtual date boards; VirtualBoardItem shows video counts (localized
tooltip) and falls back to the video thumbnail for video covers.
- tests/conftest.py wires a real SqliteGalleryService so router tests exercise
the filter SQL; service + router tests cover video-only dates, mixed dates,
cover selection, and per-user isolation.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): don't advance gallery selection for videos whose delete failed
handleDeletions treated every requested video as deleted when picking the
post-delete selection, so a 403/500 on deleteVideo could jump the Viewer away
from a video that still exists, and a surviving neighbour was skipped as a
replacement candidate. Only successfully deleted names now count: a failed
displayed video keeps its selection, and failed neighbours remain valid
replacements. Covered by state.test.ts with rejected deleteVideo dispatches.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): count uncategorized videos when deciding the gallery has content
useHasImages only looked at boards and the uncategorized *image* total, so a
gallery whose only content was an uncategorized video rendered the
new-user/get-started view instead of the normal no-selection state. The hook
now also reads the uncategorized video total (getBoardVideosTotal('none'));
the decision logic is extracted as getHasGalleryContent and unit-tested.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): stop labeling board video counts as images in tooltips
Board tooltips folded video_count into image_count and rendered
boards.imagesWithCount, so a video-only board read e.g. '1 image, 0 assets'.
Tooltips now show split image/video/asset counts using the new
boards.videosWithCount translation; the compact unlabeled headline count in
the boards list stays combined so video-only boards don't read as empty.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): restrict client-side video upload acceptance to MP4 only
The dropzone accept map advertised .webm/.mov and isVideoFile treated
.webm/.mov/.mkv as videos, but the upload router accepts MP4 only, so
those files were accepted client-side and then rejected with 415 after
the bytes were uploaded. Consolidate the accepted-media lists into
common/util/uploadMediaAccept.ts (single source of truth shared by
useImageUploadButton and FullscreenDropzone) and pin them to the
backend contract with a regression test.
Also split the accept map: image-only upload fields (board covers,
style presets, model images, workflow thumbnails) no longer advertise
video/mp4, which they had inherited when video entries were added to
the shared map.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(api): bulk video star/unstar returns partial successes instead of 403 mid-batch
star_videos_in_list and unstar_videos_in_list re-raised the ownership
HTTPException mid-loop, so a batch containing one foreign (or stale)
name mutated the earlier owned videos and then returned 403 with no
payload — the client never invalidated caches for the videos that did
change. Skip unauthorized names and return 200 with the actually
starred/unstarred videos, mirroring delete_videos_from_list. Router
tests cover the mixed-ownership batch for both routes.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): localize virtual board section header and toggle
The 'By Date' header and the Collapse/Expand aria-label in
VirtualBoardSection were hardcoded English. Add boards.byDate and
common.collapse/common.expand translation keys and use them.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(app): failed video saves no longer orphan files on disk
DiskVideoFileStorage.save() moves the source MP4 into permanent storage before
writing the thumbnail and sidecar, so a failure in either later step used to
leave the moved MP4 (and partial artifacts) on disk with no DB record through
which they could be managed. save() now removes its destination files before
raising, and VideoService.create()'s rollback also deletes files to cover
failures after a successful file save (e.g. building the DTO).
Also documents why board attachment during create is best-effort (mirroring
ImageService.create: a board deleted mid-generation must not destroy the
render) and pins the explicit fallback — DTO reports the actual missing board
association and a warning is logged — with a service test.
Co-Authored-By: Claude Fable 5 <[email protected]>
* docs(app): document videos.user_id lifecycle and pin user-deletion behavior
videos.user_id deliberately has no FK to users, matching images/boards/
workflows (migration_27 adds those user_id columns index-only): deleting a
user leaves their media in place for admin review/cleanup rather than
cascading a row delete that would strand files on disk. A migration comment
now states the parallel, and a migration-backed test creates a user and a
video, deletes the user, and asserts the record survives, stays attributed to
the deleted owner, and remains visible only to admins.
Co-Authored-By: Claude Fable 5 <[email protected]>
* feat(app): support VideoField values in session queue batch data
Batch.data previously allowed ImageField but not VideoField, so submitting
multiple VideoField values through the generic batching capability failed
Pydantic validation before enqueueing. VideoField now joins the
BatchScalarDataType union; a test asserts a VideoField batch validates and
expands into separate sessions. schema.ts/openapi.json regenerated.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): video uploads are opt-in per consumer; upload validation fixes
- useImageUploadButton gains an allowVideos opt-in (default off). Only the
gallery uploader accepts videos; image-only consumers (ref images, board
covers, launchpad buttons, image-to-prompt, etc.) no longer let a selected
MP4 upload into the gallery while the requested image action goes nowhere.
Videos are excluded from their accept map and rejected at runtime if the
file dialog bypasses it, with tests via partitionUploadFiles.
- The hook's loading state now covers both the image and video mutations, so
an in-flight MP4 upload shows a loading button and blocks resubmission.
- The fullscreen drag-drop/paste validator accepts a file when either its
MIME type or its extension is recognized — a clip.mp4 with an empty
File.type used to be rejected even though the backend accepts it. The
validator moved to a pure module with tests.
- Failed video uploads no longer toast "Image Upload Failed": video-only
batches use a new toast.videoUploadFailed key, mixed batches the neutral
toast.uploadFailed.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(app): bound untrusted video decoding with a killable subprocess timeout
probe_video / extract_video_frame / decoder_frame_count now run in a
short-lived child process (video_decode_worker.py) killed after a hard
timeout. Previously a crafted MP4 that failed the imageio probe and then
hung inside cv2.VideoCapture()/read() would pin the FastAPI request
worker that called it forever; repeated uploads could exhaust the pool.
The worker is run by file path (not -m) and imports only imageio/PIL/cv2
so it starts without pulling in the invokeai package or torch. Tests
substitute a never-returning worker command and assert the helpers fail
within a bounded interval, plus happy-path tests against a real
synthetic MP4 to validate the subprocess plumbing end to end.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(app): stream frames through the video concat/trim nodes
video_concat and extract_video_range fully decoded their inputs into
lists of uncompressed frames before encoding; with the 1 GB upload cap a
long 1080p source can expand to tens of gigabytes of RAM, so any user
able to enqueue these nodes could exhaust server memory.
Frames now stream from the decoder straight into an incremental FFMPEG
writer. The concat node buffers only the transition windows (bounded by
transition_frames), and the range node holds one frame at a time and
stops decoding at the end of the requested range. Tests use lazy frame
iterators to pin that encoding begins before the inputs are exhausted
and that look-ahead stays bounded.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): video upload feedback parity with images
- Add uploadVideo matchFulfilled/matchRejected listeners mirroring the
image upload listeners: success toasts name the destination board and
navigate the gallery on the first upload of a batch; failure toasts
name the failed file, which is what makes partially failed
Promise.allSettled batches attributable (uploadVideos and the
fullscreen dropzone aggregate without rethrowing, same as images).
- GalleryUploadButton now uses the hook's combined isUploading, so an
in-flight MP4 shows a spinner and blocks resubmission.
- Media-neutral labels on the two video-enabled surfaces: gallery
uploader aria/tooltip says Upload Media, the fullscreen overlay says
uploaded items (not images) will be added, and its invalid-file toast
mentions MP4.
Co-Authored-By: Claude Fable 5 <[email protected]>
* docs: include videos in the user-deletion data-loss note
The admin guide's user-deletion warning enumerated boards, images,
workflows, queue items, and style presets but not videos. State that
video records survive with the deleted user_id, that files remain under
outputs/videos, and that administrators keep gallery visibility of the
orphaned media for review/cleanup — matching the behavior pinned by the
video_records user-deletion lifecycle test.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix video processing resource bounds
* fix video lifecycle edge cases
* stabilize decoder inactivity test
* chore(deps): declare psutil as a direct dependency
video_thumbnails.py now imports psutil for decode-worker process-tree
termination, but it was only present transitively (via accelerate and
friends). Declare it so the import can't silently break when an
upstream package drops it.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix video security and lifecycle regressions
* chore: regenerate OpenAPI schema
* fix: preserve exact frame dimensions in video encoders
imageio's FFMPEG writer defaults to macro_block_size=16, which makes ffmpeg
silently rescale frames to the next multiple of 16 — a 1920x1080 upload
trimmed by Frame Range from Video came back as 1920x1088 while the DTO
recorded 1080, so concatenating the trim with its own source failed the
same-dimensions check.
- New invokeai/app/util/video_encoding.make_mp4_writer single-sources the
encoder settings (libx264, macro_block_size=1) for wan_latents_to_video,
video_concat, and video_frame_extract_range.
- yuv420p requires even dimensions, so concat and extract-range now reject
odd-dimension sources up front with a clear error instead of an opaque
ffmpeg failure mid-encode.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix: correct A14B fallback scheduler and stop LoRA leakage to low-noise expert
Two silent-wrong-output bugs on the GGUF A14B path:
- The no-scheduler-dir fallback returned FlowMatchEulerDiscreteScheduler for
A14B, but both Wan-AI/Wan2.2-{T2V,I2V}-A14B-Diffusers repos ship
UniPCMultistepScheduler with flow_shift=3.0 / flow_prediction /
use_flow_sigmas (verified against the upstream scheduler_config.json).
Every A14B GGUF render ran an unshifted first-order Euler schedule,
degrading output and skewing how many steps land above the MoE boundary.
An unreadable on-disk config now also falls back to the variant default
instead of bare FlowMatchEuler.
- low_loras fell back to the primary list when loras_low_noise was empty,
but the Wan LoRA loader deliberately routes expert-tagged LoRAs to exactly
one list — so a high-noise-only LoRA (e.g. a Lightning high-noise distill)
was silently applied to the low-noise expert too, and high-only targeting
was impossible. An empty low list now means no LoRAs on the low expert.
Also (here and in the previous commit): the Wan VAE decode nodes now raise a
clear latent-channel mismatch error (16-channel A14B latents vs 48-channel
TI2V-5B VAE and vice versa) instead of an opaque tensor-size RuntimeError
when the wrong VAE is selected.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix: keep viewer selection on surviving item after image deletion
The image-side handleDeletions cleared the gallery selection
(imageSelected(null)) whenever the deletion intersected the multi-selection
but the displayed item was not among the deleted names — e.g. a video
displayed while only images were deleted from a mixed selection, or a
hover-delete of a non-displayed selected image. It also treated every
requested name as deleted, ignoring the server's deleted_images response,
so a partial failure could jump the selection away from an image that still
exists.
Port the deleteVideoModal logic: only server-confirmed deletions count, a
surviving displayed item stays selected, and the usage-reset sweep
(nodes/canvas/ref-image layers) runs only for actually-deleted images.
Regression tests mirror deleteVideoModal/store/state.test.ts.
Co-Authored-By: Claude Fable 5 <[email protected]>
* perf: ~48x faster Wan VAE decode on ROCm via conv2d decomposition
MIOpen has no implicit-GEMM 3D-convolution kernels for the Wan VAE's shapes
on RDNA3 and falls back to Im3d2Col (61% of decode GPU time in a torch
profile). An 81-frame 832x480 decode took 730s on a W7900 vs 13s on an RTX
5060; dtype changes and cudnn.benchmark kernel search were all within +/-7%.
A stride-1 kTxkHxkW conv3d is exactly the sum of kT conv2d taps over shifted
temporal slices, and MIOpen's conv2d kernels are well optimized. This rebinds
WanCausalConv3d.forward (class-level, idempotent, ROCm builds only) to that
decomposition:
- same 3-latent-frame decode: 81.6s -> 1.71s (~48x), extrapolating to ~12s
for the 81-frame workload — matching NVIDIA wall-clock
- numerically equivalent up to summation order: ~1e-6 max error vs F.conv3d
in fp32; full bf16 decode differs by <=3/255 in pixel space (0.1% of pixels
by more than 1/255)
- strided encoder downsample convs keep the stock F.conv3d path (temporal
taps couple under stride)
- applied from every AutoencoderKLWan load site (Wan checkpoint/diffusers
VAE loaders, Wan main-model VAE submodel, Anima VAE), so decode, encode,
and ref-image conditioning all benefit; CUDA builds are untouched
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix: self-heal the media cookie for sessions that predate or outlive it
Video playback authenticates via an HttpOnly cookie (media elements can't
send Authorization headers) that was only issued at login. A session restored
from localStorage can hold a valid JWT without the cookie — the session may
predate the cookie's introduction, or the cookie may have been cleared while
the JWT survived. Every API call works, but each <video> request 401s and the
player silently renders black with 0:00 duration (hit during PR #9163
functional testing).
- New POST /api/v1/auth/media-cookie re-issues the cookie from a valid Bearer
token: same live-user check as get_current_user, cookie lifetime clamped to
the token's remaining validity, successful no-op in single-user mode.
Cookie attributes are shared with login via _set_media_cookie so they can't
drift.
- Frontend calls it once per app load when an authenticated session exists
(useMediaCookieRefresh in GlobalHookIsolator); failures are left to the
existing global 401 handling.
Co-Authored-By: Claude Fable 5 <[email protected]>
* feat: use Apply LoRA Collection node in Wan concept-LoRA workflow templates
Replace the per-slot concept-LoRA plumbing in the three 'w/ Concept LoRAs'
video templates (Text to Video, Image to Video, Extend Video) with the single
wan_lora_collection_loader node: users now add any number of concept LoRAs
through one multi-LoRA form field instead of two fixed slots (T2V/Extend) or
the lora_selector + collect chain (I2V).
Chain in all three: model loader -> Lightning high-noise LoRA -> Lightning
low-noise LoRA -> Apply LoRA Collection (concept LoRAs, ships empty) ->
denoise. Also prunes exposedFields entries that referenced nodes deleted in
an earlier revision of these templates (pre-existing; the frontend ignored
them, but they were dead weight).
Validated: backend WorkflowValidator + default-sync asserts, node versions
current, every edge/form/exposedFields reference resolves, frontend
parseAndMigrateWorkflow accepts all three, no machine-specific model
identifiers ship.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix: enforce multiuser image authorization
* fix: address remaining Wan video review findings
* test: give closed-stream decoder test headroom for slow Windows spawn
The 0.2s decode timeout raced against Python subprocess startup on the
Windows CI runner: the inactivity deadline fired before the worker could
close its stdout, raising the generic decode timeout instead of the
expected 'decoder worker' one. A generous timeout makes the EOF path
deterministic; proc.wait still bounds the test at ~1s.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix: address adversarial Wan video review findings
* fix: resolve remaining Wan video review issues
* feat(ui): rename gallery/board strings from Images to Images/Videos
The gallery grid, selections, board operations, and related settings now
operate on both images and videos, so the user-facing strings that
describe them say so. Image-only surfaces (compare, reference images,
progress previews, image storage maintenance, upload-format errors) are
unchanged, as are unused legacy keys.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(api): harden video/media API per PR review
- Scope the media cookie (set + delete) and the sliding-token middleware's
auth-route exclusions to the reverse-proxy root_path, so media auth works
behind sub-path proxies and a proxied logout can't mint a replacement token.
- Video upload: run filesystem writes, MP4 validation, ffmpeg probing, and
create() in the thread pool; add VideoUploadLimitASGIMiddleware to bound
request size before multipart spooling and cap concurrent uploads.
- Add GET /videos/i/{name}/workflow (mirrors the image route) so persisted
video workflows/graphs are retrievable, with read-access checks.
- Add DELETE /videos/uncategorized so the "Delete All Uncategorized
Images/Videos" action can cover both media kinds.
- Make polymorphic gallery ordering deterministic on created_at ties with
kind+name tie-breakers, and pick virtual-board covers via ROW_NUMBER
instead of a bare-column MAX() aggregate.
- Add cpu_only to WanT5Encoder_WanT5Encoder_Config (parity with the other
standalone text-encoder configs; the loader already honors the field).
- Regenerate schema.ts/openapi.json.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(backend): Wan VAE effective device + generalized working-memory estimation
- Move VAE inputs to get_effective_device(vae) instead of the globally
selected device — a cpu_only Wan VAE previously crashed every Wan VAE
invocation on GPU hosts.
- Add estimate_vae_working_memory_wan (per-frame conv working set + resident
RGB clip, config-driven spatial scale for TI2V's 16x compression) and
reserve working memory in all four Wan VAE paths, replacing the Flux
estimator / missing reservations.
- Fall back to spatial tiling for video decodes whose full-frame working set
exceeds the execution device's VRAM, and move the decoded clip to the CPU
before MP4 encoding so VRAM isn't held for the encode's duration.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): video gallery/deletion/workflow fixes per PR review
- Video deletion: use the batch endpoint (one request per invocation), clear
workflow-node VideoField inputs only for server-confirmed deletions, and
invalidate per-video DTO/metadata/workflow caches on delete (including
board-cascade deletions in the board mutations).
- VideoFieldInputComponent resets its value only on a confirmed 404, not on
transient auth/server/network errors.
- Global Delete hotkey partitions the polymorphic selection and routes videos
through the video delete flow.
- "Delete All Uncategorized Images/Videos" now deletes both media kinds;
"Download Board" relabeled "Download Board Images" (image-only endpoint).
- Translation splits: image-only multi-select actions revert to "Images";
polymorphic gallery search + star hotkey become media-neutral; the multi-
drag preview counts the whole mixed selection.
- Expose video metadata + workflow in the viewer: new video details overlay
(metadata/workflow/graph tabs), a Load Workflow toolbar action for videos,
and a 'video' source for the load-workflow dialog.
- Model Manager: wan_t5_encoder gets the encoder settings panel (Run on CPU).
Co-Authored-By: Claude Fable 5 <[email protected]>
* docs: make gallery docs video-aware; fix video workflow count
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix: address remaining video review findings
* fix(backend): Wan inference fixes from full-PR self-review
- Force bfloat16 in the standalone Wan VAE checkpoint loader: `precision: auto`
resolves to fp16 on CUDA, and fp16 is unstable on the Wan VAE (the diffusers
folder path already forced bf16). Both starter VAEs route through this loader.
- Count the decoded RGB clip twice in the Wan working-memory estimator:
diffusers' frame accumulation transiently holds ~2x the clip at peak, which
the tiled-decode fallback previously undercounted by up to ~2 GB.
- Ignore a wired 'Transformer (Low Noise)' for TI2V-5B (warn instead of raising
a misleading A14B error), matching the field's documented behavior.
- Release the expert swapper's device context even when LoRA weight-restore
raises, so a failed unwind can't pin an 8-9 GB expert in VRAM.
- Validate LoRA variant (A14B vs 5B) against the wired transformer in both Wan
LoRA loaders — a mismatch previously crashed mid-denoise with an opaque
layer-patcher shape error.
- Fix the WanDiffusersModel exception ladder: the old-diffusers torch_dtype
retry now also gets the missing-variant OSError fallback, with the matching
dtype kwarg.
- Mark both Wan ideal-dimensions nodes Prototype like every other Wan node;
correct the text-encoder docstring (seq_len 512, not 226).
- Add CPU tests for the multi-frame WanVideoDenoise loop (T_lat>1 shapes,
zero-velocity invariant, A14B I2V 36-channel concat across frames, TI2V-5B
expand-timesteps mask blend incl. per-token timesteps and frame-0 restore).
Node version bumps: wan_model_loader, wan_lora_loader,
wan_lora_collection_loader -> 1.0.1.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(api): video service fixes from full-PR self-review
- Purge cached invocation outputs on video deletion: the memory invocation
cache registered images/tensors/conditioning on_deleted hooks but not videos,
so re-running an identical graph after deleting its output "succeeded" with a
cached VideoOutput naming a 404 video.
- Add the single-user early-return to VideosInterface's read-access and
board-save checks, matching ImagesInterface — after a multiuser->single-user
switch, video workflows no longer fail with PermissionError where identical
image operations succeed.
- Restructure staged-delete recovery to match the image side: video_records
.get() raises rather than returning None, so the explicit commit branch was
unreachable and recovery semantics lived in the exception handler by luck.
- Return 416 (not 206 with "bytes 0--1/0") for any Range request against a
zero-length video file; add tests for the whole Range-parser matrix.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): video UX fixes from full-PR self-review
- Add 'video' to the invocation-complete passthrough denylist: a Video
Primitive completing mid-run invalidated gallery caches and auto-switched the
user's selection/board to the node's *input* video.
- Show the multi-selection context menu only when the clicked item is part of
the selection (both image and video menus): right-clicking a video with 2+
images selected previously produced a menu with every action disabled.
- Clear workflow VideoField references when videos are cascade-deleted via
board deletion or delete-uncategorized, matching the direct-delete flow.
- Toast on total video-delete-batch failure (the untracked mutation was
otherwise silent) and on failed logout (the button previously did nothing
when the server was unreachable).
- Check resp.ok in useDownloadItem so an expired media cookie can't save error
bodies as .mp4/.png files.
- Provide the LIST_TAG-scoped VideoList tag from listVideos so the star/board
invalidations that reference it actually match; fix the misleading comment;
dedupe the doubled BoardVideosTotal tag type.
- Validate VideoField access on workflow load (checkVideoAccess), resetting
stale refs with a warning like image fields.
- Wire middle-click-open-in-new-tab for gallery videos (the setting label
already promised it).
- Show the effective fallback (primary CFG) in the low-noise guidance slider
when unset, instead of a constant the run never uses.
- "Moving 1 image/video to board:" singular form for mixed-media moves.
- Regenerate schema.ts/openapi.json (node version bumps, classification,
docstring fixes).
Co-Authored-By: Claude Fable 5 <[email protected]>
* chore: workflows/docs/deps fixes from full-PR self-review
- Update all 12 bundled Wan workflows to current node versions (wan_model_loader
/ wan_lora_loader / wan_lora_collection_loader 1.0.1, wan_ref_image_encoder
1.2.0, backfilling the optional end_image/num_frames inputs) so fresh installs
don't open with "node needs update" badges; add a registry-consistency test
over the bundled Wan/video workflows so stale embeds can't recur.
- Docs: the A14B auto scheduler is UniPC (not FlowMatchEuler); note that the
bundled TI2V-5B workflows ship 20 steps as a speed compromise vs the 40-50
quality recommendation.
- Pin imageio[ffmpeg]>=2.37 and psutil>=6 (imageio encode behavior is
version-sensitive enough that we carry a regression test for it); relock.
- De-flake the thumbnail worker descendant-kill test (0.5s was the only tight
ceiling in the file; a loaded runner could kill the worker before the child
pid file existed).
Co-Authored-By: Claude Fable 5 <[email protected]>
* ci: raise python-tests job timeout to 30 minutes
Main already runs 9-11 min per platform and this PR pushed py3.11 windows-cpu
past the 15-minute cap (cancelled mid-pytest at 15m10s on the last run).
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix: resolve CI failures from main's route-auth audit and knip
The route-authorization audit merged from main (#9367) only recognized the
two Bearer-token dependencies, so it flagged the video media routes (which
authenticate via get_current_media_user_or_default) and the media-cookie
endpoint (which validated its Bearer token inline). Teach the audit about
the media dependency, drop the image media routes from PUBLIC_ROUTES (they
now carry cookie auth on this branch), and give refresh_media_cookie a
CurrentUserOrDefault dependency in place of its duplicated inline
validation. The media-cookie tests now patch auth_dependencies'
ApiDependencies like every other auth-dependent route test.
knip: getDeletedVideosFromDeleteBoardAction was exported but only used
in-module; cover it in the listener unit tests like its image twin.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(backend): Wan invocation fixes from JPPhoto's 2026-07-21 review
- Both Wan LoRA loaders now validate the *resolved* config (type=LoRA,
base=Wan) instead of trusting the client-supplied identifier fields; a
mislabeled Flux/SDXL/main key is rejected up front instead of reaching
the layer patcher.
- The collection loader rejects LoRAs already applied upstream on either
expert list (same invariant as the single loader) instead of silently
doubling their effective weight.
- A LoRA routed only to the low-noise list of a TI2V-5B main now logs a
warning — the single-transformer path never consumes that list, so the
routing was a silent no-op.
- _ExpertSwapper._release clears its slots in a nested finally, so a
device-context exit failure can no longer leave stale contexts that a
later close() would double-exit.
- WanLatentsToImage rejects multi-frame (T>1) video latents with a clear
error pointing at wan_l2v, before the VAE is even loaded — previously it
ran the full multi-frame decode and died in an opaque einops rank error.
- wan_ref_image_encoder docstrings now describe both the 20-channel A14B
and 48-channel TI2V-5B condition paths (they claimed A14B-only and told
users to omit the node for TI2V-5B, contradicting the implementation).
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(api): per-user upload slots, probe validation, stream decode fallback
- VideoUploadLimitASGIMiddleware now accounts upload slots per user (cap 2)
on top of the global cap, so one tenant's slow chunked uploads can no
longer hold all four slots and starve other users into 429s. Single-user
mode keeps the whole global capacity (no per-user quota).
- probe_video validates decoder-reported metadata: non-positive or
over-limit dimensions (> 64 MP) and non-finite/negative durations are
rejected before the upload path persists them; garbage fps degrades to
None (unknown). The decode worker refuses to decode frames from files
whose probed dimensions exceed the bound — a small crafted container
claiming 100k x 100k would otherwise trigger a ~30 GB allocation.
- The worker's stream command falls back to cv2 like probe/frame/count do,
so an MP4 accepted at upload via the cv2 path now also works in the
frame-range and concat nodes. The fallback only engages before the first
emitted frame; a mid-stream decoder death still surfaces as an error.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(ui): media resilience fixes from JPPhoto's 2026-07-21 review
- Partial board deletion: the boardAndImagesDeleted listener now invalidates
the per-item Image*/Video* tags for the confirmed-deleted names it parses
out of the 500 detail — the rejected mutation runs invalidatesTags with no
result, so those caches previously stayed readable.
- Media-cookie refresh retries transient failures on a bounded backoff
(2s, 10s) instead of latching before the request and giving up forever;
401 still bails (session genuinely expired).
- Thumbnail 404s degrade gracefully: BoardTooltip, GalleryBoard,
VirtualBoardItem, and VideoFieldInputComponent show an icon fallback via
fallbackStrategy="onError" (thumbnail generation is best-effort server-
side), and GalleryVideoThumbnail's <video> fallback does the near-zero
seek on loadedmetadata so browsers that don't auto-paint the first frame
no longer show a black tile.
- CurrentVideoPreview handles play() rejection (rolls isPlaying back) and
media element errors (drops back to the play overlay) instead of hiding
the overlay over a dead element with an unhandled promise rejection.
- Hardening from the disputed items: changeVideoIsIntermediate also
invalidates the VideoList LIST_TAG (covers a future omitted-board_id
list); logout clears gallery.selection and the logout mutation documents
that resetApiState in store.ts is what actually clears cross-user caches.
- Typegen regenerated for the wan_lora_loader / wan_ref_image_encoder
docstring updates.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix: address final video review findings
* fix: harden video workflows and auth refresh
* chore: regenerate OpenAPI schema
* fix: close video workflow review gaps
* fix: address self-review findings on video workflows and auth refresh
Fixes the confirmed findings from the 2026-07-22 self-review round
(github.com/invoke-ai/InvokeAI/pull/9163#issuecomment-5051225515):
Frontend:
- DeleteVideoModal: detach the dialog promise callbacks before the async
deletion so the accept path's synchronous onClose (bound to cancel) no
longer rejects a confirmed deletion as "User canceled"; dismissal still
rejects. Adds behavioral tests for both paths.
- Sliding-window refresh: bound the media-cookie sync fetch with a 10s
AbortSignal timeout so a stalled request can't hold the exclusive
cross-tab media-auth lock (shared with login/logout) forever; commit the
refreshed token even when the cookie sync fails with a 5xx/network error
(only a 401/403 rejection of the token blocks the commit); throttle
acceptance to once per minute so bulk mutations don't pay a serialized
cookie round trip per request.
- Fallback media-auth lock: waiters renew their ticket lease while queueing
so a >30s wait no longer lets a later ticket enter concurrently.
- useMediaCookieRefresh: a pause() abort now resumes the same attempt
instead of consuming a retry slot (and no longer permanently disables
self-heal when the final attempt was paused); effect cleanup aborts
in-flight refreshes so every logout path (sessionExpiredLogout, direct
logout) stops a pending refresh from re-minting the cookie post-logout.
- CurrentVideoPreview: a benign AbortError from play() rolls back silently,
and load errors during the pending media-cookie self-heal window no
longer raise a spurious "Unable to Play Video" toast.
Backend:
- Decode-worker memory bounds resized for legal near-cap frames: worker
RLIMIT_AS headroom 1->4 GiB, parent RSS kill threshold 1->3 GiB (with
keep-in-sync cross-references), monitor poll 50->250 ms.
- Upload probe: a decode-worker timeout is now inconclusive (upload
proceeds) instead of a 415; the probe's decoded first frame is reused as
the thumbnail source, dropping one worker subprocess per upload.
- delete_images_on_board / delete_videos_on_board return (deleted, failed)
and delete_board reports the services' ground truth instead of a racy
router-side listing diff (which also doubled the DB work).
- Video list/uncategorized delete endpoints skip HTTPException (ownership
skips, 404 races) silently instead of reporting them as failures,
matching the image endpoints; delete_images_from_list now populates
failed_images for genuine failures, matching the video path.
- video_concat: an unknown probed fps mixed with agreeing known rates uses
the known rate again instead of hard-erroring; disagreeing known rates
still require an explicit Output FPS.
- SlidingWindowTokenMiddleware runs its synchronous SQLite user lookup via
run_in_threadpool so a contended DB lock can't stall the event loop.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(tests): deflake drip-feed upload timeout test on Windows
The test gave the middleware a 20 ms absolute upload deadline and delivered a
chunk every 5 ms — but Windows event-loop timers have ~15.6 ms granularity, so
the deadline could expire before the first chunk was ever delivered. The
request then ended at receive_calls == 1 and the `receive_calls > 1` assertion
failed (py3.12 windows-cpu CI).
Widen the margins so the scenario the test describes actually occurs on coarse
timers: 250 ms absolute deadline (many chunks flow first on every platform)
with a 1 s idle timeout that never fires between 5 ms chunks.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(tests): sort imports in test_video_upload_limits (ruff I001)
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix: address in-PR items from JPPhoto's non-merge-blocker list
Fixes the subset of JPPhoto's 2026-07-22 "Still Open, Non-Merge Blockers"
that are small, self-contained, and scoped to surfaces this PR introduced;
the rest are deferred to a follow-on PR (triage rationale posted on the PR).
- video_thumbnails._run_worker: an unexpected exception from communicate()
(e.g. OSError) now terminates the worker process tree unconditionally —
previously the finally stopped the RSS-monitor backstop while the except
path left the worker and its ffmpeg child running forever. Adds the
injected-OSError test JPPhoto asked for.
- reduxRemember driver: client-state persistence POSTs now commit
X-Refreshed-Token via the same acceptance flow as dynamicBaseQuery
(extracted as acceptRefreshedToken, sharing the cross-tab lock, cookie
sync, throttle, and generation guards), so persistence-only sessions no
longer hard-expire mid-activity.
- delete_videos_from_list / delete_images_from_list: dedup request names —
a repeated name was processed twice and landed in both deleted_* and
failed_* under the admin ownership bypass, toasting a spurious partial
failure. Regression test added.
- gallery + videos list endpoints: bound offset (ge=0) and limit
(ge=0, le=MAX_PAGE_SIZE=1000) — these flowed verbatim into SQL, where a
negative LIMIT means unlimited in SQLite, so one request could
materialize the entire gallery. openapi.json regenerated (schema.ts is
unchanged — constraints don't alter the generated types).
- get_video_full: open the file once and serve HEAD/range/full from the fd
(full downloads now stream chunked from the handle instead of
FileResponse's lazy path-based open), eliminating the delete-race 500;
deletion's atomic rename can no longer invalidate a path between check
and open.
- upload_video: close the multipart spool immediately after the body copy,
shrinking the double-temp-disk window (2 x 1 GiB x 4 concurrent worst
case) to the copy loop itself.
- docs/gallery.mdx: document shared-board deletion semantics (only your
own media is permanently deleted; admins delete everything) and the
kept-on-failure -> Uncategorized behavior with its UI warning.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix video gallery review findings
* fix Windows video thumbnail handling
* test: cover remaining video review findings
* test: cover adversarial video and Wan findings
* fix: address remaining video and Wan review findings
* test: call now-sync star/unstar routes directly
11b38696bf converted the video batch routes from async def to sync def
(so FastAPI offloads them to its threadpool), but the star/unstar
dedupe test still drove them through asyncio.run(), which requires a
coroutine.
Co-Authored-By: Claude Fable 5 <[email protected]>
* test: cover Wan conditioning and video link regressions
* fix: validate Wan conditions and video links
---------
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Co-authored-by: Jonathan <[email protected]>
Co-authored-by: JPPhoto <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5
Jonathan
JPPhoto
parent
527d984b05
commit
eb9a951248
@@ -219,10 +219,13 @@ If no arguments are provided, the script will run in interactive mode.
|
||||
|
||||
:::danger[Data Loss]
|
||||
Deleting a user removes the user record and cascades to their sessions, board shares, sent
|
||||
invitations, and per-user client state. It does **not** delete the boards, images, workflows, queue
|
||||
items, or style presets they created — those rows remain in the database, owned by a user_id that no
|
||||
longer exists, and will not appear in any user's gallery. Physical image files in `outputs/images`
|
||||
are also left in place until a gallery maintenance script is run to remove orphan images.
|
||||
invitations, and per-user client state. It does **not** delete the boards, images, videos,
|
||||
workflows, queue items, or style presets they created — those rows remain in the database, owned by
|
||||
a user_id that no longer exists, and will not appear in other (non-admin) users' galleries.
|
||||
Administrators keep gallery visibility of the orphaned images and videos and can review or delete
|
||||
them from there. Physical media files in `outputs/images` and `outputs/videos` are also left in
|
||||
place: orphan image files can be removed with a gallery maintenance script, while video files should
|
||||
be deleted through the gallery (which removes the file along with the record) or cleaned up manually.
|
||||
|
||||
If you want their content gone as well, reassign or delete it before deleting the user. Back up the
|
||||
database first if recovery might be needed.
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
---
|
||||
title: Gallery Panel
|
||||
description: Learn how to manage, organize, and use your generated images and assets with the Gallery Panel in InvokeAI.
|
||||
lastUpdated: 2026-02-19
|
||||
description: Learn how to manage, organize, and use your generated images, videos, and assets with the Gallery Panel in InvokeAI.
|
||||
lastUpdated: 2026-07-19
|
||||
sidebar:
|
||||
order: 1
|
||||
---
|
||||
|
||||
import { Card, CardGrid, Steps } from '@astrojs/starlight/components';
|
||||
|
||||
The Gallery Panel is a fast way to review, find, and make use of images you've generated and loaded. The Gallery is divided into **Boards**. The *Uncategorized* board is always present, but you can create your own for better organization.
|
||||
The Gallery Panel is a fast way to review, find, and make use of images and videos you've generated and loaded. The Gallery is divided into **Boards**. The *Uncategorized* board is always present, but you can create your own for better organization. Boards are polymorphic — images and videos coexist on the same board and appear together in the gallery, sorted by creation time.
|
||||
|
||||

|
||||
|
||||
@@ -50,11 +50,17 @@ Each board has a context menu accessible via right-click (or Ctrl+click).
|
||||

|
||||
|
||||
- **Auto-add to this Board:** If *Auto-Assign Board on Click* is disabled in settings, use this option to quickly set the selected board as the default destination for new images.
|
||||
- **Download Board:** Packages all images within the board into a `.zip` file. A notification link will be provided when the download is ready.
|
||||
- **Delete Board:** Permanently removes the board and all of its contents.
|
||||
- **Download Board Images:** Packages all images within the board into a `.zip` file. A notification link will be provided when the download is ready. Videos are not included; download them individually from the gallery or viewer.
|
||||
- **Delete Board:** Opens a confirmation dialog. **Delete Board Only** removes the board and leaves its contents uncategorized; **Delete Board and Assets** permanently removes the board, its images, and its videos.
|
||||
|
||||
:::danger
|
||||
Deleting a board will **permanently delete all images** contained within it. Proceed with caution!
|
||||
Only **Delete Board and Assets** permanently deletes the board's images and videos. This action cannot be undone.
|
||||
:::
|
||||
|
||||
:::note
|
||||
In multi-user deployments, **Delete Board and Assets** only permanently deletes *your own* media on a shared board — other users' contributions are preserved and move to Uncategorized. (Administrators delete everything on the board.)
|
||||
|
||||
If any item's files cannot be removed (for example, due to a transient storage error), that item is kept rather than left orphaned on disk: its record moves to Uncategorized, and the app shows a warning with the number of items that could not be deleted.
|
||||
:::
|
||||
|
||||
### Image Storage Maintenance
|
||||
@@ -67,22 +73,22 @@ Every board is organized into two distinct tabs:
|
||||
|
||||

|
||||
|
||||
1. **Images:** Images generated directly within InvokeAI.
|
||||
1. **Images/Videos:** Images and videos generated directly within InvokeAI, shown interleaved by creation time.
|
||||
2. **Assets:** External images you have uploaded to use as an [Image Prompt](https://support.invoke.ai/support/solutions/articles/151000159340-using-the-image-prompt-adapter-ip-adapter-) or within the Image to Image tab.
|
||||
|
||||
---
|
||||
|
||||
## Virtual Boards
|
||||
|
||||
Virtual boards are read-only board groupings that Invoke computes on-the-fly from your image metadata rather than storing in the database. The first available type groups images **By Date**, creating one sub-board per day on which you generated images.
|
||||
Virtual boards are read-only board groupings that Invoke computes on-the-fly from your gallery records rather than storing in the database. The first available type groups media **By Date**, creating one sub-board per day on which you generated images or videos.
|
||||
|
||||
Virtual boards are **off by default**. To enable them:
|
||||
|
||||
1. Open the **board settings** (gear icon at the top of the Gallery).
|
||||
2. Toggle **Virtual Boards** on.
|
||||
3. A collapsible **By Date** section appears in the board list, with a sub-board for each day that has images. Each sub-board shows the date, image / asset counts, and a cover thumbnail.
|
||||
3. A collapsible **By Date** section appears in the board list, with a sub-board for each day that has images or videos. Each sub-board shows the date, image / asset / video counts, and a cover thumbnail (the newest item of that day, image or video).
|
||||
|
||||
Selecting a date sub-board filters the gallery to just the images from that day. The collapse state of the By Date section persists across reloads.
|
||||
Selecting a date sub-board filters the gallery to just the images and videos from that day. The collapse state of the By Date section persists across reloads.
|
||||
|
||||
### Limits
|
||||
|
||||
@@ -90,7 +96,7 @@ Because virtual boards are derived, not stored:
|
||||
|
||||
- They are **read-only**: no drag-and-drop, no context menu, no auto-add destination.
|
||||
- You cannot rename or delete them.
|
||||
- Generating a new image updates the counts immediately, but the image is still saved to your regular auto-add board — virtual boards are a *view*, not a destination.
|
||||
- Generating a new image or video updates the counts immediately, but the item is still saved to your regular auto-add board — virtual boards are a *view*, not a destination.
|
||||
- Disabling the **Virtual Boards** toggle hides the section and resets the selection to *Uncategorized* if you were viewing a virtual sub-board.
|
||||
|
||||
---
|
||||
@@ -134,6 +140,29 @@ Additionally, each image has a context menu (right-click or Ctrl+click) with pow
|
||||
|
||||
---
|
||||
|
||||
## Videos in the Gallery
|
||||
|
||||
Videos generated by InvokeAI (currently from the Wan 2.2 model family) appear alongside images in the same gallery view. Each video item displays a first-frame still as its thumbnail with a play badge in the corner; selecting it opens the video in the viewer where you can play it back inline.
|
||||
|
||||
### Uploading Videos
|
||||
|
||||
You can upload existing videos to a board via the standard drop-or-upload affordance. The upload pipeline accepts **MP4 files encoded with H.264/AVC only** so uploaded videos can play in supported browsers. Other containers (`.mov`, `.webm`, `.mkv`) and codecs such as HEVC/H.265 are not transcoded on upload and are rejected at the API boundary. Re-encode them before uploading, for example with `ffmpeg -i input.mov -c:v libx264 output.mp4`.
|
||||
|
||||
The **Concatenate Videos** and **Extract Video Range** nodes encode video frames only; audio tracks are not preserved. Concatenation requires matching source frame rates unless an explicit output FPS is selected, which retimes the source frames.
|
||||
|
||||
### Video Context Menu
|
||||
|
||||
Each video has a context menu with the same organization actions as images, plus video-appropriate variants:
|
||||
|
||||
- **Open in New Tab / Download:** Opens or saves the raw MP4 file.
|
||||
- **Star Video:** Pins the video to the top of the gallery.
|
||||
- **Change Board:** Moves the video to a different board. *(Drag-and-drop onto board thumbnails also works.)*
|
||||
- **Delete Video:** Permanently deletes the video and its thumbnail.
|
||||
|
||||
Videos count toward board contents: a board with two images and three videos shows five items in the polymorphic gallery list and reports both totals in its stats.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
This walkthrough covers the Gallery interface and Boards. For guidance on prompting and generation workflows, please refer to the [Prompting Guide](/concepts/prompting-guide/) and [AI Image Generation](/concepts/image-generation/).
|
||||
|
||||
@@ -0,0 +1,267 @@
|
||||
---
|
||||
title: Video Generation
|
||||
description: Generate short videos with the Wan 2.2 model family — text-to-video, image-to-video, and the trick for stitching longer sequences.
|
||||
lastUpdated: 2026-07-19
|
||||
---
|
||||
|
||||
import { Card, CardGrid, Steps } from '@astrojs/starlight/components';
|
||||
|
||||
InvokeAI ships **experimental support for the Wan 2.2 model family**, which lets you generate short MP4 clips from a text prompt, an image, or both. Output ranges from a few-second loop (the model's training distribution) up to longer sequences assembled with the [concat trick](#making-longer-videos) below.
|
||||
|
||||
:::caution[Experimental]
|
||||
Video generation is a prototype feature. Workflows, node fields, and starter-model packaging may change between releases. The underlying models are also new — expect rough edges in coherence and artifacts at longer durations.
|
||||
:::
|
||||
|
||||
:::tip[Just want to make a video?]
|
||||
Start with the [**Video Workflows** guide](/features/video-workflows/) — a plain-language tour of the ten ready-made workflows that ship with InvokeAI and how to run them. This page is the deeper technical reference behind them.
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## Models
|
||||
|
||||
Wan 2.2 ships three transformer variants, plus a shared text encoder and two VAEs. All share the same diffusion-style sampling but differ in size, conditioning, and intended task.
|
||||
|
||||
### The variants
|
||||
|
||||
| Variant | Task | Params | VAE | Conditioning |
|
||||
|---|---|---|---|---|
|
||||
| **T2V-A14B** | Text → Video | 14B × 2 experts | A14B VAE (16-ch, 8× spatial) | Text only |
|
||||
| **I2V-A14B** | Image+Text → Video | 14B × 2 experts | A14B VAE (16-ch, 8× spatial) | Text + reference image (36-channel concat) |
|
||||
| **TI2V-5B** | Text → Video OR Image+Text → Video | 5B (single) | Wan 2.2-VAE (48-ch, 16× spatial) | Text, optionally with reference image (first-frame mask blend) |
|
||||
|
||||
* **T2V-A14B** generates videos from a text prompt alone. Best motion coherence and prompt-following of the three.
|
||||
* **I2V-A14B** locks the first frame to a reference image you supply. Best subject-preservation; the image is concatenated to the noise latents at every step so the model "sees" the reference throughout denoising.
|
||||
* **TI2V-5B** is the small single-expert variant. It can do **both** text-to-video and image-to-video with the same checkpoint, at substantially lower VRAM, but with somewhat less stable long-range coherence than the A14B variants.
|
||||
|
||||
### High-noise and low-noise transformers (A14B variants only)
|
||||
|
||||
The A14B models are a **mixture-of-experts (MoE)** pair. There are actually *two* 14B transformers on disk per variant — a "high-noise" expert and a "low-noise" expert — and the denoise loop swaps between them at a model-defined boundary timestep:
|
||||
|
||||
* **High-noise expert** runs early in denoising, when the latents are still mostly noise. It's responsible for composition, layout, and broad motion.
|
||||
* **Low-noise expert** runs later, when the latents are close to clean. It refines detail and texture.
|
||||
|
||||
InvokeAI handles the swap automatically — both experts have to be installed (the starter bundle handles this), but the workflow only references the "high-noise" model as the main and the "low-noise" model is wired alongside it via the loader node. You don't manage the boundary yourself.
|
||||
|
||||
**TI2V-5B is single-expert** — no swap, no boundary, just one model that runs every step. Workflows for TI2V-5B are correspondingly simpler.
|
||||
|
||||
### Lightning LoRAs (4-step inference for A14B)
|
||||
|
||||
The default A14B variants need ~40–50 denoise steps for clean output. The Wan team also released **Lightning distillation LoRAs** that collapse that to 4 steps with minimal quality loss — about a 10× speedup. There's a pair per variant (one LoRA for the high-noise expert, one for the low-noise), wired through the LoRA loader nodes in the starter workflows.
|
||||
|
||||
:::tip
|
||||
**TI2V-5B doesn't have a Lightning LoRA.** Its smaller size means each step is cheap; you typically run it at 40–50 steps and end up in a similar wall-clock ballpark as A14B + Lightning. (The bundled TI2V-5B workflows ship with 20 steps as a speed/quality compromise — raise the step count if you want cleaner output.)
|
||||
:::
|
||||
|
||||
### Installing models
|
||||
|
||||
The model manager ships two **starter bundles** for video work:
|
||||
|
||||
* **Wan 2.2 Text-to-Video** (~36 GB) — UMT5-XXL text encoder, both VAEs, TI2V-5B Q4_K_M, T2V-A14B Q4_K_M (high + low), T2V Lightning (high + low).
|
||||
* **Wan 2.2 Image-to-Video** (~32 GB) — UMT5-XXL, A14B VAE, I2V-A14B Q4_K_M (high + low), I2V Lightning (high + low).
|
||||
|
||||
The bundles are independent. Installing both ends up at ~56 GB total (shared components — UMT5-XXL and the A14B VAE — are deduplicated on the second install). A 12 GB VRAM card can install only the Text-to-Video bundle and have **TI2V-5B available for both T2V and image-to-video** without ever touching the I2V bundle.
|
||||
|
||||
Higher-quality Q8_0 quantizations of every transformer, plus full Diffusers builds of all three variants, are available as a-la-carte installs in the starter models list.
|
||||
|
||||
:::tip
|
||||
**On a 12 GB VRAM card**: install just the Text-to-Video bundle and use TI2V-5B. The A14B variants will technically run via aggressive offloading but are slow and prone to OOM. TI2V-5B Q4_K_M fits comfortably and is what we recommend for that tier.
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## Workflow setup
|
||||
|
||||
The shipped starter workflows ("Text to Video - Wan 2.2 Lightning", "Image to Video - Wan 2.2 Lightning") are the easiest starting point — load them from the workflow library, pick your models, set a prompt, and Invoke. The [Video Workflows guide](/features/video-workflows/) walks through all ten and how to choose between them; the sections below describe what's happening inside so you can build your own.
|
||||
|
||||
### Constraints that apply to every video workflow
|
||||
|
||||
**Frame count**: `num_frames - 1` must be divisible by **4**. This is dictated by the Wan VAE's temporal compression (4 pixel-frames → 1 latent-frame). Valid values: 5, 9, 13, … **81** (the training default, 5 seconds at 16 fps), 85, 89, etc.
|
||||
|
||||
**Pixel dimensions**: must be a multiple of **16** for T2V-A14B and I2V-A14B, and a multiple of **32** for TI2V-5B. The constraint comes from the VAE's spatial downsample × the transformer's 2×2 patch size:
|
||||
|
||||
| Variant | VAE spatial | Pixel multiple of |
|
||||
|---|---|---|
|
||||
| T2V-A14B, I2V-A14B | 8× | **16** |
|
||||
| TI2V-5B | 16× | **32** |
|
||||
|
||||
Reference values that work: 832×480 (480p), 1280×720 (720p, A14B only — TI2V-5B needs 1280×704 instead since 720 isn't divisible by 32).
|
||||
|
||||
**Encoder and denoise dimensions must match**: the `Reference Image - Wan 2.2` encoder and the `Denoise Video - Wan 2.2` node both have their own `width` and `height` fields. They have to be identical or the denoise loop will reject the condition tensor.
|
||||
|
||||
:::tip[Use Wan 2.2 I2V Ideal Dimensions]
|
||||
The **Wan 2.2 I2V Ideal Dimensions** node takes a source image's W×H and a target preset (480p / 720p / 1080p) and outputs valid (width, height) for the encoder + denoise inputs. Wire it once and feed its outputs into both nodes. Saves the manual snap-to-16/snap-to-32 math.
|
||||
:::
|
||||
|
||||
### Text-to-video workflow
|
||||
|
||||
The minimum node chain for T2V:
|
||||
|
||||
```
|
||||
Wan Main Model Loader ──┐
|
||||
│
|
||||
Wan T5 Text Encoder ────┤
|
||||
▼
|
||||
Wan Compel Conditioning (positive)
|
||||
│
|
||||
▼
|
||||
Denoise Video - Wan 2.2 ──→ Latents to Video - Wan 2.2 ──→ MP4
|
||||
▲
|
||||
Wan Compel Conditioning (negative) ─┘
|
||||
```
|
||||
|
||||
For **TI2V-5B T2V** this is the entire graph — load the TI2V-5B model and the TI2V-5B VAE, set width/height/num_frames, and run.
|
||||
|
||||
For **T2V-A14B** the main model loader also exposes the low-noise expert slot, and you typically add the **Lightning LoRA pair** (one for each expert) to bring step count down to 4. Recommended:
|
||||
|
||||
* Steps: **4** (with Lightning) or **40–50** (without)
|
||||
* CFG: **5.0** high-noise / **4.0** low-noise (the dedicated `Guidance Scale (Low Noise)` field on the denoise node)
|
||||
* Width × Height: 832×480 (faster, default) or 1280×720 (sharper, 4× the memory)
|
||||
|
||||
### Image-to-video workflow
|
||||
|
||||
I2V adds a **Reference Image** branch alongside the denoise. The reference image gets VAE-encoded into a conditioning tensor that the denoise loop uses to anchor the video's content:
|
||||
|
||||
```
|
||||
Wan Main Model Loader ──┐
|
||||
Wan T5 Text Encoder ────┤
|
||||
Wan Compel Conditioning ┤
|
||||
│
|
||||
Image Primitive ──→ Reference Image - Wan 2.2 ──┐
|
||||
│ │
|
||||
▼ ▼
|
||||
Denoise Video - Wan 2.2 ──→ Latents to Video - Wan 2.2 ──→ MP4
|
||||
```
|
||||
|
||||
For **I2V-A14B**, both the reference encoder and the denoise node need to use the same width/height. The encoder also takes a `num_frames` parameter that must match the denoise's `num_frames` — set both to 81 by default.
|
||||
|
||||
For **TI2V-5B image-to-video**, the conditioning math is different (the model uses a first-frame-mask blend rather than channel concatenation), but the workflow shape is the same. The encoder auto-detects TI2V-5B from the VAE's 48 latent channels and emits the right condition tensor.
|
||||
|
||||
:::caution[TI2V-5B I2V dimensional constraint]
|
||||
TI2V-5B image-to-video requires **width and height divisible by 32** (not just 16). The encoder will refuse the workflow with a clear error if not. 832×480 works; 1280×720 does not (720 is not divisible by 32). Use 1280×704 for 720p-ish on TI2V-5B.
|
||||
:::
|
||||
|
||||
### First-last-frame interpolation (FLF2V)
|
||||
|
||||
The **Reference Image - Wan 2.2** node has an optional **End Image** input. Provide a start image on the regular `image` field and an end image on `End Image (FLF2V)`, and the model interpolates a clip that **begins on the first image and animates to the second** — handy for morphing between two stills or bridging two shots.
|
||||
|
||||
```
|
||||
Image Primitive (start) ──→ image ────┐
|
||||
├─→ Reference Image - Wan 2.2 ──→ Denoise Video ──→ MP4
|
||||
Image Primitive (end) ──→ End Image ───┘
|
||||
```
|
||||
|
||||
This is **I2V-A14B only** (`num_frames > 1`); it is not supported on TI2V-5B or single-frame I2V (the node raises a clear error). Wan 2.2 has no dedicated first-last-frame checkpoint — that was a Wan 2.1 model — so the stock I2V-A14B model accepts the end-frame anchor through its existing conditioning channels. Quality is good in practice but not guaranteed for every image pair, so eyeball your results. The shipped **"Interpolate 2 Images to Video - Wan 2.2 Lightning"** workflow wires this up end to end.
|
||||
|
||||
### Recommended starting parameters
|
||||
|
||||
| | T2V-A14B + Lightning | T2V-A14B | I2V-A14B + Lightning | TI2V-5B (T2V or I2V) |
|
||||
|---|---|---|---|---|
|
||||
| Steps | 4 | 40–50 | 4 | 40–50 (bundled workflows use 20 for speed) |
|
||||
| CFG (high) | 1.0 | 5.0 | 1.0 | 5.0–5.5 |
|
||||
| CFG (low) | 1.0 | 4.0 | 1.0 | n/a (single expert) |
|
||||
| Num frames | 81 | 81 | 81 | 81 |
|
||||
| Width × Height | 832×480 | 832×480 | 832×480 | 832×480 |
|
||||
| Scheduler | Auto (UniPC) | Auto (UniPC) | Auto (UniPC) | Auto (UniPC) |
|
||||
|
||||
---
|
||||
|
||||
## Making longer videos
|
||||
|
||||
The Wan 2.2 models were trained on **81-frame** clips (5 seconds at 16 fps). Outputs much longer than that suffer rapidly degrading coherence — the temporal positional encoding goes out of distribution and the model loses track of scene content. So instead of asking for `num_frames=200`, the recommended pattern is **chaining**: render a sequence of 81-frame clips where each one's first frame matches the previous clip's last frame, then concatenate them with the `Concatenate Videos` node.
|
||||
|
||||
### The basic chain
|
||||
|
||||
<Steps>
|
||||
|
||||
1. **Render the first clip** with I2V or T2V, ending on whatever subject/scene you want to continue.
|
||||
|
||||
2. **Extract the last frame** of clip 1 using the `Frame from Video` node. Use `frame_index = -1` for the literal last frame, or `-3` / `-5` to step back a few frames (last frames sometimes have boundary artifacts — see the [troubleshooting](#late-frame-artifacts-text-color-blobs) note).
|
||||
|
||||
3. **Feed that frame as the reference image** for an I2V run that becomes clip 2. Adjust the prompt for whatever motion you want next.
|
||||
|
||||
4. **Repeat** as many times as you want clips.
|
||||
|
||||
5. **Concatenate** all the clips into a single MP4 with `Concatenate Videos`. Pick a transition mode based on whether you want a seamless join (`cut` if the bridge frame matches perfectly), a smooth blend (`crossfade`), or a punctuated scene change (`fade_through_black`).
|
||||
|
||||
</Steps>
|
||||
|
||||
### Transition modes
|
||||
|
||||
The `Concatenate Videos` node offers three:
|
||||
|
||||
* **`cut`** — hard splice. Fastest. Total length = sum of inputs. Use this when the bridge frame is genuinely shared (clip 2's first frame = clip 1's last frame) — the seam is invisible.
|
||||
* **`crossfade`** — linear A→B dissolve over `transition_frames`. Consumes `transition_frames` from both sides of each boundary. Total length = `sum(inputs) - transition_frames × (n-1)`. Use this when bridge frames don't quite match.
|
||||
* **`fade_through_black`** — A fades to black, then B fades in from black. Total length is preserved. Use this for explicit scene changes.
|
||||
|
||||
### Quality degradation across iterations
|
||||
|
||||
A real failure mode of long chains: each iteration's reference image is itself a *generation output*, so artifacts compound. The model treats codec artifacts and VAE softness in the bridge frame as "style" and reproduces them in the next clip. By the 4th or 5th iteration you can see noticeable softening or color drift.
|
||||
|
||||
**Mitigations**:
|
||||
|
||||
1. **Pick a bridge frame a few back from the end** (e.g., `frame_index = -3` or `-5`). The very last frame is often the worst frame of a clip due to boundary effects in the temporal attention.
|
||||
2. **Refresh the bridge frame with a low-strength img2img pass** before feeding it into the next I2V. An SDXL or FLUX img2img at strength ~0.2 with a quality-focused negative prompt (`blurry, low quality, compression artifacts`) noticeably suppresses the cumulative drift.
|
||||
3. **Don't chain more than 4–5 clips** unless you're explicitly doing img2img refinement between each.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### OOM errors
|
||||
|
||||
Video denoise is memory-intensive — attention scales roughly as `(T_lat × H/16 × W/16)²`, so resolution and frame count both quadratically affect peak VRAM.
|
||||
|
||||
* **Drop resolution before frame count.** Going from 1280×720 to 832×480 is a ~2.4× memory drop and visually subtle in most content. Going from 81 frames to 65 only saves ~20%.
|
||||
* **TI2V-5B before A14B.** TI2V-5B Q4_K_M peaks around ~6–8 GB at 832×480, versus ~12–14 GB for A14B Q4_K_M. If you're at the OOM edge, switch model family.
|
||||
* **OOM at the *reference image encoder* step** is usually allocator fragmentation from a previous run rather than absolute memory pressure. Restart the dev server and try again; if it recurs reproducibly, file an issue.
|
||||
|
||||
### Late-frame artifacts (text, color blobs)
|
||||
|
||||
If your video looks great for most of its duration but the last ~20% develops Asian text, watermarks, or floating colored shapes, **that's the model's training-data prior leaking through** as temporal coherence weakens at long temporal distance. It's particularly common on TI2V-5B (smaller model, less capacity to hold scene).
|
||||
|
||||
**Mitigations**:
|
||||
|
||||
* Add to the negative prompt: `text, watermark, logo, subtitles, chinese characters, kanji, ticker, banner`
|
||||
* Use a more specific prompt — describe the action you want to *happen* through the clip, not just the static scene
|
||||
* Bump CFG to 5.5 (TI2V-5B tolerates this)
|
||||
* Stay at `num_frames=81`; values above push temporal RoPE out of distribution and artifacts accelerate
|
||||
|
||||
### "Dimensions must be multiples of 16/32"
|
||||
|
||||
The encoder and denoise nodes enforce these at runtime. Either:
|
||||
|
||||
* Use the **Wan 2.2 I2V Ideal Dimensions** node to compute valid (W, H) automatically from a source image, or
|
||||
* Manually round to the right multiple (16 for A14B variants, 32 for TI2V-5B)
|
||||
|
||||
### Reference image / denoise dimension mismatch
|
||||
|
||||
If the denoise refuses with `Reference-image dimensions … must match denoise dimensions`, both nodes have their own width/height fields and they need to agree. Wire the same values (or the same Ideal Dimensions output) into both.
|
||||
|
||||
### TI2V-5B VAE state-dict load error
|
||||
|
||||
If `Latents to Video - Wan 2.2` fails with `Error(s) in loading state_dict for AutoencoderKLWan: ... size mismatch for ...`, you have the **wrong VAE installed** for the chosen transformer. TI2V-5B needs the **Wan 2.2 TI2V-5B VAE** (48-channel, Wan 2.2-VAE), not the A14B VAE (16-channel). Both are in the model manager — check the VAE field on the loader and the latents-to-video node.
|
||||
|
||||
### Sampler drift on the standalone TI2V-5B GGUF
|
||||
|
||||
Standalone GGUF installs don't ship a `scheduler/` config directory. InvokeAI now defaults to `UniPCMultistepScheduler` with the correct Wan-flow params when the model is TI2V-5B and there's no on-disk scheduler — but if you have an older install behaving oddly, the safer alternative is the **full Diffusers TI2V-5B** install (which includes the scheduler config).
|
||||
|
||||
### Preview images don't appear
|
||||
|
||||
Two known causes:
|
||||
|
||||
1. **A video is already loaded in the viewer.** If the last-selected gallery item is a video, the viewer renders the video element. The progress preview overlays on top of it. If you don't see it at all, **hard-refresh the browser** (`Ctrl+Shift+R` / `Cmd+Shift+R`) — Vite's bundle cache occasionally serves a stale build.
|
||||
2. **`Show progress in viewer` is disabled.** Check the gallery settings (gear icon at the top of the gallery panel).
|
||||
|
||||
### Pipeline runs but the final MP4 is glitchy
|
||||
|
||||
This is almost always a **VAE mismatch** or a **scheduler mismatch** — both surface as garbage at the very end of the pipeline. Check that:
|
||||
|
||||
* The VAE matches the transformer family (16-ch for A14B, 48-ch for TI2V-5B)
|
||||
* You're using the default (auto-selected) scheduler — manually overriding it is currently not supported
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
Wan 2.2 model family by the Alibaba Wan-AI team. Lightning distillation LoRAs by [lightx2v](https://huggingface.co/lightx2v/Wan2.2-Lightning). GGUF quantizations by [QuantStack](https://huggingface.co/QuantStack).
|
||||
@@ -0,0 +1,166 @@
|
||||
---
|
||||
title: Video Workflows
|
||||
description: A plain-language guide to the ready-made Wan 2.2 video workflows that ship with InvokeAI — which one to pick, and how to run it.
|
||||
lastUpdated: 2026-05-30
|
||||
---
|
||||
|
||||
import { Card, CardGrid, Steps, LinkButton } from '@astrojs/starlight/components';
|
||||
|
||||
InvokeAI ships **ten ready-made video workflows** so you can start generating video without wiring up nodes yourself. Each one is a complete recipe — open it from the **Workflows** library, pick your models, type a prompt (and/or drop in an image), and press **Invoke**.
|
||||
|
||||
This page helps you choose the right workflow and run it. For how the models work under the hood, VRAM details, and troubleshooting, see the [Video Generation guide](/features/video-generation/).
|
||||
|
||||
:::caution[Experimental]
|
||||
Video generation is a prototype feature. These workflows and the models behind them are new — expect rough edges, and check the [Video Generation guide](/features/video-generation/) if something misbehaves.
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## Before you start
|
||||
|
||||
**1. Install the video models.** Open the **Model Manager** and install a Wan 2.2 starter bundle (Text-to-Video and/or Image-to-Video). The [Video Generation guide explains the bundles and what fits your GPU](/features/video-generation/#installing-models).
|
||||
|
||||
**2. You'll select models when you open a workflow.** The shipped workflows come with their model slots **left empty on purpose** — every InvokeAI install stores models a little differently, so a workflow can't point at yours automatically. When you open one, look at its **Notes** panel (the note icon in the workflow editor): it lists exactly which model to pick in each slot. Drop those in once and you're set.
|
||||
|
||||
:::tip
|
||||
Whenever a workflow won't run because a model field is empty, open its **Notes** — the shopping list of models to select is right there.
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## Which workflow should I use?
|
||||
|
||||
The workflows come in three families — **Text to Video**, **Image to Video**, and **Extend Video** — each with a high-quality option, a "concept LoRA" variant, and a lighter low-VRAM option — plus an **Interpolate** workflow that bridges two images and an **Extend Video to Image** variant that steers a continuation toward a target frame.
|
||||
|
||||
| I want to… | Use | Notes |
|
||||
|---|---|---|
|
||||
| Make a clip from a text description | **Text to Video - Wan 2.2 Lightning** | Best quality, fast. Start here. |
|
||||
| Animate an existing image | **Image to Video - Wan 2.2 Lightning** | Your image becomes the first frame. |
|
||||
| Make a video between two images | **Interpolate 2 Images to Video - Wan 2.2 Lightning** | Provide a start and end image; the model interpolates between them. |
|
||||
| Make a video longer than ~5 seconds | **Extend Video - Wan 2.2 Lightning** | Continues an existing video and stitches the pieces together. |
|
||||
| Extend a video toward a specific end frame | **Extend Video to Image - Wan 2.2 Lightning** | Continues a video and interpolates the new part to a target image. |
|
||||
| Add my own style/subject LoRAs | the **w/ Concept LoRAs** variant of any of the above | Same as the base workflow, plus slots for your LoRAs. |
|
||||
| Run on a smaller GPU (≈12–16 GB) | a **TI2V-5B (Low Quality)** workflow | Smaller model, lower memory, lower quality. |
|
||||
|
||||
If you're not sure, start with **Text to Video - Wan 2.2 Lightning** to get a feel for it, then move to Image-to-Video and Extend.
|
||||
|
||||
---
|
||||
|
||||
## Bundled video workflows
|
||||
|
||||
### Text to Video
|
||||
|
||||
<CardGrid>
|
||||
<Card title="Text to Video - Wan 2.2 Lightning" icon="pencil">
|
||||
Type a prompt, get a ~5-second clip. Uses the high-quality A14B model with the Lightning speed-up (just 4 steps). **The recommended starting point.**
|
||||
</Card>
|
||||
<Card title="Text to Video - …w/ Concept LoRAs" icon="pencil">
|
||||
The same workflow, plus extra slots for your own concept LoRAs (a style or character you've trained or downloaded). Every LoRA slot must be filled — if you don't have concept LoRAs to add, use the plain version instead.
|
||||
</Card>
|
||||
<Card title="Text to Video - Wan 2.2 TI2V-5B (Low Quality)" icon="pencil">
|
||||
Uses the smaller TI2V-5B model. Lower quality and slower per result, but fits comfortably on 12–16 GB GPUs. Good for quick drafts or smaller cards.
|
||||
</Card>
|
||||
</CardGrid>
|
||||
|
||||
### Image to Video
|
||||
|
||||
<CardGrid>
|
||||
<Card title="Image to Video - Wan 2.2 Lightning" icon="star">
|
||||
Drop in a starting image and a prompt; the model animates outward from your image as the first frame. Uses the A14B model + Lightning. **The recommended image-to-video starting point.**
|
||||
</Card>
|
||||
<Card title="Image to Video - …w/ Concept LoRAs" icon="star">
|
||||
Same as above, with slots for your own concept LoRAs.
|
||||
</Card>
|
||||
<Card title="Image to Video - Wan 2.2 TI2V-5B (Low Quality)" icon="star">
|
||||
Image-to-video on the smaller TI2V-5B model. Lower quality, lower VRAM — the low-memory option for animating an image.
|
||||
</Card>
|
||||
</CardGrid>
|
||||
|
||||
### Interpolate Between Two Images
|
||||
|
||||
<CardGrid>
|
||||
<Card title="Interpolate 2 Images to Video - Wan 2.2 Lightning" icon="random">
|
||||
Provide a **start image** and an **end image**; the model generates a clip that begins on the first and animates smoothly to the second (first-last-frame interpolation). Great for morphing between two stills or bridging two shots. Uses I2V-A14B + Lightning (4-step).
|
||||
</Card>
|
||||
</CardGrid>
|
||||
|
||||
:::caution[The "Concept LoRAs" variants need a LoRA in every slot]
|
||||
A **concept LoRA** is an add-on that teaches the model a specific style, character, or subject. The *w/ Concept LoRAs* workflows are the same as their plain counterparts but add extra LoRA slots. **Every LoRA slot is required** — if any slot, including the extra concept ones, is left empty, the workflow won't run. Only use these variants when you actually have concept LoRAs to apply; otherwise use the plain version.
|
||||
:::
|
||||
|
||||
:::note[About the "TI2V-5B (Low Quality)" variants]
|
||||
"Low quality" here means *lower than the A14B Lightning workflows*, in exchange for running on much less video memory. On a 12–16 GB GPU they may be your best (or only) option, and they're handy everywhere for fast drafts.
|
||||
:::
|
||||
|
||||
### Extend Video
|
||||
|
||||
The Wan models are trained on short (~5 second) clips, so the way to make something longer is to **continue** a video and stitch the pieces together. These workflows do that for you.
|
||||
|
||||
<CardGrid>
|
||||
<Card title="Extend Video - Wan 2.2 Lightning" icon="clock">
|
||||
Provide an existing video and use the sliders to choose where to continue from. The workflow generates a new clip starting from that point and joins it onto the original with a smooth transition. Run it repeatedly to keep growing the video.
|
||||
</Card>
|
||||
<Card title="Extend Video - …w/ Concept LoRAs" icon="clock">
|
||||
Same as above, with slots for your own concept LoRAs so the continuation keeps your style or subject.
|
||||
</Card>
|
||||
<Card title="Extend Video to Image - Wan 2.2 Lightning" icon="clock">
|
||||
Continue a video **toward a target image**. Provide a starting video and a destination image; the new segment interpolates from the video's last frame to your image (FLF2V), then joins onto the original with a smooth transition. Combines the Extend and Interpolate ideas — useful for steering where a continuation ends up.
|
||||
</Card>
|
||||
</CardGrid>
|
||||
|
||||
:::tip[Using "Extend Video to Image"]
|
||||
This variant takes **two inputs** — the source video and a destination image:
|
||||
|
||||
1. Drop your **video** into the video input and use the **start/end frame sliders** to pick the slice to continue from.
|
||||
2. Drop a **destination image** into the image input. The generated continuation interpolates from the video's last selected frame to this image, so the extended clip *ends on your image*.
|
||||
3. **Invoke.** The new segment is concatenated onto the source with a cross-fade — tune the join (transition type and frame count) on the **Concatenate Videos** node.
|
||||
|
||||
Because the endpoint is pinned to your image, it's ideal for bridging a clip to a known next shot or landing on a specific final frame.
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## Running a workflow
|
||||
|
||||
<Steps>
|
||||
|
||||
1. **Open the Workflows tab** and load one of the workflows from the library.
|
||||
|
||||
2. **Read the Notes** (note icon) and select the models it lists — the main model, VAE, and text encoder, plus the Lightning LoRA pair on the A14B workflows.
|
||||
|
||||
3. **Set your inputs:**
|
||||
- *Text to Video* — type a **positive prompt** describing the scene and motion.
|
||||
- *Image to Video* — drop a starting **image** into the image field, and optionally add a prompt.
|
||||
- *Extend Video* — provide a **starting video** and use the sliders to pick where the continuation begins.
|
||||
|
||||
4. **(Optional) adjust resolution.** The defaults are chosen to balance speed and memory. If you hit an out-of-memory error, lower the resolution preset — see the [VRAM tips](#which-one-fits-my-gpu) below.
|
||||
|
||||
5. **Press Invoke.** The finished MP4 appears in the gallery and plays inline in the viewer.
|
||||
|
||||
</Steps>
|
||||
|
||||
:::tip
|
||||
Leave the **frame count** and **steps** at their defaults to start. They're already tuned for each workflow (for example, the Lightning workflows use only 4 steps, which is what makes them fast).
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## Which one fits my GPU?
|
||||
|
||||
A rough guide — exact numbers depend on resolution and your system. See [Video Generation → OOM errors](/features/video-generation/#oom-errors) for the full picture.
|
||||
|
||||
| Your GPU | Best choice |
|
||||
|---|---|
|
||||
| 24 GB and up | Any workflow at 720p. |
|
||||
| 16–24 GB | A14B Lightning workflows at 480p; TI2V-5B at up to 720p. |
|
||||
| 12–16 GB | The **TI2V-5B (Low Quality)** workflows. |
|
||||
|
||||
If you get an **out-of-memory** error, lower the resolution first (it helps far more than reducing the frame count), and switch to a TI2V-5B workflow if you're still over budget.
|
||||
|
||||
---
|
||||
|
||||
## Going further
|
||||
|
||||
These workflows are starting points. Once you're comfortable, the [Video Generation guide](/features/video-generation/) covers what's happening inside them — the model variants, how image conditioning works, the trick for chaining clips into longer videos, recommended parameters, and troubleshooting.
|
||||
|
||||
<LinkButton href="/features/video-generation/" variant="secondary">Read the Video Generation guide</LinkButton>
|
||||
@@ -29,6 +29,8 @@ The requirements below are rough guidelines for best performance. GPUs with less
|
||||
| FLUX.2 Klein 4B | 1024x1024 | Nvidia 30xx+ | 12GB | 16GB | FP8 works with 8GB+; Diffusers + encoder |
|
||||
| FLUX.2 Klein 9B | 1024x1024 | Nvidia 40xx | 24GB | 32GB | FP8 works with 12GB+; Diffusers + encoder |
|
||||
| Z-Image Turbo | 1024x1024 | Nvidia 20xx+ | 8GB | 16GB | Q4_K 8GB; Q8/BF16 16GB+ |
|
||||
| Wan 2.2 A14B (T2V/I2V) | 1280x720 | Nvidia 30xx+ | 12GB | 32GB | Dual-expert MoE; Q4_K_M 12GB; Q8 18GB+; Diffusers requires 32GB+ |
|
||||
| Wan 2.2 TI2V-5B | 1280x720 | Nvidia 20xx+ | 8GB | 16GB | Single transformer; Q4_K_M 6GB+; Q8 8GB+; Diffusers 12GB+ |
|
||||
|
||||
:::tip[`tmpfs` on Linux]
|
||||
If your temporary directory is mounted as a `tmpfs`, ensure it has sufficient space.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from fastapi import Cookie, Depends, HTTPException, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
|
||||
from invokeai.app.api.dependencies import ApiDependencies
|
||||
@@ -13,6 +13,18 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
# HTTP Bearer token security scheme
|
||||
security = HTTPBearer(auto_error=False)
|
||||
MEDIA_TOKEN_COOKIE = "invokeai_media_token"
|
||||
|
||||
|
||||
def _validate_token(token: str, invalid_detail: str) -> TokenData:
|
||||
token_data = verify_token(token)
|
||||
if token_data is None:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail=invalid_detail)
|
||||
|
||||
user = ApiDependencies.invoker.services.users.get(token_data.user_id)
|
||||
if user is None or not user.is_active:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="User not found or inactive")
|
||||
return token_data
|
||||
|
||||
|
||||
async def get_current_user(
|
||||
@@ -116,6 +128,19 @@ async def get_current_user_or_default(
|
||||
return token_data
|
||||
|
||||
|
||||
async def get_current_media_user_or_default(
|
||||
credentials: Annotated[HTTPAuthorizationCredentials | None, Depends(security)],
|
||||
media_token: Annotated[str | None, Cookie(alias=MEDIA_TOKEN_COOKIE)] = None,
|
||||
) -> TokenData:
|
||||
"""Authenticate media requests with a Bearer token or the path-scoped login cookie."""
|
||||
if not ApiDependencies.invoker.services.configuration.multiuser:
|
||||
return TokenData(user_id="system", email="[email protected]", is_admin=True)
|
||||
token = credentials.credentials if credentials is not None else media_token
|
||||
if token is None:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Authentication required")
|
||||
return _validate_token(token, "Invalid or expired token")
|
||||
|
||||
|
||||
async def require_admin(
|
||||
current_user: Annotated[TokenData, Depends(get_current_user)],
|
||||
) -> TokenData:
|
||||
@@ -162,5 +187,6 @@ async def require_admin_or_default(
|
||||
# Type aliases for convenient use in route dependencies
|
||||
CurrentUser = Annotated[TokenData, Depends(get_current_user)]
|
||||
CurrentUserOrDefault = Annotated[TokenData, Depends(get_current_user_or_default)]
|
||||
CurrentMediaUserOrDefault = Annotated[TokenData, Depends(get_current_media_user_or_default)]
|
||||
AdminUser = Annotated[TokenData, Depends(require_admin)]
|
||||
AdminUserOrDefault = Annotated[TokenData, Depends(require_admin_or_default)]
|
||||
|
||||
@@ -10,6 +10,7 @@ from invokeai.app.services.auth.token_service import set_jwt_secret
|
||||
from invokeai.app.services.board_image_records.board_image_records_sqlite import SqliteBoardImageRecordStorage
|
||||
from invokeai.app.services.board_images.board_images_default import BoardImagesService
|
||||
from invokeai.app.services.board_records.board_records_sqlite import SqliteBoardRecordStorage
|
||||
from invokeai.app.services.board_video_records.board_video_records_sqlite import SqliteBoardVideoRecordStorage
|
||||
from invokeai.app.services.boards.boards_default import BoardService
|
||||
from invokeai.app.services.bulk_download.bulk_download_default import BulkDownloadService
|
||||
from invokeai.app.services.client_state_persistence.client_state_persistence_sqlite import ClientStatePersistenceSqlite
|
||||
@@ -24,6 +25,7 @@ from invokeai.app.services.external_generation.providers import (
|
||||
SeedreamProvider,
|
||||
)
|
||||
from invokeai.app.services.external_generation.startup import sync_configured_external_starter_models
|
||||
from invokeai.app.services.gallery.gallery_default import SqliteGalleryService
|
||||
from invokeai.app.services.image_files.image_files_disk import DiskImageFileStorage
|
||||
from invokeai.app.services.image_moves.image_moves_default import ImageMoveService
|
||||
from invokeai.app.services.image_records.image_records_sqlite import SqliteImageRecordStorage
|
||||
@@ -52,6 +54,9 @@ from invokeai.app.services.style_preset_images.style_preset_images_disk import S
|
||||
from invokeai.app.services.style_preset_records.style_preset_records_sqlite import SqliteStylePresetRecordsStorage
|
||||
from invokeai.app.services.urls.urls_default import LocalUrlService
|
||||
from invokeai.app.services.users.users_default import UserService
|
||||
from invokeai.app.services.video_files.video_files_disk import DiskVideoFileStorage
|
||||
from invokeai.app.services.video_records.video_records_sqlite import SqliteVideoRecordStorage
|
||||
from invokeai.app.services.videos.videos_default import VideoService
|
||||
from invokeai.app.services.workflow_records.workflow_records_sqlite import SqliteWorkflowRecordsStorage
|
||||
from invokeai.app.services.workflow_thumbnails.workflow_thumbnails_disk import WorkflowThumbnailFileStorageDisk
|
||||
from invokeai.backend.stable_diffusion.diffusion.conditioning_data import (
|
||||
@@ -64,6 +69,7 @@ from invokeai.backend.stable_diffusion.diffusion.conditioning_data import (
|
||||
QwenImageConditioningInfo,
|
||||
SD3ConditioningInfo,
|
||||
SDXLConditioningInfo,
|
||||
WanConditioningInfo,
|
||||
ZImageConditioningInfo,
|
||||
)
|
||||
from invokeai.backend.util.logging import InvokeAILogger
|
||||
@@ -109,6 +115,7 @@ class ApiDependencies:
|
||||
raise ValueError("Output folder is not set")
|
||||
|
||||
image_files = DiskImageFileStorage(f"{output_folder}/images")
|
||||
video_files = DiskVideoFileStorage(f"{output_folder}/videos")
|
||||
|
||||
model_images_folder = config.models_path
|
||||
style_presets_folder = config.style_presets_path
|
||||
@@ -134,6 +141,10 @@ class ApiDependencies:
|
||||
image_records = SqliteImageRecordStorage(db=db)
|
||||
image_moves = ImageMoveService(db=db, image_files=image_files, config=configuration, logger=logger)
|
||||
images = ImageService()
|
||||
video_records = SqliteVideoRecordStorage(db=db)
|
||||
videos = VideoService()
|
||||
board_video_records = SqliteBoardVideoRecordStorage(db=db)
|
||||
gallery = SqliteGalleryService(db=db)
|
||||
invocation_cache = MemoryInvocationCache(max_cache_size=config.node_cache_size)
|
||||
tensors = ObjectSerializerForwardCache(
|
||||
ObjectSerializerDisk[torch.Tensor](
|
||||
@@ -156,6 +167,7 @@ class ApiDependencies:
|
||||
Ideogram4ConditioningInfo,
|
||||
QwenImageConditioningInfo,
|
||||
AnimaConditioningInfo,
|
||||
WanConditioningInfo,
|
||||
],
|
||||
ephemeral=True,
|
||||
),
|
||||
@@ -226,6 +238,11 @@ class ApiDependencies:
|
||||
workflow_thumbnails=workflow_thumbnails,
|
||||
client_state_persistence=client_state_persistence,
|
||||
users=users,
|
||||
videos=videos,
|
||||
video_files=video_files,
|
||||
video_records=video_records,
|
||||
board_video_records=board_video_records,
|
||||
gallery=gallery,
|
||||
)
|
||||
|
||||
ApiDependencies.invoker = Invoker(services)
|
||||
|
||||
@@ -5,12 +5,23 @@ import string
|
||||
from datetime import timedelta
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import APIRouter, Body, HTTPException, Path, status
|
||||
from fastapi import APIRouter, Body, Depends, HTTPException, Path, Request, Response, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from invokeai.app.api.auth_dependencies import AdminUser, CurrentUser
|
||||
from invokeai.app.api.auth_dependencies import (
|
||||
MEDIA_TOKEN_COOKIE,
|
||||
AdminUser,
|
||||
CurrentUser,
|
||||
CurrentUserOrDefault,
|
||||
security,
|
||||
)
|
||||
from invokeai.app.api.dependencies import ApiDependencies
|
||||
from invokeai.app.services.auth.token_service import TokenData, create_access_token
|
||||
from invokeai.app.services.auth.token_service import (
|
||||
TokenData,
|
||||
create_access_token,
|
||||
get_token_remaining_seconds,
|
||||
)
|
||||
from invokeai.app.services.users.users_common import (
|
||||
UserCreateRequest,
|
||||
UserDTO,
|
||||
@@ -74,6 +85,40 @@ class LogoutResponse(BaseModel):
|
||||
success: bool = Field(description="Whether logout was successful")
|
||||
|
||||
|
||||
class MediaCookieResponse(BaseModel):
|
||||
"""Response from refreshing the media cookie."""
|
||||
|
||||
success: bool = Field(description="Whether the media cookie was set (always true in single-user mode)")
|
||||
|
||||
|
||||
def _media_cookie_path(request: Request) -> str:
|
||||
"""Public path prefix the media cookie is scoped to.
|
||||
|
||||
Behind a sub-path reverse proxy the browser sees routes under e.g. `/invoke/api/v1`,
|
||||
so the cookie path must include the proxy prefix (advertised via `root_path` by
|
||||
SubPathASGIMiddleware) or browsers will omit the cookie from media requests.
|
||||
"""
|
||||
return request.scope.get("root_path", "") + "/api/v1"
|
||||
|
||||
|
||||
def _set_media_cookie(request: Request, response: Response, token: str, max_age_seconds: int) -> None:
|
||||
"""Set the HttpOnly cookie that authenticates <video>/<img> media requests.
|
||||
|
||||
Media elements can't send Authorization headers, so the image and video media routes
|
||||
accept the JWT via this path-scoped cookie instead. Shared by login and
|
||||
/media-cookie so the cookie attributes can't drift between the two.
|
||||
"""
|
||||
response.set_cookie(
|
||||
MEDIA_TOKEN_COOKIE,
|
||||
token,
|
||||
max_age=max_age_seconds,
|
||||
httponly=True,
|
||||
secure=request.url.scheme == "https",
|
||||
samesite="lax",
|
||||
path=_media_cookie_path(request),
|
||||
)
|
||||
|
||||
|
||||
class SetupStatusResponse(BaseModel):
|
||||
"""Response for setup status check."""
|
||||
|
||||
@@ -119,7 +164,9 @@ async def get_setup_status() -> SetupStatusResponse:
|
||||
|
||||
@auth_router.post("/login", response_model=LoginResponse)
|
||||
async def login(
|
||||
request: Annotated[LoginRequest, Body(description="Login credentials")],
|
||||
login_request: Annotated[LoginRequest, Body(description="Login credentials")],
|
||||
request: Request,
|
||||
response: Response,
|
||||
) -> LoginResponse:
|
||||
"""Authenticate user and return access token.
|
||||
|
||||
@@ -143,7 +190,7 @@ async def login(
|
||||
)
|
||||
|
||||
user_service = ApiDependencies.invoker.services.users
|
||||
user = user_service.authenticate(request.email, request.password)
|
||||
user = user_service.authenticate(login_request.email, login_request.password)
|
||||
|
||||
if user is None:
|
||||
raise HTTPException(
|
||||
@@ -156,14 +203,17 @@ async def login(
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="User account is disabled")
|
||||
|
||||
# Create token with appropriate expiration
|
||||
expires_delta = timedelta(days=TOKEN_EXPIRATION_REMEMBER_ME if request.remember_me else TOKEN_EXPIRATION_NORMAL)
|
||||
expires_delta = timedelta(
|
||||
days=TOKEN_EXPIRATION_REMEMBER_ME if login_request.remember_me else TOKEN_EXPIRATION_NORMAL
|
||||
)
|
||||
token_data = TokenData(
|
||||
user_id=user.user_id,
|
||||
email=user.email,
|
||||
is_admin=user.is_admin,
|
||||
remember_me=request.remember_me,
|
||||
remember_me=login_request.remember_me,
|
||||
)
|
||||
token = create_access_token(token_data, expires_delta)
|
||||
_set_media_cookie(request, response, token, int(expires_delta.total_seconds()))
|
||||
|
||||
return LoginResponse(
|
||||
token=token,
|
||||
@@ -175,6 +225,8 @@ async def login(
|
||||
@auth_router.post("/logout", response_model=LogoutResponse)
|
||||
async def logout(
|
||||
current_user: CurrentUser,
|
||||
request: Request,
|
||||
response: Response,
|
||||
) -> LogoutResponse:
|
||||
"""Logout current user.
|
||||
|
||||
@@ -193,9 +245,57 @@ async def logout(
|
||||
"""
|
||||
# TODO: Implement token invalidation when server-side session management is added
|
||||
# For now, this is a no-op since we use stateless JWT tokens
|
||||
response.delete_cookie(MEDIA_TOKEN_COOKIE, path=_media_cookie_path(request))
|
||||
return LogoutResponse(success=True)
|
||||
|
||||
|
||||
@auth_router.post("/media-cookie", response_model=MediaCookieResponse)
|
||||
async def refresh_media_cookie(
|
||||
request: Request,
|
||||
response: Response,
|
||||
_current_user: CurrentUserOrDefault,
|
||||
credentials: Annotated[HTTPAuthorizationCredentials | None, Depends(security)],
|
||||
) -> MediaCookieResponse:
|
||||
"""Re-issue the media cookie from a valid Bearer token.
|
||||
|
||||
The media cookie is normally set at login, but a session can hold a valid JWT
|
||||
without it — the session may predate the cookie's introduction, or the cookie
|
||||
may have been cleared while the JWT (in client storage) survived. Media
|
||||
elements can't send Authorization headers, so such sessions silently fail to
|
||||
load videos (black player) while every other API call works. The frontend
|
||||
calls this on app load so an existing session self-heals without re-login.
|
||||
|
||||
The cookie's lifetime is clamped to the presented token's remaining validity —
|
||||
it is the same JWT, so a longer-lived cookie would just yield 401s after
|
||||
expiry anyway.
|
||||
|
||||
Returns:
|
||||
MediaCookieResponse indicating the cookie was set. In single-user mode the
|
||||
media routes don't require authentication, so this is a successful no-op.
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if the Bearer token is missing, invalid, or expired, or
|
||||
the user no longer exists or is inactive (raised by the auth dependency).
|
||||
"""
|
||||
config = ApiDependencies.invoker.services.configuration
|
||||
if not config.multiuser:
|
||||
return MediaCookieResponse(success=True)
|
||||
|
||||
# CurrentUserOrDefault has already validated the Bearer token (signature, expiry,
|
||||
# user exists and is active) — in multiuser mode it 401s otherwise, so credentials
|
||||
# cannot be None here. The raw token is still needed as the cookie value.
|
||||
if credentials is None:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Authentication required")
|
||||
|
||||
token = credentials.credentials
|
||||
remaining = get_token_remaining_seconds(token)
|
||||
if remaining is None:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid or expired token")
|
||||
|
||||
_set_media_cookie(request, response, token, remaining)
|
||||
return MediaCookieResponse(success=True)
|
||||
|
||||
|
||||
@auth_router.get("/me", response_model=UserDTO)
|
||||
async def get_current_user_info(
|
||||
current_user: CurrentUser,
|
||||
|
||||
@@ -22,6 +22,22 @@ class DeleteBoardResult(BaseModel):
|
||||
description="The image names of the board-images relationships that were deleted."
|
||||
)
|
||||
deleted_images: list[str] = Field(description="The names of the images that were deleted.")
|
||||
deleted_board_videos: list[str] = Field(
|
||||
default_factory=list,
|
||||
description="The video names of the board-videos relationships that were deleted.",
|
||||
)
|
||||
deleted_videos: list[str] = Field(
|
||||
default_factory=list,
|
||||
description="The names of the videos that were deleted.",
|
||||
)
|
||||
failed_images: list[str] = Field(
|
||||
default_factory=list,
|
||||
description="The names of images that could not be deleted and became uncategorized.",
|
||||
)
|
||||
failed_videos: list[str] = Field(
|
||||
default_factory=list,
|
||||
description="The names of videos that could not be deleted and became uncategorized.",
|
||||
)
|
||||
|
||||
|
||||
@boards_router.post(
|
||||
@@ -103,10 +119,12 @@ async def update_board(
|
||||
|
||||
|
||||
@boards_router.delete("/{board_id}", operation_id="delete_board", response_model=DeleteBoardResult)
|
||||
async def delete_board(
|
||||
def delete_board(
|
||||
current_user: CurrentUserOrDefault,
|
||||
board_id: str = Path(description="The id of board to delete"),
|
||||
include_images: Optional[bool] = Query(description="Permanently delete all images on the board", default=False),
|
||||
include_images: Optional[bool] = Query(
|
||||
description="Permanently delete all images and videos on the board", default=False
|
||||
),
|
||||
) -> DeleteBoardResult:
|
||||
"""Deletes a board (user must have access to it)"""
|
||||
try:
|
||||
@@ -117,20 +135,37 @@ async def delete_board(
|
||||
if not current_user.is_admin and board.user_id != current_user.user_id:
|
||||
raise HTTPException(status_code=403, detail="Not authorized to delete this board")
|
||||
|
||||
# Admins delete everything on the board; regular owners only delete their own
|
||||
# contributions so that contributions from other users to a public/shared board
|
||||
# are preserved (they cascade to "uncategorized" via FK on board_videos / board_images).
|
||||
cascade_user_id: Optional[str] = None if current_user.is_admin else current_user.user_id
|
||||
deleted_images: list[str] = []
|
||||
deleted_videos: list[str] = []
|
||||
|
||||
try:
|
||||
if include_images is True:
|
||||
assert_image_move_maintenance_inactive()
|
||||
deleted_images = ApiDependencies.invoker.services.board_images.get_all_board_image_names_for_board(
|
||||
board_id=board_id,
|
||||
categories=None,
|
||||
is_intermediate=None,
|
||||
# The services report both outcomes: records whose file delete failed are
|
||||
# preserved (they cascade to "uncategorized" via the board FKs when the
|
||||
# board is deleted below) and returned as failures. This is the ground
|
||||
# truth — reconstructing failures by diffing a router-side board listing
|
||||
# against the deleted names would double the DB work and misreport items
|
||||
# moved or deleted concurrently between the two queries.
|
||||
deleted_images, failed_images = ApiDependencies.invoker.services.images.delete_images_on_board(
|
||||
board_id=board_id, user_id=cascade_user_id
|
||||
)
|
||||
deleted_videos, failed_videos = ApiDependencies.invoker.services.videos.delete_videos_on_board(
|
||||
board_id=board_id, user_id=cascade_user_id
|
||||
)
|
||||
ApiDependencies.invoker.services.images.delete_images_on_board(board_id=board_id)
|
||||
ApiDependencies.invoker.services.boards.delete(board_id=board_id)
|
||||
return DeleteBoardResult(
|
||||
board_id=board_id,
|
||||
deleted_board_images=[],
|
||||
deleted_images=deleted_images,
|
||||
deleted_board_videos=[],
|
||||
deleted_videos=deleted_videos,
|
||||
failed_images=failed_images,
|
||||
failed_videos=failed_videos,
|
||||
)
|
||||
else:
|
||||
deleted_board_images = ApiDependencies.invoker.services.board_images.get_all_board_image_names_for_board(
|
||||
@@ -138,15 +173,34 @@ async def delete_board(
|
||||
categories=None,
|
||||
is_intermediate=None,
|
||||
)
|
||||
deleted_board_videos = (
|
||||
ApiDependencies.invoker.services.board_video_records.get_all_board_video_names_for_board(
|
||||
board_id=board_id,
|
||||
categories=None,
|
||||
is_intermediate=None,
|
||||
)
|
||||
)
|
||||
ApiDependencies.invoker.services.boards.delete(board_id=board_id)
|
||||
return DeleteBoardResult(
|
||||
board_id=board_id,
|
||||
deleted_board_images=deleted_board_images,
|
||||
deleted_images=[],
|
||||
deleted_board_videos=deleted_board_videos,
|
||||
deleted_videos=[],
|
||||
)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception:
|
||||
if include_images is True:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail={
|
||||
"message": "Failed to delete board after partially deleting media",
|
||||
"deleted_images": deleted_images,
|
||||
"deleted_videos": deleted_videos,
|
||||
"board_deleted": False,
|
||||
},
|
||||
)
|
||||
raise HTTPException(status_code=500, detail="Failed to delete board")
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import HTTPException, Query
|
||||
from fastapi.routing import APIRouter
|
||||
|
||||
from invokeai.app.api.auth_dependencies import CurrentUserOrDefault
|
||||
from invokeai.app.api.dependencies import ApiDependencies
|
||||
from invokeai.app.api.routers.images import _assert_board_read_access
|
||||
from invokeai.app.services.gallery.gallery_common import GalleryItem, GalleryItemNamesResult
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
|
||||
from invokeai.app.services.shared.pagination import MAX_PAGE_SIZE, OffsetPaginatedResults
|
||||
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
|
||||
|
||||
gallery_router = APIRouter(prefix="/v1/gallery", tags=["gallery"])
|
||||
|
||||
|
||||
@gallery_router.get(
|
||||
"/items/",
|
||||
operation_id="list_gallery_items",
|
||||
response_model=OffsetPaginatedResults[GalleryItem],
|
||||
)
|
||||
async def list_gallery_items(
|
||||
current_user: CurrentUserOrDefault,
|
||||
origin: Optional[ResourceOrigin] = Query(default=None, description="The origin of items to list."),
|
||||
categories: Optional[list[ImageCategory]] = Query(
|
||||
default=None,
|
||||
description="The categories to include. Shared between images and videos.",
|
||||
),
|
||||
is_intermediate: Optional[bool] = Query(default=None, description="Whether to list intermediate items."),
|
||||
board_id: Optional[str] = Query(
|
||||
default=None,
|
||||
description="The board id to filter by. Use 'none' to find items without a board.",
|
||||
),
|
||||
# Bounds matter: these flow verbatim into SQL, and a negative LIMIT means
|
||||
# *unlimited* in SQLite — one request would materialize the entire gallery.
|
||||
offset: int = Query(default=0, ge=0, description="The page offset"),
|
||||
limit: int = Query(default=10, ge=0, le=MAX_PAGE_SIZE, description="The number of items per page"),
|
||||
order_dir: SQLiteDirection = Query(default=SQLiteDirection.Descending, description="The order of sort"),
|
||||
starred_first: bool = Query(default=True, description="Whether to sort by starred items first"),
|
||||
search_term: Optional[str] = Query(default=None, description="The term to search for"),
|
||||
) -> OffsetPaginatedResults[GalleryItem]:
|
||||
"""Returns a paginated, time-sorted stream of polymorphic gallery items (images + videos)."""
|
||||
if board_id is not None and board_id != "none":
|
||||
_assert_board_read_access(board_id, current_user)
|
||||
|
||||
return ApiDependencies.invoker.services.gallery.list_items(
|
||||
offset=offset,
|
||||
limit=limit,
|
||||
starred_first=starred_first,
|
||||
order_dir=order_dir,
|
||||
origin=origin,
|
||||
categories=categories,
|
||||
is_intermediate=is_intermediate,
|
||||
board_id=board_id,
|
||||
search_term=search_term,
|
||||
user_id=current_user.user_id,
|
||||
is_admin=current_user.is_admin,
|
||||
)
|
||||
|
||||
|
||||
@gallery_router.get(
|
||||
"/items/names",
|
||||
operation_id="get_gallery_item_names",
|
||||
response_model=GalleryItemNamesResult,
|
||||
)
|
||||
async def get_gallery_item_names(
|
||||
current_user: CurrentUserOrDefault,
|
||||
origin: Optional[ResourceOrigin] = Query(default=None, description="The origin of items to list."),
|
||||
categories: Optional[list[ImageCategory]] = Query(
|
||||
default=None,
|
||||
description="The categories to include. Shared between images and videos.",
|
||||
),
|
||||
is_intermediate: Optional[bool] = Query(default=None, description="Whether to list intermediate items."),
|
||||
board_id: Optional[str] = Query(
|
||||
default=None,
|
||||
description="The board id to filter by. Use 'none' to find items without a board.",
|
||||
),
|
||||
order_dir: SQLiteDirection = Query(default=SQLiteDirection.Descending, description="The order of sort"),
|
||||
starred_first: bool = Query(default=True, description="Whether to sort by starred items first"),
|
||||
search_term: Optional[str] = Query(default=None, description="The term to search for"),
|
||||
) -> GalleryItemNamesResult:
|
||||
"""Returns an ordered (kind, name) list — used to drive virtualized gallery selection."""
|
||||
if board_id is not None and board_id != "none":
|
||||
_assert_board_read_access(board_id, current_user)
|
||||
|
||||
try:
|
||||
return ApiDependencies.invoker.services.gallery.list_item_names(
|
||||
starred_first=starred_first,
|
||||
order_dir=order_dir,
|
||||
origin=origin,
|
||||
categories=categories,
|
||||
is_intermediate=is_intermediate,
|
||||
board_id=board_id,
|
||||
search_term=search_term,
|
||||
user_id=current_user.user_id,
|
||||
is_admin=current_user.is_admin,
|
||||
)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=500, detail="Failed to get gallery item names")
|
||||
@@ -9,7 +9,7 @@ from fastapi.routing import APIRouter
|
||||
from PIL import Image
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from invokeai.app.api.auth_dependencies import CurrentUserOrDefault
|
||||
from invokeai.app.api.auth_dependencies import CurrentMediaUserOrDefault, CurrentUserOrDefault
|
||||
from invokeai.app.api.dependencies import ApiDependencies
|
||||
from invokeai.app.api.extract_metadata_from_image import extract_metadata_from_image
|
||||
from invokeai.app.api.routers._access import (
|
||||
@@ -48,6 +48,12 @@ images_router = APIRouter(prefix="/v1/images", tags=["images"])
|
||||
IMAGE_MAX_AGE = 31536000
|
||||
|
||||
|
||||
def _get_image_cache_control() -> str:
|
||||
if ApiDependencies.invoker.services.configuration.multiuser:
|
||||
return "private, no-store"
|
||||
return f"max-age={IMAGE_MAX_AGE}"
|
||||
|
||||
|
||||
class ResizeToDimensions(BaseModel):
|
||||
width: int = Field(..., gt=0)
|
||||
height: int = Field(..., gt=0)
|
||||
@@ -353,15 +359,15 @@ async def get_image_workflow(
|
||||
},
|
||||
)
|
||||
async def get_image_full(
|
||||
current_user: CurrentMediaUserOrDefault,
|
||||
image_name: str = Path(description="The name of full-resolution image file to get"),
|
||||
) -> Response:
|
||||
"""Gets a full-resolution image file.
|
||||
|
||||
This endpoint is intentionally unauthenticated because browsers load images
|
||||
via <img src> tags which cannot send Bearer tokens. Image names are UUIDs,
|
||||
providing security through unguessability. Returns 409 while image storage
|
||||
maintenance is active.
|
||||
Browser media requests authenticate with the path-scoped HttpOnly cookie set at login.
|
||||
Returns 409 while image storage maintenance is active.
|
||||
"""
|
||||
_assert_image_read_access(image_name, current_user)
|
||||
assert_image_move_maintenance_inactive()
|
||||
|
||||
try:
|
||||
@@ -369,7 +375,7 @@ async def get_image_full(
|
||||
with open(path, "rb") as f:
|
||||
content = f.read()
|
||||
response = Response(content, media_type="image/png")
|
||||
response.headers["Cache-Control"] = f"max-age={IMAGE_MAX_AGE}"
|
||||
response.headers["Cache-Control"] = _get_image_cache_control()
|
||||
response.headers["Content-Disposition"] = f'inline; filename="{image_name}"'
|
||||
return response
|
||||
except Exception:
|
||||
@@ -389,15 +395,15 @@ async def get_image_full(
|
||||
},
|
||||
)
|
||||
async def get_image_thumbnail(
|
||||
current_user: CurrentMediaUserOrDefault,
|
||||
image_name: str = Path(description="The name of thumbnail image file to get"),
|
||||
) -> Response:
|
||||
"""Gets a thumbnail image file.
|
||||
|
||||
This endpoint is intentionally unauthenticated because browsers load images
|
||||
via <img src> tags which cannot send Bearer tokens. Image names are UUIDs,
|
||||
providing security through unguessability. Returns 409 while image storage
|
||||
maintenance is active.
|
||||
Browser media requests authenticate with the path-scoped HttpOnly cookie set at login.
|
||||
Returns 409 while image storage maintenance is active.
|
||||
"""
|
||||
_assert_image_read_access(image_name, current_user)
|
||||
assert_image_move_maintenance_inactive()
|
||||
|
||||
try:
|
||||
@@ -405,7 +411,7 @@ async def get_image_thumbnail(
|
||||
with open(path, "rb") as f:
|
||||
content = f.read()
|
||||
response = Response(content, media_type="image/webp")
|
||||
response.headers["Cache-Control"] = f"max-age={IMAGE_MAX_AGE}"
|
||||
response.headers["Cache-Control"] = _get_image_cache_control()
|
||||
return response
|
||||
except Exception:
|
||||
raise HTTPException(status_code=404)
|
||||
@@ -492,9 +498,17 @@ async def delete_images_from_list(
|
||||
raise
|
||||
|
||||
try:
|
||||
# Skip — but do not re-raise — auth failures so a foreign name mid-batch doesn't
|
||||
# discard the response payload for items the caller had already legitimately deleted.
|
||||
# Without this, the client cache never learns about the partial successes and the
|
||||
# already-deleted records reappear in the UI until the next full refresh.
|
||||
deleted_images: set[str] = set()
|
||||
failed_images: set[str] = set()
|
||||
affected_boards: set[str] = set()
|
||||
for image_name in image_names:
|
||||
# Dedup while preserving order: a name repeated in the request would otherwise
|
||||
# be processed twice, and the second pass's not-found error would land the same
|
||||
# name in both deleted_images and failed_images.
|
||||
for image_name in dict.fromkeys(image_names):
|
||||
try:
|
||||
_assert_image_owner(image_name, current_user)
|
||||
image_dto = ApiDependencies.invoker.services.images.get_dto(image_name)
|
||||
@@ -503,11 +517,14 @@ async def delete_images_from_list(
|
||||
deleted_images.add(image_name)
|
||||
affected_boards.add(board_id)
|
||||
except HTTPException:
|
||||
raise
|
||||
continue
|
||||
except Exception:
|
||||
pass
|
||||
# A genuine deletion failure (not an auth/404 skip) — report it so the
|
||||
# client can surface a partial-failure warning, matching the video path.
|
||||
failed_images.add(image_name)
|
||||
return DeleteImagesResult(
|
||||
deleted_images=list(deleted_images),
|
||||
failed_images=list(failed_images),
|
||||
affected_boards=list(affected_boards),
|
||||
)
|
||||
except HTTPException:
|
||||
@@ -529,6 +546,7 @@ async def delete_uncategorized_images(
|
||||
|
||||
try:
|
||||
deleted_images: set[str] = set()
|
||||
failed_images: set[str] = set()
|
||||
affected_boards: set[str] = set()
|
||||
for image_name in image_names:
|
||||
try:
|
||||
@@ -540,9 +558,10 @@ async def delete_uncategorized_images(
|
||||
# Skip images not owned by the current user
|
||||
pass
|
||||
except Exception:
|
||||
pass
|
||||
failed_images.add(image_name)
|
||||
return DeleteImagesResult(
|
||||
deleted_images=list(deleted_images),
|
||||
failed_images=list(failed_images),
|
||||
affected_boards=list(affected_boards),
|
||||
)
|
||||
except Exception:
|
||||
|
||||
@@ -0,0 +1,904 @@
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
import traceback
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Annotated, BinaryIO, Optional
|
||||
|
||||
from fastapi import Body, HTTPException, Query, Request, Response, UploadFile
|
||||
from fastapi import Path as PathParam
|
||||
from fastapi.responses import StreamingResponse
|
||||
from fastapi.routing import APIRouter
|
||||
from PIL import Image as PILImage
|
||||
from pydantic import BaseModel, Field, StringConstraints, ValidationError
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
|
||||
from invokeai.app.api.auth_dependencies import CurrentMediaUserOrDefault, CurrentUserOrDefault
|
||||
from invokeai.app.api.dependencies import ApiDependencies
|
||||
from invokeai.app.api.routers.images import WorkflowAndGraphResponse, _assert_board_read_access
|
||||
from invokeai.app.invocations.fields import MetadataField, MetadataFieldValidator
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
|
||||
from invokeai.app.services.shared.pagination import MAX_PAGE_SIZE, OffsetPaginatedResults
|
||||
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
|
||||
from invokeai.app.services.video_records.video_records_common import VideoNamesResult, VideoRecordChanges
|
||||
from invokeai.app.services.videos.videos_common import (
|
||||
AddVideosToBoardResult,
|
||||
DeleteVideosResult,
|
||||
RemoveVideosFromBoardResult,
|
||||
StarredVideosResult,
|
||||
UnstarredVideosResult,
|
||||
VideoDTO,
|
||||
VideoUrlsDTO,
|
||||
)
|
||||
from invokeai.app.util.video_thumbnails import VideoDecodeTimeoutError, extract_video_frame, probe_video_with_codec
|
||||
|
||||
videos_router = APIRouter(prefix="/v1/videos", tags=["videos"])
|
||||
|
||||
# Videos are immutable; set a high max-age (1 year)
|
||||
VIDEO_MAX_AGE = 31536000
|
||||
|
||||
# MP4 only — the names service emits `{uuid}.mp4` unconditionally and we don't transcode on
|
||||
# upload. Accepting .mov/.webm/.mkv here previously caused those containers to be stored
|
||||
# under a .mp4 name and served with the .mp4 MIME type, which silently broke playback in
|
||||
# browsers when the container did not match.
|
||||
ACCEPTED_VIDEO_MIME_PREFIXES = ("video/mp4",)
|
||||
ACCEPTED_VIDEO_EXTENSIONS = (".mp4",)
|
||||
|
||||
# Per-chunk size for HTTP Range responses (1 MB)
|
||||
RANGE_CHUNK_SIZE = 1024 * 1024
|
||||
|
||||
# Upload streaming chunk size (1 MB) and a coarse per-upload size cap. The cap is generous
|
||||
# because Wan-generated MP4s for long sequences can run into the hundreds of megabytes;
|
||||
# the goal is to prevent a single client from exhausting RAM, not to be a content policy.
|
||||
UPLOAD_CHUNK_SIZE = 1024 * 1024
|
||||
MAX_UPLOAD_SIZE = 1024 * 1024 * 1024 # 1 GB
|
||||
# Pre-parse ingress cap enforced by VideoUploadLimitASGIMiddleware, applied to the whole
|
||||
# request body *before* the multipart parser spools it to temp storage. Slightly larger
|
||||
# than MAX_UPLOAD_SIZE to allow for multipart framing and the metadata form field.
|
||||
MAX_UPLOAD_REQUEST_SIZE = MAX_UPLOAD_SIZE + 10 * 1024 * 1024
|
||||
# Global bound on concurrent video uploads — each in-flight upload can hold up to two
|
||||
# copies of the file in temp storage (the multipart spool + the route's own tmp file).
|
||||
MAX_CONCURRENT_VIDEO_UPLOADS = 2
|
||||
# Per-user bound (multiuser mode only): keeps one tenant's slow uploads from holding
|
||||
# every global slot and starving the other users into 429s.
|
||||
MAX_CONCURRENT_VIDEO_UPLOADS_PER_USER = 1
|
||||
MAX_VIDEO_BATCH_SIZE = 1000
|
||||
VideoName = Annotated[str, StringConstraints(max_length=255)]
|
||||
|
||||
|
||||
class VideoNamesBatch(BaseModel):
|
||||
video_names: list[VideoName] = Field(
|
||||
max_length=MAX_VIDEO_BATCH_SIZE,
|
||||
description="The list of video names to process",
|
||||
)
|
||||
|
||||
|
||||
def _get_video_cache_control() -> str:
|
||||
if ApiDependencies.invoker.services.configuration.multiuser:
|
||||
return "private, no-store"
|
||||
return f"max-age={VIDEO_MAX_AGE}"
|
||||
|
||||
|
||||
def _assert_video_owner(video_name: str, current_user: CurrentUserOrDefault) -> None:
|
||||
"""Raise 403 if the current user does not own the video and is not an admin."""
|
||||
from invokeai.app.services.board_records.board_records_common import BoardVisibility
|
||||
|
||||
if current_user.is_admin:
|
||||
return
|
||||
owner = ApiDependencies.invoker.services.video_records.get_user_id(video_name)
|
||||
if owner is not None and owner == current_user.user_id:
|
||||
return
|
||||
|
||||
board_id = ApiDependencies.invoker.services.board_video_records.get_board_for_video(video_name)
|
||||
if board_id is not None:
|
||||
try:
|
||||
board = ApiDependencies.invoker.services.boards.get_dto(board_id=board_id)
|
||||
if board.user_id == current_user.user_id:
|
||||
return
|
||||
if board.board_visibility == BoardVisibility.Public:
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
raise HTTPException(status_code=403, detail="Not authorized to modify this video")
|
||||
|
||||
|
||||
def _assert_video_direct_owner(video_name: str, current_user: CurrentUserOrDefault) -> None:
|
||||
"""Raise 403 if the current user is not the direct owner of the video.
|
||||
|
||||
Intentionally stricter than _assert_video_owner: board-ownership and public-board
|
||||
fallbacks are NOT honored. Mirrors _assert_image_direct_owner in board_images.py —
|
||||
board-move operations need to verify the *original* owner, otherwise a user could
|
||||
move someone else's video onto their own board via the board-owner branch.
|
||||
"""
|
||||
if current_user.is_admin:
|
||||
return
|
||||
owner = ApiDependencies.invoker.services.video_records.get_user_id(video_name)
|
||||
if owner is not None and owner == current_user.user_id:
|
||||
return
|
||||
raise HTTPException(status_code=403, detail="Not authorized to move this video")
|
||||
|
||||
|
||||
def _assert_board_write_access(board_id: str, current_user: CurrentUserOrDefault) -> None:
|
||||
"""Raise 403 if the current user may not mutate the given board.
|
||||
|
||||
Mirrors _assert_board_write_access in board_images.py: admins and the board owner
|
||||
may write; public boards accept contributions from any user.
|
||||
"""
|
||||
from invokeai.app.services.board_records.board_records_common import BoardVisibility
|
||||
|
||||
try:
|
||||
board = ApiDependencies.invoker.services.boards.get_dto(board_id=board_id)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=404, detail="Board not found")
|
||||
if current_user.is_admin:
|
||||
return
|
||||
if board.user_id == current_user.user_id:
|
||||
return
|
||||
if board.board_visibility == BoardVisibility.Public:
|
||||
return
|
||||
raise HTTPException(status_code=403, detail="Not authorized to modify this board")
|
||||
|
||||
|
||||
def _assert_video_read_access(video_name: str, current_user: CurrentUserOrDefault) -> None:
|
||||
"""Raise 403 if the current user may not view the video."""
|
||||
from invokeai.app.services.board_records.board_records_common import BoardVisibility
|
||||
|
||||
if current_user.is_admin:
|
||||
return
|
||||
owner = ApiDependencies.invoker.services.video_records.get_user_id(video_name)
|
||||
if owner is not None and owner == current_user.user_id:
|
||||
return
|
||||
|
||||
board_id = ApiDependencies.invoker.services.board_video_records.get_board_for_video(video_name)
|
||||
if board_id is not None:
|
||||
try:
|
||||
board = ApiDependencies.invoker.services.boards.get_dto(board_id=board_id)
|
||||
if board.board_visibility in (BoardVisibility.Shared, BoardVisibility.Public):
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
raise HTTPException(status_code=403, detail="Not authorized to access this video")
|
||||
|
||||
|
||||
def _is_accepted_video_upload(file: UploadFile) -> bool:
|
||||
if file.content_type and file.content_type.startswith(ACCEPTED_VIDEO_MIME_PREFIXES):
|
||||
return True
|
||||
if file.filename:
|
||||
return file.filename.lower().endswith(ACCEPTED_VIDEO_EXTENSIONS)
|
||||
return False
|
||||
|
||||
|
||||
def _is_mp4_file(path: Path) -> bool:
|
||||
try:
|
||||
with open(path, "rb") as video_file:
|
||||
search_limit = min(path.stat().st_size, 64 * 1024)
|
||||
position = 0
|
||||
while position + 8 <= search_limit:
|
||||
video_file.seek(position)
|
||||
header = video_file.read(8)
|
||||
box_size = int.from_bytes(header[:4], byteorder="big")
|
||||
box_type = header[4:8]
|
||||
header_size = 8
|
||||
if box_size == 1:
|
||||
extended_size = video_file.read(8)
|
||||
if len(extended_size) != 8:
|
||||
return False
|
||||
box_size = int.from_bytes(extended_size, byteorder="big")
|
||||
header_size = 16
|
||||
if box_size < header_size:
|
||||
return False
|
||||
if box_type == b"ftyp":
|
||||
major_brand = video_file.read(4)
|
||||
return len(major_brand) == 4 and major_brand != b"qt "
|
||||
position += box_size
|
||||
except OSError:
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def _probe_decodable_video(path: Path) -> tuple[tuple[int, int, float, Optional[float]], Optional[PILImage.Image]]:
|
||||
"""Probes metadata and proves the video has a decodable first frame.
|
||||
|
||||
Returns the metadata plus the decoded frame so the save path can reuse it as the
|
||||
thumbnail source instead of spawning another decode worker. A decode timeout is
|
||||
contention on a loaded server, not evidence the video is bad — probe_video already
|
||||
succeeded — so it yields (metadata, None) and the upload proceeds, with save-time
|
||||
thumbnail extraction as the backstop.
|
||||
"""
|
||||
width, height, duration, fps, codec = probe_video_with_codec(path)
|
||||
if codec is None or codec.lower() not in {"h264", "avc", "avc1", "libx264"}:
|
||||
raise ValueError("Video must use a browser-compatible H.264/AVC codec")
|
||||
metadata = (width, height, duration, fps)
|
||||
try:
|
||||
first_frame = extract_video_frame(path, frame_index=0, raise_on_timeout=True)
|
||||
except VideoDecodeTimeoutError:
|
||||
return metadata, None
|
||||
if first_frame is None:
|
||||
raise ValueError("Video has no decodable frame")
|
||||
return metadata, first_frame
|
||||
|
||||
|
||||
@videos_router.post(
|
||||
"/upload",
|
||||
operation_id="upload_video",
|
||||
responses={
|
||||
201: {"description": "The video was uploaded successfully"},
|
||||
415: {"description": "Video upload failed"},
|
||||
},
|
||||
status_code=201,
|
||||
response_model=VideoDTO,
|
||||
)
|
||||
async def upload_video(
|
||||
current_user: CurrentUserOrDefault,
|
||||
file: UploadFile,
|
||||
request: Request,
|
||||
response: Response,
|
||||
video_category: ImageCategory = Query(description="The category of the video"),
|
||||
is_intermediate: bool = Query(description="Whether this is an intermediate video"),
|
||||
board_id: Optional[str] = Query(default=None, description="The board to add this video to, if any"),
|
||||
session_id: Optional[str] = Query(default=None, description="The session ID associated with this upload, if any"),
|
||||
metadata: Optional[str] = Body(
|
||||
default=None,
|
||||
description="The metadata to associate with the video, must be a stringified JSON dict",
|
||||
embed=True,
|
||||
),
|
||||
) -> VideoDTO:
|
||||
"""Uploads a video for the current user."""
|
||||
if metadata is not None:
|
||||
try:
|
||||
MetadataFieldValidator.validate_json(metadata)
|
||||
except ValidationError as e:
|
||||
raise HTTPException(status_code=422, detail="Metadata must be a JSON object") from e
|
||||
|
||||
# Check board access for uploads to a specific board.
|
||||
if board_id is not None:
|
||||
from invokeai.app.services.board_records.board_records_common import BoardVisibility
|
||||
|
||||
try:
|
||||
board = ApiDependencies.invoker.services.boards.get_dto(board_id=board_id)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=404, detail="Board not found")
|
||||
if (
|
||||
not current_user.is_admin
|
||||
and board.user_id != current_user.user_id
|
||||
and board.board_visibility != BoardVisibility.Public
|
||||
):
|
||||
raise HTTPException(status_code=403, detail="Not authorized to upload to this board")
|
||||
|
||||
if not _is_accepted_video_upload(file):
|
||||
raise HTTPException(status_code=415, detail="Not a supported video file")
|
||||
|
||||
# Stream the upload to a tmp file so we can probe and then hand its path to the service.
|
||||
# Reading the full body into memory first risked exhausting RAM on multi-GB uploads;
|
||||
# chunk-stream instead and enforce a hard size cap. Filesystem writes, container
|
||||
# validation, ffmpeg probing, and thumbnail extraction are all blocking — run them in
|
||||
# the thread pool so a slow (or hostile) file can't stall the event loop and every
|
||||
# other API request with it.
|
||||
tmp = tempfile.NamedTemporaryFile(prefix="invokeai_upload_", suffix=".mp4", delete=False)
|
||||
tmp_path = Path(tmp.name)
|
||||
try:
|
||||
total = 0
|
||||
while chunk := await file.read(UPLOAD_CHUNK_SIZE):
|
||||
total += len(chunk)
|
||||
if total > MAX_UPLOAD_SIZE:
|
||||
tmp.close()
|
||||
raise HTTPException(
|
||||
status_code=413,
|
||||
detail=f"Video upload exceeds maximum size ({MAX_UPLOAD_SIZE} bytes)",
|
||||
)
|
||||
await run_in_threadpool(tmp.write, chunk)
|
||||
tmp.close()
|
||||
# Release the multipart spool now that the body is copied: each in-flight
|
||||
# upload otherwise holds TWO on-disk copies (Starlette's spool + our tmp file)
|
||||
# through the probe/thumbnail/create phase — up to 2 x MAX_UPLOAD_SIZE x
|
||||
# MAX_CONCURRENT_VIDEO_UPLOADS of temp disk. Closing shrinks the double-copy
|
||||
# window to the copy loop itself.
|
||||
await file.close()
|
||||
|
||||
if not await run_in_threadpool(_is_mp4_file, tmp_path):
|
||||
raise HTTPException(status_code=415, detail="Not an MP4 video file")
|
||||
|
||||
try:
|
||||
(width, height, duration, fps), first_frame = await run_in_threadpool(_probe_decodable_video, tmp_path)
|
||||
except Exception:
|
||||
ApiDependencies.invoker.services.logger.error(traceback.format_exc())
|
||||
raise HTTPException(status_code=415, detail="Failed to read video")
|
||||
|
||||
try:
|
||||
video_dto = await run_in_threadpool(
|
||||
lambda: ApiDependencies.invoker.services.videos.create(
|
||||
source_path=tmp_path,
|
||||
width=width,
|
||||
height=height,
|
||||
duration=duration,
|
||||
fps=fps,
|
||||
first_frame=first_frame,
|
||||
video_origin=ResourceOrigin.EXTERNAL,
|
||||
video_category=video_category,
|
||||
session_id=session_id,
|
||||
board_id=board_id,
|
||||
metadata=metadata,
|
||||
workflow=None,
|
||||
graph=None,
|
||||
is_intermediate=is_intermediate,
|
||||
user_id=current_user.user_id,
|
||||
)
|
||||
)
|
||||
|
||||
response.status_code = 201
|
||||
response.headers["Location"] = video_dto.video_url
|
||||
return video_dto
|
||||
except Exception:
|
||||
ApiDependencies.invoker.services.logger.error(traceback.format_exc())
|
||||
raise HTTPException(status_code=500, detail="Failed to create video")
|
||||
finally:
|
||||
# If create() succeeded the file was moved; this unlink is a no-op then.
|
||||
try:
|
||||
tmp.close()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
@videos_router.delete("/i/{video_name}", operation_id="delete_video", response_model=DeleteVideosResult)
|
||||
async def delete_video(
|
||||
current_user: CurrentUserOrDefault,
|
||||
video_name: str = PathParam(description="The name of the video to delete"),
|
||||
) -> DeleteVideosResult:
|
||||
_assert_video_owner(video_name, current_user)
|
||||
|
||||
# Let service-level failures surface as 500s rather than swallowing them and returning a
|
||||
# success-shaped response. A previous version of this handler caught everything and
|
||||
# returned an empty ``deleted_videos`` list with HTTP 200; the frontend treated that as
|
||||
# success, dropped the item from its cache, and the video stayed on disk — a silent
|
||||
# data-consistency failure that only became visible on the next page reload.
|
||||
try:
|
||||
video_dto = ApiDependencies.invoker.services.videos.get_dto(video_name)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=404, detail="Video not found")
|
||||
|
||||
board_id = video_dto.board_id or "none"
|
||||
try:
|
||||
ApiDependencies.invoker.services.videos.delete(video_name)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=500, detail="Failed to delete video")
|
||||
|
||||
return DeleteVideosResult(
|
||||
deleted_videos=[video_name],
|
||||
failed_videos=[],
|
||||
affected_boards=[board_id],
|
||||
)
|
||||
|
||||
|
||||
@videos_router.post("/delete", operation_id="delete_videos_from_list", response_model=DeleteVideosResult)
|
||||
def delete_videos_from_list(
|
||||
current_user: CurrentUserOrDefault,
|
||||
batch: VideoNamesBatch,
|
||||
) -> DeleteVideosResult:
|
||||
# Skip — but do not re-raise — auth failures so a foreign name mid-batch doesn't
|
||||
# discard the response payload for items the caller had already legitimately deleted.
|
||||
# Without this, the client cache never learns about the partial successes and the
|
||||
# already-deleted records reappear in the UI until the next full refresh.
|
||||
#
|
||||
# HTTPException here means an intentional skip (a foreign name, or a 404 because a
|
||||
# concurrent session deleted it first) — not a failed deletion. Reporting those in
|
||||
# failed_videos would toast a spurious "could not be deleted" warning; the images
|
||||
# endpoints skip them silently, and mixed selections must behave consistently.
|
||||
deleted_videos: set[str] = set()
|
||||
failed_videos: set[str] = set()
|
||||
affected_boards: set[str] = set()
|
||||
# Dedup while preserving order: a name repeated in the request would otherwise be
|
||||
# processed twice, and the second pass's not-found error would land the same name
|
||||
# in both deleted_videos and failed_videos.
|
||||
for video_name in dict.fromkeys(batch.video_names):
|
||||
try:
|
||||
_assert_video_owner(video_name, current_user)
|
||||
video_dto = ApiDependencies.invoker.services.videos.get_dto(video_name)
|
||||
board_id = video_dto.board_id or "none"
|
||||
ApiDependencies.invoker.services.videos.delete(video_name)
|
||||
deleted_videos.add(video_name)
|
||||
affected_boards.add(board_id)
|
||||
except HTTPException:
|
||||
continue
|
||||
except Exception:
|
||||
failed_videos.add(video_name)
|
||||
return DeleteVideosResult(
|
||||
deleted_videos=list(deleted_videos),
|
||||
failed_videos=list(failed_videos),
|
||||
affected_boards=list(affected_boards),
|
||||
)
|
||||
|
||||
|
||||
@videos_router.delete("/uncategorized", operation_id="delete_uncategorized_videos", response_model=DeleteVideosResult)
|
||||
def delete_uncategorized_videos(
|
||||
current_user: CurrentUserOrDefault,
|
||||
) -> DeleteVideosResult:
|
||||
"""Deletes all uncategorized videos owned by the current user (or all if admin).
|
||||
|
||||
Mirrors ``delete_uncategorized_images`` so the "Delete All Uncategorized
|
||||
Images/Videos" board action covers both media kinds.
|
||||
"""
|
||||
names_result = ApiDependencies.invoker.services.videos.get_video_names(
|
||||
board_id="none",
|
||||
user_id=current_user.user_id,
|
||||
is_admin=current_user.is_admin,
|
||||
)
|
||||
deleted_videos: set[str] = set()
|
||||
failed_videos: set[str] = set()
|
||||
affected_boards: set[str] = set()
|
||||
for video_name in names_result.video_names:
|
||||
try:
|
||||
_assert_video_owner(video_name, current_user)
|
||||
ApiDependencies.invoker.services.videos.delete(video_name)
|
||||
deleted_videos.add(video_name)
|
||||
affected_boards.add("none")
|
||||
except HTTPException:
|
||||
# Skip videos not owned by the current user — an intentional skip, not a
|
||||
# failed deletion, so it must not be reported (and toasted) as one.
|
||||
continue
|
||||
except Exception:
|
||||
failed_videos.add(video_name)
|
||||
return DeleteVideosResult(
|
||||
deleted_videos=list(deleted_videos),
|
||||
failed_videos=list(failed_videos),
|
||||
affected_boards=list(affected_boards),
|
||||
)
|
||||
|
||||
|
||||
@videos_router.patch("/i/{video_name}", operation_id="update_video", response_model=VideoDTO)
|
||||
async def update_video(
|
||||
current_user: CurrentUserOrDefault,
|
||||
video_name: str = PathParam(description="The name of the video to update"),
|
||||
video_changes: VideoRecordChanges = Body(description="The changes to apply to the video"),
|
||||
) -> VideoDTO:
|
||||
_assert_video_owner(video_name, current_user)
|
||||
try:
|
||||
return ApiDependencies.invoker.services.videos.update(video_name, video_changes)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail="Failed to update video")
|
||||
|
||||
|
||||
@videos_router.get("/i/{video_name}", operation_id="get_video_dto", response_model=VideoDTO)
|
||||
async def get_video_dto(
|
||||
current_user: CurrentUserOrDefault,
|
||||
video_name: str = PathParam(description="The name of video to get"),
|
||||
) -> VideoDTO:
|
||||
_assert_video_read_access(video_name, current_user)
|
||||
try:
|
||||
return ApiDependencies.invoker.services.videos.get_dto(video_name)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=404)
|
||||
|
||||
|
||||
@videos_router.get(
|
||||
"/i/{video_name}/metadata", operation_id="get_video_metadata", response_model=Optional[MetadataField]
|
||||
)
|
||||
async def get_video_metadata(
|
||||
current_user: CurrentUserOrDefault,
|
||||
video_name: str = PathParam(description="The name of video to get"),
|
||||
) -> Optional[MetadataField]:
|
||||
_assert_video_read_access(video_name, current_user)
|
||||
try:
|
||||
return ApiDependencies.invoker.services.videos.get_metadata(video_name)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=404)
|
||||
|
||||
|
||||
@videos_router.get(
|
||||
"/i/{video_name}/workflow", operation_id="get_video_workflow", response_model=WorkflowAndGraphResponse
|
||||
)
|
||||
async def get_video_workflow(
|
||||
current_user: CurrentUserOrDefault,
|
||||
video_name: str = PathParam(description="The name of video whose workflow to get"),
|
||||
) -> WorkflowAndGraphResponse:
|
||||
"""Gets the workflow and graph saved with a generated video (mirrors the image route)."""
|
||||
_assert_video_read_access(video_name, current_user)
|
||||
try:
|
||||
workflow = ApiDependencies.invoker.services.videos.get_workflow(video_name)
|
||||
graph = ApiDependencies.invoker.services.videos.get_graph(video_name)
|
||||
return WorkflowAndGraphResponse(workflow=workflow, graph=graph)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=404)
|
||||
|
||||
|
||||
def _parse_range_header(range_header: str, file_size: int) -> Optional[tuple[int, int]]:
|
||||
"""Parses an HTTP Range header of the form `bytes=START-END`. Returns inclusive (start, end)
|
||||
byte offsets, or None if the header is malformed or unsatisfiable."""
|
||||
match = re.match(r"^bytes=(\d*)-(\d*)$", range_header.strip())
|
||||
if match is None:
|
||||
return None
|
||||
if file_size <= 0:
|
||||
# No byte range is satisfiable against an empty file (a suffix range would
|
||||
# otherwise "satisfy" with the invalid pair (0, -1)).
|
||||
return None
|
||||
start_str, end_str = match.group(1), match.group(2)
|
||||
if start_str == "" and end_str == "":
|
||||
return None
|
||||
if start_str == "":
|
||||
# suffix range: last N bytes
|
||||
try:
|
||||
suffix_len = int(end_str)
|
||||
except ValueError:
|
||||
return None
|
||||
if suffix_len == 0:
|
||||
return None
|
||||
start = max(file_size - suffix_len, 0)
|
||||
end = file_size - 1
|
||||
else:
|
||||
try:
|
||||
start = int(start_str)
|
||||
except ValueError:
|
||||
return None
|
||||
if end_str == "":
|
||||
end = file_size - 1
|
||||
else:
|
||||
try:
|
||||
end = int(end_str)
|
||||
except ValueError:
|
||||
return None
|
||||
if start > end or start >= file_size:
|
||||
return None
|
||||
end = min(end, file_size - 1)
|
||||
return start, end
|
||||
|
||||
|
||||
@videos_router.get(
|
||||
"/i/{video_name}/full",
|
||||
operation_id="get_video_full",
|
||||
response_class=Response,
|
||||
responses={
|
||||
200: {"description": "Return the full video file", "content": {"video/mp4": {}}},
|
||||
206: {"description": "Return a byte-range of the video file", "content": {"video/mp4": {}}},
|
||||
404: {"description": "Video not found"},
|
||||
},
|
||||
)
|
||||
@videos_router.head(
|
||||
"/i/{video_name}/full",
|
||||
operation_id="get_video_full_head",
|
||||
response_class=Response,
|
||||
responses={
|
||||
200: {"description": "Return the full video file", "content": {"video/mp4": {}}},
|
||||
404: {"description": "Video not found"},
|
||||
},
|
||||
)
|
||||
async def get_video_full(
|
||||
request: Request,
|
||||
current_user: CurrentMediaUserOrDefault,
|
||||
video_name: str = PathParam(description="The name of video file to get"),
|
||||
) -> Response:
|
||||
"""Serves the video file with HTTP Range support so HTML5 <video> seek/scrub works.
|
||||
|
||||
Browser media requests authenticate with the path-scoped HttpOnly cookie set at login.
|
||||
"""
|
||||
_assert_video_read_access(video_name, current_user)
|
||||
try:
|
||||
path_str = ApiDependencies.invoker.services.videos.get_path(video_name, thumbnail=False)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=404)
|
||||
|
||||
# Open once and serve every branch from the fd. Deletion stages files away via an
|
||||
# atomic rename, so any later path-based stat/open — including FileResponse's lazy
|
||||
# open after the route returns — races with a concurrent delete and surfaces as an
|
||||
# uncontrolled 500. An open fd is immune: the data stays readable until the handle
|
||||
# closes, even after the path is gone.
|
||||
video_file: Optional[BinaryIO] = None
|
||||
try:
|
||||
video_file = open(path_str, "rb")
|
||||
except OSError:
|
||||
raise HTTPException(status_code=404)
|
||||
|
||||
try:
|
||||
file_size = os.fstat(video_file.fileno()).st_size
|
||||
range_header = request.headers.get("range") or request.headers.get("Range")
|
||||
|
||||
common_headers = {
|
||||
"Accept-Ranges": "bytes",
|
||||
"Cache-Control": _get_video_cache_control(),
|
||||
"Content-Disposition": f'inline; filename="{video_name}"',
|
||||
}
|
||||
|
||||
# HEAD: respond with metadata only.
|
||||
if request.method == "HEAD":
|
||||
return Response(
|
||||
status_code=200,
|
||||
media_type="video/mp4",
|
||||
headers={**common_headers, "Content-Length": str(file_size)},
|
||||
)
|
||||
|
||||
if range_header is None:
|
||||
# Stream from the already-open fd in bounded chunks rather than reading the
|
||||
# file into RAM — multi-GB MP4 downloads (clients without Range, CLI tools,
|
||||
# CDN edge fetches) would otherwise allocate a multi-GB bytes object. This
|
||||
# trades FileResponse's sendfile() for delete-race immunity.
|
||||
handle = video_file
|
||||
video_file = None # ownership moves to the streaming iterator
|
||||
|
||||
def iter_video() -> Iterator[bytes]:
|
||||
try:
|
||||
while chunk := handle.read(RANGE_CHUNK_SIZE):
|
||||
yield chunk
|
||||
finally:
|
||||
handle.close()
|
||||
|
||||
return StreamingResponse(
|
||||
iter_video(),
|
||||
media_type="video/mp4",
|
||||
headers={**common_headers, "Content-Length": str(file_size)},
|
||||
)
|
||||
|
||||
parsed = _parse_range_header(range_header, file_size)
|
||||
if parsed is None:
|
||||
# Unsatisfiable range.
|
||||
return Response(
|
||||
status_code=416,
|
||||
headers={**common_headers, "Content-Range": f"bytes */{file_size}"},
|
||||
)
|
||||
start, end = parsed
|
||||
length = end - start + 1
|
||||
video_file.seek(start)
|
||||
# Read at most one chunk; clients ask for more via subsequent ranges.
|
||||
read_length = min(length, RANGE_CHUNK_SIZE)
|
||||
chunk = video_file.read(read_length)
|
||||
actual_end = start + len(chunk) - 1
|
||||
return Response(
|
||||
chunk,
|
||||
status_code=206,
|
||||
media_type="video/mp4",
|
||||
headers={
|
||||
**common_headers,
|
||||
"Content-Range": f"bytes {start}-{actual_end}/{file_size}",
|
||||
"Content-Length": str(len(chunk)),
|
||||
},
|
||||
)
|
||||
finally:
|
||||
if video_file is not None:
|
||||
video_file.close()
|
||||
|
||||
|
||||
@videos_router.get(
|
||||
"/i/{video_name}/thumbnail",
|
||||
operation_id="get_video_thumbnail",
|
||||
response_class=Response,
|
||||
responses={
|
||||
200: {"description": "Return the video thumbnail", "content": {"image/webp": {}}},
|
||||
404: {"description": "Video not found"},
|
||||
},
|
||||
)
|
||||
async def get_video_thumbnail(
|
||||
current_user: CurrentMediaUserOrDefault,
|
||||
video_name: str = PathParam(description="The name of thumbnail file to get"),
|
||||
) -> Response:
|
||||
"""Returns the first-frame WebP thumbnail of an authorized video."""
|
||||
_assert_video_read_access(video_name, current_user)
|
||||
try:
|
||||
path = ApiDependencies.invoker.services.videos.get_path(video_name, thumbnail=True)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=404)
|
||||
try:
|
||||
with open(path, "rb") as thumbnail_file:
|
||||
thumbnail = thumbnail_file.read()
|
||||
except OSError:
|
||||
raise HTTPException(status_code=404)
|
||||
|
||||
return Response(
|
||||
thumbnail,
|
||||
media_type="image/webp",
|
||||
headers={"Cache-Control": _get_video_cache_control()},
|
||||
)
|
||||
|
||||
|
||||
@videos_router.get("/i/{video_name}/urls", operation_id="get_video_urls", response_model=VideoUrlsDTO)
|
||||
async def get_video_urls(
|
||||
current_user: CurrentUserOrDefault,
|
||||
video_name: str = PathParam(description="The name of the video whose URL to get"),
|
||||
) -> VideoUrlsDTO:
|
||||
_assert_video_read_access(video_name, current_user)
|
||||
try:
|
||||
video_url = ApiDependencies.invoker.services.videos.get_url(video_name)
|
||||
thumbnail_url = ApiDependencies.invoker.services.videos.get_url(video_name, thumbnail=True)
|
||||
return VideoUrlsDTO(video_name=video_name, video_url=video_url, thumbnail_url=thumbnail_url)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=404)
|
||||
|
||||
|
||||
@videos_router.get("/", operation_id="list_video_dtos", response_model=OffsetPaginatedResults[VideoDTO])
|
||||
async def list_video_dtos(
|
||||
current_user: CurrentUserOrDefault,
|
||||
video_origin: Optional[ResourceOrigin] = Query(default=None, description="The origin of videos to list."),
|
||||
categories: Optional[list[ImageCategory]] = Query(default=None, description="The categories of video to include."),
|
||||
is_intermediate: Optional[bool] = Query(default=None, description="Whether to list intermediate videos."),
|
||||
board_id: Optional[str] = Query(
|
||||
default=None,
|
||||
description="The board id to filter by. Use 'none' to find videos without a board.",
|
||||
),
|
||||
# Bounds matter: these flow verbatim into SQL, and a negative LIMIT means
|
||||
# *unlimited* in SQLite — one request would materialize every video row.
|
||||
offset: int = Query(default=0, ge=0, description="The page offset"),
|
||||
limit: int = Query(default=10, ge=0, le=MAX_PAGE_SIZE, description="The number of videos per page"),
|
||||
order_dir: SQLiteDirection = Query(default=SQLiteDirection.Descending, description="The order of sort"),
|
||||
starred_first: bool = Query(default=True, description="Whether to sort by starred videos first"),
|
||||
search_term: Optional[str] = Query(default=None, description="The term to search for"),
|
||||
) -> OffsetPaginatedResults[VideoDTO]:
|
||||
"""Gets a list of video DTOs for the current user."""
|
||||
# Validate that the caller can read from this board. "none" is handled by the SQL layer.
|
||||
if board_id is not None and board_id != "none":
|
||||
_assert_board_read_access(board_id, current_user)
|
||||
|
||||
return ApiDependencies.invoker.services.videos.get_many(
|
||||
offset,
|
||||
limit,
|
||||
starred_first,
|
||||
order_dir,
|
||||
video_origin,
|
||||
categories,
|
||||
is_intermediate,
|
||||
board_id,
|
||||
search_term,
|
||||
current_user.user_id,
|
||||
current_user.is_admin,
|
||||
)
|
||||
|
||||
|
||||
@videos_router.get("/names", operation_id="get_video_names")
|
||||
async def get_video_names(
|
||||
current_user: CurrentUserOrDefault,
|
||||
video_origin: Optional[ResourceOrigin] = Query(default=None, description="The origin of videos to list."),
|
||||
categories: Optional[list[ImageCategory]] = Query(default=None, description="The categories of video to include."),
|
||||
is_intermediate: Optional[bool] = Query(default=None, description="Whether to list intermediate videos."),
|
||||
board_id: Optional[str] = Query(
|
||||
default=None,
|
||||
description="The board id to filter by. Use 'none' to find videos without a board.",
|
||||
),
|
||||
order_dir: SQLiteDirection = Query(default=SQLiteDirection.Descending, description="The order of sort"),
|
||||
starred_first: bool = Query(default=True, description="Whether to sort by starred videos first"),
|
||||
search_term: Optional[str] = Query(default=None, description="The term to search for"),
|
||||
) -> VideoNamesResult:
|
||||
"""Gets ordered list of video names with metadata for optimistic updates."""
|
||||
# Validate that the caller can read from this board. "none" is handled by the SQL layer.
|
||||
if board_id is not None and board_id != "none":
|
||||
_assert_board_read_access(board_id, current_user)
|
||||
|
||||
try:
|
||||
return ApiDependencies.invoker.services.videos.get_video_names(
|
||||
starred_first=starred_first,
|
||||
order_dir=order_dir,
|
||||
video_origin=video_origin,
|
||||
categories=categories,
|
||||
is_intermediate=is_intermediate,
|
||||
board_id=board_id,
|
||||
search_term=search_term,
|
||||
user_id=current_user.user_id,
|
||||
is_admin=current_user.is_admin,
|
||||
)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=500, detail="Failed to get video names")
|
||||
|
||||
|
||||
@videos_router.post("/star", operation_id="star_videos_in_list", response_model=StarredVideosResult)
|
||||
def star_videos_in_list(
|
||||
current_user: CurrentUserOrDefault,
|
||||
batch: VideoNamesBatch,
|
||||
) -> StarredVideosResult:
|
||||
# Skip — but do not re-raise — auth failures so a foreign name mid-batch doesn't
|
||||
# discard the response payload for items that were already starred. Mirrors
|
||||
# delete_videos_from_list: re-raising turned partial successes into an error-shaped
|
||||
# response, so the client never invalidated caches for the videos that did change.
|
||||
starred_videos: set[str] = set()
|
||||
failed_videos: set[str] = set()
|
||||
affected_boards: set[str] = set()
|
||||
for video_name in dict.fromkeys(batch.video_names):
|
||||
try:
|
||||
_assert_video_owner(video_name, current_user)
|
||||
updated = ApiDependencies.invoker.services.videos.update(
|
||||
video_name, changes=VideoRecordChanges(starred=True)
|
||||
)
|
||||
starred_videos.add(video_name)
|
||||
affected_boards.add(updated.board_id or "none")
|
||||
except HTTPException:
|
||||
continue
|
||||
except Exception:
|
||||
failed_videos.add(video_name)
|
||||
return StarredVideosResult(
|
||||
starred_videos=list(starred_videos),
|
||||
failed_videos=list(failed_videos),
|
||||
affected_boards=list(affected_boards),
|
||||
)
|
||||
|
||||
|
||||
@videos_router.post("/unstar", operation_id="unstar_videos_in_list", response_model=UnstarredVideosResult)
|
||||
def unstar_videos_in_list(
|
||||
current_user: CurrentUserOrDefault,
|
||||
batch: VideoNamesBatch,
|
||||
) -> UnstarredVideosResult:
|
||||
# See star_videos_in_list: skip foreign names instead of re-raising mid-batch.
|
||||
unstarred_videos: set[str] = set()
|
||||
failed_videos: set[str] = set()
|
||||
affected_boards: set[str] = set()
|
||||
for video_name in dict.fromkeys(batch.video_names):
|
||||
try:
|
||||
_assert_video_owner(video_name, current_user)
|
||||
updated = ApiDependencies.invoker.services.videos.update(
|
||||
video_name, changes=VideoRecordChanges(starred=False)
|
||||
)
|
||||
unstarred_videos.add(video_name)
|
||||
affected_boards.add(updated.board_id or "none")
|
||||
except HTTPException:
|
||||
continue
|
||||
except Exception:
|
||||
failed_videos.add(video_name)
|
||||
return UnstarredVideosResult(
|
||||
unstarred_videos=list(unstarred_videos),
|
||||
failed_videos=list(failed_videos),
|
||||
affected_boards=list(affected_boards),
|
||||
)
|
||||
|
||||
|
||||
class VideoBoardArg(BaseModel):
|
||||
board_id: str = Field(description="The id of the board to add or remove the video from")
|
||||
video_name: str = Field(description="The name of the video to add to / remove from the board")
|
||||
|
||||
|
||||
@videos_router.post(
|
||||
"/board",
|
||||
operation_id="add_video_to_board",
|
||||
response_model=AddVideosToBoardResult,
|
||||
)
|
||||
async def add_video_to_board(
|
||||
current_user: CurrentUserOrDefault,
|
||||
arg: VideoBoardArg = Body(),
|
||||
) -> AddVideosToBoardResult:
|
||||
_assert_board_write_access(arg.board_id, current_user)
|
||||
_assert_video_direct_owner(arg.video_name, current_user)
|
||||
try:
|
||||
# Capture the source board BEFORE mutating so the frontend can invalidate both
|
||||
# the old and new board caches. Mirrors add_image_to_board.
|
||||
old_board_id = (
|
||||
ApiDependencies.invoker.services.board_video_records.get_board_for_video(arg.video_name) or "none"
|
||||
)
|
||||
ApiDependencies.invoker.services.board_video_records.add_video_to_board(
|
||||
board_id=arg.board_id, video_name=arg.video_name
|
||||
)
|
||||
return AddVideosToBoardResult(
|
||||
added_videos=[arg.video_name],
|
||||
affected_boards=list({arg.board_id, old_board_id}),
|
||||
)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=500, detail="Failed to add video to board")
|
||||
|
||||
|
||||
@videos_router.delete(
|
||||
"/board",
|
||||
operation_id="remove_video_from_board",
|
||||
response_model=RemoveVideosFromBoardResult,
|
||||
)
|
||||
async def remove_video_from_board(
|
||||
current_user: CurrentUserOrDefault,
|
||||
video_name: str = Body(description="The name of the video to remove from its board", embed=True),
|
||||
) -> RemoveVideosFromBoardResult:
|
||||
# A video association can be removed by EITHER the direct video owner OR a user with
|
||||
# write access to the destination board (admin, board owner, or any contributor when the
|
||||
# board is Public). This mirrors remove_image_from_board and prevents a video from being
|
||||
# stranded when a non-owner uploads into a Public board that is later made Shared/Private:
|
||||
# without the board-write fallback, neither the uploader nor the board owner could
|
||||
# detach the video. See PR #9163 review.
|
||||
old_board_id = ApiDependencies.invoker.services.board_video_records.get_board_for_video(video_name)
|
||||
try:
|
||||
_assert_video_direct_owner(video_name, current_user)
|
||||
except HTTPException:
|
||||
if old_board_id is None:
|
||||
raise
|
||||
_assert_board_write_access(old_board_id, current_user)
|
||||
try:
|
||||
ApiDependencies.invoker.services.board_video_records.remove_video_from_board(video_name=video_name)
|
||||
return RemoveVideosFromBoardResult(
|
||||
removed_videos=[video_name],
|
||||
affected_boards=list({old_board_id or "none", "none"}),
|
||||
)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=500, detail="Failed to remove video from board")
|
||||
@@ -3,6 +3,7 @@ from fastapi.routing import APIRouter
|
||||
|
||||
from invokeai.app.api.auth_dependencies import CurrentUserOrDefault
|
||||
from invokeai.app.api.dependencies import ApiDependencies
|
||||
from invokeai.app.services.gallery.gallery_common import GalleryItemNamesResult
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory, ImageNamesResult
|
||||
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
|
||||
from invokeai.app.services.virtual_boards.virtual_boards_common import VirtualSubBoardDTO
|
||||
@@ -18,9 +19,9 @@ virtual_boards_router = APIRouter(prefix="/v1/virtual_boards", tags=["virtual_bo
|
||||
async def list_virtual_boards_by_date(
|
||||
current_user: CurrentUserOrDefault,
|
||||
) -> list[VirtualSubBoardDTO]:
|
||||
"""Gets a list of virtual sub-boards grouped by date."""
|
||||
"""Gets a list of virtual sub-boards grouped by date. Covers both images and videos."""
|
||||
try:
|
||||
return ApiDependencies.invoker.services.image_records.get_image_dates(
|
||||
return ApiDependencies.invoker.services.gallery.get_dates(
|
||||
user_id=current_user.user_id,
|
||||
is_admin=current_user.is_admin,
|
||||
)
|
||||
@@ -41,7 +42,8 @@ async def list_virtual_board_image_names_by_date(
|
||||
categories: list[ImageCategory] | None = Query(default=None, description="The categories of images to include"),
|
||||
search_term: str | None = Query(default=None, description="Search term to filter images"),
|
||||
) -> ImageNamesResult:
|
||||
"""Gets ordered image names for a specific date."""
|
||||
"""Gets ordered image names for a specific date. Image-only; kept for API compatibility —
|
||||
the UI uses the polymorphic `/by_date/{date}/item_names` endpoint."""
|
||||
try:
|
||||
return ApiDependencies.invoker.services.image_records.get_image_names_by_date(
|
||||
date=date,
|
||||
@@ -54,3 +56,32 @@ async def list_virtual_board_image_names_by_date(
|
||||
)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=500, detail="Failed to get image names for date")
|
||||
|
||||
|
||||
@virtual_boards_router.get(
|
||||
"/by_date/{date}/item_names",
|
||||
operation_id="list_virtual_board_item_names_by_date",
|
||||
response_model=GalleryItemNamesResult,
|
||||
)
|
||||
async def list_virtual_board_item_names_by_date(
|
||||
current_user: CurrentUserOrDefault,
|
||||
date: str = Path(description="The ISO date string, e.g. '2026-03-18'"),
|
||||
starred_first: bool = Query(default=True, description="Whether to sort starred items first"),
|
||||
order_dir: SQLiteDirection = Query(default=SQLiteDirection.Descending, description="The sort direction"),
|
||||
categories: list[ImageCategory] | None = Query(default=None, description="The categories of items to include"),
|
||||
search_term: str | None = Query(default=None, description="Search term to filter items"),
|
||||
) -> GalleryItemNamesResult:
|
||||
"""Gets ordered polymorphic (image + video) item refs for a specific date."""
|
||||
try:
|
||||
return ApiDependencies.invoker.services.gallery.list_item_names(
|
||||
starred_first=starred_first,
|
||||
order_dir=order_dir,
|
||||
categories=categories,
|
||||
is_intermediate=False,
|
||||
search_term=search_term,
|
||||
user_id=current_user.user_id,
|
||||
is_admin=current_user.is_admin,
|
||||
created_date=date,
|
||||
)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=500, detail="Failed to get gallery item names for date")
|
||||
|
||||
+195
-6
@@ -1,5 +1,7 @@
|
||||
import asyncio
|
||||
import inspect
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
|
||||
@@ -7,11 +9,14 @@ from fastapi import FastAPI, Request
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.middleware.gzip import GZipMiddleware
|
||||
from fastapi.openapi.docs import get_redoc_html, get_swagger_ui_html
|
||||
from fastapi.responses import HTMLResponse, RedirectResponse
|
||||
from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse
|
||||
from fastapi.security.utils import get_authorization_scheme_param
|
||||
from fastapi_events.handlers.local import local_handler
|
||||
from fastapi_events.middleware import EventHandlerASGIMiddleware
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
from starlette.datastructures import Headers
|
||||
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
|
||||
from starlette.types import ASGIApp, Receive, Scope, Send
|
||||
from starlette.types import ASGIApp, Message, Receive, Scope, Send
|
||||
|
||||
import invokeai.frontend.web as web_dir
|
||||
from invokeai.app.api.dependencies import ApiDependencies
|
||||
@@ -24,6 +29,7 @@ from invokeai.app.api.routers import (
|
||||
client_state,
|
||||
custom_nodes,
|
||||
download_queue,
|
||||
gallery,
|
||||
image_moves,
|
||||
images,
|
||||
model_manager,
|
||||
@@ -32,6 +38,7 @@ from invokeai.app.api.routers import (
|
||||
session_queue,
|
||||
style_presets,
|
||||
utilities,
|
||||
videos,
|
||||
virtual_boards,
|
||||
workflows,
|
||||
)
|
||||
@@ -99,18 +106,36 @@ class SlidingWindowTokenMiddleware(BaseHTTPMiddleware):
|
||||
# genuine user activity. GET requests are often background fetches (RTK Query
|
||||
# cache revalidation, refetch-on-focus, etc.) and should not reset the
|
||||
# inactivity timer.
|
||||
if response.status_code < 400 and request.method in ("POST", "PUT", "PATCH", "DELETE"):
|
||||
# Behind a sub-path proxy the public path carries the prefix (see SubPathASGIMiddleware),
|
||||
# so strip `root_path` before matching the auth-route exclusions — otherwise a proxied
|
||||
# logout would mint a replacement token/cookie right after the route deleted them.
|
||||
route_path = request.url.path.removeprefix(request.scope.get("root_path", ""))
|
||||
if (
|
||||
response.status_code < 400
|
||||
and request.method in ("POST", "PUT", "PATCH", "DELETE")
|
||||
and route_path not in ("/api/v1/auth/login", "/api/v1/auth/logout", "/api/v1/auth/media-cookie")
|
||||
):
|
||||
auth_header = request.headers.get("authorization", "")
|
||||
if auth_header.startswith("Bearer "):
|
||||
token = auth_header[7:]
|
||||
try:
|
||||
from datetime import timedelta
|
||||
|
||||
from invokeai.app.api.routers.auth import TOKEN_EXPIRATION_NORMAL, TOKEN_EXPIRATION_REMEMBER_ME
|
||||
from invokeai.app.services.auth.token_service import create_access_token, verify_token
|
||||
from invokeai.app.api.routers.auth import (
|
||||
TOKEN_EXPIRATION_NORMAL,
|
||||
TOKEN_EXPIRATION_REMEMBER_ME,
|
||||
)
|
||||
from invokeai.app.services.auth.token_service import TokenData, create_access_token, verify_token
|
||||
|
||||
token_data = verify_token(token)
|
||||
if token_data is not None:
|
||||
# The user lookup is a synchronous SQLite query behind a
|
||||
# process-wide lock; run it off the event loop so a contended
|
||||
# lock (e.g. generation-result writes) can't stall every
|
||||
# concurrent request from inside this per-mutation middleware.
|
||||
user = await run_in_threadpool(ApiDependencies.invoker.services.users.get, token_data.user_id)
|
||||
if user is None or not user.is_active:
|
||||
return response
|
||||
# Use the remember_me claim from the token to determine the
|
||||
# correct refresh duration. This avoids the bug where a 7-day
|
||||
# token with <24h remaining would be silently downgraded to 1 day.
|
||||
@@ -119,7 +144,13 @@ class SlidingWindowTokenMiddleware(BaseHTTPMiddleware):
|
||||
else:
|
||||
expires_delta = timedelta(days=TOKEN_EXPIRATION_NORMAL)
|
||||
|
||||
new_token = create_access_token(token_data, expires_delta)
|
||||
refreshed_data = TokenData(
|
||||
user_id=user.user_id,
|
||||
email=user.email,
|
||||
is_admin=user.is_admin,
|
||||
remember_me=token_data.remember_me,
|
||||
)
|
||||
new_token = create_access_token(refreshed_data, expires_delta)
|
||||
response.headers["X-Refreshed-Token"] = new_token
|
||||
except Exception:
|
||||
pass # Don't fail the request if token refresh fails
|
||||
@@ -148,6 +179,155 @@ class RedirectRootWithQueryStringMiddleware(BaseHTTPMiddleware):
|
||||
return response
|
||||
|
||||
|
||||
def _identify_video_upload_user(scope: Scope) -> tuple[bool, str | None]:
|
||||
"""Identify the uploader for auth and per-user slot accounting.
|
||||
|
||||
Returns (authenticated, per_user_key). In single-user mode there is only one
|
||||
tenant, so per_user_key is None — the whole global capacity is theirs and no
|
||||
per-user quota applies.
|
||||
"""
|
||||
if not ApiDependencies.invoker.services.configuration.multiuser:
|
||||
return True, None
|
||||
|
||||
scheme, token = get_authorization_scheme_param(Headers(scope=scope).get("authorization"))
|
||||
if scheme.lower() != "bearer":
|
||||
return False, None
|
||||
from invokeai.app.services.auth.token_service import verify_token
|
||||
|
||||
token_data = verify_token(token)
|
||||
if token_data is None:
|
||||
return False, None
|
||||
user = ApiDependencies.invoker.services.users.get(token_data.user_id)
|
||||
if user is None or not user.is_active:
|
||||
return False, None
|
||||
return True, token_data.user_id
|
||||
|
||||
|
||||
async def _identify_video_upload_user_async(scope: Scope) -> tuple[bool, str | None]:
|
||||
return await run_in_threadpool(_identify_video_upload_user, scope)
|
||||
|
||||
|
||||
class VideoUploadLimitASGIMiddleware:
|
||||
"""Bound video-upload ingress *before* FastAPI's multipart parser runs.
|
||||
|
||||
The upload route's own MAX_UPLOAD_SIZE check only fires after the multipart body has
|
||||
been fully parsed (and spooled to temp storage), so oversized, chunked, or many
|
||||
concurrent uploads could exhaust temp space before ever being rejected. This
|
||||
middleware rejects oversized requests from the Content-Length header, aborts
|
||||
chunked bodies that exceed the cap mid-stream, and bounds concurrent uploads
|
||||
both globally and per user (so one tenant's slow uploads cannot starve the
|
||||
others into 429s).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
app: ASGIApp,
|
||||
max_body_bytes: int,
|
||||
max_concurrent: int,
|
||||
max_concurrent_per_user: int | None = None,
|
||||
identify_user: Callable[[Scope], tuple[bool, str | None] | Awaitable[tuple[bool, str | None]]] | None = None,
|
||||
idle_timeout_seconds: float = 120.0,
|
||||
max_upload_duration_seconds: float = 30 * 60,
|
||||
) -> None:
|
||||
self.app = app
|
||||
self.max_body_bytes = max_body_bytes
|
||||
self.max_concurrent = max_concurrent
|
||||
self.max_concurrent_per_user = max_concurrent_per_user
|
||||
self.identify_user = identify_user
|
||||
self.idle_timeout_seconds = idle_timeout_seconds
|
||||
self.max_upload_duration_seconds = max_upload_duration_seconds
|
||||
self._active = 0
|
||||
self._active_by_user: dict[str, int] = {}
|
||||
|
||||
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
|
||||
if scope["type"] != "http":
|
||||
return await self.app(scope, receive, send)
|
||||
# Behind a sub-path proxy the public path carries the prefix (see SubPathASGIMiddleware).
|
||||
route_path: str = scope.get("path", "").removeprefix(scope.get("root_path", ""))
|
||||
if not (scope.get("method") == "POST" and route_path == "/api/v1/videos/upload"):
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
per_user_key: str | None = None
|
||||
if self.identify_user is not None:
|
||||
identity = self.identify_user(scope)
|
||||
if inspect.isawaitable(identity):
|
||||
identity = await identity
|
||||
authenticated, per_user_key = identity
|
||||
if not authenticated:
|
||||
response = JSONResponse(
|
||||
{"detail": "Authentication required"},
|
||||
status_code=401,
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
return await response(scope, receive, send)
|
||||
|
||||
content_length = Headers(scope=scope).get("content-length")
|
||||
if content_length is not None and content_length.isdigit() and int(content_length) > self.max_body_bytes:
|
||||
response = JSONResponse(
|
||||
{"detail": f"Video upload exceeds maximum request size ({self.max_body_bytes} bytes)"},
|
||||
status_code=413,
|
||||
)
|
||||
return await response(scope, receive, send)
|
||||
|
||||
if self._active >= self.max_concurrent:
|
||||
response = JSONResponse(
|
||||
{"detail": "Too many concurrent video uploads; try again shortly"},
|
||||
status_code=429,
|
||||
headers={"Retry-After": "5"},
|
||||
)
|
||||
return await response(scope, receive, send)
|
||||
|
||||
if (
|
||||
per_user_key is not None
|
||||
and self.max_concurrent_per_user is not None
|
||||
and self._active_by_user.get(per_user_key, 0) >= self.max_concurrent_per_user
|
||||
):
|
||||
response = JSONResponse(
|
||||
{"detail": "Too many concurrent video uploads for this user; try again shortly"},
|
||||
status_code=429,
|
||||
headers={"Retry-After": "5"},
|
||||
)
|
||||
return await response(scope, receive, send)
|
||||
|
||||
self._active += 1
|
||||
if per_user_key is not None:
|
||||
self._active_by_user[per_user_key] = self._active_by_user.get(per_user_key, 0) + 1
|
||||
received = 0
|
||||
upload_started_at = asyncio.get_running_loop().time()
|
||||
|
||||
async def limited_receive() -> Message:
|
||||
# Backstop for clients that omit Content-Length (chunked transfer): count the
|
||||
# streamed body and abort the request once it exceeds the cap, so the multipart
|
||||
# parser stops spooling. A clean 413 isn't possible mid-parse; the aborted
|
||||
# request surfaces to the client as a dropped connection.
|
||||
nonlocal received
|
||||
remaining_duration = self.max_upload_duration_seconds - (
|
||||
asyncio.get_running_loop().time() - upload_started_at
|
||||
)
|
||||
if remaining_duration <= 0:
|
||||
return {"type": "http.disconnect"}
|
||||
try:
|
||||
message = await asyncio.wait_for(receive(), timeout=min(self.idle_timeout_seconds, remaining_duration))
|
||||
except TimeoutError:
|
||||
return {"type": "http.disconnect"}
|
||||
if message["type"] == "http.request":
|
||||
received += len(message.get("body", b""))
|
||||
if received > self.max_body_bytes:
|
||||
return {"type": "http.disconnect"}
|
||||
return message
|
||||
|
||||
try:
|
||||
await self.app(scope, limited_receive, send)
|
||||
finally:
|
||||
self._active -= 1
|
||||
if per_user_key is not None:
|
||||
remaining = self._active_by_user.get(per_user_key, 0) - 1
|
||||
if remaining > 0:
|
||||
self._active_by_user[per_user_key] = remaining
|
||||
else:
|
||||
self._active_by_user.pop(per_user_key, None)
|
||||
|
||||
|
||||
class SubPathASGIMiddleware:
|
||||
"""Make the app work behind a reverse proxy that serves it under a sub-path (e.g. `/invoke`).
|
||||
|
||||
@@ -191,6 +371,13 @@ class SubPathASGIMiddleware:
|
||||
# Add the middleware
|
||||
app.add_middleware(RedirectRootWithQueryStringMiddleware)
|
||||
app.add_middleware(SlidingWindowTokenMiddleware)
|
||||
app.add_middleware(
|
||||
VideoUploadLimitASGIMiddleware,
|
||||
max_body_bytes=videos.MAX_UPLOAD_REQUEST_SIZE,
|
||||
max_concurrent=videos.MAX_CONCURRENT_VIDEO_UPLOADS,
|
||||
max_concurrent_per_user=videos.MAX_CONCURRENT_VIDEO_UPLOADS_PER_USER,
|
||||
identify_user=_identify_video_upload_user_async,
|
||||
)
|
||||
|
||||
|
||||
# Add event handler
|
||||
@@ -223,6 +410,8 @@ app.include_router(model_manager.model_manager_router, prefix="/api")
|
||||
app.include_router(download_queue.download_queue_router, prefix="/api")
|
||||
app.include_router(image_moves.image_moves_router, prefix="/api")
|
||||
app.include_router(images.images_router, prefix="/api")
|
||||
app.include_router(videos.videos_router, prefix="/api")
|
||||
app.include_router(gallery.gallery_router, prefix="/api")
|
||||
app.include_router(boards.boards_router, prefix="/api")
|
||||
app.include_router(board_images.board_images_router, prefix="/api")
|
||||
app.include_router(virtual_boards.virtual_boards_router, prefix="/api")
|
||||
|
||||
@@ -141,6 +141,7 @@ class UIComponent(str, Enum, metaclass=MetaEnum):
|
||||
None_ = "none"
|
||||
Textarea = "textarea"
|
||||
Slider = "slider"
|
||||
VideoFrameIndex = "video-frame-index"
|
||||
|
||||
|
||||
class FieldDescriptions:
|
||||
@@ -174,6 +175,9 @@ class FieldDescriptions:
|
||||
z_image_model = "Z-Image model (Transformer) to load"
|
||||
qwen_image_model = "Qwen Image Edit model (Transformer) to load"
|
||||
qwen_vl_encoder = "Qwen2.5-VL tokenizer, processor and text/vision encoder"
|
||||
wan_model = "Wan 2.2 model (Transformer) to load"
|
||||
wan_t5_encoder = "UMT5-XXL tokenizer and text encoder for Wan 2.2"
|
||||
wan_ref_image = "Reference-image (VAE-latent) conditioning for Wan 2.2 I2V."
|
||||
sdxl_main_model = "SDXL Main model (UNet, VAE, CLIP1, CLIP2) to load"
|
||||
sdxl_refiner_model = "SDXL Refiner Main Modde (UNet, VAE, CLIP2) to load"
|
||||
onnx_main_model = "ONNX Main model (UNet, VAE, CLIP) to load"
|
||||
@@ -241,6 +245,12 @@ class ImageField(BaseModel):
|
||||
image_name: str = Field(description="The name of the image")
|
||||
|
||||
|
||||
class VideoField(BaseModel):
|
||||
"""A video primitive field"""
|
||||
|
||||
video_name: str = Field(description="The name of the video")
|
||||
|
||||
|
||||
class BoardField(BaseModel):
|
||||
"""A board primitive field"""
|
||||
|
||||
@@ -371,6 +381,39 @@ class AnimaConditioningField(BaseModel):
|
||||
)
|
||||
|
||||
|
||||
class WanConditioningField(BaseModel):
|
||||
"""A Wan 2.2 conditioning tensor primitive value.
|
||||
|
||||
Wan conditioning is the UMT5-XXL hidden state for the prompt plus an attention
|
||||
mask marking valid (non-padding) tokens.
|
||||
"""
|
||||
|
||||
conditioning_name: str = Field(description="The name of conditioning tensor")
|
||||
|
||||
|
||||
class WanRefImageConditioningField(BaseModel):
|
||||
"""Reference-image conditioning for Wan 2.2 I2V.
|
||||
|
||||
Carries the 20-channel VAE-latent condition tensor (4-channel first-frame
|
||||
mask + 16-channel ref-image latents). The denoise loop concatenates this
|
||||
to the 16-channel noise latents along the channel dim each step, producing
|
||||
the 36-channel input the I2V-A14B transformer expects.
|
||||
|
||||
Also carries the spatial dims and frame count used to encode the image so
|
||||
the denoise node can sanity-check the user's width/height/num_frames — a
|
||||
latent temporal-dim mismatch is hard to debug from the downstream error.
|
||||
"""
|
||||
|
||||
condition_tensor_name: str = Field(description="Name of the saved [1, 20, T_lat, H/8, W/8] condition tensor.")
|
||||
width: int = Field(description="Image width used during VAE encoding (matches denoise width).")
|
||||
height: int = Field(description="Image height used during VAE encoding (matches denoise height).")
|
||||
num_frames: int = Field(
|
||||
default=1,
|
||||
description="Pixel-frame count the condition was built for. 1 for single-frame I2V "
|
||||
"(image output), 81+ for video.",
|
||||
)
|
||||
|
||||
|
||||
class ConditioningField(BaseModel):
|
||||
"""A conditioning tensor primitive value"""
|
||||
|
||||
|
||||
@@ -175,6 +175,11 @@ GENERATION_MODES = Literal[
|
||||
"anima_img2img",
|
||||
"anima_inpaint",
|
||||
"anima_outpaint",
|
||||
"wan_txt2img",
|
||||
"wan_img2img",
|
||||
"wan_inpaint",
|
||||
"wan_outpaint",
|
||||
"wan_i2v",
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -87,6 +87,14 @@ class Qwen3EncoderField(BaseModel):
|
||||
loras: List[LoRAField] = Field(default_factory=list, description="LoRAs to apply on model loading")
|
||||
|
||||
|
||||
class WanT5EncoderField(BaseModel):
|
||||
"""Field for the UMT5-XXL text encoder used by Wan 2.2 models."""
|
||||
|
||||
tokenizer: ModelIdentifierField = Field(description="Info to load tokenizer submodel")
|
||||
text_encoder: ModelIdentifierField = Field(description="Info to load text_encoder submodel")
|
||||
loras: List[LoRAField] = Field(default_factory=list, description="LoRAs to apply on model loading")
|
||||
|
||||
|
||||
class VAEField(BaseModel):
|
||||
vae: ModelIdentifierField = Field(description="Info to load vae submodel")
|
||||
seamless_axes: List[str] = Field(default_factory=list, description='Axes("x" and "y") to which apply seamless')
|
||||
@@ -101,6 +109,47 @@ class TransformerField(BaseModel):
|
||||
loras: List[LoRAField] = Field(description="LoRAs to apply on model loading")
|
||||
|
||||
|
||||
class WanTransformerField(BaseModel):
|
||||
"""Transformer field for Wan 2.2 models.
|
||||
|
||||
Wan 2.2 A14B is a Mixture-of-Experts model with two transformer experts:
|
||||
a high-noise expert (active at large timesteps) and a low-noise expert
|
||||
(active at small timesteps). TI2V-5B is a single-transformer model and only
|
||||
populates ``transformer``.
|
||||
|
||||
``boundary_ratio`` matches Diffusers' ``WanPipeline`` semantics: it's the
|
||||
boundary timestep as a fraction of ``num_train_timesteps`` (typically 1000),
|
||||
so ``boundary_ratio=0.875`` means the high-noise expert handles t >= 875 and
|
||||
the low-noise expert handles t < 875.
|
||||
"""
|
||||
|
||||
transformer: ModelIdentifierField = Field(
|
||||
description="Primary transformer submodel. For A14B this is the high-noise expert."
|
||||
)
|
||||
transformer_low_noise: ModelIdentifierField | None = Field(
|
||||
default=None,
|
||||
description="Low-noise transformer expert (Wan 2.2 A14B only). None for TI2V-5B.",
|
||||
)
|
||||
loras: List[LoRAField] = Field(
|
||||
default_factory=list,
|
||||
description="LoRAs to apply to the primary transformer. For A14B applied to the high-noise expert.",
|
||||
)
|
||||
loras_low_noise: List[LoRAField] = Field(
|
||||
default_factory=list,
|
||||
description="LoRAs to apply to the low-noise expert (Wan 2.2 A14B). The Wan LoRA loader "
|
||||
"routes 'both'- and 'low'-targeted LoRAs here; if empty, no LoRAs are applied to the "
|
||||
"low-noise expert (a 'high'-targeted LoRA must not leak onto it).",
|
||||
)
|
||||
boundary_ratio: float = Field(
|
||||
default=0.875,
|
||||
ge=0.0,
|
||||
le=1.0,
|
||||
description="Boundary timestep as a fraction of num_train_timesteps (Wan 2.2 A14B only). "
|
||||
"High-noise expert: t >= boundary_ratio * num_train_timesteps. Low-noise expert: t below. "
|
||||
"Ignored for TI2V-5B.",
|
||||
)
|
||||
|
||||
|
||||
@invocation_output("unet_output")
|
||||
class UNetOutput(BaseInvocationOutput):
|
||||
"""Base class for invocations that output a UNet field."""
|
||||
|
||||
@@ -30,10 +30,14 @@ from invokeai.app.invocations.fields import (
|
||||
SD3ConditioningField,
|
||||
TensorField,
|
||||
UIComponent,
|
||||
VideoField,
|
||||
WanConditioningField,
|
||||
WanRefImageConditioningField,
|
||||
ZImageConditioningField,
|
||||
)
|
||||
from invokeai.app.services.images.images_common import ImageDTO
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.app.services.videos.videos_common import VideoDTO
|
||||
|
||||
"""
|
||||
Primitives: Boolean, Integer, Float, String, Image, Latents, Conditioning, Color
|
||||
@@ -522,6 +526,44 @@ class AnimaConditioningOutput(BaseInvocationOutput):
|
||||
return cls(conditioning=AnimaConditioningField(conditioning_name=conditioning_name))
|
||||
|
||||
|
||||
@invocation_output("wan_conditioning_output")
|
||||
class WanConditioningOutput(BaseInvocationOutput):
|
||||
"""Base class for nodes that output a Wan 2.2 text conditioning tensor."""
|
||||
|
||||
conditioning: WanConditioningField = OutputField(description=FieldDescriptions.cond)
|
||||
|
||||
@classmethod
|
||||
def build(cls, conditioning_name: str) -> "WanConditioningOutput":
|
||||
return cls(conditioning=WanConditioningField(conditioning_name=conditioning_name))
|
||||
|
||||
|
||||
@invocation_output("wan_ref_image_output")
|
||||
class WanRefImageOutput(BaseInvocationOutput):
|
||||
"""Output of a Wan 2.2 reference-image VAE-encoder."""
|
||||
|
||||
ref_image: WanRefImageConditioningField = OutputField(
|
||||
description="VAE-latent reference-image conditioning for Wan 2.2 I2V.",
|
||||
title="Reference Image",
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def build(
|
||||
cls,
|
||||
condition_tensor_name: str,
|
||||
width: int,
|
||||
height: int,
|
||||
num_frames: int = 1,
|
||||
) -> "WanRefImageOutput":
|
||||
return cls(
|
||||
ref_image=WanRefImageConditioningField(
|
||||
condition_tensor_name=condition_tensor_name,
|
||||
width=width,
|
||||
height=height,
|
||||
num_frames=num_frames,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@invocation_output("conditioning_output")
|
||||
class ConditioningOutput(BaseInvocationOutput):
|
||||
"""Base class for nodes that output a single conditioning tensor"""
|
||||
@@ -533,6 +575,57 @@ class ConditioningOutput(BaseInvocationOutput):
|
||||
return cls(conditioning=ConditioningField(conditioning_name=conditioning_name))
|
||||
|
||||
|
||||
@invocation_output("video_output")
|
||||
class VideoOutput(BaseInvocationOutput):
|
||||
"""Output of a node that produces a video file (e.g. Wan 2.2 latents-to-video)."""
|
||||
|
||||
video: VideoField = OutputField(description="The output video")
|
||||
width: int = OutputField(description="The width of the video in pixels")
|
||||
height: int = OutputField(description="The height of the video in pixels")
|
||||
num_frames: int = OutputField(description="The number of frames in the video")
|
||||
fps: float = OutputField(description="The frames-per-second of the video")
|
||||
duration: float = OutputField(description="The duration of the video in seconds")
|
||||
|
||||
@classmethod
|
||||
def build(cls, video_dto: VideoDTO, num_frames: Optional[int] = None) -> "VideoOutput":
|
||||
# Frame count isn't stored on the DTO; derive it from duration * fps when fps is known.
|
||||
fps = video_dto.fps or 0.0
|
||||
if num_frames is None:
|
||||
num_frames = int(round(video_dto.duration * fps)) if fps > 0 else 0
|
||||
return cls(
|
||||
video=VideoField(video_name=video_dto.video_name),
|
||||
width=video_dto.width,
|
||||
height=video_dto.height,
|
||||
num_frames=num_frames,
|
||||
fps=fps,
|
||||
duration=video_dto.duration,
|
||||
)
|
||||
|
||||
|
||||
@invocation(
|
||||
"video",
|
||||
title="Video Primitive",
|
||||
tags=["primitives", "video"],
|
||||
category="primitives",
|
||||
version="1.0.0",
|
||||
)
|
||||
class VideoInvocation(BaseInvocation):
|
||||
"""A video primitive value. Drop a video onto the field to make it available as an input
|
||||
to downstream nodes (e.g. Frame from Video, Concatenate Videos)."""
|
||||
|
||||
video: VideoField = InputField(description="The video to load")
|
||||
|
||||
# Return annotation is a real class (not a forward-ref string) because a previous
|
||||
# `from __future__ import annotations` left wan_l2v with a stringified annotation
|
||||
# and crashed the output-class registry on startup (commit cac366229a).
|
||||
def invoke(self, context: InvocationContext) -> VideoOutput:
|
||||
from invokeai.app.util.video_thumbnails import decoder_frame_count
|
||||
|
||||
video_dto = context.videos.get_dto(self.video.video_name)
|
||||
num_frames = decoder_frame_count(context.videos.get_path(self.video.video_name))
|
||||
return VideoOutput.build(video_dto=video_dto, num_frames=num_frames)
|
||||
|
||||
|
||||
@invocation_output("conditioning_collection_output")
|
||||
class ConditioningCollectionOutput(BaseInvocationOutput):
|
||||
"""Base class for nodes that output a collection of conditioning tensors"""
|
||||
|
||||
@@ -0,0 +1,286 @@
|
||||
"""Concatenate two or more videos with an optional transition.
|
||||
|
||||
Pairs naturally with the I2V chaining workflow: feed several Wan-generated
|
||||
clips into this node to glue them into one longer video. The transition
|
||||
options hide the seam between independently-denoised clips.
|
||||
|
||||
Implementation uses imageio (FFMPEG plugin) for both decode and encode, matching
|
||||
``wan_latents_to_video`` and ``video_thumbnails`` — so we can read our own
|
||||
output without surprises. Frames stream from the decoders into the encoder one
|
||||
at a time, buffering only the transition windows, so peak memory stays
|
||||
O(transition_frames) even when the inputs are long uploads (the upload cap
|
||||
admits files whose decoded frames would run to tens of gigabytes).
|
||||
"""
|
||||
|
||||
import math
|
||||
import tempfile
|
||||
from collections import deque
|
||||
from pathlib import Path
|
||||
from typing import Callable, Iterable, Iterator, Literal, Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
|
||||
from invokeai.app.invocations.fields import (
|
||||
InputField,
|
||||
VideoField,
|
||||
WithBoard,
|
||||
WithMetadata,
|
||||
)
|
||||
from invokeai.app.invocations.primitives import VideoOutput
|
||||
from invokeai.app.services.session_processor.session_processor_common import CanceledException
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.app.util.video_encoding import make_mp4_writer
|
||||
from invokeai.app.util.video_thumbnails import iter_video_frames, probe_video
|
||||
|
||||
TransitionMode = Literal["cut", "crossfade", "fade_through_black"]
|
||||
MAX_TRANSITION_MEMORY_BYTES = 512 * 1024 * 1024
|
||||
_BLEND_WORKING_FRAMES = 13
|
||||
|
||||
|
||||
def _crossfade(a_tail: list[np.ndarray], b_head: list[np.ndarray]) -> Iterator[np.ndarray]:
|
||||
"""Yields a linear A to B cross-dissolve without retaining the blended frames."""
|
||||
n = len(a_tail)
|
||||
for i in range(n):
|
||||
alpha = (i + 1) / (n + 1)
|
||||
blended = a_tail[i].astype(np.float32) * (1.0 - alpha) + b_head[i].astype(np.float32) * alpha
|
||||
yield np.clip(blended, 0, 255).astype(np.uint8)
|
||||
|
||||
|
||||
def _fade_through_black(a_tail: list[np.ndarray], b_head: list[np.ndarray]) -> Iterator[np.ndarray]:
|
||||
"""A fades to black, then black fades to B. Consumes N/2 frames from each side and returns N output frames.
|
||||
|
||||
Asymmetric framing: the first ``len(a_tail)`` output frames are the trailing A frames scaled
|
||||
toward zero brightness; the next ``len(b_head)`` are the leading B frames scaled up from zero.
|
||||
"""
|
||||
n_a = len(a_tail)
|
||||
for i, fa in enumerate(a_tail):
|
||||
# 1.0 at i=0 (fully visible) → near 0 at i=n_a-1 (essentially black).
|
||||
alpha = 1.0 - (i + 1) / (n_a + 1)
|
||||
yield np.clip(fa.astype(np.float32) * alpha, 0, 255).astype(np.uint8)
|
||||
n_b = len(b_head)
|
||||
for j, fb in enumerate(b_head):
|
||||
alpha = (j + 1) / (n_b + 1)
|
||||
yield np.clip(fb.astype(np.float32) * alpha, 0, 255).astype(np.uint8)
|
||||
|
||||
|
||||
@invocation(
|
||||
"video_concat",
|
||||
title="Concatenate Videos",
|
||||
tags=["video", "concat", "transition"],
|
||||
category="video",
|
||||
version="1.0.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class VideoConcatInvocation(BaseInvocation, WithMetadata, WithBoard):
|
||||
"""Join two or more videos into a single MP4.
|
||||
|
||||
Transitions:
|
||||
|
||||
* ``cut`` — hard splice, no blending. Fastest; total length is the sum of inputs.
|
||||
* ``crossfade`` — linear A→B cross-dissolve over ``transition_frames``. Each boundary
|
||||
consumes ``transition_frames`` from both adjacent clips, so total length is
|
||||
``sum(inputs) - transition_frames * (n - 1)``.
|
||||
* ``fade_through_black`` — A fades to black, then B fades in from black. Each boundary
|
||||
consumes ``transition_frames // 2`` frames from the preceding clip's tail and the
|
||||
remainder (``transition_frames - transition_frames // 2``) from the next clip's head,
|
||||
so the total emitted is exactly ``transition_frames`` per boundary — even for odd
|
||||
``transition_frames`` — and the overall length equals the sum of inputs.
|
||||
|
||||
All inputs must share the same pixel dimensions. Output frame rate defaults to the
|
||||
first input's fps; override with ``fps`` to force a specific rate (the frames are not
|
||||
resampled, only the container is encoded at the new rate).
|
||||
"""
|
||||
|
||||
videos: list[VideoField] = InputField(
|
||||
min_length=2,
|
||||
description="Videos to concatenate, in order. At least two are required.",
|
||||
)
|
||||
transition: TransitionMode = InputField(
|
||||
default="cut",
|
||||
description="Transition between consecutive clips.",
|
||||
)
|
||||
transition_frames: int = InputField(
|
||||
default=8,
|
||||
ge=0,
|
||||
le=240,
|
||||
description="Length of each transition in frames. Ignored when transition is 'cut'.",
|
||||
)
|
||||
fps: Optional[int] = InputField(
|
||||
default=None,
|
||||
ge=1,
|
||||
le=120,
|
||||
description="Output frame rate. Defaults to the first input's fps.",
|
||||
)
|
||||
|
||||
def invoke(self, context: InvocationContext) -> VideoOutput:
|
||||
if len(self.videos) < 2:
|
||||
raise ValueError("video_concat requires at least two input videos.")
|
||||
|
||||
paths: list[Path] = [context.videos.get_path(v.video_name) for v in self.videos]
|
||||
|
||||
# Probe inputs up front: enforce matching dims and pick the default output fps.
|
||||
probes = [probe_video(p) for p in paths]
|
||||
widths = {(w, h) for (w, h, _, _) in probes}
|
||||
if len(widths) > 1:
|
||||
raise ValueError(
|
||||
f"All inputs must share the same dimensions. Got: "
|
||||
f"{sorted(widths)}. Re-render at a single resolution before concatenating."
|
||||
)
|
||||
width, height, _, _first_fps = probes[0]
|
||||
# libx264 + yuv420p needs even dimensions; we encode with macro_block_size=1 to
|
||||
# preserve the source dimensions exactly, so reject odd sources with a clear error.
|
||||
if width % 2 or height % 2:
|
||||
raise ValueError(
|
||||
f"Input videos are {width}x{height}; H.264 encoding requires even dimensions. "
|
||||
"Re-encode or crop the sources to even width and height first."
|
||||
)
|
||||
self._validate_transition_memory(width, height)
|
||||
output_fps = self._resolve_output_fps([probe[3] for probe in probes])
|
||||
|
||||
context.util.signal_progress(f"Joining {len(self.videos)} clip(s) ({self.transition}) @ {output_fps:.2f} fps")
|
||||
|
||||
tmp = tempfile.NamedTemporaryFile(prefix="invokeai_video_concat_", suffix=".mp4", delete=False)
|
||||
tmp.close()
|
||||
tmp_path = Path(tmp.name)
|
||||
try:
|
||||
# Frames stream from the decoders straight into the encoder; only the
|
||||
# transition windows are buffered. See _iter_joined_frames.
|
||||
writer = make_mp4_writer(tmp_path, output_fps)
|
||||
num_frames = 0
|
||||
try:
|
||||
clip_iters = [iter_video_frames(p, is_canceled=context.util.is_canceled) for p in paths]
|
||||
for frame in self._iter_joined_frames(clip_iters, is_canceled=context.util.is_canceled):
|
||||
writer.append_data(frame)
|
||||
num_frames += 1
|
||||
finally:
|
||||
writer.close()
|
||||
|
||||
if num_frames == 0:
|
||||
raise ValueError("Concatenation produced zero output frames.")
|
||||
|
||||
duration = num_frames / output_fps
|
||||
context.logger.info(
|
||||
f"Encoded concatenated MP4: {num_frames} frames @ {output_fps:.2f} fps "
|
||||
f"({duration:.2f}s) at {width}x{height}"
|
||||
)
|
||||
video_dto = context.videos.save(
|
||||
source_path=tmp_path,
|
||||
width=width,
|
||||
height=height,
|
||||
duration=duration,
|
||||
fps=output_fps,
|
||||
)
|
||||
context.logger.info(f"Saved concatenated video: {video_dto.video_name}")
|
||||
return VideoOutput.build(video_dto)
|
||||
finally:
|
||||
try:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _estimate_transition_memory(self, width: int, height: int) -> int:
|
||||
if self.transition == "cut" or self.transition_frames == 0:
|
||||
return 0
|
||||
buffered_frames = self.transition_frames * (2 if self.transition == "crossfade" else 1)
|
||||
frame_bytes = width * height * 3
|
||||
return frame_bytes * (buffered_frames + _BLEND_WORKING_FRAMES)
|
||||
|
||||
def _resolve_output_fps(self, source_rates: list[Optional[float]]) -> float:
|
||||
if self.fps is not None:
|
||||
return float(self.fps)
|
||||
known_rates = [rate for rate in source_rates if rate is not None and rate > 0]
|
||||
if not known_rates:
|
||||
return 16.0
|
||||
if any(not math.isclose(rate, known_rates[0], rel_tol=1e-3) for rate in known_rates[1:]):
|
||||
raise ValueError("Input videos have different frame rates; set Output FPS to retime them.")
|
||||
# An unknown rate mixed with agreeing known ones is not an error: probe_video
|
||||
# deliberately reports None for metadata-poor containers (VFR flags, missing
|
||||
# avg_frame_rate) whose real rate is usually the same as their neighbours'.
|
||||
# Erroring here would break previously-working concat workflows over a metadata
|
||||
# quirk; disagreement between *known* rates is the case that silently retimes.
|
||||
return known_rates[0]
|
||||
|
||||
def _validate_transition_memory(self, width: int, height: int) -> None:
|
||||
estimated_bytes = self._estimate_transition_memory(width, height)
|
||||
if estimated_bytes > MAX_TRANSITION_MEMORY_BYTES:
|
||||
estimated_mib = estimated_bytes / (1024 * 1024)
|
||||
limit_mib = MAX_TRANSITION_MEMORY_BYTES / (1024 * 1024)
|
||||
raise ValueError(
|
||||
f"The requested transition needs an estimated {estimated_mib:.0f} MiB, "
|
||||
f"which exceeds the {limit_mib:.0f} MiB transition memory budget. "
|
||||
"Lower transition_frames or use lower-resolution clips."
|
||||
)
|
||||
|
||||
def _iter_joined_frames(
|
||||
self,
|
||||
clips: list[Iterable[np.ndarray]],
|
||||
is_canceled: Optional[Callable[[], bool]] = None,
|
||||
) -> Iterator[np.ndarray]:
|
||||
"""Yields the joined output frames, pulling lazily from each clip's frame iterator.
|
||||
|
||||
A frame is emitted as soon as it can no longer participate in a transition, so at
|
||||
most one transition window (the previous clip's tail plus the current clip's head,
|
||||
each bounded by ``transition_frames``) is buffered at a time — never a whole clip.
|
||||
|
||||
Transition layout matches the class docstring:
|
||||
|
||||
* ``crossfade`` consumes ``tf`` frames from both sides of each boundary and emits
|
||||
``tf`` blended frames in their place.
|
||||
* ``fade_through_black`` splits ``tf`` asymmetrically (``tf // 2`` from the previous
|
||||
clip's tail, the remainder from the next clip's head) so an odd ``tf`` still emits
|
||||
exactly ``tf`` frames per boundary.
|
||||
|
||||
Raises ValueError if a clip decodes to zero frames or is too short to supply its
|
||||
transition windows. Because frames stream straight into the encoder, that error can
|
||||
surface mid-encode; the caller discards the partial output file.
|
||||
"""
|
||||
if self.transition == "crossfade" and self.transition_frames > 0:
|
||||
tail_need = head_need = self.transition_frames
|
||||
blend = _crossfade
|
||||
elif self.transition == "fade_through_black" and self.transition_frames > 0:
|
||||
tail_need = self.transition_frames // 2
|
||||
head_need = self.transition_frames - tail_need
|
||||
blend = _fade_through_black
|
||||
else:
|
||||
# "cut", or a zero-length transition: plain concatenation.
|
||||
tail_need = head_need = 0
|
||||
blend = None
|
||||
|
||||
a_tail: list[np.ndarray] = []
|
||||
for i, clip in enumerate(clips):
|
||||
head_want = 0 if i == 0 else head_need
|
||||
tail_keep = 0 if i == len(clips) - 1 else tail_need
|
||||
b_head: list[np.ndarray] = []
|
||||
head_complete = head_want == 0
|
||||
tail_buf: deque[np.ndarray] = deque()
|
||||
n_frames = 0
|
||||
for frame in clip:
|
||||
if is_canceled is not None and is_canceled():
|
||||
raise CanceledException
|
||||
frame = np.ascontiguousarray(frame)
|
||||
n_frames += 1
|
||||
# The clip's first head_want frames are consumed into the boundary blend
|
||||
# with the previous clip's tail rather than emitted directly.
|
||||
if not head_complete:
|
||||
b_head.append(frame)
|
||||
if len(b_head) == head_want and blend is not None:
|
||||
yield from blend(a_tail, b_head)
|
||||
a_tail = []
|
||||
b_head = []
|
||||
head_complete = True
|
||||
continue
|
||||
# Hold back the last tail_keep frames seen so far; anything older is
|
||||
# guaranteed not to be part of the next boundary and can be emitted.
|
||||
tail_buf.append(frame)
|
||||
if len(tail_buf) > tail_keep:
|
||||
yield tail_buf.popleft()
|
||||
if n_frames == 0:
|
||||
raise ValueError(f"Input video {i} ({self.videos[i].video_name}) decoded to zero frames.")
|
||||
if n_frames < head_want + tail_keep:
|
||||
raise ValueError(
|
||||
f"Clip {i} has {n_frames} frames but the requested transitions need "
|
||||
f"{head_want} from its head + {tail_keep} from its tail. Lower "
|
||||
f"transition_frames or use longer clips."
|
||||
)
|
||||
a_tail = list(tail_buf)
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Extract a single frame from a video as an image.
|
||||
|
||||
Enables I2V "shot extension": take the last frame of one clip and feed it back
|
||||
in as the reference image for the next clip, then concatenate the MP4s
|
||||
externally to get a video longer than the model's single-shot frame budget.
|
||||
Also useful as a general-purpose video-to-image step.
|
||||
"""
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
|
||||
from invokeai.app.invocations.fields import (
|
||||
InputField,
|
||||
UIComponent,
|
||||
VideoField,
|
||||
WithBoard,
|
||||
WithMetadata,
|
||||
)
|
||||
from invokeai.app.invocations.primitives import ImageOutput
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.app.util.video_thumbnails import decoder_frame_count, extract_video_frame, probe_video
|
||||
|
||||
|
||||
@invocation(
|
||||
"video_frame_extract",
|
||||
title="Frame from Video",
|
||||
tags=["video", "image", "frame"],
|
||||
category="image",
|
||||
version="1.1.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class VideoFrameExtractInvocation(BaseInvocation, WithMetadata, WithBoard):
|
||||
"""Extract a single frame from a video and save it as an image.
|
||||
|
||||
``frame_index`` is 0-based. Negative indices count from the end, so the
|
||||
default of -1 returns the final frame — the typical setup for chaining
|
||||
I2V clips into a longer sequence.
|
||||
"""
|
||||
|
||||
video: VideoField = InputField(description="The video to extract a frame from.")
|
||||
frame_index: int = InputField(
|
||||
default=-1,
|
||||
description="Index of the frame to extract. 0 = first frame, -1 = last frame, -2 = second-to-last, etc.",
|
||||
ui_component=UIComponent.VideoFrameIndex,
|
||||
)
|
||||
|
||||
def invoke(self, context: InvocationContext) -> ImageOutput:
|
||||
video_path = context.videos.get_path(self.video.video_name)
|
||||
|
||||
# Resolve negative indices against the actual frame count rather than
|
||||
# trusting imageio plugins to accept index=-1 uniformly. Use the decoder's
|
||||
# frame count when available — duration*fps can be off-by-one for VFR
|
||||
# uploads or containers with approximate metadata, causing frame_index=-1
|
||||
# to point past the final frame.
|
||||
index = self.frame_index
|
||||
if index < 0:
|
||||
n_frames = decoder_frame_count(video_path)
|
||||
if n_frames is None:
|
||||
_, _, duration, fps = probe_video(video_path)
|
||||
if not fps or duration <= 0:
|
||||
raise ValueError(
|
||||
f"Cannot resolve negative frame index for video {self.video.video_name}: "
|
||||
f"probe returned duration={duration}, fps={fps}."
|
||||
)
|
||||
n_frames = int(round(duration * fps))
|
||||
if n_frames <= 0:
|
||||
raise ValueError(f"Video {self.video.video_name} has no decodable frames (probed {n_frames}).")
|
||||
index = n_frames + index
|
||||
if index < 0:
|
||||
raise ValueError(f"frame_index {self.frame_index} is out of range for a {n_frames}-frame video.")
|
||||
|
||||
frame = extract_video_frame(video_path, frame_index=index)
|
||||
if frame is None:
|
||||
raise ValueError(f"Failed to extract frame {index} from {self.video.video_name}.")
|
||||
|
||||
image_dto = context.images.save(image=frame)
|
||||
return ImageOutput.build(image_dto=image_dto)
|
||||
@@ -0,0 +1,244 @@
|
||||
"""Extract a contiguous range of frames from a video and re-encode as MP4.
|
||||
|
||||
Companion to ``video_frame_extract`` (single frame → image) and
|
||||
``video_concat`` (many videos → one). This node takes a slice of an input
|
||||
video and emits a new MP4, so the output can be fed straight into
|
||||
Concatenate Videos to splice clips together — e.g. trim a generated clip
|
||||
to a usable middle section before chaining it to another shot.
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
from itertools import islice
|
||||
from pathlib import Path
|
||||
from typing import Callable, Iterator, Optional, Protocol
|
||||
|
||||
import numpy as np
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import (
|
||||
BaseInvocation,
|
||||
BaseInvocationOutput,
|
||||
Classification,
|
||||
invocation,
|
||||
invocation_output,
|
||||
)
|
||||
from invokeai.app.invocations.fields import (
|
||||
InputField,
|
||||
OutputField,
|
||||
UIComponent,
|
||||
VideoField,
|
||||
WithBoard,
|
||||
WithMetadata,
|
||||
)
|
||||
from invokeai.app.invocations.primitives import VideoOutput
|
||||
from invokeai.app.services.session_processor.session_processor_common import CanceledException
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.app.util.video_encoding import make_mp4_writer
|
||||
from invokeai.app.util.video_thumbnails import decoder_frame_count, iter_video_frames, probe_video
|
||||
|
||||
|
||||
class _FrameWriter(Protocol):
|
||||
def append_data(self, frame: np.ndarray) -> None: ...
|
||||
|
||||
|
||||
def _validate_even_dimensions(width: int, height: int, video_name: str) -> None:
|
||||
"""Raises if the source dimensions can't be encoded as-is with libx264 + yuv420p.
|
||||
|
||||
We encode with ``macro_block_size=1`` to preserve the source dimensions exactly
|
||||
(imageio's default of 16 silently rescales), which requires even width and height.
|
||||
"""
|
||||
if width % 2 or height % 2:
|
||||
raise ValueError(
|
||||
f"Video {video_name} is {width}x{height}; H.264 encoding requires even dimensions. "
|
||||
"Re-encode or crop the source to even width and height first."
|
||||
)
|
||||
|
||||
|
||||
def _write_frame_range(
|
||||
frames: Iterator[np.ndarray],
|
||||
writer: _FrameWriter,
|
||||
start: int,
|
||||
end: int,
|
||||
is_canceled: Optional[Callable[[], bool]] = None,
|
||||
) -> int:
|
||||
"""Streams frames[start..end] (inclusive) from a lazy decoder into the writer.
|
||||
|
||||
Frames are appended one at a time as they stream past — the full range is never
|
||||
materialized in RAM. The upload cap admits files whose decoded frames would run to
|
||||
tens of gigabytes, so peak memory here must stay one-frame-sized regardless of the
|
||||
requested range. Decoding stops as soon as ``end`` has been written. Returns the
|
||||
number of frames written.
|
||||
"""
|
||||
written = 0
|
||||
for idx, frame in enumerate(islice(frames, end + 1)):
|
||||
if is_canceled is not None and is_canceled():
|
||||
raise CanceledException
|
||||
if idx < start:
|
||||
continue
|
||||
writer.append_data(np.ascontiguousarray(frame))
|
||||
written += 1
|
||||
return written
|
||||
|
||||
|
||||
@invocation_output("extract_video_range_output")
|
||||
class ExtractVideoRangeOutput(BaseInvocationOutput):
|
||||
"""Output of ``extract_video_range``: a trimmed video plus the resolved frame indices.
|
||||
|
||||
Mirrors ``VideoOutput`` so the video can be piped directly into Concatenate Videos or
|
||||
any other ``VideoField``-consuming node, and additionally exposes the resolved
|
||||
(positive, clamped) start and end indices so chained workflows can feed them back in
|
||||
— e.g. drive a downstream Frame from Video to pull the same boundary frame.
|
||||
"""
|
||||
|
||||
video: VideoField = OutputField(description="The trimmed video")
|
||||
width: int = OutputField(description="The width of the video in pixels")
|
||||
height: int = OutputField(description="The height of the video in pixels")
|
||||
num_frames: int = OutputField(description="The number of frames in the trimmed video")
|
||||
fps: float = OutputField(description="The frames-per-second of the trimmed video")
|
||||
duration: float = OutputField(description="The duration of the trimmed video in seconds")
|
||||
start_frame: int = OutputField(description="The resolved (positive, 0-based) start frame index in the source video")
|
||||
end_frame: int = OutputField(description="The resolved (positive, 0-based) end frame index in the source video")
|
||||
|
||||
|
||||
@invocation(
|
||||
"extract_video_range",
|
||||
title="Frame Range from Video",
|
||||
tags=["video", "trim", "range", "frames"],
|
||||
category="video",
|
||||
version="1.1.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class ExtractVideoRangeInvocation(BaseInvocation, WithMetadata, WithBoard):
|
||||
"""Trim a video to a contiguous frame range and re-encode as MP4.
|
||||
|
||||
Both bounds are inclusive and 0-based — ``start_frame=10, end_frame=50``
|
||||
emits 41 frames. Negative indices count from the end (``end_frame=-1``
|
||||
is the final frame), matching ``video_frame_extract``. The output frame
|
||||
rate defaults to the source video's frame rate; set ``fps=0`` to inherit
|
||||
it (or 16 fps if the source rate can't be probed).
|
||||
|
||||
The resolved (positive) ``start_frame`` and ``end_frame`` are also emitted as
|
||||
outputs, so chained workflows can re-use the boundary indices — e.g. feeding
|
||||
them into a downstream Frame from Video to extract the same boundary frame.
|
||||
"""
|
||||
|
||||
video: VideoField = InputField(description="The video to extract a frame range from.")
|
||||
start_frame: int = InputField(
|
||||
default=0,
|
||||
description=("First frame to keep, inclusive. 0 = first frame. Negative indices count from the end."),
|
||||
ui_component=UIComponent.VideoFrameIndex,
|
||||
)
|
||||
end_frame: int = InputField(
|
||||
default=-1,
|
||||
description=("Last frame to keep, inclusive. -1 = last frame. Negative indices count from the end."),
|
||||
ui_component=UIComponent.VideoFrameIndex,
|
||||
)
|
||||
fps: int = InputField(
|
||||
default=0,
|
||||
ge=0,
|
||||
le=120,
|
||||
description="Output frame rate. 0 = match the source video's frame rate "
|
||||
"(falls back to 16 fps if the source rate can't be probed).",
|
||||
)
|
||||
|
||||
def invoke(self, context: InvocationContext) -> ExtractVideoRangeOutput:
|
||||
video_path = context.videos.get_path(self.video.video_name)
|
||||
width, height, duration, source_fps = probe_video(video_path)
|
||||
|
||||
n_frames = decoder_frame_count(video_path)
|
||||
if n_frames is None:
|
||||
if not source_fps or duration <= 0:
|
||||
raise ValueError(
|
||||
f"Cannot determine frame count for {self.video.video_name}: "
|
||||
f"probe returned duration={duration}, fps={source_fps}."
|
||||
)
|
||||
n_frames = int(round(duration * source_fps))
|
||||
if n_frames <= 0:
|
||||
raise ValueError(f"Video {self.video.video_name} has no decodable frames (probed {n_frames}).")
|
||||
|
||||
start = self._resolve_index(self.start_frame, n_frames, "start_frame")
|
||||
end = self._resolve_index(self.end_frame, n_frames, "end_frame")
|
||||
if end < start:
|
||||
raise ValueError(
|
||||
f"end_frame ({self.end_frame} → {end}) must be >= start_frame "
|
||||
f"({self.start_frame} → {start}) after resolving negative indices."
|
||||
)
|
||||
|
||||
# Derive the output frame rate from the source video when ``fps`` is 0
|
||||
# (the default), so a trimmed clip plays back at the same speed as its
|
||||
# source. Fall back to 16 fps (the Wan video default) when the source
|
||||
# rate couldn't be probed.
|
||||
if self.fps > 0:
|
||||
output_fps = float(self.fps)
|
||||
elif source_fps and source_fps > 0:
|
||||
output_fps = float(source_fps)
|
||||
else:
|
||||
output_fps = 16.0
|
||||
|
||||
# libx264 + yuv420p needs even dimensions. Reject odd sources up front with a
|
||||
# clear message instead of surfacing an opaque ffmpeg error mid-encode.
|
||||
_validate_even_dimensions(width, height, self.video.video_name)
|
||||
|
||||
context.util.signal_progress(f"Transcoding frames {start}-{end} of {n_frames} @ {output_fps:.2f} fps")
|
||||
|
||||
tmp = tempfile.NamedTemporaryFile(prefix="invokeai_video_range_", suffix=".mp4", delete=False)
|
||||
tmp.close()
|
||||
tmp_path = Path(tmp.name)
|
||||
try:
|
||||
# imageio's iter_index isn't exposed by iio.imiter, so we enumerate and skip.
|
||||
# Frames stream straight from the decoder into the encoder; see _write_frame_range.
|
||||
writer = make_mp4_writer(tmp_path, output_fps)
|
||||
try:
|
||||
num_frames = _write_frame_range(
|
||||
iter_video_frames(video_path, is_canceled=context.util.is_canceled),
|
||||
writer,
|
||||
start,
|
||||
end,
|
||||
is_canceled=context.util.is_canceled,
|
||||
)
|
||||
finally:
|
||||
writer.close()
|
||||
|
||||
expected_frames = end - start + 1
|
||||
if num_frames != expected_frames:
|
||||
raise ValueError(
|
||||
f"Decoded only {num_frames} of {expected_frames} requested frames for range {start}-{end} "
|
||||
f"of {self.video.video_name} "
|
||||
f"(probed {n_frames} frames). The container's metadata may be inaccurate."
|
||||
)
|
||||
|
||||
out_duration = num_frames / output_fps
|
||||
context.logger.info(
|
||||
f"Encoded trimmed MP4: {num_frames} frames @ {output_fps:.2f} fps "
|
||||
f"({out_duration:.2f}s) at {width}x{height}"
|
||||
)
|
||||
video_dto = context.videos.save(
|
||||
source_path=tmp_path,
|
||||
width=width,
|
||||
height=height,
|
||||
duration=out_duration,
|
||||
fps=output_fps,
|
||||
)
|
||||
context.logger.info(f"Saved trimmed video: {video_dto.video_name}")
|
||||
base = VideoOutput.build(video_dto)
|
||||
return ExtractVideoRangeOutput(
|
||||
video=base.video,
|
||||
width=base.width,
|
||||
height=base.height,
|
||||
num_frames=base.num_frames,
|
||||
fps=base.fps,
|
||||
duration=base.duration,
|
||||
start_frame=start,
|
||||
end_frame=end,
|
||||
)
|
||||
finally:
|
||||
try:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def _resolve_index(value: int, n_frames: int, field_name: str) -> int:
|
||||
resolved = value + n_frames if value < 0 else value
|
||||
if resolved < 0 or resolved >= n_frames:
|
||||
raise ValueError(f"{field_name}={value} is out of range for a {n_frames}-frame video.")
|
||||
return resolved
|
||||
@@ -0,0 +1,777 @@
|
||||
"""Wan 2.2 denoise invocation.
|
||||
|
||||
Supports both single-transformer (TI2V-5B) and dual-expert MoE (A14B) denoising.
|
||||
For A14B the high-noise expert handles timesteps ``t >= boundary_timestep`` and
|
||||
the low-noise expert handles ``t < boundary_timestep``, where
|
||||
``boundary_timestep = boundary_ratio * num_train_timesteps`` (typically 1000).
|
||||
|
||||
To keep VRAM usage manageable both experts are pinned in the model cache
|
||||
(system RAM) but only one is GPU-resident at a time. The boundary is normally
|
||||
crossed once per denoise, so the swap incurs a single CPU→GPU transfer.
|
||||
|
||||
Phase 8 will add inpaint via :class:`RectifiedFlowInpaintExtension`.
|
||||
|
||||
The transformer call signature mirrors Diffusers' ``WanPipeline``:
|
||||
|
||||
transformer(
|
||||
hidden_states=latents_5d, # [B, C, 1, H/s, W/s]
|
||||
timestep=t.expand(B), # scheduler-time
|
||||
encoder_hidden_states=prompt_embeds, # [B, seq_len, 4096]
|
||||
attention_kwargs=None,
|
||||
return_dict=False,
|
||||
)[0]
|
||||
"""
|
||||
|
||||
from contextlib import ExitStack
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Iterable, Iterator, Optional, Tuple
|
||||
|
||||
import torch
|
||||
import torchvision.transforms as tv_transforms
|
||||
from torchvision.transforms.functional import resize as tv_resize
|
||||
from tqdm import tqdm
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
|
||||
from invokeai.app.invocations.fields import (
|
||||
DenoiseMaskField,
|
||||
FieldDescriptions,
|
||||
Input,
|
||||
InputField,
|
||||
LatentsField,
|
||||
WanConditioningField,
|
||||
WanRefImageConditioningField,
|
||||
)
|
||||
from invokeai.app.invocations.model import LoRAField, WanTransformerField
|
||||
from invokeai.app.invocations.primitives import LatentsOutput
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.backend.model_manager.taxonomy import BaseModelType, ModelFormat, WanVariantType
|
||||
from invokeai.backend.patches.layer_patcher import LayerPatcher
|
||||
from invokeai.backend.patches.lora_conversions.wan_lora_constants import WAN_LORA_TRANSFORMER_PREFIX
|
||||
from invokeai.backend.patches.model_patch_raw import ModelPatchRaw
|
||||
from invokeai.backend.rectified_flow.rectified_flow_inpaint_extension import RectifiedFlowInpaintExtension
|
||||
from invokeai.backend.stable_diffusion.diffusers_pipeline import PipelineIntermediateState
|
||||
from invokeai.backend.stable_diffusion.diffusion.conditioning_data import WanConditioningInfo
|
||||
from invokeai.backend.util.devices import TorchDevice
|
||||
from invokeai.backend.wan.sampling_utils import get_spatial_scale_factor, make_noise
|
||||
|
||||
# Type alias: a factory that produces a fresh iterator of (LoRA patch, weight)
|
||||
# pairs each time it is called. We need fresh iterators because the patcher
|
||||
# consumes the iterator once per ``apply_smart_model_patches`` invocation, and
|
||||
# the expert may be swapped (and re-entered) multiple times in a render.
|
||||
LoRAIteratorFactory = Callable[[], Iterable[Tuple[ModelPatchRaw, float]]]
|
||||
|
||||
|
||||
def _resolve_variant(context: InvocationContext, transformer_field: WanTransformerField) -> WanVariantType:
|
||||
"""Look up the Wan variant from the main model config that produced this transformer."""
|
||||
config = context.models.get_config(transformer_field.transformer)
|
||||
variant = getattr(config, "variant", None)
|
||||
if not isinstance(variant, WanVariantType):
|
||||
raise ValueError(f"Could not determine Wan variant from model {config.name!r}: variant is {variant!r}.")
|
||||
return variant
|
||||
|
||||
|
||||
def _validate_spatial_dimensions(variant: WanVariantType, width: int, height: int) -> None:
|
||||
if variant == WanVariantType.TI2V_5B and (width % 32 or height % 32):
|
||||
raise ValueError(
|
||||
f"TI2V-5B requires width and height to be multiples of 32 (got {width}x{height}). "
|
||||
"Wan 2.2-VAE 16x spatial * transformer patch_size 2 = pixel dims must divide by 32."
|
||||
)
|
||||
|
||||
|
||||
def _validate_ref_condition_shape(
|
||||
condition: torch.Tensor,
|
||||
*,
|
||||
channels: int,
|
||||
frames: int,
|
||||
height: int,
|
||||
width: int,
|
||||
) -> None:
|
||||
if condition.ndim != 5:
|
||||
raise ValueError(f"Wan reference condition must be a 5D tensor; got shape {tuple(condition.shape)}.")
|
||||
if condition.shape[0] != 1:
|
||||
raise ValueError(f"Wan reference condition requires batch size 1; got {condition.shape[0]}.")
|
||||
if condition.shape[1] != channels:
|
||||
raise ValueError(f"Wan reference condition requires {channels} channels; got {condition.shape[1]}.")
|
||||
if condition.shape[2] != frames:
|
||||
expected = "a single latent frame" if frames == 1 else f"{frames} latent frames"
|
||||
raise ValueError(f"Wan reference condition requires {expected}; got {condition.shape[2]}.")
|
||||
if condition.shape[3:] != (height, width):
|
||||
raise ValueError(
|
||||
f"Wan reference condition requires {width}x{height} latent spatial dimensions; "
|
||||
f"got {condition.shape[4]}x{condition.shape[3]}."
|
||||
)
|
||||
|
||||
|
||||
def _scheduler_path_for_transformer(context: InvocationContext, transformer_field: WanTransformerField) -> Path | None:
|
||||
"""Return the on-disk ``scheduler/`` directory for the main model, or None."""
|
||||
config = context.models.get_config(transformer_field.transformer)
|
||||
model_root = context.models.get_absolute_path(config)
|
||||
if model_root.is_file():
|
||||
return None
|
||||
candidate = model_root / "scheduler"
|
||||
if (candidate / "scheduler_config.json").exists():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _default_scheduler_for_variant(variant: WanVariantType):
|
||||
"""Build a variant-appropriate scheduler when no on-disk config is available.
|
||||
|
||||
Standalone GGUF / single-file installs don't ship a ``scheduler/`` directory,
|
||||
so we have to reconstruct the scheduler from variant knowledge. Values are
|
||||
verbatim from each variant's ``scheduler/scheduler_config.json`` in the
|
||||
matching ``Wan-AI/Wan2.2-*-Diffusers`` repo.
|
||||
"""
|
||||
from diffusers import UniPCMultistepScheduler
|
||||
|
||||
# Both Wan 2.2 families ship UniPCMultistepScheduler with flow_prediction +
|
||||
# use_flow_sigmas; only the exponential flow_shift differs (5.0 for TI2V-5B,
|
||||
# 3.0 for the A14B pair). Without these settings samples drift — an unshifted
|
||||
# schedule also skews how many A14B steps land above the MoE expert boundary.
|
||||
flow_shift = 5.0 if variant == WanVariantType.TI2V_5B else 3.0
|
||||
return UniPCMultistepScheduler(
|
||||
num_train_timesteps=1000,
|
||||
solver_order=2,
|
||||
prediction_type="flow_prediction",
|
||||
flow_shift=flow_shift,
|
||||
use_flow_sigmas=True,
|
||||
solver_type="bh2",
|
||||
final_sigmas_type="zero",
|
||||
)
|
||||
|
||||
|
||||
class _ExpertSwapper:
|
||||
"""Manages GPU residency and LoRA patching of one or two Wan transformer experts.
|
||||
|
||||
Both experts are kept in the model cache (system RAM); only one is on
|
||||
device at a time. ``get(label)`` returns the model for the requested label,
|
||||
swapping GPU residency when the label changes and applying that expert's
|
||||
LoRA patches via ``LayerPatcher.apply_smart_model_patches``.
|
||||
|
||||
Ordering on swap: exit the active expert's LoRA context (restores weights)
|
||||
-> exit ``model_on_device`` (returns expert to RAM) -> load the new expert
|
||||
(fresh handle) -> enter its device context -> apply its LoRAs. This
|
||||
mirrors the pattern used by ``flux_denoise``/``anima_denoise`` but adds
|
||||
the extra context layer needed for dual experts.
|
||||
|
||||
Model handles are obtained lazily inside ``get()`` rather than cached at
|
||||
construction. With dual ~9 GB GGUF experts plus a UMT5-XXL encoder
|
||||
competing for the RAM cache, holding both ``LoadedModel`` handles upfront
|
||||
can leave one of them stale by the time the swap happens — InvokeAI's
|
||||
model cache emits a ``has already been dropped from the RAM cache``
|
||||
warning and reloads from disk per swap. See issue #7513 for the broader
|
||||
pattern.
|
||||
"""
|
||||
|
||||
HIGH = "high"
|
||||
LOW = "low"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
context: InvocationContext,
|
||||
high_model: Any,
|
||||
low_model: Any | None,
|
||||
inference_dtype: torch.dtype,
|
||||
high_lora_factory: LoRAIteratorFactory | None = None,
|
||||
low_lora_factory: LoRAIteratorFactory | None = None,
|
||||
high_is_quantized: bool = False,
|
||||
low_is_quantized: bool = False,
|
||||
) -> None:
|
||||
self._context = context
|
||||
self._high_model = high_model
|
||||
self._low_model = low_model
|
||||
self._inference_dtype = inference_dtype
|
||||
self._high_lora_factory = high_lora_factory
|
||||
self._low_lora_factory = low_lora_factory
|
||||
self._high_is_quantized = high_is_quantized
|
||||
self._low_is_quantized = low_is_quantized
|
||||
self._active_label: str | None = None
|
||||
self._active_info: Any | None = None
|
||||
self._active_device_ctx: Any | None = None
|
||||
self._active_lora_ctx: Any | None = None
|
||||
self._active_model: Any | None = None
|
||||
|
||||
def get(self, label: str) -> Any:
|
||||
if label not in (self.HIGH, self.LOW):
|
||||
raise ValueError(f"Unknown expert label: {label!r}")
|
||||
if label == self.LOW and self._low_model is None:
|
||||
raise ValueError("Low-noise expert was requested but is not available.")
|
||||
if label == self._active_label:
|
||||
assert self._active_model is not None
|
||||
return self._active_model
|
||||
|
||||
# Capture the outgoing expert's cache record before _release() drops our handle.
|
||||
# We need it to force-unload below.
|
||||
outgoing_cached_model = None
|
||||
if self._active_info is not None:
|
||||
# ``LoadedModel`` exposes its cache_record only via a private attribute. There
|
||||
# is no public ``unload_from_vram`` on the LoadedModel today, and we don't want
|
||||
# to take on a broader backend refactor in this fix; tolerate AttributeError
|
||||
# so a future refactor doesn't break the swap.
|
||||
outgoing_cached_model = getattr(self._active_info, "_cache_record", None)
|
||||
if outgoing_cached_model is not None:
|
||||
outgoing_cached_model = getattr(outgoing_cached_model, "cached_model", None)
|
||||
|
||||
# Release current GPU residency before bringing the other expert on device.
|
||||
self._release()
|
||||
|
||||
# Force the outgoing expert off GPU. The model cache's automatic offload
|
||||
# (inside lock() -> _offload_unlocked_models) decides how much to free based on
|
||||
# ``torch.cuda.memory_allocated()`` minus a 3 GB working-memory budget. With Wan
|
||||
# 81-frame video the intermediate activations from the previous denoise step are
|
||||
# still allocated alongside the just-unlocked high-noise expert, so the cache
|
||||
# underestimates how much room the new expert really needs and partial-loads
|
||||
# most of its layers to CPU. The user-visible symptom: log line "Loaded model
|
||||
# ... VRAM: 2381 MB (25.9%)" instead of ~100% for the incoming expert.
|
||||
#
|
||||
# Sidestep the heuristic by explicitly unloading every weight of the outgoing
|
||||
# expert to RAM. This is safe even if the cache evicted the entry between unlock
|
||||
# and now — the cached_model object still owns the tensors.
|
||||
if outgoing_cached_model is not None:
|
||||
try:
|
||||
outgoing_cached_model.full_unload_from_vram()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Hand the PyTorch allocator a clean slate before partial_load_to_vram measures
|
||||
# free space — the freed blocks stay pinned in the caching allocator until
|
||||
# empty_cache is called.
|
||||
TorchDevice.empty_cache()
|
||||
|
||||
# Load the requested expert lazily so its ``LoadedModel`` handle is
|
||||
# always fresh — see class docstring for the cache-eviction reasoning.
|
||||
model_id = self._high_model if label == self.HIGH else self._low_model
|
||||
info = self._context.models.load(model_id)
|
||||
device_ctx = info.model_on_device()
|
||||
cached_weights, model = device_ctx.__enter__()
|
||||
|
||||
# Stash the device-context state immediately. If anything below fails (most
|
||||
# likely the LoRA patcher), the surrounding ExitStack will eventually call
|
||||
# ``swapper.close() -> _release()`` to clean up, and ``_release`` needs to
|
||||
# see the device context to exit it. Without this early stash the expert's
|
||||
# weights stay GPU-resident (8-9 GB for GGUF experts) until the cache's LRU
|
||||
# finally evicts them, masquerading as a memory leak.
|
||||
self._active_label = label
|
||||
self._active_info = info
|
||||
self._active_device_ctx = device_ctx
|
||||
self._active_model = model
|
||||
|
||||
# Apply LoRA patches for this expert. GGUF transformers need sidecar
|
||||
# patching since direct patching of GGMLTensors isn't supported.
|
||||
lora_factory = self._high_lora_factory if label == self.HIGH else self._low_lora_factory
|
||||
is_quantized = self._high_is_quantized if label == self.HIGH else self._low_is_quantized
|
||||
lora_ctx: Any | None = None
|
||||
if lora_factory is not None:
|
||||
lora_ctx = LayerPatcher.apply_smart_model_patches(
|
||||
model=model,
|
||||
patches=lora_factory(),
|
||||
prefix=WAN_LORA_TRANSFORMER_PREFIX,
|
||||
dtype=self._inference_dtype,
|
||||
cached_weights=cached_weights,
|
||||
force_sidecar_patching=is_quantized,
|
||||
)
|
||||
lora_ctx.__enter__()
|
||||
|
||||
self._active_lora_ctx = lora_ctx
|
||||
return model
|
||||
|
||||
def _release(self) -> None:
|
||||
# LoRA context first so weights are restored before the model leaves GPU. The
|
||||
# device context must exit even if weight-restore raises — otherwise the cache
|
||||
# record stays locked and the expert is pinned in VRAM for the whole process.
|
||||
# The field-clearing sits in its own finally so that a device-exit failure
|
||||
# cannot leave stale contexts behind: a later close() would double-exit the
|
||||
# already-exited LoRA context, and get() could hand back a stale model.
|
||||
try:
|
||||
if self._active_lora_ctx is not None:
|
||||
self._active_lora_ctx.__exit__(None, None, None)
|
||||
finally:
|
||||
try:
|
||||
if self._active_device_ctx is not None:
|
||||
self._active_device_ctx.__exit__(None, None, None)
|
||||
finally:
|
||||
self._active_label = None
|
||||
self._active_info = None
|
||||
self._active_device_ctx = None
|
||||
self._active_lora_ctx = None
|
||||
self._active_model = None
|
||||
|
||||
def close(self) -> None:
|
||||
self._release()
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_denoise",
|
||||
title="Denoise - Wan 2.2",
|
||||
tags=["image", "wan"],
|
||||
category="image",
|
||||
version="1.0.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanDenoiseInvocation(BaseInvocation):
|
||||
"""Run the denoising process with a Wan 2.2 model.
|
||||
|
||||
Drives the flow-matching schedule the model ships with (UniPC with flow
|
||||
sigmas for the Wan 2.2 reference repos; see ``_build_scheduler``). CFG is
|
||||
supported when negative conditioning is provided and ``guidance_scale != 1.0``.
|
||||
|
||||
For Wan 2.2 A14B the high-noise expert handles timesteps at and above
|
||||
``boundary_ratio * num_train_timesteps``; the low-noise expert handles
|
||||
timesteps below. Both experts share the model cache; only the active one is
|
||||
GPU-resident at any time.
|
||||
"""
|
||||
|
||||
transformer: WanTransformerField = InputField(
|
||||
description="Wan transformer field (transformer + optional dual-expert metadata).",
|
||||
input=Input.Connection,
|
||||
title="Transformer",
|
||||
)
|
||||
positive_conditioning: WanConditioningField = InputField(
|
||||
description=FieldDescriptions.positive_cond, input=Input.Connection
|
||||
)
|
||||
negative_conditioning: Optional[WanConditioningField] = InputField(
|
||||
default=None, description=FieldDescriptions.negative_cond, input=Input.Connection
|
||||
)
|
||||
|
||||
ref_image: Optional[WanRefImageConditioningField] = InputField(
|
||||
default=None,
|
||||
description=FieldDescriptions.wan_ref_image,
|
||||
input=Input.Connection,
|
||||
title="Reference Image",
|
||||
)
|
||||
|
||||
latents: Optional[LatentsField] = InputField(
|
||||
default=None,
|
||||
description=FieldDescriptions.latents,
|
||||
input=Input.Connection,
|
||||
)
|
||||
denoise_mask: Optional[DenoiseMaskField] = InputField(
|
||||
default=None,
|
||||
description=FieldDescriptions.denoise_mask,
|
||||
input=Input.Connection,
|
||||
)
|
||||
|
||||
denoising_start: float = InputField(default=0.0, ge=0, le=1, description=FieldDescriptions.denoising_start)
|
||||
denoising_end: float = InputField(default=1.0, ge=0, le=1, description=FieldDescriptions.denoising_end)
|
||||
add_noise: bool = InputField(default=True, description="Add noise based on denoising start.")
|
||||
|
||||
guidance_scale: float = InputField(
|
||||
default=4.0,
|
||||
ge=1.0,
|
||||
description="Classifier-free guidance scale. 4.0 is the Wan 2.2 default for A14B; "
|
||||
"TI2V-5B can tolerate higher values up to ~5.5.",
|
||||
title="Guidance Scale",
|
||||
)
|
||||
guidance_scale_low_noise: Optional[float] = InputField(
|
||||
default=None,
|
||||
ge=0.0,
|
||||
description="Optional separate CFG scale for the low-noise expert (Wan 2.2 A14B only). "
|
||||
"Values below 1.0 (including 0) fall back to the primary 'Guidance Scale'. "
|
||||
"Ignored for TI2V-5B.",
|
||||
title="Guidance Scale (Low Noise)",
|
||||
)
|
||||
# Wan transformer has ``patch_size=(1, 2, 2)``: combined with the VAE's
|
||||
# 8x spatial scale, generated H/W must be a multiple of 16 (not just 8)
|
||||
# or the patch round-trip lands off-by-one and the scheduler step fails
|
||||
# with a spatial-dim mismatch.
|
||||
width: int = InputField(default=1024, gt=0, multiple_of=16, description="Width of the generated image.")
|
||||
height: int = InputField(default=1024, gt=0, multiple_of=16, description="Height of the generated image.")
|
||||
steps: int = InputField(default=40, gt=0, description="Number of denoising steps.")
|
||||
seed: int = InputField(default=0, description="Randomness seed for reproducibility.")
|
||||
|
||||
@torch.no_grad()
|
||||
def invoke(self, context: InvocationContext) -> LatentsOutput:
|
||||
latents = self._run_diffusion(context)
|
||||
latents = latents.detach().to("cpu")
|
||||
name = context.tensors.save(tensor=latents)
|
||||
return LatentsOutput.build(latents_name=name, latents=latents, seed=None)
|
||||
|
||||
def _run_diffusion(self, context: InvocationContext) -> torch.Tensor:
|
||||
if self.denoising_start >= self.denoising_end:
|
||||
raise ValueError(
|
||||
f"denoising_start ({self.denoising_start}) must be less than denoising_end ({self.denoising_end})."
|
||||
)
|
||||
|
||||
device = TorchDevice.choose_torch_device()
|
||||
inference_dtype = TorchDevice.choose_bfloat16_safe_dtype(device)
|
||||
|
||||
variant = _resolve_variant(context, self.transformer)
|
||||
_validate_spatial_dimensions(variant, self.width, self.height)
|
||||
spatial_scale = get_spatial_scale_factor(variant)
|
||||
|
||||
scheduler = self._build_scheduler(context, device)
|
||||
|
||||
pos_cond = self._load_conditioning(context, self.positive_conditioning, device=device, dtype=inference_dtype)
|
||||
low_cfg_enabled = (
|
||||
self.transformer.transformer_low_noise is not None
|
||||
and self.guidance_scale_low_noise is not None
|
||||
and self.guidance_scale_low_noise >= 1.0
|
||||
and self.guidance_scale_low_noise != 1.0
|
||||
)
|
||||
do_cfg = self.negative_conditioning is not None and (self.guidance_scale != 1.0 or low_cfg_enabled)
|
||||
neg_cond: WanConditioningInfo | None = None
|
||||
if do_cfg:
|
||||
assert self.negative_conditioning is not None
|
||||
neg_cond = self._load_conditioning(
|
||||
context, self.negative_conditioning, device=device, dtype=inference_dtype
|
||||
)
|
||||
|
||||
# Reference-image conditioning (Wan 2.2 I2V-A14B only). The condition
|
||||
# tensor is 20 channels (4 mask + 16 VAE-encoded image latents); it
|
||||
# gets concatenated to the 16-channel noise latents each step,
|
||||
# yielding the 36-channel input the I2V transformer expects.
|
||||
ref_condition: torch.Tensor | None = None
|
||||
if self.ref_image is not None:
|
||||
if variant != WanVariantType.I2V_A14B:
|
||||
raise ValueError(
|
||||
f"Reference-image conditioning is only supported by the Wan 2.2 I2V variant. "
|
||||
f"The selected transformer is {variant.value!r}. Remove the Reference Image input "
|
||||
"or load an I2V model."
|
||||
)
|
||||
if self.ref_image.width != self.width or self.ref_image.height != self.height:
|
||||
raise ValueError(
|
||||
f"Reference-image dimensions ({self.ref_image.width}x{self.ref_image.height}) must "
|
||||
f"match denoise dimensions ({self.width}x{self.height})."
|
||||
)
|
||||
if self.ref_image.num_frames > 1:
|
||||
# The image denoise produces single-frame output; concatenating a multi-frame
|
||||
# condition to a single-frame noise tensor mismatches the temporal dim and the
|
||||
# downstream tensor-shape error would be unhelpful.
|
||||
raise ValueError(
|
||||
f"This denoise node produces a single-frame image but the reference image was "
|
||||
f"encoded for {self.ref_image.num_frames} frames. Use the Denoise Video - Wan 2.2 "
|
||||
"node for video I2V, or set num_frames=1 on the Reference Image node."
|
||||
)
|
||||
ref_condition = context.tensors.load(self.ref_image.condition_tensor_name).to(
|
||||
device=device, dtype=inference_dtype
|
||||
)
|
||||
_validate_ref_condition_shape(
|
||||
ref_condition,
|
||||
channels=20,
|
||||
frames=1,
|
||||
height=self.height // spatial_scale,
|
||||
width=self.width // spatial_scale,
|
||||
)
|
||||
|
||||
# Schedule timesteps. set_timesteps populates scheduler.timesteps and
|
||||
# scheduler.sigmas (where sigmas is in [0, 1] flow-matching space).
|
||||
scheduler.set_timesteps(num_inference_steps=self.steps, device=device)
|
||||
timesteps = scheduler.timesteps
|
||||
# sigmas has length steps + 1.
|
||||
sigmas = scheduler.sigmas
|
||||
|
||||
# Apply denoising_start / denoising_end clipping.
|
||||
if self.denoising_start > 0 or self.denoising_end < 1:
|
||||
start_idx = int(self.denoising_start * self.steps)
|
||||
end_idx = int(self.denoising_end * self.steps)
|
||||
timesteps = timesteps[start_idx:end_idx]
|
||||
sigmas = sigmas[start_idx : end_idx + 1]
|
||||
total_steps = len(timesteps)
|
||||
|
||||
# Latents stay in fp32 throughout the denoise loop to avoid accumulating
|
||||
# bf16 quantization across the scheduler's small per-step deltas. We
|
||||
# cast to bf16 only when calling the transformer, matching Diffusers'
|
||||
# WanPipeline (which calls ``prepare_latents(..., dtype=torch.float32)``
|
||||
# then ``latent_model_input = latents.to(transformer_dtype)``).
|
||||
latent_dtype = torch.float32
|
||||
|
||||
# Load init latents (img2img) and convert 4D → 5D.
|
||||
init_latents_5d: torch.Tensor | None = None
|
||||
if self.latents is not None:
|
||||
loaded = context.tensors.load(self.latents.latents_name).to(device=device, dtype=latent_dtype)
|
||||
if loaded.ndim not in (4, 5):
|
||||
raise ValueError(
|
||||
f"Wan image denoise expects a 4D or 5D initial latent tensor; got {tuple(loaded.shape)}."
|
||||
)
|
||||
if loaded.shape[0] != 1:
|
||||
raise ValueError(
|
||||
f"Wan image denoise requires initial latents with batch size 1; got {loaded.shape[0]}."
|
||||
)
|
||||
if loaded.ndim == 5 and loaded.shape[2] != 1:
|
||||
raise ValueError(
|
||||
f"Wan image denoise requires single-frame initial latents; got {loaded.shape[2]} frames."
|
||||
)
|
||||
if loaded.ndim == 4:
|
||||
loaded = loaded.unsqueeze(2)
|
||||
expected_channels = 48 if variant == WanVariantType.TI2V_5B else 16
|
||||
if loaded.shape[1] != expected_channels:
|
||||
raise ValueError(
|
||||
f"Wan {variant.value} image denoise expects {expected_channels} channels in initial latents; "
|
||||
f"got {loaded.shape[1]}."
|
||||
)
|
||||
expected_height = self.height // spatial_scale
|
||||
expected_width = self.width // spatial_scale
|
||||
if loaded.shape[-2:] != (expected_height, expected_width):
|
||||
raise ValueError(
|
||||
f"Wan image denoise expects initial latent dimensions {expected_height}x{expected_width}; "
|
||||
f"got {loaded.shape[-2]}x{loaded.shape[-1]}."
|
||||
)
|
||||
init_latents_5d = loaded
|
||||
|
||||
# Determine the latent channel count. Prefer init_latents shape; otherwise
|
||||
# fall back to the variant default. (We avoid loading the transformer just
|
||||
# to read .config.in_channels; the variant gives us the right answer.)
|
||||
latent_channels = (
|
||||
init_latents_5d.shape[1]
|
||||
if init_latents_5d is not None
|
||||
else (48 if variant == WanVariantType.TI2V_5B else 16)
|
||||
)
|
||||
|
||||
noise = make_noise(
|
||||
batch_size=1,
|
||||
latent_channels=latent_channels,
|
||||
height=self.height,
|
||||
width=self.width,
|
||||
spatial_scale_factor=spatial_scale,
|
||||
device=device,
|
||||
dtype=latent_dtype,
|
||||
seed=self.seed,
|
||||
)
|
||||
|
||||
# Combine init latents + noise per the schedule's starting sigma.
|
||||
if init_latents_5d is not None:
|
||||
if self.add_noise:
|
||||
s_0 = float(sigmas[0])
|
||||
latents = s_0 * noise + (1.0 - s_0) * init_latents_5d
|
||||
else:
|
||||
latents = init_latents_5d
|
||||
else:
|
||||
if self.denoising_start > 1e-5:
|
||||
raise ValueError("denoising_start should be 0 when initial latents are not provided.")
|
||||
latents = noise
|
||||
|
||||
if total_steps <= 0:
|
||||
return latents.squeeze(2)
|
||||
|
||||
# Inpaint extension (4D space — the existing extension is shape-agnostic
|
||||
# but operates on the squeezed-T shape we use for masks).
|
||||
inpaint_mask = self._prep_inpaint_mask(context, latents.squeeze(2))
|
||||
inpaint_extension: RectifiedFlowInpaintExtension | None = None
|
||||
if inpaint_mask is not None:
|
||||
if init_latents_5d is None:
|
||||
raise ValueError("Initial latents are required when using an inpaint mask (img2img inpainting).")
|
||||
inpaint_extension = RectifiedFlowInpaintExtension(
|
||||
init_latents=init_latents_5d.squeeze(2),
|
||||
inpaint_mask=inpaint_mask,
|
||||
noise=noise.squeeze(2),
|
||||
)
|
||||
|
||||
step_callback = self._build_step_callback(context)
|
||||
|
||||
# Resolve experts and the boundary timestep that triggers the MoE swap.
|
||||
#
|
||||
# We deliberately do NOT call ``context.models.load(...)`` for the
|
||||
# transformer experts here — that would put both ~9 GB GGUF handles
|
||||
# in the model cache concurrently. With UMT5-XXL (~10 GB) competing
|
||||
# for the same cache, the LRU policy can drop one of them by the
|
||||
# time the denoise loop swaps in, producing the
|
||||
# "has already been dropped from the RAM cache" warning and forcing
|
||||
# a disk reload per swap. The swapper calls ``models.load`` lazily
|
||||
# inside each ``get()`` instead, so handles are always fresh.
|
||||
#
|
||||
# The config metadata (variant / format) is fine to read upfront —
|
||||
# ``get_config`` doesn't touch the weights cache.
|
||||
high_model = self.transformer.transformer
|
||||
low_model = self.transformer.transformer_low_noise
|
||||
low_config = context.models.get_config(low_model) if low_model is not None else None
|
||||
# FlowMatchEulerDiscreteScheduler stores num_train_timesteps in its config
|
||||
# (default 1000). Diffusers' WanPipeline computes:
|
||||
# boundary_timestep = boundary_ratio * num_train_timesteps
|
||||
num_train_timesteps = int(scheduler.config.num_train_timesteps)
|
||||
boundary_timestep = self.transformer.boundary_ratio * num_train_timesteps if low_model is not None else None
|
||||
|
||||
# LoRA wiring. The high-noise expert uses ``transformer.loras``; the
|
||||
# low-noise expert uses ``transformer.loras_low_noise``. No fallback
|
||||
# between the lists: the Wan LoRA loader routes untagged LoRAs to both
|
||||
# and expert-tagged LoRAs to exactly one, so an empty low list means
|
||||
# "no LoRAs on the low expert" — reusing the primary list here would
|
||||
# silently apply high-expert-only LoRAs (e.g. Lightning high-noise
|
||||
# distills) to the wrong expert.
|
||||
# Quantized (GGUF) experts force sidecar patching so GGMLTensor weights
|
||||
# aren't touched directly.
|
||||
high_loras = self.transformer.loras
|
||||
low_loras = self.transformer.loras_low_noise
|
||||
high_config = context.models.get_config(high_model)
|
||||
high_is_quantized = high_config.format == ModelFormat.GGUFQuantized
|
||||
low_is_quantized = low_config.format == ModelFormat.GGUFQuantized if low_config is not None else False
|
||||
|
||||
def high_lora_factory() -> Iterable[Tuple[ModelPatchRaw, float]]:
|
||||
return self._lora_iterator(context, high_loras)
|
||||
|
||||
def low_lora_factory() -> Iterable[Tuple[ModelPatchRaw, float]]:
|
||||
return self._lora_iterator(context, low_loras)
|
||||
|
||||
with ExitStack() as exit_stack:
|
||||
swapper = _ExpertSwapper(
|
||||
context=context,
|
||||
high_model=high_model,
|
||||
low_model=low_model,
|
||||
inference_dtype=inference_dtype,
|
||||
high_lora_factory=high_lora_factory if high_loras else None,
|
||||
low_lora_factory=low_lora_factory if low_loras else None,
|
||||
high_is_quantized=high_is_quantized,
|
||||
low_is_quantized=low_is_quantized,
|
||||
)
|
||||
exit_stack.callback(swapper.close)
|
||||
|
||||
for step_idx, t in enumerate(tqdm(timesteps, desc="Denoising (Wan 2.2)", total=total_steps)):
|
||||
timestep = t.expand(latents.shape[0])
|
||||
|
||||
# Pick the active expert: high-noise for t >= boundary_timestep,
|
||||
# low-noise below. Single-transformer models always use HIGH.
|
||||
if low_model is not None and float(t) < float(boundary_timestep):
|
||||
active_label = _ExpertSwapper.LOW
|
||||
# Treat None or values below 1.0 (incl. the FE's default 0)
|
||||
# as "use the primary guidance_scale".
|
||||
low_cfg = self.guidance_scale_low_noise
|
||||
active_cfg = low_cfg if (low_cfg is not None and low_cfg >= 1.0) else self.guidance_scale
|
||||
else:
|
||||
active_label = _ExpertSwapper.HIGH
|
||||
active_cfg = self.guidance_scale
|
||||
|
||||
transformer = swapper.get(active_label)
|
||||
|
||||
# Cast latents to the transformer's dtype only for the forward
|
||||
# pass; keep the scheduler-level latents in fp32.
|
||||
latent_model_input = latents.to(dtype=inference_dtype)
|
||||
|
||||
# For I2V, concatenate the ref-image condition (4-ch mask + 16-ch
|
||||
# image latents) along the channel dim, producing the 36-channel
|
||||
# input the I2V transformer's patch_embedding expects.
|
||||
if ref_condition is not None:
|
||||
latent_model_input = torch.cat([latent_model_input, ref_condition], dim=1)
|
||||
|
||||
noise_pred_cond = transformer(
|
||||
hidden_states=latent_model_input,
|
||||
timestep=timestep,
|
||||
encoder_hidden_states=pos_cond.prompt_embeds.unsqueeze(0),
|
||||
attention_kwargs=None,
|
||||
return_dict=False,
|
||||
)[0]
|
||||
|
||||
if neg_cond is not None and active_cfg != 1.0:
|
||||
noise_pred_uncond = transformer(
|
||||
hidden_states=latent_model_input,
|
||||
timestep=timestep,
|
||||
encoder_hidden_states=neg_cond.prompt_embeds.unsqueeze(0),
|
||||
attention_kwargs=None,
|
||||
return_dict=False,
|
||||
)[0]
|
||||
noise_pred = noise_pred_uncond + active_cfg * (noise_pred_cond - noise_pred_uncond)
|
||||
else:
|
||||
noise_pred = noise_pred_cond
|
||||
|
||||
latents = scheduler.step(noise_pred, t, latents, return_dict=False)[0]
|
||||
|
||||
if inpaint_extension is not None:
|
||||
sigma_prev = float(sigmas[step_idx + 1])
|
||||
latents_4d = latents.squeeze(2)
|
||||
latents_4d = inpaint_extension.merge_intermediate_latents_with_init_latents(latents_4d, sigma_prev)
|
||||
latents = latents_4d.unsqueeze(2)
|
||||
|
||||
step_callback(
|
||||
PipelineIntermediateState(
|
||||
step=step_idx + 1,
|
||||
order=1,
|
||||
total_steps=total_steps,
|
||||
timestep=int(t.item()),
|
||||
latents=latents.squeeze(2),
|
||||
)
|
||||
)
|
||||
|
||||
# Squeeze T for downstream 4D consumers.
|
||||
return latents.squeeze(2)
|
||||
|
||||
def _build_scheduler(self, context: InvocationContext, device: torch.device):
|
||||
"""Construct the scheduler matching the model's on-disk ``scheduler_config.json``.
|
||||
|
||||
All Wan 2.2 variants ship ``UniPCMultistepScheduler`` with flow-matching
|
||||
settings, but with different exponential shifts (TI2V-5B: ``flow_shift=5.0``,
|
||||
A14B: ``3.0``) — and future variants may differ further, so when a
|
||||
``scheduler/`` directory exists we dispatch on its ``_class_name`` and load
|
||||
it verbatim. When no on-disk config is available (standalone GGUF /
|
||||
single-file installs don't ship one) — or the config is unreadable — fall
|
||||
back to the variant-aware default so the noise schedule still matches what
|
||||
the model was trained against.
|
||||
"""
|
||||
import json
|
||||
|
||||
import diffusers
|
||||
|
||||
scheduler_dir = _scheduler_path_for_transformer(context, self.transformer)
|
||||
scheduler_cls = None
|
||||
if scheduler_dir is not None:
|
||||
# Read the on-disk class name and instantiate that class. Diffusers'
|
||||
# SchedulerMixin.from_pretrained does class dispatch internally, but
|
||||
# only when called from the abstract base; calling a concrete subclass
|
||||
# silently builds the wrong type. Resolve it explicitly.
|
||||
config_path = scheduler_dir / "scheduler_config.json"
|
||||
try:
|
||||
with config_path.open("r", encoding="utf-8") as f:
|
||||
cfg = json.load(f)
|
||||
class_name = cfg.get("_class_name")
|
||||
scheduler_cls = getattr(diffusers, class_name, None) if class_name else None
|
||||
except (OSError, json.JSONDecodeError):
|
||||
scheduler_cls = None
|
||||
|
||||
if scheduler_dir is None or scheduler_cls is None:
|
||||
variant = _resolve_variant(context, self.transformer)
|
||||
return _default_scheduler_for_variant(variant)
|
||||
|
||||
return scheduler_cls.from_pretrained(str(scheduler_dir), local_files_only=True)
|
||||
|
||||
def _load_conditioning(
|
||||
self,
|
||||
context: InvocationContext,
|
||||
cond_field: WanConditioningField,
|
||||
*,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
) -> WanConditioningInfo:
|
||||
cond_data = context.conditioning.load(cond_field.conditioning_name)
|
||||
assert len(cond_data.conditionings) == 1
|
||||
cond_info = cond_data.conditionings[0]
|
||||
assert isinstance(cond_info, WanConditioningInfo)
|
||||
return cond_info.to(device=device, dtype=dtype)
|
||||
|
||||
def _prep_inpaint_mask(self, context: InvocationContext, latents_4d: torch.Tensor) -> torch.Tensor | None:
|
||||
"""Resize the user-supplied mask down to latent resolution.
|
||||
|
||||
Convention matches Anima/FLUX: the original mask has 0 = preserve and
|
||||
1 = denoise; the extension expects the inverted form.
|
||||
"""
|
||||
if self.denoise_mask is None:
|
||||
return None
|
||||
mask = context.tensors.load(self.denoise_mask.mask_name)
|
||||
mask = 1.0 - mask
|
||||
_, _, latent_h, latent_w = latents_4d.shape
|
||||
mask = tv_resize(
|
||||
img=mask,
|
||||
size=[latent_h, latent_w],
|
||||
interpolation=tv_transforms.InterpolationMode.BILINEAR,
|
||||
antialias=False,
|
||||
)
|
||||
return mask.to(device=latents_4d.device, dtype=latents_4d.dtype)
|
||||
|
||||
def _build_step_callback(self, context: InvocationContext) -> Callable[[PipelineIntermediateState], None]:
|
||||
def step_callback(state: PipelineIntermediateState) -> None:
|
||||
context.util.sd_step_callback(state, BaseModelType.Wan)
|
||||
|
||||
return step_callback
|
||||
|
||||
def _lora_iterator(
|
||||
self, context: InvocationContext, loras: list[LoRAField]
|
||||
) -> Iterator[Tuple[ModelPatchRaw, float]]:
|
||||
"""Yield (ModelPatchRaw, weight) pairs for the given LoRA list.
|
||||
|
||||
The caller passes either ``transformer.loras`` (high-noise expert) or
|
||||
``transformer.loras_low_noise`` (low-noise expert) — the fallback to
|
||||
the primary list when low-noise is empty is handled at the call site.
|
||||
"""
|
||||
for lora_field in loras:
|
||||
lora_info = context.models.load(lora_field.lora)
|
||||
assert isinstance(lora_info.model, ModelPatchRaw), (
|
||||
f"Wan LoRA model must be ModelPatchRaw, got {type(lora_info.model).__name__}"
|
||||
)
|
||||
yield (lora_info.model, lora_field.weight)
|
||||
del lora_info
|
||||
@@ -0,0 +1,209 @@
|
||||
"""Compute Wan 2.2-compatible pixel dimensions for a target short-side resolution.
|
||||
|
||||
Wan's transformer ``patch_size=(1, 2, 2)`` adds a 2x patchify on top of the VAE's
|
||||
spatial compression, so pixel dimensions must be a multiple of (2 × VAE scale):
|
||||
|
||||
- I2V-A14B / T2V (8x VAE) → multiples of 16
|
||||
- TI2V-5B (Wan 2.2-VAE, 16x VAE) → multiples of 32
|
||||
|
||||
This module exposes one node per family. Each takes a source image's W×H and a
|
||||
target short-side preset (480p / 720p / 1080p) and returns the scaled, snapped
|
||||
(width, height) that can be fed directly into the matching ``wan_ref_image_encoder``
|
||||
/ ``wan_denoise`` inputs. Both nodes share :func:`_scale_and_snap`; they differ
|
||||
only in the pixel multiple they snap to.
|
||||
"""
|
||||
|
||||
import math
|
||||
from typing import Literal
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
|
||||
from invokeai.app.invocations.fields import InputField
|
||||
from invokeai.app.invocations.ideal_size import IdealSizeOutput
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
|
||||
WanTargetResolution = Literal["480p", "720p", "1080p"]
|
||||
WanRounding = Literal["nearest", "floor", "ceiling"]
|
||||
|
||||
# Pixel-grid multiple = 2 (transformer patch) × VAE spatial scale. The 8x-VAE
|
||||
# I2V/T2V models need multiples of 16; the 16x-VAE TI2V-5B needs multiples of 32.
|
||||
WAN_I2V_PIXEL_MULTIPLE = 16
|
||||
WAN_TI2V_PIXEL_MULTIPLE = 32
|
||||
|
||||
# Short-side pixel count for each preset. "p" notation is by convention the *short*
|
||||
# dimension in modern video (so a portrait 720p video is 720 wide × 1280 tall).
|
||||
WAN_TARGET_RESOLUTION_PX: dict[str, int] = {
|
||||
"480p": 480,
|
||||
"720p": 720,
|
||||
"1080p": 1080,
|
||||
}
|
||||
|
||||
WAN_TARGET_RESOLUTION_LABELS: dict[str, str] = {
|
||||
"480p": "480p (Wan native)",
|
||||
"720p": "720p (Wan native, default)",
|
||||
"1080p": "1080p (extrapolated — not a Wan training size)",
|
||||
}
|
||||
|
||||
|
||||
def _scale_and_snap(
|
||||
width: int,
|
||||
height: int,
|
||||
target_short_side: int,
|
||||
rounding: WanRounding,
|
||||
multiple: int,
|
||||
) -> tuple[int, int]:
|
||||
"""Scale a source W×H so its shorter side equals ``target_short_side``, then
|
||||
snap each dimension to ``multiple`` using the requested rounding mode.
|
||||
|
||||
``multiple`` is the Wan pixel-grid constraint (16 for the 8x-VAE I2V/T2V
|
||||
models, 32 for the 16x-VAE TI2V-5B). Shared by both ideal-dimensions nodes.
|
||||
"""
|
||||
short = min(width, height)
|
||||
if short <= 0:
|
||||
raise ValueError("Source dimensions must be positive.")
|
||||
|
||||
# Reject sources so narrow that the scaled long side is still under one Wan
|
||||
# pixel grid. The downstream clamp to ``max(w, multiple)`` would otherwise
|
||||
# silently return multiple×multiple, which has no relation to the requested
|
||||
# aspect ratio — better to fail fast and have the workflow author fix inputs.
|
||||
long_side = max(width, height)
|
||||
if long_side < multiple:
|
||||
raise ValueError(
|
||||
f"Source longer side ({long_side}px) is smaller than the Wan pixel grid ({multiple}px). "
|
||||
f"Use an input image at least {multiple}px on its longer side."
|
||||
)
|
||||
|
||||
scale = target_short_side / short
|
||||
raw_w = width * scale
|
||||
raw_h = height * scale
|
||||
|
||||
if rounding == "floor":
|
||||
w = int(raw_w // multiple) * multiple
|
||||
h = int(raw_h // multiple) * multiple
|
||||
elif rounding == "ceiling":
|
||||
w = int(math.ceil(raw_w / multiple)) * multiple
|
||||
h = int(math.ceil(raw_h / multiple)) * multiple
|
||||
else: # nearest
|
||||
w = round(raw_w / multiple) * multiple
|
||||
h = round(raw_h / multiple) * multiple
|
||||
|
||||
# Belt-and-suspenders clamp against floor-of-<multiple — unreachable now that
|
||||
# the long_side guard runs first, but keeps the contract that returned
|
||||
# dimensions are always valid Wan inputs.
|
||||
w = max(w, multiple)
|
||||
h = max(h, multiple)
|
||||
return w, h
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_i2v_ideal_dimensions",
|
||||
title="Wan 2.2 I2V Ideal Dimensions (A14B)",
|
||||
tags=["wan", "video", "dimensions", "math"],
|
||||
category="video",
|
||||
version="1.1.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanI2VIdealDimensionsInvocation(BaseInvocation):
|
||||
"""Ideal dimensions for the Wan 2.2 A14B models (I2V-A14B and T2V-A14B).
|
||||
|
||||
Use this node for the A14B family. For the TI2V-5B model use "Wan 2.2 TI2V
|
||||
Ideal Dimensions" instead — TI2V-5B requires multiples of 32, and feeding it
|
||||
these multiples-of-16 dims fails the patchify step.
|
||||
|
||||
Scales the input W×H so the shorter side equals the chosen preset (480 / 720 /
|
||||
1080 px), then snaps each dimension to a multiple of 16 (the A14B pixel-grid
|
||||
constraint). Wire from ``Image Primitive``'s width/height outputs and into
|
||||
``wan_ref_image_encoder`` / ``wan_denoise``.
|
||||
"""
|
||||
|
||||
width: int = InputField(
|
||||
default=1024,
|
||||
gt=0,
|
||||
description="Source image width in pixels.",
|
||||
)
|
||||
height: int = InputField(
|
||||
default=1024,
|
||||
gt=0,
|
||||
description="Source image height in pixels.",
|
||||
)
|
||||
target_resolution: WanTargetResolution = InputField(
|
||||
default="720p",
|
||||
description=(
|
||||
"Short-side resolution preset. 480p and 720p are Wan 2.2's native training "
|
||||
"resolutions; 1080p works but is extrapolation and costs ~2.25x the memory "
|
||||
"of 720p."
|
||||
),
|
||||
ui_choice_labels=WAN_TARGET_RESOLUTION_LABELS,
|
||||
)
|
||||
rounding: WanRounding = InputField(
|
||||
default="nearest",
|
||||
description=(
|
||||
"How to snap each dimension to a multiple of 16. 'floor' rounds down — "
|
||||
"safest for VRAM, guaranteed not to exceed the unsnapped target. "
|
||||
"'ceiling' rounds up. 'nearest' minimizes aspect-ratio drift (default)."
|
||||
),
|
||||
)
|
||||
|
||||
def invoke(self, context: InvocationContext) -> IdealSizeOutput:
|
||||
target_short_side = WAN_TARGET_RESOLUTION_PX[self.target_resolution]
|
||||
w, h = _scale_and_snap(
|
||||
self.width, self.height, target_short_side, self.rounding, multiple=WAN_I2V_PIXEL_MULTIPLE
|
||||
)
|
||||
return IdealSizeOutput(width=w, height=h)
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_ti2v_ideal_dimensions",
|
||||
title="Wan 2.2 TI2V Ideal Dimensions (5B)",
|
||||
tags=["wan", "video", "dimensions", "math"],
|
||||
category="video",
|
||||
version="1.0.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanTI2VIdealDimensionsInvocation(BaseInvocation):
|
||||
"""Ideal dimensions for the Wan 2.2 TI2V-5B model.
|
||||
|
||||
Use this node for TI2V-5B only. For the A14B models (I2V-A14B / T2V-A14B) use
|
||||
"Wan 2.2 I2V Ideal Dimensions" instead — those need multiples of 16, and this
|
||||
node's multiples-of-32 dims would overshoot their pixel grid.
|
||||
|
||||
Identical to the A14B node but snaps each dimension to a multiple of 32 instead
|
||||
of 16: the Wan 2.2-VAE used by TI2V-5B applies 16x spatial compression and the
|
||||
transformer adds a 2x patch on top, so pixel dims must divide by 32 for the
|
||||
patchify step. Wire from ``Image Primitive``'s width/height outputs and into
|
||||
the matching ``wan_denoise`` inputs.
|
||||
"""
|
||||
|
||||
width: int = InputField(
|
||||
default=1024,
|
||||
gt=0,
|
||||
description="Source image width in pixels.",
|
||||
)
|
||||
height: int = InputField(
|
||||
default=1024,
|
||||
gt=0,
|
||||
description="Source image height in pixels.",
|
||||
)
|
||||
target_resolution: WanTargetResolution = InputField(
|
||||
default="720p",
|
||||
description=(
|
||||
"Short-side resolution preset. 480p and 720p are Wan 2.2's native training "
|
||||
"resolutions; 1080p works but is extrapolation and costs ~2.25x the memory "
|
||||
"of 720p."
|
||||
),
|
||||
ui_choice_labels=WAN_TARGET_RESOLUTION_LABELS,
|
||||
)
|
||||
rounding: WanRounding = InputField(
|
||||
default="nearest",
|
||||
description=(
|
||||
"How to snap each dimension to a multiple of 32. 'floor' rounds down — "
|
||||
"safest for VRAM, guaranteed not to exceed the unsnapped target. "
|
||||
"'ceiling' rounds up. 'nearest' minimizes aspect-ratio drift (default)."
|
||||
),
|
||||
)
|
||||
|
||||
def invoke(self, context: InvocationContext) -> IdealSizeOutput:
|
||||
target_short_side = WAN_TARGET_RESOLUTION_PX[self.target_resolution]
|
||||
w, h = _scale_and_snap(
|
||||
self.width, self.height, target_short_side, self.rounding, multiple=WAN_TI2V_PIXEL_MULTIPLE
|
||||
)
|
||||
return IdealSizeOutput(width=w, height=h)
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Wan 2.2 image-to-latents invocation.
|
||||
|
||||
Encodes an image to latent space using the Wan VAE (AutoencoderKLWan). The Wan
|
||||
VAE expects 5D ``[B, C, T, H, W]`` input with ``T=1`` for single images. After
|
||||
encoding, latents are normalised against the per-channel ``latents_mean`` and
|
||||
``latents_std`` stored in the VAE config — this matches the Diffusers
|
||||
``WanPipeline`` reference and is the inverse of the denormalisation in
|
||||
``wan_latents_to_image.py``.
|
||||
"""
|
||||
|
||||
import einops
|
||||
import torch
|
||||
from diffusers.models.autoencoders import AutoencoderKLWan
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
|
||||
from invokeai.app.invocations.fields import (
|
||||
FieldDescriptions,
|
||||
ImageField,
|
||||
Input,
|
||||
InputField,
|
||||
WithBoard,
|
||||
WithMetadata,
|
||||
)
|
||||
from invokeai.app.invocations.model import VAEField
|
||||
from invokeai.app.invocations.primitives import LatentsOutput
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.backend.model_manager.load.load_base import LoadedModel
|
||||
from invokeai.backend.model_manager.load.model_cache.utils import get_effective_device
|
||||
from invokeai.backend.stable_diffusion.diffusers_pipeline import image_resized_to_grid_as_tensor
|
||||
from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_wan
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_i2l",
|
||||
title="Image to Latents - Wan 2.2",
|
||||
tags=["image", "latents", "vae", "i2l", "wan"],
|
||||
category="image",
|
||||
version="1.0.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanImageToLatentsInvocation(BaseInvocation, WithMetadata, WithBoard):
|
||||
"""Encodes an image with the Wan VAE (AutoencoderKLWan).
|
||||
|
||||
The output latents have the temporal dimension squeezed out, so downstream
|
||||
nodes see 4D ``[B, C, H, W]``. The denoise loop re-adds ``T=1`` before
|
||||
feeding the transformer.
|
||||
"""
|
||||
|
||||
image: ImageField = InputField(description="The image to encode.")
|
||||
vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection)
|
||||
|
||||
@staticmethod
|
||||
def vae_encode(vae_info: LoadedModel, image_tensor: torch.Tensor) -> torch.Tensor:
|
||||
if not isinstance(vae_info.model, AutoencoderKLWan):
|
||||
raise TypeError(f"Expected AutoencoderKLWan for Wan VAE, got {type(vae_info.model).__name__}.")
|
||||
|
||||
estimated_working_memory = estimate_vae_working_memory_wan(
|
||||
operation="encode",
|
||||
vae=vae_info.model,
|
||||
pixel_height=image_tensor.shape[-2],
|
||||
pixel_width=image_tensor.shape[-1],
|
||||
pixel_frames=image_tensor.shape[2] if image_tensor.ndim == 5 else 1,
|
||||
)
|
||||
|
||||
with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae):
|
||||
assert isinstance(vae, AutoencoderKLWan)
|
||||
|
||||
vae_dtype = next(iter(vae.parameters())).dtype
|
||||
image_tensor = image_tensor.to(device=get_effective_device(vae), dtype=vae_dtype)
|
||||
|
||||
with torch.inference_mode():
|
||||
# Wan VAE expects 5D [B, C, T, H, W].
|
||||
if image_tensor.ndim == 4:
|
||||
image_tensor = image_tensor.unsqueeze(2) # [B, C, H, W] -> [B, C, 1, H, W]
|
||||
|
||||
encoded = vae.encode(image_tensor, return_dict=False)[0]
|
||||
latents = encoded.sample().to(dtype=vae_dtype)
|
||||
|
||||
# Normalise to the denoiser's expected zero-centred space:
|
||||
# (latents - mean) / std
|
||||
latents_mean = torch.tensor(vae.config.latents_mean).view(1, -1, 1, 1, 1).to(latents)
|
||||
latents_std = torch.tensor(vae.config.latents_std).view(1, -1, 1, 1, 1).to(latents)
|
||||
latents = (latents - latents_mean) / latents_std
|
||||
|
||||
# Drop the temporal dim to keep the rest of the InvokeAI pipeline 4D.
|
||||
if latents.ndim == 5:
|
||||
latents = latents.squeeze(2)
|
||||
|
||||
return latents
|
||||
|
||||
@torch.no_grad()
|
||||
def invoke(self, context: InvocationContext) -> LatentsOutput:
|
||||
image = context.images.get_pil(self.image.image_name)
|
||||
|
||||
image_tensor = image_resized_to_grid_as_tensor(image.convert("RGB"))
|
||||
if image_tensor.dim() == 3:
|
||||
image_tensor = einops.rearrange(image_tensor, "c h w -> 1 c h w")
|
||||
|
||||
vae_info = context.models.load(self.vae.vae)
|
||||
|
||||
context.util.signal_progress("Running Wan VAE encode")
|
||||
latents = self.vae_encode(vae_info=vae_info, image_tensor=image_tensor)
|
||||
|
||||
latents = latents.to("cpu")
|
||||
name = context.tensors.save(tensor=latents)
|
||||
return LatentsOutput.build(latents_name=name, latents=latents, seed=None)
|
||||
@@ -0,0 +1,121 @@
|
||||
"""Wan 2.2 latents-to-image invocation.
|
||||
|
||||
Decodes Wan latents using the Wan VAE (AutoencoderKLWan).
|
||||
|
||||
Latents from the denoise loop are in normalised space (zero-centred). Before
|
||||
VAE decode they are denormalised using the VAE config's per-channel
|
||||
``latents_mean`` / ``latents_std`` (matching Diffusers ``WanPipeline``).
|
||||
|
||||
The VAE expects 5D ``[B, C, T, H, W]``; downstream nodes work with 4D, so this
|
||||
node re-adds ``T=1`` before decode and squeezes it back out afterwards.
|
||||
"""
|
||||
|
||||
import torch
|
||||
from diffusers.models.autoencoders import AutoencoderKLWan
|
||||
from einops import rearrange
|
||||
from PIL import Image
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
|
||||
from invokeai.app.invocations.fields import (
|
||||
FieldDescriptions,
|
||||
Input,
|
||||
InputField,
|
||||
LatentsField,
|
||||
WithBoard,
|
||||
WithMetadata,
|
||||
)
|
||||
from invokeai.app.invocations.model import VAEField
|
||||
from invokeai.app.invocations.primitives import ImageOutput
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.backend.model_manager.load.model_cache.utils import get_effective_device
|
||||
from invokeai.backend.util.devices import TorchDevice
|
||||
from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_wan
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_l2i",
|
||||
title="Latents to Image - Wan 2.2",
|
||||
tags=["latents", "image", "vae", "l2i", "wan"],
|
||||
category="latents",
|
||||
version="1.0.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanLatentsToImageInvocation(BaseInvocation, WithMetadata, WithBoard):
|
||||
"""Decodes Wan latents back to RGB."""
|
||||
|
||||
latents: LatentsField = InputField(description=FieldDescriptions.latents, input=Input.Connection)
|
||||
vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection)
|
||||
|
||||
@torch.no_grad()
|
||||
def invoke(self, context: InvocationContext) -> ImageOutput:
|
||||
latents = context.tensors.load(self.latents.latents_name)
|
||||
|
||||
if latents.ndim not in (4, 5):
|
||||
raise ValueError(
|
||||
f"Wan latents-to-image expects a 4D or 5D latent tensor [B, C, (T), H, W]; got {tuple(latents.shape)}."
|
||||
)
|
||||
if latents.shape[0] != 1:
|
||||
raise ValueError(f"Wan latents-to-image requires batch size 1; got {latents.shape[0]}.")
|
||||
|
||||
# This node decodes exactly one image. Multi-frame video latents would otherwise
|
||||
# run the full (expensive) multi-frame VAE decode — under a working-memory
|
||||
# estimate that assumed one frame — and then die in an opaque einops rank error
|
||||
# at the final rearrange. Checked before the VAE is even loaded.
|
||||
if latents.ndim == 5 and latents.shape[2] != 1:
|
||||
raise ValueError(
|
||||
f"These latents hold {latents.shape[2]} frames of video; this node decodes a single "
|
||||
"image. Use 'Latents to Video - Wan 2.2' (wan_l2v) for video latents."
|
||||
)
|
||||
|
||||
vae_info = context.models.load(self.vae.vae)
|
||||
if not isinstance(vae_info.model, AutoencoderKLWan):
|
||||
raise TypeError(f"Expected AutoencoderKLWan for Wan VAE, got {type(vae_info.model).__name__}.")
|
||||
|
||||
spatial_scale = getattr(vae_info.model.config, "scale_factor_spatial", None) or 8
|
||||
estimated_working_memory = estimate_vae_working_memory_wan(
|
||||
operation="decode",
|
||||
vae=vae_info.model,
|
||||
pixel_height=latents.shape[-2] * spatial_scale,
|
||||
pixel_width=latents.shape[-1] * spatial_scale,
|
||||
pixel_frames=1,
|
||||
)
|
||||
|
||||
with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae):
|
||||
context.util.signal_progress("Running Wan VAE decode")
|
||||
assert isinstance(vae, AutoencoderKLWan)
|
||||
|
||||
vae_dtype = next(iter(vae.parameters())).dtype
|
||||
latents = latents.to(device=get_effective_device(vae), dtype=vae_dtype)
|
||||
|
||||
TorchDevice.empty_cache()
|
||||
|
||||
with torch.inference_mode():
|
||||
# Re-add the temporal dim if upstream squeezed it out.
|
||||
if latents.ndim == 4:
|
||||
latents = latents.unsqueeze(2)
|
||||
|
||||
if latents.shape[1] != vae.config.z_dim:
|
||||
raise ValueError(
|
||||
f"Latent channel mismatch: these latents have {latents.shape[1]} channels but the "
|
||||
f"selected VAE expects {vae.config.z_dim}. A14B models need the 16-channel Wan 2.1 "
|
||||
"VAE; TI2V-5B needs the 48-channel Wan 2.2 VAE."
|
||||
)
|
||||
|
||||
# Denormalise from denoiser space back to raw VAE space.
|
||||
latents_mean = torch.tensor(vae.config.latents_mean).view(1, -1, 1, 1, 1).to(latents)
|
||||
latents_std = torch.tensor(vae.config.latents_std).view(1, -1, 1, 1, 1).to(latents)
|
||||
latents = latents * latents_std + latents_mean
|
||||
|
||||
decoded = vae.decode(latents, return_dict=False)[0]
|
||||
|
||||
if decoded.ndim == 5:
|
||||
decoded = decoded.squeeze(2)
|
||||
|
||||
img = decoded.clamp(-1, 1)
|
||||
img = rearrange(img[0], "c h w -> h w c")
|
||||
img_pil = Image.fromarray((127.5 * (img + 1.0)).byte().cpu().numpy())
|
||||
|
||||
TorchDevice.empty_cache()
|
||||
|
||||
image_dto = context.images.save(image=img_pil)
|
||||
return ImageOutput.build(image_dto)
|
||||
@@ -0,0 +1,230 @@
|
||||
"""Wan 2.2 latents-to-video invocation.
|
||||
|
||||
Decodes multi-frame Wan latents with the Wan VAE and encodes the result to an
|
||||
MP4 file via :mod:`imageio` (backed by the bundled FFmpeg binary from
|
||||
``imageio-ffmpeg``). The video is then persisted through ``context.videos.save``,
|
||||
which moves the temp file into ``outputs/videos/`` and records the DTO.
|
||||
|
||||
Latent shape on input is 5D ``[B, C, T_lat, H_lat, W_lat]`` (typically B=1).
|
||||
The VAE expands the temporal dim by 4× during decode minus the initial offset:
|
||||
``T_pixel = (T_lat - 1) * 4 + 1`` (e.g. T_lat=21 → 81 pixel frames).
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
from collections.abc import Iterable, Iterator
|
||||
from pathlib import Path
|
||||
from typing import Callable, Protocol
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from diffusers.models.autoencoders import AutoencoderKLWan
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
|
||||
from invokeai.app.invocations.fields import (
|
||||
FieldDescriptions,
|
||||
Input,
|
||||
InputField,
|
||||
LatentsField,
|
||||
WithBoard,
|
||||
WithMetadata,
|
||||
)
|
||||
from invokeai.app.invocations.model import VAEField
|
||||
from invokeai.app.invocations.primitives import VideoOutput
|
||||
from invokeai.app.services.session_processor.session_processor_common import CanceledException
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.app.util.video_encoding import make_mp4_writer
|
||||
from invokeai.backend.model_manager.load.model_cache.utils import get_effective_device
|
||||
from invokeai.backend.util.devices import TorchDevice
|
||||
from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_wan
|
||||
|
||||
|
||||
class _FrameWriter(Protocol):
|
||||
def append_data(self, frame: np.ndarray) -> None: ...
|
||||
|
||||
|
||||
def _validate_video_latent_batch(latents: torch.Tensor) -> None:
|
||||
if latents.ndim in (4, 5) and latents.shape[0] != 1:
|
||||
raise ValueError(f"Wan latents-to-video requires batch size 1; got {latents.shape[0]}.")
|
||||
|
||||
|
||||
def _iter_decoded_frames(decoded: torch.Tensor) -> Iterator[np.ndarray]:
|
||||
for index in range(decoded.shape[1]):
|
||||
frame = decoded[:, index]
|
||||
frame = frame.clamp(-1, 1).permute(1, 2, 0).cpu().float()
|
||||
yield (127.5 * (frame + 1.0)).round().clamp(0, 255).byte().numpy()
|
||||
|
||||
|
||||
def _write_video_frames(writer: _FrameWriter, frames: Iterable[np.ndarray], is_canceled: Callable[[], bool]) -> None:
|
||||
frames_iter = iter(frames)
|
||||
while True:
|
||||
if is_canceled():
|
||||
raise CanceledException
|
||||
try:
|
||||
frame = next(frames_iter)
|
||||
except StopIteration:
|
||||
return
|
||||
writer.append_data(np.ascontiguousarray(frame))
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_l2v",
|
||||
title="Latents to Video - Wan 2.2",
|
||||
tags=["latents", "video", "vae", "l2v", "wan"],
|
||||
category="latents",
|
||||
version="1.0.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanLatentsToVideoInvocation(BaseInvocation, WithMetadata, WithBoard):
|
||||
"""Decode 5D Wan latents to RGB frames and encode an MP4."""
|
||||
|
||||
latents: LatentsField = InputField(description=FieldDescriptions.latents, input=Input.Connection)
|
||||
vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection)
|
||||
fps: int = InputField(
|
||||
default=16,
|
||||
ge=1,
|
||||
le=120,
|
||||
description="Frames-per-second for the encoded MP4. Wan 2.2 was trained at 16 FPS.",
|
||||
)
|
||||
|
||||
@torch.no_grad()
|
||||
def invoke(self, context: InvocationContext) -> VideoOutput:
|
||||
latents = context.tensors.load(self.latents.latents_name)
|
||||
_validate_video_latent_batch(latents)
|
||||
if latents.ndim == 4:
|
||||
# Promote 4D (single-frame) to 5D so this node can also serve as a
|
||||
# one-frame "video" encode if someone wires it that way.
|
||||
latents = latents.unsqueeze(2)
|
||||
if latents.ndim != 5:
|
||||
raise ValueError(
|
||||
f"Wan latents-to-video expects a 5D latent tensor [B, C, T, H, W]; got {tuple(latents.shape)}."
|
||||
)
|
||||
if any(size == 0 for size in latents.shape[2:]):
|
||||
raise ValueError("Wan latents-to-video requires non-empty temporal and spatial dimensions.")
|
||||
|
||||
vae_info = context.models.load(self.vae.vae)
|
||||
if not isinstance(vae_info.model, AutoencoderKLWan):
|
||||
raise TypeError(f"Expected AutoencoderKLWan for Wan VAE, got {type(vae_info.model).__name__}.")
|
||||
|
||||
if latents.shape[1] != vae_info.model.config.z_dim:
|
||||
raise ValueError(
|
||||
f"Latent channel mismatch: these latents have {latents.shape[1]} channels but the "
|
||||
f"selected VAE expects {vae_info.model.config.z_dim}. A14B models need the 16-channel Wan 2.1 VAE; "
|
||||
"TI2V-5B needs the 48-channel Wan 2.2 VAE."
|
||||
)
|
||||
|
||||
_, _, t_lat, h_lat, w_lat = latents.shape
|
||||
spatial_scale = getattr(vae_info.model.config, "scale_factor_spatial", None) or 8
|
||||
temporal_scale = getattr(vae_info.model.config, "scale_factor_temporal", None) or 4
|
||||
t_pixel = (t_lat - 1) * temporal_scale + 1
|
||||
h_pixel, w_pixel = h_lat * spatial_scale, w_lat * spatial_scale
|
||||
|
||||
estimated_working_memory = estimate_vae_working_memory_wan(
|
||||
operation="decode",
|
||||
vae=vae_info.model,
|
||||
pixel_height=h_pixel,
|
||||
pixel_width=w_pixel,
|
||||
pixel_frames=t_pixel,
|
||||
)
|
||||
# Long/high-res clips can need a working set no card fits. When the full-frame
|
||||
# estimate exceeds the execution device's total VRAM, fall back to spatial tiling
|
||||
# and budget for the tiled working set instead. (A cpu_only VAE runs in system
|
||||
# RAM, where the working set is not the constraint.)
|
||||
use_tiling = False
|
||||
if not getattr(vae_info.config, "cpu_only", None):
|
||||
exec_device = TorchDevice.choose_torch_device()
|
||||
if exec_device.type == "cuda":
|
||||
total_vram = torch.cuda.get_device_properties(exec_device).total_memory
|
||||
if estimated_working_memory > 0.9 * total_vram:
|
||||
use_tiling = True
|
||||
tile_size = int(getattr(vae_info.model, "tile_sample_min_height", 256))
|
||||
estimated_working_memory = estimate_vae_working_memory_wan(
|
||||
operation="decode",
|
||||
vae=vae_info.model,
|
||||
pixel_height=h_pixel,
|
||||
pixel_width=w_pixel,
|
||||
pixel_frames=t_pixel,
|
||||
tile_size=tile_size,
|
||||
)
|
||||
|
||||
with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae):
|
||||
assert isinstance(vae, AutoencoderKLWan)
|
||||
context.logger.info(
|
||||
f"Running Wan VAE decode: {t_lat} latent frames -> {t_pixel} pixel frames at {w_pixel}x{h_pixel}"
|
||||
+ (" (tiled)" if use_tiling else "")
|
||||
)
|
||||
context.util.signal_progress("Running Wan VAE decode (video)")
|
||||
|
||||
vae_dtype = next(iter(vae.parameters())).dtype
|
||||
latents = latents.to(device=get_effective_device(vae), dtype=vae_dtype)
|
||||
|
||||
TorchDevice.empty_cache()
|
||||
|
||||
if use_tiling:
|
||||
vae.enable_tiling()
|
||||
try:
|
||||
with torch.inference_mode():
|
||||
# Denormalise from denoiser space back to VAE space.
|
||||
latents_mean = torch.tensor(vae.config.latents_mean).view(1, -1, 1, 1, 1).to(latents)
|
||||
latents_std = torch.tensor(vae.config.latents_std).view(1, -1, 1, 1, 1).to(latents)
|
||||
latents = latents * latents_std + latents_mean
|
||||
|
||||
# [B, C=3, T_pixel, H, W] in [-1, 1] (roughly).
|
||||
decoded = vae.decode(latents, return_dict=False)[0]
|
||||
del latents, latents_mean, latents_std
|
||||
finally:
|
||||
if use_tiling:
|
||||
# The VAE instance is cached and shared; don't leak tiling into other nodes.
|
||||
vae.disable_tiling()
|
||||
|
||||
# Take batch 0 (we generate one video at a time) and move the clip off the
|
||||
# accelerator now — MP4 encoding can take a while, and holding the full
|
||||
# decoded clip in VRAM for its duration starves the next node's model load.
|
||||
decoded = decoded[0].cpu() # [C, T, H, W]
|
||||
|
||||
TorchDevice.empty_cache()
|
||||
|
||||
if context.util.is_canceled():
|
||||
raise CanceledException
|
||||
|
||||
num_frames = decoded.shape[1]
|
||||
if num_frames == 0:
|
||||
raise ValueError("Wan VAE decode produced zero frames.")
|
||||
|
||||
height, width = decoded.shape[2:]
|
||||
duration = num_frames / float(self.fps)
|
||||
|
||||
# Encode to a temporary MP4 (libx264 + yuv420p, exact frame dimensions —
|
||||
# see make_mp4_writer for why macro_block_size matters).
|
||||
tmp = tempfile.NamedTemporaryFile(prefix="invokeai_wan_video_", suffix=".mp4", delete=False)
|
||||
tmp.close()
|
||||
tmp_path = Path(tmp.name)
|
||||
try:
|
||||
context.logger.info(
|
||||
f"Encoding MP4: {num_frames} frames @ {self.fps} fps ({duration:.2f}s) at {width}x{height} via libx264"
|
||||
)
|
||||
context.util.signal_progress(f"Encoding MP4 ({num_frames} frames @ {self.fps} fps)")
|
||||
writer = make_mp4_writer(tmp_path, self.fps)
|
||||
try:
|
||||
_write_video_frames(writer, _iter_decoded_frames(decoded), context.util.is_canceled)
|
||||
finally:
|
||||
writer.close()
|
||||
del decoded
|
||||
TorchDevice.empty_cache()
|
||||
encoded_bytes = tmp_path.stat().st_size
|
||||
context.logger.info(f"MP4 encode complete: {encoded_bytes / 1024:.1f} KB")
|
||||
video_dto = context.videos.save(
|
||||
source_path=tmp_path,
|
||||
width=width,
|
||||
height=height,
|
||||
duration=duration,
|
||||
fps=float(self.fps),
|
||||
)
|
||||
context.logger.info(f"Saved video: {video_dto.video_name}")
|
||||
return VideoOutput.build(video_dto)
|
||||
finally:
|
||||
# If save() moved the file this is a no-op; if it failed earlier, we
|
||||
# don't want a lingering temp file.
|
||||
try:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
@@ -0,0 +1,265 @@
|
||||
from typing import Literal, Optional
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import (
|
||||
BaseInvocation,
|
||||
BaseInvocationOutput,
|
||||
Classification,
|
||||
invocation,
|
||||
invocation_output,
|
||||
)
|
||||
from invokeai.app.invocations.fields import FieldDescriptions, Input, InputField, OutputField
|
||||
from invokeai.app.invocations.model import LoRAField, ModelIdentifierField, WanTransformerField
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.backend.model_manager.taxonomy import (
|
||||
BaseModelType,
|
||||
ModelType,
|
||||
WanLoRAVariantType,
|
||||
WanVariantType,
|
||||
)
|
||||
|
||||
# Target option for routing a LoRA to one or both Wan A14B expert lists.
|
||||
#
|
||||
# - ``auto``: read the LoRA config's ``expert`` field (set by the probe / from
|
||||
# filename). ``"high"`` -> primary list only, ``"low"`` -> low-noise list
|
||||
# only, ``None`` -> both lists.
|
||||
# - ``both``: append to both lists regardless of the config.
|
||||
# - ``high``: append only to the primary list (high-noise expert).
|
||||
# - ``low``: append only to the low-noise list (low-noise expert).
|
||||
WanLoRATarget = Literal["auto", "both", "high", "low"]
|
||||
|
||||
|
||||
def _assert_is_wan_lora(lora_config: object, lora_key: str) -> None:
|
||||
"""Reject an identifier whose *resolved* config is not a Wan LoRA.
|
||||
|
||||
The identifier's own ``base``/``type`` fields are client-supplied and cannot be
|
||||
trusted: a hand-authored workflow can label any existing model key as a Wan LoRA
|
||||
and reach model patching (or fail minutes in, after expensive loading). Only the
|
||||
config the key actually resolves to is authoritative.
|
||||
"""
|
||||
config_type = getattr(lora_config, "type", None)
|
||||
config_base = getattr(lora_config, "base", None)
|
||||
if config_type is not ModelType.LoRA or config_base is not BaseModelType.Wan:
|
||||
raise ValueError(
|
||||
f"Model '{lora_key}' is not a Wan LoRA (resolved to "
|
||||
f"type={getattr(config_type, 'value', config_type)}, "
|
||||
f"base={getattr(config_base, 'value', config_base)})."
|
||||
)
|
||||
|
||||
|
||||
def _assert_lora_variant_matches_main(lora_config: object, main_config: object, lora_key: str) -> None:
|
||||
"""Reject an A14B LoRA wired against a 5B main (and vice versa).
|
||||
|
||||
A mismatch otherwise crashes deep in the layer patcher mid-denoise with an opaque
|
||||
tensor-shape error, after minutes of model loading. Skips silently when either
|
||||
variant is unrecorded (e.g. a LoRA whose targeted layers don't pin the inner dim).
|
||||
"""
|
||||
lora_variant = getattr(lora_config, "variant", None)
|
||||
main_variant = getattr(main_config, "variant", None)
|
||||
if lora_variant is None or main_variant is None:
|
||||
return
|
||||
lora_is_5b = lora_variant == WanLoRAVariantType.Wan5B
|
||||
main_is_5b = main_variant == WanVariantType.TI2V_5B
|
||||
if lora_is_5b != main_is_5b:
|
||||
raise ValueError(
|
||||
f"LoRA '{lora_key}' targets Wan {lora_variant.value.upper()} models, but the "
|
||||
f"transformer is a {main_variant.value} model. A14B and 5B LoRAs are not interchangeable."
|
||||
)
|
||||
|
||||
|
||||
def _warn_if_low_routing_is_inert(
|
||||
context: InvocationContext, main_config: object, lora_key: str, to_primary: bool, to_low_noise: bool
|
||||
) -> None:
|
||||
"""Warn when a LoRA is routed only to the low-noise list of a TI2V-5B main.
|
||||
|
||||
The single-transformer TI2V-5B denoise path consumes only the primary list, so
|
||||
such a LoRA silently has no effect — the node would otherwise report success
|
||||
while doing nothing.
|
||||
"""
|
||||
if to_primary or not to_low_noise:
|
||||
return
|
||||
if getattr(main_config, "variant", None) == WanVariantType.TI2V_5B:
|
||||
context.logger.warning(
|
||||
f"LoRA '{lora_key}' is routed only to the low-noise expert, which the single-transformer "
|
||||
"TI2V-5B variant never uses — the LoRA will have no effect."
|
||||
)
|
||||
|
||||
|
||||
def _resolve_target(target: WanLoRATarget, lora_expert: str | None) -> tuple[bool, bool]:
|
||||
"""Return (apply_to_primary, apply_to_low_noise) based on the requested
|
||||
target and the LoRA's recorded expert tag."""
|
||||
if target == "both":
|
||||
return True, True
|
||||
if target == "high":
|
||||
return True, False
|
||||
if target == "low":
|
||||
return False, True
|
||||
# auto
|
||||
if lora_expert == "high":
|
||||
return True, False
|
||||
if lora_expert == "low":
|
||||
return False, True
|
||||
return True, True
|
||||
|
||||
|
||||
@invocation_output("wan_lora_loader_output")
|
||||
class WanLoRALoaderOutput(BaseInvocationOutput):
|
||||
"""Wan 2.2 LoRA loader output."""
|
||||
|
||||
transformer: Optional[WanTransformerField] = OutputField(
|
||||
default=None, description=FieldDescriptions.transformer, title="Wan Transformer"
|
||||
)
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_lora_loader",
|
||||
title="Apply LoRA - Wan 2.2",
|
||||
tags=["lora", "model", "wan"],
|
||||
category="model",
|
||||
version="1.0.1",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanLoRALoaderInvocation(BaseInvocation):
|
||||
"""Apply a LoRA to the Wan 2.2 transformer(s).
|
||||
|
||||
For A14B (dual expert) the LoRA's recorded ``expert`` field determines
|
||||
which expert list it lands in: ``"high"`` -> primary list, ``"low"`` ->
|
||||
low-noise list, ``None`` (untagged) -> both lists. Use the ``target``
|
||||
field to override.
|
||||
|
||||
For TI2V-5B (single transformer) only the primary list is used at denoise
|
||||
time; a LoRA routed only to the low-noise list would be inert, so that
|
||||
routing logs a warning.
|
||||
"""
|
||||
|
||||
lora: ModelIdentifierField = InputField(
|
||||
description=FieldDescriptions.lora_model,
|
||||
title="LoRA",
|
||||
ui_model_base=BaseModelType.Wan,
|
||||
ui_model_type=ModelType.LoRA,
|
||||
)
|
||||
weight: float = InputField(default=0.75, description=FieldDescriptions.lora_weight)
|
||||
target: WanLoRATarget = InputField(
|
||||
default="auto",
|
||||
description="Which expert(s) to apply this LoRA to. 'auto' uses the LoRA's "
|
||||
"recorded expert tag (or both if untagged); 'both'/'high'/'low' override it.",
|
||||
)
|
||||
transformer: WanTransformerField | None = InputField(
|
||||
default=None,
|
||||
description=FieldDescriptions.transformer,
|
||||
input=Input.Connection,
|
||||
title="Wan Transformer",
|
||||
)
|
||||
|
||||
def invoke(self, context: InvocationContext) -> WanLoRALoaderOutput:
|
||||
lora_key = self.lora.key
|
||||
|
||||
if not context.models.exists(lora_key):
|
||||
raise ValueError(f"Unknown lora: {lora_key}!")
|
||||
|
||||
lora_config = context.models.get_config(self.lora)
|
||||
_assert_is_wan_lora(lora_config, lora_key)
|
||||
|
||||
output = WanLoRALoaderOutput()
|
||||
if self.transformer is None:
|
||||
return output
|
||||
|
||||
main_config = context.models.get_config(self.transformer.transformer)
|
||||
_assert_lora_variant_matches_main(lora_config, main_config, lora_key)
|
||||
|
||||
lora_expert = getattr(lora_config, "expert", None)
|
||||
to_primary, to_low_noise = _resolve_target(self.target, lora_expert)
|
||||
_warn_if_low_routing_is_inert(context, main_config, lora_key, to_primary, to_low_noise)
|
||||
|
||||
# Reject duplicates on whichever list(s) we're about to append to.
|
||||
if to_primary and any(item.lora.key == lora_key for item in self.transformer.loras):
|
||||
raise ValueError(f'LoRA "{lora_key}" already applied to primary transformer list.')
|
||||
if to_low_noise and any(item.lora.key == lora_key for item in self.transformer.loras_low_noise):
|
||||
raise ValueError(f'LoRA "{lora_key}" already applied to low-noise transformer list.')
|
||||
|
||||
output.transformer = self.transformer.model_copy(deep=True)
|
||||
new_lora = LoRAField(lora=self.lora, weight=self.weight)
|
||||
if to_primary:
|
||||
output.transformer.loras.append(new_lora)
|
||||
if to_low_noise:
|
||||
output.transformer.loras_low_noise.append(new_lora)
|
||||
|
||||
return output
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_lora_collection_loader",
|
||||
title="Apply LoRA Collection - Wan 2.2",
|
||||
tags=["lora", "model", "wan"],
|
||||
category="model",
|
||||
version="1.0.1",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanLoRACollectionLoader(BaseInvocation):
|
||||
"""Apply a collection of LoRAs to the Wan 2.2 transformer(s).
|
||||
|
||||
Each LoRA is routed to the primary and/or low-noise list based on its
|
||||
recorded ``expert`` tag (set by the probe from the filename). Untagged
|
||||
LoRAs go to both lists.
|
||||
"""
|
||||
|
||||
loras: Optional[LoRAField | list[LoRAField]] = InputField(
|
||||
default=None,
|
||||
description="LoRAs to apply. May be a single LoRA or a collection.",
|
||||
title="LoRAs",
|
||||
ui_model_base=[BaseModelType.Wan],
|
||||
ui_model_type=ModelType.LoRA,
|
||||
)
|
||||
transformer: Optional[WanTransformerField] = InputField(
|
||||
default=None,
|
||||
description=FieldDescriptions.transformer,
|
||||
input=Input.Connection,
|
||||
title="Wan Transformer",
|
||||
)
|
||||
|
||||
def invoke(self, context: InvocationContext) -> WanLoRALoaderOutput:
|
||||
output = WanLoRALoaderOutput()
|
||||
|
||||
if self.transformer is None:
|
||||
return output
|
||||
|
||||
output.transformer = self.transformer.model_copy(deep=True)
|
||||
|
||||
if self.loras is None:
|
||||
return output
|
||||
|
||||
loras = self.loras if isinstance(self.loras, list) else [self.loras]
|
||||
added: set[str] = set()
|
||||
main_config = context.models.get_config(self.transformer.transformer)
|
||||
|
||||
for lora in loras:
|
||||
if lora is None or lora.lora.key in added:
|
||||
continue
|
||||
|
||||
lora_key = lora.lora.key
|
||||
if not context.models.exists(lora_key):
|
||||
raise ValueError(f"Unknown lora: {lora_key}!")
|
||||
|
||||
lora_config = context.models.get_config(lora.lora)
|
||||
_assert_is_wan_lora(lora_config, lora_key)
|
||||
_assert_lora_variant_matches_main(lora_config, main_config, lora_key)
|
||||
|
||||
lora_expert = getattr(lora_config, "expert", None)
|
||||
to_primary, to_low_noise = _resolve_target("auto", lora_expert)
|
||||
_warn_if_low_routing_is_inert(context, main_config, lora_key, to_primary, to_low_noise)
|
||||
|
||||
# Reject LoRAs already applied upstream (same invariant the single loader
|
||||
# enforces) — re-appending would silently double the effective weight.
|
||||
# Intra-collection duplicates are skipped via `added` before reaching here.
|
||||
if to_primary and any(item.lora.key == lora_key for item in output.transformer.loras):
|
||||
raise ValueError(f'LoRA "{lora_key}" already applied to primary transformer list.')
|
||||
if to_low_noise and any(item.lora.key == lora_key for item in output.transformer.loras_low_noise):
|
||||
raise ValueError(f'LoRA "{lora_key}" already applied to low-noise transformer list.')
|
||||
|
||||
added.add(lora_key)
|
||||
|
||||
if to_primary:
|
||||
output.transformer.loras.append(lora)
|
||||
if to_low_noise:
|
||||
output.transformer.loras_low_noise.append(lora)
|
||||
|
||||
return output
|
||||
@@ -0,0 +1,300 @@
|
||||
from typing import Optional
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import (
|
||||
BaseInvocation,
|
||||
BaseInvocationOutput,
|
||||
Classification,
|
||||
invocation,
|
||||
invocation_output,
|
||||
)
|
||||
from invokeai.app.invocations.fields import FieldDescriptions, Input, InputField, OutputField
|
||||
from invokeai.app.invocations.model import (
|
||||
ModelIdentifierField,
|
||||
VAEField,
|
||||
WanT5EncoderField,
|
||||
WanTransformerField,
|
||||
)
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.backend.model_manager.taxonomy import BaseModelType, ModelFormat, ModelType, SubModelType, WanVariantType
|
||||
|
||||
|
||||
@invocation_output("wan_model_loader_output")
|
||||
class WanModelLoaderOutput(BaseInvocationOutput):
|
||||
"""Wan 2.2 model loader output."""
|
||||
|
||||
transformer: WanTransformerField = OutputField(
|
||||
description="Wan transformer (one or two experts depending on the variant)",
|
||||
title="Transformer",
|
||||
)
|
||||
wan_t5_encoder: WanT5EncoderField = OutputField(
|
||||
description=FieldDescriptions.wan_t5_encoder,
|
||||
title="UMT5-XXL Encoder",
|
||||
)
|
||||
vae: VAEField = OutputField(description=FieldDescriptions.vae, title="VAE")
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_model_loader",
|
||||
title="Main Model - Wan 2.2",
|
||||
tags=["model", "wan"],
|
||||
category="model",
|
||||
version="1.0.1",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanModelLoaderInvocation(BaseInvocation):
|
||||
"""Loads a Wan 2.2 model, outputting its submodels.
|
||||
|
||||
Components can be mixed and matched, mirroring the Qwen Image loader pattern:
|
||||
|
||||
- Transformer(s):
|
||||
* Diffusers main: emits ``transformer/`` and (for A14B) ``transformer_2/``
|
||||
from the same model record.
|
||||
* GGUF main: emits the single GGUF as the primary transformer; for A14B
|
||||
the second-expert GGUF must be wired to ``Transformer (Low Noise)``.
|
||||
- VAE: standalone Wan VAE > main (if Diffusers) > Component Source (Diffusers).
|
||||
- UMT5-XXL encoder: standalone Wan T5 encoder > main (if Diffusers) >
|
||||
Component Source (Diffusers).
|
||||
|
||||
The Component Source slot lets users supply a Diffusers Wan main model purely
|
||||
for VAE / encoder extraction when the actual transformer is in a single-file
|
||||
format. Together, the standalone VAE + standalone encoder let a GGUF
|
||||
transformer run without a full ~30 GB Diffusers install.
|
||||
"""
|
||||
|
||||
model: ModelIdentifierField = InputField(
|
||||
description=FieldDescriptions.wan_model,
|
||||
input=Input.Direct,
|
||||
ui_model_base=BaseModelType.Wan,
|
||||
ui_model_type=ModelType.Main,
|
||||
title="Transformer",
|
||||
)
|
||||
|
||||
transformer_low_noise_model: Optional[ModelIdentifierField] = InputField(
|
||||
default=None,
|
||||
description="Optional second GGUF transformer for the A14B low-noise expert. "
|
||||
"Only relevant when the main model is a single-file GGUF and the variant is A14B; "
|
||||
"ignored when the main is a Diffusers A14B (both experts are pulled from "
|
||||
"transformer/ and transformer_2/ already) or when the variant is TI2V-5B.",
|
||||
input=Input.Direct,
|
||||
ui_model_base=BaseModelType.Wan,
|
||||
ui_model_type=ModelType.Main,
|
||||
ui_model_format=ModelFormat.GGUFQuantized,
|
||||
title="Transformer (Low Noise)",
|
||||
)
|
||||
|
||||
vae_model: Optional[ModelIdentifierField] = InputField(
|
||||
default=None,
|
||||
description="Standalone Wan VAE model. If not set, the VAE is loaded from the main model "
|
||||
"(when in Diffusers format) or from the Component Source.",
|
||||
input=Input.Direct,
|
||||
ui_model_base=BaseModelType.Wan,
|
||||
ui_model_type=ModelType.VAE,
|
||||
title="VAE",
|
||||
)
|
||||
|
||||
wan_t5_encoder_model: Optional[ModelIdentifierField] = InputField(
|
||||
default=None,
|
||||
description="Standalone Wan UMT5-XXL encoder. If not set, the encoder is loaded from the main "
|
||||
"model (when in Diffusers format) or from the Component Source.",
|
||||
input=Input.Direct,
|
||||
ui_model_type=ModelType.WanT5Encoder,
|
||||
title="Wan T5 Encoder",
|
||||
)
|
||||
|
||||
component_source: Optional[ModelIdentifierField] = InputField(
|
||||
default=None,
|
||||
description="Diffusers Wan main model to extract VAE and/or encoder from. "
|
||||
"Use this if you don't have separate VAE/encoder models. "
|
||||
"Ignored for any submodel that is provided separately.",
|
||||
input=Input.Direct,
|
||||
ui_model_base=BaseModelType.Wan,
|
||||
ui_model_type=ModelType.Main,
|
||||
ui_model_format=ModelFormat.Diffusers,
|
||||
title="Component Source (Diffusers)",
|
||||
)
|
||||
|
||||
def invoke(self, context: InvocationContext) -> WanModelLoaderOutput:
|
||||
main_config = context.models.get_config(self.model)
|
||||
self._validate_main_config(main_config, "Wan main")
|
||||
main_format = main_config.format
|
||||
main_is_diffusers = main_format == ModelFormat.Diffusers
|
||||
main_is_gguf = main_format == ModelFormat.GGUFQuantized
|
||||
main_variant = getattr(main_config, "variant", None)
|
||||
if main_is_gguf and self.component_source is not None:
|
||||
self._validate_component_source_format(context, self.component_source)
|
||||
|
||||
# Resolve transformer + dual-expert wiring + boundary_ratio.
|
||||
#
|
||||
# Diffusers main: transformer/ is the primary, transformer_2/ is the
|
||||
# low-noise expert (A14B only). boundary_ratio comes from the probed
|
||||
# model_index.json.
|
||||
#
|
||||
# GGUF main: the file itself is one expert (high or low). For A14B,
|
||||
# the user wires the other expert to transformer_low_noise_model.
|
||||
# We swap so the *high*-noise expert is always the primary if needed.
|
||||
# boundary_ratio falls back to 0.875 unless a Diffusers component_source
|
||||
# provides a recorded value.
|
||||
boundary_ratio = 0.9 if main_variant == WanVariantType.I2V_A14B else 0.875
|
||||
transformer_low_noise: Optional[ModelIdentifierField] = None
|
||||
|
||||
if main_is_diffusers:
|
||||
transformer = self.model.model_copy(update={"submodel_type": SubModelType.Transformer})
|
||||
if getattr(main_config, "has_dual_expert", False):
|
||||
transformer_low_noise = self.model.model_copy(update={"submodel_type": SubModelType.Transformer2})
|
||||
recorded = getattr(main_config, "boundary_ratio", None)
|
||||
if recorded is not None:
|
||||
boundary_ratio = float(recorded)
|
||||
elif main_is_gguf:
|
||||
primary_expert = getattr(main_config, "expert", "none")
|
||||
primary_id = self.model.model_copy(update={"submodel_type": SubModelType.Transformer})
|
||||
|
||||
if self.transformer_low_noise_model is not None and main_variant == WanVariantType.TI2V_5B:
|
||||
# The field's own docs promise this input is ignored for the
|
||||
# single-expert TI2V-5B — e.g. a leftover wire from an A14B session.
|
||||
context.logger.warning("'Transformer (Low Noise)' is ignored for the single-expert TI2V-5B variant.")
|
||||
|
||||
if self.transformer_low_noise_model is not None and main_variant != WanVariantType.TI2V_5B:
|
||||
low_config = context.models.get_config(self.transformer_low_noise_model)
|
||||
self._validate_main_config(low_config, "Transformer (Low Noise)")
|
||||
if low_config.format != ModelFormat.GGUFQuantized:
|
||||
raise ValueError(
|
||||
f"'Transformer (Low Noise)' must be a GGUF-format Wan model. "
|
||||
f"'{low_config.name}' is in {low_config.format.value} format."
|
||||
)
|
||||
low_id = self.transformer_low_noise_model.model_copy(update={"submodel_type": SubModelType.Transformer})
|
||||
low_expert = getattr(low_config, "expert", "none")
|
||||
|
||||
if getattr(low_config, "variant", None) != main_variant:
|
||||
raise ValueError("The high-noise and low-noise GGUF models must use the same Wan variant.")
|
||||
if {primary_expert, low_expert} != {"high", "low"}:
|
||||
raise ValueError("A Wan A14B GGUF expert pair must contain one high and one low expert.")
|
||||
|
||||
# Make sure 'transformer' is the high-noise expert and
|
||||
# 'transformer_low_noise' is the low-noise expert. If the user
|
||||
# accidentally swapped them, swap back.
|
||||
if primary_expert == "low" and low_expert == "high":
|
||||
transformer = low_id
|
||||
transformer_low_noise = primary_id
|
||||
else:
|
||||
transformer = primary_id
|
||||
transformer_low_noise = low_id
|
||||
else:
|
||||
transformer = primary_id
|
||||
if main_variant in (WanVariantType.T2V_A14B, WanVariantType.I2V_A14B) and primary_expert != "high":
|
||||
raise ValueError("An unpaired Wan A14B GGUF model must be the high-noise expert.")
|
||||
# A14B without a paired low-noise GGUF will produce degraded
|
||||
# quality (only the high-noise expert runs). Warn but don't
|
||||
# abort — TI2V-5B GGUFs are single-expert and totally fine.
|
||||
if main_variant in (WanVariantType.T2V_A14B, WanVariantType.I2V_A14B):
|
||||
context.logger.warning(
|
||||
"A14B GGUF main was provided without a paired 'Transformer (Low Noise)'. "
|
||||
"Only the high-noise expert will run; image quality will be reduced."
|
||||
)
|
||||
|
||||
# Borrow the boundary_ratio recorded on the optional Diffusers
|
||||
# component_source, when one is wired.
|
||||
if self.component_source is not None:
|
||||
src_cfg = context.models.get_config(self.component_source)
|
||||
src_boundary = getattr(src_cfg, "boundary_ratio", None)
|
||||
if (
|
||||
src_cfg.format == ModelFormat.Diffusers
|
||||
and getattr(src_cfg, "variant", None) == main_variant
|
||||
and src_boundary is not None
|
||||
):
|
||||
boundary_ratio = float(src_boundary)
|
||||
else:
|
||||
raise ValueError(
|
||||
f"Unsupported main model format for Wan: {main_format.value}. "
|
||||
"Use a Diffusers folder or a GGUF single-file checkpoint."
|
||||
)
|
||||
|
||||
# VAE: standalone override > main (if Diffusers) > component source.
|
||||
if self.vae_model is not None:
|
||||
self._validate_standalone_vae(context, self.vae_model, main_variant)
|
||||
vae = self.vae_model.model_copy(update={"submodel_type": SubModelType.VAE})
|
||||
elif main_is_diffusers:
|
||||
vae = self.model.model_copy(update={"submodel_type": SubModelType.VAE})
|
||||
elif self.component_source is not None:
|
||||
self._validate_component_source_vae(context, self.component_source, main_variant)
|
||||
vae = self.component_source.model_copy(update={"submodel_type": SubModelType.VAE})
|
||||
else:
|
||||
raise ValueError(
|
||||
"No source for VAE. Either set 'VAE' to a standalone Wan VAE, "
|
||||
"or set 'Component Source' to a Diffusers Wan main model."
|
||||
)
|
||||
|
||||
# Tokenizer + text encoder: standalone override > main (if Diffusers) > component source.
|
||||
if self.wan_t5_encoder_model is not None:
|
||||
t5_config = context.models.get_config(self.wan_t5_encoder_model)
|
||||
if t5_config.type != ModelType.WanT5Encoder or t5_config.format != ModelFormat.WanT5Encoder:
|
||||
raise ValueError("The Wan T5 Encoder must resolve to a standalone Wan T5 encoder model.")
|
||||
tokenizer = self.wan_t5_encoder_model.model_copy(update={"submodel_type": SubModelType.Tokenizer})
|
||||
text_encoder = self.wan_t5_encoder_model.model_copy(update={"submodel_type": SubModelType.TextEncoder})
|
||||
elif main_is_diffusers:
|
||||
tokenizer = self.model.model_copy(update={"submodel_type": SubModelType.Tokenizer})
|
||||
text_encoder = self.model.model_copy(update={"submodel_type": SubModelType.TextEncoder})
|
||||
elif self.component_source is not None:
|
||||
self._validate_component_source_format(context, self.component_source)
|
||||
tokenizer = self.component_source.model_copy(update={"submodel_type": SubModelType.Tokenizer})
|
||||
text_encoder = self.component_source.model_copy(update={"submodel_type": SubModelType.TextEncoder})
|
||||
else:
|
||||
raise ValueError(
|
||||
"No source for Wan T5 encoder. "
|
||||
"Either set 'Wan T5 Encoder' to a standalone UMT5-XXL encoder, "
|
||||
"or set 'Component Source' to a Diffusers Wan main model."
|
||||
)
|
||||
|
||||
return WanModelLoaderOutput(
|
||||
transformer=WanTransformerField(
|
||||
transformer=transformer,
|
||||
transformer_low_noise=transformer_low_noise,
|
||||
boundary_ratio=boundary_ratio,
|
||||
),
|
||||
wan_t5_encoder=WanT5EncoderField(tokenizer=tokenizer, text_encoder=text_encoder),
|
||||
vae=VAEField(vae=vae),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _validate_main_config(config: object, label: str) -> None:
|
||||
if getattr(config, "base", None) != BaseModelType.Wan or getattr(config, "type", None) != ModelType.Main:
|
||||
raise ValueError(f"The {label} model must resolve to a Wan main model.")
|
||||
|
||||
@staticmethod
|
||||
def _validate_component_source_format(context: InvocationContext, model: ModelIdentifierField) -> None:
|
||||
source_config = context.models.get_config(model)
|
||||
if source_config.base != BaseModelType.Wan or source_config.type != ModelType.Main:
|
||||
raise ValueError("The Component Source model must resolve to a Wan main model.")
|
||||
if source_config.format != ModelFormat.Diffusers:
|
||||
raise ValueError(
|
||||
f"The Component Source model must be in Diffusers format. "
|
||||
f"The selected model '{source_config.name}' is in {source_config.format.value} format."
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _validate_component_source_vae(
|
||||
context: InvocationContext, model: ModelIdentifierField, main_variant: WanVariantType
|
||||
) -> None:
|
||||
WanModelLoaderInvocation._validate_component_source_format(context, model)
|
||||
source_config = context.models.get_config(model)
|
||||
source_variant = getattr(source_config, "variant", None)
|
||||
main_is_ti2v = main_variant == WanVariantType.TI2V_5B
|
||||
source_is_ti2v = source_variant == WanVariantType.TI2V_5B
|
||||
if main_is_ti2v != source_is_ti2v:
|
||||
raise ValueError(
|
||||
"The Component Source VAE is incompatible with the selected transformer. "
|
||||
"TI2V-5B requires the 48-channel Wan 2.2 VAE; A14B models require the 16-channel Wan 2.1 VAE."
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _validate_standalone_vae(
|
||||
context: InvocationContext, model: ModelIdentifierField, main_variant: WanVariantType
|
||||
) -> None:
|
||||
vae_config = context.models.get_config(model)
|
||||
if vae_config.base != BaseModelType.Wan or vae_config.type != ModelType.VAE:
|
||||
raise ValueError("The VAE must resolve to a standalone Wan VAE model.")
|
||||
expected_channels = 48 if main_variant == WanVariantType.TI2V_5B else 16
|
||||
if vae_config.latent_channels != expected_channels:
|
||||
raise ValueError(
|
||||
"The standalone VAE is incompatible with the selected transformer. "
|
||||
"TI2V-5B requires the 48-channel Wan 2.2 VAE; A14B models require the 16-channel Wan 2.1 VAE."
|
||||
)
|
||||
@@ -0,0 +1,209 @@
|
||||
"""Reference-image (VAE-latent) encoder for Wan 2.2 image-to-video conditioning.
|
||||
|
||||
Wan 2.2 I2V conditions on a reference image by VAE-encoding it. The condition
|
||||
shape depends on the wired VAE (which selects the model family):
|
||||
|
||||
- **A14B** (16-channel Wan 2.1 VAE): a 20-channel condition tensor (4-ch
|
||||
first-frame mask + 16-ch image latents) that the denoise loop concatenates
|
||||
to the noise latents along the channel dim each step.
|
||||
- **TI2V-5B** (48-channel Wan 2.2 VAE): a single-frame 48-channel condition
|
||||
that the denoise loop blends with the noisy latents at every step
|
||||
(expand_timesteps path).
|
||||
|
||||
Supports both single-frame (image I2V, ``num_frames=1``) and multi-frame
|
||||
(video I2V, e.g. ``num_frames=81``) condition tensors; multi-frame and FLF2V
|
||||
(end-image interpolation) are A14B-only.
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
import torch
|
||||
from diffusers.models.autoencoders import AutoencoderKLWan
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
|
||||
from invokeai.app.invocations.fields import (
|
||||
FieldDescriptions,
|
||||
ImageField,
|
||||
Input,
|
||||
InputField,
|
||||
)
|
||||
from invokeai.app.invocations.model import VAEField
|
||||
from invokeai.app.invocations.primitives import WanRefImageOutput
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.backend.model_manager.load.model_cache.utils import get_effective_device
|
||||
from invokeai.backend.util.devices import TorchDevice
|
||||
from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_wan
|
||||
from invokeai.backend.wan.extensions.wan_ref_image_extension import (
|
||||
encode_reference_image_to_condition,
|
||||
encode_reference_image_to_ti2v_condition,
|
||||
encode_reference_image_to_video_condition,
|
||||
)
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_ref_image_encoder",
|
||||
title="Reference Image - Wan 2.2",
|
||||
tags=["image", "conditioning", "wan", "i2v"],
|
||||
category="conditioning",
|
||||
version="1.2.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanRefImageEncoderInvocation(BaseInvocation):
|
||||
"""VAE-encode a reference image into Wan 2.2 I2V conditioning.
|
||||
|
||||
The wired VAE selects the output shape. With the 16-channel A14B VAE the
|
||||
output is a ``[1, 20, T_lat, height // 8, width // 8]`` condition (4-ch
|
||||
first-frame mask + 16-ch image latents) that the denoise loop concatenates
|
||||
to the noise latents each step, producing the 36-channel input the
|
||||
I2V-A14B transformer expects. With the 48-channel TI2V-5B VAE the output
|
||||
is a ``[1, 48, 1, height // 16, width // 16]`` condition that the denoise
|
||||
loop blends into the noisy latents at every step.
|
||||
|
||||
For image (single-frame) I2V leave ``num_frames=1`` (T_lat=1). For A14B
|
||||
video I2V set ``num_frames`` to match the value on the video-denoise node
|
||||
(e.g. 81 for the Wan 2.2 reference defaults); TI2V-5B always uses the
|
||||
single-frame condition.
|
||||
|
||||
Supply an optional ``end_image`` for **first-last-frame interpolation
|
||||
(FLF2V)** — the model then interpolates the motion from ``image`` (first
|
||||
frame) to ``end_image`` (final frame). FLF2V is I2V-A14B video only
|
||||
(``num_frames > 1``); it is not supported for TI2V-5B or single-frame I2V.
|
||||
|
||||
For T2V (no reference image), omit this node entirely.
|
||||
"""
|
||||
|
||||
image: ImageField = InputField(description="Reference image to condition on (the first frame).")
|
||||
vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection, title="VAE")
|
||||
# Must match wan_denoise's width/height. multiple_of=16 (not 8) because
|
||||
# Wan's transformer patch_size=(1, 2, 2) needs latent H/W to be even.
|
||||
width: int = InputField(
|
||||
default=1024,
|
||||
gt=0,
|
||||
multiple_of=16,
|
||||
description="Width to resize the reference image to (must match denoise width).",
|
||||
)
|
||||
height: int = InputField(
|
||||
default=1024,
|
||||
gt=0,
|
||||
multiple_of=16,
|
||||
description="Height to resize the reference image to (must match denoise height).",
|
||||
)
|
||||
num_frames: int = InputField(
|
||||
default=1,
|
||||
ge=1,
|
||||
description="Pixel-frame count to build the condition for. Use 1 for single-frame image "
|
||||
"I2V. For video I2V, set this to match the video-denoise node's num_frames (and ensure "
|
||||
"(num_frames - 1) %% 4 == 0, e.g. 81).",
|
||||
title="Number of Frames",
|
||||
)
|
||||
end_image: Optional[ImageField] = InputField(
|
||||
default=None,
|
||||
description="Optional end frame for first-last-frame interpolation (FLF2V). When set, the "
|
||||
"video interpolates from the reference image (first frame) to this image (final frame). "
|
||||
"I2V-A14B video only (num_frames > 1); not supported for TI2V-5B or single-frame I2V.",
|
||||
title="End Image (FLF2V)",
|
||||
)
|
||||
|
||||
@torch.no_grad()
|
||||
def invoke(self, context: InvocationContext) -> WanRefImageOutput:
|
||||
if self.num_frames > 1 and (self.num_frames - 1) % 4 != 0:
|
||||
raise ValueError(
|
||||
f"num_frames must satisfy (num_frames - 1) %% 4 == 0 for the Wan VAE's temporal "
|
||||
f"compression (got {self.num_frames}). Try 5, 9, 13, ..., 81, 85, ..."
|
||||
)
|
||||
|
||||
pil_image = context.images.get_pil(self.image.image_name, "RGB")
|
||||
end_pil_image = context.images.get_pil(self.end_image.image_name, "RGB") if self.end_image is not None else None
|
||||
|
||||
vae_info = context.models.load(self.vae.vae)
|
||||
if not isinstance(vae_info.model, AutoencoderKLWan):
|
||||
raise TypeError(f"Reference-image encoder requires AutoencoderKLWan, got {type(vae_info.model).__name__}.")
|
||||
|
||||
estimated_working_memory = estimate_vae_working_memory_wan(
|
||||
operation="encode",
|
||||
vae=vae_info.model,
|
||||
pixel_height=self.height,
|
||||
pixel_width=self.width,
|
||||
pixel_frames=self.num_frames,
|
||||
)
|
||||
|
||||
with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae):
|
||||
assert isinstance(vae, AutoencoderKLWan)
|
||||
# A cpu_only VAE stays in system RAM even when an accelerator is selected —
|
||||
# run the encode where the weights actually live.
|
||||
device = get_effective_device(vae)
|
||||
target_dtype = TorchDevice.choose_bfloat16_safe_dtype(device)
|
||||
context.util.signal_progress(
|
||||
("VAE-encoding FLF2V start+end images" if end_pil_image is not None else "VAE-encoding reference image")
|
||||
+ (f" ({self.num_frames} frames)" if self.num_frames > 1 else "")
|
||||
)
|
||||
# Free cached allocator blocks left over from earlier nodes (denoise expert
|
||||
# swaps in particular can leave the cache fragmented in ways that look like
|
||||
# free VRAM but fail a single large contiguous request). Mirrors the
|
||||
# pattern used in wan_latents_to_image.py / wan_latents_to_video.py.
|
||||
TorchDevice.empty_cache()
|
||||
# Pick the encoder path by VAE z_dim: 48 means the Wan 2.2-VAE (TI2V-5B),
|
||||
# which uses a single-frame 48-channel condition that the denoise loop
|
||||
# blends with the noisy latents at every step (expand_timesteps path).
|
||||
# 16 means the standard Wan VAE (A14B), which uses the 20-channel
|
||||
# mask + latent condition concatenated to noise along the channel dim.
|
||||
is_ti2v_5b = getattr(vae.config, "z_dim", 16) == 48
|
||||
if end_pil_image is not None and (is_ti2v_5b or self.num_frames <= 1):
|
||||
raise ValueError(
|
||||
"End-image (FLF2V) interpolation is only supported for I2V-A14B video "
|
||||
f"(num_frames > 1). Got {'TI2V-5B' if is_ti2v_5b else 'single-frame I2V'}. "
|
||||
"Remove the End Image input, or use an A14B VAE with num_frames > 1."
|
||||
)
|
||||
if is_ti2v_5b:
|
||||
# TI2V-5B I2V needs latent H/W to be even for the transformer
|
||||
# patch_size=(1,2,2), so pixel dims must be multiples of 32
|
||||
# (16x VAE * 2 transformer patch). A14B's 8x VAE only needed
|
||||
# multiples of 16.
|
||||
if self.width % 32 != 0 or self.height % 32 != 0:
|
||||
raise ValueError(
|
||||
f"TI2V-5B I2V requires width and height to be multiples of 32 "
|
||||
f"(got {self.width}x{self.height}). The Wan 2.2-VAE uses 16x "
|
||||
f"spatial compression and the transformer adds a 2x patch on "
|
||||
f"top, so pixel dims must divide by 32 for the patchify step."
|
||||
)
|
||||
condition = encode_reference_image_to_ti2v_condition(
|
||||
image=pil_image,
|
||||
vae=vae,
|
||||
width=self.width,
|
||||
height=self.height,
|
||||
device=device,
|
||||
dtype=target_dtype,
|
||||
)
|
||||
elif self.num_frames <= 1:
|
||||
condition = encode_reference_image_to_condition(
|
||||
image=pil_image,
|
||||
vae=vae,
|
||||
width=self.width,
|
||||
height=self.height,
|
||||
device=device,
|
||||
dtype=target_dtype,
|
||||
)
|
||||
else:
|
||||
condition = encode_reference_image_to_video_condition(
|
||||
image=pil_image,
|
||||
vae=vae,
|
||||
width=self.width,
|
||||
height=self.height,
|
||||
num_frames=self.num_frames,
|
||||
device=device,
|
||||
dtype=target_dtype,
|
||||
last_image=end_pil_image,
|
||||
)
|
||||
|
||||
condition = condition.detach().to("cpu")
|
||||
# Release this node's VAE-encode intermediates before the next node tries to
|
||||
# partial-load the denoise transformer — the OOM we saw in PR #9163 review
|
||||
# was the I2V expert load racing against still-cached encode activations.
|
||||
TorchDevice.empty_cache()
|
||||
name = context.tensors.save(tensor=condition)
|
||||
return WanRefImageOutput.build(
|
||||
condition_tensor_name=name,
|
||||
width=self.width,
|
||||
height=self.height,
|
||||
num_frames=self.num_frames,
|
||||
)
|
||||
@@ -0,0 +1,112 @@
|
||||
import torch
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
|
||||
from invokeai.app.invocations.fields import (
|
||||
FieldDescriptions,
|
||||
Input,
|
||||
InputField,
|
||||
UIComponent,
|
||||
)
|
||||
from invokeai.app.invocations.model import WanT5EncoderField
|
||||
from invokeai.app.invocations.primitives import WanConditioningOutput
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.backend.model_manager.load.model_cache.utils import get_effective_device
|
||||
from invokeai.backend.stable_diffusion.diffusion.conditioning_data import (
|
||||
ConditioningFieldData,
|
||||
WanConditioningInfo,
|
||||
)
|
||||
|
||||
# Wan models are trained with 512-token text sequences (matches the
|
||||
# upstream config.json's ``text_len: 512`` and the WanPipeline.__call__
|
||||
# default). Diffusers' ``_get_t5_prompt_embeds`` has a stale 226 default
|
||||
# that gets overridden by ``__call__``; using 512 here matches the actual
|
||||
# pipeline behaviour.
|
||||
WAN_T5_MAX_SEQ_LEN = 512
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_text_encoder",
|
||||
title="Prompt - Wan 2.2",
|
||||
tags=["prompt", "conditioning", "wan"],
|
||||
category="conditioning",
|
||||
version="1.0.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanTextEncoderInvocation(BaseInvocation):
|
||||
"""Encodes a text prompt for Wan 2.2 using the UMT5-XXL encoder.
|
||||
|
||||
Output is the encoder's last hidden state (shape: [seq_len=512, 4096]) plus
|
||||
an attention mask marking valid (non-padding) tokens. The Wan transformer
|
||||
consumes these directly as ``encoder_hidden_states``.
|
||||
"""
|
||||
|
||||
prompt: str = InputField(description="Text prompt for Wan 2.2.", ui_component=UIComponent.Textarea)
|
||||
wan_t5_encoder: WanT5EncoderField = InputField(
|
||||
title="UMT5-XXL Encoder",
|
||||
description=FieldDescriptions.wan_t5_encoder,
|
||||
input=Input.Connection,
|
||||
)
|
||||
|
||||
@torch.no_grad()
|
||||
def invoke(self, context: InvocationContext) -> WanConditioningOutput:
|
||||
prompt_embeds, attention_mask = self._encode(context)
|
||||
|
||||
# Persist on CPU; the denoise loop will move to device as needed.
|
||||
prompt_embeds = prompt_embeds.detach().to("cpu")
|
||||
attention_mask = attention_mask.detach().to("cpu") if attention_mask is not None else None
|
||||
|
||||
conditioning_data = ConditioningFieldData(
|
||||
conditionings=[WanConditioningInfo(prompt_embeds=prompt_embeds, prompt_attention_mask=attention_mask)]
|
||||
)
|
||||
conditioning_name = context.conditioning.save(conditioning_data)
|
||||
return WanConditioningOutput.build(conditioning_name)
|
||||
|
||||
def _encode(self, context: InvocationContext) -> tuple[torch.Tensor, torch.Tensor | None]:
|
||||
from diffusers.pipelines.wan.pipeline_wan import prompt_clean
|
||||
from transformers import UMT5EncoderModel
|
||||
|
||||
cleaned = prompt_clean(self.prompt)
|
||||
|
||||
# Tokenizer + text encoder both routed through the model cache so the
|
||||
# registered loaders handle the nested-vs-flat directory layout for us
|
||||
# (main-model layout: <root>/tokenizer/ + <root>/text_encoder/;
|
||||
# standalone WanT5Encoder layout may also be flat).
|
||||
tokenizer_info = context.models.load(self.wan_t5_encoder.tokenizer)
|
||||
with tokenizer_info.model_on_device() as (_, tokenizer):
|
||||
text_inputs = tokenizer(
|
||||
[cleaned],
|
||||
padding="max_length",
|
||||
max_length=WAN_T5_MAX_SEQ_LEN,
|
||||
truncation=True,
|
||||
add_special_tokens=True,
|
||||
return_attention_mask=True,
|
||||
return_tensors="pt",
|
||||
)
|
||||
|
||||
text_encoder_info = context.models.load(self.wan_t5_encoder.text_encoder)
|
||||
with text_encoder_info.model_on_device() as (_, text_encoder):
|
||||
assert isinstance(text_encoder, UMT5EncoderModel)
|
||||
device = get_effective_device(text_encoder)
|
||||
|
||||
input_ids = text_inputs.input_ids.to(device)
|
||||
attention_mask = text_inputs.attention_mask.to(device)
|
||||
|
||||
context.util.signal_progress("Running UMT5-XXL text encoder")
|
||||
outputs = text_encoder(input_ids, attention_mask)
|
||||
# Drop the batch dim (we always encode one prompt at a time).
|
||||
prompt_embeds = outputs.last_hidden_state.squeeze(0)
|
||||
attention_mask_out = attention_mask.squeeze(0)
|
||||
|
||||
# Match the Diffusers reference: zero out the embeddings past the valid
|
||||
# token count so the transformer sees clean padding.
|
||||
valid_len = int(attention_mask_out.sum().item())
|
||||
if valid_len < prompt_embeds.shape[0]:
|
||||
prompt_embeds = prompt_embeds.clone()
|
||||
prompt_embeds[valid_len:] = 0
|
||||
|
||||
# If every token is valid we don't need the mask downstream.
|
||||
mask_out: torch.Tensor | None = attention_mask_out
|
||||
if attention_mask_out.all():
|
||||
mask_out = None
|
||||
|
||||
return prompt_embeds.to(dtype=torch.bfloat16), mask_out
|
||||
@@ -0,0 +1,400 @@
|
||||
"""Wan 2.2 video denoise invocation (T2V / I2V).
|
||||
|
||||
Multi-frame counterpart to :mod:`wan_denoise`. Drives the same flow-matching
|
||||
schedule + expert-swap MoE logic, but the noise tensor has a real temporal
|
||||
dimension (``T_lat = (num_frames - 1) // 4 + 1``) and the I2V conditioning is
|
||||
built across all latent frames (first frame conditioned, rest zero).
|
||||
|
||||
Kept as a separate file rather than parameterizing ``WanDenoiseInvocation``
|
||||
so the working single-frame T2I path is not risked by the video work; the
|
||||
shared bits (expert swapper, scheduler construction, conditioning loading,
|
||||
LoRA iteration) live in ``wan_denoise`` and are imported here.
|
||||
"""
|
||||
|
||||
from contextlib import ExitStack
|
||||
from typing import Callable, Iterable, Optional, Tuple
|
||||
|
||||
import torch
|
||||
from tqdm import tqdm
|
||||
|
||||
from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
|
||||
from invokeai.app.invocations.fields import (
|
||||
FieldDescriptions,
|
||||
Input,
|
||||
InputField,
|
||||
WanConditioningField,
|
||||
WanRefImageConditioningField,
|
||||
)
|
||||
from invokeai.app.invocations.model import WanTransformerField
|
||||
from invokeai.app.invocations.primitives import LatentsOutput
|
||||
from invokeai.app.invocations.wan_denoise import (
|
||||
WanDenoiseInvocation,
|
||||
_ExpertSwapper,
|
||||
_resolve_variant,
|
||||
_validate_ref_condition_shape,
|
||||
_validate_spatial_dimensions,
|
||||
)
|
||||
from invokeai.app.services.shared.invocation_context import InvocationContext
|
||||
from invokeai.backend.model_manager.taxonomy import BaseModelType, ModelFormat, WanVariantType
|
||||
from invokeai.backend.patches.model_patch_raw import ModelPatchRaw
|
||||
from invokeai.backend.stable_diffusion.diffusers_pipeline import PipelineIntermediateState
|
||||
from invokeai.backend.stable_diffusion.diffusion.conditioning_data import WanConditioningInfo
|
||||
from invokeai.backend.util.devices import TorchDevice
|
||||
from invokeai.backend.wan.sampling_utils import (
|
||||
get_default_latent_channels,
|
||||
get_spatial_scale_factor,
|
||||
make_noise,
|
||||
num_latent_frames_for,
|
||||
)
|
||||
|
||||
|
||||
@invocation(
|
||||
"wan_video_denoise",
|
||||
title="Denoise Video - Wan 2.2",
|
||||
tags=["video", "wan"],
|
||||
category="latents",
|
||||
version="1.0.0",
|
||||
classification=Classification.Prototype,
|
||||
)
|
||||
class WanVideoDenoiseInvocation(BaseInvocation):
|
||||
"""Run the Wan 2.2 denoising loop on a multi-frame latent tensor.
|
||||
|
||||
The output is a 5D ``[1, C, T_lat, H/8, W/8]`` latent tensor ready for
|
||||
:class:`WanLatentsToVideoInvocation` to VAE-decode and encode as MP4.
|
||||
|
||||
Mirrors :class:`WanDenoiseInvocation` for the per-step logic (CFG, MoE
|
||||
expert swap at the boundary timestep, LoRA patching, scheduler selection).
|
||||
Differences from the image denoise:
|
||||
|
||||
* The noise tensor has a real temporal dim built from ``num_frames``.
|
||||
* The I2V condition is built across all latent frames (frame 0
|
||||
conditioned, rest zero) via
|
||||
:func:`encode_reference_image_to_video_condition` upstream — the
|
||||
``ref_image`` field on this node carries a tensor of shape
|
||||
``[1, 20, T_lat, H_lat, W_lat]`` instead of ``[1, 20, 1, ...]``.
|
||||
* No ``denoising_start`` / ``denoising_end`` / initial-latents inputs.
|
||||
The image denoise node uses those for img2img (noise injection on an
|
||||
existing latent), but image-conditioned video generation flows through
|
||||
the reference-frame conditioning mechanism instead — the first frame
|
||||
drives subsequent frames at every step, so a partial-schedule run from
|
||||
an initial latent has no analogue here. Run the schedule from t=1
|
||||
to t=0 every time. The base ``WanDenoiseInvocation`` still handles
|
||||
the img2img case for stills.
|
||||
"""
|
||||
|
||||
transformer: WanTransformerField = InputField(
|
||||
description=(
|
||||
"Wan transformer field. Supported: T2V-A14B / I2V-A14B (dual-expert) and "
|
||||
"TI2V-5B (single-expert, handles both T2V and I2V). All three accept a "
|
||||
"Reference Image input for image-to-video; A14B uses the 36-channel concat "
|
||||
"scheme while TI2V-5B uses the expand_timesteps first-frame-mask blend."
|
||||
),
|
||||
input=Input.Connection,
|
||||
title="Transformer",
|
||||
)
|
||||
positive_conditioning: WanConditioningField = InputField(
|
||||
description=FieldDescriptions.positive_cond, input=Input.Connection
|
||||
)
|
||||
negative_conditioning: Optional[WanConditioningField] = InputField(
|
||||
default=None, description=FieldDescriptions.negative_cond, input=Input.Connection
|
||||
)
|
||||
ref_image: Optional[WanRefImageConditioningField] = InputField(
|
||||
default=None,
|
||||
description=FieldDescriptions.wan_ref_image,
|
||||
input=Input.Connection,
|
||||
title="Reference Image",
|
||||
)
|
||||
|
||||
guidance_scale: float = InputField(
|
||||
default=5.0,
|
||||
ge=1.0,
|
||||
description="Classifier-free guidance scale. Wan 2.2 video reference uses 5.0 for the "
|
||||
"high-noise expert and 4.0 for the low-noise expert.",
|
||||
title="Guidance Scale",
|
||||
)
|
||||
guidance_scale_low_noise: Optional[float] = InputField(
|
||||
default=4.0,
|
||||
ge=0.0,
|
||||
description="Optional separate CFG scale for the low-noise expert (Wan 2.2 A14B only). "
|
||||
"Values below 1.0 fall back to the primary 'Guidance Scale'.",
|
||||
title="Guidance Scale (Low Noise)",
|
||||
)
|
||||
|
||||
# Wan transformer patch_size=(1, 2, 2) × VAE spatial 8x => H/W multiple of 16.
|
||||
width: int = InputField(default=832, gt=0, multiple_of=16, description="Width of the generated video.")
|
||||
height: int = InputField(default=480, gt=0, multiple_of=16, description="Height of the generated video.")
|
||||
num_frames: int = InputField(
|
||||
default=81,
|
||||
ge=5,
|
||||
description="Number of output frames. Must satisfy (num_frames - 1) %% 4 == 0 so the latent "
|
||||
"temporal dim divides cleanly. Wan 2.2 was trained at 81 frames @ 16 FPS (~5 s).",
|
||||
title="Number of Frames",
|
||||
)
|
||||
steps: int = InputField(default=40, gt=0, description="Number of denoising steps.")
|
||||
seed: int = InputField(default=0, description="Randomness seed for reproducibility.")
|
||||
|
||||
@torch.no_grad()
|
||||
def invoke(self, context: InvocationContext) -> LatentsOutput:
|
||||
latents = self._run_diffusion(context)
|
||||
# Keep the 5D shape (B, C, T, H, W) — wan_latents_to_video expects it.
|
||||
latents = latents.detach().to("cpu")
|
||||
name = context.tensors.save(tensor=latents)
|
||||
# LatentsOutput.build uses latents.size()[3] / [2] for width / height.
|
||||
# For 5D the spatial dims are at indices 4 / 3 instead of 3 / 2, so we
|
||||
# call the constructor directly with the actual H/W from the inputs.
|
||||
from invokeai.app.invocations.fields import LatentsField
|
||||
|
||||
return LatentsOutput(
|
||||
latents=LatentsField(latents_name=name, seed=self.seed),
|
||||
width=self.width,
|
||||
height=self.height,
|
||||
)
|
||||
|
||||
def _run_diffusion(self, context: InvocationContext) -> torch.Tensor:
|
||||
if (self.num_frames - 1) % 4 != 0:
|
||||
raise ValueError(
|
||||
f"num_frames must satisfy (num_frames - 1) %% 4 == 0 for the Wan VAE's temporal "
|
||||
f"compression (got {self.num_frames}). Try 5, 9, 13, ..., 81, 85, ..."
|
||||
)
|
||||
|
||||
device = TorchDevice.choose_torch_device()
|
||||
inference_dtype = TorchDevice.choose_bfloat16_safe_dtype(device)
|
||||
|
||||
variant = _resolve_variant(context, self.transformer)
|
||||
_validate_spatial_dimensions(variant, self.width, self.height)
|
||||
spatial_scale = get_spatial_scale_factor(variant)
|
||||
|
||||
# Reuse the image denoise's scheduler construction so we pick up whatever
|
||||
# scheduler the variant ships with (UniPCMultistepScheduler with the
|
||||
# variant's flow_shift, or whatever an on-disk config specifies).
|
||||
scheduler_builder = WanDenoiseInvocation._build_scheduler # bound on instance below
|
||||
# Bind a minimal instance to call _build_scheduler — it only reads
|
||||
# self.transformer, which is shape-compatible.
|
||||
proxy = WanDenoiseInvocation.model_construct(
|
||||
transformer=self.transformer,
|
||||
positive_conditioning=self.positive_conditioning,
|
||||
)
|
||||
scheduler = scheduler_builder(proxy, context, device)
|
||||
|
||||
pos_cond = self._load_conditioning(context, self.positive_conditioning, device=device, dtype=inference_dtype)
|
||||
low_cfg_enabled = (
|
||||
self.transformer.transformer_low_noise is not None
|
||||
and self.guidance_scale_low_noise is not None
|
||||
and self.guidance_scale_low_noise >= 1.0
|
||||
and self.guidance_scale_low_noise != 1.0
|
||||
)
|
||||
do_cfg = self.negative_conditioning is not None and (self.guidance_scale != 1.0 or low_cfg_enabled)
|
||||
neg_cond: WanConditioningInfo | None = None
|
||||
if do_cfg:
|
||||
assert self.negative_conditioning is not None
|
||||
neg_cond = self._load_conditioning(
|
||||
context, self.negative_conditioning, device=device, dtype=inference_dtype
|
||||
)
|
||||
|
||||
# I2V condition tensor. Two flavours:
|
||||
# * A14B I2V — [1, 20, T_lat, H_lat, W_lat] (4 mask + 16 latent channels).
|
||||
# Concatenated to noise latents along the channel dim each step → 36ch.
|
||||
# * TI2V-5B I2V — [1, 48, 1, H_lat, W_lat] (single latent frame, same
|
||||
# channel count as the noise latents). Blended with noise via a
|
||||
# first_frame_mask at every step (expand_timesteps path).
|
||||
# Variant dispatch happens via the condition tensor's channel count below.
|
||||
ref_condition: torch.Tensor | None = None
|
||||
if self.ref_image is not None:
|
||||
if variant not in (WanVariantType.I2V_A14B, WanVariantType.TI2V_5B):
|
||||
raise ValueError(
|
||||
f"Reference-image conditioning is only supported by Wan 2.2 I2V variants "
|
||||
f"(I2V-A14B or TI2V-5B). The selected transformer is {variant.value!r}. "
|
||||
"Remove the Reference Image input or load an I2V variant."
|
||||
)
|
||||
if self.ref_image.width != self.width or self.ref_image.height != self.height:
|
||||
raise ValueError(
|
||||
f"Reference-image dimensions ({self.ref_image.width}x{self.ref_image.height}) must "
|
||||
f"match denoise dimensions ({self.width}x{self.height})."
|
||||
)
|
||||
# A14B encodes one condition tensor per pixel-frame count, so the
|
||||
# encoder's num_frames must match. TI2V-5B's condition is always
|
||||
# single-frame regardless of the output length, so the field's
|
||||
# num_frames is informational only and we skip this check.
|
||||
if variant == WanVariantType.I2V_A14B and self.ref_image.num_frames != self.num_frames:
|
||||
raise ValueError(
|
||||
f"Reference-image num_frames ({self.ref_image.num_frames}) must match denoise "
|
||||
f"num_frames ({self.num_frames}). Re-run the Reference Image - Wan 2.2 node with "
|
||||
f"num_frames={self.num_frames}."
|
||||
)
|
||||
ref_condition = context.tensors.load(self.ref_image.condition_tensor_name).to(
|
||||
device=device, dtype=inference_dtype
|
||||
)
|
||||
_validate_ref_condition_shape(
|
||||
ref_condition,
|
||||
channels=48 if variant == WanVariantType.TI2V_5B else 20,
|
||||
frames=1 if variant == WanVariantType.TI2V_5B else num_latent_frames_for(self.num_frames),
|
||||
height=self.height // spatial_scale,
|
||||
width=self.width // spatial_scale,
|
||||
)
|
||||
|
||||
scheduler.set_timesteps(num_inference_steps=self.steps, device=device)
|
||||
timesteps = scheduler.timesteps
|
||||
total_steps = len(timesteps)
|
||||
|
||||
# fp32 latents through the loop; cast to inference_dtype only when
|
||||
# calling the transformer (same as wan_denoise).
|
||||
latent_dtype = torch.float32
|
||||
# 48 for TI2V-5B (Wan 2.2-VAE z_dim=48), 16 for A14B variants.
|
||||
latent_channels = get_default_latent_channels(variant)
|
||||
t_lat = num_latent_frames_for(self.num_frames)
|
||||
|
||||
latents = make_noise(
|
||||
batch_size=1,
|
||||
latent_channels=latent_channels,
|
||||
height=self.height,
|
||||
width=self.width,
|
||||
spatial_scale_factor=spatial_scale,
|
||||
device=device,
|
||||
dtype=latent_dtype,
|
||||
seed=self.seed,
|
||||
num_latent_frames=t_lat,
|
||||
)
|
||||
|
||||
if total_steps <= 0:
|
||||
return latents
|
||||
|
||||
# Build the TI2V-5B first-frame mask once: 0 at frame 0 (locked to the
|
||||
# condition), 1 elsewhere (free to denoise). Broadcasts across channel dim.
|
||||
first_frame_mask: torch.Tensor | None = None
|
||||
if ref_condition is not None and variant == WanVariantType.TI2V_5B:
|
||||
_, _, _, h_lat, w_lat = latents.shape
|
||||
first_frame_mask = torch.ones(1, 1, t_lat, h_lat, w_lat, device=device, dtype=inference_dtype)
|
||||
first_frame_mask[:, :, 0] = 0
|
||||
|
||||
step_callback = self._build_step_callback(context)
|
||||
|
||||
high_model = self.transformer.transformer
|
||||
low_model = self.transformer.transformer_low_noise
|
||||
low_config = context.models.get_config(low_model) if low_model is not None else None
|
||||
num_train_timesteps = int(scheduler.config.num_train_timesteps)
|
||||
boundary_timestep = self.transformer.boundary_ratio * num_train_timesteps if low_model is not None else None
|
||||
|
||||
# No fallback between the LoRA lists — see the matching comment in
|
||||
# wan_denoise: an empty low list means "no LoRAs on the low expert".
|
||||
high_loras = self.transformer.loras
|
||||
low_loras = self.transformer.loras_low_noise
|
||||
high_config = context.models.get_config(high_model)
|
||||
high_is_quantized = high_config.format == ModelFormat.GGUFQuantized
|
||||
low_is_quantized = low_config.format == ModelFormat.GGUFQuantized if low_config is not None else False
|
||||
|
||||
def high_lora_factory() -> Iterable[Tuple[ModelPatchRaw, float]]:
|
||||
return proxy._lora_iterator(context, high_loras)
|
||||
|
||||
def low_lora_factory() -> Iterable[Tuple[ModelPatchRaw, float]]:
|
||||
return proxy._lora_iterator(context, low_loras)
|
||||
|
||||
with ExitStack() as exit_stack:
|
||||
swapper = _ExpertSwapper(
|
||||
context=context,
|
||||
high_model=high_model,
|
||||
low_model=low_model,
|
||||
inference_dtype=inference_dtype,
|
||||
high_lora_factory=high_lora_factory if high_loras else None,
|
||||
low_lora_factory=low_lora_factory if low_loras else None,
|
||||
high_is_quantized=high_is_quantized,
|
||||
low_is_quantized=low_is_quantized,
|
||||
)
|
||||
exit_stack.callback(swapper.close)
|
||||
|
||||
for step_idx, t in enumerate(
|
||||
tqdm(timesteps, desc=f"Denoising Wan 2.2 video ({self.num_frames} frames)", total=total_steps)
|
||||
):
|
||||
if low_model is not None and float(t) < float(boundary_timestep):
|
||||
active_label = _ExpertSwapper.LOW
|
||||
low_cfg = self.guidance_scale_low_noise
|
||||
active_cfg = low_cfg if (low_cfg is not None and low_cfg >= 1.0) else self.guidance_scale
|
||||
else:
|
||||
active_label = _ExpertSwapper.HIGH
|
||||
active_cfg = self.guidance_scale
|
||||
|
||||
transformer = swapper.get(active_label)
|
||||
|
||||
latent_model_input = latents.to(dtype=inference_dtype)
|
||||
|
||||
# Per-variant conditioning. Two distinct mechanisms:
|
||||
if first_frame_mask is not None:
|
||||
# TI2V-5B I2V (expand_timesteps): blend the condition into frame 0
|
||||
# and the noisy latents elsewhere. Per-token timestep tensor
|
||||
# gates the model so it sees timestep=0 at frame-0 positions
|
||||
# (nothing to denoise) and the normal `t` everywhere else.
|
||||
assert ref_condition is not None
|
||||
latent_model_input = (1 - first_frame_mask) * ref_condition + first_frame_mask * latent_model_input
|
||||
# Strided slice matches the transformer's spatial patch_size=2;
|
||||
# flatten gives per-token timesteps. Shape: [1, T_lat * H_lat//2 * W_lat//2].
|
||||
temp_ts = (first_frame_mask[0, 0, :, ::2, ::2] * t).flatten()
|
||||
timestep = temp_ts.unsqueeze(0).expand(latents.shape[0], -1).to(dtype=inference_dtype)
|
||||
elif ref_condition is not None:
|
||||
# A14B I2V: concat 20-ch condition along channel dim → 36-ch input.
|
||||
latent_model_input = torch.cat([latent_model_input, ref_condition], dim=1)
|
||||
timestep = t.expand(latents.shape[0])
|
||||
else:
|
||||
# T2V (any variant): scalar timestep per batch.
|
||||
timestep = t.expand(latents.shape[0])
|
||||
|
||||
noise_pred_cond = transformer(
|
||||
hidden_states=latent_model_input,
|
||||
timestep=timestep,
|
||||
encoder_hidden_states=pos_cond.prompt_embeds.unsqueeze(0),
|
||||
attention_kwargs=None,
|
||||
return_dict=False,
|
||||
)[0]
|
||||
|
||||
if neg_cond is not None and active_cfg != 1.0:
|
||||
noise_pred_uncond = transformer(
|
||||
hidden_states=latent_model_input,
|
||||
timestep=timestep,
|
||||
encoder_hidden_states=neg_cond.prompt_embeds.unsqueeze(0),
|
||||
attention_kwargs=None,
|
||||
return_dict=False,
|
||||
)[0]
|
||||
noise_pred = noise_pred_uncond + active_cfg * (noise_pred_cond - noise_pred_uncond)
|
||||
else:
|
||||
noise_pred = noise_pred_cond
|
||||
|
||||
latents = scheduler.step(noise_pred, t, latents, return_dict=False)[0]
|
||||
|
||||
step_callback(
|
||||
PipelineIntermediateState(
|
||||
step=step_idx + 1,
|
||||
order=1,
|
||||
total_steps=total_steps,
|
||||
timestep=int(t.item()),
|
||||
# Preview shows the middle frame for video.
|
||||
latents=latents[:, :, t_lat // 2],
|
||||
)
|
||||
)
|
||||
|
||||
# TI2V-5B: frame 0's latents drifted through the scheduler step at each
|
||||
# iteration; restore them to the clean condition before VAE-decoding so
|
||||
# the first output frame matches the reference image. Mirrors
|
||||
# ``WanImageToVideoPipeline`` (pipeline_wan_i2v.py:813-814).
|
||||
if first_frame_mask is not None:
|
||||
assert ref_condition is not None
|
||||
latents = (1 - first_frame_mask) * ref_condition.to(dtype=latents.dtype) + first_frame_mask * latents
|
||||
|
||||
return latents
|
||||
|
||||
def _load_conditioning(
|
||||
self,
|
||||
context: InvocationContext,
|
||||
cond_field: WanConditioningField,
|
||||
*,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
) -> WanConditioningInfo:
|
||||
cond_data = context.conditioning.load(cond_field.conditioning_name)
|
||||
assert len(cond_data.conditionings) == 1
|
||||
cond_info = cond_data.conditionings[0]
|
||||
assert isinstance(cond_info, WanConditioningInfo)
|
||||
return cond_info.to(device=device, dtype=dtype)
|
||||
|
||||
def _build_step_callback(self, context: InvocationContext) -> Callable[[PipelineIntermediateState], None]:
|
||||
def step_callback(state: PipelineIntermediateState) -> None:
|
||||
context.util.sd_step_callback(state, BaseModelType.Wan)
|
||||
|
||||
return step_callback
|
||||
@@ -1,5 +1,22 @@
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
# Suppress the HuggingFace tokenizers fork-after-parallelism warning. The Rust
|
||||
# ``tokenizers`` library warms a thread pool the first time a tokenizer is used
|
||||
# (e.g. the UMT5 / T5 text encoder during Wan / FLUX / SD3 conditioning), then
|
||||
# complains every time we fork() afterwards — which we do, on every MP4 encode,
|
||||
# because imageio's FFMPEG plugin shells out to ffmpeg via subprocess.Popen.
|
||||
# The warning is harmless (the child correctly falls back to single-threaded
|
||||
# tokenization before exec()) but it spams the log on every video generation.
|
||||
#
|
||||
# This MUST execute before any HF library is imported. The pyproject console-script
|
||||
# (``invokeai-web = invokeai.app.run_app:run_app``) reaches this module first via
|
||||
# ``from invokeai.app.run_app import run_app``, so setting the env var at module
|
||||
# level — not inside ``run_app()`` — guarantees it lands before any transitive HF
|
||||
# import. Use ``setdefault`` so anyone who explicitly exports ``true`` upstream
|
||||
# keeps their value.
|
||||
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
|
||||
|
||||
|
||||
def get_app():
|
||||
"""Import the app and event loop. We wrap this in a function to more explicitly control when it happens, because
|
||||
|
||||
@@ -104,3 +104,23 @@ def verify_token(token: str) -> TokenData | None:
|
||||
except Exception:
|
||||
# Catch any other exceptions (e.g., Pydantic validation errors)
|
||||
return None
|
||||
|
||||
|
||||
def get_token_remaining_seconds(token: str) -> int | None:
|
||||
"""Return the number of seconds until a *valid* token expires.
|
||||
|
||||
Verifies the token first (signature + expiry + payload shape); returns None if
|
||||
it fails verification. A valid token without an ``exp`` claim gets the default
|
||||
expiration window, matching what ``create_access_token`` would have assigned.
|
||||
"""
|
||||
if verify_token(token) is None:
|
||||
return None
|
||||
try:
|
||||
claims = jwt.get_unverified_claims(token)
|
||||
except JWTError:
|
||||
return None
|
||||
exp = claims.get("exp")
|
||||
if exp is None:
|
||||
return int(timedelta(hours=DEFAULT_EXPIRATION_HOURS).total_seconds())
|
||||
remaining = int(cast(float, exp) - datetime.now(timezone.utc).timestamp())
|
||||
return remaining if remaining > 0 else None
|
||||
|
||||
@@ -30,8 +30,13 @@ class BoardImageRecordStorageBase(ABC):
|
||||
board_id: str,
|
||||
categories: list[ImageCategory] | None,
|
||||
is_intermediate: bool | None,
|
||||
user_id: Optional[str] = None,
|
||||
) -> list[str]:
|
||||
"""Gets all board images for a board, as a list of the image names."""
|
||||
"""Gets all board images for a board, as a list of the image names.
|
||||
|
||||
When ``user_id`` is provided, results are restricted to images owned by that user;
|
||||
pass ``None`` for the admin path (no per-user restriction).
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
|
||||
@@ -80,6 +80,7 @@ class SqliteBoardImageRecordStorage(BoardImageRecordStorageBase):
|
||||
board_id: str,
|
||||
categories: list[ImageCategory] | None,
|
||||
is_intermediate: bool | None,
|
||||
user_id: Optional[str] = None,
|
||||
) -> list[str]:
|
||||
with self._db.transaction() as cursor:
|
||||
params: list[str | bool] = []
|
||||
@@ -124,6 +125,13 @@ class SqliteBoardImageRecordStorage(BoardImageRecordStorageBase):
|
||||
"""
|
||||
params.append(is_intermediate)
|
||||
|
||||
# Per-user filter — admins pass user_id=None to skip this clause.
|
||||
if user_id is not None:
|
||||
stmt += """--sql
|
||||
AND images.user_id = ?
|
||||
"""
|
||||
params.append(user_id)
|
||||
|
||||
# Put a ring on it
|
||||
stmt += ";"
|
||||
|
||||
|
||||
@@ -30,8 +30,13 @@ class BoardImagesServiceABC(ABC):
|
||||
board_id: str,
|
||||
categories: list[ImageCategory] | None,
|
||||
is_intermediate: bool | None,
|
||||
user_id: Optional[str] = None,
|
||||
) -> list[str]:
|
||||
"""Gets all board images for a board, as a list of the image names."""
|
||||
"""Gets all board images for a board, as a list of the image names.
|
||||
|
||||
When ``user_id`` is provided, results are restricted to images owned by that user;
|
||||
pass ``None`` for the admin path (no per-user restriction).
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
|
||||
@@ -29,11 +29,13 @@ class BoardImagesService(BoardImagesServiceABC):
|
||||
board_id: str,
|
||||
categories: list[ImageCategory] | None,
|
||||
is_intermediate: bool | None,
|
||||
user_id: Optional[str] = None,
|
||||
) -> list[str]:
|
||||
return self.__invoker.services.board_image_records.get_all_board_image_names_for_board(
|
||||
board_id,
|
||||
categories,
|
||||
is_intermediate,
|
||||
user_id=user_id,
|
||||
)
|
||||
|
||||
def get_board_for_image(
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Optional
|
||||
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory
|
||||
|
||||
|
||||
class BoardVideoRecordStorageBase(ABC):
|
||||
"""Abstract base class for the one-to-many board-video relationship record storage."""
|
||||
|
||||
@abstractmethod
|
||||
def add_video_to_board(self, board_id: str, video_name: str) -> None:
|
||||
"""Adds a video to a board."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def remove_video_from_board(self, video_name: str) -> None:
|
||||
"""Removes a video from a board."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_all_board_video_names_for_board(
|
||||
self,
|
||||
board_id: str,
|
||||
categories: list[ImageCategory] | None,
|
||||
is_intermediate: bool | None,
|
||||
user_id: Optional[str] = None,
|
||||
) -> list[str]:
|
||||
"""Gets all board videos for a board, as a list of the video names.
|
||||
|
||||
When ``user_id`` is provided, results are restricted to videos owned by that user;
|
||||
pass ``None`` for the admin path (no per-user restriction).
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_board_for_video(self, video_name: str) -> Optional[str]:
|
||||
"""Gets a video's board id, if it has one."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_video_count_for_board(self, board_id: str) -> int:
|
||||
"""Gets the number of videos for a board."""
|
||||
pass
|
||||
@@ -0,0 +1,97 @@
|
||||
import sqlite3
|
||||
from typing import Optional, cast
|
||||
|
||||
from invokeai.app.services.board_video_records.board_video_records_base import BoardVideoRecordStorageBase
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory
|
||||
from invokeai.app.services.shared.sqlite.sqlite_database import SqliteDatabase
|
||||
|
||||
|
||||
class SqliteBoardVideoRecordStorage(BoardVideoRecordStorageBase):
|
||||
def __init__(self, db: SqliteDatabase) -> None:
|
||||
super().__init__()
|
||||
self._db = db
|
||||
|
||||
def add_video_to_board(self, board_id: str, video_name: str) -> None:
|
||||
with self._db.transaction() as cursor:
|
||||
cursor.execute(
|
||||
"""--sql
|
||||
INSERT INTO board_videos (board_id, video_name)
|
||||
VALUES (?, ?)
|
||||
ON CONFLICT (video_name) DO UPDATE SET board_id = ?;
|
||||
""",
|
||||
(board_id, video_name, board_id),
|
||||
)
|
||||
|
||||
def remove_video_from_board(self, video_name: str) -> None:
|
||||
with self._db.transaction() as cursor:
|
||||
cursor.execute(
|
||||
"DELETE FROM board_videos WHERE video_name = ?;",
|
||||
(video_name,),
|
||||
)
|
||||
|
||||
def get_all_board_video_names_for_board(
|
||||
self,
|
||||
board_id: str,
|
||||
categories: list[ImageCategory] | None,
|
||||
is_intermediate: bool | None,
|
||||
user_id: Optional[str] = None,
|
||||
) -> list[str]:
|
||||
with self._db.transaction() as cursor:
|
||||
params: list[str | bool] = []
|
||||
stmt = """
|
||||
SELECT videos.video_name
|
||||
FROM videos
|
||||
LEFT JOIN board_videos ON board_videos.video_name = videos.video_name
|
||||
WHERE 1=1
|
||||
"""
|
||||
if board_id == "none":
|
||||
stmt += " AND board_videos.board_id IS NULL "
|
||||
else:
|
||||
stmt += " AND board_videos.board_id = ? "
|
||||
params.append(board_id)
|
||||
|
||||
if categories is not None:
|
||||
category_strings = [c.value for c in set(categories)]
|
||||
placeholders = ",".join("?" * len(category_strings))
|
||||
stmt += f" AND videos.video_category IN ( {placeholders} ) "
|
||||
for c in category_strings:
|
||||
params.append(c)
|
||||
|
||||
if is_intermediate is not None:
|
||||
stmt += " AND videos.is_intermediate = ? "
|
||||
params.append(is_intermediate)
|
||||
|
||||
if user_id is not None:
|
||||
stmt += " AND videos.user_id = ? "
|
||||
params.append(user_id)
|
||||
|
||||
stmt += ";"
|
||||
cursor.execute(stmt, params)
|
||||
result = cast(list[sqlite3.Row], cursor.fetchall())
|
||||
return [r[0] for r in result]
|
||||
|
||||
def get_board_for_video(self, video_name: str) -> Optional[str]:
|
||||
with self._db.transaction() as cursor:
|
||||
cursor.execute(
|
||||
"SELECT board_id FROM board_videos WHERE video_name = ?;",
|
||||
(video_name,),
|
||||
)
|
||||
result = cursor.fetchone()
|
||||
if result is None:
|
||||
return None
|
||||
return cast(str, result[0])
|
||||
|
||||
def get_video_count_for_board(self, board_id: str) -> int:
|
||||
with self._db.transaction() as cursor:
|
||||
cursor.execute(
|
||||
"""--sql
|
||||
SELECT COUNT(*)
|
||||
FROM board_videos
|
||||
INNER JOIN videos ON board_videos.video_name = videos.video_name
|
||||
WHERE videos.is_intermediate = FALSE
|
||||
AND board_videos.board_id = ?;
|
||||
""",
|
||||
(board_id,),
|
||||
)
|
||||
count = cast(int, cursor.fetchone()[0])
|
||||
return count
|
||||
@@ -10,8 +10,14 @@ class BoardDTO(BoardRecord):
|
||||
|
||||
cover_image_name: Optional[str] = Field(description="The name of the board's cover image.")
|
||||
"""The URL of the thumbnail of the most recent image in the board."""
|
||||
cover_video_name: Optional[str] = Field(
|
||||
default=None, description="The name of the board's cover video, when the most recent item is a video."
|
||||
)
|
||||
"""The name of the cover video, set when the most-recent item on the board is a video rather than an image."""
|
||||
image_count: int = Field(description="The number of images in the board.")
|
||||
"""The number of images in the board."""
|
||||
video_count: int = Field(default=0, description="The number of videos in the board.")
|
||||
"""The number of videos in the board."""
|
||||
asset_count: int = Field(description="The number of assets in the board.")
|
||||
"""The number of assets in the board."""
|
||||
owner_username: Optional[str] = Field(default=None, description="The username of the board owner (for admin view).")
|
||||
@@ -24,12 +30,16 @@ def board_record_to_dto(
|
||||
image_count: int,
|
||||
asset_count: int,
|
||||
owner_username: Optional[str] = None,
|
||||
cover_video_name: Optional[str] = None,
|
||||
video_count: int = 0,
|
||||
) -> BoardDTO:
|
||||
"""Converts a board record to a board DTO."""
|
||||
return BoardDTO(
|
||||
**board_record.model_dump(exclude={"cover_image_name"}),
|
||||
cover_image_name=cover_image_name,
|
||||
cover_video_name=cover_video_name,
|
||||
image_count=image_count,
|
||||
video_count=video_count,
|
||||
asset_count=asset_count,
|
||||
owner_username=owner_username,
|
||||
)
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
from typing import Optional
|
||||
|
||||
from invokeai.app.services.board_records.board_records_common import BoardChanges, BoardRecordOrderBy
|
||||
from invokeai.app.services.boards.boards_base import BoardServiceABC
|
||||
from invokeai.app.services.boards.boards_common import BoardDTO, board_record_to_dto
|
||||
@@ -12,6 +14,38 @@ class BoardService(BoardServiceABC):
|
||||
def start(self, invoker: Invoker) -> None:
|
||||
self.__invoker = invoker
|
||||
|
||||
def _resolve_cover(self, board_id: str) -> tuple[Optional[str], Optional[str]]:
|
||||
"""Pick the cover item for a board, considering both images and videos.
|
||||
|
||||
Returns ``(cover_image_name, cover_video_name)`` — at most one is set.
|
||||
The winner is chosen by ``(starred DESC, created_at DESC)`` across both
|
||||
tables so a recent video can supersede an older image (and vice versa).
|
||||
"""
|
||||
cover_image = self.__invoker.services.image_records.get_most_recent_image_for_board(board_id)
|
||||
cover_video = self.__invoker.services.video_records.get_most_recent_video_for_board(board_id)
|
||||
|
||||
if cover_image is None and cover_video is None:
|
||||
return None, None
|
||||
if cover_video is None:
|
||||
assert cover_image is not None
|
||||
return cover_image.image_name, None
|
||||
if cover_image is None:
|
||||
return None, cover_video.video_name
|
||||
|
||||
# Match the gallery's deterministic (starred, created_at, kind, name) ordering.
|
||||
image_key = (cover_image.starred, cover_image.created_at, "image", cover_image.image_name)
|
||||
video_key = (cover_video.starred, cover_video.created_at, "video", cover_video.video_name)
|
||||
if video_key > image_key:
|
||||
return None, cover_video.video_name
|
||||
return cover_image.image_name, None
|
||||
|
||||
def _get_counts(self, board_id: str) -> tuple[int, int, int]:
|
||||
"""Return ``(image_count, video_count, asset_count)`` for a board."""
|
||||
image_count = self.__invoker.services.board_image_records.get_image_count_for_board(board_id)
|
||||
asset_count = self.__invoker.services.board_image_records.get_asset_count_for_board(board_id)
|
||||
video_count = self.__invoker.services.board_video_records.get_video_count_for_board(board_id)
|
||||
return image_count, video_count, asset_count
|
||||
|
||||
def create(
|
||||
self,
|
||||
board_name: str,
|
||||
@@ -22,14 +56,16 @@ class BoardService(BoardServiceABC):
|
||||
|
||||
def get_dto(self, board_id: str) -> BoardDTO:
|
||||
board_record = self.__invoker.services.board_records.get(board_id)
|
||||
cover_image = self.__invoker.services.image_records.get_most_recent_image_for_board(board_record.board_id)
|
||||
if cover_image:
|
||||
cover_image_name = cover_image.image_name
|
||||
else:
|
||||
cover_image_name = None
|
||||
image_count = self.__invoker.services.board_image_records.get_image_count_for_board(board_id)
|
||||
asset_count = self.__invoker.services.board_image_records.get_asset_count_for_board(board_id)
|
||||
return board_record_to_dto(board_record, cover_image_name, image_count, asset_count)
|
||||
cover_image_name, cover_video_name = self._resolve_cover(board_record.board_id)
|
||||
image_count, video_count, asset_count = self._get_counts(board_id)
|
||||
return board_record_to_dto(
|
||||
board_record,
|
||||
cover_image_name,
|
||||
image_count,
|
||||
asset_count,
|
||||
cover_video_name=cover_video_name,
|
||||
video_count=video_count,
|
||||
)
|
||||
|
||||
def update(
|
||||
self,
|
||||
@@ -37,15 +73,16 @@ class BoardService(BoardServiceABC):
|
||||
changes: BoardChanges,
|
||||
) -> BoardDTO:
|
||||
board_record = self.__invoker.services.board_records.update(board_id, changes)
|
||||
cover_image = self.__invoker.services.image_records.get_most_recent_image_for_board(board_record.board_id)
|
||||
if cover_image:
|
||||
cover_image_name = cover_image.image_name
|
||||
else:
|
||||
cover_image_name = None
|
||||
|
||||
image_count = self.__invoker.services.board_image_records.get_image_count_for_board(board_id)
|
||||
asset_count = self.__invoker.services.board_image_records.get_asset_count_for_board(board_id)
|
||||
return board_record_to_dto(board_record, cover_image_name, image_count, asset_count)
|
||||
cover_image_name, cover_video_name = self._resolve_cover(board_record.board_id)
|
||||
image_count, video_count, asset_count = self._get_counts(board_id)
|
||||
return board_record_to_dto(
|
||||
board_record,
|
||||
cover_image_name,
|
||||
image_count,
|
||||
asset_count,
|
||||
cover_video_name=cover_video_name,
|
||||
video_count=video_count,
|
||||
)
|
||||
|
||||
def delete(self, board_id: str) -> None:
|
||||
self.__invoker.services.board_records.delete(board_id)
|
||||
@@ -63,16 +100,12 @@ class BoardService(BoardServiceABC):
|
||||
board_records = self.__invoker.services.board_records.get_many(
|
||||
user_id, is_admin, order_by, direction, offset, limit, include_archived
|
||||
)
|
||||
summaries = self.__invoker.services.gallery.get_board_media_summaries(
|
||||
[record.board_id for record in board_records.items]
|
||||
)
|
||||
board_dtos = []
|
||||
for r in board_records.items:
|
||||
cover_image = self.__invoker.services.image_records.get_most_recent_image_for_board(r.board_id)
|
||||
if cover_image:
|
||||
cover_image_name = cover_image.image_name
|
||||
else:
|
||||
cover_image_name = None
|
||||
|
||||
image_count = self.__invoker.services.board_image_records.get_image_count_for_board(r.board_id)
|
||||
asset_count = self.__invoker.services.board_image_records.get_asset_count_for_board(r.board_id)
|
||||
summary = summaries[r.board_id]
|
||||
|
||||
# For admin users, include owner username
|
||||
owner_username = None
|
||||
@@ -81,7 +114,17 @@ class BoardService(BoardServiceABC):
|
||||
if owner:
|
||||
owner_username = owner.display_name or owner.email
|
||||
|
||||
board_dtos.append(board_record_to_dto(r, cover_image_name, image_count, asset_count, owner_username))
|
||||
board_dtos.append(
|
||||
board_record_to_dto(
|
||||
r,
|
||||
summary.cover_image_name,
|
||||
summary.image_count,
|
||||
summary.asset_count,
|
||||
owner_username,
|
||||
cover_video_name=summary.cover_video_name,
|
||||
video_count=summary.video_count,
|
||||
)
|
||||
)
|
||||
|
||||
return OffsetPaginatedResults[BoardDTO](items=board_dtos, offset=offset, limit=limit, total=len(board_dtos))
|
||||
|
||||
@@ -96,16 +139,12 @@ class BoardService(BoardServiceABC):
|
||||
board_records = self.__invoker.services.board_records.get_all(
|
||||
user_id, is_admin, order_by, direction, include_archived
|
||||
)
|
||||
summaries = self.__invoker.services.gallery.get_board_media_summaries(
|
||||
[record.board_id for record in board_records]
|
||||
)
|
||||
board_dtos = []
|
||||
for r in board_records:
|
||||
cover_image = self.__invoker.services.image_records.get_most_recent_image_for_board(r.board_id)
|
||||
if cover_image:
|
||||
cover_image_name = cover_image.image_name
|
||||
else:
|
||||
cover_image_name = None
|
||||
|
||||
image_count = self.__invoker.services.board_image_records.get_image_count_for_board(r.board_id)
|
||||
asset_count = self.__invoker.services.board_image_records.get_asset_count_for_board(r.board_id)
|
||||
summary = summaries[r.board_id]
|
||||
|
||||
# For admin users, include owner username
|
||||
owner_username = None
|
||||
@@ -114,6 +153,16 @@ class BoardService(BoardServiceABC):
|
||||
if owner:
|
||||
owner_username = owner.display_name or owner.email
|
||||
|
||||
board_dtos.append(board_record_to_dto(r, cover_image_name, image_count, asset_count, owner_username))
|
||||
board_dtos.append(
|
||||
board_record_to_dto(
|
||||
r,
|
||||
summary.cover_image_name,
|
||||
summary.image_count,
|
||||
summary.asset_count,
|
||||
owner_username,
|
||||
cover_video_name=summary.cover_video_name,
|
||||
video_count=summary.video_count,
|
||||
)
|
||||
)
|
||||
|
||||
return board_dtos
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Optional
|
||||
|
||||
from invokeai.app.services.gallery.gallery_common import BoardMediaSummary, GalleryItem, GalleryItemNamesResult
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
|
||||
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
|
||||
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
|
||||
from invokeai.app.services.virtual_boards.virtual_boards_common import VirtualSubBoardDTO
|
||||
|
||||
|
||||
class GalleryServiceABC(ABC):
|
||||
"""High-level service producing a polymorphic stream of images and videos."""
|
||||
|
||||
@abstractmethod
|
||||
def list_items(
|
||||
self,
|
||||
offset: int = 0,
|
||||
limit: int = 10,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> OffsetPaginatedResults[GalleryItem]:
|
||||
"""Lists a paginated, time-sorted stream of image + video items."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def list_item_names(
|
||||
self,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
created_date: Optional[str] = None,
|
||||
) -> GalleryItemNamesResult:
|
||||
"""Returns ordered (kind, name) refs for optimistic UI / virtualized lists.
|
||||
|
||||
`created_date` restricts the result to items created on the given ISO date — used by
|
||||
date-based virtual boards.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_dates(
|
||||
self,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> list[VirtualSubBoardDTO]:
|
||||
"""Returns date-based virtual sub-boards covering both images and videos."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_board_media_summaries(self, board_ids: list[str]) -> dict[str, BoardMediaSummary]:
|
||||
"""Returns covers and media counts for each requested board."""
|
||||
pass
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Polymorphic gallery types: images and videos appearing in a single time-sorted stream."""
|
||||
|
||||
import datetime
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Optional, Union
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory
|
||||
from invokeai.app.util.metaenum import MetaEnum
|
||||
from invokeai.app.util.model_exclude_null import BaseModelExcludeNull
|
||||
|
||||
|
||||
class GalleryItemKind(str, Enum, metaclass=MetaEnum):
|
||||
"""Discriminator for polymorphic gallery items."""
|
||||
|
||||
IMAGE = "image"
|
||||
VIDEO = "video"
|
||||
|
||||
|
||||
class GalleryItemRef(BaseModel):
|
||||
"""A thin reference to a gallery item — used for ordered name lists."""
|
||||
|
||||
kind: GalleryItemKind = Field(description="Whether the item is an image or video.")
|
||||
name: str = Field(description="The unique name of the image or video.")
|
||||
|
||||
|
||||
class GalleryItem(BaseModelExcludeNull):
|
||||
"""A gallery item — either an image or a video, with shared fields and a discriminator.
|
||||
|
||||
Frontend code should dispatch on `kind` to render image- vs video-specific UI.
|
||||
"""
|
||||
|
||||
kind: GalleryItemKind = Field(description="Whether the item is an image or video.")
|
||||
name: str = Field(description="The unique name of the image or video.")
|
||||
full_url: str = Field(description="URL to the full-resolution image PNG or the full-quality video MP4.")
|
||||
thumbnail_url: str = Field(description="URL to the static (WebP) thumbnail.")
|
||||
width: int = Field(description="The width of the item in pixels.")
|
||||
height: int = Field(description="The height of the item in pixels.")
|
||||
category: ImageCategory = Field(description="The category of the item (images and videos share the same enum).")
|
||||
starred: bool = Field(description="Whether the item is starred.")
|
||||
is_intermediate: bool = Field(description="Whether the item is an intermediate output.")
|
||||
board_id: Optional[str] = Field(default=None, description="Owning board id, if any.")
|
||||
created_at: Union[datetime.datetime, str] = Field(description="The created timestamp of the item.")
|
||||
# Video-only fields. None for images.
|
||||
duration: Optional[float] = Field(default=None, description="Video duration in seconds. None for images.")
|
||||
fps: Optional[float] = Field(default=None, description="Video frames per second. None for images.")
|
||||
|
||||
|
||||
class GalleryItemNamesResult(BaseModel):
|
||||
"""Ordered list of gallery item references plus counts for optimistic UI."""
|
||||
|
||||
items: list[GalleryItemRef] = Field(description="Ordered list of (kind, name) references.")
|
||||
starred_count: int = Field(description="Number of starred items (when starred_first=True).")
|
||||
total_count: int = Field(description="Total number of items matching the query.")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BoardMediaSummary:
|
||||
cover_image_name: Optional[str] = None
|
||||
cover_video_name: Optional[str] = None
|
||||
image_count: int = 0
|
||||
video_count: int = 0
|
||||
asset_count: int = 0
|
||||
@@ -0,0 +1,471 @@
|
||||
import sqlite3
|
||||
from typing import Optional, Union, cast
|
||||
|
||||
from invokeai.app.services.gallery.gallery_base import GalleryServiceABC
|
||||
from invokeai.app.services.gallery.gallery_common import (
|
||||
BoardMediaSummary,
|
||||
GalleryItem,
|
||||
GalleryItemKind,
|
||||
GalleryItemNamesResult,
|
||||
GalleryItemRef,
|
||||
)
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
|
||||
from invokeai.app.services.invoker import Invoker
|
||||
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
|
||||
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
|
||||
from invokeai.app.services.shared.sqlite.sqlite_database import SqliteDatabase
|
||||
from invokeai.app.services.virtual_boards.virtual_boards_common import VirtualSubBoardDTO
|
||||
|
||||
|
||||
class SqliteGalleryService(GalleryServiceABC):
|
||||
"""Implements a polymorphic gallery via UNION ALL across the `images` and `videos` tables.
|
||||
|
||||
Filters are applied identically on each half. The two halves expose a common column set so
|
||||
the result is shape-compatible (a literal `kind` discriminator + a `name` alias + duration/fps
|
||||
that are NULL for images).
|
||||
"""
|
||||
|
||||
__invoker: Invoker
|
||||
|
||||
def __init__(self, db: SqliteDatabase) -> None:
|
||||
super().__init__()
|
||||
self._db = db
|
||||
|
||||
def start(self, invoker: Invoker) -> None:
|
||||
self.__invoker = invoker
|
||||
|
||||
def list_items(
|
||||
self,
|
||||
offset: int = 0,
|
||||
limit: int = 10,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> OffsetPaginatedResults[GalleryItem]:
|
||||
image_half, image_params, image_count_query = self._build_half(
|
||||
kind="image",
|
||||
origin=origin,
|
||||
categories=categories,
|
||||
is_intermediate=is_intermediate,
|
||||
board_id=board_id,
|
||||
search_term=search_term,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
video_half, video_params, video_count_query = self._build_half(
|
||||
kind="video",
|
||||
origin=origin,
|
||||
categories=categories,
|
||||
is_intermediate=is_intermediate,
|
||||
board_id=board_id,
|
||||
search_term=search_term,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
|
||||
order_clause = self._build_order_clause(starred_first, order_dir)
|
||||
|
||||
union_query = f"""--sql
|
||||
SELECT * FROM (
|
||||
{image_half}
|
||||
UNION ALL
|
||||
{video_half}
|
||||
)
|
||||
{order_clause}
|
||||
LIMIT ? OFFSET ?
|
||||
;
|
||||
"""
|
||||
|
||||
with self._db.transaction() as cursor:
|
||||
cursor.execute(union_query, image_params + video_params + [limit, offset])
|
||||
rows = cast(list[sqlite3.Row], cursor.fetchall())
|
||||
|
||||
cursor.execute(image_count_query, image_params)
|
||||
image_count = cast(int, cursor.fetchone()[0])
|
||||
cursor.execute(video_count_query, video_params)
|
||||
video_count = cast(int, cursor.fetchone()[0])
|
||||
|
||||
urls = self.__invoker.services.urls
|
||||
items = [self._row_to_item(row, urls) for row in rows]
|
||||
return OffsetPaginatedResults[GalleryItem](
|
||||
items=items,
|
||||
offset=offset,
|
||||
limit=limit,
|
||||
total=image_count + video_count,
|
||||
)
|
||||
|
||||
def list_item_names(
|
||||
self,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
created_date: Optional[str] = None,
|
||||
) -> GalleryItemNamesResult:
|
||||
image_half, image_params, _ = self._build_half(
|
||||
kind="image",
|
||||
origin=origin,
|
||||
categories=categories,
|
||||
is_intermediate=is_intermediate,
|
||||
board_id=board_id,
|
||||
search_term=search_term,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
names_only=True,
|
||||
created_date=created_date,
|
||||
)
|
||||
video_half, video_params, _ = self._build_half(
|
||||
kind="video",
|
||||
origin=origin,
|
||||
categories=categories,
|
||||
is_intermediate=is_intermediate,
|
||||
board_id=board_id,
|
||||
search_term=search_term,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
names_only=True,
|
||||
created_date=created_date,
|
||||
)
|
||||
|
||||
order_clause = self._build_order_clause(starred_first, order_dir)
|
||||
|
||||
union_query = f"""--sql
|
||||
SELECT * FROM (
|
||||
{image_half}
|
||||
UNION ALL
|
||||
{video_half}
|
||||
)
|
||||
{order_clause}
|
||||
;
|
||||
"""
|
||||
|
||||
with self._db.transaction() as cursor:
|
||||
cursor.execute(union_query, image_params + video_params)
|
||||
rows = cast(list[sqlite3.Row], cursor.fetchall())
|
||||
|
||||
starred_count = 0
|
||||
if starred_first:
|
||||
starred_count = sum(1 for r in rows if r["starred"])
|
||||
|
||||
refs = [GalleryItemRef(kind=GalleryItemKind(row["kind"]), name=row["name"]) for row in rows]
|
||||
return GalleryItemNamesResult(items=refs, starred_count=starred_count, total_count=len(refs))
|
||||
|
||||
def get_dates(
|
||||
self,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> list[VirtualSubBoardDTO]:
|
||||
image_conditions = " AND images.is_intermediate = 0 "
|
||||
video_conditions = " AND videos.is_intermediate = 0 "
|
||||
image_params: list[Union[int, str, bool]] = []
|
||||
video_params: list[Union[int, str, bool]] = []
|
||||
|
||||
if user_id is not None and not is_admin:
|
||||
image_conditions += " AND images.user_id = ? "
|
||||
image_params.append(user_id)
|
||||
video_conditions += " AND videos.user_id = ? "
|
||||
video_params.append(user_id)
|
||||
|
||||
union = f"""--sql
|
||||
SELECT
|
||||
images.created_at AS created_at,
|
||||
'image' AS kind,
|
||||
images.image_name AS name,
|
||||
images.image_category AS category
|
||||
FROM images
|
||||
WHERE 1=1 {image_conditions}
|
||||
UNION ALL
|
||||
SELECT
|
||||
videos.created_at AS created_at,
|
||||
'video' AS kind,
|
||||
videos.video_name AS name,
|
||||
videos.video_category AS category
|
||||
FROM videos
|
||||
WHERE 1=1 {video_conditions}
|
||||
"""
|
||||
|
||||
counts_query = f"""--sql
|
||||
SELECT
|
||||
DATE(created_at) AS date,
|
||||
SUM(CASE WHEN kind = 'image' AND category = 'general' THEN 1 ELSE 0 END) AS image_count,
|
||||
SUM(CASE WHEN kind = 'image' AND category != 'general' THEN 1 ELSE 0 END) AS asset_count,
|
||||
SUM(CASE WHEN kind = 'video' THEN 1 ELSE 0 END) AS video_count
|
||||
FROM ({union})
|
||||
GROUP BY DATE(created_at)
|
||||
ORDER BY date DESC;
|
||||
"""
|
||||
|
||||
# The cover is the newest item of each date. ROW_NUMBER with kind/name tie-breakers
|
||||
# (rather than a bare-column MAX() aggregate) keeps the choice deterministic when
|
||||
# several items share the maximum timestamp — otherwise the cover could flicker
|
||||
# between equally-new items across refetches.
|
||||
covers_query = f"""--sql
|
||||
SELECT date, kind, name FROM (
|
||||
SELECT
|
||||
DATE(created_at) AS date,
|
||||
kind,
|
||||
name,
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY DATE(created_at)
|
||||
ORDER BY created_at DESC, kind DESC, name DESC
|
||||
) AS rn
|
||||
FROM ({union})
|
||||
)
|
||||
WHERE rn = 1;
|
||||
"""
|
||||
|
||||
with self._db.transaction() as cursor:
|
||||
cursor.execute(counts_query, image_params + video_params)
|
||||
count_rows = cast(list[sqlite3.Row], cursor.fetchall())
|
||||
cursor.execute(covers_query, image_params + video_params)
|
||||
cover_rows = cast(list[sqlite3.Row], cursor.fetchall())
|
||||
|
||||
covers = {row["date"]: (row["kind"], row["name"]) for row in cover_rows}
|
||||
|
||||
boards: list[VirtualSubBoardDTO] = []
|
||||
for row in count_rows:
|
||||
date = row["date"]
|
||||
cover_kind, cover_name = covers.get(date, (None, None))
|
||||
boards.append(
|
||||
VirtualSubBoardDTO(
|
||||
virtual_board_id=f"by_date:{date}",
|
||||
board_name=date,
|
||||
date=date,
|
||||
image_count=row["image_count"],
|
||||
asset_count=row["asset_count"],
|
||||
video_count=row["video_count"],
|
||||
cover_image_name=cover_name if cover_kind == "image" else None,
|
||||
cover_video_name=cover_name if cover_kind == "video" else None,
|
||||
)
|
||||
)
|
||||
return boards
|
||||
|
||||
def get_board_media_summaries(self, board_ids: list[str]) -> dict[str, BoardMediaSummary]:
|
||||
summaries = {board_id: BoardMediaSummary() for board_id in board_ids}
|
||||
if not board_ids:
|
||||
return summaries
|
||||
|
||||
placeholders = ",".join("?" for _ in board_ids)
|
||||
query = f"""--sql
|
||||
SELECT
|
||||
board_id,
|
||||
SUM(CASE WHEN kind = 'image' AND category = 'general' THEN 1 ELSE 0 END) AS image_count,
|
||||
SUM(CASE WHEN kind = 'image' AND category != 'general' THEN 1 ELSE 0 END) AS asset_count,
|
||||
SUM(CASE WHEN kind = 'video' THEN 1 ELSE 0 END) AS video_count,
|
||||
MAX(CASE WHEN rank = 1 AND kind = 'image' THEN name END) AS cover_image_name,
|
||||
MAX(CASE WHEN rank = 1 AND kind = 'video' THEN name END) AS cover_video_name
|
||||
FROM (
|
||||
SELECT
|
||||
board_id,
|
||||
kind,
|
||||
name,
|
||||
category,
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY board_id
|
||||
ORDER BY starred DESC, created_at DESC, kind DESC, name DESC
|
||||
) AS rank
|
||||
FROM (
|
||||
SELECT
|
||||
board_images.board_id AS board_id,
|
||||
'image' AS kind,
|
||||
images.image_name AS name,
|
||||
images.image_category AS category,
|
||||
images.starred AS starred,
|
||||
images.created_at AS created_at
|
||||
FROM board_images
|
||||
INNER JOIN images ON board_images.image_name = images.image_name
|
||||
WHERE images.is_intermediate = FALSE
|
||||
AND board_images.board_id IN ({placeholders})
|
||||
UNION ALL
|
||||
SELECT
|
||||
board_videos.board_id AS board_id,
|
||||
'video' AS kind,
|
||||
videos.video_name AS name,
|
||||
videos.video_category AS category,
|
||||
videos.starred AS starred,
|
||||
videos.created_at AS created_at
|
||||
FROM board_videos
|
||||
INNER JOIN videos ON board_videos.video_name = videos.video_name
|
||||
WHERE videos.is_intermediate = FALSE
|
||||
AND board_videos.board_id IN ({placeholders})
|
||||
)
|
||||
)
|
||||
GROUP BY board_id;
|
||||
"""
|
||||
with self._db.transaction() as cursor:
|
||||
cursor.execute(query, [*board_ids, *board_ids])
|
||||
rows = cast(list[sqlite3.Row], cursor.fetchall())
|
||||
for row in rows:
|
||||
summaries[row["board_id"]] = BoardMediaSummary(
|
||||
cover_image_name=row["cover_image_name"],
|
||||
cover_video_name=row["cover_video_name"],
|
||||
image_count=row["image_count"],
|
||||
video_count=row["video_count"],
|
||||
asset_count=row["asset_count"],
|
||||
)
|
||||
return summaries
|
||||
|
||||
@staticmethod
|
||||
def _build_order_clause(starred_first: bool, order_dir: SQLiteDirection) -> str:
|
||||
"""Ordering with `kind` + `name` tie-breakers.
|
||||
|
||||
Images and videos created within the same timestamp granularity would otherwise
|
||||
have no defined relative order — rows could reorder across refetches or shift
|
||||
between offset pages. The tie-breakers follow `order_dir` so ascending and
|
||||
descending remain exact mirror images.
|
||||
"""
|
||||
direction = order_dir.value
|
||||
tie_breakers = f"kind {direction}, name {direction}"
|
||||
if starred_first:
|
||||
return f"ORDER BY starred DESC, created_at {direction}, {tie_breakers}"
|
||||
return f"ORDER BY created_at {direction}, {tie_breakers}"
|
||||
|
||||
def _build_half(
|
||||
self,
|
||||
kind: str,
|
||||
origin: Optional[ResourceOrigin],
|
||||
categories: Optional[list[ImageCategory]],
|
||||
is_intermediate: Optional[bool],
|
||||
board_id: Optional[str],
|
||||
search_term: Optional[str],
|
||||
user_id: Optional[str],
|
||||
is_admin: bool,
|
||||
names_only: bool = False,
|
||||
created_date: Optional[str] = None,
|
||||
) -> tuple[str, list[Union[int, str, bool]], str]:
|
||||
"""Builds one half of the union (either `images` or `videos`).
|
||||
|
||||
Returns `(query_with_select, params, count_query)`. Both halves emit the same columns so
|
||||
UNION ALL is shape-compatible: `kind`, `name`, `width`, `height`, `category`, `starred`,
|
||||
`is_intermediate`, `board_id`, `created_at`, `duration`, `fps`.
|
||||
|
||||
`names_only=True` selects only `kind`, `name`, `starred`, `created_at` (the minimum needed
|
||||
for ordering + the counts result).
|
||||
"""
|
||||
if kind == "image":
|
||||
base_table = "images"
|
||||
join_table = "board_images"
|
||||
name_col = "image_name"
|
||||
category_col = "image_category"
|
||||
origin_col = "image_origin"
|
||||
duration_expr = "NULL"
|
||||
fps_expr = "NULL"
|
||||
elif kind == "video":
|
||||
base_table = "videos"
|
||||
join_table = "board_videos"
|
||||
name_col = "video_name"
|
||||
category_col = "video_category"
|
||||
origin_col = "video_origin"
|
||||
duration_expr = f"{base_table}.duration"
|
||||
fps_expr = f"{base_table}.fps"
|
||||
else:
|
||||
raise ValueError(f"Unknown kind: {kind}")
|
||||
|
||||
if names_only:
|
||||
select_cols = (
|
||||
f"'{kind}' AS kind, "
|
||||
f"{base_table}.{name_col} AS name, "
|
||||
f"{base_table}.starred AS starred, "
|
||||
f"{base_table}.created_at AS created_at"
|
||||
)
|
||||
else:
|
||||
select_cols = (
|
||||
f"'{kind}' AS kind, "
|
||||
f"{base_table}.{name_col} AS name, "
|
||||
f"{base_table}.width AS width, "
|
||||
f"{base_table}.height AS height, "
|
||||
f"{base_table}.{category_col} AS category, "
|
||||
f"{base_table}.starred AS starred, "
|
||||
f"{base_table}.is_intermediate AS is_intermediate, "
|
||||
f"{join_table}.board_id AS board_id, "
|
||||
f"{base_table}.created_at AS created_at, "
|
||||
f"{duration_expr} AS duration, "
|
||||
f"{fps_expr} AS fps"
|
||||
)
|
||||
|
||||
from_clause = f"FROM {base_table} LEFT JOIN {join_table} ON {join_table}.{name_col} = {base_table}.{name_col}"
|
||||
|
||||
conditions = ""
|
||||
params: list[Union[int, str, bool]] = []
|
||||
|
||||
if origin is not None:
|
||||
conditions += f" AND {base_table}.{origin_col} = ? "
|
||||
params.append(origin.value)
|
||||
|
||||
if categories is not None:
|
||||
category_strings = [c.value for c in set(categories)]
|
||||
placeholders = ",".join("?" * len(category_strings))
|
||||
conditions += f" AND {base_table}.{category_col} IN ( {placeholders} ) "
|
||||
for c in category_strings:
|
||||
params.append(c)
|
||||
|
||||
if is_intermediate is not None:
|
||||
conditions += f" AND {base_table}.is_intermediate = ? "
|
||||
params.append(is_intermediate)
|
||||
|
||||
if created_date is not None:
|
||||
conditions += f" AND DATE({base_table}.created_at) = ? "
|
||||
params.append(created_date)
|
||||
|
||||
if board_id == "none":
|
||||
conditions += f" AND {join_table}.board_id IS NULL "
|
||||
if user_id is not None and not is_admin:
|
||||
conditions += f" AND {base_table}.user_id = ? "
|
||||
params.append(user_id)
|
||||
elif board_id is not None:
|
||||
conditions += f" AND {join_table}.board_id = ? "
|
||||
params.append(board_id)
|
||||
elif user_id is not None and not is_admin:
|
||||
# No board_id supplied — still enforce per-user isolation so
|
||||
# non-admin callers cannot enumerate other users' items.
|
||||
conditions += f" AND {base_table}.user_id = ? "
|
||||
params.append(user_id)
|
||||
|
||||
if search_term:
|
||||
conditions += f" AND ({base_table}.metadata LIKE ? OR {base_table}.created_at LIKE ?) "
|
||||
params.append(f"%{search_term.lower()}%")
|
||||
params.append(f"%{search_term.lower()}%")
|
||||
|
||||
half_query = f"SELECT {select_cols} {from_clause} WHERE 1=1 {conditions}"
|
||||
count_query = f"SELECT COUNT(*) {from_clause} WHERE 1=1 {conditions}"
|
||||
return half_query, params, count_query
|
||||
|
||||
def _row_to_item(self, row: sqlite3.Row, urls) -> GalleryItem:
|
||||
kind = GalleryItemKind(row["kind"])
|
||||
name = row["name"]
|
||||
if kind == GalleryItemKind.IMAGE:
|
||||
full_url = urls.get_image_url(name)
|
||||
thumbnail_url = urls.get_image_url(name, thumbnail=True)
|
||||
duration = None
|
||||
fps = None
|
||||
else:
|
||||
full_url = urls.get_video_url(name)
|
||||
thumbnail_url = urls.get_video_url(name, thumbnail=True)
|
||||
duration = row["duration"]
|
||||
fps = row["fps"]
|
||||
return GalleryItem(
|
||||
kind=kind,
|
||||
name=name,
|
||||
full_url=full_url,
|
||||
thumbnail_url=thumbnail_url,
|
||||
width=row["width"],
|
||||
height=row["height"],
|
||||
category=ImageCategory(row["category"]),
|
||||
starred=bool(row["starred"]),
|
||||
is_intermediate=bool(row["is_intermediate"]),
|
||||
board_id=row["board_id"],
|
||||
created_at=row["created_at"],
|
||||
duration=duration,
|
||||
fps=fps,
|
||||
)
|
||||
@@ -61,6 +61,21 @@ class ImageFileStorageBase(ABC):
|
||||
"""Deletes an image and its thumbnail (if one exists)."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def stage_delete(self, image_name: str, image_subfolder: str = "") -> object:
|
||||
"""Moves an image's files out of service and returns a rollback token."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def commit_delete(self, token: object) -> None:
|
||||
"""Permanently removes files represented by a staged-delete token."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def rollback_delete(self, token: object) -> None:
|
||||
"""Restores files represented by a staged-delete token."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_workflow(self, image_name: str, image_subfolder: str = "") -> Optional[str]:
|
||||
"""Gets the workflow of an image."""
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
# Copyright (c) 2022 Kyle Schouviller (https://github.com/kyle0654) and the InvokeAI Team
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import zlib
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from queue import Queue
|
||||
from typing import Optional, Union
|
||||
@@ -16,6 +21,7 @@ from invokeai.app.services.image_files.image_files_common import (
|
||||
)
|
||||
from invokeai.app.services.invoker import Invoker
|
||||
from invokeai.app.util.thumbnails import get_thumbnail_name, make_thumbnail
|
||||
from invokeai.backend.util.logging import InvokeAILogger
|
||||
|
||||
_PNG_RLE_MIN_PIXELS = 512 * 512
|
||||
_PNG_RLE_SAMPLE_TILE = 32
|
||||
@@ -23,6 +29,12 @@ _PNG_RLE_MIN_RAW_SIZE_PERCENT = 30
|
||||
_PNG_RLE_MAX_SAMPLE_SIZE_PERCENT = 102
|
||||
|
||||
|
||||
@dataclass
|
||||
class _StagedDelete:
|
||||
directory: Path
|
||||
files: list[tuple[Path, Path]]
|
||||
|
||||
|
||||
def _get_png_size(image: PILImageType, compress_type: Optional[int] = None) -> int:
|
||||
output = io.BytesIO()
|
||||
options = {"compress_level": 1}
|
||||
@@ -73,6 +85,7 @@ class DiskImageFileStorage(ImageFileStorageBase):
|
||||
|
||||
def start(self, invoker: Invoker) -> None:
|
||||
self.__invoker = invoker
|
||||
self.__recover_staged_deletes()
|
||||
|
||||
@property
|
||||
def image_root(self) -> Path:
|
||||
@@ -157,20 +170,53 @@ class DiskImageFileStorage(ImageFileStorageBase):
|
||||
raise ImageFileSaveException from e
|
||||
|
||||
def delete(self, image_name: str, image_subfolder: str = "") -> None:
|
||||
token = self.stage_delete(image_name, image_subfolder)
|
||||
self.commit_delete(token)
|
||||
|
||||
def stage_delete(self, image_name: str, image_subfolder: str = "") -> _StagedDelete:
|
||||
candidates = [
|
||||
self.get_path(image_name, image_subfolder=image_subfolder),
|
||||
self.get_path(image_name, thumbnail=True, image_subfolder=image_subfolder),
|
||||
]
|
||||
staging_dir = Path(tempfile.mkdtemp(prefix=".delete_", dir=self.__output_folder))
|
||||
staged: list[tuple[Path, Path]] = []
|
||||
try:
|
||||
image_path = self.get_path(image_name, image_subfolder=image_subfolder)
|
||||
with open(staging_dir / "manifest.json", "w", encoding="utf-8") as manifest:
|
||||
manifest.write(json.dumps({"image_name": image_name, "image_subfolder": image_subfolder}))
|
||||
manifest.flush()
|
||||
os.fsync(manifest.fileno())
|
||||
for index, source in enumerate(candidates):
|
||||
self.__cache.pop(source, None)
|
||||
if source.exists():
|
||||
destination = staging_dir / str(index)
|
||||
source.replace(destination)
|
||||
staged.append((source, destination))
|
||||
return _StagedDelete(directory=staging_dir, files=staged)
|
||||
except Exception as e:
|
||||
for source, destination in reversed(staged):
|
||||
if destination.exists():
|
||||
source.parent.mkdir(parents=True, exist_ok=True)
|
||||
destination.replace(source)
|
||||
shutil.rmtree(staging_dir, ignore_errors=True)
|
||||
raise ImageFileDeleteException from e
|
||||
|
||||
if image_path.exists():
|
||||
image_path.unlink()
|
||||
if image_path in self.__cache:
|
||||
del self.__cache[image_path]
|
||||
def commit_delete(self, token: object) -> None:
|
||||
if not isinstance(token, _StagedDelete):
|
||||
raise ImageFileDeleteException("Invalid staged-delete token")
|
||||
try:
|
||||
shutil.rmtree(token.directory)
|
||||
except Exception as e:
|
||||
raise ImageFileDeleteException from e
|
||||
|
||||
thumbnail_path = self.get_path(image_name, True, image_subfolder=image_subfolder)
|
||||
|
||||
if thumbnail_path.exists():
|
||||
thumbnail_path.unlink()
|
||||
if thumbnail_path in self.__cache:
|
||||
del self.__cache[thumbnail_path]
|
||||
def rollback_delete(self, token: object) -> None:
|
||||
if not isinstance(token, _StagedDelete):
|
||||
raise ImageFileDeleteException("Invalid staged-delete token")
|
||||
try:
|
||||
for source, destination in reversed(token.files):
|
||||
if destination.exists():
|
||||
source.parent.mkdir(parents=True, exist_ok=True)
|
||||
destination.replace(source)
|
||||
shutil.rmtree(token.directory, ignore_errors=True)
|
||||
except Exception as e:
|
||||
raise ImageFileDeleteException from e
|
||||
|
||||
@@ -240,6 +286,37 @@ class DiskImageFileStorage(ImageFileStorageBase):
|
||||
for folder in folders:
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def __recover_staged_deletes(self) -> None:
|
||||
logger = InvokeAILogger.get_logger()
|
||||
for staging_dir in self.__output_folder.glob(".delete_*"):
|
||||
manifest_path = staging_dir / "manifest.json"
|
||||
if not manifest_path.is_file():
|
||||
if not any(staging_dir.iterdir()):
|
||||
staging_dir.rmdir()
|
||||
continue
|
||||
try:
|
||||
with open(manifest_path, encoding="utf-8") as manifest:
|
||||
data = json.load(manifest)
|
||||
image_name = data["image_name"]
|
||||
image_subfolder = data.get("image_subfolder", "")
|
||||
candidates = [
|
||||
self.get_path(image_name, image_subfolder=image_subfolder),
|
||||
self.get_path(image_name, thumbnail=True, image_subfolder=image_subfolder),
|
||||
]
|
||||
token = _StagedDelete(
|
||||
directory=staging_dir,
|
||||
files=[(source, staging_dir / str(index)) for index, source in enumerate(candidates)],
|
||||
)
|
||||
self.__invoker.services.image_records.get(image_name)
|
||||
self.rollback_delete(token)
|
||||
except Exception as error:
|
||||
from invokeai.app.services.image_records.image_records_common import ImageRecordNotFoundException
|
||||
|
||||
if isinstance(error, ImageRecordNotFoundException):
|
||||
shutil.rmtree(staging_dir, ignore_errors=True)
|
||||
else:
|
||||
logger.error(f"Failed to recover staged image deletion {staging_dir}: {error}")
|
||||
|
||||
def __get_cache(self, image_name: Path) -> Optional[PILImageType]:
|
||||
return None if image_name not in self.__cache else self.__cache[image_name]
|
||||
|
||||
|
||||
@@ -148,8 +148,17 @@ class ImageServiceABC(ABC):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete_images_on_board(self, board_id: str):
|
||||
"""Deletes all images on a board."""
|
||||
def delete_images_on_board(self, board_id: str, user_id: Optional[str] = None) -> tuple[list[str], list[str]]:
|
||||
"""Deletes all images on a board; returns ``(deleted_names, failed_names)``.
|
||||
|
||||
When ``user_id`` is provided, only images owned by that user are deleted (other users'
|
||||
contributions to a public/shared board are preserved). Pass ``None`` for the admin
|
||||
path to delete every image on the board regardless of uploader.
|
||||
|
||||
``failed_names`` is the service's ground truth for per-image failures — callers must
|
||||
not reconstruct it by diffing their own board listing against ``deleted_names``, which
|
||||
races with concurrent moves/deletes.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
|
||||
@@ -47,6 +47,10 @@ class ResultWithAffectedBoards(BaseModel):
|
||||
|
||||
class DeleteImagesResult(ResultWithAffectedBoards):
|
||||
deleted_images: list[str] = Field(description="The names of the images that were deleted")
|
||||
failed_images: list[str] = Field(
|
||||
default_factory=list,
|
||||
description="The names of authorized images that could not be deleted",
|
||||
)
|
||||
|
||||
|
||||
class StarredImagesResult(ResultWithAffectedBoards):
|
||||
|
||||
@@ -291,22 +291,51 @@ class ImageService(ImageServiceABC):
|
||||
self.__invoker.services.logger.error("Problem deleting image record and file")
|
||||
raise e
|
||||
|
||||
def delete_images_on_board(self, board_id: str):
|
||||
def delete_images_on_board(self, board_id: str, user_id: Optional[str] = None) -> tuple[list[str], list[str]]:
|
||||
try:
|
||||
# When ``user_id`` is set the lookup filters to images owned by that user so the
|
||||
# cascade doesn't destroy other users' contributions to a public/shared board.
|
||||
image_names = self.__invoker.services.board_image_records.get_all_board_image_names_for_board(
|
||||
board_id,
|
||||
categories=None,
|
||||
is_intermediate=None,
|
||||
user_id=user_id,
|
||||
)
|
||||
deleted_image_names: list[str] = []
|
||||
failed_image_names: list[str] = []
|
||||
staged_deletes: list[tuple[str, object]] = []
|
||||
for image_name in image_names:
|
||||
try:
|
||||
record = self.__invoker.services.image_records.get(image_name)
|
||||
self.__invoker.services.image_files.delete(image_name, image_subfolder=record.image_subfolder)
|
||||
except Exception:
|
||||
pass
|
||||
self.__invoker.services.image_records.delete_many(image_names)
|
||||
for image_name in image_names:
|
||||
token = self.__invoker.services.image_files.stage_delete(
|
||||
image_name, image_subfolder=record.image_subfolder
|
||||
)
|
||||
staged_deletes.append((image_name, token))
|
||||
deleted_image_names.append(image_name)
|
||||
except Exception as e:
|
||||
failed_image_names.append(image_name)
|
||||
self.__invoker.services.logger.error(
|
||||
f"Failed to delete image file {image_name}; keeping record: {str(e)}"
|
||||
)
|
||||
try:
|
||||
self.__invoker.services.image_records.delete_many(deleted_image_names)
|
||||
except Exception:
|
||||
for image_name, token in staged_deletes:
|
||||
try:
|
||||
self.__invoker.services.image_files.rollback_delete(token)
|
||||
except Exception as rollback_error:
|
||||
self.__invoker.services.logger.error(
|
||||
f"Failed to restore staged image files for {image_name}: {rollback_error}"
|
||||
)
|
||||
raise
|
||||
for _, token in staged_deletes:
|
||||
try:
|
||||
self.__invoker.services.image_files.commit_delete(token)
|
||||
except Exception as cleanup_error:
|
||||
self.__invoker.services.logger.error(f"Failed to purge staged image files: {cleanup_error}")
|
||||
for image_name in deleted_image_names:
|
||||
self._on_deleted(image_name)
|
||||
return deleted_image_names, failed_image_names
|
||||
except ImageRecordDeleteException:
|
||||
self.__invoker.services.logger.error("Failed to delete image records")
|
||||
raise
|
||||
|
||||
@@ -37,6 +37,7 @@ class MemoryInvocationCache(InvocationCacheBase):
|
||||
if self._max_cache_size == 0:
|
||||
return
|
||||
self._invoker.services.images.on_deleted(self._delete_by_match)
|
||||
self._invoker.services.videos.on_deleted(self._delete_by_match)
|
||||
self._invoker.services.tensors.on_deleted(self._delete_by_match)
|
||||
self._invoker.services.conditioning.on_deleted(self._delete_by_match)
|
||||
|
||||
|
||||
@@ -15,6 +15,7 @@ if TYPE_CHECKING:
|
||||
from invokeai.app.services.board_image_records.board_image_records_base import BoardImageRecordStorageBase
|
||||
from invokeai.app.services.board_images.board_images_base import BoardImagesServiceABC
|
||||
from invokeai.app.services.board_records.board_records_base import BoardRecordStorageBase
|
||||
from invokeai.app.services.board_video_records.board_video_records_base import BoardVideoRecordStorageBase
|
||||
from invokeai.app.services.boards.boards_base import BoardServiceABC
|
||||
from invokeai.app.services.bulk_download.bulk_download_base import BulkDownloadBase
|
||||
from invokeai.app.services.client_state_persistence.client_state_persistence_base import ClientStatePersistenceABC
|
||||
@@ -22,6 +23,7 @@ if TYPE_CHECKING:
|
||||
from invokeai.app.services.download import DownloadQueueServiceBase
|
||||
from invokeai.app.services.events.events_base import EventServiceBase
|
||||
from invokeai.app.services.external_generation.external_generation_base import ExternalGenerationServiceBase
|
||||
from invokeai.app.services.gallery.gallery_base import GalleryServiceABC
|
||||
from invokeai.app.services.image_files.image_files_base import ImageFileStorageBase
|
||||
from invokeai.app.services.image_moves.image_moves_default import ImageMoveService
|
||||
from invokeai.app.services.image_records.image_records_base import ImageRecordStorageBase
|
||||
@@ -39,6 +41,9 @@ if TYPE_CHECKING:
|
||||
from invokeai.app.services.session_queue.session_queue_base import SessionQueueBase
|
||||
from invokeai.app.services.urls.urls_base import UrlServiceBase
|
||||
from invokeai.app.services.users.users_base import UserServiceBase
|
||||
from invokeai.app.services.video_files.video_files_base import VideoFileStorageBase
|
||||
from invokeai.app.services.video_records.video_records_base import VideoRecordStorageBase
|
||||
from invokeai.app.services.videos.videos_base import VideoServiceABC
|
||||
from invokeai.app.services.workflow_records.workflow_records_base import WorkflowRecordsStorageBase
|
||||
from invokeai.app.services.workflow_thumbnails.workflow_thumbnails_base import WorkflowThumbnailServiceBase
|
||||
from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ConditioningFieldData
|
||||
@@ -80,6 +85,11 @@ class InvocationServices:
|
||||
workflow_thumbnails: "WorkflowThumbnailServiceBase",
|
||||
client_state_persistence: "ClientStatePersistenceABC",
|
||||
users: "UserServiceBase",
|
||||
videos: "VideoServiceABC",
|
||||
video_files: "VideoFileStorageBase",
|
||||
video_records: "VideoRecordStorageBase",
|
||||
board_video_records: "BoardVideoRecordStorageBase",
|
||||
gallery: "GalleryServiceABC",
|
||||
image_moves: "ImageMoveService | None" = None,
|
||||
):
|
||||
self.board_images = board_images
|
||||
@@ -114,3 +124,8 @@ class InvocationServices:
|
||||
self.workflow_thumbnails = workflow_thumbnails
|
||||
self.client_state_persistence = client_state_persistence
|
||||
self.users = users
|
||||
self.videos = videos
|
||||
self.video_files = video_files
|
||||
self.video_records = video_records
|
||||
self.board_video_records = board_video_records
|
||||
self.gallery = gallery
|
||||
|
||||
@@ -33,6 +33,8 @@ from invokeai.backend.model_manager.taxonomy import (
|
||||
Qwen3VariantType,
|
||||
QwenImageVariantType,
|
||||
SchedulerPredictionType,
|
||||
WanLoRAVariantType,
|
||||
WanVariantType,
|
||||
ZImageVariantType,
|
||||
)
|
||||
|
||||
@@ -134,6 +136,8 @@ class ModelRecordChanges(BaseModelExcludeNull):
|
||||
| Flux2VariantType
|
||||
| ZImageVariantType
|
||||
| QwenImageVariantType
|
||||
| WanVariantType
|
||||
| WanLoRAVariantType
|
||||
| Qwen3VariantType
|
||||
] = Field(description="The variant of the model.", default=None)
|
||||
prediction_type: Optional[SchedulerPredictionType] = Field(
|
||||
|
||||
@@ -9,3 +9,8 @@ class NameServiceBase(ABC):
|
||||
def create_image_name(self) -> str:
|
||||
"""Creates a name for an image."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def create_video_name(self) -> str:
|
||||
"""Creates a name for a video."""
|
||||
pass
|
||||
|
||||
@@ -10,3 +10,8 @@ class SimpleNameService(NameServiceBase):
|
||||
uuid_str = uuid_string()
|
||||
filename = f"{uuid_str}.png"
|
||||
return filename
|
||||
|
||||
def create_video_name(self) -> str:
|
||||
uuid_str = uuid_string()
|
||||
filename = f"{uuid_str}.mp4"
|
||||
return filename
|
||||
|
||||
@@ -15,7 +15,7 @@ from pydantic import (
|
||||
)
|
||||
from pydantic_core import to_jsonable_python
|
||||
|
||||
from invokeai.app.invocations.fields import ImageField
|
||||
from invokeai.app.invocations.fields import ImageField, VideoField
|
||||
from invokeai.app.services.shared.graph import Graph, GraphExecutionState, NodeNotFoundError
|
||||
from invokeai.app.services.workflow_records.workflow_records_common import (
|
||||
WorkflowWithoutID,
|
||||
@@ -51,7 +51,7 @@ class SessionQueueItemNotFoundError(ValueError):
|
||||
|
||||
# region Batch
|
||||
|
||||
BatchScalarDataType = Union[StrictStr, float, int, ImageField]
|
||||
BatchScalarDataType = Union[StrictStr, float, int, ImageField, VideoField]
|
||||
BatchDataType = Union[BatchScalarDataType, list[BatchScalarDataType]]
|
||||
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ from torch import Tensor
|
||||
|
||||
from invokeai.app.invocations.constants import IMAGE_MODES
|
||||
from invokeai.app.invocations.fields import MetadataField, WithBoard, WithMetadata
|
||||
from invokeai.app.services.board_records.board_records_common import BoardRecordOrderBy
|
||||
from invokeai.app.services.board_records.board_records_common import BoardRecordOrderBy, BoardVisibility
|
||||
from invokeai.app.services.boards.boards_common import BoardDTO
|
||||
from invokeai.app.services.config.config_default import InvokeAIAppConfig
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
|
||||
@@ -18,6 +18,7 @@ from invokeai.app.services.invocation_services import InvocationServices
|
||||
from invokeai.app.services.model_records.model_records_base import UnknownModelException
|
||||
from invokeai.app.services.session_processor.session_processor_common import ProgressImage
|
||||
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
|
||||
from invokeai.app.services.videos.videos_common import VideoDTO
|
||||
from invokeai.app.util.step_callback import diffusion_step_callback
|
||||
from invokeai.backend.model_manager.configs.base import Config_Base
|
||||
from invokeai.backend.model_manager.configs.factory import AnyModelConfig
|
||||
@@ -169,6 +170,22 @@ class ImagesInterface(InvocationContextInterface):
|
||||
super().__init__(services, data)
|
||||
self._util = util
|
||||
|
||||
def _assert_read_access(self, image_name: str) -> None:
|
||||
if not self._services.configuration.multiuser:
|
||||
return
|
||||
user_id = self._data.queue_item.user_id
|
||||
user = self._services.users.get(user_id)
|
||||
if user is None:
|
||||
raise PermissionError("Queue user is not authorized to access this image")
|
||||
if user.is_admin or self._services.image_records.get_user_id(image_name) == user_id:
|
||||
return
|
||||
board_id = self._services.board_image_records.get_board_for_image(image_name)
|
||||
if board_id is not None:
|
||||
board = self._services.boards.get_dto(board_id)
|
||||
if board.board_visibility in (BoardVisibility.Shared, BoardVisibility.Public):
|
||||
return
|
||||
raise PermissionError("Queue user is not authorized to access this image")
|
||||
|
||||
def save(
|
||||
self,
|
||||
image: Image,
|
||||
@@ -211,6 +228,16 @@ class ImagesInterface(InvocationContextInterface):
|
||||
elif isinstance(self._data.invocation, WithBoard) and self._data.invocation.board:
|
||||
board_id_ = self._data.invocation.board.board_id
|
||||
|
||||
if board_id_ is not None and self._services.configuration.multiuser:
|
||||
board = self._services.boards.get_dto(board_id_)
|
||||
user = self._services.users.get(self._data.queue_item.user_id)
|
||||
if user is None or (
|
||||
not user.is_admin
|
||||
and board.user_id != self._data.queue_item.user_id
|
||||
and board.board_visibility != BoardVisibility.Public
|
||||
):
|
||||
raise PermissionError("Queue user is not authorized to save images to this board")
|
||||
|
||||
workflow_ = None
|
||||
if self._data.queue_item.workflow:
|
||||
workflow_ = self._data.queue_item.workflow.model_dump_json()
|
||||
@@ -243,6 +270,7 @@ class ImagesInterface(InvocationContextInterface):
|
||||
Returns:
|
||||
The image as a PIL Image object.
|
||||
"""
|
||||
self._assert_read_access(image_name)
|
||||
image = self._services.images.get_pil_image(image_name)
|
||||
if mode and mode != image.mode:
|
||||
try:
|
||||
@@ -266,6 +294,7 @@ class ImagesInterface(InvocationContextInterface):
|
||||
Returns:
|
||||
The image's metadata, if it has any.
|
||||
"""
|
||||
self._assert_read_access(image_name)
|
||||
return self._services.images.get_metadata(image_name)
|
||||
|
||||
def get_dto(self, image_name: str) -> ImageDTO:
|
||||
@@ -277,6 +306,7 @@ class ImagesInterface(InvocationContextInterface):
|
||||
Returns:
|
||||
The image as an ImageDTO object.
|
||||
"""
|
||||
self._assert_read_access(image_name)
|
||||
return self._services.images.get_dto(image_name)
|
||||
|
||||
def get_path(self, image_name: str, thumbnail: bool = False) -> Path:
|
||||
@@ -289,9 +319,121 @@ class ImagesInterface(InvocationContextInterface):
|
||||
Returns:
|
||||
The local path of the image or thumbnail.
|
||||
"""
|
||||
self._assert_read_access(image_name)
|
||||
return Path(self._services.images.get_path(image_name, thumbnail))
|
||||
|
||||
|
||||
class VideosInterface(InvocationContextInterface):
|
||||
"""Save and look up videos produced by invocations.
|
||||
|
||||
Mirrors :class:`ImagesInterface` but consumes a path to an already-encoded
|
||||
MP4 (or other supported container) rather than an in-memory PIL image —
|
||||
video encoding is the caller's responsibility (e.g. the
|
||||
``wan_latents_to_video`` node uses ``imageio[ffmpeg]``).
|
||||
"""
|
||||
|
||||
def __init__(self, services: InvocationServices, data: InvocationContextData, util: "UtilInterface") -> None:
|
||||
super().__init__(services, data)
|
||||
self._util = util
|
||||
|
||||
def _assert_read_access(self, video_name: str) -> None:
|
||||
if not self._services.configuration.multiuser:
|
||||
return
|
||||
user_id = self._data.queue_item.user_id
|
||||
user = self._services.users.get(user_id)
|
||||
if user is None:
|
||||
raise PermissionError("Queue user is not authorized to access this video")
|
||||
if user.is_admin or self._services.video_records.get_user_id(video_name) == user_id:
|
||||
return
|
||||
board_id = self._services.board_video_records.get_board_for_video(video_name)
|
||||
if board_id is not None:
|
||||
board = self._services.boards.get_dto(board_id)
|
||||
if board.board_visibility in (BoardVisibility.Shared, BoardVisibility.Public):
|
||||
return
|
||||
raise PermissionError("Queue user is not authorized to access this video")
|
||||
|
||||
def save(
|
||||
self,
|
||||
source_path: Path,
|
||||
width: int,
|
||||
height: int,
|
||||
duration: float,
|
||||
fps: Optional[float] = None,
|
||||
board_id: Optional[str] = None,
|
||||
image_category: ImageCategory = ImageCategory.GENERAL,
|
||||
metadata: Optional[MetadataField] = None,
|
||||
) -> VideoDTO:
|
||||
"""Save a video produced by an invocation. The file at ``source_path`` is moved into
|
||||
the videos output folder; the caller should treat the path as consumed after this
|
||||
returns.
|
||||
|
||||
``board_id`` falls back to the invocation's :class:`WithBoard` mixin if unset, and
|
||||
``metadata`` falls back to the :class:`WithMetadata` mixin. Both can be overridden
|
||||
explicitly. ``image_category`` reuses the image enum since the gallery's category
|
||||
filter is shared between kinds.
|
||||
"""
|
||||
|
||||
self._util.signal_progress("Saving video")
|
||||
|
||||
metadata_ = None
|
||||
if metadata:
|
||||
metadata_ = metadata.model_dump_json()
|
||||
elif isinstance(self._data.invocation, WithMetadata) and self._data.invocation.metadata:
|
||||
metadata_ = self._data.invocation.metadata.model_dump_json()
|
||||
|
||||
board_id_ = None
|
||||
if board_id:
|
||||
board_id_ = board_id
|
||||
elif isinstance(self._data.invocation, WithBoard) and self._data.invocation.board:
|
||||
board_id_ = self._data.invocation.board.board_id
|
||||
|
||||
if board_id_ is not None and self._services.configuration.multiuser:
|
||||
board = self._services.boards.get_dto(board_id_)
|
||||
user = self._services.users.get(self._data.queue_item.user_id)
|
||||
if user is None or (
|
||||
not user.is_admin
|
||||
and board.user_id != self._data.queue_item.user_id
|
||||
and board.board_visibility != BoardVisibility.Public
|
||||
):
|
||||
raise PermissionError("Queue user is not authorized to save videos to this board")
|
||||
|
||||
workflow_ = None
|
||||
if self._data.queue_item.workflow:
|
||||
workflow_ = self._data.queue_item.workflow.model_dump_json()
|
||||
|
||||
graph_ = None
|
||||
if self._data.queue_item.session.graph:
|
||||
graph_ = self._data.queue_item.session.graph.model_dump_json()
|
||||
|
||||
return self._services.videos.create(
|
||||
source_path=source_path,
|
||||
width=width,
|
||||
height=height,
|
||||
duration=duration,
|
||||
fps=fps,
|
||||
is_intermediate=self._data.invocation.is_intermediate,
|
||||
video_category=image_category,
|
||||
board_id=board_id_,
|
||||
metadata=metadata_,
|
||||
video_origin=ResourceOrigin.INTERNAL,
|
||||
workflow=workflow_,
|
||||
graph=graph_,
|
||||
session_id=self._data.queue_item.session_id,
|
||||
node_id=self._data.invocation.id,
|
||||
user_id=self._data.queue_item.user_id,
|
||||
)
|
||||
|
||||
def get_dto(self, video_name: str) -> VideoDTO:
|
||||
"""Get a video DTO by name."""
|
||||
self._assert_read_access(video_name)
|
||||
return self._services.videos.get_dto(video_name)
|
||||
|
||||
def get_path(self, video_name: str, thumbnail: bool = False) -> Path:
|
||||
"""Get the on-disk path to a video file or its WebP thumbnail."""
|
||||
self._assert_read_access(video_name)
|
||||
return Path(self._services.videos.get_path(video_name, thumbnail=thumbnail))
|
||||
|
||||
|
||||
class TensorsInterface(InvocationContextInterface):
|
||||
def save(self, tensor: Tensor) -> str:
|
||||
"""Saves a tensor, returning its name.
|
||||
@@ -738,6 +880,7 @@ class InvocationContext:
|
||||
def __init__(
|
||||
self,
|
||||
images: ImagesInterface,
|
||||
videos: VideosInterface,
|
||||
tensors: TensorsInterface,
|
||||
conditioning: ConditioningInterface,
|
||||
models: ModelsInterface,
|
||||
@@ -750,6 +893,8 @@ class InvocationContext:
|
||||
) -> None:
|
||||
self.images = images
|
||||
"""Methods to save, get and update images and their metadata."""
|
||||
self.videos = videos
|
||||
"""Methods to save and get videos produced by invocations."""
|
||||
self.tensors = tensors
|
||||
"""Methods to save and get tensors, including image, noise, masks, and masked images."""
|
||||
self.conditioning = conditioning
|
||||
@@ -792,10 +937,12 @@ def build_invocation_context(
|
||||
conditioning = ConditioningInterface(services=services, data=data)
|
||||
models = ModelsInterface(services=services, data=data, util=util)
|
||||
images = ImagesInterface(services=services, data=data, util=util)
|
||||
videos = VideosInterface(services=services, data=data, util=util)
|
||||
boards = BoardsInterface(services=services, data=data)
|
||||
|
||||
ctx = InvocationContext(
|
||||
images=images,
|
||||
videos=videos,
|
||||
logger=logger,
|
||||
config=config,
|
||||
tensors=tensors,
|
||||
|
||||
@@ -4,6 +4,11 @@ from pydantic import BaseModel, Field
|
||||
|
||||
GenericBaseModel = TypeVar("GenericBaseModel", bound=BaseModel)
|
||||
|
||||
# Upper bound for offset-paginated `limit` query params. Values flow verbatim into
|
||||
# SQL LIMIT clauses, where SQLite treats a negative limit as unlimited and a huge one
|
||||
# materializes every row into DTOs — routes must clamp with ge=0/le=MAX_PAGE_SIZE.
|
||||
MAX_PAGE_SIZE = 1000
|
||||
|
||||
|
||||
class CursorPaginatedResults(BaseModel, Generic[GenericBaseModel]):
|
||||
"""
|
||||
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
"""Migration: Add `videos` and `board_videos` tables for minimal video support.
|
||||
|
||||
The `videos` table parallels `images` but with extra `duration` and `fps` columns.
|
||||
The `board_videos` table parallels `board_images`, providing one-to-many board↔video association.
|
||||
Foreign-key cascades from `boards` mirror the image side, so deleting a board also removes its videos' associations.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
|
||||
from invokeai.app.services.shared.sqlite_migrator.sqlite_migrator_common import Migration
|
||||
|
||||
|
||||
class AddVideosTablesCallback:
|
||||
def __call__(self, cursor: sqlite3.Cursor) -> None:
|
||||
self._create_videos(cursor)
|
||||
self._create_board_videos(cursor)
|
||||
|
||||
def _create_videos(self, cursor: sqlite3.Cursor) -> None:
|
||||
tables = [
|
||||
"""--sql
|
||||
CREATE TABLE IF NOT EXISTS videos (
|
||||
video_name TEXT NOT NULL PRIMARY KEY,
|
||||
video_origin TEXT NOT NULL,
|
||||
video_category TEXT NOT NULL,
|
||||
width INTEGER NOT NULL,
|
||||
height INTEGER NOT NULL,
|
||||
duration REAL NOT NULL DEFAULT 0.0,
|
||||
fps REAL,
|
||||
session_id TEXT,
|
||||
node_id TEXT,
|
||||
metadata TEXT,
|
||||
is_intermediate BOOLEAN DEFAULT FALSE,
|
||||
starred BOOLEAN DEFAULT FALSE,
|
||||
has_workflow BOOLEAN DEFAULT FALSE,
|
||||
-- Deliberately no FK to users(user_id), matching images/boards/workflows
|
||||
-- (migration_27 adds those user_id columns with an index only): deleting a
|
||||
-- user leaves their media in place for admin review/cleanup rather than
|
||||
-- cascading a row delete that would strand the files on disk.
|
||||
user_id TEXT NOT NULL DEFAULT 'system',
|
||||
video_subfolder TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME NOT NULL DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
|
||||
-- Updated via trigger
|
||||
updated_at DATETIME NOT NULL DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
|
||||
-- Soft delete, currently unused
|
||||
deleted_at DATETIME
|
||||
);
|
||||
"""
|
||||
]
|
||||
|
||||
indices = [
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS idx_videos_video_name ON videos(video_name);",
|
||||
"CREATE INDEX IF NOT EXISTS idx_videos_video_origin ON videos(video_origin);",
|
||||
"CREATE INDEX IF NOT EXISTS idx_videos_video_category ON videos(video_category);",
|
||||
"CREATE INDEX IF NOT EXISTS idx_videos_created_at ON videos(created_at);",
|
||||
"CREATE INDEX IF NOT EXISTS idx_videos_starred ON videos(starred);",
|
||||
"CREATE INDEX IF NOT EXISTS idx_videos_user_id ON videos(user_id);",
|
||||
]
|
||||
|
||||
triggers = [
|
||||
"""--sql
|
||||
CREATE TRIGGER IF NOT EXISTS tg_videos_updated_at
|
||||
AFTER UPDATE
|
||||
ON videos FOR EACH ROW
|
||||
BEGIN
|
||||
UPDATE videos SET updated_at = STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')
|
||||
WHERE video_name = old.video_name;
|
||||
END;
|
||||
"""
|
||||
]
|
||||
|
||||
for stmt in tables + indices + triggers:
|
||||
cursor.execute(stmt)
|
||||
|
||||
def _create_board_videos(self, cursor: sqlite3.Cursor) -> None:
|
||||
tables = [
|
||||
"""--sql
|
||||
CREATE TABLE IF NOT EXISTS board_videos (
|
||||
board_id TEXT NOT NULL,
|
||||
video_name TEXT NOT NULL,
|
||||
created_at DATETIME NOT NULL DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
|
||||
-- updated via trigger
|
||||
updated_at DATETIME NOT NULL DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
|
||||
-- Soft delete, currently unused
|
||||
deleted_at DATETIME,
|
||||
-- enforce one-to-many board↔video using PK on video_name
|
||||
PRIMARY KEY (video_name),
|
||||
FOREIGN KEY (board_id) REFERENCES boards (board_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (video_name) REFERENCES videos (video_name) ON DELETE CASCADE
|
||||
);
|
||||
"""
|
||||
]
|
||||
|
||||
indices = [
|
||||
"CREATE INDEX IF NOT EXISTS idx_board_videos_board_id ON board_videos (board_id);",
|
||||
"CREATE INDEX IF NOT EXISTS idx_board_videos_board_id_created_at ON board_videos (board_id, created_at);",
|
||||
]
|
||||
|
||||
triggers = [
|
||||
"""--sql
|
||||
CREATE TRIGGER IF NOT EXISTS tg_board_videos_updated_at
|
||||
AFTER UPDATE
|
||||
ON board_videos FOR EACH ROW
|
||||
BEGIN
|
||||
UPDATE board_videos SET updated_at = STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')
|
||||
WHERE board_id = new.board_id AND video_name = old.video_name;
|
||||
END;
|
||||
"""
|
||||
]
|
||||
|
||||
for stmt in tables + indices + triggers:
|
||||
cursor.execute(stmt)
|
||||
|
||||
|
||||
def build_migration() -> Migration:
|
||||
"""Builds the migration that adds the videos and board_videos tables.
|
||||
|
||||
Depends on migration_27, which last reshaped the boards table and created the 'system' user
|
||||
that videos.user_id defaults to.
|
||||
"""
|
||||
return Migration(
|
||||
id="2026_07_01_add_videos_tables",
|
||||
depends_on="migration_27",
|
||||
callback=AddVideosTablesCallback(),
|
||||
)
|
||||
@@ -9,6 +9,11 @@ class UrlServiceBase(ABC):
|
||||
"""Gets the URL for an image or thumbnail."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_video_url(self, video_name: str, thumbnail: bool = False) -> str:
|
||||
"""Gets the URL for a video or its first-frame thumbnail."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_model_image_url(self, model_key: str) -> str:
|
||||
"""Gets the URL for a model image"""
|
||||
|
||||
@@ -17,6 +17,15 @@ class LocalUrlService(UrlServiceBase):
|
||||
|
||||
return f"{self._base_url}/images/i/{image_basename}/full"
|
||||
|
||||
def get_video_url(self, video_name: str, thumbnail: bool = False) -> str:
|
||||
video_basename = os.path.basename(video_name)
|
||||
|
||||
# These paths are determined by the routes in invokeai/app/api/routers/videos.py
|
||||
if thumbnail:
|
||||
return f"{self._base_url}/videos/i/{video_basename}/thumbnail"
|
||||
|
||||
return f"{self._base_url}/videos/i/{video_basename}/full"
|
||||
|
||||
def get_model_image_url(self, model_key: str) -> str:
|
||||
return f"{self._base_url_v2}/models/i/{model_key}/image"
|
||||
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
from abc import ABC, abstractmethod
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from PIL import Image
|
||||
|
||||
|
||||
class VideoFileStorageBase(ABC):
|
||||
"""Low-level service responsible for storing and retrieving video files."""
|
||||
|
||||
@abstractmethod
|
||||
def get_path(self, video_name: str, thumbnail: bool = False, video_subfolder: str = "") -> Path:
|
||||
"""Gets the internal path to a video or its thumbnail."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def save(
|
||||
self,
|
||||
source_path: Path,
|
||||
video_name: str,
|
||||
thumbnail_size: int = 256,
|
||||
video_subfolder: str = "",
|
||||
metadata: Optional[str] = None,
|
||||
workflow: Optional[str] = None,
|
||||
graph: Optional[str] = None,
|
||||
first_frame: Optional[Image.Image] = None,
|
||||
) -> None:
|
||||
"""Saves a video by moving/copying the file at `source_path` into storage, then writes a sibling
|
||||
WEBP thumbnail extracted from the first frame, plus an optional sidecar JSON of metadata/workflow/graph.
|
||||
A caller that already decoded frame 0 can pass it as `first_frame` to skip the extraction.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete(self, video_name: str, video_subfolder: str = "") -> None:
|
||||
"""Deletes a video file and its thumbnail (if one exists)."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def stage_delete(self, video_name: str, video_subfolder: str = "") -> object:
|
||||
"""Moves a video's files out of service and returns a rollback token."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def commit_delete(self, token: object) -> None:
|
||||
"""Permanently removes files represented by a staged-delete token."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def rollback_delete(self, token: object) -> None:
|
||||
"""Restores files represented by a staged-delete token."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_workflow(self, video_name: str, video_subfolder: str = "") -> Optional[str]:
|
||||
"""Gets the workflow JSON sidecar of a video, if any."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_graph(self, video_name: str, video_subfolder: str = "") -> Optional[str]:
|
||||
"""Gets the graph JSON sidecar of a video, if any."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def validate_path(self, path: str) -> bool:
|
||||
"""Validates the path given for a video or thumbnail."""
|
||||
pass
|
||||
@@ -0,0 +1,19 @@
|
||||
class VideoFileNotFoundException(Exception):
|
||||
"""Raised when a video file is not found in storage."""
|
||||
|
||||
def __init__(self, message="Video file not found"):
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class VideoFileSaveException(Exception):
|
||||
"""Raised when a video file cannot be saved."""
|
||||
|
||||
def __init__(self, message="Video file not saved"):
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class VideoFileDeleteException(Exception):
|
||||
"""Raised when a video file cannot be deleted."""
|
||||
|
||||
def __init__(self, message="Video file not deleted"):
|
||||
super().__init__(message)
|
||||
@@ -0,0 +1,281 @@
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional, Union
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from invokeai.app.services.invoker import Invoker
|
||||
from invokeai.app.services.video_files.video_files_base import VideoFileStorageBase
|
||||
from invokeai.app.services.video_files.video_files_common import (
|
||||
VideoFileDeleteException,
|
||||
VideoFileNotFoundException,
|
||||
VideoFileSaveException,
|
||||
)
|
||||
from invokeai.app.util.thumbnails import make_thumbnail
|
||||
from invokeai.app.util.video_thumbnails import extract_video_frame, get_video_thumbnail_name
|
||||
from invokeai.backend.util.logging import InvokeAILogger
|
||||
|
||||
|
||||
@dataclass
|
||||
class _StagedDelete:
|
||||
directory: Path
|
||||
files: list[tuple[Path, Path]]
|
||||
|
||||
|
||||
class DiskVideoFileStorage(VideoFileStorageBase):
|
||||
"""Stores video files on disk under {outputs}/videos/, with first-frame WebP thumbnails under
|
||||
{outputs}/videos/thumbnails/ and optional JSON sidecars for metadata/workflow/graph under
|
||||
{outputs}/videos/sidecars/."""
|
||||
|
||||
def __init__(self, output_folder: Union[str, Path]):
|
||||
self.__output_folder = output_folder if isinstance(output_folder, Path) else Path(output_folder)
|
||||
self.__thumbnails_folder = self.__output_folder / "thumbnails"
|
||||
self.__sidecars_folder = self.__output_folder / "sidecars"
|
||||
self.__validate_storage_folders()
|
||||
|
||||
def start(self, invoker: Invoker) -> None:
|
||||
self.__invoker = invoker
|
||||
self.__recover_staged_deletes()
|
||||
|
||||
def save(
|
||||
self,
|
||||
source_path: Path,
|
||||
video_name: str,
|
||||
thumbnail_size: int = 256,
|
||||
video_subfolder: str = "",
|
||||
metadata: Optional[str] = None,
|
||||
workflow: Optional[str] = None,
|
||||
graph: Optional[str] = None,
|
||||
first_frame: Optional[Image.Image] = None,
|
||||
) -> None:
|
||||
logger = InvokeAILogger.get_logger()
|
||||
try:
|
||||
self.__validate_storage_folders()
|
||||
video_path = self.get_path(video_name, video_subfolder=video_subfolder)
|
||||
video_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Move if the source is on the same filesystem; otherwise copy then unlink.
|
||||
try:
|
||||
shutil.move(str(source_path), str(video_path))
|
||||
except Exception:
|
||||
shutil.copy2(str(source_path), str(video_path))
|
||||
try:
|
||||
Path(source_path).unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
logger.info(f"Video file written: {video_path}")
|
||||
|
||||
thumbnail_name = get_video_thumbnail_name(video_name)
|
||||
thumbnail_path = self.get_path(thumbnail_name, thumbnail=True, video_subfolder=video_subfolder)
|
||||
thumbnail_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Thumbnail extraction is best-effort — if both imageio and cv2 fail, we still want
|
||||
# the video record + file in place and the invocation to complete. A missing
|
||||
# thumbnail leaves the gallery with a broken-image placeholder for that item, which
|
||||
# is annoying but not fatal. The upload path already decoded frame 0 to prove
|
||||
# decodability and passes it in, saving a decode-worker subprocess per upload.
|
||||
frame = first_frame
|
||||
if frame is None:
|
||||
try:
|
||||
frame = extract_video_frame(video_path, frame_index=0)
|
||||
except Exception as e:
|
||||
logger.warning(f"Thumbnail extraction raised for {video_name}: {e}")
|
||||
frame = None
|
||||
if frame is not None:
|
||||
thumbnail = make_thumbnail(frame, thumbnail_size)
|
||||
thumbnail.save(thumbnail_path, "WEBP")
|
||||
logger.info(f"Thumbnail written: {thumbnail_path}")
|
||||
else:
|
||||
logger.warning(
|
||||
f"Could not extract a thumbnail frame for {video_name}; gallery thumbnail will be missing."
|
||||
)
|
||||
|
||||
if metadata is not None or workflow is not None or graph is not None:
|
||||
sidecar_path = self.__get_sidecar_path(video_name, video_subfolder=video_subfolder)
|
||||
sidecar_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
sidecar = {
|
||||
"invokeai_metadata": metadata,
|
||||
"invokeai_workflow": workflow,
|
||||
"invokeai_graph": graph,
|
||||
}
|
||||
with open(sidecar_path, "w", encoding="utf-8") as f:
|
||||
json.dump(sidecar, f)
|
||||
logger.info(f"Sidecar written: {sidecar_path}")
|
||||
except Exception as e:
|
||||
# By this point the source MP4 has usually already been moved into permanent
|
||||
# storage, so bailing out without cleanup would orphan the video (and any
|
||||
# partially written thumbnail/sidecar) on disk with no DB record through which
|
||||
# it can be managed — the caller rolls the record back on this exception.
|
||||
try:
|
||||
self.delete(video_name, video_subfolder=video_subfolder)
|
||||
except Exception as cleanup_err:
|
||||
logger.error(f"Failed to clean up partially saved files for {video_name}: {cleanup_err}")
|
||||
raise VideoFileSaveException from e
|
||||
|
||||
def delete(self, video_name: str, video_subfolder: str = "") -> None:
|
||||
token = self.stage_delete(video_name, video_subfolder)
|
||||
self.commit_delete(token)
|
||||
|
||||
def stage_delete(self, video_name: str, video_subfolder: str = "") -> _StagedDelete:
|
||||
candidates = [
|
||||
self.get_path(video_name, video_subfolder=video_subfolder),
|
||||
self.get_path(video_name, thumbnail=True, video_subfolder=video_subfolder),
|
||||
self.__get_sidecar_path(video_name, video_subfolder=video_subfolder),
|
||||
]
|
||||
staging_dir = Path(tempfile.mkdtemp(prefix=".delete_", dir=self.__output_folder))
|
||||
staged: list[tuple[Path, Path]] = []
|
||||
try:
|
||||
with open(staging_dir / "manifest.json", "w", encoding="utf-8") as manifest:
|
||||
manifest.write(json.dumps({"video_name": video_name, "video_subfolder": video_subfolder}))
|
||||
manifest.flush()
|
||||
os.fsync(manifest.fileno())
|
||||
for index, source in enumerate(candidates):
|
||||
if source.exists():
|
||||
destination = staging_dir / str(index)
|
||||
source.replace(destination)
|
||||
staged.append((source, destination))
|
||||
return _StagedDelete(directory=staging_dir, files=staged)
|
||||
except Exception as e:
|
||||
for source, destination in reversed(staged):
|
||||
if destination.exists():
|
||||
source.parent.mkdir(parents=True, exist_ok=True)
|
||||
destination.replace(source)
|
||||
shutil.rmtree(staging_dir, ignore_errors=True)
|
||||
raise VideoFileDeleteException from e
|
||||
|
||||
def commit_delete(self, token: object) -> None:
|
||||
if not isinstance(token, _StagedDelete):
|
||||
raise VideoFileDeleteException("Invalid staged-delete token")
|
||||
shutil.rmtree(token.directory)
|
||||
|
||||
def rollback_delete(self, token: object) -> None:
|
||||
if not isinstance(token, _StagedDelete):
|
||||
raise VideoFileDeleteException("Invalid staged-delete token")
|
||||
try:
|
||||
for source, destination in reversed(token.files):
|
||||
if destination.exists():
|
||||
source.parent.mkdir(parents=True, exist_ok=True)
|
||||
destination.replace(source)
|
||||
shutil.rmtree(token.directory, ignore_errors=True)
|
||||
except Exception as e:
|
||||
raise VideoFileDeleteException from e
|
||||
|
||||
def get_path(self, video_name: str, thumbnail: bool = False, video_subfolder: str = "") -> Path:
|
||||
base_folder = self.__thumbnails_folder if thumbnail else self.__output_folder
|
||||
filename = get_video_thumbnail_name(video_name) if thumbnail else video_name
|
||||
|
||||
basename = Path(filename).name
|
||||
if basename != filename:
|
||||
raise ValueError("Invalid video name, potential directory traversal detected")
|
||||
|
||||
if video_subfolder:
|
||||
self._validate_subfolder(video_subfolder)
|
||||
video_path = base_folder / video_subfolder / basename
|
||||
else:
|
||||
video_path = base_folder / basename
|
||||
|
||||
resolved_base = base_folder.resolve()
|
||||
resolved_video_path = video_path.resolve()
|
||||
if not resolved_video_path.is_relative_to(resolved_base):
|
||||
raise ValueError("Video path outside outputs folder, potential directory traversal detected")
|
||||
return resolved_video_path
|
||||
|
||||
def get_workflow(self, video_name: str, video_subfolder: str = "") -> Optional[str]:
|
||||
sidecar = self.__read_sidecar(video_name, video_subfolder)
|
||||
if sidecar is None:
|
||||
return None
|
||||
workflow = sidecar.get("invokeai_workflow")
|
||||
return workflow if isinstance(workflow, str) else None
|
||||
|
||||
def get_graph(self, video_name: str, video_subfolder: str = "") -> Optional[str]:
|
||||
sidecar = self.__read_sidecar(video_name, video_subfolder)
|
||||
if sidecar is None:
|
||||
return None
|
||||
graph = sidecar.get("invokeai_graph")
|
||||
return graph if isinstance(graph, str) else None
|
||||
|
||||
def validate_path(self, path: Union[str, Path]) -> bool:
|
||||
path = path if isinstance(path, Path) else Path(path)
|
||||
return path.exists()
|
||||
|
||||
@staticmethod
|
||||
def _validate_subfolder(subfolder: str) -> None:
|
||||
"""Validates a subfolder path to prevent directory traversal."""
|
||||
if not subfolder:
|
||||
return
|
||||
if "\\" in subfolder:
|
||||
raise ValueError("Backslashes not allowed in subfolder path")
|
||||
if subfolder.startswith("/"):
|
||||
raise ValueError("Absolute paths not allowed in subfolder path")
|
||||
for part in subfolder.split("/"):
|
||||
if part == "..":
|
||||
raise ValueError("Parent directory references not allowed in subfolder path")
|
||||
if part == "":
|
||||
raise ValueError("Empty path segments not allowed in subfolder path")
|
||||
|
||||
def __get_sidecar_path(self, video_name: str, video_subfolder: str = "") -> Path:
|
||||
sidecar_name = Path(video_name).stem + ".json"
|
||||
if video_subfolder:
|
||||
self._validate_subfolder(video_subfolder)
|
||||
sidecar_path = self.__sidecars_folder / video_subfolder / sidecar_name
|
||||
else:
|
||||
sidecar_path = self.__sidecars_folder / sidecar_name
|
||||
resolved_base = self.__sidecars_folder.resolve()
|
||||
resolved_sidecar_path = sidecar_path.resolve()
|
||||
if not resolved_sidecar_path.is_relative_to(resolved_base):
|
||||
raise ValueError("Sidecar path outside outputs folder, potential directory traversal detected")
|
||||
return resolved_sidecar_path
|
||||
|
||||
def __read_sidecar(self, video_name: str, video_subfolder: str = "") -> Optional[dict]:
|
||||
path = self.__get_sidecar_path(video_name, video_subfolder=video_subfolder)
|
||||
if not path.exists():
|
||||
return None
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except Exception as e:
|
||||
raise VideoFileNotFoundException from e
|
||||
|
||||
def __validate_storage_folders(self) -> None:
|
||||
for folder in (self.__output_folder, self.__thumbnails_folder, self.__sidecars_folder):
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def __recover_staged_deletes(self) -> None:
|
||||
logger = InvokeAILogger.get_logger()
|
||||
for staging_dir in self.__output_folder.glob(".delete_*"):
|
||||
manifest_path = staging_dir / "manifest.json"
|
||||
if not manifest_path.is_file():
|
||||
if not any(staging_dir.iterdir()):
|
||||
staging_dir.rmdir()
|
||||
continue
|
||||
try:
|
||||
with open(manifest_path, encoding="utf-8") as manifest:
|
||||
data = json.load(manifest)
|
||||
video_name = data["video_name"]
|
||||
video_subfolder = data.get("video_subfolder", "")
|
||||
candidates = [
|
||||
self.get_path(video_name, video_subfolder=video_subfolder),
|
||||
self.get_path(video_name, thumbnail=True, video_subfolder=video_subfolder),
|
||||
self.__get_sidecar_path(video_name, video_subfolder=video_subfolder),
|
||||
]
|
||||
token = _StagedDelete(
|
||||
directory=staging_dir,
|
||||
files=[(source, staging_dir / str(index)) for index, source in enumerate(candidates)],
|
||||
)
|
||||
# get() raises VideoRecordNotFoundException when the record is gone —
|
||||
# handled below by purging the staging dir (the delete had committed in
|
||||
# the DB, so finish it). A surviving record means the delete never
|
||||
# committed: restore the files.
|
||||
self.__invoker.services.video_records.get(video_name)
|
||||
self.rollback_delete(token)
|
||||
except Exception as error:
|
||||
from invokeai.app.services.video_records.video_records_common import VideoRecordNotFoundException
|
||||
|
||||
if isinstance(error, VideoRecordNotFoundException):
|
||||
shutil.rmtree(staging_dir, ignore_errors=True)
|
||||
else:
|
||||
logger.error(f"Failed to recover staged video deletion {staging_dir}: {error}")
|
||||
@@ -0,0 +1,108 @@
|
||||
from abc import ABC, abstractmethod
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
from invokeai.app.invocations.fields import MetadataField
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
|
||||
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
|
||||
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
|
||||
from invokeai.app.services.video_records.video_records_common import (
|
||||
VideoNamesResult,
|
||||
VideoRecord,
|
||||
VideoRecordChanges,
|
||||
)
|
||||
|
||||
|
||||
class VideoRecordStorageBase(ABC):
|
||||
"""Low-level service responsible for interfacing with the video record store."""
|
||||
|
||||
@abstractmethod
|
||||
def get(self, video_name: str) -> VideoRecord:
|
||||
"""Gets a video record."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_metadata(self, video_name: str) -> Optional[MetadataField]:
|
||||
"""Gets a video's metadata."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def update(self, video_name: str, changes: VideoRecordChanges) -> None:
|
||||
"""Updates a video record."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_many(
|
||||
self,
|
||||
offset: int = 0,
|
||||
limit: int = 10,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
video_origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> OffsetPaginatedResults[VideoRecord]:
|
||||
"""Gets a page of video records."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete(self, video_name: str) -> None:
|
||||
"""Deletes a video record."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete_many(self, video_names: list[str]) -> None:
|
||||
"""Deletes many video records."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def save(
|
||||
self,
|
||||
video_name: str,
|
||||
video_origin: ResourceOrigin,
|
||||
video_category: ImageCategory,
|
||||
width: int,
|
||||
height: int,
|
||||
duration: float,
|
||||
fps: Optional[float],
|
||||
has_workflow: bool,
|
||||
is_intermediate: Optional[bool] = False,
|
||||
starred: Optional[bool] = False,
|
||||
session_id: Optional[str] = None,
|
||||
node_id: Optional[str] = None,
|
||||
metadata: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
video_subfolder: str = "",
|
||||
) -> datetime:
|
||||
"""Saves a video record."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_user_id(self, video_name: str) -> Optional[str]:
|
||||
"""Gets the user_id of the video owner. Returns None if video not found."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_most_recent_video_for_board(self, board_id: str) -> Optional[VideoRecord]:
|
||||
"""Gets the most recent non-intermediate video on a board, preferring starred videos."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_video_names(
|
||||
self,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
video_origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> VideoNamesResult:
|
||||
"""Gets ordered list of video names with metadata for optimistic updates."""
|
||||
pass
|
||||
@@ -0,0 +1,137 @@
|
||||
import datetime
|
||||
from typing import Optional, Union
|
||||
|
||||
from pydantic import BaseModel, Field, StrictBool, StrictStr
|
||||
|
||||
from invokeai.app.services.image_records.image_records_common import (
|
||||
ImageCategory,
|
||||
ResourceOrigin,
|
||||
)
|
||||
from invokeai.app.util.misc import get_iso_timestamp
|
||||
from invokeai.app.util.model_exclude_null import BaseModelExcludeNull
|
||||
|
||||
|
||||
class VideoRecordNotFoundException(Exception):
|
||||
"""Raised when a video record is not found."""
|
||||
|
||||
def __init__(self, message="Video record not found"):
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class VideoRecordSaveException(Exception):
|
||||
"""Raised when a video record cannot be saved."""
|
||||
|
||||
def __init__(self, message="Video record not saved"):
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class VideoRecordDeleteException(Exception):
|
||||
"""Raised when a video record cannot be deleted."""
|
||||
|
||||
def __init__(self, message="Video record not deleted"):
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
VIDEO_DTO_COLS = ", ".join(
|
||||
[
|
||||
"videos." + c
|
||||
for c in [
|
||||
"video_name",
|
||||
"video_origin",
|
||||
"video_category",
|
||||
"width",
|
||||
"height",
|
||||
"duration",
|
||||
"fps",
|
||||
"session_id",
|
||||
"node_id",
|
||||
"has_workflow",
|
||||
"is_intermediate",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
"deleted_at",
|
||||
"starred",
|
||||
"video_subfolder",
|
||||
]
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
class VideoRecord(BaseModelExcludeNull):
|
||||
"""Deserialized video record without metadata."""
|
||||
|
||||
video_name: str = Field(description="The unique name of the video.")
|
||||
video_origin: ResourceOrigin = Field(description="The origin of the video.")
|
||||
video_category: ImageCategory = Field(description="The category of the video (reuses ImageCategory).")
|
||||
width: int = Field(description="The pixel width of the video.")
|
||||
height: int = Field(description="The pixel height of the video.")
|
||||
duration: float = Field(description="The duration of the video in seconds.")
|
||||
fps: Optional[float] = Field(default=None, description="The frames-per-second of the video, if known.")
|
||||
created_at: Union[datetime.datetime, str] = Field(description="The created timestamp of the video.")
|
||||
updated_at: Union[datetime.datetime, str] = Field(description="The updated timestamp of the video.")
|
||||
deleted_at: Optional[Union[datetime.datetime, str]] = Field(
|
||||
default=None, description="The deleted timestamp of the video."
|
||||
)
|
||||
is_intermediate: bool = Field(description="Whether this is an intermediate video.")
|
||||
session_id: Optional[str] = Field(default=None, description="The session ID that produced this video, if any.")
|
||||
node_id: Optional[str] = Field(default=None, description="The node ID that produced this video, if any.")
|
||||
starred: bool = Field(description="Whether this video is starred.")
|
||||
has_workflow: bool = Field(description="Whether this video has a workflow associated.")
|
||||
video_subfolder: str = Field(default="", description="The subfolder where the video is stored on disk.")
|
||||
|
||||
|
||||
class VideoRecordChanges(BaseModelExcludeNull, extra="allow"):
|
||||
"""Allowed mutations on a video record."""
|
||||
|
||||
video_category: Optional[ImageCategory] = Field(default=None, description="The video's new category.")
|
||||
session_id: Optional[StrictStr] = Field(default=None, description="The video's new session ID.")
|
||||
is_intermediate: Optional[StrictBool] = Field(default=None, description="The video's new `is_intermediate` flag.")
|
||||
starred: Optional[StrictBool] = Field(default=None, description="The video's new `starred` state.")
|
||||
|
||||
|
||||
def deserialize_video_record(video_dict: dict) -> VideoRecord:
|
||||
"""Deserializes a video record from a sqlite row dict."""
|
||||
video_name = video_dict.get("video_name", "unknown")
|
||||
video_origin = ResourceOrigin(video_dict.get("video_origin", ResourceOrigin.INTERNAL.value))
|
||||
video_category = ImageCategory(video_dict.get("video_category", ImageCategory.GENERAL.value))
|
||||
width = video_dict.get("width", 0)
|
||||
height = video_dict.get("height", 0)
|
||||
duration = video_dict.get("duration", 0.0)
|
||||
fps_raw = video_dict.get("fps", None)
|
||||
fps = float(fps_raw) if fps_raw is not None else None
|
||||
session_id = video_dict.get("session_id", None)
|
||||
node_id = video_dict.get("node_id", None)
|
||||
created_at = video_dict.get("created_at", get_iso_timestamp())
|
||||
updated_at = video_dict.get("updated_at", get_iso_timestamp())
|
||||
deleted_at = video_dict.get("deleted_at", None)
|
||||
is_intermediate = video_dict.get("is_intermediate", False)
|
||||
starred = video_dict.get("starred", False)
|
||||
has_workflow = video_dict.get("has_workflow", False)
|
||||
video_subfolder = video_dict.get("video_subfolder", "")
|
||||
|
||||
return VideoRecord(
|
||||
video_name=video_name,
|
||||
video_origin=video_origin,
|
||||
video_category=video_category,
|
||||
width=width,
|
||||
height=height,
|
||||
duration=float(duration),
|
||||
fps=fps,
|
||||
session_id=session_id,
|
||||
node_id=node_id,
|
||||
created_at=created_at,
|
||||
updated_at=updated_at,
|
||||
deleted_at=deleted_at,
|
||||
is_intermediate=is_intermediate,
|
||||
starred=starred,
|
||||
has_workflow=has_workflow,
|
||||
video_subfolder=video_subfolder,
|
||||
)
|
||||
|
||||
|
||||
class VideoNamesResult(BaseModel):
|
||||
"""Response containing ordered video names with metadata for optimistic updates."""
|
||||
|
||||
video_names: list[str] = Field(description="Ordered list of video names")
|
||||
starred_count: int = Field(description="Number of starred videos (when starred_first=True)")
|
||||
total_count: int = Field(description="Total number of videos matching the query")
|
||||
@@ -0,0 +1,380 @@
|
||||
import sqlite3
|
||||
from datetime import datetime
|
||||
from typing import Optional, Union, cast
|
||||
|
||||
from invokeai.app.invocations.fields import MetadataField, MetadataFieldValidator
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
|
||||
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
|
||||
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
|
||||
from invokeai.app.services.shared.sqlite.sqlite_database import SqliteDatabase
|
||||
from invokeai.app.services.video_records.video_records_base import VideoRecordStorageBase
|
||||
from invokeai.app.services.video_records.video_records_common import (
|
||||
VIDEO_DTO_COLS,
|
||||
VideoNamesResult,
|
||||
VideoRecord,
|
||||
VideoRecordChanges,
|
||||
VideoRecordDeleteException,
|
||||
VideoRecordNotFoundException,
|
||||
VideoRecordSaveException,
|
||||
deserialize_video_record,
|
||||
)
|
||||
|
||||
|
||||
class SqliteVideoRecordStorage(VideoRecordStorageBase):
|
||||
def __init__(self, db: SqliteDatabase) -> None:
|
||||
super().__init__()
|
||||
self._db = db
|
||||
|
||||
def get(self, video_name: str) -> VideoRecord:
|
||||
with self._db.transaction() as cursor:
|
||||
try:
|
||||
cursor.execute(
|
||||
f"""--sql
|
||||
SELECT {VIDEO_DTO_COLS} FROM videos
|
||||
WHERE video_name = ?;
|
||||
""",
|
||||
(video_name,),
|
||||
)
|
||||
result = cast(Optional[sqlite3.Row], cursor.fetchone())
|
||||
except sqlite3.Error as e:
|
||||
raise VideoRecordNotFoundException from e
|
||||
|
||||
if not result:
|
||||
raise VideoRecordNotFoundException
|
||||
return deserialize_video_record(dict(result))
|
||||
|
||||
def get_user_id(self, video_name: str) -> Optional[str]:
|
||||
with self._db.transaction() as cursor:
|
||||
cursor.execute(
|
||||
"""--sql
|
||||
SELECT user_id FROM videos
|
||||
WHERE video_name = ?;
|
||||
""",
|
||||
(video_name,),
|
||||
)
|
||||
result = cast(Optional[sqlite3.Row], cursor.fetchone())
|
||||
if not result:
|
||||
return None
|
||||
return cast(Optional[str], dict(result).get("user_id"))
|
||||
|
||||
def get_most_recent_video_for_board(self, board_id: str) -> Optional[VideoRecord]:
|
||||
with self._db.transaction() as cursor:
|
||||
cursor.execute(
|
||||
f"""--sql
|
||||
SELECT {VIDEO_DTO_COLS}
|
||||
FROM videos
|
||||
JOIN board_videos ON videos.video_name = board_videos.video_name
|
||||
WHERE board_videos.board_id = ?
|
||||
AND videos.is_intermediate = FALSE
|
||||
ORDER BY videos.starred DESC, videos.created_at DESC, videos.video_name DESC
|
||||
LIMIT 1;
|
||||
""",
|
||||
(board_id,),
|
||||
)
|
||||
result = cast(Optional[sqlite3.Row], cursor.fetchone())
|
||||
if result is None:
|
||||
return None
|
||||
return deserialize_video_record(dict(result))
|
||||
|
||||
def get_metadata(self, video_name: str) -> Optional[MetadataField]:
|
||||
with self._db.transaction() as cursor:
|
||||
try:
|
||||
cursor.execute(
|
||||
"""--sql
|
||||
SELECT metadata FROM videos
|
||||
WHERE video_name = ?;
|
||||
""",
|
||||
(video_name,),
|
||||
)
|
||||
result = cast(Optional[sqlite3.Row], cursor.fetchone())
|
||||
except sqlite3.Error as e:
|
||||
raise VideoRecordNotFoundException from e
|
||||
|
||||
if not result:
|
||||
raise VideoRecordNotFoundException
|
||||
|
||||
as_dict = dict(result)
|
||||
metadata_raw = cast(Optional[str], as_dict.get("metadata", None))
|
||||
return MetadataFieldValidator.validate_json(metadata_raw) if metadata_raw is not None else None
|
||||
|
||||
def update(self, video_name: str, changes: VideoRecordChanges) -> None:
|
||||
with self._db.transaction() as cursor:
|
||||
try:
|
||||
if changes.video_category is not None:
|
||||
cursor.execute(
|
||||
"UPDATE videos SET video_category = ? WHERE video_name = ?;",
|
||||
(changes.video_category.value, video_name),
|
||||
)
|
||||
if changes.session_id is not None:
|
||||
cursor.execute(
|
||||
"UPDATE videos SET session_id = ? WHERE video_name = ?;",
|
||||
(changes.session_id, video_name),
|
||||
)
|
||||
if changes.is_intermediate is not None:
|
||||
cursor.execute(
|
||||
"UPDATE videos SET is_intermediate = ? WHERE video_name = ?;",
|
||||
(changes.is_intermediate, video_name),
|
||||
)
|
||||
if changes.starred is not None:
|
||||
cursor.execute(
|
||||
"UPDATE videos SET starred = ? WHERE video_name = ?;",
|
||||
(changes.starred, video_name),
|
||||
)
|
||||
except sqlite3.Error as e:
|
||||
raise VideoRecordSaveException from e
|
||||
|
||||
def get_many(
|
||||
self,
|
||||
offset: int = 0,
|
||||
limit: int = 10,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
video_origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> OffsetPaginatedResults[VideoRecord]:
|
||||
with self._db.transaction() as cursor:
|
||||
count_query = """--sql
|
||||
SELECT COUNT(*)
|
||||
FROM videos
|
||||
LEFT JOIN board_videos ON board_videos.video_name = videos.video_name
|
||||
WHERE 1=1
|
||||
"""
|
||||
videos_query = f"""--sql
|
||||
SELECT {VIDEO_DTO_COLS}
|
||||
FROM videos
|
||||
LEFT JOIN board_videos ON board_videos.video_name = videos.video_name
|
||||
WHERE 1=1
|
||||
"""
|
||||
|
||||
query_conditions = ""
|
||||
query_params: list[Union[int, str, bool]] = []
|
||||
|
||||
if video_origin is not None:
|
||||
query_conditions += " AND videos.video_origin = ? "
|
||||
query_params.append(video_origin.value)
|
||||
|
||||
if categories is not None:
|
||||
category_strings = [c.value for c in set(categories)]
|
||||
placeholders = ",".join("?" * len(category_strings))
|
||||
query_conditions += f" AND videos.video_category IN ( {placeholders} ) "
|
||||
for c in category_strings:
|
||||
query_params.append(c)
|
||||
|
||||
if is_intermediate is not None:
|
||||
query_conditions += " AND videos.is_intermediate = ? "
|
||||
query_params.append(is_intermediate)
|
||||
|
||||
if board_id == "none":
|
||||
query_conditions += " AND board_videos.board_id IS NULL "
|
||||
if user_id is not None and not is_admin:
|
||||
query_conditions += " AND videos.user_id = ? "
|
||||
query_params.append(user_id)
|
||||
elif board_id is not None:
|
||||
query_conditions += " AND board_videos.board_id = ? "
|
||||
query_params.append(board_id)
|
||||
elif user_id is not None and not is_admin:
|
||||
# No board_id supplied — still enforce per-user isolation so
|
||||
# non-admin callers cannot enumerate other users' videos.
|
||||
query_conditions += " AND videos.user_id = ? "
|
||||
query_params.append(user_id)
|
||||
|
||||
if search_term:
|
||||
query_conditions += " AND (videos.metadata LIKE ? OR videos.created_at LIKE ?) "
|
||||
query_params.append(f"%{search_term.lower()}%")
|
||||
query_params.append(f"%{search_term.lower()}%")
|
||||
|
||||
if starred_first:
|
||||
query_pagination = (
|
||||
f" ORDER BY videos.starred DESC, videos.created_at {order_dir.value}, "
|
||||
f"videos.video_name {order_dir.value} LIMIT ? OFFSET ? "
|
||||
)
|
||||
else:
|
||||
query_pagination = (
|
||||
f" ORDER BY videos.created_at {order_dir.value}, videos.video_name {order_dir.value} "
|
||||
"LIMIT ? OFFSET ? "
|
||||
)
|
||||
|
||||
videos_query += query_conditions + query_pagination + ";"
|
||||
videos_params = query_params.copy()
|
||||
videos_params.extend([limit, offset])
|
||||
cursor.execute(videos_query, videos_params)
|
||||
result = cast(list[sqlite3.Row], cursor.fetchall())
|
||||
videos = [deserialize_video_record(dict(r)) for r in result]
|
||||
|
||||
count_query += query_conditions + ";"
|
||||
cursor.execute(count_query, query_params.copy())
|
||||
count = cast(int, cursor.fetchone()[0])
|
||||
|
||||
return OffsetPaginatedResults(items=videos, offset=offset, limit=limit, total=count)
|
||||
|
||||
def delete(self, video_name: str) -> None:
|
||||
with self._db.transaction() as cursor:
|
||||
try:
|
||||
cursor.execute("DELETE FROM videos WHERE video_name = ?;", (video_name,))
|
||||
except sqlite3.Error as e:
|
||||
raise VideoRecordDeleteException from e
|
||||
|
||||
def delete_many(self, video_names: list[str]) -> None:
|
||||
with self._db.transaction() as cursor:
|
||||
try:
|
||||
placeholders = ",".join("?" for _ in video_names)
|
||||
cursor.execute(f"DELETE FROM videos WHERE video_name IN ({placeholders})", video_names)
|
||||
except sqlite3.Error as e:
|
||||
raise VideoRecordDeleteException from e
|
||||
|
||||
def save(
|
||||
self,
|
||||
video_name: str,
|
||||
video_origin: ResourceOrigin,
|
||||
video_category: ImageCategory,
|
||||
width: int,
|
||||
height: int,
|
||||
duration: float,
|
||||
fps: Optional[float],
|
||||
has_workflow: bool,
|
||||
is_intermediate: Optional[bool] = False,
|
||||
starred: Optional[bool] = False,
|
||||
session_id: Optional[str] = None,
|
||||
node_id: Optional[str] = None,
|
||||
metadata: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
video_subfolder: str = "",
|
||||
) -> datetime:
|
||||
with self._db.transaction() as cursor:
|
||||
try:
|
||||
cursor.execute(
|
||||
"""--sql
|
||||
INSERT OR IGNORE INTO videos (
|
||||
video_name,
|
||||
video_origin,
|
||||
video_category,
|
||||
width,
|
||||
height,
|
||||
duration,
|
||||
fps,
|
||||
node_id,
|
||||
session_id,
|
||||
metadata,
|
||||
is_intermediate,
|
||||
starred,
|
||||
has_workflow,
|
||||
user_id,
|
||||
video_subfolder
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
|
||||
""",
|
||||
(
|
||||
video_name,
|
||||
video_origin.value,
|
||||
video_category.value,
|
||||
width,
|
||||
height,
|
||||
float(duration),
|
||||
float(fps) if fps is not None else None,
|
||||
node_id,
|
||||
session_id,
|
||||
metadata,
|
||||
is_intermediate,
|
||||
starred,
|
||||
has_workflow,
|
||||
user_id or "system",
|
||||
video_subfolder,
|
||||
),
|
||||
)
|
||||
|
||||
cursor.execute(
|
||||
"SELECT created_at FROM videos WHERE video_name = ?;",
|
||||
(video_name,),
|
||||
)
|
||||
created_at = datetime.fromisoformat(cursor.fetchone()[0])
|
||||
except sqlite3.Error as e:
|
||||
raise VideoRecordSaveException from e
|
||||
return created_at
|
||||
|
||||
def get_video_names(
|
||||
self,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
video_origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> VideoNamesResult:
|
||||
with self._db.transaction() as cursor:
|
||||
query_conditions = ""
|
||||
query_params: list[Union[int, str, bool]] = []
|
||||
|
||||
if video_origin is not None:
|
||||
query_conditions += " AND videos.video_origin = ? "
|
||||
query_params.append(video_origin.value)
|
||||
|
||||
if categories is not None:
|
||||
category_strings = [c.value for c in set(categories)]
|
||||
placeholders = ",".join("?" * len(category_strings))
|
||||
query_conditions += f" AND videos.video_category IN ( {placeholders} ) "
|
||||
for c in category_strings:
|
||||
query_params.append(c)
|
||||
|
||||
if is_intermediate is not None:
|
||||
query_conditions += " AND videos.is_intermediate = ? "
|
||||
query_params.append(is_intermediate)
|
||||
|
||||
if board_id == "none":
|
||||
query_conditions += " AND board_videos.board_id IS NULL "
|
||||
if user_id is not None and not is_admin:
|
||||
query_conditions += " AND videos.user_id = ? "
|
||||
query_params.append(user_id)
|
||||
elif board_id is not None:
|
||||
query_conditions += " AND board_videos.board_id = ? "
|
||||
query_params.append(board_id)
|
||||
elif user_id is not None and not is_admin:
|
||||
# No board_id supplied — still enforce per-user isolation so
|
||||
# non-admin callers cannot enumerate other users' videos.
|
||||
query_conditions += " AND videos.user_id = ? "
|
||||
query_params.append(user_id)
|
||||
|
||||
if search_term:
|
||||
query_conditions += " AND (videos.metadata LIKE ? OR videos.created_at LIKE ?) "
|
||||
query_params.append(f"%{search_term.lower()}%")
|
||||
query_params.append(f"%{search_term.lower()}%")
|
||||
|
||||
starred_count = 0
|
||||
if starred_first:
|
||||
cursor.execute(
|
||||
f"""--sql
|
||||
SELECT COUNT(*)
|
||||
FROM videos
|
||||
LEFT JOIN board_videos ON board_videos.video_name = videos.video_name
|
||||
WHERE videos.starred = TRUE AND (1=1{query_conditions})
|
||||
""",
|
||||
query_params,
|
||||
)
|
||||
starred_count = cast(int, cursor.fetchone()[0])
|
||||
|
||||
order_clause = (
|
||||
f" ORDER BY videos.starred DESC, videos.created_at {order_dir.value}, "
|
||||
f"videos.video_name {order_dir.value} "
|
||||
if starred_first
|
||||
else f" ORDER BY videos.created_at {order_dir.value}, videos.video_name {order_dir.value} "
|
||||
)
|
||||
cursor.execute(
|
||||
f"""--sql
|
||||
SELECT videos.video_name
|
||||
FROM videos
|
||||
LEFT JOIN board_videos ON board_videos.video_name = videos.video_name
|
||||
WHERE 1=1{query_conditions}
|
||||
{order_clause}
|
||||
""",
|
||||
query_params,
|
||||
)
|
||||
result = cast(list[sqlite3.Row], cursor.fetchall())
|
||||
video_names = [row[0] for row in result]
|
||||
return VideoNamesResult(video_names=video_names, starred_count=starred_count, total_count=len(video_names))
|
||||
@@ -0,0 +1,166 @@
|
||||
from abc import ABC, abstractmethod
|
||||
from pathlib import Path
|
||||
from typing import Callable, Optional
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from invokeai.app.invocations.fields import MetadataField
|
||||
from invokeai.app.services.image_records.image_records_common import ImageCategory, ResourceOrigin
|
||||
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
|
||||
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
|
||||
from invokeai.app.services.video_records.video_records_common import (
|
||||
VideoNamesResult,
|
||||
VideoRecord,
|
||||
VideoRecordChanges,
|
||||
)
|
||||
from invokeai.app.services.videos.videos_common import VideoDTO
|
||||
|
||||
|
||||
class VideoServiceABC(ABC):
|
||||
"""High-level service for video management."""
|
||||
|
||||
_on_changed_callbacks: list[Callable[[VideoDTO], None]]
|
||||
_on_deleted_callbacks: list[Callable[[str], None]]
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._on_changed_callbacks = []
|
||||
self._on_deleted_callbacks = []
|
||||
|
||||
def on_changed(self, on_changed: Callable[[VideoDTO], None]) -> None:
|
||||
"""Register a callback for when a video is changed."""
|
||||
self._on_changed_callbacks.append(on_changed)
|
||||
|
||||
def on_deleted(self, on_deleted: Callable[[str], None]) -> None:
|
||||
"""Register a callback for when a video is deleted."""
|
||||
self._on_deleted_callbacks.append(on_deleted)
|
||||
|
||||
def _on_changed(self, item: VideoDTO) -> None:
|
||||
for callback in self._on_changed_callbacks:
|
||||
callback(item)
|
||||
|
||||
def _on_deleted(self, item_id: str) -> None:
|
||||
for callback in self._on_deleted_callbacks:
|
||||
callback(item_id)
|
||||
|
||||
@abstractmethod
|
||||
def create(
|
||||
self,
|
||||
source_path: Path,
|
||||
width: int,
|
||||
height: int,
|
||||
duration: float,
|
||||
fps: Optional[float],
|
||||
video_origin: ResourceOrigin,
|
||||
video_category: ImageCategory,
|
||||
node_id: Optional[str] = None,
|
||||
session_id: Optional[str] = None,
|
||||
board_id: Optional[str] = None,
|
||||
is_intermediate: Optional[bool] = False,
|
||||
metadata: Optional[str] = None,
|
||||
workflow: Optional[str] = None,
|
||||
graph: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
first_frame: Optional[Image.Image] = None,
|
||||
) -> VideoDTO:
|
||||
"""Creates a video by moving/copying the file at `source_path` into storage and recording it.
|
||||
|
||||
``first_frame``, when provided (e.g. the upload path already decoded frame 0 to
|
||||
prove decodability), is used as the thumbnail source instead of spawning another
|
||||
decode worker.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def update(self, video_name: str, changes: VideoRecordChanges) -> VideoDTO:
|
||||
"""Updates a video."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_record(self, video_name: str) -> VideoRecord:
|
||||
"""Gets a video record."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_dto(self, video_name: str) -> VideoDTO:
|
||||
"""Gets a video DTO."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_metadata(self, video_name: str) -> Optional[MetadataField]:
|
||||
"""Gets a video's metadata."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_workflow(self, video_name: str) -> Optional[str]:
|
||||
"""Gets a video's workflow."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_graph(self, video_name: str) -> Optional[str]:
|
||||
"""Gets a video's graph."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_path(self, video_name: str, thumbnail: bool = False) -> str:
|
||||
"""Gets a video's on-disk path."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_url(self, video_name: str, thumbnail: bool = False) -> str:
|
||||
"""Gets a video's URL."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_many(
|
||||
self,
|
||||
offset: int = 0,
|
||||
limit: int = 10,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
video_origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> OffsetPaginatedResults[VideoDTO]:
|
||||
"""Gets a paginated list of video DTOs."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete(self, video_name: str) -> None:
|
||||
"""Deletes a video."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete_videos_on_board(self, board_id: str, user_id: Optional[str] = None) -> tuple[list[str], list[str]]:
|
||||
"""Deletes all videos on a board; returns ``(deleted_names, failed_names)``.
|
||||
|
||||
When ``user_id`` is provided, only videos owned by that user are deleted (other users'
|
||||
contributions to a public/shared board are preserved). Pass ``None`` for the admin
|
||||
path to delete every video on the board regardless of uploader.
|
||||
|
||||
Videos whose backing file deletion fails are intentionally retained (their DB records
|
||||
survive and cascade to "uncategorized" via the board_videos FK), so the returned lists
|
||||
are the authoritative ``deleted_videos``/``failed_videos`` for the caller's response —
|
||||
callers must not reconstruct failures by diffing their own board listing, which races
|
||||
with concurrent moves/deletes.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_video_names(
|
||||
self,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
video_origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> VideoNamesResult:
|
||||
"""Gets ordered list of video names."""
|
||||
pass
|
||||
@@ -0,0 +1,64 @@
|
||||
from typing import Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from invokeai.app.services.video_records.video_records_common import VideoRecord
|
||||
from invokeai.app.util.model_exclude_null import BaseModelExcludeNull
|
||||
|
||||
|
||||
class VideoUrlsDTO(BaseModelExcludeNull):
|
||||
"""The URLs for a video and its thumbnail."""
|
||||
|
||||
video_name: str = Field(description="The unique name of the video.")
|
||||
video_url: str = Field(description="The URL of the video file (MP4).")
|
||||
thumbnail_url: str = Field(description="The URL of the video's first-frame thumbnail (WebP).")
|
||||
|
||||
|
||||
class VideoDTO(VideoRecord, VideoUrlsDTO):
|
||||
"""Deserialized video record, enriched for the frontend."""
|
||||
|
||||
board_id: Optional[str] = Field(
|
||||
default=None, description="The id of the board the video belongs to, if one exists."
|
||||
)
|
||||
|
||||
|
||||
def video_record_to_dto(
|
||||
video_record: VideoRecord,
|
||||
video_url: str,
|
||||
thumbnail_url: str,
|
||||
board_id: Optional[str],
|
||||
) -> VideoDTO:
|
||||
"""Converts a video record to a video DTO."""
|
||||
return VideoDTO(
|
||||
**video_record.model_dump(),
|
||||
video_url=video_url,
|
||||
thumbnail_url=thumbnail_url,
|
||||
board_id=board_id,
|
||||
)
|
||||
|
||||
|
||||
class VideoResultWithAffectedBoards(BaseModel):
|
||||
affected_boards: list[str] = Field(description="The ids of boards affected by the operation")
|
||||
|
||||
|
||||
class DeleteVideosResult(VideoResultWithAffectedBoards):
|
||||
deleted_videos: list[str] = Field(description="The names of the videos that were deleted")
|
||||
failed_videos: list[str] = Field(description="The names of videos that were not deleted")
|
||||
|
||||
|
||||
class StarredVideosResult(VideoResultWithAffectedBoards):
|
||||
starred_videos: list[str] = Field(description="The names of the videos that were starred")
|
||||
failed_videos: list[str] = Field(description="The names of videos that were not starred")
|
||||
|
||||
|
||||
class UnstarredVideosResult(VideoResultWithAffectedBoards):
|
||||
unstarred_videos: list[str] = Field(description="The names of the videos that were unstarred")
|
||||
failed_videos: list[str] = Field(description="The names of videos that were not unstarred")
|
||||
|
||||
|
||||
class AddVideosToBoardResult(VideoResultWithAffectedBoards):
|
||||
added_videos: list[str] = Field(description="The video names that were added to the board")
|
||||
|
||||
|
||||
class RemoveVideosFromBoardResult(VideoResultWithAffectedBoards):
|
||||
removed_videos: list[str] = Field(description="The video names that were removed from their board")
|
||||
@@ -0,0 +1,406 @@
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from invokeai.app.invocations.fields import MetadataField
|
||||
from invokeai.app.services.image_records.image_records_common import (
|
||||
ImageCategory,
|
||||
InvalidImageCategoryException,
|
||||
InvalidOriginException,
|
||||
ResourceOrigin,
|
||||
)
|
||||
from invokeai.app.services.invoker import Invoker
|
||||
from invokeai.app.services.shared.pagination import OffsetPaginatedResults
|
||||
from invokeai.app.services.shared.sqlite.sqlite_common import SQLiteDirection
|
||||
from invokeai.app.services.video_files.video_files_common import (
|
||||
VideoFileDeleteException,
|
||||
VideoFileNotFoundException,
|
||||
VideoFileSaveException,
|
||||
)
|
||||
from invokeai.app.services.video_records.video_records_common import (
|
||||
VideoNamesResult,
|
||||
VideoRecord,
|
||||
VideoRecordChanges,
|
||||
VideoRecordDeleteException,
|
||||
VideoRecordNotFoundException,
|
||||
VideoRecordSaveException,
|
||||
)
|
||||
from invokeai.app.services.videos.videos_base import VideoServiceABC
|
||||
from invokeai.app.services.videos.videos_common import VideoDTO, video_record_to_dto
|
||||
|
||||
|
||||
class VideoService(VideoServiceABC):
|
||||
__invoker: Invoker
|
||||
|
||||
def start(self, invoker: Invoker) -> None:
|
||||
self.__invoker = invoker
|
||||
|
||||
def create(
|
||||
self,
|
||||
source_path: Path,
|
||||
width: int,
|
||||
height: int,
|
||||
duration: float,
|
||||
fps: Optional[float],
|
||||
video_origin: ResourceOrigin,
|
||||
video_category: ImageCategory,
|
||||
node_id: Optional[str] = None,
|
||||
session_id: Optional[str] = None,
|
||||
board_id: Optional[str] = None,
|
||||
is_intermediate: Optional[bool] = False,
|
||||
metadata: Optional[str] = None,
|
||||
workflow: Optional[str] = None,
|
||||
graph: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
first_frame: Optional[Image.Image] = None,
|
||||
) -> VideoDTO:
|
||||
if video_origin not in ResourceOrigin:
|
||||
raise InvalidOriginException
|
||||
if video_category not in ImageCategory:
|
||||
raise InvalidImageCategoryException
|
||||
|
||||
video_name = self.__invoker.services.names.create_video_name()
|
||||
|
||||
# Reuse the image subfolder strategy for video organization.
|
||||
from invokeai.app.services.image_files.image_subfolder_strategy import create_subfolder_strategy
|
||||
|
||||
strategy_name = self.__invoker.services.configuration.image_subfolder_strategy
|
||||
strategy = create_subfolder_strategy(strategy_name)
|
||||
video_subfolder = strategy.get_subfolder(video_name, video_category, is_intermediate or False)
|
||||
|
||||
record_saved = False
|
||||
board_attached = False
|
||||
try:
|
||||
self.__invoker.services.video_records.save(
|
||||
video_name=video_name,
|
||||
video_origin=video_origin,
|
||||
video_category=video_category,
|
||||
width=width,
|
||||
height=height,
|
||||
duration=duration,
|
||||
fps=fps,
|
||||
has_workflow=workflow is not None or graph is not None,
|
||||
is_intermediate=is_intermediate,
|
||||
node_id=node_id,
|
||||
metadata=metadata,
|
||||
session_id=session_id,
|
||||
user_id=user_id,
|
||||
video_subfolder=video_subfolder,
|
||||
)
|
||||
record_saved = True
|
||||
if board_id is not None:
|
||||
# Board attachment is deliberately best-effort, mirroring ImageService.create:
|
||||
# this is reachable when the board is deleted between the caller's access
|
||||
# check and this insert, and failing the whole create here would destroy a
|
||||
# just-generated video over a cosmetic categorization problem. The returned
|
||||
# DTO reports the video's *actual* board (None on fallback), so callers are
|
||||
# not told the attachment succeeded.
|
||||
try:
|
||||
self.__invoker.services.board_video_records.add_video_to_board(
|
||||
board_id=board_id, video_name=video_name
|
||||
)
|
||||
board_attached = True
|
||||
except Exception as e:
|
||||
self.__invoker.services.logger.warning(f"Failed to add video to board {board_id}: {str(e)}")
|
||||
|
||||
self.__invoker.services.video_files.save(
|
||||
source_path=source_path,
|
||||
video_name=video_name,
|
||||
video_subfolder=video_subfolder,
|
||||
metadata=metadata,
|
||||
workflow=workflow,
|
||||
graph=graph,
|
||||
first_frame=first_frame,
|
||||
)
|
||||
|
||||
video_dto = self.get_dto(video_name)
|
||||
self._on_changed(video_dto)
|
||||
return video_dto
|
||||
except VideoRecordSaveException:
|
||||
self.__invoker.services.logger.error("Failed to save video record")
|
||||
raise
|
||||
except Exception as e:
|
||||
# Roll back any DB-side state we created so the gallery doesn't end up with a
|
||||
# ghost record whose file endpoints 404. Most commonly triggered by
|
||||
# VideoFileSaveException (disk save or sidecar write failure), but we also
|
||||
# need to unwind on any unexpected post-record failure.
|
||||
if board_attached:
|
||||
try:
|
||||
self.__invoker.services.board_video_records.remove_video_from_board(video_name=video_name)
|
||||
except Exception as rollback_err:
|
||||
self.__invoker.services.logger.error(
|
||||
f"Failed to roll back board attachment for {video_name}: {str(rollback_err)}"
|
||||
)
|
||||
if record_saved:
|
||||
try:
|
||||
self.__invoker.services.video_records.delete(video_name)
|
||||
except Exception as rollback_err:
|
||||
self.__invoker.services.logger.error(
|
||||
f"Failed to roll back video record for {video_name}: {str(rollback_err)}"
|
||||
)
|
||||
# The disk layer cleans up after itself when the save fails, but a failure
|
||||
# after a successful file save (e.g. building the DTO) would still leave the
|
||||
# files on disk with no record pointing at them. delete() skips files that
|
||||
# don't exist, so this is a no-op when nothing was written.
|
||||
try:
|
||||
self.__invoker.services.video_files.delete(video_name, video_subfolder=video_subfolder)
|
||||
except Exception as rollback_err:
|
||||
self.__invoker.services.logger.error(
|
||||
f"Failed to roll back video files for {video_name}: {str(rollback_err)}"
|
||||
)
|
||||
if isinstance(e, VideoFileSaveException):
|
||||
self.__invoker.services.logger.error("Failed to save video file")
|
||||
else:
|
||||
self.__invoker.services.logger.error(f"Problem saving video record and file: {str(e)}")
|
||||
raise
|
||||
|
||||
def update(self, video_name: str, changes: VideoRecordChanges) -> VideoDTO:
|
||||
try:
|
||||
self.__invoker.services.video_records.update(video_name, changes)
|
||||
video_dto = self.get_dto(video_name)
|
||||
self._on_changed(video_dto)
|
||||
return video_dto
|
||||
except VideoRecordSaveException:
|
||||
self.__invoker.services.logger.error("Failed to update video record")
|
||||
raise
|
||||
except Exception as e:
|
||||
self.__invoker.services.logger.error("Problem updating video record")
|
||||
raise e
|
||||
|
||||
def get_record(self, video_name: str) -> VideoRecord:
|
||||
try:
|
||||
return self.__invoker.services.video_records.get(video_name)
|
||||
except VideoRecordNotFoundException:
|
||||
self.__invoker.services.logger.error("Video record not found")
|
||||
raise
|
||||
except Exception as e:
|
||||
self.__invoker.services.logger.error("Problem getting video record")
|
||||
raise e
|
||||
|
||||
def get_dto(self, video_name: str) -> VideoDTO:
|
||||
try:
|
||||
video_record = self.__invoker.services.video_records.get(video_name)
|
||||
return video_record_to_dto(
|
||||
video_record=video_record,
|
||||
video_url=self.__invoker.services.urls.get_video_url(video_name),
|
||||
thumbnail_url=self.__invoker.services.urls.get_video_url(video_name, thumbnail=True),
|
||||
board_id=self.__invoker.services.board_video_records.get_board_for_video(video_name),
|
||||
)
|
||||
except VideoRecordNotFoundException:
|
||||
self.__invoker.services.logger.error("Video record not found")
|
||||
raise
|
||||
except Exception as e:
|
||||
self.__invoker.services.logger.error("Problem getting video DTO")
|
||||
raise e
|
||||
|
||||
def get_metadata(self, video_name: str) -> Optional[MetadataField]:
|
||||
try:
|
||||
return self.__invoker.services.video_records.get_metadata(video_name)
|
||||
except VideoRecordNotFoundException:
|
||||
self.__invoker.services.logger.error("Video record not found")
|
||||
raise
|
||||
except Exception as e:
|
||||
self.__invoker.services.logger.error("Problem getting video metadata")
|
||||
raise e
|
||||
|
||||
def get_workflow(self, video_name: str) -> Optional[str]:
|
||||
try:
|
||||
record = self.__invoker.services.video_records.get(video_name)
|
||||
return self.__invoker.services.video_files.get_workflow(video_name, video_subfolder=record.video_subfolder)
|
||||
except VideoFileNotFoundException:
|
||||
self.__invoker.services.logger.error("Video file not found")
|
||||
raise
|
||||
except Exception:
|
||||
self.__invoker.services.logger.error("Problem getting video workflow")
|
||||
raise
|
||||
|
||||
def get_graph(self, video_name: str) -> Optional[str]:
|
||||
try:
|
||||
record = self.__invoker.services.video_records.get(video_name)
|
||||
return self.__invoker.services.video_files.get_graph(video_name, video_subfolder=record.video_subfolder)
|
||||
except VideoFileNotFoundException:
|
||||
self.__invoker.services.logger.error("Video file not found")
|
||||
raise
|
||||
except Exception:
|
||||
self.__invoker.services.logger.error("Problem getting video graph")
|
||||
raise
|
||||
|
||||
def get_path(self, video_name: str, thumbnail: bool = False) -> str:
|
||||
try:
|
||||
record = self.__invoker.services.video_records.get(video_name)
|
||||
return str(
|
||||
self.__invoker.services.video_files.get_path(
|
||||
video_name, thumbnail=thumbnail, video_subfolder=record.video_subfolder
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
self.__invoker.services.logger.error("Problem getting video path")
|
||||
raise e
|
||||
|
||||
def get_url(self, video_name: str, thumbnail: bool = False) -> str:
|
||||
try:
|
||||
return self.__invoker.services.urls.get_video_url(video_name, thumbnail=thumbnail)
|
||||
except Exception as e:
|
||||
self.__invoker.services.logger.error("Problem getting video URL")
|
||||
raise e
|
||||
|
||||
def get_many(
|
||||
self,
|
||||
offset: int = 0,
|
||||
limit: int = 10,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
video_origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> OffsetPaginatedResults[VideoDTO]:
|
||||
try:
|
||||
results = self.__invoker.services.video_records.get_many(
|
||||
offset,
|
||||
limit,
|
||||
starred_first,
|
||||
order_dir,
|
||||
video_origin,
|
||||
categories,
|
||||
is_intermediate,
|
||||
board_id,
|
||||
search_term,
|
||||
user_id,
|
||||
is_admin,
|
||||
)
|
||||
video_dtos = [
|
||||
video_record_to_dto(
|
||||
video_record=r,
|
||||
video_url=self.__invoker.services.urls.get_video_url(r.video_name),
|
||||
thumbnail_url=self.__invoker.services.urls.get_video_url(r.video_name, thumbnail=True),
|
||||
board_id=self.__invoker.services.board_video_records.get_board_for_video(r.video_name),
|
||||
)
|
||||
for r in results.items
|
||||
]
|
||||
return OffsetPaginatedResults[VideoDTO](
|
||||
items=video_dtos, offset=results.offset, limit=results.limit, total=results.total
|
||||
)
|
||||
except Exception as e:
|
||||
self.__invoker.services.logger.error("Problem getting paginated video DTOs")
|
||||
raise e
|
||||
|
||||
def delete(self, video_name: str) -> None:
|
||||
token: object | None = None
|
||||
record_deleted = False
|
||||
try:
|
||||
record = self.__invoker.services.video_records.get(video_name)
|
||||
token = self.__invoker.services.video_files.stage_delete(video_name, video_subfolder=record.video_subfolder)
|
||||
self.__invoker.services.video_records.delete(video_name)
|
||||
record_deleted = True
|
||||
try:
|
||||
self.__invoker.services.video_files.commit_delete(token)
|
||||
except Exception as cleanup_error:
|
||||
self.__invoker.services.logger.error(f"Failed to purge staged video files: {cleanup_error}")
|
||||
self._on_deleted(video_name)
|
||||
except VideoRecordDeleteException:
|
||||
if token is not None:
|
||||
self.__invoker.services.video_files.rollback_delete(token)
|
||||
self.__invoker.services.logger.error("Failed to delete video record")
|
||||
raise
|
||||
except VideoFileDeleteException:
|
||||
self.__invoker.services.logger.error("Failed to delete video file")
|
||||
raise
|
||||
except Exception as e:
|
||||
if token is not None and not record_deleted:
|
||||
try:
|
||||
self.__invoker.services.video_files.rollback_delete(token)
|
||||
except Exception as rollback_error:
|
||||
self.__invoker.services.logger.error(f"Failed to restore video files: {rollback_error}")
|
||||
self.__invoker.services.logger.error("Problem deleting video record and file")
|
||||
raise e
|
||||
|
||||
def delete_videos_on_board(self, board_id: str, user_id: Optional[str] = None) -> tuple[list[str], list[str]]:
|
||||
try:
|
||||
# When ``user_id`` is set the lookup filters to videos owned by that user so the
|
||||
# cascade doesn't destroy other users' contributions to a public/shared board.
|
||||
video_names = self.__invoker.services.board_video_records.get_all_board_video_names_for_board(
|
||||
board_id, categories=None, is_intermediate=None, user_id=user_id
|
||||
)
|
||||
# Only delete records for files we actually managed to remove. Otherwise a
|
||||
# transient FS error would leave the file orphaned on disk with no record
|
||||
# pointing at it — the API would report success and the user would have no
|
||||
# way to clean up the leak. The board itself will still be deleted by the
|
||||
# caller, so any preserved records cascade to "uncategorized" via the
|
||||
# board_videos FK.
|
||||
deleted_video_names: list[str] = []
|
||||
failed_video_names: list[str] = []
|
||||
staged_deletes: list[tuple[str, object]] = []
|
||||
for video_name in video_names:
|
||||
try:
|
||||
record = self.__invoker.services.video_records.get(video_name)
|
||||
token = self.__invoker.services.video_files.stage_delete(
|
||||
video_name, video_subfolder=record.video_subfolder
|
||||
)
|
||||
staged_deletes.append((video_name, token))
|
||||
deleted_video_names.append(video_name)
|
||||
except Exception as e:
|
||||
failed_video_names.append(video_name)
|
||||
self.__invoker.services.logger.error(
|
||||
f"Failed to delete video file {video_name}; keeping record: {str(e)}"
|
||||
)
|
||||
try:
|
||||
self.__invoker.services.video_records.delete_many(deleted_video_names)
|
||||
except Exception:
|
||||
for video_name, token in staged_deletes:
|
||||
try:
|
||||
self.__invoker.services.video_files.rollback_delete(token)
|
||||
except Exception as rollback_error:
|
||||
self.__invoker.services.logger.error(
|
||||
f"Failed to restore staged video files for {video_name}: {rollback_error}"
|
||||
)
|
||||
raise
|
||||
for _, token in staged_deletes:
|
||||
try:
|
||||
self.__invoker.services.video_files.commit_delete(token)
|
||||
except Exception as cleanup_error:
|
||||
self.__invoker.services.logger.error(f"Failed to purge staged video files: {cleanup_error}")
|
||||
for video_name in deleted_video_names:
|
||||
self._on_deleted(video_name)
|
||||
return deleted_video_names, failed_video_names
|
||||
except VideoRecordDeleteException:
|
||||
self.__invoker.services.logger.error("Failed to delete video records")
|
||||
raise
|
||||
except VideoFileDeleteException:
|
||||
self.__invoker.services.logger.error("Failed to delete video files")
|
||||
raise
|
||||
except Exception as e:
|
||||
self.__invoker.services.logger.error(f"Problem deleting video records and files: {str(e)}")
|
||||
raise e
|
||||
|
||||
def get_video_names(
|
||||
self,
|
||||
starred_first: bool = True,
|
||||
order_dir: SQLiteDirection = SQLiteDirection.Descending,
|
||||
video_origin: Optional[ResourceOrigin] = None,
|
||||
categories: Optional[list[ImageCategory]] = None,
|
||||
is_intermediate: Optional[bool] = None,
|
||||
board_id: Optional[str] = None,
|
||||
search_term: Optional[str] = None,
|
||||
user_id: Optional[str] = None,
|
||||
is_admin: bool = False,
|
||||
) -> VideoNamesResult:
|
||||
try:
|
||||
return self.__invoker.services.video_records.get_video_names(
|
||||
starred_first=starred_first,
|
||||
order_dir=order_dir,
|
||||
video_origin=video_origin,
|
||||
categories=categories,
|
||||
is_intermediate=is_intermediate,
|
||||
board_id=board_id,
|
||||
search_term=search_term,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
except Exception as e:
|
||||
self.__invoker.services.logger.error("Problem getting video names")
|
||||
raise e
|
||||
@@ -4,11 +4,16 @@ from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class VirtualSubBoardDTO(BaseModel):
|
||||
"""A virtual sub-board computed from image metadata, not stored in the database."""
|
||||
"""A virtual sub-board computed from image/video metadata, not stored in the database."""
|
||||
|
||||
virtual_board_id: str = Field(description="The virtual board ID, e.g. 'by_date:2026-03-18'.")
|
||||
board_name: str = Field(description="The display name of the virtual sub-board, e.g. '2026-03-18'.")
|
||||
date: str = Field(description="The ISO date string, e.g. '2026-03-18'.")
|
||||
image_count: int = Field(description="The number of general images for this date.")
|
||||
asset_count: int = Field(description="The number of asset images for this date.")
|
||||
video_count: int = Field(default=0, description="The number of videos for this date.")
|
||||
cover_image_name: Optional[str] = Field(default=None, description="The most recent image name for this date.")
|
||||
cover_video_name: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The most recent video name for this date. Set instead of cover_image_name when the newest item for the date is a video.",
|
||||
)
|
||||
|
||||
+1622
File diff suppressed because it is too large
Load Diff
+1556
File diff suppressed because it is too large
Load Diff
+1686
File diff suppressed because it is too large
Load Diff
+1193
File diff suppressed because it is too large
Load Diff
+1224
File diff suppressed because it is too large
Load Diff
+872
@@ -0,0 +1,872 @@
|
||||
{
|
||||
"name": "Image to Video - Wan 2.2 TI2V (Low Quality)",
|
||||
"author": "InvokeAI",
|
||||
"description": "Low quality but fast image-to-video generation with Wan 2.2 TI2V 5B model. Provide an initial image and a target resolution for the video.",
|
||||
"version": "1.0.0",
|
||||
"contact": "",
|
||||
"tags": "wan2.2, image to video, video",
|
||||
"notes": "This workflow needs the Wan 2.2 TI2V-5B model. Each InvokeAI install assigns its own internal model IDs, so the model fields are left blank -- select these in the form after loading:\n\nMain Model node:\n- Main Model: Wan 2.2 TI2V-5B (Q4_K_M GGUF recommended for <=16 GB VRAM)\n- Transformer (Low Noise): leave empty (TI2V-5B is a single-expert model)\n- VAE: Wan 2.2 TI2V-5B VAE\n- T5 Encoder: Wan T5 Encoder (UMT5-XXL)\n\n720p should work on GPUs with 16 GB VRAM and up.",
|
||||
"exposedFields": [
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "transformer_low_noise_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "component_source"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "vae_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "wan_t5_encoder_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
|
||||
"fieldName": "lora"
|
||||
},
|
||||
{
|
||||
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
|
||||
"fieldName": "weight"
|
||||
},
|
||||
{
|
||||
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
|
||||
"fieldName": "target"
|
||||
},
|
||||
{
|
||||
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
|
||||
"fieldName": "lora"
|
||||
},
|
||||
{
|
||||
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
|
||||
"fieldName": "weight"
|
||||
},
|
||||
{
|
||||
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
|
||||
"fieldName": "target"
|
||||
},
|
||||
{
|
||||
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "steps"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale_low_noise"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "width"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "height"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "num_frames"
|
||||
},
|
||||
{
|
||||
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"fieldName": "fps"
|
||||
}
|
||||
],
|
||||
"meta": {
|
||||
"version": "4.0.0",
|
||||
"category": "default"
|
||||
},
|
||||
"nodes": [
|
||||
{
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_model_loader",
|
||||
"inputs": {
|
||||
"model": {
|
||||
"name": "model",
|
||||
"label": "Transformer (High Noise)",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"transformer_low_noise_model": {
|
||||
"name": "transformer_low_noise_model",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"vae_model": {
|
||||
"name": "vae_model",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"wan_t5_encoder_model": {
|
||||
"name": "wan_t5_encoder_model",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"component_source": {
|
||||
"name": "component_source",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false
|
||||
},
|
||||
"position": {
|
||||
"x": 0,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "Positive Prompt",
|
||||
"notes": "",
|
||||
"type": "wan_text_encoder",
|
||||
"inputs": {
|
||||
"prompt": {
|
||||
"name": "prompt",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": ""
|
||||
},
|
||||
"wan_t5_encoder": {
|
||||
"name": "wan_t5_encoder",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 686.9799980678728,
|
||||
"y": 210.13999972398182
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "Negative Prompt (unused at CFG=1.0)",
|
||||
"notes": "",
|
||||
"type": "wan_text_encoder",
|
||||
"inputs": {
|
||||
"prompt": {
|
||||
"name": "prompt",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": " "
|
||||
},
|
||||
"wan_t5_encoder": {
|
||||
"name": "wan_t5_encoder",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 700,
|
||||
"y": 392
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "rand_int",
|
||||
"inputs": {
|
||||
"low": {
|
||||
"name": "low",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 0
|
||||
},
|
||||
"high": {
|
||||
"name": "high",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 2147483647
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false
|
||||
},
|
||||
"position": {
|
||||
"x": 700,
|
||||
"y": 572
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_video_denoise",
|
||||
"inputs": {
|
||||
"transformer": {
|
||||
"name": "transformer",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"positive_conditioning": {
|
||||
"name": "positive_conditioning",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"negative_conditioning": {
|
||||
"name": "negative_conditioning",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"ref_image": {
|
||||
"name": "ref_image",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"guidance_scale": {
|
||||
"name": "guidance_scale",
|
||||
"label": "CFG (High Noise)",
|
||||
"description": "",
|
||||
"value": 1
|
||||
},
|
||||
"guidance_scale_low_noise": {
|
||||
"name": "guidance_scale_low_noise",
|
||||
"label": "CFG (Low Noise)",
|
||||
"description": "",
|
||||
"value": 1
|
||||
},
|
||||
"width": {
|
||||
"name": "width",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 832
|
||||
},
|
||||
"height": {
|
||||
"name": "height",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 480
|
||||
},
|
||||
"num_frames": {
|
||||
"name": "num_frames",
|
||||
"label": "Frames",
|
||||
"description": "",
|
||||
"value": 81
|
||||
},
|
||||
"steps": {
|
||||
"name": "steps",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 20
|
||||
},
|
||||
"seed": {
|
||||
"name": "seed",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 0
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 1050,
|
||||
"y": 230
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_l2v",
|
||||
"inputs": {
|
||||
"board": {
|
||||
"name": "board",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"metadata": {
|
||||
"name": "metadata",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"latents": {
|
||||
"name": "latents",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"vae": {
|
||||
"name": "vae",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"fps": {
|
||||
"name": "fps",
|
||||
"label": "FPS",
|
||||
"description": "",
|
||||
"value": 16
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": false,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 1400,
|
||||
"y": 33
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
|
||||
"version": "1.2.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_ref_image_encoder",
|
||||
"inputs": {
|
||||
"image": {
|
||||
"name": "image",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"end_image": { "name": "end_image", "label": "" },
|
||||
"vae": {
|
||||
"name": "vae",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"width": {
|
||||
"name": "width",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 1024
|
||||
},
|
||||
"height": {
|
||||
"name": "height",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 1024
|
||||
},
|
||||
"num_frames": {
|
||||
"name": "num_frames",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 1
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 669.4858247907592,
|
||||
"y": -383.75896955550337
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
|
||||
"version": "1.0.2",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "image",
|
||||
"inputs": {
|
||||
"image": {
|
||||
"name": "image",
|
||||
"label": "Initial Image",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": -165.5308813883526,
|
||||
"y": -364.6304014144227
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_ti2v_ideal_dimensions",
|
||||
"inputs": {
|
||||
"width": {
|
||||
"name": "width",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 1024
|
||||
},
|
||||
"height": {
|
||||
"name": "height",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 1024
|
||||
},
|
||||
"target_resolution": {
|
||||
"name": "target_resolution",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": "720p"
|
||||
},
|
||||
"rounding": {
|
||||
"name": "rounding",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": "nearest"
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 261.9930762062073,
|
||||
"y": -602.4138188363532
|
||||
}
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"id": "edge-loader-t5-pos",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-t5-neg",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-vae-l2v",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "vae",
|
||||
"targetHandle": "vae"
|
||||
},
|
||||
{
|
||||
"id": "edge-pos-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "positive_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-neg-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "negative_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-rand-seed-denoise",
|
||||
"type": "default",
|
||||
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "value",
|
||||
"targetHandle": "seed"
|
||||
},
|
||||
{
|
||||
"id": "edge-denoise-latents-l2v",
|
||||
"type": "default",
|
||||
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "latents",
|
||||
"targetHandle": "latents"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7etransformer-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabtransformer",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-1473db93-45eb-4468-bba1-d0f9d1c21aedref_image-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabref_image",
|
||||
"type": "default",
|
||||
"source": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "ref_image",
|
||||
"targetHandle": "ref_image"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7evae-1473db93-45eb-4468-bba1-d0f9d1c21aedvae",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
|
||||
"sourceHandle": "vae",
|
||||
"targetHandle": "vae"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-0771d3a4-7f0b-45bd-9c73-a99b40e24b7bimage-1473db93-45eb-4468-bba1-d0f9d1c21aedimage",
|
||||
"type": "default",
|
||||
"source": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
|
||||
"target": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
|
||||
"sourceHandle": "image",
|
||||
"targetHandle": "image"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-0771d3a4-7f0b-45bd-9c73-a99b40e24b7bwidth-55cb8b4f-b7fc-4d37-9861-d5c84a57353bwidth",
|
||||
"type": "default",
|
||||
"source": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
|
||||
"target": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
|
||||
"sourceHandle": "width",
|
||||
"targetHandle": "width"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-0771d3a4-7f0b-45bd-9c73-a99b40e24b7bheight-55cb8b4f-b7fc-4d37-9861-d5c84a57353bheight",
|
||||
"type": "default",
|
||||
"source": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
|
||||
"target": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
|
||||
"sourceHandle": "height",
|
||||
"targetHandle": "height"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-55cb8b4f-b7fc-4d37-9861-d5c84a57353bwidth-1473db93-45eb-4468-bba1-d0f9d1c21aedwidth",
|
||||
"type": "default",
|
||||
"source": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
|
||||
"target": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
|
||||
"sourceHandle": "width",
|
||||
"targetHandle": "width"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-55cb8b4f-b7fc-4d37-9861-d5c84a57353bheight-1473db93-45eb-4468-bba1-d0f9d1c21aedheight",
|
||||
"type": "default",
|
||||
"source": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
|
||||
"target": "1473db93-45eb-4468-bba1-d0f9d1c21aed",
|
||||
"sourceHandle": "height",
|
||||
"targetHandle": "height"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-55cb8b4f-b7fc-4d37-9861-d5c84a57353bwidth-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabwidth",
|
||||
"type": "default",
|
||||
"source": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "width",
|
||||
"targetHandle": "width"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-55cb8b4f-b7fc-4d37-9861-d5c84a57353bheight-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabheight",
|
||||
"type": "default",
|
||||
"source": "55cb8b4f-b7fc-4d37-9861-d5c84a57353b",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "height",
|
||||
"targetHandle": "height"
|
||||
}
|
||||
],
|
||||
"form": {
|
||||
"elements": {
|
||||
"root": {
|
||||
"id": "root",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"heading-XDPaQofdEE",
|
||||
"node-field-KcT0glAFDK",
|
||||
"node-field-e3hxuqgTgS",
|
||||
"container-mUeBcw2tYf",
|
||||
"heading-wuPtY8B4vR",
|
||||
"container-AkpUH1WtID",
|
||||
"heading-10PuWMKJg9",
|
||||
"container-C0z6ru33to"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"node-field-zQA9XuRZRW": {
|
||||
"id": "node-field-zQA9XuRZRW",
|
||||
"parentId": "container-mUeBcw2tYf",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"fieldName": "fps"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-ok9GStueKC": {
|
||||
"id": "node-field-ok9GStueKC",
|
||||
"parentId": "container-mUeBcw2tYf",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "num_frames"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-PdU0Sz2m5z": {
|
||||
"id": "node-field-PdU0Sz2m5z",
|
||||
"parentId": "container-y0w7TfJC7e",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "float-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-UiQ7MaseDR": {
|
||||
"id": "node-field-UiQ7MaseDR",
|
||||
"parentId": "container-mUeBcw2tYf",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "steps"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-KcT0glAFDK": {
|
||||
"id": "node-field-KcT0glAFDK",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "string-field-config",
|
||||
"component": "textarea"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-N9mexEKkYg": {
|
||||
"id": "node-field-N9mexEKkYg",
|
||||
"parentId": "container-C0z6ru33to",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "wan_t5_encoder_model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-2u9lAbMBnD": {
|
||||
"id": "node-field-2u9lAbMBnD",
|
||||
"parentId": "container-C0z6ru33to",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "vae_model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-CTGqrO3x8o": {
|
||||
"id": "node-field-CTGqrO3x8o",
|
||||
"parentId": "container-C0z6ru33to",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "component_source"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-Q345WyNzc4": {
|
||||
"id": "node-field-Q345WyNzc4",
|
||||
"parentId": "container-y0w7TfJC7e",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"container-mUeBcw2tYf": {
|
||||
"id": "container-mUeBcw2tYf",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-UiQ7MaseDR",
|
||||
"node-field-ok9GStueKC",
|
||||
"node-field-zQA9XuRZRW"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"heading-XDPaQofdEE": {
|
||||
"id": "heading-XDPaQofdEE",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Generation"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"heading-wuPtY8B4vR": {
|
||||
"id": "heading-wuPtY8B4vR",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Model"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"container-AkpUH1WtID": {
|
||||
"id": "container-AkpUH1WtID",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-y0w7TfJC7e"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-y0w7TfJC7e": {
|
||||
"id": "container-y0w7TfJC7e",
|
||||
"parentId": "container-AkpUH1WtID",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"node-field-Q345WyNzc4",
|
||||
"node-field-PdU0Sz2m5z"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-C0z6ru33to": {
|
||||
"id": "container-C0z6ru33to",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-N9mexEKkYg",
|
||||
"node-field-2u9lAbMBnD",
|
||||
"node-field-CTGqrO3x8o"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"heading-10PuWMKJg9": {
|
||||
"id": "heading-10PuWMKJg9",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Advanced"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"node-field-e3hxuqgTgS": {
|
||||
"id": "node-field-e3hxuqgTgS",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "0771d3a4-7f0b-45bd-9c73-a99b40e24b7b",
|
||||
"fieldName": "image"
|
||||
},
|
||||
"showDescription": true
|
||||
},
|
||||
"type": "node-field"
|
||||
}
|
||||
},
|
||||
"rootElementId": "root"
|
||||
},
|
||||
"id": "default_wan22_i2v_ti2v5b"
|
||||
}
|
||||
+1354
File diff suppressed because it is too large
Load Diff
+991
@@ -0,0 +1,991 @@
|
||||
{
|
||||
"name": "Text to Video - Wan 2.2 Lightning w/ Concept LoRAs",
|
||||
"author": "InvokeAI",
|
||||
"description": "Fast text-to-video generation with Wan 2.2 T2V A14B + the Lightning LoRA pair. Distillation lets the model converge in 4 steps with CFG 1.0 (no negative branch), roughly 20x faster than the standard T2V workflow at the cost of some image quality. Pick the high-noise Lightning LoRA on 'Apply LoRA (High)' and the low-noise one on 'Apply LoRA (Low)' — the 'auto' target routes each to the right expert when the LoRAs are tagged. Defaults: 832x480, 81 frames @ 16 FPS (~5 s).",
|
||||
"version": "1.0.0",
|
||||
"contact": "",
|
||||
"tags": "wan2.2, text to video, video, lightning, lora",
|
||||
"notes": "Each InvokeAI install assigns its own internal model IDs, so the model fields are left blank -- select these in the form after loading:\n\nMain Model node:\n- Main Model: Wan 2.2 T2V A14B High Noise (Q4_K_M GGUF recommended for <=16 GB VRAM)\n- Transformer (Low Noise): Wan 2.2 T2V A14B Low Noise (Q4_K_M)\n- VAE: Wan 2.2 A14B VAE\n- T5 Encoder: Wan T5 Encoder (UMT5-XXL)\n\nLightning LoRAs (4-step distillation - use the T2V variant, NOT I2V):\n- Apply Lightning LoRA (High Noise): Wan 2.2 T2V Lightning High Noise (4-step)\n- Apply Lightning LoRA (Low Noise): Wan 2.2 T2V Lightning Low Noise (4-step)\n\nThis workflow also has two optional concept-LoRA loaders (routed via 'target' = high/low) -- leave them empty or select your own high/low-noise concept LoRA pair.\n\nThe Lightning LoRAs are A14B-specific and distinct for T2V vs I2V. If your LoRAs are untagged (no 'expert' field), use each LoRA loader's 'target' dropdown to route the high-noise LoRA to 'high' and the low-noise to 'low'. CFG=1.0 skips the negative-conditioning branch, so the Negative Prompt is ignored at runtime.",
|
||||
"exposedFields": [
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "transformer_low_noise_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "component_source"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "vae_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "wan_t5_encoder_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "steps"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale_low_noise"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "width"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "height"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "num_frames"
|
||||
},
|
||||
{
|
||||
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"fieldName": "fps"
|
||||
}
|
||||
],
|
||||
"meta": {
|
||||
"version": "4.0.0",
|
||||
"category": "default"
|
||||
},
|
||||
"nodes": [
|
||||
{
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_model_loader",
|
||||
"inputs": {
|
||||
"model": {
|
||||
"name": "model",
|
||||
"label": "Transformer (High Noise)",
|
||||
"description": ""
|
||||
},
|
||||
"transformer_low_noise_model": {
|
||||
"name": "transformer_low_noise_model",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"vae_model": {
|
||||
"name": "vae_model",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"wan_t5_encoder_model": {
|
||||
"name": "wan_t5_encoder_model",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"component_source": {
|
||||
"name": "component_source",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"dynamicInputTemplates": {},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false
|
||||
},
|
||||
"position": {
|
||||
"x": 0,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "Positive Prompt",
|
||||
"notes": "",
|
||||
"type": "wan_text_encoder",
|
||||
"inputs": {
|
||||
"prompt": {
|
||||
"name": "prompt",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": ""
|
||||
},
|
||||
"wan_t5_encoder": {
|
||||
"name": "wan_t5_encoder",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"dynamicInputTemplates": {},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 1400,
|
||||
"y": 212
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "Negative Prompt (unused at CFG=1.0)",
|
||||
"notes": "",
|
||||
"type": "wan_text_encoder",
|
||||
"inputs": {
|
||||
"prompt": {
|
||||
"name": "prompt",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": " "
|
||||
},
|
||||
"wan_t5_encoder": {
|
||||
"name": "wan_t5_encoder",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"dynamicInputTemplates": {},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 1400,
|
||||
"y": 392
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "rand_int",
|
||||
"inputs": {
|
||||
"low": {
|
||||
"name": "low",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 0
|
||||
},
|
||||
"high": {
|
||||
"name": "high",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 2147483647
|
||||
}
|
||||
},
|
||||
"dynamicInputTemplates": {},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false
|
||||
},
|
||||
"position": {
|
||||
"x": 1400,
|
||||
"y": 572
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_video_denoise",
|
||||
"inputs": {
|
||||
"transformer": {
|
||||
"name": "transformer",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"positive_conditioning": {
|
||||
"name": "positive_conditioning",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"negative_conditioning": {
|
||||
"name": "negative_conditioning",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"ref_image": {
|
||||
"name": "ref_image",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"guidance_scale": {
|
||||
"name": "guidance_scale",
|
||||
"label": "CFG (High Noise)",
|
||||
"description": "",
|
||||
"value": 1
|
||||
},
|
||||
"guidance_scale_low_noise": {
|
||||
"name": "guidance_scale_low_noise",
|
||||
"label": "CFG (Low Noise)",
|
||||
"description": "",
|
||||
"value": 1
|
||||
},
|
||||
"width": {
|
||||
"name": "width",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 832
|
||||
},
|
||||
"height": {
|
||||
"name": "height",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 480
|
||||
},
|
||||
"num_frames": {
|
||||
"name": "num_frames",
|
||||
"label": "Frames",
|
||||
"description": "",
|
||||
"value": 81
|
||||
},
|
||||
"steps": {
|
||||
"name": "steps",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 4
|
||||
},
|
||||
"seed": {
|
||||
"name": "seed",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 0
|
||||
}
|
||||
},
|
||||
"dynamicInputTemplates": {},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 1750,
|
||||
"y": 230
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_l2v",
|
||||
"inputs": {
|
||||
"board": {
|
||||
"name": "board",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"metadata": {
|
||||
"name": "metadata",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"latents": {
|
||||
"name": "latents",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"vae": {
|
||||
"name": "vae",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"fps": {
|
||||
"name": "fps",
|
||||
"label": "FPS",
|
||||
"description": "",
|
||||
"value": 16
|
||||
}
|
||||
},
|
||||
"dynamicInputTemplates": {},
|
||||
"isOpen": true,
|
||||
"isIntermediate": false,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 2100,
|
||||
"y": 33
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "5d18cf53-905d-4eb6-afd4-6c19f790b374",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "5d18cf53-905d-4eb6-afd4-6c19f790b374",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_lora_loader",
|
||||
"inputs": {
|
||||
"lora": {
|
||||
"name": "lora",
|
||||
"label": "Lightning LoRA (High Noise)",
|
||||
"description": ""
|
||||
},
|
||||
"weight": {
|
||||
"name": "weight",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 1
|
||||
},
|
||||
"target": {
|
||||
"name": "target",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": "high"
|
||||
},
|
||||
"transformer": {
|
||||
"name": "transformer",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"dynamicInputTemplates": {},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 350,
|
||||
"y": 772
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_lora_loader",
|
||||
"inputs": {
|
||||
"lora": {
|
||||
"name": "lora",
|
||||
"label": "Lightning LoRA (Low Noise)",
|
||||
"description": ""
|
||||
},
|
||||
"weight": {
|
||||
"name": "weight",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 1
|
||||
},
|
||||
"target": {
|
||||
"name": "target",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": "low"
|
||||
},
|
||||
"transformer": {
|
||||
"name": "transformer",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"dynamicInputTemplates": {},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 700,
|
||||
"y": 772
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "f37bd1f0-f0d8-4105-8e07-5ec484298d36",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "f37bd1f0-f0d8-4105-8e07-5ec484298d36",
|
||||
"type": "wan_lora_collection_loader",
|
||||
"version": "1.0.1",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"nodePack": "invokeai",
|
||||
"inputs": {
|
||||
"loras": {
|
||||
"name": "loras",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": []
|
||||
},
|
||||
"transformer": {
|
||||
"name": "transformer",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"dynamicInputTemplates": {}
|
||||
},
|
||||
"position": {
|
||||
"x": 1053.289904257177,
|
||||
"y": 770.7485394743496
|
||||
}
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"id": "edge-loader-t5-pos",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-t5-neg",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-vae-l2v",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "vae",
|
||||
"targetHandle": "vae"
|
||||
},
|
||||
{
|
||||
"id": "edge-pos-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "positive_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-neg-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "negative_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-rand-seed-denoise",
|
||||
"type": "default",
|
||||
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "value",
|
||||
"targetHandle": "seed"
|
||||
},
|
||||
{
|
||||
"id": "edge-denoise-latents-l2v",
|
||||
"type": "default",
|
||||
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "latents",
|
||||
"targetHandle": "latents"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7etransformer-5d18cf53-905d-4eb6-afd4-6c19f790b374transformer",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "5d18cf53-905d-4eb6-afd4-6c19f790b374",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-5d18cf53-905d-4eb6-afd4-6c19f790b374transformer-eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75transformer",
|
||||
"type": "default",
|
||||
"source": "5d18cf53-905d-4eb6-afd4-6c19f790b374",
|
||||
"target": "eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75transformer-f37bd1f0-f0d8-4105-8e07-5ec484298d36transformer",
|
||||
"type": "default",
|
||||
"source": "eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75",
|
||||
"target": "f37bd1f0-f0d8-4105-8e07-5ec484298d36",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-f37bd1f0-f0d8-4105-8e07-5ec484298d36transformer-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabtransformer",
|
||||
"type": "default",
|
||||
"source": "f37bd1f0-f0d8-4105-8e07-5ec484298d36",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
}
|
||||
],
|
||||
"form": {
|
||||
"elements": {
|
||||
"root": {
|
||||
"id": "root",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"heading-XDPaQofdEE",
|
||||
"node-field-KcT0glAFDK",
|
||||
"container-i8V1oJeQCP",
|
||||
"container-mUeBcw2tYf",
|
||||
"heading-wuPtY8B4vR",
|
||||
"container-AkpUH1WtID",
|
||||
"heading-TwsoE3wQi9",
|
||||
"container-0u7wyxEdNa",
|
||||
"heading-vSOD5q5gvL",
|
||||
"node-field-EbS4ZMqDPZ",
|
||||
"heading-10PuWMKJg9",
|
||||
"container-C0z6ru33to"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"node-field-zQA9XuRZRW": {
|
||||
"id": "node-field-zQA9XuRZRW",
|
||||
"parentId": "container-mUeBcw2tYf",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"fieldName": "fps"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-ok9GStueKC": {
|
||||
"id": "node-field-ok9GStueKC",
|
||||
"parentId": "container-mUeBcw2tYf",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "num_frames"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-58JRDH0dfF": {
|
||||
"id": "node-field-58JRDH0dfF",
|
||||
"parentId": "container-i8V1oJeQCP",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "height"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-jIfJ3RUDyT": {
|
||||
"id": "node-field-jIfJ3RUDyT",
|
||||
"parentId": "container-i8V1oJeQCP",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "width"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-QDqQ4XOKvg": {
|
||||
"id": "node-field-QDqQ4XOKvg",
|
||||
"parentId": "container-7g1Xedgg8T",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale_low_noise"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "float-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-PdU0Sz2m5z": {
|
||||
"id": "node-field-PdU0Sz2m5z",
|
||||
"parentId": "container-y0w7TfJC7e",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "float-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-UiQ7MaseDR": {
|
||||
"id": "node-field-UiQ7MaseDR",
|
||||
"parentId": "container-mUeBcw2tYf",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "steps"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-KcT0glAFDK": {
|
||||
"id": "node-field-KcT0glAFDK",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "string-field-config",
|
||||
"component": "textarea"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-N9mexEKkYg": {
|
||||
"id": "node-field-N9mexEKkYg",
|
||||
"parentId": "container-C0z6ru33to",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "wan_t5_encoder_model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-2u9lAbMBnD": {
|
||||
"id": "node-field-2u9lAbMBnD",
|
||||
"parentId": "container-C0z6ru33to",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "vae_model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-CTGqrO3x8o": {
|
||||
"id": "node-field-CTGqrO3x8o",
|
||||
"parentId": "container-C0z6ru33to",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "component_source"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-7G9srQT2Uw": {
|
||||
"id": "node-field-7G9srQT2Uw",
|
||||
"parentId": "container-7g1Xedgg8T",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "transformer_low_noise_model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-Q345WyNzc4": {
|
||||
"id": "node-field-Q345WyNzc4",
|
||||
"parentId": "container-y0w7TfJC7e",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"container-mUeBcw2tYf": {
|
||||
"id": "container-mUeBcw2tYf",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-UiQ7MaseDR",
|
||||
"node-field-ok9GStueKC",
|
||||
"node-field-zQA9XuRZRW"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"heading-XDPaQofdEE": {
|
||||
"id": "heading-XDPaQofdEE",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Generation"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"container-i8V1oJeQCP": {
|
||||
"id": "container-i8V1oJeQCP",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-jIfJ3RUDyT",
|
||||
"node-field-58JRDH0dfF"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"heading-wuPtY8B4vR": {
|
||||
"id": "heading-wuPtY8B4vR",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Model"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"container-AkpUH1WtID": {
|
||||
"id": "container-AkpUH1WtID",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-y0w7TfJC7e",
|
||||
"container-7g1Xedgg8T"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-y0w7TfJC7e": {
|
||||
"id": "container-y0w7TfJC7e",
|
||||
"parentId": "container-AkpUH1WtID",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"node-field-Q345WyNzc4",
|
||||
"node-field-PdU0Sz2m5z"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-7g1Xedgg8T": {
|
||||
"id": "container-7g1Xedgg8T",
|
||||
"parentId": "container-AkpUH1WtID",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"node-field-7G9srQT2Uw",
|
||||
"node-field-QDqQ4XOKvg"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-C0z6ru33to": {
|
||||
"id": "container-C0z6ru33to",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-N9mexEKkYg",
|
||||
"node-field-2u9lAbMBnD",
|
||||
"node-field-CTGqrO3x8o"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"heading-10PuWMKJg9": {
|
||||
"id": "heading-10PuWMKJg9",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Advanced"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"container-0u7wyxEdNa": {
|
||||
"id": "container-0u7wyxEdNa",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-Um9gcFv0lo",
|
||||
"container-4owWd0thWk"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-Um9gcFv0lo": {
|
||||
"id": "container-Um9gcFv0lo",
|
||||
"parentId": "container-0u7wyxEdNa",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-lF8o60KuVb"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-4owWd0thWk": {
|
||||
"id": "container-4owWd0thWk",
|
||||
"parentId": "container-0u7wyxEdNa",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-rvE0oy63qL"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-lF8o60KuVb": {
|
||||
"id": "container-lF8o60KuVb",
|
||||
"parentId": "container-Um9gcFv0lo",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"node-field-tpqKlLmMtv"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-rvE0oy63qL": {
|
||||
"id": "container-rvE0oy63qL",
|
||||
"parentId": "container-4owWd0thWk",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"node-field-yuWwWH1ucK"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"heading-TwsoE3wQi9": {
|
||||
"id": "heading-TwsoE3wQi9",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Lightning LoRAs"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"heading-vSOD5q5gvL": {
|
||||
"id": "heading-vSOD5q5gvL",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Concept LoRAs"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"node-field-tpqKlLmMtv": {
|
||||
"id": "node-field-tpqKlLmMtv",
|
||||
"parentId": "container-lF8o60KuVb",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "5d18cf53-905d-4eb6-afd4-6c19f790b374",
|
||||
"fieldName": "lora"
|
||||
},
|
||||
"showDescription": true
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-yuWwWH1ucK": {
|
||||
"id": "node-field-yuWwWH1ucK",
|
||||
"parentId": "container-rvE0oy63qL",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "eb6b4c66-0bfb-4e0d-bc39-9d8814f3cd75",
|
||||
"fieldName": "lora"
|
||||
},
|
||||
"showDescription": true
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-EbS4ZMqDPZ": {
|
||||
"id": "node-field-EbS4ZMqDPZ",
|
||||
"type": "node-field",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "f37bd1f0-f0d8-4105-8e07-5ec484298d36",
|
||||
"fieldName": "loras"
|
||||
},
|
||||
"showDescription": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"rootElementId": "root"
|
||||
},
|
||||
"id": "default_wan22_t2v_lightning_concept"
|
||||
}
|
||||
+986
@@ -0,0 +1,986 @@
|
||||
{
|
||||
"name": "Text to Video - Wan 2.2 Lightning",
|
||||
"author": "InvokeAI",
|
||||
"description": "Fast text-to-video generation with Wan 2.2 T2V A14B + the Lightning LoRA pair. Distillation lets the model converge in 4 steps with CFG 1.0 (no negative branch), roughly 20x faster than the standard T2V workflow at the cost of some image quality. Pick the high-noise Lightning LoRA on 'Apply LoRA (High)' and the low-noise one on 'Apply LoRA (Low)' — the 'auto' target routes each to the right expert when the LoRAs are tagged. Defaults: 832x480, 81 frames @ 16 FPS (~5 s).",
|
||||
"version": "1.0.0",
|
||||
"contact": "",
|
||||
"tags": "wan2.2, text to video, video, lightning, lora",
|
||||
"notes": "Each InvokeAI install assigns its own internal model IDs, so the model fields are left blank -- select these in the form after loading:\n\nMain Model node:\n- Main Model: Wan 2.2 T2V A14B High Noise (Q4_K_M GGUF recommended for <=16 GB VRAM)\n- Transformer (Low Noise): Wan 2.2 T2V A14B Low Noise (Q4_K_M)\n- VAE: Wan 2.2 A14B VAE\n- T5 Encoder: Wan T5 Encoder (UMT5-XXL)\n\nLightning LoRAs (4-step distillation - use the T2V variant, NOT I2V):\n- Apply Lightning LoRA (High Noise): Wan 2.2 T2V Lightning High Noise (4-step)\n- Apply Lightning LoRA (Low Noise): Wan 2.2 T2V Lightning Low Noise (4-step)\n\nThe Lightning LoRAs are A14B-specific and distinct for T2V vs I2V. If your LoRAs are untagged (no 'expert' field), use each LoRA loader's 'target' dropdown to route the first to 'high' and the second to 'low'. CFG=1.0 skips the negative-conditioning branch, so the Negative Prompt is ignored at runtime.",
|
||||
"exposedFields": [
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "transformer_low_noise_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "component_source"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "vae_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "wan_t5_encoder_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
|
||||
"fieldName": "lora"
|
||||
},
|
||||
{
|
||||
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
|
||||
"fieldName": "weight"
|
||||
},
|
||||
{
|
||||
"nodeId": "3466ad11-a931-4adc-b394-9ec287f0c23b",
|
||||
"fieldName": "target"
|
||||
},
|
||||
{
|
||||
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
|
||||
"fieldName": "lora"
|
||||
},
|
||||
{
|
||||
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
|
||||
"fieldName": "weight"
|
||||
},
|
||||
{
|
||||
"nodeId": "9b4cde65-699f-42c1-8889-76ac91fcb62f",
|
||||
"fieldName": "target"
|
||||
},
|
||||
{
|
||||
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "steps"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale_low_noise"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "width"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "height"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "num_frames"
|
||||
},
|
||||
{
|
||||
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"fieldName": "fps"
|
||||
}
|
||||
],
|
||||
"meta": {
|
||||
"version": "4.0.0",
|
||||
"category": "default"
|
||||
},
|
||||
"nodes": [
|
||||
{
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_model_loader",
|
||||
"inputs": {
|
||||
"model": {
|
||||
"name": "model",
|
||||
"label": "Transformer (High Noise)",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"transformer_low_noise_model": {
|
||||
"name": "transformer_low_noise_model",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"vae_model": {
|
||||
"name": "vae_model",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"wan_t5_encoder_model": {
|
||||
"name": "wan_t5_encoder_model",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"component_source": {
|
||||
"name": "component_source",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false
|
||||
},
|
||||
"position": {
|
||||
"x": 0,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "Positive Prompt",
|
||||
"notes": "",
|
||||
"type": "wan_text_encoder",
|
||||
"inputs": {
|
||||
"prompt": {
|
||||
"name": "prompt",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": ""
|
||||
},
|
||||
"wan_t5_encoder": {
|
||||
"name": "wan_t5_encoder",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 700,
|
||||
"y": 212
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "Negative Prompt (unused at CFG=1.0)",
|
||||
"notes": "",
|
||||
"type": "wan_text_encoder",
|
||||
"inputs": {
|
||||
"prompt": {
|
||||
"name": "prompt",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": " "
|
||||
},
|
||||
"wan_t5_encoder": {
|
||||
"name": "wan_t5_encoder",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 700,
|
||||
"y": 392
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "rand_int",
|
||||
"inputs": {
|
||||
"low": {
|
||||
"name": "low",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 0
|
||||
},
|
||||
"high": {
|
||||
"name": "high",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 2147483647
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false
|
||||
},
|
||||
"position": {
|
||||
"x": 700,
|
||||
"y": 572
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_video_denoise",
|
||||
"inputs": {
|
||||
"transformer": {
|
||||
"name": "transformer",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"positive_conditioning": {
|
||||
"name": "positive_conditioning",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"negative_conditioning": {
|
||||
"name": "negative_conditioning",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"ref_image": {
|
||||
"name": "ref_image",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"guidance_scale": {
|
||||
"name": "guidance_scale",
|
||||
"label": "CFG (High Noise)",
|
||||
"description": "",
|
||||
"value": 1
|
||||
},
|
||||
"guidance_scale_low_noise": {
|
||||
"name": "guidance_scale_low_noise",
|
||||
"label": "CFG (Low Noise)",
|
||||
"description": "",
|
||||
"value": 1
|
||||
},
|
||||
"width": {
|
||||
"name": "width",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 832
|
||||
},
|
||||
"height": {
|
||||
"name": "height",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 480
|
||||
},
|
||||
"num_frames": {
|
||||
"name": "num_frames",
|
||||
"label": "Frames",
|
||||
"description": "",
|
||||
"value": 81
|
||||
},
|
||||
"steps": {
|
||||
"name": "steps",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 4
|
||||
},
|
||||
"seed": {
|
||||
"name": "seed",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 0
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 1050,
|
||||
"y": 230
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_l2v",
|
||||
"inputs": {
|
||||
"board": {
|
||||
"name": "board",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"metadata": {
|
||||
"name": "metadata",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"latents": {
|
||||
"name": "latents",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"vae": {
|
||||
"name": "vae",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"fps": {
|
||||
"name": "fps",
|
||||
"label": "FPS",
|
||||
"description": "",
|
||||
"value": 16
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": false,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 1400,
|
||||
"y": 33
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "291abec8-b996-4c7b-959c-360f56a3ed60",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "291abec8-b996-4c7b-959c-360f56a3ed60",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "Apply Lightning LoRA (High Noise)",
|
||||
"notes": "",
|
||||
"type": "wan_lora_loader",
|
||||
"inputs": {
|
||||
"lora": {
|
||||
"name": "lora",
|
||||
"label": "Lightning LoRA (High Noise)",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"weight": {
|
||||
"name": "weight",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 1
|
||||
},
|
||||
"target": {
|
||||
"name": "target",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": "high"
|
||||
},
|
||||
"transformer": {
|
||||
"name": "transformer",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 350,
|
||||
"y": 772
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_lora_loader",
|
||||
"inputs": {
|
||||
"lora": {
|
||||
"name": "lora",
|
||||
"label": "Lightning LoRA (Low Noise)",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"weight": {
|
||||
"name": "weight",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 1
|
||||
},
|
||||
"target": {
|
||||
"name": "target",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": "low"
|
||||
},
|
||||
"transformer": {
|
||||
"name": "transformer",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 700,
|
||||
"y": 772
|
||||
}
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"id": "edge-loader-t5-pos",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-t5-neg",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-vae-l2v",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "vae",
|
||||
"targetHandle": "vae"
|
||||
},
|
||||
{
|
||||
"id": "edge-pos-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "positive_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-neg-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "negative_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-rand-seed-denoise",
|
||||
"type": "default",
|
||||
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "value",
|
||||
"targetHandle": "seed"
|
||||
},
|
||||
{
|
||||
"id": "edge-denoise-latents-l2v",
|
||||
"type": "default",
|
||||
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "latents",
|
||||
"targetHandle": "latents"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7etransformer-291abec8-b996-4c7b-959c-360f56a3ed60transformer",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "291abec8-b996-4c7b-959c-360f56a3ed60",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-291abec8-b996-4c7b-959c-360f56a3ed60transformer-ae8f115d-a3b8-4f76-b280-7313366a94adtransformer",
|
||||
"type": "default",
|
||||
"source": "291abec8-b996-4c7b-959c-360f56a3ed60",
|
||||
"target": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-ae8f115d-a3b8-4f76-b280-7313366a94adtransformer-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabtransformer",
|
||||
"type": "default",
|
||||
"source": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
}
|
||||
],
|
||||
"form": {
|
||||
"elements": {
|
||||
"root": {
|
||||
"id": "root",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"heading-XDPaQofdEE",
|
||||
"node-field-KcT0glAFDK",
|
||||
"container-i8V1oJeQCP",
|
||||
"container-mUeBcw2tYf",
|
||||
"heading-wuPtY8B4vR",
|
||||
"container-AkpUH1WtID",
|
||||
"heading-TwsoE3wQi9",
|
||||
"container-0u7wyxEdNa",
|
||||
"heading-10PuWMKJg9",
|
||||
"container-C0z6ru33to"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"node-field-zQA9XuRZRW": {
|
||||
"id": "node-field-zQA9XuRZRW",
|
||||
"parentId": "container-mUeBcw2tYf",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"fieldName": "fps"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-ok9GStueKC": {
|
||||
"id": "node-field-ok9GStueKC",
|
||||
"parentId": "container-mUeBcw2tYf",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "num_frames"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-58JRDH0dfF": {
|
||||
"id": "node-field-58JRDH0dfF",
|
||||
"parentId": "container-i8V1oJeQCP",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "height"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-jIfJ3RUDyT": {
|
||||
"id": "node-field-jIfJ3RUDyT",
|
||||
"parentId": "container-i8V1oJeQCP",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "width"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-QDqQ4XOKvg": {
|
||||
"id": "node-field-QDqQ4XOKvg",
|
||||
"parentId": "container-7g1Xedgg8T",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale_low_noise"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "float-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-PdU0Sz2m5z": {
|
||||
"id": "node-field-PdU0Sz2m5z",
|
||||
"parentId": "container-y0w7TfJC7e",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "float-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-UiQ7MaseDR": {
|
||||
"id": "node-field-UiQ7MaseDR",
|
||||
"parentId": "container-mUeBcw2tYf",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "steps"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-KcT0glAFDK": {
|
||||
"id": "node-field-KcT0glAFDK",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "string-field-config",
|
||||
"component": "textarea"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-N9mexEKkYg": {
|
||||
"id": "node-field-N9mexEKkYg",
|
||||
"parentId": "container-C0z6ru33to",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "wan_t5_encoder_model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-2u9lAbMBnD": {
|
||||
"id": "node-field-2u9lAbMBnD",
|
||||
"parentId": "container-C0z6ru33to",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "vae_model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-CTGqrO3x8o": {
|
||||
"id": "node-field-CTGqrO3x8o",
|
||||
"parentId": "container-C0z6ru33to",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "component_source"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-7G9srQT2Uw": {
|
||||
"id": "node-field-7G9srQT2Uw",
|
||||
"parentId": "container-7g1Xedgg8T",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "transformer_low_noise_model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-Q345WyNzc4": {
|
||||
"id": "node-field-Q345WyNzc4",
|
||||
"parentId": "container-y0w7TfJC7e",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"container-mUeBcw2tYf": {
|
||||
"id": "container-mUeBcw2tYf",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-UiQ7MaseDR",
|
||||
"node-field-ok9GStueKC",
|
||||
"node-field-zQA9XuRZRW"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"heading-XDPaQofdEE": {
|
||||
"id": "heading-XDPaQofdEE",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Generation"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"container-i8V1oJeQCP": {
|
||||
"id": "container-i8V1oJeQCP",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-jIfJ3RUDyT",
|
||||
"node-field-58JRDH0dfF"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"heading-wuPtY8B4vR": {
|
||||
"id": "heading-wuPtY8B4vR",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Model"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"container-AkpUH1WtID": {
|
||||
"id": "container-AkpUH1WtID",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-y0w7TfJC7e",
|
||||
"container-7g1Xedgg8T"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-y0w7TfJC7e": {
|
||||
"id": "container-y0w7TfJC7e",
|
||||
"parentId": "container-AkpUH1WtID",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"node-field-Q345WyNzc4",
|
||||
"node-field-PdU0Sz2m5z"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-7g1Xedgg8T": {
|
||||
"id": "container-7g1Xedgg8T",
|
||||
"parentId": "container-AkpUH1WtID",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"node-field-7G9srQT2Uw",
|
||||
"node-field-QDqQ4XOKvg"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-C0z6ru33to": {
|
||||
"id": "container-C0z6ru33to",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-N9mexEKkYg",
|
||||
"node-field-2u9lAbMBnD",
|
||||
"node-field-CTGqrO3x8o"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"heading-10PuWMKJg9": {
|
||||
"id": "heading-10PuWMKJg9",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Advanced"
|
||||
},
|
||||
"type": "heading"
|
||||
},
|
||||
"container-0u7wyxEdNa": {
|
||||
"id": "container-0u7wyxEdNa",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-Um9gcFv0lo",
|
||||
"container-4owWd0thWk"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-Um9gcFv0lo": {
|
||||
"id": "container-Um9gcFv0lo",
|
||||
"parentId": "container-0u7wyxEdNa",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-lF8o60KuVb"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-4owWd0thWk": {
|
||||
"id": "container-4owWd0thWk",
|
||||
"parentId": "container-0u7wyxEdNa",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-rvE0oy63qL"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"node-field-a8nZxgz9u9": {
|
||||
"id": "node-field-a8nZxgz9u9",
|
||||
"parentId": "container-lF8o60KuVb",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "291abec8-b996-4c7b-959c-360f56a3ed60",
|
||||
"fieldName": "lora"
|
||||
},
|
||||
"showDescription": true
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-NwR9yOugEz": {
|
||||
"id": "node-field-NwR9yOugEz",
|
||||
"parentId": "container-rvE0oy63qL",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
|
||||
"fieldName": "lora"
|
||||
},
|
||||
"showDescription": true
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"container-lF8o60KuVb": {
|
||||
"id": "container-lF8o60KuVb",
|
||||
"parentId": "container-Um9gcFv0lo",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"node-field-a8nZxgz9u9",
|
||||
"node-field-0al3LtpcGU"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"container-rvE0oy63qL": {
|
||||
"id": "container-rvE0oy63qL",
|
||||
"parentId": "container-4owWd0thWk",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"node-field-NwR9yOugEz",
|
||||
"node-field-lgLAjS4nks"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"node-field-0al3LtpcGU": {
|
||||
"id": "node-field-0al3LtpcGU",
|
||||
"parentId": "container-lF8o60KuVb",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "291abec8-b996-4c7b-959c-360f56a3ed60",
|
||||
"fieldName": "weight"
|
||||
},
|
||||
"showDescription": true,
|
||||
"settings": {
|
||||
"type": "float-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-lgLAjS4nks": {
|
||||
"id": "node-field-lgLAjS4nks",
|
||||
"parentId": "container-rvE0oy63qL",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "ae8f115d-a3b8-4f76-b280-7313366a94ad",
|
||||
"fieldName": "weight"
|
||||
},
|
||||
"showDescription": true,
|
||||
"settings": {
|
||||
"type": "float-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"heading-TwsoE3wQi9": {
|
||||
"id": "heading-TwsoE3wQi9",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"content": "Lightning LoRAs"
|
||||
},
|
||||
"type": "heading"
|
||||
}
|
||||
},
|
||||
"rootElementId": "root"
|
||||
},
|
||||
"id": "default_wan22_t2v_lightning"
|
||||
}
|
||||
+869
@@ -0,0 +1,869 @@
|
||||
{
|
||||
"name": "Text to Video - Wan 2.2 TI2V-5B (Low Quality)",
|
||||
"author": "InvokeAI",
|
||||
"description": "Text-to-video generation with Wan 2.2 TI2V-5B. Produces an MP4 of `num_frames` frames at `fps` frames-per-second (defaults: 81 frames @ 16 FPS = ~5 s, 720p ~1248x704). The output lands in the gallery alongside images and plays inline in the viewer.",
|
||||
"version": "1.0.0",
|
||||
"contact": "",
|
||||
"tags": "wan2.2, text to video, video",
|
||||
"notes": "This workflow needs the Wan 2.2 TI2V-5B model. Each InvokeAI install assigns its own internal model IDs, so the model fields are left blank -- select these in the form after loading:\n\nMain Model node:\n- Main Model: Wan 2.2 TI2V-5B (Q4_K_M GGUF recommended for <=16 GB VRAM)\n- Transformer (Low Noise): leave empty (TI2V-5B is a single-expert model)\n- VAE: Wan 2.2 TI2V-5B VAE\n- T5 Encoder: Wan T5 Encoder (UMT5-XXL)\n\nDefault output is 720p (~1248x704); reduce the \"Ideal Dimensions\" preset to 480p for less VRAM.",
|
||||
"exposedFields": [
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "transformer_low_noise_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "component_source"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "vae_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "wan_t5_encoder_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "steps"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale_low_noise"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "width"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "height"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "num_frames"
|
||||
},
|
||||
{
|
||||
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"fieldName": "fps"
|
||||
}
|
||||
],
|
||||
"meta": {
|
||||
"version": "4.0.0",
|
||||
"category": "default"
|
||||
},
|
||||
"nodes": [
|
||||
{
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_model_loader",
|
||||
"inputs": {
|
||||
"model": {
|
||||
"name": "model",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"transformer_low_noise_model": {
|
||||
"name": "transformer_low_noise_model",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"vae_model": {
|
||||
"name": "vae_model",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"wan_t5_encoder_model": {
|
||||
"name": "wan_t5_encoder_model",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": null
|
||||
},
|
||||
"component_source": {
|
||||
"name": "component_source",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false
|
||||
},
|
||||
"position": {
|
||||
"x": 200,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "Positive Prompt",
|
||||
"notes": "",
|
||||
"type": "wan_text_encoder",
|
||||
"inputs": {
|
||||
"prompt": {
|
||||
"name": "prompt",
|
||||
"label": "Positive Prompt",
|
||||
"description": "",
|
||||
"value": ""
|
||||
},
|
||||
"wan_t5_encoder": {
|
||||
"name": "wan_t5_encoder",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 700,
|
||||
"y": -200
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "Negative Prompt",
|
||||
"notes": "",
|
||||
"type": "wan_text_encoder",
|
||||
"inputs": {
|
||||
"prompt": {
|
||||
"name": "prompt",
|
||||
"label": "Negative Prompt",
|
||||
"description": "",
|
||||
"value": ""
|
||||
},
|
||||
"wan_t5_encoder": {
|
||||
"name": "wan_t5_encoder",
|
||||
"label": "",
|
||||
"description": ""
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 683.3585813389434,
|
||||
"y": 19.353124950263965
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"version": "1.0.1",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "rand_int",
|
||||
"inputs": {
|
||||
"low": {
|
||||
"name": "low",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 0
|
||||
},
|
||||
"high": {
|
||||
"name": "high",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 2147483647
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false
|
||||
},
|
||||
"position": {
|
||||
"x": 382.5329363121502,
|
||||
"y": 709.7864089212082
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_video_denoise",
|
||||
"inputs": {
|
||||
"transformer": {
|
||||
"name": "transformer",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"positive_conditioning": {
|
||||
"name": "positive_conditioning",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"negative_conditioning": {
|
||||
"name": "negative_conditioning",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"ref_image": {
|
||||
"name": "ref_image",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"guidance_scale": {
|
||||
"name": "guidance_scale",
|
||||
"label": "CFG",
|
||||
"description": "",
|
||||
"value": 5
|
||||
},
|
||||
"guidance_scale_low_noise": {
|
||||
"name": "guidance_scale_low_noise",
|
||||
"label": "CFG (Low)",
|
||||
"description": "",
|
||||
"value": 4
|
||||
},
|
||||
"width": {
|
||||
"name": "width",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 720
|
||||
},
|
||||
"height": {
|
||||
"name": "height",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 480
|
||||
},
|
||||
"num_frames": {
|
||||
"name": "num_frames",
|
||||
"label": "Frames",
|
||||
"description": "",
|
||||
"value": 81
|
||||
},
|
||||
"steps": {
|
||||
"name": "steps",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 20
|
||||
},
|
||||
"seed": {
|
||||
"name": "seed",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 0
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 1165.285565516453,
|
||||
"y": -39.7591269778113
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"version": "1.0.0",
|
||||
"nodePack": "invokeai",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"type": "wan_l2v",
|
||||
"inputs": {
|
||||
"board": {
|
||||
"name": "board",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"metadata": {
|
||||
"name": "metadata",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"latents": {
|
||||
"name": "latents",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"vae": {
|
||||
"name": "vae",
|
||||
"label": "",
|
||||
"description": ""
|
||||
},
|
||||
"fps": {
|
||||
"name": "fps",
|
||||
"label": "FPS",
|
||||
"description": "",
|
||||
"value": 16
|
||||
}
|
||||
},
|
||||
"isOpen": true,
|
||||
"isIntermediate": false,
|
||||
"useCache": true
|
||||
},
|
||||
"position": {
|
||||
"x": 1531.0370931112793,
|
||||
"y": -63.90613171839515
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "0cc328ea-b3bf-4ac1-98f3-844de0e072c8",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "0cc328ea-b3bf-4ac1-98f3-844de0e072c8",
|
||||
"type": "integer",
|
||||
"version": "1.0.1",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"nodePack": "invokeai",
|
||||
"inputs": {
|
||||
"value": {
|
||||
"name": "value",
|
||||
"label": "Width",
|
||||
"description": "Width of the video. Will be adjusted to meet the constraints of the selected target resolution.",
|
||||
"value": 832
|
||||
}
|
||||
}
|
||||
},
|
||||
"position": {
|
||||
"x": 370.6719737658655,
|
||||
"y": 436.76341584866606
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "e20a326b-a312-4cc4-89cf-cfc13887cd4a",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "e20a326b-a312-4cc4-89cf-cfc13887cd4a",
|
||||
"type": "integer",
|
||||
"version": "1.0.1",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"nodePack": "invokeai",
|
||||
"inputs": {
|
||||
"value": {
|
||||
"name": "value",
|
||||
"label": "Height",
|
||||
"description": "Height of the video. Will be adjusted to meet the constraints of the selected target resolution.",
|
||||
"value": 480
|
||||
}
|
||||
}
|
||||
},
|
||||
"position": {
|
||||
"x": 373.58106252303446,
|
||||
"y": 567.1240792266004
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
|
||||
"type": "wan_ti2v_ideal_dimensions",
|
||||
"version": "1.0.0",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"nodePack": "invokeai",
|
||||
"inputs": {
|
||||
"width": {
|
||||
"name": "width",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 1024
|
||||
},
|
||||
"height": {
|
||||
"name": "height",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": 480
|
||||
},
|
||||
"target_resolution": {
|
||||
"name": "target_resolution",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": "720p"
|
||||
},
|
||||
"rounding": {
|
||||
"name": "rounding",
|
||||
"label": "",
|
||||
"description": "",
|
||||
"value": "nearest"
|
||||
}
|
||||
}
|
||||
},
|
||||
"position": {
|
||||
"x": 773.8083794848407,
|
||||
"y": 420.1537431373451
|
||||
}
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"id": "edge-loader-transformer-denoise",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-t5-pos",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-t5-neg",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-vae-l2v",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "vae",
|
||||
"targetHandle": "vae"
|
||||
},
|
||||
{
|
||||
"id": "edge-pos-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "positive_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-neg-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "negative_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-rand-seed-denoise",
|
||||
"type": "default",
|
||||
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "value",
|
||||
"targetHandle": "seed"
|
||||
},
|
||||
{
|
||||
"id": "edge-denoise-latents-l2v",
|
||||
"type": "default",
|
||||
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "latents",
|
||||
"targetHandle": "latents"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-0cc328ea-b3bf-4ac1-98f3-844de0e072c8value-0f05c4e0-f92f-4b23-9a57-c69ccc50c2cdwidth",
|
||||
"type": "default",
|
||||
"source": "0cc328ea-b3bf-4ac1-98f3-844de0e072c8",
|
||||
"target": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
|
||||
"sourceHandle": "value",
|
||||
"targetHandle": "width"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-0f05c4e0-f92f-4b23-9a57-c69ccc50c2cdwidth-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabwidth",
|
||||
"type": "default",
|
||||
"source": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "width",
|
||||
"targetHandle": "width"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-e20a326b-a312-4cc4-89cf-cfc13887cd4avalue-0f05c4e0-f92f-4b23-9a57-c69ccc50c2cdheight",
|
||||
"type": "default",
|
||||
"source": "e20a326b-a312-4cc4-89cf-cfc13887cd4a",
|
||||
"target": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
|
||||
"sourceHandle": "value",
|
||||
"targetHandle": "height"
|
||||
},
|
||||
{
|
||||
"id": "reactflow__edge-0f05c4e0-f92f-4b23-9a57-c69ccc50c2cdheight-4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dabheight",
|
||||
"type": "default",
|
||||
"source": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "height",
|
||||
"targetHandle": "height"
|
||||
}
|
||||
],
|
||||
"form": {
|
||||
"elements": {
|
||||
"root": {
|
||||
"id": "root",
|
||||
"data": {
|
||||
"layout": "column",
|
||||
"children": [
|
||||
"heading-NVsTvi64mE",
|
||||
"node-field-LCSk79Cykw",
|
||||
"node-field-8eaiK9h1E6",
|
||||
"node-field-z8AjtazZ1C",
|
||||
"container-yTCxecjcVW",
|
||||
"container-tL8WPBUtAl",
|
||||
"container-WfDQ0EqgcT",
|
||||
"divider-dWeIfMmqLs",
|
||||
"heading-4KzzU0571E",
|
||||
"container-ZPG1DkfsBF",
|
||||
"container-LtEu7q4UHX"
|
||||
]
|
||||
},
|
||||
"type": "container"
|
||||
},
|
||||
"node-field-mz64Xfk59R": {
|
||||
"id": "node-field-mz64Xfk59R",
|
||||
"parentId": "container-tL8WPBUtAl",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"fieldName": "fps"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-dADhnNIONx": {
|
||||
"id": "node-field-dADhnNIONx",
|
||||
"parentId": "container-tL8WPBUtAl",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "num_frames"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-GM02XkVswM": {
|
||||
"id": "node-field-GM02XkVswM",
|
||||
"parentId": "container-WfDQ0EqgcT",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "float-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-RN6aZuubTt": {
|
||||
"id": "node-field-RN6aZuubTt",
|
||||
"parentId": "container-WfDQ0EqgcT",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "steps"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-8eaiK9h1E6": {
|
||||
"id": "node-field-8eaiK9h1E6",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "string-field-config",
|
||||
"component": "textarea"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-LCSk79Cykw": {
|
||||
"id": "node-field-LCSk79Cykw",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
"showDescription": false,
|
||||
"settings": {
|
||||
"type": "string-field-config",
|
||||
"component": "textarea"
|
||||
}
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-tmweV5koO8": {
|
||||
"id": "node-field-tmweV5koO8",
|
||||
"parentId": "container-YUjkEQOaZH",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "wan_t5_encoder_model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-Ny6XBv2KCR": {
|
||||
"id": "node-field-Ny6XBv2KCR",
|
||||
"parentId": "container-LtEu7q4UHX",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "vae_model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-zmxixZsriN": {
|
||||
"id": "node-field-zmxixZsriN",
|
||||
"parentId": "container-LtEu7q4UHX",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "component_source"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"node-field-FnSN39ojJs": {
|
||||
"id": "node-field-FnSN39ojJs",
|
||||
"parentId": "container-YUjkEQOaZH",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "model"
|
||||
},
|
||||
"showDescription": false
|
||||
},
|
||||
"type": "node-field"
|
||||
},
|
||||
"heading-NVsTvi64mE": {
|
||||
"id": "heading-NVsTvi64mE",
|
||||
"parentId": "root",
|
||||
"type": "heading",
|
||||
"data": {
|
||||
"content": "Generation"
|
||||
}
|
||||
},
|
||||
"container-WfDQ0EqgcT": {
|
||||
"id": "container-WfDQ0EqgcT",
|
||||
"parentId": "root",
|
||||
"type": "container",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-RN6aZuubTt",
|
||||
"node-field-GM02XkVswM"
|
||||
]
|
||||
}
|
||||
},
|
||||
"container-bOWWWaSIsY": {
|
||||
"id": "container-bOWWWaSIsY",
|
||||
"parentId": "container-yTCxecjcVW",
|
||||
"type": "container",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-41iwUAOdbd",
|
||||
"node-field-C8Yua39ccI"
|
||||
]
|
||||
}
|
||||
},
|
||||
"container-yTCxecjcVW": {
|
||||
"id": "container-yTCxecjcVW",
|
||||
"parentId": "root",
|
||||
"type": "container",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-bOWWWaSIsY"
|
||||
]
|
||||
}
|
||||
},
|
||||
"container-tL8WPBUtAl": {
|
||||
"id": "container-tL8WPBUtAl",
|
||||
"parentId": "root",
|
||||
"type": "container",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-dADhnNIONx",
|
||||
"node-field-mz64Xfk59R"
|
||||
]
|
||||
}
|
||||
},
|
||||
"heading-4KzzU0571E": {
|
||||
"id": "heading-4KzzU0571E",
|
||||
"parentId": "root",
|
||||
"type": "heading",
|
||||
"data": {
|
||||
"content": "Model"
|
||||
}
|
||||
},
|
||||
"divider-dWeIfMmqLs": {
|
||||
"id": "divider-dWeIfMmqLs",
|
||||
"parentId": "root",
|
||||
"type": "divider"
|
||||
},
|
||||
"container-YUjkEQOaZH": {
|
||||
"id": "container-YUjkEQOaZH",
|
||||
"parentId": "container-ZPG1DkfsBF",
|
||||
"type": "container",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-FnSN39ojJs",
|
||||
"node-field-tmweV5koO8"
|
||||
]
|
||||
}
|
||||
},
|
||||
"container-ZPG1DkfsBF": {
|
||||
"id": "container-ZPG1DkfsBF",
|
||||
"parentId": "root",
|
||||
"type": "container",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"container-YUjkEQOaZH"
|
||||
]
|
||||
}
|
||||
},
|
||||
"container-LtEu7q4UHX": {
|
||||
"id": "container-LtEu7q4UHX",
|
||||
"parentId": "root",
|
||||
"type": "container",
|
||||
"data": {
|
||||
"layout": "row",
|
||||
"children": [
|
||||
"node-field-Ny6XBv2KCR",
|
||||
"node-field-zmxixZsriN"
|
||||
]
|
||||
}
|
||||
},
|
||||
"node-field-z8AjtazZ1C": {
|
||||
"id": "node-field-z8AjtazZ1C",
|
||||
"type": "node-field",
|
||||
"parentId": "root",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "0f05c4e0-f92f-4b23-9a57-c69ccc50c2cd",
|
||||
"fieldName": "target_resolution"
|
||||
},
|
||||
"showDescription": true
|
||||
}
|
||||
},
|
||||
"node-field-41iwUAOdbd": {
|
||||
"id": "node-field-41iwUAOdbd",
|
||||
"type": "node-field",
|
||||
"parentId": "container-bOWWWaSIsY",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "0cc328ea-b3bf-4ac1-98f3-844de0e072c8",
|
||||
"fieldName": "value"
|
||||
},
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
},
|
||||
"showDescription": true
|
||||
}
|
||||
},
|
||||
"node-field-C8Yua39ccI": {
|
||||
"id": "node-field-C8Yua39ccI",
|
||||
"type": "node-field",
|
||||
"parentId": "container-bOWWWaSIsY",
|
||||
"data": {
|
||||
"fieldIdentifier": {
|
||||
"nodeId": "e20a326b-a312-4cc4-89cf-cfc13887cd4a",
|
||||
"fieldName": "value"
|
||||
},
|
||||
"settings": {
|
||||
"type": "integer-field-config",
|
||||
"showShuffle": false,
|
||||
"component": "number-input"
|
||||
},
|
||||
"showDescription": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"rootElementId": "root"
|
||||
},
|
||||
"id": "default_wan22_t2v_ti2v5b"
|
||||
}
|
||||
@@ -0,0 +1,307 @@
|
||||
{
|
||||
"id": "default_wan22_i2v_c2d5e1b3-3e4f-5b6c-af7d-8e9f0a1b2c3e",
|
||||
"name": "Image to Image - Wan 2.2",
|
||||
"author": "InvokeAI",
|
||||
"description": "Image-to-image generation with Wan 2.2 I2V A14B. The reference image is VAE-encoded and concatenated to the noise latents each step (the I2V transformer has in_channels=36). Drop a reference image into the 'Reference Image' input, then invoke. Only the I2V A14B variant is supported — T2V and TI2V-5B don't consume reference images.",
|
||||
"version": "1.0.0",
|
||||
"contact": "",
|
||||
"tags": "wan2.2, image to image",
|
||||
"notes": "Prerequisite model downloads: a Wan 2.2 I2V A14B main (Diffusers or GGUF expert pair). For GGUF mains, also install the Component Source (Diffusers Wan I2V) OR the standalone Wan VAE + UMT5-XXL encoder. Wan 2.2 I2V was trained for video — at single-frame inference it tends to anchor strongly to the reference. Recommended settings: 30-40 steps and CFG 5-7 (or 4 steps and CFG 1 with the Wan I2V Lightning LoRA pair).",
|
||||
"exposedFields": [
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "transformer_low_noise_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "component_source"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "vae_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "wan_t5_encoder_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "7a6edc2d-f38e-a0c1-e27a-8f9dcbb20fce",
|
||||
"fieldName": "image"
|
||||
},
|
||||
{
|
||||
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "steps"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale_low_noise"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "width"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "height"
|
||||
}
|
||||
],
|
||||
"meta": {
|
||||
"version": "3.0.0",
|
||||
"category": "default"
|
||||
},
|
||||
"nodes": [
|
||||
{
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"type": "wan_model_loader",
|
||||
"version": "1.0.1",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false,
|
||||
"inputs": {
|
||||
"model": { "name": "model", "label": "" },
|
||||
"transformer_low_noise_model": { "name": "transformer_low_noise_model", "label": "" },
|
||||
"vae_model": { "name": "vae_model", "label": "" },
|
||||
"wan_t5_encoder_model": { "name": "wan_t5_encoder_model", "label": "" },
|
||||
"component_source": { "name": "component_source", "label": "" }
|
||||
}
|
||||
},
|
||||
"position": { "x": 200, "y": 0 }
|
||||
},
|
||||
{
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"type": "wan_text_encoder",
|
||||
"version": "1.0.0",
|
||||
"label": "Positive Prompt",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"inputs": {
|
||||
"prompt": { "name": "prompt", "label": "", "value": "" },
|
||||
"wan_t5_encoder": { "name": "wan_t5_encoder", "label": "" }
|
||||
}
|
||||
},
|
||||
"position": { "x": 700, "y": -200 }
|
||||
},
|
||||
{
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"type": "wan_text_encoder",
|
||||
"version": "1.0.0",
|
||||
"label": "Negative Prompt",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"inputs": {
|
||||
"prompt": { "name": "prompt", "label": "", "value": " " },
|
||||
"wan_t5_encoder": { "name": "wan_t5_encoder", "label": "" }
|
||||
}
|
||||
},
|
||||
"position": { "x": 700, "y": 100 }
|
||||
},
|
||||
{
|
||||
"id": "7a6edc2d-f38e-a0c1-e27a-8f9dcbb20fce",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "7a6edc2d-f38e-a0c1-e27a-8f9dcbb20fce",
|
||||
"type": "wan_ref_image_encoder",
|
||||
"version": "1.2.0",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"inputs": {
|
||||
"image": { "name": "image", "label": "Reference Image" },
|
||||
"end_image": { "name": "end_image", "label": "" },
|
||||
"num_frames": { "name": "num_frames", "label": "", "value": 1 },
|
||||
"vae": { "name": "vae", "label": "" },
|
||||
"width": { "name": "width", "label": "", "value": 1024 },
|
||||
"height": { "name": "height", "label": "", "value": 1024 }
|
||||
}
|
||||
},
|
||||
"position": { "x": 700, "y": 300 }
|
||||
},
|
||||
{
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"type": "rand_int",
|
||||
"version": "1.0.1",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false,
|
||||
"inputs": {
|
||||
"low": { "name": "low", "label": "", "value": 0 },
|
||||
"high": { "name": "high", "label": "", "value": 2147483647 }
|
||||
}
|
||||
},
|
||||
"position": { "x": 700, "y": 550 }
|
||||
},
|
||||
{
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"type": "wan_denoise",
|
||||
"version": "1.0.0",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"inputs": {
|
||||
"transformer": { "name": "transformer", "label": "" },
|
||||
"positive_conditioning": { "name": "positive_conditioning", "label": "" },
|
||||
"negative_conditioning": { "name": "negative_conditioning", "label": "" },
|
||||
"ref_image": { "name": "ref_image", "label": "" },
|
||||
"latents": { "name": "latents", "label": "" },
|
||||
"denoise_mask": { "name": "denoise_mask", "label": "" },
|
||||
"denoising_start": { "name": "denoising_start", "label": "", "value": 0 },
|
||||
"denoising_end": { "name": "denoising_end", "label": "", "value": 1 },
|
||||
"add_noise": { "name": "add_noise", "label": "", "value": true },
|
||||
"guidance_scale": { "name": "guidance_scale", "label": "CFG", "value": 5.0 },
|
||||
"guidance_scale_low_noise": { "name": "guidance_scale_low_noise", "label": "CFG (Low)" },
|
||||
"width": { "name": "width", "label": "", "value": 1024 },
|
||||
"height": { "name": "height", "label": "", "value": 1024 },
|
||||
"steps": { "name": "steps", "label": "", "value": 30 },
|
||||
"seed": { "name": "seed", "label": "", "value": 0 }
|
||||
}
|
||||
},
|
||||
"position": { "x": 1100, "y": -50 }
|
||||
},
|
||||
{
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"type": "wan_l2i",
|
||||
"version": "1.0.0",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": false,
|
||||
"useCache": true,
|
||||
"inputs": {
|
||||
"board": { "name": "board", "label": "" },
|
||||
"metadata": { "name": "metadata", "label": "" },
|
||||
"latents": { "name": "latents", "label": "" },
|
||||
"vae": { "name": "vae", "label": "" }
|
||||
}
|
||||
},
|
||||
"position": { "x": 1550, "y": -50 }
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"id": "edge-loader-transformer-denoise",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-t5-pos",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-t5-neg",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-vae-l2i",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "vae",
|
||||
"targetHandle": "vae"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-vae-refenc",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "7a6edc2d-f38e-a0c1-e27a-8f9dcbb20fce",
|
||||
"sourceHandle": "vae",
|
||||
"targetHandle": "vae"
|
||||
},
|
||||
{
|
||||
"id": "edge-pos-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "positive_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-neg-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "negative_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-refenc-refimage-denoise",
|
||||
"type": "default",
|
||||
"source": "7a6edc2d-f38e-a0c1-e27a-8f9dcbb20fce",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "ref_image",
|
||||
"targetHandle": "ref_image"
|
||||
},
|
||||
{
|
||||
"id": "edge-rand-seed-denoise",
|
||||
"type": "default",
|
||||
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "value",
|
||||
"targetHandle": "seed"
|
||||
},
|
||||
{
|
||||
"id": "edge-denoise-latents-l2i",
|
||||
"type": "default",
|
||||
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "latents",
|
||||
"targetHandle": "latents"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
{
|
||||
"id": "default_wan22_t2v_b1c4f0a2-2d3e-4a5b-9f6c-7d8e0a1b2c3d",
|
||||
"name": "Text to Image - Wan 2.2",
|
||||
"author": "InvokeAI",
|
||||
"description": "Text-to-image generation with Wan 2.2 (T2V A14B or TI2V-5B). For A14B GGUFs, wire the second-expert transformer into 'Transformer (Low Noise)' and pick a Diffusers Wan as the Component Source (or use standalone VAE + UMT5-XXL encoder). TI2V-5B is a single-transformer model — leave the low-noise slot empty.",
|
||||
"version": "1.0.0",
|
||||
"contact": "",
|
||||
"tags": "wan2.2, text to image",
|
||||
"notes": "Prerequisite model downloads: a Wan 2.2 main model (Diffusers or GGUF). For GGUF mains, also install the Component Source (Diffusers Wan) OR the standalone Wan VAE + UMT5-XXL encoder. The Wan 2.2 starter bundle in the Model Manager pulls everything you need for T2V A14B Q4_K_M/Q8_0. Recommended settings: 30-40 steps and CFG 5-7 (or 4 steps and CFG 1 with the Wan Lightning LoRA pair).",
|
||||
"exposedFields": [
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "transformer_low_noise_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "component_source"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "vae_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"fieldName": "wan_t5_encoder_model"
|
||||
},
|
||||
{
|
||||
"nodeId": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"fieldName": "prompt"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "steps"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "guidance_scale_low_noise"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "width"
|
||||
},
|
||||
{
|
||||
"nodeId": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"fieldName": "height"
|
||||
}
|
||||
],
|
||||
"meta": {
|
||||
"version": "3.0.0",
|
||||
"category": "default"
|
||||
},
|
||||
"nodes": [
|
||||
{
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"type": "wan_model_loader",
|
||||
"version": "1.0.1",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false,
|
||||
"inputs": {
|
||||
"model": { "name": "model", "label": "" },
|
||||
"transformer_low_noise_model": { "name": "transformer_low_noise_model", "label": "" },
|
||||
"vae_model": { "name": "vae_model", "label": "" },
|
||||
"wan_t5_encoder_model": { "name": "wan_t5_encoder_model", "label": "" },
|
||||
"component_source": { "name": "component_source", "label": "" }
|
||||
}
|
||||
},
|
||||
"position": { "x": 200, "y": 0 }
|
||||
},
|
||||
{
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"type": "wan_text_encoder",
|
||||
"version": "1.0.0",
|
||||
"label": "Positive Prompt",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"inputs": {
|
||||
"prompt": { "name": "prompt", "label": "", "value": "a cat" },
|
||||
"wan_t5_encoder": { "name": "wan_t5_encoder", "label": "" }
|
||||
}
|
||||
},
|
||||
"position": { "x": 700, "y": -200 }
|
||||
},
|
||||
{
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"type": "wan_text_encoder",
|
||||
"version": "1.0.0",
|
||||
"label": "Negative Prompt",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"inputs": {
|
||||
"prompt": { "name": "prompt", "label": "", "value": " " },
|
||||
"wan_t5_encoder": { "name": "wan_t5_encoder", "label": "" }
|
||||
}
|
||||
},
|
||||
"position": { "x": 700, "y": 100 }
|
||||
},
|
||||
{
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"type": "rand_int",
|
||||
"version": "1.0.1",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": false,
|
||||
"inputs": {
|
||||
"low": { "name": "low", "label": "", "value": 0 },
|
||||
"high": { "name": "high", "label": "", "value": 2147483647 }
|
||||
}
|
||||
},
|
||||
"position": { "x": 700, "y": 400 }
|
||||
},
|
||||
{
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"type": "wan_denoise",
|
||||
"version": "1.0.0",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": true,
|
||||
"useCache": true,
|
||||
"inputs": {
|
||||
"transformer": { "name": "transformer", "label": "" },
|
||||
"positive_conditioning": { "name": "positive_conditioning", "label": "" },
|
||||
"negative_conditioning": { "name": "negative_conditioning", "label": "" },
|
||||
"ref_image": { "name": "ref_image", "label": "" },
|
||||
"latents": { "name": "latents", "label": "" },
|
||||
"denoise_mask": { "name": "denoise_mask", "label": "" },
|
||||
"denoising_start": { "name": "denoising_start", "label": "", "value": 0 },
|
||||
"denoising_end": { "name": "denoising_end", "label": "", "value": 1 },
|
||||
"add_noise": { "name": "add_noise", "label": "", "value": true },
|
||||
"guidance_scale": { "name": "guidance_scale", "label": "CFG", "value": 5.0 },
|
||||
"guidance_scale_low_noise": { "name": "guidance_scale_low_noise", "label": "CFG (Low)" },
|
||||
"width": { "name": "width", "label": "", "value": 1024 },
|
||||
"height": { "name": "height", "label": "", "value": 1024 },
|
||||
"steps": { "name": "steps", "label": "", "value": 30 },
|
||||
"seed": { "name": "seed", "label": "", "value": 0 }
|
||||
}
|
||||
},
|
||||
"position": { "x": 1100, "y": -50 }
|
||||
},
|
||||
{
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"type": "invocation",
|
||||
"data": {
|
||||
"id": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"type": "wan_l2i",
|
||||
"version": "1.0.0",
|
||||
"label": "",
|
||||
"notes": "",
|
||||
"isOpen": true,
|
||||
"isIntermediate": false,
|
||||
"useCache": true,
|
||||
"inputs": {
|
||||
"board": { "name": "board", "label": "" },
|
||||
"metadata": { "name": "metadata", "label": "" },
|
||||
"latents": { "name": "latents", "label": "" },
|
||||
"vae": { "name": "vae", "label": "" }
|
||||
}
|
||||
},
|
||||
"position": { "x": 1550, "y": -50 }
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"id": "edge-loader-transformer-denoise",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "transformer",
|
||||
"targetHandle": "transformer"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-t5-pos",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-t5-neg",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"sourceHandle": "wan_t5_encoder",
|
||||
"targetHandle": "wan_t5_encoder"
|
||||
},
|
||||
{
|
||||
"id": "edge-loader-vae-l2i",
|
||||
"type": "default",
|
||||
"source": "1a0e6c7b-9d2f-4b3c-8e1a-2f3d4c5b6a7e",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "vae",
|
||||
"targetHandle": "vae"
|
||||
},
|
||||
{
|
||||
"id": "edge-pos-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "2b1f7d8c-ae3f-5c4d-9f2b-3a4e5d6c7b8f",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "positive_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-neg-cond-denoise",
|
||||
"type": "default",
|
||||
"source": "3c2a8e9d-bf4a-6d5e-af3c-4b5f6e7d8c9a",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "conditioning",
|
||||
"targetHandle": "negative_conditioning"
|
||||
},
|
||||
{
|
||||
"id": "edge-rand-seed-denoise",
|
||||
"type": "default",
|
||||
"source": "5e4cab0b-d16c-8faf-c05e-6d7baf90ebbc",
|
||||
"target": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"sourceHandle": "value",
|
||||
"targetHandle": "seed"
|
||||
},
|
||||
{
|
||||
"id": "edge-denoise-latents-l2i",
|
||||
"type": "default",
|
||||
"source": "4d3b9faf-c05b-7e6f-bf4d-5c6a7f8e9dab",
|
||||
"target": "6f5dcb1c-e27d-9fb0-d16f-7e8cbaa1fcbd",
|
||||
"sourceHandle": "latents",
|
||||
"targetHandle": "latents"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -179,6 +179,84 @@ ANIMA_LATENT_RGB_FACTORS = [
|
||||
|
||||
ANIMA_LATENT_RGB_BIAS = [-0.1835, -0.0868, -0.3360]
|
||||
|
||||
# Wan 2.2 A14B uses the standard 16-channel Wan VAE.
|
||||
# Factors come from ComfyUI's Wan21 latent_format (same VAE as A14B).
|
||||
WAN_LATENT_RGB_FACTORS = [
|
||||
[-0.1299, -0.1692, 0.2932],
|
||||
[0.0671, 0.0406, 0.0442],
|
||||
[0.3568, 0.2548, 0.1747],
|
||||
[0.0372, 0.2344, 0.1420],
|
||||
[0.0313, 0.0189, -0.0328],
|
||||
[0.0296, -0.0956, -0.0665],
|
||||
[-0.3477, -0.4059, -0.2925],
|
||||
[0.0166, 0.1902, 0.1975],
|
||||
[-0.0412, 0.0267, -0.1364],
|
||||
[-0.1293, 0.0740, 0.1636],
|
||||
[0.0680, 0.3019, 0.1128],
|
||||
[0.0032, 0.0581, 0.0639],
|
||||
[-0.1251, 0.0927, 0.1699],
|
||||
[0.0060, -0.0633, 0.0005],
|
||||
[0.3477, 0.2275, 0.2950],
|
||||
[0.1984, 0.0913, 0.1861],
|
||||
]
|
||||
|
||||
WAN_LATENT_RGB_BIAS = [-0.1835, -0.0868, -0.3360]
|
||||
|
||||
# Wan 2.2 TI2V-5B uses Wan2.2-VAE with 48 latent channels and 16x spatial downscale.
|
||||
# Factors come from ComfyUI's Wan22 latent_format.
|
||||
WAN22_LATENT_RGB_FACTORS = [
|
||||
[0.0119, 0.0103, 0.0046],
|
||||
[-0.1062, -0.0504, 0.0165],
|
||||
[0.0140, 0.0409, 0.0491],
|
||||
[-0.0813, -0.0677, 0.0607],
|
||||
[0.0656, 0.0851, 0.0808],
|
||||
[0.0264, 0.0463, 0.0912],
|
||||
[0.0295, 0.0326, 0.0590],
|
||||
[-0.0244, -0.0270, 0.0025],
|
||||
[0.0443, -0.0102, 0.0288],
|
||||
[-0.0465, -0.0090, -0.0205],
|
||||
[0.0359, 0.0236, 0.0082],
|
||||
[-0.0776, 0.0854, 0.1048],
|
||||
[0.0564, 0.0264, 0.0561],
|
||||
[0.0006, 0.0594, 0.0418],
|
||||
[-0.0319, -0.0542, -0.0637],
|
||||
[-0.0268, 0.0024, 0.0260],
|
||||
[0.0539, 0.0265, 0.0358],
|
||||
[-0.0359, -0.0312, -0.0287],
|
||||
[-0.0285, -0.1032, -0.1237],
|
||||
[0.1041, 0.0537, 0.0622],
|
||||
[-0.0086, -0.0374, -0.0051],
|
||||
[0.0390, 0.0670, 0.2863],
|
||||
[0.0069, 0.0144, 0.0082],
|
||||
[0.0006, -0.0167, 0.0079],
|
||||
[0.0313, -0.0574, -0.0232],
|
||||
[-0.1454, -0.0902, -0.0481],
|
||||
[0.0714, 0.0827, 0.0447],
|
||||
[-0.0304, -0.0574, -0.0196],
|
||||
[0.0401, 0.0384, 0.0204],
|
||||
[-0.0758, -0.0297, -0.0014],
|
||||
[0.0568, 0.1307, 0.1372],
|
||||
[-0.0055, -0.0310, -0.0380],
|
||||
[0.0239, -0.0305, 0.0325],
|
||||
[-0.0663, -0.0673, -0.0140],
|
||||
[-0.0416, -0.0047, -0.0023],
|
||||
[0.0166, 0.0112, -0.0093],
|
||||
[-0.0211, 0.0011, 0.0331],
|
||||
[0.1833, 0.1466, 0.2250],
|
||||
[-0.0368, 0.0370, 0.0295],
|
||||
[-0.3441, -0.3543, -0.2008],
|
||||
[-0.0479, -0.0489, -0.0420],
|
||||
[-0.0660, -0.0153, 0.0800],
|
||||
[-0.0101, 0.0068, 0.0156],
|
||||
[-0.0690, -0.0452, -0.0927],
|
||||
[-0.0145, 0.0041, 0.0015],
|
||||
[0.0421, 0.0451, 0.0373],
|
||||
[0.0504, -0.0483, -0.0356],
|
||||
[-0.0837, 0.0168, 0.0055],
|
||||
]
|
||||
|
||||
WAN22_LATENT_RGB_BIAS = [0.0317, -0.0878, -0.1388]
|
||||
|
||||
|
||||
def sample_to_lowres_estimated_image(
|
||||
samples: torch.Tensor,
|
||||
@@ -270,6 +348,15 @@ def diffusion_step_callback(
|
||||
# Anima uses Wan 2.1 VAE with 16 latent channels
|
||||
latent_rgb_factors = ANIMA_LATENT_RGB_FACTORS
|
||||
latent_rgb_bias = ANIMA_LATENT_RGB_BIAS
|
||||
elif base_model == BaseModelType.Wan:
|
||||
# A14B (16-ch standard Wan VAE, 8x spatial) vs TI2V-5B (48-ch Wan2.2-VAE,
|
||||
# 16x spatial). The latent channel count uniquely identifies the variant.
|
||||
if sample.shape[-3] == 48:
|
||||
latent_rgb_factors = WAN22_LATENT_RGB_FACTORS
|
||||
latent_rgb_bias = WAN22_LATENT_RGB_BIAS
|
||||
else:
|
||||
latent_rgb_factors = WAN_LATENT_RGB_FACTORS
|
||||
latent_rgb_bias = WAN_LATENT_RGB_BIAS
|
||||
else:
|
||||
raise ValueError(f"Unsupported base model: {base_model}")
|
||||
|
||||
@@ -287,8 +374,13 @@ def diffusion_step_callback(
|
||||
latent_rgb_bias=latent_rgb_bias_torch,
|
||||
)
|
||||
|
||||
width = image.width * 8
|
||||
height = image.height * 8
|
||||
# Spatial downscale ratio: 8x is the SD/SDXL/FLUX/Wan-A14B default;
|
||||
# Wan TI2V-5B's Wan2.2-VAE uses 16x.
|
||||
spatial_scale = 8
|
||||
if base_model == BaseModelType.Wan and sample.shape[-3] == 48:
|
||||
spatial_scale = 16
|
||||
width = image.width * spatial_scale
|
||||
height = image.height * spatial_scale
|
||||
percentage = calc_percentage(intermediate_state)
|
||||
|
||||
signal_progress("Denoising", percentage, image, (width, height))
|
||||
|
||||
@@ -0,0 +1,288 @@
|
||||
"""Standalone decode worker for untrusted video files.
|
||||
|
||||
This script is executed as a short-lived child process by
|
||||
``invokeai.app.util.video_thumbnails`` — never imported by the server at runtime — so
|
||||
that a decoder hang on a crafted or malformed container can be bounded by a timeout and
|
||||
killed. cv2 wheels have historically hung on certain codec/container combinations, and
|
||||
a hung *thread* cannot be killed from Python, so process isolation is the only reliable
|
||||
bound. It is run by file path (not ``-m``) and deliberately imports only the stdlib plus
|
||||
imageio/PIL/cv2, so it starts quickly without pulling in the invokeai package or torch.
|
||||
|
||||
Protocol: ``python video_decode_worker.py <command> <args...>``. On success a single
|
||||
JSON object is written to stdout and the exit code is 0; on any failure the exit code is
|
||||
non-zero with a message on stderr.
|
||||
|
||||
Commands:
|
||||
probe <video_path> -> {"width", "height", "duration", "fps", "codec"}
|
||||
frame <video_path> <index> <out_path> -> {"ok": true}; the frame is written to out_path as PNG
|
||||
count <video_path> -> {"count": <int or null>}
|
||||
stream <video_path> -> consecutive numpy arrays on stdout
|
||||
"""
|
||||
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import struct
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import imageio.v3 as iio
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
# Upper bound on the decoded frame size we are willing to allocate (~8K video). A small
|
||||
# crafted container can claim absurd dimensions in cheap metadata while the full frame
|
||||
# is only allocated at decode time, so dimensions are checked before any decode.
|
||||
# Keep in sync with MAX_VIDEO_FRAME_PIXELS in video_thumbnails.py (this script must not
|
||||
# import the invokeai package).
|
||||
MAX_FRAME_PIXELS = 64 * 1024 * 1024
|
||||
# Address-space headroom for everything allocated after startup. Must accommodate a
|
||||
# legal max-size decode, not just a typical one: a MAX_FRAME_PIXELS RGB frame is
|
||||
# 192 MiB and the emit path holds ~4 frame-sized copies at once (~768 MiB), on top of
|
||||
# which RLIMIT_AS also counts the lazily-imported cv2's library mappings, glibc
|
||||
# per-thread arenas, and the codec's reference-frame buffers — address space routinely
|
||||
# exceeds resident memory by a wide margin. Keep in sync with
|
||||
# MAX_VIDEO_DECODE_WORKER_RSS_BYTES in video_thumbnails.py (the parent-side resident
|
||||
# bound; this script must not import the invokeai package).
|
||||
WORKER_MEMORY_HEADROOM_BYTES = 4 * 1024 * 1024 * 1024
|
||||
|
||||
|
||||
def _limit_worker_memory() -> None:
|
||||
"""Bound allocations made after worker startup on Linux."""
|
||||
if sys.platform != "linux":
|
||||
return
|
||||
try:
|
||||
import os
|
||||
import resource
|
||||
|
||||
pages = int(Path("/proc/self/statm").read_text().split()[0])
|
||||
limit = pages * os.sysconf("SC_PAGE_SIZE") + WORKER_MEMORY_HEADROOM_BYTES
|
||||
_soft, hard = resource.getrlimit(resource.RLIMIT_AS)
|
||||
if hard != resource.RLIM_INFINITY:
|
||||
limit = min(limit, hard)
|
||||
resource.setrlimit(resource.RLIMIT_AS, (limit, hard))
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def _validate_decoded_frame(frame: np.ndarray) -> None:
|
||||
if frame.ndim != 3 or frame.shape[2] != 3:
|
||||
raise ValueError(f"Decoded frame must be RGB; got shape {frame.shape}")
|
||||
height, width = frame.shape[:2]
|
||||
if height <= 0 or width <= 0 or height * width > MAX_FRAME_PIXELS:
|
||||
raise ValueError(f"Decoded frame dimensions {width}x{height} exceed the maximum decodable size")
|
||||
|
||||
|
||||
def _assert_decodable_dims(video_path: Path) -> None:
|
||||
"""Refuses to decode frames from a video whose reported dimensions exceed the bound.
|
||||
|
||||
If the dimensions cannot be probed, decoding is refused because the parent's frame
|
||||
record bound and PIL checks run only after the decoder has allocated the frame.
|
||||
"""
|
||||
try:
|
||||
width, height, _duration, _fps = _probe(video_path)[:4]
|
||||
except Exception as error:
|
||||
raise ValueError(f"Unable to validate video dimensions for {video_path}") from error
|
||||
if width <= 0 or height <= 0:
|
||||
raise ValueError(f"Video reports invalid dimensions {width}x{height}")
|
||||
if width * height > MAX_FRAME_PIXELS:
|
||||
raise ValueError(f"Video dimensions {width}x{height} exceed the maximum decodable size")
|
||||
|
||||
|
||||
def _extract_frame(video_path: Path, frame_index: int) -> Optional[Image.Image]:
|
||||
"""Extracts a single frame from a video file as a PIL Image. Returns None on failure.
|
||||
|
||||
Tries imageio's FFMPEG plugin first since it's the same encoder we use for output,
|
||||
then falls back to cv2 — uploaded videos with unusual codecs may need that path.
|
||||
"""
|
||||
try:
|
||||
# iio.imread with index=N seeks to that frame directly. Returns RGB HxWxC uint8.
|
||||
frame = iio.imread(video_path, plugin="FFMPEG", index=frame_index)
|
||||
_validate_decoded_frame(frame)
|
||||
return Image.fromarray(frame)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
import cv2 # local import so the imageio-only path doesn't pay the cv2 import cost
|
||||
|
||||
capture = cv2.VideoCapture(str(video_path))
|
||||
if not capture.isOpened():
|
||||
capture.release()
|
||||
return None
|
||||
try:
|
||||
if frame_index > 0:
|
||||
capture.set(cv2.CAP_PROP_POS_FRAMES, frame_index)
|
||||
ok, frame_bgr = capture.read()
|
||||
if not ok or frame_bgr is None:
|
||||
return None
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
_validate_decoded_frame(frame_rgb)
|
||||
return Image.fromarray(frame_rgb)
|
||||
finally:
|
||||
capture.release()
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _probe(video_path: Path) -> tuple[int, int, float, Optional[float], Optional[str]]:
|
||||
"""Returns (width, height, duration_seconds, fps_or_none, codec_or_none) for a video file.
|
||||
|
||||
Tries imageio's FFMPEG plugin first; falls back to cv2.VideoCapture. Raises if
|
||||
neither backend can read the file.
|
||||
"""
|
||||
try:
|
||||
meta = iio.immeta(video_path, plugin="FFMPEG")
|
||||
fps_raw = meta.get("fps")
|
||||
duration = float(meta.get("duration", 0.0)) if meta.get("duration") is not None else 0.0
|
||||
size = meta.get("size")
|
||||
if size is None:
|
||||
# Fall through to cv2 — imageio didn't give us dimensions.
|
||||
raise ValueError("imageio probe missing 'size'")
|
||||
width, height = int(size[0]), int(size[1])
|
||||
fps: Optional[float] = float(fps_raw) if fps_raw and fps_raw > 0 else None
|
||||
codec_raw = meta.get("codec")
|
||||
codec = str(codec_raw).lower() if codec_raw else None
|
||||
return width, height, duration, fps, codec
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
import cv2
|
||||
|
||||
capture = cv2.VideoCapture(str(video_path))
|
||||
if not capture.isOpened():
|
||||
capture.release()
|
||||
raise FileNotFoundError(f"Unable to open video at {video_path}")
|
||||
try:
|
||||
width = int(capture.get(cv2.CAP_PROP_FRAME_WIDTH))
|
||||
height = int(capture.get(cv2.CAP_PROP_FRAME_HEIGHT))
|
||||
frame_count = int(capture.get(cv2.CAP_PROP_FRAME_COUNT))
|
||||
fps_raw = capture.get(cv2.CAP_PROP_FPS)
|
||||
fps_v2: Optional[float] = float(fps_raw) if fps_raw and fps_raw > 0 else None
|
||||
duration = (frame_count / fps_v2) if (fps_v2 and frame_count > 0) else 0.0
|
||||
fourcc = int(capture.get(cv2.CAP_PROP_FOURCC))
|
||||
codec = "".join(chr((fourcc >> (8 * index)) & 0xFF) for index in range(4)).strip().lower() or None
|
||||
finally:
|
||||
capture.release()
|
||||
return width, height, duration, fps_v2, codec
|
||||
|
||||
|
||||
def _count(video_path: Path) -> Optional[int]:
|
||||
"""Return the exact decoded frame count, or None if neither backend can determine it.
|
||||
|
||||
Tries imageio's improps first (works for a handful of codecs that expose nframes in
|
||||
container metadata). For libx264 streams imageio reports ``inf``, so we fall through
|
||||
to cv2's ``CAP_PROP_FRAME_COUNT`` which reads the actual packet count. Both sources
|
||||
are preferred over a ``duration * fps`` estimate, which can overshoot by one on VFR
|
||||
uploads or containers with imprecise metadata.
|
||||
"""
|
||||
try:
|
||||
props = iio.improps(video_path, plugin="FFMPEG")
|
||||
except Exception:
|
||||
props = None
|
||||
shape = getattr(props, "shape", None) if props is not None else None
|
||||
if shape:
|
||||
n = shape[0]
|
||||
if not (isinstance(n, float) and not math.isfinite(n)):
|
||||
try:
|
||||
return int(n)
|
||||
except (TypeError, ValueError, OverflowError):
|
||||
pass
|
||||
|
||||
try:
|
||||
import cv2
|
||||
|
||||
capture = cv2.VideoCapture(str(video_path))
|
||||
if not capture.isOpened():
|
||||
capture.release()
|
||||
return None
|
||||
try:
|
||||
count = int(capture.get(cv2.CAP_PROP_FRAME_COUNT))
|
||||
finally:
|
||||
capture.release()
|
||||
return count if count > 0 else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _emit_stream_frame(frame: np.ndarray) -> None:
|
||||
_validate_decoded_frame(frame)
|
||||
record = io.BytesIO()
|
||||
np.save(record, np.ascontiguousarray(frame), allow_pickle=False)
|
||||
payload = record.getvalue()
|
||||
sys.stdout.buffer.write(struct.pack(">Q", len(payload)))
|
||||
sys.stdout.buffer.write(payload)
|
||||
sys.stdout.buffer.flush()
|
||||
|
||||
|
||||
def _stream(video_path: Path) -> None:
|
||||
"""Writes decoded frames as length-prefixed, non-pickled numpy records.
|
||||
|
||||
Tries imageio's FFMPEG plugin first and falls back to cv2, mirroring
|
||||
``_extract_frame``/``_probe``/``_count`` — a file accepted at upload via the cv2
|
||||
fallback must also stream in the frame-range and concat nodes. The fallback only
|
||||
engages when no frame has been emitted yet: a mid-stream decoder failure must
|
||||
surface as an error, not restart the stream from frame 0.
|
||||
"""
|
||||
emitted = False
|
||||
try:
|
||||
for frame in iio.imiter(video_path, plugin="FFMPEG"):
|
||||
_emit_stream_frame(frame)
|
||||
emitted = True
|
||||
except Exception:
|
||||
if emitted:
|
||||
raise
|
||||
if emitted:
|
||||
return
|
||||
|
||||
import cv2
|
||||
|
||||
capture = cv2.VideoCapture(str(video_path))
|
||||
if not capture.isOpened():
|
||||
capture.release()
|
||||
raise FileNotFoundError(f"Unable to open video at {video_path}")
|
||||
try:
|
||||
while True:
|
||||
ok, frame_bgr = capture.read()
|
||||
if not ok or frame_bgr is None:
|
||||
break
|
||||
_emit_stream_frame(cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB))
|
||||
emitted = True
|
||||
finally:
|
||||
capture.release()
|
||||
if not emitted:
|
||||
raise ValueError(f"No frames decoded from {video_path}")
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
try:
|
||||
_limit_worker_memory()
|
||||
command = argv[1]
|
||||
if command == "stream":
|
||||
_assert_decodable_dims(Path(argv[2]))
|
||||
_stream(Path(argv[2]))
|
||||
elif command == "probe":
|
||||
width, height, duration, fps, codec = _probe(Path(argv[2]))
|
||||
print(json.dumps({"width": width, "height": height, "duration": duration, "fps": fps, "codec": codec}))
|
||||
elif command == "frame":
|
||||
_assert_decodable_dims(Path(argv[2]))
|
||||
image = _extract_frame(Path(argv[2]), int(argv[3]))
|
||||
if image is None:
|
||||
print("no frame decoded", file=sys.stderr)
|
||||
return 1
|
||||
image.save(argv[4], format="PNG")
|
||||
print(json.dumps({"ok": True}))
|
||||
elif command == "count":
|
||||
print(json.dumps({"count": _count(Path(argv[2]))}))
|
||||
else:
|
||||
print(f"unknown command {command!r}", file=sys.stderr)
|
||||
return 1
|
||||
except Exception as e:
|
||||
print(str(e) or repr(e), file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv))
|
||||
@@ -0,0 +1,21 @@
|
||||
"""Shared MP4 writer construction for video-producing invocations.
|
||||
|
||||
All video nodes encode through this helper so the encoder settings stay in one
|
||||
place. libx264 + yuv420p (imageio's defaults for the FFMPEG plugin) give
|
||||
broadly-compatible browser playback. ``macro_block_size=1`` is load-bearing:
|
||||
imageio's default of 16 makes ffmpeg silently *rescale* frames to the next
|
||||
multiple of 16 (e.g. 1920x1080 -> 1920x1088), which desynchronizes the encoded
|
||||
file from the dimensions recorded in the video DTO and breaks same-dimension
|
||||
checks downstream (e.g. concatenating a trimmed clip with its source).
|
||||
|
||||
yuv420p requires even dimensions; callers validate that before encoding.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import imageio.v2 as iio2
|
||||
|
||||
|
||||
def make_mp4_writer(path: Path | str, fps: float):
|
||||
"""Returns an imageio FFMPEG writer that preserves frame dimensions exactly."""
|
||||
return iio2.get_writer(str(path), format="FFMPEG", mode="I", fps=fps, codec="libx264", macro_block_size=1)
|
||||
@@ -0,0 +1,442 @@
|
||||
"""Video frame/probe helpers used by the video file store, upload router, and video nodes.
|
||||
|
||||
Decoding runs in a short-lived child process (``video_decode_worker.py``) with a hard
|
||||
timeout. Files reaching these helpers are user uploads, and both decode backends can
|
||||
hang indefinitely on crafted or malformed containers (cv2 wheels historically, ffmpeg in
|
||||
degenerate cases). An in-process hang would tie up the FastAPI request worker that
|
||||
called us — repeated crafted uploads could exhaust the worker pool — and a hung thread
|
||||
cannot be killed from Python, so process isolation is the only reliable bound.
|
||||
The parent explicitly terminates the worker and its descendants when the timeout
|
||||
expires, so a hostile file costs at most ``timeout`` seconds and cannot leak a stuck
|
||||
FFmpeg process.
|
||||
"""
|
||||
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import queue
|
||||
import signal
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Iterator, Optional
|
||||
|
||||
import numpy as np
|
||||
import psutil
|
||||
from PIL import Image
|
||||
|
||||
from invokeai.app.services.session_processor.session_processor_common import CanceledException
|
||||
|
||||
# Generous — a healthy decode of a single frame or of container metadata takes well
|
||||
# under a second even for large files. The timeout exists to bound adversarial or hung
|
||||
# decodes, not to police slow ones.
|
||||
VIDEO_DECODE_TIMEOUT_SECONDS = 30.0
|
||||
MAX_DECODED_FRAME_RECORD_BYTES = 256 * 1024 * 1024
|
||||
MAX_DECODE_STDERR_BYTES = 16 * 1024
|
||||
# Resident-memory kill threshold for a decode worker *tree* (the worker plus its
|
||||
# ffmpeg child). Sized for a legal max-size decode, not a typical one: a
|
||||
# MAX_VIDEO_FRAME_PIXELS RGB frame is 192 MiB, the worker's emit path holds ~4
|
||||
# frame-sized copies at once (~768 MiB), and the ffmpeg child keeps its own output
|
||||
# frame plus YUV reference frames (several hundred MiB more) — so a 1 GiB bound would
|
||||
# kill decodes of videos the frame-size validators explicitly accept. This is an
|
||||
# anti-runaway bound, not a budget. Keep in sync with WORKER_MEMORY_HEADROOM_BYTES in
|
||||
# video_decode_worker.py (the worker-side address-space bound).
|
||||
MAX_VIDEO_DECODE_WORKER_RSS_BYTES = 3 * 1024 * 1024 * 1024
|
||||
# A bound this size doesn't need 50 ms granularity; each poll walks the process tree
|
||||
# via psutil, so a coarser period keeps sustained streaming decodes cheap.
|
||||
WORKER_MEMORY_POLL_SECONDS = 0.25
|
||||
# Upper bound on decoded frame size (~8K video). Decoder-reported dimensions are
|
||||
# untrusted metadata: a tiny crafted container can claim absurd dimensions whose full
|
||||
# frame is only allocated at decode time. probe_video rejects such files before the
|
||||
# upload path persists them, and the worker refuses to decode frames from them.
|
||||
# Keep in sync with MAX_FRAME_PIXELS in video_decode_worker.py.
|
||||
MAX_VIDEO_FRAME_PIXELS = 64 * 1024 * 1024
|
||||
MAX_CONCURRENT_VIDEO_DECODERS = 2
|
||||
_VIDEO_DECODER_SLOTS = threading.BoundedSemaphore(MAX_CONCURRENT_VIDEO_DECODERS)
|
||||
_VIDEO_STREAM_SLOTS = threading.BoundedSemaphore(1)
|
||||
|
||||
_WORKER_PATH = Path(__file__).parent / "video_decode_worker.py"
|
||||
|
||||
|
||||
class VideoDecodeTimeoutError(TimeoutError):
|
||||
"""The decode worker did not finish within its time budget.
|
||||
|
||||
A timeout is contention (or an adversarial file), not evidence the video is
|
||||
undecodable — callers that gate acceptance on decodability must treat it as
|
||||
inconclusive rather than as a failed decode.
|
||||
"""
|
||||
|
||||
|
||||
def get_video_thumbnail_name(video_name: str) -> str:
|
||||
"""Given a video file name (e.g. <uuid>.mp4), returns the matching thumbnail name (e.g. <uuid>.webp)."""
|
||||
return os.path.splitext(video_name)[0] + ".webp"
|
||||
|
||||
|
||||
def _worker_command(*args: str) -> list[str]:
|
||||
"""Command line for one decode-worker invocation (patchable in tests).
|
||||
|
||||
The worker is run by file path rather than ``-m`` so the child process doesn't
|
||||
import the invokeai package (and transitively torch) just to decode a frame.
|
||||
"""
|
||||
return [sys.executable, str(_WORKER_PATH), *args]
|
||||
|
||||
|
||||
def _spawn_worker(*args: str, **kwargs: Any) -> subprocess.Popen[Any]:
|
||||
"""Starts a worker in an independently killable process group on POSIX."""
|
||||
if os.name != "nt":
|
||||
kwargs["start_new_session"] = True
|
||||
return subprocess.Popen(_worker_command(*args), **kwargs)
|
||||
|
||||
|
||||
def _is_process_running(pid: int) -> bool:
|
||||
"""Returns whether a process exists and is not a zombie."""
|
||||
try:
|
||||
process = psutil.Process(pid)
|
||||
return process.is_running() and process.status() != psutil.STATUS_ZOMBIE
|
||||
except psutil.Error:
|
||||
return False
|
||||
|
||||
|
||||
def _terminate_process_tree(proc: subprocess.Popen[Any]) -> None:
|
||||
"""Kills a worker and every descendant it spawned."""
|
||||
if os.name != "nt":
|
||||
try:
|
||||
os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
|
||||
proc.wait(timeout=5)
|
||||
return
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
pass
|
||||
try:
|
||||
parent = psutil.Process(proc.pid)
|
||||
processes = parent.children(recursive=True)
|
||||
processes.append(parent)
|
||||
for process in processes:
|
||||
try:
|
||||
process.kill()
|
||||
except psutil.Error:
|
||||
pass
|
||||
psutil.wait_procs(processes, timeout=5)
|
||||
except psutil.Error:
|
||||
try:
|
||||
proc.kill()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _worker_tree_rss(proc: subprocess.Popen[Any]) -> int:
|
||||
"""Returns resident memory used by the worker and its decoder children."""
|
||||
try:
|
||||
worker = psutil.Process(proc.pid)
|
||||
processes = [worker, *worker.children(recursive=True)]
|
||||
except psutil.Error:
|
||||
return 0
|
||||
total = 0
|
||||
for process in processes:
|
||||
try:
|
||||
total += process.memory_info().rss
|
||||
except psutil.Error:
|
||||
pass
|
||||
return total
|
||||
|
||||
|
||||
def _start_worker_memory_monitor(
|
||||
proc: subprocess.Popen[Any],
|
||||
) -> tuple[threading.Event, threading.Event, threading.Thread]:
|
||||
"""Kills a decoder process tree when its cross-platform RSS bound is exceeded."""
|
||||
stopped = threading.Event()
|
||||
exceeded = threading.Event()
|
||||
|
||||
def monitor() -> None:
|
||||
while not stopped.is_set():
|
||||
if proc.poll() is not None:
|
||||
return
|
||||
if _worker_tree_rss(proc) > MAX_VIDEO_DECODE_WORKER_RSS_BYTES:
|
||||
exceeded.set()
|
||||
_terminate_process_tree(proc)
|
||||
return
|
||||
stopped.wait(WORKER_MEMORY_POLL_SECONDS)
|
||||
|
||||
thread = threading.Thread(target=monitor, name="video-memory-monitor", daemon=True)
|
||||
thread.start()
|
||||
return stopped, exceeded, thread
|
||||
|
||||
|
||||
def _run_worker_unbounded(args: list[str], timeout: float, raise_on_timeout: bool = False) -> Optional[dict[str, Any]]:
|
||||
"""Runs the decode worker; returns its parsed JSON output, or None on failure or timeout.
|
||||
|
||||
With ``raise_on_timeout``, a timeout raises VideoDecodeTimeoutError instead of
|
||||
returning None, letting callers distinguish "could not decode" from "ran out of
|
||||
time on a loaded machine".
|
||||
"""
|
||||
monitor_stop: threading.Event | None = None
|
||||
monitor: threading.Thread | None = None
|
||||
try:
|
||||
proc = _spawn_worker(*args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
except Exception:
|
||||
return None
|
||||
try:
|
||||
monitor_stop, _memory_exceeded, monitor = _start_worker_memory_monitor(proc)
|
||||
stdout, _ = proc.communicate(timeout=timeout)
|
||||
except subprocess.TimeoutExpired as error:
|
||||
_terminate_process_tree(proc)
|
||||
proc.communicate()
|
||||
if raise_on_timeout:
|
||||
raise VideoDecodeTimeoutError(f"Video decode worker timed out after {timeout}s") from error
|
||||
return None
|
||||
except Exception:
|
||||
# An unexpected failure (e.g. OSError from communicate()) must not leak the
|
||||
# worker tree: the finally below stops the RSS-monitor backstop, so nothing
|
||||
# else would ever reap a still-running worker and its ffmpeg child.
|
||||
_terminate_process_tree(proc)
|
||||
return None
|
||||
finally:
|
||||
if monitor_stop is not None and monitor is not None:
|
||||
monitor_stop.set()
|
||||
monitor.join(timeout=1)
|
||||
if proc.returncode != 0:
|
||||
return None
|
||||
try:
|
||||
result = json.loads(stdout)
|
||||
except ValueError:
|
||||
return None
|
||||
return result if isinstance(result, dict) else None
|
||||
|
||||
|
||||
def _run_worker(args: list[str], timeout: float, raise_on_timeout: bool = False) -> Optional[dict[str, Any]]:
|
||||
deadline = time.monotonic() + timeout
|
||||
if not _VIDEO_DECODER_SLOTS.acquire(timeout=timeout):
|
||||
if raise_on_timeout:
|
||||
raise VideoDecodeTimeoutError(f"Video decode worker timed out after {timeout}s")
|
||||
return None
|
||||
try:
|
||||
remaining = max(0.0, deadline - time.monotonic())
|
||||
return _run_worker_unbounded(args, remaining, raise_on_timeout)
|
||||
finally:
|
||||
_VIDEO_DECODER_SLOTS.release()
|
||||
|
||||
|
||||
def _iter_video_frames_unbounded(
|
||||
video_path: Path,
|
||||
timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS,
|
||||
is_canceled: Optional[Callable[[], bool]] = None,
|
||||
) -> Iterator[np.ndarray]:
|
||||
"""Streams decoded frames from an isolated worker with bounded memory and wait time."""
|
||||
proc = _spawn_worker(
|
||||
"stream",
|
||||
str(video_path),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
if proc.stdout is None or proc.stderr is None:
|
||||
_terminate_process_tree(proc)
|
||||
raise RuntimeError("Unable to open video decoder output stream")
|
||||
memory_monitor_stop, _memory_exceeded, memory_monitor = _start_worker_memory_monitor(proc)
|
||||
|
||||
results: queue.Queue[tuple[str, object]] = queue.Queue(maxsize=1)
|
||||
stopped = threading.Event()
|
||||
stderr_tail = bytearray()
|
||||
|
||||
def read_stderr() -> str:
|
||||
return bytes(stderr_tail).decode(errors="replace").strip()
|
||||
|
||||
def drain_stderr() -> None:
|
||||
while chunk := proc.stderr.read(4096):
|
||||
stderr_tail.extend(chunk)
|
||||
if len(stderr_tail) > MAX_DECODE_STDERR_BYTES:
|
||||
del stderr_tail[:-MAX_DECODE_STDERR_BYTES]
|
||||
|
||||
def read_exactly(size: int) -> bytes:
|
||||
chunks: list[bytes] = []
|
||||
remaining = size
|
||||
while remaining > 0:
|
||||
chunk = proc.stdout.read(remaining)
|
||||
if not chunk:
|
||||
if remaining == size:
|
||||
raise EOFError
|
||||
raise OSError("Truncated frame record from video decoder")
|
||||
chunks.append(chunk)
|
||||
remaining -= len(chunk)
|
||||
return b"".join(chunks)
|
||||
|
||||
def put_result(result: tuple[str, object]) -> None:
|
||||
while not stopped.is_set():
|
||||
try:
|
||||
results.put(result, timeout=0.1)
|
||||
return
|
||||
except queue.Full:
|
||||
continue
|
||||
|
||||
def read_frames() -> None:
|
||||
try:
|
||||
while not stopped.is_set():
|
||||
record_size = struct.unpack(">Q", read_exactly(8))[0]
|
||||
if record_size > MAX_DECODED_FRAME_RECORD_BYTES:
|
||||
raise ValueError(f"Decoded frame record exceeds {MAX_DECODED_FRAME_RECORD_BYTES} bytes")
|
||||
payload = read_exactly(record_size)
|
||||
put_result(("frame", np.load(io.BytesIO(payload), allow_pickle=False)))
|
||||
except (EOFError, ValueError, OSError) as error:
|
||||
put_result(("done", error))
|
||||
|
||||
reader = threading.Thread(target=read_frames, name="video-frame-reader", daemon=True)
|
||||
stderr_reader = threading.Thread(target=drain_stderr, name="video-stderr-reader", daemon=True)
|
||||
reader.start()
|
||||
stderr_reader.start()
|
||||
deadline = time.monotonic() + timeout
|
||||
try:
|
||||
while True:
|
||||
if is_canceled is not None and is_canceled():
|
||||
raise CanceledException
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise TimeoutError(f"Timed out decoding frames from {video_path}")
|
||||
try:
|
||||
kind, value = results.get(timeout=min(0.1, remaining))
|
||||
except queue.Empty:
|
||||
continue
|
||||
if kind == "frame":
|
||||
if not isinstance(value, np.ndarray):
|
||||
raise ValueError(f"Decoder returned an invalid frame for {video_path}")
|
||||
yield value
|
||||
deadline = time.monotonic() + timeout
|
||||
continue
|
||||
try:
|
||||
return_code = proc.wait(timeout=min(1, timeout))
|
||||
except subprocess.TimeoutExpired as error:
|
||||
_terminate_process_tree(proc)
|
||||
stderr_reader.join(timeout=1)
|
||||
detail = read_stderr()
|
||||
message = f"Timed out waiting for video decoder worker for {video_path}"
|
||||
raise TimeoutError(f"{message}: {detail}" if detail else message) from error
|
||||
if return_code != 0:
|
||||
stderr_reader.join(timeout=1)
|
||||
detail = read_stderr()
|
||||
message = f"Unable to decode video at {video_path}"
|
||||
raise ValueError(f"{message}: {detail}" if detail else message) from value
|
||||
return
|
||||
finally:
|
||||
memory_monitor_stop.set()
|
||||
memory_monitor.join(timeout=1)
|
||||
stopped.set()
|
||||
if proc.poll() is None:
|
||||
_terminate_process_tree(proc)
|
||||
proc.stdout.close()
|
||||
proc.wait()
|
||||
reader.join(timeout=1)
|
||||
stderr_reader.join(timeout=1)
|
||||
proc.stderr.close()
|
||||
|
||||
|
||||
def iter_video_frames(
|
||||
video_path: Path,
|
||||
timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS,
|
||||
is_canceled: Optional[Callable[[], bool]] = None,
|
||||
) -> Iterator[np.ndarray]:
|
||||
acquired: list[threading.BoundedSemaphore] = []
|
||||
capacity_deadline = time.monotonic() + timeout
|
||||
try:
|
||||
for slot in (_VIDEO_STREAM_SLOTS, _VIDEO_DECODER_SLOTS):
|
||||
while True:
|
||||
if is_canceled is not None and is_canceled():
|
||||
raise CanceledException
|
||||
remaining = capacity_deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise TimeoutError(f"Timed out waiting to decode frames from {video_path}")
|
||||
if slot.acquire(timeout=min(0.1, remaining)):
|
||||
acquired.append(slot)
|
||||
break
|
||||
yield from _iter_video_frames_unbounded(video_path, timeout, is_canceled)
|
||||
finally:
|
||||
for slot in reversed(acquired):
|
||||
slot.release()
|
||||
|
||||
|
||||
def extract_video_frame(
|
||||
video_path: Path,
|
||||
frame_index: int = 0,
|
||||
timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS,
|
||||
raise_on_timeout: bool = False,
|
||||
) -> Optional[Image.Image]:
|
||||
"""Extracts a single frame from a video file as a PIL Image. Returns None on failure or timeout.
|
||||
|
||||
With ``raise_on_timeout``, a timeout raises VideoDecodeTimeoutError instead of
|
||||
returning None.
|
||||
"""
|
||||
fd, tmp_name = tempfile.mkstemp(prefix="invokeai_frame_", suffix=".png")
|
||||
os.close(fd)
|
||||
try:
|
||||
result = _run_worker(["frame", str(video_path), str(frame_index), tmp_name], timeout, raise_on_timeout)
|
||||
if result is None:
|
||||
return None
|
||||
with Image.open(tmp_name) as image:
|
||||
image.load()
|
||||
return image
|
||||
except VideoDecodeTimeoutError:
|
||||
raise
|
||||
except Exception:
|
||||
return None
|
||||
finally:
|
||||
Path(tmp_name).unlink(missing_ok=True)
|
||||
|
||||
|
||||
def probe_video_with_codec(
|
||||
video_path: Path, timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS
|
||||
) -> tuple[int, int, float, Optional[float], Optional[str]]:
|
||||
"""Returns (width, height, duration_seconds, fps_or_none, codec_or_none) for a video file.
|
||||
|
||||
Raises FileNotFoundError if the file cannot be read — including when the decode
|
||||
times out, since a file we cannot probe within the bound is treated as unreadable —
|
||||
or when the decoder reports metadata no sane video has (non-positive or over-limit
|
||||
dimensions, non-finite or negative duration). Decoder-reported values are untrusted:
|
||||
they come from the uploaded container, and the upload path persists them and sizes
|
||||
thumbnail decoding by them. A non-finite or non-positive fps is coerced to None
|
||||
(unknown) rather than rejected, matching the decoder's own unknown-fps behavior.
|
||||
"""
|
||||
result = _run_worker(["probe", str(video_path)], timeout)
|
||||
if result is None:
|
||||
raise FileNotFoundError(f"Unable to open video at {video_path}")
|
||||
try:
|
||||
width = int(result["width"])
|
||||
height = int(result["height"])
|
||||
duration = float(result["duration"])
|
||||
fps_raw = result.get("fps")
|
||||
fps: Optional[float] = float(fps_raw) if fps_raw else None
|
||||
codec_raw = result.get("codec")
|
||||
codec = str(codec_raw).lower() if codec_raw else None
|
||||
except (KeyError, TypeError, ValueError, OverflowError) as e:
|
||||
raise FileNotFoundError(f"Unable to open video at {video_path}") from e
|
||||
if width <= 0 or height <= 0 or width * height > MAX_VIDEO_FRAME_PIXELS:
|
||||
raise FileNotFoundError(f"Video at {video_path} reports invalid dimensions {width}x{height}")
|
||||
if not math.isfinite(duration) or duration < 0:
|
||||
raise FileNotFoundError(f"Video at {video_path} reports an invalid duration {duration}")
|
||||
if fps is not None and (not math.isfinite(fps) or fps <= 0):
|
||||
fps = None
|
||||
return width, height, duration, fps, codec
|
||||
|
||||
|
||||
def probe_video(
|
||||
video_path: Path, timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS
|
||||
) -> tuple[int, int, float, Optional[float]]:
|
||||
"""Returns validated video metadata without the codec."""
|
||||
width, height, duration, fps, _codec = probe_video_with_codec(video_path, timeout)
|
||||
return width, height, duration, fps
|
||||
|
||||
|
||||
def decoder_frame_count(video_path: Path, timeout: float = VIDEO_DECODE_TIMEOUT_SECONDS) -> Optional[int]:
|
||||
"""Returns the exact decoded frame count, or None if it cannot be determined in time.
|
||||
|
||||
Preferred over a ``duration * fps`` estimate, which can overshoot by one on VFR
|
||||
uploads or containers with imprecise metadata; callers fall back to that estimate
|
||||
when this returns None.
|
||||
"""
|
||||
result = _run_worker(["count", str(video_path)], timeout)
|
||||
if result is None:
|
||||
return None
|
||||
count = result.get("count")
|
||||
if isinstance(count, bool) or not isinstance(count, (int, float)):
|
||||
return None
|
||||
return int(count) if count > 0 else None
|
||||
@@ -55,6 +55,7 @@ from invokeai.backend.model_manager.configs.lora import (
|
||||
LoRA_LyCORIS_SD1_Config,
|
||||
LoRA_LyCORIS_SD2_Config,
|
||||
LoRA_LyCORIS_SDXL_Config,
|
||||
LoRA_LyCORIS_Wan_Config,
|
||||
LoRA_LyCORIS_ZImage_Config,
|
||||
LoRA_OMI_FLUX_Config,
|
||||
LoRA_OMI_SDXL_Config,
|
||||
@@ -81,10 +82,12 @@ from invokeai.backend.model_manager.configs.main import (
|
||||
Main_Diffusers_SD3_Config,
|
||||
Main_Diffusers_SDXL_Config,
|
||||
Main_Diffusers_SDXLRefiner_Config,
|
||||
Main_Diffusers_Wan_Config,
|
||||
Main_Diffusers_ZImage_Config,
|
||||
Main_GGUF_Flux2_Config,
|
||||
Main_GGUF_FLUX_Config,
|
||||
Main_GGUF_QwenImage_Config,
|
||||
Main_GGUF_Wan_Config,
|
||||
Main_GGUF_ZImage_Config,
|
||||
MainModelDefaultSettings,
|
||||
)
|
||||
@@ -126,10 +129,13 @@ from invokeai.backend.model_manager.configs.vae import (
|
||||
VAE_Checkpoint_SD1_Config,
|
||||
VAE_Checkpoint_SD2_Config,
|
||||
VAE_Checkpoint_SDXL_Config,
|
||||
VAE_Checkpoint_Wan_Config,
|
||||
VAE_Diffusers_Flux2_Config,
|
||||
VAE_Diffusers_SD1_Config,
|
||||
VAE_Diffusers_SDXL_Config,
|
||||
VAE_Diffusers_Wan_Config,
|
||||
)
|
||||
from invokeai.backend.model_manager.configs.wan_t5_encoder import WanT5Encoder_WanT5Encoder_Config
|
||||
from invokeai.backend.model_manager.model_on_disk import ModelOnDisk
|
||||
from invokeai.backend.model_manager.taxonomy import (
|
||||
BaseModelType,
|
||||
@@ -180,6 +186,7 @@ AnyModelConfig = Annotated[
|
||||
Annotated[Main_Diffusers_Flux2_Config, Main_Diffusers_Flux2_Config.get_tag()],
|
||||
Annotated[Main_Diffusers_CogView4_Config, Main_Diffusers_CogView4_Config.get_tag()],
|
||||
Annotated[Main_Diffusers_QwenImage_Config, Main_Diffusers_QwenImage_Config.get_tag()],
|
||||
Annotated[Main_Diffusers_Wan_Config, Main_Diffusers_Wan_Config.get_tag()],
|
||||
Annotated[Main_Diffusers_ZImage_Config, Main_Diffusers_ZImage_Config.get_tag()],
|
||||
Annotated[Main_Diffusers_Ideogram4_Config, Main_Diffusers_Ideogram4_Config.get_tag()],
|
||||
# Main (Pipeline) - checkpoint format
|
||||
@@ -201,6 +208,7 @@ AnyModelConfig = Annotated[
|
||||
Annotated[Main_GGUF_Flux2_Config, Main_GGUF_Flux2_Config.get_tag()],
|
||||
Annotated[Main_GGUF_FLUX_Config, Main_GGUF_FLUX_Config.get_tag()],
|
||||
Annotated[Main_GGUF_QwenImage_Config, Main_GGUF_QwenImage_Config.get_tag()],
|
||||
Annotated[Main_GGUF_Wan_Config, Main_GGUF_Wan_Config.get_tag()],
|
||||
Annotated[Main_GGUF_ZImage_Config, Main_GGUF_ZImage_Config.get_tag()],
|
||||
# VAE - checkpoint format
|
||||
Annotated[VAE_Checkpoint_SD1_Config, VAE_Checkpoint_SD1_Config.get_tag()],
|
||||
@@ -208,12 +216,18 @@ AnyModelConfig = Annotated[
|
||||
Annotated[VAE_Checkpoint_SDXL_Config, VAE_Checkpoint_SDXL_Config.get_tag()],
|
||||
Annotated[VAE_Checkpoint_FLUX_Config, VAE_Checkpoint_FLUX_Config.get_tag()],
|
||||
Annotated[VAE_Checkpoint_Flux2_Config, VAE_Checkpoint_Flux2_Config.get_tag()],
|
||||
# IMPORTANT: VAE_Checkpoint_Wan_Config must be checked BEFORE QwenImage —
|
||||
# both share the AutoencoderKLWan architecture and the Wan config relies
|
||||
# on a filename heuristic to claim 16-channel files; ordering here lets
|
||||
# Wan win when the filename suggests it.
|
||||
Annotated[VAE_Checkpoint_Wan_Config, VAE_Checkpoint_Wan_Config.get_tag()],
|
||||
Annotated[VAE_Checkpoint_QwenImage_Config, VAE_Checkpoint_QwenImage_Config.get_tag()],
|
||||
Annotated[VAE_Checkpoint_Anima_Config, VAE_Checkpoint_Anima_Config.get_tag()],
|
||||
# VAE - diffusers format
|
||||
Annotated[VAE_Diffusers_SD1_Config, VAE_Diffusers_SD1_Config.get_tag()],
|
||||
Annotated[VAE_Diffusers_SDXL_Config, VAE_Diffusers_SDXL_Config.get_tag()],
|
||||
Annotated[VAE_Diffusers_Flux2_Config, VAE_Diffusers_Flux2_Config.get_tag()],
|
||||
Annotated[VAE_Diffusers_Wan_Config, VAE_Diffusers_Wan_Config.get_tag()],
|
||||
# ControlNet - checkpoint format
|
||||
Annotated[ControlNet_Checkpoint_SD1_Config, ControlNet_Checkpoint_SD1_Config.get_tag()],
|
||||
Annotated[ControlNet_Checkpoint_SD2_Config, ControlNet_Checkpoint_SD2_Config.get_tag()],
|
||||
@@ -236,6 +250,13 @@ AnyModelConfig = Annotated[
|
||||
Annotated[LoRA_LyCORIS_FLUX_Config, LoRA_LyCORIS_FLUX_Config.get_tag()],
|
||||
Annotated[LoRA_LyCORIS_ZImage_Config, LoRA_LyCORIS_ZImage_Config.get_tag()],
|
||||
Annotated[LoRA_LyCORIS_QwenImage_Config, LoRA_LyCORIS_QwenImage_Config.get_tag()],
|
||||
# Wan and Anima both target ``blocks.X`` shapes; their LoRA probes are
|
||||
# mutually exclusive — Wan rejects Anima's ``_proj``/``mlp``/
|
||||
# ``adaln_modulation`` markers, Anima requires at least one of those
|
||||
# markers (see ``has_cosmos_dit_*_keys_strict``). Order between these
|
||||
# two doesn't affect correctness; mutual exclusivity is locked in by
|
||||
# ``test_wan_lora_probe_independence.py``.
|
||||
Annotated[LoRA_LyCORIS_Wan_Config, LoRA_LyCORIS_Wan_Config.get_tag()],
|
||||
Annotated[LoRA_LyCORIS_Anima_Config, LoRA_LyCORIS_Anima_Config.get_tag()],
|
||||
# LoRA - OMI format
|
||||
Annotated[LoRA_OMI_SDXL_Config, LoRA_OMI_SDXL_Config.get_tag()],
|
||||
@@ -262,6 +283,8 @@ AnyModelConfig = Annotated[
|
||||
# Qwen VL Encoder (Qwen2.5-VL multimodal encoder for Qwen Image)
|
||||
Annotated[QwenVLEncoder_Diffusers_Config, QwenVLEncoder_Diffusers_Config.get_tag()],
|
||||
Annotated[QwenVLEncoder_Checkpoint_Config, QwenVLEncoder_Checkpoint_Config.get_tag()],
|
||||
# Wan T5 Encoder (UMT5-XXL for Wan 2.2)
|
||||
Annotated[WanT5Encoder_WanT5Encoder_Config, WanT5Encoder_WanT5Encoder_Config.get_tag()],
|
||||
# TI - file format
|
||||
Annotated[TI_File_SD1_Config, TI_File_SD1_Config.get_tag()],
|
||||
Annotated[TI_File_SD2_Config, TI_File_SD2_Config.get_tag()],
|
||||
|
||||
@@ -28,14 +28,23 @@ from invokeai.backend.model_manager.taxonomy import (
|
||||
FluxLoRAFormat,
|
||||
ModelFormat,
|
||||
ModelType,
|
||||
WanLoRAVariantType,
|
||||
ZImageVariantType,
|
||||
)
|
||||
from invokeai.backend.model_manager.util.model_util import lora_token_vector_length
|
||||
from invokeai.backend.patches.lora_conversions.anima_lora_constants import (
|
||||
has_cosmos_dit_kohya_keys,
|
||||
has_cosmos_dit_kohya_keys_strict,
|
||||
has_cosmos_dit_peft_keys,
|
||||
has_cosmos_dit_peft_keys_strict,
|
||||
)
|
||||
from invokeai.backend.patches.lora_conversions.flux_control_lora_utils import is_state_dict_likely_flux_control
|
||||
from invokeai.backend.patches.lora_conversions.wan_lora_constants import (
|
||||
detect_wan_lora_variant,
|
||||
has_non_wan_architecture_keys,
|
||||
has_wan_kohya_keys,
|
||||
has_wan_peft_keys,
|
||||
)
|
||||
|
||||
# Defaults used to compute the effective slider range when one or both bounds
|
||||
# are unset. These intentionally mirror the frontend's DEFAULT_LORA_WEIGHT_CONFIG
|
||||
@@ -909,16 +918,20 @@ class LoRA_LyCORIS_Anima_Config(LoRA_LyCORIS_Config_Base, Config_Base):
|
||||
Anima LoRAs have keys like:
|
||||
- lora_unet_blocks_0_cross_attn_k_proj.lora_down.weight (Kohya format)
|
||||
- diffusion_model.blocks.0.cross_attn.k_proj.lora_A.weight (diffusers PEFT format)
|
||||
- transformer.blocks.0.cross_attn.k_proj.lora_A.weight (diffusers PEFT format)
|
||||
- transformer.blocks.0.mlp.layer_0.lora_A.weight (Anima-only MLP layer)
|
||||
|
||||
Detection requires Cosmos DiT-specific subcomponent names (cross_attn,
|
||||
self_attn, mlp, adaln_modulation) to avoid false-positives on other
|
||||
architectures that also use ``blocks`` in their paths.
|
||||
Uses the **strict** Cosmos-DiT detectors, which require an
|
||||
Anima-exclusive subcomponent name (``mlp``, ``adaln_modulation``, or
|
||||
``_proj``-suffixed attention). The loose detectors would also accept
|
||||
Wan-native LoRAs (which use ``cross_attn``/``self_attn`` too but with
|
||||
bare ``.q``/``.k``/``.v``/``.o`` rather than ``_proj``), so they're not
|
||||
safe for first-match-wins probing — see the regression tests in
|
||||
``test_wan_lora_probe_independence.py``.
|
||||
"""
|
||||
state_dict = mod.load_state_dict()
|
||||
str_keys = [k for k in state_dict.keys() if isinstance(k, str)]
|
||||
|
||||
has_cosmos_keys = has_cosmos_dit_kohya_keys(str_keys) or has_cosmos_dit_peft_keys(str_keys)
|
||||
has_cosmos_keys = has_cosmos_dit_kohya_keys_strict(str_keys) or has_cosmos_dit_peft_keys_strict(str_keys)
|
||||
|
||||
# Also check for LoRA/LoKR weight suffixes
|
||||
has_lora_suffix = state_dict_has_any_keys_ending_with(
|
||||
@@ -941,19 +954,112 @@ class LoRA_LyCORIS_Anima_Config(LoRA_LyCORIS_Config_Base, Config_Base):
|
||||
|
||||
@classmethod
|
||||
def _get_base_or_raise(cls, mod: ModelOnDisk) -> BaseModelType:
|
||||
"""Anima LoRAs target Cosmos DiT blocks (blocks.X.cross_attn, blocks.X.self_attn, etc.).
|
||||
"""Anima LoRAs target Cosmos DiT blocks (blocks.X.mlp, blocks.X.adaln_modulation,
|
||||
blocks.X.cross_attn.q_proj, etc.).
|
||||
|
||||
Uses Cosmos DiT-specific subcomponent names to avoid false-positives.
|
||||
Uses the strict Cosmos-DiT detectors to be mutually exclusive with
|
||||
Wan-LoRA detection — see ``_validate_looks_like_lora`` for rationale.
|
||||
"""
|
||||
state_dict = mod.load_state_dict()
|
||||
str_keys = [k for k in state_dict.keys() if isinstance(k, str)]
|
||||
|
||||
if has_cosmos_dit_kohya_keys(str_keys) or has_cosmos_dit_peft_keys(str_keys):
|
||||
if has_cosmos_dit_kohya_keys_strict(str_keys) or has_cosmos_dit_peft_keys_strict(str_keys):
|
||||
return BaseModelType.Anima
|
||||
|
||||
raise NotAMatchError("model does not look like an Anima LoRA")
|
||||
|
||||
|
||||
class LoRA_LyCORIS_Wan_Config(LoRA_LyCORIS_Config_Base, Config_Base):
|
||||
"""Model config for Wan 2.2 LoRA models in LyCORIS format.
|
||||
|
||||
Wan LoRAs target ``WanTransformer3DModel`` blocks. The Wan 2.2 A14B family
|
||||
is dual-expert (high-noise + low-noise) — LoRAs are typically trained
|
||||
against one expert. ``expert`` records which one so the model loader
|
||||
invocation can wire it to the correct ``loras`` / ``loras_low_noise`` list.
|
||||
Many LoRAs are expert-agnostic (TI2V-5B family, or community LoRAs that
|
||||
just don't tag the expert) — these get ``expert=None`` and are applied to
|
||||
both experts by default.
|
||||
"""
|
||||
|
||||
base: Literal[BaseModelType.Wan] = Field(default=BaseModelType.Wan)
|
||||
expert: Literal["high", "low"] | None = Field(
|
||||
default=None,
|
||||
description="For Wan 2.2 A14B dual-expert LoRAs: 'high' targets the high-noise expert, "
|
||||
"'low' targets the low-noise expert. None means the LoRA is expert-agnostic "
|
||||
"(TI2V-5B, or community LoRAs without explicit tagging) and is applied to both.",
|
||||
)
|
||||
variant: WanLoRAVariantType | None = Field(
|
||||
default=None,
|
||||
description="The Wan model family this LoRA targets, detected from its inner-dim "
|
||||
"(5120 -> A14B, 3072 -> TI2V-5B). A14B LoRAs are incompatible with TI2V-5B mains "
|
||||
"(and vice versa) — they crash with a shape mismatch in the layer patcher. The "
|
||||
"linear-view graph builder filters LoRAs on variant when building the LoRA "
|
||||
"collection. None means the LoRA's inner-dim couldn't be identified.",
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _validate_looks_like_lora(cls, mod: ModelOnDisk) -> None:
|
||||
"""Wan LoRAs target attn1/attn2/ffn.net (diffusers form) or self_attn/cross_attn/ffn.N (native form)."""
|
||||
state_dict = mod.load_state_dict()
|
||||
str_keys = [k for k in state_dict.keys() if isinstance(k, str)]
|
||||
|
||||
has_wan_keys = has_wan_kohya_keys(str_keys) or has_wan_peft_keys(str_keys)
|
||||
has_lora_suffix = state_dict_has_any_keys_ending_with(
|
||||
state_dict,
|
||||
{
|
||||
"lora_A.weight",
|
||||
"lora_B.weight",
|
||||
"lora_down.weight",
|
||||
"lora_up.weight",
|
||||
"dora_scale",
|
||||
".lokr_w1",
|
||||
".lokr_w2",
|
||||
},
|
||||
)
|
||||
|
||||
# Reject if any non-Wan architecture signature is present. Without this
|
||||
# guard a Wan LoRA could be falsely identified by Anima (cross_attn /
|
||||
# self_attn name collision) or vice versa.
|
||||
if has_wan_keys and has_lora_suffix and not has_non_wan_architecture_keys(str_keys):
|
||||
return
|
||||
|
||||
raise NotAMatchError("model does not match Wan LoRA heuristics")
|
||||
|
||||
@classmethod
|
||||
def _get_base_or_raise(cls, mod: ModelOnDisk) -> BaseModelType:
|
||||
state_dict = mod.load_state_dict()
|
||||
str_keys = [k for k in state_dict.keys() if isinstance(k, str)]
|
||||
|
||||
if (has_wan_kohya_keys(str_keys) or has_wan_peft_keys(str_keys)) and not has_non_wan_architecture_keys(
|
||||
str_keys
|
||||
):
|
||||
return BaseModelType.Wan
|
||||
|
||||
raise NotAMatchError("model does not look like a Wan LoRA")
|
||||
|
||||
@classmethod
|
||||
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
|
||||
# Run the base-class probe (file-check, lora-suffix, base detection).
|
||||
instance = super().from_model_on_disk(mod, override_fields)
|
||||
|
||||
# Auto-detect the expert tag from the filename if the user didn't
|
||||
# override it. ``high_noise`` / ``low_noise`` / hyphenated / concatenated
|
||||
# variants — mirrors the GGUF transformer probe's heuristic.
|
||||
if instance.expert is None:
|
||||
name = mod.path.stem.lower()
|
||||
if any(s in name for s in ("high_noise", "high-noise", "highnoise")):
|
||||
instance.expert = "high"
|
||||
elif any(s in name for s in ("low_noise", "low-noise", "lownoise")):
|
||||
instance.expert = "low"
|
||||
|
||||
# Auto-detect the model-family variant from inner_dim in the state
|
||||
# dict. The override field skips this if the user has set it.
|
||||
if instance.variant is None:
|
||||
instance.variant = detect_wan_lora_variant(mod.load_state_dict())
|
||||
|
||||
return instance
|
||||
|
||||
|
||||
class ControlAdapter_Config_Base(ABC, BaseModel):
|
||||
default_settings: ControlAdapterDefaultSettings | None = Field(None)
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ from invokeai.backend.model_manager.taxonomy import (
|
||||
QwenImageVariantType,
|
||||
SchedulerPredictionType,
|
||||
SubModelType,
|
||||
WanVariantType,
|
||||
ZImageVariantType,
|
||||
)
|
||||
from invokeai.backend.quantization.gguf.ggml_tensor import GGMLTensor
|
||||
@@ -65,7 +66,12 @@ class MainModelDefaultSettings(BaseModel):
|
||||
def from_base(
|
||||
cls,
|
||||
base: BaseModelType,
|
||||
variant: Flux2VariantType | FluxVariantType | ModelVariantType | ZImageVariantType | None = None,
|
||||
variant: Flux2VariantType
|
||||
| FluxVariantType
|
||||
| ModelVariantType
|
||||
| WanVariantType
|
||||
| ZImageVariantType
|
||||
| None = None,
|
||||
) -> Self | None:
|
||||
match base:
|
||||
case BaseModelType.StableDiffusion1:
|
||||
@@ -99,6 +105,12 @@ class MainModelDefaultSettings(BaseModel):
|
||||
return cls(steps=4, cfg_scale=1.0, width=1024, height=1024)
|
||||
case BaseModelType.QwenImage:
|
||||
return cls(steps=40, cfg_scale=4.0, width=1024, height=1024)
|
||||
case BaseModelType.Wan:
|
||||
# Wan 2.2 recommended defaults differ by variant.
|
||||
if variant == WanVariantType.TI2V_5B:
|
||||
return cls(steps=30, cfg_scale=5.0, width=1024, height=1024)
|
||||
# Default to A14B settings (also used when variant is unknown).
|
||||
return cls(steps=40, cfg_scale=4.0, width=1024, height=1024)
|
||||
case _:
|
||||
# TODO(psyche): Do we want defaults for other base types?
|
||||
return None
|
||||
@@ -1474,6 +1486,277 @@ class Main_GGUF_QwenImage_Config(Checkpoint_Config_Base, Main_Config_Base, Confi
|
||||
return cls(**override_fields, variant=explicit_variant)
|
||||
|
||||
|
||||
def _has_wan_keys(state_dict: dict[str | int, Any]) -> bool:
|
||||
"""Check if state dict contains Wan 2.2 transformer keys.
|
||||
|
||||
Two layouts are accepted:
|
||||
|
||||
* **Diffusers** (city96-style GGUF, Wan-AI/*-Diffusers safetensors): the text
|
||||
projection is named ``condition_embedder.text_embedder.linear_1``.
|
||||
* **Native upstream** (QuantStack-style GGUF, ComfyUI, Wan-AI's non-Diffusers
|
||||
releases): the text projection is named ``text_embedding.0``.
|
||||
|
||||
Both layouts share ``patch_embedding.weight`` as the input conv. Combined with
|
||||
the text-projection fingerprint, this won't collide with FLUX
|
||||
(``double_blocks/single_blocks``), Qwen Image (``txt_in/img_in``), Z-Image
|
||||
(``cap_embedder``), or Anima (``llm_adapter``).
|
||||
|
||||
Tolerates both bare keys and the ComfyUI ``model.diffusion_model.`` /
|
||||
``diffusion_model.`` prefixes.
|
||||
"""
|
||||
text_proj_options = (
|
||||
"condition_embedder.text_embedder.linear_1.weight",
|
||||
"text_embedding.0.weight",
|
||||
)
|
||||
prefixes = ("", "model.diffusion_model.", "diffusion_model.")
|
||||
keys = state_dict.keys()
|
||||
if not any((p + "patch_embedding.weight") in keys for p in prefixes):
|
||||
return False
|
||||
return any((p + needle) in keys for p in prefixes for needle in text_proj_options)
|
||||
|
||||
|
||||
def _is_native_wan_layout(state_dict: dict[str | int, Any]) -> bool:
|
||||
"""True if the state dict uses the native upstream Wan key layout.
|
||||
|
||||
Native layout uses ``text_embedding.0/2``, ``self_attn``/``cross_attn``,
|
||||
``ffn.0/2``, ``head.head``, ``head.modulation``, etc. — what ComfyUI and
|
||||
QuantStack ship. Diffusers layout uses ``condition_embedder.*``, ``attn1``/
|
||||
``attn2``, ``ffn.net.*``, ``proj_out``, ``scale_shift_table``.
|
||||
"""
|
||||
prefixes = ("", "model.diffusion_model.", "diffusion_model.")
|
||||
keys = state_dict.keys()
|
||||
return any((p + "text_embedding.0.weight") in keys for p in prefixes)
|
||||
|
||||
|
||||
def _detect_wan_gguf_variant(state_dict: dict[str | int, Any]) -> WanVariantType | None:
|
||||
"""Determine A14B (T2V vs I2V) vs TI2V-5B from the GGUF state dict.
|
||||
|
||||
``patch_embedding.weight`` has shape ``[inner_dim, in_channels, T, H, W]``;
|
||||
``in_channels`` uniquely identifies the Wan 2.2 variant:
|
||||
|
||||
- 16 → T2V-A14B (noise latents only).
|
||||
- 36 → I2V-A14B (16 noise + 16 ref-image latents + 4 first-frame mask,
|
||||
concatenated along the channel dim — see diffusers
|
||||
``WanImageToVideoPipeline.prepare_latents``).
|
||||
- 48 → TI2V-5B (Wan2.2-VAE z_dim=48).
|
||||
|
||||
Returns None if the tensor is missing or the channel count is unrecognised.
|
||||
"""
|
||||
candidates = (
|
||||
"patch_embedding.weight",
|
||||
"model.diffusion_model.patch_embedding.weight",
|
||||
"diffusion_model.patch_embedding.weight",
|
||||
)
|
||||
for key in candidates:
|
||||
if key in state_dict:
|
||||
tensor = state_dict[key]
|
||||
shape = getattr(tensor, "tensor_shape", None) or getattr(tensor, "shape", None)
|
||||
if shape is None or len(shape) < 2:
|
||||
return None
|
||||
in_channels = int(shape[1])
|
||||
if in_channels == 16:
|
||||
return WanVariantType.T2V_A14B
|
||||
if in_channels == 36:
|
||||
return WanVariantType.I2V_A14B
|
||||
if in_channels == 48:
|
||||
return WanVariantType.TI2V_5B
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _detect_wan_gguf_expert(filename: str) -> Literal["high", "low", "none"]:
|
||||
"""Filename heuristic for the A14B dual-expert MoE.
|
||||
|
||||
Community releases tag each expert in the filename — typically
|
||||
``high_noise`` / ``low_noise`` (or hyphenated/concatenated variants).
|
||||
Returns 'none' when neither marker is present (single-expert model or
|
||||
ambiguous filename).
|
||||
"""
|
||||
name = filename.lower()
|
||||
if any(s in name for s in ("high_noise", "high-noise", "highnoise")):
|
||||
return "high"
|
||||
if any(s in name for s in ("low_noise", "low-noise", "lownoise")):
|
||||
return "low"
|
||||
return "none"
|
||||
|
||||
|
||||
class Main_GGUF_Wan_Config(Checkpoint_Config_Base, Main_Config_Base, Config_Base):
|
||||
"""Model config for GGUF-quantized Wan 2.2 transformer models.
|
||||
|
||||
A14B's MoE ships as two GGUF files (one per expert); ``expert`` records
|
||||
which one this is so the model loader invocation can pair them. TI2V-5B
|
||||
is a single-transformer model and stores ``expert='none'``.
|
||||
"""
|
||||
|
||||
base: Literal[BaseModelType.Wan] = Field(default=BaseModelType.Wan)
|
||||
format: Literal[ModelFormat.GGUFQuantized] = Field(default=ModelFormat.GGUFQuantized)
|
||||
variant: WanVariantType = Field()
|
||||
expert: Literal["high", "low", "none"] = Field(
|
||||
default="none",
|
||||
description="For Wan 2.2 A14B's dual-expert MoE: 'high' for the high-noise expert, "
|
||||
"'low' for the low-noise expert. 'none' for single-transformer models (TI2V-5B).",
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
|
||||
raise_if_not_file(mod)
|
||||
raise_for_override_fields(cls, override_fields)
|
||||
|
||||
sd = mod.load_state_dict()
|
||||
|
||||
if not _has_ggml_tensors(sd):
|
||||
raise NotAMatchError("state dict does not look like GGUF quantized")
|
||||
if not _has_wan_keys(sd):
|
||||
raise NotAMatchError("state dict does not look like a Wan transformer")
|
||||
gguf_name = mod.metadata().get("general.name", "")
|
||||
normalized_identity = "".join(
|
||||
character for character in f"{mod.path.stem} {gguf_name}".lower() if character.isalnum()
|
||||
)
|
||||
if "wan21" in normalized_identity:
|
||||
raise NotAMatchError("Wan 2.1 GGUF models are not supported by the Wan 2.2 loader")
|
||||
|
||||
explicit_variant = override_fields.pop("variant", None)
|
||||
variant = explicit_variant or _detect_wan_gguf_variant(sd)
|
||||
if variant is None:
|
||||
raise NotAMatchError("could not determine Wan variant from state dict")
|
||||
if variant in (WanVariantType.T2V_A14B, WanVariantType.I2V_A14B) and "wan22" not in normalized_identity:
|
||||
raise NotAMatchError("Wan A14B GGUF filename or metadata must identify the model as Wan 2.2")
|
||||
|
||||
explicit_expert = override_fields.pop("expert", None)
|
||||
expert = explicit_expert or _detect_wan_gguf_expert(mod.path.stem)
|
||||
|
||||
return cls(**override_fields, variant=variant, expert=expert)
|
||||
|
||||
|
||||
class Main_Diffusers_Wan_Config(Diffusers_Config_Base, Main_Config_Base, Config_Base):
|
||||
"""Model config for Wan 2.2 diffusers models.
|
||||
|
||||
Covers both the dual-expert T2V-A14B family and the single-transformer TI2V-5B
|
||||
family. Variant is detected from the on-disk transformer config (latent channel
|
||||
count) plus the presence of a sibling ``transformer_2/`` directory.
|
||||
"""
|
||||
|
||||
base: Literal[BaseModelType.Wan] = Field(default=BaseModelType.Wan)
|
||||
variant: WanVariantType = Field()
|
||||
has_dual_expert: bool = Field(
|
||||
default=False,
|
||||
description="Whether this model ships two transformer experts (Wan 2.2 A14B MoE). False for TI2V-5B.",
|
||||
)
|
||||
boundary_ratio: float | None = Field(
|
||||
default=None,
|
||||
description="MoE expert switch point as a fraction of num_train_timesteps (typically 1000). "
|
||||
"None for single-transformer models. Read from model_index.json by Diffusers' WanPipeline.",
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
|
||||
raise_if_not_dir(mod)
|
||||
|
||||
raise_for_override_fields(cls, override_fields)
|
||||
|
||||
# Wan repos ship with WanPipeline (T2V) or WanImageToVideoPipeline (I2V/TI2V).
|
||||
# Either class name is sufficient to identify a Wan diffusers model.
|
||||
raise_for_class_name(
|
||||
common_config_paths(mod.path),
|
||||
{
|
||||
"WanPipeline",
|
||||
"WanImageToVideoPipeline",
|
||||
},
|
||||
)
|
||||
|
||||
repo_variant = override_fields.pop("repo_variant", None) or cls._get_repo_variant_or_raise(mod)
|
||||
|
||||
explicit_variant = override_fields.pop("variant", None)
|
||||
has_dual_expert = (mod.path / "transformer_2" / "config.json").exists()
|
||||
variant = explicit_variant or cls._detect_wan_variant(mod, has_dual_expert)
|
||||
boundary_ratio = override_fields.pop("boundary_ratio", None)
|
||||
if boundary_ratio is None:
|
||||
boundary_ratio = cls._read_boundary_ratio(mod)
|
||||
|
||||
return cls(
|
||||
**override_fields,
|
||||
repo_variant=repo_variant,
|
||||
variant=variant,
|
||||
has_dual_expert=has_dual_expert,
|
||||
boundary_ratio=boundary_ratio,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _read_boundary_ratio(cls, mod: ModelOnDisk) -> float | None:
|
||||
"""Pull ``boundary_ratio`` from ``model_index.json`` if present.
|
||||
|
||||
Diffusers' ``WanPipeline.__init__`` registers it via ``register_to_config``,
|
||||
which persists it as a top-level key in the saved pipeline config.
|
||||
"""
|
||||
try:
|
||||
model_index = get_config_dict_or_raise(mod.path / "model_index.json")
|
||||
except NotAMatchError:
|
||||
return None
|
||||
value = model_index.get("boundary_ratio")
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
def _detect_wan_variant(cls, mod: ModelOnDisk, has_dual_expert: bool) -> WanVariantType:
|
||||
"""Detect Wan variant from transformer + VAE config.
|
||||
|
||||
- T2V-A14B: dual transformer experts, standard Wan VAE (z_dim=16),
|
||||
transformer ``in_channels=16`` (text-only conditioning).
|
||||
- I2V-A14B: dual transformer experts, standard Wan VAE,
|
||||
transformer ``in_channels=36`` (text + VAE-encoded reference image
|
||||
+ first-frame mask concatenated along the channel dim).
|
||||
- TI2V-5B: single transformer, Wan2.2-VAE (z_dim=48).
|
||||
"""
|
||||
if has_dual_expert:
|
||||
# Disambiguate T2V vs I2V via the transformer's input channel count.
|
||||
# Wan 2.2 I2V uses VAE-latent concatenation: 16 noise + 16 ref-image
|
||||
# latents + 4 first-frame mask = 36. (Wan 2.1 I2V used CLIP-vision
|
||||
# via ``image_dim``; that mechanism is absent in Wan 2.2.)
|
||||
in_channels = cls._transformer_in_channels(mod)
|
||||
if in_channels == 36:
|
||||
return WanVariantType.I2V_A14B
|
||||
return WanVariantType.T2V_A14B
|
||||
|
||||
# Single-transformer model: distinguish TI2V-5B from any future single-expert
|
||||
# A14B-derived release by inspecting the VAE latent dimension.
|
||||
try:
|
||||
vae_config = get_config_dict_or_raise(mod.path / "vae" / "config.json")
|
||||
z_dim = vae_config.get("z_dim")
|
||||
if z_dim is not None and int(z_dim) >= 32:
|
||||
return WanVariantType.TI2V_5B
|
||||
except NotAMatchError:
|
||||
# No VAE config to inspect — fall through to the heuristic path below.
|
||||
pass
|
||||
|
||||
# Filename / repo-name heuristic as a last resort.
|
||||
name = mod.path.name.lower()
|
||||
if "5b" in name or "ti2v" in name:
|
||||
return WanVariantType.TI2V_5B
|
||||
return WanVariantType.T2V_A14B
|
||||
|
||||
@staticmethod
|
||||
def _transformer_in_channels(mod: ModelOnDisk) -> int | None:
|
||||
"""Read ``in_channels`` from ``transformer/config.json``.
|
||||
|
||||
For Wan 2.2 A14B, this is the canonical discriminator between T2V
|
||||
(``in_channels=16``) and I2V (``in_channels=36``). Returns None if the
|
||||
config can't be read.
|
||||
"""
|
||||
try:
|
||||
transformer_config = get_config_dict_or_raise(mod.path / "transformer" / "config.json")
|
||||
except NotAMatchError:
|
||||
return None
|
||||
value = transformer_config.get("in_channels")
|
||||
try:
|
||||
return int(value) if value is not None else None
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
class Main_Checkpoint_Anima_Config(Checkpoint_Config_Base, Main_Config_Base, Config_Base):
|
||||
"""Model config for Anima single-file checkpoint models (safetensors).
|
||||
|
||||
|
||||
@@ -40,6 +40,11 @@ def _is_qwen_image_vae(state_dict: dict[str | int, Any]) -> bool:
|
||||
1. Diffusers-format encoder/decoder keys (`encoder.conv_in`, `decoder.conv_in`)
|
||||
2. 5-dimensional convolution weights (3D causal convolutions vs. standard 2D conv in SD/SDXL/FLUX VAEs)
|
||||
3. 16-dimensional latent space (z_dim=16)
|
||||
|
||||
Note: Wan 2.2 A14B reuses the same architecture (AutoencoderKLWan with z_dim=16),
|
||||
so this function returns True for both. Disambiguation between the two for
|
||||
standalone files relies on the filename heuristic in :func:`_is_wan_vae` and
|
||||
config registration order.
|
||||
"""
|
||||
decoder_conv_in_key = "decoder.conv_in.weight"
|
||||
if decoder_conv_in_key not in state_dict:
|
||||
@@ -52,6 +57,34 @@ def _is_qwen_image_vae(state_dict: dict[str | int, Any]) -> bool:
|
||||
return shape[1] == 16
|
||||
|
||||
|
||||
def _wan_vae_z_dim(state_dict: dict[str | int, Any]) -> int | None:
|
||||
"""Return ``z_dim`` for a Wan-family VAE state dict, or ``None`` if it isn't one.
|
||||
|
||||
Wan-family VAEs (AutoencoderKLWan) have 5D convolution weights and a
|
||||
decoder.conv_in input channel count of 16 (Wan 2.1 / A14B / Qwen Image) or
|
||||
48 (Wan 2.2 TI2V-5B's Wan2.2-VAE).
|
||||
"""
|
||||
decoder_conv_in_key = "decoder.conv_in.weight"
|
||||
if decoder_conv_in_key not in state_dict:
|
||||
return None
|
||||
weight = state_dict[decoder_conv_in_key]
|
||||
shape = getattr(weight, "shape", None)
|
||||
if shape is None or len(shape) != 5:
|
||||
return None
|
||||
z = int(shape[1])
|
||||
return z if z in (16, 48) else None
|
||||
|
||||
|
||||
def _filename_suggests_wan(mod: ModelOnDisk) -> bool:
|
||||
"""Filename heuristic to distinguish standalone Wan VAE files from Qwen Image VAEs.
|
||||
|
||||
Both use the same ``AutoencoderKLWan`` architecture for 16-channel files, so the
|
||||
state dict alone can't tell them apart. Filenames in the wild (community ports,
|
||||
ComfyUI repacks) typically include ``wan`` for Wan releases.
|
||||
"""
|
||||
return "wan" in mod.path.name.lower()
|
||||
|
||||
|
||||
def _is_flux2_vae(state_dict: dict[str | int, Any]) -> bool:
|
||||
"""Check if state dict is a FLUX.2 VAE (AutoencoderKLFlux2).
|
||||
|
||||
@@ -114,9 +147,10 @@ class VAE_Checkpoint_Config_Base(Checkpoint_Config_Base):
|
||||
if _is_flux2_vae(state_dict):
|
||||
raise NotAMatchError("model is a FLUX.2 VAE, not a standard VAE")
|
||||
|
||||
# Exclude Qwen Image VAEs - they have their own config class
|
||||
if _is_qwen_image_vae(state_dict):
|
||||
raise NotAMatchError("model is a Qwen Image VAE, not a standard VAE")
|
||||
# Exclude Qwen Image / Wan VAEs - they share the AutoencoderKLWan
|
||||
# architecture and each has its own config class.
|
||||
if _is_qwen_image_vae(state_dict) or _wan_vae_z_dim(state_dict) is not None:
|
||||
raise NotAMatchError("model is a Wan-family VAE, not a standard VAE")
|
||||
|
||||
@classmethod
|
||||
def _get_base_or_raise(cls, mod: ModelOnDisk) -> BaseModelType:
|
||||
@@ -218,9 +252,98 @@ class VAE_Checkpoint_QwenImage_Config(Checkpoint_Config_Base, Config_Base):
|
||||
if not _is_qwen_image_vae(state_dict):
|
||||
raise NotAMatchError("state dict does not look like a Qwen Image VAE")
|
||||
|
||||
# Defer to VAE_Checkpoint_Wan_Config for files whose names indicate Wan
|
||||
# (both architectures are 16-channel AutoencoderKLWan and otherwise
|
||||
# indistinguishable from the state dict alone).
|
||||
if _filename_suggests_wan(mod):
|
||||
raise NotAMatchError("filename suggests a Wan VAE, not Qwen Image")
|
||||
|
||||
return cls(**override_fields)
|
||||
|
||||
|
||||
class VAE_Checkpoint_Wan_Config(Checkpoint_Config_Base, Config_Base):
|
||||
"""Model config for Wan 2.2 VAE checkpoint models (AutoencoderKLWan).
|
||||
|
||||
Distinguishes A14B (z_dim=16, standard Wan VAE) from TI2V-5B (z_dim=48,
|
||||
Wan2.2-VAE) via the input channel count of ``decoder.conv_in.weight``.
|
||||
"""
|
||||
|
||||
type: Literal[ModelType.VAE] = Field(default=ModelType.VAE)
|
||||
format: Literal[ModelFormat.Checkpoint] = Field(default=ModelFormat.Checkpoint)
|
||||
base: Literal[BaseModelType.Wan] = Field(default=BaseModelType.Wan)
|
||||
cpu_only: bool | None = Field(default=None, description="Whether this model should run on CPU only")
|
||||
latent_channels: Literal[16, 48] = Field(
|
||||
description="VAE latent channel count: 16 for A14B (standard Wan VAE) or 48 for TI2V-5B (Wan2.2-VAE)."
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
|
||||
raise_if_not_file(mod)
|
||||
|
||||
raise_for_override_fields(cls, override_fields)
|
||||
|
||||
state_dict = mod.load_state_dict()
|
||||
z_dim = _wan_vae_z_dim(state_dict)
|
||||
if z_dim is None:
|
||||
raise NotAMatchError("state dict does not look like a Wan VAE")
|
||||
|
||||
# 48-channel files are unambiguously Wan2.2-VAE (TI2V-5B). 16-channel
|
||||
# files are architecturally identical to Qwen Image's VAE; require the
|
||||
# filename to suggest Wan to claim them, otherwise let the QwenImage
|
||||
# config win.
|
||||
latent_channels: int = z_dim
|
||||
if latent_channels == 16 and not _filename_suggests_wan(mod):
|
||||
raise NotAMatchError(
|
||||
"16-channel AutoencoderKLWan VAE without 'wan' in filename — deferring to Qwen Image VAE config."
|
||||
)
|
||||
|
||||
explicit = override_fields.pop("latent_channels", None)
|
||||
if explicit is not None:
|
||||
latent_channels = int(explicit)
|
||||
|
||||
return cls(**override_fields, latent_channels=latent_channels)
|
||||
|
||||
|
||||
class VAE_Diffusers_Wan_Config(Diffusers_Config_Base, Config_Base):
|
||||
"""Model config for Wan 2.2 VAE in diffusers folder layout (AutoencoderKLWan)."""
|
||||
|
||||
type: Literal[ModelType.VAE] = Field(default=ModelType.VAE)
|
||||
format: Literal[ModelFormat.Diffusers] = Field(default=ModelFormat.Diffusers)
|
||||
base: Literal[BaseModelType.Wan] = Field(default=BaseModelType.Wan)
|
||||
cpu_only: bool | None = Field(default=None, description="Whether this model should run on CPU only")
|
||||
latent_channels: Literal[16, 48] = Field(
|
||||
default=16,
|
||||
description="VAE latent channel count: 16 for A14B or 48 for TI2V-5B's Wan2.2-VAE.",
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
|
||||
raise_if_not_dir(mod)
|
||||
|
||||
raise_for_override_fields(cls, override_fields)
|
||||
|
||||
raise_for_class_name(
|
||||
common_config_paths(mod.path),
|
||||
{"AutoencoderKLWan"},
|
||||
)
|
||||
|
||||
# Read z_dim from the diffusers config to set latent_channels.
|
||||
latent_channels: int = 16
|
||||
try:
|
||||
config = get_config_dict_or_raise(common_config_paths(mod.path))
|
||||
z = config.get("z_dim")
|
||||
if z is not None and int(z) in (16, 48):
|
||||
latent_channels = int(z)
|
||||
except NotAMatchError:
|
||||
pass
|
||||
|
||||
explicit = override_fields.pop("latent_channels", None)
|
||||
if explicit is not None:
|
||||
latent_channels = int(explicit)
|
||||
|
||||
return cls(**override_fields, latent_channels=latent_channels)
|
||||
|
||||
|
||||
def _has_anima_vae_keys(state_dict: dict[str | int, Any]) -> bool:
|
||||
"""Check if state dict looks like an Anima QwenImage VAE (AutoencoderKLQwenImage).
|
||||
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Configurations for the UMT5-XXL text encoder used by Wan 2.2.
|
||||
|
||||
Wan ships a UMT5-XXL encoder (not the more common T5-XXL). The two are not
|
||||
weight-compatible — UMT5 has a different vocabulary and ``model_type``. We
|
||||
register a dedicated config + ModelType so users can't accidentally wire a
|
||||
FLUX/SD3-style T5-XXL into a Wan slot.
|
||||
|
||||
For Phase 3 we accept the diffusers-folder layout only. Single-file UMT5
|
||||
checkpoints are uncommon; if they show up later, a checkpoint config can be
|
||||
added alongside this one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal, Self
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
from invokeai.backend.model_manager.configs.base import Config_Base
|
||||
from invokeai.backend.model_manager.configs.identification_utils import (
|
||||
NotAMatchError,
|
||||
raise_for_override_fields,
|
||||
raise_if_not_dir,
|
||||
)
|
||||
from invokeai.backend.model_manager.model_on_disk import ModelOnDisk
|
||||
from invokeai.backend.model_manager.taxonomy import BaseModelType, ModelFormat, ModelType
|
||||
|
||||
|
||||
def _read_text_encoder_model_type(mod: ModelOnDisk) -> str | None:
|
||||
"""Return ``model_type`` from the encoder's ``config.json``.
|
||||
|
||||
Diffusers encoder folders may live at the root (``config.json``) or under a
|
||||
``text_encoder/`` subdirectory. UMT5-XXL sets ``model_type`` to ``"umt5"``;
|
||||
a regular T5-XXL would be ``"t5"``.
|
||||
"""
|
||||
candidates: list[Path] = [mod.path / "text_encoder" / "config.json", mod.path / "config.json"]
|
||||
for path in candidates:
|
||||
if path.exists():
|
||||
try:
|
||||
with path.open("r", encoding="utf-8") as f:
|
||||
config = json.load(f)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
continue
|
||||
mt = config.get("model_type")
|
||||
if isinstance(mt, str):
|
||||
return mt.lower()
|
||||
return None
|
||||
|
||||
|
||||
class WanT5Encoder_WanT5Encoder_Config(Config_Base):
|
||||
"""UMT5-XXL encoder in diffusers folder layout.
|
||||
|
||||
Accepts either:
|
||||
- A directory containing ``text_encoder/`` (and typically ``tokenizer/``) ─ the
|
||||
shape produced by ``Wan-AI/Wan2.2-T2V-A14B::text_encoder+tokenizer``.
|
||||
- A bare ``text_encoder/`` directory whose own ``config.json`` declares
|
||||
``model_type: umt5``.
|
||||
"""
|
||||
|
||||
base: Literal[BaseModelType.Any] = Field(default=BaseModelType.Any)
|
||||
type: Literal[ModelType.WanT5Encoder] = Field(default=ModelType.WanT5Encoder)
|
||||
format: Literal[ModelFormat.WanT5Encoder] = Field(default=ModelFormat.WanT5Encoder)
|
||||
|
||||
cpu_only: bool | None = Field(default=None, description="Whether this model should run on CPU only")
|
||||
|
||||
@classmethod
|
||||
def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -> Self:
|
||||
raise_if_not_dir(mod)
|
||||
raise_for_override_fields(cls, override_fields)
|
||||
|
||||
# Refuse to claim full Wan pipelines — they should match Main_Diffusers_Wan_Config.
|
||||
if (mod.path / "model_index.json").exists() or (mod.path / "transformer").exists():
|
||||
raise NotAMatchError(
|
||||
"directory looks like a full Wan pipeline (model_index.json or transformer/), "
|
||||
"not a standalone Wan T5 encoder"
|
||||
)
|
||||
|
||||
model_type = _read_text_encoder_model_type(mod)
|
||||
if model_type is None:
|
||||
raise NotAMatchError("no encoder config.json found at root or text_encoder/")
|
||||
if model_type != "umt5":
|
||||
raise NotAMatchError(f"encoder model_type is {model_type!r}, not 'umt5'")
|
||||
|
||||
return cls(**override_fields)
|
||||
@@ -63,6 +63,7 @@ from invokeai.backend.patches.lora_conversions.qwen_image_lora_conversion_utils
|
||||
)
|
||||
from invokeai.backend.patches.lora_conversions.sd_lora_conversion_utils import lora_model_from_sd_state_dict
|
||||
from invokeai.backend.patches.lora_conversions.sdxl_lora_conversion_utils import convert_sdxl_keys_to_diffusers_format
|
||||
from invokeai.backend.patches.lora_conversions.wan_lora_conversion_utils import lora_model_from_wan_state_dict
|
||||
from invokeai.backend.patches.lora_conversions.z_image_lora_conversion_utils import lora_model_from_z_image_state_dict
|
||||
|
||||
|
||||
@@ -175,6 +176,10 @@ class LoRALoader(ModelLoader):
|
||||
elif self._model_base == BaseModelType.Anima:
|
||||
# Anima LoRAs use Kohya-style or diffusers PEFT format targeting Cosmos DiT blocks.
|
||||
model = lora_model_from_anima_state_dict(state_dict=state_dict, alpha=None)
|
||||
elif self._model_base == BaseModelType.Wan:
|
||||
# Wan LoRAs use Kohya / diffusers PEFT / native PEFT formats targeting
|
||||
# WanTransformer3DModel attention (attn1/attn2) and FFN blocks.
|
||||
model = lora_model_from_wan_state_dict(state_dict=state_dict, alpha=None)
|
||||
else:
|
||||
raise ValueError(f"Unsupported LoRA base model: {self._model_base}")
|
||||
|
||||
|
||||
@@ -10,6 +10,8 @@ from invokeai.backend.model_manager.configs.vae import (
|
||||
VAE_Checkpoint_Anima_Config,
|
||||
VAE_Checkpoint_Config_Base,
|
||||
VAE_Checkpoint_QwenImage_Config,
|
||||
VAE_Checkpoint_Wan_Config,
|
||||
VAE_Diffusers_Wan_Config,
|
||||
)
|
||||
from invokeai.backend.model_manager.load.model_loader_registry import ModelLoaderRegistry
|
||||
from invokeai.backend.model_manager.load.model_loaders.generic_diffusers import GenericDiffusersLoader
|
||||
@@ -21,6 +23,133 @@ from invokeai.backend.model_manager.taxonomy import (
|
||||
SubModelType,
|
||||
)
|
||||
|
||||
# Architectural defaults for the Wan 2.2-VAE (TI2V-5B). Verbatim from the
|
||||
# vae/config.json shipped with Wan-AI/Wan2.2-TI2V-5B-Diffusers — only the
|
||||
# values that differ from diffusers' AutoencoderKLWan defaults are listed.
|
||||
# latents_mean / latents_std are required because the model normalises latents
|
||||
# against them at encode/decode time; the wrong arrays produce silent garbage.
|
||||
_WAN_TI2V_5B_VAE_CONFIG: dict = {
|
||||
"base_dim": 160,
|
||||
"decoder_base_dim": 256,
|
||||
"z_dim": 48,
|
||||
"in_channels": 12,
|
||||
"out_channels": 12,
|
||||
"patch_size": 2,
|
||||
"scale_factor_spatial": 16,
|
||||
"is_residual": True,
|
||||
"latents_mean": [
|
||||
-0.2289,
|
||||
-0.0052,
|
||||
-0.1323,
|
||||
-0.2339,
|
||||
-0.2799,
|
||||
0.0174,
|
||||
0.1838,
|
||||
0.1557,
|
||||
-0.1382,
|
||||
0.0542,
|
||||
0.2813,
|
||||
0.0891,
|
||||
0.1570,
|
||||
-0.0098,
|
||||
0.0375,
|
||||
-0.1825,
|
||||
-0.2246,
|
||||
-0.1207,
|
||||
-0.0698,
|
||||
0.5109,
|
||||
0.2665,
|
||||
-0.2108,
|
||||
-0.2158,
|
||||
0.2502,
|
||||
-0.2055,
|
||||
-0.0322,
|
||||
0.1109,
|
||||
0.1567,
|
||||
-0.0729,
|
||||
0.0899,
|
||||
-0.2799,
|
||||
-0.1230,
|
||||
-0.0313,
|
||||
-0.1649,
|
||||
0.0117,
|
||||
0.0723,
|
||||
-0.2839,
|
||||
-0.2083,
|
||||
-0.0520,
|
||||
0.3748,
|
||||
0.0152,
|
||||
0.1957,
|
||||
0.1433,
|
||||
-0.2944,
|
||||
0.3573,
|
||||
-0.0548,
|
||||
-0.1681,
|
||||
-0.0667,
|
||||
],
|
||||
"latents_std": [
|
||||
0.4765,
|
||||
1.0364,
|
||||
0.4514,
|
||||
1.1677,
|
||||
0.5313,
|
||||
0.4990,
|
||||
0.4818,
|
||||
0.5013,
|
||||
0.8158,
|
||||
1.0344,
|
||||
0.5894,
|
||||
1.0901,
|
||||
0.6885,
|
||||
0.6165,
|
||||
0.8454,
|
||||
0.4978,
|
||||
0.5759,
|
||||
0.3523,
|
||||
0.7135,
|
||||
0.6804,
|
||||
0.5833,
|
||||
1.4146,
|
||||
0.8986,
|
||||
0.5659,
|
||||
0.7069,
|
||||
0.5338,
|
||||
0.4889,
|
||||
0.4917,
|
||||
0.4069,
|
||||
0.4999,
|
||||
0.6866,
|
||||
0.4093,
|
||||
0.5709,
|
||||
0.6065,
|
||||
0.6415,
|
||||
0.4944,
|
||||
0.5726,
|
||||
1.2042,
|
||||
0.5458,
|
||||
1.6887,
|
||||
0.3971,
|
||||
1.0600,
|
||||
0.3943,
|
||||
0.5537,
|
||||
0.5444,
|
||||
0.4089,
|
||||
0.7468,
|
||||
0.7744,
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _wan_vae_init_kwargs_for(latent_channels: int) -> dict:
|
||||
"""Return the AutoencoderKLWan constructor kwargs for a given z_dim.
|
||||
|
||||
z_dim=48 means TI2V-5B's Wan 2.2-VAE (different base dim, patchified IO,
|
||||
16x spatial). Anything else falls back to the A14B / Wan 2.1 defaults.
|
||||
"""
|
||||
if latent_channels == 48:
|
||||
return dict(_WAN_TI2V_5B_VAE_CONFIG)
|
||||
return {"z_dim": latent_channels}
|
||||
|
||||
|
||||
@ModelLoaderRegistry.register(base=BaseModelType.Any, type=ModelType.VAE, format=ModelFormat.Diffusers)
|
||||
@ModelLoaderRegistry.register(base=BaseModelType.Any, type=ModelType.VAE, format=ModelFormat.Checkpoint)
|
||||
@@ -35,10 +164,17 @@ class VAELoader(GenericDiffusersLoader):
|
||||
if isinstance(config, VAE_Checkpoint_Anima_Config):
|
||||
from diffusers.models.autoencoders import AutoencoderKLWan
|
||||
|
||||
from invokeai.backend.wan.rocm_causal_conv3d import patch_wan_causal_conv3d_for_rocm
|
||||
|
||||
patch_wan_causal_conv3d_for_rocm()
|
||||
return AutoencoderKLWan.from_single_file(
|
||||
config.path,
|
||||
torch_dtype=self._torch_dtype,
|
||||
)
|
||||
elif isinstance(config, VAE_Checkpoint_Wan_Config):
|
||||
return self._load_wan_vae(config)
|
||||
elif isinstance(config, VAE_Diffusers_Wan_Config):
|
||||
return self._load_wan_vae_diffusers(config)
|
||||
elif isinstance(config, VAE_Checkpoint_QwenImage_Config):
|
||||
return self._load_qwen_image_vae(config)
|
||||
elif isinstance(config, VAE_Checkpoint_Config_Base):
|
||||
@@ -49,6 +185,77 @@ class VAELoader(GenericDiffusersLoader):
|
||||
else:
|
||||
return super()._load_model(config, submodel_type)
|
||||
|
||||
def _load_wan_vae(self, config: VAE_Checkpoint_Wan_Config) -> AnyModel:
|
||||
"""Load a Wan 2.2 VAE from a single safetensors file.
|
||||
|
||||
Picks the correct ``AutoencoderKLWan`` config based on ``z_dim``. The Wan
|
||||
ecosystem ships two distinct VAE architectures:
|
||||
|
||||
* ``z_dim=16`` — the Wan 2.1 / Wan 2.2 A14B VAE. Diffusers' defaults match
|
||||
this one (base_dim=96, 8x spatial, no patchify, 3 in/out channels).
|
||||
* ``z_dim=48`` — the Wan 2.2-VAE used by TI2V-5B. Larger (base_dim=160,
|
||||
decoder_base_dim=256), 16x spatial, patchify with patch_size=2 (so
|
||||
in/out channels are 12 = 3 RGB x 2x2 patch), residual blocks, and
|
||||
its own latents_mean / latents_std.
|
||||
|
||||
Without overriding those params at construction time, the state dict
|
||||
from the TI2V-5B VAE checkpoint won't load (channel and shape mismatches
|
||||
throughout the encoder + decoder).
|
||||
|
||||
Forces bfloat16 (same as ``WanDiffusersModel`` and ``_load_wan_vae_diffusers``) —
|
||||
fp16 is unstable on the Wan VAE, and the default ``precision: auto`` resolves to
|
||||
float16 on CUDA.
|
||||
"""
|
||||
import accelerate
|
||||
import torch
|
||||
from diffusers.models.autoencoders.autoencoder_kl_wan import AutoencoderKLWan
|
||||
from safetensors.torch import load_file
|
||||
|
||||
from invokeai.backend.wan.rocm_causal_conv3d import patch_wan_causal_conv3d_for_rocm
|
||||
|
||||
patch_wan_causal_conv3d_for_rocm()
|
||||
sd = load_file(config.path)
|
||||
|
||||
for k in list(sd.keys()):
|
||||
if sd[k].is_floating_point():
|
||||
sd[k] = sd[k].to(torch.bfloat16)
|
||||
|
||||
new_sd_size = sum(t.nelement() * t.element_size() for t in sd.values())
|
||||
self._ram_cache.make_room(new_sd_size)
|
||||
|
||||
init_kwargs = _wan_vae_init_kwargs_for(config.latent_channels)
|
||||
with accelerate.init_empty_weights():
|
||||
model = AutoencoderKLWan(**init_kwargs)
|
||||
|
||||
model.load_state_dict(sd, strict=True, assign=True)
|
||||
model.eval()
|
||||
return model
|
||||
|
||||
def _load_wan_vae_diffusers(self, config: VAE_Diffusers_Wan_Config) -> AnyModel:
|
||||
"""Load a Wan 2.2 VAE from a flat diffusers folder (AutoencoderKLWan).
|
||||
|
||||
The standalone install ``Wan-AI/Wan2.2-T2V-A14B-Diffusers::vae`` lands as a
|
||||
single-class folder (``config.json`` + ``diffusion_pytorch_model.safetensors``,
|
||||
no ``model_index.json``). The generic loader rejects this when a
|
||||
``submodel_type`` is requested — we always pass ``SubModelType.VAE`` from
|
||||
the model loader invocation since that's how cached entries are keyed.
|
||||
Loading ``AutoencoderKLWan`` directly here sidesteps the submodel check.
|
||||
|
||||
Forces bfloat16 (same as ``WanDiffusersModel``) — fp16 is unstable on the
|
||||
Wan VAE.
|
||||
"""
|
||||
import torch
|
||||
from diffusers.models.autoencoders.autoencoder_kl_wan import AutoencoderKLWan
|
||||
|
||||
from invokeai.backend.wan.rocm_causal_conv3d import patch_wan_causal_conv3d_for_rocm
|
||||
|
||||
patch_wan_causal_conv3d_for_rocm()
|
||||
return AutoencoderKLWan.from_pretrained(
|
||||
config.path,
|
||||
torch_dtype=torch.bfloat16,
|
||||
local_files_only=True,
|
||||
)
|
||||
|
||||
def _load_qwen_image_vae(self, config: VAE_Checkpoint_QwenImage_Config) -> AnyModel:
|
||||
"""Load a Qwen Image VAE from a single safetensors file.
|
||||
|
||||
|
||||
@@ -0,0 +1,357 @@
|
||||
"""Loader registrations for Wan 2.2 image-generation models.
|
||||
|
||||
Currently covers:
|
||||
- Main: Diffusers format (T2V-A14B with dual experts via Transformer +
|
||||
Transformer2 submodels, plus TI2V-5B). Phase 4 will add a GGUFQuantized loader.
|
||||
- WanT5Encoder: standalone UMT5-XXL encoder folder (``text_encoder/`` +
|
||||
``tokenizer/`` subdirs, or a flat ``text_encoder/`` folder).
|
||||
- VAE: handled in ``vae.py`` (registered for type=VAE generically).
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import torch
|
||||
|
||||
from invokeai.backend.model_manager.configs.base import Checkpoint_Config_Base, Diffusers_Config_Base
|
||||
from invokeai.backend.model_manager.configs.factory import AnyModelConfig
|
||||
from invokeai.backend.model_manager.configs.main import Main_GGUF_Wan_Config, _is_native_wan_layout
|
||||
from invokeai.backend.model_manager.load.load_default import ModelLoader
|
||||
from invokeai.backend.model_manager.load.model_loader_registry import ModelLoaderRegistry
|
||||
from invokeai.backend.model_manager.load.model_loaders.generic_diffusers import GenericDiffusersLoader
|
||||
from invokeai.backend.model_manager.taxonomy import (
|
||||
AnyModel,
|
||||
BaseModelType,
|
||||
ModelFormat,
|
||||
ModelType,
|
||||
SubModelType,
|
||||
WanVariantType,
|
||||
)
|
||||
from invokeai.backend.quantization.gguf.ggml_tensor import GGMLTensor
|
||||
from invokeai.backend.quantization.gguf.loaders import gguf_sd_loader
|
||||
from invokeai.backend.quantization.gguf.utils import TORCH_COMPATIBLE_QTYPES
|
||||
from invokeai.backend.util.devices import TorchDevice
|
||||
|
||||
|
||||
@ModelLoaderRegistry.register(base=BaseModelType.Wan, type=ModelType.Main, format=ModelFormat.Diffusers)
|
||||
class WanDiffusersModel(GenericDiffusersLoader):
|
||||
"""Loader for Wan 2.2 diffusers-format models (T2V-A14B and TI2V-5B).
|
||||
|
||||
Forces bfloat16 for the transformer and VAE — fp16 is unstable on Wan VAE
|
||||
(same issue affects the Flux VAE). Resolves the appropriate Hugging Face
|
||||
class for each submodel via the parent loader's ``get_hf_load_class``.
|
||||
"""
|
||||
|
||||
def _load_model(
|
||||
self,
|
||||
config: AnyModelConfig,
|
||||
submodel_type: Optional[SubModelType] = None,
|
||||
) -> AnyModel:
|
||||
if isinstance(config, Checkpoint_Config_Base):
|
||||
raise NotImplementedError("Single-file checkpoint format is not yet supported for Wan models.")
|
||||
|
||||
if submodel_type is None:
|
||||
raise Exception("A submodel type must be provided when loading Wan main pipelines.")
|
||||
|
||||
if submodel_type is SubModelType.VAE:
|
||||
from invokeai.backend.wan.rocm_causal_conv3d import patch_wan_causal_conv3d_for_rocm
|
||||
|
||||
patch_wan_causal_conv3d_for_rocm()
|
||||
|
||||
model_path = Path(config.path)
|
||||
load_class = self.get_hf_load_class(model_path, submodel_type)
|
||||
repo_variant = config.repo_variant if isinstance(config, Diffusers_Config_Base) else None
|
||||
variant = repo_variant.value if repo_variant else None
|
||||
model_path = model_path / submodel_type.value
|
||||
|
||||
def _load_with_variant_fallback(dtype_kwarg: dict[str, torch.dtype]) -> AnyModel:
|
||||
# Some Wan repos ship without a fp16 variant suffix on every submodel.
|
||||
# If the requested variant isn't on disk, fall back to the default weights.
|
||||
try:
|
||||
return load_class.from_pretrained(
|
||||
model_path,
|
||||
**dtype_kwarg,
|
||||
variant=variant,
|
||||
local_files_only=True,
|
||||
)
|
||||
except OSError as e:
|
||||
if variant and "no file named" in str(e):
|
||||
return load_class.from_pretrained(model_path, **dtype_kwarg, local_files_only=True)
|
||||
raise
|
||||
|
||||
# bfloat16 across the board: matches Diffusers WanPipeline reference and
|
||||
# avoids the fp16 instability seen in the Wan VAE.
|
||||
try:
|
||||
result: AnyModel = _load_with_variant_fallback({"dtype": torch.bfloat16})
|
||||
except TypeError:
|
||||
# Older diffusers releases use torch_dtype instead of dtype.
|
||||
result = _load_with_variant_fallback({"torch_dtype": torch.bfloat16})
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# Native (upstream) -> Diffusers key rename rules.
|
||||
#
|
||||
# Mirrors diffusers.loaders.single_file_utils.convert_wan_transformer_to_diffusers
|
||||
# (T2V subset; we don't ship VACE / motion / face-adapter conversion). Order
|
||||
# matters — `cross_attn`/`self_attn` must come before `.q. .k. .v. .o.` so the
|
||||
# attention blocks are renamed before the projection suffix swap. The norm2/3
|
||||
# swap uses a placeholder to avoid collisions during the substring rewrite.
|
||||
_WAN_NATIVE_TO_DIFFUSERS_RENAMES: tuple[tuple[str, str], ...] = (
|
||||
("time_embedding.0", "condition_embedder.time_embedder.linear_1"),
|
||||
("time_embedding.2", "condition_embedder.time_embedder.linear_2"),
|
||||
("text_embedding.0", "condition_embedder.text_embedder.linear_1"),
|
||||
("text_embedding.2", "condition_embedder.text_embedder.linear_2"),
|
||||
("time_projection.1", "condition_embedder.time_proj"),
|
||||
("cross_attn", "attn2"),
|
||||
("self_attn", "attn1"),
|
||||
(".o.", ".to_out.0."),
|
||||
(".q.", ".to_q."),
|
||||
(".k.", ".to_k."),
|
||||
(".v.", ".to_v."),
|
||||
(".k_img.", ".add_k_proj."),
|
||||
(".v_img.", ".add_v_proj."),
|
||||
(".norm_k_img.", ".norm_added_k."),
|
||||
("head.modulation", "scale_shift_table"),
|
||||
("head.head", "proj_out"),
|
||||
("modulation", "scale_shift_table"),
|
||||
("ffn.0", "ffn.net.0.proj"),
|
||||
("ffn.2", "ffn.net.2"),
|
||||
# norm2 <-> norm3 swap via placeholder
|
||||
("norm2", "norm__placeholder"),
|
||||
("norm3", "norm2"),
|
||||
("norm__placeholder", "norm3"),
|
||||
# I2V-only keys (harmless on T2V)
|
||||
("img_emb.proj.0", "condition_embedder.image_embedder.norm1"),
|
||||
("img_emb.proj.1", "condition_embedder.image_embedder.ff.net.0.proj"),
|
||||
("img_emb.proj.3", "condition_embedder.image_embedder.ff.net.2"),
|
||||
("img_emb.proj.4", "condition_embedder.image_embedder.norm2"),
|
||||
)
|
||||
|
||||
|
||||
def _convert_wan_native_to_diffusers(state_dict: dict) -> dict:
|
||||
"""Rename native upstream Wan keys (ComfyUI / QuantStack) to diffusers names.
|
||||
|
||||
Pure substring replacement — no tensor manipulation — so it's safe to apply
|
||||
to a dict of GGMLTensors. Returns a new dict; the input is not mutated.
|
||||
"""
|
||||
converted: dict = {}
|
||||
for key, value in state_dict.items():
|
||||
if not isinstance(key, str):
|
||||
converted[key] = value
|
||||
continue
|
||||
new_key = key
|
||||
for needle, replacement in _WAN_NATIVE_TO_DIFFUSERS_RENAMES:
|
||||
new_key = new_key.replace(needle, replacement)
|
||||
converted[new_key] = value
|
||||
return converted
|
||||
|
||||
|
||||
def _unwrap_unquantized_to_compute_dtype(state_dict: dict) -> dict:
|
||||
"""Replace non-quantized GGMLTensor entries with plain tensors at compute_dtype.
|
||||
|
||||
Why: QuantStack-style GGUFs store biases (and other small tensors) as F16,
|
||||
while Wan's ``patch_embedding`` is an ``nn.Conv3d``. ``conv3d`` isn't in
|
||||
GGMLTensor's dispatch table, so PyTorch reads the wrapper's underlying F16
|
||||
storage directly and crashes against bf16 latents
|
||||
(``Input type (c10::BFloat16) and bias type (c10::Half) should be the same``).
|
||||
|
||||
For compatible qtypes (F16/F32/BF16) we just pre-cast to compute_dtype here —
|
||||
they're not quantized, there's no benefit to keeping them wrapped, and
|
||||
unwrapping them sidesteps the missing-op problem entirely. Genuinely
|
||||
quantized tensors (Q4_K, Q6_K, etc.) stay wrapped — their on-demand
|
||||
dequantization through the linear/addmm dispatch path still works.
|
||||
"""
|
||||
unwrapped: dict = {}
|
||||
for key, value in state_dict.items():
|
||||
if isinstance(value, GGMLTensor) and value._ggml_quantization_type in TORCH_COMPATIBLE_QTYPES:
|
||||
# GGMLTensor.get_dequantized_tensor() already casts to compute_dtype.
|
||||
unwrapped[key] = value.get_dequantized_tensor()
|
||||
else:
|
||||
unwrapped[key] = value
|
||||
return unwrapped
|
||||
|
||||
|
||||
@ModelLoaderRegistry.register(base=BaseModelType.Wan, type=ModelType.Main, format=ModelFormat.GGUFQuantized)
|
||||
class WanGGUFCheckpointModel(ModelLoader):
|
||||
"""Loader for GGUF-quantized Wan 2.2 transformer models.
|
||||
|
||||
The community typically distributes Wan A14B as two files (one per expert
|
||||
— high-noise + low-noise). Each file is loaded independently here; the
|
||||
pairing happens at the WanModelLoaderInvocation layer. TI2V-5B ships as a
|
||||
single file.
|
||||
|
||||
Mirrors the QwenImage GGUF loader pattern: ``gguf_sd_loader`` -> strip the
|
||||
ComfyUI ``model.diffusion_model.`` / ``diffusion_model.`` prefix if present
|
||||
-> auto-detect arch from state-dict shapes -> ``init_empty_weights`` +
|
||||
``load_state_dict(strict=False, assign=True)``.
|
||||
"""
|
||||
|
||||
def _load_model(
|
||||
self,
|
||||
config: AnyModelConfig,
|
||||
submodel_type: Optional[SubModelType] = None,
|
||||
) -> AnyModel:
|
||||
if not isinstance(config, Main_GGUF_Wan_Config):
|
||||
raise TypeError(f"Expected Main_GGUF_Wan_Config, got {type(config).__name__}.")
|
||||
|
||||
if submodel_type != SubModelType.Transformer:
|
||||
raise ValueError(
|
||||
"Only the Transformer submodel is available from a GGUF Wan checkpoint. "
|
||||
"Pair with a standalone Wan VAE and Wan T5 encoder for the other components."
|
||||
)
|
||||
|
||||
return self._load_from_singlefile(config)
|
||||
|
||||
def _load_from_singlefile(self, config: Main_GGUF_Wan_Config) -> AnyModel:
|
||||
import accelerate
|
||||
from diffusers import WanTransformer3DModel
|
||||
|
||||
model_path = Path(config.path)
|
||||
target_device = TorchDevice.choose_torch_device()
|
||||
compute_dtype = TorchDevice.choose_bfloat16_safe_dtype(target_device)
|
||||
|
||||
sd = gguf_sd_loader(model_path, compute_dtype=compute_dtype)
|
||||
|
||||
# Strip ComfyUI-style prefixes if present.
|
||||
for prefix in ("model.diffusion_model.", "diffusion_model."):
|
||||
if any(isinstance(k, str) and k.startswith(prefix) for k in sd.keys()):
|
||||
sd = {
|
||||
(k[len(prefix) :] if isinstance(k, str) and k.startswith(prefix) else k): v for k, v in sd.items()
|
||||
}
|
||||
break
|
||||
|
||||
# QuantStack and other community releases ship the native upstream Wan key
|
||||
# layout (text_embedding.0, self_attn/cross_attn, ffn.0/2, head.head, ...);
|
||||
# diffusers' WanTransformer3DModel expects condition_embedder.*, attn1/attn2,
|
||||
# ffn.net.*, proj_out. Convert in place if needed.
|
||||
if _is_native_wan_layout(sd):
|
||||
sd = _convert_wan_native_to_diffusers(sd)
|
||||
|
||||
# Pre-cast non-quantized tensors (F16/F32/BF16 biases, scale_shift_table,
|
||||
# patch_embedding.weight, etc.) to compute_dtype. This avoids dtype
|
||||
# mismatches in conv3d at the input (patch_embedding is the only Conv3d
|
||||
# in WanTransformer3DModel; conv3d isn't in GGMLTensor's dispatch table
|
||||
# so the wrapper's underlying storage dtype reaches PyTorch directly).
|
||||
sd = _unwrap_unquantized_to_compute_dtype(sd)
|
||||
|
||||
# Auto-detect architecture from the state dict.
|
||||
num_layers = 0
|
||||
for key in sd.keys():
|
||||
if isinstance(key, str) and key.startswith("blocks."):
|
||||
parts = key.split(".")
|
||||
if len(parts) >= 2:
|
||||
try:
|
||||
num_layers = max(num_layers, int(parts[1]) + 1)
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
# Patch embedding gives us in_channels (16=A14B, 48=TI2V-5B) and inner dim.
|
||||
patch_w = sd.get("patch_embedding.weight")
|
||||
if patch_w is None:
|
||||
raise RuntimeError("GGUF state dict missing patch_embedding.weight after prefix strip")
|
||||
patch_shape = patch_w.tensor_shape if isinstance(patch_w, GGMLTensor) else patch_w.shape
|
||||
inner_dim = int(patch_shape[0])
|
||||
in_channels = int(patch_shape[1])
|
||||
|
||||
# Wan uses head_dim=128 throughout the family; num_heads = inner_dim / 128.
|
||||
attention_head_dim = 128
|
||||
num_attention_heads = inner_dim // attention_head_dim
|
||||
|
||||
ffn_w = sd.get("blocks.0.ffn.net.0.proj.weight")
|
||||
if ffn_w is None:
|
||||
raise RuntimeError("GGUF state dict missing blocks.0.ffn.net.0.proj.weight after prefix strip")
|
||||
ffn_shape = ffn_w.tensor_shape if isinstance(ffn_w, GGMLTensor) else ffn_w.shape
|
||||
ffn_dim = int(ffn_shape[0])
|
||||
|
||||
text_w = sd.get("condition_embedder.text_embedder.linear_1.weight")
|
||||
text_dim = 4096
|
||||
if text_w is not None:
|
||||
text_shape = text_w.tensor_shape if isinstance(text_w, GGMLTensor) else text_w.shape
|
||||
text_dim = int(text_shape[1])
|
||||
|
||||
# out_channels is read from proj_out.weight directly rather than assumed
|
||||
# equal to in_channels: I2V-A14B has in_channels=36 (16 noise + 16
|
||||
# ref-image latents + 4 mask, concatenated by the denoise loop) but
|
||||
# out_channels=16 (only the noise prediction comes back). proj_out is
|
||||
# ``nn.Linear(inner_dim, out_channels * prod(patch_size))`` and
|
||||
# patch_size is (1, 2, 2) → prod = 4 for the Wan 2.2 family.
|
||||
proj_out_w = sd.get("proj_out.weight")
|
||||
if proj_out_w is None:
|
||||
raise RuntimeError("GGUF state dict missing proj_out.weight after prefix strip")
|
||||
proj_out_shape = proj_out_w.tensor_shape if isinstance(proj_out_w, GGMLTensor) else proj_out_w.shape
|
||||
out_channels = int(proj_out_shape[0]) // 4
|
||||
|
||||
# Layer count fallback (only triggers if the auto-count loop above
|
||||
# found zero blocks, which shouldn't happen for a valid GGUF). T2V/I2V
|
||||
# A14B have 40 layers; TI2V-5B has 30.
|
||||
layer_count_fallback = 30 if config.variant == WanVariantType.TI2V_5B else 40
|
||||
|
||||
model_config: dict = {
|
||||
"patch_size": (1, 2, 2),
|
||||
"in_channels": in_channels,
|
||||
"out_channels": out_channels,
|
||||
"num_layers": num_layers if num_layers > 0 else layer_count_fallback,
|
||||
"attention_head_dim": attention_head_dim,
|
||||
"num_attention_heads": num_attention_heads,
|
||||
"ffn_dim": ffn_dim,
|
||||
"text_dim": text_dim,
|
||||
}
|
||||
|
||||
with accelerate.init_empty_weights():
|
||||
model = WanTransformer3DModel(**model_config)
|
||||
|
||||
incompatible_keys = model.load_state_dict(sd, strict=False, assign=True)
|
||||
if incompatible_keys.missing_keys:
|
||||
raise RuntimeError(f"GGUF state dict is missing model parameters: {incompatible_keys.missing_keys}")
|
||||
return model
|
||||
|
||||
|
||||
@ModelLoaderRegistry.register(base=BaseModelType.Any, type=ModelType.WanT5Encoder, format=ModelFormat.WanT5Encoder)
|
||||
class WanT5EncoderLoader(ModelLoader):
|
||||
"""Loader for the standalone Wan UMT5-XXL encoder.
|
||||
|
||||
Accepts two on-disk layouts:
|
||||
1. Parent dir with ``text_encoder/`` (and typically ``tokenizer/``) subdirs —
|
||||
what ``Wan-AI/Wan2.2-T2V-A14B::text_encoder+tokenizer`` produces.
|
||||
2. A flat ``text_encoder/`` folder with ``config.json`` declaring
|
||||
``model_type: umt5`` directly at the root. In this case the tokenizer
|
||||
is loaded from the same folder via ``AutoTokenizer.from_pretrained``.
|
||||
"""
|
||||
|
||||
def _load_model(
|
||||
self,
|
||||
config: AnyModelConfig,
|
||||
submodel_type: Optional[SubModelType] = None,
|
||||
) -> AnyModel:
|
||||
if submodel_type is None:
|
||||
raise ValueError("A submodel type (Tokenizer or TextEncoder) must be provided.")
|
||||
|
||||
root = Path(config.path)
|
||||
nested_text_encoder = root / "text_encoder"
|
||||
nested_tokenizer = root / "tokenizer"
|
||||
|
||||
if submodel_type == SubModelType.TextEncoder:
|
||||
from transformers import UMT5EncoderModel
|
||||
|
||||
target = nested_text_encoder if nested_text_encoder.exists() else root
|
||||
return UMT5EncoderModel.from_pretrained(
|
||||
str(target),
|
||||
torch_dtype=torch.bfloat16,
|
||||
local_files_only=True,
|
||||
)
|
||||
if submodel_type == SubModelType.Tokenizer:
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
# Prefer a sibling tokenizer/ directory; fall back to the encoder dir
|
||||
# itself, which is normal for "flat" downloads.
|
||||
target = (
|
||||
nested_tokenizer
|
||||
if nested_tokenizer.exists()
|
||||
else (nested_text_encoder if nested_text_encoder.exists() else root)
|
||||
)
|
||||
return AutoTokenizer.from_pretrained(str(target), local_files_only=True)
|
||||
|
||||
raise ValueError(
|
||||
f"Unsupported submodel type for WanT5Encoder: {submodel_type.value if submodel_type else 'None'}"
|
||||
)
|
||||
@@ -3,13 +3,14 @@ from typing import Any, Optional, TypeAlias
|
||||
|
||||
import safetensors.torch
|
||||
import torch
|
||||
from gguf import GGUFValueType
|
||||
from picklescan.scanner import scan_file_path
|
||||
from safetensors import safe_open
|
||||
|
||||
from invokeai.app.services.config.config_default import get_config
|
||||
from invokeai.backend.model_hash.model_hash import HASHING_ALGORITHMS, ModelHash
|
||||
from invokeai.backend.model_manager.taxonomy import ModelRepoVariant
|
||||
from invokeai.backend.quantization.gguf.loaders import gguf_sd_loader
|
||||
from invokeai.backend.quantization.gguf.loaders import WrappedGGUFReader, gguf_sd_loader
|
||||
from invokeai.backend.util.logging import InvokeAILogger
|
||||
from invokeai.backend.util.silence_warnings import SilenceWarnings
|
||||
|
||||
@@ -52,9 +53,19 @@ class ModelOnDisk:
|
||||
if path in self._metadata_cache:
|
||||
return self._metadata_cache[path]
|
||||
try:
|
||||
with safe_open(self.path, framework="pt", device="cpu") as f:
|
||||
metadata = f.metadata()
|
||||
assert isinstance(metadata, dict)
|
||||
if path.suffix == ".gguf":
|
||||
with WrappedGGUFReader(path) as reader:
|
||||
metadata = {
|
||||
name: value
|
||||
for name, field in reader.fields.items()
|
||||
if field.types
|
||||
and field.types[0] == GGUFValueType.STRING
|
||||
and isinstance(value := field.contents(), str)
|
||||
}
|
||||
else:
|
||||
with safe_open(path, framework="pt", device="cpu") as f:
|
||||
metadata = f.metadata()
|
||||
assert isinstance(metadata, dict)
|
||||
except Exception:
|
||||
metadata = {}
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user