chore(startup): silence HF tokenizers fork-after-parallelism warning

Set TOKENIZERS_PARALLELISM=false at startup (via os.environ.setdefault
so users can override) before any HF library is imported.

The Rust ``tokenizers`` library warms a thread pool the first time a
tokenizer runs — for us that's UMT5 / T5 text encoding during Wan /
FLUX / SD3 conditioning. Every subsequent fork() then logs

  huggingface/tokenizers: The current process just got forked, after
  parallelism has already been used. Disabling parallelism to avoid
  deadlocks...

In video generation we fork on every MP4 encode (imageio's FFMPEG
plugin uses subprocess.Popen → fork+exec), so this warning lands once
per generation in the server log. The advisory is benign — the child
correctly falls back to single-threaded tokenization before exec(),
and the parent's thread pool is unaffected — but the noise obscures
real warnings.

Setting the env var before any HF import prevents the thread pool
from warming up at all, so the fork detector stays quiet without
sacrificing anything: tokenization happens once per generation and
isn't a hot path for us.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
Lincoln Stein
2026-05-14 09:54:18 -04:00
co-authored by Claude Opus 4.7
parent a421f9c66c
commit 2106f10ec4
+13
View File
@@ -10,10 +10,23 @@ def get_app():
def run_app() -> None:
"""The main entrypoint for the app."""
import asyncio
import os
import sys
import threading
import traceback
# Suppress the HuggingFace tokenizers fork-after-parallelism warning. The Rust
# ``tokenizers`` library warms a thread pool the first time a tokenizer is used
# (e.g. the UMT5 / T5 text encoder during Wan / FLUX / SD3 conditioning), then
# complains every time we fork() afterwards — which we do, on every MP4 encode,
# because imageio's FFMPEG plugin shells out to ffmpeg via subprocess.Popen.
# The warning is harmless (the child correctly falls back to single-threaded
# tokenization before exec()), but it spams the log on every video generation.
# Setting the env var BEFORE any HF library import keeps the thread pool from
# warming up in the first place, so the fork detector stays quiet. Use
# ``setdefault`` so anyone who explicitly sets ``true`` upstream keeps their value.
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
from invokeai.frontend.cli.arg_parser import InvokeAIArgs
# Parse the CLI arguments before doing anything else, which ensures CLI args correctly override settings from other