mirror of
https://github.com/invoke-ai/InvokeAI.git
synced 2026-10-12 02:16:43 +00:00
chore(startup): silence HF tokenizers fork-after-parallelism warning
Set TOKENIZERS_PARALLELISM=false at startup (via os.environ.setdefault so users can override) before any HF library is imported. The Rust ``tokenizers`` library warms a thread pool the first time a tokenizer runs — for us that's UMT5 / T5 text encoding during Wan / FLUX / SD3 conditioning. Every subsequent fork() then logs huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... In video generation we fork on every MP4 encode (imageio's FFMPEG plugin uses subprocess.Popen → fork+exec), so this warning lands once per generation in the server log. The advisory is benign — the child correctly falls back to single-threaded tokenization before exec(), and the parent's thread pool is unaffected — but the noise obscures real warnings. Setting the env var before any HF import prevents the thread pool from warming up at all, so the fork detector stays quiet without sacrificing anything: tokenization happens once per generation and isn't a hot path for us. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
a421f9c66c
commit
2106f10ec4
@@ -10,10 +10,23 @@ def get_app():
|
||||
def run_app() -> None:
|
||||
"""The main entrypoint for the app."""
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
import traceback
|
||||
|
||||
# Suppress the HuggingFace tokenizers fork-after-parallelism warning. The Rust
|
||||
# ``tokenizers`` library warms a thread pool the first time a tokenizer is used
|
||||
# (e.g. the UMT5 / T5 text encoder during Wan / FLUX / SD3 conditioning), then
|
||||
# complains every time we fork() afterwards — which we do, on every MP4 encode,
|
||||
# because imageio's FFMPEG plugin shells out to ffmpeg via subprocess.Popen.
|
||||
# The warning is harmless (the child correctly falls back to single-threaded
|
||||
# tokenization before exec()), but it spams the log on every video generation.
|
||||
# Setting the env var BEFORE any HF library import keeps the thread pool from
|
||||
# warming up in the first place, so the fork detector stays quiet. Use
|
||||
# ``setdefault`` so anyone who explicitly sets ``true`` upstream keeps their value.
|
||||
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
|
||||
|
||||
from invokeai.frontend.cli.arg_parser import InvokeAIArgs
|
||||
|
||||
# Parse the CLI arguments before doing anything else, which ensures CLI args correctly override settings from other
|
||||
|
||||
Reference in New Issue
Block a user