Files

5.5 KiB
Raw Permalink Blame History

Control maps: control-map extraction

thenoise can extract control maps from a plain image — the pictures that ControlNet-style pipelines condition on. It works exactly like /upscale: image in, map out, no diffusion model loaded. A map is always the size of the image it was made from.

Five kinds ship today: canny, lineart and softedge need nothing extra, while depth and pose need a small download each. Every one of them is available from the CLI (controlmap), the web UI's Control Maps tab and the /controlmap endpoint, with the same options.

Input Canny Edge Depth map Pose Lineart Soft-edged map
thenoise-input-img canny depth pose lineart softedge

Installing and downloading what each kind needs

Canny, lineart and softedge work out of the box. Depth and pose need checkpoints:

python scripts/download.py --model controlmaps                # depth + pose + person detector
python scripts/download.py --model controlmaps --variant base  # depth, crisper
python scripts/download.py --model controlmaps --variant large # depth, crispest
python scripts/download.py --model controlmaps --variant pose       # pose + person detector
python scripts/download.py --model controlmaps --variant pose-r50vd # the better detector

Everything lands in ./models/controlmaps, one directory per checkpoint. Point --controlmap-dir at that directory and the engine works out which checkpoint is which:

./thenoise.sh controlmap --input photo.jpg --type depth \
  --controlmap-dir ./models/controlmaps --out depth.png
./thenoise.sh controlmap --input photo.jpg --type pose \
  --controlmap-dir ./models/controlmaps --out pose.png

Pose needs both of its checkpoints: one finds the people, the other measures their joints. With only one in the directory pose is not offered, and --list says what is missing rather than staying quiet about it:

./thenoise.sh controlmap --list --controlmap-dir ./models/controlmaps

Ask for more than one --type and each map gets its own file (map_canny.png, map_pose.png, …).

The depth map

Relative depth — bright is near, dark is far — rather than distance in metres.

Option Default Meaning
invert false Swap near/far (dark = near)
colormap gray gray, or turbo: a vivid rainbow that is easier to read by eye

The pose map

An openpose-style skeleton — coloured bones and joints on a black canvas. It is drawn rather than stretched, so a limb keeps its angle whatever size the map ends up. A joint the model is unsure about is left out together with the bones hanging from it, so a half-detected person loses that limb instead of gaining a stray line.

Bodies only: no hands, no face.

Option Default Meaning
detector first found Which person detector to use (see the variants above)
threshold 0.5 How sure the engine has to be that there is a person there at all
kp_threshold 0.3 How sure it has to be about a joint before drawing it
line_width 4 Bone thickness, in pixels
radius 4 Radius of the joint dots
background black black, or green — the canvas the original openpose maps used

The canny map

White edges on black: the classic hard-edged edge map.

Option Default Meaning
low 100 How faint an edge may be and still be drawn — lower finds more
high 200 Where an edge counts as strong; faint ones survive only where they run into a strong edge
thickness 1 Line thickness, in pixels

The lineart map

Thin dark lines on white: cleaner and more drawing-like than canny, with a single threshold instead of two. An approximation of the model-based lineart annotator, with nothing to download.

Option Default Meaning
threshold 0.5 0–1: how strong a line has to be to be drawn, measured against the strongest line in the image
thickness 1 Line thickness, in pixels

The softedge map

Edges as a soft grey gradient instead of a hard line — anti-aliased dark edges on white, close to the softedge annotator, with nothing to download. Good when you want the generation to keep soft shading rather than snap to outlines.

Option Default Meaning
threshold 0.2 0–1: where an edge starts to count
blur 1.5 Extra smoothing, in pixels; 0 leaves edges crisp

What the map remembers

Every map is saved with a controlmap_data chunk recording its type, the model that made it and any option you changed, next to whatever metadata the input image already carried. A map therefore says where it came from later on — on disk and in the web UI's info dialog.