mirror of
https://github.com/piabrota/mu.git
synced 2026-11-07 06:15:59 -10:00
ygg-models.sh setup and clean: the local model server in one command (#90)
* ygg-models.sh setup and clean; the example names its files and sources (local-models-setup LS-T3) setup does the first-time setup and is safe to run again: the GPU, the list from the example (--price required for a new one, refused before anything changes), the missing download files (hf, else uvx), the image, start, load, and the chat template when the list names a missing one (the first start leaves that line out of the preset, then template, stop, start and load). clean removes the container, the network (refused while another container is on it) and the image; --state and --weights remove only the files they name. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01H5rHSQVuXrzFdTaNXZdSde * ygg-models.sh clean decides first, and --weights removes hf's cache folder (local-models-setup LS-T3) clean lists the other containers on the model network before it stops anything: a refusal names them and leaves the server, the network and the image as they are. --weights also removes <weights>/.cache/huggingface, the state hf download leaves there, and no other path. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01H5rHSQVuXrzFdTaNXZdSde * ygg-models.sh setup never restarts the server under a run; gungnir's patterns (local-models-setup LS-T3) setup checks the model network before step 1 when making the template would stop and start the server, and refuses, changing nothing, while a run is on it; it checks again before the restart. A failed template step says the server runs without the chat template. The download repo, revision and paths use gungnir's patterns, so none starts with a dash. clean --weights refuses a symlinked .cache or .cache/huggingface. The network refusal points at ygg-clean.sh, and the example says to comment out chat-template-file until template has run. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01H5rHSQVuXrzFdTaNXZdSde --------- Co-authored-by: Bruno Guilhermo de Barros Valério <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
This commit is contained in:
4 files changed
+830
-179
No files matched your search
@@ -47,7 +47,7 @@ folder.
|
||||
| Build `ygg` from the umbrella's checkouts, so a smoke or proof run never uses a stale one | `scripts/ygg-build.sh [--yggdrasil DIR] [--out FILE] [--allow-behind] [--force]`. Refuses a dirty module or one behind its local `origin/main` (it fetches nothing); builds to the `ygg` first on PATH (else `$(go env GOPATH)/bin/ygg`) and writes the commits to `OUT.build.txt`; a link is followed to its target; an error if `command -v ygg` is not OUT. Never overwrites a file that is not a Go binary without `--force`. The smoke and proof scripts refuse a `ygg` from PATH whose facts do not name odin's HEAD (`--any-ygg` skips that) |
|
||||
| Clean up after runs that have ended: the empty pods and leftover containers | `scripts/ygg-clean.sh --dry-run [--folder DIR] [--keep PROJECT]... [PROJECT...]`, then without `--dry-run`. Runs `ygg doctor` on each project under DIR (default `~/ygg-it`) with the config its runs used; skips a project with a live run (its lock in `/proc/locks`, or open in a process) and refuses when it cannot tell; never deletes a run, a project or an image, and never starts or stops the engine |
|
||||
| Set up and check the GPU for the engine: the NVIDIA toolkit (pinned 1.20.1) and the CDI spec in the Podman machine's distro on WSL2, repeatable after a driver update or on a new machine | `scripts/ygg-gpu.sh setup\|check [--machine NAME] [--folder DIR]`. `setup` installs only what is missing and regenerates a stale spec, then runs `check`; `check` runs the CUDA image (by digest) through `ygg-engine.sh` and prints `gpu: <name> <memory>`, or exits 3 with `missing: toolkit\|spec\|stale spec` and the fix, or 2 when the engine does not answer. Root commands go through `wsl.exe -u root`, never sudo; it never starts, stops or re-creates the machine |
|
||||
| Start, stop and check the local model server (llama-server and the model gate in one container, on the engine's GPU), from the operator's model list; the qwen scripts find it through it | `scripts/ygg-models.sh [--models FILE] [--folder DIR] start\|stop\|status\|load NAME`. FILE defaults to `${XDG_CONFIG_HOME:-~/.config}/ygg/models.yaml` and DIR (holds `factory.yaml`, which names the engine) to `~/ygg-it` (example list: `scripts/testdata/models/models.example.yaml`). `start` runs `ygg-gpu.sh check` first (a stale spec is set up again), builds the image only when its tag is missing (long, CUDA), creates the internal network when absent, writes the preset and runs the container with `127.0.0.1:<host_port>` as its only published port; `status` prints `running on <url>, loaded: <name\|none>` or exits 3; `stop` keeps the network and the image. Engine calls go through `ygg-engine.sh` |
|
||||
| Set up, start, stop, check and clean the local model server (llama-server and the model gate in one container, on the engine's GPU), from the operator's model list; the qwen scripts find it through it | `scripts/ygg-models.sh [--models FILE] [--folder DIR] setup [--weights DIR] [--state DIR] [--price IN:OUT[:CACHE_READ]] [--model NAME]` does the first-time setup and is safe to run again: the GPU (`ygg-gpu.sh`), the list from the example when FILE is missing (`--price` required then; folders default to `DIR/models/weights` and `DIR/models/state`), the missing `download` files (`hf`, else `uvx`), the image, start, load (the first model unless `--model`) and the chat template when it is missing. `clean [--state] [--weights]` removes the container, the network and the image, and refuses, changing nothing, while another container is on the network; `--state` the preset and allow files; `--weights` only the files the list downloads and `.cache/huggingface` under the weights folder. Also `start\|stop\|status\|load NAME\|template NAME`. FILE defaults to `${XDG_CONFIG_HOME:-~/.config}/ygg/models.yaml` and DIR (holds `factory.yaml`, which names the engine) to `~/ygg-it` (example list: `scripts/testdata/models/models.example.yaml`). `start` runs `ygg-gpu.sh check` first (a stale spec is set up again), builds the image only when its tag is missing (long, CUDA), creates the internal network when absent, writes the preset and runs the container with `127.0.0.1:<host_port>` as its only published port; `status` prints `running on <url>, loaded: <name\|none>` or exits 3; `stop` keeps the network and the image. Engine calls go through `ygg-engine.sh` |
|
||||
| Any real-engine run (Podman, devcontainer, the sandbox): the engine, registry config and checks | `scripts/ygg-engine.sh check\|env\|run --folder DIR`. Never set `CONTAINER_HOST` by hand: the root socket is the wrong one |
|
||||
| The realms smoke runs (v0-H13, v0-H15) and their evidence | `scripts/ygg-smoke-realms.sh`, `scripts/ygg-smoke-loop.sh`, `scripts/ygg-evidence.sh` |
|
||||
| The grader smoke on a real engine (v0-H16), with the public sample suite | `scripts/ygg-smoke-grade.sh [--build]`. Never a hidden suite by default; `--task ID --suite-dir DIR --reference DIR [--suite-engine DIR]` points the grader at a suite store as it is (never copied or read) and grades the task's reference solution; its output holds numbers only |
|
||||
|
||||
+13
-8
@@ -1,5 +1,8 @@
|
||||
# An example model list for scripts/ygg-models.sh and `ygg models`. Copy it to ~/.config/ygg/models.yaml and
|
||||
# change the two folders to absolute paths on your machine. Never commit your own list.
|
||||
# An example model list for scripts/ygg-models.sh and `ygg models`. `scripts/ygg-models.sh setup --price IN:OUT`
|
||||
# writes it to ~/.config/ygg/models.yaml with your folders and price; or copy it there and change the two folders
|
||||
# to absolute paths on your machine. Never commit your own list. When you copy it, comment out chat-template-file
|
||||
# below until `scripts/ygg-models.sh template NAME` has made the file: a model whose template file is missing does not
|
||||
# load (setup does this for you).
|
||||
# The engine is a Podman machine that sees host files only through the `sandbox.mounts` of factory.yaml, so both
|
||||
# folders must be under a folder of those mounts (for the founder's setup, under ~/ygg-it).
|
||||
server:
|
||||
@@ -11,7 +14,7 @@ server:
|
||||
state: /abs/path/to/state # the preset and the allow file; mounted read-only at /etc/ygg-models
|
||||
models:
|
||||
qwen3.8-27b:
|
||||
file: model.gguf # under weights
|
||||
file: Qwen3.8-27B-UD-Q4_K_M.gguf # under weights
|
||||
context: 102400 # the context size; becomes ctx-size in the preset
|
||||
output: 32768 # the most output tokens OpenCode asks for
|
||||
args: # llama-server preset keys, written as given
|
||||
@@ -29,14 +32,16 @@ models:
|
||||
# Out-of-memory-safe values for a WSL2 machine with limited RAM: a small prompt cache and few checkpoints.
|
||||
cache-ram: "2048"
|
||||
ctx-checkpoints: "4"
|
||||
# MTP: the draft model speeds up decoding. The draft file sits next to the model, under weights.
|
||||
# MTP: the draft model speeds up decoding. The draft file is under weights, at the path it has in the repo.
|
||||
spec-type: draft-mtp
|
||||
model-draft: /models/mtp.gguf
|
||||
model-draft: /models/MTP/mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
n-gpu-layers-draft: all
|
||||
cache-type-k-draft: q8_0
|
||||
cache-type-v-draft: q8_0
|
||||
# Claude Code sends a later system message that the model's own template refuses. `scripts/ygg-models.sh
|
||||
# template NAME` writes a fixed template to weights/chat-template.jinja; then uncomment the next line, and
|
||||
# stop and start the server.
|
||||
# chat-template-file: /models/chat-template.jinja
|
||||
# setup` (or `template NAME`) writes a fixed template to weights/chat-template.jinja, which this line uses.
|
||||
chat-template-file: /models/chat-template.jinja
|
||||
download: # where setup downloads the files from (Hugging Face), each to its own path under weights
|
||||
repo: unsloth/Qwen3.8-27B-GGUF
|
||||
files: [Qwen3.8-27B-UD-Q4_K_M.gguf, MTP/mtp-Qwen3.8-27B-Q4_0.gguf]
|
||||
price: {input: 0.10, output: 0.40} # example values: OpenRouter's price, dollars per million tokens
|
||||
@@ -1,8 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tests for ygg-models.sh. scripts/tests/ygg-models.test.sh
|
||||
# Fakes stand in for the tools: ygg (YGG_CMD) answers `models server|preset|image|build`; ygg-engine.sh
|
||||
# Fakes stand in for the tools: ygg (YGG_CMD) answers `models server|preset|sources|image|build`; ygg-engine.sh
|
||||
# (YGG_ENGINE_SH) runs the fake podman; ygg-gpu.sh (YGG_GPU_SH) is a fake; curl is a fake whose server is a few
|
||||
# files. The whole test runs under env -i (hermetic.sh): no real engine, GPU or ygg is ever called.
|
||||
# files; hf and uvx are fakes in their own folders, put on PATH by the cases that want them. The whole test runs
|
||||
# under env -i (hermetic.sh): no real engine, GPU, network or ygg is ever called. The fakes of the steps also write
|
||||
# one line each to all.log, so a case can check the order of the steps.
|
||||
# Each check is a condition string run by eval: $rc and the helpers are read there, so the linter cannot see them used.
|
||||
# shellcheck disable=SC2016,SC2034,SC2329
|
||||
set -uo pipefail
|
||||
@@ -47,9 +49,11 @@ case "$2" in
|
||||
[ -z "$FAKE_YGG_REFUSE" ] || { echo "ygg: $FAKE_YGG_REFUSE" >&2; exit 1; }
|
||||
[ -f "$5" ] || { echo "ygg: $5: no such file" >&2; exit 1; }
|
||||
cat "$FAKE_T/server.out" ;;
|
||||
preset) [ -f "$3" ] || exit 1; printf 'version = 1\n[qwen3.8-27b]\nmodel = /models/model.gguf\n' ;;
|
||||
preset) [ -f "$3" ] || exit 1
|
||||
if [ -f "$FAKE_T/preset.out" ]; then cat "$FAKE_T/preset.out"; else printf 'version = 1\n[qwen3.8-27b]\nmodel = /models/model.gguf\n'; fi ;;
|
||||
sources) [ -f "$3" ] || { echo "ygg: $3: no such file" >&2; exit 1; }; cat "$FAKE_T/sources.out" 2>/dev/null; exit 0 ;;
|
||||
image) echo "ygg-models:abc123" ;;
|
||||
build) touch "$FAKE_T/st/image" ;;
|
||||
build) touch "$FAKE_T/st/image"; echo build >>"$FAKE_T/all.log" ;;
|
||||
*) exit 64 ;;
|
||||
esac
|
||||
FAKE
|
||||
@@ -72,6 +76,9 @@ echo "$*" >>"$FAKE_T/podman.log"
|
||||
{ printf '%s\n' "$@"; echo ---; } >>"$FAKE_T/podman.args" # one argument per line
|
||||
case "$1 $2" in
|
||||
"container ls") # --filter name=^NAME$ matches the name exactly; FAKE_ID_CLASH lists another container's name
|
||||
case "$5" in # --filter network=NAME: the server when it exists, and FAKE_NET_USERS
|
||||
network=*) [ -e "$st/container" ] && echo ygg-models; [ -z "$FAKE_NET_USERS" ] || printf '%s\n' $FAKE_NET_USERS; exit 0 ;;
|
||||
esac
|
||||
[ -e "$st/container" ] && echo ygg-models
|
||||
[ -z "$FAKE_ID_CLASH" ] || echo "$FAKE_ID_CLASH"
|
||||
exit 0 ;;
|
||||
@@ -79,14 +86,20 @@ case "$1 $2" in
|
||||
"network exists") [ -e "$st/network" ] ;;
|
||||
"network inspect") cat "$st/internal" ;;
|
||||
"network create") touch "$st/network"; echo true >"$st/internal" ;;
|
||||
"network rm") rm -f "$st/network" ;;
|
||||
"image rm") rm -f "$st/image" ;;
|
||||
"container inspect") cat "$st/running" ;;
|
||||
"logs --tail") echo "fake log: unknown key 'bogus' in the preset" ;;
|
||||
"rm -f") rm -f "$st/container" "$st/running" "$st/up" ;;
|
||||
"stop "*) echo false >"$st/running"; rm -f "$st/up" ;;
|
||||
"stop "*) echo false >"$st/running"; rm -f "$st/up"; echo stop >>"$FAKE_T/all.log" ;;
|
||||
"rm "*) rm -f "$st/container" "$st/running" ;;
|
||||
"run -d")
|
||||
[ -z "$FAKE_RUN_CDI" ] || { echo "Error: setting up CDI devices: unresolvable CDI devices nvidia.com/gpu=all" >&2; exit 125; }
|
||||
touch "$st/container"
|
||||
echo run >>"$FAKE_T/all.log"
|
||||
# the preset each run started with, and a fresh server: nothing loaded
|
||||
cp "$FAKE_ST/preset.ini" "$FAKE_T/preset.run.$(grep -c '^run$' "$FAKE_T/all.log")" 2>/dev/null
|
||||
[ ! -f "$FAKE_T/models.unloaded.json" ] || cp "$FAKE_T/models.unloaded.json" "$FAKE_T/models.json"
|
||||
if [ -n "$FAKE_RUN_EXITS" ]; then echo false >"$st/running"; else echo true >"$st/running"; touch "$st/up"; fi
|
||||
echo "0123456789abcdef" ;;
|
||||
*) echo "fake podman: unexpected $*" >&2; exit 65 ;;
|
||||
@@ -95,6 +108,7 @@ FAKE
|
||||
cat >"$T/fakes/gpu.sh" <<'FAKE'
|
||||
#!/bin/bash
|
||||
echo "gpu $*" >>"$FAKE_T/gpu.log"
|
||||
if [ -n "$FAKE_LIST" ] && [ -e "$FAKE_LIST" ]; then echo "gpu $1 list=present"; else echo "gpu $1 list=absent"; fi >>"$FAKE_T/all.log"
|
||||
case "$1" in
|
||||
check)
|
||||
if [ -e "$FAKE_T/gpu.state" ]; then cat "$FAKE_T/gpu.state"; echo "fix: scripts/ygg-gpu.sh setup"; exit 3; fi
|
||||
@@ -114,9 +128,10 @@ url="${args[$((${#args[@]} - 1))]}"
|
||||
echo "$method $url $body" >>"$FAKE_T/curl.log"
|
||||
[ -e "$FAKE_T/st/up" ] || exit 7
|
||||
case "$method $url" in
|
||||
"GET "*/props\?model=*) [ -z "$FAKE_NO_PROPS" ] || { printf 404; exit 0; }; cat "$FAKE_T/props.json" >"$out"; printf 200 ;;
|
||||
"GET "*/props\?model=*) echo props >>"$FAKE_T/all.log"; [ -z "$FAKE_NO_PROPS" ] || { printf 404; exit 0; }; cat "$FAKE_T/props.json" >"$out"; printf 200 ;;
|
||||
"GET "*/models) cat "$FAKE_T/models.json" >"$out"; printf 200 ;;
|
||||
"POST "*/models/load)
|
||||
echo load >>"$FAKE_T/all.log"
|
||||
code="$(cat "$FAKE_T/load.code" 2>/dev/null || echo 200)"
|
||||
cat "$FAKE_T/load.body" >"$out" 2>/dev/null || : >"$out"
|
||||
case "$code" in 200 | 400) cp "$FAKE_T/models.after.json" "$FAKE_T/models.json" ;; esac
|
||||
@@ -124,15 +139,44 @@ case "$method $url" in
|
||||
*) exit 64 ;;
|
||||
esac
|
||||
FAKE
|
||||
chmod +x "$T/fakes/"* "$T/engine.sh" "$T/bin/curl"
|
||||
export YGG_CMD="$T/fakes/ygg" YGG_ENGINE_SH="$T/engine.sh" YGG_GPU_SH="$T/fakes/gpu.sh"
|
||||
# hf and uvx: each logs its call to dl.log, then dl-impl writes the file (hf download REPO PATH --revision REV
|
||||
# --local-dir DIR); FAKE_DL_FAIL makes the download fail and write nothing.
|
||||
mkdir -p "$T/hfbin" "$T/uvxbin"
|
||||
cat >"$T/dl-impl" <<'FAKE'
|
||||
#!/bin/bash
|
||||
[ "$1" = download ] || exit 64
|
||||
repo="$2" path="$3" dir="" rev=""
|
||||
shift 3
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in --local-dir) dir="$2"; shift 2 ;; --revision) rev="$2"; shift 2 ;; *) exit 64 ;; esac
|
||||
done
|
||||
[ -n "$dir" ] && [ -n "$rev" ] || exit 64
|
||||
[ -z "$FAKE_DL_FAIL" ] || { echo "hf: 404 Not Found" >&2; exit 1; }
|
||||
if [ -n "$FAKE_LIST" ] && [ -e "$FAKE_LIST" ]; then echo "download $path list=present"; else echo "download $path list=absent"; fi >>"$FAKE_T/all.log"
|
||||
mkdir -p "$(dirname "$dir/$path")"
|
||||
echo "weights of $repo@$rev" >"$dir/$path"
|
||||
FAKE
|
||||
cat >"$T/hfbin/hf" <<'FAKE'
|
||||
#!/bin/bash
|
||||
echo "hf $*" >>"$FAKE_T/dl.log"
|
||||
exec "$FAKE_T/dl-impl" "$@"
|
||||
FAKE
|
||||
cat >"$T/uvxbin/uvx" <<'FAKE'
|
||||
#!/bin/bash
|
||||
echo "uvx $*" >>"$FAKE_T/dl.log"
|
||||
[ "$1 $2 $3" = "--from huggingface_hub hf" ] || exit 64
|
||||
shift 3
|
||||
exec "$FAKE_T/dl-impl" "$@"
|
||||
FAKE
|
||||
chmod +x "$T/fakes/"* "$T/engine.sh" "$T/bin/curl" "$T/dl-impl" "$T/hfbin/hf" "$T/uvxbin/uvx"
|
||||
export YGG_CMD="$T/fakes/ygg" YGG_ENGINE_SH="$T/engine.sh" YGG_GPU_SH="$T/fakes/gpu.sh" FAKE_ST="$ST"
|
||||
export YGG_MODELS_POLL=0.1 YGG_MODELS_TIMEOUT=2
|
||||
LIST="$T/models.yaml"; : >"$LIST"
|
||||
|
||||
reset() { # a clean engine, and a list that names $W
|
||||
rm -rf "${SD:?}" "${ST:?}" "${T:?}"/*.log "${T:?}"/podman.args "${T:?}/gpu.state" "${T:?}/load.code" "${T:?}/load.body"; mkdir -p "$SD" "$W"
|
||||
rm -f "$W/chat-template.jinja"
|
||||
unset FAKE_RUN_EXITS FAKE_ENGINE_DOWN FAKE_ID_CLASH FAKE_IMAGE_ERR FAKE_YGG_REFUSE FAKE_NO_PROPS FAKE_RUN_CDI
|
||||
rm -f "$W/chat-template.jinja" "$T/preset.out" "$T/sources.out" "$T/models.unloaded.json" "$T"/preset.run.*
|
||||
unset FAKE_RUN_EXITS FAKE_ENGINE_DOWN FAKE_ID_CLASH FAKE_IMAGE_ERR FAKE_YGG_REFUSE FAKE_NO_PROPS FAKE_RUN_CDI FAKE_NET_USERS FAKE_DL_FAIL FAKE_LIST
|
||||
printf 'NAME=ygg-models\nNETWORK=ygg-models\nPORT=8089\nHOST_PORT=8088\nWEIGHTS=%s\nSTATE=%s\nENGINE_WEIGHTS=%s\nENGINE_STATE=%s\n' "$W" "$ST" "$EW" "$EST" >"$T/server.out"
|
||||
printf '{"data":[{"id":"qwen3.8-27b","status":{"value":"unloaded","preset":"version = 1\n[qwen]\nmodel = x"}}]}' >"$T/models.json"
|
||||
printf '{"data":[{"id":"qwen3.8-27b","status":{"value":"loaded","preset":"version = 1\n[qwen]\nmodel = x"}}]}' >"$T/models.after.json"
|
||||
@@ -385,6 +429,227 @@ reset
|
||||
run bash "$M" --models "$LIST" stop
|
||||
check "stop: with no container it says so and exits 0" eval '[ $rc = 0 ] && has "not running"'
|
||||
|
||||
# --- setup
|
||||
# The setup cases: a weights folder without a space (setup writes it into a new list, and ygg refuses a space), a list
|
||||
# in a folder that does not exist yet, the founder's two files, and a preset that names the chat template.
|
||||
W2="$T/w2"
|
||||
NL="$T/cfg/models.yaml"
|
||||
GG=Qwen3.8-27B-UD-Q4_K_M.gguf
|
||||
MTP=MTP/mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
TPL_LINE="chat-template-file = /models/chat-template.jinja"
|
||||
sreset() { # reset, then: no list, no weights, the GPU without its toolkit, no image, nothing running
|
||||
reset; rm -rf "${W2:?}" "${T:?}/cfg"; echo "missing: toolkit" >"$T/gpu.state"
|
||||
sed -i "s|^WEIGHTS=.*|WEIGHTS=$W2|" "$T/server.out"
|
||||
printf 'qwen3.8-27b unsloth/Qwen3.8-27B-GGUF main %s\nqwen3.8-27b unsloth/Qwen3.8-27B-GGUF main %s\n' "$GG" "$MTP" >"$T/sources.out"
|
||||
printf 'version = 1\n\n[qwen3.8-27b]\nmodel = /models/%s\nctx-size = 102400\n%s\nmodel-draft = /models/%s\n' "$GG" "$TPL_LINE" "$MTP" >"$T/preset.out"
|
||||
cp "$T/models.json" "$T/models.unloaded.json"
|
||||
mkprops "A
|
||||
$TPL_OLD
|
||||
B"
|
||||
export FAKE_LIST="$NL"
|
||||
}
|
||||
steps() { tr '\n' '|' <"$T/all.log" 2>/dev/null; }
|
||||
lf() { sed -n "s/^ $1: \([^ ]*\).*/\1/p" "$2"; } # lf KEY FILE: the value of a server key of a list
|
||||
SETUP=(--weights "$W2" --state "$ST" --price 0.30:1.20:0.06)
|
||||
|
||||
sreset
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: a fresh setup exits 0" eval '[ $rc = 0 ]'
|
||||
check "setup: the steps run in order: GPU check and setup (before the list), the two downloads (after it), the image, start, load, then the template, stop, start and load" \
|
||||
eval '[ "$(steps)" = "gpu check list=absent|gpu setup list=absent|download $GG list=present|download $MTP list=present|build|run|load|props|stop|run|load|" ]'
|
||||
check "setup: the new list is the example with the given folders and price, and nothing else changed" \
|
||||
eval 'diff <(grep -v -E "^ (weights|state):|^ price:|^# Written by" "$S/testdata/models/models.example.yaml") <(grep -v -E "^ (weights|state):|^ price:|^# Written by" "$NL") >/dev/null && [ "$(lf weights "$NL")" = "$W2" ] && [ "$(lf state "$NL")" = "$ST" ] && grep -q -E "^ price: \{input: 0.30, output: 1.20, cache_read: 0.06\}" "$NL"'
|
||||
check "setup: the list is written atomically: no temporary file is left next to it" eval '[ "$(ls -A "$T/cfg")" = models.yaml ]'
|
||||
check "setup: both folders exist" eval '[ -d "$W2" ] && [ -d "$ST" ]'
|
||||
check "setup: each file is downloaded with hf download REPO PATH --revision REV --local-dir WEIGHTS" \
|
||||
eval 'grep -q -x -F "hf download unsloth/Qwen3.8-27B-GGUF $GG --revision main --local-dir $W2" "$T/dl.log" && grep -q -x -F "hf download unsloth/Qwen3.8-27B-GGUF $MTP --revision main --local-dir $W2" "$T/dl.log" && [ -f "$W2/$MTP" ]'
|
||||
check "setup: the first start leaves the missing chat template out of the preset; the second has it" \
|
||||
eval '! grep -q -F "$TPL_LINE" "$T/preset.run.1" && grep -q -x -F "$TPL_LINE" "$T/preset.run.2" && [ "$(cat "$ST/preset.ini")" = "$(cat "$T/preset.out")" ]'
|
||||
check "setup: the template is saved in the weights folder, with the fix" \
|
||||
eval '[ -f "$W2/chat-template.jinja" ] && grep -q -F "<|im_start|>system" "$W2/chat-template.jinja"'
|
||||
check "setup: the build goes through the engine with the folder's factory.yaml, and the server starts from the list given" \
|
||||
eval 'grep -q "^engine run.* -- $T/fakes/ygg models build --config $HOME/ygg-it/factory.yaml" "$T/engine.log" && grep -q -F "ygg models preset $NL" "$T/ygg.log" && grep -q -F "ygg models server --config $HOME/ygg-it/factory.yaml " "$T/ygg.log"'
|
||||
check "setup: it says what each step did" \
|
||||
eval 'has "gpu: set up" && has "model list: written" && has "download: $GG: done" && has "image: built" && has "server: started" && has "load: qwen3.8-27b: loaded" && has "template: saved"'
|
||||
cp "$NL" "$T/list.before"; : >"$T/all.log"; : >"$T/dl.log"; : >"$T/podman.log"; : >"$T/curl.log"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup again: exits 0, and only checks: no GPU setup, download, build, start or load" \
|
||||
eval '[ $rc = 0 ] && [ "$(steps)" = "gpu check list=present|" ] && [ ! -s "$T/dl.log" ] && ! logged "run -d" && ! logged "stop" && ! grep -q POST "$T/curl.log"'
|
||||
check "setup again: the list is as it was, and the flags are said to matter only for a new list" \
|
||||
eval 'cmp -s "$NL" "$T/list.before" && has "only for a new list"'
|
||||
check "setup again: every step says it is already in place" \
|
||||
eval 'has "gpu: already in place" && has "model list: already in place" && has "download: $GG: already in place" && has "download: $MTP: already in place" && has "image: already in place" && has "server: already running" && has "load: qwen3.8-27b: already loaded" && has "template: already in place"'
|
||||
|
||||
# --price on a new list
|
||||
sreset
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup --weights "$W2" --state "$ST"
|
||||
check "setup: a new list without --price refuses naming the flag, and nothing is created or called" \
|
||||
eval '[ $rc != 0 ] && has "--price" && [ ! -e "$T/cfg" ] && [ ! -e "$W2" ] && [ ! -e "$ST" ] && [ ! -e "$T/gpu.log" ] && [ ! -e "$T/podman.log" ] && [ ! -e "$T/dl.log" ]'
|
||||
for bad in 0.30 abc:1 0.30:1.20:0.06:9 "1:2;rm"; do
|
||||
sreset
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup --weights "$W2" --state "$ST" --price "$bad"
|
||||
check "setup: --price '$bad' is refused, and nothing is created" eval '[ $rc = 2 ] && has "--price" && [ ! -e "$T/cfg" ] && [ ! -e "$T/gpu.log" ]'
|
||||
done
|
||||
sreset
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup --weights "$T/a b" --state "$ST" --price 1:2
|
||||
check "setup: a weights folder ygg would refuse (a space) is refused before anything changes" \
|
||||
eval '[ $rc = 2 ] && has "--weights" && [ ! -e "$T/cfg" ] && [ ! -e "$T/gpu.log" ]'
|
||||
sreset; export FAKE_YGG_REFUSE="the folder $W2 is under no sandbox.mounts entry"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: a new list that ygg refuses (no sandbox.mounts covers it) stops before anything changes" \
|
||||
eval '[ $rc != 0 ] && has "sandbox.mounts" && [ ! -e "$T/cfg" ] && [ ! -e "$T/gpu.log" ]'
|
||||
unset FAKE_YGG_REFUSE
|
||||
|
||||
# the default folders
|
||||
sreset; sed -i "s|^WEIGHTS=.*|WEIGHTS=$HOME/ygg-it/models/weights|; s|^STATE=.*|STATE=$HOME/ygg-it/models/state|" "$T/server.out"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup --price 1:2
|
||||
check "setup: the folders default to <folder>/models/weights and <folder>/models/state, and the price without cache_read" \
|
||||
eval '[ $rc = 0 ] && [ "$(lf weights "$NL")" = "$HOME/ygg-it/models/weights" ] && [ "$(lf state "$NL")" = "$HOME/ygg-it/models/state" ] && grep -q -E "^ price: \{input: 1, output: 2\}" "$NL" && [ -d "$HOME/ygg-it/models/weights" ] && [ -d "$HOME/ygg-it/models/state" ]'
|
||||
rm -rf "${HOME:?}/ygg-it"
|
||||
|
||||
# downloads
|
||||
sreset; mkdir -p "$W2"; echo mine >"$W2/$GG"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: with one file there, only the missing one is downloaded, and the one there is untouched" \
|
||||
eval '[ $rc = 0 ] && [ "$(grep -c "^hf download" "$T/dl.log")" = 1 ] && grep -q -F " $MTP " "$T/dl.log" && [ "$(cat "$W2/$GG")" = mine ]'
|
||||
sreset
|
||||
PATH="$T/uvxbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: without hf on PATH, the download goes through uvx --from huggingface_hub hf" \
|
||||
eval '[ $rc = 0 ] && grep -q -x -F "uvx --from huggingface_hub hf download unsloth/Qwen3.8-27B-GGUF $GG --revision main --local-dir $W2" "$T/dl.log" && [ -f "$W2/$GG" ]'
|
||||
sreset
|
||||
run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: with neither hf nor uvx, it refuses naming both, before anything changes" \
|
||||
eval '[ $rc != 0 ] && has "neither hf nor uvx" && [ ! -e "$T/gpu.log" ] && [ ! -e "$T/cfg" ] && [ ! -e "$T/podman.log" ]'
|
||||
sreset; export FAKE_DL_FAIL=1
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: a download that fails stops setup with hf's message" eval '[ $rc != 0 ] && has "404" && ! logged "run -d"'
|
||||
unset FAKE_DL_FAIL
|
||||
sreset; : >"$T/sources.out"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: a list with no download says so, and downloads nothing" eval '[ $rc = 0 ] && has "download: none" && [ ! -e "$T/dl.log" ]'
|
||||
for bad in "../evil.gguf" "/etc/passwd" "a/../../b"; do
|
||||
sreset; printf 'qwen3.8-27b unsloth/Qwen3.8-27B-GGUF main %s\n' "$bad" >"$T/sources.out"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: a source path '$bad' is refused, and nothing is downloaded or set up" eval '[ $rc != 0 ] && has "not a plain path" && [ ! -e "$T/dl.log" ] && [ ! -e "$T/gpu.log" ]'
|
||||
done
|
||||
|
||||
# the template
|
||||
sreset; mkdir -p "$W2"; echo "my template" >"$W2/chat-template.jinja"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: a template that is there is kept: no props, one start, and the preset has it" \
|
||||
eval '[ $rc = 0 ] && ! grep -q props "$T/all.log" && [ "$(grep -c "^run$" "$T/all.log")" = 1 ] && grep -q -x -F "$TPL_LINE" "$T/preset.run.1" && [ "$(cat "$W2/chat-template.jinja")" = "my template" ] && has "template: already in place"'
|
||||
sreset; sed -i "/^chat-template-file/d" "$T/preset.out"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: a list that names no template: no props, one start" \
|
||||
eval '[ $rc = 0 ] && ! grep -q props "$T/all.log" && [ "$(grep -c "^run$" "$T/all.log")" = 1 ] && [ ! -e "$W2/chat-template.jinja" ] && has "template: none"'
|
||||
sreset; sed -i "s|^chat-template-file = .*|chat-template-file = /models/other.jinja|" "$T/preset.out"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: a missing template of another name is refused before the download, naming it" \
|
||||
eval '[ $rc != 0 ] && has "/models/other.jinja" && [ ! -e "$T/dl.log" ] && ! logged "run -d"'
|
||||
sreset; touch "$SD/image"; up; mkdir -p "$W2/MTP"; touch "$W2/$GG" "$W2/$MTP"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: a running server with the template missing is stopped, started without it, then made and started with it" \
|
||||
eval '[ $rc = 0 ] && [ "$(steps)" = "gpu check list=absent|gpu setup list=absent|stop|run|load|props|stop|run|load|" ]'
|
||||
|
||||
# a live run on the network: setup never stops or restarts the server under it
|
||||
sreset; touch "$SD/image" "$SD/network"; echo true >"$SD/internal"; up; mkdir -p "$W2/MTP"; touch "$W2/$GG" "$W2/$MTP"
|
||||
export FAKE_NET_USERS="run-live-sandbox"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: with the template missing and a run on the network, it refuses naming the run, before anything changes" \
|
||||
eval '[ $rc != 0 ] && has run-live-sandbox && has ygg-clean.sh && ! logged "stop" && ! logged "rm -f" && ! logged "run -d" && [ ! -e "$T/gpu.log" ] && [ ! -e "$T/cfg" ] && [ "$(cat "$SD/running")" = true ]'
|
||||
unset FAKE_NET_USERS
|
||||
sreset; mkdir -p "$W2/MTP"; touch "$W2/$GG" "$W2/$MTP" "$W2/chat-template.jinja"; touch "$SD/image" "$SD/network"; echo true >"$SD/internal"; up
|
||||
cp "$T/models.after.json" "$T/models.json"; export FAKE_NET_USERS="run-live-sandbox"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: with everything in place, a run on the network does not stop setup, and nothing is stopped" \
|
||||
eval '[ $rc = 0 ] && ! logged "stop" && has "server: already running"'
|
||||
unset FAKE_NET_USERS
|
||||
sreset; export FAKE_NO_PROPS=1
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: a template step that fails says the server runs without the chat template, and to run setup again" \
|
||||
eval '[ $rc != 0 ] && has "runs without the chat template" && has "run setup again" && [ "$(grep -c "^run$" "$T/all.log")" = 1 ]'
|
||||
unset FAKE_NO_PROPS
|
||||
|
||||
# a leading dash, or another character gungnir refuses, in a source
|
||||
for bad in "x -bad/r main a.gguf" "x o/-r main a.gguf" "x -o/r main a.gguf" "x o/r -main a.gguf" "x o/r main -a.gguf" "x o/r main d/-a.gguf" "x o/r refs/pr/1 a.gguf" "x o/r main .hidden"; do
|
||||
sreset; echo "$bad" >"$T/sources.out"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: the source line '$bad' is refused before anything changes" eval '[ $rc != 0 ] && [ ! -e "$T/dl.log" ] && [ ! -e "$T/gpu.log" ] && [ ! -e "$T/cfg" ]'
|
||||
done
|
||||
|
||||
# the model to load
|
||||
sreset; printf '\n[a-model]\nmodel = /models/a.gguf\nctx-size = 1\n' >>"$T/preset.out"
|
||||
printf '{"data":[{"id":"a-model","status":{"value":"unloaded"}},{"id":"qwen3.8-27b","status":{"value":"unloaded"}}]}' >"$T/models.unloaded.json"
|
||||
printf '{"data":[{"id":"a-model","status":{"value":"loaded"}},{"id":"qwen3.8-27b","status":{"value":"unloaded"}}]}' >"$T/models.after.json"
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}"
|
||||
check "setup: it loads the first model in sorted order" \
|
||||
eval '[ $rc = 0 ] && grep -q -F "{\"model\":\"a-model\"}" "$T/curl.log" && ! grep -q -F "{\"model\":\"qwen3.8-27b\"}" "$T/curl.log"'
|
||||
sreset
|
||||
PATH="$T/hfbin:$PATH" run bash "$M" --models "$NL" setup "${SETUP[@]}" --model nothere
|
||||
check "setup --model: a model the list does not have is refused before anything changes" \
|
||||
eval '[ $rc != 0 ] && has nothere && [ ! -e "$T/cfg" ] && [ ! -e "$T/gpu.log" ]'
|
||||
run bash "$M" --models "$NL" setup --frob
|
||||
check "setup: an unknown flag is a usage error" eval '[ $rc = 2 ] && has usage'
|
||||
|
||||
# --- clean
|
||||
creset() { # a server that runs, its network and image, state files, and the two downloaded files plus others
|
||||
sreset; rm -f "${T:?}/gpu.state"; touch "$SD/network" "$SD/image"; echo true >"$SD/internal"; up
|
||||
mkdir -p "$ST" "$W2/MTP" "$T/cfg"
|
||||
for f in preset.ini allowed preset.ini.tmp .allowed-123 notes.txt; do echo x >"$ST/$f"; done
|
||||
for f in "$GG" "$MTP" chat-template.jinja extra.gguf MTP/other.gguf; do echo x >"$W2/$f"; done
|
||||
mkdir -p "$W2/.cache/huggingface/download/MTP" "$W2/.cache/other" # hf download's state, and another cache
|
||||
echo x >"$W2/.cache/huggingface/download/MTP/mtp.metadata"; echo x >"$W2/.cache/other/keep"
|
||||
: >"$NL"
|
||||
}
|
||||
creset
|
||||
run bash "$M" --models "$NL" clean
|
||||
check "clean: stops and removes the container, then the network, then the image, saying each" \
|
||||
eval '[ $rc = 0 ] && [ "$(grep -E "^(stop |rm -f |network rm |image rm )" "$T/podman.log" | cut -d" " -f1,2 | tr "\n" "|")" = "stop -t|rm -f|network rm|image rm|" ] && has "container ygg-models: removed" && has "network ygg-models: removed" && has "image ygg-models:abc123: removed"'
|
||||
check "clean: without --state or --weights, the state, the weights and the list stay" \
|
||||
eval '[ "$(ls -A "$ST" | wc -l)" = 5 ] && [ -f "$W2/$GG" ] && [ -f "$W2/$MTP" ] && [ -f "$NL" ] && [ -d "$W2/.cache/huggingface" ]'
|
||||
reset; sed -i "s|^WEIGHTS=.*|WEIGHTS=$W2|" "$T/server.out"
|
||||
run bash "$M" --models "$NL" clean
|
||||
check "clean: with nothing there, it says so for each and removes nothing" \
|
||||
eval '[ $rc = 0 ] && ! grep -q -E "^(stop |rm |network rm |image rm )" "$T/podman.log" && has "container ygg-models: not there" && has "network ygg-models: not there" && has "image ygg-models:abc123: not there"'
|
||||
creset
|
||||
run bash "$M" --models "$NL" clean --state
|
||||
check "clean --state: removes preset.ini, allowed and their temporary files, and keeps the folder and other files" \
|
||||
eval '[ $rc = 0 ] && [ -d "$ST" ] && [ "$(ls -A "$ST")" = notes.txt ] && [ -f "$W2/$GG" ]'
|
||||
creset
|
||||
run bash "$M" --models "$NL" clean --weights
|
||||
check "clean --weights: removes hf download's .cache/huggingface folder, and only that path" \
|
||||
eval '[ ! -e "$W2/.cache/huggingface" ] && [ -f "$W2/.cache/other/keep" ] && [ -d "$W2/.cache" ] && has "$W2/.cache/huggingface"'
|
||||
check "clean --weights: deletes the files the list downloads, and only those" \
|
||||
eval '[ $rc = 0 ] && [ ! -e "$W2/$GG" ] && [ ! -e "$W2/$MTP" ] && [ -f "$W2/extra.gguf" ] && [ -f "$W2/chat-template.jinja" ] && [ -f "$W2/MTP/other.gguf" ] && [ -d "$W2/MTP" ] && [ "$(ls -A "$ST" | wc -l)" = 5 ] && has "deleted $W2/$GG"'
|
||||
creset; rm -f "${W2:?}/MTP/mtp-Qwen3.8-27B-Q4_0.gguf"
|
||||
run bash "$M" --models "$NL" clean --weights --state
|
||||
check "clean --weights --state: a named file already gone is fine" eval '[ $rc = 0 ] && [ ! -e "$W2/$GG" ] && [ "$(ls -A "$ST")" = notes.txt ]'
|
||||
creset; printf 'qwen3.8-27b unsloth/Qwen3.8-27B-GGUF main ../outside.gguf\n' >"$T/sources.out"; echo x >"$T/outside.gguf"
|
||||
run bash "$M" --models "$NL" clean --weights
|
||||
check "clean --weights: a source path that leaves the weights folder is refused before anything changes" \
|
||||
eval '[ $rc != 0 ] && has "not a plain path" && [ -f "$T/outside.gguf" ] && [ -f "$W2/$GG" ] && [ -d "$W2/.cache/huggingface" ] && ! logged "stop" && ! logged "rm -f" && [ -e "$SD/container" ]'
|
||||
creset; export FAKE_NET_USERS="run-abc-sandbox run-def-sandbox"
|
||||
run bash "$M" --models "$NL" clean --state --weights
|
||||
check "clean: another container on the network is named, and nothing is touched: the server still runs, the network and the image stay" \
|
||||
eval '[ $rc != 0 ] && has run-abc-sandbox && has run-def-sandbox && has ygg-clean.sh && ! logged "stop" && ! logged "rm -f" && ! logged "network rm" && ! logged "image rm" && [ -e "$SD/container" ] && [ "$(cat "$SD/running")" = true ] && [ -e "$SD/network" ] && [ -e "$SD/image" ] && [ "$(ls -A "$ST" | wc -l)" = 5 ] && [ -f "$W2/$GG" ]'
|
||||
creset; rm -rf "${W2:?}/.cache"; mkdir -p "$T/hfhome/hub"; echo secret >"$T/hfhome/token"; ln -s "$T/hfhome" "$W2/.cache"
|
||||
run bash "$M" --models "$NL" clean --weights
|
||||
check "clean --weights: a symlinked .cache is refused before anything changes, and what it points to stays" \
|
||||
eval '[ $rc != 0 ] && has "symlink" && [ -f "$T/hfhome/token" ] && [ -d "$T/hfhome/hub" ] && ! logged "stop" && [ -f "$W2/$GG" ]'
|
||||
creset; rm -rf "${W2:?}/.cache/huggingface"; ln -s "$T/hfhome" "$W2/.cache/huggingface"
|
||||
run bash "$M" --models "$NL" clean --weights
|
||||
check "clean --weights: a symlinked .cache/huggingface is refused before anything changes, and what it points to stays" \
|
||||
eval '[ $rc != 0 ] && has "symlink" && [ -f "$T/hfhome/token" ] && ! logged "stop" && [ -f "$W2/$GG" ]'
|
||||
creset; rm -rf "${W2:?}/.cache/huggingface"; ln -s "$T/hfhome" "$W2/.cache/huggingface"
|
||||
run bash "$M" --models "$NL" clean
|
||||
check "clean without --weights: a symlinked .cache/huggingface does not matter" eval '[ $rc = 0 ] && [ -f "$T/hfhome/token" ]'
|
||||
creset; export FAKE_NET_USERS="ygg-models"
|
||||
run bash "$M" --models "$NL" clean
|
||||
check "clean: the server itself on the network does not stop the network's removal" eval '[ $rc = 0 ] && logged "network rm ygg-models"'
|
||||
unset FAKE_NET_USERS
|
||||
run bash "$M" --models "$NL" clean --frob
|
||||
check "clean: an unknown flag is a usage error" eval '[ $rc = 2 ] && has usage'
|
||||
|
||||
# --- usage and hygiene
|
||||
run bash "$M"
|
||||
check "no command is a usage error" eval '[ $rc = 2 ] && has usage'
|
||||
@@ -396,6 +661,13 @@ EX="$S/testdata/models/models.example.yaml"
|
||||
check "the example list exists, and names no local path or operator" \
|
||||
eval '[ -f "$EX" ] && ! grep -E "/home/|/Users/|/mnt/|/root/" "$EX" && ! grep -w -F -e "$(id -un)" -e "$(basename "$HOME")" "$EX" && grep -q "weights: /abs/" "$EX" && grep -q "state: /abs/" "$EX"'
|
||||
check "the example list holds the MTP, memory-safe and template keys, and every args key is a valid preset key name" \
|
||||
eval 'grep -q "cache-ram: \"2048\"" "$EX" && grep -q "ctx-checkpoints: \"4\"" "$EX" && grep -q "spec-type: draft-mtp" "$EX" && grep -q "model-draft: /models/mtp.gguf" "$EX" && grep -q "n-gpu-layers-draft: all" "$EX" && grep -q "cache-type-k-draft: q8_0" "$EX" && grep -q "cache-type-v-draft: q8_0" "$EX" && grep -q "# chat-template-file: /models/chat-template.jinja" "$EX" && ! grep -E "^ [A-Za-z0-9_-]+:" "$EX" | grep -v -E "^ [a-z][a-z0-9-]*:"'
|
||||
eval 'grep -q "cache-ram: \"2048\"" "$EX" && grep -q "ctx-checkpoints: \"4\"" "$EX" && grep -q "spec-type: draft-mtp" "$EX" && grep -q "model-draft: /models/MTP/mtp-Qwen3.8-27B-Q4_0.gguf" "$EX" && grep -q "n-gpu-layers-draft: all" "$EX" && grep -q "cache-type-k-draft: q8_0" "$EX" && grep -q "cache-type-v-draft: q8_0" "$EX" && grep -q "^ chat-template-file: /models/chat-template.jinja" "$EX" && ! grep -E "^ [A-Za-z0-9_-]+:" "$EX" | grep -v -E "^ [a-z][a-z0-9-]*:"'
|
||||
check "the example list names the founder's files and where they come from" \
|
||||
eval 'grep -q "^ file: Qwen3.8-27B-UD-Q4_K_M.gguf" "$EX" && grep -q "^ repo: unsloth/Qwen3.8-27B-GGUF" "$EX" && grep -q -F "files: [Qwen3.8-27B-UD-Q4_K_M.gguf, MTP/mtp-Qwen3.8-27B-Q4_0.gguf]" "$EX"'
|
||||
check "the example header says to use setup, or to comment out chat-template-file until template has run" \
|
||||
eval 'sed -n "1,8p" "$EX" | grep -q "setup" && sed -n "1,8p" "$EX" | grep -q "comment out" && sed -n "1,8p" "$EX" | grep -q "chat-template-file"'
|
||||
check "the example price is marked as an example" eval 'grep -E "^ price:" "$EX" | grep -q "example"'
|
||||
check "the header names setup and clean and their flags" \
|
||||
eval '(for w in "setup " "clean " --weights --state --price --model; do grep -q -F -- "$w" <(sed -n "2,60p" "$M") || exit 1; done)'
|
||||
hermetic_done
|
||||
exit $fail
|
||||
+533
-159
@@ -1,6 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start, stop and check the local model server: llama-server (router mode) and the model gate in one container, on
|
||||
# the factory's engine, with the GPU (yggdrasil docs/2-engineering/specs/2026-10-25-local-models.md, 3.4 and 3.10).
|
||||
# Set up, start, stop, check and clean the local model server: llama-server (router mode) and the model gate in one
|
||||
# container, on the factory's engine, with the GPU (yggdrasil docs/2-engineering/specs/2026-10-25-local-models.md, 3.4
|
||||
# and 3.10).
|
||||
# ygg-models.sh [--models FILE] [--folder DIR] setup [--weights DIR] [--state DIR] [--price IN:OUT[:CACHE_READ]] [--model NAME]
|
||||
# the whole first-time setup, safe to run again: each step is skipped when
|
||||
# it is in place, and prints one line that says done or already in place.
|
||||
# In order: ygg-gpu.sh check (and ygg-gpu.sh setup when it reports a
|
||||
# missing step); the list: when FILE is missing, it is written atomically
|
||||
# from scripts/testdata/models/models.example.yaml with the weights folder
|
||||
# (default DIR/models/weights), the state folder (default
|
||||
# DIR/models/state) and the price in dollars per million tokens, and both
|
||||
# folders are made; --price is required then, and a missing or bad one is
|
||||
# refused before anything changes; an existing FILE is used as is (the three
|
||||
# flags only matter for a new list); the downloads: each file of the list's
|
||||
# `download` (`ygg models sources`) that is missing under the weights folder,
|
||||
# with `hf download REPO PATH --revision REV --local-dir WEIGHTS` (hf, else
|
||||
# `uvx --from huggingface_hub hf`, else refused); a file that is there is
|
||||
# never deleted or overwritten; the image, built when its tag is missing;
|
||||
# start; load NAME (default: the first model of the list in sorted order).
|
||||
# When NAME's chat-template-file is /models/chat-template.jinja and that file
|
||||
# is missing under the weights folder, the first start leaves that line out
|
||||
# of the preset (llama-server cannot load a model whose template file is
|
||||
# missing), then template NAME saves it, and setup stops, starts and loads
|
||||
# again with the whole preset. Any other missing template file is refused
|
||||
# before anything changes, and so is making the template while a container
|
||||
# other than the server is on the network (a run uses the server: it is
|
||||
# never stopped under it). A failed template step says the server runs
|
||||
# without the template. Everything setup refuses is checked before the
|
||||
# GPU step: the price, the folders, the new list (ygg reads it), the model,
|
||||
# the template, the download paths (gungnir's patterns: each part starts
|
||||
# with a letter or digit), hf or uvx when a file is missing, and the network.
|
||||
# ygg-models.sh [--models FILE] [--folder DIR] clean [--state] [--weights]
|
||||
# decides first, then changes: while a container other than the server is
|
||||
# on the network (a run still uses it), it refuses and names them, and
|
||||
# touches nothing (the server keeps running); with --weights, a download
|
||||
# path that is not plain, or a symlinked <weights>/.cache or
|
||||
# .cache/huggingface (it may be the operator's own Hugging Face cache, with
|
||||
# its token), is refused the same way. Then it stops and removes
|
||||
# the container, removes the network, then the image, each only when it is
|
||||
# there, each saying what it did. --state also removes preset.ini and
|
||||
# allowed from the state folder, and their temporary files, never the
|
||||
# folder. --weights also deletes the files the list's `download` names under
|
||||
# the weights folder, and <weights>/.cache/huggingface (the state hf
|
||||
# download leaves there): never another file or folder. The list stays.
|
||||
# ygg-models.sh [--models FILE] [--folder DIR] start build the image if its tag is missing, create the internal network if it is
|
||||
# absent, write the preset into the state folder, run the container, wait
|
||||
# until the server answers
|
||||
@@ -12,8 +54,8 @@
|
||||
# model's chat template from the server (GET /props?model=NAME), replace the
|
||||
# line that refuses a later system message with a system turn (refused
|
||||
# unless that line is found exactly once) and save it as
|
||||
# <WEIGHTS>/chat-template.jinja; then uncomment `chat-template-file:
|
||||
# /models/chat-template.jinja` in the list's args, and stop and start
|
||||
# <WEIGHTS>/chat-template.jinja, which the list's `chat-template-file:
|
||||
# /models/chat-template.jinja` uses at the next start (stop, then start)
|
||||
# DIR (default $HOME/ygg-it) holds factory.yaml, which names the engine (sandbox.engine): it is passed to ygg-engine.sh
|
||||
# and ygg-gpu.sh as --folder, and `ygg models server` and `ygg models build` read DIR/factory.yaml. The engine is a
|
||||
# Podman machine that sees the host's files only through factory.yaml's `sandbox.mounts`: both the weights and the
|
||||
@@ -23,8 +65,10 @@
|
||||
# scripts/testdata/models/models.example.yaml). This script never parses it: `ygg models server --config
|
||||
# DIR/factory.yaml FILE` prints the server's eight lines NAME=, NETWORK=, PORT=, HOST_PORT=, WEIGHTS=, STATE=,
|
||||
# ENGINE_WEIGHTS=, ENGINE_STATE= (read as data, never evaluated; the last two are the folders as the engine sees
|
||||
# them, mapped through sandbox.mounts; this script checks and writes the host ones), and
|
||||
# `ygg models preset FILE` prints the preset, which `start` writes to <STATE>/preset.ini.
|
||||
# them, mapped through sandbox.mounts; this script checks and writes the host ones),
|
||||
# `ygg models preset FILE` prints the preset, which `start` writes to <STATE>/preset.ini (setup reads the models
|
||||
# and their chat-template-file from it), and `ygg models sources FILE` prints `<model> <repo> <revision> <path>`
|
||||
# per file to download (each a word; this script checks each path is relative and stays under the weights folder).
|
||||
# The container: named NAME, on NETWORK alone (an internal network), --device nvidia.com/gpu=all, --cap-drop=ALL and
|
||||
# --security-opt=no-new-privileges, the engine-side weights at /models and state at /etc/ygg-models (both read-only), 127.0.0.1:<HOST_PORT> published to 8080 (the only
|
||||
# published port), and YGG_MODELGATE_PORT=<PORT> for the gate. `start` first runs ygg-gpu.sh check: a stale CDI spec
|
||||
@@ -42,12 +86,19 @@ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ygg="${YGG_CMD:-ygg}"
|
||||
engine_sh="${YGG_ENGINE_SH:-$HERE/ygg-engine.sh}"
|
||||
gpu_sh="${YGG_GPU_SH:-$HERE/ygg-gpu.sh}"
|
||||
example="$HERE/testdata/models/models.example.yaml"
|
||||
start_timeout="${YGG_MODELS_START_TIMEOUT:-120}"
|
||||
load_timeout="${YGG_MODELS_TIMEOUT:-300}"
|
||||
poll="${YGG_MODELS_POLL:-1}"
|
||||
|
||||
usage() { echo "usage: ygg-models.sh [--models FILE] [--folder DIR] start|stop|status|load NAME|template NAME" >&2; exit 2; }
|
||||
usage() {
|
||||
echo "usage: ygg-models.sh [--models FILE] [--folder DIR] start|stop|status|load NAME|template NAME" >&2
|
||||
echo " ygg-models.sh [--models FILE] [--folder DIR] setup [--weights DIR] [--state DIR] [--price IN:OUT[:CACHE_READ]] [--model NAME]" >&2
|
||||
echo " ygg-models.sh [--models FILE] [--folder DIR] clean [--state] [--weights]" >&2
|
||||
exit 2
|
||||
}
|
||||
die() { echo "ygg-models: $*" >&2; exit 1; }
|
||||
bad() { echo "ygg-models: $*; nothing was changed" >&2; exit 2; }
|
||||
list="${XDG_CONFIG_HOME:-$HOME/.config}/ygg/models.yaml"
|
||||
dir="$HOME/ygg-it"
|
||||
while [ $# -gt 0 ]; do
|
||||
@@ -59,31 +110,65 @@ while [ $# -gt 0 ]; do
|
||||
done
|
||||
[ $# -ge 1 ] || usage
|
||||
cmd="$1"; shift
|
||||
new_weights="" new_state="" price="" want_model="" clean_state="" clean_weights=""
|
||||
case "$cmd" in
|
||||
start | stop | status) [ $# -eq 0 ] || usage ;;
|
||||
load | template) [ $# -eq 1 ] || usage ;;
|
||||
setup)
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--weights) [ $# -ge 2 ] || usage; new_weights="$2"; shift 2 ;;
|
||||
--state) [ $# -ge 2 ] || usage; new_state="$2"; shift 2 ;;
|
||||
--price) [ $# -ge 2 ] || usage; price="$2"; shift 2 ;;
|
||||
--model) [ $# -ge 2 ] || usage; want_model="$2"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done ;;
|
||||
clean)
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--state) clean_state=1; shift ;;
|
||||
--weights) clean_weights=1; shift ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
|
||||
# The server's lines are data: read by key, never evaluated, so a path may hold spaces.
|
||||
name="" network="" port="" host_port="" weights="" state="" engine_weights="" engine_state=""
|
||||
server_lines="$("$ygg" models server --config "$dir/factory.yaml" "$list")" || exit 1
|
||||
while IFS='=' read -r key value; do
|
||||
case "$key" in
|
||||
NAME) name="$value" ;;
|
||||
NETWORK) network="$value" ;;
|
||||
PORT) port="$value" ;;
|
||||
HOST_PORT) host_port="$value" ;;
|
||||
WEIGHTS) weights="$value" ;;
|
||||
STATE) state="$value" ;;
|
||||
ENGINE_WEIGHTS) engine_weights="$value" ;;
|
||||
ENGINE_STATE) engine_state="$value" ;;
|
||||
# read_server FILE: the server's lines are data, read by key, never evaluated, so a path may hold spaces.
|
||||
name="" network="" port="" host_port="" weights="" state="" engine_weights="" engine_state="" url=""
|
||||
read_server() {
|
||||
local lines key value v
|
||||
lines="$("$ygg" models server --config "$dir/factory.yaml" "$1")" || exit 1
|
||||
while IFS='=' read -r key value; do
|
||||
case "$key" in
|
||||
NAME) name="$value" ;;
|
||||
NETWORK) network="$value" ;;
|
||||
PORT) port="$value" ;;
|
||||
HOST_PORT) host_port="$value" ;;
|
||||
WEIGHTS) weights="$value" ;;
|
||||
STATE) state="$value" ;;
|
||||
ENGINE_WEIGHTS) engine_weights="$value" ;;
|
||||
ENGINE_STATE) engine_state="$value" ;;
|
||||
esac
|
||||
done <<<"$lines"
|
||||
for v in name network port host_port weights state engine_weights engine_state; do
|
||||
[ -n "${!v}" ] || die "ygg models server $1 did not print ${v^^}"
|
||||
done
|
||||
url="http://127.0.0.1:$host_port"
|
||||
}
|
||||
|
||||
# check_names: names go to podman as they are: a network such as host would ignore -p, and a name that starts with -
|
||||
# is a flag.
|
||||
check_names() {
|
||||
local v
|
||||
for v in "$name" "$network"; do
|
||||
[[ "$v" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]*$ ]] || die "the name '$v' in $list is not a plain container or network name"
|
||||
done
|
||||
case "$network" in
|
||||
host | none | bridge | pasta | private | slirp4netns) die "the network '$network' in $list is a reserved podman network, not an internal one; name your own" ;;
|
||||
esac
|
||||
done <<<"$server_lines"
|
||||
for v in name network port host_port weights state engine_weights engine_state; do
|
||||
[ -n "${!v}" ] || die "ygg models server $list did not print ${v^^}"
|
||||
done
|
||||
url="http://127.0.0.1:$host_port"
|
||||
}
|
||||
|
||||
pm() { "$engine_sh" run --folder "$dir" -- podman "$@"; }
|
||||
|
||||
@@ -106,6 +191,12 @@ exists() {
|
||||
case "$rc" in 0 | 1) return "$rc" ;; *) echo "ygg-models: podman $1 exists $2 failed (exit $rc)" >&2; exit 2 ;; esac
|
||||
}
|
||||
|
||||
# build_image TAG: ygg models build, through the engine.
|
||||
build_image() {
|
||||
echo "ygg-models: building the image $1 (the first build takes a long time: CUDA)"
|
||||
"$engine_sh" run --folder "$dir" -- "$ygg" models build --config "$dir/factory.yaml" || die "ygg models build failed"
|
||||
}
|
||||
|
||||
# http METHOD PATH [BODY]: sets code and body (the answer's text). A server that does not answer sets code to 000.
|
||||
code="" body=""
|
||||
http() {
|
||||
@@ -142,6 +233,7 @@ else:
|
||||
|
||||
# The model's template refuses a system message after the first (Claude Code sends one). This renders it as a system
|
||||
# turn, and refuses a template that does not hold the refusing line exactly once. Reads /props on stdin, writes argv[1].
|
||||
# A template with the fix already prints ALREADY and writes nothing, unless argv[2] is keep: then it is written as it is.
|
||||
fix_template='
|
||||
import json, sys
|
||||
t = json.loads(sys.stdin.read(), strict=False)["chat_template"]
|
||||
@@ -149,101 +241,40 @@ old = "{{- raise_exception(\x27System message must be at the beginning.\x27) }}"
|
||||
new = "{{- \x27<|im_start|>system\\n\x27 + content + \x27<|im_end|>\\n\x27 }}"
|
||||
if t.count(old) == 0 and t.count(new) >= 1:
|
||||
print("ALREADY")
|
||||
if sys.argv[2] == "keep":
|
||||
open(sys.argv[1], "w").write(t)
|
||||
sys.exit(0)
|
||||
if t.count(old) != 1:
|
||||
sys.exit("ygg-models: did not find the line that refuses later system messages in the chat template exactly once; check the template by hand")
|
||||
open(sys.argv[1], "w").write(t.replace(old, new))
|
||||
'
|
||||
|
||||
case "$cmd" in
|
||||
status)
|
||||
cstate="$(container_state)" || exit $?
|
||||
if [ "$cstate" != running ]; then echo "not running"; exit 3; fi
|
||||
http GET /models
|
||||
loaded=unknown
|
||||
[ "$code" != 200 ] || loaded="$(printf '%s' "$body" | python3 -I -c "$parse" loaded 2>/dev/null)" || loaded=unknown
|
||||
echo "running on $url, loaded: $loaded"
|
||||
;;
|
||||
stop)
|
||||
cstate="$(container_state)" || exit $?
|
||||
if [ "$cstate" = absent ]; then echo "not running"; exit 0; fi
|
||||
pm stop -t 30 "$name" >/dev/null || true
|
||||
pm rm -f "$name" >/dev/null
|
||||
echo "stopped $name"
|
||||
;;
|
||||
template)
|
||||
model="$1"
|
||||
[[ "$model" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] || die "the model name '$model' is not a plain name"
|
||||
cstate="$(container_state)" || exit $?
|
||||
[ "$cstate" = running ] || die "the model server is not running; start it: scripts/ygg-models.sh start"
|
||||
[ -d "$weights" ] || die "the weights folder $weights does not exist"
|
||||
http GET /models
|
||||
[ "$code" = 200 ] || die "model $model: GET /models answered $code at $url"
|
||||
st="$(printf '%s' "$body" | python3 -I -c "$parse" entry "$model")" || die "model $model: the model list is not JSON"
|
||||
case "$st" in
|
||||
absent) die "model $model: the model server does not list it" ;;
|
||||
loaded) ;;
|
||||
*) die "model $model is not loaded (status $st); load it first: scripts/ygg-models.sh load $model" ;;
|
||||
esac
|
||||
# The router forwards GET /props?model=NAME to the loaded model's server.
|
||||
http GET "/props?model=$model"
|
||||
[ "$code" = 200 ] || die "model $model: GET /props?model=$model answered $code at $url"
|
||||
tmp="$weights/chat-template.jinja.tmp"
|
||||
res="$(printf '%s' "$body" | python3 -I -c "$fix_template" "$tmp")" || { rm -f -- "$tmp"; exit 1; }
|
||||
if [ "$res" = ALREADY ]; then
|
||||
rm -f -- "$tmp"
|
||||
echo "model $model: the fix is already in place (the server's template has the system turn); nothing saved"
|
||||
exit 0
|
||||
fi
|
||||
mv -f "$tmp" "$weights/chat-template.jinja"
|
||||
echo "saved $weights/chat-template.jinja; in the list, uncomment chat-template-file: /models/chat-template.jinja, then:"
|
||||
echo " scripts/ygg-models.sh stop && scripts/ygg-models.sh start"
|
||||
;;
|
||||
load)
|
||||
model="$1"
|
||||
cstate="$(container_state)" || exit $?
|
||||
[ "$cstate" = running ] || die "the model server is not running; start it: scripts/ygg-models.sh start"
|
||||
deadline=$((SECONDS + load_timeout))
|
||||
st=""
|
||||
sent=""
|
||||
while :; do
|
||||
http GET /models
|
||||
[ "$code" = 200 ] || die "model $model: GET /models answered $code at $url (start the server: scripts/ygg-models.sh start)"
|
||||
st="$(printf '%s' "$body" | python3 -I -c "$parse" entry "$model")" || die "model $model: the model list is not JSON"
|
||||
case "$st" in
|
||||
absent) die "model $model: the model server does not list it" ;;
|
||||
loaded) echo "loaded $model"; exit 0 ;;
|
||||
esac
|
||||
# The router keeps `failed: true` until a new load attempt, so a model that is not loaded is asked to load
|
||||
# (as odin's Ready does); a failure counts only after that.
|
||||
if [ -z "$sent" ]; then
|
||||
http POST /models/load "$(python3 -I -c 'import json,sys; print(json.dumps({"model": sys.argv[1]}, separators=(",", ":")))' "$model")"
|
||||
case "$code" in
|
||||
200) ;;
|
||||
400) case "$body" in *"model is already running"*) ;; *) die "model $model: the load request answered 400: $body" ;; esac ;;
|
||||
*) die "model $model: the load request answered $code: $body" ;;
|
||||
esac
|
||||
sent=1
|
||||
sleep "$poll"
|
||||
continue
|
||||
fi
|
||||
case "$st" in failed*) die "model $model: the load failed, exit_code ${st#failed }" ;; esac
|
||||
[ "$SECONDS" -lt "$deadline" ] || die "model $model: not loaded within ${load_timeout}s (status $st, wanted loaded)"
|
||||
sleep "$poll"
|
||||
done
|
||||
;;
|
||||
start)
|
||||
# Names go to podman as they are: a network such as host would ignore -p, and a name that starts with - is a flag.
|
||||
for v in "$name" "$network"; do
|
||||
[[ "$v" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]*$ ]] || die "the name '$v' in $list is not a plain container or network name"
|
||||
done
|
||||
case "$network" in
|
||||
host | none | bridge | pasta | private | slirp4netns) die "the network '$network' in $list is a reserved podman network, not an internal one; name your own" ;;
|
||||
esac
|
||||
[ -d "$weights" ] || die "the weights folder $weights does not exist (weights in $list); put the model files there"
|
||||
tag="$("$ygg" models image)" || exit 1
|
||||
preset="$("$ygg" models preset "$list")" || exit 1
|
||||
# The GPU: the engine resolves the CDI device before the container exists.
|
||||
# do_stop: stop and remove the container; sets stop_result to stopped or absent.
|
||||
stop_result=""
|
||||
do_stop() {
|
||||
local cstate
|
||||
cstate="$(container_state)" || exit $?
|
||||
if [ "$cstate" = absent ]; then stop_result=absent; return 0; fi
|
||||
pm stop -t 30 "$name" >/dev/null || true
|
||||
pm rm -f "$name" >/dev/null
|
||||
stop_result=stopped
|
||||
}
|
||||
|
||||
# do_start [GPU] [NO_TEMPLATE]: start the server; sets start_result to started or running. With GPU set to skip, the GPU
|
||||
# is not checked (setup checked it). With NO_TEMPLATE set, the preset leaves out `chat-template-file =
|
||||
# /models/chat-template.jinja` (setup makes that file once the model is loaded).
|
||||
start_result=""
|
||||
do_start() {
|
||||
local gpu="${1:-check}" no_template="${2:-}" tag preset rc gpu_out cstate out deadline
|
||||
check_names
|
||||
[ -d "$weights" ] || die "the weights folder $weights does not exist (weights in $list); put the model files there"
|
||||
tag="$("$ygg" models image)" || exit 1
|
||||
preset="$("$ygg" models preset "$list")" || exit 1
|
||||
if [ -n "$no_template" ]; then
|
||||
preset="$(printf '%s\n' "$preset" | grep -v -x -F 'chat-template-file = /models/chat-template.jinja')" || true
|
||||
fi
|
||||
# The GPU: the engine resolves the CDI device before the container exists.
|
||||
if [ "$gpu" = check ]; then
|
||||
rc=0
|
||||
gpu_out="$("$gpu_sh" check --folder "$dir" 2>&1)" || rc=$?
|
||||
case "$rc" in
|
||||
@@ -258,55 +289,398 @@ case "$cmd" in
|
||||
fi ;;
|
||||
*) printf '%s\n' "$gpu_out" >&2; exit "$rc" ;;
|
||||
esac
|
||||
cstate="$(container_state)" || exit $?
|
||||
case "$cstate" in
|
||||
running) echo "already running on $url (stop first to apply list changes: scripts/ygg-models.sh stop)"; exit 0 ;;
|
||||
stopped) pm rm -f "$name" >/dev/null ;;
|
||||
fi
|
||||
cstate="$(container_state)" || exit $?
|
||||
case "$cstate" in
|
||||
running) start_result=running; return 0 ;;
|
||||
stopped) pm rm -f "$name" >/dev/null ;;
|
||||
esac
|
||||
exists image "$tag" || build_image "$tag"
|
||||
if exists network "$network"; then
|
||||
[ "$(pm network inspect -f '{{.Internal}}' "$network")" = true ] ||
|
||||
die "the network $network exists but is not internal; remove it (podman network rm $network) or name another in $list"
|
||||
else
|
||||
pm network create --internal "$network" >/dev/null
|
||||
fi
|
||||
mkdir -p "$state"
|
||||
printf '%s\n' "$preset" >"$state/preset.ini.tmp"
|
||||
mv -f "$state/preset.ini.tmp" "$state/preset.ini"
|
||||
rc=0
|
||||
# The engine sees the folders at their engine-side paths (sandbox.mounts); the host paths are for this script.
|
||||
# ,Z relabels the mounts for SELinux, as the sandbox driver does for its own read-only mounts.
|
||||
out="$(pm run -d --name "$name" --network "$network" --device nvidia.com/gpu=all \
|
||||
--cap-drop=ALL --security-opt=no-new-privileges \
|
||||
-v "$engine_weights:/models:ro,Z" -v "$engine_state:/etc/ygg-models:ro,Z" \
|
||||
-p "127.0.0.1:$host_port:8080" -e "YGG_MODELGATE_PORT=$port" "$tag" 2>&1)" || rc=$?
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
printf '%s\n' "$out" >&2
|
||||
case "$out" in
|
||||
*"unresolvable CDI devices"*) die "the engine cannot resolve the GPU; run scripts/ygg-gpu.sh setup" ;;
|
||||
esac
|
||||
if ! exists image "$tag"; then
|
||||
echo "ygg-models: building the image $tag (the first build takes a long time: CUDA)"
|
||||
"$engine_sh" run --folder "$dir" -- "$ygg" models build --config "$dir/factory.yaml" || die "ygg models build failed"
|
||||
fi
|
||||
if exists network "$network"; then
|
||||
[ "$(pm network inspect -f '{{.Internal}}' "$network")" = true ] ||
|
||||
die "the network $network exists but is not internal; remove it (podman network rm $network) or name another in $list"
|
||||
else
|
||||
pm network create --internal "$network" >/dev/null
|
||||
fi
|
||||
mkdir -p "$state"
|
||||
printf '%s\n' "$preset" >"$state/preset.ini.tmp"
|
||||
mv -f "$state/preset.ini.tmp" "$state/preset.ini"
|
||||
rc=0
|
||||
# The engine sees the folders at their engine-side paths (sandbox.mounts); the host paths are for this script.
|
||||
# ,Z relabels the mounts for SELinux, as the sandbox driver does for its own read-only mounts.
|
||||
out="$(pm run -d --name "$name" --network "$network" --device nvidia.com/gpu=all \
|
||||
--cap-drop=ALL --security-opt=no-new-privileges \
|
||||
-v "$engine_weights:/models:ro,Z" -v "$engine_state:/etc/ygg-models:ro,Z" \
|
||||
-p "127.0.0.1:$host_port:8080" -e "YGG_MODELGATE_PORT=$port" "$tag" 2>&1)" || rc=$?
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
printf '%s\n' "$out" >&2
|
||||
case "$out" in
|
||||
*"unresolvable CDI devices"*) die "the engine cannot resolve the GPU; run scripts/ygg-gpu.sh setup" ;;
|
||||
esac
|
||||
exit 1
|
||||
fi
|
||||
deadline=$((SECONDS + start_timeout))
|
||||
while :; do
|
||||
cstate="$(container_state)" || exit $?
|
||||
if [ "$cstate" != running ]; then
|
||||
echo "ygg-models: the container exited while starting; the end of its log:" >&2
|
||||
pm logs --tail 20 "$name" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
deadline=$((SECONDS + start_timeout))
|
||||
while :; do
|
||||
cstate="$(container_state)" || exit $?
|
||||
if [ "$cstate" != running ]; then
|
||||
echo "ygg-models: the container exited while starting; the end of its log:" >&2
|
||||
pm logs --tail 20 "$name" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
http GET /models
|
||||
[ "$code" != 200 ] || break
|
||||
if [ "$SECONDS" -ge "$deadline" ]; then
|
||||
echo "ygg-models: the server did not answer on $url within ${start_timeout}s; the end of its log:" >&2
|
||||
pm logs --tail 20 "$name" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
http GET /models
|
||||
[ "$code" != 200 ] || break
|
||||
if [ "$SECONDS" -ge "$deadline" ]; then
|
||||
echo "ygg-models: the server did not answer on $url within ${start_timeout}s; the end of its log:" >&2
|
||||
pm logs --tail 20 "$name" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
sleep "$poll"
|
||||
done
|
||||
start_result=started
|
||||
}
|
||||
|
||||
# do_load NAME: load the model and wait until it is loaded; sets load_result to loaded or already.
|
||||
load_result=""
|
||||
do_load() {
|
||||
local model="$1" cstate deadline st="" sent=""
|
||||
cstate="$(container_state)" || exit $?
|
||||
[ "$cstate" = running ] || die "the model server is not running; start it: scripts/ygg-models.sh start"
|
||||
deadline=$((SECONDS + load_timeout))
|
||||
while :; do
|
||||
http GET /models
|
||||
[ "$code" = 200 ] || die "model $model: GET /models answered $code at $url (start the server: scripts/ygg-models.sh start)"
|
||||
st="$(printf '%s' "$body" | python3 -I -c "$parse" entry "$model")" || die "model $model: the model list is not JSON"
|
||||
case "$st" in
|
||||
absent) die "model $model: the model server does not list it" ;;
|
||||
loaded) if [ -n "$sent" ]; then load_result=loaded; else load_result=already; fi; return 0 ;;
|
||||
esac
|
||||
# The router keeps `failed: true` until a new load attempt, so a model that is not loaded is asked to load
|
||||
# (as odin's Ready does); a failure counts only after that.
|
||||
if [ -z "$sent" ]; then
|
||||
http POST /models/load "$(python3 -I -c 'import json,sys; print(json.dumps({"model": sys.argv[1]}, separators=(",", ":")))' "$model")"
|
||||
case "$code" in
|
||||
200) ;;
|
||||
400) case "$body" in *"model is already running"*) ;; *) die "model $model: the load request answered 400: $body" ;; esac ;;
|
||||
*) die "model $model: the load request answered $code: $body" ;;
|
||||
esac
|
||||
sent=1
|
||||
sleep "$poll"
|
||||
continue
|
||||
fi
|
||||
case "$st" in failed*) die "model $model: the load failed, exit_code ${st#failed }" ;; esac
|
||||
[ "$SECONDS" -lt "$deadline" ] || die "model $model: not loaded within ${load_timeout}s (status $st, wanted loaded)"
|
||||
sleep "$poll"
|
||||
done
|
||||
}
|
||||
|
||||
# do_template NAME [keep]: save the loaded model's fixed template as <weights>/chat-template.jinja; sets template_result
|
||||
# to saved, or already (the server's template has the fix; with keep, it is saved as it is).
|
||||
template_result=""
|
||||
do_template() {
|
||||
local model="$1" keep="${2:-}" cstate st tmp res
|
||||
[[ "$model" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] || die "the model name '$model' is not a plain name"
|
||||
cstate="$(container_state)" || exit $?
|
||||
[ "$cstate" = running ] || die "the model server is not running; start it: scripts/ygg-models.sh start"
|
||||
[ -d "$weights" ] || die "the weights folder $weights does not exist"
|
||||
http GET /models
|
||||
[ "$code" = 200 ] || die "model $model: GET /models answered $code at $url"
|
||||
st="$(printf '%s' "$body" | python3 -I -c "$parse" entry "$model")" || die "model $model: the model list is not JSON"
|
||||
case "$st" in
|
||||
absent) die "model $model: the model server does not list it" ;;
|
||||
loaded) ;;
|
||||
*) die "model $model is not loaded (status $st); load it first: scripts/ygg-models.sh load $model" ;;
|
||||
esac
|
||||
# The router forwards GET /props?model=NAME to the loaded model's server.
|
||||
http GET "/props?model=$model"
|
||||
[ "$code" = 200 ] || die "model $model: GET /props?model=$model answered $code at $url"
|
||||
tmp="$weights/chat-template.jinja.tmp"
|
||||
res="$(printf '%s' "$body" | python3 -I -c "$fix_template" "$tmp" "${keep:-no}")" || { rm -f -- "$tmp"; exit 1; }
|
||||
if [ "$res" = ALREADY ] && [ -z "$keep" ]; then
|
||||
rm -f -- "$tmp"
|
||||
template_result=already
|
||||
return 0
|
||||
fi
|
||||
mv -f "$tmp" "$weights/chat-template.jinja"
|
||||
template_result=saved
|
||||
}
|
||||
|
||||
# A path the list downloads: relative, and each part starts with a letter or digit, so none is an option, . or ..
|
||||
# (gungnir's pattern; the script checks it too, so a ygg that let one through still cannot reach outside weights).
|
||||
plain_rel() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._+-]*(/[A-Za-z0-9][A-Za-z0-9._+-]*)*$ ]]; }
|
||||
|
||||
# network_users: print the containers on the model network other than the server, one per line (nothing when the
|
||||
# network is absent). Reading only: it changes nothing.
|
||||
network_users() {
|
||||
local users
|
||||
exists network "$network" || return 0
|
||||
users="$(pm container ls -a --filter "network=$network" --format '{{.Names}}')" || exit 2
|
||||
printf '%s\n' "$users" | grep -v -x -F -e "$name" -e '' || true
|
||||
}
|
||||
# refuse_if_used WHAT: stop with exit 1, naming the containers, when another container is on the network. WHAT says
|
||||
# what was not done.
|
||||
refuse_if_used() {
|
||||
local others
|
||||
others="$(network_users)" || exit $?
|
||||
[ -z "$others" ] ||
|
||||
die "other containers are on the network $network, so $1: $(printf '%s\n' "$others" | tr '\n' ' ')- let their runs end, then run this again (scripts/ygg-clean.sh removes the leftover containers of runs that have ended)"
|
||||
}
|
||||
|
||||
# read_sources FILE: fill src_model, src_repo, src_rev and src_path from `ygg models sources FILE`, each checked.
|
||||
src_model=() src_repo=() src_rev=() src_path=()
|
||||
read_sources() {
|
||||
local out m repo rev path extra
|
||||
out="$("$ygg" models sources "$1")" || exit 1
|
||||
while read -r m repo rev path extra; do
|
||||
[ -n "$m" ] || continue
|
||||
[ -n "$path" ] && [ -z "$extra" ] || die "ygg models sources $1 printed a line that is not <model> <repo> <revision> <path>: $m $repo $rev $path $extra"
|
||||
[[ "$repo" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$ ]] || die "model $m: the download repo '$repo' is not a plain <owner>/<name>"
|
||||
[[ "$rev" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$ ]] || die "model $m: the download revision '$rev' is not plain"
|
||||
plain_rel "$path" || die "model $m: the download file '$path' is not a plain path under the weights folder"
|
||||
src_model+=("$m") src_repo+=("$repo") src_rev+=("$rev") src_path+=("$path")
|
||||
done <<<"$out"
|
||||
}
|
||||
|
||||
# setup: the whole first-time setup (the header says the steps); everything it refuses is checked before step 1.
|
||||
do_setup() {
|
||||
local src tmpd="" abs p_in p_out p_cr extra v nv rc gpu_out preset line sec model tpl tpl_host="" tpl_make="" i missing=0 hf=()
|
||||
local -a models=()
|
||||
local -A tpls=()
|
||||
local num='^[0-9]+(\.[0-9]+)?$'
|
||||
if [ -e "$list" ]; then
|
||||
src="$list"
|
||||
else
|
||||
[ -n "$price" ] || bad "$list does not exist, and a new list needs --price IN:OUT[:CACHE_READ] (OpenRouter's price of the model, dollars per million tokens)"
|
||||
IFS=: read -r p_in p_out p_cr extra <<<"$price"
|
||||
[[ "$p_in" =~ $num ]] && [[ "$p_out" =~ $num ]] && { [ -z "$p_cr" ] || [[ "$p_cr" =~ $num ]]; } && [ -z "$extra" ] ||
|
||||
bad "--price '$price' is not IN:OUT or IN:OUT:CACHE_READ (numbers, dollars per million tokens)"
|
||||
case "$dir" in /*) abs="$dir" ;; *) abs="$PWD/$dir" ;; esac
|
||||
new_weights="${new_weights:-$abs/models/weights}" new_state="${new_state:-$abs/models/state}"
|
||||
for v in weights state; do
|
||||
nv="new_$v"
|
||||
[[ "${!nv}" =~ ^/[A-Za-z0-9._/@+-]*[A-Za-z0-9._@+-]$ ]] ||
|
||||
bad "--$v '${!nv}' is not an absolute path of letters, digits and . _ / @ + - only (ygg refuses others)"
|
||||
done
|
||||
echo "ready on $url (container $name; log: podman logs $name)"
|
||||
[ -f "$example" ] || die "the example list $example is missing"
|
||||
for v in '^ weights: /abs/path/to/weights' '^ state: /abs/path/to/state' '^ price: '; do
|
||||
[ "$(grep -c -E -- "$v" "$example")" = 1 ] || die "the example list $example does not hold one line that matches '$v'"
|
||||
done
|
||||
tmpd="$(mktemp -d)"
|
||||
# shellcheck disable=SC2064 # the folder is known now
|
||||
trap "rm -rf -- '$tmpd'" EXIT
|
||||
src="$tmpd/models.yaml"
|
||||
{
|
||||
echo "# Written by scripts/ygg-models.sh setup from scripts/testdata/models/models.example.yaml."
|
||||
sed -E -e "s|^( weights: )/abs/path/to/weights|\1$new_weights|" -e "s|^( state: )/abs/path/to/state|\1$new_state|" \
|
||||
-e "s|^ price: .*| price: {input: $p_in, output: $p_out${p_cr:+, cache_read: $p_cr}} # OpenRouter's price, dollars per million tokens (setup --price)|" \
|
||||
"$example"
|
||||
} >"$src"
|
||||
fi
|
||||
# ygg reads the list (a new one too, before it is written): the server, the folders under sandbox.mounts.
|
||||
read_server "$src"
|
||||
check_names
|
||||
preset="$("$ygg" models preset "$src")" || exit 1
|
||||
sec=""
|
||||
while IFS= read -r line; do
|
||||
case "$line" in
|
||||
"["*"]") sec="${line#[}"; sec="${sec%]}"; models+=("$sec") ;;
|
||||
"chat-template-file = "*) [ -z "$sec" ] || tpls[$sec]="${line#chat-template-file = }" ;;
|
||||
esac
|
||||
done <<<"$preset"
|
||||
[ "${#models[@]}" -gt 0 ] || die "the list $list has no model"
|
||||
if [ -n "$want_model" ]; then
|
||||
model=""
|
||||
for v in "${models[@]}"; do [ "$v" != "$want_model" ] || model="$v"; done
|
||||
[ -n "$model" ] || bad "--model $want_model is not in the list (it has: ${models[*]})"
|
||||
else
|
||||
model="$(printf '%s\n' "${models[@]}" | LC_ALL=C sort | head -n 1)"
|
||||
fi
|
||||
tpl="${tpls[$model]:-}"
|
||||
case "$tpl" in
|
||||
"") ;;
|
||||
/models/*)
|
||||
plain_rel "${tpl#/models/}" || die "model $model: chat-template-file $tpl is not a plain path under /models"
|
||||
tpl_host="$weights/${tpl#/models/}"
|
||||
if [ ! -e "$tpl_host" ]; then
|
||||
[ "$tpl" = /models/chat-template.jinja ] ||
|
||||
die "model $model: chat-template-file $tpl is missing ($tpl_host); setup makes only /models/chat-template.jinja: put the file there, or name that one in $list"
|
||||
tpl_make=1
|
||||
fi ;;
|
||||
esac
|
||||
read_sources "$src"
|
||||
for i in "${!src_path[@]}"; do [ -e "$weights/${src_path[$i]}" ] || missing=1; done
|
||||
if [ "$missing" = 1 ]; then
|
||||
if command -v hf >/dev/null 2>&1; then hf=(hf)
|
||||
elif command -v uvx >/dev/null 2>&1; then hf=(uvx --from huggingface_hub hf)
|
||||
else die "a file of the list is missing, and neither hf nor uvx is on PATH to download it: install one (uv: https://docs.astral.sh/uv/; hf: pip install -U huggingface_hub)"
|
||||
fi
|
||||
fi
|
||||
# Making the template stops and starts the server: never under a run that uses it.
|
||||
[ -z "$tpl_make" ] || refuse_if_used "setup changed nothing (making the chat template stops and starts the server)"
|
||||
|
||||
# 1. The GPU.
|
||||
rc=0
|
||||
gpu_out="$("$gpu_sh" check --folder "$dir" 2>&1)" || rc=$?
|
||||
case "$rc" in
|
||||
0) gpu_out="$(printf '%s\n' "$gpu_out" | tail -n 1)"; echo "gpu: already in place (${gpu_out#gpu: })" ;;
|
||||
3)
|
||||
printf '%s\n' "$gpu_out"
|
||||
"$gpu_sh" setup --folder "$dir" || die "scripts/ygg-gpu.sh setup failed"
|
||||
echo "gpu: set up" ;;
|
||||
*) printf '%s\n' "$gpu_out" >&2; exit "$rc" ;;
|
||||
esac
|
||||
# 2. The list, and its folders.
|
||||
if [ -n "$tmpd" ]; then
|
||||
mkdir -p "$(dirname "$list")"
|
||||
v="$(mktemp "$(dirname "$list")/.models.yaml.XXXXXX")"
|
||||
cat "$src" >"$v"
|
||||
mv -f "$v" "$list"
|
||||
mkdir -p "$weights" "$state"
|
||||
echo "model list: written $list (weights $weights, state $state)"
|
||||
else
|
||||
[ -z "$new_weights$new_state$price" ] ||
|
||||
echo "ygg-models: $list exists: --weights, --state and --price count only for a new list, so it is used as is"
|
||||
mkdir -p "$weights"
|
||||
echo "model list: already in place ($list)"
|
||||
fi
|
||||
# 3. The downloads: a file that is there is never touched.
|
||||
[ "${#src_path[@]}" -gt 0 ] || echo "download: none in the list"
|
||||
for i in "${!src_path[@]}"; do
|
||||
if [ -e "$weights/${src_path[$i]}" ]; then echo "download: ${src_path[$i]}: already in place"; continue; fi
|
||||
"${hf[@]}" download "${src_repo[$i]}" "${src_path[$i]}" --revision "${src_rev[$i]}" --local-dir "$weights" ||
|
||||
die "the download of ${src_path[$i]} from ${src_repo[$i]} failed"
|
||||
[ -f "$weights/${src_path[$i]}" ] || die "the download of ${src_path[$i]} from ${src_repo[$i]} left no file at $weights/${src_path[$i]}"
|
||||
echo "download: ${src_path[$i]}: done"
|
||||
done
|
||||
# 4. The image.
|
||||
v="$("$ygg" models image)" || exit 1
|
||||
if exists image "$v"; then echo "image: already in place ($v)"; else build_image "$v"; echo "image: built $v"; fi
|
||||
# 5. Start and load. A server that runs while the template is missing cannot load the model: it starts again
|
||||
# without that line first.
|
||||
if [ -n "$tpl_make" ]; then
|
||||
do_stop
|
||||
[ "$stop_result" = absent ] || echo "server: stopped, to start without the missing chat template"
|
||||
fi
|
||||
do_start skip "$tpl_make"
|
||||
if [ "$start_result" = running ]; then
|
||||
echo "server: already running on $url"
|
||||
[ "$(cat "$state/preset.ini" 2>/dev/null)" = "$preset" ] ||
|
||||
echo "ygg-models: the server runs an older preset than the list's; to apply it: scripts/ygg-models.sh stop && scripts/ygg-models.sh start"
|
||||
elif [ -n "$tpl_make" ]; then
|
||||
echo "server: started on $url, without the chat template for now"
|
||||
else
|
||||
echo "server: started on $url"
|
||||
fi
|
||||
do_load "$model"
|
||||
if [ "$load_result" = already ]; then echo "load: $model: already loaded"; else echo "load: $model: loaded"; fi
|
||||
# 6. The template.
|
||||
if [ -z "$tpl" ]; then
|
||||
echo "template: none in the list for $model"
|
||||
elif [ -z "$tpl_host" ]; then
|
||||
echo "template: the list names $tpl, which is not under /models; setup leaves it as it is"
|
||||
elif [ -z "$tpl_make" ]; then
|
||||
echo "template: already in place ($tpl_host)"
|
||||
else
|
||||
( do_template "$model" keep ) ||
|
||||
die "the server runs without the chat template: Claude Code's later system messages fail; run setup again once the template step works (the error is above)"
|
||||
echo "template: saved $tpl_host"
|
||||
refuse_if_used "the server was not restarted: it runs without the chat template, which Claude Code's later system messages need; run setup again"
|
||||
do_stop
|
||||
do_start skip
|
||||
echo "server: started again on $url, with the chat template"
|
||||
do_load "$model"
|
||||
if [ "$load_result" = already ]; then echo "load: $model: already loaded"; else echo "load: $model: loaded"; fi
|
||||
fi
|
||||
}
|
||||
|
||||
# clean: the container, the network and the image; with --state and --weights, those files too.
|
||||
do_clean() {
|
||||
local tag f n i
|
||||
read_server "$list"
|
||||
check_names
|
||||
# Decide first: a refusal changes nothing.
|
||||
if [ -n "$clean_weights" ]; then
|
||||
read_sources "$list"
|
||||
# A symlinked cache may be the operator's own Hugging Face cache, with its token: never removed through a link.
|
||||
for f in "$weights/.cache" "$weights/.cache/huggingface"; do
|
||||
[ ! -L "$f" ] || die "$f is a symlink (it may be your own Hugging Face cache, with its token), so nothing was removed; remove the link yourself, then run clean again"
|
||||
done
|
||||
fi
|
||||
refuse_if_used "nothing was removed (the server, the network and the image stay)"
|
||||
do_stop
|
||||
if [ "$stop_result" = absent ]; then echo "container $name: not there"; else echo "container $name: removed"; fi
|
||||
if exists network "$network"; then
|
||||
pm network rm "$network" >/dev/null
|
||||
echo "network $network: removed"
|
||||
else
|
||||
echo "network $network: not there"
|
||||
fi
|
||||
tag="$("$ygg" models image)" || exit 1
|
||||
if exists image "$tag"; then
|
||||
pm image rm "$tag" >/dev/null
|
||||
echo "image $tag: removed"
|
||||
else
|
||||
echo "image $tag: not there"
|
||||
fi
|
||||
if [ -n "$clean_state" ]; then
|
||||
n=0
|
||||
for f in "$state/preset.ini" "$state/preset.ini.tmp" "$state/allowed" "$state"/.allowed-*; do
|
||||
if [ -f "$f" ] || [ -L "$f" ]; then rm -f -- "$f"; n=$((n + 1)); fi
|
||||
done
|
||||
echo "state: removed $n files from $state (the folder stays)"
|
||||
fi
|
||||
if [ -n "$clean_weights" ]; then
|
||||
[ "${#src_path[@]}" -gt 0 ] || echo "weights: the list downloads no file; nothing deleted"
|
||||
for i in "${!src_path[@]}"; do
|
||||
f="$weights/${src_path[$i]}"
|
||||
if [ -d "$f" ] && [ ! -L "$f" ]; then echo "weights: $f is a folder; left as it is"
|
||||
elif [ -f "$f" ] || [ -L "$f" ]; then rm -f -- "$f"; echo "deleted $f"
|
||||
else echo "weights: ${src_path[$i]}: not there"
|
||||
fi
|
||||
done
|
||||
# hf download keeps its own state under the folder it downloads to: that exact folder, never another.
|
||||
f="${weights:?}/.cache/huggingface"
|
||||
if [ -d "$f" ] && [ ! -L "$f" ]; then rm -rf -- "$f"; echo "deleted $f (hf download's state)"; fi
|
||||
fi
|
||||
}
|
||||
|
||||
case "$cmd" in
|
||||
setup) do_setup; exit 0 ;;
|
||||
clean) do_clean; exit 0 ;;
|
||||
esac
|
||||
read_server "$list"
|
||||
|
||||
case "$cmd" in
|
||||
status)
|
||||
cstate="$(container_state)" || exit $?
|
||||
if [ "$cstate" != running ]; then echo "not running"; exit 3; fi
|
||||
http GET /models
|
||||
loaded=unknown
|
||||
[ "$code" != 200 ] || loaded="$(printf '%s' "$body" | python3 -I -c "$parse" loaded 2>/dev/null)" || loaded=unknown
|
||||
echo "running on $url, loaded: $loaded"
|
||||
;;
|
||||
stop)
|
||||
do_stop
|
||||
if [ "$stop_result" = absent ]; then echo "not running"; else echo "stopped $name"; fi
|
||||
;;
|
||||
template)
|
||||
do_template "$1"
|
||||
if [ "$template_result" = already ]; then
|
||||
echo "model $1: the fix is already in place (the server's template has the system turn); nothing saved"
|
||||
else
|
||||
echo "saved $weights/chat-template.jinja; the list's chat-template-file: /models/chat-template.jinja uses it from the next start:"
|
||||
echo " scripts/ygg-models.sh stop && scripts/ygg-models.sh start"
|
||||
fi
|
||||
;;
|
||||
load)
|
||||
do_load "$1"
|
||||
echo "loaded $1"
|
||||
;;
|
||||
start)
|
||||
do_start
|
||||
if [ "$start_result" = running ]; then
|
||||
echo "already running on $url (stop first to apply list changes: scripts/ygg-models.sh stop)"
|
||||
else
|
||||
echo "ready on $url (container $name; log: podman logs $name)"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
Reference in new issue
Block a user