mirror of
https://github.com/piabrota/mu.git
synced 2026-11-09 12:21:32 -10:00
dc71c33042
* ygg-models.sh setup and clean; the example names its files and sources (local-models-setup LS-T3) setup does the first-time setup and is safe to run again: the GPU, the list from the example (--price required for a new one, refused before anything changes), the missing download files (hf, else uvx), the image, start, load, and the chat template when the list names a missing one (the first start leaves that line out of the preset, then template, stop, start and load). clean removes the container, the network (refused while another container is on it) and the image; --state and --weights remove only the files they name. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01H5rHSQVuXrzFdTaNXZdSde * ygg-models.sh clean decides first, and --weights removes hf's cache folder (local-models-setup LS-T3) clean lists the other containers on the model network before it stops anything: a refusal names them and leaves the server, the network and the image as they are. --weights also removes <weights>/.cache/huggingface, the state hf download leaves there, and no other path. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01H5rHSQVuXrzFdTaNXZdSde * ygg-models.sh setup never restarts the server under a run; gungnir's patterns (local-models-setup LS-T3) setup checks the model network before step 1 when making the template would stop and start the server, and refuses, changing nothing, while a run is on it; it checks again before the restart. A failed template step says the server runs without the chat template. The download repo, revision and paths use gungnir's patterns, so none starts with a dash. clean --weights refuses a symlinked .cache or .cache/huggingface. The network refusal points at ygg-clean.sh, and the example says to comment out chat-template-file until template has run. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01H5rHSQVuXrzFdTaNXZdSde --------- Co-authored-by: Bruno Guilhermo de Barros Valério <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
48 lines
3.0 KiB
YAML
48 lines
3.0 KiB
YAML
# An example model list for scripts/ygg-models.sh and `ygg models`. `scripts/ygg-models.sh setup --price IN:OUT`
|
|
# writes it to ~/.config/ygg/models.yaml with your folders and price; or copy it there and change the two folders
|
|
# to absolute paths on your machine. Never commit your own list. When you copy it, comment out chat-template-file
|
|
# below until `scripts/ygg-models.sh template NAME` has made the file: a model whose template file is missing does not
|
|
# load (setup does this for you).
|
|
# The engine is a Podman machine that sees host files only through the `sandbox.mounts` of factory.yaml, so both
|
|
# folders must be under a folder of those mounts (for the founder's setup, under ~/ygg-it).
|
|
server:
|
|
name: ygg-models # the container's name and its name on the network
|
|
network: ygg-models # an internal podman network
|
|
port: 8089 # the gate's port on that network
|
|
host_port: 8088 # llama-server on 127.0.0.1 of the host
|
|
weights: /abs/path/to/weights # mounted read-only at /models
|
|
state: /abs/path/to/state # the preset and the allow file; mounted read-only at /etc/ygg-models
|
|
models:
|
|
qwen3.8-27b:
|
|
file: Qwen3.8-27B-UD-Q4_K_M.gguf # under weights
|
|
context: 102400 # the context size; becomes ctx-size in the preset
|
|
output: 32768 # the most output tokens OpenCode asks for
|
|
args: # llama-server preset keys, written as given
|
|
n-gpu-layers: "999"
|
|
flash-attn: "on"
|
|
cache-type-k: q8_0
|
|
cache-type-v: q8_0
|
|
parallel: "1"
|
|
jinja: "true"
|
|
# Sampling: llama-server takes temp, top-k and top-p from the model file (its general.sampling.* keys)
|
|
# unless a key here sets them, so the list sets only what the file lacks. Add temp, top-p or top-k here to
|
|
# override the model's own values.
|
|
min-p: "0" # Qwen's recommendation; the model file has no min_p, and llama-server's own default is 0.05
|
|
reasoning-effort: low
|
|
# Out-of-memory-safe values for a WSL2 machine with limited RAM: a small prompt cache and few checkpoints.
|
|
cache-ram: "2048"
|
|
ctx-checkpoints: "4"
|
|
# MTP: the draft model speeds up decoding. The draft file is under weights, at the path it has in the repo.
|
|
spec-type: draft-mtp
|
|
model-draft: /models/MTP/mtp-Qwen3.8-27B-Q4_0.gguf
|
|
n-gpu-layers-draft: all
|
|
cache-type-k-draft: q8_0
|
|
cache-type-v-draft: q8_0
|
|
# Claude Code sends a later system message that the model's own template refuses. `scripts/ygg-models.sh
|
|
# setup` (or `template NAME`) writes a fixed template to weights/chat-template.jinja, which this line uses.
|
|
chat-template-file: /models/chat-template.jinja
|
|
download: # where setup downloads the files from (Hugging Face), each to its own path under weights
|
|
repo: unsloth/Qwen3.8-27B-GGUF
|
|
files: [Qwen3.8-27B-UD-Q4_K_M.gguf, MTP/mtp-Qwen3.8-27B-Q4_0.gguf]
|
|
price: {input: 0.10, output: 0.40} # example values: OpenRouter's price, dollars per million tokens
|