Run 100 GB-class AI models on one desktop machine.
Open source. OpenAI API. Built on ExLlamaV3.
Quick start · Speed · Models · Docs
 |  |  |  |
Decode and prefill tok/s at 8K (RTX: 5K), served. Kyojin 1.8.10 on Strix Halo, 1.8.9 on RTX and Mac.
Quick start
Strix Halo (AMD): install ROCm torch and build once (doc/install.md), then GLM-5.3-Flash:
git clone https://github.com/Yamz-Labs/kyojin && cd kyojin
hf download yamz-labs/GLM-5.3-Flash-EXL3-Yamz --local-dir ./glm-pack
python tools/glm/serve.py --model ./glm-pack --port 8000 -c 131072 --num-draft 2
RTX 3090 (NVIDIA): build for CUDA once (tools/cuda/README.md), then Qwen3.8-27B with 245K of context (the served settings, one session), with turboderp's 4.0 bpw pack:
hf download turboderp/Qwen3.8-27B-exl3 --revision SC_4.00bpw_H5_V6 --local-dir ./qwen27-pack
python tools/qwen/serve.py --model ./qwen27-pack -c 245760 --k-bits 6 --v-bits 6 --draft-policy mix
Mac Studio (Apple): follow the kyojin-mac install, then DeepSeek V4.1 Flash:
hf download yamz-labs/DeepSeek-V4.1-Flash-EXL3-Yamz --local-dir ./ds41-pack
python -m kyojin_mlx.serve.server ./ds41-pack --sessions 4 --ctx 32768 --port 8080
Every server speaks the OpenAI API. More commands: doc/models.md.
Test it
With a server running on port 8000 (the Mac command above uses 8080):
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": "Say hello in five words."}]}'
Speed by context
Tokens per second against context length (log scale), served through the real server with real text. Each chart says its machine, Kyojin version and date. Higher is better.
Strix Halo (AMD), GLM-5.3-Flash and Qwen3.8-Flash-Next
 |  |
RTX 3090 (NVIDIA), Qwen3.8-27B
 |  |
Mac Studio (Apple), DeepSeek V4.1 Flash
 |  |
Benchmark it yourself
With a server running:
tools/bench.sh --base http://localhost:8000/v1 --reps 3
It prints a Markdown block. Paste it in a benchmark report to share your results (how the figures are measured).
What is inside
- Speculative decoding, on by default; it returns the same tokens as plain decoding.
- Long context: up to 983K tokens (DeepSeek V4.1 Flash on a Mac Studio), 500K on GLM-5.3-Flash.
- Image input on Qwen3.8-27B.
- Optional uncensored mode: a small run-time preset on the normal packs, off by default; it edits no weights. How it works: doc/refusal_hook.md.
Made with our models
One prompt each, written by the model running on Kyojin and published as it came out. Open the gallery.
 Voxel pagoda GLM-5.3-Flash |  Voxel pagoda Qwen3.8-Flash-Next |  Voxel pagoda MiMo-V2.6-Flash |
Models
All packs are EXL3 on Hugging Face: ours on yamz-labs, Qwen3.8-27B from turboderp. Serve commands: doc/models.md.
| Pack | Machine | Size |
|---|---|---|
| GLM-5.3-Flash | Strix Halo | 100.1 GB |
| Qwen3.8-Flash-Next | Strix Halo | 95.4 GB |
| Qwen3.8-27B-exl3 (turboderp) | RTX 3090 | 15 GiB |
| DeepSeek V4.1 Flash | Mac Studio, Strix Halo (SSD tier) | 184.7 GB |
| MiMo-V2.6-Flash-MOPD | Strix Halo | 105.0 GB |
| Flint-40B-A2.5B | RTX 3090 | 17.9 GB |
| Nanbeige4.3-33B | RTX 3090 | 18.7 GB |
Known limits
- Strix Halo needs Linux (Ubuntu/Debian tested), a gfx1151 machine with 128 GB and ROCm 7.0 or newer; ROCm 6.4 has no gfx1151 code (install).
- NVIDIA needs sm_80 or newer; tested on one RTX 3090 with 24 GB (RTX).
- The first GLM launch tunes its kernels for about 10 minutes, and the port stays closed until it finishes (install).
- GLM at
-c 131072leaves about 10 GiB free on a 128 GB machine (install). - The 763B model on one 128 GB machine is experimental: real decode is about 5 to 11 tokens per second (Strix Halo).
Contributors
Thank you to everyone who tests Kyojin on their own machine and writes precise reports. Who did what: doc/credits.md.
- Code: @felladrin, @twstagg, @resynth, @razethion, @MangetsuC, @mostlygeek, @zaxbysauce, @thebigen01, @nobert, @Idriss091, @JoaoFay.
- Reports and benchmarks: @kamjin3086, @OleksKon, @spawnax, @alanoo81, @dturini12, @ChrisC381, @morrisfamily, @xXm-mXx.
Read more
- Install and first launch: requirements, build, per-shell setup, first-launch tuning, system settings for full speed
- Strix Halo, RTX and Mac pages
- Models: weights, serve commands, options
- Benchmarks: how the figures are measured, where each comes from, full tables, share yours with
tools/bench.sh - Troubleshooting, refusal hook, development, credits and licence
Reports and pull requests are welcome: see CONTRIBUTING.md. Kernel notes: doc/strix-halo.md. Upstream guide: doc/upstream.md.
MIT licence. Built on ExLlamaV3 by turboderp, with the first AMD ports by vcruz305 and sdougbrown: full credits. News: @YamzLabs on X.