llama.cpp in router mode serving ~10 model presets from one 16 GB GPU. Last full tune: 2026-07-16 (decode numbers measured then via bench.py, unless a preset comment in config.ini says otherwise). Last full clean sweep: 2026-08-08 03:22 — all 11 presets loaded and ran green (baseline 2.0 GB) after the week's coder-race/KV-fix/gemma-sidecar round.

Box: AMD RX 7600 XT 16 GB (~288 GB/s; backend = Arch llama-cpp 0.5.0 + ggml-vulkan / RADV = llama.cpp b11146 since 2026-10-02, not ROCm — rocm-smi is only the VRAM monitor) + Ryzen 5600X (6 cores) + 48 GB RAM. ⚠ The GPU also drives the display — see VRAM safety.
Endpoints: LAN http://$SERVER:11343/v1 · remote https://<own-domain>/api/v1 (own reverse proxy to the same server; requires an API key. Only the LAN endpoint is keyless). $SERVER = the box's LAN address.
Files here: config.ini (model presets — the section names ARE the API model ids), bench.py (safety-first benchmark), bench-results.md / bench-history.md (ledgers, generated).

How it runs

systemd unit llama.service (on the server, under a dedicated service user) runs a script wrapping:

llama-server --port 11343 --host 0.0.0.0 --models-max 1 --models-preset ~/.config/llamacpp/config.ini

Router mode: each requested model loads on demand in a child process.
--models-max 1 is deliberate and load-bearing: at most one model resident; requesting another LRU-evicts the current one first. Anything ≥2 lets two ~13 GB models stack → VRAM overcommit → GTT spill → whole-PC freeze. Do not raise it casually. (sleep-idle-seconds per preset is the second line of defense.)
Models live in ~/software/models/ (paths in config.ini must stay absolute — llama-server does not expand ~ or env vars in preset values). mmproj/draft files must match the exact filename in the preset (watch for trailing spaces when renaming downloads!).

Deploying config changes: this dir is stowed into ~/.config/llamacpp/ as per-file symlinks, so on the server dotsync (pull; re-stows only if files were added/removed) then sudo systemctl restart llama.service — the router reads presets once at startup. After ANY preset change: restart + ./bench.py -m <changed-ids> and check the guards.

Upgrading llama.cpp itself (2026-10-02, b10615 → b11146): Arch replaced llama.cpp-vulkan with a unified llama-cpp package on top of the system ggml, whose backends are split out. sudo pacman -Syu ggml llama-cpp ggml-vulkan — name ggml explicitly (it conflicts with the old ggml-cpu without replacing it, and the removal prompt defaults to N), and ggml-vulkan explicitly (only an optdepend — without it llama-server silently runs on the CPU backend). Then restart, and llama-server --list-devices must show Vulkan0. whisper-cpp moves to the same ggml (CPU backends are inside the base package) — restart its unit too.

VRAM safety (the golden rules)

The amdgpu driver does not OOM cleanly — an overcommitted model spills into GTT (system RAM), starves the desktop and freezes the PC (reboot). Hard-won rules:

Keep every preset's benched VRAM free ≥ 2.5 GB (idle desktop uses ~1.3 of 17.2).
File size > VRAM is fine for MoE — n-cpu-moe N keeps the first N layers' experts in system RAM. It is the main tuning knob: higher = safer/slower, lower = faster.
Bench with a clean baseline (close the browser): a dirty baseline both skews the margins and costs ~3–4 t/s of decode (GPU compute contention — measured 2026-07-08).
Bench after every change; config-edit without restart = numbers from the old config.

Model roster (reference numbers = clean night sweeps, 2026-07-16)

model id	decode t/s	ctx	extras	role
`gemma-4-E4B-it-UD-Q8_K_XL`	57.3	96k	vision, MTP, prefill 185 t/s	fast small generalist, long docs, images (ctx 96k ✓ 08-08, n_ctx_train 131072)
`Qwen3.6-35B-A3B-Thinking`	38.8	24k	MTP, vision, reasoning (budget 8k = "medium")	default model in opencode/pi/aichat since 09-21; hard problems, slow-but-smart answers
`gpt-oss-20b`	38.4	64k	reasoning, tools	fast reasoning + tool use · fast long-context (37 t/s vs Coder-Next 16; 131k ruled out)
`gpt-oss-20b-low`	37.2	64k	reasoning LOW, TTFT 0.8 s	same model, snappy answers (no long preamble)
`Qwen3.6-35B-A3B-MTP-UD-IQ3_XXS`	35.9	24k	MTP, vision, thinking OFF	daily driver — compact instant answers
`gemma-4-26B-A4B-it-UD-IQ4_XS`	33.9	24k	vision, MTP	quality generalist + best vision
`Qwen3.5-9B-UD-Q6_K_XL`	32.1	32k	vision, MTP, reasoning	small Qwen, quick tasks
`Qwen3-Coder-30B-Instruct-UD-Q3_K_XL`	~30 clean / 27.0 evening @moe12	32k	—	main agent coder — won the 2026-08-06 quant race; moe12 verified evening 08-07 + clean 08-08
`GLM-4.7-Flash-UD-Q4_K_XL`	21.5 @moe22	24k	reasoning	quality coder (opencode subagents); KV fix + moe22 ✓ verified clean 08-08
`GLM-4.7-Flash-UD-Q3_K_XL`	26.0 @moe12 (2.0 daytime baseline; ~28 clean)	24k	reasoning	fast coder — re-added 09-21 because Q4's 21.5 dragged in agent loops; 3-bit, evening-safe at moe12 (13.4 used / 3.8 free @2.0); Q4 stays for hard reviews
`Qwen3-Coder-Next-UD-IQ3_XXS`	16.0	128k	80B-A3B	long-session coder (128k ctx)
`Qwen3.8-27B-UD-IQ3_XXS`	36–37 on b11429 (10-22 ×2; was 15.5 on b11146, 10.9 base)	24k	vision, MTP, hybrid reasoner @ xhigh	⚠ clean-desktop specialty (kept 09-10 over the ≥22 gate — the speed half is now passed 1.7× over, since llama.cpp 0.6.0 halved the per-step MTP cost; role re-rank pending the user) — dense-27B quality; prefill still GDN-bound (~20–35). 2.7 GB free @1.3 baseline @24k (✓ 10-22 on b11429, `-n 512`, acceptance 0.89, len 2.78). The felt slowness is the xhigh thinking preamble (17.5 s before the first answer token on a trivial prompt; `reasoning_effort=low` measured 6.45 s at the same decode on 10-02 — siblings benched then removed by decision, the kwarg is one line if ever wanted); never on a busy evening desktop (~1 GB free = GTT freeze)
`Qwen3-Embedding-0.6B`	33 (CPU)	8k	CPU-only, `/v1/embeddings`	RAG/search embedder (not a chat model)
`Ornith-1.5-35B-A3B-IQ3_XXS`	32.1 @moe16	32k (10-22, unbenched)	MTP, vision, thinking	agentic coder (added 09-21) — Qwen3.5-MoE derivative claiming +15 Terminal-Bench / +5.6 SWE-bench over Qwen3.6-35B; the only coder with MTP; 13.8 used / 3.3 free @2.0

Expected run-to-run spread: MTP models swing ±15% with draft acceptance rate (content- dependent); CPU-heavy presets (Coder-Next, embedder) dip under daytime CPU contention. Treat clean night runs as the reference; don't retune on daytime deltas.

Which model, when

Agent coding loops (edit/test cycles): Qwen3-Coder-30B — best speed/quality balance at 32k.
Hard code, reviews, tricky bugs: GLM-4.7-Flash — strongest 30B-class coder, slightly slower.
GLM at agent-loop speed: GLM-4.7-Flash-UD-Q3_K_XL — same model, 3-bit, ~28 t/s; pick it when the Q4's 21 t/s drags, keep Q4 for the review that has to be right.
Agentic coding with reasoning (⚗ new, judge over the next weeks): Ornith-1.5-35B-A3B — Qwen3.5-MoE derivative, 32 t/s with MTP and a thinking preamble (~4 s TTFT), vision. If its launch numbers hold in practice it displaces GLM/Coder-30B for agent loops.
Marathon sessions / huge conversation history: Qwen3-Coder-Next — 128k ctx at only 6 GB VRAM (hybrid attention). Caveat: ~33 t/s prefill means it's for growing sessions (cache-reuse makes turns incremental), not for cold-dumping 100k tokens.
Everyday questions: Qwen3.6-35B-A3B — no reasoning preamble, 36 t/s.
Hard reasoning: 35B-Thinking (quality) or gpt-oss-20b (speed + tool use). Their multi-second TTFT is the reasoning phase streaming first — not a slow load. For quick interactive gpt-oss answers use gpt-oss-20b-low (reasoning effort low).
Best single-model quality (quiet desktop only): Qwen3.8-27B — dense 27B hybrid reasoner + vision at 37 t/s since llama.cpp 0.6.0 (was ~15 on 0.5.0; prefill still GDN-bound on Vulkan), but a busy evening desktop pushes it into GTT-freeze territory — check rocm-smi first.
Images: gemma-4-26B for quality, gemma-4-E4B or Qwen3.5-9B for speed, Qwen3.6-35B when you want the daily driver to see the screenshot.
Long one-shot documents: gemma-4-E4B — 185 t/s prefill eats 64k in ~6 min (Coder-Next would take ~30 min to prefill the same).
RAG embeddings: point Open WebUI etc. at Qwen3-Embedding-0.6B. Runs on CPU by design — 0 VRAM, can never contribute to an overcommit.

Speech-to-text (separate service)

STT deliberately does not live in this router — it is a CPU-only whisper.cpp service on port 11345, documented in ../whisper/README.md. Two reasons: models-max 1 would make every transcription LRU-evict the resident chat model, and llama.cpp decodes audio via miniaudio (wav/mp3/flac only) while browsers record opus.

The router does answer POST /v1/audio/transcriptions — it rewrites the request into a chat completion — so gemma-4-E4B-it-UD-Q8_K_XL (audio-capable via its mmproj) transcribes uploaded wav/mp3/flac with no config change. Keep that as the fallback; mind the eviction.

Text-to-speech (separate service)

TTS is not servable from this router at all — unlike STT, there is no fallback. llama-server has --model-vocoder / --tts-use-guide-tokens, but exposes no /v1/audio/speech route: the router 404s on it exactly like an undefined path, and grep -rn "audio/speech" over the b10216 tree returns nothing — the only audio route registered is /v1/audio/transcriptions (STT). Worse, --model-vocoder is accepted by the server and then silently ignored: it is registered in the arg table and documented, but no server code path consumes it. --help listing it is not evidence it works. llama-tts is a CLI. (All verified 2026-08-09.)

So TTS is a CPU-only Kokoro-82M service on port 11347, documented in ../tts/README.md (repo-side only — not installed on fl yet). The models-max 1 argument applies even harder there than for STT: TTS fires on every response, so a resident preset would evict the chat model every single turn.

Client wiring

The same catalog is mirrored in every client — when adding/removing a preset, update all:

common/.config/opencode/opencode.json (both providers + agent model overrides)
common/.pi/agent/models.json (both providers; contextWindow = server ctx-size)
common/.config/aichat/config.yaml (both clients)
wm/.config/aichat/config.yaml — full-file overlay for the off-LAN host (default client duskadiy); every aichat change must be mirrored here too (missed for GLM Q3 + Ornith until 09-21). With the lw/ overlays that makes seven client files.

Rule: client model id = config.ini section name, client context ≤ server ctx-size. The embedder is deliberately absent from chat clients. 2026-08-06: the coder id changed …-IQ4_XS → …-UD-Q3_K_XL in all six client files (common + lw overlays) after the quant race; OWUI picks the new id up automatically from the router, but chats/presets saved against the old id need re-picking. 2026-08-16: Qwen3.8-27B-UD-IQ3_XXS (⚗ candidate) added to all six client files. 2026-09-10: kept as ⚠ clean-desktop specialty after the gate bench finally ran on the v3 gguf (14.5 t/s MTP / 10.9 base, 2.8 GB free @1.5); server ctx restored to 24576 = client ctx, no client edits. 2026-09-21: GLM-4.7-Flash-UD-Q3_K_XL (⚡ fast coder) added to all six client files next to the Q4. 2026-10-22: Qwen3.8-27B quant race opened (ISTA GSQ-RCO IQ3_XXS-mtp + DFlash2 sibling) — bench-only, NO client edits until a candidate wins; see the watch list. Same night: fl found on llama.cpp 0.6.0 (b11429) and the Qwen3.8 incumbent re-measured 37.1 t/s (15.5 on b11146) — every MTP/hybrid preset on the roster needs a re-sweep on this build; ledger numbers above it are from 0.5.0 unless dated 10-22. 2026-09-21: Ornith-1.5-35B-A3B-IQ3_XXS (agentic coder, thinking + vision) added to all six client files after passing its gate (32.1 t/s @moe16). 2026-09-21: default model → Qwen3.6-35B-A3B-Thinking everywhere (opencode global + plan/build/auto, pi defaultModel + thinking level medium, aichat common/lw/wm) — gemma-4-26B refused routine sysadmin work. Thinking preset gets reasoning-budget 8192. wm aichat caught up (GLM Q3, Ornith). 2026-10-22: Ornith in pi: Compacting in OVERFLOW recovery right after a truncated reply is pi's truncated-reply recovery, not a full window — pi clamps each request's max_tokens to contextWindow − estimate − 4096, which on 24k/4096 starts exactly at the compaction threshold, so the first request after a compaction (estimate gains the system prompt + tool schemas) can leave a thinking reply ~1024 answer tokens → length → compact + retry. compaction-watch.ts now names the case with numbers; common/.pi/CLAUDE.md > Compaction tuning has the arithmetic. Applied the same day, UNBENCHED: Ornith ctx-size 24576 → 32768 + the seven client files (pi/opencode/aichat, common+lw+wm; predicted ≈ 0.1 GB of KV on this hybrid arch) — restart llama.service, then ./bench.py -m Ornith-1.5-35B-A3B-IQ3_XXS; gate and the revert path are in the preset comment. 2026-10-12: Ornith in pi: "Response was truncated before completion." = finish_reason: length — the <think> block counts against pi's 4096 max_tokens and the preset has no reasoning-budget. Fixed client-side: pi sends thinking_budget_tokens per thinking level on Ornith (compat.thinkingTokenBudgetField + settings.json > thinkingBudgets, common + lw). The preset deliberately keeps NO CLI reasoning-budget — it would override the per-request field (see the preset comment).

Tuning cheat-sheet

n-cpu-moe (MoE only): experts→CPU. The speed/VRAM dial. Measured curve is gentle — tune in steps of 2–4 layers, re-bench, keep free ≥2.5 GB.
ctx-size: KV cost varies wildly by arch — Coder-Next 64k→128k cost 0.5 GB (hybrid attn); dense Qwens pay ~1 GB per 16k. Raise only after a bench shows the headroom.
Quants: dense = bandwidth-bound, quant size sets speed directly (9B: Q8→Q6 = +87% with MTP). MoE = buy quality with a bigger quant, pay in CPU offload (both coders run 4-bit now; 3-bit was only ~2 t/s faster). Coders want ≥4-bit; chat tolerates UD 3-bit.
MTP / spec decode (spec-type draft-mtp): ~1.5–2× decode. Qwen3.5/3.6 embed the head in the MTP-repo GGUF (same filename as plain repo — size is the tell); gemma-4 uses a separate mtp-*.gguf draft file; Qwen3.8 ships the head in the main-repo GGUFs for ≥ Q2_K_XL (≤ IQ2_S are stripped — use the repo's MTP/mtp-*.gguf draft instead). On the hybrid 27B it is only +33% (10.9→14.5) and costs 1.5 GB: every speculative step re-runs the Gated-DeltaNet layers. No MTP possible yet for: GLM-4.7-Flash (conversion drops the head — expected ~1.5× when llama.cpp lands it), Qwen3-Coder-Next (Qwen3-Next head exists upstream, no GGUF ships it), gpt-oss and Qwen3-Coder-30B (no head exists). Recheck releases occasionally.
KV cache: q8_0 K / q4_0 V everywhere except gpt-oss (attention-sink issues with quantized KV — keep defaults there).

Benchmarking

./bench.py                     # full sweep (add -x Qwen3-Embedding-0.6B — it can't chat)
./bench.py -m id1,id2          # just the changed presets
./bench.py -n 512 --ctx 16000  # longer gen + long-context decode column

⚠ Benches need the server to themselves: with models-max 1, a request for another model arriving mid-load LRU force-kills the loading instance → fake "failed to load" rows (and a dirty baseline). Close chat clients / OWUI before sweeping.

Safety-first: force-unloads resident models via the router API (POST /models/unload), verifies a clean card with rocm-smi before every load, records a ⚠ row and skips generation when a model lands too tight (free < 1.5 GB or GTT ballooning). Results: bench-results.md (latest) + bench-history.md (append-only; diff runs there).

Reading TTFT: bench.py stamps it on the first non-empty content delta, so for thinking models TTFT = prompt eval plus the whole reasoning preamble, and 0.00 means the answer never started within -n tokens (thinking ate the budget). It is the column that compares reasoning-effort siblings; for stable MTP decode numbers use -n 512.

Watch list

GLM MTP in llama.cpp — the single biggest pending win (primary coder ~22 → ~30+).
Qwen3.8-27B quant race (opened 2026-10-22, bench-only, not client-wired) — two candidate sections in config.ini after the incumbent: Qwen3.8-27B-GSQ-RCO-IQ3_XXS (ISTA-DASLab GSQ-RCO file, 10.44 GB with MTP head = −0.49 GB vs unsloth v3) and its -dflash text-only sibling (z-lab/Qwen3.8-27B-DFlash2-GGUF Q4_K_M drafter, spec-draft-n-max 3). Expected: +4 % decode and +0.5 GB free from the file; +5–15 % over MTP from DFlash2 on RDNA3/Vulkan (not the paper's 1.8×). Quality is unmeasured against our file — the user's own prompts decide. First run 10-22 03:14: both candidates HTTP 400 = fl's config.ini not pulled yet (router: "model not found"); the incumbent in the same run read 37.1 t/s on b11429 (see its row). Re-run 03:25, one session: incumbent 36.2 / 2.7 free, GSQ 37.5 / 3.2 free (+3.6 %, +0.5 GB — speed+VRAM passed, quality call open), DFlash2 15.4 / 1.9 free → dropped (tombstoned in config.ini). Ruled out by the same research: ByteShape IQ4_XS (13.1 GB), ISTA IQ3_S (12.1 GB with MTP, under the floor), the adaptive KV-cache-streaming fork (CUDA-only, PCIe 5.0). Still open: #27408 (spec + mmproj → 500), #28158 (Vulkan OOB draft token). Research + verdicts + download lines: the config block.
llama.cpp 0.7.0 (b11669) — HOLD fl at 0.6.0: #30688, Vulkan draft-mtp decode 4× / prefill 6× slower (target compute buffer lands in GTT = this box's freeze precursor). Every MTP preset on the roster is exposed. When pacman -Si llama-cpp on fl shows 0.7.0, IgnorePkg it until fixed. The 0.5.0 → 0.6.0 jump itself (Qwen3.8 15.5 → 37) is llama-side, best explained by #29856 (hybrid graph reserve; the old 10.9 base was probably GTT-starved, not kernel-bound — see the preset block). ggml 0.26.0 (next -Syu): #30039 reports a −45 % prefill regression on a GDN hybrid — bench Qwen3.8 before and after; downgrade ggml/ggml-vulkan from the pacman cache if prefill tanks.
Coder-Next 256k: if the load log's n_ctx_train says 262144, ctx can likely double again for ~1 GB (KV measured nearly flat).

vLLM on this box (bench rig, docker/ROCm, never resident together with the router): see ../vllm/README.md.

README.md Unescape Escape

Local LLM server — setup & usage