llama.cpp in router mode on the Wingman work desktop. This is the NVIDIA counterpart of fl (fl/.config/llamacpp/, RX 7600 XT 16 GB) and lw (lw/…, CPU laptop). Same recipe as fl — MoE experts in RAM, quantized KV, MTP — on a smaller, older card.

Box: GTX 1080 Ti 11 GB (Pascal, compute 6.1: int8 MMQ path, no tensor cores) on the legacy nvidia-580xx driver · i7-8700 (6 cores / 12 threads) · 31 GiB DDR4 (~35 GB/s = the ceiling for experts kept in RAM).
Backend: upstream prebuilt CUDA 12.8 tarball (~/.local/opt/llama.cpp-cuda/current, build b11227) through the ~/.local/bin/llama-server-cuda wrapper. CUDA 13.x cannot target Pascal, so the Arch ggml-cuda / llama.cpp-cuda-git packages are useless on this box; the prebuilt carries Pascal as PTX and the driver JITs it (first start of a new build ≈ 50 s, then cached). A Vulkan prebuilt sits beside it (llama-server-vulkan) as the no-CUDA fallback — same build, ~half the prompt speed, same decode.
Endpoint: http://127.0.0.1:11343/v1 on the box, http://192.168.50.24:11343/v1 from the LAN (wired IP, DHCP — may change). Bound to 0.0.0.0 and keyless: anyone on the office network (and ZeroTier) can use it. IPv4 only — use 127.0.0.1, not localhost (which also resolves to ::1, where nothing listens). The chat endpoint is POST-only: a browser GET gets 404.
Files here (~/.config/llamacpp/, per-file stow links from the repo's wm/.config/llamacpp/): config.ini (presets — section names ARE the API model ids), llama.service (user unit), bench.py (nvidia-smi benchmark), bench-results.md / bench-history.md (ledgers, generated next to the script, not tracked). The GGUFs live in ~/Documents/models/ (llama-server -hf drops downloads into ~/.cache/huggingface/hub/ — move them out).

How it runs

A user systemd unit, llama.service (this dir; linked into ~/.config/systemd/user/ and enabled, starts with the login session — loginctl enable-linger coja makes it start at boot without a login). Its ExecStart is the command the old abbr ran:

nice -n 19 llama-server-cuda --host 0.0.0.0 --port 11343 --cors-origins localhost \
  --models-max 1 --models-preset ~/.config/llamacpp/config.ini

The fish abbr llamaserver now means systemctl --user restart llama; and journalctl --user -fu llama. Install once after stowing (a symlinked unit can't be linked directly, hence readlink -f):

systemctl --user link (readlink -f ~/.config/llamacpp/llama.service); systemctl --user enable --now llama

Router mode: each requested model loads on demand in a child process; /v1/models lists the presets.
--models-max 1: one model resident; a new request LRU-evicts the previous. Two presets never fit together in the ~6.5–7.5 GB the desktop leaves free. sleep-idle-seconds per preset frees idle models either way.
nice -n 19: the MoE presets run their expert matmuls on all 6 cores — niceness keeps the desktop responsive; the GPU part is unaffected.
--cors-origins localhost: same-machine profile (llama.cpp PR 25655) — blocks browser pages from calling the keyless API. Non-browser clients unaffected.
Presets are read once at startup: edit config.ini → systemctl --user restart llama → ./bench.py -m <ids>.
Logs: journalctl --user -u llama -f (each model child logs under the router).
"couldn't bind HTTP server socket … 11343" = a router is already running (the service, another shell, or a bench run) — systemctl --user status llama, pgrep -af llama-server.

VRAM (the golden rule)

11,264 MiB total, but Hyprland + Xorg + electron apps hold 3.5–4.5 GB, so a preset gets ~6.5–7.5 GB and must be sized for the low end. Unlike fl's amdgpu, an NVIDIA overcommit does not freeze the PC — cudaMalloc fails and the load errors out cleanly. So the guard is "loads reliably on a busy desktop": keep every benched VRAM free ≥ ~1 GB (bench.py flags < 0.8). If a preset is tight: raise n-cpu-moe (MoE), lower ctx-size, or use a smaller quant.

Dense models are the trap here: anything that doesn't fit entirely (> ~6 GB file) streams the leftover layers from DDR4 and collapses (14B Q3 = 7.6 t/s, 8B Q8 = 3.4, 24B IQ3 = 1.8 — see the tombstones in config.ini). MoE with the experts in RAM is 2–5× faster at the same size.

Model roster (router benches 2026-09-28 → 10-19, `bench.py` through the service, desktop baseline 3.3–4.6 GB, gen 256)

Three conditions appear below: quiet = ~3.3 GB desktop, nothing else on the CPU · busy = 4.2–4.6 GB desktop · sim = the 2026-10-19 sweep with the AirSim/PX4 simulation stack running (details after the table).

model id	decode t/s	ctx	VRAM free	extras	role
`gemma-4-E4B-it-UD-Q4_K_XL`	85–101 (MTP 64–84%; 91 sim)	32k	3.6–4.8 GB	vision (projector on CPU), MTP draft, thinking OFF	⭐ fastest + instant tier — quick generalist, images, long documents; pi's default local model
`Mellum2.1-12B-A2.5B-Thinking-IQ4_XS`	63 @moe4 / 87 @moe0 quiet · 42 sim	16k	1.6–2.3 GB	code MoE 2.5B active, thinking ON, prefill 1.1–1.5k t/s	⭐ fast reasoning coder (added 10-12)
`Qwen3.5-9B-MTP`	51–61 (MTP 90%)	8k	0.4–1.8 GB	vision, MTP	⚠ quiet-desktop preferred (loaded at 1.0 GB free on a ~4.5 GB desktop 10-19) — the daily driver at +75%
`Qwen3.5-9B-Thinking`	34–38	16k	1.0–2.5 GB	vision, reasoning budget 8k	hard problems (TTFT ≈ think block)
`Qwen3.5-9B-Q4_K_M`	31–33 (one unexplained 24 on 10-12)	16k	1.0–2.8 GB	vision, thinking OFF	⭐ daily driver — best quality that always fits
`gpt-oss-20b` / `-low`	21 quiet · 17 busy · 10–12 sim	32k	2.1–3.7 GB	MoE @moe18, reasoning + tools	reasoning and tool use; `-low` cuts the 8.8 s think TTFT to 1.8 s
`Ornith-1.5-35B-A3B-instant`	14–31 (MTP 59–67%; 15 sim)	16k	1.4–3.5 GB	MoE @moe36, MTP, thinking OFF	⭐ agentic coder for agent loops; 31.5 and 15.1 t/s on two quiet-desktop runs 16 min apart (prefill 63 → 9) — CPU contention, the server runs at nice 19
`Ornith-1.5-35B-A3B-IQ4_XS`	10–26 (MTP 47–61%; 15 sim)	16k	1.8–3.3 GB	same file, thinking ON	coder with reasoning; TTFT ~20 s is the think block
`Ornith-1.5-35B-A3B-long`	10 busy · 16.9 @35k prompt (MTP 59–65%)	128k	1.8 GB	same file, @moe40, thinking OFF, prefill ~250 t/s	long-context tier (added 10-28) — 35k prompt = 2.4 min to prefill, 100k ≈ 7 min

Read across: everything that fits entirely on the GPU runs 31–101 t/s; everything that keeps experts in RAM runs 10–31 and tracks the CPU's idle share. The bench guard (--min-free-gb 0.8) skipped four presets in the first sweep; they were re-tuned (coder moe 28→34, gpt-oss moe 14→18, 9B draft split into its own alias) and re-benched green. Expected spread: MTP rows swing with draft acceptance (content-dependent); every MoE row dips when the desktop is busy (the experts share the 6 cores).

The sim condition (2026-10-19 12:55 sweep). Unreal Blocks (≈ 3 cores at nice 5, 0.56 GB VRAM, GPU 41% busy) plus the PX4/planner nodes at realtime priority had been running since 12:08; load average ~9 on the 6-core box. Every CPU-expert preset sat at its floor (Ornith 15, gpt-oss 10–12, Mellum 42) and the GPU-only presets lost 5–15% to the shared GPU. Nothing in the presets caused it and nothing in them fixes it: the service runs at nice 19 and yields by design. Judge local-model speed with the sim stopped. Two things the run did establish: all nine presets loaded with ≥ 1.0 GB free at a ~4–4.5 GB desktop (the busy-desktop sizing holds), and the run's "baseline 7.5 GB" header was pi's resident gemma, not the desktop — bench.py now drains before it records the baseline.

Which model, when

Quick questions, images, long one-shot documents: gemma-4-E4B — 85 t/s, 32k, vision.
Best always-available quality: Qwen3.5-9B-Q4_K_M (33 t/s). On a quiet desktop pick Qwen3.5-9B-MTP instead — same model, 57 t/s; if it fails to load, the desktop is holding too much VRAM (electron apps), fall back to the plain preset.
Autocomplete-class / cheapest agent steps: gemma-4-E4B again — LFM2.5 was faster (119 t/s) but its output was unusable in practice and it was retired 2026-10-12.
Reasoning coder at speed: Mellum2.1-12B-A2.5B-Thinking — 63 t/s (87 on a quiet desktop) with a think block and >1k t/s prefill; the first stop for a bug that needs thought.
Agent coding loops (opencode/pi edit-test cycles): Ornith-1.5-35B-A3B-instant — 14–31 t/s depending on what else is using the CPU (nice 19), no think preamble. Switch to …-IQ4_XS (thinking) for the bug that needs reasoning.
Reasoning + tool use at long context: gpt-oss-20b (32k); -low for interactive use.
Really long context (to 128k): Ornith-1.5-35B-A3B-long — the 35B with every expert in RAM so the KV fits; prefill ~250 t/s (100k ≈ 7 min). fl's Qwen3-Coder-Next 80B needs ≥ 28 GB of RAM for its experts and does not fit next to this desktop.
No separate fast coder: the 2-bit Qwen3-Coder was deleted 2026-10-12 (dominated by Ornith-instant); a K-quant Qwen3-Coder stays listed in config.ini candidates if a second coder is ever wanted.
Anything bigger/denser (14B+ dense, 27B, 80B-A3B) does not belong on this card — see the candidates list in config.ini for what does.

Client wiring

These clients run on this box (other machines use http://192.168.50.24:11343/v1). Rule: client model id = config.ini section name, client context ≤ server ctx-size.

tool	overlay file	provider / client id
aichat	`wm/.config/aichat/config.yaml`	`wm:` (e.g. `wm:Ornith-1.5-35B-A3B-instant`) — default model still `duskadiy:`; `aichat -m wm:…` or flip `model:`
opencode	`wm/.config/opencode/opencode.json`	`wmcpp/` (e.g. `wmcpp/Ornith-1.5-35B-A3B-instant`); defaults + agent models point at `duskadiy/` (the common file's `llama.cpp/` LAN provider is unreachable here)
pi	`wm/.pi/agent/models.json` + `settings.json`	`wmcpp`; default model `wmcpp/gemma-4-E4B-it-UD-Q4_K_XL` since 10-12 (was `duskadiy/Ornith…`)

Two ways a model "doesn't respond" in pi (both seen 2026-10-12):

The router was started before the preset existed — it reads config.ini once, so pi's request fails as an unknown model. systemctl --user restart llama after every preset change.
A small coder (Mellum 2.1, and AesCode-8B before its removal) run with pi's full tool set from ~ keeps calling read/bash to explore the home directory and never finishes (router log: the same conversation re-served with a growing context, 2.3k → 5.4k tokens). Run pi inside a project directory, or pi --no-tools for plain questions; the same prompt from an empty directory finishes in four rounds.

When adding/removing a preset, update all three wm overlays (aichat, opencode, pi). They are full-file copies of the common configs plus the wm/wmcpp provider — when a common client config changes, mirror the change here (same rule as lw).

Tuning cheat-sheet (Pascal + CPU experts)

n-cpu-moe N (MoE only): experts of the first N layers stay in RAM. The VRAM/speed dial; each 4-bit expert layer on the GPU ≈ 0.4 GB (35B-A3B). Measured: gpt-oss @moe12 = 29.8 vs @moe24 = 18.6 t/s; Coder-30B @moe24 = 29.9 vs @moe48 = 20.2. Tune in steps of 2–4, re-bench. Except Ornith: moe30 vs moe36 on the same quiet desktop = 32.1 vs 31.5 t/s (10-12) — its experts on the GPU buy nothing; give the VRAM to the desktop instead.
Quant type for CPU experts: I-quants (IQ4_XS/IQ3_XXS) dequantize slowly on CPU and 4-bit experts stream 2× the bytes of 2-bit → Ornith IQ4_XS with 80% of experts in RAM = 11–15 t/s base. Prefer K-quants (Q4_K_M / UD-Q4_K_XL / UD-Q3_K_XL) and the smallest quant the quality allows when the experts live in RAM.
MTP / spec decode (spec-type = draft-mtp) works here: Qwen3.5-9B 32.6 → 57.3 t/s at 90% acceptance; Ornith 11–15 → 14–21 at 60%; gemma-4-E4B 84.7 with its draft. Cost: a flat ~0.75–1.1 GB VRAM per model — which is why the 9B draft lives in its own quiet-desktop alias. Thinking mode drafts worse (Ornith 47–53%, 10 t/s). Qwen3.5/3.6 heads live in the -MTP repo files; gemma-4 needs a separate mtp-*.gguf draft; gpt-oss and Qwen3-Coder-30B have no head.
threads = 6 = physical cores. Never 12: decode collapses 14 → 1.3 t/s (HT contention).
KV cache: q8_0 K and V everywhere except gpt-oss (attention sinks — defaults). flash-attn on everywhere (required for quantized V). Dense Qwens pay ~0.6–0.8 GB per 16k; MoE 48-layer Qwen3-Coder ~1.6 GB per 16k (why it stays at 16k).
First value after a download: llama-fit-params -m <gguf> -c <ctx> -fa on prints the split that fits the VRAM free right now; take it, then add margin (moe +2, or ngl −4).
CPU governor is powersave (intel_pstate still boosts); performance might buy a little on the CPU-expert presets — unmeasured.
Vision: keep the projector on the CPU (no-mmproj-offload = true) — ~0.9 GB VRAM back.

Benchmarking

Run on this box against the running llamaserver (it evicts whatever is resident — close chat clients first):

./bench.py                     # full sweep of everything the router lists
./bench.py -m id1,id2          # just the changed presets
./bench.py -n 512 --ctx 12000  # longer gen + a long-context decode column

Safety-first like fl's: force-unloads resident models via the router API, waits for the card to drain (--drain-gb 5.5 = the desktop's own usage plus slack — raise it if your desktop alone sits higher), records a ⚠ row instead of generating when a model leaves < 0.8 GB free. The MTP acceptance % lands in the spec column. Ledgers: bench-results.md (latest) + bench-history.md (append-only) next to the script.

Reading the columns (changed 2026-10-19):

baseline used is measured after draining anything of ours; the header adds "(was X GB with a resident model, unloaded first)" when it had to. A high baseline now really is the desktop.
TTFT = time to the first answer token — for a thinking model that is the think block. all-think = the whole gen budget (256 tokens) was reasoning and no answer arrived; decode t/s on that row is still valid (server timings count every token). -n 512 or the -low/-instant alias gets an answer.
prefill t/s at the default ~30-token prompt is fixed-overhead noise (Mellum showed 15 and 100 on two runs; a 1.7k-token probe gave 1,160). Use --ctx for a real prefill number.
fl's and lw's bench.py still carry the content-only TTFT (0.00 on an all-think row); port the two changes when their ledgers next matter.

Watch list

Free VRAM is the whole game: the desktop's 4.2–4.6 GB (Hyprland ~0.4, Xorg, Ferdium/electron ~0.6+) costs the 9B its MTP and every MoE two or three expert layers. Closing electron apps before a session is worth more than any preset knob.
Qwen3.5-9B ctx 32768 for the plain preset (KV is cheap on the hybrid arch) — bench first.
Qwen3.6-35B-A3B-MTP UD-IQ3_XXS (14 GB) — the one "bigger" model worth a test: 28% fewer bytes per token than Ornith's file and a draft that accepts well on non-thinking Qwen; likely ≥ Ornith's 31 t/s quiet.
2026-10-12 candidate round: AesCode-8B and Mellum 2.1 kept; FrogNano-4B (87.9 t/s with draft, vision) retired — gemma-4-E4B and LFM2.5 cover that slot faster.
2026-10-19: AesCode-8B removed (preset and the aichat/opencode/pi entries); tombstone in config.ini.
gemma sleep-idle-seconds = 30 vs pi (journal 2026-10-19 12:44–12:56): a pi session slept and woke its default model 3× in 12 min, each wake a full reload ≈ 6 s before the first token (cold load 17 s). Resident, gemma holds 3.7 GB and still leaves ~3 GB beside a 4.5 GB desktop; 120–300 s (fl uses 300 on its coders and 30 on this same gemma) would remove the stall. Not changed — the VRAM-vs-latency trade is the user's call.
The sim stack is the busy floor (10-19): with Blocks/PX4 running, CPU-expert presets drop 40–55% and GPU-only ones 5–15%. Unmeasured: threads 3–4 on the CPU-expert presets while the sim holds 3 cores — 6 threads then share physical cores with it, and HT sharing is exactly what collapsed decode at 12.
gemma logs Gemma4Assistant requires ctx_other … (this warning is normal during memory fitting) and failed to measure the memory of the extra model at every load, including each wake: the auto-fitter probing the MTP draft. Harmless (ngl 99, everything on the GPU); noted in the preset so nobody chases it.
Dead files were deleted 2026-10-12 (66 GB): ~/Documents/models now holds exactly the nine presets' files (46 GB).

README.md Unescape Escape

Local LLM server — wm (GTX 1080 Ti), setup & usage