# llama.cpp model config — WM (Wingman work desktop) · NVIDIA GTX 1080 Ti 11 GB (Pascal, legacy) # backend = upstream prebuilt CUDA 12.8 build (~/.local/opt/llama.cpp-cuda, launched via the # `llama-server-cuda` wrapper in ~/.local/bin) — NOT the Arch package, NOT a local build. # ═════════════════════════════════════════════════════════════════════════════ # HARDWARE: i7-8700 (6 cores / 12 threads, AVX2, no AVX-512) · 31 GiB DDR4 dual-channel # (~35 GB/s effective = the CPU-expert ceiling) · GTX 1080 Ti 11,264 MiB, compute # capability 6.1 (Pascal): no tensor cores, no usable fp16 → CUDA runs its int8 MMQ # path, which is what llama.cpp is best at on this card. Driver nvidia-580xx (AUR): # the LAST branch for Pascal; CUDA 13.x toolkits can no longer target this GPU, so # the repo `ggml-cuda`/`llama.cpp-cuda-git` builds are USELESS here — the prebuilt # CUDA 12.8 tarball carries Pascal as PTX and the driver JITs it (first run of a new # build ≈ 50 s, then cached in ~/.nv/ComputeCache; the wrapper raises the cache cap). # # This is the NVIDIA analog of fl/.config/llamacpp/config.ini (RX 7600 XT 16 GB). Same # recipe (MoE experts on CPU, quantized KV, MTP), different limits: # # • VRAM BUDGET: 11.2 GB card, but the desktop (Hyprland + Xorg + electron apps) holds # 3.5–4.5 GB → ~6.5–7.5 GB usable. Every preset below is sized for ~6.5 free. # ⚠ Unlike fl's amdgpu, an NVIDIA overcommit does NOT freeze the PC — cudaMalloc fails and # the load errors out cleanly ("failed to allocate buffer for kv cache"). The guard here is # therefore "loads reliably on a busy desktop", target ≥ ~1 GB free after load (bench.py # records it). Still keep `--models-max 1`: two presets never fit together. # • MoE IS THE BOX. Dense models that don't fit VRAM fall off a cliff (every layer left on # the CPU streams from DDR4): Qwen3-14B Q3 = 7.6 t/s @ngl34, Qwen3-8B Q8 = 3.4, Ministral-14B # = 3.8, Devstral-24B = 1.8 (measured 2026-09-28, tombstones below). A 20–35B MoE with its # experts in RAM decodes at 15–30 t/s because only ~3B params are read per token. # • `n-cpu-moe N` = experts of the first N layers stay in RAM. LOWER = more VRAM, faster. # Measured 2026-09-28 (llama-bench, ~9 GB free at the time): gpt-oss-20b @moe12 = 29.8 t/s, # @moe24 (all) = 18.6; Qwen3-Coder-30B IQ2_XXS @moe24 = 29.9, @moe48 (all) = 20.2. # • QUANT TYPE MATTERS FOR CPU EXPERTS: I-quants (IQ4_XS/IQ3_XXS) dequantize slowly on CPU and # 4-bit experts stream twice the bytes of 2-bit ones — Ornith IQ4_XS with 80% of experts in # RAM = 11–15 t/s base. When choosing a file for the experts-in-RAM pattern prefer K-quants # (Q4_K_M/UD-Q4_K_XL/UD-Q3_K_XL) and the smallest quant the quality allows. # • `threads = 6` = PHYSICAL cores. NEVER 12: measured 2026-09-28, tg collapses 14.3 → 1.3 t/s # (HT contention on the expert matmuls). The default (6) is right; the key is pinned anyway. # • MTP / speculative decode WORKS here (`spec-type = draft-mtp`, Qwen3.5-family heads): # Ornith @moe34: 11–15 → 18.5–19.0 t/s at 61% acceptance for ~1.1 GB extra VRAM (09-28). # Unlike lw (2 cores) the 6 cores have compute to spare for the batched verify. # • KV cache: `cache-type-k/v = q8_0` everywhere except gpt-oss (attention sinks — keep the # defaults there, same as fl). flash-attn = on everywhere (required for quantized V). # • CPU governor is `powersave` (intel_pstate still boosts). The service runs at Nice=19 so the # 6-core expert matmuls never lag the desktop; GPU work is unaffected by niceness. THE FLIP SIDE # (full sweep 2026-10-19 with the AirSim/PX4 sim stack up: Unreal Blocks ≈ 3 cores at nice 5, # root nodes at realtime priority, load avg ~9, GPU 41% busy): every CPU-expert preset sat at # its floor — Ornith 15 t/s (31 quiet), gpt-oss 10–12 (21), Mellum 42 (69) — and the GPU-only # presets lost 5–15% to the shared GPU (9B-MTP 51 vs 57–61). Nothing to tune: the server yields # by design; stop the sim (or accept the floor) rather than touch the presets. All nine still # loaded with ≥ 1.0 GB free at that ~4–4.5 GB desktop — the busy-desktop sizing holds. # # Router: user systemd unit llama.service (this dir, Nice=19) = # llama-server-cuda --host 0.0.0.0 --port 11343 --cors-origins localhost \ # --models-max 1 --models-preset ~/.config/llamacpp/config.ini # abbr `llamaserver` = restart it + tail the journal. Presets are read ONCE → restart after edits. # LAN-reachable (http://192.168.50.24:11343/v1, DHCP) and keyless: anyone on the office network can use it. # This file stows to ~/.config/llamacpp/ (like fl/lw); the GGUFs live in ~/Documents/models/. # Section names ARE the API model ids. Paths must stay absolute (no ~ expansion in presets). # ═════════════════════════════════════════════════════════════════════════════ # ───────────────────────────────────────────────────────────────────────────── # Fast — the only class that fits ENTIRELY on the GPU (≤ ~6 GB file): dense 9B, tiny-active MoE, E4B # ───────────────────────────────────────────────────────────────────────────── [Qwen3.5-9B-Q4_K_M] # ⭐ DAILY DRIVER — Qwen3.5-9B, Q4_K_M (5.87 GB, the -MTP repo file; identical weights to the plain # repo's Q4_K_M plus an MTP head that is only used by the [Qwen3.5-9B-MTP] alias). Fully on GPU. # llama-bench 2026-09-28: pp512 828 t/s / tg128 38.3 t/s — vision via the mmproj (projector on # CPU: no-mmproj-offload). Thinking hard-OFF for instant answers; -Thinking alias below. # VRAM PROBES 2026-09-28 (process VRAM, nvidia-smi, 4.6 GB desktop baseline): # no draft @16k (this preset) 5.63 GB → ~1.0 GB free ← fits # draft-mtp n=3 @8k (±mmproj) 6.38 GB → ~0.3 GB free ← MTP alias: quiet desktop only # draft-mtp @8k/16k, ub 256 6.29 GB (ctx and the CPU-side projector cost ~nothing: # hybrid attention → tiny KV; MTP is a flat +0.75 GB) # ✓ ROUTER 2026-10-19 (sim stack up, ~4–4.5 GB desktop): 30.7 t/s / 197 pp / 10.1 used / 1.7 free — GPU-only, # so only the shared GPU (Unreal at 41%) shows: −6% vs 09-28. # ⬇ SOURCE unsloth/Qwen3.5-9B-MTP-GGUF → Qwen3.5-9B-Q4_K_M.gguf saved as *-MTP-Q4_K_M.gguf (sha256 # verified); mmproj = the plain repo's mmproj-BF16 (same encoder). The plain Qwen3.5-9B-Q4_K_M.gguf # (5.68 GB, fetched by `llama-server -hf` earlier) was redundant → DELETED 2026-10-12. model = /home/coja/Documents/models/Qwen3.5-9B-MTP-Q4_K_M.gguf mmproj = /home/coja/Documents/models/Qwen3.5-9B-mmproj-BF16.gguf no-mmproj-offload = true # vision encoder on CPU reasoning-budget = 0 # instant answers; delete to allow /think (or use the alias) ctx-size = 16384 # ✓ ROUTER 09-28: 32.6 t/s / 184 pp / 10.8 used / 1.0 free @4.6 desktop. # 5.63 GB process VRAM @16k (probe); n_ctx_train 262144, KV is # cheap on this hybrid-attention arch — 32768 likely fits, bench first n-gpu-layers = 99 threads = 6 flash-attn = on cache-type-k = q8_0 cache-type-v = q8_0 batch-size = 2048 ubatch-size = 512 cache-reuse = 256 temp = 0.7 # Qwen non-thinking card: 0.7 / 0.8 / 20 / 0, rep 1.05 top-p = 0.8 top-k = 20 min-p = 0 repeat-penalty = 1.05 jinja = on sleep-idle-seconds = 60 [Qwen3.5-9B-Thinking] # Same GGUF, reasoning ON with a medium budget (fl's pattern) — zero disk cost. Slower per # answer (thinking preamble streams first as reasoning_content); use for hard problems. # No MTP here on purpose: thinking tokens draft poorly (Ornith: 60% → 47% acceptance) and the # draft's 0.75 GB does not fit beside the desktop anyway. ✓ ROUTER 09-28: 37.8 t/s / 219 pp / TTFT # 2.3 s (the think block) / 1.0 free · 10-19 (sim stack up): 33.7 / 195 pp / TTFT 2.6 s / 1.7 free. model = /home/coja/Documents/models/Qwen3.5-9B-MTP-Q4_K_M.gguf mmproj = /home/coja/Documents/models/Qwen3.5-9B-mmproj-BF16.gguf no-mmproj-offload = true reasoning-budget = 8192 # caps runaway think loops; leaves half the ctx for prompt+answer ctx-size = 16384 n-gpu-layers = 99 threads = 6 flash-attn = on cache-type-k = q8_0 cache-type-v = q8_0 batch-size = 2048 ubatch-size = 512 cache-reuse = 256 temp = 0.6 # Qwen thinking card: 0.6 / 0.95 / 20 / 0, no rep-penalty top-p = 0.95 top-k = 20 min-p = 0 jinja = on sleep-idle-seconds = 60 [Qwen3.5-9B-MTP] # ⚠ QUIET-DESKTOP SPECIALTY (fl's Qwen3.8-27B pattern) — the daily driver + its MTP draft. The draft # is a flat +0.75 GB (6.29 GB process VRAM @ub 256, probe 09-28): needs a desktop ≤ ~4.5 GB # (1.0 GB free there on 10-19, 0.4 free at 4.8 — close the electron apps on a fat desktop), # otherwise the load fails cleanly and the router reports it. Worth it when it fits: fl # measured nearly 2× decode on this model. # ✓ ROUTER 2026-09-28 (guard lowered to 0.2 for the measurement): 57.3 t/s / 164 pp / 90% draft # acceptance / 11.4 used / 0.4 free @4.6 desktop — +75% over the plain preset's 32.6. # ✓ ROUTER 2026-10-19 (sim stack up, ~4–4.5 GB desktop, default guard): 51.3 t/s / 140 pp / 90% / 10.8 used / # 1.0 free — loaded at the guard on a normal busy desktop, so the "≤ ~3.9 GB" rule above is conservative; # the −10% vs 57–61 is the GPU shared with Unreal (the MTP verify is GPU work too). model = /home/coja/Documents/models/Qwen3.5-9B-MTP-Q4_K_M.gguf mmproj = /home/coja/Documents/models/Qwen3.5-9B-mmproj-BF16.gguf no-mmproj-offload = true spec-type = draft-mtp spec-draft-n-max = 3 # fl: n=3 beat n=2 by ~1 t/s on this model; drop to 2 if acceptance is low reasoning-budget = 0 ctx-size = 8192 # ctx barely moves the footprint here; kept small out of caution n-gpu-layers = 99 threads = 6 flash-attn = on cache-type-k = q8_0 cache-type-v = q8_0 batch-size = 512 ubatch-size = 256 # −0.09 GB vs 512 (probe) — every bit counts for this one cache-reuse = 256 temp = 0.7 top-p = 0.8 top-k = 20 min-p = 0 repeat-penalty = 1.05 jinja = on sleep-idle-seconds = 60 # ─── RETIRED 2026-10-12: [LFM2.5-8B-A1B-UD-Q4_K_XL] ─────────────────────────────────────────── # Liquid LFM2.5-8B-A1B UD-Q4_K_XL (5.35 GB): fastest decode on the box (119 t/s quiet / 69 busy) but # the user found its OUTPUT QUALITY unusable in pi → retired the same day it topped the chart. The # instant tier is gemma-4-E4B (101 t/s quiet / 85 busy, vision, MTP). File deleted. [gemma-4-E4B-it-UD-Q4_K_XL] # Small VISION model + MTP draft — gemma-4 E4B (~4B effective), QAT UD-Q4_K_XL 4.22 GB, fully on # GPU. ✓ ROUTER BENCH 2026-09-28: 84.7 t/s WITH the draft (64% acceptance) / 297 pp / TTFT 0.15 s / # 8.2 used / 3.6 free @32k — the FASTEST model on the box (fl's Q8 does 57). Roles: images (projector on CPU # to save ~1 GB VRAM), long one-shot documents (prefill is fast), a second quick generalist. # reasoning-budget 0: since the Jul-17 template gemma-4 deliberates on hard prompts and can # return an empty answer under a small max_tokens (fl note) — fast models answer NOW. # (id drops "qat" like lw's convention; the file IS the qat build.) # ⬇ SOURCE unsloth/gemma-4-E4B-it-qat-GGUF (all sha256 verified 09-28), local ← upstream: # gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf ← same name (4.22 GB) # gemma-4-E4B-it-mmproj-F16.gguf ← mmproj-F16.gguf (0.99 GB) # gemma-4-E4B-it-mtp-Q8_0.gguf ← MTP/mtp-gemma-4-E4B-it-Q8_0.gguf (0.10 GB) # Main and drafter from the SAME repo (fl's cross-repo drafter mismatch = "Invalid input batch"). # ✓ ROUTER 2026-10-19 (sim stack up): 91.0 t/s / 84% acceptance / 292 pp / TTFT 0.10 s / 7.6 used / 4.2 free — # the t/s follows the acceptance (64–84% across runs, content-dependent); GPU-only, unbothered by the CPU load. # ℹ LOG LINES AT EVERY LOAD (incl. each wake from sleep) — harmless, do not chase: # "E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be # set (this warning is normal during memory fitting)" + "W failed to measure the memory of the extra # model, fitting without it" = the auto-fitter probing the MTP draft and giving up; everything is on the # GPU anyway (ngl 99) so the fit result is irrelevant. The model serves fine right after. # ⚠ sleep-idle 30 s vs pi (journal 2026-10-19 12:44–12:56): a pi session slept/woke this preset 3× in 12 min; # each wake = full reload ≈ 6 s before the first token (cold load 17 s). Resident it holds 3.7 GB and still # leaves ~3 GB beside a 4.5 GB desktop → 120–300 s (fl runs 300 on its coders, 30 on this same gemma) would # remove that stall for pi's default model. Left at 30 pending the user's call. model = /home/coja/Documents/models/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf mmproj = /home/coja/Documents/models/gemma-4-E4B-it-mmproj-F16.gguf no-mmproj-offload = true model-draft = /home/coja/Documents/models/gemma-4-E4B-it-mtp-Q8_0.gguf spec-type = draft-mtp spec-draft-n-max = 2 reasoning-budget = 0 # ⚠ remove if startup says the option is not recognized / not honored ctx-size = 32768 # n_ctx_train 131072; fl runs 98304 with room — raise after a bench n-gpu-layers = 99 threads = 6 flash-attn = on cache-type-k = q8_0 cache-type-v = q8_0 batch-size = 2048 ubatch-size = 512 cache-reuse = 256 # inert under mmproj ("not supported by multimodal") — kept for parity temp = 1.0 # Gemma defaults: 1.0 / 0.95 / 64 top-p = 0.95 top-k = 64 jinja = on sleep-idle-seconds = 30 # ───────────────────────────────────────────────────────────────────────────── # Coding — MoE, experts in RAM # ───────────────────────────────────────────────────────────────────────────── [Ornith-1.5-35B-A3B-IQ4_XS] # ⭐ AGENTIC CODER — Ornith-1.5-35B-A3B (ornith-ai, MIT): Qwen3.5-MoE arch (40 layers, 256 # experts / 8 active + shared, ~3B active, MTP head embedded, vision, n_ctx_train 262144). # Card claims it beats Qwen3.6-35B-A3B on coding/agent benches (SWE-bench Verified 79, # Terminal-Bench 68.5) — unverified, judge it in use. Thinking ON by default (coder role); # the -instant alias below turns it off. # MEASURED 2026-09-28 (server, ctx 16384, q8/q8 KV, 6.8 GB free baseline): # @moe34 no MTP : 11.2–15.2 t/s decode, 4.6 GB VRAM # @moe34 MTP n=2 : 18.5–19.0 t/s, acceptance 0.61 (mean len 2.2), 5.7 GB VRAM ← +25–60% # llama-bench @moe40 (all experts CPU) = 13.7 / @moe32 = 12.7 (VRAM-starved) → the wall is # DDR4 streaming 4-bit I-quant experts; MTP is the lever, not moe. # moe36 chosen for a busy-desktop margin (~5.2 GB predicted); moe34 = +~0.5 GB for ~+1 t/s. # ✓ ROUTER BENCH 2026-09-28 15:20 (bench.py, 4.5 GB desktop baseline, gen 256): # [instant] 21.2 t/s decode / 36 pp / TTFT 1.0 s / 60% MTP acceptance / 10.4 used / 1.4 free # [thinking] 10.0 t/s / 20 pp / TTFT 21.5 s (the think block) / 47% acceptance / 10.1 / 1.8 # → thinking mode halves MTP's payoff (reasoning tokens draft worse); the instant alias is # the agent-loop pick, thinking for hard problems only. 1.4 GB free = at the margin; if a # load ever fails on a busy desktop, step moe 36→38 (≈ −0.4 GB, ≈ −0.5 t/s). # 2nd sweep 15:5x (4.2 GB baseline): instant 14.0 t/s (60%) / 2.0 free · thinking 10.5 (53%) / 2.2 # free → the instant alias ranges 14–21 t/s with desktop CPU load (the experts share the 6 cores). # ✓ QUIET DESKTOP 2026-10-12 15:36 (3.3 GB baseline, user run): instant 31.5 t/s / 63 pp / TTFT 0.6 s / # 67% acceptance / 8.3 used / 3.5 free. Same preset — the 1 GB the desktop gave back is worth # +10–17 t/s here. # ✗ moe30 TESTED 2026-10-12 (same quiet desktop, 3.1 GB baseline, one-off load): 32.1 / 31.7 t/s at # 59% acceptance, 7.35 GB process VRAM, 0.7 GB free on the card → +0.5 t/s for 2.4 GB of margin. # GPU-resident experts barely move this model (fl saw the same on Coder-Next); the wall is the # MTP verify + CPU-side work, not expert bandwidth. A -fast alias was added and REMOVED the same # day on this evidence — moe36 stays; lower moe is not the lever here, free VRAM for the # desktop is. # ⚠ VARIANCE 2026-10-12 15:52 (user run, SAME quiet desktop: 3.4 GB baseline / 3.5 free): 15.1 t/s / # 9 pp / TTFT 3.4 s / 59% acceptance — vs 31.5 / 63 pp sixteen minutes earlier. Prefill is pure # compute, so a 7× prefill drop = CPU contention, not VRAM: the abbr runs the server at nice 19 and # it yields to any other CPU load (browser/electron bursts, a build, a cold mmap re-read of the # 19 GB file). Expect 14–31 t/s on this preset and judge it over several runs (`top` during one). # ✓ ROUTER 2026-10-19 (sim stack up — header; ~4–4.5 GB desktop): instant 15.0 t/s / 37 pp / 60% / 9.0 used / # 2.9 free · thinking 14.7 / 34 pp / 57% / 2.9 free (TTFT = all-think: 256 tokens of reasoning, no answer # yet). The busy floor again, this time with a named cause: Unreal Blocks + PX4 on 3+ of the 6 cores. # ⬇ SOURCE bartowski/Ornith-1.5-35B-A3B-GGUF (imatrix): Ornith-1.5-35B-A3B-IQ4_XS.gguf # 19,278,554,784 B, sha256 verified 09-28. fl runs the IQ3_XXS (15.34 GB) of the same repo # at 32 t/s with only 16 expert layers in RAM — here it would also stream fewer bytes per # token; swap to it if speed matters more than the 4-bit quality. # mmproj-Ornith-1.5-35B-A3B-f16.gguf (0.9 GB) is available in the repo — not fetched (add # `mmproj = …` + `no-mmproj-offload = true` if vision is wanted). model = /home/coja/Documents/models/Ornith-1.5-35B-A3B-IQ4_XS.gguf spec-type = draft-mtp spec-draft-n-max = 2 ctx-size = 16384 # KV ≈ 1.3 GB @16k q8/q8 (measured OOM at 8k with ngl 99 and NO # moe on the 14B — with experts in RAM the KV is the only GPU cost # besides attention). 24576 needs a bench first. n-gpu-layers = 99 n-cpu-moe = 36 # 40 MoE layers total. ↓ toward 34 on a quiet desktop (bench free ≥ 1 GB) threads = 6 flash-attn = on cache-type-k = q8_0 cache-type-v = q8_0 batch-size = 2048 ubatch-size = 512 cache-reuse = 256 temp = 0.6 # model card, general use: 0.6 / 0.95 / 20 / 0 top-p = 0.95 top-k = 20 min-p = 0 jinja = on sleep-idle-seconds = 120 [Ornith-1.5-35B-A3B-instant] # Same GGUF with thinking hard-OFF (reasoning-budget 0) for agent loops that can't afford the # thinking TTFT. Otherwise identical to the preset above. model = /home/coja/Documents/models/Ornith-1.5-35B-A3B-IQ4_XS.gguf spec-type = draft-mtp spec-draft-n-max = 2 reasoning-budget = 0 ctx-size = 16384 n-gpu-layers = 99 n-cpu-moe = 36 threads = 6 flash-attn = on cache-type-k = q8_0 cache-type-v = q8_0 batch-size = 2048 ubatch-size = 512 cache-reuse = 256 temp = 0.6 top-p = 0.95 top-k = 20 min-p = 0 jinja = on sleep-idle-seconds = 120 [Ornith-1.5-35B-A3B-long] # The long-context tier (added 2026-10-28): the same GGUF at 128k (n_ctx_train 262144) — wm's answer # to fl's Qwen3-Coder-Next 80B, which is NOT viable here (RAM, see candidates). Thinking OFF like # -instant and like Coder-Next: a think block would eat the context this preset exists for; set # reasoning-budget = -1 to bring it back. # All 40 expert layers in RAM: llama-bench measured moe40 ≈ moe32 on this model (13.7 vs 12.7 t/s), so # the ~1.6 GB moe36→40 frees pays for the KV. Qwen3.5-MoE is hybrid — only every 4th layer holds KV — # so 128k costs ~1.2 GB over 16k. # llama-fit-params 2026-10-28 (moe40, q8/q8, fa on, no MTP; CUDA0 MiB model / KV / compute): # 16k 1342/232/530 · 64k 1342/742/530 · 128k 1342/1422/569 · 256k 1342/2782/953 (+1.7 GB: no room) # ✓ ROUTER BENCH 2026-10-28 16:13 (bench.py --ctx 32000, 4.7 GB baseline, load avg ~11 = busy desktop): # 10.0 t/s short-prompt decode / 59% MTP / 10.0 used / 1.8 free · @35.6k prompt: prefill 249 t/s # (142.7 s), decode 16.9 t/s at 65% acceptance. → a 100k prompt ≈ 7 min to prefill: long context # here is for patient one-shots and growing agent sessions (cache-reuse), not dumping 100k at once. # The 19 GB mmap pushed the desktop ~3.5 GB further into swap (9.5 → 13 GiB) — RAM is the limit. model = /home/coja/Documents/models/Ornith-1.5-35B-A3B-IQ4_XS.gguf spec-type = draft-mtp spec-draft-n-max = 2 reasoning-budget = 0 ctx-size = 131072 n-gpu-layers = 99 n-cpu-moe = 40 # = all 40 MoE layers (see header) threads = 6 flash-attn = on cache-type-k = q8_0 cache-type-v = q8_0 batch-size = 2048 ubatch-size = 512 cache-reuse = 256 temp = 0.6 top-p = 0.95 top-k = 20 min-p = 0 jinja = on sleep-idle-seconds = 120 # ─── RETIRED 2026-10-12: [Qwen3-Coder-30B-Instruct-UD-IQ2_XXS] ────────────────────────────── # Qwen3-Coder-30B-A3B, UD-IQ2_XXS (10.33 GB, 2-bit). Router bench 09-28 @moe34/16k: 14.0 t/s, 1.9 GB # free (moe28 left 0.6 GB). Dominated by Ornith-instant (same speed, 4-bit, MTP) → removed from the # three wm client overlays and DELETED from ~/Documents/models/ 2026-10-12. The K-quant upgrade # (UD-Q3_K_XL 13.8 GB) stays in the candidates list if a second coder is ever wanted. # ───────────────────────────────────────────────────────────────────────────── # Reasoning + tools — gpt-oss # ───────────────────────────────────────────────────────────────────────────── [gpt-oss-20b] # OpenAI gpt-oss-20b — MoE 21B / 3.6B active, 24 layers, harmony template (jinja required). # File is the Q4_K_M requant (11.62 GB); the experts are MXFP4 in every quant, so this is # effectively the native file (fl uses ggml-org's mxfp4.gguf, 12.1 GB — same thing). # llama-bench 2026-09-28 (~9 GB free): @moe12 = 648 pp / 29.8 tg · @moe24 (all CPU) = 455 / 18.6. # The fitter picked ~10 expert layers on CPU @16k with 7.5 free; moe14 = margin for a busy # desktop. Sliding-window attention → KV is cheap, so ctx is generous. # KV types left at default on purpose (attention sinks + quantized KV = known trouble, fl note). # ⬇ On disk since before this config (user download). TTFT of several seconds = the model # THINKING (harmony reasoning streams first) — use the -low alias for snappy answers. # ✓ ROUTER 2026-10-19 (sim stack up): 11.7 t/s / 24 pp / 8.8 used / 3.0 free (TTFT all-think) · -low 9.8 / # 23 pp / TTFT 3.1 s / 2.6 free — the lowest yet (21 quiet 10-12, 17.4 busy 09-28): all 18 expert layers # stream from RAM on a CPU the sim already half-owns. Not a preset problem; see the header. model = /home/coja/Documents/models/gpt-oss-20b-Q4_K_M.gguf ctx-size = 32768 # ⚗ unbenched at 32k — fl measured 32k→64k = +1.2 GB on this arch; # here 16k is the safe fallback if bench.py shows < 1 GB free n-gpu-layers = 99 n-cpu-moe = 18 # 24 MoE layers. ✗ moe14 @32k = 0.3 GB free (router sweep 09-28) → 18 (+4 layers # ≈ +1.8 GB). ✓ moe18 @32k: 17.4 t/s / 30 pp / TTFT 8.8 s (= thinking) / # 9.7 used / 2.1 free (router 09-28). moe12 = 29.8 needs ~9 GB free threads = 6 flash-attn = on batch-size = 2048 ubatch-size = 512 cache-reuse = 256 temp = 1.0 top-p = 1.0 jinja = on sleep-idle-seconds = 120 [gpt-oss-20b-low] # Same GGUF, reasoning effort LOW — zero disk cost. Cuts the thinking preamble for interactive # use; keep [gpt-oss-20b] (medium) for hard problems. model = /home/coja/Documents/models/gpt-oss-20b-Q4_K_M.gguf chat-template-kwargs = {"reasoning_effort":"low"} ctx-size = 32768 n-gpu-layers = 99 n-cpu-moe = 18 # = the main preset. ✓ 17.5 t/s / TTFT 1.8 s / 2.1 free (router 09-28) threads = 6 flash-attn = on batch-size = 2048 ubatch-size = 512 cache-reuse = 256 temp = 1.0 top-p = 1.0 jinja = on sleep-idle-seconds = 60 # ─── RETIRED 2026-09-28: [Qwen3-14B-UD-Q3_K_XL] ────────────────────────────────────────── # Dense 14B, UD-Q3_K_XL (7.59 GB, 40 layers): does not fit beside the desktop. ngl 34 @16k was # the fitter's split at 7.5 GB free and gave 7.6 t/s (llama-bench); through the router at a # 4.5 GB desktop baseline it FAILED TO LOAD (context creation, 15:2x sweep). A fitting split # (~ngl 28 @8k) lands ≈ 5 t/s — Qwen3.5-9B covers dense at 38 t/s. Not wired to clients. # DELETED from ~/Documents/models/ 2026-10-12. # ─── RETIRED 2026-09-28 (never wired) — more dense files that cannot fit this card ─────── # All measured with llama-bench at the fitter's ngl for 16k ctx (p128/n64), 6.8 GB free: # Qwen3-8B-Q8_0.gguf (8.71 GB) ngl 19 → 3.4 t/s # → a Q4_K_M (~5 GB, unsloth/Qwen3-8B-GGUF) WOULD fit fully and run ~40 t/s; but # Qwen3.5-9B already covers that slot. Delete or re-quant. # Ministral-3-14B-Instruct-2512-Q4_K_M.gguf (8.24 GB) ngl 21 → 3.8 t/s # Devstral-Small-2-24B-Instruct-2512-UD-IQ3_XXS.gguf (9.41) ngl 28 → 1.8 t/s # Lexi-Llama-3-8B-Uncensored_F16.gguf (16.07 GB, F16!) ngl 9 → 1.0 t/s # → an 8B at F16 is 2× the bytes of Q8 for no quality; a Q4_K_M of it would fit fully. # All four DELETED from ~/Documents/models/ 2026-10-12 (66 GB freed together with the 14B, the plain 9B # and the 2-bit coder). # ───────────────────────────────────────────────────────────────────────────── # Coding — fits ENTIRELY on the GPU (added 2026-10-12 after passing its gate bench) # ───────────────────────────────────────────────────────────────────────────── [Mellum2.1-12B-A2.5B-Thinking-IQ4_XS] # ⚗ JetBrains Mellum 2.1 — code MoE, 12B total / 2.5B active, 28 layers × 64 experts, THINKING on. # IQ4_XS 6.97 GB: fits entirely on a quiet desktop (fitter @16k: no offload at 8.3 GB free); # n-cpu-moe 4 keeps ~0.85 GB in RAM as busy-desktop insurance (each layer's experts ≈ 0.2 GB) — # drop to 0 on a quiet desktop (see probe). # ✓ ROUTER BENCH 2026-10-12 @moe4: 68.7 t/s decode / 9.5 used / 2.3 free. The bench's "15 pp" is the # short-prompt artifact — PROBE with a 1,713-token prompt: moe4 = 1,160 pp / 62.8–63.7 t/s / 6.2 GB # process VRAM; moe0 (all on GPU) = 1,549 pp / 86.7–87.4 t/s / 6.9 GB. KEPT at moe4 (loads beside a # 4.5 GB desktop: ~1.1 GB free); moe0 is +24 t/s when the desktop holds ≤ ~3.9 GB. Role: the fast # REASONING coder — 63–87 t/s with a think block, prefill > 1k t/s. Wired into aichat/opencode/pi. # Note n_ctx_train is 131072 via RoPE scaling over an 8192 base — keep ctx modest until tested. # ✓ ROUTER 2026-10-19 (sim stack up): 41.9 t/s / TTFT 5.2 s (= the think block) / 10.2 used / 1.6 free — −40% vs # quiet: even 4 CPU-expert layers hurt once the cores are taken. moe0 would dodge it, on a quiet desktop only. # ⬇ SOURCE bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF: Mellum2.1-12B-A2.5B-Thinking-IQ4_XS.gguf (sha verified). model = /home/coja/Documents/models/Mellum2.1-12B-A2.5B-Thinking-IQ4_XS.gguf ctx-size = 16384 n-gpu-layers = 99 n-cpu-moe = 4 # 28 MoE layers; 0 = all on GPU (quiet desktop only) threads = 6 flash-attn = on cache-type-k = q8_0 cache-type-v = q8_0 batch-size = 2048 ubatch-size = 512 cache-reuse = 256 temp = 0.6 # thinking-model defaults (0.6 / 0.95 / 20) top-p = 0.95 top-k = 20 min-p = 0 jinja = on sleep-idle-seconds = 60 # ─── RETIRED 2026-10-19 (user's call): [AesCode-8B-Q4_K_M] ───────────────────────────────────── # Microsoft AesCode-8B (qwen3vl, 36 layers, vision), bartowski Q4_K_M 5.21 GB + mmproj f16 1.16 GB. # Router bench 2026-10-12: 36.4 t/s / 386 pp / TTFT 0.07 s / 9.4 used / 2.4 free, fully on the GPU at # ctx 16384 (mmproj on CPU). Was the quick-edit / screenshot-aware coder; removed from aichat/opencode/pi. # ─── RETIRED 2026-10-12 (same day as its gate bench): [FrogNano-4B-2609-Q4_K_M] ────────────────── # Microsoft FrogNano-4B-2609 (qwen35, MTP head, vision), Q4_K_M 2.80 GB + mmproj 0.67. Router bench: # 87.9 t/s with the draft (84% acceptance) / 244 pp / 7.9 used / 3.9 free — good, but no slot: on the # same quiet desktop gemma-4-E4B does 101 t/s WITH vision and LFM2.5-8B-A1B 119 without. Both files # deleted from ~/Documents/models/ 2026-10-12. # ───────────────────────────────────────────────────────────────────────────── # Candidates — ⬇ NOT ON DISK. Sized for this box (≤ ~20 GB file, 2–4B active). Add a section # when downloaded; measured expectations are extrapolations from the presets above. # ───────────────────────────────────────────────────────────────────────────── # (2026-09-28: user dropped Qwen3.6-35B-A3B, GLM-4.7-Flash, LFM2-24B-A2B, the coder upgrades and the # embedder from the shortlist; LFM2.5-8B-A1B and gemma-4-E4B were downloaded and have presets above.) # unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-IQ3_XXS 14.1 GB — newest general Qwen MoE + MTP; est. ~25 t/s here. # unsloth/Qwen3-Coder-30B-A3B-Instruct UD-Q3_K_XL 13.8 GB — K-quant coder (est. ~15–18 t/s); the 2-bit file was deleted 10-12. # unsloth/GLM-4.7-Flash-GGUF UD-Q3_K_XL 13.8 GB — K-quant coder; deepseek2/MLA → KV k/v types must match. # LiquidAI/LFM2-24B-A2B-GGUF Q4_K_M 14.4 GB — 2B active; est. ~20–25 t/s, unproven arch. # unsloth/gemma-4-26B-A4B-it-qat-GGUF UD-Q4_K_XL 14.2 GB — 4B active in RAM = slower than the 3B-active Qwens. # NOT viable (RAM): Qwen3-Coder-Next 80B (≥ 28 GB), Qwen3.8-Flash-Next, GLM-5.3-Flash, Kimi-Linear-48B # (≥ 23 GB), Qwen3.6/3.8-27B and Muse-Glimmer-30B (dense — see the tombstones above). # ───────────────────────────────────────────────────────────────────────────── # Notes # ───────────────────────────────────────────────────────────────────────────── # n-cpu-moe N : experts of the first N MoE layers stay in system RAM. Higher = less VRAM # (loads on a busier desktop) but slower; each layer on the GPU ≈ 0.4 GB for a # 35B-A3B at 4-bit. Dense models ignore it — use n-gpu-layers / ctx instead. # Sizing rule : file GB × (1 − N/layers) + KV + ~0.7 GB compute ≤ VRAM free at load time. # Fitter : `llama-fit-params -m -c -fa on` prints the split that fits RIGHT # NOW (it reads current free VRAM) — a good first value after a download. # After ANY change: restart the router (presets are read once) and run ./bench.py — keep # every "VRAM free" ≥ ~1 GB; a load failure here is a clean error, not a freeze.