490 lines
34 KiB
INI
490 lines
34 KiB
INI
# llama.cpp model config — WM (Wingman work desktop) · NVIDIA GTX 1080 Ti 11 GB (Pascal, legacy)
|
||
# backend = upstream prebuilt CUDA 12.8 build (~/.local/opt/llama.cpp-cuda, launched via the
|
||
# `llama-server-cuda` wrapper in ~/.local/bin) — NOT the Arch package, NOT a local build.
|
||
# ═════════════════════════════════════════════════════════════════════════════
|
||
# HARDWARE: i7-8700 (6 cores / 12 threads, AVX2, no AVX-512) · 31 GiB DDR4 dual-channel
|
||
# (~35 GB/s effective = the CPU-expert ceiling) · GTX 1080 Ti 11,264 MiB, compute
|
||
# capability 6.1 (Pascal): no tensor cores, no usable fp16 → CUDA runs its int8 MMQ
|
||
# path, which is what llama.cpp is best at on this card. Driver nvidia-580xx (AUR):
|
||
# the LAST branch for Pascal; CUDA 13.x toolkits can no longer target this GPU, so
|
||
# the repo `ggml-cuda`/`llama.cpp-cuda-git` builds are USELESS here — the prebuilt
|
||
# CUDA 12.8 tarball carries Pascal as PTX and the driver JITs it (first run of a new
|
||
# build ≈ 50 s, then cached in ~/.nv/ComputeCache; the wrapper raises the cache cap).
|
||
#
|
||
# This is the NVIDIA analog of fl/.config/llamacpp/config.ini (RX 7600 XT 16 GB). Same
|
||
# recipe (MoE experts on CPU, quantized KV, MTP), different limits:
|
||
#
|
||
# • VRAM BUDGET: 11.2 GB card, but the desktop (Hyprland + Xorg + electron apps) holds
|
||
# 3.5–4.5 GB → ~6.5–7.5 GB usable. Every preset below is sized for ~6.5 free.
|
||
# ⚠ Unlike fl's amdgpu, an NVIDIA overcommit does NOT freeze the PC — cudaMalloc fails and
|
||
# the load errors out cleanly ("failed to allocate buffer for kv cache"). The guard here is
|
||
# therefore "loads reliably on a busy desktop", target ≥ ~1 GB free after load (bench.py
|
||
# records it). Still keep `--models-max 1`: two presets never fit together.
|
||
# • MoE IS THE BOX. Dense models that don't fit VRAM fall off a cliff (every layer left on
|
||
# the CPU streams from DDR4): Qwen3-14B Q3 = 7.6 t/s @ngl34, Qwen3-8B Q8 = 3.4, Ministral-14B
|
||
# = 3.8, Devstral-24B = 1.8 (measured 2026-09-28, tombstones below). A 20–35B MoE with its
|
||
# experts in RAM decodes at 15–30 t/s because only ~3B params are read per token.
|
||
# • `n-cpu-moe N` = experts of the first N layers stay in RAM. LOWER = more VRAM, faster.
|
||
# Measured 2026-09-28 (llama-bench, ~9 GB free at the time): gpt-oss-20b @moe12 = 29.8 t/s,
|
||
# @moe24 (all) = 18.6; Qwen3-Coder-30B IQ2_XXS @moe24 = 29.9, @moe48 (all) = 20.2.
|
||
# • QUANT TYPE MATTERS FOR CPU EXPERTS: I-quants (IQ4_XS/IQ3_XXS) dequantize slowly on CPU and
|
||
# 4-bit experts stream twice the bytes of 2-bit ones — Ornith IQ4_XS with 80% of experts in
|
||
# RAM = 11–15 t/s base. When choosing a file for the experts-in-RAM pattern prefer K-quants
|
||
# (Q4_K_M/UD-Q4_K_XL/UD-Q3_K_XL) and the smallest quant the quality allows.
|
||
# • `threads = 6` = PHYSICAL cores. NEVER 12: measured 2026-09-28, tg collapses 14.3 → 1.3 t/s
|
||
# (HT contention on the expert matmuls). The default (6) is right; the key is pinned anyway.
|
||
# • MTP / speculative decode WORKS here (`spec-type = draft-mtp`, Qwen3.5-family heads):
|
||
# Ornith @moe34: 11–15 → 18.5–19.0 t/s at 61% acceptance for ~1.1 GB extra VRAM (09-28).
|
||
# Unlike lw (2 cores) the 6 cores have compute to spare for the batched verify.
|
||
# • KV cache: `cache-type-k/v = q8_0` everywhere except gpt-oss (attention sinks — keep the
|
||
# defaults there, same as fl). flash-attn = on everywhere (required for quantized V).
|
||
# • CPU governor is `powersave` (intel_pstate still boosts). The service runs at Nice=19 so the
|
||
# 6-core expert matmuls never lag the desktop; GPU work is unaffected by niceness. THE FLIP SIDE
|
||
# (full sweep 2026-10-19 with the AirSim/PX4 sim stack up: Unreal Blocks ≈ 3 cores at nice 5,
|
||
# root nodes at realtime priority, load avg ~9, GPU 41% busy): every CPU-expert preset sat at
|
||
# its floor — Ornith 15 t/s (31 quiet), gpt-oss 10–12 (21), Mellum 42 (69) — and the GPU-only
|
||
# presets lost 5–15% to the shared GPU (9B-MTP 51 vs 57–61). Nothing to tune: the server yields
|
||
# by design; stop the sim (or accept the floor) rather than touch the presets. All nine still
|
||
# loaded with ≥ 1.0 GB free at that ~4–4.5 GB desktop — the busy-desktop sizing holds.
|
||
#
|
||
# Router: user systemd unit llama.service (this dir, Nice=19) =
|
||
# llama-server-cuda --host 0.0.0.0 --port 11343 --cors-origins localhost \
|
||
# --models-max 1 --models-preset ~/.config/llamacpp/config.ini
|
||
# abbr `llamaserver` = restart it + tail the journal. Presets are read ONCE → restart after edits.
|
||
# LAN-reachable (http://192.168.50.24:11343/v1, DHCP) and keyless: anyone on the office network can use it.
|
||
# This file stows to ~/.config/llamacpp/ (like fl/lw); the GGUFs live in ~/Documents/models/.
|
||
# Section names ARE the API model ids. Paths must stay absolute (no ~ expansion in presets).
|
||
# ═════════════════════════════════════════════════════════════════════════════
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Fast — the only class that fits ENTIRELY on the GPU (≤ ~6 GB file): dense 9B, tiny-active MoE, E4B
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
|
||
[Qwen3.5-9B-Q4_K_M]
|
||
# ⭐ DAILY DRIVER — Qwen3.5-9B, Q4_K_M (5.87 GB, the -MTP repo file; identical weights to the plain
|
||
# repo's Q4_K_M plus an MTP head that is only used by the [Qwen3.5-9B-MTP] alias). Fully on GPU.
|
||
# llama-bench 2026-09-28: pp512 828 t/s / tg128 38.3 t/s — vision via the mmproj (projector on
|
||
# CPU: no-mmproj-offload). Thinking hard-OFF for instant answers; -Thinking alias below.
|
||
# VRAM PROBES 2026-09-28 (process VRAM, nvidia-smi, 4.6 GB desktop baseline):
|
||
# no draft @16k (this preset) 5.63 GB → ~1.0 GB free ← fits
|
||
# draft-mtp n=3 @8k (±mmproj) 6.38 GB → ~0.3 GB free ← MTP alias: quiet desktop only
|
||
# draft-mtp @8k/16k, ub 256 6.29 GB (ctx and the CPU-side projector cost ~nothing:
|
||
# hybrid attention → tiny KV; MTP is a flat +0.75 GB)
|
||
# ✓ ROUTER 2026-10-19 (sim stack up, ~4–4.5 GB desktop): 30.7 t/s / 197 pp / 10.1 used / 1.7 free — GPU-only,
|
||
# so only the shared GPU (Unreal at 41%) shows: −6% vs 09-28.
|
||
# ⬇ SOURCE unsloth/Qwen3.5-9B-MTP-GGUF → Qwen3.5-9B-Q4_K_M.gguf saved as *-MTP-Q4_K_M.gguf (sha256
|
||
# verified); mmproj = the plain repo's mmproj-BF16 (same encoder). The plain Qwen3.5-9B-Q4_K_M.gguf
|
||
# (5.68 GB, fetched by `llama-server -hf` earlier) was redundant → DELETED 2026-10-12.
|
||
model = /home/coja/Documents/models/Qwen3.5-9B-MTP-Q4_K_M.gguf
|
||
mmproj = /home/coja/Documents/models/Qwen3.5-9B-mmproj-BF16.gguf
|
||
no-mmproj-offload = true # vision encoder on CPU
|
||
reasoning-budget = 0 # instant answers; delete to allow /think (or use the alias)
|
||
ctx-size = 16384 # ✓ ROUTER 09-28: 32.6 t/s / 184 pp / 10.8 used / 1.0 free @4.6 desktop.
|
||
# 5.63 GB process VRAM @16k (probe); n_ctx_train 262144, KV is
|
||
# cheap on this hybrid-attention arch — 32768 likely fits, bench first
|
||
n-gpu-layers = 99
|
||
threads = 6
|
||
flash-attn = on
|
||
cache-type-k = q8_0
|
||
cache-type-v = q8_0
|
||
batch-size = 2048
|
||
ubatch-size = 512
|
||
cache-reuse = 256
|
||
temp = 0.7 # Qwen non-thinking card: 0.7 / 0.8 / 20 / 0, rep 1.05
|
||
top-p = 0.8
|
||
top-k = 20
|
||
min-p = 0
|
||
repeat-penalty = 1.05
|
||
jinja = on
|
||
sleep-idle-seconds = 60
|
||
|
||
[Qwen3.5-9B-Thinking]
|
||
# Same GGUF, reasoning ON with a medium budget (fl's pattern) — zero disk cost. Slower per
|
||
# answer (thinking preamble streams first as reasoning_content); use for hard problems.
|
||
# No MTP here on purpose: thinking tokens draft poorly (Ornith: 60% → 47% acceptance) and the
|
||
# draft's 0.75 GB does not fit beside the desktop anyway. ✓ ROUTER 09-28: 37.8 t/s / 219 pp / TTFT
|
||
# 2.3 s (the think block) / 1.0 free · 10-19 (sim stack up): 33.7 / 195 pp / TTFT 2.6 s / 1.7 free.
|
||
model = /home/coja/Documents/models/Qwen3.5-9B-MTP-Q4_K_M.gguf
|
||
mmproj = /home/coja/Documents/models/Qwen3.5-9B-mmproj-BF16.gguf
|
||
no-mmproj-offload = true
|
||
reasoning-budget = 8192 # caps runaway think loops; leaves half the ctx for prompt+answer
|
||
ctx-size = 16384
|
||
n-gpu-layers = 99
|
||
threads = 6
|
||
flash-attn = on
|
||
cache-type-k = q8_0
|
||
cache-type-v = q8_0
|
||
batch-size = 2048
|
||
ubatch-size = 512
|
||
cache-reuse = 256
|
||
temp = 0.6 # Qwen thinking card: 0.6 / 0.95 / 20 / 0, no rep-penalty
|
||
top-p = 0.95
|
||
top-k = 20
|
||
min-p = 0
|
||
jinja = on
|
||
sleep-idle-seconds = 60
|
||
|
||
[Qwen3.5-9B-MTP]
|
||
# ⚠ QUIET-DESKTOP SPECIALTY (fl's Qwen3.8-27B pattern) — the daily driver + its MTP draft. The draft
|
||
# is a flat +0.75 GB (6.29 GB process VRAM @ub 256, probe 09-28): needs a desktop ≤ ~4.5 GB
|
||
# (1.0 GB free there on 10-19, 0.4 free at 4.8 — close the electron apps on a fat desktop),
|
||
# otherwise the load fails cleanly and the router reports it. Worth it when it fits: fl
|
||
# measured nearly 2× decode on this model.
|
||
# ✓ ROUTER 2026-09-28 (guard lowered to 0.2 for the measurement): 57.3 t/s / 164 pp / 90% draft
|
||
# acceptance / 11.4 used / 0.4 free @4.6 desktop — +75% over the plain preset's 32.6.
|
||
# ✓ ROUTER 2026-10-19 (sim stack up, ~4–4.5 GB desktop, default guard): 51.3 t/s / 140 pp / 90% / 10.8 used /
|
||
# 1.0 free — loaded at the guard on a normal busy desktop, so the "≤ ~3.9 GB" rule above is conservative;
|
||
# the −10% vs 57–61 is the GPU shared with Unreal (the MTP verify is GPU work too).
|
||
model = /home/coja/Documents/models/Qwen3.5-9B-MTP-Q4_K_M.gguf
|
||
mmproj = /home/coja/Documents/models/Qwen3.5-9B-mmproj-BF16.gguf
|
||
no-mmproj-offload = true
|
||
spec-type = draft-mtp
|
||
spec-draft-n-max = 3 # fl: n=3 beat n=2 by ~1 t/s on this model; drop to 2 if acceptance is low
|
||
reasoning-budget = 0
|
||
ctx-size = 8192 # ctx barely moves the footprint here; kept small out of caution
|
||
n-gpu-layers = 99
|
||
threads = 6
|
||
flash-attn = on
|
||
cache-type-k = q8_0
|
||
cache-type-v = q8_0
|
||
batch-size = 512
|
||
ubatch-size = 256 # −0.09 GB vs 512 (probe) — every bit counts for this one
|
||
cache-reuse = 256
|
||
temp = 0.7
|
||
top-p = 0.8
|
||
top-k = 20
|
||
min-p = 0
|
||
repeat-penalty = 1.05
|
||
jinja = on
|
||
sleep-idle-seconds = 60
|
||
|
||
# ─── RETIRED 2026-10-12: [LFM2.5-8B-A1B-UD-Q4_K_XL] ───────────────────────────────────────────
|
||
# Liquid LFM2.5-8B-A1B UD-Q4_K_XL (5.35 GB): fastest decode on the box (119 t/s quiet / 69 busy) but
|
||
# the user found its OUTPUT QUALITY unusable in pi → retired the same day it topped the chart. The
|
||
# instant tier is gemma-4-E4B (101 t/s quiet / 85 busy, vision, MTP). File deleted.
|
||
|
||
[gemma-4-E4B-it-UD-Q4_K_XL]
|
||
# Small VISION model + MTP draft — gemma-4 E4B (~4B effective), QAT UD-Q4_K_XL 4.22 GB, fully on
|
||
# GPU. ✓ ROUTER BENCH 2026-09-28: 84.7 t/s WITH the draft (64% acceptance) / 297 pp / TTFT 0.15 s /
|
||
# 8.2 used / 3.6 free @32k — the FASTEST model on the box (fl's Q8 does 57). Roles: images (projector on CPU
|
||
# to save ~1 GB VRAM), long one-shot documents (prefill is fast), a second quick generalist.
|
||
# reasoning-budget 0: since the Jul-17 template gemma-4 deliberates on hard prompts and can
|
||
# return an empty answer under a small max_tokens (fl note) — fast models answer NOW.
|
||
# (id drops "qat" like lw's convention; the file IS the qat build.)
|
||
# ⬇ SOURCE unsloth/gemma-4-E4B-it-qat-GGUF (all sha256 verified 09-28), local ← upstream:
|
||
# gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf ← same name (4.22 GB)
|
||
# gemma-4-E4B-it-mmproj-F16.gguf ← mmproj-F16.gguf (0.99 GB)
|
||
# gemma-4-E4B-it-mtp-Q8_0.gguf ← MTP/mtp-gemma-4-E4B-it-Q8_0.gguf (0.10 GB)
|
||
# Main and drafter from the SAME repo (fl's cross-repo drafter mismatch = "Invalid input batch").
|
||
# ✓ ROUTER 2026-10-19 (sim stack up): 91.0 t/s / 84% acceptance / 292 pp / TTFT 0.10 s / 7.6 used / 4.2 free —
|
||
# the t/s follows the acceptance (64–84% across runs, content-dependent); GPU-only, unbothered by the CPU load.
|
||
# ℹ LOG LINES AT EVERY LOAD (incl. each wake from sleep) — harmless, do not chase:
|
||
# "E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be
|
||
# set (this warning is normal during memory fitting)" + "W failed to measure the memory of the extra
|
||
# model, fitting without it" = the auto-fitter probing the MTP draft and giving up; everything is on the
|
||
# GPU anyway (ngl 99) so the fit result is irrelevant. The model serves fine right after.
|
||
# ⚠ sleep-idle 30 s vs pi (journal 2026-10-19 12:44–12:56): a pi session slept/woke this preset 3× in 12 min;
|
||
# each wake = full reload ≈ 6 s before the first token (cold load 17 s). Resident it holds 3.7 GB and still
|
||
# leaves ~3 GB beside a 4.5 GB desktop → 120–300 s (fl runs 300 on its coders, 30 on this same gemma) would
|
||
# remove that stall for pi's default model. Left at 30 pending the user's call.
|
||
model = /home/coja/Documents/models/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf
|
||
mmproj = /home/coja/Documents/models/gemma-4-E4B-it-mmproj-F16.gguf
|
||
no-mmproj-offload = true
|
||
model-draft = /home/coja/Documents/models/gemma-4-E4B-it-mtp-Q8_0.gguf
|
||
spec-type = draft-mtp
|
||
spec-draft-n-max = 2
|
||
reasoning-budget = 0 # ⚠ remove if startup says the option is not recognized / not honored
|
||
ctx-size = 32768 # n_ctx_train 131072; fl runs 98304 with room — raise after a bench
|
||
n-gpu-layers = 99
|
||
threads = 6
|
||
flash-attn = on
|
||
cache-type-k = q8_0
|
||
cache-type-v = q8_0
|
||
batch-size = 2048
|
||
ubatch-size = 512
|
||
cache-reuse = 256 # inert under mmproj ("not supported by multimodal") — kept for parity
|
||
temp = 1.0 # Gemma defaults: 1.0 / 0.95 / 64
|
||
top-p = 0.95
|
||
top-k = 64
|
||
jinja = on
|
||
sleep-idle-seconds = 30
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Coding — MoE, experts in RAM
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
|
||
[Ornith-1.5-35B-A3B-IQ4_XS]
|
||
# ⭐ AGENTIC CODER — Ornith-1.5-35B-A3B (ornith-ai, MIT): Qwen3.5-MoE arch (40 layers, 256
|
||
# experts / 8 active + shared, ~3B active, MTP head embedded, vision, n_ctx_train 262144).
|
||
# Card claims it beats Qwen3.6-35B-A3B on coding/agent benches (SWE-bench Verified 79,
|
||
# Terminal-Bench 68.5) — unverified, judge it in use. Thinking ON by default (coder role);
|
||
# the -instant alias below turns it off.
|
||
# MEASURED 2026-09-28 (server, ctx 16384, q8/q8 KV, 6.8 GB free baseline):
|
||
# @moe34 no MTP : 11.2–15.2 t/s decode, 4.6 GB VRAM
|
||
# @moe34 MTP n=2 : 18.5–19.0 t/s, acceptance 0.61 (mean len 2.2), 5.7 GB VRAM ← +25–60%
|
||
# llama-bench @moe40 (all experts CPU) = 13.7 / @moe32 = 12.7 (VRAM-starved) → the wall is
|
||
# DDR4 streaming 4-bit I-quant experts; MTP is the lever, not moe.
|
||
# moe36 chosen for a busy-desktop margin (~5.2 GB predicted); moe34 = +~0.5 GB for ~+1 t/s.
|
||
# ✓ ROUTER BENCH 2026-09-28 15:20 (bench.py, 4.5 GB desktop baseline, gen 256):
|
||
# [instant] 21.2 t/s decode / 36 pp / TTFT 1.0 s / 60% MTP acceptance / 10.4 used / 1.4 free
|
||
# [thinking] 10.0 t/s / 20 pp / TTFT 21.5 s (the think block) / 47% acceptance / 10.1 / 1.8
|
||
# → thinking mode halves MTP's payoff (reasoning tokens draft worse); the instant alias is
|
||
# the agent-loop pick, thinking for hard problems only. 1.4 GB free = at the margin; if a
|
||
# load ever fails on a busy desktop, step moe 36→38 (≈ −0.4 GB, ≈ −0.5 t/s).
|
||
# 2nd sweep 15:5x (4.2 GB baseline): instant 14.0 t/s (60%) / 2.0 free · thinking 10.5 (53%) / 2.2
|
||
# free → the instant alias ranges 14–21 t/s with desktop CPU load (the experts share the 6 cores).
|
||
# ✓ QUIET DESKTOP 2026-10-12 15:36 (3.3 GB baseline, user run): instant 31.5 t/s / 63 pp / TTFT 0.6 s /
|
||
# 67% acceptance / 8.3 used / 3.5 free. Same preset — the 1 GB the desktop gave back is worth
|
||
# +10–17 t/s here.
|
||
# ✗ moe30 TESTED 2026-10-12 (same quiet desktop, 3.1 GB baseline, one-off load): 32.1 / 31.7 t/s at
|
||
# 59% acceptance, 7.35 GB process VRAM, 0.7 GB free on the card → +0.5 t/s for 2.4 GB of margin.
|
||
# GPU-resident experts barely move this model (fl saw the same on Coder-Next); the wall is the
|
||
# MTP verify + CPU-side work, not expert bandwidth. A -fast alias was added and REMOVED the same
|
||
# day on this evidence — moe36 stays; lower moe is not the lever here, free VRAM for the
|
||
# desktop is.
|
||
# ⚠ VARIANCE 2026-10-12 15:52 (user run, SAME quiet desktop: 3.4 GB baseline / 3.5 free): 15.1 t/s /
|
||
# 9 pp / TTFT 3.4 s / 59% acceptance — vs 31.5 / 63 pp sixteen minutes earlier. Prefill is pure
|
||
# compute, so a 7× prefill drop = CPU contention, not VRAM: the abbr runs the server at nice 19 and
|
||
# it yields to any other CPU load (browser/electron bursts, a build, a cold mmap re-read of the
|
||
# 19 GB file). Expect 14–31 t/s on this preset and judge it over several runs (`top` during one).
|
||
# ✓ ROUTER 2026-10-19 (sim stack up — header; ~4–4.5 GB desktop): instant 15.0 t/s / 37 pp / 60% / 9.0 used /
|
||
# 2.9 free · thinking 14.7 / 34 pp / 57% / 2.9 free (TTFT = all-think: 256 tokens of reasoning, no answer
|
||
# yet). The busy floor again, this time with a named cause: Unreal Blocks + PX4 on 3+ of the 6 cores.
|
||
# ⬇ SOURCE bartowski/Ornith-1.5-35B-A3B-GGUF (imatrix): Ornith-1.5-35B-A3B-IQ4_XS.gguf
|
||
# 19,278,554,784 B, sha256 verified 09-28. fl runs the IQ3_XXS (15.34 GB) of the same repo
|
||
# at 32 t/s with only 16 expert layers in RAM — here it would also stream fewer bytes per
|
||
# token; swap to it if speed matters more than the 4-bit quality.
|
||
# mmproj-Ornith-1.5-35B-A3B-f16.gguf (0.9 GB) is available in the repo — not fetched (add
|
||
# `mmproj = …` + `no-mmproj-offload = true` if vision is wanted).
|
||
model = /home/coja/Documents/models/Ornith-1.5-35B-A3B-IQ4_XS.gguf
|
||
spec-type = draft-mtp
|
||
spec-draft-n-max = 2
|
||
ctx-size = 16384 # KV ≈ 1.3 GB @16k q8/q8 (measured OOM at 8k with ngl 99 and NO
|
||
# moe on the 14B — with experts in RAM the KV is the only GPU cost
|
||
# besides attention). 24576 needs a bench first.
|
||
n-gpu-layers = 99
|
||
n-cpu-moe = 36 # 40 MoE layers total. ↓ toward 34 on a quiet desktop (bench free ≥ 1 GB)
|
||
threads = 6
|
||
flash-attn = on
|
||
cache-type-k = q8_0
|
||
cache-type-v = q8_0
|
||
batch-size = 2048
|
||
ubatch-size = 512
|
||
cache-reuse = 256
|
||
temp = 0.6 # model card, general use: 0.6 / 0.95 / 20 / 0
|
||
top-p = 0.95
|
||
top-k = 20
|
||
min-p = 0
|
||
jinja = on
|
||
sleep-idle-seconds = 120
|
||
|
||
[Ornith-1.5-35B-A3B-instant]
|
||
# Same GGUF with thinking hard-OFF (reasoning-budget 0) for agent loops that can't afford the
|
||
# thinking TTFT. Otherwise identical to the preset above.
|
||
model = /home/coja/Documents/models/Ornith-1.5-35B-A3B-IQ4_XS.gguf
|
||
spec-type = draft-mtp
|
||
spec-draft-n-max = 2
|
||
reasoning-budget = 0
|
||
ctx-size = 16384
|
||
n-gpu-layers = 99
|
||
n-cpu-moe = 36
|
||
threads = 6
|
||
flash-attn = on
|
||
cache-type-k = q8_0
|
||
cache-type-v = q8_0
|
||
batch-size = 2048
|
||
ubatch-size = 512
|
||
cache-reuse = 256
|
||
temp = 0.6
|
||
top-p = 0.95
|
||
top-k = 20
|
||
min-p = 0
|
||
jinja = on
|
||
sleep-idle-seconds = 120
|
||
|
||
[Ornith-1.5-35B-A3B-long]
|
||
# The long-context tier (added 2026-10-28): the same GGUF at 128k (n_ctx_train 262144) — wm's answer
|
||
# to fl's Qwen3-Coder-Next 80B, which is NOT viable here (RAM, see candidates). Thinking OFF like
|
||
# -instant and like Coder-Next: a think block would eat the context this preset exists for; set
|
||
# reasoning-budget = -1 to bring it back.
|
||
# All 40 expert layers in RAM: llama-bench measured moe40 ≈ moe32 on this model (13.7 vs 12.7 t/s), so
|
||
# the ~1.6 GB moe36→40 frees pays for the KV. Qwen3.5-MoE is hybrid — only every 4th layer holds KV —
|
||
# so 128k costs ~1.2 GB over 16k.
|
||
# llama-fit-params 2026-10-28 (moe40, q8/q8, fa on, no MTP; CUDA0 MiB model / KV / compute):
|
||
# 16k 1342/232/530 · 64k 1342/742/530 · 128k 1342/1422/569 · 256k 1342/2782/953 (+1.7 GB: no room)
|
||
# ✓ ROUTER BENCH 2026-10-28 16:13 (bench.py --ctx 32000, 4.7 GB baseline, load avg ~11 = busy desktop):
|
||
# 10.0 t/s short-prompt decode / 59% MTP / 10.0 used / 1.8 free · @35.6k prompt: prefill 249 t/s
|
||
# (142.7 s), decode 16.9 t/s at 65% acceptance. → a 100k prompt ≈ 7 min to prefill: long context
|
||
# here is for patient one-shots and growing agent sessions (cache-reuse), not dumping 100k at once.
|
||
# The 19 GB mmap pushed the desktop ~3.5 GB further into swap (9.5 → 13 GiB) — RAM is the limit.
|
||
model = /home/coja/Documents/models/Ornith-1.5-35B-A3B-IQ4_XS.gguf
|
||
spec-type = draft-mtp
|
||
spec-draft-n-max = 2
|
||
reasoning-budget = 0
|
||
ctx-size = 131072
|
||
n-gpu-layers = 99
|
||
n-cpu-moe = 40 # = all 40 MoE layers (see header)
|
||
threads = 6
|
||
flash-attn = on
|
||
cache-type-k = q8_0
|
||
cache-type-v = q8_0
|
||
batch-size = 2048
|
||
ubatch-size = 512
|
||
cache-reuse = 256
|
||
temp = 0.6
|
||
top-p = 0.95
|
||
top-k = 20
|
||
min-p = 0
|
||
jinja = on
|
||
sleep-idle-seconds = 120
|
||
|
||
# ─── RETIRED 2026-10-12: [Qwen3-Coder-30B-Instruct-UD-IQ2_XXS] ──────────────────────────────
|
||
# Qwen3-Coder-30B-A3B, UD-IQ2_XXS (10.33 GB, 2-bit). Router bench 09-28 @moe34/16k: 14.0 t/s, 1.9 GB
|
||
# free (moe28 left 0.6 GB). Dominated by Ornith-instant (same speed, 4-bit, MTP) → removed from the
|
||
# three wm client overlays and DELETED from ~/Documents/models/ 2026-10-12. The K-quant upgrade
|
||
# (UD-Q3_K_XL 13.8 GB) stays in the candidates list if a second coder is ever wanted.
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Reasoning + tools — gpt-oss
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
|
||
[gpt-oss-20b]
|
||
# OpenAI gpt-oss-20b — MoE 21B / 3.6B active, 24 layers, harmony template (jinja required).
|
||
# File is the Q4_K_M requant (11.62 GB); the experts are MXFP4 in every quant, so this is
|
||
# effectively the native file (fl uses ggml-org's mxfp4.gguf, 12.1 GB — same thing).
|
||
# llama-bench 2026-09-28 (~9 GB free): @moe12 = 648 pp / 29.8 tg · @moe24 (all CPU) = 455 / 18.6.
|
||
# The fitter picked ~10 expert layers on CPU @16k with 7.5 free; moe14 = margin for a busy
|
||
# desktop. Sliding-window attention → KV is cheap, so ctx is generous.
|
||
# KV types left at default on purpose (attention sinks + quantized KV = known trouble, fl note).
|
||
# ⬇ On disk since before this config (user download). TTFT of several seconds = the model
|
||
# THINKING (harmony reasoning streams first) — use the -low alias for snappy answers.
|
||
# ✓ ROUTER 2026-10-19 (sim stack up): 11.7 t/s / 24 pp / 8.8 used / 3.0 free (TTFT all-think) · -low 9.8 /
|
||
# 23 pp / TTFT 3.1 s / 2.6 free — the lowest yet (21 quiet 10-12, 17.4 busy 09-28): all 18 expert layers
|
||
# stream from RAM on a CPU the sim already half-owns. Not a preset problem; see the header.
|
||
model = /home/coja/Documents/models/gpt-oss-20b-Q4_K_M.gguf
|
||
ctx-size = 32768 # ⚗ unbenched at 32k — fl measured 32k→64k = +1.2 GB on this arch;
|
||
# here 16k is the safe fallback if bench.py shows < 1 GB free
|
||
n-gpu-layers = 99
|
||
n-cpu-moe = 18 # 24 MoE layers. ✗ moe14 @32k = 0.3 GB free (router sweep 09-28) → 18 (+4 layers
|
||
# ≈ +1.8 GB). ✓ moe18 @32k: 17.4 t/s / 30 pp / TTFT 8.8 s (= thinking) /
|
||
# 9.7 used / 2.1 free (router 09-28). moe12 = 29.8 needs ~9 GB free
|
||
threads = 6
|
||
flash-attn = on
|
||
batch-size = 2048
|
||
ubatch-size = 512
|
||
cache-reuse = 256
|
||
temp = 1.0
|
||
top-p = 1.0
|
||
jinja = on
|
||
sleep-idle-seconds = 120
|
||
|
||
[gpt-oss-20b-low]
|
||
# Same GGUF, reasoning effort LOW — zero disk cost. Cuts the thinking preamble for interactive
|
||
# use; keep [gpt-oss-20b] (medium) for hard problems.
|
||
model = /home/coja/Documents/models/gpt-oss-20b-Q4_K_M.gguf
|
||
chat-template-kwargs = {"reasoning_effort":"low"}
|
||
ctx-size = 32768
|
||
n-gpu-layers = 99
|
||
n-cpu-moe = 18 # = the main preset. ✓ 17.5 t/s / TTFT 1.8 s / 2.1 free (router 09-28)
|
||
threads = 6
|
||
flash-attn = on
|
||
batch-size = 2048
|
||
ubatch-size = 512
|
||
cache-reuse = 256
|
||
temp = 1.0
|
||
top-p = 1.0
|
||
jinja = on
|
||
sleep-idle-seconds = 60
|
||
|
||
# ─── RETIRED 2026-09-28: [Qwen3-14B-UD-Q3_K_XL] ──────────────────────────────────────────
|
||
# Dense 14B, UD-Q3_K_XL (7.59 GB, 40 layers): does not fit beside the desktop. ngl 34 @16k was
|
||
# the fitter's split at 7.5 GB free and gave 7.6 t/s (llama-bench); through the router at a
|
||
# 4.5 GB desktop baseline it FAILED TO LOAD (context creation, 15:2x sweep). A fitting split
|
||
# (~ngl 28 @8k) lands ≈ 5 t/s — Qwen3.5-9B covers dense at 38 t/s. Not wired to clients.
|
||
# DELETED from ~/Documents/models/ 2026-10-12.
|
||
|
||
# ─── RETIRED 2026-09-28 (never wired) — more dense files that cannot fit this card ───────
|
||
# All measured with llama-bench at the fitter's ngl for 16k ctx (p128/n64), 6.8 GB free:
|
||
# Qwen3-8B-Q8_0.gguf (8.71 GB) ngl 19 → 3.4 t/s
|
||
# → a Q4_K_M (~5 GB, unsloth/Qwen3-8B-GGUF) WOULD fit fully and run ~40 t/s; but
|
||
# Qwen3.5-9B already covers that slot. Delete or re-quant.
|
||
# Ministral-3-14B-Instruct-2512-Q4_K_M.gguf (8.24 GB) ngl 21 → 3.8 t/s
|
||
# Devstral-Small-2-24B-Instruct-2512-UD-IQ3_XXS.gguf (9.41) ngl 28 → 1.8 t/s
|
||
# Lexi-Llama-3-8B-Uncensored_F16.gguf (16.07 GB, F16!) ngl 9 → 1.0 t/s
|
||
# → an 8B at F16 is 2× the bytes of Q8 for no quality; a Q4_K_M of it would fit fully.
|
||
# All four DELETED from ~/Documents/models/ 2026-10-12 (66 GB freed together with the 14B, the plain 9B
|
||
# and the 2-bit coder).
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Coding — fits ENTIRELY on the GPU (added 2026-10-12 after passing its gate bench)
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
|
||
[Mellum2.1-12B-A2.5B-Thinking-IQ4_XS]
|
||
# ⚗ JetBrains Mellum 2.1 — code MoE, 12B total / 2.5B active, 28 layers × 64 experts, THINKING on.
|
||
# IQ4_XS 6.97 GB: fits entirely on a quiet desktop (fitter @16k: no offload at 8.3 GB free);
|
||
# n-cpu-moe 4 keeps ~0.85 GB in RAM as busy-desktop insurance (each layer's experts ≈ 0.2 GB) —
|
||
# drop to 0 on a quiet desktop (see probe).
|
||
# ✓ ROUTER BENCH 2026-10-12 @moe4: 68.7 t/s decode / 9.5 used / 2.3 free. The bench's "15 pp" is the
|
||
# short-prompt artifact — PROBE with a 1,713-token prompt: moe4 = 1,160 pp / 62.8–63.7 t/s / 6.2 GB
|
||
# process VRAM; moe0 (all on GPU) = 1,549 pp / 86.7–87.4 t/s / 6.9 GB. KEPT at moe4 (loads beside a
|
||
# 4.5 GB desktop: ~1.1 GB free); moe0 is +24 t/s when the desktop holds ≤ ~3.9 GB. Role: the fast
|
||
# REASONING coder — 63–87 t/s with a think block, prefill > 1k t/s. Wired into aichat/opencode/pi.
|
||
# Note n_ctx_train is 131072 via RoPE scaling over an 8192 base — keep ctx modest until tested.
|
||
# ✓ ROUTER 2026-10-19 (sim stack up): 41.9 t/s / TTFT 5.2 s (= the think block) / 10.2 used / 1.6 free — −40% vs
|
||
# quiet: even 4 CPU-expert layers hurt once the cores are taken. moe0 would dodge it, on a quiet desktop only.
|
||
# ⬇ SOURCE bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF: Mellum2.1-12B-A2.5B-Thinking-IQ4_XS.gguf (sha verified).
|
||
model = /home/coja/Documents/models/Mellum2.1-12B-A2.5B-Thinking-IQ4_XS.gguf
|
||
ctx-size = 16384
|
||
n-gpu-layers = 99
|
||
n-cpu-moe = 4 # 28 MoE layers; 0 = all on GPU (quiet desktop only)
|
||
threads = 6
|
||
flash-attn = on
|
||
cache-type-k = q8_0
|
||
cache-type-v = q8_0
|
||
batch-size = 2048
|
||
ubatch-size = 512
|
||
cache-reuse = 256
|
||
temp = 0.6 # thinking-model defaults (0.6 / 0.95 / 20)
|
||
top-p = 0.95
|
||
top-k = 20
|
||
min-p = 0
|
||
jinja = on
|
||
sleep-idle-seconds = 60
|
||
|
||
# ─── RETIRED 2026-10-19 (user's call): [AesCode-8B-Q4_K_M] ─────────────────────────────────────
|
||
# Microsoft AesCode-8B (qwen3vl, 36 layers, vision), bartowski Q4_K_M 5.21 GB + mmproj f16 1.16 GB.
|
||
# Router bench 2026-10-12: 36.4 t/s / 386 pp / TTFT 0.07 s / 9.4 used / 2.4 free, fully on the GPU at
|
||
# ctx 16384 (mmproj on CPU). Was the quick-edit / screenshot-aware coder; removed from aichat/opencode/pi.
|
||
|
||
# ─── RETIRED 2026-10-12 (same day as its gate bench): [FrogNano-4B-2609-Q4_K_M] ──────────────────
|
||
# Microsoft FrogNano-4B-2609 (qwen35, MTP head, vision), Q4_K_M 2.80 GB + mmproj 0.67. Router bench:
|
||
# 87.9 t/s with the draft (84% acceptance) / 244 pp / 7.9 used / 3.9 free — good, but no slot: on the
|
||
# same quiet desktop gemma-4-E4B does 101 t/s WITH vision and LFM2.5-8B-A1B 119 without. Both files
|
||
# deleted from ~/Documents/models/ 2026-10-12.
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Candidates — ⬇ NOT ON DISK. Sized for this box (≤ ~20 GB file, 2–4B active). Add a section
|
||
# when downloaded; measured expectations are extrapolations from the presets above.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# (2026-09-28: user dropped Qwen3.6-35B-A3B, GLM-4.7-Flash, LFM2-24B-A2B, the coder upgrades and the
|
||
# embedder from the shortlist; LFM2.5-8B-A1B and gemma-4-E4B were downloaded and have presets above.)
|
||
# unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-IQ3_XXS 14.1 GB — newest general Qwen MoE + MTP; est. ~25 t/s here.
|
||
# unsloth/Qwen3-Coder-30B-A3B-Instruct UD-Q3_K_XL 13.8 GB — K-quant coder (est. ~15–18 t/s); the 2-bit file was deleted 10-12.
|
||
# unsloth/GLM-4.7-Flash-GGUF UD-Q3_K_XL 13.8 GB — K-quant coder; deepseek2/MLA → KV k/v types must match.
|
||
# LiquidAI/LFM2-24B-A2B-GGUF Q4_K_M 14.4 GB — 2B active; est. ~20–25 t/s, unproven arch.
|
||
# unsloth/gemma-4-26B-A4B-it-qat-GGUF UD-Q4_K_XL 14.2 GB — 4B active in RAM = slower than the 3B-active Qwens.
|
||
# NOT viable (RAM): Qwen3-Coder-Next 80B (≥ 28 GB), Qwen3.8-Flash-Next, GLM-5.3-Flash, Kimi-Linear-48B
|
||
# (≥ 23 GB), Qwen3.6/3.8-27B and Muse-Glimmer-30B (dense — see the tombstones above).
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Notes
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# n-cpu-moe N : experts of the first N MoE layers stay in system RAM. Higher = less VRAM
|
||
# (loads on a busier desktop) but slower; each layer on the GPU ≈ 0.4 GB for a
|
||
# 35B-A3B at 4-bit. Dense models ignore it — use n-gpu-layers / ctx instead.
|
||
# Sizing rule : file GB × (1 − N/layers) + KV + ~0.7 GB compute ≤ VRAM free at load time.
|
||
# Fitter : `llama-fit-params -m <gguf> -c <ctx> -fa on` prints the split that fits RIGHT
|
||
# NOW (it reads current free VRAM) — a good first value after a download.
|
||
# After ANY change: restart the router (presets are read once) and run ./bench.py — keep
|
||
# every "VRAM free" ≥ ~1 GB; a load failure here is a clean error, not a freeze.
|