coja/dots
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[Sync] adopt unified stow layout from the private repo

Browse Source 
Mirrors the private dots tree at 900bdda: one shared base plus per-host
overlays, replacing the old flat .config/ layout (last synced 2026-06-28).

- packages: common/ gui/ wm/ lw/ fl/ + install.sh and bin/ tooling (dotsync,
  reconcile-hyde.sh)
- new README (layout, deploy order, HyDE dependency), plus ToDo.md and
  HYDE-UPDATE.md
- current HyDE waybar rig (layouts/, cava), pi agent extensions, claude/
  config, tmux, presenterm, aichat roles
- drops stale duplicates and generated cruft that should never have been
  tracked: the second top-level .pi/ copy, btop.log, zellij config.kdl.bak,
  fish_variables, nvim codecompanion.lua
- .pi/agent/auth.json is gitignored now; auth.json.example ships instead
- fl/ and wm/ hypr themes/ stay untracked (HyDE-generated per machine, per
  the root .gitignore)
This commit is contained in:
coja
2026-08-12 22:50:18 +02:00
parent f62cb40499
commit a700e23e0a
318 changed files with 12564 additions and 10772 deletions
Download Patch File Download Diff File Expand all files Collapse all files
fl/.config/llamacpp/README.md
+165
View File
@@ -0,0 +1,165 @@
# Local LLM server — setup & usage
llama.cpp in **router mode** serving ~10 model presets from one 16 GB GPU.
Last full tune: **2026-07-16** (decode numbers measured then via `bench.py`, unless a preset comment in `config.ini` says otherwise).
Last full clean sweep: **2026-08-08 03:22** — all 11 presets loaded and ran green (baseline 2.0 GB) after the week's coder-race/KV-fix/gemma-sidecar round.
- **Box**: AMD RX 7600 XT 16 GB (ROCm, ~288 GB/s) + Ryzen 5600X (6 cores) + 48 GB RAM.
⚠ The GPU **also drives the display** — see [VRAM safety](#vram-safety-the-golden-rules).
- **Endpoints**: LAN `http://192.168.0.204:11343/v1` · remote `https://llm.duskadiy.com/api/v1`
(own reverse proxy to the same server, no key).
- **Files here**: `config.ini` (model presets — the section names ARE the API model ids),
`bench.py` (safety-first benchmark), `bench-results.md` / `bench-history.md` (ledgers, generated).
## How it runs
systemd unit **`llama.service`** (on the server, user `anon`) runs a script wrapping:
```
llama-server --port 11343 --host 0.0.0.0 --models-max 1 --models-preset ~/.config/llamacpp/config.ini
```
- Router mode: each requested model loads on demand in a child process.
- **`--models-max 1` is deliberate and load-bearing**: at most one model resident; requesting
another LRU-evicts the current one *first*. Anything ≥2 lets two ~13 GB models stack →
VRAM overcommit → GTT spill → **whole-PC freeze**. Do not raise it casually.
(`sleep-idle-seconds` per preset is the second line of defense.)
- Models live in `~/software/models/` (paths in `config.ini` must stay absolute — llama-server
does not expand `~` or env vars in preset values). mmproj/draft files must match the exact
filename in the preset (watch for trailing spaces when renaming downloads!).
**Deploying config changes**: this dir is stowed into `~/.config/llamacpp/` as per-file
symlinks, so on the server `dotsync` (pull; re-stows only if files were added/removed) then
`sudo systemctl restart llama.service` — the router reads presets once at startup.
After ANY preset change: restart + `./bench.py -m <changed-ids>` and check the guards.
## VRAM safety (the golden rules)
ROCm does not OOM cleanly — an overcommitted model spills into GTT (system RAM),
starves the desktop and freezes the PC (reboot). Hard-won rules:
1. Keep every preset's benched **VRAM free ≥ 2.5 GB** (idle desktop uses ~1.3 of 17.2).
2. File size > VRAM is **fine** for MoE — `n-cpu-moe N` keeps the first N layers' experts
in system RAM. It is the main tuning knob: higher = safer/slower, lower = faster.
3. Bench with a **clean baseline** (close the browser): a dirty baseline both skews the
margins and costs ~3–4 t/s of decode (GPU compute contention — measured 2026-07-08).
4. Bench after every change; config-edit without restart = numbers from the *old* config.
## Model roster (reference numbers = clean night sweeps, 2026-07-16)
| model id | decode t/s | ctx | extras | role |
|---|---|---|---|---|
| `gemma-4-E4B-it-UD-Q8_K_XL` | **57.3** | **96k** | vision, MTP, prefill 185 t/s | fast small generalist, long docs, images (ctx 96k ✓ 08-08, n_ctx_train 131072) |
| `Qwen3.6-35B-A3B-Thinking` | 38.8 | 24k | MTP, vision, reasoning | hard problems, slow-but-smart answers |
| `gpt-oss-20b` | 38.4 | **64k** | reasoning, tools | fast reasoning + tool use · fast long-context (37 t/s vs Coder-Next 16; 131k ruled out) |
| `gpt-oss-20b-low` | 37.2 | **64k** | reasoning LOW, TTFT 0.8 s | same model, snappy answers (no long preamble) |
| `Qwen3.6-35B-A3B-MTP-UD-IQ3_XXS` | 35.9 | 24k | MTP, vision, thinking OFF | **daily driver** — compact instant answers |
| `gemma-4-26B-A4B-it-UD-IQ4_XS` | 33.9 | 24k | vision, MTP | quality generalist + best vision |
| `Qwen3.5-9B-UD-Q6_K_XL` | 32.1 | 32k | vision, MTP, reasoning | small Qwen, quick tasks |
| `Qwen3-Coder-30B-Instruct-UD-Q3_K_XL` | **~30** clean / 27.0 evening @moe12 | 32k | — | **main agent coder** — won the 2026-08-06 quant race; moe12 verified evening 08-07 + clean 08-08 |
| `GLM-4.7-Flash-UD-Q4_K_XL` | **21.5** @moe22 | 24k | reasoning | quality coder (opencode subagents); KV fix + moe22 ✓ verified clean 08-08 |
| `Qwen3-Coder-Next-UD-IQ3_XXS` | 16.0 | **128k** | 80B-A3B | long-session coder (128k ctx) |
| `Qwen3-Embedding-0.6B` | 33 (CPU) | 8k | CPU-only, `/v1/embeddings` | RAG/search embedder (not a chat model) |
Expected run-to-run spread: MTP models swing ±15% with draft **acceptance rate** (content-
dependent); CPU-heavy presets (Coder-Next, embedder) dip under daytime CPU contention.
Treat clean night runs as the reference; don't retune on daytime deltas.
## Which model, when
- **Agent coding loops** (edit/test cycles): `Qwen3-Coder-30B` — best speed/quality balance at 32k.
- **Hard code, reviews, tricky bugs**: `GLM-4.7-Flash` — strongest 30B-class coder, slightly slower.
- **Marathon sessions / huge conversation history**: `Qwen3-Coder-Next` — 128k ctx at only
6 GB VRAM (hybrid attention). Caveat: ~33 t/s prefill means it's for *growing* sessions
(`cache-reuse` makes turns incremental), **not** for cold-dumping 100k tokens.
- **Everyday questions**: `Qwen3.6-35B-A3B` — no reasoning preamble, 36 t/s.
- **Hard reasoning**: `35B-Thinking` (quality) or `gpt-oss-20b` (speed + tool use).
Their multi-second TTFT is the reasoning phase streaming first — not a slow load.
For quick interactive gpt-oss answers use `gpt-oss-20b-low` (reasoning effort low).
- **Images**: `gemma-4-26B` for quality, `gemma-4-E4B` or `Qwen3.5-9B` for speed,
`Qwen3.6-35B` when you want the daily driver to see the screenshot.
- **Long one-shot documents**: `gemma-4-E4B` — 185 t/s prefill eats 64k in ~6 min
(Coder-Next would take ~30 min to prefill the same).
- **RAG embeddings**: point Open WebUI etc. at `Qwen3-Embedding-0.6B`. Runs on CPU by
design — 0 VRAM, can never contribute to an overcommit.
## Speech-to-text (separate service)
STT deliberately does **not** live in this router — it is a CPU-only whisper.cpp service on port
11345, documented in `../whisper/README.md`. Two reasons: `models-max 1` would make every
transcription LRU-evict the resident chat model, and llama.cpp decodes audio via miniaudio
(wav/mp3/flac only) while browsers record opus.
The router *does* answer `POST /v1/audio/transcriptions` — it rewrites the request into a chat
completion — so `gemma-4-E4B-it-UD-Q8_K_XL` (audio-capable via its mmproj) transcribes uploaded
wav/mp3/flac with no config change. Keep that as the fallback; mind the eviction.
## Text-to-speech (separate service)
TTS is **not servable from this router at all** — unlike STT, there is no fallback. `llama-server`
has `--model-vocoder` / `--tts-use-guide-tokens`, but exposes **no `/v1/audio/speech` route**:
the router 404s on it exactly like an undefined path, and `grep -rn "audio/speech"` over the
b10216 tree returns nothing — the only audio route registered is `/v1/audio/transcriptions` (STT).
Worse, `--model-vocoder` is accepted by the server and then **silently ignored**: it is registered
in the arg table and documented, but no server code path consumes it. `--help` listing it is not
evidence it works. `llama-tts` is a CLI. (All verified 2026-08-09.)
So TTS is a CPU-only Kokoro-82M service on port **11347**, documented in `../tts/README.md`
(repo-side only — not installed on fl yet). The `models-max 1` argument applies even harder there
than for STT: TTS fires on *every* response, so a resident preset would evict the chat model every
single turn.
## Client wiring
The same catalog is mirrored in every client — when adding/removing a preset, update all:
- `common/.config/opencode/opencode.json` (both providers + agent model overrides)
- `common/.pi/agent/models.json` (both providers; `contextWindow` = server `ctx-size`)
- `common/.config/aichat/config.yaml` (both clients)
Rule: client model **id = config.ini section name**, client context ≤ server `ctx-size`.
The embedder is deliberately absent from chat clients.
2026-08-06: the coder id changed `…-IQ4_XS` → `…-UD-Q3_K_XL` in all six client files
(common + lw overlays) after the quant race; OWUI picks the new id up automatically from
the router, but chats/presets saved against the old id need re-picking.
## Tuning cheat-sheet
- **`n-cpu-moe`** (MoE only): experts→CPU. The speed/VRAM dial. Measured curve is gentle —
tune in steps of 2–4 layers, re-bench, keep free ≥2.5 GB.
- **`ctx-size`**: KV cost varies wildly by arch — Coder-Next 64k→128k cost 0.5 GB (hybrid
attn); dense Qwens pay ~1 GB per 16k. Raise only after a bench shows the headroom.
- **Quants**: dense = bandwidth-bound, quant size sets speed directly (9B: Q8→Q6 = +87%
with MTP). MoE = buy quality with a bigger quant, pay in CPU offload (both coders run
4-bit now; 3-bit was only ~2 t/s faster). Coders want ≥4-bit; chat tolerates UD 3-bit.
- **MTP / spec decode** (`spec-type draft-mtp`): ~1.5–2× decode. Qwen3.5/3.6 embed the
head in the *MTP-repo* GGUF (same filename as plain repo — size is the tell);
gemma-4 uses a separate `mtp-*.gguf` draft file. **No MTP possible yet for**:
GLM-4.7-Flash (conversion drops the head — expected ~1.5× when llama.cpp lands it),
Qwen3-Coder-Next (Qwen3-Next head exists upstream, no GGUF ships it), gpt-oss and
Qwen3-Coder-30B (no head exists). Recheck releases occasionally.
- **KV cache**: `q8_0` K / `q4_0` V everywhere except gpt-oss (attention-sink issues with
quantized KV — keep defaults there).
## Benchmarking
```
./bench.py # full sweep (add -x Qwen3-Embedding-0.6B — it can't chat)
./bench.py -m id1,id2 # just the changed presets
./bench.py -n 512 --ctx 16000 # longer gen + long-context decode column
```
⚠ **Benches need the server to themselves**: with `models-max 1`, a request for another
model arriving mid-load LRU **force-kills the loading instance** → fake "failed to load"
rows (and a dirty baseline). Close chat clients / OWUI before sweeping.
Safety-first: force-unloads resident models via the router API (`POST /models/unload`),
verifies a clean card with rocm-smi before every load, records a ⚠ row and skips
generation when a model lands too tight (free < 1.5 GB or GTT ballooning).
Results: `bench-results.md` (latest) + `bench-history.md` (append-only; diff runs there).
## Watch list
- **GLM MTP in llama.cpp** — the single biggest pending win (primary coder ~22 → ~30+).
- **Coder-Next 256k**: if the load log's `n_ctx_train` says 262144, ctx can likely double
again for ~1 GB (KV measured nearly flat).
fl/.config/llamacpp/bench.py
Executable
+320
View File
@@ -0,0 +1,320 @@
#!/usr/bin/env python3
"""
llama.cpp router benchmark → writes a results ledger for later tuning.
SAFETY-FIRST: this GPU also drives the display — VRAM overcommit spills to GTT and can
freeze the whole PC. The script therefore:
- records the BASELINE VRAM before touching anything (leftover model / browser skews everything)
- ensures a CLEAN card before EVERY load (incl. the first): resident models are force-unloaded
via the router API (POST /models/unload), then VRAM is verified with rocm-smi — the poll
covers driver reclaim lag, older builds without the endpoint (sleep-idle fallback), and
desktop apps the router doesn't own. Aborts if the card never drains.
- checks free VRAM right after a model mounts; too tight (--min-free-gb) or GTT ballooning
(--max-gtt-gb, measured against a PER-MODEL baseline — GTT reclaims slowly) → records a
⚠ row and skips generating; the sweep continues after the drain-wait
- refuses to sweep multiple models blind (no rocm-smi) unless --no-vram-ok
Per model it records decode/prefill t/s, TTFT, VRAM used/free, and the preset's knobs
(ctx, n-cpu-moe, spec-type) parsed from config.ini.
Output: bench-results.md (latest run, overwritten) + bench-history.md (every run, appended)
— both next to this script. Run ON the GPU box. Stdlib only.
./bench.py # all models the server lists
./bench.py -m id1,id2 # only these
./bench.py -x Qwen3-Coder-Next-UD-IQ3_XXS # skip some (e.g. not downloaded yet)
./bench.py -n 512 --ctx 16000 # longer gen + a long-context decode column
Compare a change: edit config.ini → reload server → re-run → diff runs in bench-history.md.
"""
import argparse, json, os, re, subprocess, sys, time, urllib.request, urllib.error
from datetime import datetime
HERE = os.path.dirname(os.path.abspath(__file__))
_smi_warned = False
def rocm_mem(kind="vram"):
"""(used_bytes, total_bytes) from rocm-smi for 'vram' or 'gtt'; (None, None) if unavailable."""
global _smi_warned
try:
out = subprocess.run(["rocm-smi", "--showmeminfo", kind],
capture_output=True, text=True, timeout=15).stdout
except Exception:
out = ""
def grab(pat):
m = re.search(pat + r"\s*:?\s*(\d+)", out)
return int(m.group(1)) if m else None
label = "VRAM" if kind == "vram" else "GTT"
used = grab(label + r" Total Used Memory \(B\)") or grab(r"Used Memory \(B\)")
total = grab(label + r" Total Memory \(B\)")
if kind == "vram" and total is None and not _smi_warned:
_smi_warned = True
print(" (rocm-smi not readable here — VRAM columns and freeze guards are OFF)", file=sys.stderr)
return used, total
def with_retries(fn, retries, wait, label=""):
"""Retry on 5xx/connection errors (model still mounting). A deterministic 'failed to
load' from the router is NOT retried — that's a broken preset, not a slow mount."""
last = None
for i in range(retries + 1):
try:
return fn()
except urllib.error.HTTPError as e:
if e.code < 500: # 4xx = real client error
raise
body = ""
try:
body = e.read().decode("utf-8", "ignore")
except Exception:
pass
if "failed to load" in body: # deterministic → don't burn retries
raise RuntimeError("model failed to load — check `journalctl -u llama.service`") from None
last = e
except (urllib.error.URLError, ConnectionError, TimeoutError) as e:
last = e
if i < retries:
print(f" … {label} not ready ({last}); waiting {wait}s (try {i + 1}/{retries})", file=sys.stderr)
time.sleep(wait)
raise last
def _open(url, key, payload=None, timeout=600):
data = json.dumps(payload).encode() if payload is not None else None
hdrs = {"Authorization": f"Bearer {key}"}
if data:
hdrs["Content-Type"] = "application/json"
return urllib.request.urlopen(urllib.request.Request(url, data=data, headers=hdrs), timeout=timeout)
def list_models(base, key):
d = json.load(_open(base.rstrip("/") + "/models", key))
return [m["id"] for m in d.get("data", [])]
def force_unload(base, key):
"""Ask the router to unload every resident model (POST /models/unload on the ROOT api,
not /v1). Harmless no-op on builds without the endpoint — callers fall back to polling."""
root = re.sub(r"/v1/?$", "", base.rstrip("/"))
try:
d = json.load(_open(root + "/models", key, timeout=30))
except Exception as e:
print(f" (router /models not readable: {e} — relying on sleep-idle drain)", file=sys.stderr)
return
for m in d.get("data", []):
if m.get("status", "loaded") != "loaded": # builds without 'status': try them all
continue
try:
_open(root + "/models/unload", key, {"model": m["id"]}, timeout=30).read()
except Exception:
pass # not loaded / endpoint missing — poll covers it
def parse_config(path):
"""{preset: {ctx, ncpumoe, spec}} from the router config.ini."""
sections, sec = {}, None
try:
lines = open(path, encoding="utf-8").read().splitlines()
except OSError:
return {}
for ln in lines:
s = ln.strip()
if s.startswith("[") and s.endswith("]"):
sec = s[1:-1]; sections[sec] = {}
elif sec and "=" in s and not s.startswith("#"):
k, v = s.split("=", 1)
sections[sec][k.strip()] = v.split("#", 1)[0].strip()
return {sec: {"ctx": kv.get("ctx-size", "-"),
"ncpumoe": kv.get("n-cpu-moe", "-"),
"spec": kv.get("spec-type", "-") or "-"}
for sec, kv in sections.items()}
def run(base, key, model, prompt, n, temp, timeout):
payload = {"model": model, "messages": [{"role": "user", "content": prompt}],
"max_tokens": n, "temperature": temp, "stream": True,
"stream_options": {"include_usage": True},
# llama.cpp extension: makes the server embed its own timings in the stream →
# the table shows the SAME predicted_per_second as the llama.cpp web UI
# (without it we fall back to wall-clock estimates, which read lower).
"timings_per_token": True}
t0 = time.perf_counter(); ttft = None; ntok = 0; usage = None; timings = None
for raw in _open(base.rstrip("/") + "/chat/completions", key, payload, timeout):
line = raw.decode("utf-8", "ignore").strip()
if not line.startswith("data:"):
continue
body = line[5:].strip()
if body == "[DONE]":
break
try:
c = json.loads(body)
except ValueError:
continue
ch = c.get("choices") or [{}]
if ch and ch[0].get("delta", {}).get("content"):
if ttft is None:
ttft = time.perf_counter() - t0
ntok += 1
if c.get("usage"):
usage = c["usage"]
if c.get("timings"):
timings = c["timings"]
total = time.perf_counter() - t0
comp = (usage or {}).get("completion_tokens") or ntok
ptok = (usage or {}).get("prompt_tokens")
gen_s = total - (ttft or total)
tg = comp / gen_s if gen_s > 0 else 0.0
pp = (ptok / ttft) if (ptok and ttft) else None
if timings: # llama.cpp's own numbers are authoritative when the router forwards them
tg = timings.get("predicted_per_second", tg)
pp = timings.get("prompt_per_second", pp)
ptok = timings.get("prompt_n", ptok)
return {"tg": tg, "pp": pp, "ttft": ttft or 0.0, "ptok": ptok or 0}
def wait_drain(drain_gb, window_s, label="", base=None, key=None):
"""Force-unload resident models via the router API, then poll until VRAM used drops
below drain_gb (card is clean) or the window expires."""
used, _ = rocm_mem()
if base and used is not None and used > drain_gb * 1e9:
force_unload(base, key)
end = time.time() + window_s
announced = False
while used is not None and used > drain_gb * 1e9 and time.time() < end:
if not announced:
print(f" … waiting for VRAM to drain before {label} ({gb(used)} used)", file=sys.stderr)
announced = True
time.sleep(5)
used, _ = rocm_mem()
return used
def gb(b):
return f"{b / 1e9:.1f} GB" if b else "-"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("-u", "--url", default="http://192.168.0.204:11343/v1")
ap.add_argument("-k", "--key", default="no-key-required")
ap.add_argument("-m", "--models", default="all", help="comma-separated ids, or 'all'")
ap.add_argument("-x", "--skip", default="", help="comma-separated ids to skip")
ap.add_argument("-n", "--tokens", type=int, default=256)
ap.add_argument("-t", "--temp", type=float, default=0.3)
ap.add_argument("--ctx", type=int, default=0, help="also record decode at ~this many prompt tokens")
ap.add_argument("--timeout", type=int, default=600, help="per-request stall timeout (s)")
ap.add_argument("--config", default=os.path.join(HERE, "config.ini"))
ap.add_argument("--out", default=os.path.join(HERE, "bench-results.md"))
ap.add_argument("--history", default=os.path.join(HERE, "bench-history.md"))
ap.add_argument("--retries", type=int, default=4, help="retries while a model is still mounting")
ap.add_argument("--retry-wait", type=int, default=15, help="seconds between retries")
ap.add_argument("--drain-gb", type=float, default=4.0,
help="VRAM considered 'clean' below this (raise it if your desktop alone uses more)")
ap.add_argument("--min-free-gb", type=float, default=1.5,
help="SAFETY: stop the sweep if a loaded model leaves less VRAM free than this")
ap.add_argument("--max-gtt-gb", type=float, default=2.0,
help="SAFETY: flag a model if GTT grows more than this over baseline (= VRAM spilling to RAM)")
ap.add_argument("--unload-wait", type=int, default=360,
help="after a too-tight model, wait up to this long for it to unload before continuing "
"(sleep-idle-seconds keeps models resident up to 300s)")
ap.add_argument("--no-vram-ok", action="store_true",
help="allow a multi-model sweep even though rocm-smi (and thus the freeze guards) is unavailable")
a = ap.parse_args()
models = list_models(a.url, a.key) if a.models == "all" else [m.strip() for m in a.models.split(",")]
skip = {s.strip() for s in a.skip.split(",") if s.strip()}
models = [m for m in models if m not in skip]
cfg = parse_config(a.config)
used0, total_vram = rocm_mem("vram")
if total_vram is None and len(models) > 1 and not a.no_vram_ok:
sys.exit("ABORT: rocm-smi is not readable here, so the VRAM/GTT freeze guards can't work.\n"
"Run this ON the GPU box, bench a single model with -m <id>, or pass --no-vram-ok to accept the risk.")
if used0 is not None and used0 > a.drain_gb * 1e9:
print(f"⚠ baseline VRAM used is already {gb(used0)} — a leftover model (sleep-idle keeps them "
f"up to 5 min) or desktop apps (browser!). Results/safety margins will be skewed; "
f"ideally close GPU apps or wait, then re-run.", file=sys.stderr)
task = "Write a Python function that merges two sorted lists, with a short docstring and one example."
filler = ("The quick brown fox jumps over the lazy dog. " * max(1, a.ctx // 9)) if a.ctx else None
cols = ["model", "cfg ctx", "n-cpu-moe", "spec", "decode t/s", "prefill t/s", "TTFT s", "VRAM used", "VRAM free"]
if a.ctx:
cols.append(f"decode@{a.ctx // 1000}k")
latest = open(a.out, "w", encoding="utf-8")
hist = open(a.history, "a", encoding="utf-8")
meta = (f"_Run {datetime.now():%Y-%m-%d %H:%M} · server `{a.url}` · VRAM total {gb(total_vram)} · "
f"baseline used {gb(used0)} · gen {a.tokens} tok · guards: free≥{a.min_free_gb} GB, GTT+≤{a.max_gtt_gb} GB_")
def emit(line, both=True):
print(line)
latest.write(line + "\n"); latest.flush()
if both:
hist.write(line + "\n"); hist.flush()
latest.write("# llama.cpp benchmark results (latest run)\n\n")
hist.write(f"\n## run {datetime.now():%Y-%m-%d %H:%M}\n\n")
emit(meta + "\n")
emit("| " + " | ".join(cols) + " |")
emit("|" + "|".join(["---"] * len(cols)) + "|")
for m in models:
c = cfg.get(m, {})
base_cells = [m, c.get("ctx", "-"), c.get("ncpumoe", "-"), c.get("spec", "-")]
# A CLEAN card before EVERY load (incl. the first): force-unload via the router API,
# then verify with rocm-smi — double residency = bogus numbers or a freeze.
u = wait_drain(a.drain_gb, a.unload_wait, m, a.url, a.key)
if u is not None and u > a.drain_gb * 1e9:
emit("")
emit(f"_⚠ aborted before {m}: VRAM still {gb(u)} used after {a.unload_wait}s — something "
f"won't unload (leftover model / GPU apps). If that's your normal desktop, raise --drain-gb._")
print(f"\n⚠ aborted: card not clean ({gb(u)} used).", file=sys.stderr)
break
gtt_pre, _ = rocm_mem("gtt") # per-model GTT baseline — GTT reclaims slowly across swaps
try:
# warmup — retries wait out the mount; 'failed to load' aborts immediately
with_retries(lambda: run(a.url, a.key, m, "hi", 8, a.temp, a.timeout), a.retries, a.retry_wait, m)
used, _ = rocm_mem("vram")
gtt, _ = rocm_mem("gtt")
free = (total_vram - used) if (total_vram and used) else None
gtt_delta = (gtt - gtt_pre) if (gtt is not None and gtt_pre is not None) else None
reason = None
if free is not None and free < a.min_free_gb * 1e9:
reason = f"only {gb(free)} VRAM free"
elif gtt_delta is not None and gtt_delta > a.max_gtt_gb * 1e9:
reason = f"GTT grew {gb(gtt_delta)} = spilling to system RAM"
if reason:
# Too tight — do NOT generate (allocates more). Record the row; the sweep continues
# once this model has fully unloaded (see below).
hint = ("raise n-cpu-moe" if c.get("ncpumoe", "-") != "-"
else "lower ctx / smaller quant / add n-cpu-moe if MoE")
cells = base_cells + [f"⚠ {reason} — SKIPPED ({hint})", "", "", gb(used), gb(free)]
if a.ctx:
cells.append("")
else:
r = with_retries(lambda: run(a.url, a.key, m, task, a.tokens, a.temp, a.timeout),
a.retries, a.retry_wait, m)
cells = base_cells + [f"{r['tg']:.1f}", (f"{r['pp']:.0f}" if r['pp'] else "-"),
f"{r['ttft']:.2f}", gb(used), gb(free)]
if a.ctx:
rc = with_retries(lambda: run(a.url, a.key, m, filler + "\n\n" + task, a.tokens, a.temp, a.timeout),
a.retries, a.retry_wait, m)
cells.append(f"{rc['tg']:.1f}")
except Exception as e:
err = f"HTTP {e.code} (check server log)" if isinstance(e, urllib.error.HTTPError) else f"ERROR {e}"
cells = base_cells + [err] + [""] * (len(cols) - len(base_cells) - 1)
emit("| " + " | ".join(str(x) for x in cells) + " |")
# (no post-row wait needed: the pre-load drain above protects the next iteration,
# including after ⚠-tight rows where generation was skipped)
latest.write("\n_Tuning hints: **VRAM free** = headroom to raise `ctx-size` or lower `n-cpu-moe` "
"(more experts on GPU → faster). Low decode t/s on an offloaded MoE → lower `n-cpu-moe` "
"if free allows. High **baseline used** invalidates the run. For spec/MTP models, check "
"the server log's acceptance rate._\n")
latest.close(); hist.close()
print(f"\nwrote {a.out} (+ appended {a.history})")
if __name__ == "__main__":
main()
fl/.config/llamacpp/config.ini
+457
View File
@@ -0,0 +1,457 @@
# llama.cpp model config — AMD RX 7600 XT (ROCm) · 16 GB (17.16 GB total)
# ═════════════════════════════════════════════════════════════════════════════
# ⚠⚠ VRAM SAFETY — READ THIS. This GPU ALSO DRIVES THE DISPLAY. If a model asks for
# more VRAM than is free, ROCm does NOT OOM cleanly — it spills into system RAM
# (GTT), which starves the desktop and FREEZES THE WHOLE PC (reboot required).
# → TARGET ≥ ~2.5 GB free (≤ ~14.5 GB used). HARD FLOOR 1.5 GB (bench.py guard stops there).
# Idle desktop uses ~1.3 GB of 17.16 — but a browser/leftover model can hold 1-3 GB more,
# so measure with a CLEAN baseline (bench.py prints "baseline used"; close GPU apps first;
# sleep-idle keeps a model resident up to 5 min after last use).
# → Verify with bench.py → ledger: bench-results.md (latest) + bench-history.md (all runs).
# → If a model is tight: LOWER ctx-size, or RAISE n-cpu-moe (MoE), or use a smaller quant.
# The values below are deliberately CONSERVATIVE after freeze incidents — raise ctx
# only after bench.py confirms the headroom.
# ═════════════════════════════════════════════════════════════════════════════
# Speed notes: decode is memory-bandwidth-bound (~288 GB/s). Prefer MoE (few active
# params). For MoE that doesn't fit, n-cpu-moe offloads expert FFNs to CPU (48 GB RAM) —
# this is FASTER than letting VRAM spill to GTT, and frees VRAM. threads = 6 (5600X cores).
# Runtime: `sudo rocm-smi --setperflevel high`; `watch -n1 rocm-smi --showmeminfo vram`.
# ─────────────────────────────────────────────────────────────────────────────
# Coding models
# ─────────────────────────────────────────────────────────────────────────────
[Qwen3-Coder-Next-UD-IQ3_XXS]
# ⭐ Big-context coder: Qwen3-Coder-Next 80B-A3B (3B active), Qwen3-Next hybrid attn → cheap KV.
# Weights ~30 GB: ALL experts on CPU (n-cpu-moe 99), only attention+KV on GPU → low VRAM, safe.
# Needs a CURRENT llama.cpp build (Qwen3-Next arch). ⬇ DOWNLOAD unsloth/Qwen3-Coder-Next-GGUF:UD-IQ3_XXS.
# Froze the PC before at n-cpu-moe 48 / ctx 131072 (overcommit) — keep it conservative, verify, then raise ctx.
model = /home/anon/software/models/Qwen3-Coder-Next-UD-IQ3_XXS.gguf
ctx-size = 131072 # ✓ VERIFIED @131072/moe99 (2026-07-16): 15.7 t/s, 6.0 GB used / 11.2 free —
# full 128k at a comfortable margin. Settled.
n-gpu-layers = 99
n-cpu-moe = 99 # ALL experts on CPU — settled AGAIN 2026-08-08 after a two-round
# experiment, CLOSED: GPU experts barely move this model.
# moe56 (04:11) = NO-OP: 6.0 used ≈ moe99's 5.8, all numbers flat →
# only ~48 MoE layers exist; moe ≥ 48 ≡ all-on-CPU. (Also: the old
# "moe48/131k froze the PC" lore was ctx-KV overcommit in a pre-
# quantized-KV config, not expert placement.)
# moe40 (04:37) = 8 layers on GPU, 10.2 used / 6.9 free: prefill
# 33→36 (+9%), decode 16.4 (noise) — 4.2 GB for a rounding error.
# The shallow curve says don't chase it: cache-reuse already makes
# growing sessions incremental, and the 6 GB footprint IS the niche.
# MEASURED (07-16): moe99 = 14.3-16.4 t/s vs moe36@65536 = 16.9.
threads = 6
# no-mmap TRIED AND REVERTED 2026-07-18: llama.cpp's log hint suggested it (CPU experts +
# mmap = page-fault churn), but reading 26 GB fully resident exceeds the router's load
# patience → "failed to load". Retry only if the router ever gets a load-timeout knob.
flash-attn = on
cache-type-k = q8_0
cache-type-v = q4_0
batch-size = 2048
ubatch-size = 256 # small compute buffer = less VRAM
cache-reuse = 256
defrag-thold = 0.1
temp = 0.7 # unsloth default 1.0/top-p0.95/top-k40/min-p0.01 — lower = more deterministic
top-p = 0.95
top-k = 40
min-p = 0.01
jinja = on
sleep-idle-seconds = 300
# ─── RETIRED 2026-07-16: [DeepSeek-Coder-V2-Lite-Instruct-Q8_0] ──────────────────────────
# Mid-2024 model, outclassed by GLM-4.7-Flash + both Qwen coders. Bench kept skipping it
# (~15.3 GB fixed footprint; measured 2026-07-03 that ctx/moe knobs barely move it — only fix
# was a Q4_K_M requant, not worth it as the 4th-best coder). Removed from opencode/pi/aichat.
# File can be deleted from the models dir: DeepSeek-Coder-V2-Lite-Instruct-Q5_K_M.gguf
# ─── RETIRED 2026-08-06: [Qwen3-Coder-30B-Instruct-IQ4_XS] ───────────────────────────────
# Lost the 2026-08-06 quant race to UD-Q3_K_XL@moe10 (31.3 vs 23.0 t/s, +36%, three
# consistent sweeps; it also failed twice to load on ~3.2 GB evening baselines at moe20 =
# 14.1-14.3 used). Clients rewired to Qwen3-Coder-30B-Instruct-UD-Q3_K_XL same day — the
# extended quality trial was deliberately skipped (user call; revert path = git, one commit).
# ⚠ OWUI chats/presets saved against this id need re-picking after the restart.
# File can be deleted from the models dir: Qwen3-Coder-30B-A3B-Instruct-IQ4_XS.gguf
# Reference numbers: moe24 = 21.1 t/s, 11.9/5.3 · moe20 = 23.2 (2026-07-16), 22.0-23.0 (2026-08-05/06)
[Qwen3-Coder-30B-Instruct-UD-Q3_K_XL]
# ⭐ MAIN AGENT CODER — took the slot 2026-08-06 after winning the quant race: 31.3 t/s @moe10
# vs IQ4_XS@moe20 = 23.0 (+36%, three consistent sweeps); Q3_K_S dominated (tombstone below).
# Same 32k ctx and sampling as the retired IQ4_XS — clients only needed the id swap.
# ⚠ QUALITY TRIAL SKIPPED (user call, 2026-08-06): Q3-on-MoE damage looks like wrong API
# signatures / off-by-one logic, NOT word salad — if opencode sessions smell off, the
# IQ4_XS preset + client wiring are one `git revert` away (and the .gguf may still be on
# disk). UD protects attn/shared tensors; the experts are still 3-bit.
# RACE NUMBERS: moe24 = 22.2 · moe16 = 25.1 (07-16) / 25.5 / 26.3 · moe10 = 31.3 (15.0 used /
# 2.1 free @3.1 GB baseline; replicate 30.9 @1.8 free under a 13.9 leftover-model baseline).
model = /home/anon/software/models/Qwen3-Coder-30B-A3B-Instruct-UD-Q3_K_XL.gguf
ctx-size = 32768
n-gpu-layers = 99
n-cpu-moe = 12 # ✓ VERIFIED @moe12 (2026-08-07, 2.9 GB EVENING baseline): 27.0 t/s,
# 14.2 used / 3.0 free — loaded fine in exactly the regime that killed
# IQ4_XS@moe20 twice; the insurance works. Cost vs moe10 (31.3 clean race)
# was ~4 t/s, more than the ~1 estimated — evening compute contention
# likely eats 2-3 of that; confirmed: CLEAN (2026-08-08 02:45, 1.9
# baseline) = 29.5 t/s, 13.2 used / 4.0 free — evening cost was indeed
# contention. Settled: moe12 = 29.5 clean / 27.0 evening, always loads.
threads = 6
flash-attn = on
cache-type-k = q8_0
cache-type-v = q4_0
batch-size = 2048
ubatch-size = 1024 # ✓ SETTLED 2026-08-08: prefill 74 → 89-92 t/s (+24%), decode unchanged,
# buffer +0.5 GB. FINAL GATE PASSED (14:43 sweep, 3.0 GB dirty baseline =
# evening-class): loaded and ran at 14.6 used / 2.6 free. If a worse
# evening (3.3+) ever fails the load: revert to 512, or pay one moe
# step (12→13).
cache-reuse = 256
defrag-thold = 0.1
temp = 0.15
top-p = 0.8
top-k = 20
min-p = 0
repeat-penalty = 1.05
jinja = on
sleep-idle-seconds = 300
# ─── RETIRED 2026-08-06: [Qwen3-Coder-30B-Instruct-Q3_K_S] ───────────────────────────────
# Dominated in the quant race: 30.9 t/s @moe8 (15.2 used / 2.0 free, replicate 30.9) = a tie
# with UD-Q3_K_XL@moe10 (31.3) at equal headroom, from a strictly worse quant (plain Q3_K_S,
# no UD-protected attn/shared tensors). No axis where it wins. Was never client-wired.
# File can be deleted from the models dir: Qwen3-Coder-30B-A3B-Instruct-Q3_K_S.gguf
# ─────────────────────────────────────────────────────────────────────────────
# General Qwen models (35B-A3B is fast MoE; the dense 27Bs are inherently VRAM-tight)
# ─────────────────────────────────────────────────────────────────────────────
[Qwen3.6-35B-A3B-MTP-UD-IQ3_XXS]
# ⭐ Fast NON-THINKING daily driver: Qwen3.6-35B-A3B (MoE, 3B active) — the "bigger Qwen3
# Instruct". Compact precise answers, no reasoning preamble. (Alias kept for clients.)
# File is flat + named without "MTP" (unsloth MTP-repo naming). If the load log complains
# there is no MTP head in this file, comment the two spec- lines out (plain repo file).
model = /home/anon/software/models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf
mmproj = /home/anon/software/models/mmproj-Qwen3.6-35B-A3B-F16.gguf # vision ✓ VERIFIED 2026-07-18:
# 13.9 used / 3.3 free (2.8 dirty baseline), decode unchanged at 36-37 t/s.
spec-type = draft-mtp
spec-draft-n-max = 2
reasoning-budget = 0 # hard-disable thinking → instant compact answers. Delete to allow /think.
# ⚠ if startup says "option 'reasoning_budget' not recognized" → remove this line.
ctx-size = 24576 # raised from 16384 (bench @16384: 4.8 GB free) — bench to confirm ≥2.5 free
n-gpu-layers = 99
n-cpu-moe = 16 # MEASURED @24: 19.8 t/s (MTP working!), 10.6 used / 6.6 free → pull experts onto GPU
threads = 6
flash-attn = on
cache-type-k = q8_0
cache-type-v = q4_0
batch-size = 2048
ubatch-size = 512
cache-reuse = 256
defrag-thold = 0.1
temp = 0.7
top-p = 0.8
top-k = 20
min-p = 0
repeat-penalty = 1.05
jinja = on
sleep-idle-seconds = 120
[Qwen3.6-35B-A3B-Thinking]
# Same GGUF as above with reasoning ENABLED (no reasoning-budget line) — fills the "thinking
# model" slot at zero disk cost. Qwen thinking-mode sampling: temp 0.6 / top-p 0.95, no
# repeat-penalty. Slower per answer (reasoning preamble) — use for hard problems.
model = /home/anon/software/models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf
mmproj = /home/anon/software/models/mmproj-Qwen3.6-35B-A3B-F16.gguf # vision ✓ VERIFIED 2026-07-18:
# 13.9 used / 3.3 free (2.8 dirty baseline), decode unchanged at 36-37 t/s.
spec-type = draft-mtp
spec-draft-n-max = 2
ctx-size = 24576 # thinking eats ctx — same footprint as the non-thinking preset
n-gpu-layers = 99
n-cpu-moe = 16
threads = 6
flash-attn = on
cache-type-k = q8_0
cache-type-v = q4_0
batch-size = 2048
ubatch-size = 512
cache-reuse = 256
defrag-thold = 0.1
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0
jinja = on
sleep-idle-seconds = 120
# ─── RETIRED 2026-07-03: [Qwen3.6-27B-UD-IQ3_XXS] ────────────────────────────────────────
# The 27B files (plain AND MTP) were deleted from the models dir — preset removed so the
# router doesn't advertise a dead model. To bring it back: re-download (MTP-repo files are
# named WITHOUT "MTP"), restore the preset, re-add the id to opencode/pi/aichat lists.
# ─── RETIRED 2026-07-16: [Qwen3.5-27B-UD-Q3_K_XL] ────────────────────────────────────────
# Dense 27B: 16.2 GB used / 0.9 free even @8192 (under the 1.5 floor) and ~10 t/s ceiling on
# this card. Role covered by Qwen3.6-35B-A3B (MoE, 2× faster). Removed from opencode/pi/aichat.
# File can be deleted from the models dir: Qwen3.5-27B-UD-Q3_K_XL.gguf
[Qwen3.5-9B-UD-Q6_K_XL]
# Small fast Qwen + MTP. Replaces Qwen3.5-9B-Q8_0 (retired 2026-07-16: Q8 was quality-overkill
# and bandwidth-slow at 17.2 t/s — delete Qwen3.5-9B-Q8_0.gguf; also delete the interim
# Qwen3.5-4B-UD-Q6_K_XL.gguf, superseded by this).
# ⬇ FILE MUST COME FROM unsloth/Qwen3.5-9B-MTP-GGUF (8.99 GB — MTP head embedded, filename
# identical to the plain repo!). If the load log complains there is no MTP head in this
# file, it's the plain-repo file — comment the two spec- lines out.
# ✓ VERIFIED (2026-07-16): 32.2 t/s (MTP working — nearly 2× the Q8's 17.2), prefill 83,
# 12.4 used / 4.8 free @32768. TTFT ~3.7 s = the model thinking (reasoning streams first).
model = /home/anon/software/models/Qwen3.5-9B-UD-Q6_K_XL.gguf
mmproj = /home/anon/software/models/Qwen3.5-9B-MTP-mmproj-F16.gguf # vision (from the MTP repo)
spec-type = draft-mtp
spec-draft-n-max = 3 # ⚗→✓ KEPT (04:11): 33.8 t/s vs 32.0-33.1 @n=2 — small real win.
# Revert to 2 if a future sweep drops below 32.
ctx-size = 32768 # raise toward 65536 once bench confirms free ≥2.5 GB
n-gpu-layers = 99
flash-attn = on
cache-type-k = q8_0
cache-type-v = q4_0 # q4 V to save VRAM
batch-size = 2048
ubatch-size = 512
cache-reuse = 256
defrag-thold = 0.1
temp = 0.7
top-p = 0.8
top-k = 20
min-p = 0
repeat-penalty = 1.05
jinja = on
sleep-idle-seconds = 60
# ─── RETIRED 2026-07-16: [Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-IQ3_M] ──────────────
# Dense: 10 t/s, 2.0 GB free (under the 2.5 target) and unfixable — bandwidth-bound, smaller
# quant would cost the quality that justified it. Removed from opencode/pi/aichat.
# File can be deleted from the models dir: Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-IQ3_M.gguf
# ─────────────────────────────────────────────────────────────────────────────
# Gemma models
# ─────────────────────────────────────────────────────────────────────────────
[gemma-4-26B-A4B-it-UD-IQ4_XS]
# MoE (4B active) + MTP + vision. ~13.5 GB weights → tight. Projector on CPU (no-mmproj-offload).
# From unsloth/gemma-4-26B-A4B-it-qat-GGUF: mmproj-F16.gguf + mtp-gemma-4-26B-A4B-it.gguf.
# ⬇ SOURCES — local name ← upstream (PLAIN repo unsloth/gemma-4-26B-A4B-it-GGUF; the qat
# repo has NO IQ4_XS, its only main is qat-UD-Q4_K_XL 14.25 GB — see decision note below):
# gemma-4-26B-A4B-it-UD-IQ4_XS.gguf ← gemma-4-26B-A4B-it-UD-IQ4_XS.gguf (13.6 GB)
# ✓ current: the 2026-08-05 dl has the Jul 17 template
# mmproj-gemma-4-26B-A4B-it-F16.gguf ← mmproj-F16.gguf (1.19 GB)
# mtp-gemma-4-26B-A4B-it.gguf ← MTP/mtp-gemma-4-26B-A4B-it-Q8_0.gguf (462 MB)
# Keep LOCAL names stable; re-fetch the upstream names above when repos update. Pair main
# and drafter from the SAME repo — the two repos' drafters are different extractions.
# ✓ FIXED 2026-08-08 03:15: matched plain-repo sidecars cured the "Invalid input batch."
# 500s — root cause CONFIRMED as the cross-repo drafter mismatch (old qat-repo mtp under
# the new plain main). Re-benched 32.8 t/s @moe13, 12.1 used / 5.0 free, MTP ✓ — same
# speed as pre-update (32.7), template fix now in place. The *.gguf.bak sidecars are
# deletable once vision + opencode tool-calling are smoke-tested.
# (QAT quality upgrade = separate decision: main → qat-UD-Q4_K_XL, +0.7 GB, moe 13→~16, and
# a section/client id rename round. Not part of this fix.)
# ⚠ vision + MTP together is heavy — if it OOMs/freezes, add n-cpu-moe or drop MTP (keep vision).
model = /home/anon/software/models/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf
mmproj = /home/anon/software/models/mmproj-gemma-4-26B-A4B-it-F16.gguf
model-draft = /home/anon/software/models/mtp-gemma-4-26B-A4B-it.gguf
spec-type = draft-mtp
spec-draft-n-max = 2
reasoning-budget = 0 # added 2026-08-09: the Jul-17 official template makes gemma-4 DELIBERATE
# on hard prompts (probe: 700 tok = 100% reasoning_content, content
# EMPTY, finish=length) — that ate pi's 4096 maxTokens cap and returned
# truncated nothing. Same fix as the 35B daily driver. ⚠ if startup says
# option not recognized, remove; if it loads but a hard probe still emits
# reasoning_content, the gemma template ignores the budget → remove and
# raise client maxTokens to 8192 instead.
ctx-size = 24576 # MEASURED @16384/moe16: 29.6 t/s, 10.7 used / 6.5 free → sliding-window KV is
# cheap, take the ctx back (clients already assume 24576)
# n_ctx_train = 262144 (08-08 load log) — huge ctx headroom exists if a
# long-context vision need ever appears; bench stepwise, free ≥ 2.5.
n-gpu-layers = 99
n-cpu-moe = 13 # MEASURED @16: 26.4 t/s, 11.9 used / 5.2 free → pull ~3 expert layers onto GPU for speed
threads = 6
flash-attn = on
jinja = on # OpenAI-style tool calling (opencode build/plan default to this preset)
cache-type-k = q8_0
cache-type-v = q4_0
batch-size = 2048
ubatch-size = 512
cache-reuse = 256
defrag-thold = 0.1
no-mmproj-offload = true # vision encoder on CPU → frees ~1 GB
temp = 1.0
top-p = 0.95
top-k = 64
sleep-idle-seconds = 60
[gemma-4-E4B-it-UD-Q8_K_XL]
# ~5-7 GB — small/fast, lots of headroom, safe at big ctx.
# Vision: the HF repo ships mmproj-{BF16,F16,F32}.gguf — use F16 (990 MB), saved AS the exact
# path below (rename it; if the path doesn't exist the model fails to load / vision silently
# missing depending on router). Verify: startup log shows the mmproj load, and an image request
# answers instead of erroring "image input is not supported".
# NOTE: repo got Google's chat-template update ~2026-05 — if the main gguf predates that, re-download it too.
model = /home/anon/software/models/gemma-4-E4B-it-UD-Q8_K_XL.gguf
mmproj = /home/anon/software/models/mmproj-gemma-4-E4B-it.gguf
model-draft = /home/anon/software/models/mtp-gemma-4-E4B-it.gguf # ✓ MEASURED (2026-07-18): 57.3 t/s —
# +50% over 38.4 without the draft, 10.5 used / 6.6 free. Fastest model on the box.
spec-type = draft-mtp
spec-draft-n-max = 2
reasoning-budget = 0 # added 2026-08-09, same reason as the 26B: gemma-4 deliberates on hard
# prompts since the Jul-17 template (this is what caused the 20-second
# aichat -e "think" preambles on 2026-08-05). Fast models answer NOW.
# ⚠ same caveats as the 26B — remove if not recognized / not honored.
ctx-size = 98304 # ⚗→✓ CLOSED 2026-08-08: 58.2-60.4 t/s, 10.3-10.6 used / 6.5-6.9 free
# @98304, and n_ctx_train = 131072 confirmed (upstream config.json
# max_position_embeddings) — 98304 is inside training range, no RoPE
# risk. 131072 is the hard ceiling if ever needed (~+0.5-1 GB KV).
n-gpu-layers = 99
flash-attn = on
jinja = on # OpenAI-style tool calling (doc-writer agent defaults to this preset)
cache-type-k = q8_0
cache-type-v = q8_0
batch-size = 2048
ubatch-size = 1024
cache-reuse = 256
defrag-thold = 0.1
temp = 1.0
top-p = 0.95
top-k = 64
sleep-idle-seconds = 30
# ─────────────────────────────────────────────────────────────────────────────
# GLM
# ─────────────────────────────────────────────────────────────────────────────
[GLM-4.7-Flash-UD-Q4_K_XL]
# 30B-A3B MoE, primary coder. Upgraded UD-Q3_K_XL (~14 GB) → UD-Q4_K_XL (17.5 GB, unsloth's
# recommended quant) 2026-07-16, benched clean same day → the old GLM-4.7-Flash-UD-Q3_K_XL.gguf
# can be deleted.
# Tuning trajectory: Q3@moe9 = 28.2 t/s → Q4@moe24 = 17.9 (measured 2026-07-16) → now moe18,
# lower stepwise toward 16 while bench keeps free ≥2.5 GB.
model = /home/anon/software/models/GLM-4.7-Flash-UD-Q4_K_XL.gguf
ctx-size = 24576 # MEASURED @16384/moe24 (Q3): 9.5 GB used, 7.6 free
# n_ctx_train = 202752 (08-08 load log) — MLA KV is compact, ctx could
# go far higher if opencode subagent sessions ever outgrow 24k.
n-gpu-layers = 99
n-cpu-moe = 22 # lowered 24→22 on 2026-08-08: both step-down gates passed post-KV-fix —
# clean sweep @24 = 20.5 t/s, 12.0 used / 5.2 free (02:45, 1.9 baseline);
# elevated-baseline load @24 = 18.7, 13.0/4.2 (@2.9). moe22 est ~21.5
# clean / ~19.5-20 evening, ~12.7 used clean / ~14.0 evening → inside the
# coder-proven evening envelope (14.2 loads). ✓ CLEAN VERIFIED (02:58,
# 2.0 baseline): 21.5 t/s, 12.8 used / 4.4 free — est was 21.5, exact.
# Evening observation pending; revert to 24 if an evening bench shows
# free < 2.5. moe20 stays gated (~14.4-14.7 evening = borderline vs the
# coder's 15.2 evening death).
# (08-05/07 "failures" were the mixed-KV rejection below, not VRAM.)
# Trajectory: Q3@moe9 = 28.2 → Q4@moe24 = 17.9 (07-16) → 18.7/20.5 post-KV-fix.
threads = 6
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0 # was q4_0 — BROKE 2026-08-08: a llama.cpp upgrade (the one deprecating
# defrag-thold) enforces same-type K/V for this model (deepseek2/MLA
# conversion; MLA KV is compact, mixed types unsupported). Every load
# since failed at context creation regardless of VRAM. Only THIS preset
# was affected — the other k-q8/v-q4 presets are regular-attention archs.
# If q8/q8 still refuses: delete both cache-type lines (f16 default; MLA
# KV is small enough that the cost is minor at 24576 ctx).
batch-size = 2048
ubatch-size = 512
cache-reuse = 256
defrag-thold = 0.1
# MTP: GLM-4.7-Flash HAS an MTP head upstream (num_nextn_predict_layers: 1) but the GGUF
# conversion (deepseek2 arch) drops those tensors and llama.cpp draft-mtp only supports
# Qwen3.5/3.6 + Gemma 4 so far (PRs #22673/#23398). No mtp-*.gguf exists to download —
# recheck when llama.cpp lands GLM MTP support.
temp = 0.6 # unsloth rec: 0.7/top-p 1.0 (tool-calling) or 1.0/0.95 (general); no repeat-penalty
top-p = 0.95
min-p = 0.01
jinja = on
reasoning-preserve = true # ⚗ option ACCEPTED by the build (loaded + ran 20.9 t/s, 04:11 sweep).
# BEHAVIOR flag — judge multi-turn subagent quality in opencode over the
# next days; remove if reasoning bleed ever pollutes final answers.
sleep-idle-seconds = 60
# ─────────────────────────────────────────────────────────────────────────────
# Utility & candidates
# ─────────────────────────────────────────────────────────────────────────────
[Qwen3-Embedding-0.6B]
# Embeddings-only, for RAG/semantic search (OWUI etc. via /v1/embeddings) — NOT a chat model,
# so it is deliberately absent from the opencode/pi/aichat chat lists.
# Qwen3-Embedding requires last-token pooling. ~0.7 GB — tiny, loads fast.
# (Q8_0 is effectively lossless for a 0.6B embedder — the f16 file is redundant, delete it.)
# CPU-ONLY on purpose: a 0.6B embeds fast on 6 cores, and 0 VRAM means it can never stack
# with a chat model into an overcommit (and never suffers/causes eviction pressure).
model = /home/anon/software/models/Qwen3-Embedding-0.6B-Q8_0.gguf
embeddings = true
pooling = last
ctx-size = 8192
n-gpu-layers = 0
flash-attn = on
batch-size = 8192 # embed long chunks in one pass
ubatch-size = 1024
threads = 6
sleep-idle-seconds = 60
[gpt-oss-20b]
# OpenAI gpt-oss-20b — MoE 21B / 3.6B active, reasoning + tool use (enabled 2026-07-16, NOT
# BENCHED yet). OpenAI post-trained it in MXFP4, so there is exactly ONE sensible GGUF:
# ggml-org/gpt-oss-20b-GGUF → gpt-oss-20b-mxfp4.gguf (~12.1 GB). Requants are never better —
# the old gpt-oss-20b-Q3_K_M.gguf can be deleted.
# Harmony chat template → jinja required. Default KV types kept (gpt-oss attention sinks have
# had issues with quantized KV — verify before adding cache-type lines).
model = /home/anon/software/models/gpt-oss-20b-mxfp4.gguf
ctx-size = 65536 # ⚗→✓ VERIFIED @65536 (04:11): 37.3/37.4 t/s unchanged, 14.0 used / 3.2
# free — the fast long-context option is real (37 t/s vs Coder-Next 16).
# 131072 is OFF the table: 32k→65k cost +1.2 GB (KV slope steepens past
# the sliding window) → 131k would land under 2.5 free. Client caps
# raised 32k→64k on 2026-08-08.
n-gpu-layers = 99
n-cpu-moe = 4 # ✓ SETTLED. MEASURED @6: 31.3 t/s · @4 (2026-07-16): 38.1 t/s, 12.7 used /
# 4.5 free — fastest model in the lineup. TTFT (~3-7 s) is the model THINKING
# (harmony reasoning streams as reasoning_content first) — not a slow load.
threads = 6
flash-attn = on
batch-size = 2048
ubatch-size = 512
cache-reuse = 256
defrag-thold = 0.1
temp = 1.0
top-p = 1.0
jinja = on
sleep-idle-seconds = 120
[gpt-oss-20b-low]
# Same GGUF, reasoning effort LOW — zero disk cost (like the 35B-Thinking alias, inverted).
# Cuts the multi-second thinking preamble for light/interactive use; keep [gpt-oss-20b]
# (default = medium effort) for hard problems. ⚠ if startup says the option is not
# recognized, the llama.cpp build predates chat-template-kwargs — comment that line out.
model = /home/anon/software/models/gpt-oss-20b-mxfp4.gguf
chat-template-kwargs = {"reasoning_effort":"low"}
ctx-size = 65536 # ⚗→✓ VERIFIED @65536 (04:11): 37.3/37.4 t/s unchanged, 14.0 used / 3.2
# free — the fast long-context option is real (37 t/s vs Coder-Next 16).
# 131072 is OFF the table: 32k→65k cost +1.2 GB (KV slope steepens past
# the sliding window) → 131k would land under 2.5 free. Client caps
# raised 32k→64k on 2026-08-08.
n-gpu-layers = 99
n-cpu-moe = 4
threads = 6
flash-attn = on
batch-size = 2048
ubatch-size = 512
cache-reuse = 256
defrag-thold = 0.1
temp = 1.0
top-p = 1.0
jinja = on
sleep-idle-seconds = 60
# ─────────────────────────────────────────────────────────────────────────────
# Notes
# ─────────────────────────────────────────────────────────────────────────────
# n-cpu-moe N : offload the first N MoE layers' experts to CPU. Higher = less VRAM (safer) but
# slower. On this display-driving GPU, a bit of offload is FAR better than a freeze.
# Only applies to MoE models (dense models ignore it — reduce ctx instead).
# override-tensor : finer control, e.g. `blk\.(1[2-9]|[2-9][0-9])\.ffn_.*_exps=CPU`.
# After ANY change: reload the server and run bench.py — keep every "VRAM free" ≥ ~2.5 GB.