223 lines
17 KiB
Markdown
223 lines
17 KiB
Markdown
# Local LLM server — wm (GTX 1080 Ti), setup & usage
|
||
|
||
llama.cpp in **router mode** on the Wingman work desktop. This is the **NVIDIA** counterpart of
|
||
fl (`fl/.config/llamacpp/`, RX 7600 XT 16 GB) and lw (`lw/…`, CPU laptop). Same recipe as fl —
|
||
MoE experts in RAM, quantized KV, MTP — on a smaller, older card.
|
||
|
||
- **Box**: GTX 1080 Ti **11 GB** (Pascal, compute 6.1: int8 MMQ path, no tensor cores) on the
|
||
legacy `nvidia-580xx` driver · i7-8700 (6 cores / 12 threads) · **31 GiB DDR4** (~35 GB/s =
|
||
the ceiling for experts kept in RAM).
|
||
- **Backend**: upstream prebuilt **CUDA 12.8** tarball (`~/.local/opt/llama.cpp-cuda/current`,
|
||
build b11227) through the `~/.local/bin/llama-server-cuda` wrapper. CUDA 13.x cannot target
|
||
Pascal, so the Arch `ggml-cuda` / `llama.cpp-cuda-git` packages are useless on this box; the
|
||
prebuilt carries Pascal as PTX and the driver JITs it (first start of a new build ≈ 50 s, then
|
||
cached). A Vulkan prebuilt sits beside it (`llama-server-vulkan`) as the no-CUDA fallback —
|
||
same build, ~half the prompt speed, same decode.
|
||
- **Endpoint**: `http://127.0.0.1:11343/v1` on the box, `http://192.168.50.24:11343/v1` from the LAN
|
||
(wired IP, DHCP — may change). Bound to `0.0.0.0` and **keyless**: anyone on the office network
|
||
(and ZeroTier) can use it. IPv4 only — use `127.0.0.1`, not `localhost` (which also resolves to
|
||
`::1`, where nothing listens). The chat endpoint is POST-only: a browser GET gets 404.
|
||
- **Files here** (`~/.config/llamacpp/`, per-file stow links from the repo's `wm/.config/llamacpp/`):
|
||
`config.ini` (presets — section names ARE the API model ids), `llama.service` (user unit), `bench.py` (nvidia-smi benchmark),
|
||
`bench-results.md` / `bench-history.md` (ledgers, generated next to the script, not tracked).
|
||
The GGUFs live in `~/Documents/models/` (`llama-server -hf` drops downloads into
|
||
`~/.cache/huggingface/hub/` — move them out).
|
||
|
||
## How it runs
|
||
|
||
A **user** systemd unit, `llama.service` (this dir; linked into `~/.config/systemd/user/` and enabled,
|
||
starts with the login session — `loginctl enable-linger coja` makes it start at boot without a login).
|
||
Its `ExecStart` is the command the old abbr ran:
|
||
|
||
```
|
||
nice -n 19 llama-server-cuda --host 0.0.0.0 --port 11343 --cors-origins localhost \
|
||
--models-max 1 --models-preset ~/.config/llamacpp/config.ini
|
||
```
|
||
|
||
The fish abbr **`llamaserver`** now means `systemctl --user restart llama; and journalctl --user -fu llama`.
|
||
Install once after stowing (a symlinked unit can't be linked directly, hence `readlink -f`):
|
||
|
||
```
|
||
systemctl --user link (readlink -f ~/.config/llamacpp/llama.service); systemctl --user enable --now llama
|
||
```
|
||
|
||
- Router mode: each requested model loads on demand in a child process; `/v1/models` lists the presets.
|
||
- **`--models-max 1`**: one model resident; a new request LRU-evicts the previous. Two presets
|
||
never fit together in the ~6.5–7.5 GB the desktop leaves free. `sleep-idle-seconds` per
|
||
preset frees idle models either way.
|
||
- **`nice -n 19`**: the MoE presets run their expert matmuls on all 6 cores — niceness keeps the
|
||
desktop responsive; the GPU part is unaffected.
|
||
- **`--cors-origins localhost`**: same-machine profile (llama.cpp PR 25655) — blocks browser pages
|
||
from calling the keyless API. Non-browser clients unaffected.
|
||
- Presets are read once at startup: edit `config.ini` → `systemctl --user restart llama` → `./bench.py -m <ids>`.
|
||
- Logs: `journalctl --user -u llama -f` (each model child logs under the router).
|
||
- "couldn't bind HTTP server socket … 11343" = a router is already running (the service, another
|
||
shell, or a bench run) — `systemctl --user status llama`, `pgrep -af llama-server`.
|
||
|
||
## VRAM (the golden rule)
|
||
|
||
11,264 MiB total, but Hyprland + Xorg + electron apps hold **3.5–4.5 GB**, so a preset gets
|
||
**~6.5–7.5 GB** and must be sized for the low end. Unlike fl's amdgpu, an NVIDIA overcommit does
|
||
**not** freeze the PC — `cudaMalloc` fails and the load errors out cleanly. So the guard is
|
||
"loads reliably on a busy desktop": keep every benched **VRAM free ≥ ~1 GB** (`bench.py` flags
|
||
< 0.8). If a preset is tight: raise `n-cpu-moe` (MoE), lower `ctx-size`, or use a smaller quant.
|
||
|
||
Dense models are the trap here: anything that doesn't fit entirely (> ~6 GB file) streams the
|
||
leftover layers from DDR4 and collapses (14B Q3 = 7.6 t/s, 8B Q8 = 3.4, 24B IQ3 = 1.8 — see the
|
||
tombstones in `config.ini`). MoE with the experts in RAM is 2–5× faster at the same size.
|
||
|
||
## Model roster (router benches 2026-09-28 → 10-19, `bench.py` through the service, desktop baseline 3.3–4.6 GB, gen 256)
|
||
|
||
Three conditions appear below: **quiet** = ~3.3 GB desktop, nothing else on the CPU · **busy** = 4.2–4.6 GB
|
||
desktop · **sim** = the 2026-10-19 sweep with the AirSim/PX4 simulation stack running (details after the table).
|
||
|
||
| model id | decode t/s | ctx | VRAM free | extras | role |
|
||
|---|---|---|---|---|---|
|
||
| `gemma-4-E4B-it-UD-Q4_K_XL` | **85–101** (MTP 64–84%; 91 sim) | 32k | 3.6–4.8 GB | vision (projector on CPU), MTP draft, thinking OFF | ⭐ **fastest + instant tier** — quick generalist, images, long documents; pi's default local model |
|
||
| `Mellum2.1-12B-A2.5B-Thinking-IQ4_XS` | **63** @moe4 / 87 @moe0 quiet · 42 sim | 16k | 1.6–2.3 GB | code MoE 2.5B active, thinking ON, prefill 1.1–1.5k t/s | ⭐ **fast reasoning coder** (added 10-12) |
|
||
| `Qwen3.5-9B-MTP` | **51–61** (MTP 90%) | 8k | **0.4–1.8 GB** | vision, MTP | ⚠ quiet-desktop preferred (loaded at 1.0 GB free on a ~4.5 GB desktop 10-19) — the daily driver at +75% |
|
||
| `Qwen3.5-9B-Thinking` | 34–38 | 16k | 1.0–2.5 GB | vision, reasoning budget 8k | hard problems (TTFT ≈ think block) |
|
||
| `Qwen3.5-9B-Q4_K_M` | 31–33 (one unexplained 24 on 10-12) | 16k | 1.0–2.8 GB | vision, thinking OFF | ⭐ **daily driver** — best quality that always fits |
|
||
| `gpt-oss-20b` / `-low` | 21 quiet · 17 busy · 10–12 sim | 32k | 2.1–3.7 GB | MoE @moe18, reasoning + tools | reasoning and tool use; `-low` cuts the 8.8 s think TTFT to 1.8 s |
|
||
| `Ornith-1.5-35B-A3B-instant` | **14–31** (MTP 59–67%; 15 sim) | 16k | 1.4–3.5 GB | MoE @moe36, MTP, thinking OFF | ⭐ **agentic coder** for agent loops; 31.5 and 15.1 t/s on two quiet-desktop runs 16 min apart (prefill 63 → 9) — CPU contention, the server runs at nice 19 |
|
||
| `Ornith-1.5-35B-A3B-IQ4_XS` | 10–26 (MTP 47–61%; 15 sim) | 16k | 1.8–3.3 GB | same file, thinking ON | coder with reasoning; TTFT ~20 s is the think block |
|
||
| `Ornith-1.5-35B-A3B-long` | 10 busy · 16.9 @35k prompt (MTP 59–65%) | **128k** | 1.8 GB | same file, @moe40, thinking OFF, prefill ~250 t/s | **long-context tier** (added 10-28) — 35k prompt = 2.4 min to prefill, 100k ≈ 7 min |
|
||
|
||
Read across: everything that fits **entirely on the GPU** runs 31–101 t/s; everything
|
||
that keeps experts in RAM runs 10–31 and tracks the CPU's idle share. The bench guard (`--min-free-gb 0.8`) skipped four presets
|
||
in the first sweep; they were re-tuned (coder moe 28→34, gpt-oss moe 14→18, 9B draft split into
|
||
its own alias) and re-benched green. Expected spread: MTP rows swing with draft acceptance
|
||
(content-dependent); every MoE row dips when the desktop is busy (the experts share the 6 cores).
|
||
|
||
**The sim condition (2026-10-19 12:55 sweep).** Unreal `Blocks` (≈ 3 cores at nice 5, 0.56 GB VRAM, GPU 41%
|
||
busy) plus the PX4/planner nodes at realtime priority had been running since 12:08; load average ~9 on the
|
||
6-core box. Every CPU-expert preset sat at its floor (Ornith 15, gpt-oss 10–12, Mellum 42) and the GPU-only
|
||
presets lost 5–15% to the shared GPU. Nothing in the presets caused it and nothing in them fixes it: the
|
||
service runs at nice 19 and yields by design. Judge local-model speed with the sim stopped. Two things
|
||
the run did establish: all nine presets loaded with **≥ 1.0 GB free at a ~4–4.5 GB desktop** (the
|
||
busy-desktop sizing holds), and the run's "baseline 7.5 GB" header was pi's resident gemma, not the desktop
|
||
— `bench.py` now drains before it records the baseline.
|
||
|
||
## Which model, when
|
||
|
||
- **Quick questions, images, long one-shot documents**: `gemma-4-E4B` — 85 t/s, 32k, vision.
|
||
- **Best always-available quality**: `Qwen3.5-9B-Q4_K_M` (33 t/s). On a quiet desktop pick
|
||
`Qwen3.5-9B-MTP` instead — same model, 57 t/s; if it fails to load, the desktop is holding too
|
||
much VRAM (electron apps), fall back to the plain preset.
|
||
- **Autocomplete-class / cheapest agent steps**: `gemma-4-E4B` again — LFM2.5 was faster (119 t/s) but
|
||
its output was unusable in practice and it was retired 2026-10-12.
|
||
- **Reasoning coder at speed**: `Mellum2.1-12B-A2.5B-Thinking` — 63 t/s (87 on a quiet desktop) with a
|
||
think block and >1k t/s prefill; the first stop for a bug that needs thought.
|
||
- **Agent coding loops** (opencode/pi edit-test cycles): `Ornith-1.5-35B-A3B-instant` — 14–31 t/s
|
||
depending on what else is using the CPU (nice 19), no think preamble. Switch to `…-IQ4_XS` (thinking) for the bug that needs reasoning.
|
||
- **Reasoning + tool use at long context**: `gpt-oss-20b` (32k); `-low` for interactive use.
|
||
- **Really long context (to 128k)**: `Ornith-1.5-35B-A3B-long` — the 35B with every expert in RAM so
|
||
the KV fits; prefill ~250 t/s (100k ≈ 7 min). fl's Qwen3-Coder-Next 80B needs ≥ 28 GB of RAM for
|
||
its experts and does not fit next to this desktop.
|
||
- No separate fast coder: the 2-bit Qwen3-Coder was deleted 2026-10-12 (dominated by Ornith-instant);
|
||
a K-quant Qwen3-Coder stays listed in `config.ini` candidates if a second coder is ever wanted.
|
||
- Anything bigger/denser (14B+ dense, 27B, 80B-A3B) does **not** belong on this card — see the
|
||
candidates list in `config.ini` for what does.
|
||
|
||
## Client wiring
|
||
|
||
These clients run on this box (other machines use `http://192.168.50.24:11343/v1`). Rule: client model **id = config.ini section
|
||
name**, client context ≤ server `ctx-size`.
|
||
|
||
| tool | overlay file | provider / client id |
|
||
|---|---|---|
|
||
| aichat | `wm/.config/aichat/config.yaml` | `wm:` (e.g. `wm:Ornith-1.5-35B-A3B-instant`) — default model still `duskadiy:`; `aichat -m wm:…` or flip `model:` |
|
||
| opencode | `wm/.config/opencode/opencode.json` | `wmcpp/` (e.g. `wmcpp/Ornith-1.5-35B-A3B-instant`); defaults + agent models point at `duskadiy/` (the common file's `llama.cpp/` LAN provider is unreachable here) |
|
||
| pi | `wm/.pi/agent/models.json` + `settings.json` | `wmcpp`; default model `wmcpp/gemma-4-E4B-it-UD-Q4_K_XL` since 10-12 (was `duskadiy/Ornith…`) |
|
||
|
||
**Two ways a model "doesn't respond" in pi** (both seen 2026-10-12):
|
||
|
||
1. The router was started before the preset existed — it reads `config.ini` once, so pi's request fails
|
||
as an unknown model. `systemctl --user restart llama` after every preset change.
|
||
2. A small coder (Mellum 2.1, and AesCode-8B before its removal) run with pi's full tool set **from `~`** keeps calling
|
||
read/bash to explore the home directory and never finishes (router log: the same conversation
|
||
re-served with a growing context, 2.3k → 5.4k tokens). Run pi inside a project directory, or
|
||
`pi --no-tools` for plain questions; the same prompt from an empty directory finishes in four rounds.
|
||
|
||
When adding/removing a preset, update all three wm overlays (aichat, opencode, pi). They are full-file
|
||
copies of the common configs plus the `wm`/`wmcpp` provider — when a common client config changes,
|
||
mirror the change here (same rule as lw).
|
||
|
||
## Tuning cheat-sheet (Pascal + CPU experts)
|
||
|
||
- **`n-cpu-moe N`** (MoE only): experts of the first N layers stay in RAM. The VRAM/speed dial;
|
||
each 4-bit expert layer on the GPU ≈ 0.4 GB (35B-A3B). Measured: gpt-oss @moe12 = 29.8 vs
|
||
@moe24 = 18.6 t/s; Coder-30B @moe24 = 29.9 vs @moe48 = 20.2. Tune in steps of 2–4, re-bench.
|
||
**Except Ornith**: moe30 vs moe36 on the same quiet desktop = 32.1 vs 31.5 t/s (10-12) — its experts
|
||
on the GPU buy nothing; give the VRAM to the desktop instead.
|
||
- **Quant type for CPU experts**: I-quants (IQ4_XS/IQ3_XXS) dequantize slowly on CPU and 4-bit
|
||
experts stream 2× the bytes of 2-bit → Ornith IQ4_XS with 80% of experts in RAM = 11–15 t/s
|
||
base. Prefer **K-quants** (Q4_K_M / UD-Q4_K_XL / UD-Q3_K_XL) and the smallest quant the
|
||
quality allows when the experts live in RAM.
|
||
- **MTP / spec decode** (`spec-type = draft-mtp`) **works here**: Qwen3.5-9B 32.6 → 57.3 t/s at
|
||
90% acceptance; Ornith 11–15 → 14–21 at 60%; gemma-4-E4B 84.7 with its draft. Cost: a flat
|
||
~0.75–1.1 GB VRAM per model — which is why the 9B draft lives in its own quiet-desktop alias.
|
||
Thinking mode drafts worse (Ornith 47–53%, 10 t/s). Qwen3.5/3.6 heads
|
||
live in the `-MTP` repo files; gemma-4 needs a separate `mtp-*.gguf` draft; gpt-oss and
|
||
Qwen3-Coder-30B have no head.
|
||
- **`threads = 6`** = physical cores. **Never 12**: decode collapses 14 → 1.3 t/s (HT contention).
|
||
- **KV cache**: `q8_0` K and V everywhere except gpt-oss (attention sinks — defaults). `flash-attn`
|
||
on everywhere (required for quantized V). Dense Qwens pay ~0.6–0.8 GB per 16k; MoE 48-layer
|
||
Qwen3-Coder ~1.6 GB per 16k (why it stays at 16k).
|
||
- **First value after a download**: `llama-fit-params -m <gguf> -c <ctx> -fa on` prints the split
|
||
that fits the VRAM free *right now*; take it, then add margin (moe +2, or ngl −4).
|
||
- **CPU governor** is `powersave` (intel_pstate still boosts); `performance` might buy a little
|
||
on the CPU-expert presets — unmeasured.
|
||
- **Vision**: keep the projector on the CPU (`no-mmproj-offload = true`) — ~0.9 GB VRAM back.
|
||
|
||
## Benchmarking
|
||
|
||
Run on this box against the running `llamaserver` (it evicts whatever is resident — close chat clients first):
|
||
|
||
```
|
||
./bench.py # full sweep of everything the router lists
|
||
./bench.py -m id1,id2 # just the changed presets
|
||
./bench.py -n 512 --ctx 12000 # longer gen + a long-context decode column
|
||
```
|
||
|
||
Safety-first like fl's: force-unloads resident models via the router API, waits for the card
|
||
to drain (`--drain-gb 5.5` = the desktop's own usage plus slack — raise it if your desktop
|
||
alone sits higher), records a ⚠ row instead of generating when a model leaves < 0.8 GB free.
|
||
The MTP acceptance % lands in the spec column. Ledgers: `bench-results.md` (latest) +
|
||
`bench-history.md` (append-only) next to the script.
|
||
|
||
Reading the columns (changed 2026-10-19):
|
||
|
||
- **baseline used** is measured *after* draining anything of ours; the header adds "(was X GB with a
|
||
resident model, unloaded first)" when it had to. A high baseline now really is the desktop.
|
||
- **TTFT** = time to the first *answer* token — for a thinking model that is the think block.
|
||
`all-think` = the whole gen budget (256 tokens) was reasoning and no answer arrived; decode t/s on that
|
||
row is still valid (server timings count every token). `-n 512` or the `-low`/`-instant` alias gets an answer.
|
||
- **prefill t/s** at the default ~30-token prompt is fixed-overhead noise (Mellum showed 15 and 100 on two
|
||
runs; a 1.7k-token probe gave 1,160). Use `--ctx` for a real prefill number.
|
||
- fl's and lw's `bench.py` still carry the content-only TTFT (0.00 on an all-think row); port the two
|
||
changes when their ledgers next matter.
|
||
|
||
## Watch list
|
||
|
||
- **Free VRAM is the whole game**: the desktop's 4.2–4.6 GB (Hyprland ~0.4, Xorg, Ferdium/electron
|
||
~0.6+) costs the 9B its MTP and every MoE two or three expert layers. Closing electron apps before
|
||
a session is worth more than any preset knob.
|
||
- **Qwen3.5-9B ctx 32768** for the plain preset (KV is cheap on the hybrid arch) — bench first.
|
||
- **Qwen3.6-35B-A3B-MTP UD-IQ3_XXS** (14 GB) — the one "bigger" model worth a test: 28% fewer bytes per
|
||
token than Ornith's file and a draft that accepts well on non-thinking Qwen; likely ≥ Ornith's 31 t/s quiet.
|
||
- 2026-10-12 candidate round: AesCode-8B and Mellum 2.1 kept; FrogNano-4B (87.9 t/s with draft, vision)
|
||
retired — gemma-4-E4B and LFM2.5 cover that slot faster.
|
||
- 2026-10-19: AesCode-8B removed (preset and the aichat/opencode/pi entries); tombstone in `config.ini`.
|
||
- **gemma `sleep-idle-seconds = 30` vs pi** (journal 2026-10-19 12:44–12:56): a pi session slept and woke
|
||
its default model 3× in 12 min, each wake a full reload ≈ 6 s before the first token (cold load 17 s).
|
||
Resident, gemma holds 3.7 GB and still leaves ~3 GB beside a 4.5 GB desktop; 120–300 s (fl uses 300 on its
|
||
coders and 30 on this same gemma) would remove the stall. Not changed — the VRAM-vs-latency trade is the user's call.
|
||
- **The sim stack is the busy floor** (10-19): with Blocks/PX4 running, CPU-expert presets drop 40–55% and
|
||
GPU-only ones 5–15%. Unmeasured: `threads 3`–`4` on the CPU-expert presets *while the sim holds 3 cores*
|
||
— 6 threads then share physical cores with it, and HT sharing is exactly what collapsed decode at 12.
|
||
- gemma logs `Gemma4Assistant requires ctx_other … (this warning is normal during memory fitting)` and
|
||
`failed to measure the memory of the extra model` at every load, including each wake: the auto-fitter
|
||
probing the MTP draft. Harmless (ngl 99, everything on the GPU); noted in the preset so nobody chases it.
|
||
- Dead files were deleted 2026-10-12 (66 GB): ~/Documents/models now holds exactly the nine presets' files (46 GB).
|