134 lines
8.6 KiB
Markdown
134 lines
8.6 KiB
Markdown
# Local LLM server — laptop (lw), setup & usage
|
||
|
||
llama.cpp in **router mode** on the laptop. This is the **CPU** counterpart of the fl box
|
||
(`fl/.config/llamacpp/`) — lw has **no discrete GPU**, so everything runs on the CPU and
|
||
the only resource ceiling is **system RAM** (not VRAM). None of fl's VRAM/freeze rules apply.
|
||
|
||
- **Box**: Intel Core i7-5600U (Broadwell 2015, **2 cores / 4 threads**, AVX2) · iGPU HD Graphics
|
||
5500 (no compute) · **15.49 GiB RAM** · DDR3L-1600 dual-channel: ~20 GB/s theoretical, **~8–10
|
||
GB/s effective measured** — the dense-decode bottleneck → prefer **small quants + MoE**.
|
||
- **Endpoint**: `http://127.0.0.1:11343/v1` — **local only** (server binds `127.0.0.1`; deliberate:
|
||
the laptop roams onto untrusted wifi and the API has no key, unlike the stationary fl box).
|
||
- **Files here** (`~/.config/llamacpp/`, stowed from the repo's `lw/.config/llamacpp/`):
|
||
`config.ini` (presets — section names ARE the API model ids), `bench.py` (CPU benchmark),
|
||
`bench-results.md` / `bench-history.md` (ledgers, generated by bench.py). The GGUF blobs
|
||
themselves live in `~/software/models/` and are not tracked.
|
||
|
||
## How it runs
|
||
|
||
Launched by the fish abbr **`llamaserver`** (defined in `lw/.config/fish/host.fish`;
|
||
llama.cpp is pacman-installed, `llama-server` is on PATH):
|
||
|
||
```
|
||
nice -n 19 llama-server --host 127.0.0.1 --port 11343 --cors-origins localhost \
|
||
--models-max 1 --models-preset ~/.config/llamacpp/config.ini
|
||
```
|
||
|
||
- Router mode: each requested model loads on demand in a child process.
|
||
- **`--models-max 1`**: one model resident; a new request LRU-evicts the previous. There's no
|
||
fl-style freeze risk here (no display-driving GPU) — this is purely the safest RAM posture on
|
||
15 GB with no co-residency swap. Raise to 2–3 only to keep small models hot together
|
||
(E2B+4B+1.7B ≈ 6 GB fits). `sleep-idle-seconds` per preset frees idle models either way.
|
||
- **`nice -n 19`**: on 2 cores generation pins the whole CPU — niceness makes inference yield to
|
||
interactive apps (full speed when the desktop is idle).
|
||
- **`--cors-origins localhost`**: same-machine profile from llama.cpp PR 25655 — the default CORS
|
||
reflects any Origin, so without this a malicious page in the laptop's own browser could call
|
||
the (keyless) API on 127.0.0.1. Browser-only enforcement: bench.py/CLI clients are unaffected.
|
||
- `config.ini` here is the stowed repo copy (`lw/.config/llamacpp/`), so editing it edits the
|
||
repo (no deploy step).
|
||
|
||
## RAM (the golden rule)
|
||
|
||
There is no VRAM and no freeze risk. The constraint is plain: **N resident models must fit in
|
||
system RAM together.** `bench.py` records each model's RAM footprint (MemAvailable delta) and
|
||
the remaining free RAM — keep enough headroom for the desktop (and for co-resident models if you
|
||
ever raise `--models-max` above 1). If RAM gets tight: shorten `sleep-idle-seconds`, use smaller
|
||
quants, or reduce `ctx-size`. Note: the 4B/E2B presets are small enough to co-reside, but a big
|
||
MoE (e.g. Qwen3-30B-A3B at ~12 GB) fills most of 15 GB → it runs **1-at-a-time** regardless.
|
||
|
||
`threads = 2` (physical cores) on the dense presets — they measure at the practical bandwidth
|
||
wall (~8–10 GB/s effective) and the 2026-07-26 sweep confirmed more threads don't lift them.
|
||
The **compute-bound** pair runs `threads = 3` (same-run sweep: Granite 4.5→4.8, 30B 1.9→2.0;
|
||
`threads = 4` LOSES to HT contention on both). Every preset sets `threads-batch = 4`: prefill
|
||
is compute-bound on all models — benched +8% prefill / −9% TTFT with no decode cost, and the
|
||
extra HT load only bursts during prefill.
|
||
|
||
## Tuning cheat-sheet (CPU)
|
||
|
||
- **Quant size is the speed dial**: decode is bandwidth-bound, so a smaller quant is directly
|
||
faster. Prefer Q4_K_M / Q5_K_M; F16/Q8 are usually quality-overkill AND slow on CPU.
|
||
- **`threads`** = **physical** core count (`lscpu` → Core(s)/socket × Socket(s)). Hyperthreads
|
||
rarely help decode; oversubscribing can hurt. Set this per preset (default 4 is a guess).
|
||
- **Model size**: stick to ≤ ~8B. **MoE** models (few active params, e.g. Qwen3-A3B) are ideal
|
||
IF they fit RAM — they decode at their *active* size while giving bigger-model quality. There
|
||
is no `n-cpu-moe` tradeoff here — it's all CPU already.
|
||
- **MTP / speculative decode** (`spec-type = draft-mtp`): a clear win on the fl GPU (+50–100%) but
|
||
**TESTED AND REJECTED on this box** (2026-07-26): the draft lost the final CPU-pinned
|
||
(`gpu-layers-draft = 0`) same-run A/Bs on both gemmas — E2B 4.0 vs 4.9 t/s at 47% acceptance,
|
||
E4B 2.3 vs 2.4 at 62%. Batched verify triples the FLOPs per weight-read and 2 cores have
|
||
no spare compute, so spec decode can't pay here at any setting. MTP stays an fl-only trick.
|
||
(Only gemma-4 has drafts anyway; Qwen 2507/2.5 have no MTP path — that needs Qwen 3.5/3.6.)
|
||
- **KV cache**: leave at default (f16). Quantized KV (`cache-type-*`) adds dequant overhead with
|
||
no VRAM to reclaim → usually slower on CPU. (This is the opposite of the fl config.)
|
||
- **iGPU / Vulkan**: the pacman build **does** have a Vulkan backend and by default auto-offloads
|
||
layers to the HD 5500 — that's why every preset pins `n-gpu-layers = 0`. The iGPU shares the
|
||
same DDR3, is slower than the cores, and big allocations die (`ErrorOutOfDeviceMemory`, SIGSEGV
|
||
in llama-cli) — it broke the 30B load outright. Only re-enable offload if a bench proves the
|
||
iGPU wins (it won't).
|
||
|
||
## Benchmarking
|
||
|
||
Run **on the laptop** (against its own localhost server):
|
||
|
||
```
|
||
./bench.py # full sweep of everything the server lists
|
||
./bench.py -m id1,id2 # just the changed presets
|
||
./bench.py -n 512 --ctx 8000 # longer gen + a long-context decode column
|
||
```
|
||
|
||
For a clean per-model RAM footprint, restart the server first (with `--models-max 1` only a
|
||
re-bench of the *same* still-resident model reads a `~0` RAM Δ, but a restart keeps runs comparable). Results land in
|
||
`bench-results.md` (latest) + `bench-history.md` (append-only — diff runs there).
|
||
|
||
⚠ Absolute numbers swing **±20–30%** with desktop load and thermals (E2B has measured 6.1 idle
|
||
vs 4.9 warm/busy). Only rows from the **same run** are directly comparable — A/B via extra
|
||
preset ids in one `./bench.py -m a,b` command (see the rig sections in config.ini), never by
|
||
comparing across runs. For spec/MTP rows the draft **acceptance %** appears in the spec column.
|
||
|
||
## Model roster (reference numbers = clean sweep 2026-07-25, n-gpu-layers=0; Granite/30B since
|
||
promoted to threads=3 and all presets to threads-batch=4, 2026-07-26)
|
||
|
||
| model id | decode t/s | prefill t/s | ctx | extras | role |
|
||
|---|---|---|---|---|---|
|
||
| `gemma-4-E2B-it-UD-Q4_K_XL` | **6.1** | 8 | 4k | vision-capable (mmproj) | fastest — quick Q&A, askllama-class |
|
||
| `Qwen3-1.7B` | 4.8 | 11 | 8k | thinking | snappy small tasks |
|
||
| `Granite-4.0-H-Tiny` | 4.3–4.8 | 11–14 | 8k | 7B/~1B-active hybrid MoE, threads 3 | ⭐ speed AND brains |
|
||
| `Qwen3-4B-Instruct-2507` | 3.9 | 7 | 4k | non-thinking | daily driver |
|
||
| `Qwen2.5-Coder-3B-Instruct` | 3.3 | 6 | 8k | temp 0.2 | small coder |
|
||
| `Qwen3-30B-A3B-Instruct-2507-UD-IQ3_XXS` | 1.9–2.7 | 4 | 4k | ~12 GB, TTFT ~6 s, threads 3 | ⭐ quality when you can wait |
|
||
| `gemma-4-E4B-it-UD-Q4_K_XL` | 2.4–2.5 | 4 | 4k | vision (mmproj) | quality vision; else the 30B is smarter at the same speed |
|
||
| `Jan-v3-4b` | 2.1¹ | 4 | 4k | agentic tune | agentic variant of the 4B (~0.9× its speed) |
|
||
|
||
¹ variance-depressed in the sweep; same-run A/B vs the base (2026-07-26) shows ~0.9×.
|
||
|
||
## Client wiring
|
||
|
||
Clients run **on the laptop itself** (the server is localhost-only, see Endpoint above); same
|
||
rule as fl otherwise: client model **id = config.ini section name**, client context ≤ server
|
||
`ctx-size`. Point it at `http://127.0.0.1:11343/v1` (no key). If LAN access is ever needed,
|
||
switch the abbr back to `--host 0.0.0.0` **and add `--api-key`** — never open it keyless.
|
||
|
||
**Wired (2026-07-26), lw-only via stow host overlays** — these are full-file `--override`
|
||
copies of the common configs with an added local provider; when the common client configs
|
||
change, mirror the change here:
|
||
|
||
| tool | overlay file | provider / client id |
|
||
|---|---|---|
|
||
| aichat | `lw/.config/aichat/config.yaml` | `lw:` (e.g. `lw:gemma-4-E2B-it-UD-Q4_K_XL`) |
|
||
| opencode | `lw/.config/opencode/opencode.json` | `lwcpp/` |
|
||
| pi | `lw/.pi/agent/models.json` + `settings.json` | `lwcpp` (⚠ pi hides providers not in settings.json `enabledModels`) |
|
||
|
||
All 8 presets are listed in each, with client context capped to the server `ctx-size` and the
|
||
benched t/s in the display names. Defaults still point at the fl LAN models — switch to a local
|
||
model in-tool when roaming. Re-run `./install.sh` on lw once so stow links the new files.
|