# Local LLM server — laptop (lw), setup & usage

llama.cpp in **router mode** on the laptop. This is the **CPU** counterpart of the fl box
(`fl/.config/llamacpp/`) — lw has **no discrete GPU**, so everything runs on the CPU and
the only resource ceiling is **system RAM** (not VRAM). None of fl's VRAM/freeze rules apply.

- **Box**: Intel Core i7-5600U (Broadwell 2015, **2 cores / 4 threads**, AVX2) · iGPU HD Graphics
  5500 (no compute) · **15.49 GiB RAM** · DDR3L-1600 dual-channel: ~20 GB/s theoretical, **~8–10
  GB/s effective measured** — the dense-decode bottleneck → prefer **small quants + MoE**.
- **Endpoint**: `http://127.0.0.1:11343/v1` — **local only** (server binds `127.0.0.1`; deliberate:
  the laptop roams onto untrusted wifi and the API has no key, unlike the stationary fl box).
- **Files here** (`~/.config/llamacpp/`, stowed from the repo's `lw/.config/llamacpp/`):
  `config.ini` (presets — section names ARE the API model ids), `bench.py` (CPU benchmark),
  `bench-results.md` / `bench-history.md` (ledgers, generated by bench.py). The GGUF blobs
  themselves live in `~/software/models/` and are not tracked.

## How it runs

Launched by the fish abbr **`llamaserver`** (defined in `lw/.config/fish/host.fish`;
llama.cpp is pacman-installed, `llama-server` is on PATH):

```
nice -n 19 llama-server --host 127.0.0.1 --port 11343 --cors-origins localhost \
  --models-max 1 --models-preset ~/.config/llamacpp/config.ini
```

- Router mode: each requested model loads on demand in a child process.
- **`--models-max 1`**: one model resident; a new request LRU-evicts the previous. There's no
  fl-style freeze risk here (no display-driving GPU) — this is purely the safest RAM posture on
  15 GB with no co-residency swap. Raise to 2–3 only to keep small models hot together
  (E2B+4B+1.7B ≈ 6 GB fits). `sleep-idle-seconds` per preset frees idle models either way.
- **`nice -n 19`**: on 2 cores generation pins the whole CPU — niceness makes inference yield to
  interactive apps (full speed when the desktop is idle).
- **`--cors-origins localhost`**: same-machine profile from llama.cpp PR 25655 — the default CORS
  reflects any Origin, so without this a malicious page in the laptop's own browser could call
  the (keyless) API on 127.0.0.1. Browser-only enforcement: bench.py/CLI clients are unaffected.
- `config.ini` here is the stowed repo copy (`lw/.config/llamacpp/`), so editing it edits the
  repo (no deploy step).

## RAM (the golden rule)

There is no VRAM and no freeze risk. The constraint is plain: **N resident models must fit in
system RAM together.** `bench.py` records each model's RAM footprint (MemAvailable delta) and
the remaining free RAM — keep enough headroom for the desktop (and for co-resident models if you
ever raise `--models-max` above 1). If RAM gets tight: shorten `sleep-idle-seconds`, use smaller
quants, or reduce `ctx-size`. Note: the 4B/E2B presets are small enough to co-reside, but a big
MoE (e.g. Qwen3-30B-A3B at ~12 GB) fills most of 15 GB → it runs **1-at-a-time** regardless.

`threads = 2` (physical cores) on the dense presets — they measure at the practical bandwidth
wall (~8–10 GB/s effective) and the 2026-07-26 sweep confirmed more threads don't lift them.
The **compute-bound** pair runs `threads = 3` (same-run sweep: Granite 4.5→4.8, 30B 1.9→2.0;
`threads = 4` LOSES to HT contention on both). Every preset sets `threads-batch = 4`: prefill
is compute-bound on all models — benched +8% prefill / −9% TTFT with no decode cost, and the
extra HT load only bursts during prefill.

## Tuning cheat-sheet (CPU)

- **Quant size is the speed dial**: decode is bandwidth-bound, so a smaller quant is directly
  faster. Prefer Q4_K_M / Q5_K_M; F16/Q8 are usually quality-overkill AND slow on CPU.
- **`threads`** = **physical** core count (`lscpu` → Core(s)/socket × Socket(s)). Hyperthreads
  rarely help decode; oversubscribing can hurt. Set this per preset (default 4 is a guess).
- **Model size**: stick to ≤ ~8B. **MoE** models (few active params, e.g. Qwen3-A3B) are ideal
  IF they fit RAM — they decode at their *active* size while giving bigger-model quality. There
  is no `n-cpu-moe` tradeoff here — it's all CPU already.
- **MTP / speculative decode** (`spec-type = draft-mtp`): a clear win on the fl GPU (+50–100%) but
  **TESTED AND REJECTED on this box** (2026-07-26): the draft lost the final CPU-pinned
  (`gpu-layers-draft = 0`) same-run A/Bs on both gemmas — E2B 4.0 vs 4.9 t/s at 47% acceptance,
  E4B 2.3 vs 2.4 at 62%. Batched verify triples the FLOPs per weight-read and 2 cores have
  no spare compute, so spec decode can't pay here at any setting. MTP stays an fl-only trick.
  (Only gemma-4 has drafts anyway; Qwen 2507/2.5 have no MTP path — that needs Qwen 3.5/3.6.)
- **KV cache**: leave at default (f16). Quantized KV (`cache-type-*`) adds dequant overhead with
  no VRAM to reclaim → usually slower on CPU. (This is the opposite of the fl config.)
- **iGPU / Vulkan**: the pacman build **does** have a Vulkan backend and by default auto-offloads
  layers to the HD 5500 — that's why every preset pins `n-gpu-layers = 0`. The iGPU shares the
  same DDR3, is slower than the cores, and big allocations die (`ErrorOutOfDeviceMemory`, SIGSEGV
  in llama-cli) — it broke the 30B load outright. Only re-enable offload if a bench proves the
  iGPU wins (it won't).

## Benchmarking

Run **on the laptop** (against its own localhost server):

```
./bench.py                     # full sweep of everything the server lists
./bench.py -m id1,id2          # just the changed presets
./bench.py -n 512 --ctx 8000   # longer gen + a long-context decode column
```

For a clean per-model RAM footprint, restart the server first (with `--models-max 1` only a
re-bench of the *same* still-resident model reads a `~0` RAM Δ, but a restart keeps runs comparable). Results land in
`bench-results.md` (latest) + `bench-history.md` (append-only — diff runs there).

⚠ Absolute numbers swing **±20–30%** with desktop load and thermals (E2B has measured 6.1 idle
vs 4.9 warm/busy). Only rows from the **same run** are directly comparable — A/B via extra
preset ids in one `./bench.py -m a,b` command (see the rig sections in config.ini), never by
comparing across runs. For spec/MTP rows the draft **acceptance %** appears in the spec column.

## Model roster (reference numbers = clean sweep 2026-07-25, n-gpu-layers=0; Granite/30B since
promoted to threads=3 and all presets to threads-batch=4, 2026-07-26)

| model id | decode t/s | prefill t/s | ctx | extras | role |
|---|---|---|---|---|---|
| `gemma-4-E2B-it-UD-Q4_K_XL` | **6.1** | 8 | 4k | vision-capable (mmproj) | fastest — quick Q&A, askllama-class |
| `Qwen3-1.7B` | 4.8 | 11 | 8k | thinking | snappy small tasks |
| `Granite-4.0-H-Tiny` | 4.3–4.8 | 11–14 | 8k | 7B/~1B-active hybrid MoE, threads 3 | ⭐ speed AND brains |
| `Qwen3-4B-Instruct-2507` | 3.9 | 7 | 4k | non-thinking | daily driver |
| `Qwen2.5-Coder-3B-Instruct` | 3.3 | 6 | 8k | temp 0.2 | small coder |
| `Qwen3-30B-A3B-Instruct-2507-UD-IQ3_XXS` | 1.9–2.7 | 4 | 4k | ~12 GB, TTFT ~6 s, threads 3 | ⭐ quality when you can wait |
| `gemma-4-E4B-it-UD-Q4_K_XL` | 2.4–2.5 | 4 | 4k | vision (mmproj) | quality vision; else the 30B is smarter at the same speed |
| `Jan-v3-4b` | 2.1¹ | 4 | 4k | agentic tune | agentic variant of the 4B (~0.9× its speed) |

¹ variance-depressed in the sweep; same-run A/B vs the base (2026-07-26) shows ~0.9×.

## Client wiring

Clients run **on the laptop itself** (the server is localhost-only, see Endpoint above); same
rule as fl otherwise: client model **id = config.ini section name**, client context ≤ server
`ctx-size`. Point it at `http://127.0.0.1:11343/v1` (no key). If LAN access is ever needed,
switch the abbr back to `--host 0.0.0.0` **and add `--api-key`** — never open it keyless.

**Wired (2026-07-26), lw-only via stow host overlays** — these are full-file `--override`
copies of the common configs with an added local provider; when the common client configs
change, mirror the change here:

| tool | overlay file | provider / client id |
|---|---|---|
| aichat | `lw/.config/aichat/config.yaml` | `lw:` (e.g. `lw:gemma-4-E2B-it-UD-Q4_K_XL`) |
| opencode | `lw/.config/opencode/opencode.json` | `lwcpp/` |
| pi | `lw/.pi/agent/models.json` + `settings.json` | `lwcpp` (⚠ pi hides providers not in settings.json `enabledModels`) |

All 8 presets are listed in each, with client context capped to the server `ctx-size` and the
benched t/s in the display names. Defaults still point at the fl LAN models — switch to a local
model in-tool when roaming. Re-run `./install.sh` on lw once so stow links the new files.