dots/fl/.config/vllm/README.md

# vLLM on fl (docker, ROCm) — side by side with the llama.cpp router, never at the same time

Added 2026-10-21 to answer one question: **how does vLLM compare with llama.cpp on this card, on the
same models?** It is a bench rig, not a second production endpoint. Nothing here was run on fl yet —
fl has no sshd, so every command below is for you to run there; the first run will find the rough edges.

- **Box**: the same fl as the router — RX 7600 XT 16 GB (gfx1102, RDNA3), 5600X, 48 GB RAM. The card
  drives the display. llama.cpp runs the Arch Vulkan build; vLLM needs ROCm, which comes **inside the
  docker image** — nothing ROCm is installed on the host.
- **Files**: `vllm.ini` (presets, same `key = --flag` convention as `../llamacpp/config.ini`),
  `vllmctl.py` (start/stop with the VRAM guards), `bench-vs.py` (one table for both engines).
- **Port 11348**. Not in the router, not in any client list (opencode/pi/aichat/OWUI) on purpose.

## ⚠ The one rule: one engine resident at a time

vLLM pre-allocates `gpu-memory-utilization` × 16 GiB at startup and keeps it, idle or not. The router
loads models lazily on request. So while vLLM is up, **any client request to :11343** makes the router
load a model on top of it → overcommit → GTT spill → frozen PC (the same failure `--models-max 1`
guards against). Unloading resident models is not enough; the clients on the LAN are the problem.

`vllmctl.py up` therefore refuses while `llama.service` is active. The sequence is always:

```fish
~/.config/vllm/vllmctl.py up <preset> --stop-router    # sudo systemctl stop llama.service, then start vLLM
# … use / bench vLLM …
~/.config/vllm/vllmctl.py down                         # stop container, wait for VRAM to drain
sudo systemctl start llama.service                     # router back for the LAN
```

The other order is the only one that should fail safely: vLLM compares its requested budget against
free VRAM at startup and aborts when a router model is resident ("free memory … is less than desired
GPU memory utilization" — reported by users, not something verified on this card). Everything else freezes.

## What "same models" means here, and the size surprise

llama.cpp runs GGUF; vLLM runs Hugging Face safetensors (bf16 / AWQ / GPTQ / compressed-tensors).
The GGUF files cannot be loaded natively, so the pairs below are **the same model at a vLLM-native
quant**, not the same files (same-file GGUF loading is phase 2, see the end).

Sizes from the HF API, 2026-10-21 — the "4-bit" vLLM checkpoints are **10–11.5 GB** because embeddings,
lm_head and the vision tower stay bf16. They are bigger than the Q6/Q8 GGUFs they pair with, so on
16 GB they are **clean-desktop only**, at `gpu-memory-utilization` ≈ 0.78 and 16k context:

| pair | llama.cpp preset (ledger) | vLLM preset (`vllm.ini`) | vLLM weights | fits? |
|---|---|---|---|---|
| smoke | — | `Qwen3-0.6B-smoke` = `Qwen/Qwen3-0.6B` | 1.5 GB | yes — proves docker+GPU+image |
| gemma-4-E4B | `gemma-4-E4B-it-UD-Q8_K_XL` — 57 t/s **with MTP**, 96k ctx, ~7 GB | ~~`google/gemma-4-E4B-it-qat-w4a16-ct`~~ | 11.5 GB | **✗ no** (2026-10-21): vLLM needs ~16 GiB before any KV — the bf16 per-layer embedding table (262144 × 42 × 256 × 2 B = 5.6 GB) is held twice. Tombstone in the ini. |
| Qwen3.5-9B | `Qwen3.5-9B-UD-Q6_K_XL` — 32 t/s **with MTP**, 32k ctx, 9 GB | `Qwen3.5-9B-w4a16` = `RedHatAI/Qwen3.5-9B-quantized.w4a16` | 11.0 GB | the last viable pair: 0.75 on a 2.1 GB desktop, 0.78 clean; prefill batch 512 |

**Cannot be compared on this card at all**: every 30B+ MoE (Coder-30B, GLM-4.7-Flash, Qwen3.6-35B,
Ornith, gemma-4-26B), Coder-Next 80B and Qwen3.8-27B — they only fit llama.cpp through `n-cpu-moe`
expert offload into RAM, which vLLM does not have (its MoE CPU-offload RFC died stale; the only
option is whole-weight `cpu_offload_gb`, which is slow). gpt-oss-20b (MXFP4, ~13 GB) is marginal and
its RDNA3 kernel status is unknown — not in the ini.

**Why the "docker should just work" expectation does not hold here**: of the three first-day failures,
one was the image (no gfx1102 kernels → the override, solved) and two were memory. vLLM keeps
everything in VRAM — bf16 embeddings, fp32 logits during profiling, CUDA-graph memory — on a card that
also drives the desktop. No image changes a model's memory need; the ROCm images on Docker Hub are the
right ones.

So the comparison is **not apples to apples** and the table says so: llama.cpp rows run Q8/Q6 with a
speculative MTP draft at a bigger context; vLLM rows run int4 without speculation at 16k. Read it as
"what each engine delivers on this card for this model", not as a kernel shoot-out.

## Setup on fl (once)

```fish
sudo pacman -S docker                          # ~20 GB free disk for the ROCm image + 25 GB for two models
sudo systemctl enable --now docker
sudo usermod -aG docker anon                   # then log out / in (or `newgrp docker`)
cd ~/.dots && ./install.sh                     # links ~/.config/vllm/* (new files are not live until re-stow)
~/.config/vllm/vllmctl.py pull                 # vllm/vllm-openai-rocm:v0.32.0 (ROCm 10.0; 11.6 GB download, ~25 GB on disk)
~/.config/vllm/vllmctl.py gpu-check            # torch sees the card? prints device name + gfx arch
```

`gpu-check` must print `cuda available: True`, a `gfx1100` device (that is the override at work) and
`torch kernel: ok`. **The override is required** (found 2026-10-21): `v0.32.0` carries no gfx1102 code
objects, and a kernel launch without one aborts the process *silently* — the engine core vanishes, the
API server only says `Failed core proc(s): {}`, and even the `print`s before the abort are lost to
stdout buffering. `[global] env.HSA_OVERRIDE_GFX_VERSION = 11.0.0` presents the card as gfx1100 (RX 7900),
whose RDNA3 code runs on this chip. Nothing else on the host changes.

Then the smoke test (tiny model, 30 % of the card, proves the whole path before any big download —
✓ passed 2026-10-21: 154.9 t/s decode, 1462 t/s prefill, TTFT 0.02 s, 7.7 GB used incl. a 2.9 GB desktop):

```fish
~/.config/vllm/vllmctl.py up Qwen3-0.6B-smoke --stop-router
curl -s localhost:11348/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"Qwen3-0.6B-smoke","messages":[{"role":"user","content":"hi"}],"max_tokens":20}'
~/.config/vllm/vllmctl.py down; sudo systemctl start llama.service
```

First start of any model compiles Triton kernels: minutes, with `vllmctl.py up` echoing the log tail
every 30 s.

**Tags and updates.** `<tag>` is the part after the colon: `latest` = newest stable release (was
`v0.32.0` on 2026-10-18, ROCm 10.0), `nightly` = today's main branch (18 GB), `latest-rocm72` /
`nightly-rocm72` = the same on the older ROCm 7.2 stack (smaller, 12 GB), `…-base` = without vLLM
itself. The ini pins `v0.32.0` so a re-pull cannot silently change the engine under a bench run. To
update: pick the new `vX.Y.Z` on hub.docker.com/r/vllm/vllm-openai-rocm/tags, put it in `[global] image`,
`vllmctl.py pull`, then `vllmctl.py down` / `up` — the container is recreated from the image on every
`up`. Afterwards `docker image prune -f` drops the superseded layers (`docker images` shows what is kept).
`HF_TOKEN` is only needed for gated repos; none of the three presets is gated. If one ever is,
`hf auth login` on fl writes `~/.cache/huggingface/token`, which the mounted cache hands to the container.

## Running the benchmark

Two phases, because the engines cannot be resident together. Do both on a **clean desktop** (close the
browser; `vramx` must show ≤ ~1.5 GB used) — the vLLM presets leave ~2 GB free at 0.78 and an evening
desktop does not fit.

```fish
# phase 1 — vLLM (router stopped by --stop-router; first run also downloads the model)
~/.config/vllm/vllmctl.py up gemma-4-E4B-it-w4a16 --stop-router
~/.config/vllm/bench-vs.py -t vllm:gemma-4-E4B-it-w4a16 --ctx 8000 --concurrency 4
~/.config/vllm/vllmctl.py down
~/.config/vllm/vllmctl.py up Qwen3.5-9B-w4a16            # router is still stopped
~/.config/vllm/bench-vs.py -t vllm:Qwen3.5-9B-w4a16 --ctx 8000 --concurrency 4
~/.config/vllm/vllmctl.py down

# phase 2 — llama.cpp (vLLM down, router back)
sudo systemctl start llama.service
~/.config/vllm/bench-vs.py -t llamacpp:gemma-4-E4B-it-UD-Q8_K_XL -t llamacpp:Qwen3.5-9B-UD-Q6_K_XL --ctx 8000 --concurrency 4
```

Every row lands in `bench-vs-history.md` (append) and the latest run in `bench-vs-results.md`, same
folder. The guards: a `llamacpp` target is skipped while vLLM answers `/health`; a `vllm` target is
skipped while the router has a model loaded or `llama.service` is active (`--force` overrides both).

**How to read it**

- `decode t/s` is usage tokens over wall time after the first token, measured the same way on both
  engines; llama.cpp's own `timings` number is in parentheses so rows match `../llamacpp/bench-results.md`.
- Thinking is **off** on both (`chat_template_kwargs.enable_thinking=false`) and a reasoning delta counts
  as first token, so TTFT compares the engines, not the preamble lengths. `--think` reverts that.
- `spec` shows the MTP draft on the llama.cpp side; vLLM rows have none unless the preset sets
  `speculative-config`. Expect the gemma/Qwen llama.cpp rows to carry a 1.5–2× draft bonus.
- `agg t/s @N` is continuous batching's home turf — and llama-server still won it (85 vs 61 at 4
  streams): its default slot count on b11146 is evidently > 1, so no `parallel` key is needed.
- `--ctx 8000` adds decode at an 8k prompt — the number that matters for agent loops.

## Results

**2026-10-21, Qwen3.5-9B, same evening, same script, thinking off, 256 tok, 0.6 GB desktop:**

| engine · checkpoint | spec | decode t/s | prefill t/s | TTFT s | decode @8k prompt | 4 streams agg t/s | VRAM used |
|---|---|---|---|---|---|---|---|
| vLLM · RedHatAI w4a16 @0.80, 16k, batch 512 | — | 29.4 | **181** | **0.17** | 16.0 | 60.6 | 12.5 GB |
| llama.cpp · UD-Q6_K_XL, 32k | MTP draft | **61.3** | 114 | 0.27 | **51.1** | **85.4** | 10.9 GB |

What it says, for this card:

- **Decode**: llama.cpp wins 2× single-stream and 3× at an 8k prompt. Most of the 2× is the MTP draft
  (the ledger's pre-MTP Q8 number was 17 t/s in July; a no-draft Q6 today is probably ~30, i.e. parity
  with vLLM). The 8k gap is not MTP: vLLM's decode halves with context here (16.0 vs 29.4) while
  llama.cpp loses 17 % — on RDNA3 vLLM's attention path is the Triton generic one, llama.cpp's is a
  tuned Vulkan kernel.
- **Prefill**: vLLM +59 % even with its batch capped at 512 for memory. This is its real strength on
  the box, and it shows in TTFT.
- **Concurrency**: the column that was supposed to be vLLM's — it batched 4 streams to 2× its single
  (60.6), but llama-server served the same 4 at 85.4. Its default slot count is evidently > 1 on this
  build, so the README's "llama.cpp serializes" caveat does not hold; it does not need a `parallel` key.
- **Memory**: vLLM holds 1.6 GB more for a model with 2 GB less context, and only runs on a ≤ 0.6 GB
  desktop. 0.75 was 0.11 GiB short of KV room; 0.80 is the setting.

**Verdict**: no vLLM model earns a roster slot on fl; the bench rig stays, the router stays, llama-swap
is not needed. The one follow-up that could move the decode column is MTP in vLLM (the knob is in
the ini) — but it cannot touch the memory column or the MoE roster, so it is curiosity, not a decision.

## When it does not start

- **`No available memory for the cache blocks`** (seen 2026-10-21, gemma @0.73/8k on a 2.1 GB desktop):
  the model loaded, but budget − weights − activation reservations left nothing for KV. What vLLM
  reserved is in the log: `docker logs vllm 2>&1 | grep -iE 'profil|kv cache|weights take|peak'`.
  Cheapest fixes first: text-only (`limit-mm-per-prompt` all zero — skips the image/audio encoder
  warmup, now in both presets), smaller `max-num-batched-tokens` (2048, now in both presets), lower
  `max-num-seqs`; then a cleaner desktop so the projection allows 0.78–0.82; last resort the smaller
  GPTQ checkpoint. `--set key=val` tries any of these without editing the ini.
- **`ABORT: projected free …`**: the desktop is using more than the preset assumes. Close GPU apps or
  take the suggested utilization it prints. `--force` exists; the freeze is real.
- **compressed-tensors / "no kernel" for int4 on ROCm**: both real presets use compressed-tensors
  pack-quantized. If this image lacks a ROCm kernel for it, swap in the GPTQ alternative noted in the
  ini (`Vishva007/gemma-4-E4B-it-W4A16-AutoRound-GPTQ`, 10.0 GB) — different kernel path.
- **Hangs / garbage output**: RDNA3 attention backend. Set `env.VLLM_ATTENTION_BACKEND = TRITON_ATTN`
  explicitly; option names moved between versions — `vllmctl.py exec vllm serve --help` shows this image's.
- **GTT growing during a run** (`vramx`, the GTT line): the budget is leaking into system RAM — stop,
  lower the utilization. Do not wait for the desktop to stutter.
- **Root-owned files in `~/.cache/huggingface`**: the container runs as root, so its downloads land
  root-owned in anon's HF cache. Harmless for the rig; `sudo chown -R anon: ~/.cache/huggingface` if
  the host-side `hf` tooling complains.
- **Container died**: `docker logs vllm` (the container is kept after a crash; `vllmctl.py up` shows
  the tail and removes it on the next start).

## Phase 2 — if the numbers justify more

1. **Same-file GGUF in vLLM**: GGUF loading moved to an out-of-tree `vllm-gguf-plugin`; it is
   "highly experimental and under-optimized" per vLLM's docs and needs a custom image layer
   (`pip install vllm-gguf-plugin`), plus `--tokenizer <base repo>`. Untested on ROCm. It would make the
   weights identical — and show vLLM's unoptimized path, so interpret carefully.
2. **MTP in vLLM**: `RedHatAI/Qwen3.5-9B-quantized.w4a16` ships `model_mtp.safetensors`; the
   commented `speculative-config` in the ini is the knob (syntax per this image's `--help`).
3. **Permanent coexistence**: replace the router's `--models-preset` mode with llama-swap (AUR
   `llama-swap-bin`), which spawns llama-server *or* the vLLM container per model id, one at a time,
   with a `ttl` idle unload. Same host/port/model ids for every client. Only worth it if a vLLM model
   earns a place in the daily roster.