vLLM on fl (docker, ROCm) — side by side with the llama.cpp router, never at the same time

Added 2026-10-21 to answer one question: how does vLLM compare with llama.cpp on this card, on the same models? It is a bench rig, not a second production endpoint. Nothing here was run on fl yet — fl has no sshd, so every command below is for you to run there; the first run will find the rough edges.

Box: the same fl as the router — RX 7600 XT 16 GB (gfx1102, RDNA3), 5600X, 48 GB RAM. The card drives the display. llama.cpp runs the Arch Vulkan build; vLLM needs ROCm, which comes inside the docker image — nothing ROCm is installed on the host.
Files: vllm.ini (presets, same key = --flag convention as ../llamacpp/config.ini), vllmctl.py (start/stop with the VRAM guards), bench-vs.py (one table for both engines).
Port 11348. Not in the router, not in any client list (opencode/pi/aichat/OWUI) on purpose.

⚠ The one rule: one engine resident at a time

vLLM pre-allocates gpu-memory-utilization × 16 GiB at startup and keeps it, idle or not. The router loads models lazily on request. So while vLLM is up, any client request to :11343 makes the router load a model on top of it → overcommit → GTT spill → frozen PC (the same failure --models-max 1 guards against). Unloading resident models is not enough; the clients on the LAN are the problem.

vllmctl.py up therefore refuses while llama.service is active. The sequence is always:

~/.config/vllm/vllmctl.py up <preset> --stop-router    # sudo systemctl stop llama.service, then start vLLM
# … use / bench vLLM …
~/.config/vllm/vllmctl.py down                         # stop container, wait for VRAM to drain
sudo systemctl start llama.service                     # router back for the LAN

The other order is the only one that should fail safely: vLLM compares its requested budget against free VRAM at startup and aborts when a router model is resident ("free memory … is less than desired GPU memory utilization" — reported by users, not something verified on this card). Everything else freezes.

What "same models" means here, and the size surprise

llama.cpp runs GGUF; vLLM runs Hugging Face safetensors (bf16 / AWQ / GPTQ / compressed-tensors). The GGUF files cannot be loaded natively, so the pairs below are the same model at a vLLM-native quant, not the same files (same-file GGUF loading is phase 2, see the end).

Sizes from the HF API, 2026-10-21 — the "4-bit" vLLM checkpoints are 10–11.5 GB because embeddings, lm_head and the vision tower stay bf16. They are bigger than the Q6/Q8 GGUFs they pair with, so on 16 GB they are clean-desktop only, at gpu-memory-utilization ≈ 0.78 and 16k context:

pair	llama.cpp preset (ledger)	vLLM preset (`vllm.ini`)	vLLM weights	fits?
smoke	—	`Qwen3-0.6B-smoke` = `Qwen/Qwen3-0.6B`	1.5 GB	yes — proves docker+GPU+image
gemma-4-E4B	`gemma-4-E4B-it-UD-Q8_K_XL` — 57 t/s with MTP, 96k ctx, ~7 GB	~~`google/gemma-4-E4B-it-qat-w4a16-ct`~~	11.5 GB	✗ no (2026-10-21): vLLM needs ~16 GiB before any KV — the bf16 per-layer embedding table (262144 × 42 × 256 × 2 B = 5.6 GB) is held twice. Tombstone in the ini.
Qwen3.5-9B	`Qwen3.5-9B-UD-Q6_K_XL` — 32 t/s with MTP, 32k ctx, 9 GB	`Qwen3.5-9B-w4a16` = `RedHatAI/Qwen3.5-9B-quantized.w4a16`	11.0 GB	the last viable pair: 0.75 on a 2.1 GB desktop, 0.78 clean; prefill batch 512

Cannot be compared on this card at all: every 30B+ MoE (Coder-30B, GLM-4.7-Flash, Qwen3.6-35B, Ornith, gemma-4-26B), Coder-Next 80B and Qwen3.8-27B — they only fit llama.cpp through n-cpu-moe expert offload into RAM, which vLLM does not have (its MoE CPU-offload RFC died stale; the only option is whole-weight cpu_offload_gb, which is slow). gpt-oss-20b (MXFP4, ~13 GB) is marginal and its RDNA3 kernel status is unknown — not in the ini.

Why the "docker should just work" expectation does not hold here: of the three first-day failures, one was the image (no gfx1102 kernels → the override, solved) and two were memory. vLLM keeps everything in VRAM — bf16 embeddings, fp32 logits during profiling, CUDA-graph memory — on a card that also drives the desktop. No image changes a model's memory need; the ROCm images on Docker Hub are the right ones.

So the comparison is not apples to apples and the table says so: llama.cpp rows run Q8/Q6 with a speculative MTP draft at a bigger context; vLLM rows run int4 without speculation at 16k. Read it as "what each engine delivers on this card for this model", not as a kernel shoot-out.

Setup on fl (once)

sudo pacman -S docker                          # ~20 GB free disk for the ROCm image + 25 GB for two models
sudo systemctl enable --now docker
sudo usermod -aG docker anon                   # then log out / in (or `newgrp docker`)
cd ~/.dots && ./install.sh                     # links ~/.config/vllm/* (new files are not live until re-stow)
~/.config/vllm/vllmctl.py pull                 # vllm/vllm-openai-rocm:v0.32.0 (ROCm 10.0; 11.6 GB download, ~25 GB on disk)
~/.config/vllm/vllmctl.py gpu-check            # torch sees the card? prints device name + gfx arch

gpu-check must print cuda available: True, a gfx1100 device (that is the override at work) and torch kernel: ok. The override is required (found 2026-10-21): v0.32.0 carries no gfx1102 code objects, and a kernel launch without one aborts the process silently — the engine core vanishes, the API server only says Failed core proc(s): {}, and even the prints before the abort are lost to stdout buffering. [global] env.HSA_OVERRIDE_GFX_VERSION = 11.0.0 presents the card as gfx1100 (RX 7900), whose RDNA3 code runs on this chip. Nothing else on the host changes.

Then the smoke test (tiny model, 30 % of the card, proves the whole path before any big download — ✓ passed 2026-10-21: 154.9 t/s decode, 1462 t/s prefill, TTFT 0.02 s, 7.7 GB used incl. a 2.9 GB desktop):

~/.config/vllm/vllmctl.py up Qwen3-0.6B-smoke --stop-router
curl -s localhost:11348/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"Qwen3-0.6B-smoke","messages":[{"role":"user","content":"hi"}],"max_tokens":20}'
~/.config/vllm/vllmctl.py down; sudo systemctl start llama.service

First start of any model compiles Triton kernels: minutes, with vllmctl.py up echoing the log tail every 30 s.

Tags and updates. <tag> is the part after the colon: latest = newest stable release (was v0.32.0 on 2026-10-18, ROCm 10.0), nightly = today's main branch (18 GB), latest-rocm72 / nightly-rocm72 = the same on the older ROCm 7.2 stack (smaller, 12 GB), …-base = without vLLM itself. The ini pins v0.32.0 so a re-pull cannot silently change the engine under a bench run. To update: pick the new vX.Y.Z on hub.docker.com/r/vllm/vllm-openai-rocm/tags, put it in [global] image, vllmctl.py pull, then vllmctl.py down / up — the container is recreated from the image on every up. Afterwards docker image prune -f drops the superseded layers (docker images shows what is kept). HF_TOKEN is only needed for gated repos; none of the three presets is gated. If one ever is, hf auth login on fl writes ~/.cache/huggingface/token, which the mounted cache hands to the container.

Running the benchmark

Two phases, because the engines cannot be resident together. Do both on a clean desktop (close the browser; vramx must show ≤ ~1.5 GB used) — the vLLM presets leave ~2 GB free at 0.78 and an evening desktop does not fit.

# phase 1 — vLLM (router stopped by --stop-router; first run also downloads the model)
~/.config/vllm/vllmctl.py up gemma-4-E4B-it-w4a16 --stop-router
~/.config/vllm/bench-vs.py -t vllm:gemma-4-E4B-it-w4a16 --ctx 8000 --concurrency 4
~/.config/vllm/vllmctl.py down
~/.config/vllm/vllmctl.py up Qwen3.5-9B-w4a16            # router is still stopped
~/.config/vllm/bench-vs.py -t vllm:Qwen3.5-9B-w4a16 --ctx 8000 --concurrency 4
~/.config/vllm/vllmctl.py down

# phase 2 — llama.cpp (vLLM down, router back)
sudo systemctl start llama.service
~/.config/vllm/bench-vs.py -t llamacpp:gemma-4-E4B-it-UD-Q8_K_XL -t llamacpp:Qwen3.5-9B-UD-Q6_K_XL --ctx 8000 --concurrency 4

Every row lands in bench-vs-history.md (append) and the latest run in bench-vs-results.md, same folder. The guards: a llamacpp target is skipped while vLLM answers /health; a vllm target is skipped while the router has a model loaded or llama.service is active (--force overrides both).

How to read it

decode t/s is usage tokens over wall time after the first token, measured the same way on both engines; llama.cpp's own timings number is in parentheses so rows match ../llamacpp/bench-results.md.
Thinking is off on both (chat_template_kwargs.enable_thinking=false) and a reasoning delta counts as first token, so TTFT compares the engines, not the preamble lengths. --think reverts that.
spec shows the MTP draft on the llama.cpp side; vLLM rows have none unless the preset sets speculative-config. Expect the gemma/Qwen llama.cpp rows to carry a 1.5–2× draft bonus.
agg t/s @N is continuous batching's home turf — and llama-server still won it (85 vs 61 at 4 streams): its default slot count on b11146 is evidently > 1, so no parallel key is needed.
--ctx 8000 adds decode at an 8k prompt — the number that matters for agent loops.

Results

2026-10-21, Qwen3.5-9B, same evening, same script, thinking off, 256 tok, 0.6 GB desktop:

engine · checkpoint	spec	decode t/s	prefill t/s	TTFT s	decode @8k prompt	4 streams agg t/s	VRAM used
vLLM · RedHatAI w4a16 @0.80, 16k, batch 512	—	29.4	181	0.17	16.0	60.6	12.5 GB
llama.cpp · UD-Q6_K_XL, 32k	MTP draft	61.3	114	0.27	51.1	85.4	10.9 GB

What it says, for this card:

Decode: llama.cpp wins 2× single-stream and 3× at an 8k prompt. Most of the 2× is the MTP draft (the ledger's pre-MTP Q8 number was 17 t/s in July; a no-draft Q6 today is probably ~30, i.e. parity with vLLM). The 8k gap is not MTP: vLLM's decode halves with context here (16.0 vs 29.4) while llama.cpp loses 17 % — on RDNA3 vLLM's attention path is the Triton generic one, llama.cpp's is a tuned Vulkan kernel.
Prefill: vLLM +59 % even with its batch capped at 512 for memory. This is its real strength on the box, and it shows in TTFT.
Concurrency: the column that was supposed to be vLLM's — it batched 4 streams to 2× its single (60.6), but llama-server served the same 4 at 85.4. Its default slot count is evidently > 1 on this build, so the README's "llama.cpp serializes" caveat does not hold; it does not need a parallel key.
Memory: vLLM holds 1.6 GB more for a model with 2 GB less context, and only runs on a ≤ 0.6 GB desktop. 0.75 was 0.11 GiB short of KV room; 0.80 is the setting.

Verdict: no vLLM model earns a roster slot on fl; the bench rig stays, the router stays, llama-swap is not needed. The one follow-up that could move the decode column is MTP in vLLM (the knob is in the ini) — but it cannot touch the memory column or the MoE roster, so it is curiosity, not a decision.

When it does not start

No available memory for the cache blocks (seen 2026-10-21, gemma @0.73/8k on a 2.1 GB desktop): the model loaded, but budget − weights − activation reservations left nothing for KV. What vLLM reserved is in the log: docker logs vllm 2>&1 | grep -iE 'profil|kv cache|weights take|peak'. Cheapest fixes first: text-only (limit-mm-per-prompt all zero — skips the image/audio encoder warmup, now in both presets), smaller max-num-batched-tokens (2048, now in both presets), lower max-num-seqs; then a cleaner desktop so the projection allows 0.78–0.82; last resort the smaller GPTQ checkpoint. --set key=val tries any of these without editing the ini.
ABORT: projected free …: the desktop is using more than the preset assumes. Close GPU apps or take the suggested utilization it prints. --force exists; the freeze is real.
compressed-tensors / "no kernel" for int4 on ROCm: both real presets use compressed-tensors pack-quantized. If this image lacks a ROCm kernel for it, swap in the GPTQ alternative noted in the ini (Vishva007/gemma-4-E4B-it-W4A16-AutoRound-GPTQ, 10.0 GB) — different kernel path.
Hangs / garbage output: RDNA3 attention backend. Set env.VLLM_ATTENTION_BACKEND = TRITON_ATTN explicitly; option names moved between versions — vllmctl.py exec vllm serve --help shows this image's.
GTT growing during a run (vramx, the GTT line): the budget is leaking into system RAM — stop, lower the utilization. Do not wait for the desktop to stutter.
Root-owned files in ~/.cache/huggingface: the container runs as root, so its downloads land root-owned in anon's HF cache. Harmless for the rig; sudo chown -R anon: ~/.cache/huggingface if the host-side hf tooling complains.
Container died: docker logs vllm (the container is kept after a crash; vllmctl.py up shows the tail and removes it on the next start).

Phase 2 — if the numbers justify more

Same-file GGUF in vLLM: GGUF loading moved to an out-of-tree vllm-gguf-plugin; it is "highly experimental and under-optimized" per vLLM's docs and needs a custom image layer (pip install vllm-gguf-plugin), plus --tokenizer <base repo>. Untested on ROCm. It would make the weights identical — and show vLLM's unoptimized path, so interpret carefully.
MTP in vLLM: RedHatAI/Qwen3.5-9B-quantized.w4a16 ships model_mtp.safetensors; the commented speculative-config in the ini is the knob (syntax per this image's --help).
Permanent coexistence: replace the router's --models-preset mode with llama-swap (AUR llama-swap-bin), which spawns llama-server or the vLLM container per model id, one at a time, with a ttl idle unload. Same host/port/model ids for every client. Only worth it if a vLLM model earns a place in the daily roster.

14 KiB Raw Blame History Unescape Escape