[Sync] Lua Hyprland chain, licence cleanup

fl/.config/vllm/README.md

@@ -0,0 +1,206 @@
+# vLLM on fl (docker, ROCm) — side by side with the llama.cpp router, never at the same time
+
+Added 2026-10-21 to answer one question: **how does vLLM compare with llama.cpp on this card, on the
+same models?** It is a bench rig, not a second production endpoint. Nothing here was run on fl yet —
+fl has no sshd, so every command below is for you to run there; the first run will find the rough edges.
+
+- **Box**: the same fl as the router — RX 7600 XT 16 GB (gfx1102, RDNA3), 5600X, 48 GB RAM. The card
+  drives the display. llama.cpp runs the Arch Vulkan build; vLLM needs ROCm, which comes **inside the
+  docker image** — nothing ROCm is installed on the host.
+- **Files**: `vllm.ini` (presets, same `key = --flag` convention as `../llamacpp/config.ini`),
+  `vllmctl.py` (start/stop with the VRAM guards), `bench-vs.py` (one table for both engines).
+- **Port 11348**. Not in the router, not in any client list (opencode/pi/aichat/OWUI) on purpose.
+
+## ⚠ The one rule: one engine resident at a time
+
+vLLM pre-allocates `gpu-memory-utilization` × 16 GiB at startup and keeps it, idle or not. The router
+loads models lazily on request. So while vLLM is up, **any client request to :11343** makes the router
+load a model on top of it → overcommit → GTT spill → frozen PC (the same failure `--models-max 1`
+guards against). Unloading resident models is not enough; the clients on the LAN are the problem.
+
+`vllmctl.py up` therefore refuses while `llama.service` is active. The sequence is always:
+
+```fish
+~/.config/vllm/vllmctl.py up <preset> --stop-router    # sudo systemctl stop llama.service, then start vLLM
+# … use / bench vLLM …
+~/.config/vllm/vllmctl.py down                         # stop container, wait for VRAM to drain
+sudo systemctl start llama.service                     # router back for the LAN
+```
+
+The other order is the only one that should fail safely: vLLM compares its requested budget against
+free VRAM at startup and aborts when a router model is resident ("free memory … is less than desired
+GPU memory utilization" — reported by users, not something verified on this card). Everything else freezes.
+
+## What "same models" means here, and the size surprise
+
+llama.cpp runs GGUF; vLLM runs Hugging Face safetensors (bf16 / AWQ / GPTQ / compressed-tensors).
+The GGUF files cannot be loaded natively, so the pairs below are **the same model at a vLLM-native
+quant**, not the same files (same-file GGUF loading is phase 2, see the end).
+
+Sizes from the HF API, 2026-10-21 — the "4-bit" vLLM checkpoints are **10–11.5 GB** because embeddings,
+lm_head and the vision tower stay bf16. They are bigger than the Q6/Q8 GGUFs they pair with, so on
+16 GB they are **clean-desktop only**, at `gpu-memory-utilization` ≈ 0.78 and 16k context:
+
+| pair | llama.cpp preset (ledger) | vLLM preset (`vllm.ini`) | vLLM weights | fits? |
+|---|---|---|---|---|
+| smoke | — | `Qwen3-0.6B-smoke` = `Qwen/Qwen3-0.6B` | 1.5 GB | yes — proves docker+GPU+image |
+| gemma-4-E4B | `gemma-4-E4B-it-UD-Q8_K_XL` — 57 t/s **with MTP**, 96k ctx, ~7 GB | ~~`google/gemma-4-E4B-it-qat-w4a16-ct`~~ | 11.5 GB | **✗ no** (2026-10-21): vLLM needs ~16 GiB before any KV — the bf16 per-layer embedding table (262144 × 42 × 256 × 2 B = 5.6 GB) is held twice. Tombstone in the ini. |
+| Qwen3.5-9B | `Qwen3.5-9B-UD-Q6_K_XL` — 32 t/s **with MTP**, 32k ctx, 9 GB | `Qwen3.5-9B-w4a16` = `RedHatAI/Qwen3.5-9B-quantized.w4a16` | 11.0 GB | the last viable pair: 0.75 on a 2.1 GB desktop, 0.78 clean; prefill batch 512 |
+
+**Cannot be compared on this card at all**: every 30B+ MoE (Coder-30B, GLM-4.7-Flash, Qwen3.6-35B,
+Ornith, gemma-4-26B), Coder-Next 80B and Qwen3.8-27B — they only fit llama.cpp through `n-cpu-moe`
+expert offload into RAM, which vLLM does not have (its MoE CPU-offload RFC died stale; the only
+option is whole-weight `cpu_offload_gb`, which is slow). gpt-oss-20b (MXFP4, ~13 GB) is marginal and
+its RDNA3 kernel status is unknown — not in the ini.
+
+**Why the "docker should just work" expectation does not hold here**: of the three first-day failures,
+one was the image (no gfx1102 kernels → the override, solved) and two were memory. vLLM keeps
+everything in VRAM — bf16 embeddings, fp32 logits during profiling, CUDA-graph memory — on a card that
+also drives the desktop. No image changes a model's memory need; the ROCm images on Docker Hub are the
+right ones.
+
+So the comparison is **not apples to apples** and the table says so: llama.cpp rows run Q8/Q6 with a
+speculative MTP draft at a bigger context; vLLM rows run int4 without speculation at 16k. Read it as
+"what each engine delivers on this card for this model", not as a kernel shoot-out.
+
+## Setup on fl (once)
+
+```fish
+sudo pacman -S docker                          # ~20 GB free disk for the ROCm image + 25 GB for two models
+sudo systemctl enable --now docker
+sudo usermod -aG docker anon                   # then log out / in (or `newgrp docker`)
+cd ~/.dots && ./install.sh                     # links ~/.config/vllm/* (new files are not live until re-stow)
+~/.config/vllm/vllmctl.py pull                 # vllm/vllm-openai-rocm:v0.32.0 (ROCm 10.0; 11.6 GB download, ~25 GB on disk)
+~/.config/vllm/vllmctl.py gpu-check            # torch sees the card? prints device name + gfx arch
+```
+
+`gpu-check` must print `cuda available: True`, a `gfx1100` device (that is the override at work) and
+`torch kernel: ok`. **The override is required** (found 2026-10-21): `v0.32.0` carries no gfx1102 code
+objects, and a kernel launch without one aborts the process *silently* — the engine core vanishes, the
+API server only says `Failed core proc(s): {}`, and even the `print`s before the abort are lost to
+stdout buffering. `[global] env.HSA_OVERRIDE_GFX_VERSION = 11.0.0` presents the card as gfx1100 (RX 7900),
+whose RDNA3 code runs on this chip. Nothing else on the host changes.
+
+Then the smoke test (tiny model, 30 % of the card, proves the whole path before any big download —
+✓ passed 2026-10-21: 154.9 t/s decode, 1462 t/s prefill, TTFT 0.02 s, 7.7 GB used incl. a 2.9 GB desktop):
+
+```fish
+~/.config/vllm/vllmctl.py up Qwen3-0.6B-smoke --stop-router
+curl -s localhost:11348/v1/chat/completions -H 'content-type: application/json' \
+  -d '{"model":"Qwen3-0.6B-smoke","messages":[{"role":"user","content":"hi"}],"max_tokens":20}'
+~/.config/vllm/vllmctl.py down; sudo systemctl start llama.service
+```
+
+First start of any model compiles Triton kernels: minutes, with `vllmctl.py up` echoing the log tail
+every 30 s.
+
+**Tags and updates.** `<tag>` is the part after the colon: `latest` = newest stable release (was
+`v0.32.0` on 2026-10-18, ROCm 10.0), `nightly` = today's main branch (18 GB), `latest-rocm72` /
+`nightly-rocm72` = the same on the older ROCm 7.2 stack (smaller, 12 GB), `…-base` = without vLLM
+itself. The ini pins `v0.32.0` so a re-pull cannot silently change the engine under a bench run. To
+update: pick the new `vX.Y.Z` on hub.docker.com/r/vllm/vllm-openai-rocm/tags, put it in `[global] image`,
+`vllmctl.py pull`, then `vllmctl.py down` / `up` — the container is recreated from the image on every
+`up`. Afterwards `docker image prune -f` drops the superseded layers (`docker images` shows what is kept).
+`HF_TOKEN` is only needed for gated repos; none of the three presets is gated. If one ever is,
+`hf auth login` on fl writes `~/.cache/huggingface/token`, which the mounted cache hands to the container.
+
+## Running the benchmark
+
+Two phases, because the engines cannot be resident together. Do both on a **clean desktop** (close the
+browser; `vramx` must show ≤ ~1.5 GB used) — the vLLM presets leave ~2 GB free at 0.78 and an evening
+desktop does not fit.
+
+```fish
+# phase 1 — vLLM (router stopped by --stop-router; first run also downloads the model)
+~/.config/vllm/vllmctl.py up gemma-4-E4B-it-w4a16 --stop-router
+~/.config/vllm/bench-vs.py -t vllm:gemma-4-E4B-it-w4a16 --ctx 8000 --concurrency 4
+~/.config/vllm/vllmctl.py down
+~/.config/vllm/vllmctl.py up Qwen3.5-9B-w4a16            # router is still stopped
+~/.config/vllm/bench-vs.py -t vllm:Qwen3.5-9B-w4a16 --ctx 8000 --concurrency 4
+~/.config/vllm/vllmctl.py down
+
+# phase 2 — llama.cpp (vLLM down, router back)
+sudo systemctl start llama.service
+~/.config/vllm/bench-vs.py -t llamacpp:gemma-4-E4B-it-UD-Q8_K_XL -t llamacpp:Qwen3.5-9B-UD-Q6_K_XL --ctx 8000 --concurrency 4
+```
+
+Every row lands in `bench-vs-history.md` (append) and the latest run in `bench-vs-results.md`, same
+folder. The guards: a `llamacpp` target is skipped while vLLM answers `/health`; a `vllm` target is
+skipped while the router has a model loaded or `llama.service` is active (`--force` overrides both).
+
+**How to read it**
+
+- `decode t/s` is usage tokens over wall time after the first token, measured the same way on both
+  engines; llama.cpp's own `timings` number is in parentheses so rows match `../llamacpp/bench-results.md`.
+- Thinking is **off** on both (`chat_template_kwargs.enable_thinking=false`) and a reasoning delta counts
+  as first token, so TTFT compares the engines, not the preamble lengths. `--think` reverts that.
+- `spec` shows the MTP draft on the llama.cpp side; vLLM rows have none unless the preset sets
+  `speculative-config`. Expect the gemma/Qwen llama.cpp rows to carry a 1.5–2× draft bonus.
+- `agg t/s @N` is continuous batching's home turf — and llama-server still won it (85 vs 61 at 4
+  streams): its default slot count on b11146 is evidently > 1, so no `parallel` key is needed.
+- `--ctx 8000` adds decode at an 8k prompt — the number that matters for agent loops.
+
+## Results
+
+**2026-10-21, Qwen3.5-9B, same evening, same script, thinking off, 256 tok, 0.6 GB desktop:**
+
+| engine · checkpoint | spec | decode t/s | prefill t/s | TTFT s | decode @8k prompt | 4 streams agg t/s | VRAM used |
+|---|---|---|---|---|---|---|---|
+| vLLM · RedHatAI w4a16 @0.80, 16k, batch 512 | — | 29.4 | **181** | **0.17** | 16.0 | 60.6 | 12.5 GB |
+| llama.cpp · UD-Q6_K_XL, 32k | MTP draft | **61.3** | 114 | 0.27 | **51.1** | **85.4** | 10.9 GB |
+
+What it says, for this card:
+
+- **Decode**: llama.cpp wins 2× single-stream and 3× at an 8k prompt. Most of the 2× is the MTP draft
+  (the ledger's pre-MTP Q8 number was 17 t/s in July; a no-draft Q6 today is probably ~30, i.e. parity
+  with vLLM). The 8k gap is not MTP: vLLM's decode halves with context here (16.0 vs 29.4) while
+  llama.cpp loses 17 % — on RDNA3 vLLM's attention path is the Triton generic one, llama.cpp's is a
+  tuned Vulkan kernel.
+- **Prefill**: vLLM +59 % even with its batch capped at 512 for memory. This is its real strength on
+  the box, and it shows in TTFT.
+- **Concurrency**: the column that was supposed to be vLLM's — it batched 4 streams to 2× its single
+  (60.6), but llama-server served the same 4 at 85.4. Its default slot count is evidently > 1 on this
+  build, so the README's "llama.cpp serializes" caveat does not hold; it does not need a `parallel` key.
+- **Memory**: vLLM holds 1.6 GB more for a model with 2 GB less context, and only runs on a ≤ 0.6 GB
+  desktop. 0.75 was 0.11 GiB short of KV room; 0.80 is the setting.
+
+**Verdict**: no vLLM model earns a roster slot on fl; the bench rig stays, the router stays, llama-swap
+is not needed. The one follow-up that could move the decode column is MTP in vLLM (the knob is in
+the ini) — but it cannot touch the memory column or the MoE roster, so it is curiosity, not a decision.
+
+## When it does not start
+
+- **`No available memory for the cache blocks`** (seen 2026-10-21, gemma @0.73/8k on a 2.1 GB desktop):
+  the model loaded, but budget − weights − activation reservations left nothing for KV. What vLLM
+  reserved is in the log: `docker logs vllm 2>&1 | grep -iE 'profil|kv cache|weights take|peak'`.
+  Cheapest fixes first: text-only (`limit-mm-per-prompt` all zero — skips the image/audio encoder
+  warmup, now in both presets), smaller `max-num-batched-tokens` (2048, now in both presets), lower
+  `max-num-seqs`; then a cleaner desktop so the projection allows 0.78–0.82; last resort the smaller
+  GPTQ checkpoint. `--set key=val` tries any of these without editing the ini.
+- **`ABORT: projected free …`**: the desktop is using more than the preset assumes. Close GPU apps or
+  take the suggested utilization it prints. `--force` exists; the freeze is real.
+- **compressed-tensors / "no kernel" for int4 on ROCm**: both real presets use compressed-tensors
+  pack-quantized. If this image lacks a ROCm kernel for it, swap in the GPTQ alternative noted in the
+  ini (`Vishva007/gemma-4-E4B-it-W4A16-AutoRound-GPTQ`, 10.0 GB) — different kernel path.
+- **Hangs / garbage output**: RDNA3 attention backend. Set `env.VLLM_ATTENTION_BACKEND = TRITON_ATTN`
+  explicitly; option names moved between versions — `vllmctl.py exec vllm serve --help` shows this image's.
+- **GTT growing during a run** (`vramx`, the GTT line): the budget is leaking into system RAM — stop,
+  lower the utilization. Do not wait for the desktop to stutter.
+- **Root-owned files in `~/.cache/huggingface`**: the container runs as root, so its downloads land
+  root-owned in anon's HF cache. Harmless for the rig; `sudo chown -R anon: ~/.cache/huggingface` if
+  the host-side `hf` tooling complains.
+- **Container died**: `docker logs vllm` (the container is kept after a crash; `vllmctl.py up` shows
+  the tail and removes it on the next start).
+
+## Phase 2 — if the numbers justify more
+
+1. **Same-file GGUF in vLLM**: GGUF loading moved to an out-of-tree `vllm-gguf-plugin`; it is
+   "highly experimental and under-optimized" per vLLM's docs and needs a custom image layer
+   (`pip install vllm-gguf-plugin`), plus `--tokenizer <base repo>`. Untested on ROCm. It would make the
+   weights identical — and show vLLM's unoptimized path, so interpret carefully.
+2. **MTP in vLLM**: `RedHatAI/Qwen3.5-9B-quantized.w4a16` ships `model_mtp.safetensors`; the
+   commented `speculative-config` in the ini is the knob (syntax per this image's `--help`).
+3. **Permanent coexistence**: replace the router's `--models-preset` mode with llama-swap (AUR
+   `llama-swap-bin`), which spawns llama-server *or* the vLLM container per model id, one at a time,
+   with a `ttl` idle unload. Same host/port/model ids for every client. Only worth it if a vLLM model
+   earns a place in the daily roster.