[Sync] Lua Hyprland chain, licence cleanup
This commit is contained in:
Coja committed
1 parent
93b4f695ce
commit
47166d877f
183 files changed
+5618
-5839
No files matched your search
@@ -0,0 +1,206 @@
|
||||
# vLLM on fl (docker, ROCm) — side by side with the llama.cpp router, never at the same time
|
||||
|
||||
Added 2026-10-21 to answer one question: **how does vLLM compare with llama.cpp on this card, on the
|
||||
same models?** It is a bench rig, not a second production endpoint. Nothing here was run on fl yet —
|
||||
fl has no sshd, so every command below is for you to run there; the first run will find the rough edges.
|
||||
|
||||
- **Box**: the same fl as the router — RX 7600 XT 16 GB (gfx1102, RDNA3), 5600X, 48 GB RAM. The card
|
||||
drives the display. llama.cpp runs the Arch Vulkan build; vLLM needs ROCm, which comes **inside the
|
||||
docker image** — nothing ROCm is installed on the host.
|
||||
- **Files**: `vllm.ini` (presets, same `key = --flag` convention as `../llamacpp/config.ini`),
|
||||
`vllmctl.py` (start/stop with the VRAM guards), `bench-vs.py` (one table for both engines).
|
||||
- **Port 11348**. Not in the router, not in any client list (opencode/pi/aichat/OWUI) on purpose.
|
||||
|
||||
## ⚠ The one rule: one engine resident at a time
|
||||
|
||||
vLLM pre-allocates `gpu-memory-utilization` × 16 GiB at startup and keeps it, idle or not. The router
|
||||
loads models lazily on request. So while vLLM is up, **any client request to :11343** makes the router
|
||||
load a model on top of it → overcommit → GTT spill → frozen PC (the same failure `--models-max 1`
|
||||
guards against). Unloading resident models is not enough; the clients on the LAN are the problem.
|
||||
|
||||
`vllmctl.py up` therefore refuses while `llama.service` is active. The sequence is always:
|
||||
|
||||
```fish
|
||||
~/.config/vllm/vllmctl.py up <preset> --stop-router # sudo systemctl stop llama.service, then start vLLM
|
||||
# … use / bench vLLM …
|
||||
~/.config/vllm/vllmctl.py down # stop container, wait for VRAM to drain
|
||||
sudo systemctl start llama.service # router back for the LAN
|
||||
```
|
||||
|
||||
The other order is the only one that should fail safely: vLLM compares its requested budget against
|
||||
free VRAM at startup and aborts when a router model is resident ("free memory … is less than desired
|
||||
GPU memory utilization" — reported by users, not something verified on this card). Everything else freezes.
|
||||
|
||||
## What "same models" means here, and the size surprise
|
||||
|
||||
llama.cpp runs GGUF; vLLM runs Hugging Face safetensors (bf16 / AWQ / GPTQ / compressed-tensors).
|
||||
The GGUF files cannot be loaded natively, so the pairs below are **the same model at a vLLM-native
|
||||
quant**, not the same files (same-file GGUF loading is phase 2, see the end).
|
||||
|
||||
Sizes from the HF API, 2026-10-21 — the "4-bit" vLLM checkpoints are **10–11.5 GB** because embeddings,
|
||||
lm_head and the vision tower stay bf16. They are bigger than the Q6/Q8 GGUFs they pair with, so on
|
||||
16 GB they are **clean-desktop only**, at `gpu-memory-utilization` ≈ 0.78 and 16k context:
|
||||
|
||||
| pair | llama.cpp preset (ledger) | vLLM preset (`vllm.ini`) | vLLM weights | fits? |
|
||||
|---|---|---|---|---|
|
||||
| smoke | — | `Qwen3-0.6B-smoke` = `Qwen/Qwen3-0.6B` | 1.5 GB | yes — proves docker+GPU+image |
|
||||
| gemma-4-E4B | `gemma-4-E4B-it-UD-Q8_K_XL` — 57 t/s **with MTP**, 96k ctx, ~7 GB | ~~`google/gemma-4-E4B-it-qat-w4a16-ct`~~ | 11.5 GB | **✗ no** (2026-10-21): vLLM needs ~16 GiB before any KV — the bf16 per-layer embedding table (262144 × 42 × 256 × 2 B = 5.6 GB) is held twice. Tombstone in the ini. |
|
||||
| Qwen3.5-9B | `Qwen3.5-9B-UD-Q6_K_XL` — 32 t/s **with MTP**, 32k ctx, 9 GB | `Qwen3.5-9B-w4a16` = `RedHatAI/Qwen3.5-9B-quantized.w4a16` | 11.0 GB | the last viable pair: 0.75 on a 2.1 GB desktop, 0.78 clean; prefill batch 512 |
|
||||
|
||||
**Cannot be compared on this card at all**: every 30B+ MoE (Coder-30B, GLM-4.7-Flash, Qwen3.6-35B,
|
||||
Ornith, gemma-4-26B), Coder-Next 80B and Qwen3.8-27B — they only fit llama.cpp through `n-cpu-moe`
|
||||
expert offload into RAM, which vLLM does not have (its MoE CPU-offload RFC died stale; the only
|
||||
option is whole-weight `cpu_offload_gb`, which is slow). gpt-oss-20b (MXFP4, ~13 GB) is marginal and
|
||||
its RDNA3 kernel status is unknown — not in the ini.
|
||||
|
||||
**Why the "docker should just work" expectation does not hold here**: of the three first-day failures,
|
||||
one was the image (no gfx1102 kernels → the override, solved) and two were memory. vLLM keeps
|
||||
everything in VRAM — bf16 embeddings, fp32 logits during profiling, CUDA-graph memory — on a card that
|
||||
also drives the desktop. No image changes a model's memory need; the ROCm images on Docker Hub are the
|
||||
right ones.
|
||||
|
||||
So the comparison is **not apples to apples** and the table says so: llama.cpp rows run Q8/Q6 with a
|
||||
speculative MTP draft at a bigger context; vLLM rows run int4 without speculation at 16k. Read it as
|
||||
"what each engine delivers on this card for this model", not as a kernel shoot-out.
|
||||
|
||||
## Setup on fl (once)
|
||||
|
||||
```fish
|
||||
sudo pacman -S docker # ~20 GB free disk for the ROCm image + 25 GB for two models
|
||||
sudo systemctl enable --now docker
|
||||
sudo usermod -aG docker anon # then log out / in (or `newgrp docker`)
|
||||
cd ~/.dots && ./install.sh # links ~/.config/vllm/* (new files are not live until re-stow)
|
||||
~/.config/vllm/vllmctl.py pull # vllm/vllm-openai-rocm:v0.32.0 (ROCm 10.0; 11.6 GB download, ~25 GB on disk)
|
||||
~/.config/vllm/vllmctl.py gpu-check # torch sees the card? prints device name + gfx arch
|
||||
```
|
||||
|
||||
`gpu-check` must print `cuda available: True`, a `gfx1100` device (that is the override at work) and
|
||||
`torch kernel: ok`. **The override is required** (found 2026-10-21): `v0.32.0` carries no gfx1102 code
|
||||
objects, and a kernel launch without one aborts the process *silently* — the engine core vanishes, the
|
||||
API server only says `Failed core proc(s): {}`, and even the `print`s before the abort are lost to
|
||||
stdout buffering. `[global] env.HSA_OVERRIDE_GFX_VERSION = 11.0.0` presents the card as gfx1100 (RX 7900),
|
||||
whose RDNA3 code runs on this chip. Nothing else on the host changes.
|
||||
|
||||
Then the smoke test (tiny model, 30 % of the card, proves the whole path before any big download —
|
||||
✓ passed 2026-10-21: 154.9 t/s decode, 1462 t/s prefill, TTFT 0.02 s, 7.7 GB used incl. a 2.9 GB desktop):
|
||||
|
||||
```fish
|
||||
~/.config/vllm/vllmctl.py up Qwen3-0.6B-smoke --stop-router
|
||||
curl -s localhost:11348/v1/chat/completions -H 'content-type: application/json' \
|
||||
-d '{"model":"Qwen3-0.6B-smoke","messages":[{"role":"user","content":"hi"}],"max_tokens":20}'
|
||||
~/.config/vllm/vllmctl.py down; sudo systemctl start llama.service
|
||||
```
|
||||
|
||||
First start of any model compiles Triton kernels: minutes, with `vllmctl.py up` echoing the log tail
|
||||
every 30 s.
|
||||
|
||||
**Tags and updates.** `<tag>` is the part after the colon: `latest` = newest stable release (was
|
||||
`v0.32.0` on 2026-10-18, ROCm 10.0), `nightly` = today's main branch (18 GB), `latest-rocm72` /
|
||||
`nightly-rocm72` = the same on the older ROCm 7.2 stack (smaller, 12 GB), `…-base` = without vLLM
|
||||
itself. The ini pins `v0.32.0` so a re-pull cannot silently change the engine under a bench run. To
|
||||
update: pick the new `vX.Y.Z` on hub.docker.com/r/vllm/vllm-openai-rocm/tags, put it in `[global] image`,
|
||||
`vllmctl.py pull`, then `vllmctl.py down` / `up` — the container is recreated from the image on every
|
||||
`up`. Afterwards `docker image prune -f` drops the superseded layers (`docker images` shows what is kept).
|
||||
`HF_TOKEN` is only needed for gated repos; none of the three presets is gated. If one ever is,
|
||||
`hf auth login` on fl writes `~/.cache/huggingface/token`, which the mounted cache hands to the container.
|
||||
|
||||
## Running the benchmark
|
||||
|
||||
Two phases, because the engines cannot be resident together. Do both on a **clean desktop** (close the
|
||||
browser; `vramx` must show ≤ ~1.5 GB used) — the vLLM presets leave ~2 GB free at 0.78 and an evening
|
||||
desktop does not fit.
|
||||
|
||||
```fish
|
||||
# phase 1 — vLLM (router stopped by --stop-router; first run also downloads the model)
|
||||
~/.config/vllm/vllmctl.py up gemma-4-E4B-it-w4a16 --stop-router
|
||||
~/.config/vllm/bench-vs.py -t vllm:gemma-4-E4B-it-w4a16 --ctx 8000 --concurrency 4
|
||||
~/.config/vllm/vllmctl.py down
|
||||
~/.config/vllm/vllmctl.py up Qwen3.5-9B-w4a16 # router is still stopped
|
||||
~/.config/vllm/bench-vs.py -t vllm:Qwen3.5-9B-w4a16 --ctx 8000 --concurrency 4
|
||||
~/.config/vllm/vllmctl.py down
|
||||
|
||||
# phase 2 — llama.cpp (vLLM down, router back)
|
||||
sudo systemctl start llama.service
|
||||
~/.config/vllm/bench-vs.py -t llamacpp:gemma-4-E4B-it-UD-Q8_K_XL -t llamacpp:Qwen3.5-9B-UD-Q6_K_XL --ctx 8000 --concurrency 4
|
||||
```
|
||||
|
||||
Every row lands in `bench-vs-history.md` (append) and the latest run in `bench-vs-results.md`, same
|
||||
folder. The guards: a `llamacpp` target is skipped while vLLM answers `/health`; a `vllm` target is
|
||||
skipped while the router has a model loaded or `llama.service` is active (`--force` overrides both).
|
||||
|
||||
**How to read it**
|
||||
|
||||
- `decode t/s` is usage tokens over wall time after the first token, measured the same way on both
|
||||
engines; llama.cpp's own `timings` number is in parentheses so rows match `../llamacpp/bench-results.md`.
|
||||
- Thinking is **off** on both (`chat_template_kwargs.enable_thinking=false`) and a reasoning delta counts
|
||||
as first token, so TTFT compares the engines, not the preamble lengths. `--think` reverts that.
|
||||
- `spec` shows the MTP draft on the llama.cpp side; vLLM rows have none unless the preset sets
|
||||
`speculative-config`. Expect the gemma/Qwen llama.cpp rows to carry a 1.5–2× draft bonus.
|
||||
- `agg t/s @N` is continuous batching's home turf — and llama-server still won it (85 vs 61 at 4
|
||||
streams): its default slot count on b11146 is evidently > 1, so no `parallel` key is needed.
|
||||
- `--ctx 8000` adds decode at an 8k prompt — the number that matters for agent loops.
|
||||
|
||||
## Results
|
||||
|
||||
**2026-10-21, Qwen3.5-9B, same evening, same script, thinking off, 256 tok, 0.6 GB desktop:**
|
||||
|
||||
| engine · checkpoint | spec | decode t/s | prefill t/s | TTFT s | decode @8k prompt | 4 streams agg t/s | VRAM used |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| vLLM · RedHatAI w4a16 @0.80, 16k, batch 512 | — | 29.4 | **181** | **0.17** | 16.0 | 60.6 | 12.5 GB |
|
||||
| llama.cpp · UD-Q6_K_XL, 32k | MTP draft | **61.3** | 114 | 0.27 | **51.1** | **85.4** | 10.9 GB |
|
||||
|
||||
What it says, for this card:
|
||||
|
||||
- **Decode**: llama.cpp wins 2× single-stream and 3× at an 8k prompt. Most of the 2× is the MTP draft
|
||||
(the ledger's pre-MTP Q8 number was 17 t/s in July; a no-draft Q6 today is probably ~30, i.e. parity
|
||||
with vLLM). The 8k gap is not MTP: vLLM's decode halves with context here (16.0 vs 29.4) while
|
||||
llama.cpp loses 17 % — on RDNA3 vLLM's attention path is the Triton generic one, llama.cpp's is a
|
||||
tuned Vulkan kernel.
|
||||
- **Prefill**: vLLM +59 % even with its batch capped at 512 for memory. This is its real strength on
|
||||
the box, and it shows in TTFT.
|
||||
- **Concurrency**: the column that was supposed to be vLLM's — it batched 4 streams to 2× its single
|
||||
(60.6), but llama-server served the same 4 at 85.4. Its default slot count is evidently > 1 on this
|
||||
build, so the README's "llama.cpp serializes" caveat does not hold; it does not need a `parallel` key.
|
||||
- **Memory**: vLLM holds 1.6 GB more for a model with 2 GB less context, and only runs on a ≤ 0.6 GB
|
||||
desktop. 0.75 was 0.11 GiB short of KV room; 0.80 is the setting.
|
||||
|
||||
**Verdict**: no vLLM model earns a roster slot on fl; the bench rig stays, the router stays, llama-swap
|
||||
is not needed. The one follow-up that could move the decode column is MTP in vLLM (the knob is in
|
||||
the ini) — but it cannot touch the memory column or the MoE roster, so it is curiosity, not a decision.
|
||||
|
||||
## When it does not start
|
||||
|
||||
- **`No available memory for the cache blocks`** (seen 2026-10-21, gemma @0.73/8k on a 2.1 GB desktop):
|
||||
the model loaded, but budget − weights − activation reservations left nothing for KV. What vLLM
|
||||
reserved is in the log: `docker logs vllm 2>&1 | grep -iE 'profil|kv cache|weights take|peak'`.
|
||||
Cheapest fixes first: text-only (`limit-mm-per-prompt` all zero — skips the image/audio encoder
|
||||
warmup, now in both presets), smaller `max-num-batched-tokens` (2048, now in both presets), lower
|
||||
`max-num-seqs`; then a cleaner desktop so the projection allows 0.78–0.82; last resort the smaller
|
||||
GPTQ checkpoint. `--set key=val` tries any of these without editing the ini.
|
||||
- **`ABORT: projected free …`**: the desktop is using more than the preset assumes. Close GPU apps or
|
||||
take the suggested utilization it prints. `--force` exists; the freeze is real.
|
||||
- **compressed-tensors / "no kernel" for int4 on ROCm**: both real presets use compressed-tensors
|
||||
pack-quantized. If this image lacks a ROCm kernel for it, swap in the GPTQ alternative noted in the
|
||||
ini (`Vishva007/gemma-4-E4B-it-W4A16-AutoRound-GPTQ`, 10.0 GB) — different kernel path.
|
||||
- **Hangs / garbage output**: RDNA3 attention backend. Set `env.VLLM_ATTENTION_BACKEND = TRITON_ATTN`
|
||||
explicitly; option names moved between versions — `vllmctl.py exec vllm serve --help` shows this image's.
|
||||
- **GTT growing during a run** (`vramx`, the GTT line): the budget is leaking into system RAM — stop,
|
||||
lower the utilization. Do not wait for the desktop to stutter.
|
||||
- **Root-owned files in `~/.cache/huggingface`**: the container runs as root, so its downloads land
|
||||
root-owned in anon's HF cache. Harmless for the rig; `sudo chown -R anon: ~/.cache/huggingface` if
|
||||
the host-side `hf` tooling complains.
|
||||
- **Container died**: `docker logs vllm` (the container is kept after a crash; `vllmctl.py up` shows
|
||||
the tail and removes it on the next start).
|
||||
|
||||
## Phase 2 — if the numbers justify more
|
||||
|
||||
1. **Same-file GGUF in vLLM**: GGUF loading moved to an out-of-tree `vllm-gguf-plugin`; it is
|
||||
"highly experimental and under-optimized" per vLLM's docs and needs a custom image layer
|
||||
(`pip install vllm-gguf-plugin`), plus `--tokenizer <base repo>`. Untested on ROCm. It would make the
|
||||
weights identical — and show vLLM's unoptimized path, so interpret carefully.
|
||||
2. **MTP in vLLM**: `RedHatAI/Qwen3.5-9B-quantized.w4a16` ships `model_mtp.safetensors`; the
|
||||
commented `speculative-config` in the ini is the knob (syntax per this image's `--help`).
|
||||
3. **Permanent coexistence**: replace the router's `--models-preset` mode with llama-swap (AUR
|
||||
`llama-swap-bin`), which spawns llama-server *or* the vLLM container per model id, one at a time,
|
||||
with a `ttl` idle unload. Same host/port/model ids for every client. Only worth it if a vLLM model
|
||||
earns a place in the daily roster.
|
||||
Executable
+258
@@ -0,0 +1,258 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
bench-vs — the SAME prompt through llama.cpp (router :11343) and vLLM (:11348), one table.
|
||||
|
||||
Measures per target: decode t/s, prefill t/s, TTFT, VRAM used/free (amdgpu sysfs, no root needed),
|
||||
optionally decode at a long prompt (--ctx) and aggregate throughput under N parallel requests
|
||||
(--concurrency). Method = ../llamacpp/bench.py (streamed chat completion, usage.completion_tokens
|
||||
over wall time after the first token), applied identically to both engines; llama.cpp's own
|
||||
`timings` figure is shown in parentheses so rows stay comparable with bench-results.md.
|
||||
|
||||
Thinking is OFF by default on both engines (chat_template_kwargs.enable_thinking=false) and the
|
||||
first reasoning_content delta counts as the first token anyway — a reasoning preamble of different
|
||||
length per engine would otherwise poison TTFT. --think re-enables it.
|
||||
|
||||
GUARDS (the card drives the display; two resident engines = GTT spill = frozen PC):
|
||||
- a llamacpp target is refused while vLLM answers /health (stop it: ./vllmctl.py down)
|
||||
- a vllm target is refused while the router reports a loaded model or llama.service is active
|
||||
|
||||
./bench-vs.py -t vllm:gemma-4-E4B-it-w4a16 # vLLM up (vllmctl.py up …), router stopped
|
||||
./bench-vs.py -t llamacpp:gemma-4-E4B-it-UD-Q8_K_XL # vLLM down, router started
|
||||
./bench-vs.py -t llamacpp:Qwen3.5-9B-UD-Q6_K_XL --ctx 8000 --concurrency 4
|
||||
Output: bench-vs-results.md (latest) + bench-vs-history.md (append), next to this script. Stdlib only.
|
||||
"""
|
||||
import argparse, json, os, re, subprocess, sys, threading, time, urllib.request, urllib.error
|
||||
from datetime import datetime
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
DRM = "/sys/class/drm/renderD128/device"
|
||||
URLS = {"llamacpp": "http://127.0.0.1:11343/v1", "vllm": "http://127.0.0.1:11348/v1"}
|
||||
TASK = "Write a Python function that merges two sorted lists, with a short docstring and one example."
|
||||
|
||||
|
||||
def sysfs(name):
|
||||
try:
|
||||
return int(open(os.path.join(DRM, name)).read().strip())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def gb(b):
|
||||
return f"{b / 1e9:.1f} GB" if b is not None else "-"
|
||||
|
||||
|
||||
def http(url, key, payload=None, timeout=600):
|
||||
data = json.dumps(payload).encode() if payload is not None else None
|
||||
hdrs = {"Authorization": f"Bearer {key}"}
|
||||
if data:
|
||||
hdrs["Content-Type"] = "application/json"
|
||||
return urllib.request.urlopen(urllib.request.Request(url, data=data, headers=hdrs), timeout=timeout)
|
||||
|
||||
|
||||
def parse_ini(path):
|
||||
sections, sec = {}, None
|
||||
try:
|
||||
lines = open(path, encoding="utf-8").read().splitlines()
|
||||
except OSError:
|
||||
return {}
|
||||
for ln in lines:
|
||||
s = ln.strip()
|
||||
if not s or s.startswith("#"):
|
||||
continue
|
||||
if s.startswith("[") and s.endswith("]"):
|
||||
sec = s[1:-1]; sections[sec] = {}
|
||||
elif sec and "=" in s:
|
||||
k, v = s.split("=", 1)
|
||||
sections[sec][k.strip()] = v.split(" #", 1)[0].split("\t#", 1)[0].strip()
|
||||
return sections
|
||||
|
||||
|
||||
def run(base, key, model, prompt, n, temp, timeout, think):
|
||||
payload = {"model": model, "messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": n, "temperature": temp, "stream": True,
|
||||
"stream_options": {"include_usage": True},
|
||||
"timings_per_token": True} # llama.cpp extension, ignored by vLLM
|
||||
if not think:
|
||||
payload["chat_template_kwargs"] = {"enable_thinking": False} # both engines accept it; templates without it ignore it
|
||||
t0 = time.perf_counter(); ttft = None; ntok = 0; usage = None; timings = None
|
||||
for raw in http(base.rstrip("/") + "/chat/completions", key, payload, timeout):
|
||||
line = raw.decode("utf-8", "ignore").strip()
|
||||
if not line.startswith("data:"):
|
||||
continue
|
||||
body = line[5:].strip()
|
||||
if body == "[DONE]":
|
||||
break
|
||||
try:
|
||||
c = json.loads(body)
|
||||
except ValueError:
|
||||
continue
|
||||
ch = c.get("choices") or [{}]
|
||||
d = ch[0].get("delta", {}) if ch else {}
|
||||
if d.get("content") or d.get("reasoning_content") or d.get("reasoning"):
|
||||
if ttft is None:
|
||||
ttft = time.perf_counter() - t0
|
||||
ntok += 1
|
||||
if c.get("usage"):
|
||||
usage = c["usage"]
|
||||
if c.get("timings"):
|
||||
timings = c["timings"]
|
||||
total = time.perf_counter() - t0
|
||||
comp = (usage or {}).get("completion_tokens") or ntok
|
||||
ptok = (usage or {}).get("prompt_tokens")
|
||||
gen_s = total - (ttft or total)
|
||||
tg = comp / gen_s if gen_s > 0 else 0.0
|
||||
pp = (ptok / ttft) if (ptok and ttft) else None
|
||||
srv = timings.get("predicted_per_second") if timings else None # llama.cpp's own number, for the ledger
|
||||
return {"tg": tg, "pp": pp, "ttft": ttft or 0.0, "ptok": ptok or 0, "ctok": comp, "wall": total, "srv": srv,
|
||||
"t0": t0, "t1": t0 + total}
|
||||
|
||||
|
||||
def with_retries(fn, retries, wait, label=""):
|
||||
last = None
|
||||
for i in range(retries + 1):
|
||||
try:
|
||||
return fn()
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", "ignore")[:300]
|
||||
if e.code < 500 or "failed to load" in body.lower():
|
||||
raise
|
||||
last = e
|
||||
except (urllib.error.URLError, ConnectionError, TimeoutError) as e:
|
||||
last = e
|
||||
if i < retries:
|
||||
print(f" … {label}: not ready ({last}); retry in {wait}s", file=sys.stderr)
|
||||
time.sleep(wait)
|
||||
raise last
|
||||
|
||||
|
||||
def vllm_up(url):
|
||||
try:
|
||||
return http(re.sub(r"/v1/?$", "", url) + "/health", "x", timeout=3).status == 200
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _status(m):
|
||||
"""Router /models status: newer builds send {'value': 'loaded'|'sleeping'|'unloaded'|…}, older a string."""
|
||||
st = m.get("status", "loaded")
|
||||
return st.get("value", "loaded") if isinstance(st, dict) else st
|
||||
|
||||
|
||||
def router_loaded(url):
|
||||
try:
|
||||
d = json.load(http(re.sub(r"/v1/?$", "", url) + "/models", "x", timeout=3))
|
||||
return [m["id"] for m in d.get("data", []) if _status(m) in ("loaded", "loading")]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
def router_active():
|
||||
try:
|
||||
r = subprocess.run(["systemctl", "is-active", "llama.service"], capture_output=True, text=True, timeout=10)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
return r.stdout.strip() == "active"
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("-t", "--target", action="append", required=True,
|
||||
help="engine:model, engine = llamacpp | vllm (repeatable)")
|
||||
ap.add_argument("--url", action="append", default=[], help="engine=URL override, e.g. vllm=http://host:11348/v1")
|
||||
ap.add_argument("-k", "--key", default="no-key-required")
|
||||
ap.add_argument("-n", "--tokens", type=int, default=256)
|
||||
ap.add_argument("--temp", type=float, default=0.3)
|
||||
ap.add_argument("--ctx", type=int, default=0, help="also record decode at ~this many prompt tokens")
|
||||
ap.add_argument("--concurrency", type=int, default=0, help="also record aggregate t/s with N parallel requests")
|
||||
ap.add_argument("--think", action="store_true", help="leave the model's thinking ON (default: off on both engines)")
|
||||
ap.add_argument("--timeout", type=int, default=600)
|
||||
ap.add_argument("--retries", type=int, default=4); ap.add_argument("--retry-wait", type=int, default=15)
|
||||
ap.add_argument("--force", action="store_true", help="skip the two-engines-resident guards")
|
||||
ap.add_argument("--out", default=os.path.join(HERE, "bench-vs-results.md"))
|
||||
ap.add_argument("--history", default=os.path.join(HERE, "bench-vs-history.md"))
|
||||
a = ap.parse_args()
|
||||
|
||||
urls = dict(URLS)
|
||||
for u in a.url:
|
||||
k, v = u.split("=", 1); urls[k] = v
|
||||
llcfg = parse_ini(os.path.join(HERE, "..", "llamacpp", "config.ini"))
|
||||
vlcfg = parse_ini(os.path.join(HERE, "vllm.ini"))
|
||||
total = sysfs("mem_info_vram_total")
|
||||
if total is None:
|
||||
print(" (amdgpu sysfs not readable here — VRAM columns are OFF; run this ON fl for them)", file=sys.stderr)
|
||||
|
||||
filler = ("The quick brown fox jumps over the lazy dog. " * max(1, a.ctx // 9)) if a.ctx else None
|
||||
cols = ["engine", "model", "cfg ctx", "spec", "decode t/s", "prefill t/s", "TTFT s", "VRAM used", "VRAM free"]
|
||||
if a.ctx:
|
||||
cols.append(f"decode@{a.ctx // 1000}k")
|
||||
if a.concurrency:
|
||||
cols.append(f"agg t/s @{a.concurrency}")
|
||||
|
||||
latest = open(a.out, "w", encoding="utf-8"); hist = open(a.history, "a", encoding="utf-8")
|
||||
|
||||
def emit(line):
|
||||
print(line); latest.write(line + "\n"); latest.flush(); hist.write(line + "\n"); hist.flush()
|
||||
|
||||
latest.write("# llama.cpp vs vLLM (latest run)\n\n"); hist.write(f"\n## run {datetime.now():%Y-%m-%d %H:%M}\n\n")
|
||||
emit(f"_Run {datetime.now():%Y-%m-%d %H:%M} · gen {a.tokens} tok · temp {a.temp} · thinking "
|
||||
f"{'ON' if a.think else 'OFF'} · VRAM total {gb(total)} · baseline used {gb(sysfs('mem_info_vram_used'))} · "
|
||||
f"decode = usage tokens / wall after first token on BOTH engines (llama.cpp's own timings in parentheses)_\n")
|
||||
emit("| " + " | ".join(cols) + " |"); emit("|" + "|".join(["---"] * len(cols)) + "|")
|
||||
|
||||
for t in a.target:
|
||||
eng, _, model = t.partition(":")
|
||||
if eng not in urls or not model:
|
||||
emit(f"| {eng} | {model} | bad target — use engine:model with engine in {list(urls)} |"); continue
|
||||
base = urls[eng]
|
||||
# guards
|
||||
if not a.force:
|
||||
if eng == "llamacpp" and vllm_up(urls["vllm"]):
|
||||
emit(f"| llamacpp | {model} | ⚠ SKIPPED: vLLM is up on {urls['vllm']} — loading a router model on top = freeze. "
|
||||
f"`vllmctl.py down` first |"); continue
|
||||
if eng == "vllm":
|
||||
loaded = router_loaded(urls["llamacpp"])
|
||||
if loaded or router_active():
|
||||
emit(f"| vllm | {model} | ⚠ SKIPPED: router loaded={loaded or '-'} active={router_active()} — "
|
||||
f"stop llama.service before benching vLLM |"); continue
|
||||
if eng == "llamacpp":
|
||||
c = llcfg.get(model, {})
|
||||
cfgctx, spec = c.get("ctx-size", "-"), (c.get("spec-type") or "-")
|
||||
else:
|
||||
c = vlcfg.get(model, {})
|
||||
cfgctx = c.get("max-model-len", "-")
|
||||
spec = "mtp" if c.get("speculative-config") else "-"
|
||||
cells = [eng, model, cfgctx, spec]
|
||||
try:
|
||||
with_retries(lambda: run(base, a.key, model, "hi", 8, a.temp, a.timeout, a.think), a.retries, a.retry_wait, model)
|
||||
used = sysfs("mem_info_vram_used")
|
||||
r = with_retries(lambda: run(base, a.key, model, TASK, a.tokens, a.temp, a.timeout, a.think), a.retries, a.retry_wait, model)
|
||||
dec = f"{r['tg']:.1f}" + (f" ({r['srv']:.1f})" if r["srv"] else "")
|
||||
cells += [dec, f"{r['pp']:.0f}" if r["pp"] else "-", f"{r['ttft']:.2f}", gb(used),
|
||||
gb(total - used) if (total and used is not None) else "-"]
|
||||
if a.ctx:
|
||||
rc = run(base, a.key, model, filler + "\n\n" + TASK, a.tokens, a.temp, a.timeout, a.think)
|
||||
cells.append(f"{rc['tg']:.1f}")
|
||||
if a.concurrency:
|
||||
res = [None] * a.concurrency
|
||||
def worker(i):
|
||||
res[i] = run(base, a.key, model, TASK + f" (variant {i})", a.tokens, a.temp, a.timeout, a.think)
|
||||
th = [threading.Thread(target=worker, args=(i,)) for i in range(a.concurrency)]
|
||||
for x in th: x.start()
|
||||
for x in th: x.join()
|
||||
ok = [x for x in res if x]
|
||||
wall = max(x["t1"] for x in ok) - min(x["t0"] for x in ok)
|
||||
cells.append(f"{sum(x['ctok'] for x in ok) / wall:.1f}" if wall > 0 else "-")
|
||||
except Exception as e:
|
||||
err = f"HTTP {e.code} {e.read().decode('utf-8', 'ignore')[:120]!r}" if isinstance(e, urllib.error.HTTPError) else f"ERROR {e}"
|
||||
cells = cells[:4] + [err] + [""] * (len(cols) - 5) # same width whether it died early or in the --ctx/--concurrency pass
|
||||
emit("| " + " | ".join(str(x) for x in cells) + " |")
|
||||
|
||||
latest.write("\n_llama.cpp rows run with the preset's MTP draft when `spec` says so; vLLM rows have no speculation "
|
||||
"unless `spec` = mtp. `agg t/s @N` = N parallel requests, completion tokens over the wall time of "
|
||||
"the whole batch (llama-server's default slot count serves them in parallel too)._\n")
|
||||
latest.close(); hist.close()
|
||||
print(f"\nwrote {a.out} (+ appended {a.history})")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,92 @@
|
||||
# vLLM presets for fl — AMD RX 7600 XT · 16 GB · docker image vllm/vllm-openai-rocm
|
||||
# Same convention as ../llamacpp/config.ini: section name = served model id, `key = value` = `--key value`.
|
||||
#
|
||||
# ⚠ ONE ENGINE AT A TIME. vLLM pre-allocates `gpu-memory-utilization` × 16 GiB at startup and KEEPS it.
|
||||
# Any llama.cpp load on top (a client hitting :11343 is enough) overcommits → GTT spill → PC freeze.
|
||||
# `vllmctl.py up` refuses to start while llama.service is active — stop the router first
|
||||
# (`vllmctl.py up <preset> --stop-router`), start it again after `vllmctl.py down`.
|
||||
#
|
||||
# Sections:
|
||||
# [global] container/runtime knobs (NOT vllm flags) — see vllmctl.py
|
||||
# [<preset>] one `vllmctl.py up <preset>`:
|
||||
# model HF repo id, or /models/<dir> (models-dir is mounted at /models, read-only)
|
||||
# any other key → `--key value` for `vllm serve`; `true` → bare `--key`; `false` → omitted
|
||||
# served-model-name defaults to the section name (what clients and bench-vs.py use)
|
||||
# x-* vllmctl-only metadata (x-weights-gb feeds the VRAM projection)
|
||||
# env.NAME = value container environment (per preset; [global] env.* apply to all)
|
||||
#
|
||||
# VRAM MATH (vllmctl prints it before every start, aborts under reserve-gb unless --force):
|
||||
# budget = gpu-memory-utilization × 17.16 GB (vLLM's total = the card's total, same as sysfs)
|
||||
# budget must hold weights + activations + KV cache; everything above it + the desktop must stay
|
||||
# under 16 GB with ≥ reserve-gb free. 0.75 → 12.9 GB budget → ~2.9 GB free on a 1.3 GB clean desktop.
|
||||
# A busy evening desktop (3+ GB) does NOT fit any of the real models below — clean desktop only.
|
||||
# Sizes measured 2026-10-21 from the HF API: the 4-bit vLLM checkpoints are 10-11.5 GB because
|
||||
# embeddings, lm_head and the vision tower stay bf16 — bigger than the Q6/Q8 GGUFs they pair with.
|
||||
|
||||
[global]
|
||||
image = vllm/vllm-openai-rocm:v0.32.0 # = `latest` on 2026-10-18 (ROCm 10.0, 11.6 GB download); update = bump this + `vllmctl.py pull`
|
||||
container = vllm
|
||||
port = 11348 # whisper = 11345/11346, kokoro = 11347
|
||||
models-dir = /home/anon/software/models # mounted at /models (ro) — share the GGUF dir
|
||||
hf-cache = /home/anon/.cache/huggingface # mounted at /root/.cache/huggingface (downloads + token)
|
||||
router = http://127.0.0.1:11343 # llama.cpp router root (for the resident-model gate)
|
||||
reserve-gb = 2.0 # abort if projected free VRAM < this (bench.py floor 1.5, target 2.5)
|
||||
drain-gb = 4.0 # "card is clean" threshold before starting (same as bench.py)
|
||||
health-timeout = 1800 # s — first start compiles Triton kernels, be patient
|
||||
gpu-memory-utilization = 0.75 # default; presets override
|
||||
# REQUIRED on this card (✓ verified 2026-10-21): v0.32.0 ships no gfx1102 code objects — without the
|
||||
# override every GPU kernel launch aborts the process SILENTLY (the engine core just vanishes, the
|
||||
# API server reports "Failed core proc(s): {}"). gfx1100 (RX 7900) code runs on gfx1102, same RDNA3 ISA.
|
||||
env.HSA_OVERRIDE_GFX_VERSION = 11.0.0
|
||||
# Attention backend knobs (ROCm; names moved between versions — `vllmctl.py exec vllm serve --help`):
|
||||
# env.VLLM_ATTENTION_BACKEND = TRITON_ATTN # the default on RDNA; set explicitly if a hang points at attention
|
||||
|
||||
# ─── 1. smoke test — tiny, certain to be supported; proves docker + GPU + image + override ────
|
||||
# ✓ 2026-10-21: 154.9 t/s decode · 1462 t/s prefill · TTFT 0.02 s · 7.7 GB used incl. a 2.9 GB desktop.
|
||||
[Qwen3-0.6B-smoke]
|
||||
model = Qwen/Qwen3-0.6B
|
||||
gpu-memory-utilization = 0.30
|
||||
max-model-len = 4096
|
||||
x-weights-gb = 1.5
|
||||
|
||||
# ─── 2. gemma-4-E4B — ✗ DOES NOT FIT vLLM ON THIS CARD (2026-10-21), tombstone kept for the math ──
|
||||
# Pairs with llama.cpp [gemma-4-E4B-it-UD-Q8_K_XL] (57 t/s with MTP, 96k ctx). google/gemma-4-E4B-it-qat-w4a16-ct
|
||||
# is 11.5 GB, and the second try (text-only, batch 2048, @0.73 on a 2.1 GB desktop) ended with
|
||||
# "Available KV cache memory: -4.44 GiB" after an OOM on a 5,637,144,576-byte allocation. That is exactly
|
||||
# Gemma 4's per-layer embedding table: 262144 vocab × 42 layers × 256 × 2 bytes (bf16). vLLM holds it
|
||||
# in bf16 AND needs a second buffer of the same size → ~16 GiB before any KV block. No util, batch or
|
||||
# desktop fixes that; the GPTQ/AWQ alternatives carry the same bf16 table. llama.cpp fits because the
|
||||
# GGUF stores the table quantized, once. A different image changes nothing here.
|
||||
# [gemma-4-E4B-it-w4a16]
|
||||
# model = google/gemma-4-E4B-it-qat-w4a16-ct
|
||||
# gpu-memory-utilization = 0.78
|
||||
# max-model-len = 16384
|
||||
# max-num-seqs = 4
|
||||
# max-num-batched-tokens = 2048
|
||||
# limit-mm-per-prompt = {"image": 0, "audio": 0, "video": 0}
|
||||
# x-weights-gb = 11.5
|
||||
|
||||
# ─── 3. Qwen3.5-9B — pairs with llama.cpp [Qwen3.5-9B-UD-Q6_K_XL] (32 t/s with MTP, 32k ctx) ──
|
||||
# Red Hat's llm-compressor int4 (compressed-tensors). 10.95 GB + a 0.49 GB MTP head file.
|
||||
# Alternative (real AWQ, Triton kernels on ROCm): QuantTrio/Qwen3.5-9B-AWQ — 12.4 GB, too big for 0.78.
|
||||
# The last viable "same model" pair: no per-layer table, but a 248k vocab → the fp32 logits buffer
|
||||
# during profiling is max-num-batched-tokens × 248k × 4 B (2048 → 2.0 GB, 512 → 0.5 GB). Start small.
|
||||
# On a 2.1 GB desktop the projection allows 0.75 (`--set gpu-memory-utilization=0.75`) → ~1.9 GB for
|
||||
# activations + graphs + KV. If "No available memory for the cache blocks" again: `--set enforce-eager=true`
|
||||
# (drops the CUDA-graph memory, costs decode speed), then a cleaner desktop for 0.78.
|
||||
# ✓ RUNS 2026-10-21 @0.80 (0.75 was 0.11 GiB short of KV room): 29.4 t/s decode · 181 prefill · TTFT 0.17 ·
|
||||
# decode@8k 16.0 · 4-stream aggregate 60.6 · 12.5 GB used on a 0.6 GB desktop. Same evening, llama.cpp
|
||||
# [Qwen3.5-9B-UD-Q6_K_XL] with its MTP draft: 61.3 · 114 · 0.27 · 51.1 · 85.4 · 10.9 GB. See README.
|
||||
[Qwen3.5-9B-w4a16]
|
||||
model = RedHatAI/Qwen3.5-9B-quantized.w4a16
|
||||
gpu-memory-utilization = 0.80 # needs a ≤ 0.6 GB desktop (projection: 2.9 GB free); 0.75 fails on KV room
|
||||
max-model-len = 16384
|
||||
max-num-seqs = 4
|
||||
max-num-batched-tokens = 512 # caps the profiling logits buffer (prefill t/s cost — note it in the ledger)
|
||||
limit-mm-per-prompt = {"image": 0, "video": 0} # text-only bench (Qwen3.5 has no audio input)
|
||||
x-weights-gb = 11.0
|
||||
# NEXT (the one knob that could change the verdict): MTP speculative decoding — the repo ships
|
||||
# model_mtp.safetensors and llama.cpp's 2× comes from exactly this. Costs ~0.5 GB weights + draft
|
||||
# state → try @0.82 on a clean desktop. Syntax per `vllmctl.py exec vllm serve --help`:
|
||||
# ./vllmctl.py up Qwen3.5-9B-w4a16 --set gpu-memory-utilization=0.82 --set 'speculative-config={"method": "mtp", "num_speculative_tokens": 2}'
|
||||
# speculative-config = {"method": "mtp", "num_speculative_tokens": 2}
|
||||
Executable
+398
@@ -0,0 +1,398 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
vllmctl — run vLLM (docker, ROCm) on fl WITHOUT freezing the box.
|
||||
|
||||
The RX 7600 XT drives the display; VRAM overcommit spills to GTT and freezes the PC. vLLM grabs
|
||||
`gpu-memory-utilization` × total VRAM at startup and keeps it, so this script gates every start:
|
||||
|
||||
1. llama.service must be STOPPED (or --stop-router): a LAN client hitting :11343 would make the
|
||||
router load a model on top of vLLM = the freeze. --unsafe-keep-router skips this gate.
|
||||
2. the card must be clean (VRAM used ≤ drain-gb; resident router models are unloaded via its API).
|
||||
3. projected free VRAM = total − used − budget must be ≥ reserve-gb (else abort, --force overrides).
|
||||
|
||||
Presets live in vllm.ini next to this file (same key = --flag convention as llamacpp/config.ini).
|
||||
|
||||
./vllmctl.py up <preset> [--stop-router] [--force] [--dry-run] start, wait for /health
|
||||
./vllmctl.py down stop, wait for VRAM to drain
|
||||
./vllmctl.py status | logs | pull | gpu-check | exec <cmd...>
|
||||
Stdlib only. Run ON fl.
|
||||
"""
|
||||
import argparse, grp, json, os, re, shlex, subprocess, sys, time, urllib.request, urllib.error
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
INI = os.path.join(HERE, "vllm.ini")
|
||||
DRM = "/sys/class/drm/renderD128/device"
|
||||
|
||||
|
||||
# ─── ini ────────────────────────────────────────────────────────────────────────────────────
|
||||
def parse_ini(path):
|
||||
"""{section: {key: value}} — inline `# comments` stripped, order kept."""
|
||||
sections, sec = {}, None
|
||||
for ln in open(path, encoding="utf-8").read().splitlines():
|
||||
s = ln.strip()
|
||||
if not s or s.startswith("#"):
|
||||
continue
|
||||
if s.startswith("[") and s.endswith("]"):
|
||||
sec = s[1:-1]; sections[sec] = {}
|
||||
elif sec and "=" in s:
|
||||
k, v = s.split("=", 1)
|
||||
sections[sec][k.strip()] = v.split(" #", 1)[0].split("\t#", 1)[0].strip()
|
||||
return sections
|
||||
|
||||
|
||||
def build_args(preset):
|
||||
"""vllm serve args from a preset section (x-* and env.* are not flags)."""
|
||||
args = []
|
||||
for k, v in preset.items():
|
||||
if k.startswith("x-") or k.startswith("env."):
|
||||
continue
|
||||
if v.lower() == "false":
|
||||
continue
|
||||
if v.lower() == "true":
|
||||
args.append(f"--{k}")
|
||||
else:
|
||||
args += [f"--{k}", v]
|
||||
return args
|
||||
|
||||
|
||||
# ─── gpu / router ───────────────────────────────────────────────────────────────────────────
|
||||
def sysfs(name):
|
||||
try:
|
||||
return int(open(os.path.join(DRM, name)).read().strip())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def vram():
|
||||
"""(used, total, gtt_used) bytes from amdgpu sysfs; None where unreadable."""
|
||||
return sysfs("mem_info_vram_used"), sysfs("mem_info_vram_total"), sysfs("mem_info_gtt_used")
|
||||
|
||||
|
||||
def gb(b):
|
||||
return f"{b / 1e9:.1f} GB" if b is not None else "-"
|
||||
|
||||
|
||||
def http(url, payload=None, timeout=10):
|
||||
data = json.dumps(payload).encode() if payload is not None else None
|
||||
hdrs = {"Content-Type": "application/json"} if data else {}
|
||||
return urllib.request.urlopen(urllib.request.Request(url, data=data, headers=hdrs), timeout=timeout)
|
||||
|
||||
|
||||
def _status(m):
|
||||
"""Router /models status: newer builds send {'value': 'loaded'|'sleeping'|'unloaded'|…}, older a string."""
|
||||
st = m.get("status", "loaded")
|
||||
return st.get("value", "loaded") if isinstance(st, dict) else st
|
||||
|
||||
|
||||
def router_loaded(root):
|
||||
"""ids the llama.cpp router reports as loaded ([] if unreachable)."""
|
||||
try:
|
||||
d = json.load(http(root.rstrip("/") + "/models", timeout=5))
|
||||
except Exception:
|
||||
return []
|
||||
return [m["id"] for m in d.get("data", []) if _status(m) in ("loaded", "loading")]
|
||||
|
||||
|
||||
def router_unload(root):
|
||||
for m in router_loaded(root):
|
||||
try:
|
||||
http(root.rstrip("/") + "/models/unload", {"model": m}, timeout=30).read()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def router_active():
|
||||
try:
|
||||
r = subprocess.run(["systemctl", "is-active", "llama.service"], capture_output=True, text=True, timeout=10)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
return r.stdout.strip() == "active"
|
||||
|
||||
|
||||
def router_hint():
|
||||
return "" if router_active() else "\n (llama.service is STOPPED → sudo systemctl start llama.service to bring the router back)"
|
||||
|
||||
|
||||
def wait_drain(drain_gb, window_s, label):
|
||||
used, _, _ = vram()
|
||||
end = time.time() + window_s
|
||||
said = False
|
||||
while used is not None and used > drain_gb * 1e9 and time.time() < end:
|
||||
if not said:
|
||||
print(f" … waiting for VRAM to drain {label} ({gb(used)} used)"); said = True
|
||||
time.sleep(5)
|
||||
used, _, _ = vram()
|
||||
return used
|
||||
|
||||
|
||||
# ─── docker ─────────────────────────────────────────────────────────────────────────────────
|
||||
def docker(*args, check=True, capture=True):
|
||||
return subprocess.run(["docker", *args], check=check, capture_output=capture, text=True)
|
||||
|
||||
|
||||
def container_state(name):
|
||||
r = docker("inspect", "-f", "{{.State.Status}}", name, check=False)
|
||||
return r.stdout.strip() if r.returncode == 0 else None
|
||||
|
||||
|
||||
def gpu_flags():
|
||||
flags = ["--device", "/dev/kfd", "--device", "/dev/dri", "--group-add", "video",
|
||||
"--cap-add", "SYS_PTRACE", "--security-opt", "seccomp=unconfined", "--ipc=host"]
|
||||
try:
|
||||
flags += ["--group-add", str(grp.getgrnam("render").gr_gid)] # Arch: /dev/dri/renderD128 is group render
|
||||
except KeyError:
|
||||
pass
|
||||
return flags
|
||||
|
||||
|
||||
def run_cmd(g, name, preset, envs):
|
||||
cmd = ["docker", "run", "-d", "--name", g["container"], *gpu_flags(),
|
||||
"-p", f'{g["port"]}:8000',
|
||||
"-v", f'{g["models-dir"]}:/models:ro',
|
||||
"-v", f'{g["hf-cache"]}:/root/.cache/huggingface']
|
||||
for k, v in envs.items():
|
||||
cmd += ["-e", f"{k}={v}"]
|
||||
cmd.append(g["image"])
|
||||
cmd += ["--host", "0.0.0.0", "--port", "8000", "--served-model-name", preset.get("served-model-name", name)]
|
||||
cmd += build_args({k: v for k, v in preset.items() if k != "served-model-name"})
|
||||
return cmd
|
||||
|
||||
|
||||
ROOT_CAUSE = re.compile(r"hipError|HSA_STATUS|code object|gfx\d+|no kernel image|EngineCore.*(ERROR|Error|error)"
|
||||
r"|^\S*(ERROR|CRITICAL)|RuntimeError|ValueError|NotImplementedError|OutOfMemory|Killed|Segmentation"
|
||||
r"|free memory|not supported|Unsupported|Traceback|memory profiling|kv cache memory|weights take|peak", re.I)
|
||||
|
||||
|
||||
def root_cause(lines):
|
||||
"""The lines that explain a dead engine core: known ROCm/vLLM failure signatures, the FIRST
|
||||
traceback's error line, and the last 8 lines — not the parent's 40-line stack trace."""
|
||||
out, seen = [], set()
|
||||
for l in lines:
|
||||
if ROOT_CAUSE.search(l) and "api_server" not in l and "core_client" not in l and "launch_core_engines" not in l:
|
||||
key = l.strip()[:120]
|
||||
if key not in seen:
|
||||
seen.add(key); out.append(l.rstrip())
|
||||
if len(out) >= 30:
|
||||
break
|
||||
return out + ["…"] + [l.rstrip() for l in lines[-8:]]
|
||||
|
||||
|
||||
# ─── commands ───────────────────────────────────────────────────────────────────────────────
|
||||
def cmd_up(a, cfg):
|
||||
g = cfg["global"]
|
||||
if a.preset not in cfg or a.preset == "global":
|
||||
sys.exit(f"unknown preset {a.preset!r}; have: {', '.join(k for k in cfg if k != 'global')}")
|
||||
preset = dict(cfg[a.preset])
|
||||
preset.update(dict(kv.split("=", 1) for kv in a.set)) # --set gpu-memory-utilization=0.73 --set max-model-len=8192
|
||||
preset.setdefault("gpu-memory-utilization", g.get("gpu-memory-utilization", "0.75"))
|
||||
envs = {k[4:]: v for k, v in g.items() if k.startswith("env.")}
|
||||
envs.update({k[4:]: v for k, v in preset.items() if k.startswith("env.")})
|
||||
envs.update(dict(e.split("=", 1) for e in a.env))
|
||||
util = float(preset["gpu-memory-utilization"])
|
||||
reserve = float(a.reserve if a.reserve is not None else g.get("reserve-gb", 2.0))
|
||||
drain = float(g.get("drain-gb", 4.0))
|
||||
|
||||
cmd = run_cmd(g, a.preset, preset, envs)
|
||||
print("docker command:\n " + " ".join(shlex.quote(c) for c in cmd) + "\n")
|
||||
if a.dry_run:
|
||||
return
|
||||
|
||||
st = container_state(g["container"])
|
||||
if st == "running":
|
||||
sys.exit(f"container {g['container']} is already running — `vllmctl.py down` first")
|
||||
if st: # leftover from a crash: its log is the evidence, show the tail, then clear the name
|
||||
r = docker("logs", "--tail", "15", g["container"], check=False)
|
||||
print(f"removing old container ({st}); its last lines were:\n" + "\n".join(" | " + l for l in (r.stdout + r.stderr).strip().splitlines()[-15:]))
|
||||
docker("rm", g["container"], check=False)
|
||||
|
||||
# 1. clean card — unload resident router models through its API while it is still up, then drain
|
||||
if router_active():
|
||||
router_unload(g["router"])
|
||||
used = wait_drain(drain, 180, "before start")
|
||||
used, total, gtt = vram()
|
||||
if used is None or total is None:
|
||||
if not a.force:
|
||||
sys.exit(f"ABORT: cannot read {DRM}/mem_info_vram_* — no VRAM guard, not starting blind (--force overrides)")
|
||||
print("⚠ FORCED without a VRAM guard (sysfs unreadable)"); used, total = 0, 1
|
||||
if used > drain * 1e9 and not a.force:
|
||||
sys.exit(f"ABORT: card not clean — {gb(used)} VRAM used (> drain-gb {drain}). Leftover model / GPU apps?")
|
||||
|
||||
# 2. projection — abort here costs nothing, llama.service is still running
|
||||
budget = util * total
|
||||
free_after = total - used - budget
|
||||
weights = float(preset.get("x-weights-gb", 0) or 0) * 1e9
|
||||
print(f"VRAM: total {gb(total)} · used now {gb(used)} (GTT {gb(gtt)}) · vLLM budget {util:.2f} × total = {gb(budget)}"
|
||||
f"\n → projected free {gb(free_after)} (reserve {reserve} GB)")
|
||||
if weights:
|
||||
room = budget - weights
|
||||
print(f" budget − weights {gb(weights)} = {gb(room)} for activations + KV cache"
|
||||
+ (" ⚠ < 1 GB: expect 'no memory for KV cache' → raise util or lower max-model-len" if room < 1e9 else ""))
|
||||
if free_after < reserve * 1e9:
|
||||
suggest = max(0.0, (total - used - reserve * 1e9) / total)
|
||||
msg = (f"projected free {gb(free_after)} < reserve {reserve} GB. Max safe gpu-memory-utilization right now "
|
||||
f"= {suggest:.2f} (desktop is using {gb(used)}).")
|
||||
if not a.force:
|
||||
sys.exit("ABORT: " + msg + f"\n → free the desktop (vramx) or `--set gpu-memory-utilization={suggest:.2f}` "
|
||||
f"(+ `--set max-model-len=8192` if weights leave < 1 GB); --force to start anyway.")
|
||||
print("⚠ FORCED: " + msg)
|
||||
|
||||
# 3. router gate — the only thing that stops a LAN client from loading a model on top of vLLM
|
||||
if router_active():
|
||||
if a.stop_router:
|
||||
print("stopping llama.service (sudo) …")
|
||||
subprocess.run(["sudo", "systemctl", "stop", "llama.service"], check=True)
|
||||
elif a.unsafe_keep_router:
|
||||
print("⚠ llama.service stays ACTIVE: any client request to :11343 while vLLM runs = GTT freeze.")
|
||||
else:
|
||||
sys.exit("ABORT: llama.service is active. Any client request to :11343 would load a model on top "
|
||||
"of vLLM and freeze the PC.\n → re-run with --stop-router (sudo systemctl stop llama.service), "
|
||||
"or --unsafe-keep-router if you really know nothing will call the router.")
|
||||
used2, _, _ = vram()
|
||||
if used2 is not None and used2 > used + 1e9: # a client loaded something between projection and stop
|
||||
print(f"⚠ VRAM grew to {gb(used2)} meanwhile — vLLM's own free-memory check should refuse; watch the start")
|
||||
|
||||
# 4. start + wait
|
||||
cid = subprocess.run(cmd, check=True, capture_output=True, text=True).stdout.strip()
|
||||
print(f"started {cid[:12]} — waiting for http://127.0.0.1:{g['port']}/health (timeout {g.get('health-timeout', 1800)} s; "
|
||||
f"first start compiles kernels)")
|
||||
end = time.time() + int(g.get("health-timeout", 1800))
|
||||
last_log, shown = 0, set()
|
||||
while time.time() < end:
|
||||
st = container_state(g["container"])
|
||||
if st != "running":
|
||||
r = docker("logs", g["container"], check=False)
|
||||
print("root cause candidates (full log: docker logs " + g["container"] + "):")
|
||||
for l in root_cause((r.stdout + r.stderr).splitlines()):
|
||||
print(" | " + l[:200])
|
||||
sys.exit(f"ABORT: container is {st or 'gone'}. If a line names gfx1102 / 'code object' / hipErrorNoBinaryForGpu: "
|
||||
f"uncomment env.HSA_OVERRIDE_GFX_VERSION = 11.0.0 in [global] and retry." + router_hint())
|
||||
try:
|
||||
if http(f"http://127.0.0.1:{g['port']}/health", timeout=3).status == 200:
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
if time.time() - last_log > 30:
|
||||
r = docker("logs", "--tail", "3", g["container"], check=False)
|
||||
tail = (r.stdout + r.stderr).strip().splitlines()[-3:]
|
||||
for t in tail:
|
||||
if t not in shown: # only lines not printed before (compile phases are quiet for minutes)
|
||||
print(" | " + t[:160]); shown.add(t)
|
||||
last_log = time.time()
|
||||
time.sleep(3)
|
||||
else:
|
||||
sys.exit("ABORT: no /health within the timeout — `vllmctl.py logs`, then `vllmctl.py down`" + router_hint())
|
||||
used, total, gtt = vram()
|
||||
print(f"UP. VRAM used {gb(used)} / free {gb(total - used)} · GTT {gb(gtt)}")
|
||||
try:
|
||||
ids = [m["id"] for m in json.load(http(f"http://127.0.0.1:{g['port']}/v1/models")).get("data", [])]
|
||||
print(f"models: {', '.join(ids)} → http://192.168.0.204:{g['port']}/v1")
|
||||
except Exception as e:
|
||||
print(f"(/v1/models not readable yet: {e})")
|
||||
print(f"next: ./bench-vs.py -t vllm:{preset.get('served-model-name', a.preset)} · when done: ./vllmctl.py down")
|
||||
|
||||
|
||||
def cmd_down(a, cfg):
|
||||
g = cfg["global"]
|
||||
st = container_state(g["container"])
|
||||
if st:
|
||||
print(f"stopping {g['container']} ({st}) …")
|
||||
docker("stop", "-t", "60", g["container"], check=False)
|
||||
docker("rm", g["container"], check=False)
|
||||
else:
|
||||
print("container not running")
|
||||
used = wait_drain(float(g.get("drain-gb", 4.0)), 120, "after stop")
|
||||
used, total, gtt = vram()
|
||||
print(f"VRAM used {gb(used)} / free {gb(total - used) if total and used is not None else '-'} · GTT {gb(gtt)}")
|
||||
if not router_active():
|
||||
print("llama.service is stopped → `sudo systemctl start llama.service` to bring the router back")
|
||||
|
||||
|
||||
def cmd_status(a, cfg):
|
||||
g = cfg["global"]
|
||||
used, total, gtt = vram()
|
||||
print(f"container {g['container']}: {container_state(g['container']) or 'not running'}")
|
||||
print(f"VRAM used {gb(used)} / total {gb(total)} · GTT {gb(gtt)}")
|
||||
print(f"llama.service: {'ACTIVE' if router_active() else 'stopped'} · loaded: {router_loaded(g['router']) or '-'}")
|
||||
try:
|
||||
ids = [m["id"] for m in json.load(http(f"http://127.0.0.1:{g['port']}/v1/models", timeout=3)).get("data", [])]
|
||||
print(f"vLLM :{g['port']} serving: {', '.join(ids)}")
|
||||
except Exception:
|
||||
print(f"vLLM :{g['port']}: not answering")
|
||||
|
||||
|
||||
def cmd_logs(a, cfg):
|
||||
os.execvp("docker", ["docker", "logs", "-f", "--tail", "100", cfg["global"]["container"]])
|
||||
|
||||
|
||||
def cmd_pull(a, cfg):
|
||||
os.execvp("docker", ["docker", "pull", cfg["global"]["image"]])
|
||||
|
||||
|
||||
GPU_CHECK = r"""
|
||||
import os, torch
|
||||
print('HSA_OVERRIDE_GFX_VERSION =', os.environ.get('HSA_OVERRIDE_GFX_VERSION', '(unset)'))
|
||||
print('cuda available:', torch.cuda.is_available())
|
||||
p = torch.cuda.get_device_properties(0)
|
||||
print('device:', torch.cuda.get_device_name(0) or '(no name)', p.gcnArchName, f'{p.total_memory / 1e9:.1f} GB')
|
||||
print('torch kernel …', end=' ', flush=True) # a hard abort here (no code object for this arch) prints nothing more
|
||||
x = torch.ones(1024, device='cuda'); torch.cuda.synchronize(); print('ok', x.sum().item())
|
||||
import vllm._C # noqa
|
||||
ops = sorted({s.name.split('::', 1)[1] for s in torch._C._jit_get_all_schemas() if s.name.startswith('_C::')})
|
||||
print(f'vllm _C ops registered: {len(ops)}')
|
||||
f16 = dict(device='cuda', dtype=torch.float16)
|
||||
probes = {'silu_and_mul': lambda: torch.ops._C.silu_and_mul(torch.empty(4, 8, **f16), torch.randn(4, 16, **f16)),
|
||||
'gelu_fast': lambda: torch.ops._C.gelu_fast(torch.empty(4, 8, **f16), torch.randn(4, 8, **f16)),
|
||||
'rms_norm': lambda: torch.ops._C.rms_norm(torch.empty(4, 8, **f16), torch.randn(4, 8, **f16), torch.ones(8, **f16), 1e-6)}
|
||||
for name, fn in probes.items():
|
||||
if name in ops:
|
||||
print(f'vllm custom op {name} …', end=' ', flush=True); fn(); torch.cuda.synchronize(); print('ok -> the image has kernels for this arch')
|
||||
break
|
||||
else:
|
||||
print('none of the probe ops exist in this build; first ops:', ops[:15])
|
||||
"""
|
||||
|
||||
|
||||
def cmd_gpu_check(a, cfg):
|
||||
"""Does the image see the card? Prints torch's device name + arch (needs no model)."""
|
||||
g = cfg["global"]
|
||||
envs = [x for k, v in g.items() if k.startswith("env.") for x in ("-e", f"{k[4:]}={v}")]
|
||||
envs += [x for e in a.env for x in ("-e", e)]
|
||||
code = GPU_CHECK
|
||||
cmd = ["docker", "run", "--rm", *gpu_flags(), *envs, "--entrypoint", "python3", g["image"], "-u", "-c", code]
|
||||
print(" ".join(shlex.quote(c) for c in cmd))
|
||||
os.execvp("docker", cmd)
|
||||
|
||||
|
||||
def cmd_exec(a, cfg):
|
||||
g = cfg["global"]
|
||||
if container_state(g["container"]) == "running":
|
||||
os.execvp("docker", ["docker", "exec", "-it", g["container"], *a.cmd])
|
||||
envs = [x for k, v in g.items() if k.startswith("env.") for x in ("-e", f"{k[4:]}={v}")]
|
||||
os.execvp("docker", ["docker", "run", "--rm", "-it", *gpu_flags(), *envs, "--entrypoint", a.cmd[0], g["image"], *a.cmd[1:]])
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--ini", default=INI)
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
up = sub.add_parser("up"); up.add_argument("preset")
|
||||
up.add_argument("--stop-router", action="store_true", help="sudo systemctl stop llama.service first")
|
||||
up.add_argument("--unsafe-keep-router", action="store_true", help="leave llama.service running (freeze risk)")
|
||||
up.add_argument("--force", action="store_true", help="ignore the VRAM projection / clean-card checks")
|
||||
up.add_argument("--reserve", type=float, help="override reserve-gb")
|
||||
up.add_argument("--dry-run", action="store_true", help="print the docker command only")
|
||||
up.add_argument("--env", action="append", default=[], metavar="KEY=VAL", help="extra container env (e.g. HSA_OVERRIDE_GFX_VERSION=11.0.0)")
|
||||
up.add_argument("--set", action="append", default=[], metavar="KEY=VAL", help="override a preset key for this start (e.g. gpu-memory-utilization=0.73)")
|
||||
sub.add_parser("down"); sub.add_parser("status"); sub.add_parser("logs"); sub.add_parser("pull")
|
||||
gc = sub.add_parser("gpu-check"); gc.add_argument("--env", action="append", default=[], metavar="KEY=VAL")
|
||||
ex = sub.add_parser("exec"); ex.add_argument("cmd", nargs=argparse.REMAINDER)
|
||||
a = ap.parse_args()
|
||||
cfg = parse_ini(a.ini)
|
||||
if "global" not in cfg:
|
||||
sys.exit(f"{a.ini}: no [global] section")
|
||||
{"up": cmd_up, "down": cmd_down, "status": cmd_status, "logs": cmd_logs, "pull": cmd_pull,
|
||||
"gpu-check": cmd_gpu_check, "exec": cmd_exec}[a.cmd](a, cfg)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in new issue
Block a user