[Sync] Lua Hyprland chain, licence cleanup
This commit is contained in:
Coja committed
1 parent
93b4f695ce
commit
47166d877f
183 files changed
+5618
-5839
No files matched your search
@@ -70,9 +70,9 @@ starves the desktop and freezes the PC (reboot). Hard-won rules:
|
||||
| `GLM-4.7-Flash-UD-Q4_K_XL` | **21.5** @moe22 | 24k | reasoning | quality coder (opencode subagents); KV fix + moe22 ✓ verified clean 08-08 |
|
||||
| `GLM-4.7-Flash-UD-Q3_K_XL` | **26.0** @moe12 (2.0 daytime baseline; ~28 clean) | 24k | reasoning | **fast coder** — re-added 09-21 because Q4's 21.5 dragged in agent loops; 3-bit, evening-safe at moe12 (13.4 used / 3.8 free @2.0); Q4 stays for hard reviews |
|
||||
| `Qwen3-Coder-Next-UD-IQ3_XXS` | 16.0 | **128k** | 80B-A3B | long-session coder (128k ctx) |
|
||||
| `Qwen3.8-27B-UD-IQ3_XXS` | **15.5** (10.9 w/o MTP) | 24k | vision, MTP, hybrid reasoner @ **xhigh** | ⚠ **clean-desktop specialty** (kept 09-10 over the ≥22 gate) — dense-27B quality, slow: its Gated-DeltaNet layers are Vulkan-bound (prefill ~35). 2.5 GB free @1.7 baseline @24k (✓ 10-02 on b11146, `-n 512`, acceptance 0.84). The *felt* slowness is the **xhigh thinking preamble** (17.5 s before the first answer token on a trivial prompt; `reasoning_effort=low` measured 6.45 s at the same decode on 10-02 — siblings benched then removed by decision, the kwarg is one line if ever wanted); **never on a busy evening desktop** (~1 GB free = GTT freeze) |
|
||||
| `Qwen3.8-27B-UD-IQ3_XXS` | **36–37** on b11429 (10-22 ×2; was 15.5 on b11146, 10.9 base) | 24k | vision, MTP, hybrid reasoner @ **xhigh** | ⚠ **clean-desktop specialty** (kept 09-10 over the ≥22 gate — **the speed half is now passed 1.7× over**, since llama.cpp 0.6.0 halved the per-step MTP cost; role re-rank pending the user) — dense-27B quality; prefill still GDN-bound (~20–35). 2.7 GB free @1.3 baseline @24k (✓ 10-22 on b11429, `-n 512`, acceptance 0.89, len 2.78). The *felt* slowness is the **xhigh thinking preamble** (17.5 s before the first answer token on a trivial prompt; `reasoning_effort=low` measured 6.45 s at the same decode on 10-02 — siblings benched then removed by decision, the kwarg is one line if ever wanted); **never on a busy evening desktop** (~1 GB free = GTT freeze) |
|
||||
| `Qwen3-Embedding-0.6B` | 33 (CPU) | 8k | CPU-only, `/v1/embeddings` | RAG/search embedder (not a chat model) |
|
||||
| `Ornith-1.5-35B-A3B-IQ3_XXS` | **32.1** @moe16 | 24k | MTP, vision, thinking | **agentic coder** (added 09-21) — Qwen3.5-MoE derivative claiming +15 Terminal-Bench / +5.6 SWE-bench over Qwen3.6-35B; the only coder with MTP; 13.8 used / 3.3 free @2.0 |
|
||||
| `Ornith-1.5-35B-A3B-IQ3_XXS` | **32.1** @moe16 | **32k** (10-22, unbenched) | MTP, vision, thinking | **agentic coder** (added 09-21) — Qwen3.5-MoE derivative claiming +15 Terminal-Bench / +5.6 SWE-bench over Qwen3.6-35B; the only coder with MTP; 13.8 used / 3.3 free @2.0 |
|
||||
|
||||
Expected run-to-run spread: MTP models swing ±15% with draft **acceptance rate** (content-
|
||||
dependent); CPU-heavy presets (Coder-Next, embedder) dip under daytime CPU contention.
|
||||
@@ -94,9 +94,9 @@ Treat clean night runs as the reference; don't retune on daytime deltas.
|
||||
- **Hard reasoning**: `35B-Thinking` (quality) or `gpt-oss-20b` (speed + tool use).
|
||||
Their multi-second TTFT is the reasoning phase streaming first — not a slow load.
|
||||
For quick interactive gpt-oss answers use `gpt-oss-20b-low` (reasoning effort low).
|
||||
- **Best single-model quality, speed no object** (quiet desktop only): `Qwen3.8-27B` — dense
|
||||
27B hybrid reasoner + vision at ~14.5 t/s; it cannot go faster on this card (GDN layers on
|
||||
Vulkan), and a busy evening desktop pushes it into GTT-freeze territory — check `rocm-smi` first.
|
||||
- **Best single-model quality** (quiet desktop only): `Qwen3.8-27B` — dense 27B hybrid reasoner +
|
||||
vision at **37 t/s since llama.cpp 0.6.0** (was ~15 on 0.5.0; prefill still GDN-bound on Vulkan),
|
||||
but a busy evening desktop pushes it into GTT-freeze territory — check `rocm-smi` first.
|
||||
- **Images**: `gemma-4-26B` for quality, `gemma-4-E4B` or `Qwen3.5-9B` for speed,
|
||||
`Qwen3.6-35B` when you want the daily driver to see the screenshot.
|
||||
- **Long one-shot documents**: `gemma-4-E4B` — 185 t/s prefill eats 64k in ~6 min
|
||||
@@ -150,8 +150,13 @@ the router, but chats/presets saved against the old id need re-picking.
|
||||
2026-09-10: kept as ⚠ clean-desktop specialty after the gate bench finally ran on the v3 gguf
|
||||
(14.5 t/s MTP / 10.9 base, 2.8 GB free @1.5); server ctx restored to 24576 = client ctx, no client edits.
|
||||
2026-09-21: `GLM-4.7-Flash-UD-Q3_K_XL` (⚡ fast coder) added to all six client files next to the Q4.
|
||||
2026-10-22: Qwen3.8-27B quant race opened (ISTA GSQ-RCO IQ3_XXS-mtp + DFlash2 sibling) — bench-only,
|
||||
NO client edits until a candidate wins; see the watch list. Same night: fl found on llama.cpp 0.6.0
|
||||
(b11429) and the Qwen3.8 incumbent re-measured 37.1 t/s (15.5 on b11146) — every MTP/hybrid preset
|
||||
on the roster needs a re-sweep on this build; ledger numbers above it are from 0.5.0 unless dated 10-22.
|
||||
2026-09-21: `Ornith-1.5-35B-A3B-IQ3_XXS` (agentic coder, thinking + vision) added to all six client files after passing its gate (32.1 t/s @moe16).
|
||||
2026-09-21: default model → `Qwen3.6-35B-A3B-Thinking` everywhere (opencode global + plan/build/auto, pi defaultModel + thinking level medium, aichat common/lw/wm) — gemma-4-26B refused routine sysadmin work. Thinking preset gets `reasoning-budget 8192`. wm aichat caught up (GLM Q3, Ornith).
|
||||
2026-10-22: Ornith in pi: `Compacting in OVERFLOW recovery` right after a truncated reply is pi's *truncated-reply* recovery, not a full window — pi clamps each request's `max_tokens` to `contextWindow − estimate − 4096`, which on 24k/4096 starts exactly at the compaction threshold, so the first request after a compaction (estimate gains the system prompt + tool schemas) can leave a thinking reply ~1024 answer tokens → `length` → compact + retry. `compaction-watch.ts` now names the case with numbers; `common/.pi/CLAUDE.md` > Compaction tuning has the arithmetic. Applied the same day, UNBENCHED: Ornith `ctx-size` 24576 → 32768 + the seven client files (pi/opencode/aichat, common+lw+wm; predicted ≈ 0.1 GB of KV on this hybrid arch) — restart llama.service, then `./bench.py -m Ornith-1.5-35B-A3B-IQ3_XXS`; gate and the revert path are in the preset comment.
|
||||
2026-10-12: Ornith in pi: "Response was truncated before completion." = `finish_reason: length` — the `<think>` block counts against pi's 4096 `max_tokens` and the preset has no reasoning-budget. Fixed client-side: pi sends `thinking_budget_tokens` per thinking level on Ornith (`compat.thinkingTokenBudgetField` + `settings.json > thinkingBudgets`, common + lw). The preset deliberately keeps NO CLI reasoning-budget — it would override the per-request field (see the preset comment).
|
||||
|
||||
## Tuning cheat-sheet
|
||||
@@ -200,9 +205,27 @@ reasoning-effort siblings; for stable MTP decode numbers use `-n 512`.
|
||||
## Watch list
|
||||
|
||||
- **GLM MTP in llama.cpp** — the single biggest pending win (primary coder ~22 → ~30+).
|
||||
- **DFlash2 for Qwen3.8-27B** — `z-lab/Qwen3.8-27B-DFlash2-GGUF` drafter, upstream 1.81× vs the
|
||||
MTP head's 1.33× here (b11146 has the support). Blockers: #27408 (spec + mmproj → image requests
|
||||
500, so only as a text-only sibling preset) and #28158 (Vulkan OOB draft token, open). Trial =
|
||||
`spec-type draft-dflash`, `spec-draft-n-max 3`, no mmproj; n-max 7 does not fit the VRAM envelope.
|
||||
- **Qwen3.8-27B quant race (opened 2026-10-22, bench-only, not client-wired)** — two candidate
|
||||
sections in `config.ini` after the incumbent: `Qwen3.8-27B-GSQ-RCO-IQ3_XXS` (ISTA-DASLab GSQ-RCO
|
||||
file, 10.44 GB with MTP head = −0.49 GB vs unsloth v3) and its `-dflash` text-only sibling
|
||||
(`z-lab/Qwen3.8-27B-DFlash2-GGUF` Q4_K_M drafter, `spec-draft-n-max 3`). Expected: +4 % decode and
|
||||
+0.5 GB free from the file; +5–15 % over MTP from DFlash2 on RDNA3/Vulkan (not the paper's 1.8×).
|
||||
Quality is unmeasured against our file — the user's own prompts decide. First run 10-22 03:14: both
|
||||
candidates HTTP 400 = fl's config.ini not pulled yet (router: "model not found"); the incumbent in
|
||||
the same run read **37.1 t/s on b11429** (see its row). Re-run 03:25, one session: incumbent 36.2 /
|
||||
2.7 free, **GSQ 37.5 / 3.2 free** (+3.6 %, +0.5 GB — speed+VRAM passed, **quality call open**),
|
||||
**DFlash2 15.4 / 1.9 free → dropped** (tombstoned in `config.ini`). Ruled out by the same
|
||||
research: ByteShape IQ4_XS (13.1 GB), ISTA IQ3_S (12.1 GB with MTP, under the floor), the
|
||||
adaptive KV-cache-streaming fork (CUDA-only, PCIe 5.0). Still open: #27408 (spec + mmproj → 500),
|
||||
#28158 (Vulkan OOB draft token). Research + verdicts + download lines: the config block.
|
||||
- **llama.cpp 0.7.0 (b11669) — HOLD fl at 0.6.0**: #30688, Vulkan draft-mtp decode 4× / prefill 6×
|
||||
slower (target compute buffer lands in GTT = this box's freeze precursor). Every MTP preset on the
|
||||
roster is exposed. When `pacman -Si llama-cpp` on fl shows 0.7.0, `IgnorePkg` it until fixed.
|
||||
The 0.5.0 → 0.6.0 jump itself (Qwen3.8 15.5 → 37) is llama-side, best explained by #29856 (hybrid
|
||||
graph reserve; the old 10.9 base was probably GTT-starved, not kernel-bound — see the preset block).
|
||||
**ggml 0.26.0** (next `-Syu`): #30039 reports a −45 % prefill regression on a GDN hybrid — bench
|
||||
Qwen3.8 before and after; downgrade ggml/ggml-vulkan from the pacman cache if prefill tanks.
|
||||
- **Coder-Next 256k**: if the load log's `n_ctx_train` says 262144, ctx can likely double
|
||||
again for ~1 GB (KV measured nearly flat).
|
||||
|
||||
> **vLLM on this box** (bench rig, docker/ROCm, never resident together with the router): see `../vllm/README.md`.
|
||||
@@ -102,7 +102,9 @@ def force_unload(base, key):
|
||||
print(f" (router /models not readable: {e} — relying on sleep-idle drain)", file=sys.stderr)
|
||||
return
|
||||
for m in d.get("data", []):
|
||||
if m.get("status", "loaded") != "loaded": # builds without 'status': try them all
|
||||
st = m.get("status", "loaded") # b11146 sends {'value': 'loaded'|'sleeping'|'unloaded'}; older a string
|
||||
st = st.get("value", "loaded") if isinstance(st, dict) else st
|
||||
if st not in ("loaded", "loading"): # builds without 'status': try them all
|
||||
continue
|
||||
try:
|
||||
_open(root + "/models/unload", key, {"model": m["id"]}, timeout=30).read()
|
||||
|
||||
@@ -379,6 +379,36 @@ sleep-idle-seconds = 60
|
||||
# stall/500) → any trial is a separate TEXT-ONLY preset; this one keeps MTP + vision.
|
||||
# Open on Vulkan for MTP and DFlash alike: #28158 — intermittent draft token id 248320
|
||||
# (= n_vocab) → HTTP 500 "Invalid input batch", mostly after very long prompts. If seen: that.
|
||||
# ✦ 2026-10-22 QUANT RACE OPENED (reddit tip: ByteShape / ISTA 3-bit quants + DFlash2 + KV streaming):
|
||||
# two bench-only candidate sections follow this preset — [Qwen3.8-27B-GSQ-RCO-IQ3_XXS] (ISTA file,
|
||||
# −0.49 GB, MTP) and its -dflash text-only sibling; the research + verdicts live in that block.
|
||||
# ⚠ llama.cpp 0.7.0 (b11669) = #30688, Vulkan draft decode 4× slower (compute buffer lands in GTT):
|
||||
# hold fl at 0.6.0 until fixed — hits EVERY MTP preset here, not only this one.
|
||||
# ✦ BUILD DATUM 2026-10-22 03:15 — fl turned out to be on 0.6.0 (b11429) + ggml-vulkan 0.25.3-2 (the
|
||||
# SAME kernels as 10-02): bench 37.1 t/s / 21 pp / TTFT 3.3 / 14.4 used / 2.7 free @1.3 (-n 512).
|
||||
# Server: 221 tok @ 37.07, draft acceptance 0.892 (141/158), mean len 2.78 → ~75 ms per
|
||||
# speculative step vs ~173 on b11146 — a llama.cpp-side change (b11147..b11429), NOT the Vulkan
|
||||
# kernels (ggml is a separate package, unchanged). A 4096-token think-run: 27.3 t/s @ acceptance
|
||||
# 0.51 / len 2.02 = the same ~74 ms step; acceptance sags over long thinking. Footprint 13.1,
|
||||
# unchanged. → the ≥22 gate is PASSED 1.7× over; the 'clean-desktop specialty' verdict's SPEED half
|
||||
# is obsolete (the VRAM half — never on a busy desktop — still stands). Role re-rank = user's call.
|
||||
# 03:25 re-run: 36.2 / 45 pp / TTFT 8.6 / 14.5 / 2.7 — reproducible.
|
||||
# CAUSE (researched 10-22; a hypothesis — no PR in b11147..b11429 claims any such gain): Arch's llama-cpp
|
||||
# links the SYSTEM ggml (-DLLAMA_USE_SYSTEM_GGML=ON), so every ggml-side Vulkan PR (#29476 GDN tune,
|
||||
# #29280 descriptor reuse, FA work) is EXCLUDED — those ship in ggml 0.26.0, not on fl yet. Best fit:
|
||||
# #29856 (10-03) "gather the recurrent states once so the reserve covers every split" — on 0.5.0 the
|
||||
# hybrid graph was silently re-reserved after the first speculative step and the target compute
|
||||
# buffer could land partly in GTT = the #30688 mechanism seen from the other side. The arithmetic
|
||||
# agrees: the old no-MTP base (10.9 t/s = 92 ms/token) was SLOWER than today's whole MTP step (75 ms
|
||||
# for ≥3 verified tokens + 2 draft passes), so the "10.9 base = the GDN kernel wall" story above is
|
||||
# SUSPECT: that base was most likely GTT-starved, not kernel-bound. Cheap check whenever the card is
|
||||
# free: a TEMP -nomtp alias on 0.6.0 — ~22–26 t/s confirms it, ~11 refutes it. #27694 (probabilistic
|
||||
# drafter, 10-02) explains acceptance 0.84 → 0.89, not the step time. (Also learned: the 9B is hybrid
|
||||
# too, 24 GDN + 8 attn per #30039 — the "regular-attention qwen35" remark further down is wrong.)
|
||||
# ⚠ ggml 0.26.0 (the next -Syu brings ggml + ggml-vulkan 0.26.0-1, SOVERSION-compatible, unrebuilt):
|
||||
# #30039 reports −45 % PREFILL on a GDN hybrid with b11429 binaries, TG flat → bench this preset right
|
||||
# before and after that upgrade; if prefill tanks, downgrade ggml/ggml-vulkan from the pacman cache.
|
||||
# REFERENCE from here: 37.1 (b11429). Every other MTP/hybrid preset needs a re-sweep on this build.
|
||||
# ⚠ Needs a CURRENT llama.cpp build: first "qwen35" file HERE with ssm.*/hybrid-attn keys
|
||||
# (the 9B is regular-attention qwen35). If the load fails with unknown-architecture /
|
||||
# missing-tensor errors, update llama.cpp on fl before touching this preset.
|
||||
@@ -418,6 +448,150 @@ min-p = 0
|
||||
jinja = on
|
||||
sleep-idle-seconds = 120
|
||||
|
||||
# ═══ QUANT RACE 2026-10-22 — Qwen3.8-27B third-party 3-bit quants + DFlash2 (bench-only, NOT client-wired) ═══
|
||||
# Trigger: r/LocalLLaMA 2026-09-24 "Qwen-3.8-27B is good enough that I stopped using API": a 16 GB user
|
||||
# (RTX 5060 Ti, CUDA, PCIe 5.0 x8, display on the iGPU) runs ByteShape IQ4_XS + DFlash2 + Raymond's
|
||||
# KV-cache streaming; a reply says "try the ISTA IQ3_S / XXS — more speed and context, couldn't tell a
|
||||
# difference in quality". What survives contact with THIS box (16 GB + display, Vulkan/RADV, PCIe 4.0 x8):
|
||||
# · KV-cache streaming (troed → RaymondHuang210129/llama.cpp-adaptive-kv-streaming): OUT. CUDA-only —
|
||||
# the README lists Vulkan as "memory-infrastructure tests only" (no attention path), ROCm = one
|
||||
# unreviewed PR for gfx1201; both authors say PCIe 5.0 is required (PCIe 4.0 users: 2–4 t/s @192k);
|
||||
# no upstream PR (discussion #28216 only). And it is a 160k+ tool — clients run this model at 24k.
|
||||
# · ByteShape (byteshape/Qwen3.8-27B-GGUF, proprietary "ShapeLearn" per-tensor mixed types, the IQ*
|
||||
# labels are size classes; MTP head kept, quantized IQ3/IQ4 vs Q6_K/Q8_0 elsewhere): IQ4_XS-3.84bpw
|
||||
# 13.08 GB → OUT (ceiling here ≈ 11 GB file); the commenter's own file is troed's 12.25 GB
|
||||
# ASCII-vocab-pruned build (248320 → 129006 tokens) → OUT twice. Same size class as the incumbent:
|
||||
# IQ3_S-3.23bpw 11.035 GB (vendor blog: 98.7 % of BF16 task avg vs 93.6 % for unsloth UD-IQ3_XXS —
|
||||
# vendor numbers; the one independent KLD run ranks their IQ4_XS BELOW unsloth's UD-IQ4_XS, 0.050 vs
|
||||
# 0.019), IQ3_XS-3.01bpw 10.353 GB, IQ3_XXS-2.88bpw 9.919 GB. Optional third runner, see below.
|
||||
# · ISTA (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF: GSQ = learned per-coordinate grids, RCO = per-tensor
|
||||
# type under a size budget, standard GGUF block types, imatrix ~90 % English — discussion #7 reports
|
||||
# degraded Chinese; "-mtp" files carry the head, +0.35 GB, weights identical):
|
||||
# IQ3_S-mtp 12.12 GB → footprint ≈ 14.2 → ~1.4 free @1.6 = UNDER THE 1.5 FLOOR. Plain IQ3_S 11.77
|
||||
# fits only WITHOUT any drafter (~3.4 free) at ~10 t/s → no. Both reddit comments measured "more
|
||||
# speed and context" against a 13 GB file, not against a 3-bit one.
|
||||
# IQ3_XXS-mtp 10.443 GB = 0.49 GB under the incumbent → the one ISTA file with a case here.
|
||||
# Quality evidence (nobody measured it against OUR unsloth v3 file): ISTA card wiki-ppl 7.20 vs
|
||||
# BF16 7.05, AIME25/GPQA-D/LCBv6 within 1–2 pts of BF16; ByteShape's blog scores it 94.4 vs 93.6
|
||||
# for UD-IQ3_XXS (= equal); Agention's KLD table puts it at 0.11, ~2× their own IQ3_XXS. Thread:
|
||||
# one user "ISTA even at lower quants did better", troed "tried it, not as good for my use".
|
||||
# → bench.py settles speed + VRAM only; the quality half is the user's own prompts, both files.
|
||||
# · DFlash2 (z-lab/Qwen3.8-27B-DFlash2-GGUF: 1.92B block-diffusion drafter, 5 layers, 2048-token
|
||||
# sliding window, no own embeddings/output head; Q4_K_M 1.143 GB; llama.cpp support since b10658 =
|
||||
# already in fl's b11146): acceptance length ~5.4 vs the MTP head's 2.6 here. RDNA3/Vulkan reports
|
||||
# (PR #27342 thread): 7900 XTX RADV n-max 3 = +4–12 % decode over MTP, prefill −1 to −12 %; Strix
|
||||
# Halo RADV n-max 4 = 19–23 t/s vs MTP 20; R9700 n-max 4 = 52 vs 47; a net LOSS once layers spill.
|
||||
# Expect +5–15 % over the incumbent (37.1 on b11429, 10-22), NOT the paper's 1.8× — every verified draft token
|
||||
# re-runs the 48 GDN layers. Acceptance falls with context (5090 datum: 31–36 % at 23–63k vs MTP
|
||||
# 55 %) = why the redditor says "at low context".
|
||||
# VRAM: drafter 1.14 GB + (1 + n-max) × ~150 MiB recurrent-state snapshots (n-max 3 ≈ 0.6 GB) +
|
||||
# drafter KV — the MTP head measured 1.5 GB against a ~0.7 estimate, so budget ~2 GB and let the
|
||||
# bench.py guard decide; the ISTA file's −0.49 GB is what buys the margin. Default n-max 7 = OUT.
|
||||
# Text-only: #27408 (spec + mmproj → image requests 500) still open → NO mmproj on that preset.
|
||||
# #28158 (Vulkan: draft token id 248320 → HTTP 500, rare, very long prompts) still open.
|
||||
# Build window: #29226 (b11132) broke drafters WITH embeddings but WITHOUT an output head (fixed by
|
||||
# #30111, b11512); the z-lab drafter has neither → should not be hit. FIRST GATE regardless:
|
||||
# "draft acceptance" in the server log > 0 (0.00 = that class of breakage → drop the sibling).
|
||||
# ⚠ BUILD: fl = llama-cpp 0.6.0-1 (b11429) + ggml 0.25.3-2 + ggml-vulkan 0.25.3-2, CONFIRMED 10-22 (the
|
||||
# b11146 → b11429 move alone took the incumbent 15.5 → 37.1, see its block). Arch extra also has
|
||||
# ggml/ggml-vulkan 0.26.0 (#27952 int8 coopmat prefill, #29476 GDN tune) — the next -Syu brings them
|
||||
# = another variable, bench before/after. Upstream 0.7.0 (b11669) = #30688: Vulkan
|
||||
# draft-mtp decode 4× / prefill 6× SLOWER, the target compute buffer re-allocated into GTT — on this
|
||||
# box that is the freeze precursor, and it hits every MTP preset on the roster. When
|
||||
# `pacman -Si llama-cpp` on fl shows 0.7.0: HOLD (IgnorePkg = llama-cpp ggml ggml-vulkan) until fixed.
|
||||
# Race on the current build first (one variable per run); the ggml 0.26.0 bump is a separate round.
|
||||
# PLAN (clean baseline, -n 512, incumbent re-run in the SAME session — the 10-02 band was 12.5–15.5):
|
||||
# 1. download (below) · 2. `sudo systemctl restart llama.service` ·
|
||||
# 3. ./bench.py -n 512 -m Qwen3.8-27B-UD-IQ3_XXS,Qwen3.8-27B-GSQ-RCO-IQ3_XXS,Qwen3.8-27B-GSQ-RCO-IQ3_XXS-dflash
|
||||
# 4. quality: the same 3–4 real prompts (incl. non-English) against incumbent vs GSQ.
|
||||
# GATE: GSQ-mtp ≥ incumbent decode (expected ~+4 %) with free ≥ 2.5 → quality is the user's call →
|
||||
# if it wins, either point the incumbent section's `model =` at the GSQ file (keeps the client id,
|
||||
# accepted misnomer) or rename = the seven-client-file round. DFlash sibling: acceptance > 0 AND
|
||||
# decode ≥ 1.10 × the SAME-SESSION incumbent AND free ≥ 2.5 → keep as the text-only fast sibling
|
||||
# (wire into clients then); else drop. Loser files: delete from ~/software/models (10.4 + 1.1 GB).
|
||||
# ✦ RACE RUN 2026-10-22 03:25 @1.3 baseline, -n 512, all three in ONE session (rows in bench-history.md):
|
||||
# incumbent UD-IQ3_XXS ....... 36.2 t/s / 45 pp / TTFT 8.6 / 14.5 used / 2.7 free
|
||||
# GSQ-RCO-IQ3_XXS (MTP) ...... 37.5 / 44 / 7.0 / 14.0 / 3.2 → +3.6 % decode, +0.5 GB free = exactly the
|
||||
# file-size prediction. Speed + VRAM half of the gate PASSED. OPEN: the quality half (same 3–4 real
|
||||
# prompts, one non-English, against both; nobody has measured ISTA vs unsloth v3 at this size).
|
||||
# GSQ -dflash ................ 15.4 / 32 / 18.0 / 15.3 / 1.9 → −59 % vs its own MTP row and under the
|
||||
# 2.5 target → DROPPED (tombstone below). Read the acceptance line once: `journalctl -u llama.service
|
||||
# --since 03:25 | grep 'draft acceptance'` — 0.00 = the #29226-window bug (fix #30111 = b11512 sits in
|
||||
# 0.7.0 behind #30688 → re-probe only on a fixed 0.7.x); > 0 = block-drafting re-runs the 48 GDN
|
||||
# layers per verified block and loses on this card (the RX 9070 XT datum in #25117 said the same)
|
||||
# → closed for good. Either way nothing to tune: n-max 2 only shrinks the loss.
|
||||
# NEXT: quality call → if GSQ wins, point the incumbent's `model =` at the GSQ file (keeps the client
|
||||
# id; the section name becomes a misnomer) or do the seven-client rename round, then delete the
|
||||
# unsloth file; if not, delete the GSQ file and the section. The DFlash2 drafter can go now.
|
||||
# ⬇ DOWNLOAD on fl — curl straight into the models dir (the HF cache has root-owned dirs from docker):
|
||||
# cd ~/software/models
|
||||
# curl -L -o Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF/resolve/main/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf
|
||||
# (dropped 10-22, only for a re-probe) curl -L -o Qwen3.8-27B-DFlash2-Q4_K_M.gguf https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/resolve/main/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
|
||||
# stat -c '%s %n' Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf Qwen3.8-27B-DFlash2-Q4_K_M.gguf # 10442827840 · 1143006816
|
||||
# (mmproj: ISTA ships the same BF16 encoder — the existing mmproj-Qwen3.8-27B-F16.gguf is reused)
|
||||
|
||||
[Qwen3.8-27B-GSQ-RCO-IQ3_XXS]
|
||||
# ⚗ RACE CANDIDATE 2026-10-22 — bench-only, NOT in any client file. = the incumbent preset with the ISTA
|
||||
# file swapped in (MTP head Q6_K, same as unsloth's). Predicted @1.6 baseline @24k: ~14.1 used /
|
||||
# ~3.0 free (incumbent 14.4 / 2.7), decode ~38–39 (incumbent 37.1 on b11429) — bandwidth × file size, nothing
|
||||
# else moves. Knobs deliberately identical to the incumbent (n-max 2) for a clean A/B.
|
||||
# ✓ MEASURED 2026-10-22 03:25 @1.3: 37.5 t/s / 44 pp / TTFT 7.0 / 14.0 used / 3.2 free (incumbent 36.2 /
|
||||
# 2.7 in the same run). Speed + VRAM passed; QUALITY CALL PENDING — not client-wired until then.
|
||||
model = /home/anon/software/models/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf
|
||||
mmproj = /home/anon/software/models/mmproj-Qwen3.8-27B-F16.gguf
|
||||
spec-type = draft-mtp
|
||||
spec-draft-n-max = 2
|
||||
ctx-size = 24576
|
||||
n-gpu-layers = 99
|
||||
threads = 6
|
||||
flash-attn = on
|
||||
cache-type-k = q8_0
|
||||
cache-type-v = q4_0
|
||||
batch-size = 2048
|
||||
ubatch-size = 256
|
||||
cache-reuse = 256
|
||||
defrag-thold = 0.1
|
||||
no-mmproj-offload = true
|
||||
temp = 1.0
|
||||
top-p = 0.95
|
||||
top-k = 20
|
||||
min-p = 0
|
||||
jinja = on
|
||||
sleep-idle-seconds = 120
|
||||
|
||||
# ─── DROPPED 2026-10-22 03:25: [Qwen3.8-27B-GSQ-RCO-IQ3_XXS-dflash] (DFlash2 text-only sibling) ─────
|
||||
# 15.4 t/s / 32 pp / 15.3 used / 1.9 free vs 37.5 / 3.2 for the same file with its MTP head — the
|
||||
# z-lab block drafter is a net loss on this card (see the RACE RUN entry above for the two possible
|
||||
# causes and the only re-probe condition: a 0.7.x build that carries #30111 without #30688).
|
||||
# Removed as a loadable section on purpose: OWUI lists every router id, and 1.9 free on a busy
|
||||
# desktop is the GTT freeze. File can be deleted from the models dir: Qwen3.8-27B-DFlash2-Q4_K_M.gguf
|
||||
# Keys as benched (n-max 3, no mmproj — #27408):
|
||||
# model = /home/anon/software/models/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf
|
||||
# model-draft = /home/anon/software/models/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
|
||||
# spec-type = draft-dflash
|
||||
# spec-draft-n-max = 3
|
||||
# ctx-size = 24576
|
||||
# n-gpu-layers = 99
|
||||
# threads = 6
|
||||
# flash-attn = on
|
||||
# cache-type-k = q8_0
|
||||
# cache-type-v = q4_0
|
||||
# batch-size = 2048
|
||||
# ubatch-size = 256
|
||||
# cache-reuse = 256
|
||||
# defrag-thold = 0.1
|
||||
# temp = 1.0
|
||||
# top-p = 0.95
|
||||
# top-k = 20
|
||||
# min-p = 0
|
||||
# jinja = on
|
||||
# sleep-idle-seconds = 120
|
||||
|
||||
# ─── optional 3rd runner (not added — copy the GSQ section if wanted): ByteShape IQ3_S-3.23bpw ────
|
||||
# 11.035 GB = the incumbent's size, vendor-claimed best quality in the class, MTP head kept (IQ3-quantized).
|
||||
# model = /home/anon/software/models/Qwen3.8-27B-IQ3_S-3.23bpw.gguf
|
||||
# curl -L -o Qwen3.8-27B-IQ3_S-3.23bpw.gguf https://huggingface.co/byteshape/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-IQ3_S-3.23bpw.gguf # 11035354496 B
|
||||
|
||||
# ─── RETIRED 2026-07-16: [Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-IQ3_M] ──────────────
|
||||
# Dense: 10 t/s, 2.0 GB free (under the 2.5 target) and unfixable — bandwidth-bound, smaller
|
||||
# quant would cost the quality that justified it. Removed from opencode/pi/aichat.
|
||||
@@ -552,6 +726,22 @@ sleep-idle-seconds = 30
|
||||
# here "like the Thinking preset" silently disables the pi knob and brings back pi's
|
||||
# "Response was truncated before completion." (= finish_reason length: think block + answer
|
||||
# > the client's 4096 max_tokens). An instruct alias (budget 0) in a SEPARATE section is fine.
|
||||
# ⚠ 2026-10-22: pi's "Compacting in OVERFLOW recovery" on this preset is NOT this window filling
|
||||
# up — pi clamps each request's max_tokens to contextWindow − estimate − 4096, which on
|
||||
# 24k/4096 starts exactly at its compaction threshold; the first request after a compaction
|
||||
# (estimate gains the system prompt + tool schemas) can leave the thinking reply ~1024 answer
|
||||
# tokens → finish_reason length → pi compacts again and retries. Arithmetic + options in
|
||||
# common/.pi/CLAUDE.md > Compaction tuning > max_tokens clamp.
|
||||
# APPLIED 2026-10-22 (same day, unbenched): ctx-size 24576 → 32768 here + 32768 in the seven
|
||||
# client files (pi contextWindow both blocks common+lw, opencode limit.context common+lw,
|
||||
# aichat max_input_tokens 30000 common+lw+wm). Threshold and clamp in pi now sit at 24576,
|
||||
# above anything a fresh post-compaction context reaches. Predicted cost ≈ 0.1 GB: hybrid
|
||||
# GDN 3:1 = 10 attention layers with small GQA, ~8 KB/token of q8/q4 KV → ~70 MB for +8192,
|
||||
# plus a few MB of FA scratch — read the "KV self size" line in the load log to confirm.
|
||||
# GATE on the first load: `sudo systemctl restart llama.service`, then
|
||||
# `./bench.py -m Ornith-1.5-35B-A3B-IQ3_XXS` — free ≥ 2.5 at the 2.0 baseline and decode
|
||||
# ≈ 32 t/s → keep; else back to 24576 in all eight files (the router reads presets once at
|
||||
# startup, and pi at 32k against a 24k server produces REAL overflow recoveries).
|
||||
# ⬇ DOWNLOAD from bartowski/Ornith-1.5-35B-A3B-GGUF (imatrix; the official repo has only Q4_K_M+
|
||||
# ≥ 21.7 GB, unsloth has no 1.5 quants yet):
|
||||
# Ornith-1.5-35B-A3B-IQ3_XXS.gguf 15,340,447,392 B · sha256 8918ccb9ee29abe3875efec0c3e86f0e35ef0b8a03e3f0e1c422869859a518d0
|
||||
@@ -560,7 +750,9 @@ model = /home/anon/software/models/Ornith-1.5-35B-A3B-IQ3_XXS.gguf
|
||||
mmproj = /home/anon/software/models/mmproj-Ornith-1.5-35B-A3B-f16.gguf
|
||||
spec-type = draft-mtp
|
||||
spec-draft-n-max = 2
|
||||
ctx-size = 24576 # = the daily driver; n_ctx_train 262144 — raise only after a bench
|
||||
ctx-size = 32768 # 24576 → 32768 on 2026-10-22 for pi's max_tokens clamp (⚠ note above);
|
||||
# UNBENCHED until the next restart — gate free ≥ 2.5 @2.0, decode unchanged;
|
||||
# n_ctx_train 262144, KV is cheap here (hybrid GDN, ~8 KB/token)
|
||||
n-gpu-layers = 99
|
||||
n-cpu-moe = 16 # ✓ KEEP (09-21: 32.1 t/s, 13.8 used / 3.3 free @2.0; moe20 was 27.9 /
|
||||
# 12.4 / 4.8). = daily-driver envelope; moe14 not worth the margin.
|
||||
|
||||
Reference in new issue
Block a user