[Fix] cleanup
This commit is contained in:
coja
@@ -0,0 +1,214 @@
|
||||
# llama.cpp model config — LAPTOP (lw) · CPU inference, NO discrete GPU
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
# HARDWARE: Intel Core i7-5600U (Broadwell 2015, 2 cores / 4 threads, AVX2, no AVX-512)
|
||||
# · iGPU HD Graphics 5500 (i915, not used for compute) · 15.49 GiB RAM
|
||||
# · DDR3L-1600 dual-channel ≈ ~20 GB/s effective → THE bottleneck.
|
||||
#
|
||||
# This is the laptop analog of fl/.config/llamacpp/config.ini (the RX 7600 XT box), but on a
|
||||
# CPU-only machine almost every fl rule inverts:
|
||||
#
|
||||
# • NO VRAM / GTT / freeze risk — no display-driving GPU to starve. The only ceiling is SYSTEM
|
||||
# RAM. The `llamaserver` abbr uses `--models-max 1` (one model resident; a new request LRU-
|
||||
# evicts the previous — safest on this 15 GB laptop, no co-residency swap). Raise to 2-3 only
|
||||
# to keep small models hot together (E2B+4B+1.7B ≈ 6 GB fits).
|
||||
# • Decode speed ≈ effective bandwidth ÷ active-bytes, so SMALL QUANTS are directly faster and
|
||||
# MoE (few active params) is the single biggest trick. Clean bench 2026-07-25 (n-gpu-layers=0):
|
||||
# dense models reach ~8-10 GB/s effective (decode t/s × model GB) ≈ the PRACTICAL DDR3L wall,
|
||||
# i.e. dense decode IS bandwidth-bound as designed. (The earlier "core-starved ~5-7 GB/s"
|
||||
# reading was the Vulkan-offload taint.) Granite (~2.6 GB/s eff) and the 30B (~3.5) sit far
|
||||
# BELOW the wall → COMPUTE-bound: threads 3/4 might lift those two (dense won't gain).
|
||||
# • `threads = 2` = PHYSICAL cores for the dense presets (at the wall — swept 2026-07-26, no
|
||||
# gain). `threads = 3` on the compute-bound Granite/30B: +~5-7% same-run (t4 LOSES to HT
|
||||
# contention on both — never promote it). `threads-batch = 4` EVERYWHERE: prefill is
|
||||
# compute-bound on all models — benched +8% prefill / −9% TTFT on Granite, no decode cost
|
||||
# (HT only bursts during prefill, so no sustained desktop pressure).
|
||||
# • ⚠ RESPONSIVENESS: on 2 cores, generation pins the whole CPU → the desktop can freeze. The
|
||||
# `llamaserver` abbr runs the server under `nice -n 19`, so inference YIELDS to interactive apps
|
||||
# (still full speed when the desktop is idle). Still laggy? set `threads = 1` (≈half speed, always
|
||||
# responsive).
|
||||
# • `flash-attn = on` EVERYWHERE EXCEPT the two gemma-4 presets: benched ~+11% here (gen 3.7→4.1,
|
||||
# prompt 1.7→1.9 t/s) so it's worth keeping — BUT this build (b10068) FAILS TO LOAD the gemma3n
|
||||
# arch (E2B/E4B) with flash-attn, so those two stay OFF. (If another model also won't load with
|
||||
# it, drop it there too.) No cache-type-k/v (quantized KV) on CPU: dequant overhead, no VRAM to
|
||||
# reclaim → slower. KV stays f16.
|
||||
# • ⚠ `n-gpu-layers = 0` REQUIRED in every preset — the pacman build has a VULKAN backend and
|
||||
# by default AUTO-OFFLOADS layers to the HD 5500 iGPU (common_fit_params). Seen live
|
||||
# 2026-07-25: Granite/30B loads died with "ggml_vulkan ... ErrorOutOfDeviceMemory" (llama-cli
|
||||
# even SIGSEGVs). The iGPU shares the same DDR3 and is slower than the cores — offload also
|
||||
# silently poisons benchmarks ("omit it for CPU-only" was wrong on this build).
|
||||
#
|
||||
# Clean baseline 2026-07-25 17:41 (n-gpu-layers=0, threads=2, gen 256 — bench-history.md);
|
||||
# decode noted per preset below, roster table in the README. Prefill is slow on 2 cores, so
|
||||
# huge ctx is slow to FILL (not RAM-limited) — 4096 is a sane default; raise only if needed.
|
||||
#
|
||||
# Router: llama-server --host 127.0.0.1 --port 11343 --cors-origins localhost --models-max 1 \
|
||||
# (localhost-only: the laptop roams onto untrusted networks and there's no API key —
|
||||
# unlike fl, nothing on the LAN targets this server. --cors-origins localhost blocks
|
||||
# browser pages from calling the keyless API; non-browser clients unaffected — PR 25655)
|
||||
# --models-preset ~/.config/llamacpp/config.ini (abbr `llamaserver`; llama.cpp is
|
||||
# pacman-installed → `llama-server` is on PATH)
|
||||
# This file stows to ~/.config/llamacpp/ (like fl); the GGUFs live in ~/software/models/.
|
||||
# Section names ARE the API model ids.
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Fast / small — the "instant" tier (quick Q&A, askllama, autocomplete)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
[gemma-4-E2B-it-UD-Q4_K_XL]
|
||||
# Gemma 3n E2B (~2B effective/active) — FASTEST here: benched 6.1 t/s (clean 2026-07-25). Add an mmproj for vision.
|
||||
model = /home/anon/software/models/gemma-4-E2B-it-UD-Q4_K_XL.gguf
|
||||
# flash-attn OFF: gemma3n fails to load with it on this build (b10068).
|
||||
ctx-size = 4096
|
||||
threads = 2
|
||||
threads-batch = 4
|
||||
n-gpu-layers = 0
|
||||
jinja = on
|
||||
temp = 1.0 # Gemma default (1.0 / top-p 0.95 / top-k 64); drop to 0.7 for determinism
|
||||
top-p = 0.95
|
||||
top-k = 64
|
||||
sleep-idle-seconds = 30
|
||||
|
||||
[Qwen3-1.7B]
|
||||
# Qwen3-1.7B, Q8_0 (~1.8 GB — Q8 is fine at this size). Benched 4.8 t/s (clean 2026-07-25) —
|
||||
# ~8.8 GB/s effective ≈ at the bandwidth wall; more threads won't lift this one.
|
||||
model = /home/anon/software/models/Qwen3-1.7B-Q8_0.gguf
|
||||
ctx-size = 8192 # tiny KV — can afford more ctx
|
||||
threads = 2
|
||||
threads-batch = 4
|
||||
n-gpu-layers = 0
|
||||
flash-attn = on
|
||||
jinja = on
|
||||
temp = 0.7
|
||||
top-p = 0.8
|
||||
top-k = 20
|
||||
min-p = 0
|
||||
repeat-penalty = 1.05
|
||||
sleep-idle-seconds = 30
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Daily drivers — 4B-class (best quality/speed balance; benched 2.1-3.9 t/s, clean 2026-07-25)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
[Qwen3-4B-Instruct-2507]
|
||||
# Qwen3-4B Instruct 2507, Q4_K_M (~2.5 GB). Benched 3.9 t/s (clean 2026-07-25). Best small
|
||||
# all-rounder; non-thinking, compact answers. The natural DAILY DRIVER — askllama points here.
|
||||
model = /home/anon/software/models/Qwen3-4B-Instruct-2507.Q4_K_M.gguf
|
||||
ctx-size = 4096
|
||||
threads = 2
|
||||
threads-batch = 4
|
||||
n-gpu-layers = 0
|
||||
flash-attn = on
|
||||
jinja = on
|
||||
temp = 0.7
|
||||
top-p = 0.8
|
||||
top-k = 20
|
||||
min-p = 0
|
||||
repeat-penalty = 1.05
|
||||
sleep-idle-seconds = 30
|
||||
|
||||
[gemma-4-E4B-it-UD-Q4_K_XL]
|
||||
# Gemma 3n E4B (~4B active), QAT Q4_K_XL (~4-7 GB). Quality step up from E2B; add an mmproj for vision.
|
||||
# (File is the -qat- build — QAT keeps 4-bit quality; the id drops "qat" to match fl's convention.)
|
||||
model = /home/anon/software/models/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf
|
||||
# flash-attn OFF: gemma3n fails to load with it on this build (b10068).
|
||||
# Benched 2.4 t/s (nodraft, clean 2026-07-26) — same speed as the far-smarter 30B MoE, so E4B's
|
||||
# niche is vision (mmproj). MTP draft TESTED AND REMOVED 2026-07-26: lost the final CPU-pinned
|
||||
# same-run A/Bs on both gemmas (E2B 4.0 vs 4.9 at 47% acc; E4B 2.3 vs 2.4 at 62% acc) — batched
|
||||
# verify is compute-bound on 2 cores; fl's +50% is a GPU result. The mtp-*.gguf drafts in
|
||||
# ~/software/models/ can be deleted.
|
||||
ctx-size = 4096
|
||||
threads = 2
|
||||
threads-batch = 4
|
||||
n-gpu-layers = 0
|
||||
jinja = on
|
||||
temp = 1.0
|
||||
top-p = 0.95
|
||||
top-k = 64
|
||||
sleep-idle-seconds = 30
|
||||
|
||||
[Jan-v3-4b]
|
||||
# Jan v3 4B (Qwen3-4B-based, tuned for agentic/reasoning use). Same-run A/B vs its base
|
||||
# (2026-07-26): 2.6 vs 2.9 t/s — NOT anomalous, just ~10% slower (Q4_K_XL reads a bit more per
|
||||
# token than the base's Q4_K_M). The earlier 2.1-vs-3.9 reading was cross-run variance.
|
||||
model = /home/anon/software/models/Jan-v3-4b-base-instruct-Q4_K_XL.gguf
|
||||
ctx-size = 4096 # init had 256 — unusably small for reasoning; raised
|
||||
threads = 2
|
||||
threads-batch = 4
|
||||
n-gpu-layers = 0
|
||||
flash-attn = on
|
||||
jinja = on
|
||||
temp = 0.7
|
||||
top-p = 0.95
|
||||
top-k = 20
|
||||
sleep-idle-seconds = 30
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Coding
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
[Qwen2.5-Coder-3B-Instruct]
|
||||
# Qwen2.5-Coder-3B Instruct, Q5_K_M (~2.3 GB, official Qwen GGUF). Small dedicated coder — benched 3.3 t/s (clean 2026-07-25).
|
||||
model = /home/anon/software/models/qwen2.5-coder-3b-instruct-q5_k_m.gguf
|
||||
ctx-size = 8192
|
||||
threads = 2
|
||||
threads-batch = 4
|
||||
n-gpu-layers = 0
|
||||
flash-attn = on
|
||||
jinja = on
|
||||
temp = 0.2 # low temp for code
|
||||
top-p = 0.9
|
||||
sleep-idle-seconds = 30
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# ⭐ Quality when you can wait — 30B MoE, only 3B ACTIVE → ~30B quality at ~4B speed
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
[Qwen3-30B-A3B-Instruct-2507-UD-IQ3_XXS]
|
||||
# Qwen3-30B-A3B Instruct 2507, UD-IQ3_XXS (~12 GB). Benched 1.9-2.7 t/s decode (cross-run
|
||||
# variance) / 4 t/s prefill / TTFT ~6.5 s — ~30B quality at E4B speed: THE quality pick when you
|
||||
# can wait. Compute-bound (~3.5 GB/s eff) → thread sweep 2026-07-26: t3 2.0 > t2 1.9 > t4 1.8
|
||||
# same-run → threads=3 promoted. (Earlier load failures were the Vulkan auto-offload OOM.)
|
||||
# ⚠ RAM: ~12 GB on 15.5 GB. With `--models-max 1` (the abbr default) it's always alone — no
|
||||
# co-residency swap risk — but 12 GB + desktop is still tight, so keep ctx small and close the
|
||||
# browser. If it swaps even alone, step the quant down to UD-IQ2_M (~10.5 GB).
|
||||
model = /home/anon/software/models/Qwen3-30B-A3B-Instruct-2507-UD-IQ3_XXS.gguf
|
||||
ctx-size = 4096
|
||||
threads = 3
|
||||
threads-batch = 4
|
||||
n-gpu-layers = 0
|
||||
flash-attn = on
|
||||
jinja = on
|
||||
temp = 0.7
|
||||
top-p = 0.8
|
||||
top-k = 20
|
||||
min-p = 0
|
||||
sleep-idle-seconds = 30
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# ⭐ Faster "speed AND brains" — small-active MoE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
[Granite-4.0-H-Tiny]
|
||||
# IBM Granite 4.0 H Tiny — 7B total / ~1B ACTIVE (hybrid Mamba-2 + MoE). Benched 4.3-4.8 t/s
|
||||
# decode / 11-14 t/s prefill: faster than every 4B AND ~7B-class quality, tiny RAM Δ (mmap).
|
||||
# COMPUTE-bound (~2.6 GB/s eff) → thread sweep 2026-07-26: t3 4.8 > t2 4.5 > t4 4.3 same-run →
|
||||
# threads=3 promoted (t4 loses to HT contention). Revert to 2 if the desktop drags.
|
||||
# QUANT: Q4_K_M (~4.2 GB) — goal here is speed, and Q4_K_M is the sweet spot (Q5_K_M/Q6_K fine
|
||||
# too, RAM allows, small speed cost since decode reads only the ~1B active).
|
||||
# Needs a current llama.cpp (hybrid arch) — the pacman build should be fine.
|
||||
model = /home/anon/software/models/granite-4.0-h-tiny-Q4_K_M.gguf
|
||||
ctx-size = 8192
|
||||
threads = 3
|
||||
threads-batch = 4
|
||||
n-gpu-layers = 0
|
||||
flash-attn = on # confirmed loading fine on the hybrid arch (2026-07-25)
|
||||
jinja = on
|
||||
temp = 0.7
|
||||
top-p = 0.95
|
||||
top-k = 20
|
||||
sleep-idle-seconds = 30
|
||||
|
||||
# (Bench-rig history: the MTP A/B rigs and the thread-sweep rigs that lived here were deleted
|
||||
# 2026-07-26 after their verdicts — MTP rejected (E4B comment), threads=3 promoted to
|
||||
# Granite/30B, threads-batch=4 everywhere. Raw legs in bench-history.md.)
|
||||
Reference in New Issue
Block a user