dots/fl/.config/whisper/README.md

Speech-to-text — setup & usage

whisper.cpp serving Whisper models over an OpenAI-compatible transcription endpoint, so Open WebUI's mic button works against local hardware. Added 2026-08-08.

• Box: same fl server as the LLM router — but this runs entirely on CPU (Ryzen 5600X, 6 cores).
• Files here: whisper@.service (one template unit) + one <instance>.env per model. Models live with the GGUFs in ~/software/models/.

instance	endpoint	model	speed / use
whisper@turbo	http://$SERVER:11345/v1	ggml-large-v3-turbo-q5_0 (574 MB)	default — best accuracy, multilingual, ~15 s per request. This is what OWUI points at.
whisper@small	http://$SERVER:11346/v1	ggml-small.en-q5_1 (181 MB)	English-only, ~5x lighter. Point a client here when latency beats accuracy.

Both run at once — together they're well under 1 GB of RAM and zero VRAM, so there is no reason to stop one to use the other. Adding a third is a new .env file plus systemctl enable --now whisper@<name>.

How it runs

Template unit whisper@.service (a dedicated service user), from /usr/bin/whisper-server (Arch extra/whisper-cpp). Each instance reads ~/.config/whisper/<instance>.env:

WHISPER_MODEL=/home/<user>/software/models/ggml-large-v3-turbo-q5_0.bin
WHISPER_PORT=11345
WHISPER_LANG=auto

which expands to:

whisper-server --model $WHISPER_MODEL \
               --host 0.0.0.0 --port $WHISPER_PORT \
               --inference-path /v1/audio/transcriptions \
               --convert --tmp-dir /run/whisper-<instance> \
               --no-gpu --threads 6 --language $WHISPER_LANG

It is a template rather than N copied units on purpose: the --tmp-dir bug below had to be found once and fixed once, and that only stays true while there is exactly one ExecStart in the repo.

flag	why
--no-gpu	Load-bearing. 0 VRAM — see below.
--convert	ffmpeg transcodes webm/opus/ogg → 16 kHz wav. Browsers record opus; whisper.cpp reads wav. Requires ffmpeg on the box.
--inference-path	--request-path defaults to empty, so this alone puts the route at /v1/audio/transcriptions — exactly where OWUI posts.
--tmp-dir	Required. Defaults to ".", i.e. the CWD — which is / here, and unwritable. Without it every request returns FFmpeg conversion failed., plain wav included. RuntimeDirectory=whisper-%i creates and owns /run/whisper-<instance>.
--language	auto for multilingual models; en for .en models, where detection is pointless.
--threads 6	all cores; Nice=5 yields to the router's CPU-offloaded experts.
ports 11345/11346	clear of 11343 (router), 11344 (the standalone embedding llama-server) and 11434 (Ollama). Router children take random ephemeral ports (32768+), so low ports are safe from them — but run ss -ltnp before claiming one.

Keyless on the LAN, same posture as the router. If this ever leaves the trusted LAN it needs a proxy with auth in front — whisper-server has no --api-key.

Deploying changes: /etc/systemd/system/whisper@.service is a symlink to the file in the repo, so a git pull edit is on disk immediately — but systemd caches unit files, so it still needs a reload. (Copying into /etc instead would work, but then every repo edit silently drifts.) On the server:

dotsync                                    # nudges you if this dir changed
sudo systemctl daemon-reload
sudo systemctl restart whisper@turbo whisper@small

Editing only an .env needs no daemon-reload — just restart that instance.

First install (stow does not install units):

yay -S whisper-cpp ffmpeg
curl -L -o ~/software/models/ggml-large-v3-turbo-q5_0.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-turbo-q5_0.bin
curl -L -o ~/software/models/ggml-small.en-q5_1.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.en-q5_1.bin

# Point systemd at the REAL file in the repo, NOT the stowed symlink: stow already made
# ~/.config/whisper/whisper@.service a symlink, and systemctl rejects those with
#   "Failed to link unit: Refusing to operate on linked unit file"
sudo systemctl link (readlink -f ~/.config/whisper/whisper@.service)   # fish
sudo systemctl enable --now whisper@turbo whisper@small

Why this is NOT a llama.cpp router preset

The router does expose POST /v1/audio/transcriptions (it rewrites the request into a chat completion with the prompt "Transcribe audio to text"), and gemma-4-E4B genuinely accepts audio input. Tempting — but two reasons it stays a separate service. Don't re-litigate this without re-checking both:

1. --models-max 1 eviction. Every transcription would LRU-evict the resident chat model, and the next chat message would reload it (~17.5 GB for GLM) with a cold prompt cache. Fine for a one-off file; unusable for talking to a model.
2. Format. llama.cpp decodes audio via miniaudio — wav/mp3/flac only. OWUI's mic records audio/webm; codecs=opus and its own transcode-to-mp3 step depends on ffprobe being usable inside the OWUI AppImage; when that fails it silently forwards the opus file. --convert here makes the question moot.

Third reason, softer: an LLM asked to transcribe can paraphrase or answer the audio instead of transcribing it, and the gemma-4-E4B preset samples at temp 1.0. Whisper is purpose-built and boring.

Still available as a fallback for wav/mp3/flac (no config change needed), if these services are down:

curl -F file=@clip.wav -F model=gemma-4-E4B-it-UD-Q8_K_XL \
     http://$SERVER:11343/v1/audio/transcriptions

Open WebUI wiring

Admin Settings → Audio → Speech-to-Text:

field	value
STT Engine	OpenAI
Base URL	http://$SERVER:11345/v1 — no trailing path, OWUI appends /audio/transcriptions. Swap 11345→11346 for the fast instance.
API Key	any non-empty string (none) — ignored by whisper-server, but OWUI needs it set
STT Model	whisper-1 — ignored by whisper-server; must not be blank
Request format	leave default (multipart)

OWUI sends multipart model + optional language + file; whisper-server reads the fields it knows and ignores the rest. Note it joins segments with newlines, so long dictations arrive in the composer with line breaks mid-sentence.

Verifying

systemctl status whisper@turbo whisper@small
rocm-smi --showmeminfo vram                     # must be UNCHANGED vs before the services started

curl -F file=@sample.wav http://$SERVER:11345/v1/audio/transcriptions        # → {"text": "…"}
ffmpeg -i sample.wav -c:a libopus sample.webm   # the case that actually matters:
curl -F file=@sample.webm http://$SERVER:11345/v1/audio/transcriptions       # proves --convert
curl -F file=@sample.webm -F model=whisper-1 -F language=en \
     http://$SERVER:11346/v1/audio/transcriptions                            # OWUI's exact shape

Then: with a chat model loaded, transcribe and confirm http://$SERVER:11343/v1/models still shows it ready/sleeping — not unloaded. That is the whole point of the separate service.

Measured 2026-08-08 (6 threads, CPU, 11 s JFK sample, --convert in the path):

model	per request	verdict
large-v3-turbo-q5_0 (574 MB)	~15.2 s	accurate; too slow to dictate into comfortably
small.en-q5_1 (181 MB)	~2.2 s	~7x faster. Word-for-word identical to turbo on the JFK sample — only punctuation differs (And so my fellow Americans vs turbo's And so, my fellow Americans,).

Runs were tight: small.en 2.16 / 2.16 / 2.20 s over three passes, turbo 15.1-15.2 s over three. The webm/opus path costs nothing extra (2.19 s with --convert + OWUI's field shape).

On this evidence turbo is hard to justify as the default — 13 seconds per utterance buys a comma on clean audio. Where it should still win is names, jargon, accents and non-English speech, which the JFK sample cannot show. If small.en holds up on real dictation, point OWUI at :11346 and demote turbo to the second instance (or just swap the ports in the two .env files).

The cost is flat for both models: whisper pads every request to a 30 s window, so a 3 s clip costs what a 30 s one does (turbo 15.1 s vs 15.2 s; small.en 2.08 s vs 2.20 s). It is a fixed toll per utterance, not a rate — which is why the model is the only thing that moves it. Flash-attention is already on by default in whisper-server, and the parakeet-* binaries in the package have no server integration, so model size is the only real dial.

A/B any candidate without touching a service:

curl -LO https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav   # known text
time whisper-cli -m ~/software/models/ggml-small.en-q5_1.bin -f jfk.wav -t 6 -nt
time whisper-cli -m ~/software/models/ggml-base.en-q5_1.bin  -f jfk.wav -t 6 -nt

Troubleshooting

• Every request returns {"error":"FFmpeg conversion failed."}, wav included → the temp dir is not writable by the service user. Check --tmp-dir /run/whisper-<instance> is in ExecStart and that RuntimeDirectory= created it (ls -ld /run/whisper-turbo). ffmpeg itself is fine — the startup log says ffmpeg is available. either way.
• Failed to load environment files → the instance has no .env (whisper@foo needs foo.env), or the repo was not re-stowed after adding it (install.sh links new files; dotsync does it for you).
• webm/opus uploads fail, wav works → ffmpeg missing or --convert dropped. ffmpeg -version on the box; the journal logs the conversion command on each non-wav request.
• VRAM moved when transcribing → --no-gpu lost, or the package was rebuilt with a GPU backend. Fix immediately; this is the freeze scenario.
• Empty / hallucinated text on near-silence → whisper's classic failure. Try VAD: download ggml-silero-v5.1.2.bin and add --vad --vad-model … to the template.
• Still too slow → next step down is ggml-base.en-q5_1.bin (56 MB, ~4x lighter again) at a real cost in accuracy on names and jargon. New .env, or edit small.env.
• OWUI: "Server Connection Error" → base URL has a trailing /audio/transcriptions (it appends that itself), or the port is not reachable from the client host.

Watch list

• whisper-cpp also ships parakeet-cli — NVIDIA Parakeet is much faster than Whisper on CPU for English. No server front-end upstream yet (examples/ has parakeet-cli but no server path); worth rechecking, it would be the real fix for the 15 s toll.
• TTS (the other half of voice chat) is unconfigured — OWUI can drive its own, or llama.cpp can serve OuteTTS via --model-vocoder. Separate job.