[Fix] cleanup
This commit is contained in:
coja
@@ -0,0 +1,193 @@
|
||||
# Speech-to-text — setup & usage
|
||||
|
||||
whisper.cpp serving Whisper models over an **OpenAI-compatible** transcription endpoint, so Open WebUI's
|
||||
mic button works against local hardware. Added **2026-08-08**.
|
||||
|
||||
- **Box**: same fl server as the LLM router — but this runs **entirely on CPU** (Ryzen 5600X, 6 cores).
|
||||
- **Files here**: `whisper@.service` (one template unit) + one `<instance>.env` per model.
|
||||
Models live with the GGUFs in `~/software/models/`.
|
||||
|
||||
| instance | endpoint | model | speed / use |
|
||||
|---|---|---|---|
|
||||
| **`whisper@turbo`** | `http://$SERVER:11345/v1` | `ggml-large-v3-turbo-q5_0` (574 MB) | **default** — best accuracy, multilingual, ~15 s per request. This is what OWUI points at. |
|
||||
| `whisper@small` | `http://$SERVER:11346/v1` | `ggml-small.en-q5_1` (181 MB) | English-only, ~5x lighter. Point a client here when latency beats accuracy. |
|
||||
|
||||
Both run at once — together they're well under 1 GB of RAM and **zero VRAM**, so there is no reason to
|
||||
stop one to use the other. Adding a third is a new `.env` file plus `systemctl enable --now whisper@<name>`.
|
||||
|
||||
## How it runs
|
||||
|
||||
Template unit **`whisper@.service`** (a dedicated service user), from `/usr/bin/whisper-server` (Arch `extra/whisper-cpp`).
|
||||
Each instance reads `~/.config/whisper/<instance>.env`:
|
||||
|
||||
```
|
||||
WHISPER_MODEL=/home/<user>/software/models/ggml-large-v3-turbo-q5_0.bin
|
||||
WHISPER_PORT=11345
|
||||
WHISPER_LANG=auto
|
||||
```
|
||||
|
||||
which expands to:
|
||||
|
||||
```
|
||||
whisper-server --model $WHISPER_MODEL \
|
||||
--host 0.0.0.0 --port $WHISPER_PORT \
|
||||
--inference-path /v1/audio/transcriptions \
|
||||
--convert --tmp-dir /run/whisper-<instance> \
|
||||
--no-gpu --threads 6 --language $WHISPER_LANG
|
||||
```
|
||||
|
||||
It is a template rather than N copied units on purpose: the `--tmp-dir` bug below had to be found once
|
||||
and fixed once, and that only stays true while there is exactly one `ExecStart` in the repo.
|
||||
|
||||
| flag | why |
|
||||
|---|---|
|
||||
| `--no-gpu` | **Load-bearing.** 0 VRAM — see below. |
|
||||
| `--convert` | ffmpeg transcodes webm/opus/ogg → 16 kHz wav. Browsers record opus; whisper.cpp reads wav. **Requires ffmpeg on the box.** |
|
||||
| `--inference-path` | `--request-path` defaults to empty, so this alone puts the route at `/v1/audio/transcriptions` — exactly where OWUI posts. |
|
||||
| `--tmp-dir` | **Required.** Defaults to `"."`, i.e. the CWD — which is `/` here, and unwritable. Without it every request returns `FFmpeg conversion failed.`, plain wav included. `RuntimeDirectory=whisper-%i` creates and owns `/run/whisper-<instance>`. |
|
||||
| `--language` | `auto` for multilingual models; `en` for `.en` models, where detection is pointless. |
|
||||
| `--threads 6` | all cores; `Nice=5` yields to the router's CPU-offloaded experts. |
|
||||
| ports `11345`/`11346` | clear of `11343` (router), **`11344` (the standalone embedding llama-server)** and `11434` (Ollama). Router *children* take random ephemeral ports (32768+), so low ports are safe from them — but run `ss -ltnp` before claiming one. |
|
||||
|
||||
Keyless on the LAN, same posture as the router. If this ever leaves the trusted LAN it needs a proxy
|
||||
with auth in front — whisper-server has no `--api-key`.
|
||||
|
||||
**Deploying changes**: `/etc/systemd/system/whisper@.service` is a symlink to the file in the repo, so a
|
||||
`git pull` edit is on disk immediately — but systemd caches unit files, so it still needs a reload.
|
||||
(Copying into `/etc` instead would work, but then every repo edit silently drifts.) On the server:
|
||||
|
||||
```
|
||||
dotsync # nudges you if this dir changed
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart whisper@turbo whisper@small
|
||||
```
|
||||
|
||||
Editing only an `.env` needs no `daemon-reload` — just restart that instance.
|
||||
|
||||
First install (stow does not install units):
|
||||
|
||||
```
|
||||
yay -S whisper-cpp ffmpeg
|
||||
curl -L -o ~/software/models/ggml-large-v3-turbo-q5_0.bin \
|
||||
https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-turbo-q5_0.bin
|
||||
curl -L -o ~/software/models/ggml-small.en-q5_1.bin \
|
||||
https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.en-q5_1.bin
|
||||
|
||||
# Point systemd at the REAL file in the repo, NOT the stowed symlink: stow already made
|
||||
# ~/.config/whisper/whisper@.service a symlink, and systemctl rejects those with
|
||||
# "Failed to link unit: Refusing to operate on linked unit file"
|
||||
sudo systemctl link (readlink -f ~/.config/whisper/whisper@.service) # fish
|
||||
sudo systemctl enable --now whisper@turbo whisper@small
|
||||
```
|
||||
|
||||
## Why this is NOT a llama.cpp router preset
|
||||
|
||||
The router *does* expose `POST /v1/audio/transcriptions` (it rewrites the request into a chat completion
|
||||
with the prompt "Transcribe audio to text"), and `gemma-4-E4B` genuinely accepts audio input. Tempting —
|
||||
but two reasons it stays a separate service. Don't re-litigate this without re-checking both:
|
||||
|
||||
1. **`--models-max 1` eviction.** Every transcription would LRU-evict the resident chat model, and the
|
||||
next chat message would reload it (~17.5 GB for GLM) with a cold prompt cache. Fine for a one-off
|
||||
file; unusable for talking to a model.
|
||||
2. **Format.** llama.cpp decodes audio via miniaudio — **wav/mp3/flac only**. OWUI's mic records
|
||||
`audio/webm; codecs=opus` and its own transcode-to-mp3 step depends on ffprobe being usable inside
|
||||
the OWUI AppImage; when that fails it silently forwards the opus file. `--convert` here makes the
|
||||
question moot.
|
||||
|
||||
Third reason, softer: an LLM asked to transcribe can paraphrase or answer the audio instead of
|
||||
transcribing it, and the `gemma-4-E4B` preset samples at temp 1.0. Whisper is purpose-built and boring.
|
||||
|
||||
**Still available as a fallback** for wav/mp3/flac (no config change needed), if these services are down:
|
||||
|
||||
```
|
||||
curl -F file=@clip.wav -F model=gemma-4-E4B-it-UD-Q8_K_XL \
|
||||
http://$SERVER:11343/v1/audio/transcriptions
|
||||
```
|
||||
|
||||
## Open WebUI wiring
|
||||
|
||||
Admin Settings → **Audio** → Speech-to-Text:
|
||||
|
||||
| field | value |
|
||||
|---|---|
|
||||
| STT Engine | `OpenAI` |
|
||||
| Base URL | `http://$SERVER:11345/v1` — no trailing path, OWUI appends `/audio/transcriptions`. Swap `11345`→`11346` for the fast instance. |
|
||||
| API Key | any non-empty string (`none`) — ignored by whisper-server, but OWUI needs it set |
|
||||
| STT Model | `whisper-1` — ignored by whisper-server; must not be blank |
|
||||
| Request format | leave default (`multipart`) |
|
||||
|
||||
OWUI sends multipart `model` + optional `language` + `file`; whisper-server reads the fields it knows
|
||||
and ignores the rest. Note it joins segments with newlines, so long dictations arrive in the composer
|
||||
with line breaks mid-sentence.
|
||||
|
||||
## Verifying
|
||||
|
||||
```
|
||||
systemctl status whisper@turbo whisper@small
|
||||
rocm-smi --showmeminfo vram # must be UNCHANGED vs before the services started
|
||||
|
||||
curl -F file=@sample.wav http://$SERVER:11345/v1/audio/transcriptions # → {"text": "…"}
|
||||
ffmpeg -i sample.wav -c:a libopus sample.webm # the case that actually matters:
|
||||
curl -F file=@sample.webm http://$SERVER:11345/v1/audio/transcriptions # proves --convert
|
||||
curl -F file=@sample.webm -F model=whisper-1 -F language=en \
|
||||
http://$SERVER:11346/v1/audio/transcriptions # OWUI's exact shape
|
||||
```
|
||||
|
||||
Then: with a chat model loaded, transcribe and confirm `http://$SERVER:11343/v1/models` still
|
||||
shows it `ready`/`sleeping` — **not** `unloaded`. That is the whole point of the separate service.
|
||||
|
||||
**Measured 2026-08-08** (6 threads, CPU, 11 s JFK sample, `--convert` in the path):
|
||||
|
||||
| model | per request | verdict |
|
||||
|---|---|---|
|
||||
| `large-v3-turbo-q5_0` (574 MB) | **~15.2 s** | accurate; too slow to dictate into comfortably |
|
||||
| `small.en-q5_1` (181 MB) | **~2.2 s** | **~7x faster.** Word-for-word identical to turbo on the JFK sample — only punctuation differs (`And so my fellow Americans` vs turbo's `And so, my fellow Americans,`). |
|
||||
|
||||
Runs were tight: small.en 2.16 / 2.16 / 2.20 s over three passes, turbo 15.1-15.2 s over three.
|
||||
The webm/opus path costs nothing extra (2.19 s with `--convert` + OWUI's field shape).
|
||||
|
||||
**On this evidence turbo is hard to justify as the default** — 13 seconds per utterance buys a comma on
|
||||
clean audio. Where it should still win is names, jargon, accents and non-English speech, which the JFK
|
||||
sample cannot show. If small.en holds up on real dictation, point OWUI at `:11346` and demote turbo to
|
||||
the second instance (or just swap the ports in the two `.env` files).
|
||||
|
||||
The cost is **flat** for both models: whisper pads every request to a 30 s window, so a 3 s clip costs
|
||||
what a 30 s one does (turbo 15.1 s vs 15.2 s; small.en 2.08 s vs 2.20 s). It is a fixed toll per
|
||||
utterance, not a rate — which is why the model is the only thing that moves it.
|
||||
Flash-attention is already on by default in whisper-server, and the `parakeet-*` binaries in the package
|
||||
have no server integration, so **model size is the only real dial**.
|
||||
|
||||
A/B any candidate without touching a service:
|
||||
|
||||
```
|
||||
curl -LO https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav # known text
|
||||
time whisper-cli -m ~/software/models/ggml-small.en-q5_1.bin -f jfk.wav -t 6 -nt
|
||||
time whisper-cli -m ~/software/models/ggml-base.en-q5_1.bin -f jfk.wav -t 6 -nt
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Every request returns `{"error":"FFmpeg conversion failed."}`, wav included** → the temp dir is not
|
||||
writable by the service user. Check `--tmp-dir /run/whisper-<instance>` is in ExecStart and that
|
||||
`RuntimeDirectory=` created it (`ls -ld /run/whisper-turbo`). ffmpeg itself is fine — the startup log
|
||||
says `ffmpeg is available.` either way.
|
||||
- **`Failed to load environment files`** → the instance has no `.env` (`whisper@foo` needs `foo.env`),
|
||||
or the repo was not re-stowed after adding it (`install.sh` links new files; `dotsync` does it for you).
|
||||
- **webm/opus uploads fail, wav works** → ffmpeg missing or `--convert` dropped. `ffmpeg -version` on
|
||||
the box; the journal logs the conversion command on each non-wav request.
|
||||
- **VRAM moved when transcribing** → `--no-gpu` lost, or the package was rebuilt with a GPU backend.
|
||||
Fix immediately; this is the freeze scenario.
|
||||
- **Empty / hallucinated text on near-silence** → whisper's classic failure. Try VAD:
|
||||
download `ggml-silero-v5.1.2.bin` and add `--vad --vad-model …` to the template.
|
||||
- **Still too slow** → next step down is `ggml-base.en-q5_1.bin` (56 MB, ~4x lighter again) at a real
|
||||
cost in accuracy on names and jargon. New `.env`, or edit `small.env`.
|
||||
- **OWUI: "Server Connection Error"** → base URL has a trailing `/audio/transcriptions` (it appends that
|
||||
itself), or the port is not reachable from the client host.
|
||||
|
||||
## Watch list
|
||||
|
||||
- `whisper-cpp` also ships **`parakeet-cli`** — NVIDIA Parakeet is much faster than Whisper on CPU for
|
||||
English. No server front-end upstream yet (`examples/` has `parakeet-cli` but no server path); worth
|
||||
rechecking, it would be the real fix for the 15 s toll.
|
||||
- **TTS** (the other half of voice chat) is unconfigured — OWUI can drive its own, or llama.cpp can
|
||||
serve OuteTTS via `--model-vocoder`. Separate job.
|
||||
Reference in New Issue
Block a user