coja/dots
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[Docs] sanitize locators in the published infra guides

Browse Source
This commit is contained in:
coja
2026-08-13 00:04:37 +02:00
parent a099091b6c
commit 885a62520d
2 changed files with 16 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
fl/.config/llamacpp/README.md
+4 -3
View File
@@ -6,14 +6,15 @@ Last full clean sweep: **2026-08-08 03:22** — all 11 presets loaded and ran gr
- **Box**: AMD RX 7600 XT 16 GB (ROCm, ~288 GB/s) + Ryzen 5600X (6 cores) + 48 GB RAM. - **Box**: AMD RX 7600 XT 16 GB (ROCm, ~288 GB/s) + Ryzen 5600X (6 cores) + 48 GB RAM.
⚠ The GPU **also drives the display** — see [VRAM safety](#vram-safety-the-golden-rules). ⚠ The GPU **also drives the display** — see [VRAM safety](#vram-safety-the-golden-rules).
- **Endpoints**: LAN `http://192.168.0.204:11343/v1` · remote `https://llm.duskadiy.com/api/v1` - **Endpoints**: LAN `http://$SERVER:11343/v1` · remote `https://<own-domain>/api/v1`
(own reverse proxy to the same server, no key). (own reverse proxy to the same server; keyless — put auth in front before exposing it).
`$SERVER` = the box's LAN address.
- **Files here**: `config.ini` (model presets — the section names ARE the API model ids), - **Files here**: `config.ini` (model presets — the section names ARE the API model ids),
`bench.py` (safety-first benchmark), `bench-results.md` / `bench-history.md` (ledgers, generated). `bench.py` (safety-first benchmark), `bench-results.md` / `bench-history.md` (ledgers, generated).
## How it runs ## How it runs
systemd unit **`llama.service`** (on the server, user `anon`) runs a script wrapping: systemd unit **`llama.service`** (on the server, under a dedicated service user) runs a script wrapping:
``` ```
llama-server --port 11343 --host 0.0.0.0 --models-max 1 --models-preset ~/.config/llamacpp/config.ini llama-server --port 11343 --host 0.0.0.0 --models-max 1 --models-preset ~/.config/llamacpp/config.ini
fl/.config/whisper/README.md
+12 -12
View File
@@ -9,19 +9,19 @@ mic button works against local hardware. Added **2026-08-08**.
| instance | endpoint | model | speed / use | | instance | endpoint | model | speed / use |
|---|---|---|---| |---|---|---|---|
| **`whisper@turbo`** | `http://192.168.0.204:11345/v1` | `ggml-large-v3-turbo-q5_0` (574 MB) | **default** — best accuracy, multilingual, ~15 s per request. This is what OWUI points at. | | **`whisper@turbo`** | `http://$SERVER:11345/v1` | `ggml-large-v3-turbo-q5_0` (574 MB) | **default** — best accuracy, multilingual, ~15 s per request. This is what OWUI points at. |
| `whisper@small` | `http://192.168.0.204:11346/v1` | `ggml-small.en-q5_1` (181 MB) | English-only, ~5x lighter. Point a client here when latency beats accuracy. | | `whisper@small` | `http://$SERVER:11346/v1` | `ggml-small.en-q5_1` (181 MB) | English-only, ~5x lighter. Point a client here when latency beats accuracy. |
Both run at once — together they're well under 1 GB of RAM and **zero VRAM**, so there is no reason to Both run at once — together they're well under 1 GB of RAM and **zero VRAM**, so there is no reason to
stop one to use the other. Adding a third is a new `.env` file plus `systemctl enable --now whisper@<name>`. stop one to use the other. Adding a third is a new `.env` file plus `systemctl enable --now whisper@<name>`.
## How it runs ## How it runs
Template unit **`whisper@.service`** (user `anon`), from `/usr/bin/whisper-server` (Arch `extra/whisper-cpp`). Template unit **`whisper@.service`** (a dedicated service user), from `/usr/bin/whisper-server` (Arch `extra/whisper-cpp`).
Each instance reads `~/.config/whisper/<instance>.env`: Each instance reads `~/.config/whisper/<instance>.env`:
``` ```
WHISPER_MODEL=/home/anon/software/models/ggml-large-v3-turbo-q5_0.bin WHISPER_MODEL=/home/<user>/software/models/ggml-large-v3-turbo-q5_0.bin
WHISPER_PORT=11345 WHISPER_PORT=11345
WHISPER_LANG=auto WHISPER_LANG=auto
``` ```
@@ -101,7 +101,7 @@ transcribing it, and the `gemma-4-E4B` preset samples at temp 1.0. Whisper is pu
``` ```
curl -F file=@clip.wav -F model=gemma-4-E4B-it-UD-Q8_K_XL \ curl -F file=@clip.wav -F model=gemma-4-E4B-it-UD-Q8_K_XL \
http://192.168.0.204:11343/v1/audio/transcriptions http://$SERVER:11343/v1/audio/transcriptions
``` ```
## Open WebUI wiring ## Open WebUI wiring
@@ -111,7 +111,7 @@ Admin Settings → **Audio** → Speech-to-Text:
| field | value | | field | value |
|---|---| |---|---|
| STT Engine | `OpenAI` | | STT Engine | `OpenAI` |
| Base URL | `http://192.168.0.204:11345/v1` — no trailing path, OWUI appends `/audio/transcriptions`. Swap `11345`→`11346` for the fast instance. | | Base URL | `http://$SERVER:11345/v1` — no trailing path, OWUI appends `/audio/transcriptions`. Swap `11345`→`11346` for the fast instance. |
| API Key | any non-empty string (`none`) — ignored by whisper-server, but OWUI needs it set | | API Key | any non-empty string (`none`) — ignored by whisper-server, but OWUI needs it set |
| STT Model | `whisper-1` — ignored by whisper-server; must not be blank | | STT Model | `whisper-1` — ignored by whisper-server; must not be blank |
| Request format | leave default (`multipart`) | | Request format | leave default (`multipart`) |
@@ -126,14 +126,14 @@ with line breaks mid-sentence.
systemctl status whisper@turbo whisper@small systemctl status whisper@turbo whisper@small
rocm-smi --showmeminfo vram # must be UNCHANGED vs before the services started rocm-smi --showmeminfo vram # must be UNCHANGED vs before the services started
curl -F file=@sample.wav http://192.168.0.204:11345/v1/audio/transcriptions # → {"text": "…"} curl -F file=@sample.wav http://$SERVER:11345/v1/audio/transcriptions # → {"text": "…"}
ffmpeg -i sample.wav -c:a libopus sample.webm # the case that actually matters: ffmpeg -i sample.wav -c:a libopus sample.webm # the case that actually matters:
curl -F file=@sample.webm http://192.168.0.204:11345/v1/audio/transcriptions # proves --convert curl -F file=@sample.webm http://$SERVER:11345/v1/audio/transcriptions # proves --convert
curl -F file=@sample.webm -F model=whisper-1 -F language=en \ curl -F file=@sample.webm -F model=whisper-1 -F language=en \
http://192.168.0.204:11346/v1/audio/transcriptions # OWUI's exact shape http://$SERVER:11346/v1/audio/transcriptions # OWUI's exact shape
``` ```
Then: with a chat model loaded, transcribe and confirm `http://192.168.0.204:11343/v1/models` still Then: with a chat model loaded, transcribe and confirm `http://$SERVER:11343/v1/models` still
shows it `ready`/`sleeping` — **not** `unloaded`. That is the whole point of the separate service. shows it `ready`/`sleeping` — **not** `unloaded`. That is the whole point of the separate service.
**Measured 2026-08-08** (6 threads, CPU, 11 s JFK sample, `--convert` in the path): **Measured 2026-08-08** (6 threads, CPU, 11 s JFK sample, `--convert` in the path):
@@ -168,7 +168,7 @@ time whisper-cli -m ~/software/models/ggml-base.en-q5_1.bin -f jfk.wav -t 6 -nt
## Troubleshooting ## Troubleshooting
- **Every request returns `{"error":"FFmpeg conversion failed."}`, wav included** → the temp dir is not - **Every request returns `{"error":"FFmpeg conversion failed."}`, wav included** → the temp dir is not
writable by `anon`. Check `--tmp-dir /run/whisper-<instance>` is in ExecStart and that writable by the service user. Check `--tmp-dir /run/whisper-<instance>` is in ExecStart and that
`RuntimeDirectory=` created it (`ls -ld /run/whisper-turbo`). ffmpeg itself is fine — the startup log `RuntimeDirectory=` created it (`ls -ld /run/whisper-turbo`). ffmpeg itself is fine — the startup log
says `ffmpeg is available.` either way. says `ffmpeg is available.` either way.
- **`Failed to load environment files`** → the instance has no `.env` (`whisper@foo` needs `foo.env`), - **`Failed to load environment files`** → the instance has no `.env` (`whisper@foo` needs `foo.env`),
@@ -182,7 +182,7 @@ time whisper-cli -m ~/software/models/ggml-base.en-q5_1.bin -f jfk.wav -t 6 -nt
- **Still too slow** → next step down is `ggml-base.en-q5_1.bin` (56 MB, ~4x lighter again) at a real - **Still too slow** → next step down is `ggml-base.en-q5_1.bin` (56 MB, ~4x lighter again) at a real
cost in accuracy on names and jargon. New `.env`, or edit `small.env`. cost in accuracy on names and jargon. New `.env`, or edit `small.env`.
- **OWUI: "Server Connection Error"** → base URL has a trailing `/audio/transcriptions` (it appends that - **OWUI: "Server Connection Error"** → base URL has a trailing `/audio/transcriptions` (it appends that
itself), or the port is not reachable from `192.168.0.72`. itself), or the port is not reachable from the client host.
## Watch list ## Watch list