93 lines
3.3 KiB
Markdown
93 lines
3.3 KiB
Markdown
|
# llama
|
|||
|
|
|
|||
|
|
llama.cpp server with **native gfx1151** kernels via kyuz0's ROCm 7.2.2
|
|||
|
|
toolbox. Sits beside Ollama (11434) and vLLM (8000) on port 8080. Same
|
|||
|
|
Qwen3-Coder model as Ollama, faster path.
|
|||
|
|
|
|||
|
|
## Why this exists
|
|||
|
|
|
|||
|
|
Ollama's bundled ROCm doesn't ship native gfx1151 — we coerce gfx1100
|
|||
|
|
kernels via `HSA_OVERRIDE_GFX_VERSION=11.0.0`. kyuz0's image is built
|
|||
|
|
against gfx1151 with rocWMMA acceleration. Expected eval_tps delta on
|
|||
|
|
Qwen3-Coder-30B-A3B-Q4: **~30-50 % faster**, with ~2× prefill speedup.
|
|||
|
|
The compose stub used to be vulkan-radv with a placeholder model path;
|
|||
|
|
this rewrite makes it the second working coding endpoint.
|
|||
|
|
|
|||
|
|
## Bring up (LL-P0 verification)
|
|||
|
|
|
|||
|
|
```sh
|
|||
|
|
# 1. Pull the Unsloth UD-Q4_K_XL Qwen3-Coder GGUF on the box.
|
|||
|
|
# Verify the actual filename in the HF repo first — Unsloth's naming
|
|||
|
|
# sometimes splits into shards. As of 2026-05 the single-file
|
|||
|
|
# UD-Q4_K_XL is ~17-19 GB.
|
|||
|
|
hf download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
|
|||
|
|
'Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf' \
|
|||
|
|
--local-dir /models/qwen
|
|||
|
|
|
|||
|
|
# 2. Stand up the container.
|
|||
|
|
cd /srv/docker/llama
|
|||
|
|
docker compose pull # ~6-10 GB image
|
|||
|
|
docker compose up -d
|
|||
|
|
docker compose logs -f # wait for "main: server is listening on http://0.0.0.0:8080"
|
|||
|
|
|
|||
|
|
# 3. Smoke + perf measure.
|
|||
|
|
./smoke.sh
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
If `predicted_per_second` is meaningfully higher than what Ollama
|
|||
|
|
reports for the same prompt, the migration is justified. If it's the
|
|||
|
|
same or worse, leave Ollama as the default and treat llama.cpp as a
|
|||
|
|
secondary option.
|
|||
|
|
|
|||
|
|
## Comparison test (vs Ollama)
|
|||
|
|
|
|||
|
|
Run the same prompt against both for a clean A/B:
|
|||
|
|
|
|||
|
|
```sh
|
|||
|
|
# Ollama
|
|||
|
|
curl -s http://framework:11434/api/generate \
|
|||
|
|
-d '{"model":"qwen3-coder:30b","prompt":"Write a Python fibonacci function with type hints.","stream":false}' \
|
|||
|
|
| jq '{eval_tps:(.eval_count/(.eval_duration/1e9)), prompt_tps:(.prompt_eval_count/(.prompt_eval_duration/1e9))}'
|
|||
|
|
|
|||
|
|
# llama.cpp (this stack)
|
|||
|
|
curl -s http://framework:8080/completion \
|
|||
|
|
-d '{"prompt":"Write a Python fibonacci function with type hints.","n_predict":200,"temperature":0}' \
|
|||
|
|
| jq '.timings | {predicted_per_second, prompt_per_second}'
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Coexistence with Ollama
|
|||
|
|
|
|||
|
|
Both can run simultaneously — different ports, different model files on
|
|||
|
|
disk (Ollama's content-addressed store at `/models/ollama/` vs the raw
|
|||
|
|
GGUF at `/models/qwen/`). They will compete for GPU memory if both have
|
|||
|
|
their models hot. With `OLLAMA_KEEP_ALIVE=24h` Ollama keeps Qwen3
|
|||
|
|
resident; if you want to A/B without contention, `docker exec ollama
|
|||
|
|
ollama stop qwen3-coder:30b` while testing llama.cpp.
|
|||
|
|
|
|||
|
|
If LL-P0 confirms the perf win, LL-P1 wires this as a third opencode
|
|||
|
|
provider (`framework-llama/qwen3-coder` alongside `framework/qwen3-coder:30b`
|
|||
|
|
and `framework-vllm/kimi-linear`).
|
|||
|
|
|
|||
|
|
## Pin manifest
|
|||
|
|
|
|||
|
|
| Component | Pin |
|
|||
|
|
|---|---|
|
|||
|
|
| Image | `kyuz0/amd-strix-halo-toolboxes:rocm-7.2.2` |
|
|||
|
|
| Weights | `unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF` (UD-Q4_K_XL variant) |
|
|||
|
|
| Default port | 8080 |
|
|||
|
|
| Context | 65536 (matches Ollama config) |
|
|||
|
|
|
|||
|
|
## Operations
|
|||
|
|
|
|||
|
|
```sh
|
|||
|
|
docker compose logs -f # tail
|
|||
|
|
docker compose restart llama # reload
|
|||
|
|
docker compose down # stop
|
|||
|
|
docker compose exec llama bash # shell in
|
|||
|
|
./smoke.sh # health + perf check
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Status
|
|||
|
|
|
|||
|
|
LL-P0 in progress. LL-P1 (opencode provider wire-up) pending verification.
|