# llama

llama.cpp server with **native gfx1151** kernels via kyuz0's ROCm 7.2.2
toolbox. Sits beside Ollama (11434) and vLLM (8000) on port 8080. Same
Qwen3-Coder model as Ollama, faster path.

## Why this exists

Ollama's bundled ROCm doesn't ship native gfx1151 — we coerce gfx1100
kernels via `HSA_OVERRIDE_GFX_VERSION=11.0.0`. kyuz0's image is built
against gfx1151 with rocWMMA acceleration. Expected eval_tps delta on
Qwen3-Coder-30B-A3B-Q4: **~30-50 % faster**, with ~2× prefill speedup.
The compose stub used to be vulkan-radv with a placeholder model path;
this rewrite makes it the second working coding endpoint.

## Bring up (LL-P0 verification)

```sh
# 1. Pull the Unsloth UD-Q4_K_XL Qwen3-Coder GGUF on the box.
#    Verify the actual filename in the HF repo first — Unsloth's naming
#    sometimes splits into shards. As of 2026-05 the single-file
#    UD-Q4_K_XL is ~17-19 GB.
hf download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
    'Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf' \
    --local-dir /models/qwen

# 2. Stand up the container.
cd /srv/docker/llama
docker compose pull       # ~6-10 GB image
docker compose up -d
docker compose logs -f    # wait for "main: server is listening on http://0.0.0.0:8080"

# 3. Smoke + perf measure.
./smoke.sh
```

If `predicted_per_second` is meaningfully higher than what Ollama
reports for the same prompt, the migration is justified. If it's the
same or worse, leave Ollama as the default and treat llama.cpp as a
secondary option.

## Comparison test (vs Ollama)

Run the same prompt against both for a clean A/B:

```sh
# Ollama
curl -s http://framework:11434/api/generate \
  -d '{"model":"qwen3-coder:30b","prompt":"Write a Python fibonacci function with type hints.","stream":false}' \
  | jq '{eval_tps:(.eval_count/(.eval_duration/1e9)), prompt_tps:(.prompt_eval_count/(.prompt_eval_duration/1e9))}'

# llama.cpp (this stack)
curl -s http://framework:8080/completion \
  -d '{"prompt":"Write a Python fibonacci function with type hints.","n_predict":200,"temperature":0}' \
  | jq '.timings | {predicted_per_second, prompt_per_second}'
```

## Coexistence with Ollama

Both can run simultaneously — different ports, different model files on
disk (Ollama's content-addressed store at `/models/ollama/` vs the raw
GGUF at `/models/qwen/`). They will compete for GPU memory if both have
their models hot. With `OLLAMA_KEEP_ALIVE=24h` Ollama keeps Qwen3
resident; if you want to A/B without contention, `docker exec ollama
ollama stop qwen3-coder:30b` while testing llama.cpp.

If LL-P0 confirms the perf win, LL-P1 wires this as a third opencode
provider (`framework-llama/qwen3-coder` alongside `framework/qwen3-coder:30b`
and `framework-vllm/kimi-linear`).

## Pin manifest

| Component | Pin |
|---|---|
| Image | `kyuz0/amd-strix-halo-toolboxes:rocm-7.2.2` |
| Weights | `unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF` (UD-Q4_K_XL variant) |
| Default port | 8080 |
| Context | 65536 (matches Ollama config) |

## Operations

```sh
docker compose logs -f                              # tail
docker compose restart llama                        # reload
docker compose down                                 # stop
docker compose exec llama bash                      # shell in
./smoke.sh                                          # health + perf check
```

## Status

LL-P0 in progress. LL-P1 (opencode provider wire-up) pending verification.