added qwable and orinth

2026-06-26 11:33:35 -04:00
parent 224afbb3a6
commit 705421470a
6 changed files with 384 additions and 7 deletions
--- a/pyinfra/framework/compose/ornith/README.md
+++ b/pyinfra/framework/compose/ornith/README.md
@@ -0,0 +1,152 @@
+# ornith
+
+Ornith-1.0-35B on Strix Halo via `kyuz0:rocm-7.2.2`. DeepReinforce's
+MIT-licensed **agentic-coding** model — a self-improving RL fine-tune of
+**Qwen3.5-35B-A3B** that co-trains its own task scaffolds with the policy.
+Strong on Terminal-Bench 2.1 / SWE-Bench Verified, emits OpenAI-style
+`tool_calls`, opens each answer with a `<think>` reasoning block.
+
+OpenAI-compatible endpoint at `http://framework:8083` once running.
+
+## MoE, not dense (read first — this is why it's worth a slot)
+
+Despite "35B" in the name, Ornith-1.0-35B is **MoE with only ~3B active
+params per token** (256 routed experts, 8 active + a shared expert, 40
+layers). On this bandwidth-bound box (256 GB/s) decode speed tracks
+*active* params, so it runs like the 30B-A3B workhorse (**~80-100 tok/s**),
+not like a dense 27/31B (~10-15 tok/s). That's the whole point: near
+frontier-class agentic-coding quality at interactive speed. Candidate to
+replace `qwen3-coder:30b` (Ollama) as the opencode daily driver — A/B
+before promoting.
+
+## Quant choice moves speed here
+
+For MoE, decode bandwidth ∝ *active bytes per token*, so quant tier
+changes t/s (~2x across the range), unlike a model where everything is
+read every token:
+
+| Quant | Size | When |
+|---|---|---|
+| **Q4_K_M** | **21.2 GB** | **default** — fastest, huge arena headroom |
+| Q6_K | 28.5 GB | bump here only if Q4 quality disappoints (~slower) |
+| Q8_0 | 36.9 GB | max quality, ~half the decode speed — rarely worth it for A3B |
+
+## Coexistence notes
+
+At ~21.2 GB (Q4_K_M) Ornith fits the merged arena easily:
+
+| Concurrent service | Coexists? |
+|---|---|
+| `llama` (Qwen3-Coder-30B, 8080) | ✅ yes |
+| `ollama` (11434) | ✅ yes |
+| `kimi-linear` (vLLM, 8000) | ✅ yes |
+| `qwable` (8082) | ✅ yes (~38 GB total) |
+| `qwen3-235b` (88.8 GB, 8081) | ❌ no — swap-model stops it |
+| `comfyui` (8188) | ❌ no — swap-model stops it |
+
+`restart: "no"`: you bring it up deliberately (via `swap-model ornith`),
+it won't auto-start after a reboot and surprise-collide with a big model.
+
+## Prereqs
+
+- Pyinfra deploy has run (creates `/srv/docker/ornith/` with right perms).
+- BIOS UMA at 0.5 GB + `ttm.pages_limit=33554432` kernel cmdline active.
+  Verify: `cat /proc/cmdline | grep ttm.pages_limit`.
+
+## Download weights (~21.2 GB, single file)
+
+```sh
+# /models/qwen exists via pyinfra; just create the model subdir.
+mkdir -p /models/qwen/Ornith-1.0-35B
+
+hf download deepreinforce-ai/Ornith-1.0-35B-GGUF \
+    'ornith-1.0-35b-Q4_K_M.gguf' \
+    --local-dir /models/qwen/Ornith-1.0-35B
+
+# File lands at:
+#   /models/qwen/Ornith-1.0-35B/ornith-1.0-35b-Q4_K_M.gguf  (~21.2 GB)
+```
+
+Single-file GGUF (not sharded) — point `--model` straight at it. Disk:
+needs ~22 GB free on `/models`. Verify the exact filename in the HF repo
+before downloading (casing matters).
+
+## Bring up
+
+Easy path — `swap-model` handles stop-conflicting-services + waits for
+`/health`:
+
+```sh
+ssh framework swap-model ornith     # ~1-2 min cold load (21.2 GB)
+ssh framework /srv/docker/ornith/smoke.sh    # /health + perf
+```
+
+Manual equivalent (first-ever bring-up, before the image is cached):
+
+```sh
+cd /srv/docker/ornith
+docker compose pull       # already-cached image if you ran llama first
+docker compose up -d
+docker compose logs -f    # wait for "server is listening on http://0.0.0.0:8083"
+
+./smoke.sh                # /health + tiny generation + perf
+```
+
+If `./smoke.sh` reports `predicted_per_second` in the ~80-100 tok/s band,
+it's healthy. <30 tok/s = investigate (likely arena < 100 GB — see
+qwen3-235b/README.md "Troubleshooting" for the arena checks).
+
+## Reasoning + tool calls
+
+Ornith emits a `<think>...</think>` block before the final answer and
+OpenAI-style `tool_calls`. `--jinja` (set in the compose file) uses the
+model's embedded Qwen3.5 chat template, which both rely on. If opencode
+shows raw `<think>` content in responses, the box's llama.cpp build is
+too old to split reasoning — bump the `kyuz0` image tag or add the
+build's reasoning-format flag. Recommended sampling (set server-side):
+temp 0.6 / top_p 0.95 / top_k 20.
+
+## Ramping context
+
+Defaults to 64K to match the other llama.cpp stacks (keeps opencode
+auto-compaction consistent across providers). Ornith's native context is
+262144, and the model is small relative to the arena, so there's room to
+push far higher:
+
+| Stage | `--ctx-size` | Margin in arena |
+|---|---|---|
+| **Current default** | **65536** | huge |
+| Stretch | 131072 | comfortable |
+| Native max | 262144 | watch KV cache size (q8_0 KV helps) |
+
+Edit `--ctx-size` in `docker-compose.yml`, `docker compose down && up -d`,
+re-run `./smoke.sh`.
+
+## Operations
+
+```sh
+docker compose logs -f                  # tail
+docker compose down                     # stop
+docker compose exec ornith bash         # shell in
+./smoke.sh                              # health + perf
+amdgpu_top                              # GPU view on host
+```
+
+## Pin manifest
+
+| Component | Pin |
+|---|---|
+| Image | `kyuz0/amd-strix-halo-toolboxes:rocm-7.2.2` (shared with `llama`/`qwable`) |
+| Weights | `deepreinforce-ai/Ornith-1.0-35B-GGUF` → `ornith-1.0-35b-Q4_K_M.gguf` (~21.2 GB) |
+| Base | Qwen3.5-35B-A3B (MoE: 256 experts, 8 active + shared, 40 layers) |
+| Default port | 8083 |
+| Default context | 65536 (native 262144) |
+| KV cache type | q8_0 (k and v) |
+| License | MIT (model); Qwen3.5 base license also applies |
+
+## Status
+
+Compose artifacts written; awaiting box-side weight pull + bring-up.
+Wired as a `swap-model ornith` target and as the `framework-ornith`
+opencode provider. A/B against `qwen3-coder:30b`; promote to opencode
+default if the agentic-coding quality proves out.
--- a/pyinfra/framework/compose/ornith/smoke.sh
+++ b/pyinfra/framework/compose/ornith/smoke.sh
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+# Smoke-test the running ornith llama-server (port 8083). Hits /health
+# for liveness, then a tiny OpenAI-compatible chat completion, then
+# measures eval_tps via /completion. MoE 35B-A3B (~3B active) → expect
+# ~80-100 tok/s, like the 30B-A3B workhorse (NOT a dense 27/31B).
+set -euo pipefail
+
+HOST="${ORNITH_HOST:-127.0.0.1:8083}"
+MODEL="${ORNITH_MODEL:-ornith}"
+
+echo "[smoke] GET /health on $HOST"
+curl -fsS "http://$HOST/health" | python3 -m json.tool
+
+echo
+echo "[smoke] POST /v1/chat/completions ($MODEL) — tiny generation"
+# Ornith opens with a <think> block; ask for a terse final answer.
+curl -fsS "http://$HOST/v1/chat/completions" \
+    -H 'Content-Type: application/json' \
+    -d "{
+        \"model\": \"$MODEL\",
+        \"messages\": [{\"role\": \"user\", \"content\": \"Reply with exactly: ok\"}],
+        \"max_tokens\": 256,
+        \"temperature\": 0.0
+    }" | python3 -m json.tool
+
+echo
+echo "[smoke] perf measure — eval_tps and prompt_tps (n_predict=128)"
+curl -fsS "http://$HOST/completion" \
+    -H 'Content-Type: application/json' \
+    -d '{
+        "prompt": "Write a Python function that computes the Fibonacci sequence iteratively. Include type hints and a brief docstring.",
+        "n_predict": 128,
+        "temperature": 0.0,
+        "stream": false
+    }' | python3 -c "
+import json, sys
+r = json.load(sys.stdin)
+t = r.get('timings', {})
+print(f'predicted_per_second:  {t.get(\"predicted_per_second\", \"?\"):.2f} tok/s')
+print(f'prompt_per_second:     {t.get(\"prompt_per_second\", \"?\"):.2f} tok/s')
+print(f'predicted_n:           {t.get(\"predicted_n\", \"?\")}')
+print(f'prompt_n:              {t.get(\"prompt_n\", \"?\")}')
+"
+
+echo
+echo "[smoke] passed — expected band ~80-100 tok/s decode (35B-A3B MoE Q4)."
+echo "          <30 tok/s = investigate arena (see qwen3-235b/README.md)."