# kimi-linear — resumption guide

Open this first when picking the work back up.

## What this project is

Kimi-Linear-48B-A3B-Instruct on the Strix Halo box via vLLM, ROCm/TheRock 7.x,
gfx1151. Sits beside Ollama+Qwen3-Coder. Goal: long-context (256K-1M)
local inference using the model architecturally best-suited to the box's
unified-memory shape.

Roadmap entry: `localgenai/Roadmap.md` → "Layer 0: Inference + tools"
(to be added after BIOS unblock).

Container artifacts: `pyinfra/framework/compose/kimi-linear.yml` +
`pyinfra/framework/compose/kimi-linear/` (Dockerfile, build.sh,
patch-tokenizer.sh, smoke.sh, README.md). Deploy push:
`cd pyinfra/framework && ./run.sh`.

## Where we are (2026-05-10)

**P0 — DONE, constrained.** First-ever locally-served Kimi-Linear
generation on a Strix Halo iGPU. Smoke test passes:
`/v1/models` returns `kimi-linear`; tiny generation returns "ok".

Current runtime cap: `--max-model-len 4096`, `--max-num-seqs 1`,
`--num-gpu-blocks-override 32`. The long-context point of the model is
locked behind a **VRAM ceiling** — see "What blocks progress" below.

**P1-P4 — pending.**

## The gauntlet of fixes that got us here (don't re-derive)

All of these are baked into the repo. Reproducing P0 from a clean box is
push the repo + run the steps in `compose/kimi-linear/README.md`.

1. **Image entrypoint missing.** kyuz0 toolboxes drop into a shell;
   compose's `command:` gets exec'd as the program. Fix: explicit
   `entrypoint: ["vllm", "serve"]` in the compose file, model path as
   positional first arg.

2. **Tokenizer ImportError.** `tokenization_kimi.py` imports
   `bytes_to_unicode` from a transformers internal that's been removed.
   Fix: `patch-tokenizer.sh` inlines the function. Idempotent.

3. **Missing AITER gfx1151 GEMM configs.** kyuz0 image is built for
   gfx1151 but doesn't ship the AITER autotuning JSONs for every op
   Kimi's MLA layers hit (validated against Qwen/MiniMax, not
   MLA-heavy models). Fix: derived `Dockerfile` copies gfx1100 (RDNA3)
   configs into gfx1151-named slots — kernels compile + run, tile
   sizes not optimal but functional.

4. **MLA AITER FP8 BMM tries to materialize a 30 GB intermediate.**
   On top of resident weights, that's ~58 GB needed and we have 31 GB.
   Fix: `VLLM_ROCM_USE_AITER_MLA=0` — bypasses AITER for just the MLA
   path, keeps it for everything else.

5. **`HSA_XNACK=1` is a trap with vLLM.** It enables HIP demand-paging
   into GTT (115 GB ceiling per kernel cmdline `amdgpu.gttsize=117760`)
   so vLLM computes "Available KV cache memory: 73.6 GiB", but
   PyTorch's actual allocator stays capped at the GPU pool (~31 GB).
   vLLM then OOMs trying to allocate the budget it computed. Fix: turn
   XNACK *off*, live within the 31 GB pool until BIOS UMA gives a
   single bigger pool.

6. **`--swap-space` was removed in modern vLLM.** Don't pass it.

7. **`--num-gpu-blocks-override 32`** as belt-and-braces against
   vLLM's KV pool auto-discovery picking a too-big number even without
   XNACK.

## What blocks long context

PyTorch's discoverable VRAM equals the **BIOS UMA Frame Buffer Size**,
not `amdgpu.gttsize`. The kernel cmdline is necessary but insufficient.

- 128 GB physical → 64 GB UMA → ~62 GB visible to Linux.
- `rocminfo` reports two ~31 GB GPU pools (Pool 1 coarse / Pool 2 fine).
- PyTorch's allocator only uses one pool (~31 GB) → OOMs at ~30 GB
  Kimi weights with little KV headroom.
- User has previously found Framework's BIOS caps UMA at 64 GB.

**Research outcome (2026-05-10):** the right unblock isn't a higher BIOS
UMA cap — it's the inverse. Set UMA *small* and merge the two pools.

| Layer | Setting |
| --- | --- |
| BIOS | Update to **3.05 stable** (Apr 2026); set UMA Frame Buffer = **0.5 GB** or **8 GB** (counter-intuitive but documented — frees pages for GTT) |
| Kernel | **≥ 6.16.9.** Earlier kernels cap ROCm visibility at 15.5 GB. pyinfra's `linux-generic-hwe-24.04` may be on 6.8/6.11 — verify with `uname -r` and upgrade if needed. |
| Cmdline | `amd_iommu=off amdgpu.gttsize=131072 ttm.pages_limit=33554432` (`ttm.pages_limit` in 4 KiB pages = 128 GiB) |
| Env | `HSA_XNACK=1` **+** `HSA_FORCE_FINE_GRAIN_PCIE=1` (the piece our earlier XNACK attempt was missing) |
| Env | `PYTORCH_HIP_ALLOC_CONF="backend:native,expandable_segments:True,garbage_collection_threshold:0.9"` (HIP variant, not CUDA) |
| PyTorch | TheRock gfx1151 wheels — kyuz0:stable image already uses these |

Confirmed working in kyuz0/amd-strix-halo-vllm-toolboxes (DeepWiki:
"Kernel Parameters and Unified Memory") and the Framework community
"Linux + ROCm: January 2026 Stable Configurations Update" thread.
Single-process allocation budget after this: ~110 GB.

**The hard ceiling**, if all of the above doesn't yield enough: 96 GB
direct UMA. AMD AGESA / StrixHaloPI limit, not Framework's. No path
past 96 GB on signed firmware as of May 2026; 192 GB Strix Halo refresh
is rumored but unreleased.

**Fallback if the merge approach doesn't work for vLLM specifically**:
llama.cpp ROCm + bartowski Q4_K_M GGUF. Different memory model, splits
layers across CPU/GPU. Lower throughput, more flexible.

## When you come back

1. Read research output for BIOS UMA limits. If it landed in the chat,
   it's the most recent note in this project's session. If not yet,
   re-dispatch (prompt is in conversation history — Framework Desktop
   May 2026 UMA Frame Buffer cap research).
2. Decide path:
   - **BIOS bump available** → flash, reboot, drop the constraints in
     `compose/kimi-linear.yml` (max-model-len, num-gpu-blocks-override),
     re-`./smoke.sh`, ramp context.
   - **BIOS capped at 64 GB** → pivot to llama.cpp ROCm path or accept
     4K context as the long-term reality.
3. Then advance through P1 → P2 → P3 per the roadmap.

## Files of record

- `pyinfra/framework/compose/kimi-linear.yml` — service def, all the
  flag/env tradeoffs documented inline.
- `pyinfra/framework/compose/kimi-linear/` — Dockerfile + scripts.
- `pyinfra/framework/deploy.py` — wired into the service loop +
  asset-copy block.
- `Roadmap.md` — strategy.
- `StrixHaloMemory.md` — the UMA-vs-GTT discussion that needs a
  follow-up paragraph from this work (PyTorch caps at UMA, XNACK is a
  trap).

## Decisions worth not relitigating

- **vLLM 0.19.x via kyuz0:stable** chosen over source-build with
  `v0.11.2` pin (build.sh exists as fallback). The recipe pin advice
  was based on an earlier Moonshot doc; current upstream works.
- **4-bit compressed-tensors** (cyankiwi) chosen over 8-bit. With the
  31 GB ceiling, 8-bit wouldn't even fit weights resident.
- **VLLM_ROCM_USE_AITER_MLA=0**, NOT `VLLM_MLA_DISABLE=1`. Granular
  disable preserves AITER for non-MLA paths. The full disable is the
  next escalation if needed.
- **No upstream filings.** Findings stay in this repo per project
  policy (memory: `feedback_private_findings`).