llama_qwen3.6_A3B/README.md

# llama_qwen3.6_A3B

llama.cpp CUDA runner for **Qwen3.6-35B-A3B** (UD-Q4_K_M) at **128k context** on 4n4rch02.

Port **18008**, OpenAI-compatible API at `http://192.168.3.189:18008/v1`.

## Host

- 4n4rch02 (192.168.3.189), Arch Linux, driver 610.43.03
- AMD Ryzen 7 3700X — **8 physical cores** / 16 SMT threads
- 62 GiB RAM, 15 GiB swap
- CUDA0: RTX 3060 12 GB — **shared with the KDE/Wayland desktop** (~1.1 GiB at idle)
- CUDA1: RTX 3080 10 GB — dedicated

## Modell

`/mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`

20.6 GiB, unsloth Dynamic Q4_K_M (imatrix). The compose mounts the **repo dir**, not
the snapshot dir — the snapshot entry is a relative symlink into `../../blobs/`, so
both have to be inside the mount.

### Architektur (`qwen35moe`) — warum 128k hier billig ist

This is a **hybrid SSM + attention MoE**, not a dense-attention model:

| | |
|---|---|
| Layers | 40 |
| `full_attention_interval` | 4 → **only 10 layers use full attention** |
| Remaining 30 layers | gated-delta SSM, constant-size recurrent state |
| Attention heads | 16 Q / 2 KV, `key_length` = `value_length` = 256 |
| Experts | 256 total, 8 active, expert FFN 512, shared expert 512 |
| Native context | 262144 |

Only the 10 full-attention layers grow a KV cache, so 128k of KV costs just

```
10 layers x 2 kv-heads x 256 dim x 2 (K+V) x 1.0625 B/elem (q8_0) x 131072 tok = 1.33 GiB
```

Measured: the whole runtime footprint with *all* experts on CPU is 4.65 GB at 128k ctx.
**The context is not the constraint here — the 20.6 GiB of weights against 22 GiB of
VRAM is.** That is what the tuning below is about.

The model is multimodal-capable (the chat template emits `<|vision_start|>` /
`<|image_pad|>` tokens), but **no mmproj file is present**, so this deployment is
text-only. It is a reasoning model with `<think>` tags and XML-style tool calls, so
`--jinja` is mandatory.

## Tuning

### 1. `--n-cpu-moe` collides with `-sm layer` — `-ts` is not optional

`--n-cpu-moe N` moves the expert tensors of the **first N layers** to host RAM.
`-sm layer` assigns the **first** layers to CUDA0. Those are the same layers, so
CUDA0 gets the lightweight ones and CUDA1 ends up holding every heavy expert layer.

With the default split, **every** value of `--n-cpu-moe` from 12 down to 4 died the
same way — CUDA1 out of memory while allocating the KV cache:

```
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 680.00 MiB on device 1: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA1 buffer of size 713031680
llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
```

`-ts 24,16` moves the layer boundary back toward CUDA0 and fixes it. A heavy
(expert-bearing) layer is ~469 MiB; a layer whose experts are on CPU is ~59 MiB.

### 2. `--threads 8`, not 16

The CPU-side expert matmuls are memory-bandwidth-bound, so the 8 SMT siblings only
add contention. Measured at `ncmoe=8`, 8k depth:

| threads | prefill | generation |
|---|---|---|
| 8 | 556 t/s | **57.3 t/s** |
| 16 | 556 t/s | 39.8 t/s |

**+44% generation.** Prefill is unaffected because it runs on the GPU.

### 3. Expert offload vs. VRAM headroom

All at 128k ctx, q8_0 KV, `--threads 8`, `-ts 24,16`:

| `--n-cpu-moe` | gen @512 | gen @8k | CUDA0 free | CUDA1 free |
|---|---|---|---|---|
| 8 | 59.9 t/s | 57.3 t/s | 388 MiB | 531 MiB |
| **10 (default)** | **55.6 t/s** | **53.3 t/s** | **1312 MiB** | **529 MiB** |
| 12 | 51.3 t/s | 49.1 t/s | 2236 MiB | 527 MiB |
| 40 (all experts on CPU) | 15.3 t/s | 16.1 t/s | — | — |

`ncmoe=10` is the shipped default: it gives up 7% throughput for **~1.3 GiB of spare
VRAM on CUDA0**, which the desktop shares. At `ncmoe=8` only 388 MiB is left there,
and a browser opening a few video tabs is enough to OOM a `restart: unless-stopped`
service into a crash loop. If the desktop is idle, `ncmoe=8` is the faster setting.

CUDA1 is the binding constraint in every case — 529 MiB free is not enough for another
469 MiB expert layer, which is why `ncmoe` cannot go below 8 at this context size.

### 4. Throughput over depth (shipped config)

`ncmoe=10`, `-ts 24,16`, `--threads 8`, 128k ctx allocated, q8_0 KV:

| prompt depth | prefill | generation |
|---|---|---|
| 512 | 420 t/s | 56.3 t/s |
| 32k (27169 tok) | 486 t/s | 47.8 t/s |
| ~97k (99109 tok) | 425 t/s | 34.1 t/s |

Generation falls off ~40% between empty and ~97k, which is the attention cost on the
10 full-attention layers. Prefill stays flat around 420–490 t/s, so filling the whole
128k window takes roughly 4–5 minutes.

### 5. Rejected

- `-ub 256`: frees only ~80 MiB per GPU (not the 469 MiB an extra layer needs) and
  costs **38% prefill** (556 → 347 t/s). Keep the default `-ub 512`.
- Lower `-ts` toward CUDA1 (e.g. `23,17`): would leave CUDA1 at ~60 MiB free.

## Deploy

```bash
cd ~/projects/llama_qwen3.6_A3B
docker compose up -d
```

Loads in ~15 s (`--no-mmap`, weights read from NVMe).

## Benchmarks

```bash
python3 bench/bench.py --port 18008 --depths 512,8192,32768 --n-predict 128
./bench/sweep.sh 10:24,16:8 8:24,16:8      # <ncmoe>:<tensor-split>:<threads>
```

`bench.py` issues a warmup request first — without it the first measurement reads
~30% low because of one-off CUDA graph setup.

## Image

`ghcr.io/ggml-org/llama.cpp:server-cuda`, tested at build **b10121**
(commit `555881ebc8b0`, 2026-07-25). Updated from b10068 during this deployment;
b10121 is the first build here that supports the `qwen35moe` architecture end to end.

## VRAM-Exklusivitaet

Uses ~20 GB of the 22 GB total. It **cannot** run alongside `llama-gemma4` (18006) or
qwen-prism (18004). Ollama on 11434 loads models on demand and will fight for VRAM —
stop it or let its `keep_alive` expire before starting this runner.