# llama_qwen3.6_A3B

llama.cpp CUDA runner for **Qwen3.6-35B-A3B** (UD-Q4_K_M) at **128k context** on 4n4rch02.

Port **18008**, OpenAI-compatible API at `http://192.168.3.189:18008/v1`.

## Host

- 4n4rch02 (192.168.3.189), Arch Linux, driver 610.43.03
- AMD Ryzen 7 3700X — **8 physical cores** / 16 SMT threads
- 62 GiB RAM, 15 GiB swap
- CUDA0: RTX 3060 12 GB — **shared with the KDE/Wayland desktop** (~1.1 GiB at idle)
- CUDA1: RTX 3080 10 GB — dedicated

## Modell

`/mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`

20.6 GiB, unsloth Dynamic Q4_K_M (imatrix). The compose mounts the **repo dir**, not
the snapshot dir — the snapshot entry is a relative symlink into `../../blobs/`, so
both have to be inside the mount.

### Architektur (`qwen35moe`) — warum 128k hier billig ist

This is a **hybrid SSM + attention MoE**, not a dense-attention model:

| | |
|---|---|
| Layers | 40 |
| `full_attention_interval` | 4 → **only 10 layers use full attention** |
| Remaining 30 layers | gated-delta SSM, constant-size recurrent state |
| Attention heads | 16 Q / 2 KV, `key_length` = `value_length` = 256 |
| Experts | 256 total, 8 active, expert FFN 512, shared expert 512 |
| Native context | 262144 |

Only the 10 full-attention layers grow a KV cache, so 128k of KV costs just

```
10 layers x 2 kv-heads x 256 dim x 2 (K+V) x 1.0625 B/elem (q8_0) x 131072 tok = 1.33 GiB
```

Measured: the whole runtime footprint with *all* experts on CPU is 4.65 GB at 128k ctx.
**The context is not the constraint here — the 20.6 GiB of weights against 22 GiB of
VRAM is.** That is what the tuning below is about.

The model is multimodal-capable (the chat template emits `<|vision_start|>` /
`<|image_pad|>` tokens), but **no mmproj file is present**, so this deployment is
text-only. It is a reasoning model with `<think>` tags and XML-style tool calls, so
`--jinja` is mandatory.

## Tuning

### 1. `--n-cpu-moe` collides with `-sm layer` — `-ts` is not optional

`--n-cpu-moe N` moves the expert tensors of the **first N layers** to host RAM.
`-sm layer` assigns the **first** layers to CUDA0. Those are the same layers, so
CUDA0 gets the lightweight ones and CUDA1 ends up holding every heavy expert layer.

With the default split, **every** value of `--n-cpu-moe` from 12 down to 4 died the
same way — CUDA1 out of memory while allocating the KV cache:

```
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 680.00 MiB on device 1: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA1 buffer of size 713031680
llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
```

`-ts 24,16` moves the layer boundary back toward CUDA0 and fixes it. A heavy
(expert-bearing) layer is ~469 MiB; a layer whose experts are on CPU is ~59 MiB.

### 2. `--threads 8`, not 16

The CPU-side expert matmuls are memory-bandwidth-bound, so the 8 SMT siblings only
add contention. Measured at `ncmoe=8`, 8k depth:

| threads | prefill | generation |
|---|---|---|
| 8 | 556 t/s | **57.3 t/s** |
| 16 | 556 t/s | 39.8 t/s |

**+44% generation.** Prefill is unaffected because it runs on the GPU.

### 3. Expert offload vs. VRAM headroom

All at 128k ctx, q8_0 KV, `--threads 8`, `-ts 24,16`:

| `--n-cpu-moe` | gen @512 | gen @8k | CUDA0 free | CUDA1 free |
|---|---|---|---|---|
| 8 | 59.9 t/s | 57.3 t/s | 388 MiB | 531 MiB |
| **10 (default)** | **55.6 t/s** | **53.3 t/s** | **1312 MiB** | **529 MiB** |
| 12 | 51.3 t/s | 49.1 t/s | 2236 MiB | 527 MiB |
| 40 (all experts on CPU) | 15.3 t/s | 16.1 t/s | — | — |

`ncmoe=10` is the shipped default: it gives up 7% throughput for **~1.3 GiB of spare
VRAM on CUDA0**, which the desktop shares. At `ncmoe=8` only 388 MiB is left there,
and a browser opening a few video tabs is enough to OOM a `restart: unless-stopped`
service into a crash loop. If the desktop is idle, `ncmoe=8` is the faster setting.

CUDA1 is the binding constraint in every case — 529 MiB free is not enough for another
469 MiB expert layer, which is why `ncmoe` cannot go below 8 at this context size.

### 4. Throughput over depth (shipped config)

`ncmoe=10`, `-ts 24,16`, `--threads 8`, 128k ctx allocated, q8_0 KV:

| prompt depth | prefill | generation |
|---|---|---|
| 512 | 420 t/s | 56.3 t/s |
| 32k (27169 tok) | 486 t/s | 47.8 t/s |
| ~97k (99109 tok) | 425 t/s | 34.1 t/s |

Generation falls off ~40% between empty and ~97k, which is the attention cost on the
10 full-attention layers. Prefill stays flat around 420–490 t/s, so filling the whole
128k window takes roughly 4–5 minutes.

### 5. Speculative decoding makes it slower — don't enable it

Counter-intuitive but consistent, `--n-predict 400`, thinking disabled:

| workload | none | ngram-mod | ngram-simple | ngram-cache |
|---|---|---|---|---|
| rewrite (output ≈ copy of prompt) | **55.6** | 40.6 | 39.7 | 47.0 |
| extend (structured code) | **55.3** | 32.7 | 27.2 | 38.1 |
| prose (novel text, control) | **56.1** | 54.7 | 52.9 | 48.4 |

Every mode loses, and it loses *most* on exactly the structured workloads
speculation is supposed to win. The reason is the CPU-side experts: verifying a
K-token draft costs about K times the CPU expert work, because each token routes to
its own subset of 8 out of 256 experts, so there is no weight reuse to amortize.
On a fully GPU-resident model batch verification is nearly free; with
`--n-cpu-moe` it is not. Reconsider only if the model ever fits entirely in VRAM.

### 6. Rejected

- `-ub 256`: frees only ~80 MiB per GPU (not the 469 MiB an extra layer needs) and
  costs **38% prefill** (556 → 347 t/s). Keep the default `-ub 512`.
- Lower `-ts` toward CUDA1 (e.g. `23,17`): would leave CUDA1 at ~60 MiB free.

## Vergleich: A3B (MoE) vs. Qwen3.6-27B (dense)

Same box, same llama.cpp image, same `-ctk/-ctv q8_0 -fa on --threads 8`, same
benchmark, measured at identical prompt depths:

| Prompt-Tiefe | A3B gen | 27B gen | A3B prefill | 27B prefill |
|---|---|---|---|---|
| 512 | **56,3 t/s** | 24,1 t/s | 420 t/s | **539 t/s** |
| 8k | **53,3 t/s** | 23,2 t/s | 497 t/s | **817 t/s** |
| 32k | **47,8 t/s** | 20,5 t/s | 486 t/s | **767 t/s** |

**Generierung: A3B ist durchgehend ~2,3x schneller.** Nur 3B der 35B Parameter sind
pro Token aktiv, das dichte 27B muss alle 27B lesen — auch wenn es komplett im VRAM
liegt und das A3B ein Drittel seiner Experten im RAM hat.

**Prefill: das 27B ist ~1,6x schneller**, weil es vollständig GPU-resident ist,
während beim A3B jeder Prefill-Batch durch die CPU-Experten muss.

**Kontext: das 27B schafft die 128k hier gar nicht.** Es hat 65 Layer mit 4 KV-Heads
gegen 40 Layer mit 2 — bei `full_attention_interval=4` also 16 statt 10
Full-Attention-Layer und damit **4,25 GiB KV bei 128k statt 1,33 GiB**. Zusammen mit
~15 GiB Gewichten reicht das nicht: schon das kleinere Q4_K_S scheitert beim
Compute-Buffer:

```
allocating 1145.13 MiB on device 0: cudaMalloc failed: out of memory
graph_reserve: failed to allocate compute buffers
```

Gemessen wurde es deshalb bei 64k ctx.

### Caveats zu diesem Vergleich

- Gemessen wurde **Q4_K_S (16,1 GB)**, nicht Q4_K_M (17 GB) — die einzige lokal
  vorhandene, mit upstream llama.cpp ladbare 27B-Datei dieser Klasse. Q4_K_M ist ~5%
  größer und wäre entsprechend etwas langsamer, der Abstand also eher noch größer.
- Der ollama-Blob von `qwen3.6:27b-q4_K_M` lässt sich mit upstream llama.cpp **nicht**
  laden: `key qwen35.rope.dimension_sections has wrong array length; expected 4, got 3`.
  Ollama fährt einen eigenen Fork; ein Vergleich exakt dieser Datei ist im selben
  Runner nicht möglich.
- Ältere 27B-Zahlen unter `PrismQuant/benchmarks/` (34,3 t/s) stammen von einer
  **RX 7900 XTX, gedrosselt** — nicht mit dieser Maschine vergleichbar.

### A note on the CUDA0 headroom

Free VRAM on CUDA0 was observed drifting between ~1100 and ~1300 MiB across restarts
purely from desktop compositor churn — a ~200 MiB swing with nothing else changing.
That is why the default keeps >1 GiB spare there rather than the 388 MiB that
`ncmoe=8` leaves.

## `--predict` ist ein Default, kein Deckel

Despite the name, `--predict` does **not** cap what a client can ask for. Measured
against `--predict 8192`:

| Request | Result |
|---|---|
| `max_tokens: 9000` | **8893** tokens, `finish_reason: stop` — the flag is ignored |
| no `max_tokens` | **8192** tokens, `finish_reason: length` — the flag applies |

So it only binds clients that send no `max_tokens` at all. That is still worth having
here: without it such a client generates until the 128k window is full, which at
~40 t/s is **roughly 50 minutes**, and with `-np 1` the single slot is blocked for all
of it. Set to `32768` — a runaway is bounded to ~12 minutes, while no realistic long
answer gets truncated. Many OpenAI clients do not send `max_tokens`, and at 8192 they
were silently cut off with `finish_reason: length`.

Note the sibling `gemma4-26b-llama-runner` README describes `--predict` as a "harte
Cap fuer Gesamt-Generierung" — by this measurement that is wrong there too.

## `--reasoning-budget` dagegen ist ein echter Deckel

Same flag family, opposite semantics — measured with `bench/reasoning_budget_probe.py`,
which counts the thinking block exactly via the server's `/tokenize` endpoint:

| Request | Reasoning tokens | Answer |
|---|---|---|
| server default | **4095** | 1227 tok, complete |
| `reasoning_budget: 16384` in the request body | **4095** | 531 tok, complete |

A client **cannot** raise it. Where `--predict` was a default anyone could override,
this one is a hard server-side ceiling, so changing it means editing the compose file
and restarting.

It does bite: on a moderately hard number-theory problem the thinking was guillotined
mid-derivation (`...Since $x^2 \equiv`) at exactly the budget. But the model recovered
— it produced the correct answer (empty solution set, verified by brute force) and
rebuilt the parity argument cleanly in the visible response. So 4096 truncates
reasoning without necessarily damaging the result.

Raising it does not weaken the runaway protection, since total generation is already
bounded by `--predict` / the client's `max_tokens`. For heavier math or agent
workloads, 16384 is the obvious next step; there is no measured evidence here that
4096 is actively breaking anything.

## Deploy

```bash
cd ~/projects/llama_qwen3.6_A3B
docker compose up -d
```

Loads in ~15 s (`--no-mmap`, weights read from NVMe).

## Benchmarks

```bash
python3 bench/bench.py --port 18008 --depths 512,8192,32768 --n-predict 128
./bench/sweep.sh 10:24,16:8 8:24,16:8      # <ncmoe>:<tensor-split>:<threads>
python3 bench/spec_bench.py --no-think     # copy/code/prose generation workloads
```

`bench.py` issues a warmup request first — without it the first measurement reads
~30% low because of one-off CUDA graph setup.

## Image

`ghcr.io/ggml-org/llama.cpp:server-cuda`, tested at build **b10121**
(commit `555881ebc8b0`, 2026-07-25). Updated from b10068 during this deployment;
b10121 is the first build here that supports the `qwen35moe` architecture end to end.

## VRAM-Exklusivitaet

Uses ~20 GB of the 22 GB total. It **cannot** run alongside `llama-gemma4` (18006) or
qwen-prism (18004). Ollama on 11434 loads models on demand and will fight for VRAM —
stop it or let its `keep_alive` expire before starting this runner.