DATAandClaude Opus 5
e41dc2ba6f
Measured: with --predict 8192 a request sending max_tokens=9000 returned 8893 tokens (finish_reason stop), while a request sending no max_tokens was cut at exactly 8192 (finish_reason length). The flag is a default for clients that omit max_tokens, not a ceiling anyone can hit. That default still matters on a -np 1 server, where an unbounded client would fill the 128k window and block the only slot for ~50 minutes. 32768 bounds that to ~12 minutes without truncating realistic long answers, which 8192 was doing silently to any client that omits max_tokens. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
243 lines
10 KiB
Markdown
243 lines
10 KiB
Markdown
# llama_qwen3.6_A3B
|
||
|
||
llama.cpp CUDA runner for **Qwen3.6-35B-A3B** (UD-Q4_K_M) at **128k context** on 4n4rch02.
|
||
|
||
Port **18008**, OpenAI-compatible API at `http://192.168.3.189:18008/v1`.
|
||
|
||
## Host
|
||
|
||
- 4n4rch02 (192.168.3.189), Arch Linux, driver 610.43.03
|
||
- AMD Ryzen 7 3700X — **8 physical cores** / 16 SMT threads
|
||
- 62 GiB RAM, 15 GiB swap
|
||
- CUDA0: RTX 3060 12 GB — **shared with the KDE/Wayland desktop** (~1.1 GiB at idle)
|
||
- CUDA1: RTX 3080 10 GB — dedicated
|
||
|
||
## Modell
|
||
|
||
`/mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`
|
||
|
||
20.6 GiB, unsloth Dynamic Q4_K_M (imatrix). The compose mounts the **repo dir**, not
|
||
the snapshot dir — the snapshot entry is a relative symlink into `../../blobs/`, so
|
||
both have to be inside the mount.
|
||
|
||
### Architektur (`qwen35moe`) — warum 128k hier billig ist
|
||
|
||
This is a **hybrid SSM + attention MoE**, not a dense-attention model:
|
||
|
||
| | |
|
||
|---|---|
|
||
| Layers | 40 |
|
||
| `full_attention_interval` | 4 → **only 10 layers use full attention** |
|
||
| Remaining 30 layers | gated-delta SSM, constant-size recurrent state |
|
||
| Attention heads | 16 Q / 2 KV, `key_length` = `value_length` = 256 |
|
||
| Experts | 256 total, 8 active, expert FFN 512, shared expert 512 |
|
||
| Native context | 262144 |
|
||
|
||
Only the 10 full-attention layers grow a KV cache, so 128k of KV costs just
|
||
|
||
```
|
||
10 layers x 2 kv-heads x 256 dim x 2 (K+V) x 1.0625 B/elem (q8_0) x 131072 tok = 1.33 GiB
|
||
```
|
||
|
||
Measured: the whole runtime footprint with *all* experts on CPU is 4.65 GB at 128k ctx.
|
||
**The context is not the constraint here — the 20.6 GiB of weights against 22 GiB of
|
||
VRAM is.** That is what the tuning below is about.
|
||
|
||
The model is multimodal-capable (the chat template emits `<|vision_start|>` /
|
||
`<|image_pad|>` tokens), but **no mmproj file is present**, so this deployment is
|
||
text-only. It is a reasoning model with `<think>` tags and XML-style tool calls, so
|
||
`--jinja` is mandatory.
|
||
|
||
## Tuning
|
||
|
||
### 1. `--n-cpu-moe` collides with `-sm layer` — `-ts` is not optional
|
||
|
||
`--n-cpu-moe N` moves the expert tensors of the **first N layers** to host RAM.
|
||
`-sm layer` assigns the **first** layers to CUDA0. Those are the same layers, so
|
||
CUDA0 gets the lightweight ones and CUDA1 ends up holding every heavy expert layer.
|
||
|
||
With the default split, **every** value of `--n-cpu-moe` from 12 down to 4 died the
|
||
same way — CUDA1 out of memory while allocating the KV cache:
|
||
|
||
```
|
||
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 680.00 MiB on device 1: cudaMalloc failed: out of memory
|
||
alloc_tensor_range: failed to allocate CUDA1 buffer of size 713031680
|
||
llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
|
||
```
|
||
|
||
`-ts 24,16` moves the layer boundary back toward CUDA0 and fixes it. A heavy
|
||
(expert-bearing) layer is ~469 MiB; a layer whose experts are on CPU is ~59 MiB.
|
||
|
||
### 2. `--threads 8`, not 16
|
||
|
||
The CPU-side expert matmuls are memory-bandwidth-bound, so the 8 SMT siblings only
|
||
add contention. Measured at `ncmoe=8`, 8k depth:
|
||
|
||
| threads | prefill | generation |
|
||
|---|---|---|
|
||
| 8 | 556 t/s | **57.3 t/s** |
|
||
| 16 | 556 t/s | 39.8 t/s |
|
||
|
||
**+44% generation.** Prefill is unaffected because it runs on the GPU.
|
||
|
||
### 3. Expert offload vs. VRAM headroom
|
||
|
||
All at 128k ctx, q8_0 KV, `--threads 8`, `-ts 24,16`:
|
||
|
||
| `--n-cpu-moe` | gen @512 | gen @8k | CUDA0 free | CUDA1 free |
|
||
|---|---|---|---|---|
|
||
| 8 | 59.9 t/s | 57.3 t/s | 388 MiB | 531 MiB |
|
||
| **10 (default)** | **55.6 t/s** | **53.3 t/s** | **1312 MiB** | **529 MiB** |
|
||
| 12 | 51.3 t/s | 49.1 t/s | 2236 MiB | 527 MiB |
|
||
| 40 (all experts on CPU) | 15.3 t/s | 16.1 t/s | — | — |
|
||
|
||
`ncmoe=10` is the shipped default: it gives up 7% throughput for **~1.3 GiB of spare
|
||
VRAM on CUDA0**, which the desktop shares. At `ncmoe=8` only 388 MiB is left there,
|
||
and a browser opening a few video tabs is enough to OOM a `restart: unless-stopped`
|
||
service into a crash loop. If the desktop is idle, `ncmoe=8` is the faster setting.
|
||
|
||
CUDA1 is the binding constraint in every case — 529 MiB free is not enough for another
|
||
469 MiB expert layer, which is why `ncmoe` cannot go below 8 at this context size.
|
||
|
||
### 4. Throughput over depth (shipped config)
|
||
|
||
`ncmoe=10`, `-ts 24,16`, `--threads 8`, 128k ctx allocated, q8_0 KV:
|
||
|
||
| prompt depth | prefill | generation |
|
||
|---|---|---|
|
||
| 512 | 420 t/s | 56.3 t/s |
|
||
| 32k (27169 tok) | 486 t/s | 47.8 t/s |
|
||
| ~97k (99109 tok) | 425 t/s | 34.1 t/s |
|
||
|
||
Generation falls off ~40% between empty and ~97k, which is the attention cost on the
|
||
10 full-attention layers. Prefill stays flat around 420–490 t/s, so filling the whole
|
||
128k window takes roughly 4–5 minutes.
|
||
|
||
### 5. Speculative decoding makes it slower — don't enable it
|
||
|
||
Counter-intuitive but consistent, `--n-predict 400`, thinking disabled:
|
||
|
||
| workload | none | ngram-mod | ngram-simple | ngram-cache |
|
||
|---|---|---|---|---|
|
||
| rewrite (output ≈ copy of prompt) | **55.6** | 40.6 | 39.7 | 47.0 |
|
||
| extend (structured code) | **55.3** | 32.7 | 27.2 | 38.1 |
|
||
| prose (novel text, control) | **56.1** | 54.7 | 52.9 | 48.4 |
|
||
|
||
Every mode loses, and it loses *most* on exactly the structured workloads
|
||
speculation is supposed to win. The reason is the CPU-side experts: verifying a
|
||
K-token draft costs about K times the CPU expert work, because each token routes to
|
||
its own subset of 8 out of 256 experts, so there is no weight reuse to amortize.
|
||
On a fully GPU-resident model batch verification is nearly free; with
|
||
`--n-cpu-moe` it is not. Reconsider only if the model ever fits entirely in VRAM.
|
||
|
||
### 6. Rejected
|
||
|
||
- `-ub 256`: frees only ~80 MiB per GPU (not the 469 MiB an extra layer needs) and
|
||
costs **38% prefill** (556 → 347 t/s). Keep the default `-ub 512`.
|
||
- Lower `-ts` toward CUDA1 (e.g. `23,17`): would leave CUDA1 at ~60 MiB free.
|
||
|
||
## Vergleich: A3B (MoE) vs. Qwen3.6-27B (dense)
|
||
|
||
Same box, same llama.cpp image, same `-ctk/-ctv q8_0 -fa on --threads 8`, same
|
||
benchmark, measured at identical prompt depths:
|
||
|
||
| Prompt-Tiefe | A3B gen | 27B gen | A3B prefill | 27B prefill |
|
||
|---|---|---|---|---|
|
||
| 512 | **56,3 t/s** | 24,1 t/s | 420 t/s | **539 t/s** |
|
||
| 8k | **53,3 t/s** | 23,2 t/s | 497 t/s | **817 t/s** |
|
||
| 32k | **47,8 t/s** | 20,5 t/s | 486 t/s | **767 t/s** |
|
||
|
||
**Generierung: A3B ist durchgehend ~2,3x schneller.** Nur 3B der 35B Parameter sind
|
||
pro Token aktiv, das dichte 27B muss alle 27B lesen — auch wenn es komplett im VRAM
|
||
liegt und das A3B ein Drittel seiner Experten im RAM hat.
|
||
|
||
**Prefill: das 27B ist ~1,6x schneller**, weil es vollständig GPU-resident ist,
|
||
während beim A3B jeder Prefill-Batch durch die CPU-Experten muss.
|
||
|
||
**Kontext: das 27B schafft die 128k hier gar nicht.** Es hat 65 Layer mit 4 KV-Heads
|
||
gegen 40 Layer mit 2 — bei `full_attention_interval=4` also 16 statt 10
|
||
Full-Attention-Layer und damit **4,25 GiB KV bei 128k statt 1,33 GiB**. Zusammen mit
|
||
~15 GiB Gewichten reicht das nicht: schon das kleinere Q4_K_S scheitert beim
|
||
Compute-Buffer:
|
||
|
||
```
|
||
allocating 1145.13 MiB on device 0: cudaMalloc failed: out of memory
|
||
graph_reserve: failed to allocate compute buffers
|
||
```
|
||
|
||
Gemessen wurde es deshalb bei 64k ctx.
|
||
|
||
### Caveats zu diesem Vergleich
|
||
|
||
- Gemessen wurde **Q4_K_S (16,1 GB)**, nicht Q4_K_M (17 GB) — die einzige lokal
|
||
vorhandene, mit upstream llama.cpp ladbare 27B-Datei dieser Klasse. Q4_K_M ist ~5%
|
||
größer und wäre entsprechend etwas langsamer, der Abstand also eher noch größer.
|
||
- Der ollama-Blob von `qwen3.6:27b-q4_K_M` lässt sich mit upstream llama.cpp **nicht**
|
||
laden: `key qwen35.rope.dimension_sections has wrong array length; expected 4, got 3`.
|
||
Ollama fährt einen eigenen Fork; ein Vergleich exakt dieser Datei ist im selben
|
||
Runner nicht möglich.
|
||
- Ältere 27B-Zahlen unter `PrismQuant/benchmarks/` (34,3 t/s) stammen von einer
|
||
**RX 7900 XTX, gedrosselt** — nicht mit dieser Maschine vergleichbar.
|
||
|
||
### A note on the CUDA0 headroom
|
||
|
||
Free VRAM on CUDA0 was observed drifting between ~1100 and ~1300 MiB across restarts
|
||
purely from desktop compositor churn — a ~200 MiB swing with nothing else changing.
|
||
That is why the default keeps >1 GiB spare there rather than the 388 MiB that
|
||
`ncmoe=8` leaves.
|
||
|
||
## `--predict` ist ein Default, kein Deckel
|
||
|
||
Despite the name, `--predict` does **not** cap what a client can ask for. Measured
|
||
against `--predict 8192`:
|
||
|
||
| Request | Result |
|
||
|---|---|
|
||
| `max_tokens: 9000` | **8893** tokens, `finish_reason: stop` — the flag is ignored |
|
||
| no `max_tokens` | **8192** tokens, `finish_reason: length` — the flag applies |
|
||
|
||
So it only binds clients that send no `max_tokens` at all. That is still worth having
|
||
here: without it such a client generates until the 128k window is full, which at
|
||
~40 t/s is **roughly 50 minutes**, and with `-np 1` the single slot is blocked for all
|
||
of it. Set to `32768` — a runaway is bounded to ~12 minutes, while no realistic long
|
||
answer gets truncated. Many OpenAI clients do not send `max_tokens`, and at 8192 they
|
||
were silently cut off with `finish_reason: length`.
|
||
|
||
Note the sibling `gemma4-26b-llama-runner` README describes `--predict` as a "harte
|
||
Cap fuer Gesamt-Generierung" — by this measurement that is wrong there too.
|
||
|
||
`--reasoning-budget 4096` is a separate knob for the thinking block and was **not**
|
||
verified the same way; do not assume its semantics from the flag name either.
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
cd ~/projects/llama_qwen3.6_A3B
|
||
docker compose up -d
|
||
```
|
||
|
||
Loads in ~15 s (`--no-mmap`, weights read from NVMe).
|
||
|
||
## Benchmarks
|
||
|
||
```bash
|
||
python3 bench/bench.py --port 18008 --depths 512,8192,32768 --n-predict 128
|
||
./bench/sweep.sh 10:24,16:8 8:24,16:8 # <ncmoe>:<tensor-split>:<threads>
|
||
python3 bench/spec_bench.py --no-think # copy/code/prose generation workloads
|
||
```
|
||
|
||
`bench.py` issues a warmup request first — without it the first measurement reads
|
||
~30% low because of one-off CUDA graph setup.
|
||
|
||
## Image
|
||
|
||
`ghcr.io/ggml-org/llama.cpp:server-cuda`, tested at build **b10121**
|
||
(commit `555881ebc8b0`, 2026-07-25). Updated from b10068 during this deployment;
|
||
b10121 is the first build here that supports the `qwen35moe` architecture end to end.
|
||
|
||
## VRAM-Exklusivitaet
|
||
|
||
Uses ~20 GB of the 22 GB total. It **cannot** run alongside `llama-gemma4` (18006) or
|
||
qwen-prism (18004). Ollama on 11434 loads models on demand and will fight for VRAM —
|
||
stop it or let its `keep_alive` expire before starting this runner.
|