llama_qwen3.6_A3B/README.md

llama_qwen3.6_A3B

llama.cpp CUDA runner for Qwen3.6-35B-A3B (UD-Q4_K_M) at 128k context on 4n4rch02.

Port 18008, OpenAI-compatible API at http://192.168.3.189:18008/v1.

Host

• 4n4rch02 (192.168.3.189), Arch Linux, driver 610.43.03
• AMD Ryzen 7 3700X — 8 physical cores / 16 SMT threads
• 62 GiB RAM, 15 GiB swap
• CUDA0: RTX 3060 12 GB — shared with the KDE/Wayland desktop (~1.1 GiB at idle)
• CUDA1: RTX 3080 10 GB — dedicated

Modell

/mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

20.6 GiB, unsloth Dynamic Q4_K_M (imatrix). The compose mounts the repo dir, not the snapshot dir — the snapshot entry is a relative symlink into ../../blobs/, so both have to be inside the mount.

Architektur (qwen35moe) — warum 128k hier billig ist

This is a hybrid SSM + attention MoE, not a dense-attention model:

Layers	40
full_attention_interval	4 → only 10 layers use full attention
Remaining 30 layers	gated-delta SSM, constant-size recurrent state
Attention heads	16 Q / 2 KV, key_length = value_length = 256
Experts	256 total, 8 active, expert FFN 512, shared expert 512
Native context	262144

Only the 10 full-attention layers grow a KV cache, so 128k of KV costs just

10 layers x 2 kv-heads x 256 dim x 2 (K+V) x 1.0625 B/elem (q8_0) x 131072 tok = 1.33 GiB

Measured: the whole runtime footprint with all experts on CPU is 4.65 GB at 128k ctx. The context is not the constraint here — the 20.6 GiB of weights against 22 GiB of VRAM is. That is what the tuning below is about.

The model is multimodal-capable (the chat template emits <|vision_start|> / <|image_pad|> tokens), but no mmproj file is present, so this deployment is text-only. It is a reasoning model with <think> tags and XML-style tool calls, so --jinja is mandatory.

Tuning

1. --n-cpu-moe collides with -sm layer — -ts is not optional

--n-cpu-moe N moves the expert tensors of the first N layers to host RAM. -sm layer assigns the first layers to CUDA0. Those are the same layers, so CUDA0 gets the lightweight ones and CUDA1 ends up holding every heavy expert layer.

With the default split, every value of --n-cpu-moe from 12 down to 4 died the same way — CUDA1 out of memory while allocating the KV cache:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 680.00 MiB on device 1: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA1 buffer of size 713031680
llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache

-ts 24,16 moves the layer boundary back toward CUDA0 and fixes it. A heavy (expert-bearing) layer is ~469 MiB; a layer whose experts are on CPU is ~59 MiB.

2. --threads 8, not 16

The CPU-side expert matmuls are memory-bandwidth-bound, so the 8 SMT siblings only add contention. Measured at ncmoe=8, 8k depth:

threads	prefill	generation
8	556 t/s	57.3 t/s
16	556 t/s	39.8 t/s

+44% generation. Prefill is unaffected because it runs on the GPU.

3. Expert offload vs. VRAM headroom

All at 128k ctx, q8_0 KV, --threads 8, -ts 24,16:

--n-cpu-moe	gen @512	gen @8k	CUDA0 free	CUDA1 free
8	59.9 t/s	57.3 t/s	388 MiB	531 MiB
10 (default)	55.6 t/s	53.3 t/s	1312 MiB	529 MiB
12	51.3 t/s	49.1 t/s	2236 MiB	527 MiB
40 (all experts on CPU)	15.3 t/s	16.1 t/s	—	—

ncmoe=10 is the shipped default: it gives up 7% throughput for ~1.3 GiB of spare VRAM on CUDA0, which the desktop shares. At ncmoe=8 only 388 MiB is left there, and a browser opening a few video tabs is enough to OOM a restart: unless-stopped service into a crash loop. If the desktop is idle, ncmoe=8 is the faster setting.

CUDA1 is the binding constraint in every case — 529 MiB free is not enough for another 469 MiB expert layer, which is why ncmoe cannot go below 8 at this context size.

4. Throughput over depth (shipped config)

ncmoe=10, -ts 24,16, --threads 8, 128k ctx allocated, q8_0 KV:

prompt depth	prefill	generation
512	420 t/s	56.3 t/s
32k (27169 tok)	486 t/s	47.8 t/s
~97k (99109 tok)	425 t/s	34.1 t/s

Generation falls off ~40% between empty and ~97k, which is the attention cost on the 10 full-attention layers. Prefill stays flat around 420–490 t/s, so filling the whole 128k window takes roughly 4–5 minutes.

5. Speculative decoding makes it slower — don't enable it

Counter-intuitive but consistent, --n-predict 400, thinking disabled:

workload	none	ngram-mod	ngram-simple	ngram-cache
rewrite (output ≈ copy of prompt)	55.6	40.6	39.7	47.0
extend (structured code)	55.3	32.7	27.2	38.1
prose (novel text, control)	56.1	54.7	52.9	48.4

Every mode loses, and it loses most on exactly the structured workloads speculation is supposed to win. The reason is the CPU-side experts: verifying a K-token draft costs about K times the CPU expert work, because each token routes to its own subset of 8 out of 256 experts, so there is no weight reuse to amortize. On a fully GPU-resident model batch verification is nearly free; with --n-cpu-moe it is not. Reconsider only if the model ever fits entirely in VRAM.

6. Rejected

• -ub 256: frees only ~80 MiB per GPU (not the 469 MiB an extra layer needs) and costs 38% prefill (556 → 347 t/s). Keep the default -ub 512.
• Lower -ts toward CUDA1 (e.g. 23,17): would leave CUDA1 at ~60 MiB free.

A note on the CUDA0 headroom

Free VRAM on CUDA0 was observed drifting between ~1100 and ~1300 MiB across restarts purely from desktop compositor churn — a ~200 MiB swing with nothing else changing. That is why the default keeps >1 GiB spare there rather than the 388 MiB that ncmoe=8 leaves.

Deploy

cd ~/projects/llama_qwen3.6_A3B
docker compose up -d

Loads in ~15 s (--no-mmap, weights read from NVMe).

Benchmarks

python3 bench/bench.py --port 18008 --depths 512,8192,32768 --n-predict 128
./bench/sweep.sh 10:24,16:8 8:24,16:8      # <ncmoe>:<tensor-split>:<threads>
python3 bench/spec_bench.py --no-think     # copy/code/prose generation workloads

bench.py issues a warmup request first — without it the first measurement reads ~30% low because of one-off CUDA graph setup.

Image

ghcr.io/ggml-org/llama.cpp:server-cuda, tested at build b10121 (commit 555881ebc8b0, 2026-07-25). Updated from b10068 during this deployment; b10121 is the first build here that supports the qwen35moe architecture end to end.

VRAM-Exklusivitaet

Uses ~20 GB of the 22 GB total. It cannot run alongside llama-gemma4 (18006) or qwen-prism (18004). Ollama on 11434 loads models on demand and will fight for VRAM — stop it or let its keep_alive expire before starting this runner.