llama_qwen3.6_A3B/README.md

llama_qwen3.6_A3B

llama.cpp CUDA runner for Qwen3.6-35B-A3B (UD-Q4_K_M) at 128k context on 4n4rch02.

Port 18008, OpenAI-compatible API at http://192.168.3.189:18008/v1.

Host

• 4n4rch02 (192.168.3.189), Arch Linux, driver 610.43.03
• AMD Ryzen 7 3700X — 8 physical cores / 16 SMT threads
• 62 GiB RAM, 15 GiB swap
• CUDA0: RTX 3060 12 GB — shared with the KDE/Wayland desktop (~1.1 GiB at idle)
• CUDA1: RTX 3080 10 GB — dedicated

Modell

/mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

20.6 GiB, unsloth Dynamic Q4_K_M (imatrix). The compose mounts the repo dir, not the snapshot dir — the snapshot entry is a relative symlink into ../../blobs/, so both have to be inside the mount.

Architektur (qwen35moe) — warum 128k hier billig ist

This is a hybrid SSM + attention MoE, not a dense-attention model:

Layers	40
full_attention_interval	4 → only 10 layers use full attention
Remaining 30 layers	gated-delta SSM, constant-size recurrent state
Attention heads	16 Q / 2 KV, key_length = value_length = 256
Experts	256 total, 8 active, expert FFN 512, shared expert 512
Native context	262144

Only the 10 full-attention layers grow a KV cache, so 128k of KV costs just

10 layers x 2 kv-heads x 256 dim x 2 (K+V) x 1.0625 B/elem (q8_0) x 131072 tok = 1.33 GiB

Measured: the whole runtime footprint with all experts on CPU is 4.65 GB at 128k ctx. The context is not the constraint here — the 20.6 GiB of weights against 22 GiB of VRAM is. That is what the tuning below is about.

The model is multimodal-capable (the chat template emits <|vision_start|> / <|image_pad|> tokens), but no mmproj file is present, so this deployment is text-only. It is a reasoning model with <think> tags and XML-style tool calls, so --jinja is mandatory.

Tuning

1. --n-cpu-moe collides with -sm layer — -ts is not optional

--n-cpu-moe N moves the expert tensors of the first N layers to host RAM. -sm layer assigns the first layers to CUDA0. Those are the same layers, so CUDA0 gets the lightweight ones and CUDA1 ends up holding every heavy expert layer.

With the default split, every value of --n-cpu-moe from 12 down to 4 died the same way — CUDA1 out of memory while allocating the KV cache:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 680.00 MiB on device 1: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA1 buffer of size 713031680
llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache

-ts 24,16 moves the layer boundary back toward CUDA0 and fixes it. A heavy (expert-bearing) layer is ~469 MiB; a layer whose experts are on CPU is ~59 MiB.

2. --threads 8, not 16

The CPU-side expert matmuls are memory-bandwidth-bound, so the 8 SMT siblings only add contention. Measured at ncmoe=8, 8k depth:

threads	prefill	generation
8	556 t/s	57.3 t/s
16	556 t/s	39.8 t/s

+44% generation. Prefill is unaffected because it runs on the GPU.

3. Expert offload vs. VRAM headroom

All at 128k ctx, q8_0 KV, --threads 8, -ts 24,16:

--n-cpu-moe	gen @512	gen @8k	CUDA0 free	CUDA1 free
8	59.9 t/s	57.3 t/s	388 MiB	531 MiB
10 (default)	55.6 t/s	53.3 t/s	1312 MiB	529 MiB
12	51.3 t/s	49.1 t/s	2236 MiB	527 MiB
40 (all experts on CPU)	15.3 t/s	16.1 t/s	—	—

ncmoe=10 is the shipped default: it gives up 7% throughput for ~1.3 GiB of spare VRAM on CUDA0, which the desktop shares. At ncmoe=8 only 388 MiB is left there, and a browser opening a few video tabs is enough to OOM a restart: unless-stopped service into a crash loop. If the desktop is idle, ncmoe=8 is the faster setting.

CUDA1 is the binding constraint in every case — 529 MiB free is not enough for another 469 MiB expert layer, which is why ncmoe cannot go below 8 at this context size.

4. Throughput over depth (shipped config)

ncmoe=10, -ts 24,16, --threads 8, 128k ctx allocated, q8_0 KV:

prompt depth	prefill	generation
512	420 t/s	56.3 t/s
32k (27169 tok)	486 t/s	47.8 t/s
~97k (99109 tok)	425 t/s	34.1 t/s

Generation falls off ~40% between empty and ~97k, which is the attention cost on the 10 full-attention layers. Prefill stays flat around 420–490 t/s, so filling the whole 128k window takes roughly 4–5 minutes.

5. Rejected

• -ub 256: frees only ~80 MiB per GPU (not the 469 MiB an extra layer needs) and costs 38% prefill (556 → 347 t/s). Keep the default -ub 512.
• Lower -ts toward CUDA1 (e.g. 23,17): would leave CUDA1 at ~60 MiB free.

Deploy

cd ~/projects/llama_qwen3.6_A3B
docker compose up -d

Loads in ~15 s (--no-mmap, weights read from NVMe).

Benchmarks

python3 bench/bench.py --port 18008 --depths 512,8192,32768 --n-predict 128
./bench/sweep.sh 10:24,16:8 8:24,16:8      # <ncmoe>:<tensor-split>:<threads>

bench.py issues a warmup request first — without it the first measurement reads ~30% low because of one-off CUDA graph setup.

Image

ghcr.io/ggml-org/llama.cpp:server-cuda, tested at build b10121 (commit 555881ebc8b0, 2026-07-25). Updated from b10068 during this deployment; b10121 is the first build here that supports the qwen35moe architecture end to end.

VRAM-Exklusivitaet

Uses ~20 GB of the 22 GB total. It cannot run alongside llama-gemma4 (18006) or qwen-prism (18004). Ollama on 11434 loads models on demand and will fight for VRAM — stop it or let its keep_alive expire before starting this runner.