llama.cpp CUDA runner for Qwen3.6-35B-A3B at 128k context

Hybrid SSM+attention MoE (qwen35moe): only 10 of 40 layers use full attention, so 128k of KV costs ~1.3 GiB. The binding constraint is the 20.6 GiB of weights against 22 GiB of VRAM, handled with --n-cpu-moe. Two findings drive the config: - --n-cpu-moe strips experts from the first N layers, which -sm layer assigns to CUDA0, so CUDA1 inherits every heavy layer and OOMs at any offload level. -ts 24,16 rebalances it. - --threads 8 (physical cores) beats 16 by 44% on generation; the expert matmuls are bandwidth-bound and SMT siblings only contend. Ships ncmoe=10 (53 t/s @8k, 34 t/s @97k) over the faster ncmoe=8 to keep ~1.3 GiB spare on CUDA0, which is shared with the desktop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-25 13:49:33 +02:00
commit 2c9cd04e96
6 changed files with 421 additions and 0 deletions
@@ -0,0 +1,5 @@
+*.log
+*.gguf
+bench/results/*.json
+__pycache__/
+.venv/
@@ -0,0 +1,151 @@
+# llama_qwen3.6_A3B
+
+llama.cpp CUDA runner for **Qwen3.6-35B-A3B** (UD-Q4_K_M) at **128k context** on 4n4rch02.
+
+Port **18008**, OpenAI-compatible API at `http://192.168.3.189:18008/v1`.
+
+## Host
+
+- 4n4rch02 (192.168.3.189), Arch Linux, driver 610.43.03
+- AMD Ryzen 7 3700X — **8 physical cores** / 16 SMT threads
+- 62 GiB RAM, 15 GiB swap
+- CUDA0: RTX 3060 12 GB — **shared with the KDE/Wayland desktop** (~1.1 GiB at idle)
+- CUDA1: RTX 3080 10 GB — dedicated
+
+## Modell
+
+`/mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`
+
+20.6 GiB, unsloth Dynamic Q4_K_M (imatrix). The compose mounts the **repo dir**, not
+the snapshot dir — the snapshot entry is a relative symlink into `../../blobs/`, so
+both have to be inside the mount.
+
+### Architektur (`qwen35moe`) — warum 128k hier billig ist
+
+This is a **hybrid SSM + attention MoE**, not a dense-attention model:
+
+| | |
+|---|---|
+| Layers | 40 |
+| `full_attention_interval` | 4 → **only 10 layers use full attention** |
+| Remaining 30 layers | gated-delta SSM, constant-size recurrent state |
+| Attention heads | 16 Q / 2 KV, `key_length` = `value_length` = 256 |
+| Experts | 256 total, 8 active, expert FFN 512, shared expert 512 |
+| Native context | 262144 |
+
+Only the 10 full-attention layers grow a KV cache, so 128k of KV costs just
+
+```
+10 layers x 2 kv-heads x 256 dim x 2 (K+V) x 1.0625 B/elem (q8_0) x 131072 tok = 1.33 GiB
+```
+
+Measured: the whole runtime footprint with *all* experts on CPU is 4.65 GB at 128k ctx.
+**The context is not the constraint here — the 20.6 GiB of weights against 22 GiB of
+VRAM is.** That is what the tuning below is about.
+
+The model is multimodal-capable (the chat template emits `<|vision_start|>` /
+`<|image_pad|>` tokens), but **no mmproj file is present**, so this deployment is
+text-only. It is a reasoning model with `<think>` tags and XML-style tool calls, so
+`--jinja` is mandatory.
+
+## Tuning
+
+### 1. `--n-cpu-moe` collides with `-sm layer` — `-ts` is not optional
+
+`--n-cpu-moe N` moves the expert tensors of the **first N layers** to host RAM.
+`-sm layer` assigns the **first** layers to CUDA0. Those are the same layers, so
+CUDA0 gets the lightweight ones and CUDA1 ends up holding every heavy expert layer.
+
+With the default split, **every** value of `--n-cpu-moe` from 12 down to 4 died the
+same way — CUDA1 out of memory while allocating the KV cache:
+
+```
+ggml_backend_cuda_buffer_type_alloc_buffer: allocating 680.00 MiB on device 1: cudaMalloc failed: out of memory
+alloc_tensor_range: failed to allocate CUDA1 buffer of size 713031680
+llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
+```
+
+`-ts 24,16` moves the layer boundary back toward CUDA0 and fixes it. A heavy
+(expert-bearing) layer is ~469 MiB; a layer whose experts are on CPU is ~59 MiB.
+
+### 2. `--threads 8`, not 16
+
+The CPU-side expert matmuls are memory-bandwidth-bound, so the 8 SMT siblings only
+add contention. Measured at `ncmoe=8`, 8k depth:
+
+| threads | prefill | generation |
+|---|---|---|
+| 8 | 556 t/s | **57.3 t/s** |
+| 16 | 556 t/s | 39.8 t/s |
+
+**+44% generation.** Prefill is unaffected because it runs on the GPU.
+
+### 3. Expert offload vs. VRAM headroom
+
+All at 128k ctx, q8_0 KV, `--threads 8`, `-ts 24,16`:
+
+| `--n-cpu-moe` | gen @512 | gen @8k | CUDA0 free | CUDA1 free |
+|---|---|---|---|---|
+| 8 | 59.9 t/s | 57.3 t/s | 388 MiB | 531 MiB |
+| **10 (default)** | **55.6 t/s** | **53.3 t/s** | **1312 MiB** | **529 MiB** |
+| 12 | 51.3 t/s | 49.1 t/s | 2236 MiB | 527 MiB |
+| 40 (all experts on CPU) | 15.3 t/s | 16.1 t/s | — | — |
+
+`ncmoe=10` is the shipped default: it gives up 7% throughput for **~1.3 GiB of spare
+VRAM on CUDA0**, which the desktop shares. At `ncmoe=8` only 388 MiB is left there,
+and a browser opening a few video tabs is enough to OOM a `restart: unless-stopped`
+service into a crash loop. If the desktop is idle, `ncmoe=8` is the faster setting.
+
+CUDA1 is the binding constraint in every case — 529 MiB free is not enough for another
+469 MiB expert layer, which is why `ncmoe` cannot go below 8 at this context size.
+
+### 4. Throughput over depth (shipped config)
+
+`ncmoe=10`, `-ts 24,16`, `--threads 8`, 128k ctx allocated, q8_0 KV:
+
+| prompt depth | prefill | generation |
+|---|---|---|
+| 512 | 420 t/s | 56.3 t/s |
+| 32k (27169 tok) | 486 t/s | 47.8 t/s |
+| ~97k (99109 tok) | 425 t/s | 34.1 t/s |
+
+Generation falls off ~40% between empty and ~97k, which is the attention cost on the
+10 full-attention layers. Prefill stays flat around 420–490 t/s, so filling the whole
+128k window takes roughly 4–5 minutes.
+
+### 5. Rejected
+
+- `-ub 256`: frees only ~80 MiB per GPU (not the 469 MiB an extra layer needs) and
+  costs **38% prefill** (556 → 347 t/s). Keep the default `-ub 512`.
+- Lower `-ts` toward CUDA1 (e.g. `23,17`): would leave CUDA1 at ~60 MiB free.
+
+## Deploy
+
+```bash
+cd ~/projects/llama_qwen3.6_A3B
+docker compose up -d
+```
+
+Loads in ~15 s (`--no-mmap`, weights read from NVMe).
+
+## Benchmarks
+
+```bash
+python3 bench/bench.py --port 18008 --depths 512,8192,32768 --n-predict 128
+./bench/sweep.sh 10:24,16:8 8:24,16:8      # <ncmoe>:<tensor-split>:<threads>
+```
+
+`bench.py` issues a warmup request first — without it the first measurement reads
+~30% low because of one-off CUDA graph setup.
+
+## Image
+
+`ghcr.io/ggml-org/llama.cpp:server-cuda`, tested at build **b10121**
+(commit `555881ebc8b0`, 2026-07-25). Updated from b10068 during this deployment;
+b10121 is the first build here that supports the `qwen35moe` architecture end to end.
+
+## VRAM-Exklusivitaet
+
+Uses ~20 GB of the 22 GB total. It **cannot** run alongside `llama-gemma4` (18006) or
+qwen-prism (18004). Ollama on 11434 loads models on demand and will fight for VRAM —
+stop it or let its `keep_alive` expire before starting this runner.
@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+"""Benchmark llama.cpp server: prefill (pp) and generation (tg) throughput."""
+import json, sys, urllib.request, argparse, random, string
+
+def post(port, path, payload, timeout=1800):
+    req = urllib.request.Request(
+        f"http://localhost:{port}{path}",
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"},
+    )
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.load(r)
+
+def make_prompt(approx_tokens):
+    # ~0.75 words/token for english-ish filler; use varied words to avoid trivial cache hits
+    words = ["system", "kernel", "vector", "matrix", "gradient", "tensor", "buffer",
+             "context", "attention", "expert", "router", "layer", "cache", "token",
+             "inference", "throughput", "latency", "quantize", "offload", "memory"]
+    rnd = random.Random(1234)
+    n_words = int(approx_tokens * 0.75)
+    return " ".join(rnd.choice(words) for _ in range(n_words))
+
+def bench(port, prompt_tokens, n_predict, label):
+    prompt = make_prompt(prompt_tokens)
+    r = post(port, "/completion", {
+        "prompt": prompt,
+        "n_predict": n_predict,
+        "temperature": 0.7,
+        "cache_prompt": False,
+    })
+    t = r.get("timings", {})
+    pp_n = t.get("prompt_n", 0); pp_ms = t.get("prompt_ms", 0)
+    tg_n = t.get("predicted_n", 0); tg_ms = t.get("predicted_ms", 0)
+    pp_tps = t.get("prompt_per_second") or 0
+    tg_tps = t.get("predicted_per_second") or 0
+    print(f"{label:<28} pp: {pp_n:>7} tok @ {pp_tps:>8.1f} t/s | tg: {tg_n:>4} tok @ {tg_tps:>6.2f} t/s",
+          flush=True)
+    return {"label": label, "pp_n": pp_n, "pp_tps": pp_tps, "tg_n": tg_n, "tg_tps": tg_tps,
+            "pp_ms": pp_ms, "tg_ms": tg_ms}
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--port", type=int, default=18008)
+    ap.add_argument("--label", default="run")
+    ap.add_argument("--depths", default="512,8192,32768,131072")
+    ap.add_argument("--n-predict", type=int, default=128)
+    ap.add_argument("--json-out", default=None)
+    a = ap.parse_args()
+
+    # Warm up: the first request after load pays one-off CUDA graph / kernel
+    # setup that otherwise shows up as a bogus ~30% slower first data point.
+    try:
+        post(a.port, "/completion", {"prompt": make_prompt(64), "n_predict": 16,
+                                     "temperature": 0, "cache_prompt": False})
+    except Exception as e:
+        print(f"warmup failed: {e}")
+
+    results = []
+    for d in [int(x) for x in a.depths.split(",")]:
+        try:
+            results.append(bench(a.port, d, a.n_predict, f"{a.label} @{d//1024}k" if d >= 1024 else f"{a.label} @{d}"))
+        except Exception as e:
+            print(f"{a.label} @{d}: FAILED {type(e).__name__}: {e}")
+    if a.json_out:
+        with open(a.json_out, "w") as f:
+            json.dump(results, f, indent=2)
@@ -0,0 +1,60 @@
+#!/bin/bash
+# Sweep llama.cpp runner configs for Qwen3.6-35B-A3B; report VRAM + throughput.
+#
+# Usage:   ./sweep.sh <ncmoe>:<tensor-split>[:<threads>] ...
+# Example: ./sweep.sh 10:24,16:8 8:24,16:8
+#
+# Env: CTX (default 131072), DEPTHS (default 512,8192), PORT (default 18008)
+
+HUB=${HUB:-/mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF}
+SNAP=${SNAP:-snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf}
+CTX=${CTX:-131072}
+PORT=${PORT:-18008}
+NAME=qwen36-sweep
+BENCH=${BENCH:-$(dirname "$0")/bench.py}
+
+run_cfg() {
+  local spec="$1"
+  local ncmoe="${spec%%:*}"
+  local rest="${spec#*:}"
+  local ts="${rest%%:*}"
+  local threads="${rest#*:}"
+  [ "$threads" = "$ts" ] && threads=8
+
+  echo "########## n-cpu-moe=$ncmoe  -ts $ts  threads=$threads  ctx=$CTX ##########"
+  docker rm -f $NAME >/dev/null 2>&1
+  docker run -d --name $NAME --runtime nvidia --network host \
+    -e NVIDIA_VISIBLE_DEVICES=0,1 \
+    -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
+    -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
+    -v "$HUB":/models:ro \
+    ghcr.io/ggml-org/llama.cpp:server-cuda \
+    -m /models/$SNAP \
+    --ctx-size $CTX -ctk q8_0 -ctv q8_0 -fa on \
+    -ngl 99 --n-cpu-moe "$ncmoe" -sm layer -ts "$ts" \
+    --threads "$threads" -np 1 --no-mmap $EXTRA \
+    --host 0.0.0.0 --port $PORT >/dev/null 2>&1
+
+  local ok=0
+  for _ in $(seq 1 200); do
+    if docker logs $NAME 2>&1 | grep -q "listening on"; then ok=1; break; fi
+    if docker logs $NAME 2>&1 | grep -qiE "out of memory|CUDA error|failed to allocate|terminate called|error loading model"; then ok=2; break; fi
+    if ! docker ps --format '{{.Names}}' | grep -q "^$NAME$"; then ok=3; break; fi
+    sleep 3
+  done
+
+  if [ "$ok" != "1" ]; then
+    echo "RESULT ncmoe=$ncmoe ts=$ts threads=$threads -> FAILED"
+    docker logs $NAME 2>&1 | grep -iE "out of memory|failed to allocate" | head -2
+    docker rm -f $NAME >/dev/null 2>&1; echo; return 1
+  fi
+
+  echo "--- VRAM used/free (MiB) ---"
+  nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader
+  python3 "$BENCH" --port $PORT --label "n${ncmoe}_ts${ts}_t${threads}" \
+    --depths "${DEPTHS:-512,8192}" --n-predict 128
+  docker rm -f $NAME >/dev/null 2>&1
+  echo
+}
+
+for c in "$@"; do run_cfg "$c"; done
@@ -0,0 +1,72 @@
+services:
+  llama-qwen36-a3b:
+    image: ghcr.io/ggml-org/llama.cpp:server-cuda
+    container_name: llama-qwen36-a3b
+    restart: unless-stopped
+    runtime: nvidia
+    network_mode: host
+    environment:
+      - NVIDIA_VISIBLE_DEVICES=0,1
+      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
+      - CUDA_DEVICE_ORDER=PCI_BUS_ID
+    cap_add:
+      - IPC_LOCK
+    ipc: host
+    volumes:
+      # HF hub repo dir, NOT the snapshot dir: the snapshot entry is a relative
+      # symlink into ../../blobs/, so the mount has to contain both.
+      - /mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF:/models:ro
+      - /mnt/1TVi550s3/datas-docker-space/slot-cache/qwen36-a3b:/slots
+    command:
+      - -m
+      - /models/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
+      - --alias
+      - qwen3.6-35b-a3b-q4-k-m
+      - --ctx-size
+      - "131072"
+      # Only 10 of 40 layers are full-attention (full_attention_interval=4), so
+      # 128k of KV costs just ~1.3 GiB at q8_0. The other 30 layers are SSM and
+      # carry a constant-size recurrent state.
+      - -ctk
+      - q8_0
+      - -ctv
+      - q8_0
+      - -fa
+      - "on"
+      - -ngl
+      - "99"
+      # Weights are 20.6 GiB vs 22 GiB total VRAM, so the experts of the first
+      # N layers live in host RAM. See README for the ncmoe/-ts interaction.
+      - --n-cpu-moe
+      - "10"
+      - -sm
+      - layer
+      - -ts
+      - "24,16"
+      # 8 = physical cores on the Ryzen 7 3700X. Using all 16 SMT threads costs
+      # ~30% generation throughput (measured), because the CPU-side expert
+      # matmuls are memory-bound and SMT siblings just contend for bandwidth.
+      - --threads
+      - "8"
+      - -np
+      - "1"
+      - --no-mmap
+      - --predict
+      - "8192"
+      - --reasoning-budget
+      - "4096"
+      - --slot-save-path
+      - /slots
+      - --jinja
+      - --reasoning-format
+      - auto
+      - --host
+      - 0.0.0.0
+      - --port
+      - "18008"
+    healthcheck:
+      test: ["CMD", "curl", "-sf", "http://localhost:18008/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 180s
@@ -0,0 +1,67 @@
+import sys, struct
+
+GGUF_MAGIC = 0x46554747
+# value types
+T_UINT8,T_INT8,T_UINT16,T_INT16,T_UINT32,T_INT32,T_FLOAT32,T_BOOL,T_STRING,T_ARRAY,T_UINT64,T_INT64,T_FLOAT64 = range(13)
+
+f = open(sys.argv[1],'rb')
+
+def rd(n): return f.read(n)
+def u32(): return struct.unpack('<I', rd(4))[0]
+def u64(): return struct.unpack('<Q', rd(8))[0]
+def i32(): return struct.unpack('<i', rd(4))[0]
+def i64(): return struct.unpack('<q', rd(8))[0]
+def f32(): return struct.unpack('<f', rd(4))[0]
+def f64(): return struct.unpack('<d', rd(8))[0]
+def string():
+    n = u64()
+    return rd(n).decode('utf-8', errors='replace')
+
+def value(t):
+    if t == T_UINT8:  return struct.unpack('<B', rd(1))[0]
+    if t == T_INT8:   return struct.unpack('<b', rd(1))[0]
+    if t == T_UINT16: return struct.unpack('<H', rd(2))[0]
+    if t == T_INT16:  return struct.unpack('<h', rd(2))[0]
+    if t == T_UINT32: return u32()
+    if t == T_INT32:  return i32()
+    if t == T_FLOAT32:return f32()
+    if t == T_BOOL:   return struct.unpack('<?', rd(1))[0]
+    if t == T_STRING: return string()
+    if t == T_UINT64: return u64()
+    if t == T_INT64:  return i64()
+    if t == T_FLOAT64:return f64()
+    if t == T_ARRAY:
+        et = u32(); n = u64()
+        if et == T_STRING:
+            # don't materialize huge token lists
+            if n > 16:
+                for _ in range(n):
+                    ln = u64(); f.seek(ln, 1)
+                return f'<array string x{n}>'
+            return [string() for _ in range(n)]
+        sizes = {T_UINT8:1,T_INT8:1,T_UINT16:2,T_INT16:2,T_UINT32:4,T_INT32:4,T_FLOAT32:4,T_BOOL:1,T_UINT64:8,T_INT64:8,T_FLOAT64:8}
+        if n > 16:
+            f.seek(sizes[et]*n, 1)
+            return f'<array t{et} x{n}>'
+        return [value(et) for _ in range(n)]
+    raise ValueError(f'unknown type {t}')
+
+magic = u32()
+assert magic == GGUF_MAGIC, hex(magic)
+ver = u32()
+n_tensors = u64()
+n_kv = u64()
+print(f'gguf_version={ver} n_tensors={n_tensors} n_kv={n_kv}')
+print('---')
+kv = {}
+for _ in range(n_kv):
+    k = string(); t = u32(); v = value(t)
+    kv[k] = v
+
+for k, v in kv.items():
+    if k.startswith('tokenizer.ggml.') and k not in ('tokenizer.ggml.model','tokenizer.ggml.pre','tokenizer.ggml.bos_token_id','tokenizer.ggml.eos_token_id','tokenizer.ggml.padding_token_id','tokenizer.ggml.add_bos_token'):
+        continue
+    if k == 'tokenizer.chat_template':
+        print(f'{k} = <len {len(str(v))}>')
+        continue
+    print(f'{k} = {v}')