commit 2c9cd04e96d164e022e4b3232698c2703002fb49 Author: DATA Date: Sat Jul 25 13:49:33 2026 +0200 llama.cpp CUDA runner for Qwen3.6-35B-A3B at 128k context Hybrid SSM+attention MoE (qwen35moe): only 10 of 40 layers use full attention, so 128k of KV costs ~1.3 GiB. The binding constraint is the 20.6 GiB of weights against 22 GiB of VRAM, handled with --n-cpu-moe. Two findings drive the config: - --n-cpu-moe strips experts from the first N layers, which -sm layer assigns to CUDA0, so CUDA1 inherits every heavy layer and OOMs at any offload level. -ts 24,16 rebalances it. - --threads 8 (physical cores) beats 16 by 44% on generation; the expert matmuls are bandwidth-bound and SMT siblings only contend. Ships ncmoe=10 (53 t/s @8k, 34 t/s @97k) over the faster ncmoe=8 to keep ~1.3 GiB spare on CUDA0, which is shared with the desktop. Co-Authored-By: Claude Opus 5 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..9affe70 --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +*.log +*.gguf +bench/results/*.json +__pycache__/ +.venv/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..5299244 --- /dev/null +++ b/README.md @@ -0,0 +1,151 @@ +# llama_qwen3.6_A3B + +llama.cpp CUDA runner for **Qwen3.6-35B-A3B** (UD-Q4_K_M) at **128k context** on 4n4rch02. + +Port **18008**, OpenAI-compatible API at `http://192.168.3.189:18008/v1`. + +## Host + +- 4n4rch02 (192.168.3.189), Arch Linux, driver 610.43.03 +- AMD Ryzen 7 3700X — **8 physical cores** / 16 SMT threads +- 62 GiB RAM, 15 GiB swap +- CUDA0: RTX 3060 12 GB — **shared with the KDE/Wayland desktop** (~1.1 GiB at idle) +- CUDA1: RTX 3080 10 GB — dedicated + +## Modell + +`/mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf` + +20.6 GiB, unsloth Dynamic Q4_K_M (imatrix). The compose mounts the **repo dir**, not +the snapshot dir — the snapshot entry is a relative symlink into `../../blobs/`, so +both have to be inside the mount. + +### Architektur (`qwen35moe`) — warum 128k hier billig ist + +This is a **hybrid SSM + attention MoE**, not a dense-attention model: + +| | | +|---|---| +| Layers | 40 | +| `full_attention_interval` | 4 → **only 10 layers use full attention** | +| Remaining 30 layers | gated-delta SSM, constant-size recurrent state | +| Attention heads | 16 Q / 2 KV, `key_length` = `value_length` = 256 | +| Experts | 256 total, 8 active, expert FFN 512, shared expert 512 | +| Native context | 262144 | + +Only the 10 full-attention layers grow a KV cache, so 128k of KV costs just + +``` +10 layers x 2 kv-heads x 256 dim x 2 (K+V) x 1.0625 B/elem (q8_0) x 131072 tok = 1.33 GiB +``` + +Measured: the whole runtime footprint with *all* experts on CPU is 4.65 GB at 128k ctx. +**The context is not the constraint here — the 20.6 GiB of weights against 22 GiB of +VRAM is.** That is what the tuning below is about. + +The model is multimodal-capable (the chat template emits `<|vision_start|>` / +`<|image_pad|>` tokens), but **no mmproj file is present**, so this deployment is +text-only. It is a reasoning model with `` tags and XML-style tool calls, so +`--jinja` is mandatory. + +## Tuning + +### 1. `--n-cpu-moe` collides with `-sm layer` — `-ts` is not optional + +`--n-cpu-moe N` moves the expert tensors of the **first N layers** to host RAM. +`-sm layer` assigns the **first** layers to CUDA0. Those are the same layers, so +CUDA0 gets the lightweight ones and CUDA1 ends up holding every heavy expert layer. + +With the default split, **every** value of `--n-cpu-moe` from 12 down to 4 died the +same way — CUDA1 out of memory while allocating the KV cache: + +``` +ggml_backend_cuda_buffer_type_alloc_buffer: allocating 680.00 MiB on device 1: cudaMalloc failed: out of memory +alloc_tensor_range: failed to allocate CUDA1 buffer of size 713031680 +llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache +``` + +`-ts 24,16` moves the layer boundary back toward CUDA0 and fixes it. A heavy +(expert-bearing) layer is ~469 MiB; a layer whose experts are on CPU is ~59 MiB. + +### 2. `--threads 8`, not 16 + +The CPU-side expert matmuls are memory-bandwidth-bound, so the 8 SMT siblings only +add contention. Measured at `ncmoe=8`, 8k depth: + +| threads | prefill | generation | +|---|---|---| +| 8 | 556 t/s | **57.3 t/s** | +| 16 | 556 t/s | 39.8 t/s | + +**+44% generation.** Prefill is unaffected because it runs on the GPU. + +### 3. Expert offload vs. VRAM headroom + +All at 128k ctx, q8_0 KV, `--threads 8`, `-ts 24,16`: + +| `--n-cpu-moe` | gen @512 | gen @8k | CUDA0 free | CUDA1 free | +|---|---|---|---|---| +| 8 | 59.9 t/s | 57.3 t/s | 388 MiB | 531 MiB | +| **10 (default)** | **55.6 t/s** | **53.3 t/s** | **1312 MiB** | **529 MiB** | +| 12 | 51.3 t/s | 49.1 t/s | 2236 MiB | 527 MiB | +| 40 (all experts on CPU) | 15.3 t/s | 16.1 t/s | — | — | + +`ncmoe=10` is the shipped default: it gives up 7% throughput for **~1.3 GiB of spare +VRAM on CUDA0**, which the desktop shares. At `ncmoe=8` only 388 MiB is left there, +and a browser opening a few video tabs is enough to OOM a `restart: unless-stopped` +service into a crash loop. If the desktop is idle, `ncmoe=8` is the faster setting. + +CUDA1 is the binding constraint in every case — 529 MiB free is not enough for another +469 MiB expert layer, which is why `ncmoe` cannot go below 8 at this context size. + +### 4. Throughput over depth (shipped config) + +`ncmoe=10`, `-ts 24,16`, `--threads 8`, 128k ctx allocated, q8_0 KV: + +| prompt depth | prefill | generation | +|---|---|---| +| 512 | 420 t/s | 56.3 t/s | +| 32k (27169 tok) | 486 t/s | 47.8 t/s | +| ~97k (99109 tok) | 425 t/s | 34.1 t/s | + +Generation falls off ~40% between empty and ~97k, which is the attention cost on the +10 full-attention layers. Prefill stays flat around 420–490 t/s, so filling the whole +128k window takes roughly 4–5 minutes. + +### 5. Rejected + +- `-ub 256`: frees only ~80 MiB per GPU (not the 469 MiB an extra layer needs) and + costs **38% prefill** (556 → 347 t/s). Keep the default `-ub 512`. +- Lower `-ts` toward CUDA1 (e.g. `23,17`): would leave CUDA1 at ~60 MiB free. + +## Deploy + +```bash +cd ~/projects/llama_qwen3.6_A3B +docker compose up -d +``` + +Loads in ~15 s (`--no-mmap`, weights read from NVMe). + +## Benchmarks + +```bash +python3 bench/bench.py --port 18008 --depths 512,8192,32768 --n-predict 128 +./bench/sweep.sh 10:24,16:8 8:24,16:8 # :: +``` + +`bench.py` issues a warmup request first — without it the first measurement reads +~30% low because of one-off CUDA graph setup. + +## Image + +`ghcr.io/ggml-org/llama.cpp:server-cuda`, tested at build **b10121** +(commit `555881ebc8b0`, 2026-07-25). Updated from b10068 during this deployment; +b10121 is the first build here that supports the `qwen35moe` architecture end to end. + +## VRAM-Exklusivitaet + +Uses ~20 GB of the 22 GB total. It **cannot** run alongside `llama-gemma4` (18006) or +qwen-prism (18004). Ollama on 11434 loads models on demand and will fight for VRAM — +stop it or let its `keep_alive` expire before starting this runner. diff --git a/bench/bench.py b/bench/bench.py new file mode 100755 index 0000000..da8d23e --- /dev/null +++ b/bench/bench.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +"""Benchmark llama.cpp server: prefill (pp) and generation (tg) throughput.""" +import json, sys, urllib.request, argparse, random, string + +def post(port, path, payload, timeout=1800): + req = urllib.request.Request( + f"http://localhost:{port}{path}", + data=json.dumps(payload).encode(), + headers={"Content-Type": "application/json"}, + ) + with urllib.request.urlopen(req, timeout=timeout) as r: + return json.load(r) + +def make_prompt(approx_tokens): + # ~0.75 words/token for english-ish filler; use varied words to avoid trivial cache hits + words = ["system", "kernel", "vector", "matrix", "gradient", "tensor", "buffer", + "context", "attention", "expert", "router", "layer", "cache", "token", + "inference", "throughput", "latency", "quantize", "offload", "memory"] + rnd = random.Random(1234) + n_words = int(approx_tokens * 0.75) + return " ".join(rnd.choice(words) for _ in range(n_words)) + +def bench(port, prompt_tokens, n_predict, label): + prompt = make_prompt(prompt_tokens) + r = post(port, "/completion", { + "prompt": prompt, + "n_predict": n_predict, + "temperature": 0.7, + "cache_prompt": False, + }) + t = r.get("timings", {}) + pp_n = t.get("prompt_n", 0); pp_ms = t.get("prompt_ms", 0) + tg_n = t.get("predicted_n", 0); tg_ms = t.get("predicted_ms", 0) + pp_tps = t.get("prompt_per_second") or 0 + tg_tps = t.get("predicted_per_second") or 0 + print(f"{label:<28} pp: {pp_n:>7} tok @ {pp_tps:>8.1f} t/s | tg: {tg_n:>4} tok @ {tg_tps:>6.2f} t/s", + flush=True) + return {"label": label, "pp_n": pp_n, "pp_tps": pp_tps, "tg_n": tg_n, "tg_tps": tg_tps, + "pp_ms": pp_ms, "tg_ms": tg_ms} + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("--port", type=int, default=18008) + ap.add_argument("--label", default="run") + ap.add_argument("--depths", default="512,8192,32768,131072") + ap.add_argument("--n-predict", type=int, default=128) + ap.add_argument("--json-out", default=None) + a = ap.parse_args() + + # Warm up: the first request after load pays one-off CUDA graph / kernel + # setup that otherwise shows up as a bogus ~30% slower first data point. + try: + post(a.port, "/completion", {"prompt": make_prompt(64), "n_predict": 16, + "temperature": 0, "cache_prompt": False}) + except Exception as e: + print(f"warmup failed: {e}") + + results = [] + for d in [int(x) for x in a.depths.split(",")]: + try: + results.append(bench(a.port, d, a.n_predict, f"{a.label} @{d//1024}k" if d >= 1024 else f"{a.label} @{d}")) + except Exception as e: + print(f"{a.label} @{d}: FAILED {type(e).__name__}: {e}") + if a.json_out: + with open(a.json_out, "w") as f: + json.dump(results, f, indent=2) diff --git a/bench/sweep.sh b/bench/sweep.sh new file mode 100755 index 0000000..7776e93 --- /dev/null +++ b/bench/sweep.sh @@ -0,0 +1,60 @@ +#!/bin/bash +# Sweep llama.cpp runner configs for Qwen3.6-35B-A3B; report VRAM + throughput. +# +# Usage: ./sweep.sh :[:] ... +# Example: ./sweep.sh 10:24,16:8 8:24,16:8 +# +# Env: CTX (default 131072), DEPTHS (default 512,8192), PORT (default 18008) + +HUB=${HUB:-/mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF} +SNAP=${SNAP:-snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf} +CTX=${CTX:-131072} +PORT=${PORT:-18008} +NAME=qwen36-sweep +BENCH=${BENCH:-$(dirname "$0")/bench.py} + +run_cfg() { + local spec="$1" + local ncmoe="${spec%%:*}" + local rest="${spec#*:}" + local ts="${rest%%:*}" + local threads="${rest#*:}" + [ "$threads" = "$ts" ] && threads=8 + + echo "########## n-cpu-moe=$ncmoe -ts $ts threads=$threads ctx=$CTX ##########" + docker rm -f $NAME >/dev/null 2>&1 + docker run -d --name $NAME --runtime nvidia --network host \ + -e NVIDIA_VISIBLE_DEVICES=0,1 \ + -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \ + -e CUDA_DEVICE_ORDER=PCI_BUS_ID \ + -v "$HUB":/models:ro \ + ghcr.io/ggml-org/llama.cpp:server-cuda \ + -m /models/$SNAP \ + --ctx-size $CTX -ctk q8_0 -ctv q8_0 -fa on \ + -ngl 99 --n-cpu-moe "$ncmoe" -sm layer -ts "$ts" \ + --threads "$threads" -np 1 --no-mmap $EXTRA \ + --host 0.0.0.0 --port $PORT >/dev/null 2>&1 + + local ok=0 + for _ in $(seq 1 200); do + if docker logs $NAME 2>&1 | grep -q "listening on"; then ok=1; break; fi + if docker logs $NAME 2>&1 | grep -qiE "out of memory|CUDA error|failed to allocate|terminate called|error loading model"; then ok=2; break; fi + if ! docker ps --format '{{.Names}}' | grep -q "^$NAME$"; then ok=3; break; fi + sleep 3 + done + + if [ "$ok" != "1" ]; then + echo "RESULT ncmoe=$ncmoe ts=$ts threads=$threads -> FAILED" + docker logs $NAME 2>&1 | grep -iE "out of memory|failed to allocate" | head -2 + docker rm -f $NAME >/dev/null 2>&1; echo; return 1 + fi + + echo "--- VRAM used/free (MiB) ---" + nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader + python3 "$BENCH" --port $PORT --label "n${ncmoe}_ts${ts}_t${threads}" \ + --depths "${DEPTHS:-512,8192}" --n-predict 128 + docker rm -f $NAME >/dev/null 2>&1 + echo +} + +for c in "$@"; do run_cfg "$c"; done diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..19495cc --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,72 @@ +services: + llama-qwen36-a3b: + image: ghcr.io/ggml-org/llama.cpp:server-cuda + container_name: llama-qwen36-a3b + restart: unless-stopped + runtime: nvidia + network_mode: host + environment: + - NVIDIA_VISIBLE_DEVICES=0,1 + - NVIDIA_DRIVER_CAPABILITIES=compute,utility + - CUDA_DEVICE_ORDER=PCI_BUS_ID + cap_add: + - IPC_LOCK + ipc: host + volumes: + # HF hub repo dir, NOT the snapshot dir: the snapshot entry is a relative + # symlink into ../../blobs/, so the mount has to contain both. + - /mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF:/models:ro + - /mnt/1TVi550s3/datas-docker-space/slot-cache/qwen36-a3b:/slots + command: + - -m + - /models/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf + - --alias + - qwen3.6-35b-a3b-q4-k-m + - --ctx-size + - "131072" + # Only 10 of 40 layers are full-attention (full_attention_interval=4), so + # 128k of KV costs just ~1.3 GiB at q8_0. The other 30 layers are SSM and + # carry a constant-size recurrent state. + - -ctk + - q8_0 + - -ctv + - q8_0 + - -fa + - "on" + - -ngl + - "99" + # Weights are 20.6 GiB vs 22 GiB total VRAM, so the experts of the first + # N layers live in host RAM. See README for the ncmoe/-ts interaction. + - --n-cpu-moe + - "10" + - -sm + - layer + - -ts + - "24,16" + # 8 = physical cores on the Ryzen 7 3700X. Using all 16 SMT threads costs + # ~30% generation throughput (measured), because the CPU-side expert + # matmuls are memory-bound and SMT siblings just contend for bandwidth. + - --threads + - "8" + - -np + - "1" + - --no-mmap + - --predict + - "8192" + - --reasoning-budget + - "4096" + - --slot-save-path + - /slots + - --jinja + - --reasoning-format + - auto + - --host + - 0.0.0.0 + - --port + - "18008" + healthcheck: + test: ["CMD", "curl", "-sf", "http://localhost:18008/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 180s diff --git a/scripts/gguf_meta.py b/scripts/gguf_meta.py new file mode 100755 index 0000000..b4e2425 --- /dev/null +++ b/scripts/gguf_meta.py @@ -0,0 +1,67 @@ +import sys, struct + +GGUF_MAGIC = 0x46554747 +# value types +T_UINT8,T_INT8,T_UINT16,T_INT16,T_UINT32,T_INT32,T_FLOAT32,T_BOOL,T_STRING,T_ARRAY,T_UINT64,T_INT64,T_FLOAT64 = range(13) + +f = open(sys.argv[1],'rb') + +def rd(n): return f.read(n) +def u32(): return struct.unpack(' 16: + for _ in range(n): + ln = u64(); f.seek(ln, 1) + return f'' + return [string() for _ in range(n)] + sizes = {T_UINT8:1,T_INT8:1,T_UINT16:2,T_INT16:2,T_UINT32:4,T_INT32:4,T_FLOAT32:4,T_BOOL:1,T_UINT64:8,T_INT64:8,T_FLOAT64:8} + if n > 16: + f.seek(sizes[et]*n, 1) + return f'' + return [value(et) for _ in range(n)] + raise ValueError(f'unknown type {t}') + +magic = u32() +assert magic == GGUF_MAGIC, hex(magic) +ver = u32() +n_tensors = u64() +n_kv = u64() +print(f'gguf_version={ver} n_tensors={n_tensors} n_kv={n_kv}') +print('---') +kv = {} +for _ in range(n_kv): + k = string(); t = u32(); v = value(t) + kv[k] = v + +for k, v in kv.items(): + if k.startswith('tokenizer.ggml.') and k not in ('tokenizer.ggml.model','tokenizer.ggml.pre','tokenizer.ggml.bos_token_id','tokenizer.ggml.eos_token_id','tokenizer.ggml.padding_token_id','tokenizer.ggml.add_bos_token'): + continue + if k == 'tokenizer.chat_template': + print(f'{k} = ') + continue + print(f'{k} = {v}')