llama.cpp CUDA runner for Qwen3.6-35B-A3B at 128k context

Hybrid SSM+attention MoE (qwen35moe): only 10 of 40 layers use full
attention, so 128k of KV costs ~1.3 GiB. The binding constraint is the
20.6 GiB of weights against 22 GiB of VRAM, handled with --n-cpu-moe.

Two findings drive the config:
- --n-cpu-moe strips experts from the first N layers, which -sm layer
  assigns to CUDA0, so CUDA1 inherits every heavy layer and OOMs at any
  offload level. -ts 24,16 rebalances it.
- --threads 8 (physical cores) beats 16 by 44% on generation; the expert
  matmuls are bandwidth-bound and SMT siblings only contend.

Ships ncmoe=10 (53 t/s @8k, 34 t/s @97k) over the faster ncmoe=8 to keep
~1.3 GiB spare on CUDA0, which is shared with the desktop.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docker-compose.yml

@@ -0,0 +1,72 @@
+services:
+  llama-qwen36-a3b:
+    image: ghcr.io/ggml-org/llama.cpp:server-cuda
+    container_name: llama-qwen36-a3b
+    restart: unless-stopped
+    runtime: nvidia
+    network_mode: host
+    environment:
+      - NVIDIA_VISIBLE_DEVICES=0,1
+      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
+      - CUDA_DEVICE_ORDER=PCI_BUS_ID
+    cap_add:
+      - IPC_LOCK
+    ipc: host
+    volumes:
+      # HF hub repo dir, NOT the snapshot dir: the snapshot entry is a relative
+      # symlink into ../../blobs/, so the mount has to contain both.
+      - /mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF:/models:ro
+      - /mnt/1TVi550s3/datas-docker-space/slot-cache/qwen36-a3b:/slots
+    command:
+      - -m
+      - /models/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
+      - --alias
+      - qwen3.6-35b-a3b-q4-k-m
+      - --ctx-size
+      - "131072"
+      # Only 10 of 40 layers are full-attention (full_attention_interval=4), so
+      # 128k of KV costs just ~1.3 GiB at q8_0. The other 30 layers are SSM and
+      # carry a constant-size recurrent state.
+      - -ctk
+      - q8_0
+      - -ctv
+      - q8_0
+      - -fa
+      - "on"
+      - -ngl
+      - "99"
+      # Weights are 20.6 GiB vs 22 GiB total VRAM, so the experts of the first
+      # N layers live in host RAM. See README for the ncmoe/-ts interaction.
+      - --n-cpu-moe
+      - "10"
+      - -sm
+      - layer
+      - -ts
+      - "24,16"
+      # 8 = physical cores on the Ryzen 7 3700X. Using all 16 SMT threads costs
+      # ~30% generation throughput (measured), because the CPU-side expert
+      # matmuls are memory-bound and SMT siblings just contend for bandwidth.
+      - --threads
+      - "8"
+      - -np
+      - "1"
+      - --no-mmap
+      - --predict
+      - "8192"
+      - --reasoning-budget
+      - "4096"
+      - --slot-save-path
+      - /slots
+      - --jinja
+      - --reasoning-format
+      - auto
+      - --host
+      - 0.0.0.0
+      - --port
+      - "18008"
+    healthcheck:
+      test: ["CMD", "curl", "-sf", "http://localhost:18008/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 180s