DATAandClaude Opus 5
e41dc2ba6f
Measured: with --predict 8192 a request sending max_tokens=9000 returned 8893 tokens (finish_reason stop), while a request sending no max_tokens was cut at exactly 8192 (finish_reason length). The flag is a default for clients that omit max_tokens, not a ceiling anyone can hit. That default still matters on a -np 1 server, where an unbounded client would fill the 128k window and block the only slot for ~50 minutes. 32768 bounds that to ~12 minutes without truncating realistic long answers, which 8192 was doing silently to any client that omits max_tokens. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
77 lines
2.5 KiB
YAML
77 lines
2.5 KiB
YAML
services:
|
|
llama-qwen36-a3b:
|
|
image: ghcr.io/ggml-org/llama.cpp:server-cuda
|
|
container_name: llama-qwen36-a3b
|
|
restart: unless-stopped
|
|
runtime: nvidia
|
|
network_mode: host
|
|
environment:
|
|
- NVIDIA_VISIBLE_DEVICES=0,1
|
|
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
- CUDA_DEVICE_ORDER=PCI_BUS_ID
|
|
cap_add:
|
|
- IPC_LOCK
|
|
ipc: host
|
|
volumes:
|
|
# HF hub repo dir, NOT the snapshot dir: the snapshot entry is a relative
|
|
# symlink into ../../blobs/, so the mount has to contain both.
|
|
- /mnt/2TSAM990nvme/docker-volume-outsource/llm-models/hf/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF:/models:ro
|
|
- /mnt/1TVi550s3/datas-docker-space/slot-cache/qwen36-a3b:/slots
|
|
command:
|
|
- -m
|
|
- /models/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
|
|
- --alias
|
|
- qwen3.6-35b-a3b-q4-k-m
|
|
- --ctx-size
|
|
- "131072"
|
|
# Only 10 of 40 layers are full-attention (full_attention_interval=4), so
|
|
# 128k of KV costs just ~1.3 GiB at q8_0. The other 30 layers are SSM and
|
|
# carry a constant-size recurrent state.
|
|
- -ctk
|
|
- q8_0
|
|
- -ctv
|
|
- q8_0
|
|
- -fa
|
|
- "on"
|
|
- -ngl
|
|
- "99"
|
|
# Weights are 20.6 GiB vs 22 GiB total VRAM, so the experts of the first
|
|
# N layers live in host RAM. See README for the ncmoe/-ts interaction.
|
|
- --n-cpu-moe
|
|
- "10"
|
|
- -sm
|
|
- layer
|
|
- -ts
|
|
- "24,16"
|
|
# 8 = physical cores on the Ryzen 7 3700X. Using all 16 SMT threads costs
|
|
# ~30% generation throughput (measured), because the CPU-side expert
|
|
# matmuls are memory-bound and SMT siblings just contend for bandwidth.
|
|
- --threads
|
|
- "8"
|
|
- -np
|
|
- "1"
|
|
- --no-mmap
|
|
# Default only, NOT a ceiling: a client sending max_tokens overrides it
|
|
# (verified — max_tokens=9000 returned 8893 tokens against --predict 8192).
|
|
# It bounds clients that send no max_tokens at all, which would otherwise
|
|
# generate until the 128k window is full: ~50 min blocking the single slot.
|
|
- --predict
|
|
- "32768"
|
|
- --reasoning-budget
|
|
- "4096"
|
|
- --slot-save-path
|
|
- /slots
|
|
- --jinja
|
|
- --reasoning-format
|
|
- auto
|
|
- --host
|
|
- 0.0.0.0
|
|
- --port
|
|
- "18008"
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-sf", "http://localhost:18008/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 180s
|