Counted the thinking block exactly via /tokenize. Server default stops it at
4095 tokens; a client sending reasoning_budget=16384 still gets 4095, so it
cannot be raised per request. Changing it means editing the compose file.
It truncates mid-derivation on a moderately hard problem, but the model
recovered and answered correctly (empty solution set, verified by brute
force), so there is no evidence 4096 is actually damaging output.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured: with --predict 8192 a request sending max_tokens=9000 returned
8893 tokens (finish_reason stop), while a request sending no max_tokens was
cut at exactly 8192 (finish_reason length). The flag is a default for
clients that omit max_tokens, not a ceiling anyone can hit.
That default still matters on a -np 1 server, where an unbounded client
would fill the 128k window and block the only slot for ~50 minutes. 32768
bounds that to ~12 minutes without truncating realistic long answers, which
8192 was doing silently to any client that omits max_tokens.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Generation: A3B ~2.3x faster at every depth (3B of 35B params active per
token vs all 27B). Prefill: 27B ~1.6x faster, since it is fully GPU-resident
while every A3B prefill batch goes through the CPU experts.
The 27B cannot do 128k here at all: 65 layers x 4 KV heads gives 4.25 GiB of
KV at 128k against the A3B's 1.33 GiB, so even Q4_K_S OOMs on the compute
buffer. Measured at 64k instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Verifying a K-token draft costs ~K times the CPU expert work, because each
token routes to its own 8-of-256 experts and nothing is amortized. The loss
is largest on structured code output (-51% for ngram-simple), i.e. exactly
where speculation should have won.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hybrid SSM+attention MoE (qwen35moe): only 10 of 40 layers use full
attention, so 128k of KV costs ~1.3 GiB. The binding constraint is the
20.6 GiB of weights against 22 GiB of VRAM, handled with --n-cpu-moe.
Two findings drive the config:
- --n-cpu-moe strips experts from the first N layers, which -sm layer
assigns to CUDA0, so CUDA1 inherits every heavy layer and OOMs at any
offload level. -ts 24,16 rebalances it.
- --threads 8 (physical cores) beats 16 by 44% on generation; the expert
matmuls are bandwidth-bound and SMT siblings only contend.
Ships ncmoe=10 (53 t/s @8k, 34 t/s @97k) over the faster ncmoe=8 to keep
~1.3 GiB spare on CUDA0, which is shared with the desktop.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DATAandClaude Opus 5