DATAandClaude Opus 5 f628eedbaa Measure speculative decoding: all ngram modes lose to plain decoding ...

Verifying a K-token draft costs ~K times the CPU expert work, because each
token routes to its own 8-of-256 experts and nothing is amortized. The loss
is largest on structured code output (-51% for ngram-simple), i.e. exactly
where speculation should have won.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-07-25 13:54:08 +02:00

..

bench.py

llama.cpp CUDA runner for Qwen3.6-35B-A3B at 128k context

2026-07-25 13:49:33 +02:00

spec_bench.py

Measure speculative decoding: all ngram modes lose to plain decoding

2026-07-25 13:54:08 +02:00

sweep.sh

llama.cpp CUDA runner for Qwen3.6-35B-A3B at 128k context

2026-07-25 13:49:33 +02:00