Commit Graph
1 Commits
Author SHA1 Message Date
DATAandClaude Opus 5 f628eedbaa Measure speculative decoding: all ngram modes lose to plain decoding
Verifying a K-token draft costs ~K times the CPU expert work, because each
token routes to its own 8-of-256 experts and nothing is amortized. The loss
is largest on structured code output (-51% for ngram-simple), i.e. exactly
where speculation should have won.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-25 13:54:08 +02:00