Ship reasoning-budget 8192 as the compromise

Third data point: 8192 also gives the correct answer, at 9041 tokens
between 4096's 5325 and 16384's 17417. All three runs hit the ceiling
exactly -- the model never terminates thinking on its own, it fills
whatever budget it gets. Only the content changes: derivation at 4096,
second-guessing at 8192, a flat repetition loop at 16384.

On this problem the budget is a pure latency dial with no measured quality
effect. 8192 leaves room to finish a derivation without paying for the spin.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-25 15:32:51 +02:00
co-authored by Claude Opus 5
parent 9daabf3223
commit e2157e9226
2 changed files with 19 additions and 12 deletions
+14 -8
View File
@@ -228,17 +228,23 @@ force), one run per setting at `temperature 0.3`:
| budget | thinking | total tokens | wall clock | answer |
|---|---|---|---|---|
| 4096 | 4095, cut mid-derivation | 5325 | ~2 min | **correct** |
| **8192 (shipped)** | 8191, cut mid-self-doubt | 9041 | ~3 min | **correct** |
| 16384 | 16383, cut mid-**repetition-loop** | 17417 | ~6 min | **correct** |
At 4096 the thinking is guillotined mid-derivation (`...Since $x^2 \equiv`) and the
model recovers, rebuilding the parity argument cleanly in the visible answer. At
16384 it never converges — it spends the extra ~12k tokens looping on
`Maybe it's $a^2 + b^2 = c^2 + 7$?` over and over, then answers correctly anyway.
**The model never stops thinking on its own — it exhausts whatever budget it is
given.** All three runs hit the ceiling exactly, and all three produced the same
correct answer. What changes is only what the thinking is spent on:
So the extra budget cost **3.3x the tokens and wall clock for an identical answer**,
and the degenerate loop is exactly the rambling the brake exists to cut off. The
intuition that "it was truncated, so it needs more room" did not survive measurement:
this model does not use more thinking budget productively, it fills it.
- 4096 — guillotined mid-derivation (`...Since $x^2 \equiv`); the model recovers and
rebuilds the parity argument cleanly in the visible answer
- 8192 — already past the proof and into second-guessing (`Let's check if I missed
any "positive integers" definition`)
- 16384 — a flat repetition loop, `Maybe it's $a^2 + b^2 = c^2 + 7$?` over and over
So on this problem the budget is a **pure latency dial with no measured quality
effect**. The intuition that "it was truncated, so it needs more room" did not
survive measurement. 8192 is shipped as a compromise: enough to finish a derivation,
short of the point where it just spins.
Caveat: one problem, one sample per setting. Treat the direction as indicative, not
as a tuned value. Raising the budget does not weaken runaway protection either way,