diff --git a/README.md b/README.md index 35c39da..091d25e 100644 --- a/README.md +++ b/README.md @@ -220,16 +220,29 @@ A client **cannot** raise it. Where `--predict` was a default anyone could overr this one is a hard server-side ceiling, so changing it means editing the compose file and restarting. -It does bite: on a moderately hard number-theory problem the thinking was guillotined -mid-derivation (`...Since $x^2 \equiv`) at exactly the budget. But the model recovered -— it produced the correct answer (empty solution set, verified by brute force) and -rebuilt the parity argument cleanly in the visible response. So 4096 truncates -reasoning without necessarily damaging the result. +### More budget did not buy better answers -Raising it does not weaken the runaway protection, since total generation is already -bounded by `--predict` / the client's `max_tokens`. For heavier math or agent -workloads, 16384 is the obvious next step; there is no measured evidence here that -4096 is actively breaking anything. +Same number-theory problem (correct answer: empty solution set, verified by brute +force), one run per setting at `temperature 0.3`: + +| budget | thinking | total tokens | wall clock | answer | +|---|---|---|---|---| +| 4096 | 4095, cut mid-derivation | 5325 | ~2 min | **correct** | +| 16384 | 16383, cut mid-**repetition-loop** | 17417 | ~6 min | **correct** | + +At 4096 the thinking is guillotined mid-derivation (`...Since $x^2 \equiv`) and the +model recovers, rebuilding the parity argument cleanly in the visible answer. At +16384 it never converges — it spends the extra ~12k tokens looping on +`Maybe it's $a^2 + b^2 = c^2 + 7$?` over and over, then answers correctly anyway. + +So the extra budget cost **3.3x the tokens and wall clock for an identical answer**, +and the degenerate loop is exactly the rambling the brake exists to cut off. The +intuition that "it was truncated, so it needs more room" did not survive measurement: +this model does not use more thinking budget productively, it fills it. + +Caveat: one problem, one sample per setting. Treat the direction as indicative, not +as a tuned value. Raising the budget does not weaken runaway protection either way, +since total generation is bounded by `--predict` / the client's `max_tokens`. ## Deploy diff --git a/docker-compose.yml b/docker-compose.yml index 1ef3758..907e2ac 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -57,8 +57,12 @@ services: # generate until the 128k window is full: ~50 min blocking the single slot. - --predict - "32768" + # Hard server-side ceiling on the thinking block — a client CANNOT raise it + # per request (verified). Note the README measurement: on the test problem + # 4096 and 16384 produced the SAME correct answer, but 16384 burned 3.3x the + # tokens looping. 4096 is defensible; revisit if real workloads disagree. - --reasoning-budget - - "4096" + - "16384" - --slot-save-path - /slots - --jinja