DATA/llama_qwen3.6_A3B
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Measure --reasoning-budget: a real cap, unlike --predict

Browse Source 
Counted the thinking block exactly via /tokenize. Server default stops it at
4095 tokens; a client sending reasoning_budget=16384 still gets 4095, so it
cannot be raised per request. Changing it means editing the compose file.

It truncates mid-derivation on a moderately hard problem, but the model
recovered and answered correctly (empty solution set, verified by brute
force), so there is no evidence 4096 is actually damaging output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
DATA
2026-07-25 15:03:49 +02:00
co-authored by Claude Opus 5
parent e41dc2ba6f
commit 09230bc499
2 changed files with 92 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+24 -2
View File
@@ -206,8 +206,30 @@ were silently cut off with `finish_reason: length`.
Note the sibling `gemma4-26b-llama-runner` README describes `--predict` as a "harte
Cap fuer Gesamt-Generierung" — by this measurement that is wrong there too.
`--reasoning-budget 4096` is a separate knob for the thinking block and was **not**
verified the same way; do not assume its semantics from the flag name either.
## `--reasoning-budget` dagegen ist ein echter Deckel
Same flag family, opposite semantics — measured with `bench/reasoning_budget_probe.py`,
which counts the thinking block exactly via the server's `/tokenize` endpoint:
| Request | Reasoning tokens | Answer |
|---|---|---|
| server default | **4095** | 1227 tok, complete |
| `reasoning_budget: 16384` in the request body | **4095** | 531 tok, complete |
A client **cannot** raise it. Where `--predict` was a default anyone could override,
this one is a hard server-side ceiling, so changing it means editing the compose file
and restarting.
It does bite: on a moderately hard number-theory problem the thinking was guillotined
mid-derivation (`...Since $x^2 \equiv`) at exactly the budget. But the model recovered
— it produced the correct answer (empty solution set, verified by brute force) and
rebuilt the parity argument cleanly in the visible response. So 4096 truncates
reasoning without necessarily damaging the result.
Raising it does not weaken the runaway protection, since total generation is already
bounded by `--predict` / the client's `max_tokens`. For heavier math or agent
workloads, 16384 is the obvious next step; there is no measured evidence here that
4096 is actively breaking anything.
## Deploy