From e41dc2ba6fb06ee54ed85b36d7772d7bdef96266 Mon Sep 17 00:00:00 2001 From: DATA Date: Sat, 25 Jul 2026 14:53:03 +0200 Subject: [PATCH] Raise --predict default to 32768 and document that it is not a cap Measured: with --predict 8192 a request sending max_tokens=9000 returned 8893 tokens (finish_reason stop), while a request sending no max_tokens was cut at exactly 8192 (finish_reason length). The flag is a default for clients that omit max_tokens, not a ceiling anyone can hit. That default still matters on a -np 1 server, where an unbounded client would fill the 128k window and block the only slot for ~50 minutes. 32768 bounds that to ~12 minutes without truncating realistic long answers, which 8192 was doing silently to any client that omits max_tokens. Co-Authored-By: Claude Opus 5 --- README.md | 23 +++++++++++++++++++++++ docker-compose.yml | 6 +++++- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 1151690..59bfe4c 100644 --- a/README.md +++ b/README.md @@ -186,6 +186,29 @@ purely from desktop compositor churn — a ~200 MiB swing with nothing else chan That is why the default keeps >1 GiB spare there rather than the 388 MiB that `ncmoe=8` leaves. +## `--predict` ist ein Default, kein Deckel + +Despite the name, `--predict` does **not** cap what a client can ask for. Measured +against `--predict 8192`: + +| Request | Result | +|---|---| +| `max_tokens: 9000` | **8893** tokens, `finish_reason: stop` — the flag is ignored | +| no `max_tokens` | **8192** tokens, `finish_reason: length` — the flag applies | + +So it only binds clients that send no `max_tokens` at all. That is still worth having +here: without it such a client generates until the 128k window is full, which at +~40 t/s is **roughly 50 minutes**, and with `-np 1` the single slot is blocked for all +of it. Set to `32768` — a runaway is bounded to ~12 minutes, while no realistic long +answer gets truncated. Many OpenAI clients do not send `max_tokens`, and at 8192 they +were silently cut off with `finish_reason: length`. + +Note the sibling `gemma4-26b-llama-runner` README describes `--predict` as a "harte +Cap fuer Gesamt-Generierung" — by this measurement that is wrong there too. + +`--reasoning-budget 4096` is a separate knob for the thinking block and was **not** +verified the same way; do not assume its semantics from the flag name either. + ## Deploy ```bash diff --git a/docker-compose.yml b/docker-compose.yml index 19495cc..1ef3758 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -51,8 +51,12 @@ services: - -np - "1" - --no-mmap + # Default only, NOT a ceiling: a client sending max_tokens overrides it + # (verified — max_tokens=9000 returned 8893 tokens against --predict 8192). + # It bounds clients that send no max_tokens at all, which would otherwise + # generate until the 128k window is full: ~50 min blocking the single slot. - --predict - - "8192" + - "32768" - --reasoning-budget - "4096" - --slot-save-path