diff --git a/README.md b/README.md index 1151690..59bfe4c 100644 --- a/README.md +++ b/README.md @@ -186,6 +186,29 @@ purely from desktop compositor churn — a ~200 MiB swing with nothing else chan That is why the default keeps >1 GiB spare there rather than the 388 MiB that `ncmoe=8` leaves. +## `--predict` ist ein Default, kein Deckel + +Despite the name, `--predict` does **not** cap what a client can ask for. Measured +against `--predict 8192`: + +| Request | Result | +|---|---| +| `max_tokens: 9000` | **8893** tokens, `finish_reason: stop` — the flag is ignored | +| no `max_tokens` | **8192** tokens, `finish_reason: length` — the flag applies | + +So it only binds clients that send no `max_tokens` at all. That is still worth having +here: without it such a client generates until the 128k window is full, which at +~40 t/s is **roughly 50 minutes**, and with `-np 1` the single slot is blocked for all +of it. Set to `32768` — a runaway is bounded to ~12 minutes, while no realistic long +answer gets truncated. Many OpenAI clients do not send `max_tokens`, and at 8192 they +were silently cut off with `finish_reason: length`. + +Note the sibling `gemma4-26b-llama-runner` README describes `--predict` as a "harte +Cap fuer Gesamt-Generierung" — by this measurement that is wrong there too. + +`--reasoning-budget 4096` is a separate knob for the thinking block and was **not** +verified the same way; do not assume its semantics from the flag name either. + ## Deploy ```bash diff --git a/docker-compose.yml b/docker-compose.yml index 19495cc..1ef3758 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -51,8 +51,12 @@ services: - -np - "1" - --no-mmap + # Default only, NOT a ceiling: a client sending max_tokens overrides it + # (verified — max_tokens=9000 returned 8893 tokens against --predict 8192). + # It bounds clients that send no max_tokens at all, which would otherwise + # generate until the 128k window is full: ~50 min blocking the single slot. - --predict - - "8192" + - "32768" - --reasoning-budget - "4096" - --slot-save-path