Raise --predict default to 32768 and document that it is not a cap

Measured: with --predict 8192 a request sending max_tokens=9000 returned
8893 tokens (finish_reason stop), while a request sending no max_tokens was
cut at exactly 8192 (finish_reason length). The flag is a default for
clients that omit max_tokens, not a ceiling anyone can hit.

That default still matters on a -np 1 server, where an unbounded client
would fill the 128k window and block the only slot for ~50 minutes. 32768
bounds that to ~12 minutes without truncating realistic long answers, which
8192 was doing silently to any client that omits max_tokens.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-25 14:53:03 +02:00
co-authored by Claude Opus 5
parent e9a091c35f
commit e41dc2ba6f
2 changed files with 28 additions and 1 deletions
+5 -1
View File
@@ -51,8 +51,12 @@ services:
- -np
- "1"
- --no-mmap
# Default only, NOT a ceiling: a client sending max_tokens overrides it
# (verified — max_tokens=9000 returned 8893 tokens against --predict 8192).
# It bounds clients that send no max_tokens at all, which would otherwise
# generate until the 128k window is full: ~50 min blocking the single slot.
- --predict
- "8192"
- "32768"
- --reasoning-budget
- "4096"
- --slot-save-path