Raise --predict default to 32768 and document that it is not a cap
Measured: with --predict 8192 a request sending max_tokens=9000 returned 8893 tokens (finish_reason stop), while a request sending no max_tokens was cut at exactly 8192 (finish_reason length). The flag is a default for clients that omit max_tokens, not a ceiling anyone can hit. That default still matters on a -np 1 server, where an unbounded client would fill the 128k window and block the only slot for ~50 minutes. 32768 bounds that to ~12 minutes without truncating realistic long answers, which 8192 was doing silently to any client that omits max_tokens. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+5
-1
@@ -51,8 +51,12 @@ services:
|
||||
- -np
|
||||
- "1"
|
||||
- --no-mmap
|
||||
# Default only, NOT a ceiling: a client sending max_tokens overrides it
|
||||
# (verified — max_tokens=9000 returned 8893 tokens against --predict 8192).
|
||||
# It bounds clients that send no max_tokens at all, which would otherwise
|
||||
# generate until the 128k window is full: ~50 min blocking the single slot.
|
||||
- --predict
|
||||
- "8192"
|
||||
- "32768"
|
||||
- --reasoning-budget
|
||||
- "4096"
|
||||
- --slot-save-path
|
||||
|
||||
Reference in New Issue
Block a user