You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix: honor tokenizer turn-end EOS in batched HTTP serving #229
serve_batched.factory passes only model.generation_config.eos_token_id to BatchedService. A chat checkpoint can declare a different tokenizer EOS, causing generated turn-end tokens to be ignored.
Evidence
KONI-Llama3.1-8B-Instruct-20241024 generation_config EOS=128001, tokenizer EOS=128009 (<|eot_id|>). #227 direct GPU run continued into synthetic dialogue after answering; unioning both EOS IDs stopped at the actual turn end. Current factory has the same config-only selection.
Scope
Normalize the union of configured and tokenizer EOS at the shared HTTP service factory; preserve explicit ignore_eos. Add regression tests and a real KONI HTTP stop/reuse check. No model weights, SDK or tokenizer files changed.
Done when
Config-only baseline reproduces the missed turn end; factory regression fails before/passes after; EOS stop and ignore_eos behavior are tested, with hardware HTTP evidence where available.
Activity
serve: no SIGPIPE on disconnect, tokenizer EOS, producer cache flags (#…
gspark-etri commentedon Oct 30, 2026
Fixed by #281, squash-merged into optimus-rm.