Skip to content

fix: honor tokenizer turn-end EOS in batched HTTP serving #229

Closed

Description

@gspark-etri

Problem

serve_batched.factory passes only model.generation_config.eos_token_id to BatchedService. A chat checkpoint can declare a different tokenizer EOS, causing generated turn-end tokens to be ignored.

Evidence

KONI-Llama3.1-8B-Instruct-20241024 generation_config EOS=128001, tokenizer EOS=128009 (<|eot_id|>). #227 direct GPU run continued into synthetic dialogue after answering; unioning both EOS IDs stopped at the actual turn end. Current factory has the same config-only selection.

Scope

Normalize the union of configured and tokenizer EOS at the shared HTTP service factory; preserve explicit ignore_eos. Add regression tests and a real KONI HTTP stop/reuse check. No model weights, SDK or tokenizer files changed.

Done when

Config-only baseline reproduces the missed turn end; factory regression fails before/passes after; EOS stop and ignore_eos behavior are tested, with hardware HTTP evidence where available.

Owner

Codex. Related #201, #227 and #211.

Activity

  1. gspark-etri commented on Oct 30, 2026

    @gspark-etri
    CollaboratorAuthor

    Fixed by #281, squash-merged into optimus-rm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions