SentencePieceTokenizer (Unigram) emits one <unk> per unknown character — SentencePiece merges a run of unknown characters into one <unk> #7859

Summary

SentencePieceTokenizer.Create(stream) on a SentencePiece Unigram model returns one <unk> id for every character the vocabulary lacks. The reference implementation (google/sentencepiece unigram_model.cc, "merges continuous run of unknown pieces") and HF tokenizers return a single <unk> for a run of such characters. A model trained with SentencePiece therefore receives different ids from .NET than it was trained on.

Repro (Microsoft.ML.Tokenizers 2.0.1, net10.0, osx-arm64)

1. Train a small Unigram model on a few Korean/English sentences that contain no Latin capitals:

   spm.SentencePieceTrainer.train(input=corpus, model_type="unigram", vocab_size=80,
       normalization_rule_name="nmt_nfkc", unk_id=0, bos_id=1, eos_id=2, pad_id=-1)

2. Encode "ＡＢＣ 전각 ①" (NFKC → ABC …, three characters absent from the vocabulary):
   • sentencepiece (Python), HF XLMRobertaTokenizer and XLMRobertaTokenizerFast: ▁ <unk> ▁ <unk> ▁ <unk> — one <unk> per run.
   • SentencePieceTokenizer.EncodeToIds(text, addBeginningOfSentence: false, addEndOfSentence: false): ▁ <unk> <unk> <unk> ▁ <unk> <unk> ▁ <unk> <unk> — one per character.
3. Same for "처음 보는 글자 魑魅 😀": each unseen word becomes N <unk> instead of 1.

With a large vocabulary (XLM-RoBERTa's 250k sentencepiece.bpe.model) the two paths agreed on 12 everyday texts, because unknown runs are rare there — the difference shows on rare scripts and characters.

Expected

EncodeToIds merges consecutive unknown pieces into one <unk>, as SentencePiece's Unigram encoder does. (The BPE model path may be worth checking for the same behaviour.)

Workaround

Collapsing consecutive UnknownId entries after EncodeToIds. With that, a sentence encoder's vectors match sentence-transformers to cosine 1.000000 on 14 texts.