TorchSharp text trainers (TextClassification, SentenceSimilarity, NER, QA) drop every input character above U+00FF — non-Latin text reaches the model as nothing #7858

Summary

The tokenizer every NAS-BERT / RoBERTa trainer in Microsoft.ML.TorchSharp builds (TokenizerExtensions.GetInstance(IChannel) — an EnglishRobertaTokenizer created with unsupported characters filtered) removes every input character above U+00FF on encode, before BPE. No <unk> id is produced; the characters vanish silently, so nothing during training reports it. Affected: TextClassificationTrainer, SentenceSimilarityTrainer, NerTrainer, QATrainer.

#7749 covers the decode side of the same filter (QA answers); this is the input side.

Observed (Microsoft.ML.TorchSharp 0.23.1)

"음악을 들으면서"            -> tokens [Ġ]                       decode " "
"2026년 10월 25일 오전 9시"  -> [20 | 26 | Ġ10 | Ġ25 | Ġ | Ġ9]   decode "2026 10 25  9"
"Привет мир"                -> [Ġ]
"日本語のテキスト"            -> []
"café naïve résumé"         -> kept whole

A character-by-character check over U+0021–U+2FFF: a character is kept exactly when it is ≤ U+00FF.

Impact

On Korean news, dialogue and sentence corpora, 82–89% of non-space characters (73–81% of words) are removed before the model sees them. Text classification, sentence similarity and NER then train on the digits, Latin letters and punctuation that are left, and report ordinary-looking metrics.

Repro

var ext = typeof(Microsoft.ML.TorchSharp.NasBert.NerTrainer).Assembly
    .GetType("Microsoft.ML.TorchSharp.Extensions.TokenizerExtensions");
using var ch = ((IHostEnvironment)new MLContext()).Register("t").Start("t");
var tok = (Tokenizer)ext.GetMethod("GetInstance",
        BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public)
    .Invoke(null, new object[] { ch });
tok.Decode(tok.EncodeToIds("음악을 들으면서")); // " "

Expected

Either byte-level encoding of characters outside Latin-1 (what GPT-2/RoBERTa byte-level BPE does by design), or an explicit error/warning when input text loses characters — plus documentation that these trainers are Latin-1-only until then.

Related: #7749