All pull requests

Pull requests

• fix(unigram): preserve byte fallback during training

  #2584·original4422 opened Oct 24, 2026·

  ·
  1

• fix(tk-encode): update ptr_hash to 2.1.3, fix forked Tokenizer build

  #2581·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

  2

• feat(composition): component handles

  #2579·SBrandeis opened Oct 22, 2026Collaborator·2/4·

  ·
  1

• feat(composition): ComponentSequence

  #2577·SBrandeis opened Oct 22, 2026Collaborator·4/4·

  ·
  1

• feat(composition): post-processors

  #2573·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

• feat(composition): decoders

  #2572·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

• feat(composition): pre-tokenizers

  #2571·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

• feat(composition): normalizers

  #2570·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

• feat(composition): models and Tokenizer(model)

  #2569·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

• Respect recognized cl100k numeric group sizes

  #2568·Aurantium-47 opened Oct 22, 2026·

  ·

• Fix Kimi pretokenization for overlapping Han categories

  #2567·Aurantium-47 opened Oct 22, 2026·

  ·
  1

• feat(composition): serialization utils

  #2563·SBrandeis opened Oct 21, 2026Collaborator·3/4·

  ·
  1

• feat(composition): TokenizerSpec: JSON config as source of truth

  #2562·SBrandeis opened Oct 21, 2026Collaborator·1/4·

  ·
  1

• fix(bitcannon): backwards compatibility: use onig to generate atom_tables

  #2558·SBrandeis opened Oct 21, 2026Collaborator·

  ·
  1

  1

• Restore Python tokenizer JSON serialization for v1

  #2556·xgbtxy opened Oct 20, 2026·

  ·
  1

  1

• fix(vocab): load an empty vocab

  #2554·SBrandeis opened Oct 19, 2026Collaborator·

  ·

  · v1.0.0
  3

• feat(python): Add support for DecodeStream in Python

  #2551·SBrandeis opened Oct 19, 2026Collaborator·

  ·
  1

  1

• fix(byte-level): drop unknown bytes + warn

  #2550·SBrandeis opened Oct 19, 2026Collaborator·

  ·
  1

  1

• tk-encode, tk-serialize: harden BPE merge loading, BucketVocabStore sizing, WordPiece scratch capacity, and JSON duplicate-key order

  #2549·professor-moody opened Oct 18, 2026·

  ·

• fix(parallelism): honor RAYON_NUM_THREADS, expose set_num_threads in Python

  #2546·nov3o opened Oct 17, 2026·

  ·

• fix(decoders): Strip panics on short tokens made only of content

  #2545·spicysharkco opened Oct 16, 2026·

  ·
  4

• feat(python): update PyO3 and rust-numpy to 0.30 and test Python 3.15(t)

  #2544·ngoldbaum opened Oct 16, 2026Contributor·

  ·

• feat(python): Support Python 3.15 via abi3t wheels

  #2543·ngoldbaum opened Oct 16, 2026Contributor·

  ·
  1

• perf(encode): encode one long input on every thread, and stream a file from Python with encode_file

  #2542·nov3o opened Oct 16, 2026·

  ·
  1

• perf(bpe trainer): a small parallel trainer (a compact take on #2501)

  #2541·nov3o opened Oct 16, 2026·

  ·
  1