Skip to content

Establish the published 4.8.1 Python text API contract - #177

Merged
sidmohan0 merged 2 commits into
devfrom
codex/datafog-481-contract
Sep 28, 2026
Merged

sidmohan0 merged 2 commits into
devfrom
codex/datafog-481-contract

Conversation

@sidmohan0

@sidmohan0 sidmohan0 commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

The Rust migration needs a reproducible baseline of what the released Python package actually does. This freezes 111 black-box observations from the published datafog==4.8.1 wheel and checks the development checkout against them without regenerating expectations.

The cases cover structured synchronous detection, Unicode offsets, selection and aliases, allowlists, German locale activation, explicit-span overlap handling, all legacy transformation strategies, result shapes, selected signatures, warnings/errors, convenience APIs, guardrails, and regex TextService calls. The migration guide separates 102 compatibility cases from nine observations requiring review, including invalid-checksum cards and mismatched explicit span text. ML, OCR, Spark, CLI, async, and application-adapter acceptance remain explicitly outside this baseline's scope.

The capture tool requires isolated imports, verifies the pinned PyPI wheel SHA-256 and installed package bytes, and writes only a separate candidate file. The existing CI test discovery automatically includes the contract suite. Detection and transformation behavior are unchanged. The nlp-advanced extra now explicitly includes SentencePiece and protobuf so the multilingual GLiNER tokenizer can load without the OCR extra. Its install-profile smoke test also checks that the tokenizer protobuf schema loads.

Validation:

  • Published wheel: 111/111 observations matched; independent recapture reproduced the full fixture.
  • Checkout contract: 111 passed on Python 3.10, 3.12, and 3.14.
  • Contract plus engine, bridge, allowlist, German PII, agent, and telemetry suites: 260 passed.
  • Altered-wheel rejection and deliberate mismatch detection verified.
  • All applicable configured pre-commit hooks passed.
  • Reproduced the missing-protobuf error with Transformers 5.16.1, then loaded and exercised the actual microsoft/mdeberta-v3-base tokenizer after installing protobuf.

@CLAassistant

CLAassistant commented Sep 28, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@sidmohan0
sidmohan0 merged commit 082a92a into dev Sep 28, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants