Repository navigation
feat(ar): add Arabic normalization - #36
Merged
Merged
Conversation
Contributor
|
Review before undraft:
Core harakat/alef/digit path looks solid. Keep draft until those three are covered. |
jqueguiner
force-pushed
the
feat/arabic-normalization
branch
from
November 11, 2026 15:27
93c777c to
5e32243
Compare
jqueguiner
marked this pull request as ready for review
November 11, 2026 15:27
jqueguiner
approved these changes
Nov 11, 2026
jqueguiner
left a comment
jqueguiner
left a comment
Contributor
There was a problem hiding this comment.
Ready: number_words wired, Arabic comma in punct step, e2e for صفر، خمسة، واحد.
jqueguiner
approved these changes
Nov 11, 2026
jqueguiner
left a comment
jqueguiner
left a comment
Contributor
There was a problem hiding this comment.
Ready: number_words + Arabic comma + e2e.
Adds an optional character_replacements field to LanguageConfig and a step that applies it first in gladia-3 text_pre. The step is a no-op for every language that leaves the field as None, so existing outputs are unchanged.
Registers the 'ar' language with symbol and currency words, Arabic filler words and an empty word replacement table.
Vowel marks were turned into spaces by remove_symbols, splitting vocalized words into single letters (مَرْحَبًا -> م ر ح ب ا). They are now deleted up front, together with the typographic tatweel.
Folds hamzated/madda/wasla alef to bare alef, alef maqsura to yeh and ta marbuta to heh, plus Persian yeh/keheh lookalikes, so spelling variants of the same word no longer count as WER errors.
Maps Arabic-Indic and Eastern Arabic-Indic digits to 0-9 and the Arabic decimal separator, thousands separator and percent sign to their ASCII counterparts. Previously ١٢٫٥ collapsed to ١٢٥.
Once alef is folded, the plural "أمم" (nations) normalizes to "امم" and was deleted as a hesitation. "اه" is also the dialectal "yes" in Egyptian and Levantine Arabic. Only the elongated forms (اهه, اممم, ممم) are fillers now.
…ized form remove_diacritics already folded ؤ to و and ئ to ي through NFKD, but only as a side effect, so symbol words such as "زائد" and "بالمئه" did not match what the pipeline outputs and replacement keys with hamza seats could never match. The folding is now part of character_replacements, the config words are written in their final form, and a test guards that invariant.
Adds ArabicNumberNormalizer behind expand_written_numbers, so references and hypotheses that write the same number as words or digits normalize alike (خمسة وعشرون and ٢٥ both give 25). Covers MSA cardinals up to billions with gender and case variants, two-word teens, units-before-tens compounds chained with و, glued and separate hundreds, ألف/آلاف/مليون/ملايين groups, digits followed by a scale word, and common Egyptian and Levantine spellings. و is only consumed where it builds a compound, so ثلاثة وأربعة stays "3 و4" like ٣ و٤. Standalone واحد/أحد stay words, as English keeps "one". digit_words enables digit-by-digit readings. Ordinals, fractions (نص, ربع) and dual nouns are out of scope.
Wire number_words from digit vocabulary and treat U+060C like ASCII commas so ASR sequences such as صفر، خمسة fold before digit conversion. Co-authored-by: Cursor <cursoragent@cursor.com>
jqueguiner
force-pushed
the
feat/arabic-normalization
branch
from
November 11, 2026 15:27
5e32243 to
1edc137
Compare
jqueguiner
approved these changes
Nov 11, 2026
jqueguiner
left a comment
jqueguiner
left a comment
Contributor
There was a problem hiding this comment.
Re-approve after rebase onto docs merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds Arabic (
ar) normalization: it strips vowel marks (harakat) and tatweel, unifies alef / alef maqsura / ta marbuta variants, and converts Arabic-Indic digits and separators. This is driven by a new genericapply_character_replacementsstep. Previously Arabic fell back todefault, which split vocalized words into single letters (مَرْحَبًا→م ر ح ب ا) and dropped the Arabic decimal separator (١٢٫٥→١٢٥).Not covered yet: spelled-out Arabic numbers (
ثلاثة→3), and the word replacement list is still empty.Type of change
Checklist
New language
normalization/languages/{lang}/withoperators.py,replacements.py,__init__.pyreplacements.py(not hardcoded inoperators.py)LanguageConfigis filled in with the language's data (separators, currency words, digit words, …)LanguageOperators— only override methods where the logic changes, not just the data@register_languageand imported innormalization/languages/__init__.pytests/unit/languages/tests/e2e/files/{preset}/{lang}.csv(e.g.tests/e2e/files/gladia-3/fr.csv)New step
nameclass attribute set (this is the key used in YAML presets)@register_stepand imported insteps/text/__init__.pyorsteps/word/__init__.pyoperators.config.*insteadsteps/text/placeholders.pyandpipeline/base.py'svalidate()is updated (N/A)tests/unit/steps/uv run scripts/generate_step_docs.pyHow was this tested?
1093 passed. New tests:
tests/unit/steps/text/apply_character_replacements_test.py,tests/unit/languages/arabic_operators_test.py,tests/e2e/files/gladia-3/ar.csv.