Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ Pipelines are defined declaratively in **YAML presets**. Each preset lists the s

| Code | Language |
| ---- | -------- |
| `ar` | Arabic |
| `da` | Danish |
| `de` | German |
| `en` | English |
Expand Down
10 changes: 10 additions & 0 deletions docs/reference/steps.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,16 @@ Auto-generated — do not edit by hand. Run `scripts/generate_step_docs.py` to u

## Text Steps

### `apply_character_replacements`

**Base class:** `TextStep`

Replace or delete single characters using ``character_replacements`` from the language config.

Runs first in the pipeline so that script-level variants (letter forms, native
digits, vowel marks) are unified before protection, numeric, and symbol steps.
Skipped when the language does not define ``character_replacements``.

### `apply_sentence_level_replacements`

**Base class:** `TextStep`
Expand Down
2 changes: 2 additions & 0 deletions normalization/languages/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
from . import (
arabic,
danish,
dutch,
english,
Expand All @@ -18,6 +19,7 @@
register_language(LanguageOperators)

__all__ = [
"arabic",
"danish",
"dutch",
"english",
Expand Down
7 changes: 7 additions & 0 deletions normalization/languages/arabic/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
from .operators import ArabicOperators
from .replacements import ARABIC_REPLACEMENTS

__all__ = [
"ArabicOperators",
"ARABIC_REPLACEMENTS",
]
Loading
Loading