Skip to content

About

Garhwali (gbm) language corpus, speech, and research resources with provenance, rights, and training-ready views.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

175 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

🏔️ Garhwali Language Lab

A provenance-first corpus and research project for Garhwali (gbm).

We organize Garhwali text, speech, folklore, scholarship, and local knowledge into reusable resources with source, rights, and quality information.

Python Tests Hugging Face corpus Hugging Face speech

About

Garhwali resources are scattered across archives, books, websites, research, and speech collections. This project brings them together and prepares reproducible text, speech, vocabulary, and evaluation resources for language research and future language tools.

Primary language identifier: garhwali:gbm (Garhwali, ISO 639-3). English (eng) appears in documentation, glosses, and translation/evaluation references; Hindi (hin) appears in bilingual glosses and some translation/evaluation material. Kumaoni (kfy) occurs in a small number of mixed-source entries, not as a standalone training subset. This is a Garhwali-focused corpus, not a balanced multilingual dataset; use row-level language and quality labels when selecting data.

Current versions

The corpus version below was published on 20 November 2026; the speech and Kaggle entries were last checked on 8 October 2026 and have not changed since. This table shows current versions only; dated release reports preserve earlier history.

Resource Current version What it contains
Hugging Face corpus v0.2.9 Mixed-source corpus with provenance and rights fields, plus purpose-labeled text views.
Hugging Face speech audio v0.2.1; training views v0.2 VAANI and Meta audio, with separate text-only indexes for existing ASR references.
Kaggle screened text v1 — public 1,841 screened training candidates and 110 short context rows.
Kaggle ASR references v2 — private 2,718 existing VAANI audio/reference pairs with train, validation, and test splits.
Kaggle ASR notebook v3 — saved, not run Baseline notebook configured to use the v2 reference dataset.

Recent changes

Corpus v0.2.9 was published to Hugging Face on 20 November 2026. It is the first release built with the pipeline fixes merged between 11 and 20 November; the speech and Kaggle releases above were built before them and are unchanged. Full figures are in the 20 November metrics page.

  • Held-out splits. Text segments now split 90/5/5 (182,611 train / 10,145 validation / 10,145 test). Whole duplicate components are moved into validation and test until each reaches 5%, and every leakage check stays at 0. Very short or very common segments, such as a lone ., no longer chain unrelated documents together.
  • Text cleaning. A danda followed directly by a letter, as in …है।अब…, now ends a segment, and Unicode normalization changes are logged per row.
  • PahariLI. Sentences already held by another source are counted and reported instead of stopping the build.
  • Hub card. Additive uploads keep the documentation for configs carried forward from earlier releases, and the generated upload command is quoted so paths with spaces work.
  • Tests and CI. CI enforces a 60% line-coverage floor; 14 unused scripts moved to scripts/archive/.

Earlier fixes, merged on 11 November 2026:

  • Language tagging. A free-text source label that mentions Garhwali only in a negated clause or as a topic no longer counts as a Garhwali label, and underscore scopes such as garhwali_focused_mixed_reference are treated as mixed. Devanagari rows labeled Garhwali but dominated by Hindi function words are kept, drop to medium confidence, and are sent to native review.
  • Segmentation. The segmenter now splits verse on the double danda (॥), keeps verse numbers with their verse, and no longer splits after Devanagari abbreviations such as डॉ.
  • Native review. Adjudication uses a majority rule, so a third reviewer settles a tie, and corrected text is compared after NFC and whitespace normalization, with dialect labels compared as sets.
  • Splits. Conflicting duplicate components are placed by a hash of their root instead of always going to train, so duplicates no longer push the 90/5/5 target toward train. Experimental ASR/TTS rows whose speaker or normalized transcript also appears in another split are flagged and counted. A missing semantic-edge file is reported and can be made a hard error.
  • Text intake. prepare_text_corpus decodes input strictly and counts invalid UTF-8.
  • Shared helpers. Thirteen scripts now import the SHA-256 helpers from garhwali_lab.hashing, and a test fails if a script redefines them.
  • Release bundle. release/versions.json names v0.2.0 as the current frozen GitHub bundle, so CI's release snapshot check now runs against release/v0.2.0 instead of v0.1.1.

Training starting points

For fresh-clone setup and runnable CPU-first text training, start at the developer quick start. Its ASR section links to a Kaggle Whisper fine-tuning notebook. Both examples use existing data and unreviewed labels; their loss and WER/CER outputs are experimental, not independent quality claims. See the developer readiness checklist for the current release boundary.

Text experiments

The corpus offers 1,841 automatically screened sentence-length candidates (47,566 whitespace-separated words) in the screened_meta_gbm configuration. These transcript values already appeared in earlier corpus views: the new configuration makes them easier to select, but adds no unique text examples. A separate 110-row short_utterances_meta_gbm configuration is for context and is not recommended for general language-model training.

from datasets import load_dataset

train = load_dataset(
    "rushilrawat/garhwali-corpus",
    "screened_meta_gbm",
    split="train",
)

Speech recognition experiments

The speech dataset contains 113,363 audio/source rows, 113,350 unique audio hashes, and about 154.65 hours. Its asr_reference index points to 2,718 existing VAANI clips and transcripts: 2,202 train / 373 validation / 143 test. Join the index to audio using source_record_id; see the speech training guide.

The optional asr_meta_extra_train index adds 2,294 train-only Meta references. Do not use it for evaluation. VAANI and Meta references are upstream labels, not native-speaker-adjudicated ground truth. SraVaani outputs are machine-generated hypotheses and are not used as reference labels.

Current scale and limits

The v0.2.9 corpus has 20 configurations and 30 config/split views holding 976,343 rows: 790,404 source/reference rows and 185,939 overlapping content-view rows. These are view counts, not unique passages or training examples. The Hub repository is 8.53 GB across all release paths. (The Dataset Viewer was still processing v0.2.9 when these figures were taken from the release manifest.) The speech page displays 118,375 rows because it totals the 113,363 audio/source rows and both text-only indexes.

Source terms vary; there is no single license for the entire corpus. Check each source and the row-level rights fields before reuse. The main catalog gap is 8,444 distinct full texts retained locally pending rights evidence (from the v0.2.7 closeout scan); they are not part of the public corpus. See the license policy, attribution guide, and current metrics page. Build-level figures for v0.2.9 (parent texts, segments, splits) are on the local candidate metrics page.

Automated screening does not verify Garhwali spelling, meaning, dialect, or transcript accuracy. The native-speaker review workflow is in place, but 0 records have been adjudicated so far (see the review guide). The benchmark remains a local draft, and independent-final evaluation eligibility is 0 of 5 task areas. Existing model scores are development results, not independent confirmation. See the model roadmap and final review.

Work with the project

Create an environment, install the developer requirements, and run the test suite:

python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements-dev.txt
.venv/bin/python -m pip install --no-deps -e .   # optional: import garhwali_lab from notebooks or other folders
.venv/bin/python -m unittest

Run commands from the repository root. Pipeline commands run directly (.venv/bin/python scripts/<name>.py) and need no PYTHONPATH.

On 20 November 2026 GitHub CI ran the full suite: 968 tests passed (1 skipped), with 62% line coverage against a 60% floor. Tests for the Kaggle, speech, and ingestion-graph builders need pyarrow and langgraph from the developer requirements and error out without them.

The requirements-*.txt files are hash-pinned lock files generated from the extras in pyproject.toml, so pip installs exactly the tested versions. After changing a dependency, regenerate them with uv pip compile pyproject.toml --extra <extra> --python-version 3.12 --universal --generate-hashes -o requirements-<name>.txt (the command is also recorded at the top of each file). External tools are found on PATH; set GARHWALI_PDFTOPPM to point at a specific pdftoppm binary.

Load and inspect the dataset with the Hugging Face Datasets library. To rebuild local Kaggle exports or refresh derived corpus views, follow the pipeline guide. These commands write local artifacts; they do not publish data.

Raw downloads, PDFs, caches, and generated data are excluded from Git. Review the PDF intake guide before adding scans.

Project guides

Contributions

Source leads, corrections, and review contributions are welcome. Please include the source and any relevant rights, attribution, dialect, or regional context.

About

Garhwali (gbm) language corpus, speech, and research resources with provenance, rights, and training-ready views.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages