chunking-algorithm

🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines

🦛 CHONK your texts with Chonkie ✨ Type-friendly, light-weight, fast and super-simple chunking library

FastCDC implementation in Rust

Open-source toolkit for reliable RAG pipelines: convert PDFs to Markdown, clean documents, inspect chunks, compare chunking strategies, and enrich metadata for LLM applications.

FastCDC implementation in Python https://pypi.org/project/fastcdc/

Structure-aware semantic chunking toolkit for RAG pipelines with semantic boundaries, metadata preservation, deduplication, and CLI/package tooling.

Android Resumable Uploads SDK from Fastpix

Chunkpad is designed to prepare documents for Retrieval-Augmented Generation (RAG) pipelines and AI applications.

Go implementation of the AE chunking algorithm.

Structure-aware document splitter for RAG — parses any document into a unified AST and chunks by section/block boundaries. 面向 RAG 的结构感知文档分割器——将任何文档解析为统一的 AST，并按节/块边界进行分割。

Chunk Skills: Frontmatter; Index of markdown headers 2 nodes; recompose with PageIndex natural page form

A high-performance FastCDC 2020 implementation written in Python + Cython

Chunk MCP tools: Tool with required properties; optional Properties and Enums

A Rust library for Content-Defined Chunking (CDC).

A nodejs chunking system

Implementation of an interactive chatbot for summarizing legal and policy documents. Includes data preprocessing (cleaning, tokenization, hierarchical chunking), extractive TF-IDF baselines, and fine-tuned abstractive models (DistilBART, LED). Integrates a retrieval layer for document relevance and uses ROUGE, BLEU, and cosine similarity metrics.

Detailed overview on chunking

[2024-2] AI 기반 회의 지원 플랫폼 서비스 "Clerker"

Cross-industry RAG benchmarking platform, 10 chunking strategies, 3 retrieval modes (BM25/Vector/Hybrid RRF), 5-node LangGraph agent, NDCG@K · MRR · MAP · Ragas · adversarial robustness evaluation across healthcare, finance & legal domains

Explore and benchmark the world of data chunking algorithms in 'ChunkingChampions' - a competitive arena to determine the most efficient and effective chunking strategies for varied data sizes.