A clean, modular, and educational Retrieval-Augmented Generation (RAG) engine built from the ground up — without restrictive third-party framework lock-ins, black-box abstractions, or bloated dependencies.
Large Language Models (LLMs) excel at reasoning but suffer from temporal knowledge cutoffs and hallucination. This repository demonstrates how a production-ready RAG system solves these limitations by turning open-ended generation into an open-book synthesis problem:
- Retrieve: Extract the most contextually relevant passages using Hybrid Search (combining Dense Semantic Vectors with Sparse Lexical BM25).
- Fuse: Rank and synthesize multi-modal retrieval candidates using Reciprocal Rank Fusion (RRF).
- Augment: Ground LLM synthesis strictly on retrieved evidence with formatted source citations.
- Evaluate: Score the output automatically for Faithfulness (factual grounding) and Answer Relevancy.
flowchart TD
subgraph Ingestion["1. Document Ingestion & Chunking"]
A["Raw Knowledge Corpus<br/>(Markdown, JSONL, Text)"] --> B["Document Loader & Normalizer"]
B --> C["Sliding-Window Chunking<br/>(Overlapping Boundaries)"]
end
subgraph Indexing["2. Dual Hybrid Indexing"]
C --> D["Dense Vector Index<br/>(Cosine Semantic Embeddings)"]
C --> E["Sparse Lexical Index<br/>(Okapi BM25 Term Matching)"]
end
subgraph Retrieval["3. Hybrid Retrieval & Rank Fusion"]
Q["User Query"] --> F["Vector Similarity Search"]
Q --> G["BM25 Keyword Search"]
D -.-> F
E -.-> G
F --> H["Reciprocal Rank Fusion (RRF)<br/>Score = Σ w / (k + rank)"]
G --> H
H --> I["Top-K Ranked Passages"]
end
subgraph Generation["4. Synthesis & Evaluation"]
I --> J["Context Prompt Injector"]
Q --> J
J --> K["OpenAI LLM / Offline Extractive Fallback"]
K --> L["Cited Grounded Response"]
L --> M["Automated Evaluation<br/>(Faithfulness & Relevancy)"]
end
style Ingestion fill:#0d1117,stroke:#00d9ff,stroke-width:2px,color:#ffffff
style Indexing fill:#0d1117,stroke:#34d399,stroke-width:2px,color:#ffffff
style Retrieval fill:#0d1117,stroke:#7c3aed,stroke-width:2px,color:#ffffff
style Generation fill:#0d1117,stroke:#fbbf24,stroke-width:2px,color:#ffffff
Embedding entire documents into single vectors dilutes granular details. We decompose documents into focused passages (e.g., 300 words) with overlapping spans (50 words) to guarantee conceptual continuity across chunk boundaries.
- Dense Semantic Embeddings: Captures conceptual affinity (e.g.,
"climate variation"matches"global warming"). Includes zero-download offline hash embeddings and optional SentenceTransformers. - Sparse Okapi BM25: Excels at verbatim keywords, technical acronyms, serial numbers, and exact entities (e.g.,
NIRCam,15 millikelvin,ISO-8601) where pure vector models often struggle.
To unify rankings without requiring brittle calibration of raw vector distances and BM25 scores, we combine rankings using Reciprocal Rank Fusion:
If an OPENAI_API_KEY is provided, the engine utilizes GPT models to synthesize fluent answers. When running locally without API credentials, the pipeline gracefully defaults to an offline extractive synthesis engine, making the entire codebase 100% free and runnable offline.
- Faithfulness: Measures whether all assertions made in the generated answer are strictly substantiated by the retrieved context.
- Answer Relevancy: Verifies that the synthesized answer directly addresses the user query.
git clone https://github.com/JayanGupta/RAG-Engine.git
cd RAG-Enginepython -m venv venv
# On Windows:
.\venv\Scripts\activate
# On Linux/macOS:
source venv/bin/activate
pip install -r requirements.txtCopy the template configuration:
cp .env.example .env(Optional: Insert your OPENAI_API_KEY in .env for generative answers. If left empty, the engine uses local extractive synthesis).
streamlit run app.pyOpen http://localhost:8501 in your browser to interact with the visual dashboard, adjust retrieval weights, and inspect chunk rankings.
from src.rag_engine.pipeline import RAGPipeline
from src.rag_engine.documents import load_documents_from_jsonl
# 1. Initialize Pipeline & Ingest Knowledge Base
pipeline = RAGPipeline()
docs = load_documents_from_jsonl("data/documents.jsonl")
pipeline.index_documents(docs)
# 2. Query the Hybrid Search Engine
query = "What temperature must superconducting qubits be maintained at?"
result = pipeline.query(query, top_k=3)
# 3. Inspect Response, Citations & Metrics
print("Generated Answer:\n", result.answer)
print("\nSources Cited:")
for source in result.sources:
print(f"- {source.title} (RRF Score: {source.score:.4f})")
print(f"\nFaithfulness: {result.metrics['faithfulness']:.2f}")
print(f"Relevancy: {result.metrics['relevancy']:.2f}")
Run unit and integration tests across chunking, dual indexing, BM25, and hybrid fusion:
# Run via pytest
pytest tests/ -v
# Or use the standalone runner
python run_tests.py
RAG-Engine/
|-- app.py # Streamlit visual interactive application
|-- pyproject.toml # Project metadata and tool configurations
|-- requirements.txt # Python dependencies
|-- run_tests.py # Standalone test runner script
|-- start.sh # Container entrypoint script
|-- README.md # Master repository documentation
|-- .env.example # Template environment configuration
|
|-- config/ # System configuration & prompts
| |-- settings.py # Search weights, chunk sizes & defaults
| `-- prompts.py # LLM system & context augmentation prompts
|
|-- data/ # Demo corpus and benchmark knowledge bases
| `-- documents.jsonl # Multi-domain articles (Science, Tech, Quantum)
|
|-- experiment/ # Research & educational scratchpads
| `-- rag_pipeline.ipynb # Interactive Jupyter step-by-step notebook
|
|-- src/rag_engine/ # Core Modular RAG Engine
| |-- chunking.py # Overlapping sliding window text splitter
| |-- documents.py # Document abstractions & serializers
| |-- embeddings.py # Dense vector embedding adapters
| |-- vector_store.py # In-memory cosine similarity store
| |-- bm25_search.py # Okapi BM25 sparse keyword search engine
| |-- hybrid_search.py # Reciprocal Rank Fusion (RRF) algorithm
| |-- generator.py # LLM and local extractive synthesis
| |-- eval.py # Faithfulness & relevancy metrics
| `-- pipeline.py # Unified end-to-end orchestration pipeline
|
`-- tests/ # Complete Pytest test suite
|-- test_chunking.py # Chunk split & overlap unit tests
|-- test_documents.py # Document validation tests
|-- test_embeddings_and_vector_store.py# Cosine similarity verification
|-- test_bm25_and_hybrid.py # BM25 & RRF fusion scoring tests
`-- test_pipeline.py # End-to-end integration tests
Contributions, bug fixes, and suggestions are welcome:
- Fork the Project (
https://github.com/JayanGupta/RAG-Engine/fork) - Create your Feature Branch (
git checkout -b feature/AdvancedReranker) - Commit your Changes (
git commit -m "feat: add cross-encoder reranker support") - Push to the Branch (
git push origin feature/AdvancedReranker) - Open a Pull Request
Give a star if this repository helped your understanding of Retrieval-Augmented Generation!
