A local code intelligence engine that indexes Python, TypeScript, JavaScript, C#, C++ and Rockwell L5X (PLC) projects into a hybrid semantic search system, then exposes query capabilities through a Model Context Protocol (MCP) server for use with AI assistants.
Instead of grepping for strings, the Codebase Indexer:
- Parses source code with tree-sitter to extract real AST symbols (functions, classes, interfaces)
- Embeds every symbol at three granularities using
BAAI/bge-code-v1 - Stores embeddings in FAISS and symbol relationships in SQLite
- Serves 14 AI-facing MCP tools so any compatible assistant (Claude Code, Continue.dev, etc.) can query it
Splitting files every N tokens or at blank lines cuts function bodies in half, strands a method's doc comment in a different chunk from its code, and leaves the embedder unable to see where one symbol ends and the next begins. This indexer parses real AST symbols instead: each function or method is its own chunk, a class is carved into a skeleton (fields and signatures, with bodies stubbed out) plus one chunk per member carrying its own docs, and calls between symbols are recorded as graph edges the retriever can follow. That structure is measurable: moving each member's doc comment into its own chunk (ADR-034) raised MRR@10 on one TypeScript repo from 0.554 to 0.709, and by +0.39 on a second repo the design was never tuned on. Whole-file slices are still kept as tiers 2 and 3, for questions about a file as a whole.
Built to run on one 8 GB consumer GPU, measured on an 8 GB RTX PRO 1000 laptop card:
- Embedder:
bge-code-v1loads in bf16 on a GPU that supports it, about 2.9 GB, instead of 6.2 GB in fp32 (ADR-035). - Summarizer:
Qwen2.5-Coder-1.5Bin fp16, batched under a self-imposed GPU memory cap with a 1 GB reserve. It sizes batches by a token budget and backs off rather than spill into system RAM, where Windows would run it about 50 times slower without an error (ADR-027). - One model on the card at a time: indexing summarizes every chunk first, unloads the summarizer, then embeds, so peak memory is the larger model rather than both (about 5.8 GB during summarization).
- No GPU? Everything runs on the CPU, just slower;
CODE_INDEXER_DEVICE=cpuforces it (see below).
flowchart LR
A[Source files] --> B["Language adapters<br/>Python · TS/JS · C# · C++ · L5X"]
B --> C["AST chunker<br/>tier 1: one chunk per symbol,<br/>class skeletons + members<br/>tiers 2-3: whole-file slices"]
B --> D[("SQLite<br/>files · symbols · edges · chunks")]
C --> D
C --> E["Summarizer (optional)<br/>Qwen2.5-Coder-1.5B"]
C --> F["Embedder<br/>bge-code-v1"]
E --> G[("summary.faiss")]
F --> H[("tier1 / tier2 / tier3 .faiss")]
D --> I["Hybrid retriever<br/>RRF fusion +<br/>call-graph expansion"]
G --> I
H --> I
I --> J["MCP server<br/>semantic_code_search, find_dead_code, ..."]
Every symbol is embedded at three chunk sizes, each stored in a separate FAISS index under .code-index/:
| Tier | Token budget | FAISS file | Granularity |
|---|---|---|---|
| 1 — Surgical | ~500 | tier1_surgical.faiss |
One chunk per AST symbol |
| 2 — Component | ~1,500 | tier2_component.faiss |
Module-level sliding window |
| 3 — Architectural | ~4,000 | tier3_architectural.faiss |
System-level sliding window |
At query time, all three tiers are searched and results are fused with Reciprocal Rank Fusion (RRF) for consensus ranking.
Source files
→ ast_chunker.py (tree-sitter AST → symbols + edges + references)
→ incremental_indexer.py (content-hash change detection → stale removals + new embeddings)
→ core.py (bge-code-v1 embeddings, FAISS, token counting)
→ db.py (SQLite: files, symbols, chunks, edges, symbol_references)
→ MCPServer.py (MCP tools over the combined index)
hybrid_retriever.py implements a three-step Retrieve → Traverse → Rerank pipeline:
- Semantic — FAISS top-50 candidates from tier-1
- Structural — One-hop call-graph expansion via SQLite (bidirectional CTEs, cycle-guarded)
- Reranking —
Qwen/Qwen3-Reranker-0.6Bcross-encoder. Off by default ([reranker].enabled = false); when disabled the fused retrieval scores stand.
iterative_retriever.py wraps this in multi-round loops with confidence-based early stopping and query enrichment from prior findings.
On the reranker, honestly. It is a supported opt-in, not a recommended default, and the research thread behind it is closed. Three reasons it stays off: on CPU it costs ~90 s/query against 0.3–2 s without it; the last measurement predates the current embedder, and on Python and C# plain dense retrieval with
bge-code-v1now scores above the old embedder with reranking; and when these tools are consumed over MCP the calling agent re-ranks the results anyway. Turn it on if you have a GPU and a Python-heavy repo — it measured a real lift there (+0.187 mrr@10 on contamination-free fixtures) — but expect nothing for free. Full reasoning in ADR-009 §P4.
From the repository root, into the Python environment that will run the indexer (for a GPU, one with a CUDA build of PyTorch):
pip install -e . # installs the dependencies and two commands: code-indexer, code-indexer-servepip install -r requirements.txt installs the same dependencies without the commands. Two pins
matter: mcp[cli]<2 (2.x removed the FastMCP server this uses, B-036) and accelerate (without it
the summarizer can't load, and summaries are skipped).
Optional:
pip install leidenalg python-igraphenables the higher-quality Leiden community-detection backend formap_module_communities. Without it the engine uses NetworkX's built-in Louvain — no functionality is lost.
Run the first build from a terminal, in the project's root folder:
code-indexer # or: python /path/to/indexer/src/incremental_indexer.pyExpect a long first run. The first use downloads the models (about 3 GB for the embedder plus the summarizer). The build then summarizes every chunk and embeds it: about 17.6 min for a 120-file repository on an 8 GB laptop GPU, and roughly an hour for a 500-file one. Much slower on a CPU. Later runs re-process only changed files, and summaries are cached by chunk text. The build ends "Done successfully", or "Done with warnings" when summaries were skipped.
The index goes in .code-index/ in the project folder. Configure it per project with an
indexer.toml in the project root (see this repository's own indexer.toml for every key).
With parallel branches or git worktrees, index one shared ref instead of whatever is checked out (ADR-042):
[indexer]
source = "git:origin/main"The index then lives in <git common dir>/code-index, shared by every worktree of the repository,
and follows the ref: after a git fetch moves it, the running server reindexes only the files that
changed. Edits and branch switches don't touch it, so read the files your branch changed directly.
A local symbolic ref works as a movable alias for a series of dated branches:
git symbolic-ref refs/code-index/target refs/remotes/origin/<branch>, then
source = "git:refs/code-index/target".
code-indexer-serve # or: python /path/to/indexer/src/MCPServer.pyRun it with the project root as the working directory (MCP clients do this for you). The server starts instantly (indexes load lazily on first tool call) and starts a file watchdog that reindexes changed files in the background. The watchdog never starts a first build: until the terminal build above has finished, tools report that the index is empty.
Only one server per index writes to it and runs the watchdog; other sessions on the same index serve read-only tools (ADR-038).
On a CUDA machine the models run in one shared background process, the model host
(ADR-028), which keeps one model on the card at a time
for every project and session. It starts on first use. [model_host] enabled = "auto" (the
default) turns it on when the models run on CUDA; false loads them in each process instead.
By default every model auto-selects CUDA when a GPU is present. To force the entire local stack (embedder, reranker, summarizer, and eval) onto the CPU — e.g. on a machine where GPU use is unstable — set one environment variable:
CODE_INDEXER_DEVICE=cpu # authoritative override for all local model loads (ADR-020/ADR-024)
This is the supported force-CPU control. It is resolved by src/device.py (resolve_device())
and honoured by all four components, so a CPU-only index run is CPU-safe by construction. Any
other value (cuda, mps, …) is passed through as an explicit device choice; unset means
auto-CUDA-else-CPU.
Add to your MCP config (~/.claude/claude_mcp_config.json or .mcp.json in your project):
{
"mcpServers": {
"codebase-indexer": {
"command": "python",
"args": ["/absolute/path/to/indexer/src/MCPServer.py"]
}
}
}Use the interpreter the indexer is installed in (the absolute path to its python, if it isn't the
first python on PATH): a CPU-only interpreter puts the models on the CPU.
| Tool | Category | Description |
|---|---|---|
semantic_code_search |
Search | FAISS + RRF search across all three tiers |
find_similar_code |
Search | Find duplicates and callers of a code snippet |
analyze_blast_radius |
Impact | Map callers, dependents, and parallel implementations of a symbol |
detect_pattern_violations |
Impact | Find files that should follow a pattern but don't |
trace_data_flow |
Tracing | Full lifecycle of a data symbol (definitions → producers → consumers) |
investigate_architecture |
Tracing | Full RTR pipeline with Markdown report and architectural risk analysis |
find_test_coverage |
Discovery | Locate test files (Vitest/Jest and pytest) covering a source file or symbol |
find_dead_code |
Discovery | Determine if a symbol has any consumers in the codebase |
verify_candidate_edges |
Discovery | Check the unconfirmed references behind an ADVISORY or INSUFFICIENT verdict |
find_unabstracted_collection_reads |
Discovery | Enforce "reads of X must go through Y" patterns |
map_module_communities |
Discovery | Whole-graph structural view: community detection + god-object analysis + a DSM visualization |
what_writes |
PLC | Every rung that writes a tag, in an L5X project |
index_status |
Maintenance | What the index covers: its source, commit, counts and recently changed files |
reindex |
Maintenance | Rebuild from the index's source; rarely needed, since the watchdog keeps the index current |
Where the other tools answer questions about a single symbol, map_module_communities
analyzes the shape of the whole graph. It reads the stored symbol/edge graph and runs
Louvain community detection + betweenness centrality over it (pure
NetworkX — no LLM, deterministic under a fixed seed), then returns:
- a community map — natural module clusters, each with a heuristic label and a raw cohesion score;
- god-objects — high-coupling chokepoints whose owned members sprawl across many coupling communities (the "this class has grown too large" signal), ranked by a composite score;
- a Design Structure Matrix (DSM) written to
.code-index/architecture_matrix.html— a self-contained, single-file interactive matrix (no web server, no CDN) where god-objects show up as dense rows/columns and healthy communities as bright diagonal blocks.
map_module_communities() # descriptive report + DSM (default)
map_module_communities(target_path="src/") # scope analysis to a subtree
map_module_communities(suggest_splits=True) # also emit heuristic module-split proposals
suggest_splits=True additionally proposes how an over-large class might decompose, grouping
its members by coupling community. These proposals are descriptive heuristics, not
recommendations — every one is stamped [HEURISTIC — unverified] and should be validated
against the real call graph before acting. The whole report is explicitly exploratory: it
maps the EXTRACTED graph honestly but is not a verified accuracy claim (a measured quality bar
is tracked separately).
Attribution. This capability is inspired by Graphify (Safi Shamsi, MIT) — specifically its demonstration of community detection + betweenness "god-node" analysis over a code graph. It is reimplemented natively over this engine's own EXTRACTED edges (no Graphify source is used), and the DSM is a deliberately different visual idiom from Graphify's force-directed node-link graph. See
docs/adr/ADR-006.
codebase-indexer/
├── indexer.toml # Per-repo configuration (embedder, retrieval, eval, ignores)
├── pyproject.toml # Package, dependencies and the two commands
├── requirements.txt
├── benchmarks/ # Eval deliverables: baselines, qrels, fixtures (ADR-007/019)
├── cloud/ # GPU eval VM launcher + startup script
├── docs/
│ ├── ARCHITECTURE.md # Full architecture walkthrough
│ ├── adr/ # Architecture Decision Records — the living source of truth
│ └── adr-backlog.md # Proposed/queued decisions
├── examples/ # Sample per-language rule configs
├── scripts/ # Benchmark launchers (run from the repo root)
├── tests/
├── tools/ # Eval + CI harnesses (coir_eval, conformance_eval, …)
└── src/
├── MCPServer.py # MCP server entry point — all 14 tools, and the watchdog
├── config.py # Locates and parses the per-repo indexer.toml
├── index_location.py # Where the index lives: .code-index or <git dir>/code-index (ADR-042)
├── index_lock.py # One writer and one watchdog per index (ADR-038)
├── source.py # Reads files from the folder or from a git commit (ADR-042)
├── model_host.py # Shared background process that runs the models (ADR-028)
├── model_client.py # Talks to the model host, or loads models in-process
├── device.py # CPU/CUDA resolution (ADR-020)
├── core.py # Embeddings (bge-code-v1), FAISS management, token counting
├── ast_chunker.py # tree-sitter AST → symbols, edges, chunks
├── adapters/ # Per-language extraction adapters (ADR-003)
│ ├── base.py # Adapter interface + shared tree-sitter helpers
│ └── … # python, ts, csharp, cpp, l5x
├── scan_policy.py # Which files are indexed (ADR-026)
├── db.py # SQLite schema and queries (WAL mode)
├── stable_id.py # The 60-bit compound-key hash every symbol identity uses
├── call_resolver.py # Baseline CALLS-edge resolution (ADR-021)
├── graph_analytics.py # Community detection + centrality + god-objects (ADR-006)
├── graph_report.py # Markdown report rendering for map_module_communities
├── graph_viz.py # Design Structure Matrix (DSM) HTML visualization
├── incremental_indexer.py # Change detection and the build (the code-indexer command)
├── hybrid_retriever.py # Retrieve → Traverse → Rerank pipeline
├── iterative_retriever.py # Multi-round retrieval with early stopping
├── fusion.py # Sparse tokenization + score-normalized fusion (ADR-009)
├── reranker.py # Cross-encoder rescoring — off by default
├── import_resolver.py # tsconfig path aliases, barrel files, relative imports
├── category_tagger.py # Symbol classification
├── summarizer.py # Chunk summaries (Qwen2.5-Coder), for the separate summary index
└── tui/ # Textual terminal UI
├── app.py
├── backend.py
└── tools.py
The .code-index/ directory (FAISS indexes and graph.db) is generated at runtime and is git-ignored.
docs/README.md is the map. Three working lists, three jobs, no overlap:
| Doc | Holds |
|---|---|
docs/backlog.md |
wants — problems and requests, with a source and a date |
docs/roadmap.md |
order — sequencing and dependency edges. Start here to find what's next. |
docs/adr/ |
decisions — one committed solution per file, plus its Implementation Log |
A backlog item asserts a problem; an ADR asserts a solution you are building. Contribution standards,
including the branch-born ADR rule, are in CONTRIBUTING.md.
For depth this README intentionally skips, docs/ARCHITECTURE.md is the
long-form walkthrough of every part of the engine and why it is shaped that way — a dated snapshot,
not a living document. Each ADR's Status: line is the source of truth for anything summarized here;
where the two disagree, the ADR is right.
- Stable FAISS IDs — 60-bit deterministic IDs (file hash × offset) allow surgical removes without full rebuilds.
- Dual persistence — FAISS handles ANN speed; SQLite handles graph queries. They are complementary, not redundant.
- Monster-line shredding —
ast_chunker.pydetects and strips Base64 inline SVGs and similar noise before chunking to prevent embedding corruption. - Import resolution —
import_resolver.pyhandles tsconfig path aliases, relative imports, barrel files, and extension inference so graph edges reflect real module boundaries. - Lazy cross-encoder — When the reranker is enabled,
Qwen/Qwen3-Reranker-0.6B(~1.2 GB) loads only on first use rather than at startup, so the MCP handshake completes instantly.
| Language | Extensions | Symbols extracted | Graph edges |
|---|---|---|---|
| Python | .py |
module, class, function, method, decorated function | import, call, owns, extends |
| TypeScript | .ts, .tsx |
class, interface, function, arrow function, type alias, React component | import, call, owns, extends, context |
| JavaScript | .js, .jsx |
class, function, arrow function, React component | import, call, owns, extends, context |
| C# | .cs, .csproj, .sln |
namespace, class, interface, struct, record, enum, method, constructor, property | import (using), call, owns, extends, implements |
| C++ | .cpp, .cc, .cxx, .h, .hpp, .hxx |
namespace, class, struct, enum, function, method, constructor, template, typedef, using alias | import (#include), call, owns, extends |
| Rockwell L5X (PLC) | .L5X, .l5x |
module, program, routine, add-on instruction (AOI), its parameters and local tags, tag | owns, alias of, reads, writes (with rung-level references, ADR-013) |
These are known, documented constraints — not bugs. They reflect the fundamental difference between a heuristic structural indexer and a compiler frontend.
- Dynamic attribute assignment (
self.x = ...in__init__) is not indexed as a symbol — only statically declared class attributes are. __getattr__/__getattribute__overrides make attribute resolution invisible to the graph.- Decorators are captured as symbol metadata but decorator-generated methods (e.g.
@property,@staticmethod) are not separately indexed as symbols.
- Extension methods (prototype augmentation) resolve to candidates only.
- Dynamic
require()andimport()with non-literal paths are not resolved. - Type-narrowed dispatch (discriminated unions) produces candidate call edges, not single-target edges.
tsconfig.jsonpath aliases are resolved;node_modulesedges terminate at the package boundary.
- Extension methods resolve to candidates only (no receiver-type inference without full type resolution).
- LINQ query syntax is indexed as text at chunk level, not as call edges.
dynamicis invisible to the graph.- Cross-file partial-class EXTENDS/IMPLEMENTS/OWNS edges are not cleaned on incremental re-index; full re-index always produces a correct graph.
Position statement: This is a heuristic semantic/structural indexer, not a compiler frontend. For precise C++ navigation use clangd or SCIP. This tool's value is hybrid search and a good-enough graph over codebases where no compile-accurate index exists.
- Preprocessor macros: macro-generated functions are invisible to the graph. Only source-visible function definitions and declarations are indexed.
- Template instantiations: template definitions index correctly; instantiations (
std::vector<int>) are invisible. - Function pointers and virtual dispatch: call edges resolve to the declared type only, not the runtime type.
- Operator overloads: indexed as symbols but rarely earn call edges (call sites use operator syntax, not a function name).
- Header/impl unification: relies on FQN identity across files. If a declaration and definition use different parameter spellings (e.g.
const string&vsconst std::string&) they will produce different FQNs and not unify. Normalize includes in your codebase to prevent this.
| Language | Symbols P/R | Edges P/R | Call edges P/R | Fixtures |
|---|---|---|---|---|
| cpp | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 6 |
| csharp | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 6 |
| python | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 6 |
| typescript | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 6 |
Measured on hand-authored feature fixtures (Tier-A adapters), not an exhaustive corpus — a row is "precision/recall on the fixtures we wrote," never a language's true precision (ADR-008 §7). Regenerate with python tools/conformance_eval.py --write-readme.
Known extraction gaps (encoded as correct ground truth; reported, not gated — the ruler catching real, documented limitations):
- csharp/interface_impl_gap — Class Foo implements two interfaces (IReader, IWriter) with no base class; both are correctly
implements. The adapter lacks type resolution and applies the base-list heuristic 'first entry -> extends, rest -> implements', mislabeling IReader asextends. Expected edges encode the correct semantics, so all-edges scores below 1.0.