fix(index): a repeat full index keeps the name words of files it didn't change

CHANGELOG.md

@@ -65,6 +65,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335)
 - `codegraph index` no longer stops with advice to run `codegraph daemon stop`, a command that doesn't exist, when the project's background-server lock names a process that isn't CodeGraph, such as a lock left by a crash whose process number the system has since given to another program: it now clears that stale lock, leaving the process alone, and rebuilds, and `codegraph unlock` clears it too. When a CodeGraph process that doesn't answer holds the lock, such as an older background server, the message names that process and the command that stops it, and the message about another server already serving a project now points to `codegraph daemon`. Thanks @inth3shadows for the report and @danusha2345 for the first fix. (#2770)
 - A program that uses CodeGraph as a library and indexes chosen files with `indexFiles()` now keeps other CodeGraph processes, like a `codegraph sync` run by a git hook, out of the index until those files are stored. Before, it let go of the write lock that keeps two writers apart as soon as indexing started, so another process could write to the index at the same time.
+- A program that uses CodeGraph as a library and runs `indexAll()` again on a project it has already indexed now keeps every symbol findable by the words in its name, which the prompt hook and `codegraph_explore` use to match a question like "how does the order state machine work" to `OrderStateMachine`. Before, only the files changed since the last index kept their names' words, and a cancelled run lost them all; `codegraph index` was not affected. Thanks @rudycelekli.
 - The shared background server is no longer killed with "Main thread unresponsive" while it brings another project's index up to date, such as one a session names with `projectPath`: only its own project's index counted as progress, so a long update of any other looked like a hang, and the server was stopped and restarted again and again. Every index the server keeps up to date now counts, and so does the one a session updates on its own when the background server is unavailable. Thanks @mradwankhalil for the report and the first fix. (#2404)
 - A long-running session, like the background server behind your agent, now notices a framework that arrives or goes while it runs, such as Express in a new `package.json`, or one app replaced by another with as many files. Files it indexes from then on are read with the frameworks the project has now; before, they were read with the ones it had when the session started, so a new Express app's routes were missing until a full `codegraph index`. Files indexed before the change are not read again, so `codegraph status` now says which frameworks changed since the last full index and recommends running one. Thanks @nk33-dev for the report and the first fix. (#2516, #2519)
 - On Windows, deleting the folder a link in your project points to, like a junction to a library kept outside the project, no longer makes CodeGraph's server use a full CPU core until it is restarted, or stop picking up your edits. The linked files now drop out of the index while the rest of the project keeps updating, and if the project folder itself disappears while watched, live updates turn off with a message that says so. Thanks @L0garithmic.

__tests__/segment-vocab.test.ts

@@ -3,7 +3,7 @@ import * as fs from 'node:fs';
 import * as path from 'node:path';
 import * as os from 'node:os';
 import { CodeGraph } from '../src';
-import { extractProseCandidates } from '../src/search/identifier-segments';
+import { extractProseCandidates, splitIdentifierSegments } from '../src/search/identifier-segments';
 
 /**
  * The graph-derived gate behind the prompt hook's MEDIUM tier: symbol names
@@ -61,6 +61,24 @@ export function writeConfig(): void {}
     fs.rmSync(dir, { recursive: true, force: true });
   });
 
+  type Rows<T> = { prepare(sql: string): { all(): T[] } };
+  const graphDb = <T>(): Rows<T> => (cg as unknown as { db: { getDb(): Rows<T> } }).db.getDb();
+  /** Every vocabulary row as "segment name", sorted. The table itself:
+   *  getSegmentMatches hides orphan rows, so a prune shows only here. */
+  const vocabRows = (): string[] =>
+    graphDb<{ segment: string; name: string }>()
+      .prepare('SELECT segment, name FROM name_segment_vocab')
+      .all()
+      .map((r) => `${r.segment} ${r.name}`)
+      .sort();
+  /** What a fresh index writes: the segments of every symbol name in the graph. */
+  const expectedVocab = (): string[] =>
+    graphDb<{ name: string }>()
+      .prepare("SELECT DISTINCT name FROM nodes WHERE kind NOT IN ('file', 'import')")
+      .all()
+      .flatMap((r) => splitIdentifierSegments(r.name).map((s) => `${s} ${r.name}`))
+      .sort();
+
   it('co-occurrence: two prose words on one name find it — the reported-prompt shape', () => {
     // The words a French prompt would produce: "comment marche la state
     // machine des commandes ?" — no keyword list knows any of them.
@@ -158,6 +176,76 @@ export function writeConfig(): void {}
     await expect(cg.healSegmentVocabIfEmpty()).resolves.toBe(true);
   });
 
+  it('a second full index keeps the words of the names it did not rewrite (#2252)', async () => {
+    // Over an existing graph the store skips unchanged files, so their names
+    // are never written again: the run must not lose the rows it found.
+    const before = vocabRows();
+    expect(before).toEqual(expectedVocab());
+    for (let run = 0; run < 2; run++) {
+      expect((await cg.indexAll()).success).toBe(true);
+      expect(vocabRows()).toEqual(before);
+    }
+    expect(cg.getSegmentMatches(['state', 'machine']).map((m) => m.name)).toContain('OrderStateMachine');
+    expect(cg.getSegmentMatches(['checkout']).map((m) => m.name)).toContain('CheckoutService');
+  });
+
+  it('a full index after an edit keeps unchanged files\' words and prunes a renamed name (#2252)', async () => {
+    const touched = path.join(dir, 'src', 'state-machine.ts');
+    fs.writeFileSync(touched, fs.readFileSync(touched, 'utf8').replace('OrderStateMachine', 'RenamedWorkflowEngine'));
+    expect((await cg.indexAll()).success).toBe(true);
+    // Exactly the graph's names: the untouched files' rows kept, the new name
+    // added, and the old one's rows gone from the table itself.
+    expect(vocabRows()).toEqual(expectedVocab());
+    expect(vocabRows().filter((r) => r.endsWith(' OrderStateMachine'))).toEqual([]);
+    expect(cg.getSegmentMatches(['renamed', 'workflow']).map((m) => m.name)).toContain('RenamedWorkflowEngine');
+    expect(cg.getSegmentMatches(['checkout']).map((m) => m.name)).toContain('CheckoutService');
+    expect(cg.getSegmentMatches(['state', 'machine'])).toEqual([]);
+  });
+
+  it('a name a full index pruned is written again when it comes back in the same session (#2252)', async () => {
+    // The write path skips names it has already written this session; after
+    // the prune drops a name's rows, that memory must not hide its return.
+    const touched = path.join(dir, 'src', 'state-machine.ts');
+    const original = fs.readFileSync(touched, 'utf8');
+    const renamed = original.replace('OrderStateMachine', 'RenamedWorkflowEngine');
+    fs.writeFileSync(touched, renamed);
+    expect((await cg.indexAll()).success).toBe(true);
+    fs.writeFileSync(touched, original);
+    expect((await cg.indexAll()).success).toBe(true);
+    expect(vocabRows()).toEqual(expectedVocab());
+    expect(vocabRows().filter((r) => r.endsWith(' RenamedWorkflowEngine'))).toEqual([]);
+    fs.writeFileSync(touched, renamed);
+    await cg.sync();
+    expect(cg.getSegmentMatches(['renamed', 'workflow']).map((m) => m.name)).toContain('RenamedWorkflowEngine');
+  });
+
+  it('a cancelled full index keeps the vocabulary it found (#2252)', async () => {
+    const before = vocabRows();
+    const controller = new AbortController();
+    controller.abort();
+    const result = await cg.indexAll({ signal: controller.signal });
+    expect(result.success).toBe(false);
+    expect(result.errors.map((e) => e.message)).toContain('Aborted');
+    expect(vocabRows()).toEqual(before);
+    expect(cg.getSegmentMatches(['state', 'machine']).map((m) => m.name)).toContain('OrderStateMachine');
+  });
+
+  it('a full index into a fresh database still starts the vocabulary over (`codegraph index`)', async () => {
+    // `codegraph index` recreates the database, then runs a full index.
+    const before = vocabRows();
+    cg.close();
+    cg = await CodeGraph.recreate(dir);
+    expect((await cg.indexAll()).success).toBe(true);
+    expect(vocabRows()).toEqual(before);
+    // An emptied graph counts as fresh too: rows it no longer backs are dropped
+    // at the start, as before, rather than kept for a prune.
+    const queries = (cg as unknown as { queries: { insertNameSegmentsBatch(names: string[]): void } }).queries;
+    queries.insertNameSegmentsBatch(['GhostSymbolMachine']);
+    cg.clear();
+    expect((await cg.indexAll()).success).toBe(true);
+    expect(vocabRows()).toEqual(before);
+  });
+
   it('a rename through updateNode reaches the vocab — the framework post-extract path (#1141)', () => {
     // Framework resolvers rewrite node names after extraction (NestJS route
     // prefixing) via updateNode. The new name must become prose-searchable;

src/db/migrations.ts

@@ -109,8 +109,9 @@ const migrations: Migration[] = [
       // DDL only — instant on any size database (the row-churn hazards of #1067
       // don't apply). The table starts EMPTY on migrated databases; `sync`
       // detects that over a populated graph and backfills batched+yielding
-      // (CodeGraph.rebuildNameSegmentVocab), and any full index rebuilds it
-      // from scratch. Keep the definition in lockstep with schema.sql.
+      // (CodeGraph.rebuildNameSegmentVocab), and a full index into a fresh
+      // database fills it from scratch. Keep the definition in lockstep with
+      // schema.sql.
       db.exec(`
         CREATE TABLE IF NOT EXISTS name_segment_vocab (
           segment TEXT NOT NULL,

src/db/queries.ts

@@ -644,8 +644,9 @@ export class QueryBuilder {
   }
 
   /** Which node kinds contribute their name to the segment vocabulary — the
-   *  single gate shared by insertNode, updateNode, and the rebuild page query
-   *  (getDistinctNodeNames), so the write paths can't drift apart. */
+   *  single gate shared by insertNode, updateNode, the rebuild page query
+   *  (getDistinctNodeNames) and the prune (pruneNameSegmentVocab), so the
+   *  write paths can't drift apart. */
   private isSegmentableKind(kind: string): boolean {
     return kind !== 'file' && kind !== 'import';
   }
@@ -901,14 +902,30 @@ export class QueryBuilder {
   // Name-segment vocabulary (prompt-hook graph-derived gate)
   // ===========================================================================
 
-  /** Wipe the segment vocabulary. A full index calls this at its start; the
-   *  node write path repopulates it as files (re-)index, so the end state is
-   *  exactly the current names with no orphan rows. */
+  /** Wipe the segment vocabulary. A full index of a fresh database calls this
+   *  at its start; the node write path repopulates it as files index, so the
+   *  end state is exactly the current names with no orphan rows. */
   clearNameSegmentVocab(): void {
     this.db.exec('DELETE FROM name_segment_vocab');
     this.segmentedNames.clear();
   }
 
+  /**
+   * Drop the rows of names that no symbol carries any more, deleted or renamed
+   * since they were written. A full index over an existing graph calls this
+   * once it has stored everything, instead of clearing the table up front: it
+   * never rewrites the names of its unchanged files, so a clear lost those
+   * until a rebuild (#2252). The session's dedupe set goes too, so a dropped
+   * name that comes back later is written again rather than skipped as
+   * already seen.
+   */
+  pruneNameSegmentVocab(): void {
+    this.db.exec(
+      "DELETE FROM name_segment_vocab WHERE name NOT IN (SELECT name FROM nodes WHERE kind NOT IN ('file', 'import'))"
+    );
+    this.segmentedNames.clear();
+  }
+
   /** True when the vocab has no rows — an index built before the table existed.
    *  `sync` uses this to heal such databases (see rebuildNameSegmentVocabFrom). */
   isNameSegmentVocabEmpty(): boolean {

src/db/schema.sql

@@ -153,8 +153,9 @@ END;
 -- materialized on the node write path.
 -- Deletions leave orphan rows ON PURPOSE: rows are PROPOSALS, always
 -- re-verified against nodes before being surfaced (CodeGraph.getSegmentMatches),
--- and a full index clears the table at its start. Populated lazily on old
--- databases (empty until the next index/sync heals it).
+-- and a full index prunes them once it completes (a fresh one starts the table
+-- empty). Populated lazily on old databases (empty until the next index/sync
+-- heals it).
 CREATE TABLE IF NOT EXISTS name_segment_vocab (
     segment TEXT NOT NULL,
     name TEXT NOT NULL,

src/index.ts

@@ -650,10 +650,15 @@ export class CodeGraph {
         // marker behind, so `codegraph status` can tell a truncated index
         // from a completed one instead of silently serving partial results.
         try { this.queries.setMetadata('index_state', 'indexing'); } catch { /* metadata is advisory */ }
-        // Segment vocabulary starts empty and is repopulated by the node write
-        // path as every file (re-)indexes below — so a full index is also the
-        // orphan-cleanup pass for names deleted since the last one.
-        try { this.queries.clearNameSegmentVocab(); } catch { /* vocab is advisory — never fail an index over it */ }
+        // Segment vocabulary. On a fresh database it starts empty, and the node
+        // write path fills it as every file indexes below. Over an existing
+        // graph the store skips unchanged files, which never write their names
+        // again, so clearing here lost them until a rebuild, and a cancelled or
+        // failed run left the table empty (#2252). There the table is kept, and
+        // the names no symbol carries any more are pruned once the run completes.
+        if (freshDb) {
+          try { this.queries.clearNameSegmentVocab(); } catch { /* vocab is advisory — never fail an index over it */ }
+        }
         // Bulk FTS mode for the mass-insert phase: drop the per-row FTS sync
         // triggers, rebuild nodes_fts once from the nodes table afterwards.
         // Crash inside the window is healed on the next DatabaseConnection.open.
@@ -785,6 +790,16 @@ export class CodeGraph {
           if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[synth-timing] deferredThisMember: ${Date.now() - tDeferred}ms`);
         }
 
+        // Over an existing graph the vocabulary was kept (see the top of the
+        // run). Every write of this run has landed, so drop the rows of names
+        // deleted or renamed since they were written — the orphan cleanup the
+        // up-front clear used to do. A cancelled run keeps them all.
+        if (!freshDb && result.success && !options.signal?.aborted) {
+          const tPrune = Date.now();
+          try { this.queries.pruneNameSegmentVocab(); } catch { /* vocab is advisory — never fail an index over it */ }
+          if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] vocab-prune: ${Date.now() - tPrune}ms`);
+        }
+
         // Refresh planner stats + checkpoint the WAL after bulk writes.
         // Off-thread (worker connection): on a multi-GB index this is minutes
         // of IO, and inline it starved the #850 watchdog AFTER a fully