lore

Institutional knowledge on your filesystem.

lore manages personal knowledge vaults and shared team libraries backed by markdown and git. It handles the plumbing — subscriptions, publishing, search, graph traversal, and agent-driven maintenance — so your editor and your LLM agent can focus on the content.

A persistent daemon indexes your vault and libraries into SQLite (FTS5 for BM25-ranked search, typed graph edges for relationship traversal). Query from the CLI, a terminal UI, or let your agent use it as a knowledge backend.

Install

go install github.com/gregbuehler/lore@latest

Or build from source:

git clone https://github.com/gregbuehler/lore.git
cd lore
go install ./...

Quick Start

1. Create a vault

lore vault init ~/vault
cd ~/vault

This creates a personal vault with .lore/config.yaml and the default directory structure. If you already have an Obsidian vault or markdown directory:

lore vault init --adopt ~/my-existing-vault

2. Subscribe to a library

lore subscribe git@git.example.com:team/services.git

Or subscribe to a local directory:

lore subscribe /path/to/library --local

If the lore content lives inside a normal project repository, subscribe with an explicit root:

lore subscribe git@github.com:gregbuehler/citizen.git --root docs

3. Start the daemon

export LORE_VAULT=~/vault
lore daemon start

The daemon watches your vault and libraries, maintains a SQLite FTS5 index, and serves queries over a Unix socket. It auto-starts on first query if LORE_VAULT is set.

To auto-start at login:

lore daemon install   # macOS launchd or Linux systemd --user

4. Search and explore

lore query "gateway mTLS"                    # BM25 full-text search
lore query --graph Wiki/Services/gateway     # outgoing graph edges
lore query --backlinks Wiki/Services/gateway # incoming references
lore context Wiki/Services/gateway --brief   # assembled LLM context
lore ui                                     # terminal UI

5. Generate agent context

lore vault context

This writes .lore/LORE.md with your library subscriptions, skills, and workflows, and imports it into .claude/CLAUDE.md so Claude Code sees everything on session start.

For Codex-based vaults, the same generated .lore/LORE.md is imported from AGENTS.md.

6. Explore further

lore vault status              # see what you're subscribed to
lore search "authx mTLS"       # raw grep-style markdown search
lore library skills             # list available skills

Authenticated document HTTP API

The existing Unix-socket and CLI interfaces remain available. Set LORE_API_TOKEN to enable the HTTP listener alongside lore daemon start; LORE_API_ADDR defaults to 127.0.0.1:7520. Every /api/v1 route requires Authorization: Bearer <token>. /healthz and /readyz are unauthenticated health probes. Use TLS at the reverse proxy when exposing the service.

The API operates on the same Markdown files and subscription configuration as the CLI. Source IDs remain vault and library:<subscription-name>; discovery includes configured subscriptions and does not hide the root vault. Libraries retain their configured read-only or read-write permissions.

Method and route	Request / result
GET /api/v1/sources	{sources:[{source_id,name,kind,writable}]}
POST /api/v1/sources	{name} → {source}; persists a local read-write Lore subscription
GET /api/v1/documents?source=ID&q=text&cursor=CURSOR	{documents,hasMore,nextCursor}; follow all pages; query uses the existing source-scoped FTS index
GET /api/v1/document?source=ID&path=note.md	{document}; id=DOCUMENT_ID may replace path
PUT /api/v1/document	{source,path,content,baseRevision,documentID?,operationID?} → {document}
DELETE /api/v1/document?source=ID&path=note.md&baseRevision=REV	{ok:true}; optional id=DOCUMENT_ID asserts identity
POST /api/v1/document/move	{source,path,newPath,baseRevision,documentID?} → {document}
GET /api/v1/folders?source=ID	{folders:[{path}]}, including empty nested folders
POST /api/v1/folders	{source,path} → {path}; creates nested directories
GET /api/v1/resolve?source=ID&from=note.md&target=TARGET	{resolved,document_id?,source_id?,rel_path?,title?}
GET /api/v1/backlinks?source=ID&path=note.md	{documents:[document summary]}; same-library references only
GET /api/v1/asset?source=ID&path=images/photo.png	Attachment bytes with safe type/disposition headers
GET /api/v1/assets?source=ID	{assets:[{path,revision,size,modified_at}]}; sorted canonical portable attachment manifest
POST /api/v1/asset?source=ID&path=images/photo.png	Binary upload; existing paths are never replaced
GET /api/v1/export?source=ID&path=optional/folder	ZIP preserving relative Markdown, attachments and empty folders
POST /api/v1/import?source=ID&folder=optional/folder	ZIP bytes → {imported:number}; preflight rejects collisions rather than replacing files

URL-encode every query value. Errors are {code,error}; stale or missing revisions on an existing document and destination collisions return HTTP 409. An empty baseRevision creates a document only when its path does not exist. Delete and move require the current SHA256 revision. API mutations are serialized; clients must retain local edits on conflicts and reload the latest revision before retrying. API move preserves identity and updates same-library wiki/Markdown inbound references and the moved note's relative outbound links, without editing fenced/inline code. External editors remain supported through the filesystem watcher; they should not concurrently rename/write files while a multi-file move is being applied.

For synchronization, supply the optional documentID on updates/moves and id on deletion. Identity and content revision are checked together under the writer lock, so a replacement at a reused path is protected even when its bytes match. Existing clients that omit the new identity precondition remain compatible.

Creation accepts an optional UUID operationID. Persist the UUID before sending, then replay the exact original path and content after an uncertain response. The identity sidecar atomically reserves a receipt and canonical identity before publishing the Markdown file. A replay returns that identity's current document, including later edits and moves; it never recreates the old path. Reusing the UUID with a different path/content returns 409. If the accepted document was deleted, replay returns 410 creation_gone, preserving the caller's opportunity to recover its local draft explicitly. Receipts are source-scoped, survive document deletion, and store only the creation content hash. Interrupted reserved creates can resume after a daemon restart.

Keep receipts with the canonical identity backup. An older daemon that drops receipt fields is not a safe storage-server rollback after offline writes; restore a compatible coordinated backup. Historical HTTP/socket clients still work with the new daemon.

Wiki-link filename paths are percent-encoded independently of heading and alias suffixes and decoded once during resolution. For example, a file named new#draft.md is linked as [[new%23draft]], not [[new#draft]]. Rename/move preserves this distinction, including links inside ordinary list containers; fenced and inline code examples remain literal.

Offline clients use the additive, read-only assets endpoint to discover all existing attachments in a selected source, including unreferenced files, without unpacking an export ZIP or maintaining another backend cache/index. Each entry has a source-relative path, lowercase SHA-256 revision of the file bytes, byte size, and RFC3339Nano modified_at. Results are sorted by path and include only regular portable files other than .md (case-insensitive); hidden paths, symlinks and special files are excluded. Empty manifests return {"assets":[]}. Hashing streams canonical bytes with a fixed buffer and honors cancellation; read errors or detected concurrent file changes fail the entire enumeration, not a partial-success response. Clients download changed entries using asset, verify the revision before declaring offline availability, and retry failed passes without treating failures as deletions. Existing upload, download, import and export contracts remain unchanged.

Stable document identity and migration

Documents retain source_id, rel_path, title, content and revision. Read/list responses additionally include document_id, created_at and modified_at (RFC3339); reads include parsed YAML metadata when available. Summaries include excerpt instead of full content. Normal API reads lazily assign existing files random Lore-owned identities. Initial created_at uses the existing file modification time (historical creation dates cannot be recovered); subsequent writes preserve it, while modified_at follows the canonical file's modification time.

The HTTP metadata projection stringifies YAML mapping keys and represents nonfinite scalars as ".nan", ".inf" or "-.inf", recursively. This keeps valid YAML frontmatter JSON-readable without rewriting the author's Markdown.

Identity is stored atomically in <vault>/.lore/document-identities.json, separately from the disposable SQLite search index. This is a narrow additive migration: no Markdown is relocated, no author frontmatter is rewritten, and existing path-addressed HTTP/socket clients continue working. API rename/move, restart and reindex preserve IDs. An external filesystem rename is not an API move and cannot reliably preserve identity. Portable ZIP import creates new identities rather than importing another vault's private identity registry.

API-created libraries live in <vault>/.lore/libraries/<name> and are recorded in <vault>/.lore/config.yaml as ordinary local: subscriptions. Existing subscriptions retain their paths. Hidden placement prevents double indexing as vault documents; each library is still indexed and watched as its own root. Restarting with the vault config rediscovers these libraries. Source discovery also reloads config subscriptions, so CLI subscribe/unsubscribe changes appear without restarting; new roots are indexed and watched on discovery. Hidden-path filtering is relative to the most-specific registered source root: a library beneath .lore/libraries receives external-edit notifications, while hidden metadata beneath that library, symlinks and out-of-source paths remain excluded.

Back up canonical Markdown/attachments, all configured subscription roots, .lore/config.yaml and .lore/document-identities.json together. Do not delete the identity registry when rebuilding the index. LORE_DB selects the base directory for the derived per-vault SQLite index and is not canonical content.

Portable file safety

Document/attachment/folder paths are confined to their selected source. Traversal and symlink components are rejected. The existing explicit document exceptions .lore/LORE.md and .lore/skills/*.md remain readable as Markdown; config and credentials are not exposed as assets. ZIP transfer excludes hidden files/directories (including .git, .lore and .env) and rejects unsafe archive names, symlinks, duplicate entries and existing destination conflicts. Import validates and expands the entire archive before applying files. Limits are 64 MiB compressed ZIP, 256 MiB expanded ZIP, 32 MiB per archive file or attachment upload, and 4096 archive entries (including implicit parent folders). imported counts files, not directories. Export/import preserve ordinary relative file layout, not Lore configuration, credentials or subscription registration. Ordinary POSIX colon and significant-space filenames roundtrip unchanged. Windows drive/UNC paths, traversal, control characters and hidden metadata are not accepted portable paths.

Concepts

Vaults are personal. Your notes, daily logs, drafts, and context. A vault is a directory of markdown files with .lore/config.yaml. Your agent works here.

Libraries are shared. Team knowledge bases that anyone can subscribe to. Each library is a git repo (or local directory) containing Wiki pages, skills, and a library.yaml schema. Subscribing clones it locally; your agent reads it as local markdown alongside your vault. By default the library root is the repository root; use --root docs when the lore content lives in a subdirectory of a larger project repo.

Skills are procedural knowledge curated in libraries. They encode retrieval paths — not facts, but how to find out. When a library has a skill like deployed-versions, any agent with access to that library knows how to answer "what versions are deployed on staging?" without re-deriving the answer path.

Excerpts are self-descriptions that libraries publish. Each library generates an excerpt.md summarizing its pages, skills, and sources. lore vault context assembles these excerpts into your vault's agent context — so libraries own their own descriptions and the vault doesn't need to parse library internals.

Commands

See docs/commands.md, generated from the Cobra command tree. Refresh it with:

lore docs commands docs/commands.md

Daemon Architecture

The lore daemon is a lightweight background process that provides fast search and graph queries:

┌─────────────┐     Unix socket      ┌──────────────────────┐
│  lore query │ ───────────────────── │     lore daemon      │
│  lore ui    │  length-prefixed JSON │                      │
│  agent      │                       │  fsnotify watcher    │
└─────────────┘                       │  SQLite FTS5 index   │
                                      │  typed graph edges   │
                                      └──────────────────────┘

• Index: SQLite with FTS5 virtual table for BM25-ranked full-text search
• Graph: edges table storing typed relationships (owner, depends_on, deployed_in, mentions) extracted from wikilinks in section context
• Watcher: fsnotify-based recursive file watching with 500ms debounce
• Resolver: Obsidian-style shortest-path wikilink resolution (folder expansion, proximity ranking, ancestor walking)
• Protocol: Length-prefixed JSON over Unix socket (~/.local/share/lore/daemon.sock, or LORE_SOCKET)

The daemon auto-starts on first query when LORE_VAULT is set. All CLI commands fall back to direct SQLite access if the daemon is unavailable.

Index Health

A note on concurrency first: only one daemon may write a vault's index. The daemon takes an exclusive flock on <index>.lock before it starts indexing, because the socket — the other thing that keeps duplicates out — is only bound after the initial build. Two daemons writing one SQLite index is the fastest way to damage the FTS5 index.

FTS5 damage is not all-or-nothing, and the obvious checks lie. An index can hold every row, answer plain MATCH, bm25() and snippet() correctly, and pass FTS5's own integrity-check, while the ranked traversal behind ORDER BY rank — which every lore query uses — still fails with database disk image is malformed (267). Repopulating the documents table (a reindex) does not necessarily clear the damaged segment; only an FTS5-level rebuild does.

lore guards this in four places:

• On open: VerifyFTS runs integrity-check and a ranked probe against tokens taken from indexed titles. A failure triggers an automatic rebuild. A damaged FTS index warns instead of failing — graph and entity commands do not need FTS.
• On query: Search retries once after rebuilding when a query hits corruption mid-session, instead of surfacing "error 267".
• On reindex: full index builds finish with an FTS rebuild and verification, so lore daemon reindex genuinely repairs rather than repopulating around the damage.
• On demand: lore doctor verifies, lore doctor --repair rebuilds.

lore doctor            # verify integrity-check + ranked probe
lore doctor --repair   # rebuild the FTS index if verification fails

The index is a derived cache — deleting ~/.local/share/lore/vaults/*/index.db and restarting the daemon always rebuilds it from the markdown files.

macOS: vaults in ~/Documents

macOS gates ~/Documents, ~/Desktop and ~/Downloads behind privacy consent (TCC). A LaunchAgent has no such consent, and an ungranted open() inside those directories blocks in the kernel rather than failing — so a daemon installed with lore daemon install hangs before it logs anything, and KeepAlive respawns more hung copies. A daemon started from a terminal works, because it inherits the terminal's grant; that difference invites hand-started duplicates, which is how indexes get corrupted.

If your vault lives in one of those directories, grant the binary access:

System Settings → Privacy & Security → Full Disk Access → +
/path/to/lore

Then reload the agent and confirm with lore daemon status. Re-granting may be needed after upgrading the binary. Keeping the vault outside those directories avoids the grant entirely.

Library Anatomy

my-library/
  library.yaml           # schema, tone rules, TTLs, sources, skills
  CLAUDE.md              # agent instructions for this library
  excerpt.md             # self-description (generated by lore library index)
  Wiki/
    index.md             # navigation index (generated)
    Services/            # entity pages grouped by type
      gateway.md
      storage.md
  skills/
    deployed-versions.md # procedural knowledge
  sources/
    incoming/            # contributor drop zone
  log.md                 # activity log

library.yaml

name: "services"
description: "Platform service knowledge"

publishing: pr-required

tone:
  voice: third-person
  prohibited:
    - personal characterizations
  required:
    - factual sourcing

default_ttl:
  service: 90d
  environment-config: 14d

skills:
  - name: deployed-versions
    file: skills/deployed-versions.md
    description: Look up deployed software versions

sources:
  - repo: git.example.com/myorg/deployment
    local: ~/src/git.example.com/myorg/deployment
    watch:
      - path: deployments/{entity}/**
        maps_to: environment

Skills

A skill is a markdown file in skills/ with frontmatter and a step-by-step procedure:

---
name: deployed-versions
description: Look up deployed software versions
trigger: "what versions are deployed on {environment}"
inputs:
  - name: environment
    required: true
---

# Deployed Versions

## Fast Path: Service Catalog
...

## Detailed Path: Source Repos
...

Skills teach agents how to answer questions, not what the answer is. The library's CLAUDE.md references skills so agents discover them automatically.

Maintenance Pipeline

lore's maintenance system turns raw evidence into synthesized library knowledge:

Daily Logs ──→ lore library review ──→ surface new evidence
             → lore library maintain ──→ agent synthesizes into pages

Source Repos ──→ lore library watch ──→ agent updates pages from IaC changes

Sync ──→ lore sync ──→ pull all libraries + reindex
Push ──→ lore publish ──→ commit+push pending changes to library repos

How maintain works

1. Scans daily log files for entity mentions newer than each page's last_updated
2. Matches mentions using entity names and aliases (word-boundary-aware)
3. Assembles a context package: current page + new evidence + tone rules + format rules
4. Invokes the configured agent to rewrite the page
5. Cleans up the context package and rebuilds indexes

How watch works

1. Reads sources: from library.yaml — repos and path patterns to track
2. Maps entities to repo directories using {entity} placeholders and inventory data
3. Runs git log --since=<last_updated> to find relevant commits
4. Assembles a context package with commits, authors, file lists, and instructions
5. Invokes the agent to synthesize changes into the page

Agent providers

Maintenance commands use agent.provider to decide how agent work is executed. Shared defaults live in .lore/config.yaml; machine-local preferences can live in .lore/local.yaml, so the same synced vault can use Codex at home and Claude at work without editing the shared config.

Provider	Behavior
claude	Run Claude non-interactively with claude -p <prompt>.
codex	Run Codex non-interactively with codex exec, send the prompt on stdin, and pass --cd <workdir>. Sandbox and approval settings are configurable.
custom	Run a user-supplied command for sites that wrap or proxy an agent. Requires an explicit agent.command.
none	Do not invoke an agent; commands stop before synthesis.

Dangerous permission bypass is always opt-in. When enabled, lore maps it to the provider-specific dangerous flag, such as Claude's --dangerously-skip-permissions or Codex's equivalent unsafe bypass flag. Leave it disabled for normal local or CI maintenance.

Agent selection precedence is:

1. Command flag, for example lore maintain services --agent codex
2. Environment, for example LORE_AGENT_PROVIDER=codex
3. Machine-local .lore/local.yaml
4. Shared .lore/config.yaml
5. Backward-compatible default: Claude

Example .lore/local.yaml for a Codex machine:

agent:
  provider: codex
  command: codex
  sandbox: workspace-write
  approval: never

Generate those local overrides with:

lore agent local codex
lore agent local claude
lore agent local none
lore agent local status

lore agent local only generates built-in provider configs. For custom, edit .lore/local.yaml manually and include agent.command plus optional agent.args.

CI Automation

When a library is a git repo, maintenance runs as GitHub Actions:

on:
  schedule:
    - cron: '0 6 * * 1'
  workflow_dispatch:
jobs:
  maintain:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: lore library watch my-library
      - run: lore library lint my-library --fix
      - run: lore library index my-library
      - uses: peter-evans/create-pull-request@v6

Linting

lore vault lint checks vault pages. lore library lint checks library pages with additional rules:

Check	Vault	Library	Fixable
Missing frontmatter	yes	yes	yes
Empty files	yes	yes	no
Missing entity_type		yes	no
Stale pages (TTL)		yes	no
Stale index		yes	no
Local filesystem paths		yes	yes
Section names		yes	yes
Change log format		yes	yes
Required sections		yes	no
Frontmatter field order		yes	no

Agent Context

lore generates context so your agent understands the knowledge system:

.lore/LORE.md             ← generated library/skills discovery

.claude/CLAUDE.md         ← Claude vault operating manual
  @../.lore/LORE.md       ← import: generated context

AGENTS.md                 ← Codex vault operating manual
  @.lore/LORE.md          ← import: generated context

Library CLAUDE.md          ← per-library agent instructions
  excerpt.md               ← per-library self-description

Run lore vault context after subscribing to new libraries or adding skills. It regenerates .lore/LORE.md and wires imports idempotently. By default, it updates existing provider context files (.claude/CLAUDE.md and/or AGENTS.md); if none exist, it falls back to the effective agent provider. Use --agent claude, --agent codex, --agent all, or --agent none to be explicit.