Thorondor

A self-hosted, data-sovereign semantic web-search service for agents — discovers URLs through SearXNG, crawls pages via Crawl4AI, chunks content with a first-party semantic chunker, reranks passages against the original query, and returns cited evidence through REST and MCP with no mandatory hosted vendor.

What It Is

Thorondor is a local semantic web-search stack designed to replace hosted search-for-agents APIs (Exa, Tavily, and equivalents) with an on-premises, operator-controlled pipeline. It uses SearXNG for multi-engine URL discovery, Crawl4AI for JavaScript-capable page crawling, a first-party ClusterSemanticChunker backed by BGE-M3 embeddings for globally-optimal chunk boundaries, and BGE-reranker-v2-m3 to score passages against the original query before assembly. The orchestrator exposes versioned search, known-URL fetch, site-map, and bounded site-crawl contracts through REST and MCP. Search remains live and non-persistent; operators may separately enable a bounded local page cache for known-URL fetches and change monitoring.

The result is an agent-facing web intelligence layer rather than only a search endpoint: an agent can discover cited evidence, inspect a known page, request bounded links/tables/typed metadata or schema-shaped fields, detect that a page changed, and evaluate a specific target transition such as “out of stock” to “available”. Whole-page noise such as advertisements may still produce a general change signal, but cannot satisfy a target watch. Scheduling, alerts, and follow-up actions remain the responsibility of Tengwar or the calling agent runtime.

Architecture Overview

query
  -> optional LLM query planner (subquery expansion)
  -> SearXNG multi-engine URL discovery
  -> URL safety pre-filter (SSRF/IP blocklist)
  -> domain allow/block policy + selection scoring
  -> Crawl4AI page extraction (concurrent, per-host bounded)
  -> trafilatura HTML-to-Markdown cleaning + content dedup
  -> semantic chunking service (ClusterSemanticChunker + BGE-M3 embeddings)
  -> lexical candidate prefilter (keeps top-50 before reranking)
  -> batched reranking (BGE-reranker-v2-m3)
  -> relevance floor filter
  -> token-budget assembly -> passages + citations
  -> SearchResponse (REST or MCP)

All inter-service traffic travels over internal Compose networks. Crawl4AI's outbound HTTP exits through its built-in DNS-pinning proxy on a dedicated egress network.

Repository Layout

Path	Type	Description
orchestrator/	service	FastAPI app: /v1/search, /search (compat), /v1/fetch, /v1/map, /v1/crawl, /health, MCP /mcp
semantic-chunking-service/	service	FastAPI chunker: /chunk, /health — ClusterSemanticChunker + OpenAI-compatible embedding client
thorondor_cli/	tool	Textual configurator, env/deploy helpers, native thorondor-mcp proxy, and harness writers
ssrf-proxy/	service	Retained async HTTP CONNECT proxy for deployment compatibility; Crawl4AI 0.9.2 uses its own DNS-pinning egress
searxng/	config	SearXNG settings.yml mounted read-only by Compose
models/	artifact	Local GGUF model files consumed by the llama.cpp profile (not committed)
scripts/	scripts	install.*, deploy.ps1, deploy-llamacpp.ps1, smoke.ps1, check-release-guard.ps1 (PowerShell) + Bash equivalents
docker-compose.yml	config	Core stack: orchestrator, chunker, SearXNG, Crawl4AI, and the retained first-party egress-proxy service; bundled-models profile adds TEI embedding + reranker
docker-compose.llamacpp.yml	config	Override: swaps embedding and reranker to local llama.cpp containers using GGUF files under models/
docker-compose.production.yml	config	Image-only production slice for Tengwar-style deployments; no build: blocks or model containers
.env.example	config	Template for .env — every key the Python settings loader requires
.env.llamacpp.example	config	Template for .env.llamacpp — llama.cpp image SHA, GGUF paths, and batch sizes
.env.production.example	config	Template overlay for released GHCR image refs and production service names
docs/architecture/	docs	Operational architecture reference (overview, pipeline, deployment, deps, security, config)
docs/plans/	docs	Active implementation plans; completed plans are removed after implementation
.agents/skills/thorondor-web-search/	skill	Repo-local agent skill for calling the running service
LICENSE	license	MIT
THIRD-PARTY-NOTICES.md	license	Third-party runtime image and package notices

Prerequisites

• Docker Engine 28+ with Docker Compose 2.33.1+, or a Docker Desktop release containing them — installed, running, and configured for Linux containers. Thorondor uses Compose gateway priority to select Crawl4AI's isolated outbound network deterministically.
• GGUF model files (llama.cpp profile only) — bge-m3.gguf and bge-reranker-v2-m3.gguf placed under models/. Verify model license terms before downloading weights.
• Hardware — tested on Windows 10/11 with Docker Desktop (WSL2 backend) and on ARM64 (NVIDIA DGX Spark). The llama.cpp image is pinned to a SHA known to work on both architectures and supports CPU inference. The bundled TEI image requires an AMD64 NVIDIA CUDA host.
• PowerShell 7+ for the deploy and smoke scripts. Bash equivalents exist under scripts/.

Quickstart — Textual Configurator

The interactive path is a one-line host install. It creates the local thorondor and thorondor-mcp commands; the first thorondor run initializes the managed deployment directory under ~/.thorondor.

curl -LsSf https://raw.githubusercontent.com/FeanorsCodeSL/thorondor/main/scripts/install.sh | sh
thorondor

PowerShell:

irm https://raw.githubusercontent.com/FeanorsCodeSL/thorondor/main/scripts/install.ps1 | iex
thorondor

The thorondor dashboard writes complete .env files, can probe endpoints, generates SEARXNG_SECRET and CRAWL4AI_API_KEY when blank, and drives the existing Compose stack. The dashboard uses the same keyboard model as the Imladris TUI: ↑/↓ to move through the action menu, Enter to open a screen, ←/→ to focus the action menu or the components table, and Esc to back out of a form. The left rail lists the operator actions; the main panel shows the live component table (searxng / crawl4ai / chunker / embedding / reranker / llm planner) plus status and harness status:

Action	Purpose
Mode	Choose BYO endpoints, bundled TEI containers, or llama.cpp GGUF containers. Picking llamacpp auto-downloads missing GGUF files into models/ with a progress bar.
Endpoints	Edit embedding, reranker, and optional LLM-operation endpoints.
Search/crawl	Tune ports, budgets, crawl limits, robots, and domain filters.
Validate	Check env completeness and show the exact Compose command.
Deploy	Run config, build, up -d, /health, and smoke search.
Wire MCP	Wire Claude Code, Codex, or OpenCode to thorondor-mcp or Docker HTTP /mcp.
Refresh state	Re-read .env and the harness detection without restarting.
Quit	Exit the dashboard.

The llama.cpp mode also exposes a Download models button (and a thorondor download-models subcommand for non-interactive use) that fetches the two Q8 GGUF files from immutable Hugging Face revisions into models/ and verifies their SHA-256 checksums without changing the active mode. Both are wired into scripts/deploy-llamacpp.ps1 and scripts/deploy-llamacpp.sh so a fresh clone can deploy the llamacpp profile with no manual file placement.

thorondor doctor is the non-interactive status path. thorondor-mcp is a native stdio MCP proxy that forwards web_search, web_fetch, web_map, and web_crawl to their matching versioned REST endpoints. The Dockerized HTTP MCP surface remains available at http://localhost:8080/mcp.

thorondor uninstall stops the managed stack, removes ~/.thorondor, and then removes the installed thorondor tool. Use thorondor uninstall --keep-tool when you only want to reset the local deployment files.

When a BYO embedding endpoint points outside the Compose internal network, the configurator generates docker-compose.host-endpoints.yml and includes it during deploy. That overlay gives only the chunker egress for the external embedding call; host-rewritten localhost endpoints also get host.docker.internal:host-gateway entries for Linux Docker Engine.

Quickstart — Windows with llama.cpp (fully local)

# 1. Clone and enter the repo
git clone <repo-url> thorondor
Set-Location thorondor

# 2. Create environment files from examples
Copy-Item .env.example .env
Copy-Item .env.llamacpp.example .env.llamacpp

# 3. Create the models directory and place GGUF files
New-Item -ItemType Directory -Force models | Out-Null
# Copy bge-m3.gguf and bge-reranker-v2-m3.gguf into models\

# 4. Deploy: copies env files, generates internal service secrets, validates Compose,
#    builds first-party images, starts all services, runs a smoke search
.\scripts\deploy-llamacpp.ps1

The script exits non-zero on any failure. After a successful deploy:

# Health check
Invoke-RestMethod http://localhost:8080/health

# Quick search
$body = @{ query = "latest Python packaging tools 2025"; token_budget = 3000 } | ConvertTo-Json
Invoke-RestMethod http://localhost:8080/v1/search -Method Post -ContentType "application/json" -Body $body

To run the investigation-mode smoke test (broader queries + URL diagnostics):

.\scripts\smoke.ps1 `
  -ComposeFiles @("docker-compose.yml","docker-compose.llamacpp.yml") `
  -EnvFiles @(".env",".env.llamacpp") `
  -Profile llamacpp-models `
  -Investigation

Quickstart — BYO Embedding and Reranker Endpoints

If you already operate OpenAI-compatible embedding and reranker servers (vLLM, TEI, Infinity, etc.), skip the llama.cpp profile entirely and point the core stack at your endpoints:

# 1. Copy base env only
Copy-Item .env.example .env

# 2. Edit .env — set your external endpoints
# EMBEDDING_ENDPOINT=http://your-embedding-host:8082
# EMBEDDING_MODEL=BAAI/bge-m3
# RERANKER_ENDPOINT=http://your-reranker-host:8081
# RERANKER_MODEL=BAAI/bge-reranker-v2-m3
# RERANKER_PATH=/rerank

# 3. Deploy without the llama.cpp override
.\scripts\deploy.ps1 -Profile ""

The bundled-models profile (TEI containers) is still available if you want to run embedding and reranker inside Docker without GGUF files:

This profile requires an AMD64 host with NVIDIA Container Toolkit GPU access.

.\scripts\deploy.ps1 -Profile bundled-models

Production Images for Tengwar

Thorondor production deployments should consume released images rather than building from a source checkout on the headless host. The release workflow publishes:

• ghcr.io/feanorscodesl/thorondor-orchestrator
• ghcr.io/feanorscodesl/thorondor-chunker
• ghcr.io/feanorscodesl/thorondor-egress-proxy

Create a Git tag such as v0.1.0 or run the publish-images workflow manually. The workflow pushes multi-arch linux/amd64 and linux/arm64 images, uploads digest artifacts, and attaches THORONDOR_IMAGE_DIGESTS.md to tag releases.

For Tengwar, copy .env.production.example to .env.production, replace the three first-party image refs with the release digest refs, and run the image-only Compose file:

docker compose `
  --env-file .env --env-file .env.production `
  -f docker-compose.production.yml `
  config

docker-compose.production.yml expects Tengwar's external app-network and externally managed embedding and reranker services. It does not start extra model containers.

Tengwar development integration

For the integrated development stack, run the Tengwar-owned recipe from the Tengwar checkout:

cd ../Tengwar
just thorondor-deploy

The recipe builds the three first-party :local images from the sibling Thorondor checkout when selected, attaches the Thorondor orchestrator and chunker to tengwar-shared, points them at Tengwar's embedding and reranker services, and waits for Thorondor /health. The endpoint checks only the orchestrator process and performs no dependency or public-web request. The recipe also generates and preserves the managed CRAWL4AI_API_KEY. It recreates only the Thorondor Compose project; do not use down -v or remove the shared network when switching versions.

Use /health for recurring service health. Tengwar's MCP registry separately pins the discovered web_search descriptor checksum and withholds the tool when that descriptor changes; reapprove a changed descriptor through the admin MCP workflow before expecting mcp_thorondor_web_search to be exposed.

REST API

POST /v1/search

POST http://localhost:8080/v1/search
Content-Type: application/json

{
  "query": "what changed in the EU AI Act timeline in 2025",
  "search_profile": "research",
  "token_budget": 4000,
  "max_urls": 8,
  "max_passages": 10,
  "freshness": "year",
  "domains": ["eur-lex.europa.eu", "europarl.europa.eu"],
  "exclude_domains": ["example.com"],
  "decompose": true,
  "include_raw_markdown": false
}

PowerShell example:

$body = @{
  query         = "what changed in the EU AI Act timeline in 2025"
  search_profile = "research"
  token_budget  = 4000
} | ConvertTo-Json

Invoke-RestMethod http://localhost:8080/v1/search `
  -Method Post `
  -ContentType "application/json" `
  -Body $body

curl example:

curl -s -X POST http://localhost:8080/v1/search \
  -H "Content-Type: application/json" \
  -d '{"query":"EU AI Act 2025 timeline","search_profile":"research","token_budget":4000}' \
  | python -m json.tool

Request fields

Field	Type	Default	Description
query	string (required)	—	Information need, max 500 chars. Reranking always scores against this string, not sub-queries.
search_profile	"quick" | "research" | "deep"	null	Selects a preset bundle of token_budget, max_urls, max_passages defaults.
token_budget	int 1–16000	profile or server default	Controls passage assembly, not crawl cost.
max_urls	int 1–20	profile or server default	Cap on URLs selected before crawling.
max_passages	int 1–50	profile default	Hard cap on returned passages after token budgeting.
decompose	bool	true	Whether the optional LLM planner may expand the query into sub-queries. No-op when LLM_ENDPOINT is blank.
freshness	"day" | "week" | "month" | "year"	null	Discovery time-range hint passed to SearXNG.
domains	list[str]	null	Per-call domain allowlist. Combined with ALLOWLIST_ONLY if set.
exclude_domains	list[str]	null	Per-call domain blocklist merged with DOMAIN_BLOCKLIST.
include_raw_markdown	bool	false	Attach original and exact cleaned Markdown plus document identity for cited pages in raw_markdown[].

Response envelope (SearchResponse)

{
  "query": "...",
  "schema_version": "thorondor.search.v1",
  "passages": [
    {
      "text": "...",
      "score": 0.87,
      "token_count": 134,
      "citation_id": 1,
      "start_index": 412,
      "end_index": 1187,
      "verbatim": true,
      "document_id": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
      "evidence_id": "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
      "section_heading": "Implementation timeline",
      "provenance": "external_web",
      "trust": "untrusted",
      "score_components": [
        {
          "name": "reranker",
          "score": 0.87,
          "strategy": "external-reranker@1"
        }
      ]
    }
  ],
  "citations": [
    {
      "id": 1,
      "url": "https://...",
      "title": "...",
      "published": "2026-08-04T10:15:00Z",
      "modified_at": null,
      "document_id": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
      "evidence_spans": [
        {
          "start_index": 412,
          "end_index": 1187,
          "verbatim": true,
          "evidence_id": "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
          "section_heading": "Implementation timeline"
        }
      ],
      "metadata": {
        "published_at": {
          "value": "2026-08-04T10:15:00Z",
          "source": "json_ld",
          "confidence": "high"
        },
        "conflicts": []
      }
    }
  ],
  "stats": {
    "sub_queries": ["..."],
    "subquery_diagnostics": [
      {
        "query": "...",
        "status": "ok",
        "elapsed_ms": 124,
        "result_count": 10,
        "failure_reason": null
      }
    ],
    "discovery_status": "degraded",
    "unresponsive_engines": [
      { "engine": "mojeek", "reason": "access denied" }
    ],
    "urls_discovered": 18,
    "urls_selected": 6,
    "evidence_quality_strategy": null,
    "evidence_quality_chunks_dropped": 0,
    "urls_crawled_ok": 5,
    "urls_crawled_failed": 1,
    "chunks_produced": 43,
    "chunks_sent_to_reranker": 43,
    "reranked": true,
    "tokens_returned": 3812,
    "elapsed_ms": 4200,
    "reason": null
  },
  "raw_markdown": null
}

Key stats fields:

Field	Meaning
discovery_status	ok when SearXNG reports no engine failures, degraded when results remain usable despite failed engines, or unavailable when engine failures leave no usable discovery results.
unresponsive_engines	SearXNG engine names and reported failure or suspension reasons.
subquery_diagnostics	Bounded status, elapsed time, and result count for each SearXNG sub-query attempt. A failed attempt does not discard successful sibling attempts.
engine_contributions	Result contributions before URL merging, counted per sub-query and plural SearXNG engine. One URL found in three sub-queries contributes three times. SearXNG does not expose true per-engine latency, so Thorondor does not invent it.
reranked	false when the reranker was unreachable; passages are still returned sorted by position.
reason	Non-null closed enum when the search ended before normal assembly: search_provider_unavailable, no_results_from_discovery, no_urls_after_selection, all_crawls_failed, no_chunks_after_dedup, no_chunks_after_rerank, no_evidence_after_quality_gate, or no_evidence_after_output_budget.
embedding_degraded	true when the chunker fell back to token-based splitting because embeddings failed.
url_diagnostics	Selected-first URL decisions with contributing sub-queries, plural engines, positions, best upstream score, lexical score, and selection/filter reason. The list is bounded to 50 items and 64 KiB.
url_diagnostics_omitted	Counts of decisions omitted from the URL diagnostic item/byte budget, grouped by reason.
evidence_quality_drops	Bounded counts for deterministic navigation, footer, and generic-link evidence rejection when EVIDENCE_QUALITY_ENABLED=true. The gate is disabled by default pending broader quality measurement.
evidence_items_omitted / evidence_bytes_omitted	Evidence rejected by the independent 50-item, 64-KiB-serialized-item, and 256-KiB-serialized-list response envelope. A top-ranked oversized item is shortened safely before omission is considered.
raw_markdown_omitted	Cited raw-Markdown entries omitted by the independent 20-item, 256-KiB-per-item, and 512-KiB-total envelope.

An empty-passage response with stats.reason set is a normal 200, not an error. The caller should read reason rather than interpreting passages.length == 0 alone. search_provider_unavailable means SearXNG reported at least one failed engine and returned no usable discovery results; a healthy empty search remains no_results_from_discovery.

For verbatim=true, start_index and end_index are Unicode code-point offsets into the exact cleaned Markdown and the end is exclusive. evidence_id is emitted only when passage.text == cleaned_markdown[start_index:end_index] has been verified. document_id remains stable across ranking and chunk ordering when the final URL and exact cleaned document bytes are unchanged. Citation metadata keeps publication and modification separate, identifies the selected field source/confidence, and retains bounded conflicts instead of silently discarding them. passage.score remains the final ordering score; score_components identifies only the reranker, partial-batch floor, or position fallback actually used to produce it.

POST /search is a backwards-compatible alias for POST /v1/search.

POST /v1/fetch

Use POST /v1/fetch when the agent already knows one to four target URLs. The response is a thorondor.fetch.v1 envelope with one bounded result per URL, aggregate terminal-outcome counts, retryability, final URL, status, content type, allowlisted validators, metadata, and links. Markdown, JavaScript rendering, links, and metadata are requested by default; PDF, document, and raw HTML capabilities are explicit. Raw HTML is size-capped and never appears in ordinary search responses.

Optional structured_formats accepts the closed values links, tables, json_ld, and json_schema. Each format has its own result entry, status, omissions, untrusted labels, and source-document identity. Links and table elements carry exact source-HTML spans; each typed JSON-LD field addresses the exact <script> element that declared it. json_ld emits only Article-family and Product documents. json_schema requires a bounded extraction_schema, validates model output against a small local subset, and accepts a leaf only when its value occurs in an exact cleaned-Markdown evidence span. Extraction is all-or-nothing: any schema or evidence failure suppresses model data and returns only closed validation diagnostics. It rejects remote references, combinators, arbitrary keywords, schemas over 16 KiB, depth over five, more than 32 properties, prompts over 128 KiB, outputs over 64 KiB, and crawl requests over four pages. Model work has a 20-second deadline and a process-owned four-request concurrency limit, using the configured LLM_ENDPOINT, LLM_MODEL, and optional LLM_API_KEY.

links requires the links capability, tables and json_schema require Markdown, and json_ld requires metadata. Duplicate, incompatible, or schema-less formats are rejected. Deterministic structured requests fetch live source HTML; schema-only requests use current cleaned Markdown without requesting an additional raw-HTML copy. Every structured request bypasses the page cache so a cached record without the addressed source cannot produce structured evidence. Consequently, watch, change, diff, stale reads, and revalidation are unavailable on the same request. All four profiles are explicit opt-ins. Omission counts are profile-specific: eligible HTTP links, table cells, JSON-LD blocks or fields, and schema leaves suppressed by response budgeting.

Terminal outcomes distinguish content from challenge or empty shells, robots refusal, timeout, rate limiting, unsafe redirects or targets, unsupported content or capabilities, oversized content, malformed upstream data, and local processing failures. Returned web content is always marked provenance: "external_web" and trust: "untrusted".

When PAGE_CACHE_ENABLED=true, each result also reports cache.state (fresh, stale, revalidated, or bypass), a new/same/changed/removed page transition, and bounded changed-section and line summaries when content changed. force_refresh=true performs a live comparison. stale_while_revalidate=true may return bounded stale content while refreshing only for ordinary non-watch reads. Background refreshes reacquire fetch admission and run under the fetch deadline. Browser-rendered requests use a full refetch and hash comparison because the hardened Crawl4AI API cannot accept request-supplied validators; an extractor that explicitly supports conditional revalidation may reuse the stored document after 304 Not Modified without extending its absolute retention deadline.

An optional watch evaluates one declarative target by a bounded compound CSS selector, role/name, or normalized text inside a named section. It compares text or one declared attribute and sets condition_met=true only for a uniquely resolved transition from the expected state to the desired state. Missing, ambiguous, unsupported, fetch-failure, and first-observation cases fail closed. No caller JavaScript or regular expression is accepted. Whole-page changes such as advertisements may still set change.state=changed, but cannot satisfy a target watch.

POST /v1/map and POST /v1/crawl

Use POST /v1/map to discover a deterministic, robots-aware URL set for one site. Use POST /v1/crawl for the same bounded traversal plus typed page results. Both resolve the seed first, scope the operation to its effective origin and seed directory by default, combine declared and common sitemaps with bounded breadth-first link traversal, and optionally add SearXNG site: discovery. Parent paths and subdomains require explicit opt-in; arbitrary external-origin crawling is not supported.

The shared request policy supports sitemap modes include, only, and skip; path include/exclude globs; query preservation, stripping, or exclusion; file-extension allowlists; and explicit depth, page, and discovered-URL limits. Crawl requests may use the same structured_formats contract for each successful page. Responses report every retained URL's sources, state transitions, terminal reason, untrusted sitemap modification metadata, aggregate fetch outcomes, robots state and network/cache source, warnings, and omissions. These operations are synchronous and intentionally limited to small site slices.

Durable crawl jobs

When CRAWL_JOBS_ENABLED=true, POST /v1/crawl/jobs runs the same bounded crawl policy through one durable local worker. The caller must send Idempotency-Key and X-Thorondor-Job-Scope; an identical scoped replay returns the existing job, while the same key with a different request returns 409. The scope is an ownership label for an already authenticated internal ingress, not authentication by itself.

Use GET /v1/crawl/jobs/{job_id} for status, GET /v1/crawl/jobs/{job_id}/results for byte-and-item-bounded cursor pages, and POST /v1/crawl/jobs/{job_id}/cancel for cooperative cancellation. All reads and cancellation require the same scope. Results survive an orchestrator restart, are deduplicated, and expire at an absolute deadline that polling does not extend. Recovery repeats the bounded crawl from its seed and suppresses already stored final-URL results; it is not checkpointed frontier resumption. A job attempt has its own bounded deadline and consumes one shared MAX_INFLIGHT_CRAWLS slot while running. The default advertised synchronous boundary is 10 pages; durable mode remains explicit and can be selected for any crawl when recovery or polling matters.

curl -s -X POST http://localhost:8080/v1/crawl/jobs \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: product-catalog-2026-08-07' \
  -H 'X-Thorondor-Job-Scope: internal-agent' \
  -d '{"url":"https://example.com/docs","max_pages":20}'

This REST-only job API does not add MCP tools, schedules, or webhooks. Status exposes bounded progress, observed failure summaries, warnings, and page results, but not the synchronous crawl route's full frontier URL records or aggregate statistics. The feature is disabled by default and uses SQLite under /var/lib/thorondor; enable only one orchestrator replica. New jobs return 429 when the configured record cap is full. Raw HTML requests are rejected unless CRAWL_JOB_RAW_HTML_ENABLED=true explicitly permits persistence.

MCP

Mount the MCP endpoint in your agent's configuration:

http://localhost:8080/mcp

Transport: streamable HTTP (stateless_http=True). The server ID is thorondor.

The exposed tools are web_search, web_fetch, web_map, and web_crawl. Each mirrors its versioned REST request and response contract. web_map returns URL-only discovery outcomes; web_crawl adds typed page evidence for the bounded site slice.

MCP product contract

The selected conformance claim is limited to the 2026-07-28 selected tool-server profile evidence. Modern server/discover currently exposes the pinned SDK capability map:

{
  "prompts": {"listChanged": true},
  "resources": {"subscribe": true, "listChanged": true},
  "tools": {"listChanged": true}
}

The prompt and resource handlers are retained SDK-provided empty primitives; Thorondor's product claim is for the four tools, not prompt/resource content, subscriptions, or list-change notifications. tools/list returns exactly web_search, web_fetch, web_map, and web_crawl.

Tool annotations are identical on the HTTP and native stdio surfaces:

Tools	Annotations
web_search, web_map, web_crawl	readOnlyHint=true, openWorldHint=true
web_fetch	readOnlyHint=false, destructiveHint=false, idempotentHint=false, openWorldHint=true

Successful and controlled operational tool outcomes are explicit CallToolResult values. They carry resultType="complete", the corresponding REST body in structuredContent, and isError=false for success or isError=true for an operational failure. Every modern result carries _meta["io.modelcontextprotocol/serverInfo"] with {"name":"thorondor", "version": THORONDOR_VERSION}. Cacheable SDK hints are explicit ttlMs=0 and cacheScope="private"; these hints do not enable persistence. Only the optional known-URL page cache can persist web_fetch data, and search, map, crawl, and structured requests remain live or bypass that cache.

Protocol and dispatch failures remain top-level JSON-RPC errors. The selected surface uses -32020 for an MCP header/envelope mismatch, -32022 for an unsupported protocol version with data.supported and data.requested, -32602 for invalid protocol parameters or an unknown tool, -32601 for a removed or unknown method, and sanitized -32603 for unexpected internal defects. Request-model validation inside a known tool is a structured invalid_request tool error with status 422; signature-level SDK validation before tool entry remains an SDK-generated isError=true tool result. HTTP status, route, dependency, retry, and typed outcome information for a known tool remains in its structured CallToolResult, rather than being converted into a transport-level exception. The final MAX_RESPONSE_BODY_BYTES check covers the complete serialized MCP result object, including content, structuredContent, isError, resultType, and identity metadata; it excludes JSON-RPC envelopes, HTTP headers, and transport framing. Oversized results become a bounded error and are not truncated into invalid structured evidence.

Modern 2026-07-28 requests carry the required protocol version and client capabilities, plus optional client information, in the reserved per-request _meta envelope. HTTP requests also require a matching MCP-Protocol-Version, Mcp-Method matching the JSON-RPC method, and, for tools/call, Mcp-Name matching the tool name. Accept must cover both application/json and text/event-stream, or use */*. A protocol, method, or tool-name mismatch is an HTTP 400 JSON-RPC -32020 error; an unacceptable Accept value is an HTTP 406 transport response. Optional whitespace around MCP header values is removed before comparison. Modern responses are sessionless and do not issue mcp-session-id. A legacy 2025 handshake remains valid without these modern routing headers; a non-handshake modern envelope without them is rejected rather than silently treated as legacy. The native thorondor-mcp command uses stdio framing and forwards the same four versioned REST contracts.

Selected conformance evidence

The pinned runner is Node 24.19.0 with @modelcontextprotocol/conformance@0.2.0-alpha.10. Install and run it from a fresh, empty output directory:

npm ci --prefix tools/mcp-conformance --ignore-scripts
PYTHONPATH=. .venv/bin/python scripts/run-mcp-conformance.py \
  --output-dir /tmp/thorondor-mcp-conformance

The profile contains 37 required scenarios. tools-list, server-sse-multiple-streams, and dns-rebinding-protection are direct gates; server-stateless and caching are mixed and evaluated check by check. Tool call, completion, prompt, resource, and input-required scenarios remain fixture-dependent or outside this selected product profile. Their output is retained for inspection but cannot establish a Thorondor production pass. The runner rejects reused output directories, records the manifest and exact result directories, and evaluates only artifacts from that manifest. Manifest claim/version drift, timeouts, invalid runner statuses, and scenario/artifact mismatches fail closed. A focused run such as --scenario tools-list is an explicitly scoped claim, not a full-profile result. Individual fixture scenarios may return nonzero while the evaluator still reports a clean selected gate result; those observations are not hidden. Mixed scenarios can also return nonzero because their informational failures and SHOULD observations remain visible; the evaluator uses the profile's per-check dispositions rather than treating that status as a blanket production failure.

The two list-change SHOULD observations are reported separately and are intentional non-gating deviations: Thorondor does not add autonomous subscription notifications, so clients refresh the advertised lists when needed. OAuth, resources, prompts, completion, Tasks, MCP Apps, sampling, roots, MRTR, and production diagnostic tools are outside this claim.

Example MCP tool call (Claude SDK style):

{
  "name": "web_search",
  "input": {
    "query": "EU AI Act enforcement 2025",
    "search_profile": "research",
    "token_budget": 4000
  }
}

The native stdio surface is the supported thorondor-mcp command. Set THORONDOR_BASE_URL when the orchestrator is not at its default http://localhost:8080. If the orchestrator uses a non-default MAX_RESPONSE_BODY_BYTES, provide the same value to the proxy process; the command applies it both while reading REST responses and while constructing the final MCP result.

Health and Observability

/health

{
  "status": "ok"
}

GET /health is the only health endpoint on both Thorondor-owned services. It reports local process availability and never calls SearXNG, Crawl4AI, the chunker, an embedding provider, a reranker, or the public web. Docker and deployment harnesses can poll it repeatedly without causing search, crawl, or model inference. Dependency failures are detected and returned by real search, fetch, map, and crawl requests through their normal typed outcomes and degradation behavior.

X-Request-ID

Every REST and MCP request generates or accepts X-Request-ID. The same ID is returned to the caller and forwarded to all downstream seams (SearXNG, Crawl4AI, chunker, embedding, reranker, and configured LLM operations).

Structured logging

One JSON log line is emitted per completed search:

{
  "level": "INFO",
  "logger": "orchestrator.pipeline",
  "message": "search_completed",
  "request_id": "a3f7...",
  "event": "search_completed",
  "query_hash": "4a8b1c2d",
  "sub_query_count": 2,
  "urls_discovered": 18,
  "urls_selected": 6,
  "urls_crawled_ok": 5,
  "urls_crawled_failed": 1,
  "reranked": true,
  "reason": null,
  "elapsed_ms": 4200
}

Logs never contain raw query text, page markdown, secret values, userinfo components, or URL query strings.

Configuration Reference

All keys must be present in .env (leave optional keys blank rather than deleting them). The Python settings loader raises RuntimeError on startup for any missing key.

Service URLs

Variable	Default	Description
SEARXNG_URL	http://searxng:8080	Internal SearXNG base URL used by the orchestrator.
CRAWL4AI_URL	http://crawl4ai:11235	Internal Crawl4AI API base URL.
CHUNKER_URL	http://chunker:8000	Internal semantic chunking service base URL.
SEARXNG_BASE_URL	http://searxng:8080/	External base URL passed to SearXNG for self-referencing links (Compose env).
ORCHESTRATOR_HOST	127.0.0.1	Host interface bound by Compose. Keep localhost for personal use; set 0.0.0.0 only behind firewall/auth.
ORCHESTRATOR_PORT	8080	Host port bound to the orchestrator container (Compose env).

Logging

Variable	Default	Description
LOG_LEVEL	INFO	Orchestrator log level.
CHUNKER_LOG_LEVEL	INFO	Chunker service log level.
PROXY_LOG_LEVEL	INFO	SSRF egress proxy log level.

Resource Envelope

Variable	Default	Description
MAX_REQUEST_BODY_BYTES	32768	Maximum REST or in-process MCP request payload.
MAX_RESPONSE_BODY_BYTES	2097152	Maximum serialized REST, MCP, Crawl4AI, or stdio-proxy response payload.
SEARCH_ROUTE_DEADLINE_S	120	End-to-end search deadline.
FETCH_ROUTE_DEADLINE_S	60	End-to-end known-URL fetch deadline.
MAP_ROUTE_DEADLINE_S	90	End-to-end site-map deadline.
SITE_CRAWL_ROUTE_DEADLINE_S	120	End-to-end bounded site-crawl deadline.
DISCOVERY_TIMEOUT_S	20	Query-planning and per-subquery discovery stage deadline.
CHUNK_TIMEOUT_S	45	Total chunking stage deadline.
MAX_INFLIGHT_SEARCHES	4	Process-wide search admission slots.
MAX_INFLIGHT_FETCHES	8	Process-wide fetch admission slots.
MAX_INFLIGHT_MAPS	4	Process-wide site-map admission slots.
MAX_INFLIGHT_CRAWLS	2	Process-wide bounded site-crawl admission slots.
ADMISSION_WAIT_S	0.05	Maximum wait for a route or crawler slot before rejection.
ADMISSION_RETRY_AFTER_S	1	Bounded Retry-After value returned with HTTP 429.
MAX_INTERNAL_FANOUT	20	Shared cap covering URL, subquery, crawl, chunk, and profile fan-out.
MAX_CONTENT_BYTES	262144	Maximum combined Markdown and HTML bytes retained per fetched page.
CHUNK_CONCURRENCY	4	Process-owned concurrent chunk-service requests.

Search Profiles

Variable	Default	Description
MAX_SUBQUERIES	3	Maximum sub-queries the planner may produce (1–8).
DEFAULT_TOKEN_BUDGET	4000	Token budget used when no profile and no explicit token_budget is given.
SEARCH_PROFILE_QUICK_TOKEN_BUDGET	2000	Token budget for the quick profile.
SEARCH_PROFILE_QUICK_MAX_URLS	5	Max URLs for quick.
SEARCH_PROFILE_QUICK_MAX_PASSAGES	5	Max passages for quick.
SEARCH_PROFILE_RESEARCH_TOKEN_BUDGET	8000	Token budget for research.
SEARCH_PROFILE_RESEARCH_MAX_URLS	12	Max URLs for research.
SEARCH_PROFILE_RESEARCH_MAX_PASSAGES	20	Max passages for research.
SEARCH_PROFILE_DEEP_TOKEN_BUDGET	16000	Token budget for deep.
SEARCH_PROFILE_DEEP_MAX_URLS	20	Max URLs for deep.
SEARCH_PROFILE_DEEP_MAX_PASSAGES	40	Max passages for deep.

Crawl Tuning

Variable	Default	Description
MAX_URLS	6	Default max URLs when no profile and no explicit max_urls is given.
CRAWL_CONCURRENCY	4	Total concurrent Crawl4AI requests (1–20).
CRAWL_PER_HOST_CONCURRENCY	1	Concurrent requests per hostname.
CRAWL_TIMEOUT_S	15	Per-URL crawl timeout in seconds.
CRAWL_RESPECT_ROBOTS_TXT	true	Whether Crawl4AI observes robots.txt.
CRAWLER_USER_AGENT	ThorondorBot/1.0 (+https://github.com/FeanorsCodeSL/thorondor)	Honest outbound crawler identity with a contact URL.
CRAWLER_ROBOTS_USER_AGENT	ThorondorBot	Token reserved for robots policy matching and required to appear in the outbound identity.
SITE_DEFAULT_DELAY_S	0.5	Minimum spacing between target requests to one host.
SITE_MAX_JITTER_S	0.25	Maximum deterministic per-host spacing jitter.
SITE_MAX_COOLDOWN_S	300	Maximum adaptive or Retry-After cooldown in seconds.
ROBOTS_CACHE_TTL_S	86400	Absolute in-memory robots snapshot TTL, capped at 24 hours.
MAX_ROBOTS_BYTES	262144	Maximum robots response bytes parsed by map/crawl.
MAX_SITEMAP_BYTES	262144	Maximum bytes parsed from one sitemap document.
MAX_SITEMAP_ENTRIES	500	Maximum retained entries per sitemap document.
MAX_SITEMAP_DOCUMENTS	16	Maximum sitemap documents fetched per operation.

Optional Page Cache and Change Detection

Variable	Default	Description
PAGE_CACHE_ENABLED	false	Enable persistent caching for /v1/fetch and web_fetch; ordinary search is never persisted.
PAGE_CACHE_PATH	/var/lib/thorondor/page-cache.sqlite3	SQLite file in the dedicated thorondor-page-cache volume.
PAGE_CACHE_TTL_S	300	Fresh lifetime from fetch time; reads never extend it.
PAGE_CACHE_STALE_S	900	Additional bounded stale-while-revalidate window.
PAGE_CACHE_RETENTION_S	604800	Absolute page and target-snapshot retention from the full content fetch; reads and 304 responses do not extend it.
PAGE_CACHE_RAW_HTML_ENABLED	false	Permit raw HTML persistence when the request also asks for raw_html.
PAGE_DIFF_MAX_INPUT_LINES	2000	Per-document line cap before returning summary-only truncated diff metadata.
PAGE_DIFF_MAX_OPERATIONS	1000000	Maximum old-line × new-line comparison work.
PAGE_DIFF_MAX_OUTPUT_LINES	24	Maximum added and removed lines returned; hard cap 24.

Optional Durable Crawl Jobs

Variable	Default	Description
CRAWL_JOBS_ENABLED	false	Enable the REST-only, single-replica SQLite worker.
CRAWL_JOB_PATH	/var/lib/thorondor/crawl-jobs.sqlite3	Durable job and result database.
CRAWL_SYNC_MAX_PAGES	10	Advertised synchronous/durable product boundary.
CRAWL_JOB_RETENTION_S	86400	Absolute terminal-result lifetime.
CRAWL_JOB_EXPIRED_TOMBSTONE_S	3600	Typed expired-status lifetime after result removal.
CRAWL_JOB_MAX_ATTEMPTS	3	Total attempts for retryable whole-job failures.
CRAWL_JOB_RETRY_BASE_S	1	Exponential retry base in seconds.
CRAWL_JOB_ATTEMPT_DEADLINE_S	600	Maximum wall time for one durable attempt.
CRAWL_JOB_MAX_RECORDS	100	Maximum retained jobs, including terminal and expired records.
CRAWL_JOB_RAW_HTML_ENABLED	false	Permit raw HTML in durable results when requested.
CRAWL_JOB_MAX_INFLIGHT_REQUESTS	16	Concurrent job create/status/results/cancel database operations.
CRAWL_JOB_RESULT_PAGE_MAX_ITEMS	10	Maximum items per cursor page.
CRAWL_JOB_RESULT_PAGE_MAX_BYTES	524288	Maximum stored result item and cursor-page bytes.

Markdown Extraction

Variable	Default	Description
MARKDOWN_EXTRACTOR	trafilatura	Must be trafilatura (only supported value).
MARKDOWN_EXTRACTOR_FAVOR_RECALL	true	Trafilatura include_tables-style recall mode.
MARKDOWN_EXTRACTOR_INCLUDE_COMMENTS	false	Include comment sections in extracted text.
MARKDOWN_EXTRACTOR_INCLUDE_TABLES	true	Include HTML table content.
MARKDOWN_EXTRACTOR_DEDUPLICATE	true	Enable trafilatura's internal deduplication.

Chunker Tuning

Variable	Default	Description
CHUNKER_DEFAULT_STRATEGY_VERSION	cluster-semantic@1	Default chunking strategy version sent to the chunker service.
CHUNKER_MAX_SEGMENTS_DP	10000	Segment count above which the chunker falls back from DP to greedy-semantic (OOM guard).
REWARD_CACHE_MAX_SIZE	100000	LRU cache bound for the DP reward function.
CHUNKER_MEM_LIMIT	768m	Docker memory limit for the chunker container (Compose env).

Embedding (used by the chunker service)

Variable	Default	Description
EMBEDDING_ENDPOINT	http://embedding:80	OpenAI-compatible /v1/embeddings server base URL.
EMBEDDING_MODEL	BAAI/bge-m3	Model name sent in embedding requests.
EMBEDDING_MODEL_REVISION	5617a9f61b028005a4858fdac845db406aefb181	Immutable Hub revision used by the bundled TEI embedding container.
EMBEDDING_API_KEY	(blank)	Optional bearer token for the embedding server.
EMBEDDING_BATCH_SIZE	64	Texts per embedding API call.
EMBEDDING_TIMEOUT_S	60	Embedding request timeout in seconds.

Reranker

Variable	Default	Description
RERANKER_ENDPOINT	http://reranker:80	Reranker server base URL.
RERANKER_MODEL	BAAI/bge-reranker-v2-m3	Model name sent in rerank requests.
RERANKER_MODEL_REVISION	953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e	Immutable Hub revision used by the bundled TEI reranker container.
RERANKER_PATH	/rerank	Rerank endpoint path.
RERANKER_API_KEY	(blank)	Optional bearer token for the reranker.
RERANKER_BATCH_SIZE	32	Chunks per reranker API call.
RERANKER_TIMEOUT_S	30	Reranker request timeout in seconds.
RELEVANCE_SCORE_FLOOR	0.0	Passages with reranker score ≤ this value are dropped after reranking (0.0 disables the filter).
EVIDENCE_QUALITY_ENABLED	false	Enable the measured evidence-quality@1 structural filter. Disabled by default until a broader independent corpus supports activation.

Optional LLM Operations

Variable	Default	Description
LLM_ENDPOINT	(blank)	Optional OpenAI-compatible chat completions server for query decomposition and explicit JSON-Schema extraction. Leave blank to use IdentityPlanner and report model extraction as unsupported.
LLM_MODEL	(blank)	Model name sent for configured LLM operations. Required when LLM_ENDPOINT is set.
LLM_API_KEY	(blank)	Optional bearer token for configured LLM operations.

URL Safety and SSRF Protection

Variable	Default	Description
URL_SAFETY_BLOCKED_IP_CATEGORIES	loopback,link_local,private,reserved,multicast,unspecified	IP address categories blocked before crawl.
URL_SAFETY_BLOCKED_SPECIAL_IPS	169.254.169.254,fd00:ec2::254	Specific IPs blocked regardless of category (AWS/cloud metadata endpoints).
URL_SAFETY_NAT64_NETWORKS	64:ff9b::/96	IPv6 NAT64 networks; embedded IPv4 is extracted and re-checked.
URL_SAFETY_SIX_TO_FOUR_NETWORKS	2002::/16	IPv6 6-to-4 networks; embedded IPv4 is extracted and re-checked.
URL_SAFETY_IPV4_COMPAT_NETWORKS	::/96	IPv4-compatible IPv6 networks; embedded IPv4 is extracted and re-checked.

MCP Transport Security

Variable	Default	Description
MCP_ALLOWED_HOSTS	127.0.0.1:*,localhost:*,[::1]:*,thorondor:*,orchestrator:*	Host headers accepted by the MCP streamable HTTP endpoint.
MCP_ALLOWED_ORIGINS	http://127.0.0.1:*,http://localhost:*,http://[::1]:*,http://thorondor:*,http://orchestrator:*	Origin headers accepted by the MCP streamable HTTP endpoint when an Origin header is present.

Domain Allow/Block Lists

Variable	Default	Description
DOMAIN_BLOCKLIST	(blank)	Comma-separated domains excluded from all searches. Supports subdomain matching.
DOMAIN_ALLOWLIST	(blank)	Comma-separated domains permitted when ALLOWLIST_ONLY=true.
ALLOWLIST_ONLY	false	When true, restricts all crawls to DOMAIN_ALLOWLIST + per-call domains.

SSRF Egress Proxy

Variable	Default	Description
PROXY_HOST	0.0.0.0	Proxy listen address.
PROXY_PORT	8888	Proxy listen port.
PROXY_MAX_HEADER_BYTES	65536	Max bytes read for HTTP request headers.
PROXY_READ_CHUNK_BYTES	4096	Read chunk size for incoming data.
PROXY_RELAY_CHUNK_BYTES	65536	Relay chunk size for proxied data.
PROXY_BLOCKED_IP_CATEGORIES	(same as URL_SAFETY)	IP categories blocked by the proxy itself.
PROXY_BLOCKED_SPECIAL_IPS	(same as URL_SAFETY)	Special IPs blocked by the proxy.
PROXY_NAT64_NETWORKS	64:ff9b::/96	NAT64 networks for the proxy's IP expansion.
PROXY_SIX_TO_FOUR_NETWORKS	2002::/16	6-to-4 networks for the proxy's IP expansion.
PROXY_IPV4_COMPAT_NETWORKS	::/96	IPv4-compat networks for the proxy's IP expansion.

Crawl4AI Egress Safety

Variable	Default	Description
CRAWL4AI_ALLOW_INTERNAL_URLS	false	Keeps Crawl4AI 0.9.2's DNS-pinning proxy restricted to globally routable targets. Do not enable it in managed deployments.

Crawl4AI 0.9.2 owns its browser's connect-time DNS pinning and therefore has a dedicated outbound network selected explicitly as its default gateway. Its API remains unpublished on a separate internal control network shared only with the orchestrator. External proxy variables are intentionally absent because the hardened server replaces caller proxy configuration with its loopback pinning proxy.

API Keys

Variable	Default	Description
SEARXNG_SECRET	(blank — generated by deploy script)	SearXNG instance secret. Generated by deploy.ps1; never commit a real value.
SEARXNG_API_KEY	(blank)	Optional bearer token accepted by SearXNG for its JSON API.
CRAWL4AI_API_KEY	(blank — generated by deploy script)	Bearer token shared by the orchestrator and managed Crawl4AI container. BYO endpoints may leave it blank only when unauthenticated.
CHUNKER_API_KEY	(blank)	Optional bearer token for the chunking service.
RERANKER_API_KEY	(blank)	Optional bearer token for the reranker.
EMBEDDING_API_KEY	(blank)	Optional bearer token for the embedding server (read by chunker).
LLM_API_KEY	(blank)	Optional bearer token for configured LLM operations.

llama.cpp Overrides (.env.llamacpp)

Variable	Default	Description
LLAMACPP_IMAGE	ghcr.io/ggml-org/llama.cpp:server@sha256:2244d2b2...	Pinned llama.cpp build b10454 server image SHA.
LLAMACPP_EMBEDDING_MODEL	/models/bge-m3.gguf	Container path to the embedding GGUF (mounted from ./models).
LLAMACPP_EMBEDDING_ALIAS	bge-m3	Model alias used in embedding API requests.
LLAMACPP_EMBEDDING_POOLING	cls	Pooling strategy for the embedding model.
LLAMACPP_EMBEDDING_CONTEXT	8192	Context length for the embedding server.
LLAMACPP_RERANKER_MODEL	/models/bge-reranker-v2-m3.gguf	Container path to the reranker GGUF.
LLAMACPP_RERANKER_ALIAS	bge-reranker-v2-m3	Model alias used in reranker API requests.
LLAMACPP_RERANKER_CONTEXT	8192	Context length for the reranker server.
LLAMACPP_BATCH	8192	Batch size for llama.cpp.
LLAMACPP_UBATCH	8192	Micro-batch size for llama.cpp.

Development Checks

# Create or activate a Python environment, then install test dependencies
python -m pip install `
  -r orchestrator\requirements-dev.lock `
  -r semantic-chunking-service\requirements-dev.lock

# Run all tests
python -m pytest semantic-chunking-service\tests orchestrator\tests -v

# Run local SonarQube analysis
# Requires SONAR_HOST_URL and SONAR_TOKEN in the environment.
.\scripts\sonar.ps1

# Validate base Compose config
docker compose --env-file .env -f docker-compose.yml config

# Validate llama.cpp overlay config
docker compose --env-file .env --env-file .env.llamacpp `
  -f docker-compose.yml -f docker-compose.llamacpp.yml `
  --profile llamacpp-models config

# Check release guard (CI parity check)
.\scripts\check-release-guard.ps1

Troubleshooting

Docker pipe error (dockerDesktopLinuxEngine not found) Start Docker Desktop and wait until its system tray icon reports "running". Then re-run the deploy script.

Missing GGUF files deploy-llamacpp.ps1 validates model paths before starting Compose. Either place the files under models\ matching the paths in .env.llamacpp, or edit .env.llamacpp to point to your actual filenames.

thorondor download-models verifies the pinned SHA-256 for both default GGUF files. On a mismatch it preserves the existing file and exits non-zero; remove or replace that file only after confirming its provenance.

stats.reranked=false in search response The reranker was unreachable at query time. Passages are returned sorted by position (no semantic ranking). The service continues to function; fix the reranker and passages will be reranked again.

Port conflict on 8080 Change ORCHESTRATOR_PORT in .env to an available port, then restart: docker compose up -d orchestrator.

Expose beyond localhost Compose binds the orchestrator to 127.0.0.1 by default. To expose it to another host, set ORCHESTRATOR_HOST=0.0.0.0 and put the service behind TLS, authentication, and rate limiting first. Thorondor does not implement endpoint authentication itself.

SEARXNG_SECRET not set error Run .\scripts\deploy.ps1 (or deploy-llamacpp.ps1) rather than docker compose up directly — the deploy script generates the secret when the field is blank.

Crawl4AI returns 401 or is unreachable Run the deploy script so CRAWL4AI_API_KEY is generated and passed to both services. Keep CRAWL4AI_ALLOW_INTERNAL_URLS=false; the API is reachable only through the dedicated internal control network.

All crawls failing (urls_crawled_ok=0) Check Crawl4AI logs: docker compose logs crawl4ai. Common causes are missing crawl-egress routing, DNS failures, robots.txt refusals (CRAWL_RESPECT_ROBOTS_TXT=true), or slow sites. Increase CRAWL_TIMEOUT_S only after confirming outbound routing and DNS.

No results from SearXNG SearXNG may be rate-limited or blocking engines may be unavailable. Check docker compose logs searxng. The SearXNG web UI is not published by the default Compose stack; inspect it from inside the Compose network or add a temporary local-only port mapping during debugging.

License

MIT — see LICENSE. Third-party notices: THIRD-PARTY-NOTICES.md.