Skip to content

Add opt-in explainable query workflows, evidence capture and snapshot replay - #68

Open
kennethbruskiewicz wants to merge 9 commits into
mainfrom
codex/explainable-query-workflows
Open

kennethbruskiewicz wants to merge 9 commits into
mainfrom
codex/explainable-query-workflows

Conversation

@kennethbruskiewicz

@kennethbruskiewicz kennethbruskiewicz commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This branch adds an experimental, opt-in layer for explainable queries on top of the toolkit. It covers composing queries, keeping the evidence behind each answer, and replaying that evidence without live services. Everything is off by default. With nothing enabled, existing functions return the same values as before.

The work follows the CLI/MCP telemetry split that landed in #55 and is now rebased onto it.

What's included

Query workflows (TCT/workflow.py, TCT/interfaces/WORKFLOWS.md)

Recipes are JSON DAGs of named operations. References pass data between steps, and no arbitrary Python runs. Each output row keeps its lineage, and remote TRAPI responses are kept whole rather than flattened into merged edges. A run can be saved as a portable evidence snapshot. examples/workflows/offline_pathway.json shows a recipe that runs fully offline.

Evidence capture and rollup (TCT/evidence.py, TCT/service_capture.py)

Buffered HTTP calls to NameRes, NodeNorm, MetaKG, KP info and TRAPI endpoints are observed without replacing their response objects. A deterministic rollup summarizes the captured evidence per query, and golden fixtures under tests/fixtures/rollup/ pin its output.

Execution controls and recording (TCT/execution.py, TCT/journal.py)

A scoped execution context sets timeouts, recording and caching for a block of calls. Recording writes immutable artifacts and a transactional index to a local store when it's turned on.

TRAPI representations and validation (TCT/trapi_representation.py, TCT/trapi_validation.py)

Shared query tools can return optional TRAPI views of their native results. Each view lists the limits of the conversion, such as merged edges that can't recover provider bindings. Validation runs offline against the upstream TRAPI 1.5.0 and 2.0.0-beta2 schemas. Those schemas are vendored under TCT/schemas/ with their Apache-2.0 license and checked against pinned SHA-256 checksums. They account for about 3.4k of the added lines.

Response caching and snapshot replay (TCT/response_cache.py)

HTTP response reuse is opt-in. The cache key canonicalizes only JSON object key order. Arrays, query IDs, predicate order, URLs and literal values are kept as sent. Snapshots replay deterministically, though a replayed snapshot isn't proof that an answer is correct.

The MCP and CLI tool surfaces (TCT/interfaces/tools.py, mcp.py, cli.py) expose the new options. The MCP contract tests cover the added parameters.

Testing

A local pytest run gives 336 passed and 1 skipped, and codespell passes.

Three live-service tests started failing after main last went green on Sep 16. The cause is upstream changes, not this branch. A plain requests.get shows the same responses.

  • NameRes on Babel 2026jul22 (CI, test and RENCI) reports Solr stats at the top level and indexes about 331M names. Prod is still on 2025sep1 with about 425M under solr. test_nameres_status now accepts both shapes and uses a 300M floor.
  • Node Annotator no longer returns clinical_approval for CHEBI:15377. The expectation drops the volatile warning and clinical fields, the same change as draft test: review runtime isolation and Node Annotator expectations #63.

Codespell skips TCT/schemas/, since those files are checksum-pinned upstream copies and their typos come from TRAPI itself.

Open questions

  • Whether the workflow and evidence layer belongs in the core TCT package or in a separate optional extra.
  • Whether vendoring the TRAPI schemas is acceptable, or whether they should be fetched and pinned at install time.
  • Which TRAPI versions need to stay supported as 2.0 moves out of beta.

🤖 Generated with Claude Code

kennethbruskiewicz and others added 9 commits November 13, 2026 12:14
… tests

The vendored TRAPI schemas are checksum-pinned upstream files, so codespell
now skips TCT/schemas instead of editing them.

NameRes on Babel 2026jul22 reports Solr stats at the top level and indexes
~331M names (2025sep1 had ~425M under "solr"). The status test accepts both
shapes and uses a 300M sanity floor.

Node Annotator no longer returns clinical_approval for CHEBI:15377. The
expectation drops the volatile warning and clinical fields, matching #63.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the conflict in TCT/interfaces/mcp.py with #64 (FastMCP-only server).

The tool wrapper keeps this branch's optional `execution` argument and its
handling of recording, validation and I/O errors. It now raises FastMCP's
ToolError and registers the readOnlyHint/openWorldHint annotations from #64.

The four tools added on this branch (validate_query_workflow,
run_query_workflow, replay_query_workflow, evidence) join _READ_ONLY_TOOLS.
None of them changes the structures passed in, which is the criterion #64
uses, and its annotation test expects every non-mutating tool to be listed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant