Skip to content

Omnicorp: build curie pairs in numpy, fix set-node KeyError, disable GC - #267

Open
maximusunc wants to merge 1 commit into
develfrom
claude/keen-mccarthy-ptmnxx
Open

maximusunc wants to merge 1 commit into
develfrom
claude/keen-mccarthy-ptmnxx

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

The overlay held every curie pair as a dict of curie-tuple keys to sets
of (result, analysis) tuples, built each LMDB key with make_key, and
read shared counts one transaction per 1000-key batch. On a large
creative query (20k results, 5.2M pairs, 1.2M co-occurrence edges) that
took ~29s, about 12s of it the cyclic GC rescanning those objects.

  • generate_curie_pairs encodes pairs as ints over local node ids
    (assigned in sorted curie order) and returns a CuriePairs that groups
    the distinct (pair, analysis) links with numpy.
  • read_shared_counts does all lookups in one read txn, in LMDB key
    order; shared_count_query wraps it.
  • add_shared_pmid_counts takes (subject, object, count, refs) tuples and
    resolves each analysis's OMNICORP support graph once, not per pair.
  • aragorn_omnicorp disables the cyclic GC for the task; the message and
    everything built from it are acyclic.

Same benchmark: ~29s -> ~10s, and the overlay's peak memory on top of
the loaded message drops from ~2.3 GB to ~0.8 GB. Output is unchanged
(same co-occurrence edges per analysis and node counts).

Fix: curies bound to a set-interpretation qnode were paired without
checking they have omnicorp data, so one missing from the curies LMDB
failed the whole query with a KeyError. They are now filtered like
every other node, and self-pairs are skipped.

Drops generate_curie_pairs' unused max_pairs cap and make_key.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01S2yuRD7fu2tQ9a89t5V54p

The overlay held every curie pair as a dict of curie-tuple keys to sets
of (result, analysis) tuples, built each LMDB key with make_key, and
read shared counts one transaction per 1000-key batch. On a large
creative query (20k results, 5.2M pairs, 1.2M co-occurrence edges) that
took ~29s, about 12s of it the cyclic GC rescanning those objects.

- generate_curie_pairs encodes pairs as ints over local node ids
  (assigned in sorted curie order) and returns a CuriePairs that groups
  the distinct (pair, analysis) links with numpy.
- read_shared_counts does all lookups in one read txn, in LMDB key
  order; shared_count_query wraps it.
- add_shared_pmid_counts takes (subject, object, count, refs) tuples and
  resolves each analysis's OMNICORP support graph once, not per pair.
- aragorn_omnicorp disables the cyclic GC for the task; the message and
  everything built from it are acyclic.

Same benchmark: ~29s -> ~10s, and the overlay's peak memory on top of
the loaded message drops from ~2.3 GB to ~0.8 GB. Output is unchanged
(same co-occurrence edges per analysis and node counts).

Fix: curies bound to a set-interpretation qnode were paired without
checking they have omnicorp data, so one missing from the curies LMDB
failed the whole query with a KeyError. They are now filtered like
every other node, and self-pairs are skipped.

Drops generate_curie_pairs' unused max_pairs cap and make_key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRD7fu2tQ9a89t5V54p
@codecov

codecov Bot commented Nov 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.72180% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.88%. Comparing base (3a1e010) to head (e6b0251).
⚠️ Report is 127 commits behind head on devel.

Files with missing lines Patch % Lines
workers/aragorn_omnicorp/worker.py 88.72% 8 Missing and 7 partials ⚠️
Files with missing lines Coverage Δ
workers/aragorn_omnicorp/worker.py 88.44% <88.72%> (+6.80%) ⬆️

... and 27 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0532fcd...e6b0251. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants