Repository navigation
Omnicorp: build curie pairs in numpy, fix set-node KeyError, disable GC - #267
Open
maximusunc wants to merge 1 commit into
Open
maximusunc wants to merge 1 commit into
maximusunc wants to merge 1 commit into
Conversation
The overlay held every curie pair as a dict of curie-tuple keys to sets of (result, analysis) tuples, built each LMDB key with make_key, and read shared counts one transaction per 1000-key batch. On a large creative query (20k results, 5.2M pairs, 1.2M co-occurrence edges) that took ~29s, about 12s of it the cyclic GC rescanning those objects. - generate_curie_pairs encodes pairs as ints over local node ids (assigned in sorted curie order) and returns a CuriePairs that groups the distinct (pair, analysis) links with numpy. - read_shared_counts does all lookups in one read txn, in LMDB key order; shared_count_query wraps it. - add_shared_pmid_counts takes (subject, object, count, refs) tuples and resolves each analysis's OMNICORP support graph once, not per pair. - aragorn_omnicorp disables the cyclic GC for the task; the message and everything built from it are acyclic. Same benchmark: ~29s -> ~10s, and the overlay's peak memory on top of the loaded message drops from ~2.3 GB to ~0.8 GB. Output is unchanged (same co-occurrence edges per analysis and node counts). Fix: curies bound to a set-interpretation qnode were paired without checking they have omnicorp data, so one missing from the curies LMDB failed the whole query with a KeyError. They are now filtered like every other node, and self-pairs are skipped. Drops generate_curie_pairs' unused max_pairs cap and make_key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRD7fu2tQ9a89t5V54p
Codecov Report❌ Patch coverage is
... and 27 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The overlay held every curie pair as a dict of curie-tuple keys to sets
of (result, analysis) tuples, built each LMDB key with make_key, and
read shared counts one transaction per 1000-key batch. On a large
creative query (20k results, 5.2M pairs, 1.2M co-occurrence edges) that
took ~29s, about 12s of it the cyclic GC rescanning those objects.
(assigned in sorted curie order) and returns a CuriePairs that groups
the distinct (pair, analysis) links with numpy.
order; shared_count_query wraps it.
resolves each analysis's OMNICORP support graph once, not per pair.
everything built from it are acyclic.
Same benchmark: ~29s -> ~10s, and the overlay's peak memory on top of
the loaded message drops from ~2.3 GB to ~0.8 GB. Output is unchanged
(same co-occurrence edges per analysis and node counts).
Fix: curies bound to a set-interpretation qnode were paired without
checking they have omnicorp data, so one missing from the curies LMDB
failed the whole query with a KeyError. They are now filtered like
every other node, and self-pairs are skipped.
Drops generate_curie_pairs' unused max_pairs cap and make_key.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01S2yuRD7fu2tQ9a89t5V54p