Repository navigation
Port Relay PR #891/#897/#898/#899: confidence ordering and RRF fusion - #260
Merged
Merged
Conversation
Upstream Relay merged nine PRs since the 2b2121d pin. Ported: - #891: appraise_confidence sorts results by confidence; ScoreStatCalc summarizes ordering_components.confidence instead of analysis scores. - #894/#895: a response with no knowledge graph nodes counts as having no results (premerge.response_results). - #897: weighted RRF ranker fusion (Aragorn 0.9 / ARAX 0.1, c=40, on by default, ARS_RRF_* env vars) in the merge post-process, reading TRAPI 2.0 node bindings. Adds rrf_score/rrf_rank/rrf_ranker_ranks/ rrf_ranker_contributions to merged results. - #898: premerge no longer normalizes scores or strips phantom support graphs; the infores and phantom-support-graph reports go with them. - #899: last_merged_completed (live and replayed) carries results_stat. #896, #903, #892 need nothing here. Goldens regenerated by running the new upstream commit; RRF gets its own corpus cases and upstream goldens. The parity register records each accepted change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRBRjXSHEEg7ZHSLVX7
Codecov Report❌ Patch coverage is
... and 3 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR re-pins the upstream Relay commit from
2b2121dfto84a66c1and ports several behavioral changes:Summary
Updates the ARS merge and pre-merge workers to implement upstream's recent changes for result ordering by confidence, weighted reciprocal rank fusion (RRF) of ARA rankings, and removal of pre-merge validation steps.
Key Changes
Relay PR #891 (orderingByConfidence)
appraise_confidencenow sorts merged results byordering_components.confidence(highest first) instead of by scoreScoreStatCalccomputes statistics over confidence values rather than analysis scores{}statsRelay PR #897 (Weighted RRF Ranker Fusion)
shepherd_utils/ars/ranker_fusion.pymodule implements weighted reciprocal rank fusionrrf_score,rrf_rank,rrf_ranker_ranks, andrrf_ranker_contributionsfields to resultsARS_RRF_ENABLED,ARS_RRF_ARAGORN_WEIGHT,ARS_RRF_Cars_mergeworker's post-processing pipelineRelay PR #898 (Validation Simplification)
pre_merge_processandremove_phantom_support_graphscalls fromars_premergeworkerRelay PR #899 (Result Stats in Completion)
last_merged_completednotification now carriesresults_statfrom the final merged versionTest Coverage
Notable Implementation Details
{"id": ...}objects) and 2.0 ({"ids": [...]}objects) node binding formatsordering_components.confidence