Skip to content

Port Relay PR #891/#897/#898/#899: confidence ordering and RRF fusion - #260

Merged
maximusunc merged 1 commit into
develfrom
claude/lucid-carson-237knk
Nov 13, 2026
Merged

maximusunc merged 1 commit into
develfrom
claude/lucid-carson-237knk

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

This PR re-pins the upstream Relay commit from 2b2121df to 84a66c1 and ports several behavioral changes:

Summary

Updates the ARS merge and pre-merge workers to implement upstream's recent changes for result ordering by confidence, weighted reciprocal rank fusion (RRF) of ARA rankings, and removal of pre-merge validation steps.

Key Changes

Relay PR #891 (orderingByConfidence)

  • appraise_confidence now sorts merged results by ordering_components.confidence (highest first) instead of by score
  • ScoreStatCalc computes statistics over confidence values rather than analysis scores
  • Results without a confidence value contribute nothing to stats
  • ARA child responses (which lack ordering components) have empty {} stats

Relay PR #897 (Weighted RRF Ranker Fusion)

  • New shepherd_utils/ars/ranker_fusion.py module implements weighted reciprocal rank fusion
  • Fuses rankings from multiple ARAs (Aragorn and ARAX) using configurable weights
  • Adds rrf_score, rrf_rank, rrf_ranker_ranks, and rrf_ranker_contributions fields to results
  • Configuration via environment variables: ARS_RRF_ENABLED, ARS_RRF_ARAGORN_WEIGHT, ARS_RRF_C
  • Integrated into ars_merge worker's post-processing pipeline
  • Supports both TRAPI 1.5 and 2.0 node binding formats

Relay PR #898 (Validation Simplification)

  • Removed pre_merge_process and remove_phantom_support_graphs calls from ars_premerge worker
  • These functions are retained in code for reference but no longer executed on arriving responses
  • Validation now runs directly on the payload as sent

Relay PR #899 (Result Stats in Completion)

  • last_merged_completed notification now carries results_stat from the final merged version

Test Coverage

  • New test fixtures for ranker fusion cases (TRAPI 1.5 and 2.0)
  • New test cases for confidence-based scoring
  • Unit tests for RRF configuration parsing and environment variable handling
  • Golden parity tests updated to validate RRF output against upstream

Notable Implementation Details

  • RRF key matching handles both TRAPI 1.5 (list of {"id": ...} objects) and 2.0 ({"ids": [...]} objects) node binding formats
  • Confidence stats exclude results without ordering_components.confidence
  • RRF is applied post-annotation and post-confidence-ordering in the merge pipeline
  • Environment variable parsing includes validation and fallback to defaults with logging

Upstream Relay merged nine PRs since the 2b2121d pin. Ported:

- #891: appraise_confidence sorts results by confidence; ScoreStatCalc
  summarizes ordering_components.confidence instead of analysis scores.
- #894/#895: a response with no knowledge graph nodes counts as having
  no results (premerge.response_results).
- #897: weighted RRF ranker fusion (Aragorn 0.9 / ARAX 0.1, c=40, on by
  default, ARS_RRF_* env vars) in the merge post-process, reading TRAPI
  2.0 node bindings. Adds rrf_score/rrf_rank/rrf_ranker_ranks/
  rrf_ranker_contributions to merged results.
- #898: premerge no longer normalizes scores or strips phantom support
  graphs; the infores and phantom-support-graph reports go with them.
- #899: last_merged_completed (live and replayed) carries results_stat.

#896, #903, #892 need nothing here. Goldens regenerated by running the
new upstream commit; RRF gets its own corpus cases and upstream goldens.
The parity register records each accepted change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRBRjXSHEEg7ZHSLVX7
@codecov

codecov Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.66667% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.68%. Comparing base (38c9c0a) to head (20feeb3).
⚠️ Report is 1 commits behind head on devel.

Files with missing lines Patch % Lines
shepherd_utils/ars/ranker_fusion.py 92.37% 4 Missing and 5 partials ⚠️
workers/ars_merge/worker.py 90.19% 2 Missing and 3 partials ⚠️
shepherd_utils/ars/notify.py 70.00% 3 Missing ⚠️
Files with missing lines Coverage Δ
shepherd_utils/ars/lifecycle.py 92.40% <100.00%> (ø)
shepherd_utils/ars/premerge.py 82.45% <100.00%> (+1.53%) ⬆️
shepherd_utils/config.py 100.00% <ø> (ø)
workers/ars_premerge/worker.py 87.71% <100.00%> (-0.57%) ⬇️
shepherd_utils/ars/notify.py 81.81% <70.00%> (-2.12%) ⬇️
workers/ars_merge/worker.py 84.77% <90.19%> (+0.66%) ⬆️
shepherd_utils/ars/ranker_fusion.py 92.37% <92.37%> (ø)

... and 3 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 893a3d8...20feeb3. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc
maximusunc merged commit 0318504 into devel Nov 13, 2026
2 checks passed
@maximusunc
maximusunc deleted the claude/lucid-carson-237knk branch November 13, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants