Skip to content

Score Pathfinder path rankings with NDCG - #66

Open
mohsenht wants to merge 9 commits into
mainfrom
NDCG-for-pathfinder
Open

mohsenht wants to merge 9 commits into
mainfrom
NDCG-for-pathfinder

Conversation

@mohsenht

@mohsenht mohsenht commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

mohsenht and others added 8 commits November 19, 2025 12:50
…s" it cannot install runner module inside test_harness module
…l_run

# Conflicts:
#	test_harness/main.py
#	test_harness/run.py
Main now has a built-in --local mode and --target_url/--target overrides,
so drop local_run's hard-wired MockReporter/MockSlacker/MockQueryRunner and
forced "ara" component in favor of main's code. Keep local_run's packaging
fix so test_harness.runner gets installed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mohsenht
mohsenht requested a review from maximusunc November 9, 2026 17:22

@maximusunc maximusunc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plumbing from harness to radiator looks complete and clean. I left four comments on the scoring itself. The first two decide whether the number means what the schema says ("0 to 1"). The other two are about NDCG and pass/fail reading the response differently, and about SKIPPED rows still carrying a score.


Generated by Claude Code

Comment thread test_harness/pathfinder_test_runner.py Outdated
]


def ideal_dcg(n_expected: int) -> float:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ideal assumes any 3 expected nodes can share one path. Expected nodes are pooled from every asset that shares the query, and those assets often describe alternative paths. Say asset A expects KIT and asset B expects mast cell + histamine: the ideal packs all three onto path 1, which may not be achievable. An agent that returns the best real answer (KIT path first, then the mast cell/histamine path) gets dcg(1, 3) / 7 ≈ 0.42, not 1. That's the value test_ndcg_is_scored_per_query asserts.

As a result, scores have a query-dependent ceiling below 1 and can't be compared across queries with different numbers of assets. Could the ideal be built from each asset's own set of nodes (e.g. one ideal path per asset, best first)? Or should each asset be scored separately? If the pooled ideal is intended, the docstring and the schema comment ("from 0 to 1") should say the ceiling can be lower.


Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to scoring each asset on its own. A path is relevant when it has at least the asset's minimum_required_path_nodes, and NDCG is 1/log₂(rank + 1) of the first such path, so a perfect answer always scores 1.

Comment thread test_harness/pathfinder_test_runner.py Outdated
if index not in seen and curies & on_path
}
seen |= first_visits
dcg += (2 ** len(first_visits) - 1) / log2(rank + 1)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relevance here isn't capped, but ideal_dcg caps it at MAX_PATH_NODES = 3 per path, so NDCG can exceed 1. on_path is every subject/object of every edge in the path's auxiliary graph, endpoints included. If that touches 4 expected nodes, the top path's gain is 2**4 - 1 = 15 against an ideal of 7 + 1/log2(3) ≈ 7.63, giving NDCG ≈ 1.97. This can happen through an extra edge in the auxiliary graph, a longer path, or an expected node that is also a query endpoint.

The schema documents the value as 0 to 1. Capping len(first_visits) at MAX_PATH_NODES, or clamping the result to 1.0, would keep that true. A test with a path that touches more than 3 expected nodes would cover it.


Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relevance is now either a hit or not, so the score is between 0 and 1.

Comment thread test_harness/run.py
# NDCG is extra to pass/fail: a response it
# can't score mustn't fail the test.
try:
agent_report.ndcg = pathfinder_ndcg(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NDCG and the pass/fail analysis just above read the response differently:

  • pathfinder_pass_fail_analysis only looks at results[0]["analyses"], but ranked_paths looks at analyses in every result. An agent can pass with NDCG 0, or fail with a high NDCG, depending on where its paths sit.
  • Pass/fail looks up CURIEs with normalized_curies[path_node_id], which raises if the node normalizer didn't return one. query_path_nodes uses .get(path_node_id, path_node_id) instead. When a CURIE is missing, pass/fail raises, the outer except marks the agent "Test Error", and this NDCG call never runs, so the more forgiving fallback never takes effect.

Can both use the same set of results and the same normalization? If the difference is intentional, a comment saying so would help.


Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maximusunc
Pathfinder always returns a single result, so results[0] already holds all paths.

Both now share ranked_paths and the same normalization, which falls back to the CURIE as given. The one intended difference is that NDCG only scores the top 500 paths, so a match ranked below that still passes but scores 0.

Comment thread test_harness/run.py Outdated
query_path_nodes.get(test_asset_hash)
):
# no paths, so none of them are relevant
agent_report.ndcg = 0.0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the status agent (normally the ARS) produces no result, force_skipped further down makes every agent SKIPPED in the collector. Any ndcg already set on the reports is still sent, though, including this 0.0 for NO_RESULTS and any score computed for other ARAs. The radiator would then show SKIPPED rows with an NDCG value (often 0.0), which reads like a real score. Clearing ndcg (to None) when force_skipped is set, in the same place the status is overridden, would keep the two consistent.


Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already covered: with force_skipped, _asset_result (result_collector.py:208) ignores every agent's report, so ndcg is sent as None. I added test_collector_force_skipped_drops_ndcg to pin it down.

Addresses review feedback on the NDCG scoring:

- Score each asset on its own instead of pooling expected nodes across
  every asset sharing a query. The pooled ideal assumed all expected
  nodes could share one path, so the best real answer could score well
  below 1. A path is now relevant when it visits at least the asset's
  minimum_required_path_nodes, and only the first relevant path counts,
  so NDCG is 1 / log2(rank + 1) of that path: always 0 to 1, with 1 for
  a relevant top path. Drops ideal_dcg and MAX_PATH_NODES.
- This also stops NDCG exceeding 1 when a path visits more expected
  nodes than the ideal allowed for (eg an expected query endpoint).
- Pass/fail and NDCG now read the same paths (ranked_paths over
  results[0], the single result Pathfinder returns), match nodes the
  same way, and share one normalization that falls back to the CURIE as
  given. Before, a CURIE the node normalizer missed raised in pass/fail
  and turned the agent into a Test Error. A path missing from the
  response is skipped rather than raising.
- Add a test that a force-skipped agent's NDCG is never sent to the
  radiator; the collector already drops it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants