Repository navigation
Conversation
…s" it cannot install runner module inside test_harness module
…l_run # Conflicts: # test_harness/main.py # test_harness/run.py
Main now has a built-in --local mode and --target_url/--target overrides, so drop local_run's hard-wired MockReporter/MockSlacker/MockQueryRunner and forced "ara" component in favor of main's code. Keep local_run's packaging fix so test_harness.runner gets installed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
maximusunc
left a comment
There was a problem hiding this comment.
The plumbing from harness to radiator looks complete and clean. I left four comments on the scoring itself. The first two decide whether the number means what the schema says ("0 to 1"). The other two are about NDCG and pass/fail reading the response differently, and about SKIPPED rows still carrying a score.
Generated by Claude Code
| ] | ||
|
|
||
|
|
||
| def ideal_dcg(n_expected: int) -> float: |
There was a problem hiding this comment.
The ideal assumes any 3 expected nodes can share one path. Expected nodes are pooled from every asset that shares the query, and those assets often describe alternative paths. Say asset A expects KIT and asset B expects mast cell + histamine: the ideal packs all three onto path 1, which may not be achievable. An agent that returns the best real answer (KIT path first, then the mast cell/histamine path) gets dcg(1, 3) / 7 ≈ 0.42, not 1. That's the value test_ndcg_is_scored_per_query asserts.
As a result, scores have a query-dependent ceiling below 1 and can't be compared across queries with different numbers of assets. Could the ideal be built from each asset's own set of nodes (e.g. one ideal path per asset, best first)? Or should each asset be scored separately? If the pooled ideal is intended, the docstring and the schema comment ("from 0 to 1") should say the ceiling can be lower.
Generated by Claude Code
There was a problem hiding this comment.
Switched to scoring each asset on its own. A path is relevant when it has at least the asset's minimum_required_path_nodes, and NDCG is 1/log₂(rank + 1) of the first such path, so a perfect answer always scores 1.
| if index not in seen and curies & on_path | ||
| } | ||
| seen |= first_visits | ||
| dcg += (2 ** len(first_visits) - 1) / log2(rank + 1) |
There was a problem hiding this comment.
Relevance here isn't capped, but ideal_dcg caps it at MAX_PATH_NODES = 3 per path, so NDCG can exceed 1. on_path is every subject/object of every edge in the path's auxiliary graph, endpoints included. If that touches 4 expected nodes, the top path's gain is 2**4 - 1 = 15 against an ideal of 7 + 1/log2(3) ≈ 7.63, giving NDCG ≈ 1.97. This can happen through an extra edge in the auxiliary graph, a longer path, or an expected node that is also a query endpoint.
The schema documents the value as 0 to 1. Capping len(first_visits) at MAX_PATH_NODES, or clamping the result to 1.0, would keep that true. A test with a path that touches more than 3 expected nodes would cover it.
Generated by Claude Code
There was a problem hiding this comment.
Relevance is now either a hit or not, so the score is between 0 and 1.
| # NDCG is extra to pass/fail: a response it | ||
| # can't score mustn't fail the test. | ||
| try: | ||
| agent_report.ndcg = pathfinder_ndcg( |
There was a problem hiding this comment.
NDCG and the pass/fail analysis just above read the response differently:
pathfinder_pass_fail_analysisonly looks atresults[0]["analyses"], butranked_pathslooks at analyses in every result. An agent can pass with NDCG 0, or fail with a high NDCG, depending on where its paths sit.- Pass/fail looks up CURIEs with
normalized_curies[path_node_id], which raises if the node normalizer didn't return one.query_path_nodesuses.get(path_node_id, path_node_id)instead. When a CURIE is missing, pass/fail raises, the outerexceptmarks the agent "Test Error", and this NDCG call never runs, so the more forgiving fallback never takes effect.
Can both use the same set of results and the same normalization? If the difference is intentional, a comment saying so would help.
Generated by Claude Code
There was a problem hiding this comment.
@maximusunc
Pathfinder always returns a single result, so results[0] already holds all paths.
Both now share ranked_paths and the same normalization, which falls back to the CURIE as given. The one intended difference is that NDCG only scores the top 500 paths, so a match ranked below that still passes but scores 0.
| query_path_nodes.get(test_asset_hash) | ||
| ): | ||
| # no paths, so none of them are relevant | ||
| agent_report.ndcg = 0.0 |
There was a problem hiding this comment.
When the status agent (normally the ARS) produces no result, force_skipped further down makes every agent SKIPPED in the collector. Any ndcg already set on the reports is still sent, though, including this 0.0 for NO_RESULTS and any score computed for other ARAs. The radiator would then show SKIPPED rows with an NDCG value (often 0.0), which reads like a real score. Clearing ndcg (to None) when force_skipped is set, in the same place the status is overridden, would keep the two consistent.
Generated by Claude Code
There was a problem hiding this comment.
Already covered: with force_skipped, _asset_result (result_collector.py:208) ignores every agent's report, so ndcg is sent as None. I added test_collector_force_skipped_drops_ndcg to pin it down.
Addresses review feedback on the NDCG scoring: - Score each asset on its own instead of pooling expected nodes across every asset sharing a query. The pooled ideal assumed all expected nodes could share one path, so the best real answer could score well below 1. A path is now relevant when it visits at least the asset's minimum_required_path_nodes, and only the first relevant path counts, so NDCG is 1 / log2(rank + 1) of that path: always 0 to 1, with 1 for a relevant top path. Drops ideal_dcg and MAX_PATH_NODES. - This also stops NDCG exceeding 1 when a path visits more expected nodes than the ideal allowed for (eg an expected query endpoint). - Pass/fail and NDCG now read the same paths (ranked_paths over results[0], the single result Pathfinder returns), match nodes the same way, and share one normalization that falls back to the CURIE as given. Before, a CURIE the node normalizer missed raised in pass/fail and turned the agent into a Test Error. A path missing from the response is skipped rather than raising. - Add a test that a force-skipped agent's NDCG is never sent to the radiator; the collector already drops it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No description provided.