Skip to content

feat: pass captured tool declarations to ADK judges - #264

Merged
krisztianfekete merged 3 commits into
agentevals-dev:mainfrom
erauner12:feature/tool-definitions-app-details
Nov 4, 2026
Merged

krisztianfekete merged 3 commits into
agentevals-dev:mainfrom
erauner12:feature/tool-definitions-app-details

Conversation

@erauner12

@erauner12 erauner12 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Closes #263.

Summary

ADK judges need to know which tools an agent had available, not just which ones it called.

This PR passes captured tool definitions into Invocation.app_details, including descriptions and parameter schemas, so judges have the context needed to evaluate tool use.

It also:

  • Uses upstream's source-owned tool events.
  • Groups declarations by agent and skips non-function tools.
  • Falls back to observed tool names when definitions aren't available.
  • Keeps exported eval-set JSON unchanged.

Testing

Added regression coverage using OTel GenAI semconv fixtures, including rubric_based_final_response_quality_v1 with tool calls.

Validation after rebase: 2,059 model-free tests passed on Python 3.13.15, including 29 integration tests; 14 skipped. E2E tests excluded. Ruff, lockfile, and diff checks passed. No live model calls.

Base

Rebased directly onto upstream main at 890dc65, which includes merged #267. The identity repair is inherited from upstream; this PR now adds only captured tool declarations and their tests. Producers use canonical GenAI semconv; no legacy producer fixture or private extraction override is added.

AI assistance: non-trivial code and test assistance.

@erauner12
erauner12 marked this pull request as ready for review October 29, 2026 18:09
@erauner12 erauner12 changed the title feat(adk-bridge): pass captured tool definitions to judges feat: preserve captured tool declarations and ADK rounds for judges Oct 30, 2026
@erauner12
erauner12 force-pushed the feature/tool-definitions-app-details branch from b22d890 to 9c93c10 Compare October 30, 2026 21:11

@krisztianfekete krisztianfekete left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, couple of small things:

  1. rubric_based_final_response_quality_v1 now errors on every ADK trace with a tool call (see inline). CI misses it because no test runs that metric on a converted trace.
  2. We're dropping legacy formats in favour of OTel GenAI semconv, which is also where ADK telemetry v2 is going. Please remove the parts built on OpenLLMetry gen_ai.prompt.N, ADK's deprecated call_llm span and the Gemini Schema format. Please also use the semconv fixtures for tests.

Comment thread src/agentevals/adk_bridge.py Resolved
Comment thread src/agentevals/adk_bridge.py Resolved
Comment thread src/agentevals/genai/extract.py Resolved
Comment thread tests/test_builtin_metrics.py Resolved
Comment thread tests/test_adk_bridge.py Resolved
@erauner12
erauner12 force-pushed the feature/tool-definitions-app-details branch from 9c93c10 to c982ece Compare November 2, 2026 21:11
@erauner12 erauner12 changed the title feat: preserve captured tool declarations and ADK rounds for judges feat: pass captured tool declarations to ADK judges Nov 2, 2026
@erauner12

erauner12 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased this PR onto #267. Removed the legacy compatibility branch and dedicated tests, converted only type == "function" entries, dropped the OBJECT case, and switched the judge fixture to v2proj_span_only/S2.json, including a rubric_based_final_response_quality_v1 run asserting result.error is None.

@krisztianfekete Would you accept a minimal synthetic kagent 0.10 compatibility fixture, or should semconv be the only supported path?

@krisztianfekete

Copy link
Copy Markdown
Member

@krisztianfekete Would you accept a minimal synthetic kagent 0.10 compatibility fixture, or should semconv be the only supported path?

Optimally no, semconv is the only supported path. Older producers should upgrade or normalize to OTel GenAI before export. Would that work with you?

@krisztianfekete krisztianfekete left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's rebase it to main, then it LGTM!

@erauner12
erauner12 force-pushed the feature/tool-definitions-app-details branch from 50c25af to cfbf71f Compare November 3, 2026 20:51
@krisztianfekete
krisztianfekete merged commit 6d633a8 into agentevals-dev:main Nov 4, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pass captured tool definitions to ADK judges as app_details

2 participants