You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gooddata-eval: fix dropped checks and over-broad exception catches from PR #1845 review #1942
Three CodeRabbit findings on #1845, not addressed before merge (non-blocking, tracked here instead):
packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py:1266-1284 — feat(gooddata-eval): populate best_run_latency_s for agentic evaluators #1845 (comment)
The best-run detail now hardcodes 9 fixed checks instead of **ev.checks(). Fixture-conditional checks (comparison_correct, title_correct, dashboard_correct) are silently dropped, defaulting to True in the report and inflating quality_score. The per-run _run_detail still uses checks(), so winning and failed runs now have different keys. Fix: use **ev.checks() and include the conditional checks in the failure message too.
packages/gooddata-eval/src/gooddata_eval/core/agentic/anomaly_detection.py:415-424 — feat(gooddata-eval): populate best_run_latency_s for agentic evaluators #1845 (comment) except Exception around send_message labels any failure LoopExit.CHAT_ERROR, including non-chat bugs (e.g. a TypeError in _accumulate), mislabeling them as a GoodData infra fault. Should catch only ChatError (and httpx.HTTPError if transport faults must be recorded), matching kda_skill.py's existing pattern.
Activity
feat(gooddata-eval): populate best_run_latency_s for agentic evaluators