Skip to content

gooddata-eval: fix dropped checks and over-broad exception catches from PR #1845 review #1942

Open

Description

@Tomkess

Three CodeRabbit findings on #1845, not addressed before merge (non-blocking, tracked here instead):

  1. packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py:1266-1284 — feat(gooddata-eval): populate best_run_latency_s for agentic evaluators #1845 (comment)
    The best-run detail now hardcodes 9 fixed checks instead of **ev.checks(). Fixture-conditional checks (comparison_correct, title_correct, dashboard_correct) are silently dropped, defaulting to True in the report and inflating quality_score. The per-run _run_detail still uses checks(), so winning and failed runs now have different keys. Fix: use **ev.checks() and include the conditional checks in the failure message too.

  2. packages/gooddata-eval/src/gooddata_eval/core/agentic/anomaly_detection.py:415-424 — feat(gooddata-eval): populate best_run_latency_s for agentic evaluators #1845 (comment)
    except Exception around send_message labels any failure LoopExit.CHAT_ERROR, including non-chat bugs (e.g. a TypeError in _accumulate), mislabeling them as a GoodData infra fault. Should catch only ChatError (and httpx.HTTPError if transport faults must be recorded), matching kda_skill.py's existing pattern.

  3. packages/gooddata-eval/src/gooddata_eval/core/agentic/what_if.py:372-381 — feat(gooddata-eval): populate best_run_latency_s for agentic evaluators #1845 (comment)
    Same over-broad except Exception → CHAT_ERROR pattern as (2).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions