Skip to content

chat eval (#1078): gold set, offline scorer and first staging baseline - #1219

Draft
PRAteek-singHWY wants to merge 5 commits into
OWASP:stagingfrom
PRAteek-singHWY:chat_eval_1078
Draft

PRAteek-singHWY wants to merge 5 commits into
OWASP:stagingfrom
PRAteek-singHWY:chat_eval_1078

Conversation

@PRAteek-singHWY

@PRAteek-singHWY PRAteek-singHWY commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1078.

Adds a 24-question gold set for chat (in the ask_regressions format) and a small scorer: right CRE found, SAMM cited when the question asks for SAMM, refuses when it should, and every [n] points at a real source. No LLM, so it runs in make test.

scripts/chat_eval_gold.py --live asks a running OpenCRE the questions and saves the answers. After that, scoring is offline.

Ran it on staging.opencre.org: 22/24. Misses: the 2031 Global AppSec question gets the 2025 DC event instead of "I don't know", and the SMS 2FA answer is right but misses CRE 354-752.

For "does the source really support the sentence" you need an LLM judge, so I'd keep Ragas as an optional local command, not CI.

@SomyaJain0308, review and more questions welcome.

24 questions in the ask_regressions format (the OWASP#962 threat-modeling pair,
password storage, AI prompt injection, two abstain cases), each with the CRE
ids a good answer should surface. CRE ids are checked against the CRE table;
the two AI ids come from OWASP#1124 and are newer than the Aug cache.

ask_eval/gold.py scores a /rest/v1/completion response with no LLM and no
network: CRE recall, required cite groups through multicite (SAMM must be
cited when the question names SAMM), abstain correctness, and [n] markers
that point past the returned passages. to_phase6_cases feeds the same gold
into phase6_eval's hit@k so the Ragas/DeepEval bridge runs on real cases.

Faithfulness beyond marker validity needs an LLM judge and is left out.
…cores them

Live mode asks a running OpenCRE each gold question and writes the answers
to JSONL, so the run can be re-scored later without paying for the LLM again.
Recorded mode scores a JSONL with no network. --baseline compares two runs
and lists which cases flipped (current chat vs OWASP#1161 top-k).

Failed requests and missing rows are reported apart and kept out of the
metrics, so a 503 does not read as a wrong answer.
First live run against staging.opencre.org (24 questions) showed two scorer
bugs and two gold rows that were too narrow:
- [n] markers point into owasp_agent.citations on staging, not into table,
  so every marker was flagged as bad. Count citations when present.
- staging prefixes replies with "Answer: " and keeps accurate=true on
  refusals, so a correct refusal was scored as an answer. Strip the label.
- secrets_in_code accepts 508-702 (Use key vaults), dependencies accepts
  715-223 / 715-334. Both are correct CREs the first gold missed; noted in
  the rows.

Runner sends a named User-Agent: Cloudflare in front of opencre.org answers
urllib's default agent with 403 / error 1010.
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@PRAteek-singHWY
PRAteek-singHWY marked this pull request as draft November 10, 2026 17:04
@SomyaJain0308

Copy link
Copy Markdown

One improvement that came to mind while going through the scorer: would it be useful to track CRE precision and recall separately? Currently, 'cre_hit' only checks whether at least one expected CRE ID appears, so it doesn't tell us whether the response also retrieved irrelevant CREs which excludes extra unnecessary info being passed into the answers. Having both metrics could give us a better picture of retrieval quality. What's your take on it?

@PRAteek-singHWY

PRAteek-singHWY commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

One improvement that came to mind while going through the scorer: would it be useful to track CRE precision and recall separately? Currently, 'cre_hit' only checks whether at least one expected CRE ID appears, so it doesn't tell us whether the response also retrieved irrelevant CREs which excludes extra unnecessary info being passed into the answers. Having both metrics could give us a better picture of retrieval quality. What's your take on it?

Hi @SomyaJain0308 Good idea, added cre_precision in 9b12e58. It's reported but not pass/fail: on the staging run it's 0.71, and some of the "extra" CREs are correct siblings the gold doesn't list yet.

Share of returned CREs the gold row lists. Reported only, not part of
pass/fail, since gold rows list accepted CREs rather than every correct
sibling. Staging run 2026-11-09: 0.705. Suggested by @SomyaJain0308.
…verdict

- Gold set 24 -> 29: two compound questions where both halves must be
  cited (password storage + credential stuffing, session cookies + XSS)
  and three more refusals (future Top 10 edition, off topic, unknown CVE).
- application/utils/ask_eval/faithfulness.py: hermetic side of the
  faithfulness eval. Picks the judgeable answers, pulls the evidence
  passages, counts passages cut at the API's 500 char cap, summarises
  judge verdicts. Tested with a fake judge.
- scripts/chat_eval_faithfulness.py: optional local Ragas faithfulness
  run over a recording, listing the claims the judge found unsupported.
  Not in make test or any requirements file.
- docs/chat_eval.md: metrics, how to run, staging results (26/29), the
  evidence truncation finding (62% of passages cut) and the verdict:
  hermetic scorer in make test, Ragas as an optional local script, not CI.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants