Skip to content

Generate Cheat Sheet rows in golden dataset builder - #1198

Open
KaavyaIndhu wants to merge 1 commit into
OWASP:mainfrom
KaavyaIndhu:fix/1194-golden-dataset-drift
Open

KaavyaIndhu wants to merge 1 commit into
OWASP:mainfrom
KaavyaIndhu:fix/1194-golden-dataset-drift

Conversation

@KaavyaIndhu

Copy link
Copy Markdown

Part of #1194.

#1036 added 3 Cheat Sheet rows to golden_dataset.json (authorization, rest-security, ssrf-prevention), but build_golden_dataset.py had no code for them, so --check couldn’t match the fixture.

What changed:

  • Added build_positive_cheatsheets(), which builds the 3 rows and takes the CRE IDs from owasp_cheatsheets_supplement.json (the source named in the rows’ provenance).
  • The excerpt text is kept as constants copied from the fixture, since the cheat sheet markdown isn’t in the repo.
  • The rows are appended last in build(), matching their position in the fixture.
  • No changes to the fixture or tests.

Verification:

@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration
  • Source mode: Auto
  • Repository: OWASP/OpenCRE
  • Branch: main
  • File: .coderabbit.yml
  • Commit: 6b8fcc0
  • Layers: Repository YAML
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e39c12fc-453d-4c1b-a467-da880cdb1073
Commits

Reviewing files that changed from the base of the PR and between 1936b7f and 3309aed.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: OWASP/OpenCRE
  • Branch: main
  • File: .coderabbit.yml
  • Commit: 6b8fcc0
  • Layers: Repository YAML
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 398f4eee-a93f-43d7-826c-fa14986dbfa9
Commits

Reviewing files that changed from the base of the PR and between 6b8fcc0 and 1936b7f.

Files selected for processing: 1
  • scripts/build_golden_dataset.py

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

Summary by CodeRabbit

  • New Features
    • The golden dataset now includes three positive examples based on OWASP Cheat Sheets. Each example includes its title, section path, fixture text, and related CRE mapping information. The dataset build now incorporates these examples and reports an error when a fixture section lacks a mapping.

Walkthrough

The dataset builder adds three OWASP Cheat Sheet fixtures and generates positive rows linked to CRE identifiers through section mappings. The build process appends these rows after the existing dataset slices.

Changes

Cheat Sheet positive rows

Layer / File(s) Summary
Generate and append positive rows
scripts/build_golden_dataset.py
Adds Authorization, REST Security, and Server Side Request Forgery Prevention fixtures. build_positive_cheatsheets() maps their sections to CRE identifiers and raises ValueError when a fixture section has no mapping. build() appends the generated rows.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Possibly related issues

Possibly related PRs

  • OWASP/OpenCRE#1088: Includes golden-dataset work and identifies a fixture follow-up that this change addresses.

Suggested reviewers: northdpole, pa04rth

Merge Risk: Minimal · previous review 1936b

The existing formatting drift is not caused or worsened by this change. No actionable merge-blocking risk remains after normal checks.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: generating Cheat Sheet rows in the golden dataset builder.
Description check Passed The description directly explains the added builder function, its data source, fixture handling, row ordering, and verification results.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 1 files.

Finishing Touches
Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

OWASP#1036 added three OWASP Cheat Sheet rows to golden_dataset.json, but
build_golden_dataset.py had no code to produce them, so --check could
never match the committed fixture. The builder now generates them,
reading CRE mappings from owasp_cheatsheets_supplement.json.

Part of OWASP#1194
@KaavyaIndhu
KaavyaIndhu force-pushed the fix/1194-golden-dataset-drift branch from 1936b7f to 3309aed Compare November 2, 2026 23:38
@Pa04rth

Pa04rth commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

@KaavyaIndhu I think adding build_positive_cheatsheets() addresses the missing Cheat Sheet rows, but the CWE drift exposes a broader issue with the determinism of the generator.

I would suggest separating the deterministic dataset generation from live database retrieval.

We could maintain version-controlled input fixtures for CWE records, their CRE relationships, and Cheat Sheet cases, and have the generator consume those fixtures. The generated output can then be compared against golden_dataset.json.

Separately, we should retain integration tests against the actual database to ensure that changes in extraction, adapters, and relationship retrieval are still handled correctly.

A few additional things worth considering:

  • Ensure all fields influencing generated records are included in the fixtures, not just CWE identifiers.
  • Avoid duplicating expected fixture content inside generator code.
  • Define stable ordering and uniqueness guarantees.
  • Provide a controlled process for refreshing fixtures when upstream data changes.

This would solve the current nondeterminism without sacrificing database integration coverage. @PRAteek-singHWY , review this PR and also suggest what you think ....

@KaavyaIndhu

Copy link
Copy Markdown
Author

Thanks for the review @Pa04rth ! That makes total sense, and I’ll start working on it right away and put up a PR.

One more thing I thought I could add is once the generator only uses fixtures, the --check test no longer needs a DB, so it could run in CI on every PR and catch drift early.
If you’re okay with it, I can include that too.

@KaavyaIndhu

Copy link
Copy Markdown
Author

Opened #1216 with the fixture-based approach. Thanks for the suggestion @Pa04rth!

@PRAteek-singHWY

Copy link
Copy Markdown
Contributor

Thanks @KaavyaIndhu. Adding build_positive_cheatsheets() and taking the CRE ids from owasp_cheatsheets_supplement.json is right; that's the source the rows' provenance names.

I ran it locally against my standards_cache.sqlite (an August snapshot):

  • The builder produces 322 rows, and every row is content-identical to golden_dataset.json: same ids, same order, same fields.
  • --check still reports DRIFT, and dataset_test.TestDatasetDeterminism fails when a DB is present. On my DB the only difference is whitespace: the 3 Cheat Sheet rows from feat: add adapter for CheatSheetRecord and Librarian integration with pipeline dry run #1036 are indented differently from what the builder writes. Your description puts the drift on the CWE rows. That's true on a newer DB, but on mine the CWE picks match, so there are two separate causes, not one.
  • CI is green here only because CI has no DB, so the determinism test is skipped. It doesn't show the fix works yet.

#1216 already includes this commit and also fixes the CWE half, so I'd treat #1216 as the PR to review and merge. Once it lands this one has nothing left to add and can be closed. My review of the combined change is on #1216.

@northdpole for merge order: #1216 contains this commit, so this one can close once #1216 lands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants