Skip to content

Loosen the goldens tolerance for deterministic cases to 5e-6 - #7

Merged
wrignj08 merged 1 commit into
mainfrom
fix/goldens-tolerance
Nov 16, 2026
Merged

wrignj08 merged 1 commit into
mainfrom
fix/goldens-tolerance

Conversation

@wrignj08

Copy link
Copy Markdown
Contributor

The weekly slow tier failed last night (run 43738950785) because of one golden case:

  • test_goldens_reproduce_with_ocm, case array_s2_overlap0_logits_tiny0:output
  • 182 of 128,000 values (0.14%) were outside atol=1e-6, by at most 1.9e-6
  • class outputs matched exactly

The archive repo's slow tier failed the same way on 8 November (by up to 2.1e-6). Both runs used ubuntu-26.04 runners. The goldens were generated on a local CPU with torch 2.14.1.

This PR raises the tolerance for deterministic cases from 1e-6 to 5e-6. Cases that aren't deterministic keep 1e-5, and class outputs are still compared exactly.

A manually triggered run that includes the slow tier passed on this branch: run 43790201601.

🤖 Generated with Claude Code

ubuntu-26.04 runners differ from the goldens by up to ~2e-6 on
array_s2_overlap0_logits_tiny0, failing the weekly slow tier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wrignj08
wrignj08 merged commit 3038bb7 into main Nov 16, 2026
17 checks passed
@wrignj08
wrignj08 deleted the fix/goldens-tolerance branch November 16, 2026 02:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant