Skip to content

Add model-comparison tooling and structured MMLU reporting - #1041

Open
Mikyx-1 wants to merge 1 commit into
google:devfrom
Mikyx-1:eval/model-comparison
Open

Mikyx-1 wants to merge 1 commit into
google:devfrom
Mikyx-1:eval/model-comparison

Conversation

@Mikyx-1

@Mikyx-1 Mikyx-1 commented Sep 25, 2026

Copy link
Copy Markdown

Implements step 1 of the series proposed in #1002.

Adds a serial baseline-versus-target runner with Markdown and JSON reports covering MMLU accuracy, correctness flips, full-vocabulary KL divergence, timing, memory usage, and optional cross entropy.

  • Scores the first answer token using A–D label logits.
  • Stores reference logits in JSON Lines with exact dataset, tokenizer, and prompt validation.
  • Adds CMake/Bazel integration, regression tests, and usage documentation.
  • Preserves configured inference settings.

Validation

  • Clean CMake build.
  • 12 C++ and 18 Python tests passed.
  • C++ tests passed under AddressSanitizer and UndefinedBehaviorSanitizer.
  • Completed an 83-question Gemma 3 270M self-comparison and cross-entropy evaluation.
  • Verified rejection of mismatched reference prompts.

Existing autotuning produced one correctness flip during self-comparison. A temporary diagnostic using fixed MatMul candidates produced zero KL across five questions. Autotuner controls remain scheduled for step 2.

Bazel rules were updated but not executed because Bazel was unavailable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant