Skip to content

Give every benchmark the same shape and lint it - #242

Merged
JuanVqz merged 3 commits into
mainfrom
feature/benchmark-file-consistency
Sep 24, 2026
Merged

JuanVqz merged 3 commits into
mainfrom
feature/benchmark-file-consistency

Conversation

@JuanVqz

@JuanVqz JuanVqz commented Sep 24, 2026

Copy link
Copy Markdown
Member

Summary

  • Every benchmark now has the same shape: default timing, a comparison at the end, and a Benchmark.ips that actually runs. A new lint enforces it in CI.
  • Two files didn't fit. One measured with its own timing, and one never benchmarked anything.

Changes

Step 1: select-first-vs-detect.rb uses the default timing

  • It was the only file measured for 20 s per report (Benchmark.ips(20)) instead of the default 5 s, since it was added in 87ff8f5 with no reason given. That made it about 4x slower than the others in every CI job.
  • The result holds at the default timing, with margins no worse: detect is 3.82x faster than select.first on Ruby 3.4 (3.55x at 20 s), and 4.83x on Ruby 2.1 (4.53x at 20 s).

Step 2: remove concatenation_randomized.rb, which never ran

Step 3: lint every benchmark's shape in CI

  • .github/scripts/lint-benchmarks.rb reads each file under code/ with Prism, without running it, and fails when a Benchmark.ips block:
    • doesn't call x.compare!;
    • sets its own timing (Benchmark.ips(20), x.time =, x.warmup =, or x.config with time/warmup in any hash syntax);
    • sits inside a method that never runs from the top of the file.
  • A new lint job runs it on ruby_4.0. The benchmark jobs wait for it, so a failing lint doesn't hold 26 of the org's job slots. benchmarks-ok, the required check, fails when the lint fails.
  • CONTRIBUTING's "Note on entry" lists the three rules.

Test plan

  • The lint passes all 67 files, locally and under Docker ruby_4.0 (the command CI runs).
  • 15 test cases (kept locally), checked against what each file really does by running it with a stand-in Benchmark.ips:
    • Flagged: each timing form, a benchmark only an uncalled helper reaches, a method named report that nothing calls, the uncalled one of two same-named class methods, a top-level call that reaches a different method with the same name, and files without Benchmark.ips.
    • Pass: Foo.new.run, class methods calling each other, class << self, A::B.run, inherited and mixed-in methods.
  • benchmarks-ok passes only when changes and lint succeed and the benchmark jobs succeed or are skipped (all 6 combinations checked).
  • actionlint and shellcheck are clean.
  • This PR changes the workflow and .github/scripts/, so its own CI runs the full suite behind the new lint job.

It was the only benchmark measured for 20 s per report (Benchmark.ips(20))
instead of the default 5 s, since it was added in 87ff8f5 with no reason
given. That made it about 4x slower than others in every CI job, and its
numbers were measured differently from every other file.

At the default timing the result holds, with margins no worse: detect is
3.82x faster than select.first on Ruby 3.4 (3.55x at 20 s) and 4.83x on
Ruby 2.1 (4.53x at 20 s).
Its three Benchmark.ips blocks are inside methods (bench_100_to_100,
bench_100_to_1000, bench_10_to_100) that nothing calls, so running it
only built its random strings: no report, no comparison. CI logs show
only its `ruby -v` line, and results collection found nothing from it.
The results in its comments came from a manual run.

concatenation.rb already covers every method it compared (String#+,
interpolation, String#concat, String#append). It came from #216 for
issue #158 and was never linked from the README.
.github/scripts/lint-benchmarks.rb reads each file under code/ with Prism,
without running it, and fails when a Benchmark.ips block:

- does not call x.compare!
- sets its own timing (Benchmark.ips(20), x.time =, x.warmup =, or
  x.config with time or warmup in any hash syntax)
- sits inside a method that never runs from the top of the file, which is
  how concatenation_randomized.rb measured nothing for years

A new lint job runs it on ruby_4.0. The benchmark jobs wait for it, so a
failing lint does not hold 26 jobs of the org's slots, and benchmarks-ok,
the required check, fails when it fails. CONTRIBUTING lists the three
rules.
@JuanVqz
JuanVqz marked this pull request as ready for review September 24, 2026 05:20
@JuanVqz
JuanVqz merged commit 6e6e567 into main Sep 24, 2026
29 checks passed
@JuanVqz
JuanVqz deleted the feature/benchmark-file-consistency branch September 24, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant