Give every benchmark the same shape and lint it - #242
Merged
Merged
Conversation
It was the only benchmark measured for 20 s per report (Benchmark.ips(20)) instead of the default 5 s, since it was added in 87ff8f5 with no reason given. That made it about 4x slower than others in every CI job, and its numbers were measured differently from every other file. At the default timing the result holds, with margins no worse: detect is 3.82x faster than select.first on Ruby 3.4 (3.55x at 20 s) and 4.83x on Ruby 2.1 (4.53x at 20 s).
Its three Benchmark.ips blocks are inside methods (bench_100_to_100, bench_100_to_1000, bench_10_to_100) that nothing calls, so running it only built its random strings: no report, no comparison. CI logs show only its `ruby -v` line, and results collection found nothing from it. The results in its comments came from a manual run. concatenation.rb already covers every method it compared (String#+, interpolation, String#concat, String#append). It came from #216 for issue #158 and was never linked from the README.
.github/scripts/lint-benchmarks.rb reads each file under code/ with Prism, without running it, and fails when a Benchmark.ips block: - does not call x.compare! - sets its own timing (Benchmark.ips(20), x.time =, x.warmup =, or x.config with time or warmup in any hash syntax) - sits inside a method that never runs from the top of the file, which is how concatenation_randomized.rb measured nothing for years A new lint job runs it on ruby_4.0. The benchmark jobs wait for it, so a failing lint does not hold 26 jobs of the org's slots, and benchmarks-ok, the required check, fails when it fails. CONTRIBUTING lists the three rules.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Benchmark.ipsthat actually runs. A new lint enforces it in CI.Changes
Step 1:
select-first-vs-detect.rbuses the default timingBenchmark.ips(20)) instead of the default 5 s, since it was added in 87ff8f5 with no reason given. That made it about 4x slower than the others in every CI job.detectis 3.82x faster thanselect.firston Ruby 3.4 (3.55x at 20 s), and 4.83x on Ruby 2.1 (4.53x at 20 s).Step 2: remove
concatenation_randomized.rb, which never ranBenchmark.ipsblocks sit inside methods that nothing calls, so running it only builds its random strings: no report, no comparison. CI logs show only itsruby -vline, and results collection (Save results as JSON and run YJIT and ZJIT variants #241) found nothing from it. The results in its comments came from a manual run.concatenation.rbalready covers every method it compared (String#+, interpolation,String#concat,String#append). It came from Randomized string concat #216 for String concat contains dummy comparision. #158 and was never linked from the README.Step 3: lint every benchmark's shape in CI
.github/scripts/lint-benchmarks.rbreads each file undercode/with Prism, without running it, and fails when aBenchmark.ipsblock:x.compare!;Benchmark.ips(20),x.time =,x.warmup =, orx.configwithtime/warmupin any hash syntax);lintjob runs it on ruby_4.0. The benchmark jobs wait for it, so a failing lint doesn't hold 26 of the org's job slots.benchmarks-ok, the required check, fails when the lint fails.Test plan
ruby_4.0(the command CI runs).Benchmark.ips:reportthat nothing calls, the uncalled one of two same-named class methods, a top-level call that reaches a different method with the same name, and files withoutBenchmark.ips.Foo.new.run, class methods calling each other,class << self,A::B.run, inherited and mixed-in methods.benchmarks-okpasses only whenchangesandlintsucceed and the benchmark jobs succeed or are skipped (all 6 combinations checked)..github/scripts/, so its own CI runs the full suite behind the newlintjob.