Repository navigation
Centralize tokenizer buffer availability checks - #205
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tokenizer currently checks whether a byte is buffered before calling
ensureBuffer()innextToken()andskipWhitespace(). Extending that optimization to the remaining readers would repeat the same check throughout the class.This moves the fast availability check into
ensureBuffer(), where it also handles requests for multiple bytes. Readers can call the method directly, and buffer availability stays in one place. The existing retry after an empty stream read is preserved.I've chosen the simpler implementation despite the measured CPU cost. In a local comparison against guards in all readers, the centralized implementation produced these average parser CPU times across six runs of 35,798 synthetic responses on PHP 8.4.27. CLI OPcache and JIT were disabled.
The inline version used for this comparison was an experimental implementation that extended #204 to the other readers. These figures compare the two designs and do not measure a regression against current
masteror production request times.Centralizing the check trades some parser CPU time for simpler readers and one place to maintain buffer availability logic. The bulk-scanning improvements remain in place.
All 1,025 unit tests pass, and 3,125 token/exception comparisons match the inline implementation.
Refs #200.