Skip to content

Centralize tokenizer buffer availability checks - #205

Merged
stevebauman merged 1 commit into
masterfrom
refactor/centralize-buffer-checks
Nov 16, 2026
Merged

stevebauman merged 1 commit into
masterfrom
refactor/centralize-buffer-checks

Conversation

@stevebauman

Copy link
Copy Markdown
Member

The tokenizer currently checks whether a byte is buffered before calling ensureBuffer() in nextToken() and skipWhitespace(). Extending that optimization to the remaining readers would repeat the same check throughout the class.

This moves the fast availability check into ensureBuffer(), where it also handles requests for multiple bytes. Readers can call the method directly, and buffer availability stays in one place. The existing retry after an empty stream read is preserved.

I've chosen the simpler implementation despite the measured CPU cost. In a local comparison against guards in all readers, the centralized implementation produced these average parser CPU times across six runs of 35,798 synthetic responses on PHP 8.4.27. CLI OPcache and JIT were disabled.

Workload Inline guards in all readers Centralized check CPU increase
Mixed FETCH 1.452 seconds 1.593 seconds 9.7%
Numeric-heavy FETCH 0.333 seconds 0.354 seconds 6.3%

The inline version used for this comparison was an experimental implementation that extended #204 to the other readers. These figures compare the two designs and do not measure a regression against current master or production request times.

Centralizing the check trades some parser CPU time for simpler readers and one place to maintain buffer availability logic. The bulk-scanning improvements remain in place.

All 1,025 unit tests pass, and 3,125 token/exception comparisons match the inline implementation.

Refs #200.

@stevebauman
stevebauman merged commit ff7f756 into master Nov 16, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant