Skip to content

Avoid redundant tokenizer buffer checks - #204

Merged
stevebauman merged 2 commits into
masterfrom
fix/reduce-buffer-checks
Nov 16, 2026
Merged

stevebauman merged 2 commits into
masterfrom
fix/reduce-buffer-checks

Conversation

@stevebauman

Copy link
Copy Markdown
Member

The tokenizer calls ensureBuffer() while skipping whitespace and again before reading the next token, even when the current byte is already available.

This adds a direct check in skipWhitespace() and nextToken() so ensureBuffer() is only called when the current byte is missing. Both methods retain their existing read attempts, including the retry when the stream temporarily returns no data.

The following benchmarks compare this change against the implementation after #203, using PHP 8.4.27 and 35,798 synthetic responses per workload. Each result is the average CPU time across six runs.

Workload Before CPU time After CPU time CPU time reduction
Mixed FETCH 1.718 seconds 1.562 seconds 9.0%
Numeric-heavy FETCH 0.383 seconds 0.345 seconds 9.9%

All 924 unit tests pass, and 3,125 comparisons against the previous tokenizer produced matching token and exception results.

Refs #200.

@stevebauman
stevebauman merged commit 8cfc491 into master Nov 16, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant