Skip to content

Remove redundant parser token cloning - #203

Merged
stevebauman merged 1 commit into
masterfrom
fix/remove-token-cloning
Nov 16, 2026
Merged

stevebauman merged 1 commit into
masterfrom
fix/remove-token-cloning

Conversation

@stevebauman

Copy link
Copy Markdown
Member

The parser clones each token before adding it to a response. The built-in tokenizer already creates a fresh object for every token, and advancing the parser replaces its current token without modifying previously returned objects.

This removes the four clone operations and lets responses retain the tokens directly.

The following benchmarks compare this change against the implementation after #202, using PHP 8.4.27 and 35,798 synthetic responses per workload. Each result is the average CPU time across three runs.

Workload With cloning Without cloning CPU time reduction
Mixed FETCH 1.650 seconds 1.626 seconds 1.5%
Numeric-heavy FETCH 0.381 seconds 0.378 seconds 0.8%

The measured improvement is small, especially for the numeric-heavy workload where individual runs varied in both directions.

All 920 unit tests pass.

Custom tokenizers that reuse mutable token objects, or retain and later modify emitted tokens, would lose the isolation previously provided by cloning. The built-in tokenizer has neither behavior, and no such custom implementation exists in this repository.

Refs #200.

@stevebauman
stevebauman merged commit 8e976cc into master Nov 16, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant