Optimize Lexer::readString by OpaqueRock · Pull Request #1948 · webonyx/graphql-php

OpaqueRock · 2026-07-16T10:26:24Z

Summary

Bulk-scan runs of ordinary bytes with strcspn() instead of decoding/validating one UTF-8 character at a time via readChar(). Falls back to per-character logic only for quotes, line terminators, control characters, and escape sequences. TAB (0x09) is excluded from the stop-byte set since it is a legal, unescaped SourceCharacter.

Why

readString() is on the hot path for every request in a GraphQL server. The per-call overhead of character-by-character decoding compounds across large queries with many string literals.

Benchmarks

Measured with an isolated benchmark harness (PHP 8.3.6, x86_64, Xdebug disabled, 500 iterations averaged over 3 runs):

Metric	master	This PR	Speedup
`Parser::parse` (Lexer)	10,907 μs/op	4,026 μs/op	2.71x

Testing

Full test suite passes (1990+ tests)
PHPStan passes
New tests cover all single-char escape sequences, multi-byte UTF-8 in readChar(), and the & punctuator — achieving 100% line coverage on Lexer.php

Lexer::readString previously decoded and validated one UTF-8 character at a time via Lexer::readChar(). Replace this with a bulk scan using strcspn() to copy runs of bytes that need no special handling (i.e. anything other than a quote, backslash, or control character) in a single native call, falling back to the original per-character logic only for quotes, line terminators, control characters, and escape sequences. Care is taken to keep $this->position (a decoded-character count, not a byte count) accurate for multi-byte UTF-8 content. TAB (0x09), a legal, unescaped SourceCharacter per the GraphQL spec, is excluded from the stop-byte set so it is scanned through like any other ordinary character instead of being individually inspected. Visitor::visitInParallel previously called extractVisitFn() to resolve the enter/leave callback for the current visitor and node kind on every single node, for every wrapped visitor - an O(nodes x visitors) number of array lookups, even though a callback's identity for a given (visitor, kind, direction) triple never changes during a traversal. Since NodeKind exposes a small, fixed set of kinds, precompute this mapping once per call (O(visitors x kinds)) instead. Additionally, replace func_get_args() plus argument unpacking in the enter/leave dispatcher closures with the five explicit, typed parameters they are always called with (matching Visitor::visit()'s single call site), avoiding func_get_args()'s per-call overhead. Add a regression test for the TAB-handling edge case described above.

Copilot

Pull request overview

This PR introduces targeted performance optimizations in two hot-path components of the library: lexing string literals and running multiple validation visitors in a single traversal.

Changes:

Optimize Lexer::readString() by bulk-scanning “ordinary” bytes via strcspn() and only falling back to per-character handling for escape/control/terminator cases (explicitly allowing unescaped TAB per spec).
Optimize Visitor::visitInParallel() by precomputing per-(visitor, kind, direction) callbacks once per traversal and avoiding func_get_args()/argument unpacking in the dispatcher.
Add a regression test ensuring strings containing an unescaped TAB are lexed correctly.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
tests/Language/LexerTest.php	Adds regression coverage for unescaped TAB in string literals.
src/Language/Visitor.php	Reduces per-node overhead in parallel visitor dispatch by caching callbacks and avoiding `func_get_args()`.
src/Language/Lexer.php	Speeds up string literal lexing by scanning runs of non-special bytes in bulk while preserving escape/control handling.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

The enter/leave callback caches were previously prefilled only for NodeKind constants, silently skipping generic and kind-specific visitors for Nodes with custom (non-NodeKind) kinds. Switch to lazy, per-direction caches that resolve and store a callback (or `false` as a "no callback" sentinel) on first use, keyed by any kind string. Also fixes a pre-existing undefined-array-key warning in visit() when traversing a Node with a custom kind. Add regression tests covering generic and kind-specific visitors for a custom Node kind.

spawnia

Are those two independent changes? Please split if so.

@var

- Use a HEREDOC for the tab-character test source, deriving the expected value from it via substr() so input and expectation can't drift apart. - Type-hint $path/$ancestors as array in visitInParallel()'s enter/leave closures, and document $key/$parent via @phpstan-param since PHP 7.4 doesn't support union types. - Add @var docblocks describing the shape of the enter/leave callback caches. - Replace the lazily-built $stopBytes runtime computation in Lexer::readString() with a precomputed STRING_STOP_BYTES constant.

The visit() undefined-array-key warning fix for custom Node kinds, along with its regression tests, has been split out into a separate PR (webonyx#1952) since it is unrelated to the Lexer/Visitor optimization work here.

OpaqueRock · 2026-07-22T14:47:29Z

Thanks for the review! I've pushed two updates:

Addressed the requested changes: converted the tab-character Lexer test to a HEREDOC, added strict param typing (array for $path/$ancestors) plus @phpstan-param docblocks for the enter/leave closures in visitInParallel() (union types aren't available since this targets PHP 7.4), added @var docblocks for the $enterFns/$leaveFns caches, and replaced the runtime-computed $stopBytes string with a STRING_STOP_BYTES class constant.
Split out the unrelated visit() fix: per your suggestion, the visit() undefined-array-key fix for custom Node kinds (and its two regression tests) is no longer part of this PR — it's now a standalone PR: Fix undefined-array-key warning in visit() for custom Node kinds #1952.

This PR is now scoped solely to the Lexer::readString() / Visitor::visitInParallel() optimizations. Let me know if anything else needs adjusting.

spawnia · 2026-07-22T15:12:42Z

I would like the Lexer and Visitor optimizations as separate PRs.

The Visitor::visitInParallel() enter/leave callback caching optimization has been split out into a separate PR (webonyx#1953), so it can be reviewed independently of the Lexer::readString optimization here.

OpaqueRock · 2026-07-22T17:27:47Z

The Visitor::visitInParallel() optimization has been now split out into its own PR, #1953

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

PHP's switch uses loose comparison, so `case null` also matches code 0, causing a backslash followed by a raw NUL byte to be treated as EOF instead of an invalid escape sequence. Check for EOF with a strict comparison before the switch instead.

OpaqueRock · 2026-07-24T04:16:59Z

I looked into the null value comparison and it's a real, reachable bug, though it pre-dates this PR (the switch/case null block existed identically on master; this PR just moved/reformatted it).

The problem: switch in PHP uses loose (==) comparison for each case. So case null: also matches integer 0, not just true EOF. In readString(), after consuming a backslash, $code is 0 for a literal NUL byte (0x00) in the input, and null only on genuine EOF. Because of the loose comparison, a literal NUL byte after a backslash was being misidentified as EOF.

Impact: For input containing \ followed by a raw NUL byte inside a string literal, the lexer silently dropped both bytes and kept parsing, instead of raising a syntax error. E.g. "ab\<NUL>cd" produced the token value 'abcd' with no error at all.

Per the GraphQL spec, a NUL byte is not a valid raw string character and has no dedicated escape (\0 isn't in the EscapedCharacter grammar) — the only spec-compliant way to include it is \u0000. So \<NUL> should always be rejected as an invalid escape sequence, the same as \q or any other unrecognized escape.

Fix: Added an explicit if ($code === null) { break; } strict-comparison check immediately before the switch, and removed the now-dead case null: branch inside the switch. Considered using PHP 8's match expression instead, but the project's floor is PHP 7.4, so that's not available; the strict pre-check is also simpler here since the \u case has multi-statement logic that doesn't map cleanly to match anyway.

Added a regression test (reportsUsefulStringErrors in LexerTest.php) covering \ followed by a raw NUL byte, verified it failed before the fix and passes after.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

spawnia

Comment.

The PR is a sound performance optimization with no correctness bugs. The three surviving comments are minor improvements: a simplification opportunity (mb_strlen over preg_match_all), an open reviewer request for named constants, and a test hygiene suggestion. None rise to the level of requesting changes.

spawnia · 2026-07-24T09:28:47Z

Reviewed at 031cb52 (standard, full)

🤖 Posted by Claude Code

OpaqueRock · 2026-07-25T07:37:08Z

Replaced preg_match_all codepoint-counting with mb_strlen and hardcoded the tab-character test's expected string/length.

Cover the single-char escape sequences (\/, \b, \f, \n, \r, \t), the & punctuator token, and multi-byte UTF-8 in readChar(). 🤖 Generated with Claude Code

spawnia · 2026-07-28T08:47:37Z

Thanks @OpaqueRock! Released as v15.36.0.

OpaqueRock and others added 2 commits July 16, 2026 13:00

Merge remote-tracking branch 'origin/master' into pr-1948-OpaqueRock

ef0969c

spawnia requested a review from Copilot July 21, 2026 13:14

Copilot started reviewing on behalf of spawnia July 21, 2026 13:15 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

spawnia requested a review from Copilot July 21, 2026 13:35

Copilot started reviewing on behalf of spawnia July 21, 2026 13:36 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/Language/Visitor.php Outdated

Comment thread src/Language/Visitor.php Outdated

spawnia reviewed Jul 22, 2026

View reviewed changes

Comment thread tests/Language/VisitorTest.php Outdated

Comment thread tests/Language/LexerTest.php Outdated

Comment thread src/Language/Visitor.php Outdated

Comment thread src/Language/Visitor.php Outdated

Comment thread src/Language/Lexer.php Outdated

OpaqueRock mentioned this pull request Jul 22, 2026

Fix undefined-array-key warning in visit() for custom Node kinds #1952

Open

Remove visit() custom-kind bugfix, split into webonyx#1952

cf27f7e

The visit() undefined-array-key warning fix for custom Node kinds, along with its regression tests, has been split out into a separate PR (webonyx#1952) since it is unrelated to the Lexer/Visitor optimization work here.

OpaqueRock mentioned this pull request Jul 22, 2026

Optimize Visitor::visitInParallel #1953

Open

Remove Visitor optimization, split into webonyx#1953

65089c9

The Visitor::visitInParallel() enter/leave callback caching optimization has been split out into a separate PR (webonyx#1953), so it can be reviewed independently of the Lexer::readString optimization here.

OpaqueRock changed the title ~~Optimize Lexer::readString and Visitor::visitInParallel~~ Optimize Lexer::readString Jul 22, 2026

spawnia requested a review from Copilot July 23, 2026 07:29

Copilot started reviewing on behalf of spawnia July 23, 2026 07:30 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread src/Language/Lexer.php Outdated

spawnia requested a review from Copilot July 24, 2026 08:12

Copilot started reviewing on behalf of spawnia July 24, 2026 08:13 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread tests/Language/LexerTest.php

spawnia reviewed Jul 24, 2026

View reviewed changes

Comment thread src/Language/Lexer.php Outdated

Comment thread src/Language/Lexer.php

Comment thread tests/Language/LexerTest.php Outdated

Use mb_strlen for codepoint counting, simplify tab-character test

cc722fd

spawnia approved these changes Jul 28, 2026

View reviewed changes

Add tests for full Lexer.php coverage

ab0ccf8

Cover the single-char escape sequences (\/, \b, \f, \n, \r, \t), the & punctuator token, and multi-byte UTF-8 in readChar(). 🤖 Generated with Claude Code

spawnia merged commit 47ded87 into webonyx:master Jul 28, 2026
22 checks passed

OpaqueRock deleted the optimize-input-processing branch August 5, 2026 09:09

Uh oh!

Uh oh!

Conversation

OpaqueRock commented Jul 16, 2026 • edited by spawnia Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Benchmarks

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

spawnia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OpaqueRock commented Jul 22, 2026

Uh oh!

spawnia commented Jul 22, 2026

Uh oh!

OpaqueRock commented Jul 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

OpaqueRock commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

spawnia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spawnia commented Jul 24, 2026

Uh oh!

OpaqueRock commented Jul 25, 2026

Uh oh!

Uh oh!

spawnia commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

OpaqueRock commented Jul 16, 2026 •

edited by spawnia

Loading