perf: fast path for absolute http(s) URLs, and better join buffer sizing - #1145
Closed
bartlomieju wants to merge 6 commits into
Closed
perf: fast path for absolute http(s) URLs, and better join buffer sizing#1145bartlomieju wants to merge 6 commits into
bartlomieju wants to merge 6 commits into
Conversation
`ParseOptions::parse` sized the serialization buffer from `input.len()`. That is exact for an absolute URL, but `Url::join` routes through the same path with the *relative reference* as `input`, so the buffer was sized for the reference while the result is roughly the base plus the reference. Every join therefore grew the buffer a reallocation at a time. Measured with a counting global allocator, `join` performed 2 allocations and 3 reallocations against `parse`'s 1 and 0, on every relative form tested, including the short bare specifiers that dominate module resolution. Size the buffer for base + input when the input has no scheme. Inputs that do have a scheme ignore the base, so they keep their exact `input.len()` capacity and are unaffected. join, base pre-parsed 181.5 ns -> 150.0 ns (-17.3%) five module specifiers 778.2 ns -> 533.1 ns (-31.6%) Plain `Url::parse` is unchanged, still 1 allocation and 0 reallocations. The scheme check only sizes the buffer, so a wrong answer costs a little capacity and never correctness.
The general parser walks the input character by character through a `Chars` iterator that re-tests every character for ASCII tab/newline, runs the host through IDNA ToASCII, and re-checks every path byte against a percent-encode set. For the common case -- an already-canonical absolute http(s) URL with a lowercase ASCII domain and a path needing no encoding -- none of that work changes the output. Recognize that case up front and build the `Url` directly: match the scheme with byte compares, validate the host and path against conservative byte classes, and copy the input verbatim as the serialization. The fast path declines anything unusual -- uppercase, userinfo, a port, a query, a fragment, percent-encoding, dot segments, backslashes, IP literals, punycode, non-ASCII, empty labels, a host ending in a number -- so it never has to reproduce the general parser's handling of those. It is attempted only when no base URL, encoding override or violation callback is set. single short parse 159.8 ns -> 54.2 ns (-66%) bulk parse, 100k 33.2 ms -> 23.8 ms (-30%) A differential test asserts that every input the fast path accepts yields a field-for-field identical `Url` to the general parser. Over a 100k real-world URL corpus, 75,526 inputs take the fast path and all agree. WPT stays green.
The fast path scanned the whole host looking for the path separator before validating any of it, so an input it was always going to decline paid for the scan first. Review of the benchmark run flagged a possible ~2-3% regression on `parse_idn` from exactly this. A host that does not begin with a lowercase ASCII letter can never be verbatim, so check that byte before scanning. One compare now covers the common declines -- non-ASCII and punycode hosts, IPv4 (leading digit), IPv6, uppercase -- which previously walked the host only to be rejected afterwards. Purely a reordering: the set of inputs the fast path accepts is unchanged, at 75,526 of the 100,025 corpus URLs, and the differential test, the full suite and WPT all still pass.
Mutation-checked the existing coverage by deliberately breaking the fast path, which found two mutations that only WPT caught and the dedicated tests missed: allowing uppercase inside a host label, and passing `xn--` through verbatim. The first slipped through because the existing uppercase case, `EXAMPLE.com`, is still rejected by the leading-byte check, so it never reached the label scan; the second because the existing punycode case is valid and round-trips unchanged. Added hosts that are uppercase only after a lowercase first byte, and punycode that fails IDNA validation, so both are now caught without relying on WPT. Also broadened the differential cases to cover length boundaries, inputs shorter than the scheme literal, hyphens and digits at label edges, every byte the path class admits and several just outside it, dot segments in each position, and repeated separators. Added public-API tests: one parses each input twice, once with a syntax violation callback to force the general parser, and compares every observable; one mutates a fast-path `Url` through the setters, which is what would catch a wrong component offset that the getters alone would not reveal; and one checks that `join`'s buffer sizing did not change what it resolves to.
Categorizing why the fast path declined 24,499 of the 100,025 corpus URLs found one dominant reason: 22,679 of them (22.7% of the corpus) carry a query string. Everything else combined -- path bytes needing encoding, fragments, ports, userinfo, uppercase hosts, punycode, dot segments -- accounts for under 2%. Accept a query when a path precedes it, validating its bytes against the complement of the special-query percent-encode set. `%` is included, since the parser does not re-encode existing escapes in a query. Fragments stay declined: '#' is outside both byte classes, so they fall back without extra handling. Corpus coverage rises from 75,526 to 96,980 of 100,025 URLs, 75.5% to 97.0%. bulk parse, 100k -22.9% single short parse +1.4% The single-URL cost is the tradeoff and is deliberate: locating the query is fused into the path validation loop rather than run as a separate scan, so a URL without a query pays one extra byte comparison per path byte instead of a second pass. Given queries appear in 22.7% of real URLs, that trade is clearly worth making, but it is not free. `https://host?q` is still declined -- it gains a "/" in its serialization, so the input is no longer copied verbatim, and the shape is rare enough not to justify a second output form.
The fast path refused to run whenever a base URL was set, which meant `Url::join` never used it -- including for an absolute input, where the base is ignored anyway. That guard was stricter than necessary. `parse_with_scheme` consults the base only in the "special relative" state, entered when fewer than two slashes follow the scheme. The fast path accepts nothing but `http://` and `https://`, which is exactly two, so the base is provably ignored for every input it accepts and excluding it bought nothing. An encoding override still changes how a query is serialized and a violation callback still expects to be called, so both remain excluded. join with an absolute specifier 537.4 ns -> 176.6 ns (-66.9%) Relative resolution is unaffected, as it must be: `https:/d` and `https:d` carry a scheme but fewer than two slashes, so they stay in the special-relative state and resolve against the base. Both are covered by a test, alongside one asserting that `join` and `parse` agree for absolute inputs across every base shape -- special, non-special, cannot-be-a-base, and one with userinfo and a port.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1145 +/- ##
=======================================
Coverage ? 87.50%
=======================================
Files ? 26
Lines ? 5424
Branches ? 0
=======================================
Hits ? 4746
Misses ? 678
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
|
Hi! Current Servo AI policy does not allow AI contributions https://book.servo.org/contributing/getting-started#ai-contributions Thank you for working on this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes to speed up parsing.
First,
Url::joinsized its parse buffer from the relative reference, but theresult is roughly the base plus that reference, so every join grew the buffer
through three reallocations. It now sizes for both.
Second, and this is where most of the win is, a fast path for plain absolute
http(s) URLs. Most URLs are already canonical, so there is nothing to normalise,
but the parser still walks them character by character, calls IDNA, and checks
every path byte against a percent-encode set. The fast path spots that case up
front and builds the
Urldirectly. It only takes a lowercase ASCII domain witha path and query that need no encoding, and bails on anything else (uppercase,
userinfo, ports, fragments, percent-encoding, dot segments, IP literals,
punycode, non-ASCII). On a 100k corpus of real URLs, 97% take it.
Url::joinuses it too when the specifier is already absolute, since the base is ignored
there anyway. A differential test checks that everything the fast path accepts
comes out identical to the general parser, field for field. WPT is green.
Benchmarks on an Apple M5, criterion with paired baselines:
join, absolute specifierjoin, relative, base reparsedjoin, relative, base pre-parsedThat last row is the cost of trying the fast path and bailing, which is within
noise here. Happy to share the benchmark harness if it would help review.
I tested all of these changes against
adaand on my machine in most ofthe benchmarks
rust-urlis now faster. Both parsers still give identical resultin the ada test corpus, the only difference is performance.
I made sure this change doesn't clash with #1142
or potential redesign in #1135.
Disclaimer: I used AI to research, implement and test this change.