perf: store per-module constant prefix trees in oleans#14362
Draft
Kha wants to merge 21 commits into
Draft
Conversation
Member
Author
|
!bench |
|
Benchmark results for 7a68001 against 8006bb0 are in. There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (55✅, 329🟥) Too many entries to display here. View the full report on radar instead. Medium changes (7✅, 1528🟥) Too many entries to display here. View the full report on radar instead. Small changes (5✅, 816🟥) Too many entries to display here. View the full report on radar instead. |
github-actions
Bot
added
toolchain-available
Collaborator
|
Reference manual CI status:
|
leanprover-bot
added
the
breaks-manual
Member
Author
|
!bench mathlib |
|
Benchmark results for leanprover-community/mathlib4-nightly-testing@48fbe78 against leanprover-community/mathlib4-nightly-testing@5edfd4c are in. There are significant results. @Kha
No significant changes detected. |
Member
Author
|
!bench |
|
Benchmark results for 736e8af against 8006bb0 are in. There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (56✅, 232🟥) Too many entries to display here. View the full report on radar instead. Medium changes (5✅, 1101🟥) Too many entries to display here. View the full report on radar instead. Small changes (6✅, 1324🟥) Too many entries to display here. View the full report on radar instead. |
This PR replaces the eager construction of the imported-constants hash maps in `finalizeImport` (`privateConstantMap`, `publicConstantMap`, `const2ModIdx`; ~530k insertions for `import Lean`) with prefix trees over each module's constant names that are precomputed at olean write time and remain in the memory-mapped compacted regions. Import now only merges the per-module trees, allocating fresh nodes solely for name prefixes shared between modules (~3k nodes / ~81k edges for the stdlib), and lookups walk the merged view, disambiguating children by cached per-prefix `Name` hash. `import Lean` drops from ~527ms to ~393ms; olean sizes grow by ~6%. Details: * New `Lean.ConstTrie`: generic per-module tree (`ConstTrie α`), merged import view (`ImportedConsts α`) with single-module subtrees borrowed from the regions and module-index tagging, and the new `ConstMap` structure (`imported` view + `locals` map). * `ModuleData` gains `constTrie`/`extraConstTrie` fields, filled centrally in `saveModuleData(Parts)`. The extra-const tree replaces the 325k codegen-name entries of `const2ModIdx` (`Kernel.Environment` now holds `allImportedConsts`/`importedExtraConsts` instead, serving `getModuleIdxFor?`). * Duplicate imported constants are resolved (incl. `subsumesInfo` realizability subsumption and `throwAlreadyImported`) from the merge's reported collisions. * In `saveModuleDataParts`, all parts' data must stay alive across the chained compactor saves; the tries are therefore attached to all parts up front (the compactor tracks compacted objects by heap address, so per-part temporaries would corrupt later parts via address reuse). * `stage0/src/stdlib_flags.h` temporarily enables `interpreter.prefer_native` as this is an ABI-breaking change to `Kernel.Environment`/`ConstMap` (to be reverted by `update-stage0`). Stage 1 tests are meaningless on this commit since the stage 1 oleans lack the new `ModuleData` fields; validate against stage 2. Co-Authored-By: Claude
This PR speeds up constant lookups in the prefix-tree-backed environment by storing each trie node's child key hashes in a `ByteArray` alongside the children, so the per-level binary search runs over contiguous memory instead of chasing child and key pointers, in the memory-mapped per-module trees as well as the merged import spine. This supersedes the `indexed` hash-map wrappers on wide merged nodes, which could not cover wide nodes inside single-module region subtrees and made `Runtime.markPersistent` walk their entries on every import. Co-Authored-By: Claude
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/batteries
that referenced
this pull request
Jul 11, 2026
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/mathlib4-nightly-testing
that referenced
this pull request
Jul 11, 2026
leanprover-bot
added a commit
to leanprover/reference-manual
that referenced
this pull request
Jul 11, 2026
leanprover-bot
added
builds-manual
Member
Author
|
!bench |
|
Benchmark results for 2abb3e2 against 8006bb0 are in. There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (49✅, 433🟥) Too many entries to display here. View the full report on radar instead. Medium changes (9✅, 1889🟥) Too many entries to display here. View the full report on radar instead. Small changes (4✅, 369🟥) Too many entries to display here. View the full report on radar instead. |
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/batteries
that referenced
this pull request
Jul 11, 2026
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/mathlib4-nightly-testing
that referenced
this pull request
Jul 11, 2026
leanprover-bot
added a commit
to leanprover/reference-manual
that referenced
this pull request
Jul 11, 2026
This PR restores the intended final form of the trie child index, which the previous commit accidentally carried in an interim form due to a workspace snapshot race: instead of a binary search that composes each probed hash from eight byte loads, the child index stores `[k][2^k + 1 u16 offsets][one fingerprint byte per child]`; the query hash's top `k` bits select a window of on average at most one child, a fingerprint byte filters it, and the final name component comparison (sound on its own, as siblings always differ in their final component) confirms the match. Co-Authored-By: Claude
Member
Author
|
!bench |
|
Benchmark results for e957cab against 8006bb0 are in. There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (52✅, 126🟥) Too many entries to display here. View the full report on radar instead. Medium changes (7✅, 560🟥) Too many entries to display here. View the full report on radar instead. Small changes (6✅, 1925🟥) Too many entries to display here. View the full report on radar instead. |
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/batteries
that referenced
this pull request
Jul 11, 2026
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/mathlib4-nightly-testing
that referenced
this pull request
Jul 11, 2026
leanprover-bot
added a commit
to leanprover/reference-manual
that referenced
this pull request
Jul 11, 2026
|
Benchmark results for leanprover-community/mathlib4-nightly-testing@50dff9c against leanprover-community/mathlib4-nightly-testing@5edfd4c are in. There are significant results. @Kha
Large changes (1🟥)
Medium changes (2✅)
Small changes (262🟥)
|
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/batteries
that referenced
this pull request
Jul 16, 2026
github-actions
Bot
added
the
mathlib4-nightly-available
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/mathlib4-nightly-testing
that referenced
this pull request
Jul 16, 2026
leanprover-bot
added a commit
to leanprover/reference-manual
that referenced
this pull request
Jul 16, 2026
Member
Author
|
!bench |
|
Benchmark results for 7e710c4 against 8006bb0 are in. (These commits have already been benchmarked in a previous command.) There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (66✅, 12🟥)
Medium changes (27✅, 12🟥)
Small changes (52✅, 2434🟥)
|
Member
Author
|
!bench |
|
Benchmark results for 4716d7a against 8006bb0 are in. There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (63✅, 11🟥)
Medium changes (16✅, 17🟥)
Small changes (46✅, 2450🟥)
|
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/batteries
that referenced
this pull request
Jul 16, 2026
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/mathlib4-nightly-testing
that referenced
this pull request
Jul 16, 2026
leanprover-bot
added a commit
to leanprover/reference-manual
that referenced
this pull request
Jul 16, 2026
Member
Author
|
!bench |
|
Benchmark results for 4dd5587 against 8006bb0 are in. There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (62✅, 10🟥)
Medium changes (22✅, 20🟥)
Small changes (43✅, 2446🟥)
|
Member
Author
|
!bench mathlib |
|
Benchmark results for leanprover-community/mathlib4-nightly-testing@e583f21 against leanprover-community/mathlib4-nightly-testing@5edfd4c are in. There are significant results. @Kha
Large changes (1🟥)
Medium changes (2✅)
Small changes (3862🟥)
|
Member
Author
|
!bench |
Member
Author
|
!bench mathlib |
|
Benchmark results for 1bc40e1 against 8006bb0 are in. There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (61✅, 12🟥)
Medium changes (23✅, 16🟥)
Small changes (45✅, 2445🟥)
|
|
Benchmark results for leanprover-community/mathlib4-nightly-testing@717c4e4 against leanprover-community/mathlib4-nightly-testing@5edfd4c are in. There are significant results. @Kha
Large changes (1🟥)
Medium changes (2✅)
Small changes (3171🟥)
|
Member
Author
|
!bench |
|
Benchmark results for 61fdbb4 against 8006bb0 are in. There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (73✅, 12🟥)
Medium changes (42✅, 9🟥)
Small changes (45✅, 2388🟥)
|
Member
Author
|
!bench |
|
Benchmark results for 9d845af against 8006bb0 are in. There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (65✅, 12🟥)
Medium changes (20✅, 12🟥)
Small changes (52✅, 2444🟥)
|
The constants view was still merged eagerly at import because conflicting declarations must be reported then. Split the two concerns: the view is now merged lazily like the extra-constants view, with the subsumption rule for names declared by several modules applied inline when a node's entry is first accessed, and conflict detection runs as a separate check over a duplicate sweep that walks only name prefixes occurring in several modules. The check exempts module pairs certified by one of the import's direct imports: replaying the loader's level propagation from each direct import's perspective (bit-parallel over one reverse pass of the topologically sorted modules) shows which modules' data that import's own compile loaded, and at which level; pairs it loaded at the current import's levels were checked by that compile, recursively down to each pair's first co-importer. The sweep skips certified subtrees, so module-system compiles pay almost nothing; compiles without the module system re-check name universes no prior compile loaded, such as non-public names of module-system dependencies. Locally, module-file compiles drop to the lazy merge's numbers (`Init.Data.Sum` -4.5% instructions vs the eager merge) and a clean stdlib rebuild is unchanged; a non-module `import Lean` pays the full check (+1.6%) as its `all`-level universe has no certifier. Validated by the full-stdlib trie oracle, the duplicate-sweep-vs-eager-merge equivalence check, and the leanchecker conflict suite including the level-escalation case `PrivateConflictC`; `ReplaceAxiom` now passes the environment's IR extension state through to its rebuilt environment as code generation of its own module accesses it. Co-Authored-By: Claude
Member
Author
|
!bench |
|
Benchmark results for 5d27b34 against 8006bb0 are in. There are significant results. @Kha Warning These warnings may indicate that the benchmark results are not directly comparable, for example due to changes in the runner configuration or hardware.
Large changes (67✅, 12🟥)
Medium changes (24✅, 17🟥)
Small changes (50✅, 2420🟥)
|
Member
Author
|
!bench mathlib |
|
Benchmark results for leanprover-community/mathlib4-nightly-testing@8d7209e against leanprover-community/mathlib4-nightly-testing@5edfd4c are in. There are significant results. @Kha
Large changes (1🟥)
Medium changes (2✅)
Small changes (2✅, 103🟥)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR replaces the eager construction of the imported-constants hash maps at import time with per-module prefix trees that are precomputed at olean write time, live in the memory-mapped compacted regions, and are merged lazily on first access. Imports get substantially faster and lighter —
import Leanwall clock -28%, mathlib's import phase -21%, build memory -5% on stdlib and mathlib alike, header snapshot saves -17% wall at -23% snapshot size — in exchange for ~6% larger stdlib oleans (~4% for mathlib) and 1-3% more build instructions.Details:
Lean.ConstTrie: per-module prefix trees (ConstTrie α) over constant names, stored inModuleData(constTrie/extraConstTrie, filled centrally insaveModuleData(Parts)), with children disambiguated by cached per-prefixNamehashes through a compact per-node byte index.ImportedConsts α, the merged import view: single-module subtrees are borrowed directly from the olean regions, prefixes shared between modules get nodes computed lazily on first access, and the subsumption rule for names declared by several modules is applied inline when a node's entry is first accessed.lean_mark_persistent_unshared, which skips multi-threaded subgraphs.Environment.const2ModIdxis memoized in a process-global cache instead of aThunkin the kernel environment, and--incr-header-savepre-forces the lazy views intruncateToHeader, keeping thunks, closures, and the memoized map out of compacted snapshots; the lookup cache is reset after persisting.getModuleIdxFor?.interpretationprofile category.