feat: look for .ilean.mmap files on watchdog load - #14421
Open
robsimmons wants to merge 4 commits into
Open
Conversation
github-actions
Bot
added
the
toolchain-available
|
Mathlib CI status (docs):
|
Collaborator
|
Reference manual CI status:
|
github-actions
Bot
added
the
mathlib4-nightly-available
leanprover-bot
added
the
builds-manual
mathlib-lean-pr-testing
Bot
added
the
builds-mathlib
robsimmons
marked this pull request as ready for review
July 17, 2026 12:55
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/batteries
that referenced
this pull request
Jul 17, 2026
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/mathlib4-nightly-testing
that referenced
this pull request
Jul 17, 2026
leanprover-bot
added a commit
to leanprover/reference-manual
that referenced
this pull request
Jul 17, 2026
robsimmons
added
the
changelog-no
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/batteries
that referenced
this pull request
Jul 19, 2026
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/mathlib4-nightly-testing
that referenced
this pull request
Jul 19, 2026
leanprover-bot
added a commit
to leanprover/reference-manual
that referenced
this pull request
Jul 19, 2026
Vtec234
requested changes
Jul 27, 2026
Vtec234
left a comment
Vtec234
left a comment
Member
There was a problem hiding this comment.
On Linux the effect is even more dramatic, taking us from 1.1G unshared memory for the watchdog to about 67M unshared memory for the watchdog.
Do you know why the unshared amount is larger on macOS?
Contributor
Author
No, but I believe it has to do with larger minimum page size and the fact that we are loading thousands of sometimes tiny little files. |
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/batteries
that referenced
this pull request
Jul 29, 2026
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/mathlib4-nightly-testing
that referenced
this pull request
Jul 29, 2026
leanprover-bot
added a commit
to leanprover/reference-manual
that referenced
this pull request
Jul 29, 2026
Vtec234
reviewed
Jul 29, 2026
Vtec234
reviewed
Jul 29, 2026
robsimmons
requested review from
TwoFX,
datokrat,
hargoniX and
sgraf812
as code owners
July 29, 2026 19:57
mathlib-lean-pr-testing
Bot
added
breaks-mathlib
robsimmons
removed request for
Garmelon,
TwoFX,
datokrat,
digama0,
hargoniX,
kim-em,
kmill,
leodemoura,
sgraf812 and
tydeu
July 29, 2026 20:32
github-actions
Bot
removed
the
changes-stage0
Vtec234
approved these changes
Jul 30, 2026
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/batteries
that referenced
this pull request
Jul 30, 2026
leanprover-bot
added a commit
to leanprover/reference-manual
that referenced
this pull request
Jul 30, 2026
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/mathlib4-nightly-testing
that referenced
this pull request
Jul 30, 2026
leanprover-bot
added
builds-manual
mathlib-lean-pr-testing
Bot
added
builds-mathlib
Comment about compactedIleanErrors > 0 warning.
robsimmons
commented
Jul 30, 2026
Comment on lines
+1715
to
+1719
|
|
||
| -- This warning message only makes sense if we're only building .ilean.mmap files for read-only toolchain and library files. | ||
| -- For other use cases, this will be too noisy. | ||
| if compactedIleanErrors > 0 then | ||
| (← getStderr).putStrLn s!"Attempted to load {compactedIleanCount} .{Ilean.compactedExt} {compactedIleanCount.plural "file"}, but failed to load {compactedIleanErrors} of them" |
Contributor
Author
There was a problem hiding this comment.
I would be happy to simplify this PR even further by removing this error reporting — but given the intended workbench use case of generated .ilean.mmap files for read-only toolchain and library files, I thought the warning was prudent.
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/batteries
that referenced
this pull request
Jul 30, 2026
mathlib-nightly-testing Bot
pushed a commit
to leanprover-community/mathlib4-nightly-testing
that referenced
this pull request
Jul 30, 2026
leanprover-bot
added a commit
to leanprover/reference-manual
that referenced
this pull request
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces a new, memory-mapped route to loading
ileanobjects into memory. This feature should be treated as undocumented and subject to change without notice in future releases.Lean4web and Workbench memory requirements are, at present, somewhat dominated by the watchdog's ileans. Mmapping ileans has a similar unshared-memory reduction to just deleting the ileans completely. (On live.lean-lang.org we've just deleted them, which slightly degrades the user experience.)
This design looks for
.ilean.mmapfiles aside.ileanfiles one time, on watchdog start, and prefers the.ilean.mmapfile if it has reports a last-modified time that is greater than or equal to the.ilean's last-modified time.There's no mechanism for creating
.ilean.mmapfiles, which is intentional: we only intend for this feature to be used for lean4web and workbench environments that we control at first so that we can back out of this design later on. (The robsimmons/milean repository shows one way of building these.) Unless someone goes out of their way to create.ilean.mmapfiles in their.lakedirectories, this PR adds the cost of one failed file lookup per ilean.Benchmarking
These numbers were generated with a slightly different design, but the changes I've made shouldn't make a difference here.
In my benchmarking (on OSX, with a mathlib nightly-testing 2026-07-15 compiled against this branch):
.ilean.mmapfiles for just the.ileanfiles in Mathlib: watchdog needs 1.1G memory -> watchdog needs 496M memory.ilean.mmapfiles for all.ileanfiles, including those in the core release: watchdog needs 1.1G memory -> watchdog needs 393M memory.On Linux the effect is even more dramatic, taking us from 1.1G unshared memory for the watchdog to about 67M unshared memory for the watchdog. (This measurement includes the additional effect of 085ab16 — without that change, Linux unshared memory usage "only" goes down to about 171M.)