feat: Lock-free apply_block refactor by sergerad · Pull Request #2345 · 0xMiden/node

sergerad · 2026-07-15T03:05:41Z

Summary

Closes #1539.
Closes #1853.
Closes #2414.

Makes the store's block-write path lock-free for readers, and makes the resulting read-consistency rules type-enforced. Reads previously contended on RwLocks over the in-memory trees (and blocked during apply_block's DB-commit window); they now load an immutable snapshot via ArcSwap and are never blocked by writes. All reads flow through a request-scoped StateView, so a query combining tree and DB data at different chain heights is no longer expressible.

                                   LoadedState::start()
                                           │ spawns worker task, hands out one of each
             ┌─────────────────────┬───────┴────────────┬─────────────────────┐
             ▼                     ▼                    ▼                     ▼
      ┌────────────┐        ┌─────────────┐      ┌─────────────┐      ╔══════════════╗
      │ Arc<State> │        │ BlockWriter │      │ ProofWriter │      ║  WriteWorker ║
      │ (read-only,│        │ (write cap, │      │ (write cap, │      ║ (tokio task) ║
      │  shared)   │        │  1 holder)  │      │  1 holder)  │      ╚══════════════╝
      └────────────┘        └─────────────┘      └─────────────┘        owns MUTABLE
             │                     │                    │               trees: nullifier,
             │       apply_block() │      apply_proof() │               account, MMR,
             │                     │                    │               forest
             │         WriteRequest│                    ├─ commit proof       │
             │             mpsc(1) │                    │  to block store     │ per committed
             │                     ▼                    │                     │ block: builds
             │                     ═══════════▶ ════════╪═══════════▶         │ + publishes
             │                                          │                     ▼
             │                                          │              ┌────────────────┐
      State fields                                      │              │ StateSnapshot  │
      ┌───────────────────────────────────────┐         │              │  (immutable,   │
      │ latest_snapshot: Arc<ArcSwap<─────────── swap on commit ──────▶│  per block N)  │
      │                        StateSnapshot>>│         │              │  tree READER   │
      │ committed_tip_tx: watch ◀── fired by worker     │              │  views + MMR   │
      │ proven_tip:       watch ◀── advanced by ────────┘              │ + SnapshotGuard│
      │ db, block_store, block/proof caches   │                        └────────────────┘
      └───────────────────────────────────────┘                                ▲
             │                                                                 │ pins ONE
             │ view()  (wait-free ArcSwap load, one per request)               │ generation
             ▼                                                                 │
      ┌─────────────────────────────────────────────┐                          │
      │ StateView  { snapshot: Arc<StateSnapshot> ──┼──────────────────────────┘
      │              db:       Arc<Db>            } │
      │  • tip() = snapshot height                  │   ALL reads live here:
      │  • DB queries scoped by tip                 │   get_account, get_*_inputs,
      │    (ScopedBlockNum / ScopedBlockRange)      │   sync_*, get_block_header, …
      │  • trees only via block_in_place helpers    │
      └─────────────────────────────────────────────┘

  Writes flow LEFT→RIGHT (capability → worker → snapshot); reads flow DOWN (State → view →
  pinned snapshot + tip-scoped DB). Live tips (committed/proven) bypass snapshots via watch
  channels. Readers never block writes; a view keeps serving block N while the worker
  publishes N+1.

Why:

Read endpoints (sync, account/nullifier proofs, chain tip) no longer stall while a block is applied.
Removes the fragile cross-task lock choreography in apply_block (oneshot handshakes between the DB task and the in-memory update).
Snapshot-scoped reads were previously enforced only by convention — several paths read the tip and the data from different snapshots. StateView makes the scoping structural instead.

How:

Lock-free write path (state/writer/)

A single WriteWorker task owns the mutable nullifier tree, account tree, blockchain MMR, and account-state forest, processing blocks serially from an mpsc channel — no locks. In-flight writes always complete; shutdown is only observed between requests.
After each DB commit, the worker builds an immutable StateSnapshot (trees backed by read-only RocksDB snapshot views) and publishes it atomically via ArcSwap, so readers keep a consistent frozen view while the next block commits.
Db::apply_block is now a plain transaction — the oneshot allow_acquire/acquire_done synchronization is removed.

Write capabilities (state/lifecycle.rs)

LoadedState::start spawns the worker and returns the read-only Arc<State> plus non-cloneable BlockWriter/ProofWriter capabilities and a WriterTask handle, statically limiting each write path to one task. The capabilities expose no read access; tasks that read and write get Arc<State> alongside their capability.
BlockWriter::stop drains and joins the worker so tree storage is released deterministically before the data directory is re-opened or deleted (used by recover and stress-test seeding).

Type-enforced reads (state/view/)

All tree and DB reads live on StateView, pinned to one snapshot per request (State::view()). DB queries are scoped by the view's tip internally; callers cannot supply their own. RocksDB-backed trees are only reachable through block_in_place helpers; snapshot fields are only visible inside the view module.
Tip-scoped Db queries require view-issued proof types (ScopedBlockNum / ScopedBlockRange), constructible only by a StateView after validating the bound against its tip — extending the enforcement to the DB boundary itself.
Range-scoped sync queries validate range.end() <= tip themselves via a new RangeBeyondTip error (same InvalidArgument response as before); the RPC layer's range_bounds_check is deleted and pagination's chain_tip is now the tip the query actually ran against.
Fixes paths that previously took two snapshots per request (get_account, the block producer's get_tx_inputs); sync_chain_mmr clamps the proven tip to the view's tip.

Live tips (state/tip.rs)

State::committed_tip() / proven_tip() read the watch channels their writers publish to (mirroring subscribe_committed_tip / subscribe_proven_tip); the Finality enum is removed. The committed tip is published after the snapshot, so it never reports a block a fresh view cannot serve.

Observability: SnapshotGuard tracks live snapshot generations and lifetimes; warns when a snapshot outlives 10s or more than 4 generations are pinned (a leaked/slow reader pins a RocksDB snapshot).

Supporting changes: read-only reader() views for AccountStateForest / AccountTreeWithHistory (relaxed to BackendReader/SmtStorageReader bounds); state module restructured into view/ (read endpoints) and writer/ (worker + capabilities); new tracing field names allowlisted.

Changelog

[[entry]]
scope       = "node"
impact      = "changed"
description = "Store reads are lock-free: readers use atomically published in-memory snapshots and are no longer blocked while blocks are applied."

…ckfree-store-state

sergerad · 2026-08-02T21:22:44Z

@@ -0,0 +1,419 @@
+use std::collections::HashSet;


Much of the diffs here are a move of impl but partial / involving a split to other files. The only changes should be w.r.t BlockNumber -> ScopedBlockNumber and ScopedBlockRange.

sergerad · 2026-08-02T21:46:39Z

+                conn,
+                note_commitments.as_slice(),
+                up_to_block,
+            )


This up_to_block bound may technically not be required but I think it might still be a good idea to use it for any path that involves multiple DB calls - some of which are block bound sensitive. If anything, to catch / not be effected by desync-like bugs we would not expect.

…ckfree-store-state

Mirko-von-Leipzig

Some comments; but the size of the PR does somewhat force our hand towards merging this and fixing things in post.

A more staggered/stacked approach could have been:

snapshot API + one or two example impls
more impls
lastly update apply block

I do think we can do better once we have sqlite added; but this is a decent stepping stone.

Mirko-von-Leipzig · 2026-08-03T11:42:29Z

Does recovery still have a place in a multi-validator world? I guess if they each resign each block?

Current impl is maybe one piece of what we would need. Lets followup #2427

Mirko-von-Leipzig · 2026-08-03T14:25:24Z

Should we even offer this API outside of a view?

Yea it seems wrong at first glance but is actually the right thing to do. StateViews are intended to be short-lived because they hold StateSnapshots which we want to release ASAP. The committed / proven tips are for long-held stream use cases. And they don't involve the rest of data in StateSnapshot at all.

Updated comment to elaborate on that.

Mirko-von-Leipzig · 2026-08-03T14:35:26Z

+    let (current_block_height, store_inputs) = state
+        .with_view(async |view| {


Why do we use with_state as a closure?

Do you mean with_view? Mainly so that a set of view calls are ensured to use the same snapshot (because state.view() could return a different snapshot different calls).

Mirko-von-Leipzig · 2026-08-03T14:40:23Z

+    /// Store write capabilities consumed by the full-node sync loop.
+    ///
+    /// Must be provided in full-node mode and omitted in sequencer mode. The RPC service itself
+    /// only ever reads through `state`; these are passed through untouched to the sync tasks.


Then this should probably form part of the enum?

Yea it might be a little bit involved so will do it as a followup #2428

Mirko-von-Leipzig · 2026-08-03T14:44:27Z

+    /// Runs a read operation over a view pinned at the current chain tip, dropping the view — and
+    /// releasing its snapshot generation — as soon as the operation completes.
+    ///
+    /// This is the required form whenever multiple reads must observe the *same* snapshot: the
+    /// closure's view is one consistent generation, whereas consecutive [`Self::view`] calls may
+    /// straddle a commit. The typical case is pairing a query with [`StateView::tip`] so a
+    /// response reports exactly the height it was served at.


tbh I don't quite understand the need for view and with_view. Why can't the caller hold the view for as many calls as they want?

Both are intended to be used so that the lifetime of the StateView (and underlying Arc<StateSnapshot>) are as short as possible.

So for example

// Bad let view = state.view(); let tip = view.tip(); // more unrelated instructions... // scope finally ends } // Good let tip = state.view().tip();

And

// Bad let view = state.view(); let x = view.get_accounts(); let y = view.compute_something(x); // more unrelated instructions... // scope finally ends } // Good // Snapshot tightly scoped state.with_view(...); // unrelated instructions don't affect lifetime of snapshot

I understand in general, but because we offer both APIs there isn't really any safety in that; its an improvement in cases where we should otherwise manually drop.

But I don't know of any such cases? I would expect a view to live for 99% of an RPC request's lifespan.

Mirko-von-Leipzig · 2026-08-03T16:21:21Z

+    /// moving the closure body into `block_in_place`.
+    fn with_inner_read_blocking<R>(&self, f: impl FnOnce(&StateSnapshot) -> R) -> R {
+        let span = Span::current();
+        tokio::task::block_in_place(|| span.in_scope(|| f(&self.snapshot)))


There are very few places on should use block_in_place - ideally never / only when manually starting a new async runtime.

Can we make this spawn block instead?

@kkovaacs I believe you originally implemented with_inner_read_blocking let us know your thoughts. The 'static lifetime bound of spawn_blocking might be an issue here.

Mirko-von-Leipzig · 2026-08-03T16:27:19Z

+
+        // The DB is committed at this point, so the prepared mutations must be applied and any
+        // failure to do so aborts the process.
+        let snapshot = tokio::task::block_in_place(|| {


Prefer spawn blocking

We can't move self into spawn_blocking (requires 'static lifetime). Also I think block_in_place is fine given this is all called serially by one task.

Mirko-von-Leipzig · 2026-08-03T16:30:22Z

+            .unwrap_or_else(|error| {
+                Self::abort_after_post_commit_failure("nullifier tree", &error)
+            });


Are we aborting because the apply should always succeed?

Why not just panic?

The idea is to exit the entire application instantly rather than panic this task and have upstream join handle it. Unsure if that has real world value however.

Yes but why do we want to abort at all? Why not panic and log? If we abort we get zero telemetry.

…ckfree-store-state

igamigo

Not a full review (hopefully by tomorrow morning), but so far it's looking good. Left just one comment for now

igamigo · 2026-08-04T03:57:05Z

+                .reader()
+                .expect("nullifier tree snapshot creation should not fail"),
+            account_tree: self.account_tree.reader(),
+            blockchain: self.blockchain.clone(),


Is this fine to do? I haven't done the math but wouldn't this self.blockchain grow into the hundreds of MBs fairly quickly (weeks/months)? Not sure there is an easy way to work around this though. This might have been accounted for already

sergerad added 18 commits July 15, 2026 15:04

Initial apply_block lock-free refactor

7195581

Add snapshot count observability

5c7dbeb

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-lo…

b7699fc

…ckfree-store-state

Fix ui tests

46b8df6

Lint

bb3cf85

Warn about long lived snapshots

891d702

Warn on snapshot count

8c2bc02

Scape all requests

8373d9b

Lint

3a27431

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-lo…

8fa1502

…ckfree-store-state

Replace result with control flow

c21a950

Compartmentalize apply_block

0b562a8

snapshots_live fn

ddf4f15

Shutdown writer

6556067

Lint

2ad2bbf

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-lo…

09b11f8

…ckfree-store-state

Fix comments

dc1e08a

Lint

d7e6c52

sergerad requested review from Mirko-von-Leipzig and kkovaacs July 23, 2026 01:18

sergerad marked this pull request as ready for review July 23, 2026 01:18

sergerad added 8 commits July 23, 2026 09:31

Add branch comment

d14362e

Fix shutdown

d6bf7bd

Shutdown integration without token

220f911

integrate cancellation token

8a531c2

Refactor writer task management

3b648ff

Split module

aea62ce

WriterTask

7f8ebc5

Add benchmarks

c528f19

sergerad mentioned this pull request Jul 23, 2026

test: Benchmarks for lock free apply_block #2370

Merged

sergerad added 4 commits August 3, 2026 09:04

Fix comments

3d972cb

RM unnecessary comment

f4e54ad

Fix comment field names

8e9f20c

Move import

7de6dcd

sergerad commented Aug 2, 2026

View reviewed changes

More scoped usage

9166468

sergerad commented Aug 2, 2026

View reviewed changes

sergerad added 11 commits August 3, 2026 09:50

Fix comments

f50bc3c

Scope before mmr query

eacb766

ScopedBlockNum deref

59927a6

RM scoped_tip

3ea0392

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-lo…

6dbe464

…ckfree-store-state

Fix entries_limit

b5e3b57

Shadow blocks

f807adb

Fix var name

fde0d90

RM unnecessary db()

bd36c48

Fix scope_block

d990f4c

Fix committed tip w/o subs

adab95a

sergerad requested a review from Mirko-von-Leipzig August 3, 2026 02:21

sergerad added 2 commits August 3, 2026 19:20

Update with_view comment

862f5d9

Fmt comment

dddafb3

Mirko-von-Leipzig approved these changes Aug 3, 2026

View reviewed changes

bobbinth requested a review from igamigo August 3, 2026 17:30

sergerad mentioned this pull request Aug 3, 2026

Re-design the recovery command in the context of multi-validators #2427

Open

sergerad added 3 commits August 4, 2026 08:40

Update tip api comment

8c13ffd

Move use and mod statements

f593daf

Move cancellation token

5c8b840

sergerad mentioned this pull request Aug 3, 2026

Integrate Rpc::sync_writers into RpcMode::FullNode #2428

Open

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-lo…

1b8a079

…ckfree-store-state

igamigo reviewed Aug 4, 2026

View reviewed changes

		let (current_block_height, store_inputs) = state
		.with_view(async \|view\| {

Uh oh!

Conversation

sergerad commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changelog

Uh oh!

sergerad Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mirko-von-Leipzig left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sergerad Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

igamigo left a comment

Choose a reason for hiding this comment

Uh oh!

igamigo Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sergerad commented Jul 15, 2026 •

edited

Loading

sergerad Aug 2, 2026 •

edited

Loading

sergerad Aug 3, 2026 •

edited

Loading

igamigo Aug 4, 2026 •

edited

Loading