core, miner, consensus/bor, eth, triedb/pathdb: pipelined state root computation for block import

[pratikspatil024]

Description

Overlaps state-root computation (SRC) of block N with transaction execution of block N+1 during block import. Execution of N+1 opens state at the last committed trie root plus an in-memory FlatDiff overlay of block N's mutations, while a background SRC goroutine commits block N and computes its root. Disabled by default — enabled with [pipeline] enable-import-src = true.

Built on top of the delayed SRC PoC: instead of just deferring SRC, it pipelines SRC with the next block's work.

Measured impact

Mainnet full node (n2d-standard-16, path scheme, BlockSTM enforce), ~64k-block catch-up windows, block-weight and traffic-character controlled across adjacent weekday ranges:

configuration	agg mgasps	vs baseline
pipeline off	291.0	—
pipeline on	346.4	+19.0%
pipeline off + producewitnesses	255.3	—
pipeline on + producewitnesses	293.7	+15.0%

Per-block overhead beyond raw execution dropped from ~45–50ms to ~18ms. Zero root_mismatch, pipeline fallbacks, or execution errors across the benchmark campaign (~500k blocks imported under the pipeline).

How it works — import

When importing block N (pipeline active):

1. Execute block N at the last committed root + FlatDiff overlay from block N−1
2. Run ValidateStateCheap (gas, bloom, receipt root — no IntermediateRoot)
3. Extract block N's FlatDiff via CommitSnapshot (~1ms, no trie hashing)
4. Collect the previous SRC(N−1): verify its root, write + announce the witness, trie GC
5. Write block metadata and advance the head immediately (sync protocol sees it)
6. Store the FlatDiff so StateAt/RPC reads (eth_call, eth_estimateGas, pending reads) are correct during the pipeline window
7. Spawn SRC(N) in the background — it overlaps with block N+1's execution
8. Continue to the next block without waiting for SRC(N)

Two supporting optimizations ship with the pipeline:

• pathdb trie-node index (triedb/pathdb/lookup_nodes.go): a reference-counted (owner, path, hash) → blob map over live diff layers. reader.Node resolves in one probe instead of walking up to 128 diff layers per read (the walk measured ~44% of process CPU during catch-up). The hash is part of the key, so a hit is correct by construction; a definitive miss reads the disk layer directly. This index is always on (not gated by the pipeline flag) — it is a read-path cache whose misses fall through to the existing walk, adding ~250MB steady-state at 128 diff layers.
• Warm-node handoff (core/state/warm_snapshot.go, pipeline.warm-snapshot, default on): when witnesses are produced, the execution-side trie prefetcher is detached and handed to SRC, which builds an immutable hash-verified snapshot of the loaded trie nodes and consults it before pathdb. No effect when witnesses are off.

Review-sensitive properties

• The head advances before the state root is verified. ValidateStateCheap gates insertion; the full root check happens asynchronously in the SRC goroutine. A mismatch fires chain/imports/pipelined/root_mismatch (a hard alarm that must stay 0) and errors the pipeline. Sync compensates: hasPendingPipelinedHeadState keeps the node in full-sync while a head-state commit is in flight.
• WIT protocol behavior: WitnessReadyEvent now push-announces witness availability to stateless peers (replacing a 10s poll); GetWitness serving waits for in-flight SRC (bounded, with a header-existence DoS guard) and reads uncached so peer traffic can't evict import-critical cache entries.
• Witness completeness: witness validity under the pipeline reduces to the completeness of the FlatDiff read surface — execution reads at the last committed root (root_{N-2}) plus the parent FlatDiff overlay, so the correct-generation (root_{N-1}) proof nodes can only come from the SRC's re-read at the parent root, driven by that read surface. The review found (comment thread) that the surface was originally rebuilt from finalDB.stateObjects and undercounted three read classes: BlockSTM worker reads, overlay-served reads, and reader-walked read-only slots of accounts the block also mutates. Fixed in 12bd14abe: CommitSnapshot drains the shared reader cache and an overlay-read tracker (both attribution-free, complete by construction), and the SRC statedb drains its reader tracers via CollectStateWitness (the same call core, core/state: produce complete witnesses under BlockSTM v2 #2333 added on the execution side). Pinned by TestV2WitnessRegenerationPipelinedSRCChained, a chained round trip over all 222 consecutive mainnet fixture pairs that reproduces the steady-state generation offset (block N executed over the parent's FlatDiff overlay) and stateless-replays the SRC-completed witness against real mainnet roots — it failed on every sampled pair before the fix. The single-block oracles (TestV2WitnessRegenerationAllBlocks, TestV2WitnessRegenerationPipelinedSRCAllBlocks, 241 blocks) remain as same-generation regression checks; they are structurally blind to the cross-block offset, which is why they alone were insufficient. Caveat: three fixture blocks carry a known pre-existing witness gap (EIP-7702 authority pre-state code missing from the fixture archive — see the PR discussion) and are pinned by exact hash; a live-node witness-replay A/B on the fixed build is in progress to confirm end-to-end, and prewalk volume / settle-drain numbers will be re-measured during the soak on the fixed build. PropagateReadsTo in checkAndCommitSpan captures the validator-contract proof nodes read via a copied statedb; EIP-2935 accounts are touched into the read set.
• Lifecycle: one SRC goroutine + one auto-collection goroutine per pipelined block, registered on bc.wg. Pending SRC is flushed on reorg, gap, shutdown, and a witness-fed block; on a collect failure (SRC error or root mismatch) the collecting thread runs a full rollback — clears the pending entry, drops the FlatDiff overlay, deletes the rejected block's canonical hash and tx lookups, moves the head back, dereferences the divergent root, and emits a corrective ChainHeadEvent. SetHead/setHeadBeyondRoot rewinds do not flush; they rewind to committed state, which supersedes any in-flight SRC.
• No database schema change (BlockChainVersion unchanged); nothing forces a resync.

Production-side (miner) pipelining: landed but disabled

miner/pipeline.go contains the speculative-sealing counterpart (FlatDiff extraction after FinalizeForPipeline, background SRC, speculative N+1 build, async chain write). It is hard-disabled: isPipelineEligible returns false unconditionally and no config exposes it (the worker/pipeline/enabled gauge is always 0). Pre-Rio seal-recovery interaction makes speculative Prepare fail; re-enablement is future work and the gating logic is preserved in comments. Reviewers can treat the miner path as dormant code with test coverage.

Config

[pipeline]
enable-import-src = false   # master gate (default: disabled)
import-src-logs   = false   # verbose pipeline logging
warm-snapshot     = true    # witness-producing nodes only; leave on

Notable negative results baked into the design (so nobody re-litigates them): the exec-side trie prefetcher and the speculative block prefetcher are load-bearing via shared-cache warming — disabling either regresses 25–75% — so neither is configurable (pipeline.exec-prefetch existed briefly on the feature branch and was removed).

Executed tests

• Parity suites (hash + path schemes, -race): pipelined vs sequential roots/hashes, witness regeneration round trips (241 mainnet fixture blocks, mainnet-root-anchored, inline + pipelined-SRC completion) plus proof-node-set parity suites, FlatDiff mutation/overlay suites, self-destruct integration, pathdb node-index unit tests (fork refcounting, deletion markers, definitive-miss semantics), tests/bor integration.
• Mainnet catch-up benchmark campaign (2026-07-10 → 2026-07-21): 12 instrumented legs covering the pipeline × witness matrix with per-leg lane analysis (exec / SRC / commit / overlap) and CPU+mutex profiling.
• RPC correctness under pipelined import: a deterministic window test (core/pipelined_window_test.go, SRC goroutine held open via test hook) pins the overlay/trie semantics of the "head advanced, root not yet committed" interval; an end-to-end battery (tests/bor/pipelined_rpc_test.go) drives 12 read methods against a live-syncing importer at pinned heights and latest, checks sync-time vs settled response identity, and verifies full per-height parity vs a non-pipelined BP including cryptographic account-proof verification. Handlers that need committed trie nodes (eth_getProof, debug_storageRangeAt) gate on WaitForPipelinedStateCommit instead of erroring transiently inside the window.
• Diffguard mutation testing (50% sampling): 89.9% overall kill rate, 93.6% on tier-1 logic.
• Pending before ready-for-review: 24h tip-following soak, kurtosis devnet e2e with reorgs, witness-sync devnet check.

Quality gates — declared deviations

One CI gate fails for structural reasons; it is intentional and listed here rather than suppressed:

• Cognitive complexity / function size (diffguard, Quality metrics job): insertChainWithWitnesses, IntermediateRoot, getStateObject, and updateTrie exceed thresholds. All four are legacy hot-path giants that this PR extends in place; refactoring them is high-risk consensus-path churn that belongs in a dedicated cleanup, not a feature PR. New-code violations were extracted below thresholds (applyFlatMutationFast, runSRCCompute, FinalizeForPipeline, witness-collection loops deduped into addObjectWitness). nodeWalk is a rename of pre-existing upstream logic. The reported core -> core dependency cycle is a tool artifact (self-cycle).
  codecov/patch and SonarCloud were listed here previously and are no longer red — patch coverage now passes after the added test coverage, and SonarCloud's duplication gate passes after the production-code dedup (addObjectWitness, commitSprintWork reuse). Quality metrics (diffguard) is the only remaining red check.

Rollout notes

Backwards-compatible; off by default; no coordinated upgrade required. Not consensus-affecting: the pipeline computes the identical root on the identical commit path, one block later in wall-clock, and aborts on mismatch. Operator-facing changes when enabled: witness-ready latency trails the block by one SRC cycle (~210ms end-to-end vs ~143ms inline). Related: #2312 implements an alternative trie-node index on develop; this PR lands first and #2312 rebases.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

_{Tip: disable this comment in your organization's Code Review settings.}

[claude]

Code Review

Found 6 issues: 4 bugs and 2 security concerns.

Bugs

1. miner/worker.go:1117 — writeElapsed always measures ~0 (broken metric)
   writeElapsed is computed immediately after writeStart, before either write call executes. The original code had the write call between writeStart and writeElapsed. The writeBlockAndSetHeadTimer metric will always report approximately zero. Fix: move writeElapsed := time.Since(writeStart) to after the if/else block.
   

   bor/miner/worker.go

   Lines 1116 to 1123 in 07345ad

   	writeStart := time.Now()
   	writeElapsed := time.Since(writeStart)
   	if task.pipelined {
   	_, err = w.chain.WriteBlockAndSetHeadPipelined(block, receipts, logs, task.state, true, task.witnessBytes)
   	} else {
   	_, err = w.chain.WriteBlockAndSetHead(block, receipts, logs, task.state, true)
   	}
   	writeBlockAndSetHeadTimer.Update(writeElapsed)

2. miner/pipeline.go:380-383 — nil pointer dereference when chainHead == nil
   When chainHead is nil, the || short-circuits to true and enters the if-body, where chainHead.Number.Uint64() panics. This is in the block production path. Per security-common.md: No panics in consensus, sync, or block production paths. Fix: split the nil check from the number check into separate if-blocks.
   

   bor/miner/pipeline.go

   Lines 379 to 384 in 07345ad

   	chainHead := w.chain.CurrentBlock()
   	if chainHead == nil || chainHead.Number.Uint64() != blockNNum {
   	log.Error("Pipelined SRC: chain head mismatch after waiting", "expected", blockNNum,
   	"got", chainHead.Number.Uint64())
   	return
   	}

3. core/stateless/witness.go:101 — NewWitness no longer copies the context header (mutation risk)
   The old code did ctx := types.CopyHeader(context) and zeroed Root/ReceiptHash. The new code stores the caller pointer directly. In miner/worker.go:1196, the raw header pointer is passed — this header is later mutated in place. The Witness will silently see those mutations. See state-security.md threat model.
   

   bor/core/stateless/witness.go

   Lines 88 to 107 in 07345ad

   	func NewWitness(context *types.Header, chain HeaderReader) (*Witness, error) {
   	// When building witnesses, retrieve the parent header, which will *always*
   	// be included to act as a trustless pre-root hash container
   	var headers []*types.Header
   	if chain != nil {
   	parent := chain.GetHeader(context.ParentHash, context.Number.Uint64()-1)
   	if parent == nil {
   	return nil, errors.New("failed to retrieve parent header")
   	}
   	headers = append(headers, parent)
   	}
   	// Create the witness with a reconstructed gutted out block
   	return &Witness{
   	context: context,
   	Headers: headers,
   	Codes: make(map[string]struct{}),
   	State: make(map[string]struct{}),
   	chain: chain,
   	}, nil
   	}

4. miner/pipeline.go:124 — SetLastFlatDiff stores a provisional header hash that never matches
   env.header.Hash() lacks both Root and the seal signature. In PostExecutionStateAt, the comparison uses the sealed header — so FlatDiff overlay path is never taken. The txpool falls back to StateAt(header.Root) which may fail if SRC hasn't committed. Same issue at lines 521 and 783.
   

   bor/miner/pipeline.go

   Lines 123 to 125 in 07345ad

   	w.chain.SetLastFlatDiff(flatDiff, env.header.Hash())
   	// Note: this counts block N as "entering the pipeline." If Prepare() fails

Security Concerns

5. core/stateless/witness.go:56 — pre-state root validation anchored to untrusted witness data
   The old ValidateWitnessPreState took a caller-supplied expectedPreStateRoot. The new version fetches the parent using witness.context.ParentHash (from the witness itself). For peer-received witnesses, no call site verifies witness.context.ParentHash == block.ParentHash(). A malicious peer could bypass the pre-state root check. Per state-security.md and security-common.md peer-triggerable escalation.
   

   bor/core/stateless/witness.go

   Lines 49 to 71 in 07345ad

   	// Get the witness context header (the block this witness is for).
   	contextHeader := witness.Header()
   	if contextHeader == nil {
   	return fmt.Errorf("witness context header is nil")
   	}
   	// Get the parent block header from the chain.
   	parentHeader := headerReader.GetHeader(contextHeader.ParentHash, contextHeader.Number.Uint64()-1)
   	if parentHeader == nil {
   	return fmt.Errorf("parent block header not found: parentHash=%x, parentNumber=%d",
   	contextHeader.ParentHash, contextHeader.Number.Uint64()-1)
   	}
   	// Get witness pre-state root (from first header which should be parent).
   	witnessPreStateRoot := witness.Root()
   	// Compare with actual parent block's state root.
   	if witnessPreStateRoot != parentHeader.Root {
   	return fmt.Errorf("witness pre-state root mismatch: witness=%x, parent=%x, blockNumber=%d",
   	witnessPreStateRoot, parentHeader.Root, contextHeader.Number.Uint64())
   	}
   	return nil

6. core/blockchain.go:4402 — SpawnSRCGoroutine uses raw go func() without panic recovery
   The old code used bc.wg.Go(func() { ... }) for lifecycle-safe goroutine management. The new code uses bc.wg.Add(1) + raw go func(). If the goroutine panics, the process crashes without graceful shutdown. Per security-common.md: No panics in block production paths.
   

   bor/core/blockchain.go

   Lines 4399 to 4410 in 07345ad

   	pending.wg.Add(1)
   	bc.wg.Add(1)
   	go func() {
   	defer bc.wg.Done()
   	defer pending.wg.Done()
   	tmpDB, err := state.New(parentRoot, bc.statedb)
   	if err != nil {
   	log.Error("Pipelined SRC: failed to open tmpDB", "parentRoot", parentRoot, "err", err)
   	pending.err = err
   	return

[claude]

Code Review

Found 5 issues in miner/worker.go and miner/pipeline.go. Checked for bugs and CLAUDE.md compliance.

---

1. Bug: writeElapsed always ~0ns (miner/worker.go L1116-L1123)

writeElapsed := time.Since(writeStart) is computed immediately after writeStart := time.Now(), before either WriteBlockAndSetHeadPipelined or WriteBlockAndSetHead executes. writeBlockAndSetHeadTimer always records ~0, and workerMgaspsTimer (line 1148) reports inflated MGas/s. Fix: move writeElapsed := time.Since(writeStart) to after the if/else block.

---

2. Bug: nil pointer dereference (miner/pipeline.go L379-L384)

When chainHead is nil, the || short-circuits into the if-body, but chainHead.Number.Uint64() in log.Error dereferences nil and panics. Per CLAUDE.md: No panics in consensus, sync, or block production paths. Fix: split into two if-checks.

---

3. Bug: unchecked type assertion (miner/pipeline.go L335-L341)

borEngine, _ := w.engine.(*bor.Bor) discards the ok boolean. If w.engine is not *bor.Bor, borEngine is nil and borEngine.AssembleBlock(...) panics. The same assertion at line 96 correctly checks ok. Fix: check ok and return early.

---

4. Bug: goroutine leak on 5 return paths (miner/pipeline.go L293-L345)

initialFillDone channel (line 293) goroutine is not drained on return paths at lines 345, 357, 371, 373, 383. Only WaitForSRC error (line 331) and happy path (line 390) drain it. Fix: defer drain after line 293.

---

5. Bug: trie DB race after SpawnSRCGoroutine (miner/pipeline.go L206-L229)

SpawnSRCGoroutine called at line 213 launches a goroutine doing CommitWithUpdate. If StateAtWithFlatDiff fails (line 219) or GetHeader returns nil (line 228), fallbackToSequential does IntermediateRoot inline on the same parent root concurrently. The comments at lines 206-211 identify this as causing missing trie node / layer stale errors but only guard the Prepare() case. Fix: WaitForSRC() before fallbackToSequential, or move spawn after preconditions.

[codecov]

Codecov Report

❌ Patch coverage is 87.85079% with 355 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.19%. Comparing base (12555f8) to head (12bd14a).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
miner/pipeline.go	89.89%	66 Missing and 14 partials ⚠️
core/state/statedb.go	85.87%	73 Missing and 2 partials ⚠️
miner/worker.go	83.33%	40 Missing and 10 partials ⚠️
tests/bor/helper.go	63.15%	21 Missing and 7 partials ⚠️
core/blockchain_reader.go	89.63%	12 Missing and 5 partials ⚠️
core/state/reader.go	46.42%	14 Missing and 1 partial ⚠️
eth/backend.go	59.37%	13 Missing ⚠️
consensus/bor/bor.go	89.58%	8 Missing and 2 partials ⚠️
core/state/trie_prefetcher.go	86.11%	6 Missing and 4 partials ⚠️
core/state/state_object.go	81.39%	6 Missing and 2 partials ⚠️
... and 11 more

❌ Your patch check has failed because the patch coverage (87.85%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2180      +/-   ##
===========================================
+ Coverage    54.23%   55.19%   +0.96%     
===========================================
  Files          908      912       +4     
  Lines       162242   165877    +3635     
===========================================
+ Hits         87985    91561    +3576     
- Misses       68818    68849      +31     
- Partials      5439     5467      +28     

Files with missing lines	Coverage Δ
core/block_validator.go	58.69% <100.00%> (+9.13%)	⬆️
core/blockchain.go	71.87% <ø> (+8.79%)	⬆️
core/blockchain_insert.go	78.50% <100.00%> (+1.27%)	⬆️
core/evm.go	93.67% <100.00%> (+19.65%)	⬆️
core/state/database.go	57.46% <100.00%> (+4.33%)	⬆️
core/state/parallel_statedb_settle.go	100.00% <100.00%> (ø)
core/state/warm_snapshot.go	100.00% <100.00%> (ø)
core/stateless/encoding.go	63.49% <ø> (ø)
core/stateless/witness.go	47.89% <100.00%> (+9.28%)	⬆️
core/txpool/blobpool/blobpool.go	54.87% <100.00%> (ø)
... and 33 more

... and 27 files with indirect coverage changes

Files with missing lines	Coverage Δ
core/block_validator.go	58.69% <100.00%> (+9.13%)	⬆️
core/blockchain.go	71.87% <ø> (+8.79%)	⬆️
core/blockchain_insert.go	78.50% <100.00%> (+1.27%)	⬆️
core/evm.go	93.67% <100.00%> (+19.65%)	⬆️
core/state/database.go	57.46% <100.00%> (+4.33%)	⬆️
core/state/parallel_statedb_settle.go	100.00% <100.00%> (ø)
core/state/warm_snapshot.go	100.00% <100.00%> (ø)
core/stateless/encoding.go	63.49% <ø> (ø)
core/stateless/witness.go	47.89% <100.00%> (+9.28%)	⬆️
core/txpool/blobpool/blobpool.go	54.87% <100.00%> (ø)
... and 33 more

... and 27 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[lucca30]

Additionally, the 500ms buffer previously reserved after transaction execution for SRC is removed when the pipeline is active. Transactions now get the full block time for inclusion since SRC runs in the background.

I am okay with the idea of removing the remaining 100ms.

We already reduced this buffer from 500ms to 100ms in v2.7.1, and from what we have seen so far, this remaining time looks small enough that removing it seems reasonable.

My main concern is not the removal of the 100ms itself. My concern is the cost of pipelining SRC with the next block production.

In other words: by doing SRC in parallel with block building, how much do we impact SRC time itself?

Do we expect SRC to remain roughly the same, or does it become meaningfully slower because it is now competing with the next block production? That is the part I would like to understand better.

I think this is basically a TPS vs finality question:

• on one side, we gain more block time for transaction inclusion, which is good for TPS
• on the other side, if SRC takes longer to complete, we may delay block completion, which could hurt finality

So I am supportive of the direction, but I think the key question is still:

How much TPS do we gain, and how much finality do we lose, if any, by making SRC fully pipelined with block production?

If the impact on SRC time is only slight, then the tradeoff is probably clearly worth it.

But if SRC time increases materially once it is pipelined with block production, then we should make that tradeoff explicit

[cffls]

Do we expect SRC to remain roughly the same, or does it become meaningfully slower because it is now competing with the next block production? That is the part I would like to understand better.

I think SRC will be roughly the same, because the time consuming part, trie nodes prefetching, is already running at the same time with tx execution today, and this PR doesn't change this behavior.

[claude]

Code Review

Found 3 issues in core/blockchain.go. Checked for bugs and CLAUDE.md compliance.

1. Missing ValidateReorg guard in pipelined import path (HIGH) — The non-pipelined path checks forker.ValidateReorg per-block before writing to prevent a race with newly-whitelisted Heimdall milestones/checkpoints. The pipelined path skips this check, allowing a block that conflicts with a freshly-anchored milestone to be accepted as chain head.

2. writeHeadBlock called without chainmu in auto-collection goroutine (HIGH) — On root mismatch, the recovery goroutine calls writeHeadBlock(parentBlock) which has a documented contract requiring the mu mutex to be held. The goroutine does not acquire it, creating a data race on chain head state.

3. flushPendingImportSRC error silently discarded (HIGH) — The function returns a meaningful error (SRC failure or root mismatch from a previously-committed block) but the error is discarded with _ in the block-error cleanup path. Every other call site handles this error.

[claude]

Inline Review Comments

Since inline comments could not be posted via the review API, here are the detailed findings with line references:

---

Issue 1 (HIGH): Missing ValidateReorg guard in pipelined import path — core/blockchain.go:3457

The non-pipelined path (lines 3618-3631) calls forker.ValidateReorg per-block immediately before writing to DB to prevent a race where a Heimdall milestone/checkpoint is whitelisted during block execution. The pipelined path skips this check entirely, calling writeBlockAndSetHeadPipelined without a preceding ValidateReorg. A block that conflicts with a freshly-anchored Heimdall milestone or checkpoint can be accepted as chain head in pipelined mode. The upfront check at line 3103 runs before execution begins and does not close this window.

Fix: Add the same per-block ValidateReorg guard before writeBlockAndSetHeadPipelined.

CLAUDE.md: blockchain-security.md and consensus-security.md

---

Issue 2 (HIGH): writeHeadBlock called without chainmu in auto-collection goroutine — core/blockchain.go:3507

writeHeadBlock has a documented contract at line 1733: "this function assumes that the mu mutex is held!". The auto-collection goroutine calls it without acquiring chainmu. After insertChainWithWitnesses returns and releases chainmu, this goroutine may still be running. If a root mismatch is detected, writeHeadBlock is called without the mutex — while another goroutine could concurrently acquire chainmu for a new InsertChain call. This is a data race on chain head state.

Fix: Acquire bc.chainmu.Lock() before calling writeHeadBlock in the error recovery path.

CLAUDE.md: security-common.md — "Shared mutable state protected by mutex or atomic operations"

---

Issue 3 (HIGH): flushPendingImportSRC error silently discarded — core/blockchain.go:3407

flushPendingImportSRC() returns a meaningful error (state root mismatch or SRC failure from a previously-committed block). Discarding it with _ means a block with a bad state root could persist undetected. Every other call site handles this error (line 1784, line 3328).

Fix: Replace _ = bc.flushPendingImportSRC() with if err := bc.flushPendingImportSRC(); err != nil { log.Error(...) } consistent with other call sites.

CLAUDE.md: security-common.md — "Error values checked — never discard errors with _ in security-sensitive paths"

[pratikspatil024]

Thanks — this was a genuinely valuable review. I verified all seven findings independently against the source (three separate deep-dives plus a direct read of the mutex semantics for #1), and every one is real. All are fixed in 6463a59, along with the three test gaps and the side observations.

Three places where my dig went further than the write-up, and each one changes the fix — worth reading before the re-review:

#2's obvious fix would have been a new bug

Hoisting the destruct check above the overlay probe is exactly what I was about to do, and it's wrong: ApplyFlatDiff (and applyFlatMutation / resolveFlatMutationObject) deliberately seed parent-block destructs into the same stateObjectsDestruct map, and those accounts' rewritten slots legitimately live in FlatDiff.Storage. Reordering would zero valid parent destruct-and-resurrect state — trading your bug for a mirror-image one.

The fix tracks this block's own destructs in a separate currentBlockDestructs set, populated only from the three execution-path writers (Finalise, finaliseDelete, SetStorage) and never from the FlatDiff replay paths. GetCommittedState consults that set before the overlay probe; the existing stateObjectsDestruct check stays where it is for the non-overlay path.

Also confirmed your EIP-6780 caveat holds structurally, not just probabilistically: a SelfDestruct6780 requires newContract (same-tx creation), while a FlatDiff.Storage entry requires an account that was alive with code at the end of the parent block — CREATE onto such an address fails, so the two preconditions are mutually exclusive. Post-Napoli tip is genuinely shielded; pre-Napoli replay (~54.8M mainnet blocks, i.e. precisely what this feature is for) is not.

#6 fails in both directions, not one

You flagged SRC(N) starving on the swapped backend. The main thread's own execution of N+1 breaks too: in FlatDiff mode it executes against CommittedParentRoot = root_{N-1}, but the witness memdb only carries nodes rooted at root_N, and DisableSnapInReader removes the snapshot fallback. So the combination is broken regardless of what the SRC goroutine is doing.

Fixed by computing witnessFed before pipelineActive and excluding it, plus a flushPendingImportSRC before the backend swap so a cross-call in-flight SRC is collected first. I went with the per-block fallback rather than rejecting the flag combination at startup — degrades gracefully instead of refusing to boot.

#5: one sub-claim doesn't hold

HasState / pendingImportSRC is not a lasting leak — hasRecentPipelinedState returns false once collectedCh closes and is grace-bounded, so it's only true in the brief pre-close window. The pending entry does linger, but that's #3's wedge rather than a state-advertisement bug. Everything else in #5 confirmed: canonical hash and tx lookups never deleted, events already emitted with no corrective event, txpool left reset at the bad root, divergent trie root committed and never dereferenced, lastFlatDiff still serving the rejected block.

---

What changed

#1 / #3 / #5 — one rewrite, since they share a cause. The collector no longer touches chainmu at all; you're right that TryLock is a blocking receive that only reports failure once the mutex is closed. Recovery moved to the collecting thread (which already holds chainmu) as recoverFailedPipelinedImport, and now runs for SRC errors too, not just mismatches. It does the complete rollback: clears the pending entry, drops the FlatDiff overlay, deletes the rejected block's canonical hash and tx lookups, moves the head back, dereferences the divergent root in hash mode, and emits a corrective ChainHeadEvent so the txpool re-resets off committed state.

One extra trigger for #3 you didn't list: stateless.NewWitness failure is only logged, leaving computeWitness=true with a nil witness, which then hits the makeWitness && execWitness == nil hard-fail.

#4 — clamped at the source (adjustBack && idx > 0) and hardened the downloader guard to index >= 0 && index < len(results).

#7 — treated as the most urgent since it ships regardless of the flag. Two changes: a lookup-index rejection is authoritative again (accountTip fails precisely when the reader's state is neither the disk layer nor a descendant of it, so it's structural, not a race — no fallback), and the remaining fallback for the genuine located-layer-went-stale race now goes through bottomIfAncestorOf, serving the base only when it is an ancestor of or equal to the reader's state. Both of your cases now return errSnapshotStale instead of another state's values.

Side items — a locally unavailable parent state no longer writes the block to the bad-block DB (isLocalStateUnavailable matches trie.MissingNodeError and the "is not available" shape). You were right that the peer drop itself is pre-existing — the downloader wraps any insert error — so the PR's actual delta was the bad-block record and log. TestV2GasDeterminism now distinguishes "no candidates" (legitimate skip) from "all candidates failed base reads" (t.Fatalf).

Tests, and I checked they actually fail

Your first suggested test was the highest-value one — it caught #1, #4 and #5 exactly as you predicted.

• TestPipelinedImportSRC_RootMismatchRollback — two-block batch, first block's root mutated, both schemes. I temporarily restored the old collector shape to confirm it has teeth: the suite hung and had to be killed, and couldn't even reach shutdown, matching your note that Stop() hangs.
• TestFlatDiffOverlay_DestructByExecutingBlock — destruct performed by the executing block over an overlay. Without the guard it returns 0xbeef where a non-pipelined node returns zero.
• TestPipelinedImportSRC_PipelineActuallyRuns — the positive control you asked for: the pipelined block counter must advance once per block under the pipelined config and stay at zero under the default. The suite can no longer pass with the pipeline silently disabled.

Two of your new coverage tests needed updating

Heads-up since they're yours (11636e8) — both encoded the old behavior and correctly failed against the fixes:

• TestPendingImportCollectionHelpers expected flushPendingImportSRC to still find the pending entry after a failed collect. Clearing it is the anti-wedge fix, so the test now asserts the entry is gone and a follow-up flush is a no-op; I kept flush-surfaces-the-error covered with a fresh pending.
• TestReaderStaleFallbackHelpers and TestReaderPublicFallbackBranches/stale lookup... expected the unguarded fallback. They now assert errSnapshotStale, which is the point of the ancestry check.

Claim corrections and CI

Both accepted. The description's "flushed on reorg, gap, or error" and the mismatch-recovery claim were wrong as written; I'll rewrite those and refresh the deviations section — you're right that it's stale, only Quality metrics (diffguard) is red now, codecov/patch passes and SonarCloud isn't reported as a check.

Full sweep green on 6463a59: core, core/state, triedb/pathdb, eth, eth/downloader, consensus/bor, miner, legacypool, the pipelined integration tests, and -race on the pipelined/FlatDiff suites.

Given the size of what this surfaced in the failure paths, I don't think the earlier "ready modulo soak" framing holds — I'd like your re-review of the recovery path specifically before this goes near the soak.

[Copilot]

Pull request overview

Copilot reviewed 82 out of 83 changed files in this pull request and generated no new comments.

Suppressed comments (2)

core/stateless/witness.go:76

• ValidateWitnessPreState computes contextHeader.Number.Uint64()-1 without guarding against Number == 0 (genesis) or Number == nil. For a genesis-context witness this underflows to MaxUint64 and produces a misleading lookup/error. Add an explicit check and return a clear error before subtracting.

This issue also appears on line 115 of the same file.
core/stateless/witness.go:123

• NewWitness now calls types.CopyHeader(context) (which dereferences context) without guarding against context == nil. This is a behavior change: previously a nil context could slip through and be handled later, but now it will panic immediately. Also, context.Number.Uint64()-1 underflows for genesis headers. Add explicit nil/genesis checks before accessing context/subtracting.

[cffls]

Follow-up: V2 witness completeness — this branch predates #2333

Separate topic from the failure-path findings, and it only affects the witness-producing configuration. #2333 ("produce complete witnesses under BlockSTM v2") merged into develop yesterday as 12555f8f4. I checked, and it is not an ancestor of 6463a592e — so this branch still carries the pre-fix witness collection.

To be clear about what is and isn't already handled here, because there are two layers to it:

Already covered on this branch. CollectStateWitness() exists (core/state/statedb.go:372, called from core/parallel_state_processor.go:1175) and pulls the read-side witness directly off the shared reader. Its comment already spells out the SafeBase mechanism — V2 worker reads go through pool copies that share statedb's reader by reference, so finalDB.IntermediateRoot's object walk missing read-only addresses is compensated for. #2333 only moves that call (to after engine.Finalize, whose state-sync reads at sprint boundaries are also part of the witness contract). So "worker-only reads are invisible to the witness" is not the gap.

Not covered on this branch. CollectStateWitness harvests trie tracers, and in production most reads never touch a trie: the flat reader (snapshot in hash mode, pathdb state reader in path mode) satisfies hot keys directly, and the shared reader cache — warmed by the import prefetcher — serves subsequent reads with no trie resolution at all. The tracers are therefore empty for precisely the hottest accounts. #2333 fixes this with a per-key walked flag plus a prewalker that re-resolves unwalked cached keys through the trie reader while workers execute. Grepping this branch: StartWitnessReadSetPrewalk, resolveCachedKeysIntoTrie and the walked flag have zero occurrences.

The reason I'm raising it against this PR rather than just noting the merge: the witness validation here structurally cannot see this failure.

• The mainnet campaign ran 2026-07-10 → 07-21 with pipeline on + producewitnesses under BlockSTM enforce — well before core, core/state: produce complete witnesses under BlockSTM v2 #2333, so those legs measured the broken path.
• The parity suites generate fresh chains with cold state, where every read falls through to the trie and gets traced. That is exactly the condition under which the bug is invisible, which is why core, core/state: produce complete witnesses under BlockSTM v2 #2333 needed mainnet A/B scans to surface it (162/162 witnesses failed stateless replay under V2 enforce vs 197/197 passing serial).

So "witness proof-node-set parity + stateless replay across all configurations" doesn't currently establish V2 witness completeness for the pipelined path. Worth softening that line in the description until it's re-established on a merged branch.

Merge overlap

The two changes collide in the witness path specifically:

file	#2333	this PR
core/state/statedb.go	+119	+786
core/state/reader.go	+15	+41
core/parallel_state_processor.go	+29	±26

Two things I'd check deliberately rather than trust the merge resolution on:

1. wireStorageCaches removal. Dropping SafeBase's SharedStorageCache and OverlayStorageCache should help core, core/state: produce complete witnesses under BlockSTM v2 #2333's coverage rather than hurt it — core, core/state: produce complete witnesses under BlockSTM v2 #2333 sweeps readerWithCache at the reader level, so routing more worker storage reads through the reader puts more keys into the swept set. But it shifts which layer serves reads, so core, core/state: produce complete witnesses under BlockSTM v2 #2333's measured "~1,600 keys/block prewalked, ~0.008 left for the settle drain" won't transfer. The prewalker volume and settle-drain tail need re-measuring here before the "off the block's critical path" property can be assumed.

2. SRC must extend the exec-side witness, not regenerate it. Ordering is fine: Process (now calling CollectStateWitness after engine.Finalize) returns before persistPipelinedImport captures statedb.Witness(). But SRC then completes the witness on its own tmpDB with NewTrieOnly, driven by the FlatDiff ReadSet/NonExistentReads that CommitSnapshot builds from the executing statedb's objects. Keys that only ever lived in the reader cache — exactly the ones the prewalker rescues — won't appear in that ReadSet. So the recovered nodes survive only if SRC appends to the handed-in execWitness rather than rebuilding from the read set. Worth an explicit assertion post-merge.

Cheapest way to settle it

#2333 already ships the oracle: TestV2WitnessRegenerationAllBlocks round-trips 241 mainnet fixture blocks and asserts regenerated witnesses replay with identical gas, receipt and state roots (-short-skipped, ~2.5 min). After merging develop, pointing that suite at the pipelined path — pipeline on, witnesses on — would settle this empirically instead of by inspection, and would cover the SRC-extends-vs-rebuilds question in (2) at the same time.

One sequencing note: per #2333's rollout notes, v2.9.2 ships a guard forcing the serial processor for witness-recording blocks. I don't see that guard in develop (it's on the release line), so it shouldn't affect this merge — but it does mean "pipeline + witnesses under V2" is currently disabled in the released client, which is worth knowing when planning the soak configuration.

[pratikspatil024]

@cffls Thanks — this follow-up was extremely productive. All three parts are now addressed on the branch (5e1712b4c merge + dbcf6daa6), and chasing the last failing fixture block turned up two real bugs, one of them in your domain. Details below.

#2333 merged, with one deliberate collision resolution

develop (through 12555f8f4) is merged. The prewalker, resolveCachedKeysIntoTrie, and the per-key walked flag are all active on this branch, and TestV2WitnessRegenerationPipelinedPrewalkFires guards that the prewalker actually fires on our read routing (3,206 keys on the fixture block — so the round trip genuinely exercises the cache-served blind spot, not a vacuous pass).

The collision: this branch tracks base-read failures in V2 workers (develop's SafeBase doesn't observe them at all), and the merge initially promoted every pooled-copy failure into one block-global error. That over-triggers under witness-backed replay — a speculative incarnation chasing stale values may legitimately read outside the canonical state and then get invalidated. Resolution: SafeBase read methods return errors, each ParallelStateDB records its own BaseReadErr (cleared on Reset), and only an error observed by the incarnation that settles aborts the block.

Your point 2 (SRC must extend, not rebuild) — settled empirically

witnessRegenPipelinedRoundTrip runs your oracle through the pipelined split: V2 exec (prewalker included) → CommitSnapshot extracts the FlatDiff exactly as persistPipelinedImport does → SRC completion on a trie-only statedb carrying the same witness object, mirroring runSRCCompute. Keys the prewalker rescues only ever lived in the reader cache and are absent from the FlatDiff read set, so this fails if SRC rebuilds instead of extends. TestV2WitnessRegenerationPipelinedSRCAllBlocks: 241/241 with stateless replay to identical gas/receipt/state roots, and the SRC-computed root checked against the reference.

Bug 1 (ours): the settled-incarnation check was reading recycled pdbs

One fixture block (83014065) kept failing both AllBlocks oracles with a missing trie node under WMATIC's storage. Root cause was not witness completeness: a speculative incarnation of tx 265 read stale pool reserves from base, took a phantom branch, touched a WMATIC balanceOf slot canonical execution never reads (hence legitimately absent from the witness), and was then correctly invalidated and re-executed by validation — same behavior as develop.

The abort came from our gate: it scanned the executor's states array after execution, but settlement recycles each pdb into the pool where a later tx's Reset reuses it — stale states[i] slots alias the mutated object. Instrumentation showed ten different indices all pointing at tx 265's invalidated incarnation. The check now runs inside the settle callback, the only point where a pdb is provably the settled incarnation (regression test: TestV2SettleFn_RecordsBaseReadErr).

Bug 2 (yours, I think): witnesses are missing EIP-7702 authority pre-state code

With the gate fixed, three other fixture blocks (83014074, 83014100, 83020871) started failing deterministically with code is not found <hash>, from validateAuthorization (state_transition.go:702). The read is canonical: validating a set-code authorization reads the authority's current code to ParseDelegation it. The witness/code archive for these blocks doesn't contain those pre-state blobs — the fixtures were captured by a client whose V2 base reads silently nil-serve the miss, so the blob was never observed.

The uncomfortable part: serial stateless replay of these witnesses anchors to the real mainnet roots anyway, purely by convergent divergence — the authority's real code is a delegation designator (authorization accepted) and a missing blob reads as empty code (also accepted). The two paths happen to produce identical receipts and roots. The day an authority holds real contract code, that convergence breaks: a full node rejects the authorization (ErrAuthorizationDestinationHasCode) while stateless replay of an incomplete witness accepts and applies it → state-root divergence in stateless verification. Since witnesses cross node versions in a mixed fleet, this seems worth tracking as a witness-completeness gap independent of this PR — happy to file it separately if you agree it's real.

For this PR I've pinned the three blocks as knownIncompleteWitnessFixtures (exact missing hash asserted; any other failure on them still fails the sweep; entries drop when fixtures are regenerated by a client that records these reads). Whether a given run trips the gap is scheduling-dependent — the read only reaches base when the speculative incarnation runs early enough and still validates — so a pass is tolerated, a different error is not.

Oracle hardening that fell out of this

Both round trips now anchor the stage-1 replay against each block's real roots from the fixture header. Previously the round trip was self-consistent only — a fixture whose replay silently diverged from mainnet (zero-ish reads swallowed symmetrically on both sides) would pass against itself, which is exactly how the 7702 gap stayed invisible. With the anchor: serial 241/241, pipelined 241/241.

On your point 1 (wireStorageCaches / prewalk volume)

Agreed the ~1,600 keys/block and settle-drain tail numbers don't transfer to this branch's routing. The prewalk-fires test proves the mechanism is live here; for the volumes we'll watch chain/witness/readset/prewalk/keys and the settle-drain during the 24h tip soak (witness-producing config) and report the numbers on this thread rather than assume yours.

I'll also soften the PR description's "witness proof-node-set parity + stateless replay across all configurations" claim as you suggested, replacing it with what's now actually established: mainnet-anchored replay on 241 fixture blocks through both the inline and pipelined-SRC paths on the merged branch, with the 7702 fixture caveat noted.

[Copilot]

Pull request overview

Copilot reviewed 87 out of 88 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 87 out of 88 changed files in this pull request and generated no new comments.

Suppressed comments (1)

core/v2_blockstm_test.go:275

• The test filters incomplete embedded witness fixtures by string-matching err.Error() for "v2: base read: missing trie node". This is brittle (error text/wrapping can change) and can misclassify failures. Prefer a typed check (e.g., errors.As(err, *trie.MissingNodeError) against the wrapped cause) so the skip-vs-fail decision is stable across error formatting changes.

[cffls]

One remaining gap from the re-review: rollback is still collect-path-only

Re-checked on 1bec1712e. The recovery rewrite is sound and I'm satisfied on the deadlock, the wedge and the durable-marker cleanup — but recoverFailedPipelinedImport is still reachable from exactly one place, collectPendingImportSRC (core/blockchain.go:5506). flushPendingImportSRC (:5467) clears the pending entry and returns collectedErr without any rollback, and none of its four call sites compensate:

call site	holds chainmu	what happens on a failed SRC
:5977 buildPipelineImportOpts (reorg / gap)	yes	logs "flush failed on mismatch", then builds fresh opts and keeps importing
:3812 insertChainWithWitnesses (ProcessBlock error)	yes	logs only; outer error is returned
:3720 insertChainWithWitnesses (witnessFed)	yes	returns the error, aborts the insert
:2274 stopWithoutSaving	no	logs only

So on those paths the rejected block keeps its canonical hash and tx-lookup entries, lastFlatDiff may keep serving its post-state, the divergent root is never dereferenced, and no corrective event fires. The :5977 one is the one I'd actually fix: a reorg or gap arriving while an SRC is in flight is both the least-tested path in the suite and the only one that continues importing after discovering the failure.

Partly self-healing — a genuine reorg rewrites canonical hashes through the normal reorg machinery, and a restart rewinds a head whose state is missing — but "self-heals on restart" isn't the property you want backing the mechanism that exists to contain an unverified head.

Suggested shape: call recoverFailedPipelinedImport from the three under-chainmu sites (or hoist the call into flushPendingImportSRC behind a flag the shutdown path passes as false, since stopWithoutSaving can't hold chainmu and startup rewind covers it anyway). The function is already idempotent with respect to the pending entry, so the collect path is unaffected.

Three smaller items from the same pass, all optional:

• idx > 0 clamp (:3835) now misattributes rather than panics. Both index consumers are negative-safe today — the downloader's new index >= 0 && index < len(results) guard and cmd/utils/cmd.go:252's pre-existing failindex > 0. With the clamp, idx = 0 routes the downloader into reportBadBlock(blocks[0]), blaming the batch's first block for a failure that belonged to a previous batch's block. Returning -1 lands in the else branch, which exists precisely for out-of-band indices and only logs.
• No RemovedLogsEvent on rollback. rmLogsFeed is used by the reorg path (:4529); recovery sends only ChainHeadEvent, so eth_subscribe("logs") clients and the filter indexer keep the rejected block's logs with Removed: false.
• Nothing pins the mirror-image invariant currentBlockDestructs protects — that a parent-block destruct seeded by the replay paths still serves overlay storage. That's the regression a future "simplify these two sets into one" would reintroduce silently, and it's the reason the separate set exists.

---

Follow-up suggestion: enable delayed SRC in the kurtosis e2e legs

Not for this PR necessarily, but worth tracking as a follow-up item: because pipeline.enable-import-src defaults to false, every kurtosis leg in CI currently exercises only the OFF path. Nothing in CI runs the pipeline in a real multi-node network — and the one suite that does drive it end to end (tests/bor) is the flaky-peering one that's red on this run.

That's an awkward coverage shape for this feature specifically, because the interactions I'd least trust to unit tests are exactly the ones kurtosis covers well:

• reorgs arriving while an SRC is in flight (the R1 path above, and the gap my first review flagged as untested)
• real peering and sync behaviour while hasPendingPipelinedHeadState holds the node in full-sync
• witness production and stateless sync against a pipelined producer — kurtosis-stateless-e2e.yml is the more valuable of the two legs here, since pipeline + witness is the riskiest combination and now also the one carrying the freshly-merged core, core/state: produce complete witnesses under BlockSTM v2 #2333 prewalker
• restart across the "head advanced, root not yet committed" window

Wiring-wise the args file is external — .github/workflows/kurtosis-e2e.yml:101 copies configs/kurtosis-e2e.yml from 0xPolygon/pos-workflows and passes it to kurtosis run --args-file=. So either add the flag to the pos-workflows config or inject an override step after the copy. I'd suggest a separate pipeline-enabled leg rather than flipping the existing one, so the default-off path keeps its coverage and a failure tells you immediately which mode broke.

Doing this before the 24h soak would also mean the soak isn't the first time the pipeline meets a multi-node network.

[pratikspatil024]

@cffls All four items from the re-review are addressed in 7c6fb6b85 (kept as a standalone commit — the kurtosis work discussed in the follow-up comment below is deliberately separate).

R1 — rollback on every chainmu flush path

Took your suggested shape: flushPendingImportSRC(rollback bool). The three callers that hold chainmu pass true — including the reorg/gap site at buildPipelineImportOpts, which you correctly singled out as the one that kept importing after discovering the failure; shutdown passes false (it can't take chainmu, and the startup rewind covers it). Your note that recoverFailedPipelinedImport is idempotent on the pending entry made the hoist clean — the collect path is unaffected.

Pinned by TestFlushPendingImportSRCRollsBack: head moves back to the parent, the rejected block loses its canonical hash, and the FlatDiff overlay stops being served. Verified to fail without the fix (head stays on the rejected block).

The three smaller items — all taken, each verified first

• idx clamp → -1. Confirmed your reachability analysis on both consumers: the downloader's else branch exists precisely for out-of-band indices and only logs, and cmd/utils/cmd.go falls back to a generic message. The clamp really did route a previous batch's failure into reportBadBlock(blocks[0]).
• RemovedLogsEvent on rollback. Recovery now collects the rejected block's logs (removed=true, before its indexes are deleted) and emits on rmLogsFeed after the head rewrite — same retraction shape as the reorg path at :4529.
• The mirror-image invariant test. TestFlatDiffOverlay_ParentDestructKeepsOverlayStorage: a parent-block destruct seeded by the FlatDiff replay paths must not shadow overlay storage written after the resurrection. It documents in the test name exactly the regression a future "merge the two destruct sets" would reintroduce.

One related hardening that grew out of exercising R1 on a live network (commit adaa2f032, details in the kurtosis comment): a batch can publish descendants of a failed block before its SRC verdict lands, so recoverFailedPipelinedImport now clears canonical hashes and tx lookups for the published descendants too, not just the rejected block.

On the kurtosis follow-up

Agreed, and we went one step further: we ran both e2e legs locally with pipeline.enable-import-src=true injected into the bor config template — and it immediately caught a real consensus bug (producer-side, not import-side). Full report in the separate comment below; short version is that your instinct about "the interactions I'd least trust to unit tests" was exactly right. We'll propose the separate pipeline-enabled CI leg (override step after the pos-workflows config copy, so it stays self-contained in this repo).

[pratikspatil024]

Local kurtosis run of both e2e legs with the pipeline enabled — found and fixed a producer-side consensus bug

Following up on the CI-coverage discussion: we replicated both kurtosis e2e legs locally (kurtosis-pos v1.3.4, bor:local from this branch, heimdall-v2:local from develop, the pos-workflows args topologies) with one deliberate delta CI doesn't have — [pipeline] enable-import-src = true injected into every bor node's config template, validators included.

The bug (fixed in adaa2f032)

Pipeline-enabled validators deterministically split the network at the first sprint boundary (block 16; 3/3 runs; control run with pipeline off on identical images: clean lockstep, zero bad blocks).

Root cause, pinned by elimination: both competing block-16 candidates shared the identical parent block; state-sync fetches returned zero events everywhere; the span-1 commit was byte-identical on every execution — all consensus inputs deterministic. The two header roots differed by exactly the parent block's state write (the EIP-2935 block-hash slot, the only per-block state change on an idle devnet).

Mechanism: commitWork opened sealing state via StateAtWithReaders(parent.Root), which serves the FlatDiff overlay whenever the parent is the latest pipelined import — and that interception matched on root alone, so it wasn't even scoped to the uncommitted window. An overlay statedb is rooted at the grandparent with the parent's writes installed as unjournaled read-only objects, so the root sealed into the header omits every parent write the new block doesn't rewrite. Every importer — the pipelined SRC and the stock 2.8.0 baseline node byte-identically — recomputed the correct root and rejected the block. The importers were right; the producer was wrong.

Why nothing caught it before: the mainnet soak node never mines, and no unit suite exercises miner + pipelined-import interplay. It needed a real multi-node network — precisely the coverage gap flagged in review.

Fix: SealingStateAt / SealingStateAtWithReaders — block production waits (bounded) for the parent's pending SRC to commit and opens the committed trie, never the overlay; commitWork re-checks the head after the wait since a failed SRC rolls it back. The overlay-serving accessors remain for RPC readers and pending-block serving, where no header root is derived from them. Pinned by TestSealingStateNeverServesOverlay.

Fallout also observed and fixed: with verification trailing the insert, a batch can publish descendants of a failed block before the first SRC verdict lands, and the single-block rollback left their canonical hashes resolving above the head — recovery now clears the published descendants' markers too.

Note on scope: the feature has no hardfork gate — the flag alone activates it, and post-Rio span rotation still has producers sealing on pipelined-imported heads, so this fix is required regardless of any Rio-scoping decision.

Verification on the fixed build

Leg 1 (e2e topology: 4 validators + rpc + stock-2.8.0 baseline, 1s blocks, Rio at 128, pipeline on everywhere):

• 4,003 blocks, all six nodes in lockstep, pre- and post-Rio
• chain_imports_pipelined_src_count ≈ 2,950–3,070 per validator (pipeline active on essentially every imported block); root_mismatch = 0 fleet-wide
• pos-workflows smoke suite: 96/96 opcode/precompile txs, milestones ✅, plasma bridge (POL + ERC20 + ERC721) ✅, checkpoints ✅
• pos-workflows RPC suite: 46/46

Leg 2 (stateless topology: 3 witness-producing validators + 2 stateless-sync validators + bor RPC matrix + erigon, pipeline on for all full-sync bor nodes):

• 375+ blocks, all nine nodes in lockstep (heads within 5 blocks under load), pre- and post-Rio, erigon included
• root_mismatch = 0 on every full-sync node; witness-producing validators ran 220–316 pipelined SRCs each
• The witness-producing RPC node shows chain_witness_size_bytes_count == chain_imports_pipelined_src_count — every witness on this leg was produced by the pipelined SRC completion path (the pipeline + witness combination this branch changes most, now carrying the merged core, core/state: produce complete witnesses under BlockSTM v2 #2333 prewalker)
• Both stateless-sync validators stayed at head throughout — i.e. witnesses produced by pipelined nodes were consumed successfully by stateless syncers, covering the mixed-fleet production/consumption direction end to end
• The stateless-sync nodes correctly report src_count = 0: the pipeline gates itself off under stateless sync
• 200-transaction polycli load burst through a witness-producing node: 0 errors, lockstep maintained

Environment notes for anyone rerunning this on a Mac: the smoke suite needs bash 4+ (declare -A) and coreutils timeout (absent on macOS — three runs "failed" with all 96 transactions reported unmined while they had actually mined within seconds; a two-line shim fixes it). Happy to contribute the portability fixes to pos-workflows.

CI follow-up

This is strong evidence for the separate pipeline-enabled kurtosis leg suggested in review — it would have caught this bug on the first push. Wiring-wise we'd inject the [pipeline] block via an override step after the pos-workflows config copy, keeping the leg self-contained in this repo and the default-off leg's coverage intact.

[Copilot]

Pull request overview

Copilot reviewed 87 out of 88 changed files in this pull request and generated no new comments.

Suppressed comments (2)

core/state/v2_executor_differential_test.go:93

• BaseNonce ignores the error returned by SafeBase.GetNonce. If a base-state read fails (e.g., missing trie node in a fixture), this helper will silently treat the nonce as 0 and the differential harness may proceed with incorrect inputs, masking regressions.

Since this is test-only code, it’s better to fail fast on any SafeBase read error (panic is fine here) so the harness can’t pass on zero-ish substituted values.
core/stateless/witness_test.go:223

• This assertion slices err.Error() without checking its length. If ValidateWitnessPreState returns a different/shorter error string, the test will panic with an out-of-range slice instead of failing with a helpful message.

Guard the length before slicing so the test fails cleanly.

[cffls]

Pipelined SRC produces witnesses that fail stateless replay (live dev-node A/B on adaa2f032)

Ran a witness-validity check on a dev node syncing mainnet at the current branch head. Result: with enable-import-src = true, essentially every block's witness fails stateless replay with missing state trie nodes; with the flag off, every block passes. Root cause looks like the deferred-commit design interacting with how the FlatDiff read-set is captured — details and a reproduction path below.

Node config (otherwise stock):

syncmode = "full"     # path scheme, snapshot = true, triesinmemory = 1
[parallelevm]  enable = true, procs = 8, enforce = true
[pipeline]     enable-import-src = true
[witness]      enable = true, producewitnesses = true, witnessapi = true

First, a harness correction that matters for reading these numbers

The replay tool I used had its own bug, and I fixed it before trusting any of this. It hand-rolled the replay as an ApplyTransaction loop, which skips the pre-block system calls — on bor mainnet PragueBlock = 73440256, so every block since then runs ProcessParentBlockHash (EIP-2935 parent-hash history). That mutates state but consumes no block gas, which produced matching gas, no missing nodes, wrong post-state root on every post-Prague block regardless of configuration.

The fix was to reuse this PR's own regen-test approach — drive the real core.NewStateProcessor(chainCtx).Process(...) with a minimal stub engine, exactly as executeStatelessSerial does. After that the tool reports full stateless equivalence (gas + receipt root + state root) on a known-good configuration, which is what makes the comparison below meaningful. Anything previously concluded from that tool's root check should be treated as unreliable; its gas-based results are unaffected.

The A/B

Same binary, same tool, both sides measured from a fresh restart so reader-cache warmth is comparable:

configuration	blocks	pass	failing	missing trie nodes
enable-import-src = false, V2 enforce	100	100	0	none (dbErr: <nil>)
enable-import-src = true, V2 enforce (cold)	99 (+1 state-sync skip)	0	99	every block
enable-import-src = true, V2 enforce (warm, 80 min uptime)	150	0	150	every block
enable-import-src = true, V2 off (serial)	81	12	69	most blocks

Failures are both account-trie (getStateObject (addr) error: missing trie node ... (path 0b0102)) and storage-trie (missing trie node ... (owner 36ecd3bd...) (path 09)). The ~50% gas shortfall in the raw output is downstream: once a read fails, the statedb returns zero-ish values and execution diverges.

Witness density rules out "collection dropped nodes" — pipelined witnesses are larger, not smaller:

configuration	nodes/Mgas	KB/Mgas
pipeline off + V2	249.9	93.1
pipeline on + V2	279.1	104.7
pipeline on + serial	257.9	96.8

More nodes, wrong nodes — the signature of a generation mismatch rather than a coverage shortfall.

Diagnosis

The root offset. Importing block N, execution runs against the last committed root — root_{N-2} — plus the FlatDiff overlay of N-1, because SRC(N-1) hasn't been collected yet. The overlay makes the values correct, but trie nodes fetched from the reader are of the root_{N-2} generation. The published witness for block N must contain nodes rooted at root_{N-1}. So for any subtree modified in block N-1, the exec-side witness carries the stale generation and lacks the correct one. That matches the density numbers exactly: stale nodes ride along as dead weight while needed ones are absent.

Single point of failure. preloadFlatDiffReads (core/blockchain.go:5383) is the only thing that makes SRC re-read at the correct root to collect valid proof nodes, and it is driven entirely off flatDiff.ReadSet / ReadStorage / NonExistentReads. Witness completeness therefore reduces to "is the FlatDiff read-set a complete record of everything execution read?"

One confirmed contributor — V2 worker reads. Those sets are built exclusively from the final statedb: captureReadOnlyAccount, captureObjectStorage and captureNonExistentRead all iterate s.stateObjects and each object's originStorage. Under BlockSTM v2, worker reads live on ParallelStateDB instances backed by SafeBase pool copies that are discarded after settle and never reach finalDB.stateObjects. This is the same defect #2333 identified for the witness and fixed with a shared-reader drain (CollectStateWitness) — the FlatDiff read-set has the same defect and no equivalent drain. Disabling V2 moved the pass rate from 0/99 to 12/81, confirming it as a real contributor.

Not the whole story. 69/81 still fail in serial mode, where the read-set should be complete, so there is a second gap in read-set capture I did not pin down. Candidates I traced but could not eliminate by inspection: slots of N-1-mutated accounts that fall through the overlay to the stale storage trie, and objects no longer in s.stateObjects by the time CommitSnapshot runs.

Why the round-trip test passes

TestV2WitnessRegenerationPipelinedSRC / PipelinedSRCAllBlocks do drive the real split via witnessRegenPipelinedRoundTrip — the flag isn't the issue. They pass because the harness is single-block: execution and SRC both open at pb.witness.Root(). With one block there is no N-2, the offset is zero, the exec witness is already the right generation, and the read-set gap is fully masked. That's how 241/241 green coexists with 0/99 on a live node.

Reproducing it in go test

The minimum shape needs three block-heights: state committed at root_{N-2}; execute N-1 and CommitSnapshot without committing; execute N with the reader bound to root_{N-2} plus the FlatDiff_{N-1} overlay; run SRC(N-1); then round-trip block N's collected witness.

No new fixture capture needed — a consecutive pair supplies both roots, since witness_{N-1}.Root() is root_{N-2} and witness_N.Root() is root_{N-1}. The LFS set in core/blockstm/testdata isn't contiguous overall but contains 13 runs of ≥3 consecutive blocks:

run	blocks
83017816–83017860	45
83014081–83014113	33
83017760–83017792	33
83020826–83020843	18

A chained variant of witnessRegenPipelinedRoundTrip over one of those runs should fail in-tree. Two caveats: build the memdb from the union of the consecutive witnesses so both root generations are available, and anchor each block against its real state root from the fixture header (the anchoring already added here) rather than self-consistency — otherwise the same convergent-divergence blind spot that hid the 7702 gap could hide this too. Also worth avoiding the 83020871–83020898 run, since 83020871 is one of the knownIncompleteWitnessFixtures and would tangle the two issues.

Suggested fix direction

Deriving witness completeness from a reconstructed read-set is the fragile part — it depends on which statedb happened to observe each read. The robust fix is the same shape as #2333's: drain the shared reader's caches into the FlatDiff read-set (the reader already knows every key read, worker or not) instead of rebuilding it from finalDB.stateObjects. That closes the confirmed V2 contributor and plausibly the residual one, since it removes the dependency on read attribution entirely.

Happy to hand over the replay tooling if useful. Note this does contradict the "witness proof-node-set parity + stateless replay across all configurations" claim in the description, and it's the concrete failure mode the earlier "SRC must extend, not rebuild" concern was circling — extending turns out to be necessary but not sufficient, because the extended nodes are the wrong generation.

[pratikspatil024]

@cffls Confirmed on all counts — reproduced in-tree at exactly the shape you suggested, root-caused, and fixed in 12bd14abe. Your diagnosis was right, including the part you flagged as unexplained: there was a third capture gap behind the serial residual.

The repro

TestV2WitnessRegenerationPipelinedSRCChained is the chained variant you proposed: union hash-db of two consecutive witnesses, block N-1 through the pipeline in direct mode, SRC(N-1) commits root_{N-1} into the shared triedb, block N executed over the FlatDiff overlay with readers pinned at root_{N-2} (the setupBlockReaders + SetFlatDiffRef production shape), SRC(N) completes the witness at root_{N-1}, and the completed witness stateless-replays anchored against the real header roots. It runs over all 222 consecutive pairs in the existing LFS set (the 7702-incomplete fixtures excluded, per your note, so the two issues stay untangled). Before the fix: every sampled pair failed with your exact signature — SRC root correct, replay diverging with large gas shortfalls.

Your read on why the single-block oracles passed was also exactly right — with one block there is no N-2, so they are structurally blind to the offset. They stay in the suite as same-generation regression checks; the chained sweep is now the one that carries the load.

Three gaps, not two

1. V2 worker reads (your confirmed contributor) — fixed with the drain you suggested: readerWithCache.CollectReadSet enumerates every key resolved through the shared reader cache (worker or finalDB, whichever layer served it), merged into the read-set at CommitSnapshot. Attribution-free by construction.
2. Overlay-served reads — keys served by the parent FlatDiff never reach the reader at all, and by definition those are keys the parent block just wrote, so their subtrees are wrong-generation guaranteed. The overlay probes (accountOverlay/storageOverlay) now record every hit into a tracker on the shared FlatDiff — workers and finalDB record into the same object — drained at the next CommitSnapshot. Armed only when a witness is being produced, so witness-off imports pay nothing.
3. The serial residual you couldn't pin: read-only slots of accounts the block also mutates. IntermediateRoot's witness harvest collects only obj.trie.Witness() (write paths) for mutated accounts, and its read-only loop explicitly skips objects with mutations. The preload walks those read-only slots through the SRC statedb's reader, whose tracers nothing on the SRC side ever drained — core, core/state: produce complete witnesses under BlockSTM v2 #2333 added CollectStateWitness on the execution side, but the SRC statedb never got the mirror call. We proved this one with a slot-level trace on a stubborn pair (every EVM-touched slot was already in the read-set; the missing nodes were reader-tracer orphans on a mutated account) and fixed it with the mirror call in runSRCCompute. This one is exec-mode-independent, which is why disabling V2 only moved your numbers from 0/99 to 12/81.

Verification so far

• Chained sweep: 222/222 pairs, stateless replay to identical gas/receipt/state roots against real mainnet anchors.
• Both AllBlocks oracles unchanged (241/241), prewalk-participation guard green, core/state suite green, race pass over the chained pairs + pipelined import tests green.
• In progress: a live dev-node witness-replay A/B on the fixed build (mirroring yours), and a rerun of the stateless kurtosis leg. On that leg — a correction to our earlier report: the devnet result ("stateless validators consumed pipelined-produced witnesses successfully") was real but devnet-masked. On an idle devnet essentially all state access is system-call work on finalDB and the one hot subtree (the EIP-2935 contract) is mutated every block, so the read-set happened to be complete there. It could not have caught this; your live-node A/B is the coverage that matters, and we'd gladly take the replay tooling off your hands to make it a standing check.

One expected side effect worth noting for your density metric: the drained read-set includes speculative prefetcher reads (the cache is shared), so witnesses get somewhat larger rather than smaller — a superset costs size, never correctness. If the bloat is measurable we can role-tag the cache entries in a follow-up.

The PR description's witness-completeness claim has been rewritten accordingly.

[Copilot]

Pull request overview

Copilot reviewed 87 out of 88 changed files in this pull request and generated no new comments.

Suppressed comments (1)

core/stateless/witness_test.go:223

• This assertion can panic if the returned error string is shorter than expectedError (slicing err.Error() without a length guard). Even if today’s error messages are longer, keeping the test panic-free makes failures easier to diagnose.

[pratikspatil024]

Follow-up: both in-progress verification items from the previous comment are done, and the live A/B is decisive.

Live dev-node witness replay A/B (mainnet, syncing at tip)

Same methodology as yours — replay every produced witness through the real serial processor with a stub engine, anchored against the canonical block roots (gas + receipt root + state root). Same node, same config, only the binary differs; the pre-fix witnesses were still on disk, so the control and the fixed run share everything else.

build	blocks	pass	fail
pre-fix (adaa2f032, control)	91582500–91582699	0	200
fixed (12bd14abe)	91585500–91585699	200	0

The 0/200 control reproduces your 0/99 and validates the harness; the 200/200 confirms the fix end-to-end on live mainnet blocks.

On density: the fixed build measures 294.0 nodes/Mgas, 110.0 KB/Mgas over the passing range (vs your 279.1 / 104.7 for the broken pipelined variant and 249.9 / 93.1 for pipeline-off — different block samples, so indicative only). That's the expected superset cost of the attribution-free drain (~+5% nodes over the broken variant, ~+18% over pipeline-off). If that overhead matters for witness-serving nodes we can role-tag the reader-cache entries in a follow-up and drop the speculative-prefetcher share.

Stateless kurtosis leg, rerun on the fixed build

459 blocks (through Rio at 128), all nine nodes — 3 witness-producing validators, 2 stateless-sync validators, the bor RPC witness-flag matrix, and erigon — at identical head, root_mismatch = 0 everywhere. Load this time was targeted at the fix's hardest read class: 20 transactions calling a contract that reads the parent block's hash from the EIP-2935 history contract, i.e. user-level reads of the slot the previous block's system call just wrote — overlay-served, wrong-generation-guaranteed. All consumed successfully by both stateless-sync validators.

The replay harness from the A/B is a ~100-line self-contained test (opens chaindata + witness filestore read-only, replays a block range); happy to compare notes against your tooling or fold them together into the standing check discussed for CI.