Bor Deadlock Issue

[gaia]

System information

• Bor client version: v2.9.0 (GitCommit ad85c473567f411d7ddfd4739ef314064b2fdeb6, Go 1.26.3; bor version reports bor/matic-bor/v2.9.0/linux-amd64/go1.26.3)
• Heimdall client version: v0.9.0 (Heimdall V2)
• OS & Version: Linux (Ubuntu), running as an unprivileged LXC container on Proxmox VE (host kernel 7.0.6-2-pve)
• Environment: Polygon Mainnet (ChainID 137)
• Type of node: Validator
• Additional Information: syncmode=stateless, "db.engine"="pebble", "state.scheme"="path", cache=8192, maxpeers=80. Note: the binary is built from a private fork (stack-trace paths read /home/runner/work/bor-private/bor-private/...); the affected code paths (eth/downloader, triedb/pathdb) are go-ethereum-derived.

Overview of the problem

A mainnet validator running Bor v2.9.0 in syncmode=stateless intermittently deadlocks its sync subsystem: the block head freezes, no new blocks are imported for hours, the process stays alive with 0% CPU (wchan=futex_do_wait) and RPC still responsive, and it does not self-recover — only a process restart clears it. On this node it recurs roughly every ~30 min–3 h of uptime, across multiple process lifetimes (not a one-off corrupt state).

The freeze correlates with a mode=stateless peer-synchronization timeout followed by a peer drop. A SIGQUIT goroutine dump captured during the wedge (taken twice, in two independent occurrences — same result) shows the three sync movers permanently parked: the header skeleton (eth/downloader/skeleton.go:259, select), the state fetcher (eth/downloader/statesync.go:48, select), and the pathdb trie generator (triedb/pathdb/generate.go:828, chan receive, starved).

Expected behaviour: after a peer-sync timeout Bor should drop the peer and continue syncing from other peers (it has ~79). Actual behaviour: the sync coordinator wedges and the head stays frozen until a restart.

Downstream effect: because the head is frozen, Bor cannot serve the block referenced by a checkpoint vote extension with 512 confirmations, so Heimdall logs bor block not found locally: end=… confirmations=512 target=… and the validator misses checkpoints until Bor is restarted and re-synced.

Reproduction Steps

1. Run Bor v2.9.0 on mainnet with syncmode=stateless (config excerpt below), behind a healthy Heimdall v0.9.0.
2. After some uptime (~30 min–3 h), observe a stateless peer-sync failure in the logs:
   WARN Synchronisation failed, dropping peer err=timeout mode=stateless.
3. Shortly after, eth_blockNumber stops advancing while the chain moves on; eth_syncing reports a currentBlock that never changes and a highestBlock that keeps creeping; CPU drops to 0%; the journal shows only a Fetching state sync events from Heimdall poll loop and Whitelisting milestone deferred err="chain out of sync" every ~2 s.
4. The node never recovers on its own. systemctl restart bor (or a SIGQUIT-dump + restart) restores sync; the deadlock then recurs after the next uptime interval.

Logs / Traces / Output / Error Messages

Trigger (same minute the head starts lagging):

WARN [..] Network response id collision      have=…eth/69:3  want=…eth/69:4
WARN [..] Synchronisation failed, dropping peer   peer=a7a27f…  err=timeout  mode=stateless

Stuck state after the freeze (repeats indefinitely, no Imported lines):

INFO [..] Fetching state updates from Heimdall   fromID=3,179,029 ...
INFO [..] Fetching state sync events             queryParams="from_id=...&to_time=...&pagination.limit=50"
INFO [..] Whitelisting milestone deferred        err="chain out of sync"

Goroutine dump (captured via systemctl kill -s QUIT while wedged; full dumps linked below) — the sync pipeline is parked for the entire stuck duration:

goroutine 469 [select, 218 minutes]:
  runtime.selectgo
  github.com/ethereum/go-ethereum/eth/downloader.(*skeleton).startup      // skeleton.go:259
  github.com/ethereum/go-ethereum/eth/downloader.newSkeleton.gowrap1      // skeleton.go:241

goroutine 470 [select, 201 minutes]:
  github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).stateFetcher   // statesync.go:48
  github.com/ethereum/go-ethereum/eth/downloader.New.gowrap1

goroutine 422 [chan receive, 218 minutes]:
  runtime.chanrecv1
  github.com/ethereum/go-ethereum/triedb/pathdb.(*generator).generate      // generate.go:828
  github.com/ethereum/go-ethereum/triedb/pathdb.(*generator).run.gowrap1   // generate.go:137

Full goroutine dumps (838 goroutines / ~15k lines each, captured in two independent occurrences — identical deadlock site):

• Dump 1 (2026-06-29T17:37Z): <PASTE GIST URL — upload bor-2.9.0-stateless-deadlock-goroutines.txt>
• Dump 2 (2026-06-29T18:40Z): <PASTE GIST URL — upload dump-2026-06-29T18:40:01Z.txt>

Additional Information

1. bor.service (no modifications; standard):

   [Unit]
     Description=bor
     StartLimitIntervalSec=500
     StartLimitBurst=5
   [Service]
     Restart=on-failure
     RestartSec=5s
     ExecStart=/usr/bin/bor server -config "/var/lib/bor/config.toml"
     Type=simple
     KillSignal=SIGINT
     User=bor
     TimeoutStopSec=300
   [Install]
     WantedBy=multi-user.target

   Relevant config.toml (secrets redacted):

   chain = "mainnet"
   syncmode = "stateless"
   "db.engine" = "pebble"
   "state.scheme" = "path"
   [cache]
     cache = 8192
   [p2p]
     maxpeers = 80
     maxpendpeers = 20
     port = 30303
     nodiscover = false
     nodekeyhex = "<REDACTED>"
     [p2p.discovery]
       dns = ["enrtree://AKUEZKN7PSKVNR65FZDHECMKOJQSGPARGTPPBI7WS2VUL4EGR6XPC@pos.polygon-peers.io"]
       bootnodes = [ <standard Polygon mainnet bootnodes> ]
   [heimdall]
     url = "http://<HEIMDALL_HOST>:1317"
   [jsonrpc.http]
     api = ["eth", "net", "web3", "txpool", "bor", "admin"]

   Chain config: mainnet, ChainID 137, Engine bor.

2. Syncing/peers/node info (IPs/IDs redacted):

   • maxpeers = 80; admin.peers.length ≈ 79 (stable, healthy peer count throughout the stall).
   • admin.nodeInfo:

     name: bor/matic-bor/v2.9.0/linux-amd64/go1.26.3
     enode: enode://821e5bb017d4a2b37b056c…<ID redacted>@[<IP REDACTED>]:30303
     listenAddr: [::]:30303   ports: discovery=30303, listener=30303
     protocols: eth, snap, wit
     

   • eth.syncing while wedged (representative; currentBlock frozen, highestBlock advancing, no heal progress):

     {"currentBlock":"0x5535fce","startingBlock":"0x5535fce","highestBlock":"0x5536a3c",
      "healedTrienodes":"0x32c72831","healingTrienodes":"0x0","stateIndexRemaining":"0x0", ...}

     currentBlock (0x5535fce) does not change across repeated calls; highestBlock creeps upward as the chain advances.

3. Heimdall config (/var/lib/heimdall/config/app.toml, relevant lines, keys redacted):

   eth_rpc_url = "https://mainnet.infura.io/v3/<REDACTED>"
   bor_rpc_url = "http://<BOR_HOST>:8545"      # local Bor
   bor_grpc_flag = "false"
   comet_bft_rpc_url = "http://0.0.0.0:26657"

   Heimdall is healthy throughout (latest_block_time fresh within ~5 s, catching_up=false); all heimdall services (heimdalld, bridge, rest-server) running. The communication Bor→Heimdall works (Bor keeps fetching state-sync events); the failure is internal to Bor's sync coordinator.

4. CURL: N/A — not an RPC-request bug. The behaviour is observable via eth_blockNumber (frozen) / eth_syncing (non-advancing currentBlock) over time.

[gaia]

logs.zip

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[gaia]

ping

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[gaia]

ping