[Bug]: Nested-paging (EPT) ring-0 assertion in pgmPoolTracDerefGCPhys plus RCU stalls booting a Linux 7.0.x guest with nested VT-x (VirtualBox 7.2.x, Windows host)

[zoomdbz]

Update: this also reproduces as a ring-0 VMM assertion (Guru Meditation), not only an RCU stall, and it lands in the nested-paging (EPT) code

Description

Intermittent permanent RCU expedited stall in a Linux SMP guest, beginning 2-4 seconds after kernel start. One secondary vCPU (varies per boot; observed CPU 1 and CPU 4) stops responding to expedited grace period IPIs while idle in pv_native_safe_halt and never wakes again. The stall never resolves; observed for 68+ minutes, with the jiffies counter in the stall reports tracking uptime back to roughly t+2s.

Depending on when it lands, the guest either hangs visibly during boot, or comes up half-wedged: the lightdm greeter renders black while ModemManager, systemd-hostnamed, fail2ban-server and dockerd accumulate hung-task warnings (blocked > 241s) behind synchronize_rcu_expedited.

The NMI watchdog backtrace DOES reach the stuck vCPU (RIP: pv_native_safe_halt+0xf/0x20, trace default_idle -> default_idle_call -> do_idle -> cpu_startup_entry -> start_secondary), so NMI delivery works while the normal IPI wakeup path does not. The host-side VBox.log for stalled sessions is clean: native VT-x ("VT-x w/ nested paging and unrestricted guest execution hw support", no NEM fallback), no guru meditation, no timer catch-up warnings.

Guest kernel 6.19.14+kali-amd64 on the identical VM has never shown this across months of daily use, including clean boots today interleaved with 7.0.12 failures. Upgrading VirtualBox 7.2.8 -> 7.2.12 did not change behavior; the attached log is a 7.2.12 stall session.

Frequency

2026-07-14: at least five stalled boots and one fully clean boot of guest kernel 7.0.12, across VirtualBox 7.2.8 and 7.2.12 on the same host. Guest kernel 6.19.14: zero failures ever. No deterministic trigger found; behavior is consistent with a timing race in waking a halted vCPU.

Guest dmesg excerpts

Early-hang boot (7.2.8 host, victim CPU 1, 68 minutes into the stall):

rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 1-...D } 1021209 jiffies s: 249 root: 0x2/.
rcu: blocking rcu_node structures (internal RCU debug):
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G        W  OE 7.0.12+kali-amd64 #1 PREEMPT(lazy)  Kali 7.0.12-2kali1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
RIP: 0010:pv_native_safe_halt+0xf/0x20
 default_idle+0x9/0x20
 default_idle_call+0x2f/0x130
 do_idle+0x27a/0x2b0
 cpu_startup_entry+0x29/0x30
 start_secondary+0x126/0x180
 common_startup_64+0x13e/0x141

Late-stall boot (7.2.12 host, victim CPU 4; reports repeat every ~63s with jiffies tracking uptime):

[   25.596153] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 4-...D } 5463 jiffies s: 541 root: 0x10/.
[   89.054636] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 4-...D } 21335 jiffies s: 541 root: 0x10/.
[  152.507686] ... 37207 jiffies ...
[  342.877738] ... 84823 jiffies ...
[  363.347161] INFO: task ModemManager:896 blocked for more than 241 seconds.
[  363.347560] INFO: task ModemManager:896 is blocked on a mutex likely owned by task (systemd-hostn):899.
[  363.347656] INFO: task (systemd-hostn):899 blocked for more than 241 seconds.
[  363.347991] INFO: task (fail2ban-serv):1017 blocked for more than 241 seconds.
[  363.348299] INFO: task dockerd:1086 blocked for more than 241 seconds.

Blocked tasks show synchronize_rcu_expedited+0xd8/0x220 in their stacks.

Environment

• Host: Windows 11 Pro 10.0.26200.8655, Intel Core i9-13900K (hybrid, 8 P + 16 E cores), 64 GB RAM
• VirtualBox: 7.2.8 r173730 and 7.2.12 r174389 (both reproduce)
• Guest: Kali rolling, kernel 7.0.12-2kali1 (7.0.12+kali-amd64, PREEMPT_DYNAMIC, lazy preemption). Known good on same VM: 6.19.14+kali-amd64
• Guest Additions: 7.2.8 r173730 (ISO install)
• VM config: 8 vCPUs, ICH9, BIOS firmware, 16 GB RAM, IO-APIC on, x2APIC on, paravirt provider KVM (GIM KVM active, TSC mode RealTSCOffset), nested paging on, unrestricted guest on, large pages on, VMSVGA with 3D disabled. Nested VT-x was enabled during all failures above.

Workaround

Pinning the guest to kernel 6.19.14 on the identical VM configuration avoids the issue reliably (months of use, zero failures, including clean boots interleaved with 7.0.12 failures on the same day). No VM configuration change (vCPU count, x2APIC, nested VT-x, paravirt provider) was tested as a workaround.

Attachments

VBox.log of the 7.2.12 stall session (victim CPU 4) is attached in the first comment. Guest-side screenshots of both stall variants available, plus full guest kernel journal on request.

[zoomdbz]

Full host-side VBox.log of the 7.2.12 stall session (victim vCPU 4, guest kernel 7.0.12-2kali1): https://gist.github.com/zoomdbz/0b3b55b760dd0723f863bb9ff981584f

[zoomdbz]

VBox-stall-CPU4-7.2.12-20260714.log

[zoomdbz]

• boot -2 (7.0.12), lasted 25 seconds: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 5-...D } 5526 jiffies with RIP: pv_native_safe_halt+0xf/0x20. CPU 5 wedged in the paravirt idle-halt, expedited RCU stalled waiting on it, boot died in 25s. That is the race, caught red-handed.
• boot -3 (7.0.12), lasted 57 seconds: no stall printed before it died, but a sub-minute boot that got rebooted is the same failure not reaching the console.
• boot -4: had at least one hung-task (kernel started suppressing further reports).
• boot -1 (6.19.14), ran 81 minutes clean: zero race hits. Your stable kernel, behaving.
• boot 0 (7.0.12, current): clean so far, but only 4 minutes up. Boots -2/-3 died in under a minute, so 4 minutes means dodged it this time, not that it's fixed.

[zoomdbz]

Update: this also reproduces as a ring-0 VMM assertion (Guru Meditation), not only an RCU stall, and it lands in the nested-paging (EPT) code.

Same setup as the report: VirtualBox 7.2.12 r174389, Windows 11 host, Intel i9-13900K, guest Kali 7.0.12-2kali1, 8 vCPU, VT-x + nested paging + unrestricted guest, nested-hw-virt on, GIM = KVM paravirt, x2APIC on.

On a 7.0.12 cold boot (2026-07-16), about 2m24s in, VCPU2 took a Guru Meditation:

!! VCPU2: Guru Meditation -2701 (VERR_VMM_RING0_ASSERTION)
!!R0-Assertion Failed!!
Location : PGMAllPool-x86.cpp(4501) pgmPoolTracDerefGCPhys(...)
HCPhys=00000002cfc2d000 GCPhys=00000002324ef000; found page has HCPhys=0000000ec2eef000 iPte=0 fIsNested=true

Top stack frames:

pgmPoolTracDerefGCPhys              <- assertion, PGMAllPool-x86.cpp:4501
...
pgmR0BthEPTProtNestedTrap0eHandler  <- EPT nested-guest page-fault handler

So while servicing a nested-guest EPT page fault, the shadow/nested page-pool tracker found a host-physical mismatch for a guest-physical page in the fIsNested=true branch and asserted.

I believe the RCU expedited stalls in the original report and this pool-tracking assertion are the same nested-paging inconsistency surfacing two ways under guest kernel 7.0.12: on some boots a secondary vCPU silently wedges in the paravirt idle-halt and never takes its wakeup IPI (guest prints the RCU stall), on other boots the PGM pool tracker catches the bad HCPhys first and the hypervisor asserts.

Repro, unchanged: guest 7.0.12-2kali1 fails within about 1-3 minutes on most cold boots (RCU stall or this Guru); guest 6.19.14 on the identical VM has never failed (last clean run 81 minutes). Requires nested-hw-virt = on. x2APIC was on for this crash; toggling it off in an earlier session did not prevent the stall either. Both 7.2.8 and 7.2.12 are affected.

Full VBox.log for the Guru session (complete ring-0 dump, VMCS, EPTP, and the stack at the assertion): https://gist.github.com/zoomdbz/21a1e98a878f0a1e87a6179d7ba114a3

[zoomdbz]

Additional data point: reproduces at 2 vCPUs (guest side), so this is not a high-SMP scaling race.

Dropped the guest to 2 vCPUs. It boots successfully more often (fewer EPT operations per boot), but still fails intermittently. Guest journal (journalctl -b -1 -k) from a failed 2-vCPU boot, kernel 7.0.12-2kali1, same host (VirtualBox 7.2.12, i9-13900K, nested-hw-virt on, x2APIC on):

rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=2.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
...
WARNING: kernel/rcu/tree_exp.h:803 at rcu_exp_handler+0x4a/0x160, CPU#0: (udev-worker)/616
RIP: 0010:rcu_exp_handler+0x4a/0x160
Call Trace:
 ? __pfx_rcu_exp_handler+0x10/0x10
WARNING: kernel/rcu/tree_plugin.h:823 at rcu_sched_clock_irq+0x872/0xc80, CPU#0: (udev-worker)/616
RIP: 0010:rcu_sched_clock_irq+0x872/0xc80
Call Trace:
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 0-...D } 5443 jiffies s: 521 root: 0x1/.
rcu: blocking rcu_node structures (internal RCU debug):
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 0-...D } 21315 jiffies s: 521 root: 0x1/.

Two differences from the 8-vCPU captures: at 2 vCPUs the victim is CPU 0 (the boot CPU), not a secondary vCPU; and the stall is preceded by two kernel WARNINGs in the RCU expedited path itself, rcu_exp_handler (tree_exp.h:803) and rcu_sched_clock_irq (tree_plugin.h:823), on the same CPU that then stalls. The expedited-grace-period IPI handler is warning about an unexpected state on that CPU immediately before the grace period stalls on it.

Picture so far:

• Guest kernel 7.0.12-2kali1 fails on most cold boots; 6.19.14 on the identical VM never fails (last clean run 81 minutes). Requires nested-hw-virt on.
• Fails at both 8 and 2 vCPUs, so it is not an SMP-scaling race. The victim vCPU varies (a secondary vCPU at 8, CPU 0 at 2).
• Two faces of one nested-paging fault: host-side, the ring-0 Guru assertion in pgmPoolTracDerefGCPhys (fIsNested=true) reached via the EPT nested-guest page-fault handler; guest-side, this RCU expedited stall (a vCPU wedged, never taking its wakeup IPI). x2APIC on or off, both fail. VirtualBox 7.2.8 and 7.2.12 both affected.