PVE kernel 7.0.14-4-pve can-t mount ext4 FS on drbd

[bajizs]

After upgrading the PVE kernel to 7.0.14-4-pve (Proxmox 9.x), we can no longer mount the ext4 filesystem on DRBD storage. This prevents LINSTOR HA from working.

After updating the Proxmox node, we can no longer start the LINSTOR controller managed by drbd-reactor. The problem occurs when mounting the /var/lib/linstor filesystem from the DRBD device /dev/drbd1000. fsck does not find any problems, and when I move the filesystem to an image file, it mounts normally.

Kernel reports error: EXT4-fs (drbd1000): bad block size 1024

#dumpe2fs  /dev/drbd1000
Filesystem features:      has_journal ext_attr resize_inode dir_index orphan_file filetype extent 64bit flex_bg metadata_csum_seed sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Block size:               1024
Journal inode:            8
Journal backup:           inode blocks
Journal features:         journal_64bit journal_checksum_v3
Journal sequence:         0x00000170
Journal start:            0
Journal checksum type:    crc32c
Journal checksum:         0x380a54b0

# blkid /dev/drbd1000
/dev/drbd1000: UUID="4fac6cbc-7f2c-4a8a-80cc-e2bda4ed014d" BLOCK_SIZE="1024" TYPE="ext4"

#uname --kernel-release 
7.0.14-4-pve
# linstor -v
linstor-client 1.28.1; GIT-hash: UNKNOWN
#linstor controller version
linstor controller 1.34.1; GIT-hash: c4ba920da9c8d533315725e8252fa806ca39153d
#dkms status
drbd/9.3.3-1, 7.0.14-4-pve, x86_64: installed (Original modules exist)

[bajizs]

New information, interesting event. I restarted one of the two Proxmox nodes where linstor_db is located. Unexpectedly, the old node that had not been restarted mounted the filesystem — unfortunately, drbd_reactor had remained enabled. To my surprise, I found that the filesystem worked normally. This state remained until we restarted the other node as well; since then, the filesystem can no longer be mounted again. Maybe this helps uncover the issue, but it is frustrating that such a basic function has such a bug.

[ghernadi]

Sorry for the delay. With 1.33.0 we introduced an auto-block-size feature that should manage blocksizes in case satellites have different local block sizes (we had a few customers that were running into issues regarding 512 vs 4k block sizes).

However, when introducing this new feature we also disabled it for all existing resource-definitions in order to avoid issues similar to what you describe here. So that brings us back to the beginning of the search.

I would still ask you to double-check: linstor rd lp linstor_db and just to be sure linstor vd lp linstor_db 0 if the DrbdOptions/auto-block-size is effectively disabled.

If the feature is disabled (as I would expect), can you please provide some logs, or details. I see that a linstor c version succeeded, so the controller is apparently running. Where exactly could you not start the controller? I think a more detailed overview could help here.

[bajizs]

Hello,
LINSTOR is now running because we are not using DRBD storage for linstor_db at the moment (it can not be mounted). We have disabled drbd-reactor and copied the files directly to the /var/lib/linstor folder. We dumped the linstord_db DRBD storage to an image file, mounted it without any problems, and copied all the files to the /var/lib/linstor folder. So the system is usable now, but without LINSTOR HA.

# linstor rd lp linstor_db
╭────────────────────────────────────────╮
│ Key                         │ Value    │
╞════════════════════════════════════════╡
│ DrbdOptions/Resource/quorum │ majority │
│ DrbdOptions/auto-verify-alg │ sha512   │
╰────────────────────────────────────────╯

# linstor vd lp linstor_db 0
╭───────────────────────────────────────────────────────────────╮
│ Key                                        │ Value            │
╞═══════════════════════════════════════════════════════════════╡
│ DrbdCurrentGi                              │ D94D5B043B0FD3FE │
│ DrbdOptions/Disk/block-size                │ 4096             │
│ DrbdOptions/Disk/discard-granularity       │ 16384            │
│ DrbdOptions/Disk/discard-zeroes-if-aligned │ yes              │
│ DrbdOptions/Disk/rs-discard-granularity    │ 16384            │
│ Linstor/Drbd/InitialUptoDateOn             │ RU01             │
│ Linstor/Drbd/freeze-block-size             │ True             │
│ StorDriver/internal/AllocationGranularity  │ 16               │
╰───────────────────────────────────────────────────────────────╯
``

When we try to mount the DRBD storage, we get the error I described.

# drbdadm status linstor_db 
linstor_db role:Secondary
  disk:UpToDate open:no
  ru02 role:Secondary
    peer-disk:UpToDate

# mount /dev/drbd/by-res/linstor_db/0 /tmp/TEST
mount: /tmp/TEST: fsconfig() failed: /dev/drbd1000: Can't open blockdev.
       dmesg(1) may have more information after failed mount system call.

This will finish without any message in dmesg.

# drbdadm primary linstor_db 
# mount /dev/drbd/by-res/linstor_db/0 /tmp/TEST

Now success, messages in dmesg:

[972830.784298] drbd linstor_db: Preparing cluster-wide state change 813222424: 0->all role( Primary )
[972830.785767] drbd linstor_db: State change 813222424: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC
[972830.786151] drbd linstor_db: Committing cluster-wide state change 813222424 (2ms)
[972830.786568] drbd linstor_db: role( Secondary -> Primary ) [primary]
[972895.577889] EXT4-fs (drbd1000): mounted filesystem a2304839-b942-4ff3-9f71-057c743d52b6 r/w with ordered data mode. Quota mode: none.

Honestly, I don’t know what happened now, because as you can see, the mount suddenly succeeded.

The only difference from this morning’s attempts is that before that I manually set the DRBD storage to primary. But I also tried that when reporting the issue, and then nothing helped until we restarted the server today. After that, I tried switching between primary and secondary manually, and even the disconnect state, just to make it work on at least one machine, but none of it worked. Meanwhile, using the dd command, I was able to copy the contents into an .img file without any problem, and from that I started the LINSTOR controller.

I can’t explain these events. There must be a bug somewhere that came up during the reboot after the update, while this issue seems to have disappeared in the meantime, as far as I can tell. So for now, let’s put this on hold. I’ll also try moving other servers to kernel 7 and see what happens there.

[ghernadi]

Thanks! My previous assumption that DrbdOptions/auto-block-size is disabled was apparently wrong. Since the property is nowhere set for you, it is True by default, aka the feature is enabled. That could have caused the issues.
In fact we do see that DrbdOptions/Disk/block-size is set to 4096 so it looks like auto-block-size did run.

If the FS was now created originally with 1k block-size but someone who enabled auto-block-size (by removing the property for example) caused LINSTOR to change the block-size to 4k for the DRBD device, it might happen that mount is not particularly happy about that.

I think you have two options now:

1. Recreate the linstor_db using a 4k bs this time (apparently this gets to be the default on your system either way) and copy into that newly created resource the files you are currently using for the up and running controller.
2. Keep the 1k FS and configure LINSTOR to leave the blocksize alone:

• Set DrbdOptions/auto-block-size to False on the resource-definition,
• Set DrbdOptions/Disk/block-size to the value your FS supports, I'd try 1024 first.

I'd recommend option 1, but - as always - please create backups (LINSTOR db as well as the content of the device itself) in case you need to rollback.

---

One other things to mention, regarding the failed mount:

# mount /dev/drbd/by-res/linstor_db/0 /tmp/TEST
mount: /tmp/TEST: fsconfig() failed: /dev/drbd1000: Can't open blockdev.
       dmesg(1) may have more information after failed mount system call.

This is as designed. The linstor_db has explicitly DRBD's "auto-promote" feature disabled, for very good reasons (do not change that!). It is drbd-reactor that is supposed to (manually) set the DRBD resource to primary before mounting it and make it secondary again when it no longer needs it.

[bajizs]

I also think the best long-term solution is to reformat the partition with a 4K block size. I restored the full functionality of drbd-reactor based on that, and I just tested it — everything is fine. Thank you for the information.

I was also thinking about what could have triggered this situation and why the reboot solved it. During the update, as always, we updated only one node at a time, so the servers were offline one by one. We didn’t track it separately during the process, but when I checked, linstor_db had been on the node that was updated last the whole time. In this situation, the expected behavior is that when the node is shut down, it moves to the other node. However, due to the block size, it could no longer move there. But after the last node came back up with the new kernel and the new DRBD KMS module, it’s not entirely clear to me why it still remained in the state of trying the wrong block size.

Today, we restarted the last node for other reasons — the very one that had linstor_db on it last before the problem — and suddenly the system recovered. Presumably, this is when the automatic block size got corrected.

However, we do not know why it could not recover, for example, when we intentionally disconnected the nodes, even at the risk of split brain.

Note: before we upgraded the kernel, all nodes had already been updated and restarted for the LINSTOR satellite and LINSTOR controller, specifically to ensure that after the server reboot, the same version would definitely be running on every location.