AKS nodepool issue with xfs_growfs only on CentralUS region
https://askubuntu.com/questions/1559618/aks-nodepool-issue-with-xfs-growfs-only-on-centralus-region
Our problem occur only on CentralUS region, the same exact configuration is working on other regions (uksouth, eastus2) - we reached previously Azure support for help, but they could not identify the issue, and pointed us to reach maybe help here:
I think the error fundamentally can be something with disk geometry/(some driver) as we think the sunit=0 should be always there ? Not sure, when we create lv and sunit is not 0 it is always fail to xfs_grows.
We have spined up multiple AKS cluster, with newest (and previously tested older) Ubuntu Image: Ubuntu Linux: Node image version AKSUbuntu-2204gen2containerd-202510.29.0 - basically what we want to achieve it is extending our created Logical Volume with xfs_grows: (behind it there is TopolVM which does exactly the same commands as below creating pv and vg):
set -e
export RUNLEVEL=1
printf '#!/bin/sh\nexit 0' > /usr/sbin/policy-rc.d
apt update && apt install -y lvm2
if ! vgdisplay nvme-vg &>/dev/null; then
pvcreate -ff -y /dev/nvme0n1
vgcreate nvme-vg /dev/nvme0n1
else
echo "Volume group nvme-vg already exists. Skipping."
fi
).
After that we use topolVM as CSI driver, but esentially it is doing some standard Linux commands like below:
On some CentralUS region nodes is not working on some of them it is working:
NOT WORKING EXAMPLE:
root@aks-az2f60c-14008962-vmss000001:/# pvs
PV VG Fmt Attr PSize PFree
/dev/nvme0n1 nvme-vg lvm2 a-- <1.75t <1.72t
root@aks-az2f60c-14008962-vmss000001:/# mkdir /live-demo
root@aks-az2f60c-14008962-vmss000001:/# vgcreate nvme-vg /dev/nvme0n1
/dev/nvme-vg: already exists in filesystem
Run `vgcreate --help' for more information.
root@aks-az2f60c-14008962-vmss000001:/# lvcreate -n live-test -L 40G nvme-vg
Logical volume "live-test" created.
root@aks-az2f60c-14008962-vmss000001:/# mkfs.xfs /dev/nvme-vg/live-test
meta-data=/dev/nvme-vg/live-test isize=512 agcount=16, agsize=655359 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=10485744, imaxpct=25
= sunit=1 swidth=128 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=5119, version=2
= sectsz=512 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Discarding blocks...Done.
root@aks-az2f60c-14008962-vmss000001:/# mount /dev/nvme-vg/live-test /live-demo/
root@aks-az2f60c-14008962-vmss000001:/# xfs_growfs /live-demo/
meta-data=/dev/mapper/nvme--vg-live--test isize=512 agcount=16, agsize=655359 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=10485744, imaxpct=25
= sunit=1 swidth=128 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=5119, version=2
= sectsz=512 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device
WORKING EXAMPLE
root@aks-az2f60c-14008962-vmss000002:/# pvs
PV VG Fmt Attr PSize PFree
/dev/nvme0n1 nvme-vg lvm2 a-- <1.75t <1.75t
root@aks-az2f60c-14008962-vmss000002:/# mkdir /live-demo
root@aks-az2f60c-14008962-vmss000002:/# vgcreate nvme-vg /dev/nvme0n1
A volume group called nvme-vg already exists.
root@aks-az2f60c-14008962-vmss000002:/# lvcreate -n live-test -L 40G nvme-vg
Logical volume "live-test" created.
root@aks-az2f60c-14008962-vmss000002:/# mkfs.xfs /dev/nvme-vg/live-test
meta-data=/dev/nvme-vg/live-test isize=512 agcount=4, agsize=2621440 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=10485760, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=5120, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Discarding blocks...Done.
root@aks-az2f60c-14008962-vmss000002:/# mount /dev/nvme-vg/live-test /live-demo/
root@aks-az2f60c-14008962-vmss000002:/# xfs_growfs /live-demo/
meta-data=/dev/mapper/nvme--vg-live--test isize=512 agcount=4, agsize=2621440 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=10485760, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=5120, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@aks-az2f60c-14008962-vmss000002:/#
