Resizing qcow2 Files Used by el8 VMs on a el7 Hypervisor

This is my braindump of using virt-resize to migrate 3 OpenShift 4 master VMs' qcow2 disk files, on a CentOS 7 hypervisor, from 70G each in one libvirt storage pool to 150G each in another pool.

Analyse old qcow2 files

As always when using guestfs-tools, the VMs were shut down before operating on the qcow2 files.

Checking usage of old qcow2 files (click to expand details)
root@epyc ~ # virt-filesystems -a /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2 --all --long -h
Name       Type        VFS      Label       MBR  Size  Parent
/dev/sda1  filesystem  unknown  -           -    1.0M  -
/dev/sda2  filesystem  vfat     EFI-SYSTEM  -    127M  -
/dev/sda3  filesystem  ext4     boot        -    384M  -
/dev/sda4  filesystem  xfs      root        -    69G   -
/dev/sda1  partition   -        -           -    1.0M  /dev/sda
/dev/sda2  partition   -        -           -    127M  /dev/sda
/dev/sda3  partition   -        -           -    384M  /dev/sda
/dev/sda4  partition   -        -           -    69G   /dev/sda
/dev/sda   device      -        -           -    70G   -

showed me that I want to extend /dev/sda4.

Create (sparse) qcow2 Target Files

Virt-resize cannot do in-place disk modifications. You have to have space to store the resized output disk.

To store the resized disk image in a file, create a file of a suitable size:

(from the Manual page virt-resize(1))

I simply used virt-manager to create 3, each 150G large but thin allocated.

I could also have used qemu-img create -f qcow2… (click for details).
qemu-img create -f qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2 150G
qemu-img create -f qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master02.qcow2 150G
qemu-img create -f qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master03.qcow2 150G
chown qemu.qemu /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master0?.qcow2

Failed attempt on CentOS7

My initial attempt failed.

Resizing the xfs does not work out of the box on my CentOS 7 hypervisor (click to expand details).
root@epyc ~ # rpm -qf $(which virt-resize)
libguestfs-tools-c-1.40.2-10.el7.x86_64
root@epyc ~ # virt-resize --expand /dev/sda4 /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_
master01.qcow2                                                                                                                                    
[   0.0] Examining /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2
**********

Summary of changes:

/dev/sda1: This partition will be left alone.

/dev/sda2: This partition will be left alone.

/dev/sda3: This partition will be left alone.

/dev/sda4: This partition will be resized from 69.5G to 149.5G.  The 
filesystem xfs on /dev/sda4 will be expanded using the ‘xfs_growfs’ 
method.

**********
[   2.8] Setting up initial partition table on /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
[  16.5] Copying /dev/sda1
[  16.5] Copying /dev/sda2
[  16.7] Copying /dev/sda3
[  17.3] Copying /dev/sda4
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 159.4] Expanding /dev/sda4 using the ‘xfs_growfs’ method
virt-resize: error: libguestfs error: mount: mount exited with status 32: 
mount: wrong fs type, bad option, bad superblock on /dev/sda4,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

If reporting bugs, run virt-resize with debugging enabled and include the 
complete output:

  virt-resize -v -x [...]
root@epyc images # qemu-img info /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2
image: /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2
file format: qcow2
virtual size: 70G (75161927680 bytes)
disk size: 70G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: true
root@epyc images # qemu-img info /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2 
image: /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
file format: qcow2
virtual size: 150G (161061273600 bytes)
disk size: 70G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: true
root@epyc images # qemu-img check -r all /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
No errors were found on the image.
2457600/2457600 = 100.00% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 161086111744
And without expanding, using virt-resize makes no sense. That would add a new partition and I'd need to touch the CoreOS that the OCP4 master nodes use (click to expand details).
root@epyc ~ # virt-resize  /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.
qcow2                                                                                                                                             
[   0.0] Examining /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2
**********

Summary of changes:

/dev/sda1: This partition will be left alone.

/dev/sda2: This partition will be left alone.

/dev/sda3: This partition will be left alone.

/dev/sda4: This partition will be left alone.

There is a surplus of 80.0G.  An extra partition will be created for the 
surplus.

**********
[   2.8] Setting up initial partition table on /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
[  16.7] Copying /dev/sda1
[  16.7] Copying /dev/sda2
[  16.9] Copying /dev/sda3
[  17.6] Copying /dev/sda4
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00

Resize operation completed with no errors.  Before deleting the old disk, 
carefully check that the resized disk boots and works correctly.

This is a known issue with an el7 libvirt host attempting to work with el8 images.

Workaround, use Upstream’s Appliance

One can either use an el8 host to do the resize or apply the workaround from access.redhat.com solution Using RHEL 8 virtual disk images on a RHEL 7 host sometimes fails.

See also Bug 1671895 and How can I switch to a fixed / prebuilt appliance? in the libguestfs Frequently Asked Questions (FAQ).

tl;dr: obtain latest appliance from http://download.libguestfs.org/binaries/appliance/, unpack, set LIBGUESTFS_PATH, run virt-resize using the downloaded appliance (click to expand details).
root@epyc ~ # mkdir /export
root@epyc ~ # cd /export
root@epyc export # wget https://download.libguestfs.org/binaries/appliance/appliance-1.40.1.tar.xz
root@epyc export # tar -Jxf appliance-1.40.1.tar.xz 
root@epyc export # ls appliance
initrd  kernel  README.fixed  root
root@epyc export # chmod -R a+rwX /export/appliance                                                                                               
root@epyc export # export LIBGUESTFS_PATH=/export/appliance
root@epyc export # virt-resize --expand /dev/sda4 /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2 
[   0.0] Examining /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2
**********

Summary of changes:

/dev/sda1: This partition will be left alone.

/dev/sda2: This partition will be left alone.

/dev/sda3: This partition will be left alone.

/dev/sda4: This partition will be resized from 69.5G to 149.5G.  The 
filesystem xfs on /dev/sda4 will be expanded using the ‘xfs_growfs’ 
method.

**********
[   4.2] Setting up initial partition table on /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
[  18.6] Copying /dev/sda1
[  18.7] Copying /dev/sda2
[  18.9] Copying /dev/sda3
[  19.4] Copying /dev/sda4
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 155.1] Expanding /dev/sda4 using the ‘xfs_growfs’ method

Resize operation completed with no errors.  Before deleting the old disk, 
carefully check that the resized disk boots and works correctly.
root@epyc export # virt-resize --expand /dev/sda4 /var/lib/libvirt/images/on_SSD/ocp4_master02.qcow2 /var/lib/libvirt/images/for_OCP4_on_
SSD/ocp4_master02.qcow2                                                                                                                           
[   0.0] Examining /var/lib/libvirt/images/on_SSD/ocp4_master02.qcow2
**********

Summary of changes:

/dev/sda1: This partition will be left alone.

/dev/sda2: This partition will be left alone.

/dev/sda3: This partition will be left alone.

/dev/sda4: This partition will be resized from 69.5G to 149.5G.  The 
filesystem xfs on /dev/sda4 will be expanded using the ‘xfs_growfs’ 
method.

**********
[   4.0] Setting up initial partition table on /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master02.qcow2
[  18.3] Copying /dev/sda1
[  18.3] Copying /dev/sda2
[  18.4] Copying /dev/sda3
[  19.3] Copying /dev/sda4
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 179.1] Expanding /dev/sda4 using the ‘xfs_growfs’ method

Resize operation completed with no errors.  Before deleting the old disk, 
carefully check that the resized disk boots and works correctly.
root@epyc export # virt-resize --expand /dev/sda4 /var/lib/libvirt/images/on_SSD/ocp4_master03.qcow2 /var/lib/libvirt/images/for_OCP4_on_
SSD/ocp4_master03.qcow2                                                                                                                           
[   0.0] Examining /var/lib/libvirt/images/on_SSD/ocp4_master03.qcow2
**********

Summary of changes:

/dev/sda1: This partition will be left alone.

/dev/sda2: This partition will be left alone.

/dev/sda3: This partition will be left alone.

/dev/sda4: This partition will be resized from 69.5G to 149.5G.  The 
filesystem xfs on /dev/sda4 will be expanded using the ‘xfs_growfs’ 
method.

**********
[   4.2] Setting up initial partition table on /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master03.qcow2
[  18.5] Copying /dev/sda1
[  18.6] Copying /dev/sda2
[  18.8] Copying /dev/sda3
[  19.7] Copying /dev/sda4
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 252.2] Expanding /dev/sda4 using the ‘xfs_growfs’ method

Resize operation completed with no errors.  Before deleting the old disk, 
carefully check that the resized disk boots and works correctly.

Cleanup

Now all that was left to do was editing the 4 VMs' config, to use the new qcow2 location, and clean up the previously used file (click to expand details).
root@epyc for_OCP4_on_SSD # virsh edit --domain ocp4_master01 
Domain ocp4_master01 XML configuration edited.

root@epyc for_OCP4_on_SSD # virsh edit --domain ocp4_master02
Domain ocp4_master02 XML configuration edited.

root@epyc for_OCP4_on_SSD # virsh edit --domain ocp4_master03
Domain ocp4_master03 XML configuration edited.

root@epyc for_OCP4_on_SSD # cd ../on_SSD/
root@epyc on_SSD # mv ocp4_master01.qcow2 old-ocp4_master01.qcow2
root@epyc on_SSD # mv ocp4_master02.qcow2 old-ocp4_master02.qcow2
root@epyc on_SSD # mv ocp4_master03.qcow2 old-ocp4_master03.qcow2

Verify

As expected, the OCP4 nodes now see a larger /sysroot (without any manual intervention in CoreOS).

$ ssh core@master03.ocp4.pcfe.net df -h -x tmpfs -x devtmpfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       150G   25G  125G  17% /sysroot
/dev/sda3       364M  194M  148M  57% /boot
overlay          24G   64M   24G   1% /etc/NetworkManager/systemConnectionsMerged
And the new files are not yet using the full 150G eache (click to expand details).
root@epyc ~ # du -hs /var/lib/libvirt/images/on_SSD/old-ocp4_master0* /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master0*
71G     /var/lib/libvirt/images/on_SSD/old-ocp4_master01.qcow2
71G     /var/lib/libvirt/images/on_SSD/old-ocp4_master02.qcow2
71G     /var/lib/libvirt/images/on_SSD/old-ocp4_master03.qcow2
74G     /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
76G     /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master02.qcow2
79G     /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master03.qcow2
root@epyc ~ # ls -lh /var/lib/libvirt/images/on_SSD/old-ocp4_master0* /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master0*
-rw-------. 1 qemu qemu 151G Aug  8 17:02 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
-rw-------. 1 qemu qemu 151G Aug  8 17:02 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master02.qcow2
-rw-------. 1 qemu qemu 151G Aug  8 17:02 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master03.qcow2
-rw-------. 1 qemu qemu  71G Aug  8 12:37 /var/lib/libvirt/images/on_SSD/old-ocp4_master01.qcow2
-rw-------. 1 qemu qemu  71G Aug  8 12:37 /var/lib/libvirt/images/on_SSD/old-ocp4_master02.qcow2
-rw-------. 1 qemu qemu  71G Aug  8 12:37 /var/lib/libvirt/images/on_SSD/old-ocp4_master03.qcow2

But I will want to double check on discards at some future point.