Resizing qcow2 Files Used by el8 VMs on a el7 Hypervisor
Table of Contents
This is my braindump of using virt-resize to migrate 3 OpenShift 4 master VMs’ qcow2 disk files, on a CentOS 7 hypervisor,
from 70G each in one libvirt storage pool to 150G each in another pool.
Analyse old qcow2 files
As always when using guestfs-tools, the VMs were shut down before operating on the qcow2 files.
Checking usage of old qcow2 files (click to expand details)
root@epyc ~ # virt-filesystems -a /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2 --all --long -h
Name Type VFS Label MBR Size Parent
/dev/sda1 filesystem unknown - - 1.0M -
/dev/sda2 filesystem vfat EFI-SYSTEM - 127M -
/dev/sda3 filesystem ext4 boot - 384M -
/dev/sda4 filesystem xfs root - 69G -
/dev/sda1 partition - - - 1.0M /dev/sda
/dev/sda2 partition - - - 127M /dev/sda
/dev/sda3 partition - - - 384M /dev/sda
/dev/sda4 partition - - - 69G /dev/sda
/dev/sda device - - - 70G -
showed me that I want to extend /dev/sda4.
Create (sparse) qcow2 Target Files
Virt-resize cannot do in-place disk modifications. You have to have space to store the resized output disk.
To store the resized disk image in a file, create a file of a suitable size:
(from the Manual page virt-resize(1))
I simply used virt-manager to create 3, each 150G large but thin allocated.
I could also have used qemu-img create -f qcow2… (click for details).
qemu-img create -f qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2 150G
qemu-img create -f qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master02.qcow2 150G
qemu-img create -f qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master03.qcow2 150G
chown qemu.qemu /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master0?.qcow2
Failed attempt on CentOS7
My initial attempt failed.
Resizing the xfs does not work out of the box on my CentOS 7 hypervisor (click to expand details).
root@epyc ~ # rpm -qf $(which virt-resize)
libguestfs-tools-c-1.40.2-10.el7.x86_64
root@epyc ~ # virt-resize --expand /dev/sda4 /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_
master01.qcow2
[ 0.0] Examining /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2
**********
Summary of changes:
/dev/sda1: This partition will be left alone.
/dev/sda2: This partition will be left alone.
/dev/sda3: This partition will be left alone.
/dev/sda4: This partition will be resized from 69.5G to 149.5G. The
filesystem xfs on /dev/sda4 will be expanded using the ‘xfs_growfs’
method.
**********
[ 2.8] Setting up initial partition table on /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
[ 16.5] Copying /dev/sda1
[ 16.5] Copying /dev/sda2
[ 16.7] Copying /dev/sda3
[ 17.3] Copying /dev/sda4
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 159.4] Expanding /dev/sda4 using the ‘xfs_growfs’ method
virt-resize: error: libguestfs error: mount: mount exited with status 32:
mount: wrong fs type, bad option, bad superblock on /dev/sda4,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
If reporting bugs, run virt-resize with debugging enabled and include the
complete output:
virt-resize -v -x [...]
root@epyc images # qemu-img info /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2
image: /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2
file format: qcow2
virtual size: 70G (75161927680 bytes)
disk size: 70G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true
root@epyc images # qemu-img info /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
image: /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
file format: qcow2
virtual size: 150G (161061273600 bytes)
disk size: 70G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true
root@epyc images # qemu-img check -r all /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
No errors were found on the image.
2457600/2457600 = 100.00% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 161086111744
And without expanding, using virt-resize makes no sense. That would add a new partition and I'd need to touch the CoreOS that the OCP4 master nodes use (click to expand details).
root@epyc ~ # virt-resize /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.
qcow2
[ 0.0] Examining /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2
**********
Summary of changes:
/dev/sda1: This partition will be left alone.
/dev/sda2: This partition will be left alone.
/dev/sda3: This partition will be left alone.
/dev/sda4: This partition will be left alone.
There is a surplus of 80.0G. An extra partition will be created for the
surplus.
**********
[ 2.8] Setting up initial partition table on /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
[ 16.7] Copying /dev/sda1
[ 16.7] Copying /dev/sda2
[ 16.9] Copying /dev/sda3
[ 17.6] Copying /dev/sda4
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
Resize operation completed with no errors. Before deleting the old disk,
carefully check that the resized disk boots and works correctly.
This is a known issue with an el7 libvirt host attempting to work with el8 images.
Workaround, use Upstream’s Appliance
One can either use an el8 host to do the resize or apply the workaround from access.redhat.com solution Using RHEL 8 virtual disk images on a RHEL 7 host sometimes fails.
See also Bug 1671895 and How can I switch to a fixed / prebuilt appliance? in the libguestfs Frequently Asked Questions (FAQ).
tl;dr: obtain latest appliance from http://download.libguestfs.org/binaries/appliance/, unpack, set LIBGUESTFS_PATH, run virt-resize using the downloaded appliance (click to expand details).
root@epyc ~ # mkdir /export
root@epyc ~ # cd /export
root@epyc export # wget https://download.libguestfs.org/binaries/appliance/appliance-1.40.1.tar.xz
root@epyc export # tar -Jxf appliance-1.40.1.tar.xz
root@epyc export # ls appliance
initrd kernel README.fixed root
root@epyc export # chmod -R a+rwX /export/appliance
root@epyc export # export LIBGUESTFS_PATH=/export/appliance
root@epyc export # virt-resize --expand /dev/sda4 /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
[ 0.0] Examining /var/lib/libvirt/images/on_SSD/ocp4_master01.qcow2
**********
Summary of changes:
/dev/sda1: This partition will be left alone.
/dev/sda2: This partition will be left alone.
/dev/sda3: This partition will be left alone.
/dev/sda4: This partition will be resized from 69.5G to 149.5G. The
filesystem xfs on /dev/sda4 will be expanded using the ‘xfs_growfs’
method.
**********
[ 4.2] Setting up initial partition table on /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
[ 18.6] Copying /dev/sda1
[ 18.7] Copying /dev/sda2
[ 18.9] Copying /dev/sda3
[ 19.4] Copying /dev/sda4
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 155.1] Expanding /dev/sda4 using the ‘xfs_growfs’ method
Resize operation completed with no errors. Before deleting the old disk,
carefully check that the resized disk boots and works correctly.
root@epyc export # virt-resize --expand /dev/sda4 /var/lib/libvirt/images/on_SSD/ocp4_master02.qcow2 /var/lib/libvirt/images/for_OCP4_on_
SSD/ocp4_master02.qcow2
[ 0.0] Examining /var/lib/libvirt/images/on_SSD/ocp4_master02.qcow2
**********
Summary of changes:
/dev/sda1: This partition will be left alone.
/dev/sda2: This partition will be left alone.
/dev/sda3: This partition will be left alone.
/dev/sda4: This partition will be resized from 69.5G to 149.5G. The
filesystem xfs on /dev/sda4 will be expanded using the ‘xfs_growfs’
method.
**********
[ 4.0] Setting up initial partition table on /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master02.qcow2
[ 18.3] Copying /dev/sda1
[ 18.3] Copying /dev/sda2
[ 18.4] Copying /dev/sda3
[ 19.3] Copying /dev/sda4
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 179.1] Expanding /dev/sda4 using the ‘xfs_growfs’ method
Resize operation completed with no errors. Before deleting the old disk,
carefully check that the resized disk boots and works correctly.
root@epyc export # virt-resize --expand /dev/sda4 /var/lib/libvirt/images/on_SSD/ocp4_master03.qcow2 /var/lib/libvirt/images/for_OCP4_on_
SSD/ocp4_master03.qcow2
[ 0.0] Examining /var/lib/libvirt/images/on_SSD/ocp4_master03.qcow2
**********
Summary of changes:
/dev/sda1: This partition will be left alone.
/dev/sda2: This partition will be left alone.
/dev/sda3: This partition will be left alone.
/dev/sda4: This partition will be resized from 69.5G to 149.5G. The
filesystem xfs on /dev/sda4 will be expanded using the ‘xfs_growfs’
method.
**********
[ 4.2] Setting up initial partition table on /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master03.qcow2
[ 18.5] Copying /dev/sda1
[ 18.6] Copying /dev/sda2
[ 18.8] Copying /dev/sda3
[ 19.7] Copying /dev/sda4
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 252.2] Expanding /dev/sda4 using the ‘xfs_growfs’ method
Resize operation completed with no errors. Before deleting the old disk,
carefully check that the resized disk boots and works correctly.
Cleanup
Now all that was left to do was editing the 4 VMs' config, to use the new qcow2 location, and clean up the previously used file (click to expand details).
root@epyc for_OCP4_on_SSD # virsh edit --domain ocp4_master01
Domain ocp4_master01 XML configuration edited.
root@epyc for_OCP4_on_SSD # virsh edit --domain ocp4_master02
Domain ocp4_master02 XML configuration edited.
root@epyc for_OCP4_on_SSD # virsh edit --domain ocp4_master03
Domain ocp4_master03 XML configuration edited.
root@epyc for_OCP4_on_SSD # cd ../on_SSD/
root@epyc on_SSD # mv ocp4_master01.qcow2 old-ocp4_master01.qcow2
root@epyc on_SSD # mv ocp4_master02.qcow2 old-ocp4_master02.qcow2
root@epyc on_SSD # mv ocp4_master03.qcow2 old-ocp4_master03.qcow2
Verify
As expected, the OCP4 nodes now see a larger /sysroot (without any manual intervention in CoreOS).
$ ssh core@master03.ocp4.pcfe.net df -h -x tmpfs -x devtmpfs
Filesystem Size Used Avail Use% Mounted on
/dev/sda4 150G 25G 125G 17% /sysroot
/dev/sda3 364M 194M 148M 57% /boot
overlay 24G 64M 24G 1% /etc/NetworkManager/systemConnectionsMerged
And the new files are not yet using the full 150G eache (click to expand details).
root@epyc ~ # du -hs /var/lib/libvirt/images/on_SSD/old-ocp4_master0* /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master0*
71G /var/lib/libvirt/images/on_SSD/old-ocp4_master01.qcow2
71G /var/lib/libvirt/images/on_SSD/old-ocp4_master02.qcow2
71G /var/lib/libvirt/images/on_SSD/old-ocp4_master03.qcow2
74G /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
76G /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master02.qcow2
79G /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master03.qcow2
root@epyc ~ # ls -lh /var/lib/libvirt/images/on_SSD/old-ocp4_master0* /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master0*
-rw-------. 1 qemu qemu 151G Aug 8 17:02 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master01.qcow2
-rw-------. 1 qemu qemu 151G Aug 8 17:02 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master02.qcow2
-rw-------. 1 qemu qemu 151G Aug 8 17:02 /var/lib/libvirt/images/for_OCP4_on_SSD/ocp4_master03.qcow2
-rw-------. 1 qemu qemu 71G Aug 8 12:37 /var/lib/libvirt/images/on_SSD/old-ocp4_master01.qcow2
-rw-------. 1 qemu qemu 71G Aug 8 12:37 /var/lib/libvirt/images/on_SSD/old-ocp4_master02.qcow2
-rw-------. 1 qemu qemu 71G Aug 8 12:37 /var/lib/libvirt/images/on_SSD/old-ocp4_master03.qcow2
But I will want to double check on discards at some future point.