Half of 960evo recycled from LVM cache for OCP4
Table of Contents
This is my braindump of shrinking the existing 1TB LVM cache (on a Samsung 960evo) to half size and using the freed up space to host qcow2 files used by our OpenShift 4 VMs
Why
8 TB spinning rust plus 1TB SSD cache is nice and all, but we need more SSD backed qcow2 files for OCP4.
Note
I did not stop any VMs, whose qcow2 files are on the currently cached LV, why would I? I can live with the performance hit for the short while the LV is uncached and I have no wish to have a downtime on about a dozen VMs.
If the below is too terse for you, then see these previous posts of mine
What
root@epyc ~ # lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
[...]
LV_var_lib_libvirt_images_HDD VG_epyc Cwi-aoC--- 8.00t [LV_cache] [LV_var_lib_libvirt_images_HDD_corig] 99.95 1.14 0.00
[...]
How
Remove Existing Cache
root@epyc ~ # lvremove VG_epyc/LV_cache
Flushing 0 blocks for cache VG_epyc/LV_var_lib_libvirt_images_HDD.
Logical volume "LV_cache" successfully removed
root@epyc ~ # pvs -o+tags
Error reading device /dev/sda1 at 0 length 4.
Error reading device /dev/sda1 at 4096 length 4.
PV VG Fmt Attr PSize PFree PV Tags
/dev/md127 VG_epyc lvm2 a-- 14.55t 4.89t hdd
/dev/nvme0n1p3 VG_epyc lvm2 a-- 952.67g 107.67g 970pro,ssd
/dev/nvme1n1p1 VG_epyc lvm2 a-- <931.51g <931.51g 960evo,ssd
Create Cache with Half of 960evo
root@epyc ~ # lvcreate -L 1G -n LV_cache_metadata VG_epyc @960evo
Logical volume "LV_cache_metadata" created.
root@epyc ~ # lvcreate -l 50%PVS -n LV_cache VG_epyc @960evo
Logical volume "LV_cache" created.
Check Before Assembly
root@epyc ~ # lvdisplay --maps VG_epyc/LV_cache_metadata
--- Logical volume ---
LV Path /dev/VG_epyc/LV_cache_metadata
LV Name LV_cache_metadata
VG Name VG_epyc
LV UUID 3pJQ3a-RU0l-3gFV-A28H-M3aq-yQdN-sumFXu
LV Write Access read/write
LV Creation host, time epyc.internal.pcfe.net, 2021-04-24 10:18:18 +0200
LV Status available
# open 0
LV Size 1.00 GiB
Current LE 256
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:6
--- Segments ---
Logical extents 0 to 255:
Type linear
Physical volume /dev/nvme1n1p1
Physical extents 0 to 255
root@epyc ~ # lvdisplay --maps VG_epyc/LV_cache
--- Logical volume ---
LV Path /dev/VG_epyc/LV_cache
LV Name LV_cache
VG Name VG_epyc
LV UUID 1bW097-HXZ1-Qk3u-4u16-wReg-kzCQ-Cl965s
LV Write Access read/write
LV Creation host, time epyc.internal.pcfe.net, 2021-04-24 10:18:31 +0200
LV Status available
# open 0
LV Size 465.25 GiB
Current LE 119105
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:7
--- Segments ---
Logical extents 0 to 119104:
Type linear
Physical volume /dev/nvme1n1p1
Physical extents 256 to 119360
root@epyc ~ # lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
[...]
LV_cache VG_epyc -wi-a----- 465.25g
LV_cache_metadata VG_epyc -wi-a----- 1.00g
[...]
LV_var_lib_libvirt_images_HDD VG_epyc -wi-ao---- 8.00t
Assemble
Create Cache Pool
root@epyc ~ # lvconvert --type cache-pool --poolmetadata VG_epyc/LV_cache_metadata VG_epyc/LV_cache
Using 512.00 KiB chunk size instead of default 64.00 KiB, so cache pool has less than 1000000 chunks.
WARNING: Converting VG_epyc/LV_cache and VG_epyc/LV_cache_metadata to cache pool's data and metadata volumes with metadata wiping.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert VG_epyc/LV_cache and VG_epyc/LV_cache_metadata? [y/n]: y
Converted VG_epyc/LV_cache and VG_epyc/LV_cache_metadata to cache pool.
Use Cache Pool for the LV
root@epyc ~ # lvconvert --type cache --cachemode writeback --cachepool VG_epyc/LV_cache VG_epyc/LV_var_lib_libvirt_images_HDD
Do you want wipe existing metadata of cache pool VG_epyc/LV_cache? [y/n]: y
Logical volume VG_epyc/LV_var_lib_libvirt_images_HDD is now cached.
Check Cache Usage
Right after it was created, the cache is of course mostly empty
root@epyc ~ # lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
[...]
LV_var_lib_libvirt_images_HDD VG_epyc Cwi-aoC--- 8.00t [LV_cache] [LV_var_lib_libvirt_images_HDD_corig] 0.06 0.74 0.18
[...]
root@epyc ~ # lvdisplay VG_epyc/LV_var_lib_libvirt_images_HDD
--- Logical volume ---
LV Path /dev/VG_epyc/LV_var_lib_libvirt_images_HDD
LV Name LV_var_lib_libvirt_images_HDD
VG Name VG_epyc
LV UUID AbOUd3-Dw2u-jdyL-D4Ff-MjrM-IEy1-qcfycf
LV Write Access read/write
LV Creation host, time epyc.internal.pcfe.net, 2018-08-31 09:44:31 +0200
LV Cache pool name LV_cache
LV Cache origin name LV_var_lib_libvirt_images_HDD_corig
LV Status available
# open 1
LV Size 8.00 TiB
Cache used blocks 0.06%
Cache metadata blocks 0.74%
Cache dirty blocks 17.52%
Cache read hits/misses 4 / 2
Cache wrt hits/misses 2759 / 1688
Cache demotions 0
Cache promotions 565
Current LE 2097152
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:9
root@epyc ~ # date
Sat Apr 24 10:24:50 CEST 2021
And about 90 minutes later;
root@epyc ~ # date ; lvdisplay VG_epyc/LV_var_lib_libvirt_images_HDD
Sat Apr 24 11:56:58 CEST 2021
--- Logical volume ---
LV Path /dev/VG_epyc/LV_var_lib_libvirt_images_HDD
LV Name LV_var_lib_libvirt_images_HDD
VG Name VG_epyc
LV UUID AbOUd3-Dw2u-jdyL-D4Ff-MjrM-IEy1-qcfycf
LV Write Access read/write
LV Creation host, time epyc.internal.pcfe.net, 2018-08-31 09:44:31 +0200
LV Cache pool name LV_cache
LV Cache origin name LV_var_lib_libvirt_images_HDD_corig
LV Status available
# open 1
LV Size 8.00 TiB
Cache used blocks 4.73%
Cache metadata blocks 0.74%
Cache dirty blocks 0.00%
Cache read hits/misses 101249 / 205887
Cache wrt hits/misses 375962 / 495289
Cache demotions 0
Cache promotions 45101
Current LE 2097152
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:9
New LV for OCP4 qcow2 Files
After a normal LVc creation, I now have;
root@epyc ~ # lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
[...]
LV_OCP4_qcow2s VG_epyc -wi-ao---- 465.25g
[...]
root@epyc ~ # pvs -o+tags
Error reading device /dev/sda1 at 0 length 4.
Error reading device /dev/sda1 at 4096 length 4.
PV VG Fmt Attr PSize PFree PV Tags
/dev/md127 VG_epyc lvm2 a-- 14.55t 4.89t hdd
/dev/nvme0n1p3 VG_epyc lvm2 a-- 952.67g 107.67g 970pro,ssd
/dev/nvme1n1p1 VG_epyc lvm2 a-- <931.51g 0 960evo,ssd
root@epyc ~ # df -h /var/lib/libvirt/images/for_OCP4_on_SSD/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VG_epyc-LV_OCP4_qcow2s 466G 33M 465G 1% /var/lib/libvirt/images/for_OCP4_on_SSD