Half of 960evo recycled from LVM cache for OCP4

This is my braindump of shrinking the existing 1TB LVM cache (on a Samsung 960evo) to half size and using the freed up space to host qcow2 files used by our OpenShift 4 VMs

Why

8 TB spinning rust plus 1TB SSD cache is nice and all, but we need more SSD backed qcow2 files for OCP4.

Note

I did not stop any VMs, whose qcow2 files are on the currently cached LV, why would I? I can live with the performance hit for the short while the LV is uncached and I have no wish to have a downtime on about a dozen VMs.

If the below is too terse for you, then see these previous posts of mine

What

root@epyc ~ # lvs
  LV                                    VG      Attr       LSize   Pool       Origin                                Data%  Meta%  Move Log Cpy%Sync Convert
[...]
  LV_var_lib_libvirt_images_HDD         VG_epyc Cwi-aoC---   8.00t [LV_cache] [LV_var_lib_libvirt_images_HDD_corig] 99.95  1.14            0.00            
[...]

How

Remove Existing Cache

root@epyc ~ # lvremove VG_epyc/LV_cache
  Flushing 0 blocks for cache VG_epyc/LV_var_lib_libvirt_images_HDD.
  Logical volume "LV_cache" successfully removed
root@epyc ~ # pvs -o+tags
  Error reading device /dev/sda1 at 0 length 4.
  Error reading device /dev/sda1 at 4096 length 4.
  PV             VG      Fmt  Attr PSize    PFree    PV Tags   
  /dev/md127     VG_epyc lvm2 a--    14.55t    4.89t hdd       
  /dev/nvme0n1p3 VG_epyc lvm2 a--   952.67g  107.67g 970pro,ssd
  /dev/nvme1n1p1 VG_epyc lvm2 a--  <931.51g <931.51g 960evo,ssd

Create Cache with Half of 960evo

root@epyc ~ # lvcreate -L 1G -n LV_cache_metadata VG_epyc @960evo
  Logical volume "LV_cache_metadata" created.
root@epyc ~ # lvcreate -l 50%PVS -n LV_cache VG_epyc @960evo
  Logical volume "LV_cache" created.

Check Before Assembly

root@epyc ~ # lvdisplay --maps VG_epyc/LV_cache_metadata
  --- Logical volume ---
  LV Path                /dev/VG_epyc/LV_cache_metadata
  LV Name                LV_cache_metadata
  VG Name                VG_epyc
  LV UUID                3pJQ3a-RU0l-3gFV-A28H-M3aq-yQdN-sumFXu
  LV Write Access        read/write
  LV Creation host, time epyc.internal.pcfe.net, 2021-04-24 10:18:18 +0200
  LV Status              available
  # open                 0
  LV Size                1.00 GiB
  Current LE             256
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:6
   
  --- Segments ---
  Logical extents 0 to 255:
    Type                linear
    Physical volume     /dev/nvme1n1p1
    Physical extents    0 to 255
root@epyc ~ # lvdisplay --maps VG_epyc/LV_cache
  --- Logical volume ---
  LV Path                /dev/VG_epyc/LV_cache
  LV Name                LV_cache
  VG Name                VG_epyc
  LV UUID                1bW097-HXZ1-Qk3u-4u16-wReg-kzCQ-Cl965s
  LV Write Access        read/write
  LV Creation host, time epyc.internal.pcfe.net, 2021-04-24 10:18:31 +0200
  LV Status              available
  # open                 0
  LV Size                465.25 GiB
  Current LE             119105
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:7
   
  --- Segments ---
  Logical extents 0 to 119104:
    Type                linear
    Physical volume     /dev/nvme1n1p1
    Physical extents    256 to 119360
root@epyc ~ # lvs
  LV                                    VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
[...]
  LV_cache                              VG_epyc -wi-a----- 465.25g                                                    
  LV_cache_metadata                     VG_epyc -wi-a-----   1.00g                                                    
[...]
  LV_var_lib_libvirt_images_HDD         VG_epyc -wi-ao----   8.00t                                                    

Assemble

Create Cache Pool

root@epyc ~ # lvconvert --type cache-pool --poolmetadata VG_epyc/LV_cache_metadata VG_epyc/LV_cache                                                                                                                                                                                                   
  Using 512.00 KiB chunk size instead of default 64.00 KiB, so cache pool has less than 1000000 chunks.
  WARNING: Converting VG_epyc/LV_cache and VG_epyc/LV_cache_metadata to cache pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert VG_epyc/LV_cache and VG_epyc/LV_cache_metadata? [y/n]: y
  Converted VG_epyc/LV_cache and VG_epyc/LV_cache_metadata to cache pool.

Use Cache Pool for the LV

root@epyc ~ # lvconvert --type cache --cachemode writeback --cachepool VG_epyc/LV_cache VG_epyc/LV_var_lib_libvirt_images_HDD                                                                                                                                                             
Do you want wipe existing metadata of cache pool VG_epyc/LV_cache? [y/n]: y
  Logical volume VG_epyc/LV_var_lib_libvirt_images_HDD is now cached.

Check Cache Usage

Right after it was created, the cache is of course mostly empty

root@epyc ~ # lvs
  LV                                    VG      Attr       LSize   Pool       Origin                                Data%  Meta%  Move Log Cpy%Sync Convert
[...]
  LV_var_lib_libvirt_images_HDD         VG_epyc Cwi-aoC---   8.00t [LV_cache] [LV_var_lib_libvirt_images_HDD_corig] 0.06   0.74            0.18
[...]               
root@epyc ~ # lvdisplay VG_epyc/LV_var_lib_libvirt_images_HDD
  --- Logical volume ---
  LV Path                /dev/VG_epyc/LV_var_lib_libvirt_images_HDD
  LV Name                LV_var_lib_libvirt_images_HDD
  VG Name                VG_epyc
  LV UUID                AbOUd3-Dw2u-jdyL-D4Ff-MjrM-IEy1-qcfycf
  LV Write Access        read/write
  LV Creation host, time epyc.internal.pcfe.net, 2018-08-31 09:44:31 +0200
  LV Cache pool name     LV_cache
  LV Cache origin name   LV_var_lib_libvirt_images_HDD_corig
  LV Status              available
  # open                 1
  LV Size                8.00 TiB
  Cache used blocks      0.06%
  Cache metadata blocks  0.74%
  Cache dirty blocks     17.52%
  Cache read hits/misses 4 / 2
  Cache wrt hits/misses  2759 / 1688
  Cache demotions        0
  Cache promotions       565
  Current LE             2097152
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:9
   
root@epyc ~ # date
Sat Apr 24 10:24:50 CEST 2021

And about 90 minutes later;

root@epyc ~ # date ; lvdisplay VG_epyc/LV_var_lib_libvirt_images_HDD
Sat Apr 24 11:56:58 CEST 2021
  --- Logical volume ---
  LV Path                /dev/VG_epyc/LV_var_lib_libvirt_images_HDD
  LV Name                LV_var_lib_libvirt_images_HDD
  VG Name                VG_epyc
  LV UUID                AbOUd3-Dw2u-jdyL-D4Ff-MjrM-IEy1-qcfycf
  LV Write Access        read/write
  LV Creation host, time epyc.internal.pcfe.net, 2018-08-31 09:44:31 +0200
  LV Cache pool name     LV_cache
  LV Cache origin name   LV_var_lib_libvirt_images_HDD_corig
  LV Status              available
  # open                 1
  LV Size                8.00 TiB
  Cache used blocks      4.73%
  Cache metadata blocks  0.74%
  Cache dirty blocks     0.00%
  Cache read hits/misses 101249 / 205887
  Cache wrt hits/misses  375962 / 495289
  Cache demotions        0
  Cache promotions       45101
  Current LE             2097152
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:9

New LV for OCP4 qcow2 Files

After a normal LVc creation, I now have;

root@epyc ~ # lvs
  LV                                    VG      Attr       LSize   Pool       Origin                                Data%  Meta%  Move Log Cpy%Sync Convert
[...]
  LV_OCP4_qcow2s                        VG_epyc -wi-ao---- 465.25g                                                                                         
[...]                   
root@epyc ~ # pvs -o+tags                                                                                                                         
  Error reading device /dev/sda1 at 0 length 4.
  Error reading device /dev/sda1 at 4096 length 4.
  PV             VG      Fmt  Attr PSize    PFree   PV Tags   
  /dev/md127     VG_epyc lvm2 a--    14.55t   4.89t hdd       
  /dev/nvme0n1p3 VG_epyc lvm2 a--   952.67g 107.67g 970pro,ssd
  /dev/nvme1n1p1 VG_epyc lvm2 a--  <931.51g      0  960evo,ssd
root@epyc ~ # df -h /var/lib/libvirt/images/for_OCP4_on_SSD/
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VG_epyc-LV_OCP4_qcow2s  466G   33M  465G   1% /var/lib/libvirt/images/for_OCP4_on_SSD