Categories: braindump

Dell T7910 with Proxmox VE, HDD to SSD

Table of Contents

Since I mostly do LVM and sometimes btrfs, here’s a short braindump of my first ZFS on Linux rodeo.

Why

With a recent storage upgrade to my Ceph cluster, 4x 500GB SATA SSD became available. As my Proxmox VE playground still has 3x 500 GB SATA HDD, it made sense to re-use the SATA SSDs there.

Required reading

tl;dr

  1. Add SATA SSD to free HDD carrier
  2. Verify in dmesg -T that the drive is noticed
  3. Copy parttable from a known good disk with sgdisk <healthy bootable device> -R <new device>
  4. Issue a new GUID as we want no duplicates with sgdisk -G <new device>
  5. Find name of a HDD to replace with zpool status -v
  6. Find name of new SSD with ls -l /dev/disk/by-id/
  7. Initiate transfer with zpool replace rpool ata-SAMSUNG_HE502IJ_[REDACTED]-part3 ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3
  8. Set up the new SSD to be bootable with proxmox-boot-tool format /dev/sdd2 and proxmox-boot-tool init /dev/sdd2
  9. Wait for resilver to complete, checking status with zpool status -v from time to time
  10. Physically remove the HDD
  11. repeat

In detail

bash history from replacement of first HDD by an SSD

dmesg -T | tail
lsblk
sgdisk /dev/sda  -R /dev/sdd
sgdisk -G /dev/sdd
proxmox-boot-tool format /dev/sdd2 --force
proxmox-boot-tool init /dev/sdd2
zpool status -v
ls -l /dev/disk/by-id/
zpool replace rpool ata-SAMSUNG_HE502IJ_[REDACTED]-part3 ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3
date ; zpool status -v
date ; zpool status -v
date ; zpool status -v

NOTE: proxmox-boot-tool format required --force because while I had not fully zeroed out the SSD, only wipefs -af but PVE still detected that it was a luks before (it was used in a crypted at rest Ceph cluster)

NOTE: While the Proxmox VE docs use -f with zpool replace, I did not need to force that part.

State while resilvering

root@t7910:~# zpool status -v
  pool: r1nvme
 state: ONLINE
  scan: scrub repaired 0B in 00:02:46 with 0 errors on Sun Mar 12 00:26:47 2023
config:

        NAME                                             STATE     READ WRITE CKSUM
        r1nvme                                           ONLINE       0     0     0
          mirror-0                                       ONLINE       0     0     0
            nvme-KINGSTON_SA2000M8500G_[REDACTED]-part2  ONLINE       0     0     0
            nvme-KINGSTON_SA2000M8500G_[REDACTED]-part2  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Mar 15 16:18:15 2023
        776G scanned at 250M/s, 523G issued at 169M/s, 776G total
        138G resilvered, 67.42% done, 00:25:34 to go
config:

        NAME                                                  STATE     READ WRITE CKSUM
        rpool                                                 ONLINE       0     0     0
          raidz1-0                                            ONLINE       0     0     0
            replacing-0                                       ONLINE       0     0     0
              ata-SAMSUNG_HE502IJ_[REDACTED]-part3            ONLINE       0     0     0
              ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3  ONLINE       0     0     0  (resilvering)
            ata-SAMSUNG_HE502IJ_[REDACTED]-part3              ONLINE       0     0     0
            ata-SAMSUNG_HE502IJ_[REDACTED]-part3              ONLINE       0     0     0
        logs
          mirror-1                                            ONLINE       0     0     0
            nvme-KINGSTON_SA2000M8500G_[REDACTED]-part1       ONLINE       0     0     0
            nvme-KINGSTON_SA2000M8500G_[REDACTED]-part1       ONLINE       0     0     0

errors: No known data errors

rinse, wash, repeat

After resilvering finished, the HDD was physicallyt removed. Repeat the process for the remaining 2 HDDs

second HDD

sgdisk /dev/sdb -R /dev/sda
sgdisk -G /dev/sda
proxmox-boot-tool format /dev/sda2 --force
proxmox-boot-tool init /dev/sda2
zpool status -v
ls -l /dev/disk/by-id
zpool replace rpool ata-SAMSUNG_HE502IJ_[REDACTED]-part3 ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3
date ; zpool status -v
date ; zpool status -v

third HDD

in dmesg

[Sat Mar 18 13:12:24 2023] scsi 0:0:1:0: Direct-Access     ATA      Samsung SSD 860  4B6Q PQ: 0 ANSI: 6
[Sat Mar 18 13:12:24 2023] sd 0:0:1:0: Attached scsi generic sg1 type 0
[Sat Mar 18 13:12:24 2023] sd 0:0:1:0: [sdb] 976773168 512-byte logical blocks: (500 GB/466 GiB)
[Sat Mar 18 13:12:24 2023] sd 0:0:1:0: [sdb] Write Protect is off
[Sat Mar 18 13:12:24 2023] sd 0:0:1:0: [sdb] Mode Sense: 9b 00 10 08
[Sat Mar 18 13:12:24 2023] sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Sat Mar 18 13:12:24 2023] sd 0:0:1:0: [sdb] Attached SCSI disk

sgdisk --zap /dev/sdb
wipefs -af /dev/sdb
sgdisk /dev/sda -R /dev/sdb
sgdisk -G /dev/sdb
lsblk
ls -l /dev/disk/by-id|less
proxmox-boot-tool format /dev/sdb2 --force
proxmox-boot-tool init /dev/sdb2
zpool status -v
ls -l /dev/disk/by-id
zpool replace rpool ata-SAMSUNG_HE502IJ_[REDACTED]-part3 ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3

progressing nicely

root@t7910:~# date ; zpool status -v
Sat 18 Mar 2023 02:08:15 PM CET
  pool: r1nvme
 state: ONLINE
  scan: scrub repaired 0B in 00:02:46 with 0 errors on Sun Mar 12 00:26:47 2023
config:

	NAME                                             STATE     READ WRITE CKSUM
	r1nvme                                           ONLINE       0     0     0
	  mirror-0                                       ONLINE       0     0     0
	    nvme-KINGSTON_SA2000M8500G_[REDACTED]-part2  ONLINE       0     0     0
	    nvme-KINGSTON_SA2000M8500G_[REDACTED]-part2  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Mar 18 13:25:15 2023
	784G scanned at 311M/s, 388G issued at 154M/s, 785G total
	101G resilvered, 49.42% done, 00:44:00 to go
config:

	NAME                                                 STATE     READ WRITE CKSUM
	rpool                                                ONLINE       0     0     0
	  raidz1-0                                           ONLINE       0     0     0
	    ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3   ONLINE       0     0     0
	    ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3   ONLINE       0     0     0
	    replacing-2                                      ONLINE       0     0     0
	      ata-SAMSUNG_HE502IJ_[REDACTED]-part3           ONLINE       0     0     0
	      ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3 ONLINE       0     0     0  (resilvering)
	logs	
	  mirror-1                                           ONLINE       0     0     0
	    nvme-KINGSTON_SA2000M8500G_[REDACTED]-part1      ONLINE       0     0     0
	    nvme-KINGSTON_SA2000M8500G_[REDACTED]-part1      ONLINE       0     0     0

errors: No known data errors

Once it had finished, physically removed the last of the 3 HDDs and cleaned up.

root@t7910:~# date ; zpool status -v
Sat 18 Mar 2023 03:21:23 PM CET
  pool: r1nvme
 state: ONLINE
  scan: scrub repaired 0B in 00:02:46 with 0 errors on Sun Mar 12 00:26:47 2023
config:

	NAME                                             STATE     READ WRITE CKSUM
	r1nvme                                           ONLINE       0     0     0
	  mirror-0                                       ONLINE       0     0     0
	    nvme-KINGSTON_SA2000M8500G_[REDACTED]-part2  ONLINE       0     0     0
	    nvme-KINGSTON_SA2000M8500G_[REDACTED]-part2  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: resilvered 206G in 01:29:43 with 0 errors on Sat Mar 18 14:54:58 2023
config:

	NAME                                               STATE     READ WRITE CKSUM
	rpool                                              ONLINE       0     0     0
	  raidz1-0                                         ONLINE       0     0     0
	    ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3 ONLINE       0     0     0
	    ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3 ONLINE       0     0     0
	    ata-Samsung_SSD_860_EVO_500GB_[REDACTED]-part3 ONLINE       0     0     0
	logs	
	  mirror-1                                         ONLINE       0     0     0
	    nvme-KINGSTON_SA2000M8500G_[REDACTED]-part1    ONLINE       0     0     0
	    nvme-KINGSTON_SA2000M8500G_[REDACTED]-part1    ONLINE       0     0     0

errors: No known data errors

root@t7910:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
3FBA-2766 is configured with: uefi (versions: 5.13.19-6-pve, 5.15.83-1-pve, 5.15.85-1-pve)
WARN: /dev/disk/by-uuid/BDB4-FA0A does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
WARN: /dev/disk/by-uuid/BDB6-355E does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
WARN: /dev/disk/by-uuid/BDB7-5EF9 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
D2B8-226C is configured with: uefi (versions: 5.13.19-6-pve, 5.15.83-1-pve, 5.15.85-1-pve)
FE1D-D9DE is configured with: uefi (versions: 5.13.19-6-pve, 5.15.83-1-pve, 5.15.85-1-pve)

root@t7910:~# proxmox-boot-tool clean --dry-run
Checking whether ESP '3FBA-2766' exists.. Found!
Checking whether ESP 'BDB4-FA0A' exists.. Not found!
Checking whether ESP 'BDB6-355E' exists.. Not found!
Checking whether ESP 'BDB7-5EF9' exists.. Not found!
Checking whether ESP 'D2B8-226C' exists.. Found!
Checking whether ESP 'FE1D-D9DE' exists.. Found!
Sorting and removing duplicate ESPs..
root@t7910:~# proxmox-boot-tool clean
Checking whether ESP '3FBA-2766' exists.. Found!
Checking whether ESP 'BDB4-FA0A' exists.. Not found!
Checking whether ESP 'BDB6-355E' exists.. Not found!
Checking whether ESP 'BDB7-5EF9' exists.. Not found!
Checking whether ESP 'D2B8-226C' exists.. Found!
Checking whether ESP 'FE1D-D9DE' exists.. Found!
Sorting and removing duplicate ESPs..
root@t7910:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
3FBA-2766 is configured with: uefi (versions: 5.13.19-6-pve, 5.15.83-1-pve, 5.15.85-1-pve)
D2B8-226C is configured with: uefi (versions: 5.13.19-6-pve, 5.15.83-1-pve, 5.15.85-1-pve)
FE1D-D9DE is configured with: uefi (versions: 5.13.19-6-pve, 5.15.83-1-pve, 5.15.85-1-pve)
root@t7910:~#

Lastly, rename the pool in PVE

While purely cosmetic, I also wanted to adjust the name in the ProxmoxVE interface.

I came across https://www.reddit.com/r/Proxmox/comments/hndzcn/how_to_rename_storage_spaces/.

So I cheated and did as follows

  1. move all VM storage from local-zfs-hdd to ceph-rbd-external
  2. edit /etc/pve/storage.cfg to change the name from local-zfs-hdd to local-zfs-sata
  3. reboot
  4. move VM storage back from ceph-rbd-external to local-zfs-sata

Mainly because this was also a good excuse to give both the Ceph and the ZFS storage, now that they are both migrated from SATA HDDs to SATA SSDs, a load test. But also because I could not be asked to edit all VM configs. A reboot test was anyway planned as I had replaced 3 devices I boot from.

root@t7910:~# vi  /etc/pve/storage.cfg
root@t7910:~# reboot
Connection to t7910.mgmt.pcfe.net closed by remote host.
Connection to t7910.mgmt.pcfe.net closed.
pcfe@t3600 ~ $ ssh -l root t7910.mgmt.pcfe.net
Linux t7910 5.15.102-1-pve #1 SMP PVE 5.15.102-1 (2023-03-14T13:48Z) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Mar 18 17:28:09 2023 from 192.168.50.59
root@t7910:~# cat /etc/pve/storage.cfg
[…]
zfspool: local-zfs-sata
	pool rpool/data
	content rootdir,images
	mountpoint /rpool/data
	sparse 1

root@t7910:~# pvesm status
Name                     Type     Status           Total            Used       Available        %
ceph-rbd-external         rbd     active      6398450789       255838821      6142611968    4.00%
cephfs-external        cephfs     active      6470918144       328306688      6142611456    5.07%
local                     dir     active       768343424         8776448       759566976    1.14%
local-zfs-nvme        zfspool     active       455213056       205647444       249565612   45.18%
local-zfs-sata        zfspool     active       916273542       156706523       759567019   17.10%

Related