December 16, 2021

Categories: braindump

QNAP TS-473A Initial Bringup

Table of Contents

I purchased a QNAP TS-473. This comes with a AMD Ryzen Embedded V1500B 4-core/8-thread @ 2.2 GHz CPU.

my QNAP TS-473A, cover removed, showing motherboard and back of PCIe slots

These are my notes from initial bringup.

The chassis has 4x 3.5 inch hot-swap trays (which of course also accept 2.5" drives).

The board (TS x73A MB V2.1) has:

2x M.2 2280 PCIe Gen3 x1 slots
2x PCIe 3.0 x4 slots
2x DDR4 SO-DIMM slots, that accept both ECC and non ECC modules, maximum 2x 32 GB

I’ll only be using the QNAP provided QTS 5.0.0 to verify the hardware works.

This is actually meant to become a node in my Ceph cluster cluster.

Budget Impact

count	descriptioon	total €
1	Samsung SSD 980 NVMe M.2 2280 PCIe 3.0 V-NAND MLC 500GB	58 €
1	Crucial P2 M.2 NVMe 2TB	144 €
1	QNAP QXG-10G2SF-CX4	176 €
1	QNAP TS-473A-8G	806 €
1	ASUS GeForce GT 1030 BRK 2GB	116 €
4	old 4TB HDD, from the shelf	0 €
2	G.Skill F4-3200C22-32GRS, left over from another project	0 €

Which comes up to 1'300 € with recycled RAM and HDDs.

Caveats

It would be nicer if:

the PCIe slots were PCIe 3.0 x8 instead of x4
the NVMe slots were PCIe Gen3 x4 instead of x1
the NVMe slots also accepted cards shorter than 2280
I could enable SecureBoot

But well, this is a consumer NAS, not an enterprise storage node.

FIXME: WTF, that lstopo looks odd, 16 on PCI 09:00.0 for 2 SATA but only 1,0 on PCI 01:00.0 for the GPU?!? Investigate.

lstopo --physical --cpuset --no-legend --of png lstopo-fully-loaded-ts-473a.png

output of

lspci  -vvv | grep -e LnkSta: -e "^0"

(click to expand).

[root@ts-473a-01 ~]# lspci  -vvv | grep -e LnkSta: -e "^0"
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] (prog-if 00 [Normal decode])
		LnkSta:	Speed 2.5GT/s (downgraded), Width x4 (ok)
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] (prog-if 00 [Normal decode])
		LnkSta:	Speed 8GT/s (ok), Width x4 (ok)
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] (prog-if 00 [Normal decode])
		LnkSta:	Speed 8GT/s (ok), Width x1 (ok)
00:01.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] (prog-if 00 [Normal decode])
		LnkSta:	Speed 8GT/s (ok), Width x1 (ok)
00:01.5 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] (prog-if 00 [Normal decode])
		LnkSta:	Speed 5GT/s (downgraded), Width x1 (ok)
00:01.6 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] (prog-if 00 [Normal decode])
		LnkSta:	Speed 5GT/s (downgraded), Width x1 (ok)
00:01.7 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] (prog-if 00 [Normal decode])
		LnkSta:	Speed 8GT/s (ok), Width x1 (downgraded)
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus A (prog-if 00 [Normal decode])
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus B (prog-if 00 [Normal decode])
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 7
01:00.0 VGA compatible controller: NVIDIA Corporation GP108 [GeForce GT 1030] (rev a1) (prog-if 00 [VGA controller])
		LnkSta:	Speed 2.5GT/s (downgraded), Width x4 (ok)
01:00.1 Audio device: NVIDIA Corporation GP108 High Definition Audio Controller (rev a1)
		LnkSta:	Speed 2.5GT/s (downgraded), Width x4 (ok)
02:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
		LnkSta:	Speed 8GT/s (ok), Width x4 (downgraded)
02:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
		LnkSta:	Speed 8GT/s (ok), Width x4 (downgraded)
03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a809 (prog-if 02 [NVM Express])
		LnkSta:	Speed 8GT/s (ok), Width x1 (downgraded)
04:00.0 Non-Volatile memory controller: Micron/Crucial Technology P2 NVMe PCIe SSD (rev 01) (prog-if 02 [NVM Express])
		LnkSta:	Speed 8GT/s (ok), Width x1 (downgraded)
05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
		LnkSta:	Speed 5GT/s (ok), Width x1 (ok)
06:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
		LnkSta:	Speed 5GT/s (ok), Width x1 (ok)
07:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller (prog-if 01 [AHCI 1.0])
		LnkSta:	Speed 8GT/s (ok), Width x1 (ok)
08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
08:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
08:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1 (prog-if 30 [XHCI])
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
08:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1 (prog-if 30 [XHCI])
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
08:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
08:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
08:00.7 Non-VGA unclassified device: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/Renoir Non-Sensor Fusion Hub KMDF driver
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
09:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 61) (prog-if 01 [AHCI 1.0])
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)

Hardware Bought

QNAP TS-473 4-bay NAS
QNAP QXG-10G2SF-CX4 2x 10 GbE SFP+ network card
ASUS GeForce GT 1030 BRK 2.0 GB GPU
8 pack of QNAP HS-M2SSD-01 Heatsink was actually not needed, my NAS came with 2 NVMe heatsinks
500GB Samsung SSD 980 NVMe M.2 2280 PCIe 3.0 V-NAND MLC
2TB Crucial P2 M.2 NVMe

Hardware Recycled

64 GiB RAM, G.Skill F4-3200C22D-64GRS (that is a kit containing 2x 32G SO-DIMMs that each report as F4-3200C22-32GRS), good as new, Danke Schatz.
4x 1TB HDD, used Seagate ST4000VN000, I had on the shelf since decommissioning my old haswell hypervisor.

Notes on Hardware Choices

CPU

4C/8T and the ability to have 64 GiB ECC RAM is what makes this QNAP interesting to me.

PCIe slots

Make my life easy when it comes to modding a NAS case for Ceph use.

The TS-473A User Guide labels the top slot as PCIe slot 1 (PCIe 3.0 x4) and the bottom slot as PCIe slot 2 (PCIe 3.0 x4).

But a small spanner in my plan to remove the GPU after install is that on the 4 disk model of the TS-x73A series one needs to first remove the power supply unit to add/remove PCIe cards. :-(

If I get more of these, I’ll probably go for the 6 tray model since both the TS-673A and the TS-873A do not need the owner to remove the PSU for PCIe card access.

Graphics Card

The GPU was simply the smallest, actively cooled GPU, that I found at the vendor where I bought the TS-473A. AFAIK it has a TDP of 30W and connects as PCIe 3.0 x4 (I do wish ASUS did not give it a 16x connector, luckily the PCIe slots of the TS-473A are open to accommodate this 16x connector).

I would have preferred AMD chipset (because IMO they do a much better open source job) but found no AMD based card, of any performance level, that was only one slot wide.

¯\_(ツ)_/¯

I need graphical output to go over the BIOS settings (which is hopefully in UEFI and SecureBoot mode already) and initial RHEL installation tests.

Network Card

My Ceph’s cluster_network runs on 10 Gigabit/s and I wanted an actively cooled card. QNAP’s card with a NVIDIA Mellanox ConnectX-4 Lx ticks these boxes.

Since normally I happily use Intel NICs, my plan B is a passively cooled Intel X520-DA2.

NVMe

For my Ceph use, I want one small NVMe to hold the OS that runs the Ceph containers (RHEL 8) and one large, cheap, NVMe that will hold the DB devices for my 4 OSDs. I want the NVMe large enough to hold at least 4% of each OSD in the node and have old drives on the shelf I plan to recycle. These range from 1TB to 8TB.

NVMe is a consumable (like toner, paper, etc) to me, I’ll see how long the Crucial P2 lasts, with my pools all being replica 3 I am not really fussed when it dies and takes all 4 OSDs down. I’ll just get a replacement NVMe and live with a reduced total capacity until the replacement is here.

Initial Boot

Initial boot was done after adding a couple of old disk from the shelf and was eventless, apart from taking longer than I expected until QTS was ready to use.

Add-ons

Next, I

upgraded the memory to 64 GiB
put both NVMe in
removed the PSU so I could access the PCIe slots
moved a strand of power cabling to clear the area for the GPU ()
inserted the network card in the bottom slot
inserted the GPU in the upper slot
re-mounted the PSU

All Add-Ons OK

As expected, every component reports in QTS and appears functional.

Poking Around

sudo parted /dev/sdd

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Password: 
GNU Parted 3.1
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
p
Model:  USB DISK MODULE (scsi)
Disk /dev/sdd: 4983MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type      File system  Flags
 1      4096B   5374kB  5370kB  primary   ext2
 2      5374kB  517MB   512MB   primary   ext2
 3      517MB   1030MB  512MB   primary   ext2
 4      1030MB  3964MB  2934MB  extended
 5      1030MB  1038MB  8516kB  logical   ext2
 6      1038MB  1047MB  8909kB  logical   ext2
 7      1047MB  3964MB  2917MB  logical   ext2

(parted)

[foo@ts-473a-01 /]$ cat /proc/mounts |grep sdd
/dev/sdd5 /mnt/boot_config ext2 rw,relatime,errors=continue 0 0
[foo@ts-473a-01 /]$ ls -la /mnt/boot_config
total 14
drwxr-xr-x 4 admin administrators  1024 2021-12-07 22:25 ./
drwxr-xr-x 8 admin administrators   180 2021-12-12 21:06 ../
drwx------ 2 admin administrators 12288 2018-06-27 10:06 lost+found/
drwxr-xr-x 2 admin administrators  1024 2021-12-07 22:25 qlicense/
[foo@ts-473a-01 /]$ ls -la /mnt/boot_config/qlicense/
total 2
drwxr-xr-x 2 admin administrators 1024 2021-12-07 22:25 ./
drwxr-xr-x 4 admin administrators 1024 2021-12-07 22:25 ../
[foo@ts-473a-01 /]$ umount /mnt/boot_config
[foo@ts-473a-01 /]$ df -h                     
Filesystem                Size      Used Available Use% Mounted on
[...]
/dev/mapper/cachedev1
                        294.3G     16.4G    277.4G   6% /share/CACHEDEV1_DATA
[foo@ts-473a-01 /]$ ls /share/CACHEDEV1_DATA
aquota.user  homes/  lost+found/  Public/
[foo@ts-473a-01 /]$ mkdir /share/CACHEDEV1_DATA/foo
[foo@ts-473a-01 /]$ ls -la /share/CACHEDEV1_DATA/foo
total 8
drwxr-xr-x  2 foo    everyone       4096 2021-12-12 21:11 ./
drwxrwxrwx 24 admin  administrators 4096 2021-12-12 21:11 ../
[foo@ts-473a-01 /]$ man dd 
-sh: man: command not found
[foo@ts-473a-01 /]$ dd --help
BusyBox v1.24.1 (2021-12-01 03:08:35 CST) multi-call binary.

Usage: dd [if=FILE] [of=FILE] [ibs=N] [obs=N] [bs=N] [count=N] [skip=N]
	[seek=N] [conv=notrunc|noerror|sync|fsync]
[...]
[foo@ts-473a-01 /]$ sudo dd if=/dev/sdd bs=512 status=progress of=/share/CACHEDEV1_DATA/foo/TS-273A-Flash_Memory_dualboot.img
Password: 
dd: invalid argument 'progress' to 'status'
[foo@ts-473a-01 /]$ sudo dd if=/dev/sdd bs=512 of=/share/CACHEDEV1_DATA/foo/TS-273A-Flash_Memory_dualboot.img
9732096+0 records in
9732096+0 records out
4982833152 bytes (4.6GB) copied, 133.978201 seconds, 35.5MB/s
[foo@ts-473a-01 /]$ cd /share/CACHEDEV1_DATA/foo/
[foo@ts-473a-01 foo]$ sha256sum TS-273A-Flash_Memory_dualboot.img > TS-273A-Flash_Memory_dualboot.img.sha256sum
[foo@ts-473a-01 foo]$ sudo sha256sum /dev/sdd # checksum matched what I got on the previous command

[root@fileserver QNAP]# scp foo@ts-473a-01:/share/CACHEDEV1_DATA/foo/TS-273A-Flash_Memory_dualboot.img .
[...]
[root@fileserver QNAP]# sha256sum -c TS-273A-Flash_Memory_dualboot.img.sha256sum 
TS-273A-Flash_Memory_dualboot.img: OK

Next up

Now that I know nothing was dead on arrival and I saved an image of the QTS boot medium, I’ll be installing RHEL 8 on this TS-473A, but that’s for another post.
Check if https://github.com/guedou/TS-453Be also applies to x73A.

Temporarily reinstalled QTS 5.0.0.1891 to backup the binaries and libs. Saved to workstation

pcfe@workstation Desktop $ pwd
/home/pcfe/Desktop
pcfe@workstation Desktop $ ls -lh TS-473A-QTS-bits/
total 115M
-rw-r--r--. 1 pcfe pcfe 1,7M Jan 22 15:23 144856_2022-01-22_TS-473A_dump.tgz
-rw-r--r--. 1 pcfe pcfe 3,8K Jan 22 15:23 model_Q07D0_Q07N0_10_10.conf
-rw-r--r--. 1 pcfe pcfe  298 Jan 22 15:23 overall
-rw-r--r--. 1 pcfe pcfe 4,5M Jan 22 15:23 QTS-5.0.0.1891-bin.tar
-rw-r--r--. 1 pcfe pcfe  80M Jan 22 15:23 QTS-5.0.0.1891-lib.tar
-rw-r--r--. 1 pcfe pcfe  29M Jan 22 15:23 QTS-5.0.0.1891-sbin.tar
pcfe@workstation Desktop $

Update 2021-12-18

After an initial RHEL8 install from USB stick went fine, some closing notes before I really do start a new post for RHEL8

There is no option to PXE boot.
RHEL 8.5’s anaconda was not too happy when probing disks and finding what QTS had set up. This was dealt with by booting into rescue mode, not mounting any disks, and running wipefs --all --force /dev/… against HDD bays 1 through 4 and both NVMe (but not the 5GB built-in USB used by QTS).

slot	by-path	note
NVMe slot 1	`/dev/disk/by-path/pci-0000:03:00.0-nvme-1`	the top slot
NVMe slot 2	`/dev/disk/by-path/pci-0000:04:00.0-nvme-1`	the bottom slot
HDD bay 1	`/dev/disk/by-path/pci-0000:07:00.0-ata-1`	bay furthest away from the power button
HDD bay 2	`/dev/disk/by-path/pci-0000:07:00.0-ata-2`
HDD bay 3	`/dev/disk/by-path/pci-0000:09:00.0-ata-1`
HDD bay 4	`/dev/disk/by-path/pci-0000:09:00.0-ata-2`	bay closest to the power button

I attached a heatsink to each NVMe’s controller

my QNAP TS-473A, cover removed, showing the motherboard and thus the NVMes with their heatsinks mounted

Update 2023-03-11

I just stumbled across https://www.reddit.com/r/qnap/comments/11lfqgn/serial_console_and_truenas_on_qnap_ts473a/. not sure yet if I’ll bother taking it all apart (PSU needs to be removed to remove/add PCIe cards) unless I see thermal issues. Wish I’d known from the start that there’s serial on the motherboard, should have looked more carefully.