QNAP TS-473A with CentOS Stream 9

Table of Contents

In happy distro hopping, I installed CentOS Stream 9 on my QNAP TS-473A.

These are my installation notes. They are similar to my RHEL8 notes and my Fedora Server notes.

Used Hardware

I assembled the following:

  • QNAP TS-473 4-bay NAS with AMD Ryzen Embedded V1500B 4-core/8-thread @ 2.2 GHz CPU
  • QNAP QXG-10G2SF-CX4 2x 10 GbE SFP+ network card
  • ASUS GeForce GT 1030 BRK 2.0 GB GPU
  • 500GB Samsung SSD 980 NVMe M.2 2280 PCIe 3.0 V-NAND MLC
  • 2TB Crucial P2 M.2 NVMe
  • 64 GiB RAM, G.Skill F4-3200C22D-64GRS (that is a kit containing 2x 32G SO-DIMMs that each report as F4-3200C22-32GRS)
  • 4x 1TB HDD, for now, these might be replaced later with something larger

Firmware Settings

  1. ensure you have added a GPU to the TS-x73A
  2. connect screen and keyboard
  3. enter firmware setup by pressing Del or Esc during power on self test (POST)
  4. Boot / Quiet Boot: Disabled (simply so I get shown on screen which key to press during POST to enter UEFI)
  5. Boot / Boot Option Priorities: as you see fit. I disabled USB DISK MODULE PMAP and reordered the others to my liking.
  6. Save & Exit: Save Changes and Exit

Note that if you ever want to return to QTS, you must re-enable the USB DISK MODULE PMAP to be able to successfully boot from it by selecting it at Save & Exit / Boot Override.

Firmware Details

As of 2021-12-18 I have Aptio Setup Utility Version 2.20.1274:

description value
BIOS Vendor American Megatrends
Core Version 5.14
Compliancy UEFI 2.7; PI 1.6
Project Version Q07DAR12
Build Date and Time 05/03/2021 10:59:15
Total Memory Total Memory 65536 MB (DDR4)
Memory Frequency 2400 MHz
EC Version Q07DE008

Kickstart Install of CentOS Stream 9

Much like previous installs, I started my kickstart from a previous linux install.

While the QNAP TS-473A boots from a Fedora Server USB stick just fine, like it does from a RHEL stick, and one can instyyall interactively just fine, I prefer to automatically install with kickstart. While I could just modify the boot entry when starting from a stick, I find it easier to simply put the kernel and initrd from Fedora Everything onto the QNAP’s /boot/ partition and add a custom menu entry to grub.

While I do this with Ansible and my local Fedora Everything mirro, any method is fine. The Ansible tasks should be self explanatory. Create the file, run grub2-mkconfig -o …, reboot, choose the kickstart target.

    - name: "GRUB | ensure initrd for CentOS Stream 9 kickstart is present"
        url: "http://fileserver.internal.pcfe.net/ftp/distributions/CentOS/9-stream/DVD/x86_64/images/pxeboot/initrd.img"
        dest: "/boot/initrd-kickstart-cos9.img"
        mode: "0600"
    - name: "GRUB | ensure kernel for CentOS Stream 9 kickstart is present"
        url: "http://fileserver.internal.pcfe.net/ftp/distributions/CentOS/9-stream/DVD/x86_64/images/pxeboot/vmlinuz"
        dest: "/boot/vmlinuz-kickstart-cos9"
        mode: "0755"
    - name: "GRUB | ensure kickstarting CentOS Stream 9 entry is present"
        dest: "/etc/grub.d/12_cos9_kickstart"
        owner: "root"
        group: "root"
        mode: 0755
        content: |
          exec tail -n +3 $0
          # This file provides an easy way to add custom menu entries.  Simply type the
          # menu entries you want to add after this comment.  Be careful not to change
          # the 'exec tail' line above.
          menuentry "WARNING Kickstart this box with CentOS Stream 9 as a TS-473A ceph node WARNING" {
              linuxefi /vmlinuz-kickstart-cos9 inst.kdump_addon=on ip=enp6s0:dhcp inst.repo=http://fileserver.internal.pcfe.net/ftp/distributions/CentOS/9-stream/DVD/x86_64/ inst.ks=http://fileserver.internal.pcfe.net/ftp/kickstart/CentOSstream9-x86_64-QNAP-TS-473A-ks.cfg
              initrdefi /initrd-kickstart-cos9.img
      notify: grub2-mkconfig | run
My Kickstart file F35-QNAP-TS-473A-ks.cfg (click the triangle to expand)
# Generated by Anaconda 35.22.2
# Generated by pykickstart v3.34
# changed by pcfe, 2022-01-29

# avoid using half arsed names like sda, sdb, etc
# TS-473A User Guide, page 10, says
#   top is M.2 SSD slot 1
# lower is M.2 SSD slot 2
# Disks bays are numbered starting from 1, bay furthest away from the power button.
# for PCIe slots, the user guide says top is slot 1, bottom is slot 2
# NVMe slot 1 /dev/disk/by-path/pci-0000:03:00.0-nvme-1 (the top slot, contains a Samsung 980 500GB)
# NVMe slot 2 /dev/disk/by-path/pci-0000:04:00.0-nvme-1 (the bottom slot, contains a Crucial P2 2TB)
# HDD bay 1   /dev/disk/by-path/pci-0000:07:00.0-ata-1  (bay furthest away from the power button)
# HDD bay 2   /dev/disk/by-path/pci-0000:07:00.0-ata-2
# HDD bay 3   /dev/disk/by-path/pci-0000:09:00.0-ata-1
# HDD bay 4   /dev/disk/by-path/pci-0000:09:00.0-ata-2  (bay closest to the power button)

# reboot after installation is complete?

# Use graphical install

# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'

# System language
lang en_US.UTF-8 --addsupport=de_DE.UTF-8,de_LU.UTF-8,en_DK.UTF-8,en_GB.UTF-8,en_IE.UTF-8,fr_FR.UTF-8,fr_LU.UTF-8

# Network information
# all switch ports have the respective VLAN as native
# 2.5 Gig on-board 1 ('access' network)
network  --bootproto=dhcp --device=enp6s0                --ipv6=auto --activate
# 2.5 Gig on-board 2 (will go on 'storage' via ansible)
network  --bootproto=dhcp --device=enp5s0   --onboot=off --ipv6=auto --no-activate
# 10 Gig on PCIe (will go on 'ceph' via ansible)
network  --bootproto=dhcp --device=enp2s0f0np0 --onboot=off --ipv6=auto --no-activate
# 10 Gig on PCIe slot 2 (PCIe 3.0 x4), currently unused
network  --bootproto=dhcp --device=enp2s0f1np0 --onboot=off --ipv6=auto --no-activate

# Use network installation
url --url="http://fileserver.internal.pcfe.net/ftp/distributions/CentOS/9-stream/DVD/x86_64"
# The AppStream repo
repo --name="AppStream" --baseurl="http://fileserver.internal.pcfe.net/ftp/distributions/CentOS/9-stream/DVD/x86_64/AppStream"

# Package groups to install
# see https://docs.fedoraproject.org/en-US/fedora/f35/install-guide/appendixes/Kickstart_Syntax_Reference/#sect-kickstart-packages
# For Ceph use, '@^server-product-environment' should be enough. The Ceph installer pulls in what is needed.
# For general Fedora Server use, I also had '@container-management' and '@domain-client'.


# Run the Setup Agent on first boot
firstboot --enable

# we only install to the 500GB Samsung NVMe, that is in _M.2 SSD slot 1_, the top slot.
ignoredisk --only-use=/dev/disk/by-path/pci-0000:03:00.0-nvme-1

# Partition clearing information
# note that  OS goes on a small portion os the device in bay 1, the rest will be allocated to Ceph in a separtate VG.
# so kickstarting with the below clearpart line will nuke the Ceph bits on SSD !!!
clearpart --all --initlabel --drives=/dev/disk/by-path/pci-0000:03:00.0-nvme-1

# Disk partitioning information
# the 500GB Samsung NVMe in slot 1 will be fully used for the OS
# the   2TB Crucial NVMe in slot 2 and the HDDs in slots 1 through 4
# will be fed to ceph-ansible as devices
# c.f. https://docs.ceph.com/ceph-ansible/master/osds/scenarios.html
part /boot        --fstype="ext4"  --ondisk=/dev/disk/by-path/pci-0000:03:00.0-nvme-1 --size=1024
part /boot/efi    --fstype="efi"   --ondisk=/dev/disk/by-path/pci-0000:03:00.0-nvme-1 --size=200    --fsoptions="umask=0077,shortname=winnt"
part pv.01        --fstype="lvmpv" --ondisk=/dev/disk/by-path/pci-0000:03:00.0-nvme-1 --size=18500  --grow
volgroup VG_OS    --pesize=4096 pv.01
logvol /                   --fstype="xfs"  --size=4096  --name=LV_root       --vgname=VG_OS
logvol swap                --fstype="swap" --size=1024  --name=LV_swap       --vgname=VG_OS
logvol /var                --fstype="xfs"  --size=2048  --name=LV_var        --vgname=VG_OS
logvol /var/crash          --fstype="xfs"  --size=1024  --name=LV_var_crash  --vgname=VG_OS
logvol /var/log            --fstype="xfs"  --size=2048  --name=LV_var_log    --vgname=VG_OS
logvol /var/lib/ceph       --fstype="xfs"  --size=1024  --name=LV_cephd_data --vgname=VG_OS
logvol /var/lib/containers --fstype="xfs"  --size=4096  --name=LV_containers --vgname=VG_OS
logvol /home               --fstype="xfs"  --size=1024  --name=LV_home       --vgname=VG_OS

timesource --ntp-server=epyc.internal.pcfe.net
timesource --ntp-server=edgerouter-6p.internal.pcfe.net
# System timezone
timezone Europe/Berlin --utc

# Root password
rootpw --iscrypted $6$tojQNyCl8NsOXJ0h$YiKGtd1StvpOxVoAFtGGwFv9.ArUAuZ4v0J1KXfi2pSNji9Wm.Q.B4C8rANM5i/zvWSegIVz8oduzccVnslU/.

# Ansible user
user --uid=1100 --gid=1100 --name=ansible --lock --gecos="Ansible User"

# pcfe user
user --uid=1000 --gid=1000 --groups=wheel --name=pcfe --password=$y$j9T$ZWDidv6BLl.N4DxKVv0aY1$ct5WbCcT5e/hVBlW0u/mqCDyWwRPB6B5/jWGGPtCPF4 --iscrypted --gecos="Patrick C. F. Ernzer"

# no kdump
%addon com_redhat_kdump --disable

%post --log=/root/ks-post.log
# dump pcfe's ssh key to the root user
# obviously change this to your own pubkey unless you want to grant me root access
mkdir /root/.ssh
chown root.root /root/.ssh
chmod 700 /root/.ssh
cat <>/root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAvNDSbbViufkQdqHfI4lrF3utwd028ndTJspdiOZ2JtdIVBjUokQRoVFY8+DXjTpBIBKWd/WciqMc02gYUXG94pDZkxHe9Z0xD/SdpoGng2XodVUVEnNWImVSbpPFDwqFZWOqyC8QVEp7AMhj+4AdWp9JmTQUeYWLssrmvnY9m0dB3K2CL6G532y7cZ9cEl73kBIxPVOHkdRZGUyTC05fZL7Ldd2eepi/oWRpDXmfWn/rN6zl1vKaYq5TaOcnATCL1tmP/t8yOdodMCgqYHRbhh8zuFcsMxl7b+eenjhlsh87V/pdKrWZFcfeWxamj7CdEQA79r3Sw/7h6Y2OGvKYKzofGtnjPzJnu63Hzdu7oQcQTXQpuMgoSMkhS+MbJOfiJUONK1tfTKiN29NJZ90biSonu7XpOpemIRAlx/vhpVXkKcN2PY12fRy7wL0A9yghb6M1Hkw1bHK7tlw/cpQiHhEPJuTbBWTZJ3OWSLXx+EMRfdn8cHx1yckaqXzMLoGh52OkgVbNeN52bbrwDrelOc237zknPnSzbnB7wIwZwmRE0GDvl/Ta+AM5A8N7FMC5K9wbOgP9qObTbUQGwP0hwg/Xai2kR/7QUwSB3/y2ja2wZNCSP5aSGszLkJd3X5M0yALcQFVzNyqUKy5wQhQEpUKnteAvwbwpmUmuU6WQPNk= private key 2008-05-22
chown root.root /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
restorecon /root/.ssh/authorized_keys

cat <>/etc/udev/rules.d/75-disable-5GB-on-board-stick.rules
# The on-board 5GB stick should be disabled
# I currently have no use for it and leaving it untouched allows a reset to the shipped state
# by choosing the USB stick as boot target during POST
# c.f. https://projectgus.com/2014/09/blacklisting-a-single-usb-device-from-linux/
SUBSYSTEM=="usb", ATTRS{idVendor}=="1005", ATTRS{idProduct}=="b155", ATTR{authorized}="0"
chown root.root /etc/udev/rules.d/75-disable-5GB-on-board-stick.rules
chmod 644 /etc/udev/rules.d/75-disable-5GB-on-board-stick.rules
restorecon /etc/udev/rules.d/75-disable-5GB-on-board-stick.rules

# pull check-mk-agent from my monitoring server (checkmk Raw edition)
dnf -y install http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/check-mk-agent-2.0.0p17-1.noarch.rpm
echo "check-mk-agent installed from monitoring server" >> /etc/motd

# disable Red Hat graphical boot (rhgb)
sed --in-place "s/rhgb//g" /etc/default/grub
echo "removed graphical boot from grub defaults" >> /etc/motd

echo "kickstarted at `date` for CentOS Stream 9 and Ceph on QNAP TS-473A" >> /etc/motd


Interactive Installation

Alternatively, create a bootable installation medium to install CentOS Stream interactively.

All other Setup via Ansible

All other configuration tasks were done with the same Ansible Playbooks shown in this post: