Install four TerraMaster F5-422 with CentOS Stream

Table of Contents

I received my 3 additional TerraMaster F5-422.

Time to install CentOS Stream in preparation for Ceph Luminous.

Please also see the earlier post TerraMaster F5-422 with CentOS Stream for more info, it described my initial CentOS Stream install on my first F5-422.

Upgrade RAM

Like the first F5-422, on the three new ones I proceeded as follows;

  1. remove 6 screws on the back
  2. carefully remove back cover (mind the 2 cables connecting the the fans to the motherboard)
  3. add a Crucial CT102464BF160B 8GB SO-DIMM.
  4. close case in reverse order

Install SSD for OS and Ceph use

  1. put a Samsung SSD 860 EVO 500GB, SATA in drive bay 1
  2. only use e.g. 65GiB for base OS, keep some unpartitioned space for Ceph use.

Firmware Settings

As so often, I was not happy with the vendor’s UEFI settings and made the following changes.

  1. connect screen, keyboard, mouse (and USB boot medium)
  2. enter firmware setup
  3. Advanced / Power Management Features: enable Wake on LAN
  4. Advanced / Power Management Features: disable Wake Up on Alarm
  5. Chipset / South Bridge / OS Selection: change from Windows to Intel Linux
  6. Boot / Setup Prompt Timeout: change to 5 seconds or more
  7. Boot / Boot Mode: change from Legcy+UEFI to UEFI Only
  8. save and exit

Install CentOS Stream

nodes 2, 3 and 4 needing initial install and node 1 currently running F29, I processed all 4 as follows;

  1. create USB boot medium. I used isoimagewriter to write CentOS-Stream-x86_64-dvd1.iso on a USB stick
  2. boot from USB
  3. do a kistart install by adding inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/CentOS-Stream-TerraMaster-F5-422-ks.cfg in the USB medium’s bootloader phase
  4. while mediacheck ran, I adjusted /etc/dhcp/dhcpd.conf on the dhcp server, my DNS was set up previously.

Used Kickstart File

#version=RHEL8

# avoid using half arsed names like sda, sdb, etc
# mapping seems to be:
# bay 1 /dev/disk/by-path/pci-0000:00:12.0-ata-1
# bay 2 /dev/disk/by-path/pci-0000:00:12.0-ata-2
# bay 3 /dev/disk/by-path/pci-0000:03:00.0-ata-1
# bay 4 /dev/disk/by-path/pci-0000:03:00.0-ata-2
# bay 5 /dev/disk/by-path/pci-0000:04:00.0-ata-1

# we only use the SATA SSD in bay 1
ignoredisk --only-use=/dev/disk/by-path/pci-0000:00:12.0-ata-1

# Partition clearing information
# note that  OS goes on a small portion os the device in bay 1, the rest will be allocated to Ceph in a separtate VG.
# so kickstarting with the below clearpart line will nuke the Ceph bits on SSD !!!
clearpart --all --initlabel --drives=/dev/disk/by-path/pci-0000:00:12.0-ata-1

# Use graphical install
graphical

# Use CDROM installation media
cdrom

# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'

# System language
lang en_US.UTF-8

# Network information
# 1 Gig on USB
network  --bootproto=dhcp --device=enp0s21f0u3 --onboot=off --ipv6=auto --no-activate
# 10 Gig on PCIe
network  --bootproto=dhcp --device=enp1s0 --onboot=off --ipv6=auto
# 1 Gig on PCIe
network  --bootproto=dhcp --device=enp2s0 --ipv6=auto --activate
network  --hostname=localhost.localdomain

repo --name="AppStream" --baseurl=file:///run/install/repo/AppStream

# Root password
rootpw --iscrypted $6$X55cCQsXt3xeCukD$qJhkHV0x8cHjBc8zBbRldKq7YdMIf52lfM2GvaNW13HEPBIllwsXfIzmD1s4ZxstUTO5u4JzxAuLjz27SSmy6/

# Run the Setup Agent on first boot?
firstboot --disable

# Do not configure the X Window System
skipx

# System services
services --enabled="chronyd"

# System timezone
timezone Europe/Berlin --isUtc

# Ansible user
user --groups=wheel --name=ansible --password=$6$1XAtqKKCAXHi17l/$Yosu3OEblUaM2SvrbeBH6wlzV8TAv7eO.3jI9CpTG3gbRfGnXlbuLPmgw.IBacL3km/ArjEjKbd9Ki2LTxYpu. --iscrypted --gecos="ansible"

# Disk partitioning information
# we use a 65G PV (SATA SSD in bay 1 is 465.8G, the rest will be allocated to Ceph in VG_SSD_bay1_OS via Ansible)
# but keep free PE
part /boot     --fstype="ext4"  --ondisk=/dev/disk/by-path/pci-0000:00:12.0-ata-1 --size=1024
part /boot/efi --fstype="efi"   --ondisk=/dev/disk/by-path/pci-0000:00:12.0-ata-1 --size=512    --fsoptions="umask=0077,shortname=winnt"
part pv.65     --fstype="lvmpv" --ondisk=/dev/disk/by-path/pci-0000:00:12.0-ata-1 --size=66560
volgroup VG_SSD_bay1_OS --pesize=4096 pv.65
logvol /        --fstype="xfs"  --size=4096 --name=LV_root    --vgname=VG_SSD_bay1_OS
logvol swap     --fstype="swap" --size=4096 --name=LV_swap    --vgname=VG_SSD_bay1_OS
logvol /var     --fstype="xfs"  --size=4096 --name=LV_var     --vgname=VG_SSD_bay1_OS
logvol /var/log --fstype="xfs"  --size=2048 --name=LV_var_log --vgname=VG_SSD_bay1_OS
logvol /home    --fstype="xfs"  --size=2048 --name=LV_home    --vgname=VG_SSD_bay1_OS

%packages
@^minimal-environment
@container-management
chrony

%end

%addon com_redhat_kdump --disable --reserve-mb='auto'

%end

%anaconda
pwpolicy root --minlen=6 --minquality=1 --notstrict --nochanges --notempty
pwpolicy user --minlen=6 --minquality=1 --notstrict --nochanges --emptyok
pwpolicy luks --minlen=6 --minquality=1 --notstrict --nochanges --notempty
%end

%post --log=/root/ks-post.log
# dump pcfe's ssh key to the root user
mkdir /root/.ssh
chown root.root /root/.ssh
chmod 700 /root/.ssh
cat <<EOF >>/root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAvNDSbbViufkQdqHfI4lrF3utwd028ndTJspdiOZ2JtdIVBjUokQRoVFY8+DXjTpBIBKWd/WciqMc02gYUXG94pDZkxHe9Z0xD/SdpoGng2XodVUVEnNWImVSbpPFDwqFZWOqyC8QVEp7AMhj+4AdWp9JmTQUeYWLssrmvnY9m0dB3K2CL6G532y7cZ9cEl73kBIxPVOHkdRZGUyTC05fZL7Ldd2eepi/oWRpDXmfWn/rN6zl1vKaYq5TaOcnATCL1tmP/t8yOdodMCgqYHRbhh8zuFcsMxl7b+eenjhlsh87V/pdKrWZFcfeWxamj7CdEQA79r3Sw/7h6Y2OGvKYKzofGtnjPzJnu63Hzdu7oQcQTXQpuMgoSMkhS+MbJOfiJUONK1tfTKiN29NJZ90biSonu7XpOpemIRAlx/vhpVXkKcN2PY12fRy7wL0A9yghb6M1Hkw1bHK7tlw/cpQiHhEPJuTbBWTZJ3OWSLXx+EMRfdn8cHx1yckaqXzMLoGh52OkgVbNeN52bbrwDrelOc237zknPnSzbnB7wIwZwmRE0GDvl/Ta+AM5A8N7FMC5K9wbOgP9qObTbUQGwP0hwg/Xai2kR/7QUwSB3/y2ja2wZNCSP5aSGszLkJd3X5M0yALcQFVzNyqUKy5wQhQEpUKnteAvwbwpmUmuU6WQPNk= private key 2008-05-22
EOF
chown root.root /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
restorecon /root/.ssh/authorized_keys

cat <<EOF >>/etc/udev/rules.d/75-disable-120MB-on-board-stick.rules 
# The on-board 120MB stick should be disabled
# it's tiny and leaving it untouched allows a reset to the shipped state
# by choosing the USB stick as boot target during POST and booting UEFI+Legacy mode
# c.f. https://projectgus.com/2014/09/blacklisting-a-single-usb-device-from-linux/
SUBSYSTEM=="usb", ATTRS{idVendor}=="13fe", ATTRS{idProduct}=="4100", ATTR{authorized}="0"
EOF
chown root.root /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules
chmod 644 /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules
restorecon /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules

# Since Ceph and EPEL should not be mixed,
# pull check-mk-agent from my monitoring server (checkmk Raw edition)
dnf -y install http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/check-mk-agent-1.6.0p9-1.noarch.rpm
echo "check-mk-agent installed from monitoring server" >> /etc/motd

echo "kickstarted at `date` for CentOS Stream on TerraMaster F5-422" >> /etc/motd

%end

Since the machine is supposed to be mainly a Ceph OSD, I only used some 65G of the SSD for the OS.

Everything on top of this minimal kickstart install is handled via Ansible. You could also do a manual minimal install, I simply could not be asked to click around interactively for 4 installs.

ToDo

the following still need to be added to the playbook.

Test WOL

Still need to test if Wake on LAN works.

Clen up NetworkManmager connections

TODO: delete left behind ifcfg-enp… after terramaster-network.yml ran.

Set Up smartd

TODO: still need to set up smartd to send alerts to my sms gateway.

Disable the 120 MB USB stick

note: the NAS has a 120MB stick in an on-board connector, I decided to not touch it for now. In case I ever need to go back to the vendor OS. Note that enabling EUFI only boot mode makes the 120MB stick not show up as a valid boot target with the shipped image. Seems someone forgot to get their boot medium signed.

I disabled it with a blacklist entry.

[root@f5-422-01 ~]# cat /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules
# The on-board 120MB stick should be disabled
# it's tiny and I have 0 use for it
# c.f. https://projectgus.com/2014/09/blacklisting-a-single-usb-device-from-linux/
SUBSYSTEM=="usb", ATTRS{idVendor}=="13fe", ATTRS{idProduct}=="4100", ATTR{authorized}="0"

TODO: put the above cleanly in a playbook, at the moment it’s handled in kickstart.

After the next boot, as expected the tiny flash drive is no longer visible

[root@f5-422-01 ~]# lsusb
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13fe:4100 Kingston Technology Company Inc. Flash drive
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
[root@f5-422-01 ~]# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/7p, 5000M
    |__ Port 3: Dev 3, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M

Drive Bay Mapping

To avoid using half arsed names like sda, sdb, etc I address the drive bays by their /dev/disk/by-path/ entries.

FIXME: double check. The mapping seems to be

bay number /dev/disk/by-path/ entry
bay 1 /dev/disk/by-path/pci-0000:00:12.0-ata-1
bay 2 /dev/disk/by-path/pci-0000:00:12.0-ata-2
bay 3 /dev/disk/by-path/pci-0000:03:00.0-ata-1
bay 4 /dev/disk/by-path/pci-0000:03:00.0-ata-2
bay 5 /dev/disk/by-path/pci-0000:04:00.0-ata-1

Initial Setup

Temporary repo foo on 2020-02-29

On 2020-02-29, I got on dnf upgrade

[...]
No available modular metadata for modular package 'oci-systemd-hook-1:0.1.15-2.git2d0b8a3.module_el8.1.0+237+63e26edc.x86_64', it cannot be installed on the system
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: No available modular metadata for modular package

So I quickly ripped that out to avoid going down that rabbit hole.

As root, on a F5-422:

dnf remove oci-systemd-hook

Playbooks will take care of installing needed packages and RPM takes care of dependencies.

Resolved on 2020-03-01

The above no longer occurred on Sunday 2020-03-01.

check-mk-agent

Since Ceph and EPEL are not meant to be mixed, I install check-mk-agent from my monitoring server.

FIXME: that’s in kickstart, not in Ansibe

FIXME: adjust role pcfe.checkmk, paths between package in EPEL and the one on Check MK Raw are different. For now I just hammered it into the F5-422 kickstart as all other boxes use the RPM from EPEL or Fedora.

terramaster-initial-setup-centos8.yml

takes care of

  • creating an ansible user
  • basic-security-setup
  • local settings (mirros etc)

As user, on my F31 x96_64 workstation.

ansible-playbook terramaster-initial-setup-centos8.yml
---
# sets up a CentoOS Stream minimal install to be ready for ceph-ansible
#
# this is for my home setup, not for production!
- hosts:
  - TerraMaster_boxes
  become: false
  roles:
    - pcfe.user_owner
    - pcfe.basic-security-setup
    - pcfe.housenet
    #- pcfe.checkmk
    # note though that check-mk-agent is in EPEL but RHCS 3 docs say EPEL needs to be off
    # for now I manually installed the rpm with
    # dnf install http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/check-mk-agent-1.6.0p5-1.noarch.rpm


  vars:
    ansible_user: root
    user_owner: ansible
    common_timezone: Europe/Berlin

  tasks:
    # !!!
    #
    # if I ever enable EPEL, then I MUST exclude
    # - ansible
    # - ceph
    # in the EPEL repo files to ensure no newer versions of those packages are pulled in from EPEL
    #
    # exclude = *ceph* nfs-ganesha-rgw rbd-mirror *ansible*
    #
    # !!!
    #
    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/installation_guide/index#enabling-the-red-hat-ceph-storage-repositories-install
    - name: "REPOS | ensure EPEL is disabled"
      yum_repository:
        name: epel
        state: absent

    # start by enabling time sync, RHSM operations will fail on too large time delta
    # note that this uses chronyd, not ntpd.
    - name: "CHRONYD | ensure chrony is installed"
      package:
        name:       chrony
        state:      present
    - name:         "CHRONYD | ensure chrony-wait is enabled"
      service:
        name:       chrony-wait
        enabled:    true
    - name:         "CHRONYD | ensure chronyd is enabled and running"
      service:
        name:       chronyd
        enabled:    true
        state:      started
    
    # enable persistent journal
    - name: "JOURNAL | ensure persistent logging for the systemd journal is possible"
      file:
        path: /var/log/journal
        state: directory
        owner: root
        group: systemd-journal
        mode: u=rwx,g=rs,o=rx

    # 2.10. Enabling Password-less SSH for Ansible
    - name: "SUDO | enable passwordless sudo for ansible user"
      copy:
        dest: /etc/sudoers.d/ansible
        content: |
          ansible   ALL=NOPASSWD:   ALL
        owner: root
        group: root
        mode: 0440

    # Install prerequisites
    - name: "package | ensure prerequisites needed in addition to minimal install are present"
      package:
        name:
          - vim
        state: present

    # I admit, the regexp is a search engine hit
    - name: "GRUB | ensure console blanking is disabled in defaults file"
      lineinfile:
        state: present
        dest: /etc/default/grub
        backrefs: yes
        regexp: '^(GRUB_CMDLINE_LINUX=(?!.* consoleblank)\"[^\"]+)(\".*)'
        line: '\1 consoleblank=0\2'

    # upgrade the box
    - name: "package | ensure all updates are applied"
      package:
        update_cache: yes
        name: '*'
        state: latest
      tags: apply_errata


terramaster-general-setup.yml

Besides running some of my roles, it takes care of

  • enabled wake on LAN
  • setup watchdog
  • set up rngd

As user, on my F31 x96_64 workstation.

ansible-playbook terramaster-general-setup.yml
---
- hosts:
  - TerraMaster_boxes
  become: true
  roles:
    - linux-system-roles.network
    - pcfe.user_owner
    - pcfe.basic-security-setup
    - pcfe.housenet
    - pcfe.comfort
#    - pcfe.checkmk

  tasks:
    # Install some tools
    - name: "PACKAGE | tool installation"
      package:
        name:
          - pciutils
          - usbutils
          - fio
          - powertop
          - tuned
          - tuned-utils
          - numactl
          - mailx
          - teamd
          - NetworkManager-team
          - iperf3
        state: present
        update_cache: no

    # linux-system-roles.network sets static network config (from host_vars)
    # but I want the static hostname nailed down too
    - name: "set hostname"
      hostname:
        name:          "{{ ansible_fqdn }}"
        use:           systemd

    # enable WOL manually until https://github.com/linux-system-roles/network/issues/150 is fixed
    - name: "ensure Wake On LAN is enable for the upper network card"
      lineinfile:
        path:         /etc/sysconfig/network-scripts/ifcfg-1G_PCIe
        create:       false
        regexp:       '^ETHTOOL_OPTS= '
        insertafter:  '^TYPE=Ethernet'
        line:         'ETHTOOL_OPTS="wol g"'

    # enable watchdog
    # it's a Jun 22 13:12:09 localhost kernel: sbsa-gwdt e0bb0000.gwdt: Initialized with 10s timeout @ 250000000 Hz, action=0.
    - name: "WATCHDOG | ensure kernel module wdat_wdt has correct options configured"
      lineinfile:
        path:         /etc/modprobe.d/wdat_wdt.conf
        create:       true
        regexp:       '^options '
        insertafter:  '^#options'
        line:         'options wdat_wdt nowayout=0'

    # configure both watchdog.service and systemd watchdog, but only use the latter
    - name: "PACKAGE | ensure watchdog package is installed"
      package:
        name:         watchdog
        state:        present
        update_cache: no
    - name: "WATCHDOG | ensure correct watchdog-device is used by watchdog.service"
      lineinfile:
        path:         /etc/watchdog.conf
        regexp:       '^watchdog-device'
        insertafter:  '^#watchdog-device'
        line:         'watchdog-device = /dev/watchdog'
    - name: "WATCHDOG | ensure timeout is set to 30 seconds for watchdog.service"
      lineinfile:
        path:         /etc/watchdog.conf
        regexp:       '^watchdog-timeout'
        insertafter:  '^#watchdog-timeout'
        line:         'watchdog-timeout = 30'
    # Using systemd watchdog rather than watchdog.service
    - name: "WATCHDOG | ensure watchdog.service is disabled"
      systemd:
        name:         watchdog.service
        state:        stopped
        enabled:      false
    # configure systemd watchdog
    # c.f. http://0pointer.de/blog/projects/watchdog.html
    - name: "SYSTEMD | ensure systemd watchdog is enabled"
      lineinfile:
        path:         /etc/systemd/system.conf
        regexp:       '^RuntimeWatchdogSec'
        insertafter:  'EOF'
        line:         'RuntimeWatchdogSec=30'
    - name: "SYSTEMD | ensure systemd shutdown watchdog is enabled"
      lineinfile:
        path:         /etc/systemd/system.conf
        regexp:       '^ShutdownWatchdogSec'
        insertafter:  'EOF'
        line:         'ShutdownWatchdogSec=30'

    # install and enable rngd
    - name: "PACKAGE | ensure rng-tools package is installed"
      package:
        name:         rng-tools
        state:        present
        update_cache: no
    - name: "RNGD | ensure rngd.service is enabled and started"
      systemd:
        name:         rngd.service
        state:        started
        enabled:      true

    - name: "TUNED | ensure tuned.service is enabled and running"
      systemd:
        name:           tuned.service
        state:          started
        enabled:        true
    - block:
      - name: "TUNED | check which tuned profile is active"
        shell:          tuned-adm active
        register:       tuned_active_profile
        ignore_errors:  yes
        changed_when:   no
      - name: "TUNED | activate tuned profile throughput-performance if not yet enabled"
        shell:          tuned-adm profile throughput-performance
        when:           "tuned_active_profile.stdout.find('Current active profile: throughput-performance') != 0"

    # install cockpit, but disabled for now
    - name: "COCKPIT | ensure packages for https://cockpit-project.org/ are installed"
      package:
        name:
          - cockpit
          - cockpit-selinux
          - cockpit-machines
          - cockpit-kdump
          - cockpit-storaged
          - cockpit-system
        state: present
        update_cache: no
    - name: "COCKPIT | ensure cockpit.socket is stopped and disabled"
      systemd:
        name:       cockpit.socket
        state:      stopped
        enabled:    False
    - name: "COCKPIT | ensure firewalld forbids service cockpit in zone public"
      firewalld:
        service:    cockpit
        zone:       public
        permanent:  True
        state:      disabled
        immediate:  True

    # disable libvirtd
    - name: "Ensure libvirtd.service is disabled and stopped"
      systemd:
        name:         libvirtd.service
        state:        stopped
        enabled:      False

    # enable kdump.service since kickstart now creates a sufficiently large /var/crash
    # alternatively, you could set up netdump
    - name: "Ensure kdump.service is enabled and started"
      systemd:
        name:         kdump.service
        state:        started
        enabled:      True

    # podman
    - name: "PACKAGE | ensure podman is installed"
      package:
        name:
          - podman
          - podman-docker
        state: present

    # setroubleshoot, see also https://danwalsh.livejournal.com/20931.html
    - name: "PACKAGE | ensure setroubleshoot for headless server is installed"
      package:
        name:
          - setroubleshoot-server
          - setroubleshoot-plugins
        state: present

    - name: "MONITORING | ensure packages for monitoring are installed"
      package:
        name:
          - smartmontools
          - hdparm
          - check-mk-agent
        state: present

    - name: "MONITORING | ensure firewalld permits 6556 in zone public for check-mk-agent"
      firewalld:
        port:       6556/tcp
        permanent:  True
        state:      enabled
        immediate:  True
        zone:       public
    - name: "MONITORING | ensure tarsnap cache is in fileinfo"
      lineinfile:
        path: /etc/check_mk/fileinfo.cfg
        line: "/usr/local/tarsnap-cache/cache"
        create: yes
    - name: "MONITORING | ensure entropy_avail plugin for Check_MK is present"
      template:
        src:        templates/check-mk-agent-plugin-entropy_avail.j2
        dest:       /usr/lib/check_mk_agent/plugins/entropy_avail
        mode:       0755
        group:      root
        owner:      root
    - name: "MONITORING | plugins from running CEE instance"
      get_url:
        url: "http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/plugins/{{ item }}"
        dest: "/usr/lib/check_mk_agent/plugins/{{ item }}"
        mode: "0755"
      loop:
        - smart
        - lvm
    - name: "MONITORING | ensure check_mk.socket is started and enabled"
      systemd:
        name:       check_mk.socket
        state:      started
        enabled:    True

    # I admit, the regexp is a search engine hit
    # maybe using grubby(8) would be more readable
    # - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/configuring-kernel-command-line-parameters_managing-monitoring-and-updating-the-kernel#what-is-grubby_configuring-kernel-command-line-parameters
    # - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sec-Making_Persistent_Changes_to_a_GRUB_2_Menu_Using_the_grubby_Tool
    - name: "GRUB | ensure console blanking is disabled in defaults file plus handler"
      lineinfile:
        state: present
        dest: /etc/default/grub
        backrefs: yes
        regexp: '^(GRUB_CMDLINE_LINUX=(?!.* consoleblank)\"[^\"]+)(\".*)'
        line: '\1 consoleblank=0\2'
      notify: grub2-mkconfig | run

    # Since I do not manage to get these F5-422 to PXE boot, add an entry into grub
    # so that I can kickstart the box after this without fiddling with a USB stick
    - name: "GRUB | ensure initrd for RHEL 8.1 kickstart is present"
      get_url:
        url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.1/Server/x86_64/os/images/pxeboot/initrd.img"
        dest: "/boot/initrd-kickstart-rhel81.img"
        mode: "0600"
    - name: "GRUB | ensure kernel for RHEL 8.1 kickstart is present"
      get_url:
        url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.1/Server/x86_64/os/images/pxeboot/vmlinuz"
        dest: "/boot/vmlinuz-kickstart-rhel81"
        mode: "0755"
    - name: "GRUB | ensure kickstarting RHEL 8.1 entry is present"
      copy:
        dest: "/etc/grub.d/11_RHEL81_kickstart"
        owner: "root"
        group: "root"
        mode: 0755
        content: |
          #!/bin/sh
          exec tail -n +3 $0
          # This file provides an easy way to add custom menu entries.  Simply type the
          # menu entries you want to add after this comment.  Be careful not to change
          # the 'exec tail' line above.
          menuentry "WARNING Kickstart this box as a F5-422 ceph node WARNING" {
              linuxefi /vmlinuz-kickstart-rhel81 ip=dhcp inst.repo=ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.1/Server/x86_64/os inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/RHEL81-TerraMaster-F5-422-ks.cfg
              initrdefi /initrd-kickstart-rhel81.img
          }
      notify: grub2-mkconfig | run


    # upgrade the box
    - name: "package | ensure all updates are applied"
      package:
        update_cache: yes
        name: '*'
        state: latest
      tags: apply_errata

  handlers:
    - name: grub2-mkconfig | run
      command: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

test forcefully crashing the box.

Verify that the watchdog works as expected.

As root, on a F5-422:

echo '1' > /proc/sys/kernel/sysrq
echo 'c' > /proc/sysrq-trigger

As expected, I get an ops outut on the console acter the c and, as previously tested, the box reboots itself after 30 seconds.

Hardware Info

dmidecode -t baseboard

[root@f5-422-01 ~]# dmidecode -t baseboard
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
	Manufacturer: GIFA Industrial Control
	Product Name: TM-J3455-4G3L
	Version: Default string
	Serial Number: Default string
	Asset Tag: Default string
	Features:
		Board is a hosting board
		Board is replaceable
	Location In Chassis: Default string
	Chassis Handle: 0x0003
	Type: Motherboard
	Contained Object Handles: 0

Handle 0x0025, DMI type 10, 6 bytes
On Board Device Information
	Type: Video
	Status: Enabled
	Description:    To Be Filled By O.E.M.

Handle 0x002D, DMI type 41, 11 bytes
Onboard Device
	Reference Designation:  Onboard IGD
	Type: Video
	Status: Enabled
	Type Instance: 1
	Bus Address: 0000:00:02.0

Handle 0x002E, DMI type 41, 11 bytes
Onboard Device
	Reference Designation:  Onboard LAN
	Type: Ethernet
	Status: Enabled
	Type Instance: 1
	Bus Address: 0000:00:19.0

Handle 0x002F, DMI type 41, 11 bytes
Onboard Device
	Reference Designation:  Onboard 1394
	Type: Other
	Status: Enabled
	Type Instance: 1
	Bus Address: 0000:03:1c.2

lspci

[root@f5-422-01 ~]# lspci
00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
00:02.0 VGA compatible controller: Intel Corporation Device 5a85 (rev 0b)
00:0e.0 Audio device: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster (rev 0b)
00:0f.0 Communication controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine (rev 0b)
00:12.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b)
00:13.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 (rev fb)
00:13.2 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3 (rev fb)
00:13.3 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #4 (rev fb)
00:14.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port B #1 (rev fb)
00:14.1 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port B #2 (rev fb)
00:15.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b)
00:1a.0 Serial bus controller [0c80]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PWM Pin Controller (rev 0b)
00:1f.0 ISA bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface (rev 0b)
00:1f.1 SMBus: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller (rev 0b)
01:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)

lsusb

[root@f5-422-01 ~]# lsusb
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13fe:4100 Kingston Technology Company Inc. Flash drive
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

lsusb -t

[root@f5-422-01 ~]# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/7p, 5000M
    |__ Port 3: Dev 3, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M

lscpu

[root@f5-422-01 ~]# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               92
Model name:          Intel(R) Celeron(R) CPU J3455 @ 1.50GHz
Stepping:            9
CPU MHz:             1582.479
CPU max MHz:         2300.0000
CPU min MHz:         800.0000
BogoMIPS:            2995.20
Virtualization:      VT-x
L1d cache:           24K
L1i cache:           32K
L2 cache:            1024K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault cat_l2 ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust smep erms mpx rdt_a rdseed smap clflushopt intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts md_clear arch_capabilities

free -m, free -h

[root@f5-422-01 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:          11854         268       11345           1         239       11330
Swap:          4095           0        4095
[root@f5-422-01 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           11Gi       268Mi        11Gi       1.0Mi       239Mi        11Gi
Swap:         4.0Gi          0B       4.0Gi

dmidecode -t memory

[root@f5-422-01 ~]# dmidecode -t memory
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.

Handle 0x0030, DMI type 16, 23 bytes
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: None
	Maximum Capacity: 32 GB
	Error Information Handle: Not Provided
	Number Of Devices: 4

Handle 0x0031, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: 64 bits
	Data Width: 64 bits
	Size: 4096 MB
	Form Factor: SODIMM
	Set: None
	Locator: ChannelA-DIMM0
	Bank Locator: BANK 0
	Type: DDR3
	Type Detail: Synchronous
	Speed: 1600 MT/s
	Manufacturer: Undefi
	Serial Number: 88000000
	Asset Tag: 9876543210
	Part Number:                /
	Rank: Unknown
	Configured Memory Speed: 1600 MT/s
	Minimum Voltage: 44.975 V
	Maximum Voltage: 44.975 V
	Configured Voltage: 1.5 V

Handle 0x0032, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: 64 bits
	Data Width: 64 bits
	Size: 8192 MB
	Form Factor: SODIMM
	Set: None
	Locator: ChannelB-DIMM0
	Bank Locator: BANK 1
	Type: DDR3
	Type Detail: Synchronous
	Speed: 1600 MT/s
	Manufacturer: Unde
	Serial Number: [REDACTED]
	Asset Tag: 9876543210
	Part Number: CT102464BF160B.M16
	Rank: Unknown
	Configured Memory Speed: 1600 MT/s
	Minimum Voltage: 1.35 V
	Maximum Voltage: 1.5 V
	Configured Voltage: 1.5 V

Handle 0x0033, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: Unknown
	Data Width: Unknown
	Size: No Module Installed
	Form Factor: DIMM
	Set: None
	Locator: ChannelC-DIMM0
	Bank Locator: BANK 2
	Type: Unknown
	Type Detail: None
	Speed: Unknown
	Manufacturer: Not Specified
	Serial Number: Not Specified
	Asset Tag: 9876543210
	Part Number: Not Specified
	Rank: Unknown
	Configured Memory Speed: 1600 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x0034, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: Unknown
	Data Width: Unknown
	Size: No Module Installed
	Form Factor: DIMM
	Set: None
	Locator: ChannelD-DIMM0
	Bank Locator: BANK 3
	Type: Unknown
	Type Detail: None
	Speed: Unknown
	Manufacturer: Not Specified
	Serial Number: Not Specified
	Asset Tag: 9876543210
	Part Number: Not Specified
	Rank: Unknown
	Configured Memory Speed: 1600 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

The vendor’s filling of the DMI registers does leave to be desired…

	Minimum Voltage: 44.975 V
	Maximum Voltage: 44.975 V
	Configured Voltage: 1.5 V

yeah, right. But well, Configured Voltage is right.

Network Devices

upper NIC, 1Gig, PCIe

[root@f5-422-01 ~]# ethtool -i enp2s0
driver: r8169
version:
firmware-version: rtl8168h-2_0.0.2 02/26/15
expansion-rom-version:
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@f5-422-01 ~]# lspci -s 02:00.0
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

middle NIC, 1Gig, USB

[root@f5-422-01 ~]# ethtool -i enp0s21f0u3
driver: r8152
version: v1.09.9
firmware-version:
expansion-rom-version:
bus-info: usb-0000:00:15.0-3
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
[root@f5-422-01 ~]# lsusb | grep -i rtl
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter

lower NIC, 10Gig, PCIe

[root@f5-422-01 ~]# lsusb | grep -i rtl
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
[root@f5-422-01 ~]# ethtool -i enp1s0
driver: atlantic
version: 2.0.4.0-kern
firmware-version: 3.1.57
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@f5-422-01 ~]# lspci -s 01:00.0
01:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)

iperf3 between 2 10G NICs

In c’t 02/2020 S. 102 they were measured at roughly 7 GBit/s (Article here but paywalled. As a subscriber I just read it on paper.)

7,0 gigabit are 6,07 gibibit, so I seem to be pretty close with 5,8 on jumbo frames.

MTU 1500

With the default MTU, I get 5.4 Gbits/sec out of the interfaces.

iperf3 -c 192.168.30.181 -i 1 -t 20 -P 4 | tee $(date +%F)-iperf3-4parallel.log

Connecting to host 192.168.30.181, port 5201
[  5] local 192.168.30.182 port 48790 connected to 192.168.30.181 port 5201
[  7] local 192.168.30.182 port 48792 connected to 192.168.30.181 port 5201
[  9] local 192.168.30.182 port 48794 connected to 192.168.30.181 port 5201
[ 11] local 192.168.30.182 port 48796 connected to 192.168.30.181 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   111 MBytes   927 Mbits/sec    0    479 KBytes
[ 7] 0.00-1.00 sec 326 MBytes 2.74 Gbits/sec 0 516 KBytes
[ 9] 0.00-1.00 sec 110 MBytes 926 Mbits/sec 0 485 KBytes
[ 11] 0.00-1.00 sec 111 MBytes 933 Mbits/sec 0 488 KBytes
[SUM] 0.00-1.00 sec 658 MBytes 5.52 Gbits/sec 0


[ 5] 1.00-2.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 1.00-2.00 sec 326 MBytes 2.73 Gbits/sec 0 516 KBytes
[ 9] 1.00-2.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 1.00-2.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 1.00-2.00 sec 651 MBytes 5.46 Gbits/sec 0


[ 5] 2.00-3.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 2.00-3.00 sec 324 MBytes 2.72 Gbits/sec 0 516 KBytes
[ 9] 2.00-3.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 2.00-3.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 2.00-3.00 sec 649 MBytes 5.44 Gbits/sec 0


[ 5] 3.00-4.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 3.00-4.00 sec 326 MBytes 2.73 Gbits/sec 0 516 KBytes
[ 9] 3.00-4.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 3.00-4.00 sec 107 MBytes 901 Mbits/sec 0 488 KBytes
[SUM] 3.00-4.00 sec 650 MBytes 5.45 Gbits/sec 0


[ 5] 4.00-5.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 4.00-5.00 sec 325 MBytes 2.72 Gbits/sec 0 516 KBytes
[ 9] 4.00-5.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 4.00-5.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 4.00-5.00 sec 650 MBytes 5.45 Gbits/sec 0


[ 5] 5.00-6.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 5.00-6.00 sec 325 MBytes 2.72 Gbits/sec 0 516 KBytes
[ 9] 5.00-6.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 5.00-6.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 5.00-6.00 sec 650 MBytes 5.45 Gbits/sec 0


[ 5] 6.00-7.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 6.00-7.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 6.00-7.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 6.00-7.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 6.00-7.00 sec 651 MBytes 5.46 Gbits/sec 0


[ 5] 7.00-8.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 7.00-8.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 7.00-8.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 7.00-8.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 7.00-8.00 sec 651 MBytes 5.46 Gbits/sec 0


[ 5] 8.00-9.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 8.00-9.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 8.00-9.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 8.00-9.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 8.00-9.00 sec 650 MBytes 5.45 Gbits/sec 0


[ 5] 9.00-10.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 9.00-10.00 sec 324 MBytes 2.72 Gbits/sec 0 542 KBytes
[ 9] 9.00-10.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 9.00-10.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 9.00-10.00 sec 650 MBytes 5.45 Gbits/sec 0


[ 5] 10.00-11.00 sec 107 MBytes 901 Mbits/sec 0 479 KBytes
[ 7] 10.00-11.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 10.00-11.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 10.00-11.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 10.00-11.00 sec 649 MBytes 5.45 Gbits/sec 0


[ 5] 11.00-12.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 11.00-12.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 11.00-12.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 11.00-12.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 11.00-12.00 sec 650 MBytes 5.46 Gbits/sec 0


[ 5] 12.00-13.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 12.00-13.00 sec 324 MBytes 2.72 Gbits/sec 0 542 KBytes
[ 9] 12.00-13.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 12.00-13.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 12.00-13.00 sec 649 MBytes 5.45 Gbits/sec 0


[ 5] 13.00-14.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 13.00-14.00 sec 326 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 13.00-14.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 13.00-14.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 13.00-14.00 sec 651 MBytes 5.46 Gbits/sec 0


[ 5] 14.00-15.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 14.00-15.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 14.00-15.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 14.00-15.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 14.00-15.00 sec 650 MBytes 5.45 Gbits/sec 0


[ 5] 15.00-16.00 sec 109 MBytes 918 Mbits/sec 0 1.23 MBytes
[ 7] 15.00-16.00 sec 325 MBytes 2.73 Gbits/sec 0 1.27 MBytes
[ 9] 15.00-16.00 sec 108 MBytes 910 Mbits/sec 0 1.18 MBytes
[ 11] 15.00-16.00 sec 109 MBytes 918 Mbits/sec 0 617 KBytes
[SUM] 15.00-16.00 sec 653 MBytes 5.47 Gbits/sec 0


[ 5] 16.00-17.00 sec 108 MBytes 902 Mbits/sec 0 1.23 MBytes
[ 7] 16.00-17.00 sec 325 MBytes 2.73 Gbits/sec 0 1.27 MBytes
[ 9] 16.00-17.00 sec 109 MBytes 912 Mbits/sec 0 1.18 MBytes
[ 11] 16.00-17.00 sec 108 MBytes 908 Mbits/sec 0 617 KBytes
[SUM] 16.00-17.00 sec 649 MBytes 5.45 Gbits/sec 0


[ 5] 17.00-18.00 sec 109 MBytes 912 Mbits/sec 0 1.23 MBytes
[ 7] 17.00-18.00 sec 325 MBytes 2.73 Gbits/sec 0 1.27 MBytes
[ 9] 17.00-18.00 sec 108 MBytes 902 Mbits/sec 0 1.18 MBytes
[ 11] 17.00-18.00 sec 108 MBytes 910 Mbits/sec 0 617 KBytes
[SUM] 17.00-18.00 sec 650 MBytes 5.45 Gbits/sec 0


[ 5] 18.00-19.00 sec 109 MBytes 912 Mbits/sec 0 1.23 MBytes
[ 7] 18.00-19.00 sec 325 MBytes 2.73 Gbits/sec 0 1.27 MBytes
[ 9] 18.00-19.00 sec 109 MBytes 912 Mbits/sec 0 1.18 MBytes
[ 11] 18.00-19.00 sec 108 MBytes 909 Mbits/sec 0 617 KBytes
[SUM] 18.00-19.00 sec 651 MBytes 5.46 Gbits/sec 0


[ 5] 19.00-20.00 sec 108 MBytes 902 Mbits/sec 0 1.23 MBytes
[ 7] 19.00-20.00 sec 325 MBytes 2.73 Gbits/sec 0 1.27 MBytes
[ 9] 19.00-20.00 sec 109 MBytes 912 Mbits/sec 0 1.18 MBytes
[ 11] 19.00-20.00 sec 108 MBytes 908 Mbits/sec 0 617 KBytes
[SUM] 19.00-20.00 sec 649 MBytes 5.45 Gbits/sec 0


[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 2.12 GBytes 910 Mbits/sec 0 sender [ 5] 0.00-20.02 sec 2.12 GBytes 908 Mbits/sec receiver [ 7] 0.00-20.00 sec 6.35 GBytes 2.73 Gbits/sec 0 sender [ 7] 0.00-20.02 sec 6.35 GBytes 2.72 Gbits/sec receiver [ 9] 0.00-20.00 sec 2.12 GBytes 910 Mbits/sec 0 sender [ 9] 0.00-20.02 sec 2.12 GBytes 908 Mbits/sec receiver [ 11] 0.00-20.00 sec 2.12 GBytes 910 Mbits/sec 0 sender [ 11] 0.00-20.02 sec 2.12 GBytes 908 Mbits/sec receiver [SUM] 0.00-20.00 sec 12.7 GBytes 5.46 Gbits/sec 0 sender [SUM] 0.00-20.02 sec 12.7 GBytes 5.45 Gbits/sec receiver

iperf Done.

MTU 9000

With jumbo frames, I get 5.8 Gbits/sec out of the interfaces.

iperf3 -c 192.168.30.181 -i 1 -t 20 -P 4 | tee $(date +%F)-iperf3-4parallel.log

Connecting to host 192.168.30.181, port 5201
[  5] local 192.168.30.182 port 60392 connected to 192.168.30.181 port 5201
[  7] local 192.168.30.182 port 60394 connected to 192.168.30.181 port 5201
[  9] local 192.168.30.182 port 60396 connected to 192.168.30.181 port 5201
[ 11] local 192.168.30.182 port 60398 connected to 192.168.30.181 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   233 MBytes  1.95 Gbits/sec    0    638 KBytes
[ 7] 0.00-1.00 sec 112 MBytes 941 Mbits/sec 0 577 KBytes
[ 9] 0.00-1.00 sec 232 MBytes 1.95 Gbits/sec 0 612 KBytes
[ 11] 0.00-1.00 sec 123 MBytes 1.03 Gbits/sec 0 568 KBytes
[SUM] 0.00-1.00 sec 700 MBytes 5.87 Gbits/sec 0


[ 5] 1.00-2.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 1.00-2.00 sec 116 MBytes 972 Mbits/sec 0 577 KBytes
[ 9] 1.00-2.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 1.00-2.00 sec 115 MBytes 964 Mbits/sec 0 568 KBytes
[SUM] 1.00-2.00 sec 693 MBytes 5.81 Gbits/sec 0


[ 5] 2.00-3.00 sec 231 MBytes 1.93 Gbits/sec 0 638 KBytes
[ 7] 2.00-3.00 sec 115 MBytes 967 Mbits/sec 0 577 KBytes
[ 9] 2.00-3.00 sec 231 MBytes 1.93 Gbits/sec 0 612 KBytes
[ 11] 2.00-3.00 sec 116 MBytes 973 Mbits/sec 0 568 KBytes
[SUM] 2.00-3.00 sec 693 MBytes 5.81 Gbits/sec 0


[ 5] 3.00-4.00 sec 230 MBytes 1.93 Gbits/sec 0 638 KBytes
[ 7] 3.00-4.00 sec 116 MBytes 969 Mbits/sec 0 577 KBytes
[ 9] 3.00-4.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 3.00-4.00 sec 114 MBytes 960 Mbits/sec 0 568 KBytes
[SUM] 3.00-4.00 sec 691 MBytes 5.80 Gbits/sec 0


[ 5] 4.00-5.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 4.00-5.00 sec 115 MBytes 968 Mbits/sec 0 577 KBytes
[ 9] 4.00-5.00 sec 230 MBytes 1.93 Gbits/sec 0 612 KBytes
[ 11] 4.00-5.00 sec 115 MBytes 968 Mbits/sec 0 568 KBytes
[SUM] 4.00-5.00 sec 692 MBytes 5.80 Gbits/sec 0


[ 5] 5.00-6.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 5.00-6.00 sec 116 MBytes 972 Mbits/sec 0 577 KBytes
[ 9] 5.00-6.00 sec 230 MBytes 1.93 Gbits/sec 0 612 KBytes
[ 11] 5.00-6.00 sec 115 MBytes 966 Mbits/sec 0 568 KBytes
[SUM] 5.00-6.00 sec 692 MBytes 5.81 Gbits/sec 0


[ 5] 6.00-7.00 sec 230 MBytes 1.93 Gbits/sec 0 638 KBytes
[ 7] 6.00-7.00 sec 115 MBytes 966 Mbits/sec 0 577 KBytes
[ 9] 6.00-7.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 6.00-7.00 sec 115 MBytes 968 Mbits/sec 0 568 KBytes
[SUM] 6.00-7.00 sec 692 MBytes 5.80 Gbits/sec 0


[ 5] 7.00-8.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 7.00-8.00 sec 115 MBytes 968 Mbits/sec 0 577 KBytes
[ 9] 7.00-8.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 7.00-8.00 sec 115 MBytes 965 Mbits/sec 0 568 KBytes
[SUM] 7.00-8.00 sec 692 MBytes 5.81 Gbits/sec 0


[ 5] 8.00-9.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 8.00-9.00 sec 115 MBytes 968 Mbits/sec 0 577 KBytes
[ 9] 8.00-9.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 8.00-9.00 sec 115 MBytes 963 Mbits/sec 0 568 KBytes
[SUM] 8.00-9.00 sec 692 MBytes 5.80 Gbits/sec 0


[ 5] 9.00-10.00 sec 231 MBytes 1.93 Gbits/sec 0 638 KBytes
[ 7] 9.00-10.00 sec 116 MBytes 970 Mbits/sec 0 577 KBytes
[ 9] 9.00-10.00 sec 230 MBytes 1.93 Gbits/sec 0 612 KBytes
[ 11] 9.00-10.00 sec 115 MBytes 968 Mbits/sec 0 568 KBytes
[SUM] 9.00-10.00 sec 692 MBytes 5.80 Gbits/sec 0


[ 5] 10.00-11.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 10.00-11.00 sec 115 MBytes 963 Mbits/sec 0 577 KBytes
[ 9] 10.00-11.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 10.00-11.00 sec 115 MBytes 968 Mbits/sec 0 568 KBytes
[SUM] 10.00-11.00 sec 692 MBytes 5.81 Gbits/sec 0


[ 5] 11.00-12.00 sec 230 MBytes 1.93 Gbits/sec 0 638 KBytes
[ 7] 11.00-12.00 sec 116 MBytes 970 Mbits/sec 0 577 KBytes
[ 9] 11.00-12.00 sec 231 MBytes 1.93 Gbits/sec 0 612 KBytes
[ 11] 11.00-12.00 sec 116 MBytes 970 Mbits/sec 0 568 KBytes
[SUM] 11.00-12.00 sec 692 MBytes 5.81 Gbits/sec 0


[ 5] 12.00-13.00 sec 239 MBytes 2.00 Gbits/sec 0 638 KBytes
[ 7] 12.00-13.00 sec 119 MBytes 1.00 Gbits/sec 0 577 KBytes
[ 9] 12.00-13.00 sec 215 MBytes 1.80 Gbits/sec 0 926 KBytes
[ 11] 12.00-13.00 sec 119 MBytes 996 Mbits/sec 0 568 KBytes
[SUM] 12.00-13.00 sec 691 MBytes 5.80 Gbits/sec 0


[ 5] 13.00-14.00 sec 232 MBytes 1.95 Gbits/sec 0 961 KBytes
[ 7] 13.00-14.00 sec 115 MBytes 966 Mbits/sec 0 577 KBytes
[ 9] 13.00-14.00 sec 232 MBytes 1.95 Gbits/sec 0 926 KBytes
[ 11] 13.00-14.00 sec 117 MBytes 980 Mbits/sec 0 848 KBytes
[SUM] 13.00-14.00 sec 696 MBytes 5.84 Gbits/sec 0


[ 5] 14.00-15.00 sec 229 MBytes 1.92 Gbits/sec 0 961 KBytes
[ 7] 14.00-15.00 sec 116 MBytes 974 Mbits/sec 0 577 KBytes
[ 9] 14.00-15.00 sec 231 MBytes 1.93 Gbits/sec 0 926 KBytes
[ 11] 14.00-15.00 sec 116 MBytes 969 Mbits/sec 0 848 KBytes
[SUM] 14.00-15.00 sec 691 MBytes 5.80 Gbits/sec 0


[ 5] 15.00-16.00 sec 231 MBytes 1.94 Gbits/sec 0 961 KBytes
[ 7] 15.00-16.00 sec 115 MBytes 967 Mbits/sec 0 577 KBytes
[ 9] 15.00-16.00 sec 231 MBytes 1.94 Gbits/sec 0 926 KBytes
[ 11] 15.00-16.00 sec 116 MBytes 969 Mbits/sec 0 848 KBytes
[SUM] 15.00-16.00 sec 693 MBytes 5.82 Gbits/sec 0


[ 5] 16.00-17.00 sec 230 MBytes 1.93 Gbits/sec 0 961 KBytes
[ 7] 16.00-17.00 sec 115 MBytes 962 Mbits/sec 0 577 KBytes
[ 9] 16.00-17.00 sec 230 MBytes 1.93 Gbits/sec 0 926 KBytes
[ 11] 16.00-17.00 sec 116 MBytes 970 Mbits/sec 0 848 KBytes
[SUM] 16.00-17.00 sec 690 MBytes 5.79 Gbits/sec 0


[ 5] 17.00-18.00 sec 231 MBytes 1.94 Gbits/sec 0 961 KBytes
[ 7] 17.00-18.00 sec 116 MBytes 970 Mbits/sec 0 577 KBytes
[ 9] 17.00-18.00 sec 231 MBytes 1.94 Gbits/sec 0 926 KBytes
[ 11] 17.00-18.00 sec 115 MBytes 968 Mbits/sec 0 848 KBytes
[SUM] 17.00-18.00 sec 694 MBytes 5.82 Gbits/sec 0


[ 5] 18.00-19.00 sec 230 MBytes 1.93 Gbits/sec 0 961 KBytes
[ 7] 18.00-19.00 sec 115 MBytes 963 Mbits/sec 0 577 KBytes
[ 9] 18.00-19.00 sec 230 MBytes 1.93 Gbits/sec 0 926 KBytes
[ 11] 18.00-19.00 sec 115 MBytes 961 Mbits/sec 0 848 KBytes
[SUM] 18.00-19.00 sec 689 MBytes 5.78 Gbits/sec 0


[ 5] 19.00-20.00 sec 231 MBytes 1.94 Gbits/sec 0 961 KBytes
[ 7] 19.00-20.00 sec 114 MBytes 959 Mbits/sec 0 577 KBytes
[ 9] 19.00-20.00 sec 231 MBytes 1.94 Gbits/sec 0 926 KBytes
[ 11] 19.00-20.00 sec 117 MBytes 983 Mbits/sec 0 848 KBytes
[SUM] 19.00-20.00 sec 694 MBytes 5.82 Gbits/sec 0


[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 4.51 GBytes 1.94 Gbits/sec 0 sender [ 5] 0.00-20.02 sec 4.51 GBytes 1.94 Gbits/sec receiver [ 7] 0.00-20.00 sec 2.25 GBytes 968 Mbits/sec 0 sender [ 7] 0.00-20.02 sec 2.25 GBytes 966 Mbits/sec receiver [ 9] 0.00-20.00 sec 4.49 GBytes 1.93 Gbits/sec 0 sender [ 9] 0.00-20.02 sec 4.49 GBytes 1.93 Gbits/sec receiver [ 11] 0.00-20.00 sec 2.27 GBytes 973 Mbits/sec 0 sender [ 11] 0.00-20.02 sec 2.26 GBytes 971 Mbits/sec receiver [SUM] 0.00-20.00 sec 13.5 GBytes 5.81 Gbits/sec 0 sender [SUM] 0.00-20.02 sec 13.5 GBytes 5.80 Gbits/sec receiver

iperf Done.