TerraMaster F5-422 with RHEL8

Table of Contents

In further happy distro hopping (since Ansible makes this quite painless), I installed RHEL8 on my four TerraMaster F5-422.

Please also see the earlier posts TerraMaster F5-422 with CentOS Stream and Install four TerraMaster F5-422 with CentOS Stream for more info.

Upgrade RAM

All my F5-422 had their RAM upgraded to 12 GiB. Proceeded as follows;

  1. remove 6 screws on the back
  2. carefully remove back cover (mind the 2 cables connecting the the fans to the motherboard)
  3. add a Crucial CT102464BF160B 8GB SO-DIMM or similar.
  4. close case in reverse order

Install SSD for OS and Ceph use

  1. I put a Samsung SSD 860 EVO 500GB, SATA in drive bay 1
  2. I only less than 100GiB for base OS, I keep the rest as unpartitioned space for Ceph use.

Firmware Settings

As so often, I was not happy with the vendor’s UEFI settings and made the following changes (it is probably a good idea to reset to defaults before you start).

  1. connect screen, keyboard, mouse (and USB boot medium)
  2. enter firmware setup
  3. Advanced / Power Management Features: enable Wake on LAN
  4. Advanced / Power Management Features: disable Wake Up on Alarm
  5. Chipset / South Bridge / OS Selection: change from Windows to Intel Linux
  6. Boot / Setup Prompt Timeout: change to 5 seconds or more
  7. Boot / Boot Mode: change from Legcy+UEFI to UEFI Only
  8. save and exit

Install RHEL8

See also https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_an_advanced_rhel_installation/index

  1. create USB boot medium as per docs with dd if=rhel-8.1-x86_64-boot.iso of=/dev/sdd status=progress
  2. connect screen, keyboard, mouse and USB boot medium
  3. enter firmware setup
  4. change OS type from Windows to Intel Linux
  5. set boot to UEFI only
  6. save and exit
  7. boot from USB
  8. do a minimal install or a kickstart install

Create Kickstart Entry on USB Stick

[root@t7910 iso]# mount /dev/sdd2 /mnt/import/
[root@t7910 iso]# vim /mnt/import/EFI/BOOT/grub.cfg 
[root@t7910 iso]# grep -A 3 Kickstart /mnt/import/EFI/BOOT/grub.cfg
menuentry 'Kickstart a F5-422 with Red Hat Enterprise Linux 8.1.0' --class fedora --class gnu-linux --class gnu --class os {
        linuxefi /images/pxeboot/vmlinuz ip=dhcp inst.repo=ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.1/Server/x86_64/os inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/RHEL81-TerraMaster-F5-422-ks.cfg
        initrdefi /images/pxeboot/initrd.img
}
  1. <Del> to enter Setup
  2. Go to Save & Exit / Boot Override and select UEFI: stickname, Partition 1
  3. Selected the entry Kickstart a F5-422 with Red Hat Enterprise Linux 8.1.0'

Used Kickstart File

#version=RHEL8

# avoid using half arsed names like sda, sdb, etc
# mapping seems to be:
# bay 1 /dev/disk/by-path/pci-0000:00:12.0-ata-1
# bay 2 /dev/disk/by-path/pci-0000:00:12.0-ata-2
# bay 3 /dev/disk/by-path/pci-0000:03:00.0-ata-1
# bay 4 /dev/disk/by-path/pci-0000:03:00.0-ata-2
# bay 5 /dev/disk/by-path/pci-0000:04:00.0-ata-1

# reboot after installation is complete?
#reboot

# we only use the SATA SSD in bay 1
ignoredisk --only-use=/dev/disk/by-path/pci-0000:00:12.0-ata-1

# Partition clearing information
# note that  OS goes on a small portion os the device in bay 1, the rest will be allocated to Ceph in a separtate VG.
# so kickstarting with the below clearpart line will nuke the Ceph bits on SSD !!!
clearpart --all --initlabel --drives=/dev/disk/by-path/pci-0000:00:12.0-ata-1

# Use graphical install
graphical

# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'

# System language
lang en_US.UTF-8

# Network information all switch ports have the respective VLAN as native
# 1 Gig on PCIe (will go on 'storage' via ansible)
network  --bootproto=dhcp --device=enp2s0 --onboot=off --ipv6=auto --no-activate
# 10 Gig on PCIe (will go on 'ceph' via ansible)
network  --bootproto=dhcp --device=enp1s0 --onboot=off --ipv6=auto  --no-activate
# 1 Gig on USB ('access' network)
network  --bootproto=dhcp --device=enp0s21f0u3 --ipv6=auto --activate
network  --hostname=localhost.localdomain

# Use network installation
url --url="ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.2/Server/x86_64/os/BaseOS"
repo --name="AppStream" --baseurl=ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.2/Server/x86_64/os/AppStream

# Root password
rootpw --iscrypted $6$X55cCQsXt3xeCukD$qJhkHV0x8cHjBc8zBbRldKq7YdMIf52lfM2GvaNW13HEPBIllwsXfIzmD1s4ZxstUTO5u4JzxAuLjz27SSmy6/

# Run the Setup Agent on first boot?
firstboot --disable

# Do not configure the X Window System
skipx

# System services
services --enabled="chronyd"

# Intended system purpose
syspurpose --role="Red Hat Enterprise Linux Server" --sla="Self-Support" --usage="Development/Test"

# System timezone
timezone Europe/Berlin --isUtc --ntpservers=epyc.internal.pcfe.net,edgerouter-6p.internal.pcfe.net

# Ansible user
user --groups=wheel --name=ansible --password=$6$1XAtqKKCAXHi17l/$Yosu3OEblUaM2SvrbeBH6wlzV8TAv7eO.3jI9CpTG3gbRfGnXlbuLPmgw.IBacL3km/ArjEjKbd9Ki2LTxYpu. --iscrypted --gecos="ansible"

# Disk partitioning information
# the 120 GB SATA SSD in slot 1 will be fully used for the OS
# the 500 GB SATA SSD in slot 2 and the HDDs in slots 3 through 5
# will be fed top ceph-ansible as devices
# c.f. https://docs.ceph.com/ceph-ansible/master/osds/scenarios.html
part /boot        --fstype="ext4"  --ondisk=/dev/disk/by-path/pci-0000:00:12.0-ata-1 --size=1024
part /boot/efi    --fstype="efi"   --ondisk=/dev/disk/by-path/pci-0000:00:12.0-ata-1 --size=512    --fsoptions="umask=0077,shortname=winnt"
part pv.01        --fstype="lvmpv" --ondisk=/dev/disk/by-path/pci-0000:00:12.0-ata-1 --size=61440 --grow
volgroup VG_OS_SSD_bay1 --pesize=4096 pv.01
logvol /                   --fstype="xfs"  --size=4096  --name=LV_root       --vgname=VG_OS_SSD_bay1
logvol swap                --fstype="swap" --size=4096  --name=LV_swap       --vgname=VG_OS_SSD_bay1
logvol /var                --fstype="xfs"  --size=4096  --name=LV_var        --vgname=VG_OS_SSD_bay1
logvol /var/log            --fstype="xfs"  --size=4096  --name=LV_var_log    --vgname=VG_OS_SSD_bay1
logvol /var/crash          --fstype="xfs"  --size=15360 --name=LV_var_crash  --vgname=VG_OS_SSD_bay1
logvol /var/lib/containers --fstype="xfs"  --size=8192  --name=LV_containers --vgname=VG_OS_SSD_bay1
logvol /home               --fstype="xfs"  --size=1024  --name=LV_home       --vgname=VG_OS_SSD_bay1

%packages
@^minimal-environment
chrony
kexec-tools

%end

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

%anaconda
pwpolicy root --minlen=6 --minquality=1 --notstrict --nochanges --notempty
pwpolicy user --minlen=6 --minquality=1 --notstrict --nochanges --emptyok
pwpolicy luks --minlen=6 --minquality=1 --notstrict --nochanges --notempty
%end

%post --log=/root/ks-post.log
# dump pcfe's ssh key to the root user
# obviously change this to your own pubkey unless you want to grant me root access
mkdir /root/.ssh
chown root.root /root/.ssh
chmod 700 /root/.ssh
cat <>/root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAvNDSbbViufkQdqHfI4lrF3utwd028ndTJspdiOZ2JtdIVBjUokQRoVFY8+DXjTpBIBKWd/WciqMc02gYUXG94pDZkxHe9Z0xD/SdpoGng2XodVUVEnNWImVSbpPFDwqFZWOqyC8QVEp7AMhj+4AdWp9JmTQUeYWLssrmvnY9m0dB3K2CL6G532y7cZ9cEl73kBIxPVOHkdRZGUyTC05fZL7Ldd2eepi/oWRpDXmfWn/rN6zl1vKaYq5TaOcnATCL1tmP/t8yOdodMCgqYHRbhh8zuFcsMxl7b+eenjhlsh87V/pdKrWZFcfeWxamj7CdEQA79r3Sw/7h6Y2OGvKYKzofGtnjPzJnu63Hzdu7oQcQTXQpuMgoSMkhS+MbJOfiJUONK1tfTKiN29NJZ90biSonu7XpOpemIRAlx/vhpVXkKcN2PY12fRy7wL0A9yghb6M1Hkw1bHK7tlw/cpQiHhEPJuTbBWTZJ3OWSLXx+EMRfdn8cHx1yckaqXzMLoGh52OkgVbNeN52bbrwDrelOc237zknPnSzbnB7wIwZwmRE0GDvl/Ta+AM5A8N7FMC5K9wbOgP9qObTbUQGwP0hwg/Xai2kR/7QUwSB3/y2ja2wZNCSP5aSGszLkJd3X5M0yALcQFVzNyqUKy5wQhQEpUKnteAvwbwpmUmuU6WQPNk= private key 2008-05-22
EOF
chown root.root /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
restorecon /root/.ssh/authorized_keys

cat <>/etc/udev/rules.d/75-disable-120MB-on-board-stick.rules
# The on-board 120MB stick should be disabled
# it's tiny and leaving it untouched allows a reset to the shipped state
# by choosing the USB stick as boot target during POST and booting UEFI+Legacy mode
# c.f. https://projectgus.com/2014/09/blacklisting-a-single-usb-device-from-linux/
SUBSYSTEM=="usb", ATTRS{idVendor}=="13fe", ATTRS{idProduct}=="4100", ATTR{authorized}="0"
EOF
chown root.root /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules
chmod 644 /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules
restorecon /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules

# Since Ceph and EPEL should not be mixed,
# pull check-mk-agent from my monitoring server (checkmk Raw edition)
dnf -y install http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/check-mk-agent-2.0.0p17-1.noarch.rpm
echo "check-mk-agent installed from monitoring server" >> /etc/motd

# seems I can NOT specify a connection name for the network setup
# https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#network
# so just remove the line setting the name "System enp…" and then move the files
# linux-system-roles.network will do the rest later
# I do this because I find speaking names so much more pleasant
sed --in-place "s/^NAME//g" /etc/sysconfig/network-scripts/ifcfg-enp*
mv /etc/sysconfig/network-scripts/ifcfg-enp0s21f0u3 /etc/sysconfig/network-scripts/ifcfg-1G_USB
mv /etc/sysconfig/network-scripts/ifcfg-enp1s0 /etc/sysconfig/network-scripts/ifcfg-10G_PCIe
mv /etc/sysconfig/network-scripts/ifcfg-enp2s0 /etc/sysconfig/network-scripts/ifcfg-1G_PCIe

# disable Red Hat graphical boot (rhgb)
sed --in-place "s/ rhgb//g" /etc/default/grub

echo "kickstarted at `date` for RHEL 8.2 on TerraMaster F5-422" >> /etc/motd

%end


Since the machine is supposed to be mainly a Ceph OSD, initially (der RHEL 8.1) I only used some 50GiB of the SSD for the OS.

But since then I’ve gone to

bay 1: 120 GB SSD used completely for OS bay 2: 500 GB HDD split up decided by cep-ansible bays 3 through 5: assorted HDD I had in my homelab

Everything on top of this minimal kickstart install is handled via Ansible. You could also do a manual minimal install, I simply could not be asked to click around interactively for 4 installs.

ToDo

the following still need to be added to the playbook.

Set Up smartd

TODO: still need to set up smartd to send alerts to my sms gateway.

Disable the 120 MB USB stick

note: the NAS has a 120MB stick in an on-board connector, I decided to not touch it for now. In case I ever need to go back to the vendor OS. Note that enabling EUFI only boot mode makes the 120MB stick not show up as a valid boot target with the shipped image. Seems someone forgot to get their boot medium signed.

I disabled it with a blacklist entry.

[root@f5-422-01 ~]# cat /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules
# The on-board 120MB stick should be disabled
# it's tiny and I have 0 use for it
# c.f. https://projectgus.com/2014/09/blacklisting-a-single-usb-device-from-linux/
SUBSYSTEM=="usb", ATTRS{idVendor}=="13fe", ATTRS{idProduct}=="4100", ATTR{authorized}="0"

TODO: put the above cleanly in a playbook, at the moment it’s handled in kickstart.

After the next boot, as expected the tiny flash drive is no longer visible

[root@f5-422-01 ~]# lsusb
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13fe:4100 Kingston Technology Company Inc. Flash drive
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
[root@f5-422-01 ~]# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/7p, 5000M
    |__ Port 3: Dev 3, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M

Drive Bay Mapping

To avoid using half arsed names like sda, sdb, etc I address the drive bays by their /dev/disk/by-path/ entries.

FIXME: double check. The mapping seems to be

bay number /dev/disk/by-path/ entry
bay 1 /dev/disk/by-path/pci-0000:00:12.0-ata-1
bay 2 /dev/disk/by-path/pci-0000:00:12.0-ata-2
bay 3 /dev/disk/by-path/pci-0000:03:00.0-ata-1
bay 4 /dev/disk/by-path/pci-0000:03:00.0-ata-2
bay 5 /dev/disk/by-path/pci-0000:04:00.0-ata-1

Initial Setup

check-mk-agent

Since Ceph and EPEL are not meant to be mixed, I install check-mk-agent from my monitoring server.

For now I just hammered it into the F5-422 kickstart as all other boxes use the RPM from EPEL or Fedora.

terramaster-initial-setup-centos8.yml

takes care of

  • creating an ansible user
  • basic-security-setup
  • local settings (mirros etc)
  • adding consoleblank=0 to my default boot options

It’s the same as terramaster-initial-setup-centos8.yml but with Subscription handling added.

As user, on my F31 x96_64 workstation.

ansible-playbook -i ../inventories/ceph-F5-422-cluster.ini terramaster-initial-setup-rhel8.yml
---
# sets up a RHEL 8 minimal install to be ready for ceph-ansible
#
# this is for my home setup, not for production!
- hosts:
  - TerraMaster_boxes

  become: false

  roles:
    - pcfe.user_owner
    - pcfe.basic_security_setup
    - pcfe.housenet
    #- pcfe.checkmk
    # note though that check-mk-agent is in EPEL but RHCS 4 docs say EPEL needs to be off
    # for now I manually installed the rpm with
    # dnf install http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/check-mk-agent-1.6.0p5-1.noarch.rpm

  # no need for double indirect if you are OK with checking in ak details into git
  # this is OK to do if you use an in-house Satellite server and your security policies allow it
  # this is not a good idea if you register your systems directly to redhat.com and cannot guarantee that your git remains private
  vars_files:
    - "vars/subscription-manager-autoattach-ak-secrets.yml"
  vars:
    ansible_user: root
    user_owner: ansible
    common_timezone: Europe/Berlin
    rhsm_activationkey: "{{ vaulted_rhsm_activationkey }}"
    rhsm_org_id: "{{ vaulted_rhsm_org_id }}"
    rhsm_pool_ids: "{{ vaulted_rhsm_pool_ids }}"

  pre_tasks:
    # work around identical UUID on each host as per https://theforeman.org/plugins/katello/nightly/troubleshooting/content_hosts.html
    # The TerraMaster F5-422 boxes all have the same system-uuid :-(
    # RHSM does not like that, so override to avoid both
    # "HTTP error (409 - Conflict): Request failed due to concurrent modification, please re-try.\n"
    # and all 4 boxes overriding each other in insights inventory
    - name: "RHSM | ensure uuid override is derived from fqdn, F5-422 all have idential uuid in DMI"
      copy:
        dest: "/etc/rhsm/facts/uuid_override.facts"
        owner: "root"
        group: "root"
        mode: 0644
        content: |
          {"dmi.system.uuid": "{{ ansible_fqdn | to_uuid }}"}


    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/installation_guide/index#enabling-the-red-hat-ceph-storage-repositories-install
    - name: "RHSM on the RHEL8 boxes"
      block:
        - name: "RHSM | ensure system is registered with my activation key"
          redhat_subscription:
            activationkey:  "{{ rhsm_activationkey }}"
            org_id:         "{{ rhsm_org_id }}"
            syspurpose:
              usage: "Development/Test"
              role: ""
              service_level_agreement: "Self-Support"

        - name: "RHSM | ensure RHEL8 BaseOS repos needed for Ceph are enabled"
          rhsm_repository:
            name:
              - rhel-8-for-x86_64-baseos-rpms
              - rhel-8-for-x86_64-appstream-rpms
            state: enabled
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
      tags: do_subsmgr_register
    - name: "RHSM on the RHEL7 boxes"
      block:
        - name: "RHSM | ensure system is registered with my activation key"
          redhat_subscription:
            activationkey:  "{{ rhsm_activationkey }}"
            org_id:         "{{ rhsm_org_id }}"
        - name: "RHSM | disable all repositories, next task will enable needed repos"
          rhsm_repository:
            name: '*'
            state: disabled
        - name: "RHSM | ensure RHEL7 BaseOS repos needed for Ceph are enabled"
          rhsm_repository:
            name:
              - rhel-7-server-rpms
              - rhel-7-server-extras-rpms
            state: enabled
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "7"
      tags: do_subsmgr_register

  tasks:
    # !!!
    #
    # if I ever enable EPEL, then I MUST exclude
    # - ansible
    # - ceph
    # in the EPEL repo files to ensure no newer versions of those packages are pulled in from EPEL
    #
    # exclude = *ceph* nfs-ganesha-rgw rbd-mirror *ansible*
    #
    # !!!
    #
    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/installation_guide/index#enabling-the-red-hat-ceph-storage-repositories-install
    - name: "REPOS | ensure EPEL is disabled"
      yum_repository:
        name: epel
        state: absent

    # start by enabling time sync, RHSM operations will fail on too large time delta
    # note that this uses chronyd, not ntpd.
    - name: "CHRONYD | ensure chrony is installed"
      package:
        name:       chrony
        state:      present
    - name:         "CHRONYD | ensure chrony-wait is enabled"
      service:
        name:       chrony-wait
        enabled:    true
    - name:         "CHRONYD | ensure chronyd is enabled and running"
      service:
        name:       chronyd
        enabled:    true
        state:      started
    
    # enable persistent journal
    - name: "JOURNAL | ensure persistent logging for the systemd journal is possible"
      file:
        path: /var/log/journal
        state: directory
        owner: root
        group: systemd-journal
        mode: u=rwx,g=rs,o=rx

    # 2.10. Enabling Password-less SSH for Ansible
    - name: "SUDO | enable passwordless sudo for ansible user"
      copy:
        dest: /etc/sudoers.d/ansible
        content: |
          ansible   ALL=NOPASSWD:   ALL
        owner: root
        group: root
        mode: 0440

    # I admit, the regexp is a search engine hit
    - name: "GRUB | ensure console blanking is disabled in defaults file"
      lineinfile:
        state: present
        dest: /etc/default/grub
        backrefs: yes
        regexp: '^(GRUB_CMDLINE_LINUX=(?!.* consoleblank)\"[^\"]+)(\".*)'
        line: '\1 consoleblank=0\2'

terramaster-general-setup.yml

Besides running some of my roles, it takes care of

  • ensuring Wake on LAN is set up for the 1Gig PCIe network interface
  • set up watchdog, rngd, tuned
  • ensuring cockbit is installed but disabled
  • ensuring libvirtd is not running
  • ensuring kdump is on
  • ensuring I can monitor the host with my Check MK instance
  • ensuring consoleblank=0 is set in my default boot options
  • ensuring I can kickstart the node again via a target in grub
  • ensuring the system has the latest updates applied

As user, on my F31 x96_64 workstation.

ansible-playbook -i ../inventories/ceph-F5-422-cluster.ini terramaster-general-setup.yml
---
- name: General setup of my Terramaster F5-422 Ceph nodes

  hosts:
    - TerraMaster_boxes

  become: true
  roles:
    - fedora.linux_system_roles.network
    - pcfe.user_owner
    - pcfe.basic_security_setup
    - pcfe.housenet
    - pcfe.comfort
#    - pcfe.checkmk

  handlers:
    - name: Handle running grub2-mkconfig for dist RedHat major vers 9
      ansible.builtin.command: grub2-mkconfig -o /boot/grub2/grub.cfg
      changed_when: true
    - name: Handle running grub2-mkconfig for dist RedHat major vers 8
      ansible.builtin.command: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
      changed_when: true

  tasks:
    # work around identical UUID on each host as per https://theforeman.org/plugins/katello/nightly/troubleshooting/content_hosts.html
    # The TerraMaster F5-422 boxes all have the same system-uuid :-(
    # RHSM does not like that, so override to avoid both
    # "HTTP error (409 - Conflict): Request failed due to concurrent modification, please re-try.\n"
    # and all 4 boxes overriding each other in insights inventory
    - name: "RHSM | ensure uuid override is derived from fqdn, F5-422 all have idential uuid in DMI"
      ansible.builtin.copy:
        dest: /etc/rhsm/facts/uuid_override.facts
        owner: root
        group: root
        mode: u=rw,g=r,o=r
        content: |
          {"dmi.system.uuid": "{{ ansible_fqdn | to_uuid }}"}

    # Ensure the packages that the RHEL8 only preflight play from cephadm-ansible would install
    # also ensure ceph-selinux is installed, docs say that's in the default set
    - name: Ensure needed packages for RHCS5 on RHEL9 are installed
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "9"
      ansible.builtin.package:
        name:
          - chrony
          - cephadm
          - podman
          - ceph-common
          - ceph-selinux
        state: present
        update_cache: false

    # Install some tools
    - name: "PACKAGE | tool installation, el8 and el9"
      ansible.builtin.package:
        name:
          - pciutils
          - usbutils
          - fio
          - powertop
          - tuned
          - tuned-utils
          - numactl
          - iperf3
          - tcpdump
          - hwloc-gui
          - fwupd
        state: present
        update_cache: false

    - name: "PACKAGE | tool installation el9"
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "9"
      ansible.builtin.package:
        name:
          - s-nail
        state: present
        update_cache: false

    - name: "PACKAGE | tool installation el8"
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
      ansible.builtin.package:
        name:
          - teamd
          - NetworkManager-team
          - mailx
        state: present
        update_cache: false

    # linux-system-roles.network sets static network config (from host_vars)
    # but I want the static hostname nailed down too
    #
    # FIXME; ansible_fqdn is a fact, so that does not help when it's a freshly kickstarted localhost.localdomain
    - name: Ensure static hostname is set to FQDN
      ansible.builtin.hostname:
        name: "{{ ansible_fqdn }}"
        use: systemd

    # enable WOL manually until https://github.com/linux-system-roles/network/issues/150 is fixed
    - name: Ensure Wake On LAN is enable for the upper network card
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
      ansible.builtin.lineinfile:
        path: /etc/sysconfig/network-scripts/ifcfg-1G_PCIe
        create: false
        regexp: '^ETHTOOL_OPTS= '
        insertafter: '^TYPE=Ethernet'
        line: 'ETHTOOL_OPTS="wol g"'

    # enable watchdog
    # it's a Jun 22 13:12:09 localhost kernel: sbsa-gwdt e0bb0000.gwdt: Initialized with 10s timeout @ 250000000 Hz, action=0.
    - name: "WATCHDOG | ensure kernel module wdat_wdt has correct options configured"
      ansible.builtin.lineinfile:
        path: /etc/modprobe.d/wdat_wdt.conf
        create: true
        regexp: '^options '
        insertafter: '^#options'
        line: 'options wdat_wdt nowayout=0'
        mode: u=rw,g=r,o=r

    # configure both watchdog.service and systemd watchdog, but only use the latter
    - name: "PACKAGE | ensure watchdog package is installed"
      ansible.builtin.package:
        name: watchdog
        state: present
        update_cache: false
    - name: "WATCHDOG | ensure correct watchdog-device is used by watchdog.service"
      ansible.builtin.lineinfile:
        path: /etc/watchdog.conf
        regexp: '^watchdog-device'
        insertafter: '^#watchdog-device'
        line: 'watchdog-device = /dev/watchdog'
    - name: "WATCHDOG | ensure timeout is set to 30 seconds for watchdog.service"
      ansible.builtin.lineinfile:
        path: /etc/watchdog.conf
        regexp: '^watchdog-timeout'
        insertafter: '^#watchdog-timeout'
        line: 'watchdog-timeout = 30'
    # Using systemd watchdog rather than watchdog.service
    - name: "WATCHDOG | ensure watchdog.service is disabled"
      ansible.builtin.systemd:
        name: watchdog.service
        state: stopped
        enabled: false
    # configure systemd watchdog
    # c.f. http://0pointer.de/blog/projects/watchdog.html
    - name: "SYSTEMD | ensure systemd watchdog is enabled"
      ansible.builtin.lineinfile:
        path: /etc/systemd/system.conf
        regexp: '^RuntimeWatchdogSec'
        insertafter: 'EOF'
        line: 'RuntimeWatchdogSec=30'
    - name: "SYSTEMD | ensure systemd shutdown watchdog is enabled"
      ansible.builtin.lineinfile:
        path: /etc/systemd/system.conf
        regexp: '^ShutdownWatchdogSec'
        insertafter: 'EOF'
        line: 'ShutdownWatchdogSec=30'

    # install and enable rngd
    - name: "PACKAGE | ensure rng-tools package is installed"
      ansible.builtin.package:
        name: rng-tools
        state: present
        update_cache: false
    - name: "RNGD | ensure rngd.service is enabled and started"
      ansible.builtin.systemd:
        name: rngd.service
        state: started
        enabled: true

    # ensure tuned is set up as I wish
    - name: "TUNED | ensure tuned.service is enabled and running"
      ansible.builtin.systemd:
        name: tuned.service
        state: started
        enabled: true
    - name: "TUNED | check which tuned profile is active"
      ansible.builtin.command: tuned-adm active
      register: tuned_active_profile
      ignore_errors: true
      changed_when: false
    - name: "TUNED | activate tuned profile {{ tuned_profile }}"
      ansible.builtin.command: "tuned-adm profile {{ tuned_profile }}"
      when: not tuned_active_profile.stdout is search('Current active profile:' ~ ' ' ~ tuned_profile)
      changed_when: true

    # install cockpit, but disabled for now
    - name: "COCKPIT | ensure packages for https://cockpit-project.org/ are installed"
      ansible.builtin.package:
        name:
          - cockpit
          - cockpit-selinux
          - cockpit-kdump
          - cockpit-storaged
          - cockpit-system
        state: present
        update_cache: false
    - name: "COCKPIT | ensure cockpit.socket is stopped and disabled"
      ansible.builtin.systemd:
        name: cockpit.socket
        state: stopped
        enabled: false
    - name: "COCKPIT | ensure firewalld forbids service cockpit in zone public"
      ansible.posix.firewalld:
        service: cockpit
        zone: public
        permanent: true
        state: disabled
        immediate: true

    # disable libvirtd
    - name: "Ensure libvirtd.service is disabled and stopped"
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
      ansible.builtin.systemd:
        name: libvirtd.service
        state: stopped
        enabled: false

    # enable kdump.service since kickstart now creates a sufficiently large /var/crash
    # alternatively, you could set up netdump
    - name: "Ensure kdump.service is enabled and started"
      ansible.builtin.systemd:
        name: kdump.service
        state: started
        enabled: true

    # podman
    - name: "PACKAGE | ensure podman is installed"
      ansible.builtin.package:
        name:
          - podman
          - podman-docker
        state: present

    # setroubleshoot, see also https://danwalsh.livejournal.com/20931.html
    - name: "PACKAGE | ensure setroubleshoot for headless server is installed"
      ansible.builtin.package:
        name:
          - setroubleshoot-server
          - setroubleshoot-plugins
        state: present

    - name: "MONITORING | ensure packages for monitoring are installed"
      ansible.builtin.package:
        name:
          - smartmontools
          - hdparm
          - check-mk-agent
          - lm_sensors
        state: present

    - name: "MONITORING | ensure firewalld permits 6556 in zone public for check-mk-agent"
      ansible.posix.firewalld:
        port: 6556/tcp
        permanent: true
        state: enabled
        immediate: true
        zone: public
    - name: "MONITORING | ensure tarsnap cache is in fileinfo"
      ansible.builtin.lineinfile:
        path: /etc/check_mk/fileinfo.cfg
        line: "/usr/local/tarsnap-cache/cache"
        create: true
        mode: u=rw,g=r,o=r
    - name: "MONITORING | ensure entropy_avail plugin for Check_MK is present"
      ansible.builtin.template:
        src: templates/check-mk-agent-plugin-entropy_avail.j2
        dest: /usr/lib/check_mk_agent/plugins/entropy_avail
        mode: u=rwx,g=rx,o=rx
        group: root
        owner: root
    - name: "MONITORING | plugins from running CEE instance"
      ansible.builtin.get_url:
        url: "http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/plugins/{{ item }}"
        dest: "/usr/lib/check_mk_agent/plugins/{{ item }}"
        mode: u=rwx,g=rx,o=rx
      loop:
        - smart
        - lvm
    - name: "MONITORING | ensure check_mk.socket is started and enabled"
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
      ansible.builtin.systemd:
        name: check_mk.socket
        state: started
        enabled: true

    - name: "MONITORING | ensure check-mk-agent.socket is started and enabled"
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "9"
      ansible.builtin.systemd:
        name: check-mk-agent.socket
        state: started
        enabled: true


    # I admit, the regexp is a search engine hit
    # maybe using grubby(8) would be more readable
    # yamllint disable rule:line-length
    # - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/configuring-kernel-command-line-parameters_managing-monitoring-and-updating-the-kernel#what-is-grubby_configuring-kernel-command-line-parameters
    # - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sec-Making_Persistent_Changes_to_a_GRUB_2_Menu_Using_the_grubby_Tool
    # yamllint enable rule:line-length
    - name: "GRUB | ensure console blanking is disabled in defaults file plus handler"
      ansible.builtin.lineinfile:
        state: present
        dest: /etc/default/grub
        backrefs: true
        regexp: '^(GRUB_CMDLINE_LINUX=(?!.* consoleblank)\"[^\"]+)(\".*)'
        line: '\1 consoleblank=0\2'
      notify: "Handle running grub2-mkconfig for dist {{ ansible_distribution }} major vers {{ ansible_distribution_major_version }}"

    # Since I do not manage to get these F5-422 to PXE boot, add an entry into grub
    # so that I can kickstart the box after this without fiddling with a USB stick
    - name: "GRUB | ensure initrd for RHEL 8.9 kickstart is present"
      ansible.builtin.get_url:
        url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.9/Server/x86_64/os/images/pxeboot/initrd.img"
        dest: "/boot/initrd-kickstart-rhel89.img"
        mode: u=rw
    - name: "GRUB | ensure kernel for RHEL 8.9 kickstart is present"
      ansible.builtin.get_url:
        url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.9/Server/x86_64/os/images/pxeboot/vmlinuz"
        dest: "/boot/vmlinuz-kickstart-rhel89"
        mode: u=rwx,g=rx,o=rx
    # yamllint disable rule:line-length
    - name: "GRUB | ensure kickstarting RHEL 8.9 entry is present"
      ansible.builtin.copy:
        dest: "/etc/grub.d/11_RHEL8x_kickstart"
        owner: "root"
        group: "root"
        mode: u=rwx,g=rx,o=rx
        content: |
          #!/bin/sh
          exec tail -n +3 $0
          # This file provides an easy way to add custom menu entries.  Simply type the
          # menu entries you want to add after this comment.  Be careful not to change
          # the 'exec tail' line above.
          menuentry "WARNING Kickstart this box with RHEL 8.9 as a F5-422 ceph node WARNING" {
              linuxefi /vmlinuz-kickstart-rhel89 ip=dhcp inst.repo=ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.9/Server/x86_64/os inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/RHEL89-TerraMaster-F5-422-ks.cfg
              initrdefi /initrd-kickstart-rhel89.img
          }
      notify: "Handle running grub2-mkconfig for dist {{ ansible_distribution }} major vers {{ ansible_distribution_major_version }}"
    # yamllint enable rule:line-length

    - name: "GRUB | ensure initrd for RHEL 9.3 kickstart is present"
      ansible.builtin.get_url:
        url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-9.3/Server/x86_64/os/images/pxeboot/initrd.img"
        dest: "/boot/initrd-kickstart-rhel93.img"
        mode: u=rw
    - name: "GRUB | ensure kernel for RHEL 9.3 kickstart is present"
      ansible.builtin.get_url:
        url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-9.3/Server/x86_64/os/images/pxeboot/vmlinuz"
        dest: "/boot/vmlinuz-kickstart-rhel93"
        mode: u=rwx,g=rx,o=rx
    # yamllint disable rule:line-length
    - name: "GRUB | ensure kickstarting RHEL 9.3 entry is present"
      ansible.builtin.copy:
        dest: "/etc/grub.d/12_RHEL9x_kickstart"
        owner: "root"
        group: "root"
        mode: u=rwx,g=rx,o=rx
        content: |
          #!/bin/sh
          exec tail -n +3 $0
          # This file provides an easy way to add custom menu entries.  Simply type the
          # menu entries you want to add after this comment.  Be careful not to change
          # the 'exec tail' line above.
          menuentry "WARNING Kickstart this box with RHEL 9.3 as a F5-422 ceph node WARNING" {
              linuxefi /vmlinuz-kickstart-rhel93 ip=dhcp inst.repo=ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-9.3/Server/x86_64/os inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/RHEL93-TerraMaster-F5-422-ks.cfg
              initrdefi /initrd-kickstart-rhel93.img
          }
      notify: "Handle running grub2-mkconfig for dist {{ ansible_distribution }} major vers {{ ansible_distribution_major_version }}"
    # yamllint enable rule:line-length

    # upgrade the box
    - name: Package | ensure all updates are applied
      ansible.builtin.package:
        update_cache: true
        name: '*'
        state: latest  # noqa package-latest
      tags: apply_errata

test forcefully crashing the box.

Verify that the watchdog works as expected.

As root, on a F5-422:

echo '1' > /proc/sys/kernel/sysrq
echo 'c' > /proc/sysrq-trigger

As expected, I get an oops output on the console after the c and, as previously tested, the box reboots itself after 30 seconds.

Hardware Info

dmidecode -t baseboard

[root@f5-422-01 ~]# dmidecode -t baseboard
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
	Manufacturer: GIFA Industrial Control
	Product Name: TM-J3455-4G3L
	Version: Default string
	Serial Number: Default string
	Asset Tag: Default string
	Features:
		Board is a hosting board
		Board is replaceable
	Location In Chassis: Default string
	Chassis Handle: 0x0003
	Type: Motherboard
	Contained Object Handles: 0

Handle 0x0025, DMI type 10, 6 bytes
On Board Device Information
	Type: Video
	Status: Enabled
	Description:    To Be Filled By O.E.M.

Handle 0x002D, DMI type 41, 11 bytes
Onboard Device
	Reference Designation:  Onboard IGD
	Type: Video
	Status: Enabled
	Type Instance: 1
	Bus Address: 0000:00:02.0

Handle 0x002E, DMI type 41, 11 bytes
Onboard Device
	Reference Designation:  Onboard LAN
	Type: Ethernet
	Status: Enabled
	Type Instance: 1
	Bus Address: 0000:00:19.0

Handle 0x002F, DMI type 41, 11 bytes
Onboard Device
	Reference Designation:  Onboard 1394
	Type: Other
	Status: Enabled
	Type Instance: 1
	Bus Address: 0000:03:1c.2

lspci

[root@f5-422-04 ~]# lspci
00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
00:02.0 VGA compatible controller: Intel Corporation Device 5a85 (rev 0b)
00:0e.0 Audio device: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster (rev 0b)
00:0f.0 Communication controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine (rev 0b)
00:12.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b)
00:13.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 (rev fb)
00:13.2 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3 (rev fb)
00:13.3 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #4 (rev fb)
00:14.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port B #1 (rev fb)
00:14.1 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port B #2 (rev fb)
00:15.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b)
00:1a.0 Serial bus controller [0c80]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PWM Pin Controller (rev 0b)
00:1f.0 ISA bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface (rev 0b)
00:1f.1 SMBus: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller (rev 0b)
01:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)

lsusb

[root@f5-422-04 ~]# lsusb
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13fe:4100 Kingston Technology Company Inc. Flash drive
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

lsusb -t

[root@f5-422-04 ~]# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/7p, 5000M
    |__ Port 3: Dev 3, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M

lscpu

[root@f5-422-04 ~]# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               92
Model name:          Intel(R) Celeron(R) CPU J3455 @ 1.50GHz
Stepping:            9
CPU MHz:             892.928
CPU max MHz:         2300.0000
CPU min MHz:         800.0000
BogoMIPS:            2995.20
Virtualization:      VT-x
L1d cache:           24K
L1i cache:           32K
L2 cache:            1024K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault cat_l2 ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust smep erms mpx rdt_a rdseed smap clflushopt intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts md_clear arch_capabilities

free -m and free -h

[root@f5-422-04 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:          11691         429        9855           1        1406       10980
Swap:          4095           0        4095
[root@f5-422-04 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           11Gi       430Mi       9.6Gi       1.0Mi       1.4Gi        10Gi
Swap:         4.0Gi          0B       4.0Gi

dmidecode -t memory

[root@f5-422-04 ~]# dmidecode -t memory
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.

Handle 0x0030, DMI type 16, 23 bytes
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: None
	Maximum Capacity: 32 GB
	Error Information Handle: Not Provided
	Number Of Devices: 4

Handle 0x0031, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: 64 bits
	Data Width: 64 bits
	Size: 4096 MB
	Form Factor: SODIMM
	Set: None
	Locator: ChannelA-DIMM0
	Bank Locator: BANK 0
	Type: DDR3
	Type Detail: Synchronous
	Speed: 1600 MT/s
	Manufacturer: Undefi
	Serial Number: 88000000
	Asset Tag: 9876543210
	Part Number:                /  
	Rank: Unknown
	Configured Memory Speed: 1600 MT/s
	Minimum Voltage: 44.975 V
	Maximum Voltage: 44.975 V
	Configured Voltage: 1.5 V

Handle 0x0032, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: 64 bits
	Data Width: 64 bits
	Size: 8192 MB
	Form Factor: SODIMM
	Set: None
	Locator: ChannelB-DIMM0
	Bank Locator: BANK 1
	Type: DDR3
	Type Detail: Synchronous
	Speed: 1600 MT/s
	Manufacturer: Unde
	Serial Number: [REDACTED]
	Asset Tag: 9876543210
	Part Number: CT102464BF160B.C16
	Rank: Unknown
	Configured Memory Speed: 1600 MT/s
	Minimum Voltage: 1.35 V
	Maximum Voltage: 1.5 V
	Configured Voltage: 1.5 V

Handle 0x0033, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: Unknown
	Data Width: Unknown
	Size: No Module Installed
	Form Factor: DIMM
	Set: None
	Locator: ChannelC-DIMM0
	Bank Locator: BANK 2
	Type: Unknown
	Type Detail: None
	Speed: Unknown
	Manufacturer: Not Specified
	Serial Number: Not Specified
	Asset Tag: 9876543210
	Part Number: Not Specified
	Rank: Unknown
	Configured Memory Speed: 1600 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x0034, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: Unknown
	Data Width: Unknown
	Size: No Module Installed
	Form Factor: DIMM
	Set: None
	Locator: ChannelD-DIMM0
	Bank Locator: BANK 3
	Type: Unknown
	Type Detail: None
	Speed: Unknown
	Manufacturer: Not Specified
	Serial Number: Not Specified
	Asset Tag: 9876543210
	Part Number: Not Specified
	Rank: Unknown
	Configured Memory Speed: 1600 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

The vendor’s filling of the DMI registers does leave to be desired…

	Minimum Voltage: 44.975 V
	Maximum Voltage: 44.975 V
	Configured Voltage: 1.5 V

yeah, right. But well, Configured Voltage is right.

Network Devices

upper NIC, 1Gig, PCIe

[root@f5-422-04 ~]# ethtool -i enp2s0
driver: r8169
version: 
firmware-version: rtl8168h-2_0.0.2 02/26/15
expansion-rom-version: 
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@f5-422-04 ~]# lspci -s 02:00.0
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

middle NIC, 1Gig, USB

[root@f5-422-04 ~]# ethtool -i enp0s21f0u3
driver: r8152
version: v1.09.9
firmware-version: 
expansion-rom-version: 
bus-info: usb-0000:00:15.0-3
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
[root@f5-422-04 ~]# lsusb -s 002:003
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter

lower NIC, 10Gig, PCIe

[root@f5-422-04 ~]# ethtool -i enp1s0
driver: atlantic
version: 2.0.4.0-kern
firmware-version: 3.1.57
expansion-rom-version: 
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@f5-422-04 ~]# lspci -s 01:00.0
01:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)

iperf3 between 2 10G NICs

FIXME: get new measurements. These are from CentOS Stream

In c’t 02/2020 S. 102 they were measured at roughly 7 GBit/s (Article here but paywalled. As a subscriber I just read it on paper.)

7,0 gigabit are 6,07 gibibit, so I seem to be pretty close with FIXME on jumbo frames.

Setup of iperf3

[root@f5-422-01 ~]# firewall-cmd --add-port 5201/tcp
success
[root@f5-422-01 ~]# iperf3 --server --bind 192.168.30.181

MTU 1500

With the default MTU, I get 5.4 Gbits/sec out of the interfaces.

iperf3 -c 192.168.30.181 -i 1 -t 20  -P 4 | tee $(date +%F)-iperf3-mtu1500-4parallel.log
Connecting to host 192.168.30.181, port 5201
[  5] local 192.168.30.182 port 39590 connected to 192.168.30.181 port 5201
[  7] local 192.168.30.182 port 39592 connected to 192.168.30.181 port 5201
[  9] local 192.168.30.182 port 39594 connected to 192.168.30.181 port 5201
[ 11] local 192.168.30.182 port 39596 connected to 192.168.30.181 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   949 Mbits/sec    0    585 KBytes       
[  7]   0.00-1.00   sec   215 MBytes  1.80 Gbits/sec    0    642 KBytes       
[  9]   0.00-1.00   sec   214 MBytes  1.80 Gbits/sec    0    634 KBytes       
[ 11]   0.00-1.00   sec   115 MBytes   961 Mbits/sec    0    665 KBytes       
[SUM]   0.00-1.00   sec   657 MBytes  5.51 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   119 MBytes   995 Mbits/sec    0    643 KBytes       
[  7]   1.00-2.00   sec   204 MBytes  1.71 Gbits/sec    0    679 KBytes       
[  9]   1.00-2.00   sec   202 MBytes  1.70 Gbits/sec    0    672 KBytes       
[ 11]   1.00-2.00   sec   120 MBytes  1.01 Gbits/sec    0    696 KBytes       
[SUM]   1.00-2.00   sec   645 MBytes  5.41 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec    0    643 KBytes       
[  7]   2.00-3.00   sec   215 MBytes  1.80 Gbits/sec    0    713 KBytes       
[  9]   2.00-3.00   sec   212 MBytes  1.78 Gbits/sec    0    707 KBytes       
[ 11]   2.00-3.00   sec   111 MBytes   933 Mbits/sec    0    696 KBytes       
[SUM]   2.00-3.00   sec   650 MBytes  5.45 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   108 MBytes   902 Mbits/sec    0    643 KBytes       
[  7]   3.00-4.00   sec   216 MBytes  1.81 Gbits/sec    0    713 KBytes       
[  9]   3.00-4.00   sec   216 MBytes  1.81 Gbits/sec    0    707 KBytes       
[ 11]   3.00-4.00   sec   108 MBytes   902 Mbits/sec    0    696 KBytes       
[SUM]   3.00-4.00   sec   648 MBytes  5.43 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   109 MBytes   912 Mbits/sec    0    643 KBytes       
[  7]   4.00-5.00   sec   216 MBytes  1.81 Gbits/sec    0    713 KBytes       
[  9]   4.00-5.00   sec   216 MBytes  1.81 Gbits/sec    0    707 KBytes       
[ 11]   4.00-5.00   sec   109 MBytes   912 Mbits/sec    0    696 KBytes       
[SUM]   4.00-5.00   sec   650 MBytes  5.45 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   110 MBytes   922 Mbits/sec    0    710 KBytes       
[  7]   5.00-6.00   sec   210 MBytes  1.76 Gbits/sec    0    789 KBytes       
[  9]   5.00-6.00   sec   210 MBytes  1.76 Gbits/sec    0    786 KBytes       
[ 11]   5.00-6.00   sec   110 MBytes   922 Mbits/sec    0    776 KBytes       
[SUM]   5.00-6.00   sec   640 MBytes  5.37 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   108 MBytes   902 Mbits/sec    0    803 KBytes       
[  7]   6.00-7.00   sec   215 MBytes  1.80 Gbits/sec    0    885 KBytes       
[  9]   6.00-7.00   sec   214 MBytes  1.79 Gbits/sec    0    875 KBytes       
[ 11]   6.00-7.00   sec   108 MBytes   902 Mbits/sec    0    875 KBytes       
[SUM]   6.00-7.00   sec   644 MBytes  5.40 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   109 MBytes   913 Mbits/sec    0    803 KBytes       
[  7]   7.00-8.00   sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]   7.00-8.00   sec   216 MBytes  1.81 Gbits/sec    0    875 KBytes       
[ 11]   7.00-8.00   sec   109 MBytes   913 Mbits/sec    0    875 KBytes       
[SUM]   7.00-8.00   sec   650 MBytes  5.46 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   109 MBytes   912 Mbits/sec    0    803 KBytes       
[  7]   8.00-9.00   sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]   8.00-9.00   sec   216 MBytes  1.81 Gbits/sec    0    875 KBytes       
[ 11]   8.00-9.00   sec   109 MBytes   912 Mbits/sec    0    875 KBytes       
[SUM]   8.00-9.00   sec   650 MBytes  5.45 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   108 MBytes   902 Mbits/sec    0    803 KBytes       
[  7]   9.00-10.00  sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]   9.00-10.00  sec   218 MBytes  1.82 Gbits/sec    0    875 KBytes       
[ 11]   9.00-10.00  sec   108 MBytes   902 Mbits/sec    0    875 KBytes       
[SUM]   9.00-10.00  sec   649 MBytes  5.44 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-11.00  sec   109 MBytes   912 Mbits/sec    0    803 KBytes       
[  7]  10.00-11.00  sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]  10.00-11.00  sec   216 MBytes  1.81 Gbits/sec    0    875 KBytes       
[ 11]  10.00-11.00  sec   109 MBytes   912 Mbits/sec    0    875 KBytes       
[SUM]  10.00-11.00  sec   650 MBytes  5.45 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  11.00-12.00  sec   108 MBytes   902 Mbits/sec    0    803 KBytes       
[  7]  11.00-12.00  sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]  11.00-12.00  sec   216 MBytes  1.81 Gbits/sec    0    875 KBytes       
[ 11]  11.00-12.00  sec   108 MBytes   902 Mbits/sec    0    875 KBytes       
[SUM]  11.00-12.00  sec   648 MBytes  5.43 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  12.00-13.00  sec   109 MBytes   912 Mbits/sec    0    803 KBytes       
[  7]  12.00-13.00  sec   218 MBytes  1.82 Gbits/sec    0    885 KBytes       
[  9]  12.00-13.00  sec   216 MBytes  1.81 Gbits/sec    0    875 KBytes       
[ 11]  12.00-13.00  sec   109 MBytes   912 Mbits/sec    0    875 KBytes       
[SUM]  12.00-13.00  sec   651 MBytes  5.46 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  13.00-14.00  sec   108 MBytes   902 Mbits/sec    0    803 KBytes       
[  7]  13.00-14.00  sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]  13.00-14.00  sec   216 MBytes  1.81 Gbits/sec    0    875 KBytes       
[ 11]  13.00-14.00  sec   108 MBytes   902 Mbits/sec    0    875 KBytes       
[SUM]  13.00-14.00  sec   648 MBytes  5.43 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  14.00-15.00  sec   109 MBytes   912 Mbits/sec    0    803 KBytes       
[  7]  14.00-15.00  sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]  14.00-15.00  sec   216 MBytes  1.81 Gbits/sec    0   1.28 MBytes       
[ 11]  14.00-15.00  sec   109 MBytes   912 Mbits/sec    0    875 KBytes       
[SUM]  14.00-15.00  sec   650 MBytes  5.45 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  15.00-16.00  sec   109 MBytes   912 Mbits/sec    0    803 KBytes       
[  7]  15.00-16.00  sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]  15.00-16.00  sec   218 MBytes  1.82 Gbits/sec    0   1.28 MBytes       
[ 11]  15.00-16.00  sec   109 MBytes   912 Mbits/sec    0    875 KBytes       
[SUM]  15.00-16.00  sec   651 MBytes  5.46 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  16.00-17.00  sec   108 MBytes   902 Mbits/sec    0    803 KBytes       
[  7]  16.00-17.00  sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]  16.00-17.00  sec   216 MBytes  1.81 Gbits/sec    0   1.28 MBytes       
[ 11]  16.00-17.00  sec   108 MBytes   902 Mbits/sec    0    875 KBytes       
[SUM]  16.00-17.00  sec   648 MBytes  5.43 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  17.00-18.00  sec   109 MBytes   912 Mbits/sec    0    803 KBytes       
[  7]  17.00-18.00  sec   218 MBytes  1.82 Gbits/sec    0    885 KBytes       
[  9]  17.00-18.00  sec   216 MBytes  1.81 Gbits/sec    0   1.28 MBytes       
[ 11]  17.00-18.00  sec   109 MBytes   912 Mbits/sec    0    875 KBytes       
[SUM]  17.00-18.00  sec   651 MBytes  5.46 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  18.00-19.00  sec   109 MBytes   913 Mbits/sec    0    803 KBytes       
[  7]  18.00-19.00  sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]  18.00-19.00  sec   216 MBytes  1.81 Gbits/sec    0   1.28 MBytes       
[ 11]  18.00-19.00  sec   109 MBytes   913 Mbits/sec    0    875 KBytes       
[SUM]  18.00-19.00  sec   650 MBytes  5.45 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  19.00-20.00  sec   108 MBytes   902 Mbits/sec    0    803 KBytes       
[  7]  19.00-20.00  sec   216 MBytes  1.81 Gbits/sec    0    885 KBytes       
[  9]  19.00-20.00  sec   216 MBytes  1.81 Gbits/sec    0   1.28 MBytes       
[ 11]  19.00-20.00  sec   108 MBytes   902 Mbits/sec    0    875 KBytes       
[SUM]  19.00-20.00  sec   648 MBytes  5.43 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.13 GBytes   916 Mbits/sec    0             sender
[  5]   0.00-20.02  sec  2.13 GBytes   914 Mbits/sec                  receiver
[  7]   0.00-20.00  sec  4.20 GBytes  1.81 Gbits/sec    0             sender
[  7]   0.00-20.02  sec  4.20 GBytes  1.80 Gbits/sec                  receiver
[  9]   0.00-20.00  sec  4.20 GBytes  1.80 Gbits/sec    0             sender
[  9]   0.00-20.02  sec  4.20 GBytes  1.80 Gbits/sec                  receiver
[ 11]   0.00-20.00  sec  2.14 GBytes   917 Mbits/sec    0             sender
[ 11]   0.00-20.02  sec  2.13 GBytes   915 Mbits/sec                  receiver
[SUM]   0.00-20.00  sec  12.7 GBytes  5.44 Gbits/sec    0             sender
[SUM]   0.00-20.02  sec  12.7 GBytes  5.43 Gbits/sec                  receiver

iperf Done.

MTU 9000

With jumbo frames, I get 5.8 Gbits/sec out of the interfaces.

iperf3 -c 192.168.30.181 -i 1 -t 20  -P 4 | tee $(date +%F)-iperf3-mtu9000-4parallel.log
Connecting to host 192.168.30.181, port 5201
[  5] local 192.168.30.182 port 39576 connected to 192.168.30.181 port 5201
[  7] local 192.168.30.182 port 39578 connected to 192.168.30.181 port 5201
[  9] local 192.168.30.182 port 39580 connected to 192.168.30.181 port 5201
[ 11] local 192.168.30.182 port 39582 connected to 192.168.30.181 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   953 Mbits/sec    0    585 KBytes       
[  7]   0.00-1.00   sec   234 MBytes  1.96 Gbits/sec    0    673 KBytes       
[  9]   0.00-1.00   sec   233 MBytes  1.96 Gbits/sec    0    699 KBytes       
[ 11]   0.00-1.00   sec   121 MBytes  1.01 Gbits/sec    0    559 KBytes       
[SUM]   0.00-1.00   sec   701 MBytes  5.88 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   121 MBytes  1.02 Gbits/sec    0    585 KBytes       
[  7]   1.00-2.00   sec   231 MBytes  1.94 Gbits/sec    0    673 KBytes       
[  9]   1.00-2.00   sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]   1.00-2.00   sec   111 MBytes   929 Mbits/sec    0    612 KBytes       
[SUM]   1.00-2.00   sec   694 MBytes  5.82 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   119 MBytes  1000 Mbits/sec    0    673 KBytes       
[  7]   2.00-3.00   sec   229 MBytes  1.92 Gbits/sec    0    708 KBytes       
[  9]   2.00-3.00   sec   229 MBytes  1.92 Gbits/sec    0    699 KBytes       
[ 11]   2.00-3.00   sec   118 MBytes   988 Mbits/sec    0    699 KBytes       
[SUM]   2.00-3.00   sec   695 MBytes  5.83 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   135 MBytes  1.13 Gbits/sec    0    673 KBytes       
[  7]   3.00-4.00   sec   212 MBytes  1.78 Gbits/sec    0    708 KBytes       
[  9]   3.00-4.00   sec   213 MBytes  1.78 Gbits/sec    0    699 KBytes       
[ 11]   3.00-4.00   sec   136 MBytes  1.14 Gbits/sec    0    699 KBytes       
[SUM]   3.00-4.00   sec   695 MBytes  5.83 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   115 MBytes   968 Mbits/sec    0    673 KBytes       
[  7]   4.00-5.00   sec   232 MBytes  1.94 Gbits/sec    0    708 KBytes       
[  9]   4.00-5.00   sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]   4.00-5.00   sec   115 MBytes   967 Mbits/sec    0    699 KBytes       
[SUM]   4.00-5.00   sec   694 MBytes  5.82 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   116 MBytes   972 Mbits/sec    0    673 KBytes       
[  7]   5.00-6.00   sec   230 MBytes  1.93 Gbits/sec    0    708 KBytes       
[  9]   5.00-6.00   sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]   5.00-6.00   sec   116 MBytes   969 Mbits/sec    0    699 KBytes       
[SUM]   5.00-6.00   sec   693 MBytes  5.82 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   116 MBytes   970 Mbits/sec    0    673 KBytes       
[  7]   6.00-7.00   sec   231 MBytes  1.94 Gbits/sec    0    708 KBytes       
[  9]   6.00-7.00   sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]   6.00-7.00   sec   115 MBytes   967 Mbits/sec    0    699 KBytes       
[SUM]   6.00-7.00   sec   693 MBytes  5.82 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   116 MBytes   972 Mbits/sec    0    673 KBytes       
[  7]   7.00-8.00   sec   231 MBytes  1.94 Gbits/sec    0    708 KBytes       
[  9]   7.00-8.00   sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]   7.00-8.00   sec   116 MBytes   973 Mbits/sec    0    699 KBytes       
[SUM]   7.00-8.00   sec   694 MBytes  5.82 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   115 MBytes   969 Mbits/sec    0    673 KBytes       
[  7]   8.00-9.00   sec   231 MBytes  1.94 Gbits/sec    0    708 KBytes       
[  9]   8.00-9.00   sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]   8.00-9.00   sec   115 MBytes   964 Mbits/sec    0    699 KBytes       
[SUM]   8.00-9.00   sec   692 MBytes  5.81 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   115 MBytes   968 Mbits/sec    0    673 KBytes       
[  7]   9.00-10.00  sec   231 MBytes  1.94 Gbits/sec    0    708 KBytes       
[  9]   9.00-10.00  sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]   9.00-10.00  sec   116 MBytes   970 Mbits/sec    0    699 KBytes       
[SUM]   9.00-10.00  sec   693 MBytes  5.81 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-11.00  sec   116 MBytes   972 Mbits/sec    0    673 KBytes       
[  7]  10.00-11.00  sec   232 MBytes  1.94 Gbits/sec    0    708 KBytes       
[  9]  10.00-11.00  sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]  10.00-11.00  sec   116 MBytes   974 Mbits/sec    0    699 KBytes       
[SUM]  10.00-11.00  sec   695 MBytes  5.83 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  11.00-12.00  sec   115 MBytes   962 Mbits/sec    0    673 KBytes       
[  7]  11.00-12.00  sec   230 MBytes  1.93 Gbits/sec    0    708 KBytes       
[  9]  11.00-12.00  sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]  11.00-12.00  sec   115 MBytes   967 Mbits/sec    0    699 KBytes       
[SUM]  11.00-12.00  sec   691 MBytes  5.80 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  12.00-13.00  sec   116 MBytes   970 Mbits/sec    0    673 KBytes       
[  7]  12.00-13.00  sec   232 MBytes  1.94 Gbits/sec    0    708 KBytes       
[  9]  12.00-13.00  sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]  12.00-13.00  sec   116 MBytes   969 Mbits/sec    0    699 KBytes       
[SUM]  12.00-13.00  sec   694 MBytes  5.82 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  13.00-14.00  sec   116 MBytes   974 Mbits/sec    0    673 KBytes       
[  7]  13.00-14.00  sec   231 MBytes  1.94 Gbits/sec    0    708 KBytes       
[  9]  13.00-14.00  sec   232 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]  13.00-14.00  sec   116 MBytes   970 Mbits/sec    0    699 KBytes       
[SUM]  13.00-14.00  sec   694 MBytes  5.83 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  14.00-15.00  sec   115 MBytes   967 Mbits/sec    0    673 KBytes       
[  7]  14.00-15.00  sec   232 MBytes  1.94 Gbits/sec    0    708 KBytes       
[  9]  14.00-15.00  sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]  14.00-15.00  sec   115 MBytes   966 Mbits/sec    0    699 KBytes       
[SUM]  14.00-15.00  sec   693 MBytes  5.81 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  15.00-16.00  sec   117 MBytes   978 Mbits/sec    0    673 KBytes       
[  7]  15.00-16.00  sec   231 MBytes  1.93 Gbits/sec    0    708 KBytes       
[  9]  15.00-16.00  sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]  15.00-16.00  sec   116 MBytes   974 Mbits/sec    0    699 KBytes       
[SUM]  15.00-16.00  sec   694 MBytes  5.82 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  16.00-17.00  sec   115 MBytes   968 Mbits/sec    0    673 KBytes       
[  7]  16.00-17.00  sec   231 MBytes  1.93 Gbits/sec    0    708 KBytes       
[  9]  16.00-17.00  sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]  16.00-17.00  sec   116 MBytes   974 Mbits/sec    0    699 KBytes       
[SUM]  16.00-17.00  sec   694 MBytes  5.82 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  17.00-18.00  sec   116 MBytes   972 Mbits/sec    0    673 KBytes       
[  7]  17.00-18.00  sec   232 MBytes  1.95 Gbits/sec    0    708 KBytes       
[  9]  17.00-18.00  sec   231 MBytes  1.93 Gbits/sec    0    699 KBytes       
[ 11]  17.00-18.00  sec   115 MBytes   968 Mbits/sec    0    699 KBytes       
[SUM]  17.00-18.00  sec   694 MBytes  5.82 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  18.00-19.00  sec   116 MBytes   970 Mbits/sec    0    673 KBytes       
[  7]  18.00-19.00  sec   230 MBytes  1.93 Gbits/sec    0    708 KBytes       
[  9]  18.00-19.00  sec   232 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]  18.00-19.00  sec   116 MBytes   970 Mbits/sec    0    699 KBytes       
[SUM]  18.00-19.00  sec   693 MBytes  5.82 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  19.00-20.00  sec   115 MBytes   968 Mbits/sec    0    673 KBytes       
[  7]  19.00-20.00  sec   231 MBytes  1.94 Gbits/sec    0    708 KBytes       
[  9]  19.00-20.00  sec   231 MBytes  1.94 Gbits/sec    0    699 KBytes       
[ 11]  19.00-20.00  sec   116 MBytes   971 Mbits/sec    0    699 KBytes       
[SUM]  19.00-20.00  sec   693 MBytes  5.81 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.28 GBytes   981 Mbits/sec    0             sender
[  5]   0.00-20.02  sec  2.28 GBytes   979 Mbits/sec                  receiver
[  7]   0.00-20.00  sec  4.50 GBytes  1.93 Gbits/sec    0             sender
[  7]   0.00-20.02  sec  4.49 GBytes  1.93 Gbits/sec                  receiver
[  9]   0.00-20.00  sec  4.50 GBytes  1.93 Gbits/sec    0             sender
[  9]   0.00-20.02  sec  4.49 GBytes  1.93 Gbits/sec                  receiver
[ 11]   0.00-20.00  sec  2.28 GBytes   979 Mbits/sec    0             sender
[ 11]   0.00-20.02  sec  2.28 GBytes   977 Mbits/sec                  receiver
[SUM]   0.00-20.00  sec  13.6 GBytes  5.82 Gbits/sec    0             sender
[SUM]   0.00-20.02  sec  13.5 GBytes  5.81 Gbits/sec                  receiver

iperf Done.