Install four TerraMaster F5-422 with CentOS Stream
Table of Contents
I received my 3 additional TerraMaster F5-422.
Time to install CentOS Stream in preparation for Ceph Luminous.
Please also see the earlier post TerraMaster F5-422 with CentOS Stream for more info, it described my initial CentOS Stream install on my first F5-422.
Upgrade RAM
Like the first F5-422, on the three new ones I proceeded as follows;
- remove 6 screws on the back
- carefully remove back cover (mind the 2 cables connecting the the fans to the motherboard)
- add a Crucial CT102464BF160B 8GB SO-DIMM.
- close case in reverse order
Install SSD for OS and Ceph use
- put a Samsung SSD 860 EVO 500GB, SATA in drive bay 1
- only use e.g. 65GiB for base OS, keep some unpartitioned space for Ceph use.
Firmware Settings
As so often, I was not happy with the vendor’s UEFI settings and made the following changes.
- connect screen, keyboard, mouse (and USB boot medium)
- enter firmware setup
- Advanced / Power Management Features: enable Wake on LAN
- Advanced / Power Management Features: disable Wake Up on Alarm
- Chipset / South Bridge / OS Selection: change from Windows to Intel Linux
- Boot / Setup Prompt Timeout: change to 5 seconds or more
- Boot / Boot Mode: change from Legcy+UEFI to UEFI Only
- save and exit
Install CentOS Stream
nodes 2, 3 and 4 needing initial install and node 1 currently running F29, I processed all 4 as follows;
- create USB boot medium. I used
isoimagewriterto writeCentOS-Stream-x86_64-dvd1.isoon a USB stick - boot from USB
- do a kistart install by adding
inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/CentOSstream8-x86_64-TerraMaster-F5-422-ks.cfgin the USB medium’s bootloader phase - while mediacheck ran, I adjusted
/etc/dhcp/dhcpd.confon the dhcp server, my DNS was set up previously.
Used Kickstart File
#version=RHEL8 # avoid using half arsed names like sda, sdb, etc # mapping seems to be: # bay 1 /dev/disk/by-path/pci-0000:00:12.0-ata-1 # bay 2 /dev/disk/by-path/pci-0000:00:12.0-ata-2 # bay 3 /dev/disk/by-path/pci-0000:03:00.0-ata-1 # bay 4 /dev/disk/by-path/pci-0000:03:00.0-ata-2 # bay 5 /dev/disk/by-path/pci-0000:04:00.0-ata-1 # we only use the SATA SSD in bay 1 ignoredisk --only-use=/dev/disk/by-path/pci-0000:00:12.0-ata-1 # Partition clearing information # note that OS goes on a small portion os the device in bay 1, the rest will be allocated to Ceph in a separtate VG. # so kickstarting with the below clearpart line will nuke the Ceph bits on SSD !!! clearpart --all --initlabel --drives=/dev/disk/by-path/pci-0000:00:12.0-ata-1 # Use graphical install graphical # Use CDROM installation media cdrom # Keyboard layouts keyboard --vckeymap=us --xlayouts='us' # System language lang en_US.UTF-8 # Network information # 1 Gig on USB network --bootproto=dhcp --device=enp0s21f0u3 --onboot=off --ipv6=auto --no-activate # 10 Gig on PCIe network --bootproto=dhcp --device=enp1s0 --onboot=off --ipv6=auto # 1 Gig on PCIe network --bootproto=dhcp --device=enp2s0 --ipv6=auto --activate network --hostname=localhost.localdomain repo --name="AppStream" --baseurl=file:///run/install/repo/AppStream # Root password rootpw --iscrypted $6$X55cCQsXt3xeCukD$qJhkHV0x8cHjBc8zBbRldKq7YdMIf52lfM2GvaNW13HEPBIllwsXfIzmD1s4ZxstUTO5u4JzxAuLjz27SSmy6/ # Run the Setup Agent on first boot? firstboot --disable # Do not configure the X Window System skipx # System services services --enabled="chronyd" # System timezone timezone Europe/Berlin --isUtc # Ansible user user --groups=wheel --name=ansible --password=$6$1XAtqKKCAXHi17l/$Yosu3OEblUaM2SvrbeBH6wlzV8TAv7eO.3jI9CpTG3gbRfGnXlbuLPmgw.IBacL3km/ArjEjKbd9Ki2LTxYpu. --iscrypted --gecos="ansible" # Disk partitioning information # we use a 65G PV (SATA SSD in bay 1 is 465.8G, the rest will be allocated to Ceph in VG_SSD_bay1_OS via Ansible) # but keep free PE part /boot --fstype="ext4" --ondisk=/dev/disk/by-path/pci-0000:00:12.0-ata-1 --size=1024 part /boot/efi --fstype="efi" --ondisk=/dev/disk/by-path/pci-0000:00:12.0-ata-1 --size=512 --fsoptions="umask=0077,shortname=winnt" part pv.65 --fstype="lvmpv" --ondisk=/dev/disk/by-path/pci-0000:00:12.0-ata-1 --size=66560 volgroup VG_SSD_bay1_OS --pesize=4096 pv.65 logvol / --fstype="xfs" --size=4096 --name=LV_root --vgname=VG_SSD_bay1_OS logvol swap --fstype="swap" --size=4096 --name=LV_swap --vgname=VG_SSD_bay1_OS logvol /var --fstype="xfs" --size=4096 --name=LV_var --vgname=VG_SSD_bay1_OS logvol /var/log --fstype="xfs" --size=2048 --name=LV_var_log --vgname=VG_SSD_bay1_OS logvol /home --fstype="xfs" --size=2048 --name=LV_home --vgname=VG_SSD_bay1_OS %packages @^minimal-environment @container-management chrony %end %addon com_redhat_kdump --disable --reserve-mb='auto' %end %anaconda pwpolicy root --minlen=6 --minquality=1 --notstrict --nochanges --notempty pwpolicy user --minlen=6 --minquality=1 --notstrict --nochanges --emptyok pwpolicy luks --minlen=6 --minquality=1 --notstrict --nochanges --notempty %end %post --log=/root/ks-post.log # dump pcfe's ssh key to the root user mkdir /root/.ssh chown root.root /root/.ssh chmod 700 /root/.ssh cat <>/root/.ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAvNDSbbViufkQdqHfI4lrF3utwd028ndTJspdiOZ2JtdIVBjUokQRoVFY8+DXjTpBIBKWd/WciqMc02gYUXG94pDZkxHe9Z0xD/SdpoGng2XodVUVEnNWImVSbpPFDwqFZWOqyC8QVEp7AMhj+4AdWp9JmTQUeYWLssrmvnY9m0dB3K2CL6G532y7cZ9cEl73kBIxPVOHkdRZGUyTC05fZL7Ldd2eepi/oWRpDXmfWn/rN6zl1vKaYq5TaOcnATCL1tmP/t8yOdodMCgqYHRbhh8zuFcsMxl7b+eenjhlsh87V/pdKrWZFcfeWxamj7CdEQA79r3Sw/7h6Y2OGvKYKzofGtnjPzJnu63Hzdu7oQcQTXQpuMgoSMkhS+MbJOfiJUONK1tfTKiN29NJZ90biSonu7XpOpemIRAlx/vhpVXkKcN2PY12fRy7wL0A9yghb6M1Hkw1bHK7tlw/cpQiHhEPJuTbBWTZJ3OWSLXx+EMRfdn8cHx1yckaqXzMLoGh52OkgVbNeN52bbrwDrelOc237zknPnSzbnB7wIwZwmRE0GDvl/Ta+AM5A8N7FMC5K9wbOgP9qObTbUQGwP0hwg/Xai2kR/7QUwSB3/y2ja2wZNCSP5aSGszLkJd3X5M0yALcQFVzNyqUKy5wQhQEpUKnteAvwbwpmUmuU6WQPNk= private key 2008-05-22 EOF chown root.root /root/.ssh/authorized_keys chmod 600 /root/.ssh/authorized_keys restorecon /root/.ssh/authorized_keys cat < >/etc/udev/rules.d/75-disable-120MB-on-board-stick.rules # The on-board 120MB stick should be disabled # it's tiny and leaving it untouched allows a reset to the shipped state # by choosing the USB stick as boot target during POST and booting UEFI+Legacy mode # c.f. https://projectgus.com/2014/09/blacklisting-a-single-usb-device-from-linux/ SUBSYSTEM=="usb", ATTRS{idVendor}=="13fe", ATTRS{idProduct}=="4100", ATTR{authorized}="0" EOF chown root.root /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules chmod 644 /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules restorecon /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules # Since Ceph and EPEL should not be mixed, # pull check-mk-agent from my monitoring server (checkmk Raw edition) dnf -y install http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/check-mk-agent-2.0.0p17-1.noarch.rpm echo "check-mk-agent installed from monitoring server" >> /etc/motd echo "kickstarted at `date` for CentOS Stream on TerraMaster F5-422" >> /etc/motd %end
Since the machine is supposed to be mainly a Ceph OSD, I only used some 65G of the SSD for the OS.
Everything on top of this minimal kickstart install is handled via Ansible. You could also do a manual minimal install, I simply could not be asked to click around interactively for 4 installs.
ToDo
the following still need to be added to the playbook.
Test WOL
Still need to test if Wake on LAN works.
Clen up NetworkManmager connections
TODO: delete left behind ifcfg-enp… after terramaster-network.yml ran.
Set Up smartd
TODO: still need to set up smartd to send alerts to my sms gateway.
Disable the 120 MB USB stick
note: the NAS has a 120MB stick in an on-board connector, I decided to not touch it for now. In case I ever need to go back to the vendor OS. Note that enabling EUFI only boot mode makes the 120MB stick not show up as a valid boot target with the shipped image. Seems someone forgot to get their boot medium signed.
I disabled it with a blacklist entry.
[root@f5-422-01 ~]# cat /etc/udev/rules.d/75-disable-120MB-on-board-stick.rules
# The on-board 120MB stick should be disabled
# it's tiny and I have 0 use for it
# c.f. https://projectgus.com/2014/09/blacklisting-a-single-usb-device-from-linux/
SUBSYSTEM=="usb", ATTRS{idVendor}=="13fe", ATTRS{idProduct}=="4100", ATTR{authorized}="0"
TODO: put the above cleanly in a playbook, at the moment it’s handled in kickstart.
After the next boot, as expected the tiny flash drive is no longer visible
[root@f5-422-01 ~]# lsusb
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13fe:4100 Kingston Technology Company Inc. Flash drive
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
[root@f5-422-01 ~]# lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/7p, 5000M
|__ Port 3: Dev 3, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M
Drive Bay Mapping
To avoid using half arsed names like sda, sdb, etc
I address the drive bays by their /dev/disk/by-path/ entries.
FIXME: double check. The mapping seems to be
| bay number | /dev/disk/by-path/ entry |
|---|---|
| bay 1 | /dev/disk/by-path/pci-0000:00:12.0-ata-1 |
| bay 2 | /dev/disk/by-path/pci-0000:00:12.0-ata-2 |
| bay 3 | /dev/disk/by-path/pci-0000:03:00.0-ata-1 |
| bay 4 | /dev/disk/by-path/pci-0000:03:00.0-ata-2 |
| bay 5 | /dev/disk/by-path/pci-0000:04:00.0-ata-1 |
Initial Setup
Temporary repo foo on 2020-02-29
On 2020-02-29, I got on dnf upgrade
[...]
No available modular metadata for modular package 'oci-systemd-hook-1:0.1.15-2.git2d0b8a3.module_el8.1.0+237+63e26edc.x86_64', it cannot be installed on the system
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: No available modular metadata for modular package
So I quickly ripped that out to avoid going down that rabbit hole.
As root, on a F5-422:
dnf remove oci-systemd-hook
Playbooks will take care of installing needed packages and RPM takes care of dependencies.
Resolved on 2020-03-01
The above no longer occurred on Sunday 2020-03-01.
check-mk-agent
Since Ceph and EPEL are not meant to be mixed, I install check-mk-agent from my monitoring server.
FIXME: that’s in kickstart, not in Ansibe
FIXME: adjust role pcfe.checkmk, paths between package in EPEL and the one on Check MK Raw are different. For now I just hammered it into the F5-422 kickstart as all other boxes use the RPM from EPEL or Fedora.
terramaster-initial-setup-centos8.yml
takes care of
- creating an ansible user
- basic-security-setup
- local settings (mirros etc)
As user, on my F31 x96_64 workstation.
ansible-playbook terramaster-initial-setup-centos8.yml
---
# sets up a CentoOS Stream minimal install to be ready for ceph-ansible
#
# this is for my home setup, not for production!
- hosts:
- TerraMaster_boxes
become: false
roles:
- pcfe.user_owner
- pcfe.basic_security_setup
- pcfe.housenet
#- pcfe.checkmk
# note though that check-mk-agent is in EPEL but RHCS 3 docs say EPEL needs to be off
# for now I manually installed the rpm with
# dnf install http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/check-mk-agent-1.6.0p5-1.noarch.rpm
vars:
ansible_user: root
user_owner: ansible
common_timezone: Europe/Berlin
tasks:
# !!!
#
# if I ever enable EPEL, then I MUST exclude
# - ansible
# - ceph
# in the EPEL repo files to ensure no newer versions of those packages are pulled in from EPEL
#
# exclude = *ceph* nfs-ganesha-rgw rbd-mirror *ansible*
#
# !!!
#
# https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/installation_guide/index#enabling-the-red-hat-ceph-storage-repositories-install
- name: "REPOS | ensure EPEL is disabled"
yum_repository:
name: epel
state: absent
# start by enabling time sync, RHSM operations will fail on too large time delta
# note that this uses chronyd, not ntpd.
- name: "CHRONYD | ensure chrony is installed"
package:
name: chrony
state: present
- name: "CHRONYD | ensure chrony-wait is enabled"
service:
name: chrony-wait
enabled: true
- name: "CHRONYD | ensure chronyd is enabled and running"
service:
name: chronyd
enabled: true
state: started
# enable persistent journal
- name: "JOURNAL | ensure persistent logging for the systemd journal is possible"
file:
path: /var/log/journal
state: directory
owner: root
group: systemd-journal
mode: u=rwx,g=rs,o=rx
# 2.10. Enabling Password-less SSH for Ansible
- name: "SUDO | enable passwordless sudo for ansible user"
copy:
dest: /etc/sudoers.d/ansible
content: |
ansible ALL=NOPASSWD: ALL
owner: root
group: root
mode: 0440
# Install prerequisites
- name: "package | ensure prerequisites needed in addition to minimal install are present"
package:
name:
- vim
state: present
# I admit, the regexp is a search engine hit
- name: "GRUB | ensure console blanking is disabled in defaults file"
lineinfile:
state: present
dest: /etc/default/grub
backrefs: yes
regexp: '^(GRUB_CMDLINE_LINUX=(?!.* consoleblank)\"[^\"]+)(\".*)'
line: '\1 consoleblank=0\2'
# upgrade the box
- name: "package | ensure all updates are applied"
package:
update_cache: yes
name: '*'
state: latest
tags: apply_errata
terramaster-general-setup.yml
Besides running some of my roles, it takes care of
- enabled wake on LAN
- setup watchdog
- set up rngd
As user, on my F31 x96_64 workstation.
ansible-playbook terramaster-general-setup.yml
---
- name: General setup of my Terramaster F5-422 Ceph nodes
hosts:
- TerraMaster_boxes
become: true
roles:
- fedora.linux_system_roles.network
- pcfe.user_owner
- pcfe.basic_security_setup
- pcfe.housenet
- pcfe.comfort
# - pcfe.checkmk
handlers:
- name: Handle running grub2-mkconfig for dist RedHat major vers 9
ansible.builtin.command: grub2-mkconfig -o /boot/grub2/grub.cfg
changed_when: true
- name: Handle running grub2-mkconfig for dist RedHat major vers 8
ansible.builtin.command: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
changed_when: true
tasks:
# work around identical UUID on each host as per https://theforeman.org/plugins/katello/nightly/troubleshooting/content_hosts.html
# The TerraMaster F5-422 boxes all have the same system-uuid :-(
# RHSM does not like that, so override to avoid both
# "HTTP error (409 - Conflict): Request failed due to concurrent modification, please re-try.\n"
# and all 4 boxes overriding each other in insights inventory
- name: "RHSM | ensure uuid override is derived from fqdn, F5-422 all have idential uuid in DMI"
ansible.builtin.copy:
dest: /etc/rhsm/facts/uuid_override.facts
owner: root
group: root
mode: u=rw,g=r,o=r
content: |
{"dmi.system.uuid": "{{ ansible_fqdn | to_uuid }}"}
# Ensure the packages that the RHEL8 only preflight play from cephadm-ansible would install
# also ensure ceph-selinux is installed, docs say that's in the default set
- name: Ensure needed packages for RHCS5 on RHEL9 are installed
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "9"
ansible.builtin.package:
name:
- chrony
- cephadm
- podman
- ceph-common
- ceph-selinux
state: present
update_cache: false
# Install some tools
- name: "PACKAGE | tool installation, el8 and el9"
ansible.builtin.package:
name:
- pciutils
- usbutils
- fio
- powertop
- tuned
- tuned-utils
- numactl
- iperf3
- tcpdump
- hwloc-gui
- fwupd
state: present
update_cache: false
- name: "PACKAGE | tool installation el9"
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "9"
ansible.builtin.package:
name:
- s-nail
state: present
update_cache: false
- name: "PACKAGE | tool installation el8"
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
ansible.builtin.package:
name:
- teamd
- NetworkManager-team
- mailx
state: present
update_cache: false
# linux-system-roles.network sets static network config (from host_vars)
# but I want the static hostname nailed down too
#
# FIXME; ansible_fqdn is a fact, so that does not help when it's a freshly kickstarted localhost.localdomain
- name: Ensure static hostname is set to FQDN
ansible.builtin.hostname:
name: "{{ ansible_fqdn }}"
use: systemd
# enable WOL manually until https://github.com/linux-system-roles/network/issues/150 is fixed
- name: Ensure Wake On LAN is enable for the upper network card
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
ansible.builtin.lineinfile:
path: /etc/sysconfig/network-scripts/ifcfg-1G_PCIe
create: false
regexp: '^ETHTOOL_OPTS= '
insertafter: '^TYPE=Ethernet'
line: 'ETHTOOL_OPTS="wol g"'
# enable watchdog
# it's a Jun 22 13:12:09 localhost kernel: sbsa-gwdt e0bb0000.gwdt: Initialized with 10s timeout @ 250000000 Hz, action=0.
- name: "WATCHDOG | ensure kernel module wdat_wdt has correct options configured"
ansible.builtin.lineinfile:
path: /etc/modprobe.d/wdat_wdt.conf
create: true
regexp: '^options '
insertafter: '^#options'
line: 'options wdat_wdt nowayout=0'
mode: u=rw,g=r,o=r
# configure both watchdog.service and systemd watchdog, but only use the latter
- name: "PACKAGE | ensure watchdog package is installed"
ansible.builtin.package:
name: watchdog
state: present
update_cache: false
- name: "WATCHDOG | ensure correct watchdog-device is used by watchdog.service"
ansible.builtin.lineinfile:
path: /etc/watchdog.conf
regexp: '^watchdog-device'
insertafter: '^#watchdog-device'
line: 'watchdog-device = /dev/watchdog'
- name: "WATCHDOG | ensure timeout is set to 30 seconds for watchdog.service"
ansible.builtin.lineinfile:
path: /etc/watchdog.conf
regexp: '^watchdog-timeout'
insertafter: '^#watchdog-timeout'
line: 'watchdog-timeout = 30'
# Using systemd watchdog rather than watchdog.service
- name: "WATCHDOG | ensure watchdog.service is disabled"
ansible.builtin.systemd:
name: watchdog.service
state: stopped
enabled: false
# configure systemd watchdog
# c.f. http://0pointer.de/blog/projects/watchdog.html
- name: "SYSTEMD | ensure systemd watchdog is enabled"
ansible.builtin.lineinfile:
path: /etc/systemd/system.conf
regexp: '^RuntimeWatchdogSec'
insertafter: 'EOF'
line: 'RuntimeWatchdogSec=30'
- name: "SYSTEMD | ensure systemd shutdown watchdog is enabled"
ansible.builtin.lineinfile:
path: /etc/systemd/system.conf
regexp: '^ShutdownWatchdogSec'
insertafter: 'EOF'
line: 'ShutdownWatchdogSec=30'
# install and enable rngd
- name: "PACKAGE | ensure rng-tools package is installed"
ansible.builtin.package:
name: rng-tools
state: present
update_cache: false
- name: "RNGD | ensure rngd.service is enabled and started"
ansible.builtin.systemd:
name: rngd.service
state: started
enabled: true
# ensure tuned is set up as I wish
- name: "TUNED | ensure tuned.service is enabled and running"
ansible.builtin.systemd:
name: tuned.service
state: started
enabled: true
- name: "TUNED | check which tuned profile is active"
ansible.builtin.command: tuned-adm active
register: tuned_active_profile
ignore_errors: true
changed_when: false
- name: "TUNED | activate tuned profile {{ tuned_profile }}"
ansible.builtin.command: "tuned-adm profile {{ tuned_profile }}"
when: not tuned_active_profile.stdout is search('Current active profile:' ~ ' ' ~ tuned_profile)
changed_when: true
# install cockpit, but disabled for now
- name: "COCKPIT | ensure packages for https://cockpit-project.org/ are installed"
ansible.builtin.package:
name:
- cockpit
- cockpit-selinux
- cockpit-kdump
- cockpit-storaged
- cockpit-system
state: present
update_cache: false
- name: "COCKPIT | ensure cockpit.socket is stopped and disabled"
ansible.builtin.systemd:
name: cockpit.socket
state: stopped
enabled: false
- name: "COCKPIT | ensure firewalld forbids service cockpit in zone public"
ansible.posix.firewalld:
service: cockpit
zone: public
permanent: true
state: disabled
immediate: true
# disable libvirtd
- name: "Ensure libvirtd.service is disabled and stopped"
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
ansible.builtin.systemd:
name: libvirtd.service
state: stopped
enabled: false
# enable kdump.service since kickstart now creates a sufficiently large /var/crash
# alternatively, you could set up netdump
- name: "Ensure kdump.service is enabled and started"
ansible.builtin.systemd:
name: kdump.service
state: started
enabled: true
# podman
- name: "PACKAGE | ensure podman is installed"
ansible.builtin.package:
name:
- podman
- podman-docker
state: present
# setroubleshoot, see also https://danwalsh.livejournal.com/20931.html
- name: "PACKAGE | ensure setroubleshoot for headless server is installed"
ansible.builtin.package:
name:
- setroubleshoot-server
- setroubleshoot-plugins
state: present
- name: "MONITORING | ensure packages for monitoring are installed"
ansible.builtin.package:
name:
- smartmontools
- hdparm
- check-mk-agent
- lm_sensors
state: present
- name: "MONITORING | ensure firewalld permits 6556 in zone public for check-mk-agent"
ansible.posix.firewalld:
port: 6556/tcp
permanent: true
state: enabled
immediate: true
zone: public
- name: "MONITORING | ensure tarsnap cache is in fileinfo"
ansible.builtin.lineinfile:
path: /etc/check_mk/fileinfo.cfg
line: "/usr/local/tarsnap-cache/cache"
create: true
mode: u=rw,g=r,o=r
- name: "MONITORING | ensure entropy_avail plugin for Check_MK is present"
ansible.builtin.template:
src: templates/check-mk-agent-plugin-entropy_avail.j2
dest: /usr/lib/check_mk_agent/plugins/entropy_avail
mode: u=rwx,g=rx,o=rx
group: root
owner: root
- name: "MONITORING | plugins from running CEE instance"
ansible.builtin.get_url:
url: "http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/plugins/{{ item }}"
dest: "/usr/lib/check_mk_agent/plugins/{{ item }}"
mode: u=rwx,g=rx,o=rx
loop:
- smart
- lvm
- name: "MONITORING | ensure check_mk.socket is started and enabled"
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
ansible.builtin.systemd:
name: check_mk.socket
state: started
enabled: true
- name: "MONITORING | ensure check-mk-agent.socket is started and enabled"
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "9"
ansible.builtin.systemd:
name: check-mk-agent.socket
state: started
enabled: true
# I admit, the regexp is a search engine hit
# maybe using grubby(8) would be more readable
# yamllint disable rule:line-length
# - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/configuring-kernel-command-line-parameters_managing-monitoring-and-updating-the-kernel#what-is-grubby_configuring-kernel-command-line-parameters
# - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sec-Making_Persistent_Changes_to_a_GRUB_2_Menu_Using_the_grubby_Tool
# yamllint enable rule:line-length
- name: "GRUB | ensure console blanking is disabled in defaults file plus handler"
ansible.builtin.lineinfile:
state: present
dest: /etc/default/grub
backrefs: true
regexp: '^(GRUB_CMDLINE_LINUX=(?!.* consoleblank)\"[^\"]+)(\".*)'
line: '\1 consoleblank=0\2'
notify: "Handle running grub2-mkconfig for dist {{ ansible_distribution }} major vers {{ ansible_distribution_major_version }}"
# Since I do not manage to get these F5-422 to PXE boot, add an entry into grub
# so that I can kickstart the box after this without fiddling with a USB stick
- name: "GRUB | ensure initrd for RHEL 8.9 kickstart is present"
ansible.builtin.get_url:
url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.9/Server/x86_64/os/images/pxeboot/initrd.img"
dest: "/boot/initrd-kickstart-rhel89.img"
mode: u=rw
- name: "GRUB | ensure kernel for RHEL 8.9 kickstart is present"
ansible.builtin.get_url:
url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.9/Server/x86_64/os/images/pxeboot/vmlinuz"
dest: "/boot/vmlinuz-kickstart-rhel89"
mode: u=rwx,g=rx,o=rx
# yamllint disable rule:line-length
- name: "GRUB | ensure kickstarting RHEL 8.9 entry is present"
ansible.builtin.copy:
dest: "/etc/grub.d/11_RHEL8x_kickstart"
owner: "root"
group: "root"
mode: u=rwx,g=rx,o=rx
content: |
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
menuentry "WARNING Kickstart this box with RHEL 8.9 as a F5-422 ceph node WARNING" {
linuxefi /vmlinuz-kickstart-rhel89 ip=dhcp inst.repo=ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.9/Server/x86_64/os inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/RHEL89-TerraMaster-F5-422-ks.cfg
initrdefi /initrd-kickstart-rhel89.img
}
notify: "Handle running grub2-mkconfig for dist {{ ansible_distribution }} major vers {{ ansible_distribution_major_version }}"
# yamllint enable rule:line-length
- name: "GRUB | ensure initrd for RHEL 9.3 kickstart is present"
ansible.builtin.get_url:
url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-9.3/Server/x86_64/os/images/pxeboot/initrd.img"
dest: "/boot/initrd-kickstart-rhel93.img"
mode: u=rw
- name: "GRUB | ensure kernel for RHEL 9.3 kickstart is present"
ansible.builtin.get_url:
url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-9.3/Server/x86_64/os/images/pxeboot/vmlinuz"
dest: "/boot/vmlinuz-kickstart-rhel93"
mode: u=rwx,g=rx,o=rx
# yamllint disable rule:line-length
- name: "GRUB | ensure kickstarting RHEL 9.3 entry is present"
ansible.builtin.copy:
dest: "/etc/grub.d/12_RHEL9x_kickstart"
owner: "root"
group: "root"
mode: u=rwx,g=rx,o=rx
content: |
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
menuentry "WARNING Kickstart this box with RHEL 9.3 as a F5-422 ceph node WARNING" {
linuxefi /vmlinuz-kickstart-rhel93 ip=dhcp inst.repo=ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-9.3/Server/x86_64/os inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/RHEL93-TerraMaster-F5-422-ks.cfg
initrdefi /initrd-kickstart-rhel93.img
}
notify: "Handle running grub2-mkconfig for dist {{ ansible_distribution }} major vers {{ ansible_distribution_major_version }}"
# yamllint enable rule:line-length
# upgrade the box
- name: Package | ensure all updates are applied
ansible.builtin.package:
update_cache: true
name: '*'
state: latest # noqa package-latest
tags: apply_errata
test forcefully crashing the box.
Verify that the watchdog works as expected.
As root, on a F5-422:
echo '1' > /proc/sys/kernel/sysrq
echo 'c' > /proc/sysrq-trigger
As expected, I get an ops outut on the console acter the c and,
as previously tested, the box reboots itself after 30 seconds.
Hardware Info
dmidecode -t baseboard
[root@f5-422-01 ~]# dmidecode -t baseboard
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.
Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: GIFA Industrial Control
Product Name: TM-J3455-4G3L
Version: Default string
Serial Number: Default string
Asset Tag: Default string
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: Default string
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0
Handle 0x0025, DMI type 10, 6 bytes
On Board Device Information
Type: Video
Status: Enabled
Description: To Be Filled By O.E.M.
Handle 0x002D, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Onboard IGD
Type: Video
Status: Enabled
Type Instance: 1
Bus Address: 0000:00:02.0
Handle 0x002E, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Onboard LAN
Type: Ethernet
Status: Enabled
Type Instance: 1
Bus Address: 0000:00:19.0
Handle 0x002F, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Onboard 1394
Type: Other
Status: Enabled
Type Instance: 1
Bus Address: 0000:03:1c.2
lspci
[root@f5-422-01 ~]# lspci
00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
00:02.0 VGA compatible controller: Intel Corporation Device 5a85 (rev 0b)
00:0e.0 Audio device: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster (rev 0b)
00:0f.0 Communication controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine (rev 0b)
00:12.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b)
00:13.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 (rev fb)
00:13.2 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3 (rev fb)
00:13.3 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #4 (rev fb)
00:14.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port B #1 (rev fb)
00:14.1 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port B #2 (rev fb)
00:15.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b)
00:1a.0 Serial bus controller [0c80]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PWM Pin Controller (rev 0b)
00:1f.0 ISA bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface (rev 0b)
00:1f.1 SMBus: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller (rev 0b)
01:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
lsusb
[root@f5-422-01 ~]# lsusb
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13fe:4100 Kingston Technology Company Inc. Flash drive
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
lsusb -t
[root@f5-422-01 ~]# lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/7p, 5000M
|__ Port 3: Dev 3, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M
lscpu
[root@f5-422-01 ~]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 92
Model name: Intel(R) Celeron(R) CPU J3455 @ 1.50GHz
Stepping: 9
CPU MHz: 1582.479
CPU max MHz: 2300.0000
CPU min MHz: 800.0000
BogoMIPS: 2995.20
Virtualization: VT-x
L1d cache: 24K
L1i cache: 32K
L2 cache: 1024K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault cat_l2 ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust smep erms mpx rdt_a rdseed smap clflushopt intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts md_clear arch_capabilities
free -m and free -h
[root@f5-422-01 ~]# free -m
total used free shared buff/cache available
Mem: 11854 268 11345 1 239 11330
Swap: 4095 0 4095
[root@f5-422-01 ~]# free -h
total used free shared buff/cache available
Mem: 11Gi 268Mi 11Gi 1.0Mi 239Mi 11Gi
Swap: 4.0Gi 0B 4.0Gi
dmidecode -t memory
[root@f5-422-01 ~]# dmidecode -t memory
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.
Handle 0x0030, DMI type 16, 23 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: None
Maximum Capacity: 32 GB
Error Information Handle: Not Provided
Number Of Devices: 4
Handle 0x0031, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x0030
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 4096 MB
Form Factor: SODIMM
Set: None
Locator: ChannelA-DIMM0
Bank Locator: BANK 0
Type: DDR3
Type Detail: Synchronous
Speed: 1600 MT/s
Manufacturer: Undefi
Serial Number: 88000000
Asset Tag: 9876543210
Part Number: /
Rank: Unknown
Configured Memory Speed: 1600 MT/s
Minimum Voltage: 44.975 V
Maximum Voltage: 44.975 V
Configured Voltage: 1.5 V
Handle 0x0032, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x0030
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 8192 MB
Form Factor: SODIMM
Set: None
Locator: ChannelB-DIMM0
Bank Locator: BANK 1
Type: DDR3
Type Detail: Synchronous
Speed: 1600 MT/s
Manufacturer: Unde
Serial Number: [REDACTED]
Asset Tag: 9876543210
Part Number: CT102464BF160B.M16
Rank: Unknown
Configured Memory Speed: 1600 MT/s
Minimum Voltage: 1.35 V
Maximum Voltage: 1.5 V
Configured Voltage: 1.5 V
Handle 0x0033, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x0030
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: DIMM
Set: None
Locator: ChannelC-DIMM0
Bank Locator: BANK 2
Type: Unknown
Type Detail: None
Speed: Unknown
Manufacturer: Not Specified
Serial Number: Not Specified
Asset Tag: 9876543210
Part Number: Not Specified
Rank: Unknown
Configured Memory Speed: 1600 MT/s
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Handle 0x0034, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x0030
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: DIMM
Set: None
Locator: ChannelD-DIMM0
Bank Locator: BANK 3
Type: Unknown
Type Detail: None
Speed: Unknown
Manufacturer: Not Specified
Serial Number: Not Specified
Asset Tag: 9876543210
Part Number: Not Specified
Rank: Unknown
Configured Memory Speed: 1600 MT/s
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
The vendor’s filling of the DMI registers does leave to be desired…
Minimum Voltage: 44.975 V
Maximum Voltage: 44.975 V
Configured Voltage: 1.5 V
yeah, right. But well, Configured Voltage is right.
Network Devices
upper NIC, 1Gig, PCIe
[root@f5-422-01 ~]# ethtool -i enp2s0
driver: r8169
version:
firmware-version: rtl8168h-2_0.0.2 02/26/15
expansion-rom-version:
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@f5-422-01 ~]# lspci -s 02:00.0
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
middle NIC, 1Gig, USB
[root@f5-422-01 ~]# ethtool -i enp0s21f0u3
driver: r8152
version: v1.09.9
firmware-version:
expansion-rom-version:
bus-info: usb-0000:00:15.0-3
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
[root@f5-422-01 ~]# lsusb | grep -i rtl
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
lower NIC, 10Gig, PCIe
[root@f5-422-01 ~]# lsusb | grep -i rtl
Bus 002 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
[root@f5-422-01 ~]# ethtool -i enp1s0
driver: atlantic
version: 2.0.4.0-kern
firmware-version: 3.1.57
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@f5-422-01 ~]# lspci -s 01:00.0
01:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
iperf3 between 2 10G NICs
In c’t 02/2020 S. 102 they were measured at roughly 7 GBit/s (Article here but paywalled. As a subscriber I just read it on paper.)
7,0 gigabit are 6,07 gibibit, so I seem to be pretty close with 5,8 on jumbo frames.
MTU 1500
With the default MTU, I get 5.4 Gbits/sec out of the interfaces.
iperf3 -c 192.168.30.181 -i 1 -t 20 -P 4 | tee $(date +%F)-iperf3-4parallel.log
Connecting to host 192.168.30.181, port 5201
[ 5] local 192.168.30.182 port 48790 connected to 192.168.30.181 port 5201
[ 7] local 192.168.30.182 port 48792 connected to 192.168.30.181 port 5201
[ 9] local 192.168.30.182 port 48794 connected to 192.168.30.181 port 5201
[ 11] local 192.168.30.182 port 48796 connected to 192.168.30.181 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 111 MBytes 927 Mbits/sec 0 479 KBytes
[ 7] 0.00-1.00 sec 326 MBytes 2.74 Gbits/sec 0 516 KBytes
[ 9] 0.00-1.00 sec 110 MBytes 926 Mbits/sec 0 485 KBytes
[ 11] 0.00-1.00 sec 111 MBytes 933 Mbits/sec 0 488 KBytes
[SUM] 0.00-1.00 sec 658 MBytes 5.52 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 1.00-2.00 sec 326 MBytes 2.73 Gbits/sec 0 516 KBytes
[ 9] 1.00-2.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 1.00-2.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 1.00-2.00 sec 651 MBytes 5.46 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 2.00-3.00 sec 324 MBytes 2.72 Gbits/sec 0 516 KBytes
[ 9] 2.00-3.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 2.00-3.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 2.00-3.00 sec 649 MBytes 5.44 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 3.00-4.00 sec 326 MBytes 2.73 Gbits/sec 0 516 KBytes
[ 9] 3.00-4.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 3.00-4.00 sec 107 MBytes 901 Mbits/sec 0 488 KBytes
[SUM] 3.00-4.00 sec 650 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 4.00-5.00 sec 325 MBytes 2.72 Gbits/sec 0 516 KBytes
[ 9] 4.00-5.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 4.00-5.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 4.00-5.00 sec 650 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 5.00-6.00 sec 325 MBytes 2.72 Gbits/sec 0 516 KBytes
[ 9] 5.00-6.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 5.00-6.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 5.00-6.00 sec 650 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 6.00-7.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 6.00-7.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 6.00-7.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 6.00-7.00 sec 651 MBytes 5.46 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 7.00-8.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 7.00-8.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 7.00-8.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 7.00-8.00 sec 651 MBytes 5.46 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 8.00-9.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 8.00-9.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 8.00-9.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 8.00-9.00 sec 650 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 9.00-10.00 sec 324 MBytes 2.72 Gbits/sec 0 542 KBytes
[ 9] 9.00-10.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 9.00-10.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 9.00-10.00 sec 650 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 10.00-11.00 sec 107 MBytes 901 Mbits/sec 0 479 KBytes
[ 7] 10.00-11.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 10.00-11.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 10.00-11.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 10.00-11.00 sec 649 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 11.00-12.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 11.00-12.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 11.00-12.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 11.00-12.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 11.00-12.00 sec 650 MBytes 5.46 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 12.00-13.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 12.00-13.00 sec 324 MBytes 2.72 Gbits/sec 0 542 KBytes
[ 9] 12.00-13.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 12.00-13.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 12.00-13.00 sec 649 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 13.00-14.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 13.00-14.00 sec 326 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 13.00-14.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 13.00-14.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 13.00-14.00 sec 651 MBytes 5.46 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 14.00-15.00 sec 108 MBytes 909 Mbits/sec 0 479 KBytes
[ 7] 14.00-15.00 sec 325 MBytes 2.73 Gbits/sec 0 542 KBytes
[ 9] 14.00-15.00 sec 108 MBytes 909 Mbits/sec 0 485 KBytes
[ 11] 14.00-15.00 sec 108 MBytes 909 Mbits/sec 0 488 KBytes
[SUM] 14.00-15.00 sec 650 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 15.00-16.00 sec 109 MBytes 918 Mbits/sec 0 1.23 MBytes
[ 7] 15.00-16.00 sec 325 MBytes 2.73 Gbits/sec 0 1.27 MBytes
[ 9] 15.00-16.00 sec 108 MBytes 910 Mbits/sec 0 1.18 MBytes
[ 11] 15.00-16.00 sec 109 MBytes 918 Mbits/sec 0 617 KBytes
[SUM] 15.00-16.00 sec 653 MBytes 5.47 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 16.00-17.00 sec 108 MBytes 902 Mbits/sec 0 1.23 MBytes
[ 7] 16.00-17.00 sec 325 MBytes 2.73 Gbits/sec 0 1.27 MBytes
[ 9] 16.00-17.00 sec 109 MBytes 912 Mbits/sec 0 1.18 MBytes
[ 11] 16.00-17.00 sec 108 MBytes 908 Mbits/sec 0 617 KBytes
[SUM] 16.00-17.00 sec 649 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 17.00-18.00 sec 109 MBytes 912 Mbits/sec 0 1.23 MBytes
[ 7] 17.00-18.00 sec 325 MBytes 2.73 Gbits/sec 0 1.27 MBytes
[ 9] 17.00-18.00 sec 108 MBytes 902 Mbits/sec 0 1.18 MBytes
[ 11] 17.00-18.00 sec 108 MBytes 910 Mbits/sec 0 617 KBytes
[SUM] 17.00-18.00 sec 650 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 18.00-19.00 sec 109 MBytes 912 Mbits/sec 0 1.23 MBytes
[ 7] 18.00-19.00 sec 325 MBytes 2.73 Gbits/sec 0 1.27 MBytes
[ 9] 18.00-19.00 sec 109 MBytes 912 Mbits/sec 0 1.18 MBytes
[ 11] 18.00-19.00 sec 108 MBytes 909 Mbits/sec 0 617 KBytes
[SUM] 18.00-19.00 sec 651 MBytes 5.46 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 19.00-20.00 sec 108 MBytes 902 Mbits/sec 0 1.23 MBytes
[ 7] 19.00-20.00 sec 325 MBytes 2.73 Gbits/sec 0 1.27 MBytes
[ 9] 19.00-20.00 sec 109 MBytes 912 Mbits/sec 0 1.18 MBytes
[ 11] 19.00-20.00 sec 108 MBytes 908 Mbits/sec 0 617 KBytes
[SUM] 19.00-20.00 sec 649 MBytes 5.45 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 2.12 GBytes 910 Mbits/sec 0 sender
[ 5] 0.00-20.02 sec 2.12 GBytes 908 Mbits/sec receiver
[ 7] 0.00-20.00 sec 6.35 GBytes 2.73 Gbits/sec 0 sender
[ 7] 0.00-20.02 sec 6.35 GBytes 2.72 Gbits/sec receiver
[ 9] 0.00-20.00 sec 2.12 GBytes 910 Mbits/sec 0 sender
[ 9] 0.00-20.02 sec 2.12 GBytes 908 Mbits/sec receiver
[ 11] 0.00-20.00 sec 2.12 GBytes 910 Mbits/sec 0 sender
[ 11] 0.00-20.02 sec 2.12 GBytes 908 Mbits/sec receiver
[SUM] 0.00-20.00 sec 12.7 GBytes 5.46 Gbits/sec 0 sender
[SUM] 0.00-20.02 sec 12.7 GBytes 5.45 Gbits/sec receiver
iperf Done.
MTU 9000
With jumbo frames, I get 5.8 Gbits/sec out of the interfaces.
iperf3 -c 192.168.30.181 -i 1 -t 20 -P 4 | tee $(date +%F)-iperf3-4parallel.log
Connecting to host 192.168.30.181, port 5201
[ 5] local 192.168.30.182 port 60392 connected to 192.168.30.181 port 5201
[ 7] local 192.168.30.182 port 60394 connected to 192.168.30.181 port 5201
[ 9] local 192.168.30.182 port 60396 connected to 192.168.30.181 port 5201
[ 11] local 192.168.30.182 port 60398 connected to 192.168.30.181 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 233 MBytes 1.95 Gbits/sec 0 638 KBytes
[ 7] 0.00-1.00 sec 112 MBytes 941 Mbits/sec 0 577 KBytes
[ 9] 0.00-1.00 sec 232 MBytes 1.95 Gbits/sec 0 612 KBytes
[ 11] 0.00-1.00 sec 123 MBytes 1.03 Gbits/sec 0 568 KBytes
[SUM] 0.00-1.00 sec 700 MBytes 5.87 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 1.00-2.00 sec 116 MBytes 972 Mbits/sec 0 577 KBytes
[ 9] 1.00-2.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 1.00-2.00 sec 115 MBytes 964 Mbits/sec 0 568 KBytes
[SUM] 1.00-2.00 sec 693 MBytes 5.81 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 231 MBytes 1.93 Gbits/sec 0 638 KBytes
[ 7] 2.00-3.00 sec 115 MBytes 967 Mbits/sec 0 577 KBytes
[ 9] 2.00-3.00 sec 231 MBytes 1.93 Gbits/sec 0 612 KBytes
[ 11] 2.00-3.00 sec 116 MBytes 973 Mbits/sec 0 568 KBytes
[SUM] 2.00-3.00 sec 693 MBytes 5.81 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 230 MBytes 1.93 Gbits/sec 0 638 KBytes
[ 7] 3.00-4.00 sec 116 MBytes 969 Mbits/sec 0 577 KBytes
[ 9] 3.00-4.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 3.00-4.00 sec 114 MBytes 960 Mbits/sec 0 568 KBytes
[SUM] 3.00-4.00 sec 691 MBytes 5.80 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 4.00-5.00 sec 115 MBytes 968 Mbits/sec 0 577 KBytes
[ 9] 4.00-5.00 sec 230 MBytes 1.93 Gbits/sec 0 612 KBytes
[ 11] 4.00-5.00 sec 115 MBytes 968 Mbits/sec 0 568 KBytes
[SUM] 4.00-5.00 sec 692 MBytes 5.80 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 5.00-6.00 sec 116 MBytes 972 Mbits/sec 0 577 KBytes
[ 9] 5.00-6.00 sec 230 MBytes 1.93 Gbits/sec 0 612 KBytes
[ 11] 5.00-6.00 sec 115 MBytes 966 Mbits/sec 0 568 KBytes
[SUM] 5.00-6.00 sec 692 MBytes 5.81 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 230 MBytes 1.93 Gbits/sec 0 638 KBytes
[ 7] 6.00-7.00 sec 115 MBytes 966 Mbits/sec 0 577 KBytes
[ 9] 6.00-7.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 6.00-7.00 sec 115 MBytes 968 Mbits/sec 0 568 KBytes
[SUM] 6.00-7.00 sec 692 MBytes 5.80 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 7.00-8.00 sec 115 MBytes 968 Mbits/sec 0 577 KBytes
[ 9] 7.00-8.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 7.00-8.00 sec 115 MBytes 965 Mbits/sec 0 568 KBytes
[SUM] 7.00-8.00 sec 692 MBytes 5.81 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 8.00-9.00 sec 115 MBytes 968 Mbits/sec 0 577 KBytes
[ 9] 8.00-9.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 8.00-9.00 sec 115 MBytes 963 Mbits/sec 0 568 KBytes
[SUM] 8.00-9.00 sec 692 MBytes 5.80 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 231 MBytes 1.93 Gbits/sec 0 638 KBytes
[ 7] 9.00-10.00 sec 116 MBytes 970 Mbits/sec 0 577 KBytes
[ 9] 9.00-10.00 sec 230 MBytes 1.93 Gbits/sec 0 612 KBytes
[ 11] 9.00-10.00 sec 115 MBytes 968 Mbits/sec 0 568 KBytes
[SUM] 9.00-10.00 sec 692 MBytes 5.80 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 10.00-11.00 sec 231 MBytes 1.94 Gbits/sec 0 638 KBytes
[ 7] 10.00-11.00 sec 115 MBytes 963 Mbits/sec 0 577 KBytes
[ 9] 10.00-11.00 sec 231 MBytes 1.94 Gbits/sec 0 612 KBytes
[ 11] 10.00-11.00 sec 115 MBytes 968 Mbits/sec 0 568 KBytes
[SUM] 10.00-11.00 sec 692 MBytes 5.81 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 11.00-12.00 sec 230 MBytes 1.93 Gbits/sec 0 638 KBytes
[ 7] 11.00-12.00 sec 116 MBytes 970 Mbits/sec 0 577 KBytes
[ 9] 11.00-12.00 sec 231 MBytes 1.93 Gbits/sec 0 612 KBytes
[ 11] 11.00-12.00 sec 116 MBytes 970 Mbits/sec 0 568 KBytes
[SUM] 11.00-12.00 sec 692 MBytes 5.81 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 12.00-13.00 sec 239 MBytes 2.00 Gbits/sec 0 638 KBytes
[ 7] 12.00-13.00 sec 119 MBytes 1.00 Gbits/sec 0 577 KBytes
[ 9] 12.00-13.00 sec 215 MBytes 1.80 Gbits/sec 0 926 KBytes
[ 11] 12.00-13.00 sec 119 MBytes 996 Mbits/sec 0 568 KBytes
[SUM] 12.00-13.00 sec 691 MBytes 5.80 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 13.00-14.00 sec 232 MBytes 1.95 Gbits/sec 0 961 KBytes
[ 7] 13.00-14.00 sec 115 MBytes 966 Mbits/sec 0 577 KBytes
[ 9] 13.00-14.00 sec 232 MBytes 1.95 Gbits/sec 0 926 KBytes
[ 11] 13.00-14.00 sec 117 MBytes 980 Mbits/sec 0 848 KBytes
[SUM] 13.00-14.00 sec 696 MBytes 5.84 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 14.00-15.00 sec 229 MBytes 1.92 Gbits/sec 0 961 KBytes
[ 7] 14.00-15.00 sec 116 MBytes 974 Mbits/sec 0 577 KBytes
[ 9] 14.00-15.00 sec 231 MBytes 1.93 Gbits/sec 0 926 KBytes
[ 11] 14.00-15.00 sec 116 MBytes 969 Mbits/sec 0 848 KBytes
[SUM] 14.00-15.00 sec 691 MBytes 5.80 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 15.00-16.00 sec 231 MBytes 1.94 Gbits/sec 0 961 KBytes
[ 7] 15.00-16.00 sec 115 MBytes 967 Mbits/sec 0 577 KBytes
[ 9] 15.00-16.00 sec 231 MBytes 1.94 Gbits/sec 0 926 KBytes
[ 11] 15.00-16.00 sec 116 MBytes 969 Mbits/sec 0 848 KBytes
[SUM] 15.00-16.00 sec 693 MBytes 5.82 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 16.00-17.00 sec 230 MBytes 1.93 Gbits/sec 0 961 KBytes
[ 7] 16.00-17.00 sec 115 MBytes 962 Mbits/sec 0 577 KBytes
[ 9] 16.00-17.00 sec 230 MBytes 1.93 Gbits/sec 0 926 KBytes
[ 11] 16.00-17.00 sec 116 MBytes 970 Mbits/sec 0 848 KBytes
[SUM] 16.00-17.00 sec 690 MBytes 5.79 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 17.00-18.00 sec 231 MBytes 1.94 Gbits/sec 0 961 KBytes
[ 7] 17.00-18.00 sec 116 MBytes 970 Mbits/sec 0 577 KBytes
[ 9] 17.00-18.00 sec 231 MBytes 1.94 Gbits/sec 0 926 KBytes
[ 11] 17.00-18.00 sec 115 MBytes 968 Mbits/sec 0 848 KBytes
[SUM] 17.00-18.00 sec 694 MBytes 5.82 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 18.00-19.00 sec 230 MBytes 1.93 Gbits/sec 0 961 KBytes
[ 7] 18.00-19.00 sec 115 MBytes 963 Mbits/sec 0 577 KBytes
[ 9] 18.00-19.00 sec 230 MBytes 1.93 Gbits/sec 0 926 KBytes
[ 11] 18.00-19.00 sec 115 MBytes 961 Mbits/sec 0 848 KBytes
[SUM] 18.00-19.00 sec 689 MBytes 5.78 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 19.00-20.00 sec 231 MBytes 1.94 Gbits/sec 0 961 KBytes
[ 7] 19.00-20.00 sec 114 MBytes 959 Mbits/sec 0 577 KBytes
[ 9] 19.00-20.00 sec 231 MBytes 1.94 Gbits/sec 0 926 KBytes
[ 11] 19.00-20.00 sec 117 MBytes 983 Mbits/sec 0 848 KBytes
[SUM] 19.00-20.00 sec 694 MBytes 5.82 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 4.51 GBytes 1.94 Gbits/sec 0 sender
[ 5] 0.00-20.02 sec 4.51 GBytes 1.94 Gbits/sec receiver
[ 7] 0.00-20.00 sec 2.25 GBytes 968 Mbits/sec 0 sender
[ 7] 0.00-20.02 sec 2.25 GBytes 966 Mbits/sec receiver
[ 9] 0.00-20.00 sec 4.49 GBytes 1.93 Gbits/sec 0 sender
[ 9] 0.00-20.02 sec 4.49 GBytes 1.93 Gbits/sec receiver
[ 11] 0.00-20.00 sec 2.27 GBytes 973 Mbits/sec 0 sender
[ 11] 0.00-20.02 sec 2.26 GBytes 971 Mbits/sec receiver
[SUM] 0.00-20.00 sec 13.5 GBytes 5.81 Gbits/sec 0 sender
[SUM] 0.00-20.02 sec 13.5 GBytes 5.80 Gbits/sec receiver
iperf Done.