January 27, 2019

Categories: howto

Fedora on the ODROID-HC2

Table of Contents

This blog post is about installing Fedora 29 on an ODROID-HC2 (from a Fedora 29 x86_64 workstation).

The aim is to get the ODROID in a state where one can ssh as root to the host.

Both initial setup over serial and installing without using a serial console are described.

Ansible and further uses will be described in separate posts.

Note on Other OS Images

Images for the ODROID-XU4 are fully compatible with the HC2 (source: manual from Pollin, the German shop I bought mine from). So you are defiitely not limited to running Fedora on the ODROID-HC2 Single Board Computer (SBC).

Overview of Steps

obtain Fedora for armhfp
write Fedora image with fedora-arm-image-installer
fuse signed blobs from ODROID into SD card
potentially disable need for serial console (initial-setup.service)
boot ODROD-HC2 from prepared card

General Information on Fedora for ARM

Architectures/ARM on the Fedora Project Wiki
Installing Fedora on your ARM device on the Fedora Project Wiki
The Discussions about the Fedora ARM Project Mailing-List

Hardware Overview


SOC	Samsung Exynos 5 Octa (5422)
CPU	2.1GHz Quad-Core (Cortex®-A15) + 1.4GHz Quad-Core (Cortex®-A7)
RAM	2GB LPDDR3
storage	1 SATA port
µSD slot	UHS-1 capable micro-SD card slot for boot media

Note that while you can plug in UHS-1 cards, do not expect the controller on the ODROID-HC2 to use them to their full potential. I get a tad over 21MiB/s with my µSD cards in the ODROID-HC2, while the same cards gived me a tad over 82MiB/s on my workstation.

Hardware Notes

The HC2 is closely ralated to Hardkernel’s ODROID-XU4 and ODROID-HC1. Kernels for XU4 are bootable on HC2.

On the PCB, my REV 0.1 20171017 is actually identified as MSIP-REM-HKL-ODROID-XU4S

Note though that the HC2 has no graphics output, so you can drop the console=tty1 you see in some guides.

FIXME: Since the SOC does have a Mali™-T628 MP6, check specs to see if it can be used for something besides gfx out.

Also, the HC2 has no eMMC, just a plain µSD. So eMMC related step in XU4 guides do not apply. The parts not referring to XU4 connectors not present on the HC2 should apply.

Performance wise the XU4 is way better than Raspberry Pi 3, ODROID-C1+, ODROID-C2, …

Detailed Specifications

To get proper details on specification, please see all of the following:

Hardkernel’s ODROID-HC2 shop page
Hardkernel’s ODOID-XU4 wiki section since the HC2 is a XU4 on a different PCB and ther is no HC2 category as of 2019-01-26
Hardkernel’s ODOID-XU4 boot sequence
Samsung’s Exynos 5 Octa (5422) page
Wikipedia’s Comparison of single-board computers entry

Board Specific Notes

Things You Need to Keep in Mind

Now at the start of 2019 it is sadly still too often the case that ARM boards have their quirks that need to be taken into account during initial bring up.

I found the below with regards to the ODROID-HC2, once you deal with them and have Fedora up, the rest is sompletely pain free for me so far.

Unfortunately (as of XU3/4), the FBL (equivalent of u-boot’s SPL) must be signed to run on Exynos5 SoCs. As a result EXYNOS5 devices must explicitly reuse the pre-built binary FPL code from HardKernel and can only use u-boot only as a second stage bootloader. this is also true for the HC2.
While Hardkernel has their U-Boot fork on github,
- as of 2019-01-26 the signed bits needed are in the odroidxu4-v2017.05 branch, not master
I prefer to use as much as possible the more actively maintained upstream U-Boot.
- See the section on fusing to use packaged uboot-images-armv7 with signed FBL.
U-Boot 2018.09 does not recognise the HC2
- you have to ln -s exynos5422-odroidxu4.dtb exynos5422-odroidunknown.dtb.
U-Boot 2019-01 has the necessary change. No more symlinking necessary.
How to fuse the signed blobs
not all SATA drives have screw holes matching those of the HC2 case
hard drive needs parking at shutdown
this is a big.LITTLE desig, not all CPU cores are the same speed.

See the ODROID Wiki for a description of the boot sequence.

Goodies the ODROID-HC2 Board offers

It’s not all quirky though, you can also

control the blue LED with the trigger mode in sysfs
- the instructions below set it up for heartbeat, so you can see if your headless HC2 is alive.
if you have a working RTC battery, use RTC Wakeup
you can run the ODROID-HC2 with mini DC UPS

Check the ODROID WIKI for more. Until there is a separate category for HC2, keep in mind that the HC2 is has the same Exynos 5422 as the XU4 but on a different PCB (and thus connectors differ).

Until there is a more detailed Manual for the HC2, you may want to look at the ODROID-XU4 User Manual.

Tested From a Fedora 29 x86_64 Workstation

As you might be aware, my OS choices are (by order of preference);

Fedora
Red Hat Enterprise Linux
CentOS
Debian

So the x86_64 workstation used to prepare the µSD card is currently running Fedora 29, since it’s the current version. If you use RHEL or CentOS they will probably also apply. Same for different architectures of Fedora 29, again let me know how you fared. Let me know if you tried. For Debian setting up an ODROID (XU4, HC1, HC2) are well documented. The latest Linux Kernel 4.14 LTS runs on the XU4 and thus also the HC2. If you are inflicted with Windows or OS X on your workstation, you are on your own.

Install Needed Software

On your workstation

sudo dnf install arm-image-installer uboot-images-armv7

I used

arm-image-installer-2.10-1.fc29.noarch
uboot-images-armv7-2019.01-1.fc30.noarch

Download Fedora for ARMv7

Get the armhfs image I chose the Minimal image for ARM Servers (and it’s checksum file) since I plan to add what’s missing with Ansible.

Verify the file with

On your workstation

sha256sum -c Fedora-Spins-29-1.2-armhfp-CHECKSUM

Write Fedora Image to µSD card

Attach a µSD card to your workstation and check with lsblk what the name is. In the following examples I’ll be using /dev/sdi.

Change to the directory where you downloaded the Minimal image for ARM Servers.

Obviously adjust the path to your ssh pubkey.

Also I immediately inject my ssh key. For initial bringup, I’ll be using --norootpass.

On your workstation

sudo fedora-arm-image-installer \
  --target=none \
  --image=Fedora-Minimal-armhfp-29-1.2-sda.raw.xz \
  --addkey=/home/pcfe/.ssh/id_USBkey.pub \
  --norootpass \
  --resizefs \
  --args "console=ttySAC2,115200n8 cpuidle.off=1 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd no_bL_switcher" \
  --media=/dev/sdi

If you use --resizefs the whole remaining space on your µSD card will be used to grow /. Alternatively you can drop that parameter and comfortably resize partition 3 on the card on your linux workstation with gnome-disks. Remember to re-plug the card if told to do so by a script.

Comments on the Used kernel cmdline Parameters

cpuidle.off=1 is needed for my board to boot up.
console=ttySAC2,115200n8 for the USB-UART Module Kit.
rd.driver.pre=ledtrig-heartbeat and /etc/dracut.conf.d/ledtrig-heartbeat.conf and dracut --force --verbose are all needed to get heartbeat LED working
rd.driver.pre=xhci-plat-hcd is still needed to get the NIC (r8152) to a USB3 connection (5000M)
no_bL_switcher boot with Big.Little.Switcher off (and thus all 8 cores online). You can toggle it at runtime too.

check these fedora-arm mailing list threads

the thread F28 on odroid XU4
the thread f29 uboot & odroid hc2
the thread how to make a working f28 sd card for odroid-hc2

kernel cmdline Used by Hardkernel’s latest Ubuntu image

I wrote the ubuntu-18.04.1-4.14-minimal-odroid-xu4-20181203 to a card, mounted it and grep-ed; it seems to use;

I got the following result:

boot/boot.ini:setenv bootrootfs "console=tty1 console=ttySAC2,115200n8 root=UUID=e139ce78-9841-40fe-8823-96a304a09859 rootwait ro fsck.repair=yes net.ifnames=0"

FIXME: boot that and check /proc/cmdline. boot initiated 2019-01-27 16:48 CET, heartbeat blinked, still not pinging at 16:54 CET probably also has some first boot setup re-try when I have a working serial console again.

Fuse SD card

Why this is needed is well explained in the following two

U-Boot’s README.odroid
The post [fedora-arm] Fedora 27 on ODroid UX4 in the Discussions about the Fedora ARM Project

Preparation; download signed blobs and tool

A big thank you to Chris for this blogpost. That gives straightforward commands one can copypasta.

Download the required files from Hardkernel.

On your workstation

mkdir hardkernel ; cd hardkernel

wget https://raw.githubusercontent.com/hardkernel/u-boot/odroidxu4-v2017.05/sd_fuse/sd_fusing.sh \
https://raw.githubusercontent.com/hardkernel/u-boot/odroidxu4-v2017.05/sd_fuse/bl1.bin.hardkernel \
https://raw.githubusercontent.com/hardkernel/u-boot/odroidxu4-v2017.05/sd_fuse/bl2.bin.hardkernel.720k_uboot \
https://raw.githubusercontent.com/hardkernel/u-boot/odroidxu4-v2017.05/sd_fuse/tzsw.bin.hardkernel

chmod a+x sd_fusing.sh

Use U-Boot files provided by Fedora (since they are more modern)

Copy the Fedora U-Boot files into the local dir.

On your workstation

cp /usr/share/uboot/odroid-xu3/u-boot.bin .

Fuse your SD card

Finally, run the fusing script to embed the files onto the SD card, passing in the device for your SD card.

On your workstation

sudo ./sd_fusing.sh /dev/sdi

Again, be sure to write to the correct device (my setup uses /dev/sdi)

It should end with

[...]
U-boot image is fused successfully.
Eject /dev/sdi and insert it again.

Do as instructed, re-plug the µSD card.

Mount partition 3 of the card (/), for the next step.

Fix for heartbeat LED

When Dracut makes the initramfs, it must find instructions to load ledtrig-heartbeat, in addition to the rd.driver.pre=ledtrig-heartbeat

Create the file while you have / mounted anyway

On your workstation, as root

cat <<EOF >/run/media/pcfe/__/etc/dracut.conf.d/ledtrig-heartbeat.conf
add_drivers+=" ledtrig-heartbeat "
EOF

Do not forget to umount the card

unmount all partitions from the µSD card, remove from your workstation, insert into ODROID. Do not apply power yet.

Either re-do the initramfs once your ODROID has booted or ijust wait for dracut to trigger on a kernel update.

On the ODROID, as root

dracut --force --verbose

Physically Connect Serial Console

Wire up your USB-UART Module Kit

Check with dmesg or journalctl --lines=50 --follow --full what ttyUSB got assigned. Use screen (or minicom or …) to connect to that serial interface. For most people the device will be /dev/ttyUSB0

Persistent names for multiple UART Kits

If, like me, you end up with multiple UART Kits permanently connected to various ODROIDs, it is useful to give them unique names. To do this, you need to determine their serial number, this will be used to differentiate them.

udevadm info -a -p $(udevadm info -q path -n /dev/ttyUSB0)|less

[...]
  looking at parent device '/devices/[...]
    [...]
    DRIVERS=="usb"
    [...]
    ATTRS{idVendor}=="10c4"
    [...]
    ATTRS{serial}=="01234567"
    [...]
    ATTRS{idProduct}=="ea60"
    [...]
    ATTRS{product}=="CP2104 USB to UART Bridge Controller"
[...]

Use that information to create unique symlinks, like so;

DRIVERS=="usb" ATTRS{idProduct}=="ea60", ATTRS{idVendor}=="10c4", ATTRS{serial}=="01234567", SYMLINK+="ODROID-UART-00", GROUP="dialout", MODE="0660"
DRIVERS=="usb" ATTRS{idProduct}=="ea60", ATTRS{idVendor}=="10c4", ATTRS{serial}=="89012345", SYMLINK+="ODROID-UART-01", GROUP="dialout", MODE="0660"

Remember to udevadm control --reload after changing a file under /etc/udev/rules.d/. Test with udevadm test $(udevadm info -q path -n /dev/ttyUSB0) 2>&1

Viev Serial Console on Your Workstation

using screen

The following example uses screen. On your workstation

sudo screen -fn /dev/ttyUSB0 115200

(-fn to disable flow control, speed must match what you set up in extlinux.conf, 115200 is a good choice that is known working).

Exit with ctrl-a \

You should see the ODROID boot when you apply power to it.

Verify that you hve the intended version of U-Boot (since that’s printed soon after power up, the ODROID-HC2 was powered up only after the serial connection was up)

View the serial output on your workstation

U-Boot 2019.01 (Jan 15 2019 - 09:25:07 +0000) for ODROID-XU3/XU4/HC1/HC2

CPU:   Exynos5422 @ 800 MHz
Model: Odroid XU3 based on EXYNOS5422
Board: Odroid XU3 based on EXYNOS5422
Type:  hc1
DRAM:  2 GiB
MMC:   EXYNOS DWMMC: 0, EXYNOS DWMMC: 2
Loading Environment from MMC... Card did not respond to voltage select!
*** Warning - No block device, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  0
MMC Device 1 not found
no mmc device at slot 1
Card did not respond to voltage select!
switch to partitions #0, OK
mmc2 is current device
Scanning mmc 2:2...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
603 bytes read in 4 ms (146.5 KiB/s)
Ignoring unknown command: ui
Ignoring malformed menu command:  autoboot
Ignoring malformed menu command:  hidden
Ignoring unknown command: totaltimeout
Fedora-Minimal-armhfp-29-1.2 Boot Options.
1:      Fedora-Minimal-armhfp-29-1.2 (4.18.16-300.fc29.armv7hl)
Enter choice: 1:        Fedora-Minimal-armhfp-29-1.2 (4.18.16-300.fc29.armv7hl)
Retrieving file: /initramfs-4.18.16-300.fc29.armv7hl.img
41153549 bytes read in 3026 ms (13 MiB/s)
Retrieving file: /vmlinuz-4.18.16-300.fc29.armv7hl
7012864 bytes read in 513 ms (13 MiB/s)
append: console=ttySAC2,115200n8 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd no_bL_switcher ro root=UUID=2161061e-8612-4e18-a4e1-0e95aca6d2ff
Retrieving file: /dtb-4.18.16-300.fc29.armv7hl/exynos5422-odroidhc1.dtb
53395 bytes read in 163 ms (319.3 KiB/s)
Kernel image @ 0x42000000 [ 0x000000 - 0x6b0200 ]
## Flattened Device Tree blob at 43000000
   Booting using the fdt blob at 0x43000000
   Loading Ramdisk to 4d8c0000, end 4ffff40d ... OK
   Loading Device Tree to 4d8af000, end 4d8bf092 ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x100
[    0.000000] Linux version 4.18.16-300.fc29.armv7hl (mockbuild@buildvm-armv7-06.arm.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sun Oct 21 00:56:28 UTC 2018
[    0.000000] CPU: ARMv7 Processor [410fc073] revision 3 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: Hardkernel Odroid HC1
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 64 MiB at 0xba800000
[    0.000000] Samsung CPU ID: 0xe5422001
[    0.000000] Running under secure firmware.
[    0.000000] percpu: Embedded 16 pages/cpu @(ptrval) s34956 r8192 d22388 u65536
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 516928
[    0.000000] Kernel command line: console=ttySAC2,115200n8 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd no_bL_switcher ro root=UUID=2161061e-8612-4e18-a4e1-0e95aca6d2ff
[ ... ]

using conserver

Since I have 5 boards, it makes more sense to wire up 5 UART Kits permanently to the ODROID-HC2 nodes on one end and my server on the other. Using conserver allows me to have permanent serial logs.

Currently 2 ODROIDs are permanently wired up, I play to get one UARM module per ODROID.

conserver.cf

root@epyc ~ # grep "^[^#;]" /etc/conserver.cf
group sysadmin {
    users root, pcfe;
}
group helpers {
    users *, !sysadmin;
}
default global {
        logfile /var/consoles/&;        # '&' is replaced with console name
        timestamp 1hab;                 # write timestamps
        rw sysadmin;                    # allow sysadmins full access
        ro helpers;                     # allow helpers to watch
}
default odroid {
        type device;
        device /dev/ODROID-UART-0.;
        devicesubst .=Pd;
        portbase 0;
        portinc 1;
        host none;
        baud 115200;
        parity none;
        options autoreinit;
}
default * { include global; master localhost; }
console odroid-hc2-00 { include odroid; port 0; }
console odroid-hc2-01 { include odroid; port 1; }
access * {
        trusted 127.0.0.1;
        allowed 192.168.50.35/24;
}
config * {
        defaultaccess rejected;
        daemonmode on;
        logfile /var/log/conserver;
}

/etc/console.cf

config * {
    master localhost;
}

sample usage

pcfe@epyc ~ $ console odroid-hc2-00
[Enter `^Ec?' for help]

Finish Initial Setup via Serial Console

Apart from debugging when something is broken, by default you need a serial console to go through initial setup questions at first boot.

When prompted, set language, timezone, rootpassword, etc.

Disable Initial Setup if you have no Serial Console

My first UART Dongle broke shortly after I started suing it. So I had to do a few installs without a serial connection while waiting on 2 new modules to arrive in the post.

To be able to initially boot without a serial console connected, you need to

write the image to the µSD card
fuse the card
mount the card on your workstation
remove 2 symlinks to disable initial-setup.service
umount the card

Write the Image

Do leave the console=ttySAC2,115200n8 statement in so that you can connect a serial console at a later stage.

I also decided to drop --norootpass, I drop in my ssh pubkey and will anyway handle users and passwords with Ansible.

On your workstation

sudo fedora-arm-image-installer \
  --target=none \
  --image=Fedora-Minimal-armhfp-29-1.2-sda.raw.xz \
  --addkey=/home/pcfe/.ssh/id_USBkey.pub \
  --resizefs \
  --args "console=ttySAC2,115200n8 cpuidle.off=1 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd no_bL_switcher" \
  --media=/dev/sdi

Fuse as usual

Use U-Boot armv7 ≥ 2019.01

Verify intended image (U-Boot armv7 should be ≥ 2019.01) is in place;

On your workstation

pcfe@karhu ~ $ cd work/hardkernel/
pcfe@karhu hardkernel $ rpm -qf /usr/share/uboot/odroid-xu3/u-boot.bin
uboot-images-armv7-2019.01-1.fc30.noarch
pcfe@karhu hardkernel $ sha256sum /usr/share/uboot/odroid-xu3/u-boot.bin u-boot.bin
359e68e3cf865f5c36d128290b7327d49e00cd17ffbedcb1e57e45bf467327ab  /usr/share/uboot/odroid-xu3/u-boot.bin
359e68e3cf865f5c36d128290b7327d49e00cd17ffbedcb1e57e45bf467327ab  u-boot.bin

do the fusing

On your workstation

sudo ./sd_fusing.sh /dev/sdi

It should end with

[...]
U-boot image is fused successfully.
Eject /dev/sdi and insert it again.

Do as instructed, re-plug the µSD card.

disable initial-setup (and fix heartbeat too)

Mount / (the 3rd partition of the card), I lazily used my KDE Plasma desktop Device Notifier ;-) Adjust the path according to your mount point.

On your workstation, as root

find /run/media/pcfe/__/ -name "initial-setup.service" -type l -exec /bin/rm -i {} \;

you should get

/bin/rm: remove symbolic link '/run/media/pcfe/__/etc/systemd/system/graphical.target.wants/initial-setup.service'?
/bin/rm: remove symbolic link '/run/media/pcfe/__/etc/systemd/system/multi-user.target.wants/initial-setup.service'?

Obviously, answer y to both questions.

do not forget the heartbeat fix

On your workstation, as root

cat <<EOF >/run/media/pcfe/__/etc/dracut.conf.d/ledtrig-heartbeat.conf
add_drivers+=" ledtrig-heartbeat "
EOF

Umount

Do not forget to umount!

The Rest is as Usual

Transfer card to ODROID-HC2 (not powered, all LEDs off) and apply power to boot.

Expect it to

grab an IP via DHCP
respond to ping after less than a minute
respond to ssh -v root@<IP> after less than two minutes

If these do not happen you will want to connect a serial console.

FIXME: find on odroid wiki the details so that any USB serial (e.g. buspirate) works at the correct voltage. Probably 1.8V, not 3.3, so check specs before wiring up

All done

You should now be able to log in via ssh to your ODROID-HC2 running Fedora ARM

Additional Testing and Configuration Performed

My test logs and notes follow. This section is mainly a braindump for my own reference.

Completed, Watchdog Timer

Fedora supports the ODROID-HC2’s hardware watchdog out of the box, all you need to do is configure the daemon.

[root@odroid-hc2-00 ~]# grep -i s3c2410 /boot/config-4.18.16-300.fc29.armv7hl
CONFIG_HAVE_S3C2410_I2C=y
CONFIG_I2C_S3C2410=m
CONFIG_HAVE_S3C2410_WATCHDOG=y
CONFIG_S3C2410_WATCHDOG=m
[root@odroid-hc2-00 ~]# grep -i s3c2410 /boot/config-4.20.5-200.fc29.armv7hl
CONFIG_HAVE_S3C2410_I2C=y
CONFIG_I2C_S3C2410=m
CONFIG_HAVE_S3C2410_WATCHDOG=y
CONFIG_S3C2410_WATCHDOG=m

I set this up in my Site-Specific Setup Playbook See https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdShutdownWatchdog and http://0pointer.de/blog/projects/watchdog.html for details.

Also see the following links;

Testing the watchdog

Do not attempt the below on your ODROID-HC2, as root, unless you

want to verify a crashed ODROID gets reset by the hardware watchdog within 30 seconds
ideally have serial console wired up, so that you can follow what os going on
are OK with this test hard-resetting your ODROID-HC2

# echo '1' > /proc/sys/kernel/sysrq
# echo 'c' > /proc/sysrq-trigger

Completed, Heartbeat LED Functionality

heartbeat LED needs kernel cmdline entry and preloading by Dracut. Posted my findings to ARM list.

Completed, `rd.driver.pre=xhci-plat-hcd`

rd.driver.pre=xhci-plat-hcd is needed (to get Driver=r8152, 5000M instead of Driver=r8152, 480M). Posted my findings to ARM list.

[root@odroid-hc2-00 ~]# cat /proc/cmdline
console=ttySAC2,115200n8 cpuidle.off=1 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd no_bL_switcher ro root=UUID=2161061e-8612-4e18-a4e1-0e95aca6d2ff LANG=en_US.UTF-8
[root@odroid-hc2-00 ~]# lsusb -t
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=exynos-ohci/3p, 12M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=exynos-ehci/3p, 480M

[root@odroid-hc2-00 ~]# cat /proc/cmdline
console=ttySAC2,115200n8 cpuidle.off=1 rd.driver.pre=ledtrig-heartbeat no_bL_switcher ro root=UUID=2161061e-8612-4e18-a4e1-0e95aca6d2ff LANG=en_US.UTF-8
[root@odroid-hc2-00 ~]# lsusb -t
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=r8152, 480M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=exynos-ehci/3p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=exynos-ohci/3p, 12M

Completed, SATA HDD performance

FIXME: these perf tests done with big.LITTLE switcher off and no cgroup setup. Re-run after at least nailing IRQs to fast cores.

All the above was actually done without a SATA drive connected. There is no technical reason to not do all the steps above with a SATA drive already plugged in. I simply was concentrating on the bring up first as I expected no issues with the SATA side

I now plugged one in. As expected I can talk to the disk at a decent speed

on the ODROID-HC2, as root

[root@localhost ~]# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 149.1G  0 disk
mmcblk0     179:0    0  29.7G  0 disk
|-mmcblk0p1 179:1    0    76M  0 part /boot/efi
|-mmcblk0p2 179:2    0   489M  0 part /boot
`-mmcblk0p3 179:3    0  19.9G  0 part /
zram0       252:0    0 948.9M  0 disk [SWAP]

Performance is decent, considering this is a cheap SBC and I used an old SATA drive from the shelf Seriously old drive;

Samsung HD161HJ
160 GB SATA
decommissioned in 2010

FIXME: re-test on one of the other 4 ODROID-HC2 nodes. Since they will eventually become OSDs, they will get much more modern drives

FIXME: also re-run tests after cgroups big.LITTLE tuning.

Sequential write comes in at 77.8MiB/s

[root@odroid-hc2-00 ~]# fio --rw=write --name=test -size=10G --filename=/dev/sda --bs=4k
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=76.9MiB/s][r=0,w=19.7k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=6404: Mon Jan 28 15:38:17 2019
  write: IOPS=19.9k, BW=77.8MiB/s (81.5MB/s)(10.0GiB/131679msec)
    clat (usec): min=11, max=16473, avg=40.63, stdev=434.28
     lat (usec): min=12, max=16474, avg=42.57, stdev=434.33
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   13], 20.00th=[   13],
     | 30.00th=[   14], 40.00th=[   15], 50.00th=[   16], 60.00th=[   18],
     | 70.00th=[   20], 80.00th=[   23], 90.00th=[   29], 95.00th=[   38],
     | 99.00th=[   80], 99.50th=[  111], 99.90th=[ 8586], 99.95th=[10159],
     | 99.99th=[12649]
   bw (  KiB/s): min=63392, max=166666, per=99.98%, avg=79615.45, stdev=6101.43, samples=263
   iops        : min=15848, max=41666, avg=19903.85, stdev=1525.32, samples=263
  lat (usec)   : 20=70.81%, 50=26.43%, 100=2.17%, 250=0.29%, 500=0.02%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.19%, 20=0.05%
  cpu          : usr=10.91%, sys=43.21%, ctx=32771, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2621440,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=77.8MiB/s (81.5MB/s), 77.8MiB/s-77.8MiB/s (81.5MB/s-81.5MB/s), io=10.0GiB (10.7GB), run=131679-131679msec

Disk stats (read/write):
  sda: ios=73/20294, merge=0/2594160, ticks=801/12803196, in_queue=12861324, util=99.83%

Sequential read comes in at 77.9MiB/s

[root@odroid-hc2-00 ~]# fio --rw=read --name=test -size=10G --filename=/dev/sda --bs=4k
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=77.0MiB/s,w=0KiB/s][r=19.0k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=7804: Mon Jan 28 15:40:43 2019
   read: IOPS=19.9k, BW=77.9MiB/s (81.6MB/s)(10.0GiB/131510msec)
    clat (usec): min=4, max=145646, avg=41.13, stdev=312.41
     lat (usec): min=6, max=145648, avg=42.98, stdev=312.43
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    6], 10.00th=[    7], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    8], 50.00th=[    8], 60.00th=[    9],
     | 70.00th=[    9], 80.00th=[    9], 90.00th=[   10], 95.00th=[   12],
     | 99.00th=[ 1663], 99.50th=[ 2409], 99.90th=[ 2999], 99.95th=[ 3097],
     | 99.99th=[ 3326]
   bw (  KiB/s): min=56984, max=81488, per=99.99%, avg=79721.12, stdev=2430.65, samples=263
   iops        : min=14246, max=20374, avg=19930.28, stdev=607.63, samples=263
  lat (usec)   : 10=92.11%, 20=5.09%, 50=0.42%, 100=0.77%, 250=0.05%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=1.02%, 4=0.52%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=9.28%, sys=38.09%, ctx=55743, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=77.9MiB/s (81.6MB/s), 77.9MiB/s-77.9MiB/s (81.6MB/s-81.6MB/s), io=10.0GiB (10.7GB), run=131510-131510msec

Disk stats (read/write):
  sda: ios=40956/0, merge=2/0, ticks=202697/0, in_queue=202244, util=99.70%

Mixed read (80%) / write (20%) is not so hot

[root@odroid-hc2-00 ~]# fio --rw=readwrite --name=test -size=10G --filename=/dev/sda --bs=4k --rwmixread=80
test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3650: Mon Jan 28 15:32:09 2019
   read: IOPS=15.1k, BW=59.1MiB/s (61.9MB/s)(8191MiB/138651msec)
    clat (usec): min=5, max=755962, avg=49.30, stdev=1438.18
     lat (usec): min=6, max=755963, avg=51.03, stdev=1438.19
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    6], 10.00th=[    7], 20.00th=[    7],
     | 30.00th=[    7], 40.00th=[    8], 50.00th=[    8], 60.00th=[    8],
     | 70.00th=[    9], 80.00th=[    9], 90.00th=[   11], 95.00th=[   15],
     | 99.00th=[ 1205], 99.50th=[ 1631], 99.90th=[ 2802], 99.95th=[ 6980],
     | 99.99th=[55837]
   bw (  KiB/s): min= 8192, max=81920, per=99.95%, avg=60466.40, stdev=18053.74, samples=277
   iops        : min= 2048, max=20480, avg=15116.58, stdev=4513.44, samples=277
  write: IOPS=3782, BW=14.8MiB/s (15.5MB/s)(2049MiB/138651msec)
    clat (usec): min=9, max=207831, avg=22.54, stdev=456.09
     lat (usec): min=10, max=207833, avg=24.48, stdev=456.12
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   13], 10.00th=[   14], 20.00th=[   15],
     | 30.00th=[   16], 40.00th=[   17], 50.00th=[   18], 60.00th=[   19],
     | 70.00th=[   21], 80.00th=[   24], 90.00th=[   30], 95.00th=[   41],
     | 99.00th=[   81], 99.50th=[  104], 99.90th=[  210], 99.95th=[  262],
     | 99.99th=[  474]
   bw (  KiB/s): min= 2024, max=21024, per=99.95%, avg=15122.70, stdev=4515.05, samples=277
   iops        : min=  506, max= 5256, avg=3780.64, stdev=1128.77, samples=277
  lat (usec)   : 10=70.40%, 20=20.58%, 50=6.38%, 100=1.20%, 250=0.18%
  lat (usec)   : 500=0.07%, 750=0.13%, 1000=0.12%
  lat (msec)   : 2=0.61%, 4=0.29%, 10=0.02%, 20=0.02%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%
  cpu          : usr=8.91%, sys=36.80%, ctx=52647, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2096980,524460,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=59.1MiB/s (61.9MB/s), 59.1MiB/s-59.1MiB/s (61.9MB/s-61.9MB/s), io=8191MiB (8589MB), run=138651-138651msec
  WRITE: bw=14.8MiB/s (15.5MB/s), 14.8MiB/s-14.8MiB/s (15.5MB/s-15.5MB/s), io=2049MiB (2148MB), run=138651-138651msec

Disk stats (read/write):
  sda: ios=32021/3951, merge=785/506832, ticks=178900/9342712, in_queue=9550467, util=99.79%

Completed, µSD card, UHS-1, performance

The SD card controller does not manage to use the card’s full potential. Performance is well below that of the old SATA HDD tested above.

One test with same card but plugged into my workstation further down below.

Sequential write comes in at 21.6MiB/s

[root@odroid-hc2-00 ~]# fio --rw=write --name=test -size=5G --filename=/dev/mmcblk0p4 --bs=4k
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=10582: Thu Feb  7 12:51:56 2019
  write: IOPS=5466, BW=21.4MiB/s (22.4MB/s)(5120MiB/239764msec)
    clat (usec): min=11, max=48850, avg=168.40, stdev=1095.40
     lat (usec): min=13, max=48851, avg=171.22, stdev=1095.45
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   13], 10.00th=[   14], 20.00th=[   15],
     | 30.00th=[   17], 40.00th=[   22], 50.00th=[   26], 60.00th=[   33],
     | 70.00th=[   37], 80.00th=[   44], 90.00th=[   56], 95.00th=[   84],
     | 99.00th=[ 7177], 99.50th=[ 8717], 99.90th=[12387], 99.95th=[13173],
     | 99.99th=[15795]
   bw (  KiB/s): min=16479, max=152176, per=99.92%, avg=21847.72, stdev=6033.41, samples=479
   iops        : min= 4119, max=38046, avg=5461.91, stdev=1508.45, samples=479
  lat (usec)   : 20=37.50%, 50=49.69%, 100=9.02%, 250=1.91%, 500=0.14%
  lat (usec)   : 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.03%, 10=1.28%, 20=0.37%, 50=0.01%
  cpu          : usr=9.32%, sys=37.90%, ctx=35456, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=21.4MiB/s (22.4MB/s), 21.4MiB/s-21.4MiB/s (22.4MB/s-22.4MB/s), io=5120MiB (5369MB), run=239764-239764msec

Disk stats (read/write):
  mmcblk0: ios=259/5680, merge=1283/1298306, ticks=139984/12804254, in_queue=638570, util=91.23%
[root@odroid-hc2-00 ~]# fio --rw=read --name=test -size=5G --filename=/dev/mmcblk0p4 --bs=4k

Same SD card in my workstation performs roughly as expected

[root@karhu ~]# fio --rw=write --name=test -size=5G --filename=/dev/sdi4 --bs=4k
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]       
test: (groupid=0, jobs=1): err= 0: pid=30604: Fri Feb  8 12:51:14 2019
  write: IOPS=21.1k, BW=82.3MiB/s (86.3MB/s)(5120MiB/62233msec)
    clat (usec): min=2, max=261960, avg=45.57, stdev=612.94
     lat (usec): min=2, max=261960, avg=45.85, stdev=612.94
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    7],
     | 30.00th=[    7], 40.00th=[    7], 50.00th=[    8], 60.00th=[    8],
     | 70.00th=[    8], 80.00th=[   10], 90.00th=[   12], 95.00th=[   15],
     | 99.00th=[   57], 99.50th=[ 6194], 99.90th=[ 7635], 99.95th=[ 7963],
     | 99.99th=[ 9241]
   bw (  KiB/s): min=64992, max=481632, per=99.99%, avg=84239.67, stdev=39340.51, samples=124
   iops        : min=16248, max=120408, avg=21059.86, stdev=9835.14, samples=124
  lat (usec)   : 4=8.66%, 10=74.10%, 20=14.80%, 50=1.30%, 100=0.51%
  lat (usec)   : 250=0.11%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 10=0.52%, 20=0.01%, 100=0.01%, 500=0.01%
  cpu          : usr=6.36%, sys=14.24%, ctx=8175, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=82.3MiB/s (86.3MB/s), 82.3MiB/s-82.3MiB/s (86.3MB/s-86.3MB/s), io=5120MiB (5369MB), run=62233-62233msec

Disk stats (read/write):
  sdi: ios=26/4945, merge=0/1283655, ticks=73/8372115, in_queue=8480586, util=99.93%

Sequential read comes in at 21.6MiB/s

[root@odroid-hc2-00 ~]# fio --rw=read --name=test -size=5G --filename=/dev/mmcblk0p4 --bs=4k
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=21.7MiB/s,w=0KiB/s][r=5551,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=11500: Thu Feb  7 12:56:31 2019
   read: IOPS=5519, BW=21.6MiB/s (22.6MB/s)(5120MiB/237465msec)
    clat (usec): min=5, max=46294, avg=168.34, stdev=1252.21
     lat (usec): min=6, max=46296, avg=170.82, stdev=1252.21
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    7], 20.00th=[    8],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   13], 60.00th=[   13],
     | 70.00th=[   13], 80.00th=[   15], 90.00th=[   18], 95.00th=[   23],
     | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[10814], 99.95th=[11863],
     | 99.99th=[20579]
   bw (  KiB/s): min=17408, max=23040, per=99.95%, avg=22066.79, stdev=665.87, samples=474
   iops        : min= 4352, max= 5760, avg=5516.64, stdev=166.46, samples=474
  lat (usec)   : 10=32.91%, 20=61.09%, 50=2.95%, 100=1.15%, 250=0.30%
  lat (usec)   : 500=0.03%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.97%, 20=0.58%, 50=0.01%
  cpu          : usr=5.35%, sys=21.00%, ctx=29534, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=21.6MiB/s (22.6MB/s), 21.6MiB/s-21.6MiB/s (22.6MB/s-22.6MB/s), io=5120MiB (5369MB), run=237465-237465msec

Disk stats (read/write):
  mmcblk0: ios=20537/1418, merge=44/503, ticks=408088/24423, in_queue=418421, util=98.08%

Mixed read (80%) / write (20%)

[root@odroid-hc2-00 ~]# fio --rw=readwrite --name=test -size=10G --filename=/dev/mmcblk0p4 --bs=4k --rwmixread=80
test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]  
test: (groupid=0, jobs=1): err= 0: pid=12804: Thu Feb  7 13:07:58 2019
   read: IOPS=4423, BW=17.3MiB/s (18.1MB/s)(8191MiB/474104msec)
    clat (usec): min=5, max=911064, avg=196.57, stdev=5226.08
     lat (usec): min=6, max=911067, avg=199.14, stdev=5226.08
    clat percentiles (usec):
     |  1.00th=[     7],  5.00th=[     8], 10.00th=[     9], 20.00th=[    10],
     | 30.00th=[    11], 40.00th=[    11], 50.00th=[    12], 60.00th=[    13],
     | 70.00th=[    13], 80.00th=[    15], 90.00th=[    17], 95.00th=[    21],
     | 99.00th=[  8717], 99.50th=[  9110], 99.90th=[ 10028], 99.95th=[ 10814],
     | 99.99th=[116917]
   bw (  KiB/s): min=  184, max=23040, per=100.00%, avg=18403.13, stdev=7324.80, samples=911
   iops        : min=   44, max= 5760, avg=4600.72, stdev=1831.20, samples=911
  write: IOPS=1106, BW=4425KiB/s (4531kB/s)(2049MiB/474104msec)
    clat (usec): min=9, max=31192, avg=48.79, stdev=303.98
     lat (usec): min=11, max=31195, avg=51.64, stdev=304.02
    clat percentiles (usec):
     |  1.00th=[   14],  5.00th=[   18], 10.00th=[   21], 20.00th=[   25],   
     | 30.00th=[   27], 40.00th=[   30], 50.00th=[   31], 60.00th=[   34],   
     | 70.00th=[   36], 80.00th=[   41], 90.00th=[   51], 95.00th=[   73],   
     | 99.00th=[  129], 99.50th=[  190], 99.90th=[ 5145], 99.95th=[ 7504],   
     | 99.99th=[10683]
   bw (  KiB/s): min=   40, max= 6440, per=100.00%, avg=4602.59, stdev=1840.08, samples=911
   iops        : min=   10, max= 1610, avg=1150.62, stdev=460.04, samples=911
  lat (usec)   : 10=19.99%, 20=57.59%, 50=17.83%, 100=2.71%, 250=0.53%
  lat (usec)   : 500=0.05%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=1.17%, 20=0.07%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=5.25%, sys=16.66%, ctx=53587, majf=0, minf=31
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2096980,524460,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=17.3MiB/s (18.1MB/s), 17.3MiB/s-17.3MiB/s (18.1MB/s-18.1MB/s), io=8191MiB (8589MB), run=474104-474104msec
  WRITE: bw=4425KiB/s (4531kB/s), 4425KiB/s-4425KiB/s (4531kB/s-4531kB/s), io=2049MiB (2148MB), run=474104-474104msec

Disk stats (read/write):
  mmcblk0: ios=32760/4615, merge=443/513208, ticks=744889/1221329, in_queue=925209, util=96.85%

Completed, big.LITTLE switching

For now I’ve decided to boot my ODROID-HC2 nodes with big.LITTLE switcher disabled. I can easily enable it at run time if I ever feel the need to save power.

If you do not use no_bL_switcher on the kernel command line, then the big.LITTLE switcher initialises automatically and only the 4 little cores are online at boot. This allows the SOC to save power. At higher load, the 4 big cores should get enabled.

Jan 11 12:40:53 localhost kernel: Kernel command line: console=ttySAC2,115200n8 cpuidle.off=1 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd ro root=UUID=2161061e-8612-4e18-a4e1-0e95aca6d2ff LANG=en_US.UTF-8
[...]
Jan 11 12:40:53 localhost kernel: ThumbEE CPU extension supported.
Jan 11 12:40:53 localhost kernel: Registering SWP/SWPB emulation handler
Jan 11 12:40:53 localhost kernel: big.LITTLE switcher initializing
Jan 11 12:40:53 localhost kernel: CPU0 paired with CPU7
Jan 11 12:40:53 localhost kernel: CPU1 paired with CPU6
Jan 11 12:40:53 localhost kernel: CPU2 paired with CPU5
Jan 11 12:40:53 localhost kernel: CPU3 paired with CPU4
Jan 11 12:40:53 localhost kernel: GIC ID for CPU 0 cluster 1 is 4
Jan 11 12:40:53 localhost kernel: GIC ID for CPU 1 cluster 1 is 5
Jan 11 12:40:53 localhost kernel: GIC ID for CPU 2 cluster 1 is 6
Jan 11 12:40:53 localhost kernel: GIC ID for CPU 3 cluster 1 is 7
Jan 11 12:40:53 localhost kernel: GIC ID for CPU 0 cluster 0 is 0
Jan 11 12:40:53 localhost kernel: IRQ54 no longer affine to CPU4
Jan 11 12:40:53 localhost kernel: GIC ID for CPU 1 cluster 0 is 1
Jan 11 12:40:53 localhost kernel: IRQ55 no longer affine to CPU5
Jan 11 12:40:53 localhost kernel: GIC ID for CPU 2 cluster 0 is 2
Jan 11 12:40:53 localhost kernel: IRQ56 no longer affine to CPU6
Jan 11 12:40:53 localhost kernel: GIC ID for CPU 3 cluster 0 is 3
Jan 11 12:40:53 localhost kernel: IRQ57 no longer affine to CPU7
Jan 11 12:40:53 localhost kernel: big.LITTLE switcher initialized
[...]

[root@odroid-hc2-00 ~]# cat /proc/cmdline
console=ttySAC2,115200n8 cpuidle.off=1 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd ro root=UUID=2161061e-8612-4e18-a4e1-0e95aca6d2ff LANG=en_US.UTF-8
[root@odroid-hc2-00 ~]# uptime
 03:11:30 up 3 min,  1 user,  load average: 0,58, 0,44, 0,20
[root@odroid-hc2-00 ~]# cat /sys/kernel/bL_switcher/active
1
[root@odroid-hc2-00 ~]# lscpu
Architecture:         armv7l
Byte Order:           Little Endian
CPU(s):               8
On-line CPU(s) list:  0-3
Off-line CPU(s) list: 4-7
Thread(s) per core:   1
Core(s) per socket:   4
Socket(s):            1
Vendor ID:            ARM
Model:                3
Model name:           Cortex-A7
Stepping:             r0p3
CPU max MHz:          1300,0000
CPU min MHz:          200,0000
BogoMIPS:             42.00
Flags:                half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae

But you can easily online all 8 cores by disabling the big.LITTLE switcher.

[root@odroid-hc2-00 ~]# cat /sys/kernel/bL_switcher/active
1
[root@odroid-hc2-00 ~]# echo '0' > /sys/kernel/bL_switcher/active
[root@odroid-hc2-00 ~]# cat /sys/kernel/bL_switcher/active
0
[root@odroid-hc2-00 ~]# lscpu
Architecture:        armv7l
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           2
Vendor ID:           ARM
Model:               3
Model name:          Cortex-A7
Stepping:            r0p3
CPU max MHz:         1800,0000
CPU min MHz:         200,0000
BogoMIPS:            48.00
Flags:               half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae

If OTOH, you do use no_bL_switcher in the kernel command line, then all 8 cores are available at boot,

Jan 11 12:40:53 localhost kernel: Kernel command line: console=ttySAC2,115200n8 cpuidle.off=1 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd no_bL_switcher ro root=UUID=2161061e-8612-4e18-a4e1-0e95aca6d2ff LANG=en_US.UTF-8
[...]
Jan 11 12:40:53 localhost kernel: ThumbEE CPU extension supported.
Jan 11 12:40:53 localhost kernel: Registering SWP/SWPB emulation handler
Jan 11 12:40:53 localhost kernel: registered taskstats version 1

[root@odroid-hc2-00 ~]# cat /proc/cmdline
console=ttySAC2,115200n8 cpuidle.off=1 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd no_bL_switcher ro root=UUID=2161061e-8612-4e18-a4e1-0e95aca6d2ff LANG=en_US.UTF-8
[root@odroid-hc2-00 ~]# uptime
 03:24:19 up 9 min,  1 user,  load average: 0,76, 0,46, 0,26
[root@odroid-hc2-00 ~]# cat /sys/kernel/bL_switcher/active
0
[root@odroid-hc2-00 ~]# lscpu
Architecture:        armv7l
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           2
Vendor ID:           ARM
Model:               3
Model name:          Cortex-A7
Stepping:            r0p3
CPU max MHz:         1800,0000
CPU min MHz:         200,0000
BogoMIPS:            42.00
Flags:               half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae

You can enable the big.LITTLE switcher at runtime with echo 1 > /sys/kernel/bL_switcher/active. This will, of course, offline 4 cores to save power.

The big cores should get enabled when the bL_switcher deems necessary.

[root@odroid-hc2-00 ~]# cat /sys/kernel/bL_switcher/active
0
[root@odroid-hc2-00 ~]# echo '1' > /sys/kernel/bL_switcher/active

[  674.616963] big.LITTLE switcher initializing
[  674.619928] CPU0 paired with CPU7
[  674.623300] CPU1 paired with CPU6
[  674.626483] CPU2 paired with CPU5
[  674.629701] CPU3 paired with CPU4
[  674.632992] GIC ID for CPU 0 cluster 1 is 4
[  674.637179] GIC ID for CPU 1 cluster 1 is 5
[  674.641322] GIC ID for CPU 2 cluster 1 is 6
[  674.645483] GIC ID for CPU 3 cluster 1 is 7
[  674.649644] GIC ID for CPU 0 cluster 0 is 0
[  674.677122] IRQ54 no longer affine to CPU4
[  674.709511] GIC ID for CPU 1 cluster 0 is 1
[  674.733704] IRQ55 no longer affine to CPU5
[  674.750436] GIC ID for CPU 2 cluster 0 is 2
[  674.771594] IRQ56 no longer affine to CPU6
[  674.786313] GIC ID for CPU 3 cluster 0 is 3
[  674.818131] cpu cpu4: Dropping the link to regulator.40
[  674.825420] IRQ57 no longer affine to CPU7
[  674.839465] big.LITTLE switcher initialized

[root@odroid-hc2-00 ~]# cat /sys/kernel/bL_switcher/active
1
[root@odroid-hc2-00 ~]# lscpu
Architecture:         armv7l
Byte Order:           Little Endian
CPU(s):               8
On-line CPU(s) list:  0-3
Off-line CPU(s) list: 4-7
Thread(s) per core:   1
Core(s) per socket:   4
Socket(s):            1
Vendor ID:            ARM
Model:                3
Model name:           Cortex-A7
Stepping:             r0p3
CPU max MHz:          1800,0000
CPU min MHz:          200,0000
BogoMIPS:             78.00
Flags:                half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae

See the following links for more details;

IKS (In kernel Switcher) page at Linaro.
Linux support for ARM big.LITTLE on LWN.net
General-Purpose NAS in ODROID Magazine of February 2017

Completed, Check Temperature Zone 0 - 4 in the check-mk-agent output

bug

check-mk-agent from Fedora ARm 29 reports a crash At the moment Check_MK reports a crash.

from agent output

<<<lnx_thermal>>>
thermal_zone0 enabled cpu0-thermal 43000 10000 70000 active 10000 85000 active 10000 110000 active 0 120000 critical 
thermal_zone1 enabled cpu1-thermal 43000 10000 70000 active 10000 85000 active 10000 110000 active 0 120000 critical 
thermal_zone2 enabled cpu2-thermal 47000 10000 70000 active 10000 85000 active 10000 110000 active 0 120000 critical 
thermal_zone3 enabled cpu3-thermal 45000 10000 70000 active 10000 85000 active 10000 110000 active 0 120000 critical 
thermal_zone4 enabled gpu-thermal 43000 10000 85000 active 10000 103000 active 10000 110000 active 0 120000 critical 
<<<local>>>

but Exception ValueError (invalid literal for int() with base 10: ‘active’) in Check_MK 1.5.0p11

Submitted a bug report through the Check_MK Raw webUI. I stupidly browsed away from the page without noting any potential ID :-(

workaround

check-mk-agent grabbed from my check-mk-server’s agents page works just fine WRT temps, Implemented Würgaround in My Site-Specific Setup Playbook.

Completed, Check_MK logging S.M.A.R.T. data

reported bug via email, tracked as “FEED-3415: linux smart plugin und JMicron USB nach SATA bridges”
deploying patched version via playbook odroid-general-setup.yml
- FIXME: paste new log output
- updated playbook will go out on next ./deploy.sh

--- smart       2019-02-19 20:45:56.477618050 +0100
+++ /root/smart 2019-02-19 20:47:37.678256369 +0100
@@ -142,6 +142,9 @@
             MODEL=${MODEL// /-}
             DNAME=${DNAME#AMCC_}
             DNAME="AMCC_${MODEL}_${DNAME%000000000000}"
+       # 2019-02-19 Patrick C. F. Ernzer - special option in case vendor is JMicron
+       elif [ "$VEND" == "JMicron" ]; then
+               CMD="smartctl -d sat -v 9,raw48 -A $D"
         elif [ "$VEND" != "ATA" ] ; then
             TEMP=
             # create temperature output as expected by checks/smart

Completed, fix HDD click on shutdown

Since my disks clicked at shutdown, I rolled out the fix documented in the ODROID Wiki via Ansible. See the used template and used playbook

Undecided, Frequency Governors

For now I have decided to use the ondemand governor and set it up via Ansible. I set the governor with a tuned profile (amongst other tuned settings). See the used template and used playbook

I’m not yet 100% set on that governor though.

My testing notes follow;

[root@odroid-hc2-00 ~]# lscpu
Architecture:        armv7l
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           2
Vendor ID:           ARM
Model:               3
Model name:          Cortex-A7
Stepping:            r0p3
CPU max MHz:         1800,0000
CPU min MHz:         200,0000
BogoMIPS:            24.00
Flags:               half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
[root@odroid-hc2-00 ~]# cpupower --cpu all frequency-info
analyzing CPU 0:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency: 154 us
  hardware limits: 200 MHz - 1.30 GHz
  available frequency steps:  200 MHz, 300 MHz, 400 MHz, 500 MHz, 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz
  available cpufreq governors: conservative userspace powersave ondemand performance schedutil
  current policy: frequency should be within 200 MHz and 1.30 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency: 400 MHz (asserted by call to hardware)
analyzing CPU 1:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency: 154 us
  hardware limits: 200 MHz - 1.30 GHz
  available frequency steps:  200 MHz, 300 MHz, 400 MHz, 500 MHz, 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz
  available cpufreq governors: conservative userspace powersave ondemand performance schedutil
  current policy: frequency should be within 200 MHz and 1.30 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency: 700 MHz (asserted by call to hardware)
analyzing CPU 2:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency: 154 us
  hardware limits: 200 MHz - 1.30 GHz
  available frequency steps:  200 MHz, 300 MHz, 400 MHz, 500 MHz, 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz
  available cpufreq governors: conservative userspace powersave ondemand performance schedutil
  current policy: frequency should be within 200 MHz and 1.30 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency: 700 MHz (asserted by call to hardware)
analyzing CPU 3:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency: 154 us
  hardware limits: 200 MHz - 1.30 GHz
  available frequency steps:  200 MHz, 300 MHz, 400 MHz, 500 MHz, 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz
  available cpufreq governors: conservative userspace powersave ondemand performance schedutil
  current policy: frequency should be within 200 MHz and 1.30 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency: 700 MHz (asserted by call to hardware)
analyzing CPU 4:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 4 5 6 7
  CPUs which need to have their frequency coordinated by software: 4 5 6 7
  maximum transition latency: 154 us
  hardware limits: 200 MHz - 1.80 GHz
  available frequency steps:  200 MHz, 300 MHz, 400 MHz, 500 MHz, 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80 GHz
  available cpufreq governors: conservative userspace powersave ondemand performance schedutil
  current policy: frequency should be within 200 MHz and 1.80 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency: 1.80 GHz (asserted by call to hardware)
analyzing CPU 5:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 4 5 6 7
  CPUs which need to have their frequency coordinated by software: 4 5 6 7
  maximum transition latency: 154 us
  hardware limits: 200 MHz - 1.80 GHz
  available frequency steps:  200 MHz, 300 MHz, 400 MHz, 500 MHz, 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80 GHz
  available cpufreq governors: conservative userspace powersave ondemand performance schedutil
  current policy: frequency should be within 200 MHz and 1.80 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency: 1.80 GHz (asserted by call to hardware)
analyzing CPU 6:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 4 5 6 7
  CPUs which need to have their frequency coordinated by software: 4 5 6 7
  maximum transition latency: 154 us
  hardware limits: 200 MHz - 1.80 GHz
  available frequency steps:  200 MHz, 300 MHz, 400 MHz, 500 MHz, 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80 GHz
  available cpufreq governors: conservative userspace powersave ondemand performance schedutil
  current policy: frequency should be within 200 MHz and 1.80 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency: 1.80 GHz (asserted by call to hardware)
analyzing CPU 7:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 4 5 6 7
  CPUs which need to have their frequency coordinated by software: 4 5 6 7
  maximum transition latency: 154 us
  hardware limits: 200 MHz - 1.80 GHz
  available frequency steps:  200 MHz, 300 MHz, 400 MHz, 500 MHz, 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80 GHz
  available cpufreq governors: conservative userspace powersave ondemand performance schedutil
  current policy: frequency should be within 200 MHz and 1.80 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency: 1.80 GHz (asserted by call to hardware)

Check cpufreq governors together with big.LITLE

See here and here for measurement results on odroid forum and here

Plan: put his findings into a tuned profile. Look at tuned-profiles-cpu-partitioning.noarch and the Fedora Power Management Guide and the power management list and this LWN article

Turns out that ODROID Magazine Feb 2017 has what I want on page 12

decide on a governor. performance is too brute force, base one on the article General-Purpose NAS
- in ODROID Magazine of February 2017.
- also available here, where it’s easier to copypaste from
write cgroups rules, same article, page 12 says

# mkdir -p /sys/fs/cgroup/cpuset/littlecores /sys/fs/cgroup/cpuset/bigcores
# echo '0-3' > /sys/fs/cgroup/cpuset/littlecores/cpuset.cpus
# echo '0' > /sys/fs/cgroup/cpuset/littlecores/cpuset.mems
# chmod -R 777 /sys/fs/cgroup/cpuset/littlecores
# echo '4-7' > /sys/fs/cgroup/cpuset/bigcores/cpuset.cpus
# echo '0' > /sys/fs/cgroup/cpuset/bigcores/cpuset.mems
# chmod -R 777 /sys/fs/cgroup/cpuset/bigcores

Make a playbook entry, I guess with geturl and service to do these steps from the article

sudo wget -O /etc/systemd/system/cpuset.service https://raw.githubusercontent.com/mad-ady/odroid-xu4-optimizations/master/cpuset.service
sudo systemctl enable cpuset
sudo systemctl start cpuset

or use snapshot, write to file, read in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-cgsnapshot

[root@odroid-hc2-00 ~]# cgsnapshot --silent cpuset 
# Configuration file generated by cgsnapshot
mount {
        cpuset = /sys/fs/cgroup/cpuset;
}

group bigcores {
        cpuset {
                cpuset.mems="0";
                cpuset.sched_relax_domain_level="-1";
                cpuset.mem_exclusive="0";
                cpuset.cpus="4-7";
                cpuset.mem_hardwall="0";
                cpuset.memory_migrate="0";
                cpuset.memory_spread_page="0";
                cpuset.sched_load_balance="1";
                cpuset.cpu_exclusive="0";
                cpuset.memory_spread_slab="0";
        }
}

group littlecores {
        cpuset {
                cpuset.mems="0";
                cpuset.sched_relax_domain_level="-1";
                cpuset.mem_exclusive="0";
                cpuset.cpus="0-3";
                cpuset.mem_hardwall="0";
                cpuset.memory_migrate="0";
                cpuset.memory_spread_page="0";
                cpuset.sched_load_balance="1";
                cpuset.cpu_exclusive="0";
                cpuset.memory_spread_slab="0";
        }
}

[root@odroid-hc2-00 ~]# grep -E 'CPU0|usb' /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
 84:          1          0          0          0          0          0          0          0     GICv2 103 Level     ohci_hcd:usb1, ehci_hcd:usb2
146:      13045          0          0          0          0          0          0          0     GICv2 104 Level     xhci-hcd:usb3
147:     885840          0          0          0          0          0          0          0     GICv2 105 Level     xhci-hcd:usb5
[root@odroid-hc2-00 ~]# systemctl status irqbalance
Unit irqbalance.service could not be found.
[root@odroid-hc2-00 ~]# cat /proc/irq/84/smp_affinity_list 
0-7
[root@odroid-hc2-00 ~]# cat /proc/irq/146/smp_affinity_list 
0-7
[root@odroid-hc2-00 ~]# cat /proc/irq/147/smp_affinity_list 
0-7
[root@odroid-hc2-00 ~]# echo 4 > /proc/irq/84/smp_affinity_list
[root@odroid-hc2-00 ~]# echo 5 > /proc/irq/146/smp_affinity_list
[root@odroid-hc2-00 ~]# echo 6 > /proc/irq/147/smp_affinity_list
[root@odroid-hc2-00 ~]# grep -E 'CPU0|usb' /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
 84:          1          0          0          0          0          0          0          0     GICv2 103 Level     ohci_hcd:usb1, ehci_hcd:usb2
146:      13156          0          0          0          0         57          0          0     GICv2 104 Level     xhci-hcd:usb3
147:     896693          0          0          0          0          0        692          0     GICv2 105 Level     xhci-hcd:usb5

re-test storage speeds after cgroupd are set up

cgexec -g cpuset:bigcores <command> but remember that currently the disk are in use for Ceph. Also test in single user mode, since Ceph is live on all 5 ODROID-HC2s at the moment but initial test were done early in initial bringup, way before Ceph was even running, let alone configured.

Alternatrively, use https://access.redhat.com/solutions/480473 to disable cores 0-3 or https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/7/html/tuning_guide/interrupt_and_process_binding

Link Collection

The following links came in handy;

Fedora on the ODROID-HC2

Note on Other OS Images

Overview of Steps

General Information on Fedora for ARM

Hardware Overview

Hardware Notes

Detailed Specifications

Board Specific Notes

Things You Need to Keep in Mind

Goodies the ODROID-HC2 Board offers

Tested From a Fedora 29 x86_64 Workstation

Install Needed Software

Download Fedora for ARMv7

Write Fedora Image to µSD card

Comments on the Used kernel cmdline Parameters

check these fedora-arm mailing list threads

kernel cmdline Used by Hardkernel’s latest Ubuntu image

Fuse SD card

Preparation; download signed blobs and tool

Use U-Boot files provided by Fedora (since they are more modern)

Fuse your SD card

Fix for heartbeat LED

Do not forget to umount the card

Physically Connect Serial Console

Persistent names for multiple UART Kits

Viev Serial Console on Your Workstation

using screen

using conserver

conserver.cf

/etc/console.cf

sample usage

Finish Initial Setup via Serial Console

Disable Initial Setup if you have no Serial Console

Write the Image

Fuse as usual

Use U-Boot armv7 ≥ 2019.01

do the fusing

disable initial-setup (and fix heartbeat too)

do not forget the heartbeat fix

Umount

The Rest is as Usual

All done

Additional Testing and Configuration Performed

Completed, Watchdog Timer

Testing the watchdog

Completed, Heartbeat LED Functionality

Completed, rd.driver.pre=xhci-plat-hcd

Completed, SATA HDD performance

Sequential write comes in at 77.8MiB/s

Sequential read comes in at 77.9MiB/s

Mixed read (80%) / write (20%) is not so hot

Completed, µSD card, UHS-1, performance

Sequential write comes in at 21.6MiB/s

Same SD card in my workstation performs roughly as expected

Sequential read comes in at 21.6MiB/s

Mixed read (80%) / write (20%)

Completed, big.LITTLE switching

Completed, Check Temperature Zone 0 - 4 in the check-mk-agent output

bug

workaround

Completed, Check_MK logging S.M.A.R.T. data

Completed, fix HDD click on shutdown

Undecided, Frequency Governors

re-test storage speeds after cgroupd are set up

Link Collection

Related

Completed, `rd.driver.pre=xhci-plat-hcd`