Including ODROID-HC2 Nodes in My Ceph Cluster

I built a Ceph Luminous cluster containing seven ODROID-HC2 nodes. This cluster also contains some x86 VMs running on my hypervisor.

I use this cluster mainly as a Ceph playground.

For Ceph use I definitely did not want to mess about with serial console on the ODROID-HC2 (there are 7 nodes to install, there could be more). The instructions include getting rid of the need for serial console during first boot.

6 ODROID-HC2 nodes

As of this writing, Fedora ARM 29 is the current Fedora version. Expect these steps to continue working with future Fedora and Ceph releases. If something breaks, let me know.

This post gives you all the steps to go from unpacking your ODROID-HC2 to taking control with ceph-ansible.

This post does not explain Ceph.

You do not need a serial console, although I recommend owning one for debugging unexpected failures.

What This Post is About

This post is about setting up one or more ODROID-HC2 so that I can run ceph-ansible against them. This is achieved without interactively doing anything on the serial console.

On my ODROID-HC2 nodes, under Fedora 29, I run:

The following Ceph components I run on x86_64 VMs, running RHEL7 that hosted on my hypervisor:

Note that setting up Ceph with ceph-ansible will be handled in a separate port. This post is just about the ODROID specific steps. Once I can ansible the nodes, for my ceph-ansible use, I’ll treat them no different than x86_64 nodes.

The following Ceph components may be set up in the near future:

This post will be updated when I do.

Overview of Steps

  1. obtain ODROID-HC2 nodes (minimum 3, I recommend at least 5)
  2. obtain Fedora for armhfp
  3. write Fedora image with fedora-arm-image-installer
  4. fuse signed blobs from Hardkernel co., Ltd. into the Fedora ARM image on µSD card
  5. re-plug card when instructed to do so by the fuse script
  6. mount card
  7. disable Fedora’s initial-setup.service (this enables you to boot without interaction on the serial console)
  8. enable the Heartbeat LED
  9. unmount card
  10. boot ODROD-HC2 from prepared card (using DHCP to set up network)
  11. ssh in to install packages needed for Ansible to control the ODROID-HC2
  12. create ansible user using Ansible as user root via ssh
  13. optional: set up root password (I disable password based login via network)
  14. configure HDD sleep timeout
  15. implement some work-arounds

Hardware Bought

Since Ceph needs a minimum of three nodes to be even remotely useful, I bought seven ODROID-HC2 plus needed peripherals but no disks. I had enough old disks on the shelf that I could recycle.

I currently live in Germany, buying from a Eropean supplier saves me the hassle of physically going to customs to pay my dues. More importantly, it grants me consumer rights I expect as a European citizen.

Pollin Electronic GmbH is a distributor in my country. I had used that vendor in the past and had no cause for complaints. I have no relation to Pollin, I just wanted to be able to order from within the EU and my usual vendor does not stock the ODROID-HC2.

Use your distributor or vendor of choice. Since the ODROID-HC2 is usually sold naked without a PSU;

  • potentially add one RTC battery to every ODROID
  • ensure you can suppply 12 V - 2 A to each board
  • consider investing in at least one USB-UART module (aka serial console) for debugging
  • consider adding a cover to the top of your ODROID stack
  • consider adding a low noise fan or two, mainly to cool the disks

My Shopping List

Here are the exact parts I purchased;

Quantity Description Product Page at Pollin
7 ODROID-HC2 ODROID-HC2 Einplatinen Computer für NAS und Cluster Anwendungen
0 power supply Tischnetzteil 12 V- 2 A für Odroid-HC2
0 power cable Euro-Netzkabel mit Doppelnutkupplung, 1,5 m
7 µSD card microSDHC Speicherkarte SANDISK Extreme Pro, 32 GB, UHS-I U3
7 RTC battery ODROID BACKUP BATTERY
1 plastic cover ODROID-HC2 Gehäuse, transparent
1 serial console ODROID USB-UART MODULE, Schnittstellenkarte
7 network cables whichever length suits your setup
7 SATA storage whichever SATA storage you want to use. 2.5” and 3.5” fit in the chassis. SSD or HDD can be used

For SATA storage, as this will be a test cluster, I’ll use old SATA HDD drives that I decommissioned from other machines a while ago.

Notes on My PSU Choice

Both wiring the cluster to my UPS and moving the cluster would be a hassle with 7 power bricks. So I went with;

Still, in the shopping list above I have listed the ODROID power bricks and needed power cables at Pollin. I figured not everyone wants to do custom wiring.

At 5 nodes, the wiring looked as follows;

Since then I have added 2 more nodes but also had to RMA a node. As such, beginning of March 2019 there are 6 OSDs.

Notes on the ODROID-HC2 Single Board Computers

Hardware Notes

This hardware is not exactly what I would have chosen with a bigger budget, but it sure beats having Ceph OSDs with multiple VMs on a single hypervisor that’s hosting qcow2 files on local storage (Which is what I was doig before when testing Ceph installs).

For a 7 node storage cluster made of real hardware, this build is surprisingly compact.

Downsides

General Downsides

Ceph Relevant Downsides

  • There is only one network interface, Gigabit Ethernet connected via USB 3.0
  • There is only one SATA interface, JMicron JMS578 USB 3.0 to SATA Bridge with UAS
  • There is no NVMe interface

Upsides

  • It is near identical to the ODROID-XU4, many guides found on the internet apply.
    • OS images for the XU4 are fully compatibe
    • OS images for the HC1 are fully compatibe
  • The boards are a lot cheaper than current x86_64 solutions.
  • They use little power.
  • As far as Single Board Computers (SBC) go, the ODROID-HC2 is currently (early 2019) comparatively powerful.
  • The whole 7 node cluster will be rather compact since the ODROID-HC2 is stackable.
  • one can fuse the signed blobs and upstream U-Boot bits. This means you still get to use upstream U-Boot, it’s just an extra step.
  • the boards have a hardware watchdog.

OS and Ceph Support

I can run Fedora-Minimal-armhfp-29-1.2 just fine on the boards. Updating to the latest kernel (4.20.3-200.fc29) was completely hassle free.

Because of the architecture, I can not use RHEL 7. Maybe I should have gone with aarch64 hardware instead.

There also is no packaged Red Hat Ceph Storage 3 for ARMv7 but since there are armv7hl packages in Fedora I can use ceph_origin: distro in my ceph-ansible all.yml.

I have not tried CentOS 7 for ARM.

I plan to try el8 once an ARMv7 build (RHEL or CentOS) is available.

As of January 2019, the latest Linux image for HC2 / XU4 from the vendor is Ubuntu 18.04 not something I even considered running.

This is Just a Test Cluster

For anything more than a test cluster I would have chosen

  • boards on which I can install RHEL
  • boards which have more and faster NICs (2 if not 4. Plus, 10 GiG would be nice between the OSDs)
  • more storage connectors and bays (probably 4 HDD and one NVMe for home use)
  • at least one NVMe

I’ve been seriously considering something compact from Supermicro or, as recommended by a work colleague, InWin cases with ASRock boards. Or maybe the upcoming X470D4U. But as I just started with Ceph and different architectures are enjoyable to me, I went with the cheap solution (I paid about 850€ for everything (no disks though!) instead of the same or more for a single Xeon based node, of which I’d realistically want five).

Note on Getting Fedora 29 on the ODROID-HC2 (not Ceph specific)

Normally I install computers by booting over the network and then feed the installer (mostly anaconda) both the configuration (kickstart) file and the OS to be installed via network.

That’s not possible (a naked ODROID will AFAIK not PXE boot), so my choices are:

  1. install using ARM Tools for writing disk images
  2. use Hardkernel’s official images for the ODROID-HC2

My favourite server OS is RHEL / CentOS, closely followed by Fedora (which is my default for workstations, desktops and laptops), none of these are made available by the vendor. Plus, I do prefer to do the OS install myself.

Please see the post Fedora on the ODROID-HC2 for more details on the Fedora 29 ARM installation.

Please see the post ODROID-HC2 and Ansible for more details on how to get an ODROID-HC2’s Fedora install into the state where you can control it with Ansible. Once that is achieved, you can run ceph-ansible against them.

Install Needed Software on Your Workstation

On your workstation

sudo dnf install arm-image-installer uboot-images-armv7

I used

  • arm-image-installer-2.10-1.fc29.noarch
  • uboot-images-armv7-2019.01-1.fc30.noarch

If you use an older version of U-Boot, you will have to create a symlink in /boot/dtb-<version>/ after writing the fedora image and fusing the µSD card plus every time you update the kernel.

The commit making the symlink superfluous is present in U-Boot 2019.01

example (on a booted ODROID)

[root]# cd /boot/dtb-4.18.16-300.fc29.armv7hl/
[root]# ln -s exynos5422-odroidhc1.dtb exynos5422-odroidunknown.dtb

Download Fedora for ARMv7

Get an armhfs image from the Fedora Images for ARM®-based Computers page. I chose the Minimal image for ARM Servers (and it’s checksum file) since I plan to add what’s missing with Ansible.

Verify the file with

sha256sum -c Fedora-Spins-29-1.2-armhfp-CHECKSUM

Determine Device Name of µSD Card

You need to check (e.g. with lsblk or with journalctl --lines=50 --follow --full or with dmesg) for the device name your card has.

Do not blindly copypaste my code if your card is not /dev/sdi like mine is!

Write Fedora Image to µSD card

Important

  • Ensure you adjust --media=/dev/sdi, writing to the wrong disk can result in you having to restore from backup.
  • Obviously adjust the path to the ssh public key given to --addkey=…
  • See Fedora on the ODROID-HC2 for details on --args ….

Exact Command Used

Write the image with the following command.

sudo fedora-arm-image-installer \
  --target=none \
  --image=Fedora-Minimal-armhfp-29-1.2-sda.raw.xz \
  --addkey=/home/pcfe/.ssh/id_USBkey.pub \
  --resizefs \
  --args "console=ttySAC2,115200n8 cpuidle.off=1 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd no_bL_switcher" \
  --media=/dev/sdi
=====================================================
= Selected Image:                                 
= Fedora-Minimal-armhfp-29-1.2-sda.raw.xz
= Selected Media : /dev/sdi
= U-Boot Target : none
= Root partition will be resized
= SSH Public Key /home/pcfe/.ssh/id_USBkey.pub will be added.
=====================================================
 
*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
 
 Type 'YES' to proceed, anything else to exit now 
 
= Proceed? YES
= Writing: 
= Fedora-Minimal-armhfp-29-1.2-sda.raw.xz 
= To: /dev/sdi ....
0+232039 records in
0+232039 records out
1937768448 bytes (1,9 GB, 1,8 GiB) copied, 29,7358 s, 65,2 MB/s
= Writing image complete!
= Resizing /dev/sdi ....
Checking that no-one is using this disk right now ... OK

Disk /dev/sdi: 29,7 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x45c031c3

Old situation:

Device     Boot   Start     End Sectors  Size Id Type
/dev/sdi1          2048  157695  155648   76M  c W95 FAT32 (LBA)
/dev/sdi2  *     157696 1159167 1001472  489M 83 Linux
/dev/sdi3       1159168 3610623 2451456  1,2G 83 Linux

/dev/sdi3: 
New situation:
Disklabel type: dos
Disk identifier: 0x45c031c3

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sdi1          2048   157695   155648   76M  c W95 FAT32 (LBA)
/dev/sdi2  *     157696  1159167  1001472  489M 83 Linux
/dev/sdi3       1159168 62333951 61174784 29,2G 83 Linux

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
e2fsck 1.44.3 (10-July-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
_/: 35219/76640 files (0.8% non-contiguous), 254141/306432 blocks
resize2fs 1.44.3 (10-July-2018)
Resizing the filesystem on /dev/sdi3 to 7646848 (4k) blocks.
The filesystem on /dev/sdi3 is now 7646848 (4k) blocks long.

= No U-Boot files found for none.
= Adding SSH key to authorized keys.
= Adding optional kernel parameters for none : 
= console=ttySAC2,115200n8 cpuidle.off=1 rd.driver.pre=ledtrig-heartbeat,xhci-plat-hcd no_bL_switcher

= Installation Complete! Insert into the none and boot.

Difference to Plain Fedora on ODROID-HC2

The options passed to fedora-arm-image-installer differ as follows from what I used when I initially installed Fedora 29 on the ODROID-HC2;

  1. I removed --norootpass since I do not use initial-setup.service (because the initial setup interrupts boot and needs you to be connected to a serial console).

Set you root password later. Ideally with Ansible, but interactively over ssh is also fine. For both connections (Ansible or ssh interactive), use the privkey matching the pubkey you added with --addkey=…. It is highly recommended to use ssh-agent rather than having a privkey without a password!

Alternatively; disable password based login alltogether instead of changing the root password. That’s whay I do since I always log in with ssh keys.

Note on Using the SD Card as Journal

I thought about keeping some space free on the µSD card, but testing the card inside the ODRID-HC2 with fio showed that mine underperform my HDDs by quite a margin.

I got some 20 MiB/s in both sequential read and write.

Points Verified

See Fedora on the ODROID-HC2 for details.

  • cpuidle.off=1 is needed for the board to boot
  • heartbeat LED needs both kernel cmdline entry rd.driver.pre=ledtrig-heartbeat and preloading by Dracut.
  • rd.driver.pre=xhci-plat-hcd is needed.
  • no_bL_switcher is used to have all 8 cores available at boot.

Points I Might Revisit

  • decide on another governor. performance is too brute force, base one on the article General-Purpose NAS in ODROID Magazine of February 2017, I went with the ondemenad governor and set it via tuned.
  • revisit cgroups rules, same article, page 12, that are currently rolled out via Ansible. maybe use cgsnapshot

Fuse SD card

The process of including the signed blobs from the vendor as well as the latest U-Boot is called fusing. Fedora on the ODROID-HC2 has more details.

Preparation for Fusing; Download Signed Blobs and Tool

A big thank you to Chris for this blogpost. That gives straightforward commands one can copypasta.

Download the required files from Hardkernel.

On your workstation

mkdir hardkernel ; cd hardkernel

wget https://raw.githubusercontent.com/hardkernel/u-boot/odroidxu4-v2017.05/sd_fuse/sd_fusing.sh \
https://raw.githubusercontent.com/hardkernel/u-boot/odroidxu4-v2017.05/sd_fuse/bl1.bin.hardkernel \
https://raw.githubusercontent.com/hardkernel/u-boot/odroidxu4-v2017.05/sd_fuse/bl2.bin.hardkernel.720k_uboot \
https://raw.githubusercontent.com/hardkernel/u-boot/odroidxu4-v2017.05/sd_fuse/tzsw.bin.hardkernel

chmod a+x sd_fusing.sh

Use U-Boot Files Provided by Fedora (Since They Are More Modern than the U-Boot Hardkernel Provides)

Copy the Fedora U-Boot files into the local dir.

On your workstation

cp /usr/share/uboot/odroid-xu3/u-boot.bin .

Fuse Your SD Card

Finally, run the fusing script to embed the files onto the SD card, passing in the device for your SD card.

On your workstation

Again, be sure to write to the correct device (my setup uses /dev/sdi)

sudo ./sd_fusing.sh /dev/sdi
[...]
Eject /dev/sdi and insert it again.

Do as instructed, re-plug the µSD card.

There are more steps to perform, please scroll past the next subsection on subsequent fusing.

Subsequent Fusing

If you fuse again later (after messing up the µSD card maybe), first, verify that the intended image (U-Boot ≥ armv7-2019.01) is in place. A dnf upgrade on your workstation might have gifted you a newer U-Boot.

on your workstation, as user

cd hardkernel/
rpm -qf /usr/share/uboot/odroid-xu3/u-boot.bin
sha256sum /usr/share/uboot/odroid-xu3/u-boot.bin u-boot.bin

If the sha256sums are not identical, you want to cp /usr/share/uboot/odroid-xu3/u-boot.bin . This will happen when a dnf upgrade of your workstation gifts you an updated U-Boot. If that happens, you definitely want the newer U-Boot.

Do the fusing, on your workstation

sudo ./sd_fusing.sh /dev/sdi
[...]
Eject /dev/sdi and insert it again.

Re-plug the card as instructed.

Mount rootfs

There are 2 changes you need to do to the rootfs (/) on the card, which is partition 3 and should (thanks to `--resizefs) be the largest partition on the SD card.

I was lazy and mounted simply with my KDE Plasma Desktop Device Notifier ;-) but mounting from the commandline is just as valid. Adjust the path according to your mount point.

Disable initial-setup

If you disable initial-setup, then you do not need a serial connection. If you leave initial setup enabled, Fedora ARM will interrupt the initial boot to ask questions interactively. You would need to connect via serial console for that interaction as the HC2 has no graphical output.

So we’ll disable initial-setup by mounting / (3rd partition) and then deleting 2 symlinks

on the workstation, as root (obviously adjust the path to where you mounted, mine’s at /run/media/pcfe/__/)

find /run/media/pcfe/__/ -name "initial-setup.service" -type l -exec /bin/rm -i {} \;

you should get

/bin/rm: remove symbolic link '/run/media/pcfe/__/etc/systemd/system/graphical.target.wants/initial-setup.service'?
/bin/rm: remove symbolic link '/run/media/pcfe/__/etc/systemd/system/multi-user.target.wants/initial-setup.service'?

Obviously, answer y to both questions.

Enable Heartbeat LED

In addition to the kernel parameter rd.driver.pre=ledtrig-heartbeat, I want the module preloaded as soon as possible during boot so that I get early heartbeat LED (as opposed to loading the module only when the board has finished booting).

Installing a kernel update on Fedora will trigger Dracut to generate the initramfs. Dracut must be given instructions to load ledtrig-heartbeat early during boot.

This is achieved by creating the required file while / is mounted anyway from the previousl step.

On the workstation, as root (again, adjust your patch accordingly!)

cat <<EOF >/run/media/pcfe/__/etc/dracut.conf.d/ledtrig-heartbeat.conf
add_drivers+=" ledtrig-heartbeat "
EOF

You may want to verify that you created the file correctly;

[root@karhu ~]# cat /run/media/pcfe/__/etc/dracut.conf.d/ledtrig-heartbeat.conf
add_drivers+=" ledtrig-heartbeat "

Note: Shortly after initial boot, I will dnf upgrade the installed Fedora, so I’m not going to bother running dracut --force on the ODROID for a kernel I’ll boot once and then gets superceded by a newer kernel.

Unmount

Do not forget to unmount

On your workstation, as root

umount /run/media/pcfe/__

Remove µSD Card from Workstation and Insert in ODROID

Not forgetting the previous step of unmouting, remove the SD card from your workstation and transfer it to the ODROID-HC2 (not powered, all LEDs off) once it is firmly seated, apply power to boot.

Expect it to

  1. grab an IP via DHCP
  2. respond to ping within two minutes
  3. respond to ssh -v root@<IP> within five minutes

If these do not happen you will want to connect a serial console. See Fedora on the ODROID-HC2 for details.

Determine Which IP Your ODROID-HC2 Got Via DHCP

Look at the machine that acts as the DHCP server in your network to determin the IP you need to ssh to.

You may want to switch the ODROID to static IP configuration instead of the default DHCP. I do just that with the ansible role linux-system-roles.network in my Site-Specific Setup Playbook.

Log In Via ssh

Log onto your ODROID-HC2 using your ssh key, ideally your privkey has a passphrase and you use ssh-agent.

Expect to be able to ssh within five minutes of booting the ODROOID-HC2. I can generally ssh to mine 90 seconds after applying power.

On you workstation

ssh -l root -v odroid-hc2-00
[ ... ]
debug1: Will attempt key: … agent
[ … ]
[root@odroid-hc2-00 ~]# 

Should you be unable to ssh at this step, then you will definitely need a serial console to debug. See installing Fedora 29 on my ODROID-HC2 for details on the serial console.

Enable Heartbeat LED for Currently Booted Kernel

If you want to use the heartbeat LED to notice board hangs during your set up of the ODROID-HC2, manually load the module for now.

On the ODROID, as root

modprobe ledtrig-heartbeat

We’ll install a kernel update momentarily, there is no real need to rebuild initramfs for the kernel shipped with Fedora-Minimal-armhfp-29-1.2 If you followed the instructions for writing /etc/dracut.conf.d/ledtrig-heartbeat.conf earlier in this post, all newly installed kernels will have LED heartbeat enabled early on in the boot process.

Prepare for Ansible Control as User root

Is described in greater detail in ODROID-HC2 and Ansible

On the ODROID-HC2, as root

date
dnf -y install python libselinux-python
date

Be patient, installing these 2 packages and their dependencies takes up to 8 minutes for me (and rarely under 4). Hence the two date commands around the yum install, they help when I wonder “this feels like it’s taking forever”

Verify Ansible Connectivity

(At this stage there is no ansible user available on the ODROID-HC2 yet. That will be added momentarily.)

Verify that you can reach the ODROID-HC2 with Ansible (obviously adjust to your inventory location and group name)

On the Ansible control node

ansible -i ../inventories/ceph-HouseNet-cluster ceph-arm-nodes -m ping -e ansible_user=root

It should return this

odroid-hc2-03 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
odroid-hc2-01 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
odroid-hc2-04 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
odroid-hc2-00 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
odroid-hc2-02 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

If this fails, you will need to ssh -v (with your key) to the ODROID as root and debug why Ansible can not connect.

Initially Configure Your ODROID-HC2

Do whatever is needed to the ODROID-HC2 in order to be able to use ceph-ansible against the ODROID-HC2.

Note that I have no intention of running ceph-ansible itself on the ODROID-HC2.

This mainly boils down to creating a user that Ansible can connect as and granting that user password-less sudo.

While this initial configuration can be achieved manually over ssh, ceph-ansible needs to talk Ansible to the ODROID-HC2 nodes anyway. So I chose to do this initial config via Ansible too.

Notes:

  • since I installed an ssh public key for the root user on the ODROID-HC2, I ensure my Ansible configuration uses it.

    • I recommend using ssh-agent on your Ansible control node when you run Ansible from a shell.
    • I recommend a credentials store if you use Ansible Tower.
  • as the instructions above create only a root user, you may want to start with an Ansible play that connects as root and sets up an ansible user.

Running My Initial Setup Playbook

See ODROID-HC2 and Ansible for details. The main use of that is to create an ansible user on the ODROID-HC2.

You probably want to use your own playbooks and roles.

This subsection mainly serves as a note to myself.

On my Ansible control node (Fedora x86_64 workstation)

ansible-playbook -i ../inventories/ceph-HouseNet-cluster arm-fedora-initial-setup.yml

Perform Site-Specific Setup

After initial setup, you may have some site-specific playbooks and roles you want to run before using ceph-ansible. If that is the case, do so now.

One of the things worth doing is to install one or more langpacks-… RPMs, this stops dnf from complaining about Failed to set locale, defaulting to C.

The warning will not impede your ability to use the node, it’s just an annoyance when working interactively with dnf. Plus, I like to have language packs for those that I speak.

Running My Site-Specific Setup Playbook

See My Site-Specific Setup Playbook for a log output and a description.

This subsection mainly serves as a note to myself.

On my Ansible control node (Fedora x86_64 workstation)

ansible-playbook -i ../inventories/ceph-HouseNet-cluster odroid-general-setup.yml
ansible-playbook odroid-general-setup.yml

Reboot if You Installed a New kernel or glibc

If, like mine, your initial setup includes applying all errata to the OS (highly recommended) then reboot if you got a new kernel or glibc.

If you did not install a new kernel, remember to dracut --force on the ODROID-HC2 to enable your heartbeat LED functionality at the next boot into the old kernel.

I will (amongst others) have gotten a new kernel (4.20.3-200.fc29.armv7hl as of 2019-01-28) and fully intend to use the newest one available.

On the ODROID-HC2, as root

systemctl reboot

Note on Reboots

Sometimes, when I warm boot an ODROID-HC2 running Fedora 29, the heartbeat LED blinks ‘alive’ but the host is not reachable over ssh or even reacting to ping.

Power cycling the node in that case has always allowed the node to come up when this happened.

If you worry about this happening to you, consider simply doing systemctl poweroff followed by a manual power cycle (wait a few 10s of seconds after heartbeat LED turns off, unplug, wait for all ODROID LEDs to turn off, then apply power again) instead of systemctl reboot.

Analysis of Hung Reboots

Turns out USB3 does not always get initialised properly on warm boot. https://forum.odroid.com/viewtopic.php?f=146&t=29188 has a patch that does not seem to be upstream yet.

When this happens, one can see the following on serial console:

[*     ] A start job is running for udev Wai…vice Initialization (8s / 3min 1s)[   30.693007] xhci-hcd xhci-hcd.1.auto: Timeout while waiting for setup device command
[...]
         Starting Hostname Service...
[   42.336595] fbcon: Taking over console

Fedora 29 (Twenty Nine)
Kernel 4.20.6-200.fc29.armv7hl on an armv7l (ttySAC2)

localhost login:

Workaround for Unreliable Reboots

A soft reboot or the watchdog triggering a reboot does not fix the issue, a cold boot is required. Note that I have configured the hardware watchdog via My Site-Specific Setup Playbook,

If this happened and one has serial consoles wired up, then one can cleanly issue systemctl poweroff via the serial console, followed by a manual power cycle (wait a few 10s of seconds after heartbeat LED turns off, unplug, wait for all ODROID-HC2 LEDs to turn off, then apply power again).

But without a serial console, you have to power cycle the ODROID-HC2 when it got into this state.

I’ve resorted to not using systemctl reboot when I need to reboot an ODROID-HC2. Instead I issue a systemctl poweroff, wait for powerdown to complete (heartbeat LED off, then wait for the green LED on the board to also turn off and remain off) followed by a power cycle (unplug, wait for all ODROID-HC2 LEDs to turn off, then apply power again).

That way I am sure the board will boot fine.

Check Kernel Version After Reboot

You heartbeat LED should now function without further manual intervention. Expect it to start blinking within 30 seconds of applying power.

After a two or three minutes you should be able to ssh in again to verify the new kernel got picked at boot time.

On the workstation

pcfe@karhu ~ $ ssh -l root odroid-hc2-00 uname -r
4.20.3-200.fc29.armv7hl

HDD go to sleep a tad early

c.f. Automatic Spin-Down of SATA Drive thread on the ODROID Forum.

Setting via hdparm -S 242 /dev/sda does not help (drives still spin down within a few minutes)

Workaround for HDD Spin Down Timer

Update firmware and write spin-down timer to JMS578 Firmware I set a timeout of 60 minutes, adjust as fit for your purposes.

pcfe@karhu tmp $ ssh root@odroid-hc2-01
Last login: Tue Mar 12 11:28:18 2019 from 192.168.50.35
No Sockets found in /run/screen/S-root.

[root@odroid-hc2-01 ~]# cd /tmp/JMS578FwUpdater/
[root@odroid-hc2-01 JMS578FwUpdater]# ll
total 4700
-rwxr-xr-x. 1 root root 4130828 Apr 19  2018 JMS578FwUpdate
-rwxr-xr-x. 1 root root  519032 Nov  1  2017 JMS578FwUpdate.v1.00
-rwxr-xr-x. 1 root root   50688 Mär  7 10:33 JMS578-Hardkenel-Release-v173.01.00.02-20190306.bin
-rw-r--r--. 1 root root   50688 Dez  5  2017 JMS578_Hardkernel_v173.01.00.01.bin
-rwxr-xr-x. 1 root root   50688 Nov  1  2017 JMS578-v0.1.0.5.bin
[root@odroid-hc2-01 JMS578FwUpdater]# ./JMS578FwUpdate -d /dev/sda -v
Bridge Firmware Version: v173.1.0.1

[root@odroid-hc2-01 JMS578FwUpdater]# ./JMS578FwUpdate -d /dev/sda -f ./JMS578-Hardkenel-Release-v173.01.00.02-20190306.bin -b ./backup.bin -t 60
Update Firmware file name: ./JMS578-Hardkenel-Release-v173.01.00.02-20190306.bin
Backup Firmware file name: ./backup.bin
Auto spin-down timer: 60 min.
Backup the ROM code sucessfully.
Programming & Compare Success!!

[root@odroid-hc2-01 JMS578FwUpdater]# systemctl poweroff
[root@odroid-hc2-01 JMS578FwUpdater]# Connection to odroid-hc2-01 closed by remote host.
Connection to odroid-hc2-01 closed.

Possible Future Improvements

Fixups Before Running ceph-ansible

ansible-playbook -i ../inventories/ceph-HouseNet-cluster ceph-prepare-arm.yml

While ceph-ansible 3.2-stable works just fine against RHEL7 targets, for my Fedora OSDs I had to implement the following work-arounds;

  • MUST disable firewall adjustments in all.yml with configure_firewall: False
  • MUST open firewall ports on OSDs before running site.yml
    • ansible-playbook -i ../inventories/ceph-HouseNet-cluster ceph-prepare-arm.yml
  • MUST unset LC_MESSAGES LANG LC_PAPER LC_MEASUREMENT LC_NUMERIC LC_MONETARY LC_TIME LC_DATE LC_DATE_TIME
  • MUST remember to check on chronyd
  • MUST zap disks and reboot before trying a fresh install!
    • ceph-disk zap or ceph-volume zap
    • partprobe was not sufficient, so do remember to reboot in order to hoover in new partition table
  • MUST obtain prometheus-node_exporter for Fedora

    • this works

      mock -r fedora-29-armhfp /var/tmp/golang-github-prometheus-node_exporter-0.15.2-2.el7cp.src.rpm
      • this does not work

        cd /home/pcfe/work/git/github.com/golang-github-prometheus-node_exporter
        # remote is git@github.com:pcfe/golang-github-prometheus-node_exporter.git
        # WIP in branch pcfe-add-armv7hl
        mock --buildsrpm --spec golang-github-prometheus-node_exporter.spec --sources $(pwd) --root fedora-29-armhfp
        cp /var/lib/mock/fedora-29-armhfp/result/golang-github-prometheus-node_exporter-0.17.0-9.fc29.src.rpm .
        mock -r fedora-29-armhfp golang-github-prometheus-node_exporter-0.17.0-9.fc29.src.rpm
  • MUST override dnf, see RFE: please support dnf, when running cephmetrics playbook by using [ansible@ceph-ansible cephmetrics-ansible]$ ansible-playbook playbook.yml -e ansible_pkg_mgr=yum

  • MUST set interpreter to python3 if running the cephmetrics play against a F29 node

My Ceph-Specific Playbook

Before running ceph-ansible on my control node (a RHEL7 x86_64 VM), I do some preparations from my Fedora x86_64 control node. This is mainly due to Python3 seems to break TASK [ceph-mon : create monitor initial keyring] #3565

But there are other things that need working around for these unsupported by ceph-ansible` Fedora ARM nodes.

# sets up a Fedora 29 ARM minimal install to be ready for Ceph
# this is for the ceph-arm-nodes in my own test cluster, not for production!
- hosts:
  - ceph-arm-nodes
  become: yes

  tasks:
    # steps taken from install guide and adjusted for ceph-ansible 3.2 with specifics for ARM nodes running Fedora instead of RHEL
    # 2.8. Configuring a Firewall for Red Hat Ceph Storage (optional)
    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/installation_guide_for_red_hat_enterprise_linux/requirements-for-installing-rhcs#configuring-a-firewall-for-red-hat-ceph-storage
    ### since this is for demo setups, enable ports for all possible Ceph services
    ### on prod this should not be done
    #
    # as of RHCS 3.2 The ceph-ansible utility now configures the firewalld service by default
    # but this does NOT work on Fedora 29 (https://github.com/ceph/ceph-ansible/issues/3565)
    #
    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3.2/html/release_notes/enhancements#the_literal_ceph_ansible_literal_utility
    - name:         "FIREWALLD | ensure firewall is enabled and running"
      service:
        name:       firewalld
        enabled:    true
        state:      started
    # ceph.xml opens 6800-7300 tcp. ceph-mon.xml 3300 tcp and 6789 tcp
    - name: "FIREWALLD | ensure firewalld permits ceph and ceph-mon services in zone public"
      firewalld:
        service:    "{{ item }}"
        zone:       public
        permanent:  True
        state:      enabled
        immediate:  True
      with_items:
        - ceph
        - ceph-mon
    - name:         "FIREWALLD | The Ceph Object Gateway nodes use port 7480 by default. Since we use the SSL/TLS service, open port 443."
      firewalld:
        port:       443/tcp
        permanent:  true
        zone:       public
        state:      enabled
        immediate:  true
    # prometheur node exporter not present in Fedora repos.
    # for now, just do:
    #     yumdownloader --source on a RHEL7 node
    #     mock -r fedora-29-armhfp /var/tmp/golang-github-prometheus-node_exporter-0.15.2-2.el7cp.src.rpm
    #
    # cephmetrics-ansible sets this fine if i override the ansible interpreter by a group_var
    - name:         "FIREWALLD | Prometheus node-exporter daemon uses 9100"
      firewalld:
        port:       9100/tcp
        permanent:  true
        zone:       public
        state:      enabled
        immediate:  true
    - name:         "FIREWALLD | MGR nodes need to allow gathering of Ceph data (9283) for cephmetrics"
      # cephmetrics-ansible sets this fine if i override the ansible interpreter to python3 for the Fedora ARM 29 nodes via a group_var
      firewalld:
        port:       9283/tcp
        permanent:  true
        zone:       public
        state:      enabled
        immediate:  true
    - name:         "FIREWALLD | iSCSI gateway nodes need to allow gathering of Ceph iSCSI gateway data (9287) for cephmetrics"
      firewalld:
        port:       9287/tcp
        permanent:  true
        zone:       public
        state:      enabled
        immediate:  true

    # 3.2. Installing a Red Hat Ceph Storage Cluster
    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/installation_guide_for_red_hat_enterprise_linux/deploying-red-hat-ceph-storage#installing-a-red-hat-ceph-storage-cluster    - name: Nextcloud | ensure packages are installed
    #
    # is done from ceph-ansible-rhel7.internal.pcfe.net after ceph-prepare-ceph-ansible-control-VM.yml was run

    # MONITORING
    # strange, collectd-ceph was not installed when I did `ansible-playbook playbook.yml -e ansible_pkg_mgr=yum`
    # it did work at the beginning though, guess I must have installed that package by hand
    # anyway, add pkg here

    - name: "PACKAGE | ensure collectd-ceph package is installed"
      package:
        name:         collectd-ceph
        state:        present
ansible-playbook -i ../inventories/ceph-HouseNet-cluster -l odroid-hc2-00 ceph-prepare-arm.yml | tee ../../hugo-blog.pcfe.net/static/code/ceph-prepare-arm.log

PLAY [ceph-arm-nodes] ***********************************************************************************************

TASK [Gathering Facts] **********************************************************************************************
ok: [odroid-hc2-00]

TASK [FIREWALLD | ensure firewall is enabled and running] ***********************************************************
ok: [odroid-hc2-00]

TASK [FIREWALLD | ensure firewalld permits ceph and ceph-mon services in zone public] *******************************
ok: [odroid-hc2-00] => (item=ceph)
ok: [odroid-hc2-00] => (item=ceph-mon)

TASK [FIREWALLD | The Ceph Object Gateway nodes use port 7480 by default. Since we use the SSL/TLS service, open port 443.] ***
ok: [odroid-hc2-00]

TASK [FIREWALLD | Prometheus node-exporter daemon uses 9100] ********************************************************
ok: [odroid-hc2-00]

TASK [FIREWALLD | MGR nodes need to allow gathering of Ceph data (9283) for cephmetrics] ****************************
ok: [odroid-hc2-00]

TASK [FIREWALLD | iSCSI gateway nodes need to allow gathering of Ceph iSCSI gateway data (9287) for cephmetrics] ****
ok: [odroid-hc2-00]

TASK [PACKAGE | ensure collectd-ceph package is installed] **********************************************************
ok: [odroid-hc2-00]

PLAY RECAP **********************************************************************************************************
odroid-hc2-00              : ok=8    changed=0    unreachable=0    failed=0   


Performance Baseline

I gathered a performance baseline as per 7. Benchmarking Performance of the RHCS 3 Administration Guide.

Write Test

[root@odroid-hc2-00 ~]# rados bench -p testbench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_odroid-hc2-00.internal.pcfe.n_12521
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        16         0         0         0           -           0
    2      16        24         8   15.9955        16     1.84685     1.56863
    3      16        36        20    26.659        48     1.23306     1.79863
    4      16        45        29   28.9916        36     1.58812     1.71253
    5      16        58        42   33.5897        52     1.25309     1.67093
    6      16        67        51   33.9896        36     1.61167     1.60464
    7      16        79        63   35.9896        48     1.30261     1.54448
    8      16        93        77   38.4891        56     1.43842     1.53261
    9      16       103        87   38.6561        40     1.26403     1.52164
   10      16       110        94     37.59        28     2.10354     1.53743
Total time run:         10.857355
Total writes made:      111
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     40.8939
Stddev Bandwidth:       17.3845
Max bandwidth (MB/sec): 56
Min bandwidth (MB/sec): 0
Average IOPS:           10
Stddev IOPS:            4
Max IOPS:               14
Min IOPS:               0
Average Latency(s):     1.56195
Stddev Latency(s):      0.438288
Max latency(s):         2.90667
Min latency(s):         0.743802

Sequential Read

[root@odroid-hc2-00 ~]# rados bench -p testbench 10 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15        32        17   67.6993        68   0.0895419    0.429525
    2      15        48        33   65.6877        64     1.08895    0.707737
    3      15        72        57   75.2167        96     1.73453    0.687807
    4      16        95        79   78.3703        88     1.20438    0.700034
    5      16       111        95   75.5099        64     1.19779     0.71015
Total time run:       5.683982
Total reads made:     111
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   78.1142
Average IOPS:         19
Stddev IOPS:          3 
Max IOPS:             24
Min IOPS:             16
Average Latency(s):   0.776547
Max latency(s):       2.19314
Min latency(s):       0.0628356

Random Read

[root@odroid-hc2-00 ~]# rados bench -p testbench 10 rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15        41        26   103.949       104   0.0142945    0.356068
    2      16        65        49   97.9598        92    0.489051    0.498968
    3      16        86        70   93.3013        84   0.0239526    0.534059
    4      16       108        92   91.9708        88     1.37579    0.605005
    5      16       127       111   88.7737        76    0.739931    0.629072
    6      16       151       135   89.9747        96     1.07922    0.647909
    7      16       174       158    90.261        92    0.711373    0.636545
    8      16       193       177   88.4766        76     2.09042    0.648356
    9      16       212       196   87.0876        76      2.2776    0.652152
   10      16       239       223   89.1764       108     2.49155    0.654033
Total time run:       10.955053
Total reads made:     240
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   87.6308
Average IOPS:         21
Stddev IOPS:          2 
Max IOPS:             27
Min IOPS:             19
Average Latency(s):   0.712775
Max latency(s):       2.49155
Min latency(s):       0.0137742

Recycling an old tablet to display cephmetrics

FIXME: make new picture, with more light, on a sunny day.