Installing cephadm on CentOS Stream 9

Table of Contents

This is my braindump on installing cephadm on CentOS Stream 9.

While I use a QNAP TS-473A for my tests plus 2 VMs, the below applies to any machine or VM running CentOS Stream 9 FWIW: My installation and initial configuration is described in the post QNAP TS-473A with CentOS Stream 9.

Install CentOS Stream 9

Installation and initial configuration is described in a separate post.

It really boiled down to installing CentOS Stream 9 with the same method and post-install setup that I installed Fedora Server 35 with and then running my Ansible Playbooks.

I expect the installation method to be just as painless with Red Hat Enterprise Linux 9 once it is generally available.

[root@ts-473a-01 ~]# cat /etc/os-release 
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

Install cephadm

As per https://docs.ceph.com/en/pacific/cephadm/install/#distribution-specific-installations.

The CentOS Storage Special Interest Group has repos which you can install straight from CentOS Stream 9;

[root@ts-473a-01 ~]# dnf search release-ceph
Last metadata expiration check: []
================================== Name Matched: release-ceph ===================================
centos-release-ceph-pacific.noarch : Ceph Pacific packages from the CentOS Storage SIG repository
centos-release-ceph-quincy.noarch : Ceph Quincy packages from the CentOS Storage SIG repository

Pick the Ceph release you want to use. For me, today, that is Ceph Pacific (version 16).

[root@ts-473a-01 ~]# dnf install --assumeyes centos-release-ceph-pacific.noarch
Click to show the output from the dnf install command.
Last metadata expiration check: 1:58:43 ago on Sat 12 Feb 2022 10:30:42 PM CET.
Dependencies resolved.
==================================================================================================================================================
 Package                                          Architecture              Version                        Repository                        Size
==================================================================================================================================================
Installing:
 centos-release-ceph-pacific                      noarch                    1.0-2.el9s                     extras-common                    7.4 k
Installing dependencies:
 centos-release-storage-common                    noarch                    2-4.el9s                       extras-common                    8.2 k

Transaction Summary
==================================================================================================================================================
Install  2 Packages

Total download size: 16 k
Installed size: 2.3 k
Downloading Packages:
(1/2): centos-release-ceph-pacific-1.0-2.el9s.noarch.rpm                                                           25 kB/s | 7.4 kB     00:00    
(2/2): centos-release-storage-common-2-4.el9s.noarch.rpm                                                           27 kB/s | 8.2 kB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                              13 kB/s |  16 kB     00:01     
CentOS Stream 9 - Extras packages                                                                                 2.1 MB/s | 2.1 kB     00:00    
Importing GPG key 0x1D997668:
 Userid     : "CentOS Extras SIG (https://wiki.centos.org/SpecialInterestGroup) <security@centos.org>"
 Fingerprint: 363F C097 2F64 B699 AED3 968E 1FF6 A217 1D99 7668
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Extras
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                          1/1 
  Installing       : centos-release-storage-common-2-4.el9s.noarch                                                                            1/2 
  Installing       : centos-release-ceph-pacific-1.0-2.el9s.noarch                                                                            2/2 
  Verifying        : centos-release-ceph-pacific-1.0-2.el9s.noarch                                                                            1/2 
  Verifying        : centos-release-storage-common-2-4.el9s.noarch                                                                            2/2 

Installed:
  centos-release-ceph-pacific-1.0-2.el9s.noarch                           centos-release-storage-common-2-4.el9s.noarch                          

Complete!

Once the relevant Ceph repo file is installed via the above, then you can simply install cephadm.

[root@ts-473a-01 ~]# dnf install --assumeyes cephadm
Click to show the output from the dnf install command.
Last metadata expiration check: 0:00:15 ago on Sun 13 Feb 2022 12:30:06 AM CET.
Dependencies resolved.
==================================================================================================================================================
 Package                       Architecture                 Version                               Repository                                 Size
==================================================================================================================================================
Installing:
 cephadm                       noarch                       2:16.2.7-1.el9s                       centos-ceph-pacific                        75 k

Transaction Summary
==================================================================================================================================================
Install  1 Package

Total download size: 75 k
Installed size: 332 k
Downloading Packages:
cephadm-16.2.7-1.el9s.noarch.rpm                                                                                  432 kB/s |  75 kB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                              76 kB/s |  75 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                          1/1 
  Running scriptlet: cephadm-2:16.2.7-1.el9s.noarch                                                                                           1/1 
  Installing       : cephadm-2:16.2.7-1.el9s.noarch                                                                                           1/1 
  Verifying        : cephadm-2:16.2.7-1.el9s.noarch                                                                                           1/1 

Installed:
  cephadm-2:16.2.7-1.el9s.noarch                                                                                                                  

Complete!

Ensure all Hosts use Short Hostname

Since my initial test with fqdn-hostname were not so nice, added hosts gave me errors (not on public network) when trying to deploy daemons, I went fot short hostnames this time.

If you care, click to show the Ansible task I use to do this.
    # linux-system-roles.network sets static network config (from host_vars)
    # but I want the static hostname nailed down too
    # note that cephadm prefers a short hostname (`ansible_hostname` in my inventory), not the long one (`ansible_fqdn`)
    # unless given `--allow-fqdn-hostname`
    # see https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html/installation_guide/red-hat-ceph-storage-installation#recommended-cephadm-bootstrap-command-options_install
    # and https://docs.ceph.com/en/latest/cephadm/host-management/#fully-qualified-domain-names-vs-bare-host-names
    - name: "set hostname"
      hostname:
        name:          "{{ ansible_hostname }}"
        use:           systemd

Quick Check

I simply run hostname (should show short name) and hostname -f (should show fqdn-hostname) on ech Ceph Pacific host.

pcfe@workstation ~ $ for i in ts-473a-01 pacific-pve pacific-epyc ; \
  do echo ; \
  ssh -l root ${i} hostname ; \
  ssh -l root ${i} hostname -f ; \
  done

ts-473a-01
ts-473a-01.internal.pcfe.net

pacific-pve
pacific-pve.internal.pcfe.net

pacific-epyc
pacific-epyc.internal.pcfe.net

Bootstrap Cluster

I specify --cluster-network 192.168.30.0/24 because I have a separate network card for the cluster network, 10 GbE and with an MTU of 9000.

I specify --single-host-defaults because I just want to play with cephadm and Ceph Pacific, my other nodes (4x TerraMaster F5-422) currently run Ceph Nautilus, specifically Red Hat Ceph Storage 4.

I do NOT specify --allow-fqdn-hostname even though I read section 3.8.1. Recommended cephadm bootstrap command options in the RHCS5 Installation Guide and section Fully qualified domain names vs bare host names of the latest (not Pacific) Ceph Docs.

I’ll see if these choices are wise, this is a playground for now.

Click to show hostname and IP checking done before bootstrapping.
[root@ts-473a-01 ~]# hostname
ts-473a-01
[root@ts-473a-01 ~]# hostname -f
ts-473a-01.internal.pcfe.net
[root@ts-473a-01 ~]# hostname -s
ts-473a-01
[root@ts-473a-01 ~]# for i in internal storage ceph ; do host ts-473a-01.${i}.pcfe.net;done
ts-473a-01.internal.pcfe.net has address 192.168.50.185
ts-473a-01.storage.pcfe.net has address 192.168.40.185
ts-473a-01.ceph.pcfe.net has address 192.168.30.185
[root@ts-473a-01 ~]# for i in internal storage ceph ; do host pacific-epyc.${i}.pcfe.net;done
pacific-epyc.internal.pcfe.net has address 192.168.50.61
pacific-epyc.storage.pcfe.net has address 192.168.40.61
Host pacific-epyc.ceph.pcfe.net not found: 3(NXDOMAIN)
[root@ts-473a-01 ~]# for i in internal storage ceph ; do host pacific-pve.${i}.pcfe.net;done
pacific-pve.internal.pcfe.net has address 192.168.50.60
pacific-pve.storage.pcfe.net has address 192.168.40.60
Host pacific-pve.ceph.pcfe.net not found: 3(NXDOMAIN)
[root@ts-473a-01 ~]# ip r s
default via 192.168.50.254 dev enp6s0 proto static metric 102 
192.168.30.0/24 dev enp2s0f0np0 proto kernel scope link src 192.168.30.185 metric 100 
192.168.40.0/24 dev enp5s0 proto kernel scope link src 192.168.40.185 metric 101 
192.168.50.0/24 dev enp6s0 proto kernel scope link src 192.168.50.185 metric 102 
[root@ts-473a-01 ~]# nmcli connection show 
NAME           UUID                                  TYPE      DEVICE      
System 2.5G_1  56897b97-fc4b-4fac-a7ee-87d3268a0ed4  ethernet  enp6s0      
System 10G_1   de94a3ff-71a6-4bfe-88d5-a5832ea7a635  ethernet  enp2s0f0np0 
System 2.5G_2  b176f25b-8b27-4079-9f2d-5ad31f6def9d  ethernet  enp5s0      
System 10G_2   472e11cc-e20f-483d-bac2-af2bfa546bbf  ethernet  --  
[root@ts-473a-01 ~]# cephadm bootstrap \
  --mon-ip 192.168.40.185 \
  --cluster-network 192.168.30.0/24 \
  --single-host-defaults
Click to show the output of cephadm bootstrap.
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 3.4.5 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 2570b1c0-8c5f-11ec-88d0-245ebe4b8fc0
Verifying IP 192.168.40.185 port 3300 ...
Verifying IP 192.168.40.185 port 6789 ...
Mon IP `192.168.40.185` is in CIDR network `192.168.40.0/24`
Adjusting default settings to suit single-host cluster...
Pulling container image quay.io/ceph/ceph:v16...
Ceph version: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
firewalld ready
Enabling firewalld service ceph-mon in current zone...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting mon public_network to 192.168.40.0/24
Setting cluster_network to 192.168.30.0/24
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 9283 ...
firewalld ready
Enabling firewalld service ceph in current zone...
firewalld ready
Enabling firewalld port 9283/tcp in current zone...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host ts-473a-01...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for mgr epoch 9...
mgr epoch 9 is available
Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
firewalld ready
Enabling firewalld port 8443/tcp in current zone...
Ceph Dashboard is now available at:

             URL: https://ts-473a-01.internal.pcfe.net:8443/
            User: admin
        Password: [REDACTED]

Enabling client.admin keyring and conf on hosts with "admin" label
You can access the Ceph CLI with:

        sudo /usr/sbin/cephadm shell --fsid 2570b1c0-8c5f-11ec-88d0-245ebe4b8fc0 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

Please consider enabling telemetry to help improve Ceph:

        ceph telemetry on

For more information see:

        https://docs.ceph.com/docs/pacific/mgr/telemetry/

Bootstrap complete.

After bootstrap completed, I pointed FireFox (on my workstation) at the node, changed my password (as reasonably enforced by the webUI) and logged in.

No screenshot this time, look at my near identical post but on Fedora Server if you crave a screenshot :-)

Cephadm Shell

Just a quick smoke test if it is functional, as you can see I have not yet set up OSDs, pools etc.

While I could just use cephadm shell, I used the full command provided by the bootstrap output.

[root@ts-473a-01 ~]# /usr/sbin/cephadm shell --fsid 2570b1c0-8c5f-11ec-88d0-245ebe4b8fc0 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
Using recent ceph image quay.io/ceph/ceph@sha256:a39107f8d3daab4d756eabd6ee1630d1bc7f31eaa76fff41a77fa32d0b903061
[ceph: root@ts-473a-01 /]# ceph -s
  cluster:
    id:     2570b1c0-8c5f-11ec-88d0-245ebe4b8fc0
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 2
 
  services:
    mon: 1 daemons, quorum ts-473a-01 (age 8m)
    mgr: ts-473a-01.dvkmdy(active, since 5m), standbys: ts-473a-01.pypagy
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     
 
[ceph: root@ts-473a-01 /]# exit
exit
[root@ts-473a-01 ~]# 

That’s all for this post.