Installing cephadm on CentOS Stream 9
Table of Contents
This is my braindump on installing cephadm on CentOS Stream 9.
While I use a QNAP TS-473A for my tests plus 2 VMs, the below applies to any machine or VM running CentOS Stream 9 FWIW: My installation and initial configuration is described in the post QNAP TS-473A with CentOS Stream 9.
Install CentOS Stream 9
Installation and initial configuration is described in a separate post.
It really boiled down to installing CentOS Stream 9 with the same method and post-install setup that I installed Fedora Server 35 with and then running my Ansible Playbooks.
I expect the installation method to be just as painless with Red Hat Enterprise Linux 9 once it is generally available.
[root@ts-473a-01 ~]# cat /etc/os-release
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
Install cephadm
As per https://docs.ceph.com/en/pacific/cephadm/install/#distribution-specific-installations.
The CentOS Storage Special Interest Group has repos which you can install straight from CentOS Stream 9;
[root@ts-473a-01 ~]# dnf search release-ceph
Last metadata expiration check: […]
================================== Name Matched: release-ceph ===================================
centos-release-ceph-pacific.noarch : Ceph Pacific packages from the CentOS Storage SIG repository
centos-release-ceph-quincy.noarch : Ceph Quincy packages from the CentOS Storage SIG repository
Pick the Ceph release you want to use. For me, today, that is Ceph Pacific (version 16).
[root@ts-473a-01 ~]# dnf install --assumeyes centos-release-ceph-pacific.noarch
Click to show the output from the dnf install command.
Last metadata expiration check: 1:58:43 ago on Sat 12 Feb 2022 10:30:42 PM CET.
Dependencies resolved.
==================================================================================================================================================
Package Architecture Version Repository Size
==================================================================================================================================================
Installing:
centos-release-ceph-pacific noarch 1.0-2.el9s extras-common 7.4 k
Installing dependencies:
centos-release-storage-common noarch 2-4.el9s extras-common 8.2 k
Transaction Summary
==================================================================================================================================================
Install 2 Packages
Total download size: 16 k
Installed size: 2.3 k
Downloading Packages:
(1/2): centos-release-ceph-pacific-1.0-2.el9s.noarch.rpm 25 kB/s | 7.4 kB 00:00
(2/2): centos-release-storage-common-2-4.el9s.noarch.rpm 27 kB/s | 8.2 kB 00:00
--------------------------------------------------------------------------------------------------------------------------------------------------
Total 13 kB/s | 16 kB 00:01
CentOS Stream 9 - Extras packages 2.1 MB/s | 2.1 kB 00:00
Importing GPG key 0x1D997668:
Userid : "CentOS Extras SIG (https://wiki.centos.org/SpecialInterestGroup) <security@centos.org>"
Fingerprint: 363F C097 2F64 B699 AED3 968E 1FF6 A217 1D99 7668
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Extras
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : centos-release-storage-common-2-4.el9s.noarch 1/2
Installing : centos-release-ceph-pacific-1.0-2.el9s.noarch 2/2
Verifying : centos-release-ceph-pacific-1.0-2.el9s.noarch 1/2
Verifying : centos-release-storage-common-2-4.el9s.noarch 2/2
Installed:
centos-release-ceph-pacific-1.0-2.el9s.noarch centos-release-storage-common-2-4.el9s.noarch
Complete!
Once the relevant Ceph repo file is installed via the above, then you can simply install cephadm.
[root@ts-473a-01 ~]# dnf install --assumeyes cephadm
Click to show the output from the dnf install command.
Last metadata expiration check: 0:00:15 ago on Sun 13 Feb 2022 12:30:06 AM CET.
Dependencies resolved.
==================================================================================================================================================
Package Architecture Version Repository Size
==================================================================================================================================================
Installing:
cephadm noarch 2:16.2.7-1.el9s centos-ceph-pacific 75 k
Transaction Summary
==================================================================================================================================================
Install 1 Package
Total download size: 75 k
Installed size: 332 k
Downloading Packages:
cephadm-16.2.7-1.el9s.noarch.rpm 432 kB/s | 75 kB 00:00
--------------------------------------------------------------------------------------------------------------------------------------------------
Total 76 kB/s | 75 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: cephadm-2:16.2.7-1.el9s.noarch 1/1
Installing : cephadm-2:16.2.7-1.el9s.noarch 1/1
Verifying : cephadm-2:16.2.7-1.el9s.noarch 1/1
Installed:
cephadm-2:16.2.7-1.el9s.noarch
Complete!
Ensure all Hosts use Short Hostname
Since my initial test with fqdn-hostname were not so nice, added hosts gave me errors (not on public network) when trying to deploy daemons, I went fot short hostnames this time.
If you care, click to show the Ansible task I use to do this.
# linux-system-roles.network sets static network config (from host_vars)
# but I want the static hostname nailed down too
# note that cephadm prefers a short hostname (`ansible_hostname` in my inventory), not the long one (`ansible_fqdn`)
# unless given `--allow-fqdn-hostname`
# see https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html/installation_guide/red-hat-ceph-storage-installation#recommended-cephadm-bootstrap-command-options_install
# and https://docs.ceph.com/en/latest/cephadm/host-management/#fully-qualified-domain-names-vs-bare-host-names
- name: "set hostname"
hostname:
name: "{{ ansible_hostname }}"
use: systemd
Quick Check
I simply run hostname (should show short name) and hostname -f (should show fqdn-hostname) on ech Ceph Pacific host.
pcfe@workstation ~ $ for i in ts-473a-01 pacific-pve pacific-epyc ; \
do echo ; \
ssh -l root ${i} hostname ; \
ssh -l root ${i} hostname -f ; \
done
ts-473a-01
ts-473a-01.internal.pcfe.net
pacific-pve
pacific-pve.internal.pcfe.net
pacific-epyc
pacific-epyc.internal.pcfe.net
Bootstrap Cluster
I specify --cluster-network 192.168.30.0/24 because I have a separate network card for the cluster network,
10 GbE and with an MTU of 9000.
I specify --single-host-defaults because I just want to play with cephadm and Ceph Pacific, my other nodes (4x TerraMaster F5-422)
currently run Ceph Nautilus, specifically Red Hat Ceph Storage 4.
I do NOT specify --allow-fqdn-hostname even though I read section 3.8.1. Recommended cephadm bootstrap command options in the RHCS5 Installation Guide
and section Fully qualified domain names vs bare host names of the latest (not Pacific) Ceph Docs.
I’ll see if these choices are wise, this is a playground for now.
Click to show hostname and IP checking done before bootstrapping.
[root@ts-473a-01 ~]# hostname
ts-473a-01
[root@ts-473a-01 ~]# hostname -f
ts-473a-01.internal.pcfe.net
[root@ts-473a-01 ~]# hostname -s
ts-473a-01
[root@ts-473a-01 ~]# for i in internal storage ceph ; do host ts-473a-01.${i}.pcfe.net;done
ts-473a-01.internal.pcfe.net has address 192.168.50.185
ts-473a-01.storage.pcfe.net has address 192.168.40.185
ts-473a-01.ceph.pcfe.net has address 192.168.30.185
[root@ts-473a-01 ~]# for i in internal storage ceph ; do host pacific-epyc.${i}.pcfe.net;done
pacific-epyc.internal.pcfe.net has address 192.168.50.61
pacific-epyc.storage.pcfe.net has address 192.168.40.61
Host pacific-epyc.ceph.pcfe.net not found: 3(NXDOMAIN)
[root@ts-473a-01 ~]# for i in internal storage ceph ; do host pacific-pve.${i}.pcfe.net;done
pacific-pve.internal.pcfe.net has address 192.168.50.60
pacific-pve.storage.pcfe.net has address 192.168.40.60
Host pacific-pve.ceph.pcfe.net not found: 3(NXDOMAIN)
[root@ts-473a-01 ~]# ip r s
default via 192.168.50.254 dev enp6s0 proto static metric 102
192.168.30.0/24 dev enp2s0f0np0 proto kernel scope link src 192.168.30.185 metric 100
192.168.40.0/24 dev enp5s0 proto kernel scope link src 192.168.40.185 metric 101
192.168.50.0/24 dev enp6s0 proto kernel scope link src 192.168.50.185 metric 102
[root@ts-473a-01 ~]# nmcli connection show
NAME UUID TYPE DEVICE
System 2.5G_1 56897b97-fc4b-4fac-a7ee-87d3268a0ed4 ethernet enp6s0
System 10G_1 de94a3ff-71a6-4bfe-88d5-a5832ea7a635 ethernet enp2s0f0np0
System 2.5G_2 b176f25b-8b27-4079-9f2d-5ad31f6def9d ethernet enp5s0
System 10G_2 472e11cc-e20f-483d-bac2-af2bfa546bbf ethernet --
[root@ts-473a-01 ~]# cephadm bootstrap \
--mon-ip 192.168.40.185 \
--cluster-network 192.168.30.0/24 \
--single-host-defaults
Click to show the output of cephadm bootstrap.
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 3.4.5 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 2570b1c0-8c5f-11ec-88d0-245ebe4b8fc0
Verifying IP 192.168.40.185 port 3300 ...
Verifying IP 192.168.40.185 port 6789 ...
Mon IP `192.168.40.185` is in CIDR network `192.168.40.0/24`
Adjusting default settings to suit single-host cluster...
Pulling container image quay.io/ceph/ceph:v16...
Ceph version: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
firewalld ready
Enabling firewalld service ceph-mon in current zone...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting mon public_network to 192.168.40.0/24
Setting cluster_network to 192.168.30.0/24
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 9283 ...
firewalld ready
Enabling firewalld service ceph in current zone...
firewalld ready
Enabling firewalld port 9283/tcp in current zone...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host ts-473a-01...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for mgr epoch 9...
mgr epoch 9 is available
Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
firewalld ready
Enabling firewalld port 8443/tcp in current zone...
Ceph Dashboard is now available at:
URL: https://ts-473a-01.internal.pcfe.net:8443/
User: admin
Password: [REDACTED]
Enabling client.admin keyring and conf on hosts with "admin" label
You can access the Ceph CLI with:
sudo /usr/sbin/cephadm shell --fsid 2570b1c0-8c5f-11ec-88d0-245ebe4b8fc0 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
Please consider enabling telemetry to help improve Ceph:
ceph telemetry on
For more information see:
https://docs.ceph.com/docs/pacific/mgr/telemetry/
Bootstrap complete.
After bootstrap completed, I pointed FireFox (on my workstation) at the node, changed my password (as reasonably enforced by the webUI) and logged in.
No screenshot this time, look at my near identical post but on Fedora Server if you crave a screenshot :-)
Cephadm Shell
Just a quick smoke test if it is functional, as you can see I have not yet set up OSDs, pools etc.
While I could just use cephadm shell, I used the full command provided by the bootstrap output.
[root@ts-473a-01 ~]# /usr/sbin/cephadm shell --fsid 2570b1c0-8c5f-11ec-88d0-245ebe4b8fc0 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
Using recent ceph image quay.io/ceph/ceph@sha256:a39107f8d3daab4d756eabd6ee1630d1bc7f31eaa76fff41a77fa32d0b903061
[ceph: root@ts-473a-01 /]# ceph -s
cluster:
id: 2570b1c0-8c5f-11ec-88d0-245ebe4b8fc0
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 2
services:
mon: 1 daemons, quorum ts-473a-01 (age 8m)
mgr: ts-473a-01.dvkmdy(active, since 5m), standbys: ts-473a-01.pypagy
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[ceph: root@ts-473a-01 /]# exit
exit
[root@ts-473a-01 ~]#
That’s all for this post.