Ceph Nautilus and NFS-Ganesha
Table of Contents
this is my braindump on setting up NFS-Ganesha to serve 3 separate directories on my CephFS using 3 separate cephx users.
Preparations
As per section 4.9. Installing the NFS-Ganesha Gateway of the RHCS 4.1 install guide.
User Creation
Create users, fetch keyrings and copy to ganesha machine.
[root@f5-422-01 ~]# podman exec --tty --interactive ceph-mon-f5-422-01 ceph auth get-or-create client.nfs_ganesha_HouseNet_shared mon 'allow r' mds 'allow rw path=/HouseNet_shared' osd 'allow rw'
[root@f5-422-01 ~]# podman exec --tty --interactive ceph-mon-f5-422-01 ceph auth get-or-create client.nfs_ganesha_nfs4ocp mon 'allow r' mds 'allow rw path=/nfs4ocp' osd 'allow rw'
[root@f5-422-01 ~]# podman exec --tty --interactive ceph-mon-f5-422-01 ceph auth get-or-create client.nfs_ganesha_OCP4_testing mon 'allow r' mds 'allow rw path=/OCP4_testing' osd 'allow rw'
[root@f5-422-01 ~]# podman exec --interactive --tty ceph-mon-f5-422-01 ceph auth get client.nfs_ganesha_HouseNet_shared -o /root/ceph.client.nfs_ganesha_HouseNet_shared.keyring
exported keyring for client.nfs_ganesha_HouseNet_shared
[root@f5-422-01 ~]# podman exec --interactive --tty ceph-mon-f5-422-01 ceph auth get client.nfs_ganesha_OCP4_testing -o /root/ceph.client.nfs_ganesha_OCP4_testing.keyring
exported keyring for client.nfs_ganesha_OCP4_testing
[root@f5-422-01 ~]# podman exec --interactive --tty ceph-mon-f5-422-01 ceph auth get client.nfs_ganesha_nfs4ocp -o /root/ceph.client.nfs_ganesha_nfs4ocp.keyring
exported keyring for client.nfs_ganesha_nfs4ocp
[root@f5-422-01 ~]# podman cp ceph-mon-f5-422-01:/root/ceph.client.nfs_ganesha_HouseNet_shared.keyring ~/tmp/
[root@f5-422-01 ~]# podman cp ceph-mon-f5-422-01:/root/ceph.client.nfs_ganesha_OCP4_testing.keyring ~/tmp/
[root@f5-422-01 ~]# podman cp ceph-mon-f5-422-01:/root/ceph.client.nfs_ganesha_nfs4ocp.keyring ~/tmp/
[root@f5-422-01 ~]# ls -1 ~/tmp/ceph.client.nfs_ganesha*.keyring
/root/tmp/ceph.client.nfs_ganesha_HouseNet_shared.keyring
/root/tmp/ceph.client.nfs_ganesha.keyring
/root/tmp/ceph.client.nfs_ganesha_nfs4ocp.keyring
/root/tmp/ceph.client.nfs_ganesha_OCP4_testing.keyring
[root@f5-422-01 ~]# scp ~/tmp/ceph.client.nfs_ganesha*.keyring ganesha:/etc/ceph/
ceph-ansible
I currently use this group_vars/nfss.yml
FIXME: read up on ceph_nfs_rados_backend and then enable.
nfs_file_gw: false because I do not want to export the whole CephFS, only some select subdirectories.
[ansible@ceph-ansible ceph-ansible]$ grep ^[^#] group_vars/nfss.yml
---
dummy:
copy_admin_key: false
ceph_nfs_enable_service: true
nfs_file_gw: false
nfs_obj_gw: false
ceph_nfs_log_file: "/var/log/ganesha/ganesha.log"
ceph_nfs_rados_backend: false
ceph_nfs_rados_export_index: "ganesha-export-index"
ceph_nfs_disable_caching: true
ganesha_conf_overrides: |
EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
Export_Id = 10001;
# Exported path (mandatory)
Path = /HouseNet_shared;
# Pseudo Path (required for NFSv4 or if mount_path_pseudo = true)
Pseudo = /HouseNet_shared;
# Restrict the protocols that may use this export. This cannot allow
# access that is denied in NFS_CORE_PARAM.
#Protocols = 3,4;
Protocols = 4;
# Access type for clients. Default is None, so some access must be
# given. It can be here, in the EXPORT_DEFAULTS, or in a CLIENT block
Access_Type = RO;
# Whether to squash various users.
Squash = root_squash;
# Allowed security types for this export
Sectype = sys,krb5,krb5i,krb5p;
# Exporting FSAL
FSAL {
Name = CEPH;
User_Id = nfs_ganesha_HouseNet_shared;
}
}
EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
Export_Id = 10002;
# Exported path (mandatory)
Path = /OCP4_testing;
# Pseudo Path (required for NFSv4 or if mount_path_pseudo = true)
Pseudo = /OCP4_testing;
# Restrict the protocols that may use this export. This cannot allow
# access that is denied in NFS_CORE_PARAM.
Protocols = 3,4;
#Protocols = 4;
# Access type for clients. Default is None, so some access must be
# given. It can be here, in the EXPORT_DEFAULTS, or in a CLIENT block
Access_Type = RW;
# Whether to squash various users.
Squash = root_squash;
# Allowed security types for this export
Sectype = sys,krb5,krb5i,krb5p;
# Exporting FSAL
FSAL {
Name = CEPH;
User_Id = nfs_ganesha_OCP4_testing;
}
}
EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
Export_Id = 10003;
# Exported path (mandatory)
Path = /nfs4ocp;
# Pseudo Path (required for NFSv4 or if mount_path_pseudo = true)
Pseudo = /nfs4ocp;
# Restrict the protocols that may use this export. This cannot allow
# access that is denied in NFS_CORE_PARAM.
Protocols = 3,4;
#Protocols = 4;
# Access type for clients. Default is None, so some access must be
# given. It can be here, in the EXPORT_DEFAULTS, or in a CLIENT block
Access_Type = RW;
# Whether to squash various users.
Squash = root_squash;
# Allowed security types for this export
Sectype = sys,krb5,krb5i,krb5p;
# Exporting FSAL
FSAL {
Name = CEPH;
User_Id = nfs_ganesha_nfs4ocp;
}
}
Quick Smoke Test
The machine t3600 has access to the whole CephFS. That is mounted at /mnt/cephfs/.
/mnt/import/ is used for testing NFS access via the machine where the ganesha container runs.
Here that is a VM, but I could also have deployed that to an OSD, it’s just that mine are a bit on the lightweight side,
so I elected to use a VM on a beefy hypervisor for this.
[root@t3600 ~]# ll /etc/ceph/
total 20
-rw-------. 1 root root 40 Aug 15 19:42 ceph.client.libvirt_t3600.secret
-rw-------. 1 root root 159 Aug 15 18:46 ceph.client.workstation_pcfe.keyring
-rw-------. 1 root root 40 Aug 15 18:46 ceph.client.workstation_pcfe.secret
-rw-r--r--. 1 root root 297 Aug 15 18:49 ceph.conf
drwxr-xr-x. 2 root root 4096 Aug 15 18:46 old
[root@t3600 ~]# mount ganesha.internal.pcfe.net:/ /mnt/import/
[root@t3600 ~]# ls /mnt/cephfs/
GitLab_backup HouseNet_shared nfs4ocp OCP4_testing pcfe
[root@t3600 ~]# ls /mnt/import/
HouseNet_shared nfs4ocp OCP4_testing
[root@t3600 ~]# umount /mnt/import
[root@t3600 ~]# showmount -e ganesha.internal.pcfe.net
clnt_create: RPC: Unable to receive
[root@t3600 ~]# showmount -e ganesha.storage.pcfe.net
clnt_create: RPC: Unable to receive
[root@t3600 ~]# mount ganesha.internal.pcfe.net:/OCP4_testing /mnt/import/
[root@t3600 ~]# ls /mnt/import/
banana
[root@t3600 ~]# umount /mnt/import
[root@t3600 ~]# mount ganesha.internal.pcfe.net:/nfs4ocp /mnt/import/
[root@t3600 ~]# ls /mnt/import/
[root@t3600 ~]# ls -la /mnt/import/
total 5
drwxrwxrwx. 2 root root 0 Aug 25 21:07 .
drwxr-xr-x. 6 root root 4096 Jun 23 23:52 ..
[root@t3600 ~]# umount /mnt/import/
[root@t3600 ~]# mount ganesha.internal.pcfe.net:/HouseNet_shared /mnt/import/
[root@t3600 ~]# touch /mnt/import/banana
touch: cannot touch '/mnt/import/banana': Read-only file system
[root@t3600 ~]# touch /mnt/cephfs/HouseNet_shared/00-readme.txt
[root@t3600 ~]# ll /mnt/import/
total 0
-rw-r--r--. 1 root root 0 Aug 25 22:35 00-readme.txt
[root@t3600 ~]# umount /mnt/import
[root@t3600 ~]# mount ganesha.internal.pcfe.net:/OCP4_testing /mnt/import/
[root@t3600 ~]# ls /mnt/import/banana/
rhel-8.2-x86_64-boot.iso rhel-8.2-x86_64-kvm.qcow2 supp-supplementary-8.2-rhel-8-x86_64-dvd.iso
rhel-8.2-x86_64-dvd.iso SHA256SUMS
[root@t3600 ~]# umount /mnt/import
[root@t3600 ~]# mount ganesha.internal.pcfe.net:/OCP4_testing/banana /mnt/import/
[root@t3600 ~]# ls /mnt/import/
rhel-8.2-x86_64-boot.iso rhel-8.2-x86_64-kvm.qcow2 supp-supplementary-8.2-rhel-8-x86_64-dvd.iso
rhel-8.2-x86_64-dvd.iso SHA256SUMS
[root@t3600 ~]# cd /mnt/import/
[root@t3600 import]# sha256sum -c SHA256SUMS
rhel-8.2-x86_64-boot.iso: OK
rhel-8.2-x86_64-dvd.iso: OK
rhel-8.2-x86_64-kvm.qcow2: OK
supp-supplementary-8.2-rhel-8-x86_64-dvd.iso: OK
[root@t3600 import]# cd
[root@t3600 ~]#
[root@t3600 ~]# umount /mnt/import/
ToDo
https://docs.ceph.com/docs/nautilus/mgr/dashboard/#nfs-ganesha-management
and ceph_nfs_rados_backend: true
first try not so good;
[root@ganesha ~]# podman logs -f ceph-nfs-ganesha
2020-08-26 06:41:06 /opt/ceph-container/bin/entrypoint.sh: static: does not generate config
2020-08-26 06:41:06 /opt/ceph-container/bin/entrypoint.sh: SUCCESS
exec: PID 56: spawning /usr/bin/ganesha.nfsd -F -L STDOUT
exec: Waiting 56 to quit
26/08/2020 06:41:06 : epoch 5f45e7e2 : ganesha.internal.pcfe.net : ganesha.nfsd-56[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /builddir/build/BUILD/nfs-ganesha-2.8.3/src, built at Jun 3 2020 20:45:02 on
26/08/2020 06:41:06 : epoch 5f45e7e2 : ganesha.internal.pcfe.net : ganesha.nfsd-56[main] rados_url_client_setup :CONFIG :EVENT :rados_url_client_setup: Failed to connect to cluster
26/08/2020 06:41:06 : epoch 5f45e7e2 : ganesha.internal.pcfe.net : ganesha.nfsd-56[main] cu_rados_url_fetch :CONFIG :EVENT :cu_rados_url_fetch: Failed to create ioctx
26/08/2020 06:41:06 : epoch 5f45e7e2 : ganesha.internal.pcfe.net : ganesha.nfsd-56[main] main :NFS STARTUP :CRIT :Error (token scan) while parsing (/etc/ganesha/ganesha.conf)
26/08/2020 06:41:06 : epoch 5f45e7e2 : ganesha.internal.pcfe.net : ganesha.nfsd-56[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:23): new url (rados://cephfs_data/ganesha-export-index) open error (Unknown error -107), ignored
26/08/2020 06:41:06 : epoch 5f45e7e2 : ganesha.internal.pcfe.net : ganesha.nfsd-56[main] main :NFS STARTUP :FATAL :Fatal errors. Server exiting...
teardown: managing teardown after SIGCHLD
teardown: Waiting PID 56 to terminate
teardown: Process 56 is terminated
teardown: Bye Bye, container will die with return code 0
retry with admin key copied, better, ganesha container works.
Also needed setting of a dashboard option
[root@f5-422-01 ~]# podman exec --interactive --tty ceph-mon-f5-422-01 ceph dashboard get-ganesha-clusters-rados-pool-namespace
[root@f5-422-01 ~]# podman exec --interactive --tty ceph-mon-f5-422-01 ceph dashboard set-ganesha-clusters-rados-pool-namespace cephfs_data/ganesha-export-index
Option GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE updated
But the dashboard does not show me my 3 exports (could be because they came in via ganesha_conf_overrides)
AND when I try to create an export in the dashboard, I get No cluster available, so I’m still missing a setting it seems.