Categories: braindump

GitLab Backup to CephFS

Table of Contents

This is a quick braindump of setting up our home GitLab instance (on CentOS 8) to backup to an existing CephFS but with access limited to the directory /GitLab_backup.

[root@gitlab ~]# df -h /mnt/cephfs-GitLab_backup/
Filesystem      Size  Used Avail Use% Mounted on
ceph-fuse       5.3T  175G  5.1T   4% /mnt/cephfs-GitLab_backup

The Ceph cluster is a containerised Ceph Nautilus, specifically Red Hat Ceph Storage (RHCS) 4.1.

Required Reading

Ceph User Setup

Access to the CephFS needs a client to authenticate.

Ceph User Creation

I want this user to only have access to /GitLab_backup on the CephFS.

[root@f5-422-01 ~]# podman exec  --tty --interactive ceph-mon-f5-422-01 \
ceph auth get-or-create client.gitlab_server mon 'allow r' mds 'allow rw path=/GitLab_backup' osd 'allow rw'

Obtain Keyring from Container onto MON node

The credentials will need to be copied to the client machine.

On a Ceph MON:

First create a client keyring inside the container:

[root@f5-422-01 ~]# podman exec  --tty --interactive ceph-mon-f5-422-01 \
ceph auth get client.gitlab_server -o /root/ceph.client.gitlab_server.keyring

Then extract keyring from the container:

[root@f5-422-01 ~]# podman cp ceph-mon-f5-422-01:/root/ceph.client.gitlab_server.keyring ~/tmp/

In the next step we will, amongst others, make that keyring available to the client machine. Obviously, delete ~/tmp/ceph.client.gitlab_server.keyring once it’s copied onto the client machine.

GitLab as CephFS Client

The CephFS client machine is our home GitLab server. It’s running CentOS 8.

Set Up FUSE Client

While I chose to use the FUSE Client, you could also try the kernel Client. Both have their own advantages, check the upstream docs to help you decide.

For other OSes, obtaining and configuring a FUSE Client will be different. Check out the [clients] capability of ceph-ansible for automated steps.

[root@gitlab ~]# dnf install centos-release-ceph-nautilus.noarch
[root@gitlab ~]# dnf install ceph-fuse
[root@gitlab ~]# ls /etc/ceph/
[root@gitlab ~]# scp root@f5-422-01:tmp/ceph.client.gitlab_server.keyring /etc/ceph/ceph.client.gitlab_server.keyring
[root@gitlab ~]# scp root@f5-422-01:/etc/ceph.conf /etc/ceph/ceph.conf
[root@gitlab ~]# chmod 600 /etc/ceph/ceph.client.gitlab_server.keyring

Test FUSE Client

Mounting once manually to verify everything is set up correctly and create a subdirectory to contain the backup of repositories and metadata.

[root@gitlab ~]# mkdir /mnt/cephfs-GitLab_backup
[root@gitlab ~]# ceph-fuse -n client.gitlab_server /mnt/cephfs-GitLab_backup --client_mountpoint /GitLab_backup
[root@gitlab ~]# ls /mnt/cephfs-GitLab_backup
[root@gitlab ~]# mkdir /mnt/cephfs-GitLab_backup/repos-and-metadata
[root@gitlab ~]# chown git.git /mnt/cephfs-GitLab_backup/repos-and-metadata/
[root@gitlab ~]# fusermount -u /mnt/cephfs-GitLab_backup

fstab Entry on the Client

This should be mounted automatically at boot.

none  /mnt/cephfs-GitLab_backup  fuse.ceph  ceph.id=gitlab_server,ceph.client_mountpoint=/GitLab_backup,_netdev,defaults  0 0

GitLab Server Config Changes

This backs up repositories and metadata to /var/opt/gitlab/backups/, followed by an copy to the locally mounted /mnt/cephfs-GitLab_backup/repos-and-metadata.

Backing up directly to /mnt/cephfs-GitLab_backup/repos-and-metadata, I got tar: … file changed as we read it, even with gitlab-backup create STRATEGY=copy.

Make GitLab Copy to a Locally Mounted Directory after Backup

As per the GitLab documentation.

In /etc/gitlab/gitlab.rb

# gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"
# gitlab_rails['backup_path'] = '/mnt/cephfs-GitLab_backup/repos-and-metadata'
# limit backup lifetime to 7 days - 604800 seconds
gitlab_rails['backup_keep_time'] = 604800
gitlab_rails['backup_upload_connection'] = {
  :provider => 'Local',
  :local_root => '/mnt/cephfs-GitLab_backup'
}

# The directory inside the mounted folder to copy backups to
# Use '.' to store them in the root directory
gitlab_rails['backup_upload_remote_directory'] = 'repos-and-metadata'

followed by

[root@gitlab ~]# gitlab-ctl reconfigure

Test GitLab Backup to CephFS

[root@gitlab ~]# gitlab-backup create
[root@gitlab ~]# ls -l /mnt/cephfs-GitLab_backup/repos-and-metadata/
total 669590
-rw-------. 1 git git 685660160 Jul 18 18:21 1595089287_2020_07_18_13.1.4-ee_gitlab_backup.tar
[root@gitlab ~]# tar tvf /mnt/cephfs-GitLab_backup/repos-and-metadata/1595089287_2020_07_18_13.1.4-ee_gitlab_backup.tar

GitLab Server Crontab

Since one should not store the backup of sensitive data (gitlab-ctl backup-etc) alongside the normal backup of repos and metadata (gitlab-backup create), I have the following two entries in my crontab; once a month the secrets and every day the repos and metadata

15 4 1 * * /usr/bin/gitlab-ctl backup-etc /net/fileserver/var/crypt/backups/GitLab/backup_secrets/config_backup
30 4 * * * /opt/gitlab/bin/gitlab-backup create CRON=1

Related