Minisforum MS-01, CentOS Stream 9, Ceph Squid install

Table of Contents

doh!, install of my Ceph Squid 19 itself was in early March, but I forgot to publish the braindump I’d written to this blog.

It was really just following the official Using cephadm to Deploy a New Ceph Cluster for the latest version (squid) at the time of install.

Ceph dashboard teaser pic showing 5 host, 3 MONs, 3 MGRs and 10 OSDs

Additioonally, on the hottest summer days this year the MS-01 nodes were among the hardware nodes I switched off to keep the temperature in the work room at least below 40°C.

I actually ran a few minor update rounds (Dashboard / Administration / Upgrade) since the install in MArch happened. The original install was 19.2.1 today, 2025-09-06, it’s at 19.2.3 both GUI led updates were pleasantly hassle free. Nice!

Documentation pages to RTFM

Bootstrap

As per the section Running the bootstrap command of the Ceph Squid docs.

[root@ms-01-01 ~]# cephadm bootstrap --mon-ip 192.168.40.151 --allow-fqdn-hostname --cluster-network "192.168.30.0/24"
Click to see full bootstrap output.
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...    
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 5.4.0 is present
systemctl is present               
lvcreate is present
Unit chronyd.service is enabled and running              
Host looks OK          
Cluster fsid: cee8622c-fb45-11ef-9938-5847ca797c06
Verifying IP 192.168.40.151 port 3300 ...
Verifying IP 192.168.40.151 port 6789 ...                         
Mon IP `192.168.40.151` is in CIDR network `192.168.40.0/24`                                                                                       
Mon IP `192.168.40.151` is in CIDR network `192.168.40.0/24`                                                                                       
Pulling container image quay.io/ceph/ceph:v19...
Ceph version: ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)                                                   
Extracting ceph user uid/gid from container image...
Creating initial keys...                                  
Creating initial monmap...
Creating mon...                      
firewalld ready
Enabling firewalld service ceph-mon in current zone...  
Waiting for mon to start...
Waiting for mon...       
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...                             
Setting public_network to 192.168.40.0/24 in mon config section
Setting cluster_network to 192.168.30.0/24
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...                                                                                                                                    
Verifying port 0.0.0.0:9283 ...                                                                                                                    
Verifying port 0.0.0.0:8765 ...      
Verifying port 0.0.0.0:8443 ...                                                                                                                    
firewalld ready                           
Enabling firewalld service ceph in current zone...                                                                                                 
firewalld ready                                                                                                                                    
Enabling firewalld port 9283/tcp in current zone...             
Enabling firewalld port 8765/tcp in current zone...
Enabling firewalld port 8443/tcp in current zone...
Waiting for mgr to start...         
Waiting for mgr...                                                                                                                                                                                                                                                                                    
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr not available, waiting (4/15)...
mgr not available, waiting (5/15)...
mgr not available, waiting (6/15)...
mgr not available, waiting (7/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host ms-01-01.storage.pcfe.net...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Deploying ceph-exporter service with default placement...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for mgr epoch 9...
mgr epoch 9 is available
Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
firewalld ready
Ceph Dashboard is now available at:

             URL: https://ms-01-01.storage.pcfe.net:8443/
            User: admin
        Password: [REDACTED]

Enabling client.admin keyring and conf on hosts with "admin" label
Saving cluster configuration to /var/lib/ceph/cee8622c-fb45-11ef-9938-5847ca797c06/config directory
You can access the Ceph CLI as following in case of multi-cluster or non-default config:

        sudo /usr/sbin/cephadm shell --fsid cee8622c-fb45-11ef-9938-5847ca797c06 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

Or, if you are only running a single cluster on this host:

        sudo /usr/sbin/cephadm shell 

Please consider enabling telemetry to help improve Ceph:

        ceph telemetry on

For more information see:

        https://docs.ceph.com/en/latest/mgr/telemetry/

Bootstrap complete.
[root@ms-01-01 ~]# 

Distribute ssh pubkey to all other nodes

Distribute the key that got generated during bootstrapping to the other 4 nodes.

[root@ms-01-01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ms-01-…

If you are not in a homelab, then please RTFM at both https://docs.ceph.com/en/squid/cephadm/install/#deployment-with-custom-ssh-keys and https://docs.ceph.com/en/squid/cephadm/install/#deployment-with-ca-signed-ssh-keys before you copy ssh keys willy nilly in prod.

Use cephadm shell on the bootstrap node

Now that it was bootsrtrapped, all further ceph command swill happen in a cephadm shell

For a start, I share some telemetry data with the community.

[root@ms-01-01 ~]# cephadm shell
[…]
[ceph: root@ms-01-01 /]# 
[ceph: root@ms-01-01 /]# ceph version
ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)
[ceph: root@ms-01-01 /]# ceph telemetry on
Error EPERM: Telemetry data is licensed under the Community Data License Agreement - Sharing - Version 1.0 (https://cdla.io/sharing-1-0/).
To enable, add '--license sharing-1-0' to the 'ceph telemetry on' command.
[ceph: root@ms-01-01 /]# ceph telemetry on --license sharing-1-0
Telemetry is on.
Some channels are disabled, please enable with:
`ceph telemetry enable channel perf`
[ceph: root@ms-01-01 /]# ceph orch host ls
                                                                                                      
HOST                       ADDR            LABELS  STATUS  
ms-01-01.storage.pcfe.net  192.168.40.151  _admin          
1 hosts in cluster

Add hosts

[ceph: root@ms-01-01 /]# ceph orch host add ms-01-02.storage.pcfe.net 192.168.40.152 --labels _admin
Added host 'ms-01-02.storage.pcfe.net' with addr '192.168.40.152'
[ceph: root@ms-01-01 /]# ceph orch host add ms-01-03.storage.pcfe.net 192.168.40.153 --labels _admin
                                                                                                      
Added host 'ms-01-03.storage.pcfe.net' with addr '192.168.40.153'
[ceph: root@ms-01-01 /]# ceph orch host add ms-01-04.storage.pcfe.net 192.168.40.154 --labels _admin
                                                                                                      
Added host 'ms-01-04.storage.pcfe.net' with addr '192.168.40.154'
[ceph: root@ms-01-01 /]# ceph orch host add ms-01-05.storage.pcfe.net 192.168.40.155 --labels _admin
                                                                                                      
Added host 'ms-01-05.storage.pcfe.net' with addr '192.168.40.155'
[ceph: root@ms-01-01 /]# ceph log last cephadm ; ceph -W cephadm

NOTE:: later more labels were added, some in the Dashboard some in the cephadm shell.

Add unsused storage

What I want is to consume all available and unused devices on my 5 nodes and that the OSDs are encrypted, the performance hit for luks is small and when one of the cheap consumer NVMe dies, I want to not have to faff around with wiping the device before I put it on the electronics recycling heap.

See also https://docs.ceph.com/en/squid/cephadm/install/#adding-storage

I do not want to use the all available devices default

What did not work for me was the default for all available devices.

Click to see details of me trying that
[ceph: root@ms-01-01 /]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@ms-01-01 /]# ceph log last cephadm ; ceph -W cephadm
[…]
2025-03-07T11:40:08.153124+0000 mgr.ms-01-01.pnupxa (mgr.14199) 704 : cephadm [INF] Marking host: ms-01-01.storage.pcfe.net for OSDSpec preview refresh.
2025-03-07T11:40:08.153163+0000 mgr.ms-01-01.pnupxa (mgr.14199) 705 : cephadm [INF] Marking host: ms-01-02.storage.pcfe.net for OSDSpec preview refresh.
2025-03-07T11:40:08.153190+0000 mgr.ms-01-01.pnupxa (mgr.14199) 706 : cephadm [INF] Marking host: ms-01-03.storage.pcfe.net for OSDSpec preview refresh.
2025-03-07T11:40:08.153211+0000 mgr.ms-01-01.pnupxa (mgr.14199) 707 : cephadm [INF] Marking host: ms-01-04.storage.pcfe.net for OSDSpec preview refresh.
2025-03-07T11:40:08.153233+0000 mgr.ms-01-01.pnupxa (mgr.14199) 708 : cephadm [INF] Marking host: ms-01-05.storage.pcfe.net for OSDSpec preview refresh.
2025-03-07T11:40:08.153476+0000 mgr.ms-01-01.pnupxa (mgr.14199) 709 : cephadm [INF] Saving service osd.all-available-devices spec with placement *
2025-03-07T11:40:19.586771+0000 mgr.ms-01-01.pnupxa (mgr.14199) 715 : cephadm [INF] Deploying daemon osd.4 on ms-01-03.storage.pcfe.net
2025-03-07T11:40:19.651087+0000 mgr.ms-01-01.pnupxa (mgr.14199) 717 : cephadm [INF] Deploying daemon osd.2 on ms-01-02.storage.pcfe.net
2025-03-07T11:40:19.697048+0000 mgr.ms-01-01.pnupxa (mgr.14199) 718 : cephadm [INF] Deploying daemon osd.3 on ms-01-05.storage.pcfe.net
2025-03-07T11:40:19.810508+0000 mgr.ms-01-01.pnupxa (mgr.14199) 719 : cephadm [INF] Deploying daemon osd.0 on ms-01-01.storage.pcfe.net
2025-03-07T11:40:19.951175+0000 mgr.ms-01-01.pnupxa (mgr.14199) 720 : cephadm [INF] Deploying daemon osd.1 on ms-01-04.storage.pcfe.net
2025-03-07T11:40:28.379302+0000 mgr.ms-01-01.pnupxa (mgr.14199) 725 : cephadm [INF] Deploying daemon osd.5 on ms-01-02.storage.pcfe.net
2025-03-07T11:40:28.510591+0000 mgr.ms-01-01.pnupxa (mgr.14199) 726 : cephadm [INF] Deploying daemon osd.6 on ms-01-03.storage.pcfe.net
2025-03-07T11:40:28.558902+0000 mgr.ms-01-01.pnupxa (mgr.14199) 727 : cephadm [INF] Deploying daemon osd.8 on ms-01-01.storage.pcfe.net
2025-03-07T11:40:28.669190+0000 mgr.ms-01-01.pnupxa (mgr.14199) 728 : cephadm [INF] Deploying daemon osd.7 on ms-01-05.storage.pcfe.net
2025-03-07T11:40:28.980062+0000 mgr.ms-01-01.pnupxa (mgr.14199) 729 : cephadm [INF] Deploying daemon osd.9 on ms-01-04.storage.pcfe.net
  cluster:
    id:     cee8622c-fb45-11ef-9938-5847ca797c06
    health: HEALTH_OK
 
  services:
    mon: 5 daemons, quorum ms-01-01,ms-01-02,ms-01-05,ms-01-03,ms-01-04 (age 4m)
    mgr: ms-01-01.pnupxa(active, since 19m), standbys: ms-01-02.yuzhvl
    osd: 10 osds: 1 up (since 1.01027s), 10 in (since 19s)
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     
 

2025-03-07T11:40:47.650454+0000 mgr.ms-01-01.pnupxa [INF] Detected new or changed devices on ms-01-04.storage.pcfe.net
2025-03-07T11:40:47.660575+0000 mgr.ms-01-01.pnupxa [INF] Detected new or changed devices on ms-01-03.storage.pcfe.net
2025-03-07T11:40:47.689086+0000 mgr.ms-01-01.pnupxa [INF] Adjusting osd_memory_target on ms-01-04.storage.pcfe.net to  9447M
2025-03-07T11:40:47.698657+0000 mgr.ms-01-01.pnupxa [INF] Adjusting osd_memory_target on ms-01-03.storage.pcfe.net to  9449M
2025-03-07T11:40:47.724974+0000 mgr.ms-01-01.pnupxa [INF] Detected new or changed devices on ms-01-05.storage.pcfe.net
2025-03-07T11:40:47.744682+0000 mgr.ms-01-01.pnupxa [INF] Adjusting osd_memory_target on ms-01-05.storage.pcfe.net to  9449M
2025-03-07T11:40:47.831136+0000 mgr.ms-01-01.pnupxa [INF] Detected new or changed devices on ms-01-02.storage.pcfe.net
2025-03-07T11:40:47.851474+0000 mgr.ms-01-01.pnupxa [INF] Adjusting osd_memory_target on ms-01-02.storage.pcfe.net to  7401M
2025-03-07T11:40:48.522158+0000 mgr.ms-01-01.pnupxa [INF] Detected new or changed devices on ms-01-01.storage.pcfe.net
2025-03-07T11:40:48.559680+0000 mgr.ms-01-01.pnupxa [INF] Adjusting osd_memory_target on ms-01-01.storage.pcfe.net to  5863M

Awww, frack, that ends up with NOT encrypted devices.

It’s not what I want because this does not enable at rest encryption, so I disable that by setting it to unmanaged.

[ceph: root@ms-01-01 /]# ceph orch apply osd --all-available-devices --unmanaged=true
Scheduled osd.all-available-devices update...

Using a custom drive specification

And whipped up a primitive drive spec, mainly to get encrypted: true in.

See also: https://docs.ceph.com/en/squid/cephadm/services/osd/#advanced-osd-service-specifications

Create the drive spec

  • node name starts with ms-01-
  • device does not rotate
  • OSD will be encrypted
[root@ms-01-01 ~]# cat ms-01-SSD-as-OSD-spec.yml
service_type: osd
service_id: osd_spec_MS-01_SSDs
placement:
  host_pattern: 'ms-01-*'
encrypted: true
data_devices:
  rotational: 0

Apply drive spec

Work around my earlier mistake

Because I first applied a drive spec that gave me non-encrypted OSDs I no longer had unused devices, now I could of course throw away the whole cluster, wipe all storage and start from zero, but why would I do that when I have many OSDs and very very little capacity used. I’ll just do them in a semi rolling fashion

Click to see details of me first applying the wrong spec and then doing nuke&pave on one OSD to free it as a test
[root@ms-01-01 ~]#  cephadm shell --mount ms-01-SSD-as-OSD-spec.yml:/var/lib/ceph/osd/ms-01-SSD-as-OSD-spec.yml
Inferring fsid cee8622c-fb45-11ef-9938-5847ca797c06
Inferring config /var/lib/ceph/cee8622c-fb45-11ef-9938-5847ca797c06/mon.ms-01-01/config
Not using image 'f2efb0401a30ec7eda97b6da76b314bd081fcb910cc5dcd826bc7c72c9dfdd7d' as it's not in list of non-dangling images with ceph=True label
[ceph: root@ms-01-01 /]# cd /var/lib/ceph/osd/
[ceph: root@ms-01-01 osd]# ls
ms-01-SSD-as-OSD-spec.yml
[ceph: root@ms-01-01 osd]# cat ms-01-SSD-as-OSD-spec.yml 
service_type: osd
service_id: osd_spec_MS-01_SSDs
placement:
  host_pattern: 'ms-01-*'
encrypted: true
data_devices:
  rotational: 0
[ceph: root@ms-01-01 osd]# ceph orch apply -i ms-01-SSD-as-OSD-spec.yml --dry-run
WARNING! Dry-Runs are snapshots of a certain point in time and are bound 
to the current inventory setup. If any of these conditions change, the 
preview will be invalid. Please make sure to have a minimal 
timeframe between planning and applying the specs.
####################
SERVICESPEC PREVIEWS
####################
+---------+------+--------+-------------+
|SERVICE  |NAME  |ADD_TO  |REMOVE_FROM  |
+---------+------+--------+-------------+
+---------+------+--------+-------------+
################
OSDSPEC PREVIEWS
################
Preview data is being generated.. Please re-run this command in a bit.

Ah well with no devices free the dry run could not give anything, so remove the first of the non crypted OSDs and check if drive spec picks it up later

[ceph: root@ms-01-01 osd]# ceph orch osd rm 0  --zap
Scheduled OSD(s) for removal.
[ceph: root@ms-01-01 osd]# ceph log last cephadm ; ceph -W cephadm
[…]
2025-03-08T11:01:53.573439+0000 mgr.ms-01-02.yuzhvl [INF] osd.0 now down
2025-03-08T11:01:53.573568+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.0 from ms-01-01.storage.pcfe.net -- ports []
2025-03-08T11:01:56.001333+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.0
2025-03-08T11:01:56.016993+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.0 on ms-01-01.storage.pcfe.net
2025-03-08T11:01:56.032853+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.0 on ms-01-01.storage.pcfe.net
2025-03-08T11:01:56.032882+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.0 on ms-01-01.storage.pcfe.net
2025-03-08T11:01:58.489295+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.0 on ms-01-01.storage.pcfe.net
2025-03-08T11:02:03.008587+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-01.storage.pcfe.net

another dry run

[ceph: root@ms-01-01 osd]# ceph orch apply -i ms-01-SSD-as-OSD-spec.yml --dry-run
WARNING! Dry-Runs are snapshots of a certain point in time and are bound 
to the current inventory setup. If any of these conditions change, the 
preview will be invalid. Please make sure to have a minimal 
timeframe between planning and applying the specs.
####################
SERVICESPEC PREVIEWS
####################
+---------+------+--------+-------------+
|SERVICE  |NAME  |ADD_TO  |REMOVE_FROM  |
+---------+------+--------+-------------+
+---------+------+--------+-------------+
################
OSDSPEC PREVIEWS
################
+---------+---------------------+---------------------------+--------------+----+-----+
|SERVICE  |NAME                 |HOST                       |DATA          |DB  |WAL  |
+---------+---------------------+---------------------------+--------------+----+-----+
|osd      |osd_spec_MS-01_SSDs  |ms-01-01.storage.pcfe.net  |/dev/nvme0n1  |-   |-    |
+---------+---------------------+---------------------------+--------------+----+-----+
[ceph: root@ms-01-01 osd]# 

looks good, apply

[ceph: root@ms-01-01 osd]# ceph orch apply -i ms-01-SSD-as-OSD-spec.yml 
Scheduled osd.osd_spec_MS-01_SSDs update...

Check at OS level that the OSD 0 was created as crypted

[root@ms-01-01 ~]# lsblk
NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1                                                                                               259:0    0   1.9T  0 disk  
└─ceph--39fa7697--930a--4e88--8280--c82f04bd4b42-osd--block--6af44afd--e59e--4ce3--8906--bf790f4a8e4d 253:2    0   1.9T  0 lvm   
  └─TyyWnn-bjFU-X7ua-Asd5-sKin-Jsh3-w5EJX4                                                            253:9    0   1.9T  0 crypt 
nvme1n1                                                                                               259:1    0   1.8T  0 disk  
└─ceph--5c22e2bb--67ec--4c1b--a59d--203e2083a0dd-osd--block--24f804d8--c3d1--4512--adf1--90b438072061 253:3    0   1.8T  0 lvm   
nvme2n1                                                                                               259:2    0 953.9G  0 disk  
├─nvme2n1p1                                                                                           259:3    0   600M  0 part  /boot/efi
├─nvme2n1p2                                                                                           259:4    0     1G  0 part  /boot
└─nvme2n1p3                                                                                           259:5    0   200G  0 part  
  ├─VG_OS-LV_root                                                                                     253:0    0     4G  0 lvm   /
  ├─VG_OS-LV_swap                                                                                     253:1    0     1G  0 lvm   [SWAP]
  ├─VG_OS-LV_var_crash                                                                                253:4    0    40G  0 lvm   /var/crash
  ├─VG_OS-LV_var                                                                                      253:5    0    12G  0 lvm   /var
  ├─VG_OS-LV_home                                                                                     253:6    0     1G  0 lvm   /home
  ├─VG_OS-LV_var_log                                                                                  253:7    0    10G  0 lvm   /var/log
  └─VG_OS-LV_containers                                                                               253:8    0    10G  0 lvm   /var/lib/containers/storage/overlay
                                                                                                                                 /var/lib/containers
[root@ms-01-01 ~]# 

yupp looks much better, nvme0n1 is indeed crypted.

as expected the freshly rolled ous OSD is encrypted.

Click to see details of me deleting more of the non crypted ones, did it in 2 halves
[ceph: root@ms-01-01 /]# ceph orch osd rm 1  --zap
Scheduled OSD(s) for removal.
[ceph: root@ms-01-01 /]# ceph orch osd rm 2  --zap
Scheduled OSD(s) for removal.
[ceph: root@ms-01-01 /]# ceph orch osd rm 3  --zap
Scheduled OSD(s) for removal.
[ceph: root@ms-01-01 /]# ceph orch osd rm 4  --zap
Scheduled OSD(s) for removal.

wait a bit, then check on progress

[ceph: root@ms-01-01 /]# ceph orch osd rm status
OSD  HOST                       STATE     PGS  REPLACE  FORCE  ZAP   DRAIN STARTED AT            
1    ms-01-04.storage.pcfe.net  draining    7  False    False  True  2025-03-08 11:08:10.780531  
2    ms-01-02.storage.pcfe.net  draining    4  False    False  True  2025-03-08 11:08:13.833413  
3    ms-01-05.storage.pcfe.net  draining    6  False    False  True  2025-03-08 11:09:58.385193  
4    ms-01-03.storage.pcfe.net  draining    2  False    False  True  2025-03-08 11:08:57.038480  
[ceph: root@ms-01-01 /]# 

Went well, as expected

[ceph: root@ms-01-01 /]# ceph -W cephadm
  cluster:
    id:     cee8622c-fb45-11ef-9938-5847ca797c06
    health: HEALTH_OK
 
  services:
    mon: 5 daemons, quorum ms-01-01,ms-01-02,ms-01-05,ms-01-03,ms-01-04 (age 19h)
    mgr: ms-01-02.yuzhvl(active, since 20h), standbys: ms-01-01.pnupxa
    osd: 10 osds: 10 up (since 10m), 10 in (since 19h); 12 remapped pgs
 
  data:
    pools:   3 pools, 65 pgs
    objects: 30.58k objects, 119 GiB
    usage:   376 GiB used, 18 TiB / 18 TiB avail
    pgs:     19208/91746 objects misplaced (20.936%)
             53 active+clean
             10 active+remapped+backfill_wait
             2  active+remapped+backfilling
 
  io:
    recovery: 168 MiB/s, 41 objects/s
 

2025-03-08T11:19:10.697465+0000 mgr.ms-01-02.yuzhvl [INF] osd.1 now down
2025-03-08T11:19:10.697653+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.1 from ms-01-04.storage.pcfe.net -- ports []
2025-03-08T11:19:13.292850+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.1
2025-03-08T11:19:13.309298+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.1 on ms-01-04.storage.pcfe.net
2025-03-08T11:19:13.325684+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.1 on ms-01-04.storage.pcfe.net
2025-03-08T11:19:13.325714+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.1 on ms-01-04.storage.pcfe.net
2025-03-08T11:19:15.753203+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.1 on ms-01-04.storage.pcfe.net
2025-03-08T11:19:16.770071+0000 mgr.ms-01-02.yuzhvl [INF] osd.4 now down
2025-03-08T11:19:16.770239+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.4 from ms-01-03.storage.pcfe.net -- ports []
2025-03-08T11:19:19.413494+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.4
2025-03-08T11:19:19.428111+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.4 on ms-01-03.storage.pcfe.net
2025-03-08T11:19:19.442515+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.4 on ms-01-03.storage.pcfe.net
2025-03-08T11:19:19.442577+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.4 on ms-01-03.storage.pcfe.net
2025-03-08T11:19:21.889073+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.4 on ms-01-03.storage.pcfe.net
2025-03-08T11:19:26.229376+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-04.storage.pcfe.net
2025-03-08T11:19:26.239572+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-03.storage.pcfe.net
2025-03-08T11:19:41.571565+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.1 on ms-01-03.storage.pcfe.net
2025-03-08T11:19:47.716121+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.4 on ms-01-04.storage.pcfe.net
2025-03-08T11:20:07.614992+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-03.storage.pcfe.net
2025-03-08T11:20:07.791014+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-04.storage.pcfe.net
2025-03-08T11:21:05.066704+0000 mgr.ms-01-02.yuzhvl [INF] osd.2 now down
2025-03-08T11:21:05.066860+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.2 from ms-01-02.storage.pcfe.net -- ports []
2025-03-08T11:21:07.644311+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.2
2025-03-08T11:21:07.659369+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.2 on ms-01-02.storage.pcfe.net
2025-03-08T11:21:07.673985+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.2 on ms-01-02.storage.pcfe.net
2025-03-08T11:21:07.674032+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.2 on ms-01-02.storage.pcfe.net
2025-03-08T11:21:10.122593+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.2 on ms-01-02.storage.pcfe.net
2025-03-08T11:21:13.973207+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-02.storage.pcfe.net
2025-03-08T11:21:15.757390+0000 mgr.ms-01-02.yuzhvl [INF] Adjusting osd_memory_target on ms-01-02.storage.pcfe.net to 14802M
2025-03-08T11:21:33.589860+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.2 on ms-01-02.storage.pcfe.net
2025-03-08T11:21:53.611449+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-02.storage.pcfe.net
2025-03-08T11:21:55.474212+0000 mgr.ms-01-02.yuzhvl [INF] Adjusting osd_memory_target on ms-01-02.storage.pcfe.net to  7401M
2025-03-08T11:35:29.777052+0000 mgr.ms-01-02.yuzhvl [INF] osd.3 now down
2025-03-08T11:35:29.777269+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.3 from ms-01-05.storage.pcfe.net -- ports []
2025-03-08T11:35:32.284114+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.3
2025-03-08T11:35:32.300918+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.3 on ms-01-05.storage.pcfe.net
2025-03-08T11:35:32.316676+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.3 on ms-01-05.storage.pcfe.net
2025-03-08T11:35:32.316708+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.3 on ms-01-05.storage.pcfe.net
2025-03-08T11:35:34.790736+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.3 on ms-01-05.storage.pcfe.net
2025-03-08T11:35:38.702871+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-05.storage.pcfe.net
2025-03-08T11:35:53.638463+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.3 on ms-01-05.storage.pcfe.net
2025-03-08T11:36:13.618714+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-05.storage.pcfe.net

then wait for all PGs to be active+clean again before removing the second half.

[ceph: root@ms-01-01 /]# ceph -w 
  cluster:
    id:     cee8622c-fb45-11ef-9938-5847ca797c06
    health: HEALTH_OK
 
  services:
    mon: 5 daemons, quorum ms-01-01,ms-01-02,ms-01-05,ms-01-03,ms-01-04 (age 20h)
    mgr: ms-01-02.yuzhvl(active, since 21h), standbys: ms-01-01.pnupxa
    osd: 10 osds: 10 up (since 11m), 10 in (since 20h)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 30.58k objects, 119 GiB
    usage:   367 GiB used, 18 TiB / 18 TiB avail
    pgs:     65 active+clean
 
  io:
    recovery: 49 MiB/s, 12 objects/s

and remove the second half of not encrypted OSDs so that the new drive spec (with encrypted: true) can pick it up.

[ceph: root@ms-01-01 /]# ceph orch osd rm 5 6 7 8 9 --zap
Scheduled OSD(s) for removal.
[ceph: root@ms-01-01 /]# ceph orch osd rm status
OSD  HOST                       STATE    PGS  REPLACE  FORCE  ZAP   DRAIN STARTED AT  
5    ms-01-02.storage.pcfe.net  started   18  False    False  True                    
6    ms-01-03.storage.pcfe.net  started   18  False    False  True                    
7    ms-01-05.storage.pcfe.net  started   18  False    False  True                    
8    ms-01-01.storage.pcfe.net  started   20  False    False  True                    
9    ms-01-04.storage.pcfe.net  started   18  False    False  True                    
[ceph: root@ms-01-01 /]# 
[ceph: root@ms-01-01 /]# ceph log last cephadm ; ceph -W cephadm
[…]
2025-03-08T11:51:10.129970+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53411 : cephadm [INF] osd.8 crush weight is 1.81939697265625
2025-03-08T11:51:10.133867+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53412 : cephadm [INF] osd.5 crush weight is 1.81939697265625
2025-03-08T11:51:10.137239+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53413 : cephadm [INF] osd.6 crush weight is 1.81939697265625
2025-03-08T11:51:10.140274+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53414 : cephadm [INF] osd.9 crush weight is 1.81939697265625
2025-03-08T11:51:10.143226+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53415 : cephadm [INF] osd.7 crush weight is 1.81939697265625
2025-03-08T11:51:17.802807+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53444 : cephadm [INF] osd.9 weight is now 0.0
  cluster:
    id:     cee8622c-fb45-11ef-9938-5847ca797c06
    health: HEALTH_OK
 
  services:
    mon: 5 daemons, quorum ms-01-01,ms-01-02,ms-01-05,ms-01-03,ms-01-04 (age 20h)
    mgr: ms-01-02.yuzhvl(active, since 21h), standbys: ms-01-01.pnupxa
    osd: 10 osds: 10 up (since 15m), 10 in (since 20h); 25 remapped pgs
 
  data:
    pools:   3 pools, 65 pgs
    objects: 30.58k objects, 119 GiB
    usage:   370 GiB used, 18 TiB / 18 TiB avail
    pgs:     21144/91746 objects misplaced (23.046%)
             40 active+clean
             20 active+remapped+backfill_wait
             5  active+remapped+backfilling
 
  io:
    recovery: 294 MiB/s, 73 objects/s
 

2025-03-08T11:52:21.699945+0000 mgr.ms-01-02.yuzhvl [INF] osd.8 weight is now 0.0
2025-03-08T11:54:34.358011+0000 mgr.ms-01-02.yuzhvl [INF] osd.7 weight is now 0.0
2025-03-08T11:55:37.250217+0000 mgr.ms-01-02.yuzhvl [INF] osd.6 weight is now 0.0
2025-03-08T11:56:38.679752+0000 mgr.ms-01-02.yuzhvl [INF] osd.5 weight is now 0.0
2025-03-08T12:12:15.978654+0000 mgr.ms-01-02.yuzhvl [INF] osd.9 now down
2025-03-08T12:12:15.978804+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.9 from ms-01-04.storage.pcfe.net -- ports []
2025-03-08T12:12:18.509455+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.9
2025-03-08T12:12:18.526608+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.9 on ms-01-04.storage.pcfe.net
2025-03-08T12:12:18.541942+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.9 on ms-01-04.storage.pcfe.net
2025-03-08T12:12:18.541979+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.9 on ms-01-04.storage.pcfe.net
2025-03-08T12:12:21.000208+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.9 on ms-01-04.storage.pcfe.net
2025-03-08T12:12:24.896547+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-04.storage.pcfe.net
2025-03-08T12:12:44.852913+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.9 on ms-01-04.storage.pcfe.net
2025-03-08T12:13:05.216192+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-04.storage.pcfe.net
2025-03-08T12:22:09.446113+0000 mgr.ms-01-02.yuzhvl [INF] osd.7 now down
2025-03-08T12:22:09.446277+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.7 from ms-01-05.storage.pcfe.net -- ports []
2025-03-08T12:22:11.919234+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.7
2025-03-08T12:22:11.936494+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.7 on ms-01-05.storage.pcfe.net
2025-03-08T12:22:11.952321+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.7 on ms-01-05.storage.pcfe.net
2025-03-08T12:22:11.952350+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.7 on ms-01-05.storage.pcfe.net
2025-03-08T12:22:14.389828+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.7 on ms-01-05.storage.pcfe.net
2025-03-08T12:22:18.292163+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-05.storage.pcfe.net
2025-03-08T12:22:40.156873+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.7 on ms-01-05.storage.pcfe.net
2025-03-08T12:23:01.152178+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-05.storage.pcfe.net
2025-03-08T12:25:17.113831+0000 mgr.ms-01-02.yuzhvl [INF] osd.8 now down
2025-03-08T12:25:17.113965+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.8 from ms-01-01.storage.pcfe.net -- ports []
2025-03-08T12:25:19.572978+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.8
2025-03-08T12:25:19.586673+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.8 on ms-01-01.storage.pcfe.net
2025-03-08T12:25:19.601872+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.8 on ms-01-01.storage.pcfe.net
2025-03-08T12:25:19.601901+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.8 on ms-01-01.storage.pcfe.net
2025-03-08T12:25:21.987384+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.8 on ms-01-01.storage.pcfe.net
2025-03-08T12:25:22.598562+0000 mgr.ms-01-02.yuzhvl [INF] osd.5 now down
2025-03-08T12:25:22.598706+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.5 from ms-01-02.storage.pcfe.net -- ports []
2025-03-08T12:25:25.216537+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.5
2025-03-08T12:25:25.232876+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.5 on ms-01-02.storage.pcfe.net
2025-03-08T12:25:25.248294+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.5 on ms-01-02.storage.pcfe.net
2025-03-08T12:25:25.248325+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.5 on ms-01-02.storage.pcfe.net
2025-03-08T12:25:27.666190+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.5 on ms-01-02.storage.pcfe.net
2025-03-08T12:25:28.680404+0000 mgr.ms-01-02.yuzhvl [INF] osd.6 now down
2025-03-08T12:25:28.680544+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.6 from ms-01-03.storage.pcfe.net -- ports []
2025-03-08T12:25:31.282911+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.6
2025-03-08T12:25:31.300272+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.6 on ms-01-03.storage.pcfe.net
2025-03-08T12:25:31.317684+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.6 on ms-01-03.storage.pcfe.net
2025-03-08T12:25:31.317714+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.6 on ms-01-03.storage.pcfe.net
2025-03-08T12:25:33.716134+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.6 on ms-01-03.storage.pcfe.net
2025-03-08T12:25:37.583541+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-02.storage.pcfe.net
2025-03-08T12:25:38.037671+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-03.storage.pcfe.net
2025-03-08T12:25:39.588782+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-01.storage.pcfe.net
2025-03-08T12:25:55.777847+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.8 on ms-01-03.storage.pcfe.net
2025-03-08T12:25:56.142570+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.6 on ms-01-02.storage.pcfe.net
2025-03-08T12:25:56.356299+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.5 on ms-01-01.storage.pcfe.net
2025-03-08T12:26:16.769890+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-03.storage.pcfe.net
2025-03-08T12:26:16.972150+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-02.storage.pcfe.net
2025-03-08T12:26:17.467832+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-01.storage.pcfe.net
Standard procedure for custom drive spec

If I’d started with empty drives to use for OSDs, then this would just have been the commands

  • cephadm shell --mount <my YAML>
  • ceph orch apply -i ms-01-SSD-as-OSD-spec.yml (maybe first a --dry-run though)
[root@ms-01-01 ~]#  cephadm shell --mount ms-01-SSD-as-OSD-spec.yml:/var/lib/ceph/osd/ms-01-SSD-as-OSD-spec.yml
[ceph: root@ms-01-01 /]# cd /var/lib/ceph/osd/
[ceph: root@ms-01-01 osd]# ls
ms-01-SSD-as-OSD-spec.yml
[ceph: root@ms-01-01 osd]# cat ms-01-SSD-as-OSD-spec.yml 
service_type: osd
service_id: osd_spec_MS-01_SSDs
placement:
  host_pattern: 'ms-01-*'
encrypted: true
data_devices:
  rotational: 0
[ceph: root@ms-01-01 osd]# ceph orch apply -i ms-01-SSD-as-OSD-spec.yml 
Scheduled osd.osd_spec_MS-01_SSDs update...

Check that OSDs are crypted

After the above song and dance, of course I double checked. All OSDs are now crypted at rest. Meaning that if I wear one out, I can remove it drom the node and throw it into recycling without worrying about anyone getting access to the data stored on that device.

Obviously only as long as I do not thror a not encrypted BaseOS storage of the node into the same recycling run.

NOTE:: this obviously does not protect the data in case of theft/recyclint/sale of a whole node, the point here is really just peace of mind when I physically remove an OSD storage device from a cluster node mostly because I wore it out.

Click to see lsbkl output of all 5 nodes
pcfe@t3600 ~ $ for i in $(seq 1 5) ; do echo MS-01 number ${i} ; ssh ms-01-0${i} -l root lsblk ;echo ; done
MS-01 number 1
NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1                                                                                               259:0    0   1.9T  0 disk  
└─ceph--39fa7697--930a--4e88--8280--c82f04bd4b42-osd--block--6af44afd--e59e--4ce3--8906--bf790f4a8e4d 253:2    0   1.9T  0 lvm   
  └─TyyWnn-bjFU-X7ua-Asd5-sKin-Jsh3-w5EJX4                                                            253:9    0   1.9T  0 crypt 
nvme1n1                                                                                               259:1    0   1.8T  0 disk  
└─ceph--394db331--20da--401c--84c5--50397547d3c3-osd--block--e47aef1e--585c--4555--95a8--f8a06943097c 253:3    0   1.8T  0 lvm   
  └─Appi83-m73V-FjvK-pLJH-J8Ob-HsN0-uJpIaZ                                                            253:10   0   1.8T  0 crypt 
nvme2n1                                                                                               259:2    0 953.9G  0 disk  
├─nvme2n1p1                                                                                           259:3    0   600M  0 part  /boot/efi
├─nvme2n1p2                                                                                           259:4    0     1G  0 part  /boot
└─nvme2n1p3                                                                                           259:5    0   200G  0 part  
  ├─VG_OS-LV_root                                                                                     253:0    0     4G  0 lvm   /
  ├─VG_OS-LV_swap                                                                                     253:1    0     1G  0 lvm   [SWAP]
  ├─VG_OS-LV_var_crash                                                                                253:4    0    40G  0 lvm   /var/crash
  ├─VG_OS-LV_var                                                                                      253:5    0    12G  0 lvm   /var
  ├─VG_OS-LV_home                                                                                     253:6    0     1G  0 lvm   /home
  ├─VG_OS-LV_var_log                                                                                  253:7    0    10G  0 lvm   /var/log
  └─VG_OS-LV_containers                                                                               253:8    0    10G  0 lvm   /var/lib/containers/storage/overlay
                                                                                                                                 /var/lib/containers

MS-01 number 2
NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme2n1                                                                                               259:0    0   1.8T  0 disk  
└─ceph--487b4a4c--b0b3--4a81--8f03--4f10c70be9d9-osd--block--ffbfb311--08e7--4d25--a397--ceea88027638 253:8    0   1.8T  0 lvm   
  └─hBVzZT-KBP3-okfr-UUxw-zIQb-9Nxo-HRMYBD                                                            253:10   0   1.8T  0 crypt 
nvme1n1                                                                                               259:1    0   1.8T  0 disk  
└─ceph--8d3763d6--e03a--4443--a1c5--0ab11b2d7f6b-osd--block--aba0fdee--23fd--4672--8cd6--65a61c082b8c 253:7    0   1.8T  0 lvm   
  └─zEGV9m-4iUT-p3RH-vaCX-oNFx-lpVv-SM9AUa                                                            253:9    0   1.8T  0 crypt 
nvme0n1                                                                                               259:2    0 953.9G  0 disk  
├─nvme0n1p1                                                                                           259:3    0   600M  0 part  /boot/efi
├─nvme0n1p2                                                                                           259:4    0     1G  0 part  /boot
└─nvme0n1p3                                                                                           259:5    0   200G  0 part  
  ├─VG_OS-LV_root                                                                                     253:0    0     4G  0 lvm   /
  ├─VG_OS-LV_swap                                                                                     253:1    0     1G  0 lvm   [SWAP]
  ├─VG_OS-LV_var_crash                                                                                253:2    0    40G  0 lvm   /var/crash
  ├─VG_OS-LV_var                                                                                      253:3    0    12G  0 lvm   /var
  ├─VG_OS-LV_home                                                                                     253:4    0     1G  0 lvm   /home
  ├─VG_OS-LV_var_log                                                                                  253:5    0    10G  0 lvm   /var/log
  └─VG_OS-LV_containers                                                                               253:6    0    10G  0 lvm   /var/lib/containers/storage/overlay
                                                                                                                                 /var/lib/containers

MS-01 number 3
NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1                                                                                               259:0    0   1.8T  0 disk  
└─ceph--9930a217--399a--452d--9089--110255862676-osd--block--88f72a42--f96d--4463--9be1--2a3e0f980212 253:7    0   1.8T  0 lvm   
  └─IKDjmd-Tmno-TPvv-zwDk-IaVz-ZclB-8lpdhA                                                            253:9    0   1.8T  0 crypt 
nvme1n1                                                                                               259:1    0   1.8T  0 disk  
└─ceph--a20aa91d--cea4--42f8--b1d1--0faab214a74d-osd--block--6f962829--1f98--4cb5--afc6--ba29e5367475 253:8    0   1.8T  0 lvm   
  └─CuKsDQ-EUhh-osRb-hVdC-syPC-p3lz-coVA4K                                                            253:10   0   1.8T  0 crypt 
nvme2n1                                                                                               259:2    0 953.9G  0 disk  
├─nvme2n1p1                                                                                           259:3    0   600M  0 part  /boot/efi
├─nvme2n1p2                                                                                           259:4    0     1G  0 part  /boot
└─nvme2n1p3                                                                                           259:5    0   200G  0 part  
  ├─VG_OS-LV_root                                                                                     253:0    0     4G  0 lvm   /
  ├─VG_OS-LV_swap                                                                                     253:1    0     1G  0 lvm   [SWAP]
  ├─VG_OS-LV_var_crash                                                                                253:2    0    40G  0 lvm   /var/crash
  ├─VG_OS-LV_var                                                                                      253:3    0    12G  0 lvm   /var
  ├─VG_OS-LV_home                                                                                     253:4    0     1G  0 lvm   /home
  ├─VG_OS-LV_var_log                                                                                  253:5    0    10G  0 lvm   /var/log
  └─VG_OS-LV_containers                                                                               253:6    0    10G  0 lvm   /var/lib/containers/storage/overlay
                                                                                                                                 /var/lib/containers

MS-01 number 4
NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme2n1                                                                                               259:0    0   1.8T  0 disk  
└─ceph--5ac5826d--1ae5--4c18--a3b8--982d5e3cb33f-osd--block--994ef9b8--c453--4f61--9c91--8df05ac9ddd3 253:8    0   1.8T  0 lvm   
  └─mWGrQW-SEOu-i31F-DzrU-yEPv-xz2g-H28f3l                                                            253:10   0   1.8T  0 crypt 
nvme1n1                                                                                               259:1    0 953.9G  0 disk  
├─nvme1n1p1                                                                                           259:2    0   600M  0 part  /boot/efi
├─nvme1n1p2                                                                                           259:3    0     1G  0 part  /boot
└─nvme1n1p3                                                                                           259:4    0   200G  0 part  
  ├─VG_OS-LV_root                                                                                     253:0    0     4G  0 lvm   /
  ├─VG_OS-LV_swap                                                                                     253:1    0     1G  0 lvm   [SWAP]
  ├─VG_OS-LV_var_crash                                                                                253:2    0    40G  0 lvm   /var/crash
  ├─VG_OS-LV_var                                                                                      253:3    0    12G  0 lvm   /var
  ├─VG_OS-LV_home                                                                                     253:4    0     1G  0 lvm   /home
  ├─VG_OS-LV_var_log                                                                                  253:5    0    10G  0 lvm   /var/log
  └─VG_OS-LV_containers                                                                               253:6    0    10G  0 lvm   /var/lib/containers/storage/overlay
                                                                                                                                 /var/lib/containers
nvme0n1                                                                                               259:5    0   1.9T  0 disk  
└─ceph--ee5e0a69--45c3--4e49--8b6e--0023c1c16257-osd--block--67b39a1b--6b31--4daf--9fd2--ec70a734624d 253:7    0   1.9T  0 lvm   
  └─9aHi7q-XIep-d50G-ejp5-0XIg-GiaS-N2yrWE                                                            253:9    0   1.9T  0 crypt 

MS-01 number 5
NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme1n1                                                                                               259:0    0   1.8T  0 disk  
└─ceph--2491bd1c--43e8--4476--b6cf--b41c00d6699e-osd--block--d7617b6e--5435--4a8e--9bb3--4260c7ccc9d8 253:8    0   1.8T  0 lvm   
  └─4BW01s-srfz-kLEw-y3Vq-iMiW-xJ0Z-Hup5Rb                                                            253:10   0   1.8T  0 crypt 
nvme0n1                                                                                               259:1    0   1.8T  0 disk  
└─ceph--e538ff2e--7854--4735--a9c3--be1aa5e04668-osd--block--f4a7f625--aa52--4c61--be87--a182252927cc 253:5    0   1.8T  0 lvm   
  └─NyhkOy-bB2Z-f9aT-YgY1-5z3d-MM2C-7zFMz3                                                            253:9    0   1.8T  0 crypt 
nvme2n1                                                                                               259:2    0 953.9G  0 disk  
├─nvme2n1p1                                                                                           259:3    0   600M  0 part  /boot/efi
├─nvme2n1p2                                                                                           259:4    0     1G  0 part  /boot
└─nvme2n1p3                                                                                           259:5    0   200G  0 part  
  ├─VG_OS-LV_root                                                                                     253:0    0     4G  0 lvm   /
  ├─VG_OS-LV_swap                                                                                     253:1    0     1G  0 lvm   [SWAP]
  ├─VG_OS-LV_var_crash                                                                                253:2    0    40G  0 lvm   /var/crash
  ├─VG_OS-LV_var                                                                                      253:3    0    12G  0 lvm   /var
  ├─VG_OS-LV_home                                                                                     253:4    0     1G  0 lvm   /home
  ├─VG_OS-LV_var_log                                                                                  253:6    0    10G  0 lvm   /var/log
  └─VG_OS-LV_containers                                                                               253:7    0    10G  0 lvm   /var/lib/containers/storage/overlay
                                                                                                                                 /var/lib/containers

pcfe@t3600 ~ $ 

MGR redirects to FQDN

I want this setting enabled so that when I access e.g. ms-01-01 but the dashboard is currently running on -02, that I get redirected to a fully qualified domain name, not an IP.

This makes my browser happier for TLS verification and it make my life easier WRT readability.

I honestly do not have the foggiest WTH this is not the default.

[ceph: root@ms-01-01 /]# ceph config set mgr mgr/dashboard/redirect_resolve_ip_addr True

Performance baseline

The poiint here is to get numbers to compare to if I ever make changes. e.g. take the public_network interfaces from 1 Gigabity to 2.5 Gigabit if I ever get a 2.5 capable switch.

See also https://docs.redhat.com/en/documentation/red_hat_ceph_storage/8/html/administration_guide/ceph-performance-benchmark#performance-baseline_admin

[ceph: root@ms-01-01 /]# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 10.00
pool 2 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 389 lfor 0/385/383 flags hashpspool stripe_width 0 application rbd read_balance_score 1.56
pool 3 'rbdbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 397 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88
pool 4 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 2630 lfor 0/0/2600 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 3.76
pool 5 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 2606 lfor 0/0/2604 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.17

[ceph: root@ms-01-01 /]# 

Rados, sequential write test, pool with 32 PGs

[ceph: root@ms-01-01 /]# date
Sat Sep  6 15:38:15 UTC 2025
[ceph: root@ms-01-01 /]# rados bench -p testbench 900 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 900 seconds or 0 objects
Object prefix: benchmark_data_ms-01-01.storage.pcfe.net_47
Click to see full run
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        46        30   119.986       120    0.891861    0.283605
    2      16        81        65   129.986       140    0.728704    0.373012
    3      16       110        94    125.32       116    0.357917    0.413796
    4      16       147       131   130.986       148    0.151139    0.427921
    5      16       179       163   130.387       128   0.0169029    0.460131
    6      16       210       194   129.319       124    0.121417    0.456936
    7      16       242       226   129.129       128    0.862833    0.473473
    8      16       273       257   128.486       124     1.16294    0.473367
    9      16       304       288   127.986       124   0.0874029    0.475227
   10      16       336       320   127.986       128   0.0159942    0.473775
   11      16       367       351   127.623       124   0.0881047    0.480436
   12      16       397       381   126.987       120     1.24982    0.482688
   13      16       429       413   127.064       128    0.241703     0.48436
   14      16       459       443   126.558       120     0.12791    0.484436
   15      16       495       479    127.72       144   0.0896826    0.480814
   16      16       527       511   127.737       128   0.0123545    0.478949
   17      16       563       547   128.691       144    0.182025    0.476148
   18      16       595       579   128.652       128    0.275573    0.479161
   19      16       625       609   128.196       120    0.247677    0.481757
2025-09-06T15:39:18.769105+0000 min lat: 0.0120121 max lat: 2.16865 avg lat: 0.48376
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       655       639   127.786       120    0.186276     0.48376
   21      16       685       669   127.415       120     0.29716    0.483704
   22      16       717       701   127.441       128   0.0879459    0.486324
   23      16       745       729   126.769       112     1.52949    0.492499
   24      16       780       764   127.319       140     0.15574    0.489639
   25      16       809       793   126.866       116     1.76825    0.493005
   26      16       842       826   127.063       132     1.79154    0.491684
   27      16       873       857    126.95       124   0.0183032    0.490702
   28      16       906       890    127.13       132    0.706151    0.490792
   29      16       938       922   127.159       128    0.582968    0.493053
   30      16       973       957   127.587       140   0.0132536    0.491336
   31      16      1004       988   127.471       124    0.169061    0.494953
   32      16      1033      1017   127.112       116   0.0919299    0.494588
   33      16      1068      1052   127.502       140     0.15031    0.492852
   34      16      1098      1082   127.281       120   0.0898387    0.491945
   35      16      1131      1115   127.415       132    0.139664    0.492299
   36      16      1166      1150   127.765       140    0.573958    0.494613
   37      16      1199      1183   127.879       132    0.128009    0.493121
   38      16      1228      1212   127.566       116   0.0936423    0.492605
   39      16      1259      1243   127.474       124     1.80107     0.49548
2025-09-06T15:39:38.770982+0000 min lat: 0.0115416 max lat: 2.50556 avg lat: 0.493221
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1296      1280   127.987       148      1.5973    0.493221
   41      16      1327      1311   127.889       124    0.213491    0.494597
   42      16      1357      1341   127.701       120   0.0147104    0.494833
   43      16      1389      1373   127.708       128   0.0211331     0.49667
   44      16      1426      1410   128.169       148   0.0949964    0.494086
   45      16      1455      1439   127.898       116    0.155738    0.494075
   46      16      1484      1468   127.639       116     0.34259    0.496389
   47      16      1513      1497   127.391       116    0.132117    0.495907
   48      16      1548      1532   127.654       140   0.0129881    0.494806
   49      16      1580      1564    127.66       128   0.0882796    0.496133
   50      16      1614      1598   127.827       136   0.0916736    0.495361
   51      16      1647      1631   127.908       132     1.11173    0.495569
   52      16      1677      1661   127.756       120    0.507702    0.496079
   53      16      1709      1693    127.76       128    0.129273    0.497577
   54      16      1740      1724   127.691       124   0.0254375    0.495974
   55      16      1772      1756   127.696       128   0.0227182      0.4963
   56      16      1806      1790   127.844       136    0.329753    0.496575
   57      16      1837      1821   127.776       124   0.0167499    0.495306
   58      16      1872      1856   127.987       140    0.196348    0.494583
   59      16      1905      1889   128.055       132    0.322432     0.49442
2025-09-06T15:39:58.773032+0000 min lat: 0.0112911 max lat: 2.50556 avg lat: 0.494783
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1936      1920   127.987       124   0.0991983    0.494783
   61      16      1973      1957   128.315       148   0.0123943    0.494378
   62      16      2005      1989    128.31       128     1.01152     0.49553
   63      16      2037      2021   128.304       128    0.127416      0.4931
   64      16      2072      2056   128.487       140   0.0861447    0.491937
   65      16      2108      2092   128.725       144     1.97594    0.492938
   66      16      2139      2123   128.653       124    0.996572    0.493387
   67      16      2169      2153   128.524       120    0.897124    0.495383
   68      16      2197      2181   128.281       112    0.721435    0.495552
   69      16      2226      2210   128.103       116      1.3611     0.49589
   70      16      2260      2244   128.215       136     1.05833    0.496308
   71      16      2287      2271   127.931       108   0.0877596    0.497459
   72      16      2319      2303   127.931       128     1.23147    0.496159
   73      16      2354      2338   128.097       140     1.76149    0.495753
   74      16      2389      2373   128.257       140   0.0179643    0.495055
   75      16      2423      2407    128.36       136   0.0859759    0.494071
   76      16      2451      2435   128.145       112   0.0231612    0.494963
   77      16      2480      2464   127.987       116    0.419215    0.495413
   78      16      2515      2499   128.141       140     1.77961     0.49505
   79      16      2548      2532   128.189       132    0.433241    0.495388
2025-09-06T15:40:18.775246+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.49461
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2581      2565   128.237       132    0.139117     0.49461
   81      16      2615      2599   128.332       136   0.0945735     0.49366
   82      16      2647      2631   128.328       128    0.190526    0.493379
   83      16      2677      2661   128.227       120    0.380594    0.494329
   84      16      2706      2690   128.082       116   0.0924989    0.495199
   85      16      2743      2727   128.316       148   0.0829465    0.494778
   86      16      2774      2758   128.266       124     2.22245     0.49439
   87      16      2803      2787   128.124       116     0.36343    0.495131
   88      16      2835      2819   128.123       128    0.368028    0.495404
   89      16      2873      2857   128.391       152    0.275797    0.495344
   90      16      2902      2886   128.253       116    0.156844    0.495956
   91      16      2933      2917   128.206       124     1.61382    0.495995
   92      16      2964      2948    128.16       124    0.200034    0.496516
   93      16      2994      2978   128.073       120     0.74711    0.497577
   94      16      3024      3008   127.986       120   0.0832374    0.497393
   95      16      3058      3042   128.071       136    0.319327    0.497268
   96      16      3089      3073   128.028       124    0.998517    0.497572
   97      16      3122      3106   128.069       132     0.36057    0.497108
   98      16      3148      3132   127.823       104     1.19455     0.49836
   99      16      3180      3164   127.825       128    0.460973    0.498516
2025-09-06T15:40:38.777483+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.498469
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3214      3198   127.906       136    0.190703    0.498469
  101      16      3244      3228   127.828       120   0.0891681    0.498488
  102      16      3279      3263   127.947       140     1.01252    0.498182
  103      16      3309      3293    127.87       120    0.755003    0.497799
  104      16      3341      3325   127.871       128    0.329617    0.498093
  105      16      3371      3355   127.796       120    0.462132    0.497717
  106      16      3405      3389   127.873       136   0.0921742    0.497935
  107      16      3433      3417   127.725       112   0.0923818    0.498832
  108      16      3462      3446   127.616       116     1.41373    0.498909
  109      16      3493      3477   127.583       124    0.280848    0.498269
  110      16      3525      3509   127.586       128    0.198916     0.49795
  111      16      3554      3538   127.482       116    0.085032    0.498829
  112      16      3586      3570   127.486       128    0.198039    0.498902
  113      16      3614      3598   127.349       112    0.248161    0.498954
  114      16      3650      3634   127.495       144   0.0151987    0.498726
  115      16      3678      3662    127.36       112    0.337411    0.499597
  116      16      3715      3699   127.538       148    0.101546    0.498792
  117      16      3745      3729   127.474       120    0.139778    0.498472
  118      16      3776      3760   127.444       124   0.0985602    0.499527
  119      16      3807      3791   127.415       124   0.0877263    0.499744
2025-09-06T15:40:58.779686+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.499195
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      3837      3821   127.353       120    0.143885    0.499195
  121      16      3869      3853   127.358       128    0.145949    0.499272
  122      16      3901      3885   127.364       128     2.06391    0.499918
  123      16      3931      3915   127.304       120    0.105985    0.500572
  124      16      3970      3954   127.535       156   0.0309444    0.500174
  125      16      4005      3989   127.634       140    0.536079    0.499227
  126      16      4037      4021   127.637       128     0.56835    0.499251
  127      16      4070      4054   127.671       132    0.257916    0.499163
  128      16      4106      4090   127.799       144    0.194968    0.498771
  129      16      4137      4121   127.769       124    0.281637    0.499064
  130      16      4166      4150   127.679       116    0.233604    0.499265
  131      16      4197      4181    127.65       124    0.422676    0.499544
  132      16      4229      4213   127.653       128    0.136296    0.499425
  133      16      4262      4246   127.685       132    0.165794    0.499302
  134      16      4297      4281   127.777       140    0.611767    0.499259
  135      16      4330      4314   127.808       132     1.05761    0.499475
  136      16      4360      4344   127.751       120   0.0177166    0.499404
  137      16      4390      4374   127.694       120    0.169101     0.49937
  138      16      4425      4409   127.783       140     1.37626    0.498999
  139      16      4456      4440   127.756       124   0.0868818    0.498992
2025-09-06T15:41:18.781998+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.4981
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      4488      4472   127.758       128    0.146172      0.4981
  141      16      4523      4507   127.844       140     2.37268    0.498882
  142      16      4553      4537   127.789       120    0.149856     0.49874
  143      16      4584      4568   127.762       124   0.0153841    0.498249
  144      16      4614      4598   127.708       120    0.189612    0.498561
  145      16      4644      4628   127.655       120    0.087014    0.498361
  146      16      4674      4658   127.603       120    0.101273    0.498314
  147      16      4704      4688   127.551       120    0.349187    0.499679
  148      16      4732      4716   127.446       112    0.932191    0.500301
  149      16      4769      4753   127.584       148   0.0166335    0.500164
  150      16      4802      4786   127.613       132   0.0146001    0.499685
  151      16      4832      4816   127.562       120     2.05424    0.500096
  152      16      4864      4848   127.565       128   0.0185872    0.500373
  153      16      4891      4875   127.437       108     1.15425    0.499997
  154      16      4929      4913   127.597       152    0.597443    0.500299
  155      16      4959      4943   127.548       120   0.0184109    0.500234
  156      16      4991      4975    127.55       128    0.647753    0.500019
  157      16      5022      5006   127.528       124   0.0846472     0.49948
  158      16      5054      5038   127.531       128   0.0179906    0.499787
  159      16      5084      5068   127.483       120     1.78151    0.500272
2025-09-06T15:41:38.784176+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.50063
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      5112      5096   127.386       112    0.229029     0.50063
  161      16      5147      5131   127.464       140    0.205066    0.500724
  162      16      5181      5165   127.517       136    0.139766     0.49996
  163      16      5214      5198   127.544       132   0.0151517    0.499815
  164      16      5245      5229   127.523       124     0.18035    0.499337
  165      16      5279      5263   127.574       136    0.212684    0.499407
  166      16      5314      5298   127.649       140   0.0900562    0.498887
  167      16      5349      5333   127.723       140    0.161326    0.498775
  168      16      5385      5369    127.82       144       2.114    0.499046
  169      16      5418      5402   127.844       132     1.50679    0.498985
  170      16      5450      5434   127.845       128   0.0835925    0.498632
  171      16      5481      5465   127.822       124     2.17135    0.498477
  172      16      5509      5493    127.73       112    0.461062    0.499395
  173      16      5540      5524   127.709       124    0.170932    0.499317
  174      16      5570      5554   127.664       120     1.72015    0.499776
  175      16      5605      5589   127.735       140    0.016511    0.499746
  176      16      5635      5619   127.691       120   0.0188613    0.499901
  177      16      5670      5654    127.76       140   0.0262272    0.499686
  178      16      5705      5689   127.829       140   0.0185254    0.499313
  179      16      5737      5721    127.83       128   0.0142448    0.499583
2025-09-06T15:41:58.786375+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.499317
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      5771      5755   127.875       136   0.0949172    0.499317
  181      16      5803      5787   127.876       128   0.0187945     0.49914
  182      16      5838      5822   127.942       140    0.403885     0.49917
  183      16      5871      5855   127.964       132   0.0830073    0.498966
  184      16      5902      5886   127.943       124   0.0830302    0.499013
  185      16      5934      5918   127.943       128    0.868037     0.49916
  186      16      5965      5949   127.921       124     1.06085    0.499356
  187      16      5996      5980     127.9       124   0.0854325    0.499033
  188      16      6026      6010   127.858       120    0.206039    0.498886
  189      16      6057      6041   127.838       124   0.0976375     0.49875
  190      16      6090      6074    127.86       132    0.513917    0.498319
  191      16      6122      6106    127.86       128   0.0880232    0.498888
  192      16      6154      6138   127.861       128     1.73397    0.499036
  193      16      6183      6167     127.8       116    0.267093    0.499223
  194      16      6221      6205   127.924       152     1.63174    0.498967
  195      16      6254      6238   127.945       132    0.533932    0.498767
  196      16      6290      6274   128.027       144    0.101415    0.498503
  197      16      6319      6303   127.966       116   0.0139238    0.498474
  198      16      6353      6337   128.006       136     1.13811    0.499147
  199      16      6383      6367   127.966       120    0.536272    0.499058
2025-09-06T15:42:18.788755+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.498567
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      6414      6398   127.946       124    0.301265    0.498567
  201      16      6445      6429   127.926       124    0.568706    0.498866
  202      16      6485      6469   128.085       160    0.192172    0.498296
  203      16      6517      6501   128.085       128    0.460974    0.498558
  204      16      6546      6530   128.025       116     1.01028    0.498661
  205      16      6576      6560   127.986       120   0.0855708    0.498712
  206      16      6605      6589   127.928       116    0.621335    0.498904
  207      16      6643      6627   128.044       152    0.227024    0.498146
  208      16      6672      6656   127.986       116    0.119048    0.498013
  209      16      6710      6694   128.101       152   0.0153679     0.49794
  210      16      6743      6727   128.119       132    0.738219    0.498168
  211      16      6772      6756   128.062       116    0.100046    0.498637
  212      16      6806      6790   128.099       136    0.754633    0.498487
  213      16      6840      6824   128.136       136    0.147949    0.497936
  214      16      6873      6857   128.154       132   0.0148063    0.498241
  215      16      6905      6889   128.154       128    0.214634    0.498032
  216      16      6939      6923    128.19       136    0.226452    0.498139
  217      16      6970      6954    128.17       124     1.42773    0.498302
  218      16      7002      6986    128.17       128    0.170738    0.497888
  219      16      7032      7016   128.132       120     1.66462    0.498185
2025-09-06T15:42:38.790733+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.498067
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      7064      7048   128.132       128     1.31962    0.498067
  221      16      7094      7078   128.095       120    0.278874    0.498398
  222      16      7123      7107    128.04       116     0.11889    0.498375
  223      16      7156      7140   128.058       132     1.75955    0.498411
  224      16      7184      7168   127.986       112    0.629593    0.498789
  225      16      7219      7203   128.039       140    0.157019    0.498765
  226      16      7252      7236   128.057       132    0.163003    0.498666
  227      16      7283      7267   128.039       124    0.463427    0.498789
  228      16      7318      7302   128.091       140    0.585858    0.498435
  229      16      7354      7338   128.161       144    0.242101    0.498385
  230      16      7385      7369   128.143       124    0.222028    0.498215
  231      16      7414      7398    128.09       116    0.280185    0.498316
  232      16      7447      7431   128.107       132    0.343408    0.498583
  233      16      7479      7463   128.106       128     1.14125    0.498413
  234      16      7506      7490    128.02       108   0.0839596    0.498283
  235      16      7537      7521   128.003       124    0.411275    0.498843
  236      16      7570      7554    128.02       132     1.71495    0.498826
  237      16      7602      7586    128.02       128    0.546866    0.498875
  238      16      7633      7617   128.003       124    0.616218    0.498814
  239      16      7666      7650    128.02       132    0.808912    0.498968
2025-09-06T15:42:58.792669+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.499115
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      7697      7681   128.003       124     1.01153    0.499115
  241      16      7729      7713   128.003       128    0.169586    0.499001
  242      16      7763      7747   128.036       136   0.0836757    0.498608
  243      16      7797      7781   128.069       136    0.974431     0.49876
  244      16      7828      7812   128.052       124   0.0933906    0.498493
  245      16      7862      7846   128.084       136   0.0264225    0.498804
  246      16      7896      7880   128.116       136     1.35708    0.498611
  247      16      7932      7916   128.181       144    0.953986    0.498218
  248      16      7961      7945   128.131       116     0.53038    0.498478
  249      16      7992      7976   128.115       124   0.0904629    0.498187
  250      16      8022      8006   128.082       120     1.44914    0.498691
  251      16      8055      8039   128.098       132    0.478966    0.498668
  252      16      8083      8067   128.034       112     1.27091    0.498785
  253      16      8121      8105   128.129       152    0.771445    0.498566
  254      16      8157      8141   128.191       144    0.514055    0.498057
  255      16      8195      8179   128.284       152    0.860361    0.497881
  256      16      8229      8213   128.314       136   0.0925479    0.497722
  257      16      8263      8247   128.344       136   0.0913154    0.497788
  258      16      8296      8280   128.358       132   0.0166663    0.497453
  259      16      8326      8310   128.326       120    0.740491    0.497972
2025-09-06T15:43:18.794988+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.497889
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      8356      8340   128.294       120      1.3518    0.497889
  261      16      8386      8370   128.262       120   0.0823364     0.49791
  262      16      8417      8401   128.246       124     1.54553     0.49786
  263      16      8450      8434    128.26       132   0.0838941    0.497792
  264      16      8482      8466   128.259       128     1.72307    0.497549
  265      16      8515      8499   128.273       132    0.356187    0.497651
  266      16      8543      8527   128.212       112   0.0153906    0.497953
  267      16      8581      8565   128.301       152    0.298913    0.497681
  268      16      8620      8604   128.404       156   0.0991551    0.497596
  269      16      8651      8635   128.388       124   0.0174738     0.49748
  270      16      8684      8668   128.401       132    0.632525    0.497568
  271      16      8720      8704   128.458       144   0.0179211    0.497476
  272      16      8752      8736   128.457       128    0.355126    0.497472
  273      16      8784      8768   128.455       128    0.083615     0.49719
  274      16      8816      8800   128.453       128    0.404962    0.497328
  275      16      8853      8837   128.524       148   0.0822306    0.497097
  276      16      8884      8868   128.508       124    0.137836    0.497125
  277      16      8915      8899   128.492       124    0.938899    0.497151
  278      16      8946      8930   128.475       124     1.57643    0.497049
  279      16      8974      8958   128.416       112    0.201861    0.497013
2025-09-06T15:43:38.797064+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.497314
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      9007      8991   128.429       132    0.267373    0.497314
  281      16      9037      9021   128.399       120    0.230352    0.497376
  282      16      9075      9059   128.483       152     1.33239    0.497304
  283      16      9104      9088   128.438       116    0.119168    0.497561
  284      16      9137      9121   128.451       132    0.627274    0.497566
  285      16      9167      9151   128.421       120    0.192958    0.497364
  286      16      9200      9184   128.434       132    0.158574    0.497227
  287      16      9234      9218    128.46       136     0.16397    0.497378
  288      16      9264      9248   128.431       120    0.729445    0.497506
  289      16      9296      9280   128.429       128     1.05838    0.497642
  290      16      9328      9312   128.428       128    0.327132    0.497384
  291      16      9361      9345    128.44       132    0.318189    0.497225
  292      16      9390      9374   128.397       116    0.219333    0.497388
  293      16      9423      9407   128.409       132    0.425543    0.497768
  294      16      9451      9435   128.354       112     1.43109    0.497962
  295      16      9485      9469    128.38       136    0.189871    0.497842
  296      16      9516      9500   128.365       124    0.230003    0.497657
  297      16      9544      9528    128.31       112    0.955559    0.498058
  298      16      9576      9560   128.308       128   0.0124968    0.497977
  299      16      9613      9597   128.374       148   0.0123177    0.497772
2025-09-06T15:43:58.798869+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.497815
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      9647      9631     128.4       136    0.811663    0.497815
  301      16      9678      9662   128.385       124   0.0884557     0.49765
  302      16      9710      9694   128.384       128   0.0216289    0.497641
  303      16      9742      9726   128.382       128    0.174616    0.497752
  304      16      9777      9761    128.42       140    0.015451    0.497691
  305      16      9809      9793   128.419       128    0.216147     0.49751
  306      16      9842      9826   128.431       132    0.101977    0.497521
  307      16      9875      9859   128.442       132    0.112421    0.497377
  308      16      9906      9890   128.428       124     0.11111     0.49751
  309      16      9942      9926   128.478       144    0.649199    0.497281
  310      16      9979      9963   128.541       148     1.69193    0.496995
  311      16     10009      9993   128.514       120    0.220615    0.497058
  312      16     10047     10031   128.589       152    0.197369     0.49661
  313      16     10077     10061   128.561       120    0.424528    0.496527
  314      16     10113     10097    128.61       144    0.698878    0.496785
  315      16     10145     10129   128.609       128    0.409623    0.496865
  316      16     10175     10159   128.581       120   0.0907244    0.496909
  317      16     10211     10195    128.63       144    0.151109    0.496434
  318      16     10245     10229   128.653       136     0.92775    0.496823
  319      16     10277     10261   128.651       128     1.18775     0.49678
2025-09-06T15:44:18.801007+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.49664
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     10313     10297   128.699       144    0.577956     0.49664
  321      16     10344     10328   128.684       124   0.0136668     0.49665
  322      16     10373     10357   128.645       116    0.128139    0.496726
  323      16     10407     10391   128.667       136      0.1921    0.496567
  324      16     10442     10426   128.702       140   0.0853432    0.496328
  325      16     10474     10458     128.7       128    0.227602    0.496532
  326      16     10503     10487   128.661       116   0.0931666    0.496592
  327      16     10535     10519   128.659       128     1.43914    0.496656
  328      16     10565     10549   128.633       120     1.21759    0.496883
  329      16     10594     10578   128.594       116   0.0168948    0.496886
  330      16     10623     10607   128.556       116    0.127112    0.497045
  331      16     10654     10638   128.542       124    0.468855    0.497003
  332      16     10688     10672   128.565       136    0.318109    0.496861
  333      16     10718     10702   128.539       120   0.0895297      0.4968
  334      16     10746     10730   128.489       112   0.0931543    0.496932
  335      16     10781     10765   128.524       140    0.102196    0.497246
  336      16     10810     10794   128.486       116    0.309746    0.497038
  337      16     10840     10824   128.461       120     1.48846    0.497417
  338      16     10875     10859   128.495       140   0.0125648    0.497074
  339      16     10903     10887   128.447       112     0.24323       0.497
2025-09-06T15:44:38.803036+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.497293
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     10932     10916    128.41       116    0.159828    0.497293
  341      16     10969     10953   128.467       148    0.512563    0.497498
  342      16     11000     10984   128.454       124    0.430946     0.49749
  343      16     11034     11018   128.476       136   0.0190478      0.4975
  344      16     11064     11048   128.451       120     1.43754    0.497451
  345      16     11096     11080    128.45       128    0.123126    0.497735
  346      16     11126     11110   128.426       120   0.0199677    0.497568
  347      16     11160     11144   128.447       136    0.248343    0.497401
  348      16     11193     11177   128.458       132     1.33479    0.497461
  349      16     11225     11209   128.456       128     1.55478    0.497472
  350      16     11255     11239   128.432       120    0.402596    0.497454
  351      16     11284     11268   128.397       116    0.209659    0.497276
  352      16     11327     11311    128.52       172   0.0162636    0.496935
  353      16     11357     11341   128.496       120    0.303009    0.496808
  354      16     11389     11373   128.495       128     2.09929      0.4972
  355      16     11417     11401   128.448       112    0.329918    0.497181
  356      16     11450     11434   128.458       132    0.201195    0.497147
  357      16     11483     11467   128.468       132    0.012181    0.497155
  358      16     11513     11497   128.445       120    0.270215    0.497146
  359      16     11544     11528   128.432       124     2.08333     0.49738
2025-09-06T15:44:58.804827+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.497608
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     11574     11558   128.409       120     0.13663    0.497608
  361      16     11604     11588   128.385       120    0.143828    0.497511
  362      16     11640     11624   128.428       144     1.93239    0.497602
  363      16     11673     11657   128.438       132     1.44862    0.497344
  364      16     11709     11693   128.481       144     0.45765    0.497235
  365      16     11738     11722   128.447       116    0.187503    0.497381
  366      16     11769     11753   128.435       124     1.54588    0.497409
  367      16     11801     11785   128.433       128     0.14537    0.497391
  368      16     11834     11818   128.443       132   0.0837397     0.49768
  369      16     11863     11847   128.409       116     1.19477    0.497808
  370      16     11896     11880   128.419       132   0.0154446    0.497747
  371      16     11926     11910   128.396       120    0.291201    0.497643
  372      16     11958     11942   128.395       128    0.543976    0.497705
  373      16     11995     11979   128.448       148   0.0158311    0.497739
  374      16     12027     12011   128.446       128     1.44982    0.497627
  375      16     12057     12041   128.424       120   0.0830153    0.497706
  376      16     12088     12072   128.412       124    0.839145    0.497897
  377      16     12122     12106   128.432       136   0.0169174    0.497736
  378      16     12155     12139   128.441       132      1.1392     0.49759
  379      16     12185     12169   128.419       120    0.175248    0.497634
2025-09-06T15:45:18.806935+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.49747
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     12221     12205    128.46       144    0.133709     0.49747
  381      16     12254     12238   128.469       132    0.734467    0.497619
  382      16     12285     12269   128.458       124   0.0242917    0.497503
  383      16     12317     12301   128.456       128    0.148004    0.497484
  384      16     12351     12335   128.476       136    0.372017     0.49757
  385      16     12385     12369   128.496       136   0.0842423    0.497467
  386      16     12419     12403   128.515       136    0.610525     0.49743
  387      16     12450     12434   128.503       124    0.622395    0.497484
  388      16     12480     12464   128.481       120     0.16895    0.497589
  389      16     12515     12499   128.511       140    0.107047    0.497332
  390      16     12556     12540   128.602       164   0.0124197    0.497078
  391      16     12592     12576   128.641       144     0.33531    0.496892
  392      16     12621     12605   128.609       116   0.0831915    0.496986
  393      16     12652     12636   128.597       124    0.231673    0.497114
  394      16     12683     12667   128.585       124     1.42598    0.496965
  395      16     12713     12697   128.564       120    0.185326      0.4969
  396      16     12744     12728   128.552       124   0.0329641    0.496685
  397      16     12778     12762   128.571       136   0.0830932    0.496686
  398      16     12810     12794   128.569       128    0.797652     0.49691
  399      16     12840     12824   128.548       120     2.05504    0.497196
2025-09-06T15:45:38.809001+0000 min lat: 0.0112911 max lat: 2.90998 avg lat: 0.497298
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     12871     12855   128.536       124      0.6905    0.497298
  401      16     12902     12886   128.525       124    0.254844    0.497479
  402      16     12933     12917   128.514       124    0.205747    0.497421
  403      16     12966     12950   128.522       132    0.370429    0.497308
  404      16     12995     12979   128.491       116    0.677479    0.497547
  405      16     13026     13010    128.48       124   0.0870828     0.49738
  406      16     13058     13042   128.479       128     0.95667    0.497583
  407      16     13096     13080   128.537       152      1.6651      0.4974
  408      16     13135     13119   128.604       156     1.11176    0.497185
  409      16     13164     13148   128.573       116    0.177138     0.49717
  410      16     13195     13179   128.562       124    0.399098    0.497271
  411      16     13222     13206   128.512       108   0.0220697    0.497366
  412      16     13265     13249   128.618       172   0.0120533    0.497085
  413      16     13297     13281   128.616       128     1.09701     0.49718
  414      16     13329     13313   128.614       128     1.28757    0.497078
  415      16     13360     13344   128.603       124    0.567254    0.497114
  416      16     13391     13375   128.592       124    0.229586    0.497204
  417      16     13423     13407   128.591       128    0.115248    0.496987
  418      16     13459     13443   128.628       144    0.192905    0.497026
  419      16     13492     13476   128.636       132     1.24071    0.496936
2025-09-06T15:45:58.811246+0000 min lat: 0.0112911 max lat: 2.90998 avg lat: 0.497053
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     13519     13503   128.586       108     1.47235    0.497053
  421      16     13550     13534   128.575       124    0.212801      0.4972
  422      16     13579     13563   128.546       116     1.08266    0.497322
  423      16     13611     13595   128.544       128     1.52997    0.497483
  424      16     13641     13625   128.524       120    0.150946    0.497302
  425      16     13676     13660   128.551       140    0.596906    0.497405
  426      16     13711     13695   128.578       140      0.2281    0.497193
  427      16     13742     13726   128.567       124   0.0839347    0.497224
  428      16     13775     13759   128.575       132    0.201946    0.497209
  429      16     13810     13794   128.602       140     0.32731    0.497051
  430      16     13841     13825   128.591       124    0.949474    0.497035
  431      16     13873     13857    128.59       128    0.279448    0.497047
  432      16     13907     13891   128.607       136    0.789029    0.497195
  433      16     13939     13923   128.605       128     0.88506    0.497013
  434      16     13976     13960    128.65       148    0.221014    0.496969
  435      16     14012     13996   128.685       144   0.0118398     0.49664
  436      16     14043     14027   128.674       124    0.188171    0.496684
  437      16     14074     14058   128.664       124    0.412753     0.49664
  438      16     14109     14093    128.69       140   0.0876013    0.496772
  439      16     14137     14121   128.651       112   0.0227811    0.496833
2025-09-06T15:46:18.813698+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496803
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     14171     14155   128.668       136    0.629819    0.496803
  441      16     14201     14185   128.648       120   0.0876253    0.496817
  442      16     14233     14217   128.647       128    0.522567    0.497065
  443      16     14266     14250   128.654       132   0.0923337    0.496944
  444      16     14299     14283   128.662       132    0.239631    0.496894
  445      16     14333     14317   128.678       136   0.0167242    0.496877
  446      16     14366     14350   128.686       132   0.0132008    0.496753
  447      16     14400     14384   128.702       136    0.516377    0.496719
  448      16     14433     14417    128.71       132   0.0906252     0.49649
  449      16     14466     14450   128.717       132   0.0875175    0.496337
  450      16     14501     14485   128.742       140     1.70374    0.496583
  451      16     14533     14517    128.74       128     0.13055    0.496465
  452      16     14564     14548    128.73       124    0.421772    0.496476
  453      16     14597     14581   128.737       132    0.480648    0.496586
  454      16     14631     14615   128.753       136    0.728483    0.496505
  455      16     14660     14644   128.725       116   0.0258437    0.496583
  456      16     14699     14683   128.785       156     1.38215    0.496443
  457      16     14726     14710   128.739       108    0.225671    0.496422
  458      16     14755     14739   128.711       116    0.935365    0.496658
  459      16     14787     14771    128.71       128    0.185154    0.496743
2025-09-06T15:46:38.815971+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496528
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     14818     14802   128.699       124    0.629326    0.496528
  461      16     14853     14837   128.724       140    0.382282    0.496571
  462      16     14884     14868   128.714       124    0.236527    0.496563
  463      16     14914     14898   128.695       120    0.416287    0.496686
  464      16     14946     14930   128.693       128    0.159529    0.496667
  465      16     14975     14959   128.666       116     1.65493    0.496633
  466      16     15005     14989   128.647       120     0.39065    0.496549
  467      16     15043     15027   128.697       152    0.318626    0.496765
  468      16     15072     15056    128.67       116     0.27181    0.496826
  469      16     15104     15088   128.669       128   0.0873646    0.496884
  470      16     15135     15119   128.659       124     1.11515    0.496908
  471      16     15165     15149    128.64       120   0.0867459    0.496877
  472      16     15196     15180    128.63       124     0.20517    0.497097
  473      16     15228     15212   128.629       128    0.196976    0.497028
  474      16     15260     15244   128.628       128    0.155721    0.496837
  475      16     15288     15272   128.593       112     0.11416    0.496959
  476      16     15320     15304   128.591       128   0.0880278     0.49698
  477      16     15351     15335   128.582       124    0.599031    0.497028
  478      16     15382     15366   128.572       124    0.165262     0.49717
  479      16     15412     15396   128.554       120   0.0910894    0.497152
2025-09-06T15:46:58.818123+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497084
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     15444     15428   128.553       128    0.177656    0.497084
  481      16     15481     15465   128.593       148    0.194738    0.497015
  482      16     15510     15494   128.567       116     1.85793     0.49712
  483      16     15543     15527   128.574       132    0.324268    0.496883
  484      16     15579     15563   128.606       144    0.220855    0.496859
  485      16     15608     15592    128.58       116    0.157102    0.496885
  486      16     15641     15625   128.587       132     1.02381    0.497195
  487      16     15672     15656   128.578       124   0.0873863    0.497203
  488      16     15702     15686    128.56       120    0.221675    0.497358
  489      16     15737     15721   128.583       140     1.19506    0.497225
  490      16     15768     15752   128.574       124     1.58886    0.497159
  491      16     15807     15791    128.63       156    0.120142    0.496906
  492      16     15838     15822    128.62       124    0.208816    0.496872
  493      16     15874     15858   128.651       144     1.70134    0.496848
  494      16     15905     15889   128.642       124    0.013919     0.49679
  495      16     15938     15922   128.649       132    0.107668    0.496578
  496      16     15969     15953   128.639       124     0.20202    0.496691
  497      16     16000     15984    128.63       124     2.19179    0.496876
  498      16     16030     16014   128.613       120   0.0935627      0.4969
  499      16     16063     16047   128.619       132    0.586365    0.496987
2025-09-06T15:47:18.820825+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497189
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     16093     16077   128.602       120    0.928535    0.497189
  501      16     16125     16109   128.601       128    0.587212    0.497124
  502      16     16157     16141     128.6       128   0.0149436    0.496984
  503      16     16188     16172    128.59       124    0.525985    0.497092
  504      16     16223     16207   128.613       140    0.177496     0.49698
  505      16     16256     16240    128.62       132     1.49771    0.497001
  506      16     16284     16268   128.587       112    0.204646    0.497055
  507      16     16323     16307   128.641       156    0.162624    0.497101
  508      16     16353     16337   128.624       120     1.14793    0.497145
  509      16     16381     16365   128.591       112    0.258723    0.497014
  510      16     16415     16399   128.606       136   0.0191178     0.49716
  511      16     16450     16434   128.628       140   0.0163753    0.497114
  512      16     16480     16464   128.611       120     1.45301    0.496958
  513      16     16513     16497   128.618       132    0.299722      0.4968
  514      16     16545     16529   128.616       128    0.470141    0.497005
  515      16     16575     16559     128.6       120   0.0836169    0.497205
  516      16     16604     16588   128.575       116    0.795945    0.497203
  517      16     16636     16620   128.574       128    0.152648    0.497248
  518      16     16668     16652   128.573       128      1.7621    0.497177
  519      16     16701     16685    128.58       132   0.0286827    0.497207
2025-09-06T15:47:38.823024+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497116
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     16733     16717   128.578       128    0.121051    0.497116
  521      16     16764     16748    128.57       124     1.92792    0.497185
  522      16     16796     16780   128.568       128    0.223131    0.497065
  523      16     16829     16813   128.575       132     2.03957    0.497055
  524      16     16860     16844   128.566       124    0.368705    0.497162
  525      16     16892     16876   128.565       128   0.0291149    0.497224
  526      16     16927     16911   128.587       140    0.289156    0.497117
  527      16     16956     16940   128.563       116    0.154645    0.497165
  528      16     16987     16971   128.554       124   0.0191086    0.497211
  529      16     17018     17002   128.546       124    0.016989    0.497038
  530      16     17052     17036    128.56       136   0.0873115    0.496995
  531      16     17085     17069   128.566       132    0.261964    0.497159
  532      16     17115     17099    128.55       120    0.271947    0.497064
  533      16     17148     17132   128.556       132   0.0223805    0.497098
  534      16     17179     17163   128.548       124     2.03793    0.497349
  535      16     17209     17193   128.532       120    0.380499    0.497416
  536      16     17241     17225   128.531       128    0.395876    0.497297
  537      16     17273     17257    128.53       128     1.56496    0.497343
  538      16     17310     17294   128.566       148   0.0836351     0.49699
  539      16     17339     17323   128.543       116    0.261275    0.497154
2025-09-06T15:47:58.825300+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497092
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     17374     17358   128.564       140    0.158989    0.497092
  541      16     17405     17389   128.555       124    0.215266    0.497162
  542      16     17438     17422   128.562       132    0.352414     0.49733
  543      16     17472     17456   128.575       136      1.7696    0.497264
  544      16     17506     17490   128.589       136   0.0252598    0.497077
  545      16     17538     17522   128.588       128   0.0955861    0.497009
  546      16     17573     17557   128.609       140     1.66585    0.497175
  547      16     17607     17591   128.622       136    0.189143    0.496961
  548      16     17638     17622   128.614       124    0.108806    0.496882
  549      16     17671     17655    128.62       132     2.31094    0.496916
  550      16     17701     17685   128.604       120   0.0906144    0.497151
  551      16     17728     17712   128.567       108   0.0891716     0.49709
  552      16     17761     17745   128.573       132    0.093646    0.497087
  553      16     17789     17773   128.543       112    0.143058    0.497338
  554      16     17823     17807   128.557       136    0.603812    0.497307
  555      16     17853     17837   128.541       120    0.109883    0.497243
  556      16     17884     17868   128.533       124   0.0156527    0.497444
  557      16     17913     17897    128.51       116   0.0838089    0.497448
  558      16     17948     17932   128.531       140      1.5958    0.497477
  559      16     17978     17962   128.516       120    0.202312    0.497409
2025-09-06T15:48:18.827278+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497323
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     18014     17998   128.543       144    0.611702    0.497323
  561      16     18045     18029   128.535       124   0.0833382    0.497345
  562      16     18079     18063   128.548       136    0.156749    0.497305
  563      16     18114     18098   128.569       140   0.0152491      0.4973
  564      16     18143     18127   128.546       116     1.95806    0.497362
  565      16     18176     18160   128.553       132    0.379853    0.497464
  566      16     18209     18193   128.559       132   0.0878705    0.497461
  567      16     18240     18224   128.551       124    0.083344    0.497533
  568      16     18270     18254   128.535       120    0.432692    0.497496
  569      16     18299     18283   128.513       116   0.0146442    0.497573
  570      16     18332     18316    128.52       132     1.27941    0.497523
  571      16     18366     18350   128.533       136    0.421052     0.49734
  572      16     18394     18378   128.504       112     0.20878    0.497499
  573      16     18422     18406   128.475       112   0.0953588    0.497533
  574      16     18454     18438   128.474       128     1.88759    0.497579
  575      16     18486     18470   128.473       128   0.0906426     0.49748
  576      16     18525     18509   128.521       156   0.0117905    0.497391
  577      16     18554     18538   128.499       116    0.026821    0.497546
  578      16     18585     18569   128.491       124     0.55262    0.497686
  579      16     18625     18609   128.546       160     1.18513    0.497467
2025-09-06T15:48:38.829200+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497211
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     18660     18644   128.566       140   0.0887577    0.497211
  581      16     18694     18678   128.578       136     1.86063    0.497239
  582      16     18726     18710   128.577       128   0.0204899    0.497217
  583      16     18754     18738   128.549       112   0.0166127    0.497212
  584      16     18786     18770   128.548       128   0.0827811    0.497199
  585      16     18818     18802   128.547       128    0.129177    0.497309
  586      16     18852     18836    128.56       136     2.06812    0.497305
  587      16     18881     18865   128.538       116    0.192596     0.49741
  588      16     18916     18900   128.558       140    0.021311    0.497254
  589      16     18952     18936   128.584       144   0.0842292    0.497262
  590      16     18982     18966   128.569       120    0.203662    0.497172
  591      16     19013     18997   128.561       124       2.305    0.497252
  592      16     19046     19030   128.567       132     1.88673    0.497218
  593      16     19084     19068   128.607       152    0.355858    0.496991
  594      16     19114     19098   128.592       120    0.191087    0.497129
  595      16     19147     19131   128.598       132    0.260669    0.497217
  596      16     19180     19164   128.604       132    0.911384    0.497343
  597      16     19212     19196   128.603       128   0.0959994    0.497227
  598      16     19243     19227   128.595       124    0.112064    0.497089
  599      16     19278     19262   128.614       140    0.285722    0.497112
2025-09-06T15:48:58.831279+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497174
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     19310     19294   128.613       128    0.564131    0.497174
  601      16     19341     19325   128.605       124     1.99262    0.497139
  602      16     19374     19358   128.611       132    0.493302     0.49707
  603      16     19411     19395   128.643       148    0.553165    0.496956
  604      16     19442     19426   128.635       124     0.15799    0.497093
  605      16     19470     19454   128.608       112    0.706058    0.497189
  606      16     19504     19488    128.62       136   0.0877895    0.497176
  607      16     19534     19518   128.606       120     1.39541     0.49723
  608      16     19566     19550   128.605       128    0.780547    0.497219
  609      16     19598     19582   128.604       128    0.958955    0.497244
  610      16     19627     19611   128.583       116    0.899976    0.497336
  611      16     19663     19647   128.608       144     1.23997    0.497237
  612      16     19698     19682   128.627       140   0.0131782    0.497181
  613      16     19733     19717   128.645       140     0.15652    0.497012
  614      16     19764     19748   128.638       124    0.819587    0.497135
  615      16     19795     19779    128.63       124    0.529949     0.49701
  616      16     19826     19810   128.623       124   0.0120411    0.497145
  617      16     19858     19842   128.621       128     1.73184    0.497184
  618      16     19890     19874    128.62       128    0.702031    0.497218
  619      16     19923     19907   128.626       132     1.12445    0.497201
2025-09-06T15:49:18.833681+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.49716
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  620      16     19957     19941   128.638       136     0.15215     0.49716
  621      16     19991     19975    128.65       136     1.10722    0.497151
  622      16     20023     20007   128.649       128   0.0845331    0.496995
  623      16     20052     20036   128.628       116   0.0900372    0.496987
  624      16     20087     20071   128.646       140   0.0856471    0.496966
  625      16     20119     20103   128.645       128     1.80381    0.497067
  626      16     20151     20135   128.644       128    0.101881     0.49704
  627      16     20186     20170   128.662       140    0.149242    0.496841
  628      16     20222     20206   128.687       144    0.366895    0.496914
  629      16     20252     20236   128.673       120     1.78885       0.497
  630      16     20284     20268   128.672       128    0.350274    0.497013
  631      16     20316     20300   128.671       128    0.227148    0.497012
  632      16     20348     20332    128.67       128     1.06538     0.49703
  633      16     20376     20360   128.643       112     0.20394    0.497043
  634      16     20409     20393   128.649       132   0.0166973    0.497111
  635      16     20443     20427    128.66       136    0.219014    0.497036
  636      16     20475     20459   128.659       128    0.180861    0.496958
  637      16     20508     20492   128.664       132    0.227043     0.49701
  638      16     20542     20526   128.676       136    0.268174    0.496931
  639      16     20578     20562     128.7       144    0.293443    0.496817
2025-09-06T15:49:38.835859+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496911
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  640      16     20607     20591    128.68       116   0.0308784    0.496911
  641      16     20640     20624   128.685       132    0.125695    0.496928
  642      16     20676     20660   128.709       144    0.402102    0.496851
  643      16     20709     20693   128.714       132    0.121779    0.496815
  644      16     20742     20726   128.719       132    0.298877    0.496934
  645      16     20771     20755   128.699       116     1.01643    0.496949
  646      16     20802     20786   128.692       124     1.59932    0.496923
  647      16     20834     20818   128.691       128    0.228951    0.496876
  648      16     20864     20848   128.677       120    0.127424    0.496905
  649      16     20898     20882   128.689       136    0.012764    0.496891
  650      16     20931     20915   128.694       132    0.555876    0.497001
  651      16     20962     20946   128.687       124    0.501452    0.497003
  652      16     20994     20978   128.686       128    0.357937    0.496874
  653      16     21027     21011   128.691       132    0.712629    0.496922
  654      16     21059     21043   128.689       128   0.0919336    0.496951
  655      16     21089     21073   128.676       120    0.227276    0.496975
  656      16     21117     21101   128.651       112    0.163457    0.497062
  657      16     21149     21133    128.65       128    0.452582    0.497044
  658      16     21175     21159   128.612       104    0.345423    0.497064
  659      16     21207     21191   128.611       128    0.359749    0.497043
2025-09-06T15:49:58.838143+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496999
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  660      16     21243     21227   128.635       144   0.0119636    0.496999
  661      16     21281     21265    128.67       152   0.0834842    0.496886
  662      16     21313     21297   128.669       128   0.0859965    0.496878
  663      16     21347     21331    128.68       136   0.0868878    0.496864
  664      16     21382     21366   128.697       140    0.137291    0.496798
  665      16     21414     21398   128.696       128   0.0899432     0.49699
  666      16     21444     21428   128.683       120    0.158628    0.496937
  667      16     21476     21460   128.682       128   0.0854752    0.496916
  668      16     21510     21494   128.693       136     1.10989    0.496973
  669      16     21543     21527   128.698       132    0.973859    0.496867
  670      16     21579     21563    128.72       144     1.01193    0.496886
  671      16     21609     21593   128.707       120    0.199175      0.4969
  672      16     21642     21626   128.712       132    0.455941    0.496838
  673      16     21678     21662   128.735       144    0.319036    0.496787
  674      16     21711     21695    128.74       132     0.35557    0.496824
  675      16     21742     21726   128.733       124      1.4648    0.496785
  676      16     21781     21765   128.773       156    0.086695    0.496519
  677      16     21813     21797   128.772       128   0.0984366    0.496508
  678      16     21847     21831   128.783       136     1.52147    0.496547
  679      16     21875     21859   128.758       112    0.123523    0.496519
2025-09-06T15:50:18.840588+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496447
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  680      16     21906     21890   128.751       124    0.388716    0.496447
  681      16     21945     21929   128.791       156    0.462731    0.496491
  682      16     21977     21961    128.79       128    0.193746    0.496538
  683      16     22011     21995     128.8       136    0.244371    0.496526
  684      16     22041     22025   128.787       120    0.111875    0.496502
  685      16     22076     22060   128.804       140     0.16684     0.49648
  686      16     22106     22090   128.791       120    0.890448     0.49659
  687      16     22141     22125   128.807       140    0.191691    0.496505
  688      16     22172     22156     128.8       124    0.366525    0.496479
  689      16     22206     22190    128.81       136     1.53857    0.496555
  690      16     22237     22221   128.803       124    0.669494    0.496476
  691      16     22269     22253   128.802       128   0.0871688    0.496537
  692      16     22303     22287   128.813       136    0.236077    0.496521
  693      16     22331     22315   128.788       112     1.42682    0.496554
  694      16     22362     22346   128.781       124     1.11015    0.496622
  695      16     22398     22382   128.803       144    0.384562    0.496515
  696      16     22431     22415   128.808       132   0.0832375    0.496519
  697      16     22464     22448   128.812       132    0.146332    0.496472
  698      16     22494     22478     128.8       120     1.40816    0.496536
  699      16     22527     22511   128.804       132   0.0936015    0.496516
2025-09-06T15:50:38.842907+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496485
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  700      16     22561     22545   128.815       136    0.174489    0.496485
  701      16     22590     22574   128.796       116      1.4611    0.496474
  702      16     22621     22605   128.789       124   0.0871898    0.496475
  703      16     22652     22636   128.783       124   0.0838748    0.496395
  704      16     22679     22663   128.753       108    0.535887    0.496632
  705      16     22712     22696   128.758       132    0.087955     0.49666
  706      16     22747     22731   128.774       140   0.0890702    0.496638
  707      15     22779     22764   128.778       132   0.0875567    0.496649
  708      16     22813     22797   128.783       132     1.37929    0.496529
  709      16     22846     22830   128.787       132   0.0217707    0.496489
  710      16     22877     22861    128.78       124    0.420075    0.496555
  711      16     22908     22892   128.774       124   0.0914168     0.49651
  712      16     22940     22924   128.773       128    0.194288    0.496498
  713      16     22976     22960   128.794       144   0.0834247    0.496492
  714      16     23012     22996   128.815       144   0.0167785    0.496477
  715      16     23047     23031   128.831       140     1.31584    0.496381
  716      16     23082     23066   128.846       140     1.26216    0.496318
  717      16     23112     23096   128.834       120    0.128126    0.496309
  718      16     23142     23126   128.822       120    0.495135    0.496318
  719      16     23173     23157   128.815       124    0.295205    0.496372
2025-09-06T15:50:58.845264+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496362
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  720      16     23205     23189   128.814       128   0.0133885    0.496362
  721      16     23239     23223   128.824       136     0.78904    0.496277
  722      16     23268     23252   128.806       116    0.667653    0.496527
  723      16     23301     23285    128.81       132     0.15411    0.496543
  724      16     23334     23318   128.815       132    0.250982    0.496471
  725      16     23365     23349   128.808       124    0.521842    0.496339
  726      16     23402     23386   128.834       148   0.0838884    0.496444
  727      16     23430     23414   128.811       112   0.0912053     0.49645
  728      16     23465     23449   128.827       140   0.0160954    0.496404
  729      16     23499     23483   128.836       136    0.158878    0.496505
  730      16     23530     23514    128.83       124   0.0175636    0.496409
  731      16     23563     23547   128.834       132    0.234138    0.496404
  732      16     23594     23578   128.827       124     1.15021    0.496508
  733      16     23627     23611   128.832       132    0.417956    0.496433
  734      16     23655     23639   128.809       112    0.190337    0.496495
  735      16     23685     23669   128.797       120    0.550822    0.496505
  736      16     23719     23703   128.807       136    0.561391      0.4965
  737      16     23754     23738   128.822       140    0.209948    0.496347
  738      16     23787     23771   128.826       132   0.0179907    0.496361
  739      16     23817     23801   128.814       120   0.0881252    0.496386
2025-09-06T15:51:18.847793+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496551
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  740      16     23847     23831   128.802       120    0.124325    0.496551
  741      16     23882     23866   128.817       140    0.018145    0.496493
  742      16     23914     23898   128.816       128   0.0880226    0.496466
  743      16     23950     23934   128.837       144   0.0881598    0.496383
  744      16     23983     23967   128.841       132    0.186385    0.496365
  745      16     24014     23998   128.834       124   0.0176771    0.496384
  746      16     24050     24034   128.855       144   0.0833495    0.496319
  747      16     24083     24067   128.859       132     1.48442    0.496403
  748      16     24115     24099   128.858       128     1.43534    0.496401
  749      16     24149     24133   128.867       136    0.262983    0.496309
  750      16     24180     24164   128.861       124   0.0874134    0.496286
  751      16     24211     24195   128.854       124    0.205091    0.496339
  752      16     24241     24225   128.842       120    0.246256    0.496232
  753      16     24271     24255   128.831       120     2.21482    0.496318
  754      16     24303     24287   128.829       128    0.119742    0.496375
  755      16     24333     24317   128.818       120   0.0904597    0.496338
  756      16     24366     24350   128.822       132     0.42795    0.496291
  757      16     24399     24383   128.826       132     2.59693    0.496464
  758      16     24430     24414    128.82       124   0.0178529     0.49651
  759      16     24463     24447   128.824       132   0.0846903    0.496464
2025-09-06T15:51:38.850062+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496402
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  760      16     24493     24477   128.812       120    0.509347    0.496402
  761      16     24524     24508   128.806       124    0.357603    0.496541
  762      16     24552     24536   128.784       112     1.64092     0.49657
  763      16     24584     24568   128.783       128    0.435011    0.496585
  764      16     24614     24598   128.771       120    0.916258    0.496624
  765      16     24645     24629   128.765       124   0.0132452    0.496643
  766      16     24679     24663   128.774       136    0.086325    0.496644
  767      16     24709     24693   128.763       120    0.182785    0.496711
  768      16     24736     24720   128.736       108     1.30047    0.496805
  769      16     24768     24752   128.735       128     0.16885    0.496846
  770      16     24799     24783   128.729       124   0.0908904    0.496743
  771      16     24831     24815   128.728       128    0.513463    0.496755
  772      16     24864     24848   128.732       132     1.78014    0.496769
  773      16     24897     24881   128.736       132    0.135393    0.496665
  774      16     24928     24912    128.73       124    0.141492    0.496685
  775      16     24959     24943   128.724       124   0.0250499    0.496823
  776      16     24991     24975   128.723       128   0.0119032    0.496847
  777      16     25020     25004   128.707       116   0.0857588    0.496821
  778      16     25054     25038   128.716       136   0.0164854     0.49677
  779      16     25085     25069    128.71       124    0.388014     0.49686
2025-09-06T15:51:58.852300+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496877
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  780      16     25118     25102   128.714       132    0.111373    0.496877
  781      16     25147     25131   128.698       116    0.396779    0.496892
  782      16     25180     25164   128.702       132    0.133056    0.496941
  783      16     25211     25195   128.696       124     1.01611    0.497033
  784      16     25242     25226    128.69       124    0.101869    0.496959
  785      16     25272     25256   128.679       120   0.0874754    0.496972
  786      16     25302     25286   128.668       120    0.723337     0.49711
  787      16     25331     25315   128.652       116   0.0168556    0.497178
  788      16     25357     25341    128.62       104    0.886662    0.497275
  789      16     25390     25374   128.625       132      1.2442    0.497306
  790      16     25423     25407   128.629       132     1.26531    0.497244
  791      16     25454     25438   128.623       124    0.149689    0.497251
  792      16     25487     25471   128.627       132    0.873668    0.497355
  793      16     25518     25502   128.621       124    0.430606     0.49723
  794      16     25551     25535   128.626       132    0.528231    0.497298
  795      16     25580     25564    128.61       116    0.137408     0.49729
  796      16     25614     25598   128.619       136    0.559987    0.497242
  797      16     25646     25630   128.618       128    0.581304    0.497287
  798      16     25677     25661   128.612       124     0.43385    0.497208
  799      16     25709     25693   128.612       128     1.33966    0.497326
2025-09-06T15:52:18.854533+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497211
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  800      16     25744     25728   128.626       140     1.43384    0.497211
  801      16     25776     25760   128.625       128    0.172351    0.497136
  802      16     25808     25792   128.624       128    0.977078    0.497281
  803      16     25838     25822   128.614       120     1.72357     0.49721
  804      16     25873     25857   128.628       140    0.269135    0.497191
  805      16     25902     25886   128.612       116   0.0883461    0.497236
  806      16     25938     25922   128.631       144   0.0169808    0.497166
  807      16     25974     25958    128.65       144     1.75482    0.497093
  808      16     26002     25986   128.629       112    0.408635     0.49712
  809      16     26037     26021   128.644       140   0.0162427    0.497082
  810      16     26066     26050   128.628       116    0.267328    0.497039
  811      16     26103     26087   128.652       148    0.268523    0.497128
  812      16     26132     26116   128.636       116   0.0926032     0.49717
  813      16     26160     26144   128.616       112    0.703411    0.497313
  814      16     26194     26178   128.625       136    0.698206    0.497333
  815      16     26227     26211   128.629       132    0.197742    0.497272
  816      16     26258     26242   128.623       124    0.135187    0.497255
  817      16     26289     26273   128.618       124      0.1126    0.497193
  818      16     26322     26306   128.622       132    0.762397    0.497249
  819      16     26363     26347   128.665       164   0.0171378    0.497178
2025-09-06T15:52:38.856545+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497187
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  820      16     26395     26379   128.664       128    0.599347    0.497187
  821      16     26428     26412   128.668       132      0.6571    0.497151
  822      16     26463     26447   128.682       140     1.16671    0.497104
  823      16     26498     26482   128.696       140    0.338615    0.496984
  824      16     26530     26514   128.695       128    0.717878    0.497026
  825      16     26564     26548   128.704       136   0.0133314    0.497018
  826      16     26597     26581   128.707       132    0.109907    0.496872
  827      16     26630     26614   128.711       132    0.961285    0.496981
  828      16     26660     26644   128.701       120    0.965745    0.497026
  829      16     26690     26674    128.69       120   0.0881294    0.497023
  830      16     26722     26706    128.69       128     1.18792    0.497033
  831      16     26753     26737   128.684       124   0.0835127    0.496988
  832      16     26788     26772   128.697       140     1.25384    0.496967
  833      16     26815     26799   128.673       108    0.131317    0.497008
  834      16     26847     26831   128.672       128       1.285    0.497141
  835      16     26882     26866   128.685       140    0.103099    0.496908
  836      16     26914     26898   128.685       128    0.599605    0.496912
  837      16     26946     26930   128.684       128    0.219788    0.497017
  838      16     26979     26963   128.688       132    0.280519    0.496895
  839      16     27009     26993   128.677       120   0.0848683     0.49695
2025-09-06T15:52:58.858510+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496968
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  840      16     27045     27029   128.695       144    0.227195    0.496968
  841      16     27077     27061   128.695       128    0.338462    0.496926
  842      16     27113     27097   128.713       144     1.83516    0.496881
  843      16     27144     27128   128.707       124   0.0830275    0.496864
  844      16     27175     27159   128.702       124   0.0138857    0.496822
  845      16     27210     27194   128.715       140    0.184648    0.496845
  846      16     27242     27226   128.714       128    0.105456     0.49683
  847      16     27272     27256   128.704       120    0.124289    0.496928
  848      16     27304     27288   128.703       128      0.2104    0.496874
  849      16     27337     27321   128.707       132    0.330146    0.496863
  850      16     27373     27357   128.725       144     1.21452    0.496957
  851      16     27405     27389   128.724       128   0.0160896     0.49694
  852      16     27438     27422   128.728       132   0.0870598    0.496865
  853      16     27468     27452   128.717       120     1.13649    0.496912
  854      16     27502     27486   128.726       136     1.30123    0.496931
  855      16     27536     27520   128.734       136    0.621463    0.496879
  856      16     27565     27549    128.72       116    0.648204    0.496945
  857      16     27592     27576   128.695       108     1.12094    0.496994
  858      16     27622     27606   128.685       120   0.0867658    0.496989
  859      16     27656     27640   128.694       136   0.0176688    0.496998
2025-09-06T15:53:18.860908+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497107
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  860      16     27686     27670   128.684       120     0.87022    0.497107
  861      16     27716     27700   128.674       120    0.202417    0.497041
  862      16     27749     27733   128.677       132    0.104684    0.496995
  863      16     27778     27762   128.663       116    0.231146    0.497071
  864      16     27810     27794   128.662       128   0.0961636    0.497177
  865      16     27839     27823   128.647       116    0.957915    0.497136
  866      16     27872     27856   128.651       132    0.621128    0.497192
  867      16     27905     27889   128.655       132    0.214006     0.49715
  868      16     27936     27920    128.65       124    0.818133    0.497258
  869      16     27967     27951   128.644       124    0.246626    0.497097
  870      16     28001     27985   128.653       136   0.0870381    0.497146
  871      16     28033     28017   128.652       128    0.431196    0.497206
  872      16     28065     28049   128.651       128    0.115195    0.497104
  873      16     28096     28080   128.646       124     1.56901    0.497268
  874      16     28131     28115   128.659       140    0.610961    0.497198
  875      16     28160     28144   128.644       116    0.646003    0.497296
  876      16     28191     28175   128.639       124   0.0868408    0.497228
  877      16     28221     28205   128.629       120   0.0193528    0.497262
  878      16     28251     28235   128.619       120   0.0830888    0.497271
  879      16     28287     28271   128.637       144   0.0879058    0.497193
2025-09-06T15:53:38.863231+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497356
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  880      16     28314     28298   128.613       108     1.79126    0.497356
  881      16     28343     28327   128.599       116    0.017847    0.497383
  882      16     28374     28358   128.594       124   0.0885008    0.497391
  883      16     28405     28389   128.588       124    0.236418    0.497454
  884      16     28441     28425   128.606       144   0.0180073    0.497363
  885      16     28475     28459   128.614       136     0.18639    0.497292
  886      16     28510     28494   128.627       140    0.340808    0.497261
  887      16     28536     28520   128.599       104   0.0854572    0.497357
  888      16     28566     28550    128.59       120   0.0878152    0.497391
  889      16     28595     28579   128.575       116    0.136222    0.497443
  890      16     28629     28613   128.584       136    0.232702     0.49743
  891      16     28664     28648   128.596       140     1.58094    0.497404
  892      16     28694     28678   128.587       120   0.0206609    0.497411
  893      16     28727     28711   128.591       132   0.0944843    0.497407
  894      16     28759     28743    128.59       128   0.0942855    0.497424
  895      16     28792     28776   128.594       132    0.172671     0.49738
  896      16     28821     28805    128.58       116   0.0936381     0.49748
  897      16     28853     28837   128.579       128     1.50884    0.497508
  898      16     28886     28870   128.583       132   0.0136993    0.497517
  899      16     28918     28902   128.582       128   0.0891547    0.497503
2025-09-06T15:53:58.865532+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497532
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  900      13     28947     28934   128.581       128    0.804857    0.497532
Total time run:         900.508
Total writes made:      28947
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     128.581
Stddev Bandwidth:       10.3096
Max bandwidth (MB/sec): 172
Min bandwidth (MB/sec): 104
Average IOPS:           32
Stddev IOPS:            2.5774
Max IOPS:               43
Min IOPS:               26
Average Latency(s):     0.497607
Stddev Latency(s):      0.560887
Max latency(s):         2.90998
Min latency(s):         0.0112637
[ceph: root@ms-01-01 /]# 

Rados, sequential read test, pool with 32 PGs

[ceph: root@ms-01-01 /]# date
Sat Sep  6 18:02:39 UTC 2025
[ceph: root@ms-01-01 /]# rados bench -p testbench 900 seq
hints = 1
Click to see full run
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       1         1         0         0         0           -           0
    1      16        51        35   139.971       140  0.00576719    0.279735
    2      16        81        65   129.978       120    0.615285    0.400994
    3      16       111        95   126.646       120    0.262298    0.440092
    4      16       147       131    130.98       144     1.06346    0.449015
    5      16       176       160   127.981       116     1.16096     0.45495
    6      16       209       193   128.648       132  0.00941819    0.460813
    7      16       241       225   128.553       128    0.278738    0.471168
    8      16       272       256   127.982       124    0.585682    0.473817
    9      16       305       289   128.423       132    0.397055    0.475108
   10      16       336       320   127.979       124   0.0100224    0.479312
   11      16       368       352   127.979       128    0.349567    0.479956
   12      16       398       382   127.313       120    0.233218    0.482224
   13      16       427       411   126.442       116    0.141797    0.484991
   14      16       461       445   127.123       136   0.0940371    0.486826
   15      16       494       478   127.447       132    0.984991    0.487235
   16      16       525       509   127.231       124    0.909859    0.488067
   17      16       563       547   128.686       152    0.165475    0.482307
   18      16       593       577   128.203       120    0.436253    0.486661
   19      16       625       609   128.192       128    0.811796    0.487417
2025-09-06T18:03:02.348388+0000 min lat: 0.00306773 max lat: 1.89244 avg lat: 0.49023
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       655       639   127.781       120    0.281749     0.49023
   21      16       685       669    127.41       120    0.437773    0.488087
   22      16       716       700   127.254       124    0.410089    0.489592
   23      16       748       732   127.286       128    0.362424    0.492715
   24      16       780       764   127.315       128    0.256071    0.492451
   25      16       810       794   127.022       120    0.294221    0.496227
   26      16       842       826   127.059       128    0.280833    0.492168
   27      16       871       855   126.649       116     1.64619    0.495698
   28      16       905       889   126.982       136    0.349129    0.495225
   29      16       938       922   127.154       132     1.37743    0.495949
   30      16       973       957   127.582       140  0.00500597    0.492614
   31      16      1004       988   127.466       124    0.205431    0.495119
   32      16      1033      1017   127.107       116     1.08622    0.496186
   33      16      1068      1052   127.497       140    0.125499    0.493873
   34      16      1097      1081   127.159       116    0.187805    0.492131
   35      16      1131      1115   127.411       136    0.535902    0.495211
   36      16      1165      1149   127.649       136    0.648156    0.495292
   37      16      1196      1180    127.55       124    0.174556    0.493856
   38      16      1228      1212   127.561       128    0.465592    0.493075
   39      16      1262      1246   127.777       136   0.0102957    0.495957
2025-09-06T18:03:22.351012+0000 min lat: 0.00306773 max lat: 2.0271 avg lat: 0.494969
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1295      1279   127.882       132    0.912497    0.494969
   41      16      1323      1307   127.494       112    0.912498    0.496502
   42      16      1356      1340   127.601       132  0.00948159    0.497076
   43      16      1389      1373   127.703       132   0.0108206    0.497531
   44      16      1425      1409   128.073       144    0.227257    0.494487
   45      16      1455      1439   127.893       120    0.397512    0.494719
   46      16      1483      1467   127.548       112    0.808184     0.49758
   47      16      1512      1496   127.302       116     0.61087    0.498028
   48      16      1547      1531   127.566       140     0.20288    0.497715
   49      16      1578      1562   127.493       124    0.620353    0.498016
   50      16      1616      1600   127.982       152  0.00426473    0.495593
   51      16      1646      1630   127.825       120     1.04344    0.495678
   52      16      1676      1660   127.674       120    0.339092    0.496449
   53      16      1709      1693   127.755       132    0.434183    0.498095
   54      16      1737      1721   127.463       112  0.00930642    0.497088
   55      16      1771      1755   127.618       136  0.00946941    0.497972
   56      16      1804      1788   127.696       132    0.259619    0.497103
   57      16      1837      1821   127.771       132  0.00950038     0.49549
   58      16      1869      1853   127.775       128    0.229229    0.495607
   59      16      1903      1887   127.914       136    0.377198    0.495499
2025-09-06T18:03:42.353886+0000 min lat: 0.00300177 max lat: 2.0271 avg lat: 0.495681
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1937      1921   128.049       136    0.229569    0.495681
   61      16      1971      1955   128.179       136    0.179857    0.495987
   62      16      2003      1987   128.176       128    0.148077    0.496047
   63      16      2036      2020   128.236       132  0.00321485    0.494439
   64      16      2073      2057   128.545       148  0.00506035    0.492693
   65      16      2105      2089   128.536       128  0.00996371    0.493904
   66      16      2137      2121   128.528       128     0.21248    0.494269
   67      16      2166      2150    128.34       116     0.33767    0.495082
   68      16      2195      2179   128.159       116    0.737051    0.496149
   69      16      2224      2208   127.982       116    0.543227    0.496498
   70      16      2256      2240   127.982       128    0.176781    0.496147
   71      16      2285      2269   127.813       116    0.266752    0.497433
   72      16      2316      2300    127.76       124    0.372841    0.497255
   73      16      2353      2337   128.037       148    0.139255    0.496314
   74      16      2385      2369   128.036       128    0.266525    0.496781
   75      16      2420      2404   128.196       140    0.297324    0.496606
   76      16      2448      2432   127.982       112    0.717249    0.497082
   77      16      2480      2464   127.982       128    0.694474    0.497304
   78      16      2514      2498   128.085       136     1.20036    0.496787
   79      16      2547      2531   128.134       132     1.22903    0.497212
2025-09-06T18:04:02.356418+0000 min lat: 0.00300177 max lat: 2.0271 avg lat: 0.495893
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2579      2563   128.132       128     0.60856    0.495893
   81      16      2614      2598   128.279       140    0.216403     0.49574
   82      16      2647      2631   128.324       132     1.40591    0.495954
   83      16      2675      2659   128.127       112    0.183819    0.495189
   84      16      2706      2690   128.078       124    0.192226    0.495735
   85      16      2740      2724   128.171       136  0.00416184    0.494977
   86      16      2773      2757   128.215       132     2.12443    0.494323
   87      16      2802      2786   128.075       116    0.356535    0.495284
   88      16      2834      2818   128.074       128     1.81285    0.496497
   89      16      2870      2854   128.252       144    0.889957    0.496552
   90      16      2902      2886    128.25       128     0.63383    0.496893
   91      16      2931      2915   128.115       116    0.285331    0.496523
   92      16      2963      2947   128.113       128    0.481716    0.497276
   93      16      2992      2976   127.983       116    0.405642    0.497421
   94      16      3024      3008   127.983       128    0.221838    0.497783
   95      16      3057      3041   128.025       132    0.593946    0.497604
   96      16      3090      3074   128.066       132  0.00412563    0.497358
   97      16      3119      3103   127.942       116  0.00427057    0.497278
   98      16      3147      3131   127.779       112    0.479791    0.498667
   99      16      3178      3162   127.741       124    0.277126    0.498719
2025-09-06T18:04:22.358522+0000 min lat: 0.00300177 max lat: 2.36401 avg lat: 0.498278
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3214      3198   127.903       144     1.15525    0.498278
  101      16      3245      3229   127.865       124  0.00952695    0.498026
  102      16      3278      3262   127.905       132    0.191092     0.49801
  103      16      3308      3292   127.828       120    0.640412    0.497861
  104      16      3339      3323   127.791       124     1.95322    0.497655
  105      16      3370      3354   127.755       124     0.36248    0.497762
  106      16      3401      3385   127.719       124    0.230054    0.497732
  107      16      3432      3416   127.685       124      1.6113    0.498838
  108      16      3461      3445   127.576       116     1.37972     0.49905
  109      16      3491      3475   127.507       120     1.53033    0.498807
  110      16      3523      3507   127.511       128    0.256755     0.49812
  111      16      3554      3538   127.479       124     1.62511    0.499213
  112      16      3586      3570   127.484       128     0.15323    0.498856
  113      16      3613      3597   127.311       108    0.197853    0.499316
  114      16      3647      3631   127.387       136    0.699708    0.500205
  115      16      3677      3661   127.323       120  0.00527622    0.499934
  116      16      3714      3698   127.501       148    0.184277    0.498855
  117      16      3743      3727   127.403       116     0.74199    0.499486
  118      16      3774      3758   127.374       124    0.197858    0.500192
  119      16      3806      3790   127.379       128    0.106141    0.500039
2025-09-06T18:04:42.360597+0000 min lat: 0.00299179 max lat: 2.36401 avg lat: 0.499846
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      3833      3817   127.217       108    0.209887    0.499846
  121      16      3865      3849   127.224       128     0.28772    0.499589
  122      16      3900      3884   127.328       140    0.369694    0.499959
  123      16      3928      3912   127.203       112    0.901468    0.501408
  124      16      3966      3950   127.403       152    0.246088    0.500682
  125      16      4002      3986   127.536       144    0.513193    0.499997
  126      16      4034      4018    127.54       128    0.152934    0.499787
  127      16      4066      4050   127.543       128    0.580802    0.500011
  128      16      4104      4088   127.734       152    0.208105    0.499095
  129      16      4133      4117   127.643       116    0.121422    0.499389
  130      16      4163      4147   127.584       120   0.0093297    0.499208
  131      16      4196      4180   127.618       132    0.503652    0.499427
  132      16      4227      4211    127.59       124  0.00991318    0.499936
  133      16      4259      4243   127.593       128    0.678579    0.499814
  134      16      4296      4280   127.745       148    0.687167    0.499521
  135      16      4325      4309   127.658       116   0.0100017    0.499755
  136      16      4357      4341   127.661       128    0.681882    0.499721
  137      16      4389      4373   127.663       128    0.437386    0.499762
  138      15      4422      4407   127.723       136    0.266809    0.499486
  139      16      4454      4438   127.696       124    0.785598    0.499721
2025-09-06T18:05:02.363174+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.499186
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      4486      4470   127.698       128    0.978208    0.499186
  141      16      4519      4503   127.729       132    0.383326    0.499622
  142      16      4551      4535    127.73       128    0.392751    0.499306
  143      16      4580      4564   127.648       116     1.63569    0.499092
  144      16      4612      4596   127.651       128    0.154411    0.499328
  145      16      4641      4625    127.57       116    0.117135     0.49903
  146      16      4670      4654   127.491       116    0.149681    0.499358
  147      16      4701      4685   127.467       124     1.44813    0.499939
  148      16      4729      4713   127.362       112    0.193016    0.500454
  149      16      4762      4746   127.393       132  0.00999606     0.50068
  150      16      4796      4780   127.451       136     1.49075    0.500327
  151      16      4829      4813   127.481       132  0.00429479    0.500366
  152      15      4860      4845   127.484       128    0.487669    0.500598
  153      16      4890      4874   127.409       116    0.253615    0.500241
  154      16      4926      4910   127.516       144     1.58565    0.500283
  155      16      4956      4940   127.468       120     1.02842     0.50047
  156      16      4989      4973   127.497       132    0.250503    0.500153
  157      16      5019      5003   127.449       120    0.811717     0.49989
  158      16      5051      5035   127.452       128     1.79872    0.500299
  159      16      5082      5066   127.431       124    0.242845    0.500416
2025-09-06T18:05:22.365656+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.501239
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      15      5111      5096   127.384       120    0.932433    0.501239
  161      16      5145      5129   127.413       132    0.588867    0.500704
  162      16      5180      5164    127.49       140    0.184245    0.500433
  163      16      5209      5193    127.42       116    0.648174    0.500812
  164      16      5243      5227   127.472       136     0.35894    0.500498
  165      16      5275      5259   127.475       128    0.498188    0.500551
  166      16      5312      5296   127.598       148    0.229462    0.500109
  167      16      5345      5329   127.625       132    0.648958    0.499956
  168      16      5382      5366   127.746       148  0.00464392    0.499642
  169      16      5412      5396     127.7       120  0.00487543    0.499385
  170      16      5448      5432   127.796       144    0.483526    0.499331
  171      16      5477      5461   127.727       116    0.134459    0.498981
  172      16      5508      5492   127.705       124     1.20563     0.49972
  173      16      5536      5520   127.614       112     1.41229    0.499839
  174      16      5567      5551   127.593       124    0.150071      0.5001
  175      16      5598      5582   127.573       124       1.337    0.500327
  176      16      5632      5616    127.62       136    0.227541    0.499891
  177      16      5663      5647     127.6       124    0.880482    0.500197
  178      16      5700      5684   127.714       148    0.686781     0.49984
  179      16      5733      5717   127.738       132    0.689276    0.499841
2025-09-06T18:05:42.367930+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.499441
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      5769      5753   127.828       144  0.00934018    0.499441
  181      16      5798      5782   127.763       116    0.417504    0.499759
  182      16      5833      5817    127.83       140    0.487476    0.499255
  183      16      5866      5850   127.853       132  0.00436459    0.498963
  184      16      5896      5880    127.81       120    0.724651    0.499236
  185      16      5931      5915   127.876       140    0.609206    0.498968
  186      16      5961      5945   127.834       120    0.158926    0.498613
  187      16      5993      5977   127.834       128    0.158989    0.499061
  188      16      6022      6006   127.771       116    0.216444    0.499332
  189      16      6054      6038   127.772       128    0.301984    0.499349
  190      16      6087      6071   127.795       132    0.317744    0.499137
  191      16      6119      6103   127.796       128    0.936912    0.499401
  192      16      6151      6135   127.797       128     1.02494    0.499482
  193      16      6180      6164   127.735       116    0.544841    0.499793
  194      16      6216      6200   127.819       144  0.00325828    0.499267
  195      16      6247      6231     127.8       124  0.00964512    0.499378
  196      16      6284      6268   127.903       148    0.527441    0.498964
  197      16      6315      6299   127.883       124    0.653334    0.499203
  198      16      6348      6332   127.903       132     0.60937    0.499184
  199      16      6379      6363   127.884       124    0.702848      0.4991
2025-09-06T18:06:02.370072+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.498833
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      6411      6395   127.884       128  0.00439229    0.498833
  201      16      6441      6425   127.845       120    0.188819    0.498802
  202      16      6479      6463   127.964       152    0.254095    0.498431
  203      16      6510      6494   127.945       124     1.70776    0.498496
  204      16      6542      6526   127.945       128     1.91442    0.498379
  205      16      6574      6558   127.945       128    0.236667    0.498629
  206      16      6602      6586   127.868       112    0.378327    0.498789
  207      16      6638      6622   127.946       144    0.217619    0.498294
  208      16      6669      6653   127.927       124     2.02291    0.498433
  209      16      6707      6691   128.042       152    0.170598     0.49804
  210      16      6738      6722   128.022       124    0.179312    0.498469
  211      16      6769      6753   128.003       124      1.0306     0.49887
  212      16      6802      6786   128.022       132    0.563567    0.498485
  213      16      6836      6820   128.059       136    0.335909    0.498259
  214      16      6868      6852   128.059       128     1.41516    0.498427
  215      16      6901      6885   128.077       132    0.372781    0.498417
  216      16      6935      6919   128.114       136  0.00354265    0.498329
  217      16      6965      6949   128.076       120     1.28527    0.498382
  218      16      6998      6982   128.094       132    0.152524    0.498051
  219      16      7027      7011   128.039       116    0.286006    0.498115
2025-09-06T18:06:22.372424+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.498139
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      7059      7043   128.039       128  0.00392951    0.498139
  221      16      7090      7074   128.021       124    0.278302    0.498314
  222      16      7119      7103   127.966       116  0.00933923    0.498315
  223      16      7150      7134   127.948       124      1.5835    0.498636
  224      16      7181      7165   127.931       124     0.60661    0.498852
  225      16      7214      7198   127.949       132    0.162822    0.498868
  226      16      7248      7232   127.984       136    0.432917    0.498761
  227      16      7279      7263   127.967       124    0.312608    0.498858
  228      16      7316      7300   128.055       148    0.379037     0.49854
  229      16      7348      7332   128.054       128    0.222594    0.498587
  230      16      7381      7365   128.071       132    0.638524    0.498586
  231      16      7410      7394   128.019       116   0.0102252    0.498793
  232      16      7442      7426   128.019       128    0.342715    0.498807
  233      16      7475      7459   128.036       132    0.283528    0.498647
  234      16      7502      7486    127.95       108     0.32061    0.498819
  235      16      7532      7516   127.916       120    0.423818    0.499084
  236      16      7564      7548   127.917       128     1.05935    0.499046
  237      16      7597      7581   127.934       132      0.3387    0.499032
  238      16      7628      7612   127.917       124    0.470145    0.499028
  239      16      7661      7645   127.934       132    0.276346    0.499079
2025-09-06T18:06:42.374509+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.499308
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      7691      7675   127.901       120    0.630996    0.499308
  241      16      7724      7708   127.918       132    0.653379    0.499234
  242      16      7759      7743   127.968       140    0.382368    0.499024
  243      16      7794      7778   128.017       140  0.00389913    0.498749
  244      16      7823      7807   127.968       116     1.42538     0.49891
  245      16      7855      7839   127.968       128     1.20369    0.498941
  246      16      7892      7876    128.05       148    0.391859    0.498501
  247      16      7925      7909   128.066       132     0.15492    0.498133
  248      16      7956      7940   128.049       124     1.61748     0.49845
  249      16      7987      7971   128.033       124  0.00471052    0.498296
  250      16      8017      8001   128.001       120      1.9789    0.498616
  251      16      8049      8033   128.001       128    0.719657     0.49844
  252      16      8079      8063   127.969       120    0.146483    0.498867
  253      16      8116      8100   128.048       148    0.328742    0.498303
  254      16      8153      8137   128.126       148    0.287973    0.497678
  255      16      8186      8170   128.141       132    0.208681    0.498128
  256      16      8223      8207   128.219       148     1.42616    0.497903
  257      16      8258      8242   128.265       140    0.157802    0.497907
  258      16      8288      8272   128.233       120    0.227412     0.49754
  259      16      8321      8305   128.247       132    0.134095    0.497593
2025-09-06T18:07:02.376904+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.497811
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      8350      8334     128.2       116    0.597617    0.497811
  261      16      8381      8365   128.184       124     1.46212    0.497966
  262      16      8412      8396   128.168       124    0.159406    0.497709
  263      16      8446      8430   128.197       136     2.19868    0.497516
  264      16      8477      8461   128.181       124    0.147281    0.497576
  265      16      8509      8493   128.181       128    0.621998    0.497842
  266      16      8538      8522   128.135       116    0.267175    0.498179
  267      16      8573      8557   128.179       140     1.26651    0.498039
  268      16      8616      8600   128.343       172  0.00420792    0.497629
  269      16      8647      8631   128.327       124  0.00442937    0.497774
  270      16      8680      8664    128.34       132    0.010723    0.497621
  271      16      8711      8695   128.324       124    0.576568     0.49769
  272      16      8746      8730   128.367       140    0.931825    0.497696
  273      16      8777      8761   128.351       124    0.387395    0.497583
  274      16      8814      8798   128.423       148   0.0101271    0.497352
  275      16      8846      8830   128.421       128    0.728713      0.4974
  276      16      8879      8863   128.434       132    0.760732    0.497348
  277      16      8909      8893   128.403       120    0.581887    0.497206
  278      16      8941      8925   128.402       128     0.57315    0.497442
  279      16      8970      8954   128.357       116     1.18111    0.497594
2025-09-06T18:07:22.378922+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.497688
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      8999      8983   128.313       116     1.12803    0.497688
  281      16      9033      9017   128.341       136     1.15891    0.497727
  282      16      9064      9048   128.325       124  0.00320911    0.497727
  283      16      9098      9082   128.352       136    0.234972    0.497717
  284      16      9130      9114   128.351       128    0.404145    0.497589
  285      16      9163      9147   128.364       132    0.313078    0.497574
  286      16      9195      9179   128.362       128    0.185359    0.497249
  287      16      9230      9214   128.403       140    0.349621    0.497439
  288      16      9257      9241   128.332       108    0.489215    0.497554
  289      16      9291      9275   128.358       136    0.889491    0.497648
  290      16      9321      9305    128.33       120     1.13856    0.497582
  291      16      9355      9339   128.356       136    0.118936    0.497343
  292      16      9386      9370   128.341       124    0.488898    0.497485
  293      16      9417      9401   128.326       124     0.20372    0.497714
  294      16      9445      9429    128.27       112    0.427066    0.497997
  295      16      9479      9463   128.297       136    0.265628    0.497932
  296      16      9511      9495   128.296       128    0.418079    0.497851
  297      16      9537      9521   128.214       104    0.429063    0.498098
  298      16      9568      9552     128.2       124     0.23329    0.498192
  299      16      9606      9590   128.279       152    0.910775    0.497995
2025-09-06T18:07:42.381048+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.497992
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      9637      9621   128.265       124    0.608789    0.497992
  301      16      9673      9657   128.317       144    0.255794    0.497664
  302      16      9703      9687   128.289       120     1.12719    0.497914
  303      16      9737      9721   128.315       136    0.737864    0.497867
  304      16      9767      9751   128.287       120    0.262968    0.497586
  305      16      9804      9788   128.352       148     0.94377    0.497755
  306      16      9835      9819   128.338       124    0.487529    0.497775
  307      16      9870      9854   128.376       140  0.00387504     0.49759
  308      16      9899      9883   128.335       116    0.569328    0.497772
  309      16      9934      9918   128.373       140    0.442329    0.497707
  310      16      9974      9958   128.475       160  0.00426233     0.49715
  311      16     10004      9988   128.448       120    0.154078    0.497264
  312      16     10041     10025    128.51       148  0.00293449    0.496826
  313      16     10071     10055   128.483       120    0.520624    0.496829
  314      16     10107     10091   128.533       144     1.78664    0.496991
  315      16     10137     10121   128.505       120     1.25159      0.4971
  316      16     10170     10154   128.516       132    0.680811    0.497233
  317      16     10207     10191   128.578       148   0.0030961    0.496805
  318      16     10239     10223   128.576       128     0.12769    0.496662
  319      16     10270     10254   128.562       124    0.273306    0.496537
2025-09-06T18:08:02.383278+0000 min lat: 0.00293449 max lat: 2.36401 avg lat: 0.496638
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     10305     10289   128.597       140    0.646938    0.496638
  321      16     10338     10322   128.608       132     0.35997    0.496654
  322      16     10366     10350   128.556       112    0.116863    0.496595
  323      16     10402     10386   128.604       144     1.58206    0.496481
  324      16     10436     10420   128.627       136  0.00312157    0.496438
  325      16     10467     10451   128.613       124     0.41607    0.496649
  326      16     10497     10481   128.586       120    0.281999    0.496806
  327      16     10528     10512   128.572       124    0.167662    0.496854
  328      16     10558     10542   128.546       120    0.459874     0.49711
  329      16     10589     10573   128.532       124    0.536428    0.497116
  330      16     10617     10601   128.482       112    0.289528    0.497259
  331      16     10647     10631   128.456       120    0.186033    0.497293
  332      16     10682     10666   128.491       140    0.990862     0.49706
  333      16     10713     10697   128.477       124    0.250918    0.497082
  334      16     10742     10726    128.44       116    0.243726    0.497182
  335      16     10774     10758   128.439       128    0.822844     0.49746
  336      16     10805     10789   128.425       124    0.147175    0.497183
  337      16     10833     10817   128.377       112    0.179403    0.497404
  338      16     10867     10851   128.399       136    0.130057    0.497052
  339      16     10897     10881   128.374       120    0.162936    0.496973
2025-09-06T18:08:22.385343+0000 min lat: 0.00293449 max lat: 2.37439 avg lat: 0.497198
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     10927     10911    128.35       120    0.321889    0.497198
  341      16     10962     10946   128.384       140    0.318759    0.497307
  342      16     10995     10979   128.394       132  0.00998479    0.497422
  343      16     11024     11008   128.358       116  0.00958292    0.497612
  344      16     11058     11042    128.38       136  0.00394575    0.497569
  345      16     11087     11071   128.344       116    0.802075    0.497839
  346      16     11120     11104   128.355       132    0.781992    0.497708
  347      16     11152     11136   128.354       128    0.361345    0.497657
  348      16     11183     11167   128.341       124     1.19181    0.497768
  349      16     11219     11203   128.386       144    0.462863    0.497449
  350      16     11248     11232   128.351       116     1.18204     0.49769
  351      16     11275     11259   128.293       108     0.23239    0.497753
  352      16     11311     11295   128.337       144  0.00412308    0.497561
  353      16     11351     11335   128.427       160    0.241098    0.497104
  354      16     11380     11364   128.392       116    0.105852     0.49726
  355      16     11411     11395   128.379       124    0.004077    0.497323
  356      16     11440     11424   128.345       116    0.295806    0.497335
  357      16     11474     11458   128.366       136     1.36883    0.497611
  358      16     11505     11489   128.354       124    0.347334    0.497475
  359      16     11537     11521   128.353       128     1.76748    0.497587
2025-09-06T18:08:42.387535+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497728
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     11568     11552   128.341       124    0.251515    0.497728
  361      16     11596     11580   128.295       112     0.58145    0.497844
  362      16     11630     11614   128.317       136  0.00938028     0.49783
  363      16     11665     11649   128.349       140     1.10464    0.497648
  364      16     11699     11683    128.37       136    0.230716     0.49737
  365      16     11732     11716    128.38       132    0.336426    0.497389
  366      16     11762     11746   128.357       120    0.356947    0.497469
  367      16     11792     11776   128.334       120    0.246938    0.497574
  368      16     11822     11806   128.311       120    0.126272    0.497851
  369      16     11856     11840   128.332       136    0.678884     0.49775
  370      16     11887     11871    128.32       124     1.51797    0.497687
  371      16     11920     11904    128.33       132    0.138966    0.497656
  372      16     11952     11936   128.329       128     1.99888    0.497817
  373      16     11983     11967   128.318       124  0.00486749    0.497818
  374      16     12017     12001   128.338       136    0.247335    0.497399
  375      16     12051     12035   128.358       136  0.00401746    0.497509
  376      16     12081     12065   128.336       120    0.302152    0.497711
  377      16     12114     12098   128.346       132   0.0033209    0.497667
  378      16     12147     12131   128.356       132        0.24    0.497573
  379      16     12177     12161   128.333       120     1.52235     0.49764
2025-09-06T18:09:02.389397+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497534
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     12212     12196   128.364       140   0.0046087    0.497534
  381      16     12246     12230   128.384       136    0.746804    0.497757
  382      16     12277     12261   128.373       124    0.628044    0.497782
  383      16     12310     12294   128.382       132    0.222153    0.497561
  384      16     12342     12326   128.381       128    0.154559     0.49742
  385      16     12377     12361   128.411       140    0.272726    0.497449
  386      16     12410     12394    128.42       132   0.0103072    0.497426
  387      16     12440     12424   128.399       120    0.391835    0.497366
  388      16     12470     12454   128.377       120    0.470033    0.497568
  389      16     12504     12488   128.397       136     1.45165    0.497424
  390      16     12541     12525   128.447       148     1.66159    0.497426
  391      16     12583     12567   128.548       168    0.249008    0.496928
  392      16     12613     12597   128.526       120    0.611948    0.497084
  393      16     12646     12630   128.535       132   0.0900817    0.497108
  394      16     12672     12656   128.473       104    0.144182    0.497042
  395      16     12705     12689   128.481       132    0.173422    0.496869
  396      16     12736     12720    128.47       124    0.167057    0.496983
  397      16     12767     12751   128.459       124    0.413334    0.497113
  398      16     12802     12786   128.488       140  0.00421749     0.49736
  399      16     12832     12816   128.467       120    0.547067    0.497295
2025-09-06T18:09:22.391081+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497417
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     12860     12844   128.425       112     0.20035    0.497417
  401      16     12894     12878   128.444       136    0.300884    0.497506
  402      16     12924     12908   128.423       120    0.650455    0.497617
  403      16     12955     12939   128.412       124    0.483085    0.497552
  404      16     12984     12968   128.381       116     0.37074    0.497539
  405      16     13017     13001    128.39       132     1.02046    0.497653
  406      16     13049     13033   128.389       128    0.381209    0.497661
  407      16     13084     13068   128.418       140  0.00510433    0.497403
  408      16     13123     13107   128.485       156    0.898466    0.497262
  409      16     13156     13140   128.494       132    0.274762    0.497247
  410      16     13187     13171   128.483       124    0.451803    0.497308
  411      16     13216     13200   128.453       116    0.362619     0.49745
  412      16     13252     13236    128.49       144    0.278338    0.497328
  413      16     13285     13269   128.499       132    0.422513    0.497206
  414      16     13319     13303   128.517       136    0.751218    0.497228
  415      16     13349     13333   128.496       120    0.679691     0.49728
  416      16     13382     13366   128.505       132    0.198557    0.497268
  417      16     13413     13397   128.494       124     0.42791    0.497144
  418      16     13446     13430   128.502       132     0.18337    0.497163
  419      16     13483     13467   128.549       148  0.00940943    0.497045
2025-09-06T18:09:42.393164+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497132
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     13512     13496   128.519       116    0.188586    0.497132
  421      16     13539     13523    128.47       108    0.412928    0.497319
  422      16     13570     13554   128.459       124      1.1894     0.49742
  423      16     13602     13586   128.458       128    0.759103    0.497457
  424      16     13634     13618   128.457       128   0.0039798    0.497371
  425      16     13665     13649   128.447       124    0.445506    0.497423
  426      16     13701     13685   128.483       144    0.449915    0.497339
  427      16     13734     13718   128.491       132  0.00442037    0.497332
  428      16     13767     13751   128.499       132    0.225952    0.497251
  429      16     13802     13786   128.526       140    0.289564    0.497016
  430      16     13832     13816   128.506       120     1.51509    0.497158
  431      16     13863     13847   128.496       124    0.611599    0.497208
  432      16     13896     13880   128.504       132  0.00442923    0.497194
  433      16     13930     13914   128.521       136    0.991144    0.497074
  434      16     13965     13949   128.548       140    0.210871    0.497006
  435      16     14000     13984   128.574       140     1.36111    0.496777
  436      16     14033     14017   128.582       132    0.593111    0.496784
  437      16     14064     14048   128.571       124     1.79177    0.496686
  438      16     14096     14080    128.57       128      1.9373    0.496929
  439      16     14127     14111    128.56       124    0.409606    0.497037
2025-09-06T18:10:02.395110+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.496873
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     14162     14146   128.585       140    0.333377    0.496873
  441      16     14194     14178   128.584       128    0.659246     0.49701
  442      16     14223     14207   128.556       116    0.235254    0.497142
  443      16     14257     14241   128.572       136    0.566517     0.49702
  444      16     14289     14273   128.571       128     1.06056    0.497045
  445      16     14322     14306   128.579       132    0.123148    0.496853
  446      16     14355     14339   128.586       132    0.167563    0.496948
  447      16     14390     14374   128.612       140  0.00942157    0.496858
  448      16     14423     14407   128.619       132    0.198323    0.496648
  449      16     14456     14440   128.627       132     1.74442     0.49655
  450      16     14493     14477    128.67       148     0.25956    0.496533
  451      16     14525     14509   128.668       128  0.00961006    0.496509
  452      16     14554     14538    128.64       116    0.260049    0.496455
  453      16     14588     14572   128.657       136    0.795083    0.496583
  454      16     14620     14604   128.655       128    0.719707    0.496504
  455      16     14652     14636   128.654       128    0.665609    0.496614
  456      16     14689     14673   128.696       148    0.331472    0.496422
  457      16     14718     14702   128.668       116    0.516364    0.496535
  458      16     14746     14730   128.632       112    0.709876    0.496753
  459      16     14776     14760   128.613       120    0.600357    0.496775
2025-09-06T18:10:22.397169+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.496844
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     14806     14790   128.594       120    0.184671    0.496844
  461      16     14842     14826   128.628       144     0.16644     0.49679
  462      16     14872     14856   128.609       120     1.36522    0.496756
  463      16     14906     14890   128.625       136      1.3291    0.496853
  464      16     14934     14918   128.589       112    0.107218    0.496809
  465      16     14966     14950   128.588       128    0.182398     0.49665
  466      16     14995     14979   128.561       116    0.216869    0.496707
  467      15     15031     15016   128.602       148  0.00317107    0.496749
  468      16     15061     15045   128.575       116    0.753498    0.496893
  469      16     15097     15081   128.608       144  0.00512218    0.496919
  470      16     15126     15110   128.581       116     1.00771    0.496943
  471      16     15156     15140   128.563       120      1.2514    0.496968
  472      16     15185     15169   128.536       116    0.408152     0.49715
  473      16     15217     15201   128.535       128    0.293741    0.497095
  474      16     15248     15232   128.526       124   0.0103043    0.497054
  475      16     15279     15263   128.516       124    0.301884    0.497099
  476      16     15310     15294   128.506       124     1.28326    0.497195
  477      16     15342     15326   128.505       128    0.304381    0.497129
  478      16     15371     15355   128.479       116    0.248527    0.497386
  479      16     15404     15388   128.487       132    0.298936    0.497271
2025-09-06T18:10:42.399750+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497136
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     15435     15419   128.477       124    0.195316    0.497136
  481      16     15468     15452   128.484       132    0.346396    0.497212
  482      16     15500     15484   128.483       128     1.64571     0.49719
  483      16     15534     15518   128.499       136    0.224342    0.497128
  484      16     15569     15553   128.523       140    0.175996      0.4971
  485      16     15600     15584   128.513       124    0.492342    0.497199
  486      16     15631     15615   128.504       124  0.00989235    0.497282
  487      16     15662     15646   128.495       124    0.240696     0.49724
  488      16     15692     15676   128.477       120    0.357431    0.497365
  489      16     15722     15706    128.46       120   0.0103692    0.497439
  490      16     15754     15738   128.459       128    0.338325    0.497354
  491      16     15795     15779   128.531       164  0.00476621    0.497076
  492      16     15828     15812   128.538       132    0.278356    0.496901
  493      16     15860     15844   128.537       128    0.151095    0.496907
  494      16     15893     15877   128.544       132  0.00938923    0.496878
  495      16     15928     15912   128.567       140     1.86682    0.496876
  496      16     15959     15943   128.558       124    0.213942    0.496948
  497      16     15988     15972   128.533       116    0.112799    0.496948
  498      16     16022     16006   128.548       136    0.780793    0.497022
  499      16     16050     16034   128.515       112  0.00925413    0.497204
2025-09-06T18:11:02.401800+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497338
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     16080     16064   128.498       120    0.809226    0.497338
  501      16     16114     16098   128.512       136    0.295834    0.497283
  502      16     16147     16131   128.519       132     1.14744    0.497259
  503      16     16181     16165   128.534       136    0.746442    0.497218
  504      16     16210     16194   128.509       116    0.424045    0.497287
  505      16     16243     16227   128.516       132    0.143943     0.49709
  506      16     16275     16259   128.515       128     1.54463    0.497075
  507      16     16305     16289   128.498       120    0.168674     0.49707
  508      16     16342     16326   128.537       148     1.66319    0.497186
  509      16     16372     16356    128.52       120    0.207433    0.497178
  510      16     16400     16384   128.488       112    0.550999    0.497291
  511      16     16437     16421   128.526       148    0.164468    0.497246
  512      16     16472     16456   128.548       140        1.27    0.497025
  513      16     16503     16487   128.539       124    0.187189    0.496938
  514      16     16533     16517   128.523       120      2.1235    0.497068
  515      16     16563     16547   128.506       120     1.53499    0.497213
  516      16     16595     16579   128.505       128    0.317535    0.497196
  517      16     16626     16610   128.496       124  0.00927053    0.497198
  518      16     16655     16639   128.472       116    0.444921    0.497262
  519      16     16690     16674   128.494       140    0.127166    0.497244
2025-09-06T18:11:22.403507+0000 min lat: 0.00291441 max lat: 2.58677 avg lat: 0.497203
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     16723     16707   128.501       132    0.356817    0.497203
  521      16     16751     16735   128.469       112    0.212603     0.49732
  522      16     16787     16771   128.499       144     1.67788    0.497206
  523      16     16819     16803   128.498       128     1.70007    0.497199
  524      16     16848     16832   128.474       116    0.391044    0.497213
  525      16     16879     16863   128.466       124    0.504305     0.49739
  526      16     16914     16898   128.488       140    0.114174    0.497298
  527      16     16946     16930   128.487       128    0.164095    0.497207
  528      16     16974     16958   128.455       112    0.178093    0.497213
  529      16     17007     16991   128.462       132    0.220819    0.497258
  530      16     17036     17020   128.439       116     1.73876    0.497325
  531      16     17072     17056   128.468       144  0.00446915    0.497221
  532      16     17105     17089   128.474       132   0.0100984    0.497302
  533      16     17134     17118   128.451       116    0.442438    0.497368
  534      16     17168     17152   128.465       136    0.586345    0.497403
  535      16     17200     17184   128.464       128  0.00446841    0.497412
  536      16     17229     17213   128.441       116    0.406347    0.497432
  537      16     17260     17244   128.433       124     1.87135    0.497431
  538      16     17296     17280   128.462       144  0.00376238    0.497202
  539      16     17328     17312   128.461       128    0.134363    0.497312
2025-09-06T18:11:42.405461+0000 min lat: 0.00291441 max lat: 2.58677 avg lat: 0.497366
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     17359     17343   128.452       124    0.301831    0.497366
  541      16     17393     17377   128.466       136     1.12597    0.497437
  542      16     17422     17406   128.443       116  0.00516023    0.497583
  543      16     17460     17444   128.487       152    0.832456    0.497443
  544      16     17493     17477   128.493       132    0.668396    0.497399
  545      16     17526     17510     128.5       132     0.29177    0.497228
  546      16     17558     17542   128.499       128    0.209682    0.497328
  547      16     17594     17578   128.527       144    0.463534    0.497163
  548      16     17624     17608   128.511       120  0.00932494    0.497003
  549      16     17659     17643   128.532       140       1.938    0.497098
  550      16     17685     17669   128.488       104     1.81873    0.497165
  551      16     17717     17701   128.487       128    0.255139    0.497356
  552      16     17751     17735     128.5       136      0.1688    0.497142
  553      16     17780     17764   128.478       116    0.264821    0.497443
  554      16     17812     17796   128.477       128      1.4639    0.497441
  555      16     17842     17826   128.461       120  0.00531299    0.497339
  556      16     17872     17856   128.446       120     0.16497    0.497339
  557      16     17901     17885   128.424       116     1.26066    0.497556
  558      16     17932     17916   128.416       124     0.41316    0.497665
  559      16     17969     17953   128.451       148   0.0103564    0.497552
2025-09-06T18:12:02.407207+0000 min lat: 0.00291441 max lat: 2.58677 avg lat: 0.497469
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     18002     17986   128.457       132    0.451508    0.497469
  561      16     18029     18013   128.421       108    0.272648     0.49754
  562      16     18069     18053   128.477       160  0.00310979    0.497458
  563      16     18099     18083   128.462       120     1.04354    0.497444
  564      16     18133     18117   128.475       136  0.00432079    0.497516
  565      16     18163     18147    128.46       120     0.73703    0.497569
  566      16     18195     18179   128.459       128    0.610731    0.497535
  567      16     18228     18212   128.466       132     1.04296    0.497543
  568      16     18256     18240   128.437       112    0.221794    0.497525
  569      16     18288     18272   128.436       128    0.277707     0.49758
  570      16     18321     18305   128.442       132    0.476984    0.497555
  571      16     18355     18339   128.455       136    0.305267    0.497396
  572      16     18383     18367   128.426       112    0.215524    0.497386
  573      16     18411     18395   128.398       112     0.12944    0.497608
  574      16     18442     18426    128.39       124    0.146728    0.497584
  575      16     18471     18455   128.369       116     1.74543    0.497728
  576      16     18508     18492   128.403       148    0.637231    0.497643
  577      16     18541     18525   128.409       132    0.988765     0.49774
  578      16     18572     18556   128.401       124     1.36235    0.497728
  579      16     18608     18592   128.428       144    0.180485    0.497658
2025-09-06T18:12:22.409018+0000 min lat: 0.00291441 max lat: 2.58677 avg lat: 0.497377
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     18644     18628   128.455       144    0.225034    0.497377
  581      16     18681     18665   128.489       148    0.277628    0.497166
  582      16     18711     18695   128.474       120    0.175279    0.497284
  583      16     18742     18726   128.466       124     0.12403    0.497227
  584      16     18777     18761   128.486       140    0.393684    0.497191
  585      16     18808     18792   128.478       124    0.004276    0.497436
  586      16     18837     18821   128.457       116     0.31828    0.497418
  587      16     18867     18851   128.443       120    0.212868    0.497595
  588      16     18899     18883   128.442       128     0.26975    0.497515
  589      16     18938     18922   128.489       156    0.729405    0.497448
  590      16     18967     18951   128.467       116    0.277506    0.497323
  591      16     18999     18983   128.467       128    0.189254    0.497343
  592      16     19032     19016   128.472       132    0.371961    0.497404
  593      16     19071     19055   128.519       156      0.1841    0.497135
  594      16     19103     19087   128.518       128    0.222556    0.497222
  595      16     19133     19117   128.504       120   0.0042575    0.497356
  596      16     19165     19149   128.503       128    0.490665    0.497375
  597      16     19197     19181   128.502       128     1.10989    0.497391
  598      16     19228     19212   128.494       124     1.40869    0.497253
  599      16     19264     19248    128.52       144  0.00499039    0.497101
2025-09-06T18:12:42.410876+0000 min lat: 0.0029092 max lat: 2.58677 avg lat: 0.497095
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      15     19298     19283   128.539       140    0.215206    0.497095
  601      16     19328     19312   128.518       116    0.627554    0.497154
  602      16     19362     19346   128.531       136    0.188996    0.497192
  603      16     19393     19377   128.523       124    0.198215    0.497057
  604      15     19430     19415   128.562       152    0.185431    0.497055
  605      16     19458     19442   128.528       108    0.409243    0.497241
  606      16     19489     19473   128.521       124    0.548066    0.497261
  607      16     19521     19505    128.52       128     1.51215    0.497263
  608      16     19552     19536   128.512       124    0.214283    0.497237
  609      16     19585     19569   128.518       132    0.182495     0.49729
  610      16     19614     19598   128.497       116     0.39932    0.497339
  611      16     19651     19635   128.529       148  0.00471426     0.49734
  612      16     19684     19668   128.535       132    0.225195    0.497286
  613      16     19718     19702   128.547       136    0.368972    0.497151
  614      16     19751     19735   128.553       132  0.00518921     0.49721
  615      16     19782     19766   128.545       124    0.179398    0.497117
  616      16     19810     19794   128.518       112  0.00452079    0.497266
  617      16     19845     19829   128.537       140     1.32393    0.497254
  618      16     19877     19861   128.536       128   0.0048083    0.497315
  619      16     19908     19892   128.529       124     1.24167    0.497288
2025-09-06T18:13:02.413325+0000 min lat: 0.0029092 max lat: 2.58677 avg lat: 0.497173
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  620      16     19945     19929    128.56       148    0.178552    0.497173
  621      16     19977     19961   128.559       128  0.00954097    0.497243
  622      16     20009     19993   128.558       128    0.394515     0.49719
  623      16     20038     20022   128.538       116    0.825439    0.497259
  624      16     20073     20057   128.556       140    0.358026    0.497163
  625      16     20105     20089   128.556       128    0.337663    0.497203
  626      16     20136     20120   128.548       124    0.156441    0.497215
  627      16     20174     20158   128.586       152  0.00956967     0.49695
  628      16     20207     20191   128.591       132    0.283944    0.497063
  629      16     20239     20223    128.59       128     1.19294    0.497002
  630      15     20269     20254   128.583       124    0.898571    0.497116
  631      16     20299     20283   128.563       116     1.32028    0.497156
  632      16     20334     20318   128.581       140    0.151176    0.497075
  633      16     20364     20348   128.567       120    0.314777    0.497157
  634      16     20393     20377   128.547       116    0.559239    0.497228
  635      16     20430     20414   128.578       148    0.371777    0.497144
  636      16     20460     20444   128.565       120    0.389895    0.497118
  637      16     20494     20478   128.576       136    0.471799    0.496986
  638      16     20527     20511   128.582       132    0.125774    0.497019
  639      16     20561     20545   128.593       136    0.353528     0.49702
2025-09-06T18:13:22.415323+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.49705
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  640      16     20593     20577   128.592       128    0.242985     0.49705
  641      16     20626     20610   128.598       132    0.501938    0.497057
  642      16     20662     20646   128.621       144    0.900247     0.49696
  643      16     20696     20680   128.633       136    0.190519    0.496823
  644      16     20727     20711   128.626       124  0.00453893     0.49687
  645      16     20757     20741   128.612       120    0.356759    0.496907
  646      16     20787     20771   128.599       120      1.6677    0.496939
  647      16     20819     20803   128.598       128     0.32035    0.496891
  648      16     20852     20836   128.603       132    0.477224    0.496956
  649      16     20884     20868   128.602       128    0.153023    0.496921
  650      16     20916     20900   128.601       128  0.00959983    0.496959
  651      16     20947     20931   128.594       124    0.269298    0.497082
  652      16     20979     20963   128.593       128     0.33172    0.496983
  653      16     21013     20997   128.605       136    0.469143    0.497067
  654      16     21044     21028   128.598       124    0.122382    0.496946
  655      16     21074     21058   128.584       120    0.291679    0.497025
  656      16     21103     21087   128.565       116    0.517548    0.497162
  657      16     21134     21118   128.558       124    0.408259    0.497201
  658      16     21165     21149   128.551       124    0.523012     0.49724
  659      16     21193     21177   128.526       112     1.06902    0.497306
2025-09-06T18:13:42.417076+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.497292
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  660      16     21223     21207   128.513       120    0.191887    0.497292
  661      16     21262     21246   128.555       156  0.00956121    0.497196
  662      16     21299     21283   128.584       148   0.0650634    0.496954
  663      16     21330     21314   128.577       124    0.215255     0.49685
  664      16     21367     21351   128.606       148     2.07179    0.496859
  665      16     21398     21382     128.6       124    0.931835    0.496978
  666      16     21431     21415   128.605       132     1.30033    0.496981
  667      16     21460     21444   128.586       116  0.00987722    0.497004
  668      16     21493     21477   128.591       132    0.872785     0.49707
  669      16     21530     21514    128.62       148    0.725586    0.496934
  670      16     21561     21545   128.613       124    0.176933     0.49689
  671      16     21594     21578   128.618       132  0.00413939     0.49696
  672      16     21628     21612   128.629       136  0.00937935    0.496941
  673      16     21658     21642   128.616       120     1.67573    0.496893
  674      16     21693     21677   128.633       140    0.346008    0.496861
  675      16     21728     21712    128.65       140  0.00957494    0.496894
  676      16     21760     21744   128.649       128    0.951695     0.49686
  677      16     21799     21783   128.689       156    0.491442    0.496723
  678      16     21829     21813   128.676       120     1.14673      0.4967
  679      16     21862     21846   128.681       132    0.223762     0.49661
2025-09-06T18:14:02.419549+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496638
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  680      16     21894     21878    128.68       128      1.6374    0.496638
  681      16     21930     21914   128.703       144    0.142292    0.496563
  682      16     21963     21947   128.707       132  0.00485172    0.496554
  683      16     21994     21978     128.7       124    0.444104    0.496678
  684      16     22027     22011   128.705       132     0.51119    0.496691
  685      16     22059     22043   128.704       128    0.550832    0.496662
  686      16     22092     22076   128.709       132    0.804719    0.496635
  687      16     22126     22110    128.72       136    0.219605    0.496516
  688      16     22154     22138   128.695       112    0.436212    0.496581
  689      16     22191     22175   128.723       148    0.671074    0.496484
  690      16     22223     22207   128.722       128    0.275451    0.496467
  691      16     22251     22235   128.698       112      1.8259    0.496511
  692      16     22286     22270   128.714       140      0.4962    0.496535
  693      16     22316     22300   128.702       120    0.163581    0.496572
  694      16     22348     22332   128.701       128     1.48503    0.496635
  695      16     22380     22364     128.7       128     1.15408    0.496575
  696      16     22415     22399   128.716       140    0.494578    0.496585
  697      16     22449     22433   128.726       136    0.179676    0.496485
  698      16     22480     22464    128.72       124    0.442427    0.496593
  699      16     22513     22497   128.724       132  0.00957192    0.496549
2025-09-06T18:14:22.421243+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496549
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  700      16     22545     22529   128.723       128    0.275863    0.496549
  701      16     22574     22558   128.705       116    0.303003     0.49659
  702      16     22602     22586   128.681       112    0.534422     0.49653
  703      16     22636     22620   128.692       136     0.44303    0.496636
  704      16     22667     22651   128.685       124    0.304948    0.496726
  705      16     22699     22683   128.684       128    0.381443    0.496784
  706      16     22728     22712   128.666       116     0.29457    0.496729
  707      16     22763     22747   128.682       140    0.940405    0.496801
  708      16     22796     22780   128.687       132    0.159662    0.496719
  709      15     22826     22811    128.68       124    0.232827    0.496669
  710      16     22863     22847   128.702       144    0.163895    0.496553
  711      16     22892     22876   128.684       116    0.155947    0.496713
  712      16     22925     22909   128.688       132    0.222762    0.496598
  713      16     22959     22943   128.699       136    0.346232    0.496502
  714      16     22992     22976   128.703       132    0.897818    0.496611
  715      16     23029     23013    128.73       148    0.370455    0.496495
  716      16     23064     23048   128.746       140      1.2132    0.496396
  717      16     23098     23082   128.756       136    0.246506    0.496254
  718      16     23129     23113   128.749       124    0.365073    0.496335
  719      16     23157     23141   128.726       112     1.85838    0.496445
2025-09-06T18:14:42.423286+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496496
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  720      16     23188     23172   128.719       124   0.0957588    0.496496
  721      16     23220     23204   128.718       128  0.00445857    0.496445
  722      16     23251     23235   128.712       124    0.223392    0.496647
  723      16     23282     23266   128.705       124    0.469905    0.496617
  724      16     23317     23301   128.721       140    0.541303    0.496521
  725      16     23350     23334   128.725       132    0.273359    0.496601
  726      16     23384     23368   128.735       136    0.120779    0.496537
  727      16     23414     23398   128.723       120    0.222828    0.496453
  728      16     23448     23432   128.733       136    0.245954    0.496331
  729      16     23483     23467   128.749       140  0.00940112    0.496276
  730      16     23514     23498   128.742       124    0.232406    0.496379
  731      16     23542     23526   128.719       112     1.81049    0.496427
  732      16     23576     23560   128.729       136    0.432407     0.49647
  733      16     23607     23591   128.723       124    0.334011    0.496532
  734      16     23641     23625   128.733       136    0.574354    0.496594
  735      16     23671     23655   128.721       120    0.724521    0.496602
  736      16     23703     23687    128.72       128      1.0496    0.496614
  737      16     23735     23719   128.719       128    0.263671    0.496569
  738      16     23772     23756   128.745       148  0.00475692    0.496481
  739      16     23801     23785   128.728       116    0.189403    0.496458
2025-09-06T18:15:02.424997+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496606
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  740      16     23830     23814    128.71       116      1.0814    0.496606
  741      16     23862     23846    128.71       128    0.664569    0.496664
  742      16     23899     23883   128.735       148  0.00462682    0.496582
  743      16     23931     23915   128.734       128  0.00935809    0.496551
  744      16     23964     23948   128.739       132    0.637936     0.49652
  745      16     23998     23982   128.749       136    0.366649     0.49648
  746      16     24030     24014   128.748       128  0.00398182    0.496465
  747      16     24065     24049   128.763       140     0.36963    0.496393
  748      16     24096     24080   128.756       124    0.260242    0.496367
  749      16     24133     24117   128.782       148    0.476668    0.496355
  750      16     24164     24148   128.775       124    0.689863    0.496313
  751      16     24194     24178   128.764       120    0.553792    0.496387
  752      16     24226     24210   128.763       128    0.649896    0.496446
  753      16     24256     24240   128.751       120    0.733163    0.496483
  754      16     24287     24271   128.745       124    0.433617    0.496505
  755      16     24316     24300   128.728       116     1.11075    0.496563
  756      16     24350     24334   128.737       136    0.846018    0.496552
  757      16     24381     24365   128.731       124  0.00384142    0.496557
  758      16     24411     24395    128.72       120    0.514282    0.496632
  759      16     24446     24430   128.735       140   0.0093646    0.496599
2025-09-06T18:15:22.427243+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496444
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  760      16     24478     24462   128.734       128    0.244911    0.496444
  761      16     24507     24491   128.717       116    0.522814    0.496472
  762      16     24539     24523   128.716       128    0.413511    0.496636
  763      16     24569     24553   128.704       120     1.00365    0.496691
  764      16     24597     24581   128.683       112    0.779107    0.496755
  765      16     24627     24611   128.671       120     0.35555    0.496732
  766      16     24662     24646   128.686       140    0.310008    0.496748
  767      16     24690     24674   128.664       112    0.913338    0.496795
  768      16     24721     24705   128.658       124     1.15482    0.496861
  769      16     24750     24734   128.642       116  0.00494844    0.496869
  770      16     24783     24767   128.646       132    0.788786    0.496894
  771      16     24812     24796    128.63       116  0.00416075    0.496924
  772      16     24844     24828   128.629       128     1.53599    0.496903
  773      16     24881     24865   128.654       148    0.248943    0.496759
  774      16     24912     24896   128.648       124    0.419131     0.49676
  775      16     24940     24924   128.626       112    0.390904    0.496929
  776      16     24970     24954   128.615       120    0.215752    0.496969
  777      16     25004     24988   128.625       136     0.16379    0.496896
  778      16     25036     25020   128.624       128    0.224807     0.49689
  779      16     25067     25051   128.618       124    0.358542    0.496859
2025-09-06T18:15:42.429209+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496927
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  780      16     25099     25083   128.617       128     1.83034    0.496927
  781      16     25132     25116   128.621       132    0.414996    0.496949
  782      16     25164     25148    128.62       128    0.443258    0.496933
  783      16     25194     25178   128.609       120    0.193056    0.497022
  784      16     25223     25207   128.593       116    0.503691    0.497087
  785      16     25254     25238   128.587       124    0.612914    0.497088
  786      16     25284     25268   128.577       120    0.861878    0.497184
  787      16     25312     25296   128.555       112    0.815574    0.497204
  788      16     25342     25326   128.545       120    0.221453    0.497268
  789      16     25372     25356   128.534       120    0.299789    0.497362
  790      16     25405     25389   128.538       132  0.00440725    0.497327
  791      16     25438     25422   128.542       132    0.432757    0.497305
  792      16     25468     25452   128.532       120    0.801785    0.497335
  793      16     25501     25485   128.536       132  0.00396724    0.497345
  794      16     25533     25517   128.535       128    0.332969    0.497369
  795      15     25563     25548    128.53       124    0.244494    0.497404
  796      16     25598     25582   128.539       136     1.09812    0.497361
  797      16     25630     25614   128.538       128      1.0083    0.497367
  798      16     25660     25644   128.528       120  0.00443719    0.497309
  799      16     25690     25674   128.517       120    0.193485    0.497376
2025-09-06T18:16:02.431412+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.497351
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  800      16     25722     25706   128.516       128     1.22193    0.497351
  801      16     25760     25744   128.545       152    0.170412    0.497321
  802      16     25790     25774   128.535       120    0.330782    0.497357
  803      16     25821     25805   128.529       124    0.210534     0.49724
  804      16     25853     25837   128.528       128     1.52072    0.497341
  805      16     25886     25870   128.533       132      0.3297    0.497337
  806      16     25912     25896   128.502       104    0.781525    0.497414
  807      16     25953     25937   128.546       164    0.357451    0.497305
  808      16     25987     25971   128.556       136    0.511665    0.497295
  809      16     26018     26002    128.55       124    0.596082    0.497304
  810      16     26049     26033   128.544       124     1.28158    0.497302
  811      16     26081     26065   128.544       128    0.156761    0.497153
  812      16     26114     26098   128.548       132     0.25792    0.497219
  813      16     26144     26128   128.537       120    0.140097    0.497305
  814      16     26172     26156   128.517       112    0.420569    0.497361
  815      16     26209     26193   128.541       148     1.65198    0.497279
  816      16     26241     26225    128.54       128    0.281951    0.497258
  817      16     26273     26257   128.539       128    0.420018    0.497234
  818      16     26307     26291   128.549       136    0.187523    0.497253
  819      16     26337     26321   128.538       120    0.490617    0.497229
2025-09-06T18:16:22.433435+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.497075
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  820      16     26377     26361   128.576       160  0.00392565    0.497075
  821      16     26410     26394   128.581       132    0.734829    0.497178
  822      16     26442     26426    128.58       128  0.00407791    0.497176
  823      16     26475     26459   128.584       132     0.40133     0.49711
  824      16     26513     26497   128.612       152    0.425043     0.49704
  825      16     26547     26531   128.621       136    0.464611    0.496976
  826      15     26577     26562   128.616       124    0.937893    0.497032
  827      16     26615     26599   128.639       148    0.398004     0.49687
  828      16     26641     26625   128.609       104    0.286453    0.496906
  829      16     26672     26656   128.604       124    0.907882    0.497087
  830      16     26703     26687   128.598       124     1.26354    0.497104
  831      16     26732     26716   128.583       116    0.152784    0.497122
  832      16     26770     26754   128.611       152    0.355234    0.496969
  833      16     26799     26783   128.596       116    0.171364    0.497075
  834      16     26829     26813   128.586       120    0.346634    0.497072
  835      16     26859     26843   128.575       120    0.217705    0.497142
  836      15     26896     26881   128.603       152    0.308229    0.496968
  837      16     26927     26911   128.593       120    0.773591    0.497103
  838      16     26959     26943   128.592       128  0.00375755    0.497084
  839      16     26990     26974   128.587       124    0.203202     0.49697
2025-09-06T18:16:42.436401+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496851
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  840      16     27027     27011    128.61       148     0.16814    0.496851
  841      16     27059     27043   128.609       128  0.00439834    0.496926
  842      16     27090     27074   128.604       124     1.64512    0.497005
  843      16     27126     27110   128.622       144  0.00400234    0.496932
  844      16     27157     27141   128.616       124    0.283174     0.49694
  845      16     27190     27174    128.62       132    0.128569    0.496869
  846      16     27221     27205   128.615       124    0.199545    0.496968
  847      16     27252     27236   128.609       124     1.56657    0.496966
  848      16     27283     27267   128.604       124     1.24649    0.496999
  849      16     27321     27305   128.632       152    0.637402    0.496872
  850      16     27353     27337   128.631       128    0.223792    0.496922
  851      16     27385     27369    128.63       128    0.148846    0.496943
  852      16     27418     27402   128.634       132     1.18254    0.496925
  853      16     27451     27435   128.638       132  0.00994704    0.496933
  854      16     27482     27466   128.632       124  0.00976012    0.496957
  855      16     27512     27496   128.622       120  0.00493533    0.496998
  856      16     27548     27532    128.64       144    0.338676    0.496927
  857      16     27576     27560   128.621       112    0.153618    0.496936
  858      16     27605     27589   128.606       116    0.803978    0.497038
  859      16     27638     27622    128.61       132    0.523701    0.497039
2025-09-06T18:17:02.438696+0000 min lat: 0.00286085 max lat: 2.74757 avg lat: 0.497126
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  860      16     27666     27650   128.591       112     1.12031    0.497126
  861      16     27699     27683   128.595       132     1.19636    0.497092
  862      16     27728     27712    128.58       116  0.00924344    0.497038
  863      16     27759     27743   128.575       124     2.32524    0.497079
  864      16     27792     27776   128.579       132  0.00962865    0.497139
  865      16     27823     27807   128.573       124     0.22464     0.49715
  866      16     27853     27837   128.563       120    0.447585     0.49712
  867      16     27885     27869   128.563       128  0.00426165    0.497173
  868      16     27917     27901   128.562       128    0.514195    0.497227
  869      16     27949     27933   128.561       128    0.192849    0.497186
  870      16     27979     27963   128.552       120     1.65747    0.497172
  871      16     28014     27998   128.565       140      0.2004    0.497075
  872      16     28047     28031   128.569       132    0.394598    0.497161
  873      16     28076     28060   128.554       116    0.170698    0.497216
  874      16     28109     28093   128.558       132     0.29469    0.497259
  875      16     28141     28125   128.558       128    0.993899    0.497255
  876      16     28172     28156   128.552       124    0.243069     0.49721
  877      16     28203     28187   128.547       124     1.39689    0.497296
  878      16     28233     28217   128.537       120    0.530768    0.497293
  879      16     28264     28248   128.532       124    0.189211    0.497325
2025-09-06T18:17:22.440688+0000 min lat: 0.00286085 max lat: 2.74757 avg lat: 0.497323
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  880      16     28296     28280   128.532       128     0.16957    0.497323
  881      16     28325     28309   128.517       116    0.220706    0.497413
  882      16     28354     28338   128.503       116     1.45203     0.49747
  883      16     28387     28371   128.507       132    0.322121     0.49751
  884      16     28421     28405   128.516       136    0.330126    0.497429
  885      16     28452     28436    128.51       124     1.43398    0.497384
  886      16     28488     28472   128.528       144  0.00948231    0.497297
  887      16     28519     28503   128.523       124     0.27187    0.497184
  888      16     28549     28533   128.513       120    0.234242     0.49725
  889      16     28579     28563   128.504       120    0.811712    0.497372
  890      16     28611     28595   128.503       128    0.631672    0.497492
  891      16     28643     28627   128.502       128    0.344119     0.49747
  892      16     28675     28659   128.502       128    0.131958     0.49744
  893      16     28706     28690   128.497       124    0.382912    0.497336
  894      16     28739     28723   128.501       132    0.469578    0.497424
  895      16     28774     28758   128.513       140    0.358718    0.497343
  896      16     28804     28788   128.504       120      0.7592    0.497435
  897      16     28832     28816   128.486       112     1.15076     0.49752
  898      16     28865     28849   128.489       132    0.334928    0.497518
  899      16     28897     28881   128.489       128    0.227154    0.497494
2025-09-06T18:17:42.442778+0000 min lat: 0.00286085 max lat: 2.74757 avg lat: 0.497484
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  900      12     28925     28913   128.488       128    0.131318    0.497484
Total time run:       900.45
Total reads made:     28925
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   128.491
Average IOPS:         32
Stddev IOPS:          2.64739
Max IOPS:             43
Min IOPS:             26
Average Latency(s):   0.497611
Max latency(s):       2.74757
Min latency(s):       0.00286085

Rados, random read, pool with 32 PGs

[ceph: root@ms-01-01 /]# date
Sat Sep  6 18:20:43 UTC 2025
[ceph: root@ms-01-01 /]# rados bench -p testbench 900 rand
hints = 1
Click to see full run
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       2         2         0         0         0           -           0
    1      16        45        29   115.977       116    0.946455    0.334365
    2      16        81        65   129.978       144    0.151799    0.402858
    3      16       112        96   127.981       124     1.18352    0.432497
    4      16       143       127   126.981       124      1.3917    0.435819
    5      16       178       162   129.582       140     1.46459    0.445311
    6      16       208       192   127.982       120    0.178275     0.45001
    7      16       243       227   129.696       140  0.00718082    0.466022
    8      15       275       260   129.981       132     1.41221    0.469916
    9      16       306       290    128.87       120    0.300853     0.47135
   10      16       340       324   129.582       136  0.00488823    0.471236
   11      16       368       352   127.982       112    0.909512    0.483933
   12      16       402       386   128.649       136  0.00465737    0.477073
   13      16       437       421   129.519       140    0.239667    0.478698
   14      16       469       453    129.41       128     1.33108    0.479356
   15      16       497       481   128.248       112    0.319827    0.484032
   16      16       530       514   128.482       132    0.247962    0.483375
   17      16       559       543   127.746       116    0.400901    0.487744
   18      16       594       578   128.426       140    0.935051    0.483598
   19      16       629       613   129.034       140    0.839456    0.487638
2025-09-06T18:21:08.560249+0000 min lat: 0.00179416 max lat: 1.92525 avg lat: 0.484987
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       658       642   128.382       116    0.170675    0.484987
   21      16       693       677   128.934       140     1.64293     0.48565
   22      16       724       708   128.709       124    0.399857    0.488586
   23      16       755       739   128.503       124    0.394626    0.488468
   24      16       787       771   128.482       128    0.500534    0.489694
   25      16       819       803   128.462       128     1.36508    0.486698
   26      16       849       833   128.136       120     1.71311    0.486285
   27      16       880       864   127.982       124     0.13139    0.492606
   28      16       908       892   127.411       112     0.49139    0.493764
   29      16       940       924   127.431       128     1.33473    0.494019
   30      16       970       954   127.183       120    0.236683     0.49613
   31      16       999       983   126.822       116    0.376773    0.495851
   32      16      1029      1013   126.608       120    0.131229    0.496592
   33      16      1064      1048   127.013       140    0.721859    0.497289
   34      16      1094      1078   126.807       120    0.680327    0.498672
   35      16      1124      1108   126.612       120    0.129594     0.49848
   36      16      1156      1140    126.65       128    0.260548    0.499325
   37      16      1191      1175    127.01       140    0.950264    0.498749
   38      16      1226      1210   127.352       140  0.00300055    0.496089
   39      16      1262      1246   127.778       144   0.0096846    0.493494
2025-09-06T18:21:28.562582+0000 min lat: 0.00179416 max lat: 2.05151 avg lat: 0.492609
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1299      1283   128.283       148   0.0029702    0.492609
   41      16      1327      1311   127.886       112    0.701586    0.493137
   42      16      1362      1346   128.174       140    0.215376     0.49327
   43      16      1396      1380   128.356       136    0.316092    0.493723
   44      16      1424      1408   127.983       112      0.4967    0.495102
   45      16      1456      1440   127.983       128    0.868343    0.494754
   46      16      1489      1473    128.07       132    0.114139    0.494572
   47      16      1524      1508   128.324       140    0.622904    0.493379
   48      16      1556      1540   128.317       128    0.395353    0.493841
   49      16      1585      1569   128.065       116     1.12993    0.495359
   50      16      1618      1602   128.144       132    0.759356    0.495252
   51      16      1649      1633   128.062       124     0.20947    0.495789
   52      16      1681      1665   128.061       128    0.288825    0.495784
   53      16      1713      1697    128.06       128     1.22006    0.494936
   54      16      1744      1728   127.984       124    0.271546    0.494275
   55      16      1782      1766    128.42       152     2.33718     0.49515
   56      16      1813      1797   128.341       124    0.221193     0.49545
   57      16      1846      1830   128.405       132    0.408704    0.494118
   58      16      1871      1855   127.915       100    0.206133    0.494185
   59      16      1905      1889   128.052       136    0.608614    0.495752
2025-09-06T18:21:48.564823+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.496001
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1940      1924   128.251       140    0.638511    0.496001
   61      16      1969      1953    128.05       116    0.988119    0.496579
   62      16      2001      1985   128.049       128     1.13147     0.49677
   63      16      2041      2025   128.556       160    0.824897    0.495086
   64      16      2073      2057   128.547       128  0.00413838    0.495086
   65      16      2104      2088   128.477       124    0.940774    0.494608
   66      16      2139      2123   128.651       140    0.568543     0.49462
   67      16      2171      2155   128.641       128    0.761599    0.494895
   68      16      2206      2190   128.808       140    0.352015    0.494522
   69      16      2236      2220    128.68       120    0.691905    0.494641
   70      16      2269      2253   128.727       132    0.178285    0.493757
   71      16      2298      2282   128.548       116    0.209528    0.494817
   72      16      2331      2315   128.596       132     0.71842     0.49443
   73      16      2367      2351   128.806       144    0.633731    0.494508
   74      16      2397      2381   128.687       120    0.724632     0.49424
   75      16      2427      2411   128.571       120    0.217372    0.494385
   76      16      2461      2445   128.669       136  0.00975976    0.494507
   77      16      2497      2481   128.868       144    0.948524    0.493288
   78      16      2527      2511   128.754       120    0.474774    0.493999
   79      16      2563      2547   128.947       144  0.00296521    0.494124
2025-09-06T18:22:08.566992+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.494351
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2593      2577   128.835       120  0.00998061    0.494351
   81      16      2627      2611   128.923       136     0.26053    0.493034
   82      16      2660      2644    128.96       132   0.0041278     0.49355
   83      16      2687      2671   128.708       108    0.209411    0.494385
   84      16      2718      2702   128.651       124    0.214842    0.493519
   85      16      2750      2734   128.644       128    0.991513    0.494759
   86      16      2779      2763   128.496       116     0.48483    0.495547
   87      16      2816      2800    128.72       148      1.0391    0.494766
   88      16      2852      2836   128.894       144     1.33733    0.493983
   89      16      2886      2870   128.973       136    0.144908    0.493069
   90      16      2920      2904   129.051       136     1.44358    0.492961
   91      16      2952      2936    129.04       128  0.00351137    0.492733
   92      16      2985      2969   129.072       132     1.68695    0.492663
   93      16      3019      3003   129.146       136    0.510286    0.492775
   94      16      3053      3037   129.219       136     1.09382    0.493047
   95      16      3083      3067   129.122       120     1.17341    0.492773
   96      16      3113      3097   129.026       120    0.502812    0.493636
   97      16      3147      3131   129.098       136    0.286989    0.492621
   98      16      3180      3164   129.128       132  0.00946412    0.492579
   99      16      3218      3202   129.358       152    0.747498    0.492664
2025-09-06T18:22:28.569218+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.492763
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3249      3233   129.305       124    0.372276    0.492763
  101      16      3283      3267   129.371       136      1.0016    0.492541
  102      16      3317      3301   129.436       136     1.08775    0.492317
  103      16      3351      3335   129.499       136    0.184754    0.491702
  104      16      3381      3365   129.408       120     1.26675    0.492134
  105      16      3413      3397   129.394       128   0.0106307    0.492805
  106      16      3445      3429   129.381       128     1.04456    0.492575
  107      16      3478      3462   129.405       132    0.196917    0.491999
  108      16      3509      3493   129.355       124    0.476888    0.492411
  109      16      3539      3523   129.269       120    0.557053    0.492509
  110      16      3571      3555   129.258       128    0.277216    0.491915
  111      16      3604      3588   129.282       132    0.127825    0.492271
  112      16      3636      3620   129.271       128     1.58018    0.492753
  113      16      3664      3648   129.118       112    0.591644    0.493138
  114      16      3693      3677   129.002       116       1.497    0.493721
  115      16      3726      3710   129.028       132      0.1885      0.4936
  116      16      3760      3744   129.088       136     1.38589    0.493821
  117      16      3789      3773   128.976       116    0.882342    0.494318
  118      16      3823      3807   129.036       136  0.00511954    0.494104
  119      16      3855      3839   129.027       128   0.0095501    0.494178
2025-09-06T18:22:48.571281+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.494404
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      3882      3866   128.852       108    0.284555    0.494404
  121      16      3912      3896   128.779       120    0.290215    0.494104
  122      16      3945      3929   128.805       132    0.344239    0.494458
  123      16      3980      3964   128.896       140   0.0046088    0.494267
  124      16      4013      3997   128.921       132     1.23598    0.494125
  125      16      4045      4029   128.913       128     1.40003    0.494237
  126      16      4077      4061   128.906       128  0.00367598    0.494337
  127      16      4109      4093   128.899       128    0.894082    0.494527
  128      16      4142      4126   128.923       132    0.559329    0.494463
  129      16      4175      4159   128.946       132    0.349044    0.493989
  130      16      4208      4192    128.97       132    0.145473     0.49432
  131      16      4240      4224   128.962       128    0.956884    0.494224
  132      16      4271      4255   128.925       124     1.52672    0.494109
  133      16      4303      4287   128.917       128     1.66106    0.493823
  134      16      4333      4317   128.851       120     0.34692    0.494447
  135      16      4364      4348   128.815       124    0.157463    0.494363
  136      16      4394      4378    128.75       120     1.23643    0.495595
  137      16      4422      4406   128.628       112  0.00419456     0.49581
  138      16      4461      4445   128.826       156    0.383102    0.495293
  139      16      4492      4476   128.791       124    0.205475    0.495058
2025-09-06T18:23:08.573442+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.495406
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      4523      4507   128.757       124     1.30392    0.495406
  141      16      4556      4540    128.78       132    0.133783    0.495038
  142      16      4587      4571   128.746       124    0.426306    0.495213
  143      16      4616      4600   128.657       116    0.171095    0.495771
  144      16      4644      4628   128.541       112    0.762945    0.496191
  145      16      4676      4660   128.537       128    0.516176    0.496139
  146      15      4711      4696   128.643       144    0.233451    0.495784
  147      16      4742      4726   128.584       120    0.154194    0.495735
  148      16      4774      4758    128.58       128    0.147627    0.495399
  149      16      4803      4787   128.495       116    0.158759    0.495721
  150      16      4837      4821   128.545       136     1.72453    0.496043
  151      16      4867      4851   128.489       120    0.825901    0.496556
  152      16      4901      4885   128.538       136     1.39913    0.496193
  153      16      4934      4918    128.56       132    0.146294    0.496083
  154      16      4965      4949   128.531       124    0.507093    0.495363
  155      16      4999      4983   128.579       136     0.16174    0.495709
  156      16      5030      5014   128.549       124    0.139627    0.495535
  157      16      5063      5047   128.571       132    0.571443    0.496089
  158      16      5095      5079   128.568       128    0.347691     0.49633
  159      16      5126      5110   128.539       124    0.253666    0.496215
2025-09-06T18:23:28.575737+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.49617
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      5160      5144   128.585       136     1.08115     0.49617
  161      16      5190      5174   128.532       120    0.501026    0.496242
  162      16      5223      5207   128.553       132    0.751707    0.496462
  163      16      5258      5242   128.623       140  0.00969016    0.495906
  164      16      5288      5272   128.571       120    0.381706    0.495999
  165      16      5324      5308   128.664       144    0.169395    0.495791
  166      16      5353      5337   128.588       116    0.177165    0.495859
  167      16      5382      5366   128.512       116    0.177833    0.496177
  168      16      5413      5397   128.485       124     1.11535    0.496541
  169      16      5444      5428   128.459       124     0.12443    0.496446
  170      16      5478      5462   128.503       136     0.19199    0.495887
  171      16      5507      5491    128.43       116     1.97573    0.496706
  172      16      5539      5523   128.427       128     1.11569    0.496987
  173      16      5572      5556   128.448       132     1.22677    0.496703
  174      16      5601      5585   128.376       116    0.189057    0.496661
  175      16      5634      5618   128.397       132    0.118491    0.496933
  176      16      5666      5650   128.394       128    0.752882    0.496778
  177      16      5698      5682   128.392       128     1.61013    0.497023
  178      16      5732      5716   128.435       136    0.101837    0.496795
  179      16      5764      5748   128.432       128  0.00959418    0.496393
2025-09-06T18:23:48.577846+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.496385
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      5798      5782   128.474       136     1.61284    0.496385
  181      16      5826      5810   128.383       112    0.262099    0.496502
  182      16      5863      5847   128.491       148    0.891377    0.496596
  183      16      5893      5877   128.444       120   0.0094383    0.496525
  184      16      5925      5909   128.442       128  0.00944839    0.496628
  185      16      5955      5939   128.396       120    0.663242    0.496726
  186      16      5986      5970   128.373       124    0.516342      0.4968
  187      16      6019      6003   128.392       132    0.179988    0.496856
  188      16      6049      6033   128.347       120     1.67309    0.497213
  189      16      6082      6066   128.366       132    0.196339     0.49714
  190      16      6110      6094    128.28       112     1.55856    0.497167
  191      16      6139      6123   128.216       116    0.156225    0.497232
  192      16      6179      6163   128.381       160  0.00307982    0.497211
  193      16      6210      6194   128.358       124     0.10009    0.496853
  194      16      6239      6223   128.295       116  0.00961365    0.497516
  195      16      6268      6252   128.231       116  0.00969469    0.497593
  196      16      6303      6287   128.291       140    0.529988    0.497639
  197      16      6336      6320    128.31       132    0.850143    0.497468
  198      16      6368      6352   128.309       128    0.570711    0.497475
  199      16      6401      6385   128.327       132    0.877029    0.497499
2025-09-06T18:24:08.580258+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.497449
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      6432      6416   128.305       124    0.897945    0.497449
  201      16      6463      6447   128.284       124    0.800495    0.497522
  202      16      6491      6475   128.203       112    0.220941    0.497713
  203      16      6526      6510   128.261       140    0.169825    0.497401
  204      16      6558      6542    128.26       128    0.264377    0.497628
  205      16      6594      6578   128.337       144    0.236943    0.497295
  206      16      6629      6613   128.393       140    0.199815    0.497162
  207      16      6661      6645   128.391       128      1.4958    0.497038
  208      16      6693      6677   128.389       128    0.737803    0.497065
  209      16      6723      6707   128.349       120     0.29945     0.49744
  210      16      6754      6738   128.328       124    0.231791    0.497284
  211      16      6790      6774   128.402       144  0.00460067    0.497211
  212      16      6820      6804   128.363       120     1.25609    0.497399
  213      16      6850      6834   128.323       120  0.00418781    0.497331
  214      16      6886      6870   128.397       144    0.170761    0.497064
  215      16      6916      6900   128.357       120    0.121822    0.496989
  216      16      6948      6932   128.356       128     1.56089    0.497002
  217      16      6982      6966   128.391       136     0.43261    0.496975
  218      16      7016      7000   128.426       136     0.56955    0.496696
  219      16      7046      7030   128.387       120    0.552879    0.497116
2025-09-06T18:24:28.582424+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.497387
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      7075      7059   128.331       116    0.170507    0.497387
  221      16      7104      7088   128.275       116     1.32798    0.497532
  222      16      7133      7117    128.22       116    0.348238    0.497664
  223      16      7165      7149   128.219       128     1.11767    0.498058
  224      16      7194      7178   128.164       116    0.511028    0.497928
  225      16      7225      7209   128.146       124     1.58515    0.498057
  226      16      7255      7239   128.109       120     1.57742    0.498154
  227      16      7289      7273   128.144       136    0.159036    0.497763
  228      16      7321      7305   128.143       128     1.49895    0.497809
  229      16      7354      7338    128.16       132    0.196115    0.497368
  230      16      7385      7369   128.142       124      1.5907    0.497826
  231      16      7427      7411   128.315       168     1.06215    0.497535
  232      16      7455      7439   128.244       112     1.33585    0.497895
  233      16      7486      7470   128.226       124    0.201014    0.497985
  234      16      7517      7501   128.208       124    0.419794    0.497977
  235      16      7548      7532    128.19       124    0.642755    0.498122
  236      16      7577      7561   128.138       116    0.447203    0.498227
  237      16      7608      7592   128.121       124    0.505713    0.498515
  238      16      7644      7628   128.187       144     1.37865    0.498076
  239      16      7672      7656    128.12       112    0.215345    0.498351
2025-09-06T18:24:48.584253+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498413
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      7704      7688   128.119       128     0.44655    0.498413
  241      16      7736      7720   128.118       128   0.0095287     0.49834
  242      16      7770      7754   128.151       136    0.312365    0.498192
  243      16      7799      7783   128.101       116    0.186793    0.498197
  244      16      7831      7815     128.1       128  0.00976769    0.498387
  245      16      7866      7850   128.149       140    0.744113    0.498367
  246      16      7898      7882   128.148       128     1.09274    0.498359
  247      16      7928      7912   128.115       120    0.771925    0.498289
  248      16      7958      7942   128.082       120    0.185194    0.498479
  249      16      7991      7975   128.098       132    0.149698    0.498665
  250      16      8020      8004    128.05       116    0.145959    0.498522
  251      16      8056      8040   128.113       144     1.39841    0.498289
  252      16      8091      8075    128.16       140    0.341722    0.498246
  253      16      8123      8107    128.16       128     1.29678    0.498181
  254      16      8152      8136   128.112       116    0.271019    0.498384
  255      16      8181      8165   128.064       116    0.937672    0.498516
  256      16      8216      8200   128.111       140    0.444886     0.49827
  257      16      8250      8234   128.141       136    0.380401    0.498106
  258      16      8287      8271   128.218       148   0.0045436    0.497991
  259      16      8317      8301   128.186       120    0.611223     0.49803
2025-09-06T18:25:08.586493+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498067
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      8350      8334   128.201       132   0.0101976    0.498067
  261      16      8382      8366     128.2       128     1.04709    0.498295
  262      16      8409      8393   128.123       108     1.27154    0.498406
  263      16      8443      8427   128.153       136    0.241117    0.498411
  264      16      8472      8456   128.107       116    0.170082    0.498369
  265      16      8506      8490   128.137       136  0.00994715    0.498201
  266      16      8539      8523   128.151       132    0.715941     0.49844
  267      16      8574      8558   128.195       140    0.140435    0.498149
  268      16      8606      8590   128.195       128     1.08167     0.49798
  269      15      8636      8621   128.179       124    0.168536    0.498029
  270      16      8670      8654   128.193       132    0.377636    0.498331
  271      16      8700      8684   128.163       120    0.304312    0.498187
  272      16      8734      8718   128.192       136    0.855053    0.498077
  273      16      8764      8748   128.162       120     1.49923    0.498309
  274      16      8795      8779   128.146       124     1.07114    0.498403
  275      16      8825      8809   128.117       120     1.33039    0.498513
  276      16      8856      8840   128.102       124      1.0153    0.498665
  277      16      8888      8872   128.101       128    0.266581    0.498333
  278      16      8922      8906    128.13       136    0.150895    0.498368
  279      16      8954      8938   128.129       128  0.00399762     0.49828
2025-09-06T18:25:28.588706+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498103
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      8987      8971   128.143       132  0.00448132    0.498103
  281      16      9019      9003   128.142       128    0.202703     0.49796
  282      16      9047      9031   128.085       112    0.194229    0.498191
  283      16      9075      9059   128.028       112    0.382489    0.498693
  284      16      9107      9091   128.028       128     0.41276    0.498735
  285      16      9145      9129   128.112       152      0.3976    0.498416
  286      16      9173      9157   128.056       112  0.00943455    0.498539
  287      16      9202      9186   128.014       116    0.308213    0.498759
  288      16      9230      9214   127.958       112      0.8936    0.499062
  289      16      9263      9247   127.972       132     1.25778    0.498924
  290      16      9293      9277   127.944       120    0.255986    0.498873
  291      16      9328      9312   127.986       140    0.322974    0.499022
  292      16      9358      9342   127.958       120   0.0105642    0.498938
  293      16      9389      9373   127.945       124    0.318374    0.499173
  294      16      9420      9404   127.931       124    0.121503    0.498982
  295      16      9448      9432   127.877       112    0.167404    0.499404
  296      16      9482      9466   127.905       136    0.454141    0.499127
  297      15      9515      9500   127.932       136    0.393165    0.499284
  298      16      9543      9527   127.865       108    0.267608    0.499356
  299      16      9578      9562   127.906       140    0.934571    0.499439
2025-09-06T18:25:48.590746+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499393
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      9609      9593   127.892       124    0.265582    0.499393
  301      16      9639      9623   127.866       120    0.511249    0.499411
  302      16      9669      9653    127.84       120     0.19636    0.499404
  303      16      9702      9686   127.854       132  0.00399089    0.499379
  304      16      9738      9722   127.907       144  0.00493717     0.49907
  305      16      9771      9755    127.92       132    0.581672    0.499197
  306      16      9802      9786   127.907       124     1.30593    0.499292
  307      16      9837      9821   127.947       140    0.142791    0.499211
  308      16      9867      9851   127.921       120    0.183639    0.499245
  309      16      9900      9884   127.934       132     1.25266    0.499281
  310      16      9929      9913   127.896       116    0.865585    0.499464
  311      16      9958      9942   127.857       116  0.00999793    0.499492
  312      16      9991      9975    127.87       132   0.0094562    0.499456
  313      16     10020     10004   127.833       116    0.370611    0.499639
  314      16     10050     10034   127.807       120     1.26844    0.499674
  315      16     10085     10069   127.846       140  0.00942776    0.499424
  316      16     10115     10099   127.821       120      0.2604     0.49963
  317      16     10150     10134    127.86       140    0.554266    0.499397
  318      16     10180     10164   127.835       120     0.42475    0.499374
  319      16     10214     10198    127.86       136    0.527648     0.49932
2025-09-06T18:26:08.593006+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499472
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     10246     10230   127.861       128    0.204159    0.499472
  321      16     10277     10261   127.849       124    0.951069    0.499617
  322      16     10312     10296   127.886       140    0.179124     0.49949
  323      16     10348     10332   127.936       144     0.84076     0.49935
  324      16     10383     10367   127.973       140    0.403396    0.498935
  325      16     10417     10401   127.998       136    0.354131    0.498854
  326      16     10446     10430   127.961       116     1.58113      0.4989
  327      16     10477     10461   127.949       124     1.33911    0.499134
  328      16     10507     10491   127.925       120    0.175309    0.499179
  329      16     10539     10523   127.925       128    0.488011     0.49916
  330      16     10573     10557   127.949       136    0.163388     0.49913
  331      16     10608     10592   127.986       140    0.446794    0.499232
  332      16     10638     10622   127.962       120    0.197402    0.499172
  333      16     10670     10654   127.962       128    0.367944     0.49916
  334      16     10708     10692   128.034       152    0.231909     0.49871
  335      16     10739     10723   128.022       124    0.141185    0.498821
  336      16     10778     10762   128.105       156  0.00940077     0.49852
  337      16     10810     10794   128.104       128    0.261648    0.498504
  338      16     10847     10831   128.163       148    0.124261    0.498099
  339      16     10876     10860   128.127       116    0.164636    0.497995
2025-09-06T18:26:28.595364+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498551
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      15     10910     10895   128.162       140    0.614686    0.498551
  341      16     10942     10926    128.15       124     1.04623    0.498454
  342      16     10975     10959   128.161       132  0.00949285    0.498393
  343      16     11008     10992   128.172       132    0.308079    0.498272
  344      16     11038     11022   128.149       120     1.04958    0.498501
  345      16     11071     11055    128.16       132     1.19549    0.498422
  346      16     11106     11090   128.194       140     0.14117    0.498012
  347      16     11135     11119   128.159       116    0.269025    0.498137
  348      16     11170     11154   128.193       140    0.233673    0.498077
  349      15     11197     11182   128.146       112     1.96743    0.498424
  350      16     11233     11217    128.18       140    0.239946    0.498095
  351      16     11263     11247   128.157       120    0.141775    0.498139
  352      16     11292     11276   128.122       116    0.329235    0.498583
  353      16     11324     11308   128.122       128    0.564327    0.498772
  354      16     11356     11340   128.121       128    0.245307    0.498601
  355      16     11385     11369   128.087       116     0.36941    0.498652
  356      16     11418     11402   128.098       132    0.412727    0.498658
  357      16     11448     11432   128.075       120    0.912643    0.498826
  358      16     11480     11464   128.075       128    0.122428    0.498552
  359      16     11511     11495   128.064       124    0.417578    0.498734
2025-09-06T18:26:48.597498+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498608
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     11542     11526   128.052       124  0.00469837    0.498608
  361      16     11573     11557   128.041       124    0.277154    0.498769
  362      16     11603     11587   128.019       120    0.371347    0.498714
  363      16     11634     11618   128.008       124    0.448797    0.498882
  364      16     11665     11649   127.997       124    0.297185    0.498912
  365      16     11695     11679   127.975       120     0.21292     0.49906
  366      16     11727     11711   127.975       128    0.187083    0.499097
  367      16     11758     11742   127.964       124    0.297044    0.499262
  368      16     11792     11776   127.986       136    0.434425    0.499144
  369      16     11822     11806   127.964       120     0.28158    0.499103
  370      16     11855     11839   127.975       132    0.547389    0.499218
  371      16     11886     11870   127.964       124    0.460247    0.499317
  372      16     11917     11901   127.954       124    0.251769    0.499386
  373      16     11951     11935   127.975       136    0.224885     0.49912
  374      16     11985     11969   127.996       136    0.451716    0.499142
  375      16     12014     11998   127.964       116    0.265198    0.499209
  376      16     12045     12029   127.954       124     1.11776    0.499352
  377      16     12076     12060   127.943       124    0.319043    0.499384
  378      16     12108     12092   127.943       128    0.511691    0.499345
  379      16     12138     12122   127.922       120    0.683308    0.499455
2025-09-06T18:27:08.599777+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499403
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     12172     12156   127.944       136  0.00950364    0.499403
  381      16     12208     12192   127.986       144    0.640739    0.499298
  382      16     12238     12222   127.965       120    0.696683    0.499338
  383      16     12269     12253   127.954       124    0.743393    0.499362
  384      16     12300     12284   127.944       124    0.129444    0.499304
  385      16     12331     12315   127.934       124     0.21787     0.49913
  386      16     12363     12347   127.934       128  0.00414129    0.499346
  387      16     12397     12381   127.955       136  0.00399159    0.499033
  388      16     12436     12420   128.027       156     1.67072    0.498726
  389      16     12468     12452   128.027       128    0.219336     0.49861
  390      16     12500     12484   128.027       128    0.168562    0.498793
  391      16     12530     12514   128.006       120    0.211912    0.499008
  392      16     12564     12548   128.027       136     1.63693    0.498913
  393      16     12594     12578   128.006       120    0.135768    0.498844
  394      16     12627     12611   128.016       132  0.00404634    0.498997
  395      16     12657     12641   127.996       120    0.383596    0.498998
  396      16     12688     12672   127.986       124    0.345043    0.499044
  397      16     12722     12706   128.006       136  0.00957307     0.49891
  398      16     12752     12736   127.986       120     0.84463    0.499185
  399      16     12783     12767   127.976       124     0.13878    0.499246
2025-09-06T18:27:28.601805+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499228
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     12813     12797   127.956       120    0.188823    0.499228
  401      16     12846     12830   127.966       132    0.657651    0.499296
  402      16     12876     12860   127.946       120    0.163316    0.499298
  403      16     12906     12890   127.926       120    0.628978    0.499375
  404      16     12935     12919   127.897       116    0.516876    0.499557
  405      16     12969     12953   127.917       136   0.0104304    0.499375
  406      16     13003     12987   127.937       136    0.297675    0.499425
  407      16     13036     13020   127.946       132  0.00389872    0.499382
  408      16     13069     13053   127.956       132  0.00997506    0.499302
  409      16     13100     13084   127.947       124     1.23969    0.499421
  410      16     13136     13120   127.986       144    0.235194    0.499283
  411      16     13170     13154   128.005       136    0.132806    0.499113
  412      16     13202     13186   128.005       128    0.778281    0.499111
  413      16     13234     13218   128.005       128    0.152167    0.499098
  414      16     13268     13252   128.024       136    0.294547    0.499115
  415      16     13300     13284   128.024       128    0.473965    0.499083
  416      16     13330     13314   128.005       120     1.29628    0.499049
  417      16     13360     13344   127.986       120     1.59456    0.499016
  418      16     13396     13380   128.024       144    0.487929    0.499193
  419      16     13423     13407   127.976       108     1.14104    0.499188
2025-09-06T18:27:48.603881+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499253
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     13453     13437   127.957       120  0.00983536    0.499253
  421      16     13488     13472   127.986       140   0.0036741    0.499188
  422      16     13520     13504   127.986       128     0.71685    0.499059
  423      16     13552     13536   127.986       128     1.31663     0.49923
  424      16     13582     13566   127.967       120     1.26733     0.49928
  425      16     13614     13598   127.967       128    0.251967    0.499233
  426      16     13649     13633   127.995       140    0.163331    0.499073
  427      16     13676     13660   127.948       108    0.231181    0.498919
  428      16     13708     13692   127.948       128    0.304098    0.499214
  429      16     13745     13729   127.995       148   0.0096782    0.499198
  430      16     13774     13758   127.967       116     0.16486    0.499248
  431      16     13803     13787   127.939       116    0.271719    0.499244
  432      16     13838     13822   127.967       140    0.240628      0.4991
  433      16     13866     13850    127.93       112    0.202561    0.499172
  434      16     13899     13883    127.94       132    0.555803    0.499192
  435      16     13932     13916   127.949       132    0.341158    0.499307
  436      16     13960     13944   127.912       112    0.157427    0.499223
  437      16     13993     13977   127.922       132    0.340364     0.49903
  438      16     14023     14007   127.904       120    0.186594    0.499304
  439      16     14056     14040   127.913       132     0.53627    0.499338
2025-09-06T18:28:08.606171+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499297
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     14095     14079   127.977       156     1.04972    0.499297
  441      16     14128     14112   127.986       132    0.250034    0.499277
  442      16     14158     14142   127.968       120     1.13538    0.499214
  443      16     14192     14176   127.986       136   0.0101212    0.499129
  444      16     14223     14207   127.977       124    0.808493    0.499313
  445      16     14261     14245   128.031       152    0.910601    0.499082
  446      16     14296     14280   128.058       140     1.19419    0.499058
  447      16     14326     14310   128.039       120    0.361734    0.499009
  448      16     14356     14340   128.022       120     1.47293    0.499104
  449      16     14384     14368   127.986       112    0.276342    0.499098
  450      16     14419     14403   128.012       140    0.463872    0.499201
  451      16     14453     14437    128.03       136     1.03102    0.499199
  452      16     14481     14465   127.995       112    0.999764    0.499228
  453      16     14512     14496   127.986       124    0.207597    0.499302
  454      16     14539     14523   127.942       108     1.20829    0.499473
  455      16     14576     14560   127.986       148   0.0915569    0.499287
  456      16     14607     14591   127.977       124     0.21471    0.499308
  457      16     14640     14624   127.986       132    0.322493     0.49913
  458      16     14667     14651   127.942       108    0.207927    0.499287
  459      16     14700     14684   127.951       132  0.00461358    0.499235
2025-09-06T18:28:28.608446+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499353
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     14733     14717    127.96       132     0.48331    0.499353
  461      16     14765     14749    127.96       128    0.583011     0.49928
  462      16     14801     14785   127.994       144  0.00314487    0.499246
  463      16     14834     14818   128.003       132  0.00447125      0.4992
  464      16     14868     14852    128.02       136    0.607294    0.499118
  465      16     14903     14887   128.046       140    0.265962    0.498998
  466      16     14935     14919   128.046       128    0.179967    0.499005
  467      16     14968     14952   128.054       132  0.00427903    0.498986
  468      16     14998     14982   128.037       120    0.181298    0.499018
  469      16     15029     15013   128.028       124    0.132746    0.498882
  470      16     15063     15047   128.045       136     1.77387    0.498882
  471      16     15096     15080   128.054       132    0.103948    0.498864
  472      16     15125     15109   128.028       116    0.158716    0.498938
  473      16     15154     15138   128.003       116     0.35774    0.498838
  474      16     15184     15168   127.986       120     0.17648    0.499046
  475      16     15217     15201   127.994       132    0.240458    0.498823
  476      16     15250     15234   128.003       132     1.96903    0.499099
  477      16     15283     15267   128.011       132    0.376752    0.498993
  478      16     15315     15299   128.011       128    0.338292     0.49895
  479      16     15345     15329   127.994       120    0.387207    0.498798
2025-09-06T18:28:48.610480+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498867
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     15377     15361   127.994       128    0.215623    0.498867
  481      16     15412     15396   128.019       140     0.36446    0.498951
  482      15     15446     15431   128.044       140  0.00948299    0.499021
  483      16     15482     15466   128.069       140    0.341337    0.498909
  484      16     15512     15496   128.052       120    0.235792    0.499034
  485      16     15543     15527   128.044       124     0.69055    0.499051
  486      16     15572     15556   128.019       116    0.160791    0.499112
  487      16     15605     15589   128.027       132    0.501422    0.499121
  488      16     15641     15625    128.06       144     1.01104    0.498938
  489      15     15672     15657   128.059       128     1.81597     0.49895
  490      16     15702     15686   128.035       116    0.205763    0.499069
  491      16     15733     15717   128.027       124    0.284103    0.499165
  492      16     15769     15753   128.059       144  0.00945749    0.498934
  493      16     15806     15790   128.099       148    0.814693    0.498897
  494      16     15839     15823   128.107       132    0.343132    0.498828
  495      16     15870     15854   128.099       124  0.00992425    0.498864
  496      16     15901     15885   128.091       124    0.884654    0.498903
  497      16     15932     15916   128.082       124    0.497856    0.498763
  498      16     15966     15950   128.098       136    0.161133     0.49867
  499      16     16000     15984   128.114       136     0.25807    0.498577
2025-09-06T18:29:08.612927+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498654
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     16030     16014   128.098       120     1.88328    0.498654
  501      16     16063     16047   128.106       132  0.00955982    0.498913
  502      16     16092     16076   128.081       116    0.605273    0.498912
  503      16     16122     16106   128.065       120  0.00423439     0.49886
  504      16     16158     16142   128.097       144    0.245854    0.498815
  505      16     16189     16173   128.089       124    0.380209    0.498838
  506      16     16224     16208   128.112       140    0.596863    0.498791
  507      16     16255     16239   128.104       124    0.159978      0.4988
  508      16     16285     16269   128.088       120    0.526997    0.498869
  509      16     16320     16304   128.112       140    0.258843    0.498899
  510      16     16353     16337   128.119       132     1.15565    0.498675
  511      16     16382     16366   128.095       116  0.00555467    0.498588
  512      16     16416     16400   128.111       136    0.160095    0.498614
  513      16     16445     16429   128.087       116    0.251778    0.498571
  514      16     16477     16461   128.087       128    0.479525    0.498749
  515      16     16507     16491   128.071       120    0.139536    0.498782
  516      15     16537     16522   128.063       124   0.0053727    0.499071
  517      16     16578     16562   128.125       160     0.73311    0.498812
  518      16     16609     16593   128.117       124     1.10743    0.498809
  519      16     16640     16624   128.109       124    0.961275    0.498904
2025-09-06T18:29:28.615254+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.4989
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     16667     16651    128.07       108     1.19303      0.4989
  521      16     16701     16685   128.086       136     0.17635    0.498787
  522      16     16740     16724   128.139       156    0.336797    0.498659
  523      16     16772     16756   128.139       128   0.0039458    0.498658
  524      16     16804     16788   128.138       128  0.00398655    0.498685
  525      16     16839     16823   128.161       140     1.57302    0.498638
  526      16     16867     16851    128.13       112     1.49185    0.498578
  527      16     16900     16884   128.138       132     1.93699    0.498574
  528      16     16938     16922   128.183       152    0.786089    0.498526
  529      16     16971     16955    128.19       132    0.989095    0.498551
  530      16     17000     16984   128.167       116    0.237529    0.498528
  531      16     17037     17021   128.204       148    0.750842    0.498512
  532      16     17071     17055   128.219       136    0.521145    0.498393
  533      16     17101     17085   128.203       120    0.721035    0.498481
  534      16     17137     17121   128.233       144    0.719929    0.498413
  535      16     17171     17155   128.247       136    0.850239    0.498371
  536      16     17210     17194   128.299       156    0.354278    0.498126
  537      16     17242     17226   128.299       128    0.721432    0.498143
  538      16     17272     17256   128.283       120     1.16379    0.498125
  539      16     17309     17293    128.32       148     1.17337    0.498027
2025-09-06T18:29:48.617390+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.497789
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     17342     17326   128.326       132    0.416346    0.497789
  541      16     17371     17355   128.304       116     1.70174    0.498046
  542      16     17407     17391   128.333       144    0.267477    0.498024
  543      16     17437     17421   128.317       120    0.410465    0.497982
  544      16     17467     17451   128.302       120    0.523432    0.498179
  545      16     17498     17482   128.294       124    0.868987    0.498095
  546      16     17527     17511   128.271       116    0.286386    0.498133
  547      16     17556     17540   128.249       116    0.415641    0.498187
  548      16     17586     17570   128.234       120    0.214532    0.498338
  549      16     17624     17608   128.277       152    0.185196    0.498253
  550      16     17657     17641   128.284       132    0.166757    0.498041
  551      16     17687     17671   128.269       120    0.305845    0.498131
  552      16     17717     17701   128.254       120    0.252178     0.49813
  553      16     17750     17734   128.261       132    0.200912    0.498256
  554      16     17783     17767   128.267       132    0.524836    0.498312
  555      16     17813     17797   128.252       120    0.155246    0.498354
  556      16     17841     17825   128.223       112     0.97922    0.498394
  557      16     17874     17858    128.23       132     1.01854    0.498344
  558      16     17906     17890    128.23       128    0.609245    0.498393
  559      16     17937     17921   128.222       124     1.39075    0.498448
2025-09-06T18:30:08.619595+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498231
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      15     17971     17956   128.243       140     0.51043    0.498231
  561      16     18005     17989    128.25       132    0.587466    0.498307
  562      16     18036     18020   128.242       124     1.04433    0.498354
  563      16     18068     18052   128.242       128    0.389894    0.498337
  564      16     18100     18084   128.241       128     1.32011    0.498308
  565      16     18131     18115   128.234       124    0.437961    0.498223
  566      16     18163     18147   128.233       128    0.205272    0.498271
  567      16     18198     18182   128.254       140    0.304688    0.498214
  568      16     18226     18210   128.225       112    0.426578    0.498234
  569      16     18257     18241   128.218       124    0.241116    0.498225
  570      16     18293     18277   128.245       144  0.00453969    0.498237
  571      16     18321     18305   128.217       112     1.56114    0.498368
  572      16     18350     18334   128.196       116  0.00566492      0.4984
  573      16     18384     18368   128.209       136    0.249675    0.498398
  574      16     18417     18401   128.216       132    0.194271    0.498352
  575      16     18453     18437   128.243       144    0.295329    0.498282
  576      16     18487     18471   128.257       136     1.56923     0.49822
  577      16     18517     18501   128.242       120     1.73318    0.498215
  578      16     18549     18533   128.242       128     0.43951    0.498337
  579      16     18579     18563   128.228       120    0.212889    0.498329
2025-09-06T18:30:28.621670+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498306
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     18612     18596   128.234       132     1.70987    0.498306
  581      16     18642     18626    128.22       120  0.00404959    0.498321
  582      16     18674     18658   128.219       128    0.260859    0.498406
  583      16     18703     18687   128.199       116    0.168831    0.498363
  584      16     18733     18717   128.184       120    0.232812    0.498313
  585      16     18765     18749   128.184       128    0.257978    0.498367
  586      16     18799     18783   128.197       136      1.7413    0.498373
  587      16     18836     18820   128.231       148    0.172934    0.498331
  588      16     18870     18854   128.244       136    0.222848    0.498373
  589      16     18903     18887   128.251       132   0.0107021    0.498279
  590      16     18933     18917   128.237       120    0.205191    0.498261
  591      16     18973     18957    128.29       160  0.00932829    0.498098
  592      16     19005     18989    128.29       128    0.919792    0.498197
  593      16     19038     19022   128.296       132    0.193526    0.498153
  594      16     19072     19056   128.309       136    0.641516    0.498005
  595      16     19105     19089   128.315       132    0.388474    0.497945
  596      16     19140     19124   128.335       140    0.265263    0.497798
  597      16     19174     19158   128.348       136    0.347531    0.497679
  598      16     19205     19189    128.34       124  0.00468298    0.497917
  599      16     19234     19218    128.32       116    0.131096    0.497891
2025-09-06T18:30:48.623535+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.497848
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     19266     19250   128.319       128     0.20638    0.497848
  601      16     19298     19282   128.319       128     0.30307    0.497889
  602      16     19328     19312   128.305       120     0.69229    0.498052
  603      16     19361     19345   128.311       132    0.966773    0.498166
  604      16     19391     19375   128.297       120    0.628106    0.498154
  605      16     19424     19408   128.303       132    0.581785    0.498153
  606      16     19453     19437   128.283       116    0.245653    0.498085
  607      16     19488     19472   128.302       140    0.514374    0.498159
  608      16     19516     19500   128.275       112    0.165192    0.498207
  609      16     19548     19532   128.275       128     0.18291    0.498116
  610      16     19579     19563   128.268       124    0.487579       0.498
  611      16     19612     19596   128.274       132    0.350558     0.49813
  612      16     19648     19632     128.3       144     1.50344    0.498196
  613      15     19680     19665   128.306       132    0.497726    0.498174
  614      16     19712     19696   128.299       124    0.387109    0.498147
  615      16     19744     19728   128.298       128    0.654766    0.498222
  616      16     19777     19761   128.304       132    0.347097    0.498135
  617      16     19810     19794    128.31       132  0.00384199    0.498112
  618      16     19841     19825   128.303       124  0.00924664     0.49821
  619      16     19874     19858   128.309       132    0.317753    0.498164
2025-09-06T18:31:08.625549+0000 min lat: 0.00154492 max lat: 2.50772 avg lat: 0.498133
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  620      16     19904     19888   128.296       120    0.476462    0.498133
  621      16     19942     19926   128.334       152    0.742838    0.497973
  622      16     19973     19957   128.327       124     1.53709    0.498103
  623      16     20003     19987   128.313       120     1.31341    0.498049
  624      16     20032     20016   128.294       116    0.370344    0.498175
  625      16     20063     20047   128.287       124  0.00488616    0.498207
  626      16     20094     20078    128.28       124    0.370828    0.498267
  627      16     20124     20108   128.267       120    0.889991    0.498301
  628      16     20161     20145   128.298       148    0.407013    0.498236
  629      16     20193     20177   128.298       128   0.0045005    0.498117
  630      16     20226     20210   128.303       132    0.325955    0.498051
  631      16     20257     20241   128.297       124    0.246252    0.498093
  632      16     20288     20272    128.29       124      1.5508    0.498095
  633      16     20319     20303   128.283       124    0.261586    0.498035
  634      16     20347     20331   128.257       112    0.135072    0.498087
  635      16     20379     20363   128.257       128    0.224209    0.498247
  636      16     20411     20395   128.256       128  0.00320141    0.498235
  637      16     20443     20427   128.256       128     0.10677    0.498117
  638      16     20475     20459   128.256       128  0.00424576    0.498092
  639      16     20513     20497   128.293       152  0.00409179    0.498073
2025-09-06T18:31:28.627678+0000 min lat: 0.00154492 max lat: 2.50772 avg lat: 0.498002
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  640      16     20547     20531   128.305       136    0.217224    0.498002
  641      16     20577     20561   128.292       120  0.00941277    0.498134
  642      16     20612     20596    128.31       140    0.849566    0.498099
  643      15     20643     20628   128.309       128     1.11592     0.49816
  644      16     20681     20665    128.34       148    0.303034    0.497779
  645      16     20714     20698   128.346       132  0.00453328    0.497771
  646      16     20745     20729   128.339       124    0.162846    0.497758
  647      16     20776     20760   128.332       124    0.201974    0.497765
  648      16     20806     20790   128.319       120    0.270447    0.497867
  649      16     20841     20825   128.337       140    0.179079    0.497884
  650      16     20871     20855   128.324       120    0.178051    0.497934
  651      16     20907     20891   128.348       144    0.707323    0.497984
  652      16     20937     20921   128.336       120    0.200974    0.498017
  653      16     20969     20953   128.335       128     1.04984    0.497978
  654      15     21002     20987   128.347       136    0.332524    0.497963
  655      16     21036     21020   128.352       132  0.00292492    0.497902
  656      16     21065     21049   128.333       116    0.138455    0.497928
  657      16     21094     21078   128.315       116     0.13389    0.498032
  658      16     21128     21112   128.326       136    0.350728    0.498019
  659      16     21159     21143    128.32       124    0.206449       0.498
2025-09-06T18:31:48.630164+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.498019
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  660      16     21194     21178   128.337       140      0.1939    0.498019
  661      16     21228     21212   128.349       136    0.981908    0.498026
  662      16     21259     21243   128.342       124     0.60983    0.498078
  663      16     21293     21277   128.354       136    0.625971    0.498006
  664      16     21328     21312   128.371       140  0.00388638    0.497931
  665      16     21359     21343   128.365       124    0.206629    0.497897
  666      16     21398     21382   128.406       156     1.15015    0.497804
  667      16     21430     21414   128.406       128    0.294225    0.497747
  668      16     21461     21445   128.399       124    0.221351    0.497772
  669      16     21494     21478   128.404       132     1.10576    0.497805
  670      16     21522     21506    128.38       112     1.37844    0.497798
  671      16     21560     21544   128.415       152    0.137995    0.497606
  672      15     21591     21576   128.414       128  0.00435402    0.497663
  673      16     21622     21606   128.402       120     1.67342    0.497771
  674      16     21652     21636   128.389       120    0.340688    0.497862
  675      16     21689     21673   128.418       148    0.468005    0.497709
  676      16     21719     21703   128.406       120    0.985812    0.497749
  677      16     21749     21733   128.394       120  0.00462884    0.497805
  678      16     21782     21766   128.399       132    0.459022    0.497786
  679      16     21817     21801   128.416       140  0.00308075    0.497764
2025-09-06T18:32:08.631990+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497834
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  680      16     21847     21831   128.404       120    0.595554    0.497834
  681      16     21883     21867   128.426       144    0.231523    0.497739
  682      16     21915     21899   128.426       128     0.26586    0.497685
  683      16     21948     21932   128.431       132     1.61398    0.497591
  684      16     21978     21962   128.419       120    0.237449    0.497609
  685      16     22011     21995   128.424       132    0.317694    0.497474
  686      16     22045     22029   128.435       136     2.29904     0.49756
  687      16     22078     22062    128.44       132     1.47621    0.497654
  688      16     22105     22089    128.41       108    0.333333    0.497719
  689      16     22140     22124   128.427       140   0.0102772    0.497606
  690      16     22169     22153   128.409       116    0.268968    0.497666
  691      16     22202     22186   128.414       132    0.316142    0.497792
  692      16     22233     22217   128.408       124    0.531404    0.497859
  693      16     22262     22246    128.39       116    0.284596    0.497801
  694      16     22300     22284   128.424       152  0.00974954    0.497683
  695      16     22331     22315   128.418       124    0.878116    0.497741
  696      16     22365     22349   128.428       136  0.00411101    0.497582
  697      16     22397     22381   128.428       128     0.67671    0.497591
  698      16     22431     22415   128.439       136    0.135415    0.497601
  699      16     22466     22450   128.455       140  0.00432351    0.497468
2025-09-06T18:32:28.634440+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.4975
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  700      16     22495     22479   128.437       116  0.00410331      0.4975
  701      16     22529     22513   128.448       136  0.00340543    0.497427
  702      16     22559     22543   128.436       120    0.584921    0.497398
  703      16     22592     22576   128.441       132    0.288112    0.497582
  704      16     22624     22608    128.44       128    0.464197    0.497563
  705      16     22655     22639   128.434       124    0.249765    0.497639
  706      16     22689     22673   128.445       136     1.40698    0.497634
  707      16     22723     22707   128.455       136    0.912648    0.497608
  708      16     22753     22737   128.443       120     1.14278    0.497607
  709      16     22787     22771   128.454       136    0.274615    0.497489
  710      16     22820     22804   128.459       132     1.52785    0.497561
  711      16     22852     22836   128.458       128    0.143194    0.497524
  712      16     22882     22866   128.447       120   0.0108797    0.497514
  713      16     22916     22900   128.457       136  0.00969172    0.497626
  714      16     22944     22928   128.434       112    0.286761    0.497661
  715      16     22981     22965   128.461       148    0.202999    0.497517
  716      16     23014     22998   128.466       132    0.985751    0.497543
  717      16     23045     23029    128.46       124  0.00401166    0.497519
  718      16     23079     23063   128.471       136      1.0441    0.497587
  719      16     23109     23093   128.459       120    0.340397    0.497575
2025-09-06T18:32:48.636512+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497535
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  720      16     23140     23124   128.453       124    0.348939    0.497535
  721      16     23175     23159   128.469       140  0.00954191    0.497543
  722      16     23207     23191   128.468       128    0.141118     0.49741
  723      16     23240     23224   128.473       132    0.486684     0.49746
  724      16     23272     23256   128.472       128    0.246206    0.497543
  725      16     23303     23287   128.466       124     1.46967    0.497569
  726      16     23332     23316   128.449       116    0.210593    0.497661
  727      16     23361     23345   128.432       116    0.861538    0.497705
  728      16     23394     23378   128.436       132    0.247252    0.497672
  729      16     23433     23417   128.474       156    0.194794    0.497571
  730      16     23463     23447   128.463       120     1.26856    0.497535
  731      16     23496     23480   128.467       132    0.348161    0.497462
  732      16     23530     23514   128.478       136    0.330198    0.497413
  733      16     23561     23545   128.472       124     0.13224    0.497436
  734      16     23594     23578   128.476       132     1.69742    0.497318
  735      16     23625     23609    128.47       124    0.335757    0.497459
  736      16     23654     23638   128.453       116    0.311531    0.497478
  737      16     23686     23670   128.453       128    0.334671    0.497574
  738      16     23720     23704   128.463       136    0.275606    0.497476
  739      16     23753     23737   128.468       132     1.65541    0.497506
2025-09-06T18:33:08.638569+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497496
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  740      16     23783     23767   128.456       120    0.263774    0.497496
  741      16     23818     23802   128.472       140    0.110223    0.497351
  742      16     23847     23831   128.455       116    0.673877    0.497485
  743      16     23878     23862   128.449       124     1.80926    0.497481
  744      16     23914     23898    128.47       144      0.1978    0.497421
  745      16     23950     23934   128.491       144  0.00333044    0.497357
  746      16     23978     23962   128.468       112     1.77292    0.497421
  747      16     24011     23995   128.473       132    0.331108    0.497488
  748      16     24038     24022   128.446       108    0.245639    0.497612
  749      16     24073     24057   128.461       140    0.582493    0.497608
  750      16     24102     24086   128.445       116    0.144918    0.497601
  751      16     24136     24120   128.455       136  0.00415808    0.497652
  752      16     24167     24151   128.449       124    0.291723     0.49767
  753      16     24199     24183   128.448       128     1.23694    0.497701
  754      16     24228     24212   128.432       116    0.263751    0.497633
  755      16     24257     24241   128.415       116     0.52714    0.497729
  756      16     24296     24280   128.452       156  0.00388412     0.49759
  757      16     24329     24313   128.456       132  0.00396201    0.497566
  758      16     24361     24345   128.456       128    0.232989    0.497541
  759      16     24395     24379   128.465       136     1.53118     0.49745
2025-09-06T18:33:28.640745+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.49743
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  760      16     24430     24414   128.481       140    0.240701     0.49743
  761      16     24465     24449   128.496       140  0.00402037    0.497414
  762      16     24492     24476   128.469       108    0.220131    0.497428
  763      16     24524     24508   128.468       128  0.00948882    0.497506
  764      16     24561     24545   128.494       148    0.438075    0.497412
  765      16     24592     24576   128.488       124   0.0106415    0.497388
  766      16     24621     24605   128.472       116    0.540332    0.497383
  767      16     24650     24634   128.455       116    0.322405    0.497409
  768      16     24685     24669    128.47       140    0.671611    0.497507
  769      16     24718     24702   128.475       132    0.103512    0.497564
  770      16     24750     24734   128.474       128     1.23708    0.497494
  771      16     24783     24767   128.479       132  0.00435058    0.497537
  772      16     24814     24798   128.473       124    0.188154    0.497412
  773      16     24849     24833   128.488       140    0.152624    0.497419
  774      16     24882     24866   128.492       132     0.41205    0.497438
  775      16     24916     24900   128.502       136    0.545199    0.497474
  776      16     24947     24931   128.496       124    0.745055    0.497489
  777      16     24980     24964   128.501       132     1.27927    0.497428
  778      16     25013     24997   128.505       132    0.375195     0.49743
  779      16     25046     25030    128.51       132    0.323529    0.497435
2025-09-06T18:33:48.642586+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497426
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  780      16     25076     25060   128.499       120    0.164765    0.497426
  781      16     25107     25091   128.493       124    0.458684    0.497504
  782      16     25137     25121   128.482       120   0.0968037    0.497387
  783      16     25172     25156   128.497       140     1.36813    0.497426
  784      16     25204     25188   128.496       128    0.205059    0.497343
  785      16     25234     25218   128.485       120     1.64822    0.497457
  786      16     25264     25248   128.475       120    0.570694     0.49751
  787      16     25295     25279   128.469       124    0.428104    0.497558
  788      16     25329     25313   128.478       136    0.168965    0.497425
  789      16     25363     25347   128.488       136    0.236022    0.497381
  790      16     25395     25379   128.487       128     1.27283    0.497491
  791      16     25426     25410   128.482       124    0.182008    0.497465
  792      16     25456     25440   128.471       120     1.41367    0.497491
  793      16     25486     25470    128.46       120     1.28479    0.497525
  794      16     25522     25506    128.48       144  0.00923601    0.497468
  795      16     25550     25534   128.459       112    0.207361    0.497626
  796      16     25580     25564   128.448       120    0.553007    0.497641
  797      16     25614     25598   128.458       136     0.19253    0.497569
  798      16     25644     25628   128.447       120    0.562583    0.497637
  799      16     25678     25662   128.457       136    0.294649    0.497624
2025-09-06T18:34:08.644434+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497643
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  800      16     25710     25694   128.456       128    0.154346    0.497643
  801      16     25740     25724   128.445       120    0.212227    0.497573
  802      16     25768     25752   128.425       112    0.165633    0.497637
  803      16     25802     25786   128.434       136    0.429793    0.497596
  804      16     25836     25820   128.444       136    0.278389    0.497486
  805      16     25867     25851   128.438       124    0.144481     0.49741
  806      16     25901     25885   128.448       136    0.196789    0.497476
  807      16     25935     25919   128.457       136    0.108006    0.497483
  808      16     25967     25951   128.456       128    0.279805    0.497444
  809      16     25996     25980   128.441       116    0.314308    0.497563
  810      16     26034     26018    128.47       152     0.43413     0.49754
  811      16     26067     26051   128.474       132    0.687926    0.497588
  812      16     26097     26081   128.464       120    0.625098    0.497609
  813      16     26126     26110   128.449       116    0.207972    0.497607
  814      16     26159     26143   128.453       132  0.00996344    0.497677
  815      16     26189     26173   128.442       120    0.204718    0.497673
  816      16     26222     26206   128.447       132     1.44419    0.497655
  817      16     26254     26238   128.446       128     1.28227    0.497568
  818      16     26284     26268   128.436       120    0.530232    0.497701
  819      16     26311     26295   128.411       108    0.229182    0.497704
2025-09-06T18:34:28.646486+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497788
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  820      16     26342     26326   128.406       124    0.010523    0.497788
  821      16     26371     26355    128.39       116    0.262661    0.497859
  822      16     26402     26386   128.385       124    0.208996    0.497876
  823      16     26435     26419   128.389       132    0.588718    0.497848
  824      16     26465     26449   128.379       120   0.0042696    0.497911
  825      16     26496     26480   128.374       124    0.809936    0.497971
  826      16     26532     26516   128.393       144    0.256264    0.497897
  827      16     26561     26545   128.378       116     0.37621    0.497934
  828      16     26595     26579   128.387       136    0.665159    0.497911
  829      16     26623     26607   128.367       112    0.358751    0.497999
  830      16     26655     26639   128.367       128    0.320437    0.497905
  831      16     26691     26675   128.386       144    0.144223    0.497904
  832      16     26725     26709   128.395       136    0.543458    0.497831
  833      16     26765     26749   128.433       160  0.00432568     0.49769
  834      16     26794     26778   128.418       116    0.720505    0.497772
  835      16     26826     26810   128.417       128    0.289933    0.497783
  836      16     26857     26841   128.412       124    0.304871    0.497837
  837      16     26886     26870   128.397       116    0.137003    0.497788
  838      16     26920     26904   128.406       136   0.0095787    0.497795
  839      16     26955     26939    128.42       140    0.122133    0.497686
2025-09-06T18:34:48.648507+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497682
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  840      16     26989     26973   128.429       136    0.494781    0.497682
  841      16     27020     27004   128.424       124     1.14522    0.497765
  842      16     27052     27036   128.423       128    0.121572    0.497678
  843      16     27085     27069   128.427       132     1.78955    0.497608
  844      16     27117     27101   128.427       128     0.21996    0.497529
  845      16     27151     27135   128.436       136    0.152632    0.497576
  846      16     27183     27167   128.435       128    0.106062    0.497597
  847      16     27214     27198    128.43       124    0.227882    0.497602
  848      16     27246     27230   128.429       128    0.379913    0.497665
  849      16     27281     27265   128.443       140    0.541272    0.497682
  850      16     27312     27296   128.438       124     1.13475    0.497745
  851      16     27341     27325   128.423       116    0.147037    0.497718
  852      16     27371     27355   128.413       120    0.416081    0.497783
  853      16     27401     27385   128.403       120    0.304653    0.497831
  854      16     27434     27418   128.408       132     0.91439    0.497827
  855      16     27470     27454   128.426       144   0.0043662    0.497798
  856      16     27500     27484   128.416       120    0.224426    0.497762
  857      16     27531     27515   128.411       124    0.640894    0.497759
  858      16     27560     27544   128.396       116    0.509372    0.497834
  859      16     27593     27577   128.401       132    0.194444    0.497801
2025-09-06T18:35:08.650646+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497796
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  860      16     27629     27613   128.419       144     1.11625    0.497796
  861      16     27659     27643   128.409       120    0.483542    0.497829
  862      16     27687     27671    128.39       112    0.124585    0.497871
  863      16     27717     27701    128.38       120     0.91981    0.497903
  864      16     27749     27733    128.38       128    0.755253    0.497951
  865      16     27784     27768   128.393       140    0.345558    0.497904
  866      16     27820     27804   128.411       144    0.280595    0.497664
  867      16     27853     27837   128.415       132    0.449125    0.497641
  868      16     27882     27866   128.401       116    0.276461    0.497804
  869      16     27914     27898     128.4       128    0.280504    0.497851
  870      16     27945     27929   128.395       124    0.208588    0.497865
  871      16     27977     27961   128.395       128    0.322778    0.497902
  872      16     28011     27995   128.403       136    0.672454    0.497885
  873      16     28040     28024   128.389       116    0.177578    0.497812
  874      16     28074     28058   128.398       136    0.399873    0.497765
  875      16     28107     28091   128.402       132  0.00451468    0.497827
  876      16     28139     28123   128.402       128     1.02589    0.497839
  877      16     28173     28157    128.41       136    0.195671    0.497789
  878      16     28206     28190   128.414       132    0.497341    0.497751
  879      16     28238     28222   128.414       128    0.164813    0.497765
2025-09-06T18:35:28.653034+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497636
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  880      16     28270     28254   128.413       128    0.158231    0.497636
  881      16     28300     28284   128.404       120    0.222336    0.497723
  882      16     28331     28315   128.399       124    0.485964    0.497786
  883      16     28365     28349   128.407       136    0.187136    0.497715
  884      16     28398     28382   128.411       132  0.00931851    0.497751
  885      16     28431     28415   128.415       132    0.587317    0.497827
  886      16     28465     28449   128.424       136    0.263969    0.497762
  887      16     28494     28478    128.41       116    0.721625    0.497851
  888      16     28530     28514   128.427       144     1.07711    0.497782
  889      16     28562     28546   128.427       128    0.187841    0.497687
  890      16     28593     28577   128.422       124    0.935895    0.497767
  891      16     28627     28611    128.43       136    0.334052    0.497727
  892      16     28659     28643    128.43       128    0.358597    0.497715
  893      16     28691     28675   128.429       128    0.511986    0.497763
  894      16     28722     28706   128.425       124    0.202658    0.497784
  895      16     28756     28740   128.433       136  0.00463247    0.497765
  896      16     28784     28768   128.415       112     0.27957    0.497785
  897      16     28817     28801   128.419       132    0.301228    0.497737
  898      16     28850     28834   128.423       132    0.286105    0.497773
  899      16     28881     28865   128.418       124     0.36792    0.497791
2025-09-06T18:35:48.655233+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497846
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  900      11     28903     28892   128.395       108    0.838115    0.497846
Total time run:       900.373
Total reads made:     28903
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   128.405
Average IOPS:         32
Stddev IOPS:          2.54888
Max IOPS:             42
Min IOPS:             25
Average Latency(s):   0.497946
Max latency(s):       2.54975
Min latency(s):       0.00154492

Rados, cleanup

[ceph: root@ms-01-01 /]# time rados -p testbench cleanup
Removed 28947 objects

real    0m8.134s
user    0m0.782s
sys     0m0.628s

RBD, test setup

[ceph: root@ms-01-01 /]# ceph osd pool create rbdbench 32
pool 'rbdbench' created
[ceph: root@ms-01-01 /]# ceph osd pool application enable rbdbench rbd
enabled application 'rbd' on pool 'rbdbench'
[ceph: root@ms-01-01 /]# rbd pool init -p  rbdbench
[ceph: root@ms-01-01 /]# rbd create image1 --size 1024 --pool rbdbench

RBD, how to pick values

[ceph: root@ms-01-01 /]# rbd help bench
usage: rbd bench [--pool <pool>] [--namespace <namespace>] [--image <image>] 
                 [--io-size <io-size>] [--io-threads <io-threads>] 
                 [--io-total <io-total>] [--io-pattern <io-pattern>] 
                 [--rw-mix-read <rw-mix-read>] --io-type <io-type> 
                 <image-spec> 

Simple benchmark.

Positional arguments
  <image-spec>         image specification
                       (example: [<pool-name>/[<namespace>/]]<image-name>)

Optional arguments
  -p [ --pool ] arg    pool name
  --namespace arg      namespace name
  --image arg          image name
  --io-size arg        IO size (in B/K/M/G) (< 4G) [default: 4K]
  --io-threads arg     ios in flight [default: 16]
  --io-total arg       total size for IO (in B/K/M/G/T) [default: 1G]
  --io-pattern arg     IO pattern (rand, seq, or full-seq) [default: seq]
  --rw-mix-read arg    read proportion in readwrite (<= 100) [default: 50]
  --io-type arg        IO type (read, write, or readwrite(rw))

RBD, 4k size, random RW, 80% read, 16 parallel IO threads

[ceph: root@ms-01-01 /]# rbd bench \
                         --pool=rbdbench \
                         --image image1 \
                         --io-total 5G \
                         --io-pattern rand \
                         --rw-mix-read 80 \
                         --io-type rw
bench  type readwrite read:write=80:20 io_size 4096 io_threads 16 bytes 5368709120 pattern random
[]
Click tro see full run
[ceph: root@ms-01-01 /]# rbd bench \
>                        --pool=rbdbench \
>                        --image image1 \
>                        --io-total 5G \
>                        --io-pattern rand \
>                        --rw-mix-read 80 \
>                        --io-type rw
bench  type readwrite read:write=80:20 io_size 4096 io_threads 16 bytes 5368709120 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1     21968   22004.6    86 MiB/s
    2     44656   22345.8    87 MiB/s
    3     67744   22592.8    88 MiB/s
    4     91392   22856.3    89 MiB/s
    5    113296   22665.5    89 MiB/s
    6    134320     22469    88 MiB/s
    7    155408     22149    87 MiB/s
    8    177088   21867.4    85 MiB/s
    9    198544     21429    84 MiB/s
   10    219680   21275.5    83 MiB/s
   11    240544   21243.5    83 MiB/s
   12    262128   21342.6    83 MiB/s
   13    283568   21294.6    83 MiB/s
   14    304288   21147.5    83 MiB/s
   15    326112   21285.1    83 MiB/s
   16    347344   21362.9    83 MiB/s
   17    368048   21186.9    83 MiB/s
   18    389120   21100.6    82 MiB/s
   19    410576   21260.5    83 MiB/s
   20    431776   21135.7    83 MiB/s
   21    453376   21205.1    83 MiB/s
   22    475184   21421.6    84 MiB/s
   23    496432   21473.9    84 MiB/s
   24    517952   21473.8    84 MiB/s
   25    539056   21454.6    84 MiB/s
   26    560192   21361.8    83 MiB/s
   27    581264   21218.9    83 MiB/s
   28    602288   21169.9    83 MiB/s
   29    623376   21083.5    82 MiB/s
   30    644288   21045.1    82 MiB/s
   31    665344   21029.1    82 MiB/s
   32    686304   21010.9    82 MiB/s
   33    707568   21058.9    82 MiB/s
   34    728480   21023.7    82 MiB/s
   35    749616   21068.5    82 MiB/s
   36    770944   21122.9    83 MiB/s
   37    792560   21249.9    83 MiB/s
   38    814576   21400.2    84 MiB/s
   39    835968   21496.2    84 MiB/s
   40    857008     21477    84 MiB/s
   41    878208   21451.4    84 MiB/s
   42    899392     21365    83 MiB/s
   43    920736   21230.7    83 MiB/s
   44    942336   21272.3    83 MiB/s
   45    964000     21397    84 MiB/s
   46    984912   21339.4    83 MiB/s
   47   1006160   21352.2    83 MiB/s
   48   1027440   21343.7    83 MiB/s
   49   1049104   21356.5    83 MiB/s
   50   1070288   21256.3    83 MiB/s
   51   1091760   21368.2    83 MiB/s
   52   1114224   21615.8    84 MiB/s
   53   1135280   21566.6    84 MiB/s
   54   1156336     21445    84 MiB/s
   55   1177696   21484.5    84 MiB/s
   56   1198768   21404.5    84 MiB/s
   57   1219856   21125.1    83 MiB/s
   58   1241360   21214.7    83 MiB/s
   59   1262592   21249.9    83 MiB/s
   60   1284000   21259.5    83 MiB/s
   61   1305008   21246.7    83 MiB/s
elapsed: 61   ops: 1310720   ops/sec: 21388   bytes/sec: 84 MiB/s
read_ops: 1048933   read_ops/sec: 17116.2   read_bytes/sec: 67 MiB/s
write_ops: 261787   write_ops/sec: 4271.78   write_bytes/sec: 17 MiB/s

RBD, celanup

[ceph: root@ms-01-01 /]# rbd remove image1 --pool rbdbench
Removing image: 100% complete...done.

Current state

2025-09-06 late evening

[ceph: root@ms-01-01 /]# ceph health detail
HEALTH_OK
[ceph: root@ms-01-01 /]# ceph -s
  cluster:
    id:     cee8622c-fb45-11ef-9938-5847ca797c06
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ms-01-01,ms-01-02,ms-01-03 (age 11d)
    mgr: ms-01-01.pnupxa(active, since 11d), standbys: ms-01-02.yuzhvl, ms-01-03.ujufvv
    mds: 1/1 daemons up, 1 standby
    osd: 10 osds: 10 up (since 8d), 10 in (since 8d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 337 pgs
    objects: 28 objects, 8.5 MiB
    usage:   7.8 GiB used, 18 TiB / 18 TiB avail
    pgs:     337 active+clean
 
[ceph: root@ms-01-01 /]# ceph df
--- RAW STORAGE ---
CLASS    SIZE   AVAIL     USED  RAW USED  %RAW USED
ssd    18 TiB  18 TiB  7.8 GiB   7.8 GiB       0.04
TOTAL  18 TiB  18 TiB  7.8 GiB   7.8 GiB       0.04
 
--- POOLS ---
POOL                ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr                 1    1   16 MiB        3   48 MiB      0    5.8 TiB
testbench            2   32      0 B        0      0 B      0    5.8 TiB
rbdbench             3   32  9.4 KiB        3   40 KiB      0    5.8 TiB
cephfs.cephfs.meta   4   16   56 KiB       22  260 KiB      0    5.8 TiB
cephfs.cephfs.data   5  256      0 B        0      0 B      0    5.8 TiB
[ceph: root@ms-01-01 /]# 
[ceph: root@ms-01-01 /]# ceph orch host ls 
HOST                       ADDR            LABELS              STATUS  
ms-01-01.storage.pcfe.net  192.168.40.151  _admin,mgr,mon,osd          
ms-01-02.storage.pcfe.net  192.168.40.152  _admin,mgr,mon,osd          
ms-01-03.storage.pcfe.net  192.168.40.153  _admin,mgr,mon,osd          
ms-01-04.storage.pcfe.net  192.168.40.154  _admin,osd,mon,mds          
ms-01-05.storage.pcfe.net  192.168.40.155  _admin,osd,mon,mds          
5 hosts in cluster

Next up